[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: Split Flushing to filter command
From: |
Pádraig Brady |
Subject: |
Re: Split Flushing to filter command |
Date: |
Thu, 20 Mar 2014 10:11:01 +0000 |
User-agent: |
Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130110 Thunderbird/17.0.2 |
On 03/20/2014 02:39 AM, Eldon wrote:
> Hello all,
> I am attempting the following in a bash shell on a 3.8.13 linux kernel:
> sudo tcpdump -nn
> |grep --line-buffered NTPv2
> |split -u --lines=10 --filter=date
>
> Clearly date would be replaced with some more useful script, but for the
> mean time I am trying to use it to debug what I see as unexpected
> buffering. Since the traffic I am looking at is fairly consistent (when
> I pipe to cat instead, I see a steady stream), I would expect to see
> regular ticks as date executions each time 10 lines are sent to
> split. Instead I see flashes of many executions that seem to me to be a
> buffer flush. I looked in the code in the git repo, and it seems that
> the buffer is in fact filled prior to execution. Is it worth making a
> patch or expanding the meaning of -u to pass this through in an
> unbuffered fashion? Would there be a reason to reject such a patch if
> well-formed?
>
> Thoughts?
split(1) doesn't use stdio so we're not hitting that buffering.
What's happening is that we're explicitly buffering input
(into a 64K buffer on my x86_64 system) when using full_read() since
coreutils 4.5.8 with this commit:
http://git.sv.gnu.org/gitweb/?p=coreutils.git;a=commit;h=23f6d41
Now we really shouldn't be delaying processing like that,
so I propose we switch back to safe_read().
With the attached, the following command line gives immediate output:
while true; do seq 5; sleep 1; done | src/split --lines=5 --filter=date
I'll do a full patch later after checking if safe_read() is
appropriate elsewhere, and adding NEWS and maybe a test.
thanks,
Pádraig.
split-line-buffer.patch
Description: Text Data