Re: Split Flushing to filter command

[Pádraig Brady]

On 03/20/2014 10:11 AM, Pádraig Brady wrote:
> On 03/20/2014 02:39 AM, Eldon wrote:
>> Hello all,
>> I am attempting the following in a bash shell on a 3.8.13 linux kernel:
>> sudo tcpdump -nn 
>>     |grep --line-buffered NTPv2 
>>     |split -u --lines=10 --filter=date
>>
>> Clearly date would be replaced with some more useful script, but for the
>> mean time I am trying to use it to debug what I see as unexpected
>> buffering. Since the traffic I am looking at is fairly consistent (when
>> I pipe to cat instead, I see a steady stream), I would expect to see
>> regular ticks as date executions each time 10 lines are sent to
>> split. Instead I see flashes of many executions that seem to me to be a
>> buffer flush. I looked in the code in the git repo, and it seems that
>> the buffer is in fact filled prior to execution. Is it worth making a
>> patch or expanding the meaning of -u to pass this through in an
>> unbuffered fashion? Would there be a reason to reject such a patch if
>> well-formed?
>>
>> Thoughts?
> 
> split(1) doesn't use stdio so we're not hitting that buffering.
> What's happening is that we're explicitly buffering input
> (into a 64K buffer on my x86_64 system) when using full_read() since
> coreutils 4.5.8 with this commit:
> http://git.sv.gnu.org/gitweb/?p=coreutils.git;a=commit;h=23f6d41
> 
> Now we really shouldn't be delaying processing like that,
> so I propose we switch back to safe_read().
> 
> With the attached, the following command line gives immediate output:
> 
> while true; do seq 5; sleep 1; done | src/split --lines=5 --filter=date

Full patch attached which completely removes full_read() from all cases.

thanks,
Pádraig.

split-avoid-delay.patch
Description: Text Data