Re: line buffering in pipes

[Assaf Gordon]

Hello,

On 2019-05-02 1:22 p.m., Egmont Koblinger wrote:
> On Thu, May 2, 2019 at 9:14 PM Assaf Gordon <address@hidden> wrote:
>
> > The easiest way to avoid that is to use "stdbuf" (from coreutils),
> > forcing a flush after each line. Assuming the lines are short enough
> > (and file's output should be short enough), it should work:
> >
> >      find /usr -print0 | xargs -0r -P199 -n16 stdbuf -oL file | ...
>
> I don't think this is robust enough. If many "stdbuf -oL file"
> processes decide to produce a reasonably sized output pretty much at
> the same time, it might still suddenly clog the pipe and result in a
> short write in one of them. Or am I missing something?

That's exactly why I wrote "assuming the lines are short enough".

A filename (max 1024 bytes?) followed by a mime-type (reasonably short?)
should be enough (regardless of how many processes are writing to the
same pipe at the same time).

It will certainly fail for large outputs, but has worked well (for me)
in such (shorter) scenarios.

I will have to dig further for exact details and justification.

regards,
 - assaf