bug#13089: Wish: split every n'th into n pipes

[Pádraig Brady]

tag 13089 + notabug
close 13089

On 12/05/2012 02:11 PM, Ole Tange wrote:
> I often have data that can be processed in parallel.
>
> It would be great if split --filter could look at every n'th line
> instead of chunking into n chunks:
>
>    cat bigfile | split --every-nth -n 8 --filter "grep foo"
>
> The above should start 8 greps and give each a line in round robin manner.
>
> Ideally it should be possible to do so non-blocking so if some lines
> take longer for one instance of grep, then the rest of the greps are
> not blocked.

So that's mostly supported already (notice the r/ below):

$ seq 8000 | split -n r/8 --filter='wc -l' | uniq -c
      8 1000

The concurrency is achieved through standard I/O buffers
between split and the filters (note also the -u split option).

I'm not sure non blocking I/O would be of much benefit,
since the filters will be the same, and if we did that,
then we'd have to worry about internal buffering in split.
We had a similar question about tee, yesterday, and I
think the answer is the same here, that the complexity
doesn't seem warranted for such edge cases.

thanks,
Pádraig.