Re: GNU Parallel seems to drop

[Ole Tange]

On Tue, Sep 25, 2012 at 1:22 PM, Dirk Eddelbuettel <edd@debian.org> wrote:

>> One way to solve that is to instead have the first invocation open A1,
>> B1 and C1 while the second writes to A2, B2 and C2. You can use {#} or
>> $PARALLEL_SEQ for that by writing to A{#} or A$PARALLEL_SEQ.
>
> Hm. Then I have ~ N x cores files, and need to aggregate those. Won't win that
> over straight awk use, I fear.  Shucks.

I hear mawk is faster.

You might be able to do some trickery with fifos:

mkfifo A1 A2 A3

and then a command similar to cat that will read (non-blocking) from
A1 and when there is not longer a full record then read (non-blocking)
from A2 and when there is not longer a full record then read
(non-blocking) from A3, and then start over with A1 again until A1-A3
are closed.

nonblock-cat  A1 A2 A3 > A &

I assume order is not important.


/Ole