Re: Avoiding "Warning: Starting 10 extra processes takes > 2 sec."

[Ole Tange]

2011/3/15 Alexander Søndergaard <gammel.holte@gmail.com>:
> Hello,
>
> I've run into a small issue when using parallel.

Please always include the output of 'parallel --version' and if
possible include a complete example that reproduces the error.

> I've got a pipe that blocks due to the presence of a sort filter.
> Hence, parallel decides to avoid spawning multiple parallel tasks, as
> it doesn't receive any input till a few seconds later:
>
> cat longfile | sort | uniq | parallel -j 20 'echo {} | ruby task.rb'
>
> Warning: Starting 10 extra processes takes > 2 sec.
> Limiting to 1 jobs in parallel.

You might consider using --pipe instead of the echo:

cat longfile | sort | uniq | parallel --pipe -N 1 -j 20 ruby task.rb

> Can I avoid this warning other than simply splitting the pipeline into
> two commands or reducing the amount of parallel tasks?

My guess is that you run an old version of GNU Parallel as the warning
was changed to advisory-only at Thu Oct 14 23:13:48 2010, so any newer
release would only give the warning but not change the number of
jobslots wanted.

So: Upgrade to a newer version. If that does not help, then you will
need to give a full example including all files that reproduces the
error.

> I've checked the man page, but I haven't found any
> suitable options.
>
> Thanks.
>
> Best,
> A. S.
>

/Ole