parallel
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Avoiding "Warning: Starting 10 extra processes takes > 2 sec."


From: Ole Tange
Subject: Re: Avoiding "Warning: Starting 10 extra processes takes > 2 sec."
Date: Tue, 15 Mar 2011 17:39:34 +0100

2011/3/15 Alexander S√łndergaard <gammel.holte@gmail.com>:
> Hello,
>
> I've run into a small issue when using parallel.

Please always include the output of 'parallel --version' and if
possible include a complete example that reproduces the error.

> I've got a pipe that blocks due to the presence of a sort filter.
> Hence, parallel decides to avoid spawning multiple parallel tasks, as
> it doesn't receive any input till a few seconds later:
>
> cat longfile | sort | uniq | parallel -j 20 'echo {} | ruby task.rb'
>
> Warning: Starting 10 extra processes takes > 2 sec.
> Limiting to 1 jobs in parallel.

You might consider using --pipe instead of the echo:

cat longfile | sort | uniq | parallel --pipe -N 1 -j 20 ruby task.rb

> Can I avoid this warning other than simply splitting the pipeline into
> two commands or reducing the amount of parallel tasks?

My guess is that you run an old version of GNU Parallel as the warning
was changed to advisory-only at Thu Oct 14 23:13:48 2010, so any newer
release would only give the warning but not change the number of
jobslots wanted.

So: Upgrade to a newer version. If that does not help, then you will
need to give a full example including all files that reproduces the
error.

> I've checked the man page, but I haven't found any
> suitable options.
>
> Thanks.
>
> Best,
> A. S.
>

/Ole



reply via email to

[Prev in Thread] Current Thread [Next in Thread]