Re: Bug in tail -f jobqueue?

[Ole Tange]

On Sun, Dec 26, 2010 at 9:00 AM, Ido Tal <idotal@gmail.com> wrote:
>
> Hi,
>
> I really liked the "tail -f" trick in the documentation for turning parallel

I am happy we agree that the "tail -f" is a trick: It is clearly not
an essential feature and therefore there are a few issues surrounding
the jobqueue.

JobSlots = Number of jobs to run in parallel

* You will get a warning if you do not submit JobSlots jobs within the
first second. E.g. if you have 8 cores and use -j+2 you have to submit
10 jobs. These can be dummy jobs (e.g. echo foo). For GNU Parallel
20101014 and later this warning can be ignored.
* Jobs will be run immediately, but output from jobs will only be
printed when JobSlots more jobs has been started. E.g. if you have 10
jobslots then the output from the first completing job will only be
printed when job 11 is started.

> into a queue system/batch manager:
>
> echo >jobqueue; tail -f jobqueue | parallel
> echo my_command my_arg >> jobqueue
>
> However, I wanted to use exactly the number of cores I had, so I changed
> "parallel" in the above to "parallel -j+0"
> Upon submitting my first job to jobqueue ( echo "touch a" >> jobqueue ) I
> got:
>
> Warning: Starting 10 extra processes takes > 2 sec.
> Limiting to 1 jobs in parallel.

This warning does not exist in version 20101014 and later. So I
believe your installation of 20101222 failed (maybe it installed in
/usr/local/bin but you the one you are using is in /usr/bin/). You can
check the version with parallel --version

> The machine is not under any load. I've compiled the latest version of
> parallel, "parallel-20101222". I do not get this warning when I use
> "parallel -j+0" in a less fancy way (no tail -f). So, this seems like a
> bug...

Are you sure you are running 20101222? What does parallel --version say?

> Please write back if there is any other info needed.
>
> And, thanks! It's a great program which is helping me a lot.

I am happy GNU Parallel has helped you. You can show your appreciation
by promoting GNU Parallel. You can:

* Post the intro video http://www.youtube.com/watch?v=OpaiGYxkSuQ on
your blog/Twitter/Facebook/Linkedin/user group/mailing lists
* Write a review for your blog/Twitter/Facebook/Linkedin/user
group/mailing lists
* Help others use GNU Parallel when you see problems that can be
solved using GNU Parallel
* Request or build a package for your favourite distribution
* Join the mailing list http://lists.gnu.org/mailman/listinfo/parallel
* Invite me to speak at your next conference (Contact http://ole.tange.dk)

If GNU Parallel saves you money:

* Donate (or convince your company to donate) to FSF https://my.fsf.org/donate/

/Ole