Re: keep-order and file handle limit

[Ole Tange]

On Fri, Feb 16, 2018 at 9:53 AM, Shyam Saladi <saladi@caltech.edu> wrote:

> GNU parallel is a great piece of software. I use it all the time.

Happy to hear.

> From time to time, I run into a file handle limit issue when I have many
> jobs to run (1M+):
>
> parallel: Warning: No more file handles. parallel: Warning: Raising ulimit
> -n or /etc/security/limits.conf may help.
>
> I hadn't realized that it was related to --keep-order, but Ole's comment on
> SO suggest's that it is.

It definitely can be.

> * What the relationship is between is between keep-order, the number of
> jobs, and the number of file handles used? Is there any relationship with
> the number of cores as well? [On some machines, I don't have control over
> the file limit, so better understanding this interplay would be useful]

GNU Parallel uses a bunch of file handles for each job. It releases
these when the job is printed, but as long as the job is not printed
these file handles are kept open.

So if you have a slow job followed by a bunch of fast jobs, then GNU
Parallel will have to keep the files open for the fast jobs until the
slow job is done when using --keep-order.

To see this in action this will run 1001 jobs. The first job takes 11
seconds to run. The rest take 1 second:

# No problem: 250+ jobs running. The first job will just be printed last
seq 0 1000 | parallel -j0 'sleep {= $_=1+10*!$_ =}; echo {}'
# Ouch: Here we have to wait for the first one to finish. Then the
first 250+ will be printed and the next ones will be started.
seq 0 1000 | parallel --keep-order -j0 'sleep {= $_=0.1+10*!$_ =}; echo {}'

If you need --keep-order and you cannot raise the number of file
handles, a workaround that might work is described here:
https://www.gnu.org/software/parallel/man.html#EXAMPLE:-Running-more-than-250-jobs-workaround

> * Could someone also clarify the expected behavior on stdout? When multiple
> jobs (e.g. A and B) are running, is it possible for the output to become
> interleaved (without --keep-order)? In other words (each line represents
> output from job A or B),

> 2a) Or am I guaranteed ....
>
> A
> A
> A
> B
> B
> B
>
> 2b) OR
>
> B
> B
> B
> A
> A
> A

You are guaranteed 2a+2b.

Compare these:

parallel -j0 'echo -n {} partial;sleep {};echo {};sleep {}; echo {}'
::: 9 8 7 6 5 4 3 2 1
parallel -j0 --keep-order 'echo -n {} partial;sleep {};echo {};sleep
{}; echo {}' ::: 9 8 7 6 5 4 3 2 1
parallel -j0 --line-buffer 'echo -n {} partial;sleep {};echo {};sleep
{}; echo {}' ::: 9 8 7 6 5 4 3 2 1
parallel -j0 --line-buffer --keep-order 'echo -n {} partial;sleep
{};echo {};sleep {}; echo {}' ::: 9 8 7 6 5 4 3 2 1
parallel -j0 -u 'echo -n {} partial;sleep {};echo {};sleep {}; echo
{}' ::: 9 8 7 6 5 4 3 2 1

> For my purposes, either 2a or 2b would be acceptable, but 1 would not. Do I
> need to specify --keep-order to guarantee 1 doesn't occur.

No, you do not.

> PS. Ole, we just published a paper that used GNU parallel. Of course, it is
> cited.

Thanks for your support. It is most appreciated.

/Ole