Re: GNU Parallel and max jobs

[Ole Tange]

On Mon, Oct 8, 2018 at 1:30 AM paralleluser <paralleluser@fastmail.net> wrote:

> I decided to re-read the GNU parallel man page in full and found two 
> interesting paragraphs.  Let us assume a single user machine where the user 
> has root access.  It is the goal of the user to not limit himself in terms of 
> the number of jobs parallel can spawn (ie: full power to parallel).
:
> Overall Analysis:  Do I care about anything with "ulimit" if I make the 
> changes above?  Is there anything else to do to give parallel freedom to go 
> to the max?

There are two limits that limit the number of jobs to run in parallel:
Number of processes and number of file handles.

The limits are set in different places.

The system wide limits are set in /proc/sys/kernel/pid_max and
/proc/sys/fs/file-max

# This is the max on my system:
echo 4194303 | sudo tee /proc/sys/fs/pid_max
echo 18446744073709551615 | sudo tee /proc/sys/fs/file-max

The maximums for the user are set in /etc/security/limits.conf and
will only make sense if they are lower than the system wide limits.

The format of /etc/security/limits.conf is described in the comments
in that file. This is the max limits on my system:

*                soft    nofile          1048576
*                hard    nofile          1048576
*                soft    nproc           4194303
*                hard    nproc           4194303

If you find a way to raise the 1048576 please post an answer on
https://unix.stackexchange.com/questions/359189/fixing-ulimit-open-files-cannot-modify-limit-operation-not-permitted/359418#359418


/Ole