[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: GNU Parallel and max jobs
From: |
Ole Tange |
Subject: |
Re: GNU Parallel and max jobs |
Date: |
Wed, 10 Oct 2018 00:23:57 +0200 |
On Mon, Oct 8, 2018 at 1:30 AM paralleluser <paralleluser@fastmail.net> wrote:
> I decided to re-read the GNU parallel man page in full and found two
> interesting paragraphs. Let us assume a single user machine where the user
> has root access. It is the goal of the user to not limit himself in terms of
> the number of jobs parallel can spawn (ie: full power to parallel).
:
> Overall Analysis: Do I care about anything with "ulimit" if I make the
> changes above? Is there anything else to do to give parallel freedom to go
> to the max?
There are two limits that limit the number of jobs to run in parallel:
Number of processes and number of file handles.
The limits are set in different places.
The system wide limits are set in /proc/sys/kernel/pid_max and
/proc/sys/fs/file-max
# This is the max on my system:
echo 4194303 | sudo tee /proc/sys/fs/pid_max
echo 18446744073709551615 | sudo tee /proc/sys/fs/file-max
The maximums for the user are set in /etc/security/limits.conf and
will only make sense if they are lower than the system wide limits.
The format of /etc/security/limits.conf is described in the comments
in that file. This is the max limits on my system:
* soft nofile 1048576
* hard nofile 1048576
* soft nproc 4194303
* hard nproc 4194303
If you find a way to raise the 1048576 please post an answer on
https://unix.stackexchange.com/questions/359189/fixing-ulimit-open-files-cannot-modify-limit-operation-not-permitted/359418#359418
/Ole