parallel
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

GNU Parallel and max jobs


From: paralleluser
Subject: GNU Parallel and max jobs
Date: Sun, 07 Oct 2018 19:24:11 -0400

Friends

I decided to re-read the GNU parallel man page in full and found two 
interesting paragraphs.  Let us assume a single user machine where the user has 
root access.  It is the goal of the user to not limit himself in terms of the 
number of jobs parallel can spawn (ie: full power to parallel).

-------------------------------------------------------------------------------------------------------------
Paragraph 1 from man page:
       
       If -j0 normally spawns 252 jobs, then the above will try to spawn 2520
       jobs. On a normal GNU/Linux system you can spawn 32000 jobs using this
       technique with no problems. To raise the 32000 jobs limit raise
       /proc/sys/kernel/pid_max to 4194303.
       
Analysis:  OK I can make that change.

-------------------------------------------------------------------------------------------------------------
Paragraph 2 from man page:

       EXAMPLE: Running more than 250 jobs workaround
       If you need to run a massive amount of jobs in parallel, then you will
       likely hit the filehandle limit which is often around 250 jobs. If you
       are super user you can raise the limit in /etc/security/limits.conf
       
Analysis:  How is this different from the change above?  What number to I set 
it to?  What is the syntax for that file (it is not as clear as pid_max)

-------------------------------------------------------------------------------------------------------------

Overall Analysis:  Do I care about anything with "ulimit" if I make the changes 
above?  Is there anything else to do to give parallel freedom to go to the max?

Thanks!



reply via email to

[Prev in Thread] Current Thread [Next in Thread]