[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

How far does GNU Parallel scale?

From: Ole Tange
Subject: How far does GNU Parallel scale?
Date: Wed, 20 Aug 2014 11:37:41 +0200

I rarely run more than 200 jobs in parallel. But I was curious how
many jobs I could safely run. This starts 30000 sleeps:

  seq 1000000000 | parallel -N1000 -j120 --pipe parallel -j0 sleep

I have let those stay in the background for a day now, and the only
issue I have had so far is that 'ps aux' takes > 1 sec to run because
it has to list > 30000 processes.

I also tried going higher, but GNU Parallel does not seem to act
predictably when hitting the system hits the 32767 process limit. But
that might also be caused by the rest of the system acting weird when
hitting that limit.

So using the fan out approach (parallel --pipe parallel) GNU Parallel
scales well to 30000 processes without any changes to my GNU/Linux

If you are running GNU Parallel on another platform (and you are not
afraid to crash it), it would be interesting what the (safe) limit is
on other systems.


reply via email to

[Prev in Thread] Current Thread [Next in Thread]