Hello,
When using the --load option, if I have a long running job and many shorter
jobs, I find that new jobs are not scheduled until the longest job has
finished, even though the CPU load is below the limit set by --load.
It appears that new jobs are not started until the longer job has been "reaped".
I'm running on RHEL server 5.6, Intel Xeon 4 core.
Version info:
> parallel --version
GNU parallel 20120322
Copyright (C) 2007,2008,2009,2010,2011,2012 Ole Tange and Free Software Foundation, Inc.
To reproduce the problem, create a list of jobs where the first takes very long
compared to the others:
= load_test.sh =
sleep 600
seq 50000000 | gzip > /dev/null
seq 50000000 | gzip > /dev/null
seq 50000000 | gzip > /dev/null
seq 50000000 | gzip >
/dev/null
seq 50000000 | gzip > /dev/null
seq 50000000 | gzip > /dev/null
seq 50000000 | gzip > /dev/null
seq 50000000 | gzip > /dev/null
seq 50000000 | gzip > /dev/null
seq 50000000 | gzip > /dev/null
seq 50000000 | gzip > /dev/null
seq 50000000 | gzip > /dev/null
seq 50000000 | gzip > /dev/null
seq 50000000 | gzip > /dev/null
= eof =
Run using (assuming a 4 core cpu host):
parallel --progress --load 50% -S 4/host < load_test.sh
The first few seq jobs will start, but after they finish the load will
be above 50% for a short time. As the cpu load drops to below 50%, we'd
expect more jobs to be started, but parallel seems to wait until the
first sleep job has finished.
Same behavior on local host (not through SSH).
Great tool otherwise! Thanks!
- Joe