GNU Parallel 20130630 alpha released

[Ole Tange]

GNU Parallel 20130630 alpha has been released. It is available for
download at: http://alpha.gnu.org/gnu/parallel/

I have access to a few machines with more than 10 cores. Spawning jobs
on those remotely is a bother because /etc/ssh/sshd_config:MaxStartup
only all allows for 10 simultaneous logins, so GNU Parallel will only
spawn 10 jobs on those machines.

So far this has been patched using --sshdelay so you could tell GNU
Parallel to slow down logging in. But is a bad solution because you
could have servers in your cluster that are local and some that are on
the other side of the globe, thus having very different login times.

A year ago I devised a solution for that, but the coding
infrastructure for that was missing. That has now changed.

So when you now use --filter-hosts GNU Parallel will compute how long
it took to log in, and use a --sshdelay for that host at 1/5th of that
duration. Ideally 1/10th of the duration should also work, but that
fails too often for me.

The speedup is remarkable because more jobs get scheduled:

$ time src/parallel -j1000% -S m,e,l,o,c,f,w -j0 sleep 20\;echo {} ::: {1..500}
real    18m13.651s
user    0m9.713s
sys     0m19.885s

$ time src/parallel --filter-hosts -j1000% -S m,e,l,o,c,f,w -j0 sleep
20\;echo {} ::: {1..500}
real    1m28.680s
user    0m12.733s
sys     0m25.218s

So I am considering making --filter-hosts default, but I am not sure
if anything breaks. Therefore, dear users, could you please:

* Download the alpha release
http://alpha.gnu.org/gnu/parallel/parallel-20130630.tar.bz2
* Put '--filter-hosts' in /etc/parallel/config

If that causes problems please post a bug report.


/Ole