[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Parallel + SSH = slow

From: Christopher Sebastian
Subject: Parallel + SSH = slow
Date: Fri, 10 Sep 2010 15:12:32 -0700

Hi Ole and everyone,

GNU Parallel is very nice!  I find that it is amazingly fast for tasks
performed on the local host.  :)

...But once I start using remote hosts with --sshloginfile, the
startup time is way too long.  It takes around 30 seconds per host to
initialize (so, if I put 20 hosts into the sshloginfile, 10 minutes
are spent just on 'parallel' startup).

ALSO, I am finding that the tasks are only sent to two or three of the
remote hosts, and all the other hosts are doing 0% of the work.  (I am
sure that it is not a problem with the remote host because if I remove
the 'successful' hosts from the sshloginfile, then the work will be
sent to another two or three *different* servers instead.)

Here is the little "scanner" script that I am running.  (It attempts
to SSH into each host in my network and prints a CONNECTED message if

(for a in $(seq 144 153); do
   for b in $(seq 1 254); do
     echo 10.72.$a.$b
 done) | parallel --progress -j0 --sshloginfile parallel_sshloginfile
ssh -oBatchMode=yes -oUserKnownHostsFile=/dev/null
-oStrictHostKeyChecking=no -oConnectTimeout=3 {} bash -c \\\'echo

Here is my 'parallel_sshloginfile':


Thank you for any advice.

By the way, is there a mailing list archive anywhere so I don't ask
questions that have already been answered before?

~Christopher Sebastian
Phoenix, Arizona, USA

reply via email to

[Prev in Thread] Current Thread [Next in Thread]