Parallel + SSH = slow

[Christopher Sebastian]

Hi Ole and everyone,

GNU Parallel is very nice!  I find that it is amazingly fast for tasks
performed on the local host.  :)

...But once I start using remote hosts with --sshloginfile, the
startup time is way too long.  It takes around 30 seconds per host to
initialize (so, if I put 20 hosts into the sshloginfile, 10 minutes
are spent just on 'parallel' startup).

ALSO, I am finding that the tasks are only sent to two or three of the
remote hosts, and all the other hosts are doing 0% of the work.  (I am
sure that it is not a problem with the remote host because if I remove
the 'successful' hosts from the sshloginfile, then the work will be
sent to another two or three *different* servers instead.)



Here is the little "scanner" script that I am running.  (It attempts
to SSH into each host in my network and prints a CONNECTED message if
successful):

(for a in $(seq 144 153); do
   for b in $(seq 1 254); do
     echo 10.72.$a.$b
   done
 done) | parallel --progress -j0 --sshloginfile parallel_sshloginfile
ssh -oBatchMode=yes -oUserKnownHostsFile=/dev/null
-oStrictHostKeyChecking=no -oConnectTimeout=3 {} bash -c \\\'echo
CONNECTED:{}:\\\$\\\(hostname\\\):\\\$\\\(domainname\\\)\\\'





Here is my 'parallel_sshloginfile':

:
10.72.144.199
10.72.144.200
10.72.144.201






Thank you for any advice.

By the way, is there a mailing list archive anywhere so I don't ask
questions that have already been answered before?

~Christopher Sebastian
Phoenix, Arizona, USA