[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Running >10 jobs in parallel on the same remote machine

From: Thomas Sattler
Subject: Re: Running >10 jobs in parallel on the same remote machine
Date: Sat, 04 Aug 2012 13:25:00 +0200
User-agent: Mozilla/5.0 (X11; Linux i686; rv:14.0) Gecko/20120713 Thunderbird/14.0

> I would like to have people test the idea before I implement
> it. I need your help with the options:
>   ssh <<what options here?>> server sleep 10000000 &
>   sleep 3 # To make sure the above is logged in and set up
>   seq 1000 | parallel -j0 ssh <<what options here?>> server 'echo {};sleep 2'
>   kill %1 # Kill the sleeping ssh

According to the ssh_config manual, the extra ssh needs the
options ControlMaster and ControlPath, the others only need
ControlPath (as ControlMaster defaults to "no", which means
check for existing 'masters' but don't try to become one).

It can be tested like this:

 1) in one terminal run:

    ssh -o ControlMaster=yes -o ControlPath=/tmp/foo \
      server "date; read; last | head"

 2) as soon as you can see the output of 'date' you can
    run the following in another terminal:

    seq 100 | parallel -j0 ssh -o ControlPath=/tmp/foo \
      server 'echo {};sleep 2'

 3) Now press [Enter] in the first terminal and see the
    output of 'last': There was only one login.

ControlPath should be a "little bit better" than the one
above, something like "~/.parallel/%l -> %r@%h:%p" (see
man ssh_config for their meaning) but that would need
quoting that I want to avoid for the testing phase.
 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

The above works fine on my Gentoo machine. Afterwards I
tested on Debian Squeeze and it just doesn't work. :-(

Even so I'm still unhappy with the idea of "one separate
ssh login per job". How about letting parallel doing the
multiplexing on its own, running a shell or perl on the
remote host to spawn the jobs?


reply via email to

[Prev in Thread] Current Thread [Next in Thread]