Re: Distributed parallel: how to measure how many cores are really used on remote machine?

[Ole Tange]

On Mon, Dec 13, 2010 at 8:54 AM, Maciej Pilichowski
<pilichowski.maciej@gmail.com> wrote:
> Hello,
>
>  I have local computer with 2 cores, and remote one with 4 cores. I run
> parallel always with -j+0.
>
>  It seems that around 50% of the job is done locally, and 50% on remote
> machine. While I would expect around 20% and 80%.

This can be because it is much faster to start a job on the local
machine. If your jobs only take a few seconds or if your jobs take a
few minutes but transfer files, then the startup time can be
overwhelming.

A quick test is:

  seq 1 20 | parallel --progress -S .. sleep

The workaround may be something like:

  parallel -j+2 sem -j+0 your_command

That will start 2 extra jobs but sem will only start one per cpu. The
remaining 2 jobs are allowed to do file transfer, though.

>  I am wondering if parallel checks the cores locally, gets the answer, 2,
> and so running 2 tasks on remote machine too. Or if the transferring files
> is so limiting.
>
>  So -- how to measure how many cores are really used on remote machine?

GNU Parallel runs this command on the remote machine to figure it out:

  parallel --number-of-cores

/Ole