Re: parallel + blast + LSF

[Ole Tange]

On Tue, May 5, 2015 at 10:32 AM, Giuseppe Aprea
<giuseppe.aprea@gmail.com> wrote:
> Hi and thank you for your reply.
>
> -j issue: I used "-j 192" since 192 is the sum of all the slots the queue
> system allocates on the different hosts. Reading again the manual I see why,
> given my options and my server file, GNU parallel could run 192 jobs on the
> same host. Anyway, in my opinion, this point isn't really clear. An user
> could also get the idea that -j is for the total cores which get divided
> among the different hosts as specified by the ncpus in the server file. At
> least, I expected that.

Please help by rephrasing the man page, so you would have understood
the concept the first time. Would this help:

       -P N     Number of jobslots on each machine. Run up to N jobs
in parallel.  0 means as many as possible. Default is 100% which will
run one job per CPU core on the machine.

> --wait: I am running on a shared cluster; that means the queue system may
> give me 8 slots an a 16-cores host. The other 8 slots could be used by a
> different user at the same time. Resources fair share implies that I don't
> run more than 8 simultaneous blastp instances on that host.

Which means that for that particular host you have to lie about the
number of cores (e.g. by using the ncpu/host syntax).

If you in general only want to use 50% of the cores, then -j 50% will
do the trick.

> That is why,
> when 8 simultaneous blastp are reached, I want GNU parallel to wait for one
> of these to complete before starting another one.

And that is what GNU Parallel does normally. No extra options needed.

> That is what I expect from
> "--semaphore"; I used --wait for that and to be sure the queue system waited
> for all background gnu parallel jobs to be completed before considering the
> whole job finished. Does that make sense now?

I assume that you have understood, that --pipe makes GNU Parallel
behave very differently from not having --pipe. You could even argue
that 'parallel --pipe' would justify being a command on its own.

Have you understood that --semaphore also makes GNU Parallel behave
very differently from both --pipe and no --pipe? (Which is why it does
have its own alias, namely 'sem').

And have you understood why it does not make sense to use --semaphore
in your situation?

If yes: Please help by rephrasing the man page, so you would have
understood it in the first read.

If no: Please walk through the tutorial section on Semaphore.


/Ole