Re: parallel + blast + LSF

[Giuseppe Aprea]

Hi and thank you for your reply.

-j issue: I used "-j 192" since 192 is the sum of all the slots the queue system allocates on the different hosts. Reading again the manual I see why, given my options and my server file, GNU parallel could run 192 jobs on the same host. Anyway, in my opinion, this point isn't really clear. An user could also get the idea that -j is for the total cores which get divided among the different hosts as specified by the ncpus in the server file. At least, I expected that.

--wait: I am running on a shared cluster; that means the queue system may give me 8 slots an a 16-cores host. The other 8 slots could be used by a different user at the same time. Resources fair share implies that I don't run more than 8 simultaneous blastp instances on that host. That is why, when 8 simultaneous blastp are reached, I want GNU parallel to wait for one of these to complete before starting another one. That is what I expect from "--semaphore"; I used --wait for that and to be sure the queue system waited for all background gnu parallel jobs to be completed before considering the whole job finished. Does that make sense now?

g

On Tue, May 5, 2015 at 8:30 AM, Ole Tange <ole@tange.dk> wrote:

On Mon, May 4, 2015 at 5:31 PM, Giuseppe Aprea <giuseppe.aprea@gmail.com> wrote:

> it looks like all jobs are executed on the first server regardless of server
> file. I am using:
>
> cat proteins.fasta | parallel --no-notice -vv -j ${SLOTS} --tmpdir tmp
> --wait --slf servers --block 200k --recstart '>' --pipe blastp -evalue 1e-05
> -outfmt 6 -db proteins -query - -out result_{#}
>
> (SLOTS=192)
>
> with serverfile given by:
>
>> more servers
> 24/cresco3x045.portici.enea.it

The 24 tells GNU Parallel to treat this machine as having 24 cores.
This number is used in the -j calculation. -j 100% would mean 24 jobs
for this machine, -j 50% would mean 12 jobs, -j -1 would mean 23. And
-j 192 would mean 192.

So you have probably misunderstood how -j works. My advice: Remove -j
and let GNU Parallel run one job per core.

> It also
> looks like option "--wait" is neglected and GNU parallel is trying to run
> all 192 instances at the same time on a single server.

In the manual it says:

      --wait   Wait for all commands to complete.

                Implies --semaphore.

                See also man sem.

Did you read about --semaphore? Did you try out --wait in the
tutorial? Is that really what you want?

Maybe GNU Parallel should fail when using --pipe with
--wait/--semaphore? I cannot see a situation in which the combination
will make sense.


/Ole

PS: Consider running --bibtex once.