[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Properly scheduling multi-threaded/process jobs

From: Douglas A. Augusto
Subject: Re: Properly scheduling multi-threaded/process jobs
Date: Tue, 26 Aug 2014 14:25:08 -0300
User-agent: Mutt/1.5.23 (2014-03-12)

On 26/08/2014 at 11:04,
Rob Sargent <> wrote:

> We use tools (genetic analysis) which have explicit parameters for processor
> consumption and which when left to their own devices use variable numbers of
> core during a given run. We gauge their typical cpu load and parallelize
> accordingly.  I'm not too worried about exceeding cores/slots (within
> reason).  More runnables than cores to me just means maximal cpu usage.
> (Think back to the single cpu days: there was always more than one runnable,
> and that was a good thing.)


The ability to tell GNU Parallel that a certain job requires 'n' slots can
also, indirectly, be used when the resource is memory. In this way one could
assign 1 to the less memory hungry job and more than 1 to all others,

> Are you saying you know beforehand which job needs how many cores?  If so
> (before Ole builds your feature) could you group them into say small, medium
> and large and parallel those groups with reasonable --jobs each?

Yes, I know beforehand how many cores a job needs, this is an input parameter.
Grouping them is exactly what I did, but the drawback is that this imposes an
unnecessary synchronism point (barrier) between groups--this is specially bad
if there are unbalanced long-running jobs within a group.

Douglas A. Augusto

reply via email to

[Prev in Thread] Current Thread [Next in Thread]