[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
GNU Parallel Bug Reports Randomising Job Distribution
From: |
Alastair Andrew |
Subject: |
GNU Parallel Bug Reports Randomising Job Distribution |
Date: |
Wed, 21 Mar 2012 11:58:43 +0000 |
Hi Ole,
Is there any way to tell GNU parallel to randomly assign the jobs to machines
rather than just chunking them in sequence? I'm creating about 80 experimental
configurations on problems of varying size and distributing them across ~25
Ubuntu machines. The machines are 4 core boxes with between 8 GB RAM. I'm
assigning one job per core. If 4 of the biggest jobs get placed on the same
machine they starve each other of resources and completely kill that machine.
This is how I'm using parallel:
parallel -S .. --nice 19 --halt-on-error 0 -j+0 --noswap "solve {1}_{2}.prob"
::: small medium big ::: {1..20}
I've set the processes to be nice'd as much as possible and I've used the
--noswap flag to stop jobs being sent to heavily loaded machine but it doesn't
help. The --noswap (and I guess --load if I used it) probably won't resolve the
problem because they are concerned with preventing further jobs being started
on a stressed machine. I want to try to prevent that happening from the outset.
Instead of small_1, small_2, small_3, small_4 being sent to machine 1, and so
on. I'd like to have a flag that did that automatically.
I can sort -R the ~/.parallel/sshloginfile to randomise the order the machines
are chosen in but that doesn't help me break the order the jobs themselves are
divided in. I guess I could write the configurations out into a file, randomise
that and then read it in again. However, that seems a bit clunky compared with
GNU parallels elegant ::: syntax.
Any suggestions or comments would be welcome.
Thanks,
Alastair
- GNU Parallel Bug Reports Randomising Job Distribution,
Alastair Andrew <=