GNU Parallel Bug Reports Randomising Job Distribution

[Alastair Andrew]

Hi Ole,

Is there any way to tell GNU parallel to randomly assign the jobs to machines 
rather than just chunking them in sequence? I'm creating about 80 experimental 
configurations on problems of varying size and distributing them across ~25 
Ubuntu machines. The machines are 4 core boxes with between 8 GB RAM. I'm 
assigning one job per core. If 4 of the biggest jobs get placed on the same 
machine they starve each other of resources and completely kill that machine. 
This is how I'm using parallel:

parallel -S .. --nice 19 --halt-on-error 0 -j+0 --noswap "solve {1}_{2}.prob" 
::: small medium big ::: {1..20}

I've set the processes to be nice'd as much as possible and I've used the 
--noswap flag to stop jobs being sent to heavily loaded machine but it doesn't 
help. The --noswap (and I guess --load if I used it) probably won't resolve the 
problem because they are concerned with preventing further jobs being started 
on a stressed machine. I want to try to prevent that happening from the outset. 
Instead of small_1, small_2, small_3, small_4 being sent to machine 1, and so 
on. I'd like to have a flag that did that automatically. 

I can sort -R the ~/.parallel/sshloginfile to randomise the order the machines 
are chosen in but that doesn't help me break the order the jobs themselves are 
divided in. I guess I could write the configurations out into a file, randomise 
that and then read it in again. However, that seems a bit clunky compared with 
GNU parallels elegant ::: syntax. 

Any suggestions or comments would be welcome.
Thanks,
Alastair