|
| From: | Chevreux, Bastien |
| Subject: | GNU Parallel Bug Reports Enhancement request: having only parts of the parallel jobs run with a nice value? |
| Date: | Mon, 16 May 2016 20:08:22 +0000 |
|
Hello there, I think this is something I haven’t seen anywhere in the documentation of the current parallel version. Summary: Would it be possible to implement a feature to run a certain number of jobs “full speed” and the remaining jobs with a nice value given by “--nice”? E.g.: parallel -j 5 --numnonice 2 --nice 20 … which would run 5 jobs in parallel, making sure that 2 of them are always started without a nice value (fullspeed) and the remaining (in this case: 3) get a nice value of 20. Note that I do not expect “parallel” to constantly
monitor the number of running jobs and adapt their nice value. Setting the value just once at the start of the job would be entirely sufficient. Motivation: I’m running a program which alternates between single thread usage (for hours) and, when it is allowed to, multiple thread usage (also for hours) and sometimes uses ~100 of gigabyte RAM. Let’s say I have 12 cores and
I want to use them as efficiently as possible. I cannot start 12 jobs in parallel (and tell the program to use 1 thread) due to RAM limitations. I have enough RAM for 5 jobs, but I do not want to start 5 jobs with just 2 or 3 threads as this would severely
underutilise the machine (tested that). On the other hand, starting 5 jobs with each allowed to use, say, 6 threads without nice value is also dangerous: more often than not it happens that all 5 jobs run 6 threads in parallel. I have then 30 threads heavily
thrashing instruction pipelines and caches of my 12 cores. Not good. Starting all jobs with a nice value does not make sense either for the same reason: they’d have the same priority and the kernel would not prioritise between them. However, what works extremely well is to have parallel start all jobs without nice value, and then semi-manually babysitting the jobs to renice them as needed to always have 2 running at full speed, and the others grabbing
the CPU power left after that. I looked at the source for parallel to see whether there was a trivial way for me to implement what I’m looking for, but after some unsuccessful tinkering with a source doing lots of things I basically gave up. Best, Bastien -- DISCLAIMER: This e-mail is for the intended recipient only. If you have received it by mistake please let us know by reply and then delete it from your system; access, disclosure, copying, distribution or reliance on any of it by anyone else is prohibited. If you as intended recipient have received this e-mail incorrectly, please notify the sender (via e-mail) immediately. |
| [Prev in Thread] | Current Thread | [Next in Thread] |