bug-parallel
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Cannot specify the number of threads for parsort


From: Mario Roy
Subject: Re: Cannot specify the number of threads for parsort
Date: Fri, 10 Feb 2023 19:44:14 -0600

> Can you explain why you need this? To me it seems odd not to use all the cores you paid for.

I was trying to simulate running on a smaller machine how long the task takes for sharing with a friend. Anyway, ++ for parsort supporting the -j argument; to be consistent with parallel -j. Ditto for parsort checking the PARALLEL environment variable if it has -jN or -j N; this too for consistency with parallel.

> There's a current phobia of having a machine actually using all it's cores and a misconception that the users can do a better job of scheduling than the scheduler!

No, not at all. Having the -j option to parsort is desirable on larger NUMA-aware machines. This is true when a node is busy and wish to run on the other node. Surely, one can use nodeadm or taskset. But why default to spawning all those processes only for the OS to task switch more on that particular node.

++ for parsort supporting -j, to be consistent with parallel at the very least.


On Fri, Feb 10, 2023 at 5:07 PM Ole Tange <ole@tange.dk> wrote:
On Thu, Feb 9, 2023 at 3:30 PM Mario Roy <marioeroy@gmail.com> wrote:
:
> This is a wish list for allowing one to specify the number of threads via an ENVIRONMENT variable that works consistently using parallel or parsort. Basically, I want to specify the number of threads for parsort regardless of processing files specified via command-line arguments or STDIN.

You can use taskset:

taskset -c 1-5 parallel --number-of-threads

(parsort uses parallel --number-of-threads).

For parallel you might also use PARALLEL=-j5

Can you explain why you need this? To me it seems odd not to use all
the cores you paid for.


/Ole

reply via email to

[Prev in Thread] Current Thread [Next in Thread]