[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: How does this wait -n work to cap parallelism?

From: Greg Wooledge
Subject: Re: How does this wait -n work to cap parallelism?
Date: Mon, 29 Jul 2019 14:38:48 -0400
User-agent: Mutt/1.10.1 (2018-07-13)

On Mon, Jul 29, 2019 at 07:12:42PM +0100, Earnestly wrote:
>         #!/usr/bin/env bash
>         # number of processes to run in parallel
>         num_procs=5
>         # function that processes one item
>         my_job() {
>             printf 'Processing %s\n' "$1"
>             sleep "$(( RANDOM % 5 + 1 ))"
>         }
>         i=0
>         while IFS= read -r line; do
>             if (( i++ >= num_procs )); then
>                 wait -n   # wait for any job to complete. New in 4.3
>             fi
>             my_job "$line" &
>         done < inputlist
>         wait # wait for the remaining processes
> The question is about how the example works in order to maintain
> parallelism capped at num_proc.

"inputlist" is a file that contains some sort of data, one thing per
line.  As each line is read, it is supposed to kick off one job (process)
with that line as the input parameter.

The while loop reads a line at a time, and for every line, it runs
my_job in the background.  BUT, if it has already looped 5 or more times,
then before it does that, it calls "wait -n".

wait -n waits for one job to terminate.

Let's suppose the inputlist contains:


The first time through the loop, we run my_job 1.  The second time, my_job 2,
and so on, up to my_job 5.  All of those are run without any delay.
So there are 5 instances of my_job running simultaneously in the background.

The sixth time through the loop, i is 5 (before the increment), so
we run "wait -n".  This waits for one of the 5 instances of my_job to
finish.  Once that happens, we run my_job 6, and loop again.

The same happens for my_job 7, and my_job 8.  Each one is preceded by
a wait -n, so it waits for one of the existing jobs to terminate before
the new job is launched.

reply via email to

[Prev in Thread] Current Thread [Next in Thread]