Re: I don't understand this behaviour

[Christopher Jefferson]

Have you tried reading the documentation?

It tells you to set an id when running parallel from a script.

The purpose of id is to group your programs into groups.

It might be better if sem just refused to run from a script without an id.

On 24 Jan 2017 10:57 pm, "D. R. Evans" <doc.evans@gmail.com> wrote:

I am about to embark on a project in which the jobs will be much more
computationally intensive than the ones that precipitated this thread (see
below). I really need to understand why the "sem -- wait" does not do what I
expected, otherwise I'm liable to waste an awful lot of time by playing safe
and running the python jobs in the new project in series rather than parallel.

Can someone please point me to what the problem might be?

(Maybe my problem is simply in failing to understand what "sem --wait" is
supposed to do, but in the past it's always seemed to do what my reading of
the documentation suggests: pause the script until all the jobs started with
"sem" complete.)

For completeness, here is the original post:

D. R. Evans wrote on 01/22/2017 07:22 PM:
> Could someone please explain the following to me? I expect that I'm missing
> something simple, but I don't understand what I am seeing.
>
> I have a bash script, the relevant part of which looks like this:
>
> ----
>
> for (( yr=$MIN_YEAR; yr <= $MAX_YEAR; yr++))
> do
>
>   echo "creating maps for $yr"
>
>   sem -j-1 rbn-posters-map.py [parameters to python script]
>   sem -j-1 rbn-posters-map.py [parameters to python script]
>   sem -j-1 rbn-posters-map.py [parameters to python script]
>   sem -j-1 rbn-posters-map.py [parameters to python script]
>   sem -j-1 rbn-posters-map.py [parameters to python script]
>   sem -j-1 rbn-posters-map.py [parameters to python script]
>   sem -j-1 rbn-posters-map.py [parameters to python script]
>   sem -j-1 rbn-posters-map.py [parameters to python script]
>   sem -j-1 rbn-posters-map.py [parameters to python script]
>
> done
>
> sem --wait
>
> echo "After wait"
> ps auxw | grep rbn
>
> ----
>
> My understanding, which is obviously incomplete, was that the "sem --wait"
> command would cause the script to block until all the various "rbn-posters.py"
> jobs that had been started with sem had completed. Therefore, I expected that
> the "ps" command would find none of those jobs still to be running.
>
> However, what actually happens, consistently, is that 3 such jobs are running
> when the "ps" command is executed (on this machine there are four cores).
>
> I need to have all the jobs completed at the point where the "ps" is currently
> placed, so that the next part of the script will run correctly. But at the
> moment, because not all the "rbn-posters.py" jobs have completed, the next
> part fails.
>
> I welcome enlightenment as to why the "sem --wait" isn't doing what I
> expected. (And bonus points if someone can explain how to modify the sem
> commands in the script so as to be sure that they have all completed at the
> point where the "ps" is currently located, preferably while still allowing
> parallel execution of the "rbn-posters.py" jobs.)
>
> I thank you very much.

  Doc

--
Web:  http://www.sff.net/people/N7DR