Re: Doubts about .WAIT and .NOTPARALLEL (was: GNU Make 4.4 released!)

[Paul Smith]

On Mon, 2022-10-31 at 11:06 +0100, Alejandro Colomar wrote:
> On 10/31/22 08:01, Paul Smith wrote:
> > * New feature: The .WAIT special target
> 
> I'm curious: what is .WAIT made for?

As mentioned in the NEWS, it's _primarly_ there because the next
release of the POSIX spec will require a POSIX-conforming make utility
to support it.

> Isn't it just a workaround for broken dependencies?

Fundamentally I agree with you, that .WAIT is a poor workaround for a
true prerequisite statement, and is hard to use in a reliable way.

I made this case to the POSIX folks.  However, the .WAIT facility with
its current behavior has been available in BSD versions of make for a
very long time and is widely used there.  It tends to be used more for
things like the FreeBSD Ports build system where you're declaring
higher-level targets to be waited on so it's simpler to reason about.

>   I mean, considering the following example:
> 
>      all: one two .WAIT three
>      one two three: ; @sleep 1; echo $@
> 
> It's the same as if it was written as
> 
>      all: one two three
>      three: one two
>      one two three: ; @sleep 1; echo $@
> 
> Isn't it?

As Eddy points out it's not identical in behavior.  Fundamentally .WAIT
does not add any new edges to the DAG that make constructs, so there is
nothing inherent to the target "three" (in your example) that connects
it to "one" or "two".  The "wait-ness" is a function of the _target_
(all), not the prerequisites at all.  And, it's specific to that
target.  So if you have:

  foo: one .WAIT two
  bar: one two

then if you run "make -j foo" make will wait for "one" to complete
before running "two", but if you run "make -j bar" then it will not.

Even more annoyingly, if you have:

  all: foo bar
  foo: one .WAIT two
  bar: one two

and you run "make -j all" then "one" and "two" will be run concurrently
because even though when make wants to build "foo" it will wait to
start "two", when make wants to build "bar" concurrently with "foo"
then "one" and "two" will run concurrently.

So like I said, IMO .WAIT is a suboptimal capability that is hard to
use reliably and probably isn't very useful compared to other ordering
capabilities that GNU make already has.

But, it is very easy to use, is required by POSIX, and there are a lot
of makefiles that want to use it.

So now they can.

> > * New feature: .NOTPARALLEL accepts prerequisites
> 
> But my useful question comes with .NOTPARALLEL, which I think can be
> useful for something I asked some time ago, and can't be done with
> pre-4.4 make.
> 
> The documentation is a bit unclear on the exact details of this
> feature, by writing it in terms of .WAIT.
> 
> Let's say I have the following:
> 
>      all: a b c
>      .NOTPARALLEL: b
>      a: b c
>      a b c: ; @sleep 1; echo $@
> 
> Is it guaranteed that b will be run either before or after c but
> never at the same time?  Please clarify the manual regarding this. 
> Maybe this example is good for the docs.

No.  In the makefile above, .NOTPARALLEL has exactly zero effect.  The
manual says:

   This implicitly adds a .WAIT between each prerequisite of the listed
   targets.

I think this is pretty clear.  You might be confusing yourself by
hoping this provides some capability that it doesn't and so reading
things into the manual beyond what it actually says.

Probably an example would be helpful in the manual, but in your snippet
above you don't list any prerequisites for "b", so ".NOTPARALLEL: b",
which adds .WAIT between each prerequisite, is a no-op.

If you had this:

  .NOTPARALLEL: b
  b: x y z

it would be identical to writing:

  b: x .WAIT y .WAIT z

and exactly that, nothing else.

> The usefulness of something like this is that b might run a command
> that asks for a password, and I want literraly _everything_ else on
> hold while a program is asking for a password.

That is not how it works.

I don't believe that such a requirement can or should be met by some
form of .NOTPARALLEL.  If we wanted to create such a thing the way I
would do it is create a new special target like ".PREMAKE" or whatever,
and all the prerequisites of that special target would be guaranteed to
be updated first, before any other target, regardless of the goal
targets.

Or, something like that.  Details would need to be considered.

> I guess shuffle doesn't mess with .NOTPARALLEL or .WAIT.  Just to
> confirm.

I'm not sure what you mean by "mess with", but they have no
relationship to each other.

> > * New feature: The --jobserver-style command line option and named
> > pipes
> 
> Another curious question:  Does this fifo jobserver allow
> .NOTPARALLEL even with recursive makes?

It has no relationship to .NOTPARALLEL which continues to be confined
to a single instance of make.  I don't even really understand what it
would mean to have .NOTPARALLEL cross recursive makes.