Re: Some "fun" last weekend

[Paul Smith]

On Wed, 2022-07-20 at 20:33 -0400, Dmitry Goncharov wrote:
> if we take a step back, what is the problem? The problem is the
> presence of jobserver-auth in MAKEFLAGS in the non recursive case.
> You already implemented a solution which sets jobserver-auth=-2,-2.
> Another option is to remove jobserver-auth from MAKEFLAGS. This would
> rob make of a chance to notify the user that the program cannot
> participate in the job server protocol, the file descriptors cannot
> be opened. That may be better than the subtlety that you described
> above.  On the other hand, if we were to rewrite the current impl
> with e.g. named pipes, the problem of jobserver-auth in MAKEFLAGS
> would stay, would it not?

No, because the problem is not really with jobserver-auth itself.

The problem is fundamentally that make is using open file descriptors
to pass information from parent makes to sub-makes, combined with the
fact that we don't actually know with 100% accuracy which processes we
start are really sub-makes, and which are not.  We have a heuristic
which is not always accurate.  This leads to the following problems:

1) We have to be very careful about close-on-exec: when we invoke a
sub-make we need to disable close-on-exec for these fds, and when we
invoke a process that is not a sub-make we must enable close-on-exec.

2) Even though a sub-process may invoke a sub-make, it may do other
things as well.  Often a recipe does multiple things in the same
subshell, one of which is invoke a sub-make.  It's impossible for us to
ensure, in these cases, that the jobserver is available only to make.

3) If other processes see the open fds and start reading/writing them
(this has happened before) then they'll mess up the jobserver
completely.

4) Some processes just close all fds and this causes problems with the
jobserver.  For example we have an issue with Python where it will, by
default, set close-on-exec on all open file descriptors before it runs
any subprocess.  If you don't realize this and you use Python as part
of your build system as an intermediary between parent and child makes,
it breaks things in inscrutable ways.

5) The Savannah bugs also mention other issues:
   https://savannah.gnu.org/bugs/index.php?62397
   https://savannah.gnu.org/bugs/index.php?57242

6) There is another issue with setting blocking / non-blocking read on
the jobserver fds.  I can't remember if there's a Savannah bug or not
about this, but changing the blocking/non-blocking status on a fd is
not local to a given process and this has caused problems for some
applications used with make.  I'll have to try to locate the info on
this issue.

So, why do named pipes help?

They help because we're not expecting the open fds to be passed down;
in fact we can set close-on-exec for ALL our (non-standard) fds, which
is what you'd want.  The parent make doesn't need to use a heuristic to
figure out whether the child process is a make or not.

Instead, any sub-process can look at MAKEFLAGS, see the value of
jobserver-auth, find the name of the pipe, open it, and start to use
it.  If the sub-process doesn't know anything about MAKEFLAGS, it will
never know that the jobserver is relevant.  If the sub-process knows
about it, it can participate.  Everything is up to the sub-process and
nothing is required of the parent make.

This also allows an arbitrary number of intermediate processes to come
between the parent make and the sub-make, since we don't need to force
all the processes to preserve some resource (open fds) across exec
calls to make sure everything continues to work.

Semaphores would be the same: we pass down the NAME of the semaphore,
and each sub-process that cares would use it.  But as I discovered we
can't use semaphores because we don't have a reliable way to create an
event handler that works both with semaphores and with SIGCHLD (that I
could find), other than polling which I don't like.

The downside is we need to write code to manage the named pipe
resource.