Re: Passing jobserver file descriptors to all children

[R. Diez]

> [...]
> Originally make did pass these file descriptors to all children, but there
> were bugs filed because some tools invoked by make expected to have
> specific file descriptors available; having make pass down open FDs caused
> them to fail.

Relying on specific file descriptor numbers is asking for trouble. I do not think that allowing for such brittle scripts justifies having to add a '+' 
to the make recipe. It is easy to forget. That is probably the reason why the heuristic $(MAKE) was added to assume you wanted '+' for that recipe. 
Which means that, if you are using a single shell for all the recipe (automatically or manually joining all commands with &&), then all other child 
processes will inherit the jobserver file descriptors too. Who is going to think about that side-effect with inherited jobserver file descriptors when 
optimising recipes to fork less shells?

Furthermore, such '+' distinction force Makefile generators to become more complex. Or to just ignore it, like Autoconf/Automake seems to do. So no 
jobserver descriptors when linking with gcc then.

I guess it is too late to reverse such decision. But that should not apply to a new "--always-pass-jobserver-descriptors" flag. If you have 
compatibility issues with old scripts that expect particular file descriptors, then do not use that flag and lose some parallel performance.

From the manual page, Bash seems to use descriptors above 10 internally, as 
only the low 10 seem guarantee stability for scripts:

"Redirections using file descriptors greater than 9 should be used with care, as 
they may conflict with file descriptors the shell uses internally."

Maybe if GNU Make moved those jobserver descriptors to higher numbers, like 
Bash does internally, there would be less problems in this area.


> Are there any docs around the GCC usage of the jobserver?

Unfortunately, there does not seem to be much documentation about how GCC uses the jobserver. I asked on the mailing list about tracing jobserver 
usage, but I got no answer yet.

I am not sure that looking at GCC would help much. Like I said, parallelising build tools will probably increase in the future, and that could impact 
any build step on any language. In fact, with the advent of multicore CPUs, I am surprised that this has not developed faster.

Without a jobserver to limit the parallel load, automatic builders (like continuous integration build servers) may easily overload. For example, GCC's 
LTO uses loads of RAM. If you do not place some limit, you may blow all RAM. In fact, I already commented on this some years ago, and that was not 
about LTO:

Subject: Thoughts on limiting the parallel build load, suggestion for a new "-j 
auto" option

https://lists.gnu.org/archive/html/bug-make/2012-02/msg00031.html

Using a jobserver helps directly or indirectly limit the machine load. That is why I believe that making the jobserver available all over the place is 
the way to go. In fact, sometimes the jobserver may be the only reason GNU Make is running for (with no actual useful makefile).

I am not sure about using named pipes. Then you have to think about security, and possibly cleaning them up at the end. Inheriting file descriptors 
actually seems like a simple, good concept.

Best regards,
  rdiez