Re: parallel make and I/O of child processes

[Stefano Lattarini]

On Tuesday 25 January 2011, Ralf Wildenhues wrote:
> * Stefano Lattarini wrote on Tue, Jan 25, 2011 at 11:02:11AM CET:
> > So, what about the attached updated patch?
> 
> I have some nits, otherwise I'll let Eric approve this.
>
I agree with most of your nits, and I've amended the patch accordingly.
See my replies in-line below.

> > Subject: [PATCH] docs: another parallel make issue
> > 
> > * doc/autoconf.texi (Parallel Make): Document that some make
> > implementations, when run in parallel mode, connect stdout and
> > stderr of child processes to pipes or temporary files, and might
> > re-route stderr of spawned processes to stout.  Also document
> > that FreeBSD make in parallel mode reuses the same shell for
> > multiple commands within one recipe (like NetBSD make does).
> 
> > --- a/doc/autoconf.texi
> > +++ b/doc/autoconf.texi
> > @@ -19475,13 +19475,52 @@ sh
> >  @cindex Parallel @command{make}
> >  
> >  Support for parallel execution in @command{make} implementation varies.
> > -Generally, using GNU make is your best bet.  When NetBSD
> > address@hidden is invoked with @address@hidden, it will reuse the
> > -same shell for multiple commands within one recipe.  This can have
> > -unexpected address@hidden that GNU make has
> > -heuristics to avoid spawning a shell at all if the command is deemed
> > -safe to be executed directly.} For example, change of directories or
> > -variables persist between commands:
> > +Generally, using GNU make is your best bet.
> > +
> > +Some make implementations (among them, FreeBSD @command{make}, NetBSD
> > address@hidden, and Solaris @command{dmake}), when invoked with a
> > address@hidden@var{N}} option, connect the standard output and standard
> > +error of all their children processes to pipes or temporary regular
> 
> s/children/child/  (hasn't this been reviewed already?)
> 
Yes :-(  Sorry, stupid copy & paste blunder.

> > +files.  This can lead to slightly but subtly different semantics in
> 
> s/slightly but//
> 
Fixed.

> > +the behaviour of the spawned processes.  For example, even if the
> > address@hidden standard output is a connected to a tty, a command
> 
> s/is a/is/
>
Fixed.

> > +like @code{test -t 1} issued in a recipe will return a failure within
> > +those make implementations:
> 
> How about: ... is connected to a tty, the recipe commands will not be:
> 
Nice.  I've taken it.

> > address@hidden
> > +$ @kbd{cat Makefile}
> > +all:
> > +        @@test -t 1 && echo "Is a tty" || echo "Is not a tty"
> > +$ @kbd{make -j 2} # FreeBSD make
> > +Is not a tty
> > +$ @kbd{make -j 2} # NetBSD make
> > +--- all ---
> > +Is not a tty
> > +$ @kbd{dmake -j 2} # Solaris dmake
> > address@hidden --> 1 job
> > address@hidden --> Job output
> > +Is not a tty
> > address@hidden example
> > +
> > address@hidden
> > +while on the other hand:
> > +
> > address@hidden
> > +$ @kbd{make -j 2} # GNU make, Heirloom make
> > +Is a tty
> > address@hidden example
> > +
> > +Furthermore, parallel run of those @command{make} implementations will
> 
> "a parallel run" or "parallel runs"
> 
I've gone for the second one.

> > +route standard error from commands that they spawn into their own
> 
> s/that they spawn //
>
That was pre-existing; but I've fixed it anyway.

> > +standard output, and may remove leading whitespace from output lines.
> > +
> > +And there's more about NetBSD or FreeBSD @command{make}.  When run in
> 
> This sentence is newspaper style.  How about just
>   When NetBSD or FreeBSD @command{make} are run in parallel mode, ...
>
Fine with me.

> > +parallel mode, they will reuse the same shell for multiple commands
> > +within one recipe.  This can have unexpected
> > address@hidden that GNU make has heuristics to avoid
> > +spawning a shell at all if the command is deemed safe to be executed
> > +directly.} For example, change of directories or variables persist
> > +between commands:
> 
> Footnotes are really ugly, we should think of them as costing 5 bucks or
> so each time we use them.
>
This particular footnote was pre-existing.

> How about just omitting the footnote here,
> since this isn't about GNU make otherwise at all?
>
Agreed.  Footnote removed.

> Also, I don't know what the immediate connection here is.
>
> FWIW, newish GNU make supports .ONESHELL for reusing shells between
> recipe commands.
> 
> >  @example
> >  all:
> > @@ -19518,12 +19557,9 @@ Another consequence of this is that, if one 
> > command in a recipe uses
> >  @code{exit 0} to indicate a successful exit, the shell will be gone
> >  and the remaining commands of this recipe will not be executed.
> >  
> > -The above example also shows additional status output NetBSD
> > address@hidden produces in parallel mode for targets being updated.
> > -
> > -Furthermore, parallel NetBSD @command{make} will route standard error
> > -from commands that it spawns into its own standard output, and may
> > -remove leading whitespace from output lines.
> > +The above example also shows additional status output produced by NetBSD
> > address@hidden@footnote{but @emph{not} by FreeBSD @command{make}} in
> 
> I'd just replace the footnote with the same text in parentheses.
>
Fine with me.

> > +parallel mode for targets being updated.
> >  
> >  You can avoid these issues by using the @option{-B} option to enable
> >  compatibility semantics.  However, that will effectively also disable
> 
> Cheers,
> Ralf
> 

The updated patch is attached.

Regards,
   Stefano

From 36f9129c2daa538d169e84aa3169f9deb1b11a93 Mon Sep 17 00:00:00 2001
From: Stefano Lattarini <address@hidden>
Date: Mon, 24 Jan 2011 22:31:37 +0100
Subject: [PATCH] docs: another parallel make issue

* doc/autoconf.texi (Parallel Make): Document that some make
implementations, when run in parallel mode, connect stdout and
stderr of child processes to pipes or temporary files, and might
re-route stderr of spawned processes to stout.  Also document
that FreeBSD make in parallel mode reuses the same shell for
multiple commands within one recipe (like NetBSD make does).
---
 ChangeLog         |   10 +++++++++
 doc/autoconf.texi |   55 +++++++++++++++++++++++++++++++++++++++++-----------
 2 files changed, 53 insertions(+), 12 deletions(-)

diff --git a/ChangeLog b/ChangeLog
index 2fc5ab3..5067399 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,13 @@
+2011-01-25  Stefano Lattarini  <address@hidden>
+
+       docs: another parallel make issue
+       * doc/autoconf.texi (Parallel Make): Document that some make
+       implementations, when run in parallel mode, connect stdout and
+       stderr of child processes to pipes or temporary files, and might
+       re-route stderr of spawned processes to stout.  Also document
+       that FreeBSD make in parallel mode reuses the same shell for
+       multiple commands within one recipe (like NetBSD make does).
+
 2011-01-23  Ralf Wildenhues  <address@hidden>
 
        docs: new section about whitespace trimmmed from make command-lines.
diff --git a/doc/autoconf.texi b/doc/autoconf.texi
index ae601a0..151b70f 100644
--- a/doc/autoconf.texi
+++ b/doc/autoconf.texi
@@ -19475,12 +19475,46 @@ sh
 @cindex Parallel @command{make}
 
 Support for parallel execution in @command{make} implementation varies.
-Generally, using GNU make is your best bet.  When NetBSD
address@hidden is invoked with @address@hidden, it will reuse the
-same shell for multiple commands within one recipe.  This can have
-unexpected address@hidden that GNU make has
-heuristics to avoid spawning a shell at all if the command is deemed
-safe to be executed directly.} For example, change of directories or
+Generally, using GNU make is your best bet.
+
+Some make implementations (among them, FreeBSD @command{make}, NetBSD
address@hidden, and Solaris @command{dmake}), when invoked with a
address@hidden@var{N}} option, connect the standard output and standard
+error of all their child processes to pipes or temporary regular
+files.  This can lead to subtly different semantics in the behaviour
+of the spawned processes.  For example, even if the @command{make}
+standard output is connected to a tty, the recipe command will not be:
+
address@hidden
+$ @kbd{cat Makefile}
+all:
+        @@test -t 1 && echo "Is a tty" || echo "Is not a tty"
+$ @kbd{make -j 2} # FreeBSD make
+Is not a tty
+$ @kbd{make -j 2} # NetBSD make
+--- all ---
+Is not a tty
+$ @kbd{dmake -j 2} # Solaris dmake
address@hidden --> 1 job
address@hidden --> Job output
+Is not a tty
address@hidden example
+
address@hidden
+On the other hand:
+
address@hidden
+$ @kbd{make -j 2} # GNU make, Heirloom make
+Is a tty
address@hidden example
+
+Furthermore, parallel runs of those @command{make} implementations will
+route standard error from commands that they spawn into their own
+standard output, and may remove leading whitespace from output lines.
+
+When NetBSD or FreeBSD @command{make} are run in parallel mode, they will
+reuse the same shell for multiple commands within one recipe.  This can
+have unexpected consequences. For example, change of directories or
 variables persist between commands:
 
 @example
@@ -19518,12 +19552,9 @@ Another consequence of this is that, if one command in 
a recipe uses
 @code{exit 0} to indicate a successful exit, the shell will be gone
 and the remaining commands of this recipe will not be executed.
 
-The above example also shows additional status output NetBSD
address@hidden produces in parallel mode for targets being updated.
-
-Furthermore, parallel NetBSD @command{make} will route standard error
-from commands that it spawns into its own standard output, and may
-remove leading whitespace from output lines.
+The above example also shows additional status output produced by NetBSD
address@hidden (but @emph{not} by FreeBSD @command{make}) in parallel
+mode for targets being updated.
 
 You can avoid these issues by using the @option{-B} option to enable
 compatibility semantics.  However, that will effectively also disable
-- 
1.7.2.3