[PATCH] docs: other issues with parallel BSD make (was: Re: bug#9245: FreeBSD make in concurrent mode report spurious success in automake-generated tests harness)

[Stefano Lattarini]

[dropping bug-automake, adding autoconf-patches]

References:
 <http://debbugs.gnu.org/cgi/bugreport.cgi?bug=9245>
 <http://www.freebsd.org/cgi/query-pr.cgi?pr=159730>

On Tuesday 16 August 2011, Stefano Lattarini wrote:
> Hi Eric.
> 
> On Tuesday 16 August 2011, Eric Blake wrote:
> > On 08/16/2011 10:04 AM, Stefano Lattarini wrote:
> > > The "proper fix" is even easier BTW: just use "$$exit || exit 1" instead.
> > >
> > > The updated patch should now work (and I've tested it properly this time).
> > > This new patch hasn't been derived by "cargo-cult guessing", but from the
> > > explanation of the relevant FreeBSD make bug provided by Jilles Tjoelker:
> > >   <http://www.freebsd.org/cgi/query-pr.cgi?pr=159730>
> > >
> > > Now that's something to be documented in the Autoconf manual I'd say ...
> > 
> > Indeed.  So it all boils down to BSD make rewriting rules (but only for 
> > make -j) to be roughly:
> > 
> > sh -c 'set -v; your rules; set +v'
> > 
> > and losing your exit status (not to mention sharing shell status between 
> > commands that are supposed to be in independent shells), when it really 
> > should have been rewriting as something more complex:
> > 
> > sh -c '(your rule 1); (your rule 2); st=$?; set +v; exit $st'
> > 
> > And your workaround of an explicit exit rather than relying on $? at the 
> > end of your 'set +e' invocation is indeed the proper way to force the 
> > parallel make bug to be avoided, by exiting prior to any post-commands 
> > injected by the BSD make rewriting going on that corrupt $?.
> > 
> I'll have a "draft patch" read soonish.  There is ample room for improvements,
> but I'll post it here anyway since it can benefit from early feedback.
> 
Here it is.  As usual, comments and suggestions welcome.

Regards,
  Stefano

From a60b30ea47b2cc36d0b85bbed94a106643fba745 Mon Sep 17 00:00:00 2001
Message-Id: <address@hidden>
From: Stefano Lattarini <address@hidden>
Date: Tue, 16 Aug 2011 18:31:42 +0200
Subject: [PATCH] docs: other issues with parallel BSD make

Motivated by automake bug#9245:
 <http://debbugs.gnu.org/cgi/bugreport.cgi?bug=9245>
and FreeBSD PR bin/159730:
 <http://www.freebsd.org/cgi/query-pr.cgi?pr=159730>

* doc/autoconf.texi (Parallel Make): Document other BSD make
incompatibilities.  Reorganize the existing related documentation
accordingly.
---
 ChangeLog         |   11 ++++
 doc/autoconf.texi |  137 +++++++++++++++++++++++++++++++++++-----------------
 2 files changed, 103 insertions(+), 45 deletions(-)

diff --git a/ChangeLog b/ChangeLog
index 6ca51fd..6e36455 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,14 @@
+2011-08-16  Stefano Lattarini  <address@hidden>
+
+       docs: other issues with parallel BSD make
+       Motivated by automake bug#9245:
+        <http://debbugs.gnu.org/cgi/bugreport.cgi?bug=9245>
+       and FreeBSD PR bin/159730:
+        <http://www.freebsd.org/cgi/query-pr.cgi?pr=159730>
+       * doc/autoconf.texi (Parallel Make): Document other BSD make
+       incompatibilities.  Reorganize the existing related documentation
+       accordingly.
+
 2011-08-08  Stefano Lattarini  <address@hidden>
 
        docs: fix minor typos
diff --git a/doc/autoconf.texi b/doc/autoconf.texi
index 7c1818d..b2b0a59 100644
--- a/doc/autoconf.texi
+++ b/doc/autoconf.texi
@@ -19826,45 +19826,10 @@ sh
 Support for parallel execution in @command{make} implementation varies.
 Generally, using GNU make is your best bet.
 
-Some make implementations (among them, FreeBSD @command{make}, NetBSD
address@hidden, and Solaris @command{dmake}), when invoked with a
address@hidden@var{N}} option, connect the standard output and standard
-error of all their child processes to pipes or temporary regular
-files.  This can lead to subtly different semantics in the behaviour
-of the spawned processes.  For example, even if the @command{make}
-standard output is connected to a tty, the recipe command will not be:
-
address@hidden
-$ @kbd{cat Makefile}
-all:
-        @@test -t 1 && echo "Is a tty" || echo "Is not a tty"
-$ @kbd{make -j 2} # FreeBSD make
-Is not a tty
-$ @kbd{make -j 2} # NetBSD make
---- all ---
-Is not a tty
-$ @kbd{dmake -j 2} # Solaris dmake
address@hidden --> 1 job
address@hidden --> Job output
-Is not a tty
address@hidden example
-
address@hidden
-On the other hand:
-
address@hidden
-$ @kbd{make -j 2} # GNU make, Heirloom make
-Is a tty
address@hidden example
-
-Furthermore, parallel runs of those @command{make} implementations will
-route standard error from commands that they spawn into their own
-standard output, and may remove leading whitespace from output lines.
-
 When NetBSD or FreeBSD @command{make} are run in parallel mode, they will
 reuse the same shell for multiple commands within one recipe.  This can
-have unexpected consequences. For example, change of directories or
-variables persist between commands:
+have various unexpected consequences.  For example, changes of directories
+or variables persist between recipes, so that:
 
 @example
 all:
@@ -19876,7 +19841,6 @@ all:
 may output the following with @code{make -j1}:
 
 @example
---- all ---
 /
 value
 32235
@@ -19897,19 +19861,102 @@ value
 32239
 @end example
 
-Another consequence of this is that, if one command in a recipe uses
address@hidden 0} to indicate a successful exit, the shell will be gone
-and the remaining commands of this recipe will not be executed.
address@hidden
+Another consequence is that, if one command in a recipe uses @code{exit 0}
+to indicate a successful exit, the shell will be gone and the remaining
+commands of this recipe will not be executed.
 
-The above example also shows additional status output produced by NetBSD
address@hidden (but @emph{not} by FreeBSD @command{make}) in parallel
-mode for targets being updated.
+The BSD @command{make} implementations, when run in parallel mode,
+will also pass the @command{Makefile} recipes to the shell through
+its standard input, thus making it unusable from the recipes:
 
-You can avoid these issues by using the @option{-B} option to enable
address@hidden
+$ @kbd{cat Makefile}
+read:
+        @@read line; echo LINE: $$line
address@hidden $$ @c restore font-lock
+$ @kbd{echo foo | make read}
+LINE: foo
+$ @kbd{echo foo | make -j1 read}
+LINE:
address@hidden example
+
address@hidden
+Moreover, when FreeBSD @command{make} is run in parallel mode, it
+implements the @code{@@} and @code{-} ``recipe modifiers'' by dynamically
+modifying the active shell flags.  This behaviour has the effects of
+potentially clobbering the exit status of recipes silenced with the
address@hidden@@} modifier if they also unset the @option{errexit} shell flag,
+and of mangling the output in unexpected ways:
+
address@hidden
+$ @kbd{cat Makefile}
+a:
+        @@echo $$-; set +e; false
+b:
+        -echo $$-; false; echo set -
+$ @kbd{make a; echo status: $?}
+ehBc
+*** Error code 1
+status: 1
+$ @kbd{make -j1 a; echo status: $?}
+ehB
+status: 0
+$ @kbd{make b}
+echo $-; echo set -
+hBc
+set -
+$ @kbd{make -j1 b}
+echo $-; echo hvB
address@hidden example
+
address@hidden
+You can avoid all these issues by using the @option{-B} option to enable
 compatibility semantics.  However, that will effectively also disable
 all parallelism as that will cause prerequisites to be updated in the
 order they are listed in a rule.
 
+Some make implementations (among them, FreeBSD @command{make}, NetBSD
address@hidden, and Solaris @command{dmake}), when invoked with a
address@hidden@var{N}} option, connect the standard output and standard
+error of all their child processes to pipes or temporary regular
+files.  This can lead to subtly different semantics in the behaviour
+of the spawned processes.  For example, even if the @command{make}
+standard output is connected to a tty, the recipe command will not be:
+
address@hidden
+$ @kbd{cat Makefile}
+all:
+        @@test -t 1 && echo "Is a tty" || echo "Is not a tty"
+$ @kbd{make -j 2} # FreeBSD make
+Is not a tty
+$ @kbd{make -j 2} # NetBSD make
+--- all ---
+Is not a tty
+$ @kbd{dmake -j 2} # Solaris dmake
address@hidden --> 1 job
address@hidden --> Job output
+Is not a tty
address@hidden example
+
address@hidden
+On the other hand:
+
address@hidden
+$ @kbd{make -j 2} # GNU make, Heirloom make
+Is a tty
address@hidden example
+
address@hidden
+The above examples also show additional status output produced in parallel
+mode for targets being updated by Solaris @command{dmake} and NetBSD
address@hidden (but @emph{not} by FreeBSD @command{make}).
+
+Furthermore, parallel runs of those @command{make} implementations will
+route standard error from commands that they spawn into their own
+standard output, and may remove leading whitespace from output lines.
+
+
 @node Comments in Make Rules
 @section Comments in Make Rules
 @cindex Comments in @file{Makefile} rules
-- 
1.7.2.3