Re: Testsuite failures on AIX 5.3

[Stefano Lattarini]

On Monday 15 November 2010, Ralf Wildenhues wrote:
> * Stefano Lattarini wrote on Sun, Nov 14, 2010 at 10:14:54PM CET:
> >  <http://autobuild.josefsson.org/automake/log-201011141902214780000.txt>
> >  > + /bin/grep -F -v @SET_MAKE@ Makefile.in
> >  > + 1> Makefile.sed
> >  > + make -f Makefile.sed SHELL=/bin/sh test1
> >  >       echo ' @top_srcdir@/configure.in   @top_srcdir@/aclocal.m4 
> > @srcdir@/Makefile.am  \
> >  >              @srcdir@/Makefile.in @top_srcdir@/configure ansi2knr.1  
> > ansi2knr.c \
> >  >              depcomp install-sh missing    ' | grep ' ansi2knr\.c '
> >  > @SHELL@: not found
> >  > make: 1254-004 The error code from the last command is 1.
> >  > Stop.
> >  > + exit_status=2
> > IMHO this is a spurious failure due to the old hackish check:
> >   $FGREP -v @SET_MAKE@ Makefile.in > Makefile.sed
> >   $MAKE -f Makefile.sed SHELL=$SHELL test1
> > retained from older versions of ansi.test; since this check is superseded
> > by the later ones:
> >   $AUTOCONF
> >   ./configure
> >   $MAKE test1
> > my suggested fix is to just drop the hackish check.
> 
> But then I'm wondering why this code did not fail in the past.
> Must be a regression of some kind, either in the tests, or defs.in,
> or in other code.
Well, sort of...

By looking at the latest change in ansi.test, I see the following diffs:

 cat > Makefile.am << 'END'
 AUTOMAKE_OPTIONS = ansi2knr
-magic:
-       @echo $(DISTFILES)
+.PHONY: test1 test2
+test1:
+       echo ' $(DISTFILES) ' | grep ' ansi2knr\.c '
+       echo ' $(DISTFILES) ' | grep ' ansi2knr\.1 '
+test2: distdir
+       test -f $(distdir)/ansi2knr.c
+       test -f $(distdir)/ansi2knr.1
 END
 
 cat >> configure.in << 'END'
 AM_C_PROTOTYPES
+AC_OUTPUT
 END
 
 : > ansi2knr.c
 : > ansi2knr.1
 
-$ACLOCAL || Exit 1
-$AUTOMAKE || Exit 1
+$ACLOCAL
+$AUTOMAKE
 
 $FGREP -v @SET_MAKE@ Makefile.in > Makefile.sed
-$MAKE -s -f Makefile.sed SHELL=$SHELL magic | grep 'ansi2knr\.c'
+$MAKE -f Makefile.sed SHELL=$SHELL test1
+
+$AUTOCONF
+./configure
+$MAKE test1
+$MAKE test2
+
+:

Now, the pipeline:
  $MAKE -s -f Makefile.sed SHELL=$SHELL magic | grep 'ansi2knr\.c'
looks fishy, since it ignores the exit status from make.  And if the make
output is verbose enough in case of failure (like Solaris make's output
is), the following grep might have been succeded in spite of the make
failure.

On the other hand, the new version of the testcase doesn't ignore the
exit status of make anymore, and this might be what is causing the
failure.

Finally, a caveat: I'm proceeding mostly by guesses here, since I don't
have access to an AIX system.
 
> >  > FAIL: colon5.test (exit: 2)
> >  > FAIL: colon6.test (exit: 2)
> > 
> > These tests fail for the same resons of ansi.test;
And digging the history, I see this is my fault for having introduced the
"Makefile.sed hack" in those files too, in commit `v1.11-175-gf9fe878':

  diff --git a/tests/colon5.test b/tests/colon5.test
  ...
   $ACLOCAL
   $AUTOMAKE
  -grep 'Makefile:Makefile.in' Makefile.in
  +grep 'Makefile:Makefile\.in' Makefile.in
  +grep '^Makefile:.* \$(srcdir)/Makefile\.dep' Makefile.in
  +
  +sed '/@SET_MAKE@/d' <Makefile.in >Makefile.sed
  +$MAKE -f Makefile.sed SHELL=$SHELL test
  #^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Oops!

  diff --git a/tests/colon6.test b/tests/colon6.test
  ...
   $ACLOCAL
   $AUTOMAKE
  +
  +$EGREP 'Makefile:.*(demo|version)' demo/Makefile.in && Exit 1
  +grep 'version\.good:.*version\.gin' demo/Makefile.in
  +
  +sed '/@SET_MAKE@/d' <demo/Makefile.in >Makefile.sed
  +$MAKE -f Makefile.sed SHELL=$SHELL test
  #^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Oops!

> > I suggest to drop the older hackish checks and do more proper checks
> > with autoconf, ./configure and make.
Which is what I should have done in the first place.  Sorry about this.

> > BTW, there is a pending patch of mine aimed at making the
> > `colon*.test' tests more "semantic"; see:
> >  <http://lists.gnu.org/archive/html/automake-patches/2010-09/msg00110.html>
> 
> Changing that shouldn't be done to cover up a regression though.
Well, it wouldn't cover it, it would solve it IMHO.  After all, I don't think
the "Makefile.sed hack" is ever used in real life, right?  So no problem in
removing it in favor of more realistic checks.  WDYT?

Regards,
   Stefano