Re: 2.63b autotest vs. bison testsuite

[Eric Blake]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

According to Eric Blake on 4/6/2009 11:14 AM:
> probably need to teach more of autotest about automake's recent addition of 
> status 99 meaning hardfail (not even XFAIL can exempt it from making the 
> overall testsuite report failure), but that is why this patch below used 
> at_status=99.

As in the following.  Any objections to this patch?  In addition to adding
hard failures, it documents and tests AT_CHECK_NOESCAPE, and also tests
the ability to run post-AT_CHECK cleanup.

- --
Don't work too hard, make some time for fun as well!

Eric Blake             address@hidden
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Public key at home.comcast.net/~ericblake/eblake.gpg
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAknjLHcACgkQ84KuGfSFAYAQVgCgvrLWJRffUj3hl9tcSy4+/KLl
0HoAn33rjx0665FmwTnoZCcui/118FCh
=s3Sk
-----END PGP SIGNATURE-----

>From 05aa7f60a72577d4923b538c03f6a75d0cbf3e1b Mon Sep 17 00:00:00 2001
From: Eric Blake <address@hidden>
Date: Mon, 13 Apr 2009 06:02:42 -0600
Subject: [PATCH] Teach AT_CHECK about hard failures.

* lib/autotest/general.m4 (AT_INIT) <at_fn_check_skip>
<at_fn_check_status, at_fn_group_postprocess>: Handle hard
failures.
* doc/autoconf.texi (Writing Testsuites) <AT_CHECK>: Document
AT_CHECK_NOESCAPE and exit status 99.
* NEWS: Likewise.
* tests/autotest.at (Hard fail, Cleanup): New tests.

Signed-off-by: Eric Blake <address@hidden>
---
 ChangeLog               |    9 ++++++
 NEWS                    |    7 ++++
 doc/autoconf.texi       |   74 +++++++++++++++++++++++++++++++++++------------
 lib/autotest/general.m4 |   27 ++++++++++++-----
 tests/autotest.at       |   39 ++++++++++++++++++++++++
 5 files changed, 129 insertions(+), 27 deletions(-)

diff --git a/ChangeLog b/ChangeLog
index efa2056..062198b 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,5 +1,14 @@
 2009-04-13  Eric Blake  <address@hidden>

+       Teach AT_CHECK about hard failures.
+       * lib/autotest/general.m4 (AT_INIT) <at_fn_check_skip>
+       <at_fn_check_status, at_fn_group_postprocess>: Handle hard
+       failures.
+       * doc/autoconf.texi (Writing Testsuites) <AT_CHECK>: Document
+       AT_CHECK_NOESCAPE and exit status 99.
+       * NEWS: Likewise.
+       * tests/autotest.at (Hard fail, Cleanup): New tests.
+
        Finish upgrade to GFDL 1.3.
        * doc/autoconf.texi (copying): Use correct license; comment change
        was missed on 2008-11-04.
diff --git a/NEWS b/NEWS
index e91fc39..854533b 100644
--- a/NEWS
+++ b/NEWS
@@ -7,6 +7,13 @@ GNU Autoconf NEWS - User visible changes.
    proper m4 quoting.  For shell comments, this is a new feature; for
    non-shell comments, this fixes a regression introduced in 2.63b.

+** The macro AT_CHECK now understands the concept of hard failure.  If
+   a test exits with an unexpected status 99, cleanup actions for the
+   test are inhibited and the test is treated as a failure regardless
+   of AT_XFAIL_IF.
+
+** The autotest macro AT_CHECK_NOESCAPE is now documented.
+
 * Major changes in Autoconf 2.63b (2009-03-31) [beta]
   Released by Eric Blake, based on git versions 2.63.*.

diff --git a/doc/autoconf.texi b/doc/autoconf.texi
index 8dd22d6..57e0969 100644
--- a/doc/autoconf.texi
+++ b/doc/autoconf.texi
@@ -22273,34 +22273,70 @@ Writing Testsuites
 be a single shell word that expands into a single file name.
 @end defmac

address@hidden AT_CHECK (@var{commands}, @dvar{status, 0}, @dvar{stdout, }, @
-  @dvar{stderr, }, @ovar{run-if-fail}, @ovar{run-if-pass})
address@hidden AT_CHECK (@var{commands}, @dvar{status, 0}, @ovar{stdout}, @
+  @ovar{stderr}, @ovar{run-if-fail}, @ovar{run-if-pass})
address@hidden AT_CHECK_NOESCAPE (@var{commands}, @dvar{status, 0}, 
@ovar{stdout}, @
+  @ovar{stderr}, @ovar{run-if-fail}, @ovar{run-if-pass})
 @atindex{CHECK}
address@hidden
 Execute a test by performing given shell @var{commands}.  These commands
 should normally exit with @var{status}, while producing expected
 @var{stdout} and @var{stderr} contents.  If @var{commands} exit with
-status 77, then the whole test group is skipped.  Otherwise, if this test
+unexpected status 77, then the rest of the test group is skipped.  If
address@hidden exit with unexpected status 99, then the test group is
+immediately failed.  Otherwise, if this test
 fails, run shell commands @var{run-if-fail} or, if this test passes, run shell
 commands @var{run-if-pass}.

 This macro must be invoked in between @code{AT_SETUP} and @code{AT_CLEANUP}.

address@hidden Previously, we had this:
address@hidden  The @var{commands} @emph{must not} redirect the standard 
output, nor the
address@hidden  standard error.
address@hidden to prevent trigerring the double redirect bug on Ultrix, see
address@hidden `File Descriptors'.  This was too restricting, and Ultrix is 
pretty
address@hidden much dead, so we dropped the limitation; the obvious workaround 
on
address@hidden Ultrix is to use a working shell there.
-
-If @var{status}, or @var{stdout}, or @var{stderr} is @samp{ignore}, then
-the corresponding value is not checked.
-
-The special value @samp{expout} for @var{stdout} means the expected
-output of the @var{commands} is the content of the file @file{expout}.
-If @var{stdout} is @samp{stdout}, then the standard output of the
address@hidden is available for further tests in the file @file{stdout}.
-Similarly for @var{stderr} with @samp{experr} and @samp{stderr}.
+If @var{status} is the literal @samp{ignore}, then the corresponding
+exit status is not checked, except for the special cases of 77 (skip)
+and 99 (hard failure).  The existence of hard failures allows one to
+mark a test as an expected failure with @code{AT_XFAIL_IF} because a
+feature has not yet been implemented, but to still distinguish between
+gracefully handling the missing feature and dumping core.  A hard
+failure also inhibits post-test actions in @var{run-if-fail}.
+
+If the value of the @var{stdout} or @var{stderr} parameter is one of the
+literals in the following table, then the test treats the output
+according to the rules of that literal.  Otherwise, the value of the
+parameter is treated as text that must exactly match the output given by
address@hidden on standard out and standard error (including an empty
+parameter for no output); any differences are captured in the testsuite
+log and the test is failed.  The difference between @code{AT_CHECK} and
address@hidden is that only the latter performs shell
+expansions on comparison text given in the @var{stdout} and @var{stderr}
+arguments.
+
address@hidden @samp
address@hidden ignore
+The content of the output is ignored, but still captured in the
+testsuite log.  This is valid for both @var{stdout} and @var{stderr}.
+
address@hidden stdout
+For the @var{stdout} parameter, capture the content of standard output
+to both the file @file{stdout} and the testsuite log.  Subsequent
+commands in the test group can then post-process the file.  This action
+is often used when it is desired to use @command{grep} to look for a
+substring in the output, or when the output must be post-processed to
+normalize error messages into a common form.
+
address@hidden stderr
+Like @samp{stdout}, except that it only works for the @var{stderr}
+parameter, and the standard error capture file will be named
address@hidden
+
address@hidden expout
+For the @var{stdout} parameter, compare standard output contents with
+the previously created file @file{expout}, and list any differences in
+the testsuite log.
+
address@hidden experr
+Like @samp{expout}, except that it only works for the @var{stderr}
+parameter, and the standard error contents are compared with
address@hidden
address@hidden table
 @end defmac


diff --git a/lib/autotest/general.m4 b/lib/autotest/general.m4
index 906e481..80d7f66 100644
--- a/lib/autotest/general.m4
+++ b/lib/autotest/general.m4
@@ -324,27 +324,33 @@ at_fn_log_failure ()
   exit 1
 }

-AS_FUNCTION_DESCRIBE([at_fn_check_skip], [EXIT-CODE],
-[Check whether EXIT-CODE is the special exit code 77, and if so exit the shell
-with that same exit code.])
+AS_FUNCTION_DESCRIBE([at_fn_check_skip], [EXIT-CODE LINE],
+[Check whether EXIT-CODE is a special exit code (77 or 99), and if so exit
+the test group subshell with that same exit code.  Use LINE in any report
+about test failure.])
 at_fn_check_skip ()
 {
   case $[1] in
+    99) echo 99 > "$at_status_file"; at_failed=:
+        AS_ECHO(["$[2]: hard failure"]); exit 99;;
     77) echo 77 > "$at_status_file"; exit 77;;
   esac
 }

 AS_FUNCTION_DESCRIBE([at_fn_check_status], [EXPECTED EXIT-CODE LINE],
-[Check whether EXIT-CODE is the expected exit code, and if so do nothing.
-Otherwise, if it is 77 exit the shell with that same exit code; if it is
-anything else print an error message and fail the test.])
+[Check whether EXIT-CODE is the EXPECTED exit code, and if so do nothing.
+Otherwise, if it is 77 or 99, exit the test group subshell with that same
+exit code; if it is anything else print an error message referring to LINE,
+and fail the test.])
 at_fn_check_status ()
 {
 dnl This order ensures that we don't `skip' if we are precisely checking
-dnl $? = 77.
+dnl $? = 77 or $? = 99.
   case $[2] in
     $[1] ) ;;
     77) echo 77 > "$at_status_file"; exit 77;;
+    99) echo 99 > "$at_status_file"; at_failed=:
+        AS_ECHO(["$[3]: hard failure"]); exit 99;;
     *) AS_ECHO(["$[3]: exit code was $[2], expected $[1]"])
       at_failed=:;;
   esac
@@ -1130,11 +1136,16 @@ at_fn_group_postprocess ()
       report this failure to <AT_PACKAGE_BUGREPORT>.
 _ATEOF
     AS_ECHO(["$at_setup_line"]) >"$at_check_line_file"
-    at_xfail=no at_status=99
+    at_status=99
   fi
   $at_verbose AS_ECHO_N(["$at_group. $at_setup_line: "])
   AS_ECHO_N(["$at_group. $at_setup_line: "]) >> "$at_group_log"
   case $at_xfail:$at_status in
+    *:99)
+       at_msg='FAILED ('`cat "$at_check_line_file"`')'
+       at_res=fail
+       at_errexit=$at_errexit_p
+       ;;
     yes:0)
        at_msg="UNEXPECTED PASS"
        at_res=xpass
diff --git a/tests/autotest.at b/tests/autotest.at
index d9c6081..8992852 100644
--- a/tests/autotest.at
+++ b/tests/autotest.at
@@ -253,6 +253,16 @@ AT_CHECK_AT_TEST([Skip],
   [], [], [], [], [],
   [AT_CHECK([grep skipped micro-suite.log], [], [ignore])])

+AT_CHECK_AT_TEST([Hard fail],
+  [AT_CHECK([exit 99])
+   AT_CLEANUP
+   AT_SETUP([another test])
+   AT_XFAIL_IF([:])
+   AT_CHECK([exit 99])],
+  [], [1], [], [ignore], [],
+  [AT_CHECK([grep '2 failed unexpectedly' micro-suite.log], [], [ignore])
+   AT_CHECK([grep ok micro-suite.log], [1])])
+
 AT_CHECK_AT_TEST([Syntax error],
   [AT_CHECK([:])
    AT_CLEANUP
@@ -277,6 +287,35 @@ AT_CHECK_AT_TEST([errexit],
    AT_CHECK([grep "1 .* inhibited subsequent" stderr], [], [ignore])],
   [--errexit])

+AT_CHECK_AT_TEST([Cleanup],
+  [AT_CHECK([test ! -f cleanup.success && test ! -f cleanup.failure])
+   AT_CHECK_NOESCAPE([exit $value], [ignore], [$output],
+     [], [touch cleanup.failure], [touch cleanup.success])],
+  [], [], [], [],
+  [AT_KEYWORDS([AT@&address@hidden)
+    output=; export output],
+  [AT_CHECK([test -d micro-suite.dir/1])
+   AT_CHECK([test -f micro-suite.dir/1/cleanup.success])
+   AT_CHECK([test ! -f micro-suite.dir/1/cleanup.failure])
+
+   AT_CHECK([$CONFIG_SHELL ./micro-suite -d value=1], [], [ignore])
+   AT_CHECK([test -f micro-suite.dir/1/cleanup.success])
+   AT_CHECK([test ! -f micro-suite.dir/1/cleanup.failure])
+
+   AT_CHECK([$CONFIG_SHELL ./micro-suite -d value=1 output=mismatch],
+     [1], [ignore], [ignore])
+   AT_CHECK([test ! -f micro-suite.dir/1/cleanup.success])
+   AT_CHECK([test -f micro-suite.dir/1/cleanup.failure])
+
+   AT_CHECK([$CONFIG_SHELL ./micro-suite -d value=77], [], [ignore])
+   AT_CHECK([test ! -f micro-suite.dir/1/cleanup.success])
+   AT_CHECK([test ! -f micro-suite.dir/1/cleanup.failure])
+
+   AT_CHECK([$CONFIG_SHELL ./micro-suite -d value=99], [1], [ignore], [ignore])
+   AT_CHECK([test ! -f micro-suite.dir/1/cleanup.success])
+   AT_CHECK([test ! -f micro-suite.dir/1/cleanup.failure])
+  ], [-d value=0])
+
 ## ----------------------------------------------------- ##
 ## Newlines and command substitutions in test commands.  ##
 ## ----------------------------------------------------- ##
-- 
1.6.1.2