[PATCH] {GSoC} docs: improve, extend and fix documentation on TAP support

[Stefano Lattarini]

* doc/automake.texi ("Using the TAP test protocol"): Divide this
section into ...
("Introduction to TAP", "Use TAP with the Automake test harness",
"Incompatibilities with other TAP parsers and drivers", "Links
and external resources"): ... these subsections, extend them by
adding more information and examples, and improve them by removing
incomplete and/or temporary wordings and TODO items.
("Script-based Testsuites", "Parallel Test Harness"): Add a couple
of anchors to improve the granularity of cross-references.
* tests/tap-doc2.test: New test, verifying the correctness of the
new examples given in the manual.
* tests/Makefile.am (tap_other_tests): Add the new test.
---
 ChangeLog           |    9 ++-
 doc/automake.texi   |  181 ++++++++++++++++++++++++++++++++++++--------------
 tests/Makefile.am   |    1 +
 tests/Makefile.in   |    1 +
 tests/tap-doc2.test |  139 +++++++++++++++++++++++++++++++++++++++
 5 files changed, 276 insertions(+), 55 deletions(-)
 create mode 100755 tests/tap-doc2.test

diff --git a/ChangeLog b/ChangeLog
index d09b308..6d2a016 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -6,12 +6,13 @@
        ("Introduction to TAP", "Use TAP with the Automake test harness",
        "Incompatibilities with other TAP parsers and drivers", "Links
        and external resources"): ... these subsections, extend them by
-       adding more information, and improve them by removing incomplete
-       and/or temporary wordings and TODO items.  Also ...
-       ("Links and external resources"): ... add a link to the `tap4j'
-       project.
+       adding more information and examples, and improve them by removing
+       incomplete and/or temporary wordings and TODO items.
        ("Script-based Testsuites", "Parallel Test Harness"): Add a couple
        of anchors to improve the granularity of cross-references.
+       * tests/tap-doc2.test: New test, verifying the correctness of the
+       new examples given in the manual.
+       * tests/Makefile.am (tap_other_tests): Add the new test.
 
 2011-08-01  Stefano Lattarini  <address@hidden>
 
diff --git a/doc/automake.texi b/doc/automake.texi
index 8ac3687..c1ee0a5 100644
--- a/doc/automake.texi
+++ b/doc/automake.texi
@@ -330,6 +330,13 @@ Simple Tests
 * Serial Test Harness::         Older (and obsolescent) serial test harness
 * Parallel Test Harness::       Generic concurrent test harness
 
+Using the TAP test protocol
+
+* Introduction to TAP::
+* Use TAP with the Automake test harness::
+* Incompatibilities with other TAP parsers and drivers::
+* Links and external resources::
+
 Custom Test Drivers
 
 * Overview of Custom Test Drivers Support::
@@ -9534,7 +9541,7 @@ harness should remain untouched, and continue to work 
correctly.
 @subsection Introduction to TAP
 
 TAP, the Test Anything Protocol, is a simple text-based interface between
-a test harness and testing modules or programs.  The tests (also called
+testing modules or programs and a test harness.  The tests (also called
 ``TAP producers'' in this context) write test results in a simple format
 on standard output; a test harness (also called ``TAP consumer'') will
 parse and interpret these results, and properly present them to the user,
@@ -9543,72 +9550,144 @@ is accomplished can vary among different test 
harnesses.  The Automake
 parallel harness will present the results on the console in the usual
 fashion (@pxref{Testsuite progress on console}), and will use the
 @file{.trs} files (@pxref{Basics of test metadata}) to store the test
-results and related metadata.
+results and related metadata.  Apart from that, it will try to remain
+as much compatible as possible with pre-existing and widespread utilities,
+such as the @uref{http://search.cpan.org/~andya/Test-Harness/bin/prove,
address@hidden utility}, at least for the simpler usages.
 
 TAP started its life as part of the test harness for Perl, but today
 it has been (mostly) standardized, and has various independent
-implementations in different languages (among them, C, C++, Perl,
-Python, PHP, and Java).
+implementations in different languages; among them, C, C++, Perl,
+Python, PHP, and Java.  For a semi-official specification of the
+TAP protocol, please refer to the documentation of
address@hidden://search.cpan.org/~petdance/Test-Harness/lib/Test/Harness/TAP.pod,
+      @samp{Test::Harness::TAP}}.
 
-The most relevant real-world usages of TAP are obviously in the testsuite
-of @command{perl} and of many perl modules; but also other important
+The most relevant real-world usages of TAP are obviously in the testsuites
+of @command{perl} and of many perl modules.  Still, also other important
 third-party packages, such as @uref{http://git-scm.com/, @command{git}},
-uses TAP in their testsuite.
-
-The ``core'' of the TAP protocol is probably best summarized by this
-brief excerpt taken from the TAP specification at
address@hidden://search.cpan.org/~petdance/Test-Harness/lib/Test/Harness/TAP.pod,
-      @samp{Test::Harness::TAP}}:
address@hidden
-A harness must only read TAP output from standard output and not
-from standard error.  Lines written to standard output matching
address@hidden/^(not )?ok\b/} must be interpreted as test lines.  All other
-lines must not be considered test output.
address@hidden quotation
address@hidden
-Please refer to the above link for the complete TAP specifications.
+use TAP in their testsuite.
 
 @node Use TAP with the Automake test harness
 @subsection Use TAP with the Automake test harness
 
 Currently, the TAP driver that comes with Automake requires a perl
-interpreter to work, and requires the developer taking various steps to
-set it up.  You'll have grab the @file{tap-driver} script from the
-Automake distribution by hand, copy in in your source tree, add code
-to @file{configure.ac} to search a perl interpreter and define the
address@hidden(PERL)} variable accordingly, and use the Automake support for
-third-party test drivers to.
-
-Apart from the options that must be accepted by all the Automake test
-drivers, the  @file{tap-driver}
-Option @option{--ignore-exit} causes the driver to ignore the exit
-status of the test scripts; by default, the driver will report an
-error if the script exit with status != 0.
address@hidden: also add a @option{--no-ignore-exit} option, for
-completeness?
-
-By default, TAP diagnostic (i.e., lines beginning with the @samp{#}
-character) are copied only in the @file{.log} file.  The option
address@hidden causes the driver to display them in the testsuite
-progress output too (@emph{TODO}: give example).  The option
address@hidden restore the default behaviour.
+interpreter to work, and requires various by-hand steps on the
+developer's part (this should be fixed in future Automake versions).
+You'll have grab the @file{tap-driver} script from the Automake
+distribution by hand, copy it in your source tree, add code to
address@hidden to search a perl interpreter and to define the
address@hidden(PERL)} variable accordingly, and use the Automake support
+for third-party test drivers to instruct the harness to use the
address@hidden to run your TAP-producing tests.  See the example
+below for clarification.
+
+Apart from the options common to all the Automake test drivers
+(@pxref{Command-line arguments for test drivers}), the @file{tap-driver}
+supports the following options, whose names are chosen for enhanced
+compatibility with the @command{prove} utility.
 
address@hidden @option
address@hidden --ignore-exit
+Causes the test driver to ignore the exit status of the test scripts;
+by default, the driver will report an error if the script exit with a
+non-zero status.
address@hidden --comments
+Instruct the test driver to display TAP diagnostic (i.e., lines beginning
+with the @samp{#} character) in the testsuite progress output too; by
+default, TAP diagnostic is only copied in the @file{.log} file.
address@hidden --no-comments
+Revert the effects of @option{--comments}.
address@hidden --merge
+Instruct the test driver to merge the test scripts' standard error into
+their standard output.  This is necessary if you want to ensure that
+diagnostics from the test scripts are displayed in the correct order
+relative to test results; this can be of great help in debugging
+(especially if your test scripts are shell scripts run with shell
+tracing active).  As a downside, this option might cause the test
+harness to get confused if anything that appears on standard error
+looks like a test result.
address@hidden --no-merge
+Revert the effects of @option{--merge}.
address@hidden table
 
 @noindent
-Here is an example of how to setup  be done
+Here is an example of how the TAP driver can be set up and used.
+
address@hidden Keep in sync with tap-doc2.test.
 @example
-TEST_LOG_DRIVER = $(srcdir)/tap-driver
-TESTS = foo.test bar.test
address@hidden example
+% @kbd{cat configure.ac}
+AC_INIT([GNU Try Tap], [1.0], [bug-automake@@gnu.org])
+AC_CONFIG_AUX_DIR([build-aux])
+AM_INIT_AUTOMAKE([foreign parallel-tests -Wall -Werror])
+AC_CONFIG_FILES([Makefile])
+AC_REQUIRE_AUX_FILE([tap-driver])
+AC_PATH_PROG([PERL], [perl])
+test -n "$PERL" || AC_MSG_ERROR([perl not found])
+$PERL -MTAP::Parser -e 1 || AC_MSG_ERROR([TAP::Parser not found])
+AC_OUTPUT
+
+% @kbd{cat Makefile.am}
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/build-aux/tap-driver
+TESTS = foo.test bar.test baz.test
+EXTRA_DIST = $(TESTS)
 
address@hidden: Give example of output from @command{prove} command (when
-used in ``verbose'' mode, with option @option{-v}), to which we'd like
-to remain somewhat compatible compatible, while also (and foremost)
-being compatible with the default @option{parallel-tests} driver.
+% @kbd{cat foo.test}
+#!/bin/sh
+echo 1..4 # Number of tests to be executed.
+echo 'ok 1 - Swallows fly'
+echo 'not ok 2 - Caterpillars fly # TODO metamorphosis in progress'
+echo 'ok 3 - Pigs fly # SKIP not enough acid'
+echo '# I just love word plays ...'
+echo 'ok 4 - Flies fly too :-)'
+
+% @kbd{cat bar.test}
+#!/bin/sh
+echo 1..3
+echo 'not ok 1 - Bummer, this test has failed.'
+echo 'ok 2 - This passed though.'
+echo 'Bail out! Ennui kicking in, sorry...'
+echo 'ok 3 - This will not be seen.'
+
+% @kbd{cat baz.test}
+#!/bin/sh
+echo 1..1
+echo ok 1
+# Exit with error, even if all the test case has been successful.
+exit 7
 
address@hidden: Example of output from out own TAP testsuite driver.
-Point out similarities with the output from @command{prove --verbose}
-and from the default @option{parallel-tests} driver.
+% @kbd{cp @var{PREFIX}/share/address@hidden/tap-driver .}
+% @kbd{autoreconf -vi && ./configure && make check}
+...
+PASS: foo.test 1 - Swallows fly
+XFAIL: foo.test 2 - Caterpillars fly # TODO metamorphosis in progress
+SKIP: foo.test 3 - Pigs fly # SKIP not enough acid
+PASS: foo.test 4 - Flies fly too :-)
+FAIL: bar.test 1 - Bummer, this test has failed.
+PASS: bar.test 2 - This passed though.
+ERROR: bar.test - Bail out! Ennui kicking in, sorry...
+PASS: baz.test 1
+ERROR: baz.test - exited with status 7
+...
+Please report to bug-automake@@gnu.org
+...
+% @kbd{echo exit status: $?}
+exit status: 1
+
address@hidden Keep the "skewed" indentation below, it produces pretty PDF 
output.
+% @kbd{env TEST_LOG_DRIVER_FLAGS='--comments --ignore-exit' \
+      TESTS='foo.test baz.test' make -e check}
+...
+PASS: foo.test 1 - Swallows fly
+XFAIL: foo.test 2 - Caterpillars fly # TODO metamorphosis in progress
+SKIP: foo.test 3 - Pigs fly # SKIP not enough acid
+# foo.test: I just love word plays...
+PASS: foo.test 4 - Flies fly too :-)
+PASS: baz.test 1
+...
+% @kbd{echo exit status: $?}
+exit status: 0
address@hidden example
 
 @node Incompatibilities with other TAP parsers and drivers
 @subsection Incompatibilities with other TAP parsers and drivers
diff --git a/tests/Makefile.am b/tests/Makefile.am
index b8126f2..6a45fcf 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -1184,6 +1184,7 @@ tap-bad-prog.test \
 tap-bad-prog2.test \
 tap-basic.test \
 tap-doc.test \
+tap-doc2.test \
 tap-empty.test \
 tap-more.test \
 tap-more2.test \
diff --git a/tests/Makefile.in b/tests/Makefile.in
index 82e62a3..a085762 100644
--- a/tests/Makefile.in
+++ b/tests/Makefile.in
@@ -1417,6 +1417,7 @@ tap-bad-prog.test \
 tap-bad-prog2.test \
 tap-basic.test \
 tap-doc.test \
+tap-doc2.test \
 tap-empty.test \
 tap-more.test \
 tap-more2.test \
diff --git a/tests/tap-doc2.test b/tests/tap-doc2.test
new file mode 100755
index 0000000..63bdc23
--- /dev/null
+++ b/tests/tap-doc2.test
@@ -0,0 +1,139 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# Check that an example given in the documentation really works.
+# See section "Using the TAP test protocol", subsection "Use TAP
+# with the Automake test harness".
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+# Ensure we are run from the right directory.
+# (The last thing we want is to delete some random user files.)
+test -f ../defs-static && test -f ../defs || Exit 99
+rm -f *
+
+cat > Makefile.am <<'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/build-aux/tap-driver
+TESTS = foo.test bar.test baz.test
+EXTRA_DIST = $(TESTS)
+END
+
+cat > configure.ac <<'END'
+AC_INIT([GNU Try Tap], [1.0], address@hidden)
+AC_CONFIG_AUX_DIR([build-aux])
+AM_INIT_AUTOMAKE([foreign parallel-tests -Wall -Werror])
+AC_CONFIG_FILES([Makefile])
+AC_REQUIRE_AUX_FILE([tap-driver])
+AC_PATH_PROG([PERL], [perl])
+test -n "$PERL" || AC_MSG_ERROR([perl not found])
+$PERL -MTAP::Parser -e 1 || AC_MSG_ERROR([TAP::Parser not found])
+AC_OUTPUT
+END
+
+cat > foo.test <<'END'
+#!/bin/sh
+echo 1..4 # Number of tests to be executed.
+echo 'ok 1 - Swallows fly'
+echo 'not ok 2 - Caterpillars fly # TODO metamorphosis in progress'
+echo 'ok 3 - Pigs fly # SKIP not enough acid'
+echo '# I just love word plays...'
+echo 'ok 4 - Flies fly too :-)'
+END
+
+cat > bar.test <<'END'
+#!/bin/sh
+echo 1..3
+echo 'not ok 1 - Bummer, this test has failed.'
+echo 'ok 2 - This passed though.'
+echo 'Bail out! Ennui kicking in, sorry...'
+echo 'ok 3 - This will not be seen.'
+END
+
+cat > baz.test <<'END'
+#!/bin/sh
+echo 1..1
+echo ok 1
+# Exit with error, even if all the test case has been successful.
+exit 7
+END
+
+chmod a+x *.test
+
+mkdir build-aux
+cp "$top_testsrcdir"/lib/tap-driver build-aux
+
+(export AUTOMAKE ACLOCAL AUTOCONF && $AUTORECONF -vi) || Exit 1
+
+./configure --help # Sanity check.
+./configure || skip_ "configure failed"
+
+case $MAKE in *\ -j*) skip_ "can't work easily with concurrent make";; esac
+
+$MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+
+cat > exp <<'END'
+PASS: foo.test 1 - Swallows fly
+XFAIL: foo.test 2 - Caterpillars fly # TODO metamorphosis in progress
+SKIP: foo.test 3 - Pigs fly # SKIP not enough acid
+PASS: foo.test 4 - Flies fly too :-)
+FAIL: bar.test 1 - Bummer, this test has failed.
+PASS: bar.test 2 - This passed though.
+ERROR: bar.test - Bail out! Ennui kicking in, sorry...
+PASS: baz.test 1
+ERROR: baz.test - exited with status 7
+END
+
+sed -n '/^PASS: foo\.test/,/^ERROR: baz\.test/p' stdout > got
+
+cat exp
+cat got
+diff exp got
+
+grep '^Please report to address@hidden' stdout
+
+env \
+  TESTS='foo.test baz.test' \
+  TEST_LOG_DRIVER_FLAGS='--comments --ignore-exit' \
+  $MAKE -e check >stdout || { cat stdout; Exit 1; }
+
+cat > exp <<'END'
+PASS: foo.test 1 - Swallows fly
+XFAIL: foo.test 2 - Caterpillars fly # TODO metamorphosis in progress
+SKIP: foo.test 3 - Pigs fly # SKIP not enough acid
+# foo.test: I just love word plays...
+PASS: foo.test 4 - Flies fly too :-)
+PASS: baz.test 1
+END
+
+sed -n '/^PASS: foo\.test/,/^PASS: baz\.test/p' stdout > got
+
+cat exp
+cat got
+diff exp got
+
+# Sanity check the distribution.
+cat > bar.test <<'END'
+#!/bin/sh
+echo 1..1
+echo ok 1
+END
+echo AM_TEST_LOG_DRIVER_FLAGS = --ignore-exit >> Makefile.in
+./config.status Makefile
+$MAKE distcheck
+
+:
-- 
1.7.2.3