Gripes about Grep test suite

[Eli Zaretskii]

I'm not sure this warrants a Savannah bug report, but if tell me it
does, I will submit one.

During the past few days I needed to investigate several failures of
Grep 2.9 and 2.10 tests on MS-Windows.  I must say that I found the
test suite less user-friendly than I'd like, when it comes to digging
up the reason for a failure.  Here's why:

 . There's no README or other file in the `tests' directory with
   instructions on running and re-running the tests.

   The first thing one needs to know how to do is run a single test,
   but this is not explained anywhere.  As you-all know only too well,
   wading through the maze of autoconf-generated Makefiles trying to
   figure this out is not for the faint at heart, what with half a
   dozen levels of indirection in every target.  It took me a while to
   figure that out.  (Later, I found the recipe buried inside HACKING,
   but that file is not part of a release tarball, and I worked with
   an official release.)

 . Another annoyance is that each test script typically runs several
   individual test commands, but the diagnostics left on the log files
   is not detailed enough to pinpoint the specific command that
   failed.  For example, Spencer tests leave this laconic message:

          Spencer ere test #113 failed

   It is entirely not trivial to find out what Grep command was run
   and failed, because the shell script which runs all those tests is
   deleted, even when there's a failure.  I needed to find out how to
   produce that script, then generate it manually, and only then I
   could see the failed command.

   Other tests leave even less information.  E.g., if Grep wrote some
   error message to stderr, that error message gets written to the log
   file with no context information at all, so it is hard to know
   which part(s) of the test triggered the error.  I found myself
   adding "set -x" to various scripts to unlock these mysteries.

   May I suggest that the log file shows the full command that failed?

 . Yet another problem is that every run of "make check" wipes out the
   test-suite.log file.  So if you want to consult past results, you
   need to save the file under a different name.  WIBNI the file name
   included some unique string, like the time of the test? this would
   avoid wiping out previous results.

 . Last, but not least: the test suite requires a `timeout' command
   for some of the tests.  What it does when this command is not found
   is sub-optimal: it skips the tests that require this command.  This
   means that Grep built on a new platform cannot be fully tested
   without also building a fairly recent version of Coreutils.

   May I suggest to include in the test suite a shell script that
   would emulate this command for the simple uses needed by the tests?
   There are a couple of such scripts floating around (I used one of
   them), and I think they are good enough for this purpose.

Thank you for listening to my whining.