bug#7868: splitting up test suites

[Ralf Wildenhues]

The testsuite is too large for MSYS.

We've finally reached the point where we have more than 1000
tests, $(TESTS) expands to 15k characters, and where 'make check' will
not work at all any more on MSYS, because it cannot spawn sh any more,
presumably in 'make check TESTS="..."'.  (MSYS make doesn't export
macros to the environment of spawned processes even without .NOEXPORT,
presumably otherwise lots of Makefiles would be really unusable here.)
This also clears up the spurious failure of sed a few days ago.


Here's a preliminary plan for multiple testsuites per Makefile.am.
It would be nice if this worked somehow:

# These are all specified by the user:
TEST_SUITE_LOGS = suite1.log suite2.log suite3.log ..
TEST_EXTENSIONS = .test ...
# these undergo $(EXEEXT) autoexpansion internally:
suite1_log_SOURCES = all.test aclocal7.test ...
suite2_log_SOURCES = suffix.test ...

# These are then produced by automake:
TEST_SUITE_HTMLS = $(TEST_SUITE_LOGS:.log=.html)
suite1_log_LOGS = $(am__helper_var:.test=.log)

mostlyclean-generic:
        -test -z "$(suite1_log_LOGS)" || rm -f $(suite1_log_LOGS)
        -test -z "$(suite2_log_LOGS)" || rm -f $(suite2_log_LOGS)


In the <suite>_SOURCES, $(EXEEXT) transformation should take place
(unless no-exeext is given, of course), just as is currently done for
TESTS and *_PROGRAMS.  Hmm, alternatively we could also require all
<suite>_SOURCES to be listed in $(TESTS), that would allow reuse of
this variable as well, at the cost of some specification redundance.

Open questions: how to produce nice results, both on stdout and in suite
log files.  One way is to merge all logs into a final TEST_SUITE_LOG
(that way we could also reuse that variable).  Another is to try to even
hide the summaries of the individual suites, iff a final suite is being
made.

Internally, it would be nice if the register_language could be exploited
(but this is not a requirement for this, as some features with tests are
distinctly different from other languages).

Cheers,
Ralf