parallel-tests use case (was: split check target into check and test targets)

[Ralf Wildenhues]

* Baurzhan Ismagulov wrote on Wed, Feb 24, 2010 at 11:11:28PM CET:
> On Wed, Feb 24, 2010 at 08:49:26PM +0100, Ralf Wildenhues wrote:
> > Dunno if you've heard of the parallel-tests option
> ...
> > It also allows you to sort of rerun only tests that are out of date
> 
> Wow! Does it work with DejaGNU?

No; it is a replacement for the simple TESTS = ... driver.  The
DejaGNU handling in Automake is really separate from that.

> Which automake version is required?

1.11, but you really want 1.11.1 for a small bugfix in the driver, and a
security fix in the automake-generated dist* rules.

> > (once you have formulated dependencies for not only test programs, but
> > also test results)
> 
> Can't automake do the former automatically?

Yes.

> Could you give an example use case for the latter?

Say I have lots of tests whose outcome (of course) depends on the test
source and some library being tested, but also on some parameter file.
I would like test compilation to be done as lazily as possible, because
linking take long (think C++ template libraries), and the edit-compile-
test cycle should be as quick as possible.

For completeness, the example also has baz.test which is a shell script.

Say, foo.test and bar.test both are compiled programs and both read
foobar.dat, and when I change the latter, I'd like to see just those
tests being rerun that read the file.

Now, I could achieve that with
  make check TESTS='foo.test bar.test'

but for that I'd have to remember which test depends on foobar.dat.
Instead, the Makefile.am has the information, and I just run
  make check RECHECK_LOGS=

to only rerun those tests whose results are not up to date.  To see the
difference, try something like
  make check                 # run everything once
  touch foobar.dat           # edit ...
  make check RECHECK_LOGS=   # baz.test is not rerun
  touch foo.c                # edit ...
  make check RECHECK_LOGS=   # bar.test, baz.test are not rerun

or things like
  make clean
  make -j check              # bar.test maybe run before baz.test is linked

  make recheck               # rerun only failed (FAIL, XPASS) tests


Another example case for dependencies that I haven't shown here is when
some tests need to be run before others, because the latter use data
produced by the former.

Cheers,
Ralf

cat >configure.ac <<\EOF
AC_INIT([parallel-tests example package], [1])
AM_INIT_AUTOMAKE([foreign])
AC_PROG_CC
AC_PROG_RANLIB
AC_CONFIG_FILES([Makefile])
AC_OUTPUT
EOF

cat >Makefile.am <<\EOF
# Enable the parallel-tests driver (could also be done in AM_INIT_AUTOMAKE)
# and colored output.  The former is new in 1.11
AUTOMAKE_OPTIONS = 1.11 parallel-tests color-tests

# Some library being tested
LDADD = libfoo.a
noinst_LIBRARIES = libfoo.a

# Test suffixes: Foo.test will be turned into Foo.log by 'make check'
# (rather than Foo.test.log).  Specifying suffixes is necessary when
# we want to specifying additional dependencies
TEST_EXTENSIONS = .test
# These are the tests to be run
TESTS = foo.test bar.test baz.test

# Compiled tests go into EXTRA_PROGRAMS not check_PROGRAMS for lazy updating
EXTRA_PROGRAMS = foo.test bar.test
# EXTRA_PROGRAMS need to be cleaned manually, see automake.info(Uniform)
CLEANFILES = $(EXTRA_PROGRAMS)

# Extra data files and scripts we need to distribute
EXTRA_DIST = foobar.dat baz.test

# The tests foo.test and bar.test also read the data file foobar.dat,
# so they should be rerun when that file changes
foo.log: foobar.dat
bar.log: foobar.dat
EOF

touch libfoo.c foobar.dat

cat >foo.c <<\EOF
/* source for foo.test */
int main () { return 0; }
EOF

cat >bar.c <<\EOF
/* source for bar.test */
int main () { return 0; }
EOF

cat >baz.test <<\EOF
#! /bin/sh
exit 0
EOF

autoreconf -vi
./configure

make check                      # observe baz.test failure
make recheck                    # only baz.test gets rerun
chmod +x ./baz.test
make recheck                    # better now