Re: running tests under in a tests/ directory

[Ralf Wildenhues]

[ <http://thread.gmane.org/gmane.comp.sysutils.automake.general/10749> ]

Hello Alejandro, Martin,

* aaragon wrote on Fri, May 29, 2009 at 05:44:57AM CEST:
> 
> I found a Makefile.am file on the internet to run tests under a tests
> folder. The code looks like follows:

[ *snip*, looks an awful lot like
  <http://www.bioinf.uni-freiburg.de/~mmann/HowTo/automake.html> ]

Martin, a while ago I went over your Automake page and noted some issues
with it, and proposed a newer version that uses Automake 1.11 features.
1.11 has been released now, would you be so nice to update your web page
to reflect these fixes?

A few nits inline below, and part of my recommendation follows at the
end.  It will also help you, Alejandro, avoid the pitfall of letting
code be compiled by GNU make implicit rules (which uses different
variable names and thus does not pick up the include paths).

> check:        all

You can use check-local; and check will depend upon all-am anyway (the
part of 'all' that is updated in this directory), maybe that is
sufficient for your needs.

>       @echo "Running tests"
>       @make test -s -C $(TSTDIR)

Non-GNU make doesn't support -C, and you should use $(MAKE) so that
parallel make and make -k and make -n work; you can use
  cd $(TSTDIR) && $(MAKE) $(AM_MAKEFLAGS) -s test

but if I were you, I'd just let the automake-provided check rule do its
work in the toplevel directory.

> TESTSOURCE = $(wildcard test*.cpp)

GNU make-specific.

> TESTOBJS=$(TESTSOURCE:%.cpp=%.o)

Likewise.  More instances below.

> # additional linker and compiler flags
> LDFLAGS = -L$(top_srcdir)/yafeq -lyafeq $(LDFLAGS)

For in-tree libtool libraries, please always specify them as relative
paths to the .la file; also, don't stick them in LDFLAGS; thus, use
  LDADD = $(top_srcdir)/yafeq/libyafeq.la

(or even LIBS).

> address@hidden/Documents/workspace/yafeq/tests$mt
>  Compiling IntegrationRules.cpp                   OK
> BD Software STL Message Decryptor v3.10 for gcc 2/3/4
> In file included from testIntegrationRules.cpp:1:
> ../yafeq/quadrature.hpp:14:36: error: cpputils/map_adaptor.hpp: No such file
> or
>     directory
> 
> 
> Now, the file cpputils/map_adaptor.hpp does exist, and I actually have a
> test for detecting the cpputils library. I guess the CPPFLAGS that includes
> the correct path is not being passed to this piece of code written in the
> Makefile.am.


What follows is my suggestion that I wrote to Martin about the tests
thing.  I haven't tested it much, so if you have issues with them,
please speak up.

Cheers,
Ralf

For the tests example, I actually have a more radical suggestion:
replace it completely with a setup based on automake support for simple
tests using the TESTS variable.

Since I am lazy, and we have added several new features to the upcoming
Automake 1.11 release which can be of help here, I will further suggest
to require that next version, and use the 'parallel-tests' test driver
which has several new features.  In the following, I will also use some
more Automake 1.11 features like AM_DEFAULT_SOURCE_EXT.
(The stuff can more or less also be done with 1.10 features, but it will
be uglier, and, as already noted, I am lazy.  :-)

However, for portability, you will not be able to use wildcards.  So you
will need to mention each test once.

BTW, in your example is a line of the form
| LDFLAGS = $(LDFLAGS)

If this really were in your code, then it would provoke an endless
recursion, and an error from 'make' if referenced anywhere.

| # additional linker and compiler flags
| CXXFLAGS = -I$(top_srcdir)/src $(CXXFLAGS)

You should not override CXXFLAGS: that variable is reserved for the
user.  Automake provides AM_CXXFLAGS for that case.  However, include
paths are for the preprocessor, so AM_CPPFLAGS should be used here.

With the tests/Makefile.am and the test driver files below, there is no
need to modify toplevel Makefile.am, and you can run the testsuite with
  make check

or even
  make -j3 check

if you so prefer.  :-)

A word of caution: this code is completely untested.

Another word of caution: you need to run
  make generate-verified-files

once before you can run "make dist".  If you do not like that kind of
setup, please speak up.  (The dependencies are not completely precise
yet.)

You can update individual .verified files with something like
  make generate-verified-files TESTS='test1 test2'

Please note that 'diff -w' is not portable, at least not POSIX.
I simply used diff in the driver, but you may want to make that
configurable or so.

Cheers,
Ralf

--- tests/diff-driver ---
#! /bin/sh

# Compare output of test programs with expected output.
# Usage: diff-driver [--create] TEST-PROGRAM
#
# TEST-PROGRAM is the program to run.
# With --create, a .verified file is created.
# Without it, a .lastout file is created and compared
# to an existing .verified file; the diff is produced on stdout.
# $srcdir should contain the path to the source directory.
#
# .lastout files are created in the build tree; the TEST-PROGRAM
# may or may not live in the source tree; an existing .verified
# file may live in the source tree (if it came with the distribution)
# or the build tree, but is updated in the build tree.

if test -z "$srcdir"; then
  echo "$0: \`\$srcdir' is not set" >&2
  exit 1
fi

create=no
for arg
do
  case $arg in
    --create)
      create=yes ;;
    [!-]*)
      break ;;
  esac
  shift
done

test_program=$1
test_program_noext=${test_program%$EXEEXT}
case $test_program_noext in
  $srcdir/*)
    srcdirstrip=`echo "$srcdir" | sed 's|.|.|g'`
    test_stem=`echo "$test_program_noext" | sed "s|^$srcdirstrip/||"` ;;
  *)
    test_stem=$test_program_noext ;;
esac

output_file=$test_stem.lastout
verified_file=$test_stem.verified
if test "$create" = no && test ! -f "$verified_file"; then
  verified_file=$srcdir/$verified_file
  if test ! -f "$verified_file"; then
    echo "$0: cannot find .verified file \`$verified_file'" >&2
    exit 1
  fi
fi

if test "$create" = yes; then
  $test_program > $verified_file
  exit $?
else
  $test_program > $output_file || exit $?
  diff $verified_file $output_file
  exit $?
fi

--- tests/Makefile.am ---

# We use the new parallel-tests driver from Automake 1.11
# For nice and pretty output, we will be using color, too.
AUTOMAKE_OPTIONS = 1.10c parallel-tests color-tests

# Each program `foo' is compiled from `foo.cc', not `foo.c'.
AM_DEFAULT_SOURCE_EXT = .cc

# Add test programs here.
# If their sources do not end in .cc, or are made up of more than one
# file, then you need to provide a testX_SOURCES variable, too.
TESTEXECS = test1 test2 test3 ...

# All test executables are to be run by `make check'.
TESTS = $(TESTEXECS)

# Let automake be aware that all test executables are programs to be
# compiled (and $(EXEEXT) appended etc) using the normal PROGRAMS
# machinery.  We use EXTRA_ instead of check_, because the
# parallel-tests driver will cause the programs to be built as normal
# prerequisites just in time.
EXTRA_PROGRAMS = $(TESTEXECS)

AM_CPPFLAGS = -I$(top_srcdir)/src

# Tests are run with the diff-driver
LOG_COMPILER = ./diff-driver

# Run `make generate-verified-files' to create .verified files.
# This just runs all tests and passes --create to the diff-driver.
generate-verified-files:
        $(MAKE) $(AM_MAKEFLAGS) AM_LOG_FLAGS=--create check

# add verified output files to the distribution
EXTRA_DIST = $(TESTEXECS:=.verified)

MOSTLYCLEANFILES = $(TESTEXECS:=.lastout)
MAINTAINERCLEANFILES = $(TESTEXECS:=.verified)