Re: test infrastructure

[Francesco Salvestrini]

Hi Peter,

On Monday 03 August 2009, Peter Simons wrote:
> Hi Francesco,
>
>  > I just grabbed the same test-environment I use in my projects (that
>  > layout is in use both in savannah and in github ones) and slashed it
>  > until it worked into the archive.
>
> thank you very much for sharing that code. The patch applied fine and a
> "make check" run worked out of the box, too.

You're welcome.

> Now, I wonder how exactly I'd design a regression test for a given
> macro, say ACX_PTHREAD? I can create a small configure.ac script and
> write a test/test-acx_pthread.in driver to run that, but how do I verify
> whether that macro worked successfully?

I've added some tests and rearranged the structure slightly (patch provided in 
the "Comments, fixes and new items").

Use VERBOSE=x make check TESTS="test-<name>" to run <name> test only.

The provided tests are really simple at the moment, I had to fix some bugs 
introduced during the "port" thus I had less time to give to the test bodies.

I'll provide some more tests later, in order to drop down the learning curve 
(and fix the structure upon our requirements). A README into the tests 
directory could came handy. ACX_PTHREAD is tricky (whiteboxing) and I'll 
discuss about it later (you need to find if you can execute the test in order 
to avoid false positives).

The automake tests are good examples anyway.

Side-notes:

The "infrastructure" lays upon the automake testing suite thus:

For the exit codes:
0  -> test went ok
1  -> test failed
77 -> test to be skipped (defs.in use that exit code to skip the test without 
failure)

For the variables in Makefile.am:
FAIL_TEST    -> contains test that must fail
SUCCESS_TEST -> quite easy to understand
XFAIL_TEST   -> tests to be expecetd to fail

For test-xxx.in:
The 'required' variable has been replaced by must_have and should_have. 
must_have variable reports all the required tools needed to run the test, The 
should_have variable reports a set of tools that should be present to run the 
tools, one in the set must be present. If the conditions are not met the test 
is skipped (exit code 77).

Francesco

-- 
Did I say 2?  I lied.