Re: unit testing with autotools

[Chris Pickett]

[[ Sorry, I originally sent this from an unsubscribed account ]]

-------- Original Message --------
Subject: Re: unit testing with autotools
Date: Tue, 14 Mar 2006 11:25:06 -0500
From: Chris Pickett <address@hidden>
Reply-To: address@hidden
To: Ralf Wildenhues <address@hidden>
CC: address@hidden
References: <address@hidden> 
<address@hidden>

Hi Ralf,

Ralf Wildenhues wrote:
> Hi Chris,
>
> * Chris Pickett wrote on Mon, Mar 13, 2006 at 06:13:07PM CET:
>
> > I'd like to do unit testing for test-driven development of C programs. 
> > I'd also like to use up-to-date autotools.  I was wondering if anyone 
> > had any experience with unit testing in their projects and had 
> > recommendations, or even if autoconf was considering it for the future.
> >
> > By unit testing I mean writing multiple small tests for each function in 
> > an interface.  The prescribed development methodology is to write 
> > failing tests before you write code that will make them pass.
> >
> > Autotest would be good as a wrapper around a unit-testing framework, but 
> > not good for driving the actual low-level tests.  The same goes for 
> > DejaGNU.
>
>
> Out of curiosity: why exactly do Autotest and DejaGNU not qualify for
> this job?  Which parts are missing?

I'm sorry, I should have explained it better.  But first a disclaimer: I
only started thinking about unit testing on Friday, and I only came to
understand the basic autotools setup for starting a project from scratch
over the weekend, although in the past I have used autotools and m4 a
reasonable amount, but starting from other's work.  So I may completely
wrong here.

Autotest and DejaGNU are great for "system testing" -- running compiled
programs and checking exit success, failure, the correctness of stdout
and stderr, handling of command line options, stuff like that.

However, there isn't any support for checking the correctness of
individual functions in a module.  Ideally for each function in your
interface, you have several test functions.  Then if you want to
refactor your interface implementation at a later date, you already have
a bunch of test functions to help reassure you that the refactoring
works.  You have client code for each function from the get go.

The way a unit testing framework handles this is by making it easy to
create small test functions that exercise your real functions.
DejaGNU provides an API to such frameworks in dejagnu.h

The archetypal unit testing framework, xUnit, is described pretty
clearly here:

http://www.oreilly.com/catalog/unitest/chapter/ch03.pdf

It describes the following simple set of classes:

  -- TestCase, a  test class with several individual test methods
  -- TestFixture, providing setUp() and tearDown() functions for tests
  -- TestRunner, for running individual tests and reporting
       speed, success
  -- TestSuite, which can contain TestCases or even other TestSuites
  -- TestResult, for accumulating test outcomes

Some of the additional features that I like in check, the C unit testing
framework I'm considering:

  -- Tests can optionally be forked in their own address space
    -- you can catch signals, expected or unexpected
    -- you can stop early exits
    -- you can prevent tests using the same setup data from
       interfering with each other
  -- You can have test timeouts
  -- You can loop through a table of values with a looping test
  -- Integration with gcov, to report code coverage (percentage of code
     that your tests exercise)

So, why did I write here?

1) to find out if other people had gotten existing unit testing
frameworks to play nicely with autotools.  Check actually was designed
around the 2.13 series, and although development has dwindled, it
probably could work pretty well.  I was just hoping to hear about
other's experiences before committing to it.
  -- Braden says he uses Boost Test
  -- Ed says he doesn't bother with a heavyweight framework and
    just writes his own API test code by hand

2) to find out if people had managed to do it cleanly using just
Autotest/DejaGNU

3) (failing 2) to suggest that making it easy to unit test your code
might make a good addition to autotools.  I'm not sure about this,
writing something language-agnostic could be tricky.

> > I'm also interested in more detailed or "best practice" autotest 
> > tutorials but the advice in the archives seems to be, "look at other 
> > projects and existing tutorials and cobble something together."
>
>
> FWIW, CVS M4 and CVS Libtool use Autotest.

I checked out M4 and Libtool and looked around.  I didn't see anything
testing individual C functions (not really applicable for M4), but maybe
I missed it.  For example, there's nothing that I saw testing the
functions in libltdl/ in Libtool, and nothing testing the .c files in
src/ in M4.  Of course, they do get tested indirectly.

Cheers,
Chris