Re: Documentation for the parallel-tests driver. [4/4]

[Akim Demaille]

Le 18 oct. 08 à 08:22, Ralf Wildenhues a écrit :

> Hi Akim,

Hi Ralf!

> * Akim Demaille wrote on Fri, Oct 17, 2008 at 12:15:42PM CEST:
> > > Consequently, there is little point in showing users how to write  
> > > such
> > > a rule.
> >
> > I strongly disagree here.
>
> And your position is quite founded, if but a bit hard to swallow at
> first.  ;-)

:)

> First, I don't want to (have to) tell users about $(am__check_pre) and
> $(am__check_post).  If I did, I'd call them $(check_prepare) and
> $(check_postprocess), but I don't in the first place, as:
> - this either nails down the implementation quite severely, to the  
> point
>  that we may not easily be able to fix bugs or inconsistencies in it,
> - allowing users not to use them may prevent consistent introduction
>  of post-test cleanup macros, hard errors, and similar,
> - it is simply an Ugly API[tm].

Definitely :)  Yet I prefer an ugly feature to no feature.

> But if I understand you correctly, then that isn't the grip you have
> with this at all.  The missing bit is: you want to be able to provide
> a command executing tests; moreover, different commands for different
> sets of tests.

Actually I don't see why I could not use several commands within the  
same test suite.  I agree this is not the case currently in my test  
suites, but I see no reason not to be able to write:

        TESTS_SUITES = test-suite.log
        tests_suite_SOURCES = 1.foo 2.foo a.bar b.bar
        foo_COMPILER = ./foo
        bar_COMPILER = ../bin/bar
        bar_FLAGS = --verbose --warnings=all

to change a bit what I suggested before.  Mixing test drivers based on  
the extension is so natural that I would find it a limitation.  Also,  
take the Bison Autotest-based test suite as of today: there are  
several macros that define a full test case, and that macro is  
repeatedly called with diverse arguments (for instance many variations  
of the calculator with different features, parsing techniques, output  
language etc.).  Using this Automake framework I would have a single  
driver, say "calc-tester" and many *.calc files.  Yet I don't want a  
single "calc.log" file and several other test-suite logs: I still want  
the single test-suite.log file, but with many kinds of tests in it.


> And that part is a Good Idea[tm].
>
> I'd even go as far as: both the developer and the user might want to  
> be
> able to have a word here.  (Think valgrind, think simulator (don't  
> think
> about Libtool issues just yet), ...)

Agreed.  But in my experience either your tests are unit tests, or a  
test driver.  You don't test directly the tested tool itself.  For a  
start you certainly want to check stdout and stderr, and the expected  
exit status is not always 0 (and my test driver actually generate RST  
output for all these aspects, so the whole test-suite.html is nice to  
read).  This requires a test-driver.  Not to mention that you also  
want to be able to use this framework for installcheck.  And the added  
complexity has nothing to do the Make rule: it should all go into the  
test-driver.

So I don't think that Valgrind, libtool and so forth are relevant  
here.  They should be dealt with elsewhere.


> But I don't see the real need for users to specify the whole rule.

Agreed.  Just the test-driver associated to a given extension.


> > I wrote check.mk with a clear and simple model in mind: we are  
> > compiling
> > tests, and they can be written in several "languages".  This fits  
> > nicely
> > the Make model and gives all the rest (parallelism etc.) for free.   
> > *And*
> > it provides an opportunity to present a uniform interface a la  
> > Automake,
> > using
> >
> >         TESTS_SUITES = foo-suite.log bar-suite.log
> >         foo_suite_SOURCES = 1.foo 2.foo
> >         bar_suite_SOURCES = a.bar b.bar
> >
> >         foo_COMPILER = ./foo
> >         bar_COMPILER = ../bin/bar --verbose --warnings=all
>
> Which is a nice idea.  I like it.  I even tried to go this way when I
> first started.  It turned out that a number of purely practical issues
> needed to be addressed:
>
> - if you still want to be able to easily limit the number of tests run
>  at 'make' run time, you have to go through hoops to ensure that
>  *_OBJECTS is a direct dependent of *_SOURCES (resp. TESTS),

I don't know what you mean here.  Maybe you are referring to hard  
errors as you implemented them?

> - *for this language*, the mapping from sources to objects should  
> *not*
>   be an automake internal detail,

Well, unless you want to expose the concept of "Automake rule", say

        .log.calc = ./calc-tester $(CALC_FLAGS) $@ >$<

and process them, we either have to use regular rules, or regular  
variables.  The rules does not seem so bad to me, the fully flown rule  
will always be "something before, the command, something afterwards".   
So it's an Ugly API™, absolutely, but it's effective and can evolve  
IMHO.

> - the current code looks not really adapted to reset known extensions
>  and languages per Makefile.am, so this could be a bit of work (if
>  only to check that nothing silently breaks),

Agreed.  In fact I regret that Automake did not go further in the  
unification: Texinfo documentations should be presented with _SOURCES  
and so forth.  The model of separate compilation and linking scales  
quite well in many different context.  Tests are another.

> - then, I'm quite a fan of backward compatibility, with the aim that
>  most users would get a working parallel testsuite by simply adding
>    AUTOMAKE_OPTIONS = parallel-tests
>
>  to their Makefile.am.  Cf. testsuite addition that tests just this.

I fail to see what changes here.  By default the test-driver is nul,  
that is, you just run the test itself.


> - In your proposal, it would IMVHO still be necessary to specify the  
> set
>  of extensions that tests can have (in order to support @substed@
>  tests at least, and in order to avoid ambiguities and per-target
>  rules).

I agree that it is nicer if it is Automake that catches these errors,  
but if it doesn't Make is still here, and it will.  I don't understand  
what you mean with per-target rules here.

> I decided that this was more than I could implement in the time that I
> had for this, and it is probably more than I will have time for in the
> near future.

It looks like you're aiming for much more general than what I did/ 
need.  Yet losing the genericity I need :(


> Alternatively, is there a volunteer for implementing it?

   <- nothing :)

Thanks for spending so much time in this feature!