[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Documentation for the parallel-tests driver. [4/4]

From: Akim Demaille
Subject: Re: Documentation for the parallel-tests driver. [4/4]
Date: Mon, 20 Oct 2008 05:31:33 -0500

Le 18 oct. 08 à 08:22, Ralf Wildenhues a écrit :

Hi Akim,

Hi Ralf!

* Akim Demaille wrote on Fri, Oct 17, 2008 at 12:15:42PM CEST:
Consequently, there is little point in showing users how to write such
a rule.

I strongly disagree here.

And your position is quite founded, if but a bit hard to swallow at
first.  ;-)


First, I don't want to (have to) tell users about $(am__check_pre) and
$(am__check_post).  If I did, I'd call them $(check_prepare) and
$(check_postprocess), but I don't in the first place, as:
- this either nails down the implementation quite severely, to the point
 that we may not easily be able to fix bugs or inconsistencies in it,
- allowing users not to use them may prevent consistent introduction
 of post-test cleanup macros, hard errors, and similar,
- it is simply an Ugly API[tm].

Definitely :)  Yet I prefer an ugly feature to no feature.

But if I understand you correctly, then that isn't the grip you have
with this at all.  The missing bit is: you want to be able to provide
a command executing tests; moreover, different commands for different
sets of tests.

Actually I don't see why I could not use several commands within the same test suite. I agree this is not the case currently in my test suites, but I see no reason not to be able to write:

        TESTS_SUITES = test-suite.log
        tests_suite_SOURCES =
        foo_COMPILER = ./foo
        bar_COMPILER = ../bin/bar
        bar_FLAGS = --verbose --warnings=all

to change a bit what I suggested before. Mixing test drivers based on the extension is so natural that I would find it a limitation. Also, take the Bison Autotest-based test suite as of today: there are several macros that define a full test case, and that macro is repeatedly called with diverse arguments (for instance many variations of the calculator with different features, parsing techniques, output language etc.). Using this Automake framework I would have a single driver, say "calc-tester" and many *.calc files. Yet I don't want a single "calc.log" file and several other test-suite logs: I still want the single test-suite.log file, but with many kinds of tests in it.

And that part is a Good Idea[tm].

I'd even go as far as: both the developer and the user might want to be able to have a word here. (Think valgrind, think simulator (don't think
about Libtool issues just yet), ...)

Agreed. But in my experience either your tests are unit tests, or a test driver. You don't test directly the tested tool itself. For a start you certainly want to check stdout and stderr, and the expected exit status is not always 0 (and my test driver actually generate RST output for all these aspects, so the whole test-suite.html is nice to read). This requires a test-driver. Not to mention that you also want to be able to use this framework for installcheck. And the added complexity has nothing to do the Make rule: it should all go into the test-driver.

So I don't think that Valgrind, libtool and so forth are relevant here. They should be dealt with elsewhere.

But I don't see the real need for users to specify the whole rule.

Agreed.  Just the test-driver associated to a given extension.

I wrote with a clear and simple model in mind: we are compiling tests, and they can be written in several "languages". This fits nicely the Make model and gives all the rest (parallelism etc.) for free. *And* it provides an opportunity to present a uniform interface a la Automake,

        TESTS_SUITES = foo-suite.log bar-suite.log
        foo_suite_SOURCES =
        bar_suite_SOURCES =

        foo_COMPILER = ./foo
        bar_COMPILER = ../bin/bar --verbose --warnings=all

Which is a nice idea.  I like it.  I even tried to go this way when I
first started.  It turned out that a number of purely practical issues
needed to be addressed:

- if you still want to be able to easily limit the number of tests run
 at 'make' run time, you have to go through hoops to ensure that
 *_OBJECTS is a direct dependent of *_SOURCES (resp. TESTS),

I don't know what you mean here. Maybe you are referring to hard errors as you implemented them?

- *for this language*, the mapping from sources to objects should *not*
  be an automake internal detail,

Well, unless you want to expose the concept of "Automake rule", say

        .log.calc = ./calc-tester $(CALC_FLAGS) $@ >$<

and process them, we either have to use regular rules, or regular variables. The rules does not seem so bad to me, the fully flown rule will always be "something before, the command, something afterwards". So it's an Ugly API™, absolutely, but it's effective and can evolve IMHO.

- the current code looks not really adapted to reset known extensions
 and languages per, so this could be a bit of work (if
 only to check that nothing silently breaks),

Agreed. In fact I regret that Automake did not go further in the unification: Texinfo documentations should be presented with _SOURCES and so forth. The model of separate compilation and linking scales quite well in many different context. Tests are another.

- then, I'm quite a fan of backward compatibility, with the aim that
 most users would get a working parallel testsuite by simply adding
   AUTOMAKE_OPTIONS = parallel-tests

 to their  Cf. testsuite addition that tests just this.

I fail to see what changes here. By default the test-driver is nul, that is, you just run the test itself.

- In your proposal, it would IMVHO still be necessary to specify the set
 of extensions that tests can have (in order to support @substed@
 tests at least, and in order to avoid ambiguities and per-target

I agree that it is nicer if it is Automake that catches these errors, but if it doesn't Make is still here, and it will. I don't understand what you mean with per-target rules here.

I decided that this was more than I could implement in the time that I
had for this, and it is probably more than I will have time for in the
near future.

It looks like you're aiming for much more general than what I did/ need. Yet losing the genericity I need :(

Alternatively, is there a volunteer for implementing it?

   <- nothing :)

Thanks for spending so much time in this feature!

reply via email to

[Prev in Thread] Current Thread [Next in Thread]