Re: Documentation for the parallel-tests driver. [4/4]

[Ralf Wildenhues]

Hi Akim,

* Akim Demaille wrote on Fri, Oct 17, 2008 at 12:15:42PM CEST:
>> Consequently, there is little point in showing users how to write such
>> a rule.
>
> I strongly disagree here.

And your position is quite founded, if but a bit hard to swallow at
first.  ;-)

There are several aspects to this whole issue; I will try to keep them
as separate as possible.

First, I don't want to (have to) tell users about $(am__check_pre) and
$(am__check_post).  If I did, I'd call them $(check_prepare) and
$(check_postprocess), but I don't in the first place, as:
- this either nails down the implementation quite severely, to the point
  that we may not easily be able to fix bugs or inconsistencies in it,
- allowing users not to use them may prevent consistent introduction
  of post-test cleanup macros, hard errors, and similar,
- it is simply an Ugly API[tm].

(It essentially would amount to documenting $p, $tst, $dir, $f, $log,
possible even how and what to write to $log; tty colors etc.)

But if I understand you correctly, then that isn't the grip you have
with this at all.  The missing bit is: you want to be able to provide
a command executing tests; moreover, different commands for different
sets of tests.

And that part is a Good Idea[tm].

I'd even go as far as: both the developer and the user might want to be
able to have a word here.  (Think valgrind, think simulator (don't think
about Libtool issues just yet), ...)

But I don't see the real need for users to specify the whole rule.

> I wrote check.mk with a clear and simple model in mind: we are compiling 
> tests, and they can be written in several "languages".  This fits nicely 
> the Make model and gives all the rest (parallelism etc.) for free.  *And* 
> it provides an opportunity to present a uniform interface a la Automake, 
> using
>
>       TESTS_SUITES = foo-suite.log bar-suite.log
>       foo_suite_SOURCES = 1.foo 2.foo
>       bar_suite_SOURCES = a.bar b.bar
>
>       foo_COMPILER = ./foo
>       bar_COMPILER = ../bin/bar --verbose --warnings=all

Which is a nice idea.  I like it.  I even tried to go this way when I
first started.  It turned out that a number of purely practical issues
needed to be addressed:

- if you still want to be able to easily limit the number of tests run
  at 'make' run time, you have to go through hoops to ensure that
  *_OBJECTS is a direct dependent of *_SOURCES (resp. TESTS),
- *for this language*, the mapping from sources to objects should *not*
  be an automake internal detail,
- the current code looks not really adapted to reset known extensions
  and languages per Makefile.am, so this could be a bit of work (if
  only to check that nothing silently breaks),
- then, I'm quite a fan of backward compatibility, with the aim that
  most users would get a working parallel testsuite by simply adding
    AUTOMAKE_OPTIONS = parallel-tests

  to their Makefile.am.  Cf. testsuite addition that tests just this.
- In your proposal, it would IMVHO still be necessary to specify the set
  of extensions that tests can have (in order to support @substed@
  tests at least, and in order to avoid ambiguities and per-target
  rules).

I decided that this was more than I could implement in the time that I
had for this, and it is probably more than I will have time for in the
near future.  Alternatively, is there a volunteer for implementing it?

I'm currently trying to gauge whether the question is:

- drop my 'parallel-tests' patches now, try for or wait for something
  better?  or
- add my 'parallel-tests' bits now, and go on?

Adding the patch set now would likely mean some incompatibility in the
future when your proposed semantics are implemented, which would be bad.

Comments appreciated.  And please don't get me wrong, I'm not attached
to this patch set, but I do want to provide parallel tests eventually.

Thanks,
Ralf