Re: parallel autotest [0/3]

[Eric Blake]

Ralf Wildenhues <Ralf.Wildenhues <at> gmx.de> writes:

> > |
> > | - (a) and (b) need GNU make for parallel execution,
> 
> I should clarify: (a) is likely to cope with parallel non-GNU make,
> while (b) will ignore non-GNU make parallelism.  For example, BSD makes
> typically allow parallel execution as well, but do not implement a job
> server.

Yes, that helps me understand your intent - exposing the parallel tasks to any 
make's dependency engine, vs. hooking directly into the guts of GNU make's job 
server.

> 
> > | - AFAICS (b) currently needs a shell that understands 'read -n1',
> > | - (d) differs from (c) in that one subprocess executes more than one
> > |   test group, thus is potentially faster because it forks less,
> > | - (b) has the nice feature that it allows to parallelize across multiple
> > |   test suites, and across testing and other, independent build activity.
> > |   That means, while (c) and (d) allow
> > |       make check TESTSUITEFLAGS='-j3'
> > |   to speed up things, (b) allows
> > |       gmake -j3 check
> > |   to profit.
> >
> > Interesting trades.  I like that (c) and (d) can do parallel execution
> > when the testsuite is run manually (without make);
> 
> Good point.  I would not want to have (b) without any of (c), (d);
> rather, I thought of adding (c) and maybe also (b).

I was hoping that you could combine the good parts of (b) and (c).

> > The goal of still shipping a single 'testsuite' file that contains
> > everything needed to create the multiple tests is nice.  I agree that
> > blindly calling 'testsuite n' for n parallel tests is too much overhead.
> > On the other hand, it would sure be nice to call testsuite once with the
> > user's TESTSUITEFLAGS to generate the subset of individual files to run,
> > then turn around and run those individual files in parallel.
> 
> I don't quite understand what the last sentence in this paragraph is
> supposed to mean.

Our current recommended formula for including an autotest suite in Makefile is 
to call it with TESTSUITEFLAGS as the user's hook to select a subset of tests.  
So I was trying to convey my thoughts on a flow that could be something like 
this:

At configure time, config.status generates testsuite and also a makefile 
fragment that contains dependencies for each test's n.log, which is then 
included in the user's Makefile (automake supports makefile fragment inclusion 
even for non-GNU makes, or the fragment could be a multi-line AC_SUBST rather 
than an external file).

At make time, each n.log depends on n.test and a first stamp file.  
Running 'testsuite --parallel-start TESTSUITEFLAGS=-10' creates the stamp file, 
creates an independent shell script n.test for tests 1-10 (each with much less 
startup overhead than testsuite; for example, it can be hardcoded to run with 
the better shell detected by testsuite, rather than having to repeat the 
gyrations of detecting a better shell itself), and a no-op n.test for the 
remaining test groups (merely touching n.log).  After the stamp file exists, 
make can then run the various n.test files in parallel to create n.log files.

Finally, a second stamp file depends on all of the n.log files; 
running 'testsuite --parallel-collect' gathers the results and creates the 
second stamp.  That way, all you have to list for the dependency of check-local 
is the second stamp file.

But that still doesn't answer the question of whether the startup costs for n 
shell scripts is reasonable, in relation to forking off the main driver shell 
script that has already collected a bunch of startup state.

> 
> In each of (b), (c), you can take any TESTSUITEFLAGS you would currently
> use; with (c), add '-j3' to TESTSUITEFLAGS, with (b), add '-j3' to the 
> gmake command line directly.

And I suppose with (b) and (c) together, you could use:
gmake -j3 check TESTSUITEFLAGS=-j3

but then you have two job servers both competing for processor load without 
communicating with each other on how to balance the load.

> 
> > Is there a way to write a Makefile include fragment which gets included if
> > we detect GNU make, but is portably ignored for other makes, where we can
> > then exploit gmake features to make parallel execution easier?
> 
> As I understand this question, the at_parse_makeflags snippet shown in
> patch 3/3 does exactly that: it should be a no-op for non-GNU parallel
> make, as none of them use '--jobserver-fds='.
> 
> If your question is about a general way to include something for GNU
> make only, one possibility is to have a GNUmakefile which includes
> Makefile plus extra, GNU make-specific code.

Whatever we decide, we should probably make it easy for package writers to call 
a single macro in configure.ac to generate the AC_SUBST that then spits out all 
the Makefile magic, rather than the current approach with several lines of 
documentation on how to edit Makefile to add a testsuite.  For that matter, it 
might be worth installing /bin/autotest as a thin shell wrapper 
around /bin/autom4te --language=autotest.

> Fully agreed; I've seen such fishy behavior, too, but mostly ignored it
> up to now.  I fear that the behavior depends quite a bit on the sh and
> make implementations involved.

And I think some of it stems from the fact that we base success on the contents 
of $at_status_file, which can easily be out-of-date if a test group is killed 
and no trap exists to update the file accordingly.

> I think 1.5.25.  My usual procedure is to boot w32, fire up cygwin
> setup.exe, run its update process without changing any settings
> manually, then do testing.  I assume that, for experimental 1.7.0
> I'd have to change some setting, no?

Your guess is right - since you haven't done anything special, you're running 
cygwin 1.5.25.

-- 
Eric Blake