Re: parallel autotest [0/3]

[Eric Blake]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

According to Ralf Wildenhues on 5/25/2008 11:47 PM:
| There are many possible ways to parallelize Autotest testsuite
| execution, for example:

I'm still looking for time to review this series in depth, but appreciate
the work you've put into it so far.

|
| a) have each test group be a 'make' target in a makefile, then use
| parallel make.
| b) leverage the job server from GNU make,
| c) reimplement in shell a simplified job server a la parallel GNU make,
| d) implement a "worker thread" parallelization in shell.
|
| Features and differences of these approaches:
|
| - (a) and (b) need GNU make for parallel execution,
| - AFAICS (b) currently needs a shell that understands 'read -n1',
| - (d) differs from (c) in that one subprocess executes more than one
|   test group, thus is potentially faster because it forks less,
| - (b) has the nice feature that it allows to parallelize across multiple
|   test suites, and across testing and other, independent build activity.
|   That means, while (c) and (d) allow
|       make check TESTSUITEFLAGS='-j3'
|   to speed up things, (b) allows
|       gmake -j3 check
|   to profit.

Interesting trades.  I like that (c) and (d) can do parallel execution
when the testsuite is run manually (without make); on the other hand,
since make is usually the driver, I would tend to favor a solution along
the lines of (b) that lets the testsuite work alongside other processes.

|
| I've experimented a bit with these approaches.  I did not see an easy
| way to get (a) to work under the restrictions that it may not start the
| complete testsuite anew for each job: this has both very high overhead,
| and/or it requires that user-provided startup bits like atlocal be
| idempotent.  I must confess that I didn't try very hard, though.

The goal of still shipping a single 'testsuite' file that contains
everything needed to create the multiple tests is nice.  I agree that
blindly calling 'testsuite n' for n parallel tests is too much overhead.
On the other hand, it would sure be nice to call testsuite once with the
user's TESTSUITEFLAGS to generate the subset of individual files to run,
then turn around and run those individual files in parallel.

Is there a way to write a Makefile include fragment which gets included if
we detect GNU make, but is portably ignored for other makes, where we can
then exploit gmake features to make parallel execution easier?

|
| This patch series consists of
| 1) a preparatory patch refactoring the driver loop,

This one is probably okay, even without the others.  More comments to
follow in that mail.

| 2) an experimental patch implementing (c) if the system has mkfifo,
| 3) an *experimental* patch supporting (b) if GNU make is used, and the
|    shell can 'read -n1'.

I need more time to read through these.

|
| The patches (2) and (3) currently both have
| - an unknown number of remaining race conditions,  ;-)
| - known file descriptor leaks to test grous (not sure whether to view
|   that as a problem or not),
| - the bug^Wlimitation that, when a parallel run is interrupted, the
|   currently running test groups may still finish (see below also),

Autotest behavior when using ^C is already fishy (I often find that it
makes a test report as OK rather than failed, because it kills the test
group before any failure file could be created).  In other words, work to
improve signal handling within autotest is useful independently of this
patch series.

| - tests that are probably still too strict.
|
| Right now, the only system where I had significant problems was Cygwin
| with its seemingly limited named fifo emulation.  I expect that, given
| sufficient interest, somebody will fix this for me.  ;-)

Were you testing with cygwin 1.5.25 or the experimental cygwin 1.7.0?  I
agree that that the 1.5.x named fifos don't always work reliably.  I'll
certainly try to play more with this.

- --
Don't work too hard, make some time for fun as well!

Eric Blake             address@hidden
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Public key at home.comcast.net/~ericblake/eblake.gpg
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkg+p4wACgkQ84KuGfSFAYA3OgCfaFi1FoIrE6aeHmdOSIEC9H1I
ViUAoIGgcDesTav5mO/USn65S+2/14J2
=bNWj
-----END PGP SIGNATURE-----