Re: prepare Automake's test suite for parallelization

[Benoit SIGOURE]

On Nov 4, 2007, at 7:07 PM, Ralf Wildenhues wrote:

> I'm not entirely happy with this one yet.  It does what is needed,
> but it has warts:
> - if you interrupt the test suite, you will have leftover directories,

That's a minor problem, methinks.  As long as a subsequent `make  
clean' Does The Right Thing.

> - the check-clean stuff is supposed to be niceified in a later patch
>   (i.e., made into a proper Automake interface).
> - for XPASS tests we do the wrong thing: remove the test directory.
>   This is because our suite's defs.in has no idea that we are running
>   an XPASS.  The keep_testdirs is a cheap workaround.
>
> I guess for the latter, check-TESTS should indicate to the test being
> run that it is expected to fail.  A bit ugly.

Admittedly it's ugly, but easy to do and it "gets the job done".  I  
can't think of anything better.  Maybe the cleaning could be done by  
check-local, when it really knows whether the test is a PASS or not?

> FWIW2, we could use subdirectories named after the test name rather  
> than
> using the PID.  However, I've found myself running the same test in
> parallel sometimes:
>   make check TESTS=foo.test # Stop with C^Z, then run same command  
> again
>
> and it causes weird output if you later `fg' the first `make'.   
> Arguably
> this is PEBCAK.  So really if you don't like using the PID, please  
> speak
> up, I'm all open here.

It's not convenient.  If there's a failure, you'll have to find out  
which sub-directory to look at, right?  Most modern OSes randomize  
PIDs, so it's going to be cumbersome.  My personal opinion would be  
to use something based on the test name.  If you want, you can try to  
combine the test-name and the PID?  Or maybe increment a number for  
each test and use that?  (this will need to deal nicely with parallel  
make check though, so it might not be trivial).

> Cheers,
> Ralf
>
>     Run each test in a process-private subdirectory.  Clean up  
> afterwards.
>
>     * tests/defs.in: Employ a trap to clean up at the end of the test
>     in case the test succeeded.  This needs to be revisited because we
>     would like to keep the output of XPASSing tests.
>     Also, note when we were interrupted by a signal.
>     If $keep_testdirs is set, keep them even for successful tests.
>     * tests/Makefile.am (check-clean-local): New target.  Remove all
>     tests subdirs, but also make them writable first.  The latter was
>     already needed without per-test subdirs.
>     (distclean-local): Depend upon check-clean-local.
>     * tests/Makefile.in: Regenerate.
>     * HACKING: Mention `keep_testdirs'.
>
> diff --git a/HACKING b/HACKING
> index 5a3ed33..9f2f1f4 100644
> --- a/HACKING
> +++ b/HACKING
> @@ -119,6 +119,8 @@
>
>  * Make sure each test file is executable
>
> +* Use `keep_testdirs=yes' to keep test directories for successful
> +  tests also.
>
>  ================================================================
>  = Release procedure
> diff --git a/tests/Makefile.am b/tests/Makefile.am
> index 5d61ae7..e915f4a 100644
> --- a/tests/Makefile.am
> +++ b/tests/Makefile.am
> @@ -625,5 +625,8 @@ EXTRA_DIST = ChangeLog-old $(TESTS)
>  # Each test case depends on defs, aclocal, and automake.
>  check_SCRIPTS = defs aclocal-$(APIVERSION) automake-$(APIVERSION)
>
> -distclean-local:
> -       -rm -rf testSubDir
> +distclean-local: check-clean-local
> +
> +check-clean-local:
> +       -chmod -R a+rwx testSubDir*

u+rwx is enough, there's no need in making this open to the entire  
world ;o

> +       -rm -rf testSubDir*


> diff --git a/tests/defs.in b/tests/defs.in
> index 78cf37a..2ad4345 100644
> --- a/tests/defs.in
> +++ b/tests/defs.in
> @@ -235,16 +235,35 @@ case "$srcdir" in
>      ;;
>  esac
>
> -chmod -R a+rwx testSubDir > /dev/null 2>&1
> -rm -rf testSubDir > /dev/null 2>&1
> -mkdir testSubDir
> +curdir=`pwd`
> +testSubDir=testSubDir$$
> +chmod -R a+rwx $testSubDir > /dev/null 2>&1

Ditto.

> +rm -rf $testSubDir > /dev/null 2>&1
> +mkdir $testSubDir
> +
> +trap 'exit_status=$?
> +  cd "$curdir"
> +  case $exit_status,$keep_testdirs in
> +  0,)
> +    chmod -R a+rwx $testSubDir > /dev/null 2>&1
> +    rm -rf "$testSubDir" ;;
> +  esac
> +  test "$signal" != 0 &&
> +    echo "$as_me: caught signal $signal"
> +  echo "$as_me: exit $exit_status"
> +  exit $exit_status
> +' 0
> +for signal in 1 2 13 15; do
> +  trap 'signal='$signal'; { (exit 1); exit 1; }' $signal
> +done
> +signal=0

That's great, I'm going to use this in my own testsuites!

>  # Copy in some files we need.
>  for file in install-sh missing depcomp; do
> -   cp $srcdir/../lib/$file testSubDir/$file || exit 1
> +   cp $srcdir/../lib/$file $testSubDir/$file || exit 1
>  done
>
> -cd ./testSubDir
> +cd ./$testSubDir
>
>  # Build appropriate environment in test directory.  Eg create
>  # configure.in, touch all necessary files, etc.

--
Benoit Sigoure aka Tsuna
EPITA Research and Development Laboratory

PGP.sig
Description: This is a digitally signed message part