Re: parallel autotest [1/3]: Refactor testsuite driver loop.

[Eric Blake]

Ralf Wildenhues <Ralf.Wildenhues <at> gmx.de> writes:

> 
> 2008-04-13  Ralf Wildenhues  <Ralf.Wildenhues <at> gmx.de>
> 
>       * lib/autotest/general.m4 (AS_MESSAGE_LOG_FD): Move definition
>       earlier in the file.
>...

This patch is worth applying, once these nits are addressed...

> 
>  # The file containing the suite.
>  at_suite_log=$at_dir/$as_me.log
> -# The file containing the location of the last AT_CHECK.
> -at_check_line_file=$at_suite_dir/at-check-line
> -# The file containing the exit status of the last command.
> -at_status_file=$at_suite_dir/at-status
> -# The files containing the output of the tested commands.
> -at_stdout=$at_suite_dir/at-stdout
> -at_stder1=$at_suite_dir/at-stder1
> -at_stderr=$at_suite_dir/at-stderr
> -# The file containing the function to run a test group.
> -at_test_source=$at_suite_dir/at-test-source
> -# The file containing dates.
> -at_times_file=$at_suite_dir/at-times
> +# The directory containing helper files per test group.
> +at_helper_dir=$at_suite_dir/at-groups

You know, at this point, an ascii tree diagram might be a helpful comment 
(probably with # and not dnl, so that the comment is also present in the 
generated testsuite).  Something like:

testsuite - the testsuite
testsuite.log - summarizes the complete testsuite run
testsuite.dir/ - created during a run, remains after -d or failed test
+ at-groups/ - during a run: holds status of all groups in run
| + nnn/ - during a run: holds meta-data about a group
| | + at-check-line - during a group: holds location of current AT_CHECK
| | + at-status - during a group: holds exit status of current AT_CHECK
| | + at-stdout - during a group: captures stdout of current AT_CHECK
| | + at-stder1 - during a group: captures stderr, including trace
| | + at-stderr - during a group: captured stderr, with trace filtered out
| | + at-test-source - during a group: portion of testsuite that defines group
| | + at-times - during a group: holds timestamps for computing duration
| | + pass - during a group: created if group passed
| | + xpass - during a group: created if group xpassed
| | + fail - during a group: created if group failed
| | + xfail - during a group: created if group xfailed
| | + skip - during a group: created if group skipped
+ at-stop - during a run: created to abort the run
+ at-source-lines - during a run: caches group line numbers
+ nnn/ - created for each group, remains after -d or failed test
| + testsuite.log - summarizes the group results
| + ... - files created during the group


> +# Stop file: if it exists, do not start new jobs.
> +at_stop_file=$at_suite_dir/at-stop

Looks like a good idea.  Please mention it in the ChangeLog entry.

> @@ -959,19 +947,47 @@ BEGIN { FS="" }
>    test = substr ($ 0, 10)
>    print "at_sed" test "=\"1," start "d;" (NR-1) "q\""
>    if (test == "'"$at_group"'") exit
> -}' "$at_myself" > "$at_test_source" &&
> -. "$at_test_source" ||
> +}' "$at_myself" > "$at_suite_dir/at-source-lines" &&
> +. "$at_suite_dir/at-source-lines" ||
>    AS_ERROR([cannot create test line number cache])
> +rm -f "$at_suite_dir/at-source-lines"

I like the rename from $at_test_source to the literal at-source-lines, 
especially since the comment on the former no longer matched reality after we 
retooled things to use it as the line-number cache.

> +
> +# at_func_group_prepare
> +# ---------------------
> +# Prepare running a test group
> +at_func_group_prepare ()
> +{
> +  # The directory for additional per-group helper files.
> +  at_job_dir=$at_helper_dir/$at_group
> +  # The file containing the location of the last AT_CHECK.
> +  at_check_line_file=$at_job_dir/at-check-line

Do we still need the at- prefix on files that live under $at_helper_dir?  It 
seems a bit inconsistent to have $at_job_dir/at-stdout vs. $at_job_dir/pass, 
but I'm not sure whether the latter should be at-pass or whether we can 
simplify the former to stdout.

>    if test ! -f "$at_check_line_file"; then
> -    sed "s/^ */$as_me: warning: /" <<_ATEOF
> -     A failure happened in a test group before any test could be
> -     run. This means that test suite is improperly designed.  Please
> -     report this failure to <AT_PACKAGE_BUGREPORT>.
> +    sed "s/^ */$as_me: WARNING: /" <<_ATEOF

Are we consistent on capitalization of warning: vs. WARNING:?

> +  echo "$at_res" > "$at_job_dir/$at_res"

So both the name of the file and the contents of the file describe the result 
of the test.

> 
> +# Wrap up the test suite with summary statistics.
> +cd "$at_helper_dir"
> +
> +at_pass_list=`echo */pass | sed 's,/pass,,g; s,\*,,'`

But here, you are only using the file name, and with several forks to compute 
the summary.  What about these alternatives?

1. Keep the file name as the result, but the contents become the group id.  
This avoids a pipe, but still forks:

echo "$at_group" > "$at_job_dir/$at_res"
...
at_pass_list=`cat /dev/null */pass 2>/dev/null`

2. Keep the file name as the result but with an exploitable prefix, contents 
are irrelevant.  Use a shell loop to compute the lists.  Collecting the summary 
avoids forks altogether if we can then use an XSI construct, but is more 
expensive on deficient shells (implementation of the two versions of 
at_func_dirname left to the reader, but see libtool for hints):

touch "$at_job_dir/result-$at_res"
...
at_pass_list=
...
for at_result in */result-*; do
  at_func_dirname $at_result
  at_group=$at_func_dirname_result
  case $at_result in
    *-pass) at_pass_list="$at_pass_list $at_group" ;;
...
  esac
done

3. Like 2, but rework the loop variable to avoid needing at_func_dirname:

touch "$at_job_dir/result-$at_res"
...
at_pass_list=
...
for at_group in $at_groups; do
  case $at_group/result-* in
    *-pass) at_pass_list="$at_pass_list $at_group" ;;
...
  esac
done

-- 
Eric Blake