[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [FYI] New public temporary branch, about documentation for custom te

From: Ralf Wildenhues
Subject: Re: [FYI] New public temporary branch, about documentation for custom test drivers
Date: Mon, 27 Jun 2011 08:06:00 +0200

Hi Stefano,

* Stefano Lattarini wrote on Sun, Jun 26, 2011 at 02:20:48PM CEST:
> I've pushed the documentation work I've done so far to the temporary
> public branch "GSoC/experimental/docs-custom-test-drivers", in case
> anyone wants to take a look at it (I had hoped to finish it this morning,
> but I didn't, and I'll be AFK until this evening, so there's no point in
> delaying even more).  Notice that that branch is bound to be amended and
> rebased, and finally deleted (it won't be merged in the 'test-protocols'
> branch, rather its changes will be re-applied on the top of that).

A few comments from a cursory look:

> +2011-06-25  Stefano Lattarini  <address@hidden>
> +
> +     docs: document custom test drivers and protocols
> +     * doc/automake.texi (Simple Tests): Note that the TESTS_ENVIRONMENT
> +     use suggested here is not portable to 'parallel-tests'.
> +     (Simple Tests using parallel-tests): Document new restrictions on
> +     (Custom Test Drivers): New section and node.
> +     (Overview of Custom Test Drivers Support): New subsection.
> +     (Declaring Custom Test Drivers in @file{}): Likewise.
> +     (APIs for Custom Test Drivers): Likewise.
> +     (Options): Update description of color-tests.
> +     * lib/am/check (recheck, recheck-html): Minor adjustments to better
> +     conform to the documentation (this should cause no semantic changes
> +     w.r.t. the former behaviour); minor improvements and extensions to
> +     existing comments.
> +     * tests/test-driver-create-log-dir.test: New test.
> +     * tests/test-driver-strip-vpath.test: Likewise.
> +     * tests/test-driver-global-log.test: Likewise.
> +     * tests/test-driver-recheck.test: Likewise.
> +     * tests/ (TESTS): Update.

> --- a/doc/automake.texi
> +++ b/doc/automake.texi

> @@ -8691,10 +8708,12 @@ Simple Tests
>  TESTS =
>  @end example
> -Note that the @option{parallel-tests} driver provides a more elegant
> -way to achieve the same effect, freeing the @code{TESTS_ENVIRONMENT}
> -variable for the user to override (@pxref{Simple Tests using
> -parallel-tests}).
> +It's important to note that this last use of @code{TESTS_ENVIRONMENT}
> +is invalid when the @option{parallel-tests} option is active; however,
> +the @option{parallel-tests} driver provides a more elegant way to
> +achieve the same effect, with the further benefit of freeing the
> address@hidden variable for the user
> +(@pxref{Simple Tests using parallel-tests}).
>  @cindex Tests, expected failure
> @@ -8739,7 +8758,9 @@ Simple Tests using parallel-tests
>  the @code{check_*} variables are honored, and the environment variable
>  @env{srcdir} is set during test execution. Also, @code{TESTS_ENVIRONMENT}
>  is still honored, but is complemented by a new developer-reserved variable
> address@hidden (described below).
> address@hidden (described below), and its allowed uses are
> +somewhat restricted (other features of the @option{parallel-tests} driver
> +compensate for this, though).
>  This test driver is still experimental and may undergo changes in order
>  to satisfy additional portability requirements.
> @@ -8812,12 +8833,13 @@ Simple Tests using parallel-tests
>  but should be reserved for the user.
> -The @code{AM_TESTS_ENVIRONMENT} variable can be used to run initialization
> -code and set environment variables for the tests' runs.  The user can
> -still employ the @code{TESTS_ENVIRONMENT} variable to override settings
> -from @code{AM_TESTS_ENVIRONMENT}.  Note that, for implementation reasons,
> -if the @code{AM_TESTS_ENVIRONMENT} variable is set, its contents
> address@hidden be terminated by a semicolon.
> +The @code{AM_TESTS_ENVIRONMENT} and @code{TESTS_ENVIRONMENT} variables can
> +be used to run initialization code and set environment variables for the
> +test scripts.  The former variable is developer-reserved, and can be
> +defined in the @file{}, while the latter is reserved for the

How about s/reserved // here, to reflect prior usage?

> +user, which can employ it to extend or override the settings in the
> +former. For implementation reasons, if the @code{AM_TESTS_ENVIRONMENT}

s/\. /& /

> +variable is set, its contents @emph{must} be terminated by a semicolon.

Why is that necessarily so?

> @@ -8834,6 +8856,23 @@ Simple Tests using parallel-tests
>  AM_TESTS_ENVIRONMENT = exec 9>&2; warn_fileno=9; export warn_fileno;
>  @end example
> +It's important to note that, differently from what we've seen for the
> +serial testsuite driver (@pxref{Simple Tests using parallel-tests}), the
> address@hidden and @code{TESTS_ENVIRONMENT} variables
> address@hidden be use to define a custom test driver; the
> address@hidden and @code{LOG_FLAGS} (or their extension-specific
> +counterparts) should be used instead:
> +
> address@hidden
> +## This is WRONG!
> +AM_TESTS_ENVIRONMENT = PERL5LIB='$(srcdir)/lib' $(PERL) -Mstrict -w
> +
> +## Do this instead.
> +AM_TESTS_ENVIRONMENT = PERL5LIB='$(srcdir)/lib'; export PERL5LIB;
> +AM_LOG_FLAGS = -Mstrict -w
> address@hidden example
> +
>  @trindex mostlyclean
>  @trindex check-html
>  @vindex RST2HTML
> @@ -8959,6 +8998,290 @@ Simple Tests using parallel-tests
>  In case of doubt you may want to require to use GNU @command{make},
>  or work around the issue with inference rules to generate the tests.
> address@hidden Custom Test Drivers
> address@hidden Custom Test Drivers
> +
> address@hidden
> +* Overview of Custom Test Drivers Support::
> +* Declaring Custom Test Drivers::
> +* API for Custom Test Drivers::
> address@hidden menu
> +
> address@hidden Overview of Custom Test Drivers Support
> address@hidden Overview of Custom Test Drivers Support
> +
> +Starting form Automake version 1.12, the @option{parallel-tests} harness
> +has been augmented to allow the package authors to use third-party custom

s/has been augmented to allow/allows/

It is usually easier to read some text a couple of years down the road
if it doesn't reflect transition details that aren't necessary to know
after the fact.

> +test drivers, in case the
> address@hidden FIXME: this should become "default ones" once we have TAP and 
> Subunit
> +default one is inadequate for their purposes.
> address@hidden FIXME: add this once we have TAP and Subunit
> address@hidden "or do not support their testing protocol of choice."
> +
> +A custom test driver is expected to properly run the test programs passed
> +to it (bearing in mind that they can have command-line arguments too, which
> +can legitimately happen within the Automake testsuite harness framework),

s/(.*)/, including the command-line arguments passed to it/

> +to analyze their execution and outcome, to create the @file{.log} files
> +associated to these test runs, and to display the test results on the
> +console.

> It is the sole responsibility of the author of the test driver
> +to ensure that it implements all the above steps meaningfully and
> +correctly; Automake isn't and can't be of any help here.

I'm wondering whether it's better to omit this sentence.  It sounds like
a teacher talking to me; in programming, I am always expected to do
things correctly.

> On the other
> +hand, the Automake-provided code for testsuite summary generation offers
> +support for test drivers allowing several test results per test script,
> +if they take care of registering such results properly (@pxref{Log files

s/of registering/to register/

> +generation and test results recording}).
> +
> +The exact details of how test scripts' results are to be determined and
> +analyzed is left to the individual drivers.  Some drivers might only
> +consider the test script exit status (this is done for example by the
> +default test driver used by the @option{parallel-tests} harness, described
> +in the previous section).  Other drivers might implement more complex and
> +advanced test protocols, which might require them to parse and interpreter
> +the output emitted by the test script they're running (examples of such
> +protocols are TAP and SubUnit).
> +
> +It's very important to note that, even when using custom test drivers,
> +most of the @option{parallel-tests} infrastructure described in the
> +previous section remains in place; this comprises:

(the alternative is "this is comprised of" and needs an exhaustive list)

> address@hidden
> address@hidden
> +concurrency through the use of  @command{make}'s option @option{-j};
> address@hidden
> +per-test @file{.log} files, and generation of a summary @file{.log}
> +file from them;
> address@hidden
> address@hidden target and lazy reruns of tests;
> address@hidden
> +inter-test dependencies;
> address@hidden
> +support @code{check_*} variables (@code{check_PROGRAMS},
> address@hidden, ...);
> address@hidden
> +use of @code{VERBOSE} environment variable to get verbose output on
> +testsuite failures;
> address@hidden
> +definition and honoring of @code{TESTS_ENVIRONMENT} and
> address@hidden variables;
> address@hidden
> +definition of generic and extension-specific @code{LOG_COMPILER} and
> address@hidden variables.
> address@hidden itemize
> +


> +On the other hand, the exact semantics of how (and if)
> address@hidden, @code{XFAIL_TESTS}, and hard errors are supported
> +and handled is left to the individual test drivers.
> +
> address@hidden TODO: We should really add a working example in the doc/ 
> directory,
> address@hidden TODO: and reference if from here.

Good idea.  It could be a simplified form of the default test driver.

> address@hidden Declaring Custom Test Drivers
> address@hidden Declaring Custom Test Drivers in @file{}

Karl recommends keeping node and section names the same (modulo
formatting and comma), or at least very similar.

> address@hidden _LOG_DRIVER
> address@hidden _LOG_DRIVER_FLAGS
> address@hidden LOG_DRIVER
> address@hidden LOG_DRIVER_FLAGS
> address@hidden @var{ext}_LOG_DRIVER
> address@hidden @var{ext}_LOG_DRIVER_FLAGS
> address@hidden address@hidden
> address@hidden AM_LOG_DRIVER_FLAGS
> +Custom testsuite drivers are declared by defining the make variables
> address@hidden or @address@hidden (where @var{ext} must
> +be declared in @code{TEST_EXTENSIONS}).  They must be defined to
> +programs or scripts that will be used to drive the execution, logging,
> +and outcome report of the tests with corresponding extensions (or of
> +those with no registered extension in the case of @code{LOG_DRIVER}).
> +Clearly, multiple distinct test drivers can be declared in the same
> address@hidden  Note moreover that the @code{LOG_DRIVER} variables
> +are @emph{not} a substitute for the @code{LOG_COMPILER} variables: the
> +two sets of variables can, and often do, usefully and legitimately
> +coexist.
> +
> address@hidden TODO: We should really be able to point to a clarifying 
> example here!

Yep.  As a reader, this leaves me confused.  Why should there be two
separate script instances?

Actually, why should there (other than maybe for backward
compatibility)?  I mean, not just in the documentation, but in the
Automake code as well?  Can't we simplify this?  Thanks.

> +The developer-reserved variable @code{AM_LOG_DRIVER_FLAGS} and the
> +user-reserved variable @code{LOG_DRIVER_FLAGS} can be used to define
> +flags that will be passed to each invocation of @code{LOG_DRIVER}
> +(with the user-defined flags obviously taking precedence over the
> +developer-reserved ones).  Similarly, for each extension @var{ext}
> +declared in @code{TEST_EXTENSIONS}, flags listed in
> address@hidden@var{ext}_LOG_DRIVER_FLAGS} and
> address@hidden@var{ext}_LOG_DRIVER_FLAGS} will be passed to
> +invocations of @address@hidden
> +
> address@hidden API for Custom Test Drivers
> address@hidden API for Custom Test Drivers
> +
> +Here we'll describe the APIs currently offered to the authors of custom
> +test drivers.

Just describe things, don't describe what you describe.
(This is OK as a TODO note of course)

>  Note that they are still somewhat experimental, and might
> +undergo changes and tightenings in the future, to accommodate for new
> +features or to satisfy additional portability requirements.
> +
> +Also, note that the main characteristic of these APIs is that they are

repetition; s/Also, note that t/T/

> +designed to share as much infrastructure, semantics, and implementation
> +details as possible with the plain @option{parallel-tests} harness.  So
> +everything said in the previous section should apply here too, unless
> +explicitly noted otherwise.

repetition; s/noted/stated/

> address@hidden
> +* Command-line arguments for test drivers::
> +* Log files generation and test results recording::
> +* Console output during testsuite runs::
> +* HTML generation from testsuite logs::
> address@hidden menu
> +
> address@hidden Command-line arguments for test drivers
> address@hidden Command-line arguments for test drivers
> +
> +A custom driver can rely on various command-line options and arguments
> +being passed to it automatically by the Automake's @option{parallel-tests}
> +harness.  It is @emph{mandatory} that it understands all of them (even if
> +the exact interpretation of the associated semantics can change between a
> +test driver and another, and even be a no-op in some drivers).
> +
> address@hidden
> +Here is the list of options:
> +
> address@hidden @option
> address@hidden address@hidden
> +The name of the test, with VPATH prefix (if any) removed.  This can have a
> +suffix and a directory component (as in e.g., @file{sub/foo.test}), and is
> +mostly meant to be used in console reports about testsuite advancements and
> +results (@pxref{Console output during testsuite runs}).
> address@hidden address@hidden
> +The log file the test driver must create.  If it has a directory component
> +(as in e.g., @file{sub/foo.log}), the Automake harness will ensure that
> +such directory exist before the test driver is called.
> address@hidden address@hidden|address@hidden
> +Whether the console output should be colored or not (@pxref{Simple
> +tests and color-tests} to learn when this option gets activated and
> +when it doesn't).
> address@hidden address@hidden|address@hidden
> +Whether the tested program is expected to fail.
> address@hidden address@hidden|address@hidden
> +Whether "hard errors" in the tested program should be treated differently

Prefer ``quotes'' to "quotes" unless the quotes are part of a code
snippet.  Several instances

> +from normal failures or not (the default should be "@code{yes}").  The exact


> +meaning of "hard error" is highly dependent from the test protocols or
> +conventions in use.
> address@hidden --
> +Explicitly terminate the list of options.
> address@hidden table
> +
> +The first of the remaining arguments passed to the test driver is the
> +program to be run, and the other arguments are command-line options
> +and arguments for this program.
> +
> +Note that the exact semantics attached to the @option{--color-tests},
> address@hidden and @option{--enable-hard-errors} options are
> +left up to the individual test drivers.  Still, having a behaviour
> +compatible or at least similar to that provided by the default
> address@hidden driver is advised, as that would offer a better
> +consistency and a more pleasant user experience.
> +
> address@hidden Log files generation and test results recording
> address@hidden Log files generation and test results recording
> +
> +One of the main features of the new testsuite harness is the ability to
> +support test protocols that allow a single test script to run more
> +test cases, @emph{each with its distinct result}.  In order for the
> +testsuite summary report to be correct in this case, the test driver
> +implementing support for the protocol must follow the following
> +convention.
> +
> +TODO: In order to have a correct count of test cases in the final
> +testsuite summary, the driver must use special syntax in the created
> +.log file (see below, `:test-result:').
> +
> +The test driver must correctly generate the file specified by the
> address@hidden option (even when the tested program fails, of
> +course!).  This file is quite free-form, but still it has to conform
> +to the following conventions:
> +
> address@hidden @bullet
> address@hidden
> address@hidden Global test script result
> address@hidden Test result, global
> +The @emph{global test result} for the whole test script must be placed
> +in the first line of the @file{.log} file.
> address@hidden The following semantics is checked by tests 
> 'test-driver-recheck.test'
> address@hidden and 'test-driver-global-log.test'.  Please keep them in sync 
> whenever
> address@hidden the following is changed.
> +If this line does not begin with either @code{PASS:}, @code{XFAIL:} or
> address@hidden:}, the test script will be considered failed, and it will be
> +re-run by @code{make recheck}.  Also, if this line does not begin with
> +either @code{PASS:} or @code{XFAIL:}, the content of and its @file{.log}
> +file will be copied into the global @file{test-suite.log} (in order to
> +help in debugging and bug-report analysis).
> address@hidden
> address@hidden Single test case result
> address@hidden Test result, single
> address@hidden @code{:test-result:} reStructuredText field
> address@hidden  reStructuredText field, @code{:test-result:}
> +One of the main features of the new testsuite harness is the ability to
> +support test protocols that allow a single test script to run more
> +test cases, @emph{each with its distinct result}.  In order for the
> +testsuite report to be correct in this case, the test driver must
> +register @emph{each} such result in the generated @file{.log} file, using
> +the @code{:test-result:} reStructuredText field.
> +
> +FIXME: currently, that field is recognized anywhere in the @file{.log}
> +file; this is possibly prone to generating spurious results, in case
> +of verbose tests.  We should allow a special @code{:test-result:} that
> +stops any further parsing (maybe a line @code{:test-result:END} will
> +do?).

Well, but usually you only know the test outcome at the very end.
Requiring the result to be early in the test will make file generation
more complicated than it would need to be otherwise.

> +FIXME: we are probably going to change the current semantics of
> +testsuite summary generation, in order to allow test protocols with
> +only one result per test case not to be forced to also declare that
> +result with @code{:test-results:}; that would be a repetition, since
> +the result is already declared in the @file{.log} first line.
> address@hidden itemize

Good question.

> address@hidden: Explain why supporting the Global test result declaration
> +in the first line is useful also for test protocols that allow more
> +test cases per test script.
> +
> address@hidden
> address@hidden  * A test driver is free (but not forced) to register multiple 
> test
> address@hidden    results for a single test script, through the use of a 
> custom
> address@hidden    reStructuredText field ``test-result''.  Each of this 
> results will
> address@hidden    be accounted for separately in the final testsuite summary;
> address@hidden    TODO: give an example
> address@hidden    TODO: a special field to terminate parsing of .log?
> address@hidden    But note that the task of reporting this results on the 
> screen is
> address@hidden    left to the individual test driver though; more about this 
> below.
> address@hidden    For example, if the test script executes
> address@hidden    two test cases, one successful and one failing, and skip 
> another
> address@hidden    test case, the driver should end up writing the following 
> in the
> address@hidden    test log:
> address@hidden      :test-result: PASS [name of passed test, and/or details 
> about it]
> address@hidden      :test-result: FAIL [name of failed test, and/or details 
> about it]
> address@hidden      :test-result: SKIP [name of skipped test, and/or details 
> about it]
> address@hidden    The above lines (each of which *must* be followed by a 
> blank line
> address@hidden    in order for the HTML output generation to work) are used 
> only
> address@hidden    when generating the `test-suite.log' from the individual 
> test
> address@hidden    logs, and can be placed in any order and position within 
> the logs
> address@hidden    itself.
> address@hidden
> +
> address@hidden Console output during testsuite runs
> address@hidden Console output during testsuite runs
> +
> +TODO.
> address@hidden  * explain what kind of console output the driver is expected 
> to produce;
> address@hidden    it can have some leeway here, but IMHO should provide a 
> "look & feel"
> address@hidden    similar to the existing "parallel-tests" output.
> address@hidden  * tell that this is where the --color option is to be 
> influential.
> +
> address@hidden HTML generation from testsuite logs
> address@hidden HTML generation from testsuite logs
> +
> +If the HTML testsuite output (with @code{check-html}) is to be supported,

s/the //

> +the generated @file{.log} files must contain syntactically valid
> +reStructuredText.  If this is not the case, the HTML generation won't

will not

> +work, but all the other functionalities of the Automake testsuite harness
> +should remain untouched, and continue to work correctly.

> --- a/lib/am/
> +++ b/lib/am/
> @@ -317,7 +317,8 @@ recheck recheck-html:
>  ## test script on the first line.  See the comments in the rules of
>  ## $(TEST_SUITE_LOG) above for why we consider this acceptable and even
>  ## advisable.
> -               case $$line in FAIL*|XPASS*) echo $$f;; esac;         \
> +               case $$line in PASS:*|XFAIL:*|SKIP:*);; *) echo $$f;; esac; \
> +## A test whose log is unreadable is to be caonsidered failed.


Why is the comment below the code?
What about XPASSes?
Can test drivers implement other kinds of results?  (See dejagnu for


reply via email to

[Prev in Thread] Current Thread [Next in Thread]