Re: [GSoC] Early design discussion for TAP/SubUnit support in automake.

[Stefano Lattarini]

Hello Robert.

On Wednesday 06 July 2011, Robert Collins wrote:
> Very sorry for the slow response, been EBUSY with real-life.
>
> On Sun, May 22, 2011 at 11:42 PM, Stefano Lattarini
> <address@hidden> wrote:
> > On Sunday 22 May 2011, Ralf Wildenhues wrote:
> >> Hi Stefano, and sorry for the long delay,
> >>
> > No problem, you had warned me in due time about such possible delays this
> > month; so there's really no need to apologize.
> >
> >> * Stefano Lattarini wrote on Fri, Apr 29, 2011 at 11:21:06AM CEST:
> >> > Now that my GSoC application "automake - Interfacing with a test protocol
> >> > like TAP or subunit" has been officially accepted, I'd like to start
> >> > discussing with the community some early, high-level design and interface
> >> > decisions.
> >>
> >>
> >> >  1. Reuse parallel-tests "framework"
> >> >  -----------------------------------
> >> >
> >> >   The new TAP/SubUnit support should reuse as much of the current
> >> >   parallel-tests implementation and semantics as possible.  In 
> >> > particular,
> >> >   it should be able to run different test scripts in parallel, generate a
> >> >   `.log' file for each test script and a "summarizing" `test-suite.log'
> >> >   file, honour the make variables AM_TESTS_ENVIRONMENT, TESTS_ENVIRONMENT
> >> >   and AM_COLOR_TESTS and the environment variable VERBOSE, and support
> >> >   different extensions for the test scripts, with extension-specific "log
> >> >   compilers" and flags (the stuff enabled by TESTS_EXTENSIONS,
> >> >   <ext>_LOG_COMPILER, etc.).
> >>
> >> Sounds all sane.
> >>
> >> >   The XFAIL_TESTS variable might be still supported for the sake of
> >> >   backward-compatibility (see below for the details), but it should be
> >> >   deprecated, since TAP and SubUnit offer better and more granular ways
> >> >   to express expected failures.
> >>
> >> OK.
> >>
> >> In another mail:
> >> > Thinking again about this, it might be worth trying to be even more 
> >> > consistent
> >> > with the existing parallel-tests functionality, and use an 
> >> > `ext_TEST_PROTOCOL'
> >> > variable (or similar) instead of a global `tests-protocol' option.  With 
> >> > some
> >> > tweaking to the post-processing of `.log' files done in 
> >> > `lib/am/check.am' (to
> >> > generate `$(TEST_SUITE_LOG)'), this might allow greater code reuse and a 
> >> > more
> >> > consistent API.
> >> >
> >> > I've started experimenting with this idea, and I'm not seeing any obvious
> >> > shortcoming right now.  I'm hoping I'll be able to post some experimental
> >> > patches soon enough.
> >>
> >> Allowing to specify that per-test is a good idea for transitioning test
> >> suites.
> >>
> > About this, in my first two "tentative" patches:
> >  <http://lists.gnu.org/archive/html/automake-patches/2011-05/msg00093.html>
> > I've taken an even more general approach, allowing the developer to define
> > and use his own program(s) to:
> >  1. launch his test scripts,
> >  2. interpreter their results,
> >  3. display these results on screen, and
> >  4. format and generate the log files.
> > All of this is attainable simply by assigning a variable `LOG_WRAPPER'
> > (and extensione-specific counterparts of it), and, well, obviously
> > providing a real "driver" script that obeys to minimal command-line
> > interface (so that it can grasp the options the Automake-generated
> > Makefiles passes to it).  Then we will hopefully be able to implement
> > our TAP/SubUnit parsers on the top of this feature (thus making it
> > indirectly more tested, which is always good for a new feature).
>
Note that a slightly-improved version of this design has now been
implemented (and documented) in the public branch 'test-protocols'.

> If the subunit parser just gets all the output from the test script,
>
More than that: it gets the *command* that should be used to run the test
script (this is usually just the test script path, but might be a little
more complicated in case the test has an associated LOG_COMPILER).  This
way, the driver can also get the script exit status and standard error
(and possibly other informations, e.g., the test execution time).

> you might want to use the subunit parser itself:
>
You mean the existing python implementation, right?  Well, honestly I was
planning to do exactly that (and similarly for TAP, using the TAP::Parser
module), since I've seen I won't be able to have portable TAP/SubUnit
parsers by the midterm evaluation deadline.  Also, such parsers would
provide an existing (albeit temporary) implementation I can write test
cases against, which will offer a way to make the future "portable"
parsers more compatible with the existing "reference" implementations.

> folk in the
> bootstrap-set couldn't (obviously), but most other environments can
> use high level languages freely. (I don't object to more parser
> implementations existing, I'm just thinking about reuse where
> possible.
>
Agreed, that would be a good idea.  And the current design should make
it quite easy to implement.

>
> [SNIP]
>

> >>
> >> >  3. Console output from the test driver
> >> >  ---------------------------------------
>
> [SNIP]
>
> >> Wrt. to the details of the output, I wouldn't fix things too early.
> >> Two things two consider here:
> >> - if tests are run in parallel, you want to avoid intermixing output
> >>   from different tests as much as possible,
> >>
> > I disagree.  Once the test script name is printed in each line which report
> > the result of one of its test cases, I see no issue with intermixed lines.
> > I.e., I don't believe that this:
> >
> >  PASS: pass.test
> >  PASS: subunit.test [foo]
> >  PASS: tap.test [t1]
> >  SKIP: skip.test
> >  PASS: tap.test [t2]
> >  FAIL: subunit.test [bar]
> >  SKIP: subunit.test [baz] [reason for skipping]
> >  ERROR: tap.test [reason for the bailout]
> >  XFAIL: subunit.test [quux] [failure reason]
> >
> > is any less clear than this:
> >
> >  PASS: pass.test
> >  SKIP: skip.test
> >  PASS: subunit.test [testcase name/description]
> >  FAIL: subunit.test [testcase name/ description]
> >  SKIP: subunit.test [testcase name/description] [reason for skipping]
> >  XFAIL: subunit.test [testcase name/description] [failure reason]
> >  PASS: tap.test [testcase name/description]
> >  PASS: tap.test [testcase name/description]
> >  ERROR: tap.test [reason for the bailout]
> >
> >>   but even more than that, you want to avoid intermixing within lines.
> >>
> > OTOH, I do believe this is a real concern, to be carefully addressed and
> > tested for.  Thanks for bringing this up.
> 
> For Both TAP and subunit the test script running needs to feed into a
> single parser:
>
This is not possible with the current implementation/design of third-party
test drivers in Automake.  Every test script listed in $(TESTS) is passed
*separately* to dedicated instance of the proper test driver; this means
that the output of *each* test script in $(TESTS) will be parsed by a
different instance of the TAP (or SubUnit) parser.

> the issue with intermingling then is all about how the
> parsers structure their output.
> 
> Ideally you would use a threadsafe structure.
>
> For subunit, *if* you were working in python you could use the
> existing multithreaded logic testrepository uses, which can mingle
> multiple streams correctly, preserving datestamps - a single ui parser
> then handles all the combined output.
>
While all this seems nice and sensible, it is unfortunately incompatible
with the basic design choice of the new Automake third-party test drivers
support, which is the ability of share as much code and semantics as
possible with the pre-existing 'parallel-tests' driver, and to easily
associate a `.log' file to each test script.

> >
> >> >  4. RST support and HTML generation: should be dropped?
> >> >  ------------------------------------------------------
> >>
> >> Good question.  You could do a poll on the automake list (in a separate
> >> mail & thread, to gain visbility).  Or just defer the implementation
> >> for later.
> >>
> >> In another mail:
> >> > BTW, I also hope that new interface I'm planning will make it easier to
> >> > implement HTML report generation also for TAP and SubUnit tests, with a
> >> > consistent reuse of the existing code.  In which case my considerations
> >> > above will become moot.
> >>
> >> Cool.
> >>
> > The downside of this is that it might place additional burdens on the
> > writer of test drivers. Hmmm... maybe a new option "no-html-test" or
> > similar is warranted?
> 
> At least for subunit, and I believe TAP afficiondos will agree for
> TAP, you'd generally run subunit2html [made upname] to generate a html
> report. An example of such a formatter is subunit2junithtml, and I
> believe samba have a perl based html formatter for subunit already
> (Jelmer will know more about that).
> 
For the moment, I've managed to provide an interface that is unobtrusive and
doesn't break the existing RST->HTML conversion; i.e., test results are
registerd in the log files using the `:test-result:' reStructuredText field
(some small additional contraints are required if the RST->HTML conversion
is to be supported, but they are not required by any other testsuite harness
feature, so that the autohr of a test driver won't be forced to implement
them; this is an intended behaviour, and is now tested by the Automake's
own testsuite).

Regards,
  Stefano