Re: [PATCH] {test-protocols} parallel-tests: fix bug in Automake's own testsuite

[Stefano Lattarini]

On Thursday 23 June 2011, Stefano Lattarini wrote:
> On Thursday 23 June 2011, Ralf Wildenhues wrote:
> > * Stefano Lattarini wrote on Thu, Jun 23, 2011 at 09:52:55AM CEST:
> > >
> > > [SNIP]
> > >
> > > the authors of third-party test drivers *must* allow the direct writing
> > > of `:test-result:' directives in the driver output -- this is the only
> > > way to declare test results that are understood and accounted for by the
> > > new Automake test harness.
> > 
> > So, we are doing inband signaling.
> >
> Yes.
> 
> > But Automake's own testsuite is not a TAP or a subunit one.  Thus it
> > does not need to do inband-signaling.  Right?
> >
> Strictly speaking, it doesn't have to.  But the point of my patches so
> far is exactly to provide a minimal in-band signaling support that can
> be used by *all* test drivers that implements our published APIs -- so
> that the logs produced by those drivers can be parsed by the generic
> 'parallel-tests' harness when creating the final testsuite report.
> 
Elaborating a little more on this, we might decide that the declarations
of the test results can be placed only on the top of the log files (well,
almost -- the must go *after* the first line in the log, line which
*must* contain the declaration of the "global" test script outcome).
The parsing of these "result-declaring" directives will then stop at
the first line that is not a recognized directive.  This would retain
full backward compatibility, at the expense of a little more work on
the part of the test drivers (but *not* on the part of our default
'test-driver' script, which already does something like that anyway).
I had thought of proposing something like this in follow-up patches,
but since such a change will make the testsuite patch we are discussing
here moot, we can implement it right away instead, and save ourselves
some trouble later.  WDYT?

> > The reason I'm harping on this is that, if all existing users of
> > Automake test drivers are updated to use the same kind of inband
> > signaling, then they all are liable to similar bad signals.  That
> > is a potential source for regression.
> >
This wouldn't be anymore the case with my proposal above.

> Correct; that's possible, even if very unlikely.  BTW, I had proposed
> to use of `:am-test-result:' instead of `:test-results:' results exactly
> to make such "spurious" signals less likely (but they'd be still possible
> of course, so that's not a great solution).
> 
> Another possibility would be to make the RST field used to declare
> in-band test results configurable (at either Automake or make time);
> so that the users could protect themselves from spurious test results
> by defining that field to a value they know will never be emitted in
> the stdout/stderr of their tests.
>
And this also is made moot by my proposal above.

>
> [SNIP]
>

Regards,
  Stefano