Re: [PATCH 1/2] tests: make test runner a script, not a shell function

[Stefano Lattarini]

On Tuesday 21 June 2011, Jim Meyering wrote:
> Stefano Lattarini wrote:
> > On Tuesday 21 June 2011, Ralf Wildenhues wrote:
> >> * Stefano Lattarini wrote on Mon, Jun 20, 2011 at 11:12:23PM CEST:
> >> > Maybe we should also say that using TESTS_ENVIRONMENT to define a custom
> >> > test runner is now not only strongly deprecated (as it already was I 
> >> > hope),
> >>
> >> No it wasn't.
> >>
> > D'oh.  I'll be more explicit about that in the NEWS entry then.
> >
> >> "test runner" is not a term I would recognize, btw.
> >>
> > Suggestion on how to improve it are welcome.  As usual, I couldn't come
> > up with a better or clearer term :-(
> >
> >> > but also unsupported and not working anymore?
> >>
> >> > Also, should I look for TESTS_ENVIRONMENT usages in google code search?
> >> > I was really hoping to spae myself the pain... ;-)
> >>
> >> Not sure.  If anything, you can use regexes to avoid stuff you're not
> >> interested in.
> >>
> >> > > and if not, is it possible to have a compatible one (without too much
> >> > > maintenance effort duplication)?  No need to go the effort right away.
> >> > >
> >> > Well, we could add a new option "old-parallel-tests" or something like
> >> > that, that causes the old code in 'check.am' (with few tweaks in order
> >> > to support the new parsing of `.log' files) to be used instead of the
> >> > 'test-driver' script.  By refactoring some code in handle_tests(), we
> >> > could ensure not to add any real complexity to the automake script
> >> > (w.r.t. to my patches at least); but the duplication between 'check.am'
> >> > and 'test-driver' will unavoidable IMHO.
> >>
> >> Well, if we need to use a name, then the name should be for the new one.
> >> That way you can have full backward compatibility.
> >>
> > I'd rather give the user an incentive to move forward, by having him
> > retain the old semantics only if he really needs to.
> > [...]
> 
> Hi Stefano,
> 
> I've seen a few projects that require their automake-managed tests
> be run sequentially.  I suspect that some maintainers will not be eager
> to adapt their tests to run in parallel solely to accommodate a newer
> version of automake.  If you have only a dozen or so tests, whether they
> run in sequence or parallel doesn't make a big difference, but the work
> required to "upgrade" may be considerable.
> 
> I urge you not to impose such a change.
> There are already too many projects that require some ancient
> version of automake.  Let's not make it hard to upgrade.
> 
OK, but there's a misunderstanding here.  My proposal above was only about
the "new" and "old" implementations of the 'parallel-tests' suite (where
the "new" one uses the 'test-driver' auxiliary script); I wasn't proposing
to touch the old serial testsuite *at all*.  That is, what I was proposing
was to use the new implementation by default (even if that isn't 100%
backward-compatible w.r.t. complex uses of TESTS_ENVIRONMENT, as we've seen
with coreutils), and, if I really *really* have to, to allow the users that
still want the old semantics and implementation to get them by using a new
option, say 'old-parallel-test' or 'parallel-tests=old'.

True, I *had* thought about deprecating the older serial test harness, but
Ralf made it clear that many projects still rely on it, and might have an
hard time doing the transitions, so that's not feasible unfortunately.  And
thus I've dropped that proposal already.

Sorry for the confusion,
  Stefano