Re: tests updates

[Stefano Lattarini]

On Thursday 04 November 2010, Ralf Wildenhues wrote:
> * Stefano Lattarini wrote on Thu, Nov 04, 2010 at 01:03:17PM CET:
> > On Wednesday 03 November 2010, Ralf Wildenhues wrote:
> > > * Stefano Lattarini wrote on Wed, Nov 03, 2010 at 02:27:47PM CET:
> > > > Just one question: what about the already-existing "tests-init" branch?
> > > > Should I try to bring it to a point where it can be easily merged into
> > > > master, and then forget about it, or should it continue to live parallel
> > > > to master?
> > > 
> > > Right now the branch looks like its changes would be fairly safe to
> > > merge to master, but also, it would mean that sharing testsuite patches
> > > between branch-1.11 and master may become a bit more cumbersome.  So I'd
> > > say it depends a bit on where you want to go with this in the long run.
> > Well, I'd like to:
> > 
> >  1. reorganize the code in "tests/defs.in" in a more rational way
> >     (basically just a reordering of the code);
> 
> Seems like a good idea.
> 
> >  2. (re)introduce the `run_command' function and use it throughout;
> 
> I like it but before it can be merged it needs test exposure on
> Tru64/OSF, IRIX, and Solaris at least, ideally also AIX and HP-UX.
> You should be able to test on these systems in the near future,
> hopefully.
Hmm... more details about this off-list.

> >  3. make it easier for the user to override the aclocal "search path"
> >     in the testsuite (a new pending patch from Paolo Bonzini could
> >     help with this);
> 
> Why would the user need to mess with our testsuite?
It shouldn't; the feature proposed by Paolo (with a patch) is for
"production use", not for the testsuite -- but we could also use it
to improve our testsuite, without additiona hassle.
> Oh, is this so Libtool testing works with distcheck and with unusual
> setups?
Basically, yes.  But as I said, once the Paolo's patch is in place,
implementing point (3) should be (almost) trivial.

> >  4. improve requirements' declaration in Automake test scripts, and
> >     make it easier to run the Automake testsuite using different C,
> >     C++ and Fortran compilers, without leaning too much towards the
> >     GNU compilers as it does now.
> 
> This is a noble goal; I support it in a way that tests which are aimed
> at showing off Automake API which is supposed to be portable to
> different compilers, should also work with different compilers.  Tests
> which are aimed at verifying more internal things do not need to be made
> artificially portable to external tools.
Agreed.
> Also, we should strive to not
> make ourselves too dependent on bugs in Autoconf or Libtool; while that
> can be helpful to uncover bugs in those tools, it also means that we
> make ourselves liable to timely fixes, or version tests and all.
OK; but I'd prefer some spurious failures here and there to lurking dormant
bugs and/or a false sense of security due to lack of testsuite exposure to
a larger array of configurations.

> > These major changes would be interspesed with minor code cleanups
> > and refactoring, such as (to list a few) avoiding to leak temporary
> > variables from `tests/defs' into the test scripts, renaming `$curdir'
> > to (say) `$testbuilddir' for better clarity, and improve the messages
> > from skipped tests.
> 
> The minor cleanups sound ok, with the caveat that they have good chances
> of breaking merges, or test suite additions coming from the stable
> branch.  It is thus a good idea not to change things gratuitously here,
> and it may be prudent to do defs API changes in a branch off of maint,
> so it can be merged into other active branches before they are merged
> into master.
Good idea: taking note of it.  BTW, I think that for potentially problematic
changes I could just forget the 72-hours-gracetime, and wait for a thorough
review from you (or someone else, if anyone steps ahead ;-).

> > > We can easily move to a strategy of having new testsuite stuff added to
> > > master only by default, and only care about branch-1.11 for fixing
> > > existing regressions and serious bugs, which would probably make the
> > > situation easier.
> > Yes, I'd like to go this way.
> 
> OK then.
Good; I'll try to do the merge of "tests-init" later (or tomorrow), and
push to master if the testsuite still passes.

> > > Of course there are other active branches too, that shouldn't have a
> > > harder-than-necessary time.
> 
> > Ideally, all the tests that work with current `tests/defs' should
> > continue to work with the new one; the only problem might be that
> > some new tests committed to another branch might cause *maintcheck*
> > failures when merged to master, but that is not a big deal IMHO,
> > and easily fixed anyway.
> OK.
And as you noticed, commits that change the API of `tests/defs' can be
done in a branch off of maint, so that they can be merged into other
active branches before they are merged into master.
 
> > > We could treat merges similarly to patches in announcing the intention
> > > to merge 72h before the fact, in those cases where there is doubt over
> > > the merge (or the precise point at which a merge should be done).
> > This won't be necessary if we stick to master for tests extensions and
> > refactoring.
> 
> Well, don't let the above paragraph of mine be incentive to use less
> branches.  That's not what I intended.  Doing things in separate
> branches *when appropriate* is good!
True, but most of the changes I'm planning should be small, self-contained,
and not distruptive at all, so that thay could be easily done directly in
master, or even in maint.

> > > I must admit that I don't have all that much experience with merges of
> > > longer-term branches done by more than one person; one alternative would
> > > be that you suggest the merge and I do it (if that works out).  Testing
> > > such merges usually requires a bit of additional work, in that both
> > > branches need to be checked for global changes that need adjusting in
> > > the respective other branch.  Let's see how this works out.
> > 
> > In the end, are you OK with having me to merge "test-init" to master right
> > away, and do future testsuite work on master only?
> 
> Yes.
Good.

Thanks,
  Stefano