Re: Regtest rating project

[Graham Percival]

On Wed, Aug 08, 2012 at 12:27:40PM +0100, Phil Holmes wrote:
> We're getting to the time where the Grand Regression Test Checking
> (TM) project 
> (http://lilypond.org/doc/v2.15/Documentation/contributor/grand-regression-test-checking
> ) has got enough results to allow us to use it to fix poor regtests.
> We therefore need to agree how to get agreement on the changes
> needed to the regtests.  To put it into perspective, there are over
> 1000 regtests, of which 200 have been shown to need attention.  My
> thoughts on our options:

What does "need attention" mean?  My blind guess is that half of
those are simply typos / grammar / clarifying the text.  Then half
of the remainder need fairly simple and unproblematic changes to
the lilypond input.  Finally, half of the remainder will require
significant discussion.


> New Google Project
> ==============
> This occurred to me only recently.  What about setting up a new
> Google Project called "LilyPond Regression Test Improvement".  This
> would require a new mailing list to avoid the problems above.  We
> could then add an issue per regtest, and only people who want to
> subscribe would need to subscribe to the list.
> 
> I think the final suggestion is my favoured one.

I don't favor that option.

Let's look at this backwards: for every single change to lilypond
git, we need a patch on rietveld submitted via git-cl.  In some
cases (e.g., fixing typos) you might get a review within an hour
saying "LGTM, please push directly to staging".  But any
non-totally-trivial change is going to require a full countdown,
comments, etc.  I don't think we should be handing out blank
cheques to anybody with regards to the regtests.  I'm not
comfortable letting any "regtest improvement team" send changes
directly to git without being on a general countdown.

Who wants to be producing these patches?  I recall warning about
this situation before the whole checker started up, and I thought
that there was a volunteer to handle it.  Situations change, of
course, but it's a bit disappointing to be discussing how to
handle the feedback now.

None of the suggestions make the patch-production easier.  They
all center around discussing what types of changes to make, which
is certainly a necessary first step, but they leave the question
of creating patches unanswered.


My suggestion: make the data available, then sit back and see who
wants to deal with it.  Maybe add another column to the
spreadsheet or database for "regtest updated", then that can be
filled in as it happens.  Basically, let people handle it
manually.  I don't think that we're going to have a deluge of
developers interested in working on this, so I don't think that
we'll have synchronization problems from multi-threading.  A
global lock (i.e. person X is working on the feedback; person Y
waits until person X is finished) should suffice.

- Graham