Re: GOP2: 2 - Stable releases and roadmap (radical change)

[Graham Percival]

On Wed, Jun 27, 2012 at 05:33:47AM +0000, Keith OHara wrote:
> Graham Percival <graham <at> percival-music.ca> writes:
> > -    any regression test which fails to compile or shows incorrect
> > output. 
> 
> For any changed test then, it is probably worth reading the header, to 
> see if a subtle change that looks harmless happens to be the point of 
> the test (and would presumably cause other trouble).

Hmm.  It could be necessary to add either the texidoc to the
regtest comparison page, or add markup to the score itself to
highlight any subtle points which are vital.

> The "incorrect output" should only count where the previous stable 
> release gave correct output.  Lots of tests show behavior that some 
> people think is wrong (‘accidental-ledger.ly’ ‘ambitus.ly’) or that 
> looks bad because it is a a stress test (‘break.ly’ ‘prefatory-
> separation.ly’ ‘spacing-strict-spacing-grace.ly’).

Yes; there is a hidden assumption that the existing regtests have
been exhaustively checked and we agree that they pass (again,
possibly only after making a note in the texidoc that the output
may look "too squished" or something).

I'll make that assumption explicit.

> > *** Details: Regtests
> > 
> > The current regtests don’t cover enough – that’s why we keep on
> > finding new regression-Critical issues. I think it’s worth
> > expanding the regtests and splitting them into multiple
> > categories.
> 
> This cannot be done quickly.

Yes; I should have noted that it will likely take 100+ hours.

> Adding a few pieces of music in various styles might help, but I
> remember from my regression this cycle that my patch worked fine
> on score and parts of a full symphony, but then version 2.15.37
> failed spectacularly on two pages of guitar music.

Yes.  Under this proposal, version 2.15.37 could have became
2.16.0, even with that spectacular failure known, because that
particular edge case was not checked in the regtests.  The fix
would not occur until 2.16.1 or later -- possibly years later.

> >     Tiny: these files would test individual features, such as
> > printing accidentals or slurs, with a minimum of shared features.
> 
> As a goal, I suggest "Targeted" instead of "Tiny" -- testing performance
> in one narrow area, but thoroughly.  More often than not, after I fix 
> a bug, I find a regtest that should have caught it, and expand that 
> test rather than add a new one.

good idea!  I've modified the name.

- Graham