GOP2: 2 - Stable releases and roadmap (radical change)

[Graham Percival]

Not quite up to the ideal standard of GOP proposals, but there's a
lot of interest and this should be enough to see what way the wind
is blowing.

html-formatted version:
http://lilypond.org/~graham/gop/gop_3.html


*** Summary

Let’s drop the “any unintended change” thing, and go totally with
the regression tests. Tests pass? We can make a stable release.
Also, let’s have an official roadmap.


*** Motivation

There seems to be widespread frustration with the current system.
At the moment, any “unintended change” blocks a release (plus a
few extra conditions), so we’re at the mercy of all sorts of
behaviour that isn’t covered by the regtests. This makes it hard
to plan ahead for everybody: developers wanting to work on large
features or refactoring, users, linux distribution packagers, etc.


*** Details: Critical issues

A type-Critical issue will block a stable release, but the
definition is:

-    a reproducible failure to build either make or make doc, from
an empty build tree, in a first run, if configure does not report
any errors.

-    anything which stops contributors from helping out (e.g.
lily-git.tcl not working, source tree(s) not being available,
LilyDev being unable to compile git master, inaccurate
instructions in the Contributor’s Guide 2 Quick start).

        To limit this scope of this point, we will assume that the
contributor is using the latest LilyDev and has read the relevant
part(s) of the Contributor’s Guide. Problems in other chapters of
the CG are not sufficient to qualify as Type-Critical. 

-    any regression test which fails to compile or shows incorrect
output. 

The only change is to the third point, namely the “regression test
failure” as opposed to “any unintentional change”.


*** Details: Regtests

The current regtests don’t cover enough – that’s why we keep on
finding new regression-Critical issues. I think it’s worth
expanding the regtests and splitting them into multiple
categories.

These names don’t (deliberately) match any specific testing
methodology. If they do, then it’s probably a mistake and we
should rename these.

    Crash: we don’t care about the output of these; we just want
to make sure that lilypond doesn’t crash with this input.
    Tiny: these files would test individual features, such as
printing accidentals or slurs, with a minimum of shared features.
    Integration: these are constructed examples which combine
multiple features together.
    Pieces: musically-interesting fragments of music, such as a
few systems from a Bach sonata or Debussy piano work.
    Syntax: short fragments of music for which the .ly files are
“frozen” – we never run convert-ly on these files until LilyPond
4.0. (see below, “roadmap”) 

I figure that we’ll double the total number of regtests. There’s
probably some old ones that can be eliminated (or combined with
newer ones), but we’ll be adding a lot more.


*** Programming regtests

To avoid slowing down programming to a crawl, I figure that we’ll
identify some subset of these regtests and have a separate make
regtests-quick command which only evaluates that subset.

As a rule of thumb, I’d say that the regtests-quick target should
take as long as a make from scratch. I’m sympathetic to developers
with limited computing resources, but I think it’s reasonable to
ask everybody submitting programming patches to “double” the time
it takes to test their patch (since obviously everybody would run
make before submitting anything).

The patchy test-patches will still run the full regtest checks.


*** When breakage occurs

There will of course be functionality which breaks. When that
happens, we file a normal bug. A new regtest can only be added for
that bug when it is fixed – we won’t add the regtest first, then
try to fix it.

In other words, git master should always pass all regtests. If it
doesn’t, then reverting should be the first option.


*** Roadmap

With this change, we would no longer be committed to the same kind
of stability that we were before. As such, I think it’s worth
bumping the version up to 3.0.

The 3.x series will consist of a series of random breakage from
functionality not covered under the existing regtests and from
manual .ly changes required by GLISS. This is intentional – or
rather, we don’t intend to break stuff, but the policy accepts
that this will happen. Somebody may offer to maintain the 2.x
series to cater to users who want additional stability.

Over the next 3 months or so, we’ll discuss a number of syntax
changes in GLISS. Then discussion will cease until all the changes
have been implemented. We’ll then have release 3.2, which will
almost certainly require manual changes to all .ly files.

We’ll then have another few months of GLISS discussions, then a
pause for implementions, then 3.4. Repeat as necessary.

LilyPond 4.0 will mark the ending of GLISS, and by that point we
should have much improved regtest coverage. We can’t really plan
too much for this, since it’s likely two years away.


- Graham