Re: GOP-PROP 4: lessons from 2.14

[Carl Sorensen]

On 6/30/11 9:21 AM, "Graham Percival" <address@hidden> wrote:
> On Thu, Jun 30, 2011 at 11:17:36AM -0300, Han-Wen Nienhuys wrote:
>> We should examine other reasons for the delay of the 1st candidate
>> appearing, and the delay between the 1st and last release candidate,
>> and figure out if there is a way to prevent them from causing large
>> delays again.
> 
> Nice way to split it up!
> 
> Regular 2.13 releases did not happen until 2009-10; it was
> therefore impossible to have a release candidate before then.
> Between 2009-10 and 2011-01 (first release candidate), my vague
> sense is that roughly 50% of Critical bugs originated from before
> 2010-08 (when Phil began checking regtests).
> It would be nice if somebody checked into this... maybe select 10%
> of the bugs, then look at when they were introduced?  (most
> regression bug discussions began by finding the first version that
> broke)

There 148 issues marked with Priority=Critical in the tracker.

2.13.0 came out on 2009-02-28, according to git.

I've done an analysis, and it looks to me like there was initially a backlog
of critical issues that weren't fixed, and little work was being done to
eliminate critical issues.

Somewhere about 2010-08-01, critical issues started to disappear, but
occasional new ones appeared.

There were a couple of major changes that introduced unanticipated
regressions (new spacing code, beam collision avoidance).  These produced
more than the expected number of regressions.

It appears to me that we didn't really get serious about eliminating
critical bugs until about 2010-06-15 or so.  After that point, the number of
critical bugs more-or-less steadily decreased until we got to a release
candidate.

Of particular interest, the first release candidate of 2.14 was released on
2011-01-12.  Over the next 10 days, about a dozen bugs were reported and
fixed.  Release candidate 2 came out on 2011-02-09.   No surge of bugs
occurred with this release.  Candidate 3 came out on 2011-03-13; we got 2
bugs per week.  Candidate 4 came out on 2011-03-29; 2 new bugs.  Candidate 6
came out on 2011-04-07.  We got a couple of bugs per week.

>From this analysis, I'd say the most important thing we can do to get stable
versions released more often is to focus on keeping the count of critical
bugs at 0.  If we can do that, we can be ready to make a release candidate
at any time -- and we might even be able to release multiple times per year.

The major roadblock I see to this is the fact that we want to make big
improvements (like the revised spacing engine).  Perhaps such improvements
should be first done on a branch other than master, but eventually we have
to move it to master in order to get bug reports on the tracker.  And once
we get it to master, there will be regressions that we have to deal with.

By way of assessing our current standing, we currently have 3 critical
issues, and work is underway on all 3.  So we're very close to being stable
release-candidate ready.

Thanks,

Carl

P.S. I've attached a csv file of my analysis; I've got a .xls file if
anybody is interested.

lilypond-issues-analysis.csv
Description: lilypond-issues-analysis.csv