Re: make test

[Graham Percival]

On Wed, Aug 08, 2012 at 09:59:03AM +0100, Phil Holmes wrote:
> I've been looking at how the regression test comparison works.  The
> first thing I find is that we have 2 effectively duplicate, but
> different, pages on running regtest comparisons:
> 
> http://lilypond.org/doc/v2.15/Documentation/contributor/verify-regression-tests
> http://lilypond.org/doc/v2.15/Documentation/contributor/regtest-comparison
> 
> I think the latter is probably more accurate.  I think it would be
> best to delete one and point to the other?

Yes, the regtest-comparison page was written after the
verify-regression-tests; specifically because people didn't
understand the verifying- page.  I note that verifying- already
has a link to regtests-, though.

Hmm, it might be worth looking at the git history to see when the
"typical developer's edit/compile/test/cycle" part was added.
And/or makign the link at the top of verify- easier to see?

> I've also been benchmarking.  For example, I know that make
> CPU_COUNT=9 test is _much_ faster than make test, but the make -j9
> test isn't worth doing - most of the time is spent building the
> single regtest document, which lilypond parallelises much better
> than make.

Rather: make does not parallelize the lilypond call at all.

> I have had errors using -jX so am slightly suspicious of
> that option.  I would like to add the best way to parallelise the
> test process to the CG.

Isn't that what the "typical developer/edit/compiler" bit is
about?

The more I look at this, the worse it gets.  Neither section
discussing the regtests is completely wrong or a complete
duplicate of the other.  This can't be solved by simply deleting
one and linking to the other.

> I've also been looking at how output-distance works.  Does anyone
> now understand what this actually does?  From following the code, it
> looks to me like it doesn't actually compare images - it compares
> the .signature files, and if there's a difference over the
> threshold, it creates an image of the original and changed file, and
> then makes a "ghost" version of the change to overlay on the
> original.  Does this seem correct.  Worth documenting?

I don't understand how it works, but your description sounds very
plausible to me given the way that people talked about
output-distance 5-10 years ago.  I certainly see no problem in
documenting it.

- Graham