Re: Critical issues

[Carl Sorensen]

On 5/13/10 1:11 PM, "Karl Hammar" <address@hidden> wrote:

> Graham:
>> On Thu, May 13, 2010 at 10:06:45AM +0200, Karl Hammar wrote:
>>> How do one do a regtest?
>> 
>> Regression check; by compiling stuff.
>> 
>>>   8.1 Introduction to regression tests
>>> 
>>>   The regression tests are automatically compiled using special `make'
>>>   targets.  The output of the regression tests is also automatically
>>> 
>>> So, what targets?
>> 
>> They might be "make baseline-test", followed by applying a patch,
>> followed by "make check", but I'm not certain.  It's explained
>> somewhere in Contributing 2.
> 
> Ok, found something in "3.6.3 Testing LilyPond" (though nothing in the
> chapter on regression tests).
> 
>    * Initial test:
> 
>           make [-jX]
>           make test-baseline
>           make [-jX CPU_COUNT=X] check
> 
>    * Edit/compile/test cycle:
> 
>           _## edit source files, then..._
> 
>           make clean                    _## only if needed (see below)_
>           make [-jX]                    _## only if needed (see below)_
>           make test-redo                _## redo files differing from
> baseline_
>           make [-jX CPU_COUNT=X] check  _## CPU_COUNT here?_
> 
> Hmm, the make check seems redundant since test-redo already does it:
> 
> $ find . -name GNUmakefile | xargs grep -A 10 test-redo
> ...
> ./GNUmakefile:test-redo:
> ./GNUmakefile-  for a in `cat $(RESULT_DIR)/changed.txt` ; do \
> ./GNUmakefile-          echo removing $$a* ; \
> ./GNUmakefile-          rm -f $$a* ;\
> ./GNUmakefile-  done
> ./GNUmakefile-  $(MAKE) check
> ...
> 
> ./scripts/build/out/output-distance seems to be the workhorse of the
> regression tests. I cannot find any useful documentation of it with:
> 
>  find . -type f | xargs grep output-distance
> 
> except the source code itself.
> 
> But if I already have a known good result from the code tracker,
> how do I compare it with the new result?

What do you mean by "if I already have a known good result from the code
tracker"?

make test-baseline

followed by 

make check

will do exactly what you want.  You should then have, as described in
Contributors' 3.6.3, a file

out/test-results/index.html

that shows the results of the regression tests.

If the only snippet listed in out/test-results/index.html is
output-distance.ly, then that means you haven't broken any previous
regressions.

output-distance.ly is a snippet that has random content, so it will change
every time it's run.  That means that you will *always* get a difference
when you run the regression tests.

Ideally, in this case, you'd see changes in output-distance.ly.  And you'd
add a regression test for the current bug, so there would also be an entry
for the current bug.  Other than that, it would say that there is a large
number of snippets with no differences.

HTH,

Carl