Re: On reproducing failures of MR

[Jean Abou Samra]

Hi Paulo,


Le 14/02/2022 à 18:50, Paulo Matos a écrit :
> Hi,
>
> I am attempting to understand what are the failures in
> https://gitlab.com/lilypond/lilypond/-/merge_requests/875
>
> I decided to have a baseline of test results from HEAD before applying
> !875. For this I configured git HEAD with ./configure GUILE_FLAVOUR=guile-2.2.



I guess you mean 'git master' (HEAD is the top commit of whatever branch
you happen to be on, if you are on a branch at all).



> After building I ran the tests with `make CPU_COUNT=12 -j12 test`
>
> I get (log tail):
> Running texi2pdf on file /tmp/tmp8dctbdf6.texi to detect default page settings.
>
> Dissecting...
> Writing snippets...
> Processing...
> Processing 
> /home/pmatos/dev/lilypond/build/out/lybook-testdb/snippet-names-cc1d2dc88d557a856e363b03eeaff6cc.ly
> Command '/home/pmatos/dev/lilypond/build/out/bin/lilypond \
>         -I /home/pmatos/dev/lilypond/input/regression/abc2ly -dseparate-page-formats=ps -djob-count=12 
> -dseparate-log-files -dinclude-eps-fonts -dgs-load-fonts --header=texidoc -I 
> /home/pmatos/dev/lilypond/Documentation/included/ -dcheck-internal-types -ddump-signatures 
> -danti-alias-factor=1 -I  "/home/pmatos/dev/lilypond/input/regression/abc2ly"  -I  
> "/home/pmatos/dev/lilypond/build/input/regression/abc2ly" -daux-files  -dread-file-list 
> -dno-strip-output-dir -dtall-page-formats=eps   
> "/home/pmatos/dev/lilypond/build/out/lybook-testdb/snippet-names-cc1d2dc88d557a856e363b03eeaff6cc.ly"'
>  returned non-zero exit status 1.
> make[2]: *** [/home/pmatos/dev/lilypond/build/.././make/ly-rules.make:36: 
> out-test/collated-files.texi] Error 1
> make[1]: *** [/home/pmatos/dev/lilypond/build/.././make/lysdoc-targets.make:19: 
> lysdoc-test] Error 2
> make: *** [/home/pmatos/dev/lilypond/build/../GNUmakefile.in:316: test]
> Error 2



Phooey. Whatever the better methods to obtain what you
want, this will need figuring out since it is important
that new contributors don't get stuck in cryptic build
failures.



> Running that command with -V does not help much:
>
> [...]
> Processing `73/lily-d7b70d6a.ly'
>
> This returns with exit code 1.
>
> Has anyone seen this before?



That command is not going to work in the source tree.
You need to run it in out/lybook-testdb. Furthermore,
it has -dseparate-log-files which is the source of the
non-verbosity: the logging output is redirected to separate
.log files for each .ly files. You should find information
in out/lybook-testdb/73/lily-d7b70d6a.log.

Also, the build system gave you advice on how to find
failures at the beginning of the 'make test' run. Try
these commands.


> Or is there a better way to obtain a test
> baseline so I can compare it with the results from !875 so that I have a
> chance to understand if I can move the MR forward?


Note that 'make test' does not give you a test baseline but
just checks that all regression tests compile. To get
a test baseline, you want 'make test-baseline'. On the other
hand, if all you want is to see the result of a 'make check'
run for the MR, you can just click on the middle green
mark on the MR page (there are three of them on the right
of 'Detached merge request pipeline #... passed') and
scroll to the end of the log to find a link to the
result of the make check run done automatically in the CI
system (it will be
https://vpetzel.gitlab.io/-/lilypond/-/jobs/1500499681/artifacts/test-results/index.html).
As you will find, this is the same as what has been posted
in the form of screenshots on the MR.

Remember, however, that existing tests passing is a necessary
condition for a merge request to be accepted, but definitely
not a sufficient one.


Best,
Jean