Re: [Denemo-devel] Master is badly broken

[Richard Shann]

On Sun, 2014-06-15 at 12:53 +0200, Éloi Rivard wrote:
> Do you think the testsuite is tough enough to merge the "refactor"
> branch now ?

We seem to be so close, ready to write at least one or two tests, but I
can't get make -C tests check to work (see below).
Also, you said

>         > I create a new test that will try to execute every not
>         builtin scheme
>         > command

is this actually happening?, and if so does it execute the command on
blank file (very weak) rather than a file with music in it - that would
be a small change that would make the test far stronger...
I started trying to create a test for each new command, but I don't have
a method for creating tests that work.
To run the tests I have a separate user with user name "denemo-user". I
don't see any log file at the end of the tests or details of what
failed, I'll paste here what I see - it ends with 1 of 2 tests fail.
This is the result of:
su denemo-user
git pull
make clean
make distclean
./autogen.sh
make && make -C tests check
[...]

> Denemo - MESSAGE : Opening file 
> /home/denemo-user/denemo-git/tests/tmp/scm/SanityCheck.denemo
> OK
> PASS: integration
> /unit/run-and-quit: Denemo - MESSAGE : Loading preference file: 
> /home/denemo-user/.denemo-1.1.7/denemorc
> Denemo - MESSAGE : Denemo version 1.1.7
> Denemo - MESSAGE : Loaded keymap 
> /home/denemo-user/denemo-git/actions/Default.commands
> OK
> /unit/scheme-log: Denemo - WARNING : This is warning
> Denemo - CRITICAL: This is critical
> Denemo - MESSAGE : Loading preference file: 
> /home/denemo-user/.denemo-1.1.7/denemorc
> Denemo - MESSAGE : Denemo version 1.1.7
> Denemo - MESSAGE : Loaded keymap 
> /home/denemo-user/denemo-git/actions/Default.commands
> Denemo - INFO    : This is info
> Denemo - MESSAGE : This is message
> OK
> /unit/scheme-log-error: Denemo - MESSAGE : Loading preference file: 
> /home/denemo-user/.denemo-1.1.7/denemorc
> Denemo - MESSAGE : Denemo version 1.1.7
> Denemo - MESSAGE : Loaded keymap 
> /home/denemo-user/denemo-git/actions/Default.commands
> Denemo - ERROR   : This error is fatal
> OK
> /unit/thumbnailer: Running scheme: (d-CreateThumbnail #f 
> "/home/denemo-user/denemo-git/tests/tmp/thumbnail.png")(d-Exit) 
> /home/denemo-user/denemo-git/tests/fixtures/denemo/blank.denemo
> Denemo - MESSAGE : Loading preference file: 
> /home/denemo-user/.denemo-1.1.7/denemorc
> Denemo - MESSAGE : Denemo version 1.1.7
> Denemo - MESSAGE : Loaded keymap 
> /home/denemo-user/denemo-git/actions/Default.commands
> Success 1
> Denemo - CRITICAL: Thumbnail 128x128 file /tmp/Denemo2g5UiI/denemothumb.png 
> gave an error: Failed to open file '/tmp/Denemo2g5UiI/denemothumb.png': No 
> such file or directory
> Denemo - CRITICAL: Thumbnail 256x256 file /tmp/Denemo2g5UiI/denemothumb.png 
> gave an error: Failed to open file '/tmp/Denemo2g5UiI/denemothumb.png': No 
> such file or directory
> Denemo - MESSAGE : Opening file 
> /home/denemo-user/denemo-git/tests/fixtures/denemo/blank.denemo
> Denemo - INFO    : Creating thumbnail 
> /home/denemo-user/denemo-git/tests/tmp/thumbnail.png
> **
> ERROR:unit.c:121:test_thumbnailer: assertion failed: (g_file_test(thumbnail, 
> G_FILE_TEST_EXISTS))
> /bin/bash: line 5:  1553 Aborted                 
> G_TEST_SRCDIR="/home/denemo-user/denemo-git/tests" 
> G_TEST_BUILDDIR="/home/denemo-user/denemo-git/tests" G_DEBUG=gc-friendly 
> MALLOC_CHECK_=2 MALLOC_PERTURB_=$((${RANDOM:-256} % 256)) ${dir}$tst
> FAIL: unit
> =====================================
> 1 of 2 tests failed
> Please report to address@hidden
> =====================================
> make[2]: *** [check-TESTS] Error 1
> make[2]: Leaving directory `/home/denemo-user/denemo-git/tests'
> make[1]: *** [check-am] Error 2
> make[1]: Leaving directory `/home/denemo-user/denemo-git/tests'
> make: *** [check] Error 2
> make: Leaving directory `/home/denemo-user/denemo-git/tests'
> address@hidden:~/denemo-git$ 
> 


>         >
>         >
>         > The test parses the action directory to find the scheme
>         commands.
>         >
>         > For each command, it checks if an associated test file
>         exists, lets
>         > say tests/fixtures/scheme/THECOMMAND.scm. If so it executes
>         it.
>         
>         Is that to say it executes the script starting with a blank
>         score? Does
>         it then save the score after the script has executed and test
>         against
>         THECOMMAND.denemo ?
>         this would sound like a good framework for testing.
>         
>         
>         >  If not it just executes "(d-THECOMMAND)(d-quit)".
>         >
>         >  This would be a weak test in that case,
>         
>         It could be made quite a bit stronger by making the
>         environment in which
>         (d-THECOMMAND) is executed a more typical environment, by
>         installing a
>         couple of staffs and some chords, leaving the cursor on a
>         chord. Many
>         more commands do useful things when the cursor is on something
>         and when
>         other staffs are present than do something useful in a
>         completely empty
>         score.
>         
>         (d-AddAfter)
>         (d-A)
>         (d-CursorUp)
>         (d-CursorUp)
>         (d-AddNoteToChord)
>         (d-MoveCursorLeft)
>         (d-THECOMMAND)
>         (d-Save "filename= ....")
>         (d-Quit)
>         
>         would generate a distinctive output file for many commands (it
>         creates
>         two staffs, populates one and then executes THECOMMAND in that
>         situation).
>         
>         >  but it could at least check that the command does not
>         provoke a
>         > segfault.
>         
>         >
>         >
>         > Then the test could be a bit tougher. For example, we could
>         decide
>         > that if a scheme command return FALSE, it makes the test
>         fail.
>         
>         I'm not sure that Denemo commands return anything useful. But
>         detecting
>         scheme exceptions would be good - we could exit in the trap
>         handler if
>         noninteractive was set. We might have to fix one or two
>         commands that
>         don't expect to be executed in the given environment - well,
>         we could
>         just write a test for them.
>         >
>         > What do you think ?
>         
>         
>         I think this is excellent - it will require a rule to generate
>         the set
>         of expected output files, rather than diff them (for the
>         initial
>         creation of the expected output), and one to accept an altered
>         set of
>         files (copying them to the expected ones) would be a
>         time-saver too.
>         After a change of version in the Denemo file format all the
>         expected
>         output files would change - you would make just this change,
>         check a few
>         examples and then run the rule to overwrite all the old
>         versions with
>         the new ones.
>         
>         Richard
>         
>         
>         
>         >
>         >
>         > 2014-04-15 21:10 GMT+02:00 Richard Shann
>         <address@hidden>:
>         >         On Tue, 2014-04-15 at 19:13 +0100, Richard Shann
>         wrote:
>         >         > As you remarked, it will be good to generate a
>         new .scm
>         >         script each
>         >         > time
>         >         > a new command is made
>         >
>         >         This script could assume that a variable,
>         >         Denemo-output-filename say,
>         >         was defined which it should use via
>         >
>         >         (d-Save Denemo-output-filename)
>         >         (d-Quit)
>         >
>         >         at the end of the test. (I think I missed the
>         (d-Quit) out of
>         >         the
>         >         current script ...)

Richard