[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: unit tests best practice
Re: unit tests best practice
Thu, 3 Jan 2019 10:44:51 -0800
On Thu, Jan 03, 2019 at 05:53:33PM +0100, Catonano wrote:
> Wat's the best practice to instrument a Guile based project for running
> unit tests ?
> guile-git has this fragment in the Makefile.am file
> SCM_LOG_DRIVER = \
> $(top_builddir)/pre-inst-env \
> $(GUILE) --no-auto-compile -e main \
> as you can see, what Automake produces calls a "main" procedure in a
> "test-driver.scm" script in the "build-aux" folder
> The effect is that you can call
> make check
> in the root folder and have the tests run
> Instead, in the Makefile.am file produced by guile-hall wired projects you
> ave this fragment
> AM_TESTS_ENVIRONMENT = abs_top_srcdir=\"$(abs_top_srcdir)\"
> SCM_LOG_COMPILER = $(top_builddir)/test-env $(GUILE)
> AM_SCM_LOG_FLAGS = --no-auto-compile -L \"$(top_srcdir)\"
> so there's a SCM_LOG_COMPILER instead of a SCM_LOG_DRIVER
> What is this LOG_COMPILER doing, exactly ?
OK, this is a muddle, but, it goes like this.
LOG_DRIVER and LOG_COMPILER come from Automake. If you use them,
Automake will generate "make check" rules for all the test scripts in
When you use LOG_COMPILER, Automake considers a test to pass if it
returns an exit status of 0, perhaps by calling '(exit 0)' in Guile.
Exit code 1 = failure. 77 = skip. 99 = fatal eror. So your TESTS
scripts as run by the command line in LOG_COMPILER need to return
proper exit codes. Automake will also automatically gather anything
your tests print into .log files and .trs (test result files).
If you use LOG_DRIVER, Automake expects your scripts to manually
create its .log and .trs files. You use LOG_DRIVER if the simple
exit-code-based pass/fail criteria isn't good enough. Look
in the Automake docs under 'Custom Test Drivers' for info.
> This is what the test-env file contains
> \"@abs_top_builddir@/pre-inst-env\" \"address@hidden"
> exit $?
> what is this ?
> What does this do ?
Typically these pre-inst-env scripts set up the GUILE_LOAD_PATH for
when you are running the tests in the development tree before
installing them. (There is no standard pre-inst-env script; there are
a few different versions.) GUILE_LOAD_PATH works better with absolute
paths. It may also set compiled paths, environment variables, or
> Alex admits he goes full cargo culting when dealing with the Autotools, in
> fact there's this comment line at the beginning of the section producing
> the Autotools related files in guile-hall
> ;;;; Full on cargo cult!
> So, if there's any kind soul here willing to clarify/illustrate what is
> going on we could make the effort to condensate such knowledge in
> Improving the experience in dealing with Guile based project would be a
> great progress
> One last thing
> test-driver uses the API provided by srfi-64 to actually run the tests and
> display the results
> Does Guile provide a standard version of such script ?
> Should Guile provide one ?
I should note that srfi-64 doesn't work out of the box with
LOG_COMPILER because it doesn't return the proper exit status on
failure, either 0 = pass, 1 = fail, 77 on skip, 99 on hard error. It
isn't that complicated to add that functionality in, and that would be
a great project for someone to do.
For the one project were I tried to make a full test suite
(guile-ncurses), I went with lots of tiny scripts and used the
LOG_COMPILER exit code strategy. For that I make a small test library
here that converts #t and #f from a procedure into exit codes.