[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [PATCH 11] Fix LAZY_TEST_SUITE handling and $(TEST_SUITE_LOG) recrea

From: Akim Demaille
Subject: Re: [PATCH 11] Fix LAZY_TEST_SUITE handling and $(TEST_SUITE_LOG) recreation.
Date: Sat, 14 Mar 2009 13:39:07 +0100

Le 14 mars 09 à 12:01, Ralf Wildenhues a écrit :

Oh, what a joyful patch; cost me most brains.  Here's why:

It would have been cool if
 make check LAZY_TEST_SUITE=yes

on an up to date test directory would have no work to do at all.
Well.  Couple of things wrong with the prior semantics:
- there would be no summary output on stdout in that case;
- the exit status of `make' would not reflect the fact whether any of
 the tests failed.  I'd consider that a bug.

Hi Ralf,

Thanks for all this wonderful work!  I can't wait to use it!

For the records, here is the version I use currently, some of bugs you mentioned being fixed. I don't know which was the last I sent, so I can't send a patch. Among changes that I believe, are worth mentioning are:

- make recheck
  runs only the test that failed last time.

# Get the list of tests that failed.
LIST_FAILED_TEST_LOGS =                                                 \
  perl -e 'print join " ",                                            \
grep {open(_) && <_> =~ /^(FAIL|XPASS): /} (@ARGV);' \
       -- $(TEST_LOGS)
.PHONY: recheck recheck-html
recheck recheck-html:
        @target=$$(echo $@ | sed -e 's/^re//');                 \
        if test -f $(TEST_SUITE_LOG); then                      \
          TESTS=$$($(LIST_FAILED_TEST_LOGS));                   \
          $(MAKE) $(AM_MAKEFLAGS) $$target TESTS="$$TESTS";   \
        else                                                    \
          $(MAKE) $(AM_MAKEFLAGS) $$target;                     \

Contrary to your implementation, I still accept TESTS=foo.log whereas the genuine test is foo.test. This makes many things easier, and it is also very convenient for the user. I have a test suite which uses many different extensions, in which case it is simpler for me brains to simply enter the (base)name of the test(s).

- LAZY_TEST_SUITE is not flexible enough
It is global, it should be per test-case. I initially started with LAZY_TESTS being the list of lazy tests, but it was inconvenient because being lazy means: do not remove my *.log first. So actually you want the list of strict test logs and remove them. So it is easier to ask the user to report the set of STRICT_TESTS, which defaults to $(TESTS). I still use LAZY_TEST_SUITE in for backward compatibility issues, but I would depart from it.

# Run all the tests.
        @if test -n '$(LAZY_TEST_SUITE)'; then  \
          rm -f $(STRICT_TEST_LOGS);            \
        else                                    \
          rm -f $(TEST_LOGS);                   \
        @rm -f $(TEST_SUITE_LOG)
        @$(MAKE) $(TEST_SUITE_LOG)

I often use


which makes all failing test strict. In other words, successful tests are not rerun by "make check", but failing tests are. This is because our test suite sometimes hits the limit of the machine, and tests timeout for no sound reason. So there is no reason to rerun successful tests, but failing test might pass if the machine has a lesser load.

Description: Binary data

reply via email to

[Prev in Thread] Current Thread [Next in Thread]