[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: quiet mode for make check (silence passing tests, list failing tests

From: Chris Pickett
Subject: Re: quiet mode for make check (silence passing tests, list failing tests)
Date: Mon, 03 Mar 2008 12:13:06 -0500
User-agent: Thunderbird (Macintosh/20071210)

Hi Ralf and Akim and co.,

(Please include me in any reply, I am not subscribed.)

Ralf Wildenhues wrote:
* Chris Pickett wrote on Wed, Jan 23, 2008 at 10:56:29AM CET:
I have one test program for each module in my library. Each writes to stdout and stderr. Since I have many tests, this generates a lot of output when I run `make check'. I would like to change `make check' so that it:

1) Causes only those tests that fail to print anything
2) Additionally lists the tests that fail at the end

But it's also possible inside  In fact, there have been
patches floating around for parallel testsuites, which can hopefully be
integrated before 1.11, which deal smartly with verbose output (keeping
verbose log files of failed tests).

I found the patch (set of redefinitions) you were referring to:

The original is available here:

I adapted it for my own project here:

These are the differences in order if you diff the two files (comments are painful with trailing backslashes):

1) Remove a blank line for each individual test log:

FAIL: <test>
              <--- here
Output from test.

2) Don't print PASS: <test> or FAIL: <test> for each test as it executes. This is just too verbose for me; if everything passes, all I want is a green bar (box) at the bottom. If something fails, I want the full log, not just a message that it failed.

3) Don't show the exit status for a test.  Too verbose for me.

4) Remove the boilerplate echoing the PACKAGE_STRING and the name of the file into $(TEST_SUITE_LOG) since this is obvious from position. Also remove some idiosyncratic formatting stuff (".. contents:: :depth: 2").

5) Remove the "See $(subdir)/$(TEST_SUITE_LOG)" message (see change #7). I want to keep the red bar (box) message concise.

6) s/Please report it to/Please report to/ in line with current Automake.

7) Always send $(TEST_SUITE_LOG) to stdout. If a test fails, I'm going to check what happened anyways. Originally, a VERBOSE variable guarded that, but I think VERBOSE should be default.

8) Show $(TEST_SUITE_LOG) before the am__text_box announcing the total number of tests that pass/fail. I think you want the red/green bar to appear at the bottom of the output and not have to scroll back for it.

9) Always remove $(TEST_SUITE_LOG) so that the pass/fail output is always regenerated. This does not significantly affect running time.

Additionally, in order to use this, I added the following to my tests/

# this rule seems necessary to quiet errors
# about not knowing how to create .log

TEST_LOGS = $(TESTS:=.log)

# parallel/quiet make check from vaucanson-1.1.1
include $(top_srcdir)/build-aux/

I don't really know about the .log rule but meh, it works. Also, I tried using the EXTRA_PROGRAMS variable, and while it apparently let the tests run concurrently with the compilations, I prefer the output I get with make check (`<test> is up to date.') and also since change #2 removes the immediate feedback on pass/fail I still have to wait for everything to run anyway. Trying to look for a red message and hit Ctrl-c and then cat the appropriate tests/<test>.log is not what I want.

Anyway, thanks Akim for the hard work on It's much much better for me. By the way, my output looks like this:

gmake  check-TESTS
gmake[2]: Entering directory `$HOME/libspmt/tests'
rm -f test-suite.log
gmake[3]: Entering directory `$HOME/libspmt/tests'

FAIL: predictor
Running suite(s): predictor
Assertion failed: predictor != NULL, file  predictor.c, line 127
66%: Checks: 3, Failures: 1, Errors: 0
predictor.c:46:F:core:check_half_width_return_value:0: Assertion '(spmt_predictor_predict (predictor) == 3)' failed

 1 of 48 tests failed
 Please report to chris.pickett AT

gmake[3]: *** [test-suite.log] Error 1
gmake[3]: Leaving directory `$HOME/libspmt/tests'
gmake[2]: *** [check-TESTS] Error 2
gmake[2]: Leaving directory `$HOME/libspmt/tests'
gmake[1]: *** [check-am] Error 2
gmake[1]: Leaving directory `$HOME/libspmt/tests'
gmake: *** [check-recursive] Error 1

where the box is in red. In case you're wondering, I'm using the Check unit testing framework for C. It works quite well for me, but I'm also a maintainer. When things do work, the output looks like this now:

make  check-TESTS
rm -f test-suite.log

 All 48 tests passed

where the box is in green. That's on a different machine (OS X, not AIX), and for some reason I don't get the entering/leaving messages. Not sure which I prefer. Of course, all the .log files are available for individual tests, regardless of pass/fail status.

Finally, I noticed that is GPL, where as the Autoconf stuff in build-aux comes with this notice:

# As a special exception to the GNU General Public License, if you

# distribute this file as part of a program that contains a

# configuration script generated by Autoconf, you may include it under

# the same distribution terms that you use for the rest of that program.

I don't know if it matters. I'm assuming that using as GPL doesn't mean that my library is forced to be GPL:

## This file comes from vaucanson-1.1.1.  Since it only affects the
## build process, it is assumed that libspmt is not forced to be GPL.

but I guess I would feel better if that exception notice was there? (That `?' has a mysterious meaning.) I suppose if gets into Automake (which I hope) then everything will straighten out anyway.


reply via email to

[Prev in Thread] Current Thread [Next in Thread]