Re: Cleaner way to run all tests, despite some failing?

[Jason Pearce]

I agree, it always seems a bit ugly to do this in make, or should we say 
shell. Actually I've totally given up! I am now of the opinion that it 
is better done elsewhere.

I use a perl script. Each test can be run individually with make
> make test.log
And it actually makes a test.log  if it passes and test.log.fail if it 
fails.

Perl then calls make for each test using its system call - which is now 
well supported across a number of platforms. Linux, SunOS and Cygwin all 
work with this approach. Because we still use make to handle 
dependancies, if a test is already up to date it is not re-run. I then 
have perl to do more sophisticated control and reporting.
ie
o run a subset of tests based on regular expression matches.
o call  make as an argument to time, so I can report on the run time of 
each test.
o prety print a nice report file

Jason

Eric Hanchrow wrote:

> Here's the problem:
>
> I have a few executables in the current directory, which are tests.
> When I type "make check", I'd like them all to run -- even if some
> fail, I'd like the others to run.  And I'd like to collect some status
> while they run, which gets reported after the last one finishes.
>
> Here's how I've solved this:
>
>    TESTS := foo bar baz
>
>    check: $(TESTS)
>            @failed_tests=;                                                      
>         \
>            for t in $(TESTS);                                                   
>         \
>            do                                                                   
>         \
>                    echo -n $$t ...;                                             
>         \
>                    ./$$t || failed_tests="$$failed_tests $$t";                  
>               \
>            done;                                                                
>         \
>            test -z "$$failed_tests" || { echo Failed tests: $$failed_tests; 
> false ; }
>
>    .PHONY: check
>
> This works fine, but it's ... shall we say ... a tad ugly.  (And it
> probably subtly depends on features of the shell I happen to be
> using.)  Is there a cleaner way to do this?
>
>