Re: running tests under in a tests/ directory

[aaragon]

Hi Ralph, thanks again for replying. 


Ralf Wildenhues wrote:
> 
> Hello Alejandro,
> 
> * aaragon wrote on Sat, May 30, 2009 at 07:34:13PM CEST:
>> 
>> Thank yo Ralf for replying to my post. The code that I showed in my post
>> was
>> indeed taken from that website. The code you provided me is way too
>> advanced
>> for me, since I consider myself just a user. The main idea that I got
>> from
>> your reply is that this new code won't work with older versions of
>> Automake.
> 
> Right.  But as William already replied, you only need Automake on your
> development system and not on systems where the code is configured and
> built.
> 
>> Now, regarding the code that you wrote, I am trying to make it work, so I
>> changed the Makefile.am in the root directory to what you suggested
>> (almost):
>> 
>> # test rule
>> TSTDIR = tests
>> 
>> check:        all
>>       @echo
>> "-------------------------------------------------------------------"
>>       @echo "Running tests"
>>       cd $(TSTDIR) && $(MAKE) $(AM_MAKEFLAGS) -s test
>>       @echo
>> "-------------------------------------------------------------------"
>> 
>> Except that if I change 'check: all' to something else, then it doesn't
>> work
>> anymore.
> 
> "doesn't work"  is too vague.  What does not work, what error message do
> you get, more details please?
> 
> 
> 

In your first reply you suggested to use check-local, but if I put in the
Makefile.am in the root director:

check:  check-local
        @echo 
"-------------------------------------------------------------------"
        @echo "Running tests"
        cd $(TSTDIR) && $(MAKE) $(AM_MAKEFLAGS) -s test
        @echo 
"-------------------------------------------------------------------"

Then I get

address@hidden/Documents/workspace/cpputils$make check
make: *** No rule to make target `check-local', needed by `check'.  Stop.


Ralf Wildenhues wrote:
> 
> 
> 
>> In the tests directory, I changed the Makefile.am to reflect that I use
>> only
>> cpp files. I also made the generate-verified-files silent, so that the
>> output of that target is:
> 
>> Now, from the code that I got from the internet, I really liked the fact
>> that you don't need to specify the name of the tests in the Makefile.am
>> file
>> because of the use of wildcards. Is there a way to make something similar
>> here that is really portable?
> 
> Not really; well, at the very least you will need to have a naming
> scheme such that all tests have some specified suffix(es), for example
> '.test'.  Your sources would then match '*.test.cpp'.
> 
> 

Can you be more specific on this? All my files start with the word 'test',
but I can change the file names so that they end in 'test.cpp' if needed.
However, I tried to put 

TESTEXECS = 'test*.cpp'

in the Makefile.am under the tests directory and it gives me an error:

address@hidden/Documents/workspace/cpputils$m check
make  all-recursive
Making all in cpputils
make[2]: Nothing to be done for `all'.
Making all in tests
make[2]: Nothing to be done for `all'.
Making all in examples
make[2]: Nothing to be done for `all'.
Making check in cpputils
make[1]: Nothing to be done for `check'.
Making check in tests
make  check-TESTS
make[3]: *** No rule to make target `'test*.cpp', needed by `'test*.o'. 
Stop.
make[2]: *** [check-TESTS] Error 2
make[1]: *** [check-am] Error 2
make: *** [check-recursive] Error 1
address@hidden/Documents/workspace/cpputils$

It would be very nice if I can automate the detection of the source files.
How do I accomplish this?


Ralf Wildenhues wrote:
> 
> 
> 
>> If I make one of the tests fail (by deliberately changing one of the
>> verified files), I get a message saying that a binary operator is
>> expected:
>> 
>> address@hidden/Documents/workspace/cpputils/tests$make check
>> make  check-TESTS
>> FAIL: test001
>> PASS: testcppblas
>> /bin/sh: line 0: test: Alejandro: binary operator expected
>> ========================
>> 1 of 2 tests failed
>> See tests/test-suite.log
>> ========================
>> make[2]: *** [test-suite.log] Error 1
>> make[1]: *** [check-TESTS] Error 2
>> make: *** [check-am] Error 2
>> 
>> Is this normal?
> 
> Not sure, it could be a bug in my suggested code or in Automake.  Can
> you post the output of
> 
>   make check SHELL=/bin/sh\ -x
> 
> for that, as well as the tests/Makefile.am and the automake version
> you're using?  Thanks.
> 
> Cheers,
> Ralf
> 
> 

I'm using:
address@hidden/Documents/workspace/cpputils$automake --version
automake (GNU automake) 1.11
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv2+: GNU GPL version 2 or later
<http://gnu.org/licenses/gpl-2.0.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Tom Tromey <address@hidden>
       and Alexandre Duret-Lutz <address@hidden>.


If I type 'make check' and the tests are successful, I get the following
error at the end:

address@hidden/Documents/workspace/cpputils$make check
make  all-recursive
Making all in cpputils
make[2]: Nothing to be done for `all'.
Making all in tests
make[2]: Nothing to be done for `all'.
Making all in examples
make[2]: Nothing to be done for `all'.
Making check in cpputils
make[1]: Nothing to be done for `check'.
Making check in tests
make  check-TESTS
PASS: test001
PASS: testcppblas
==================
All 2 tests passed
==================
Making check in examples
make[1]: Nothing to be done for `check'.
-------------------------------------------------------------------
Running tests
cd tests && make  -s test
make[1]: *** No rule to make target `test'.  Stop.
make: *** [check] Error 2
address@hidden/Documents/workspace/cpputils$


I think that error is because of the following line in the Makefile.am in
the root director:

        cd $(TSTDIR) && $(MAKE) $(AM_MAKEFLAGS) -s test

Shouldn't I just use

        cd $(TSTDIR) && $(MAKE) $(AM_MAKEFLAGS) -s

?

Here I make one of the tests fail deliberately. This is the output of "make
check SHELL=/bin/sh\ -x" for this case, where you can see the error I was
referring to in my post at the end:

address@hidden/Documents/workspace/cpputils$make check SHELL=/bin/sh\ -x
+ test '!' -f config.h
+ :
make  all-recursive
+ make all-recursive
+ failcom='exit 1'
+ for f in x '$MAKEFLAGS'
+ case $f in
+ for f in x '$MAKEFLAGS'
+ case $f in
+ for f in x '$MAKEFLAGS'
+ case $f in
+ dot_seen=no
++ echo all-recursive
++ sed s/-recursive//
+ target=all
+ list='cpputils tests examples'
+ for subdir in '$list'
+ echo 'Making all in cpputils'
Making all in cpputils
+ test cpputils = .
+ local_target=all
+ CDPATH=:
+ cd cpputils
+ make all
make[2]: Nothing to be done for `all'.
+ for subdir in '$list'
+ echo 'Making all in tests'
Making all in tests
+ test tests = .
+ local_target=all
+ CDPATH=:
+ cd tests
+ make all
make[2]: Nothing to be done for `all'.
+ for subdir in '$list'
+ echo 'Making all in examples'
Making all in examples
+ test examples = .
+ local_target=all
+ CDPATH=:
+ cd examples
+ make all
make[2]: Nothing to be done for `all'.
+ test no = no
+ make all-am
+ test '!' -f config.h
+ :
+ test -z ''
+ failcom='exit 1'
+ for f in x '$MAKEFLAGS'
+ case $f in
+ for f in x '$MAKEFLAGS'
+ case $f in
+ for f in x '$MAKEFLAGS'
+ case $f in
+ dot_seen=no
++ echo check-recursive
++ sed s/-recursive//
+ target=check
+ list='cpputils tests examples'
+ for subdir in '$list'
+ echo 'Making check in cpputils'
Making check in cpputils
+ test cpputils = .
+ local_target=check
+ CDPATH=:
+ cd cpputils
+ make check
make[1]: Nothing to be done for `check'.
+ for subdir in '$list'
+ echo 'Making check in tests'
Making check in tests
+ test tests = .
+ local_target=check
+ CDPATH=:
+ cd tests
+ make check
make  check-TESTS
+ make check-TESTS
+ list='test001.log testcppblas.log'
+ test -z 'test001.log testcppblas.log'
+ rm -f test001.log testcppblas.log
+ test -z test-suite.log
+ rm -f test-suite.log
+ set_logs=
+ test 'Xtest001.log testcppblas.log' = X.log
+ make test-suite.log
+ p=test001
+ case $- in
++ echo .
++ sed 's|.|.|g'
+ srcdirstrip=.
+ case $p in
+ f=test001
+ srcdir=.
+ export srcdir
+ rm -f test001.log-t
+ trap 'st=$?; rm -f
'\''/Users/aaragon/Documents/workspace/cpputils/tests/test001.log-t'\'';
(exit $st); exit $st' 1 2 13 15
++ echo ./test001.log
++ sed 's|/[^/]*$||'
+ am__odir=.
+ test x. = x.
+ test -f ./test001
+ dir=./
+ tst=./test001
+ log=test001.log
+ __SAVED_TERM=xterm-color
+ ./diff-driver ./test001
+ estatus=1
+ test -n ''
+ TERM=xterm-color
+ export TERM
+ red=
+ grn=
+ lgn=
+ blu=
+ std=
+ test X '!=' Xno
+ test Xxterm-color '!=' Xdumb
+ test X = Xalways
+ test -t 1
+ red=''
+ grn=''
+ lgn=''
+ blu=''
+ std=''
+ xfailed=PASS
+ case "  " in
+ case $estatus:$xfailed in
+ col=''
+ res=FAIL
+ echo 'FAIL: test001'
FAIL: test001
+ echo 'FAIL: test001 (exit: 1)'
+ sed 'p;s/./=/g;p;g'
+ cat test001.log-t
+ rm -f test001.log-t
+ p=testcppblas
+ case $- in
++ echo .
++ sed 's|.|.|g'
+ srcdirstrip=.
+ case $p in
+ f=testcppblas
+ srcdir=.
+ export srcdir
+ rm -f testcppblas.log-t
+ trap 'st=$?; rm -f
'\''/Users/aaragon/Documents/workspace/cpputils/tests/testcppblas.log-t'\'';
(exit $st); exit $st' 1 2 13 15
++ echo ./testcppblas.log
++ sed 's|/[^/]*$||'
+ am__odir=.
+ test x. = x.
+ test -f ./testcppblas
+ dir=./
+ tst=./testcppblas
+ log=testcppblas.log
+ __SAVED_TERM=xterm-color
+ ./diff-driver ./testcppblas
+ estatus=0
+ test -n ''
+ TERM=xterm-color
+ export TERM
+ red=
+ grn=
+ lgn=
+ blu=
+ std=
+ test X '!=' Xno
+ test Xxterm-color '!=' Xdumb
+ test X = Xalways
+ test -t 1
+ red=''
+ grn=''
+ lgn=''
+ blu=''
+ std=''
+ xfailed=PASS
+ case "  " in
+ case $estatus:$xfailed in
+ col=''
+ res=PASS
+ echo 'PASS: testcppblas'
PASS: testcppblas
+ echo 'PASS: testcppblas (exit: 0)'
+ sed 'p;s/./=/g;p;g'
+ cat testcppblas.log-t
+ rm -f testcppblas.log-t
+ case $- in
+ list='test001.log testcppblas.log'
++ for f in '$list'
++ read line
++ echo 'FAIL: test001 (exit: 1)'
++ for f in '$list'
++ read line
++ echo 'PASS: testcppblas (exit: 0)'
+ results='FAIL: test001 (exit: 1)
PASS: testcppblas (exit: 0)'
++ echo 'FAIL: test001 (exit: 1)
PASS: testcppblas (exit: 0)'
++ sed '/^$/d'
++ sed -e 's/^[  ]*//'
++ wc -l
+ all=2
++ echo 'FAIL: test001 (exit: 1)
++ grep -c '^FAIL'
PASS: testcppblas (exit: 0)'
+ fail=1
++ echo 'FAIL: test001 (exit: 1)
PASS: testcppblas (exit: 0)'
++ grep -c '^PASS'
+ pass=1
++ echo 'FAIL: test001 (exit: 1)
++ grep -c '^SKIP'
PASS: testcppblas (exit: 0)'
+ skip=0
++ echo 'FAIL: test001 (exit: 1)
PASS: testcppblas (exit: 0)'
++ grep -c '^XFAIL'
+ xfail=0
++ echo 'FAIL: test001 (exit: 1)
PASS: testcppblas (exit: 0)'
++ grep -c '^XPASS'
+ xpass=0
++ expr 1 + 0
+ failures=1
++ expr 2 - 0
+ all=2
+ test 2 -eq 1
+ tests=tests
+ All='All '
+ case fail=$fail:xpass=$xpass:xfail=$xfail in
+ msg='1 of 2 tests failed.  '
+ exit=false
+ test 0 -ne 0
+ sed 's/.*/   &   /;h;s/./=/g;p;x;p;g;p;s/.*//'
+ echo 'cpputils 0.001: tests/test-suite.log'
+ echo '1 of 2 tests failed.  '
+ echo
+ echo '.. contents:: :depth: 2'
+ echo
+ for f in '$list'
+ read line
+ case $line in
+ echo
+ cat test001.log
+ for f in '$list'
+ read line
+ case $line in
+ mv test-suite.log.tmp test-suite.log
+ test 1 -ne 0
+ msg='1 of 2 tests failed.  See tests/test-suite.log.  '
+ test -n Alejandro 'Aragon <address@hidden>'
/bin/sh: line 0: test: Alejandro: binary operator expected
+ test x = x
+ red=
+ grn=
+ lgn=
+ blu=
+ std=
+ test X '!=' Xno
+ test Xxterm-color '!=' Xdumb
+ test X = Xalways
+ test -t 1
+ red=''
+ grn=''
+ lgn=''
+ blu=''
+ std=''
+ false
+ echo '\c'
+ echo '1 of 2 tests failed.  See tests/test-suite.log.  '
+ gawk '{ n = split($0, lines, "\\.  "); max = 0; for (i = 1; i <= n; ++i)
if (max < length(lines[i])) max = length(lines[i]); for (i = 0; i < max;
++i) line = line "="; print line; for (i = 1; i <= n; ++i) if (lines[i])
print lines[i]; print line; }'
========================
1 of 2 tests failed
See tests/test-suite.log
========================
+ echo '\c'
+ false
make[3]: *** [test-suite.log] Error 1
make[2]: *** [check-TESTS] Error 2
make[1]: *** [check-am] Error 2
+ eval exit 1
++ exit 1
make: *** [check-recursive] Error 1
address@hidden/Documents/workspace/cpputils$



-- 
View this message in context: 
http://www.nabble.com/running-tests-under-in-a-tests--directory-tp23773808p23857695.html
Sent from the Gnu - Automake - General mailing list archive at Nabble.com.