Re: script suggestion: 'check_program' to easily run multiple tests

[Bernhard Voelker]

On 08/02/2014 12:49 AM, Assaf Gordon wrote:
> Attached is a patch that adds a 'makefile' target to test a program.
> 
> Examples:
> 
>      make check-prog T=sort
>      make check-very-expensive-prog T=tail

Thanks for picking this up again.

> diff --git a/HACKING b/HACKING
> index 393a623..9acc3cf 100644
> --- a/HACKING
> +++ b/HACKING
> @@ -434,6 +434,22 @@ and moving into existing blocks.  This avoids making 
> unnecessary
>  work for translators.
>  
>  
> +Testing
> +========
> +To run all existing tests for a specific program (e.g. 'seq'), run:
> +
> +  make check-prog T=seq
> +
> +Similarly, to run expensive or very expensive tests, run:
> +
> +  make check-expensive-prog T=tail
> +  make check-very-expensive-prog T=sort
> +
> +T=[PROG] takes a basic regular expression, allowing:
> +
> +  make check-prog T='sha.*'
> +
> +
>  Add tests
>  ==========
>  Nearly every significant change must be accompanied by a test suite

IMO the "Testing" paragraph should go after "Add tests",
as it's logically after adding a test for the actual commit.
... or even after or go into
  'Run "make syntax-check", or even "make distcheck"'.

> diff --git a/Makefile.am b/Makefile.am
> index fb4af27..e238904 100644
> --- a/Makefile.am
> +++ b/Makefile.am
> @@ -85,6 +85,22 @@ check-expensive:
>  check-very-expensive:
>       $(MAKE) check-expensive RUN_VERY_EXPENSIVE_TESTS=yes
>  
> +# Shortcut to run all tests of a specific program
> +# Typical usage:
> +#     make check-prog PROG=sort
> +check-prog:
> +     $(AM_V_GEN)( test -n "$$T" || \
> +          { echo missing T=[program] parameter >&2 ; exit 1 ; } ; \
> +        TESTS=$$(find ./tests/ \( -name '*.sh' -o -name '*.pl' \) -print | \
> +          grep -w -- "$$T" | paste -s -d' ' -) ; \
> +        test -z "$$TESTS" && \
> +          { echo no tests found for program "$$T" >&2 ; exit 1 ; } ; \
> +        $(MAKE) check TESTS="$$TESTS" SUBDIRS=. VERBOSE=yes )

First of all, is there a specific reason why you added VERBOSE=yes, here?


I somehow have the impression that the approach using 'find | grep'
has some drawbacks ... and indeed:

* It doesn't find the one "*.xpl" test we have:
    tests/rm/fail-eperm.xpl
  See $(TEST_EXTENSIONS) in tests/local.mk.

* For T=factor, it would additionally run the helper program
    tests/factor/create-test.sh - which then fails.

* For T=factor it wouldn't find the $(factor_tests) if they
  don't already exist.

Therefore, I'd rather go with something based on $(built_programs).
What about the following?

diff --git a/Makefile.am b/Makefile.am
index fb4af27..2910083 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -85,6 +85,25 @@ check-expensive:
 check-very-expensive:
        $(MAKE) check-expensive RUN_VERY_EXPENSIVE_TESTS=yes

+# Shortcut to run all tests of a specific program
+# Typical usage:
+#     make check-prog PROG=sort
+check-prog:
+       $(AM_V_GEN)( test -n "$$T" || \
+           { echo "missing T=[program] parameter" >&2 ; exit 1 ; } ; \
+         case " $(built_programs) " in \
+           *" $$T "*) ;; \
+           *) { echo "program not built: '$$T'" >&2 ; exit 1 ; } ;; \
+         esac; \
+         tests=$$(grep -lw -- "$$T" $(TESTS) | paste -s -d' ' -) ; \
+         test -z "$$tests" && \
+           { echo no tests found for program "$$T" >&2 ; exit 1 ; } ; \
+         $(MAKE) check TESTS="$$tests" SUBDIRS=. )
+check-expensive-prog:
+       $(MAKE) check-prog RUN_EXPENSIVE_TESTS=yes
+check-very-expensive-prog:
+       $(MAKE) check-expensive-prog RUN_VERY_EXPENSIVE_TESTS=yes
+
 # Just prior to distribution, ...
 # transform the automake-generated rule that runs 'rm -f rm'.
 # On some systems, that command would fail with a diagnostic like

Note that check-prog would now not only run primary tests of
the given T=program (we don't have that information anywhere),
but all tests where 'program' is used.  I consider this is quite
useful, isn't it?  Admittedly, it's problematic with 'T=test'.

Have a nice day,
Berny