Re: script suggestion: 'check_program' to easily run multiple tests

[Pádraig Brady]

On 08/03/2014 10:57 PM, Bernhard Voelker wrote:
> On 08/02/2014 12:49 AM, Assaf Gordon wrote:
>> Attached is a patch that adds a 'makefile' target to test a program.
>>
>> Examples:
>>
>>      make check-prog T=sort
>>      make check-very-expensive-prog T=tail
> 
> Thanks for picking this up again.
> 
>> diff --git a/HACKING b/HACKING
>> index 393a623..9acc3cf 100644
>> --- a/HACKING
>> +++ b/HACKING
>> @@ -434,6 +434,22 @@ and moving into existing blocks.  This avoids making 
>> unnecessary
>>  work for translators.
>>  
>>  
>> +Testing
>> +========
>> +To run all existing tests for a specific program (e.g. 'seq'), run:
>> +
>> +  make check-prog T=seq
>> +
>> +Similarly, to run expensive or very expensive tests, run:
>> +
>> +  make check-expensive-prog T=tail
>> +  make check-very-expensive-prog T=sort
>> +
>> +T=[PROG] takes a basic regular expression, allowing:
>> +
>> +  make check-prog T='sha.*'
>> +
>> +
>>  Add tests
>>  ==========
>>  Nearly every significant change must be accompanied by a test suite
> 
> IMO the "Testing" paragraph should go after "Add tests",
> as it's logically after adding a test for the actual commit.
> ... or even after or go into
>   'Run "make syntax-check", or even "make distcheck"'.
> 
>> diff --git a/Makefile.am b/Makefile.am
>> index fb4af27..e238904 100644
>> --- a/Makefile.am
>> +++ b/Makefile.am
>> @@ -85,6 +85,22 @@ check-expensive:
>>  check-very-expensive:
>>      $(MAKE) check-expensive RUN_VERY_EXPENSIVE_TESTS=yes
>>  
>> +# Shortcut to run all tests of a specific program
>> +# Typical usage:
>> +#     make check-prog PROG=sort
>> +check-prog:
>> +    $(AM_V_GEN)( test -n "$$T" || \
>> +         { echo missing T=[program] parameter >&2 ; exit 1 ; } ; \
>> +       TESTS=$$(find ./tests/ \( -name '*.sh' -o -name '*.pl' \) -print | \
>> +         grep -w -- "$$T" | paste -s -d' ' -) ; \
>> +       test -z "$$TESTS" && \
>> +         { echo no tests found for program "$$T" >&2 ; exit 1 ; } ; \
>> +       $(MAKE) check TESTS="$$TESTS" SUBDIRS=. VERBOSE=yes )
> 
> First of all, is there a specific reason why you added VERBOSE=yes, here?
> 
> 
> I somehow have the impression that the approach using 'find | grep'
> has some drawbacks ... and indeed:
> 
> * It doesn't find the one "*.xpl" test we have:
>     tests/rm/fail-eperm.xpl
>   See $(TEST_EXTENSIONS) in tests/local.mk.
> 
> * For T=factor, it would additionally run the helper program
>     tests/factor/create-test.sh - which then fails.
> 
> * For T=factor it wouldn't find the $(factor_tests) if they
>   don't already exist.
> 
> Therefore, I'd rather go with something based on $(built_programs).
> What about the following?
> 
> diff --git a/Makefile.am b/Makefile.am
> index fb4af27..2910083 100644
> --- a/Makefile.am
> +++ b/Makefile.am
> @@ -85,6 +85,25 @@ check-expensive:
>  check-very-expensive:
>         $(MAKE) check-expensive RUN_VERY_EXPENSIVE_TESTS=yes
> 
> +# Shortcut to run all tests of a specific program
> +# Typical usage:
> +#     make check-prog PROG=sort
> +check-prog:
> +       $(AM_V_GEN)( test -n "$$T" || \
> +           { echo "missing T=[program] parameter" >&2 ; exit 1 ; } ; \
> +         case " $(built_programs) " in \
> +           *" $$T "*) ;; \
> +           *) { echo "program not built: '$$T'" >&2 ; exit 1 ; } ;; \
> +         esac; \
> +         tests=$$(grep -lw -- "$$T" $(TESTS) | paste -s -d' ' -) ; \
> +         test -z "$$tests" && \
> +           { echo no tests found for program "$$T" >&2 ; exit 1 ; } ; \
> +         $(MAKE) check TESTS="$$tests" SUBDIRS=. )
> +check-expensive-prog:
> +       $(MAKE) check-prog RUN_EXPENSIVE_TESTS=yes
> +check-very-expensive-prog:
> +       $(MAKE) check-expensive-prog RUN_VERY_EXPENSIVE_TESTS=yes
> +
>  # Just prior to distribution, ...
>  # transform the automake-generated rule that runs 'rm -f rm'.
>  # On some systems, that command would fail with a diagnostic like
> 
> Note that check-prog would now not only run primary tests of
> the given T=program (we don't have that information anywhere),
> but all tests where 'program' is used.  I consider this is quite
> useful, isn't it?  Admittedly, it's problematic with 'T=test'.

FWIW here's the --test bit of my ./make script,
which can test a program, directory or list of tests.
The program tests are identified from the print_ver_ tagging:

if [ "$1" = '--test' ]; then
  shift
  if test -d "$1"; then
    export tests_dir=$1
    tests="$(shopt -s nullglob; echo tests/$tests_dir/*.{sh,pl})"
    echo $tests
  elif ! printf '%s\n' "$1" | grep -qF .; then
    # No .suffix then look for program name
    prog="$1"
    [ "$prog" = 'install' ] && prog='ginstall'
    tests=$(find tests/ -name '*.sh' |
            xargs grep -El "print_ver_.* $prog" | paste -s -d' ')
    tests="$tests $(find tests/ -name "$prog*.pl" | paste -s -d' ')"
  else
    tests="$*"
  fi
  # SUBDIRS avoids gnulib tests
  make check TESTS="$tests" SUBDIRS=. VERBOSE=yes RUN_VERY_EXPENSIVE_TESTS=yes 
RUN_EXPENSIVE_TESTS=yes
  exit
fi