Re: script suggestion: 'check_program' to easily run multiple tests

[Assaf Gordon]

Hello,

On 08/03/2014 05:57 PM, Bernhard Voelker wrote:
> On 08/02/2014 12:49 AM, Assaf Gordon wrote:
> > Attached is a patch that adds a 'makefile' target to test a program.
> >
> > Examples:
> >
> >       make check-prog T=sort

> > diff --git a/HACKING b/HACKING
> > index 393a623..9acc3cf 100644
> > --- a/HACKING
> > +++ b/HACKING

<...>

> IMO the "Testing" paragraph should go after "Add tests",
> as it's logically after adding a test for the actual commit.
> ... or even after or go into
>    'Run "make syntax-check", or even "make distcheck"'.

Ok, I'll relocate it.
 
> > diff --git a/Makefile.am b/Makefile.am
> > index fb4af27..e238904 100644
> > --- a/Makefile.am
> > +++ b/Makefile.am
> > @@ -85,6 +85,22 @@ check-expensive:
> >   check-very-expensive:
> >         $(MAKE) check-expensive RUN_VERY_EXPENSIVE_TESTS=yes
> >
> > +# Shortcut to run all tests of a specific program
> > +# Typical usage:
> > +#     make check-prog PROG=sort
> > +check-prog:
> > +       $(AM_V_GEN)( test -n "$$T" || \
> > +            { echo missing T=[program] parameter >&2 ; exit 1 ; } ; \
> > +          TESTS=$$(find ./tests/ \( -name '*.sh' -o -name '*.pl' \) -print | \
> > +            grep -w -- "$$T" | paste -s -d' ' -) ; \
> > +          test -z "$$TESTS" && \
> > +            { echo no tests found for program "$$T" >&2 ; exit 1 ; } ; \
> > +          $(MAKE) check TESTS="$$TESTS" SUBDIRS=. VERBOSE=yes )
>
> First of all, is there a specific reason why you added VERBOSE=yes, here?

Yes,
The purpose of this target is to ease testing for developers.
Especially when modifying a program, and wanting to quickly verify it against 
all existing tests.
It seems like others on this list have their own version of a similar 'hack'.

For this use case, I'd anyhow add "VERBOSE=yes" if I typed "make check 
TESTS=XXXX", so I've just added it here.

I actually wanted to add the flag to save the temporary files too, but didn't 
include it yet in the patch.


<...>

> Therefore, I'd rather go with something based on $(built_programs).
> What about the following?
 
<..>

> Note that check-prog would now not only run primary tests of
> the given T=program (we don't have that information anywhere),
> but all tests where 'program' is used.  I consider this is quite
> useful, isn't it?  Admittedly, it's problematic with 'T=test'.

Looks very useful (I only read it, haven't tried it yet).

I'll test it on few systems and re-post.

Thanks!
-Assaf