bug#42883: Selecting a subset of tests in recursive make causes unnecessary failures

[Glenn Washburn]

Hi Karl,

On Wed, 19 Aug 2020 19:17:13 -0600
Karl Berry <karl@freefriends.org> wrote:

> Hi Glenn,
> 
>     I noticed that when specifying a subset of tests to run via the
> TESTS environment variable, I could have all tests complete
> successfully
> 
> Thanks for the report. That does seem suboptimal, but on the other
> hand, subsets of tests are a complication. Does it work to override
> SUBDIRS along with TESTS? As in (thanks to Jim):
>   env TESTS=grub_cmd_echo make -e check VERBOSE=yes SUBDIRS=

Yes this does work in the special case of grub, thanks for the
suggestion. I think at a minimum this behavior should be documented
with the test subsets documentation and this work around mentioned.
Ideally, the subset tests feature design should be updated to play nice
with makefile recursion.  Setting SUBDIRS= is just disabling the
makefile recusion.

If I had a test in grub-core/unittest1 and I wanted to run just the
test subset of unittest1 and grub_cmd_echo, how would I do that such
that make returns success?  If I run `TESTS="unittest1 grub_cmd_echo"
make -e check', make will fail even if both test individually succeed
because the recursive make check in grub-core will try to run
grub_cmd_echo (in addition to unittest1) and fail.

Another issue is that if there is a test X in the root makefile and a
test X in SUBDIR, how does one choose to run the X in SUBDIR and not the
X in root?  Right now TESTS=X will run both tests.  Sure one could do
`TESTS=X make -e -C SUBDIR check', but then again you're just bypassing
the makefile recursion.

And thanks Karl for helping to maintain this venerable and essential,
yet not super glamorous project.

Glenn