bug#42883: Selecting a subset of tests in recursive make causes unnecessary failures

[Glenn Washburn]

On Fri, 21 Aug 2020 16:05:24 -0600
Karl Berry <karl@freefriends.org> wrote:

>> Another issue is that if there is a test X in the root makefile
>> and a test X in SUBDIR
> 
> Isn't that fundamentally not supported, or supportable? I admit I
> don't actually know if there's any history about it, but using the
> same name for two tests in different places just seems like a basic
> source of confusion. As far as I know, the TESTS mechanism just isn't
> set up to be able to specify arbitrary test names in arbitrary
> directories.

I'm not sure if I was sufficiently clear in my example. The example is
meant to be such that the test X in root and SUBDIR are tests that
would be run when not explicitly specifying tests.  So its to be able
to run any test that would be run when not explicitly specifying tests.

I think the way to support this is to have TESTS be a list of test
paths relative to the root makefile.  So in the example above, I would
specify TESTS="X SUBDIR/X".  Before running the recursive make in
SUBDIR, the TESTS variable would be filtered to only include tests with
a relative path of the sub directory.  Also, the root makefile would
not run tests that have a directory component to the path.

I would like to get a rough idea of if this would be an approved of
approach so that I or someone else could take a stab at implementing it
with the hopes of the changes being accepted.

Glenn