bug#49309: Feature Request: Automake script based tests to print the test name before running it

[Kasper k]

Hi Peter,

Thanks for your reply.

As to the workarounds, they are just that and I had coined my own for "my machine" ™. The standard solution for "display test name before starting it" (controlled by a new AC option) would be much more palatable for consumer to add to their Makefile.am and check into their source tree.  Then if some intermittent bad situation happens in unattended/disposable CI environment, as an example, then we won't be needing to opt into any kind of workaround, it'd be just there right in the "regular" make-check logs; without introducing crazy amount of verbosity (which set -x, VERBOSE=1 etc. renders).

/K

---

From: Peter Johansson <trojkan@gmail.com>
Sent: Thursday, July 1, 2021 1:30 PM
To: Kasper k <kasperkantz@outlook.com>; 49309@debbugs.gnu.org <49309@debbugs.gnu.org>
Subject: Re: bug#49309: Feature Request: Automake script based tests to print the test name before running it

 

Hi Kasper,

I leave to the maintainers to comment on the suggestion, but in the
meantime...

On 1/7/21 1:01 pm, Kasper k wrote:
> Hello automake devs,
>
> In script based testsuites (https://www.gnu.org/software/automake/manual/html_node/Scripts_002dbased-Testsuites.html#Scripts_002dbased-Testsuites), when we run `make check` it prints one of the following string for each test: PASS <testname>, FAIL <testname>, XFAIL <testname> or SKIP <testname>.
>
> However, in some cases (long running test, flaky test, other inconclusive ones for more involved reasons), it is essential to know which test is currently in flight. Especially in CI systems, where flakiness every now and then, due to external factors, is almost unavoidable.
>
> It would be very helpful if a new option AM_TESTS_NAME=1 is provided, which would change the stdout output from:
>
>      <blank stdout and wait for test execution to complete>
>       PASS (or FAIL/XFAIL/SKIP) $name
>
> to something like:
>
>      RUNNING $name
>      <blank stdout and wait for test execution to complete>
>       PASS (or FAIL/XFAIL/SKIP) $name

Please note that that output comes from the test-driver script. If
you're not happy with the default test-driver, you can declare to use
your own my-test-driver (see
https://www.gnu.org/software/automake/manual/automake.html#Declaring-Custom-Test-Drivers),
and for example this could be a modified version of 'test-driver' script
provided by Automake. Adding a line outputting 'Running $name' is
trivial if you know any shell, or otherwise, I'm sure people here can help.

>
> this way we can easily find out the name of test (or multiple of them; in case of parallel execution) which is (are) currently running.

I usually find out which tests are running by looking at the timestamps
of log and trace files with something like 'ls -lrt *.log *.trs'.

Cheers,

Peter