[RFC PATCH 0/4] Replace custom test harness with "meson test"

[Paolo Bonzini]

Hi all,

Starting with Meson 0.57, "meson test" has all features of QEMU's
makefile-based harness and more.  In particular, some features that
were added to reach feature parity are:

* print reproducer command line for each failed test right below the test

* keep the output of multiple (non-TAP) tests together in verbose mode,
  similar to "make --output-sync target"

* report on TAP subtests as they are encountered

It also includes nicer handling of test interruption, logging of the run
in the meson-logs/ subdirectory, and a progress report/spinner.  For
these reasons it would be nice to adopt it and remove the Perl scripts
that we have to present TAP output nicely.  While at it, this series
also changes qemu-iotests to be described in tests/qemu-iotests/meson.build
and it replaces the current "-makecheck" output style with TAP.

This is an RFC just to let people try it out and because I still haven't
sorted out timeouts (e.g.  OpenBSD is insanely slow on some tests).
But in theory, if you want to try it out, the only things you need to
know are:

* while "make check" does not timeout tests like it doesn't right now,
  "meson test" does respect timeouts, so you could get failures because
  of that.  That can and probably will change from RFC to final.

* CTRL+C will only interrupt the longest running test.  Pressing
  CTRL+C repeatedly three times (which you would likely do anyway,
  that's how things work) interrupts the whole run

* Right now "make check-block" only does a single test run just like
  "../tests/check-block.sh", but it would be possible to add the thorough
  suite to "meson test --suite block" as well.

* If you were using make check-report.tap and similar, they are replaced
  by targets like make check-report.junit.xml.  This is because Gitlab
  is able to parse the resulting XML and include on the website a report
  of which tests failed.

An example of non-verbose "make check" output is available at
https://gitlab.com/bonzini/qemu/-/jobs/1680980620.  A verbose run
instead is like https://asciinema.org/a/e5irnEszSnAheOHM30exbo3F6
(does not include check-block).

Paolo

Paolo Bonzini (4):
  build: use "meson test" as the test harness
  build: make check-block a meson test
  check-block: replace -makecheck with TAP output
  configure: remove dead EXESUF variable

 .gitlab-ci.d/crossbuild-template.yml |   2 +-
 Makefile                             |   3 +-
 configure                            |   3 -
 meson.build                          |   5 +-
 scripts/mtest2make.py                | 111 +++-----
 scripts/tap-driver.pl                | 379 ---------------------------
 scripts/tap-merge.pl                 | 111 --------
 tests/Makefile.include               |  15 +-
 tests/check-block.sh                 |  28 +-
 tests/fp/meson.build                 |   2 +-
 tests/meson.build                    |   1 +
 tests/qemu-iotests/check             |   6 +-
 tests/qemu-iotests/meson.build       |  14 +
 tests/qemu-iotests/testenv.py        |  30 +--
 tests/qemu-iotests/testrunner.py     |  48 ++--
 15 files changed, 121 insertions(+), 637 deletions(-)
 delete mode 100755 scripts/tap-driver.pl
 delete mode 100755 scripts/tap-merge.pl
 create mode 100644 tests/qemu-iotests/meson.build

-- 
2.31.1