[Qemu-devel] Improving QMP test coverage

[Markus Armbruster]

I've seen enough QMP-related bugs that should've been caught in basic
testing to know we need to become better at testing this stuff.  That
means investing into tests/.

QMP is now almost eight years old, but its test coverage has always been
spotty.  We don't even really know where the holes are.

The tried-and-true approach to get such a problem under control is to
require new code to come with tests, and buckle down to write the
missing tests for old code.  This is more difficult for QMP than for
other things, because QMP is in so many places.  The QMP core is a
proper subsystem[*], but many of the 150+ commands are parts of other
subsystems.

Let's review tests we already have.

Two unit tests have been around since "forever": test-qmp-commands.c
exercises QMP command dispatch, and test-qmp-event.c exercises QMP event
sending.  The former also has a few tests of qapi_free_FOO(), but they
should really be somewhere else.  Perhaps they can even be deleted.

We additionally have quite a few tests that play with a real QEMU's QMP
monitor, using libqtest or qtest.py.

Many of them use QMP to test something else; they are not necessarily
comprehensive tests for the QMP commands they use.  For instance,
numa-test.c uses QMP command query-cpus to check command line options
have the expected effect.  It's not a comprehensive test of query-cpus.
qemu-iotests/030 uses QMP to exercise image streaming.  Uses QMP command
block-stream a lot, but whether it is a comprehensive test for it I
can't say without careful review of both the command and the test.

What do I mean by "comprehensive"?  A comprehensive test exercises the
command systematically, including its failure modes (testing how things
fail is important, because that's where we often screw up).

There's just one test dedicated to QMP: qmp-test.c.  It's quite new, and
doesn't do much, yet.  It mostly checks QMP protocol implementation,
i.e. aspects of QMP that are independent of specific commands.  For
instance, it tests QEMU cleanly rejects a command object with an
"arguments" member that isn't an object.  It also tests query-version.
It can serve as a relatively uncluttered example of how to test QMP
commands.

What can we do to improve QMP testing?  Sadly, I don't have the master
plan ready.  I can tell people their new code needs to come with tests,
but that won't help much unless subsystem maintainers pick up the habit,
too.  There are a few obvious tests to write for old code, such as a
generic test of query-like commands without arguments and without side
effects, similar to what test-hmp.c does for HMP command info (I hope to
get around to that one).  But for much of the old code, we don't even
know where the test coverage holes are.

Ideas anyone?


[*] Or could be, if we repaid technical debt in the monitor.