[Qemu-devel] [RFC][PATCH v6 00/04] qtest: qemu unit testing framework

[Michael Roth]

These patches apply to master (2-04-2011), and can also be obtained from:
git://repo.or.cz/qemu/mdroth.git qtest_v1

OVERVIEW:

QEMU currently lacks a standard means to do targeted unit testing of the device 
model. Frameworks like kvm-autotest interact via guest OS, which provide a 
highly abstracted interface to the underlying machine, and are susceptable to 
bugs in the guest OS itself. This allows for reasonable test coverage of guest 
functionality as a whole, but reduces the accuracy and specificity with which 
we can exercise paths in the underlying devices.

The following patches provide the basic beginnings of a test framework which 
replaces vcpu threads with test threads that interact with the underlying 
machine directly, allowing for directed unit/performance testing of individual 
devices. Test modules are built directly into the qemu binary, and each module 
provides the following interfaces:

init():
  Called in place of qemu's normal machine initialization to setup up devices 
explicitly. A full machine can be created here by calling into the normal init 
path, as well as minimal machines with a select set of buses/devices/IRQ 
handlers.

run():
  Test logic that interacts with the now-created machine.

cleanup():
  Currently unused, but potentially allows for chaining multiple tests 
together. Currently we run one module, then exit.

As mentioned these are very early starting points. We're mostly looking for 
input from the community on the basic approach and overall requirements for an 
acceptable framework. A basic RTC test module is provided as an example.

BUILD/EXAMPLE USAGE:

 $ ./configure --target-list=x86_64-softmmu --enable-qtest --enable-io-thread
 $ make
 $ ./x86_64-softmmu/qemu-system-x86_64 -test ?
 Available test modules:
 rtc
 $ ./x86_64-softmmu/qemu-system-x86_64 -test rtc
 ../qtest/qtest_rtc.c:test_drift():L94: hz: 2, duration_ms: 4999, exp_duration: 
5000, drift ratio: 0.000200
 ../qtest/qtest_rtc.c:test_drift():L111: hz: 1024, duration_ms: 4999, 
exp_duration: 5000, drift ratio: 0.000200

GENERAL PLAN:

 - Provide libraries for common operations like PCI device enumeration, APIC 
configuration, default-configured machine setup, interrupt handling, etc.
 - Develop tests as machine/target specific, potentially make some tests 
re-usable as interfaces are better defined
 - Do port i/o via cpu_in/cpu_out commands
 - Do guest memory access via a CPUPhysMemoryClient interface
 - Allow interrupts to be sent by writing to an FD, detection in test modules 
via select()/read()

TODO:

 - A means to propagate test returns values to main i/o thread
 - Better defined test harness for individual test cases and/or modules, likely 
via GLib
 - Support for multiple test threads in a single test module for scalability 
testing
 - Modify vl.c hooks so tests can configure their own timers/clocksources
 - More test modules, improve current rtc module
 - Further implementing/fleshing out of the overall plan

Comments/feedback are welcome!

 Makefile.objs     |    4 +-
 Makefile.target   |    1 +
 configure         |   20 ++++++
 module.h          |    5 +-
 qemu-options.hx   |    9 +++
 qtest/qtest-rtc.c |  170 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 qtest/qtest.c     |   65 ++++++++++++++++++++
 qtest/qtest.h     |   37 ++++++++++++
 roms/seabios      |    2 +-
 vl.c              |   35 +++++++++++
 10 files changed, 345 insertions(+), 3 deletions(-)