Re: [Qemu-devel] [RFC][PATCH v6 00/04] qtest: qemu unit testing framework

[Blue Swirl]

On Fri, Feb 4, 2011 at 3:49 PM, Michael Roth <address@hidden> wrote:
> These patches apply to master (2-04-2011), and can also be obtained from:
> git://repo.or.cz/qemu/mdroth.git qtest_v1
>
> OVERVIEW:
>
> QEMU currently lacks a standard means to do targeted unit testing of the 
> device model. Frameworks like kvm-autotest interact via guest OS, which 
> provide a highly abstracted interface to the underlying machine, and are 
> susceptable to bugs in the guest OS itself. This allows for reasonable test 
> coverage of guest functionality as a whole, but reduces the accuracy and 
> specificity with which we can exercise paths in the underlying devices.
>
> The following patches provide the basic beginnings of a test framework which 
> replaces vcpu threads with test threads that interact with the underlying 
> machine directly, allowing for directed unit/performance testing of 
> individual devices. Test modules are built directly into the qemu binary, and 
> each module provides the following interfaces:
>
> init():
>  Called in place of qemu's normal machine initialization to setup up devices 
> explicitly. A full machine can be created here by calling into the normal 
> init path, as well as minimal machines with a select set of buses/devices/IRQ 
> handlers.
>
> run():
>  Test logic that interacts with the now-created machine.
>
> cleanup():
>  Currently unused, but potentially allows for chaining multiple tests 
> together. Currently we run one module, then exit.
>
> As mentioned these are very early starting points. We're mostly looking for 
> input from the community on the basic approach and overall requirements for 
> an acceptable framework. A basic RTC test module is provided as an example.
>
> BUILD/EXAMPLE USAGE:
>
>  $ ./configure --target-list=x86_64-softmmu --enable-qtest --enable-io-thread
>  $ make
>  $ ./x86_64-softmmu/qemu-system-x86_64 -test ?
>  Available test modules:
>  rtc
>  $ ./x86_64-softmmu/qemu-system-x86_64 -test rtc
>  ../qtest/qtest_rtc.c:test_drift():L94: hz: 2, duration_ms: 4999, 
> exp_duration: 5000, drift ratio: 0.000200
>  ../qtest/qtest_rtc.c:test_drift():L111: hz: 1024, duration_ms: 4999, 
> exp_duration: 5000, drift ratio: 0.000200
>
> GENERAL PLAN:
>
>  - Provide libraries for common operations like PCI device enumeration, APIC 
> configuration, default-configured machine setup, interrupt handling, etc.
>  - Develop tests as machine/target specific, potentially make some tests 
> re-usable as interfaces are better defined
>  - Do port i/o via cpu_in/cpu_out commands
>  - Do guest memory access via a CPUPhysMemoryClient interface
>  - Allow interrupts to be sent by writing to an FD, detection in test modules 
> via select()/read()
>
> TODO:
>
>  - A means to propagate test returns values to main i/o thread
>  - Better defined test harness for individual test cases and/or modules, 
> likely via GLib
>  - Support for multiple test threads in a single test module for scalability 
> testing
>  - Modify vl.c hooks so tests can configure their own timers/clocksources
>  - More test modules, improve current rtc module
>  - Further implementing/fleshing out of the overall plan
>
> Comments/feedback are welcome!

Would it be possible to couple this with the tracing or Kemari somehow
so that you could capture, say, block device traces and feed them to
test setup?