Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Cleber Rosa]

On 03/08/2012 08:54 AM, Stefan Hajnoczi wrote:
> On Thu, Mar 8, 2012 at 11:44 AM, Stefan Hajnoczi<address@hidden>  wrote:
> > On Thu, Mar 8, 2012 at 4:00 AM, Lucas Meneghel Rodrigues<address@hidden>  wrote:
> > > One of our main goals is to provide useful tools for the qemu community,
> > > since we have a good number of tests and libraries written to perform
> > > integration/QA testing for that tool, being successfuly used by a number of
> > > QA teams that work on qemu. Also, we recently provided a subset of that
> > > infrastructure to test libvirt, one of our virtualization projects of
> > > interest.
> > Thanks for sharing.
> One thing I should have added is that my message is about what it
> would take for me to use autotest and contribute tests.  But now I
> realize that you might be going for a different model:
>
> If you're aiming for a different model where autotest integrates
> external test suites (i.e. tests wouldn't be written in autotest.git,
> instead autotest.git would contain snapshots of external test suites),
> then this proposal seems fine.  Upstream projects like QEMU would
> develop their own test suite and it would be dropped into autotest or
> a specific autotest instance.
>
> I'm not 100% sure which of these two models you're going for?

Autotest will continue to integrate and ship with external test suites,
even though that's an option at packaging time.

But the point here is that we also want to cover the other use cases,
which includes being able to run tests that are hosted within external
projects, such as QEMU itself. The idea is to put two things in a state
that's easier to be consumed by individual developers:

  * The test runner
  * The (optional) autotest API

So, by making the autotest API optional, and even the language of the
test script your own choice, you can keep you current test code, using
your own mini-framework and still use the autotest test runner for running
the tests and gathering the results and important system information.

By improving the API, which basically means making it more visible, better
organized and documented, we hope that users writing instrumented
tests (using serial or ssh sessions, sending either HMP or QMP monitor
commands, etc) will choose to use it.

CR.

> Stefan