Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Ademar Reis]

On Thu, Mar 08, 2012 at 01:16:58PM -0600, Anthony Liguori wrote:
> On 03/08/2012 11:59 AM, Ademar Reis wrote:

<snip>

> 
> >>I expect QEMU to grow tests for anything that involves launching
> >>QEMU directly.  Where I would not see QEMU growing tests for is
> >>things like launching QEMU through libvirt.
> >
> >Agree. For QEMU developers, libvirt should not be on the way, the
> >interaction should be minimal or non-existent.
> >
> >That's an area which will require some work in libautotest,
> >because due to previous QE requirements, it now invokes libvirt
> >methods instead of QEMU directly for a lot of stuff. But it can
> >be done and is part of the plan.
> 
> If you're talking about libautotest as an abstraction layer for
> launching QEMU, I don't think that's something we should use in
> upstream QEMU.  Abstraction layers are okay when the things you are
> abstracting are well defined and relatively static.  We're talking
> about testing the tip of a development tree though and coordinating
> changes with an abstraction layer would be counter productive.
>

Valid point. If I'm a qemu developer and I want to make huge
changes, ideally I shouldn't have to submit patches to
libautotest because the tests in qemu.org will break.

I don't know how likely such scenario is (Lucas is maitaining
kvm-autotest for years now and the integration hardly breaks)
But we do have a conflict of interests, so lets evaluate a
few options:

My main interests are:
    - remove barriers for developers to write tests
    - share the testing effort between multiple projects (qemu,
      libvirt, ovirt, spice, QE, etc...)

Now the options:

1. We keep the invocation stuff in libautotest and when writting
a complex test in qemu.git, the test writter can choose to use it
because of the goodies there.

Pros:
  - Lots of codepaths implemented both in python and as cmd-line
    utilities: less lines of code to write a test, smaller
    barrier for the developer.
  - Mature code in the test library.
  - Goodies such as video-recording and system info collection
    in a standard way for different projects.
  - Code shared with other teams.
  - The same test code can run on old platforms because
    libautotest has a compatibility layer, thus enabling a
    larger grid of tests.
  - QE teams will be closer to the development, sharing
    some of their testing code and maintenance burden.
  - It can co-exist with qemu-test.

Cons:
  - You depend on an external library for some of your tests (in
    a way, a circular dependency)

2. We "move" (or implement) all the libautotest stuff related to
qemu invocation and interaction into qemu.git, forbidding the
dependency of autotest in qemu.git.

In a way, that's what tends to happen with qemu-test if things
keep going on AS IS in qemu.org (I'm absolutely sure that you'll
replicate a lot of what autotest does as you increase the test
coverage and complexity).

Pros:
  - Everything under qemu.git control: you break it, you fix it.
  - Organic growth of the test infra-structure in QEMU

Cons:
  - Lot of code will be duplicated to cover the main code paths:
    writting tests will require writting/supporting considerable
    ammount of code (that already exists in autotest).
  - QE will be alienated from the qemu test effort. There will be
    no integration between the QE efforts and the maintenance of
    the qemu developer-level tests.
  - You don't get the goodies from autotest (standardized system
    info collection, video recording, grid support, etc).
  - The new tests will work only on the master branch.


3. A mix of (1) and (2): we "move" everything under qemu.git, but
keep the compatibility layer and provide a "libqemutest" for
third-parties.

Anybody writting a test that interacts with qemu will use this
library: qemu, libvirt, spice, ovirt, QE tests, etc.

There's current code written in python (which can be encapsulated
into cmd line utilities), but the library can provide stuff in
shell, perl, whatever. You just have to provide an API for
third-parties.

To avoid the burden of the qemu-devel list for collaborators, a
maintainer collects the contributions from third-parties.

Pros:
  - Everything under qemu.git control.
  - API available for upper layers in the stack (libvirt, QE,
    autotest, ovirt, spice, etc), keeping an unified effort on
    testing.
  - Reuse the current code from autotest (refactorings are
    planned anyway).

Cons:
  - A lot of work to be done by qemu developers.
  - A lot of work to be done by autotest: third-parties will keep
    using autotest instead of libqemutest for a while.

(1) requires work only on autotest by now... you'll wait and see
how far we can go (just raise your concerns now if you plan to
block the autotest dependency in qemu.git). In parallel, you can
keep the work on qemu-test and let the decision up to the test
writter.

(2) is the status quo of qemu.org

(3) could start with a patch submission from the autotest
developers to qemu-devel in the near future. It's not much
different than (1).

-- 
Ademar de Souza Reis Jr.
Red Hat

^[:wq!