qemu-devel
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization test


From: Ademar Reis
Subject: Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests
Date: Fri, 9 Mar 2012 11:00:04 -0300
User-agent: Mutt/1.5.21 (2010-09-15)

On Fri, Mar 09, 2012 at 09:41:05AM +0000, Stefan Hajnoczi wrote:
> On Thu, Mar 8, 2012 at 11:51 PM, Ademar Reis <address@hidden> wrote:
> > On Thu, Mar 08, 2012 at 05:21:44PM -0600, Anthony Liguori wrote:
> >> On 03/08/2012 04:24 PM, Ademar Reis wrote:
> >> >On Thu, Mar 08, 2012 at 03:24:15PM -0600, Anthony Liguori wrote:
> >> >>On 03/08/2012 03:02 PM, Ademar Reis wrote:
> >> >>>On Thu, Mar 08, 2012 at 01:16:58PM -0600, Anthony Liguori wrote:
> >> >>>>On 03/08/2012 11:59 AM, Ademar Reis wrote:
> >> >>>   - QE will be alienated from the qemu test effort. There will be
> >> >>>     no integration between the QE efforts and the maintenance of
> >> >>>     the qemu developer-level tests.
> >> >>
> >> >>I think we're a pretty friendly and open community :-)  There is no
> >> >>reason that QE should be "alienated" unless folks are choosing not
> >> >>to participate upstream.
> >> >
> >> >For the exact same reasons you as a developer don't want to
> >> >implement tests inside autotest, QE won't want to implement tests
> >> >for qemu.git. It's out of their comfort zone, just put yourself
> >> >on their shoes.
> >>
> >> This is a really, really poor argument and I hope I don't need to go
> >> into details of why.  If the primary reason for libautotest is so
> >> the people writing tests for QEMU can avoid actually working with
> >> the developers of QEMU...  we've got a problem.
> >
> > No, one of the benefits of having libautotest is to *collaborate*
> > with QE. I'll explain again:
> >
> > - As a qemu developer, I don't want to spend my time learning and
> >  getting involved in autotest, which is a complex QE project
> >  (I heard this numerous times).
> >
> > - As a Quality Engineer, I don't want to invest my time learning
> >  and getting involved into upstream qemu to test HEAD.
> 
> I think this is the key point of the whole discussion - most of the
> other topics have been distractions.  Both communities do testing but
> we test different things and have different priorities.
> 
> For me this has been the big realization from this discussion.  I felt
> kvm-autotest and qemu should share tests.  I was pushing for that but
> after following this thread I don't think it makes sense, here's why:
> 
> The Quality Engineer you describe is not a QEMU upstream QE, instead
> the QE has a broader and more downstream focus.  (This is why
> comparisons with WebKit or other upstream projects doing testing are
> not valid comparisons.)

Lucas, Cleber and the others red-hatters should remembers this
from my internal presentation, it was the first point I made:
QE and Developers have very different goals and interests.

Which is why we're pushing all these changes in autotest. We see
opportunities for collaboration, but we do realize the difference.

And look: Lucas and Cleber are not QE, they're developers working
on the autotest framework/library/whatever. We'll need similar
positions inside qemu as the test infra-structure grows.

> 
> There is not enough in common between upstream QEMU testing and
> downstream KVM QE to make convergence a win-win.  libautotest sounds
> like a technical solution to a people problem - the problem is that we
> have different priorities.  We overlap but at the end of the day we do
> different things.  We can make a best effort to converge but I don't
> see incentives that will make this a success.  Creating an abstraction
> library will be sub-optimal for both communities.

This is the part I don't agree, but in the end it's a matter of opinion.

> 
> I think what's much more valuable is for qemu.git tests to be easily
> hooked into autotest.  That way you get access to the testing that the
> qemu community is doing for free.

We get this by default in our plans: any application that can be
run and returns 0/error can be run as a test inside autotest.

What we're asking is the *possibility* of a developer using
libautotest when writting a complex test.

- We're not asking everybody to use autotest
- We're not saying autotest is better for everything
- We're not requiring you to install autotest on your machine
  (even though the process will be trivial)

- We're asking that, if a developer is going to write a test
  together with his patch to submit to qemu.git, he should be
  allowed to use libautotest at his own discretion...
- Ditto for using the test-runner: even if tests are simple
  scripts, there will be benefits in using our optional
  test-runner during development.

This is how I see the interaction with autotest-based tests:

    $ make check-full
    tests.d/my-test-using-qemu-tests.sh -- PASS
    tests.d/trivial-test.sh -- PASS
    tests.d/my-test-using-libautotest -- SKIP
    ...

    $ yum install libautotest
    # (or whatever the bootstrap procedure is)
    $ make check-full
    tests.d/my-test-using-qemu-tests.sh -- PASS
    tests.d/trivial-test.sh -- PASS
    tests.d/my-test-using-libautotest -- PASS
    ...

But for some reason that I still don't understand, the simple
acceptance of the optional dependency of libautotest in qemu is
seen as a bad thing.

If the ultimate requirement now is "we don't want to use an
external library, we don't want to maintain one ourselves and we
don't want tests to cover things out of qemu", then clearly
there's no space for autotest in qemu.git and we can move on.

For developers of libvirt, spice, ovirt and any other project up
in the stack, writting code similar to what's part of qemu-test
makes no sense at all and libautotest is a real need. We would
like qemu to embrace autotest as well thus providing
cross-project developers a similar experience and sharing the
maintenance burden, but that might be too ambitious.

Anyway, we're at the RFC stage, and this discussion has been
valuable. If for nothing else, to validate that there are no
significant technical requirements we're missing, which makes me
happy.

We have a lot of work to do and hopefully the value of autotest
to qemu will be seen as patches keep flowing and other developers
use it. In the next kvm-forum we can discuss this again in
person, maybe with some beers around us. :-)

> 
> And on the flip-side it would be awesome for kvm-autotest to cover
> upstream.  As an example, QEMU uses buildbot with a public web
> interface which shows the history of all runs and failure
> notifications are sent to qemu-devel (it's not perfect but I think it
> adds value to the community).  Can you can do daily/weekly qemu.git
> upstream kvm-autotest runs, make the results easily accessible on the
> public web, and perhaps hook into the qemu-devel mailing list?

That's also in the pipeline, but we might not have resources
anytime soon.

> That level of collaboration would allow both communities to do what
> they want to do effectively, while still getting benefits from each
> other.

Cheers,
  - Ademar

-- 
Ademar de Souza Reis Jr.
Red Hat

^[:wq!



reply via email to

[Prev in Thread] Current Thread [Next in Thread]