Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Stefan Hajnoczi]

On Fri, Mar 9, 2012 at 2:00 PM, Ademar Reis <address@hidden> wrote:
> On Fri, Mar 09, 2012 at 09:41:05AM +0000, Stefan Hajnoczi wrote:
>> On Thu, Mar 8, 2012 at 11:51 PM, Ademar Reis <address@hidden> wrote:
>> > On Thu, Mar 08, 2012 at 05:21:44PM -0600, Anthony Liguori wrote:
>> >> On 03/08/2012 04:24 PM, Ademar Reis wrote:
>> >> >On Thu, Mar 08, 2012 at 03:24:15PM -0600, Anthony Liguori wrote:
>> >> >>On 03/08/2012 03:02 PM, Ademar Reis wrote:
>> >> >>>On Thu, Mar 08, 2012 at 01:16:58PM -0600, Anthony Liguori wrote:
>> >> >>>>On 03/08/2012 11:59 AM, Ademar Reis wrote:
>> >> >>>   - QE will be alienated from the qemu test effort. There will be
>> >> >>>     no integration between the QE efforts and the maintenance of
>> >> >>>     the qemu developer-level tests.
>> >> >>
>> >> >>I think we're a pretty friendly and open community :-)  There is no
>> >> >>reason that QE should be "alienated" unless folks are choosing not
>> >> >>to participate upstream.
>> >> >
>> >> >For the exact same reasons you as a developer don't want to
>> >> >implement tests inside autotest, QE won't want to implement tests
>> >> >for qemu.git. It's out of their comfort zone, just put yourself
>> >> >on their shoes.
>> >>
>> >> This is a really, really poor argument and I hope I don't need to go
>> >> into details of why.  If the primary reason for libautotest is so
>> >> the people writing tests for QEMU can avoid actually working with
>> >> the developers of QEMU...  we've got a problem.
>> >
>> > No, one of the benefits of having libautotest is to *collaborate*
>> > with QE. I'll explain again:
>> >
>> > - As a qemu developer, I don't want to spend my time learning and
>> >  getting involved in autotest, which is a complex QE project
>> >  (I heard this numerous times).
>> >
>> > - As a Quality Engineer, I don't want to invest my time learning
>> >  and getting involved into upstream qemu to test HEAD.
>>
>> I think this is the key point of the whole discussion - most of the
>> other topics have been distractions.  Both communities do testing but
>> we test different things and have different priorities.
>>
>> For me this has been the big realization from this discussion.  I felt
>> kvm-autotest and qemu should share tests.  I was pushing for that but
>> after following this thread I don't think it makes sense, here's why:
>>
>> The Quality Engineer you describe is not a QEMU upstream QE, instead
>> the QE has a broader and more downstream focus.  (This is why
>> comparisons with WebKit or other upstream projects doing testing are
>> not valid comparisons.)
>
> Lucas, Cleber and the others red-hatters should remembers this
> from my internal presentation, it was the first point I made:
> QE and Developers have very different goals and interests.
>
> Which is why we're pushing all these changes in autotest. We see
> opportunities for collaboration, but we do realize the difference.
>
> And look: Lucas and Cleber are not QE, they're developers working
> on the autotest framework/library/whatever. We'll need similar
> positions inside qemu as the test infra-structure grows.

I don't understand this last paragraph.  If qemu.git upstream was
doing full-scale QE it would work fine because the differences that
I've described and you also have pointed out would be absent.

>> I think what's much more valuable is for qemu.git tests to be easily
>> hooked into autotest.  That way you get access to the testing that the
>> qemu community is doing for free.
>
> We get this by default in our plans: any application that can be
> run and returns 0/error can be run as a test inside autotest.

Yes, and I think that part of the plan makes sense.

> What we're asking is the *possibility* of a developer using
> libautotest when writting a complex test.
>
> - We're not asking everybody to use autotest
> - We're not saying autotest is better for everything
> - We're not requiring you to install autotest on your machine
>  (even though the process will be trivial)
>
> - We're asking that, if a developer is going to write a test
>  together with his patch to submit to qemu.git, he should be
>  allowed to use libautotest at his own discretion...
> - Ditto for using the test-runner: even if tests are simple
>  scripts, there will be benefits in using our optional
>  test-runner during development.
>
> This is how I see the interaction with autotest-based tests:
>
>    $ make check-full
>    tests.d/my-test-using-qemu-tests.sh -- PASS
>    tests.d/trivial-test.sh -- PASS
>    tests.d/my-test-using-libautotest -- SKIP
>    ...
>
>    $ yum install libautotest
>    # (or whatever the bootstrap procedure is)
>    $ make check-full
>    tests.d/my-test-using-qemu-tests.sh -- PASS
>    tests.d/trivial-test.sh -- PASS
>    tests.d/my-test-using-libautotest -- PASS
>    ...
>
> But for some reason that I still don't understand, the simple
> acceptance of the optional dependency of libautotest in qemu is
> seen as a bad thing.

I don't mind a libvirttest but I've expressed concerns that that if
qemu.git has it's own test library then the incentive to use or
contribute to an external library is not there.  That's why I
suggested a less ambitious approach.

Stefan