Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Ademar Reis]

On Thu, Mar 08, 2012 at 05:21:44PM -0600, Anthony Liguori wrote:
> On 03/08/2012 04:24 PM, Ademar Reis wrote:
> >On Thu, Mar 08, 2012 at 03:24:15PM -0600, Anthony Liguori wrote:
> >>On 03/08/2012 03:02 PM, Ademar Reis wrote:
> >>>On Thu, Mar 08, 2012 at 01:16:58PM -0600, Anthony Liguori wrote:
> >>>>On 03/08/2012 11:59 AM, Ademar Reis wrote:
> >>>
> >>><snip>
> >>>
> >>>>
> >>>>>>I expect QEMU to grow tests for anything that involves launching
> >>>>>>QEMU directly.  Where I would not see QEMU growing tests for is
> >>>>>>things like launching QEMU through libvirt.
> >>>>>
> >>>>>Agree. For QEMU developers, libvirt should not be on the way, the
> >>>>>interaction should be minimal or non-existent.
> >>>>>
> >>>>>That's an area which will require some work in libautotest,
> >>>>>because due to previous QE requirements, it now invokes libvirt
> >>>>>methods instead of QEMU directly for a lot of stuff. But it can
> >>>>>be done and is part of the plan.
> >>>>
> >>>>If you're talking about libautotest as an abstraction layer for
> >>>>launching QEMU, I don't think that's something we should use in
> >>>>upstream QEMU.  Abstraction layers are okay when the things you are
> >>>>abstracting are well defined and relatively static.  We're talking
> >>>>about testing the tip of a development tree though and coordinating
> >>>>changes with an abstraction layer would be counter productive.
> >>>>
> >>>
> >>>Valid point. If I'm a qemu developer and I want to make huge
> >>>changes, ideally I shouldn't have to submit patches to
> >>>libautotest because the tests in qemu.org will break.
> >>>
> >>>I don't know how likely such scenario is (Lucas is maitaining
> >>>kvm-autotest for years now and the integration hardly breaks)
> >>>But we do have a conflict of interests, so lets evaluate a
> >>>few options:
> >>
> >>Because kvm-autotest doesn't have thousands of tests and isn't part
> >>of 200+ developers fast paths :-)
> >>
> >>>
> >>>My main interests are:
> >>>     - remove barriers for developers to write tests
> >>
> >>This is my number one priority.
> >>
> >>>     - share the testing effort between multiple projects (qemu,
> >>>       libvirt, ovirt, spice, QE, etc...)
> >>
> >>This is my number 99 priority.
> >
> >You have the right to consider the interests of qemu only, but
> >qemu is used by several other teams who depend on it to implement
> >their own tests. Some collaboration would be extremely beneficial
> >for everybody.
> >
> >Collaboration goes both ways: you get benefits from the
> >interaction and code reuse with different projects and teams, but
> >it may require a few tradeoffs.
> 
> I think the problem here effectively boils down to "code sharing at any cost".
> 

I may agree with the "sharing at any cost" if you reference parts
of the discussion from December. But if you re-read that thread
and compare with what we're proposing now, you'll see that we're
proposing something very different, based on the requirements
previously raised.

> Let's look at it in purely practical terms.  How many lines of code
> would be saved here and what does that code do?
> 
> If the only sharing is some infrastructure bits in libautotest, then
> I suspect we're talking in the low 10ks of lines at the absolute
> maximum (and I don't think it's even close to that in reality).
> Launching QEMU and connecting to QMP isn't that hard.  It's in the
> hundreds of lines of code.  What else is there that's need to write
> a test?
> 
> I don't think this is sufficiently compelling to add an
> unconditional autotest dependency to QEMU.
> 
> >>I think you could achieve practical code sharing by decomposing
> >>kvm-autotest into a set of useful, independent tools.  For
> >>instance....
> >>
> >>>Now the options:
> >>>
> >>>1. We keep the invocation stuff in libautotest and when writting
> >>>a complex test in qemu.git, the test writter can choose to use it
> >>>because of the goodies there.
> >>>
> >>>Pros:
> >>>   - Lots of codepaths implemented both in python and as cmd-line
> >>>     utilities: less lines of code to write a test, smaller
> >>>     barrier for the developer.
> >>
> >>I've got an existence proof that this is not true.  The qemu-test
> >>tests are smaller than the corresponding autotest tests.
> >
> >You're comparing developer-level tests with the existent QA-level
> >tests (much more complex).
> 
> Let's be specific then.  Look at device-add.sh in qemu-test.  It's
> 71LOC. pci_hotplug.py in autotest is 204LOC.
> 
> What's the "much more complex" part of pci_hotplug.py that is
> enabled by having the autotest framework?
> 
> >We're talking about an approach where you'll need just a few
> >lines of code to instantiate a VM and interact with it.
> 
> This isn't that hard of a problem.  It just takes a few lines to
> launch QEMU in qemu-test too.

Lucas or Cleber: could you list everything that is involved in
launching a VM? Which kind of goodies we get from autotest, even
a very lean (and fast) future version of it?

> 
> >>>   - Mature code in the test library.
> >>>   - Goodies such as video-recording and system info collection
> >>>     in a standard way for different projects.
> >>
> >>Why does video-recording have to be part of libautotest?  Why can't
> >>you just have a simply utility that connects to a VNC session and
> >>records it?  Presumably that's all it's doing here.
> >>
> >>Likewise, there are dozens of "system info collection" such as
> >>sosreport..  why fold this all into libautotest?
> >
> >For standardization: think outside of the qemu community. A lot
> >of our developers contribute code to multiple projects. It gets
> >easier if there's a standard way to achieve the same results when
> >writting tests for different projects.
> >
> >Anyway, it's a minor issue.
> 
> Yeah, I think this is outside of the scope of the current discussion
> as it's sort of like trying to boil the ocean :-)
> 
> >>>   - Code shared with other teams.
> >>>   - The same test code can run on old platforms because
> >>>     libautotest has a compatibility layer, thus enabling a
> >>>     larger grid of tests.
> >>
> >>This is not a requirement from a QEMU perspective nor do I think it's 
> >>useful.
> >
> >Well, it's not a requirement, but you can't expect QE teams to
> >spend their time on tests that work only on HEAD. It makes no
> >sense from their point of view.
> 
> They're certainly testing some sort of HEAD, even if it's not
> qemu.git.  I don't see any conflicting goals here.  Write a test
> against upstream, backport to a downstream if necessarily.

Exactly: if you use a powerfull enough test framework/library
that allows they to extend the test. Otherwise they'll just write
their own version using autotest, which is far more powerfull.

> 
> >
> >>
> >>>   - QE teams will be closer to the development, sharing
> >>>     some of their testing code and maintenance burden.
> >>
> >>The QE team could simply contribute tests to qemu.git to achieve this same 
> >>goal.
> >
> >They won't help you if you don't help them. Again, it doesn't
> >make sense for QE to work on HEAD. At least not the QE teams I'm
> >talking about.
> >
> >And I say from my own experience: maintaining a set of thousands
> >of tests is far from trivial. You'll waste a lot of potential
> >contributions by alienating the fully-dedicated teams currently
> >working on QE.
> 
> And trying to maintain a thousand tests against any possible version
> is probably also unrealistic.

That's why there's a large team of QEs fully allocated to it. And
what you described is exactly their job description.

> 
> Is the expectation that all of the tests in WebKit work against any
> possible version of WebKit?  How do they enforce this since it's all
> in trunk?

Webkit reached a TDD level where there's no need to support
multiple releases (think of Google Chrome). Each port does its
own QE, most of the time in trunk.

And a lot of engineers are fully dedicated to keep the testing
machinery running. It's very challenging (even though it's far
easier to test a webengine than to test an emulator/hypervisor).

In other words, there's no separation between QE tests and
developer-level tests in WebKit. But I don't expect qemu to get
there anytime soon.

> 
> >>>2. We "move" (or implement) all the libautotest stuff related to
> >>>qemu invocation and interaction into qemu.git, forbidding the
> >>>dependency of autotest in qemu.git.
> >>
> >>Let me just be really clear here.  In order to do TDD, tests have to
> >>be a mandatory part of the QEMU build process.  We cannot introduce
> >>new dependencies in order to build tests.
> >>
> >>Making autotest an unconditional dependency of qemu.git is a non-starter 
> >>for me.
> >
> >You have dependencies on gtest and other libraries. it's not that
> >different.
> 
> We have a dependency on glib, which gtest happens to be a part of.
> It was a big deal introducing that dependency and gtest was not the
> only (or primary) reason to introduce that dependency.
> 
> >Plus it's not unconditional: the test runner will report tests
> >SKIPPED if a dependency is not present.
> 
> But then the tests aren't run so if most developers didn't have it
> installed, and most tests were written with it, most developers
> wouldn't be running most tests which defeats the purpose, no?

Part of a TDD approach is to have build and test bots frequently
running tests on multiple platforms with different
configurations.

You can't expect developers to run all tests all the time.

When I talk about the "grid mode", I'm talking about the
multiplexing of configuration and platforms that you get when you
run an instrummented test using autotest.

> 
> >It's actually the oposite: you'll have to forbid developers from
> >submiting tests that use libautotest if you don't want the
> >dependency.
> 
> I think part of the problem here is the abstract nature of this
> discussion. What is libautotest actually supposed to do?
> 
> This would be an easier discussion if we were talking about patches
> instead of proposals.

We're at the RFC level, patches are being worked on, but this
thread is helping us shape the work we'll do. For example, if the
usage of libautotest if forbidden in qemu.git, we can avoid a lot
of our to-do.

Ditto for the JeOS images. Based on this discussion we figured
out that we won't be able to distribute the binary images.
(thanks!)

> 
> >>>   - QE will be alienated from the qemu test effort. There will be
> >>>     no integration between the QE efforts and the maintenance of
> >>>     the qemu developer-level tests.
> >>
> >>I think we're a pretty friendly and open community :-)  There is no
> >>reason that QE should be "alienated" unless folks are choosing not
> >>to participate upstream.
> >
> >For the exact same reasons you as a developer don't want to
> >implement tests inside autotest, QE won't want to implement tests
> >for qemu.git. It's out of their comfort zone, just put yourself
> >on their shoes.
> 
> This is a really, really poor argument and I hope I don't need to go
> into details of why.  If the primary reason for libautotest is so
> the people writing tests for QEMU can avoid actually working with
> the developers of QEMU...  we've got a problem.

No, one of the benefits of having libautotest is to *collaborate*
with QE. I'll explain again:

- As a qemu developer, I don't want to spend my time learning and
  getting involved in autotest, which is a complex QE project
  (I heard this numerous times).

- As a Quality Engineer, I don't want to invest my time learning
  and getting involved into upstream qemu to test HEAD.

> >>
> >>>   - You don't get the goodies from autotest (standardized system
> >>>     info collection, video recording, grid support, etc).
> >>>   - The new tests will work only on the master branch.
> >>
> >>This last one is a legitimate point that I have considered myself.
> >>But I think the value of having tests in HEAD outweigh the cost
> >>here.
> >
> >Thinking about qemu only, agree.
> >
> >>
> >>>3. A mix of (1) and (2): we "move" everything under qemu.git, but
> >>>keep the compatibility layer and provide a "libqemutest" for
> >>>third-parties.
> >>>
> >>>Anybody writting a test that interacts with qemu will use this
> >>>library: qemu, libvirt, spice, ovirt, QE tests, etc.
> >>
> >>I really see this all as over architecting to be honest.
> >>
> >>Can someone explain in clear terms (without appealing to maturity,
> >>flexibility, or any other qualitative aspect) why it takes anything
> >>more than a few dozen shell functions to write all of the tests that
> >>are in kvm-autotest test today?
> >
> >This is close to being offensive IMO, so I won't answer. :-)
> 
> I honestly don't mean it to be so I apologize if it comes off that
> way.  But I dislike arguing in such abstract terms.  What are the
> features that you think libautotest is needed for?

As a manager relatively new to the team I lack the technical
expertise to put it in more practical terms. But check the other
e-mail from Lucas where there's a virtio-console test. I'll reply
to it.

> 
> >>You launch qemu (by just using a command), and you interact with the
> >>monitor via QMP and HMP.  Why do you ever need anything more than
> >>that?
> >
> >Well, again, I'll agree to disagree.
> >
> >>
> >>Why does libautotest even need to exist?
> >
> >You're looking at things from the sole point of view of a qemu
> >developer. If you don't want to see things from a different
> >perspective and accept any kind of tradeoffs for better
> >collaboration with other teams, then unfortunately there's not
> >much we can do. It's a pity, because collaboration goes both
> >ways.
> 
> I need something more concrete then this.  What is it about
> libautotest that would allow collaboration?
> 
> >We're trying very hard to satisfy your requirements and
> >interests. Don't you see any way you can help us satisfy
> >ours?
> 
> I'm having a hard time understanding what your requirements are
> other than that we should write our tests using libautotest (which
> has not really been explained other than being levied as a
> requirement).

I have no requirements other than the ones that interest you. I
have interests, listed in the beginning of this thread:

    - remove barriers for developers to write tests
    - share the testing effort between multiple projects
      (qemu, libvirt, ovirt, spice, QE, etc...)

Cheers,
  - Ademar

-- 
Ademar de Souza Reis Jr.
Red Hat

^[:wq!