Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Ademar Reis]

On Thu, Mar 08, 2012 at 08:56:23AM -0600, Anthony Liguori wrote:
> On 03/08/2012 08:49 AM, Ademar Reis wrote:
> >On Thu, Mar 08, 2012 at 07:36:11AM -0600, Anthony Liguori wrote:
> >>On 03/07/2012 10:00 PM, Lucas Meneghel Rodrigues wrote:
> >>>Virt/qemu tests: Minimal guest images
> >>>-------------------------------------
> >>>
> >>>In order to make development level test possible, we need the tests to run 
> >>>fast.
> >>>In order to do that, a set of minimal guest images is being developed and 
> >>>we
> >>>have a version for x86_64 ready and functional:
> >>>
> >>>https://github.com/autotest/buildroot-autotest
> >>
> >>I'm really not a fan of buildroot.  Note that in order to ship
> >>binaries, full source needs to be provided in order to comply with
> >>the GPL.  The FSF at least states that referring to another website
> >>for source that's not under your control doesn't satisfy the
> >>requirements of the GPL.
> >>
> >>Just out of curiosity, did you try to use qemu-test?  Is there a
> >>reason you created something different?
> >>
> >>I think it's good that you're thinking about how to make writing
> >>tests easier, but we have a growing test infrastructure in QEMU and
> >>that's what I'd prefer people focused on.
> >>
> >
> >You probably remember the long thread we had back in December on
> >qemu-devel on this topic. Back then our message was "we have a
> >growing test infrastructure in s/QEMU/autotest/ and that's what
> >we'd prefer people focused on". :-)
> >
> > From Dor:
> >
> >(http://lists.nongnu.org/archive/html/qemu-devel/2011-12/msg03024.html)
> >
> >"""
> >If you wish, you can challenge Lucas and Cleber w/ these type of
> >requirements and we'll all improve as a result.
> >"""
> >
> >Your response was:
> >
> >"""
> >Well consider qemu-test the challenge. It's an existence proof
> >that we can have a very easy to use framework for testing that
> >runs extremely fast and is very easy to write tests for.
> >"""
> >
> >http://knowyourmeme.com/memes/challenge-accepted ;-)
> >
> >I particularly agreed with basically everything you said on that
> >discussion regarding test simplification (I had just joined the
> >team back then). To me, autotest has been focusing on QE-level,
> >leaving the developer-level test requirements out. Now we're
> >attacking this new front, and a lot of the requirements are
> >indeed from that discussion.
> 
> If you want to talk about this in terms of "requirements", my
> requirement is for "developer-level" tests to live in qemu.git and
> be integrated into make check.
> 
> Just as we've been discussing and working on since the previous set of 
> discussions.
> 
> >By simplifying the design and bringing barriers down, we hope to
> >reach a broader audience and help developers write and maintain
> >tests, benefiting from all the instrumentation that autotest
> >brings. It's not going to be just about qemu (check the new test
> >examples).
> >
> >We have a team fully dedicated to autotest and it's used not only
> >by Qemu but also libvirt, Google, Xen, Fedora, Twitter, etc, etc
> >(these all have code contributions in autotest)
> >
> >That said, the current qemu-tests will probably be easily
> >integrated into (the new) autotest and we hope that, given enough
> >time, autotest will be good enough to relieve qemu from the
> >framework maintenance and code duplication with other projects.
> 
> autotest should not be the focal point for integration.  qemu.git should be.
> 
> I'd be perfectly happy to review patches submitting the test
> infrastructure from kvm-autotest into qemu.git (provided it didn't
> have unreasonable external dependencies and fit into QEMU).
> 
> Developer-level tests need to live where the developers live.  The
> developers live in qemu.git.  See my other response on this thread
> for the explanation of why this is so important.
> 

Excelent, we're in the same page then. This was my number 1
requirement when I was discussing the changes with Lucas and
Cleber. For convenience, I'll repeat here what I wrote in a
previous e-mail (no qemu-devel archive available yet to use as a
reference).

In summary, autotest is (or is going to be) a framework that
provides:

 - A test runner, with grid/cluster support and advanced
   instrumentation
 - A devel library and set of utilities for test writers
 - A set of pre-built images (JeOS – Just Enough OS) for
   test writers

(attached is a picture showing what we want to achieve)

If a project has an internal library or set of utilities that can
be of general use, they can be submitted to autotest.git for
inclusion, thus reaching a broader audience.

A short summary of the plans:

 - Tests can live anywhere and each devel team implements and
   maintains their own set of tests
 - Usage of the autotest library by test writers is optional
 - Tests are scripts returning 0 or error (any language)
 - Tests can be run individually or in sets
 - Tests should run fast, our target is seconds or a few minutes
 - The test runner is smart and “just works” by default
   - Trivial standard output (FAIL, PASS, SKIPPED)
   - Collect logs, OS data and other stuff (e.g. --record-video!)
   - Skip unsupported tests based on the environment they're run
   - Multiplex configurations / platforms when on the grid
   - Support to run tests “in the cloud”

There are also lots of small changes and usability improvements
in the pipeline (and feedback right now is very valuable).

Thanks,
  - Ademar

-- 
Ademar de Souza Reis Jr.
Red Hat

^[:wq!