Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Ademar Reis]

On Thu, Mar 08, 2012 at 09:17:42AM -0300, Ademar Reis wrote:
> On Thu, Mar 08, 2012 at 11:54:31AM +0000, Stefan Hajnoczi wrote:
> > On Thu, Mar 8, 2012 at 11:44 AM, Stefan Hajnoczi <address@hidden> wrote:
> > > On Thu, Mar 8, 2012 at 4:00 AM, Lucas Meneghel Rodrigues <address@hidden> 
> > > wrote:
> > >> One of our main goals is to provide useful tools for the qemu community,
> > >> since we have a good number of tests and libraries written to perform
> > >> integration/QA testing for that tool, being successfuly used by a number 
> > >> of
> > >> QA teams that work on qemu. Also, we recently provided a subset of that
> > >> infrastructure to test libvirt, one of our virtualization projects of
> > >> interest.
> > >
> > > Thanks for sharing.
> > 
> > One thing I should have added is that my message is about what it
> > would take for me to use autotest and contribute tests.  But now I
> > realize that you might be going for a different model:
> > 
> > If you're aiming for a different model where autotest integrates
> > external test suites (i.e. tests wouldn't be written in autotest.git,
> > instead autotest.git would contain snapshots of external test suites),
> > then this proposal seems fine.  Upstream projects like QEMU would
> > develop their own test suite and it would be dropped into autotest or
> > a specific autotest instance.
> > 
> 
> Yes, that's the idea. We want autotest to be the framework, not
> (just a) collection of tests. We also want each development team
> to implement and maintain their own set of tests, using (or not)
> the goodies from autotest at their discretion.
> 
> In summary, autotest is (or is going to be) a framework that
> provides:
> 
>  - A test runner, with grid/cluster support and advanced
>    instrumentation
>  - A devel library and set of utilities for test writers
>  - A set of pre-built images (JeOS – Just Enough OS) for
>    test writers 
> 
> (attached is a picture showing what we want to achieve)

Facepalm right after pressing 'y'.

> 
> If a project has an internal library or set of utilities that can
> be of general use, they can be submitted to autotest.git for
> inclusion, thus reaching a broader audience.
> 
> A short summary of the plans:
> 
>  - Tests can live anywhere and each devel team implements and
>    maintains their own set of tests
>  - Usage of the autotest library by test writers is optional
>  - Tests are scripts returning 0 or error (any language)
>  - Tests can be run individually or in sets
>  - Tests should run fast, our target is seconds or a few minutes
>  - The test runner is smart and “just works” by default
>    - Trivial standard output (FAIL, PASS, SKIPPED)
>    - Collect logs, OS data and other stuff (e.g. --record-video!)
>    - Skip unsupported tests based on the environment they're run
>    - Multiplex configurations / platforms when on the grid
>    - Support to run tests “in the cloud”
> 
> There are also lots of small changes and usability improvements
> in the pipeline (and feedback right now is very valuable).
> 
> Thanks,
>   - Ademar
> 
> -- 
> Ademar de Souza Reis Jr.
> Red Hat
> 
> ^[:wq!

-- 
Ademar de Souza Reis Jr.
Red Hat

^[:wq!

autotest-arch.jpg
Description: JPEG image