Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Ademar Reis]

On Thu, Mar 08, 2012 at 07:36:11AM -0600, Anthony Liguori wrote:
> On 03/07/2012 10:00 PM, Lucas Meneghel Rodrigues wrote:
> >Hi guys. For a while we have been discussing ways to make the virtualization
> >tests written on top of autotest useful for development level testing.
> >
> >One of our main goals is to provide useful tools for the qemu community, 
> >since
> >we have a good number of tests and libraries written to perform 
> >integration/QA
> >testing for that tool, being successfuly used by a number of QA teams that 
> >work
> >on qemu. Also, we recently provided a subset of that infrastructure to test
> >libvirt, one of our virtualization projects of interest.
> >
> >We realized that some (admittedly not very radical) changes have to be made 
> >on
> >autotest itself, so we're inviting other users of autotest to give this a 
> >good
> >read. This same document lives in the autotest wiki:
> >
> >https://github.com/autotest/autotest/wiki/FuturePlans
> >
> >Please note that splitting the virt tests from autotest is not discarded at 
> >the
> >moment, and it's not incompatible with the plan outlined below.
> >
> >====================================
> >Virt tests and autotest future goals
> >====================================
> >
> >In order to make the autotest infrastructure, and the virt tests developed on
> >top of autotest more useful for the people working on developing linux, virt 
> >and
> >other platform software, as well as the QA teams, we are working on a number 
> >of
> >goals, to streamline, simplify and make the available tools appropriate for
> >*development level testing*.
> >
> >Executing tests appropriate for *QA level testing* will continue to be
> >supported, as it's one of the biggest strenghts of autotest.
> >
> >The problem
> >-----------
> >
> >Autotest provides as of today a local engine, used to run tests on your local
> >machine (your laptop or a remote server). Currently, it runs tests properly
> >wrapped and packaged under the autotest client/tests/ folder, using specific
> >autotest rules.
> >
> >For the virt tests that live inside autotest, we have even more rules to 
> >follow,
> >causing a lot of frustration for people that are not used to how things are
> >structured and how to execute and select tests.
> >
> >The proposed solution
> >---------------------
> >
> >A solution is needed for both scenarios (virt and other general purpose 
> >tests).
> >The idea is to create specialized tools can run simple tests without 
> >packaging,
> >code that:
> >
> >* Knows nothing about the underlying infrastructure
> >* Written in any language (shell script, c, perl, you name it)
> >
> >It'll be up to the test runner to make sense of the results, provided that 
> >the
> >test writer follows some simple common sense principles while writing the 
> >code:
> >
> >1) Make the program to return 0 on success, !=0 on failure
> >2) Make the program use a test output format, mainly TAP
> 
> We're using gtest in QEMU, not TAP--for better or worse.  If
> autotest could use the gtest protocol, it would integrate much
> better with QEMU's tests.
> 
> Within QEMU, everything should be gtest when possible.
> 
> >For simple tests, we believe that option 1) will be far more popular. 
> >Autotest
> >will harness the execution of the test and put the results under the test
> >results directory, with all the sysinfo collection and other instrumentation
> >transparently available for the user.
> >
> >At some point, the test writer might want to start the framework features 
> >that
> >need to be enabled explicitly, then he/she might want to learn how to use the
> >python API to do so, but it'll not be a requirement.
> >
> >More about the test runner
> >--------------------------
> >
> >The test runner for both general and virt cases should have very simple and
> >minimal output:
> >
> >::
> >
> >Results stored in /path/to/autotest/results/default
> >my-full-test.py -- PASS
> >my-other-test.py -- PASS
> >look-mom-i-can-use-shell.sh -- PASS
> >look-mom-i-can-use-perl.pl -- FAIL
> >test-name-is-the-description.sh -- PASS
> >my-yet-another-test.sh -- SKIPPED
> >i-like-python.py -- PASS
> >whatever-test.pl -- PASS
> >
> >Both will be specialized tools that use the infrastructure of
> >client/bin/autotest-local, but with special code to attend to the output spec
> >above. They will know how to handle dependencies, and skip tests if needed.
> >
> >Directory structure
> >-------------------
> >
> >This is just to give a rough idea of how we won't depend the tests to be in 
> >the
> >autotest source code folder:
> >
> >::
> >
> >/path/to/autotest -> top level dir, that will make the autotest libs 
> >available
> >- client/bin -> Contains the test runners and auxiliary scripts
> >- client/virt/tests: Contains the virt tests that still live in autotest
> >- client/tests/kvm/tests: Contains the qemu tests that still live in autotest
> >
> >/any/path/test1: Contains tests for software foo
> >/any/path/test2: Contains tests for software bar
> >
> >/any/path/images: Contains minimal guest images for virtualization tests
> >
> >Bootstrap procedure
> >-------------------
> >
> >In order to comfortably use the framework features, some bootstrap steps 
> >will be
> >needed, along the following lines:
> >
> >::
> >
> >git clone git://github.com/autotest/autotest.git /path/to/autotest
> >export PATH='/path/to/autotest/client/bin':$PATH
> >export PYTHON_PATH='/path/to/autotest':$PYTHON_PATH
> >export AUTOTEST_DATA='/path/to/images'
> >
> >Writing tests
> >-------------
> >
> >Simple tests, general case
> >~~~~~~~~~~~~~~~~~~~~~~~~~~
> >
> >As previously mentioned, writing a trivial test is as simple as writing a
> >program that returns either 0 (true) or any other value (false). Autotest
> >returns PASS on true and FAIL on false.
> >
> >Simple tests, virt case
> >~~~~~~~~~~~~~~~~~~~~~~~
> >
> >The difference is that the program might be executed in the guest or the 
> >host,
> >so a command line flag or environment variable might be set to indicate where
> >the program should be executed (host, guest or both). Autotest returns PASS 
> >on
> >true and FAIL on false. This functionality is inspired on qemu-test, thanks 
> >to
> >Anthony Liguori.
> >
> >Instrumented tests, general case
> >~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> >
> >The test author can learn how to create an autotest wrapper for the test 
> >suite
> >and use the specialized tool to run it.
> >
> >Instrumented tests, virt case
> >~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> >
> >The test author can learn how to create an instrumented test for 
> >virtualization,
> >using the python APIs, or any other language using auxiliary scripts that
> >encapsulate high level functionality for use on shell scripts or other
> >languages. Ideas for auxiliary scripts:
> >
> >::
> >
> >virt_run_migration [params] [options]
> >virt_run_timedrift [params] [options]
> >virt_run_nic_hotplug [params] [options]
> >virt_run_block_hotplug [params] [options]
> >
> >Where tests live
> >----------------
> >
> >The tests won't need to be in the autotest tree, they can live anywhere. The
> >reason for this is that projects need in tree tests, that can be maintained 
> >by
> >the project maintainers.
> >
> >Standard use case for virt is to have both trivial and instrumented tests 
> >living
> >in the respective project's tree (qemu and libvirt). Trivial tests don't need
> >autotest libs, while instrumented tests will need to, but that's OK provided
> >that the appropriate bootstrap procedure was made.
> >
> >Test Examples
> >-------------
> >
> >simple, non instrumented
> >~~~~~~~~~~~~~~~~~~~~~~~~
> >
> >::
> >
> >uptime.sh:
> >#!/bin/sh
> >exec uptime
> >
> >uptime.py:
> >#!/usr/bin/python
> >import os, sys
> >sys.exit(os.system("uptime"))
> >
> >uptime.pl:
> >#!/usr/bin/perl
> >system("uptime");
> >exit($?);
> >
> >qemu-img-convert.sh
> >#!/bin/bash
> >qemu-img convert -O qcow2 $DATA/qemu-imgs/reference.vdi $TEMPDIR/output.qcow2
> >diff -b $TEMPDIR/output.qcow2 $DATA/qemu-imgs/reference.qcow2 > /dev/null
> >...
> >
> >uptime.py - instrummented using libautotest
> >~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> >
> >::
> >
> >#!/bin/python
> >
> >from autotest import utils, logging
> >
> >def run_uptime_host(test, params, env):
> >uptime = utils.system_output("uptime")
> >logging.info("Host uptime result is: %s", uptime)
> >
> >
> >uptime.py - host/guest mode, instrummented using libautotest
> >~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> >
> >::
> >
> >#!/bin/python
> >
> >from autotest import utils, logging
> >
> >def run_uptime_host_and_guest(test, params, env):
> >vm = env.get_vm(params["main_vm"])
> >vm.verify_alive()
> >session = vm.wait_for_login()
> >
> >uptime_guest = session.cmd("uptime")
> >logging.info("Guest uptime result is: %s", uptime_guest)
> >
> >uptime_host = utils.system_output("uptime")
> >logging.info("Host uptime result is: %s", uptime_host)
> >
> >
> >Virt/qemu tests: Minimal guest images
> >-------------------------------------
> >
> >In order to make development level test possible, we need the tests to run 
> >fast.
> >In order to do that, a set of minimal guest images is being developed and we
> >have a version for x86_64 ready and functional:
> >
> >https://github.com/autotest/buildroot-autotest
> 
> I'm really not a fan of buildroot.  Note that in order to ship
> binaries, full source needs to be provided in order to comply with
> the GPL.  The FSF at least states that referring to another website
> for source that's not under your control doesn't satisfy the
> requirements of the GPL.
> 
> Just out of curiosity, did you try to use qemu-test?  Is there a
> reason you created something different?
> 
> I think it's good that you're thinking about how to make writing
> tests easier, but we have a growing test infrastructure in QEMU and
> that's what I'd prefer people focused on.
> 

You probably remember the long thread we had back in December on
qemu-devel on this topic. Back then our message was "we have a
growing test infrastructure in s/QEMU/autotest/ and that's what
we'd prefer people focused on". :-)

>From Dor:

(http://lists.nongnu.org/archive/html/qemu-devel/2011-12/msg03024.html)

"""
If you wish, you can challenge Lucas and Cleber w/ these type of
requirements and we'll all improve as a result.
"""

Your response was:

"""
Well consider qemu-test the challenge. It's an existence proof
that we can have a very easy to use framework for testing that
runs extremely fast and is very easy to write tests for.
"""

http://knowyourmeme.com/memes/challenge-accepted ;-)

I particularly agreed with basically everything you said on that
discussion regarding test simplification (I had just joined the
team back then). To me, autotest has been focusing on QE-level,
leaving the developer-level test requirements out. Now we're
attacking this new front, and a lot of the requirements are
indeed from that discussion.

By simplifying the design and bringing barriers down, we hope to
reach a broader audience and help developers write and maintain
tests, benefiting from all the instrumentation that autotest
brings. It's not going to be just about qemu (check the new test
examples).

We have a team fully dedicated to autotest and it's used not only
by Qemu but also libvirt, Google, Xen, Fedora, Twitter, etc, etc
(these all have code contributions in autotest)

That said, the current qemu-tests will probably be easily
integrated into (the new) autotest and we hope that, given enough
time, autotest will be good enough to relieve qemu from the
framework maintenance and code duplication with other projects.

We hope in the future you see autotest the way you see gtest
right now: a library that helps you write better tests. The fully
instrummented test-runner and the hability to run tests in the
cloud will hopefully bring a lot of value as well.

Please raise your requirements. We would love to avoid code
duplication and be integrated as much as possible.

Cheers,
   - Ademar

-- 
Ademar de Souza Reis Jr.
Red Hat

^[:wq!