qemu-devel
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Qemu-devel] Functional tests (AKA Avocado-based tests)


From: Lukáš Doktor
Subject: Re: [Qemu-devel] Functional tests (AKA Avocado-based tests)
Date: Thu, 18 Jan 2018 10:25:30 +0100
User-agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.4.0

Dne 18.1.2018 v 00:41 Alistair Francis napsal(a):
> On Wed, Jan 17, 2018 at 12:05 AM, Cleber Rosa <address@hidden> wrote:
>> TL;DR
>> =====
>>
>> This is about how QEMU developers can get started with functional
>> tests that are built on top of the Avocado libraries (and meant to be
>> run with the Avocado test runner).
>>
>> The past
>> ========
>>
>> The Avocado project[1] has been working, for quite some time now, on a
>> "set of tools and libraries" with the goal of making writing tests
>> easier.  It is supposed to be a framework agnostic to the exact
>> software that will be under test.
>>
>> But, at the same time, the Avocado project cannot deny its inheritance
>> and influences.  Those come from Autotest[2], which had "KVM Autotest"
>> as its largest and most developed "test".  This large Autotest test
>> (KVM Autotest) became virt-test[3] and later got integrated into
>> Avocado and became Avocado-VT[4] which is quite relevant here,
>> together with its QEMU test provider[5].
>>
>> Avocado-VT and the QEMU test provider attempt to provide coverage
>> across platform and QEMU versions, which increases its complexity.
>> Also, it's built on a legacy set of principles and tools that makes
>> some developers stir away from it.
>>
>> What's new?
>> ===========
>>
>> A few months ago, the Avocado developers returned to its
>> "virtualization origins", in an attempt to learn from the QEMU
>> project, and try to help with a way to have more functional tests in
>> the upstream QEMU repo.
>>
>> We believe it's possible to expand the test coverage for QEMU by
>> facilitating
>> the creation of more functional tests QEMU.  This is no different than how
>> other types of tests are already included in the tree itself.
>>
>> How
>> ===
>>
>> How we did it (so far)
>> ----------------------
>>
>> We're aware that there's a dilemma here: to be able to easily write
>> more powerful tests, a lot of the complexity has to be moved
>> elsewhere.  Here, it means moving complexity from the test itself to a
>> framework.  The QEMU source tree itself has proofs of this approach,
>> being the "scripts" and "tests/qemu-iotests" some of the examples.
>>
>> Avocado itself[1] provides a lot of the code that should help to
>> absorb some of the complexities in writing tests, but not exactly
>> everything that is needed for QEMU.  The approach we believe will have
>> the best balance is to reuse upstream Avocado libraries whenever they
>> are useful and generic enough, and on top of that, libraries that are
>> part of QEMU itself.
>>
>> How can you get started with it
>> -------------------------------
>>
>> First of all, get Avocado installed.  Besides the Avocado test runner
>> itself, this will give you the basic libraries on which the other part
>> of this work was built on.  We want that to be simple and painless, so
>> here's our best bet for a one-liner installation:
>>
>>   pip install --user avocado-framework
>> avocado-framework-plugin-varianter-yaml-to-mux aexpect
>>
>> That will install Avocado within the user's home directory.  If you
>> give up on it, it can be uninstalled with another simple one-liner:
>>
>>   pip uninstall -y avocado-framework
>> avocado-framework-plugin-varianter-yaml-to-mux aexpect
>>
>> Now, suppose you're working on a given feature, and want to try your
>> luck writing a test using this work.  To avoid having you fetching and
>> rebasing from our currently in development fork[6] and branch[7], you
>> can just
>> add one commit to your tree with:
>>
>>   curl
>> https://patch-diff.githubusercontent.com/raw/apahim/qemu/pull/17.patch |
>> git am -
>>
>> This will get a simple patch from a snapshot branch[8].  You can, of course,
>> do it "the git way", fetching from that repo[6] and using the
>> non-snapshotted branch.
>>
>> After that, we'd love for you to take a look at some of the existing
>> tests[9][10] and then attempt to create test for your own use case.
>> The basic README[11] file, and the Avocado documentation[12] are also
>> important resources not to be missed.
>>
>> What's next?
>> ============
>>
>> Initially, feedback is what we're looking for.  It would be greatly
>> appreciated to understand if/how this suits (or not) use cases out
>> there.
>>
>> After feedback, further refinements, and more tests are written, the
>> Avocado developers will follow up with an initial patch series for
>> upstream QEMU.  In such a proposal, we intend to have further
>> integration.  Ideally in way that "configure" can be given a
>> "--with-functional-[avocado-]tests" parameter of sorts, and a "make
>>  [functional-]check" would seamlessly include them.
> 
> I have a few thoughts.
> 
> We use pytest/pexpect internally to kick off QEMU runs and monitor the
> output (no interaction with the QEMU source tree) and I think it is
> extremely useful. So I am all for using Python to test things and this
> looks really well done!
> 
> What I don't understand though is what this gives us compared to the
> existing QEMU test infrastructure? Besides being able to use Python
> and a better interface what are the main benefits? I think that is
> something worth documenting somewhere.
> 

Hello Alistar,

It was just briefly mentioned by Cleber in the main email, but to be more 
detailed there are 2 (3) benefits:

1. Using a test runner avoids boiler-plate code to create and provide results. 
Avocado is a test runner and can unify the execution as well as results. 
Avocado results are compatible with multiple results DBs/systems and even 
allows diffing (you can see what changed between different executions).

2. Avocado is not just a test runner, but also set of utils. This can simplify 
writing tests as one does not have to reinvent the wheel and can simply say "I 
require this pkg, install it", "I want to start this service" or run commands 
interactively without the need to worry about setting pty, reading-out the 
pipes etc.

3. There are at least three Red Hat employees based in virt team who work on 
Avocado development. This means there is a background to support you the 
community to simplify writing tests (avocado_qemu "helpers" are one example). 
In return we expect less regressions as more testing can be performed easily 
before releases/merges/during development.

Regards,
Lukáš


> Also, it looks like this will require images checked into git
> somewhere is that correct? Is there a good plan on how to handle that?
> 
> Alistair
> 
>>
>> Thanks!
>>
>> References
>> ==========
>>
>>  [1] http://avocado-framework.github.io/
>>  [2] http://autotest.github.io/
>>  [3] https://github.com/autotest/virt-test
>>  [4] https://github.com/avocado-framework/avocado-vt
>>  [5] https://github.com/autotest/tp-qemu
>>  [6] https://github.com/apahim/qemu
>>  [7] https://github.com/apahim/qemu/tree/avocado_qemu
>>  [8] https://github.com/apahim/qemu/tree/avocado_qemu_snapshot
>>  [9]
>> https://github.com/apahim/qemu/blob/avocado_qemu/tests/avocado/test_info_memdev_host_nodes.py
>> [10]
>> https://github.com/apahim/qemu/blob/avocado_qemu/tests/avocado/test_ovmf_with_240_vcpus.py
>> [11]
>> https://github.com/apahim/qemu/blob/avocado_qemu/tests/avocado/README.rst
>> [12] http://avocado-framework.readthedocs.io/
>>
>> --
>> Cleber Rosa
>> [ Sr Software Engineer - Virtualization Team - Red Hat ]
>> [ Avocado Test Framework - avocado-framework.github.io ]
>> [  7ABB 96EB 8B46 B94D 5E0F  E9BB 657E 8D33 A5F2 09F3  ]
>>


Attachment: signature.asc
Description: OpenPGP digital signature


reply via email to

[Prev in Thread] Current Thread [Next in Thread]