Re: [Qemu-devel] Functional tests (AKA Avocado-based tests)

[Cleber Rosa]

On 02/01/2018 07:10 PM, Alistair Francis wrote:
> On Wed, Jan 17, 2018 at 4:47 PM, Cleber Rosa <address@hidden> wrote:
>>
>>
>> On 01/17/2018 06:41 PM, Alistair Francis wrote:
>>> On Wed, Jan 17, 2018 at 12:05 AM, Cleber Rosa <address@hidden> wrote:
>>>> TL;DR
>>>> =====
>>>>
>>>> This is about how QEMU developers can get started with functional
>>>> tests that are built on top of the Avocado libraries (and meant to be
>>>> run with the Avocado test runner).
>>>>
>>>> The past
>>>> ========
>>>>
>>>> The Avocado project[1] has been working, for quite some time now, on a
>>>> "set of tools and libraries" with the goal of making writing tests
>>>> easier.  It is supposed to be a framework agnostic to the exact
>>>> software that will be under test.
>>>>
>>>> But, at the same time, the Avocado project cannot deny its inheritance
>>>> and influences.  Those come from Autotest[2], which had "KVM Autotest"
>>>> as its largest and most developed "test".  This large Autotest test
>>>> (KVM Autotest) became virt-test[3] and later got integrated into
>>>> Avocado and became Avocado-VT[4] which is quite relevant here,
>>>> together with its QEMU test provider[5].
>>>>
>>>> Avocado-VT and the QEMU test provider attempt to provide coverage
>>>> across platform and QEMU versions, which increases its complexity.
>>>> Also, it's built on a legacy set of principles and tools that makes
>>>> some developers stir away from it.
>>>>
>>>> What's new?
>>>> ===========
>>>>
>>>> A few months ago, the Avocado developers returned to its
>>>> "virtualization origins", in an attempt to learn from the QEMU
>>>> project, and try to help with a way to have more functional tests in
>>>> the upstream QEMU repo.
>>>>
>>>> We believe it's possible to expand the test coverage for QEMU by
>>>> facilitating
>>>> the creation of more functional tests QEMU.  This is no different than how
>>>> other types of tests are already included in the tree itself.
>>>>
>>>> How
>>>> ===
>>>>
>>>> How we did it (so far)
>>>> ----------------------
>>>>
>>>> We're aware that there's a dilemma here: to be able to easily write
>>>> more powerful tests, a lot of the complexity has to be moved
>>>> elsewhere.  Here, it means moving complexity from the test itself to a
>>>> framework.  The QEMU source tree itself has proofs of this approach,
>>>> being the "scripts" and "tests/qemu-iotests" some of the examples.
>>>>
>>>> Avocado itself[1] provides a lot of the code that should help to
>>>> absorb some of the complexities in writing tests, but not exactly
>>>> everything that is needed for QEMU.  The approach we believe will have
>>>> the best balance is to reuse upstream Avocado libraries whenever they
>>>> are useful and generic enough, and on top of that, libraries that are
>>>> part of QEMU itself.
>>>>
>>>> How can you get started with it
>>>> -------------------------------
>>>>
>>>> First of all, get Avocado installed.  Besides the Avocado test runner
>>>> itself, this will give you the basic libraries on which the other part
>>>> of this work was built on.  We want that to be simple and painless, so
>>>> here's our best bet for a one-liner installation:
>>>>
>>>>   pip install --user avocado-framework
>>>> avocado-framework-plugin-varianter-yaml-to-mux aexpect
>>>>
>>>> That will install Avocado within the user's home directory.  If you
>>>> give up on it, it can be uninstalled with another simple one-liner:
>>>>
>>>>   pip uninstall -y avocado-framework
>>>> avocado-framework-plugin-varianter-yaml-to-mux aexpect
>>>>
>>>> Now, suppose you're working on a given feature, and want to try your
>>>> luck writing a test using this work.  To avoid having you fetching and
>>>> rebasing from our currently in development fork[6] and branch[7], you
>>>> can just
>>>> add one commit to your tree with:
>>>>
>>>>   curl
>>>> https://patch-diff.githubusercontent.com/raw/apahim/qemu/pull/17.patch |
>>>> git am -
>>>>
>>>> This will get a simple patch from a snapshot branch[8].  You can, of 
>>>> course,
>>>> do it "the git way", fetching from that repo[6] and using the
>>>> non-snapshotted branch.
>>>>
>>>> After that, we'd love for you to take a look at some of the existing
>>>> tests[9][10] and then attempt to create test for your own use case.
>>>> The basic README[11] file, and the Avocado documentation[12] are also
>>>> important resources not to be missed.
>>>>
>>>> What's next?
>>>> ============
>>>>
>>>> Initially, feedback is what we're looking for.  It would be greatly
>>>> appreciated to understand if/how this suits (or not) use cases out
>>>> there.
>>>>
>>>> After feedback, further refinements, and more tests are written, the
>>>> Avocado developers will follow up with an initial patch series for
>>>> upstream QEMU.  In such a proposal, we intend to have further
>>>> integration.  Ideally in way that "configure" can be given a
>>>> "--with-functional-[avocado-]tests" parameter of sorts, and a "make
>>>>  [functional-]check" would seamlessly include them.
>>>
>>> I have a few thoughts.
>>>
>>> We use pytest/pexpect internally to kick off QEMU runs and monitor the
>>> output (no interaction with the QEMU source tree) and I think it is
>>> extremely useful. So I am all for using Python to test things and this
>>> looks really well done!
>>>
>>
>> Thanks for checking it out, and for the positive words.  Now, sorry if
>> I'm missing some obvious information, but is this work of yours with
>> pytest/pexpect publicly available?  I'd like to also take a look at
>> that, because it does look similar to the Avocado + aexpect approach
>> taken here.
> 
> Unfortunately it's not and it would take months for us to be able to
> make it available.
> 

I see.

>>
>>> What I don't understand though is what this gives us compared to the
>>> existing QEMU test infrastructure? Besides being able to use Python
>>> and a better interface what are the main benefits? I think that is
>>> something worth documenting somewhere.
>>>
>>
>> We currently intend to *add* to the QEMU test infrastructure, not
>> replace it.
> 
> Is there a benefit of integrating it into the tree then? It's always
> possible to have an out of tree testing framework.
> 

Upstream first is just the general modus operandi that we have.  Then,
we want to have as much testing as possible as early as possible.

Finally, by having tests in-tree, they can be seen in different light.
What I mean is that when tests are in-tree, they will (or should) really
map to the current state of the code they test (per commit).  If a
feature changes behavior upstream, the respective test will also need a
change at the same time, and as such, will mean an *intended* change and
not a regression.  With out-of-tree tests, it's pretty hard to keep this
synchronization and regressions will slip in, *at least* for some time.

I guess the question here is actually the opposite: do you see any
problems with in-tree tests?  Since you keep out of tree tests, I would
honestly like to hear about your experience.

>>
>> The benefits we envision are, besides hopefully easier and more capable
>> interfaces, to simply have more upstream tests.  This means avoiding new
>> regressions and improving coverage.
>>
>>> Also, it looks like this will require images checked into git
>>> somewhere is that correct? Is there a good plan on how to handle that?
>>>
>>
>> It won't require images checked into git.  Right now, tests use the
>> vmimage library:
>>
>> http://avocado-framework.readthedocs.io/en/57.0/api/utils/avocado.utils.html#avocado.utils.vmimage.get
>>
>> Which downloads (and caches) images from external sources.
> 
> Ah! That's cool. Managing images is one of the challenges we have at the 
> moment.
> 
> Alistair
> 

Nice that you like it.  We also find a library such as "vmimage" is
something simple enough, but still not quite something that would live
in the QEMU tree.

- Cleber.

>>
>> Please let me know if you have more questions!
>> - Cleber.
>>
>>> Alistair
>>>
>>>>
>>>> Thanks!
>>>>
>>>> References
>>>> ==========
>>>>
>>>>  [1] http://avocado-framework.github.io/
>>>>  [2] http://autotest.github.io/
>>>>  [3] https://github.com/autotest/virt-test
>>>>  [4] https://github.com/avocado-framework/avocado-vt
>>>>  [5] https://github.com/autotest/tp-qemu
>>>>  [6] https://github.com/apahim/qemu
>>>>  [7] https://github.com/apahim/qemu/tree/avocado_qemu
>>>>  [8] https://github.com/apahim/qemu/tree/avocado_qemu_snapshot
>>>>  [9]
>>>> https://github.com/apahim/qemu/blob/avocado_qemu/tests/avocado/test_info_memdev_host_nodes.py
>>>> [10]
>>>> https://github.com/apahim/qemu/blob/avocado_qemu/tests/avocado/test_ovmf_with_240_vcpus.py
>>>> [11]
>>>> https://github.com/apahim/qemu/blob/avocado_qemu/tests/avocado/README.rst
>>>> [12] http://avocado-framework.readthedocs.io/
>>>>
>>>> --
>>>> Cleber Rosa
>>>> [ Sr Software Engineer - Virtualization Team - Red Hat ]
>>>> [ Avocado Test Framework - avocado-framework.github.io ]
>>>> [  7ABB 96EB 8B46 B94D 5E0F  E9BB 657E 8D33 A5F2 09F3  ]
>>>>
>>
>> --
>> Cleber Rosa
>> [ Sr Software Engineer - Virtualization Team - Red Hat ]
>> [ Avocado Test Framework - avocado-framework.github.io ]
>> [  7ABB 96EB 8B46 B94D 5E0F  E9BB 657E 8D33 A5F2 09F3  ]

-- 
Cleber Rosa
[ Sr Software Engineer - Virtualization Team - Red Hat ]
[ Avocado Test Framework - avocado-framework.github.io ]
[  7ABB 96EB 8B46 B94D 5E0F  E9BB 657E 8D33 A5F2 09F3  ]