Re: [Qemu-devel] Functional tests (AKA Avocado-based tests)

[Amador Pahim]

On Thu, Jan 18, 2018 at 1:47 AM, Cleber Rosa <address@hidden> wrote:
>
>
> On 01/17/2018 06:41 PM, Alistair Francis wrote:
>> On Wed, Jan 17, 2018 at 12:05 AM, Cleber Rosa <address@hidden> wrote:
>>> TL;DR
>>> =====
>>>
>>> This is about how QEMU developers can get started with functional
>>> tests that are built on top of the Avocado libraries (and meant to be
>>> run with the Avocado test runner).
>>>
>>> The past
>>> ========
>>>
>>> The Avocado project[1] has been working, for quite some time now, on a
>>> "set of tools and libraries" with the goal of making writing tests
>>> easier.  It is supposed to be a framework agnostic to the exact
>>> software that will be under test.
>>>
>>> But, at the same time, the Avocado project cannot deny its inheritance
>>> and influences.  Those come from Autotest[2], which had "KVM Autotest"
>>> as its largest and most developed "test".  This large Autotest test
>>> (KVM Autotest) became virt-test[3] and later got integrated into
>>> Avocado and became Avocado-VT[4] which is quite relevant here,
>>> together with its QEMU test provider[5].
>>>
>>> Avocado-VT and the QEMU test provider attempt to provide coverage
>>> across platform and QEMU versions, which increases its complexity.
>>> Also, it's built on a legacy set of principles and tools that makes
>>> some developers stir away from it.
>>>
>>> What's new?
>>> ===========
>>>
>>> A few months ago, the Avocado developers returned to its
>>> "virtualization origins", in an attempt to learn from the QEMU
>>> project, and try to help with a way to have more functional tests in
>>> the upstream QEMU repo.
>>>
>>> We believe it's possible to expand the test coverage for QEMU by
>>> facilitating
>>> the creation of more functional tests QEMU.  This is no different than how
>>> other types of tests are already included in the tree itself.
>>>
>>> How
>>> ===
>>>
>>> How we did it (so far)
>>> ----------------------
>>>
>>> We're aware that there's a dilemma here: to be able to easily write
>>> more powerful tests, a lot of the complexity has to be moved
>>> elsewhere.  Here, it means moving complexity from the test itself to a
>>> framework.  The QEMU source tree itself has proofs of this approach,
>>> being the "scripts" and "tests/qemu-iotests" some of the examples.
>>>
>>> Avocado itself[1] provides a lot of the code that should help to
>>> absorb some of the complexities in writing tests, but not exactly
>>> everything that is needed for QEMU.  The approach we believe will have
>>> the best balance is to reuse upstream Avocado libraries whenever they
>>> are useful and generic enough, and on top of that, libraries that are
>>> part of QEMU itself.
>>>
>>> How can you get started with it
>>> -------------------------------
>>>
>>> First of all, get Avocado installed.  Besides the Avocado test runner
>>> itself, this will give you the basic libraries on which the other part
>>> of this work was built on.  We want that to be simple and painless, so
>>> here's our best bet for a one-liner installation:
>>>
>>>   pip install --user avocado-framework
>>> avocado-framework-plugin-varianter-yaml-to-mux aexpect
>>>
>>> That will install Avocado within the user's home directory.  If you
>>> give up on it, it can be uninstalled with another simple one-liner:
>>>
>>>   pip uninstall -y avocado-framework
>>> avocado-framework-plugin-varianter-yaml-to-mux aexpect
>>>
>>> Now, suppose you're working on a given feature, and want to try your
>>> luck writing a test using this work.  To avoid having you fetching and
>>> rebasing from our currently in development fork[6] and branch[7], you
>>> can just
>>> add one commit to your tree with:
>>>
>>>   curl
>>> https://patch-diff.githubusercontent.com/raw/apahim/qemu/pull/17.patch |
>>> git am -
>>>
>>> This will get a simple patch from a snapshot branch[8].  You can, of course,
>>> do it "the git way", fetching from that repo[6] and using the
>>> non-snapshotted branch.
>>>
>>> After that, we'd love for you to take a look at some of the existing
>>> tests[9][10] and then attempt to create test for your own use case.
>>> The basic README[11] file, and the Avocado documentation[12] are also
>>> important resources not to be missed.
>>>
>>> What's next?
>>> ============
>>>
>>> Initially, feedback is what we're looking for.  It would be greatly
>>> appreciated to understand if/how this suits (or not) use cases out
>>> there.
>>>
>>> After feedback, further refinements, and more tests are written, the
>>> Avocado developers will follow up with an initial patch series for
>>> upstream QEMU.  In such a proposal, we intend to have further
>>> integration.  Ideally in way that "configure" can be given a
>>> "--with-functional-[avocado-]tests" parameter of sorts, and a "make
>>>  [functional-]check" would seamlessly include them.
>>
>> I have a few thoughts.
>>
>> We use pytest/pexpect internally to kick off QEMU runs and monitor the
>> output (no interaction with the QEMU source tree) and I think it is
>> extremely useful. So I am all for using Python to test things and this
>> looks really well done!
>>
>
> Thanks for checking it out, and for the positive words.  Now, sorry if
> I'm missing some obvious information, but is this work of yours with
> pytest/pexpect publicly available?  I'd like to also take a look at
> that, because it does look similar to the Avocado + aexpect approach
> taken here.
>
>> What I don't understand though is what this gives us compared to the
>> existing QEMU test infrastructure? Besides being able to use Python
>> and a better interface what are the main benefits? I think that is
>> something worth documenting somewhere.
>>
>
> We currently intend to *add* to the QEMU test infrastructure, not
> replace it.
>
> The benefits we envision are, besides hopefully easier and more capable
> interfaces, to simply have more upstream tests.  This means avoiding new
> regressions and improving coverage.
>
>> Also, it looks like this will require images checked into git
>> somewhere is that correct? Is there a good plan on how to handle that?
>>
>
> It won't require images checked into git.  Right now, tests use the
> vmimage library:
>
> http://avocado-framework.readthedocs.io/en/57.0/api/utils/avocado.utils.html#avocado.utils.vmimage.get
>
> Which downloads (and caches) images from external sources.

Expanding on that, the vmimage lib downloads and caches the so called
"cloud images" from distros own repositories. Here's[1] some
documentation with usage examples, including one example on how to add
your own image provider.

Since those images usually come with no password, requiring cloudinit,
we added to the avocado_qemu "add_image()" API the ability to set the
default user and its password attaching a cloudinit cdrom to the VM,
if you include the option "cloudinit=True". You can see an example
here[2].

In the example [2], there's no username/password options in the
add_image() call, so the defaults (avocado/avocado) will be assumed.
Btw, besides setting username/password as options in the add_image(),
they can also come from parameters, so you don't have to hard code
that information in in each test. See the "image_user"/"image_pass"
parameters in the README[3] file.

[1] - http://avocado-framework.readthedocs.io/en/57.0/utils/vmimage.html
[2] - 
https://github.com/apahim/qemu/blob/avocado_qemu/tests/avocado/test_nec-usb-xhci.py#L22-L23
[3] - https://github.com/apahim/qemu/blob/avocado_qemu/tests/avocado/README.rst

--
apahim

>
> Please let me know if you have more questions!
> - Cleber.
>
>> Alistair
>>
>>>
>>> Thanks!
>>>
>>> References
>>> ==========
>>>
>>>  [1] http://avocado-framework.github.io/
>>>  [2] http://autotest.github.io/
>>>  [3] https://github.com/autotest/virt-test
>>>  [4] https://github.com/avocado-framework/avocado-vt
>>>  [5] https://github.com/autotest/tp-qemu
>>>  [6] https://github.com/apahim/qemu
>>>  [7] https://github.com/apahim/qemu/tree/avocado_qemu
>>>  [8] https://github.com/apahim/qemu/tree/avocado_qemu_snapshot
>>>  [9]
>>> https://github.com/apahim/qemu/blob/avocado_qemu/tests/avocado/test_info_memdev_host_nodes.py
>>> [10]
>>> https://github.com/apahim/qemu/blob/avocado_qemu/tests/avocado/test_ovmf_with_240_vcpus.py
>>> [11]
>>> https://github.com/apahim/qemu/blob/avocado_qemu/tests/avocado/README.rst
>>> [12] http://avocado-framework.readthedocs.io/
>>>
>>> --
>>> Cleber Rosa
>>> [ Sr Software Engineer - Virtualization Team - Red Hat ]
>>> [ Avocado Test Framework - avocado-framework.github.io ]
>>> [  7ABB 96EB 8B46 B94D 5E0F  E9BB 657E 8D33 A5F2 09F3  ]
>>>
>
> --
> Cleber Rosa
> [ Sr Software Engineer - Virtualization Team - Red Hat ]
> [ Avocado Test Framework - avocado-framework.github.io ]
> [  7ABB 96EB 8B46 B94D 5E0F  E9BB 657E 8D33 A5F2 09F3  ]