Re: [Qemu-devel] [PATCH 0/5] Acceptance/functional tests

[Cleber Rosa]

On 05/25/2018 02:08 AM, Fam Zheng wrote:
> On Thu, 05/24 20:58, Cleber Rosa wrote:
>> TL;DR
>> =====
>>
>> Another version, with a minimalist approach, to the acceptance tests
>> infrastructure for QEMU, based on the Avocado Testing Framework.
>>
>> Background
>> ==========
>>
>> The previous version, still considered an RFC, was sent to the list by
>> Eduardo[1] was based on the work held in Amador's branch[2].  After
>> reviewing in under a different light, including the experiences
>> done and reported by Philippe[3].
>>
>> Differences from previous versions
>> ==================================
>>
>> The main difference is that this series include only the minimal
>> changes deemed necessary to have a starting point.  I like to think
>> that it's better connected to the QEMU community and project needs,
>> and will hopefully allow for a more organic growth.
>>
>> Since this version has less features than the previous versions,
>> provided it's accepted, these are the next probable development tasks:
>>
>>  * Provide a simple variants mechanism to allow the same tests to be
>>    run under different targets, machine models and devices (present on
>>    the previous versions as a "YAML to Mux" file with architecture
>>    definitions)
>>  * Implement QEMUMachine migration support (present on the previous
>>    version in the "avocado_qemu.test._VM" class)
>>  * Implement Guest OS image selection and download (mostly an Avocado
>>    feature, paired with a parameter convention and cloud-init support
>>    code)
>>  * Implement interactive support for Guest OS sessions (present on
>>    the previous versions, supported by the aexpect Python module)
>>
>> Even though this version shares very little (if any) code with the
>> previous versions, the following is a list of noteworthy changes:
>>
>>  * Tests directory is now "tests/acceptance" (was "tests/avocado")
>>  * Base test class is now "avocado_qemu.Test" (was
>>    "avocado_qemu.test.QemuTest")
>>  * Base test class is now hosted in "avocado_qemu/__init__.py" (was
>>    "avocado_qemu/test.py")
>>  * Direct use of "qemu.QEMUMachine", that is, the
>>    avocado_qemu.test._VM class is gone
>>  * avocado_qemu.Test won't search for QEMU binaries on $PATH.  To use
>>    QEMU binary on a custom system location it's necessary to use the
>>    "qemu_bin" parameter
>>  * Example test in README.rst is distributed as a real test
>>    ("test_version.py")
>>  * A new "Linux boot console" test, loosely modeled after Phillipe's
>>    use case
> 
> I like the direction this series is leading us to.  Making it easy to write
> 'correct' and 'effective' tests is the most important thing IMO, and the few
> tests you add here look promising.
> 

The soft targets that these adjectives ("easy", "correct" and
"effective") bring, are kind of hard to get right, but I'm really glad
you like what you see.

> I'm also curious to see a test case involving a guest OS image. (I'm also 
> open to
> a re-implementation of tests/vm/basevm.py with Avocado, if you find it
> worthwhile.)
> 

We had such tests in the previous incarnations of this work[1].  But by
doing it in smaller steps, I believe we can achieve a better result,
specially when it comes to the exposed API.

- Cleber.

[1] https://lists.gnu.org/archive/html/qemu-devel/2018-04/msg03466.html

> Fam
> 
>>
>> Commit summary
>> ==============
>>
>> Cleber Rosa (5):
>>   Add functional/acceptance tests infrastructure
>>   scripts/qemu.py: allow adding to the list of extra arguments
>>   Acceptance tests: add quick VNC tests
>>   scripts/qemu.py: introduce set_console() method
>>   Acceptance tests: add Linux kernel boot and console checking test
>>
>>  scripts/qemu.py                             | 103 +++++++++++++++-
>>  scripts/test_qemu.py                        | 176 
>> +++++++++++++++++++++++++++
>>  tests/acceptance/README.rst                 | 141 +++++++++++++++++++++
>>  tests/acceptance/avocado_qemu/__init__.py   |  45 +++++++
>>  tests/acceptance/test_boot_linux_console.py |  37 ++++++
>>  tests/acceptance/test_version.py            |  13 ++
>>  tests/acceptance/test_vnc.py                |  50 ++++++++
>>  7 files changed, 564 insertions(+), 1 deletion(-)
>>  create mode 100644 scripts/test_qemu.py
>>  create mode 100644 tests/acceptance/README.rst
>>  create mode 100644 tests/acceptance/avocado_qemu/__init__.py
>>  create mode 100644 tests/acceptance/test_boot_linux_console.py
>>  create mode 100644 tests/acceptance/test_version.py
>>  create mode 100644 tests/acceptance/test_vnc.py
>>
>> ---
>>
>> [1] https://lists.gnu.org/archive/html/qemu-devel/2018-04/msg03443.html
>> [2] https://github.com/apahim/qemu/commits/avocado_qemu
>> [3] https://lists.gnu.org/archive/html/qemu-devel/2018-04/msg03076.html
>>
>