[Qemu-devel] [PATCH v4 0/5] Acceptance/functional tests

[Cleber Rosa]

TL;DR
=====

Another version, with a minimalist approach, of the acceptance tests
infrastructure for QEMU, based on the Avocado Testing Framework.

Background
==========

The previous version, still considered an RFC, sent to the list by
Eduardo[1] was based on the work held in Amador's branch[2].  After
reviewing it under a different light, including the experiences
done and reported by Philippe[3], it was clear to me that a different
approach would be better.

Differences from previous versions
==================================

The main difference is that this series include only the minimal
changes deemed necessary to have a starting point.  I like to think
that it's better connected to the QEMU community and project needs,
and will hopefully allow for a more organic growth.

Since this version has less features than the previous versions,
provided it's accepted, these are the next probable development tasks:

 * Provide a simple variants mechanism to allow the same tests to be
   run under different targets, machine models and devices (present on
   the previous versions as a "YAML to Mux" file with architecture
   definitions)
 * Implement QEMUMachine migration support (present on the previous
   version in the "avocado_qemu.test._VM" class)
 * Implement Guest OS image selection and download (mostly an Avocado
   feature, paired with a parameter convention and cloud-init support
   code)
 * Implement interactive support for Guest OS sessions (present on
   the previous versions, supported by the aexpect Python module)
 * Move the Python modules that are used on tests to a more proper
   "module" organization.  This should allow for a proper place
   to hold tests for the Python modules themselves.

Even though this version shares very little (if any) code with the
previous versions, the following is a list of noteworthy changes:

 * Tests directory is now "tests/acceptance" (was "tests/avocado")
 * Base test class is now "avocado_qemu.Test" (was
   "avocado_qemu.test.QemuTest")
 * Base test class is now hosted in "avocado_qemu/__init__.py" (was
   "avocado_qemu/test.py")
 * Direct use of "qemu.QEMUMachine", that is, the
   avocado_qemu.test._VM class is gone
 * avocado_qemu.Test won't search for QEMU binaries on $PATH.  To use
   QEMU binary on a custom system location it's necessary to use the
   "qemu_bin" parameter
 * Example test in README.rst is distributed as a real test
   ("test_version.py")
 * A new "Linux boot console" test, loosely modeled after Phillipe's
   use case

Changes
=======

>From v3:
--------

 * Gave more information on the meaning/intended use of acceptance
   tests
 * Added a clearer example on running tests with tags
 * Added more information on what a QEMUMachine is

>From v2
-------

 * Renamed acceptance test files, dropping "test_" prefix.
 * Dropped "scripts/test_qemu.py" until we have a proper Python module
   location on which to put their own tests.

>From v1
-------

 * Moved documentation to docs/devel/testing.rst
 * Changed code markers from "::" to ".. code::" in the documentation
   for consistency
 * Added copyright notices
 * Removed useless "qemu_bin=None" attribution on avocado_qemu.Test.setUp()
 * Removed reference about setting arguments, instead of appending them,
   on "Acceptance tests: add quick VNC tests" commit message
 * Fixed typo (s/inteligence/intelligence/) on "scripts/qemu.py:
   introduce set_console() method" commit message
 * Removed useless "return" in add_args()

Commit summary
==============

Cleber Rosa (5):
  Add functional/acceptance tests infrastructure
  scripts/qemu.py: allow adding to the list of extra arguments
  Acceptance tests: add quick VNC tests
  scripts/qemu.py: introduce set_console() method
  Acceptance tests: add Linux kernel boot and console checking test

 docs/devel/testing.rst                    | 192 +++++++++++++++++++++++++++++
 scripts/qemu.py                           | 103 +++++++++++++++-
 tests/acceptance/README.rst               |  10 ++
 tests/acceptance/avocado_qemu/__init__.py |  54 ++++++++
 tests/acceptance/boot_linux_console.py    |  47 +++++++
 tests/acceptance/version.py               |  24 ++++
 tests/acceptance/vnc.py                   |  60 +++++++++
 7 files changed, 489 insertions(+), 1 deletion(-)
 create mode 100644 tests/acceptance/README.rst
 create mode 100644 tests/acceptance/avocado_qemu/__init__.py
 create mode 100644 tests/acceptance/boot_linux_console.py
 create mode 100644 tests/acceptance/version.py
 create mode 100644 tests/acceptance/vnc.py

---

[1] https://lists.gnu.org/archive/html/qemu-devel/2018-04/msg03443.html
[2] https://github.com/apahim/qemu/commits/avocado_qemu
[3] https://lists.gnu.org/archive/html/qemu-devel/2018-04/msg03076.html