[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Qemu-devel] [PULL 12/15] Acceptance tests: add make rule for runnin
Re: [Qemu-devel] [PULL 12/15] Acceptance tests: add make rule for running them
Wed, 7 Nov 2018 00:24:55 +0100
Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.0
On 31/10/2018 01:31, Eduardo Habkost wrote:
> From: Cleber Rosa <address@hidden>
> The acceptance (aka functional, aka Avocado-based) tests are
> Python files located in "tests/acceptance" that need to be run
> with the Avocado libs and test runner.
> Let's provide a convenient way for QEMU developers to run them,
> by making use of the tests-venv with the required setup.
> Also, while the Avocado test runner will take care of creating a
> location to save test results to, it was understood that it's better
> if the results are kept within the build tree.
> Signed-off-by: Cleber Rosa <address@hidden>
> Acked-by: Stefan Hajnoczi <address@hidden>
> Acked-by: Wainer dos Santos Moschetta <address@hidden>
> Reviewed-by: Caio Carrara <address@hidden>
> Message-Id: <address@hidden>
> Signed-off-by: Eduardo Habkost <address@hidden>
> docs/devel/testing.rst | 43 +++++++++++++++++++++++++++++++++++++-----
> tests/requirements.txt | 1 +
> tests/Makefile.include | 21 +++++++++++++++++++--
> 3 files changed, 58 insertions(+), 7 deletions(-)
> diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
> index a227754f86..18e2c0868a 100644
> --- a/docs/devel/testing.rst
> +++ b/docs/devel/testing.rst
> @@ -545,10 +545,39 @@ Tests based on ``avocado_qemu.Test`` can easily:
> +Running tests
> +You can run the acceptance tests simply by executing:
> +.. code::
> + make check-acceptance
> +This involves the automatic creation of Python virtual environment
> +within the build tree (at ``tests/venv``) which will have all the
> +right dependencies, and will save tests results also within the
> +build tree (at ``tests/results``).
> -To install Avocado and its dependencies, run:
> +Note: the build environment must be using a Python 3 stack, and have
> +the ``venv`` and ``pip`` packages installed. If necessary, make sure
> +``configure`` is called with ``--python=`` and that those modules are
> +available. On Debian and Ubuntu based systems, depending on the
> +specific version, they may be on packages named ``python3-venv`` and
> +The scripts installed inside the virtual environment may be used
> +without an "activation". For instance, the Avocado test runner
> +may be invoked by running:
> + .. code::
> + tests/venv/bin/avocado run $OPTION1 $OPTION2 tests/acceptance/
> +Manual Installation
> +To manually install Avocado and its dependencies, run:
> .. code::
> @@ -689,11 +718,15 @@ The exact QEMU binary to be used on QEMUMachine.
> Uninstalling Avocado
> -If you've followed the installation instructions above, you can easily
> -uninstall Avocado. Start by listing the packages you have installed::
> +If you've followed the manual installation instructions above, you can
> +easily uninstall Avocado. Start by listing the packages you have
> pip list --user
> And remove any package you want with::
> pip uninstall <package_name>
> +If you've used ``make check-acceptance``, the Python virtual environment
> +Avocado is installed will be cleaned up as part of ``make check-clean``.
> diff --git a/tests/requirements.txt b/tests/requirements.txt
> index d39f9d1576..64c6e27a94 100644
> --- a/tests/requirements.txt
> +++ b/tests/requirements.txt
> @@ -1,3 +1,4 @@
> # Add Python module requirements, one per line, to be installed
> # in the tests/venv Python virtual environment. For more info,
> # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
> diff --git a/tests/Makefile.include b/tests/Makefile.include
> index eabc1da2f3..d2e577eabb 100644
> --- a/tests/Makefile.include
> +++ b/tests/Makefile.include
> @@ -11,6 +11,7 @@ check-help:
> @echo " $(MAKE) check-qapi-schema Run QAPI schema tests"
> @echo " $(MAKE) check-block Run block tests"
> @echo " $(MAKE) check-tcg Run TCG tests"
> + @echo " $(MAKE) check-acceptance Run all acceptance (functional)
> @echo " $(MAKE) check-report.html Generates an HTML test report"
> @echo " $(MAKE) check-venv Creates a Python venv for tests"
> @echo " $(MAKE) check-clean Clean the tests"
> @@ -902,10 +903,15 @@ check-decodetree:
> # Python venv for running tests
> -.PHONY: check-venv
> +.PHONY: check-venv check-acceptance
> +# Controls the output generated by Avocado when running tests.
> +# Any number of command separated loggers are accepted. For more
> +# information please refer to "avocado --help".
> $(shell $(PYTHON) -c 'import sys; assert sys.version_info >= (3,0)'
> >/dev/null 2>&1)
> ifeq ($(.SHELLSTATUS),0)
> @@ -922,8 +928,19 @@ $(TESTS_VENV_DIR):
> $(error "venv directory for tests requires Python 3")
> + $(call quiet-command, mkdir -p $@, \
> + MKDIR, $@)
> check-venv: $(TESTS_VENV_DIR)
> +check-acceptance: check-venv $(TESTS_RESULTS_DIR)
> + $(call quiet-command, \
> + $(TESTS_VENV_DIR)/bin/python -m avocado \
> + --show=$(AVOCADO_SHOW) run
> --job-results-dir=$(TESTS_RESULTS_DIR) \
> + --failfast=on $(SRC_PATH)/tests/acceptance, \
> + "AVOCADO", "tests/acceptance")
Right now it seems like anything that is target-dependent should be
developed under qtest, rather than as an acceptance test. On the other
hand this results in qtests that actually do not use the qtest protocol
at all---only the libqtest function to interact with QMP etc.---and as
such would look like a perfect match for "make check-acceptance".
drive_del-test is already an example of this.
How hard would it be to make it run tests for all supported targets,
rather than just the host architecture? Perhaps using a tag such as
> # Consolidated targets
> .PHONY: check-qapi-schema check-qtest check-unit check check-clean
> @@ -937,7 +954,7 @@ check-clean:
> rm -rf $(check-unit-y) tests/*.o $(QEMU_IOTESTS_HELPERS-y)
> rm -rf $(sort $(foreach target,$(SYSEMU_TARGET_LIST),
> $(check-qtest-$(target)-y)) $(check-qtest-generic-y))
> rm -f tests/test-qapi-gen-timestamp
> - rm -rf $(TESTS_VENV_DIR)
> + rm -rf $(TESTS_VENV_DIR) $(TESTS_RESULTS_DIR)
> clean: check-clean
- Re: [Qemu-devel] [PULL 12/15] Acceptance tests: add make rule for running them,
Paolo Bonzini <=