Re: [Qemu-devel] [PATCH v4 0/3] Bootstrap Python venv and acceptance/functional tests

[Philippe Mathieu-Daudé]

On 12/10/2018 18:53, Cleber Rosa wrote:
> TL;DR
> =====
> 
> Allow acceptance tests to be run with `make check-acceptance`.
> 
> Details
> =======
> 
> This introduces a Python virtual environment that will be setup within
> the QEMU build directory, that will contain the exact environment that
> tests may require.
> 
> There's one current caveat: it requires Python 3, as it's based on the
> venv module.  This was based on some discussions and perception about
> standardizing on Python 3, but can easily be made to accommodate Python
> 2 as well.
> 
> Example of bootstrap and test execution on Travis-CI:
> 
> https://travis-ci.org/qemu/qemu/jobs/439331028#L2508

If you activate Travis on your github account, you can test that in your
namespace without having to open zombie pull requests there... A simple
push to your repository will trigger a full Travis build.

This is how I use it btw, canceling the jobs I'm not interested in, to
quickly run the others.
i.e. https://travis-ci.org/philmd/qemu/jobs/439573299#L5600

> 
>    ...
>       VENV    /home/travis/build/qemu/qemu/tests/venv
>       MKDIR   /home/travis/build/qemu/qemu/tests/results
>       PIP     /home/travis/build/qemu/qemu/tests/venv-requirements.txt
>       AVOCADO tests/acceptance
>     JOB ID     : 920e4fcf55a1782f1ae77bee64b20ccdc2e1111d
>     JOB LOG    : 
> /home/travis/build/qemu/qemu/tests/results/job-2018-10-09T21.42-920e4fc/job.log
>      (1/6) 
> /home/travis/build/qemu/qemu/tests/acceptance/boot_linux_console.py:BootLinuxConsole.test:
>   PASS (3.57 s)
>      (2/6) 
> /home/travis/build/qemu/qemu/tests/acceptance/version.py:Version.test_qmp_human_info_version:
>   PASS (0.04 s)
>      (3/6) 
> /home/travis/build/qemu/qemu/tests/acceptance/vnc.py:Vnc.test_no_vnc:  PASS 
> (0.04 s)
>      (4/6) 
> /home/travis/build/qemu/qemu/tests/acceptance/vnc.py:Vnc.test_no_vnc_change_password:
>   PASS (0.04 s)
>      (5/6) 
> /home/travis/build/qemu/qemu/tests/acceptance/vnc.py:Vnc.test_vnc_change_password_requires_a_password:
>   PASS (0.04 s)
>      (6/6) 
> /home/travis/build/qemu/qemu/tests/acceptance/vnc.py:Vnc.test_vnc_change_password:
>   PASS (0.04 s)
>     RESULTS    : PASS 6 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | 
> CANCEL 0
>     JOB TIME   : 3.90 s
>    ...
> 
> Changes from v3:
> ================
> 
>  * Fixed typo in commit message (s/requiment/requirement/).  (Eric)
> 
> Changes from v2:
> ================
> 
>  * Make the $(TESTS_VENV_DIR) target depend on the
>    venv-requirements.txt file, and touch $(TESTS_VENV_DIR) after venv
>    runs.  With this, updates on the file are reflected on the
>    venv. (Philippe)
> 
>  * Run pip with "python -m pip".  It may have been installed reusing
>    the system wide packages, and then the script may not be available
>    on the venv. (Philippe)
> 
>  * Dropped Python version on Travis, and using the version supplied
>    by the distro (3.4). (Philippe)
> 
>  * Added "python3.4-venv" package requirement on Travis. (Philippe)
> 
>  * Added variable (AVOCADO_SHOW) with logging streams to be shown
>    while running the acceptance tests.  By default it's set to none,
>    the equivalent of the quiet mode used on previous versions.
>    (Philippe)
> 
>  * On Travis, set the AVOCADO_SHOW variable to "app", so that the
>    individual test results can be easily seen.  (Philippe)
> 
> Ideas discussed, but not implemented:
> 
>   * Run pip with "$(PYTHON) -m pip -q install ..." because it points
>     to the system wide Python installation. (Philippe)
> 
>   * Drop the "--system-site-packages" flag.  Waiting on another round
>     of tests to determine if they are really the cause of some package
>     installation problems.
> 
> Changes from v1:
> ================
> 
>  * TESTS_VENV_REQ (the path of "venv-requirements.txt") now points to
>    the source path ($SRC_PATH instead of $BUILD_DIR)
> 
>  * Create the venv with "--system-site-packages", which allows the
>    reuse of packages (and no additional downloads) in case there's a
>    package installed system wide providing the same package and
>    version.
> 
>  * Run Avocado with "python -m avocado".  It may have been installed
>    reusing the system wide packages, and then the script may not
>    be available on the venv.
> 
>  * Improved documentation describing the Python 3, venv and pip
>    requirements.
> 
>  * Updated avocado-framework requirement to latest released version
>    (65.0)
> 
>  * (New commit) Added support for running the acceptance tests on
>    Travis.
> 
> Ideas discussed, but not implemented:
> 
>  * Install external packages such as python3-pip on Debian based
>    systems, deemed too invasive on developer's systems.
> 
>  * Allow the use of Python 2, and consequently the "virtualenv"
>    module.
> 
> Cleber Rosa (3):
>   Bootstrap Python venv for tests
>   Acceptance tests: add make rule for running them
>   Travis support for the acceptance tests
> 
>  .travis.yml                 |  5 +++++
>  docs/devel/testing.rst      | 35 ++++++++++++++++++++++++++++++-----
>  tests/Makefile.include      | 37 +++++++++++++++++++++++++++++++++++++
>  tests/venv-requirements.txt |  4 ++++
>  4 files changed, 76 insertions(+), 5 deletions(-)
>  create mode 100644 tests/venv-requirements.txt
>