Avocado notes from KVM forum 2019

[Philippe Mathieu-Daudé]

Hi Cleber,

Here are my notes from talking about Avocado with various people during 
the KVM forum in Lyon last month.

All comments are QEMU oriented.


1) Working offline

Various people complained Avocado requires online access, and they would 
like to use it offline.

  Maintainer workflow example is:

  - run avocado
  - hack QEMU, build
  - git pull
  - build
  - hack QEMU
  (go offline)
  - hack QEMU
  - build
  - run avocado <- FAILS

Failure is because mainstream added new tests, which got pulled in, and 
the user only notice when running avocado again, but offline.
Test example is boot_linux_console.py, which added various tests from 
other subsystems, so the maintainer has to disable the new tests 
manually to be able to run his previous tests.

Expected solution: skip tests when artifact is not available, eventually 
when the --offline option is used


2) Add artifacts manually to the cache

Not all artifacts can be easily downloadable, some are public but 
require the user to accept an End User License Agreement.
Users would like to share their tests with the documentation about 
where/how to download the requisite files (accepting the EULA) to run 
the tests.


2b) Add reference to artifact to the cache

Group of users might share group of files (like NFS storage) and would 
like to use directly their remote read-only files, instead of copying it 
to their home directory.


3) Provide qemu/avocado-qemu Python packages

The mainstream project uses Avocado to test QEMU. Others projects use 
QEMU to test their code, and would like to automatize that using 
Avocado. They usually not rebuild QEMU but use a stable binary from 
distributions. The python classes are not available, so they have to 
clone QEMU to use Avocado (I guess they only need 5 python files).
When running on Continuous Integration, this is overkill, because when 
you clone QEMU you also clone various other submodules.


4) Warn the user when Avocado is too old for the tests

Some users tried Avocado following the examples on the mailing list and 
the one in some commit's descriptions where we simply show "avocado run 
...". They installed the distribution Avocado package and tried and it 
fails for few of them with no obvious reason (the .log file is hard to 
read when you are not custom to). IIUC their distribution provides a 
older Avocado (69?) while we use recent features (72).

We never noticed it because we use 'make check-venv' and do not test the 
distrib Avocado. While we can not test all distribs, we could add a 
version test if the Avocado version is too old, display a friendly 
message to the console (not the logfile).


That it for my notes.

Eduardo/Wainer, are there other topics I forgot?


Regards,

Phil.