Re: [Qemu-devel] [PATCH v2 3/3] Travis support for the acceptance tests

[Cleber Rosa]

On 10/9/18 9:46 AM, Philippe Mathieu-Daudé wrote:
> Hi Cleber,
> 
> On 09/10/2018 06:18, Cleber Rosa wrote:
>> This enables the execution of the acceptance tests on Travis.
> 
> Did you test this? =)
> 

I did have some jobs on Travis that looked promising.  But yeah, later I
found some issues. :/

>>
>> Because the Travis environment is based on Ubuntu Trusty, it requires
>> the python3-pip.
>>
>> Note: while another supposedely required component on newer versions
>> (such as on Bionic) split the Python 3 installation further on the
>> python3-venv package.
>>
>> Signed-off-by: Cleber Rosa <address@hidden>
>> ---
>>  .travis.yml | 6 ++++++
>>  1 file changed, 6 insertions(+)
>>
>> diff --git a/.travis.yml b/.travis.yml
>> index 95be6ec59f..db1a31ea51 100644
>> --- a/.travis.yml
>> +++ b/.travis.yml
>> @@ -36,6 +36,7 @@ addons:
>>        - liburcu-dev
>>        - libusb-1.0-0-dev
>>        - libvte-2.90-dev
>> +      - python3-pip
>>        - sparse
>>        - uuid-dev
>>        - gcovr
>> @@ -117,6 +118,11 @@ matrix:
>>      - env: CONFIG="--target-list=x86_64-softmmu"
>>        python:
>>          - "3.6"
>> +    # Acceptance (Functional) tests
>> +    - env: CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu"
>> +           TEST_CMD="make check-acceptance"
>> +      python:
>> +        - "3.6"
>>      # Using newer GCC with sanitizers
>>      - addons:
>>          apt:
>>
> 
> Using the following patch:
> 
> -- >8 --
> diff --git a/.travis.yml b/.travis.yml
> index 95be6ec59f..87e0c9a13f 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -118,4 +118,15 @@ matrix:
>        python:
>          - "3.6"
> +    # Acceptance (Functional) tests
> +    - env: CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu"
> +           TEST_CMD="make check-acceptance"
> +           # sudo rm /usr/local/bin/pip*
> +      python:
> +        - "3.6"
> +      addons:
> +        apt:
> +          packages:
> +            - python3-pip
> +            - python3.4-venv

This patch revealed to me that, even though we're asking Travis for a
Python 3.6 stack, we're manually pointing to the primary Python (3.4)
installation.  This is in itself a source of problems.

Read on...

>      # Using newer GCC with sanitizers
>      - addons:
> ---
> 
> I got some improvements until:
> 
>   VENV    /home/travis/build/philmd/qemu/tests/venv
>   MKDIR   /home/travis/build/philmd/qemu/tests/results
>   PIP     /home/travis/build/philmd/qemu/tests/venv-requirements.txt
> Exception:
> Traceback (most recent call last):
>   File "/usr/lib/python3/dist-packages/pip/basecommand.py", line 122, in
> main
>     status = self.run(options, args)
>   File "/usr/lib/python3/dist-packages/pip/commands/install.py", line
> 283, in run
>     requirement_set.install(install_options, global_options,
> root=options.root_path)
>   File "/usr/lib/python3/dist-packages/pip/req.py", line 1436, in install
>     requirement.install(install_options, global_options, *args, **kwargs)
>   File "/usr/lib/python3/dist-packages/pip/req.py", line 672, in install
>     self.move_wheel_files(self.source_dir, root=root)
>   File "/usr/lib/python3/dist-packages/pip/req.py", line 902, in
> move_wheel_files
>     pycompile=self.pycompile,
>   File "/usr/lib/python3/dist-packages/pip/wheel.py", line 206, in
> move_wheel_files
>     clobber(source, lib_dir, True)
>   File "/usr/lib/python3/dist-packages/pip/wheel.py", line 193, in clobber
>     os.makedirs(destsubdir)
>   File "/usr/lib/python3.4/os.py", line 237, in makedirs
>     mkdir(name, mode)
> PermissionError: [Errno 13] Permission denied:
> '/usr/local/lib/python3.4/dist-packages/avocado'
> 

This looks to me like it's caused by the use of the system wide Python
3.4 installation, as mentioned above.  That, and possibly the
"--system-wide-packages" flag to the venv.

So, in order to understand if, in those environments, we should let
"configure" pick up the right Python, I did the following experiment:

 $ python3 -m venv /tmp/py3-build
 $ . /tmp/py3-build/bin/activate
 $ which python
 /tmp/py3-build/bin/python
 $ /tmp/py3-build/bin/python --version
 Python 3.6.6

This shows that when a venv is activated, the "right" Python should be
available in the $PATH.  So, building QEMU without pointint to a
specific Python binary (when inside a venv) gives you:

 $ mkdir -p /tmp/qemu-build
 $ cd /tmp/qemu-build
 $ ~/src/qemu/configure --target-list="x86_64-softmmu"
 $ make print-PYTHON
 PYTHON=python -B
 $ which python
 /tmp/py3-build/bin/python

The whole question now is how/where Travis puts the requested Python
version (python: - "3.6").  Is it on a venv?  Is it on a specific
location? Is that reliable enough?

I'm going to look for those answers now.

- Cleber.

> See: https://travis-ci.org/philmd/qemu/jobs/439138706
>