Re: [PATCH-for-5.0 1/7] tests/acceptance/machine_sparc_leon3: Disable HelenOS test

[Philippe Mathieu-Daudé]

On 4/1/20 7:43 PM, Willian Rampazzo wrote:
> On Tue, Mar 31, 2020 at 5:07 PM Philippe Mathieu-Daudé
> <address@hidden> wrote:
> <snip>
> > First job failed by timeout, 2nd succeeded:
> > https://travis-ci.org/github/philmd/qemu/jobs/669265466
> >
> > However "Ran for 46 min 48 sec"
> >
> >   From the log:
> >
> > Fetching asset from
> > tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_mips64el_malta_5KEc_cpio
> > Fetching asset from
> > tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_mips64el_malta_5KEc_cpio
> > Fetching asset from
> > tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_orangepi
> > Fetching asset from
> > tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_orangepi_initrd
> > Fetching asset from
> > tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_orangepi_initrd
> > Fetching asset from
> > tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_orangepi_sd
> > Fetching asset from
> > tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_orangepi_sd
> > Fetching asset from
> > tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_orangepi_bionic
> > Fetching asset from
> > tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_orangepi_uboot_netbsd9
> > Fetching asset from
> > tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_orangepi_uboot_netbsd9
> > Fetching asset from
> > tests/acceptance/ppc_prep_40p.py:IbmPrep40pMachine.test_openbios_and_netbsd
> > ...
> >    (13/82)
> > tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_mips64el_malta_5KEc_cpio:
> >    SKIP: untrusted code
> >    (24/82)
> > tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_orangepi_bionic:
> >    SKIP: storage limited
> > ...
> >    (25/82)
> > tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_orangepi_uboot_netbsd9:
> >    SKIP: storage limited
> > ...
> >    (63/82)
> > tests/acceptance/ppc_prep_40p.py:IbmPrep40pMachine.test_openbios_and_netbsd:
> >    SKIP: Running on Travis-CI
> >
> > Is it possible that we are now fetching assets for tests we are not
> > running? In particular the one marked @skip because the downloading time
> > was too long on Travis?
>
> Yes, your assumption is correct, this execution of Avocado downloaded
> assets for tests that were skipped. Let me try to explain how the
> asset feature works today on Avocado.
>
> Avocado has two basic ways to work with assets:
>
> 1. Parse limited use cases of `fetch_asset` call in the test file and
> execute them. This operation can happen in two different scenarios.
> First, when using the command line `avocado assets fetch <test_file>`.

Odd, with avocado-framework==76.0 I get:

https://travis-ci.org/github/philmd/qemu/jobs/669851870#L4908

Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File 
"/home/travis/build/philmd/qemu/build/tests/venv/lib/python3.6zsite-packages/avocado/__main__.py", 
line 11, in <module>
    sys.exit(main.run())
  File 
"/home/travis/build/philmd/qemu/build/tests/venv/lib/python3.6/site-packages/avocado/core/app.py", 
line 91, in run
    return method(self.parser.config)
  File 
"/home/travis/build/philmd/qemu/build/tests/venv/lib/python3.6/site-packages/avocado/plugins/assets.py", 
line 291, in run
    success, fail = fetch_assets(test_file)
  File 
"/home/travis/build/philmd/qemu/build/tests/venv/lib/python3.6/site-packages/avocado/plugins/assets.py", 
line 200, in fetch_assets
    handler = FetchAssetHandler(test_file, klass, method)
  File 
"/home/travis/build/philmd/qemu/build/tests/venv/lib/python3.6/site-packages/avocado/plugins/assets.py", 
line 65, in __init__
    self.visit(self.tree)
  File "/usr/lib/python3.6/ast.py", line 253, in visit
    return visitor(node)
  File "/usr/lib/python3.6/ast.py", line 261, in generic_visit
    self.visit(item)
  File "/usr/lib/python3.6/ast.py", line 253, in visit
    return visitor(node)
  File 
"/home/travis/build/philmd/qemu/build/tests/venv/lib/python3.6/site-packages/avocado/plugins/assets.py", 
line 139, in visit_ClassDef
    self.generic_visit(node)
  File "/usr/lib/python3.6/ast.py", line 261, in generic_visit
    self.visit(item)
  File "/usr/lib/python3.6/ast.py", line 253, in visit
    return visitor(node)
  File 
"/home/travis/build/philmd/qemu/build/tests/venv/lib/python3.6/site-packages/avocado/plugins/assets.py", 
line 171, in visit_Assign
    self.asgmts[cur_klass][cur_method][name] = node.value.s
KeyError: 'launch_and_wait'
/home/travis/build/philmd/qemu/tests/Makefile.include:910: recipe for 
target 'fetch-acceptance-assets' failed

This launch_and_wait comes from:

tests/acceptance/boot_linux.py:88:    def launch_and_wait(self):

> In this case, it is a standalone execution of each fetch call and the
> test is not executed at all. Second, by running the test. The, enabled
> by default, plugin FetchAssetJob will do the same operation of parsing
> the test file and executing occurrences of `fetch_asset` call before
> the tests start to run. Again, the fetch time is not computed in the
> job time.
>
> 2. Execute the `fetch_asset` call from each test during the test
> execution. In this case, the FetchAssetJob plugin should be disabled.
> The fetch time is added to the job time as the asset download occurs
> during the test execution.
>
> The acceptance tests which make use of `fetch_asset` are all using the
> first method with FetchAssetJob plugin enabled. As Avocado is parsing
> the test file before it starts to run the tests, it is not aware of
> possible skips that may occur during a test execution due to possible
> dynamic dependency.
>
> This is not the desired behavior, as you mentioned, Avocado is
> downloading an asset that will not be used because its test will be
> skipped. To minimize the damage on the download side, the Travis job
> is holding the avocado cache. It means the download should happen just
> once. This does not minimize the damage to space usage.
>
> One possible workaround here is to temporarily disable the
> FetchAssetJob plugin, now that the needed assets are on Travis Avocado
> cache. The downside is that when an asset is not available in the
> cache, it will be downloaded during the test execution and the
> download time will be added to the job time. I don't know if it is
> possible to manually remove an asset from Travis Avocado cache. If so,
> this can be done for the tests that should be skipped because of space
> usage.
>
> We have been trying to make the Asset feature as flexible as possible
> to accommodate the use cases we have been identifying. Thanks for
> reporting this!
>
> > RESULTS    : PASS 65 | ERROR 0 | FAIL 0 | SKIP 14 | WARN 0 | INTERRUPT 0
> > | CANCEL 3
> > JOB TIME   : 1480.72 s
> >
> > Does this "JOB TIME" sums the 'Fetching asset' part?
>
> Answered in the comments above.

Thanks for the big clear explanation :)

Phil.