Re: [PATCH v2] tests/functional: Convert Aarch64 SBSA-Ref avocado tests

[Philippe Mathieu-Daudé]

On 22/8/24 16:06, Marcin Juszkiewicz wrote:
> On 22.08.2024 13:41, Philippe Mathieu-Daudé wrote:
> >   # Timeouts for individual tests that can be slow e.g. with debugging 
> > enabled
> >   test_timeouts = {
> > +  'aarch64_sbsaref' : 180,
>
> What kind of machine is able to run those tests in 180s? I bumped them 
> to 2400s and got timeout (Macbook with M1 Pro).

Indeed with Avocado this timeout was per test function, now it is
per test class. Having 11 test functions, for a 1-1 change we'd
need 180 * 11 = 1980 seconds.

I made a comment on Thomas/Daniel's series this patch is based on:
4b4018c6-4a2b-4250-bb53-be9cc5df7cb4@linaro.org/">https://lore.kernel.org/qemu-devel/4b4018c6-4a2b-4250-bb53-be9cc5df7cb4@linaro.org/

I could run all the tests in <300sec but for safety, should I keep
Avocado equivalent and use 2000s?

▶ 1/5 
test_aarch64_sbsaref.Aarch64SbsarefMachine.test_sbsaref_alpine_linux_max 
             OK
▶ 1/5 
test_aarch64_sbsaref.Aarch64SbsarefMachine.test_sbsaref_alpine_linux_max_pauth_impdef 
OK
▶ 1/5 
test_aarch64_sbsaref.Aarch64SbsarefMachine.test_sbsaref_alpine_linux_max_pauth_off 
   OK
▶ 1/5 
test_aarch64_sbsaref.Aarch64SbsarefMachine.test_sbsaref_alpine_linux_neoverse_n1 
     OK
▶ 1/5 
test_aarch64_sbsaref.Aarch64SbsarefMachine.test_sbsaref_edk2_firmware 
             OK
▶ 1/5 
test_aarch64_sbsaref.Aarch64SbsarefMachine.test_sbsaref_openbsd73_cortex_a57 
         OK
▶ 1/5 
test_aarch64_sbsaref.Aarch64SbsarefMachine.test_sbsaref_openbsd73_max 
             OK
▶ 1/5 
test_aarch64_sbsaref.Aarch64SbsarefMachine.test_sbsaref_openbsd73_max_pauth_impdef 
   OK
▶ 1/5 
test_aarch64_sbsaref.Aarch64SbsarefMachine.test_sbsaref_openbsd73_max_pauth_off 
      OK
▶ 1/5 
test_aarch64_sbsaref.Aarch64SbsarefMachine.test_sbsaref_openbsd73_neoverse_n1 
        OK
1/5 qemu:func-thorough+func-aarch64-thorough+thorough / 
func-aarch64-aarch64_sbsaref        OK             241.79s   11 subtests 
passed

> "make check-avocado" (with some AVOCADO_* vars to limit list of tests) 
> shown me which test is run and pass/fail for each.
>
> "make check-functional-aarch64 V=1" shows me "1/4 
> qemu:func-thorough+func-aarch64-thorough+thorough / 
> func-aarch64-aarch64_sbsaref" and timeouts without information which 
> tests pass, which fail.
>
> Maybe for QEMU project this is a progress. For me it is moving tests 
> from working ones to "sorry, timeout, find out why" ones.