Re: "make check-acceptance" takes way too long

[Cleber Rosa]

On Sat, Jul 31, 2021 at 4:33 PM Peter Maydell <peter.maydell@linaro.org> wrote:
>
> On Sat, 31 Jul 2021 at 19:43, Alex Bennée <alex.bennee@linaro.org> wrote:
> >
> >
> > Peter Maydell <peter.maydell@linaro.org> writes:
> >
> > > "make check-acceptance" takes way way too long. I just did a run
> > > on an arm-and-aarch64-targets-only debug build and it took over
> > > half an hour, and this despite it skipping or cancelling 26 out
> > > of 58 tests!
> > >
> > > I think that ~10 minutes runtime is reasonable. 30 is not;
> > > ideally no individual test would take more than a minute or so.
> > >
> > > Output saying where the time went. The first two tests take
> > > more than 10 minutes *each*. I think a good start would be to find
> > > a way of testing what they're testing that is less heavyweight.
> > >
> > >  (01/58) 
> > > tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg_gicv2:
> > > PASS (629.74 s)
> > >  (02/58) 
> > > tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg_gicv3:
> > > PASS (628.75 s)
> > >  (03/58) tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_kvm:
> > > CANCEL: kvm accelerator does not seem to be available (1.18 s)
> >
> > For these tests which purport to exercise the various GIC configurations
> > I think we would be much better served by running kvm-unit-tests which
> > at least try and exercise all the features rather than rely on the side
> > effect of booting an entire OS.
>
> I think "can we boot Linux via UEFI?" is worth testing, as is
> "can we boot Linux and do at least some stuff in userspace?"
> (there's a lot of TCG that doesn't get exercised by pure kernel boot).
> We just need to find a guest OS that isn't so overweight it takes 10
> minutes...
>
> -- PMM
>

I think using alternative guests is absolutely the way to go here.  I
had that in mind in the past, so much that I made sure to include
cirros[1] as one of the supported images[2] in avocado.utils.vmimage
(used in these tests above).  These tests are based on the LinuxTest
class[3], and they support the distro[4] and distro_version[5]
parameters.

But, cirros doesn't ship with a fully capable cloud-init package and I
deferred to support it in avocado.utils.cloudinit, and thus, support
cirrus in those tests.  I gave that idea another try, and the results
are encouraging, with reduction of runtime by almost a factor of 6.
On my system I get:

$ avocado run -p distro=fedora -p distro_version=31
tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg_gicv3
 (1/1) tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg_gicv3:
PASS (165.48 s)

And with cirros:

$ avocado run -p distro=cirros -p distro_version=0.5.2
tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg_gicv3
(1/1) tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg_gicv3:
PASS (28.80 s)

I'll work on posting the bits needed to have this working out of the
box, but it'll require new code on the Avocado side too (tentative to
version 91.0).

Regards,
- Cleber.

[1] https://github.com/cirros-dev/cirros
[2] 
https://avocado-framework.readthedocs.io/en/90.0/guides/writer/libs/vmimage.html#supported-images
[3] 
https://qemu-project.gitlab.io/qemu/devel/testing.html#the-avocado-qemu-linuxtest-base-test-class
[4] https://qemu-project.gitlab.io/qemu/devel/testing.html#distro
[5] https://qemu-project.gitlab.io/qemu/devel/testing.html#distro-version