Re: "make check-acceptance" takes way too long

[Daniel P . Berrangé]

On Tue, Feb 01, 2022 at 12:29:56AM -0500, Cleber Rosa wrote:
> 
> Assuming this is about "Testing that QEMU can boot a full distro", I wouldn't
> try to solve the problem by making the distro too slim to get to the
> point of becoming
> an unrealistic system.

At a high level our with acceptance (integration) testing is of
course to make sure that QEMU is correctly emulating a full virtual
machine, such that we have confidence that it can run real world
operating systems.

There are a number of approaches to achieve that with varying
tradeoffs.

  - Testing with very specific tailored environments, running
    very specific userspace tools and minimal kernel setup.

    This can give us a pretty decent amount of coverage of
    the core features of the emulated environment in a tightly
    controlled amount of wallclock time. When it fails it ought
    to be relatively easy to understand and debug.

    The downside is that it is the QEMU code paths it hits are
    going to be fairly static.


  - Testing with arbitrary execution of real world OS images.

    I think of this as a bit of scattergun approach. We're not
    trying to tightly control what runs, we actually want it
    to run alot of arbitrarily complex and unusual stuff.

    This is going to be time consuming and is likely to have
    higher false positive failure rates. It is worthwhile
    because it is going to find the edge cases that you simply
    won't detect any other way, because you can't even imagine
    the problems that you're trying to uncover until you uncover
    them by accident with a real OS workload.

    It is kinda like fuzzing QEMU with an entire OS :-)


Both of these approaches are valid/complementary and we should
want to have both.

Any test suite is only going to find bugs though if it is
actually executed.

As a contributor though the former is stuff I'm likely to be
willing to run myself before sending patches, while the latter
is stuff I'm just always going to punt to merge testing infra.

We want to be wary of leaving too much to be caught at time
of merge tests, because that puts a significant burden on the
person responsible for merging code in QEMU.  We need our
contributors to be motivated to run as much testing as possible
ahead of submitting patches.

> IMO the deal breaker with regards to test time can be solved more cheaply by
> having and using KVM where these tests will run, and not running them by
> default otherwise.  With the tagging mechanism we should be able to set a
> condition such as: "If using TCG, exclude tests that boot a full blown distro.
> If using KVM, do not criticize what gets booted".  Resulting in something
> like:

> Does that sound like something appropriate?

Depends whether you only care about KVM or not. From a POV of QEMU
community CI, I think it is valid to want to test TCG functionality


> BTW, on the topic of "Using something as a base OS for scripts (tests) to run
> on it", another possibility for using full blown OS would be to save
> their initialized
> state, and load it to memory for each test, saving the guest boot time.  This
> should of course be done at the framework level and transparent to tests.

There is *massive* virtue in simplicity & predictability for testing.

Building more complex infrastructure to pre-initialize caches with
clever techniques like saving running OS state is clever, but is
certainly not simple or predictable. When that kind of stuff goes
wrong, whoever gets to debug it is going to have a really bad day.

This can be worth doing if there's no other viable approach to achieve
the desired end goal. I don't think that's the case for our integration
testing needs in QEMU though. There's masses of scope for us to explore
testing with minimal tailored guest images/environments, before we need
to resort to building more complex optimization strategies.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|