Re: [PATCH] tests: improve performance of device-introspect-test

[Daniel P . Berrangé]

On Fri, Jul 10, 2020 at 10:03:56PM +0200, Markus Armbruster wrote:
> Daniel P. Berrangé <berrange@redhat.com> writes:
> 
> > Total execution time with "-m slow" and x86_64 QEMU, drops from 3
> > minutes 15 seconds, down to 54 seconds.
> >
> > Individual tests drop from 17-20 seconds, down to 3-4 seconds.
> 
> Nice!
> 
> A few observations on this test (impatient readers may skip to
> "Conclusions"):

snip

> * The number of known device types varies between targets from 33
>   (tricore) to several hundreds (x86_64+i386: 421, ppc 593, arm 667,
>   aarch64 680, ppc64 689).  Median is 215, sum is 7485.

snip

> * The test matrix is *expensive*.  Testing even a simple QMP query is
>   when you do it a quarter million times.  ARM is the greediest pig by
>   far (170k introspections, almost two thirds of the total!), followed
>   by ppc (36k), x86 (12k) and mips (11k).  Ideas on trimming excess are
>   welcome.  I'm not even sure anymore this should be a qtest.

We have 70 arm machines, 667 devices. IIUC we are roughly testing every
device against everything machine. 46,690 tests.

Most of the time devices are going to behave identically regardless of
which machine type is used. The trouble is some machines are different
enough that they can genuinely trigger different behaviour. It isn't
possible to slim the (machine, device) expansion down programatically
while still exercising the interesting combinations unless we get alot
more advanced.

eg if a have a PCI device, we only need test it in one PCI based machine,
and only need test it on one non-PCI based machine.

I would be interesting to actually get some CPU profiling data for
this test to see if it points out anything interesting about where the
time is being spent. Even if we don't reduce the complexity, reducing
a time factor will potentially greatly help. 


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|