Re: [Qemu-devel] [PATCH 0/3] script for crash-testing -device

[Marcel Apfelbaum]

On 03/23/2017 05:43 PM, Thomas Huth wrote:
> On 22.03.2017 20:13, Eduardo Habkost wrote:
> > On Wed, Mar 22, 2017 at 01:00:49PM -0300, Eduardo Habkost wrote:
> > > This series adds scripts/device-crashtest.py, that can be used to
> > > crash-test -device with multiple machine/accel/device
> > > combinations.
> > >
> > > The script found a few crashes on some machines/devices. A dump
> > > of existing cases can be seen here:
> > >   https://gist.github.com/ehabkost/503b0af0375f0d98d3e84017e8ca54eb
> > >
> > > The script contains a whitelist that can also be useful as
> > > documentation of existing ways -device can fail or crash.
> > >
> > > Note that the script takes a few hours to run on the default mode
> > > (testing all accel/machine/device combinations), but the "-r N"
> > > option can be used to make it only test N random samples.
>
> Wow, impressive script, that must have been a lot of work 'til you've
> got it in a usable shape with that huge whitelist!

+1

Great work Eduardo, thanks!

> > Something I forgot to mention: I would like to run some subset of
> > these tests on "make check", but I don't know how we could choose
> > that subset. We could run, e.g., 100 random samples, but I am not
> > sure we really want to make "make check" non-deterministic.
>
> Maybe limit the tests to the devices that have a high chance to work on
> different machines? ... that means primarily PCI, ISA and USB devices, I
> guess.

Is hard to maintain that list, it will miss new devices and so on.
We should have a "nightly" run or something, but still, maintaining
the white list of known errors is still problematic.

Thanks,
Marcel

>  Thomas