Re: [PATCH v5 1/5] iotests: remove 'linux' from default supported platforms

[Peter Maydell]

On Fri, 4 Oct 2019 at 13:45, Max Reitz <address@hidden> wrote:
> > In the end, I don't care what code these patches touched. I do an
> > innocent git pull, and when I finally see that it's master that breaks
> > iotests and not my patches on top of it, I'm annoyed.
>
> Hm.  Part of my point was that this still happens all the time.
>
> Which is why I’d prefer more tests to run in CI than a handful of not
> very useful ones in make check.
>
> (Of course, running it in a CI won’t prevent iotest failures on your
> machine, but in practice neither does the current model.)

If you want the tree to be defended against problems, then
you need to run tests in 'make check'. Nothing else gets consistently
run and has failures spotted either before code goes into the
tree or quickly afterwards. 'make check' does have the restriction
that we don't want the tests to take too long to run, but in
general the block layer should be running some reasonable subset
of tests in the project's standard "run the tests please" command.
(I have no opinion on exactly what that subset ought to be, beyond
that it would be good if that subset doesn't have known intermittent
failures in it.)

> I still think that I personally would prefer the iotests to not run as
> part of make check, but just as CI.

'make check' *is* our CI gate, to a first approximation. Most of
the various travis and other setups are simply a front-end for
"build QEMU in various configurations and on various hosts
and then run 'make check'". The travis setup at the moment is
a bit odd, because it runs tests but it's not a gate on our
merging changes. Ideally I would like to fix this so that
rather than me personally running "make check" via a bunch
of scripts we have one CI setup that we trust (gitlab seems
the current favoured contender) and that gates changes flowing
into master rather than me doing it manually. We might then turn
travis off if it's not providing anything for us that the gitlab
setup doesn't.

thanks
-- PMM