Re: [PATCH v5 1/5] iotests: remove 'linux' from default supported platforms

[Max Reitz]

On 04.10.19 15:16, Peter Maydell wrote:
> On Fri, 4 Oct 2019 at 13:45, Max Reitz <address@hidden> wrote:
>>> In the end, I don't care what code these patches touched. I do an
>>> innocent git pull, and when I finally see that it's master that breaks
>>> iotests and not my patches on top of it, I'm annoyed.
>>
>> Hm.  Part of my point was that this still happens all the time.
>>
>> Which is why I’d prefer more tests to run in CI than a handful of not
>> very useful ones in make check.
>>
>> (Of course, running it in a CI won’t prevent iotest failures on your
>> machine, but in practice neither does the current model.)
> 
> If you want the tree to be defended against problems, then
> you need to run tests in 'make check'. Nothing else gets consistently
> run and has failures spotted either before code goes into the
> tree or quickly afterwards.

Yes, but part of the problem is that what does get run as part of make
check currently does not help much.

So this doesn’t really defend the iotests from problems.

> 'make check' does have the restriction
> that we don't want the tests to take too long to run, but in
> general the block layer should be running some reasonable subset
> of tests in the project's standard "run the tests please" command.
> (I have no opinion on exactly what that subset ought to be, beyond
> that it would be good if that subset doesn't have known intermittent
> failures in it.)

Deciding the subset is difficult.  My stance was that it’s better to not
choose an arbitrary subset here but ensure that really everything gets
run as part of CI, that is asynchronously so it doesn’t block anyone and
can take as long as it wants.

If we decide to get pull requests based on that, then that’d bring even
more pain, but at least it’d be honest.  But just running half of the
qcow2 tests to me seems only like we want to pretend we ran the iotests.
 So for me, iotests still break, and we need to deal with make check
failures on top.  I’d at least prefer one or the other.


I voted for dropping make check, because running all iotests doesn’t
seem feasible to me.

>> I still think that I personally would prefer the iotests to not run as
>> part of make check, but just as CI.
> 
> 'make check' *is* our CI gate, to a first approximation. Most of
> the various travis and other setups are simply a front-end for
> "build QEMU in various configurations and on various hosts
> and then run 'make check'". The travis setup at the moment is
> a bit odd, because it runs tests but it's not a gate on our
> merging changes. Ideally I would like to fix this so that
> rather than me personally running "make check" via a bunch
> of scripts we have one CI setup that we trust (gitlab seems
> the current favoured contender) and that gates changes flowing
> into master rather than me doing it manually. We might then turn
> travis off if it's not providing anything for us that the gitlab
> setup doesn't.

Hm, but make check also serves other purposes.  I would suppose e.g.
distributions generally require make check to pass when building packages.

I assumed our CI included more things than just make check, so we could
move the iotests from out of make check but keep them as part of CI.

Max

signature.asc
Description: OpenPGP digital signature