[Qemu-devel] State of QEMU CI as we enter 4.0

[Alex Bennée]

Hi,

As we approach stabilisation for 4.0 I thought it would be worth doing a
review of the current state of CI and stimulate some discussion of where
it is working for us and what could be improved.

Testing in Build System
=======================

Things seem to be progressing well in this respect. More and more tests
have been added into the main source tree and they are only a make
invocation away. These include:

  check          (includes unit, qapi-schema, qtest and decodetree)
  check-tcg      (now with system mode tests!)
  check-softfloat
  check-block
  check-acceptance

Personally check-acceptance is the area I've looked at the least but
this seems to be the best place for "full life cycle" tests like booting
kernels and running stress and performance tests. I'm still a little
unsure how we deal with prebuilt kernels and images here though. Are
they basically provided by 3rd parties from their websites? Do we mirror
any of the artefacts we use for these tests?

One area of concern is how well this all sits with KVM (and other HW
accelerators) and how that gets tested. With my ARM hat on I don't
really see any integration between testing kernel and QEMU changes
together to catch any problems as the core OS support for KVM gets
updated.

Another area I would like to improve is how we expand testing with
existing test suites. I'm thinking things like LTP and kvm-unit-tests
which can exercise a bunch of QEMU code but are maybe a bit to big to be
included in the source tree. Although given we included TestFloat (via a
git submodule) maybe we shouldn't dismiss that approach? Or is this
something that could be done via Avocado?

Generally though I think we are doing pretty well at increasing our test
coverage while making the tests more directly available to developers
without having to rely on someones personal collection of random
binaries.

I wanted to know if we should encode this somewhere in our developer
documentation:

  There is a strong preference for new QEMU tests to be integrated with
  the build system. Developers should be able to (build and) run the new
  tests locally directly from make.

?

Testing in the Cloud
====================

After BuildBot went out-of-service we have been relying heavily on Travis
as our primary CI platform. This has been creaking somewhat under the
strain and while we have a large test matrix its coverage is fairly
Ubuntu/x86 centric. However in recent months we've expanded and we now
have:

  - Shippable, cross compilers - catches a lot of 32/64 bit isms
  - Cirrus, FreeBSD and MacOS builds
  - GitLab, Alternative x86/Debian - iotests

Currently they don't a whole lot to the diversity of our testing
although Shippable is pretty quick and does catch cross-compile missteps
quite nicely. I think there is a good argument for removing some of the
testing from Travis and trying to get its long run time down to
something a bit more useful and balancing that with more tests in on the
other services.

I'm currently looking at how easy it is to expand the build farm with
GitLab. It holds a promise of making it easy to add external runners to
the build farm with a fairly simple installation of a runner client on
the machine. We'd just need to beg and borrow for more non-x86 machines.

Cloud CI Feedback
------------------

Currently only Travis reports the current status to our IRC channel but
with the stability of the testing being up and down it sometimes feels
like unnecessary noise. I've put together a wiki template that allows
tracking of the current CI status using badges:

  https://wiki.qemu.org/Template:CIStatus

Which includes patchew and Coverity status. This can be included on any
personal pages you wish but it is also fairly prominent on the testing
page:

  https://wiki.qemu.org/Testing

For those wishing to have a central point of reference for other
branches there is:

  https://wiki.qemu.org/Template:CustomCIStatus

which is parameterised so you can enable for different branches, for
example my fpu/next branch is linked from:

  https://wiki.qemu.org/Features/Softfloat

Of course for these to be useful people need to a) look at them and b)
be confident enough that non-green is worth looking at.

I'm wary of adding a bunch more notifications onto the IRC channel. What
I really would like is a cibot which could sit on our channel and
aggregate the status from the various services and also be queried for
the state of various branches. However this would be a chunk of non-QEMU
related work to get up and running.

Test Stability
--------------

We have a slowly growing number of tests who seem to fail on a fairly
regular basis in our CI tests. Sometimes it has been possible to
replicate the failure but often it seems to be a feature of running in
the CI system that is hard to replicate on developer machines. I've had
some success in replicating some and getting the appropriate developers
to debug on a tmate session. However we really we need more eyes on
these troublesome tests helping with the effort. For now I've created a
table to track them so the details are not lost on the ephemeral IRC:

  https://wiki.qemu.org/Testing/CI/TroublesomeTests

If we start collecting data on this page we can maybe spot some patterns
or gain some clues on how to replicate the failures locally.

As you may notices I've created the a Testing/CI page and am slowly
populating the sub pages with more information. Please feel free to help
in the effort ;-)

Docker Images
=============

The addition of docker has unlocked the ability to build a lot more
tests as well as compile testing on a much wider range of distros. I
think there are two outstanding areas that need improvement

Daniel has been looking at building and hosting the images somewhere.
This would be useful as it would stop us slamming the distros
repositories constantly rebuilding the same images and also help reduce
the time to test.

The other area that needs some work is better supporting non-x86 hosts.
While Docker's multi-arch story is much better (docker run debian:stable
will DTRT on any main architecture) we get stumped by things like
Debian's uneven support of cross compilers. For 4.1 I'd like to
reorganise the dockerfiles subdirectory into multiarch and arch specific
directories so we approach this is a less ad-hoc way. It would also be
nice to have the ability to gracefully fallback to linux-user powered
images where the host architecture doesn't have what we need.

Conclusion
==========

I think generally the state of QEMU's CI has improved over the last few
years but we still have a number of challenges caused by its distributed
nature and test stability. We are still re-active to failures rather
than having something fast and reliable enough to gate changes going
into the code base. This results in fairly long periods when one
or more parts of the testing mosaic are stuck on red waiting for fixes
to finally get merged back into master.

So what do people think? Have I missed anything out? What else can we do
to improve the situation?

Let the discussion begin ;-)

--
Alex Bennée