Re: [PATCH 0/5] QEMU Gating CI

[Cleber Rosa]

On Fri, 24 Apr 2020 08:57:46 +0200
Erik Skultety <address@hidden> wrote:

> On Thu, Apr 23, 2020 at 11:28:21PM +0200, Philippe Mathieu-Daudé
> wrote:
> > On 4/23/20 7:13 PM, Daniel P. Berrangé wrote:
> > > On Thu, Apr 23, 2020 at 01:04:13PM -0400, Cleber Rosa wrote:
> > > > ----- Original Message -----
> > > > > From: "Peter Maydell" <address@hidden>
> > > > > To: "Markus Armbruster" <address@hidden>
> > > > > Cc: "Fam Zheng" <address@hidden>, "Thomas Huth"
> > > > > <address@hidden>, "Beraldo Leal" <address@hidden>, "Erik
> > > > > Skultety" <address@hidden>, "Alex Bennée"
> > > > > <address@hidden>, "Wainer Moschetta"
> > > > > <address@hidden>, "QEMU Developers"
> > > > > <address@hidden>, "Wainer dos Santos Moschetta"
> > > > > <address@hidden>, "Willian Rampazzo"
> > > > > <address@hidden>, "Cleber Rosa" <address@hidden>,
> > > > > "Philippe Mathieu-Daudé" <address@hidden>, "Eduardo
> > > > > Habkost" <address@hidden> Sent: Tuesday, April 21, 2020
> > > > > 8:53:49 AM Subject: Re: [PATCH 0/5] QEMU Gating CI
> > > > > 
> > > > > On Thu, 19 Mar 2020 at 16:33, Markus Armbruster
> > > > > <address@hidden> wrote:
> > > > > > Peter Maydell <address@hidden> writes:
> > > > > > > I think we should start by getting the gitlab setup
> > > > > > > working for the basic "x86 configs" first. Then we can
> > > > > > > try adding a runner for s390 (that one's logistically
> > > > > > > easiest because it is a project machine, not one owned by
> > > > > > > me personally or by Linaro) once the basic framework is
> > > > > > > working, and expand from there.
> > > > > > 
> > > > > > Makes sense to me.
> > > > > > 
> > > > > > Next steps to get this off the ground:
> > > > > > 
> > > > > > * Red Hat provides runner(s) for x86 stuff we care about.
> > > > > > 
> > > > > > * If that doesn't cover 'basic "x86 configs" in your
> > > > > > judgement, we fill the gaps as described below under
> > > > > > "Expand from there".
> > > > > > 
> > > > > > * Add an s390 runner using the project machine you
> > > > > > mentioned.
> > > > > > 
> > > > > > * Expand from there: identify the remaining gaps, map them
> > > > > > to people / organizations interested in them, and solicit
> > > > > > contributions from these guys.
> > > > > > 
> > > > > > A note on contributions: we need both hardware and people.
> > > > > > By people I mean maintainers for the infrastructure, the
> > > > > > tools and all the runners. Cleber & team are willing to
> > > > > > serve for the infrastructure, the tools and the Red Hat
> > > > > > runners.
> > > > > 
> > > > > So, with 5.0 nearly out the door it seems like a good time to
> > > > > check in on this thread again to ask where we are
> > > > > progress-wise with this. My impression is that this patchset
> > > > > provides most of the scripting and config side of the first
> > > > > step, so what we need is for RH to provide an x86 runner
> > > > > machine and tell the gitlab CI it exists. I appreciate that
> > > > > the whole coronavirus and working-from-home situation will
> > > > > have upended everybody's plans, especially when actual
> > > > > hardware might be involved, but how's it going ?
> > > > > 
> > > > 
> > > > Hi Peter,
> > > > 
> > > > You hit the nail in the head here.  We were affected indeed
> > > > with our ability to move some machines from one lab to another
> > > > (across the country), but we're actively working on it.
> > > 
> > > For x86, do we really need to be using custom runners ?
> > > 
> > > With GitLab if someone forks the repo to their personal
> > > namespace, they cannot use any custom runners setup by the origin
> > > project. So if we use custom runners for x86, people forking
> > > won't be able to run the GitLab CI jobs.
> > > 
> > > As a sub-system maintainer I wouldn't like this, because I
> > > ideally want to be able to run the same jobs on my staging tree,
> > > that Peter will run at merge time for the PULL request I send.
> > > 
> > > Thus my strong preference would be to use the GitLab runners in
> > > every scenario where they are viable to use. Only use custom
> > > runners in the cases where GitLab runners are clearly inadequate
> > > for our needs.
> > > 
> > > Based on what we've setup in GitLab for libvirt,  the shared
> > > runners they have work fine for x86. Just need the environments
> > > you are testing to be provided as Docker containers (you can
> > > actually build and cache the container images during your CI job
> > > too).  IOW, any Linux distro build and test jobs should be able
> > > to use shared runners on x86, and likewise mingw builds. Custom
> > > runners should only be needed if the jobs need todo *BSD / macOS
> > > builds, and/or have access to specific hardware devices for some
> > > reason.
> 
> Not just ^that, you also want custom VM runners to run integration
> tests, e.g. in libvirt, we'd have to put systemd and a lof of other
> cruft into the container to be able to run the tests at which point
> you must ask yourself, whyt not go with a VM instead in which case
> we're limited in terms of infrastructure...
> 

I'm completely in agreement that a lot of the jobs will be suitable to
be run on containers, but like you exemplified here Erik, we must take
into account the ones that won't be suitable.

For instance, a very real use case is testing KVM on bare metal.  One
could argue that "QEMU running on a container making use of
KVM would suffice". It may be true, it may not.  But even that 
won't be possible Today on a CentOS/RHEL 8 machine, because
gitlab-runner knows nothing about podman, so full blown x86
physical boxes may be the "cheaper" and more practical solution, at
least initially. Trying to use other architectures would surely have
similar issues.

> > 
> > Thanks to insist with that point Daniel. I'd rather see every
> > configuration reproducible, so if we loose a hardware sponsor, we
> > can find another one and start another runner.
> > Also note, if it is not easy to reproduce a runner, it will be very
> > hard to debug a reported build/test error.
> 
> (Thanks for bringing ^this point up Philippe)
> 
> ...However, what we've been actively working on in libvirt is to
> extend the lcitool we have (which can spawn local test VMs) to the
> point where we're able to generate machines that would be the
> reproducible. Right now I'm playing with cloud-init integration with
> lcitool (patches coming soon) that would allow us to use the same
> machines locally as we'd want to have in, say, OpenStack and share
> them as compressed images, so even when updated/managed by lcitool
> locally, you'd get the same environment.
> 
> Regards,
> 

This is great, and it aligns with my previous point that reproducibility
it's not *just* about the hardware, but about diligently documenting
and automating the environments, be them mundane or super specialized.
And IMO that should score some points when it comes to being
promoted/demoted as a Gating machine/job.

Thanks for the feedback Erik!
- Cleber.