Re: [PATCH RFC 0/3] gitlab: build containers to use in build jobs

[Daniel P . Berrangé]

On Thu, Jun 25, 2020 at 01:33:42PM +0200, Thomas Huth wrote:
> On 25/06/2020 13.29, Daniel P. Berrangé wrote:
> > On Thu, Jun 25, 2020 at 12:26:53PM +0100, Daniel P. Berrangé wrote:
> > > On Thu, Jun 25, 2020 at 01:15:52PM +0200, Thomas Huth wrote:
> > > > On 22/06/2020 17.33, Daniel P. Berrangé wrote:
> > > > > The current gitlab CI jobs are quite inefficient because they
> > > > > use the generic distro images and then apt-get/dnf install
> > > > > extra packages every time.
> > > > > 
> > > > > The other downside is that the container environment used is
> > > > > only defined in thte .gitlab-ci.yml file, so it tedious to
> > > > > reproduce locally.
> > > > > 
> > > > > We already have containers defined in tests/docker for use by
> > > > > developers building locally. We can use these for CI systems
> > > > > too if we just had a way to build them....
> > > > > 
> > > > > ...GitLab CI offers such a way. We can use docker-in-docker
> > > > > to build the images at the start of the CI cycle, and use
> > > > > the built images in later jobs.
> > > > > 
> > > > > These later jobs are now faster because they're not having
> > > > > to install any software.
> > > > 
> > > > Did you see any speed-up? I had a look at some pipelines, and it seems 
> > > > to me
> > > > that they rather got slower now? For example, this is the system1 
> > > > pipeline
> > > > before your change:
> > > > 
> > > >   https://gitlab.com/huth/qemu/-/jobs/610924897
> > > > 
> > > > and after your change:
> > > > 
> > > >   https://gitlab.com/huth/qemu/-/jobs/611069374
> > > > 
> > > > Duration went up from 35 minutes to 42 minutes.
> > > > 
> > > > Seems also to happen in your builds, before the change:
> > > > 
> > > >   https://gitlab.com/berrange/qemu/-/jobs/582995084
> > > > 
> > > > and after the change:
> > > > 
> > > >   https://gitlab.com/berrange/qemu/-/jobs/606175927
> > > > 
> > > > ... went from 36 minutes up to 42 minutes.
> > > > 
> > > > Could be a coincidence due to the load on the shared runners, but it 
> > > > looks
> > > > at least a little bit suspicious...
> > > 
> > > I think the difference is because we're building more features now. The
> > > dockerfiles have provided more build pre-requisites that the old gitlab
> > > recipe did.
> > > 
> > > If you compare the configure summary, I see the new build now covers
> > > SDL, curses, curl, pulseaudio, virtiofs, SASL, libjpeg, xen, docs
> > > and a few more.  So we've saved time by not intsallling many packages
> > > each time, but consumed a greater amount of time by compiling more
> > > features.
> > 
> > Oh a missed a lot more actually - there's also spice, opengl, libiscsi,
> > libnfs, libusb, seccomp, libssh, lzo, snappy, bzip, zstd, numa and udev
> > too.
> 
> Ok, that's fair, I think it's ok to spend some additional minutes for the
> extended test coverage here.

Unlike Travis which limits to 5 concurrent jobs,  GitLab doesn't seem to
have a fixed concurrency limit (at least I've not managed to hit one yet).
So if we want faster overall build time, we have alot more scope for
splitting jobs in half and making use of much better parallelism in CI
to get this to complete sooner.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|