Re: [PATCH RFC 2/3] gitlab: build all container images during CI

[Daniel P . Berrangé]

On Thu, Jun 25, 2020 at 03:25:19PM +0200, Thomas Huth wrote:
> On 25/06/2020 12.24, Daniel P. Berrangé wrote:
> > On Thu, Jun 25, 2020 at 12:14:33PM +0200, Thomas Huth wrote:
> > > On 22/06/2020 17.33, Daniel P. Berrangé wrote:
> > > > We have a number of container images in tests/docker/dockerfiles
> > > > that are intended to provide well defined environments for doing
> > > > test builds. We want our CI system to use these containers too.
> > > > 
> > > > This introduces builds of all of them as the first stage in the
> > > > CI, so that the built containers are available for later build
> > > > jobs. The containers are setup to use the GitLab container
> > > > registry as the cache, so we only pay the penalty of the full
> > > > build when the dockerfiles change. The main qemu-project/qemu
> > > > repo is used as a second cache, so that users forking QEMU will
> > > > see a fast turnaround time on their CI jobs.
> > > > 
> > > > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> > > > ---
> > > >    .gitlab-ci.d/containers.yml | 248 
> > > > ++++++++++++++++++++++++++++++++++++
> > > >    .gitlab-ci.yml              |   3 +
> > > >    2 files changed, 251 insertions(+)
> > > >    create mode 100644 .gitlab-ci.d/containers.yml
> > > > 
> > > > diff --git a/.gitlab-ci.d/containers.yml b/.gitlab-ci.d/containers.yml
> > > > new file mode 100644
> > > > index 0000000000..ea1edbb196
> > > > --- /dev/null
> > > > +++ b/.gitlab-ci.d/containers.yml
> > > > @@ -0,0 +1,248 @@
> > > > +
> > > > +
> > > > +.container_job_template: &container_job_definition
> > > > +  image: docker:stable
> > > > +  stage: containers
> > > > +  services:
> > > > +    - docker:dind
> > > > +  before_script:
> > > > +    - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest"
> > > > +    - export 
> > > > COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/ci-$NAME:latest"
> > > > +    - docker info
> > > > +    - docker login registry.gitlab.com -u "$CI_REGISTRY_USER" -p 
> > > > "$CI_REGISTRY_PASSWORD"
> > > > +  script:
> > > > +    - docker pull "$TAG" || docker pull "$COMMON_TAG" || true
> > > > +    - sed -i -e "s,FROM qemu:,FROM $CI_REGISTRY_IMAGE/ci-," 
> > > > tests/docker/dockerfiles/$NAME.docker
> > > > +    - docker build --cache-from "$TAG" --cache-from "$COMMON_TAG" 
> > > > --tag "$TAG" -f "tests/docker/dockerfiles/$NAME.docker" 
> > > > tests/docker/dockerfiles
> > > > +    - docker push "$TAG"
> > > > +  after_script:
> > > > +    - docker logout
> > > 
> > > .gitlab-ci.d/edk2.yml uses a "changes" rule to only run the pipeline if
> > > something really has been changed. Could you use something similar here?
> > > E.g.:
> > > 
> > > rules:
> > >   - changes:
> > >     - .gitlab-ci.d/containers.yml
> > >     - tests/docker/*
> > >     - tests/docker/dockerfiles/*
> > > 
> > > ?
> > 
> > If the OS distro base image changes, we'll never pick it up with that
> > kind of filtering.  For the main gitlab.com/qemu-project/qemu  you
> > could configure a nightly/weekly/whatever job to force rebuild on a
> > periodic basis to pick up base image changes.  The downside of this
> > is that any users who fork qemu won't have that periodic job and so
> > will be testing their work against potentially outdated content.
> > 
> > Having said all that, I'm not 100% convinced I'm actually picking
> > up changed base images right now anyway, given our use of caching.
> > 
> > It is possible that I would need todo an explict "docker pull" of
> > the base image to force it to trigger a refresh othrewise I have
> > a feeling we're always cached.
> 
> But currently, each of the container stages currently takes > 2 minutes,
> even with the cached containers. I had a quick look, and it takes 7 minutes
> 'till the "build" stage begins. So all the advantages of not having to do
> "yum/apt-get install" in the build containers anymore seem to be crushed by
> the time that the three container stages take now?

That's still not an apples/apples comparison though. The containers have
many mre packages installed and so test more features. Add all those extra
packages into the original apt-get/dnf commands and then compare times.

I do agree though that I don't much like the 3 containers stages. I set
them up so they would parallelize. ie stage 2 can start without waiting
for entire of stage 1 to finish - only the parent container needs to
finish. The builds stage still waits for all the containers to complete.

We could do similar optimize for the actual build jobs, so they don't have
to wait for all containers to finish - they only need wait for their own
required container to finish.

Ultimately it might be a better tradeoff to not have inheritance between
the containers, and instead just duplicate the common packages in each.
This leads to bigger container images, but simpler builds. Libvirt went
this way for its cross compiler images, just duplicating the shared parts
in each.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|