Re: [PATCH RFC 2/3] gitlab: build all container images during CI

[Philippe Mathieu-Daudé]

On 6/22/20 5:46 PM, Daniel P. Berrangé wrote:
> On Mon, Jun 22, 2020 at 05:38:16PM +0200, Philippe Mathieu-Daudé wrote:
>> On 6/22/20 5:33 PM, Daniel P. Berrangé wrote:
>>> We have a number of container images in tests/docker/dockerfiles
>>> that are intended to provide well defined environments for doing
>>> test builds. We want our CI system to use these containers too.
>>>
>>> This introduces builds of all of them as the first stage in the
>>> CI, so that the built containers are available for later build
>>> jobs. The containers are setup to use the GitLab container
>>> registry as the cache, so we only pay the penalty of the full
>>> build when the dockerfiles change. The main qemu-project/qemu
>>> repo is used as a second cache, so that users forking QEMU will
>>> see a fast turnaround time on their CI jobs.
>>
>> OMG you did it! Lovely... 😍
>>
>> Looking at https://gitlab.com/berrange/qemu/-/pipelines/158854797
>> why gitlab isn't caching the docker images? the cache isn't
>> populated on all the runners yet? Or we have to use the same
>> runner again to hit its cache?
> 
> It is caching it but it isn't obvious what to look for.
> 
> The "docker build" command is always still run, but you'll
> notice each of the actual commands in the dockerfile are
> followed by a message "  ---> Using cache", instead of the
> normal command output.

Indeed!

> 
> Compare the "amd64-debian9-container" job as an example.
> 
> Here is the output seen in the original job which actually did a real
> build:
> 
>    https://gitlab.com/berrange/qemu/-/jobs/605783351
> 
> And here is the output from the pipeline you point to above:
> 
>    https://gitlab.com/berrange/qemu/-/jobs/606175855
> 
> The cached case is still taking 2 mins 30 seconds, but the original
> full build took 4 mins 50 seconds.
> 
> The difference will be more noticable if we expand the list of packages
> install in the images to cover more of QEMU's possible dependancies.

The difference is already very noticeable on my host:

$ time docker pull registry.gitlab.com/berrange/qemu/ci-centos8:latest
latest: Pulling from berrange/qemu/ci-centos8
8a29a15cefae: Pull complete
4a748adf2909: Pull complete
3c034506a458: Pull complete
cd48de325fa1: Pull complete
Digest:
sha256:a6580433f5456d7c49e2b9b796c717c79a5fa15f4eecf71bdbcffd9df6b8217a
Status: Downloaded newer image for
registry.gitlab.com/berrange/qemu/ci-centos8:latest
registry.gitlab.com/berrange/qemu/ci-centos8:latest
real    0m53.566s
user    0m0.115s
sys     0m0.081s

> 
> Regards,
> Daniel
>