Re: [PATCH RFC 2/3] gitlab: build all container images during CI

[Thomas Huth]

On 22/06/2020 17.33, Daniel P. Berrangé wrote:
> We have a number of container images in tests/docker/dockerfiles
> that are intended to provide well defined environments for doing
> test builds. We want our CI system to use these containers too.
>
> This introduces builds of all of them as the first stage in the
> CI, so that the built containers are available for later build
> jobs. The containers are setup to use the GitLab container
> registry as the cache, so we only pay the penalty of the full
> build when the dockerfiles change. The main qemu-project/qemu
> repo is used as a second cache, so that users forking QEMU will
> see a fast turnaround time on their CI jobs.
>
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>   .gitlab-ci.d/containers.yml | 248 ++++++++++++++++++++++++++++++++++++
>   .gitlab-ci.yml              |   3 +
>   2 files changed, 251 insertions(+)
>   create mode 100644 .gitlab-ci.d/containers.yml
>
> diff --git a/.gitlab-ci.d/containers.yml b/.gitlab-ci.d/containers.yml
> new file mode 100644
> index 0000000000..ea1edbb196
> --- /dev/null
> +++ b/.gitlab-ci.d/containers.yml
> @@ -0,0 +1,248 @@
> +
> +
> +.container_job_template: &container_job_definition
> +  image: docker:stable
> +  stage: containers
> +  services:
> +    - docker:dind
> +  before_script:
> +    - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest"
> +    - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/ci-$NAME:latest"
> +    - docker info
> +    - docker login registry.gitlab.com -u "$CI_REGISTRY_USER" -p 
> "$CI_REGISTRY_PASSWORD"

I can see this in the output:

WARNING! Using --password via the CLI is insecure. Use --password-stdin.

I have to admit that I have only little knowledge about docker ... but 
could there be an issue here? Should this be done in a different way?

> +  script:
> +    - docker pull "$TAG" || docker pull "$COMMON_TAG" || true
> +    - sed -i -e "s,FROM qemu:,FROM $CI_REGISTRY_IMAGE/ci-," 
> tests/docker/dockerfiles/$NAME.docker
> +    - docker build --cache-from "$TAG" --cache-from "$COMMON_TAG" --tag "$TAG" -f 
> "tests/docker/dockerfiles/$NAME.docker" tests/docker/dockerfiles
> +    - docker push "$TAG"
> +  after_script:
> +    - docker logout
> +
> +amd64-centos7-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: centos7
> +
> +amd64-centos8-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: centos8
> +
> +amd64-debian10-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: debian10
> +
> +amd64-debian11-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: debian11
> +
> +amd64-debian9-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: debian9
> +
> +amd64-debian9-mxe-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian9-container']
> +  variables:
> +    NAME: debian9-mxe
> +
> +alpha-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-alpha-cross
> +
> +amd64-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-amd64-cross
> +
> +amd64-debian-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-amd64
> +
> +arm64-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-arm64-cross
> +
> +arm64-test-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian11-container']
> +  variables:
> +    NAME: debian-arm64-test-cross
> +
> +armel-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-armel-cross
> +
> +armhf-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-armhf-cross
> +
> +hppa-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-hppa-cross
> +
> +m68k-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-m68k-cross
> +
> +mips64-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-mips64-cross
> +
> +mips64el-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-mips64el-cross
> +
> +mips-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-mips-cross
> +
> +mipsel-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-mipsel-cross
> +
> +powerpc-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-powerpc-cross
> +
> +ppc64-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-ppc64-cross
> +
> +ppc64el-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-ppc64el-cross
> +
> +riscv64-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-riscv64-cross
> +
> +s390x-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-s390x-cross
> +
> +sh4-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-sh4-cross
> +
> +sparc64-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-sparc64-cross
> +
> +tricore-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian9-container']
> +  variables:
> +    NAME: debian-tricore-cross
> +
> +win32-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer3
> +  needs: ['amd64-debian9-mxe-container']
> +  variables:
> +    NAME: debian-win32-cross
> +
> +win64-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer3
> +  needs: ['amd64-debian9-mxe-container']
> +  variables:
> +    NAME: debian-win64-cross
> +
> +xtensa-debian-cross-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: debian-xtensa-cross
> +
> +cris-fedora-cross-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: fedora-cris-cross
> +
> +amd64-fedora-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: fedora
> +
> +i386-fedora-cross-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: fedora-i386-cross
> +
> +amd64-ubuntu1804-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: ubuntu1804
> +
> +amd64-ubuntu2004-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: ubuntu2004
> +
> +amd64-ubuntu-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: ubuntu
> +
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index 9fdc752ea6..72d688875f 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -1,10 +1,13 @@
>   stages:
>     - containers
> +  - containers-layer2
> +  - containers-layer3
>     - build
>   
>   include:
>     - local: '/.gitlab-ci.d/edk2.yml'
>     - local: '/.gitlab-ci.d/opensbi.yml'
> +  - local: '/.gitlab-ci.d/containers.yml'
>   
>   .update_apt_template: &before_script_apt
>    before_script:

Acked-by: Thomas Huth <thuth@redhat.com>