Re: [PATCH] tests/docker: Add gentoo-loongarch64-cross image and run cross builds in GitLab

[Alex Bennée]

WANG Xuerui <git@xen0n.name> writes:

> Normally this would be based on qemu/debian10 or qemu/ubuntu2004, but
> after a week-long struggle, I still cannot build stage2 gcc with the
> known-good LoongArch toolchain sources, so I chose the least-resistance
> path with Gentoo as base image. As this image is not expected to be
> re-built by CI, like hexagon, it should not take much maintenance
> effort; also it's expected to be replaced as soon as Debian is
> available.
>
> As the LoongArch *target* has not been merged yet, a check-tcg job is
> not added at the moment, but cross builds with the TCG *host* port are
> already possible, and added to CI matrix.
>
> Due to constant flux of the toolchain sources used (especially that of
> glibc), the binaries built with this image may or may not work when
> run on actual hardware, but still useful for ensuring things correctly
> build. This image is expected to be updated every once in a while,
> before everything settles down.
>
> As a reference, the image takes about 25 minutes to rebuild on a
> Threadripper 3990X system with Docker operating on HDD; YMMV but it
> probably wouldn't become significantly shorter, as everything needs to
> be built from source in our case.
>
> (In the original submission along with the rest of LoongArch TCG
> patches, I forgot to make the dependency to the container build job
> optional, thus CI was passing in my own fork but broke upstream. Fixed
> for a 2nd take, and I also took the chance to update base image versions
> and such.)
>
> Signed-off-by: WANG Xuerui <git@xen0n.name>
> ---
>
> (Note to CI maintainers: obviously this image has to be built and pushed
> manually, for everything to keep working. Sorry for the extra work
> early-on, but the community around LoongArch still hope to upstream most
> things during 2022 so eventually we will transition away from this.
> This work is no hurry though, and Happy Holidays everyone!)
>
>  .gitlab-ci.d/container-cross.yml              |  27 +++++
>  .gitlab-ci.d/crossbuilds.yml                  |  25 ++++
>  MAINTAINERS                                   |   2 +
>  tests/docker/Makefile.include                 |  21 ++++
>  .../gentoo-loongarch64-cross.docker           |  21 ++++
>  .../build-toolchain.sh                        | 109 ++++++++++++++++++
>  6 files changed, 205 insertions(+)
>  create mode 100644 tests/docker/dockerfiles/gentoo-loongarch64-cross.docker
>  create mode 100755 
> tests/docker/dockerfiles/gentoo-loongarch64-cross.docker.d/build-toolchain.sh
>
> diff --git a/.gitlab-ci.d/container-cross.yml 
> b/.gitlab-ci.d/container-cross.yml
> index a3b5b90552..7a8cc556cc 100644
> --- a/.gitlab-ci.d/container-cross.yml
> +++ b/.gitlab-ci.d/container-cross.yml
> @@ -82,6 +82,33 @@ hppa-debian-cross-container:
>    variables:
>      NAME: debian-hppa-cross
>  
> +# Similar to hexagon, we don't want to build loongarch64 in the CI either.
> +loongarch64-cross-container:
> +  image: docker:stable
> +  stage: containers
> +  rules:
> +    - if: '$CI_PROJECT_NAMESPACE == "qemu-project"'
> +      when: never
> +    - when: always
> +  variables:
> +    NAME: gentoo-loongarch64-cross
> +    GIT_DEPTH: 1
> +  services:
> +    - docker:dind
> +  before_script:
> +    - export TAG="$CI_REGISTRY_IMAGE/qemu/$NAME:latest"
> +    - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/qemu/$NAME:latest"
> +    - docker info
> +    - docker login $CI_REGISTRY -u "$CI_REGISTRY_USER" -p 
> "$CI_REGISTRY_PASSWORD"
> +  script:
> +    - echo "TAG:$TAG"
> +    - echo "COMMON_TAG:$COMMON_TAG"
> +    - docker pull $COMMON_TAG
> +    - docker tag $COMMON_TAG $TAG
> +    - docker push "$TAG"
> +  after_script:
> +    - docker logout
> +
>  m68k-debian-cross-container:
>    extends: .container_job_template
>    stage: containers-layer2
> diff --git a/.gitlab-ci.d/crossbuilds.yml b/.gitlab-ci.d/crossbuilds.yml
> index 17d6cb3e45..ef19ade554 100644
> --- a/.gitlab-ci.d/crossbuilds.yml
> +++ b/.gitlab-ci.d/crossbuilds.yml
> @@ -68,6 +68,31 @@ cross-i386-tci:
>      EXTRA_CONFIGURE_OPTS: 
> --target-list=i386-softmmu,i386-linux-user,aarch64-softmmu,aarch64-linux-user,ppc-softmmu,ppc-linux-user
>      MAKE_CHECK_ARGS: check check-tcg
>  
> +# Upstream LoongArch support is still incomplete, but toolchain is already
> +# usable and partially merged, so the host support is already testable; but
> +# don't let failures block CI.
> +#
> +# Similar to hexagon, the container image is built outside of CI and manually
> +# uploaded at the moment, so make the dependency to container build job
> +# optional.
> +cross-loongarch64-system:
> +  extends: .cross_system_build_job
> +  allow_failure: true
> +  needs:
> +    job: loongarch64-cross-container
> +    optional: true
> +  variables:
> +    IMAGE: gentoo-loongarch64-cross
> +
> +cross-loongarch64-user:
> +  extends: .cross_user_build_job
> +  allow_failure: true
> +  needs:
> +    job: loongarch64-cross-container
> +    optional: true
> +  variables:
> +    IMAGE: gentoo-loongarch64-cross
> +
>  cross-mips-system:
>    extends: .cross_system_build_job
>    needs:
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 5456536805..6b42fe16c7 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -3148,6 +3148,8 @@ LoongArch64 TCG target
>  M: WANG Xuerui <git@xen0n.name>
>  S: Maintained
>  F: tcg/loongarch64/
> +F: tests/docker/dockerfiles/gentoo-loongarch64-cross.docker
> +F: tests/docker/dockerfiles/gentoo-loongarch64-cross.docker.d/
>  
>  MIPS TCG target
>  M: Philippe Mathieu-Daudé <f4bug@amsat.org>
> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
> index f1a0c5db7a..a2cdf193bb 100644
> --- a/tests/docker/Makefile.include
> +++ b/tests/docker/Makefile.include
> @@ -176,6 +176,27 @@ docker-image-debian-hexagon-cross: 
> $(DOCKER_FILES_DIR)/debian-hexagon-cross.dock
>                               qemu/debian-hexagon-cross --add-current-user,   
>         \
>                       "PREPARE", "debian-hexagon-cross"))
>  
> +#
> +# Same for loongarch64-cross.
> +#
> +docker-image-gentoo-loongarch64-cross: 
> $(DOCKER_FILES_DIR)/gentoo-loongarch64-cross.docker
> +     $(if $(NOCACHE),                                                        
>         \
> +             $(call quiet-command,                                           
>         \
> +                     $(DOCKER_SCRIPT) build -t qemu/gentoo-loongarch64-cross 
> -f $<   \
> +                     $(if $V,,--quiet) --no-cache                            
>         \
> +                     --registry $(DOCKER_REGISTRY) --extra-files             
>         \
> +                     
> $(DOCKER_FILES_DIR)/gentoo-loongarch64-cross.docker.d/build-toolchain.sh, \
> +                     "BUILD", "gentoo-loongarch64-cross"),                   
>         \
> +             $(call quiet-command,                                           
>         \
> +                     $(DOCKER_SCRIPT) fetch $(if $V,,--quiet)                
>         \
> +                             qemu/gentoo-loongarch64-cross 
> $(DOCKER_REGISTRY),       \
> +                     "FETCH", "gentoo-loongarch64-cross")                    
>         \
> +             $(call quiet-command,                                           
>         \
> +                     $(DOCKER_SCRIPT) update $(if $V,,--quiet)               
>         \
> +                             qemu/gentoo-loongarch64-cross 
> --add-current-user,       \
> +                     "PREPARE", "gentoo-loongarch64-cross"))
> +
> +

Can this not use debian-toolchain-run (possibly renamed) like the
microblaze and nios2 toolchains? The hexagon only has the legacy setup
because we never merged the more traditional image due to signal issues.

>  debian-toolchain-run = \
>       $(if $(NOCACHE),                                                \
>               $(call quiet-command,                                   \
> diff --git a/tests/docker/dockerfiles/gentoo-loongarch64-cross.docker 
> b/tests/docker/dockerfiles/gentoo-loongarch64-cross.docker
> new file mode 100644
> index 0000000000..59846223d3
> --- /dev/null
> +++ b/tests/docker/dockerfiles/gentoo-loongarch64-cross.docker
> @@ -0,0 +1,21 @@
> +#
> +# Docker LoongArch64 cross-compiler target
> +#
> +# This docker target is used for building loongarch64 tests. As it also
> +# needs to be able to build QEMU itself in CI we include it's
> +# build-deps. It is also a "stand-alone" image so as not to be
> +# triggered by re-builds on other base images given it takes a long
> +# time to build.
> +#
> +
> +FROM docker.io/gentoo/stage3:20211229
> +
> +ADD build-toolchain.sh /root/build-toolchain.sh
> +RUN cd /root && ./build-toolchain.sh
> +
> +# Specify the cross prefix for this image (see tests/docker/common.rc)
> +# Also gcc 12 currently produces spurious warnings under our condition so
> +# disable Werror as well
> +ENV QEMU_CONFIGURE_OPTS --cross-prefix=loongarch64-unknown-linux-gnu- 
> --disable-werror
> +# We don't have target support at the moment, so just verify that things 
> build
> +ENV DEF_TARGET_LIST x86_64-softmmu,x86_64-linux-user
> diff --git 
> a/tests/docker/dockerfiles/gentoo-loongarch64-cross.docker.d/build-toolchain.sh
>  
> b/tests/docker/dockerfiles/gentoo-loongarch64-cross.docker.d/build-toolchain.sh
> new file mode 100755
> index 0000000000..654f1394e9
> --- /dev/null
> +++ 
> b/tests/docker/dockerfiles/gentoo-loongarch64-cross.docker.d/build-toolchain.sh
> @@ -0,0 +1,109 @@
> +#!/bin/bash
> +
> +set -e
> +
> +TRIPLE=loongarch64-unknown-linux-gnu
> +CROSSDEV_OV=/opt/crossdev-overlay
> +LOONGSON_OV=/opt/loongson-overlay
> +CROSS_EMERGE="${TRIPLE}-emerge"
> +
> +# this will break on non-SMP machines, but no one should build this image
> +# on such machine in the first place
> +J=$(expr $(nproc) / 2)
> +echo "MAKEOPTS=\"-j${J} -l${J}\"" >> /etc/portage/make.conf
> +echo "EGIT_CLONE_TYPE=shallow" >> /etc/portage/make.conf
> +
> +# these features are not supported in Docker
> +export FEATURES="-ipc-sandbox -network-sandbox"
> +
> +# populate Portage tree
> +GENTOO_MIRROR='https://bouncer.gentoo.org/fetch/root/all'
> +PORTAGE_SNAPSHOT_FILE=gentoo-20211228.tar.xz
> +pushd /tmp
> +    # not every mirror will have this file synced yet, so retry until success
> +    i=0
> +    max_retry=5
> +    while [[ $i -lt $max_retry ]]; do
> +        [[ $i -gt 0 ]] && echo "Retrying ($i of $max_retry)..."
> +        wget "${GENTOO_MIRROR}/snapshots/${PORTAGE_SNAPSHOT_FILE}" && break
> +        : $((i++))
> +    done
> +    [[ -f "$PORTAGE_SNAPSHOT_FILE" ]] || exit 1
> +
> +    mkdir -p /var/db/repos/gentoo
> +    pushd /var/db/repos/gentoo
> +        tar -xf "/tmp/${PORTAGE_SNAPSHOT_FILE}" --strip-components=1
> +    popd
> +
> +    rm "$PORTAGE_SNAPSHOT_FILE"
> +popd

This seems a little excessive. Wouldn't a straight emerge --sync do or
even just sticking to the state of portage in the source docker image?

Maybe:

  emerge-webrsync --revert=20211228

> +
> +emerge -j crossdev dev-vcs/git
> +
> +# prepare for crossdev
> +mkdir /etc/portage/repos.conf
> +crossdev -t "$TRIPLE" --ov-output "$CROSSDEV_OV" --init-target
> +
> +git clone https://github.com/xen0n/loongson-overlay.git "$LOONGSON_OV"
> +pushd "$LOONGSON_OV"
> +    git checkout 20b9c9f96fb5ed596bbab6bd6f274932492fb12b
> +popd
> +
> +pushd "${CROSSDEV_OV}/cross-${TRIPLE}"
> +    rm binutils gcc glibc linux-headers
> +    ln -s "${LOONGSON_OV}/sys-devel/binutils" .
> +    ln -s "${LOONGSON_OV}/sys-devel/gcc" .
> +    ln -s "${LOONGSON_OV}/sys-libs/glibc" .
> +    ln -s "${LOONGSON_OV}/sys-kernel/linux-headers" .
> +popd
> +
> +cat > "${CROSSDEV_OV}/metadata/layout.conf" <<EOF
> +masters = gentoo
> +repo-name = crossdev-overlay
> +manifest-hashes = SHA256 SHA512 WHIRLPOOL
> +thin-manifests = true
> +EOF
> +
> +chown -R portage:portage "$CROSSDEV_OV"
> +chown -R portage:portage "$LOONGSON_OV"
> +
> +# make cross toolchain
> +crossdev -t "$TRIPLE" --without-headers \
> +    --binutils 2.37_p1-r1 \
> +    --gcc 12.0.0_pre9999 \
> +    --libc 2.34-r4
> +
> +# prepare for loongarch cross emerges
> +TARGET_PROFILE="default/linux/loong/21.0/la64v100/lp64d/desktop"
> +pushd "/usr/${TRIPLE}/etc/portage"
> +    rm make.profile
> +    ln -s "${LOONGSON_OV}/profiles/${TARGET_PROFILE}" ./make.profile
> +
> +    # util-linux needs this to not depend on pam, causing circular deps later
> +    sed -i '/^USE=".*"$/s/"$/ -su"/' ./make.conf
> +
> +    mkdir repos.conf
> +    cat > repos.conf/loongson.conf <<EOF
> +[loongson]
> +priority = 50
> +location = $LOONGSON_OV
> +auto-sync = No
> +EOF
> +
> +popd
> +
> +# add build deps for qemu
> +
> +# gawk seems to have problems installing with concurrency, and its deps
> +# include ncurses that needs disabling sandbox to correctly build under
> +# Docker, so just turn off sandbox for all emerges
> +export FEATURES="$FEATURES -sandbox -usersandbox"
> +$CROSS_EMERGE -1 --onlydeps -j$J gawk
> +mkdir -p "/usr/${TRIPLE}/bin"
> +MAKEOPTS='-j1' $CROSS_EMERGE -1 gawk
> +
> +# then build the rest
> +$CROSS_EMERGE -1 --onlydeps -j$J qemu
> +
> +# clean up downloaded files and Portage tree for smaller image
> +rm -rf /var/db/repos/gentoo /var/cache/distfiles


-- 
Alex Bennée