Re: [PATCH v7 2/4] Jobs based on custom runners: build environment docs and playbook

[Alex Bennée]

Cleber Rosa <crosa@redhat.com> writes:

> To run basic jobs on custom runners, the environment needs to be
> properly set up.  The most common requirement is having the right
> packages installed.
>
> The playbook introduced here covers the QEMU's project s390x and
> aarch64 machines.  At the time this is being proposed, those machines
> have already had this playbook applied to them.
>
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>  docs/devel/ci.rst                      |  40 +++++++++
>  scripts/ci/setup/.gitignore            |   2 +
>  scripts/ci/setup/build-environment.yml | 116 +++++++++++++++++++++++++
>  scripts/ci/setup/inventory.template    |   1 +
>  4 files changed, 159 insertions(+)
>  create mode 100644 scripts/ci/setup/.gitignore
>  create mode 100644 scripts/ci/setup/build-environment.yml
>  create mode 100644 scripts/ci/setup/inventory.template
>
> diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst
> index 064ffa9988..bfedbb1025 100644
> --- a/docs/devel/ci.rst
> +++ b/docs/devel/ci.rst
> @@ -30,3 +30,43 @@ The GitLab CI jobs definition for the custom runners are 
> located under::
>  Custom runners entail custom machines.  To see a list of the machines
>  currently deployed in the QEMU GitLab CI and their maintainers, please
>  refer to the QEMU `wiki <https://wiki.qemu.org/AdminContacts>`__.
> +
> +Machine Setup Howto
> +-------------------
> +
> +For all Linux based systems, the setup can be mostly automated by the
> +execution of two Ansible playbooks.  Create an ``inventory`` file
> +under ``scripts/ci/setup``, such as this::
> +
> +  fully.qualified.domain
> +  other.machine.hostname
> +
> +You may need to set some variables in the inventory file itself.  One
> +very common need is to tell Ansible to use a Python 3 interpreter on
> +those hosts.  This would look like::
> +
> +  fully.qualified.domain ansible_python_interpreter=/usr/bin/python3
> +  other.machine.hostname ansible_python_interpreter=/usr/bin/python3

I was able to put root@foo for the machines I had in my .ssh/config

> +
> +Build environment
> +~~~~~~~~~~~~~~~~~
> +
> +The ``scripts/ci/setup/build-environment.yml`` Ansible playbook will
> +set up machines with the environment needed to perform builds and run
> +QEMU tests.  This playbook consists on the installation of various
> +required packages (and a general package update while at it).  It
> +currently covers a number of different Linux distributions, but it can
> +be expanded to cover other systems.
> +
> +The minimum required version of Ansible successfully tested in this
> +playbook is 2.8.0 (a version check is embedded within the playbook
> +itself).  To run the playbook, execute::
> +
> +  cd scripts/ci/setup
> +  ansible-playbook -i inventory build-environment.yml
> +
> +Please note that most of the tasks in the playbook require superuser
> +privileges, such as those from the ``root`` account or those obtained
> +by ``sudo``.  If necessary, please refer to ``ansible-playbook``
> +options such as ``--become``, ``--become-method``, ``--become-user``
> +and ``--ask-become-pass``.

If the above works maybe worth mentioning here because just having root
ssh is probably the easiest way to manage a box. 

> diff --git a/scripts/ci/setup/.gitignore b/scripts/ci/setup/.gitignore
> new file mode 100644
> index 0000000000..ee088604d1
> --- /dev/null
> +++ b/scripts/ci/setup/.gitignore
> @@ -0,0 +1,2 @@
> +inventory
> +
> diff --git a/scripts/ci/setup/build-environment.yml 
> b/scripts/ci/setup/build-environment.yml
> new file mode 100644
> index 0000000000..581c1c75d1
> --- /dev/null
> +++ b/scripts/ci/setup/build-environment.yml
> @@ -0,0 +1,116 @@
> +# Copyright (c) 2021 Red Hat, Inc.
> +#
> +# Author:
> +#  Cleber Rosa <crosa@redhat.com>
> +#
> +# This work is licensed under the terms of the GNU GPL, version 2 or
> +# later.  See the COPYING file in the top-level directory.
> +#
> +# This is an ansible playbook file.  Run it to set up systems with the
> +# environment needed to build QEMU.
> +---
> +- name: Installation of basic packages to build QEMU
> +  hosts: all
> +  tasks:
> +    - name: Check for suitable ansible version
> +      delegate_to: localhost
> +      assert:
> +        that:
> +          - '((ansible_version.major == 2) and (ansible_version.minor >= 8)) 
> or (ansible_version.major >= 3)'
> +        msg: "Unsuitable ansible version, please use version 2.8.0 or later"
> +
> +    - name: Update apt cache / upgrade packages via apt
> +      apt:
> +        update_cache: yes
> +        upgrade: yes
> +      when:
> +        - ansible_facts['distribution'] == 'Ubuntu'
> +
> +    - name: Install basic packages to build QEMU on Ubuntu 18.04/20.04
> +      package:
> +        name:
> +        # Originally from tests/docker/dockerfiles/ubuntu1804.docker
> +          - ccache
> +          - gcc
> +          - gettext
> +          - git
> +          - glusterfs-common
> +          - libaio-dev
> +          - libattr1-dev
> +          - libbrlapi-dev
> +          - libbz2-dev
> +          - libcacard-dev
> +          - libcap-ng-dev
> +          - libcurl4-gnutls-dev
> +          - libdrm-dev
> +          - libepoxy-dev
> +          - libfdt-dev
> +          - libgbm-dev
> +          - libgtk-3-dev
> +          - libibverbs-dev
> +          - libiscsi-dev
> +          - libjemalloc-dev
> +          - libjpeg-turbo8-dev
> +          - liblzo2-dev
> +          - libncurses5-dev
> +          - libncursesw5-dev
> +          - libnfs-dev
> +          - libnss3-dev
> +          - libnuma-dev
> +          - libpixman-1-dev
> +          - librados-dev
> +          - librbd-dev
> +          - librdmacm-dev
> +          - libsasl2-dev
> +          - libsdl2-dev
> +          - libseccomp-dev
> +          - libsnappy-dev
> +          - libspice-protocol-dev
> +          - libssh-dev
> +          - libusb-1.0-0-dev
> +          - libusbredirhost-dev
> +          - libvdeplug-dev
> +          - libvte-2.91-dev
> +          - libzstd-dev
> +          - make
> +          - python3-yaml
> +          - python3-sphinx
> +          - python3-sphinx-rtd-theme
> +          - ninja-build
> +          - sparse
> +          - xfslibs-dev
> +        state: present
> +      when:
> +        - ansible_facts['distribution'] == 'Ubuntu'
> +
> +    - name: Install packages to build QEMU on Ubuntu 18.04/20.04 on non-s390x
> +      package:
> +        name:
> +          - libspice-server-dev
> +          - libxen-dev
> +        state: present
> +      when:
> +        - ansible_facts['distribution'] == 'Ubuntu'
> +        - ansible_facts['architecture'] != 's390x'
> +
> +    - name: Install basic packages to build QEMU on Ubuntu 18.04
> +      package:
> +        name:
> +        # Originally from tests/docker/dockerfiles/ubuntu1804.docker
> +          - clang
> +      when:
> +        - ansible_facts['distribution'] == 'Ubuntu'
> +        - ansible_facts['distribution_version'] == '18.04'
> +
> +    - name: Install basic packages to build QEMU on Ubuntu 20.04
> +      package:
> +        name:
> +        # Originally from tests/docker/dockerfiles/ubuntu2004.docker
> +          - clang-10
> +          - genisoimage
> +          - liblttng-ust-dev
> +          - libslirp-dev
> +          - netcat-openbsd
> +      when:
> +        - ansible_facts['distribution'] == 'Ubuntu'
> +        - ansible_facts['distribution_version'] == '20.04'
> diff --git a/scripts/ci/setup/inventory.template 
> b/scripts/ci/setup/inventory.template
> new file mode 100644
> index 0000000000..2fbb50c4a8
> --- /dev/null
> +++ b/scripts/ci/setup/inventory.template
> @@ -0,0 +1 @@
> +localhost

Anyway:

Tested-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>

-- 
Alex Bennée