[PATCH v2 00/27] configure: create a python venv and ensure meson, sphinx

[Paolo Bonzini]

This patch series creates a mandatory python virtual environment
("venv") during configure time and uses it to ensure the availability of
meson and sphinx.

See https://www.qemu.org/2023/03/24/python/ for motivations. The summary
is that the goal of this series is to ensure that the `python` used to
run meson is the same `python` used to run Sphinx, tests, and any
build-time python scripting we have. As it stands, meson and sphinx (and
their extensions) *may* run in a different python environment than the
one configured and chosen by the user at configure/build time.

The effective change of this series is that QEMU will now
unconditionally create a venv at configure-time and will ensure that
meson (and sphinx, if docs are enabled) are available through that venv.

Some important points as a pre-emptive "FAQ":

- Works for Python 3.6 and up, on Fedora, OpenSuSE, Red Hat, CentOS,
  Alpine, Debian, Ubuntu, NetBSD, OpenBSD, and hopefully everywhere
  No new dependencies (...for most platforms. Debian and NetBSD get an
  asterisk, see patch 4).

- The venv is unconditionally created and lives at {build_dir}/pyvenv.
  The python interpreter used by this venv is always the one identified
  by configure. (Which in turn is always the one specified by --python
  or $PYTHON).  --meson and --sphinx-build disappear.

- *almost* all python scripts in qemu.git executed as part of the build
  system, meson, sphinx, avocado tests, vm tests or CI are always
  executed within this venv.  iotests are not yet integrated.

- Missing dependencies, when possible, are fetched and installed
  on-demand automatically to make developer environments "just work".
  However, it remains possible to build and test fully offline.  Right
  now, online behavior is only triggered by --enable-docs, and even
  in that case...

- ... distribution packaged 'meson' and 'sphinx' are still utilized
  whenever possible as the highest preference.

- The bundled 'meson' takes the shape of a .whl file, replacing the
  git submodule.  qemu-qmp will follow suit, while Sphinx will not be
  bundled just like it already isn't.

- about 13% of the mkvenv.py script is devoted to supporting Python
  3.7 and Debian 10. ¯\_(ツ)_/¯

Tested with "make check-minreqs" after every patch and with GitLab at
https://gitlab.com/bonzini/qemu/-/pipelines/868627762.

Enjoy!

John and Paolo

v1->v2:
- integrate distlib for script generation and also to look up
  installed packages
- print a more precise error for --diagnose when running in an
  isolated venv
- reorganized code to avoid ensurepip, so that it is not spread
  across multiple patches
- made non-fatal errors a bit less scary


John Snow (23):
  python: update pylint configuration
  python: add mkvenv.py
  mkvenv: add better error message for broken or missing ensurepip
  mkvenv: add nested venv workaround
  mkvenv: add ensure subcommand
  mkvenv: add --diagnose option to explain "ensure" failures
  mkvenv: add console script entry point generation
  mkvenv: use pip's vendored distlib as a fallback
  mkvenv: avoid ensurepip if pip is installed
  mkvenv: work around broken pip installations on Debian 10
  tests/docker: add python3-venv dependency
  tests/vm: Configure netbsd to use Python 3.10
  tests/vm: add py310-expat to NetBSD
  python: add vendor.py utility
  configure: create a python venv unconditionally
  python/wheels: add vendored meson package
  configure: use 'mkvenv ensure meson' to bootstrap meson
  qemu.git: drop meson git submodule
  tests: Use configure-provided pyvenv for tests
  configure: move --enable-docs and --disable-docs back to configure
  configure: bootstrap sphinx with mkvenv
  configure: add --enable-pypi and --disable-pypi
  configure: Add courtesy hint to Python version failure message

Paolo Bonzini (4):
  python: shut up "pip install" during "make check-minreqs"
  Python: Drop support for Python 3.6
  mkvenv: mark command as required
  python: bump some of the dependencies

 .gitlab-ci.d/buildtest-template.yml           |   4 +-
 .gitlab-ci.d/buildtest.yml                    |   6 +-
 .gitmodules                                   |   3 -
 configure                                     | 152 +--
 docs/about/build-platforms.rst                |   2 +-
 docs/conf.py                                  |   9 -
 docs/devel/acpi-bits.rst                      |   6 +-
 docs/devel/testing.rst                        |  14 +-
 docs/meson.build                              |   2 +-
 meson                                         |   1 -
 meson_options.txt                             |   2 -
 python/Makefile                               |  19 +-
 python/scripts/mkvenv.py                      | 897 ++++++++++++++++++
 python/scripts/vendor.py                      |  74 ++
 python/setup.cfg                              |  46 +-
 python/tests/flake8.sh                        |   1 +
 python/tests/isort.sh                         |   1 +
 python/tests/minreqs.txt                      |  19 +-
 python/tests/mypy.sh                          |   1 +
 python/tests/pylint.sh                        |   1 +
 python/wheels/meson-0.61.5-py3-none-any.whl   | Bin 0 -> 862509 bytes
 .../org.centos/stream/8/x86_64/test-avocado   |   4 +-
 scripts/device-crash-test                     |   2 +-
 scripts/meson-buildoptions.sh                 |   3 -
 scripts/qapi/mypy.ini                         |   2 +-
 tests/Makefile.include                        |  10 +-
 .../dockerfiles/debian-all-test-cross.docker  |   3 +-
 .../dockerfiles/debian-hexagon-cross.docker   |   3 +-
 .../dockerfiles/debian-riscv64-cross.docker   |   3 +-
 .../dockerfiles/debian-tricore-cross.docker   |   3 +-
 tests/requirements.txt                        |   7 +-
 tests/vm/netbsd                               |   2 +
 32 files changed, 1167 insertions(+), 135 deletions(-)
 delete mode 160000 meson
 create mode 100644 python/scripts/mkvenv.py
 create mode 100755 python/scripts/vendor.py
 create mode 100644 python/wheels/meson-0.61.5-py3-none-any.whl

-- 
2.40.1