[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[PATCH v5 00/25] python: create installable package

From: John Snow
Subject: [PATCH v5 00/25] python: create installable package
Date: Thu, 18 Feb 2021 14:27:08 -0500

This series factors the python/qemu directory as an installable
package. It does not yet actually change the mechanics of how any other
python source in the tree actually consumes it (yet), beyond the import
path. (some import statements change in a few places.)

git: https://gitlab.com/jsnow/qemu/-/commits/python-package-mk3

The primary motivation of this series is primarily to formalize our
dependencies on mypy, flake8, isort, and pylint alongside versions that
are known to work. It does this using the setup.cfg and setup.py
files. It also adds explicitly pinned versions (using Pipfile.lock) of
these dependencies that should behave in a repeatable and known way for
developers and CI environments both. Lastly, it enables those CI checks
such that we can enforce Python coding quality checks via the CI tests.

An auxiliary motivation is that this package is formatted in such a way
that it COULD be uploaded to https://pypi.org/project/qemu and installed
independently of qemu.git with `pip install qemu`, but that button
remains *unpushed* and this series *will not* cause any such
releases. We have time to debate finer points like API guarantees and
versioning even after this series is merged.

Some other things this enables that might be of interest:

With the python tooling as a proper package, you can install this
package in editable or production mode to a virtual environment, your
local user environment, or your system packages. The primary benefit of
this is to gain access to QMP tooling regardless of CWD, without needing
to battle sys.path (and confounding other python analysis tools).

For example: when developing, you may go to qemu/python/ and run `make
venv` followed by `pipenv shell` to activate a virtual environment that
contains the qemu python packages. These packages will always reflect
the current version of the source files in the tree. When you are
finished, you can simply exit the shell (^d) to remove these packages
from your python environment.

When not developing, you could install a version of this package to your
environment outright to gain access to the QMP and QEMUMachine classes
for lightweight scripting and testing by using pip: "pip install [--user] ."


First of all, nothing should change. Without any intervention,
everything should behave exactly as it was before. The only new
information here comes from how to interact with and run the linters
that will be enforcing code quality standards in this subdirectory.

To test those, CD to qemu/python first, and then:

1. Try "make venv && pipenv shell" to get a venv with the package
installed to it in editable mode. Ctrl+d exits this venv shell. While in
this shell, any python script that uses "from qemu.[qmp|machine] import
..." should work correctly regardless of where the script is, or what
your CWD is.

You will need Python 3.6 installed on your system to do this step. For
Fedora: "dnf install python36" will do the trick.

2. Try "make check" while still in the shell to run the Python linters
using the venv built in the previous step. This will pull some packages
from PyPI and run pytest, which will in turn execute mypy, flake8, isort
and pylint with the correct arguments.

3. Having exited the shell from above, try "make venv-check". This will
create and update the venv if needed, then run 'make check' within the
context of that shell. It should pass as long as the above did.

4. Still outside of the venv, you may try running "make check". This
will not install anything, but unless you have the right Python
dependencies installed, these tests may fail for you. You might try
using "pip install --user .[devel]" to install the development packages
needed to run the tests successfully to your local user's python
environment. Once done, you will probably want to "pip uninstall qemu"
to remove that package to avoid it interfering with other things.

5. "make distclean" will delete the venv and any temporary files that
may have been created by packaging, installing, testing, etc.

6. You may also (if you wish) create your own environment using a Python
other than 3.6, bypassing pipenv. This may be useful for environments in
which you simply do not have Python 3.6 readily available.

Using the Python of your choice:

> make distclean # (Remove the .venv, if you created one.)
> /usr/bin/python3.9 -m venv .venv
> source .venv/bin/activate  # (or activate.[fish|csh])

> make develop  # Installs linter deps to this venv
> make check    # Runs tests using current env
> deactivate    # Leaves venv.

This *should* pass, and if it doesn't, it's a bug. Due to dependencies
changing like shifting sands, it's impossible to promise support for
every last combination of dependency and python version. This is why the
"make venv-check" target uses Python 3 and a very explicit set of
packages instead. Still, this form *should* pass.



001/25:[----] [--] 'python/console_socket: avoid one-letter variable'
002/25:[----] [--] 'iotests/297: add --namespace-packages to mypy arguments'
003/25:[0009] [FC] 'python: create qemu packages'
004/25:[0012] [FC] 'python: create utils sub-package'
005/25:[0002] [FC] 'python: add qemu package installer'
006/25:[----] [--] 'python: add VERSION file'
007/25:[0012] [FC] 'python: add directory structure README.rst files'
008/25:[----] [--] 'python: Add pipenv support'
009/25:[----] [--] 'python: add pylint import exceptions'
010/25:[----] [--] 'python: move pylintrc into setup.cfg'
011/25:[----] [--] 'python: add pylint to pipenv'
012/25:[----] [--] 'python: move flake8 config to setup.cfg'
013/25:[----] [-C] 'python: add excluded dirs to flake8 config'
014/25:[----] [--] 'python: Add flake8 to pipenv'
015/25:[----] [-C] 'python: move mypy.ini into setup.cfg'
016/25:[----] [-C] 'python: add mypy to pipenv'
017/25:[----] [--] 'python: move .isort.cfg into setup.cfg'
018/25:[----] [--] 'python/qemu: add isort to pipenv'
019/25:[----] [--] 'python/qemu: add qemu package itself to pipenv'
020/25:[----] [-C] 'python: add devel package requirements to setuptools'
021/25:[----] [--] 'python: add pytest and tests'
022/25:[0010] [FC] 'python: add Makefile for some common tasks'
023/25:[----] [--] 'python: add .gitignore'
024/25:[----] [--] 'gitlab: add python linters to CI'
025/25:[down] 'python: add 'make develop' target'

-   Shuffled flake8 patch to the flake8 region of the patchset
    (Didn't move the pylint patch down: It was annoying and stands on its own.)
    Added RB/TB tags where applicable, see chart below this section.
03: Changed copyright year
    Updated import fixes for latest origin/master
04: Updated import fixes for latest origin/master
05: Updated commit message ("ReST" vs "rst")
    Added information on PEP517/518 adoption to commit message.
07: Make clear that "qemu.qmp", "qemu.machine" are not exhaustive.
    Replace "amend" by "append"
    Drastically shorten python/qemu/utils/README.rst
22: Update Makefile with new Hint for which test you should run if unsure.
    Fix typo (qemu.egg.info => qemu.egg-info)
    Have "make venv" touch .venv to avoid redundant checking where possible
    Update help text for what "make distclean" is removing.
25: New, just to facilitate creating your own venv, in the case
    that you cannot easily install python 3.6, but still want
    to run tests on your local machine easily.

Current Status

+ [01] python-console_socket-avoid    # [RB] CR [SOB] JS
+ [02] iotests-297-add-namespace      # [RB] CR [SOB] JS
+ [03] python-create-qemu-packages    # [SOB] JS
+ [04] python-create-utils-sub        # [SOB] JS
+ [05] python-add-qemu-package        # [SOB] JS
+ [06] python-add-version-file        # [RB] CR [SOB] JS
+ [07] python-add-directory-structure # [SOB] JS
+ [08] python-add-pipenv-support      # [RB] CR [SOB] JS
+ [09] python-add-pylint-import       # [RB] CR [SOB] JS
+ [10] python-move-pylintrc-into      # [RB] CR [TB] CR [SOB] JS
+ [11] python-add-pylint-to-pipenv    # [RB] CR [TB] CR [SOB] JS
+ [12] python-move-flake8-config-to   # [RB] CR [SOB] JS
+ [13] python-add-excluded-dirs-to    # [SOB] JS
+ [14] python-add-flake8-to-pipenv    # [RB] CR [TB] CR [SOB] JS
+ [15] python-move-mypy-ini-into      # [SOB] JS
+ [16] python-add-mypy-to-pipenv      # [RB] CR [TB] CR [SOB] JS
+ [17] python-move-isort-cfg-into     # [RB] CR [SOB] JS
+ [18] python-qemu-add-isort-to       # [RB] CR,PM [TB] CR [SOB] JS
+ [19] python-qemu-add-qemu-package   # [RB] CR [TB] CR [SOB] JS
+ [20] python-add-devel-package       # [SOB] JS
+ [21] python-add-pytest-and-tests    # [SOB] JS
+ [22] python-add-makefile-for-some   # [SOB] JS
+ [23] python-add-gitignore           # [SOB] JS
+ [24] gitlab-add-python-linters-to   # [SOB] JS
> [25] python-add-make-develop-target # [SOB] JS

Looking for reviews on:

+ [03] python-create-qemu-packages    # python: create qemu packages
+ [04] python-create-utils-sub        # python: create utils sub-package
+ [05] python-add-qemu-package        # python: add qemu package installer
+ [07] python-add-directory-structure # python: add directory structure 
README.rst files
+ [13] python-add-excluded-dirs-to    # python: add excluded dirs to flake8 
+ [15] python-move-mypy-ini-into      # python: move mypy.ini into setup.cfg
+ [20] python-add-devel-package       # python: add devel package requirements 
to setuptools
+ [21] python-add-pytest-and-tests    # python: add pytest and tests
+ [22] python-add-makefile-for-some   # python: add Makefile for some common 
+ [23] python-add-gitignore           # python: add .gitignore
+ [24] gitlab-add-python-linters-to   # gitlab: add python linters to CI
> [25] python-add-make-develop-target # python: add 'make develop' target

Misc Notes

Reviewer notes:

- The VERSION hack may be imperfect, but at 0.x and without uploading it
  to PyPI, we have *all* the time in the world to fine-tune it later.
- The CI integration may not be perfect, but it is better than *nothing*,
  so I think it's worth doing even in an imperfect state.
- Patch 25 is just a bonus, because I can't stop myself from making changes.

John Snow (25):
  python/console_socket: avoid one-letter variable
  iotests/297: add --namespace-packages to mypy arguments
  python: create qemu packages
  python: create utils sub-package
  python: add qemu package installer
  python: add VERSION file
  python: add directory structure README.rst files
  python: Add pipenv support
  python: add pylint import exceptions
  python: move pylintrc into setup.cfg
  python: add pylint to pipenv
  python: move flake8 config to setup.cfg
  python: add excluded dirs to flake8 config
  python: Add flake8 to pipenv
  python: move mypy.ini into setup.cfg
  python: add mypy to pipenv
  python: move .isort.cfg into setup.cfg
  python/qemu: add isort to pipenv
  python/qemu: add qemu package itself to pipenv
  python: add devel package requirements to setuptools
  python: add pytest and tests
  python: add Makefile for some common tasks
  python: add .gitignore
  gitlab: add python linters to CI
  python: add 'make develop' target

 python/PACKAGE.rst                          |  32 +++
 python/README.rst                           |  41 +++
 python/qemu/README.rst                      |   8 +
 python/qemu/machine/README.rst              |   9 +
 python/qemu/qmp/README.rst                  |   9 +
 python/qemu/utils/README.rst                |   7 +
 .gitlab-ci.yml                              |  10 +
 python/.gitignore                           |   9 +
 python/Makefile                             |  42 +++
 python/Pipfile                              |  13 +
 python/Pipfile.lock                         | 285 ++++++++++++++++++++
 python/VERSION                              |   1 +
 python/mypy.ini                             |   4 -
 python/qemu/.flake8                         |   2 -
 python/qemu/.isort.cfg                      |   7 -
 python/qemu/__init__.py                     |  11 -
 python/qemu/machine/__init__.py             |  36 +++
 python/qemu/{ => machine}/console_socket.py |  10 +-
 python/qemu/{ => machine}/machine.py        |  16 +-
 python/qemu/{ => machine}/qtest.py          |   3 +-
 python/qemu/pylintrc                        |  58 ----
 python/qemu/{qmp.py => qmp/__init__.py}     |  12 +-
 python/qemu/utils/__init__.py               |  23 ++
 python/qemu/{ => utils}/accel.py            |   0
 python/setup.cfg                            |  82 ++++++
 python/setup.py                             |  23 ++
 python/tests/test_lint.py                   |  28 ++
 tests/acceptance/avocado_qemu/__init__.py   |   4 +-
 tests/acceptance/virtio-gpu.py              |   2 +-
 tests/docker/dockerfiles/fedora.docker      |   2 +
 tests/qemu-iotests/297                      |   1 +
 tests/qemu-iotests/300                      |   4 +-
 tests/qemu-iotests/iotests.py               |   2 +-
 tests/vm/aarch64vm.py                       |   2 +-
 tests/vm/basevm.py                          |   2 +-
 35 files changed, 698 insertions(+), 102 deletions(-)
 create mode 100644 python/PACKAGE.rst
 create mode 100644 python/README.rst
 create mode 100644 python/qemu/README.rst
 create mode 100644 python/qemu/machine/README.rst
 create mode 100644 python/qemu/qmp/README.rst
 create mode 100644 python/qemu/utils/README.rst
 create mode 100644 python/.gitignore
 create mode 100644 python/Makefile
 create mode 100644 python/Pipfile
 create mode 100644 python/Pipfile.lock
 create mode 100644 python/VERSION
 delete mode 100644 python/mypy.ini
 delete mode 100644 python/qemu/.flake8
 delete mode 100644 python/qemu/.isort.cfg
 delete mode 100644 python/qemu/__init__.py
 create mode 100644 python/qemu/machine/__init__.py
 rename python/qemu/{ => machine}/console_socket.py (95%)
 rename python/qemu/{ => machine}/machine.py (98%)
 rename python/qemu/{ => machine}/qtest.py (98%)
 delete mode 100644 python/qemu/pylintrc
 rename python/qemu/{qmp.py => qmp/__init__.py} (96%)
 create mode 100644 python/qemu/utils/__init__.py
 rename python/qemu/{ => utils}/accel.py (100%)
 create mode 100644 python/setup.cfg
 create mode 100755 python/setup.py
 create mode 100644 python/tests/test_lint.py


reply via email to

[Prev in Thread] Current Thread [Next in Thread]