Re: [PATCH RFC 00/32] python/qemu: refactor as installable package

[John Snow]

Ping, please give this series a look-over. It looks big, but the changes
themselves are actually fairly tiny.

Here's a TOC:

Patch 1 moves files and renames import statements.
Patches 2-3 do some basic delinting.
Patches 4-5 do more basic delinting and add flake8/pylintrc cfg
Patches 6-10 add setup.py, pipfile, a readme, etc.
Patch 11 is a final bit of removing python2 isms.

Patches 12-32 (!) are all mypy typing fixes, and I have broken these out
in great care:

Patches 12-30 all fix one mypy issue each with bite-sized refactors each.

Patch 31 adds a mypy configuration to the package.

Patch 32 is a giant patch that has __NO__ runtime side-effects, it JUST
adds the remaining mypy annotations.


At the end of the series, you should find that mypy *strict*, flake8,
and pylint all pass 100%.

(Note: you MAY need specific versions of these tools to have them pass.
The pipfile included will help you target the correct versions.)

--js

On 5/14/20 1:53 AM, John Snow wrote:
> Hey, I got lost on my way to the store and I accidentally got 32 patches
> that convert our python library into something that passes pylint,
> flake8, and mypy --strict.
> 
> ...So, a few things:
> 
> 1. This is just an RFC. The actual design of these libraries really
> needs adjusted to be more pythonic. In general this means less
> Optional[T] return types and raising more exceptions. This could be
> handled later, but we ought to address it before publishing, if we do.
> 
> 2. We still need to think carefully about how we package this, if we
> even want to package it, what the license on the top-level package
> should be, etc.
> 
> 3. We should consider how to version it. For now, I'm using a lockstep
> versioning.
> 
> 4. You can install this package using pip3 or python3 setup.py to a
> virtual environment or to your real one. From there, any python code in
> the QEMU tree that imports these modules will work with no sys.path
> hacking or custom PYTHONPATH exports.
> 
> 5. You don't have to install it, though. I left all of the usual hacks
> in place in the rest of the tree so that everything will just keep
> working exactly as-is for right now. It's just that you COULD install it.
> 
> 6. Here's a cool trick if you don't know about it yet:
> 
>> cd qemu/python/qemu
>> pip3 install --user -e .
> 
> This will install the package in "develop" mode, which installs a
> forwarder package. When you update your source tree, the installed
> package stays "up to date" with the most recent edits.
> 
> Alright, have fun, stay safe!
> 
> John Snow (32):
>   python/qemu: create qemu.lib module
>   scripts/qmp: Fix shebang and imports
>   python//machine.py: remove bare except
>   python/qemu/lib: delint, add pylintrc
>   python/qemu/lib: delint; add flake8 config
>   python/qemu: formalize as package
>   python/qemu: add README.rst
>   python/qemu: Add Pipfile
>   python/qemu: add pylint to Pipfile
>   python/qemu: Add flake8 to Pipfile
>   python/qemu/lib: remove Python2 style super() calls
>   python/qemu/lib: fix socket.makefile() typing
>   python/qemu/lib: Adjust traceback typing
>   python//qmp.py: use True/False for non/blocking modes
>   python//qmp.py: Define common types
>   python//qmp.py: re-absorb MonitorResponseError
>   python//qmp.py: Do not return None from cmd_obj
>   python//qmp.py: add casts to JSON deserialization
>   python//qmp.py: add QMPProtocolError
>   python//qmp.py: assert sockfile is not None
>   python//machine.py: remove logging configuration
>   python//machine.py: Fix monitor address typing
>   python//machine.py: reorder __init__
>   python//machine.py: Don't modify state in _base_args()
>   python//machine.py: Handle None events in event_wait
>   python//machine.py: use qmp.command
>   python//machine.py: Add _qmp access shim
>   python//machine.py: fix _popen access
>   python//qtest.py: Check before accessing _qtest
>   python/qemu/lib: make 'args' style arguments immutable
>   python/qemu: add mypy to Pipfile
>   python/qemu/lib: Add mypy type annotations
> 
>  python/README.rst                         |   6 +
>  python/qemu/README.rst                    |   8 +
>  python/Pipfile                            |  14 +
>  python/Pipfile.lock                       | 187 +++++++++++++
>  python/qemu/__init__.py                   |  11 -
>  python/qemu/lib/.flake8                   |   2 +
>  python/qemu/lib/__init__.py               |  57 ++++
>  python/qemu/{ => lib}/accel.py            |  17 +-
>  python/qemu/{ => lib}/machine.py          | 320 +++++++++++++---------
>  python/qemu/lib/pylintrc                  |  58 ++++
>  python/qemu/{ => lib}/qmp.py              | 140 +++++++---
>  python/qemu/lib/qtest.py                  | 160 +++++++++++
>  python/qemu/qtest.py                      | 119 --------
>  python/setup.py                           |  50 ++++
>  scripts/device-crash-test                 |   2 +-
>  scripts/qmp/qemu-ga-client                |   2 +-
>  scripts/qmp/qmp                           |   4 +-
>  scripts/qmp/qmp-shell                     |   2 +-
>  scripts/qmp/qom-fuse                      |   4 +-
>  scripts/qmp/qom-get                       |   6 +-
>  scripts/qmp/qom-list                      |   4 +-
>  scripts/qmp/qom-set                       |   6 +-
>  scripts/qmp/qom-tree                      |   6 +-
>  scripts/render_block_graph.py             |   5 +-
>  scripts/simplebench/bench_block_job.py    |   4 +-
>  tests/acceptance/avocado_qemu/__init__.py |   2 +-
>  tests/acceptance/boot_linux.py            |   3 +-
>  tests/acceptance/virtio_check_params.py   |   2 +-
>  tests/acceptance/virtio_version.py        |   2 +-
>  tests/migration/guestperf/engine.py       |   2 +-
>  tests/qemu-iotests/235                    |   2 +-
>  tests/qemu-iotests/iotests.py             |   2 +-
>  tests/vm/basevm.py                        |   6 +-
>  33 files changed, 881 insertions(+), 334 deletions(-)
>  create mode 100644 python/README.rst
>  create mode 100644 python/qemu/README.rst
>  create mode 100644 python/Pipfile
>  create mode 100644 python/Pipfile.lock
>  delete mode 100644 python/qemu/__init__.py
>  create mode 100644 python/qemu/lib/.flake8
>  create mode 100644 python/qemu/lib/__init__.py
>  rename python/qemu/{ => lib}/accel.py (86%)
>  rename python/qemu/{ => lib}/machine.py (67%)
>  create mode 100644 python/qemu/lib/pylintrc
>  rename python/qemu/{ => lib}/qmp.py (70%)
>  create mode 100644 python/qemu/lib/qtest.py
>  delete mode 100644 python/qemu/qtest.py
>  create mode 100755 python/setup.py
> 

-- 
—js