On Fri, May 13, 2022 at 09:35:23AM +0100, Daniel P. Berrangé wrote:
> On Thu, May 12, 2022 at 08:06:00PM -0400, John Snow wrote:
> > RFC: This is a very early, crude attempt at switching over to an
> > external Python package dependency for QMP. This series does not
> > actually make the switch in and of itself, but instead just switches to
> > the paradigm of using a venv in general to install the QEMU python
> > packages instead of using PYTHONPATH to load them from the source tree.
> >
> > (By installing the package, we can process dependencies.)
> >
> > I'm sending it to the list so I can show you some of what's ugly so far
> > and my notes on how I might make it less ugly.
> >
> > (1) This doesn't trigger venv creation *from* iotests, it merely prints
> > a friendly error message if "make check-venv" has not been run
> > first. Not the greatest.
>
> So if we run the sequence
>
> mkdir build
> cd build
> ../configure
> make
> ./tests/qemu-iotests/check 001
>
> It won't work anymore, until we 'make check-venv' (or simply
> 'make check') ?
>
> I'm somewhat inclined to say that venv should be created
> unconditionally by default. ie a plain 'make' should always
> everything needed to be able to invoke the tests directly.
>
> > (2) This isn't acceptable for SRPM builds, because it uses PyPI to fetch
> > packages just-in-time. My thought is to use an environment variable like
> > QEMU_CHECK_NO_INTERNET that changes the behavior of the venv setup
> > process. We can use "--system-site-packages" as an argument to venv
> > creation and "--no-index" as an argument to pip installation to achieve
> > good behavior in SRPM building scenarios. It'd be up to the spec-writer
> > to opt into that behavior.
>
> I think I'd expect --system-site-packages to be the default behaviour.
> We expect QEMU to be compatible with the packages available in the
> distros that we're targetting. So if the dev has the python packages
> installed from their distro, we should be using them preferentially.
>
> This is similar to how we bundle slirp/capstone/etc, but will
> preferentially use the distro version if it is available.
AFAICT from testing it, when '--system-site-packages' is set
for the venv, then 'pip install' appears to end up being a
no-op if the package is already present in the host, but
installs it if missing.
IOW, if we default to --system-site-packages, but still
also run 'pip install', we should "do the right thing".
It'll use any distro packages that are available, and
augment with stuff from pip. In the no-op case, pip will
still try to consult the pipy servers to fetch version
info IIUC, so we need to be able to stop that. So I'd
suggest a --python-env arg taking three values
* "auto" (the default) - add --system-site-packages, but
also run 'pip install'. Good for developers day-to-day
Sounds like a decent balance...
...My only concern is that the system packages might be very old and it's possible that the qemu packages will be "too new" or have conflicts with the system deps.
I'll just have to test this.
e.g. what if I want to require mypy >= 0.900 for testing, but you have a system package that requires mypy < 0.700?
I don't *know* that this is a problem, but my python-sense is tingling.
The python dep compatibility matrix I try to enforce and support for testing is already feeling overly wide. This might force me to support an even wider matrix, which I think is the precisely wrong direction for venvs where we want tighter control as a rule.
* "system" - add --system-site-packages but never run
'pip install'. Good for formal package builds.
We still have to install the in-tree qemu ns package, but we can use --no-index to do it. It'll fail if the deps aren't met.
* "pip" - don't add --system-site-packages, always run
'pip install'. Good for testing compatibility with
bleeding edge python versions, or if explicit full
independance from host python install is desired.
as arguments to configure, this spread of options makes sense to me than paolo's version, but I've still got some doubt on mixing system and venv packages.
I am also as of yet not sold on building the venv *from* configure, see my response to Paolo on that topic.
I'll keep plugging away for now, but the big picture is still a tad murky in my head.
--js