|Subject:||Disabling PCI "hot-unplug" for a guest (and/or a single PCI device)|
|Date:||Mon, 3 Feb 2020 17:19:51 -0500|
|User-agent:||Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.4.1|
To make the above-mentioned admins happy, we need to make it possible to (easily) create guest configurations for q35-based virtual machines where the PCI devices can't be hot-unplugged by the guest OS.
Thinking in the context of a management platform (e.g. OpenStack or ovirt) that goes through libvirt to use QEMU (and forgetting about i440fx, concentrating only on q35), I can think of a few different ways this could be done:
1) Rather than leaving the task of assignung the PCI addresses of devices to libvirt (which is what essentially *all* management apps that use libvirt currently do), the management application could itself directly assign the PCI addressed of all devices to be slots on pcie.0.
This is problematic because once a management application has taken over the PCI address assignment of a single device, it must learn the rules of what type of device can be plugged into what type of PCI controller (including plugging in new controllers when necessary), and keep track of which slots on which PCI controllers are already in use - effectively tossing that part of libvirt's functionality / embedded knowledge / usefulness to management applications out the window. It's even more of a problem for management applications that have no provision for manually assigning PCI addresses - virt-manager for example only supports this by using "XML mode" where the froopy point-click UI is swapped out for an edit window where the user is simply presented with the full XML for a device and allowed to tweak it around as they see fit (including duplicate addresses, plugging the wrong kind of device into the wrong slot, referencing non-existent controllers, etc). (NB: you could argue that management could just take over PCI address assignment in the case of wanting hotplug disabled, and only care about / support pcie.0 (which makes the task much easier, since you just ignore the existence of any other PCI controllers, leaving you with a homogenous array of 32 slot x 8 functions, but becomes much more complicated if you want to allow a mix of hotpluggable and non-hotpluggable devices, and you *know* someone will)
2) libvirt could gain a knob "somewhere" in the domain XML to force a single device, or all devices, to be assigned to a PCI address on pcie.0 rather than on a pcie-root-port. This could be thought of as a "hint" about device placement, as well as extra validation in the case that a PCI address has been manually assigned. So, for example, let's say a "hotplug='disable'" option is added somewhere at the top level of the domain (maybe "<hotplug enable='no'/>" inside <features> or something like that); when PCI addresses are assigned by libvirt, it would attempt to find a slot on a controller that didn't support hotplug. And/or a similar knob could be added to each device. In both cases, the setting would be used both when assigning PCI addresses and also to validate user-provided PCI addresses to assure that the desired criterion was met (otherwise someone would manually select a PCI address on a controller that supported hotplug, but then set "hotplug='disabled'" and expect hotplug to be magically disabled on the slot).
Some of you will remember that I proposed such a knob for libvirt a few years ago when we were first fleshing out support for QEMU's PCI Express controllers and the Q35 machinetype, and it was rejected as "libvirt dictating policy". Of course at that time there weren't actual users demanding the functionality, and now there are. Aside from that, all I can say is that it isn't libvirt dictating this policy, it's the user of libvirt, and libvirt is just following directions :-) (and that I really really dislike the idea of a forced handover of the entire task of assigning/managing device PCI addresses to management apps just because they decide they want to disable guest-initiated hotplug
3) qemu could add a "hotpluggable=no" commandline option to all PCI devices (including vfio-pci) and then do whatever is necessary to make sure this is honored in the emulated hardware (is it possible to set this on a per-slot basis in a PCI controller? Or must it be done for an entire controller? I suppose it's not as much of an issue for pcie-root-port, as long as you're not using multiple functions). libvirt would then need to add this option to the XML for each device, and management applications would need to set it - it would essentially look the same to the management application, but it would be implemented differently - instead of libvirt using that flag to make a choice about which slot to assign, it would assign PCI addresses in the same manner as before, and use the libvirt XML flag to set a QEMU commandline flag for the device.
The upside of this is that we would be disabling hotplug by "disabling hotplug" rather than by "assigning the device to a slot that coincidentally doesn't support hotplug", making it all more orthogonal - everything else in a guest's config could remain exactly the same while enabling/disabling hotplug. (Another upside is that it could possibly be made to work for i440fx machine types, but we're not supposed to care about that any more, so I won't mention it :-)) The downside is that it requires a new feature in QEMU (whose difficulty/feasibility I have 0 knowledge of), so there are 3 layers of work rather than 2.
So does anyone have any different (and hopefully better) idea of how to do this? Arguments for/against the 3 possibilities I've listed here?
|[Prev in Thread]||Current Thread||[Next in Thread]|