Re: [Qemu-devel] Attaching PCI devices to the PCIe root complex

[Michael S. Tsirkin]

On Fri, Sep 27, 2013 at 07:06:44PM +0200, Markus Armbruster wrote:
> Marcel Apfelbaum <address@hidden> writes:
> 
> > On Wed, 2013-09-25 at 10:01 +0300, Michael S. Tsirkin wrote:
> >> On Tue, Sep 24, 2013 at 06:01:02AM -0400, Laine Stump wrote:
> >> > When I added support for the Q35-based machinetypes to libvirt, I
> >> > specifically prohibited attaching any PCI devices (with the exception of
> >> > graphics controllers) to the PCIe root complex,
> >> 
> >> That's wrong I think. Anything attached to RC is an integrated
> >> endpoint, and these can be PCI devices.
> > I couldn't find on PCIe spec any mention that "Root Complex Integrated 
> > EndPoint"
> > must be PCIe. But, from spec 1.3.2.3:
> > - A Root Complex Integrated Endpoint must not require I/O resources
> > claimed through BAR(s).
> > - A Root Complex Integrated Endpoint must not generate I/O Requests.
> > - A Root Complex Integrated Endpoint is required to support MSI or
> > MSI-X or both if an
> > interrupt resource is requested.
> > I suppose that this restriction can be removed for PCI devices that
> > 1. Actually work when plugged in into RC Integrated EndPoint
> > 2. Respond to the above limitations
> >
> >> 
> >> > and had planned to
> >> > prevent attaching them to PCIe root ports (ioh3420 device) and PCIe
> >> > downstream switch ports (xio-3130 device) as well. I did this because,
> >> > even though qemu currently allows attaching a normal PCI device in any
> >> > of these three places, the restriction exists for real hardware and I
> >> > didn't see any guarantee that qemu wouldn't add the restriction in the
> >> > future in order to more closely emulate real hardware.
> >> > 
> >> > However, since I did that, I've learned that many of the qemu "pci"
> >> > devices really should be considered as "pci or pcie". Gerd Hoffman lists
> >> > some of these cases in a bug he filed against libvirt:
> >> > 
> >> >    https://bugzilla.redhat.com/show_bug.cgi?id=1003983
> >> > 
> >> > I would like to loosen up the restrictions in libvirt, but want to make
> >> > sure that I don't allow something that could later be forbidden by qemu
> >> > (thus creating a compatibility problem during upgrades). Beyond Gerd's
> >> > specific requests to allow ehci, uhci, and hda controllers to attach to
> >> > PCIe ports, are there any other devices that I specifically should or
> >> > shouldn't allow? (I would rather be conservative in what I allow - it's
> >> > easy to allow more things later, but nearly impossible to revoke
> >> > permission once it's been allowed).
> > For the moment I would not remove any restrictions, but only the ones
> > requested and verified by somebody.
> >
> >> 
> >> IMO, we really need to grow an interface to query this kind of thing.
> > Basically libvirt needs to know:
> > 1. for (libvirt) controllers: what kind of devices can be plugged in
> > 2. for devices (controller is also a device)
> >     - to which controllers can it be plugged in
> >     - does it support hot-plug?
> > 3. implicit controllers of the machine types (q35 - "pcie-root", i440fx - 
> > "pci-root")
> > All the above must be exported to libvirt
> >
> > Implementation options:
> > 1. Add a compliance field on PCI/PCIe devices and controllers stating if it 
> > supports
> >    PCI/PCIe or both (and maybe hot-plug)
> >    - consider plug type + compliance to figure out whether a plug can go 
> > into a socket
> >    
> > 2. Use Markus Armbruster idea of introducing a concept of "plug and 
> > sockets":
> >    - dividing the devices into adapters and plugs
> >    - adding sockets to bridges(buses?).
> >    In this way it would be clear which devices can connect to bridges
> 
> This isn't actually my idea.  It's how things are designed to work in
> qdev, at least in my admittedly limited understanding of qdev.
> 
> In traditional qdev, a device has exactly one plug (its "bus type",
> shown by -device help), and it may have one ore more buses.  Each bus
> has a type, and you can plug a device only into a bus of the matching
> type.  This was too limiting, and is not how things work now.
> 
> As far as I know, libvirt already understands that a device can only
> plug into a matching bus.
> 
> In my understanding of where we're headed with qdev, things are / will
> become more general, yet stay really simple:
> 
> * A device can have an arbitrary number of sockets and plugs.
> 
> * Each socket / plug has a type.
> 
> * A plug can go into a socket of the same type, and only there.
> 
> Pretty straightforward generalization of traditional qdev.  I wouldn't
> expect libvirt to have serious trouble coping with it, as long as we
> provide the necessary information on device models' plugs and sockets.
> 
> In this framework, there's no such thing as a device model that can plug
> either into a PCI or a PCIe socket.  Makes sense to me, because there's
> no such thing in the physical world, either.
> Instead, and just like in the physical world, you have one separate
> device variant per desired plug type.
> 

Two types of bus is not how things work in real world though.
In real world there are 3 types of express bus besides
classical pci bus, and limitations on which devices go where
that actually apply to devices qemu emulates.
For example, a dwonstream switch port can only go on
internal virtual express bus of a switch.
Devices with multiple interfaces actually do exist
in real world - e.g. esata/usb - I never heard of a pci/pci express
one but it's not impossible I think.

> To get that, you have to split the device into a common core and bus
> adapters.  You compose the core with the PCI adapter to get the PCI
> device, with the PCIe adapter to get the PCIe device, and so forth.
> I'm not claiming that's the best way to do PCI + PCIe.  It's a purely
> theoretical approach, concerned only with conceptual cleanliness, not
> practical coding difficulties.

I don't mind if that's the internal implementation,
but I don't think we should expose this split
in the user interface.

> What we have now is entirely different: we've overloaded the existing
> PCI plug with all the other PCI-related plug types that came with the
> PCIe support, so it means pretty much nothing anymore.  In particular,
> there's no way for libvirt to figure out progragmatically whether some
> alleged "PCI" device can go into some alleged "PCI" bus.  I call that a
> mess.

There are lots of problems.

First, bus type is not the only factor that can limit
which devices go where.
For example, specific slots might not support hotplug.
Another example, all devices in same pci slot must have
"multifunction" property set.
Second, there's no way to find out what is a valid
bus address. For example, with express you can
only use slot 0.
Hotplug is only supported ATM if no two devices
share a pci slot.
Also, there's apparently no way to figure
out what kind of bus (or multiple buses) is behind each device.


Solution proposed above (separate each device into
two parts) only solves the pci versus express issue without
addressing any of the other issues.
So I'm not sure it's worth the effort.

-- 
MST