On Tue, 2016-12-13 at 14:25 +0200, Marcel Apfelbaum wrote:
Hrm, the suggestion of providing both a vanilla-PCI and PCI-E host
bridge came up before. I think one of us spotted a problem with that,
but I don't recall what it was now. I guess one is how libvirt would
map it's stupid-fake-domain-numbers to which root bus to use.
This would be a weird configuration, I never heard of something like that
on a bare metal machine, but I never worked on pseries, who knows...
That's the thing though, it's *not* bare metal ;-)
On bare metal, we use the "powernv" platform for which we haven't yet upstreamed
the PCIE support, but there we have real PCIe root ports with all what's
exepcted
on them.
They come on physically different PHB, one root complex per PHB, but they are
"proper" PCIe. Hotplug is a bit weird still because it has to go through some
FW interactions as the HW doesn't use the PCIe standard slot control bits (and
our qemu model doesn't handle it yet) but otherwise it's standard.
"pseries" on the other hand is a paravirtualized platform. It's the environment
presented to guests by our propritary hypervisor (PowerVM, aka pHyp) and
KVM/qemu. I think there's a public spec these days but to cut the story short,
it doesn't expose PCIe "properly" for bad historical reasons.
It tends to hide the parent, ie, the root port for slots connected directly to
a PHB or the entire hierarchy from the root to (and including) the downstream
switch port for slots under as switch.
So you end up with PCIe devices devoid of a proper "parent" which is a bit
weird.
Hotplug is implemented using specific firmware interfaces that are identical
with pre-PCIe (ie PCI/PCI-X) systems. The FW has interface for supporting 3
kinds of hotplug:
- Entire PHBs
- P2P Bridges
- Individual devices on an existing PHB or P2P bridge
In practice these days mostly the first one is used for everything under pHyp,
though I think we have a mix of 1 and 3 for KVM/qemu.
Cheers,
Ben.