Re: [Qemu-devel] [PATCH RFC] docs: add PCIe devices placement guidelines

[Andrea Bolognani]

On Tue, 2016-10-04 at 12:52 -0600, Alex Williamson wrote:
> > I'ts all just idle number games, but what I was thinking of was the 
> > difference between plugging  a bunch of root-port+upstream+downstreamxN 
> > combos directly into pcie-root (flat), vs. plugging the first into 
> > pcie-root, and then subsequent ones into e.g. the last downstream port 
> > of the previous set. Take the simplest case of needing 63 hotpluggable 
> > slots. In the "flat" case, you have:
> > 
> >     2 x pcie-root-port
> >     2 x pcie-switch-upstream-port
> >     63 x pcie-switch-downstream-port
> > 
> > In the "nested" or "chained" case you have:
> > 
> >     1 x pcie-root-port
> >     1 x pcie-switch-upstream-port
> >     32 x pcie-downstream-port
> >     1 x pcie-switch-upstream-port
> >     32 x pcie-switch-downstream-port
> 
> You're not thinking in enough dimensions.  A single root port can host
> multiple sub-hierarchies on it's own.  We can have a multi-function
> upstream switch, so you can have 8 upstream ports (00.{0-7}).  If we
> implemented ARI on the upstream ports, we could have 256 upstream ports
> attached to a single root port, but of course then we've run out of
> bus numbers before we've even gotten to actual devices buses.
> 
> Another option, look at the downstream ports, why do they each need to
> be in separate slots?  We have the address space of an entire bus to
> work with, so we can also create multi-function downstream ports, which
> gives us 256 downstream ports per upstream port.  Oops, we just ran out
> of bus numbers again, but at least actual devices can be attached.

What's the advantage in using ARI to stuff more than eight
of anything that's not Endpoint Devices in a single slot?

I mean, if we just fill up all 32 slots in a PCIe Root Bus
with 8 PCIe Root Ports each we already end up having 256
hotpluggable slots[1]. Why would it be preferable to use
ARI, or even PCIe Switches, instead?


[1] The last slot will have to be limited to 7 PCIe Root
    Ports if we don't want to run out of bus numbers
-- 
Andrea Bolognani / Red Hat / Virtualization