Re: [Qemu-devel] [PATCH RFC] docs: add PCIe devices placement guidelines

[Marcel Apfelbaum]

On 09/15/2016 11:38 AM, Andrew Jones wrote:
> On Wed, Sep 07, 2016 at 10:39:28PM +0300, Marcel Apfelbaum wrote:
> > On 09/07/2016 08:55 PM, Laine Stump wrote:
> > > On 09/07/2016 04:06 AM, Marcel Apfelbaum wrote:
> [snip]
> > > > Good point, maybe libvirt can avoid adding switches unless the user
> > > > explicitly
> > > > asked for them. I checked and it a actually works fine in QEMU.
> > >
> > > I'm just now writing the code that auto-adds *-ports as they are needed, and 
> > > doing it this way simplifies it *immensely*.
> > >
> > > When I had to think about the possibility of needing upstream/downstream 
> > > switches, as an endpoint device was added, I would need to check if a 
> > > (root|downstream)-port was available and if not I might
> > > be able to just add a root-port, or I might have to add a downstream-port; if 
> > > the only option was a downstream port, then *that* might require adding a new 
> > > *upstream* port.
> > >
> > > If I can limit libvirt to only auto-adding root-ports (and if there is no 
> > > downside to putting multiple root ports on a single root bus port), then I just 
> > > need to find an empty function of an empty
> > > slot on the root bus, add a root-port, and I'm done (and since 224 is *a lot*, 
> > > I think at least for now it's okay to punt once they get past that point).
> > >
> > > So, *is* there any downside to doing this?
> >
> > No downside I can think of.
> > Just be sure to emphasize the auto-add mechanism stops at 'x' devices. If the 
> > user needs more,
> > he should manually add switches and manually assign the devices to the 
> > Downstream Ports.
>
> Just catching up on mail after vacation and read this thread. Thanks
> Marcel for writing this document (I guess a v1 is coming soon).

Yes, I am sorry but I got caught up with other stuff and I am
going to be in PTO for a week, so V1 will take a little more time
than I planned.

This
> will be very useful for determining the best default configuration of
> a virtio-pci mach-virt.

It will be very good if this doc will match both x86 and match-virt PCIe 
machines.
Your review would be appreciated.

> FWIW, here is the proposal that I started formulating when I experimented
> with this several months ago;
>
>  - PCIe-only (disable-modern=off, disable-legacy=on)

If the virtio devices are plugged into PCI Express Root Ports
or Downstream Ports this is already the default configuration,
you don't need to add the disable-* anymore.

>  - No legacy PCI support, i.e. no bridges (yup, I'm a PCIe purist,
>    but don't have a leg to stand on if push came to shove)

Yes... We'll say that legacy PCI support is optional.

>  - use one or more ports for virtio-scsi controllers for disks, one is
>    probably enough
>  - use one or more ports with multifunction, allowing up to 8 functions,
>    for virtio-net, one port is probably enough

As Alex Williamson mentioned, PCI Express Root Ports are actually functions,
not devices, so you can have up to 8 Ports per slot. This will be better
than make the virtio-* multi-function devices because then you would need
to hot-plug/hot-unplug all of them.
If hot-plug is not an issue, using multi-function devices into multi-function
Ports will save a lot of bus numbers witch are, Like Laszlo mentioned a
scarce resource.

>  - Add N extra ports for hotplug, N defaulting to 2
>    - hotplug devices to first N-1 ports, reserving last for a switch
>    - if switch is needed, hotplug it with M downstream ports
>      (M defaulting to 2*(N-1)+1)

We would prefer multi-function ports to switches, since you'll go out
of bus numbers before you use all the PCI Express Root Ports anyway. (see 
previous mails)

However the switches will be supported for cases when you have a
lot of Integrated Devices and the Root Ports are not enough, or
to enable some testing scenarios.

>  - Encourage somebody to develop generic versions of ports and switches,
>    hi Marcel :-), and exclusively use those in the configuration

My goal is to try to come up with them for 2.8, but since I haven't started
to work on them yet, I can't commit :)

Thanks,
Marcel

> Thanks,
> drew