Re: [Qemu-devel] >256 Virtio-net-pci hotplug Devices

[Marcel Apfelbaum]

On 20/07/2017 3:44, Kinsella, Ray wrote:
> Hi Marcel,

Hi Ray,

> > You can use multi-function PCIe Root Ports, this will give you 8 ports
> > per slot, if you have 16 empty slots (I think we have more) you reach
> > 128 root ports.
> > Then you can use multi-function  virtio-net-pci devices, this will
> > give you 8 functions per port, so you reach the target of 1024 devices.
> >
> > You loose hot-plug granularity since you can hot-plug 8-functions group,
> > but maybe is OK, depending on your scenario.
>
> Thanks for the advice losing the hotplug granularity is something I 
> think I can live with. It would mean, I would have to track how many 
> ports are allocated to a VM, and create 8 new ports when 1 is required, 
> caching the other 7 for when they are needed.
>
> > Even so, you can use one cold-plugged pxb-pcie if you don't
> > have enough empty slots on pcie.0, in order to reach the maximum
> > number of PCIe Root Ports (256) which is the maximum for a single
> > PCI domain.
>
> Took your advice see the attached cfg, it works exactly as you 
> indicated. If you are interested, you can use it from your VM adding 
> -readconfig to your qemu cmd line. I can currently only manage to start 
> a VM with around 50 coldplugged virtio devices before something breaks. 


That's strange. Please ensure the virtio devices are working in
virtio 1.0 mode (disable-modern=0,disable-legacy=1).
Let us know any problems you see.

> Not sure what yet, I will try scaling it with hotplugging tomorrow.

Updates?

> > If you need granularity per single device (1000+ hot-pluggable),
> > you could enhance the pxb-pcie to support multiple pci domains.
>
> Do think there would be much work in this?

Not really. All you have to do is to add a property to the pxb-pci/pxb
devices: pci_domain=x; then update the ACPI table to include the pxb
domain. You also have to tweak a little the pxb-pcie/pxb devices
to not share the bus numbers if pci_domain > 0.

Thanks,
Marcel

> Thanks,
>
> Ray K