Re: [Qemu-devel] [RFC v5 0/6] pci_expander_brdige: support separate pci domain for pxb-pcie

[Michael S. Tsirkin]

Cc Laine, Eric for an opinion about the management interface.

On Mon, Sep 17, 2018 at 10:57:31PM +0800, Zihan Yang wrote:
> Hi all
> 
> Here is a minimal working version of supporting multiple pci domains.
> The next a few paragraphs will illustrate the purpose and use example.
> Current issue and limitations will be in last 2 paragraphs, followed
> by the changelog of each verison.
> 
> Currently only q35 host bridge is allocated an item in MCFG table, all
> pxb-pcie host bridges stay within pci domain 0. This series of patches
> allow each pxb-pcie to be put in separate pci domain, allocating a new
> MCFG table item for it.
> 
> Users can configure whether to put pxb host bridge into separate domain
> by specifying property 'domain_nr' of pxb-pcie device. 'bus_nr' property
> indicates the Base Bus Number(BBN) of the pxb-pcie host bridge. Another
> property max_bus specifies the maximum desired bus numbers to reduce
> MCFG space cost. Example command is
> 
>     -device pxb-pcie,id=bridge3,bus="pcie.0",domain_nr=1,max_bus=15
> 
> Then this pxb-pcie host bridge is placed at pci domain 1, and only reserve
> (15+1)=16 buses, which is much smaller than the default 256 buses.
> 
> Compared with previous version, this version is much simpler because
> mcfg of extra domain now has a relatively fixed address, as suggested
> by Marcel and Gerd. Putting extra mmconfig above 4G and let seabios
> leave them for guest os will be expected in next version. The range is
> [0x80000000, 0xb0000000), which allows us to hold 4x busses compared
> with before.
> 
> A complete command line for test is follows, you need to replace GUEST_IMAGE,
> DATA_IMAGE and SEABIOS_BIN with proper environment variable
> 
> ./x86_64-softmmu/qemu-system-x86_64 \
>     -machine q35,accel=kvm -smp 2 -m 2048 \
>     -drive file=${GUEST_IMAGE}  -netdev user,id=realnet0 \
>     -device e1000e,netdev=realnet0,mac=52:54:00:12:34:56 \
>     -device pxb-pcie,id=bridge3,bus="pcie.0",domain_nr=1 \
>     -device pcie-root-port,id=rp1,bus=bridge3,addr=1c.0,port=8,chassis=8 \
>     -drive if=none,id=drive0,file=${DATA_IMAGE} \
>     -device virtio-scsi-pci,id=scsi,bus=rp1,addr=00.0 \
>     -bios ${SEABIOS_BIN}
> 
> There are a few limitations, though
> 1. Legacy interrupt routing is not dealt with yet. There is only support for
>    devices using MSI/MSIX

That's probably a must have. What makes it hard to add?

> 2. Only 4x devices is supported, you need to be careful not to overuse

Could you elaborate on this please? What happens if you are not careful?
How does management know what the limits are?

> 3. I have not fully tested the functionality of devices under separate domain
>    yet, but Linux can recognize then when typing `lspci`
> 
> Current issue:
> * SCSI storage device will be recognized twice, one in domain 0 as 0000:01.0,
>   the other in domain 1 as 0001:01.0. I will try to fix it in next version
> 
> v5 <- v4:
> - Refactor the design and place pxb-pcie's mcfg in [0x80000000, 0xb0000000)
> - QEMU only decides the desired mcfg_size and leaves mcfg_base for seabios
> - Does not connect PXBDev and PXBPCIEHost with link property anymore, but
>   with the pcibus under them, which makes code simpler.
> 
> v4 <- v3:
> - Fix bug in setting mcfg table
> - bus_nr is not used when pxb-pcie is in a new pci domain
> 
> v3 <- v2:
> - Replace duplicate properties in pxb pcie host with link property to PXBDev
> - Allow seabios to access config space and data space of expander bridge
>   through a different ioport, because 0xcf8 is attached only to sysbus.
> - Add a new property start_bus to indicate the BBN of pxb host bridge. The
>   bus_nr property is used as the bus number of pxb-pcie device on pcie.0 bus
> 
> v2 <- v1:
> - Allow user to configure whether to put pxb-pcie into seperate domain
> - Add AML description part of each host bridge
> - Modify the location of MCFG space to between RAM hotplug and pci hole64
> 
> Many thanks to 
> Please let me know if you have any suggestions.
> 
> Zihan Yang (6):
>   pci_expander_bridge: add type TYPE_PXB_PCIE_HOST
>   pci_expander_bridge: add domain_nr and max_bus property for pxb-pcie
>   acpi-build: allocate mcfg for pxb-pcie host bridges
>   i386/acpi-build: describe new pci domain in AML
>   pci_expander_bridge: add config_write callback for pxb-pcie
>   pci_expander_bridge: inform seabios of desired mcfg size via hidden
>     bar
> 
>  hw/i386/acpi-build.c                        | 162 ++++++++++++++++++--------
>  hw/pci-bridge/pci_expander_bridge.c         | 172 
> +++++++++++++++++++++++++++-
>  hw/pci/pci.c                                |  30 ++++-
>  include/hw/pci-bridge/pci_expander_bridge.h |  25 ++++
>  include/hw/pci/pci.h                        |   2 +
>  include/hw/pci/pci_bus.h                    |   2 +
>  include/hw/pci/pci_host.h                   |   2 +-
>  7 files changed, 336 insertions(+), 59 deletions(-)
>  create mode 100644 include/hw/pci-bridge/pci_expander_bridge.h
> 
> -- 
> 2.7.4