virtio-iommu issue with VFIO device downstream to a PCIe-to-PCI bridge: VFIO devices are not assigned any iommu group

[Eric Auger]

Hi,

we have a trouble with virtio-iommu and protected assigned devices
downstream to a pcie-to-pci bridge. In that use case we observe the
assigned devices are not put to any group. This is true on both x86 and
aarch64. This use case works with intel-iommu.

*** Guest PCI topology is:
lspci -tv
-[0000:00]-+-00.0  Intel Corporation 82G33/G31/P35/P31 Express DRAM
Controller
           +-01.0  Device 1234:1111
           +-02.0-[01-02]----00.0-[02]----01.0  Broadcom Inc. and
subsidiaries BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller
           +-02.1-[03]--
           +-02.2-[04]----00.0  Red Hat, Inc. Virtio block device
           +-0a.0  Red Hat, Inc. Device 1057
           +-1f.0  Intel Corporation 82801IB (ICH9) LPC Interface Controller
           +-1f.2  Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port
SATA Controller [AHCI mode]
           \-1f.3  Intel Corporation 82801I (ICH9 Family) SMBus Controller


All the assigned devices are aliased and they get devfn=0x0.
see qemu pci_device_iommu_address_space in hw/pci.c

Initially I see the following traces
pci_device_iommu_address_space name=vfio-pci BDF=0x8 bus=0 devfn=0x8
pci_device_iommu_address_space name=vfio-pci BDF=0x8 bus=0 devfn=0x8
call iommu_fn with bus=0x55f556dde180 and devfn=0
virtio_iommu_init_iommu_mr init virtio-iommu-memory-region-0-0

Note the bus is 0 at this time and devfn that is used in the
virtio-iommu is 0. So an associated IOMMU MR is created with this bus at
devfn=0 slot. This is before bus actual numbering.

However later on, I see virtio_iommu_probe() and virtio_iommu_attach()
getting called with ep_id=520
because in the qemu virtio-iommu device, virtio_iommu_mr(pe_id) fails to
find the iommu_mr and returns -ENOENT

On guest side I see that
acpi_iommu_configure_id/iommu_probe_device() fails
(__iommu_probe_device) and also __iommu_attach_device would also fail
anyway.

I guess those get called before actual bus number recomputation?

on aarch64 I eventually see the "good" MR beeing created, ie. featuring
the right bus number:
qemu-system-aarch64: pci_device_iommu_address_space name=vfio-pci
BDF=0x208 bus=2 devfn=0x8
qemu-system-aarch64: pci_device_iommu_address_space name=vfio-pci
BDF=0x208 bus=2 devfn=0x8 call iommu_fn with bus=0xaaaaef12c450 and devfn=0

But this does not happen on x86.

Jean, do you have any idea about how to fix that? Do you think we have a
trouble in the acpi/viot setup or virtio-iommu probe sequence. It looks
like virtio probe and attach commands are called too early, before the
bus is actually correctly numbered.

Thanks

Eric