[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: virtio-iommu issue with VFIO device downstream to a PCIe-to-PCI brid
From: |
Eric Auger |
Subject: |
Re: virtio-iommu issue with VFIO device downstream to a PCIe-to-PCI bridge: VFIO devices are not assigned any iommu group |
Date: |
Mon, 9 Jan 2023 22:11:19 +0100 |
User-agent: |
Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.3.1 |
Hi,
On 1/9/23 14:24, Eric Auger wrote:
> Hi,
>
> we have a trouble with virtio-iommu and protected assigned devices
> downstream to a pcie-to-pci bridge. In that use case we observe the
> assigned devices are not put to any group. This is true on both x86 and
> aarch64. This use case works with intel-iommu.
>
> *** Guest PCI topology is:
> lspci -tv
> -[0000:00]-+-00.0 Intel Corporation 82G33/G31/P35/P31 Express DRAM
> Controller
> +-01.0 Device 1234:1111
> +-02.0-[01-02]----00.0-[02]----01.0 Broadcom Inc. and
> subsidiaries BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller
> +-02.1-[03]--
> +-02.2-[04]----00.0 Red Hat, Inc. Virtio block device
> +-0a.0 Red Hat, Inc. Device 1057
> +-1f.0 Intel Corporation 82801IB (ICH9) LPC Interface Controller
> +-1f.2 Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port
> SATA Controller [AHCI mode]
> \-1f.3 Intel Corporation 82801I (ICH9 Family) SMBus Controller
>
>
> All the assigned devices are aliased and they get devfn=0x0.
> see qemu pci_device_iommu_address_space in hw/pci.c
>
> Initially I see the following traces
> pci_device_iommu_address_space name=vfio-pci BDF=0x8 bus=0 devfn=0x8
> pci_device_iommu_address_space name=vfio-pci BDF=0x8 bus=0 devfn=0x8
> call iommu_fn with bus=0x55f556dde180 and devfn=0
> virtio_iommu_init_iommu_mr init virtio-iommu-memory-region-0-0
>
> Note the bus is 0 at this time and devfn that is used in the
> virtio-iommu is 0. So an associated IOMMU MR is created with this bus at
> devfn=0 slot. This is before bus actual numbering.
>
> However later on, I see virtio_iommu_probe() and virtio_iommu_attach()
> getting called with ep_id=520
> because in the qemu virtio-iommu device, virtio_iommu_mr(pe_id) fails to
> find the iommu_mr and returns -ENOENT
>
> On guest side I see that
> acpi_iommu_configure_id/iommu_probe_device() fails
> (__iommu_probe_device) and also __iommu_attach_device would also fail
> anyway.
>
> I guess those get called before actual bus number recomputation?
>
> on aarch64 I eventually see the "good" MR beeing created, ie. featuring
> the right bus number:
> qemu-system-aarch64: pci_device_iommu_address_space name=vfio-pci
> BDF=0x208 bus=2 devfn=0x8
> qemu-system-aarch64: pci_device_iommu_address_space name=vfio-pci
> BDF=0x208 bus=2 devfn=0x8 call iommu_fn with bus=0xaaaaef12c450 and devfn=0
>
> But this does not happen on x86.
>
> Jean, do you have any idea about how to fix that? Do you think we have a
> trouble in the acpi/viot setup or virtio-iommu probe sequence. It looks
> like virtio probe and attach commands are called too early, before the
> bus is actually correctly numbered.
So after further investigations looks this is not a problem of bus
number, which is good at the time of the virtio cmd calls but rather a
problem related to the devfn (0 was used when creating the IOMMU MR)
whereas the virtio-iommu cmds looks for the non aliased devfn. With that
fixed, the probe and attach at least succeeds. The device still does not
work for me but I will continue my investigations and send a tentative fix.
Thanks
Eric
>
> Thanks
>
> Eric
>
>
>
>
>
>
>
- virtio-iommu issue with VFIO device downstream to a PCIe-to-PCI bridge: VFIO devices are not assigned any iommu group, Eric Auger, 2023/01/09
- Re: virtio-iommu issue with VFIO device downstream to a PCIe-to-PCI bridge: VFIO devices are not assigned any iommu group,
Eric Auger <=
- Re: virtio-iommu issue with VFIO device downstream to a PCIe-to-PCI bridge: VFIO devices are not assigned any iommu group, Jason Wang, 2023/01/11
- Re: virtio-iommu issue with VFIO device downstream to a PCIe-to-PCI bridge: VFIO devices are not assigned any iommu group, Jean-Philippe Brucker, 2023/01/13
- Re: virtio-iommu issue with VFIO device downstream to a PCIe-to-PCI bridge: VFIO devices are not assigned any iommu group, Alex Williamson, 2023/01/13
- Re: virtio-iommu issue with VFIO device downstream to a PCIe-to-PCI bridge: VFIO devices are not assigned any iommu group, Jean-Philippe Brucker, 2023/01/18
- Re: virtio-iommu issue with VFIO device downstream to a PCIe-to-PCI bridge: VFIO devices are not assigned any iommu group, Alex Williamson, 2023/01/18
- Re: virtio-iommu issue with VFIO device downstream to a PCIe-to-PCI bridge: VFIO devices are not assigned any iommu group, Eric Auger, 2023/01/18
- Re: virtio-iommu issue with VFIO device downstream to a PCIe-to-PCI bridge: VFIO devices are not assigned any iommu group, Jean-Philippe Brucker, 2023/01/20
- Re: virtio-iommu issue with VFIO device downstream to a PCIe-to-PCI bridge: VFIO devices are not assigned any iommu group, Eric Auger, 2023/01/18