[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[RFC v9 00/29] vSMMUv3/pSMMUv3 2 stage VFIO integration
From: |
Eric Auger |
Subject: |
[RFC v9 00/29] vSMMUv3/pSMMUv3 2 stage VFIO integration |
Date: |
Sun, 11 Apr 2021 14:08:43 +0200 |
Up to now vSMMUv3 has not been integrated with VFIO. VFIO
integration requires to program the physical IOMMU consistently
with the guest mappings. However, as opposed to VTD, SMMUv3 has
no "Caching Mode" which allows easy trapping of guest mappings.
This means the vSMMUV3 cannot use the same VFIO integration as VTD.
However SMMUv3 has 2 translation stages. This was devised with
virtualization use case in mind where stage 1 is "owned" by the
guest whereas the host uses stage 2 for VM isolation.
This series sets up this nested translation stage. It only works
if there is one physical SMMUv3 used along with QEMU vSMMUv3 (in
other words, it does not work if there is a physical SMMUv2).
- We force the host to use stage 2 instead of stage 1, when we
detect a vSMMUV3 is behind a VFIO device. For a VFIO device
without any virtual IOMMU, we still use stage 1 as many existing
SMMUs expect this behavior.
- We use PCIPASIDOps to propage guest stage1 config changes on
STE (Stream Table Entry) changes.
- We implement a specific UNMAP notifier that conveys guest
IOTLB invalidations to the host
- We register MSI IOVA/GPA bindings to the host so that this latter
can build a nested stage translation
- As the legacy MAP notifier is not called anymore, we must make
sure stage 2 mappings are set. This is achieved through another
prereg memory listener.
- Physical SMMU stage 1 related faults are reported to the guest
via en eventfd mechanism and exposed trhough a dedicated VFIO-PCI
region. Then they are reinjected into the guest.
Best Regards
Eric
All the patches can be found at:
https://github.com/eauger/qemu/tree/v6.0.0-rc2-2stage-rfcv9
Previous version:
v8: https://github.com/eauger/qemu/tree/v5.2.0-2stage-rfcv8
Kernel Dependencies:
[1] [PATCH v15 00/12] SMMUv3 Nested Stage Setup (IOMMU part)
[2] [PATCH v13 00/13] SMMUv3 Nested Stage Setup (VFIO part)
branch containing both:
https://github.com/eauger/linux/tree/v5.12-rc6-jean-iopf-14-2stage-v15
History:
v8 -> v9:
- added
hw/arm/smmu-common: Allow domain invalidation for NH_ALL/NSNH_ALL
following Chenxiang's report
v7 -> v8:
- adapt to changes to the kernel uapi
- Fix unregistration of MSI bindings
- applies on top of range invalidation fixes
- changes in IOTLBEntry (flags)
- addressed all the comments from reviewers/testers I hope.
Many thanks to all of you! see individual logs
Eric Auger (28):
hw/vfio/common: trace vfio_connect_container operations
update-linux-headers: Import iommu.h
header update against 5.12-rc6 and IOMMU/VFIO nested stage APIs
memory: Add new fields in IOTLBEntry
hw/arm/smmuv3: Improve stage1 ASID invalidation
hw/arm/smmu-common: Allow domain invalidation for NH_ALL/NSNH_ALL
memory: Add IOMMU_ATTR_VFIO_NESTED IOMMU memory region attribute
memory: Add IOMMU_ATTR_MSI_TRANSLATE IOMMU memory region attribute
memory: Introduce IOMMU Memory Region inject_faults API
iommu: Introduce generic header
vfio: Force nested if iommu requires it
vfio: Introduce hostwin_from_range helper
vfio: Introduce helpers to DMA map/unmap a RAM section
vfio: Set up nested stage mappings
vfio: Pass stage 1 MSI bindings to the host
vfio: Helper to get IRQ info including capabilities
vfio/pci: Register handler for iommu fault
vfio/pci: Set up the DMA FAULT region
vfio/pci: Implement the DMA fault handler
hw/arm/smmuv3: Advertise MSI_TRANSLATE attribute
hw/arm/smmuv3: Store the PASID table GPA in the translation config
hw/arm/smmuv3: Fill the IOTLBEntry arch_id on NH_VA invalidation
hw/arm/smmuv3: Fill the IOTLBEntry leaf field on NH_VA invalidation
hw/arm/smmuv3: Pass stage 1 configurations to the host
hw/arm/smmuv3: Implement fault injection
hw/arm/smmuv3: Allow MAP notifiers
pci: Add return_page_response pci ops
vfio/pci: Implement return_page_response page response callback
Liu Yi L (1):
pci: introduce PCIPASIDOps to PCIDevice
hw/arm/smmu-internal.h | 1 +
hw/vfio/pci.h | 11 +
include/exec/memory.h | 64 +-
include/hw/arm/smmu-common.h | 1 +
include/hw/iommu/iommu.h | 36 ++
include/hw/pci/pci.h | 15 +
include/hw/vfio/vfio-common.h | 19 +
include/standard-headers/drm/drm_fourcc.h | 23 +-
include/standard-headers/linux/ethtool.h | 54 +-
include/standard-headers/linux/fuse.h | 3 +-
include/standard-headers/linux/input.h | 2 +-
.../standard-headers/rdma/vmw_pvrdma-abi.h | 7 +
linux-headers/asm-generic/unistd.h | 4 +-
linux-headers/asm-mips/unistd_n32.h | 1 +
linux-headers/asm-mips/unistd_n64.h | 1 +
linux-headers/asm-mips/unistd_o32.h | 1 +
linux-headers/asm-powerpc/kvm.h | 2 +
linux-headers/asm-powerpc/unistd_32.h | 1 +
linux-headers/asm-powerpc/unistd_64.h | 1 +
linux-headers/asm-s390/unistd_32.h | 1 +
linux-headers/asm-s390/unistd_64.h | 1 +
linux-headers/asm-x86/kvm.h | 1 +
linux-headers/asm-x86/unistd_32.h | 1 +
linux-headers/asm-x86/unistd_64.h | 1 +
linux-headers/asm-x86/unistd_x32.h | 1 +
linux-headers/linux/kvm.h | 89 +++
linux-headers/linux/vfio.h | 169 +++++-
hw/arm/smmu-common.c | 3 +-
hw/arm/smmuv3.c | 224 ++++++-
hw/i386/intel_iommu.c | 6 +-
hw/pci/pci.c | 50 ++
hw/ppc/spapr_iommu.c | 2 +-
hw/vfio/common.c | 565 ++++++++++++++----
hw/vfio/pci.c | 415 ++++++++++++-
hw/virtio/virtio-iommu.c | 4 +-
softmmu/memory.c | 10 +
hw/arm/trace-events | 2 +
hw/vfio/trace-events | 11 +-
scripts/update-linux-headers.sh | 2 +-
39 files changed, 1626 insertions(+), 179 deletions(-)
create mode 100644 include/hw/iommu/iommu.h
--
2.26.3
- [RFC v9 00/29] vSMMUv3/pSMMUv3 2 stage VFIO integration,
Eric Auger <=
- [RFC v9 01/29] hw/vfio/common: trace vfio_connect_container operations, Eric Auger, 2021/04/11
- [RFC v9 02/29] update-linux-headers: Import iommu.h, Eric Auger, 2021/04/11
- [RFC v9 03/29] header update against 5.12-rc6 and IOMMU/VFIO nested stage APIs, Eric Auger, 2021/04/11
- [RFC v9 04/29] memory: Add new fields in IOTLBEntry, Eric Auger, 2021/04/11
- [RFC v9 05/29] hw/arm/smmuv3: Improve stage1 ASID invalidation, Eric Auger, 2021/04/11
- [RFC v9 06/29] hw/arm/smmu-common: Allow domain invalidation for NH_ALL/NSNH_ALL, Eric Auger, 2021/04/11
- [RFC v9 07/29] memory: Add IOMMU_ATTR_VFIO_NESTED IOMMU memory region attribute, Eric Auger, 2021/04/11
- [RFC v9 08/29] memory: Add IOMMU_ATTR_MSI_TRANSLATE IOMMU memory region attribute, Eric Auger, 2021/04/11
- [RFC v9 09/29] memory: Introduce IOMMU Memory Region inject_faults API, Eric Auger, 2021/04/11
- [RFC v9 10/29] iommu: Introduce generic header, Eric Auger, 2021/04/11