qemu-devel
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Qemu-devel] [RFC v2 00/28] vSMMUv3/pSMMUv3 2 stage VFIO integration


From: Eric Auger
Subject: [Qemu-devel] [RFC v2 00/28] vSMMUv3/pSMMUv3 2 stage VFIO integration
Date: Fri, 21 Sep 2018 10:17:51 +0200

Up to now vSMMUv3 has not been integrated with VFIO. VFIO
integration requires to program the physical IOMMU consistently
with the guest mappings. However, as opposed to VTD, SMMUv3 has
no "Caching Mode" which allows easy trapping of guest mappings.
This means the vSMMUV3 cannot use the same VFIO integration as VTD.

However SMMUv3 has 2 translation stages. This was devised with
virtualization use case in mind where stage 1 is "owned" by the
guest whereas the host uses stage 2 for VM isolation.

This series sets up this nested translation stage. It only works
if there is one physical SMMUv3 used along with QEMU vSMMUv3 (in
other words, it does not work if there is a physical SMMUv2).

The series uses a new kernel user API [1], still under definition.

- We force the host to use stage 2 instead of stage 1, when we
  detect a vSMMUV3 is behind a VFIO device. For a VFIO device
  without any virtual IOMMU, we still use stage 1 as many existing
  SMMUs expect this behavior.
- We introduce new IOTLB "config" notifiers, requested to notify
  changes in the config of a given iommu memory region. So now
  we have notifiers for IOTLB changes and config changes.
- vSMMUv3 calls config notifiers when STE (Stream Table Entries)
  are updated by the guest.
- We implement a specific UNMAP notifier that conveys guest
  IOTLB invalidations to the host
- We implement a new MAP notifiers only used for MSI IOVAs so
  that the host can build a nested stage translation for MSI IOVAs
- As the legacy MAP notifier is not called anymore, we must make
  sure stage 2 mappings are set. This is achieved through another
  memory listener.
- Physical SMMUs faults are reported to the guest via en eventfd
  mechanism and reinjected into this latter.

Note: some iommu memory notifier rework related patches are close
to those previously published by Peter and Liu. I will be pleased to
add their Signed-off-by if they agree/wish.

Note: The 2 first patches were sent separately in:
[PATCH 0/2] ARM SMMUv3: Fix event queue handling and memory region names
 as they are fixes of current SMMU emulation code.

Best Regards

Eric

This series can be found at:
https://github.com/eauger/qemu/tree/v3.0.0_2stage-rfc-v2

Testing:
- For testing use my kernel branch
  https://github.com/eauger/linux/tree/v4.19-rc4-2stage-rfc-v2 [1]
- Tested on Qualcomm HW with 1/2 assigned e1000e NICs. Hotplug
  was also tested OK.
- Known limitation:
  - currently sending an NH_ASID command instead of NH_VA
    upon guest NH_VA. This may cause important perf downgrade.
    Propagating NH_VA does not work at the moment.

References:
- [1] [RFC v2 00/20] SMMUv3 Nested Stage Setup
  (https://lkml.org/lkml/2018/9/18/1087)
  The User API still is unstable and under discussion.

History:
v1 -> v2:
- Fixed dual assignment (asid now correctly propagated on TLB invalidations)
- Integrated fault reporting

Next Steps:
- Mature the user API with people involved in SVA work (KVM forum may be a
  good opportunity to meet)
- Submit the IOMMU cfg notifier changes and some VFIO changes separately, to
  progress independently on the kernel dependency

Eric Auger (28):
  hw/arm/smmu-common: Fix the name of the iommu memory regions
  hw/arm/smmuv3: fix eventq recording and IRQ triggerring
  update-linux-headers: Import iommu.h
  linux-headers: Partial header update
  memory: add IOMMU_ATTR_VFIO_NESTED IOMMU memory region attribute
  hw/arm/smmuv3: Implement get_attr API to report IOMMU_ATTR_VFIO_NESTED
  hw/vfio/common: Refactor container initialization
  hw/vfio/common: Force nested if iommu requires it
  memory: Introduce IOMMUIOLTBNotifier
  memory: rename memory_region notify_iommu, notify_one
  memory: Add IOMMUConfigNotifier
  memory: Add arch_id in IOTLBEntry
  hw/arm/smmuv3: Store s1ctrptr in translation config data
  hw/arm/smmuv3: Implement dummy replay
  hw/arm/smmuv3: Notify on config changes
  hw/arm/smmuv3: Fill the IOTLBEntry arch_id on NH_VA invalidation
  hw/vfio/common: Introduce vfio_alloc_guest_iommu helper
  hw/vfio/common: Introduce vfio_dma_(un)map_ram_section helpers
  hw/vfio/common: Register specific nested mode notifiers and
    memory_listener
  hw/vfio/common: Register MAP notifier for MSI binding
  target/arm/kvm: Notifies IOMMU on MSI stage 1 binding
  vfio/pci: Always set up MSI route before enabling vectors
  hw/arm/smmuv3: Remove warning about unsupported MAP notifiers
  memory: Introduce IOMMU_NOTIFIER_INIT_CFG IOMMU Config Notifier
  memory: Introduce IOMMU Memory Region inject_faults API
  hw/vfio/common: Handle fault_handler
  hw/arm/smmuv3: Init fault handling
  hw/arm/smmuv3: Implement fault injection

 exec.c                          |  12 +-
 hw/arm/smmu-common.c            |  12 +-
 hw/arm/smmuv3-internal.h        |  26 +-
 hw/arm/smmuv3.c                 | 189 +++++++--
 hw/i386/intel_iommu.c           |  16 +-
 hw/misc/tz-mpc.c                |   8 +-
 hw/ppc/spapr_iommu.c            |   2 +-
 hw/s390x/s390-pci-inst.c        |   4 +-
 hw/vfio/common.c                | 666 ++++++++++++++++++++++++--------
 hw/vfio/pci.c                   |   1 +
 hw/vfio/trace-events            |   4 +-
 hw/virtio/vhost.c               |  12 +-
 include/exec/memory.h           | 140 +++++--
 include/hw/arm/smmu-common.h    |   1 +
 include/hw/vfio/vfio-common.h   |   1 +
 linux-headers/linux/iommu.h     | 237 ++++++++++++
 linux-headers/linux/vfio.h      |  48 +++
 memory.c                        |  66 +++-
 scripts/update-linux-headers.sh |   2 +-
 target/arm/kvm.c                |  46 +--
 20 files changed, 1206 insertions(+), 287 deletions(-)
 create mode 100644 linux-headers/linux/iommu.h

-- 
2.17.1




reply via email to

[Prev in Thread] Current Thread [Next in Thread]