[RFC PATCH v3 0/4] Add migration support for VFIO PCI devices in SMMUv3 nested mode

[Kunkun Jiang]

Hi all,

Since the SMMUv3's nested translation stages has been introduced by Eric, we
need to pay attention to the migration of VFIO PCI devices in SMMUv3 nested
stage mode. At present, it is not yet supported in QEMU. There are two problems
in the existing framework.

First, the current way to get dirty pages is not applicable to nested stage
mode. Because of the "Caching Mode", VTD can map the RAM through the host
single stage (giova->hpa). "vfio_listener_log_sync" gets dirty pages by
transferring "giova" to the kernel for the RAM block section of mapped MMIO
region. In nested stage mode, we setup the stage 2 (gpa->hpa) and the stage 1
(giova->gpa) separately. So it is inapplicable to get dirty pages by the
current way in nested stage mode.

Second, it also need to pass stage 1 configurations to the destination host
after the migration. In Eric's patch, it passes the stage 1 configuration to
the host on each STE update for the devices set the PASID PciOps. The
configuration will be applied at physical level. But the data of physical
level will not be sent to the destination host. So we have to pass stage 1
configurations to the destination host after the migration.

This Patch set includes patches as below:
Patch 1-2:
- Refactor the vfio_listener_log_sync and added a new function to get dirty
pages for nested mode

Patch 3:
- Added global_log_start/stop interface in vfio_memory_prereg_listener for
nested mode

Patch 4:
- Added the post_load function to vmstate_smmuv3 for passing stage 1
configuration to the destination host after the migration

Best regards,
Kunkun Jiang

History:

v2 -> v3:
- Rebase to v9 of Eric's series 'vSMMUv3/pSMMUv3 2 stage VFIO integration'[1]
- Delete smmuv3_manual_set_pci_device_pasid_table() and reuse
  smmuv3_notify_config_change() [Eric]

v1 -> v2:
- Add global_log_start/stop interface in vfio_memory_prereg_listener
- Add support for repass stage 1 configs with multiple CDs to the host

[1] [RFC v9 00/29] vSMMUv3/pSMMUv3 2 stage VFIO integration
https://lore.kernel.org/qemu-devel/20210411120912.15770-1-eric.auger@redhat.com/

Kunkun Jiang (4):
  vfio: Introduce helpers to mark dirty pages of a RAM section
  vfio: Add vfio_prereg_listener_log_sync in nested stage
  vfio: Add vfio_prereg_listener_global_log_start/stop in nested stage
  hw/arm/smmuv3: Post-load stage 1 configurations to the host

 hw/arm/smmuv3.c  | 33 ++++++++++++++++++----
 hw/vfio/common.c | 73 ++++++++++++++++++++++++++++++++++++++++++------
 2 files changed, 93 insertions(+), 13 deletions(-)

-- 
2.23.0