Re: [PATCH for-8.2 v3 0/6] vfio/migration: Add P2P support for VFIO migration

[YangHang Liu]

Test result of multiple Mellanox CX-7 VFs migration:  PASS

[1] create two VFs and bind them to mlx5_vfio_pci driver
[2] start a VM with two VFs
[3] migrate the VM
[4] check if the VM works well

Tested-by: YangHang Liu <yanghliu@redhat.com>

Best Regards,
YangHang Liu
On Wed, Aug 2, 2023 at 4:16 PM Avihai Horon <avihaih@nvidia.com> wrote:
>
> Hi all,
>
> The first patch in this series adds a small optimization to VFIO
> migration by moving the STOP_COPY->STOP transition to
> vfio_save_cleanup(). Testing with a ConnectX-7 VFIO device showed that
> this can reduce downtime by up to 6%.
>
> The rest of the series adds P2P support for VFIO migration.
>
> VFIO migration uAPI defines an optional intermediate P2P quiescent
> state. While in the P2P quiescent state, P2P DMA transactions cannot be
> initiated by the device, but the device can respond to incoming ones.
> Additionally, all outstanding P2P transactions are guaranteed to have
> been completed by the time the device enters this state.
>
> The purpose of this state is to support migration of multiple devices
> that might do P2P transactions between themselves.
>
> To implement P2P migration support, all the devices must be transitioned
> to the P2P quiescent state before being stopped or started.
>
> This behavior is achieved by adding an optional prepare callback to
> VMChangeStateEntry. These callbacks are invoked before the main VM state
> change callbacks, transitioning all the VFIO devices to the P2P state,
> and only then are the main callbacks invoked, which stop or start the
> devices.
>
> This will allow migration of multiple VFIO devices if all of them
> support P2P migration.
>
> Thanks.
>
> Changes from v2 [2]:
> * Registered VFIO prepare callback only if P2P migration is supported by
>   the device. (Cedric)
> * Added Cedric R-b tags to patches 2,3 and 6.
>
> Changes from v1 [1]:
> * Rebased on latest master branch.
> * Renamed pre_change_cb to prepare_cb and adjusted relevant code
>   accordingly. (Cedric)
> * Split VFIO vmstate change handler to two separate handlers, one for
>   prepare callback and another for main callback. (Cedric)
> * Renamed vfio_should_block_multiple_devices_migration() to
>   vfio_multiple_devices_migration_is_supported() and reverted its logic
>   accordingly. (Cedric)
> * Rephrased "that are doing P2P" to "that might do P2P" in docs and
>   commit message. (Jason)
> * Added Cedric R-b tag to patch #4.
>
> [1]
> https://lore.kernel.org/qemu-devel/20230716081541.27900-1-avihaih@nvidia.com/
>
> [2]
> https://lore.kernel.org/qemu-devel/20230731102521.15335-1-avihaih@nvidia.com/
>
> Avihai Horon (5):
>   vfio/migration: Move from STOP_COPY to STOP in vfio_save_cleanup()
>   sysemu: Add prepare callback to struct VMChangeStateEntry
>   qdev: Add qdev_add_vm_change_state_handler_full()
>   vfio/migration: Add P2P support for VFIO migration
>   vfio/migration: Allow migration of multiple P2P supporting devices
>
> Joao Martins (1):
>   vfio/migration: Refactor PRE_COPY and RUNNING state checks
>
>  docs/devel/vfio-migration.rst     | 93 +++++++++++++++++++------------
>  include/hw/vfio/vfio-common.h     |  2 +
>  include/sysemu/runstate.h         |  7 +++
>  hw/core/vm-change-state-handler.c | 14 ++++-
>  hw/vfio/common.c                  | 50 +++++++++++++----
>  hw/vfio/migration.c               | 75 ++++++++++++++++++++-----
>  softmmu/runstate.c                | 40 +++++++++++++
>  hw/vfio/trace-events              |  1 +
>  8 files changed, 218 insertions(+), 64 deletions(-)
>
> --
> 2.26.3
>
>