Since there is no device which has hardware support for system memmory
dirty bitmap tracking, right now there is no other API from vendor
driver
to VFIO IOMMU module to report dirty pages. In future, when such
hardware
support will be implemented, an API will be required in kernel such that
vendor driver could report dirty pages to VFIO module during
migration phases.
Below is the flow of state change for live migration where states in
brackets
represent VM state, migration state and VFIO device state as:
(VM state, MIGRATION_STATUS, VFIO_DEVICE_STATE)
Live migration save path:
QEMU normal running state
(RUNNING, _NONE, _RUNNING)
|
migrate_init spawns migration_thread.
(RUNNING, _SETUP, _RUNNING|_SAVING)
Migration thread then calls each device's .save_setup()
|
(RUNNING, _ACTIVE, _RUNNING|_SAVING)
If device is active, get pending bytes by .save_live_pending()
if pending bytes >= threshold_size, call save_live_iterate()
Data of VFIO device for pre-copy phase is copied.
Iterate till total pending bytes converge and are less than
threshold
|
On migration completion, vCPUs stops and calls
.save_live_complete_precopy
for each active device. VFIO device is then transitioned in
_SAVING state.
(FINISH_MIGRATE, _DEVICE, _SAVING)
For VFIO device, iterate in .save_live_complete_precopy until
pending data is 0.
(FINISH_MIGRATE, _DEVICE, _STOPPED)
|
(FINISH_MIGRATE, _COMPLETED, _STOPPED)
Migraton thread schedule cleanup bottom half and exit
Live migration resume path:
Incomming migration calls .load_setup for each device
(RESTORE_VM, _ACTIVE, _STOPPED)
|
For each device, .load_state is called for that device section data
(RESTORE_VM, _ACTIVE, _RESUMING)
|
At the end, called .load_cleanup for each device and vCPUs are
started.
|
(RUNNING, _NONE, _RUNNING)
Note that:
- Migration post copy is not supported.