Re: [Qemu-devel] [RFC PATCH V4 3/4] vfio: Add SaveVMHanlders for VFIO device to support live migration

[Zhang, Yulei]

> -----Original Message-----
> From: Alex Williamson [mailto:address@hidden
> Sent: Tuesday, April 17, 2018 4:38 AM
> To: Zhang, Yulei <address@hidden>
> Cc: address@hidden; Tian, Kevin <address@hidden>;
> address@hidden; address@hidden;
> address@hidden; Wang, Zhi A <address@hidden>;
> address@hidden; address@hidden
> Subject: Re: [RFC PATCH V4 3/4] vfio: Add SaveVMHanlders for VFIO device
> to support live migration
> 
> On Tue, 10 Apr 2018 14:03:13 +0800
> Yulei Zhang <address@hidden> wrote:
> 
> > Instead of using vm state description, add SaveVMHandlers for VFIO
> > device to support live migration.
> >
> > Introduce new Ioctl VFIO_DEVICE_GET_DIRTY_BITMAP to fetch the
> memory
> > bitmap that dirtied by vfio device during the iterative precopy stage
> > to shorten the system downtime afterward.
> >
> > For vfio pci device status migrate, during the system downtime, it
> > will save the following states 1. pci configuration space addr0~addr5
> > 2. pci configuration space msi_addr msi_data 3. pci device status
> > fetch from device driver
> >
> > And on the target side the vfio_load will restore the same states 1.
> > re-setup the pci bar configuration 2. re-setup the pci device msi
> > configuration 3. restore the pci device status
> 
> Interrupts are configured via ioctl, but I don't see any code here to 
> configure
> the device into the correct interrupt state.  How do we know the target
> device is compatible with the source device?  Do we rely on the vendor
> driver to implicitly include some kind of device and version information and
> fail at the very end of the migration?  It seems like we need to somehow
> front-load that sort of device compatibility checking since a vfio-pci device
> can be anything (ex. what happens if a user tries to migrate a GVT-g vGPU to
> an NVIDIA vGPU?).  Thanks,
> 
> Alex

We emulate the write to the pci configure space in vfio_load() which will help
setup the interrupt state. 
For compatibility I think currently the vendor driver would put version number 
or device specific info in the device state region, so during the pre-copy stage
the target side will discover the difference and call off the migration.