Re: [Qemu-devel] Make NVME device "migratable" (savevm)

[Stefan Hajnoczi]

On Tue, Aug 29, 2017 at 04:40:55PM -0400, Keith Busch wrote:
> On Tue, Aug 29, 2017 at 11:09:36AM +0100, Stefan Hajnoczi wrote:
> > On Fri, Aug 25, 2017 at 11:39:30AM +0300, Sergei Dyshel wrote:
> > > Hi all,
> > > From what I understand, I can't "savevm" a VM that uses NVME device 
> > > because
> > > it has ".unmigratable = 1" in the code. What support must be implemented 
> > > in
> > > order to make it "migratable"?
> > 
> > CCing Keith Busch, the NVMe maintainer.
> 
> To make it migratable, you'll need implement the VMStateDescription to
> save and restore the entire controller state, including all the queues.
> It doesn't actually look like there's a much state to track compared to
> other migratale devices, I just never got any time to write it out.

One thing unique to QEMU's block layer is that the -drive
werror=/rerror= option lets the user specify an error handling policy
for I/O requests.

If the policy is 'stop' or 'enospc' then the emulated storage controller
must hang on to the I/O request and re-submit it when VM execution
resumes (usually when the user issues the 'cont' command at the
monitor).

For example, if QEMU hits ENOSPC while writing a previously unwritten
sector in the disk image file, then VM execution can be paused.  The
host system administrator can arrange for more disk space and then issue
the 'cont' QEMU monitor command.  This resumes the VM and QEMU retries
the I/O request without ever reporting an I/O error to the guest.

You can see how this is implemented in hw/block/virtio-blk.c and other
emulated storage controllers.  This feature requires live migration
support so that the list of failed I/O requests can be sent to the
destination host during live migration.

If you decide to enable live migration, please consider how this should
be implemented for NVMe.

Stefan