Re: [Qemu-devel] Live migration without bdrv_drain_all()

[Felipe Franciosi]

Heya!

> On 29 Aug 2016, at 08:06, Stefan Hajnoczi <address@hidden> wrote:
> 
> At KVM Forum an interesting idea was proposed to avoid
> bdrv_drain_all() during live migration.  Mike Cui and Felipe Franciosi
> mentioned running at queue depth 1.  It needs more thought to make it
> workable but I want to capture it here for discussion and to archive
> it.
> 
> bdrv_drain_all() is synchronous and can cause VM downtime if I/O
> requests hang.  We should find a better way of quiescing I/O that is
> not synchronous.  Up until now I thought we should simply add a
> timeout to bdrv_drain_all() so it can at least fail (and live
> migration would fail) if I/O is stuck instead of hanging the VM.  But
> the following approach is also interesting...
> 
> During the iteration phase of live migration we could limit the queue
> depth so points with no I/O requests in-flight are identified.  At
> these points the migration algorithm has the opportunity to move to
> the next phase without requiring bdrv_drain_all() since no requests
> are pending.

I actually think that this "io quiesced state" is highly unlikely to _just_ 
happen on a busy guest. The main idea behind running at QD1 is to naturally 
throttle the guest and make it easier to "force quiesce" the VQs.

In other words, if the guest is busy and we run at QD1, I would expect the 
rings to be quite full of pending (ie. unprocessed) requests. At the same time, 
I would expect that a call to bdrv_drain_all() (as part of do_vm_stop()) should 
complete much quicker.

Nevertheless, you mentioned that this is still problematic as that single 
outstanding IO could block, leaving the VM paused for longer.

My suggestion is therefore that we leave the vCPUs running, but stop picking up 
requests from the VQs. Provided nothing blocks, you should reach the "io 
quiesced state" fairly quickly. If you don't, then the VM is at least still 
running (despite seeing no progress on its VQs).

Thoughts on that?

Thanks for capturing the discussion and bringing it here,
Felipe

> 
> Unprocessed requests are left in the virtio-blk/virtio-scsi virtqueues
> so that the destination QEMU can process them after migration
> completes.
> 
> Unfortunately this approach makes convergence harder because the VM
> might also be dirtying memory pages during the iteration phase.  Now
> we need to reach a spot where no I/O is in-flight *and* dirty memory
> is under the threshold.
> 
> Thoughts?
> 
> Stefan