Re: [Qemu-devel] [PATCH 0/3] vhost-scsi: Support live migration

[Liran Alon]

> On 9 Apr 2019, at 17:29, Stefan Hajnoczi <address@hidden> wrote:
> 
> On Thu, Mar 21, 2019 at 09:55:42AM +0200, Nir Weiner wrote:
>> Originally migration was not possible with vhost-scsi because
>> as part of migration, the source host target SCSI device state
>> needs to be saved and loaded into the destination host target SCSI
>> device. This cannot be done by QEMU.
>> 
>> As this can be handled either by external orchestrator or by having
>> shared-storage (i.e. iSCSI), there is no reason to limit the
>> orchestrator from being able to explictly specify it wish to enable
>> migration even when VM have a vhost-scsi device.
>> 
>> Liran Alon (1):
>>  vhost-scsi: Allow user to enable migration
>> 
>> Nir Weiner (2):
>>  vhost-scsi: The vhost device should be stopped when the VM is not
>>    running
>>  vhost-scsi: Add vmstate descriptor
>> 
>> hw/scsi/vhost-scsi.c                  | 57 ++++++++++++++++++++++-----
>> include/hw/virtio/vhost-scsi-common.h |  1 +
>> 2 files changed, 48 insertions(+), 10 deletions(-)
> 
> What happens when migration is attempted while there is in-flight I/O in
> the vring?
> 
> Stefan

What do you define as “in-flight I/O”? I think there is a need to separate the 
discussion here to multiple cases:

1) The I/O request was written to vring but guest have not yet written to 
doorbell:
This is the simplest case. No state is lost as the vring is migrated from 
source host to dest host as part of guest memory migration.
Also, the vring properties (E.g. Guest base address) is transferred from source 
host to dest host as part of vhost-scsi VMState.
(See patch 2/3 of series which adds VMSTATE_VIRTIO_DEVICE to vhost-scsi VMState 
which will cause virtio_save() to save vring on migration stream).

2) The I/O request was written to vring and the guest have written to doorbell:
The I/O request was submitted and processed by the vhost-scsi backend. 
Therefore, the I/O request was sent to the remote TGT server.
If the the TGT server has performed the write and returned ACK but the ACK was 
not received by guest, then guest is expected to initiate the write again.
(Similar to what happens for a momentary TGT server downtime / network 
downtime). So this isn’t suppose to be an issue.

An interesting case is what happens if there is an in-flight I/O write request 
(Request A) performed by source host that was sent already over the network to 
the TGT server.
But until it reaches the TGT server, the VM running on dest performs another 
I/O write request (Request B) to the same sector which results in Request B 
being handled by TGT server before Request A. In this case, it is possible that 
TGT server will actually overwrite data written by Request B with older request 
Request A that arrived to TGT later.
(This will be the result in case of using an iSCSI TGT).
But, there are various techniques that TGT server can implement to avoid this. 
For example, fencing-out requests from any older iSCSI connection than the most 
recent one.

In general, vhost-scsi migration shouldn’t be different than vhost-net or 
vhost-vsock migration.
The only thing that may be problematic in case of vhost-scsi is because the 
source host target SCSI device state may need to be saved and restored on dest 
host target SCSI device.
However, when using a shared-storage via iSCSI, this concern is irrelevant. 
Therefore, this patch series attempts to allow the admin to remove the 
migration-blocker limitation from vhost-scsi if it is using such a setup. As 
described in cover-letter.

-Liran