Re: [Qemu-devel] [Migration Bug? ] Occasionally, the content of VM's memory is inconsistent between Source and Destination of migration

[Stefan Hajnoczi]

On Fri, Mar 27, 2015 at 10:05 AM, Wen Congyang <address@hidden> wrote:
> On 03/27/2015 05:57 PM, Stefan Hajnoczi wrote:
>> On Fri, Mar 27, 2015 at 9:14 AM, Wen Congyang <address@hidden> wrote:
>>> On 03/27/2015 04:56 PM, Stefan Hajnoczi wrote:
>>>> On Thu, Mar 26, 2015 at 11:29:43AM +0100, Juan Quintela wrote:
>>>>> Wen Congyang <address@hidden> wrote:
>>>>>> On 03/25/2015 05:50 PM, Juan Quintela wrote:
>>>>>>> zhanghailiang <address@hidden> wrote:
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> We found that, sometimes, the content of VM's memory is
>>>>>>>> inconsistent between Source side and Destination side
>>>>>>>> when we check it just after finishing migration but before VM continue 
>>>>>>>> to Run.
>>>>>>>>
>>>>>>>> We use a patch like bellow to find this issue, you can find it from 
>>>>>>>> affix,
>>>>>>>> and Steps to reprduce:
>>>>>>>>
>>>>>>>> (1) Compile QEMU:
>>>>>>>>  ./configure --target-list=x86_64-softmmu  --extra-ldflags="-lssl" && 
>>>>>>>> make
>>>>>>>>
>>>>>>>> (2) Command and output:
>>>>>>>> SRC: # x86_64-softmmu/qemu-system-x86_64 -enable-kvm -cpu
>>>>>>>> qemu64,-kvmclock -netdev tap,id=hn0-device
>>>>>>>> virtio-net-pci,id=net-pci0,netdev=hn0 -boot c -drive
>>>>>>>> file=/mnt/sdb/pure_IMG/sles/sles11_sp3.img,if=none,id=drive-virtio-disk0,cache=unsafe
>>>>>>>> -device
>>>>>>>> virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0
>>>>>>>> -vnc :7 -m 2048 -smp 2 -device piix3-usb-uhci -device usb-tablet
>>>>>>>> -monitor stdio
>>>>>>>
>>>>>>> Could you try to reproduce:
>>>>>>> - without vhost
>>>>>>> - without virtio-net
>>>>>>> - cache=unsafe is going to give you trouble, but trouble should only
>>>>>>>   happen after migration of pages have finished.
>>>>>>
>>>>>> If I use ide disk, it doesn't happen.
>>>>>> Even if I use virtio-net with vhost=on, it still doesn't happen. I guess
>>>>>> it is because I migrate the guest when it is booting. The virtio net
>>>>>> device is not used in this case.
>>>>>
>>>>> Kevin, Stefan, Michael, any great idea?
>>>>
>>>> You must use -drive cache=none if you want to use live migration.  It
>>>> should not directly affect memory during migration though.
>>>
>>> Otherwise, what will happen? If the user doesn't use cache=none, and
>>> tries to use live migration, qemu doesn't output any message or trigger
>>> an event to notify the user.
>>
>> There is a risk that the destination host sees an inconsistent view of
>> the image file because the source was still accessing it towards the
>> end of migration.
>
> Does the flag BDRV_O_NOFLUSH cause it?

Partly, but the problem is worse than that.  BDRV_O_NOFLUSH means that
the host never issues fdatasync(2) and you have no guarantee that the
data reaches the physical disk safely.

The migration problem is related to the host kernel page cache.
cache=none bypasses the page cache with O_DIRECT so the destination
host doesn't see outdated cached data.

Stefan