Re: [Qemu-devel] migration question: disk images on nfs server

[Kevin Wolf]

Am 07.02.2014 um 13:47 hat Orit Wasserman geschrieben:
> On 02/07/2014 02:10 PM, Alexey Kardashevskiy wrote:
> >On 07.02.2014 18:46, Orit Wasserman wrote:
> >>On 02/07/2014 06:35 AM, Alexey Kardashevskiy wrote:
> >>>Hi!
> >>>
> >>>I have yet another problem with migration. Or NFS.
> >>>
> >>>There is one NFS server and 2 test POWER8 machines. There is a shared NFS
> >>>folder on the server, mounted to both test hosts. There is an qcow2 image
> >>>(abc.qcow2) in that shared folder.
> >>>
> >>>We start a guest with this abc.qcow2 on the test machine #1. And start
> >>>another guest on the test machine #2 with "-incoming ..." and same
> >>>abc.qcow2.
> >>>
> >>>Now we start migration. In most cases it goes fine. But if we put some
> >>>load
> >>>on machine #1, the destination guest sometime crashes.
> >>>
> >>>I blame out-of-sync NFS on the test machines. I looked a bit further in
> >>>QEMU and could not find a spot where it would fflush(abc.qcow2) or
> >>>close it
> >>>or do any other sync so it is up to the host NFS mountpoint to decide
> >>>when
> >>>to sync and it definitely does not get a clue when to do this.
> >>>
> >>>I do not really understand why the abc.qcow2 image is still open, should
> >>>not it be closed after migration succeeded?
> >>>
> >>>What do I miss here? Should we switch from NFS to GlusterFS (is it always
> >>>syncronized)? Or if we want NFS, should we just boot our guests with
> >>>"root=/dev/nfs ip=dhcp nfsroot=..." and avoid using disk images in
> >>>network
> >>>disks? Thanks!
> >>>
> >>
> >>For NFS you need to use the sync mount option to force the NFS client to
> >>sync to
> >>server on writes.
> >
> >So there is no any kind of sync in QEMU after migration finished,
> >correct? Looks too mucn to enforce "sync" option for NFS as we really
> >need it for once.
> >
> 
> It is more a NFS issue, if you have a file in NFS that two users in
> two different host are accessing (one at least write to it) you will need to 
> enforce the "sync" option.
> Even if you flush all the data and close the file the NFS client can still
> have cached data that it didn't sync to the server.

Are you sure? This is news to me. qemu does do an fsync() and that
should be enough to get the data flushed to the NFS server.

There may be another problem, though: The destination host may have stale
data in the local page cache, and qemu can't force it to drop that cache
and reload the new data from the server. In this case you need to use
cache=none (i.e. O_DIRECT). I believe this isn't strictly necessary for
NFS and the file system driver will take care of it, though we certainly
recommend cache=none with live migration.

Kevin