Re: [Qemu-devel] [RFC 00/12] Migration: Remove copying of guest ram pages

[Michael S. Tsirkin]

On Thu, Mar 21, 2013 at 11:09:19AM +0200, Orit Wasserman wrote:
> In migration all data is copied to a static buffer in QEMUFile,
> this hurts our network bandwidth and CPU usage especially with large guests.
> We switched to iovec for storing different buffers to send (even a byte field 
> is
> considered as a buffer) and use writev to send the iovec.
> writev was chosen (as apposed to sendmsg) because it supprts non socket fds.
>   
> Guest memory pages are not copied by calling a new function 
> qemu_put_buffer_no_copy.
> The page header data and device state data are still copied into the static
> buffer. This data consists of a lot of bytes and integer fields and the static
> buffer is used to store it during batching.
> Another improvement is changing qemu_putbe64/32/16 to create a single
> buffer instead of several byte sized buffer.

A recent discussion about overcommitted memory made me think:
this will still need to read pages into memory even
if all we will do is send it out on the wire immediately,
dirtying cache etc.

Now, one property of RAM writes is that if guest changes
the memory that we are migrating, we really
don't care that remote will get a new copy and not
the old copy.

So this seems like a perfect place to use vmsplice
and save the read of data from pagecache into QEMU.

For this to work, however, we need to have a way
for QEMUFile to know whether a specific iovec
references RAM (so it's ok to use vmsplice)
or a malloced buffer for device state (so it must use write
to ensure kernel copies data).

What do you think?

> Orit Wasserman (12):
>   Add iov_writev to use writev to send iovec (also for files)
>   Add QemuFileWritevBuffer QemuFileOps
>   Add socket_writev_buffer function
>   Add stdio_writev_buffer function
>   Add block_writev_buffer function
>   Update bytes_xfer in qemu_put_byte
>   Store the data to send also in iovec
>   Use writev ops instead of put_buffer ops
>   More optimized qemu_put_be64/32/16
>   Add qemu_put_buffer_no_copy
>   Use qemu_put_buffer_no_copy for guest memory pages
>   Bye Bye put_buffer
> 
>  arch_init.c                   |   2 +-
>  include/migration/qemu-file.h |  20 ++++---
>  include/qemu/iov.h            |  12 ++++
>  savevm.c                      | 130 
> +++++++++++++++++++++++++-----------------
>  util/iov.c                    |  36 ++++++++++++
>  5 files changed, 139 insertions(+), 61 deletions(-)
> 
> -- 
> 1.7.11.7