Re: [Qemu-devel] [PATCH v3 00/35] postcopy live migration

[Benoit Hudzia]

Hi Isaku,


Are you going to be at the KVM forum ( i think you have a presentation
there). It would be nice if we could meet in order to see if we can
synch our efforts .

As you know we have been developing an RDMA based solution for post
copy migration and  we demonstrated the initial proof of concept in
december 2012 ( we published some finding  in VHPC 2012 and are
working with Petter Svard from Umea on a journal paper with more
detailed performance review) .

While  RDMA post copy live migration is just of by product of our long
term effort ( i will present the project  in my talk at KVM forum)  we
grabbed the opportunity  to address problems we were facing with the
live migration of enterprise workload . Namely how to migrate in
memory database such has HANA under load.

We quickly discovered that pre copy ( even with optimization ) didn't
work with such workload. We also tried your code however the
performance where far from satisfying with large VM under load due to
the heavy cost of transferring memory between user space - kernel
multiple time ( actually it often failed)

We then tested a   pure RDMA solution we developed  ( we suport HW and
software RDMA )   and it work fine with all the  workload we tested  (
we migrated VM with 20+ GB running SAP HANA under a workload similar
to TPC-H) and we hop to test with bigger configuration soon ( 1/2 + TB
of memory) .

However the state of integration of our code with the QEMU -code base
is not as advanced and polished as the one you currently have and i
would like to know if you would be interested in trying to join our
effort or collaborate in merging our solution. Or maybe allowing us to
piggy back on your effort.

Would you bee free to meet at any time next week ? ( from Tuesday to Friday)

Ps: we would be open sourcing our project by the end of the month of
November and the post copy is only a small part of the technology
developed..


Regards
Benoit


On 30 October 2012 08:32, Isaku Yamahata <address@hidden> wrote:
>
> This is the v3 patch series of postcopy migration.
>
> The trees is available at
> git://github.com/yamahata/qemu.git qemu-postcopy-oct-30-2012
> git://github.com/yamahata/linux-umem.git linux-umem-oct-29-2012
>
> Major changes v2 -> v3:
> - implemented pre+post optimization
> - auto detection of postcopy by incoming side
> - using threads on destination instead of fork
> - using blocking io instead of select + non-blocking io loop
> - less memory overhead
> - various improvement and code simplification
> - kernel module name change umem -> uvmem to avoid name conflict.
>
> Patches organization:
> 1-2: trivial fixes
> 3-5: prepartion for threading. cherry-picked from migration tree
> 6-18: refactoring existing code and preparation
> 19-25: implement postcopy live migration itself (essential part)
> 26-35: optimization/heuristic for postcopy
>
> Usage
> =====
> You need load uvmem character device on the host before starting
> migration.
> Postcopy can be used for tcg and kvm accelarator. The implementation
> depend
> on only linux uvmem character device. But the driver dependent code is
> split
> into a file.
> I tested only host page size == guest page size case, but the
> implementation
> allows host page size != guest page size case.
>
> The following options are added with this patch series.
> - incoming part
>   use -incoming as usual. Postcopy is automatically detected.
>   example:
>   qemu -incoming tcp:0:4444 -monitor stdio -machine accel=kvm
>
> - outging part
>   options for migrate command
>   migrate [-p [-n] [-m]] URI
>           [<precopy count> [<prefault forward> [<prefault backword>]]]
>
>   Newly added options/arguments
>   -p: indicate postcopy migration
>   -n: disable background transferring pages: This is for
> benchmark/debugging
>   -m: move background transfer of postcopy mode
>   <precopy count>: The number of precopy RAM scan before postcopy.
>                    default 0 (0 means no precopy)
>   <prefault forward>: The number of forward pages which is sent with
> on-demand
>   <prefault backward>: The number of backward pages which is sent with
>                        on-demand
>
>   example:
>   migrate -p -n tcp:<dest ip address>:4444
>   migrate -p -n -m tcp:<dest ip address>:4444 42 42 0
>
>
> TODO
> ====
> - benchmark/evaluation
> - improve/optimization
>   At the moment at least what I'm aware of is
>   - pre+post case
>     On desitnation side reading dirty bitmap would cause long latency.
>     create thread for that.
> - consider on FUSE/CUSE possibility
>
> basic postcopy work flow
> ========================
>         qemu on the destination
>               |
>               V
>         open(/dev/uvmem)
>               |
>               V
>         UVMEM_INIT
>               |
>               V
>         Here we have two file descriptors to
>         umem device and shmem file
>               |
>               |                                  umem threads
>               |                                  on the destination
>               |
>               V    create pipe to communicate
>         crete threads--------------------------------,
>               |                                      |
>               V                                   mmap(shmem file)
>         mmap(uvmem device) for guest RAM          close(shmem file)
>               |                                      |
>               |                                      |
>               V                                      |
>         wait for ready from daemon <----pipe-----send ready message
>               |                                      |
>               |                                 Here the daemon takes over
>         send ok------------pipe---------------> the owner of the socket
>               |                                 to the source
>               V                                      |
>         entering post copy stage                     |
>         start guest execution                        |
>               |                                      |
>               V                                      V
>         access guest RAM                          read() to get faulted
> pages
>               |                                      |
>               V                                      V
>         page fault ------------------------------>page offset is returned
>         block                                        |
>                                                      V
>                                                   pull page from the
> source
>                                                   write the page contents
>                                                   to the shmem.
>                                                      |
>                                                      V
>         unblock     <-----------------------------write() to tell served
> pages
>         the fault handler returns the page           |
>         page fault is resolved                       |
>               |                                      V
>               |                                   touch guest RAM pages
>               |                                      |
>               |                                      V
>               |                                   release the cached page
>               |                                   madvise(MADV_REMOVE)
>               |
>               |
>               |                                   pages can be sent
>               |                                   backgroundly
>               |                                      |
>               |                                      V
>               |                                   mark page is cached
>               |                                   Thus future page fault
> is
>               |                                   avoided.
>               |                                      |
>               |                                      V
>               |                                   touch guest RAM pages
>               |                                      |
>               |                                      V
>               |                                   release the cached page
>               |                                   madvise(MADV_REMOVE)
>               |                                      |
>               V                                      V
>
>                  all the pages are pulled from the source
>
>               |                                      |
>               V                                      V
>         migration completes                        exit()
>
>
> Isaku Yamahata (32):
>   migration.c: remove redundant line in migrate_init()
>   arch_init: DPRINTF format error and typo
>   osdep: add qemu_read_full() to read interrupt-safely
>   savevm: export qemu_peek_buffer, qemu_peek_byte, qemu_file_skip,
>     qemu_fflush
>   savevm/QEMUFile: consolidate QEMUFile functions a bit
>   savevm/QEMUFile: introduce qemu_fopen_fd
>   savevm/QEMUFile: add read/write QEMUFile on memory buffer
>   savevm, buffered_file: introduce method to drain buffer of buffered
>     file
>   arch_init: export RAM_SAVE_xxx flags for postcopy
>   arch_init/ram_save: introduce constant for ram save version = 4
>   arch_init: refactor ram_save_block() and export ram_save_block()
>   arch_init/ram_save_setup: factor out bitmap alloc/free
>   arch_init/ram_load: refactor ram_load
>   arch_init: factor out logic to find ram block with id string
>   migration: export migrate_fd_completed() and migrate_fd_cleanup()
>   uvmem.h: import Linux uvmem.h and teach update-linux-headers.sh
>   osdep: add QEMU_MADV_REMOVE and tirivial fix
>   postcopy: introduce helper functions for postcopy
>   savevm: add new section that is used by postcopy
>   postcopy: implement incoming part of postcopy live migration
>   postcopy outgoing: add -p option to migrate command
>   postcopy: implement outgoing part of postcopy live migration
>   postcopy/outgoing: add -n options to disable background transfer
>   postcopy/outgoing: implement forward/backword prefault
>   arch_init: factor out setting last_block, last_offset
>   postcopy/outgoing: add movebg mode(-m) to migration command
>   arch_init: factor out ram_load
>   arch_init: export ram_save_iterate()
>   postcopy: pre+post optimization incoming side
>   arch_init: export migration_bitmap_sync and helper method to get
>     bitmap
>   postcopy/outgoing: introduce precopy_count parameter
>   postcopy: pre+post optimization outgoing side
>
> Paolo Bonzini (1):
>   split MRU ram list
>
> Umesh Deshpande (2):
>   add a version number to ram_list
>   protect the ramlist with a separate mutex
>
>  Makefile.target                 |    2 +
>  arch_init.c                     |  391 +++++---
>  arch_init.h                     |   24 +
>  buffered_file.c                 |   59 +-
>  buffered_file.h                 |    1 +
>  cpu-all.h                       |   16 +-
>  exec.c                          |   62 +-
>  hmp-commands.hx                 |   21 +-
>  hmp.c                           |   12 +-
>  linux-headers/linux/uvmem.h     |   41 +
>  migration-exec.c                |    8 +-
>  migration-fd.c                  |   23 +-
>  migration-postcopy.c            | 2019
> +++++++++++++++++++++++++++++++++++++++
>  migration-tcp.c                 |   16 +-
>  migration-unix.c                |   36 +-
>  migration.c                     |   65 +-
>  migration.h                     |   42 +
>  osdep.c                         |   24 +
>  osdep.h                         |   13 +-
>  qapi-schema.json                |    6 +-
>  qemu-common.h                   |    2 +
>  qemu-file.h                     |   12 +-
>  qmp-commands.hx                 |    4 +-
>  savevm.c                        |  223 ++++-
>  scripts/update-linux-headers.sh |    2 +-
>  sysemu.h                        |    2 +-
>  umem.c                          |  291 ++++++
>  umem.h                          |   88 ++
>  vl.c                            |    5 +-
>  29 files changed, 3265 insertions(+), 245 deletions(-)
>  create mode 100644 linux-headers/linux/uvmem.h
>  create mode 100644 migration-postcopy.c
>  create mode 100644 umem.c
>  create mode 100644 umem.h
>
> --
> 1.7.10.4
>
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to address@hidden
> More majordomo info at  http://vger.kernel.org/majordomo-info.html




--
" The production of too many useful things results in too many useless
people"