[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Qemu-devel] [PATCH v16 0/9] XBZRLE delta for live migration of large me

From: Orit Wasserman
Subject: [Qemu-devel] [PATCH v16 0/9] XBZRLE delta for live migration of large memory app
Date: Mon, 9 Jul 2012 21:22:41 +0300

Changes from v15:
        - Fix example in documentation
        - Fix identation in qmp-commands.hx
        - Fix missing comments from v13
        - Fix other comments by Eric Blake

Changes from v14:
        - rebase on top on Juan's patches
        - Use clz64 to calculate pow2floor (round down to power of 2)
        - Fix xbzrle_encode_buffer and xbzrle_decode_buffer
        - Fix QMP commands documentation 
Changes from v13:
        - Fix round to power of 2 of cache size
        - Add more checks to the XBZRLE encoding.
        - use comparison instead of XOR when calculating zrun_len
        - use strcmp trick for calculating nzrun_len (algorithm from Eric Blake)
        - Fix other comments by Eric Blake
        - Fix comments from Blue Swirl
        - Display migration statics after migration completes
Changes from v12:
        - use bool for blk and shared params
        - use long when decoding buffer
        - fix QMP commands
        - always display migration parameters in "info migrate"
        - update current_addr inside the while loop in ram_save_block
        - display statistics after migration completes
        - fix other review comments from Eric Blake

Changes from v11: 
        - divide patch 7 to several smaller patches.
        - Use an array for setting migration parameters QMP only (there
          is not support for arrays in HMP commands). parameters can be enabled
          or disabled.
        - Do not use XBZRLE in stage 3 , it is a very sensitive stage and CPU
          can be an issue.
        - Fix review comments by Juan Quintela and Eric Blake

Changes from v10:
        - Cache size will be in bytes, in case it is not a power of 2 it will be
          reduced to the nearest power of 2.
        - fix documentation
        - use cache_init with number of pages not cache size.

Changes from v9:
        - move cache implementation to separate files. Kept our own 
implementation because GCache or GHashTable have no size limit.
        - Add migrate_set_parameter function
        - removed XBZRLE option from migrate command
        - add cache size information to query_migrate command
        - add documantation file
        - write/read the exact XBZRLE header format
        - fix other review comments by Anthony and Juan

Changes from v8:
        Implement more effiecent cache_resize method
        fix set_cachesize command 

Changes from v7:
        Copy current page before encoding it, this will prevents page content
        change during the encoding.
        Allow changing the cache size during an active migration.
        Fix comments by Avi.

Changes from v6:
 1) add assert checks to ULEB encoding/decoding
 2) no need to send last zero run
Changes from v5:
1) Add migration capabilities
2) Use ULEB to encode run length
3) Do not send unmodified (dirty) page
3) Fix other patch comments

Using GCache or GHashTable requires allocating new buffer on every content 
change and have no size limit ,
so I decided to keep the simple cache implementation.

Changes from v4:
1) Rebase
2) divide patch into 9 patches
3) move memory allocation into cache_insert

Future work :
     Use SSE for encoding.
     Page ranking acording to their dirty rate and automatic 
activation/deactivation of the feature - will be sent in a separate patch 

By using XBZRLE (Xor Based Zero Run Length Encoding) we can reduce VM downtime
and total live-migration time of VMs running memory write intensive workloads
typical of large enterprise applications such as SAP ERP Systems, and generally
speaking for any application with a sparse memory update pattern.

The compression format uses the fact that we will have many zero (zero 
an unchanged value). 
We repesent the page data delta by zero and non zero runs.
We represent a zero run with it's length (in bytes). 
We represent a non zero run with it's length (in bytes) and the data.
The run length is encoded using ULEB128 (http://en.wikipedia.org/wiki/LEB128)

page = zrun nzrun
       | zrun nzrun page

zrun = length

nzrun = length byte...

length = uleb128 encoded integer

On the sender side XBZRLE is used as a compact delta encoding of page updates,
retrieving the old page content from an LRU cache (default size of 512 MB). The
receiving side uses the existing page content and XBZRLE to decode the new page

This is a more compact way to store the delta than the previous version.

This work was originally based on research results published VEE 2011: 
Evaluation of
Delta Compression Techniques for Efficient Live Migration of Large Virtual
Machines by Benoit, Svard, Tordsson and Elmroth. Additionally the delta encoder
XBRLE was improved further using XBZRLE instead.

XBZRLE has a sustained bandwidth of 2-2.5 GB/s for typical workloads making it
ideal for in-line, real-time encoding such as is needed for live-migration.

A typical usage scenario:
On incoming QEMU:
    {qemu} migrate_set_parameter xbzrle on
On outgoing QEMU:
    {qemu} migrate_set_parameter xbzrle on
    {qemu} migrate_set_cachesize 256m
    {qemu} migrate -d tcp:destination.host:4444
    {qemu} info migrate
    cache size : 64 MBytes
    transferred ram: A kbytes
    remaining ram: B kbytes
    total ram: C kbytes
    cache size: D bytes
    xbzrle transferred: E kbytes
    xbzrle pages: F pages
    xbzrle cache miss: G
    xbzrle overflow : H

Testing: live migration with XBZRLE completed in 110 seconds, without live
migration was not able to complete.

A simple synthetic memory r/w load generator:
..    include <stdlib.h>
..    include <stdio.h>
..    int main()
..    {
..        char *buf = (char *) calloc(4096, 4096);
..        while (1) {
..            int i;
..            for (i = 0; i < 4096 * 4; i++) {
..                buf[i * 4096 / 4]++;
..            }
..            printf(".");
..        }
..    }

Orit Wasserman (9):
  Add migration capabilities
  Add XBZRLE documentation
  Add cache handling functions
  Add uleb encoding/decoding functions
  Change ram_save_block to return -1 if there are no more changes
  Add xbzrle_encode_buffer and xbzrle_decode_buffer functions
  Add XBZRLE to ram_save_block and ram_save_live
  Add set_cachesize command
  Add XBZRLE statistics

 Makefile.objs             |    1 +
 arch_init.c               |  272 +++++++++++++++++++++++++++++++++++++++++++--
 cutils.c                  |   41 +++++++
 docs/xbzrle.txt           |  136 ++++++++++++++++++++++
 hmp-commands.hx           |   36 ++++++
 hmp.c                     |   90 +++++++++++++++
 hmp.h                     |    3 +
 include/qemu/page_cache.h |   79 +++++++++++++
 migration.c               |  159 ++++++++++++++++++++++++++-
 migration.h               |   21 ++++
 monitor.c                 |    7 +
 page_cache.c              |  216 +++++++++++++++++++++++++++++++++++
 qapi-schema.json          |  102 ++++++++++++++++-
 qemu-common.h             |   21 ++++
 qmp-commands.hx           |  127 ++++++++++++++++++++-
 savevm.c                  |  159 ++++++++++++++++++++++++++
 16 files changed, 1452 insertions(+), 18 deletions(-)
 create mode 100644 docs/xbzrle.txt
 create mode 100644 include/qemu/page_cache.h
 create mode 100644 page_cache.c


reply via email to

[Prev in Thread] Current Thread [Next in Thread]