[Qemu-devel] [RFC 00/13] Live memory snapshot based on userfaultfd

[zhanghailiang]

For now, we still didn't support live memory snapshot, we have discussed
a scheme which based on userfaultfd long time ago.
You can find the discussion by the follow link:
https://lists.nongnu.org/archive/html/qemu-devel/2014-11/msg01779.html

The scheme is based on userfaultfd's write-protect capability.
The userfaultfd write protection feature is available here:
http://www.spinics.net/lists/linux-mm/msg97422.html

The process of this live memory scheme is like bellow:
1. Pause VM
2. Enable write-protect fault notification by using userfaultfd to
   mark VM's memory to write-protect (readonly).
3. Save VM's static state (here is device state) to snapshot file
4. Resume VM, VM is going to run.
5. Snapshot thread begins to save VM's live state (here is RAM) into
   snapshot file.
6. During this time, all the actions of writing VM's memory will be blocked
  by kernel, and kernel will wakeup the fault treating thread in qemu to
  process this write-protect fault. The fault treating thread will deliver this
  page's address to snapshot thread.
7. snapshot thread gets this address, save this page into snasphot file,
   and then remove the write-protect by using userfaultfd API, after that,
   the actions of writing will be recovered. 
8. Repeat step 5~7 until all VM's memory is saved to snapshot file

Compared with the feature of 'migrate VM's state to file',
the main difference for live memory snapshot is it has little time delay for
catching VM's state. It just captures the VM's state while got users snapshot
command, just like take a photo of VM's state.

For now, we only support tcg accelerator, since userfaultfd is not supporting
tracking write faults for KVM.

Usage:
1. Take a snapshot
#x86_64-softmmu/qemu-system-x86_64 -machine pc-i440fx-2.5,accel=tcg,usb=off 
-drive 
file=/mnt/windows/win7_install.qcow2.bak,if=none,id=drive-ide0-0-1,format=qcow2,cache=none
 -device ide-hd,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1  -vnc :7 -m 
8192 -smp 1 -netdev tap,id=bn0 -device virtio-net-pci,id=net-pci0,netdev=bn0  
--monitor stdio
Issue snapshot command:
(qemu)migrate -d file:/home/Snapshot
2. Revert to the snapshot
#x86_64-softmmu/qemu-system-x86_64 -machine pc-i440fx-2.5,accel=tcg,usb=off 
-drive 
file=/mnt/windows/win7_install.qcow2.bak,if=none,id=drive-ide0-0-1,format=qcow2,cache=none
 -device ide-hd,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1  -vnc :7 -m 
8192 -smp 1 -netdev tap,id=bn0 -device virtio-net-pci,id=net-pci0,netdev=bn0  
--monitor stdio -incoming file:/home/Snapshot

NOTE:
The userfaultfd write protection feature does not support THP for now,
Before taking snapshot, please disable THP by:
echo never > /sys/kernel/mm/transparent_hugepage/enabled

TODO:
- Reduce the influence for VM while taking snapshot

zhanghailiang (13):
  postcopy/migration: Split fault related state into struct
    UserfaultState
  migration: Allow the migrate command to work on file: urls
  migration: Allow -incoming to work on file: urls
  migration: Create a snapshot thread to realize saving memory snapshot
  migration: implement initialization work for snapshot
  QEMUSizedBuffer: Introduce two help functions for qsb
  savevm: Split qemu_savevm_state_complete_precopy() into two helper
    functions
  snapshot: Save VM's device state into snapshot file
  migration/postcopy-ram: fix some helper functions to support
    userfaultfd write-protect
  snapshot: Enable the write-protect notification capability for VM's
    RAM
  snapshot/migration: Save VM's RAM into snapshot file
  migration/ram: Fix some helper functions' parameter to use
    PageSearchStatus
  snapshot: Remove page's write-protect and copy the content during
    setup stage

 include/migration/migration.h     |  41 +++++--
 include/migration/postcopy-ram.h  |   9 +-
 include/migration/qemu-file.h     |   3 +-
 include/qemu/typedefs.h           |   1 +
 include/sysemu/sysemu.h           |   3 +
 linux-headers/linux/userfaultfd.h |  21 +++-
 migration/fd.c                    |  51 ++++++++-
 migration/migration.c             | 101 ++++++++++++++++-
 migration/postcopy-ram.c          | 229 ++++++++++++++++++++++++++++----------
 migration/qemu-file-buf.c         |  61 ++++++++++
 migration/ram.c                   | 104 ++++++++++++-----
 migration/savevm.c                |  90 ++++++++++++---
 trace-events                      |   1 +
 13 files changed, 587 insertions(+), 128 deletions(-)

-- 
1.8.3.1