[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Qemu-devel] [PATCH 00/21][RFC] postcopy live migration
From: |
Isaku Yamahata |
Subject: |
[Qemu-devel] [PATCH 00/21][RFC] postcopy live migration |
Date: |
Thu, 29 Dec 2011 10:25:39 +0900 |
Intro
=====
This patch series implements postcopy live migration.[1]
As discussed at KVM forum 2011, dedicated character device is used for
distributed shared memory between migration source and destination.
Now we can discuss/benchmark/compare with precopy. I believe there are
much rooms for improvement.
[1] http://wiki.qemu.org/Features/PostCopyLiveMigration
Usage
=====
You need load umem character device on the host before starting migration.
Postcopy can be used for tcg and kvm accelarator. The implementation depend
on only linux umem character device. But the driver dependent code is split
into a file.
I tested only host page size == guest page size case, but the implementation
allows host page size != guest page size case.
The following options are added with this patch series.
- incoming part
command line options
-postcopy [-postcopy-flags <flags>]
where flags is for changing behavior for benchmark/debugging
Currently the following flags are available
0: default
1: enable touching page request
example:
qemu -postcopy -incoming tcp:0:4444 -monitor stdio -machine accel=kvm
- outging part
options for migrate command
migrate [-p [-n]] URI
-p: indicate postcopy migration
-n: disable background transferring pages: This is for benchmark/debugging
example:
migrate -p -n tcp:<dest ip address>:4444
TODO
====
- benchmark/evaluation. Especially how async page fault affects the result.
- improve/optimization
At the moment at least what I'm aware of is
- touching pages in incoming qemu process by fd handler seems suboptimal.
creating dedicated thread?
- making incoming socket non-blocking
- outgoing handler seems suboptimal causing latency.
- catch up memory API change
- consider on FUSE/CUSE possibility
- and more...
basic postcopy work flow
========================
qemu on the destination
|
V
open(/dev/umem)
|
V
UMEM_DEV_CREATE_UMEM
|
V
Here we have two file descriptors to
umem device and shmem file
|
| umemd
| daemon on the destination
|
V create pipe to communicate
fork()---------------------------------------,
| |
V |
close(socket) V
close(shmem) mmap(shmem file)
| |
V V
mmap(umem device) for guest RAM close(shmem file)
| |
close(umem device) |
| |
V |
wait for ready from daemon <----pipe-----send ready message
| |
| Here the daemon takes over
send ok------------pipe---------------> the owner of the socket
| to the source
V |
entering post copy stage |
start guest execution |
| |
V V
access guest RAM UMEM_GET_PAGE_REQUEST
| |
V V
page fault ------------------------------>page offset is returned
block |
V
pull page from the source
write the page contents
to the shmem.
|
V
unblock <-----------------------------UMEM_MARK_PAGE_CACHED
the fault handler returns the page
page fault is resolved
|
| pages can be sent
| backgroundly
| |
| V
| UMEM_MARK_PAGE_CACHED
| |
V V
The specified pages<-----pipe------------request to touch pages
are made present by |
touching guest RAM. |
| |
V V
reply-------------pipe-------------> release the cached page
| madvise(MADV_REMOVE)
| |
V V
all the pages are pulled from the source
| |
V V
the vma becomes anonymous<----------------UMEM_MAKE_VMA_ANONYMOUS
(note: I'm not sure if this can be implemented or not)
| |
V V
migration completes exit()
Isaku Yamahata (21):
arch_init: export sort_ram_list() and ram_save_block()
arch_init: export RAM_SAVE_xxx flags for postcopy
arch_init/ram_save: introduce constant for ram save version = 4
arch_init: refactor host_from_stream_offset()
arch_init/ram_save_live: factor out RAM_SAVE_FLAG_MEM_SIZE case
arch_init: refactor ram_save_block()
arch_init/ram_save_live: factor out ram_save_limit
arch_init/ram_load: refactor ram_load
exec.c: factor out qemu_get_ram_ptr()
exec.c: export last_ram_offset()
savevm: export qemu_peek_buffer, qemu_peek_byte, qemu_file_skip
savevm: qemu_pending_size() to return pending buffered size
savevm, buffered_file: introduce method to drain buffer of buffered
file
migration: export migrate_fd_completed() and migrate_fd_cleanup()
migration: factor out parameters into MigrationParams
umem.h: import Linux umem.h
update-linux-headers.sh: teach umem.h to update-linux-headers.sh
configure: add CONFIG_POSTCOPY option
postcopy: introduce -postcopy and -postcopy-flags option
postcopy outgoing: add -p and -n option to migrate command
postcopy: implement postcopy livemigration
Makefile.target | 4 +
arch_init.c | 260 ++++---
arch_init.h | 20 +
block-migration.c | 8 +-
buffered_file.c | 20 +-
buffered_file.h | 1 +
configure | 12 +
cpu-all.h | 9 +
exec-obsolete.h | 1 +
exec.c | 75 +-
hmp-commands.hx | 12 +-
hw/hw.h | 7 +-
linux-headers/linux/umem.h | 83 ++
migration-exec.c | 8 +
migration-fd.c | 30 +
migration-postcopy-stub.c | 77 ++
migration-postcopy.c | 1891 +++++++++++++++++++++++++++++++++++++++
migration-tcp.c | 37 +-
migration-unix.c | 32 +-
migration.c | 53 +-
migration.h | 49 +-
qemu-common.h | 2 +
qemu-options.hx | 25 +
qmp-commands.hx | 10 +-
savevm.c | 31 +-
scripts/update-linux-headers.sh | 2 +-
sysemu.h | 4 +-
umem.c | 379 ++++++++
umem.h | 105 +++
vl.c | 20 +-
30 files changed, 3086 insertions(+), 181 deletions(-)
create mode 100644 linux-headers/linux/umem.h
create mode 100644 migration-postcopy-stub.c
create mode 100644 migration-postcopy.c
create mode 100644 umem.c
create mode 100644 umem.h
- [Qemu-devel] [PATCH 00/21][RFC] postcopy live migration,
Isaku Yamahata <=
- [Qemu-devel] [PATCH 02/21] arch_init: export RAM_SAVE_xxx flags for postcopy, Isaku Yamahata, 2011/12/28
- [Qemu-devel] [PATCH 07/21] arch_init/ram_save_live: factor out ram_save_limit, Isaku Yamahata, 2011/12/28
- [Qemu-devel] [PATCH 11/21] savevm: export qemu_peek_buffer, qemu_peek_byte, qemu_file_skip, Isaku Yamahata, 2011/12/28
- [Qemu-devel] [PATCH 04/21] arch_init: refactor host_from_stream_offset(), Isaku Yamahata, 2011/12/28
- [Qemu-devel] [PATCH 06/21] arch_init: refactor ram_save_block(), Isaku Yamahata, 2011/12/28
- [Qemu-devel] [PATCH 17/21] update-linux-headers.sh: teach umem.h to update-linux-headers.sh, Isaku Yamahata, 2011/12/28
- [Qemu-devel] [PATCH 08/21] arch_init/ram_load: refactor ram_load, Isaku Yamahata, 2011/12/28
- [Qemu-devel] [PATCH 03/21] arch_init/ram_save: introduce constant for ram save version = 4, Isaku Yamahata, 2011/12/28
- [Qemu-devel] [PATCH 05/21] arch_init/ram_save_live: factor out RAM_SAVE_FLAG_MEM_SIZE case, Isaku Yamahata, 2011/12/28
- [Qemu-devel] [PATCH 10/21] exec.c: export last_ram_offset(), Isaku Yamahata, 2011/12/28