Re: [Qemu-devel] [PATCH v6 00/47] Postcopy implementation

[Dr. David Alan Gilbert]

* Li, Liang Z (address@hidden) wrote:
> Hi David,
> 
> I have tired your v6 postcopy patches and found it doesn't work. When I tried 
> to start the 
> postcopy in live migration, some errors were printed. I just did the 
> following things:
> 
> On destination side, started the qemu like this:
> 
> /root/vt-sync/post_copy_v6_qemu.git/x86_64-softmmu/qemu-system-x86_64
> -enable-kvm -smp 2 -m 1024 -net none /mnt/jinshi_ia32e_rhel6u5.qcow2
> -monitor stdio -incoming tcp:0:4444
> 
> On source side, started the qemu like this:
> 
> /root/vt-sync/post_copy_v6_qemu.git/x86_64-softmmu/qemu-system-x86_64
> -enable-kvm -smp 2 -m 1024 -net none /mnt/jinshi_ia32e_rhel6u5.qcow2
> -monitor stdio
> 
> and then
> (qemu) migrate_set_capability x-postcopy-ram on
> 
> When I started the post copy with
> (qemu) migrate -d tcp:localhost:4444
> 
> I got the error message on the source side:
> 
> (qemu) qemu-system-x86_64: socket_writev_buffer: Got err=104 for (131552/-1)
>  qemu-system-x86_64: RP: Received invalid message 0x0000 length    0x0000
> 
> and the following error on the destination side:
> 
> (qemu) qemu-system-x86_64: postcopy_ram_supported_by_host: No OS support
> qemu-system-x86_64: load of migration failed: Operation not permitted

OK, the important error here is:
           postcopy_ram_supported_by_host: No OS support

that's saying that the destination OS either:
   1) The kernel isn't the correct kernel with Andrea's userfault code compiled 
in
      (check that userfaultfd is configured into the kernel as well)
   2) That when you built the QEMU it didn't find the syscall definition for the
      userfaultfd in the header as it compiled it.

I think from that error it is (2) - so make sure that when you built the qemu
that you're using the headers from that kernel, or use the extra-cflags hack
that I mentioned in the cover letter.

Note that you need to use the kernel tree which I point to in the first message.
(The older kernel from v5 wont work).

Dave
P.S. I'm on holiday this week, so not checking work email much.

> 
> 
> the dmesg printed:
> [  233.456545] kvm: zapping shadow pages for mmio generation wraparound
> [  239.785916] kvm [11926]: vcpu0 disabled perfctr wrmsr: 0xc1 data 0xabcd
> 
> 
> The v5 patches have no such errors. Do you have any suggestion?
> 
> Liang
> 
> 
> > -----Original Message-----
> > From: address@hidden [mailto:qemu-
> > address@hidden On Behalf Of Dr. David Alan
> > Gilbert (git)
> > Sent: Wednesday, April 15, 2015 1:03 AM
> > To: address@hidden
> > Cc: address@hidden; address@hidden;
> > address@hidden; address@hidden; address@hidden;
> > address@hidden; address@hidden
> > Subject: [Qemu-devel] [PATCH v6 00/47] Postcopy implementation
> > 
> > From: "Dr. David Alan Gilbert" <address@hidden>
> > 
> >   This is the 6th cut of my version of postcopy; it is designed for use 
> > with the
> > Linux kernel additions posted by Andrea Arcangeli here:
> > 
> > git clone --reference linux -b userfault18
> > git://git.kernel.org/pub/scm/linux/kernel/git/andrea/aa.git
> > 
> > (Note this is a different API from the last version)
> > 
> > This qemu series can be found at:
> > 
> > https://github.com/orbitfp7/qemu.git
> > on the wp3-postcopy-v6 tag.
> > 
> > It addresses some but not yet all of the previous review comments; however
> > there are a couple of large simplifications, so it seems worth posting to 
> > meet
> > the new kernel API and to stop people reviewing deadcode.
> > 
> > Note: That the userfaultfd.h header is no longer included in this
> > tree:
> >       - if you're building with the appropriate kernel headers it should 
> > find it
> >       - if you're building on a host that doesn't have the kernel headers
> >         installed in the right place then:
> >            configure with:   --extra-cflags="-D__NR_userfaultfd=323"
> >            cp include/uapi/linux/userfaultfd.h into somewhere in the include
> >            path, e.g.  /usr/local/include/linux
> > 
> > v6
> >   Removed the PMI bitmaps
> >       - Andrea updated the kernel API so that userspace doesn't
> >         need to do wakeups, and thus QEMU doesn't need to keep
> >         track of which pages it's received; there is a price - which
> >         is we end up sending more dupes to the source, but it simplifies
> >         stuff a lot and makes the normal paths a lot quicker.
> >         (10s of line change in kernel, 10%-ish simplification in this code!)
> >   Changed discard message format to a simpler start/end address scheme
> >         and rework discard and chunking code to work in long's to match 
> > bitmap
> >   'qemu_get_buffer_less_copy' for postcopy pages
> >       - avoids a userspace copy since the kernel now does it
> >       - the new qemufile interface might also be useful for other places 
> > that
> >         don't need a copy (maybe xbzrle?)
> >   Changed the blockingness of the incoming fd
> >       it was incorrectly blocking during the precopy phase after a postcopy 
> > was
> >       enabled, causing the HMP to be unavailable.  It's now blocking only 
> > once
> >       the postcopy thread starts up, since it's not a coroutine it can't 
> > deal
> >       with the yields in qemu_file.
> >   An error on the return-path now marks the migration as failed
> > 
> >   Fixups from Dave Gibson's comments
> >     Removed can_postcopy, renamed save_complete to
> > save_complete_precopy
> >         added save_complete_postcopy
> >     Simplified loadvm loop exits
> >     discard message format changes above
> >     and many more smaller changes.
> > 
> >   small fixups for RCU
> > 
> > 
> > This work has been partially funded by the EU Orbit project:
> >   see http://www.orbitproject.eu/about/
> > 
> > TODO:
> >   The major work is to rework the page send/receive loops so that supporting
> >   larger host pages doesn't make it quite as messy.
> > 
> > Dr. David Alan Gilbert (47):
> >   Start documenting how postcopy works.
> >   Split header writing out of qemu_savevm_state_begin
> >   qemu_ram_foreach_block: pass up error value, and down the ramblock
> >     name
> >   Add qemu_get_counted_string to read a string prefixed by a count byte
> >   Create MigrationIncomingState
> >   Provide runtime Target page information
> >   Move copy out of qemu_peek_buffer
> >   Add qemu_get_buffer_less_copy to avoid copies some of the time
> >   Add wrapper for setting blocking status on a QEMUFile
> >   Rename save_live_complete to save_live_complete_precopy
> >   Return path: Open a return path on QEMUFile for sockets
> >   Return path: socket_writev_buffer: Block even on non-blocking fd's
> >   Migration commands
> >   Return path: Control commands
> >   Return path: Send responses from destination to source
> >   Return path: Source handling of return path
> >   ram_debug_dump_bitmap: Dump a migration bitmap as text
> >   Move loadvm_handlers into MigrationIncomingState
> >   Rework loadvm path for subloops
> >   Add migration-capability boolean for postcopy-ram.
> >   Add wrappers and handlers for sending/receiving the postcopy-ram
> >     migration messages.
> >   MIG_CMD_PACKAGED: Send a packaged chunk of migration stream
> >   migrate_init: Call from savevm
> >   Modify save_live_pending for postcopy
> >   postcopy: OS support test
> >   migrate_start_postcopy: Command to trigger transition to postcopy
> >   MIGRATION_STATUS_POSTCOPY_ACTIVE: Add new migration state
> >   Add qemu_savevm_state_complete_postcopy
> >   Postcopy: Maintain sentmap and calculate discard
> >   postcopy: Incoming initialisation
> >   postcopy: ram_enable_notify to switch on userfault
> >   Postcopy: Postcopy startup in migration thread
> >   Postcopy end in migration_thread
> >   Page request:  Add MIG_RP_MSG_REQ_PAGES reverse command
> >   Page request: Process incoming page request
> >   Page request: Consume pages off the post-copy queue
> >   postcopy_ram.c: place_page and helpers
> >   Postcopy: Use helpers to map pages during migration
> >   qemu_ram_block_from_host
> >   Don't sync dirty bitmaps in postcopy
> >   Host page!=target page: Cleanup bitmaps
> >   Postcopy; Handle userfault requests
> >   Start up a postcopy/listener thread ready for incoming page data
> >   postcopy: Wire up loadvm_postcopy_handle_ commands
> >   End of migration for postcopy
> >   Disable mlock around incoming postcopy
> >   Inhibit ballooning during postcopy
> > 
> >  arch_init.c                      | 868 
> > ++++++++++++++++++++++++++++++++++++---
> >  balloon.c                        |  11 +
> >  docs/migration.txt               | 167 ++++++++
> >  exec.c                           |  74 +++-
> >  hmp-commands.hx                  |  15 +
> >  hmp.c                            |   7 +
> >  hmp.h                            |   1 +
> >  hw/ppc/spapr.c                   |   2 +-
> >  hw/virtio/virtio-balloon.c       |   4 +-
> >  include/exec/cpu-all.h           |   2 -
> >  include/exec/cpu-common.h        |   7 +-
> >  include/migration/migration.h    | 126 +++++-
> >  include/migration/postcopy-ram.h |  88 ++++
> >  include/migration/qemu-file.h    |  15 +-
> >  include/migration/vmstate.h      |  10 +-
> >  include/qemu/typedefs.h          |   5 +
> >  include/sysemu/balloon.h         |   2 +
> >  include/sysemu/sysemu.h          |  45 +-
> >  migration/Makefile.objs          |   2 +-
> >  migration/block.c                |   9 +-
> >  migration/migration.c            | 743 +++++++++++++++++++++++++++++++--
> >  migration/postcopy-ram.c         | 715
> > ++++++++++++++++++++++++++++++++
> >  migration/qemu-file-unix.c       | 106 ++++-
> >  migration/qemu-file.c            | 100 ++++-
> >  migration/rdma.c                 |   4 +-
> >  migration/vmstate.c              |   5 +-
> >  qapi-schema.json                 |  19 +-
> >  qmp-commands.hx                  |  19 +
> >  savevm.c                         | 809 ++++++++++++++++++++++++++++++++----
> >  trace-events                     |  77 +++-
> >  30 files changed, 3832 insertions(+), 225 deletions(-)  create mode 100644
> > include/migration/postcopy-ram.h  create mode 100644 migration/postcopy-
> > ram.c
> > 
> > --
> > 2.1.0
> > 
> 
--
Dr. David Alan Gilbert / address@hidden / Manchester, UK