Re: [Qemu-devel] [RFC 00/13] Multiple fd migration support

[Dr. David Alan Gilbert]

* Juan Quintela (address@hidden) wrote:
> Hi
> 
> This patch series is "an" initial implementation of multiple fd migration.
> This is to get something out for others to comment, it is not finished at all.

I've had a quick skim:
  a) I think mst is right about the risk of getting stale pages out of order.
  b) Since you don't change the URI at all, it's a bit restricted; for example,
     it means I can't run separate sessions over different NICs unless I've
     done something clever at the routing/or bonded them.
     One thing I liked the sound of multi-fd for is NUMA; get a BIG box
     and give each numa node a separate NIC and run a separate thread on each
     node.
  c) Hmm we do still have a single thread doing all the bitmap syncing and 
scanning,
     we'll have to watch out if that is the bottleneck at all.
  d) All the zero testing is still done in the main thread which we know is
     expensive.
  e) Do we need to do something for security with having multiple ports? How
     do we check that nothing snuck in on one of our extra ports, have we got
     sanity checks to make sure it's actually the right stream.
  f) You're handing out pages to the sending threads on the basis of which one
     is free (in the same way as the multi threaded compression); but I think
     it needs some sanity adding to only hand out whole host pages - it feels
     like receiving all the chunks of one host page down separate FDs would
     be horrible.
  g) I think you might be able to combine the compression into the same threads;
     so that if multi-fd + multi-threaded-compresison is set you don't end
     up with 2 sets of threads and it might be the simplest way to make them
     work together.
  h) You've used the last free RAM_SAVE_FLAG!  And the person who takes the last
     slice^Wbit has to get some more.
     Since arm, ppc, and 68k have variants that have TARGET_PAGE_BITS 10  that
     means we're full; I suggest what you do is use that flag to mean that we
     send another 64bit word; and in that word you use the bottom 7 bits for
     the fd index and bit 7 is set to indicate it's fd.  The other bits are sent
     as zero and available for the next use.
     Either that or start combining with some other flags.
     (I may have a use for some more bits in mind!)
  i) Is this safe for xbzrle - what happens to the cache (or is it all
     still the main thread?)
  j) For postcopy I could do with a separate fd for the requested pages
     (but again that comes back to needing an easy solution to the ordering)

Dave

> 
> So far:
> 
> - we create threads for each new fd
> 
> - only for tcp of course, rest of transports are out of luck
>   I need to integrate this with daniel channel changes
> 
> - I *think* the locking is right, at least I don't get more random
>   lookups (and yes, it was not trivial).  And yes, I think that the
>   compression code locking is not completely correct.  I think it
>   would be much, much better to do the compression code on top of this
>   (will avoid a lot of copies), but I need to finish this first.
> 
> - Last patch, I add a BIG hack to try to know what the real bandwidth
>   is.
> 
> 
> Preleminar testing so far:
> 
> - quite good, the latency is much better, but was change so far, I
>   think I found the problem for the random high latencies, but more
>   testing is needed.
> 
> - under load, I think our bandwidth calculations are *not* completely
>   correct (This is the way to spell it to be allowed for a family audience).
> 
> 
> ToDo list:
> - bandwidth calculation: I am going to send another mail
>   with my ToDo list for migration, see there.
> 
> - stats: We need better stats, by thread, etc
> 
> - sincronize less times with the worker threads.
>   right now we syncronize for each page, there are two obvious optimizations
>   * send a list of pages each time we wakeup an fd
>   * if we have to sent a HUGE page, dont' do a single split, just sent the 
> whole page
>     in one send() and read things with a single recv() on destination.
>     My understanding is that this would make Transparent Huge pages trivial.
> - measure things under bigger loads
> 
> Comments, please?
> 
> Later, Juan.
> 
> Juan Quintela (13):
>   migration: create Migration Incoming State at init time
>   migration: Pass TCP args in an struct
>   migration: [HACK] Don't create decompression threads if not enabled
>   migration: Add multifd capability
>   migration: Create x-multifd-threads parameter
>   migration: create multifd migration threads
>   migration: Start of multiple fd work
>   migration: create ram_multifd_page
>   migration: Create thread infrastructure for multifd send side
>   migration: Send the fd number which we are going to use for this page
>   migration: Create thread infrastructure for multifd recv side
>   migration: Test new fd infrastructure
>   migration: [HACK]Transfer pages over new channels
> 
>  hmp.c                         |  10 ++
>  include/migration/migration.h |  13 ++
>  migration/migration.c         | 100 ++++++++----
>  migration/ram.c               | 350 
> +++++++++++++++++++++++++++++++++++++++++-
>  migration/savevm.c            |   3 +-
>  migration/tcp.c               |  76 ++++++++-
>  qapi-schema.json              |  29 +++-
>  7 files changed, 540 insertions(+), 41 deletions(-)
> 
> -- 
> 2.5.5
> 
--
Dr. David Alan Gilbert / address@hidden / Manchester, UK