[PATCH v5 0/8] Migration: Transmit and detect zero pages in the multifd threads

[Juan Quintela]

Hi

In this version:
- Rebase to latest
- Address all comments
- statistics about zero pages are right now (or at least much better than 
before)
- changed how we calculate the amount of transferred ram
- numbers, who don't like numbers.

Everything has been checked with a guest launched like the following
command.  Migration is running through localhost.  Will send numbers
with real hardware as soon as I get access to the machines that have
it (I checked with previous versions already, but not this one).

/scratch/qemu/multifd/x64/x86_64-softmmu/qemu-system-x86_64
-name guest=src,debug-threads=on
-m 16G
-smp 6
-machine q35,accel=kvm,usb=off,dump-guest-core=off
-boot strict=on
-cpu host
-no-hpet
-rtc base=utc,driftfix=slew
-global kvm-pit.lost_tick_policy=delay
-global ICH9-LPC.disable_s3=1
-global ICH9-LPC.disable_s4=1
-device pcie-root-port,id=root.1,chassis=1,addr=0x2.0,multifunction=on
-device pcie-root-port,id=root.2,chassis=2,addr=0x2.1
-device pcie-root-port,id=root.3,chassis=3,addr=0x2.2
-device pcie-root-port,id=root.4,chassis=4,addr=0x2.3
-device pcie-root-port,id=root.5,chassis=5,addr=0x2.4
-device pcie-root-port,id=root.6,chassis=6,addr=0x2.5
-device pcie-root-port,id=root.7,chassis=7,addr=0x2.6
-device pcie-root-port,id=root.8,chassis=8,addr=0x2.7
-blockdev 
driver=file,node-name=storage0,filename=/mnt/vm/test/a3.qcow2,auto-read-only=true,discard=unmap
-blockdev driver=qcow2,node-name=format0,read-only=false,file=storage0
-device virtio-blk-pci,id=virtio-disk0,drive=format0,bootindex=1,bus=root.1
-netdev tap,id=hostnet0,vhost=on,script=/etc/kvm-ifup,downscript=/etc/kvm-ifdown
-device virtio-net-pci,id=net0,netdev=hostnet0,mac=52:54:00:9d:30:23,bus=root.2
-device virtio-serial-pci,id=virtio-serial0,bus=root.3
-device virtio-balloon-pci,id=balloon0,bus=root.4 -display none
-chardev socket,id=charconsole0,path=/tmp/console-src,server=off
-device virtconsole,id=console0,chardev=charconsole0
-uuid 9d3be7da-e1ff-41a0-ac39-8b2e04de2c19
-nodefaults
-msg timestamp=on
-no-user-config
-chardev socket,id=monitor0,path=/tmp/monitor-src,server=off
-mon monitor0
-trace events=/home/quintela/tmp/events
-global migration.x-multifd=on
-global migration.multifd-channels=4
-global migration.x-max-bandwidth=6442450944

Using maximum bandwith of 6GB/s (on my test machine bigger than that
will not help, and then we are not checking that limitation works).

First, the part where multifd + zero pages don't shine.

Migrate the guest just booted and idle.

Precopy (changed multifd=off on the command line)

Migration status: completed
total time: 851 ms
downtime: 29 ms
setup: 4 ms
transferred ram: 634400 kbytes
throughput: 6139.32 mbps
duplicate: 4045125 pages
normal: 149420 pages
normal bytes: 597680 kbytes
dirty sync count: 3
multifd bytes: 0 kbytes
pages-per-second: 3088960
precopy ram: 598949 kbytes
downtime ram: 35450 kbytes

Current multifd:

Migration status: completed
total time: 922 ms
downtime: 27 ms
setup: 4 ms
transferred ram: 621342 kbytes
throughput: 5547.97 mbps
total ram: 16777992 kbytes
duplicate: 4048826 pages
normal: 146032 pages
normal bytes: 584128 kbytes
dirty sync count: 3
multifd bytes: 585757 kbytes
pages-per-second: 5918137
precopy ram: 35585 kbytes

Migration status: completed
total time: 946 ms
downtime: 60 ms
setup: 10 ms
transferred ram: 621784 kbytes
throughput: 5445.14 mbps
total ram: 16777992 kbytes
duplicate: 4048586 pages
normal: 146146 pages
normal bytes: 584584 kbytes
dirty sync count: 3
multifd bytes: 586201 kbytes
pages-per-second: 5297106
precopy ram: 35582 kbytes
downtime ram: 1 kbytes

See that times are similar, accounting is wrong for multifd of
downtime ram, for instance.  Notice that multifd is almosnt not used,
because almost everything is a zero page that is sent through the main
channel.

Multifd + zero page:

Migration status: completed
total time: 1854 ms
downtime: 150 ms
setup: 4 ms
transferred ram: 12086679 kbytes
throughput: 53522.74 mbps
total ram: 16777992 kbytes
duplicate: 4039509 pages
normal: 146476 pages
normal bytes: 585904 kbytes
dirty sync count: 3
multifd bytes: 12086679 kbytes
pages-per-second: 2274851
precopy ram: 10698366 kbytes
downtime ram: 1388313 kbytes

Migration status: completed
total time: 1547 ms
downtime: 143 ms
setup: 4 ms
transferred ram: 9877449 kbytes
throughput: 52442.68 mbps
total ram: 16777992 kbytes
duplicate: 4037502 pages
normal: 149056 pages
normal bytes: 596224 kbytes
dirty sync count: 3
multifd bytes: 9877449 kbytes
pages-per-second: 3011840
precopy ram: 8811411 kbytes
downtime ram: 1066038 kbytes

Here, we are sending zero pages through multifd channels, so the
packages that we sent through multifd channels are smaller, and the
bandwidth that we got is worse than sending them through the main
channel.  Notice that the bandwidth numbers are now correct.

Now a better scenary.  We use the same guest, but we run on it:

 stress --vm 4 --vm-keep --vm-bytes 400M

i.e. 4 threads dirtying 1.6GB in total in the 6GB test.

Precopy:

total time: 120947 ms
expected downtime: 678 ms
setup: 5 ms
transferred ram: 283655333 kbytes
throughput: 19818.30 mbps
remaining ram: 1048116 kbytes
total ram: 16777992 kbytes
duplicate: 3635761 pages
normal: 70767622 pages
normal bytes: 283070488 kbytes
dirty sync count: 286
multifd bytes: 0 kbytes
pages-per-second: 603627
dirty pages rate: 598555 pages
precopy ram: 283655333 kbytes

As you can see, it will never converge.

Current multifd:

Migration status: completed
total time: 2249 ms
downtime: 273 ms
setup: 6 ms
transferred ram: 5432984 kbytes
throughput: 19843.96 mbps
total ram: 16777992 kbytes
duplicate: 3635610 pages
normal: 1346731 pages
normal bytes: 5386924 kbytes
dirty sync count: 7
multifd bytes: 5401030 kbytes
pages-per-second: 1388326
precopy ram: 31953 kbytes

Migration status: completed
total time: 3383 ms
downtime: 230 ms
setup: 4 ms
transferred ram: 10828047 kbytes
throughput: 26252.25 mbps
total ram: 16777992 kbytes
duplicate: 3638556 pages
normal: 2691985 pages
normal bytes: 10767940 kbytes
dirty sync count: 22
page size: 4 kbytes
multifd bytes: 10796067 kbytes
pages-per-second: 1375137
precopy ram: 31980 kbytes

see, that it depends on luck what time we get, but we end converging
sooner or later (yes, I started with 512MB, but multifd will not
converge with that).

Multifd + zero page

Migration status: completed
total time: 2785 ms
downtime: 292 ms
setup: 4 ms
transferred ram: 16108184 kbytes
throughput: 47451.00 mbps
total ram: 16777992 kbytes
duplicate: 3633781 pages
normal: 957319 pages
normal bytes: 3829276 kbytes
dirty sync count: 4
multifd bytes: 16108184 kbytes
pages-per-second: 1861120
precopy ram: 14303936 kbytes
downtime ram: 1804247 kbytes

Migration status: completed
total time: 2789 ms
downtime: 302 ms
setup: 6 ms
transferred ram: 16198682 kbytes
throughput: 47683.29 mbps
total ram: 16777992 kbytes
duplicate: 3631338 pages
normal: 959689 pages
normal bytes: 3838756 kbytes
dirty sync count: 4
multifd bytes: 16198681 kbytes
pages-per-second: 1820160
precopy ram: 14385777 kbytes
downtime ram: 1812904 kbytes

Notice that we only need 4 iterations, on normal multd, it depends on luck.
Statistics are right here, and we are much more consistent in the total time 
that we need.

Please review, Juan.

[v4]
In this version
- Rebase to latest
- Address all comments from previous versions
- code cleanup

Please review.

[v2]
This is a rebase against last master.

And the reason for resend is to configure properly git-publish and
hope this time that git-publish send all the patches.

Please, review.

[v1]
Since Friday version:
- More cleanups on the code
- Remove repeated calls to qemu_target_page_size()
- Establish normal pages and zero pages
- detect zero pages on the multifd threads
- send zero pages through the multifd channels.
- reviews by Richard addressed.

It pases migration-test, so it should be perfect O:+)

ToDo for next version:
- check the version changes
  I need that 6.2 is out to check for 7.0.
  This code don't exist at all due to that reason.
- Send measurements of the differences

Please, review.

[

Friday version that just created a single writev instead of
write+writev.

]

Right now, multifd does a write() for the header and a writev() for
each group of pages.  Simplify it so we send the header as another
member of the IOV.

Once there, I got several simplifications:
* is_zero_range() was used only once, just use its body.
* same with is_zero_page().
* Be consintent and use offset insed the ramblock everywhere.
* Now that we have the offsets of the ramblock, we can drop the iov.
* Now that nothing uses iov's except NOCOMP method, move the iovs
  from pages to methods.
* Now we can use iov's with a single field for zlib/zstd.
* send_write() method is the same in all the implementaitons, so use
  it directly.
* Now, we can use a single writev() to write everything.

ToDo: Move zero page detection to the multifd thrteads.

With RAM in the Terabytes size, the detection of the zero page takes
too much time on the main thread.

Last patch on the series removes the detection of zero pages in the
main thread for multifd.  In the next series post, I will add how to
detect the zero pages and send them on multifd channels.

Please review.

Later, Juan.

Juan Quintela (8):
  migration: Export ram_transferred_ram()
  multifd: Count the number of sent bytes correctly
  migration: Make ram_save_target_page() a pointer
  multifd: Add property to enable/disable zero_page
  migration: Export ram_release_page()
  multifd: Support for zero pages transmission
  multifd: Zero pages transmission
  migration: Use multifd before we check for the zero page

 migration/migration.h  |  3 ++
 migration/multifd.h    | 24 ++++++++++-
 migration/ram.h        |  3 ++
 hw/core/machine.c      |  4 +-
 migration/migration.c  | 11 +++++
 migration/multifd.c    | 93 ++++++++++++++++++++++++++++++++++--------
 migration/ram.c        | 48 ++++++++++++++++++----
 migration/trace-events |  8 ++--
 8 files changed, 162 insertions(+), 32 deletions(-)

-- 
2.34.1