Re: [Qemu-devel] [PATCH v6 00/11] rdma: migration support

[Michael R. Hines]

All,

I have successfully performed over 1000+ back-to-back RDMA migrations 
automatically looped *in a row* using a heavy-weight memory-stress 
benchmark here at IBM.
Migration success is done by capturing the actual serial console output 
of the virtual machine while the benchmark is running and redirecting 
each migration output to a file to verify that the output matches the 
expected output of a successful migration. For half of the 1000 
migrations, I used a 14GB virtual machine size (largest VM I can create) 
and the remaining 500 migrations I used a 2GB virtual machine (to make 
sure I was testing both 32-bit and 64-bit address boundaries). The 
benchmark is configured to have 75% stores and 25% loads and is 
configured to use 80% of the allocatable free memory of the VM (i.e. no 
swapping allowed).

I have defined a successful migration per the output file as follows:

1. The memory benchmark is still running and active (CPU near 100% and 
memory usage is high)
2. There are no kernel panics in the console output (regex keywords 
"panic", "BUG", "oom", etc...)
3. The VM is still responding to network activity (pings)
4. The console is still responsive by printing periodic messages 
throughout the life of the VM to the console from inside the VM using 
the 'write' command in infinite loop.

With this method in a loop, I believe I've ironed out all the 
regression-testing bugs that I can find. You all may find the following 
bugs interesting. The original version of this patch was written in 2010 
(Before my time @ IBM).

Bug #1: In the original 2010 patch, each write operation uses the same 
"identifier". (A "Work Request ID" in infiniband terminology).
This is not typical (but allowed by the hardware) - and instead each 
operation should have its own unique identifier so that the write 
operation can be tracked properly as it completes.

Bug #2: Also in the original 2010 patch, write operations were grouped 
into separate "signaled" and "unsignaled" work requests, which is also 
not typical (but allowed by the hardware). "Signalling" is infiniband 
terminology which means to activate/deactivate notifying the sender 
whether or not the RDMA operation has already completed. (Note: the 
receiver is never notified - which is what a DMA is supposed to be). In 
normal operation per infiniband specifications, "unsignaled" operations 
(which indicate to the hardware *not* to notify the sender of 
completion) are *supposed* to be paired simultaneously with a signaled 
operation using the *same* work request identifier. Instead, the 
original patch was using *different* work requests for 
signaled/unsignaled writes, which means that most of the writes would be 
transmitted without ever being tracked for completion whatsoever. (Per 
infinband specifications, signaled and unsignaled writes must be grouped 
together because the hardware ensures that completion notification is 
not given until *all* of the writes of the same request have actually 
completed).

Bug #3: Finally, in the original 2010 patch, ordering was not being 
handled. Per infiniband specifications, writes can happen completely out 
of order. Not only that, but PCI-express itself can change the order of 
the writes as well. It was only until after the first 2 bugs were fixed 
that I could actually manifest this bug *in code*: What was happening 
was that a very large group of requests would "burst" from the QEMU 
migration thread. At which point, not all of the requests would finish. 
Then a short time later, the next iteration would start and the virtual 
machine's writable working set was still "hovering" somewhere in the 
same vicinity of the address space as the previous burst of writes that 
had not yet completed. When this happens, the new writes were much 
smaller (not a part of a larger "chunk" per our algorithms). Since the 
new writes were smaller they would complete faster than the larger, 
older writes in the same address range. Since they complete out of 
order, the newer writes would then get clobbered by the older writes - 
resulting in an inconsistent virtual machine. So, to solve this: during 
each new write, we now do a "search" to see if the address of the next 
requested write matches or overlaps with the address range of any of the 
previous "outstanding" writes that were still in transit, and I found 
several hits. This was easily solved by blocking until the conflicting 
write has completed before proceeding to issue a new write to the hardware.

- Michael


On 05/09/2013 06:45 PM, Michael R. Hines wrote:
> Some more followup questions below to help me debug before I start 
> digging in.......
>
> On 05/09/2013 06:20 PM, Chegu Vinod wrote:
>
> Setting aside the mlock() freezes for the moment, let's first fix your 
> crashing
> problem on the destination-side. Let's make that a priority before we fix
> the mlock problem.
>
> When the migration "completes", can you provide me with more detailed 
> information
> about the state of QEMU on the destination?
>
> Is it responding?
> What's on the VNC console?
> Is QEMU responding?
> Is the network responding?
> Was the VM idle? Or running an application?
> Can you attach GDB to QEMU after the migration?
>
>
> > /usr/local/bin/qemu-system-x86_64 \
> > -enable-kvm \
> > -cpu host \
> > -name vm1 \
> > -m 131072 -smp 10,sockets=1,cores=10,threads=1 \
> > -mem-path /dev/hugepages \
>
> Can you disable hugepages and re-test?
>
> I'll get back to the other mlock() issues later after we at least 
> first make sure the migration itself is working.....