Re: [Qemu-devel] [RFC PATCH RDMA support v2: 5/6] connection-setup code between client/server

[Michael R. Hines]

Orit (and anthony if you're not busy),

I forgot to respond to this very important comment:

On 02/13/2013 03:46 AM, Orit Wasserman wrote:

Are you still using the tcp for transferring device state? If so you can call the tcp functions from the migration rdma code as a first step but I would prefer it to use RDMA too.

This is the crux of the problem of using RDMA for migration: Currently all of the QEMU migration control logic and device state goes through the the QEMUFile implementation. RDMA, however is by nature a zero-copy solution and is incompatible with QEMUFile.

Using RDMA for transferring device state is not recommended: Setuping an RDMA requires registering the memory locations on both sides with the RDMA hardware. This is not ideal because this would require pinning the memory holding the device state and then issuing the RDMA transfer for *each* type of device - which would require changing the control path of every type of migrated device in QEMU.

Currently the Patch we submitted bypasses QEMUFile. It does just issues the RDMA transfer for the memory that was dirtied and then continues along with the rest of the migration call path normally.

In an ideal world, we would prefer a hyrbid approach, something like:

Begin Migration Iteration Round:
1. stop VCPU
2. start iterative pass over memory
3. send control signals (if any) / device state to QEMUFile
4. When a dirty memory page is found, do:
     a) Instruct the QEMUFile to block
     b) Issue the RDMA transfer
     c) Instruct the QEMUFile to unblock
5. resume VCPU

This would require a "smarter" QEMUFile implementation that understands when to block and for how long.

Comments?

- Michael