Re: [Qemu-devel] [RFC PATCH RDMA support v2: 5/6] connection-setup code between client/server

[Michael R. Hines]

On 02/19/2013 11:39 AM, Orit Wasserman wrote:
> Do you have some results as to the performance cost on demand pinning has?
Yes, there's a single graph on the QEMU wiki that shows this:

http://wiki.qemu.org/Features/RDMALiveMigration

Pay attention to the last group of bars, labelled "A", "B", and "C".

"A" is called "Chunked Page Registration" (i.e. on-demand pinning in 
groups of about 1MB of the surrounding memory pages)
"B" pins everything in advance

On-demand has only a slightly larger cost, but not much.

> > > For the guest memory pages, sending the pages directly without QemuFile (that 
> > > does buffering) is better, I would suggest implementing an QemuRDMAFile for 
> > > this. It will have a new API for the memory pages (zero copy) so instead of 
> > > using qemu_put_buffer we will call qemu_rdma_buffer or it can reimplement 
> > > qemu_put_buffer (you need to add offset). As for device state which is sent in 
> > > the last phase and is small you can modify the current implementation. (well 
> > > Paolo sent patches that are changing this but I think buffering is still an 
> > > option) The current migration implementation copies the device state into a 
> > > buffer and than send the data from the buffer (QemuBufferedFile). You only need 
> > > to pin this buffer, and RDMA it after all the device state was written into it. 
> > > Regards, Orit
> > I like it .......... can you help me understand - how much different is this design than 
> > the "QEMUFileOps" design that Paulo suggested?
> >
> > Or it basically the same? ....reimplementing qemu_put_buffer/get_buffer() for 
> > RDMA purposes......
> Yes it is basically the same :).
> you will need the QEMUFileRDMA to store the rdma context (look at 
> QEMUFileSocket for example) and other specific rdma parameters.
> you will the rdma_file_ops (QEMUFileOps) to implement sending guest pages 
> directly and pinning the buffer that contain the device state.
> you won't need to change qemu_put_buffer for the device state but implement a 
> rdma_put_buffer ops that will pin the buffer and send it.
> As for the guest memory pages you have two options:
> update qemu_put_buffer/get_buffer
> add new QEMUFileOps just for rdma and call it instead of qemu_put_buffer for 
> guest pages.

Understood.... I have some code reading to do with these pointers, then. 
Will have to teach myself everything you just said =)