Re: [Qemu-devel] [RFC PATCH RDMA support v2: 5/6] connection-setup code between client/server

[Michael S. Tsirkin]

I don't think people mind using RDMA specifically,
small amounts of data can be sent using SEND commands.
But it's cleaner not to use TCP for parts of data.


On Tue, Feb 19, 2013 at 01:03:59AM -0500, Michael R. Hines wrote:
> 
> This sounds great. My team was just discussing customizing
> QEMUFileOps to be more "rdma-friendly".
> 
> We weren't certain if it would be well received, so this is good to know.
> 
> I'll try to scrounge up a patch of some kind before sending out a v3
> and then do my best to get rid of the migrate_use_rdma() checks
> everywhere.......
> 
> Thanks for the suggestion,
> 
> - Michael
> 
> On 02/18/2013 05:52 AM, Paolo Bonzini wrote:
> >Il 14/02/2013 20:29, Michael R. Hines ha scritto:
> >>>Are you still using the tcp for transferring device state? If so you
> >>>can call the tcp functions from the migration rdma code as a first
> >>>step but I would prefer it to use RDMA too.
> >>This is the crux of the problem of using RDMA for migration: Currently
> >>all of the QEMU migration control logic and device state goes through
> >>the the QEMUFile implementation. RDMA, however is by nature a zero-copy
> >>solution and is incompatible with QEMUFile.
> >With the patches I sent yesterday, there is no more buffering involved
> >in migration.c.  All data goes straight from arch_init.c to a QEMUFile.
> >
> >QEMUFile still does some buffering, but this should change with other
> >patches that Orit is working on.
> >
> >>Using RDMA for transferring device state is not recommended: Setuping an
> >>RDMA requires registering the memory locations on both sides with the
> >>RDMA hardware. This is not ideal because this would require pinning the
> >>memory holding the device state and then issuing the RDMA transfer for
> >>*each* type of device - which would require changing the control path of
> >>every type of migrated device in QEMU.
> >Yes, this would not work well.  However, you can (I think) define a
> >QEMUFileOps implementation for RDMA that:
> >
> >1) registers the buffer of a QEMUFile with the RDMA hardware;
> >
> >2) in its get_buffer (receive) and put_buffer (send) callbacks, issues a
> >synchronous RDMA transfer;
> >
> >3) unregisters the buffer in the close callback.
> >
> >As a proof of concept, this would also work (though it would make no
> >sense) for transferring the RAM; in the end of course it would be used
> >only for the device state.
> >
> >It's not a problem to add more operations to QEMUFileOps or similar.
> >It's also not a problem to change the way buf is allocated, if you need
> >it to be page-aligned or something like that.
> >
> >It is much better than adding migration_use_rdma() everywhere.
> >
> >Paolo
>