Re: [Qemu-devel] [WIP] RDMA transport for COLO

[Hailiang Zhang]

On 2015/12/18 4:31, Dr. David Alan Gilbert wrote:
> Hi,
>    I've been playing with getting an RDMA setup for COLO and
> have something that mostly works, but it is very new and quite
> hacky; but I thought I'd share my work so far.

Nice work, i will look at it later. :)

> You can find it at:
> https://github.com/orbitfp7/qemu/commits/orbit-wp4-colo-dec
>
> What I've done is:
>    a) Wire up a partner TCP connection by the side of the RDMA
>      connection.
>    b) Use the TCP connection just for the responses from secondary->primary
>    c) Make the RDMA connection write to the colo-cache after
>       the first migrate
>    d) Make the RDMA connection notify the secondary when it
>      sends writes, so that the secondary can know that it needs
>      to flush those pages in the colo-cache.
>    e) Add a shutdown function and fix some other bugs
>
> I've had that working on both your current world (which is
> what that tree is based off) and your older COLO world
> from July (with a bit more hacking to make it take the newer
> patches).
>
> Looking at the speed:
>    a) The CPU load on the incoming thread is much lower - maybe
> only 10-11% instead of 30-40%.
>    b) The performance of guest code is a little slower (~10% slower?)
>      on RDMA rather than TCP (on both 10Gbps and 40Gbps links)
>      I've not worked out why yet. (My guess is it could be to do with
>      RDMA dynamic registration)
>
> Things I know I need to do:
>    1) Tidy it up - it's very messy!
>    2) Try and get rid of the TCP connection and use an RDMA
>       channel for the backwards connection
>    3) Make sure the shutdown really can cope with the other host
>       being dead.
>    4) It only deals with the dynamic registration mode of RDMA;
>       setting pin-all will probably break it.
>    5) Figure out why it's slower!
>    6) Test failover more.
>
> My work on this is part of the EU Orbit project
> ( http://www.orbitproject.eu/ )
>
> Dave
>
>
> --
> Dr. David Alan Gilbert / address@hidden / Manchester, UK
>
> .