Re: [Qemu-devel] [POC]colo-proxy in qemu

[Dr. David Alan Gilbert]

* Tkid (address@hidden) wrote:
> Hi,all
> 
> We are planning to reimplement colo proxy in userspace (Here is in qemu) to
> cache and compare net packets.This module is one of the important components
> of COLO project and now it is still in early stage, so any comments and
> feedback are warmly welcomed,thanks in advance.
> 
> ## Background
> COLO FT/HA (COarse-grain LOck-stepping Virtual Machines for Non-stop
> Service)
> project is a high availability solution. Both Primary VM (PVM) and Secondary
> VM
> (SVM) run in parallel. They receive the same request from client, and
> generate
> responses in parallel too. If the response packets from PVM and SVM are
> identical, they are released immediately. Otherwise, a VM checkpoint (on
> demand)
> is conducted.
> Paper:
> http://www.socc2013.org/home/program/a3-dong.pdf?attredirects=0
> COLO on Xen:
> http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping
> COLO on Qemu/KVM:
> http://wiki.qemu.org/Features/COLO
> 
> By the needs of capturing response packets from PVM and SVM and finding out
> whether they are identical, we introduce a new module to qemu networking
> called
> colo-proxy.
> 
> This document describes the design of the colo-proxy module
> 
> ## Glossary
>   PVM - Primary VM, which provides services to clients.
>   SVM - Secondary VM, a hot standby and replication of PVM.
>   PN - Primary Node, the host which PVM runs on
>   SN - Secondary Node, the host which SVM runs on
> 
> ## Our Idea ##
> 
> COLO-Proxy
> COLO-Proxy is a part of COLO,based on qemu net filter and it's a plugin for
> qemu net filter.the function keep SVM connect normal to PVM and compare
> PVM's packets to SVM's packets.if difference,notify COLO do checkpoint.
> 
> == Workflow ==
> 
> 
> +--+                                      +--+
> |PN|                                      |SN|
> +-----------------------+                 +-----------------------+
> | +-------------------+ |                 | +-------------------+ |
> | |                   | |                 | |                   | |
> | |        PVM        | |                 | |        SVM        | |
> | |                   | |                 | |                   | |
> | +--+-^--------------+ |                 | +-------------^----++ |
> |    | |                |                 |               |    |  |
> |    | | +------------+ |                 | +-----------+ |    |  |
> |    | | |    COLO    | |    (socket)     | |    COLO   | |    |  |
> |    | | | CheckPoint +---------------------> CheckPoint| |    |  |
> |    | | |            | |      (6)        | |           | |    |  |
> |    | | +-----^------+ |                 | +-----------+ |    |  |
> |    | |   (5) |        |                 |               |    |  |
> |    | |       |        |                 |               |    |  |
> | +--v-+--------------+ | Forward(socket) | +-------------+----v+ |
> | |COLO Proxy  |      +-------+(1)+--------->seq&ack adjust(2)| | |
> | |      +-----+------+ |                 | +-----------------+ | |
> | |      | Compare(4) <-------+(3)+---------+     COLO Proxy    | |
> | +-------------------+ | Forward(socket) | +-------------------+ |
> ++Qemu+-----------------+                 ++Qemu+-----------------+
>            | ^
>            | |
>            | |
>   +--------v-+--------+
>   |                   |
>   |      Client       |
>   |                   |
>   +-------------------+
> 
> 
> (1)When PN receive client packets,PN COLO-Proxy copy and forward packets to
> SN COLO-Proxy.
> (2)SN COLO-Proxy record PVM's packet inital seq & adjust client's ack,send
> adjusted packets to SVM
> (3)SN Qemu COLO-Proxy recieve SVM's packets and forward to PN Qemu
> COLO-Proxy.

What protocol are you using for the data carried over the Forward(socket)?
I'm just wondering if there's an existing layer2 tunneling protocol that
it would be best to use.

> (4)PN Qemu COLO-Proxy enqueue SVM's packets and enqueue PVM's packets,then
> compare PVM's packets data with SVM's packets data. If packets is different,
> compare
> module notify COLO CheckPoint module to do a checkpoint then send PVM's
> packets to
> client and drop SVM's packets, otherwise, just send PVM's packets to client
> and
> drop SVM's packets.
> (5)notify COLO-Checkpoint module checkpoint is needed
> (6)Do COLO-Checkpoint
> 
> ### QEMU space TCP/IP stack(Based on SLIRP) ###
> We need a QEMU space TCP/IP stack to help us to analysis packet. After
> looking
> into QEMU, we found that SLIRP
> 
> http://wiki.qemu.org/Documentation/Networking#User_Networking_.28SLIRP.29
> 
> is a good choice for us. SLIRP proivdes a full TCP/IP stack within QEMU, it
> can
> help use to handle the packet written to/read from backend(tap) device which
> is
> just like a link layer(L2) packet.

I still think SLIRP might be painful; but it might be an easy one to start
with.

> ### Packet enqueue and compare ###
> Together with QEMU space TCP/IP stack, we enqueue all packets sent by PVM
> and
> SVM on Primary QEMU, and then compare the packet payload for each
> connection.

Dave

--
Dr. David Alan Gilbert / address@hidden / Manchester, UK