Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter

[Wen Congyang]

On 01/20/2016 02:54 PM, Jason Wang wrote:
> 
> 
> On 01/20/2016 11:29 AM, Zhang Chen wrote:
>>
>>> Sure.
>>>
>>> Two main comments/suggestions:
>>>
>>> - TCP analysis is missed in current version, maybe you point a git tree
>>> (or another version of RFC) to me for a better understanding of the
>>> design. (Just a skeleton for TCP should be sufficient to discuss).
>>> - I prefer to make the code as reusable as possible. So it's better to
>>> split/decouple the reusable parts from the codes. So a vague idea is:
>>>
>>> 1) Decouple the packet comparing from the netfilter. You've achieved
>>> this 99% since the work has been done in a thread. Just let the thread
>>> poll sockets directly, then the comparing have the possibility to be
>>> reused by other kinds of dataplane.
>>> 2) Implement traffic mirror/redirector as filter.
>>> 3) Implement TCP seq rewriting as a filter.
>>>
>>> Then, in primary node, you need just a traffic mirror, which did:
>>> - mirror ingress traffic to secondary node
>>> - mirror outgress traffic to packet comparing thread
>>>
>>> And in secondadry node, you need two filters:
>>> - A TCP seq rewriter which adjust tcp sequence number.
>>> - A traffic redirector which redirect packet from a socket as ingress
>>> traffic, and redirect outgress traffic to the socket which could be
>>> polled by remote packet comparing thread.
>>>   Thoughts?
>>>
>>> Thanks
>>>
>>>> Thanks
>>>> zhangchen
>>>
>>
>>
>> Hi, Jason.
>> We consider your suggestion to split/decouple
>> the reusable parts from the codes.
>> Due to filter plugin are traversed one by one in order
>> we will split colo-proxy to three filters in each side.
>>
>> But in this plan,primary and secondary both have socket
>> server,startup is a problem.
> 
> I believe this issue could be solved by reusing socket chardev.
> 
>>
>>
>>  Primary qemu                                                      
>> Secondary qemu
>> +----------------------------------------------------------+      
>> +-----------------------------------------------------------+
>> | +-----------------------------------------------------+  |       | 
>> +------------------------------------------------------+ |
>> | |                                                     |  |       | 
>> |                                                      | |
>> | |                        guest                        |  |       | 
>> |                        guest                         | |
>> | |                                                     |  |       | 
>> |                                                      | |
>> | +-----------^--------------+--------------------------+  |       | 
>> +---------------------+--------+-----------------------+ |
>> |             |              |                             |      
>> |                        ^        |                         |
>> |             |              |                             |      
>> |                        |        |                         |
>> |             +-------------------------------------------------+ 
>> |                        |        |                         |
>> |  netfilter  |              |                             |    |  |  
>> netfilter            |        |                         |
>> | +-----------------------------------------------------+  |    |  | 
>> +------------------------------------------------------+ |
>> | |           |              |     filter excute order  |  |    |  | 
>> |                     |        |  filter excute order  | |
>> | |           |              |    +-------------------> |  |    |  | 
>> |                     |        | +-------------------> | |
>> | |           |              |                          |  |    |  | 
>> |                     |        |   TCP                 | |
>> | | +---------+-+     +------v-----+    +----+ +-----+  |  |    |  | 
>> | +-----------+   +---+----+---v+rewriter+  +--------+ | |
>> | | |           |     |            |    |            |  |  |    |  | 
>> | |           |   |        |             |  |        | | |
>> | | |  mirror   |     |  redirect  +---->  compare   |  |  |   
>> +--------> mirror   +---> adjust |   adjust    +-->redirect| | |
>> | | |  client   |     |  server    |    |            |  |  |       | 
>> | |  server   |   | ack    |   seq       |  |client  | | |
>> | | |           |     |            |    |            |  |  |       | 
>> | |           |   |        |             |  |        | | |
>> | | +----^------+     +----^-------+    +-----+------+  |  |       | 
>> | +-----------+   +--------+-------------+  +----+---+ | |
>> | |      |     tx          |      rx          |     rx  |  |       | 
>> |            tx                        all       |  rx | |
>> | +-----------------------------------------------------+  |       | 
>> +------------------------------------------------------+ |
>> |        |                
>> +-------------------------------------------------------------------------------------------+
>>       
>> |
>> |        |                                    |            |      
>> |                                                           |
>> +----------------------------------------------------------+      
>> +-----------------------------------------------------------+
>>          |                                    |
>>          |guest receive                       |guest send
>>          |                                    |
>> +--------+------------------------------------v------------+
>> |                                                          |
>> |                                                          |
>> |                         tap                             
>> |                              NOTE: filter direction is rx/tx/all
>> |                                                         
>> |                              rx:receive packets sent to the netdev
>> |                                                         
>> |                              tx:receive packets sent by the netdev
>> +----------------------------------------------------------+
>>
>>
>>
> 
> I still like to decouple comparer from netfilter. It have two obvious
> advantages:
> 
> - make it can be reused by other dataplane (e.g vhost)
> - secondary redirector could redirect rx to comparer on primary node
> directly which simplify the design.
> 
>>
>>
>>
>>
>> guest recv packet route
>>
>> primary
>> tap --> mirror client filter
>> mirror client will send packet to guest,at the
>> same time, copy and forward packet to secondary
>> mirror server.
>>
>> secondary
>> mirror server filter --> TCP rewriter
>> if recv packet is TCP packet,we will adjust ack
>> and update TCP checksum, then send to secondary
>> guest. else directly send to guest.
>>
>>
>> guest send packet route
>>
>> primary
>> guest --> redirect server filter
>> redirect server filter recv primary guest packet
>> but do nothing, just pass to next filter.
>>
>> redirect server filter --> compare filter
>> compare filter recv primary guest packet then
>> waiting scondary redirect packet to compare it.
>> if packet same,send primary packet and clear secondary
>> packet, else send primary packet and do
>> checkpoint.
>>
>> secondary
>> guest --> TCP rewriter filter
>> if the packet is TCP packet,we will adjust seq
>> and update TCP checksum. then send it to
>> redirect client filter. else directly send to
>> redirect client filter.
>>
>> redirect client filter --> redirect server filter
>> forward packet to primary
>>
>>
>> In failover scene（primary is down）, the TCP rewriter will keep
>> servicing
>> for the TCP connection which is established after the last checkpoint。
>>
>>
>>
>> How about this plan?
> 
> Sounds good.
> 
> And there's indeed no need to differ client/server by reusing the socket
> chardev. E.g:
> 
> In primary node:

Thanks for your suggestion.

> 
> ...
> -chardev socket,id=comparer0,host=ip_primary,port=X,server,nowait
> -chardev socket,id=comparer1,host=ip_primary,port=Y,server,nowait
> -chardev socket,id=mirrorer0,host=ip_primary,port=Z,server,nowait
> -netdev tap,id=hn0
> -traffic-mirrorer netdev=hn0,id=t0,indev=comparer0,outdev=mirrorer0
> -colo-comparer primary_traffic=comparer0,secondary_traffic=comparer1
> ...
> 
> packet comparer compares the packets from two chardev: comparer0 and
> comparer1.
> traffic-mirrorer mirror tx to secondary node through chardev mirrorer0,
> and mirror rx to packet comparer through chardev comparer0.
> 
> In secondary node:
> 
> ...
> -chardev socket,id=redirector0,host=ip_primary,port=Y
> -chardev socket,id=redirector1,host=ip_primary,port=Z
> -netdev tap,id=hn0
> -traffic-redirector netdev=hn0,id,r0,indev=redirector0,outdev=redirector1
> -colo-rewriter netdev=hn0,id=c0
> ...
> 
> traffic-redirector redirect the rx traffic from primary node through
> redirector0 and redirect the tx traffic to promary node through redirector1.
> colo-rewriter rewrite seq number as a normal netfilter.

What are traffic-mirrorer and colo-comparer, traffic-redirector, colo-rewriter?
A netfilter driver?

If not, how to get the packet from the netdev, and send back the packet to
the netdev?

Thanks
Wen Congyang

> 
> 
> 
>>
>>
>>> .
>>>
>>
> 
> 
> 
> 
> .
>