Re: [Qemu-devel] [PATCH RFC v3 19/27] COLO NIC: Implement colo nic device interface configure()

[Dr. David Alan Gilbert]

* zhanghailiang (address@hidden) wrote:
> On 2015/2/16 20:03, Dr. David Alan Gilbert wrote:
> >* zhanghailiang (address@hidden) wrote:
> >>Implement colo nic device interface configure()
> >>add a script to configure nic devices:
> >>${QEMU_SCRIPT_DIR}/colo-proxy-script.sh
> >
> >Do you have some more documentation of the new colo-proxy?  I've
> 
> Yes, gaofeng is writing it now...

Great.

> >been reading the kernel module source and I can see that it's
> >a nice idea to do the sequence number adjustment on the host,
> >that reduces the need to modify the guest kernel; I was trying to
> >figure out how you synchronise the master/slave idea of sequence numbers -
> >is that purely from the 'ack' that's duplicated back to the secondary?
> 
> Yes, you've got it :)
> 
> >If you were unlucky and the 'ack' packet was lost on the duplicated
> >link from the primary to secondary how would you recover?
> 
> The 'ack' packet will be consider to be lost, because the primary will not
> respond to this 'ack' packet until it got secondary's response,
> and client will resend it ('ack' packet).
> 
> >What about TCP connections setup before colo was activated?
> >
> 
> Actually, now, we only support activate colo before guest is startup (for 
> test procedure,
> '-S' is needed for qemu command line).

Consider this:
     1) Start primary
     2) Start secondary
     3) Start the colo pairing
     4) Primary fails
     5) Colo failover to secondary

Now we have only the old secondary running; we'd really like to get back to
having a pair of fault-tolerant hosts, so it would be good to be able to:

     6) Make the old secondary the new primary
     7) Add a new secondary
     8) Start colo-pairing to the new secondary

You could theoretically do this with colo-agent, but not with colo-proxy.

> >The other thought is that passing the 'sec_dev' as a module parameter
> >gives you an artificial limitation; it forces all of the pairs
> >to be between the same pair of hosts.   If the 'sec_dev' was a parameter
> >to the connection then you could have different slaves associated with
> >each guest on the primary host.
> >
> 
> Hmm, do you mean we should pass this 'sec_dev' as a parameter from qemu to 
> proxy module by
> maybe ioctl ?

Yes, ioctl or tc or whatever; and make it per-guest.

> Yes, it is ugly to pass this 'sec_dev' directly to module as parameter.
> We will consider this, thanks ;)

Thanks!

> >Dave
> >P.S. You probably need to clean the debug messages up in the kernel module!
> >
> 
> OK, will do that.

Thanks.

Dave

> 
> >>Signed-off-by: zhanghailiang <address@hidden>
> >>Signed-off-by: Gao feng <address@hidden>
> >>Signed-off-by: Li Zhijian <address@hidden>
> >>---
> >>  net/colo-nic.c               | 56 +++++++++++++++++++++++++++-
> >>  scripts/colo-proxy-script.sh | 88 
> >> ++++++++++++++++++++++++++++++++++++++++++++
> >>  2 files changed, 143 insertions(+), 1 deletion(-)
> >>  create mode 100755 scripts/colo-proxy-script.sh
> >>
> >>diff --git a/net/colo-nic.c b/net/colo-nic.c
> >>index 965af49..f8fc35d 100644
> >>--- a/net/colo-nic.c
> >>+++ b/net/colo-nic.c
> >>@@ -39,12 +39,66 @@ static bool colo_nic_support(NetClientState *nc)
> >>      return nc && nc->colo_script[0] && nc->colo_nicname[0];
> >>  }
> >>
> >>+static int launch_colo_script(char *argv[])
> >>+{
> >>+    int pid, status;
> >>+    char *script = argv[0];
> >>+
> >>+    /* try to launch network script */
> >>+    pid = fork();
> >>+    if (pid == 0) {
> >>+        execv(script, argv);
> >>+        _exit(1);
> >>+    } else if (pid > 0) {
> >>+        while (waitpid(pid, &status, 0) != pid) {
> >>+            /* loop */
> >>+        }
> >>+
> >>+        if (WIFEXITED(status) && WEXITSTATUS(status) == 0) {
> >>+            return 0;
> >>+        }
> >>+    }
> >>+    return -1;
> >>+}
> >>+
> >>+static int colo_nic_configure(NetClientState *nc,
> >>+            bool up, int side, int index)
> >>+{
> >>+    int i, argc = 6;
> >>+    char *argv[7], index_str[32];
> >>+    char **parg;
> >>+
> >>+    if (!nc && index <= 0) {
> >>+        error_report("Can not parse colo_script or colo_nicname");
> >>+        return -1;
> >>+    }
> >>+
> >>+    parg = argv;
> >>+    *parg++ = nc->colo_script;
> >>+    *parg++ = (char *)(side == COLO_SECONDARY_MODE ? "slave" : "master");
> >>+    *parg++ = (char *)(up ? "install" : "uninstall");
> >>+    *parg++ = nc->colo_nicname;
> >>+    *parg++ = nc->ifname;
> >>+    sprintf(index_str, "%d", index);
> >>+    *parg++ = index_str;
> >>+    *parg = NULL;
> >>+
> >>+    for (i = 0; i < argc; i++) {
> >>+        if (!argv[i][0]) {
> >>+            error_report("Can not get colo_script argument");
> >>+            return -1;
> >>+        }
> >>+    }
> >>+
> >>+    return launch_colo_script(argv);
> >>+}
> >>+
> >>  void colo_add_nic_devices(NetClientState *nc)
> >>  {
> >>      struct nic_device *nic = g_malloc0(sizeof(*nic));
> >>
> >>      nic->support_colo = colo_nic_support;
> >>-    nic->configure = NULL;
> >>+    nic->configure = colo_nic_configure;
> >>      /*
> >>       * TODO
> >>       * only support "-netdev tap,colo_scripte..."  options
> >>diff --git a/scripts/colo-proxy-script.sh b/scripts/colo-proxy-script.sh
> >>new file mode 100755
> >>index 0000000..c7aa53f
> >>--- /dev/null
> >>+++ b/scripts/colo-proxy-script.sh
> >>@@ -0,0 +1,88 @@
> >>+#!/bin/sh
> >>+#usage: ./colo-proxy-script.sh master/slave install/uninstall phy_if 
> >>virt_if index
> >>+#.e.g ./colo-proxy-script.sh master install eth2 tap0 1
> >>+
> >>+side=$1
> >>+action=$2
> >>+phy_if=$3
> >>+virt_if=$4
> >>+index=$5
> >>+br=br1
> >>+failover_br=br0
> >>+
> >>+script_usage()
> >>+{
> >>+    echo -n "usage: ./colo-proxy-script.sh master/slave "
> >>+    echo -e "install/uninstall phy_if virt_if index\n"
> >>+}
> >>+
> >>+master_install()
> >>+{
> >>+    tc qdisc add dev $virt_if root handle 1: prio
> >>+    tc filter add dev $virt_if parent 1: protocol ip prio 10 u32 match u32 
> >>0 0 flowid 1:2 action mirred egress mirror dev $phy_if
> >>+    tc filter add dev $virt_if parent 1: protocol arp prio 11 u32 match 
> >>u32 0 0 flowid 1:2 action mirred egress mirror dev $phy_if
> >>+    tc filter add dev $virt_if parent 1: protocol ipv6 prio 12 u32 match 
> >>u32 0 0 flowid 1:2 action mirred egress mirror dev $phy_if
> >>+
> >>+    modprobe nf_conntrack_ipv4
> >>+    modprobe xt_PMYCOLO sec_dev=$phy_if
> >>+
> >>+    /usr/local/sbin/iptables -t mangle -I PREROUTING -m physdev 
> >>--physdev-in $virt_if -j PMYCOLO --index $index
> >>+    /usr/local/sbin/ip6tables -t mangle -I PREROUTING -m physdev 
> >>--physdev-in $virt_if -j PMYCOLO --index $index
> >>+    /usr/local/sbin/arptables -I INPUT -i $phy_if -j MARK --set-mark $index
> >>+}
> >>+
> >>+master_uninstall()
> >>+{
> >>+    tc filter del dev $virt_if parent 1: protocol ip prio 10 u32 match u32 
> >>0 0 flowid 1:2 action mirred egress mirror dev $phy_if
> >>+    tc filter del dev $virt_if parent 1: protocol arp prio 11 u32 match 
> >>u32 0 0 flowid 1:2 action mirred egress mirror dev $phy_if
> >>+    tc filter del dev $virt_if parent 1: protocol ipv6 prio 12 u32 match 
> >>u32 0 0 flowid 1:2 action mirred egress mirror dev $phy_if
> >>+    tc qdisc del dev $virt_if root handle 1: prio
> >>+
> >>+    /usr/local/sbin/iptables -t mangle -F
> >>+    /usr/local/sbin/ip6tables -t mangle -F
> >>+    /usr/local/sbin/arptables -F
> >>+    rmmod xt_PMYCOLO
> >>+}
> >>+
> >>+slave_install()
> >>+{
> >>+    brctl addif $br $phy_if
> >>+    modprobe xt_SECCOLO
> >>+
> >>+    /usr/local/sbin/iptables -t mangle -I PREROUTING -m physdev 
> >>--physdev-in $virt_if -j SECCOLO --index $index
> >>+    /usr/local/sbin/ip6tables -t mangle -I PREROUTING -m physdev 
> >>--physdev-in $virt_if -j SECCOLO --index $index
> >>+}
> >>+
> >>+
> >>+slave_uninstall()
> >>+{
> >>+    brctl delif $br $phy_if
> >>+    brctl delif $br $virt_if
> >>+    brctl addif $failover_br $virt_if
> >>+
> >>+    /usr/local/sbin/iptables -t mangle -F
> >>+    /usr/local/sbin/ip6tables -t mangle -F
> >>+    rmmod xt_SECCOLO
> >>+}
> >>+
> >>+if [ $# -ne 5 ]; then
> >>+    script_usage
> >>+    exit 1
> >>+fi
> >>+
> >>+if [ "x$side" != "xmaster" ] && [ "x$side" != "xslave" ]; then
> >>+    script_usage
> >>+    exit 2
> >>+fi
> >>+
> >>+if [ "x$action" != "xinstall" ] && [ "x$action" != "xuninstall" ]; then
> >>+    script_usage
> >>+    exit 3
> >>+fi
> >>+
> >>+if [ $index -lt 0 ] || [ $index -gt 100 ]; then
> >>+    echo "index overflow"
> >>+    exit 4
> >>+fi
> >>+
> >>+${side}_${action}
> >>--
> >>1.7.12.4
> >>
> >>
> >--
> >Dr. David Alan Gilbert / address@hidden / Manchester, UK
> >
> >.
> >
> 
> 
--
Dr. David Alan Gilbert / address@hidden / Manchester, UK