Re: [Qemu-devel] [network performance question] only~2Gbpsthroughputbetweentwo linux guests which are running on the same hostvianetperf-tTCP_STREAM -m 1400, but xen can ac

[Zhang Haoyu]

>I see that RX checksumming is still off for you on virtio, this is
>likely what's contribution to the problem.
>
>Here's how it looks for me:
>ethtool -k eth1
>       Offload parameters for eth1:
>       rx-checksumming: on
>       tx-checksumming: on
>       scatter-gather: on
>       tcp-segmentation-offload: on
>       udp-fragmentation-offload: on
>       generic-segmentation-offload: on
>       generic-receive-offload: on
>       large-receive-offload: off
>
When I select centos-6.3 as guest os, the rx-checksuming is on, too.
After update the qemu from 1.4.0 to 2.0.0, the inter-vm throughput can achieve 
~5Gbps via netper -t TCP_STREAM -m 1400.
Here is ethtool -k eth1 on centos-6.3 guest,
ethtool -k eth1
Offload parameters for eth1:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: on
udp-fragmentation-offload: on
generic-segmentation-offload: on
generic-receive-offload: off
large-receive-offload: off

the only difference is gro, on for you, off for me,
I run 'ethtool -K eth1 gro on' on my guest, below error reported,
"Cannot set device GRO settings: Invalid argument"

>you don't supply kernel versions for host or guest kernels,
>so it's hard to judge what's going on exactly.
>
host: linux-3.10.27(directly download from kernel.org)
qemu: qemu-2.0.0(directly download from wiki.qemu.org/Download)
guest: centos-6.3(2.6.32-279.e16.x86_64), 2vcpu

>Bridge configuration also plays a huge role.
>Things like ebtables might affect performance as well,
>sometimes even if they are only loaded, not even enabled.
>
I will check it.

>Also, some old scheduler versions didn't put VMs on different
>CPUs aggressively enough, this resulted in conflicts
>when VMs compete for the same CPU.
I will check it.

>On numa systems, some older host kernels would split VM memory
>across NUMA nodes, this might lead to bad performance.
>
local first.

>On Sat, Jun 07, 2014 at 11:07:10AM +0800, Zhang Haoyu wrote:
>> After updating the qemu from 1.4 to 2.0, the inter-vm throughput can achieve 
>> ~5Gbps via netper -t TCP_STREAM -m 1400,
>> the performance gap(~2Gbps) between kvm and xen still exist.
>> 
>> Thanks,
>> Zhang Haoyu
>> 
>> ------------------                            
>> Zhang Haoyu
>> 2014-06-07
>> 
>> -----Original Message-----
>> From: Zhang Haoyu
>> Sent: 2014-06-07 09:27:16
>> To: Venkateswara Rao Nandigam; kvm; qemu-devel
>> Cc: Gleb Natapov; Paolo Bonzini; Michael S.Tsirkin; yewudi
>> Subject: Re: [network performance question] only ~2Gbpsthroughputbetweentwo 
>> linux guests which are running on the same host vianetperf-tTCP_STREAM -m 
>> 1400, but xen can ac
>> 
>> > Doesn't that answer your original question about performance gap!
 > Sorry, do you mean it's the offloadings cause the performance gap?
>> But even OFF the checksum-offload, tso, gro, .etc, the performance gap still 
>> exist,
>> if I understand correctly, kvm should have better performance than xen from 
>> the angle of implementation, because of shorter path, and fewer 
>> context-switches,
>> especially inter-vm communication.
>> 
>> And, why the performance gap is so big(~2G vs ~7G) when checksum-offload, 
>> tso, gro, .etc is on for both hypervisors?
>> Why the packes' size can be so big(65160) and stable on xen, but most 
>> packets' size is 1448, only a few part is ~65000 on kvm, when netperf -t 
>> TCP_STREAM -m 1400 ?
>> Does some TCP configurations have buissness with this? Or some virtio-net 
>> configurations?
>> 
>> Thanks,
>> Zhang Haoyu
>> 
>> -----Original Message-----
>> From: address@hidden [mailto:address@hidden On Behalf Of Zhang Haoyu
>> Sent: Friday, June 06, 2014 3:44 PM
>> To: Venkateswara Rao Nandigam; kvm; qemu-devel
>> Cc: Gleb Natapov; Paolo Bonzini; Michael S.Tsirkin; yewudi
>> Subject: Re: RE: [network performance question] only ~2Gbps 
>> throughputbetweentwo linux guests which are running on the same host via 
>> netperf-tTCP_STREAM -m 1400, but xen can ac
>> 
>> > >> Try Rx/Tx checksum offload on the all the concerned guests of  both 
>> > >> Hypervisors.
>> > >> 
>> > >Already ON on both hypervisors, so some other offloadings(e.g. tso, gso) 
>> > >can be supported.
>> > 
>> > Try Rx/Tx checksum offload "OFF" on the all the concerned guests of  
>> > both Hypervisors
>> > 
>> Off  Rx/Tx checksum offload on XEN guest, 1.6Gbps achived, tcpdump result on 
>> backend vif netdeivce shown that packets' size is 1448, stable.
>> Off Rx/Tx checksum offload on KVM guest, only ~1Gbps ahchived, tcpdump 
>> result on backend tap netdevice shown that packets' size is 1448, stable.
>> 
>> > And While launching the VM in KVM, in command line of virtio interface, 
>> > you can specify TSO, LRO, RxMergebuf. Try this instead of ethtool 
>> > interface.
>> The cuurent qemu command shown as below, and I will change the virtio-net 
>> configuration later as your advise, /usr/bin/kvm -id 8572667846472 -chardev 
>> socket,id=qmp,path=/var/run/qemu-server/8572667846472.qmp,server,nowait -mon 
>> chardev=qmp,mode=control -vnc :0,websocket,to=200,x509,password -pidfile 
>> /var/run/qemu-server/8572667846472.pid -daemonize -name centos6-196.5.5.72 
>> -smp sockets=1,cores=2 -cpu core2duo -nodefaults -vga cirrus -no-hpet -k 
>> en-us -boot menu=on,splash-time=8000 -m 4096 -usb -drive 
>> file=/sf/data/local/iso/vmtools/virtio_auto_install.iso,if=none,id=drive-ide0,media=cdrom,aio=threads,forecast=disable
>>  -device ide-cd,bus=ide.0,unit=0,drive=drive-ide0,id=ide0,bootindex=200 
>> -drive 
>> file=/sf/data/local/images/host-f8bc123b3e74/32f49b646d1e/centos6-196.5.5.72.vm/vm-disk-1.qcow2,if=none,id=drive-ide2,cache=directsync,aio=threads,forecast=disable
>>  -device ide-hd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=100 
>> -netdev type=tap,id=net0,ifname=857266784647200,sc
 r
> ip
>>  t=/sf/etc/kvm/vtp-bridge,vhost=on,vhostforce=on -device 
>> virtio-net-pci,mac=FE:FC:FE:95:EC:A7,netdev=net0,bus=p
>>  ci.0,addr=0x12,id=net0,bootindex=300 -rtc driftfix=slew,clock=rt -global 
>> kvm-pit.lost_tick_policy=discard -global PIIX4_PM.disable_s3=1 -global 
>> PIIX4_PM.disable_s4=1
>>  
>> -----Original Message-----
>> From: Zhang Haoyu [mailto:address@hidden
>> Sent: Friday, June 06, 2014 1:26 PM
>> To: Venkateswara Rao Nandigam; kvm; qemu-devel
>> Cc: Gleb Natapov; Paolo Bonzini; Michael S.Tsirkin; yewudi
>> Subject: RE: [network performance question] only ~2Gbps throughput 
>> betweentwo linux guests which are running on the same host via netperf 
>> -tTCP_STREAM -m 1400, but xen can ac
>> 
>> Thanks for reply.
>> > >>> And, vhost enabled, tx zero-copy enabled, virtio TSO enabled on kvm.
>> > 
>> > Try lro "ON" on client side. This would require mergable Rx buffers to be 
>> > ON.
>> > 
>> current setttings for gro and lro,
>> generic-receive-offload: on
>> large-receive-offload: off [fixed]
>> 
>> > And  Xen netfront to KVM virtio are not apples to apples because of their 
>> > implementation details. 
>> > 
>> You are right, I just want to make network performance comparison between 
>> the two virtualization platform from the view of user.
>> 
>> > Try Rx/Tx checksum offload on the all the concerned guests of  both 
>> > Hypervisors.
>> > 
>> Already ON on both hypervisors, so some other offloadings(e.g. tso, gso) can 
>> be supported.
>> 
>> kvm virtio-net nic:
>> ethtool -k eth0
>> Features for eth0:
>> rx-checksumming: off [fixed]
>> tx-checksumming: on
>>     tx-checksum-ipv4: off [fixed]\
>>      tx-checksum-ip-generic: on
>>      tx-checksum-ipv6: off [fixed]
>>      tx-checksum-fcoe-crc: off [fixed]
>>      tx-checksum-sctp: off [fixed]
>> scatter-gather: on
>>     tx-scatter-gather: on
>>      scatter-gather-fraglist: on
>> tcp-segmentation-offload: on
>>     tx-tcp-segmentation: on
>>      tx-tcp-ecn-segmentation: on
>>      tx-tcp6-segmentation: on
>> udp-fragmentation-offload: on
>> generic-segmentation-offload: on
>> generic-receive-offload: on
>> large-receive-offload: off [fixed]
>> rx-vlan-offload: off [fixed]
>> tx-vlan-offload: off [fixed]
>> ntuple-filters: off [fixed]
>> receive-hashing: off [fixed]
>> highdma: on [fixed]
>> rx-vlan-filter: on [fixed]
>> vlan-challenged: off [fixed]
>> tx-lockless: off [fixed]
>> netns-local: off [fixed]
>> tx-gso-robust: off [fixed]
>> tx-fcoe-segmentation: off [fixed]
>> tx-gre-segmentation: off [fixed]
>> tx-udp_tnl-segmentation: off [fixed]
>> fcoe-mtu: off [fixed]
>> tx-nocache-copy: on
>> loopback: off [fixed]
>> rx-fcs: off [fixed]
>> rx-all: off [fixed]
>> tx-vlan-stag-hw-insert: off [fixed]
>> rx-vlan-stag-hw-parse: off [fixed]
>> rx-vlan-stag-filter: off [fixed]
>> 
>> xen netfront nic:
>> ethtool -k eth0
>> Offload features for eth0:
>> rx-checksumming: on
>> tx-checksumming: on
>> scatter-gather: on
>> tcp-segmentation-offload: on
>> udp-fragmentation-offload: off
>> generic-segmentation-offload: on
>> generic-receive-offload: off
>> large-receive-offload: off
>> 
>> <piece of tcpdump result on xen backend vif netdevice >
>> 15:46:41.279954 IP 196.6.6.72.53507 > 196.6.6.71.53622: Flags [.], seq 
>> 1193138968:1193204128, ack 1, win 115, options [nop,nop,TS val 102307210 ecr 
>> 102291188], length 65160
>> 15:46:41.279971 IP 196.6.6.72.53507 > 196.6.6.71.53622: Flags [.], seq 
>> 1193204128:1193269288, ack 1, win 115, options [nop,nop,TS val 102307210 ecr 
>> 102291188], length 65160
>> 15:46:41.279987 IP 196.6.6.72.53507 > 196.6.6.71.53622: Flags [.], seq 
>> 1193269288:1193334448, ack 1, win 115, options [nop,nop,TS val 102307210 ecr 
>> 102291188], length 65160
>> 15:46:41.280003 IP 196.6.6.72.53507 > 196.6.6.71.53622: Flags [.], seq 
>> 1193334448:1193399608, ack 1, win 115, options [nop,nop,TS val 102307210 ecr 
>> 102291188], length 65160
>> 15:46:41.280020 IP 196.6.6.72.53507 > 196.6.6.71.53622: Flags [.], seq 
>> 1193399608:1193464768, ack 1, win 115, options [nop,nop,TS val 102307210 ecr 
>> 102291188], length 65160
>> 15:46:41.280213 IP 196.6.6.72.53507 > 196.6.6.71.53622: Flags [.], seq 
>> 1193464768:1193529928, ack 1, win 115, options [nop,nop,TS val 102307211 ecr 
>> 102291189], length 65160
>> 15:46:41.280233 IP 196.6.6.72.53507 > 196.6.6.71.53622: Flags [.], seq 
>> 1193529928:1193595088, ack 1, win 115, options [nop,nop,TS val 102307211 ecr 
>> 102291189], length 65160
>> 15:46:41.280250 IP 196.6.6.72.53507 > 196.6.6.71.53622: Flags [.], seq 
>> 1193595088:1193660248, ack 1, win 115, options [nop,nop,TS val 102307211 ecr 
>> 102291189], length 65160
>> 15:46:41.280239 IP 196.6.6.71.53622 > 196.6.6.72.53507: Flags [.], ack 
>> 1193138968, win 22399, options [nop,nop,TS val 102291190 ecr 102307210], 
>> length 0
>> 15:46:41.280267 IP 196.6.6.72.53507 > 196.6.6.71.53622: Flags [.], seq 
>> 1193660248:1193725408, ack 1, win 115, options [nop,nop,TS val 102307211 ecr 
>> 102291189], length 65160
>> 15:46:41.280284 IP 196.6.6.72.53507 > 196.6.6.71.53622: Flags [.], seq 
>> 1193725408:1193790568, ack 1, win 115, options [nop,nop,TS val 102307211 ecr 
>> 102291189], length 65160
>> 
>> Packets' size is very stable, 65160 Bytes.
>> 
>> Thanks,
>> Zhang Haoyu
>> 
>> -----Original Message-----
>> From: address@hidden [mailto:address@hidden On Behalf Of Zhang Haoyu
>> Sent: Friday, June 06, 2014 9:01 AM
>> To: kvm; qemu-devel
>> Cc: Gleb Natapov; Paolo Bonzini; Michael S.Tsirkin; yewudi
>> Subject: [network performance question] only ~2Gbps throughput between two 
>> linux guests which are running on the same host via netperf -t TCP_STREAM -m 
>> 1400, but xen can achieve ~7Gbps
>> 
>> Hi, all
>> 
>> I ran two linux guest on the same kvm host, then start the netserver on one 
>> vm, start netperf on the other one,  netperf command and test result shown 
>> as below, netperf -H 196.5.5.71 -t TCP_STREAM -l 60 -- -m 1400 -M 1400 
>> MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 196.5.5.71 
>> () port 0 AF_INET : nodelay
>> Recv   Send    Send                          
>> Socket Socket  Message  Elapsed              
>> Size   Size    Size     Time     Throughput  
>> bytes  bytes   bytes    secs.    10^6bits/sec  
>> 
>>  87380  16384   1400    60.01    2355.45   
>> 
>> but I ran two linux guest on the same xen hypervisor, ~7Gbps throughput 
>> achived, netperf command and test result shown as below, netperf -H 
>> 196.5.5.71 -t TCP_STREAM -l 60 -- -m 1400 -M 1400 MIGRATED TCP STREAM TEST 
>> from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 196.5.5.71 () port 0 AF_INET
>> Recv   Send    Send                          
>> Socket Socket  Message  Elapsed              
>> Size   Size    Size     Time     Throughput  
>> bytes  bytes   bytes    secs.    10^6bits/sec  
>> 
>>  87380  16384   1400    60.01    2349.82
>> 
>> many times test performed, the result is similar as above.
>> 
>> When I tcpdump backend tap netdevice, found that most packets' size is 
>> 1448bytes on kvm, and few packets are ~60000Bytes, but  I tcpdump backend 
>> vif netdevice, found that  most packets' size is >60000bytes on xen.
>> Test result of netperf -t TCP_STREAM -m 64 is similar, more larger packets 
>> on xen than kvm. 
>> 
>> And, vhost enabled, tx zero-copy enabled, virtio TSO enabled on kvm.
>> 
>> Any ideas?
>> 
>> Thanks,
>> Zhang Haoyu
>> 
>> <environment>
>> guest: linux-3.10, or centos-6.3
>> host: linux-3.10(kvm), xenserver-6.2
>> 
>> <test tool>
>> netperf-2.6
>> 
>> <test result details>
>> 1. KVM
>> netperf -H 196.5.5.71 -t TCP_STREAM -l 60 -- -m 65000 -M 65000 MIGRATED TCP 
>> STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 196.5.5.71 () port 0 
>> AF_INET
>> Recv   Send    Send                          
>> Socket Socket  Message  Elapsed              
>> Size   Size    Size     Time     Throughput  
>> bytes  bytes   bytes    secs.    10^6bits/sec  
>> 
>>  87380  16384  65000    60.00    13743.44 
>> 
>> netperf -H 196.5.5.71 -t TCP_STREAM -l 60 -- -m 1400 -M 1400 MIGRATED TCP 
>> STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 196.5.5.71 () port 0 
>> AF_INET
>> Recv   Send    Send                          
>> Socket Socket  Message  Elapsed              
>> Size   Size    Size     Time     Throughput  
>> bytes  bytes   bytes    secs.    10^6bits/sec  
>> 
>>  87380  16384   1400    60.01    2349.82   
>> 
>> netperf -H 196.5.5.71 -t TCP_STREAM -l 60 -- -m 64 -M 64 MIGRATED TCP STREAM 
>> TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 196.5.5.71 () port 0 AF_INET
>> Recv   Send    Send                          
>> Socket Socket  Message  Elapsed              
>> Size   Size    Size     Time     Throughput  
>> bytes  bytes   bytes    secs.    10^6bits/sec  
>> 
>>  87380  16384     64    60.01     294.70  
>> 
>> 2. XEN
>> netperf -H 196.6.6.71 -t TCP_STREAM -l 60 -- -m 65000 -M 65000 MIGRATED TCP 
>> STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 196.6.6.71 () port 0 
>> AF_INET
>> Recv   Send    Send                          
>> Socket Socket  Message  Elapsed              
>> Size   Size    Size     Time     Throughput  
>> bytes  bytes   bytes    secs.    10^6bits/sec  
>> 
>>  87380  16384  65000    60.01    6327.46 
>> 
>> netperf -H 196.6.6.71 -t TCP_STREAM -l 60 -- -m 1400 -M 1400 MIGRATED TCP 
>> STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 196.6.6.71 () port 0 
>> AF_INET
>> Recv   Send    Send                          
>> Socket Socket  Message  Elapsed              
>> Size   Size    Size     Time     Throughput  
>> bytes  bytes   bytes    secs.    10^6bits/sec  
>> 
>>  87380  16384   1400    60.00    7250.63  
>> 
>> netperf -H 196.6.6.71 -t TCP_STREAM -l 60 -- -m 64 -M 64 MIGRATED TCP STREAM 
>> TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 196.6.6.71 () port 0 AF_INET
>> Recv   Send    Send                          
>> Socket Socket  Message  Elapsed              
>> Size   Size    Size     Time     Throughput  
>> bytes  bytes   bytes    secs.    10^6bits/sec  
>> 
>>  87380  16384     64    60.00     668.25  
>> 
>> <piece of tcpdump result on backend tap netdevice>
>> 11:26:28.907130 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
>> 5421192:5422640, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
>> length 1448
>> 11:26:28.907146 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
>> 5422640:5424088, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
>> length 1448
>> 11:26:28.907161 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 5419328, win 24576, options [nop,nop,TS val 497985 ecr 497408], length 0
>> 11:26:28.907164 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
>> 5424088:5425536, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
>> length 1448
>> 11:26:28.907183 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
>> 5426984:5428432, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
>> length 1448
>> 11:26:28.907203 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 5421192, win 24576, options [nop,nop,TS val 497985 ecr 497408], length 0
>> 11:26:28.907217 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
>> 5428432:5429880, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
>> length 1448
>> 11:26:28.907225 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
>> 5429880:5431328, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
>> length 1448
>> 11:26:28.907238 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
>> 5431328:5432776, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
>> length 1448
>> 11:26:28.907256 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
>> 5432776:5434224, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
>> length 1448
>> 11:26:28.907268 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 5426984, win 24576, options [nop,nop,TS val 497985 ecr 497408], length 0
>> 11:26:28.907294 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [P.], seq 
>> 5434224:5434692, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
>> length 468
>> 11:26:28.907311 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 5429880, win 24576, options [nop,nop,TS val 497985 ecr 497408], length 0
>> 11:26:28.907319 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 5432776, win 24576, options [nop,nop,TS val 497985 ecr 497408], length 0
>> 11:26:28.907345 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
>> 5434692:5436140, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
>> length 1448
>> 11:26:28.907363 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
>> 5436140:5437588, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
>> length 1448
>> 11:26:28.907373 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 5434692, win 24576, options [nop,nop,TS val 497985 ecr 497408], length 0
>> 11:26:28.907385 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
>> 5437588:5439036, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
>> length 1448
 > 11:26:28.907411 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
 > 5439036:5440484, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
 > length 1448
>> 11:26:28.907427 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [P.], seq 
>> 5440484:5440692, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
>> length 208
>> 11:26:28.907427 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 5437588, win 24576, options [nop,nop,TS val 497985 ecr 497408], length 0
>> 11:26:28.907445 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
>> 5440692:5442140, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
>> length 1448
>> 11:26:28.907453 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
>> 5442140:5443588, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
>> length 1448
>> 11:26:28.907468 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
>> 5443588:5445036, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
>> length 1448
>> 11:26:28.907494 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
>> 5445036:5446484, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
>> length 1448
>> 11:26:28.907496 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 5440484, win 24576, options [nop,nop,TS val 497985 ecr 497408], length 0
>> 11:26:28.907506 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
>> 5446484:5447932, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
>> length 1448
>> 11:26:28.907539 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
>> 5447932:5449380, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
>> length 1448
>> 11:26:28.907551 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 5445036, win 24576, options [nop,nop,TS val 497985 ecr 497408], length 0
>> 11:26:28.907555 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
>> 5449380:5450828, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
>> length 1448
>> 11:26:28.907566 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [P.], seq 
>> 5450828:5451192, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
>> length 364
>> 11:26:28.907605 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
>> 5451192:5452640, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
>> length 1448
>> 11:26:28.907623 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
>> 5452640:5454088, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
>> length 1448
>> 11:26:28.907631 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
>> 5454088:5455536, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
>> length 1448
>> 11:26:28.907646 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
>> 5455536:5456984, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
>> length 1448
>> 11:26:28.907662 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
>> 5456984:5458432, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
>> length 1448
>> 11:26:28.907670 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
>> 5458432:5459880, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
>> length 1448
>> 11:26:28.907690 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
>> 5459880:5461328, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
>> length 1448
>> 11:26:28.907699 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
>> 5461328:5462776, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
>> length 1448
>> 11:26:28.907704 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 5450828, win 24576, options [nop,nop,TS val 497985 ecr 497408], length 0
>> 11:26:28.907708 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
>> 5462776:5464224, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
>> length 1448
>> 11:26:28.907721 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
>> 5464224:5465672, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
>> length 1448
>> 11:26:28.907739 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
>> 5465672:5467120, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
>> length 1448
>> 11:26:28.907764 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
>> 5468568:5470016, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
>> length 1448
>> 11:26:28.907793 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [P.], seq 
>> 5470016:5470692, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
>> length 676
>> 11:26:28.907796 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 5455536, win 24576, options [nop,nop,TS val 497985 ecr 497408], length 0
>> 11:26:28.937289 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
>> 7273240:7274688, ack 1, win 115, options [nop,nop,TS val 497416 ecr 497988], 
>> length 1448
>> 11:26:28.941798 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 6259640, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0
>> 11:26:28.941815 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 6262536, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0
>> 11:26:28.941820 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 6265432, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0
>> 11:26:28.941825 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 6268328, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0
>> 11:26:28.941829 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 6271224, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0
>> 11:26:28.941834 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 6274120, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0
>> 11:26:28.941839 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 6277016, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0
>> 11:26:28.941843 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 6279912, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0
>> 11:26:28.941848 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 6282808, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0
>> 11:26:28.941852 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 6285704, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0
>> 11:26:28.941856 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 6288600, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0
>> 11:26:28.941873 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 6291496, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0
>> 11:26:28.941888 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 6294392, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0
>> 11:26:28.941893 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 6297288, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0
>> 11:26:28.941904 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 6300184, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0
>> 11:26:28.941909 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 6303080, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0
>> 11:26:28.941919 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 6305976, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0
>> 11:26:28.941924 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 6308872, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0
>> 11:26:28.941934 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 6311768, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0
>> 11:26:28.941939 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 6314664, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0
>> 11:26:28.941950 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 6317560, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0
>> 11:26:28.941955 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 6320456, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0
>> 11:26:28.941961 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 6323352, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0
>> 11:26:28.941966 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 6326248, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0
>> 11:26:28.941967 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
>> 7274688:7276136, ack 1, win 115, options [nop,nop,TS val 497417 ecr 497993], 
>> length 1448
>> 11:26:28.941970 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 6329144, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0
>> 11:26:28.941975 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 6332040, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0
>> 11:26:28.941980 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 6334936, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0
>> 11:26:28.941985 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 6337832, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0
>> 11:26:28.941991 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 6342176, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0
>> 11:26:28.941995 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 6345072, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0
>> 11:26:28.942001 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 6362448, win 24441, options [nop,nop,TS val 497993 ecr 497411], length 0
>> 11:26:28.942118 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
>> 7276136:7338400, ack 1, win 115, options [nop,nop,TS val 497417 ecr 497993], 
>> length 62264
>> 11:26:28.942220 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 6579648, win 22745, options [nop,nop,TS val 497993 ecr 497411], length 0
>> 11:26:28.942456 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
>> 7338400:7403560, ack 1, win 115, options [nop,nop,TS val 497417 ecr 497993], 
>> length 65160
>> 11:26:28.942501 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
>> 7403560:7468720, ack 1, win 115, options [nop,nop,TS val 497417 ecr 497993], 
>> length 65160
>> 11:26:28.942547 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
>> 7468720:7533880, ack 1, win 115, options [nop,nop,TS val 497417 ecr 497993], 
>> length 65160
>> 11:26:28.942689 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 6640464, win 22964, options [nop,nop,TS val 497994 ecr 497412], length 0
>> 11:26:28.942701 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 6720104, win 23145, options [nop,nop,TS val 497994 ecr 497412], length 0
>> 11:26:28.942726 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 6743272, win 23833, options [nop,nop,TS val 497994 ecr 497412], length 0
>> 11:26:28.942864 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
>> 7533880:7599040, ack 1, win 115, options [nop,nop,TS val 497417 ecr 497994], 
>> length 65160
>> 11:26:28.942886 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 6811328, win 24159, options [nop,nop,TS val 497994 ecr 497412], length 0
>> 11:26:28.942918 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
>> 7599040:7664200, ack 1, win 115, options [nop,nop,TS val 497417 ecr 497994], 
>> length 65160
>> 11:26:28.942971 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
>> 7664200:7729360, ack 1, win 115, options [nop,nop,TS val 497417 ecr 497994], 
>> length 65160
>> 11:26:28.943019 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
>> 7729360:7794520, ack 1, win 115, options [nop,nop,TS val 497417 ecr 497994], 
>> length 65160
>> 11:26:28.943041 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 6882280, win 24268, options [nop,nop,TS val 497994 ecr 497412], length 0
>> 11:26:28.943187 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
>> 7794520:7859680, ack 1, win 115, options [nop,nop,TS val 497417 ecr 497994], 
>> length 65160
>> 11:26:28.943404 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 6928616, win 24576, options [nop,nop,TS val 497994 ecr 497412], length 0
>> 11:26:28.943415 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 6935856, win 24576, options [nop,nop,TS val 497994 ecr 497412], length 0
>> 11:26:28.943420 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 6938752, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0
>> 11:26:28.943427 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 6941648, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0
>> 11:26:28.943433 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 6944544, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0
>> 11:26:28.943438 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 6947440, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0
>> 11:26:28.943444 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 6950336, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0
>> 11:26:28.943452 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 6953232, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0
>> 11:26:28.943458 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 6956128, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0
>> 11:26:28.943463 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 6959024, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0
>> 11:26:28.943468 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 6961920, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0
>> 11:26:28.943473 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 6964816, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0
>> 11:26:28.943479 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 6969160, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0
>> 11:26:28.943487 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 6972056, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0
>> 11:26:28.943493 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 6974952, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0
>> 11:26:28.943500 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 6977848, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0
>> 11:26:28.943505 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 6980744, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0
>> 11:26:28.943511 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 6983640, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0
>> 11:26:28.943519 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 6986536, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0
>> 11:26:28.943525 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 6989432, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0
>> 11:26:28.943533 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 6992328, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0
>> 11:26:28.943542 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 6995224, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0
>> 11:26:28.943546 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 6998120, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0
>> 11:26:28.943550 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 7001016, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0
>> 11:26:28.943554 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 7003912, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0
>> 11:26:28.943558 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 7006808, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0
>> 11:26:28.943562 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 7011152, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0
>> 11:26:28.943566 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 7014048, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0
>> 11:26:28.943570 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
>> 7859680:7924840, ack 1, win 115, options [nop,nop,TS val 497417 ecr 497994], 
>> length 65160
>> 11:26:28.943607 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 7077760, win 24079, options [nop,nop,TS val 497994 ecr 497413], length 0
>> 11:26:28.943673 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
>> 7924840:7990000, ack 1, win 115, options [nop,nop,TS val 497417 ecr 497994], 
>> length 65160
>> 11:26:28.943711 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 7096584, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0
>> 11:26:28.943731 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 7099480, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0
>> 11:26:28.943750 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 7102376, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0
>> 11:26:28.943777 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
>> 7990000:8055160, ack 1, win 115, options [nop,nop,TS val 497417 ecr 497994], 
>> length 65160
>> 11:26:28.943822 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
>> 7177672, win 23988, options [nop,nop,TS val 497994 ecr 497413], length 0
>> 11:26:28.943833 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
>> 8055160:8115976, ack 1, win 115, options [nop,nop,TS val 497417 ecr 497994], 
>> length 60816
>> 11:26:28.943954 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
>> 8115976:8120320, ack 1, win 115, options [nop,nop,TS val 497417 ecr 497994], 
>> length 4344