[Qemu-devel] ping latency issue on passthrough guest

[Zhu Yijun]

Hi all,

  I booted two sr-iov guests using KVM-VFIO and pinged each other with
no-load one night. I found that most of the latency was little than 0.1ms,
but several icmp_seq greater than 10ms, even up to 1000ms;

  address@hidden:~# grep "time=[0-9][0-9]" outputfile
Mon May  7 23:05:12 201864 bytes from 192.168.0.200: icmp_seq=16703 ttl=64
time=123 ms
Mon May  7 23:26:28 201864 bytes from 192.168.0.200: icmp_seq=17979 ttl=64
time=123 ms
Mon May  7 23:45:08 201864 bytes from 192.168.0.200: icmp_seq=19099 ttl=64
time=127 ms
Tue May  8 00:19:21 201864 bytes from 192.168.0.200: icmp_seq=21152 ttl=64
time=132 ms
Tue May  8 00:20:16 201864 bytes from 192.168.0.200: icmp_seq=21207 ttl=64
time=131 ms
Tue May  8 00:20:58 201864 bytes from 192.168.0.200: icmp_seq=21249 ttl=64
time=131 ms
Tue May  8 00:24:36 201864 bytes from 192.168.0.200: icmp_seq=21467 ttl=64
time=131 ms
Tue May  8 03:17:53 201864 bytes from 192.168.0.200: icmp_seq=31864 ttl=64
time=102 ms
Tue May  8 04:51:35 201864 bytes from 192.168.0.200: icmp_seq=37486 ttl=64
time=134 ms
Tue May  8 04:51:39 201864 bytes from 192.168.0.200: icmp_seq=37490 ttl=64
time=134 ms
Tue May  8 04:55:31 201864 bytes from 192.168.0.200: icmp_seq=37722 ttl=64
time=138 ms
Tue May  8 04:59:23 201864 bytes from 192.168.0.200: icmp_seq=37954 ttl=64
time=138 ms
Tue May  8 05:15:05 201864 bytes from 192.168.0.200: icmp_seq=38896 ttl=64
time=76.4 ms
Tue May  8 05:19:05 201864 bytes from 192.168.0.200: icmp_seq=39136 ttl=64
time=146 ms
Tue May  8 07:07:40 201864 bytes from 192.168.0.200: icmp_seq=45651 ttl=64
time=10.7 ms
Tue May  8 07:12:42 201864 bytes from 192.168.0.200: icmp_seq=45953 ttl=64
time=30.7 ms
Tue May  8 07:17:40 201864 bytes from 192.168.0.200: icmp_seq=46251 ttl=64
time=166 ms
Tue May  8 08:03:33 201864 bytes from 192.168.0.200: icmp_seq=49004 ttl=64
time=14.6 ms
Tue May  8 08:13:34 201864 bytes from 192.168.0.200: icmp_seq=49605 ttl=64
time=174 ms
Tue May  8 08:13:36 201864 bytes from 192.168.0.200: icmp_seq=49607 ttl=64
time=174 ms
Tue May  8 08:42:38 201864 bytes from 192.168.0.200: icmp_seq=51349 ttl=64
time=84.0 ms
Tue May  8 08:51:50 201864 bytes from 192.168.0.200: icmp_seq=51901 ttl=64
time=92.0 ms
Tue May  8 09:34:06 201864 bytes from 192.168.0.200: icmp_seq=54437 ttl=64
time=283 ms

   VF used by these two guest from same port(eth0) of intel X710 Ethernet
controller. By contrast, I selected another two VFs and set them to
separate network namespace, this issue did not exits, all the latency
litter than 0.2ms.

   Advised by other guys, I disabled the BIOS C-State, set cpu power to
"performance", add kernel parameter "idle=poll, pcie_aspm=off", but it
makes no sense.

   I think it may be not the hardware issue, but may relate to KVM
hypervisor or guest kernel. As a result, I reported here, any advice and
suggestions will be greatly appreciated.

   The test enviroment as below:

   CPU model: Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
   HostOS: debian_9.3, kernel_version: 4.9.0-3
   QEMU 2.8.1
   libvirt 2.4.0
   NIC: intel X710
   GuestOS: debian_9.1
   xml:
   <domain type='kvm'>
  <name>ping_test01</name>
  <uuid>98132486-7f49-45bf-a6d7-6ceaf71ff4be</uuid>
  <maxMemory slots='64' unit='KiB'>134217728</maxMemory>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <blkiotune>
    <device>
      <path>/dev/sdb</path>
      <read_iops_sec>1000</read_iops_sec>
      <write_iops_sec>1000</write_iops_sec>
      <read_bytes_sec>104857600</read_bytes_sec>
      <write_bytes_sec>104857600</write_bytes_sec>
    </device>
  </blkiotune>
  <vcpu placement='static' cpuset='5-31,33-55' current='2'>64</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='9'/>
    <vcpupin vcpu='1' cpuset='37'/>
  </cputune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <sysinfo type='smbios'>
    <system>
      <entry name='manufacturer'>OpenStack Foundation</entry>
      <entry name='product'>OpenStack Nova</entry>
      <entry name='version'>2013.2.2</entry>
    </system>
  </sysinfo>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.8'>hvm</type>
    <boot dev='hd'/>
    <smbios mode='sysinfo'/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='custom' match='exact'>
    <model fallback='forbid'>Broadwell</model>
    <vendor>Intel</vendor>
    <feature policy='require' name='vme'/>
    <feature policy='require' name='ds'/>
    <feature policy='require' name='acpi'/>
    <feature policy='require' name='ss'/>
    <feature policy='require' name='ht'/>
    <feature policy='require' name='tm'/>
    <feature policy='require' name='pbe'/>
    <feature policy='require' name='dtes64'/>
    <feature policy='require' name='ds_cpl'/>
    <feature policy='require' name='vmx'/>
    <feature policy='require' name='smx'/>
    <feature policy='require' name='est'/>
    <feature policy='require' name='tm2'/>
    <feature policy='require' name='xtpr'/>
    <feature policy='require' name='pdcm'/>
    <feature policy='require' name='dca'/>
    <feature policy='require' name='osxsave'/>
    <feature policy='require' name='f16c'/>
    <feature policy='require' name='rdrand'/>
    <feature policy='require' name='arat'/>
    <feature policy='require' name='tsc_adjust'/>
    <feature policy='require' name='xsaveopt'/>
    <feature policy='require' name='pdpe1gb'/>
    <feature policy='require' name='abm'/>
    <numa>
      <cell id='0' cpus='0-63' memory='4194304' unit='KiB'/>
    </numa>
  </cpu>
  <clock offset='utc'>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='rtc' tickpolicy='catchup'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <source file='/images/debian_8.6_pub_dhcp_sr_i4_0428155812.dsk'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04'
function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='piix3-uhci'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01'
function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
function='0x0'/>
    </controller>
    <interface type='hostdev' managed='yes'>
      <mac address='52:54:00:0f:3b:b3'/>
      <driver name='vfio'/>
      <source>
        <address type='pci' domain='0x0000' bus='0x06' slot='0x03'
function='0x4'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05'
function='0x0'/>
    </interface>
    <serial type='file'>
      <source path='/tmp/vm_console.log'/>
      <target port='0'/>
    </serial>
    <serial type='pty'>
      <target port='1'/>
    </serial>
    <console type='file'>
      <source path='/tmp/zyj_console.log'/>
      <target type='serial' port='0'/>
    </console>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' listen='10.194.8.178'
keymap='en-us'>
      <listen type='address' address='10.194.8.178'/>
    </graphics>
    <video>
      <model type='cirrus' vram='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06'
function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='none' model='none'/>
  <seclabel type='dynamic' model='dac' relabel='yes'/>
</domain>


  Thanks!

Zhu Yijun