[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Qemu-devel] virtio-net-pci possible limitation on AArch64
|
From: |
Andrew Jones |
|
Subject: |
Re: [Qemu-devel] virtio-net-pci possible limitation on AArch64 |
|
Date: |
Tue, 25 Jul 2017 08:47:49 +0200 |
|
User-agent: |
Mutt/1.6.0.1 (2016-04-01) |
On Tue, Jul 25, 2017 at 12:38:52AM +0000, Alexandru Avadanii wrote:
> Hi, Andrew,
> I tried rebuilding virtinst (1.3.2), but that made no difference.
>
> However, changing "model=virtio-net-pci" to simply "model=virtio" (and
> letting the default kick in via libvirt)
> made a whole lot of difference (see domain xml at the end).
Hmm, that might be something virt-install developers will want
to consider changing. I've added Cole to CC.
>
> The interfaces are now on separate buses, all 4 work as expected, and the
> virtio legacy disable is no longer needed.
>
> I am a bit curious what goes wrong with the original XML, but not that much
> to investigate it further.
My guess is IO resource exhaustion. You might be able to see
if that's the case by checking the guest's kernel logs. I'm
not sure, but I think the IO BAR must be allocated even for
virtio-1 mode nics if they're plugged into PCI slots vs.
PCIe ports.
Thanks,
drew
>
> Thank you,
> Alex
>
> New domain XML
> ---
> <domain type='kvm' id='38'>
> <name>cfg01</name>
> <uuid>e4489f63-34a3-47f7-88c6-f6fb60a2aac0</uuid>
> <memory unit='KiB'>4194304</memory>
> <currentMemory unit='KiB'>4194304</currentMemory>
> <vcpu placement='static'>6</vcpu>
> <resource>
> <partition>/machine</partition>
> </resource>
> <os>
> <type arch='aarch64' machine='virt-2.9'>hvm</type>
> <loader readonly='yes'
> type='pflash'>/usr/share/AAVMF/AAVMF_CODE.fd</loader>
> <nvram>/var/lib/libvirt/qemu/nvram/cfg01_VARS.fd</nvram>
> <boot dev='hd'/>
> </os>
> <features>
> <acpi/>
> <gic version='3'/>
> </features>
> <cpu mode='host-passthrough' check='none'/>
> <clock offset='utc'/>
> <on_poweroff>destroy</on_poweroff>
> <on_reboot>restart</on_reboot>
> <on_crash>restart</on_crash>
> <devices>
> <emulator>/usr/bin/kvm</emulator>
> <disk type='file' device='disk'>
> <driver name='qemu' type='qcow2' cache='none' io='native'/>
> <source
> file='/home/jenkins/alav/armband/upstream/fuel/mcp/scripts/images/mcp_cfg01.qcow2'/>
> <backingStore/>
> <target dev='vda' bus='virtio'/>
> <alias name='virtio-disk0'/>
> <address type='pci' domain='0x0000' bus='0x06' slot='0x00'
> function='0x0'/>
> </disk>
> <disk type='file' device='cdrom'>
> <driver name='qemu' type='raw'/>
> <source
> file='/home/jenkins/alav/armband/upstream/fuel/mcp/scripts/images/mcp_cfg01.iso'/>
> <backingStore/>
> <target dev='sda' bus='scsi'/>
> <readonly/>
> <alias name='scsi0-0-0-0'/>
> <address type='drive' controller='0' bus='0' target='0' unit='0'/>
> </disk>
> <controller type='scsi' index='0' model='virtio-scsi'>
> <alias name='scsi0'/>
> <address type='pci' domain='0x0000' bus='0x05' slot='0x00'
> function='0x0'/>
> </controller>
> <controller type='pci' index='0' model='pcie-root'>
> <alias name='pcie.0'/>
> </controller>
> <controller type='pci' index='1' model='pcie-root-port'>
> <model name='pcie-root-port'/>
> <target chassis='1' port='0x8'/>
> <alias name='pci.1'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x01'
> function='0x0' multifunction='on'/>
> </controller>
> <controller type='pci' index='2' model='pcie-root-port'>
> <model name='pcie-root-port'/>
> <target chassis='2' port='0x9'/>
> <alias name='pci.2'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x01'
> function='0x1'/>
> </controller>
> <controller type='pci' index='3' model='pcie-root-port'>
> <model name='pcie-root-port'/>
> <target chassis='3' port='0xa'/>
> <alias name='pci.3'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x01'
> function='0x2'/>
> </controller>
> <controller type='pci' index='4' model='pcie-root-port'>
> <model name='pcie-root-port'/>
> <target chassis='4' port='0xb'/>
> <alias name='pci.4'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x01'
> function='0x3'/>
> </controller>
> <controller type='pci' index='5' model='pcie-root-port'>
> <model name='pcie-root-port'/>
> <target chassis='5' port='0xc'/>
> <alias name='pci.5'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x01'
> function='0x4'/>
> </controller>
> <controller type='pci' index='6' model='pcie-root-port'>
> <model name='pcie-root-port'/>
> <target chassis='6' port='0xd'/>
> <alias name='pci.6'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x01'
> function='0x5'/>
> </controller>
> <controller type='pci' index='7' model='dmi-to-pci-bridge'>
> <model name='i82801b11-bridge'/>
> <alias name='pci.7'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
> function='0x0'/>
> </controller>
> <controller type='pci' index='8' model='pci-bridge'>
> <model name='pci-bridge'/>
> <target chassisNr='8'/>
> <alias name='pci.8'/>
> <address type='pci' domain='0x0000' bus='0x07' slot='0x00'
> function='0x0'/>
> </controller>
> <controller type='pci' index='9' model='pcie-root-port'>
> <model name='pcie-root-port'/>
> <target chassis='9' port='0xe'/>
> <alias name='pci.9'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x01'
> function='0x6'/>
> </controller>
> <interface type='network'>
> <mac address='52:54:00:2f:9e:f0'/>
> <source network='pxe' bridge='pxe'/>
> <target dev='vnet0'/>
> <model type='virtio'/>
> <alias name='net0'/>
> <address type='pci' domain='0x0000' bus='0x01' slot='0x00'
> function='0x0'/>
> </interface>
> <interface type='network'>
> <mac address='52:54:00:87:d5:81'/>
> <source network='mgmt' bridge='mgmt'/>
> <target dev='vnet1'/>
> <model type='virtio'/>
> <alias name='net1'/>
> <address type='pci' domain='0x0000' bus='0x02' slot='0x00'
> function='0x0'/>
> </interface>
> <interface type='network'>
> <mac address='52:54:00:2f:9a:e5'/>
> <source network='internal' bridge='internal'/>
> <target dev='vnet2'/>
> <model type='virtio'/>
> <alias name='net2'/>
> <address type='pci' domain='0x0000' bus='0x03' slot='0x00'
> function='0x0'/>
> </interface>
> <interface type='network'>
> <mac address='52:54:00:64:41:45'/>
> <source network='public' bridge='public'/>
> <target dev='vnet3'/>
> <model type='virtio'/>
> <alias name='net3'/>
> <address type='pci' domain='0x0000' bus='0x04' slot='0x00'
> function='0x0'/>
> </interface>
> <serial type='pty'>
> <source path='/dev/pts/0'/>
> <target port='0'/>
> <alias name='serial0'/>
> </serial>
> <console type='pty' tty='/dev/pts/0'>
> <source path='/dev/pts/0'/>
> <target type='serial' port='0'/>
> <alias name='serial0'/>
> </console>
> <graphics type='vnc' port='5900' autoport='yes' listen='127.0.0.1'>
> <listen type='address' address='127.0.0.1'/>
> </graphics>
> <video>
> <model type='vga' vram='16384' heads='1' primary='yes'/>
> <alias name='video0'/>
> <address type='pci' domain='0x0000' bus='0x08' slot='0x01'
> function='0x0'/>
> </video>
> </devices>
> <seclabel type='dynamic' model='apparmor' relabel='yes'>
> <label>libvirt-e4489f63-34a3-47f7-88c6-f6fb60a2aac0</label>
> <imagelabel>libvirt-e4489f63-34a3-47f7-88c6-f6fb60a2aac0</imagelabel>
> </seclabel>
> <seclabel type='dynamic' model='dac' relabel='yes'>
> <label>+113:+115</label>
> <imagelabel>+113:+115</imagelabel>
> </seclabel>
> </domain>
>
> > -----Original Message-----
> > From: Alexandru Avadanii
> > Sent: Monday, July 24, 2017 6:53 PM
> > To: 'Andrew Jones'
> > Cc: address@hidden; svc-armband
> > Subject: RE: [Qemu-devel] virtio-net-pci possible limitation on AArch64
> >
> > Hi, Andrew,
> > Thank you for looking at this.
> > Comments inline.
> >
> > > -----Original Message-----
> > > From: Andrew Jones [mailto:address@hidden
> > > Sent: Monday, July 24, 2017 6:02 PM
> > > To: Alexandru Avadanii
> > > Cc: address@hidden; svc-armband
> > > Subject: Re: [Qemu-devel] virtio-net-pci possible limitation on
> > > AArch64
> > >
> > > On Fri, Jul 21, 2017 at 06:37:26PM +0000, Alexandru Avadanii wrote:
> > > > Hi,
> > > > I ran into an interesting limitation (or possibly bug) while trying
> > > > to attach
> > > more than 3 (three) virtio-net-pci NICs to an AArch64 guest.
> > > > I created 4 network with virsh (see below for xml defitions) -
> > > > nothing fancy,
> > > only 2 of them use NAT.
> > > >
> > > > Attaching a virtio-net-pci NIC for each network works well, VMs can
> > > > be
> > > spawned, all devices are properly detected by the guest OS.
> > > > The first 3 guest interfaces work perfectly, however I can't get any
> > > > traffic
> > > on the 4th one (in my case the public one).
> > > >
> > > > QEMU version: 2.9.0
> > > > Libvirt: tested with both 1.3.4 and 3.5.0, although I don't think
> > > > libvirt is to blame here
> > >
> > > 1.3.4 is way too old for AArch64 guests - 3.0.0 was the first release
> > > that contained basic support. The XML below doesn't show what I'd
> > > expect for the PCIe topology either (see [*]), and that's likely the
> > > problem. I suggest you try a fresh VM creation using latest libvirt and
> > > virt
> > tools.
> > > If it still doesn't work, then confirm it's not a bridging issue by
> > > checking that it's always the forth network by changing their order.
> >
> > I also tried libvirt 3.5.0, but maybe I missed something, like newer
> > virt-tools. I
> > will redo the test.
> > In my previous tests, I tried changing the order, and it was always the
> > fourth
> > interface having issues.
> >
> > >
> > > [*] QEMU doc: docs/config/mach-virt-graphical.cfg
> > >
> > > > Host OS: Ubuntu Xenial 16.04.2 LTS
> > > > Guest OS: Ubuntu Xenial Cloud Image (latest available)
> > > > Reproducibility
> > > > rate: 100% with below config
> > > >
> > > > VM is created with (also domain XML attached at the end of this mail):
> > > > $ virt-install --name cfg01 --ram 4096 --vcpus 6 --cpu
> > > > host-passthrough --accelerate \ --network
> > > > network:pxe,model=virtio-net-pci \ --network
> > > > network:mgmt,model=virtio-net-pci \ --network
> > > > network:internal,model=virtio-net-pci \ --network
> > > > network:public,model=virtio-net-pci \ --console pty --autostart
> > > > --noreboot --noautoconsole --video=vga --noautoconsole [...]
> > > >
> > > > You might notice I had to disable legacy virtio in the domain XML,
> > > > or the
> > > guest would complain about a missing F_... property and no guest
> > > network interfaces would be detected.
> > >
> > > You shouldn't need to do this disable-legacy stuff when using PCIe
> > > ports, as the virtio devices will automatically use virtio-1.0 ("modern").
> > >
> > > >
> > > > For now, since we only need 4 guest interfaces, we switched one of
> > > > them
> > > back to virtio-mmio, and everything is working as expected.
> > > > Unfortunately I can't investigate this myself right now, but I can
> > > > provide
> > > additional details if needed for someone else to look into it.
> > >
> > > I just checked on my setup (RH qemu based on 2.9, RH libvirt based on
> > > 3.2). I used Fedora 26's virt-manager from my x86 notebook, remotely
> > > connecting to my AArch64 host, in order to create four networks and a
> > > guest that has them all attached to four virtio nics. It worked without a
> > > hitch.
> >
> > Good to know this is specific to my setup, I was afraid this is a real
> > issue.
> > I'll retest using all suggestions above and get back with the rootcause.
> >
> > Thank you,
> > Alex
> >
> > >
> > > Thanks,
> > > drew
> > >
> > > >
> > > > BR,
> > > > Alex
> > > >
> > > > ---
> > > > Network definitions:
> > > > ---
> > > > <network>
> > > > <name>pxe</name>
> > > > <forward mode='nat'/>
> > > > <bridge name='pxe' stp='on' delay='0'/>
> > > > <ip address='192.168.10.1' netmask='255.255.255.0'>
> > > > <dhcp>
> > > > <range start='192.168.10.100' end='192.168.10.254'/>
> > > > </dhcp>
> > > > </ip>
> > > > </network>
> > > >
> > > > <network>
> > > > <name>mgmt</name>
> > > > <bridge name='mgmt' stp='on' delay='0'/>
> > > > <ip address='172.16.10.1' netmask='255.255.255.0'>
> > > > </ip>
> > > > </network>
> > > >
> > > > <network>
> > > > <name>internal</name>
> > > > <bridge name='internal' stp='on' delay='0'/> </network>
> > > >
> > > > <network>
> > > > <name>public</name>
> > > > <forward mode='nat'/>
> > > > <bridge name='public' stp='on' delay='0'/>
> > > > <ip address='10.16.0.1' netmask='255.255.255.0'>
> > > > </ip>
> > > > </network>
> > > >
> > > > ---
> > > > Domain XML:
> > > > ---
> > > > <domain type='kvm' id='28'
> > > xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
> > > > <name>cfg01</name>
> > > > <uuid>3f8415d1-963a-46aa-8a63-1c6558775b92</uuid>
> > > > <memory unit='KiB'>4194304</memory>
> > > > <currentMemory unit='KiB'>4194304</currentMemory>
> > > > <vcpu placement='static'>6</vcpu>
> > > > <resource>
> > > > <partition>/machine</partition>
> > > > </resource>
> > > > <os>
> > > > <type arch='aarch64' machine='virt-2.9'>hvm</type>
> > > > <loader readonly='yes'
> > > type='pflash'>/usr/share/AAVMF/AAVMF_CODE.fd</loader>
> > > > <nvram>/var/lib/libvirt/qemu/nvram/cfg01_VARS.fd</nvram>
> > > > <boot dev='hd'/>
> > > > </os>
> > > > <features>
> > > > <acpi/>
> > > > <gic version='3'/>
> > > > </features>
> > > > <cpu mode='host-passthrough' check='none'/>
> > > > <clock offset='utc'/>
> > > > <on_poweroff>destroy</on_poweroff>
> > > > <on_reboot>restart</on_reboot>
> > > > <on_crash>restart</on_crash>
> > > > <devices>
> > > > <emulator>/usr/bin/kvm</emulator>
> > > > <disk type='file' device='disk'>
> > > > <driver name='qemu' type='qcow2' cache='none' io='native'/>
> > > > <source
> > > file='/home/jenkins/alav/armband/upstream/fuel/mcp/scripts/images/mcp
> > > _cfg01.qcow2'/>
> > > > <backingStore/>
> > > > <target dev='vda' bus='virtio'/>
> > > > <alias name='virtio-disk0'/>
> > > > <address type='pci' domain='0x0000' bus='0x04' slot='0x00'
> > > function='0x0'/>
> > > > </disk>
> > > > <disk type='file' device='cdrom'>
> > > > <driver name='qemu' type='raw'/>
> > > > <source
> > > file='/home/jenkins/alav/armband/upstream/fuel/mcp/scripts/images/mcp
> > > _cfg01.iso'/>
> > > > <backingStore/>
> > > > <target dev='sda' bus='scsi'/>
> > > > <readonly/>
> > > > <alias name='scsi0-0-0-0'/>
> > > > <address type='drive' controller='0' bus='0' target='0' unit='0'/>
> > > > </disk>
> > > > <controller type='scsi' index='0' model='virtio-scsi'>
> > > > <alias name='scsi0'/>
> > > > <address type='pci' domain='0x0000' bus='0x03' slot='0x00'
> > > function='0x0'/>
> > > > </controller>
> > > > <controller type='pci' index='0' model='pcie-root'>
> > > > <alias name='pcie.0'/>
> > > > </controller>
> > > > <controller type='pci' index='1' model='dmi-to-pci-bridge'>
> > > > <model name='i82801b11-bridge'/>
> > > > <alias name='pci.1'/>
> > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x01'
> > > function='0x0'/>
> > > > </controller>
> > > > <controller type='pci' index='2' model='pci-bridge'>
> > > > <model name='pci-bridge'/>
> > > > <target chassisNr='2'/>
> > > > <alias name='pci.2'/>
> > > > <address type='pci' domain='0x0000' bus='0x01' slot='0x00'
> > > function='0x0'/>
> > > > </controller>
> > > > <controller type='pci' index='3' model='pcie-root-port'>
> > > > <model name='pcie-root-port'/>
> > > > <target chassis='3' port='0x10'/>
> > > > <alias name='pci.3'/>
> > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
> > > function='0x0' multifunction='on'/>
> > > > </controller>
> > > > <controller type='pci' index='4' model='pcie-root-port'>
> > > > <model name='pcie-root-port'/>
> > > > <target chassis='4' port='0x11'/>
> > > > <alias name='pci.4'/>
> > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
> > > function='0x1'/>
> > > > </controller>
> > > > <controller type='pci' index='5' model='pcie-root-port'>
> > > > <model name='pcie-root-port'/>
> > > > <target chassis='5' port='0x12'/>
> > > > <alias name='pci.5'/>
> > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
> > > function='0x2'/>
> > > > </controller>
> > > > <interface type='network'>
> > > > <mac address='52:54:00:ae:48:81'/>
> > > > <source network='pxe' bridge='pxe'/>
> > > > <target dev='vnet0'/>
> > > > <model type='virtio-net-pci'/>
> > > > <alias name='net0'/>
> > > > <address type='pci' domain='0x0000' bus='0x02' slot='0x01'
> > > function='0x0'/>
> > > > </interface>
> > > > <interface type='network'>
> > > > <mac address='52:54:00:36:2c:a9'/>
> > > > <source network='mgmt' bridge='mgmt'/>
> > > > <target dev='vnet1'/>
> > > > <model type='virtio-net-pci'/>
> > > > <alias name='net1'/>
> > > > <address type='pci' domain='0x0000' bus='0x02' slot='0x02'
> > > function='0x0'/>
> > > > </interface>
> > > > <interface type='network'>
> > > > <mac address='52:54:00:43:ce:03'/>
> > > > <source network='internal' bridge='internal'/>
> > > > <target dev='vnet2'/>
> > > > <model type='virtio-net-pci'/>
> > > > <alias name='net2'/>
> > > > <address type='pci' domain='0x0000' bus='0x02' slot='0x03'
> > > function='0x0'/>
> > > > </interface>
> > > > <interface type='network'>
> > > > <mac address='52:54:00:37:e6:0e'/>
> > > > <source network='public' bridge='public'/>
> > > > <target dev='vnet3'/>
> > > > <model type='virtio-net-pci'/>
> > > > <alias name='net3'/>
> > > > <address type='pci' domain='0x0000' bus='0x02' slot='0x04'
> > > function='0x0'/>
> > > > </interface>
> > > > <serial type='pty'>
> > > > <source path='/dev/pts/0'/>
> > > > <target port='0'/>
> > > > <alias name='serial0'/>
> > > > </serial>
> > > > <console type='pty' tty='/dev/pts/0'>
> > > > <source path='/dev/pts/0'/>
> > > > <target type='serial' port='0'/>
> > > > <alias name='serial0'/>
> > > > </console>
> > > > <graphics type='vnc' port='5900' autoport='yes' listen='127.0.0.1'>
> > > > <listen type='address' address='127.0.0.1'/>
> > > > </graphics>
> > > > <video>
> > > > <model type='vga' vram='16384' heads='1' primary='yes'/>
> > > > <alias name='video0'/>
> > > > <address type='pci' domain='0x0000' bus='0x02' slot='0x05'
> > > function='0x0'/>
> > > > </video>
> > > > </devices>
> > > > <seclabel type='dynamic' model='apparmor' relabel='yes'>
> > > > <label>libvirt-3f8415d1-963a-46aa-8a63-1c6558775b92</label>
> > > > <imagelabel>libvirt-3f8415d1-963a-46aa-8a63-
> > > 1c6558775b92</imagelabel>
> > > > </seclabel>
> > > > <seclabel type='dynamic' model='dac' relabel='yes'>
> > > > <label>+113:+115</label>
> > > > <imagelabel>+113:+115</imagelabel>
> > > > </seclabel>
> > > > <qemu:commandline>
> > > > <qemu:arg value='-global'/>
> > > > <qemu:arg value='virtio-pci.disable-legacy=on'/>
> > > > </qemu:commandline>
> > > > </domain>
> > > >
>