Re: [Qemu-devel] Multi GPU passthrough via VFIO

[Alex Williamson]

On Wed, 2014-02-05 at 19:59 +0100, Maik Broemme wrote:
> Hi,
> 
> currently VFIO with multi GPU passthrough is working partially and
> hopefully somebody has a hint about the problem. I'm doing passthrough
> of an AMD Radeon R9 290X and AMD Radeon 7870 GHz Edition to a single VM.
> 
> If the VM is running Linux this works quite well with radeon or fglrx
> driver. Please see 'dmesg' log attached, when using the radeon driver.
> If needed I can also post one with fglrx driver.
> 
> If I do the exact same passthrough to a Windows VM and use latest AMD
> Catalyst 14.1 (2/1/2014) or AMD Catalyst 13.12 (12/18/2013) I can get
> only the first device working (AMD R9 290X) with 'x-vga=on'. I don't
> enable 'x-vga=on' on second device as this should never work. :)

Why not?  The guest is able to change the VGA enable bit in the emulated
bridge registers and access VGA space of each device, just like happens
on bare metal.  You'll only get one device initialized from seabios, but
that's the same as would happen on bare metal as well.

> I see
> BIOS boot screen and everything works fine except for the second GPU.
> The windows device manager just show me "Code 12" for the second GPU
> and its HD Audio device. Code 12 means: "This device cannot find enough
> free resources that it can use".

I've seen the same using Nvidia GRID GPUs (w/o x-vga=on), but only with
the Q35 chipset model, Linux works, Windows reports Code 12.  I have no
idea why as all the PCI resources appear to be properly sized and
mapped.  FWIW, 2 GRID GPUs assigned to a guest do work with the 440FX
chipset model.  Beyond 2 we run out of MMIO resources below 4G and
something bad happens.

> QEMU is called in both cases via the following. I just replace the
> '-drive' accordingly.
> 
> /usr/bin/taskset -c 0,1,2,3 /usr/bin/qemu-system-x86_64 \
>   -machine q35,accel=kvm \
>   -enable-kvm \
>   -nodefaults \
>   -nographic \
>   -vga none \
>   -boot order=nc \
>   -cpu host \
>   -smp cores=4,threads=1,sockets=1 \
>   -m 8192 \
>   -rtc base=localtime \
>   -k de \
>   -drive 
> file=/srv/kvm/linux-drive0.img,id=drive0,if=none,cache=none,aio=threads \
>   -mon chardev=monitor0 \
>   -chardev socket,id=monitor0,path=/tmp/linux.monitor,nowait,server \
>   -netdev tap,id=net0,vhost=on,helper=/usr/lib/qemu/qemu-bridge-helper \
>   -device virtio-net-pci,netdev=net0,mac=00:00:00:02:01:04 \
>   -device virtio-blk-pci,drive=drive0,ioeventfd=on \
>   -device ioh3420,bus=pcie.0,id=pcie0,port=1,chassis=1,multifunction=on \
>   -device ioh3420,bus=pcie.0,id=pcie1,port=2,chassis=2,multifunction=on \
>   -device vfio-pci,host=01:00.0,addr=00.0,bus=pcie0,multifunction=on,x-vga=on 
> \
>   -device vfio-pci,host=01:00.1,addr=00.1,bus=pcie0 \
>   -device vfio-pci,host=02:00.0,addr=00.0,bus=pcie1,multifunction=on \
>   -device vfio-pci,host=02:00.1,addr=00.1,bus=pcie1 \
>   -no-reboot
> 
> My setup is the following:
> 
> Kernel: linux-3.13.1
> Seabios: seabios-git-rel.1.7.4.r51.g151d034 (5/2/2014)
> QEMU: qemu-git-2.0.r30666.g31db5b3 (5/2/2014)
> 
> Below is the 'lspci' output and I'm using the AMD Radeon HD 5430 as device
> for my local X server:
> 
> 00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI 
> bridge (external gfx0 port B) (rev 02)
> 00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD/ATI] RD990 I/O Memory 
> Management Unit (IOMMU)
> 00:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI 
> bridge (PCI express gpp port B)
> 00:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI 
> bridge (PCI express gpp port D)
> 00:09.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI 
> bridge (PCI express gpp port H)
> 00:0d.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI 
> bridge (external gfx1 port B)
> 00:11.0 SATA controller: Advanced Micro Devices, Inc. [AMD/ATI] 
> SB7x0/SB8x0/SB9x0 SATA Controller [AHCI mode] (rev 40)
> 00:12.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] 
> SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
> 00:12.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] 
> SB7x0/SB8x0/SB9x0 USB EHCI Controller
> 00:13.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] 
> SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
> 00:13.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] 
> SB7x0/SB8x0/SB9x0 USB EHCI Controller
> 00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 SMBus Controller 
> (rev 42)
> 00:14.2 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 Azalia 
> (Intel HDA) (rev 40)
> 00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 
> LPC host controller (rev 40)
> 00:14.4 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 PCI to PCI 
> Bridge (rev 40)
> 00:14.5 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] 
> SB7x0/SB8x0/SB9x0 USB OHCI2 Controller
> 00:15.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB700/SB800/SB900 
> PCI to PCI bridge (PCIE port 0)
> 00:15.1 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB700/SB800/SB900 
> PCI to PCI bridge (PCIE port 1)
> 00:15.2 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB900 PCI to PCI 
> bridge (PCIE port 2)
> 00:15.3 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB900 PCI to PCI 
> bridge (PCIE port 3)
> 00:16.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] 
> SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
> 00:16.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] 
> SB7x0/SB8x0/SB9x0 USB EHCI Controller
> 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor 
> Function 0
> 00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor 
> Function 1
> 00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor 
> Function 2
> 00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor 
> Function 3
> 00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor 
> Function 4
> 00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor 
> Function 5
> 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] 
> Hawaii XT [Radeon HD 8970]
> 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Device aac8
> 02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] 
> Pitcairn XT [Radeon HD 7870 GHz Edition]
> 02:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape 
> Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]
> 03:00.0 USB controller: Etron Technology, Inc. EJ168 USB 3.0 Host Controller 
> (rev 01)
> 04:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] 
> Park [Mobility Radeon HD 5430]
> 04:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cedar HDMI Audio 
> [Radeon HD 5400/6300 Series]
> 06:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. 
> RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06)
> 07:00.0 USB controller: Etron Technology, Inc. EJ168 USB 3.0 Host Controller 
> (rev 01)
> 
> Another minor issue is that the R9 290X is not reset during shutdown of
> VM (neither Linux nor Windows) but it can be tricked with doing
> "suspend-to-ram" between two starts. That's why I use '-no-reboot' option
> in QEMU. The 7870 is doing the reset properly.


Is the NoSoftRst "-" on the 290X vs "+" on the 7870 in lspci -vvv by
chance?  Thanks,

Alex