Re: [Qemu-devel] [v10][PATCH 03/10] piix: create host bridge to passthrough

[Alex Williamson]

Hi Allen,

On Wed, 10 Feb 2016 03:40:54 +0000
"Kay, Allen M" <address@hidden> wrote:

> > -----Original Message-----
> > From: Alex Williamson [mailto:address@hidden
> > Sent: Tuesday, February 09, 2016 1:33 PM
> > To: Kay, Allen M <address@hidden>
> > Cc: address@hidden; address@hidden;
> > address@hidden; address@hidden;
> > address@hidden; address@hidden; Ruan, Shuai
> > <address@hidden>
> > Subject: Re: [Qemu-devel] [v10][PATCH 03/10] piix: create host bridge to
> > passthrough
> > 
> > Hi Allen,
> >   
> Hi Alex,
> 
> > Thanks for the reply.  I'm totally onboard with you for BDW/SKL for
> > supported platforms, but I'd like to understand what the incremental
> > investment is for each back step within reason for older GPUs, at least for
> > best-effort, community support.  If we want to assume BDW/SKL and
> > Universal Passthrough Mode, then we could abandon the host bridge and
> > ISA bridge modifications altogether, right?  
> 
> Right.
> 
> > On the other hand, it's not clear
> > to me that UPT drivers are that pervasive and if we need to enable some
> > degree of host bridge/ISA bridge poke thrus, why not enable a fair chunk of
> > users, including me.
> >   
> 
> Sounds good.
> 
> > My IVB desktop seems to be working well (win10 + linux) with only poking
> > through the standard host bridge and ISA bridge identifications.  Yes, I 
> > need
> > to know about the different GMCH layout of IVB vs BDW in the IGD device,
> > but I've already taken care of that.  
> 
> Other than VendorID/DeviceID, which other PCI config fields in host bridge 
> and ISA bridge fields did you have to passthrough to get your IVB to work.  
> Are you passing IGD through as primary or secondary?

The host bridge I stripped down to:

    {PCI_REVISION_ID,         2},
    {PCI_SUBSYSTEM_VENDOR_ID, 2},
    {PCI_SUBSYSTEM_ID,        2},

And used this for the ISA bridge:

    {PCI_VENDOR_ID,           2},
    {PCI_DEVICE_ID,           2},
    {PCI_REVISION_ID,         2},
    {PCI_SUBSYSTEM_VENDOR_ID, 2},
    {PCI_SUBSYSTEM_ID,        2},

So nothing to do with GMCH/BDSM seems to be necessary for the IVB
desktop system. I didn't try to trim further.

I can run that system as either primary or secondary.  On both systems
I'm using pci-stub to keep i915 from interfering on the host, on the
SNB laptop I also use video=efifb:off since it's configured for UEFI
boot, I'm not sure if that play any role in why it's not working.

I've tried adding a bunch more registers on the SNB system to see if it
helps, for the host bridge:

    {0x50,                    2},
    {0x52,                    2},
    {0xa4,                    4},
    {0xa8,                    4},

(ie. the registers Tiejun added with this patch)

And on the ISA bridge:

    {0x40,        4},
    {0x44,        4},
    {0x48,        4},
    {0x4c,        4},
    {0xf0,        4},

These were registers I saw accessed with tracing enabled, but nothing
seemed to change by adding them. I don't see any accesses to the BDSM
register on the host bridge, but I do see GMCH accesses.  Sort of
interesting is that the guest reads 201h from that register and tries
to write 203h. In the read case we have a 2MB GGMS and GGCLCK set, the
guest tries to set IVD (IGD VGA Disable). I'm not sure why it bothers
to try to do this with GGCLCK indicated since the register is locked,
but it is slightly troubling that the spec indicates that the BIOS must
*not* set IVD to zero (VGA enabled) if GMS pre-allocates no memory,
which is exactly what we're doing to try to indicate no stolen memory.
If we could actually disable VGA on IGD, that might be a nice option,
but I know there are issues with that (iirc, there's no disable for one
of either the VGA MMIO or I/O port space, so the only option is to
disable that address space at the PCI command register, which has
obvious implications - I think it was probably I/O port space).

I'm currently trying a Linux live CD on the guest for the SNB, mostly
because I can see driver error messages and look at the driver source,
neither of which are available to me for the Windows driver.  Both of
the driver warnings I get are from the driver thinking a certain
feature should be disabled but finds it enabled.  They're also sort of
on a strange drm release path, maybe due to being in secondary mode and
Xorg not being configured to use the IGD on this image.  In case it
looks familiar:

[   12.397908] ------------[ cut here ]------------
[   12.399175] WARNING: CPU: 0 PID: 1135 at 
drivers/gpu/drm/i915/intel_display.c:1259 assert_fdi_rx_pll+0x78/0xb0 [i915]()
[   12.401720] FDI RX PLL assertion failure (expected off, current on)
[   12.403239] Modules linked in: ebtable_broute bridge stp llc ebtable_filter 
ebtable_nat ebtables ip6table_security ip6table_mangle ip6table_raw 
ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_filter 
ip6_tables iptable_security iptable_mangle iptable_raw iptable_nat 
nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iTCO_wdt 
gpio_ich iTCO_vendor_support ppdev joydev parport_pc lpc_ich i2c_piix4 parport 
acpi_cpufreq nfsd auth_rpcgss nfs_acl lockd grace isofs squashfs i915 bochs_drm 
video ttm i2c_algo_bit drm_kms_helper drm ata_generic serio_raw pata_acpi 
scsi_dh_rdac scsi_dh_emc scsi_dh_alua sunrpc loop
[   12.417994] CPU: 0 PID: 1135 Comm: Xorg Not tainted 4.2.3-300.fc23.x86_64 #1
[   12.419733] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 
rel-1.9.0-80-g2390eb5 04/01/2014
[   12.422100]  0000000000000000 00000000d92c128c ffff880078f53ab8 
ffffffff81771fca
[   12.424023]  0000000000000000 ffff880078f53b10 ffff880078f53af8 
ffffffff8109e4a6
[   12.425949]  ffff880078f53ad8 0000000000000000 ffff88007f680000 
ffff88007fb72000
[   12.427719] Call Trace:
[   12.428354]  [<ffffffff81771fca>] dump_stack+0x45/0x57
[   12.429580]  [<ffffffff8109e4a6>] warn_slowpath_common+0x86/0xc0
[   12.431037]  [<ffffffff8109e535>] warn_slowpath_fmt+0x55/0x70
[   12.432497]  [<ffffffffa01e5a98>] assert_fdi_rx_pll+0x78/0xb0 [i915]
[   12.434033]  [<ffffffffa0221433>] intel_pre_enable_lvds+0x53/0x180 [i915]
[   12.435760]  [<ffffffffa01ef729>] ironlake_crtc_enable+0x199/0xd30 [i915]
[   12.437433]  [<ffffffffa0190c80>] ? intel_runtime_pm_put+0x50/0x60 [i915]
[   12.439083]  [<ffffffffa0190d98>] ? intel_display_power_put+0x108/0x160 
[i915]
[   12.440914]  [<ffffffffa01ecff6>] __intel_set_mode+0x916/0xb60 [i915]
[   12.442458]  [<ffffffffa01f3ee6>] intel_crtc_set_config+0x2b6/0x580 [i915]
[   12.444180]  [<ffffffffa0099326>] drm_mode_set_config_internal+0x66/0x100 
[drm]
[   12.445917]  [<ffffffffa016d512>] restore_fbdev_mode+0xc2/0xf0 
[drm_kms_helper]
[   12.447695]  [<ffffffffa016f399>] 
drm_fb_helper_restore_fbdev_mode_unlocked+0x29/0x70 [drm_kms_helper]
[   12.450116]  [<ffffffffa020350e>] intel_fbdev_restore_mode+0x1e/0x50 [i915]
[   12.451794]  [<ffffffffa022c96e>] i915_driver_lastclose+0xe/0x20 [i915]
[   12.453360]  [<ffffffffa008c8ee>] drm_lastclose+0x2e/0x140 [drm]
[   12.454755]  [<ffffffffa008ccaf>] drm_release+0x2af/0x490 [drm]
[   12.456324]  [<ffffffff8121fbfc>] __fput+0xdc/0x1e0
[   12.457525]  [<ffffffff8121fd4e>] ____fput+0xe/0x10
[   12.458736]  [<ffffffff810bab65>] task_work_run+0x85/0xb0
[   12.460028]  [<ffffffff81014a2d>] do_notify_resume+0x8d/0x90
[   12.461424]  [<ffffffff81778b7c>] int_signal+0x12/0x17
[   12.462757] ---[ end trace 3278f15671a6fa94 ]---
[   12.546650] ------------[ cut here ]------------
[   12.547765] WARNING: CPU: 0 PID: 1135 at 
drivers/gpu/drm/i915/intel_display.c:1466 
assert_pch_transcoder_disabled+0x67/0x70 [i915]()
[   12.550847] transcoder assertion failed, should be off on pipe A but is 
still active
[   12.552858] Modules linked in: ebtable_broute bridge stp llc ebtable_filter 
ebtable_nat ebtables ip6table_security ip6table_mangle ip6table_raw 
ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_filter 
ip6_tables iptable_security iptable_mangle iptable_raw iptable_nat 
nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iTCO_wdt 
gpio_ich iTCO_vendor_support ppdev joydev parport_pc lpc_ich i2c_piix4 parport 
acpi_cpufreq nfsd auth_rpcgss nfs_acl lockd grace isofs squashfs i915 bochs_drm 
video ttm i2c_algo_bit drm_kms_helper drm ata_generic serio_raw pata_acpi 
scsi_dh_rdac scsi_dh_emc scsi_dh_alua sunrpc loop
[   12.568444] CPU: 0 PID: 1135 Comm: Xorg Tainted: G        W       
4.2.3-300.fc23.x86_64 #1
[   12.570599] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 
rel-1.9.0-80-g2390eb5 04/01/2014
[   12.572988]  0000000000000000 00000000d92c128c ffff880078f53b08 
ffffffff81771fca
[   12.574973]  0000000000000000 ffff880078f53b60 ffff880078f53b48 
ffffffff8109e4a6
[   12.576826]  ffff880078f53b28 0000000000000000 ffff88007f680000 
ffff88007fb72000
[   12.578804] Call Trace:
[   12.579454]  [<ffffffff81771fca>] dump_stack+0x45/0x57
[   12.580775]  [<ffffffff8109e4a6>] warn_slowpath_common+0x86/0xc0
[   12.582419]  [<ffffffff8109e535>] warn_slowpath_fmt+0x55/0x70
[   12.583924]  [<ffffffffa01e1067>] assert_pch_transcoder_disabled+0x67/0x70 
[i915]
[   12.585834]  [<ffffffffa01efb7c>] ironlake_crtc_enable+0x5ec/0xd30 [i915]
[   12.587634]  [<ffffffffa0190c80>] ? intel_runtime_pm_put+0x50/0x60 [i915]
[   12.589402]  [<ffffffffa0190d98>] ? intel_display_power_put+0x108/0x160 
[i915]
[   12.591294]  [<ffffffffa01ecff6>] __intel_set_mode+0x916/0xb60 [i915]
[   12.593019]  [<ffffffffa01f3ee6>] intel_crtc_set_config+0x2b6/0x580 [i915]
[   12.594797]  [<ffffffffa0099326>] drm_mode_set_config_internal+0x66/0x100 
[drm]
[   12.596651]  [<ffffffffa016d512>] restore_fbdev_mode+0xc2/0xf0 
[drm_kms_helper]
[   12.598574]  [<ffffffffa016f399>] 
drm_fb_helper_restore_fbdev_mode_unlocked+0x29/0x70 [drm_kms_helper]
[   12.601022]  [<ffffffffa020350e>] intel_fbdev_restore_mode+0x1e/0x50 [i915]
[   12.602888]  [<ffffffffa022c96e>] i915_driver_lastclose+0xe/0x20 [i915]
[   12.604576]  [<ffffffffa008c8ee>] drm_lastclose+0x2e/0x140 [drm]
[   12.606098]  [<ffffffffa008ccaf>] drm_release+0x2af/0x490 [drm]
[   12.607662]  [<ffffffff8121fbfc>] __fput+0xdc/0x1e0
[   12.609019]  [<ffffffff8121fd4e>] ____fput+0xe/0x10
[   12.610283]  [<ffffffff810bab65>] task_work_run+0x85/0xb0
[   12.611718]  [<ffffffff81014a2d>] do_notify_resume+0x8d/0x90
[   12.613225]  [<ffffffff81778b7c>] int_signal+0x12/0x17
[   12.614600] ---[ end trace 3278f15671a6fa95 ]---
[   12.635700] [drm:intel_set_pch_fifo_underrun_reporting [i915]] *ERROR* 
uncleared pch fifo underrun on pch transcoder A
[   12.636007] [drm:intel_pch_fifo_underrun_irq_handler [i915]] *ERROR* PCH 
transcoder A FIFO underrun
[   12.642883] [drm:intel_cpu_fifo_underrun_irq_handler [i915]] *ERROR* CPU 
pipe A FIFO underrun

Otherwise i915 seems fairly happy:

[    4.940225] i915: No ACPI video bus found
[    4.943223] [drm] Initialized i915 1.6.0 20150522 for 0000:00:03.0 on minor 1
[    5.000053] [drm] GMBUS [i915 gmbus vga] timed out, falling back to bit 
banging on pin 2
[    5.038055] i915 0000:00:03.0: fb1: inteldrmfb frame buffer device

The bit banging messages is apparently normal on this system (Lenovo
W520, Nvidia disabled in BIOS - could optimus still be getting in the
way?). I don't know that any of the above is actually fatal though, I
just know that I don't get a signal out of the LVDS panel or any of the
other outputs. However, the Xorg log file looks normal and if I disable
mmap support in vfio and move the mouse cursor around on the VNC window,
the trace log looks a lot like graphics is running ok, the panel is just
turned off. I'm thinking that makes the above warnings more relevant
since they're in the space of FDI and PCH transcoders, which I think is
how we get to a display output.

> > It seems like SNB is pretty similar to IVB (they run on the same chipsets), 
> >  
> 
> Agree.
> 
> > but I haven't yet fgured out the magic to make an SNB laptop light up with 
> > IGD
> > assignment, which would be useful to show that OpRegion passthrough is
> > actually doing something.
> >   
> 
> There might be a difference in BDSM definition.  You might want to add 0xb0 
> in host bridge passthrough and try SNB again.

I'm not seeing any accesses there in the tracing, so I don't think the
driver cares about it.

> It is also possible the difference might be caused by difference in driver 
> version.  If I google "sandybridge windows graphics driver" and "ivybridge 
> windows graphics driver" I get the following:
> 
> SNB driver: 
> https://downloadcenter.intel.com/product/81502/Intel-HD-Graphics-2000-for-2nd-Generation-Intel-Core-Processors
> IVB driver: 
> https://downloadcenter.intel.com/product/81501/Intel-HD-Graphics-2500-for-3rd-Generation-Intel-Core-Processors
> 
> SNB driver points to version 15.28 driver binary.  This driver supports SNB 
> and ILK graphics.
> IVB driver points to version 15.33 driver binary.  This driver supports IVB 
> and SNB graphics.
> 
> One experiment you can try is to install same IVB 15.33 binary on both IVB 
> and SNB systems.  To prevent Windows update messing with driver version, you 
> should do the following:
> 
> 1)  disable MSFT driver update: Computer->"Advance system 
> settings"->Hardware->"Device Installation Settings"->"No, let me choose what 
> to do"->"Never install driver software from Windows Update".
> 2) Disable Windows update to never check or download updates.
> 3) remove "c:\windows\system32\DriverStore\FileRepository\igd*"
> 4) re-install IGD driver to desired binary.
> 5) Double check driver versions of both IVB and SNB are the same after reboot 
> guest.  Unfortunately, it won't said 15.33 but have a version number starts 
> with 10.*.

There are certainly a lot of subtle differences between IVB and SNB in
the Linux driver, if I have time I'll see about setting up a Windows VM
on the SNB system.  Linux feels pretty close though if I could just get
the panel turned on.

> > If we ignore IronLake and older GPUs, how many VM chipset hacks do we
> > need?  What combinations would require GMCH mirrored in the host bridge,  
> 
> Base on original code in Xen, you can try the following host bridge offset 
> passthrough.  I have also attached pt-graphics.c from Xen/QEMU for your 
> reference.
> 
>        case 0x00:        /* vendor id */
>         case 0x02:        /* device id */
>         case 0x08:        /* revision id */
>         case 0x2c:        /* sybsystem vendor id */
>         case 0x2e:        /* sybsystem id */
>         case 0x50:        /* SNB: processor graphics control register */
>         case 0x52:        /* processor graphics control register */
>         case 0xa0:        /* top of memory */
>         case 0xb0:        /* ILK: BSM: should read from dev 2 offset 0x5c */
>         case 0x58:        /* SNB: PAVPC Offset */
>         case 0xa4:        /* SNB: graphics base of stolen memory */
>         case 0xa8:        /* SNB: base of GTT stolen memory */
> 
> > and couldn't we mirror it from the IGD device itself since it's present in 
> > the
> > same location all the way back through SNB.
> >   
> 
> Yes, you can copy content from MCCG in IGD device to same 0x50 offset in host 
> bridge.
> 
> > We certainly don't want to get into the nastiness of RMRRs in a VM, but do
> > the stolen memory use cases you've outlined explain the DMAR faults to that
> > region that I was seeing just booting a VM with a Linux live CD?  
> 
> What I described about stolen memory was based on Windows driver usage.  I 
> will need to try Linux live CD to see DMAR faults you encountered.  I will 
> try that after I finish setting up my environment.  I have been following 
> your blog.  Great instructions!

Good, glad you found it.  I'm just running a simple VM like this:

/home/alwillia/local/bin/qemu-system-x86_64 -m 2G -bios 
/home/alwillia/Work/seabios.git/out/bios.bin -net none -cdrom 
/net/store/exports/ISOs/Fedora-Live-Cinnamon-x86_64-23-10.iso -boot d -monitor 
stdio -device vfio-pci,host=00:02.0,rombar=0,x-no-mmap=on -vnc :1 -vga std 
-enable-kvm -trace events=events.txt

events.txt contains:
vfio_region_read
vfio_region_write
vfio_pci_read_config
vfio_pci_write_config
vfio_pci_igd*
pci_cfg_*

If you base on the last patch series I posted on top of Gerd's changes,
you'll need -M pc,igd_passthru=on I believe too, the code I'm working
on passes the host and ISA bridge config through vfio and initiates the
ISA bridge automatically.  I'll try to post it this week.

> > Again, it seems just as easy, if not easier to copy GMCH from the IGD 
> > itself into the
> > host bridge.
> >   
> Not sure if I follow ... how does this would solve stolen memory/RMRR 
> issue...?

In the vfio code I have now, the kernel virtualizes GMCH and BDSM on the
device to provide a pre-sanitized version, so if we need those on the
host bridge, it's easier to copy what's on the IGD instead of adding yet
more code that knows about the different GMCH layouts.  The solution is
the same, clear BDSM and GMCH.GMS to make the guest not try to access
the host stolen memory.

> > Ok, unless anyone shouts really loudly and it doesn't affect anything newer
> > than IronLake, I'm happy to let those fall off the plate.  I don't have any 
> > older
> > systems that I care to make work with this.
> >   
> Sounds good.
> 
> > 
> > Ok, so the code comment is pretty misleading for these.  Would anything
> > since SNB need these?
> >  
> We did not do much pruning pre-SNB.  Tiejun and I did checked HSW last year, 
> most (maybe all) of these register are not needed.
> 
> > What about the BDSM register in the host bridge?
> >  Is the Xen experience of not needing to copy it relevant since stolen 
> > memory is
> > identity mapped into the VM anyway?  Maybe Xen achieves the same effect
> > by not copying it and exposing it as zero.  
> 
> In SNB, BDSM is at offset 0xb0 in host bridge.  Xen/QEMU passes thru 0xb0 in 
> host bridge.  Although the comment says it is for ILK.  It is also true for 
> SNB.  In current BDW/SKL driver, BDSM is located at 0x5c in IGD device.

If I could spot any accesses to BDSM on the host bridge, I'd jump on
this, but my guest driver doesn't seem to care about it according to
the tracing.  Lighting up an output seems to be the issue.  Thanks,

Alex