Re: [Qemu-devel] VFIO-VGA Issue

[Alex Williamson]

On Wed, 2013-04-10 at 13:11 -0400, address@hidden wrote:
> >> Although with this patch I get much further, monitor still doesn't sync.
> >> I
> >> also see some effects on my host GPU (Intel HD4000). Starting qemu ruins
> >> colors (black turns green, blue turns purple, etc). Though switching to
> >> linux console and back to Xorg fixes it.
> >
> > Hmm, seems like that would imply the VGA arbitration isn't working.
> > When I test, I'm not running anything on the boot graphics device, it's
> > sitting at a vt login.
> I tried running from a vt (Xorg wasn't running). The colors were fine,
> passthrough didn't work.

Ok, maybe Xorg is making use of vga space, but not using the VGA
arbiter.  That would be unfortunate.

> >> Debug output this time is huge with most of the lines being variations
> >> of
> >> those below. On line 58515 qemu hangs again, and the last two blocks
> >> from
> >> this snip repeat indefinitely:
> >
> > Was this with or without KVM acceleration?
> I've been testing without KVM acceleration since you told it can be buggy.
> 
> > Is it hung or does the code below repeat indefinitely?
> The last five lines from the code below repeat indefinitely. The files
> which I used to pipe debug output grew in size to 500Mb+ in no time.
> 
> > It can take a long time to get something
> > drawn to the screen with all this debug output, but the monitor should
> > get signal pretty quickly.
> This monitor is connected via VGA cable, which is slow to change
> resolutions and such, so I couldn't see it even if there was a short timed
> signal before qemu hung (looped?). Also, waiting longer didn't give
> anything new.
> 
> > Check dmesg and look for errors reading the
> > ROM file, starting qemu w/ a ROM isn't going to get very far.
> There are only two lines produced on each qemu start:
> [   95.331196] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003)
> [   95.354793] vfio_ecap_init: 0000:01:00.0 hiding ecap address@hidden
> 
> >> vfio: vfio_vga_read(0x3c3, 1) = 0x0
> >> vfio: vfio_ati_3c3_quirk_read(0x3c3, 1) = 0xc0
> >> vfio: vfio_bar_read(0000:01:00.0:BAR4+0x4c, 4) = 0x0
> >> vfio: vfio_bar_write(0000:01:00.0:BAR4+0x0, 0x6e8c, 4)
> >> vfio: vfio_bar_read(0000:01:00.0:BAR4+0x4, 4) = 0x10029
> >
> > Do you mean the above 5 lines are repeated or the whole sequence?
> I mean those 5 lines are repeated.

We can really only guess what that sequence means, I don't know of any
documentation or code to look at for it.

> > It seems to be making some progress over the whole log, but I couldn't tell
> > you what it's doing.  Reading 0x3c3 to get the BAR3 address, then using
> > it to do some kind of access is pretty typical behavior.  Things I would
> > try - If the HD4000 is not the bare metal boot VGA, make it so.
> Do you mean I should boot into linux console with VESA driver and run qemu
> without starting Xorg? If starting qemu without Xorg in KMS console on
> HD4000 is fine, I did that.

I just want to make sure nothing is using the ATI card.  If your BIOS,
bootloader, and KMS are shown on the IGD and you're not loading the
radeon FB driver (as you note below), this should be the case.

> > Blacklist the radeon driver in the host so that you give the card to
> > qemu in a "fresh" state.  Thanks,
> >
> > Alex
> The radeon driver isn't installed. There's nothing to blacklist. kernel's
> .config:
> # CONFIG_DRM_RADEON is not set
> lspci -v shows no "Kernel driver in use:" for the graphic card.
> 
> 
> On a semi-releated note, today I tried passing through this card in Xen.
> With "gfx_passthru=1" Xen stopped the same way as qemu (one core was fully
> busy, with nothing on the monitor).

It's good to know we're not alone ;)

> However, turning gfx_passthru off did
> the trick. Win7 started loading with cirrus and switched to HD7750 halfway
> through boot proccess. I didn't do any testing just let Windows calculate
> its score. The result was 7.4 and Aero was working.

You should be able to do this with vfio too, use -vga cirrus and don't
use the x-vga option on pci-assign.  The x-vga enables legacy VGA
support for boot and primary console, as a secondary head normal PCI
device assignment should be sufficient.

> Also, in Xen's dmesg I saw these lines:
> (XEN) Intel VT-d Snoop Control not enabled.
> (XEN) Intel VT-d Dom0 DMA Passthrough not enabled.
> (XEN) Intel VT-d Queued Invalidation enabled.
> (XEN) Intel VT-d Interrupt Remapping enabled.
> (XEN) Intel VT-d Shared EPT tables not enabled.
> Can those "not enabled" features be the cause of my issues? Light search
> in google shows that many people have Snoop Control enabled.

Most of those are hardware features, I'd expect we're using the same.

>  And one more
> question, could the graphic card vendor (ASUS) make any changes that
> aren't present in the reference AMD design, but can make it hard to pass
> through the card?

I imagine each vendor has an opportunity to customize the BIOS and
things could certainly be done there to cause problems.  I'd hope that
the core device initialization and VESA extension code is mostly intact
from the reference version though.  Thanks,

Alex