Re: Using virtual IOMMU in guest hypervisors other than KVM and Xen?

[Jintack Lim]

On Fri, Oct 18, 2019 at 8:37 PM Peter Xu <address@hidden> wrote:
>
> On Wed, Oct 16, 2019 at 03:01:22PM -0700, Jintack Lim wrote:
> > On Mon, Oct 14, 2019 at 7:50 PM Peter Xu <address@hidden> wrote:
> > >
> > > On Mon, Oct 14, 2019 at 01:28:49PM -0700, Jintack Lim wrote:
> > > > Hi,
> > >
> > > Hello, Jintack,
> > >
> > Hi Peter,
> >
> > > >
> > > > I'm trying to pass through a physical network device to a nested VM
> > > > using virtual IOMMU. While I was able to do it successfully using KVM
> > > > and Xen guest hypervisors running in a VM respectively, I couldn't do
> > > > it with Hyper-V as I described below. I wonder if anyone have
> > > > successfully used virtual IOMMU in other hypervisors other than KVM
> > > > and Xen? (like Hyper-V or VMware)
> > > >
> > > > The issue I have with Hyper-V is that Hyper-V gives an error that the
> > > > underlying hardware is not capable of doing passthrough. The exact
> > > > error message is as follows.
> > > >
> > > > Windows Power-shell > (Get-VMHost).IovSupportReasons
> > > > The chipset on the system does not do DMA remapping, without which
> > > > SR-IOV cannot be supported.
> > > >
> > > > I'm pretty sure that Hyper-V recognizes virtual IOMMU, though; I have
> > > > enabled iommu in windows boot loader[1], and I see differences when
> > > > booing a Windows VM with and without virtual IOMMU. I also checked
> > > > that virtual IOMMU traces are printed.
> > >
> > > What traces have you checked?  More explicitly, have you seen DMAR
> > > enabled and page table setup for that specific device to be
> > > pass-throughed?
> >
> > Thanks for the pointers. I checked that DMAR is NOT enabled. The only
> > registers that Windows guest accessed were Version Register,
> > Capability Register, and Extended Capability Register. On the other
> > hand, a Linux guest accessed other registers and enabled DMAR.
> > Here's a link to the trace I got using QEMU 4.1.0. Do you see anything
> > interesting there?
> > http://paste.ubuntu.com/p/YcSyxG9Z3x/
>
> Then I feel like Windows is reluctant to enable DMAR due to lacking of
> some caps.
>
> >
> > >
> > > >
> > > > I have tried multiple KVM/QEMU versions including the latest ones
> > > > (kernel v5.3, QEMU 4.1.0) as well as two different Windows servers
> > > > (2016 and 2019), but I see the same result. [4]
> > > >
> > > > I'd love to hear if somebody is using virtual IOMMU in Hyper-V or
> > > > VMware successfully, especially for passthrough. I also appreciate if
> > > > somebody can point out any configuration errors I have.
> > > >
> > > > Here's the qemu command line I use, basically from the QEMU vt-d
> > > > page[2] and Hyper-v on KVM from kvmforum [3].
> > > >
> > > > ./qemu/x86_64-softmmu/qemu-system-x86_64 -device
> > > > intel-iommu,intremap=on,caching-mode=on -smp 6 -m 24G -M
> > >
> > > Have you tried to use 4-level IOMMU page table (aw-bits=48 on latest
> > > QEMU, or x-aw-bits=48 on some old ones)?  IIRC we've encountered
> > > issues when trying to pass the SVVP Windows test with this, in which
> > > 4-level is required.  I'm not sure whether whether that is required in
> > > general usages of vIOMMU in Windows.
> >
> > I just tried the option you mentioned, but it didn't change anything.
> > BTW, what version of Windows was it?
>
> Sorry I don't remember that. I didn't do the test but I was just
> acknowledged that with it the test passed.  I assume you're using the
> latest QEMU here because I know Windows could require another
> capability (DMA draining) and it should be on by default in latest
> qemu master.

Thanks. Yes, I plan to use v2.11.0 eventually, but I'm trying to make
things work with the latest version first.

>
> At that time the complete cmdline to pass the test should be:
>
>   -device intel-iommu,intremap=on,aw-bits=48,caching-mode=off,eim=on
>
> I also don't remember on why caching-mode needs to be off at that
> time (otherwise SVVP fails too).

Thanks for providing the cmdline. However, turning off the
caching-mode with an assigned device resulted in the following error
on VM boot.
"We need to set caching-mode=on for intel-iommu to enable device assignment."
Does this mean that we can't assign a physical device all the way to a
nested VM with a Windows L1 hypervisor as of now?

Without assigning a device, I was able to boot a Windows VM with the
cmdline above and I see that DMAR in vIOMMU is enabled. Windows still
complains about DMA remapping, though. I'll investigate further.

>
> --
> Peter Xu
>