Re: [Qemu-devel] host physical address width issues/questions for x86_64

[Peter Xu]

On Wed, Oct 18, 2017 at 10:19:31AM -0700, Prasad Singamsetty wrote:
> 
> 
> On 10/16/2017 8:56 PM, Peter Xu wrote:
> >On Mon, Oct 16, 2017 at 10:02:25AM -0700, Prasad Singamsetty wrote:
> >>
> >>
> >>On 10/14/2017 8:53 PM, Peter Xu wrote:
> >>>On Fri, Oct 13, 2017 at 11:14:03AM -0600, Alex Williamson wrote:
> >>>>On Fri, 13 Oct 2017 18:01:44 +0100
> >>>>"Dr. David Alan Gilbert" <address@hidden> wrote:
> >>>>
> >>>>>* Prasad Singamsetty (address@hidden) wrote:
> >>>>>>Hi,
> >>>>>>
> >>>>>>I am new to the alias. I have some questions on this subject
> >>>>>>and seek some clarifications from the experts in the team.
> >>>>>>I ran into a couple of issues when I tried with large configuration
> >>>>>>( >= 1TB memory, > 255 CPUs) for x86_64 guest machine.
> >>>>>>
> >>>>>>1. QEMU uses the default value of 40 (TCG_PHYS_ADDR_BITS) for address
> >>>>>>    width if user has not specified phys-bits or host-phys-bits=true
> >>>>>>    property. The default value is obviously not sufficient and
> >>>>>>    causing guest kernel to crash if configured with >= 1TB
> >>>>>>    memory. Depending on the linux kernel version in the guest the
> >>>>>>    panic was in different code paths. The workaround is for the
> >>>>>>    user to specify the phys-bits property or set the property
> >>>>>>    host-phys-bits=true.
> >>>>>>
> >>>>>>    QUESTIONS:
> >>>>...
> >>>>>>2. host_address_width in DMAR table structure
> >>>>>>
> >>>>>>    In this case, the default value is set to 39
> >>>>>>    (VTD_HOST_ADDRESS_WIDTH - 1). With interrupt remapping
> >>>>>>    enabled for the intel iommu and the guest is configured
> >>>>>>    with > 255 cpus and >= 1TB memory, the guest kernel hangs
> >>>>>>    during boot up. This need to be fixed.
> >>>>>>
> >>>>>>    QUESTION:
> >>>>>>    The question here again is can we fix this to use the
> >>>>>>    real address width from the host as the default?
> >>>>>
> >>>>>I don't know DMAR stuff; chatting to Alex (cc'd) it does sound
> >>>>>like that's an ommission that should be fixed.
> >>>>
> >>>>[CC +Peter]
> >>>>
> >>>>On physical hardware VT-d supports either 39 or 48 bit address widths
> >>>>and generally you'd expect a sufficiently capable IOMMU to be matched
> >>>>with the CPU.  Seems QEMU has only implemented a lower bit width and
> >>>>it should probably be forcing phys bits of the VM to 39 to match until
> >>>>the extended width can be implemented.  Thanks,
> >>>>
> >>>>Alex
> >>>
> >>>There were patches that tried to enable 48 bits GAW but it was
> >>>not accepted somehow:
> >>>
> >>>   https://lists.gnu.org/archive/html/qemu-devel/2016-12/msg01886.html
> >>>
> >>>Would this help in any way?
> >>>
> >>
> >>Thanks Alex for the patch info. Just curious why the patch was not
> >>accepted. Any way, I will try it.
> >
> >I don't sure I know the reason.  Anyway, it originated from one of
> >Fam's request for some NVMe tests.  If it can really help for your use
> >case as well, please feel free to revive those patches, or let me know
> >so that I can respin.  Thanks,
> >
> 
> Thanks Peter. I will start with your patch and see if I can get
> it to work first.
> 
> A quick question. Looking at the code, it doesn't look like there
> is a way to disable dma remapping. User may have a case where he
> is interested only in interrupt remapping (for > 255 cpus) and
> not DMA remapping. Is that scenario considered before?

It can be done in the guest if the guest doesn't want DMAR.

Note that there are two isolated kernel tunables for the VT-d device:

- intel_iommu: "on" to turn on DMAR, "off" to turn off DMAR
- intremap:    "on" to turn on IR, "off" to turn off IR

So even if guest has "intel_iommu=off" in its boot parameter, IR will
still be on by default (or specify it explicitly using "intremap=on").

Thanks,

-- 
Peter Xu