Re: [Qemu-devel] host physical address width issues/questions for x86_64

[Prasad Singamsetty]

On 10/16/2017 8:56 PM, Peter Xu wrote:
> On Mon, Oct 16, 2017 at 10:02:25AM -0700, Prasad Singamsetty wrote:
> > On 10/14/2017 8:53 PM, Peter Xu wrote:
> > > On Fri, Oct 13, 2017 at 11:14:03AM -0600, Alex Williamson wrote:
> > > > On Fri, 13 Oct 2017 18:01:44 +0100
> > > > "Dr. David Alan Gilbert" <address@hidden> wrote:
> > > >
> > > > > * Prasad Singamsetty (address@hidden) wrote:
> > > > > > Hi,
> > > > > >
> > > > > > I am new to the alias. I have some questions on this subject
> > > > > > and seek some clarifications from the experts in the team.
> > > > > > I ran into a couple of issues when I tried with large configuration
> > > > > > ( >= 1TB memory, > 255 CPUs) for x86_64 guest machine.
> > > > > >
> > > > > > 1. QEMU uses the default value of 40 (TCG_PHYS_ADDR_BITS) for address
> > > > > >     width if user has not specified phys-bits or host-phys-bits=true
> > > > > >     property. The default value is obviously not sufficient and
> > > > > >     causing guest kernel to crash if configured with >= 1TB
> > > > > >     memory. Depending on the linux kernel version in the guest the
> > > > > >     panic was in different code paths. The workaround is for the
> > > > > >     user to specify the phys-bits property or set the property
> > > > > >     host-phys-bits=true.
> > > > > >
> > > > > >     QUESTIONS:
> > > > ...
> > > > > > 2. host_address_width in DMAR table structure
> > > > > >
> > > > > >     In this case, the default value is set to 39
> > > > > >     (VTD_HOST_ADDRESS_WIDTH - 1). With interrupt remapping
> > > > > >     enabled for the intel iommu and the guest is configured
> > > > > >     with > 255 cpus and >= 1TB memory, the guest kernel hangs
> > > > > >     during boot up. This need to be fixed.
> > > > > >
> > > > > >     QUESTION:
> > > > > >     The question here again is can we fix this to use the
> > > > > >     real address width from the host as the default?
> > > > >
> > > > > I don't know DMAR stuff; chatting to Alex (cc'd) it does sound
> > > > > like that's an ommission that should be fixed.
> > > >
> > > > [CC +Peter]
> > > >
> > > > On physical hardware VT-d supports either 39 or 48 bit address widths
> > > > and generally you'd expect a sufficiently capable IOMMU to be matched
> > > > with the CPU.  Seems QEMU has only implemented a lower bit width and
> > > > it should probably be forcing phys bits of the VM to 39 to match until
> > > > the extended width can be implemented.  Thanks,
> > > >
> > > > Alex
> > >
> > > There were patches that tried to enable 48 bits GAW but it was
> > > not accepted somehow:
> > >
> > >    https://lists.gnu.org/archive/html/qemu-devel/2016-12/msg01886.html
> > >
> > > Would this help in any way?
> >
> > Thanks Alex for the patch info. Just curious why the patch was not
> > accepted. Any way, I will try it.
>
> I don't sure I know the reason.  Anyway, it originated from one of
> Fam's request for some NVMe tests.  If it can really help for your use
> case as well, please feel free to revive those patches, or let me know
> so that I can respin.  Thanks,

Thanks Peter. I will start with your patch and see if I can get
it to work first.

A quick question. Looking at the code, it doesn't look like there
is a way to disable dma remapping. User may have a case where he
is interested only in interrupt remapping (for > 255 cpus) and
not DMA remapping. Is that scenario considered before?

Thanks.
--Prasad