Re: [PATCH v6] xen/pt: reserve PCI slot 2 for Intel igd-passthru

[Chuck Zmudzinski]

On 1/3/2023 10:14 AM, Alex Williamson wrote:
> On Mon, 2 Jan 2023 18:10:24 -0500
> Chuck Zmudzinski <brchuckz@netscape.net> wrote:
>
> > On 1/2/23 12:46 PM, Michael S. Tsirkin wrote:
> > > On Sun, Jan 01, 2023 at 06:52:03PM -0500, Chuck Zmudzinski wrote:  
> > > > Intel specifies that the Intel IGD must occupy slot 2 on the PCI bus,
> > > > as noted in docs/igd-assign.txt in the Qemu source code.
> > > > ... 
> > > > Signed-off-by: Chuck Zmudzinski <brchuckz@aol.com>  
> > >
> > > I'm not sure why is the issue xen specific. Can you explain?
> > > Doesn't it affect kvm too?  
> > 
> > Recall from docs/igd-assign.txt that there are two modes for
> > igd passthrough: legacy and upt, and the igd needs to be
> > at slot 2 only when using legacy mode which gives one
> > single guest exclusive access to the Intel igd.
> > 
> > It's only xen specific insofar as xen does not have support
> > for the upt mode so xen must use legacy mode which
> > requires the igd to be at slot 2. I am not an expert with
>
> UPT mode never fully materialized for direct assignment, the folks at
> Intel championing this scenario left.

Thanks for clarifying that for me.

>
> > kvm, but if I understand correctly, with kvm one can use
> > the upt mode with the Intel i915 kvmgt kernel module
> > and in that case the guest will see a virtual Intel gpu
> > that can be at any arbitrary slot when using kvmgt, and
> > also, in that case, more than one guest can access the
> > igd through the kvmgt kernel module.
>
> This is true, IIRC an Intel vGPU does not need to be in slot 2.
>
> > Again, I am not an expert and do not have as much
> > experience with kvm, but if I understand correctly it is
> > possible to use the legacy mode with kvm and I think you
> > are correct that if one uses kvm in legacy mode and without
> > using the Intel i915 kvmgt kernel module, then it would be
> > necessary to reserve slot 2 for the igd on kvm.
>
> It's necessary to configure the assigned IGD at slot 2 to make it
> functional, yes, but I don't really understand this notion of
> "reserving" slot 2.  If something occupies address 00:02.0 in the
> config, it's the user's or management tool's responsibility to move it
> to make this configuration functional.  Why does QEMU need to play a
> part in reserving this bus address.  IGD devices are not generally
> hot-pluggable either, so it doesn't seem we need to reserve an address
> in case an IGD device is added dynamically later.

As I said in earlier message in this thread, the xenlight toolstack (libxl) 
that is
provided as the default toolstack for building xen guests with pci passthrough
is not the most flexible management tool, and that is why, in the case of xen,
it is simpler to reserve slot 2 while qemu assigns the slot addresses of the
qemu emulated pci devices so that the igd can use slot 2. IIRC, In hw/pci/pci.c,
once the slot value is assigned, it is constant and cannot be changed later on
by a management tool.

>  
> > Your question makes me curious, and I have not been able
> > to determine if anyone has tried igd passthrough using
> > legacy mode on kvm with recent versions of linux and qemu.
>
> Yes, it works.
>
> > I will try reproducing the problem on kvm in legacy mode with
> > current versions of linux and qemu and report my findings.
> > With kvm, there might be enough flexibility to specify the
> > slot number for every pci device in the guest. Such a
>
> I think this is always the recommendation, libvirt will do this by
> default in order to make sure the configuration is reproducible.  This
> is what we generally rely on for kvm/vfio IGD assignment to place the
> GPU at the correct address.
>
> > capability is not available using the xenlight toolstack
> > for managing xen guests, so I have been using this patch
> > to ensure that the Intel igd is at slot 2 with xen guests
> > created by the xenlight toolstack.
>
> Seems like a deficiency in xenlight.  I'm not sure why QEMU should take
> on this burden to support support tool stacks that lack such basic
> features.

So you would prefer to patch xenlight (libxl) to make it flexible enough to 
properly
handle this case of legacy igd passthrough.

>  
> > The patch as is will only fix the problem on xen, so if the
> > problem exists on kvm also, I agree that the patch should
> > be modified to also fix it on kvm.
>
> AFAICT, it's not a problem on kvm/vfio because we generally make use of
> invocations that specify bus addresses for each device by default,
> making this a configuration requirement for the user or management tool
> stack.  Thanks,

Unfortunately, and as I mentioned in an earlier message on this thread,
the xenlight management tool stack (libxl) is not so flexible and does not
make it so easy for the administrator to specify the bus address of each
device, and that is why either this patch is needed for igd legacy passtrhough
on xen, or the libxl management tool needs to be patched so it is flexible
enough to enable the slot addresses to be assigned correctly using
that tool instead of relying on qemu to reserve slot 2 for the igd.

If there is a consensus that this should be fixed in libxl instead of in qemu,
I will work on a patch to libxl, but I will wait a while for some feedback from
the xen people before I do that.

Thanks for your feedback,

Chuck