RE: [PATCH v1 20/22] intel_iommu: propagate PASID-based iotlb invalidation to host

[Liu, Yi L]

> From: Peter Xu <address@hidden>
> Sent: Thursday, March 26, 2020 9:03 PM
> To: Liu, Yi L <address@hidden>
> Subject: Re: [PATCH v1 20/22] intel_iommu: propagate PASID-based iotlb
> invalidation to host
> 
> On Thu, Mar 26, 2020 at 05:41:39AM +0000, Liu, Yi L wrote:
> > > From: Liu, Yi L
> > > Sent: Wednesday, March 25, 2020 9:22 PM
> > > To: 'Peter Xu' <address@hidden>
> > > Subject: RE: [PATCH v1 20/22] intel_iommu: propagate PASID-based
> > > iotlb invalidation to host
> > >
> > > > From: Peter Xu <address@hidden>
> > > > Sent: Wednesday, March 25, 2020 2:34 AM
> > > > To: Liu, Yi L <address@hidden>
> > > > Subject: Re: [PATCH v1 20/22] intel_iommu: propagate PASID-based
> > > > iotlb invalidation to host
> > > >
> > > > On Sun, Mar 22, 2020 at 05:36:17AM -0700, Liu Yi L wrote:
> > > > > This patch propagates PASID-based iotlb invalidation to host.
> > > > >
> > > > > Intel VT-d 3.0 supports nested translation in PASID granular.
> > > > > Guest SVA support could be implemented by configuring nested
> > > > > translation on specific PASID. This is also known as dual stage
> > > > > DMA translation.
> > > > >
> > > > > Under such configuration, guest owns the GVA->GPA translation
> > > > > which is configured as first level page table in host side for a
> > > > > specific pasid, and host owns GPA->HPA translation. As guest
> > > > > owns first level translation table, piotlb invalidation should
> > > > > be propagated to host since host IOMMU will cache first level
> > > > > page table related mappings during DMA address translation.
> > > > >
> > > > > This patch traps the guest PASID-based iotlb flush and propagate
> > > > > it to host.
> > > > >
> > > > > Cc: Kevin Tian <address@hidden>
> > > > > Cc: Jacob Pan <address@hidden>
> > > > > Cc: Peter Xu <address@hidden>
> > > > > Cc: Yi Sun <address@hidden>
> > > > > Cc: Paolo Bonzini <address@hidden>
> > > > > Cc: Richard Henderson <address@hidden>
> > > > > Cc: Eduardo Habkost <address@hidden>
> > > > > Signed-off-by: Liu Yi L <address@hidden>
> > > > > ---
> > > > >  hw/i386/intel_iommu.c          | 139
> > > > +++++++++++++++++++++++++++++++++++++++++
> > > > >  hw/i386/intel_iommu_internal.h |   7 +++
> > > > >  2 files changed, 146 insertions(+)
> > > > >
> > > > > diff --git a/hw/i386/intel_iommu.c b/hw/i386/intel_iommu.c index
> > > > > b9ac07d..10d314d 100644
> > > > > --- a/hw/i386/intel_iommu.c
> > > > > +++ b/hw/i386/intel_iommu.c
> > > > > @@ -3134,15 +3134,154 @@ static bool
> > > > vtd_process_pasid_desc(IntelIOMMUState *s,
> > > > >      return (ret == 0) ? true : false;  }
> > > > >
> > > > > +/**
> > > > > + * Caller of this function should hold iommu_lock.
> > > > > + */
> > > > > +static void vtd_invalidate_piotlb(IntelIOMMUState *s,
> > > > > +                                  VTDBus *vtd_bus,
> > > > > +                                  int devfn,
> > > > > +                                  DualIOMMUStage1Cache
> > > > > +*stage1_cache) {
> > > > > +    VTDHostIOMMUContext *vtd_dev_icx;
> > > > > +    HostIOMMUContext *host_icx;
> > > > > +
> > > > > +    vtd_dev_icx = vtd_bus->dev_icx[devfn];
> > > > > +    if (!vtd_dev_icx) {
> > > > > +        goto out;
> > > > > +    }
> > > > > +    host_icx = vtd_dev_icx->host_icx;
> > > > > +    if (!host_icx) {
> > > > > +        goto out;
> > > > > +    }
> > > > > +    if (host_iommu_ctx_flush_stage1_cache(host_icx, stage1_cache)) {
> > > > > +        error_report("Cache flush failed");
> > > >
> > > > I think this should not easily be triggered by the guest, but just
> > > > in case... Let's use
> > > > error_report_once() to be safe.
> > >
> > > Agreed.
> > >
> > > > > +    }
> > > > > +out:
> > > > > +    return;
> > > > > +}
> > > > > +
> > > > > +static inline bool vtd_pasid_cache_valid(
> > > > > +                          VTDPASIDAddressSpace *vtd_pasid_as) {
> > > > > +    return vtd_pasid_as->iommu_state &&
>                     ^^^^^^^^^^^^^^^^^^^^^^^^^
> 
> > > >
> > > > This check can be dropped because always true?
> > > >
> > > > If you agree with both the changes, please add:
> > > >
> > > > Reviewed-by: Peter Xu <address@hidden>
> > >
> > > I think the code should ensure all the pasid_as in hash table is
> > > valid. And we can since all the operations are under protection of 
> > > iommu_lock.
> > >
> > Peter,
> >
> > I think my reply was wrong. pasid_as in has table may be stale since
> > the per pasid_as cache_gen may be not identical with the cache_gen in
> > iommu_state. e.g. vtd_pasid_cache_reset() only increases the cache_gen
> > in iommu_state. So there will be pasid_as in hash table which has
> > cached pasid entry but its cache_gen is not equal to the one in
> > iommu_state. For such pasid_as, we should treat it as stale.
> > So I guess the vtd_pasid_cache_valid() is still necessary.
> 
> I guess you misread my comment. :)
> 
> I was saying the "vtd_pasid_as->iommu_state" check is not needed, because
> iommu_state was always set if the address space is created.
> vtd_pasid_cache_valid() is needed.

ok, I see.

> Also, please double confirm that vtd_pasid_cache_reset() should drop all the
> address spaces (as I think it should), not "only increase the cache_gen". 

yes, I'm just evaluating it. vtd_pasid_cache_reset() should drop all the
pasid_as and need to notify host to unbind pasid.

> IMHO you
> should only increase the cache_gen in the PSI hook (vtd_pasid_cache_psi()) 
> only.

I'm not quite get here. Why cache_gen increase only happen in PSI
hook? I think cache_gen used to avoid drop all pasid_as when a pasid
cache reset happened.

Regards,
Yi Liu