RE: [PATCH Kernel v24 0/8] Add UAPIs to support migration for VFIO devices

[Tian, Kevin]

> From: Xiang Zheng
> Sent: Wednesday, July 22, 2020 10:56 AM
> 
> Hi Alex,
> 
> Thank you for your suggestion.
> 
> On 2020/7/22 6:43, Alex Williamson wrote:
> > On Tue, 21 Jul 2020 10:43:21 +0800
> > Xiang Zheng <zhengxiang9@huawei.com> wrote:
> >
> >> Hi Kirti,
> >>
> >> Sorry to disturb you since this patch set has been merged, and I cannot
> >> receive the qemu-side emails about this patch set.
> >>
> >> We are going to support migration for VFIO devices which support dirty
> >> pages tracking.
> >>
> >> And we also plan to leverage SMMU HTTU feature to do the dirty pages
> >> tracking for the devices which don't support dirty pages tracking.
> >>
> >> For the above two cases, which side determines to choose IOMMU driver
> or
> >> vendor driver to do dirty bitmap tracking, Qemu or VFIO?
> >>
> >> In brief, if both IOMMU and VFIO devices support dirty pages tracking,
> >> we can check the capability and prefer to track dirty pages on device
> >> vendor driver which is more efficient.
> >>
> >> The qusetion is which side to do the check and selection? In my opinion,
> >> Qemu/userspace seems more suitable.
> >
> > Dirty page tracking is consolidated at the vfio container level.
> > Userspace has no basis for determining or interface for selecting a
> > dirty bitmap provider, so I would disagree that QEMU should play any
> > role here.  The container dirty bitmap tries to provide the finest
> > granularity available based on the support of all the devices/groups
> > managed by the container.  If there are groups attached to the
> > container that have not participated in page pinning, then we consider
> > all DMA mappings within the container as persistently dirty.  Once all
> > of the participants subscribe to page pinning, the dirty scope is
> > reduced to the pinned pages.  IOMMU support for dirty page logging would
> > introduce finer granularity yet, which we would probably prefer over
> > page pinning, but interfaces for this have not been devised.
> 
> Kevin and his colleagues may add these APIs in the future.
> We also plan to support these interfaces on SMMU driver and afterwards we
> can have a further discussion.

Yes, we are working on that part. Generally speaking I agree with Alex
that the kernel should decide dirty bitmap provider and IOMMU dirty
logging is likely preferred over page pinning. Page pinning either provides
only coarse-grained dirty info (e.g. 100's MB pinned pages observed in
vGPU case) or when reaching finer granularity it may affect fast-path 
performance (e.g. driver may have to intercept fast-path operations to
track dirtied pages closely). IOMMU provides an architectural and
lighter approach for tracking dirty pages.

Thanks
Kevin

> 
> >
> > Ideally userspace should be unaware of any of this, the benefit would
> > be seen transparently by having a more sparsely filled dirty bitmap,
> > which more accurately reflects how memory is actually being dirtied.
> 
> Yes, indeed.
> 
> --
> Thanks,
> Xiang