Re: [Qemu-devel] vfio for platform devices - 9/5/2012 - minutes

[Scott Wood]

On Thu, 2013-09-12 at 04:18 -0500, Bhushan Bharat-R65777 wrote:
> 
> > -----Original Message-----
> > From: Alex Williamson [mailto:address@hidden
> > Sent: Wednesday, September 11, 2013 10:45 PM
> > To: Yoder Stuart-B08248
> > Cc: Wood Scott-B07421; Sethi Varun-B16395; Bhushan Bharat-R65777; 'Peter
> > Maydell'; 'Santosh Shukla'; 'Alexander Graf'; 'Antonios Motakis'; 
> > 'Christoffer
> > Dall'; 'address@hidden'; address@hidden; kvm-
> > address@hidden; address@hidden
> > Subject: Re: vfio for platform devices - 9/5/2012 - minutes
> > 
> > On Wed, 2013-09-11 at 16:42 +0000, Yoder Stuart-B08248 wrote:
> > >
> > > > -----Original Message-----
> > > > From: Yoder Stuart-B08248
> > > > Sent: Thursday, September 05, 2013 12:51 PM
> > > > To: Wood Scott-B07421; Sethi Varun-B16395; Bhushan Bharat-R65777;
> > > > 'Peter Maydell'; 'Santosh Shukla'; 'Alex Williamson'; 'Alexander
> > > > Graf'; 'Antonios Motakis'; 'Christoffer Dall'; 'address@hidden'
> > > > Cc: address@hidden; 'address@hidden'; 'qemu-
> > > > address@hidden'
> > > > Subject: vfio for platform devices - 9/5/2012 - minutes
> > > >
> > > > We had a call with those interested and/or working on vfio for
> > > > platform devices.
> > > >
> > > > Participants: Scott Wood, Varun Sethi, Bharat Bhushan, Peter Maydell,
> > > >               Santosh Shukla, Alex Williamson, Alexander Graf,
> > > >               Antonios Motakis, Christoffer Dall, Kim Phillips,
> > > >               Stuart Yoder
> > > >
> > > > Several aspects to vfio for platform devices:
> > > >
> > > > 1. IOMMU groups
> > > >
> > > >  -iommu driver needs to register a bus notifier for the platform bus
> > > >   and create groups for relevant platform devices  -Antonios is
> > > > looking at this for several ARM IOMMUs  -PAMU (Freescale) driver
> > > > already does this
> > > >
> > > > 2. unbinding device from host
> > > >
> > > >  PCI:
> > > >    echo 0000:06:0d.0 >
> > > > /sys/bus/pci/devices/0000:06:0d.0/driver/unbind
> > > >  Platform:
> > > >    echo ffe101300.dma >
> > > > /sys/bus/platform/devices/ffe101300.dma/driver/unbind
> > > >
> > > >  -don't believe there are issues or work to do here
> > > >
> > > > 3. binding device to vfio-platform driver
> > > >
> > > >  PCI:
> > > >    echo 1102 0002 > /sys/bus/pci/drivers/vfio-pci/new_id
> > > >
> > > >  -this is probably the least understood issue-- platform drivers
> > > >   register themselves with the bus for a specific "name"
> > > >   string.  That is matched with device tree "compatible" strings
> > > >   later to bind a device to a driver  -we want is to have the
> > > > vfio-platform driver to dynamically bind
> > > >   to a variety of platform devices previously unknown to
> > > >   vfio-platform
> > > >  -ideally unbinding and binding could be an atomic operation  -Alex
> > > > W pointed out that x86 could leverage this work so
> > > >   keep that in mind in what we design  -Kim Phillips (Linaro) will
> > > > start working on this
> > >
> > > One thing we didn't discuss needs to be considered (probably by Kim
> > > who is looking at the 'binding device' issue) is around returning a
> > > passthru device back to the host.
> > >
> > > After a platform device has been bound to vfio and is in use by user
> > > space or a virtual machine, we also need to be able to unwind all that
> > > and return the device back to the host in a sane state.
> > >
> > > What happens when user space exits and the vfio file descriptors are
> > > closed?
> > 
> > For reference, expectations of how vfio-pci handles these situations:
> > 
> > For vfio-pci, when the reference count on the device fd drops to zero we 
> > call a
> > device disable function that includes disabling the bus master bit in config
> > space stop ongoing DMA.
> 
> There is no bus mastering for platform devices

Sure there is -- that's just PCI's name for DMA.

> and device disabling is not a standard like PCI. Do you think that we might 
> need to do some
> device specific handling.

It would be better if we could avoid the need for device specific
handling.  DMA can be disabled via the IOMMU.

The tricky part is how to enable the device in the next user; as I wrote
elsewhere in the thread I think this needs to be something the user is
aware of, so that the user quiesces the device prior to enabling IOMMU
mappings.

> >  I just introduced patches for v3.12 that enable a PCI bus reset
> > interface, but it's mostly useful for userspace, since on PCI it's often the
> > case that a bus contains multiple devices which don't necessarily align to 
> > iommu
> > group boundaries.
> > 
> > > When a plaform device is 'unbound' from vfio, what specifically
> > > happens to the device?
> > 
> > The driver remove function is called, which needs to call 
> > vfio_del_group_dev(),
> > which will block if the device is in use.  If it's unused, the release 
> > function
> > should have already been called and the device should be in a quiesced 
> > state.
> 
> What does it mean by quiesced state here? Does this mean:
>  - Device does not take more work from software; This will done if device is 
> not attached to any driver.
>  - Device does not create work for software (no interrupt etc). This can 
> achieved if interrupts masked at device/PIC and device in disabled.

It at least means that the device will not do any DMA or generate any
interrupts.

-Scott