Re: [Qemu-devel] [RFC PATCH] vfio: VFIO PCI driver for Qemu

[Alex Williamson]

On Thu, 2012-07-26 at 18:59 +0300, Avi Kivity wrote:
> On 07/26/2012 05:56 PM, Alex Williamson wrote:
> >> >> Let's use the same syntax as for kvm device assignment.  Then we can
> >> >> fall back on kvm when vfio is not available.  We can also have an
> >> >> optional parameter kernel-driver to explicitly select vfio or kvm.
> >> > 
> >> > This seems confusing to me, pci-assign already has options like
> >> > prefer_msi, share_intx, and configfd that vfio doesn't.  I'm sure vfio
> >> > will eventually get options that pci-assign won't have.  How is a user
> >> > supposed to figure out what options are actually available from -device
> >> > pci-assign,? 
> >> 
> >> Read the documentation.
> > 
> > And libvirt is supposed to parse the qemu-docs package matching the
> > installed qemu binary package to figure out what's supported?
> 
> I was hoping that we could avoid any change in libvirt.

I don't think that's possible...

> >> > Isn't this the same as asking to drop all model specific
> >> > devices and just use -device net,model=e1000... hey, we've been there
> >> > before ;)  Thanks,
> >> 
> >> It's not.  e1000 is a guest visible feature. vfio and kvm assignment do
> >> exactly the same thing, as far as the guest is concerned, just using a
> >> different driver.  This is more akin to -device virtio-net,vhost=on|off
> >> (where we also have a default and a fallback, which wouldn't make sense
> >> for model=e1000).
> > 
> > I understand an agree with your desire to make this transparent from the
> > user perspective, but I think the place to do that abstraction is
> > libvirt.  The qemu command line is just the final step in a process that
> > already needs to be aware of which backend will be used.  This is not
> > simply a small tweak to the qemu options and now I'm using vfio.  It
> > goes something like this:
> > 
> >    KVM                                     VFIO
> > 1. Identify the assigned device         1. Identify the assigned device
> > 2. Unbind from host driver              2. Identify the iommu group for the 
> > device
> > 3. Bind to pci-stub                     3. Evaluate all the devices for the 
> > group
> > 4. Launch qemu                          4. Unbind all devices in the group 
> > from host drivers
> >                                         5. Bind all devices in the group to 
> > vfio-pci
> >                                         6. Launch qemu
> 
> In the common case, on x86 (but I'm repeating myself), the iommu group
> includes just one device, yes?  Could we make pci-stub an alias for the
> corresponding vfio steps?

PCI bridges masking devices is not as uncommon as you'd like, that's
exactly why Andreas is using VFIO instead of KVM assignment.  Not to
mention that VFIO takes a much more strict stance on multifunction ACS
requirements, typically resulting in all function of a multifunction
device being inseparable.  So no, I don't think multi-device groups will
be unusual at all, even on x86.  Playing games with pci-stub sounds like
a nightmare.  Personally I think we have the opportunity to make libvirt
and tools like virt-manager a lot better with VFIO.  They no longer need
to do PCI bridge testing or ACS checking for VFIO and they can better
inform the user about what devices need to be removed from the host to
provide safe assignment.

> Though I generally dislike doing magic behind the user's back.  qemu and
> even more the kernel are low level interfaces and should behave as
> regularly as possible.
> 
> > 
> > I've actually already had a report from an early adopter that did
> > everything under the VFIO list on the right, but but happened to be
> > using qemu-kvm and the -device pci-assign option and couldn't figure out
> > what was going on.  Due to KVM's poor device ownership model, it was
> > more than happy to bind to a device owned by vfio-pci.  Imagine the
> > support questions we have to ask if we support both via pci-assign;
> 
> In fact we had the same experience with kvm being enabled or not.  We
> have 'info kvm' for that.

If we have both vfio and kvm assignment in the same tree there's no
reason we couldn't intermix them within a VM.  Unfortunately we have to
beware of KVM assignment's poor ownership model, but that's true whether
the device is attached to vfio-pci or some other driver.  Maybe we
should prevent that, but I see that happening by deprecating KVM
assignment and eventually disabling and removing it.  Thanks,

Alex