Re: [Qemu-devel] [PATCH for-2.10 0/3] qdev/vfio: defer DEVICE_DEL to avoid races with libvirt

[David Gibson]

On Tue, Oct 03, 2017 at 05:21:57PM -0500, Michael Roth wrote:
> Quoting Michael Roth (2017-07-26 20:30:52)
> > This series was motivated by the discussion in this thread:
> > 
> >   https://www.redhat.com/archives/libvir-list/2017-June/msg01370.html
> > 
> > The issue this series addresses is that when libvirt unplugs a VFIO PCI 
> > device,
> > it may attempt to bind the host device back to the host driver when QEMU 
> > emits
> > the DEVICE_DELETED event for the corresponding vfio-pci device. However, the
> > VFIO group FD is not actually cleaned up until vfio-pci device is 
> > *finalized*
> > by QEMU, whereas the event is emitted earlier during device_unparent.
> > Depending on the host device and how long certain operations like resetting 
> > the
> > device might take, this can in result in libvirt trying to rebind the device
> > back to the host while it is still in use by VFIO, leading to host crashes 
> > or
> > other unexpected behavior.
> > 
> > In particular, Mellanox CX4 adapters on PowerNV hosts might not be fully
> > quiesced by vfio-pci's finalize() routine until up to 6s after the
> > DEVICE_DELETED was emitted, leading to detach-device on the libvirt side 
> > pretty
> > much always crashing the host.
> > 
> > Implementing this change requires 2 prereqs to ensure the same information 
> > is
> > available when the DEVICE_DELETED is finally emitted:
> > 
> > 1) Storing the path in the composition patch, which is addressed by PATCH 1,
> >    which was plucked from another pending series from Greg Kurz:
> > 
> >    https://lists.gnu.org/archive/html/qemu-devel/2017-07/msg07922.html
> > 
> >    since we are now "disconnected" at the time the event is emitted, and
> > 
> > 2) Deferring qemu_opts_del of the DeviceState->QemuOpts till finalize, since
> >    that is where DeviceState->id is stored. This was actually how it was
> >    done in the past, so PATCH 2 simply reverts the change which moved it to
> >    device_unparent.
> > 
> > From there it's just a mechanical move of the event from device_unparent to
> > device_finalize.
> 
> Ping.

I think it got forgotten amongst the rush to get v2.10 out.

> The situation has changed somewhat since original posting as Alex now
> has a fix on the kernel side for the VFIO issue noted above:
> 
>   
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=6586b561a91cd80a91c8f107ed0d144feb3eadc2
> 
> However, I think this series would still be useful for addressing the
> issue for hosts using older kernels, and there seems to be general
> interest from the libvirt side in aligning "DEVICE_DELETED" events
> with the notion that QEMU is completely finished with a device.
> 
> Patch 1/3 is also needed for another series that Greg is working on
> and I don't want to hold that up for this series, so if it's
> preferred that we just post that patch separately in the meantime
> please let me know.

Yeah, now that I've understood the issue properly, I also think this
is a good idea.  I recommend reposting.


-- 
David Gibson                    | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au  | minimalist, thank you.  NOT _the_ _other_
                                | _way_ _around_!
http://www.ozlabs.org/~dgibson

signature.asc
Description: PGP signature