[Bug 1901440] [NEW] Instability in IVSHMEM after updating QEMU 4.2 -> 5.0

[Dave G]

Public bug reported:

Updating Ubuntu from 20.08 to 20.10 which updates QEMU from 4.2 to 5.0
results in the virtual machines freezing when the IVSHMEM interface is
active.  This workstation typically runs several windows 10 virtual
machines that are accessed locally:  two using the spice viewer and one
that uses an passthrough assigned GPU accessed through a viewer called
Looking Glass.  Looking Glass uses the IVSHMEM device interface to pass
captured frames from the windows virtual machine to the linux host for
display by a viewer application.

This workstation was 100% stable under Ubuntu 20.08 (QEMU 4.2).  It
handled a variety of heavy loads all day it never froze or crashed.  It
became unstable under Ubuntu 20.10 (QEMU 5.0), seemingly triggered by
high levels of SHM activity.  I was able to reliably reproduce the
problem when playing a video in the looking-glass vm while playing
another video in a spice vm.  Other scenarios would also trigger this
problem less reliably, but this video playback scenario would trigger it
after 3-5 minutes of playback.

The result of this new instability would manifest itself by all running
vms on the host freezing but the host was not visibly effected.  I could
find no warnings or errors in any relevant system or QEMU logs.  It
wasn't just spice, when I accessed the gpu-passthru vm via directly
assigned devices it was frozen, still outputting video of the last frame
before the crash.  All vms would have to be force-shutdown and the host
rebooted to regain vm functionality.  Just forcing shutdown and
restarting a vm would result in showing 'running' status but it would be
frozen and inaccessible until system reboot.

I suspect this is a QEMU host / kernel error for several reasons:
Having to reboot the host, insensitivity to VM changes including virtio-
win version, etc.  I suspect it's related to IVSHMEM due to the
correlation of the freeze to the looking-glass related activity.

This might be kernel / PCIe / power management related in some way.
While experimenting to troubleshoot this issue I was able to trigger the
error more quickly by disabling PCIe power management in the BIOS.

The system was 100% stable under QEMU 5.0 when not running the looking-
glass vm, quite stable when the looking-glass vm was idle or lightly
used, but appeared increasingly unstable as SHM activity increased.

Sorry if this is a bit anecdotal, this is my work machine and
unfortunately today I was forced to rollback restore to Ubuntu 20.08
(QEMU 4.2) from backup so I can work on Monday.  The system returned to
100% stability after returning to 20.08.

If requested I can restore the Ubuntu 20.10 snapshot to reproduce &
gather information as directed.

** Affects: qemu
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1901440

Title:
  Instability in IVSHMEM after updating QEMU 4.2 -> 5.0

Status in QEMU:
  New

Bug description:
  Updating Ubuntu from 20.08 to 20.10 which updates QEMU from 4.2 to 5.0
  results in the virtual machines freezing when the IVSHMEM interface is
  active.  This workstation typically runs several windows 10 virtual
  machines that are accessed locally:  two using the spice viewer and
  one that uses an passthrough assigned GPU accessed through a viewer
  called Looking Glass.  Looking Glass uses the IVSHMEM device interface
  to pass captured frames from the windows virtual machine to the linux
  host for display by a viewer application.

  This workstation was 100% stable under Ubuntu 20.08 (QEMU 4.2).  It
  handled a variety of heavy loads all day it never froze or crashed.
  It became unstable under Ubuntu 20.10 (QEMU 5.0), seemingly triggered
  by high levels of SHM activity.  I was able to reliably reproduce the
  problem when playing a video in the looking-glass vm while playing
  another video in a spice vm.  Other scenarios would also trigger this
  problem less reliably, but this video playback scenario would trigger
  it after 3-5 minutes of playback.

  The result of this new instability would manifest itself by all
  running vms on the host freezing but the host was not visibly
  effected.  I could find no warnings or errors in any relevant system
  or QEMU logs.  It wasn't just spice, when I accessed the gpu-passthru
  vm via directly assigned devices it was frozen, still outputting video
  of the last frame before the crash.  All vms would have to be force-
  shutdown and the host rebooted to regain vm functionality.  Just
  forcing shutdown and restarting a vm would result in showing 'running'
  status but it would be frozen and inaccessible until system reboot.

  I suspect this is a QEMU host / kernel error for several reasons:
  Having to reboot the host, insensitivity to VM changes including
  virtio-win version, etc.  I suspect it's related to IVSHMEM due to the
  correlation of the freeze to the looking-glass related activity.

  This might be kernel / PCIe / power management related in some way.
  While experimenting to troubleshoot this issue I was able to trigger
  the error more quickly by disabling PCIe power management in the BIOS.

  The system was 100% stable under QEMU 5.0 when not running the
  looking-glass vm, quite stable when the looking-glass vm was idle or
  lightly used, but appeared increasingly unstable as SHM activity
  increased.

  Sorry if this is a bit anecdotal, this is my work machine and
  unfortunately today I was forced to rollback restore to Ubuntu 20.08
  (QEMU 4.2) from backup so I can work on Monday.  The system returned
  to 100% stability after returning to 20.08.

  If requested I can restore the Ubuntu 20.10 snapshot to reproduce &
  gather information as directed.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1901440/+subscriptions