Re: Unpredictable performance degradation in QEMU KVMs

[Frantisek Rysanek]

On 5 Oct 2021 at 18:58, Parnell Springmeyer wrote:
> 
> Hi, we use QEMU VMs for running our integration testing 
> infrastructure and have run into a very difficult to debug problem: 
> occasionally we will see a severe performance degradation in some of 
> our QEMU VMs. 
> 
If memory serves, QEMU guests appear to run as processes in the Linux 
host instance. I'm not "in the know enough" to tell you, how much is 
possibly happening under the hood in the kernel support side of 
things, which is potentially not well described by that superficial 
abstraction visible in "top".

Esoteric issues aside (CPU arch incompatibilities between host and 
guest), have you tried inspecting what the load looks like, in the 
guest and in the host OS instance? What does "top" show? With CPU 
cores expanded? (press "1") 
Have you tried "latencytop" by any chance?

Are you sure this is a CPU performance/emulation issue?
What storage are your VM's using? Could storage be the bottleneck?
Isn't the observed "sluggishness" storage-io-bound, rather than CPU 
bound? Can you tell the difference? (Heck... apologies, that's 
probably a series of dumb questions to someone @arista.com)

Stuff can get sluggish when IRQ's don't work right. Any signs of that 
in the guest instance? Interesting messages in dmesg, interesting 
numbers in /proc/interrupts?

CPU arch emulation issues (guest vs. host) might also be an issue. If 
you specify a different CPU core for the guest than the host actually 
has, you may get some fringe parts of the instruction set, even 
within the x86_64 family, that needs to be tediously emulated for the 
guest instance... also, I'd hazard a guess 32bit vs. 64bit *might* 
play a role, albeit marginal. I have fond memories of the 387 math 
co-processor emulation (and its effects on program runtime), but 
that's a *long* time ago :-)

I've seen EXT3 and EXT4 hang for no apparent reason, on bare metal, 
under heavy IOps stress. CPU consumption at 0%, disk IOps at pure 0, 
but the filesystem would block forever in a standstill. If I recall 
correctly, I used Bonnie++ to generate that kind of stress 
reproducibly, against fast block storage (HW RAID back then). There 
was no QEMU in the game.

= feel free to add some juicy detail for us to ponder :-)

Frank