[Qemu-discuss] QEMU caching modes

[Jan Schermer]

Hi,
I am investigating a weird caching behaviour of QEMU on Ubuntu 14.04 and most 
likely even 16.04.

Our VMs are using DRBD devices as drives, like so:

file=/dev/drbd/by-res/3fa0da3f-9607-4068-b517-349c05157d9a/0,format=raw,if=none,id=drive-virtio-disk0,serial=1f59fd16-72e6-45f8-b047-e8a876bee5ff,cache=none
(we tried block instead of file, didn't make an observable difference)

I'm trying to understand when/what IO is synchronous and why, and so far what 
I'm seeing I can only describe as "weird".

With cache=none, all IO should pass from the QEMU process to the device as-is. 
Right?
If so, that's not what I see - I see (with btrace) write IO as asynchronous (W) 
in the guest, but synchronous (WS) on the host.
I tried cache=writeback, and sometimes it seems to work (IO on the host is 
async) but sometimes not. Sometimes it starts async then turns synchronous 
forever. Looks random, but there must be something that I'm missing.

cache=unsafe makes all IO reliably async, but that's not what I want, I like my 
data :-)

Guest uses virtio-blk, and I even tried virtio-scsi.
I tried disabling flushes in guest (in case virtio-blk was messing stuff up) 
with "echo temporary write through > cache_type" on the virtual drive, and it 
made no difference.

Any ideas what might be going on? Why does QEMU turn all IO into synchronous? 
In the past when I used Ceph RBD, it only turned writeback on if it observed 
flush from the guest, is there something similiar in general which  prevents 
QEMu from trusting the guest to flush properly?

Also, I believe in the past (~one year back) this worked correctly with the 
same setup, but I can't reproduce it anymore...

Host is Ubuntu 14.04, or Ubuntu 16.04 (makes no difference), kernel 4.4.0
Guest is Ubuntu 14.04, kernel 3.13 or 4.4.0.
IO is generated either using dd or fio, sync/async/direct makes no difference 
when this happens.
I'm generating the IO on a block device, not through a filesystem, if that 
makes any difference (it shouldn't).
I observe IO in the guest and on the host with btrace.
Host: WS
Guest: W
Me: WTH

I care the most about O_DIRECT writes/workloads performing well (which doesn't 
imply O_SYNC), and that's why I started investigating it in the first place. 
There are enterprise-class SSDs used in a ZFS pool below DRBD, so it is very 
fast in general, but latency suffers greatly when IO is synchronous.

Thanks

Jan