Re: [Qemu-devel] virtio-blk and virtio-scsi performance comparison

[Fam Zheng]

On Wed, 04/15 11:17, Konstantin Krotov wrote:
> Hello list!
> 
> I performed tests with fio and obtained results:
> 
> *** virtio-scsi with cache=none, io=threads, blok device is md-device from
> mdadm raid1, random r/w, 32 thread from guest (debian, kernel 3.16):
> 
> fio fio1
> readtest: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio,
> iodepth=32
> fio-2.1.11
> Starting 1 process
> Jobs: 1 (f=1): [m(1)] [100.0% done] [126.2MB/125.1MB/0KB /s] [32.3K/32.3K/0
> iops] [eta 00m:00s]
> readtest: (groupid=0, jobs=1): err= 0: pid=707: Wed Apr  8 07:35:01 2015
>   read : io=5117.4MB, bw=125830KB/s, iops=31457, runt= 41645msec
>     slat (usec): min=4, max=343, avg=11.45, stdev=10.24
>     clat (usec): min=104, max=16667, avg=484.09, stdev=121.96
>      lat (usec): min=112, max=16672, avg=495.90, stdev=123.67
>     clat percentiles (usec):
>      |  1.00th=[  302],  5.00th=[  346], 10.00th=[  374], 20.00th=[  406],
>      | 30.00th=[  426], 40.00th=[  446], 50.00th=[  462], 60.00th=[  482],
>      | 70.00th=[  506], 80.00th=[  540], 90.00th=[  596], 95.00th=[  732],
>      | 99.00th=[  948], 99.50th=[  996], 99.90th=[ 1176], 99.95th=[ 1240],
>      | 99.99th=[ 1384]
>     bw (KB  /s): min=67392, max=135216, per=99.99%, avg=125813.01,
> stdev=12524.05
>   write: io=5114.7MB, bw=125763KB/s, iops=31440, runt= 41645msec
>     slat (usec): min=4, max=388, avg=11.85, stdev=10.47
>     clat (usec): min=147, max=8968, avg=505.23, stdev=127.40
>      lat (usec): min=155, max=8973, avg=517.45, stdev=128.97
>     clat percentiles (usec):
>      |  1.00th=[  334],  5.00th=[  370], 10.00th=[  394], 20.00th=[  426],
>      | 30.00th=[  446], 40.00th=[  462], 50.00th=[  478], 60.00th=[  498],
>      | 70.00th=[  524], 80.00th=[  556], 90.00th=[  628], 95.00th=[  756],
>      | 99.00th=[  988], 99.50th=[ 1064], 99.90th=[ 1288], 99.95th=[ 1368],
>      | 99.99th=[ 2224]
>     bw (KB  /s): min=67904, max=136384, per=99.99%, avg=125746.89,
> stdev=12449.56
>     lat (usec) : 250=0.05%, 500=64.27%, 750=30.80%, 1000=4.20%
>     lat (msec) : 2=0.67%, 4=0.01%, 10=0.01%, 20=0.01%
>   cpu          : usr=18.03%, sys=76.42%, ctx=26617, majf=0, minf=7
>   IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
> >=64=0.0%
>      submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.0%
>      complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
> >=64=0.0%
>      issued    : total=r=1310044/w=1309348/d=0, short=r=0/w=0/d=0
>      latency   : target=0, window=0, percentile=100.00%, depth=32
> 
> Run status group 0 (all jobs):
>    READ: io=5117.4MB, aggrb=125829KB/s, minb=125829KB/s, maxb=125829KB/s,
> mint=41645msec, maxt=41645msec
>   WRITE: io=5114.7MB, aggrb=125762KB/s, minb=125762KB/s, maxb=125762KB/s,
> mint=41645msec, maxt=41645msec
> 
> Disk stats (read/write):
>   sda: ios=1302885/1302192, merge=55/0, ticks=281040/321660,
> in_queue=601264, util=99.29%
> 
> 
> same guest,
> *** virtio-blk with cache=none, io=threads, blok device is md-device from
> mdadm raid1, random r/w, 32 thread from guest (debian, kernel 3.16):
> 
>  fio fio1
> readtest: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio,
> iodepth=32
> fio-2.1.11
> Starting 1 process
> Jobs: 1 (f=1): [m(1)] [100.0% done] [123.7MB/123.3MB/0KB /s] [31.7K/31.6K/0
> iops] [eta 00m:00s]
> readtest: (groupid=0, jobs=1): err= 0: pid=810: Wed Apr  8 07:26:37 2015
>   read : io=5117.4MB, bw=148208KB/s, iops=37051, runt= 35357msec
>     slat (usec): min=2, max=2513, avg= 7.27, stdev=10.28
>     clat (usec): min=104, max=10716, avg=382.30, stdev=113.38
>      lat (usec): min=108, max=10719, avg=389.94, stdev=115.48
>     clat percentiles (usec):
>      |  1.00th=[  215],  5.00th=[  249], 10.00th=[  270], 20.00th=[  298],
>      | 30.00th=[  318], 40.00th=[  338], 50.00th=[  358], 60.00th=[  386],
>      | 70.00th=[  418], 80.00th=[  462], 90.00th=[  516], 95.00th=[  572],
>      | 99.00th=[  756], 99.50th=[  820], 99.90th=[  996], 99.95th=[ 1176],
>      | 99.99th=[ 2256]
>     bw (KB  /s): min=119296, max=165456, per=99.94%, avg=148124.33,
> stdev=11834.17
>   write: io=5114.7MB, bw=148129KB/s, iops=37032, runt= 35357msec
>     slat (usec): min=2, max=2851, avg= 7.55, stdev=10.53
>     clat (usec): min=172, max=11080, avg=461.92, stdev=137.02
>      lat (usec): min=178, max=11086, avg=469.86, stdev=138.05
>     clat percentiles (usec):
>      |  1.00th=[  278],  5.00th=[  318], 10.00th=[  338], 20.00th=[  366],
>      | 30.00th=[  390], 40.00th=[  414], 50.00th=[  438], 60.00th=[  466],
>      | 70.00th=[  494], 80.00th=[  532], 90.00th=[  604], 95.00th=[  716],
>      | 99.00th=[  900], 99.50th=[  980], 99.90th=[ 1336], 99.95th=[ 1704],
>      | 99.99th=[ 3408]
>     bw (KB  /s): min=119656, max=166680, per=99.93%, avg=148029.21,
> stdev=11824.30
>     lat (usec) : 250=2.71%, 500=77.22%, 750=17.60%, 1000=2.21%
>     lat (msec) : 2=0.24%, 4=0.02%, 10=0.01%, 20=0.01%
>   cpu          : usr=27.92%, sys=55.44%, ctx=91283, majf=0, minf=7
>   IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
> >=64=0.0%
>      submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.0%
>      complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
> >=64=0.0%
>      issued    : total=r=1310044/w=1309348/d=0, short=r=0/w=0/d=0
>      latency   : target=0, window=0, percentile=100.00%, depth=32
> 
> Run status group 0 (all jobs):
>    READ: io=5117.4MB, aggrb=148207KB/s, minb=148207KB/s, maxb=148207KB/s,
> mint=35357msec, maxt=35357msec
>   WRITE: io=5114.7MB, aggrb=148128KB/s, minb=148128KB/s, maxb=148128KB/s,
> mint=35357msec, maxt=35357msec
> 
> Disk stats (read/write):
>   vdb: ios=1302512/1301780, merge=0/0, ticks=294828/407184, in_queue=701380,
> util=99.51%
> 
> In my tests virtio-scsi shows worse results than virtio-blk.
> Host kernel 3.19-3, qemu-system-x86_64 -version
> QEMU emulator version 2.0.0.


Hi Konstantin,

Thanks for sharing your test result with us!

It is not surprising that virtio-blk performs better in such a test. It has a
much smaller command set, which results in both a simpler device model and
probably a simpler guest driver.

virtio-scsi, on the other hand, provides more features and means to be more
scalable (you won't need to painfully mess with pci bridges to attach 1000
disks).

Anyway, we are working on improving virtio-scsi performance, although it's
theoretically impossible to make it faster or even equally fast.

Regarding your test, I think with current code base, it generally performs
better if you use io=native. Have you compared that?

Fam