[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Qemu-devel] Disparity between host and guest CPU utilization during
From: |
Nagarajan, Padhu (HPE Storage) |
Subject: |
Re: [Qemu-devel] Disparity between host and guest CPU utilization during disk IO benchmark |
Date: |
Thu, 13 Jul 2017 23:34:56 +0000 |
Thanks Stefan. Couldn't get to this earlier. Did another run and took a diff of
/proc/interrupts before and after the run. It shows all the interrupts for
'virtio7-req.0' going to CPU1. I guess that explains the "CPU1/KVM" vcpu
utilization on the host.
34: 147 666085 0 0 PCI-MSI-edge
virtio7-req.0
The only remaining question is the high CPU utilization of the vCPU threads for
this workload. Even when I run a light fio workload (queue depth of 1 which
gives 8K IOPS), the vCPU threads are close to 100% utilization. Why is it high
and does it have an impact on guest code that could be executing on the same
CPU ?
fio command line: fio --time_based --ioengine=libaio --randrepeat=1 --direct=1
--invalidate=1 --verify=0 --offset=0 --verify_fatal=0 --group_reporting
--numjobs=1 --name=randread --rw=randread --blocksize=8K --iodepth=1
--runtime=60 --filename=/dev/vdb
qemu command line: qemu-system-x86_64 -L /usr/share/seabios/ -enable-kvm -name
node1,debug-threads=on -name node1 -S -machine pc-i440fx-2.8,accel=kvm,usb=off
-cpu SandyBridge -m 7680 -realtime mlock=off -smp 4,sockets=4,cores=1,threads=1
-object iothread,id=iothread1 -object iothread,id=iothread2 -object
iothread,id=iothread3 -object iothread,id=iothread4 -uuid XX -nographic
-no-user-config -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/node1fs.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew
-global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on
-device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device
lsi,id=scsi0,bus=pci.0,addr=0x6 -device
virtio-scsi-pci,id=scsi1,bus=pci.0,addr=0x7 -device
virtio-scsi-pci,id=scsi2,bus=pci.0,addr=0x8 -drive
file=rhel7.qcow2,if=none,id=drive-virtio-disk0,format=qcow2 -device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
-drive
file=/dev/sdc,if=none,id=drive-virtio-disk1,format=raw,cache=none,aio=native
-device
virtio-blk-pci,iothread=iothread1,scsi=off,bus=pci.0,addr=0x17,drive=drive-virtio-disk1,id=virtio-disk1
-drive
file=/dev/sdc,if=none,id=drive-scsi1-0-0-0,format=raw,cache=none,aio=native
-device
scsi-hd,bus=scsi1.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi1-0-0-0,id=scsi1-0-0-0
-netdev tap,fd=24,id=hostnet0,vhost=on,vhostfd=25 -device
virtio-net-pci,netdev=hostnet0,id=net0,mac=XXX,bus=pci.0,addr=0x2 -netdev
tap,fd=26,id=hostnet1,vhost=on,vhostfd=27 -device
virtio-net-pci,netdev=hostnet1,id=net1,mac=YYY,bus=pci.0,multifunction=on,addr=0x15
-netdev tap,fd=28,id=hostnet2,vhost=on,vhostfd=29 -device
virtio-net-pci,netdev=hostnet2,id=net2,mac=ZZZ,bus=pci.0,multifunction=on,addr=0x16
-chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 -msg timestamp=on
# qemu-system-x86_64 --version
QEMU emulator version 2.8.0(Debian 1:2.8+dfsg-3~bpo8+1)
Copyright (c) 2003-2016 Fabrice Bellard and the QEMU Project developers
Note that I had the same host block device (/dev/sdc in this case) exposed over
both virtio-scsi and virtio-blk to the guest VM for perf comparisons.
I see poor performance for 8K random reads inside the guest over both
virtio-scsi and virtio-blk, compared to the host performance. Let me open
another thread for that problem, but let me know if something obvious pops up
based on the qemu command line.
~Padhu.
-----Original Message-----
From: Stefan Hajnoczi [mailto:address@hidden
Sent: Tuesday, July 11, 2017 5:19 AM
To: Nagarajan, Padhu (HPE Storage) <address@hidden>
Cc: address@hidden
Subject: Re: [Qemu-devel] Disparity between host and guest CPU utilization
during disk IO benchmark
On Mon, Jul 10, 2017 at 05:27:15PM +0000, Nagarajan, Padhu (HPE Storage) wrote:
> Posted this in qemu-discuss and did not get a response. Hoping that someone
> here might be able to offer insights.
>
> I was running an 8K random-read fio benchmark inside the guest with
> iodepth=32. The device used inside the guest for the test was a virtio-blk
> device with iothread enabled, mapped on to a raw block device on the host.
> While this workload was running, I took a snapshot of the CPU utilization
> reported by the host and the guest. The guest had 4 cores. top inside guest
> shows 3 idle cores and one core being 74% utilized by fio (active on core 3).
> The host had 12 cores and three cores were completely consumed by three qemu
> threads. top inside host shows three qemu threads, each utilizing the CPU
> core to a near 100%. These threads are "CPU 1/KVM", "CPU 3/KVM" and "IO
> iothread1". The CPU utilization story on the host side is the same, even if I
> run a light fio workload inside the guest (for ex. iodepth=1).
>
> Why do I see two "CPU/KVM" threads occupying 100% CPU, even though only one
> core inside the guest is being utilized ? Note that I had 'accel=kvm' turned
> on for the guest.
fio might be submitting I/O requests on one vcpu and the completion
interrupts are processed on another vcpu.
To discuss further, please post:
1. Full QEMU command-line
2. Full fio command-line and job file (if applicable)
3. Output of cat /proc/interrupts inside the guest after running the
benchmark