Re: [PATCH v7 0/3] support dirty restraint on vCPU

[Hyman Huang]

1.
Start vm with kernel+initrd.img with qemu command line as following:

[root@Hyman_server1 fast_qemu]# cat vm.sh
#!/bin/bash
/usr/bin/qemu-system-x86_64 \
    -display none -vga none \
    -name guest=simple_vm,debug-threads=on \
    -monitor stdio \
    -machine pc-i440fx-2.12 \
    -accel kvm,dirty-ring-size=65536 -cpu host \
    -kernel /home/work/fast_qemu/vmlinuz-5.13.0-rc4+ \
    -initrd /home/work/fast_qemu/initrd-stress.img \
    -append "noapic edd=off printk.time=1 noreplace-smp 
cgroup_disable=memory pci=noearly console=ttyS0 debug ramsize=1500 
ratio=1 sleep=1" \
    -chardev file,id=charserial0,path=/var/log/vm_console.log \
    -serial chardev:charserial0 \
    -qmp unix:/tmp/qmp-sock,server,nowait \
    -D /var/log/vm.log \
    --trace events=/home/work/fast_qemu/events \
    -m 4096 -smp 2 -device sga

2.
Enable the dirtylimit trace event which will output to /var/log/vm.log
[root@Hyman_server1 fast_qemu]# cat /home/work/fast_qemu/events
dirtylimit_state_init
dirtylimit_vcpu
dirtylimit_impose
dirtyrate_do_calculate_vcpu


3.
Connect the qmp server with low level qmp client and set-dirty-limit

[root@Hyman_server1 my_qemu]# python3.6 ./scripts/qmp/qmp-shell -v -p 
/tmp/qmp-sock 


Welcome to the QMP low-level shell!
Connected to QEMU 6.1.92

(QEMU) set-dirty-limit cpu-index=1 dirty-rate=400 



{
    "arguments": {
        "cpu-index": 1,
        "dirty-rate": 400
    },
    "execute": "set-dirty-limit"
}

4.
observe the vcpu current dirty rate and quota dirty rate...

[root@Hyman_server1 ~]# tail -f /var/log/vm.log
dirtylimit_state_init dirtylimit state init: max cpus 2
dirtylimit_vcpu CPU[1] set quota dirtylimit 400
dirtylimit_impose CPU[1] impose dirtylimit: quota 400, current 0, 
percentage 0
dirtyrate_do_calculate_vcpu vcpu[0]: 1075 MB/s
dirtyrate_do_calculate_vcpu vcpu[1]: 1061 MB/s
dirtylimit_impose CPU[1] impose dirtylimit: quota 400, current 1061, 
percentage 62
dirtyrate_do_calculate_vcpu vcpu[0]: 1133 MB/s
dirtyrate_do_calculate_vcpu vcpu[1]: 380 MB/s
dirtylimit_impose CPU[1] impose dirtylimit: quota 400, current 380, 
percentage 57
dirtyrate_do_calculate_vcpu vcpu[0]: 1227 MB/s
dirtyrate_do_calculate_vcpu vcpu[1]: 464 MB/s

We can observe that vcpu-1's dirtyrate is about 400MB/s with dirty page 
limit set and the vcpu-0 is not affected.

5.
observe the vm stress info...
[root@Hyman_server1 fast_qemu]# tail -f /var/log/vm_console.log
[    0.838051] Run /init as init process
[    0.839216]   with arguments:
[    0.840153]     /init
[    0.840882]   with environment:
[    0.841884]     HOME=/
[    0.842649]     TERM=linux
[    0.843478]     edd=off
[    0.844233]     ramsize=1500
[    0.845079]     ratio=1
[    0.845829]     sleep=1
/init (00001): INFO: RAM 1500 MiB across 2 CPUs, ratio 1, sleep 1 us
[    1.158011] random: init: uninitialized urandom read (4096 bytes read)
[    1.448205] random: init: uninitialized urandom read (4096 bytes read)
/init (00001): INFO: 1638282593684ms copied 1 GB in 00729ms
/init (00110): INFO: 1638282593964ms copied 1 GB in 00719ms
/init (00001): INFO: 1638282594405ms copied 1 GB in 00719ms
/init (00110): INFO: 1638282594677ms copied 1 GB in 00713ms
/init (00001): INFO: 1638282595093ms copied 1 GB in 00686ms
/init (00110): INFO: 1638282595339ms copied 1 GB in 00662ms
/init (00001): INFO: 1638282595764ms copied 1 GB in 00670m

PS: the kernel and initrd images comes from:

kernel image: vmlinuz-5.13.0-rc4+, normal centos vmlinuz copied from 
/boot directory

initrd.img: initrd-stress.img, only contains a stress binary, which 
compiled from qemu source tests/migration/stress.c and run as init
in vm.

you can view README.md file of my project 
"met"(https://github.com/newfriday/met) to compile the initrd-stress.img. :)

On 11/30/21 20:57, Peter Xu wrote:
> On Tue, Nov 30, 2021 at 06:28:10PM +0800, huangy81@chinatelecom.cn wrote:
> > From: Hyman Huang(黄勇) <huangy81@chinatelecom.cn>
> >
> > The patch [2/3] has not been touched so far. Any corrections and
> > suggetions are welcome.
>
> I played with it today, but the vcpu didn't got throttled as expected.
>
> What I did was starting two workload with 500mb/s, each pinned on one vcpu
> thread:
>
> [root@fedora ~]# pgrep -fa mig_mon
> 595 ./mig_mon mm_dirty 1000 500 sequential
> 604 ./mig_mon mm_dirty 1000 500 sequential
> [root@fedora ~]# taskset -pc 595
> pid 595's current affinity list: 2
> [root@fedora ~]# taskset -pc 604
> pid 604's current affinity list: 3
>
> Then start throttle with 100mb/s:
>
> (QEMU) set-dirty-limit cpu-index=3 dirty-rate=100
> {"return": {}}
> (QEMU) set-dirty-limit cpu-index=2 dirty-rate=100
> {"return": {}}
>
> I can see the workload dropped a tiny little bit (perhaps 500mb -> 499mb), then
> it keeps going..
>
> Further throttle won't work too:
>
> (QEMU) set-dirty-limit cpu-index=2 dirty-rate=10
> {"return": {}}
>
> Funnily, the ssh client got slowed down instead... :(
>
> Yong, how did you test it?

--
Best Regards
Hyman Huang(黄勇)