On Fri, Dec 03, 2021 at 09:39:46AM +0800, huangy81@chinatelecom.cn wrote:
+static uint64_t dirtylimit_pct(unsigned int last_pct,
+ uint64_t quota,
+ uint64_t current)
+{
+ uint64_t limit_pct = 0;
+ RestrainPolicy policy;
+ bool mitigate = (quota > current) ? true : false;
+
+ if (mitigate && ((current == 0) ||
+ (last_pct <= DIRTYLIMIT_THROTTLE_SLIGHT_STEP_SIZE))) {
+ return 0;
+ }
+
+ policy = dirtylimit_policy(last_pct, quota, current);
+ switch (policy) {
+ case RESTRAIN_SLIGHT:
+ /* [90, 99] */
+ if (mitigate) {
+ limit_pct =
+ last_pct - DIRTYLIMIT_THROTTLE_SLIGHT_STEP_SIZE;
+ } else {
+ limit_pct =
+ last_pct + DIRTYLIMIT_THROTTLE_SLIGHT_STEP_SIZE;
+
+ limit_pct = MIN(limit_pct, CPU_THROTTLE_PCT_MAX);
+ }
+ break;
+ case RESTRAIN_HEAVY:
+ /* [75, 90) */
+ if (mitigate) {
+ limit_pct =
+ last_pct - DIRTYLIMIT_THROTTLE_HEAVY_STEP_SIZE;
+ } else {
+ limit_pct =
+ last_pct + DIRTYLIMIT_THROTTLE_HEAVY_STEP_SIZE;
+
+ limit_pct = MIN(limit_pct,
+ DIRTYLIMIT_THROTTLE_SLIGHT_WATERMARK);
+ }
+ break;
+ case RESTRAIN_RATIO:
+ /* [0, 75) */
+ if (mitigate) {
+ if (last_pct <= (((quota - current) * 100 / quota))) {
+ limit_pct = 0;
+ } else {
+ limit_pct = last_pct -
+ ((quota - current) * 100 / quota);
+ limit_pct = MAX(limit_pct, CPU_THROTTLE_PCT_MIN);
+ }
+ } else {
+ limit_pct = last_pct +
+ ((current - quota) * 100 / current);
+
+ limit_pct = MIN(limit_pct,
+ DIRTYLIMIT_THROTTLE_HEAVY_WATERMARK);
+ }
+ break;
+ case RESTRAIN_KEEP:
+ default:
+ limit_pct = last_pct;
+ break;
+ }
+
+ return limit_pct;
+}
+
+static void *dirtylimit_thread(void *opaque)
+{
+ int cpu_index = *(int *)opaque;
+ uint64_t quota_dirtyrate, current_dirtyrate;
+ unsigned int last_pct = 0;
+ unsigned int pct = 0;
+
+ rcu_register_thread();
+
+ quota_dirtyrate = dirtylimit_quota(cpu_index);
+ current_dirtyrate = dirtylimit_current(cpu_index);
+
+ pct = dirtylimit_init_pct(quota_dirtyrate, current_dirtyrate);
+
+ do {
+ trace_dirtylimit_impose(cpu_index,
+ quota_dirtyrate, current_dirtyrate, pct);
+
+ last_pct = pct;
+ if (pct == 0) {
+ sleep(DIRTYLIMIT_CALC_PERIOD_TIME_S);
+ } else {
+ dirtylimit_check(cpu_index, pct);
+ }
+
+ quota_dirtyrate = dirtylimit_quota(cpu_index);
+ current_dirtyrate = dirtylimit_current(cpu_index);
+
+ pct = dirtylimit_pct(last_pct, quota_dirtyrate, current_dirtyrate);
So what I had in mind is we can start with an extremely simple version of
negative feedback system. Say, firstly each vcpu will have a simple number to
sleep for some interval (this is ugly code, but just show what I meant..):
===============
diff --git a/accel/kvm/kvm-all.c b/accel/kvm/kvm-all.c
index eecd8031cf..c320fd190f 100644
--- a/accel/kvm/kvm-all.c
+++ b/accel/kvm/kvm-all.c
@@ -2932,6 +2932,8 @@ int kvm_cpu_exec(CPUState *cpu)
trace_kvm_dirty_ring_full(cpu->cpu_index);
qemu_mutex_lock_iothread();
kvm_dirty_ring_reap(kvm_state);
+ if (dirtylimit_enabled(cpu->cpu_index) &&
cpu->throttle_us_per_full)
+ usleep(cpu->throttle_us_per_full);
qemu_mutex_unlock_iothread();
ret = 0;
break;
===============
I think this will have finer granularity when throttle (for 4096 ring size,
that's per-16MB operation) than current way where we inject per-vcpu async task
to sleep, like auto-converge.
Then we have the "black box" to tune this value with below input/output:
- Input: dirty rate information, same as current algo
- Output: increase/decrease of per-vcpu throttle_us_per_full above, and
that's all
We can do the sampling per-second, then we keep doing it: we can have 1 thread
doing per-second task collecting dirty rate information for all the vcpus, then
tune that throttle_us_per_full for each of them.
The simplest linear algorithm would be as simple as (for each vcpu):
if (quota < current)
throttle_us_per_full += SOMETHING;
if (throttle_us_per_full > MAX)
throttle_us_per_full = MAX;
else
throttle_us_per_full -= SOMETHING;
if (throttle_us_per_full < 0)
throttle_us_per_full = 0;
I think your algorithm is fine, but thoroughly review every single bit of it in
one shot will be challenging, and it's also hard to prove every bit of the
algorithm is helpful, as there're a lot of hand-made macros and state changes.
I actually tested the current algorithm of yours, the dirty rate fluctuates a
bit (when I specified 200MB/s, it can go into either a few tens of MB/s or
300MB/s, normally less), neither does it respond fast (the initial throtle from
500MB/s -> 200MB/s should need 1 minute or something), so it seems not ideal
anyway. In that case I prefer we start with simple.
So IMHO we can start with this simple scheme first then it'll start working
with much less line of codes, afaict. With that scheme ready in the 1st or
initial patches, it'll be easier to either apply any better algorithm
(e.g. your current one, if you're confident with that) or other things then
it'll be much easier to review too if you could consider split your patch like
that.
Normally per my knowledge for the need on migration, we could consider add an
integral algorithm into this linear algorithm that I said above, and it should
help us reach a very stable and constant state of throttling already. But
we'll need to try it out, as I never tried.
What do you think?