[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Slowness with multi-thread TCG?

From: Frederic Barrat
Subject: Slowness with multi-thread TCG?
Date: Mon, 27 Jun 2022 20:25:35 +0200
User-agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.10.0

[ Resending as it was meant for the qemu-ppc list ]


I've been looking at why our qemu powernv model is so slow when booting a compressed linux kernel, using multiple vcpus and multi-thread tcg. With only one vcpu, the decompression time of the kernel is what it is, but when using multiple vcpus, the decompression is actually slower. And worse: it degrades very fast with the number of vcpus!

Rough measurement of the decompression time on a x86 laptop with multi-thread tcg and using the qemu powernv10 machine:
1 vcpu => 15 seconds
2 vcpus => 45 seconds
4 vcpus => 1 min 30 seconds

Looking in details, when the firmware (skiboot) hands over execution to the linux kernel, there's one main thread entering some bootstrap code and running the kernel decompression algorithm. All the other secondary threads are left spinning in skiboot (1 thread per vpcu). So on paper, with multi-thread tcg and assuming the system has enough available physical cpus, I would expect the decompression to hog one physical cpu and the time needed to be constant, no matter the number of vpcus.

All the secondary threads are left spinning in code like this:

        for (;;) {
                if (cpu_check_jobs(cpu))  // reading cpu-local data
                if (reconfigure_idle)     // global variable

The barrier is to force reading the memory with each iteration. It's defined as:

  asm volatile("" : : : "memory");

Some time later, the main thread in the linux kernel will get the secondary threads out of that loop by posting a job.

My first thought was that the translation of that code through tcg was somehow causing some abnormally slow behavior, maybe due to some non-obvious contention between the threads. However, if I send the threads spinning forever with simply:

    for (;;) ;

supposedly removing any contention, then the decompression time is the same.

Ironically, the behavior seen with single thread tcg is what I would expect: 1 thread decompressing in 15 seconds, all the other threads spinning for that same amount of time, all sharing the same physical cpu, so it all adds up nicely: I see 60 seconds decompression time with 4 vcpus (4x15). Which means multi-thread tcg is slower by quite a bit. And single thread tcg hogs one physical cpu of the laptop vs. 4 physical cpus for the slower multi-thread tcg.

Does anybody have an idea of what might happen or have suggestion to keep investigating?
Thanks for your help!


reply via email to

[Prev in Thread] Current Thread [Next in Thread]