[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [PATCH v2 0/4] NUMA: Apply socket-NUMA-node boundary for aarch64 and
Re: [PATCH v2 0/4] NUMA: Apply socket-NUMA-node boundary for aarch64 and RiscV machines
Sat, 25 Feb 2023 11:05:42 +1100
Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.2.0
On 2/25/23 1:20 AM, Igor Mammedov wrote:
On Fri, 24 Feb 2023 21:16:39 +1100
Gavin Shan <firstname.lastname@example.org> wrote:
On 2/24/23 8:26 PM, Daniel Henrique Barboza wrote:
On 2/24/23 04:09, Gavin Shan wrote:
On 2/24/23 12:18 AM, Daniel Henrique Barboza wrote:
On 2/23/23 05:13, Gavin Shan wrote:
For arm64 and RiscV architecture, the driver (/base/arch_topology.c) is
used to populate the CPU topology in the Linux guest. It's required that
the CPUs in one socket can't span mutiple NUMA nodes. Otherwise, the Linux
scheduling domain can't be sorted out, as the following warning message
indicates. To avoid the unexpected confusion, this series attempts to
rejects such kind of insane configurations.
-smp 6,maxcpus=6,sockets=2,clusters=1,cores=3,threads=1 \
-numa node,nodeid=0,cpus=0-1,memdev=ram0 \
-numa node,nodeid=1,cpus=2-3,memdev=ram1 \
-numa node,nodeid=2,cpus=4-5,memdev=ram2 \
And why is this a QEMU problem? This doesn't hurt ACPI.
Also, this restriction impacts breaks ARM guests in the wild that are running
non-Linux OSes. I don't see why we should impact use cases that has nothing to
do with Linux Kernel feelings about sockets - NUMA nodes exclusivity.
With above configuration, CPU-0/1/2 are put into socket-0-cluster-0 while
are put into socket-1-cluster-0, meaning CPU-2/3 belong to different socket and
cluster. However, CPU-2/3 are associated with NUMA node-1. In summary, multiple
CPUs in different clusters and sockets have been associated with one NUMA node.
If I'm correct, the configuration isn't sensible in a baremetal environment and
same Linux kernel is supposed to work well for baremetal and virtualized
So I think QEMU needs to emulate the topology as much as we can to match with
baremetal environment. It's the reason why I think it's a QEMU problem even it
doesn't hurt ACPI. As I said in the reply to Daniel P. Berrangé
in another thread, we may need to gurantee that the CPUs in one cluster can't be
split to multiple NUMA nodes, which matches with the baremetal environment, as I
Right, the restriction to have socket-NUMA-node or cluster-NUMA-node boundary
definitely break the configurations running in the wild.
What about a warning? If the user attempts to use an exotic NUMA configuration
like the one you mentioned we can print something like:
"Warning: NUMA topologies where a socket belongs to multiple NUMA nodes can cause
OSes like Linux to misbehave"
This would inform the user what might be going wrong in case Linux is
out on them and then the user is free to fix their topology (or the kernel). And
at the same time we wouldn't break existing stuff that happens to be working.
Yes, I think a warning message is more appropriate, so that users can fix their
irregular configurations and the existing configurations aren't disconnected.
It would be nice to get the agreements from Daniel P. Berrangé and Drew, before
I'm going to change the code and post next revision.
Honestly you (and libvirt as far as I recall) are using legacy options
to assign cpus to numa nodes.
With '-numa node,nodeid=0,cpus=0-1' you can't really be sure what/where
in topology those cpus are.
What you can do is to use '-numa cpu,...' option to assign socket/core/...
to numa node, ex:
"-numa cpu,node-id=1,socket-id=0 " or
"-numa cpu,node-id=0,socket-id=1,core-id=0 " or
to get your desired mapping.
The problem that's so far was stopping the later adoption by libvirt (Michal
is that values used by it are machine specific and to do it properly, for a
'-M x -smp ...' at least for the first time qemu should be started with-
-preconfig option and then user should query possible cpus for those values
and assign them to numa nodes via QMP.
The issue isn't related to the legacy or modern way to configure CPU-to-NUMA
Note that qtests/numa-test also use the legacy way. For example, the issue can
with the following command lines where the modern configuration is used:
-accel kvm -machine virt,gic-version=host \
-cpu host -smp 6,sockets=2,clusters=1,cores=3,threads=1 \
-m 768M,slots=16,maxmem=64G \
-object memory-backend-ram,id=mem0,size=256M \
-object memory-backend-ram,id=mem1,size=256M \
-object memory-backend-ram,id=mem2,size=256M \
-numa node,nodeid=0,memdev=mem0 \
-numa node,nodeid=1,memdev=mem1 \
-numa node,nodeid=2,memdev=mem2 \
-numa cpu,node-id=0,socket-id=0,cluster-id=0,core-id=0,thread-id=0 \
-numa cpu,node-id=0,socket-id=0,cluster-id=0,core-id=1,thread-id=0 \
-numa cpu,node-id=1,socket-id=0,cluster-id=0,core-id=2,thread-id=0 \
-numa cpu,node-id=1,socket-id=1,cluster-id=0,core-id=0,thread-id=0 \
-numa cpu,node-id=2,socket-id=1,cluster-id=0,core-id=1,thread-id=0 \
[ 0.022260] =============== sched_init_domains ===================
[ 0.022263] CPU-0: 36,56,0,-1 thread=0 core=0-2 cluster=0-2 llc=0
[ 0.022267] CPU-1: 36,56,1,-1 thread=1 core=0-2 cluster=0-2 llc=1
[ 0.022269] CPU-2: 36,56,2,-1 thread=2 core=0-2 cluster=0-2 llc=2
[ 0.022270] CPU-3: 136,156,3,-1 thread=3 core=3-5 cluster=3-5 llc=3
[ 0.022272] CPU-4: 136,156,4,-1 thread=4 core=3-5 cluster=3-5 llc=4
[ 0.022273] CPU-5: 136,156,5,-1 thread=5 core=3-5 cluster=3-5 llc=5
[ 0.022275] CPU-0: SMT=0 CLUSTER=0 MULTICORE=0-2 CPUMASK=0-1 0-1
[ 0.022277] CPU-1: SMT=1 CLUSTER=1 MULTICORE=0-2 CPUMASK=0-1 0-1
[ 0.022279] CPU-2: SMT=2 CLUSTER=0-2 MULTICORE=2-3 CPUMASK=2-3 2-3
[ 0.022281] CPU-3: SMT=3 CLUSTER=3-5 MULTICORE=2-3 CPUMASK=2-3 2-3
[ 0.022283] CPU-4: SMT=4 CLUSTER=4 MULTICORE=3-5 CPUMASK=4-5 4-5
[ 0.022284] CPU-5: SMT=5 CLUSTER=5 MULTICORE=3-5 CPUMASK=4-5 4-5
[ 0.022314] build_sched_domains: CPU-0 level-SMT
[ 0.022317] build_sched_domains: CPU-0 level-CLS
[ 0.022318] topology_span_sane: CPU-0 0 vs CPU-2 0-2
[ 0.022322] ------------[ cut here ]------------
[ 0.022323] WARNING: CPU: 0 PID: 1 at kernel/sched/topology.c:2275
btw: on x86 we also allow 'insane' configuration incl. those that do not
exist on baremetal and do not warn about it anyone (i.e. it's user's
responsibility to provide topology that specific guest OS could handle,
aka it's not QEMU business but upper layers). (I do occasionally try
introduce stricter checks in that are, but that gets push back more
often that not).
I'd do check only in case of a specific board where mapping is fixed
in specs of emulated machine, otherwise I wouldn't bother.
Right, x86 can handle the irregular configuration well and we didn't find
any triggered warning messages there. As the subject indicates, it's a
ARM64 or riscv specific issue. Unfortunately, I failed to find the requirement,
socket-to-NUMA-node or cluster-to-NUMA-node 1:1 mapping from specs. However,
sensible to split CPUs in one cluster to multiple NUMA nodes in a (ARM64 or
baremetal environment. QEMU needs to emulate the environment close to the
machine if we can.
In summary, a warning message when multiple CPUs in one cluster are split to
multiple NUMA nodes, as Daniel suggested, sounds reasonable to me. Please let me
know your thoughts, Igor :)
- Re: [PATCH v2 0/4] NUMA: Apply socket-NUMA-node boundary for aarch64 and RiscV machines, (continued)