Re: [Qemu-devel] [BUG] Redhat-6.4_64bit-guest kernel panic with cpu-passthrough and guest numa

[Gonglei (Arei)]

Cc'ing Paolo and Benoît.


Best regards,
-Gonglei


> -----Original Message-----
> From: Gonglei (Arei)
> Sent: Thursday, November 27, 2014 8:58 PM
> To: address@hidden
> Subject: [BUG] Redhat-6.4_64bit-guest kernel panic with cpu-passthrough and
> guest numa
> 
> Hi,
> 
> Running a redhat-6.4-64bit (kernel 2.6.32-358.el6.x86_64) or elder guest on
> qemu-2.1, with kvm enabled and -cpu host, non default cpu-topology and guest
> numa
> I'm seeing a reliable kernel panic from the guest shortly after boot. It is
> happening in
> find_busiest_group().
> 
> We also found it happend since commit
> 787aaf5703a702094f395db6795e74230282cd62 by git bisect.
> 
> The reproducer:
> 
> (1) full qemu cmd line:
> qemu-system-x86_64 -machine pc-i440fx-2.1,accel=kvm,usb=off \
> -cpu host -m 16384 \
> -smp 16,sockets=2,cores=4,threads=2 \
> -object memory-backend-ram,size=8192M,id=ram-node0 \
> -numa node,nodeid=0,cpus=0-7,memdev=ram-node0 \
> -object memory-backend-ram,size=8192M,id=ram-node1 \
> -numa node,nodeid=1,cpus=8-15,memdev=ram-node1 \
> -boot c -drive file=/data/wxin/vm/redhat_6.4_64 \
> -vnc 0.0.0.0:0 -device
> cirrus-vga,id=video0,vgamem_mb=8,bus=pci.0,addr=0x1.0x4 \
> -msg timestamp=on
> 
> (2)the guest kernel messages:
> 
> divide error: 0000 [#1] SMP
> last sysfs file:
> CPU 0
> Modules linked in:
> 
> Pid: 1, comm: swapper Not tainted 2.6.32-358.el6.x86_64 #1 QEMU Standard
> PC (i440FX + PIIX, 1996)
> RIP: 0010:[<ffffffff81059a9c>]  [<ffffffff81059a9c>]
> find_busiest_group+0x55c/0x9f0
> RSP: 0018:ffff88023c85f9e0  EFLAGS: 00010046
> RAX: 0000000000100000 RBX: ffff88023c85fbdc RCX: 0000000000000000
> RDX: 0000000000000000 RSI: 0000000000000010 RDI: 0000000000000010
> RBP: ffff88023c85fb50 R08: ffff88023ca16c10 R09: 0000000000000000
> R10: 0000000000000001 R11: 0000000000000000 R12: 00000000ffffff01
> R13: 0000000000016700 R14: ffffffffffffffff R15: 0000000000000000
> FS:  0000000000000000(0000) GS:ffff880028200000(0000)
> knlGS:0000000000000000
> CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
> CR2: 0000000000000000 CR3: 0000000001a85000 CR4: 00000000000407f0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Process swapper (pid: 1, threadinfo ffff88023c85e000, task ffff88043d27c040)
> Stack:
>  ffff88023c85faf0 ffff88023c85fa60 ffff88023c85fbc8 0000000200000000
> <d> 0000000100000000 ffff880028210b60 0000000100000001
> 0000000000000008
> <d> 0000000000016700 0000000000016700 ffff88023ca16c00
> 0000000000016700
> Call Trace:
>  [<ffffffff8150da2a>] thread_return+0x398/0x76e
>  [<ffffffff8150e555>] schedule_timeout+0x215/0x2e0
>  [<ffffffff81065905>] ? enqueue_entity+0x125/0x410
>  [<ffffffff8150e1d3>] wait_for_common+0x123/0x180
>  [<ffffffff81063310>] ? default_wake_function+0x0/0x20
>  [<ffffffff8150e2ed>] wait_for_completion+0x1d/0x20
>  [<ffffffff81096a89>] kthread_create+0x99/0x120
>  [<ffffffff81090950>] ? worker_thread+0x0/0x2a0
>  [<ffffffff81167769>] ? alternate_node_alloc+0xc9/0xe0
>  [<ffffffff810908d9>] create_workqueue_thread+0x59/0xd0
>  [<ffffffff8150ebce>] ? mutex_lock+0x1e/0x50
>  [<ffffffff810911bd>] __create_workqueue_key+0x14d/0x200
>  [<ffffffff81c47233>] init_workqueues+0x9f/0xb1
>  [<ffffffff81c2788c>] kernel_init+0x25e/0x2fe
>  [<ffffffff8100c0ca>] child_rip+0xa/0x20
>  [<ffffffff81c2762e>] ? kernel_init+0x0/0x2fe
>  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
> Code: 8b b5 b0 fe ff ff 48 8b bd b8 fe ff ff e8 9d 85 ff ff 0f 1f 44 00 00 48 
> 8b 95 e0
> fe ff ff 48 8b 45 a8 8b 4a 08 48 c1 e0 0a 31 d2 <48> f7 f1 48 8b 4d b0 48 89 
> 45 a0
> 31 c0 48 85 c9 74 0c 48 8b 45
> RIP  [<ffffffff81059a9c>] find_busiest_group+0x55c/0x9f0
>  RSP <ffff88023c85f9e0>
> divide error: 0000 [#2]
> ---[ end trace d7d20afc6dd05e71 ]---
> Kernel panic - not syncing: Fatal exception
> Pid: 1, comm: swapper Tainted: G      D    ---------------
> 2.6.32-358.el6.x86_64 #1
> Call Trace:
>  [<ffffffff8150cfc8>] ? panic+0xa7/0x16f
>  [<ffffffff815111f4>] ? oops_end+0xe4/0x100
>  [<ffffffff8100f19b>] ? die+0x5b/0x90
>  [<ffffffff81510a34>] ? do_trap+0xc4/0x160
>  [<ffffffff8100cf7f>] ? do_divide_error+0x8f/0xb0
>  [<ffffffff81059a9c>] ? find_busiest_group+0x55c/0x9f0
>  [<ffffffff8113b3a9>] ? zone_statistics+0x99/0xc0
>  [<ffffffff8100bdfb>] ? divide_error+0x1b/0x20
>  [<ffffffff81059a9c>] ? find_busiest_group+0x55c/0x9f0
>  [<ffffffff8150da2a>] ? thread_return+0x398/0x76e
>  [<ffffffff8150e555>] ? schedule_timeout+0x215/0x2e0
>  [<ffffffff81065905>] ? enqueue_entity+0x125/0x410
>  [<ffffffff8150e1d3>] ? wait_for_common+0x123/0x180
>  [<ffffffff81063310>] ? default_wake_function+0x0/0x20
>  [<ffffffff8150e2ed>] ? wait_for_completion+0x1d/0x20
>  [<ffffffff81096a89>] ? kthread_create+0x99/0x120
>  [<ffffffff81090950>] ? worker_thread+0x0/0x2a0
>  [<ffffffff81167769>] ? alternate_node_alloc+0xc9/0xe0
>  [<ffffffff810908d9>] ? create_workqueue_thread+0x59/0xd0
>  [<ffffffff8150ebce>] ? mutex_lock+0x1e/0x50
>  [<ffffffff810911bd>] ? __create_workqueue_key+0x14d/0x200
>  [<ffffffff81c47233>] ? init_workqueues+0x9f/0xb1
>  [<ffffffff81c2788c>] ? kernel_init+0x25e/0x2fe
>  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
>  [<ffffffff81c2762e>] ? kernel_init+0x0/0x2fe
>  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
> 
> ---
> (3)host info
> 
> /proc/cpuinfo on the host has 16 of these:
> 
> processor       : 15
> vendor_id       : GenuineIntel
> cpu family      : 6
> model           : 45
> model name      : Intel(R) Xeon(R) CPU E5-2643 0 @ 3.30GHz
> stepping        : 7
> microcode       : 1803
> cpu MHz         : 3301.000
> cache size      : 10240 KB
> physical id     : 1
> siblings        : 8
> core id         : 3
> cpu cores       : 4
> apicid          : 39
> initial apicid  : 39
> fpu             : yes
> fpu_exception   : yes
> cpuid level     : 13
> wp              : yes
> flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
> cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx
> pdpe1gb
> rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology
> nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx
> est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt
> tsc_deadline_timer aes xsave avx lahf_lm ida arat epb xsaveopt pln pts
> dtherm tpr_shadow vnmi flexpriority ept vpid
> bogomips        : 6599.83
> clflush size    : 64
> cache_alignment : 64
> address sizes   : 46 bits physical, 48 bits virtual
> power management:
> 
> 
> host numa topo:
> 
> node 0 cpus: 0 1 2 3 8 9 10 11
> node 0 size: 40936 MB
> node 0 free: 39625 MB
> node 1 cpus: 4 5 6 7 12 13 14 15
> node 1 size: 40960 MB
> node 1 free: 39876 MB
> node distances:
> node   0   1
>   0:  10  21
>   1:  21  10
> 
> (4) With "sched_debug loglevel=8" kernel parameter command line,
> you can see follow error log(those "ERROR"s):
> 
> CPU0 attaching sched-domain:
>  domain 0: span 0-15 level MC
>   groups: 0 (cpu_power = 1023) 1 2 3 4 5 6 7 8 9 10 (cpu_power = 1023) 11 12
> 13 14 15
> ERROR: parent span is not a superset of domain->span
>   domain 1: span 0-7 level CPU
> ERROR: domain->groups does not contain CPU0
>    groups: 8-15 (cpu_power = 16382)
> ERROR: groups don't span domain->span
>    domain 2: span 0-15 level NODE
>     groups:
> ERROR: domain->cpu_power not set
> 
> Any comments and help will be appreciated!
> 
> Best regards,
> -Gonglei