Re: Multiple CPU for PPC virtual machine on x86_64 host

[BERBAR Florian]

Hello,

On 8/5/22 18:39, Daniel Henrique Barboza wrote:
> On 8/2/22 19:19, BERBAR Florian wrote:
> > Hello everyone,
> >
> > I am working on 32 bit PPC systems on a qemu-ppc based virtual system 
> > but I have not been able to boot my virtual machine with multiple CPUs.
> >
> > My host has the following versions:
> > - Linux version 5.10.0-15-amd64 (debian-kernel@lists.debian.org) 
> > (gcc-10 (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for 
> > Debian) 2.35.2) #1 SMP Debian 5.10.120-1 (2022-06-09)
> > - QEMU emulator version 5.2.0 (Debian 1:5.2+dfsg-11+deb11u2)
> > Copyright (c) 2003-2020 Fabrice Bellard and the QEMU Project developers
> >
> > First, I tried to set the -smp option with the number of CPUs with 
> > the following command but the default machine type (g3beige) does not 
> > support more than one core.
> >
> > $ qemu-system-ppc -m 2047M -smp 4 -hda -cdrom 
> > install-powerpc-minimal-20220122T062244Z.iso -boot d
> > qemu-system-ppc: Invalid SMP CPUs 4. The max CPUs supported by 
> > machine 'g3beige' is 1
> >
> >
> > Then I also tried to change the machine type based on the list in the 
> > help output
> >
> > $ qemu-system-ppc -M help
> > Supported machines are:
> > 40p                  IBM RS/6000 7020 (40p)
> > bamboo               bamboo
> > g3beige              Heathrow based PowerMAC (default)
> > mac99                Mac99 based PowerMAC
> > mpc8544ds            mpc8544ds
> > none                 empty machine
> > ppce500              generic paravirt e500 platform
> > ref405ep             ref405ep
> > sam460ex             aCube Sam460ex
> > taihu                taihu
> > virtex-ml507         Xilinx Virtex ML507 reference design
>
> The mpc8544ds machine is able to boot with 2 CPUs:
>
> qemu-system-ppc -m 1G -M mpc8544ds -kernel 
> ./buildroot/qemu_ppc_mpc8544ds-latest/vmlinux \
> -net nic,model=e1000 -net user -serial mon:stdio \
> -nographic -snapshot -smp 2
This virtual system configuration worked successfully on my host with 2 
CPUs and the linux kernel from Cedric Le Goater github. My system enters 
the buildroot environment and gives the buildroot login prompt as 
indicated in your message.

The same test with the official iso image of gentoo ppc does stopped on 
U-boot prompt.

$ qemu-system-ppc -m 1G -M mpc8544ds -cdrom 
install-powerpc-minimal-20220122T062244Z.iso -serial mon:stdio 
-nographic -smp 2 -boot d

U-Boot 2021.07 (Jul 06 2021 - 09:21:42 +0800)

CPU:   Unknown, Version: 0.0, (0x00000000)
Core:  e500, Version: 3.0, (0x80210030)
Clock Configuration:
       CPU0:400  MHz,
       CCB:400  MHz,
       DDR:200  MHz (400 MT/s data rate), LBC: unknown (LCRR[CLKDIV] = 
0x00)
L1:    D-cache 32 KiB enabled
       I-cache 32 KiB enabled
DRAM:  1 GiB
L2:    disabled
Loading Environment from nowhere... OK
In:    serial@4500
Out:   serial@4500
Err:   serial@4500
Net:   eth0: virtio-net#0
Hit any key to stop autoboot:  0
=>

I forgot in my last message to mention that my version of Qemu does not 
come with the U-boot binary image. I got this message when booting the 
mpc8544ds machine with my default Qemu configuration :

qemu-system-ppc: could not find firmware/kernel file 'u-boot.e500'

I had to add it manually from the binaries on the official Qemu github 
(https://github.com/qemu/qemu/tree/master/pc-bios) to boot a mpc8544ds 
machine. This BIOS image didn't seem to be wrong during booting the 
mpc8544ds kernel from Cédric Le Goater's github.

My current U-boot is as following :
=> version
U-Boot 2021.07 (Jul 06 2021 - 09:21:42 +0800)

powerpc-linux-gcc (GCC) 10.1.0
GNU ld (GNU Binutils) 2.3

> This line brings me up to the login prompt.
>
> I can get a e500 guest started with 2 CPUs, but the boot hangs when the
> kernel tries to start the second CPU:
>
> qemu-system-ppc -m 1G -M ppce500 -cpu e500mc \
> -kernel ./buildroot/qemu_ppc_e500mc-latest/uImage -append 
> "root=/dev/vda" \
> -drive 
> file=./buildroot/qemu_ppc_e500mc-latest/rootfs.ext2,if=virtio,format=raw 
> \
> -net nic,model=virtio-net-pci -net user -serial mon:stdio -nographic 
> -snapshot \
> -smp 2
>
> e500 family performance monitor hardware support registered
> rcu: Hierarchical SRCU implementation.
> smp: Bringing up secondary CPUs ...
> (... hangs indefinitely here ...)
My attempt to start a ppce500 machine with the same command hungs with 
calltraces during rcu/scheduling after the last event you gave me in the 
trace from your previous post.

$ qemu-system-ppc -m 1G -M ppce500 -cpu e500mc -kernel /tmp/uImage 
-append "root=/dev/vda" -drive 
file=/tmp/rootfs.ext2,if=virtio,format=raw -net nic,model=virtio-net-pci 
-net user -serial mon:stdio -nographic -snapshot -smp 2
Memory CAM mapping: 256/256/256 Mb, residual: 256Mb
Linux version 5.15.18 (legoater@kal1) 
(powerpc-buildroot-linux-uclibc-gcc.br_real (Buildroot 
2022.02-4-geae5011c83) 10.3.0, GNU ld (GNU Binutils) 2.36.1) #1 SMP Wed 
Mar 9 07:54:00 EST 2022
Using QEMU e500 machine description
printk: bootconsole [udbg0] enabled
CPU maps initialized for 1 thread per core
[...]
smp: Bringing up secondary CPUs ...
smp: Brought up 1 node, 2 CPUs
devtmpfs: initialized
clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, 
max_idle_ns: 7645041785100000 ns
futex hash table entries: 512 (order: 3, 32768 bytes, linear)
[...]
clocksource: Switched to clocksource timebase
rcu: INFO: rcu_sched self-detected stall on CPU
rcu:     0-...!: (5 ticks this GP) idle=cc7/1/0x40000002 softirq=61/61 
fqs=2
    (t=274813 jiffies g=-1091 q=1)
rcu: rcu_sched kthread timer wakeup didn't happen for 274804 jiffies! 
g-1091 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402
rcu:     Possible timer handling issue on cpu=1 timer-softirq=37
rcu: rcu_sched kthread starved for 274805 jiffies! g-1091 f0x0 
RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=1
rcu:     Unless rcu_sched kthread gets sufficient CPU time, OOM is now 
expected behavior.
rcu: RCU grace-period kthread stack dump:
task:rcu_sched       state:I stack:    0 pid:   10 ppid:     2 
flags:0x00000800
Call Trace:
[c70c7c80] [84008208] 0x84008208 (unreliable)
[c70c7d70] [c0bdb83c] __schedule+0x29c/0x7c0
[c70c7dc0] [c0bdbdb0] schedule+0x50/0x130
[c70c7de0] [c0be1400] schedule_timeout+0x90/0x180
[c70c7e20] [c00c6f88] rcu_gp_fqs_loop+0x348/0x470
[c70c7e80] [c00cb1d0] rcu_gp_kthread+0x1d0/0x200
[c70c7ed0] [c0067750] kthread+0x140/0x160
[c70c7f10] [c001514c] ret_from_kernel_thread+0x14/0x1c
rcu: Stack dump where RCU GP kthread last ran:
Task dump for CPU 1:
task:swapper/1       state:R  running task     stack:    0 pid: 0 
ppid:     1 flags:0x00000800
Call Trace:
[c70cbe10] [c0f62288] cpu_core_map+0x0/0x4 (unreliable)
[c70cbf00] [c0bdb840] __schedule+0x2a0/0x7c0
[c70cbf20] [c0f62288] cpu_core_map+0x0/0x4
[c70cbf40] [c007f310] do_idle+0xe0/0x180
[c70cbf60] [c007f5a4] cpu_startup_entry+0x24/0x30
[c70cbf80] [c001717c] start_secondary+0x4dc/0x9b8
[c70cbff0] [c0002238] __secondary_start+0x90/0xdc
Task dump for CPU 0:
task:swapper/0       state:R  running task     stack:    0 pid: 1 
ppid:     0 flags:0x00000804
Call Trace:
[c70b5960] [c0072f14] sched_show_task+0x154/0x170 (unreliable)
[c70b5980] [c00cd144] rcu_dump_cpu_stacks+0xf4/0x14c
[c70b59b0] [c00cbc20] rcu_sched_clock_irq+0x870/0x9f0
[c70b5a10] [c00d8ba8] update_process_times+0xb8/0x120
[c70b5a30] [c00f2a68] tick_sched_timer+0x98/0x180
[c70b5a50] [c00d95d4] __hrtimer_run_queues+0x1f4/0x3e0
[c70b5ab0] [c00db0dc] hrtimer_interrupt+0x15c/0x370
[c70b5b00] [c000bb74] timer_interrupt+0x204/0x360
[c70b5b40] [c0001044] Decrementer+0x104/0x120
--- interrupt: 900 at __d_rehash+0x38/0xd0
NIP:  c024cc18 LR: c024da08 CTR: c0268db0
REGS: c70b5b50 TRAP: 0900   Not tainted  (5.15.18)
MSR:  00029002 <CE,EE,ME>  CR: 24002248  XER: 00000000

GPR00: c024daec c70b5c30 c70d0000 c80b9a18 00029002 00000001 00000000 
ef75b160
GPR08: 00000001 000157a0 ef7459c0 00000006 24002248 00000000 c0004350 
00000000
GPR16: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000006
GPR24: c108c308 c7035b70 fffff000 c80b36f8 00000000 c80b36f8 00000000 
c80b9a18
NIP [c024cc18] __d_rehash+0x38/0xd0
LR [c024da08] __d_add+0xf8/0x1f0
--- interrupt: 900
[c70b5c30] [c80b36f8] 0xc80b36f8 (unreliable)
[c70b5c40] [c024daec] __d_add+0x1dc/0x1f0
[c70b5c70] [c0268e00] simple_lookup+0x50/0x90
[c70b5c90] [c023d1f8] __lookup_slow+0xa8/0x1b0
[c70b5cd0] [c023dc48] lookup_one_len+0xa8/0xc0
[c70b5d10] [c04adab4] start_creating.part.0+0x54/0xf0
[c70b5d30] [c04ae260] tracefs_create_file+0x80/0x1f0
[c70b5d60] [c014ac3c] trace_create_file+0x1c/0x60
[c70b5d80] [c0153cb0] event_create_dir+0x2e0/0x530
[c70b5dc0] [c0153f4c] __trace_early_add_event_dirs+0x4c/0xa0
[c70b5de0] [c0f1d95c] event_trace_init+0xa4/0x110
[c70b5e10] [c0f1cfe8] tracer_init_tracefs+0x9c/0x324
[c70b5e40] [c0003ec8] do_one_initcall+0x58/0x248
[c70b5eb0] [c0f0b104] kernel_init_freeable+0x200/0x2b0
[c70b5ef0] [c0004374] kernel_init+0x24/0x120
[c70b5f10] [c001514c] ret_from_kernel_thread+0x14/0x1c

The complete output is added in attachment.

> So unless you want to debug why this is happening I guess you can 
> stick with
> the mpc8544ds machine.
>
>
> If you want to give it a try by yourself, the kernel and disk images I 
> used above
> can be found in this repo:
>
> https://github.com/legoater/qemu-ppc-boot
>
>
> > Then I also tried to change the machine type based on the list in the 
> > help output
> >
> > The mpc8544ds and ppce500 machines give me the "QEMU 5.2.0 monitor" 
> > prompt right after starting but I don't really understand why...
>
>
> This happens because you didn't redirect the serial console of the 
> machine
> to the terminal. What you're seeing is the QEMU monitor console.
>
> You can redirect the VM output to the terminal by adding
>
> "-serial mon:stdio -nographic"
>
> In the command line.
With the serial redirection added to my previous command, the boot of my 
machine stops at the same U-boot prompt as my test with the gentoo iso 
as cdrom device described a few lines above.

Thank you for your help.

Florian

> Thanks,
>
>
> Daniel
>
> > Could someone help me to start a PPC virtual machine with more than 
> > one CPU?
>
>
>
>
>
> > Thank you very much
> >
> > Florian

qemu_ppce500mc.output.txt
Description: Text document