Re: qemu-ppc64-linux-user: signal handler endianness in elfv2

[Vincent Fazio]

Laurent,

On 2/11/20 4:26 AM, Laurent Vivier wrote:
> Le 08/02/2020 à 19:57, Vincent Fazio a écrit :
> > I ran into a crash scenario in ppc64-linux-user when trying to run
> > openssl compiled with gcc & musl which emits an ELFv2 binary. The target
> > CPU was a big endian model (e6500)
> >
> > Example:
> >
> > vfazio@vfazio1 ~/development/buildroot/output/target :( $ gdb
> > ~/development/qemu/ppc64-linux-user/qemu-ppc64
> > (gdb) run -d guest_errors -cpu e6500 -E
> > LD_LIBRARY_PATH="/home/vfazio/development/buildroot/output/target/lib/:/home/vfazio/development/buildroot/output/target/usr/lib/"
> > lib/libc.so bin/bash
> > Starting program:
> > /home/vfazio/development/qemu/ppc64-linux-user/qemu-ppc64 -d
> > guest_errors -cpu e6500 -E
> > LD_LIBRARY_PATH="/home/vfazio/development/buildroot/output/target/lib/:/home/vfazio/development/buildroot/output/target/usr/lib/"
> > lib/libc.so bin/bash
> > [Thread debugging using libthread_db enabled]
> > Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> > [New Thread 0x7ffff7ff9700 (LWP 27186)]
> > invalid/unsupported opcode: 1f - 0c - 1a - 00 (7c005698) 0000000010028e58 0
> >
> > Thread 1 "qemu-ppc64" received signal SIGSEGV, Segmentation fault.
> > 0x00000000600a9257 in ldl_he_p (ptr=0x4c2c061000000000) at
> > /home/vfazio/development/qemu/include/qemu/bswap.h:351
> > 351        __builtin_memcpy(&r, ptr, sizeof(r));
> >
> > (gdb) !readelf -h bin/bash | grep abi
> >    Flags:                             0x2, abiv2
> > (gdb) !readelf -h lib/libc.so | grep abi
> >    Flags:                             0x2, abiv2
> > <<<<<
> >
> > openssl does "probing" for PPC features (Altivec, Crypto, etc) by trying
> > to execute possibly illegal instructions and handling them... hence that
> > SIGILL. The problem is when it comes to jumping to the signal handler
> >
> > #0  0x00000000600a9257 in ldl_he_p (ptr=0x4c2c061000000000) at
> > /home/vfazio/development/qemu/include/qemu/bswap.h:351
> > #1  0x00000000600a92fe in ldl_be_p (ptr=0x4c2c061000000000) at
> > /home/vfazio/development/qemu/include/qemu/bswap.h:449
> > #2  0x00000000600c0790 in translator_ldl_swap (env=0x62923150,
> > pc=5488768711647035392, do_swap=false) at
> > /home/vfazio/development/qemu/include/exec/translator.h:201
> > #3  0x000000006011c1ab in ppc_tr_translate_insn (dcbase=0x7fffffffd250,
> > cs=0x6291ae80) at /home/vfazio/development/qemu/target/ppc/translate.c:7856
> > #4  0x000000006005ae70 in translator_loop (ops=0x60805fc0 <ppc_tr_ops>,
> > db=0x7fffffffd250, cpu=0x6291ae80, tb=0x60a5f900
> > <static_code_gen_buffer+1681600>, max_insns=512)
> >      at /home/vfazio/development/qemu/accel/tcg/translator.c:102
> > <<<<<
> >
> > The handler has what looks like an LE address but then swaps it since
> > the host is LE and the target is BE
> > setup_rt_frame (sig=4, ka=0x628c5f80 <sigact_table+96>, info=0x62931658,
> > set=0x7fffffffd2f8, env=0x62923150) at
> > /home/vfazio/development/qemu/linux-user/ppc/signal.c:575
> > 575            qemu_log_mask(LOG_GUEST_ERROR, "sa_handler NIP to "
> > TARGET_FMT_lx "\n", ka->_sa_handler);
> > (gdb) p/x ka->_sa_handler
> > $11 = 0x10062c4c
> > 579            env->nip = tswapl((target_ulong) ka->_sa_handler);
> > (gdb) p/x env->nip
> > $12 = 0x4c2c061000000000
> > <<<<<
> >
> > the memcpy later faults because that's obviously not a valid address
> > from which to grab the PPC instructions that need to be translated.
> >
> > I'm not familiar with the ELFv2 ABI or PPC assembly in general, so I'm
> > not sure what to expect here. Typically ELFv2 was reserved for ppc64le,
> > but musl uses ELFv2 for all ppc64 targets. This likely wouldn't be an
> > issue in ppc64LE since x86_64 is LE as well and no swap would take place.
> >
> > Is the signal handler address tied to the endianess of the host? I
> > noticed there was no swapping in elfload.c so wasn't sure...removing the
> > swap gets me a little further in program execution but i do eventually
> > crash (but that may not be related to this specific issue).
> To be sure it's not a problem in the qemu linux-user part, could you run
> your binaries on a real hardware or with qemu-system-ppc64?
Good call. I ran this on a T2080E we have:

>>>>>
# /lib/libc.so --version
musl libc (powerpc64)
Version 1.1.24
Dynamic Program Loader
Usage: /lib/libc.so [options] [--] pathname [args]

# cat /proc/cpuinfo
processor       : 0
cpu             : e6500, altivec supported
clock           : 1799.999982MHz
revision        : 2.0 (pvr 8040 0120)

# openssl
OpenSSL>
<<<<<



I changed the code in setup_rt_frame to not swap:

>>>>>
vfazio@vfazio1 ~/development/buildroot/output/target $ gdb 
~/development/qemu/ppc64-linux-user/qemu-ppc64

(gdb) run -d guest_errors -cpu e6500 -E 
LD_LIBRARY_PATH="/home/vfazio/development/buildroot/output/target/lib/:/home/vfazio/development/buildroot/output/target/usr/lib/" 
lib/libc.so usr/bin/openssl
Starting program: 
/home/vfazio/development/qemu/ppc64-linux-user/qemu-ppc64 -d 
guest_errors -cpu e6500 -E 
LD_LIBRARY_PATH="/home/vfazio/development/buildroot/output/target/lib/:/home/vfazio/development/buildroot/output/target/usr/lib/" 
lib/libc.so usr/bin/openssl
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff7ff9700 (LWP 27220)]
invalid/unsupported opcode: 1f - 0c - 1a - 00 (7c000e99) 0000004000ad75f0 0
sa_handler NIP to 0000004000ad71cc
elfv2 NIP to 0000004000ad71cc
invalid/unsupported opcode: 04 - 19 - 00 - 00 (10600033) 0000004000ad7614 0
sa_handler NIP to 0000004000ad71cc
elfv2 NIP to 0000004000ad71cc
OpenSSL>
<<<<<

otherwise the segfault occurs:

>>>>>
(gdb) run -d guest_errors -cpu e6500 -E 
LD_LIBRARY_PATH="/home/vfazio/development/buildroot/output/target/lib/:/home/vfazio/development/buildroot/output/target/usr/lib/" 
lib/libc.so usr/bin/openssl
Starting program: 
/home/vfazio/development/qemu/ppc64-linux-user/qemu-ppc64 -d 
guest_errors -cpu e6500 -E 
LD_LIBRARY_PATH="/home/vfazio/development/buildroot/output/target/lib/:/home/vfazio/development/buildroot/output/target/usr/lib/" 
lib/libc.so usr/bin/openssl
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff7ff9700 (LWP 32224)]
invalid/unsupported opcode: 1f - 0c - 1a - 00 (7c000e99) 0000004000af0440 0
sa_handler NIP to 0000004000af0014
elfv2 NIP to 1400af0040000000

Thread 1 "qemu-ppc64" received signal SIGSEGV, Segmentation fault.
0x00000000600a9257 in ldl_he_p (ptr=0x1400af0040000000) at 
/home/vfazio/development/qemu/include/qemu/bswap.h:351
351        __builtin_memcpy(&r, ptr, sizeof(r));
(gdb) where
#0  0x00000000600a9257 in ldl_he_p (ptr=0x1400af0040000000) at 
/home/vfazio/development/qemu/include/qemu/bswap.h:351
#1  0x00000000600a92fe in ldl_be_p (ptr=0x1400af0040000000) at 
/home/vfazio/development/qemu/include/qemu/bswap.h:449
#2  0x00000000600c0790 in translator_ldl_swap (env=0x62923150, 
pc=1441344296367161344, do_swap=false) at 
/home/vfazio/development/qemu/include/exec/translator.h:201
#3  0x000000006011c1ab in ppc_tr_translate_insn (dcbase=0x7fffffffd250, 
cs=0x6291ae80) at /home/vfazio/development/qemu/target/ppc/translate.c:7856
#4  0x000000006005ae70 in translator_loop (ops=0x60805fc0 <ppc_tr_ops>, 
db=0x7fffffffd250, cpu=0x6291ae80, tb=0x60928100 
<static_code_gen_buffer+405696>, max_insns=512)
    at /home/vfazio/development/qemu/accel/tcg/translator.c:102
#5  0x000000006011c840 in gen_intermediate_code (cs=0x6291ae80, 
tb=0x60928100 <static_code_gen_buffer+405696>, max_insns=512) at 
/home/vfazio/development/qemu/target/ppc/translate.c:7967
#6  0x00000000600592d6 in tb_gen_code (cpu=0x6291ae80, 
pc=1441344296367161344, cs_base=0, flags=33579008, cflags=-16777216) at 
/home/vfazio/development/qemu/accel/tcg/translate-all.c:1734
#7  0x000000006005664b in tb_find (cpu=0x6291ae80, last_tb=0x0, 
tb_exit=0, cf_mask=0) at 
/home/vfazio/development/qemu/accel/tcg/cpu-exec.c:406
#8  0x0000000060056dac in cpu_exec (cpu=0x6291ae80) at 
/home/vfazio/development/qemu/accel/tcg/cpu-exec.c:730
#9  0x0000000060096d62 in cpu_loop (env=0x62923150) at 
/home/vfazio/development/qemu/linux-user/ppc/cpu_loop.c:80
#10 0x000000006006444e in main (argc=9, argv=0x7fffffffddd8, 
envp=0x7fffffffde28) at /home/vfazio/development/qemu/linux-user/main.c:865

<<<<<

The one thing i'm still not sure of is if the swap needs to happen on BE 
hosts or not at all. I'll try to get qemu compiled for the e6500 and 
rerun the tests.

I can submit a patch once I get the results.

> Or provide me a tar.gz with the binaries for the target, I will test
> them on my PowerMac G5 (64bit BE).
>
> Thanks,
> Laurent

--
Vincent Fazio
Embedded Software Engineer - Linux
Extreme Engineering Solutions, Inc
http://www.xes-inc.com