[Qemu-devel] qemu-arm user mode does not work with ARM EABI

[Martin Guy]

Summary:
 Ever since QEMU 0.8.1, qemu-arm-system mode works fine with ARM EABI
kernel and rootfs, but qemu-arm single-process mode is broken with
EABI, failing with "Unknown system call 0xf0005".
 This is ARM private system call "set_tls", called from both the
linuxthreads and the NPTL pthread implementations in glibc-2.4, and
unimplemented in QEMU.
 It is important that this mode work because scratchbox needs it to
be able to cross-compile Debian packages for the forthcoming
EABI-based Debian architecture.

Detail:
I have three kinds of ARM root filesystem:

- standard Debian ARM sarge using old-ABI: (glibc-2.3.6, linuxthreads)
- Voipio's "eabi-rootfs" which I gather uses EABI internally and glibc
shims to convert the system calls to old-ABI just before they hit the
kernel (glibc-2.3.4)
- Koen's EABI-toolchain-rootfs (from the Angstrom distro of
OpenEmbedded) which uses pure EABI (glibc-2.4,nptl)

# Here they are
martin:~/tftpboot$ ls
EABI-toolchain-rootfs/  debian-sarge-arm/  eabi-rootfs/

In qemu-system-arm (softmmu target) all three run fine (given
appropriate kernels) but qemu-arm fails on the second two.

# Even qemu-arm 0.8.1 works on old ABI
martin:~/tftpboot$ /usr/bin/qemu-arm -L `pwd`/deb* deb*/bin/echo hi
hi
martin:~/tftpboot$

# Both of the EABI scenarios fail the same way.

# With Debian QEMU (0.8.1):
martin:~/tftpboot$ /usr/bin/qemu-arm -L `pwd`/EABI* EABI*/bin/echo hi
Error: f0005
qemu: uncaught target signal 6 (Aborted) - exiting
martin:~/tftpboot$

# With qemu-arm 0.8.2 compiled from source (using gcc-3.4 of course):
martin:~/tftpboot$ /usr/local/bin/qemu-arm -L `pwd`/EABI* EABI*/bin/echo hi
qemu: Unsupported syscall: 983045
cannot set up thread-local storage: unknown error
martin:~/tftpboot$ echo 'obase=16; 983045' | bc
F0005
martin:~/tftpboot$

This is not a case of the new EABI picking up old ABI system calls,
which were encoded as SWI #(0x900000 + n).

A couple of debugging printfs show that with the pure EABI rootfs, the
system call number is picked up by QEMU from r7 as it should, and when
executing a simple binary it correctly picks up about 20 system calls
starting with 'brk; uname; access; open; fstat64; mmap2' and ending
with 'close; mmap2; [F0005]; truncate; io_submit; [fail]'.
Using the intermediate glibc-shims version, the same bogus system call
number is received via the old-ABI system call mechanism.

The Linux kernel source, while decoding ARM system calls says
in arch/arm/kernel/traps.c
*  0x9f0000 - 0x9fffff are some more esoteric system calls
and if we subtract the old syscall offset of 0x900000 from this we get
our symptom of  0xf000* The extra __ARM_NR_* system call numbers run
from __NR_SYSCALL_BASE+__ARM_NR_BASE (0x900000 + 0xf0000) and are
defined in include/asm-arm/unistd.h as:
1   __ARM_NR_breakpoint
2   __ARM_NR_cacheflush
3   __ARM_NR_usr26
4   __ARM_NR_usr32
5   __ARM_NR_set_tls   <--- this one

This is glibc-2.4 using the internal ARM system call set_tls to
initialise(?) the thread local storage (something which the shims for
glibc-2.3.4 also evidently do). It is called from the macro:

glibc-2.4/ports/sysdeps/arm/{linuxthreads,nptl}/tls.h:
/* Code to initially initialize the thread pointer.  This might need
  special attention since 'errno' is not yet available and if the
  operation can cause a failure 'errno' must not be touched.  */
# define TLS_INIT_TP(TCBP, SECONDCALL) \
 ({ INTERNAL_SYSCALL_DECL (err); \
    long result_var; \
    result_var = INTERNAL_SYSCALL_ARM (set_tls, err, 1, (TCBP)); \
    INTERNAL_SYSCALL_ERROR_P (result_var, err) \
      ? "unknown error" : NULL; })

And this is what the Linux kernel does with that:

linux-2.6.17/arch/arm/kernel/traps.c:
       case NR(set_tls):
               thread->tp_value = regs->ARM_r0;
#if defined(CONFIG_HAS_TLS_REG)
               asm ("mcr p15, 0, %0, c13, c0, 3" : : "r" (regs->ARM_r0) );
#elif !defined(CONFIG_TLS_REG_EMUL)
               /*
                * User space must never try to access this directly.
                * Expect your app to break eventually if you do so.
                * The user helper at 0xffff0fe0 must be used instead.
                * (see entry-armv.S for details)
                */
               *((unsigned int *)0xffff0ff0) = regs->ARM_r0;
#endif
               return 0;

Quite what that really means, and what QEMU should do about it, are
unclear to me. I've tried ignoring the f0005 system call, and the ARM
program segfaults straight away, so I guess *something* is necessary.
The 32-bit parameter ARM_r0 is always in the 0x7e* and 0x7f* range and
is different at each execution. Does this call set the area used for
thread-local storage?

Any ideas?

    M