Re: [Qemu-devel] dynamically linked binaries under sparc-linux-user

[Blue Swirl]

On Sun, Jun 5, 2011 at 11:28 PM, Artyom Tarasenko <address@hidden> wrote:
> On Sat, Jun 4, 2011 at 10:30 AM, Blue Swirl <address@hidden> wrote:
>> On Sun, May 29, 2011 at 2:32 AM, Artyom Tarasenko <address@hidden> wrote:
>>> On Thu, May 26, 2011 at 8:45 PM, Blue Swirl <address@hidden> wrote:
>>>> On Tue, May 24, 2011 at 10:42 PM, Artyom Tarasenko <address@hidden> wrote:
>>>>> Should it be possible to use dynamically linked binaries under
>>>>> sparc*-linux-user?
>>>>> Under qemu-system-sparc the Debian 4.08r1 initrd works fine, but:
>>>>>
>>>>> master$ sparc-linux-user/qemu-sparc -strace -L
>>>>> ../debian-4.08r1-initrd/ ../debian-4.08r1-initrd/bin/busybox
>>>>> 14004 uname(0x409ffbae) = 0
>>>>> 14004 brk(NULL) = 0x00063000
>>>>> 14004 access("/etc/ld.so.nohwcap",F_OK) = -1 errno=2 (No such file or 
>>>>> directory)
>>>>> 14004 mmap(NULL,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS,-1,0)
>>>>> = 0x40a2c000
>>>>> 14004 access("/etc/ld.so.preload",R_OK) = -1 errno=2 (No such file or 
>>>>> directory)
>>>>> 14004 open("/etc/ld.so.cache",O_RDONLY) = 3
>>>>> 14004 fstat64(3,0x409ff500) = 0
>>>>> 14004 mmap(NULL,195479,PROT_READ,MAP_PRIVATE,3,0) = 0x40a2d000
>>>>> 14004 close(3) = 0
>>>>> Segmentation fault
>>>>>
>>>>> The strange thing here is that it loads ld.so.cache. The guest fs
>>>>> doesn't have it, but the host does:
>>>>>
>>>>> master$  ll ../../debian-4.08r1-initrd/etc/ld.so.cache /etc/ld.so.cache
>>>>> ls: cannot access ../../debian-4.08r1-initrd/etc/ld.so.cache: No such
>>>>> file or directory
>>>>> -rw-r--r--. 1 root root 195479 2011-03-17 13:48 /etc/ld.so.cache
>>>>>
>>>>> Isn't this wrong?
>>>>
>>>> I'm not sure.
>>>
>>> Right. On a second thought, qemu is probably doing what is expected:
>>> the syscalls are emulated, so the host libraries must be loaded.
>>> Then the problem must be elsewhere. Here is the backtrace:
>>>
>>> master$  gdb sparc-linux-user/qemu-sparc
>>> GNU gdb (GDB) Fedora (7.0.1-50.fc12)
>>> (gdb) run -L ../debian-4.08r1-initrd/ ../debian-4.08r1-initrd/bin/busybox
>>> Starting program: sparc-linux-user/qemu-sparc -L
>>> ../debian-4.08r1-initrd/ ../debian-4.08r1-initrd/bin/busybox
>>> [Thread debugging using libthread_db enabled]
>>>
>>> Program received signal SIGSEGV, Segmentation fault.
>>> 0x00000000601c49c2 in static_code_gen_buffer ()
>>> Missing separate debuginfos, use: debuginfo-install
>>> glibc-2.11.2-3.x86_64 libattr-2.4.44-1.fc12.x86_64
>>> nspr-4.8.6-1.fc12.x86_64 nss-3.12.8-2.fc12.x86_64
>>> nss-util-3.12.8-1.fc12.x86_64 zlib-1.2.3-23.fc12.x86_64
>>> (gdb) bt
>>> #0  0x00000000601c49c2 in static_code_gen_buffer ()
>>> #1  0x00007fffffffd684 in ?? ()
>>> #2  0x00007ffff4bc8728 in ?? ()
>>> #3  0x00000000ffffffff in ?? ()
>>> #4  0x0000000060029083 in tb_alloc_page (tb=0x40a2bc00, phys_pc=<value
>>> optimized out>, phys_page2=1084406920) at exec.c:1214
>>> #5  tb_link_page (tb=0x40a2bc00, phys_pc=<value optimized out>,
>>> phys_page2=1084406920) at exec.c:1278
>>> #6  0x000000006002a037 in tb_gen_code (env=0x6223f390, pc=1084305880,
>>> cs_base=<value optimized out>, flags=<value optimized out>,
>>> cflags=<value optimized out>)
>>>    at exec.c:1004
>>> #7  0x000000006002afe8 in cpu_sparc_exec (env1=<value optimized out>)
>>> at cpu-exec.c:636
>>> #8  0x0000000060005d50 in cpu_loop (env=0x6223f390) at 
>>> linux-user/main.c:1008
>>> #9  0x00000000600069d9 in main (argc=1646342192, argv=<value optimized
>>> out>, envp=<value optimized out>) at linux-user/main.c:3533
>>> (gdb)
>>>
>>> Any ideas?
>>
>> The logs would probably show the crashing instruction.
>
> $ sparc-linux-user/qemu-sparc -g 1234 -L ../debian-4.08r1-initrd/
> ../debian-4.08r1-initrd/bin/busybox &
> $ gdb-sparc
> (gdb) target remote localhost:1234
> Remote debugging using localhost:1234
> 0x40a02c80 in ?? ()
> (gdb) c
> Continuing.
>
> Breakpoint 1, 0x40a09a2c in ?? ()
> (gdb) disas $pc-0x20,$pc+0x20
> Dump of assembler code from 0x40a09a0c to 0x40a09a4c:
>   0x40a09a0c:  srl  %l3, 0x1f, %g2
>   0x40a09a10:  add  %g2, %l3, %g2
>   0x40a09a14:  add  %l6, %g1, %l5
>   0x40a09a18:  sra  %g2, 1, %l0
>   0x40a09a1c:  add  %l0, %l0, %l1
>   0x40a09a20:  add  %l1, %l0, %g1
>   0x40a09a24:  sll  %g1, 2, %g1
>   0x40a09a28:  add  %g1, %l6, %g1
> => 0x40a09a2c:  ld  [ %g1 + 0x14 ], %g2
>   0x40a09a30:  cmp  %l4, %g2
>   0x40a09a34:  bleu  0x40a09968
>   0x40a09a38:  sethi  %hi(0x400), %g1
>   0x40a09a3c:  clr  %l2
>   0x40a09a40:  or  %g1, 0x2ac, %g1
>   0x40a09a44:  b  0x40a09a84
>   0x40a09a48:  ld  [ %l7 + %g1 ], %i3
> End of assembler dump.
> (gdb) info registers g1
> g1             0xe0d8cff4       -522661900
> (gdb) si
>
> Program received signal SIGSEGV, Segmentation fault.
> 0x40a09a2c in ?? ()
>
> I went through the arithmetical operations and got the same result for
> %g1: 0xe0d8cff4 .
>
> Is the user-emulation memory layout somewhere documented? It would be
> interesting to know what are the address spaces for the libraries and
> the user program itself .
>
>>>> It could be possible to construct a blacklist of host
>>>> files that may not be accessible or visible to the guest but that
>>>> wouldn't very robust either. Chrooting into a 100% guest architecture
>>>> system should work better.
>>>
>>> You mean some sort of mixed chrooting? At least some host libraries
>>> must be visible to the guest as if they were native.
>>
>> The simplest way is to build a statically linked emulator, chroot to
>> the guest system and execute the emulator. Then the only host
>> executable should be QEMU itself. Maybe binfmt_misc can also be used.
>
> Great idea, indeed! It works, thanks!  The breakpoint at 40a09a2c is
> never reached in this case hit though, so I can't directly see where
> the dynamically linked qemu derails.
>
> And it looks that some functionality is missing:
>
> #    chroot ../debian-4.08r1-initrd /qemu-sparc  -L . /bin/busybox ash
> BusyBox v1.1.3 (Debian 1:1.1.3-4) Built-in shell (ash)
> Enter 'help' for a list of built-in commands.
>
> ~ # echo 123
> 123
> ~ # ls
> HELPME: master/target-sparc/cpu.h:625
> ash: ls: Exec format error
>
> And here it hangs. The comment says /* FIXME: Do we also need to clear
> CF?  */ . I see no further references of CF in cpu.h - do yo know what
> is meant here?

fork() is handled on Sparc differently from other architectures, CF is
used to detect whether we are the parent or the child. Long time ago I
tried some obvious fixes here but it wouldn't help. Could be a very
simple bug somewhere.