[Qemu-devel] analysis of LTP test failures under QEMU linux-user

[Peter Maydell]

I went through and quickly analysed all our LTP failures
when running under linux-user, which I figured might be of
interest to the list...

This is with current master plus the following patches (all on list):

linux-user: Avoid possible misalignment in host_to_target_siginfo()
linux-user: Use __get_user() and __put_user() to handle structs in do_fcntl()
linux-user: Use safe_syscall wrapper for fcntl
configure: Don't override ARCH=unknown if enabling TCI
configure: Don't allow user-only targets for unknown CPU architectures
user-exec: Delete now-unused hppa and m68k cpu_signal_handler() code
user-exec: Remove unused code for OSX hosts
linux-user: Create a hostdep.h for each host architecture
linux-user: Don't use sigfillset() on uc->uc_sigmask
linux-user: Check sigsetsize argument to syscalls
linux-user: Fix wrong type used for argument to rt_sigqueueinfo
linux-user: Support F_GETPIPE_SZ and F_SETPIPE_SZ fcntls

and running a 32-bit ARM guest on x86-64 host with version
20150903 of the LTP testsuite.

Headline figures:

Total Tests: 1054
Total Skipped Tests: 76
Total Failures: 46

Categorization of the failures:

Unimplemented syscalls:
adjtimex01                    NR_adjtimex not implemented
adjtimex02                    NR_adjtimex not implemented
fanotify01                    fanotify syscalls not implemented
fanotify02                    ditto
fanotify04                    ditto
mq_notify01                   mq_notify syscall not implemented
mq_notify02                   ditto
process_vm_readv01            process_vm_readv syscall not implemented
process_vm_readv02            ditto
process_vm_readv03            ditto
process_vm_writev01           process_vm_writev syscall not implemented
process_vm_writev02           ditto
ptrace01                      ptrace not implemented
ptrace02                      ditto
ptrace03                      ditto
ptrace05                      ditto
remap_file_pages01            remap_file_pages syscall unimplemented
remap_file_pages02            ditto
futex_wait02                  wants set_robust_list and we can't do that
futex_wake03                  ditto

Unimplemented flags etc to implemented syscalls
clone05                       CLONE_VFORK not supported
clone08                       oddball clone flags not supported
setns02                       we don't support CLONE_NEWUTS, CLONE_NEWIPC

Issues with test harness assuming files/binaries/other config
ioctl03                       /dev/net/tun doesn't exist in chroot
setregid02                    test expects a "nobody" group to exist but
                              ubuntu calls it "nogroup"; this is fixed in
                              LTP upstream
setregid02_16                 ditto
utimensat01                   wants an /etc/sudoers and sudo!

Test assumes priority of one error over another, etc
(not feasible for QEMU to make most error checks happen after EFAULT check)
read02                        wants EBADF and EISDIR to take precedence over
                              EFAULT
recvmsg01                     not analysed in detail
sockioctl01                   wants ENOTTY to take precedence over EFAULT

Wants to operate on signal 64 (which under QEMU is reserved to the host libc):
rt_sigaction01                wants to test signal 64
sighold02                     wants to operate on signal 64
sigrelse01                    wants to operate on signal 64

Misc unfixed:
creat07                       creat on text's executable doesn't fail ETXTBSY
mmap12                        (not analysed)
mmap13                        (not analysed)
mprotect01                    mprotect(0) should fail ENOMEM
mprotect04                    test tries to copy and execute a lump of
                              code, unexpected SIGILL
mremap03                      mremap doesn't fail when it should
msync03                       msync should fail ENOMEM on unmapped memory
munlock02                     munlock should fail ENOMEM on unmapped memory
mincore01                     mincore on unmapped memory or overlong length
                              should fail ENOMEM
madvise02                     madvise should fail for misaligned address,
                              bad advice value, etc etc etc
readahead02                   intermittent? "failed to save any I/O"
recvfrom01                    dumps core!
shmat01                       (not analysed)

Tests which we skip because they hang or otherwise break things:
clone02                       we aren't implementing clone flags correctly so
                              the child gets the wrong return value for
                              getppid() and kills the test harness by accident
fork13                        hangs
futex_wait03                  hangs
msgctl11                      runs OK but thrashes the machine with lots
                              of processes
syslog01                      these try to restart syslogd, which is a bad
                              plan in a chroot
syslog02
syslog03
syslog04
syslog05
syslog06
syslog07
syslog08
syslog09
syslog10
syslog11
syslog12
mmap16                        gets rather confused, and in failing it doesn't
                              clean up a loopback mount it set up, which makes
                              later tests fail
mount03                       fails for a dumb reason (it assumes that
                              read(fd, NULL, 20) will return 0 not fail -EFAULT
                              if the fd happens to be at EOF, and then the
                              cleanup error handling fails and leaves the
                              loopback mount mounted, which causes some later
                              tests to fail unnecessarily.

Some of the "skipped" tests are also failing because of syscalls QEMU
doesn't implement; I haven't looked through to find out which.

thanks
-- PMM