Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc

[Grundmann, Christian]

Hi again,
got a Segfault today without virtio :-( (one IDE Disk and one virtio-scsi)

Core was generated by `/usr/libexec/qemu-kvm -name vmname -S -machine 
pc-i440fx-rhel7.2.0,accel='.
Program terminated with signal 11, Segmentation fault.
#0  0x00007fb299cbd3ab in 
tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, 
unsigned long, int) () from /lib64/libtcmalloc.so.4

Thread 6 (Thread 0x7fb28d0c5700 (LWP 29423)):
#0  0x00007fb29cc85ac3 in pread64 () at ../sysdeps/unix/syscall-template.S:81
No locals.
#1  0x00007fb29e37c2a3 in pread (__offset=<optimized out>, __nbytes=<optimized 
out>, __buf=0x7fb2a3e21a00, __fd=<optimized out>) at 
/usr/include/bits/unistd.h:99
No locals.
#2  handle_aiocb_rw_linear (address@hidden, address@hidden "QF", <incomplete 
sequence \373>) at block/raw-posix.c:909
        offset = 0
        len = <optimized out>
#3  0x00007fb29e37c3d1 in handle_aiocb_rw (aiocb=0x7fb2a1474340) at 
block/raw-posix.c:992
        nbytes = <optimized out>
        buf = 0x7fb2a3e21a00 "QF", <incomplete sequence \373>
        __PRETTY_FUNCTION__ = "handle_aiocb_rw"
#4  0x00007fb29e37d945 in aio_worker (arg=0x7fb2a1474340) at 
block/raw-posix.c:1204
        aiocb = 0x7fb2a1474340
        ret = 0
#5  0x00007fb29e33d91b in worker_thread (opaque=0x7fb2a148d450) at 
thread-pool.c:105
        req = 0x7fb2a1474b30
        ret = <optimized out>
        pool = 0x7fb2a148d450
#6  0x00007fb29cc7edf5 in start_thread (arg=0x7fb28d0c5700) at 
pthread_create.c:308
        __res = <optimized out>
        pd = 0x7fb28d0c5700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140404847302400, 
-4522449750849005939, 0, 140404847303104, 140404847302400, 26, 
4492373549408278157, 4492409237274449549}, mask_was_saved = 0}}, priv = {pad = 
{0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
#7  0x00007fb29688c1ad in clone () at 
../sysdeps/unix/sysv/linux/x86_64/clone.S:113
No locals.

Thread 5 (Thread 0x7fb104fff700 (LWP 29084)):
#0  0x00007fb296881b7d in poll () at ../sysdeps/unix/syscall-template.S:81
No locals.
#1  0x00007fb2977d6fe7 in red_worker_main () from /lib64/libspice-server.so.1
No symbol table info available.
#2  0x00007fb29cc7edf5 in start_thread (arg=0x7fb104fff700) at 
pthread_create.c:308
        __res = <optimized out>
        pd = 0x7fb104fff700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140398269822720, 
-4522449750849005939, 0, 140398269823424, 140398269822720, 140405245697216, 
4494326442046740109, 4492409237274449549}, mask_was_saved = 0}}, priv = {pad = 
{0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
#3  0x00007fb29688c1ad in clone () at 
../sysdeps/unix/sysv/linux/x86_64/clone.S:113
No locals.

Thread 4 (Thread 0x7fb28c8c4700 (LWP 29081)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at 
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x00007fb29e3b9f79 in qemu_cond_wait (cond=<optimized out>, address@hidden 
<qemu_global_mutex>) at util/qemu-thread-posix.c:132
        err = <optimized out>
        __func__ = "qemu_cond_wait"
#2  0x00007fb29e172d3b in qemu_kvm_wait_io_event (cpu=<optimized out>) at 
/usr/src/debug/qemu-2.3.0/cpus.c:912
No locals.
#3  qemu_kvm_cpu_thread_fn (arg=0x7fb2a3d2e000) at 
/usr/src/debug/qemu-2.3.0/cpus.c:949
        cpu = 0x7fb2a3d2e000
        r = <optimized out>
#4  0x00007fb29cc7edf5 in start_thread (arg=0x7fb28c8c4700) at 
pthread_create.c:308
        __res = <optimized out>
        pd = 0x7fb28c8c4700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140404838909696, 
-4522449750849005939, 0, 140404838910400, 140404838909696, 140735272359936, 
4492374652678002317, 4492409237274449549}, mask_was_saved = 0}}, priv = {pad = 
{0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
#5  0x00007fb29688c1ad in clone () at 
../sysdeps/unix/sysv/linux/x86_64/clone.S:113
No locals.

Thread 3 (Thread 0x7fb28c0c3700 (LWP 29082)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at 
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x00007fb29e3b9f79 in qemu_cond_wait (cond=<optimized out>, address@hidden 
<qemu_global_mutex>) at util/qemu-thread-posix.c:132
        err = <optimized out>
        __func__ = "qemu_cond_wait"
#2  0x00007fb29e172d3b in qemu_kvm_wait_io_event (cpu=<optimized out>) at 
/usr/src/debug/qemu-2.3.0/cpus.c:912
No locals.
#3  qemu_kvm_cpu_thread_fn (arg=0x7fb2a3d7e000) at 
/usr/src/debug/qemu-2.3.0/cpus.c:949
        cpu = 0x7fb2a3d7e000
        r = <optimized out>
#4  0x00007fb29cc7edf5 in start_thread (arg=0x7fb28c0c3700) at 
pthread_create.c:308
        __res = <optimized out>
        pd = 0x7fb28c0c3700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140404830516992, 
-4522449750849005939, 0, 140404830517696, 140404830516992, 140735272359936, 
4492375751652759181, 4492409237274449549}, mask_was_saved = 0}}, priv = {pad = 
{0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
#5  0x00007fb29688c1ad in clone () at 
../sysdeps/unix/sysv/linux/x86_64/clone.S:113
No locals.

Thread 2 (Thread 0x7fb28e6ad700 (LWP 29077)):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
No locals.
#1  0x00007fb29e3ba272 in futex_wait (val=4294967295, ev=0x7fb29ec84f44 
<rcu_call_ready_event>) at util/qemu-thread-posix.c:301
No locals.
#2  qemu_event_wait (address@hidden <rcu_call_ready_event>) at 
util/qemu-thread-posix.c:399
        value = <optimized out>
#3  0x00007fb29e3c8526 in call_rcu_thread (opaque=<optimized out>) at 
util/rcu.c:233
        tries = 0
        n = <optimized out>
        node = <optimized out>
#4  0x00007fb29cc7edf5 in start_thread (arg=0x7fb28e6ad700) at 
pthread_create.c:308
        __res = <optimized out>
        pd = 0x7fb28e6ad700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140404870272768, 
-4522449750849005939, 0, 140404870273472, 140404870272768, 140405136150080, 
4492370572995942029, 4492409237274449549}, mask_was_saved = 0}}, priv = {pad = 
{0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
#5  0x00007fb29688c1ad in clone () at 
../sysdeps/unix/sysv/linux/x86_64/clone.S:113
No locals.

Thread 1 (Thread 0x7fb29e07cc00 (LWP 29076)):
#0  0x00007fb299cbd3ab in 
tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, 
unsigned long, int) () from /lib64/libtcmalloc.so.4
No symbol table info available.
#1  0x00007fb299cbd47b in 
tcmalloc::ThreadCache::ListTooLong(tcmalloc::ThreadCache::FreeList*, unsigned 
long) () from /lib64/libtcmalloc.so.4
No symbol table info available.
#2  0x00007fb299ccc070 in tc_free () from /lib64/libtcmalloc.so.4
No symbol table info available.
#3  0x00007fb29c58d58f in g_free () from /lib64/libglib-2.0.so.0
No symbol table info available.
#4  0x00007fb29e3b7721 in parser_context_free (ctxt=0x7fb2a531e0c0) at 
qobject/json-parser.c:358
        i = <optimized out>
#5  json_parser_parse_err (tokens=<optimized out>, address@hidden, 
address@hidden) at qobject/json-parser.c:710
        result = 0x7fb2a4bdf600
#6  0x00007fb29e3b7767 in json_parser_parse (tokens=<optimized out>, 
address@hidden) at qobject/json-parser.c:694
No locals.
#7  0x00007fb29e176e04 in handle_qmp_command (parser=<optimized out>, 
tokens=<optimized out>) at /usr/src/debug/qemu-2.3.0/monitor.c:5068
        err = <optimized out>
        obj = <optimized out>
        input = 0x0
        args = 0x0
        cmd_name = <optimized out>
        mon = 0x7fb2a153e140
#8  0x00007fb29e3b64f2 in json_message_process_token (lexer=0x7fb2a1460040, 
token=0x7fb2a1424880, type=JSON_OPERATOR, x=49, y=104) at 
qobject/json-streamer.c:87
        parser = 0x7fb2a1460038
        dict = 0x7fb2a3e27200
#9  0x00007fb29e3c891f in json_lexer_feed_char (address@hidden, ch=<optimized 
out>, address@hidden) at qobject/json-lexer.c:303
        new_state = 100
#10 0x00007fb29e3c89ee in json_lexer_feed (lexer=0x7fb2a1460040, 
buffer=<optimized out>, size=<optimized out>) at qobject/json-lexer.c:356
        err = <optimized out>
        i = <optimized out>
#11 0x00007fb29e3b6689 in json_message_parser_feed (parser=<optimized out>, 
buffer=<optimized out>, size=<optimized out>) at qobject/json-streamer.c:110
No locals.
#12 0x00007fb29e1758cf in monitor_control_read (opaque=<optimized out>, 
buf=<optimized out>, size=<optimized out>) at 
/usr/src/debug/qemu-2.3.0/monitor.c:5134
        old_mon = 0x0
#13 0x00007fb29e2321b0 in qemu_chr_be_write (len=<optimized out>, 
buf=0x7fff7bea8a30 "}\212\352{\377\177", s=0x7fb2a14442e0) at qemu-char.c:305
No locals.
#14 tcp_chr_read (chan=<optimized out>, cond=<optimized out>, 
opaque=0x7fb2a14442e0) at qemu-char.c:2870
        chr = 0x7fb2a14442e0
        s = 0x7fb2a14363f0
        buf = 
"}\212\352{\377\177\000\000\360`;\236\262\177\000\000\030\003\000\000\000\000\000\000\205N;\236\262\177\000\000\240LB\241\262\177\000\000\263E;\236\262\177\000\000\240LB\241\262\177",
 '\000' <repeats 18 times>, "address@hidden"...
        len = <optimized out>
        size = <optimized out>
#15 0x00007fb29c58799a in g_main_context_dispatch () from 
/lib64/libglib-2.0.so.0
No symbol table info available.
#16 0x00007fb29e34b288 in glib_pollfds_poll () at main-loop.c:209
        context = 0x7fb2a1491140
        pfds = <optimized out>
#17 os_host_main_loop_wait (timeout=<optimized out>) at main-loop.c:254
        ret = 2
        spin_counter = 0
#18 main_loop_wait (nonblocking=<optimized out>) at main-loop.c:503
        ret = 2
        timeout = 4294967295
        timeout_ns = <optimized out>
#19 0x00007fb29e14aa4e in main_loop () at vl.c:1818
        nonblocking = <optimized out>
        last_io = 2
#20 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at 
vl.c:4394
        i = <optimized out>
        snapshot = <optimized out>
        linux_boot = <optimized out>
        initrd_filename = <optimized out>
        kernel_filename = <optimized out>
        kernel_cmdline = <optimized out>
        boot_order = 0x7fb29e3dda67 "cad"
        boot_once = 0x0
        cyls = <optimized out>
        heads = <optimized out>
        secs = <optimized out>
        translation = <optimized out>
        hda_opts = <optimized out>
        opts = <optimized out>
        machine_opts = <optimized out>
        icount_opts = <optimized out>
        olist = <optimized out>
        optind = 78
        optarg = 0x7fb2a14ef8c0 "pc-i440fx-rhel7.2.0"
        loadvm = <optimized out>
        machine_class = <optimized out>
        cpu_model = <optimized out>
        vga_model = 0x0
        qtest_chrdev = <optimized out>
        qtest_log = <optimized out>
        pid_file = <optimized out>
        incoming = <optimized out>
        show_vnc_port = <optimized out>
        defconfig = <optimized out>
        userconfig = 111
        log_mask = <optimized out>
        log_file = <optimized out>
        mem_trace = {malloc = 0x7fb29e238480 <malloc_and_trace>, realloc = 
0x7fb29e238460 <realloc_and_trace>, free = 0x7fb29e238450 <free_and_trace>, 
calloc = 0x0, try_malloc = 0x0, try_realloc = 0x0}
        trace_events = <optimized out>
        trace_file = <optimized out>
        maxram_size = <optimized out>
        ram_slots = <optimized out>
        vmstate_dump_file = <optimized out>
        main_loop_err = 0x0
        __func__ = "main"




-----Ursprüngliche Nachricht-----
Von: Paolo Bonzini [mailto:address@hidden Im Auftrag von Paolo Bonzini
Gesendet: Donnerstag, 19. November 2015 18:02
An: Grundmann, Christian <address@hidden>; 'Dr. David Alan Gilbert' 
<address@hidden>
Cc: 'address@hidden' <address@hidden>; address@hidden
Betreff: Re: WG: [ovirt-users] Segmentation fault in libtcmalloc



On 19/11/2015 17:00, Grundmann, Christian wrote:
> Hi, it seems that using virtio-scsi did the trick, But now the VMs are 
> pausing without an coredump, so the underlying Problem (no storage 
> Error) is not fixed, As I am using Snapshots (and so the disks have to 
> grow very fast) I try if tuning "volume_utilization_percent" and 
> "volume_utilization_chunk_mb" will help 
> (https://access.redhat.com/solutions/130843)

The fix for virtio-blk is probably this patch:
http://article.gmane.org/gmane.comp.emulators.qemu.block/6380/raw

Paolo