Re: [Qemu-devel] [PULL v2 for-2.4 v2 5/7] AioContext: fix broken ctx->dispatching optimization

[Stefan Hajnoczi]

On Fri, Jul 24, 2015 at 7:46 AM, Cornelia Huck <address@hidden> wrote:
> On Thu, 23 Jul 2015 21:29:05 +0200
> Christian Borntraeger <address@hidden> wrote:
>
>> Am 23.07.2015 um 21:25 schrieb Paolo Bonzini:
>> >
>> >
>> > On 23/07/2015 21:19, Christian Borntraeger wrote:
>> >> Am 23.07.2015 um 20:19 schrieb Paolo Bonzini:
>> >>>
>> >>>
>> >>> On 23/07/2015 19:20, Paolo Bonzini wrote:
>> >>>>
>> >>>>
>> >>>> On 23/07/2015 16:14, Cornelia Huck wrote:
>> >>>>> (gdb) bt
>> >>>>> #0  0x000003fffc5871b4 in pthread_cond_wait@@GLIBC_2.3.2 ()
>> >>>>>    from /lib64/libpthread.so.0
>> >>>>> #1  0x000000008024cfca in qemu_cond_wait (address@hidden,
>> >>>>>     address@hidden)
>> >>>>>     at /data/git/yyy/qemu/util/qemu-thread-posix.c:132
>> >>>>> #2  0x000000008025e83a in rfifolock_lock (r=0x9717d920)
>> >>>>>     at /data/git/yyy/qemu/util/rfifolock.c:59
>> >>>>> #3  0x00000000801b78fa in aio_context_acquire (ctx=<optimized out>)
>> >>>>>     at /data/git/yyy/qemu/async.c:331
>> >>>>> #4  0x000000008007ceb4 in virtio_blk_data_plane_start (s=0x9717d710)
>> >>>>>     at /data/git/yyy/qemu/hw/block/dataplane/virtio-blk.c:285
>> >>>>> #5  0x000000008007c64a in virtio_blk_handle_output (vdev=<optimized 
>> >>>>> out>,
>> >>>>>     vq=<optimized out>) at /data/git/yyy/qemu/hw/block/virtio-blk.c:599
>> >>>>> #6  0x00000000801c56dc in qemu_iohandler_poll (pollfds=0x97142800,
>> >>>>>     address@hidden) at /data/git/yyy/qemu/iohandler.c:126
>> >>>>> #7  0x00000000801c5178 in main_loop_wait (nonblocking=<optimized out>)
>> >>>>>     at /data/git/yyy/qemu/main-loop.c:494
>> >>>>> #8  0x0000000080013ee2 in main_loop () at /data/git/yyy/qemu/vl.c:1902
>> >>>>> #9  main (argc=<optimized out>, argv=<optimized out>, envp=<optimized 
>> >>>>> out>)
>> >>>>>     at /data/git/yyy/qemu/vl.c:4653
>> >>>>>
>> >>>>> I've stripped down the setup to the following commandline:
>> >>>>>
>> >>>>> /data/git/yyy/qemu/build/s390x-softmmu/qemu-system-s390x  -machine
>> >>>>> s390-ccw-virtio-2.4,accel=kvm,usb=off -m 1024 -smp
>> >>>>> 4,sockets=4,cores=1,threads=1 -nographic -drive
>> >>>>> file=/dev/sda,if=none,id=drive-virtio-disk0,format=raw,serial=ccwzfcp1,cache=none,aio=native
>> >>>>> -device
>> >>>>> virtio-blk-ccw,scsi=off,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1,x-data-plane=on
>> >>>>
>> >>>> What's the backtrace like for the other threads?  This is almost
>> >>>> definitely a latent bug somewhere else.
>> >>>
>> >>> BTW, I can reproduce this---I'm asking because I cannot even attach gdb
>> >>> to the hung process.
>> >>>
>> >>> The simplest workaround is to reintroduce commit a0710f7995 (iothread:
>> >>> release iothread around aio_poll, 2015-02-20), though it also comes with
>> >>> some risk.  It avoids the bug because it limits the contention on the
>> >>> RFifoLock.
>> >>
>> >> I can reproduce this with the following backtrace (with --enable-debug 
>> >> info added qemu is the tag v2.4.0-rc2)
>> >
>> > Can you check that cherry-picking a0710f7995 works for you?
>> >
>>
>> Yes, this seems to help for me. Conny, can you verify?
>
> I can't reproduce the hang with this on top, so seems to help.

Thanks Christian and Cornelia.  I'm sending a fix.

Stefan