Re: [Bug 1902470] Re: migration with TLS-MultiFD is stuck when the dst-libvirtd service restarts

[Zheng Chuan]

On 2020/11/3 4:16, Dr. David Alan Gilbert wrote:
> * zhengchuan (zhengchuan@huawei.com) wrote:
>> Anyone who could help this would be appreciated since we have stuck for 
>> three days:(
>>
>> IIUC, the client (Src) has sent first hello message to sever(Dst), however 
>> due to something happened while restarted libvirtd,
>> The messages is lost, and both of them are waiting which leading to hang 
>> forever, but I could find out how for now.
> 
> If you need to un-break things, I suggest killing the destination might
> free it; but I'm not sure.
> 
Hi, Dave.
Unfortunately, no. After killing the destination, it left Src main migration 
thread stuck at multifd_send_sync_main().

> An interesting question is if we can make migration-cancel work in this
> case.
> 
> Dave
> 
Bad thing happened, since the main qemu thread is stuck at recvmsg(), qemu 
could not respond for libvirt qmp_migrate_cancel:(

During the time, I also found another question is that the Dst socket 
connections are not closed after migration-cancel,
multifd channel would be left with status of CLOSE-WAIT if we look at them 
though 'ss' command.

This is because the multifd_save_cleanup() is simply call 
socket_send_channel_destroy and unref the ioc other than calling
qio_channel_shutdown() in multifd_recv_terminate_threads(), It is not working 
for tls channel.
Simply working around by adding qio_channel_shutdown like this
    for (i = 0; i < migrate_multifd_channels(); i++) {
        MultiFDSendParams *p = &multifd_send_state->params[i];

+       qio_channel_shutdown(p->c, QIO_CHANNEL_SHUTDOWN_BOTH, NULL);
        socket_send_channel_destroy(p->c);
    }
The residual socket is closed, but i doubt if it is the correct solution...

Back to the problem described in this issue, it is still not resolved after 
this working around, but i think it is also a similiar
cleanup issue, and i will dig it out more further...


>> -----Original Message-----
>> From: Qemu-devel 
>> [mailto:qemu-devel-bounces+zhengchuan=huawei.com@nongnu.org] On Behalf Of 
>> Yan Jin
>> Sent: 2020年11月2日 11:12
>> To: qemu-devel@nongnu.org
>> Subject: [Bug 1902470] Re: migration with TLS-MultiFD is stuck when the 
>> dst-libvirtd service restarts
>>
>> ** Description changed:
>>
>>   hi,
>>   
>>   I found that the multi-channel TLS-handshake will be stuck when the dst-
>>   libvirtd restarts, both the src and dst sockets are blocked in recvmsg.
>>   In the meantime, live_migration thread is blocked in
>>   multifd_send_sync_main, so migration cannot be cancelled though src-
>>   libvirt has delivered the QMP command.
>>   
>>   Is there any way to exit migration when the multi-channel TLS-handshake
>> - is stuck? Does setting TLS handshake timeout function take effect?
>> + is stuck? Does setting TLS-handshake timeout function take effect?
>>   
>>   The stack trace are as follows:
>>   
>>   =====src qemu-system-aar stack=====:
>>   #0  0x0000ffff87d6f28c in recvmsg () from target:/usr/lib64/libpthread.so.0
>>   #1  0x0000aaaae3817424 in qio_channel_socket_readv (ioc=0xaaaae9e30a30, 
>> iov=0xffffdb58e8a8, niov=1, fds=0x0, nfds=0x0, errp=0x0) at 
>> ../io/channel-socket.c:502
>>   #2  0x0000aaaae380f468 in qio_channel_readv_full (ioc=0xaaaae9e30a30, 
>> iov=0xffffdb58e8a8, niov=1, fds=0x0, nfds=0x0, errp=0x0) at 
>> ../io/channel.c:66
>>   #3  0x0000aaaae380f9e8 in qio_channel_read (ioc=0xaaaae9e30a30, 
>> buf=0xaaaaea204e9b "\026\003\001\001L\001", buflen=5, errp=0x0) at 
>> ../io/channel.c:217
>>   #4  0x0000aaaae380e7d4 in qio_channel_tls_read_handler (buf=0xaaaaea204e9b 
>> "\026\003\001\001L\001", len=5, opaque=0xfffd38001190) at 
>> ../io/channel-tls.c:53
>>   #5  0x0000aaaae3801114 in qcrypto_tls_session_pull (opaque=0xaaaae99d5700, 
>> buf=0xaaaaea204e9b, len=5) at ../crypto/tlssession.c:89
>>   #6  0x0000ffff8822ed30 in _gnutls_stream_read (ms=0xffffdb58eaac, 
>> pull_func=0xfffd38001870, size=5, bufel=<synthetic pointer>, 
>> session=0xaaaae983cd60) at buffers.c:346
>>   #7  _gnutls_read (ms=0xffffdb58eaac, pull_func=0xfffd38001870, size=5, 
>> bufel=<synthetic pointer>, session=0xaaaae983cd60) at buffers.c:426
>>   #8  _gnutls_io_read_buffered (session=session@entry=0xaaaae983cd60, 
>> total=5, recv_type=recv_type@entry=4294967295, ms=0xffffdb58eaac) at 
>> buffers.c:581
>>   #9  0x0000ffff88224954 in recv_headers (ms=<optimized out>, 
>> record=0xffff883cd000 <gnutls_x509_ext_export_name_constraints@got.plt>, 
>> htype=65535, type=2284006288, record_params=0xaaaae9e22a60, 
>> session=0xaaaae983cd60) at record.c:1163
>>   #10 _gnutls_recv_in_buffers (session=session@entry=0xaaaae983cd60, 
>> type=2284006288, type@entry=GNUTLS_HANDSHAKE, htype=65535, 
>> htype@entry=GNUTLS_HANDSHAKE_HELLO_RETRY_REQUEST, ms=<optimized out>, 
>> ms@entry=0) at record.c:1302
>>   #11 0x0000ffff88230568 in _gnutls_handshake_io_recv_int 
>> (session=session@entry=0xaaaae983cd60, 
>> htype=htype@entry=GNUTLS_HANDSHAKE_HELLO_RETRY_REQUEST, 
>> hsk=hsk@entry=0xffffdb58ec38, optional=optional@entry=1) at buffers.c:1445
>>   #12 0x0000ffff88232b90 in _gnutls_recv_handshake 
>> (session=session@entry=0xaaaae983cd60, 
>> type=type@entry=GNUTLS_HANDSHAKE_HELLO_RETRY_REQUEST, 
>> optional=optional@entry=1, buf=buf@entry=0x0) at handshake.c:1534
>>   #13 0x0000ffff88235b40 in handshake_client 
>> (session=session@entry=0xaaaae983cd60) at handshake.c:2925
>>   #14 0x0000ffff88237824 in gnutls_handshake (session=0xaaaae983cd60) at 
>> handshake.c:2739
>>   #15 0x0000aaaae380213c in qcrypto_tls_session_handshake 
>> (session=0xaaaae99d5700, errp=0xffffdb58ee58) at ../crypto/tlssession.c:493
>>   #16 0x0000aaaae380ea40 in qio_channel_tls_handshake_task 
>> (ioc=0xfffd38001190, task=0xaaaaea61d4e0, context=0x0) at 
>> ../io/channel-tls.c:161
>>   #17 0x0000aaaae380ec60 in qio_channel_tls_handshake (ioc=0xfffd38001190, 
>> func=0xaaaae3394d20 <multifd_tls_outgoing_handshake>, opaque=0xaaaaea189c30, 
>> destroy=0x0, context=0x0) at ../io/channel-tls.c:239
>>   #18 0x0000aaaae3394e78 in multifd_tls_channel_connect (p=0xaaaaea189c30, 
>> ioc=0xaaaae9e30a30, errp=0xffffdb58ef28) at ../migration/multifd.c:782
>>   #19 0x0000aaaae3394f30 in multifd_channel_connect (p=0xaaaaea189c30, 
>> ioc=0xaaaae9e30a30, error=0x0) at ../migration/multifd.c:804
>>   #20 0x0000aaaae33950b8 in multifd_new_send_channel_async 
>> (task=0xaaaaea6855a0, opaque=0xaaaaea189c30) at ../migration/multifd.c:858
>>   #21 0x0000aaaae3810cf8 in qio_task_complete (task=0xaaaaea6855a0) at 
>> ../io/task.c:197
>>   #22 0x0000aaaae381096c in qio_task_thread_result (opaque=0xaaaaea6855a0) 
>> at ../io/task.c:112
>>   #23 0x0000ffff88701df8 in ?? () from target:/usr/lib64/libglib-2.0.so.0
>>   #24 0x0000ffff88705a7c in g_main_context_dispatch () from 
>> target:/usr/lib64/libglib-2.0.so.0
>>   #25 0x0000aaaae3a5a29c in glib_pollfds_poll () at ../util/main-loop.c:221
>>   #26 0x0000aaaae3a5a324 in os_host_main_loop_wait (timeout=0) at 
>> ../util/main-loop.c:244
>>   #27 0x0000aaaae3a5a444 in main_loop_wait (nonblocking=0) at 
>> ../util/main-loop.c:520
>>   #28 0x0000aaaae3696b20 in qemu_main_loop () at ../softmmu/vl.c:1677
>>   #29 0x0000aaaae30949e4 in main (argc=81, argv=0xffffdb58f2c8, 
>> envp=0xffffdb58f558) at ../softmmu/main.c:50
>>   
>>   =====src live_migration stack=====:
>>   #0  0x0000ffff87d6a5d8 in pthread_cond_wait () from 
>> target:/usr/lib64/libpthread.so.0
>>   #1  0x0000aaaae3a5f3ec in qemu_sem_wait (sem=0xaaaaea189d40) at 
>> ../util/qemu-thread-posix.c:328
>>   #2  0x0000aaaae3394838 in multifd_send_sync_main (f=0xaaaae983f0e0) at 
>> ../migration/multifd.c:638
>>   #3  0x0000aaaae37de310 in ram_save_setup (f=0xaaaae983f0e0, 
>> opaque=0xaaaae4198708 <ram_state>) at ../migration/ram.c:2588
>>   #4  0x0000aaaae31cf7ac in qemu_savevm_state_setup (f=0xaaaae983f0e0) at 
>> ../migration/savevm.c:1176
>>   #5  0x0000aaaae3248360 in migration_thread (opaque=0xaaaae9829f20) at 
>> ../migration/migration.c:3521
>>   #6  0x0000aaaae3a5f8fc in qemu_thread_start (args=0xaaaaea513ee0) at 
>> ../util/qemu-thread-posix.c:521
>>   #7  0x0000ffff87d647ac in ?? () from target:/usr/lib64/libpthread.so.0
>>   #8  0x0000ffff87cba6ec in ?? () from target:/usr/lib64/libc.so.6
>>   
>>   =====dst qemu-system-aar stack=====:
>>   #0  0x0000ffff7f17d28c in recvmsg () from target:/usr/lib64/libpthread.so.0
>>   #1  0x0000aaaae263a424 in qio_channel_socket_readv (ioc=0xaaaaf998a800, 
>> iov=0xfffff5d22f78, niov=1, fds=0x0, nfds=0x0, errp=0x0) at 
>> ../io/channel-socket.c:502
>>   #2  0x0000aaaae2632468 in qio_channel_readv_full (ioc=0xaaaaf998a800, 
>> iov=0xfffff5d22f78, niov=1, fds=0x0, nfds=0x0, errp=0x0) at 
>> ../io/channel.c:66
>>   #3  0x0000aaaae26329e8 in qio_channel_read (ioc=0xaaaaf998a800, 
>> buf=0xaaaafa926dbb 
>> "q\024\335\365ȣ'\221,\\\357\246w\253\242ѠصI\247(N(K=\256\316DH\227QNf\371\"\271\017\226^\223\026\373\245z\255\227\025R.\244\205\254\002\031T\033\312:h\226\aݔ\204Ԫ\324\351K\341\365\247\032\354+\277\005O'*l\301cXx\340~?\346\b\324k\225\223D\276\252\376\257_0\036\223\022\006\212D|7h\257\226\300&n','\005zL\203M͆\023\213\237(o\272\025_\305s\372\362\351\002\367Ph\016\347\371E\n\030Y\340\002\r\362^&`\021\203}\353\324A\340ҳ(\207]\300l}h\026\037H\372\n=\"C\024\t\200\325\334&=\333>\212ƏE\214]_\372\264]"...,
>>  buflen=5, errp=0x0) at ../io/channel.c:217
>>   #4  0x0000aaaae26317d4 in qio_channel_tls_read_handler (buf=0xaaaafa926dbb 
>> "q\024\335\365ȣ'\221,\\\357\246w\253\242ѠصI\247(N(K=\256\316DH\227QNf\371\"\271\017\226^\223\026\373\245z\255\227\025R.\244\205\254\002\031T\033\312:h\226\aݔ\204Ԫ\324\351K\341\365\247\032\354+\277\005O'*l\301cXx\340~?\346\b\324k\225\223D\276\252\376\257_0\036\223\022\006\212D|7h\257\226\300&n','\005zL\203M͆\023\213\237(o\272\025_\305s\372\362\351\002\367Ph\016\347\371E\n\030Y\340\002\r\362^&`\021\203}\353\324A\340ҳ(\207]\300l}h\026\037H\372\n=\"C\024\t\200\325\334&=\333>\212ƏE\214]_\372\264]"...,
>>  len=5, opaque=0xaaaaf9c4c400) at ../io/channel-tls.c:53
>>   #5  0x0000aaaae2624114 in qcrypto_tls_session_pull (opaque=0xaaaafa4a3d90, 
>> buf=0xaaaafa926dbb, len=5) at ../crypto/tlssession.c:89
>>   #6  0x0000ffff7f63cd30 in _gnutls_stream_read (ms=0xfffff5d2317c, 
>> pull_func=0xaaaafa81a380, size=5, bufel=<synthetic pointer>, 
>> session=0xaaaafa58b9d0) at buffers.c:346
>>   #7  _gnutls_read (ms=0xfffff5d2317c, pull_func=0xaaaafa81a380, size=5, 
>> bufel=<synthetic pointer>, session=0xaaaafa58b9d0) at buffers.c:426
>>   #8  _gnutls_io_read_buffered (session=session@entry=0xaaaafa58b9d0, 
>> total=5, recv_type=recv_type@entry=4294967295, ms=0xfffff5d2317c) at 
>> buffers.c:581
>>   #9  0x0000ffff7f632954 in recv_headers (ms=<optimized out>, 
>> record=0x1ee2a9fa78, htype=65535, type=2137262992, 
>> record_params=0xaaaafa4b71a0, session=0xaaaafa58b9d0) at record.c:1163
>>   #10 _gnutls_recv_in_buffers (session=session@entry=0xaaaafa58b9d0, 
>> type=2137262992, type@entry=GNUTLS_HANDSHAKE, htype=65535, 
>> htype@entry=GNUTLS_HANDSHAKE_CLIENT_HELLO, ms=<optimized out>, ms@entry=0) 
>> at record.c:1302
>>   #11 0x0000ffff7f63e568 in _gnutls_handshake_io_recv_int 
>> (session=session@entry=0xaaaafa58b9d0, 
>> htype=htype@entry=GNUTLS_HANDSHAKE_CLIENT_HELLO, 
>> hsk=hsk@entry=0xfffff5d23308, optional=optional@entry=0) at buffers.c:1445
>>   #12 0x0000ffff7f640b90 in _gnutls_recv_handshake 
>> (session=session@entry=0xaaaafa58b9d0, 
>> type=type@entry=GNUTLS_HANDSHAKE_CLIENT_HELLO, optional=optional@entry=0, 
>> buf=buf@entry=0x0) at handshake.c:1534
>>   #13 0x0000ffff7f645f18 in handshake_server (session=<optimized out>) at 
>> handshake.c:3351
>>   #14 gnutls_handshake (session=0xaaaafa58b9d0) at handshake.c:2742
>>   #15 0x0000aaaae262513c in qcrypto_tls_session_handshake 
>> (session=0xaaaafa4a3d90, errp=0xfffff5d23478) at ../crypto/tlssession.c:493
>>   #16 0x0000aaaae2631a40 in qio_channel_tls_handshake_task 
>> (ioc=0xaaaaf9c4c400, task=0xaaaafa70e600, context=0x0) at 
>> ../io/channel-tls.c:161
>>   #17 0x0000aaaae2631c60 in qio_channel_tls_handshake (ioc=0xaaaaf9c4c400, 
>> func=0xaaaae20d4b58 <migration_tls_incoming_handshake>, opaque=0x0, 
>> destroy=0x0, context=0x0) at ../io/channel-tls.c:239
>>   #18 0x0000aaaae20d4ca8 in migration_tls_channel_process_incoming 
>> (s=0xaaaaf9b2ef20, ioc=0xaaaaf998a800, errp=0xfffff5d23548) at 
>> ../migration/tls.c:103
>>   #19 0x0000aaaae20f9f7c in migration_channel_process_incoming 
>> (ioc=0xaaaaf998a800) at ../migration/channel.c:42
>>   #20 0x0000aaaae1f484a8 in socket_accept_incoming_migration 
>> (listener=0xffff64007a40, cioc=0xaaaaf998a800, opaque=0x0) at 
>> ../migration/socket.c:130
>>   #21 0x0000aaaae2638570 in qio_net_listener_channel_func 
>> (ioc=0xaaaafa410600, condition=G_IO_IN, opaque=0xffff64007a40) at 
>> ../io/net-listener.c:54
>>   #22 0x0000aaaae263ac4c in qio_channel_fd_source_dispatch 
>> (source=0xaaaafa81a380, callback=0xaaaae26384f8 
>> <qio_net_listener_channel_func>, user_data=0xffff64007a40) at 
>> ../io/channel-watch.c:84
>>   #23 0x0000ffff7fb13a7c in g_main_context_dispatch () from 
>> target:/usr/lib64/libglib-2.0.so.0
>>   #24 0x0000aaaae287d29c in glib_pollfds_poll () at ../util/main-loop.c:221
>>   #25 0x0000aaaae287d324 in os_host_main_loop_wait (timeout=571000000) at 
>> ../util/main-loop.c:244
>>   #26 0x0000aaaae287d444 in main_loop_wait (nonblocking=0) at 
>> ../util/main-loop.c:520
>>   #27 0x0000aaaae24b9b20 in qemu_main_loop () at ../softmmu/vl.c:1677
>>   #28 0x0000aaaae1eb79e4 in main (argc=83, argv=0xfffff5d238c8, 
>> envp=0xfffff5d23b68) at ../softmmu/main.c:50
>>
>> --
>> You received this bug notification because you are a member of qemu- 
>> devel-ml, which is subscribed to QEMU.
>> https://bugs.launchpad.net/bugs/1902470
>>
>> Title:
>>   migration with TLS-MultiFD is stuck when the dst-libvirtd service
>>   restarts
>>
>> Status in QEMU:
>>   New
>>
>> Bug description:
>>   hi,
>>
>>   I found that the multi-channel TLS-handshake will be stuck when the
>>   dst-libvirtd restarts, both the src and dst sockets are blocked in
>>   recvmsg. In the meantime, live_migration thread is blocked in
>>   multifd_send_sync_main, so migration cannot be cancelled though src-
>>   libvirt has delivered the QMP command.
>>
>>   Is there any way to exit migration when the multi-channel TLS-
>>   handshake is stuck? Does setting TLS-handshake timeout function take
>>   effect?
>>
>>   The stack trace are as follows:
>>
>>   =====src qemu-system-aar stack=====:
>>   #0  0x0000ffff87d6f28c in recvmsg () from target:/usr/lib64/libpthread.so.0
>>   #1  0x0000aaaae3817424 in qio_channel_socket_readv (ioc=0xaaaae9e30a30, 
>> iov=0xffffdb58e8a8, niov=1, fds=0x0, nfds=0x0, errp=0x0) at 
>> ../io/channel-socket.c:502
>>   #2  0x0000aaaae380f468 in qio_channel_readv_full (ioc=0xaaaae9e30a30, 
>> iov=0xffffdb58e8a8, niov=1, fds=0x0, nfds=0x0, errp=0x0) at 
>> ../io/channel.c:66
>>   #3  0x0000aaaae380f9e8 in qio_channel_read (ioc=0xaaaae9e30a30, 
>> buf=0xaaaaea204e9b "\026\003\001\001L\001", buflen=5, errp=0x0) at 
>> ../io/channel.c:217
>>   #4  0x0000aaaae380e7d4 in qio_channel_tls_read_handler (buf=0xaaaaea204e9b 
>> "\026\003\001\001L\001", len=5, opaque=0xfffd38001190) at 
>> ../io/channel-tls.c:53
>>   #5  0x0000aaaae3801114 in qcrypto_tls_session_pull (opaque=0xaaaae99d5700, 
>> buf=0xaaaaea204e9b, len=5) at ../crypto/tlssession.c:89
>>   #6  0x0000ffff8822ed30 in _gnutls_stream_read (ms=0xffffdb58eaac, 
>> pull_func=0xfffd38001870, size=5, bufel=<synthetic pointer>, 
>> session=0xaaaae983cd60) at buffers.c:346
>>   #7  _gnutls_read (ms=0xffffdb58eaac, pull_func=0xfffd38001870, size=5, 
>> bufel=<synthetic pointer>, session=0xaaaae983cd60) at buffers.c:426
>>   #8  _gnutls_io_read_buffered (session=session@entry=0xaaaae983cd60, 
>> total=5, recv_type=recv_type@entry=4294967295, ms=0xffffdb58eaac) at 
>> buffers.c:581
>>   #9  0x0000ffff88224954 in recv_headers (ms=<optimized out>, 
>> record=0xffff883cd000 <gnutls_x509_ext_export_name_constraints@got.plt>, 
>> htype=65535, type=2284006288, record_params=0xaaaae9e22a60, 
>> session=0xaaaae983cd60) at record.c:1163
>>   #10 _gnutls_recv_in_buffers (session=session@entry=0xaaaae983cd60, 
>> type=2284006288, type@entry=GNUTLS_HANDSHAKE, htype=65535, 
>> htype@entry=GNUTLS_HANDSHAKE_HELLO_RETRY_REQUEST, ms=<optimized out>, 
>> ms@entry=0) at record.c:1302
>>   #11 0x0000ffff88230568 in _gnutls_handshake_io_recv_int 
>> (session=session@entry=0xaaaae983cd60, 
>> htype=htype@entry=GNUTLS_HANDSHAKE_HELLO_RETRY_REQUEST, 
>> hsk=hsk@entry=0xffffdb58ec38, optional=optional@entry=1) at buffers.c:1445
>>   #12 0x0000ffff88232b90 in _gnutls_recv_handshake 
>> (session=session@entry=0xaaaae983cd60, 
>> type=type@entry=GNUTLS_HANDSHAKE_HELLO_RETRY_REQUEST, 
>> optional=optional@entry=1, buf=buf@entry=0x0) at handshake.c:1534
>>   #13 0x0000ffff88235b40 in handshake_client 
>> (session=session@entry=0xaaaae983cd60) at handshake.c:2925
>>   #14 0x0000ffff88237824 in gnutls_handshake (session=0xaaaae983cd60) at 
>> handshake.c:2739
>>   #15 0x0000aaaae380213c in qcrypto_tls_session_handshake 
>> (session=0xaaaae99d5700, errp=0xffffdb58ee58) at ../crypto/tlssession.c:493
>>   #16 0x0000aaaae380ea40 in qio_channel_tls_handshake_task 
>> (ioc=0xfffd38001190, task=0xaaaaea61d4e0, context=0x0) at 
>> ../io/channel-tls.c:161
>>   #17 0x0000aaaae380ec60 in qio_channel_tls_handshake (ioc=0xfffd38001190, 
>> func=0xaaaae3394d20 <multifd_tls_outgoing_handshake>, opaque=0xaaaaea189c30, 
>> destroy=0x0, context=0x0) at ../io/channel-tls.c:239
>>   #18 0x0000aaaae3394e78 in multifd_tls_channel_connect (p=0xaaaaea189c30, 
>> ioc=0xaaaae9e30a30, errp=0xffffdb58ef28) at ../migration/multifd.c:782
>>   #19 0x0000aaaae3394f30 in multifd_channel_connect (p=0xaaaaea189c30, 
>> ioc=0xaaaae9e30a30, error=0x0) at ../migration/multifd.c:804
>>   #20 0x0000aaaae33950b8 in multifd_new_send_channel_async 
>> (task=0xaaaaea6855a0, opaque=0xaaaaea189c30) at ../migration/multifd.c:858
>>   #21 0x0000aaaae3810cf8 in qio_task_complete (task=0xaaaaea6855a0) at 
>> ../io/task.c:197
>>   #22 0x0000aaaae381096c in qio_task_thread_result (opaque=0xaaaaea6855a0) 
>> at ../io/task.c:112
>>   #23 0x0000ffff88701df8 in ?? () from target:/usr/lib64/libglib-2.0.so.0
>>   #24 0x0000ffff88705a7c in g_main_context_dispatch () from 
>> target:/usr/lib64/libglib-2.0.so.0
>>   #25 0x0000aaaae3a5a29c in glib_pollfds_poll () at ../util/main-loop.c:221
>>   #26 0x0000aaaae3a5a324 in os_host_main_loop_wait (timeout=0) at 
>> ../util/main-loop.c:244
>>   #27 0x0000aaaae3a5a444 in main_loop_wait (nonblocking=0) at 
>> ../util/main-loop.c:520
>>   #28 0x0000aaaae3696b20 in qemu_main_loop () at ../softmmu/vl.c:1677
>>   #29 0x0000aaaae30949e4 in main (argc=81, argv=0xffffdb58f2c8, 
>> envp=0xffffdb58f558) at ../softmmu/main.c:50
>>
>>   =====src live_migration stack=====:
>>   #0  0x0000ffff87d6a5d8 in pthread_cond_wait () from 
>> target:/usr/lib64/libpthread.so.0
>>   #1  0x0000aaaae3a5f3ec in qemu_sem_wait (sem=0xaaaaea189d40) at 
>> ../util/qemu-thread-posix.c:328
>>   #2  0x0000aaaae3394838 in multifd_send_sync_main (f=0xaaaae983f0e0) at 
>> ../migration/multifd.c:638
>>   #3  0x0000aaaae37de310 in ram_save_setup (f=0xaaaae983f0e0, 
>> opaque=0xaaaae4198708 <ram_state>) at ../migration/ram.c:2588
>>   #4  0x0000aaaae31cf7ac in qemu_savevm_state_setup (f=0xaaaae983f0e0) at 
>> ../migration/savevm.c:1176
>>   #5  0x0000aaaae3248360 in migration_thread (opaque=0xaaaae9829f20) at 
>> ../migration/migration.c:3521
>>   #6  0x0000aaaae3a5f8fc in qemu_thread_start (args=0xaaaaea513ee0) at 
>> ../util/qemu-thread-posix.c:521
>>   #7  0x0000ffff87d647ac in ?? () from target:/usr/lib64/libpthread.so.0
>>   #8  0x0000ffff87cba6ec in ?? () from target:/usr/lib64/libc.so.6
>>
>>   =====dst qemu-system-aar stack=====:
>>   #0  0x0000ffff7f17d28c in recvmsg () from target:/usr/lib64/libpthread.so.0
>>   #1  0x0000aaaae263a424 in qio_channel_socket_readv (ioc=0xaaaaf998a800, 
>> iov=0xfffff5d22f78, niov=1, fds=0x0, nfds=0x0, errp=0x0) at 
>> ../io/channel-socket.c:502
>>   #2  0x0000aaaae2632468 in qio_channel_readv_full (ioc=0xaaaaf998a800, 
>> iov=0xfffff5d22f78, niov=1, fds=0x0, nfds=0x0, errp=0x0) at 
>> ../io/channel.c:66
>>   #3  0x0000aaaae26329e8 in qio_channel_read (ioc=0xaaaaf998a800, 
>> buf=0xaaaafa926dbb 
>> "q\024\335\365ȣ'\221,\\\357\246w\253\242ѠصI\247(N(K=\256\316DH\227QNf\371\"\271\017\226^\223\026\373\245z\255\227\025R.\244\205\254\002\031T\033\312:h\226\aݔ\204Ԫ\324\351K\341\365\247\032\354+\277\005O'*l\301cXx\340~?\346\b\324k\225\223D\276\252\376\257_0\036\223\022\006\212D|7h\257\226\300&n','\005zL\203M͆\023\213\237(o\272\025_\305s\372\362\351\002\367Ph\016\347\371E\n\030Y\340\002\r\362^&`\021\203}\353\324A\340ҳ(\207]\300l}h\026\037H\372\n=\"C\024\t\200\325\334&=\333>\212ƏE\214]_\372\264]"...,
>>  buflen=5, errp=0x0) at ../io/channel.c:217
>>   #4  0x0000aaaae26317d4 in qio_channel_tls_read_handler (buf=0xaaaafa926dbb 
>> "q\024\335\365ȣ'\221,\\\357\246w\253\242ѠصI\247(N(K=\256\316DH\227QNf\371\"\271\017\226^\223\026\373\245z\255\227\025R.\244\205\254\002\031T\033\312:h\226\aݔ\204Ԫ\324\351K\341\365\247\032\354+\277\005O'*l\301cXx\340~?\346\b\324k\225\223D\276\252\376\257_0\036\223\022\006\212D|7h\257\226\300&n','\005zL\203M͆\023\213\237(o\272\025_\305s\372\362\351\002\367Ph\016\347\371E\n\030Y\340\002\r\362^&`\021\203}\353\324A\340ҳ(\207]\300l}h\026\037H\372\n=\"C\024\t\200\325\334&=\333>\212ƏE\214]_\372\264]"...,
>>  len=5, opaque=0xaaaaf9c4c400) at ../io/channel-tls.c:53
>>   #5  0x0000aaaae2624114 in qcrypto_tls_session_pull (opaque=0xaaaafa4a3d90, 
>> buf=0xaaaafa926dbb, len=5) at ../crypto/tlssession.c:89
>>   #6  0x0000ffff7f63cd30 in _gnutls_stream_read (ms=0xfffff5d2317c, 
>> pull_func=0xaaaafa81a380, size=5, bufel=<synthetic pointer>, 
>> session=0xaaaafa58b9d0) at buffers.c:346
>>   #7  _gnutls_read (ms=0xfffff5d2317c, pull_func=0xaaaafa81a380, size=5, 
>> bufel=<synthetic pointer>, session=0xaaaafa58b9d0) at buffers.c:426
>>   #8  _gnutls_io_read_buffered (session=session@entry=0xaaaafa58b9d0, 
>> total=5, recv_type=recv_type@entry=4294967295, ms=0xfffff5d2317c) at 
>> buffers.c:581
>>   #9  0x0000ffff7f632954 in recv_headers (ms=<optimized out>, 
>> record=0x1ee2a9fa78, htype=65535, type=2137262992, 
>> record_params=0xaaaafa4b71a0, session=0xaaaafa58b9d0) at record.c:1163
>>   #10 _gnutls_recv_in_buffers (session=session@entry=0xaaaafa58b9d0, 
>> type=2137262992, type@entry=GNUTLS_HANDSHAKE, htype=65535, 
>> htype@entry=GNUTLS_HANDSHAKE_CLIENT_HELLO, ms=<optimized out>, ms@entry=0) 
>> at record.c:1302
>>   #11 0x0000ffff7f63e568 in _gnutls_handshake_io_recv_int 
>> (session=session@entry=0xaaaafa58b9d0, 
>> htype=htype@entry=GNUTLS_HANDSHAKE_CLIENT_HELLO, 
>> hsk=hsk@entry=0xfffff5d23308, optional=optional@entry=0) at buffers.c:1445
>>   #12 0x0000ffff7f640b90 in _gnutls_recv_handshake 
>> (session=session@entry=0xaaaafa58b9d0, 
>> type=type@entry=GNUTLS_HANDSHAKE_CLIENT_HELLO, optional=optional@entry=0, 
>> buf=buf@entry=0x0) at handshake.c:1534
>>   #13 0x0000ffff7f645f18 in handshake_server (session=<optimized out>) at 
>> handshake.c:3351
>>   #14 gnutls_handshake (session=0xaaaafa58b9d0) at handshake.c:2742
>>   #15 0x0000aaaae262513c in qcrypto_tls_session_handshake 
>> (session=0xaaaafa4a3d90, errp=0xfffff5d23478) at ../crypto/tlssession.c:493
>>   #16 0x0000aaaae2631a40 in qio_channel_tls_handshake_task 
>> (ioc=0xaaaaf9c4c400, task=0xaaaafa70e600, context=0x0) at 
>> ../io/channel-tls.c:161
>>   #17 0x0000aaaae2631c60 in qio_channel_tls_handshake (ioc=0xaaaaf9c4c400, 
>> func=0xaaaae20d4b58 <migration_tls_incoming_handshake>, opaque=0x0, 
>> destroy=0x0, context=0x0) at ../io/channel-tls.c:239
>>   #18 0x0000aaaae20d4ca8 in migration_tls_channel_process_incoming 
>> (s=0xaaaaf9b2ef20, ioc=0xaaaaf998a800, errp=0xfffff5d23548) at 
>> ../migration/tls.c:103
>>   #19 0x0000aaaae20f9f7c in migration_channel_process_incoming 
>> (ioc=0xaaaaf998a800) at ../migration/channel.c:42
>>   #20 0x0000aaaae1f484a8 in socket_accept_incoming_migration 
>> (listener=0xffff64007a40, cioc=0xaaaaf998a800, opaque=0x0) at 
>> ../migration/socket.c:130
>>   #21 0x0000aaaae2638570 in qio_net_listener_channel_func 
>> (ioc=0xaaaafa410600, condition=G_IO_IN, opaque=0xffff64007a40) at 
>> ../io/net-listener.c:54
>>   #22 0x0000aaaae263ac4c in qio_channel_fd_source_dispatch 
>> (source=0xaaaafa81a380, callback=0xaaaae26384f8 
>> <qio_net_listener_channel_func>, user_data=0xffff64007a40) at 
>> ../io/channel-watch.c:84
>>   #23 0x0000ffff7fb13a7c in g_main_context_dispatch () from 
>> target:/usr/lib64/libglib-2.0.so.0
>>   #24 0x0000aaaae287d29c in glib_pollfds_poll () at ../util/main-loop.c:221
>>   #25 0x0000aaaae287d324 in os_host_main_loop_wait (timeout=571000000) at 
>> ../util/main-loop.c:244
>>   #26 0x0000aaaae287d444 in main_loop_wait (nonblocking=0) at 
>> ../util/main-loop.c:520
>>   #27 0x0000aaaae24b9b20 in qemu_main_loop () at ../softmmu/vl.c:1677
>>   #28 0x0000aaaae1eb79e4 in main (argc=83, argv=0xfffff5d238c8, 
>> envp=0xfffff5d23b68) at ../softmmu/main.c:50
>>
>> To manage notifications about this bug go to:
>> https://bugs.launchpad.net/qemu/+bug/1902470/+subscriptions
>>

-- 
Regards.
Chuan