Re: bad virsh save /dev/null performance (600 MiB/s max)

[Daniel P . Berrangé]

On Wed, Mar 09, 2022 at 06:37:42PM +0000, Daniel P. Berrangé wrote:
> On Wed, Mar 09, 2022 at 07:27:12PM +0100, Claudio Fontana wrote:
> > 
> > One difference I could see looking at the qmp commands issued by libvirt in 
> > the "virsh save" case,
> > is "detach:true" in the migration command (which seems to have no effect in 
> > qemu),
> 
> That is a bug in libvirt - it should not be setting that in QMP.
> 
> To quote the QAPI spec for 'migrate'
> 
>   # @detach: this argument exists only for compatibility reasons and
>   #          is ignored by QEMU
> 
> 
> > 
> > 
> > and maybe more interestingly this stuff about the "fd":
> > 
> > 
> > 2022-03-09 17:29:34.247+0000: 20390: info : qemuMonitorSend:995 : 
> > QEMU_MONITOR_SEND_MSG: mon=0x7faa9003ebf0 
> > msg={"execute":"getfd","arguments":{"fdname":"migrate"},"id":"libvirt-390"}^M
> >  fd=34
> > 2022-03-09 17:29:34.247+0000: 20387: info : qemuMonitorIOWrite:452 : 
> > QEMU_MONITOR_IO_WRITE: mon=0x7faa9003ebf0 
> > buf={"execute":"getfd","arguments":{"fdname":"migrate"},"id":"libvirt-390"}^M
> >  len=73 ret=73 errno=0
> > 2022-03-09 17:29:34.247+0000: 20387: info : qemuMonitorIOWrite:457 : 
> > QEMU_MONITOR_IO_SEND_FD: mon=0x7faa9003ebf0 fd=34 ret=73 errno=0
> > 2022-03-09 17:29:34.248+0000: 20387: info : 
> > qemuMonitorJSONIOProcessLine:240 : QEMU_MONITOR_RECV_REPLY: 
> > mon=0x7faa9003ebf0 reply={"return": {}, "id": "libvirt-390"}
> > 2022-03-09 17:29:34.249+0000: 20390: info : qemuMonitorSend:995 : 
> > QEMU_MONITOR_SEND_MSG: mon=0x7faa9003ebf0 
> > msg={"execute":"migrate","arguments":{"detach":true,"blk":false,"inc":false,"uri":"fd:migrate"},"id":"libvirt-391"}^M
> >  fd=-1
> > 2022-03-09 17:29:34.249+0000: 20387: info : qemuMonitorIOWrite:452 : 
> > QEMU_MONITOR_IO_WRITE: mon=0x7faa9003ebf0 
> > buf={"execute":"migrate","arguments":{"detach":true,"blk":false,"inc":false,"uri":"fd:migrate"},"id":"libvirt-391"}^M
> >  len=113 ret=113 errno=0
> > 
> > 
> > in qemu I am currently looking at the code in migration/socket.c
> > vs the code in migration/fd.c , wonder if the difference would
> > stem from there..
> 
> When saving to a file, libvirt passes in a pre-opened FD for
> QEU to use. IIRC this should always be a pipe FD connected to
> libvirt's iohelper program, sometimes indirectly via a compression
> program.

It occurs to me that pipes and sockets likely use very different
buffer sizes, with sockets larger IIUC, and buffer sizes can impact
throughput. Could be worth exploring a change to libvirt code to
use socketpair() instead of pipe() to see if it impacts performance
in any meaningful way, or playing with fcntl(F_SETPIPE_SZ)

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|