Re: bad virsh save /dev/null performance (600 MiB/s max)

[Claudio Fontana]

On 3/9/22 7:46 PM, Daniel P. Berrangé wrote:
> On Wed, Mar 09, 2022 at 06:37:42PM +0000, Daniel P. Berrangé wrote:
>> On Wed, Mar 09, 2022 at 07:27:12PM +0100, Claudio Fontana wrote:
>>>
>>> One difference I could see looking at the qmp commands issued by libvirt in 
>>> the "virsh save" case,
>>> is "detach:true" in the migration command (which seems to have no effect in 
>>> qemu),
>>
>> That is a bug in libvirt - it should not be setting that in QMP.
>>
>> To quote the QAPI spec for 'migrate'
>>
>>   # @detach: this argument exists only for compatibility reasons and
>>   #          is ignored by QEMU
>>
>>
>>>
>>>
>>> and maybe more interestingly this stuff about the "fd":
>>>
>>>
>>> 2022-03-09 17:29:34.247+0000: 20390: info : qemuMonitorSend:995 : 
>>> QEMU_MONITOR_SEND_MSG: mon=0x7faa9003ebf0 
>>> msg={"execute":"getfd","arguments":{"fdname":"migrate"},"id":"libvirt-390"}^M
>>>  fd=34
>>> 2022-03-09 17:29:34.247+0000: 20387: info : qemuMonitorIOWrite:452 : 
>>> QEMU_MONITOR_IO_WRITE: mon=0x7faa9003ebf0 
>>> buf={"execute":"getfd","arguments":{"fdname":"migrate"},"id":"libvirt-390"}^M
>>>  len=73 ret=73 errno=0
>>> 2022-03-09 17:29:34.247+0000: 20387: info : qemuMonitorIOWrite:457 : 
>>> QEMU_MONITOR_IO_SEND_FD: mon=0x7faa9003ebf0 fd=34 ret=73 errno=0
>>> 2022-03-09 17:29:34.248+0000: 20387: info : 
>>> qemuMonitorJSONIOProcessLine:240 : QEMU_MONITOR_RECV_REPLY: 
>>> mon=0x7faa9003ebf0 reply={"return": {}, "id": "libvirt-390"}
>>> 2022-03-09 17:29:34.249+0000: 20390: info : qemuMonitorSend:995 : 
>>> QEMU_MONITOR_SEND_MSG: mon=0x7faa9003ebf0 
>>> msg={"execute":"migrate","arguments":{"detach":true,"blk":false,"inc":false,"uri":"fd:migrate"},"id":"libvirt-391"}^M
>>>  fd=-1
>>> 2022-03-09 17:29:34.249+0000: 20387: info : qemuMonitorIOWrite:452 : 
>>> QEMU_MONITOR_IO_WRITE: mon=0x7faa9003ebf0 
>>> buf={"execute":"migrate","arguments":{"detach":true,"blk":false,"inc":false,"uri":"fd:migrate"},"id":"libvirt-391"}^M
>>>  len=113 ret=113 errno=0
>>>
>>>
>>> in qemu I am currently looking at the code in migration/socket.c
>>> vs the code in migration/fd.c , wonder if the difference would
>>> stem from there..
>>
>> When saving to a file, libvirt passes in a pre-opened FD for
>> QEU to use. IIRC this should always be a pipe FD connected to
>> libvirt's iohelper program, sometimes indirectly via a compression
>> program.
> 
> It occurs to me that pipes and sockets likely use very different
> buffer sizes, with sockets larger IIUC, and buffer sizes can impact
> throughput. Could be worth exploring a change to libvirt code to
> use socketpair() instead of pipe() to see if it impacts performance
> in any meaningful way, or playing with fcntl(F_SETPIPE_SZ)
> 
> With regards,
> Daniel
> 

Hello Daniel,

the experiment with F_SETPIPE_SZ was successful, transfer speeds are 
dramatically increased. Still digging but this is a huge benefit.

Thanks,

Claudio