Re: [Qemu-devel] Memory use with >100 virtio devices

[Alexey Kardashevskiy]

On 21/08/17 15:50, Alexey Kardashevskiy wrote:
> On 21/08/17 14:31, David Gibson wrote:
>> On Fri, Aug 18, 2017 at 02:18:53PM +0100, Stefan Hajnoczi wrote:
>>> On Fri, Aug 18, 2017 at 03:39:20PM +1000, Alexey Kardashevskiy wrote:
>>>> ==94451== -------------------- 4 of 10 --------------------
>>>> ==94451== max-live:    314,649,600 in 150 blocks
>>>> ==94451== tot-alloc:   314,649,600 in 150 blocks (avg size 2097664.00)
>>>> ==94451== deaths:      none (none of these blocks were freed)
>>>> ==94451== acc-ratios:  0.00 rd, 0.00 wr  (0 b-read, 0 b-written)
>>>> ==94451==    at 0x4895600: memalign (in
>>>> /usr/lib/valgrind/vgpreload_exp-dhat-ppc64le-linux.so)
>>>> ==94451==    by 0x48957E7: posix_memalign (in
>>>> /usr/lib/valgrind/vgpreload_exp-dhat-ppc64le-linux.so)
>>>> ==94451==    by 0xB744AB: qemu_try_memalign (oslib-posix.c:106)
>>>> ==94451==    by 0xA92053: qemu_try_blockalign (io.c:2493)
>>>> ==94451==    by 0xA34DDF: qcow2_do_open (qcow2.c:1365)
>>>> ==94451==    by 0xA35627: qcow2_open (qcow2.c:1526)
>>>> ==94451==    by 0x9FB94F: bdrv_open_driver (block.c:1109)
>>>> ==94451==    by 0x9FC413: bdrv_open_common (block.c:1365)
>>>> ==94451==    by 0x9FF823: bdrv_open_inherit (block.c:2542)
>>>> ==94451==    by 0x9FFC17: bdrv_open (block.c:2626)
>>>> ==94451==    by 0xA71027: blk_new_open (block-backend.c:267)
>>>> ==94451==    by 0x6D3E6B: blockdev_init (blockdev.c:588)
>>>
>>> This allocation is unnecessary.  Most qcow2 files are not encrypted so
>>> s->cluster_data does not need to be allocated upfront.
>>>
>>> I'll send a patch.
>>
>> Is that sufficient to explain the problem, I can't quickly see how big
>> that unnecessary allocation is - but would it account for the 10s of
>> gigabytes usage we're seeing here?
>>
>> I'm suspecting we accidentally have a O(n^2) or worse space complexity
>> going on here.
>>
> 
> No, it is a small fraction only. See "[PATCH] qcow2: allocate
> cluster_cache/cluster_data on demand" thread for more details.


The information was lost there so I'll continue in this thread.

I run QEMU again, with 2GB of RAM, -initrd+-kernel, pseries, 64 PCI
bridges, -S, no KVM, some virtio-block devices; I run it under "valgrind
--tool=exp-dhat" and exited via "c-a x" as soon as possible.

The summary of each run is:

50 virtio-block devices:
guest_insns:  2,728,740,444
max_live:     1,214,121,770 in 226,958 blocks
tot_alloc:    1,384,726,690 in 310,930 blocks


150 virtio-block devices:
guest_insns:  17,576,279,582
max_live:     7,454,182,031 in 1,286,128 blocks
tot_alloc:    7,958,747,994 in 1,469,719 blocks

250 virtio-block devices:
guest_insns:  46,100,928,249
max_live:     19,423,868,479 in 3,264,833 blocks
tot_alloc:    20,262,409,839 in 3,548,220 blocks

350 virtio-block devices:
guest_insns:  88,046,403,555
max_live:     36,994,652,991 in 6,140,203 blocks
tot_alloc:    38,167,153,779 in 6,523,206 blocks


Memory usage 1) grows a lot 2) grows out of proportion 3) QEMU becomes
incredibly slow.


With the hack (below) and 350 virtio-block devices, the summary is:
guest_insns:  7,873,805,573
max_live:     2,577,738,019 in 2,567,682 blocks
tot_alloc:    3,750,238,807 in 2,950,685 blocks
insns per allocated byte: 2


I am also attaching 2 snapshots from the valgrind's "massif" tool, with and
without the hack.

Ideas what to tweak or what valgrind tool to try?



The hack is basically excluding virtio-pci-cfg-as from the address_spaces
list (yeah, it breaks QEMU, this is just a hint):

diff --git a/memory.c b/memory.c
index 02c95d1..118ac7f 100644
--- a/memory.c
+++ b/memory.c
@@ -2589,6 +2589,7 @@ void address_space_init(AddressSpace *as,
MemoryRegion *root, const char *name)
     as->ioeventfd_nb = 0;
     as->ioeventfds = NULL;
     QTAILQ_INIT(&as->listeners);
+    if (strcmp(name, "virtio-pci-cfg-as"))
     QTAILQ_INSERT_TAIL(&address_spaces, as, address_spaces_link);
     as->name = g_strdup(name ? name : "anonymous");
     address_space_init_dispatch(as);




-- 
Alexey

massiv.bad
Description: Text document

massiv.good
Description: Text document

signature.asc
Description: OpenPGP digital signature