[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Qemu-devel] Qemu and heavily increased RSS usage
From: |
Peter Lieven |
Subject: |
Re: [Qemu-devel] Qemu and heavily increased RSS usage |
Date: |
Tue, 21 Jun 2016 17:12:57 +0200 |
User-agent: |
Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.8.0 |
Am 21.06.2016 um 15:18 schrieb Dr. David Alan Gilbert:
* Peter Lieven (address@hidden) wrote:
Hi,
while upgrading from Qemu 2.2.0 to Qemu 2.5.1.1 I noticed that the RSS memory
usage has heavily increased.
We use hugepages so the RSS memory does not include VM memory. In Qemu 2.2.0 it
used to be ~30MB per vServer
and increased to up to 300 - 400MB for Qemu 2.5.1.1 (same with master). The
memory increases over time, but seems
not to grow indefinetly. I tried to bisect, but had no result so far that made
sense. I also tried valgrind / massif, but
valgrind does not see the allocation (at least at exit) and massif fails to
rund due to - so it pretends - heap corruption.
Any help or ideas how to debug further would be appreciated.
I think I'd try stripping devices off; can you get a similar difference
to happen with a guest with no USB, no hugepages, no VGA and a simple
locally stored IDE disk?
Will do. VGA I already ruled out. Hugepages I can try, but its easier
to monitor the RSS size if the vServer memory is outside the RSS.
If you're having trouble bisecting is it possible it's a change
in one of the libraries it's linked against?
Same libraries. If I compile qemu-2.2.0 and qemu/master on the exactly
same machine I see the difference.
There was someone asking the other day on #qemu who had a setup that
was apparently using much more RAM than expected and we didn't
manage to track it down but I can't remember the version being used.
What I currently suspect are the following changes:
- We changed the coroutine pool to a per thread model. I disabled the pool.
This seems
to cut the max used RSS to about 150MB which is still a lot more than
qemu-2.2.0
- I suspect that something (e.g. the object based device tree) is creating a
lot of small allocations
which create a massive overhead. I managed to get valgrind/massif running
with attached debugger
and took snapshots of the running VM. What I see is that kernel RSS size is
much, much bigger than
what massif sees as allocated memory. I talk of massif sees 5MB heap usage
and the RSS size is 50MB or similar.
- Changing the memory allocator to tcmalloc or jemalloc seems to relax the
issue altough its not gone.
- VGA memory seems to have been moved from VM memory into Heap. But thats a
fixed 16MB allocation.
I will try to cut down devices as you proposed.
Peter
- [Qemu-devel] Qemu and heavily increased RSS usage, Peter Lieven, 2016/06/21
- Re: [Qemu-devel] Qemu and heavily increased RSS usage, Dr. David Alan Gilbert, 2016/06/21
- Re: [Qemu-devel] Qemu and heavily increased RSS usage,
Peter Lieven <=
- Re: [Qemu-devel] Qemu and heavily increased RSS usage, Stefan Hajnoczi, 2016/06/22
- Re: [Qemu-devel] Qemu and heavily increased RSS usage, Peter Lieven, 2016/06/22
- Re: [Qemu-devel] Qemu and heavily increased RSS usage, Peter Maydell, 2016/06/22
- Re: [Qemu-devel] Qemu and heavily increased RSS usage, Stefan Hajnoczi, 2016/06/24
- Re: [Qemu-devel] Qemu and heavily increased RSS usage, Peter Lieven, 2016/06/24
- Re: [Qemu-devel] Qemu and heavily increased RSS usage, Dr. David Alan Gilbert, 2016/06/24
- Re: [Qemu-devel] Qemu and heavily increased RSS usage, Peter Maydell, 2016/06/24
- Re: [Qemu-devel] Qemu and heavily increased RSS usage, Peter Lieven, 2016/06/24
- Re: [Qemu-devel] Qemu and heavily increased RSS usage, Stefan Hajnoczi, 2016/06/27
- Re: [Qemu-devel] Qemu and heavily increased RSS usage, Peter Lieven, 2016/06/27