Re: Emacs bzr memory footprint

[Nix]

On 21 Oct 2011, John Wiegley verbalised:

>>>>>> Nix  <address@hidden> writes:
>
>> Gnus is clearly driving Emacs much harder than mere cc-mode ever does.  I'm
>> particularly stunned by the number of string-chars in use. It must have some
>> huge variables defined, probably holding overview data or something... I'm
>> not sure how large some of those things are: if they're large structures
>> this might explain most or all of the memory consumption.
>
> Do you using the Gnus Registry?

I hadn't noticed it existed, so no (though it looks rather nice and I
shall probably be using it in due course, pushing my memory usage up
even further).

>                                  Which backends do you use?

nnml, nntp, a couple of tiny nndoc, and one, one single huge nnmh group
(containing filtered spam less than six months old).

>                                                              How many messages
> are in mailboxes that you open?

Some of them are quite large: my primary nnml mailbox has 16000 mails in
it, with a 6Mb overview (though only 60-odd are visible). Some of my
nntp groups never expire so are very, very large (100000-odd articles),
but, again, only a few dozen to a few hundred articles will be unread
and visible at any time. (The largest overview for a single group is
46Mb, but if Gnus is reading the entire overview database for an nntp
group in, it's doing something wrong!)

However, this cannot explain the memory consumption, because I check
most of these groups out within a few minutes of starting Emacs, and
memory consumption then is around 300Mb. The rise from then on is
inexorable, though not steady: where the figures for ten hours ago were

STIME   RSS    VSZ
Oct07 832348 1127088
Oct07 226916 499588

now they are

STIME   RSS    VSZ
Oct07 876524 1170572
Oct07 227016 499588

So the coding Emacs has hardly budged, but the newsreading one has
chewed up another 50Mb. It's a good thing this machine has 24Gb RAM :)

pmap shows thwe following sizeable anonymous regions:

0000000001f18000 850596K rw---    [ anon ]
00007fbf8ff29000      4K rw---    [ anon ]
00007fbf98000000    132K rw---    [ anon ]
00007fbf98021000  65404K -----    [ anon ]
00007fbf9f9b4000      4K -----    [ anon ]
00007fbf9f9b5000  32768K rw---    [ anon ]
00007fbfa1f6d000    772K rw---    [ anon ]
00007fbfa22b6000   1260K rw---    [ anon ]
00007fbfa266a000    236K rw---    [ anon ]
00007fbfa2ca1000     76K rw---    [ anon ]
00007fbfa2dc2000    772K rw---    [ anon ]
00007fbfa2e83000     28K rw---    [ anon ]
00007fbfa2ec9000      4K rw---    [ anon ]
00007fbfa2f0e000   1544K rw---    [ anon ]
00007fbfa309b000    168K rw---    [ anon ]

Heap fragmentation might explain this, but most of the big allocations
(e.g. for huge overviews) should be going into separately mmap()ed
regions and getting freed, not into that 850Mb pig of a heap. With luck
it's just one, but luck does not accompany me on trips like this. (I too
spent some time fruitlessly instrumenting XEmacs for signs of the cause
of its huge memory usage and came to the same conclusion as Stephen:
it's not XEmacs, it's the toolkits. The same may be true here, though at
least I'm using Lucid, not Gtk, so we can rule out *that* mountain of
code.)

Also note that XEmacs's huge memory usage was accompanied by a radical
slowdown in GC times that eventually forced a restart if I was to get
anything done. By contrast, this ballooning is not accompanied by any
slowdown in GC: a GC still takes only about 1/5s, barely slower than
when Emacs is freshly started.

Hm.


On second thought, IIRC Gnus allocates some very large lists as part of
its overview management or something: perhaps this is serving to spam
the arena with a huge number of (individually small, thus not
mmap()-allocated) atoms which, when they get freed later, produce a very
sparsely-filled, severely-fragmented heap? If so, perhaps Emacs would
benefit from a simple pool allocator accessed via a new let/setq form or
a new arg to create-buffer, so Gnus could arrange to stuff variables it
knows will be huge, or buffer-local variables of buffers it thinks may
have lots of huge buffer-local vars, into a newly-mmap()ed region?
Unfortunately that means, sigh, using our own malloc() again, which is
probably more painful than useful. I suspect actually proving my
contention first would be a good idea. Not sure how to get the addresses
of Lisp objects from a running Emacs though: gdb, presumably.

-- 
NULL && (void)