[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Compiler memory consumption

From: Andy Wingo
Subject: Re: Compiler memory consumption
Date: Tue, 16 May 2017 22:14:09 +0200
User-agent: Gnus/5.13 (Gnus v5.13) Emacs/25.1 (gnu/linux)


Definitely sounds like we have an issue we should deal with.

On Tue 16 May 2017 18:19, address@hidden (Ludovic Courtès) writes:

> here is the gcprof output

I find gcprof useful when I want to improve runtime by removing
allocation.  However I don't find it useful when dealing with memory use
issues; ymmv of course.

> while compiling gnu/packages/python.scm, which defines 841 package
> objects (structs) with 5 times more thunks of the form (lambda ()
> value):

You mention later:

> Also, for reference, loading python.go peaks at 315M RSS:
> $ \time ./pre-inst-env guile -c '(use-modules (gnu packages python))'
> 0.18user 0.02system 0:00.18elapsed 112%CPU (0avgtext+0avgdata 
> 315648maxresident)k
> 0inputs+0outputs (0major+7784minor)pagefaults 0swaps

But this I don't understand.  If I do a ./pre-inst-env guile and then
load (gnu packages python), I get a 20MB heap size, a 35MB total private
dirty memory and 52MB clean shared memory.  (Measured using smaps via  This is at commit

I just built the "time" package and I see similar numbers here.  I can
only think that the "time" package's numbers are bogus.

> time(1) reports a maximum resident set size of 3.8G (though I see
> something around 900MiB in ‘top’.)

The only way Guile's memory usage can shrink in practice is if it
recurses a lot on the stack and then returns those pages to the OS.  I
don't think libgc will return pages to the OS (though I could be
wrong).  So that would be a possibility to look into, if time can be

> When compiling python.scm #:to 'cps, we end up with 1G max RSS in 6s.

Measured with time?  If this is the case it could be that python.scm is
just a lot of code.  Any compiler would take a lot if the IR size is 1

> The only conclusion I can draw is that cps-to-bytecode compilation seems
> to be responsible for most of the memory consumption.

This is possible but I am not there yet.  I don't see why compiling this
file to CPS should cause memory usage of 1GB.  That is 9000 memory words
per textual line -- simply too much.

Many unknowns here!


reply via email to

[Prev in Thread] Current Thread [Next in Thread]