bug#41077: [feature/native-comp] virtual memory exhausted

[Andrea Corallo]

Andrea Corallo <akrl@sdf.org> writes:

> Kévin Le Gouguec <kevin.legouguec@gmail.com> writes:
>
>> Andrea Corallo <akrl@sdf.org> writes:
>>
>>> All versions should work but at this point I'd go for releases/gcc-10 or
>>> releases/gcc-9.3.0
>>
>> Thanks, I went with 9.3.0.  Now the "ELC+ELN" steps start smoothly,
>> unfortunately I don't think the little buster can finish the job :(
>>
>> I first ran a -j2 build which crashed after 14 hours while compiling
>> char-fold.elc.  I didn't get anything more precise than "virtual memory
>> exhausted: Cannot allocate memory" on the console.
>>
>> Thinking it might help to compile only one file at a time, after
>> removing the temporary files related to char-fold.el[1], I started a -j1
>> build which picked up where the previous build left off, i.e. with
>> char-fold.elc.
>>
>> Unfortunately that file alone seems to be too much for my system to
>> handle; compilation ended with the same "virtual memory exhausted" error
>> after less than an hour and a half.
>>
>> I recorded some information related to memory usage during this second
>> run (cf. attached graph[2]).  My takeway is that at some point, the
>> compilation process's memory usage skyrockets, until the system's memory
>> (2GB RAM + 2GB swap) is completely exhausted.
>>
>>
>> I haven't reopened/created a new issue because I'm not sure there's a
>> way forward; let me know if you'd like me to perform some more in-depth
>> profiling.
>>
>>
>> [1] lisp/char-fold.elc… and lisp/eln-i686-pc-linux-gnu-…/char-fold….eln.
>>
>> [2] The graph shows:
>>     - the VSZ of the process using the most virtual memory,
>>     - the RSS of the process using the most virtual memory,
>>     - the "available" column shown by free(1) on the "Mem:" line,
>>     - the "used" column shown by free(1) on the "Swap:" line,
>>     - the name of the process using the most virtual memory.
>>
>>     Measurements were taken every minute.  I can upload the sources for
>>     this graph (measurement script, measurements, and plotting script)
>>     if that helps.
>
> Hi Kévin,
>
> thanks for the great report.
>
> I suspect memory usage on certain few compilation units goes high
> correlated with compile time.
>
> At the time I've got some measures and these were my results on this.
>
>     48:32.21 leim/quail/ZIRANMA.el
>     19:52.61 leim/quail/ECDICT.el
>     19:47.17 leim/quail/ARRAY30.el
>     19:16.88 leim/quail/tsang-cns.el
>     19:07.88 leim/quail/tsang-b5.el
>     18:41.66 org/org.el
>     15:25.17 leim/quail/4Corner.el
>     14:34.79 leim/quail/PY-b5.el
>     14:18.76 char-fold.el
>     13:58.44 leim/quail/ZOZY.el
>     13:58.04 leim/quail/ETZY.el
>     13:10.27 gnus/gnus-sum.el
>     11:25.28 leim/quail/quick-cns.el
>     11:13.41 leim/quail/quick-b5.el
>     8:50.48 leim/quail/TONEPY.el
>
> I believe both measures suggests we should should black list manually
> some of these compilation units, at least the one in "leim/".  I'll push
> a patch for that.
>
> I'm not sure was mentioned in this thread but you can work around this
> now using by speed 0 or fast boot (or a combination of these).
>
> make BYTE_COMPILE_EXTRA_FLAGS='--eval "(setq comp-speed 0)"'
>
> Thanks!
>
>   Andrea
>
> --
> akrl@sdf.org

I must have replied privately to this so I'm quoting my previous mail.

That said I've just pushed a commit that exclude from native compilation
all the leim db files.

Overall compile time looks went down by a factor two!

Hope this helps.  If you could get a measure of the memory consumption as
well that would be great.

Thanks

  Andrea

-- 
akrl@sdf.org