bug#41077: [feature/native-comp] virtual memory exhausted

[Kévin Le Gouguec]

Hi Andrea!

Thank you for implementing this blacklist; it turns out that
char-fold.el was the only file my laptop could not handle[1], though as
you'll see org.el was a strong contender.

It took 3 days, but make -j1 successfully ran to completion on commit
92cf4bb.  In comparison, I've just compiled commit 9d8fc3a on master
from scratch, and it took exactly 1 hour 35 minutes.

As soon as I've figured out how to use the elisp-benchmarks package,
I'll post some figures; is there a specific place (bug number,
emacs-devel thread) where you usually collect such feedback?

I've attached some more graphs on the compilation process, as well as
the source material[2] (measurement script, measurements, and plotting
script).

The overall picture (only files which took more than 10 minutes to
compile are labeled):

monitor.pdf
Description: Adobe PDF document

Some "areas of detail" (only files which took more than 5 minutes to
compile are labeled):

monitor-swap.pdf
Description: Adobe PDF document

monitor-gnus.pdf
Description: Adobe PDF document

monitor-org.pdf
Description: Adobe PDF document

Some comments:

- As could be predicted from your previous measurements, org.el was a
  beast, but unlike with char-fold.el the little guy pulled through 🙌

- It's hard to be sure since my measurements were so imprecise[3], but
  AFAICT the compilation process for a single file seems to follow a
  memory usage pattern of "slow rise - spike - drop - spike".  See
  e.g. files.el, isearch.el, simple.el, subr.el, window.el, info.el,
  package.el, erc.el, gnus-sum.el, org.el, python.el.


Thank you for taking the time to guide me through compiling this branch.
I know that reducing the memory footprint of native compilation is
probably not your main focus right now, but I figured it would be
interesting to provide some orders of magnitude.


[1] $ git diff
    diff --git a/lisp/emacs-lisp/comp.el b/lisp/emacs-lisp/comp.el
    index 60b41f95bd..ff3c42a178 100644
    --- a/lisp/emacs-lisp/comp.el
    +++ b/lisp/emacs-lisp/comp.el
    @@ -85,7 +85,7 @@ comp-always-compile
       :group 'comp)

     (defcustom comp-bootstrap-black-list
    -  '("^leim/")
    +  '("^leim/" "^char-fold")
       "List of regexps to exclude files from native compilation during 
bootstrap.
     Skip if any is matching."
       :type 'list

[2] Measurement script:

monitor.sh
Description: application/shellscript

    Measurements:

monitor.log.tgz
Description: application/compressed-tar

    Plotting script (requires matplotlib):

plot.py
Description: Text Data

[3] I didn't instrument the build process; as can be seen in the
    scripts, I just relied on:

        ps --sort=-vsz p $pids | head 1

    $pids include the emacs session in which I was running make; this
    session was masking basically everything other than the ELC+ELN
    processes.

    Also, the sampling step being 30 seconds, a lot of interesting
    patterns may not have been recorded.

    Overall these scripts are an exercise in How Not To Collect Data 😒.