bug#27284: Memory leak in 'guix pull' or 'make' in guix source

[Ludovic Courtès]

Hello Guix!

A heads-up to share the cognitive burden related to this topic.  :-)

So, we have two problems: compilation time, and memory consumption.  I
*think* I’ve identified one of the major causes for both in Guile,
though it’s too early to say exactly how much this will impact resource
consumption for a full Guix compilation.  See
<https://lists.gnu.org/archive/html/guile-devel/2017-09/msg00031.html>
for details.

When that is fixed, we’ll still have a performance problem: building all
of Guix will still take more time than we’d like, and it won’t get
better as we add new files.  So we need to address this.

This has been discussed informally many times, and here’s a summary of
the ideas I’m aware of:

  1. Build Guix as separate derivations: the first derivation builds the
     closure of (guix packages), the second one builds the closure of
     (guix scripts *), and finally we build (gnu *).  We could also have
     derivations for the non-Scheme parts: the daemon, the manual, etc.

     (That amounts to compartmentalizing Guix in sub-packages, which in
     a way solves the bootstrapping issue that Pjotr complained about at
     some point—the fact that to build Guix one needs Guile, Guile-JSON,
     GnuTLS, Autotools, etc.)

     The advantage is that when running ‘guix pull’ frequently, you
     won’t have to recompile all of these.  However, you’ll almost
     always have to rebuild (gnu packages *), which is the longest part.

  2. Build all of Guix like the ‘guix’ package does, and hope that we
     can get a substitute.

     Bootstrapping issue: to do that, we first need compute the
     derivation of this new ‘guix’ package.  Thus, we at least need to
     build the closure of (guix packages), which should take a minute or
     so, after which we can compute the derivation, which could take a
     couple of minutes maybe.

     The problem is that building all of Guix (including running the
     test suite) takes some time, potentially more than the interval
     between two subsequent pushes to the repo.  Thus, it’s quite likely
     that the build farm would always be lagging behind.

     We could work around that by having the build farm automatically
     tag commits for which it has successfully built Guix.  That would
     introduce delays in deploying the latest Guix to users, but maybe
     that can be short enough to be acceptable.

  2b. To address the bootstrapping issue above, we also discussed the
     possibility of setting up a “meta-derivation” service: you’d give
     it a Git commit, and it’d return the derivation of ‘guix’ for that
     commit.

     Less computation would take place on the user side, but that
     doesn’t reduce the delay mentioned above.

     It also has the downside of introducing another service without
     which using Guix is more painful.

I think that’s about it.

Thoughts?  Hacks?  :-)

Ludo’.