bug#46330: Guile-provided GMP allocators interfere with GnuTLS

[Ludovic Courtès]

Ludovic Courtès <ludo@gnu.org> skribis:

> In a nutshell, Guile installs its own GMP memory allocation routines
> (when ‘scm_install_gmp_memory_functions’ is true, which is the default)
> so that GMP allocates via libgc.  GnuTLS uses Nettle, which uses GMP, so
> Nettle too ends up allocating via libgc; however, since pointers to that
> memory are not scanned by libgc, they end up being reclaimed early.

One of the solutions is to set:

  scm_install_gmp_memory_functions = 0;

in Guile, as Andy suggested on IRC, but it incurs a performance hit on
bignum-heavy applications such as the compiler:

  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=964284#78

However, since Guix now uses its own ‘guile’ binary, we can work around
the issue like so:

diff --git a/gnu/packages/aux-files/guile-launcher.c 
b/gnu/packages/aux-files/guile-launcher.c
index 1dd5d77e66..814084e032 100644
--- a/gnu/packages/aux-files/guile-launcher.c
+++ b/gnu/packages/aux-files/guile-launcher.c
@@ -1,5 +1,5 @@
 /* GNU Guix --- Functional package management for GNU
-   Copyright 1996-1997,2000-2001,2006,2008,2011,2013,2018,2020
+   Copyright 1996-1997,2000-2001,2006,2008,2011,2013,2018,2020,2021
       Free Software Foundation, Inc.
    Copyright (C) 2020 Ludovic Courtès <ludo@gnu.org>
 
@@ -82,7 +82,10 @@ main (int argc, char **argv)
   unsetenv ("GUILE_LOAD_PATH");
   unsetenv ("GUILE_LOAD_COMPILED_PATH");
 
-  scm_install_gmp_memory_functions = 1;
+  /* XXX: Do not let GMP allocate via libgc as this can lead to memory
+     corruption in GnuTLS/Nettle: <https://issues.guix.gnu.org/46330>.  */
+  scm_install_gmp_memory_functions = 0;
+
   scm_boot_guile (argc, argv, inner_main, 0);
   return 0; /* never reached */
 }

The advantage of this hack is that we still get to use upstream ‘guile’
for compilation purposes (with no performance hit), and we use our own
“safe” ‘guile’ executable for stuff that may use GnuTLS, in particular
‘guix substitute’ and ‘guix perform-download’.

There may still be a few cases where we’d use stock ‘guile’ together
with GnuTLS.  The only example that comes to mind is when calling
‘download-nar’ or ‘swh-download’ as a fallback in (guix git-download).
That’s quite rare though.

So I think that the above is a workaround we could deploy right away.
It should allow us to wait until we have Guile on mini-GMP.

Thoughts?

Ludo’.