Re: Faster "guix pull" by incremental compilation and non-circular modules?

[Ludovic Courtès]

Hi,

Maxime Devos <maximedevos@telenet.be> skribis:

>   2. Instead of building all of Guix as a single derivation,
>      create a DAG of derivations.  More concretely:
>
>      First read the *.scm files to determine which module imports
>      which modules. Then to compile, say, (gnu packages acl),
>      a derivation taking gnu/packages/acl.scm and its dependencies
>      gnu/packages/attr.go, gnu/packages/base.go, ... is made
>      compiling gnu/packages/acl.scm to a gnu/packages/acl.go.
>
>      Then to build all of Guix, 'union-build' or 'file-union' is used.

This is what (guix self), used by ‘guix pull’, is already doing.

However, currently, package modules are split in just two groups: the
“base” group is the closure of (guix packages base), and the second
group has all the rest:

--8<---------------cut here---------------start------------->8---
scheme@(guile-user)> ,use(guix modules)
scheme@(guile-user)> (length (source-module-closure '((gnu packages base))))
$4 = 417
scheme@(guile-user)> ,use(ice-9 ftw)
scheme@(guile-user)> (length (scandir "gnu/packages" (lambda (f) 
(string-suffix? ".scm" f))))
$5 = 537
scheme@(guile-user)> (- $5 $4)
$6 = 120
--8<---------------cut here---------------end--------------->8---

This is clearly uneven, especially considering that the biggest file,
crates-io.scm, is in the first group.

At its core though, the situation pretty much reflects the free software
situation: there are low-level packages (glibc, GCC, GTK, etc.) that
might depend on high-level packages (Python, Pandoc, Rust, etc.).

It’s not easy to split this spaghetti ball in smaller groups.

Thoughts?

Ludo’.