Re: Avoiding rebuilds (e.g. of mariadb, the entire Rust chain, etc.)?

[Leo Famulari]

On Sun, Dec 26, 2021 at 11:41:30PM -0600, Matthew Brooks wrote:
> Is there any way to avoid rebuilding stuff like mariadb, the entire Rust 
> chain, etc. unless one of those packages *actually* changes? It seems like 
> every few days every single package needs to rebuild for some reason, 
> including many packages that spend unbelievably long times running tests that 
> will never actually be of use to me, so I'm usually only able to update every 
> couple of weeks since so much constantly needs to be rebuilt and everything 
> takes so long.

In Guix, packages are considered to have changed when any part of their
dependency graph changes. So, no, there is not a way to avoid rebuilding
a package when one of its transitive dependencies has changed.

We have guidelines about how to make changes to the distro that are
based on how many rebuilds they cause. See item 8:

https://guix.gnu.org/manual/en/html_node/Submitting-Patches.html#Submitting-Patches

Both MariaDB and Rust have several thousand packages that depend on
them, so we change these packages seldomly, and use the build farm to
rebuild everything in advance so that users don't have to.

If users choose to avoid substitutes, that's fine too, but there's no
way to avoid doing the work.

Recently, we deployed the results of a "core-updates" project, which
updated core packages, and caused every single package to be rebuilt.
So, that's why you are having to rebuild more than usual.

> It seems to have gotten worse over time as well, as more and more base 
> packages pull in extremely computation-hungry dependency chains. At this very 
> moment, for example, I'm waiting for Rust & co. to compile simply to delete 
> old system generations from the bootloader.

Yes, Rust continues to grow in importance and has become a relatively
low-level dependency of the distro on the x86_64 platform. We are
continually finding ways to shorten the "bootstrap chain" of Rust, but
it's still fairly long.

> I've got substitutes turned off (primarily because I like the idea of every 
> package being "home built" as it were), but I can't imagine that even the 
> official substitiute servers would be able to keep up with the constant 
> rebuilds my system seems to want unless I've got something horribly 
> misconfigured somewhere.

The build farm has ~2800 powerful CPU cores for the x86_64 architecture,
so it can keep up. Other architectures are less well-resourced. And of
course very few users have access to those kinds of resources.

>    (extra-options '("--gc-keep-derivations=yes" "--gc-keep-outputs=yes" 
> "--no-substitutes"))

I would have suggested adding those first two options. Beyond that,
maybe others can suggest something that will help.