[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Speeding up archive export
From: |
Ludovic Courtès |
Subject: |
Speeding up archive export |
Date: |
Thu, 10 Sep 2020 11:04:26 +0200 |
User-agent: |
Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) |
Hello Guix!
If you’ve ever used offloading (or ‘guix copy’), you’ve probably noticed
that the time to send store items is proportional to the number of store
items to send rather than their total size. Namely:
guix archive --export coreutils
is fast, but:
guix archive --export $(guix build -d coreutils)
is slow (there are lots of small files).
Running ‘perf timechart record guix archive --export …’ confirms the
problem: guix-daemon is mostly idle, waiting for all the tiny ‘guix
authenticate’ programs it spawns to sign each every store item. Here’s
the Gantt diagram (grey = idle, blue = busy):
How can we improve on that?
Here are several solutions that come to mind:
1. Sign the whole bundle instead of each individual item.
That solves the problem, but that would prevent the receiver from
storing individual store item signatures in the future (a few years
ago Nix added signatures as part of the ‘ValidPathInfo’ table of
the store database, and I think that’s something we might want to
have too).
2. Sign fewer items: we can do that by signing only store items that
are not content-addressed—i.e., resulting from a fixed-output
derivation or being a “source” coming from ‘add-to-store’ or
similar.
That means we wouldn’t have to sign .drv and *-guile-builder, which
would make a big difference and is generally advisable.
Unfortunately, there’s no easy way to determine whether a store
item is content-addressable. Again Nix added
“certificate-addressability claims” to ‘ValidPathInfo’, which might
help, though it’s not entirely clear.
3. Reimplement ‘guix authenticate’ and a subset of (guix pki) in C++ (!).
We could load the keys and the ACL only once, and we wouldn’t have
to fork and all, I’m sure it’d be very fast… and very distracting
too: I’d rather investigate in the daemon rewrite in Scheme.
4. Spawn ‘guix authenticate’ once and talk to it over a pipe (similar
to ‘guix offload’). That might be the easiest short-term solution.
Anyway I thought I’d share and invite y’all to brainstorm. :-)
All in all, there’s more and more pressure to get our act together
regarding the daemon rewrite in Scheme. The difficulty here is to have
a series of reasonable milestones rather than all or nothing.
Ludo’.
- Speeding up archive export,
Ludovic Courtès <=