Re: Substitute retention

[zimoun]

Hey,

On Tue, 12 Oct 2021 at 18:04, Ludovic Courtès <ludovic.courtes@inria.fr> wrote:

> (Moving to guix-devel from <https://issues.guix.gnu.org/42162#43>.)

I was preparing a report.  You have been faster than me. :-)

Two questions are raising, IIUC:

 1. the “modular” derivations for all the commits
 2. long-term support for substitutes


>> Other said 3388/6996= ~50% of commits are pushed at the same time, i.e.,
>> missed by both build farms using 2 different strategies to collect the
>> thing to build (fetch every 5 minutes or fetch from guix-commits).  It
>> is a quick back to envelope so keep that with some salt. :-)
>
> OK.

To make it explicit of #1, I was talking about the “modular” Guix, i.e.,
when running “guix pull” or “guix time-machine” it leads to build the
derivations module-import.drv, guix-<hash>.drv, guix-command.drv,
guix-module-union.drv, guix-<hash>-modules.drv,
guix-packages-modules.drv, guix-system-tests-modules.drv,
guix-packages-base-modules.drv, etc.  On slow machines, it can be
unpleasant; not to say unpractical.  Even for recent commits.

The recent addition of ’channel-with-substitutes-available’ helps when
going forward (guix pull) if the build farm does not have yet these.

The issue is going backward (guix time-machine).

Basically, commit 59d10c3112 is from March 14, 2020 and it takes ~29min
on my slow laptop.  And to compare apple to apple, let take another
commit one year later from March 14, 2021, e.g., commit 7327295462.  It
takes ~5min on the same machine.

--8<---------------cut here---------------start------------->8---
$ time guix time-machine --commit=59d10c3112 -- --version
Updating channel 'guix' from Git repository at 
'https://git.savannah.gnu.org/git/guix.git'...
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
The following derivations will be built:
   /gnu/store/zvy89f9xb53fbqvfrm7lql8mbfrsfk1b-compute-guix-derivation.drv
   /gnu/store/7y80kn1bypnbm869hvcq8841mr6nqvfm-module-import-compiled.drv
   /gnu/store/amwvgaf45722k6jn4r39983zsgmbyp2g-module-import.drv
   /gnu/store/h3h0qfiw5100zkwfb919r7vn0q06ksqy-config.scm.drv
   /gnu/store/jkwhdilsbxb18hx6gi4i2rj0v06mfbab-module-import.drv
   /gnu/store/sixfy4sazai667n99pxa5h7wzzaabw79-module-import-compiled.drv

[...]

Computing Guix derivation for 'x86_64-linux'...  WARNING: (guix build 
emacs-build-system): imported module (guix build utils) overrides core binding 
`delete'
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%

[...]

substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
The following derivations will be built:
   /gnu/store/50ymxym19h8whzg3ajcl6kdjmq3p7qrg-profile.drv
   /gnu/store/l0znp7g83lbylbv97nd3ahz8rnrvxfrf-guix-59d10c311.drv
   /gnu/store/bvxrnp8bydl3zsbcg7j8j7m0qfygdhfs-guix-command.drv
   /gnu/store/xmsciylxx1j7nbry5cv1lm7595a8rilr-guix-module-union.drv
   /gnu/store/yvkw65kv9bhvx41750dchxw98qqxv64b-guix-59d10c311-modules.drv
   /gnu/store/6hrn3bpvcg8571mckzcj519xf2kqn2sl-guix-packages-base-modules.drv
   /gnu/store/nwl8z8cx9pdwn0sx5i5j5mp0bkdi64mm-guix-packages-base.drv
   /gnu/store/0j0271nbm2l526m5xs7zpd686qqrjz7w-guix-core.drv
   /gnu/store/2134jzhhckh871vlscw6dmwqqhny9zxg-guix-core-source.drv
   /gnu/store/mhik7ggrf4z5f38nsg7g0gbijm916b98-config.scm.drv
   /gnu/store/53zlyv96nrrm4h5ns7nmmndj8jys38f7-guix-extra.drv
   /gnu/store/7dn3f27i48jp6zvlwanzk8mfy828k6cm-guix-config-modules.drv
   /gnu/store/6xy3yyvjqvjyrlrwk1lzs12knbvilqpy-guix-config-source.drv
   /gnu/store/15ihwkqzyz4r4b4rppb92qcawha6a7p7-config.scm.drv
   /gnu/store/cacawv4yib8pa2ajzw0kyaihgym72mww-guix-config.drv
   /gnu/store/baivfv20hzm799v0wvdrcfaimh4aw22a-guix-extra-modules.drv
   /gnu/store/cxpkd0jkxapzbmg5vfmn4fy30yd7vlhm-guix-core-modules.drv
   /gnu/store/g3nqbybggh7dc2qd9gkj7swfmgmiigpp-guix-packages-modules.drv
   /gnu/store/nq9mzr00ny3nrsldvcq9r4va4fhb26sq-guix-packages.drv
   /gnu/store/i9dfjh5mf3r83447x7fa75hv1hnp9myv-guix-cli-modules.drv
   /gnu/store/xdsld8rhrawgngv93qx6lk9cgmql908c-guix-cli.drv
   /gnu/store/qnzm0a1gcwnfvkii7vk93rimzzp3mcf9-guix-system.drv
   /gnu/store/x2g2s3rhw0bv9qdp21yghgl2sij15dkr-guix-system-modules.drv
   /gnu/store/zandkjnylznvdj8jfsfgssvmvs6jfyph-guix-system-tests-modules.drv
   /gnu/store/45bs7y1h3gx2m7qry6r621klgkmv47wl-guix-system-tests.drv
   /gnu/store/caj7qjxvhrksk3jrkpsxqnx4kg7mlj9d-guix-daemon.drv
   /gnu/store/mah24wyy6bd51c27ww45hsqnjxhcn0yx-guix-manual.drv
   /gnu/store/002i672yl1192x7wvhkdbih94qffmcdk-guix-translated-texinfo.drv
   /gnu/store/5sf9aalvic81qlm06lq3a1pwdb2b3bm0-inferior-script.scm.drv
   /gnu/store/k612gnziiy3hn2dnrj33w4mw84kcnynm-profile.drv

11.7 MB will be downloaded

[...]

real    29m14.588s
user    0m56.126s
sys     0m1.032s
--8<---------------cut here---------------end--------------->8---

--8<---------------cut here---------------start------------->8---
$ time guix time-machine --commit=7327295462 -- --version
Updating channel 'guix' from Git repository at 
'https://git.savannah.gnu.org/git/guix.git'...
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%

[...]

building /gnu/store/6xq8vxpl51l8b3fz6sxpyspa1w5chbk9-module-import.drv...
 module-import  2KiB                                           43KiB/s 00:00 
[##################] 100.0%

 module-import-compiled  1.5MiB                               607KiB/s 00:03 
[##################] 100.0%

 module-import-compiled  1.5MiB                               606KiB/s 00:03 
[##################] 100.0%

building 
/gnu/store/pbv19dhrlqr2lnzphmydn4zrrdccghf2-compute-guix-derivation.drv...
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
@ substituter-started 
/gnu/store/fbn395nfpbp4d4fr6jsbmwcx6n10kg16-python-minimal-3.8.2 substitute
@ download-started 
/gnu/store/fbn395nfpbp4d4fr6jsbmwcx6n10kg16-python-minimal-3.8.2 
https://ci.guix.gnu.org/nar/lzip/fbn395nfpbp4d4fr6jsbmwcx6n10kg16-python-minimal-3
[...]
@ substituter-succeeded 
/gnu/store/fx0cdzzppd8jc09sianbq6gl1h7mxx3x-zziplib-0.13.72

substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
The following derivations will be built:
   /gnu/store/8kmb40b2r3cx5zxpcrwa73a4lkaxjd9l-profile.drv
   /gnu/store/la6c7m2b6izy22vv8xpyvpz1ajyq72br-profile.drv
   /gnu/store/nljv2wnw0wqkyk0am8722gdwah3b0cx2-guix-732729546.drv
   /gnu/store/bmrx03y52d8dhhcpyf9i8j4zn2fg7pip-guix-command.drv
   /gnu/store/pg26n125c9bmvk4lxxp9ssd9havk89wc-guix-module-union.drv
   /gnu/store/mp0c2ad09axrq8zwhh3ycfk4a2mrgvm2-guix-732729546-modules.drv
   /gnu/store/0gx5jr1vgkdm5ajfcscladhzjx2gz5l2-guix-system-modules.drv
   /gnu/store/1sqwx35rn2qinlzib74zfjanxzzgmza3-guix-packages-base-modules.drv
   /gnu/store/6d64jyviixsjvfjgpxv3lyzq7l35y0f3-guix-config-modules.drv
   /gnu/store/5v24gh1pbqy6jzyl9x52wzxa6qprwl6v-guix-config-source.drv
   /gnu/store/pv6rskbpg8rzq3wi92m3iw7a2524r994-config.scm.drv
   /gnu/store/i6ncmiw6agrlq290drkg94scn12sv4v8-guix-config.drv
   /gnu/store/6l3l0qsq280pwmrnb0z2dfiyc7g5ff5i-guix-extra-modules.drv
   /gnu/store/hdy7q3xm8jalssc6jim2fk1wqgxqbfm9-guix-system-tests-modules.drv
   /gnu/store/ki86v53288fnx8sih3zlfi9qnjx2lzay-guix-packages-modules.drv
   /gnu/store/l63wxbqaq98nfjkq2cyshb6q8rjxjd6h-guix-core-modules.drv
   /gnu/store/b1qky9s98cfgp88xan9pqg0k9k0rlzrm-guix-core-source.drv
   /gnu/store/wmfssg7yyz2hrwanash7yk8f86faghlf-guix-cli-modules.drv
   /gnu/store/z1xxvp4hmffrpmbl0ll5y87w5pyfma9l-guix-daemon.drv
   /gnu/store/ms6pkrkggd0rl4fh5hfh20gcva7ryip5-inferior-script.scm.drv

28.7 MB will be downloaded

[...]

real    5m56.451s
user    3m52.055s
sys     0m1.738s
--8<---------------cut here---------------end--------------->8---



To be on the same wavelength,

--8<---------------cut here---------------start------------->8---
$ git log --format="%h %cd" --after=2021-03-14 --reverse | head -n16
[...]
2babf7d831 Sun Mar 14 19:16:55 2021 +0100
b15720182e Sun Mar 14 13:24:21 2021 -0500
207aa62e6b Sun Mar 14 13:24:21 2021 -0500
30f5381487 Sun Mar 14 13:24:21 2021 -0500
af25357b7d Sun Mar 14 13:24:21 2021 -0500
7164d2105a Sun Mar 14 13:24:21 2021 -0500
078f3288e2 Sun Mar 14 13:24:21 2021 -0500
5a31eb7d35 Sun Mar 14 13:24:21 2021 -0500
620206b680 Sun Mar 14 13:24:22 2021 -0500
b76762a9b7 Sun Mar 14 13:24:22 2021 -0500
cbfcbb79df Sun Mar 14 19:43:35 2021 +0100
--8<---------------cut here---------------end--------------->8---

and Cuirass builds only one of b15720182e, 207aa62e6b, 30f5381487,
af25357b7d, 7164d2105a, 078f3288e2, 5a31eb7d35, 620206b680 or
b76762a9b7.

Considering the Build Coordinator, it uses guix-commits and from my
understanding it reads:

<https://lists.gnu.org/archive/html/guix-commits/2021-03/msg01201.html>

therefore, b15720182e would be missed but not b76762a9b7–which would be
missed by Cuirass.

Cuirass and the Build Coordinator cannot each build the both commits
b15720182e and b76762a9b7.

Cuirass check every 5 minutes and Build Coordinator reads “state” from
guix-commits.  Other said, none of them builds all these “modular”
derivations for all the commits; even for recent commits.

The rough estimate is half of commits are missed by both build farms.
Therefore, using “guix time-machine” with a random commit and one gets
1/2 probability to build something just to get the inferior – aside the
TTL policy.

(It is mitigated because the both build farms use different strategies
and thus they do not miss the same commits. \o/)

>> On that number, after 180 days (6 months), it is hard to evaluate the
>> rate of the time-machine queries.  And from my experience (no number to
>> back), running time-machine on a commit older than this 180 days implies
>> to build derivations.  Or it is a lucky day. :-)
>
> Right.
>
> So what can we do to address this issue?  I *think* we could use a
> higher TTL on berlin, and we can try that right away (9 months to being
> with?).

I *think* the issue is not TTL for question #1.  :-) But the issue that
the both build farms do not build these “modular” derivations for all
the commits.  Here, I am focused on x86_64-linux which is the case of
interest for such topic (scientific context), IMHO.

Considering to build for every commit for all architectures is not
affordable.

I agree that increasing the TTL will help for question #2 about
long-support of substitutes.

> However, there is an upper bound anyway.  To make informed decisions on
> the retention policy, we should monitor storage space on berlin/bayfront
> to better estimate what can be done.  We have Zabbix but it’s not
> accessible from the outside; maybe we could graph storage space
> somewhere so people can grab the data and work on those estimates?

Based on the size of these derivations for one commit, we could
extrapolate back to envelope.  Well, question #1 seems doable
storage-speaking.

The issue of #1 is to build these derivations for all the commits.
IMHO.

About #2, yeah if some data are available, I can try to make some
estimates.


Well, #1 seems actionable.  However, #2 raises…

> What if we decide that we need to provide substitutes for 2y old
> commits?  In that case, we need a plan to scale up.  That could be
> renting storage space somewhere.  That’s largely non-technical work that
> needs attention.

…a strong question. :-) What do “we” do for what “we” build?

Indeed, numbers are missing to make informed decisions on long-term
storage of substitutes.  What is Nix doing?


> There are also technical tweaks that could help: distinguishing between
> “important” substitutes that we want to keep, and less important
> substitutes (how?); identifying “equivalence classes” for builds of a
> given package; etc.  The outcome is unclear and it’ll take time.

I agree it will take time. :-)


I think that having 2 build farms building in parallel is a strength.
So let exploit it. :-) What one could have in mind is to challenge the
outputs; if they are identical, let keep only one version “somewhere”
and remove the other from the “elsewhere”.

For instance, we (I? with help) could resume this discussion:

<https://lists.gnu.org/archive/html/guix-devel/2020-10/msg00181.html>


Or maybe, for the identical outputs, one could imagine (dream? for) a
cooking service for missing outputs.  Well, I do not know how this is
actionable. :-)


Cheers,
simon