Re: guix weather -m etc/sources-manifest.scm and CI?

[Christopher Baines]

zimoun <zimon.toutoune@gmail.com> writes:

> Playing with the new 'etc/sources-manifest.scm', using fb32a38, I get:
>
> $ guix weather -m ~/src/guix/guix/etc/source-manifest.scm
> computing 16,831 package derivations for x86_64-linux...
> looking for 16,831 store items on https://ci.guix.gnu.org...
> https://ci.guix.gnu.org
>   74.6% substitutes available (12,556 out of 16,831)
>   at least 65,367.1 MiB of nars (compressed)
>   81,988.1 MiB on disk (uncompressed)
>   0.095 seconds per request (1,606.8 seconds in total)
>   10.5 requests per second
>
>   0.0% (0 out of 4,275) of the missing items are queued
>   5 queued builds
>       aarch64-linux: 4 (80.0%)
>       powerpc64le-linux: 1 (20.0%)
>   build rate: .00 builds per hour
>       powerpc64le-linux: 0.00 builds per hour
>       aarch64-linux: 0.00 builds per hour
>       i686-linux: 0.00 builds per hour
>       x86_64-linux: 0.00 builds per hour
> looking for 16,831 store items on https://bordeaux.guix.gnu.org...
> https://bordeaux.guix.gnu.org
>   99.8% substitutes available (16,804 out of 16,831)
>   62,195.0 MiB of nars (compressed)
>   108,212.7 MiB on disk (uncompressed)
>   0.049 seconds per request (829.2 seconds in total)
>   20.3 requests per second
>   (continuous integration information unavailable)
>
>
> The questions are:
>
> Why ci.guix.gnu.org contains only 75%?  And bordeaux almost everything?
> (I guess the missing ones on bordeaux are corner cases as icecat, 
> linux-libre).

bordeaux.guix.gnu.org is hopefully only missing substitutes for things
where there's actually an issue building them. It's possible not to
guess though and instead ask guix weather what is missing.

I used time-machine as well for the latest commit processed by
data.guix.gnu.org, just so that the list doesn't include builds which
haven't started yet.

→ guix time-machine --commit=33bc3fb2a5f30a6e21f1b8d6d43867d921bd951c -- 
weather --substitute-urls=https://bordeaux.guix.gnu.org --display-missing -m 
./etc/source-manifest.scm
computing 16,850 package derivations for x86_64-linux...
looking for 16,850 store items on https://bordeaux.guix.gnu.org...
https://bordeaux.guix.gnu.org
  99.8% substitutes available (16,822 out of 16,850)
  62,377.9 MiB of nars (compressed)
  108,593.9 MiB on disk (uncompressed)
  0.160 seconds per request (4.5 seconds in total)
  6.3 requests per second
  (continuous integration information unavailable)

Substitutes are missing for the following items:
  /gnu/store/42knh9b75m6kc30m8v247sswhdfqnn8i-xpp3-1.1.4_src.tgz                
                                                                         
i686-linux
  /gnu/store/9fgglvi13vhpc63knf15dipzmmck6ia9-mini-os-git-checkout              
                                                                         
i586-gnu
  /gnu/store/pb5jmi9zalg6xylzsjmrskwxs0kar97l-fossil-src-2.11.tar.gz            
                                                                         
armhf-linux
  /gnu/store/8s5d2hjd6nhzf6m9dlxrykk8ijkf62pi-texlive-marginnote-51265-checkout 
                                                                         
armhf-linux
  /gnu/store/cp3ka40bhb28rrmyj4mzf9xjhi0ssxjx-CombBLAS_beta_16_2.tgz            
                                                                         
x86_64-linux
  /gnu/store/f1h94axw82id0k8c2lippg73sqlibqs8-dovecot-trees-2.1.0.tar.gz        
                                                                         
aarch64-linux
  /gnu/store/f12l7dg2z1xyf2wcw9g3v7jwpgd8m5zv-tla2tools-1.8.0-checkout          
                                                                         
i686-linux
  /gnu/store/53j9996hdgnmhgzswjjggdz9wnv29p5b-jpegsrc.v9d.tar.gz                
                                                                         
i586-gnu
  /gnu/store/80s1m4q2hnjfbqzw3fhywvsyim2b00cd-gcc-4.7.4.tar.bz2                 
                                                                         
x86_64-linux
  /gnu/store/csv5xca0p8w5jqqx53szy2dja8lwxma2-unicode-blocks.txt                
                                                                         
i686-linux
  /gnu/store/xn62dzq9hw3qnvmbxyxjkvhlacs72rz7-canada1500.zip                    
                                                                         
armhf-linux
  /gnu/store/6vqin3by3nkn0sxhgwnzi9l7gflpfw1q-gcc-vc4-6.5.0-checkout            
                                                                         
x86_64-linux
  
/gnu/store/zxfkf2bzq7pp7nhmbdgzvmjp0iv46wds-propeller-gcc-b4f45a4725e0b6d0af59e594c4e3e35ca4105867-checkout
                                            i686-linux
  
/gnu/store/g98l0m8qsfdqybak2jz3ma2miv9bki4j-emacs-danneskjold-theme-0.0.0-2.e4d1f2c-checkout
                                                           x86_64-linux
  /gnu/store/bznj81ls02r7kwld6338dhba0pql7nik-Rserve_1.8-6.tar.gz               
                                                                         
i686-linux
  /gnu/store/fb0rqvv38pf02qv3zjvjsindjdhlbpca-rda_1.0.2-2.1.tar.gz              
                                                                         
x86_64-linux
  /gnu/store/m5jn626l55nwxhgs405byka7pzkg1arz-d3.v6.js                          
                                                                         
powerpc64le-linux
  /gnu/store/bf496ni94ik19gdxkgp0vddkfxbid2vf-d3.v5.js                          
                                                                         
mips64el-linux
  /gnu/store/c0kig783lv56vv0yc6i8rsvhaxdp52sd-paml4.9e.tgz                      
                                                                         
mips64el-linux
  /gnu/store/17pi83jai4wlnnv1dilqwv1xv6g35d7x-plink-1.07-src.zip                
                                                                         
aarch64-linux
  /gnu/store/c3k6gxx7yha95f4avia11ls71ndzfx34-ggtree_3.0.3.tar.gz               
                                                                         
x86_64-linux
  /gnu/store/g2fhcc16yk5asszzmih0cpkwqb3b9bb3-clusterProfiler_4.0.4.tar.gz      
                                                                         
x86_64-linux
  /gnu/store/7n2rk9n46am8qx03ppycvkcdbnh6qzkq-eye-1.4.1.tar.gz                  
                                                                         
aarch64-linux
  /gnu/store/izhvfxx0iiwv8p1xr8ag5mpvsi86x9kx-weightwatcher-1.12.tar.gz         
                                                                         
mips64el-linux
  /gnu/store/l3r5j8r4f6fh6bxa9va3681n3bclxpdq-stuff-1.26.0.tar.gz               
                                                                         
i686-linux
  /gnu/store/ymip20qmlz320r2zx0hhfxn70428n76z-skymaker-3.10.5.tar.gz            
                                                                         
armhf-linux
  /gnu/store/xqc7n8fsq5zxag37a2ay88cx6c3zsm51-mpfrcx-0.6.3.tar.gz               
                                                                         
x86_64-linux
  /gnu/store/yf2dgn6jcdwvbnzcjwcadc8lwr9ks0wr-eclib-20190909-checkout           
                                                                         
mips64el-linux


If you visit https://data.guix.gnu.org and add the store path to the
URL, you can get links to the failed builds ([1] for example).

1: 
https://data.guix.gnu.org/gnu/store/42knh9b75m6kc30m8v247sswhdfqnn8i-xpp3-1.1.4_src.tgz

> Does it make sense to duplicate the storage of all these origins?
>
> Using extensively "guix time-machine", I note that a lot of
> derivations are missing and thus they are built locally, which is
> costly on poor machine.  Could we reduce the duplication and so save
> some space in order to systematically keep these derivations?  It
> would greatly ease the Guix experience for "guix time-machine" users.
> :-)

In terms of duplication, I still think that an argument can be made that
bordeaux.guix.gnu.org is providing value, even if it's doing some of the
same stuff as ci.guix.gnu.org.

As for keeping build results, everything that's ever been built for
bordeaux.guix.gnu.org (that's only ~337739 things totalling ~1.4TiBs) is
still around. In some ways, this is because deleting things is a bit
more difficult, as the files aren't in the store, you can't just do a
guix gc.

However, I do want to try to never delete anything. That's going to
require a bit of work as the local storage on the bayfront machine will
be all used up at some point, but I do have the beginnings of a plan to
avoid this an keep building things.

signature.asc
Description: PGP signature