January (2023) update on bordeaux.guix.gnu.org

[Christopher Baines]

Hey!

The last update was sent out in September [1], so 4 and a bit months
ago.

1: https://lists.gnu.org/archive/html/guix-devel/2022-09/msg00038.html


## Numbers

bordeaux.guix.gnu.org currently provides ~2 million nars, which take up
~8.3TiB to store.

Substitute availability is still reasonable I think. Additionally,
there's some substitutes for i586-gnu (the Hurd) now.


## Non master branch substitutes

One gradual change that's been happening over the last few months is
that there's more builds/substitutes on bordeaux.guix.gnu.org for non
master branch things (so for patches and branches like staging).


## Mirrors

No progress has happened in terms of mirrors.

There's still the US mirror I host [2] plus bishan effectively acts as a
mirror [3].

2: https://bordeaux-us-east-mirror.cbaines.net/
3: https://bishan.guix.gnu.org/

I think there's still the opportunity to sort out the DNS and
certificates so that bordeaux.guix.gnu.org refers to multiple servers,
increasing reliability and hopefully performance as well.


## Serving fixed output files by hash

This has progressed recently as Guile Fibers now supports streaming
responses without storing the whole response in memory [4].

4: https://github.com/wingo/fibers/pull/63

I've enabled this in the bayfront NGinx config, so you can now make
requests like [5]

5: 
https://bordeaux.guix.gnu.org/file/texlive-20210325-texmf.tar.xz/sha256/070gczcm1h9rx29w2f02xd3nhd84c4k28nfmm8qgp69yq8vd84pz

This seems to work for me on a fast connection, but the requests drops
part way through on a slower connection. I'm not quite sure why yet but
it's probably some NGinx configuration issue. You can also try making
these requests directly against bishan.


## Next steps

There's still a whole lot to improve on in multiple areas.

While bordeaux.guix.gnu.org is now a default substitute server, and
there's been a release since this change was made, the substitute
servers list is ordered, so the obvious question is what needs to change
so that bordeaux.guix.gnu.org is the substitute server that users should
try first? In my mind, the main blocking issue is the current lack of
zstd substitutes, but that's something that's being addressed now.

There's also the content addressed mirror issue (discussed above as
serving fixed output files by hash). I think a little more testing is
required, but then it should be possible to add bordeaux.guix.gnu.org to
the list of content addressed mirrors.

In terms of resource requirements, especially now that builds for
patches and non-master branches are happening, there's a use for more
build hardware. The more pressing issue though is storage
space. Currently the two machines responsible for storing all the nars
(bishan and hatysa) each have around 10TiB of storage for this purpose,
which will probably be all used up by July this year. It should be
possible to extend the storage in hatysa, but bishan will probably need
replacing.

There's also improvements to and around the build coordinator,
particularly in the area of observability (seeing what builds are
happening and queued).

If you're interested in working on any of this, do let me know as while
I don't have time to work on much of it myself, I should be able to make
time to help others.

Let me know if you have any comments or questions!

Thanks,

Chris

signature.asc
Description: PGP signature