nar-herder design retrospective

[Christopher Baines]

Hi!

I rushed the nar-herder [1] in to existence back in December, to address
the buildup of nars on bayfront by moving the nars to another machine
with more space to store them.

1: https://git.cbaines.net/guix/nar-herder/about/

This was something I was planning for a while though, I sent an email to
guix-devel outlining some points of the design and aims around a year
ago [2].

2: https://lists.gnu.org/archive/html/guix-devel/2021-02/msg00104.html

The nar-herder is currently in a stable state. It's packaged for guix
and there's a service to use it. There are probably some bugs, but I
think I've fixed the important initial ones.

I covered some information about the deployment of the nar-herder in
this email back in December [3] and that led to some really good
replies, I'll copy/paste some of the important bits below.

3: https://lists.gnu.org/archive/html/guix-devel/2021-12/msg00140.html

https://lists.gnu.org/archive/html/guix-devel/2021-12/msg00196.html:

> Regarding nar-herder, I think it’d be nice to have a solution to
> mirroring in Guix proper, developed similarly to other components,
> because it could be a fairly central tool.

> Usually I’m the one asking for blog posts :-), but I’d really like us as
> a project to collectively engage on the topic before we publicize this
> specific approach.

https://lists.gnu.org/archive/html/guix-devel/2021-12/msg00201.html:

> Why not extend “guix archive”?
> 
> However, without all the details so my remark is totally naive, I miss
> what “nar-herder” is doing that “guix archive”+rsync+“guix publish” is
> not doing – other said I miss why another SQL database is required to
> serve stuff from one place to another.  I have read README but I did not
> get the point.

https://lists.gnu.org/archive/html/guix-devel/2021-12/msg00204.html:

> I'm quite interested in learning more and potentially trying out the
> nar-herder! Some thoughts that I'd like to add to the design space:
> 
> I think it would be great if one of the pastures to which we herd the
> nars would be a free and open source software mirror site. In my
> experience, these are usually some static web hosting in front of a
> large disk with a place to run scripts to sync the content. A database
> server may not be available. I'd like to support this use case because I
> think it is a great way to build bridges to the communities who run or
> gather around these mirrors.
> 
> I'd also like the ability fetch nars directly from the local-to-me
> mirror rather than having them be proxied through a far way server.
> 
> One of the things that I really like and find empowering about Guix is
> that the developer/system administration tools are as available, easy
> to use, and convenient as the every day tooling. To the extent
> possible, I think that we should strive to make our syncing/mirroring
> solution practical to run for local, small setups, and not require
> project-scale infrastructure or coordination between many programs
> that are not captured in a Guix service.

So, currently the nar-herder can be used to move nars between machines,
which is what I wanted the initial implementation to be capable of. This
functionality should be sufficient for operating mirrors, although I'm
not aware of any being setup yet. I'd also like to be able to get
metrics about nar requests, but this isn't supported yet.

I'm all for having a solution to mirroring in Guix itself, although I
don't have a plan for this. Maybe the nar-herder could just be moved in
to Guix, maybe with a different name? Any other ideas?

I think moving the nar-herder repository on to Savannah is probably a
good thing to do regardless.

Thanks,

Chris

signature.asc
Description: PGP signature