[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Experimental nar-herder support for serving fixed output files by ha

From: Ludovic Courtès
Subject: Re: Experimental nar-herder support for serving fixed output files by hash
Date: Mon, 27 Jun 2022 10:52:45 +0200
User-agent: Gnus/5.13 (Gnus v5.13) Emacs/28.1 (gnu/linux)


Christopher Baines <> skribis:

> With the nar-herder, the implementation is a little trickier. Since the
> nar-herder manages a collection of nars, rather than serving things from
> the store, it might have the file being requested but it's inside a
> probably compressed nar file. So, to respond to these requests, the
> nar-herder has to take the relevant nar file and then read the file out
> of it. I've now got an initial implementation of this:


> The code isn't great, there's some difficulty in extracting the single
> file from the nar, but the biggest problem is a limitation in the guile
> fibers web server. Currently, responses have to be read in to memory,
> which is fine for we pages, but not great if you're trying to serve
> files which can be multiple gigabytes in size. This also means that the
> first byte of the response is available when all the bytes are
> available, so the download is slow to start.

That, and in practice a cache (with some eviction mechanism) would be
necessary so nars don’t need to be extracted every time and so we can
use sendfile(2).

> With all of that said though, it does seem to work. For testing, I've
> enabled it on bishan, which serves the collection
> of nars. It only has IPv6 connectivity, so you'll only be able to try
> this out if you've got an IPv6 support locally:


> In terms of next steps, there's some things to do with improving the
> implementation, but it would be good to hear if this is actually
> worthwile?

IWBN to share as much code as possible with ‘guix publish’, which has
great test suite coverage and is being hammered every day.  Clearly the
bit about extracting nars is specific to the nar-herder though, so that
may prove difficult.



reply via email to

[Prev in Thread] Current Thread [Next in Thread]