Re: Experimental nar-herder support for serving fixed output files by hash

[Christopher Baines]

Ludovic Courtès <ludo@gnu.org> writes:

>> The code isn't great, there's some difficulty in extracting the single
>> file from the nar, but the biggest problem is a limitation in the guile
>> fibers web server. Currently, responses have to be read in to memory,
>> which is fine for we pages, but not great if you're trying to serve
>> files which can be multiple gigabytes in size. This also means that the
>> first byte of the response is available when all the bytes are
>> available, so the download is slow to start.
>
> That, and in practice a cache (with some eviction mechanism) would be
> necessary so nars don’t need to be extracted every time and so we can
> use sendfile(2).

I'd actually imagined that this would be used infrequently, but yeah, if
the decompression does become a bottleneck, then some caching reverse
proxy could help reduce that.

>> With all of that said though, it does seem to work. For testing, I've
>> enabled it on bishan, which serves the bordeaux.guix.gnu.org collection
>> of nars. It only has IPv6 connectivity, so you'll only be able to try
>> this out if you've got an IPv6 support locally:
>>
>>   
>> https://bishan.guix.gnu.org/file/0ad-0.0.25b-alpha.tar.xz/sha256/1p9fa8f7sjb9c5wl3mawzyfqvgr614kdkhrj2k4db9vkyisws3fp
>
> Nice!
>
>> In terms of next steps, there's some things to do with improving the
>> implementation, but it would be good to hear if this is actually
>> worthwile?
>
> IWBN to share as much code as possible with ‘guix publish’, which has
> great test suite coverage and is being hammered every day.  Clearly the
> bit about extracting nars is specific to the nar-herder though, so that
> may prove difficult.

I'm going to look at the Guile Fibers web server, hopefully that can be
improved to support streaming responses, which would allow removing a
lot of custom code from guix publish.

There isn't all that much code to the nar-herder though, and most of
waht is there is doing different things to guix publish, so I'm not sure
there's all that much to share.

What I was getting at here though, ignoring the implementation, was
whether this is worthwhile to do? As in, is there benefit to having this
and being able to extend the content addressed mirrors that Guix uses?

Thanks,

Chris

signature.asc
Description: PGP signature