Re: Dropping gzip-compressed substitutes

[Mathieu Othacehe]

Hey Ludo,

> As discussed on IRC, I’m skeptical about this because:
>
>   1. It requires the development and testing of a custom tool that’s
>      easy to get wrong—e.g., it removes a gzipped nar for something that
>      had nothing but gzip available, etc.
>
>   2. That code would have to run with privileges that give it access to
>      the signing key on berlin.
>
>   3. Those 6.5 TB are an initial constant factor; growth of the storage
>      requirements going forward probably matters more and
>      <https://issues.guix.gnu.org/53901> will give us more flexibility
>      on that.

While those are valid points, we need to keep in mind that it is
important that we manage to move the store to the new SSD array quite
quickly to start GCing again.

If we cannot manage to remove those gzip nars then, I see only two
alternatives:

* Host the nars on the HDD array only.

* Host the nars elsewhere, on a VPS as you are proposing.

> I like Chris Baines’ idea of decoupling nar distribution from nar
> building.  If we want to keep nars long enough so that ‘time-machine’ is
> usable, then storage requirements will keep growing.
>
> Perhaps that means we can regularly copy nars “elsewhere” for long-term
> storage, using nar-herder, rsync, or whatever.  The machine that stores
> nars long-term has low requirements compared to the build farm because
> we don’t need to trust it for anything other than storage.  If that
> makes things easier (and financially viable), a VPS is good enough.

Sure, the VPS would also allow us to have a less European-centric
hosting. I did not follow closely the development of the
nar-herder. Chris what improvements this tool would bring compared to a
rsync based approach?

Thanks,

Mathieu