[bug#46800] [PATCH] Allow defining multiple substituters

[Ludovic Courtès]

Hi Maxime,

Maxime Devos <maximedevos@telenet.be> skribis:

> On Tue, 2021-03-02 at 21:37 +0100, Ludovic Courtès wrote:

[...]

>> As discussed on IRC, the daemon used to have support for multiple
>> substituters, but as a built-in C++ interface, which I removed in
>> f6919ebdc6b0ce0286814cc6ab0564b1a4c67f5f.
>
> Was there any particular reason this support was removed, beyond
> moving from C++ to Scheme and the absence of any alternative substituters?

These were the main reasons, yes.

>> The Scheme interface you propose is of course nicer :-), but I’m still
>> not sure it’s necessary.  For example, in the IPFS prototype at
>> <https://issues.guix.gnu.org/33899>;, IPFS support goes hand in hand with
>> HTTP support: narinfos are retrieved over HTTP and nars can be retrieved
>> over IPFS, or HTTP.
>
> About X going hand-in-hand with Y:
>
> Note that fetching narinfos, or fetching the nar itself are separated
> A method can support both procedures, or just one of them (or none,
> but that's rather useless.)
>
> Users (well, the system administrator) can choose multiple methods, which
> will be each fetch narinfos after each other & combine the
> results into
> one large list (or maybe some other data structure, I don't recall the
> details), and each substituter will be asked to produce
> a nar until a substituter
> succeeds or all have said "sorry, I don't have that nar".

OK.

> (That's different from C++ interface for multiple substituters I think, where
> the methods are only tried sequentialy, they aren't combined.)
>
> In case of IPFS, the idea is that *both* the IPFS and HTTP substituter are
> enabled, in that order: "--substitute-methods=ipfs http".  The IPFS 
> substituter
> won't be able to produce any narinfos by itself, but that's no problem as
> the HTTP substituter can find some.  Then, the IPFS substituter will be asked
> first to download a substitute, as it's first in the "--substitute-methods" 
> list.
>
> And what if the narinfo doesn't have a IPFS URI, as the substitute server 
> doesn't
> support that?  Then "guix substitute" automatically fall-backs to HTTP.
>
> Summary: some substitution methods can't do everything on their own, but 
> that's ok,
> as "guix substitute" will just ask them to try what they can and will see if 
> some
> combination of methods works.

Alright.

> About ‘not sure it's necessary’: there presumably will be a GNUnet substituter
> at some point.  I suppose it would be possible to define all substitute 
> methods
> in (guix scripts substitute), but then you would still end up with a procedure
> that tries all methods (e.g. in wip-ipfs-substitutes, process-substitution has
> an "if" structures that tries downloading via IPFS with fall-back to HTTP; 
> this
> would become a (cond (ipfs? ipfs-code) (gnunet? gnunet-code) (#t http-code?))

I guess considerations that are more important to me (and to users, I
suppose) now than a few years back are maintainability and robustness.

Concretely, I wouldn’t want Guix to offer out of the box 4 methods, 3 of
which perform poorly or are downright buggy.  I think it would be more
fruitful if, as a project, we would focus on one or two methods or
method combinations that we have battle-tested, perform well, and a nice
long-term maintenance story, and so on.

[...]

>>  we’d rather let them choose a policy that
>> can automatically pick the “best” method, dynamically adjusting choices.
>
> Who's the user here?
> (a) the system administrator, who configuring the daemon to use a certain
>     list of substituters and defines a default list of substitute uris.
> (b) the ‘user’, that doesn't directly have the capability to modify
>     the system's guix daemon (or possibly an administrator that wants to
>     to test some things out without the possibility of accidentally messing
>     up the ‘real’ system).

I think (b) should be possible, just like users can pass
‘--substitute-urls’.

[...]

> About *automatically* dynamically adjusting choices: would be nice, but how is
> this supposed to work?  Any ideas?  The only thing I could think of is a
> allowing the user to choose which narinfo to use (e.g. from the list of found
> narinfos try to choose a narinfo that has an IPFS URI).

I think it’ll have to be fine-tuned once we have several stable
substitute methods.  After all, we have yet to figure out how to choose
between zstd and lzip for the current substitution mechanism; the
tradeoffs when very different methods are in use may be more complex!

>> All in all, I would prefer to wait until there’s a clear need for this
>> abstraction.
>
> See above responses.

I don’t think my concerns are really addressed :-), but at the same time
I think we need a playground for these things so we can actually grow
new substitute methods like those you’ve been looking at.  Hmmm tricky!

Ludo’.