[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[bug#46800] [PATCH] Allow defining multiple substituters
From: |
Ludovic Courtès |
Subject: |
[bug#46800] [PATCH] Allow defining multiple substituters |
Date: |
Fri, 12 Mar 2021 18:37:53 +0100 |
User-agent: |
Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) |
Hi Maxime,
Maxime Devos <maximedevos@telenet.be> skribis:
> On Tue, 2021-03-02 at 21:37 +0100, Ludovic Courtès wrote:
[...]
>> As discussed on IRC, the daemon used to have support for multiple
>> substituters, but as a built-in C++ interface, which I removed in
>> f6919ebdc6b0ce0286814cc6ab0564b1a4c67f5f.
>
> Was there any particular reason this support was removed, beyond
> moving from C++ to Scheme and the absence of any alternative substituters?
These were the main reasons, yes.
>> The Scheme interface you propose is of course nicer :-), but I’m still
>> not sure it’s necessary. For example, in the IPFS prototype at
>> <https://issues.guix.gnu.org/33899>;, IPFS support goes hand in hand with
>> HTTP support: narinfos are retrieved over HTTP and nars can be retrieved
>> over IPFS, or HTTP.
>
> About X going hand-in-hand with Y:
>
> Note that fetching narinfos, or fetching the nar itself are separated
> A method can support both procedures, or just one of them (or none,
> but that's rather useless.)
>
> Users (well, the system administrator) can choose multiple methods, which
> will be each fetch narinfos after each other & combine the
> results into
> one large list (or maybe some other data structure, I don't recall the
> details), and each substituter will be asked to produce
> a nar until a substituter
> succeeds or all have said "sorry, I don't have that nar".
OK.
> (That's different from C++ interface for multiple substituters I think, where
> the methods are only tried sequentialy, they aren't combined.)
>
> In case of IPFS, the idea is that *both* the IPFS and HTTP substituter are
> enabled, in that order: "--substitute-methods=ipfs http". The IPFS
> substituter
> won't be able to produce any narinfos by itself, but that's no problem as
> the HTTP substituter can find some. Then, the IPFS substituter will be asked
> first to download a substitute, as it's first in the "--substitute-methods"
> list.
>
> And what if the narinfo doesn't have a IPFS URI, as the substitute server
> doesn't
> support that? Then "guix substitute" automatically fall-backs to HTTP.
>
> Summary: some substitution methods can't do everything on their own, but
> that's ok,
> as "guix substitute" will just ask them to try what they can and will see if
> some
> combination of methods works.
Alright.
> About ‘not sure it's necessary’: there presumably will be a GNUnet substituter
> at some point. I suppose it would be possible to define all substitute
> methods
> in (guix scripts substitute), but then you would still end up with a procedure
> that tries all methods (e.g. in wip-ipfs-substitutes, process-substitution has
> an "if" structures that tries downloading via IPFS with fall-back to HTTP;
> this
> would become a (cond (ipfs? ipfs-code) (gnunet? gnunet-code) (#t http-code?))
I guess considerations that are more important to me (and to users, I
suppose) now than a few years back are maintainability and robustness.
Concretely, I wouldn’t want Guix to offer out of the box 4 methods, 3 of
which perform poorly or are downright buggy. I think it would be more
fruitful if, as a project, we would focus on one or two methods or
method combinations that we have battle-tested, perform well, and a nice
long-term maintenance story, and so on.
[...]
>> we’d rather let them choose a policy that
>> can automatically pick the “best” method, dynamically adjusting choices.
>
> Who's the user here?
> (a) the system administrator, who configuring the daemon to use a certain
> list of substituters and defines a default list of substitute uris.
> (b) the ‘user’, that doesn't directly have the capability to modify
> the system's guix daemon (or possibly an administrator that wants to
> to test some things out without the possibility of accidentally messing
> up the ‘real’ system).
I think (b) should be possible, just like users can pass
‘--substitute-urls’.
[...]
> About *automatically* dynamically adjusting choices: would be nice, but how is
> this supposed to work? Any ideas? The only thing I could think of is a
> allowing the user to choose which narinfo to use (e.g. from the list of found
> narinfos try to choose a narinfo that has an IPFS URI).
I think it’ll have to be fine-tuned once we have several stable
substitute methods. After all, we have yet to figure out how to choose
between zstd and lzip for the current substitution mechanism; the
tradeoffs when very different methods are in use may be more complex!
>> All in all, I would prefer to wait until there’s a clear need for this
>> abstraction.
>
> See above responses.
I don’t think my concerns are really addressed :-), but at the same time
I think we need a playground for these things so we can actually grow
new substitute methods like those you’ve been looking at. Hmmm tricky!
Ludo’.