[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: How did you handle making a GNU/Linux distribution?

From: Sage Gerard
Subject: Re: How did you handle making a GNU/Linux distribution?
Date: Tue, 24 Aug 2021 17:17:08 +0000

I'll answer your questions in a different thread to limit the noise in
this one. Look for the "Xiden/Guix design comparison" subject line
within the next 24h.

On 8/24/21 12:42 PM, Maxime Devos wrote:
> Sage Gerard schreef op ma 23-08-2021 om 18:24 [+0000]:
>> Thank you for the links!
>>> I miss which problem Xiden is solving and how it does.
>> You are not the first to say so, and I'm happy to learn more about why.
>> I'll try to explain in a different way here, so forgive the text wall.
>> I'll incorporate any feedback here to improve the white paper. I'd also like
>> to hear more about Guix's philosophy on what I'm going to talk about, because
>> I had reasons to end up with a different model.
> Maybe the publications at <> are 
> informative.
>> --
>> If we each use a program to distribute software, then those programs are 
>> probably
>> incompatible in some way (e.g. Cargo vs. Nexus' Vortex). Xiden addresses 
>> this by
>> decoupling the subjective elements of software distribution from the 
>> objective ones.
>>    That means modeling software distribution as (partly) a semantics 
>> problem. Racket didn't
>>   do this with its own package managers. Which is ironic, because it's hard 
>> to guarantee
>>   that that an arbitrary Racket program will get the dependencies they mean 
>> to use.
>>   I've written about this at length [1][2][3],
> About polyglot: packages in Guix can specify which version of a package they 
> use.
> E.g., Guix has multiple versions of 'mozjs', and 'mozjs@78.10.1' uses a 
> specific
> version of 'autoconf' (2.13) and 'rust' (1.41)
>      (native-inputs
>       `(("autoconf" ,autoconf-2.13)
>         ("automake" ,automake)
>         ("llvm" ,llvm)                   ;for llvm-objdump
>         ("perl" ,perl)
>         ("pkg-config" ,pkg-config)
>         ("python" ,python-3)
>         ("rust" ,rust-1.41)
>         ("cargo" ,rust-1.41 "cargo")))
> Is something missing from Guix here?  About [3]: this doesn't seem a problem 
> in Guix,
> if you're referring to the ability to define multiple versions of a package 
> at the same
> time (it's an ‘official policy’ to usually try to only keep the latest 
> version of a
> package though, but channels can ignore this completely).
>>   and spoke about it last year [4]. Xiden has changed a bit since the speech 
>> to be more flexible, and it is no longer limited for use in Racket projects. 
>> Apologies if you see something that looks contradictory.
>> What does that mean? Let's say Alice and Bob each write a Xiden launcher.
>> Alice
>> trusts SHA-384, but only if its implementation uses Keccak.
> SHA-384 and Keccak are separate things? SHA-384: a version of SHA-2,
> SHA-3: a version of Keccak.  By definition, implementations of SHA-384
> have no reason to use Keccak.
>> defines "sha384" as the canonical name of the CHF, and treat variants like 
>> "SHA-384" as aliases.
>> delegates trust in artifact signatures to GPG subprocesses.
> Guix allows changing the hash algorithm used, search for ‘content-hash’ in 
> the guix manual.
> Changing the hash algorithm globally unavoidably requires modifying every 
> package definition
> though, but that could be automated.
>> downloads content from her employer's S3 buckets.
> The ‘official’ substitute servers can be replaced:
> "guix build --substitute-urls=http://s3.bucket.employer";
> (after authorising that substitute server).
>> unconditionally prefers cached installations, such that every defined 
>> artifact
>> has exactly one stored copy on disk.
> Coming from Guix, I don't know what this means.  Does this mean only one 
> version
> of a package can be in the store at the same time, and building a newer 
> version
> deletes the older version?  How does this compare with ‘content 
> deduplication’?
>> All dependents access the stored artifact
>> using symbolic links or Windows junctions.
>> prefers declaring versions with edition and revision names.
>> Bob trusts SHA-1,
> See ‘content-hash’ above.  Also, why should Xiden let Bob trust SHA-1 in the 
> first place?
> (See 
> <>.)
>>   but only for artifacts from services authenticated by his operating 
>> system's certificates
> This is rather vague to me, unless you're referring to substitutes.
> What does this mean for origin specifications?
>      (source (origin
>                (method url-fetch)
>                (uri (string-append "mirror://gnu/hello/hello-" version
>                                    ".tar.gz"))
>                (sha256 ; imagine this was SHA-1 instead.
>                 (base32
>                  "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i"))))
> Bob got the origin specification from Xiden (or something like a Guix channel
> but for Xiden).  How would this ‘artifact’ authenticated by ‘operating system
> certificates’?  I mean, when you let your operating system build that origin
> (via guix (or Xiden?), which you could consider part of the OS), it will check
> that hash, and it will be considered valid (‘authenticated?’) if the hash 
> matches.
> There are no certificates at play here.
> I suppose the OS (guix or Xiden?) could then ‘sign’ it, but what value does 
> that
> provide to Bob?  (Unless Bob runs a substitute server on their computer.)
>> .
>> trusts digest creation and signature verification as implemented by his 
>> host's dynamically--linked `libcrypto` library for use in the same process.
>> downloads content from GitHub packages
> AFAIK there is no such thing a ‘GitHub packages’.  What is a ‘GitHub package’,
> precisely?  When I hear ‘X package’, I think of "X = python, haskell, ...’ 
> which
> have a standardised build sytem (represented by python-build-system
> and haskell-build-system in Guix), a standardised location to download them 
> from
> ( and, represented by the 'pypi' and 'hackage' 
> importers
> an refreshers in Guix) and are separate languages.
> The only thing things on GitHub have in common are:
>   * they have git repositories on GitHub
>   * some repositories on GitHub have ‘releases’, which can automatically be 
> discovered
> Notice dependency information and build system information are absent.
>> unconditionally prefers SxS installations,
> I searched for ‘SxS’, and found this: <>.
> I don't think you meant that.  What is ‘SxS’?
>>   such that packages cannot conflict in a shared namespace
> ‘packages cannot conflict in a shared namespace’ is a bit vague.
> Are you referring to ‘profile collisions’?  About having multiple versions
> of the same package on the same system?  Guix (and nix) can handle the latter.
> When the ‘propagated-inputs’ are limited, installing multiple packages each 
> using
> a different version of a package is possible.
> Is something missing from Guix here (please be very concrete)?
>>   at the cost of defeating an internal cache.
> I'm not sure what you mean here, can you illustrate this very concretely?
> And is something missing in Guix here?
>> prefers Semantic Versions
> If upstream doesn't do ‘semver’, and instead uses a versioning system like
> used by TeX (3.14, 3.141, 3.1415, 3.14159, ...), then Xiden cannot somehow
> let the package use ‘semver’ instead.  Unless you want to teach Xiden to
> convince upstream to change their versioning system?
>> These preferences are valid in Xiden's DSL for writing launchers.
> What is a launcher (compared to packages in Guix), and why would I want one?
> Why shouldn't I just install packages and run them ("guix package -i texmacs",
> start texmacs from some application chooser menu of the desktop environment).
>>   The invariants for each launcher can differ as much as Cargo and Vortex 
>> differ.
>>   What I wanted to do was allow users to declare their own invariants 
>> explicitly,
>>   but only when those invariants are wanted for subjective reasons.
> What are these invariants you're speaking of?  I don't think you're referring
> to, say, translation invariance (‘the laws of physics remain the same under a
> translation of the coordinate system’).
> Or do you mean something like ‘Invariant (computer science), an expression 
> whose
> value doesn't change during program execution’ 
> (<>)?
> But what are the ‘expressions’ in this case?
>>   I can't just write an overly-opinionated tool in this space because 
>> opinions
>>   have shelf-lives, and they can't interrupt our ability to get the content 
>> we
>>   need on demand, with our own safety guarentees.
> Is Guix too opinionated about something in particular?
>> Xiden launchers have advantages over shell scripts in that we can both still 
>> download
>> and install dependencies from the same servers, and create reproducible 
>> builds in terms
>> of the same (bit-identical) state on disk. We just differ on details that 
>> shouldn't impact
>> how we work. It also helps us patch holes faster, since if (say) a catalog 
>> maintainer tells
> What is a ‘catalog maintainer’ (maybe it's like someone with commit access to 
> a ‘guix channel’?)
> Also, what ‘shell scripts’ are you referring to?  I've been doing well 
> without writing any shell
> scripts.
>> us that a CHF was compromised, we can immediately revoke trust in the CHF 
>> independently
> Maybe guix could gain a command "guix style transition-hash OLD NEW" to 
> automagically
> rewrite package definitions to use the new hash algorithm instead of the old. 
>  It will
> need to download plenty of source code however, and it will entail a 
> world-rebuild.
> Note that, in practice, signs of the brokenness of hashes appear well in 
> advance of
> actual exploits.
>> in our own clients.
> What are these ‘clients’ you're speaking of?  I haven't been needing any 
> ‘clients’ lately.
>>   The package definitions are all expressed in terms of what the launchers 
>> allow.
> If 'package definition' = (package definition as in Guix), then this is false,
> Guix doesn't have a notion of ‘launchers’ and doesn't depend on ‘Xiden’.
> What are you meaning, exactly?  Also, I haven't found any package definitions 
> in Xiden,
> could you point me to any?
>> So what problem does this solve? I'm trying to preserve an element of 
>> walk-away power
>> in any tech community. Maybe your community goes in a questionable 
>> direction. Maybe your
>> desired software is in a different ecosystem.
> Guix channels can function independently of the main ‘guix’, augmenting 
> ‘guix’ with
> additional packages, even if they contravene official Guix policies (e.g. 
> non-free,
> bad quality ...), or simply if putting it in ‘guix proper’ takes to much time.
> There exist a few relatively well-known channels like that.
>>   Maybe the maintainers are abusive jerks.
> Technically, Guix has ‘maintainers’, but I wouldn't know where I could find 
> the list.
> Guix (the package manager) doesn't have its own infrastructure.
> It does have official substitute servers, but you can ignore them.
> So I don't see ‘maintainers are abusive jerks’ as a plausible threat to Guix.
> If they become existent, they can easily be replaced.
>> Maybe you have a `leftpad`/`event-stream` incident and the developer is 
>> unable/unwilling
>> to patch the problem.
> In Guix, if someone pushed malware (*) to, say, a package on, then 
> we (Guix)
> would simply stop using new versions of that package from  If 
> there's a healthy
> fork somewhere, we could switch over to that for new versions.
> (*) I could have confused the ‘`leftpad`/`event-stream` incident’ for another 
> incident.
>> You'd probably want a way to escape to a new community or distribution model
>>   without
> ‘Escape to a new community’ --> only if the community as a whole are ‘abusive 
> jerks’.
> If it's merely some maintainers, they can be removed or replaced.  If it's 
> all the
> maintainers, I'm sure a few well-regarded members of the community could come 
> together
> for a ‘coup’.  If it's the community as a whole, I don't see how Xiden could 
> help,
> unless you mean I would move from Guix to Xiden?
> Why would I want to ‘escape to a new distribution model’?  Writing something 
> like Guix
> from scratch seems a lot of work to me, I would rather fork Guix (with a few
> likewise-minded collaborators) than to start over again.  Or move to Debian 
> maybe.
> I don't see what ‘Xiden’ could offer here.
>>   giving up content or doing complex integration work. If Xiden can do that, 
>> users
>>   don't have to depend on the same middlemen to share work.
> Is ‘middlemen’ = guix committers here?  Do you have concrete scenario to 
> illustrate this,
>>   This is also better for
>>   developers because it doesn't oblige them to do everything their users 
>> want,
> Are we talking about upstream developers here, or guix(/Xiden) committers?
> How does this ‘obligation’ exist in the first place?
>>   while knowing users will adapt to inevitable distribution problems.
> I would rather not have to ‘adapt to problems’, I would rather that upstream 
> has something
> that works (though I can understand if occasionally mistakes are made).
>>   Everyone's consent
>>   becomes explicit, such that Xiden's "real" invariant is that consent 
>> between any
>>   two parties be mutually compatible.
> Consent about what?
>> I hope to improve user freedom from this angle,
>>   and my approach is biased by my experience with Racket's limitations in 
>> this space.
> Why are you comparing it with ‘racket’ instead of, say, Debian or Guix?
> ‘racket’ is a Scheme implementation, not a package manager.
>> The subjective parts I talk about go further into details like what name 
>> canons,
>> versioning schemes, trusted certificate chains, and even specific approaches 
>> to an exact
>> diamond dependency. The objective parts include integrity checking, 
>> signature verification,
>> safety limits, automatic dependency resolution, patches, protocols, and 
>> other concerns that
>> a decent package manager normally cares about. By keeping these separate, 
>> all of the below
>> features become cheap to write, without sacrificing trust & safety in the 
>> abstract.
>> Download archived GitHub repositories (source on Racket Discord atm)
> It can download GitHub repositories.  Why are you writing ‘archived’ here?
> Guix supports git repositories (see 'git-reference' in the guix manual), so
> I don't see what additional value Xiden brings here.  Can it also 
> automagically build
> the software?  Also, why ‘GitHub repositories’ specifically?  Why not, say,
> the git repositories at <>?
>> Manage Racket installations like how `nvm` manages Node.js installations [5]
> It would seem that your ‘racket-installation-manager’ [5] cannot resist a 
> compromise
> of  Also, I don't see how it 
> would be
> possibe to use ‘racket-installation-manager’ offline.  And how does it 
> compile the
> racket packages?  If we're talking about ‘trust’ and ‘escaping to a new 
> community’,
> maybe provide an option to override 'host-url-prefix'?  Currently, it's 
> hardcoding
> the ‘official’ racket community.
>> Produce a self-hosted Xiden installations, with implicit trust for the 
>> running Xiden instance [6]
>> Control which exact artifacts mnust be produced deterministically [7]
> Can I reproduce the ‘artifact’ (= store item?) on a separate machine?
>> Force generated bindings to be `eq?` (This one haunts Racket's package 
>> managers today) [8]
> That seems like a Scheme thing to me, not a package manager thing.
>> Source package definitions from Pastebin to download content, digests,
>> and signatures from a public S3 bucket (This does not exist today. I'm 
>> spitballing because I know
>>   it would work)
> How can this be secure?
>> I hypothesize that Xiden can "abstract over" package managers in the same 
>> way that Racket can
>> abstract over languages. I think people are used to package managers 
>> deciding things for them.
> Please provide some sane defaults.  I would rather not have to decide 
> everything myself,
> and be able to trust Guix(/Xiden?) to do something reasonable.
> Also, if you ‘abstract over package managers’, wouldn't you just end up with 
> one ‘super
> package manager’?  Why not ‘manage’ everything with a single package manager,
> like e.g. guix?
>> So what is Xiden for? Well, it's not for deciding an SOP for you.
> Searching for SOP, I find <>.
> What were you referring to here?
>>   It's for giving everyone the option to change the SOP for themselves with 
>> minimal politics.
> What are these changes you're referring to?
>>> what you would like to bootstrap?
>> All binaries related to creating a GNU/Linux distribution, such that I can 
>> reproduce an exact
>> OS, Racket installation, and Xiden instance. I want a trusted initial state 
>> on GNU/Linux.
>> To be clear, I don't need to do this for functional reasons. Xiden's already 
>> operational. I
>> just can't claim that you can trust a given Xiden instance with the same 
>> confidence as a
>>   Guix instance right now.
> If you want to get package definitions from guix, take a look at (guix 
> inferior).
> Maybe you could port the relevant code of (guix inferior) to Racket, to let 
> Xiden
> talk to Guix?   In particular, 'inferior-eval' sends an S-exp to the inferior 
> Guix,
> which is evaluated there, and the result is sent back.
> Greetings,
> Maxime.

reply via email to

[Prev in Thread] Current Thread [Next in Thread]