[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: How did you handle making a GNU/Linux distribution?
From: |
Sage Gerard |
Subject: |
Re: How did you handle making a GNU/Linux distribution? |
Date: |
Tue, 24 Aug 2021 17:17:08 +0000 |
I'll answer your questions in a different thread to limit the noise in
this one. Look for the "Xiden/Guix design comparison" subject line
within the next 24h.
On 8/24/21 12:42 PM, Maxime Devos wrote:
> Sage Gerard schreef op ma 23-08-2021 om 18:24 [+0000]:
>> Thank you for the links!
>>
>>> I miss which problem Xiden is solving and how it does.
>> You are not the first to say so, and I'm happy to learn more about why.
>>
>> I'll try to explain in a different way here, so forgive the text wall.
>> I'll incorporate any feedback here to improve the white paper. I'd also like
>> to hear more about Guix's philosophy on what I'm going to talk about, because
>> I had reasons to end up with a different model.
> Maybe the publications at <https://guix.gnu.org/en/publications/> are
> informative.
>
>> --
>>
>> If we each use a program to distribute software, then those programs are
>> probably
>> incompatible in some way (e.g. Cargo vs. Nexus' Vortex). Xiden addresses
>> this by
>> decoupling the subjective elements of software distribution from the
>> objective ones.
>> That means modeling software distribution as (partly) a semantics
>> problem. Racket didn't
>> do this with its own package managers. Which is ironic, because it's hard
>> to guarantee
>> that that an arbitrary Racket program will get the dependencies they mean
>> to use.
>>
>> I've written about this at length [1][2][3],
> About polyglot: packages in Guix can specify which version of a package they
> use.
> E.g., Guix has multiple versions of 'mozjs', and 'mozjs@78.10.1' uses a
> specific
> version of 'autoconf' (2.13) and 'rust' (1.41)
>
> (native-inputs
> `(("autoconf" ,autoconf-2.13)
> ("automake" ,automake)
> ("llvm" ,llvm) ;for llvm-objdump
> ("perl" ,perl)
> ("pkg-config" ,pkg-config)
> ("python" ,python-3)
> ("rust" ,rust-1.41)
> ("cargo" ,rust-1.41 "cargo")))
>
> Is something missing from Guix here? About [3]: this doesn't seem a problem
> in Guix,
> if you're referring to the ability to define multiple versions of a package
> at the same
> time (it's an ‘official policy’ to usually try to only keep the latest
> version of a
> package though, but channels can ignore this completely).
>
>> and spoke about it last year [4]. Xiden has changed a bit since the speech
>> to be more flexible, and it is no longer limited for use in Racket projects.
>> Apologies if you see something that looks contradictory.
>> What does that mean? Let's say Alice and Bob each write a Xiden launcher.
>>
>> Alice
>>
>> trusts SHA-384, but only if its implementation uses Keccak.
> SHA-384 and Keccak are separate things? SHA-384: a version of SHA-2,
> SHA-3: a version of Keccak. By definition, implementations of SHA-384
> have no reason to use Keccak.
>
>> defines "sha384" as the canonical name of the CHF, and treat variants like
>> "SHA-384" as aliases.
>> delegates trust in artifact signatures to GPG subprocesses.
> Guix allows changing the hash algorithm used, search for ‘content-hash’ in
> the guix manual.
> Changing the hash algorithm globally unavoidably requires modifying every
> package definition
> though, but that could be automated.
>
>> downloads content from her employer's S3 buckets.
> The ‘official’ substitute servers can be replaced:
> "guix build --substitute-urls=http://s3.bucket.employer"
> (after authorising that substitute server).
>
>> unconditionally prefers cached installations, such that every defined
>> artifact
>> has exactly one stored copy on disk.
> Coming from Guix, I don't know what this means. Does this mean only one
> version
> of a package can be in the store at the same time, and building a newer
> version
> deletes the older version? How does this compare with ‘content
> deduplication’?
>
>> All dependents access the stored artifact
>> using symbolic links or Windows junctions.
>
>> prefers declaring versions with edition and revision names.
>> Bob trusts SHA-1,
> See ‘content-hash’ above. Also, why should Xiden let Bob trust SHA-1 in the
> first place?
> (See
> <https://en.wikipedia.org/wiki/SHA-1#SHAttered_%E2%80%93_first_public_collision>.)
>
>> but only for artifacts from services authenticated by his operating
>> system's certificates
> This is rather vague to me, unless you're referring to substitutes.
> What does this mean for origin specifications?
>
> (source (origin
> (method url-fetch)
> (uri (string-append "mirror://gnu/hello/hello-" version
> ".tar.gz"))
> (sha256 ; imagine this was SHA-1 instead.
> (base32
> "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i"))))
>
> Bob got the origin specification from Xiden (or something like a Guix channel
> but for Xiden). How would this ‘artifact’ authenticated by ‘operating system
> certificates’? I mean, when you let your operating system build that origin
> (via guix (or Xiden?), which you could consider part of the OS), it will check
> that hash, and it will be considered valid (‘authenticated?’) if the hash
> matches.
> There are no certificates at play here.
>
> I suppose the OS (guix or Xiden?) could then ‘sign’ it, but what value does
> that
> provide to Bob? (Unless Bob runs a substitute server on their computer.)
>
>> .
>> trusts digest creation and signature verification as implemented by his
>> host's dynamically--linked `libcrypto` library for use in the same process.
>> downloads content from GitHub packages
> AFAIK there is no such thing a ‘GitHub packages’. What is a ‘GitHub package’,
> precisely? When I hear ‘X package’, I think of "X = python, haskell, ...’
> which
> have a standardised build sytem (represented by python-build-system
> and haskell-build-system in Guix), a standardised location to download them
> from
> (pypi.org and hackage.haskell.org, represented by the 'pypi' and 'hackage'
> importers
> an refreshers in Guix) and are separate languages.
>
> The only thing things on GitHub have in common are:
>
> * they have git repositories on GitHub
> * some repositories on GitHub have ‘releases’, which can automatically be
> discovered
>
> Notice dependency information and build system information are absent.
>
>> unconditionally prefers SxS installations,
> I searched for ‘SxS’, and found this: <https://en.wikipedia.org/wiki/SxS>.
> I don't think you meant that. What is ‘SxS’?
>
>> such that packages cannot conflict in a shared namespace
> ‘packages cannot conflict in a shared namespace’ is a bit vague.
> Are you referring to ‘profile collisions’? About having multiple versions
> of the same package on the same system? Guix (and nix) can handle the latter.
> When the ‘propagated-inputs’ are limited, installing multiple packages each
> using
> a different version of a package is possible.
>
> Is something missing from Guix here (please be very concrete)?
>
>> at the cost of defeating an internal cache.
> I'm not sure what you mean here, can you illustrate this very concretely?
> And is something missing in Guix here?
>
>> prefers Semantic Versions
> If upstream doesn't do ‘semver’, and instead uses a versioning system like
> used by TeX (3.14, 3.141, 3.1415, 3.14159, ...), then Xiden cannot somehow
> let the package use ‘semver’ instead. Unless you want to teach Xiden to
> convince upstream to change their versioning system?
>
>> These preferences are valid in Xiden's DSL for writing launchers.
> What is a launcher (compared to packages in Guix), and why would I want one?
> Why shouldn't I just install packages and run them ("guix package -i texmacs",
> start texmacs from some application chooser menu of the desktop environment).
>
>> The invariants for each launcher can differ as much as Cargo and Vortex
>> differ.
>> What I wanted to do was allow users to declare their own invariants
>> explicitly,
>> but only when those invariants are wanted for subjective reasons.
> What are these invariants you're speaking of? I don't think you're referring
> to, say, translation invariance (‘the laws of physics remain the same under a
> translation of the coordinate system’).
>
> Or do you mean something like ‘Invariant (computer science), an expression
> whose
> value doesn't change during program execution’
> (<https://en.wikipedia.org/wiki/Invariant>)?
> But what are the ‘expressions’ in this case?
>
>> I can't just write an overly-opinionated tool in this space because
>> opinions
>> have shelf-lives, and they can't interrupt our ability to get the content
>> we
>> need on demand, with our own safety guarentees.
> Is Guix too opinionated about something in particular?
>
>> Xiden launchers have advantages over shell scripts in that we can both still
>> download
>> and install dependencies from the same servers, and create reproducible
>> builds in terms
>> of the same (bit-identical) state on disk. We just differ on details that
>> shouldn't impact
>> how we work. It also helps us patch holes faster, since if (say) a catalog
>> maintainer tells
> What is a ‘catalog maintainer’ (maybe it's like someone with commit access to
> a ‘guix channel’?)
> Also, what ‘shell scripts’ are you referring to? I've been doing well
> without writing any shell
> scripts.
>
>> us that a CHF was compromised, we can immediately revoke trust in the CHF
>> independently
> Maybe guix could gain a command "guix style transition-hash OLD NEW" to
> automagically
> rewrite package definitions to use the new hash algorithm instead of the old.
> It will
> need to download plenty of source code however, and it will entail a
> world-rebuild.
>
> Note that, in practice, signs of the brokenness of hashes appear well in
> advance of
> actual exploits.
>
>> in our own clients.
> What are these ‘clients’ you're speaking of? I haven't been needing any
> ‘clients’ lately.
>
>> The package definitions are all expressed in terms of what the launchers
>> allow.
> If 'package definition' = (package definition as in Guix), then this is false,
> Guix doesn't have a notion of ‘launchers’ and doesn't depend on ‘Xiden’.
> What are you meaning, exactly? Also, I haven't found any package definitions
> in Xiden,
> could you point me to any?
>
>> So what problem does this solve? I'm trying to preserve an element of
>> walk-away power
>> in any tech community. Maybe your community goes in a questionable
>> direction. Maybe your
>> desired software is in a different ecosystem.
> Guix channels can function independently of the main ‘guix’, augmenting
> ‘guix’ with
> additional packages, even if they contravene official Guix policies (e.g.
> non-free,
> bad quality ...), or simply if putting it in ‘guix proper’ takes to much time.
> There exist a few relatively well-known channels like that.
>
>> Maybe the maintainers are abusive jerks.
> Technically, Guix has ‘maintainers’, but I wouldn't know where I could find
> the list.
> Guix (the package manager) doesn't have its own infrastructure.
> It does have official substitute servers, but you can ignore them.
> So I don't see ‘maintainers are abusive jerks’ as a plausible threat to Guix.
> If they become existent, they can easily be replaced.
>
>> Maybe you have a `leftpad`/`event-stream` incident and the developer is
>> unable/unwilling
>> to patch the problem.
> In Guix, if someone pushed malware (*) to, say, a package on pypi.org, then
> we (Guix)
> would simply stop using new versions of that package from pypi.org. If
> there's a healthy
> fork somewhere, we could switch over to that for new versions.
>
> (*) I could have confused the ‘`leftpad`/`event-stream` incident’ for another
> incident.
>
>> You'd probably want a way to escape to a new community or distribution model
>> without
> ‘Escape to a new community’ --> only if the community as a whole are ‘abusive
> jerks’.
> If it's merely some maintainers, they can be removed or replaced. If it's
> all the
> maintainers, I'm sure a few well-regarded members of the community could come
> together
> for a ‘coup’. If it's the community as a whole, I don't see how Xiden could
> help,
> unless you mean I would move from Guix to Xiden?
>
> Why would I want to ‘escape to a new distribution model’? Writing something
> like Guix
> from scratch seems a lot of work to me, I would rather fork Guix (with a few
> likewise-minded collaborators) than to start over again. Or move to Debian
> maybe.
> I don't see what ‘Xiden’ could offer here.
>
>> giving up content or doing complex integration work. If Xiden can do that,
>> users
>> don't have to depend on the same middlemen to share work.
> Is ‘middlemen’ = guix committers here? Do you have concrete scenario to
> illustrate this,
>
>> This is also better for
>> developers because it doesn't oblige them to do everything their users
>> want,
> Are we talking about upstream developers here, or guix(/Xiden) committers?
> How does this ‘obligation’ exist in the first place?
>
>> while knowing users will adapt to inevitable distribution problems.
> I would rather not have to ‘adapt to problems’, I would rather that upstream
> has something
> that works (though I can understand if occasionally mistakes are made).
>
>> Everyone's consent
>> becomes explicit, such that Xiden's "real" invariant is that consent
>> between any
>> two parties be mutually compatible.
> Consent about what?
>
>> I hope to improve user freedom from this angle,
>> and my approach is biased by my experience with Racket's limitations in
>> this space.
> Why are you comparing it with ‘racket’ instead of, say, Debian or Guix?
> ‘racket’ is a Scheme implementation, not a package manager.
>
>> The subjective parts I talk about go further into details like what name
>> canons,
>> versioning schemes, trusted certificate chains, and even specific approaches
>> to an exact
>> diamond dependency. The objective parts include integrity checking,
>> signature verification,
>> safety limits, automatic dependency resolution, patches, protocols, and
>> other concerns that
>> a decent package manager normally cares about. By keeping these separate,
>> all of the below
>> features become cheap to write, without sacrificing trust & safety in the
>> abstract.
>>
>> Download archived GitHub repositories (source on Racket Discord atm)
> It can download GitHub repositories. Why are you writing ‘archived’ here?
> Guix supports git repositories (see 'git-reference' in the guix manual), so
> I don't see what additional value Xiden brings here. Can it also
> automagically build
> the software? Also, why ‘GitHub repositories’ specifically? Why not, say,
> the git repositories at <https://git.ngyro.com/>?
>
>> Manage Racket installations like how `nvm` manages Node.js installations [5]
> It would seem that your ‘racket-installation-manager’ [5] cannot resist a
> compromise
> of https://downloads.racket-lang.org/releases/. Also, I don't see how it
> would be
> possibe to use ‘racket-installation-manager’ offline. And how does it
> compile the
> racket packages? If we're talking about ‘trust’ and ‘escaping to a new
> community’,
> maybe provide an option to override 'host-url-prefix'? Currently, it's
> hardcoding
> the ‘official’ racket community.
>
>> Produce a self-hosted Xiden installations, with implicit trust for the
>> running Xiden instance [6]
>> Control which exact artifacts mnust be produced deterministically [7]
> Can I reproduce the ‘artifact’ (= store item?) on a separate machine?
>
>> Force generated bindings to be `eq?` (This one haunts Racket's package
>> managers today) [8]
> That seems like a Scheme thing to me, not a package manager thing.
>
>> Source package definitions from Pastebin to download content, digests,
>> and signatures from a public S3 bucket (This does not exist today. I'm
>> spitballing because I know
>> it would work)
> How can this be secure?
>
>> I hypothesize that Xiden can "abstract over" package managers in the same
>> way that Racket can
>> abstract over languages. I think people are used to package managers
>> deciding things for them.
> Please provide some sane defaults. I would rather not have to decide
> everything myself,
> and be able to trust Guix(/Xiden?) to do something reasonable.
>
> Also, if you ‘abstract over package managers’, wouldn't you just end up with
> one ‘super
> package manager’? Why not ‘manage’ everything with a single package manager,
> like e.g. guix?
>
>> So what is Xiden for? Well, it's not for deciding an SOP for you.
> Searching for SOP, I find <https://en.wikipedia.org/wiki/Same-origin_policy>.
> What were you referring to here?
>
>> It's for giving everyone the option to change the SOP for themselves with
>> minimal politics.
> What are these changes you're referring to?
>
>>> what you would like to bootstrap?
>> All binaries related to creating a GNU/Linux distribution, such that I can
>> reproduce an exact
>> OS, Racket installation, and Xiden instance. I want a trusted initial state
>> on GNU/Linux.
>>
>> To be clear, I don't need to do this for functional reasons. Xiden's already
>> operational. I
>> just can't claim that you can trust a given Xiden instance with the same
>> confidence as a
>> Guix instance right now.
> If you want to get package definitions from guix, take a look at (guix
> inferior).
> Maybe you could port the relevant code of (guix inferior) to Racket, to let
> Xiden
> talk to Guix? In particular, 'inferior-eval' sends an S-exp to the inferior
> Guix,
> which is evaluated there, and the result is sent back.
>
> Greetings,
> Maxime.
- Re: How did you handle making a GNU/Linux distribution?, (continued)