[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
branch master updated: doc: Add notes from the Guix Days 2020.
branch master updated: doc: Add notes from the Guix Days 2020.
Fri, 14 Feb 2020 08:55:59 -0500
This is an automated email from the git hooks/post-receive script.
civodul pushed a commit to branch master
in repository maintenance.
The following commit(s) were added to refs/heads/master by this push:
new 358c56c doc: Add notes from the Guix Days 2020.
358c56c is described below
Author: Ludovic CourtÃ¨s <address@hidden>
AuthorDate: Fri Feb 14 14:55:32 2020 +0100
doc: Add notes from the Guix Days 2020.
* doc/guix-days-2020: New directory.
doc/guix-days-2020/README | 2 +
doc/guix-days-2020/branches-releases-buildfarm.org | 50 +++++++++
doc/guix-days-2020/clojure_session.txt | 11 ++
doc/guix-days-2020/growing-guix.org | 95 +++++++++++++++++
doc/guix-days-2020/guix-secrets.org | 113 +++++++++++++++++++++
doc/guix-days-2020/guix_on_mobiles.txt | 29 ++++++
.../parameterized-packages-guix-days.org | 61 +++++++++++
doc/guix-days-2020/patch-review.md | 58 +++++++++++
doc/guix-days-2020/powerpc.org | 75 ++++++++++++++
doc/guix-days-2020/twt_notes-herd.txt | 49 +++++++++
10 files changed, 543 insertions(+)
diff --git a/doc/guix-days-2020/README b/doc/guix-days-2020/README
new file mode 100644
@@ -0,0 +1,2 @@
+This directory contains raw notes taken during the Guix Days 2020
+in Brussels, right before FOSDEM. Enjoy!
diff --git a/doc/guix-days-2020/branches-releases-buildfarm.org
new file mode 100644
@@ -0,0 +1,50 @@
+* Branches / Releases / Build farm
+ - One core updates (GCC 5 to GCC 7), broke, then got fixed after the
+ - No Guix release with time-machine
+ - Installing from the installation image, GCC update, nss-certs
+ embeds characters, locales cause substitutes to fail
+ - Install
+ - Do a pull
+ - Reconfigure fails
+ - A long time ago, a bug was introduced in to Guix, the guix package
+ was updated, this broke the installation image
+ - What's the point of having releases?
+ - Attracting new users
+ - Build servers
+ - Substitute availability
+ - Uncertian how good this is
+ - Testing reproducible builds
+ - Redundancy
+ - All in Europe/Networking
+*** TODO Find or file bug for nss-certs failure
+*** TODO Publishing release artefacts nightly
+*** TODO Consider time of year release schedule
+ - Rough objective
+*** TODO Prioritise package builds
+ - By NGinx logs for nar requests
+ - By build time
+*** TODO Gather statistics on the build process
+ - Build time
+ - Resource usage
+*** TODO Provide tooling to easily continuously and automatically build
packages ("Derivations at home")
diff --git a/doc/guix-days-2020/clojure_session.txt
new file mode 100644
@@ -0,0 +1,11 @@
+## Intermediate step
+Deploy standalone jar in guix
+- We need Maven to actually build the clojure libraries
+- Write a importer to create the package importert
+- Leiningen keeps package description as xml
+- The maven works needs to be continued
+- `lein pom` produces xml, which can then be imported through maven
+- We need to create a light weight extractor that is able to create a plugin
xml for maven plugin plugin
+- An alternative implementation to manager maven repositories
diff --git a/doc/guix-days-2020/growing-guix.org
new file mode 100644
@@ -0,0 +1,95 @@
+#+title: Growing Guix and Guix advocacy
+#+date: <2020-01-31 Fri>
+The story of bitcoin: didn't know anything of Guix, he noticed a problem
(bitcoin reproducibility) he found hex0, then Guix and he is now using it. We
should be made it easier to let people find Guix. Stumble across it. It's not
something people ignore.
+HPC use case: Reaching out specific communities. we have only two blog-posts
+- Embedded: we have two people that have been working on it. But it's not
really easy to install guix on embedded systems. With guix is still harder.
Including uboot configuration is difficult. Pick a board that works (eg.
raspberry pi uses blobs). Prepare a bootable image, ans how that it's possible
to edit the configuration file and have it working. With Yocto it's easy to
start but then it get complicated. You don't actually edit the image. Not an
use case yet. Why didn't you ha [...]
+- Experience on getting Guix on your company: worked, but it was a lot of
work. he promised something like "your builds will be reproducible" but he
does not do it for you.. Guix helps you, its the toolbox that helps you
archiving this, but if you use it bad he won't do it for you. They did not
want to learn how to use the tool. Now everyone use it. Once you know how to
+- Using cuirass in stead of Jenkins. Trying to make it independent from Guix
(having jobs that aren't Guix jobs).
+Viral videos? We are powering bitcoin! Part of the appeal.
+Not really advocacy, other company doing something similar to bitcoin might
start using it. Maybe even Mozilla. Issue tracker that was "did you see the
video from Carlos with bitcoin, we should do the same thing!".
+Do something to reach security people. Target groups:
+1. Security Conscious people
+2. Embedded world (too soon)
+ 1. maybe parameterized packages should help (lot of time you do not need
some features. You could customize the build but...)
+3. Reduces the headache he had with all his machines to only a single config
file he has on one machine.
+4. High Performance Computing -> Academia: many colleagues uses windows or
mac. He provides docker images built with Guix (so they are reproducible) but
always using docker. When he said "containers are bad" this year for the first
time they seems to know that he is correct.
+ 1. Reproduce article results. If the packages you use are in Guix, they
are reproducible. For this specific use case, we need to tell people why it
solves reproduciblity problems. We should keep doing that. Time-machine is
awesome. We don't know for sure, we don not have the proves. But the only
piece missing is experience.
+Exploit people to work for us: the goal is that at first we do advocacy for
us, but then other people should do it for us.
+Competitions; contests with prices. If more people are involved a competition
could work. eg. 10 years reproduciblity science challenge. Reproduce an old
paper results from 10+ years ago.
+Docker: how did they manage to get so popular? It's solving some problems,
but it's not the real solution.
+Is Guix more difficult to describe in stead of docker? People might not think
they have that problems. Dockers let you move software easily. People have
this problem and docker solves that.
+VLC now is popular, but at the start Mplayer had more
+GitlabCI: multiple backend to support your computation like docker. If instead
+Gitlab-ci.yml file, specify the dependency. Better if you have Guix directly:
We should make guix an executor for gitlab-ci. Add an executor.
+Jupyter: binder -> docker image. Binder can call pip or apt .. if we had Guix
backend we could make it easier for people to discover us.
+Would be cool to have Guix working with "galaxy". in galaxy you write your
pipeline, how you process the data. In each "box" of the pipeline you have
binaries. Now, galaxy works with docker or conda. When you go to the hospital
+Galaxy is quite popular in bioinformatics.
+We have a social problem: guix jupyter works fine, but if I wanted to modify
gitlab-ci, that's ruby.
+Science and so on: really killer future is the integration is the software
heritage: automatically archived, optimal for reproducibility in science.
+Add the package link to guix packages from each package page that guix
provides. Sometimes they are not included in ubuntu, but if you just add that
they can run "guix install package" that would be free. How to hack on this
project: guix environment package
+Just inform the software that the package is in guix. Useful for security
+In the same way that sometimes we have a patch that submit upstream and is
useful to let them know they have the package for guix.
+Marketing for software things: cool looking mascots. Docker hired a
cartoonist. Every week he did a small cartoon with the whale. But it's
+Racket manual is good to look at.
+Pronounce geeks might help search.
+Remo guix is a shop, catalan guix -> choke. Also a closed source software
+One other idea that comes to mind is to promote guix to promote events: web
eyecandy. Looks very nice, web service that demonstrates specific aspect.
Website that builds you docker images given the dependencies you want. Might
it's better to be prepared to scale.
+guix import upstream -> detect cmake, pip or else. Web service, takes the url
of a package, import it, creates the package
+But for many packages that would not work.
+Guix webpage where you click and get the image of the arm board ready to use.
Alternative to raspbian and other distributions.
+Graphic designers at FOSDEM, they have a boot where you can ask to work for
you. We should get in touch with them.
+Competition for the mascot.
+Tradition of animal as a mascot: we have the gnu, does it make sense for the
communication point of view? Does not need to be a gnu. Iguana is already
+Pick an animal that have an history of why it is this kind of animal. A
+Release more often.
+1. How creator of Bitcoin had a problem (reproducibility of their software)
and happened to find guix as the solution to his problem
+ - Could serve as an example to other security-focused communities
+ - We should be more easily discoverable, people should know about us, not
find us by chance
+ - People should talk
+2. Reach specific communities:
+ - Embedded devices (but we are not ready yet)
+ - Academia (tell them about software heritage integration, integrate with
+ - Maybe Mozilla?
+3. Contact upstream to tell them that a Guix package is available
+4. How Docker became popular
+ - gitlab-ci backend, jupyter
+5. Web service to provide docker images prepared by Guix or a Guix import
upstream that auto detects how to build a package
+6. Mascot: maybe contact somebody at FOSDEM
diff --git a/doc/guix-days-2020/guix-secrets.org
new file mode 100644
@@ -0,0 +1,113 @@
+* Secrets Management in Guix
+- Thinking about how secrets management is done in Guix currently vs. other
+- Secrets in Nix:
+ - Only in NixOps
+ - Attribute list of secrets, creates systemd services for each that
+ are inactive until you run "nixops send-keys(?)
+ - Can't reboot the system by itself, as secrets won't be available
+ - Not failure-tolerant
+- Create a shepherd service in guix that uses e.g. Vault, provide an
+ integration to retrieve secrets from there
+ - Could this be generalized?
+ - Maybe don't want to depend on Vault for simple deployments
+- Use cases
+ - Wireguard private keys
+ - DB passwords
+- Need to make sure we can provide secrets to different application in
+ the ways they expect; some applications only accept secrets on the
+ command line, some as files, etc
+ - Need a simple interface to retrieve secrets - something like "guix
+ get-my-secret" in guile
+- For Nix (not NixOps), this has been a long-running discussion
+ - One proposal to integrate secrets into the store, not merged
+ - Probably better not to manage secrets in the store
+- We have one service in Guix that manages secrets out-of-band
+ - knot dns
+ - Ad-hoc, specific to package
+- Could make simplistic Shepherd service that starts, populates a
+ directory with secrets
+ - How to get secrets in here? Out of band
+ - Different Shepherd service for each secret?
+- Currently have capability to specify SSH keys from local file
+ - Doing this for secrets would put them in the store, though
+- We have "local file" for g-expressions - we could have another
+ "secret file" type that never ends up in the store
+ - How would you guarantee that that path has the secret?
+ - You could query your operating-system to make sure it doesn't
+ contain a secret file object (to make sure your secrets exist first?)
+- Some people care very much about not having secrets persist on disk
+ (files could be on an in-memory FS)
+- Could define a Guix system service that extends another service
+ (e.g. postgres that needs credentials) to reflect the fact that this
+ dependency exists
+ - "Secret service" that exists primarily to be extended by other services
+ - Can use fold-services to collect a list of secret services for an
+ - About being able to express that a service depends on a secret,
+ and get the list of secrets that an operating-system depends on
+ - "guix deploy" would look at defined secrets locally, copy files to
+ remote system's secrets dir out of band
+- Multi-stage deploys might be too complex right now, but starting to
+ think about partially evaluated services
+- G-expression referring to a secret file that expands to the path to
+ the secret is probably the most natural way to do this
+ #+BEGIN_SRC scheme
+ (system* "mysql" "password123"
+ #$(secret "/home/me/mysqlpass"))
+ - Executed on the remote host
+ - Pass the name of the secret in the gexp
+ - Need to take care to ensure that the remote secret path is the
+ same as the path being used in the "secret service"
+ - Maybe there's something fancy we could do in e.g. mysql service to
+ make sure you only have to specify secret name once
+- To avoid differing local paths on different people's laptops, have a
+ directory to look in for relative secret names
+- We could have different backends in a secret service for e.g. files,
+ vault, etc - should expect that people will want to do this
+- Reviewing Guix services and extension points - e.g. colord service
+ extends account service
+ - Guix service extensions are not inheritance, it's composition
+- What if you're not deploying to another system and you want to run
+ services that use secrets?
+ - Still having a directory for "installed" secrets as with guix
+ deploy would make sense
+- Could trick someone into running a previous system generation with
+ current secrets
+ - This is effectively a type of database, and should be outside the
+ scope of Guix
+ - Same with other databases/other stateful applications
+ - Not called out explicitly in the manual (maybe it should be)
+- Need to accomodate applications that won't retry if the secret isn't
+ available yet
+ - Extension could say "I want you to block until the secret is there"
+ - Can we update the Shepherd service so that it waits to start the
+ - Secret service that creates extensions could create one Shepherd
+ service for each secret
+ - We could require the relevant secret service by name (by
+- Where do we store the secrets?
+ - It's the responsibility of the person deploying to make sure they
+ don't store secrets in their config files
+- What about secret versioning?
+ - We don't currently restart currently-running services
+ - Ideally there should be a way to force reload these secret
+ services (and dependent services?) when the secrets change
+ - Some applications watch for updated files, some don't, some reload
+ on receiving specific signals
+ - Generalize upgrade/downgrade actions? (This is a larger discussion)
+ - Just tell people to reboot for now?
+ - Some systems (e.g. Vault) allow you to create versions of secrets;
+ ideally we'd be able to take advantage of this
+ - Vault secret backend could create multiple secrets to do this
+ - Some people might want to keep old secrets around so they can
+ roll back, some people might not want to keep them around for
+ security reasons
+ - Best left to secret service backend
+- How do we deal with this in ~guix system~ and ~guix deploy~?
+ - They should know how to deal with these secrets
+ - Don't want to duplicate code to deal with secrets between the two
+- First backend to implement should probably be the local file backend
+ - Get something that works first, don't overcomplicate
diff --git a/doc/guix-days-2020/guix_on_mobiles.txt
new file mode 100644
@@ -0,0 +1,29 @@
+* Guix on mobile
+1. a lot of options on software side (guix on top of android, etc)
+2. 2 big things we can target from hardware wise ()
+we have the toolchain, we have uboot support we have the parts
+We want to use GuixSD as a smartphone os
+1. ui issues -> kde mobile, phosh gui on phones, both are actual linux booting
on phones, not android
+2. apps or programs to have a usefull phone (phone calls, sms, etc)
+3. proprietary drivers for modem, bluetooth, etc. In the meantime we could use
the prioprietary blobs to actually boot as a proof of concept
+Target first pinephone and librem5 which actually have linux support, to test
ui and app things and continue with other phones
+1. Start with packaging with phone apps (purism apps) while waiting for actual
compatible phones to be released (top-bottom approach)
+2. Use usb modems to test that the apps work
+3. osmocom project to build free software modem
+4. Use gnuradio on a stronger hardware to emulate gsm modem, to develop the
+5. We could also use lora via usb on a phone to communicate without the need
the gsm, proprietary hardware patents will expire in 20 years
+6. We could use alternatives (ip based video/audio communication) if wifi
works with free software
+tl;dr Most of the serious issues are because of the proprietary hardware,
software work can already be done
diff --git a/doc/guix-days-2020/parameterized-packages-guix-days.org
new file mode 100644
@@ -0,0 +1,61 @@
+* Parameterized Packages
+*** USE flags
+The inspiration for Parameterized Packages in guix largely comes from
+Gentoo Linux, which is well known for its powerful
[[https://wiki.gentoo.org/wiki/USE_flag][USE flags]], which
+allow the end user to customize various properties of the package
+build process. Common USE flags include *X*, for enabling X11 support,
+*dbus* to toggle dbus integration, and *debug* to enable debugging
+A key feature of USE flags is that they can be enabled recursively for
+every installed package that supports them, allowing extensive system
+configuration to be enabled with just a few USE flag declarations.
+Another powerful feature of package parameters is the ability to
+compose multiple paramaters together, allowing packages to describe
+custom behaviors for arbitrary combinations of parameters. The full
+list of supported parameters are passed in as an argument to the
+package record, allowing package writers to write custom behaviors for
+any combination of parameters.
+** Integration with Functional Builds
+A common problem with Gentoo USE flags occurs when packages in a user
+profile themselves depend the same package, only with conflicting USE
+flags set. Fortunately, guix's functional package builds allow us to
+have multiple versions of a package in the store that differ only in
+the parameter set.
+*** Hackability / customizability
+An easy interface for package customization would further guix's goal
+of being a hackable package manager.
+The flipside of increased expressiveness in package definitions is
+increased complexity. As the number of parameters grows, there is a
+explosion in the number of possible parameter combinations. This means
+that guix may be unable to test all possibile parameter combinations
+available to the end user.
+When a user customizes their package parameters, they are creating a
+distinct version of the package that will have a unique store
+hash. Therefore unless guix dedicates resources to building
+substitutes for common package parameters, they will have to compile
+** Questions / Discussions
+*** Dealing with propagation
+We agreed that we want to to develop an alternative approach to the
+Gentoo USE flag system. Guix packages should be able to declare which
+parameters that they understand, and in addition we believe that guix
+packages should explicitly declare which parameters are allowed to
+propagate to any dependencies of the package. If a parameter is not
+allowed to propagate, then when the dependencies are evaluated, they
+will not see that the parameter has been set, instead falling back to
+the parameter default. In addition, packages will not be able to set
+any additional parameters for their dependencies that the user has not
+*** Technical Considerations
+**** Host instead of build side
+Because we want to support parameters that can add additional
+dependencies to the package, the parameter handling logic needs to
+occur on the host side. One possibility suggested was for the
+parameters to be evaluated when the package is lowered into a bag.
diff --git a/doc/guix-days-2020/patch-review.md
new file mode 100644
@@ -0,0 +1,58 @@
+# Patch review in Guix
+Tom: patch review in GDB is different
+Nicolo: why are they slow? do something from Emacs
+Lars: automating, ML does not scale
+Jonathan: automating, VMs to test patches, don't break master branch
+Chris: reducing time between submit and commit of patches
+## Process ATM
+- People with commit access look at patches, review them and eventually commit
+- now on address@hidden list, before it was address@hidden
+- debbugs is a patch backend, includes meta data (close, done, reopen)
+- people can send feedback
+- no "dedicated" maintainers for individual packages
+- some patches never sent to guix-packages, instead directly committed to git
+- difficult to apply patches from mail
+- patches breaking master get pushed
+- long time for some patches to be reviewed
+- sending patches to mailing list is a high barrier for newcomers
+- dependencies between different patches/series
+- sending a patch series can lead to one bug per patch, when done wrong
+- submitting patches by pushing to a branch -> lots of people with commit
+ -> push to a branch on a "guix-staging" repo
+ -> reviewer gets informed and can pull from it
+- automatic tests (lint, build) after submit patches
+- Chris shows his patchwork/laminar/Guix Data setup
+ -> problems with patchwork
+ -> git-hook required
+ -> GDB tried to use patchwork -> too many patches
+ -> hard to "parse" git things in e-mail
+- how can we mix e-mail and git based review?
+ -> UI to write comments to bugs
+- just use Github/Gitlab
+ -> difficult to keep track
+ -> breaks existing work flows heavily
+- "offline" review, in irc, together, once a month or so -> increasing
motivation for reviewers
+- gamification/badges for reviewing
+## Small atomic actions
+Jonathan: "read-only" branches of patches
+Nicolo: git hook for sending the patches to debbugs
+Tom: advertise Chris' stuff he as already
+Chris: avoid push flooding
+## What can we improve
+- better process -> less people with commit access
+- testing, checking the patches, commiting by robot (like Rust), robot picks
+- sign-up for reviews for particular packages/themes (e.g. R or GNOME)
+## What others do
+- GDB (patchwork, gerrit)
+- Rust ("fixing" stuff on build farms VMs)
diff --git a/doc/guix-days-2020/powerpc.org b/doc/guix-days-2020/powerpc.org
new file mode 100644
@@ -0,0 +1,75 @@
+* PowerPC notes
+** There is an old PowerPC architecture, and that is known to work
+** Modern day Power PC, big endian is problomatic, but modern day Power PC
uses little endian
+** The big problem for PowerPC, is that the bootstrapping chain doesn't
include a new enough version of GCC
+** Bootstrapping other architectures, might encounter similar issues, like
+** The hardware is quite fast
+** Power 9 is a Power architecture, this implements PowerPC little endian
+** The interest is Power PC little endian
+** For Power 8 and 9, you need GCC 6.2, as it requires float 128 support
+** The current bootstrap uses GCC 2.95 as a stepping stone
+** Find out if GCC 6 runs on Power 9
+** GCC knows big endian Power
+** Start by cheating, cross compile GCC 6 or GCC 7
+** Previously the Guix bootstrap binaries contained GCC 5
+** When you have a Guix system, work on reducing the bootstrap path
+** First get something to work, then make it maintainable
+** It'll be easier once you're running on a Power system
+** There's support QEMU
+** Guix itself works, and the guix-daemon works
+** glibc, and/or gcc requires float 128
+** We should have a stepwise recipe for getting architectures working
+** The reduced binary seed bootstrap is coming to ARM
+** Directly jumping to GCC 7 would be good
+** There will be hardware in the future where GCC 6 just doesn't run
+*** Maintaining GCC 4.7, to support new architectures
+*** If Mes could compile C++, then this could build newer GCCs
+** Start talking to the GCC maintainers
+** C++ compiler in Guile would be possible, but take work
+** Could you supervise University students to write a compiler in Guile
+*** TODO Andreas to reach out to academics
+** Mes is not fully Guile compataible, but that's the direction
+** Take to Mark Wetter about parsing C++
+** Someone has to dare to attempt this
+** TODO Open bugs to track the issues
+** Would the GCC maintainers be willing to restrict themselves to a strict
subset of C++ (if they're not doing that already)
+** cfront, could you process C++ to C (transpile)
+** CentOS is supporting GCC 4.8.5, + patches
+*** It has float 128 support, as it builds for Power 9 (big and little endian)
diff --git a/doc/guix-days-2020/twt_notes-herd.txt
new file mode 100644
@@ -0,0 +1,49 @@
+ hurd using hurd as a kernel
+ microkernel, based upon old mac version - really old - 80s
+ mach! - v3 gnu modifiewd version
+ a MICROKERNEL ONLY BASSIC STUFF MEMEORY, JUST BOOKITING SYSTEM
+ USERPSPACE SERVERS TO CARE OF EVERYTHING
+ SERVERS PROVIDES WHATEVER YOU AEXEPDCGT FOR A ASYSTYEM
+ FOR GUIX, TO GET IT WORKING YOU NEED TO
+ 1ST CROSS BUILD BIRNARIES FROM GUIX AND LINUX SYSTEM
+ BUIULD CROSS BINARIES FOR SYSTEM
+ BUSILD BINARIES
+ add bianries oforom soucrce system
+ : anmy experience?
+ Some do
+ what for Janekeke simplieis boot process of guix
+ what guix doewsa is set of biaNRIES WHICH TRIUST
+ USE THE BINARIES TO BUILD SYTEM ON NEW Architecture
+ the tools fto build the hard binaries
+ were able tyo get bianries
+ there was a brtanch which included binaries
+ because of continius evolution
+ this ended up problems catching uip with cross toolls broke
+ nativeis system deploy failure
+ hurd is different from other ardhitectures...
+ for demon to work we needs support bind mound
+ herd does not have bind mounts
+ i wrote wrapper for mount for herd firmlinks
+ bindmounts linux
+ a mount of diorector y somewhere like a hard link
+ on hurd, can't do isolated builds using guix-daemon due to lack of bind-mount
+why can't we add bind-mount support to hurd? becaues by design you are
supposed to use firmlinks in order to achieve the effect of a bind mount.
+so, to ensure teh guix-daemon works on hurd, we should adjust the daemon to
+ q: what's teh status of packages we can build on hurd?
+ a: some packages can build, but teh builds are slow.
+ commencement packages have trouble building.
+ lots of packages had "if hurd, do ..." kind of statements.
+ but this situation has improved.
+ what needs to be done (tog et guix working on hurd):
+ - visibility of package breaks on hurd, active support in fixing breaks.
ensure that when we add a new architecture in guix, it does not break hurd
support. with hydra we continuously built packages for other packages, but
cuirass does probably does not do this - which causes people not to know when
changes break packages on the hurd. we also need more people to keep an eye on
braeking packages, and fix them.
+ - fix issues with commencement files. on guix, hurd has only ever worked on
x86_64. currently but even on x86_64, it does not work currently. we need to
fix this so it works at least on x86_64. We also need to ensuer we can build
hurd and use it on other architectures.
|[Prev in Thread]
||[Next in Thread]|
- branch master updated: doc: Add notes from the Guix Days 2020.,
Ludovic Courtès <=