[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Challenge substitute servers!

From: Ludovic Courtès
Subject: Challenge substitute servers!
Date: Tue, 20 Oct 2015 01:09:33 +0200
User-agent: Gnus/5.13 (Gnus v5.13) Emacs/24.5 (gnu/linux)


I’m happy to announce the new ‘guix challenge’ command!  (Documentation

The goal of the command is, ideally, to report malicious or corrupt
substitute servers.  In practice though, many package build processes
are still non-deterministic, so chances are that, when a discrepancy
reported by ‘guix challenge’, it is due to a non-deterministic build
process–something we must fix.

Here’s what it reports on my machine (compared to,
which is one of the build machines behind

--8<---------------cut here---------------start------------->8---
$ ./pre-inst-env guix challenge --substitute-urls=
finding garbage collector roots...
determining live/dead paths...
updating list of substitutes from ''... 100.0%
/gnu/store/1c96zwwg6lnh6v9n3m0q30pg0x2m0v5c-openssl-1.0.2d contents differ:
  local hash: 0725l22r5jnzazaacncwsvp9kgf42266ayyp814v7djxs7nk963q 
/gnu/store/3b7sjkz1ps719fkl53mcxzmjysd95i5c-emacs-deferred-0.3.2 contents 
  local hash: 0hh4d5plz4ph93yqcqvyy8dfbdidwdmy93kikjgnkglxr0lw8qss
/gnu/store/5zavfkmrax6z93q85q5nifbwkfz4704m-git-2.5.0 contents differ:
  local hash: 00p3bmryhjxrhpn2gxs2fy0a15lnip05l97205pgbk5ra395hyha 
/gnu/store/76yq0qvcbjjljk8my6x06ayssph573xx-pius-2.1.1 contents differ:
  local hash: 0k4v3m9z1zp8xzzizb7d8kjj72f9172xv078sq4wl73vnq9ig3ax 
/gnu/store/avrrq8sl1ihq0vrhjrilszgmg7ifhqdw-guile-ssh-0.8.0 contents differ:
  local hash: 1qxjnirrd24djxwrh1wmsdg3qhhigymaqg673nri0d5dn87dihmc 
/gnu/store/bc081jj8rx640ism3gm6wlpb7jamg080-dmd-0.2.01 contents differ:
  local hash: 1ngmjzscwva6w0wy3ahmq4r6svdpa8pq7f5kz273qld3k2xrg54v 
/gnu/store/cmb889dhvnx9kk6gdvqjm14w4j6pqalq-guix-0.8.3.abbe2c6 contents differ:
  local hash: 0p3mr40xk9q8bw26h8gkz2nx6n0csrmdh05zb1x85i0zs9pfd2f8
/gnu/store/fkl97li2g7gqyw5bq09q2r0hkxla54lq-ath9k-htc-firmware-1.4.0 contents 
  local hash: 0r4lysb0skx7dpyh1qsk2mamkwwd4a7yldr1yy2dpr7bc0nk4z8n
/gnu/store/j0qc3ghc7ajja6k9c35y8ssxjzxrsy95-emacs-debbugs-0.7 contents differ:
  local hash: 0wfczpznl9pdalf4sp23dfp63pvrk9jsw7znvw8axj24wv9rbk4c 
/gnu/store/n5q0gbn01w5m2ic9rc2xq4kj2faglg6s-emacs-typo-1.1 contents differ:
  local hash: 1yly4y92vphdsnqnmdqwsz019fy11jkbz31sl6ysly9j3fmnhq2m 
/gnu/store/q6xnjg9fd7lfhh7rq2l98grlyq7nbcf0-emacs-butler-0.2.4 contents differ:
  local hash: 14lb6cdfngjjlxrnipq961hbgnhwp47ap904a9mm0dj4q7pj23n2
--8<---------------cut here---------------end--------------->8---

If we diff as explained in the doc below, we see that the Git
discrepancies are due to timestamp in Perl’s POD files as well as
sorted-by-inode-number ‘tclIndex’ files.  For the Emacs modes, the
problem is the autogenerated autoloads files, which include some sort of
a timestamp as well.

You’re welcome to help fix these issues!

Comments welcome!


6.12 Invoking ‘guix challenge’

Do the binaries provided by this server really correspond to the source
code it claims to build?  Is this package’s build process deterministic?
These are the questions the ‘guix challenge’ command attempts to answer.

   The former is obviously an important question: Before using a
substitute server (*note Substitutes::), you’d rather _verify_ that it
provides the right binaries, and thus _challenge_ it.  The latter is
what enables the former: If package builds are deterministic, then
independent builds of the package should yield the exact same result,
bit for bit; if a server provides a binary different from the one
obtained locally, it may be either corrupt or malicious.

   We know that the hash that shows up in ‘/gnu/store’ file names is the
hash of all the inputs of the process that built that file or
directory—compilers, libraries, build scripts, etc.  (*note
Introduction::).  Assuming deterministic build processes, one store file
name should map to exactly one build output.  ‘guix challenge’ checks
whether there is, indeed, a single mapping by comparing the build
outputs of several independent builds of any given store item.

   The command’s output looks like this:

     $ guix challenge --substitute-urls="";
     updating list of substitutes from ''... 100.0%
     updating list of substitutes from ''... 100.0%
     /gnu/store/…-openssl-1.0.2d contents differ:
       local hash: 0725l22r5jnzazaacncwsvp9kgf42266ayyp814v7djxs7nk963q…-openssl-1.0.2d: 
     /gnu/store/…-git-2.5.0 contents differ:
       local hash: 00p3bmryhjxrhpn2gxs2fy0a15lnip05l97205pgbk5ra395hyha…-git-2.5.0: 
     /gnu/store/…-pius-2.1.1 contents differ:
       local hash: 0k4v3m9z1zp8xzzizb7d8kjj72f9172xv078sq4wl73vnq9ig3ax…-pius-2.1.1: 

In this example, ‘guix challenge’ first scans the store to determine the
set of locally-built derivations—as opposed to store items that were
downloaded from a substitute server—and then queries all the substitute
servers.  It then reports those store items for which the servers
obtained a result different from the local build.

   As an example, ‘’ always gets a different answer.
Conversely, ‘’ agrees with local builds, except in the case
of Git.  This might indicate that the build process of Git is
non-deterministic, meaning that its output varies as a function of
various things that Guix does not fully control, in spite of building
packages in isolated environments (*note Features::).  Most common
sources of non-determinism include the addition of timestamps in build
results, the inclusion of random numbers, and directory listings sorted
by inode number.  See <>, for more

   To find out what’s wrong with this Git binary, we can do something
along these lines (*note Invoking guix archive::):

     $ wget -q -O -…-git-2.5.0 \
        | guix archive -x /tmp/git
     $ diff -ur /gnu/store/…-git.2.5.0 /tmp/git

   This command shows the difference between the files resulting from
the local build, and the files resulting from the build on
‘’ (*note Comparing and Merging Files:
(diffutils)Overview.).  The ‘diff’ command works great for text files.
When binary files differ, a better option is Diffoscope
(, a tool that helps visualize differences for
all kinds of files.

   Once you’ve done that work, you can tell whether the differences are
due to a non-deterministic build process or to a malicious server.  We
try hard to remove sources of non-determinism in packages to make it
easier to verify substitutes, but of course, this is a process, one that
involves not just Guix but a large part of the free software community.
In the meantime, ‘guix challenge’ is one tool to help address the

   If you are writing packages for Guix, you are encouraged to check
whether ‘’ and other substitute servers obtain the same
build result as you did with:

     $ guix challenge PACKAGE

...  where PACKAGE is a package specification such as ‘guile-2.0’ or

   The general syntax is:

     guix challenge OPTIONS [PACKAGES…]

   The one option that matters is:

     Consider URLS the whitespace-separated list of substitute source
     URLs to compare to.

reply via email to

[Prev in Thread] Current Thread [Next in Thread]