Build reproducibility metrics

[Christopher Baines]

Hey,

So there's the guix challenge command for looking at reproducibility
issues, but I've been wanting to more continuously and automatically
monitoring the reproducibility of packages.

The Guix Data Service has had the ability to do this for a few months
now, but there's not been enough data on substitutes. That's starting to
change though.

ci.guix.gnu.org has been doing a reasonable job of building packages,
but the proportion of packages built by bayfront.guix.gnu.org hasn't
always been very high. I'm hoping just having bayfront not build
core-updates and staging will help with this.

I've also been writing and trying to use the Guix Build Coordinator [1]
to build Guix packages and provide substitutes. That has got to the
point where it's not getting stuck every day at least, and there's more
than 80% of packages are available.

1: https://git.cbaines.net/guix/build-coordinator/about/

Combining that with the substitute server operated by Tobias, which has
a pretty awesome substitute availability of over 90% for recent
revisions, not only is there data from 4 different substitute servers to
use in the comparison, but the proportion of packages where there isn't
sufficient data is pretty low, below 10%.

I'm currently using the data.guix-patches.cbaines.net instance of the
Guix Data Service, you can see the package substitute availability for
the latest revision using this URL [1], and the package reproducibility
at this URL [2].

1: 
https://data.guix-patches.cbaines.net/repository/2/branch/master/latest-processed-revision/package-substitute-availability
2: 
https://data.guix-patches.cbaines.net/repository/2/branch/master/latest-processed-revision/package-reproducibility

Some caution is needed when interpreting this data. It's most probably
less up to date than what you'd get through running the guix weather or
guix challenge commands, as it takes the Guix Data Service time to query
the data, that querying process isn't very reliable at the moment
either. Additionally, the "matching" percentage could easily go down if
that output is built with a different hash in the future.

While the number itself maybe isn't the most useful thing, I like that
clicking through to the "Not matching" outputs will show a list of
outputs which didn't build reproducibly, which is something that could
help identify reproducibility issues to investigate and fix.

I think things are coming together on the substitute server side. The
goal I have in mind for this is for users of Guix to be able to have
greater trust in the substitutes they use, through trusting substitutes
only if it's been built reproducibly on multiple substitute servers. It
would be great to see work start soon on how guix as a client to
substitute servers might be enhanced to check for reproducibility when
fetching substitutes.

Thanks,

Chris

signature.asc
Description: PGP signature