[bug#70169] [PATCH 0/7] Reproducible `make dist' tarball in defiance of Autotools and Gettext

[Ludovic Courtès]

Hi!

Janneke Nieuwenhuizen <janneke@gnu.org> skribis:

> The recent XZ-utils <https://www.openwall.com/lists/oss-security/2024/03/29/4>
> debacle inspired me to resurrect and finish my patch set for creating a
> reproducible source tarball for Guix, i.e., finally have `make dist' be
> reproducible (when run from Git).  I've been using a version of these patches
> in simpler projects for some years now and stole one from Timothy Samplet's
> Gash project.

Yay, kudos to you and Timothy!

> Autotools and Gettext still make it harder than necessary to do reproducible
> (responsible?) computing, which is especially sad given the fact that the
> Reproducible Builds project recently had their 10th birthday
> <https://reproducible-builds.org/_lfs/presentations/2023-05-27-R-B-the-first-10-years/#/>.
>
> Gettext tooling embeds timestamps found in the file-system, fails to respect
> SOURCE_DATE_EPOCH, and lacks options like `--pot-creation-date' so that we
> have to resort to SED to fixup.  The caching of all sorts of information, in
> separate build stages, also doesn't help.

Sadness indeed.  Hopefully things will improve in the coming weeks, now
that there’s an impetus.

> To create a reproducible source tarball, having a reproducible build
> environment is a prerequitite, so this would have to be recorded too.
> Using this patch set, I created a tarball doing something like
>
> guix pull --commit=1dbe492b993a7629df3b35146ce0272b52913776
> guix shell
> bootstrap && ./configure --localstatedir=/var --sysconfdir=/etc && make dist
> guix hash guix-1.3.0.57425-80a228.tar.gz
> 0mk59ay5k2dxmjni9fx4i8qyfhvlgxbhqzsjpg2pbw381nskkxbj

I applied the whole series on top of
df64d48e6f9f648044aa5279c045b8d6f7bee604 (the ‘base-commit’ at the
bottom of your message).  Thus I got the same content as you but with a
different commit ID.

“make dist” gave me guix-1.3.0.57425-9f4a4a.tar.gz.  The name indeed
corresponds to the tip of my tree:

--8<---------------cut here---------------start------------->8---
$ guix hash guix-1.3.0.57425-9f4a4a.tar.gz
0z3c4f8g6rsi9n0j8cwzwvw4bc59srg6bl3jj8yi60hbr9vrz5ql
$ git log |head
commit 9f4a4adfa778b281b794b61014e06dad98b6c945
Author: Janneke Nieuwenhuizen <janneke@gnu.org>
Date:   Wed Apr 3 21:11:09 2024 +0200

    maint: Ensure generated file reproducibility for dist.
    
    * doc/local.mk (override $(srcdir)/doc/stamp-vti): New target override.
    (doc-clean, man-clean): New targets.
    (auto-clean): Depend on it in new target.
    (DIST_CONFIGURE_FLAGS): New variable.
--8<---------------cut here---------------end--------------->8---

But as a result, I get a different hash, and since the directory in the
tarball has a different name, ‘diffoscope’ isn’t very helpful.

There’s at least one relevant difference in the gzip metadata:

--8<---------------cut here---------------start------------->8---
--- guix-1.3.0.57425-9f4a4a.tar.gz
+++ /tmp/guix-1.3.0.57425-80a228.tar.gz
├── filetype from file(1)
│ @@ -1 +1 @@
│ -gzip compressed data, from Unix, original size modulo 2^32 208138240 gzip 
compressed data, rese
rved method, ASCII, extra field, encrypted, from FAT filesystem (MS-DOS, OS/2, 
NT)
│ +gzip compressed data, from Unix, original size modulo 2^32 222504960 gzip 
compressed data, rese
rved method, ASCII, has CRC, was "", has comment, encrypted, from FAT 
filesystem (MS-DOS, OS/2, NT
--8<---------------cut here---------------end--------------->8---

(Your tarball has a CRC and comment, mine doesn’t.)

Maybe we’ll have to iterate once you’ve pushed a first version, so we
can truly build the same thing.  Or we should push the branch somewhere
(or use the one from <https://data.qa.guix.gnu.org/> once it’s been
created).

Thanks!

Ludo’.