reproducible tar archives

[Simon Josefsson]

Hi

We have code to make release tar archives reproducible:

https://git.savannah.gnu.org/gitweb/?p=gnulib.git;a=blob;f=top/GNUmakefile;hb=5b92dd0a45c8d27f13a21076b57095ea5e220870#l28

https://git.savannah.gnu.org/gitweb/?p=gnulib.git;a=blob;f=top/maint.mk;h=b2baa02edff3c7cf591caf5fd24a9b7c6717122a;hb=563046ab9b3208b4b6df650451bb888fbbffc073#l53

I just noticed a well-written section on reproducible tar archives:

https://www.gnu.org/software/tar/manual/html_node/Reproducibility.html

There is gap between these recommendations and what gnulib currently
have.  Can we improve here?  Thoughts:

1) Some of the suggested flags probably requires a newer GNU tar.  Even
the --sort=name that gnulib have is not supported by some tar that I
test things on, making 'make dist' fail on those platforms because tar
is too old.  I am sympathetic to not supporting old systems for
developers to make releases on, but running 'make distcheck' on old
systems is useful for ironing out platform-dependent problems.

2) Some choices may be opinionated, but I'm not sure which ones.  Maybe
PAX archives as in --format=posix?  I recall seeing some people
recommend --format=ustar for greater compatibility, which I think should
be a factor.  Isn't there a way to use ustar and fix the large file and
timestamp problems?  Are there platforms in use that doesn't support PAX
archives?  What is the disadvantage with --format=posix?

3) The --mtime="$SOURCE_EPOCH" is the most difficult one to implement.
Thoughts on how to do it?

Regarding 3), for Libidn I am contemplating this approach:

https://git.savannah.gnu.org/cgit/libidn.git/commit/?id=504caad385a7224f1c983308f2415d06f8449406

Which adds to cfg.mk:

+TAR_OPTIONS += --mode=go+u,go-w --mtime=$(abs_top_srcdir)/NEWS

and to Makefile.am:

+dist-hook: mtime-NEWS-to-git-HEAD
+.PHONY: mtime-NEWS-to-git-HEAD
+mtime-NEWS-to-git-HEAD:
+       $(AM_V_GEN)if test -e $(srcdir)/.git \
+                       && command -v git > /dev/null; then \
+               touch -m -d @"$$(git log -1 --format=%ct)" $(srcdir)/NEWS; \
+       fi

This is an ugly hack (for several reasons), but it has sustained some
testing, and has some properties than a more naive --mtime=$(shell git
log ...).

For most packages, tar is only one of the reasons for non-reproducible
source tarballs.  Other work is usually necessary too.  But this is one
that gnulib is able to improve on.

/Simon

signature.asc
Description: PGP signature