Re: failure when rebuilding the past: long term?

[Ludovic Courtès]

Hi!

As a preamble, I’d like to say that what we’re doing in terms of time
traveling is ambitious; it’s never been done before, AFAIK.

And there are many pitfalls, as you write, mainly: disappearing source
(we’re working on it!), and, a bigger concern IMO, “non-deterministic”
build processes—in a broad sense: that includes build processes that
depend on CPU features, kernel versions, and other things not specified
in derivations.

I think we can address the latter, indirectly, with early cutoffs: if
two different derivations yield the same output, then we can consider
them equivalent and avoid rebuilding anything that depends on it,
instead just grafting them.  It helps in that it would allow users to
build ‘--without-tests’ and get an equivalent result, or to build with a
hypothetical ‘--in-virtual-machine’ or ‘--with-current-time=2020-01-01’
option.

zimoun <zimon.toutoune@gmail.com> skribis:

>  3. Having all the sources is one thing, but being able to rebuild is
>  another.  Failure of OpenBLAS [0] is one example, of some mesboot [1]
>  or of texlive [2] are others.  It appears to me that something is
>  inadequate with the current workflow pushing all to master without any
>  automated* checks. Other said, failures as 8f9fd9b70c (value "Unbound
>  variable: ~S") (value (r-biobase) seems wrong by design.  Well,
>  ac6f677249 is another recent example.  Somehow, because the package
>  collection is becoming larger and larger (which is good!)  then it is
>  becoming harder and harder to maintain the consistency both forward and
>  backward.  For the last Guix revision, my rough estimate is that ~5% of
>  packages are broken and my guess is that this number is “independant”
>  of the package collection size.  However, I already have some
>  collection of unreachable commits by the time-machine and then for some
>  reachable commits, I do not have numbers for what is effectively
>  rebuildable.  As discussed in this thread [3], maybe Guix is moving too
>  fast; or better worded, maybe the current workflow is inadequate with
>  some goals of long term and build all from sources.  I do not know…  My
>  point here: do we provide a list of commits (release, others) where we
>  apply more care for long term?

It’s hard for committers to push evidently broken code, such as unbound
variables, because ‘guix lint’, ‘make’, & co. catch it.  Still, it would
be great if it never happened at all, and for that enforcing server-side
checks could help.

The next issue is broken builds.  Automation can definitely help, and
Chris Baines’ work on automated patch handling or anything along these
lines would be a step in the right direction.

Non-deterministic builds are probably harder to automatically identify,
yet it’s a bigger concern.

Besides, I think we should do some sort of CI for time traveling, to
make sure we can travel forward and backward for different revisions.
(I would love to see research institutes invest in this work.)

My 2¢,
Ludo’.