Turning off tests leads to a different store item (was: Unable to build "Ten Years Reproducibility Challenge" paper)

[Suhail]

Felix Lechner via <help-guix@gnu.org> writes:

> On Tue, Oct 31 2023, Julien Lepiller wrote:
>
>> You could skip tests, but that would build a different derivation
>> tree, so it might affect reproducibility
>
> Those shortfalls are a consequence of our packaging habits. In an ideal
> world, would running a test suite ever build a different derivation?

Tests can fail for a number of reasons. At times tests may have some
failure modes that aren't so easily reliably overcome (as setting the
system clock).

By default, do we capture the test log and include it as part of the
build artifact? I'm not asking rhetorically.

If so, is there an existing discussion noting the pros/cons of that
decision that I can read to be better informed?

If not, why should skipping the tests result in a different derivation
tree? And if it shouldn't, but does, is there an existing bug report for
my reference?

> Please bear with me as I again belabor the same point without
> receiving public support. Building packages should be separate from
> testing them.

FWIW, I believe I agree with you. Perhaps following Simon's recently
proposed (not yet accepted) RFC process could help in this regard:
<https://issues.guix.gnu.org/issue/66844>.

Is there an existing issue which notes some of the drawbacks of the
existing approach and perhaps proposes some alternatives?

I do not know enough to know how trivial or not it would be to alter the
present behaviour (and what some of the unintended consequences might
be), but I believe this is an important topic that needs some thought
and consideration.

-- 
Suhail

This email is not an offer capable of acceptance, does not evidence an intention
to enter into an agreement, has no operative effect until a definitive agreement
is signed in writing by both parties, and that no party should act in reliance
on the email or any representations of the sender until a definitive agreement
is signed in writing by both parties.

This email may contain information that is privileged, confidential and/or
exempt from disclosure.  No waiver whatsoever is intended by sending this e-mail
which is intended only for the named recipient(s).  Unauthorized use,
dissemination or copying is prohibited.  If you receive this email in error,
please notify the sender and destroy all copies of this email.