Re: Turning off tests leads to a different store item

[Maxim Cournoyer]

Hi,

Simon Tournier <zimon.toutoune@gmail.com> writes:

> Hi,
>
> On Wed, 8 Nov 2023 at 20:20, Saku Laesvuori <saku@laesvuori.fi> wrote:
>
>> There is another way: simply preventing the tests from changing the
>> resulting store item. For example, the package could first be built
>> without tests and then that build tree could be copied to the build tree
>> of the build with tests enabled.
>
> Somehow, the store would need to keep all the build intermediary
> artifacts produced, right?  For instance, consider the extreme case
> where the build phase produces .o artifact files and the tests for
> whatever reasons relies on these temporary artifacts.
>
> Well, we had a quick chat with Josselin and Andreas about separating
> the tests at https://hpc.guix.info/events/2023/workshop/program/
> And my understanding of the rough conclusion we had: it is not easy
> and the evil are in all the details.  For example autotools: somehow
> "make check" is connected in one way or the other to "make" and/or
> "make install".  Somehow, the complete build tree (with intermediary
> artifacts as .o) should keep in the store.
>
> From a pragmatical point of view, there is packages where the tests
> cannot be totally separated from from the temporary build, therefore
> the question seems: how do these cases compare to the other regular
> cases?  What is the ratio?  Is the rule about many corner cases
> without a clear "regular"?  Or are they just few corner cases?

One easy-ish way, which would be kind of ugly because coupled to the
specific file system capabilities (e.g Btrfs), would be to leverage CoW
features and create a Btrfs snapshot at the beginning of the test suite,
and reverting to it after it's run.

But even that is not fullproof; that'd only protect the build directory,
say, not the store location (some check phases are moved after
installation).

I agree it looks tricky to get it right (and even trickier to prove/test
for it) :-).

I think the lower fruits are in looking at making the test suite of the
few common offenders more robust (using libfaketime or the likes) to
prevent (re)occurrences of time bombs in the future.

-- 
Thanks,
Maxim