Re: Treating tests as special case

[Ludovic Courtès]

Hello!

I sympathize with what you write about the inconvenience of running
tests, when substitutes aren’t available.  However, I do think running
tests has real value.

Of course sometimes we just spend time fiddling with the tests so they
would run in the isolated build environment, and they do run flawlessly
once we’ve done the usual adjustments (no networking, no /bin/sh, etc.)

However, in many packages we found integration issues that we would just
have missed had we not run the tests; that in turn can lead to very bad
user experience.  In other cases we found real upstream bugs and were
able to report them
(cf. <https://github.com/TaylanUB/scheme-bytestructures/issues/30> for
an example from today.)  Back when I contributed to Nixpkgs, tests were
not run by default and I think that it had a negative impact on QA.

So to me, not running tests is not an option.

The problem I’m more interested in is: can we provide substitutes more
quickly?  Can we grow an infrastructure such that ‘master’, by default,
contains software that has already been built?

Ricardo Wurmus <address@hidden> skribis:

> An idea that came up on #guix several months ago was to separate the
> building of packages from testing.  Testing would be a continuation of
> the build, like grafts could be envisioned as a continuation of the
> build.

I agree it would be nice, but I think there’s a significant technical
issue: test suites usually expect to run from the build tree.

Also, would a test failure invalidate the previously-built store
item(s)?

Thanks,
Ludo’.