[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Treating tests as special case

From: Mark H Weaver
Subject: Re: Treating tests as special case
Date: Thu, 05 Apr 2018 16:26:50 -0400
User-agent: Gnus/5.13 (Gnus v5.13) Emacs/25.3 (gnu/linux)

Hi Pjotr,

Pjotr Prins <address@hidden> writes:

> and he gave me a new insight which rang immediately true. He said:
> what is the point of running tests everywhere? If two people test the
> same thing, what is the added value of that? (I paraphrase)
> With Guix a reproducibly building package generates the same Hash on
> all dependencies. Running the same tests every time on that makes no
> sense.

I appreciate your thoughts on this, but I respectfully disagree.

> I know there are two 'inputs' I am not accounting for: (1) hardware
> variants and (2) the Linux kernel. But, honestly, I do not think we
> are in the business of testing those. We can assume these work.

No, we can't.  For example, I recently discovered that GNU Tar fails one
of its tests on my GuixSD system based on Btrfs.  It turned out to be a
real bug in GNU Tar that could lead to data loss when creating an
archive of recently written files, with --sparse enabled.  I fixed it in
commit 45413064c9db1712c845e5a1065aa81f66667abe on core-updates.

I would not have discovered this bug if I had simply assumed that since
GNU Tar passes its tests on ext4fs, it surely must also pass its tests
on every other file system.

> If not, any issues will be found in other ways (typically a segfault
> ;).

The GNU Tar bug on Btrfs would never produce a segfault.  The only way
the bug could be observed is by noticing that data was lost.  I don't
think that's a good way to discover a bug.  I'd much rather discover the
bug by a failing test suite.

Tests on different hardware/kernel/kernel-config/file-system
combinations are quite useful for those who care about reliability of
their systems.  I, for one, would like to keep running test suites on my
own systems.


reply via email to

[Prev in Thread] Current Thread [Next in Thread]