[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: xtest vs test
From: |
Carlo De Falco |
Subject: |
Re: xtest vs test |
Date: |
Sun, 3 Jul 2016 09:14:27 +0000 |
On 3 Jul 2016, at 08:17, LachlanA <address@hidden> wrote:
> Greetings all,
>
> In the context of bug 45507 [http://savannah.gnu.org/bugs/?45507], JWE said
> " I don't think anyone likes to see tests that fail, but removing the tests
> is definitely not what I'd like to see...
>
> I'm not even sure I like to see xtests because then people just get used to
> the idea that the tests fail, that they are expected to fail, and then don't
> even look to see why or whether they are problems that should really be
> fixed."
>
> I think this issue applies to many more bugs, so I'm replying here.
>
> I think there are four classes of people.
> 1. "Typical users" who just use the release, and trust that it works.
> 2. Those who compile a release from source, and use "make check" to see
> whether or not to trust the release.
> 3. Those who compile from dev sources, and report bugs.
> 4. Developers.
>
> Group 1 doesn't care about xtest vs test.
>
> To my understanding, xtest is to reassure group 2 that we are aware of the
> bug, and know that they can still trust Octave to work.
>
> For group 3, we'll reinstate a proper (not "x") test as soon as 4.2.0 is
> forked, and missing feedback for a few weeks doesn't matter.
>
> For group 4, we should treat xtest failures as candidates for fixing anyway.
>
> All of this means that I think that all* tests that we know will fail on
> relatively common systems, but do not mean that Octave becomes unreliable,
> should be xtests for the release (and reverted as soon as the 4.2.0 branch
> is forked from default).
>
> * That is an extreme position for argument's sake. Obviously there will be
> many exceptions.
>
> What do others think?
>
> Cheers,
> Lachlan
I like your suggestions!
What about also adding an option to test such as 'treat_xtest_as_test'
and set it on for development and off for release?
This would make the procedure you suggest more automatic ...
c.
- xtest vs test, LachlanA, 2016/07/03
- Re: xtest vs test,
Carlo De Falco <=
- Re: xtest vs test, Carnë Draug, 2016/07/31
- Re: xtest vs test, John W. Eaton, 2016/07/31
- Re: xtest vs test, John W. Eaton, 2016/07/31
- Re: xtest vs test, Doug Stewart, 2016/07/31
- Re: xtest vs test, Daniel J Sebald, 2016/07/31
- Re: xtest vs test, Daniel J Sebald, 2016/07/31
- Re: xtest vs test, Carnë Draug, 2016/07/31
- Re: xtest vs test, Daniel J Sebald, 2016/07/31
- Re: xtest vs test, Daniel J Sebald, 2016/07/31
- Re: xtest vs test, Carnë Draug, 2016/07/31