[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: Remove references to test-output-distance.ly. (issue 549780044 by ad
Re: Remove references to test-output-distance.ly. (issue 549780044 by address@hidden)
Mon, 30 Mar 2020 20:38:07 -0400
On Mar 30, 2020, at 17:34, Han-Wen Nienhuys <address@hidden> wrote:
> On Sun, Mar 29, 2020 at 11:45 PM Dan Eble <address@hidden> wrote:
>> On Mar 29, 2020, at 17:39, Han-Wen Nienhuys <address@hidden> wrote:
>>>> test-output-distance was removed on the grounds that the self-test
>>>> serves the same purpose, but I don't see how it does.
>>> Could you elaborate? What failure scenario are you worried about?
>> My question is, how does the self-test "verify that the regression tests
>> have, in fact, run"? I don't see how it could do more than verify that
>> itself has run.
> How does test-output-distance verify that the regression tests have run?
> The verification doesn't come from the test file. It comes from the
> fact that someone is looking at the test output.
(Rhetorically) How much verification comes from looking at the output since you
removed the test?
> The self-test gives us assurance that the test result index.html file
> is making sense.
Yes, and that does not overlap with the purpose of test-output-distance.
> You can verify that the tests were run by checking
> that the index.html file is there.
I think the only thing that the presence of index.html implies is that
output-distance.py was run.
> For further context, I am interested in automating our test setup
> further: I want to have CI that can post feedback ("test failed!") on
> code reviews automatically. test-output-distance is a test that always
> fails, which makes that more complicated.
This is what the description of the commit that removed test-output-distance
ought to have been.
I've been trying to point out that something of value was lost. This last
statement of yours finally gives something to weigh that value against.