Re: Remove references to test-output-distance.ly. (issue 549780044 by address@hidden)

[Dan Eble]

On Mar 30, 2020, at 17:34, Han-Wen Nienhuys <address@hidden> wrote:
> 
> On Sun, Mar 29, 2020 at 11:45 PM Dan Eble <address@hidden> wrote:
>> 
>> On Mar 29, 2020, at 17:39, Han-Wen Nienhuys <address@hidden> wrote:
>>>> test-output-distance was removed on the grounds that the self-test
>>>> serves the same purpose, but I don't see how it does.
>>> 
>>> Could you elaborate? What failure scenario are you worried about?
>> 
>> My question is, how does the self-test "verify that the regression tests 
>> have, in fact, run"?  I don't see how it could do more than verify that 
>> itself has run.
> 
> How does test-output-distance verify that the regression tests have run?
> 
> The verification doesn't come from the test file. It comes from the
> fact that someone is looking at the test output.

(Rhetorically) How much verification comes from looking at the output since you 
removed the test?

> The self-test gives us assurance that the test result index.html file
> is making sense.

Yes, and that does not overlap with the purpose of test-output-distance.

> You can verify that the tests were run by checking
> that the index.html file is there.

I think the only thing that the presence of index.html implies is that 
output-distance.py was run.

> For further context, I am interested in automating our test setup
> further: I want to have CI that can post feedback ("test failed!") on
> code reviews automatically. test-output-distance is a test that always
> fails, which makes that more complicated.

This is what the description of the commit that removed test-output-distance 
ought to have been.

I've been trying to point out that something of value was lost.  This last 
statement of yours finally gives something to weigh that value against.
-- 
Dan