Re: regtest not catched by make check

[Jonas Hahnfeld]

Am Sonntag, den 08.11.2020, 11:20 +0100 schrieb Thomas Morley:
> Am So., 8. Nov. 2020 um 09:28 Uhr schrieb Jonas Hahnfeld <hahnjo@hahnjo.de>:
> > 
> > Am Sonntag, den 08.11.2020, 01:49 +0100 schrieb Thomas Morley:
> > > To be honest, before, I had not thought very deeply about how
> > > comparing regression tests with 'make  check' works.
> > > 
> > > But now I think a regression test should actually _test_ something.
> > > The test may succeed or fail.
> > > If it fails, (but the snippet itself compiles), then the result should
> > > be catched by 'make check'. In cases where this can't be made
> > > possible, the snippet should error, stop 'make check' and a meaningful
> > > error message should be returned.
> > 
> > While this would be ideal, it might not always be possible. That's
> > exactly why visual inspection may be required.
> 
> As mentioned here
> https://gitlab.com/lilypond/lilypond/-/merge_requests/497#note_442096848
> I actually did visual inspection manually.
> It was out of interest, not because I thought it was needed. Good I
> did so otherwise I wouldn't have noticed my first attempt to fix the
> issue was insufficient.

IMHO manual testing of the feature should always be part of the
development process.

> But there are not so many regtests for fret-diagrams, so it was not much work.
> Of course I looked in the obvious regtests, what if the patch caused
> some bad elsewhere?
> There should be some script widely comparing things. Well, like 'make check'.
> Other automated visual inspection is more a safety net. Imho, it
> should not be part of the usual development process of single patches.

I mentioned scripts/auxiliar/make-regtest-pngs.sh earlier in this
thread, maybe this already does what you mean? If not, I may not
understand what you're looking for...

Jonas

signature.asc
Description: This is a digitally signed message part