Re: Test suite regressions vs expected failures

[John W. Eaton]

On 08/22/2017 02:00 PM, Mike Miller wrote:

> I think we should be consistent with the markings, so in theory the
> source tree can be scraped for bug numbers, the bug tracker can be
> scraped for the corresponding reports, and they will all be either open
> (no '*' marking) or closed as fixed (with '*' marking).

I already added a Makefile target (update-bug-status) to mostly automate 
this job.

> We could add another marking like '!' to indicate a bug that shows some
> behavior that we are not going to fix, but are intentionally adding a
> test anyway to show that we are not compatible.
>
> But since the test will always fail, what does that show? Should the
> test suite report a problem if a test marked "won't fix" actually passes
> instead?

I'm not sure exactly what to do, but it seems useful to me to somehow 
note incompatibilities that we know about but don't intend to fix so 
that we have some record of them.  That way we have some relatively easy 
to find pointer to the discussion(s) that resulted in marking them as 
"won't fix".

jwe