Re: Marking bugs as fixed in the test suite

[Mike Miller]

On Fri, Jul 07, 2017 at 14:27:39 -0400, John W. Eaton wrote:
> With this change, any failing tests that are marked this way are shown in
> the test suite summary as regressions.  Previously they were just noted as
> "expected failures" or "reported bugs" so it was easy to miss that a change
> actually reintroduced some bug that had been fixed.

I think it would be helpful to make these labels a little "louder".

With the current labeling they are still somewhat lost in the noise of
known failures, while they should be much more important. Instead of

Summary:

  PASS                            15422
  FAIL                                0
  XFAIL (expected failure)            5
  XFAIL (reported bug)               31
  XFAIL (regression)                  4
  SKIPPED (feature)                  50

maybe something like this instead

Summary:

  PASS                            15422
  FAIL                                0
  REGRESSION                          4
  XFAIL (expected failure)            5
  XFAIL (reported bug)               31
  SKIPPED (feature)                  50

> Finally, there are now 4 tests that are always failing and that are marked
> as regressions.  This is happening because the bug report is closed as
> fixed, but there are multiple parts to the report:
> 
>   http://octave.org/testfailure/?50893
> 
> The tests that fail are all items that have been marked as "won't fix". So,
> what should we do?  Open a separate bug report that won't be closed as
> fixed?  Add another marker for the tests to indicate "won't fix"?

I agree here. I went back to revision fd7a16594614 which resolved that
bug report, and the tests that were added did not pass then either. So
they were added at the time as cases that Octave did not pass. Maybe the
second asserts for each of these four tests were just typos?

This may not have been the case, but I don't think we want to add xtests
for things that we are intentionally not fixing. A better way would be
to test for what the value that Octave produces is, rather than what
Matlab does that we don't match. And maybe a FIXME or other comment
indicating why this test tests for something that is not compatible with
Matlab.

-- 
mike