bug#41824: Dejagnu's unknown proc aborts testsuite run when triggered in test-case

[Rob Savoye]

On 6/25/20 7:53 PM, Jacob Bachmeyer wrote:

> The original rationale for using UNRESOLVED here was that DejaGnu
> converts a test result to UNRESOLVED if too many errors or warnings were
> produced.  The DejaGnu manual indicates (section "A POSIX Compliant Test
> Framework") that UNRESOLVED is correct for a test where execution was
> interrupted or was set up incorrectly.  UNTESTED is specifically listed
> as a placeholder for an as-yet-unwritten testcase.  A "typical" GDB
> testsuite run has almost a hundred UNTESTED results but (without this
> patch) zero UNRESOLVED results.
> 
> To me, this seems that UNRESOLVED is correct here, or that the manual
> has an error.

  So having written both, there is always the chance I've interpreted
things differently. If the test is interrupted, or has errors/warnings,
like timing problems for example, then it's UNRESOLVED. But a bug in Tcl
code enough to trigger unknown is UNTESTED, as the test never really
ran. I'd go with which definition the toolchain teams prefer, as this
effects validation testruns. I could go either way.

  I don't know if POSIX 1003.3 is still considered a standard, that was
a long time ago. The testsuites were primarily focused on XOpen and
POSIX conformance tests, not toolchains. I was
on the standards committee, so used them as a model cause they seemed a
good standard. But the details of the interpretation can now be whatever
we want it to be. We're probably the only ones left running TET
compliant testsuites anyway. :-)

  Anyway, whether a test run aborts on an error or not is not defined,
so it's up to us to decide.

        - rob -