On 6/25/20 7:53 PM, Jacob Bachmeyer wrote:
The original rationale for using UNRESOLVED here was that DejaGnu
converts a test result to UNRESOLVED if too many errors or warnings were
produced. The DejaGnu manual indicates (section "A POSIX Compliant Test
Framework") that UNRESOLVED is correct for a test where execution was
interrupted or was set up incorrectly. UNTESTED is specifically listed
as a placeholder for an as-yet-unwritten testcase. A "typical" GDB
testsuite run has almost a hundred UNTESTED results but (without this
patch) zero UNRESOLVED results.
To me, this seems that UNRESOLVED is correct here, or that the manual
has an error.
So having written both, there is always the chance I've interpreted
things differently. If the test is interrupted, or has errors/warnings,
like timing problems for example, then it's UNRESOLVED. But a bug in Tcl
code enough to trigger unknown is UNTESTED, as the test never really
ran. I'd go with which definition the toolchain teams prefer, as this
effects validation testruns. I could go either way.
I don't know if POSIX 1003.3 is still considered a standard, that was
a long time ago. The testsuites were primarily focused on XOpen and
POSIX conformance tests, not toolchains. I was
on the standards committee, so used them as a model cause they seemed a
good standard. But the details of the interpretation can now be whatever
we want it to be. We're probably the only ones left running TET
compliant testsuites anyway. :-)