[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

bug#41824: Dejagnu's unknown proc aborts testsuite run when triggered in

From: Jacob Bachmeyer
Subject: bug#41824: Dejagnu's unknown proc aborts testsuite run when triggered in test-case
Date: Sat, 27 Jun 2020 12:29:45 -0500
User-agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv: Gecko/20090807 MultiZilla/ SeaMonkey/1.1.17 Mnenhy/

Rob Savoye wrote:
On 6/25/20 7:53 PM, Jacob Bachmeyer wrote:
The original rationale for using UNRESOLVED here was that DejaGnu
converts a test result to UNRESOLVED if too many errors or warnings were
produced.  The DejaGnu manual indicates (section "A POSIX Compliant Test
Framework") that UNRESOLVED is correct for a test where execution was
interrupted or was set up incorrectly.  UNTESTED is specifically listed
as a placeholder for an as-yet-unwritten testcase.  A "typical" GDB
testsuite run has almost a hundred UNTESTED results but (without this
patch) zero UNRESOLVED results.

To me, this seems that UNRESOLVED is correct here, or that the manual
has an error.

  So having written both, there is always the chance I've interpreted
things differently. If the test is interrupted, or has errors/warnings,
like timing problems for example, then it's UNRESOLVED. But a bug in Tcl
code enough to trigger unknown is UNTESTED, as the test never really
ran. I'd go with which definition the toolchain teams prefer, as this
effects validation testruns. I could go either way.

I see UNTESTED as a placeholder for a yet-to-be-written test; DejaGnu does not produce an error exit status after reporting UNTESTED.

Once Tcl has begun to execute "source $test_file_name", I see the test as running from the framework's perspective, so an error exit should be UNRESOLVED: we do not know the result that would have been produced had the Tcl error not occurred.

  I don't know if POSIX 1003.3 is still considered a standard, that was
a long time ago. The testsuites were primarily focused on XOpen and
POSIX conformance tests, not toolchains. I was
on the standards committee, so used them as a model cause they seemed a
good standard. But the details of the interpretation can now be whatever
we want it to be. We're probably the only ones left running TET
compliant testsuites anyway. :-)

POSIX 1003.3 appears to have been withdrawn.

  Anyway, whether a test run aborts on an error or not is not defined,
so it's up to us to decide.

Apparently, GDB has baked an assumption that the test run will not abort, ever, into their testsuite, and we had never documented circumstances that cause DejaGnu to abort, so I cannot really answer that with "your assumption is wrong".

-- Jacob

reply via email to

[Prev in Thread] Current Thread [Next in Thread]