|
From: | Jacob Bachmeyer |
Subject: | bug#41824: Dejagnu's unknown proc aborts testsuite run when triggered in test-case |
Date: | Sat, 27 Jun 2020 12:42:05 -0500 |
User-agent: | Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.8.1.22) Gecko/20090807 MultiZilla/1.8.3.4e SeaMonkey/1.1.17 Mnenhy/0.7.6.0 |
Pedro Alves wrote:
On 6/26/20 11:58 PM, Jacob Bachmeyer wrote:The issue for me is that the "gdb Summary" tally did not mention the aborted testcase. I would like to see something like this instead: # of expected passes 208 # of nasty OMG FIX! tcl errors 1 :-) Your UNRESOLVED is of course better than the status quo.UNRESOLVED is a step in the right direction, but I agree that more is needed. We are somewhat limited by POSIX, which does not define a separate result type for "test case crashed" and seems to roll that into UNRESOLVED, although POSIX appears to require a testsuite to have a strictly-defined set of tests and that a run must produce results for exactly all of them, which requires information DejaGnu does not normally have about the running testsuite. So, while UNRESOLVED is needed, we could also maintain a separate count of UNRESOLVED-due-to-crash, or simply store away the error message/errorCode/errorInfo tuples and emit them a second time at the end of the run, also easy in Tcl 8.I'm not sure how much is POSIX compliance important these days. I would hope that thatwould not limit DejaGnu's development and limit its potential to be more usable.
We already have extensions; POSIX only allowed PASS/FAIL/UNTESTED/UNRESOLVED/UNSUPPORTED. [KX]{PASS,FAIL} are GNU extensions.
Even if POSIX compliance is important, I would think that that could be put behind a --posixly_correct option, and/or POSIXLY_CORRECT environment variable, like other GNU tools do: https://www.gnu.org/prep/standards/html_node/Non_002dGNU-Standards.html Or conversely, we could have a --gnu mode, though I would say that a GNU tool should behave in GNU mode by default.
POSIX compliance in DejaGnu depended on the testsuite being run. Any results generated by the framework itself need to be in the POSIX set, though testsuites can use extensions, as GDB has done. Actually adding more test result codes is dubious, as that will interact badly with XML output and efficient database processing, both of which can reasonably enumerate the nine result codes. ({,K,X}{PASS,FAIL}, UN{TEST,RESOLV,SUPPORT}ED)
I really don't buy that adding different result types is an issue, as hinted earlier. We've actually done it in GDB, for a loooong while. We have distinct XFAIL vs KFAIL output messages show up in the .sum file (though those aren't distinguished in the summary file, doing so would be convenient, hence I see that as a GDB testsuite bug). And just very recently, we've added PATH and DUPLICATE output messages, and those do show up in the summary: # of expected passes 89593 # of unexpected failures 547 # of expected failures 146 # of known failures 104 # of untested testcases 20 # of unresolved testcases 5 # of unsupported tests 97 # of paths in test names 1 <<<<< # of duplicate test names 370 <<<<< I would think that even these two would be useful to have in DejaGnu proper for all testsuites. v1/rationale - https://sourceware.org/pipermail/gdb-patches/2020-April/167847.html v3/final version - https://sourceware.org/pipermail/gdb-patches/2020-April/168105.html
I will look at the exact details later, but this sounds like it is probably a monkeypatch that will stop working in some future version when the procedures you are overriding disappear into an internal namespace, so we need to be thinking about what would be needed to provide extensible summary line sets, just as we need some way to extend language support in default_target_compile without overriding the entire procedure.
-- Jacob
[Prev in Thread] | Current Thread | [Next in Thread] |