|
From: | Jacob Bachmeyer |
Subject: | bug#41824: Dejagnu's unknown proc aborts testsuite run when triggered in test-case |
Date: | Fri, 26 Jun 2020 17:58:09 -0500 |
User-agent: | Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.8.1.22) Gecko/20090807 MultiZilla/1.8.3.4e SeaMonkey/1.1.17 Mnenhy/0.7.6.0 |
Pedro Alves wrote:
On 6/26/20 10:53 AM, Pedro Alves wrote:to repeat at the very end of a test run with a big warning about test cases that crashed?I did not suggest to repeat anything.I misunderstood you here -- I somehow thought that that by "repeat" you meant re-running the testcase.
No, repeating output of at least the name of the file, and possibly the entire error dump.
Still, I would not suggest to repeat the error. Let me clarify -- with this: [...] Now, you already have an indication that something went wrong because make existed with error. But people may miss that.
They will easily miss that -- DejaGnu exits with a failure status if any tests produced "surprising" results: FAIL, XPASS, or UNRESOLVED. The GDB testsuite typically has multiple FAIL results, so exiting with an error code is routine.
The issue for me is that the "gdb Summary" tally did not mention the aborted testcase. I would like to see something like this instead: # of expected passes 208 # of nasty OMG FIX! tcl errors 1 :-) Your UNRESOLVED is of course better than the status quo.
UNRESOLVED is a step in the right direction, but I agree that more is needed. We are somewhat limited by POSIX, which does not define a separate result type for "test case crashed" and seems to roll that into UNRESOLVED, although POSIX appears to require a testsuite to have a strictly-defined set of tests and that a run must produce results for exactly all of them, which requires information DejaGnu does not normally have about the running testsuite. So, while UNRESOLVED is needed, we could also maintain a separate count of UNRESOLVED-due-to-crash, or simply store away the error message/errorCode/errorInfo tuples and emit them a second time at the end of the run, also easy in Tcl 8.
I am leaning towards this latter solution precisely because it would be loud and obnoxious, while still maintaining a professional tone. (Listing a count of "nasty OMG FIX! >:-( tcl errors", while certainly expressing the proper intent, probably should not find its way into a DejaGnu release, nor should an ASCII-art C'thulu reminding the user that broken testsuites drive programmers to madness, complete with randomly-chosen "your remaining sanity" score... :-) ) Repeating error dumps at the end of the run also fits well into an XML log model, where the collected errors could go into their own structure at the end of the file, without complicating the tag model used for the main test results. Simple importers would still get the inserted UNRESOLVED result, while more thorough or specialized tools could analyze the errors.
-- Jacob
[Prev in Thread] | Current Thread | [Next in Thread] |