[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

bug#41824: Dejagnu's unknown proc aborts testsuite run when triggered in

From: Jacob Bachmeyer
Subject: bug#41824: Dejagnu's unknown proc aborts testsuite run when triggered in test-case
Date: Sat, 27 Jun 2020 12:42:05 -0500
User-agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv: Gecko/20090807 MultiZilla/ SeaMonkey/1.1.17 Mnenhy/

Pedro Alves wrote:
On 6/26/20 11:58 PM, Jacob Bachmeyer wrote:

The issue for me is that the "gdb Summary" tally did not
mention the aborted testcase.  I would like to see something
like this instead:

 # of expected passes            208
 # of nasty OMG FIX! tcl errors    1


Your UNRESOLVED is of course better than the status quo.
UNRESOLVED is a step in the right direction, but I agree that more is needed.  We are 
somewhat limited by POSIX, which does not define a separate result type for "test 
case crashed" and seems to roll that into UNRESOLVED, although POSIX appears to 
require a testsuite to have a strictly-defined set of tests and that a run must produce 
results for exactly all of them, which requires information DejaGnu does not normally 
have about the running testsuite.  So, while UNRESOLVED is needed, we could also maintain 
a separate count of UNRESOLVED-due-to-crash, or simply store away the error 
message/errorCode/errorInfo tuples and emit them a second time at the end of the run, 
also easy in Tcl 8.

I'm not sure how much is POSIX compliance important these days.  I would hope 
that that
would not limit DejaGnu's development and limit its potential to be more usable.

We already have extensions; POSIX only allowed PASS/FAIL/UNTESTED/UNRESOLVED/UNSUPPORTED. [KX]{PASS,FAIL} are GNU extensions.

if POSIX compliance is important, I would think that that could be put behind
a --posixly_correct option, and/or POSIXLY_CORRECT environment variable, like 
GNU tools do:


Or conversely, we could have a --gnu mode, though I would say that a GNU tool
should behave in GNU mode by default.

POSIX compliance in DejaGnu depended on the testsuite being run. Any results generated by the framework itself need to be in the POSIX set, though testsuites can use extensions, as GDB has done. Actually adding more test result codes is dubious, as that will interact badly with XML output and efficient database processing, both of which can reasonably enumerate the nine result codes. ({,K,X}{PASS,FAIL}, UN{TEST,RESOLV,SUPPORT}ED)

I really don't buy that adding different result types is an issue, as
hinted earlier.  We've actually done it in GDB, for a loooong while.  We
have distinct XFAIL vs KFAIL output messages show up in the .sum file (though
those aren't distinguished in the summary file, doing so would be convenient,
hence I see that as a GDB testsuite bug).

And just very recently, we've added PATH and DUPLICATE output messages,
and those do show up in the summary:

# of expected passes            89593
# of unexpected failures        547
# of expected failures          146
# of known failures             104
# of untested testcases         20
# of unresolved testcases       5
# of unsupported tests          97
# of paths in test names        1      <<<<<
# of duplicate test names       370    <<<<<

I would think that even these two would be useful to have in
DejaGnu proper for all testsuites.

v1/rationale     - 
v3/final version - 

I will look at the exact details later, but this sounds like it is probably a monkeypatch that will stop working in some future version when the procedures you are overriding disappear into an internal namespace, so we need to be thinking about what would be needed to provide extensible summary line sets, just as we need some way to extend language support in default_target_compile without overriding the entire procedure.

-- Jacob

reply via email to

[Prev in Thread] Current Thread [Next in Thread]