bug#41824: Dejagnu's unknown proc aborts testsuite run when triggered in test-case

[Pedro Alves]

On 6/26/20 11:58 PM, Jacob Bachmeyer wrote:

>> The issue for me is that the "gdb Summary" tally did not
>> mention the aborted testcase.  I would like to see something
>> like this instead:
>>
>>  # of expected passes            208
>>  # of nasty OMG FIX! tcl errors    1
>>
>> :-)
>>
>> Your UNRESOLVED is of course better than the status quo.
>>   
> 
> UNRESOLVED is a step in the right direction, but I agree that more is needed. 
>  We are somewhat limited by POSIX, which does not define a separate result 
> type for "test case crashed" and seems to roll that into UNRESOLVED, although 
> POSIX appears to require a testsuite to have a strictly-defined set of tests 
> and that a run must produce results for exactly all of them, which requires 
> information DejaGnu does not normally have about the running testsuite.  So, 
> while UNRESOLVED is needed, we could also maintain a separate count of 
> UNRESOLVED-due-to-crash, or simply store away the error 
> message/errorCode/errorInfo tuples and emit them a second time at the end of 
> the run, also easy in Tcl 8.

I'm not sure how much is POSIX compliance important these days.  I would hope 
that that
would not limit DejaGnu's development and limit its potential to be more 
usable.  Even
if POSIX compliance is important, I would think that that could be put behind
a --posixly_correct option, and/or POSIXLY_CORRECT environment variable, like 
other
GNU tools do:

 https://www.gnu.org/prep/standards/html_node/Non_002dGNU-Standards.html

Or conversely, we could have a --gnu mode, though I would say that a GNU tool
should behave in GNU mode by default.

I really don't buy that adding different result types is an issue, as
hinted earlier.  We've actually done it in GDB, for a loooong while.  We
have distinct XFAIL vs KFAIL output messages show up in the .sum file (though
those aren't distinguished in the summary file, doing so would be convenient,
hence I see that as a GDB testsuite bug).

And just very recently, we've added PATH and DUPLICATE output messages,
and those do show up in the summary:

# of expected passes            89593
# of unexpected failures        547
# of expected failures          146
# of known failures             104
# of untested testcases         20
# of unresolved testcases       5
# of unsupported tests          97
# of paths in test names        1      <<<<<
# of duplicate test names       370    <<<<<

I would think that even these two would be useful to have in
DejaGnu proper for all testsuites.

v1/rationale     - 
https://sourceware.org/pipermail/gdb-patches/2020-April/167847.html
v3/final version - 
https://sourceware.org/pipermail/gdb-patches/2020-April/168105.html

Thanks,
Pedro Alves