[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

bug#41824: Dejagnu's unknown proc aborts testsuite run when triggered in

From: Pedro Alves
Subject: bug#41824: Dejagnu's unknown proc aborts testsuite run when triggered in test-case
Date: Sat, 27 Jun 2020 12:40:10 +0100
User-agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.2.1

On 6/26/20 11:58 PM, Jacob Bachmeyer wrote:

>> The issue for me is that the "gdb Summary" tally did not
>> mention the aborted testcase.  I would like to see something
>> like this instead:
>>  # of expected passes            208
>>  # of nasty OMG FIX! tcl errors    1
>> :-)
>> Your UNRESOLVED is of course better than the status quo.
> UNRESOLVED is a step in the right direction, but I agree that more is needed. 
>  We are somewhat limited by POSIX, which does not define a separate result 
> type for "test case crashed" and seems to roll that into UNRESOLVED, although 
> POSIX appears to require a testsuite to have a strictly-defined set of tests 
> and that a run must produce results for exactly all of them, which requires 
> information DejaGnu does not normally have about the running testsuite.  So, 
> while UNRESOLVED is needed, we could also maintain a separate count of 
> UNRESOLVED-due-to-crash, or simply store away the error 
> message/errorCode/errorInfo tuples and emit them a second time at the end of 
> the run, also easy in Tcl 8.

I'm not sure how much is POSIX compliance important these days.  I would hope 
that that
would not limit DejaGnu's development and limit its potential to be more 
usable.  Even
if POSIX compliance is important, I would think that that could be put behind
a --posixly_correct option, and/or POSIXLY_CORRECT environment variable, like 
GNU tools do:


Or conversely, we could have a --gnu mode, though I would say that a GNU tool
should behave in GNU mode by default.

I really don't buy that adding different result types is an issue, as
hinted earlier.  We've actually done it in GDB, for a loooong while.  We
have distinct XFAIL vs KFAIL output messages show up in the .sum file (though
those aren't distinguished in the summary file, doing so would be convenient,
hence I see that as a GDB testsuite bug).

And just very recently, we've added PATH and DUPLICATE output messages,
and those do show up in the summary:

# of expected passes            89593
# of unexpected failures        547
# of expected failures          146
# of known failures             104
# of untested testcases         20
# of unresolved testcases       5
# of unsupported tests          97
# of paths in test names        1      <<<<<
# of duplicate test names       370    <<<<<

I would think that even these two would be useful to have in
DejaGnu proper for all testsuites.

v1/rationale     - 
v3/final version - 

Pedro Alves

reply via email to

[Prev in Thread] Current Thread [Next in Thread]