bug#41824: Dejagnu's unknown proc aborts testsuite run when triggered in test-case

[Jacob Bachmeyer]

Pedro Alves wrote:
> On 6/26/20 10:53 AM, Pedro Alves wrote:
>   
> > > to repeat at the very end of a test run with a big warning about test cases 
> > > that crashed?
> > >       
> > I did not suggest to repeat anything.
> >     
>
> I misunderstood you here -- I somehow thought that that by "repeat" you meant
> re-running the testcase.
>   

No, repeating output of at least the name of the file, and possibly the 
entire error dump.

> Still, I would not suggest to repeat the error.  Let me clarify -- with this:
>
> [...]
>
> Now, you already have an indication that something went wrong
> because make existed with error.  But people may miss that.
>   

They will easily miss that -- DejaGnu exits with a failure status if any 
tests produced "surprising" results:  FAIL, XPASS, or UNRESOLVED.  The 
GDB testsuite typically has multiple FAIL results, so exiting with an 
error code is routine.

> The issue for me is that the "gdb Summary" tally did not
> mention the aborted testcase.  I would like to see something
> like this instead:
>
>  # of expected passes            208
>  # of nasty OMG FIX! tcl errors    1
>
> :-)
>
> Your UNRESOLVED is of course better than the status quo.
>   

UNRESOLVED is a step in the right direction, but I agree that more is 
needed.  We are somewhat limited by POSIX, which does not define a 
separate result type for "test case crashed" and seems to roll that into 
UNRESOLVED, although POSIX appears to require a testsuite to have a 
strictly-defined set of tests and that a run must produce results for 
exactly all of them, which requires information DejaGnu does not 
normally have about the running testsuite.  So, while UNRESOLVED is 
needed, we could also maintain a separate count of 
UNRESOLVED-due-to-crash, or simply store away the error 
message/errorCode/errorInfo tuples and emit them a second time at the 
end of the run, also easy in Tcl 8.

I am leaning towards this latter solution precisely because it would be 
loud and obnoxious, while still maintaining a professional tone.  
(Listing a count of "nasty OMG FIX! >:-( tcl errors", while certainly 
expressing the proper intent, probably should not find its way into a 
DejaGnu release, nor should an ASCII-art C'thulu reminding the user that 
broken testsuites drive programmers to madness, complete with 
randomly-chosen "your remaining sanity" score...  :-)  )  Repeating 
error dumps at the end of the run also fits well into an XML log model, 
where the collected errors could go into their own structure at the end 
of the file, without complicating the tag model used for the main test 
results.  Simple importers would still get the inserted UNRESOLVED 
result, while more thorough or specialized tools could analyze the errors.


-- Jacob