bug#41824: Dejagnu's unknown proc aborts testsuite run when triggered in test-case

[Jacob Bachmeyer]

Pedro Alves wrote:
> On 6/26/20 11:58 PM, Jacob Bachmeyer wrote:
>
>   
> > > The issue for me is that the "gdb Summary" tally did not
> > > mention the aborted testcase.  I would like to see something
> > > like this instead:
> > >
> > >  # of expected passes            208
> > >  # of nasty OMG FIX! tcl errors    1
> > >
> > > :-)
> > >
> > > Your UNRESOLVED is of course better than the status quo.
> > >   
> > >       
> > UNRESOLVED is a step in the right direction, but I agree that more is needed.  We are 
> > somewhat limited by POSIX, which does not define a separate result type for "test 
> > case crashed" and seems to roll that into UNRESOLVED, although POSIX appears to 
> > require a testsuite to have a strictly-defined set of tests and that a run must produce 
> > results for exactly all of them, which requires information DejaGnu does not normally 
> > have about the running testsuite.  So, while UNRESOLVED is needed, we could also maintain 
> > a separate count of UNRESOLVED-due-to-crash, or simply store away the error 
> > message/errorCode/errorInfo tuples and emit them a second time at the end of the run, 
> > also easy in Tcl 8.
> >     
>
> I'm not sure how much is POSIX compliance important these days.  I would hope 
> that that
> would not limit DejaGnu's development and limit its potential to be more usable.  

We already have extensions; POSIX only allowed 
PASS/FAIL/UNTESTED/UNRESOLVED/UNSUPPORTED.  [KX]{PASS,FAIL} are GNU 
extensions.

> Even
> if POSIX compliance is important, I would think that that could be put behind
> a --posixly_correct option, and/or POSIXLY_CORRECT environment variable, like 
> other
> GNU tools do:
>
>  https://www.gnu.org/prep/standards/html_node/Non_002dGNU-Standards.html
>
> Or conversely, we could have a --gnu mode, though I would say that a GNU tool
> should behave in GNU mode by default.
>   

POSIX compliance in DejaGnu depended on the testsuite being run.  Any 
results generated by the framework itself need to be in the POSIX set, 
though testsuites can use extensions, as GDB has done.  Actually adding 
more test result codes is dubious, as that will interact badly with XML 
output and efficient database processing, both of which can reasonably 
enumerate the nine result codes.  ({,K,X}{PASS,FAIL}, 
UN{TEST,RESOLV,SUPPORT}ED)

> I really don't buy that adding different result types is an issue, as
> hinted earlier.  We've actually done it in GDB, for a loooong while.  We
> have distinct XFAIL vs KFAIL output messages show up in the .sum file (though
> those aren't distinguished in the summary file, doing so would be convenient,
> hence I see that as a GDB testsuite bug).
>
> And just very recently, we've added PATH and DUPLICATE output messages,
> and those do show up in the summary:
>
> # of expected passes            89593
> # of unexpected failures        547
> # of expected failures          146
> # of known failures             104
> # of untested testcases         20
> # of unresolved testcases       5
> # of unsupported tests          97
> # of paths in test names        1      <<<<<
> # of duplicate test names       370    <<<<<
>
> I would think that even these two would be useful to have in
> DejaGnu proper for all testsuites.
>
> v1/rationale     - 
> https://sourceware.org/pipermail/gdb-patches/2020-April/167847.html
> v3/final version - 
> https://sourceware.org/pipermail/gdb-patches/2020-April/168105.html
>   

I will look at the exact details later, but this sounds like it is 
probably a monkeypatch that will stop working in some future version 
when the procedures you are overriding disappear into an internal 
namespace, so we need to be thinking about what would be needed to 
provide extensible summary line sets, just as we need some way to extend 
language support in default_target_compile without overriding the entire 
procedure.


-- Jacob