Re: [Bug-dejagnu] dejagnu-report-card (was: BUG: improper format string construction in framework.exp)

[Jacob Bachmeyer]

Ben Elliston wrote:
> On Tue, Oct 30, 2018 at 08:58:46PM -0500, Jacob Bachmeyer wrote:
>   
> > Lastly, is a soft dependency on GNU awk acceptable for DejaGnu or
> > would I need to rewrite my testsuite summarization tool in Tcl or
> > Expect before contributing it for use with the DejaGnu testsuite?
> >     
>
> I am happy for contributed tools like this to be written in GNU
> awk. Please try and use portal AWK if you can, but if not, then
> perhaps give the file a .gawk extension to make it clear that you need
> GNU awk?
>   

The AWK predecessor (written as part of a testsuite for another project 
that I intend to offer to GNU after the lowest-level basic functionality 
is ready; other parts of that testsuite already relied on GNU awk when 
this tool was written) uses both asort and asorti, so some work will be 
needed to get it into portable AWK.  Reading sum2junit.sh gave me a few 
ideas, and I have written both shell and Tcl versions of a tool that I 
am now calling "dejagnu-report-card".

An attempt to use Expect to read DejaGnu summary files had extremely 
poor performance.  The shell and Tcl versions have similar performance; 
one spends more time in the system and the other spends more time in 
user mode, but overall, they seem to have similar performance.  The AWK 
predecessor blows them both away, usually completing in about 1/2 (on 
the DejaGnu testsuite) to 1/5 (on a larger testsuite with ~2000 results) 
of the time.  (This is not noticeable on the DejaGnu testsuite, but 
~2000 unit test results can take about 200ms for the shell and Tcl 
versions to read on my system.)

Also, I would like to see this be more than another tool in contrib; I 
plan to adapt the "runtest" shell script into a "dejagnu" multi-purpose 
launcher, supporting auxiliary commands in a style similar to git(1).  
(The runtest(1) command itself would remain "for historical reasons".)  
As such, "dejagnu report-card" would be the first such command added, 
with future plans including some kind of "canned test" feature to help 
with creating new testsuites and adding tools.  Something like "dejagnu 
create testsuite" and "dejagnu add tool" would be nice.  These will need 
careful review for portability, since I only have GNU close at hand for 
testing.

With this approach, including multiple versions and selecting at 
configure time which version to install should not be a problem.  (Have 
GNU awk?  Install the .gawk version.  Have only some other AWK?  Install 
the portable .awk version.  Have no AWK?  Install the Tcl version.  The 
dejagnu(1) launcher would adapt at run-time.)

This leads to a slightly more complex "structure" for DejaGnu, with an 
"inner core" (runtest(1) and its components) and an "outer core" (the 
dejagnu(1) subcommands) that is also installed.  One of the reasons for 
this is that I would like the DejaGnu testsuite itself to use the 
dejagnu-report-card tool, and possibly for a future version of Automake 
to use it if the DEJATOOL variable indicates to test multiple tools.

Here's what the output looks like:

            __________________________________________________
           /    PASS   FAIL  ?PASS  ?FAIL  UNSUP  UNRES UNTEST
           |--------------------------------------------------
  runtest  |      77      0      0      0      0      0      0
libdejagnu  |       5      0      0      0      0      0      0
  testrun  |       0      0      0      0      0      0      0
           |--------------------------------------------------
           |--------------------------------------------------
           |      82      0      0      0      0      0      0
           \__________________________________________________


The "testrun" line will be eliminated as part of planned improvements to 
the DejaGnu testsuite.

-- Jacob

[PS:  A batch of your emails seems to have been "stuck" somewhere along 
the way for the past five days and have just now arrived in my inbox and 
at the list archives.]