Re: [Axiom-developer] Re: Test summary

[root]

Steve,

>Is this list likely to change any time in the future?
>
>If so, perhaps there is a simple way to include it in the
>axiom--main--1 sources, or on Savanna, so that developers on alternate
>branches have a reliable reference.

At IBM I incorporated automated regression testing into the makefiles.
I've yet to recover that facility but the "Daly Bug" message is part
of it. Some of the files have deliberate errors in them (demonstrating
"correct" failure cases) so the )set message test option allows the
regression software to distinguish the failures. (The "Daly Bug"
message was so named because I needed a unique string and since I
was the only one who would ever see the test results I used my name.
Apparently once IBM sold it NAG never changed it).

If you check out the src/input/Makefile.dvi you'll see that there are
other cases that do not run. In particular, the graphics test cases
do not yet run and I'm working on getting them running as I write this.
Running these cases requires sman to work properly and that only 
came into the build a few weeks ago.

Once the regression testing works we'll have a baseline of failures.
If you look at the src/input/bugs.input.pamphlet file you'll see
that I used to keep track of reported bugs, with failing examples,
as well as fixes. These were kept around to ensure that they continued
to work and that reported bug test cases were not lost.

So, the answer to your question is that, yes, we can do that and it
should happen in the near future but in a more generally useful way.
I will add both failure lists (unix and windows) to the README file
on savannah at the next push.


Also of note is that I've collected the test suites from other computer
algebra systems and plan to apply them to axiom. However I'd like to 
generalize the test suites so that they can be used by many systems
in addition to axiom. The long term win here is that we can all benefit
from a "standardized test suite" where the mathematics and the known
answers are supplied as well as the test cases. I've christened this
testing effort CATS (Computer Algebra Test Suite). In the long term
I want axiom to run all the standardized tests in CATS. The CATS
branch was on the previous arch server but I have yet to recreate
that branch here. If there is any interest in developing better
testing I'll do it now.

The challenge, of course, is to write the mathematics behind the
tests into pamphlet files so the tests are properly documented
and can be used with some confidence that the answer is correct.

Tim