[Axiom-developer] Re: [fricas-devel] Re: Project

[TimDaly]

> How does the fricas/axiom source code layout work?
> Is it all written in pamphlets that lisp is generated from?

There is a bit of a philosophical split between Axiom and
Fricas about source code layout and it is fairly fundamental.

Axiom has everything in pamphlet files and is gradually moving
to book-style layout. (As far as I know this is the only project
that is structured this way). One of Axiom's fundamental goals
is to restructure the complete source tree to be literate
programs. See the Lisp in Small Pieces or Tex, The Program books.

Fricas is a project fork and one of its fundamental goals is to
remove the literate programming structure. This is an ongoing
effort (as evidenced by the post in late March, 2008:
<http://groups.google.com/group/fricas-commit/browse_thread/thread/
ba4f3be7d257fef?hl=en>
So I really cannot comment on the non-Axiom processes.

In Axiom all of the source code lives in Knuth-style literate
documents. So the basic process is to extract the source code
from the document and then compile it. This can involve several
steps since the algebra code has its own language (Spad). Thus
algebra goes thru the steps:

extract spad from pamphlet ->
  compile code with spad compiler to lisp ->
    compile lisp code with GCL to C ->
      compile C code to machine code with GCC

> Anyway, I would love if somebody who knows what they
> are talking about regarding axiom (not me!) would
> explain what the human-written/readable code
> parts of the axiom distro are

In the near term every "source code file" is a literate document.
The idea is that you should be able to open (using a div/pdf
viewer) the file and read it like a book.

The rational for this huge change is based on experience. I
was one of the original authors of Axiom. I got my own code
back after 15 years and found it unreadable. This is despite
my best efforts at the time to write dirt-simple, clear code.
(You will encounter the same problem with Sage in the future.)

I spent some time reading and pondering this issue and eventually
discovered that Knuth had already solved it using Web technology
for literate programs. I STRONGLY encourage you to learn about
this technology because Sage is going to have the same fundamental
problems that Axiom encountered.

One problem is that the research is not associated with the source
code. A recent example is the Pollard-Rho algorithm that was part
of the Sage/Fricas discussion. It turns out that Axiom implements
Brent's algorithm which is an extension of Pollard-Rho, despite
the comments to this list. The only way to know that is to read
the code, read the literature, and "discover" what the code does.

Sage is being built by experts who are the primary source for the
material. This is excellent while it lasts but it won't last. At
some point in the future these experts will no longer be available.
If you think long term it becomes clear what this implies.

Code is guaranteed to be opaque because the world's expert in some
area (L-functions?) wrote clever, highly optimized code that does
not correspond to anything in the literature. Thus, the only person
who can properly maintain, modify, and extend the code will no
longer be available and the code will be frozen in time. My
Magnus infinite group theory project has this problem.

New people coming onto the project cannot find the literature
that corresponds to the actual code because it does not exist.
Most published results are 5 page conference papers that just
hint at the non-core, but vitally important, details if they
mention them at all. This problem has 2 parts, both of which
stem from the lack of focus on the new discipline of computational
mathematics. Part 1 is that journals and conferences only want
short papers, which are adequate for math, not the complete
implementation. Part 2 is that there is no place to describe the
actual code-level optimizations that make the implementation
practical and fast.

I believe that the Knuth's literate programming technology will
solve the long term problem and keep the code as a living,
understandable, maintainable, and extensible object. It directly
addresses the problem of bringing the research and the code together.
It gives a place to explain both the algorithm and the current
code optimizations. Further, I've been working with Carlo Traverso
(Univ. of Pisa, Math) to create a journal of these literate
programs so the algorithms and their implementations can be made
widely available for study and improvement.

This is a controversial position and led to a philosophical
split between Axiom and Fricas. Fricas has the stated goal of
competing with Mathematica. Axiom plans to change the game so
Mathematica is irrelevant.

> and roughly how big each is, in some sense.

Ah, size. Here is where Axiom parts company with most of the
other systems. Axiom is highly structured using categories
and domains. It is trivially easy to construct a domain which
overrides the categorical definition of a single function
but uses all of the rest by inheritance. Thus, "line of code"
have nothing to do with "conceptual size".

That said, Axiom is still huge by any measure. It represents
about 30 years and 300 man-years of research. It covers a
wide range of computational mathematics.



> Or just point me to an article or wiki page
> about this.  And who are some of the Axiom original
> authors?  Some files have headers like:
>
> ++ Author: Grabmeier, Gschnitzer, Williamson
> ++ Date Created: 1987
> ++ Date Last Updated: July 1990
>
> I wonder who those guys were...?

Ummm, who these guy *ARE*.

There is a very active community based around the ISSAC
(International Symposium on Scientific and Algebraic
Computation) and ECCAD (East Coast Computer Algebra Days)
which is attended by almost everyone in the computational
mathematics community. It has a long history. There are
a large number of people still active from the early Axiom
days. You can see a list of the some of the people by
starting Axiom and typing )credits. Sage may be new but
we've been at this for a long time :-)

Sage should definitely concentrate on presenting at ISSAC.
You could probably start a Sage paper track. Since I know
most of the people involved I'd be willing to help.




> There are rumblings but *definitely* no specific plans to remove
> lisp or maxima.

Lisp has already solved a lot of the problems Python has yet to
face. But since that discussion is a prelude to a language war
I'll end it here.

>
> That might sound weird, since why would an Axiom guy want to
> help with working on or improving Maxima?

While both are coded in Lisp they are philosophically wildly
different. Axiom algebra is in the spad language, not Lisp.
Thus "fixing maxima" (which works fine, btw) would be like
working in a completely separate language. I know James
(the Maxima Lead) and we're not in any kind of competition.

>    But that's the
> ** Sage WAY ** which is to get all open source software packages
> to work better.  I think any and all competition/fighting between
> open source math software is completely counterproductive.
> The goal we should all have is to provide a viable free open
> source alternative to Maple, Mathematica, Matlab, and Magma.

While I agree with "free" and "open source" (given that I've spent
the last 7 years doing that with Axiom) I have to question the
"viable" point. I think "viable" needs to be discussed within
Sage because it is vital.

If you looked at the list of credits from Axiom mentioned above
you'll notice that, of the 140+ contributors, some of them are dead.
I just attended a funeral for one in January.

Sage is too new to think about this problem but what happens when
your expertise "disappears"? What happens when you are the only
use of an open source program (such as Magnus, an infinite group
theory package)? Suppose the maintainer quits or dies? What will
Sage do (WWSD)? Do you just drop the package? Do you let it rot?
Will I find that my notebooks won't work in the next release?
If Magnus fails to compile under C++0X do you just drop it?
Do you try to recode it in Python? Do you know enough infinite
group theory to recode it? Do you know anyone willing to recode
it? Do you even know what algorithms it has?

I would hope to convince the Sage team that THE most important
contribution they can make is to provide a firm base for future
computational mathematics. Think of the science and the long
term. Insist that people document their code in a literate form
so that others can maintain and extend the code. Your REAL
issue is not the "4Ms". They are going to die because all
companies die and they are going to disappear because dying
companies take their code with them (witness Symbolics and
Macsyma). The REAL issue is the science of computational
mathematics.

Open source suffers from "project rot". Lead developers and
maintainers move on to other things and the projects die.
(Witness "Pinger", which was my open source implementation of
a closed source network management tool for Worldcom, another
company that "couldn't disappear" but did.)


> because Maxima (via pexpect) is too slow and,
> much more importantly, it is too hard for developers to improve
> and change.  This difficulty for developers has stopped dead
> promising Sage development projects related to symbolic
> calculus computation.   This symbolic manipulation rewrite
> is actually well on its way; Gary Furnish has been working on
> it day and night for a few months now, and has been funded
> by the NSF and Google (thanks!) to work on it full time at UW
> all summer.

Gee, wouldn't it be great if Gary could just read the literate
documentation of a system like Maxima and know the details of
how to implement the algorithm? This IS the late 90s and we can
construct real documentation of real computational mathematics
if we want so that the science as a whole benefits. I hope that
Gary generates literate documents so I can re-implement his
algorithms in Axiom.