[Axiom-developer] Re: Axiom source code?

[root]

David,

The Axiom source code, as delivered from NAG, is missing some parts
due to legal restrictions. The build procedures need to be reworked.
I'm in the process of reconstructing the build procedures. I'm also
rewriting the code to be "literate programs" which will be the new
format in the system. It has been a long process (I got the code in
september) and I'm only able to work on it nights, weekends and
holidays. The core of the system is mostly converted. It involves
building several layers (noweb, lisp, bootsys, depsys, interpsys,
algebra). I'm working on the last layer now. All of the code that
has been released is available on a server called tenkan.org and
can be downloaded with anonymous CVS access. The available code
will only build up to an incomplete interpsys which is the last
CVS update I did about two weeks ago.

I'm building on tenkan rather than savannah as I want to make the
initial build mistakes in private among developers. I've never run
an open source project though I've participated in a few. Once the
code will compile algebra files (I *almost* compiled the first one
at 3a.m. this morning) I'll upload the next CVS change. Once the
developers can download and build a clean copy of the system I'll
import the first version of the code into savannah and start work
there. We need to have a discussion about "task partitioning"
which will depend on people's interest and area of expertise.

What will exist at that point is a complete interpreter and all of
the algebra code. This will allow you to do algebra development.
It will not yet contain all of the other functionality like graphing,
hypertex, etc. That is the next task on my list.

If you'd like you can join the axiom-developer mailing list on
savannah. I'll set you up with an account on tenkan and you can try to
build the system.  Most of the active discusion happens on the
axiom-developer list. I'd encourage you to read the archives of
both the axiom-mail and axiom-developer lists.

Alternatively you can just do:
  cvs -d:pserver:address@hidden:/home/cvs co axiom
and you'll get all of the posted code. Once you do that
you can read the Makefile.dvi file (using xdvi) to get the
instructions. Or just type 
  make SPAD=`pwd`
which in theory should build the system from scratch and
a couple people have gotten it to work.

Please give me feedback about any failures in this process.
There is a FAQ section in the axiom/new/new/Makefile.dvi file for
simple problems others have encountered.

At the moment development has been "a one-man show" mostly for
the reason that constructing a system is very complicated. For
example, the bootsys layer is a boot-to-lisp translator but it
is written in boot and needs a bootsys layer. I'm the person 
who originally made the build procedures before Axiom became a
commercial product. There are probably only 3 people in the world
who can build Axiom from scratch and I'm trying to encode all of that
knowledge into the system. Read the Makefile.dvi file in each 
subdirectory and you'll see some of the new system documentation.

I had hoped to have the system built by the end of the year but
Christmas visits and a funeral tied up several days. However,
progress is happening and you should see a working system shortly.

Note that I update the 'homepage' on savannah to reflect changes in
the system every time I update the CVS. You'll see from that list
of tasks that there is much more going on than just programming.

As I pointed out all of the files in the system are now "literate
programs". All code exists in "pamphlet" files which are not much more
than TeX files with special tags around the code. To manipulate the
code you'll need noweb, which exists in the axiom/new/new/zips
subdirectory (or available on the web; check with google). It is built
automatically as the first stage of the system build. Noweb uses
"noweave" to extract documentation and "notangle" to extract code
source.

I urge you to actually try these instructions because it
will make sure that we share the same ability to create, use and view
pamphlets. If these instructions don't work for you let me know
because we need to make sure you've got a proper environment set up.

I've uploaded 3 examples of literate programming to illustrate styles.

An Algebra Example:

The first is called dhmatrix.pamphlet. The instructions for use are:

1)  wget http://home.earthlink.net/~jgg964/dhmatrix.pamphlet
2)  notangle dhmatrix.pamphlet >dhmatrix.spad
3)  noweave dhmatrix.pamphlet >dhmatrix.tex
4)  latex dhmatrix.tex
5)  latex dhmatrix.tex
6)  xdvi dhmatrix.dvi

step (1) will fetch the file. (wget is a useful utility for fetching
   files given by a url. how you actually get the file is up to you)
step (2) will read the pamphlet file and create a spad file.
   spad is the algebra language of Axiom. This file is extracted from
   the pamphlet file and when diff'ed against the original source
   shows no difference (an important point since we don't want to
   introduce errors by wrapping the original source files into a
   literate style).
step (3) will extract the tex output from the pamphlet file. notice
   that the dhmatrix.tex file has commands like \documentclass 
   prepended automatically by noweb. This is, in general, not what
   we want so normally we will use the -delay parameter to noweave.
   However for this particular example I am experimenting with the
   ability to collect many pamphlets into a booklet and I deliberately
   left off the \documentclass, etc. The default behavior of noweb
   works in this case.
step (4) and (5) read the dhmatrix.tex file and create dhmatrix.dvi
   We need to latex it twice to get the cross-references right. I
   don't believe there are any cross-references in this file yet but
   I do this by habit.
step (6) will let us view the dhmatrix.dvi file. You could also use
   other utilities like dvips to create a postscript file or dvipdf
   to create a pdf file. I tend to work directly from the dvi file.

The dhmatrix.pamphlet file is an example of documenting the algebra
code. The original dhmatrix.spad file (which I wrote) was taken from
Richard Paul's Ph.D. thesis which became a book (Robot Manipulators).
Richard gave me permission to quote from his thesis for documenting
the domain. There is much more work to be done but this pamphlet was
written as a first experiment.

In general I hope to search out primary sources for the algebra
that lives in Axiom and either get permission to directly quote
the relevant paper or study the paper and write a new pamphlet.
There is a large piece of work to track down the original works.

The next piece of algebra documentation is to use Barry Trager's
Ph.D. thesis to document the integration code. I have his thesis
and permission to use it for documentation purposes. There is a
long leap from the thesis to the code so I have a fair bit of
background research I need to do before I can write up the 
pamphlet. The original thesis is 85 pages.



A MAKEFILE Example:

The second file I've uploaded is actually from the sources we'll
be using. It is a Makefile in the Codemist Common Lisp (CCL) subtree.
The instructions are:

1)  wget http://home.earthlink.net/~jgg964/Makefile.pamphlet
2)  notangle -t8 Makefile.pamphlet >Makefile
3)  noweave -delay Makefile.pamphlet >Makefile.tex
4)  latex Makefile.tex
5)  latex Makefile.tex
6)  xdvi -expert -s 3 Makefile.dvi

step (1) will fetch the file.
step (2) will read the pamphlet file and create the Makefile. Notice
   the -t8 (tabs every 8 spaces) parameter to notangle. In general we
   will need this parameter and it doesn't hurt to have it if there are
   no tabs so we will use it all the time.
step (3) noweave will read the pamphlet file and create the tex output.
   Notice the -delay option to noweave. We have included the \documentstyle
   and other header information in the pamphlet file already. The -delay
   option allows us to do this. Unless the file is going to be included
   as part of a larger document (like the algebra file above) we will
   generally write our own document headers.
steps (4) and (5) are latex->dvi done twice to get reference right.
   You are certain to get a complaint about a missing noweb.sty file
   because this Makefile was ripped out of the real source tree.
   For demo purposes you can modify the line in the original pamphlet
   file from:
\usepackage{/home/axiomgnu/new/mnt/linux/bin/tex/noweb}
   to:
\usepackage{noweb}
   and rerun the command. I would ask you to resist the urge to 
   change the tex file directly as you need to think of the pamphlet
   file as the source and everything else as machine-generated files.
step (6) invokes xdvi with -expert (which eliminates the buttons as
   I know the keyboard commands) and -s 3 (which gives sufficient
   magnification so I can read it).

This makefile shows a documentation style for Makefiles in general.

Makefiles are generally very verbose in their stanzas because we will
not be doing compiles in the same directories as the source files nor
will we be depending on default stanzas to do compiles. 

We need to document the various stanzas and any special options that
we might add to the compile commands. Look for the TPD string which
I use to ifdef any changes I made to the original C sources. This
shows up as a -DTPD on the compile line for that particular stanza.

Also note that the end of the Makefile.tex contains references to
other pamphlets. The reason for these to support a future plan. We'd
like to be able to accept new code, particularly algebra code, that
we can add to the system in a reasonably automated fashion. The 
references will give us this connection. I expect to expand the
format later. These are only placeholders.

The other reason for the references is that the commands are not
actually documented in the Makefile. The Makefile only documents
build information and other special instructions. For instance,
in another Makefile (not shown here) you need to build a special
Axiom library before you build the C code. While the Makefiles
know this it would be easy for a human to overlook. Instructions
on how to use a particular command as well as documentation on
the C code that implements the command do not belong in the Makefile
but reside in the pamphlet file for the command itself.

Makefiles will also contain information about what a particular
directory contains and why it exists in the source tree.



A Non-Axiom Example:

This is an example using Java to show how to build a stand-alone
command and its Makefile

1)  wget http://home.earthlink.net/~jgg964/Magman.pamphlet
2)  notangle -t8 Magman.pamphlet >Makefile
3)  make

This illustrates a couple of useful points. First, the default
output of running notangle is the stanza marked <<*>>= in the
original source file. In Magman.pamphlet this is the Makefile.
If you read the dvi file that gets generated (or read the
generated Makefile if you must (sigh)) you will see that we
use the -R feature of notangle. The -R flag will extract particular
tags from the pamphlet file. So, if this were C code we could embed
the .h file, the .c file, and the associated Makefile into the pamphlet 
and send them as one file.

Another point is that this pamphlet file has real mathematics and
real references in it. We will need this to explain sections of Axiom.
In the long term plan we want people to submit papers to an Axiom 
Journal that includes the mathematics and executable code. That way
reviewers can test the code, readers can understand the code, the
system can import the code and maintainers can update the code.

This code is not related to Axiom but is here to illustrate the
embedded Makefile and mathematics points. If you don't have Java 
this will fail but that is not important. Note that we mix Java
C and Makefile in the same pamphlet file.

Feel free to ask questions.

Tim