RE: [Axiom-developer] Literate documentation

[Bill Page]

Ralf,

On May 18, 2007 7:47 PM you wrote:
> ... 
> Oh, maybe I made a big mistake. By "pamphlet idea" I did *not*
> refer to the production of a monolithic 1000 pages book, but
> rather on the view that documentation is more important than
> code.

Ok. I understand that.

> 
> I know that the "clone" idea is powerful. Without clones LEO
> is just a fancy way of showing the table of contents and
> clicking there to get to the corresponding section. Plus,
> LEO can extract code from the sections. Minus, there is no
> real weave command.

Oddly enough - at least from our point of view - this is a
standard but apparently undeveloped part of Leo. Ed Ream, the
developer of Leo, says that basically he was waiting for someone
to come along (like us?) who really wanted to use both Leo and
noweb together. He has invited us to prepare an example of how
we think the interaction between Leo and noweb should work.
I find that a little surprising because it would seem that Leo
already has a large number of users. It seems strange to me
that I was only able to find one posting by another user who
seemed to have concerns similar to ours.

Anyway I am thinking that we should start with something like
this:

  1) It should be possible to import a noweb file into and run
     Leo's weave command to produce an output file identical
     to that produced by applying noweave to the original file.

  2) After adding the necessary @root attribute, it should be
     possible to run Leo's tangle command to produce an output
     file identical to that produced by applying notangle to
     the original file.

  3) After importing a noweb file into Leo, saving it in Leo
     format and after opening it at a later time it should be
     possible to edit the file in Leo and then export the file
     to noweb. From the exported noweb file it should be
     possible to generate the same weave and tange outputs
     using noweb noweave and notangle.

In other words, this would make Leo a kind of interactive
replacement for noweb.

But then things get a little more complicated and interesting.
Of course Leo has some features that go beyond that supported
by the noweb file format so it would be possible and presumably
very desirable to explore the use of these features but doing
so will make it impossible to export such a Leo file to noweb
format and retain these features. At that point we would have
to make a decision if we really wanted to maintain the Axiom
source files in Leo format.

Because of Leo's untangle command, it is possible for programmers
to edit generated source files and have Leo automatically update
the associated Leo files from which they are (normally) generated.
So if we adopted Leo format as the Axiom source format, this does
*not* mean that all changes to the source would necessarily have
to be done via Leo. But the literate documentation part of the
Leo files would presumably not be updatable in this manner. On
the other hand, if Leo's weave command were to always generate
LaTeX format with the same sort of source markers that it uses
for the tangled files, then perhaps there would be some way for
an author to edit the generated LaTeX file and expect that Leo
could automatically "unweave" these changes and update the Leo
file. Maybe this would be the best of both worlds.

If we can agree that this might be very interesting to us, then
I can try to write this up as Ed Ream suggested as a guideline
for what we hope Leo might be able to do in the (near) future.

> 
> And yes, I agree, "weave" is *not* necessary, if the
> documentation can nicely be read inside LEO. Well, and that
> is exactly, what I am missing. I would have to convince LEO
> to write out a .nw file than noweave and latex and view the
> .dvi. But what happens if I see an typo in the .dvi? 
> How do I find the place in the .leo file?

I think my suggestion above is a possible answer to that
question - it would mean that you just go ahead and use the
usual dvi - LaTeX file linkage to edit the Leo-generated LaTeX
file (being careful not to distrub any of the weave-anchor
comments in the LaTeX code that where inserted by Leo). Then
you would expect Leo just to unweave these changes back into
the associated Leo source file the next time you openit in
Leo.

> 
> For my working style, LEO just adds overhead.
>

(Maybe) it is possible to use Leo in a fairly transparent manner
via untangle and unweave. Then would would use Leo to view the
integrated code and documentation in the full outline form. And
of course edit it in an integrated manner only when it seems
desirable.
 
> And the clones, I feel that they are a good thing, that
> viewing the code and documentation from different perspectives
> is desirable. However, I did not yet feel the need for
> "clones" in my own documentation efforts. I am totally
> content with hyperlinks.

One difficulty with new tools is that it is often very difficult
to imagine how their use will ultimately affect the way you
work. I think Leo's outlines and the ability to view the same
code and documentation from different points of view (via clones)
is one of those type of things.

> 
> > In particular this was a follow-up to comments from
> > Waldek, Gaby, and me that were critical of the creation
> > of 'bookvol5.pamphlet' as literate documentation of the
> > Axiom interpreter. We all expressed the opinion that
> > 'bookvol5' was more difficult to appreciate even than
> > the original code.
> 
> Sorry, I haven't even looked at bookvol5. But if it is
> recognised as worse than the actual pure code, then this
> is not a problem of whether using LEO or not, but rather a
> problem that there is no general guideline of how to write
> good literate programs, i.e. easily understandable
> documentation.

Yes, perhaps. But maybe the issue is deeper. I think Waldek
said it best in a recent email when he coined the new verb:
to "chunkify". He used this is a somewhat derogatory manner,
suggesting that to re-arrange code into chunks in order to
produce a pamphlet (noweb) style documentation of the Shoe
(new boot) compiler would be very undesirable. Clearly he
would prefer (at least in this case) that the code and the
documentation remain separate. I am sure that this view is
drive at least in part by reluctance to be reading and editing
a cumbersome and large pamphlet file every time one wants to
make a small change to the source code. Maybe Leo's untangle
and unweave features would help prevent this sort of fear of
using this literate format.

> 
> My effort in documenting my own code is probably not really
> literal programming. The best thing I've seen up to now is
> Cliff's description of cl-web.
> 

I agree that would Cliff wrote is a good example. One trouble
(and this has always been a problem with literate programming)
is that most programmers are poor or even terrible technical
writers. Thus by it's very nature I think literate programming
has to normally be a collaborative effort between the highly
technical and specialized programmers and other people who are
willing to contribute to the end result via their writing skills.
I am hopeful that Leo might be an appropriate tool for this sort
of thing.

> In general, I think, understandable documentation results by 
> constructive criticism of other people's works (+ enough time
> to actually rewrite the documentation).

I fully agree.

> 
> > Of course this is from a "programmers point of view" which
> > might not be the same as someone new just starting to use
> > Axiom and trying to understand how it works. But I think
> > programmers still need literate documentation, the question
> > is just what form this should take. I am saying that the
> > Knuth-style "book" is not a particularly good form of
> > documentation for a programmer.
> 
> What we need is not programmer, but people who understand
> their area (mathematics, physics, etc.) and who are able to
> put their knowledge into code. A traditional programmer is
> a slave. (That is overstated.) Of course, programming is a
> creative task sometimes, but I would rather like to support
> the "paper-writing people" to aquire some programming
> knowledge.
> 

I sort of agree with this however I doubt that any of the
people who have programmed fundamental parts of Axiom ever
felt like they were "slaves" even admitting that as an over
statement. These were not traditional programmers in the
conventionally structured software engineering (if these
even exist), but rather researchers who were experimenting
with ways to implement new designs for supporting abstract
mathematics (algebra) in a computer system. I expect it was
much more like what is not called "extreme programming"

http://en.wikipedia.org/wiki/Extreme_Programming

where the software specification evolves almost as rapidly
as the program code.

Another point you are making is really that there are several
different types of programming that one can expect in a system
like Axiom. Programming can occur at several different levels.
There is programming involving the "internals" and there is
"Axiom library" programming that involves the implementation
of mathematical algorithms and techniques. Perhaps the sort
of support needed for these different types of programming
is different... but in the Axiom open source project so far
we are trying to use the same tools (i.e. noweb) for both.

> 
> >> The declared goal in Axiom is to write documentation
> >> that is readable by humans. I am not an experienced LEO
> >> user, but all I see does not convince me to use LEO in
> >> order to produce an article+code about some mathematical
> >> idea.
> 
> > But of course Leo is intended to be a tool used by humans. 
> > Tools that allow one to navigate online using a web browser
> > through svn revisions and change logs is another such tool
> > intended to be used by humans.
> 
> I don't want to throw those tools away, but what I am saying
> is just that LEO does not meet my needs. I want some powerful
> editor that helps me in coding the ideas and can extract the
> code from the text.
> 
> > The question is: given current (web) technology and current 
> > programming practices, what form should a "literate program"
> > take?
> 
> >> LEO is, in my eyes, very code focused. Am I wrong?
> 
> > I think you are right. It is apparently viewed by many 
> > people who use it as a kind of high level integrated
> > development environment (IDE) that supports literature
> > programming.
> 
> For me LEO is not enough. Let me give a wish list for my dream
> IDE (whose purpose is to write extensions of Axiom's algebra
> code):
> 
> o  Code and Documentation should belong together.
> o  The IDE should hide extra markup for the doc/code and
     code/doc borders.
> o  Wysiwyg style. (I am not quite sure about that, but it
>    would avoid the weave step. And graphics and math formulae
>    inside the documentation should not be a rare thing.)
> o  Hyperlinks already inside the editor.
> o  Semantic code hyperlinks from symbol information of
>     integrated Aldor/Spad compiler.
> o  Support of (formal) API documenation.
> o  File format is revision control friendly.
> o  Automatic dependency recognition of files and code
>    (library) compilaton.
> o  Integration of test suite generation and execution.
> o  Aldor/Spad Debugger support.
> o  ... maybe some more properties ...
> 
> As I said in one of my previous mails. I somehow want an IDE
> that is as powerful as Eclipse+TeXmacs+LEO+(Emacs).
> 

Well, that seems like an excellent "wish list" to me. Maybe we
can pass some of these ideas on to the Leo developers to see
what they thing about it. In fact I have Cc'd this message to
Ed Ream and perhaps he will have time to comment on some of this.

> I think one of my next tasks is to investigate what TeXmacs
> misses.
> 

I think the biggest thing that TeXmacs lacks from out point
of view is some form of literate programming or interface to
noweb.

Have you already looked at Lyx?

http://www.lyx.org

Lyx provides WYSIWYG editing of LaTeX-like documents, import
and export of LaTeX documents and has built-in support for
noweb.

http://www.lyx.org/about/features.php

Here is an example of using Lyx and noweb to do some of the
things we are talking about in Axiom:

http://wiki.lyx.org/LyX/LyxWithRThroughSweave

Maybe what you want is something like Leo+Lyx?

Regards,
Bill Page.