RE: [Axiom-developer] Literate documentation

[Bill Page]

BTW. I am omitting Ed Ream's email address for now since his
inclusion in the Cc was uninvited and he has not replied so
far. I do not wish to appear to be "spamming" him with this
debate. I can always communicate the results to him later if
it seems relevant.

On May 19, 2007 8:35 AM Ralf Hemmecke wrote:
> ... 
> OK. But I am not convinced anymore that switching to LEO
> is such a big step forward. If the Axiom code ends up by
> something that looks like LeoPy.leo and this is understood
> as a literate program, then I would say I don't understand
> the code and the ideas behind it.
>

I did not say that Leo is a big step forward. What I said was
that it was an alternative to the monolithic book (Knuth)
style of literate programming. I personally would have not
problem at all if Axiom code looked something like this and
I certainly would consider it much more literate than Axiom
is now.

> LeoPy.leo is *not* a literate program, rather it is a program
> whose reading order is perhaps a bit different than the
> compilation order, but otherwise I miss a lot that drives
> people to write understandable code.

I *strongly* disagree. What do you think "Diary", "Notes",
"To do" and "(Project Views)" have to do with a conventional
non-literate program?

"Diary" is essentially a detailed change log. Axiom has a file
devoted to this (oddly perhaps Axiom's ChangeLog is not a
pamphlet format file). Leo's Diary nodes are considerably more
detailed than Axiom's change log.

"Notes" includes sub-nodes like: "About packages", "Announcements",
"Distribution checklist", "Leo and CVS", etc.

"To do" includes a section on "Known Bugs" and plans for "Later
versions" of Leo.

"(Project Views)" shows views of the outline related to specific
projects or tasks. It also contains some new or experimental
code that is not part of the current version of Leo.

All of these nodes contain extensive documentation and often
references active code in Leo via the clone mechanism.

Finally there is the "Code" section that contains the actual
source code of Leo. It begins with a section heading "Overview
of the code", there is also a section titled "Customizing Leo",
etc.

As far as I can see everything "that drives people to write
understandable code" is here, and more!

> And understandable code is, to a great deal, independent of
> the underlying programming language.

Yes of course.
 
> In fact, a literate program should make it very easy to
> translate a program written in one language into another
> language.

At least possible, yes but "very easy" might be an exaggeration.
I don't see anything missing in LeoPy.leo in this regard.

> 
> Maybe that is not the only guiding sentence behind LP, but
> when one documents and thinks about that, then one probably
> adds more documentation explaining the tricks needed for the
> particular language and also the goal that this or that piece
> of code actually serves. In the end a literate program might
> survive even a change in the underlying programming language.
> 

Ok. Actually as I understand it the first version of Leo was
actually written in C++ and later re-written in Python.

> ... 
> > Anyway I am thinking that we should start with something like
> > this:
> > 
> >   1) It should be possible to import a noweb file into and run
> >      Leo's weave command to produce an output file identical
> >      to that produced by applying noweave to the original file.
> > 
> >   2) After adding the necessary @root attribute, it should be
> >      possible to run Leo's tangle command to produce an output
> >      file identical to that produced by applying notangle to
> >      the original file.
> > 
> >   3) After importing a noweb file into Leo, saving it in Leo
> >      format and after opening it at a later time it should be
> >      possible to edit the file in Leo and then export the file
> >      to noweb. From the exported noweb file it should be
> >      possible to generate the same weave and tangle outputs
> >      using noweb noweave and notangle.
> > 
> > In other words, this would make Leo a kind of interactive
> > replacement for noweb.
> 
> My way of connecting noweb with something like LEO are a bit 
> different. I don't want to see myself exporting from .leo to
> .nw then call noweave+latex to get .dvi. (The tangle part can
> probably be done from within LEO.)

I did not mean to give the impression that this would be the
normal way of interacting with Leo. What I was trying to do
was to set the standard for what noweb functionality should be
available in Leo by referring directly to equivalent operations
in noweb. Once Leo has this functionality importing and exporting
to and from noweb files becomes irrelevant.

> Rather I would like to have leo work directly on the .nw files.
> A .leo file or rather a kind of index file would keep meta
> information about the outline. Let that have the extension .meta.

What you are editing in Leo *is* the contents of the noweb file.
There is no need to retain the .nw file anymore. If for some
reason you wanted it, you could export the outline (or part of
an outline) to that form.

In other respects, Leo can already operate in an mode in which
generated files are automatically updated when changes are
made in Leo (and vice versa).

> 
> Forgetting for a moment that a .nw file contains LaTeX, then
> there are
> 
> <<blah>>=
> 
> lines that start a code chunk and
> 
> @
> 
> or
> 
> @ %def ...
> 
> lines that start a documentation chunk. If I am not completely
> wrong, that is enough information to split the file into nodes
> where each node contains the documentation that comes right
> before it. (Maybe there is need for a little thought, since
> there might be code chunks that _continue_ previous code
> chunks.)

I think you are right. In fact I think that is more or less
exactly what import from noweb tries to do right now.

> If LEO reads a .meta file, then if finds there lots of 
> references to .nw files together with a precomputed outline
> from the previous save time of the file. LEO checks that the
> outline is not in conflict with the .nw files and reads the
> nodes into its memory. Then work is done as usual in leo.
> Saving any modification, saves the .mete file and all the
> various .nw files. (If the outline changed, then that might
> result in reordering the chunks inside the .nw file.---Maybe
> there should be a special nwfile-outline.)

I don't understand your invention of a ".meta" file. As far as
I can see what you describe is already implemented in Leo but
with somewhat different terminology and omitting the .nw files.

> 
> Now if the .meta file is lost or if someone makes drastic 
> changes to .nw files without using LEO, then most of or
> even all the cloned information is lost. Well, that is sad,
> but one has just lost the views on the code from different
> perspectives.

Again I don't understand .meta files, but the scenario that you
suggest is perhaps possible in Leo. If somehow you were to "lose"
the .leo file but still had all of the associated generated files
it seems clear that what you might not be able to reconstruct from
these generated files might be missing some of the information
which was not presented in any of the generated forms. If Leo
could be extended to support both weave and unweave in the manner
that I suggested in my previous email then I think the missing
information would be very small.

> 
> Before I started to look into LEO, I thought something like
> this scenario can be achieved with LEO. That would probably
> make most of the current Axiom developers happy. Everyone
> can work on .nw (.pamphlet) files as before, and there are
> people who could maintain different perspectives on the
> code+doc base.

It might be possible to build this sort of thing directly on
top of noweb but that is not what Leo does. Still, if we were
to adopt Leo it would be possible for most developers to
effectively continue to work the way they are now since the
contents of the noweb files would be in Leo. Of course there
may always be people like Tim Daly who proclaim that they are
"primitivist" who might reject anything more advanced than a
keypunch machine, but at some point Axiom has to begin to adopt
a 21st century approach if it is to have any longevity at all.
So I don't think this is really relevant.

> 
> Basically, I see LEO as a help to organise the code from a 
> higher level, but I don't want it to mix with the pamphlet
> / noweb structure.

I think trying to make pamphlet files the focus of this sort
of meta tool is wrong. It prevents one from creating the kind
of flexible outline-oriented integrated development environment
that Leo intends to provide.

> 
> > But then things get a little more complicated and interesting.
> > Of course Leo has some features that go beyond that supported
> > by the noweb file format
> 
> What exact features do you like most.
>

1) The detailed outline capability right down to the chunk
level makes the linear monolithic noweb file format obsolete and
allows one to integrate the development of new code together with
day-to-day notes and project plans as they evolve. This is
possible because of node cloning. (As illustrated by LeoPy.leo.)

2) Making it possible to work equally well from the generated
files that are actually used during the test-debug cycle seems
very desirable to me. I think applying the same principle to
the generated documentation (weave) would also make sense.
 
> [snip]
> 
> > Because of Leo's untangle command, it is possible for
> > programmers to edit generated source files and have Leo
> > automatically update the associated Leo files from which
> > they are (normally) generated.
> 
> I wonder if we should make "untangle" a desirable feature. 
> It basically means that you update code and forget about the
> documentation. I tend to say, that hasn't my support.

Of course it is possible to do that now. In fact if you look at
the changes committed to both build-improvements and wh-sandbox,
it is often the case that updating the documentation part of the
pamphlet files lags considerably behind the changes made in the
code chunks. In this case I think the tools should adapt to the
way people really work rather than trying to enforce a specific
discipline by deliberately making certain things harder to do.

> 
> My experience with developing code for Aldor-Combinat with 
> the help of ALLPROSE is that I never felt the need to edit
> the generated .as files  directly. One can easily find oneself
> around in the corresponding .as.nw file. Having need to look
> at code files directly just says that you haven't changed your
> view to programming.

I think there are many reasons why one might want to work
directly with the generated files. For example the error messages
from most compilers (even Aldor) refer to line and character
positions in the code files, not the files (or system) from
which they were generated. Most editing tools (e.g. emacs)
directly support the edit-test-debug cycle this way. Why not
make this as easy and fast as possible. It is usually not
necessary to re-build all the source files with each change.

> 
> With LP-aware debuggers I don't think that the .as or any
> other generated code file is anything else than some generated 
> thing. You can generate any additional information into such
> a file to help a debugger or any other tool to find its way
> back into the actual .as.nw source, but I don't want to look
> at .as files. That is as uninteresting (and only rarely needed)
> as the .c file that can be generated from Aldor sources.
> 

That doesn't make any sense to me. First, I do not know of any
such "LP-aware debuggers" and even if I did, I am not sure that
that would be the only way in which I would test and debug a
program. Second the comparison with .c files generated by Aldor
and GCL is wrong. The code that is being debugged is exactly
the same code that is present in the .nw file (or in the Leo
outline). When one is working intimately with the code to fix
a problem or to implement some new function this code is *all*
that the programming is focusing on. Everything else is retained
in her immediate memory. As a programmer I am quite sure that
this intimate relationship to the code is essential to that
rather mystical sense of "flow" that one feels when things are
going well in a programming session. It makes good sense to
me not to interfere with this but rather let the system take
automatically care of capturing the results.

> > Maybe this would be the best of both worlds.
> 
> Bill, can you state a few items why exactly you think LEO 
> would help Axiom?
> 

I am proposing Leo because it is available now, off-the-shelf
so as to speak. I think it is possible that using outlines in
Leo would help potential new developers understand more quickly
the parts of the code that are of immediate interest to them.
In saying this I am focusing attention on those developers who
want to get into the internals of Axiom.

But Leo outlines can also capture in exact detail the structure
of the Axiom library that is currently displayed only by a tool
like hypedoc. So it seems to me that they also make a perfect
match for someone wishing to write new library code. One of
the things that makes writing good Spad code in Axiom rather
difficult is first of all getting a good understanding of the
detailed structure of the appropriate parts of the existing
library. So I kind of thing of Leo as providing hyperdoc
functionality right in the development environment itself.

> ... 
> But I don't want my texts be stored inside .nw and inside
> .leo. There will be a point when nobody knows anymore what
> the actual sources are.

I just heard another Axiom developer say: "I don't want the boot
source code chunkified and made part of the documentation".
I think you are taking the same point of view.

But really I think most of us would agree that we want the
"actual" source code to be at as high a level as possible.
This seems to be implicit in what one means by literate
programming in the most inclusive sense. So if Leo becomes
the tool to support literate programming in Axiom then obviously
the .leo files are the sources (of both documentation and code).

> ...
> > 
> >  maybe the issue is deeper. I think Waldek said it best in
> > a recent email when he coined the new verb: to "chunkify".
> > He used this is a somewhat derogatory manner, suggesting
> > that to re-arrange code into chunks in order to produce a
> > pamphlet (noweb) style documentation of the Shoe (new boot)
> > compiler would be very undesirable. Clearly he would prefer
> > (at least in this case) that the code and the documentation
> > remain separate. I am sure that this view is driven at least
> > in part by reluctance to be reading and editing a cumbersome
> > and large pamphlet file every time one wants to make a small
> > change to the source code. Maybe Leo's untangle and unweave
> > features would help prevent this sort of fear of using this
> > literate format.
> 
> I can understand Waldek. Unfortunately, I have never read
> code in wh-sandbox, so I actually cannot judge.

??? wh-sandbox is just a branch of build-improvements which is
a branch of (old) Axiom silver. The style of the code and
documentation remains (largely) the same.

> But suppose Waldek likes having documentation here and code
> there. That does not mean that in the future somebody else
> could rearrange that to make the code even more understandable.
> Everyone has his coding style and for Axiom it is currently
> probably the best to let everyone work as they please. We need 
> something working, eventually. And if Waldek does this and puts
> the documentation different from where I would put it, I don't
> care. I cannot do Waldek's work so I am rather satisfied that
> Waldek writes documentation at all even if it lived in another
> place.

The thing I am MOST interested in is *collaboration*. I want it
to be possible for many people to work together effectively to
complete a complex task. And maintaining and improving Axiom is
certainly a complex task. I have serious doubts that it will be
possible to make substantial improvements in Axiom unless we can
find some way to solve this problem. I believe that Axiom is
currently constrained largely by the limitations of normal
individual human beings and the fact that it has gotten as far
as it has is at least in part because the people involved in
the project have often been somewhat extraordinary. But it is
not a good strategy for us to simply wait for such people to
come along and hope they become interested in Axiom. We need
proper (21st century) tools that permit the abilities of several
people to be effectively combined. 

> 
> All my LP efforts currently mainly concern the Algebra code.
> 
> >> My effort in documenting my own code is probably not really
> >> literal programming. The best thing I've seen up to now is
> >> Cliff's description of cl-web.
> 
> > I agree that would Cliff wrote is a good example. One trouble
> > (and this has always been a problem with literate programming)
> > is that most programmers are poor or even terrible technical
> > writers.
> 
> Hmmm, I really think that this is very much due to the fact
> that LP is not normally thought in university courses. Everyone
> wants to have code quickly. Nobody pays for good documentation.
> If a customer gets a sparsely documented product, one can get
> even more money out of that customer since somebody has to be
> paid for the maintenance (which might even be another company).

I disagree. I am convinced that writing prose and programming
a computer are two very different human capabilities that depend
on aptitudes which can not be easily taught. That is the reason
why many computer and other high tech companies specifically
employ technical writers to work directly with their programmers
and technical staff.

> 
> > Thus by it's very nature I think literate programming
> > has to normally be a collaborative effort between the highly
> > technical and specialized programmers and other people who
> > are willing to contribute to the end result via their writing
> > skills.
> 
> Let's see how this works out in our Axiom experiment. ;-)
> 

Well, this experiment is nearly 5 year old and I am getting a
little impatient and a little older myself. :-) I very much agree
with Martin Rubey who commented in a related thread that he was
very disappointed that so little effort (so far) has been directed
to writing literate documents for Axiom library code. If Axiom is
going to survive, I think we have to do better and we have to do
it soon.

Regards,
Bill Page.