[Axiom-developer] Re: Pamphlets and LaTex

[Stephen Wilson]

*,

A few items in this exchange were lost to axiom-devel due to an
omission in the CC field.  Am forwarding what I think are the missed
messages.


Stephen Wilson <address@hidden> writes:

> Tim,
> 
> Thanks for your detailed reply.  Im glad this discussion is happening
> as its making many things clear to me.
> 
> address@hidden writes:
> > > I have been trying to get a clear picture of the literate future for
> > > Axiom.
> > 
> > My plan, and I emphasize the "my" because I appear to be alone in this,
> > is to document axiom along the lines of the Lisp in Small Pieces book.
> > 
> > My plan shows 10 volumes,
> >  1) A tutorial -- written an published, needs rework
> >  2) Users Guide -- the Jenks book, redone with a different focus
> >  3) Programmers Guide -- how to write spad 
> >  4) Developers Notes -- how to understand the internals, etc
> >  5) The interpreter
> >  6) The spad compiler
> >  7) Hyperdoc
> >  8) Graphics
> >  9) Sman
> > 10) The algebra -- a multivolume, ongoing work
> 
> The idea of organizing axiom into a set of volumes makes perfect sense
> to me.  One needs a hierarchical structure to work with, period.  It
> should be reflected in the documentation as `volumes', the the source
> tree as `directories', etc.  I dont think your alone in that kind of
> goal.
> 
> [...]
> > > The basic issue for me is loosing the weave step. I believe that a
> > > lisp based noweb replacement could make good use of such a stage.
> > > Just like how a lisp based replacement could make use of the tangle
> > > stage (by sorting the chunks and compiling the lisp/boot/spad/code on
> > > the fly, for example).
> > 
> > Actually I have a lisp-based weave in the works. It just isn't ready
> > for publishing. It's pretty simple in concept, using the same kind
> > of scanning the tangle step uses. 
> > 
> > All you need to do is associate a lisp function with the latex
> > command. Thus 
> > 
> > \spadcommand{a+b=c} calls (spadcommand ("a+b=c")...)
> 
> This is where you loose me.  I _like_ the idea.  Its exactly the kind
> of thing I want to see.  But then the pamphlet is no longer `pure
> LaTeX', it just looks that way.
> 
> For example, we could have @(spadcommand "a+b=c").
> 
> If we need a weave stage, the file is not `pure' latex.  Using the "@
> character in the first column position" as a convention for the format
> of a _pamphlet_ file makes sense to me.  Stepping on the equivalent
> Latex construct `\' would lead to confusion, IMHO.
> 
> > The lisp function gets the latex command arguments.
> > A simple table lookup. How hard is that? If we wanted to be fancy
> > in an ANSI lisp we could use clos-like handling.
> >
> > We need this in cases like the jenks books where there are 
> > \spadcommand{some axom command}
> > and we want to process them dynamically and insert the results
> > into the final document.
> 
> Absolutely.
> 
> > But I see the weave step being used for things like Axiom-internal
> > documentation, not for things like literate papers. So you would
> > use weave 
> >   -- to create hyperdoc pages, 
> >   -- to merge pamphlets into booklets,
> >   -- to include/exclude text blocks, etc.
> > 
> > If you'll notice I've been trying to document some of the new input
> > files (see chtheorem.input, just published) in a way that will allow
> > them to become "example sections" in a tutorial book and in the
> > algebra volume. The new weave would process these documents to combine
> > them automatically. Think about this for a second... it means that the
> > documentation as well as the code becomes "chunks". This is
> > potentially a paradigm-changing difference. This chunking can be done
> > without doing special-case syntax.
> 
> noweb effectively deals with its input in this way, its just so simple
> there is nothing paradigm-changing about it.  Noweb does not delimit
> code chunks, it provides a code block header and documentation block
> header.  So chunks can follow chunks and `@'s can follow `@'s.
> 
> I wrote a noweb style processor in lisp over the weekend that respects
> this `chunking' of documents.  Its one of the reasons I wrote about
> this issue, as I too believe that it is a potentially powerful
> approach.
> 
> 
> > But this new weave use is internal-documentation only. 
> 
> Agreed. `@' should be like `\par' or some near equivalent for pure
> documentation.
> 
> > I want to be able to send and receive standard latex files that 
> > contain axiom code and process them. That requires using standard
> > latex syntax.
> 
> But your not sending and receiving standard LateX, as far as I
> understand.  What is \spadcommand{a+b=c} going to do when typeset?
> How do you write a document which uses these kinds of constructions
> yet cant make use of them without a weave step?
> 
> > You can't expect the world to adopt noweb. We can't even get the
> > developers to use it.
> 
> Its not so much about convincing the world to use noweb.  We need
> something better but nowebs simplicity serves as a good starting
> point.
> 
> What we have a hard time with is getting people excited about literate
> programming.  We are going to have an even harder time if we go the
> LaTeX route because even developers familiar with literate programming
> are going to have a hard time making the connections. I believe we
> should build off of familiar concepts if they prove sufficient.
> 
> > > In noweb, we have the basic syntax that <<chunk>>= starts a block of 
> > > code, and @ starts a block of documentation.
> > 
> > Yep. except that in NATIVE noweb << needs to be escaped (try writing
> > shift code in C which is where it bites me). Axiom's noweb is patched
> > to skip the escape, another rejected patch, another source of contention.
> 
> Right.  It would bite aldor code too and possibly spad code in the
> future. I like CWEBS @< ... @> convention more myself. Not so much
> room for conflict.
> 
> > Code IS documentation, it just needs special processing. Just like
> > quotes. Just like equations. Just like figures. Just like hyperlinks.
> > None of those other ones require non-latex preprocessors. And they
> > all work magically and correctly when made into pdf files, web pages,
> > journal files, books, etc. 
> 
> Absolutely, no disagreement there.  However, on the other side,
> pamphlets define Axiom, a program, not `just' a set of book volumes.
> We need the pamphlet format expressive to define a massively complex
> system, not just a massively extensive piece of literature.  Latex is
> not the tool for this (of course you know this already, I just felt
> the point needed to be made).
> 
> > > On the documentation side, I think having @ as an escape character is
> > > very important for long term goals. It allows us to embed commands
> > > and information at a meta level, without needing to rely on TeX macros
> > > to communicate all the information we need somehow. Similar to
> > > CWEB. Of course, we could scan the latex for individual items, but
> > > latex is complex and it would be a very difficult task, I think, to
> > > write a robust tool which copes with all the details well.
> > 
> > Actually, it gets in the way. It breaks things like latex-mode in
> > emacs or latex processing tools like lyx. It forces special handling
> > of otherwise-standard latex files.
> 
> How does \spadcommand differ in the special handling case?
> 
> Breaking latex-mode is fine.  I really dont think all files in Axiom
> should have the extension ".tex", ".pamphlet" is what we need, and a
> pamphlet.el mode to go along with it.  
> 
> 
> > Exactly how is latex complex? If the weave processor recognizes a
> > latex command and calls a lisp function why is that complex?
> 
> Because its not latex.  Take the \begin{chunk}{foo} ... \end{chunk}
> example.  You want to reference chunks with the sequence \chunk{foo}.
> I am _not_ a latex expert by any stretch, so this might be wrong in
> some details, but:
> 
>       {\chunk{foo} ...}   ==>   \begin{chunk}{foo} ... \end{chunk}
> 
> as far as latex handling of environments is concerned.  _Already_
> there is a confusion between what is latex and what is _not_ latex.
> 
> 
> > > So, the simple noweb syntax we can extend for our own purposes and we
> > > can manupulate the information contained therein directly using Axiom.
> > > I think there is some real potential here which would be awkward to
> > > achieve with latex-only pamphlets.
> > 
> > This is a standard argument, that noweb can by extended. Yes, it can.
> > 
> > In fact, Bill Page wrote a version of my special escape [[, << patch
> > in "standard noweb", that is, using perl. Of course, he didn't get the
> > "standard noweb patch" accepted upstream either. So both are "axiom-only"
> > extensions.
> 
> Sure.  I think Noweb is a good starting point.  As I said, I already
> reimplemented it in lisp.  I dont like the <<chunk>>= thing and would
> like it to go.  What I do like is the fact that an `@' character can
> serve as a reasonable escape, as reasonable a choice there is I think,
> allowing for unambiguous statements to be introduced for the purposes
> of weaving the document.
> 
> > But Bill makes the argument that we could use "standard noweb" and
> > have our own extension. How this differs from using the patched
> > version is unclear to me in any practical sense since I cannot ship
> > you a pamphlet file using "axiom noweb" and expect it to work for you.
> > Unless you are root and have perl installed you can't use it. Windows
> > users rarely have perl installed.  Nobody expects to install a perl
> > function to read a document.
> > 
> > However, latex-only pamphlets work everywhere. People expect and
> > understand style files so needing axiom.sty is normal. I have to do
> > this when I review conference papers that use a special style file.
> 
> Well, I have my doubts about it working everywhere without a special
> weave program to go along with it.
> 
> Of course, one could weave a latex file and share that.
> 
> > > In the latex only approach, I cannot find any gains in flexibility.
> > 
> > The new latex chunk mechanism allows latex format commands within
> > the chunk if desired. Thus if you want to embed equations in your
> > axiom ++ function comments it just works. If you want to use hyperlinks
> > in your code it just works. If you want to use \spadcommands they
> > also work.
> 
> We can do that anyways.  This is not a gain in flexibility.  Noweb is
> inflexible,  and so is `pure' latex.  Pamphlets (I use the term
> somewhat provocatively:) are not inflexible.  We just need a tool to
> process them.
> 
> > > In short I am advocating we stick with noweb syntax with a view
> > > towards extending the capabilities as demanded by future needs.
> > 
> > > Of course, I am still open to being convinced otherwise.
> > 
> > Noweb is "ok". I use it. But it has shortcomings. 
> > Have you tried to use it to document your work? 
> > Use it on a daily basis and I think you'll be convinced otherwise.
> 
> Noweb sucks (for my purposes).  Im going to keep extending my lisp
> tool to get the features I need.
> 
> 
> > Now for some of my motivations to get rid of noweb:
> > 
> > The ESCAPE issue
> 
> OK. Lets solve this with our own, smarter, tool. 
> 
> > The TIME issue
> > 
> > It is not obvious to everyone but noweb is slow. Very slow. 
> > My current pamphlet for work is 217905 lines long (a small book).
> > Noweb takes approximately 5 minutes to generate 53790 lines of lisp.
> > Latex takes approximately 28 seconds to generate 5278 pages of 
> > documentation.
> > (This on a 3.4Ghz machine). Latex-only documents gains 4.5 minutes for
> > every rebuild/test cycle.
> 
> The code I wrote is not designed with efficiency as the #1 priority,
> the tangle operation on a 23000 line file with 1000 chunks takes half
> a second (and I have a slow machine).  The weave step takes a second
> longer.  Good enough for me.
> 
> 
> [...]
> 
> I was going to reply to the points you follow with but I hope the main
> argument is already clear from the above.
> 
> 
> So.  We need to have a weave step, which implies the files are not
> latex files.  Lets work with pamphlets and have latex output but one
> of many possible transformations we can perform on such an object.  We
> only gain flexibility by taking that approach.
> 
> Tim, I really think we are on the same page here, we both see the same
> kinds of possibilities.  Please reconsider the LateX only approach.
> Its but one facet of a pamphlet.
> 
> 
> Sincerley,
> Steve