Re: [Axiom-developer] Heidegger, literate programming, and communication

[u1204]

Gregg and Gary,

I understand where you are coming from. Indeed, Maturana [0] is on your
side of the debate. Since even the philosophers can't agree, I doubt we
will find a common ground. 

Unfortunately, I've decided to take on the task of documenting the
Clojure internals because, yaknow, *I* don't feel I understand something
until I know what the hardware does; consider this a flaw in my
personality :-) I have to say that the posted Clojure code is
"somewhat lacking" in the communication department. Perhaps it is only
intended for an "audience of one", and I'm not "the one". :-)

Contrary to popular belief, I am reading the code. As a result, I have a
strongly held opinion that there is a lot that could be done to make
it less of a struggle.




>> From my point of view there are at least a few things that seem clear:
>>
>> 1. I think that Gregg Reynolds and I agree on a lot, but I would add to
>> his remarks that there is almost always a human audience for source code,
>> as well as the compiler/interpreter.  Sometimes, the audience is just the
>> originally programmer, perhaps at a later date.  (If I missed something,
>> Gregg, sorry, but I don't think you disagree, anyway.)
>>
>
>I agree; whoever writes the code automatically forms an "audience" of one.
>I guess I would say "reader/responder".

Hmmm. Common Lisp is about 25 years old. Assume Clojure lives that long.
What are the odds that the original authors will be maintaining the
code? Will the code still be "an audience of one"? Are you sure that's
a worthwhile goal?



>> 2. Since Clojure is a general-purpose tool, Clojure source code has no
>> single kind of human audience for the code.
>>
>> In general, I do different things with comments, or with my coding style,
>> depending on whether I expect that I will be the only maintainer of the
>> code, or expect that others with whom I'm collaborating will need to
>> interface with it, for example.  Further, I think about the skill levels
>> and background of those who will read the code.  And I think about what
>> they would want to do with it.  And then I make decisions that involve
>> tradeoffs between competing desiderata.
>

My experience "in industry" with "general-purpose tool" code is that
code does look a lot different from project to project and language to
language. But as code moved out of its childhood and off the desk, it
began to grow hair and get ugly. The authors all assumed they would be
"the only maintainer". For instance,

   I once had to maintain a C program that had 14 pages of nested ifdefs
   just to choose the correct #include files. Each include file had
   ifdefs.  The code ran everywhere, Intel, ARM, 68000s, SUNs, DEC,
   etc. but nearly every line was based on experience (e.g. compensating
   for floating-point errors on various platforms, hacking around errors
   in various C compilers and their "optimizers", etc.) with ifdefs
   around each hack. I had to run the compiler intermediate stage to
   figure out what the actual code would be for my platform. And then I
   had to reverse-engineer the fix into the appropriate include files;
   uncommented I might add. I wouldn't want to ruin the style.

   Sophisticated Lisp programmers use macros A LOT. Axiom, for instance,
   compiles code from a high-level algebra language, essentially a DSL,
   into macros that index into vectors for the function to call, or the
   category to inherit which might contain the call, and the environment
   passed to each function is a huge vector. DSLs, which make the top
   level code "so clear", often are macros generating machine-code-like
   lisp doing super-efficient vector indexing. One finds macros
   expanding into macros expanding into macros. Quick, what does the
   spadcall macro do?

   And we won't even mention that despite the use of a DSL, the DSL
   code isn't "perfectly clear" either. This is especially true when
   it gets mixed with inline, non-DSL code. For instance, Axiom's
   algebra code regularly invokes low-level lisp functions.



>Exactly.  Conclusion: it's hard, maybe impossible, to generalize about what
>all code should look like.  Maybe it's essentially pluralistic.

Yes, it is hard to generalize about what all code should look like. But
it is not hard to generalize that reading natural language explanations
is faster, more accurate, and a lot easier than reverse-engineering
code. It is MUCH easier to understand Greg Humphrey's rendering code
than it is to understand the Clojure internals.

Consider a randomly chosen paragraph from Physically Based Rendering
(p356):



  To do the permutation, this function loops over the samples, randomly
  permuting the sample points in one dimension at a time. Note that this
  is a different permutation than the earlier Shuffle() routine: that
  routine does one permutation, keeping all nDim sample points in each
  sample together, while here nDim separate permutations of a single
  dimension at a time are done. (Figure 7.21)

   for (uint32_t i = 0; i < nDim; ++1) {
     for (uint32_t j = 0; j < nSamples; ++j) {
       uint32_t other = j + (rng.RandomUInt() % (nSamples - j));
       swap(samples[nDim + j + i], samples[nDim * other + i]);
     }
   }

  Footnote: While it's not necessary to permute the first dimension of
  the LHS pattern, the implementation here does so anyway since making
  the elements of the first dimension be randomly ordered means that LHS
  patterns can be used in conjunction with sampling patterns from other
  sources without danger of correlation between their sample points.



So we learned what the code does. We also learned not to "optimize the
code" by replacing it with Shuffle(). Further, we learned that we 
shouldn't "optimize the code" by removing the apparently useless
shuffle of the first dimension. And, as a bonus, we get a figure.
NONE OF THIS INFORMATION IS IN THE CODE ITSELF.

In addition, this code lives in an organizing structure. It is 
in Chapter 7: Sampling and Reconstruction
     Section 7.3: Stratified Samples 


Heck, it is only 4 lines of C++. Why bother? *I* can read C++.  I can
even reverse engineer it (probably by inventing the diagram in Figure
2.7 on a napkin). Maybe it lives in the src/SamRecon/StratSam, which is
all the organization necessary. :-)  But I can't reverse engineer the
important information in either paragraph of text.

For comparison, refer back to the multi-page Java code I posted
from the Clojure core. As a maintenance programmer, which would you
rather maintain?
  



Estimates are that programs that "live" spend 80% or more of their
"lifetime" in maintenance. 

Companies that depend on a "crufty old program" (a l33t-speak term for
any program not authored by the current maintainer) are willing to pay
the cost to have programmers read the code, take notes, and spend time
reverse-engineering the code.

In contrast, open source code tends to die the moment the original
authors stop typing. Sourceforge, github, savannah, etc. are
wall-to-wall full of code nobody will ever use. It is very common to
see someone write:

   "Well, it hasn't been updated in the last 6 months so I don't think
    it is being maintained anymore."

Being an "audience of one" when writing the code means the program
will have an "audience of one" in github; until 6 months pass and it has
an "audience of zero".

Languages come into fashion all the time. Almost all of them disappear.
Axiom and Clojure are worthwhile efforts. We really need a mindset that
extends into the future, communicating with the maintainers.

Tim Daly


For want of a comment, the fix was lost.
For want of a fix, the routine was lost.
For want of a routine, a program was lost.
For want of a program, the customer was lost.
For want of a customer, the company was lost.

[0] Zeleny, M. (ed.)
    "Autopoiesis, a Theory of the Living Organization"
    New York, Elsevier-North Holland (1978)