Re: existing work on TODO items

[Ken Manheimer]

On 1/8/06, Dave Love <address@hidden> wrote:
> "Richard M. Stallman" <address@hidden> writes:
>
> > Could you please be more specific?  I never knew all that much about
> > allout.  All I know is that it does a more powerful kind of outlining.
>
> I never figured it out either, especially compared with Hyperbole's
> outliner (or Org).  Then when I actually tried it to investigate, I
> found it wouldn't actually run as then installed, which suggests it's
> not actually used.  [Shouldn't packages be required to compile cleanly
> when they're installed -- there was at least macro use before
> definition -- and be clean for Checkdoc?  Not that Allout is atypical
> that way.]

there was a fairly short period of time (weeks or maybe months) when
the crypt++/mailcrypt dependent version was released.  the encryption
features were not present before that.

i do believe allout is used by at least a few people besides me - i
have gotten several appreciation notes and code contributions over the
years, and there's an emacs-wiki page for it, with some addons and
"quick-start" documentation (
http://www.emacswiki.org/cgi-bin/wiki/AllOut ).  and, as far as i can
tell, allout now does compile cleanly, as of the last release.  (not
an accident - eli asked me to clean that up as i was submitting
revisions.)

> > Oh, I see allout.el was changed to use pgg.el, which was also used by
> > Gnus.  Is that the change you are criticizing?
>
> No; I didn't realize it had been changed to use the internal support.
> Originally it was updated using external libraries to do this
> encryption task (which have fundamental problems).  That's what I
> meant.  Now there seems to have been a lot of work done on pgg for
> this during some sort of non-freeze, but not for the general facility
> in TODO.  I don't understand this.

i don't understand what you mean by "the general facility in TODO".

> I didn't mean to single out allout or denigrate contributions.  The
> basic point is that development badly needs to stabilize, fix all the
> problems, and make a release.  It looks worse than the Emacs 21
> non-freeze, which I hoped would have shown what to avoid at least.

for what it's worth, i was submitting my updates not aware of the
situation.  that said, i would very much like to have the improved
code and new functionality released with emacs 22, but that's up to
the release managers.

> > Well, it is another outliner, so it would need to include the basic
> > functionality of outlines.  Are you saying that it could use
> > outline.el instead of duplicate it?
>
> Or replace it, if was a superset.  It turns out it isn't, though.

it's not based on outline functions.  it does things differently
enough - both internally and in terms of user-level nuances - that it
would take a lot of work to converge the two.  furthermore, i just did
a bit of measurement, and i would not want to try to base it on the
existing outline.el functionality, both in terms of performance and
functional nuances.

most of my editing happens in outlines, including particularly a daily
activities log, which i let get pretty big - jan 2003 to today, 1.2 MB
of text.  i often work with most everything but today's entry
collapsed.  for testing, i duplicated this outline changing the
bullets to outline.el's pure-asterisks, and compared some operations
with a time-trial function that i use.  (i'm attaching both the
time-trial function and sanitized copies of the allout and
outline-mode format outlines in case anyone else wants to fiddle with
them.)

even without the time-trial, the most striking thing is that typing in
an uncollapsed area of an large outline that otherwise has a lot of
collapsed (hidden) topics is intolerably painful in the outline-mode
buffer.  collapsing everything to the top-level topic and moving
around there with C-f, or adding characters, takes a few seconds for
each keystroke.  there is some delay in the allout-mode buffer, but
it's tolerable - much much less extreme.  (the time trial function
shows something like an order of magnitude more time for 'forward-char
in the collapsed outline-mode buffer, but it's still a matter of 100
usecs - there's something happening around the polling loops that
makes every key stroke take several seconds...)

big collapses and expands take a lot more time in the outline-mode buffer. eg:

from a fully-collapsed outline:
  outline-mode: (time-trial '(show-children)) => 1.332 - 1.704 seconds
  allout-mode: (time-trial '(allout-show-children)) => 0.264 - 0.301 seconds
    - three trials each, near an order of magnitude slower in outline-mode

hide bodies from a fully open outline:
  outline-mode: (time-trial '(hide-body)) => 35.245 seconds
  allout-mode: (time-trial '(allout-hide-bodies)) => 0.343 seconds
    - two orders of magnitude

the hide-bodies delay is extreme, but not something i'd commonly do,
while the show children operation is much more likely, and less
drastic.

there are also numerous behavioral nuances that differ in allout and
that i would not sacrifice to reuse outline-mode code, eg:

  - different concept of whitespace between items
    - allout treats trailling blank line in item as white-space separating
      it from next item, rather than part of the item's body
    - outline-mode has no notion of whitespace between items - everything
      is squashed together unless fully opened.
      even more so than with programming languages, insufficient whitespace
      between entities makes reading much more difficult.

  - isearch
   - while isearching:
    - allout just exposes immediately-containing topics of intermediate
      topics, and reconceals them if you search past
    - outline.el opens everything
   - upon arrival
    - allout opens containment context and sibling headers, plus body of
      search target if search was in target
    - outline.el just opens immediately-containing topic, leaving
      outline in incohesive state.  i see no way to operationally
      rectify that state, short of `show-all' and then somehow close
      everything down to the point you want to see.  (maybe it's not
      working right for me??)

  - opening from a hidden place - eg, a mark that was left within a
    collapsed topic
    - allout opens topics, cohesively, up to the one containing the cursor,
      whatever the depth,
    - outline.el opens from the visible topic, regardless of hidden position,
      and not necessarily showing the hidden position

there are probably several other nuances that make moment-to-moment
and day-to-day authoring of outlines with allout manageable, because
i've been tinkering with it to do my outlining for years.  i don't
think this will be reconciled trivially with another code base.

my general conclusion is that outline-mode has not been used for
editing or even navigating very extensive outlines, and that shifting
to base my code on it would not be a win, even if it were easy to do
(and i don't know that it would be easy).

> > It would be good to make that change, after the release--if it really
> > results in an improvement.  Sometimes duplicating code makes
> > maintenance harder, and sometimes it makes maintenance easier.
>
> Sure, but mostly the latter.  Just the code size (1/4 MB of allout)
> must be significant.  It's not just maintenance, though; it affects
> interactive and programmatic users too.  For instance, if allout was
> built on outline, it would be obvious to us that it was at least a
> superset, and there would be mode-specific support which doesn't seem
> to have been programmed for allout.  It can't be good to have a lot of
> duplicated work (languishing unreleased anyway) while more fundamental
> generally-useful things aren't addressed and bugs don't get fixed.

sometimes divergence is necessary.  i'm not saying allout is an
unqualified win in every direction - but it departs from outline.el
internally and in many behavioral choices so that their convergence is
non-trivial in principle, and not just in practice.

and all that said, my current interest is departing from both allout
and outline code base and basing something on emacs widgets,
specifically the tree widget.  i'm experimenting with that.  if i can
reasonably implement an elaborate feature - the allout
dynamic-exposure isearch functionality - with a derivative of the tree
widget, i will be able tell whether or not it'll be easy to implement
most of what i want, and what i expect would be a cleaner,
widget-based approach.  if that works, i would use allout format as
the externalization format, and continue to maintain allout-mode as an
alternative, fallback choice for editing such outlines.

ken manheimer
address@hidden

time-trial.el
Description: Binary data

log_outline-mode.gz
Description: GNU Zip compressed data

log_allout-mode.gz
Description: GNU Zip compressed data