Re: A vision for multiple major modes: some design notes

[Alan Mackenzie]

Hello, Eli.

On Thu, Apr 21, 2016 at 05:17:09PM +0300, Eli Zaretskii wrote:
> > Date: Wed, 20 Apr 2016 19:44:50 +0000
> > From: Alan Mackenzie <address@hidden>
> > 
> > This post describes my notion of how multiple major modes {c,sh}ould be
> > implemented.  Key notions are "islands", "island chains", and "chain
> > local" variable bindings.

> Thank you for publishing this.  A few comments and questions below.
> Please keep in mind that I never had to write any Lisp that deals with
> these issues, so apologies in advance for possibly silly questions and
> misunderstandings.

> >   o - To the user, the current major mode will be that of the island where
> >     point is.  All familiar commands will work without restriction.

> Does this mean the display of mode line, menu bar, and tool bar will
> change accordingly?

Yes, please!

> A more subtle issue is with point movements that are not shown to the
> user (those done by Lisp code of some command, before redisplay kicks
> in) -- what will be the effect of those? do they trigger redisplay,
> for example?

They shouldn't trigger redisplay, no.

> >   o - An island chain will have @dfn{chain local} variable bindings.  Such a
> >     binding will become current and accessible when point is within one of 
> > the
> >     chain's islands.  When point is not in an island, the buffer local 
> > binding
> >     of the variable will be current.

> Emacs sometimes examines buffer text without moving point, and we
> generally expect for buffer-local bindings to be in effect regardless.
> A prominent example is the display engine.  I will return to that
> later.

OK.

> >     * - [Island] will be covered by the text property `island', whose value 
> > will be
> >       the pertinent island or island chain (see section (ii)) (not yet
> >       decided).  Note that if islands are enclosed inside other islands, the
> >       value is the innermost island.  There is the possibility of using an
> >       interval tree independent of the one for text properties to increase
> >       performance.

> I don't understand the notion of "enclosed" islands: wouldn't such
> "enclosing" simply break the "outer" island into two separate islands?

If we mark island start and end with the syntax-table text properties
"{" and "}", we're going to have something like

    {     a{  }b    }

.  Simply to break the outer island into two pieces, we'd really need to
apply delimiters at a and b, giving:

    {     }{  }{    }

.  This would overwrite the previous syntaxes at a and b, and this might
be a Bad Thing.

> >   o - `scan-lists', `scan-sexps', etc. will treat a "foreign" island as
> >     whitespace, much as they do comments.  They will also treat as 
> > whitespace
> >     the gap between two islands in a chain.

> Why whitespace? why not some new category?  By overloading whitespace,
> you make things harder on the underlying infrastructure, like regexp
> search and matching.

I think it's clear that the "foreign" island's syntax has no interaction
with the current island.  If we treat it as whitespace, that should
minimise the amount of adapting we need to do to existing major modes.

I envisage that a regexp element will match the "foreign" island if that
element would match a space.  I know this sounds horrible, but I haven't
come up with a scenario where this wouldn't work well.  (This is
assuming, of course, that the magic flag `in-islands' is non-nil.)

> >   o - The regexp engine will be enhanced such that the regexps "\\s-", "\\s 
> > ",
> >     and "[[:space:]] will match an entire island.

> Extending [:space:] that way seems to be an implementation detail
> leaking to user level.  I think we should avoid that at all costs.

Why?  I don't understand your last paragraph.

> >   o - The gap between two islands in a chain will also be matched by the 
> > above
> >     regexps.
> >   o - This treatment of an island, and a gap between two islands, as WS will
> >     occur only when `in-islands' is non-nil.
> >   o - When `in-islands' is nil, there will be no reliable way of scanning 
> > over
> >     an island by regexps, since it is a potentially nested structure, and 
> > FSMs
> >     don't recognise arbitrarily nested structures.

> > (vii) Variables.
> >   o - Island chain local variable bindings will come into existence.  These
> >     bindings depend on the island point is in.  There will be lower level
> >     routines that will have "position" parameters as an alternative to using
> >     point.
> >   o - All variables which are currently buffer local will become chain local
> >     except for those whose symbols are given a non-nil `entire-buffer'
> >     property.  There will be no new functions like
> >     `make-chain-local-variable'.
> >   o - When the `entire-buffer' property is nil, the buffer local binding of 
> > a
> >     variable will hold the value pertinent to the areas of the buffer 
> > outside
> >     of islands.  When that property is non-nil, the binding holds the value
> >     for the entire buffer.
> >   o - When `in-islands' is nil, the chain local mechanism described here is
> >     not used - instead the familiar buffer local binding is used.
> >   o - The current binding for a local variable will be the chain local 
> > binding
> >     of the island chain of the island containing point.  If point is not in 
> > an
> >     island, the buffer local binding is current.
> >   o - If a chain local binding is current, and its value is unbound, the
> >     binding of an enclosing scope is NOT used in its place.  Probably the
> >     variable's default-value should be used when reading.
> >   o - In buffer.h, a new macro CVAR ("island chain variable") analogous to
> >     BVAR will be introduced.  It will use BVAR as a fall back.  Most
> >     invocations of BVAR will be changed to CVAR.
> >   o - In data.c, the mechanism for accessing local variable bindings
> >     (e.g. `swap_in_symval_forwarding') will be enhanced to test `in-islands'
> >     and handle chain local bindings appropriately.

> I'm not sure I understand the details.  E.g., where will the
> island-chain local values be stored?

In a C struct chain, analogous to struct buffer, using much the same
mechanisms.

> To remind you, buffer-local variables have a special object in their
> symbol value cell, and BVAR only works for the few buffer-local
> variables that are stored in the buffer object itself.  I'm not sure I
> understand how CVAR could solve the problem you need to solve, which
> is keeping multiple chains per buffer, each one with its values of
> these variables.

CVAR would get the current chain from the `island' (or `chain') text
property at the position.  If this is nil, it would do what BVAR does.
Otherwise it would access the appropriate named element in the struct
chain.  I think CVAR would take three parameters: the variable name, the
buffer, and the buffer position.

Other chain local variables would be accessed through an alist in the
struct chain holding miscellaneous variables, exactly as is done for
the other buffer local variables in struct buffer.

Unless there is a better solution, of course.

> > (ix) Miscellaneous commands and functions.
> >   o - `point-min' and `point-max' will, when `in-islands' is non-nil, return
> >     the max/min point in the visible region in the same chain of islands as
> >     point.
> >   o - `search-\(forward\|backward\)\(-regexp\)?' will restrict themselves to
> >     the current island chain when `in-islands' is non-nil.
> >   o - `skip-\(chars\|syntax\)-\(forward\|backward\)' will likewise operate 
> > in
> >     the current island chain (how?) when `in-islands' is non-nil.
> >   o - `\(next\|previous\)-\(single\|char\)-property-change', etc., will do 
> > the
> >     Right Thing in island chains when `in-islands' is non-nil.
> >   o - New functions `island-min', `island-max', `island-chain-min' and
> >     `island-chain-max' will do what their names say.
> >   o - There will be no restrictions on the use of widening/narrowing, as 
> > have
> >     been proposed for other support engines for multiple major modes.
> >   o - New commands like `beginning-of-island', `narrow-to-island', etc. will
> >     be wanted.  More difficultly, bindings for them will be needed.
> >   o - ??? Other commands to be amended.

> This actually sounds like a simple extension of narrowing, so I wonder
> why do we need so many new object types and notions.

I think it's more like a complicated extension of narrowing.  :-)  I
think that chain local variables are essential to multiple major modes -
you can't have m.m.m. without some sort of chain locality.  I also think
that for a major mode to work transparently over several chained
islands, all the irrelevant stuff between the islands needs to be made,
er, transparent.  That is what section (ix) is about.

> > (x) Emacs subsystems and `in-islands'.
> >   o - Redisplay will bind `in-islands' to non-nil, but will successfully
> >     display all islands wholly or partially in windows being displayed.
> >   o - Font Lock will bind `in-islands' to non-nil, but will successfully
> >     fontify all pertinent islands.
> >   o - `island-before/after-change-function' will be called with `in-islands'
> >     nil.
> >   o - `before/after-change-functions' will be called with `in-islands' bound
> >     to non-nil.
> >   o - Major modes will need to bind `in-islands' to non-nil for such things 
> > as
> >     indentation.
> >   o - For normal user interaction, `in-islands' will be nil.

> I don't see any discussion of how redisplay will deal with islands.
> To remind you, redisplay moves through portions of the buffer, without
> moving point, and access buffer-local variables for its job.  You need
> to augment the design with something that will allow redisplay see the
> correct values of variables depending on the buffer position it is at.
> The same problem exists for any features that use display simulation
> for making decisions about movement and layout, e.g. vertical-motion.

I think redisplay is mostly controlled by variables (such as
`scroll-margin') accessed by BVAR.  These calls could be replaced by
CVAR.  Problems will arise if redisplay reads the variable once, and
fails to read it again when its current position moves into or out of an
island.  Redisplay would have to be aware of island boundaries, and
re-read the controlling variables on passing a boundary.  Other than
that, I can't see any big problems.  Not yet, anyway.

[ .... ]

> Thanks.

-- 
Alan Mackenzie (Nuremberg, Germany).