Re: Syntax tables for multiple modes [was: bug#22983: syntax-ppss returns wrong result.]

[Vitalie Spinu]

>> On Sun, Mar 20 2016 04:19, Dmitry Gutov wrote:

>> So if a mode author decides to regexpf for a wiki link on a full buffer after
>> widening it, islands won't help.

> Where does widening happens in this case? First, we have font-lock-dont-widen.

Well, font-lock-dont-widen is not respected even in c-mode. Look at
c-before-context-fl-expand-region and semi-safe-place which are called directly
or indirectly from c-font-lock-fontify-region.

> For indentation, we've introduced prog-indentation-context recently. And
> indentation functions in programming modes are supposed to call prog-widen
> instead of widen now.

I was not aware of that. Not sure if it is a step in right direction though.

`prog-indentation-context` looks fine to me but multi-modes already have their
own wrappers for indentation which do just that according to their own semantics
of modes/submodes/chunks/headers etc.

The primary intent of `prog-indentation-context` is to be used in
`prog-widen`. This part seems like a major complication. All mode authors now
have to understand what is prog-widen, prog-first-column and
prog-indentation-context. Why to burden prog-mode authors with notions that
multi-mode engines can take care themselves?

It is also not clear to me why should prog-widen be used in indentation context
only? It makes perfect sense for this function to be used in font-locking and
syntax-propertize-function as well.

It's essentially a half-backed implementation of "hard widening" discussed
earlier. Why not impose the widening restriction directly in `widen` then? Maybe
bring widen to elisp and rename C widen into widen-internal. Then add generic
`prog-hard-widen-limits` which would be checked along prog-indentation-context
limits.

Multi-mode engines can then impose those hard limits whenever they need to and
adjust indentation accordingly. It's not that hard in my experience. Polymode
has a few lines to wrap indentation and it works reasonably well in pretty much
all contexts I have tried. All other problems can be solved with hard narrowing.

  https://github.com/vspinu/polymode/blob/master/polymode-methods.el#L715-L809

Unless I miss something essential it's really not worth imposing such
complexities on mode authors. Judging from the python.el, which is the only mode
using prog-first-column so far, it's not a trivial task. Each mode author will
basically have to implement indentation logic that mmm-mode or polymode already
implement. And even then, multi-mode engines will probably need to overwrite
that because the semantics of submode spans is either emacs-mode or
multi-mode-engine specific.

> syntax-propertize-function's aren't supposed to call widen at all, I think.

This should probably be in the docs then. Mode authors can decide to do loads of
work in there. One instance is `markdown-mode` which caches all font-lock
properties in syntax-propertize-function. While markdown-mode is clean and
doesn't use widen anywhere, that might not be the case for other modes.


  Vitalie