Re: Syntax tables for multiple modes [was: bug#22983: syntax-ppss returns wrong result.]

[Vitalie Spinu]

>> On Mon, Mar 21 2016 10:43, Stefan Monnier wrote:

>> parse-partial-sexp is called from code exclusively and it just happens
>> that in multi-modes it is called outside of narrow region quite often.

> How/why?  Can you give some concrete scenario?

MM engine narrows to span region for a lot of tasks, most importantly
font-lock. If inner mode fortification functions misbehaves (ignoring
font-lock-dont-widen for example) like c-mode does this leads to trouble. So to
avoid those troubles you would advice individual functions and narrow them
properly or apply other tricks like overwriting output value or input args.  It
all works fine till that function calls parse-partial-sexp (or some other low
level function) and blows with args-out-of-range error.

To be frank, the issue of parse-partial-sexp is fading because modes are now
using syntax-ppss more extensively. Most of the problems with parse-partial-sexp
from the past are now internalized in condition-case inside syntax-ppss. That
condition-case is triggered very often (almost always) from inside polymode
chunk narrowing.

>> That's a major inconvenience. Why on earth one would need to
>> take account in user narrowing for syntax parsing?

> Because you need it for *everything* that looks at the buffer.
> Why should parse-partial-sexp be different from (say) scan-sexps?

I think parse-partial-sexp, syntax-ppss and maybe some others, are special in
the sense that in order to return a correct value they need to be aware of the
whole buffer. I don't see this as an inconsistency but I might be too naive.

>> If parse-partial-sexp could be made to always widen to hard limits it
>> will automatically solve a bunch of problems. bug#22983 being one of them,

> Bug#22983 should be fixed by widening, indeed, but it should be done in
> syntax.el.  Widening in parse-partial-sexp would only address some cases
> but not all (e.g. the syntax-begin-function cases or the
> syntax-propertize-function cases).  Those other cases can only be fixed
> in syntax.el.

>> the ubiquitous out-of-range errors in font-lock in multi-modes being
>> the most important one.

> I'm not familiar with those, so if you could give some examples it
> would help us judge if they would indeed benefit from a fix in
> parse-partial-sexp rather than elsewhere.

c-mode provides an example. I don't remember where exactly and how but it has to
do with but c-before-context-fl-expand-region and c-state-semi-safe-place
because I am advising these two functions currently.

The logic is roughly like this, c-mode engine doesn't respect
font-lock-dont-widen, widens stuff in some of it's functions, then calls its
parsing, gets back some points outside font-lock range and blows when trying to
access those points from narrowed region.

I was not collecting these cases carefully but I will start doing it and will
get with more concrete examples in the following weeks.

Another directions of problems is syntax-propertize. It can be called with POS
outside of current narrowed region. Particularly from
internal--syntax-propertize. But again I don't recall how exactly that was
happening now.

> Replace all your widen calls with calls to `prog-widen' and you get the same
> result (since (nth 1 prog-indentation-context) is basically another name for
> your hard-widen-limit).  So I don't think prog-widen is that very complicated.

It's not but you have to enforce that in all known modes.

> Note that "is more general" here means that the major mode's function has to
> handle more cases, so it would seem to fundamentally require more work on the
> major mode's side.

I don't agree. Work must be done only in the generic multi-mode engines
(mmm-mode, polymode etc). Other modes should re-use that generic infrastructure,
or even better, do nothing, and leave to someone else to define a new polymode
with host chunk being *the* mode. That every mode with basic needs for inner
sub-modes tries to re-invent the wheel is a dead end.


  Vitalie