bug#31290: Fundamental bugs in syntax-propertize

[Dmitry Gutov]

On 4/28/18 12:08 AM, Alan Mackenzie wrote:

> At least that would be true if syntax-propertize--done hadn't been
> prematurely and spuriously increased, crudely to prevent an infinite
> recursion, falsely indicating to the syntax-ppss infrastructure that the
> syntax-table properties have already been applied to the region (BEGIN
> END).
>
>      .... but it should not call `syntax-ppss-flush-cache', ....
>
> Why not?  Because syntax-ppss-flush-cache sets syntax-propertize--done
> back to its true value, allowing the wrongly allowed syntax-ppss calls at
> a later position to cause a recursive loop.

Maybe we should "allow" it to loop, in certain cases? Leaving it to be 
the responsibility of the programmer, to make sure the result doesn't 
infloop, even if these rules are violated.

>      .... which means that it should not call `syntax-ppss' on some
>      position and later modify the buffer on some earlier position.
>
> This is a bad restriction, because sometimes syntax-table properties can
> only be correctly determined by examining the syntax of later buffer
> positions.  An example of this is giving the string-fence syntax-table
> text property to an unbalanced opening string quote, but not to correctly
> matched quotes.

I'm not exactly convinced by the given example (why would we use the 
string-fence in that case?), but it might be better if something like 
this was possible, indeed.

> 2. syntax-propertize-function's are banned from using syntax-ppss, the
> documentation instead directing them to use parse-partial-sexp directly.

The ones that currently call syntax-ppss, can't simply switch over to 
parse-partial-sexp without becoming slower due to the lack of cache.

Before tackling this bug, I'd rather we see a real-world problem that it 
caused, and pick a particular approach based on it.

But off the top of my head, we could introduce a "stricter but somewhat 
slower" variation of syntax-ppss to be called inside 
syntax-propertize-function's, which would treat the values in question 
more carefully, somehow.