bug#61369: Problem with keeping tree-sitter parse tree up-to-date

[Dmitry Gutov]

On 18/02/2023 03:14, Yuan Fu wrote:
> > On Feb 17, 2023, at 4:11 PM, Dmitry Gutov <dgutov@yandex.ru> wrote:
> >
> > On 18/02/2023 00:32, Yuan Fu wrote:
> > > Thank you very much! I thought that clipping the change into the fixed visible 
> > > range, and rely on treesit_sync_visible_region to add back the clipped “tail” 
> > > when we extend the visible range would be equivalent to not clipping, but I 
> > > guess clipping and re-adding affects how incremental parsing works inside 
> > > tree-sitter.
> >
> > It seems like the "repairing" sync used a different range, one that didn't 
> > include the character number 68 inserted from the beginning.
> >
> > It just synced the 1 or 2 characters at the end of the buffer, the difference 
> > between the computed visible_end and the actual BUF_ZV_BYTE.
>
> That should be enough, no? Because other text didn’t change, they just moved. 
> And tree-sitter should know that they moved. Or maybe I’m misunderstanding what 
> you mean.

But the "unsynced" character is at position 68.

And we just tell tree-sitter to update positions 134-136. So it stays 
ignorant of the changed char in the middle of the buffer.

It's not just about not knowing about the change either (the character 
in question is a newline, so its absence wouldn't lead to a syntax 
error), but about wrong offsets in the old parse tree, based on which 
the new tree is generated. That probably creates a wrong picture of the 
source text in the parser.

> > > I don’t think this change would have any adverse effect, because if you think 
> > > of it, inserting text in a narrowed region always extends the region, rather 
> > > than pushing text at the end out of the narrowed region. So the right thing to 
> > > do here is in fact not clipping new_end_offset.
> >
> > I figured it could be a problem if both old_end_byte and new_end_byte extend 
> > past the current restriction.
>
> That should be fine (ie, technically correct), since when we widen, the clipped 
> text are reparsed by tree-sitter as new text.

I guess the effect I was thinking of is that

  XTS_PARSER (lisp_parser)->visible_end

would end up with a higher value than BUF_ZV_BYTE. Not sure if it's a 
problem.

> > But I'm not sure whether that could actually happen in practice. The obvious 
> > attempts (undo a change outside of the narrowing, or revert the buffer when 
> > narrowing is in effect) didn't play out, but I'm not sure whether there is an 
> > actual hard limit on modifying the text outside of the current restriction.
>
> It is my impression that Emacs in general enforces the narrowing restriction 
> strictly. And we are still correct when exceptions occur.

Very good.