bug#60650: 30.0.50; tree-sitter: parsing (and font locking) occasionally breaks during editing

[Yuan Fu]

Mickey Petersen <mickey@masteringemacs.org> writes:

> Yuan Fu <casouri@gmail.com> writes:
>
>> Mickey Petersen <mickey@masteringemacs.org> writes:
>>
>>> It is occasionally possible to put the tree-sitter parser, I am
>>> guessing, into an erroneous state w.r.t. the contents of the buffer it
>>> is charged with parsing.
>>>
>>> This rarely presents itself, and when it does, it only seems to happen
>>> when text is deleted and inserted in short order (`delete-region' +
>>> `insert') from an elisp command.
>>>
>>> The resulting erroneous state then breaks font locking, resulting in 
>>> mis-fontified code.
>>>
>>> I must note that the parse tree is _correct_ and not invalid. There is
>>> no easy way to 'fix' this bug, as newlines and other manual edits
>>> rarely fixes the issue. There is also no easy way to insist that
>>> tree-sitter must re-do everything. Presumably deleting and recreating
>>> the parser could work as a workaround? I do not know.
>>>
>>> The only way to truly and accurately fix the issue is to save and
>>> re-open the file. This then forces a complete re-parse of the buffer
>>> and fontification, fixing the issue.
>>
>> What do you mean by "correct and not invalid"? If a complete re-parse
>> fixes the issue, then the old parse tree must be incorrect, right? As
>> for mis-fontified code, is the position wrong (everything skew left or
>> right for a few character), or there is no fontification, or something
>> else? If the position is wrong, then it’s probably a bug in syncing the
>> parser’s position wrt the buffer content.
>
> I mean that the source code is syntactically correct. Given a clean
> slate, the parser would not report any erroneous nodes.

OK, so the source code is syntactically correct, but the parse tree is
not (which lead to problems in fontification and indent).  Deleting the
parser and reparsing the whole buffer from scratch produces a correct
parse tree.

> But, somehow, both the font lock and indentation remains incorrect,
> even when I type text that would otherwise force a reparse.

That I can understand. If the parse-tree becomes erroneous, further
incremental parsing on it just continues on the wrong path (GIGO). You
have to delete the parser and re-parse from scratch to fix it, as you
observed.

> The only
> way to fix this is to save the buffer (in a correct state), close and
> re-open the file. That fixes it.
>
> That, and undoing the edited steps done by the elisp code. That snaps it back 
> to its original state, and the parse tree is back to normal.
>
> It's... weird.

Indeed, there is no "undo" in tree-sitter, the parser just parses the
edits made by Emacs’ undo. And as I mentioned above, incrementally
parsing further on an erroneous tree shouldn’t magically fix the parse
tree. So this error can be corrected by doing the reverse edit on the
parse tree, hmmm.

>>
>>> I have zero means of reliably of reproducing this, unfortunately: it is 
>>> sporadic, intermittent, and happens somewhat infrequently.
>>>
>>> So I'd love some ideas on how to go about debugging this.
>>
>> I think the first step is to better understand what’s happening. Some
>> screenshots are also welcome. Thanks for your effort in advance!
>>
>
> Yep. Will do!

Thanks. Now I understands the situation better. Next time you encounter
this, please take screenshots of the wrong fontification, the parse tree
(ie, content of treesit-explore-mode), and screenshots of the correct
fontification and parse tree (since you mentioned that undo can fix it),
and the content of C-h l (view-lossage) (unless it’s sensitive ;-).

Yuan