bug#61369: Problem with keeping tree-sitter parse tree up-to-date

[Yuan Fu]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 10/02/2023 03:22, Yuan Fu wrote:
>>>   I just want to confirm that I can reproduce this, and that if you skip
>>>   the trailing newline from the use-statement, I don't get this behavior.
>>>   So it seems like the newline is the crucial point, right?
>>>
>>> Yes, same.
>>>
>>> Thr trailing newline is necessary.
>>>
>>> The empty lines at the beginning of the buffer (being copied to) are 
>>> necessary to reproduce this as well.
>> Hmmm, it might be related to how does tree-sitter does incremental
>> parsing? If the newline is necessary, then I guess it’s not because
>> Emacs missed characters when reporting edits to tree-sitter.
>
> The newline is somewhat necessary: the scenario doesn't work, for
> example, if the pasted text doesn't include the newline but the buffer
> had an additional (third) one at the top.
>
> But the scenario also doesn't work if some other (any) character is
> removed from the yanked line before pasting: it could be even one
> after the comment instruction (//).
>
> OTOH, if I add an extra char to the yanked line, anywhere, I can skip
> the newline. E.g. I can paste
>
>   use std::path::{self, Path, PathBuf};  // good: std is a crate namee
>
> without a newline and still see the exact same syntax error.
>
> So it looks more like an off-by-one error somewhere. Maybe in our
> code, but maybe in tree-sitter somewhere.

Some progress report: I added a function that reads the buffer like a
parser would, like this:

DEFUN ("treesit--parser-view",
       Ftreesit__parser_view,
       Streesit__parser_view, 1, 1, 0,
       doc: /* Return the view of PARSER.
Read buffer like PARSER would into a string and return it.  */)
  (Lisp_Object parser)
{
  const ptrdiff_t visible_beg = XTS_PARSER (parser)->visible_beg;
  const ptrdiff_t visible_end = XTS_PARSER (parser)->visible_end;
  const ptrdiff_t view_len = visible_end - visible_beg;

  char *str_buf = xzalloc (view_len + 1);
  uint32_t read = 0;
  TSPoint pos = { 0 };
  for (int idx = 0; idx < view_len; idx++)
    {
      const char *ch = treesit_read_buffer (XTS_PARSER (parser),
                                            idx, pos, &read);
      if (read == 0)
        {
          xfree (str_buf);
          xsignal1 (Qtreesit_error, make_fixnum (idx));
        }
      else
        str_buf[idx] = *ch;
    }
  Lisp_Object ret_str = make_string (str_buf, view_len);
  xfree (str_buf);
  return ret_str;
}

After I follow the steps and got the error node, I run this function on
the parser, and the returned string looks good.

Next I’ll try to log every character actually read by the parser and see
if anything seems fishy.

Yuan