Re: treesitter local parser: huge slowdown and memory usage in a long file

[Yuan Fu]

> On May 27, 2024, at 3:24 PM, Dmitry Gutov <dmitry@gutov.dev> wrote:
> 
> On 28/05/2024 01:03, Yuan Fu wrote:
> 
>>> But if one operation just changes text in that range (keeping its length 
>>> intact, e.g. capitalizing the whole region), and another does the same 
>>> (back to lower case), then the combined range would remain 200..300.
>>> 
>>> Computing that might be difficult without having access to the kinds of 
>>> changes are being done (does tree-sitter report those?). OTOH, most of the 
>>> time the most important part is the position of the beginning of the 
>>> changes (e.g. for syntax-ppss), and we could treat the rest of the buffer 
>>> as invalidated…
>> Oh you’re absolutely right, the range will be shifted by later edits in the 
>> buffer. It’ll be hella hairy to keep track of all that—say the previous 
>> changed range is (100 . 200), and user inserted 50 chars in position 150, we 
>> need to account for that and update the range to (100 . 250) before merging 
>> the new updated ranges with this one.
>> So it seems the best way is really to move treesit--pre-redisplay entirely 
>> into the primary parser’s notifier, WDYT?
> 
> Yep, that sounds easier. And the performance should be about the same, even 
> if it'd have a bit extra overhead in those theoretical complex cases.
> 

Ok, I pushed a commit to master that does just that. I tried with C’s block 
comment, and php-ts-mode. Everything seems to work fine.

I also added treesit-primary-parser. This is supposed to be another 
configuration variable that a major mode should set. I’ve encountered various 
cases where knowing the primary parser (parser that parses the entire buffer 
rather than just a subset of it) would be very helpful. Treesit-primary-parser 
can be auto-guessed if major mode doesn’t set it, so it shouldn’t break 
anything. I’d love to know yours and Stefan’s thoughts on it.

Yuan