Re: Tree-sitter api

[Yuan Fu]

> On Dec 18, 2021, at 11:11 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Yuan Fu <casouri@gmail.com>
>> Date: Sat, 18 Dec 2021 18:51:25 -0800
>> Cc: Philipp <p.stephani2@gmail.com>,
>> ubolonton@gmail.com,
>> theo@thornhill.no,
>> cpitclaudel@gmail.com,
>> emacs-devel@gnu.org,
>> monnier@iro.umontreal.ca,
>> stephen_leake@stephe-leake.org,
>> john@yates-sheets.org
>> 
>>>> That feature in itself won't be enough.  Even with it, TreeSitter will 
>>>> have the same problem as GMP: allocation isn't allowed to fail, and 
>>>> longjmp'ing out of it isn't allowed and generally causes undefined 
>>>> behavior.
>>> 
>>> It may not be enough to satisfy purists, but it's enough to allow the
>>> user to save the session and shut down Emacs in an orderly fashion,
>>> instead of abruptly exiting and losing all the edits.
>> 
>> Uses can set tree-sitter-maximum-size to limit memory usage of tree-sitter. 
>> Buffers with size larger than that cannot enable tree-sitter. That doesn’t 
>> solve the problem directly but should let users avoid allocation failing 
>> most of the time.
> 
> Btw, we should have a good idea how frequent this out-of-memory
> problem could be with tree-sitter.  Did someone try to scroll through
> all of xdisp.c, using tree-sitter for C Mode fontifications, and
> measured the memory footprint that produces?  If not, I think it would
> be a good idea to try.
> 
> If the OOM problem happens frequently with large source files, it may
> indeed be the case that we will need to disable tree-sitter up front
> based on some size criteria.

From the author’s quote and my experiments, tree-sitter uses about 10–20x 
memory of the buffer size. So xdisp.c is fine. Also you don’t need to scroll 
through the buffer, tree-sitter parses the whole buffer up-front.

Yuan