bug#61285: (Sometimes very) slow font-lock after %w in ruby-ts-mode

[Yuan Fu]

Dmitry Gutov <dgutov@yandex.ru> writes:

> This probably involves a parser bug and/or maybe a tree-sitter one.
> But I'm posting this here anyway because this might not be the only
> way to trigger this problem. Or it could give us some optimization
> insights.
>
> Also, while it involves a node which is parsed to have a large number
> of descendants, the performance depends heavily on whether the node is
> at the top level of the program (then it's slow), or not.
>
> To repro:
>
> 1. Visit test/lisp/progmodes/ruby-mode-resources/ruby.rb
> 2. add 'a = %w' (without quotes) as a separate new line before all of
> the existing code.
> 3. Notice the delay in redisplay after you type 'w'.
>
> In you do that with a larger file, BTW, this delay may be on the order
> of a minute. Here's an example of such file:
> https://github.com/rails/rails/blob/main/activerecord/lib/active_record/associations.rb
>
> The superficial reason for this delay is that %w opens a new "array of
> strings" literal which parses every separate word in the rest of the
> buffer as a separate string. So we get a node with thousands of
> children, in the case of associations.rb. Or just ~1000 in the case of
> ruby.rb.
>
> I also tried setting treesit--font-lock-fast-mode to t: no effect.
>
> But! If we do the same not on top-level -- say, put the 'a = %w' line
> after the 'foo' line inside the first 'if' statement (i.e. on line 7),
> the delay is much smaller -- not noticeable in ruby.rb, and still
> apparent but much more bearable in associations.rb (you can put that
> statement right after 'module ActiveRecord') -- even though the size
> of the tree is changed minimally, and the number of children nodes for
> that "array of strings" still counts in the thousands (e.g. 13319).
>
> Perf report for the "bad" highlighting delay looks like this:
>
>   61.19%  emacs libtree-sitter.so.0.0  [.] ts_tree_cursor_current_status
>   30.88%  emacs libtree-sitter.so.0.0  [.] ts_tree_cursor_parent_node
>    7.44%  emacs libtree-sitter.so.0.0  [.] ts_language_symbol_metadata
>    0.06%  emacs libtree-sitter.so.0.0  [.] ts_tree_cursor_goto_first_child
>    0.05%  emacs libtree-sitter.so.0.0  [.] ts_tree_cursor_goto_next_sibling
>    0.03%  emacs libtree-sitter.so.0.0  [.] ts_node_end_byte
>
> And like this in the "good" case (with many type-backspace repetitions):
>
>   32.10%  emacs libtree-sitter.so.0.0  [.] ts_tree_cursor_current_status
>    9.50%  emacs libtree-sitter.so.0.0  [.] ts_tree_cursor_goto_first_child
>    7.89%  emacs libtree-sitter.so.0.0  [.] ts_tree_cursor_goto_next_sibling
>    7.51%  emacs libtree-sitter.so.0.0  [.] ts_language_symbol_metadata
>    6.45%  emacs libtree-sitter.so.0.0  [.] ts_tree_cursor_parent_node
>    1.87%  emacs libtree-sitter.so.0.0  [.] ts_node_start_point
>    1.85%  emacs emacs  [.] process_mark_stack
>    0.93%  emacs libtree-sitter.so.0.0  [.] ts_tree_cursor_current_node

Interesting. Perhaps it has to do with how tree-sitter implements the
"incremental" part of the parser? But the profile doesn’t look like it’s
spending time parsing, I need to look at what does
ts_tree_cursor_current_status actually do (maybe it’s used in parsing?)

Yuan