bug#61285: (Sometimes very) slow font-lock after %w in ruby-ts-mode

[Dmitry Gutov]

This probably involves a parser bug and/or maybe a tree-sitter one. But 
I'm posting this here anyway because this might not be the only way to 
trigger this problem. Or it could give us some optimization insights.

Also, while it involves a node which is parsed to have a large number of 
descendants, the performance depends heavily on whether the node is at 
the top level of the program (then it's slow), or not.

To repro:

1. Visit test/lisp/progmodes/ruby-mode-resources/ruby.rb
2. add 'a = %w' (without quotes) as a separate new line before all of 
the existing code.
3. Notice the delay in redisplay after you type 'w'.

In you do that with a larger file, BTW, this delay may be on the order 
of a minute. Here's an example of such file: 
https://github.com/rails/rails/blob/main/activerecord/lib/active_record/associations.rb

The superficial reason for this delay is that %w opens a new "array of 
strings" literal which parses every separate word in the rest of the 
buffer as a separate string. So we get a node with thousands of 
children, in the case of associations.rb. Or just ~1000 in the case of 
ruby.rb.

I also tried setting treesit--font-lock-fast-mode to t: no effect.

But! If we do the same not on top-level -- say, put the 'a = %w' line 
after the 'foo' line inside the first 'if' statement (i.e. on line 7), 
the delay is much smaller -- not noticeable in ruby.rb, and still 
apparent but much more bearable in associations.rb (you can put that 
statement right after 'module ActiveRecord') -- even though the size of 
the tree is changed minimally, and the number of children nodes for that 
"array of strings" still counts in the thousands (e.g. 13319).

Perf report for the "bad" highlighting delay looks like this:

  61.19%  emacs libtree-sitter.so.0.0  [.] ts_tree_cursor_current_status
  30.88%  emacs libtree-sitter.so.0.0  [.] ts_tree_cursor_parent_node
   7.44%  emacs libtree-sitter.so.0.0  [.] ts_language_symbol_metadata
   0.06%  emacs libtree-sitter.so.0.0  [.] ts_tree_cursor_goto_first_child
   0.05%  emacs libtree-sitter.so.0.0  [.] ts_tree_cursor_goto_next_sibling
   0.03%  emacs libtree-sitter.so.0.0  [.] ts_node_end_byte

And like this in the "good" case (with many type-backspace repetitions):

  32.10%  emacs libtree-sitter.so.0.0  [.] ts_tree_cursor_current_status
   9.50%  emacs libtree-sitter.so.0.0  [.] ts_tree_cursor_goto_first_child
   7.89%  emacs libtree-sitter.so.0.0  [.] ts_tree_cursor_goto_next_sibling
   7.51%  emacs libtree-sitter.so.0.0  [.] ts_language_symbol_metadata
   6.45%  emacs libtree-sitter.so.0.0  [.] ts_tree_cursor_parent_node
   1.87%  emacs libtree-sitter.so.0.0  [.] ts_node_start_point
   1.85%  emacs emacs  [.] process_mark_stack
   0.93%  emacs libtree-sitter.so.0.0  [.] ts_tree_cursor_current_node