bug#60691: 29.0.60; Slow tree-sitter font-lock in ruby-ts-mode

[Dmitry Gutov]

On 12/01/2023 23:58, Yuan Fu wrote:
> Dmitry Gutov <dgutov@yandex.ru> writes:
>
> > Yuan? Just making sure you got this message.
>
> Sorry for the delay :-)
>
> > On 10/01/2023 16:10, Dmitry Gutov wrote:
> > > Perhaps Yuan has some further ideas. There are some strong oddities here:
> > > - Some time into debugging and repeating the benchmark again and
> > > again, I get the "Pure Lisp storage overflowed" message. Just once
> > > per Emacs session. It doesn't seem to change much, so it might be
> > > unimportant.
>
> That sounds like 60653. The next time you encounter it, could you record
> the output of M-x memory-usage and M-x memory-report?

Managed to reproduce this after running the test in a couple of 
different files.

But 'M-x memory-usage' says no such command, and 'M-x memory-report' 
ends up with this error:

Debugger entered--Lisp error: (wrong-type-argument number-or-marker-p nil)
  memory-report--gc-elem(nil strings)
  memory-report--garbage-collect()
  memory-report()
  funcall-interactively(memory-report)
  #<subr call-interactively>(memory-report record nil)
  apply(#<subr call-interactively> memory-report (record nil))
  call-interactively@ido-cr+-record-current-command(#<subr 
call-interactively> memory-report record nil)
  apply(call-interactively@ido-cr+-record-current-command #<subr 
call-interactively> (memory-report record nil))
  call-interactively(memory-report record nil)
  command-execute(memory-report record)
  execute-extended-command(nil "memory-report" nil)
  funcall-interactively(execute-extended-command nil "memory-report" nil)
  #<subr call-interactively>(execute-extended-command nil nil)
  apply(#<subr call-interactively> execute-extended-command (nil nil))
  call-interactively@ido-cr+-record-current-command(#<subr 
call-interactively> execute-extended-command nil nil)
  apply(call-interactively@ido-cr+-record-current-command #<subr 
call-interactively> (execute-extended-command nil nil))
  call-interactively(execute-extended-command nil nil)
  command-execute(execute-extended-command)

garbage-collect's docstring says:

  However, if there was overflow in pure space, and Emacs was dumped
  using the "unexec" method, ‘garbage-collect’ returns nil, because
  real GC can’t be done.

I don't know if my Emacs was dumped using "unexec", though. ./configure 
says I'm using pdumper.

In case that matters, I'm testing the emacs-29 branch.

> > > - The profiler output looks like this:
> > >     18050  75%                    -
> > > font-lock-fontify-syntactically-region
> > >     15686  65%                     - treesit-font-lock-fontify-region
> > >      3738  15% treesit--children-covering-range-recurse
> > >       188   0%                        treesit-fontify-with-override
> > > - When running the benchmark for the first time in a buffer (such as
> > > ruby.rb), the variable treesit--font-lock-fast-mode is usually
> > > changed to t. In one Emacs session, after I changed it to nil and
> > > re-ran the benchmark, the variable stayed nil, and the benchmark ran
> > > much faster (like 10s vs 36s).
> > > In the next session, after I restarted Emacs, that didn't happen: it
> > > always stayed at t, even if I reset it to nil between runs. But if I
> > > comment out the block in treesit-font-lock-fontify-region that uses
> > > it
> > >       ;; (when treesit--font-lock-fast-mode
> > >       ;;   (setq nodes (treesit--children-covering-range-recurse
> > >       ;;                (car nodes) start end (* 4 jit-lock-chunk-size))))
> > > and evaluate the defun, the benchmark runs much faster again: 11s.
> > > (But then I brought it all back, and re-ran the tests, and the
> > > variable stayed nil that time around; to sum up: the way it's turned
> > > on is unstable.)
> > > Should treesit--font-lock-fast-mode be locally bound inside that
> > > function, so that it's reset between chunks? Or maybe the condition
> > > for its enabling should be tweaked? E.g. I don't think there are any
> > > particularly large or deep nodes in ruby.rb's parse tree. It's a
> > > very shallow file.
>
> Yeah that is a not-very-clever hack. I’ve got an idea: I can add a C
> function that checks the maximum depth of a parse tree and the maximum
> node span, and turn on the fast-mode if the depth is too large or a node
> is too wide. And we do that check once before doing any fontification.
>
> I’ll report back once I add it.

Thanks!

And if the check can be fast enough, we could probably do it in the 
beginning of fontifying every chunk.