bug#60691: 29.0.60; Slow tree-sitter font-lock in ruby-ts-mode

[Dmitry Gutov]

On 30/01/2023 01:23, Yuan Fu wrote:
> > On Jan 29, 2023, at 3:07 PM, Dmitry Gutov<dgutov@yandex.ru>  wrote:
> >
> > Hi Yuan,
> >
> > On 29/01/2023 10:25, Yuan Fu wrote:
> >
> > > > > > So if previously it happened once somehow during a certain scenario, now I have 
> > > > > > to repeat the same scenario 4 times, and the condition is met.
> > > > > I was hoping that the scenario only happen once, oh well 😄 I’ll
> > > > > change the decision based on analyzing the tree’s dimension: too
> > > > > deep or too wide activates the fast mode. Let’s see how it works.
> > > > Thank you, let me know when it's time to test again.
> > > Sorry for the delay. Now treesit-font-lock-fontify-region uses
> > > treesit-subtree-stat to determine whether to enable the "fast mode". Now
> > > it should be impossible to activate the fast mode on moderately sized
> > > buffers.
> > Thank you, it seems to work just fine in my scenario. And treesit-subtree-stat 
> > makes sense.
> >
> > I have a few more questions about the current strategy, though.
> >
> > IIUC, we only do the treesit--font-lock-fast-mode test once in 
> > treesit-font-lock-fontify-region, and then use the detected value for the whole 
> > later life of the buffer. Is that right?
> >
> > What if the buffer didn't originally have the problematic error nodes we are guarding 
> > from, and then later the user wrote enough code to have at least one of them? If they 
> > didn't close Emacs, or revert the buffer, our logic still wouldn't use the "fast 
> > node", would it?
> >
> > Or vice versa: if the buffer started out with error nodes, and consequently, "fast mode", 
> > but then the user has edited it so that those error nodes disappeared, shouldn't the buffer stop 
> > using the "fast mode"?
> >
> >  From my measurements, in ruby-mode, at least treesit-subtree-stat is 20-40x faster than 
> > refontifying the whole buffer. So one possible strategy would be to repeat the test every 
> > time. I'm not sure it's fast enough in the "problem" buffers, though, and I 
> > don't have any to test.
> >
> > In those I did test, though, it takes ~1 ms.
> >
> > But we could repeat the test only once every couple of seconds and/or after the 
> > buffer has changed again. That would hopefully make it a non-bottleneck in all 
> > cases.
> I should mention this in the comments, but the fast mode is only for very rare 
> cases, where the file is mechanically generated and has some peculiarities that 
> causes tree-sitter to work poorly. If the file is hand-written and “normal”, 
> even huge files like xdisp.c is well below the bar. Therefore I don’t think 
> “crossing the line” will realistically happen when editing source files.
>
> Here is the stats of two “problematic files”, named packet and dec_mask, 
> comparing to xdisp.c:
>
> ;;           max-depth max-width count
> ;; cut-off   100       4000
> ;; packet   (98159     46581 1895137)
> ;; dec mask (3         64301 283995)
> ;; xdisp.c  (29        985   218971)
>
> I’d say that any regular source file, even mechanically generated, wouldn’t go 
> beyond ~50 levels in depth, and hand-written files should never has a node that 
> has 4000+ direct children in the parse tree.

Oh, thanks for the explanation. Then the current strategy makes sense.

Is xdisp.c absolutely the largest C file in your experience?

According to the above numbers, a file that's only 4x as large could hit 
our current cutoff.

Though, TBH, maybe some extreme files do, and they have font-lock 
performance reduced somewhat. That's not the end of the world, and it 
shouldn't make a difference for the original scenario (diff-syntax 
fontification).

Either way, I'm closing this report. Thank you for your help.