bug#51766: 29.0.50; Return value of buffer-chars-modified-tick changes when buffer text is not yet changed before inserting a character for non-latin input methods

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: 51766@debbugs.gnu.org
> Date: Fri, 12 Nov 2021 21:39:54 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> quail changes the buffer after org-element--after-change-function call,
> >> but before org-element--before-change-function. So, all Org can see is
> >> that something has been changed in buffer, but there is no way to tell
> >> what it was. Org cannot distinguish between harmless buffer edits by
> >> quail (they do not change buffer text) and other kinds of "silent"
> >> changes.
> >
> > OK, but why does this invalidate what Org does?  All it means, AFAIU,
> > is that in some cases Org will do unnecessary processing.  Those cases
> > are probably not too frequent.
> 
> Normally, buffer changes under inhibit-modification-hooks are not
> frequent indeed. But not with quail + non-latin input methods. Every
> single self-insert-command triggers such change.

"Such change" being what exactly? the situation where
buffer-chars-modified-tick changes between post-command-hook and the
following pre-command-hook? or something else?

I'm asking because I don't think I see the problem you are describing.
With the following code:

  (defun my-before (beg end)
    (message "buf %s beg %s end %s" (current-buffer) beg end))
  (defun my-after (beg end len)
    (message "buf %s beg %s end %s len %s" (current-buffer) beg end len))
  (add-hook 'before-change-functions 'my-before)
  (add-hook 'after-change-functions 'my-after)

if I activate the chinese-py input method, then inserting any
character via the input method produces a single call to my-before and
a single call to my-after, and with the expected values, for example:

  buf *scratch* beg 440 end 440
  buf *scratch* beg 440 end 441 len 0

So what exactly is the problem with these hooks when non-latin input
methods are used?  Or what am I missing?

> > IOW, why invalidating the cache unnecessarily is such a big deal?
> 
> It is critical for Org to know which part of buffer was changed (i.e.
> beg, end, length that are normally passed as arguments of
> after-change-functions). org-element-cache can contain >100k elements
> for especially large buffers. Manually checking which elements are
> changed without knowing the changed region is inefficient. Clearing the
> cache is not too much of a big deal, but causes slowdown when user runs
> a query on Org buffers (i.e. in agenda or sparse trees) - the buffer has
> to be re-parsed. When every edit triggers cache invalidation (that's
> what happens when user uses non-latin input method), the slowdown is
> pretty much guaranteed. Moreover, too frequent cache resets increase the
> load on Emacs' garbage collector (cache size is typically a multiple of
> buffer size). Overloading garbage collector leads to overall Emacs
> slowdown.

Perhaps Org developers should ask for infrastructure changes that will
allow Org to maintain such a cache reliably and not too expensively?
It sounds like Org currently applies all kinds of heuristics based on
assumptions about how the internals work and using hooks and features
that were never designed to support this kind of caching.  Jumping
through hoops in Lisp trying to implement something that might be much
easier or even trivial in C is not the best way of getting such jobs
done.

So perhaps someone could describe on emacs-devel what does Org need to
maintain this cache, and we could then see how to provide those
features to Org.

> >> The hooks are called, but after quail already triggered
> >> buffer-chars-modified-tick increase. If quail called
> >> before-change-functions before buffer-chars-modified-tick increases, it
> >> would be useful for my scenario. Though I am not sure how feasible it
> >> is. Just an idea.
> >
> > Would it help if Org looked at both buffer-modified-tick and
> > buffer-chars-modified-tick?
> 
> When buffer-chars-modified-tick is changed, buffer-modified-tick is also
> changed. AFAIU, buffer-chars-modified-tick registers a subset of buffer
> modifications that actually change buffer text. buffer-modified-tick
> also registers text property changes. So, I do not see how
> buffer-modified-tick can help.

If you look at the implementation, you will see that when Emacs
decides that the buffer's chars-modified-tick needs to be increased,
it simply assigns to it the value of the buffer's modified-tick at
that moment.  So by tracking the value of buffer-modified-tick you
could perhaps explain why buffer-chars-modified-tick jumps by more
than you expected.