bug#51766: 29.0.50; Return value of buffer-chars-modified-tick changes when buffer text is not yet changed before inserting a character for non-latin input methods

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

> "Such change" being what exactly? the situation where
> buffer-chars-modified-tick changes between post-command-hook and the
> following pre-command-hook? or something else?

The former.

> So what exactly is the problem with these hooks when non-latin input
> methods are used?  Or what am I missing?

There is no problem with the hooks in your example. However, consider
the following:

(let ((inhibit-modification-hooks t))
  (insert "Insertion that will never trigger before/after-change-functions"))

Org cache is bound to track all the changes in buffer. Any missed change
will lead to cache corruption. So, situations like the above must be
tracked somehow. This tracking can be done using
buffer-chars-modified-tick or buffer-hash/secure-hash. The latter is too
slow.

If I understand your earlier explanation correctly, quail for non-latin
input (I tested with russian-computer) does something like

(let ((inhibit-modification-hooks t))
  (insert ?char)
  (backward-delete-char)) ;; This increases buffer-chars-modified-tick
(insert ?translated_char_according_to_input_method)

The change hooks will only be called for the last insertion. However,
the first insertion+deletion will change buffer-chars-modified-tick.

The quail's insertion+deletion itself is not a problem for Org cache -
it does not really alter the buffer text and cannot break the cache. The
problem is that it cannot be distinguished from the first example - both
cases will trigger buffer-chars-modified-tick increase.

> Perhaps Org developers should ask for infrastructure changes that will
> allow Org to maintain such a cache reliably and not too expensively?
> It sounds like Org currently applies all kinds of heuristics based on
> assumptions about how the internals work and using hooks and features
> that were never designed to support this kind of caching.  Jumping
> through hoops in Lisp trying to implement something that might be much
> easier or even trivial in C is not the best way of getting such jobs
> done.
>
> So perhaps someone could describe on emacs-devel what does Org need to
> maintain this cache, and we could then see how to provide those
> features to Org.

I am one of the Org developers.

The only assumption I had it that Emacs does not frequently change
buffer text without triggering modification hooks. Clearly, the
assumption was wrong.

Ideally, a way to track _all_ buffer modifications regardless of
inhibit-modification-hooks would be useful. Currently, Org has to work
around the possibilities that text can be inserted without triggering
modification hooks: (1) when
inhibit-modification-hooks/before-change-functions/after-change-functions
are let-bound to nil; (2) when changes are made in indirect buffer with
different buffer-local values of before/after-change-functions.

Alternatively, Emacs could support language parsers. Org cache
implements editing syntax tree generated by Org element parser. It is
very similar to what tree-sitter editing API does: 
https://tree-sitter.github.io/tree-sitter/using-parsers#editing

Native support for storing, modifying, and querying syntax trees using
efficient data structures could be a great addition to Emacs from Org's
perspective. Though it is not an easy feature to implement.

AFAIR, something similar to my last suggestion has been already
proposed: tree-sitter support. I can also propose the first idea about
reliable buffer change tracking if you think that it is something
reasonable.

Best,
Ihor