bug#69431: 30.0.50; Strange fontificaion behavior

[Andrea Corallo]

Andrea Corallo <acorallo@gnu.org> writes:

> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>>>>> I'm trying to progress on this but I've difficult time at getting a grip
>>>>> on this bug.  One of the reason why is that on my reproducer Emacs
>>>>> doesn't crash nor complain, the other reason is that I'm not really into
>>>>> our fontification.  I'll keep on looking into it but I'd appretiate any
>>>>> hint like: what should be happening to fontify the buffer that is not
>>>>> actually happening here?  With such info I could try investingating the
>>>>> issue from there comparing with the working case.
>>>>
>>>> Hmm... here, after
>>>>
>>>>     rm -rf ~/.emacs.d/eln-cache
>>>>     src/emacs -Q --eval '(setq debug-on-error t)' a.org
>>>>
>>>> I get a buffer with no fontification, that claims to be in Org mode but
>>>> whose `font-lock-keywords` is just (t nil), which explains the lack
>>>> of fontification.
>>>
>>> Where `font-lock-keywords` are supposed to be set?
>>
>> `font-lock-keywords` is normally set from `font-lock-defaults` (which is
>> set by the major-mode) by `font-lock-set-defaults` when `font-lock-mode`
>> is enabled (which usually happens soon after enabling the major mode,
>> via `after-change-major-mode-hook`).
>
> Sorry didn't had much time to progress on this.
>
> I see `font-lock-mode` and `font-lock-set-defaults` are called when
> opening an org file here on my reproducer.
>
> Also I see `font-lock-set-defaults` is actually set! So I'm wondering
> why afterwards the value changes for my test.org.
>
> Will look into it more hopefully tomorrow.

Driven by curiosity I went a little further today...

I've applied the following:

================

1 file changed, 6 insertions(+)
lisp/font-lock.el | 6 ++++++

modified   lisp/font-lock.el
@@ -1874,6 +1874,10 @@ font-lock-refresh-defaults
 (defvar-local font-lock-major-mode nil
   "Major mode for which the font-lock settings have been setup.")

+(defun k-variable-watcher (_symbol newval op _buffer)
+  (message "XXX %s" op)
+  (backtrace))
+
 (defun font-lock-set-defaults ()
   "Set fontification defaults appropriately for this mode.
 Sets various variables using `font-lock-defaults' and
@@ -1934,6 +1938,8 @@ font-lock-set-defaults
       (unless (eq (car font-lock-keywords) t)
         (setq font-lock-keywords
               (font-lock-compile-keywords font-lock-keywords))))
+    (when (string= (buffer-name) "test.org")
+      (add-variable-watcher 'font-lock-keywords #'k-variable-watcher))
     (font-lock-flush)))
 ^L
 ;;; Color etc. support.

================

And this is what it logs on *Messages* depending on the fact that the
eln-cache is wiped or not:

Warm eln-cache (working fontification)
================
For information about GNU Emacs and the GNU system, type C-h C-a.
XXX set
  backtrace()
  k-variable-watcher(font-lock-keywords ((org-font-lock-hook) 
("^\\(\\**\\)\\(\\* \\)\\(.*\\)" (1 (org-get-level-face 1)) (2 
(org-get-level-face 2)) (3 (org-get-level-face 3))) ("^[ 
\11]*\\(\\(|\\|\\+-[-+]\\).*\\S-\\)" (1 'org-table t)) ("^[ \11]*|\\(?:.*?|\\)? 
*\\(:?=[^|\n]*\\)" (1 'org-formula t)) ("^[ \11]*| *\\([#*]\\) *|" (1 
'org-formula t)) ("^[ \11]*|\\( *\\([$!_^/]\\) *|.*\\)|" (1 'org-formula t)) 
("| *\\(<[lrc]?[0-9]*>\\)" (1 'org-formula t)) ("^\\(?4:[ 
\11]*\\)\\(?1::\\(?2:\\S-+\\):\\)\\(?:\\(?3:$\\)\\|[ 
\11]+\\(?3:.*?\\)\\)\\(?5:[ \11]*\\)$" (1 'org-special-keyword t) (3 
'org-property-value t)) (org-fontify-drawers) (org-activate-links) 
(org-activate-tags (1 'org-tag prepend)) (org-activate-target-links (1 
'org-link t)) (org-activate-dates (0 'org-date t)) 
(org-activate-footnote-links)\
 ("<<<\\([^<>\n\15 \11]\\|[^<>\n\15 \11][^<>\n\15]*[^<>\n\15 \11]\\)>>>" (0 
'org-target t)) ("<<\\([^<>\n\15 \11]\\|[^<>\n\15 \11][^<>\n\15]*[^<>\n\15 
\11]\\)>>" (0 'org-target t)) ("^&?%%(.*\\|<%%([^>\n]*?>" (0 'org-sexp-date t)) 
(org-fontify-macros) ("^\\(\\*+\\)\\(?: +\\(DONE\\|TODO\\)\\)\\(?: 
+\\(.*?\\)\\)?[ \11]*$" (2 (org-get-todo-face 2) prepend)) ("^\\(\\*+\\)\\(?: 
+\\(?:DONE\\)\\)\\(?: +\\(.*?\\)\\)?[ \11]*$" (2 'org-headline-done prepend)) 
(org-font-lock-add-priority-faces) (org-font-lock-add-tag-faces) 
("\\<DEADLINE:" (0 'org-special-keyword t)) ("\\<SCHEDULED:" (0 
'org-special-keyword t)) ("\\<CLOSED:" (0 'org-special-keyword t)) ("\\<CLOCK:" 
(0 'org-special-keyword t)) (org-do-emphasis-faces) ("^[ 
\11]*\\(?:[-+*]\\|[0-9]+[.)]\\)[ \11]+\\(?:\\[@\\(?:start:\\)?[0-9]+\\][ 
\11]*\\)?\\(\\\
[[- X]\\]\\)" 1 'org-checkbox prepend) 
("\\[\\([0-9]*%\\)\\]\\|\\[\\([0-9]*\\)/\\([0-9]*\\)\\]" (0 
(org-get-checkbox-statistics-face) prepend)) ("\\(?:^[ \11]*[-+]\\|^[ 
\11]+[*]\\)[ \11]+\\(.*?[ \11]+::\\)\\([ \11]+\\|$\\)" 1 'org-list-dt prepend) 
("\\(@@\\)\\([a-z-]+:\\).*?\\(@@\\)" (1 'font-lock-comment-face t) (2 'org-tag 
t) (3 'font-lock-comment-face t)) ("^\\*+ \\(.*:ARCHIVE:.*\\)" (1 'org-archived 
prepend)) (org-do-latex-and-related) (org-fontify-entities) (org-raise-scripts) 
(org-activate-code (1 'org-code t)) ("^\\*+\\(?: +\\(DONE\\|TODO\\)\\)?\\(?: 
+\\[#[A-Z0-9]\\]\\)? +\\(?9:COMMENT\\)\\(?: \\|$\\)" (9 'org-special-keyword 
t)) (org-fontify-meta-lines-and-blocks) (org-fontify-inline-src-blocks) 
(org-cite-activate)) set #<buffer test.org>)
  normal-mode(t)
  after-find-file(nil t)
  find-file-noselect-1(#<buffer test.org> "~/test.org" nil nil "~/test.org" 
(14180533 64513))
  find-file-noselect("/home/andcor03/test.org")
  command-line-1(("-eval" "(setq native-comp-jit-compilation t)" 
"/home/andcor03/test.org"))
  command-line()
  normal-top-level()

XXX set
  backtrace()
  k-variable-watcher(font-lock-keywords (t ((org-font-lock-hook) 
("^\\(\\**\\)\\(\\* \\)\\(.*\\)" (1 (org-get-level-face 1)) (2 
(org-get-level-face 2)) (3 (org-get-level-face 3))) ("^[ 
\11]*\\(\\(|\\|\\+-[-+]\\).*\\S-\\)" (1 'org-table t)) ("^[ \11]*|\\(?:.*?|\\)? 
*\\(:?=[^|\n]*\\)" (1 'org-formula t)) ("^[ \11]*| *\\([#*]\\) *|" (1 
'org-formula t)) ("^[ \11]*|\\( *\\([$!_^/]\\) *|.*\\)|" (1 'org-formula t)) 
("| *\\(<[lrc]?[0-9]*>\\)" (1 'org-formula t)) ("^\\(?4:[ 
\11]*\\)\\(?1::\\(?2:\\S-+\\):\\)\\(?:\\(?3:$\\)\\|[ 
\11]+\\(?3:.*?\\)\\)\\(?5:[ \11]*\\)$" (1 'org-special-keyword t) (3 
'org-property-value t)) (org-fontify-drawers) (org-activate-links) 
(org-activate-tags (1 'org-tag prepend)) (org-activate-target-links (1 
'org-link t)) (org-activate-dates (0 'org-date t)) (org-activate-footnote-lin\
ks) ("<<<\\([^<>\n\15 \11]\\|[^<>\n\15 \11][^<>\n\15]*[^<>\n\15 \11]\\)>>>" (0 
'org-target t)) ("<<\\([^<>\n\15 \11]\\|[^<>\n\15 \11][^<>\n\15]*[^<>\n\15 
\11]\\)>>" (0 'org-target t)) ("^&?%%(.*\\|<%%([^>\n]*?>" (0 'org-sexp-date t)) 
(org-fontify-macros) ("^\\(\\*+\\)\\(?: +\\(DONE\\|TODO\\)\\)\\(?: 
+\\(.*?\\)\\)?[ \11]*$" (2 (org-get-todo-face 2) prepend)) ("^\\(\\*+\\)\\(?: 
+\\(?:DONE\\)\\)\\(?: +\\(.*?\\)\\)?[ \11]*$" (2 'org-headline-done prepend)) 
(org-font-lock-add-priority-faces) (org-font-lock-add-tag-faces) 
("\\<DEADLINE:" (0 'org-special-keyword t)) ("\\<SCHEDULED:" (0 
'org-special-keyword t)) ("\\<CLOSED:" (0 'org-special-keyword t)) ("\\<CLOCK:" 
(0 'org-special-keyword t)) (org-do-emphasis-faces) ("^[ 
\11]*\\(?:[-+*]\\|[0-9]+[.)]\\)[ \11]+\\(?:\\[@\\(?:start:\\)?[0-9]+\\][ 
\11]*\\)?\\\
(\\[[- X]\\]\\)" 1 'org-checkbox prepend) 
("\\[\\([0-9]*%\\)\\]\\|\\[\\([0-9]*\\)/\\([0-9]*\\)\\]" (0 
(org-get-checkbox-statistics-face) prepend)) ("\\(?:^[ \11]*[-+]\\|^[ 
\11]+[*]\\)[ \11]+\\(.*?[ \11]+::\\)\\([ \11]+\\|$\\)" 1 'org-list-dt prepend) 
("\\(@@\\)\\([a-z-]+:\\).*?\\(@@\\)" (1 'font-lock-comment-face t) (2 'org-tag 
t) (3 'font-lock-comment-face t)) ("^\\*+ \\(.*:ARCHIVE:.*\\)" (1 'org-archived 
prepend)) (org-do-latex-and-related) (org-fontify-entities) (org-raise-scripts) 
(org-activate-code (1 'org-code t)) ("^\\*+\\(?: +\\(DONE\\|TODO\\)\\)?\\(?: 
+\\[#[A-Z0-9]\\]\\)? +\\(?9:COMMENT\\)\\(?: \\|$\\)" (9 'org-special-keyword 
t)) (org-fontify-meta-lines-and-blocks) (org-fontify-inline-src-blocks) 
(org-cite-activate)) (org-font-lock-hook (0 nil)) ("^\\(\\**\\)\\(\\* 
\\)\\(.*\\)" (1 (org-\
get-level-face 1)) (2 (org-get-level-face 2)) (3 (org-get-level-face 3))) ("^[ 
\11]*\\(\\(|\\|\\+-[-+]\\).*\\S-\\)" (1 'org-table t)) ("^[ \11]*|\\(?:.*?|\\)? 
*\\(:?=[^|\n]*\\)" (1 'org-formula t)) ("^[ \11]*| *\\([#*]\\) *|" (1 
'org-formula t)) ("^[ \11]*|\\( *\\([$!_^/]\\) *|.*\\)|" (1 'org-formula t)) 
("| *\\(<[lrc]?[0-9]*>\\)" (1 'org-formula t)) ("^\\(?4:[ 
\11]*\\)\\(?1::\\(?2:\\S-+\\):\\)\\(?:\\(?3:$\\)\\|[ 
\11]+\\(?3:.*?\\)\\)\\(?5:[ \11]*\\)$" (1 'org-special-keyword t) (3 
'org-property-value t)) (org-fontify-drawers (0 nil)) (org-activate-links (0 
nil)) (org-activate-tags (1 'org-tag prepend)) (org-activate-target-links (1 
'org-link t)) (org-activate-dates (0 'org-date t)) (org-activate-footnote-links 
(0 nil)) ("<<<\\([^<>\n\15 \11]\\|[^<>\n\15 \11][^<>\n\15]*[^<>\n\15 
\11]\\)>>>" (0 'or\
g-target t)) ("<<\\([^<>\n\15 \11]\\|[^<>\n\15 \11][^<>\n\15]*[^<>\n\15 
\11]\\)>>" (0 'org-target t)) ("^&?%%(.*\\|<%%([^>\n]*?>" (0 'org-sexp-date t)) 
(org-fontify-macros (0 nil)) ("^\\(\\*+\\)\\(?: +\\(DONE\\|TODO\\)\\)\\(?: 
+\\(.*?\\)\\)?[ \11]*$" (2 (org-get-todo-face 2) prepend)) ("^\\(\\*+\\)\\(?: 
+\\(?:DONE\\)\\)\\(?: +\\(.*?\\)\\)?[ \11]*$" (2 'org-headline-done prepend)) 
(org-font-lock-add-priority-faces (0 nil)) (org-font-lock-add-tag-faces (0 
nil)) ("\\<DEADLINE:" (0 'org-special-keyword t)) ("\\<SCHEDULED:" (0 
'org-special-keyword t)) ("\\<CLOSED:" (0 'org-special-keyword t)) ("\\<CLOCK:" 
(0 'org-special-keyword t)) (org-do-emphasis-faces (0 nil)) ("^[ 
\11]*\\(?:[-+*]\\|[0-9]+[.)]\\)[ \11]+\\(?:\\[@\\(?:start:\\)?[0-9]+\\][ 
\11]*\\)?\\(\\[[- X]\\]\\)" (1 'org-checkbox prepend)) ("\\[\\
\([0-9]*%\\)\\]\\|\\[\\([0-9]*\\)/\\([0-9]*\\)\\]" (0 
(org-get-checkbox-statistics-face) prepend)) ("\\(?:^[ \11]*[-+]\\|^[ 
\11]+[*]\\)[ \11]+\\(.*?[ \11]+::\\)\\([ \11]+\\|$\\)" (1 'org-list-dt 
prepend)) ("\\(@@\\)\\([a-z-]+:\\).*?\\(@@\\)" (1 'font-lock-comment-face t) (2 
'org-tag t) (3 'font-lock-comment-face t)) ("^\\*+ \\(.*:ARCHIVE:.*\\)" (1 
'org-archived prepend)) (org-do-latex-and-related (0 nil)) 
(org-fontify-entities (0 nil)) (org-raise-scripts (0 nil)) (org-activate-code 
(1 'org-code t)) ("^\\*+\\(?: +\\(DONE\\|TODO\\)\\)?\\(?: +\\[#[A-Z0-9]\\]\\)? 
+\\(?9:COMMENT\\)\\(?: \\|$\\)" (9 'org-special-keyword t)) 
(org-fontify-meta-lines-and-blocks (0 nil)) (org-fontify-inline-src-blocks (0 
nil)) (org-cite-activate (0 nil))) set #<buffer test.org>)
  font-lock-fontify-keywords-region(1 7 nil)
  font-lock-default-fontify-region(1 7 nil)
  font-lock-fontify-region(1 7)
  #f(compiled-function (fun) #<bytecode 
0x1baf782d70c71cbf>)(font-lock-fontify-region)
  jit-lock--run-functions(1 7)
  jit-lock-fontify-now(1 7)
  jit-lock-function(1)
  redisplay_internal\ \(C\ function\)()
================


Empty eln-cache (fontification broken):
================
For information about GNU Emacs and the GNU system, type C-h C-a.
XXX makunbound
  backtrace()
  k-variable-watcher(font-lock-keywords nil makunbound #<buffer test.org>)
  kill-local-variable(font-lock-keywords)
  org-set-font-lock-defaults()
  org-mode()
  set-auto-mode-0(org-mode nil)
  set-auto-mode--apply-alist((("\\.gpg\\(~\\|\\.~[0-9]+~\\)?\\'" nil epa-file) 
("\\.elc\\'" . elisp-byte-code-mode) ("\\.zst\\'" nil jka-compr) ("\\.dz\\'" 
nil jka-compr) ("\\.xz\\'" nil jka-compr) ("\\.lzma\\'" nil jka-compr) 
("\\.lz\\'" nil jka-compr) ("\\.g?z\\'" nil jka-compr) ("\\.bz2\\'" nil 
jka-compr) ("\\.Z\\'" nil jka-compr) ("\\.vr[hi]?\\'" . vera-mode) 
("\\(?:\\.\\(?:rbw?\\|ru\\|rake\\|thor\\|axlsx\\|jbuilder\\|rabl\\|gemspec\\|podspec\\)\\|/\\(?:Gem\\|Rake\\|Cap\\|Thor\\|Puppet\\|Berks\\|Brew\\|Fast\\|Vagrant\\|Guard\\|Pod\\)file\\)\\'"
 . ruby-mode) ("\\.re?st\\'" . rst-mode) ("/\\(?:Pipfile\\|\\.?flake8\\)\\'" . 
conf-mode) ("\\.py[iw]?\\'" . python-mode) ("\\.m\\'" . octave-maybe-mode) 
("\\.less\\'" . less-css-mode) ("\\.scss\\'" . scss-mode) ("\\.cs\\'" . 
csharp-mode) ("\\.awk\\'" .\
 awk-mode) ("\\.\\(u?lpc\\|pike\\|pmod\\(\\.in\\)?\\)\\'" . pike-mode) 
("\\.idl\\'" . idl-mode) ("\\.java\\'" . java-mode) ("\\.m\\'" . objc-mode) 
("\\.ii\\'" . c++-mode) ("\\.i\\'" . c-mode) ("\\.lex\\'" . c-mode) 
("\\.y\\(acc\\)?\\'" . c-mode) ("\\.h\\'" . c-or-c++-mode) ("\\.c\\'" . c-mode) 
("\\.\\(CC?\\|HH?\\)\\'" . c++-mode) ("\\.[ch]\\(pp\\|xx\\|\\+\\+\\)\\'" . 
c++-mode) ("\\.\\(cc\\|hh\\)\\'" . c++-mode) ("\\.\\(bat\\|cmd\\)\\'" . 
bat-mode) ("\\.[sx]?html?\\(\\.[a-zA-Z_]+\\)?\\'" . mhtml-mode) ("\\.svgz?\\'" 
. image-mode) ("\\.svgz?\\'" . xml-mode) ("\\.x[bp]m\\'" . image-mode) 
("\\.x[bp]m\\'" . c-mode) ("\\.p[bpgn]m\\'" . image-mode) ("\\.tiff?\\'" . 
image-mode) ("\\.gif\\'" . image-mode) ("\\.png\\'" . image-mode) 
("\\.jpe?g\\'" . image-mode) ("\\.webp\\'" . image-mode) ("\\.te?xt\\'" . \
text-mode) ("\\.[tT]e[xX]\\'" . tex-mode) ("\\.ins\\'" . tex-mode) ("\\.ltx\\'" 
. latex-mode) ("\\.dtx\\'" . doctex-mode) ...) nil nil)
  set-auto-mode()
  normal-mode(t)
  after-find-file(nil t)
  find-file-noselect-1(#<buffer test.org> "~/test.org" nil nil "~/test.org" 
(14180533 64513))
  find-file-noselect("/home/andcor03/test.org")
  command-line-1(("-eval" "(setq native-comp-jit-compilation t)" 
"/home/andcor03/test.org"))
  command-line()
  normal-top-level()

XXX set
  backtrace()
  k-variable-watcher(font-lock-keywords (t nil) set nil)
  font-lock-fontify-keywords-region(1 7 nil)
  font-lock-default-fontify-region(1 7 nil)
  font-lock-fontify-region(1 7)
  #f(compiled-function (fun) #<bytecode 
0x1baf3a4708041cbf>)(font-lock-fontify-region)
  jit-lock--run-functions(1 7)
  jit-lock-fontify-now(1 7)
  jit-lock-function(1)
  redisplay_internal\ \(C\ function\)()

================

So from what I see in the non working example 'font-lock-keywords' is
cleared at the end of 'org-set-font-lock-defaults' which does
'(kill-local-variable 'font-lock-keywords)'.

Now why this is not happening in the working example I'm not sure ATM,
('org-set-font-lock-defaults' is not even called there AFAICS).

Maybe org maintainers can comment if this is expected or not, or explain
meanwhile the mechanism so debug will be easier.

Thanks!

  Andrea