bug#37467: 26.3; whitespace-mode doesn't update BOB/EOB highlights

[Dale Sedivec]

whitespace-mode doesn't seem to un-highlight empty lines at BOB/EOB if you 
insert non-white-space characters in the middle of a run of blank lines.

Steps to reproduce:

1. emacs -Q

2. M-x whitespace-mode RET in the *scratch* buffer you start in.

3. Type C-2 RET C-2 C-p.  The bottom three lines of the buffer are now blank 
lines, point is on the penultimate line, and whitespace-mode has thus correctly 
highlighted the three blank lines at EOB with face whitespace-empty.

4. Type x.

To be clear, the buffer contents are now as follows, with point at "|":

~~~~~~
;; This buffer is for text that is not saved, and for Lisp evaluation.
;; To create a file, visit it with C-x C-f and enter text in its buffer.

x|

~~~~~~

Expected results: The bottom line is highlighted by whitespace-mode, no other 
lines are highlighted.

Observed results: The bottom line is correctly highlighted, and the 
highlighting is cleared from line 4 with the "x", but line 3 remains 
highlighted with face whitespace-empty.

There is a second bug too:

5. Type <backspace> to delete the x.

Expected results: The bottom lines 3–5 are now highlighted with 
whitespace-empty.

Observed results: Only the bottom two blank lines 4–5 are highlighted with 
whitespace-empty.  The adjacent blank line 3 is not highlighted.

Note that you can get line 3 correctly re-highlighted by moving to EOB then up 
one line (M-> C-p).

I believe you can reproduce this same behavior at the beginning of the buffer 
as well, using similar steps.

These bugs may seem obscure, but I hit them surprisingly often at EOB when 
programming: I enter a few blank lines at EOB as I prepare to append code to 
the buffer, but then stop and move upwards to make some changes, before 
returning to near-EOB and trying to enter some lines, at which point I am left 
with some incorrect whitespace-mode highlights.

My first guess at fixing this (only at EOB so far, not BOB) was to use font 
lock multiline:

* I set the font-lock-multiline property non-nil in 
whitespace-font-lock-keywords for whitespace-empty-at-eob-regexp.

* I make whitespace-mode install a new buffer-local function to 
font-lock-extend-region-functions to correctly identify the region for 
fontification at EOB.  (Note that (whitespace-looking-back 
whitespace-empty-at-eob-regexp nil) was handy here.)

However, this only solved the first problem, removing highlighting as desired 
at the end of step 4, above.  I have not yet been able to solve the problem of 
re-highlighting line 3 at the end of my step 5.  whitespace-post-command-hook 
is doing some work to determine what to refontify, and understanding of that 
function has thus far eluded me.  I suspect a solution for one or both of these 
bugs lies in there, however.

Thanks for your consideration!

Dale


In GNU Emacs 26.3 (build 1, x86_64-apple-darwin18.7.0, NS appkit-1671.60 
Version 10.14.6 (Build 18G87))
of 2019-08-29 built on mojave.internal.macports.net
Windowing system distributor 'Apple', version 10.3.1671
Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.

Configured using:
'configure --prefix=/opt/local --without-dbus --without-gconf
--without-libotf --without-m17n-flt --without-gpm --with-gnutls
--with-xml2 --with-modules --infodir /opt/local/share/info/emacs
--with-ns --with-lcms2 'CFLAGS=-pipe -Os
-isysroot/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk
-arch x86_64' 'CPPFLAGS=-I/opt/local/include
-isysroot/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk'
'LDFLAGS=-L/opt/local/lib -Wl,-headerpad_max_install_names -Wl,-no_pie
-Wl,-syslibroot,/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk
-arch x86_64''

Configured features:
JPEG NOTIFY ACL GNUTLS LIBXML2 ZLIB TOOLKIT_SCROLL_BARS NS MODULES
THREADS LCMS2

Important settings:
  value of $LC_COLLATE: C
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Fundamental

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message rmc puny seq byte-opt gv
bytecomp byte-compile cconv cl-loaddefs cl-lib dired dired-loaddefs
format-spec rfc822 mml easymenu mml-sec password-cache epa derived epg
epg-config gnus-util rmail rmail-loaddefs mm-decode mm-bodies mm-encode
mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail rfc2047
rfc2045 ietf-drums mm-util mail-prsvr mail-utils elec-pair time-date
tooltip eldoc electric uniquify ediff-hook vc-hooks lisp-float-type
mwheel term/ns-win ns-win ucs-normalize mule-util term/common-win
tool-bar dnd fontset image regexp-opt fringe tabulated-list replace
newcomment text-mode elisp-mode lisp-mode prog-mode register page
menu-bar rfn-eshadow isearch timer select scroll-bar mouse jit-lock
font-lock syntax facemenu font-core term/tty-colors frame cl-generic
cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese composite charscript charprop
case-table epa-hook jka-cmpr-hook help simple abbrev obarray minibuffer
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote threads kqueue cocoa ns
lcms2 multi-tty make-network-process emacs)

Memory information:
((conses 16 203911 12801)
(symbols 48 20094 2)
(miscs 40 70 181)
(strings 32 28190 1477)
(string-bytes 1 764094)
(vectors 16 35016)
(vector-slots 8 723944 9708)
(floats 8 48 69)
(intervals 56 236 0)
(buffers 992 12))