bug#22761: Surprising interaction between font-lock, invisible text, and point (self-insert-command and insert behave differently)

[Clément Pit--Claudel]

Hi,

I'm running into problems using font-lock to display certain strings as 
subscripts. The basic goal is to display ‘a__b’ as ‘ab’ with ‘b’ vertically 
offset. Using font-lock to make ‘__’ invisible and to add (display (raise 
-0.15)) to ‘b’ works fine. There is a strange interaction between 
self-insert-command and font-lock, however, and the problem does not happen if 
using ‘insert’ instead of ‘self-insert-command’.

The problem happens when editing ‘a_foo_b’ to replace ‘foo’ by ‘bar’ (to get 
‘a_bar_b’). Removing ‘foo’ leaves ‘__’ in the buffer, which font-lock 
recognizes: the whole string gets displayed as ‘ab’. Even then, the point is 
still between the two underscores (it hasn't moved after font-lock added the 
invisible property to the underscores). Pressing ‘b’ to start inserting ‘bar’ 
works fine (I end up with ‘a_b_c’), except for one thing: the point gets moved 
after the second underscore. Thus, pressing ‘ar’ to complete ‘bar’ ends up 
inserting ‘a_b_arb’.

On the other hand, using ‘M-: (insert ?b)’ to insert the ‘b’ of ‘bar’ leaves 
the point in the right place.

To reproduce:

1. Open a buffer in fundamental-mode
2. Evaluate the following setup code:
    (progn
      (setq font-lock-defaults '(nil))
      (font-lock-add-keywords nil `((,(concat "[a-z]+\\(__\\)\\([a-z]+\\)")
                                     (1 '(face nil invisible 'subscript))
                                     (2 '(face nil display (raise -0.25))))))
      (add-to-invisibility-spec 'subscript)
      (make-local-variable 'font-lock-extra-managed-props)
      (add-to-list 'font-lock-extra-managed-props 'display)
      (add-to-list 'font-lock-extra-managed-props 'invisible)
      (font-lock-mode))
3. Insert the following text: before_between_after
4. Place the point after ‘between’; press <backspace> 7 times, to remove 
‘between’ entirely.
5. Type ‘between’ (using the key sequence b e t w e e n)

Expected result: buffer contains before_between_after
Actual result: buffer contains before_b_etweenafter

On the other hand, the following protocol works fine:

1-4. Same as before
5. ‘M-: (insert ?b)’
6. Type ‘etween’ (using the key sequence e t w e e n)

Expected result: buffer contains before_between_after
Actual result: buffer contains before_between_after

Cheers,
Clément.

signature.asc
Description: OpenPGP digital signature