Re: Last steps for pretesting (font-lock-extend-region-function)

[Alan Mackenzie]

Good evening, Stefan!

On Fri, 21 Apr 2006, Stefan Monnier wrote:

>> Good morning, Stefan!

>Good morning Alan,

>> The font locking is triggered by the Change.  Therefore it needs details
>> of that change  to determine what region of text needs refontifying.  It

>No, it only needs to know what the current text looks like (obviously) and
>how the text was font-locked last time (to properly remove highlighting
>where it doesn't apply any more).  This last part can be obtained via the
>font-lock-multiline property.

OK.  I think you're right here, but I'm not 100% sure.

>> Perhaps I have truly misunderstood you.  I thought you were telling me
>> that some code contained within the major mode would do the
>> (put-text-property ... 'font-lock-multiline ...).  Are you really
>> saying that the major mode merely has to set up its font-lock-keywords
>> so that font-lock.el finds the right places to apply the text
>> property?

>I've generally been assuming that your code would explicitly do
>a put-text-property, but indeed font-lock.el includes some code which tries
>to do that for you.

It would do this, presumably, in an after-change (or before-change)
function, either directly on one of these hooks or called from
font-lock-after-change.

>>> IIUC, something like the following should do:

>>> (defun c-awk-font-lock-extend-region (beg end)
>>> (cons (c-awk-beginning-of-logical-line beg)
>>> (c-awk-end-of-logical-line end))) <==================================
>>> (setq font-lock-extend-region-function 'c-awk-font-lock-extend-region)

>> That, quite demonstrably, WON'T do, since that will fail to extend the
>> region to what was the end of the logical line before the change.

>Of course it's not enough: it only takes care of making sure current
>atomic elements are properly fontified, but not that previously atomic
>elements are properly refontified.  That's what the font-lock-multiline
>property is for.

OK.  So the font-lock-multiline property is put on exactly a region of
text which needs fontifying atomically, yet straddles a line break.  An
after-change function (and/or a before-change function) is what will do
this putting and erasing of f-l-multiline.

>>>>> Here is the problems I see with your proposal:
>>>>> - an `extend-region' hook in font-lock-fontify-region is needed

>>>> It is indeed.  It is a simple patch, and I am ready and willing to write
>>>> it, both for font-core.el and modes.texi.  I do first ask that it's
>>>> accepted in principle, though.

>>> It's completely accepted.  I just hope we can call it
>>> "font-lock-extend-region-function".

>> It has been accepted (by Richard, no less), as part of the
>> font-lock-after-change-function/jit-lock-after-change.  I'm proposing to
>> enhance it to do the necessary region extension from within
>> f-l-default-fontify-region/j-l-fontify-now too.

>I consider the two as separate: the first is already installed (but I want
>it removed) and the second is not installed yet, but we agree that it should
>be added.  I.e. I want it moved from a-c-f to f-l-d-f-r (it should not be
>added to j-l-fontify-now).

I think it's more accurate just to say you want to remove the
extend-region stuff from f-l-after-change.  I want it to stay.  :-)
Why don't we call the thing we need in f-l-d-f-r
"font-lock-extend-CHUNK-function", since it's more likely to be a
jit-lock chunk than a region supplied by the user or major mode?

What is your objection to f-l-extend-region-f in a-c-f?  Is it because it
might gobble too much processor time?

It needs to be in j-l-fontify-now, so that that function knows what bytes
to apply the 'fontified property to.  Some refactoring of the code might
be helpful - say, extracting `font-lock-core-fontify-region' from
font-lock-default-fontify-region, this new function fontifying exactly
the region it is given, and doing it immediately.

>>>>> - the font-lock-multiline property should be enough in all cases to
>>>>> make it unnecessary to use an after-change-function hook.

>>>> f-l-extend-region-function also makes a (separate) after-change-f hook
>>>> unnecessary.

>>> Which f-l-extend-region-function?  The one called from after-change-f?

>> Yes.

>>> Then it's what I meant by "an after-change-function hook".  The fact that
>>> it's not called directly from a-c-f but via f-t-a-c-f is not very
>>> important for this discussion.

>> I think it's important.  It saves the major mode maintainer from having
>> to install his code as a separate a-c-function, and juggling them around
>> to make sure they get called in the correct order.

>Yes it is important in general.  But this discussion is about the need to
>have *any* a-c-f hook.  If we need one, its place is in f-l-a-c-f, indeed,
>but I argue that we don't need one, so the precise place is not really
>relevant to the discussion.

I can't see how a major mode can manipulate f-l-multiline properties
anywhere BUT an a-c-f (or a before-c-f).

>>> The code will have probably the same cost whether it's called from
>>> a-c-f or from font-lock, but in one case it'll be called (much) more
>>> often.
>> Yes.  But that calling from a-c-f is essential to correct font
>> locking.

>That's what I claim is not true.

I can't see that yet, but I'm trying to.  I think you're saying that
instead of AWK Mode recording information in c-awk-old-EOLL ("c-awk old
end of logical line") in the before-change function, it would make some
setting of font-lock-multiline over the pertinent part of the buffer.

>> #########################################################################
>> In AWK Mode:

>> Point is at EOL 3, and we delete the backslash there.

>> 1. "string \
>> 2. over \
>> 3. several \       <========= point is at EOL 3, about to delete the \
>> 4. #lines."

>OK: if the text had never been font-locked before, a hook in
>f-l-fontify-region is all we need, right?  So the interesting case is
>when the text had already been fontified.  In this case, the whole
>multiline-line has had a font-lock-multiline added, so at this point in
>your example, the multiline-line is 100% covered by a
>font-lock-multiline property.

OK.

>> 1. In c-awk-before-change (see above for the code), we calculate
>> c-awk-old-EOLL ("end of old logical line").  This give 36, (EOL 4).

>Here let's say instead we don't do anything.

OK.

>> 2. The \ gets deleted, and the text looks like this:

>> 1. "string \
>> 2. over \
>> 3. several 
>> 4. #lines."

>OK, so the whole text from line 1 to line 4 still has a font-lock-multiline
>property.

OK.

>> 3. jit-lock-after-change gets called as (jit-lock-after-change 26 26 1),
>> which in its turn calls (c-awk-font-lock-extend-region 26 26 1).  This
>> "1" is the OLD-LEN, of course.

>Here let's also say that instead we don't do anything special.

OK.

>> 4. c-awk-f-l-e-r calls (c-awk-end-of-change-region 26 26 1).  This does
>> the following:
>> (i) It determines the current position of the PREVIOUS end of logical
>> line: (+ (- c-awk-old-EOLL old-len) (- end beg)),
>>       => (+ (- 36 1) (- 26 26))
>>       => 35
>> NOTE THE USE OF old-len.

>> (ii) It determines the current end of logical line:
>>     (c-awk-end-of-logical 26)
>>      => 26.  This is EOL 3

>> (iii) It selects the greater of these two positions:
>>     (max (+ (- c-awk-old-EOLL old-len) (- end beg))
>>          (c-awk-end-of-logical-line end))
>>     => (max 35 26)
>>     => 35.
>> This 35 is the current EOL 4.

>> (iv) 35 is returned to jit-lock-after-change in the cons (1 . 35), the
>> extended fontification region.
>> #########################################################################

>Here let's also say that instead we don't do anything special.

OK.

>But when we later get to f-l-fontify-region, the code looks at the
>font-lock-multiline property at the boundary (i.e. at the beginning and end
>of line 3) and noticies that it's set, so it extends the region to be
>fontified to span all 4 lines.  Just what you wanted all along.

OK, but .....

>Look ma!  No old-len!  Tadaaa!

.... but some code, somewhere, sometime needs to adjust the f-l-m
property, removing it from L4.  In general, it will also need to _split_
the text property into L1,2,3 and L4,5,...  Otherwise, a f-l-m region
could gradually engulf an entire buffer, making a nonsense of jit-lock.
(OK, this would be an extreme situation.  :-)

>If you insert text (instead of removing it), it gets a bit more
>interesting (because the inserted text doesn't have a
>font-lock-multiline property) but it basically works along the same
>lines.

Something, somewhere, sometime has to analyse the newly inserted code,
putting f-l-m properties throughout it, I think, and adjusting those at
the boundaries of the new region.  Is this to be done at the time of the
change or at the time it gets font locked?  I think it has to be done at
the change, otherwise the f-l-m settings will be wrong when it comes time
to fontify a chunk.

#########################################################################

SUMMARY:  

1. I think font-lock-extend-region-function, called as it is from
j-l-after-change, is a good thing.  It is a correct way of giving a major
mode an opportunity to massage a fontification region.  I think it should
stay.  You think it should be removed.

2. f-l-d-f-r / j-l-f-now absolutely need a
"font-lock-extend-chunk-function" functionality (regardless of its
precise name).  We agree on this. 

3. You have convinced me that it is possible to use the f-l-m properties.

4. I think that f-l-extend-region-f, called from f-l-after-change, is
likely to be more convenient and less error prone than the direct
manipulation of the f-l-multiline property by major mode code.

5. I suspect you think that f-l-extend-region-f is likely to consume more
processor time than direct manipulation of the f-l-m property. 

Is there any reason why we can't leave both methods side by side in the
code?

>        Stefan

BTW:  I'll be away from the net over the next couple of days.  ;-)

-- 
Alan.