Re: Major modes using `widen' is a good, even essential, programming practice.

[Stefan Monnier]

>> > The intent is clearly that fontifications don't look far beyond these
>> > two points, because otherwise the whole design of jit-lock and its
>> > invocations during redisplay is basically thrown out the window.
>> Usually, font-lock rules don't look before BOL or after EOL, indeed,
> But BOL and EOL could also be very far away, so that, too, needs some
> reasonable limit.

Indeed.  But in general this is easier to address, because the use of
BOL/EOL is itself a heuristic, so we can impose some other limit (as is
done now with the forced narrowing or with `syntax-wholeline-max`) and
the harm is usually tolerable, IME.

>> *except* via `syntax-ppss` which does look at all the text from BOB
>> to point.  To make up for that, `syntax-ppss` relies heavily on caching,
>> so that it *usually* doesn't need to look very far at all (and if
>> there's no `syntax-propertize-function`, it's usually quite fast
>> because it's fully coded in C).
>> 
>> For GB-sized buffers, even the fast C code of `syntax-ppss` incurs
>> a significant delay in the "unusual" case, so have various options:
>
> I reported this for a 18MB single-line file, which is way below the GB
> bar.

On my own builds (which are slow on old machines), 18MB typically
results in a delay of just a few seconds.  Given that it's occasional,
I find it very tolerable (similar delays can occur for other reasons,
such as swapping, or a GC when the heap is large).

But this scales linearly, so sooner or later on the way to multi-GB
files the delay becomes intolerable.

> The problem is that the initial full scan can take "forever" in those
> files, and that basically means we cannot edit such files in practice.
> So if you dislike the current solution of locked narrowing, how about
> making syntax-ppss work in chunks (perhaps from an idle timer?), after
> initially scanning only the first small portion of the file.  The goal
> is to have the file displayed quickly enough, and thereafter complete
> the scan when possible.

Yes, that's probably the better long term solution.


        Stefan