bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Fri, 05 Aug 2022 11:50:56 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: gerd.moellmann@gmail.com, 56682@debbugs.gnu.org, 
> monnier@iro.umontreal.ca, 
>     dgutov@yandex.ru
> 
> > There's (at least) one more aspect of this, as long as Text mode is 
> > being used: Text mode doesn't force bidi-paragraph-direction to be 
> > left-to-right, whereas all descendants of prog-mode, including js-mode, 
> > do.  Leaving bidi-paragraph-direction at nil means Emacs needs to 
> > determine the base paragraph direction each time it's about to redisplay 
> > a window, and that might be expensive, especially in a large buffer 
> > without any paragraph breaks (by default, an empty line), because that 
> > is determined by the first strong directional character of the 
> > paragraph.  So for a more fair comparison with Text mode, you should set 
> > bidi-paragraph-direction to the value left-to-right in text-mode 
> > buffers.
> 
> Indeed, that seems to be the culprit here, I didn't know that text-mode 
> was an exception here.

Actually, it's the other way around: prog-mode and its descendants are
the exception.  The default value of bidi-paragraph-direction is nil,
but in prog-mode's we _know_ that program source code is written
left-to-right, even if some of strings and comments could include R2L
text, so we force bidi-paragraph-direction in these modes.

> If I set bidi-paragraph-direction to 'left-to-right after visiting
> the arabic-small.txt file, Emacs (mis) behaves like it does for the
> other Arabic files: it becomes slow, and C-n C-p do not work
> correctly anymore.

OK, so there's one more subtle issue about bidi, and it definitely
affects some of these files: whether the file actually includes both
L2R and R2L characters.  If all of the characters are L2R or all of
them are R2L (the latter is the case with arabic-small.txt), and
bidi-paragraph-direction is either nil or left-to-right for L2R text
and right-to-left for R2L text, then from the display engine's POV the
text is not truly bidirectional, it actually has a single direction.
Text that has a single direction doesn't require (expensive)
reordering for display, and we have optimizations in place for such
situations, to make redisplay faster in those cases, because that is
the natural situation with human-readable text: text that is
predominantly R2L is relatively rarely mixed with L2R text and is
normally displayed in right-to-left paragraphs.

By adding the "ar" prefix to the files, you made what was a
unidirectional text be bidirectional.  The effect on display is
dramatic: the first Arabic letter will now be displayed _last_,
because the paragraph now has left-to-right direction (the first
strong character in it is 'a', a L2R character), and the entire Arabic
string that follows needs to be reversed on display.

Compare arabic-small.txt with arabic-small.json after setting
bidi-paragraph-direction to be right-to-left in the latter, and you
should see a similar performance in both.  (And watch what happens on
display when you change the value of bidi-paragraph-direction in
arabic-small.json.)