Re: Message Mode and bidi

[Eric Abrahamsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Eric Abrahamsen <eric@ericabrahamsen.net>
>> Date: Mon, 19 Feb 2024 21:16:51 -0800
>> 
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> From: Eric Abrahamsen <eric@ericabrahamsen.net>
>> >> Date: Mon, 19 Feb 2024 16:46:22 -0800
>> >> 
>> >> Christopher Culver via "Emacs development discussions."
>> >> <emacs-devel@gnu.org> writes:
>> >> 
>> >> > Joost Kremers <joostkremers@fastmail.fm> writes:
>> >> >> When you compose a new message, is there a line "--text follows this 
>> >> >> line--"
>> >> >> separating the headers and the message text? In my case, there is (I 
>> >> >> use mu4e)
>> >> >> and when I type Arabic text on the line below this text, I get the 
>> >> >> effect you
>> >> >> mention. If I leave an empty line after "--text follows this line--", 
>> >> >> bidi works
>> >> >> as expected.
>> >> >
>> >> > Indeed, if I just go down one line and then begin typing, bidi works as
>> >> > expected. I am feeling very foolish that I did not even try this. Thank
>> >> > you for clearing up this problem, and for shedding light on how Emacs
>> >> > considers paragraphs.
>> >> 
>> >> This is a bit weird, because the value of `mail-header-separator'
>> >> ("--text follows this line--") is added to both `paragraph-start' and
>> >> `paragraph-separate') in `message-mode'. You'd think one of those would
>> >> do it.
>> >
>> > Emacs doesn't use paragraph-separate and paragraph-start to define
>> > where a paragraph starts and ends, for the purposes of determining the
>> > base directionality of a paragraph.  It uses separate variables for
>> > that, see the node "Bidirectional Editing" in the Emacs user manual.
>> > The reason for using separate variables is because several modes,
>> > including (but not limited to) message-mode, set the former variables
>> > to regexps that get in the way of bidi reordering, and could easily
>> > produce wrong results on display.
>> 
>> Do you think we'd stand a chance of finding values for
>> bidi-paragraph-start|separate-re that would resolve this particular
>> issue?
>
> We already have those values described in the Emacs manual:
>
>      Each paragraph of bidirectional text can have its own “base
>   direction”, either right-to-left or left-to-right.  Text in
>   left-to-right paragraphs begins on the screen at the left margin of the
>   window and is truncated or continued when it reaches the right margin.
>   By contrast, text in right-to-left paragraphs is displayed starting at
>   the right margin and is continued or truncated at the left margin.  By
>   default, paragraph boundaries are empty lines, i.e., lines consisting
>   entirely of whitespace characters.  To change that, you can customize
>   the two variables ‘bidi-paragraph-start-re’ and
>   ‘bidi-paragraph-separate-re’, whose values should be regular expressions
>   (strings); e.g., to have a single newline start a new paragraph, set
>                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   both of these variables to ‘"^"’.  These two variables are buffer-local
>   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   (*note Locals::).
> 
> But beware: doing this in Emacs will cause effects that are unpleasant
> to readers of bidirectional text, because you could have "chess-like"
> text display, like this:
>
> asasasasasasasasasasasasassa
>                                                 ASASASASASASASASASASASASA
> xcxcxcxcxcxcxcxcxcxcxcxcxc
>                                                        JKJNJKKKNKNKNKNKNK
>
> etc.  Here upper-case letters stand for RTL (like Arabic or Farsi)
> text and lower-case letters stand for LTR (like Latin or Cyrillic)
> text.

This is the effect I was imagining in the header section of the message
buffer, and why I thought we should probably skip trying to handle this.
message-mode headers can be "continuation headers", as well, effectively
line-wrapped, which would make it even harder.

> Let me explain.  UBA, the Unicode Bidirectional Algorithm which Emacs
> implements, was designed for text-processing programs that wrap long
> lines without inserting hard newlines.  For those applications, a
> single hard newline indicates a new paragraph, and thus recalculating
> the base direction of a paragraph after a newline is justified.
>
> But in Emacs, we have a lot of text filled and wrapped using hard
> newlines, and filling can insert a newline in an arbitrary place
> inside the paragraph.  If it happens that a newline was inserted
> before a strong right-to-left character, under the strict UBA rules
> the next line will be considered as a paragraph with right-to-left
> base direction, and rendered starting at the right window edge.  Which
> leads to the above "chess-like" display.  Since that is basically
> unacceptable for Emacs users, we require at least one empty line to
> separate paragraphs.  The price of a single empty line before the body
> of an email message that needs to be rendered right to left is a small
> price to pay for solving the horrible display effect of changing the
> paragraph's base direction after each newline.  An alternative would
> be to use visual-line-mode instead of wrapping using hard newlines,
> but that is still relatively rare in Emacs, especially in email
> messages.

Thanks for the background! I guess I was hoping that we could at least
support OP's original request, which is making the first paragraph of
the message body independent of the mail header separator as regards
BIDI display. I experimented with putting the value of
`mail-header-separator' into `bidi-paragraph-start|separate-re', but
couldn't get it to display that first paragraph starting on the right.
Do you think this is feasible, and worth the effort?