Re: bidi-display-reordering is now non-nil by default

[Stephen J. Turnbull]

Eli Zaretskii writes:

 > They are not irrelevant.  What you suggest runs the risk of adding or
 > removing LRM/RLM characters to/from a file against user
 > expectations.

Sure, but byte-level equality is not part of that; character-level
equality is.

 > Again, what if the user inserts another LRM?

Insert another non-character "marker" in the buffer, using whichever
non-character strategy were using.

 > > What's wrong with reparsing the buffer from the beginning, treating
 > > each change of value of the direction property as insertion of the
 > > appropriate direction mark?
 > 
 > Reparsing the whole buffer upon each insertion?  Is that the way to
 > make redisplay fast and efficient?

No, that's a proof that it's *possible*, where your words claim it's
*im*possible.  Making it fast is a SMOP.  You say it's beyond you, and
that probably means it's beyond anybody competent enough in bidi to do
the implementation.  But let's not discourage anyone from trying. ;-)

 > How do you indicate them, exactly?  Emacs has no features, except
 > again text properties, to indicate something like that.  In any case,
 > isn't it beginning to sound more and more complicated?

Sure.  And the presence of non-graphic characters in the buffer is
going to make other code more complicated.  The question is, which
complexity is preferable?  You've made your choice, and Emacs has a
bidi implementation.  That's good, very good.  Nevertheless, I am
going to reserve judgment.

 > > But if that doesn't work, I don't see how having explicit mark
 > > characters in the buffer can work either.
 > 
 > Explicit marks work because the reordering algorithm does TRT with
 > them, whether they are redundant or not.  It doesn't care.  By not
 > caring it makes it very easy to preserve the byte stream and not risk
 > changing it behind user's back.

The algorithm will be the same, except that it needs to work with a
"virtual" stream where some characters are not present in the buffer.
This is no different from handling faces, which *could* be represented
as characters in the buffer (and *are* in HTML, for example -- which
of course has been deprecated in favor of CSS!  Hmm... :-).

The necessary buffering is a relatively small amount of complexity
compared to the bidi algorithm itself.

 > The _value_ doesn't matter.  It's the property symbol that cannot be
 > the same in overlapping regions, unless the values are identical.

Of course the value matters.  A 'direction property with a sequence
value can encode the whole stack, up to 61 levels.  Again, I wouldn't
want to maintain that design (space-inefficiency and the question of
consistency of neighboring regions are killers, I think), but there
are surely lighter-weight, more efficient designs.

 > > Or you could simply replace the directional marks with a string on
 > > the preceding non-mark character containing the mark characters that
 > > were present in the source.
 > 
 > And then move that string when text is inserted after the preceding
 > non-mark character, or that character is deleted, yes?  Sounds like
 > fun.

Put that way, not at all.  But you know what?  Emacs has long ago
solved such problems, at least most of them.  IIUC, in XEmacs, this
could easily be implemented with a zero-length extent with appropriate
stickiness attributes.  If Emacs doesn't already have such a device,
it would be easy enough to add a marker-with-direction extension by
maintaining a hashtable with keys markers and values mark characters.
Not terribly efficient, of course, but proof of concept.

 > Using explicit marks does have its drawbacks, but they are minor and
 > mostly just need to get used to.

There is way too much about Emacs that users (and developers!) "mostly
just need to get used to." :-(  Whether we can do much about it, I
don't know.  But I'm not going to give up yet. :-)

Thank you very much for taking the time out to explain your reasons
for your design choices.  I have a much better grasp of the practical
issues involved in implementing bidi in Emacsen now.