[emacs-bidi] Supporting non-plain-text buffers (was: Suboptimal display-reordering in minibuffer)

[Eli Zaretskii]

> Date: Fri, 02 Jul 2010 10:04:35 +0900
> From: "Martin J. Dürst" <address@hidden>
> CC: address@hidden, address@hidden

Note that I've changed the Subject line.  It's time.

> One thing that we should think about is what people want to happen if 
> there is actual displayable text in some of these strings. I don't have 
> much of an idea where this is used, but I can imagine that at least in 
> some usage scenarios, one might want the text added via an overlay to be 
> rendered in exactly the same way as the text in the buffer.

Can you explain what do you mean by the last sentence?  Perhaps an
example will clarify that.

> > I think having a special text property that covers the text
> > that needs to be reordered is a cleaner solution.
> 
> It's definitely also a viable solution, although there also might be 
> some tricky issues. Say you have a property defining an embedding from 
> characters 10 to 30, and another such property from characters 20 to 40. 
> What exactly is that supposed to mean?

This cannot happen in Emacs, because each property can have only one
value for each character.  In effect, ranges of buffer positions of the
same text property cannot overlap.

In any case, if this were possible, it would first and foremost have
to be solved for the unidirectional display.

> > I don't intend to change the bidi reordering engine in any significant
> > way, to support these features.  All that's needed is a possibility to
> > tell it "restrict yourself to region between buffer positions P1 and
> > P2".  Actually, it just descended on me that I can easily do that with
> > `narrow-to-region', since the reordering engine already honors that,
> > it never goes out of the accessible portion of text.
> 
> I'm not sure I understand, but if it means that the bidi algorithm is 
> just applied piecewise, that won't be enough. It may be enough for some 
> simple cases, such as C programs, where the main concern is to keep text 
> within string constants together, and the rest is ASCII only and 
> therefore goes LTR. However, on the other hand, with some XML markup 
> with e.g. element and attribute names in Hebrew, in our experience 
> actual nestings (i.e. embeddings in terms of the bidi algorithm) are 
> highly desirable.

Again, an example would go a long way towards explaining what you
mean.  In general, what I wrote does not eliminate the possibility
that embeddings might be used within the reordered parts, nor that the
text outside of the markup is LTR only.

I just meant to say that, technically, reordering of just a portion of
text can be achieved by temporarily narrowing the buffer to that
portion, for as long as the display engine is processing that portion.

> I think there are also other ways of attacking the problem. What about, 
> for example, a property on characters that increases the embedding level 
> in a certain way?

This idea was actually discussed some 10 years ago, as one of the
possible means of maintaining the reordering information as part of
the buffer.  It was rejected because, as I explained above, text
properties cannot overlap, so maintaining this information would be a
pain when the buffer is edited: you would need to split and join
properties' ranges when embedding format codes are added or deleted.

> Or a property that changes the bidi category of a character?

This can be done if we need it, but I still don't see use-cases that
would benefit from such a feature.