Re: [emacs-bidi] Suboptimal display-reordering in minibuffer

[Martin J. Dürst]

Hello Beni, Eli, others,

On 2010/07/02 19:43, Beni Cherniavsky-Paskin wrote:
> Eli Zaretskii wrote:
> > From: "Martin J. Dürst"<address@hidden>
> > > Of course, if the characters in the overlay properties before-string and
> > > after-string are not currently taken into account when running the bidi
> > > algorithm, then that approach may not work very easily.
> >
> > You are right: they aren't taken into account.  I have yet to code
> > support for reordering text in display strings.  To add this feature,
> > I will need to solve quite a few problems.  Until I do, I won't know
> > whether what you suggest is even doable with a reasonable effort.
> >
> > I also think that, even if doable, this is a somewhat hackish
> > solution.  I think having a special text property that covers the text
> > that needs to be reordered is a cleaner solution.
> I'm second-guessing Martin here (correct me if I'm wrong!) but I suspect
> this proposal is based on:
>
> (A) HTML bidi experience, where CSS already supports before/after-string,
>      and browsers respect bidi control codes appearing there (?)

Well, browsers already implementing the bidi algorithm was one of the 
motivations for doing all our previous work (simulation, editing 
prototype) with HTML in browsers. I also once worked on a variant of the 
editing prototype that used CSS before/after selectors, but this is 
still not implemented in all editors, so we didn't make this the main 
line of our work.

> (B) a presumption that it's the easiest for you because you planned to
>      implement it anyway.
>
> However, since it's not implemented yet (and it's no obvious how to),
> I want to point out that having proper bidi in strings and comments
> is a much more useful goal (to users) than supporting bidi reordering of
> before/after-string properties of overlays, so if there is a better way
> to get the former, you should feel free to skip the latter.

Yes, of course the goal is to support 'proper', non-confusing bidi in 
strings and comments in programming languages, and for markup languages 
such as XML and HTML, and using before/after-string properties of 
overlays would be one means to get there.

> > > Date: Wed, 30 Jun 2010 02:04:30 +0300
> > > From: Amit Aronovitch<address@hidden>
> > > I believe that the required infrastructure has a lot in common with the
> > > coloring (font-lock) system.
> If I may add some motivation:
> In an ideal world, we would have a separate "bidi-lock" engine,
> hand-tuned for each syntax in the world, parsing it and endowing
> characters with whatever overlays/properties the display engine
> needs to show it in perfect order.
>
> But that's out of question because it'd require huge *manpower*
> to maintain, and only a fraction of the world's programmers care
> about bidi.  A much more realistic (even if less perfect) approach
> is to harness existing polished font-lock engine, defining bidi
> in terms of it output (e.g. "characters with font-lock-string-face
> should be treated as embeddings and not mix with outside text").
>
> [I'm just saying that'd be cool; see below on feasibility...]

I agree with this general assessment. If there's already some 
commonality, if only for naming, for a set of programming languages, it 
would be a pity to not reuse that for bidi.


> There is one complication that's not obvious:
>
> > It's not that simple.  The way bidi reordering is designed and
> > implemented in Emacs, the reordering itself happens _before_ faces,
> > overlays, and other display features are considered.  The bidi
> > reordering engine is totally oblivious to text properties, overlays,
> > images, etc.; it just controls which character will be considered next
> > for delivering it to the display, and all the rest, i.e. calculation
> > of the face of that character, its display metrics, etc. -- all this
> > happens _after_ reordering, in code that calls the reordering engine.

That's fine. But we understand that we need some way to tweak/control 
some aspects of bidi in order to get decent and reasonably readable 
display for program texts and markup languages. So whether it's some 
text properties, some overlay properties, or whatever, we need at least 
some hook to influence the basic bidi reordering.

> So currently there is no way whatsoever by which lisp code (even the
> hypothetical perfect bidi-lock) can affect reordering?  (Except for
> actually inserting characters into the buffer, which we agree is no-no)
>
> So to get smarter bidi, the bidi engine will have to inspect *some*
> kind of metadata on the characters it handles.  Now let's separate
> two mostly orthogonal concerns:
>
> 1. What bidi metadata exactly do we want?
>     Emacs has many kinds of metadata (face info, invisibility,
>     local vars/keymap, hooks...) but we'll need new one(s).
>     - Virtual control characters?
>     - An increment to embedding level?
>     - Overridding character's unicode bidi cathegory?
>     ...
>
>     It's an interesting discussion, but I don't have much to say now.
>
> 2. How will it be attached to the text?
>     It seems that Emacs has exactly two ways to attach metadata to text:
>     - Text properties
>     - Overlays (implemented by pairs of markers?)
>
>     Almost all metadata can be attached by both, although a few (e.g.
>     before/after-string) only work on overlays.
>
>     Also, there there are several indirection mechanisms, notably faces:
>     a lot of metadata (such as font size) is not directly attached to
>     text/overlays but to a face object (which is a property of text/overlay).
>
> Let's concentrate on question 2 here.
>
> > From the implementation POV:
> How would you like bidi information to be attached so the bidi engine
> could easily inspect it?  Is there a shortcut at all, or would *any*
> support require changing the architecture to do full-blown resolution
> of text properties before bidi?
>
> in light of the above rationale of harnessing font-lock:
> AFAIU, font-lock only sets faces as direct text properties (not overlays),
> right?  So, it would be convenient to indirect bidi metadata through faces -
> although in general it seems a weird place to put it.

I agree with most of the above. I don't think we necessarily need to 
attach bidi tweaking directly to font-lock, it indeed looks like a weird 
place. Even if we don't, there should be a way to influence bidi 
indirectly based on font-lock information.

Regards,    Martin.

> --
> Beni Cherniavsky-Paskin<address@hidden>

--
#-# Martin J. Dürst, Professor, Aoyama Gakuin University
#-# http://www.sw.it.aoyama.ac.jp   mailto:address@hidden