bug#11860: 24.1; Arabic - Harakat (diacritics, short vowels) don't appear

[Kenichi Handa]

In article <83txw0aczg.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:

> > From: Kenichi Handa <handa@gnu.org>
> > Cc: eliz@gnu.org, 11860@debbugs.gnu.org, smias@yandex.ru
> > Date: Sat, 18 Aug 2012 11:45:27 +0900
> > 
> > So, apparently Emacs on Windows and GNU/Linux uses the
> > different metrics of glyphs.  As the shaper on GNU/Linux
> > (m17n-lib library) works correctly for the same font, and
> > the other applications on Windows have no problem, I suspect
> > that the problem is in Emacs' interface with uniscribe
> > (w32font.c or w32uniscribe.c).
> > 
> > If this problem happens only for bidi scripts, one
> > possibility is that Emacs's rendering engine (xdisp.c)
> > expects glyphs in a glyph-string are rendered in that order
> > from left to right, but the returned glyph-string on Windows
> > should be rendered in reverse order.

> If this is the case, how come we display the diacriticals correctly on
> Windows in other cases, e.g. with Hebrew?

For Hebrew too, on Windows, I see the same problem as what
Steffan <smias@yandex.ru> reported:

In article <349641344144469@web8d.yandex.ru>, Steffan <smias@yandex.ru> writes:
>>> I choose "hebrew-full" as input-method.
>>> 
>>> - After typing 'f' I get KAF
>>> - then by typing d I get GIMMEL
>>> - and after typing 'D' I get "the three point sign" (HEBREW POINT QUBUTS) 
>>> not below the GIMMEL but the KAF!

If you don't face with that problem, perhaps we are using
the different font.  C-u C-x = tells that "courier new" is
used for hebrew too in my case.

> > For instance, in the above case, we may have to render glyphs in
> > this order (diacritical mark first):
> > 
> >   [0 1 1593 760 0 3 6 12 4 [1 -2 0]]
> >   [0 1 1593 969 8 1 8 12 4 nil]

> Could you propose a patch to try this isea?

I have no idea.  :-(

> > I think the further debugging must be done by those who
> > knows uniscribe, w32font.c, and w32uniscribe.c.

> Alas, I don't think we have such people on board, not with high enough
> availability, anyway.  If you could kindly suggest where to look, what
> variables to display, etc., I could try doing that, and reporting the
> results.

I've just read the function uniscribe_shape in
w32uniscribe.c.  It seems that these are the key API for
uniscribe:

* ScriptItemize -- no idea what is this
* ScriptShape -- perhaps for glyph substitution (GSUB features of opentype)
* ScriptPlace -- perhaps for glyph positioning (GPOS features of opentype)

So at first please check the documentation of ScriptShape
and figure out how it works for bidi script; i.e. what order
does it expect for input, and what order does it produce.

Next please find the meaning of this code fragment:

                  /* Detect clusters, for linking codes back to
                     characters.  */
                  if (attributes[j].fClusterStart)
                    {
                      while (from < nchars_in_run && clusters[from] < j)
                        from++;
                      if (from >= nchars_in_run)
                        from = to = nchars_in_run - 1;
                      else
                        {
                          int k;
                          to = nchars_in_run - 1;
                          for (k = from + 1; k < nchars_in_run; k++)
                            {
                              if (clusters[k] > j)
                                {
                                  to = k - 1;
                                  break;
                                }
                            }
                        }
                    }

The comment refer to "clusters".  I don't know what it
exactly means in uniscribe, but I guess it relates to
grapheme cluster, and if so, this part seems to relates to
the ordering of glyphs in this kind of grapheme clauster:

  [0 1 1593 969 8 1 8 12 4 nil]
  [0 1 1593 760 0 3 6 12 4 [1 -2 0]]

---
Kenichi Handa
handa@gnu.org