bug#13041: 24.2; diacritic-fold-search

[Eli Zaretskii]

> From: Juri Linkov <juri@jurta.org>
> Date: Sat, 01 Dec 2012 02:27:40 +0200
> Cc: 13041@debbugs.gnu.org, perin@acm.org
> 
> > In the last message of that thread, you say “Provided it doesn’t make
> > the search slow, it would be nice to add it to Emacs activating on
> > some user settings.”  Do you remember if that technique turned out to
> > be tolerably speedy?
> 
> Yes, I have no problems with the speed.  The problem is how to
> disable this feature when it is active.  We need a special key
> to toggle it in Isearch.  One variant is M-s ~ where the easy-to-type
> TILDE character represents diacritics.  Also it's unclear whether the
> Isearch prompt should indicate its active state as e.g.

I don't understand why this thread is talking only about Latin
characters with diacritics.  That is a special case of what Unicode
calls "compatibility equivalence" (q.e.).  For example, even in the
Latin environments, don't you want to find "sniff" when searching for
"sniff", and vice versa? And there are similar issues in many
non-Latin scripts.

The decomposition of a character such as 'ff' is given by the Unicode
database, for example:

  FB00;LATIN SMALL LIGATURE FF;Ll;0;L;<compat> 0066 0066;;;;N;;;;;
                                      ^^^^^^^^^^^^^^^^^^

(66 hex, or 102 decimal, is the codepoint of 'f').

Emacs already supports these decomposition properties.  E.g.:

  (get-char-code-property ?ff 'decomposition) => (compat 102 102)

Another example, closer to the issue that triggered this thread:

  (get-char-code-property ?è 'decomposition) => (101 768)

(If you want to understand why the previous example included "compat"
in the result, while this one doesn't, read more about Unicode
normalization forms.  The distinction is irrelevant for the current
discussion.)

Using these properties, every search string can be converted to a
sequence of non-decomposable characters (this process is recursive,
because the 'decomposition' property can use characters that
themselves are decomposable).  If the user wants to ignore diacritics,
then the diacritics should be dropped from the decomposition sequence
before starting the search.  E.g., for the decomposition of è above,
we will drop the 768 and will be left with 101, which is 'e'.  Then
searching for that string should apply the same decomposition
transformation to the text being searched, when comparing them.

This would be the most general way of solving this issue, a way that
is not limited to diacritics nor to Latin scripts.  And doing that
will move Emacs closer to the goal of being Unicode compatible, since
support for this is required by the Unicode Standard.

By contrast, building and using custom data bases of equivalences that
are limited to diacritics in Latin scripts is not moving Emacs towards
that goal.  It's just a hack, IMO.