Re: [PATCH] Unicode Lisp reader escapes

[Aidan Kehoe]

 Ar an séiú lá de mí Bealtaine, scríobh Richard Stallman>: 

 >     All of these are incompatibilities on the Emacs Lisp side; except for
 >     the Unicode escapes, a C programmer can use any C escape desired in
 >     Emacs Lisp.
 > 
 > That being so, I think it is useful to keep that true, and implement
 > \u and \U in a way that is compatible with C.
 > 
 > We could install this now if someone writes changes for etc/NEWS and
 > the Lisp manual, as well as the code.

Okay. I’ve already signed papers; the patch below includes updates
to the NEWS file, the code and the Lisp manual. 

One mostly open question, which the below patch takes a clear stand on, is
whether it is acceptable to call decode-char (which is implemented in Lisp)
from the Lisp reader. I share Stefan Monnier’s judgement on this:

“I'd vote to keep the code in elisp.  After all, it's there, it works, and
as mentioned: there's no evidence that the decoding time of \u escapes it
ever going to need to be fast.  And it'll become fast in Emacs-unicode
anyway, so it doesn't seem to be worth the trouble.” 

I have no objection to implementing decode-char in C in general; it would
mean that handle_one_event in xterm.c could be made much more robust, for
example. It currently is the case that Unicode keysyms are handled
inconsistently with the Unicode coding systems and that code points above
#xFFFF are simply dropped, it doesn’t even try to convert them to Emacs
characters. But integrating it into Emacs for the sake of this patch seems
too much potential instability for too little benefit.

Another thing; if this patch is to be integrated, there is some Lisp in the
source tree using \u in strings (incorrectly) that will need to be changed
to use \\u.

etc/ChangeLog addition:

2006-05-06  Aidan Kehoe  <address@hidden>

        * NEWS:
        Describe the Unicode string and character escape
        

lispref/ChangeLog addition:

2006-05-06  Aidan Kehoe  <address@hidden>

        * objects.texi (Character Type):
        Describe the Unicode character escape syntax; \uABCD or \U00ABCDEF 
        specifies Unicode characters U+ABCD and U+ABCDEF respectively.  


src/ChangeLog addition:

2006-05-06  Aidan Kehoe  <address@hidden>

        * lread.c (read_escape):
        Provide a Unicode character escape syntax; \u followed by exactly 
        four or \U followed by exactly eight hex digits in a comment or 
        string is read as a Unicode character with that code point.  
        

GNU Emacs Trunk source patch:
Diff command:   cvs -q diff -u
Files affected: src/lread.c lispref/objects.texi etc/NEWS

Index: etc/NEWS
===================================================================
RCS file: /sources/emacs/emacs/etc/NEWS,v
retrieving revision 1.1337
diff -u -u -r1.1337 NEWS
--- etc/NEWS    2 May 2006 01:47:57 -0000       1.1337
+++ etc/NEWS    6 May 2006 16:57:54 -0000
@@ -3772,6 +3772,13 @@
 been declared obsolete.
 
 +++
+*** New syntax: \uXXXX and \UXXXXXXXX specify Unicode code points in hex.
+Use "\u0428" to specify a string consisting of CYRILLIC CAPITAL LETTER SHA,
+or "\U0001D6E2" to specify one consisting of MATHEMATICAL ITALIC CAPITAL
+ALPHA (the latter is greater than #xFFFF and thus needs the longer
+syntax). Also available for characters. 
+
++++
 ** Displaying warnings to the user.
 
 See the functions `warn' and `display-warning', or the Lisp Manual.
Index: lispref/objects.texi
===================================================================
RCS file: /sources/emacs/emacs/lispref/objects.texi,v
retrieving revision 1.53
diff -u -u -r1.53 objects.texi
--- lispref/objects.texi        1 May 2006 15:05:48 -0000       1.53
+++ lispref/objects.texi        6 May 2006 16:57:56 -0000
@@ -431,6 +431,20 @@
 bit values are 2**22 for alt, 2**23 for super and 2**24 for hyper.
 @end ifnottex
 
address@hidden unicode character escape
+  Emacs provides a syntax for specifying characters by their Unicode code
+points.  @code{?\uABCD} represents a character that maps to the code
+point @samp{U+ABCD} in Unicode-based representations (UTF-8 text files,
+Unicode-oriented fonts, etc.).  There is a slightly different syntax for
+specifying characters with code points above @code{#xFFFF};
address@hidden represents an Emacs character that maps to the code
+point @samp{U+ABCDEF} in Unicode-based representations, if such an Emacs
+character exists.
+
+  Unlike in some other languages, while this syntax is available for
+character literals, and (see later) in strings, it is not available
+elsewhere in your Lisp source code.
+
 @cindex @samp{\} in character constant
 @cindex backslash in character constant
 @cindex octal character code
Index: src/lread.c
===================================================================
RCS file: /sources/emacs/emacs/src/lread.c,v
retrieving revision 1.350
diff -u -u -r1.350 lread.c
--- src/lread.c 27 Feb 2006 02:04:35 -0000      1.350
+++ src/lread.c 6 May 2006 16:57:57 -0000
@@ -1743,6 +1743,9 @@
      int *byterep;
 {
   register int c = READCHAR;
+  /* \u allows up to four hex digits, \U up to eight. Default to the
+     behaviour for \u, and change this value in the case that \U is seen. */
+  int unicode_hex_count = 4;
 
   *byterep = 0;
 
@@ -1907,6 +1910,48 @@
        return i;
       }
 
+    case 'U':
+      /* Post-Unicode-2.0: Up to eight hex chars */
+      unicode_hex_count = 8;
+    case 'u':
+
+      /* A Unicode escape. We only permit them in strings and characters,
+        not arbitrarily in the source code as in some other languages. */
+      {
+       int i = 0;
+       int count = 0;
+       Lisp_Object lisp_char;
+       while (++count <= unicode_hex_count)
+         {
+           c = READCHAR;
+           /* isdigit(), isalpha() may be locale-specific, which we don't
+              want. */
+           if      (c >= '0' && c <= '9')  i = (i << 4) + (c - '0');
+           else if (c >= 'a' && c <= 'f')  i = (i << 4) + (c - 'a') + 10;
+            else if (c >= 'A' && c <= 'F')  i = (i << 4) + (c - 'A') + 10;
+           else
+             {
+               error ("Non-hex digit used for Unicode escape");
+               break;
+             }
+         }
+
+       lisp_char = call2(intern("decode-char"), intern("ucs"),
+                         make_number(i));
+
+       if (EQ(Qnil, lisp_char))
+         {
+           /* This is ugly and horrible and trashes the user's data. */
+           XSETFASTINT (i, MAKE_CHAR (charset_katakana_jisx0201, 
+                                      34 + 128, 46 + 128));
+            return i;
+         }
+       else
+         {
+           return XFASTINT (lisp_char);
+         }
+      }
+
     default:
       if (BASE_LEADING_CODE_P (c))
        c = read_multibyte (c, readcharfun);

-- 
Aidan Kehoe, http://www.parhasard.net/