[Octave-bug-tracker] [bug #54391] Incorrect result when attempting to type or paste UTF-8 Cyrillic text into octave CLI

[Markus Mützel]

Update of bug #54391 (project octave):

                  Status:               Confirmed => Patch Submitted        
                 Release:                   4.4.0 => dev                    

    _______________________________________________________

Follow-up Comment #4:

I could track this down to GNU readline's implementation of the Meta-key and
how we use readline in Octave:

Essentially, readline uses the highest bit of an 8-bit char to indicate that
the Meta (Alt) key was pressed at the same time as a key for that character.
That is all well as long as we are only dealing with ASCII characters.
However with UTF-8, the highest bit of a 8-bit char is also set in multi-byte
characters. 

The first byte of "Ч" (and the other characters that failed pasting) is
decimal 208. It just so happens that this is the same as Latin "P" (decimal 80
in ASCII/UTF-8) with the highest bit set to 1.
We are registering Meta-P as a shortcut for readline's
"history-search-backward" function.

If I remove that shortcut, the Cyrillic example string from comment #0 can be
pasted just fine on Octave's CLI:

octave:1> x="Частота" 
x = Частота
octave:2> uint8(x)
ans =

  208  167  208  176  209  129  209  130  208  190  209  130  208  176


The attached patch removes the two functions that we are registering with the
Meta key.

I think the Ctrl key shortcuts are save when it comes to ASCII vs. UTF-8.
For reference I'll paste the corresponding code from readlines's
"chardefs.h":

/* Some character stuff. */
#define control_character_threshold 0x020   /* Smaller than this is control.
*/
#define control_character_mask 0x1f         /* 0x20 - 1 */
#define meta_character_threshold 0x07f      /* Larger than this is Meta. */
#define control_character_bit 0x40          /* 0x000000, must be off. */
#define meta_character_bit 0x080            /* x0000000, must be on. */
#define largest_char 255                    /* Largest character value. */

#define CTRL_CHAR(c) ((c) < control_character_threshold && (((c) & 0x80) ==
0))
#define META_CHAR(c) ((c) > meta_character_threshold && (c) <= largest_char)

#define CTRL(c) ((c) & control_character_mask)
#define META(c) ((c) | meta_character_bit)

#define UNMETA(c) ((c) & (~meta_character_bit))
#define UNCTRL(c) _rl_to_upper(((c)|control_character_bit))



Interestingly, the two removed shortcuts (Meta-P and Meta-N) still work for me
even with the patch applied (with GNU readline 7.0).


(file #45076)
    _______________________________________________________

Additional Item Attachment:

File name: bug54391_readline_meta.patch   Size:1 KB


    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/bugs/?54391>

_______________________________________________
  Message sent via Savannah
  https://savannah.gnu.org/