Re: Fwd: Re: Inadequate documentation of silly characters on screen.

[Davis Herring]

[I end up having to say the same thing several times here; I thought it
preferable to omitting any of Alan's questions or any aspect of the
problem.  It's not meant to be a rant.]

> No, you (all of you) are missing the point.  That point is that if an
> Emacs Lisp hacker writes "?ñ", it should work, regardless of
> what "codepoint" it has, what "bytes" represent it, whether those
> "bytes" are coded with a different codepoint, or what have you.  All of
> that stuff is uninteresting.  If it gets interesting, like now, it is
> because it is buggy.

When you wrote ?ñ, it did work -- that character has the Unicode (and
Emacs 23) code point 241, so that two-character token is entirely
equivalent to the token "241" in Emacs source.  (This is independent of
the encoding of the source file: the same two characters might be
represented by many different octet sequences in the source file, but you
always get 241 as the value (which is a code point and is distinct from
octet sequences anyway).)

But you didn't insert that object!  You forced it into a (perhaps
surprisingly: unibyte) string, which interpreted its argument (the integer
241) as a raw byte value, because that's what unibyte strings contain. 
When you then inserted the string, Emacs transformed it into a (somewhat
artificial) character whose meaning is "this was really the byte 241,
which, since it corresponds to no UTF-8 character, must merely be
reproduced literally on disk" and whose Emacs code point is 4194289. 
(That integer looks like it could be derived from 241 by sign-extension
for the convenience of Emacs hackers; the connection is unimportant to the
user.)

> OK.  Surely displaying it as "\361" is a bug?  Should it not display as
> "\17777761".  If it did, it would have saved half of my ranting.

No: characters are displayed according to their meaning, not their
internal code point.  As it happens, this character's whole meaning is
"the byte #o361", so that's what's displayed.

> So, how did the character "ñ" get turned into the illegal byte #xf1?  Is
> that the bug?

By its use in `aset' in a unibyte context (determined entirely by the
target string).

>> You assume that ?ñ is a character.
>
> I do indeed.  It is self evident.

Its characterness is determined by context, because (as you know) Emacs
has no distinct character type.  So, in the isolation of English prose, we
have no way of telling whether ?ñ "is" a character or an integer, any more
than we can guess about 241.  (We can guess about the writer's desires,
but not about the real effects.)

> Now, would you too please just agree that when I execute the three forms
> above, and "ñ" should appear?

That's Stefan's point: should common string literals generate multibyte
strings (so as to change the meaning, not of the string, but of `aset', to
what you want)?  Maybe: one could also address the issue by disallowing
`aset' on unibyte strings (or strings entirely) and introducing
`aset-unibyte' (and perhaps `aset-multibyte') so that the argument
interpretation (and the O(n) nature of the latter) would be made clear to
the programmer.  Maybe the doc-string for `aset' should just bear a really
loud warning.

It bears more consideration than merely "yes" to your question, as
reasonable as it seems.

> What is the correct Emacs internal representation for "ñ" and "ä"?  They
> surely cannot share internal representations with other
> (non-)characters?

They have the unique internal representation as (mostly) Unicode code
points (integers) 241 and 228, which happen to be identical to the
representations of bytes of those values (which interpretation prevails in
a unibyte context).

Davis

-- 
This product is sold by volume, not by mass.  If it appears too dense or
too sparse, it is because mass-energy conversion has occurred during
shipping.