[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: coding tags and utf-16

From: Kenichi Handa
Subject: Re: coding tags and utf-16
Date: Tue, 28 Feb 2006 10:08:36 +0900
User-agent: SEMI/1.14.3 (Ushinoya) FLIM/1.14.2 (Yagi-Nishiguchi) APEL/10.2 Emacs/22.0.50 (i686-pc-linux-gnu) MULE/5.0 (SAKAKI)

Sorry for the late responce.

In article <address@hidden>, Kevin Rodgers <address@hidden> writes:

>> I thought we had discussed this already.  The BOM-encodings should
>> have priority since the likelihood of a misdetection is negligible
>> (the character pair does not make sense at the start of a text in
>> latin-1 in any language): the only thing that can reasonably be
>> expected to happen is that a binary file is detected as utf-16.  Not
>> much of an issue, I'd say.

I've just digged out old mails we exchanged on this topic
(about a year ago).  To my understanding, there was no
clear conclusion.  Here are the extracts:
I wrote:
> I think BOM is not that safe because there are many charsets
> who have normal letters at 0xFE and 0xFF.

Jason wrote:
> But what are those characters, and are they likely to appear as a pair 
> at the beginning of the file, and nowhere else?

I wrote:
> Sorry, I don't know.

Dave wrote:
>> Exactly what Windows does for what?  Recognizing a utf-16 registry
>> file when opened in the registry editor?

> Auto-detecting utf-16 generally.  Although I don't think it would give
> false positives on iso-8859 text, I don't know if it could with other
> charsets.
> I could believe that Windows doesn't just go by byte-order-mark in
> some locales where there might be a problem.  If so, it could be
> useful to do the same thing.

For instance, I've just googled the two character sequence
of 0xFE 0xFF of koi8 and found several occurrences.

> Exactly.  So why haven't these entries been added to 
> auto-coding-regexp-alist?

> ("\\`\xEF\xBB\xBF" . utf-8)

As far as I know, UTF-8 should not start with this sequence
unless the text really starts with ZWNBSP (very unlikely).

> ("\\`\xFE\xFF" . utf-16-be)
> ("\\`\xFF\xFE" . utf-16-le)

Although it's not clear how safe they are, if no one objects,
I'll add them in auto-coding-regexp-alist.

> ("\\`\x00\x00\xFE\xFF" . utf-32-be)
> ("\\`\xFF\xFE\x00\x00" . utf-32-le)

Emacs doesn't support those encoding for the momemnt.

Kenichi Handa

reply via email to

[Prev in Thread] Current Thread [Next in Thread]