Re: [Openexr-devel] UNICODE support in openexr file I/O

[Yves Poissant]

From: "Drew Hess" <address@hidden>

> Yves Poissant <address@hidden> writes:
>
>> All ASCII character codes (meaning codes lower than 0x7F) are exactly the
>> same in UTF-8 so if a string is ASCII, it doesn't matter if the 
>> application
>> thinks it is UTF-8 or ASCII because they are the same. For international
>> character sets, using UTF-8 (or Unicode in general) is anyway far more
>> robust and trouble free than trying to support all the possible encodings
>> around.
>>
>> The specifications could say that any UTF-8 encoded string should start 
>> with
>> the Unicode BOM (Byte Order Mark) which is 2 special 8bits characters 
>> that
>> when found at the begining of a string signifies that the string is 
>> Unicode.
>> This way, there would be no confusion. Also, adding two API to convert 
>> from
>> UTF-8 to UTF-16 (what is usually refered as Unicode) could be done using 
>> the
>> C conversion samples that can be found on www.unicode.org.
>>
>> Any furter translations, to or from some native OS encoding for instance,
>> would need to be done on the host OS. As far as I know, every major OS in
>> existence today have an extensive set of string encoding convertions 
>> which
>> includes converting UTF-8 so I don't really think there is a real need 
>> for
>> that sort of conversion in the OpenEXR API. In fact, I even think it 
>> would
>> be detrimental to anyone who don't want to bother with Unicode but just
>> wants to store and read ASCII characters. Those users would have to 
>> convert
>> from UTF-16 to ASCII while if there are not preconvertion to and from
>> UTF-16, the string could just be used, as is, as ASCII strings.

> I agree with this, for the most part.  I'm not sure we need the
> UTF-8 -> UTF-16 conversion API, though.

That is also my conclusion.

Yves Poissant
www.hash.com