Re: [Openexr-devel] UNICODE support in openexr file I/O

[Drew Hess]

Yves Poissant <address@hidden> writes:

> All ASCII character codes (meaning codes lower than 0x7F) are exactly the 
> same in UTF-8 so if a string is ASCII, it doesn't matter if the application 
> thinks it is UTF-8 or ASCII because they are the same. For international 
> character sets, using UTF-8 (or Unicode in general) is anyway far more 
> robust and trouble free than trying to support all the possible encodings 
> around.
>
> The specifications could say that any UTF-8 encoded string should start with 
> the Unicode BOM (Byte Order Mark) which is 2 special 8bits characters that 
> when found at the begining of a string signifies that the string is Unicode. 
> This way, there would be no confusion. Also, adding two API to convert from 
> UTF-8 to UTF-16 (what is usually refered as Unicode) could be done using the 
> C conversion samples that can be found on www.unicode.org.
>
> Any furter translations, to or from some native OS encoding for instance, 
> would need to be done on the host OS. As far as I know, every major OS in 
> existence today have an extensive set of string encoding convertions which 
> includes converting UTF-8 so I don't really think there is a real need for 
> that sort of conversion in the OpenEXR API. In fact, I even think it would 
> be detrimental to anyone who don't want to bother with Unicode but just 
> wants to store and read ASCII characters. Those users would have to convert 
> from UTF-16 to ASCII while if there are not preconvertion to and from 
> UTF-16, the string could just be used, as is, as ASCII strings.


I agree with this, for the most part.  I'm not sure we need the
UTF-8 -> UTF-16 conversion API, though.

d