Re: [bug #4658], and also [bug #4624]

[Pete French]

> be fine for the conversion between UNICODE and UTF8, but what about all 
> the other conversions? If we change the internal storage of UNICODE from 
> UCS2 to UTF16 all the other conversions should know about this. No big 

Thats an interesting point - I havent yet checked what happens on OSX,
but it is a good point.

> deal for the iconv conversions, as here we would just change the name 
> used for the UNICODE conversion (somethign that is missing in your 

I havent ever used iconv. I guess I missed something somewhere (I assume
you mean that the bits of the coe which communicate with iconv need to
tell iot that the inout is now in UTF16)

> patch). But what should we do for the other converisions? As far as I 
> can tell these other encodings don't include any characters that are not 
> on the BMP, so we would just need to ignore the extra on loose 
> encodings. With does resolved I, I would see no problem with your patch.

Well, that should just come out in the was shouldnt it ? If those other
conversions do not include characters outside the BMP then they wont
include any mappings for the surrogate pairs, even if they dont know that
these particular 16 bit values should be treated separately.

I agree that there are a lot more issues with this than just the
simple UTF8 conversion, but I am not sure how much we need to attempt
to handle the non BMP codepoints in reality. OpenStep didnt understand them
at all, and OSX doesnt actually do a complete implementation of them
either (e.g. decomposition doesnt work on them).

Can you point me to the correct place to do the iconv changes pleaase
and I'll do another version of the patch at the weekend, if that would
satisfy for now ? Other bits and pieces I'll try and do as I get time
over the neext few weeks. My original idea was to revamp all of it
including case mapping stuff and also add the decompose (and possibly
precompose) methods too, time allowing. But I want to make sure I get
it right...

thanks for the advice,

-bat.