unicode normalization / unorm (was: [bug-gnu-libiconv] The utf-8-mac encoder on macOS gives incorrect output)

[Assaf Gordon]

Hello,

(cc'ing coreutils@)

On 2017-10-26 05:50 AM, Marcin Sulikowski wrote:
> I've been trying to use libiconv on macOS to convert UTF-8 strings to 
> their NFD form using libiconv's "utf-8-mac" encoding which is available 
> on macOS.

FWIW,

In GNU coreutils we are working on a unicode normalization program 
(unorm) which can perform nfd/nfc/nfkd/nfkc conversions and other 
multibyte character processing.

It is still highly experimental, but produces the following output based 
on your input:

===
$ ( printf "a%.0s" `seq 4094` ; echo -n ó ) \
     | unorm --normalization=nfd \
     | hexdump -e '8/1 "%02x " "\n"'
61 61 61 61 61 61 61 61
*
61 61 61 61 61 61 6f cc
81
===

Where U+00f3 (\xc3 \xb3) was normalized to "o" + U+0301 (\x6F \xCC \x81).


More information about the multibyte implementation progress is here:
 https://crashcourse.housegordon.org/coreutils-multibyte-support.html

If you'd like to experiment with the program, a snapshot is  here:
http://files.housegordon.org/src/coreutils-multibyte-experimental-8.28.39-79242.tar.xz
(note this is unstable and unsupported snapsot of coreutils code).

Any feedback is appreciated.

regards,
 - assaf