Re: new modules for Unicode normalization

[Pádraig Brady]

Bruno Haible wrote:
> Pádraig Brady wrote:
>>> I cannot estimate how much of these 10 MB get actually loaded into a
>>> process' working set. ...
>> $ uconv -x NFC&
>> $ sudo bin/ps_mem.py | grep uconv
>>  Private  +   Shared  =  RAM used       Program
>>   1.9 MiB + 788.0 KiB =   2.7 MiB       uconv
> 
> A great tool! 

Cheers :)
It gets a surprising amount of downloads.
I must look at getting it into procps or somewhere.

> Let's see how it compares to a normalization program built
> against gnulib:
> 
> $ /arch/x86-linux/inst-icu/3.6/bin/uconv -x NFC &
> 
> $ ./unorm NFC &
> 
> # ./bin/ps_mem.py |grep uconv
>   2.3 MiB + 126.5 KiB =   2.4 MiB       uconv
> # ./bin/ps_mem.py |grep unorm
> 100.0 KiB +  11.5 KiB = 111.5 KiB       unorm
> 
> So, it uses about 20 times less memory.

Impressive, though uconv provides more functionality
(equiv to iconv+unorm).

> Now let me try to compare the speed (averaged over 100 consecutive runs, to
> eliminate the effects of load dependent CPU frequency scaling):

I usually do `/etc/init.d/cpuspeed stop` before any benchmarking.

> $ time uconv -x NFD < $UCD51/NormalizationTest.txt > /tmp/1
> real    0m0.612s
> $ time ./unorm NFD < $UCD51/NormalizationTest.txt > /tmp/2
> real    0m0.261s
> 
> $ time uconv -x NFKC < $UCD51/NormalizationTest.txt > /tmp/1
> real    0m0.598s
> $ time ./unorm NFKC < $UCD51/NormalizationTest.txt > /tmp/2
> real    0m0.309s
> 
> So, it's also twice as fast.
> 
> The gnulib modules used are:
>   uninorm/filter
>   uninorm/nfc
>   uninorm/nfd
>   uninorm/nfkc
>   uninorm/nfkd
>   unistr/u8-uctomb
>   unistr/u8-mbtoucr

That is also impressive. Can you share the code for unorm
so I can do my own testing.

I'm now swayed a bit more in favour of another `unorm` util.
However I'm still 60/40 against. What do others think?

cheers,
Pádraig.