Re: multibyte processing - handling invalid sequences (long)

[Assaf Gordon]

Hello,

Attached an improved version of 'unorm', with unicode-normalization support for 
both line-by-line and buffer/stream methods.
The code is not yet cleaned-up, but enables comparing performances of the two 
approaches.

Briefly, it seems the buffer/stream method is more or less as fast as 
line-by-line when *not* doing unicode normalization
(only one 'mbrtowc' call is done for each input character in both cases).

When unicode normalization is used, the line-by-line is faster, likely because:
a) using u8_normalize on the entire buffer instead of uninorm_filter streaming,
b) an additional wctomb is required for each output character when using 
uninorm-filter streaming.

It's likely the buffer/streaming implementation could be improved.

However, there's an issue with uninorm-filter:
The functions (e.g. uninorm_filter_write in gnulib's uninorm.h) use 'ucs4_t',
but mbrtowc/wctomb use 'wchar_t'.
Is there a guarantee that wchar_t is actually a unicode code-point? (I couldn't 
find one.)
Currently the code assumes that they are one and the same.
If that's incorrect assumption, additional conversion will be needed.

The following commands can be used to compare implementations ('-S' uses the 
buffer/stream method instead of line-by-line):

Short lines, with/out normalization:

    yes a | head -n 10M > data1

    env time ./src/unorm    < data1 > /dev/null
    env time ./src/unorm -S < data1 > /dev/null

    env time ./src/unorm -nfkc    < data1 > /dev/null
    env time ./src/unorm -nfkc -S < data1 > /dev/null

Long lines, with/out normalization:

    yes | perl -npe '$_ = "x" x int(rand(10000)) . "\n"' | head -n 50K > data2

    env time ./src/unorm    < data2 > /dev/null
    env time ./src/unorm -S < data2 > /dev/null

    env time ./src/unorm    -nfkc < data2 > /dev/null
    env time ./src/unorm -S -nfkc < data2 > /dev/null

Comments welcomed,
 - assaf

unorm-2016-08-07.patch.xz
Description: Binary data