Re: [bug-gawk] feature request: iconv/recode dynamic extension

[Eli Zaretskii]

> Date: Sat, 22 Dec 2018 02:29:37 +0100
> From: Franta Hanzlík <address@hidden>
> 
> not sure when it is good idea, but I think this may be usefull for
> others also: I'm just doing some word processing in gawk, and it's
> part is two string comparison. These strings are plaintext ASCII
> strings obtained by removing diacritics from the original Latin-1
> and Latin-2 strings - thus I need conversion as
>  "äáéěóöščýíüúů" -> "aaeeooscyiuuu".
> For now I solve this by calling external conversion program - as
> 
> iconv -f UTF-8 -t US-ASCII//TRANSLIT <<< "üöóäěščřžýáíéúů"
>    or
> recode -f u8..flat <<< "üöóäěščřžýáíéúů"
> 
> but for thousands strings it is too slow (and resource expensive).

libiconv's TRANSLIT will only work for Latin characters, as it's not
what you want in general.  What you want is the "decomposition" of
each character into the base character and the diacriticals/combining
accents; then you want to throw out the non-base parts.  How to do
that is defined by the Unicode Standard, and needs to use the various
data files provided by the UCD, the Unicode Character Database.

> There is perhaps lot of similar text conversions cases, where gawk
> dynamic extension for this needs wil be very useful.

It could be useful for such jobs, yes.  How frequently these jobs
happen in typical Gawk usage is another question; I don't have an
answer for that.