multibyte locale patches for GNU utils available

[Bruno Haible]

Hi all,

Many GNU text processing utilities doen't work correctly in multibyte locales.
This includes UTF-8 locales and all East Asian locales.

Mitsuru Chinen and I have produced fixes for some of the utilities. Patches
are available to fix the following bugs.


fileutils-4.1.diff

  * dd
    The conv=lcase, conv=ucase options don't work correctly.

sh-utils-2.0.11.diff

  * expr
    The operators "match", "substr", "index", "length" don't work correctly.

textutils-2.0.13.diff

  * comm
    Doesn't work correctly if configured with --disable-nls.

  * cut
    Doesn't work correctly.
    Cannot be used to cut along column numbers.

  * expand
    Doesn't work correctly.

  * fold
    Doesn't work correctly.
    Cannot be used to limit the number of characters per line.
    Uses line breaking at spaces, which is not suitable for East Asian
    languages.

  * head
    Cannot be used to print the first N characters.

  * join
    Doesn't work correctly if configured with --disable-nls.
    Doesn't compare the fields according to the locale.
    Doesn't work when -t is used with a non-ASCII character.

  * paste
    Doesn't work correctly.
    Doesn't recognize ISO C 99 \uNNNN escape sequences.

  * sort
    Doesn't work correctly if configured with --disable-nls.

  * unexpand
    Doesn't work correctly.

  * uniq
    The options -f, -s, -w don't work correctly.

sed-3.02.diff

  * sed
    Ignores the LANG/LC_CTYPE/LC_ALL environment variables.
    Additionally to the patch, the package must be configured with
    "--with-regex=", in order to use the regular expression engine in glibc.

sharutils-4.2.1.diff

  * --help messages are not converted to the locale encoding.
  * shar: Internationalized shar scripts actually never cause
    internationalized messages to be printed.


Download:
  ftp://ftp.ilog.fr/pub/Users/haible/utf8/fileutils-4.1.diff
  ftp://ftp.ilog.fr/pub/Users/haible/utf8/sh-utils-2.0.11.diff
  ftp://ftp.ilog.fr/pub/Users/haible/utf8/textutils-2.0.13.diff
  ftp://ftp.ilog.fr/pub/Users/haible/utf8/sed-3.02.diff
  ftp://ftp.ilog.fr/pub/Users/haible/utf8/sharutils-4.2.1.diff

The patches assume libc support for multibyte locales, i.e. to use them
optimally, you need glibc-2.2.2 or newer.


Bruno