new module 'c32rtomb'

[Bruno Haible]

The function c32rtomb() is like wcrtomb(), except that it takes a 32-bit wide
character (char32_t) as argument, not a wchar_t.

While implementing this module, I noted a mistake in the 'mbrtoc32' module:
It assumed that when wchar_t is 32-bit and mbrtoc32() exists in libc,
mbrtoc32() is equivalent to mbrtowc(); in other words, that the char32_t
encoding and the wchar_t encoding of the same multibyte sequence are the
same. But this is not the case! On FreeBSD 12 and Solaris 11.4, the
two encodings are different. The FreeBSD 12 wchar_t encoding is apparently
based on ISO 2022 (very old).

The fix is to use mbrtoc32() on platforms where this is possible, namely
on FreeBSD.

On Solaris 11.4 and native Windows, however, it is not good to use the
system's mbrtoc32() because it refuses to convert some multibyte sequences
that mbrtowc() supports!

So, we end up using the system's mbrtoc32() and c32rtomb() functions on
  - glibc,
  - FreeBSD,
  - AIX,
and not using them on
  - Solaris 11.4,
  - mingw,
  - MSVC.


2020-01-08  Bruno Haible  <address@hidden>

        mbrtoc32: Use the system's mbrtoc32 if it exists and basically works.
        * m4/mbrtoc32.m4 (gl_MBRTOC32_SANITYCHECK): New macro.
        (gl_FUNC_MBRTOC32): Require it. Set REPLACE_MBRTOC32 if mbrtoc32 exists
        but is not working.
        * lib/mbrtoc32.c: Include hard-locale.h, <locale.h>.
        (mbrtoc32): If the char32_t encoding and the wchar_t encoding may
        differ, use the system's mbrtoc32, adding workarounds.
        * modules/mbrtoc32 (Depends-on): Add hard-locale.
        * doc/posix-functions/mbrtoc32.texi: Mention the Solaris and native
        Windows problem.
        * lib/btoc32.c: Include <stdio.h>, <string.h>.
        (btoc32): If the char32_t encoding and the wchar_t encoding may differ,
        use mbrtoc32, not btowc.
        * modules/btoc32 (Depends-on): Add mbrtoc32.
        * lib/mbsrtoc32s.c (mbsrtoc32s): If the char32_t encoding and the
        wchar_t encoding may differ, use mbrtoc32, not mbsrtowcs.
        * modules/mbsrtoc32s (Depends-on): Update conditions.
        (configure.ac): Compile mbsrtoc32s-state.c unconditionally.
        * lib/mbsnrtoc32s.c (mbsnrtoc32s): If the char32_t encoding and the
        wchar_t encoding may differ, use mbrtoc32, not mbsnrtowcs.
        * modules/mbsnrtoc32s (Depends-on): Update conditions.
        (configure.ac): Compile mbsrtoc32s-state.c unconditionally.

2020-01-08  Bruno Haible  <address@hidden>

        c32rtomb: Add tests.
        * tests/test-c32rtomb.c: New file, based on tests/test-wcrtomb.c.
        * tests/test-c32rtomb.sh: New file, based on tests/test-wcrtomb.sh.
        * tests/test-c32rtomb-w32.c: New file, based on
        tests/test-wcrtomb-w32.c.
        * tests/test-c32rtomb-w32-1.sh: New file, based on
        tests/test-wcrtomb-w32-1.sh.
        * tests/test-c32rtomb-w32-2.sh: New file, based on
        tests/test-wcrtomb-w32-2.sh.
        * tests/test-c32rtomb-w32-3.sh: New file, based on
        tests/test-wcrtomb-w32-3.sh.
        * tests/test-c32rtomb-w32-4.sh: New file, based on
        tests/test-wcrtomb-w32-4.sh.
        * tests/test-c32rtomb-w32-5.sh: New file, based on
        tests/test-wcrtomb-w32-5.sh.
        * tests/test-c32rtomb-w32-6.sh: New file, based on
        tests/test-wcrtomb-w32-6.sh.
        * tests/test-c32rtomb-w32-7.sh: New file, based on
        tests/test-wcrtomb-w32-7.sh.
        * modules/c32rtomb-tests: New file.

        c32rtomb: New module.
        * lib/uchar.in.h (c32rtomb): New declaration.
        * lib/c32rtomb.c: New file, based on lib/unistr/u8-uctomb-aux.c.
        * m4/c32rtomb.m4: New file.
        * m4/uchar.m4 (gl_UCHAR_H): Test whether c32rtomb is declared.
        (gl_UCHAR_H_DEFAULTS): Initialize GNULIB_C32RTOMB, HAVE_C32RTOMB,
        REPLACE_C32RTOMB.
        * modules/uchar (Makefile.am): Substitute GNULIB_C32RTOMB,
        HAVE_C32RTOMB, REPLACE_C32RTOMB.
        * modules/c32rtomb: New file.
        * tests/test-uchar-c++.cc: Test the signature of c32rtomb.
        * doc/posix-functions/c32rtomb.texi: Document the new module.
        * doc/posix-functions/wcrtomb.texi: Mention the new module.

2020-01-08  Bruno Haible  <address@hidden>

        c32tob: Make consistent with mbrtoc32.
        * lib/c32tob.c: Include <stdio.h>, <string.h>, <wchar.h>.
        (c32tob): If the char32_t encoding and the wchar_t encoding may differ,
        use c32rtomb, not wctob.
        * modules/c32tob (Files): Add m4/mbrtoc32.m4.
        (Depends-on): Add c32rtomb.
        (configure.ac): Require gl_MBRTOC32_SANITYCHECK.

0001-mbrtoc32-Use-the-system-s-mbrtoc32-if-it-exists-and-.patch
Description: Text Data

0002-c32rtomb-New-module.patch
Description: Text Data

0003-c32rtomb-Add-tests.patch
Description: Text Data

0004-c32tob-Make-consistent-with-mbrtoc32.patch
Description: Text Data