wcstok: fix test failure on HP-UX

[Bruno Haible]

On HP-UX 11.31 - both hppa and ia64 - I'm seeing test failures of test-wcstok.

A simple test program
=================== foo.c =================
#include <stddef.h>
#include <stdio.h>
#include <time.h>
#include <wchar.h>
int main ()
{
  wchar_t string[] = L"hello world!";
  wchar_t delim[] = L" ";
  wchar_t *ptr = NULL;
  wcstok (string, delim, &ptr);
  return (ptr == NULL);
}
===========================================
which is expected to exit with return code 0, exits with return code 1
- but only when compiled by cc, not when compiled by gcc:

$ cc -AC99 -D_XOPEN_SOURCE=500 foo.c && ./a.out ; echo $?
1
$ gcc -std=gnu99 -D_XOPEN_SOURCE=500 foo.c && ./a.out ; echo $?
0

What's going on? In order to accommodate previous standards (where the
wcstok function only took 2 arguments, not 3 arguments) [1], they merged
the 2-arguments function and the 3-arguments function into a single code
with prototype

  wchar_t * wcstok (wchar_t *ws1, const wchar_t *ws2, wchar_t **ptr);

and this code looks at bit 5 of the variable __xpg4_extended_mask in order
to determine whether it shall use or ignore the third argument 'ptr'.
More precisely, it determines the location where to store the intermediate
state as
  (__xpg4_extended_mask & 0x20
   ? ptr
   : __ismt == 0 ? &some_global_variable : __libc_get_tss(11,4))

Now, compiling with cc happens to initialize __xpg4_extended_mask to 0,
while compiling with gcc initializes it with 0x6f. And this despite the
_XOPEN_SOURCE=500 assignment specified in both compilations.

In other words, they have a broken attempt at doing symbol versioning. We're
so lucky that glibc uses ELF and has real symbol versioning! Kudos to our
giants, Eric Youngdale and Ulrich Drepper.

[1] https://pubs.opengroup.org/onlinepubs/9699919799/functions/wcstok.html

This patch provides a workaround.


2019-12-13  Bruno Haible  <address@hidden>

        wcstok: Fix test failure on HP-UX.
        * m4/wcstok.m4 (gl_FUNC_WCSTOK): Set REPLACE_WCSTOK to 1 on HP-UX.
        * doc/posix-functions/wcstok.texi: Mention the HP-UX bug.

diff --git a/doc/posix-functions/wcstok.texi b/doc/posix-functions/wcstok.texi
index 51bb656..edf30fa 100644
--- a/doc/posix-functions/wcstok.texi
+++ b/doc/posix-functions/wcstok.texi
@@ -14,6 +14,11 @@ Cygwin 1.5.x.
 @item
 This function takes only two arguments on some platforms:
 mingw, older MSVC.
+@item
+This function may use hidden state, ignoring the third argument, and thus
+exhibit a bug when two or more @code{wcstok} iteration loops are being 
performed
+in the same thread, on some platforms:
+HP-UX 11.31.
 @end itemize
 
 Portability problems not fixed by Gnulib:
diff --git -w a/m4/wcstok.m4 b/m4/wcstok.m4
index 8f0aeb0..0001eaf 100644
--- a/m4/wcstok.m4
+++ b/m4/wcstok.m4
@@ -1,4 +1,4 @@
-# wcstok.m4 serial 3
+# wcstok.m4 serial 4
 dnl Copyright (C) 2011-2019 Free Software Foundation, Inc.
 dnl This file is free software; the Free Software Foundation
 dnl gives unlimited permission to copy and/or distribute it,
@@ -7,8 +7,18 @@ dnl with or without modifications, as long as this notice is 
preserved.
 AC_DEFUN([gl_FUNC_WCSTOK],
 [
   AC_REQUIRE([gl_WCHAR_H_DEFAULTS])
+  AC_REQUIRE([AC_CANONICAL_HOST])
   AC_CHECK_FUNCS_ONCE([wcstok])
   if test $ac_cv_func_wcstok = yes; then
+    dnl On HP-UX 11.31, it depends on the compiler and linker whether wcstok()
+    dnl uses hidden state, ignoring the third argument, and thus exhibits a
+    dnl bug when two or more wcstok() iteration loops are being performed in
+    dnl the same thread.
+    case "$host_os" in
+      hpux*)
+        REPLACE_WCSTOK=1
+        ;;
+      *)
         dnl POSIX:        wchar_t *wcstok (wchar_t *, const wchar_t *, wchar_t 
**);
         dnl mingw, MSVC:  wchar_t *wcstok (wchar_t *, const wchar_t *);
         AC_CACHE_CHECK([for wcstok with POSIX signature],
@@ -33,6 +43,8 @@ wchar_t *wcstok (wchar_t *, const wchar_t *, wchar_t **);
         if test $gl_cv_func_wcstok_posix_signature = no; then
           REPLACE_WCSTOK=1
         fi
+        ;;
+    esac
   else
     HAVE_WCSTOK=0
   fi