ptsname_r: Work around ptsname_r bug on NetBSD 10.0

[Bruno Haible]

On NetBSD 10.0/i386, I see a test failure of ptsname_r. The cause is that
this function, on NetBSD, clobbers the result buffer even when it fails
with error ERANGE.

This patch adds a workaround.


2024-04-08  Bruno Haible  <bruno@clisp.org>

        ptsname_r: Work around ptsname_r bug on NetBSD 10.0.
        * m4/ptsname_r.m4 (gl_FUNC_PTSNAME_R): On NetBSD, arrange to override
        ptsname_r.
        * lib/ptsname_r.c (ptsname_r): Add workaround for NetBSD bug.
        * doc/glibc-functions/ptsname_r.texi: Mention the NetBSD bug.

diff --git a/doc/glibc-functions/ptsname_r.texi 
b/doc/glibc-functions/ptsname_r.texi
index 198ebf3fe6..785064a2fe 100644
--- a/doc/glibc-functions/ptsname_r.texi
+++ b/doc/glibc-functions/ptsname_r.texi
@@ -28,6 +28,10 @@
 When this function fails, it returns -1 instead of the error code
 on some platforms:
 macOS 11.1, FreeBSD 14.0, Android 4.3.
+@item
+When this function fails with ERANGE, it stores an empty string as result
+on some platforms:
+NetBSD 10.0.
 @end itemize
 
 Portability problems not fixed by Gnulib:
diff --git a/lib/ptsname_r.c b/lib/ptsname_r.c
index 460220b201..cc1ae495ec 100644
--- a/lib/ptsname_r.c
+++ b/lib/ptsname_r.c
@@ -68,11 +68,34 @@ ptsname_r (int fd, char *buf, size_t buflen)
 #undef ptsname_r
 {
 #if HAVE_ESSENTIALLY_WORKING_PTSNAME_R
+# if defined __NetBSD__
+  char tmpbuf[32];
+  if (buf == NULL)
+    return EINVAL;
+  if (buflen >= sizeof (tmpbuf))
+    /* ERANGE should not happen in this case.  */
+    return ptsname_r (fd, buf, buflen);
+  else
+    {
+      int ret = ptsname_r (fd, tmpbuf, sizeof (tmpbuf));
+      if (ret != 0)
+        return ret;
+      else
+        {
+          size_t len = strlen (tmpbuf);
+          if (len >= buflen)
+            return ERANGE;
+          memcpy (buf, tmpbuf, len + 1);
+          return 0;
+        }
+    }
+# else
   int ret = ptsname_r (fd, buf, buflen);
   if (ret == 0)
     return 0;
   else
     return errno;
+# endif
 #elif defined __DragonFly__
   int saved_errno = errno;
   char tmpbuf[5 + 4 + 10 + 1];
diff --git a/m4/ptsname_r.m4 b/m4/ptsname_r.m4
index 4c8826c730..286ccab8c9 100644
--- a/m4/ptsname_r.m4
+++ b/m4/ptsname_r.m4
@@ -1,5 +1,5 @@
 # ptsname_r.m4
-# serial 9
+# serial 10
 dnl Copyright (C) 2010-2024 Free Software Foundation, Inc.
 dnl This file is free software; the Free Software Foundation
 dnl gives unlimited permission to copy and/or distribute it,
@@ -74,6 +74,13 @@ AC_DEFUN([gl_FUNC_PTSNAME_R]
         *yes) ;;
         *) REPLACE_PTSNAME_R=1 ;;
       esac
+      dnl On NetBSD 10.0, when ptsname_r fails with ERANGE, it clobbers the
+      dnl result buffer. We don't use an AC_RUN_IFELSE test here, because
+      dnl while the bug exists on all platforms, only NetBSD/i386 has the
+      dnl files /dev/ptyp[01] on which the bug becomes apparent.
+      case "$host_os" in
+        netbsd*) REPLACE_PTSNAME_R=1 ;;
+      esac
     fi
   fi