Re: vasnprintf: Reject a width > INT_MAX

[Bruno Haible]

I did:
>       vasnprintf: Reject a width > INT_MAX.
>       * lib/vasnprintf.c (VASNPRINTF): If a width is > INT_MAX, fail with
>       EOVERFLOW.

This causes a test failure on Cygwin. Namely, Cygwin has vasnprintf,
but with this argument list
    "x%03000000000dy\n", -17
it returns "x-17y\n", of length 6,

It's best to ignore this vasnprintf implementation. Not because huge
widths were important by themselves, but because an implementation that
just swallows parts of the output when it cannot produce them is also
likely to have bugs in other situations, such as when malloc() fails.

This patch does that, and thus fixes the test failure.


2024-06-17  Bruno Haible  <bruno@clisp.org>

        vasnprintf: Fix test failure on Cygwin.
        * m4/vasnprintf.m4 (gl_FUNC_VASNPRINTF): Require AC_CANONICAL_HOST. Test
        whether vasnprintf works reliably; invoke gl_REPLACE_VASNPRINTF if not.

diff --git a/m4/vasnprintf.m4 b/m4/vasnprintf.m4
index 5dc3fe0c4d..a02f4e1e1d 100644
--- a/m4/vasnprintf.m4
+++ b/m4/vasnprintf.m4
@@ -1,5 +1,5 @@
 # vasnprintf.m4
-# serial 53
+# serial 54
 dnl Copyright (C) 2002-2004, 2006-2024 Free Software Foundation, Inc.
 dnl This file is free software; the Free Software Foundation
 dnl gives unlimited permission to copy and/or distribute it,
@@ -7,8 +7,40 @@
 
 AC_DEFUN([gl_FUNC_VASNPRINTF],
 [
+  AC_REQUIRE([AC_CANONICAL_HOST]) dnl for cross-compiles
   AC_CHECK_FUNCS_ONCE([vasnprintf])
-  if test $ac_cv_func_vasnprintf = no; then
+  if test $ac_cv_func_vasnprintf = yes; then
+    dnl On Cygwin, in directives with a huge width, the width is ignored, and
+    dnl the function returns a wrong result.
+    AC_CACHE_CHECK([whether vasnprintf works],
+      [gl_cv_func_vasnprintf_works],
+      [AC_RUN_IFELSE(
+         [AC_LANG_SOURCE(
+            [[#include <stdio.h>
+            ]],
+            [[size_t len;
+              char *res = vasnprintf (NULL, &len, "x%03000000000dy\n", -17);
+              /* On Cygwin 3.4.6, res is "x-17y\n" and len == 6: wrong.  */
+              return (res != NULL && len < 10);
+            ]])
+         ],
+         [gl_cv_func_vasnprintf_works=yes],
+         [gl_cv_func_vasnprintf_works=no],
+         [case "$host_os" in
+                     # Guess no on Cygwin.
+            cygwin*) gl_cv_func_vasnprintf_works="guessing no";;
+                     # If we don't know, obey --enable-cross-guesses.
+            *)       gl_cv_func_vasnprintf_works="$gl_cross_guess_normal";;
+          esac
+         ])
+      ])
+  fi
+  if test $ac_cv_func_vasnprintf != yes \
+     || case "$gl_cv_func_vasnprintf_works" in
+          *yes) false;;
+          *)    true;;
+        esac
+  then
     gl_REPLACE_VASNPRINTF
   fi
 ])