Fix calloc-gnu configure results

[Bruno Haible]

Tim Rühsen wrote:
> - gcc-10 with unset CFLAGS (*.gcc-10)
> And the same set with -fsanitize... (*.sanitize).

Looking at the differences between config.cache for the gcc-10 without and with
sanitizers, there are some changes that are due to present vs. missing -O2
options (gl_cv_c_inline_effective and possibly gl_cv_func_printf_directive_n).

Other than that:

-ac_cv_func_calloc_0_nonnull=${ac_cv_func_calloc_0_nonnull=yes}
+ac_cv_func_calloc_0_nonnull=${ac_cv_func_calloc_0_nonnull=no}

Here the problem is that in the test program, we expect
  calloc ((size_t) -1 / 8 + 1, 8)
to return NULL, but the AddressSanitizer instead makes the program
exit with code 1:

  configure:45855: ./conftest
  =================================================================
  ==462953==ERROR: AddressSanitizer: calloc parameters overflow: count * size 
(2305843009213693952 * 8) cannot be represented in type size_t (thread T0)
      #0 0x7f01a2073037 in calloc 
(/usr/lib/x86_64-linux-gnu/libasan.so.6+0xaa037)
      #1 0x5653669731b1 in main (/home/tim/src/testdir-all/conftest+0x11b1)
      #2 0x7f01a14c2e0a in __libc_start_main ../csu/libc-start.c:308

  ==462953==HINT: if you don't care about these errors you may set 
allocator_may_return_null=1
  SUMMARY: AddressSanitizer: calloc-overflow 
(/usr/lib/x86_64-linux-gnu/libasan.so.6+0xaa037) in calloc
  ==462953==ABORTING
  configure:45855: $? = 1
  configure: program exited with status 1

This patch fixed it.


2020-05-23  Bruno Haible  <address@hidden>

        calloc-gnu: Avoid wrong configure results with GCC's AddressSanitizer.
        * m4/calloc.m4 (_AC_FUNC_CALLOC_IF): Split the AC_RUN_IFELSE into two
        AC_RUN_IFELSE invocations.

diff --git a/m4/calloc.m4 b/m4/calloc.m4
index 3361cba..a93439e 100644
--- a/m4/calloc.m4
+++ b/m4/calloc.m4
@@ -1,4 +1,4 @@
-# calloc.m4 serial 21
+# calloc.m4 serial 22
 
 # Copyright (C) 2004-2020 Free Software Foundation, Inc.
 # This file is free software; the Free Software Foundation
@@ -21,33 +21,48 @@ AC_DEFUN([_AC_FUNC_CALLOC_IF],
   AC_REQUIRE([AC_CANONICAL_HOST]) dnl for cross-compiles
   AC_CACHE_CHECK([for GNU libc compatible calloc],
     [ac_cv_func_calloc_0_nonnull],
-    [AC_RUN_IFELSE(
-       [AC_LANG_PROGRAM(
-          [AC_INCLUDES_DEFAULT],
-          [[int result = 0;
-            char * volatile p = calloc (0, 0);
-            if (!p)
-              result |= 1;
-            free (p);
-            p = calloc ((size_t) -1 / 8 + 1, 8);
-            if (p)
-              result |= 2;
-            free (p);
-            return result;
-          ]])],
-       [ac_cv_func_calloc_0_nonnull=yes],
-       [ac_cv_func_calloc_0_nonnull=no],
-       [case "$host_os" in
-                         # Guess yes on glibc systems.
-          *-gnu* | gnu*) ac_cv_func_calloc_0_nonnull="guessing yes" ;;
-                         # Guess yes on musl systems.
-          *-musl*)       ac_cv_func_calloc_0_nonnull="guessing yes" ;;
-                         # Guess yes on native Windows.
-          mingw*)        ac_cv_func_calloc_0_nonnull="guessing yes" ;;
-                         # If we don't know, obey --enable-cross-guesses.
-          *)             ac_cv_func_calloc_0_nonnull="$gl_cross_guess_normal" 
;;
-        esac
-       ])])
+    [if test $cross_compiling != yes; then
+       ac_cv_func_calloc_0_nonnull=yes
+       AC_RUN_IFELSE(
+         [AC_LANG_PROGRAM(
+            [AC_INCLUDES_DEFAULT],
+            [[int result = 0;
+              char * volatile p = calloc (0, 0);
+              if (!p)
+                result |= 1;
+              free (p);
+              return result;
+            ]])],
+         [],
+         [ac_cv_func_calloc_0_nonnull=no])
+       AC_RUN_IFELSE(
+         [AC_LANG_PROGRAM(
+            [AC_INCLUDES_DEFAULT],
+            [[int result = 0;
+              char * volatile p = calloc ((size_t) -1 / 8 + 1, 8);
+              if (!p)
+                result |= 2;
+              free (p);
+              return result;
+            ]])],
+         dnl The exit code of this program is 0 if calloc() succeeded (which
+         dnl it shouldn't), 2 if calloc() failed, or 1 if some leak sanitizer
+         dnl terminated the program as a result of the calloc() call.
+         [ac_cv_func_calloc_0_nonnull=no],
+         [])
+     else
+       case "$host_os" in
+                        # Guess yes on glibc systems.
+         *-gnu* | gnu*) ac_cv_func_calloc_0_nonnull="guessing yes" ;;
+                        # Guess yes on musl systems.
+         *-musl*)       ac_cv_func_calloc_0_nonnull="guessing yes" ;;
+                        # Guess yes on native Windows.
+         mingw*)        ac_cv_func_calloc_0_nonnull="guessing yes" ;;
+                        # If we don't know, obey --enable-cross-guesses.
+         *)             ac_cv_func_calloc_0_nonnull="$gl_cross_guess_normal" ;;
+       esac
+     fi
+    ])
   case "$ac_cv_func_calloc_0_nonnull" in
     *yes)
       $1