malloc, etc.: check for ptrdiff_t overflow

In glibc 2.30 and later, malloc, realloc and calloc reject
attempts to create objects larger than PTRDIFF_MAX bytes.
This patch changes malloc-gnu etc. to support this behavior
on non-GNU hosts.  It also makes this change for malloc-posix etc.
since it’s a safety measure that ought to be in POSIX (perhaps
we can talk them into that...).

In writing this patch I found a complicated set of code that had
accumulated over the years, some written by yours truly.  I got
rid of the code I couldn’t see the need for nowadays.  Among other
things, the GNU realloc behavior is no longer incompatible with
the C standard, because in C17 the latter was relaxed to allow the
former.  If I went too far in cleaning up, the old stuff can be
resurrected.

This change is mostly for 32-bit platforms, since practical 64-bit
platforms cannot create objects larger than PTRDIFF_MAX bytes anyway.
* doc/posix-functions/calloc.texi:
* doc/posix-functions/malloc.texi:
* doc/posix-functions/realloc.texi:
Mention ptrdiff_t issues, and go into more detail about what
the gnu extension module does.
* doc/posix-functions/realloc.texi: Fix now-obsolete commentary
about C99 vs glibc, as C17 allows the glibc behavior and POSIX
will follow suit when it gets around to it.
* lib/calloc.c, lib/malloc.c, lib/realloc.c:
Simplify by always supplying a GNU-compatible version,
as that suffices for correctness and is good enough for performance.
Include xalloc-oversized.h, and use xalloc_oversized to
check for ptrdiff_t overflow.
(NEED_CALLOC_GNU, NEED_MALLOC_GNU, NEED_REALLOC_GNU): Remove.
* m4/calloc.m4 (_AC_FUNC_CALLOC_IF):
* m4/malloc.m4 (_AC_FUNC_MALLOC_IF):
* m4/realloc.m4 (_AC_FUNC_REALLOC_IF):
Don’t start with a newline.  Fix message to match behavior.
* m4/calloc.m4 (_AC_FUNC_CALLOC_IF): Don’t test for size_t overflow,
as the ptrdiff_t test is good enough.
* m4/calloc.m4 (gl_FUNC_CALLOC_GNU):
* m4/malloc.m4 (gl_FUNC_MALLOC_GNU):
* m4/realloc.m4 (gl_FUNC_REALLOC_GNU):
Do not define HAVE_CALLOC_GNU, HAVE_MALLOC_GNU, HAVE_REALLOC_GNU.
It’s not worth the aggravation of maintaining these, as they
are confusing (they don’t really mean GNU-compatible anyway).
Don’t bother testing for GNU behavior if we have already decided
to replace the function, since the replacement is always GNUish.
* m4/calloc.m4 (gl_FUNC_CALLOC_POSIX):
* m4/realloc.m4 (gl_FUNC_REALLOC_POSIX):
Defer to gl_FUNC_MALLOC_POSIX.
* m4/malloc.m4 (gl_FUNC_MALLOC_PTRDIFF, gl_CHECK_MALLOC_PTRDIFF):
New macros.
(gl_FUNC_MALLOC_POSIX): Use them to check for ptrdiff_t overflow.
* modules/calloc-gnu, modules/malloc-gnu, modules/realloc-gnu:
Remove no-longer-needed module indicators.
* modules/calloc-posix, modules/malloc-posix, modules/realloc-posix:
Depend on xalloc-oversized.
* modules/malloc-posix: Require gl_FUNC_MALLOC_POSIX instead of
calling it directly, so that other code can require it.
* modules/realloc-posix: Depend on free-posix and malloc-posix.

3 files changed, 86 insertions, 92 deletions

diff --git a/m4/calloc.m4 b/m4/calloc.m4
index eeeb033d8b..143f3bc3f8 100644
--- a/m4/calloc.m4
+++ b/m4/calloc.m4
@@ -1,4 +1,4 @@
-# calloc.m4 serial 23
+# calloc.m4 serial 24
 
 # Copyright (C) 2004-2021 Free Software Foundation, Inc.
 # This file is free software; the Free Software Foundation
@@ -14,12 +14,10 @@
 
 # _AC_FUNC_CALLOC_IF([IF-WORKS], [IF-NOT])
 # -------------------------------------
-# If 'calloc (0, 0)' is properly handled, run IF-WORKS, otherwise, IF-NOT.
+# If calloc is compatible with GNU calloc, run IF-WORKS, otherwise, IF-NOT.
 AC_DEFUN([_AC_FUNC_CALLOC_IF],
-[
-  AC_REQUIRE([AC_TYPE_SIZE_T])dnl
-  AC_REQUIRE([AC_CANONICAL_HOST]) dnl for cross-compiles
-  AC_CACHE_CHECK([for GNU libc compatible calloc],
+[ AC_REQUIRE([AC_CANONICAL_HOST]) dnl for cross-compiles
+  AC_CACHE_CHECK([whether calloc (0, n) and calloc (n, 0) return nonnull],
     [ac_cv_func_calloc_0_nonnull],
     [if test $cross_compiling != yes; then
        ac_cv_func_calloc_0_nonnull=yes
@@ -35,32 +33,6 @@ AC_DEFUN([_AC_FUNC_CALLOC_IF],
             ]])],
          [],
          [ac_cv_func_calloc_0_nonnull=no])
-       AC_RUN_IFELSE(
-         [AC_LANG_PROGRAM(
-            [AC_INCLUDES_DEFAULT],
-            [[int result;
-              typedef struct { char c[8]; } S8;
-              size_t n = (size_t) -1 / sizeof (S8) + 2;
-              S8 * volatile s = calloc (n, sizeof (S8));
-              if (s)
-                {
-                  s[0].c[0] = 1;
-                  if (s[n - 1].c[0])
-                    result = 0;
-                  else
-                    result = 2;
-                }
-              else
-                result = 3;
-              free (s);
-              return result;
-            ]])],
-         dnl The exit code of this program is 0 if calloc() succeeded with a
-         dnl wrap-around bug (which it shouldn't), 2 if calloc() succeeded in
-         dnl a non-flat address space, 3 if calloc() failed, or 1 if some leak
-         dnl sanitizer terminated the program as a result of the calloc() call.
-         [ac_cv_func_calloc_0_nonnull=no],
-         [])
      else
        case "$host_os" in
                         # Guess yes on glibc systems.
@@ -82,38 +54,31 @@ AC_DEFUN([_AC_FUNC_CALLOC_IF],
       $2
       ;;
   esac
-])# AC_FUNC_CALLOC
+])
 
 
 # gl_FUNC_CALLOC_GNU
 # ------------------
-# Report whether 'calloc (0, 0)' is properly handled, and replace calloc if
-# needed.
+# Replace calloc if it is not compatible with GNU libc.
 AC_DEFUN([gl_FUNC_CALLOC_GNU],
 [
   AC_REQUIRE([gl_STDLIB_H_DEFAULTS])
-  _AC_FUNC_CALLOC_IF(
-    [AC_DEFINE([HAVE_CALLOC_GNU], [1],
-               [Define to 1 if your system has a GNU libc compatible 'calloc'
-                function, and to 0 otherwise.])],
-    [AC_DEFINE([HAVE_CALLOC_GNU], [0])
-     REPLACE_CALLOC=1
-    ])
+  AC_REQUIRE([gl_FUNC_CALLOC_POSIX])
+  test $REPLACE_CALLOC = 1 || _AC_FUNC_CALLOC_IF([], [REPLACE_CALLOC=1])
 ])# gl_FUNC_CALLOC_GNU
 
-
 # gl_FUNC_CALLOC_POSIX
 # --------------------
 # Test whether 'calloc' is POSIX compliant (sets errno to ENOMEM when it
-# fails), and replace calloc if it is not.
+# fails, and doesn't mess up with ptrdiff_t or size_t overflow),
+# and replace calloc if it is not.
 AC_DEFUN([gl_FUNC_CALLOC_POSIX],
 [
   AC_REQUIRE([gl_STDLIB_H_DEFAULTS])
-  AC_REQUIRE([gl_CHECK_MALLOC_POSIX])
-  if test $gl_cv_func_malloc_posix = yes; then
-    AC_DEFINE([HAVE_CALLOC_POSIX], [1],
-      [Define if the 'calloc' function is POSIX compliant.])
-  else
-    REPLACE_CALLOC=1
-  fi
+  AC_REQUIRE([gl_FUNC_MALLOC_POSIX])
+  REPLACE_CALLOC=$REPLACE_MALLOC
+  dnl Although in theory we should also test for size_t overflow,
+  dnl in practice testing for ptrdiff_t overflow suffices
+  dnl since PTRDIFF_MAX <= SIZE_MAX on all known Gnulib porting targets.
+  dnl A separate size_t test would slow down 'configure'.
 ])
diff --git a/m4/malloc.m4 b/m4/malloc.m4
index 32ab42ec09..503da2cf87 100644
--- a/m4/malloc.m4
+++ b/m4/malloc.m4
@@ -1,4 +1,4 @@
-# malloc.m4 serial 22
+# malloc.m4 serial 23
 dnl Copyright (C) 2007, 2009-2021 Free Software Foundation, Inc.
 dnl This file is free software; the Free Software Foundation
 dnl gives unlimited permission to copy and/or distribute it,
@@ -7,9 +7,8 @@ dnl with or without modifications, as long as this notice is preserved.
 # This is adapted with modifications from upstream Autoconf here:
 # https://git.savannah.gnu.org/cgit/autoconf.git/commit/?id=04be2b7a29d65d9a08e64e8e56e594c91749598c
 AC_DEFUN([_AC_FUNC_MALLOC_IF],
-[
-  AC_REQUIRE([AC_CANONICAL_HOST])dnl for cross-compiles
-  AC_CACHE_CHECK([for GNU libc compatible malloc],
+[ AC_REQUIRE([AC_CANONICAL_HOST])dnl for cross-compiles
+  AC_CACHE_CHECK([whether malloc (0) returns nonnull],
     [ac_cv_func_malloc_0_nonnull],
     [AC_RUN_IFELSE(
        [AC_LANG_PROGRAM(
@@ -44,47 +43,89 @@ AC_DEFUN([_AC_FUNC_MALLOC_IF],
 
 # gl_FUNC_MALLOC_GNU
 # ------------------
-# Test whether 'malloc (0)' is handled like in GNU libc, and replace malloc if
-# it is not.
+# Replace malloc if it is not compatible with GNU libc.
 AC_DEFUN([gl_FUNC_MALLOC_GNU],
 [
   AC_REQUIRE([gl_STDLIB_H_DEFAULTS])
-  dnl _AC_FUNC_MALLOC_IF is defined in Autoconf.
-  _AC_FUNC_MALLOC_IF(
-    [AC_DEFINE([HAVE_MALLOC_GNU], [1],
-               [Define to 1 if your system has a GNU libc compatible 'malloc'
-                function, and to 0 otherwise.])],
-    [AC_DEFINE([HAVE_MALLOC_GNU], [0])
-     REPLACE_MALLOC=1
+  AC_REQUIRE([gl_FUNC_MALLOC_POSIX])
+  test $REPLACE_MALLOC = 1 || _AC_FUNC_MALLOC_IF([], [REPLACE_MALLOC=1])
+])
+
+# gl_FUNC_MALLOC_PTRDIFF
+# ----------------------
+# Test whether malloc (N) reliably fails when N exceeds PTRDIFF_MAX,
+# and replace malloc otherwise.
+AC_DEFUN([gl_FUNC_MALLOC_PTRDIFF],
+[
+  AC_REQUIRE([gl_STDLIB_H_DEFAULTS])
+  AC_REQUIRE([gl_CHECK_MALLOC_PTRDIFF])
+  test "$gl_cv_malloc_ptrdiff" = yes || REPLACE_MALLOC=1
+])
+
+# Test whether malloc, realloc, calloc refuse to create objects
+# larger than what can be expressed in ptrdiff_t.
+# Set gl_cv_func_malloc_gnu to yes or no accordingly.
+AC_DEFUN([gl_CHECK_MALLOC_PTRDIFF],
+[
+  AC_CACHE_CHECK([whether malloc is ptrdiff_t safe],
+    [gl_cv_malloc_ptrdiff],
+    [AC_COMPILE_IFELSE(
+       [AC_LANG_PROGRAM(
+          [[#include <stdint.h>
+          ]],
+          [[/* 64-bit ptrdiff_t is so wide that no practical platform
+               can exceed it.  */
+            #define WIDE_PTRDIFF (PTRDIFF_MAX >> 31 >> 31 != 0)
+
+            /* On rare machines where size_t fits in ptrdiff_t there
+               is no problem.  */
+            #define NARROW_SIZE (SIZE_MAX <= PTRDIFF_MAX)
+
+            /* glibc 2.30 and later malloc refuses to exceed ptrdiff_t
+               bounds even on 32-bit platforms.  We don't know which
+               non-glibc systems are safe.  */
+            #define KNOWN_SAFE (2 < __GLIBC__ + (30 <= __GLIBC_MINOR__))
+
+            #if WIDE_PTRDIFF || NARROW_SIZE || KNOWN_SAFE
+              return 0;
+            #else
+              #error "malloc might not be ptrdiff_t safe"
+              syntax error
+            #endif
+          ]])],
+       [gl_cv_malloc_ptrdiff=yes],
+       [gl_cv_malloc_ptrdiff=no])
     ])
 ])
 
 # gl_FUNC_MALLOC_POSIX
 # --------------------
 # Test whether 'malloc' is POSIX compliant (sets errno to ENOMEM when it
-# fails), and replace malloc if it is not.
+# fails, and doesn't mess up with ptrdiff_t overflow), and replace
+# malloc if it is not.
 AC_DEFUN([gl_FUNC_MALLOC_POSIX],
 [
   AC_REQUIRE([gl_STDLIB_H_DEFAULTS])
+  AC_REQUIRE([gl_FUNC_MALLOC_PTRDIFF])
   AC_REQUIRE([gl_CHECK_MALLOC_POSIX])
-  if test $gl_cv_func_malloc_posix = yes; then
+  if test "$gl_cv_func_malloc_posix" = yes; then
     AC_DEFINE([HAVE_MALLOC_POSIX], [1],
-      [Define if the 'malloc' function is POSIX compliant.])
+      [Define if malloc, realloc, and calloc set errno on allocation failure.])
   else
     REPLACE_MALLOC=1
   fi
 ])
 
-# Test whether malloc, realloc, calloc are POSIX compliant,
+# Test whether malloc, realloc, calloc set errno on failure.
 # Set gl_cv_func_malloc_posix to yes or no accordingly.
 AC_DEFUN([gl_CHECK_MALLOC_POSIX],
 [
-  AC_CACHE_CHECK([whether malloc, realloc, calloc are POSIX compliant],
+  AC_CACHE_CHECK([whether malloc, realloc, calloc set errno on failure],
     [gl_cv_func_malloc_posix],
     [
       dnl It is too dangerous to try to allocate a large amount of memory:
       dnl some systems go to their knees when you do that. So assume that
-      dnl all Unix implementations of the function are POSIX compliant.
+      dnl all Unix implementations of the function set errno on failure.
       AC_COMPILE_IFELSE(
         [AC_LANG_PROGRAM(
            [[]],
diff --git a/m4/realloc.m4 b/m4/realloc.m4
index a80a02a6bc..4939516a09 100644
--- a/m4/realloc.m4
+++ b/m4/realloc.m4
@@ -1,4 +1,4 @@
-# realloc.m4 serial 20
+# realloc.m4 serial 21
 dnl Copyright (C) 2007, 2009-2021 Free Software Foundation, Inc.
 dnl This file is free software; the Free Software Foundation
 dnl gives unlimited permission to copy and/or distribute it,
@@ -7,9 +7,8 @@ dnl with or without modifications, as long as this notice is preserved.
 # This is adapted with modifications from upstream Autoconf here:
 # https://git.savannah.gnu.org/cgit/autoconf.git/commit/?id=04be2b7a29d65d9a08e64e8e56e594c91749598c
 AC_DEFUN([_AC_FUNC_REALLOC_IF],
-[
-  AC_REQUIRE([AC_CANONICAL_HOST])dnl for cross-compiles
-  AC_CACHE_CHECK([for GNU libc compatible realloc],
+[ AC_REQUIRE([AC_CANONICAL_HOST])dnl for cross-compiles
+  AC_CACHE_CHECK([whether realloc (0, 0) returns nonnull],
     [ac_cv_func_realloc_0_nonnull],
     [AC_RUN_IFELSE(
        [AC_LANG_PROGRAM(
@@ -44,33 +43,22 @@ AC_DEFUN([_AC_FUNC_REALLOC_IF],
 
 # gl_FUNC_REALLOC_GNU
 # -------------------
-# Test whether 'realloc (0, 0)' is handled like in GNU libc, and replace
-# realloc if it is not.
+# Replace realloc if it is not compatible with GNU libc.
 AC_DEFUN([gl_FUNC_REALLOC_GNU],
 [
   AC_REQUIRE([gl_STDLIB_H_DEFAULTS])
-  dnl _AC_FUNC_REALLOC_IF is defined in Autoconf.
-  _AC_FUNC_REALLOC_IF(
-    [AC_DEFINE([HAVE_REALLOC_GNU], [1],
-               [Define to 1 if your system has a GNU libc compatible 'realloc'
-                function, and to 0 otherwise.])],
-    [AC_DEFINE([HAVE_REALLOC_GNU], [0])
-     REPLACE_REALLOC=1
-    ])
+  AC_REQUIRE([gl_FUNC_REALLOC_POSIX])
+  test $REPLACE_REALLOC = 1 || _AC_FUNC_REALLOC_IF([], [REPLACE_REALLOC=1])
 ])# gl_FUNC_REALLOC_GNU
 
 # gl_FUNC_REALLOC_POSIX
 # ---------------------
 # Test whether 'realloc' is POSIX compliant (sets errno to ENOMEM when it
-# fails), and replace realloc if it is not.
+# fails, and doesn't mess up with ptrdiff_t overflow),
+# and replace realloc if it is not.
 AC_DEFUN([gl_FUNC_REALLOC_POSIX],
 [
   AC_REQUIRE([gl_STDLIB_H_DEFAULTS])
-  AC_REQUIRE([gl_CHECK_MALLOC_POSIX])
-  if test $gl_cv_func_malloc_posix = yes; then
-    AC_DEFINE([HAVE_REALLOC_POSIX], [1],
-      [Define if the 'realloc' function is POSIX compliant.])
-  else
-    REPLACE_REALLOC=1
-  fi
+  AC_REQUIRE([gl_FUNC_MALLOC_POSIX])
+  REPLACE_REALLOC=$REPLACE_MALLOC
 ])