malloc, etc.: check for ptrdiff_t overflow

In glibc 2.30 and later, malloc, realloc and calloc reject
attempts to create objects larger than PTRDIFF_MAX bytes.
This patch changes malloc-gnu etc. to support this behavior
on non-GNU hosts.  It also makes this change for malloc-posix etc.
since it’s a safety measure that ought to be in POSIX (perhaps
we can talk them into that...).

In writing this patch I found a complicated set of code that had
accumulated over the years, some written by yours truly.  I got
rid of the code I couldn’t see the need for nowadays.  Among other
things, the GNU realloc behavior is no longer incompatible with
the C standard, because in C17 the latter was relaxed to allow the
former.  If I went too far in cleaning up, the old stuff can be
resurrected.

This change is mostly for 32-bit platforms, since practical 64-bit
platforms cannot create objects larger than PTRDIFF_MAX bytes anyway.
* doc/posix-functions/calloc.texi:
* doc/posix-functions/malloc.texi:
* doc/posix-functions/realloc.texi:
Mention ptrdiff_t issues, and go into more detail about what
the gnu extension module does.
* doc/posix-functions/realloc.texi: Fix now-obsolete commentary
about C99 vs glibc, as C17 allows the glibc behavior and POSIX
will follow suit when it gets around to it.
* lib/calloc.c, lib/malloc.c, lib/realloc.c:
Simplify by always supplying a GNU-compatible version,
as that suffices for correctness and is good enough for performance.
Include xalloc-oversized.h, and use xalloc_oversized to
check for ptrdiff_t overflow.
(NEED_CALLOC_GNU, NEED_MALLOC_GNU, NEED_REALLOC_GNU): Remove.
* m4/calloc.m4 (_AC_FUNC_CALLOC_IF):
* m4/malloc.m4 (_AC_FUNC_MALLOC_IF):
* m4/realloc.m4 (_AC_FUNC_REALLOC_IF):
Don’t start with a newline.  Fix message to match behavior.
* m4/calloc.m4 (_AC_FUNC_CALLOC_IF): Don’t test for size_t overflow,
as the ptrdiff_t test is good enough.
* m4/calloc.m4 (gl_FUNC_CALLOC_GNU):
* m4/malloc.m4 (gl_FUNC_MALLOC_GNU):
* m4/realloc.m4 (gl_FUNC_REALLOC_GNU):
Do not define HAVE_CALLOC_GNU, HAVE_MALLOC_GNU, HAVE_REALLOC_GNU.
It’s not worth the aggravation of maintaining these, as they
are confusing (they don’t really mean GNU-compatible anyway).
Don’t bother testing for GNU behavior if we have already decided
to replace the function, since the replacement is always GNUish.
* m4/calloc.m4 (gl_FUNC_CALLOC_POSIX):
* m4/realloc.m4 (gl_FUNC_REALLOC_POSIX):
Defer to gl_FUNC_MALLOC_POSIX.
* m4/malloc.m4 (gl_FUNC_MALLOC_PTRDIFF, gl_CHECK_MALLOC_PTRDIFF):
New macros.
(gl_FUNC_MALLOC_POSIX): Use them to check for ptrdiff_t overflow.
* modules/calloc-gnu, modules/malloc-gnu, modules/realloc-gnu:
Remove no-longer-needed module indicators.
* modules/calloc-posix, modules/malloc-posix, modules/realloc-posix:
Depend on xalloc-oversized.
* modules/malloc-posix: Require gl_FUNC_MALLOC_POSIX instead of
calling it directly, so that other code can require it.
* modules/realloc-posix: Depend on free-posix and malloc-posix.

1 files changed, 16 insertions, 39 deletions

diff --git a/lib/realloc.c b/lib/realloc.c
index c3e3cdfc55..c0d94b4399 100644
--- a/lib/realloc.c
+++ b/lib/realloc.c
@@ -21,66 +21,43 @@
 #define _GL_USE_STDLIB_ALLOC 1
 #include <config.h>
 
-/* The gnulib module 'realloc-gnu' defines HAVE_REALLOC_GNU.  */
-#if GNULIB_REALLOC_GNU && !HAVE_REALLOC_GNU
-# define NEED_REALLOC_GNU 1
-#endif
-
-/* Infer the properties of the system's malloc function.
-   The gnulib module 'malloc-gnu' defines HAVE_MALLOC_GNU.  */
-#if GNULIB_MALLOC_GNU && HAVE_MALLOC_GNU
-# define SYSTEM_MALLOC_GLIBC_COMPATIBLE 1
-#endif
-
 #include <stdlib.h>
 
-/* A function definition is only needed if NEED_REALLOC_GNU is defined above
-   or if the module 'realloc-posix' requests it.  */
-#if NEED_REALLOC_GNU || (GNULIB_REALLOC_POSIX && !HAVE_REALLOC_POSIX)
+#include <errno.h>
 
-# include <errno.h>
+#include "xalloc-oversized.h"
 
-/* Call the system's malloc and realloc below.  */
-# undef malloc
-# undef realloc
+/* Call the system's realloc below.  */
+#undef realloc
 
 /* Change the size of an allocated block of memory P to N bytes,
-   with error checking.  If N is zero, change it to 1.  If P is NULL,
-   use malloc.  */
+   with error checking.  If P is NULL, use malloc.  Otherwise if N is zero,
+   free P and return NULL.  */
 
 void *
 rpl_realloc (void *p, size_t n)
 {
-  void *result;
+  if (p == NULL)
+    return malloc (n);
 
-# if NEED_REALLOC_GNU
   if (n == 0)
     {
-      n = 1;
-
-      /* In theory realloc might fail, so don't rely on it to free.  */
       free (p);
-      p = NULL;
+      return NULL;
     }
-# endif
 
-  if (p == NULL)
+  if (xalloc_oversized (n, 1))
     {
-# if GNULIB_REALLOC_GNU && !NEED_REALLOC_GNU && !SYSTEM_MALLOC_GLIBC_COMPATIBLE
-      if (n == 0)
-        n = 1;
-# endif
-      result = malloc (n);
+      errno = ENOMEM;
+      return NULL;
     }
-  else
-    result = realloc (p, n);
 
-# if !HAVE_REALLOC_POSIX
+  void *result = realloc (p, n);
+
+#if !HAVE_MALLOC_POSIX
   if (result == NULL)
     errno = ENOMEM;
-# endif
+#endif
 
   return result;
 }
-
-#endif