malloc, etc.: check for ptrdiff_t overflow

In glibc 2.30 and later, malloc, realloc and calloc reject
attempts to create objects larger than PTRDIFF_MAX bytes.
This patch changes malloc-gnu etc. to support this behavior
on non-GNU hosts.  It also makes this change for malloc-posix etc.
since it’s a safety measure that ought to be in POSIX (perhaps
we can talk them into that...).

In writing this patch I found a complicated set of code that had
accumulated over the years, some written by yours truly.  I got
rid of the code I couldn’t see the need for nowadays.  Among other
things, the GNU realloc behavior is no longer incompatible with
the C standard, because in C17 the latter was relaxed to allow the
former.  If I went too far in cleaning up, the old stuff can be
resurrected.

This change is mostly for 32-bit platforms, since practical 64-bit
platforms cannot create objects larger than PTRDIFF_MAX bytes anyway.
* doc/posix-functions/calloc.texi:
* doc/posix-functions/malloc.texi:
* doc/posix-functions/realloc.texi:
Mention ptrdiff_t issues, and go into more detail about what
the gnu extension module does.
* doc/posix-functions/realloc.texi: Fix now-obsolete commentary
about C99 vs glibc, as C17 allows the glibc behavior and POSIX
will follow suit when it gets around to it.
* lib/calloc.c, lib/malloc.c, lib/realloc.c:
Simplify by always supplying a GNU-compatible version,
as that suffices for correctness and is good enough for performance.
Include xalloc-oversized.h, and use xalloc_oversized to
check for ptrdiff_t overflow.
(NEED_CALLOC_GNU, NEED_MALLOC_GNU, NEED_REALLOC_GNU): Remove.
* m4/calloc.m4 (_AC_FUNC_CALLOC_IF):
* m4/malloc.m4 (_AC_FUNC_MALLOC_IF):
* m4/realloc.m4 (_AC_FUNC_REALLOC_IF):
Don’t start with a newline.  Fix message to match behavior.
* m4/calloc.m4 (_AC_FUNC_CALLOC_IF): Don’t test for size_t overflow,
as the ptrdiff_t test is good enough.
* m4/calloc.m4 (gl_FUNC_CALLOC_GNU):
* m4/malloc.m4 (gl_FUNC_MALLOC_GNU):
* m4/realloc.m4 (gl_FUNC_REALLOC_GNU):
Do not define HAVE_CALLOC_GNU, HAVE_MALLOC_GNU, HAVE_REALLOC_GNU.
It’s not worth the aggravation of maintaining these, as they
are confusing (they don’t really mean GNU-compatible anyway).
Don’t bother testing for GNU behavior if we have already decided
to replace the function, since the replacement is always GNUish.
* m4/calloc.m4 (gl_FUNC_CALLOC_POSIX):
* m4/realloc.m4 (gl_FUNC_REALLOC_POSIX):
Defer to gl_FUNC_MALLOC_POSIX.
* m4/malloc.m4 (gl_FUNC_MALLOC_PTRDIFF, gl_CHECK_MALLOC_PTRDIFF):
New macros.
(gl_FUNC_MALLOC_POSIX): Use them to check for ptrdiff_t overflow.
* modules/calloc-gnu, modules/malloc-gnu, modules/realloc-gnu:
Remove no-longer-needed module indicators.
* modules/calloc-posix, modules/malloc-posix, modules/realloc-posix:
Depend on xalloc-oversized.
* modules/malloc-posix: Require gl_FUNC_MALLOC_POSIX instead of
calling it directly, so that other code can require it.
* modules/realloc-posix: Depend on free-posix and malloc-posix.

3 files changed, 50 insertions, 21 deletions

diff --git a/doc/posix-functions/calloc.texi b/doc/posix-functions/calloc.texi
index 22c51be023..9ba40c0d48 100644
--- a/doc/posix-functions/calloc.texi
+++ b/doc/posix-functions/calloc.texi
@@ -12,11 +12,22 @@ Portability problems fixed by Gnulib:
 Upon failure, the function does not set @code{errno} to @code{ENOMEM} on
 some platforms:
 mingw, MSVC 14.
-@end itemize
 
-Portability problems not fixed by Gnulib:
-@itemize
+@item
+On some platforms, @code{calloc (n, s)} can succeed even if
+multiplying @code{n} by @code{s} would exceed @code{PTRDIFF_MAX} or
+@code{SIZE_MAX}.  Although failing to check for exceeding
+@code{PTRDIFF_MAX} is arguably allowed by POSIX it can lead to
+undefined behavior later, so @code{calloc-posix} does not allow
+going over the limit.
 @end itemize
 
 Extension: Gnulib provides a module @samp{calloc-gnu} that substitutes a
 @code{calloc} implementation that behaves more like the glibc implementation.
+It fixes this portability problem:
+
+@itemize
+@item
+On some platforms, @code{calloc (0, s)} and @code{calloc (n, 0)}
+return @code{NULL} on success.
+@end itemize
diff --git a/doc/posix-functions/malloc.texi b/doc/posix-functions/malloc.texi
index 6d5995078f..829517311f 100644
--- a/doc/posix-functions/malloc.texi
+++ b/doc/posix-functions/malloc.texi
@@ -12,14 +12,19 @@ Portability problems fixed by Gnulib:
 Upon failure, the function does not set @code{errno} to @code{ENOMEM} on
 some platforms:
 mingw, MSVC 14.
-@end itemize
 
-Portability problems not fixed by Gnulib:
-@itemize
-@item @code{malloc (0)} always returns a NULL pointer on some platforms:
-AIX 5.1.
+@item
+On some platforms, @code{malloc (n)} can succeed even if @code{n}
+exceeds @code{PTRDIFF_MAX}.  Although this behavior is arguably
+allowed by POSIX it can lead to behavior not defined by POSIX later,
+so @code{malloc-posix} does not allow going over the limit.
 @end itemize
 
 Extension: Gnulib provides a module @samp{malloc-gnu} that substitutes a
 @code{malloc} implementation that behaves more like the glibc implementation,
-regarding the result of @code{malloc (0)}.
+by fixing this portability problem:
+
+@itemize
+@item
+On some platforms, @code{malloc (0)} returns @code{NULL} on success.
+@end itemize
diff --git a/doc/posix-functions/realloc.texi b/doc/posix-functions/realloc.texi
index 1f0b18e952..282e36051d 100644
--- a/doc/posix-functions/realloc.texi
+++ b/doc/posix-functions/realloc.texi
@@ -12,24 +12,37 @@ Portability problems fixed by Gnulib:
 Upon failure, the function does not set @code{errno} to @code{ENOMEM} on
 some platforms:
 mingw, MSVC 14.
-@end itemize
 
-Portability problems not fixed by Gnulib:
-@itemize
 @item
-It is not portable to call @code{realloc} with a size of 0.  With a
+On some platforms, @code{realloc (p, n)} can succeed even if @code{n}
+exceeds @code{PTRDIFF_MAX}.  Although this behavior is arguably
+allowed by POSIX it can lead to behavior not defined by POSIX later,
+so @code{realloc-posix} does not allow going over the limit.
+@end itemize
+
+Without the @samp{realloc-gnu} module described below, it is not portable
+to call @code{realloc} with a size of 0.  With a
 NULL pointer argument, this is the same ambiguity as @code{malloc (0)}
 on whether a unique zero-size object is created.  With a non-NULL
-pointer argument, C99 requires that if @code{realloc (p, 0)} returns
-@code{NULL} then @code{p} is still valid.  Among implementations that
-obey C99, behavior varies on whether @code{realloc (p, 0)} always
+pointer argument @code{p}, C17 says that it is implementation-defined
+whether @code{realloc (p, 0)} frees @code{p}.
+Behavior varies on whether @code{realloc (p, 0)} always frees @code{p}
+and successfully returns a null pointer, or always
 fails and leaves @code{p} valid, or usually succeeds and returns a
-unique zero-size object; either way, a program not suspecting these
+unique zero-size object; a program not suspecting these variations in
 semantics will leak memory (either the still-valid @code{p}, or the
-non-NULL return value).  Meanwhile, several implementations violate
-C99, by always calling @code{free (p)} but returning NULL:
-glibc, Cygwin
-@end itemize
+non-NULL return value).
 
 Extension: Gnulib provides a module @samp{realloc-gnu} that substitutes a
 @code{realloc} implementation that behaves more like the glibc implementation.
+It fixes these portability problems:
+
+@itemize
+@item
+On some platforms, @code{realloc (NULL, 0)} returns @code{NULL} on success.
+
+@item
+On some platforms, @code{realloc (p, 0)} with non-null @code{p}
+might not free @code{p}, or might clobber @code{errno},
+or might not return @code{NULL}.
+@end itemize