xpswrite - limit the memory used by images to avoid memory exhaustion

OSS-fuzz 53398 (and others)

Because xpswrite uses libtiff to write images to XPS files, and libtiff
won't offer us any way to use our memory manager, it is possible to
get a corrupted image which tries to use silly amounts of memory, which
leads to system memory being exhausted.

In this commit; change the gs_memory_status_s structure to include the
'limit' from the memory allocator. Alter the various allocators to fill
in the value (or set it to -1 if there is no limit). In the xpswrite
device calculate the amount of memory needed for an uncompressed image,
if it becomes negative we know we broke a 64-bit signed integer, so
abort. If it didn't go negative, and the memory manager has a specified
limit, check to see if the image will fit in that. If not return
VMerror.

This prevents at least some of the memory exhaustion errors with
xpswrite and OSS-fuzz, because OSS-fuzz sets -K to limit memory use.

1 files changed, 29 insertions, 0 deletions

diff --git a/devices/vector/gdevxps.c b/devices/vector/gdevxps.c
index d0f02f5d0..af0b32710 100644
--- a/devices/vector/gdevxps.c
+++ b/devices/vector/gdevxps.c
@@ -2102,6 +2102,35 @@ xps_begin_typed_image(gx_device               *dev,
 
     num_components = gs_color_space_num_components(pcs);
     bits_per_pixel = pim->BitsPerComponent * num_components;
+
+    {
+        /* This is a really hacky attempt to avoid running out of memory. This is
+         * inspired by various OSS-fuzz bugs but in particular 53398. This device
+         * uses the libtiff library to write images into the XPS file, libtiff won't
+         * let us use our own memory manager and have rejected patches to allow it
+         * to do so, so it uses system memory. If we have a badly broken file we
+         * might end up asking it to write a ridiculously large image which will
+         * cause it to eat all system memory. So here we try to limit it to any pre-defined
+         * limit. Also, if the width * height * 24 bits (RGB) goes negative then we know
+         * we've exceeded 2^(64 - 1) bytes, which is unreasonable too.
+         */
+        int64_t memory_needed = (int64_t)pim->Width * (int64_t)pim->Height * 3;
+        gs_memory_status_t status;
+
+        if (memory_needed < 0) {
+            gs_free_object(mem, pie, "xps_begin_image");
+            return_error(gs_error_VMerror);
+        }
+
+        gs_memory_status(dev->memory->gs_lib_ctx->memory, &status);
+        if (status.limit < (size_t)~1) {
+            if ( memory_needed > status.limit) {
+                gs_free_object(mem, pie, "xps_begin_image");
+                return_error(gs_error_VMerror);
+            }
+        }
+    }
+
     pie->decode_st.bps = bits_per_pixel / num_components;
     pie->bytes_comp = (pie->decode_st.bps > 8 ? 2 : 1);
     pie->decode_st.spp = num_components;