GhostPDF - clamp unsigned integers to maximum unsigned int, not signed

Bug #706607 "Old-fashioned password security not working"

The problem turns out to be on Windows, where we clamp integers to the
maximum platform integer value during parsing. Windows builds use a
32-bit maximum integer and we were clamping all integers to the maximum
signed integer value.

Ordinarily this isn't a problem because we rarely encounter values
genuinely that large. However in this file the /P value is stored as a
32-bit unsigned integer, even though e spec clearly states that it is a
2s-complement form. So the value is stored as 4294963428 when in fact
it *should* be -3868. Acrobat, of course, happily opens either way.

Alter the integer overflow detection so that we clamp signed integers
at the signed maximum and unsigned integers at the unsigned maximum.
This allows us to read the value from the file and treat it as Acrobat does.

1 files changed, 2 insertions, 2 deletions

diff --git a/pdf/pdf_int.c b/pdf/pdf_int.c
index a5b6cd935..5a78f134c 100644
--- a/pdf/pdf_int.c
+++ b/pdf/pdf_int.c
@@ -236,7 +236,7 @@ static int pdfi_read_num(pdf_context *ctx, pdf_c_stream *s, uint32_t indirect_nu
     unsigned short exponent_index = 0;
     pdf_num *num;
     int code = 0, malformed = false, doubleneg = false, recovered = false, negative = false, overflowed = false;
-    int int_val = 0, tenth_max_int = max_int / 10;
+    int int_val = 0, tenth_max_int = max_int / 10, tenth_max_uint = max_uint / 10;
 
     pdfi_skip_white(ctx, s);
 
@@ -262,7 +262,7 @@ static int pdfi_read_num(pdf_context *ctx, pdf_c_stream *s, uint32_t indirect_nu
 
         if (c >= '0' && c <= '9') {
             if  (!(malformed && recovered) && !overflowed && !real) {
-                if (int_val < tenth_max_int)
+                if ((negative && (int_val < tenth_max_int)) || int_val < tenth_max_uint)
                     int_val = int_val*10 + c - '0';
                 else {
                     pdfi_set_error(ctx, 0, NULL, E_PDF_NUMBEROVERFLOW, "pdfi_read_num", NULL);