Handle all negative int64_t values in mtree/tar

The variable last_digit_limit is negative since INT64_MIN itself is
negative as well. This means that the last digit after "limit" always
leads to maxval.

Turning last_digit_limit positive in itself is not sufficient because
it would lead to a signed integer overflow during shift operation.

If limit is reached and the last digit is last_digit_limit, the number
is at least maxval. The already existing if condition for even larger
(or smaller) values can be reused to prevent the last shift.

In my humble opinion it might make sense to reduce duplicated code and
keep it separated in a utility source file for shared use.

1 files changed, 2 insertions, 2 deletions

diff --git a/libarchive/archive_read_support_format_tar.c b/libarchive/archive_read_support_format_tar.c
index 96d81018..44dca01e 100644
--- a/libarchive/archive_read_support_format_tar.c
+++ b/libarchive/archive_read_support_format_tar.c
@@ -2643,14 +2643,14 @@ tar_atol_base_n(const char *p, size_t char_cnt, int base)
 
 		maxval = INT64_MIN;
 		limit = -(INT64_MIN / base);
-		last_digit_limit = INT64_MIN % base;
+		last_digit_limit = -(INT64_MIN % base);
 	}
 
 	l = 0;
 	if (char_cnt != 0) {
 		digit = *p - '0';
 		while (digit >= 0 && digit < base  && char_cnt != 0) {
-			if (l>limit || (l == limit && digit > last_digit_limit)) {
+			if (l>limit || (l == limit && digit >= last_digit_limit)) {
 				return maxval; /* Truncate on overflow. */
 			}
 			l = (l * base) + digit;