Bug #40761: Assert on sum function on

            IF(..., CAST(longtext AS UNSIGNED), signed_val)
            (was: LEFT JOIN on inline view crashes server)

Select from a LONGTEXT column wrapped with an expression
like "IF(..., CAST(longtext_column AS UNSIGNED), smth_signed)"
failed an assertion or crashed the server. IFNULL function was
affected too.

LONGTEXT column item has a maximum length of 32^2-1 bytes,
at the same time this is a maximum possible length of any
MySQL item. CAST(longtext_column AS UNSIGNED) returns some
unsigned numeric result of length 32^2-1, so the result of
IF/IFNULL function of this number and some other signed number
will have text length of (32^2-1)+1=32^2 (one byte for the
minus sign) - there is integer overflow, and the length is
equal to zero. That caused assert/crash.

CAST AS UNSIGNED function has been modified to limit maximal
length of resulting number to 67 (maximal length of DECIMAL
and two characters for minus sign and dot).


mysql-test/r/func_if.result:
  Added test case for bug #40761.
mysql-test/t/func_if.test:
  Added test case for bug #40761.
sql/item_func.h:
  Bug #40761: Assert on sum function on
              IF(..., CAST(longtext AS UNSIGNED), signed_val)
  
  CAST AS UNSIGNED function has been modified to limit maximal
  length of resulting number to 67 (maximal length of DECIMAL
  and two characters for minus sign and dot).

1 files changed, 14 insertions, 0 deletions

diff --git a/mysql-test/t/func_if.test b/mysql-test/t/func_if.test
index 8da10f36cbe..4efea8e195e 100644
--- a/mysql-test/t/func_if.test
+++ b/mysql-test/t/func_if.test
@@ -150,4 +150,18 @@ FROM t1;
 
 DROP TABLE t1;
 
+#
+# Bug #40761: Assert on sum func on IF(..., CAST(longtext AS UNSIGNED), signed)
+#             (was: LEFT JOIN on inline view crashes server)
+#
+
+CREATE TABLE t1 (c LONGTEXT);
+INSERT INTO t1 VALUES(1), (2), (3), (4), ('12345678901234567890');
+
+SELECT * FROM (SELECT MAX(IF(1, CAST(c AS UNSIGNED), 0)) FROM t1) AS te;
+SELECT * FROM (SELECT MAX(IFNULL(CAST(c AS UNSIGNED), 0)) FROM t1) AS te;
+
+DROP TABLE t1;
+
+
 --echo End of 5.0 tests