MDEV-9745 Crash with CASE WHEN TRUE THEN COALESCE(CAST(NULL AS UNSIGNED)) ELSE 4 END

This is a backport of the patch for MDEV-9653 (fixed earlier in 10.1.13).

The code in Item_func_case::fix_length_and_dec() did not
calculate max_length and decimals properly.

In case of any numeric result (DECIMAL, REAL, INT) a generic method
Item_func_case::agg_num_lengths() was called, which could erroneously result
into a DECIMAL item with max_length==0 and decimals==0, so the constructor of
Field_new_decimals tried to create a field of DECIMAL(0,0) type,
which caused a crash.

Unlike Item_func_case, the code responsible for merging attributes in
Item_func_coalesce::fix_length_and_dec() works fine: it has specific execution
branches for all distinct numeric types and correctly creates a DECIMAL(1,0)
column instead of DECIMAL(0,0) for the same set of arguments.

The fix does the following:
- Moves the attribute merging code from Item_func_coalesce::fix_length_and_dec()
  to a new method Item_func_hybrid_result_type::fix_attributes()
- Removes the wrong code from Item_func_case::fix_length_and_dec()
  and reuses fix_attributes() in both Item_func_coalesce::fix_length_and_dec()
  and Item_func_case::fix_length_and_dec()
- Fixes count_real_length() and count_decimal_length() to get an array
  of Items as an argument, instead of using Item::args directly.
  This is needed for Item_func_case::fix_length_and_dec().
- Moves methods Item_func::count_xxx_length() from "public" to "protected".
- Removes Item_func_case::agg_num_length(), as it's not used any more.
- Additionally removes Item_func_case::agg_str_length(),
  as it also was not used (dead code).

1 files changed, 9 insertions, 0 deletions

diff --git a/mysql-test/t/case.test b/mysql-test/t/case.test
index f536f556780..c127836d352 100644
--- a/mysql-test/t/case.test
+++ b/mysql-test/t/case.test
@@ -193,3 +193,12 @@ insert t1 values ('00:00:00'),('00:01:00');
 select case t1.f1 when '00:00:00' then 1 end from t1;
 drop table t1;
 
+--echo #
+--echo # MDEV-9745 Crash with CASE WHEN TRUE THEN COALESCE(CAST(NULL AS UNSIGNED)) ELSE 4 END
+--echo #
+CREATE TABLE t1 SELECT CASE WHEN TRUE THEN COALESCE(CAST(NULL AS UNSIGNED)) ELSE 4 END AS a;
+DESCRIBE t1;
+DROP TABLE t1;
+CREATE TABLE t1 SELECT CASE WHEN TRUE THEN COALESCE(CAST(NULL AS UNSIGNED)) ELSE 40 END AS a;
+DESCRIBE t1;
+DROP TABLE t1;