MDEV-23320 Hex hybrid constants 0xHHHH work badly in rounding functions

- Type_handler_hex_hybrid did not override
  Type_handler_string_result::Item_func_round_fix_length_and_dec(),
  so the result type of ROUND(0xFFFFFFFFFFFFFFFF) was erroneously
  calculated ad DOUBLE with a wrong length.
  Overriding Item_func_round_fix_length_and_dec(), to calculated
  the result type as INT/BIGINT.

  Also, fixing Item_func_round::fix_arg_int() to use
  args[0]->decimal_precision() instead of args[0]->max_length
  when calculating this->max_length, to get a correct result
  for hex hybrids.

- Type_handler_hex_hybrid::Item_func_int_val_fix_length_and_dec()
  called item->fix_length_and_dec_int_or_decimal(), which did not
  produce a correct result data type for hex hybrid.
  Implementing a dedicated code instead, to return INT UNSIGNED or
  BIGINT UNSIGNED depending in the number of digits in the arguments.

1 files changed, 16 insertions, 1 deletions

diff --git a/sql/sql_type.cc b/sql/sql_type.cc
index 8b6d99de621..43b742604bc 100644
--- a/sql/sql_type.cc
+++ b/sql/sql_type.cc
@@ -5650,6 +5650,14 @@ bool Type_handler_int_result::
 }
 
 
+bool Type_handler_hex_hybrid::
+       Item_func_round_fix_length_and_dec(Item_func_round *item) const
+{
+  item->fix_arg_int();
+  return false;
+}
+
+
 bool Type_handler_real_result::
        Item_func_round_fix_length_and_dec(Item_func_round *item) const
 {
@@ -5743,7 +5751,14 @@ bool Type_handler_typelib::
 bool Type_handler_hex_hybrid::
        Item_func_int_val_fix_length_and_dec(Item_func_int_val *item) const
 {
-  item->fix_length_and_dec_int_or_decimal();
+  item->collation.set_numeric();
+  item->unsigned_flag= true;
+  item->max_length= item->arguments()[0]->decimal_precision();
+#if MARIADB_VERSION_ID < 100500
+  item->set_handler(type_handler_long_or_longlong(item->max_length));
+#else
+  item->set_handler(type_handler_long_or_longlong(item->max_length, true));
+#endif
   return false;
 }