MDEV-23323 Rounding functions return a wrong data type for a BIT, ENUM, SET argument

Implementing dedicated fixing methods:
- Type_handler_bit::Item_func_round_fix_length_and_dec()
- Type_handler_bit::Item_func_int_val_fix_length_and_dec()
- Type_handler_typelib::Item_func_round_fix_length_and_dec()

because the inherited methods did not work well.

Fixing:
- Type_handler_typelib::Item_func_int_val_fix_length_and_dec
  It did not work well, because it used args[0]->max_length to
  calculate the result data type. In case of ENUM and SET it was
  not correct, because in FLOOR() and CEILING() context
  ENUM and SET return not more than 5 digits (65535 is the biggest
  possible value).

Misc:
- Changing the API of
    Type_handler_bit::Bit_decimal_notation_int_digits(const Item *item)
  to a more generic form:
    Type_handler_bit::Bit_decimal_notation_int_digits_by_nbits(uint nbits)

- Fixing Type_handler_bit::Bit_decimal_notation_int_digits_by_nbits() to
  return the exact number of decimal digits for all nbits 1..64.
  The old implementation was approximate.
  This change gives better (more precise) data types.

1 files changed, 6 insertions, 2 deletions

diff --git a/sql/item_func.cc b/sql/item_func.cc
index cf01934806c..9f934a95f4d 100644
--- a/sql/item_func.cc
+++ b/sql/item_func.cc
@@ -2218,9 +2218,13 @@ bool Item_func_int_val::fix_length_and_dec()
 {
   DBUG_ENTER("Item_func_int_val::fix_length_and_dec");
   DBUG_PRINT("info", ("name %s", func_name()));
-  if (args[0]->type_handler()->Item_func_int_val_fix_length_and_dec(this))
+  /*
+    We don't want to translate ENUM/SET to CHAR here.
+    So let's call real_type_handler(), not type_handler().
+  */
+  if (args[0]->real_type_handler()->Item_func_int_val_fix_length_and_dec(this))
     DBUG_RETURN(TRUE);
-  DBUG_PRINT("info", ("Type: %s", type_handler()->name().ptr()));
+  DBUG_PRINT("info", ("Type: %s", real_type_handler()->name().ptr()));
   DBUG_RETURN(FALSE);
 }