MDEV-23323 Rounding functions return a wrong data type for a BIT, ENUM, SET argument

Implementing dedicated fixing methods:
- Type_handler_bit::Item_func_round_fix_length_and_dec()
- Type_handler_bit::Item_func_int_val_fix_length_and_dec()
- Type_handler_typelib::Item_func_round_fix_length_and_dec()

because the inherited methods did not work well.

Fixing:
- Type_handler_typelib::Item_func_int_val_fix_length_and_dec
  It did not work well, because it used args[0]->max_length to
  calculate the result data type. In case of ENUM and SET it was
  not correct, because in FLOOR() and CEILING() context
  ENUM and SET return not more than 5 digits (65535 is the biggest
  possible value).

Misc:
- Changing the API of
    Type_handler_bit::Bit_decimal_notation_int_digits(const Item *item)
  to a more generic form:
    Type_handler_bit::Bit_decimal_notation_int_digits_by_nbits(uint nbits)

- Fixing Type_handler_bit::Bit_decimal_notation_int_digits_by_nbits() to
  return the exact number of decimal digits for all nbits 1..64.
  The old implementation was approximate.
  This change gives better (more precise) data types.

1 files changed, 13 insertions, 0 deletions

diff --git a/mysql-test/main/type_set.test b/mysql-test/main/type_set.test
index 8a81c406c63..2d0c81340ae 100644
--- a/mysql-test/main/type_set.test
+++ b/mysql-test/main/type_set.test
@@ -248,3 +248,16 @@ DROP TABLE t1;
 SET NAMES utf8;
 --error ER_ILLEGAL_VALUE_FOR_TYPE
 CREATE TABLE t1 (a SET('a,bü'));
+
+--echo #
+--echo # MDEV-23323 Rounding functions return a wrong data type for a BIT, ENUM, SET argument
+--echo #
+
+CREATE TABLE t1 (a SET('999999999999999999999999999999999999999999999999999999999999'));
+INSERT INTO t1 VALUES (1);
+CREATE TABLE t2 AS
+  SELECT a, FLOOR(a), CEILING(a), TRUNCATE(a,0), ROUND(a) FROM t1;
+SHOW CREATE TABLE t2;
+SELECT * FROM t2;
+DROP TABLE t2;
+DROP TABLE t1;