Bug#30889: filesort and order by with float/numeric crashes server

There are two problems with ROUND(X, D) on an exact numeric 
(DECIMAL, NUMERIC type) field of a table:
1) The implementation of the ROUND function would change the number of decimal
places regardless of the value decided upon in fix_length_and_dec. When the
number of decimal places is not constant, this would cause an inconsistent
state where the number of digits was less than the number of decimal places,
which crashes filesort.

Fixed by not allowing the ROUND operation to add any more decimal places than
was decided in fix_length_and_dec.

2) fix_length_and_dec would allow the number of decimals to be greater than
the maximium configured value for constant values of D. This led to the same 
crash as in (1).

Fixed by not allowing the above in fix_length_and_dec.


mysql-test/r/type_decimal.result:
  Bug#30889: Test result
mysql-test/t/type_decimal.test:
  Bug#30889: Test case
sql/item_func.cc:
  Bug#30889: 
  - Avoid setting number of digits after decimal point (scale) higher than its 
    maximum value.
  - Avoid increasing the number of decimal places in ::decimal_op
sql/item_func.h:
  Bug#30889: Added comments to the declarations of Item_func_numhybrid::<type>_op
  family of methods.

1 files changed, 31 insertions, 0 deletions

diff --git a/sql/item_func.h b/sql/item_func.h
index 43221a18a5b..a31294c0395 100644
--- a/sql/item_func.h
+++ b/sql/item_func.h
@@ -236,9 +236,40 @@ public:
   my_decimal *val_decimal(my_decimal *);
   String *val_str(String*str);
 
+  /**
+     @brief Performs the operation that this functions implements when the
+     result type is INT.
+
+     @return The result of the operation.
+  */
   virtual longlong int_op()= 0;
+
+  /**
+     @brief Performs the operation that this functions implements when the
+     result type is REAL.
+
+     @return The result of the operation.
+  */
   virtual double real_op()= 0;
+
+  /**
+     @brief Performs the operation that this functions implements when the
+     result type is DECIMAL.
+
+     @param A pointer where the DECIMAL value will be allocated.
+     @return 
+       - 0 If the result is NULL
+       - The same pointer it was given, with the area initialized to the
+         result of the operation.
+  */
   virtual my_decimal *decimal_op(my_decimal *)= 0;
+
+  /**
+     @brief Performs the operation that this functions implements when the
+     result type is a string type.
+
+     @return The result of the operation.
+  */
   virtual String *str_op(String *)= 0;
   bool is_null() { update_null_value(); return null_value; }
 };