Bug#33143: Incorrect ORDER BY for ROUND()/TRUNCATE() result

The ROUND(X, D) function would change the Item::decimals field during
execution to achieve the effect of a dynamic number of decimal digits.
This caused a series of bugs:
Bug #30617:Round() function not working under some circumstances in InnoDB
Bug #33402:ROUND with decimal and non-constant cannot round to 0 decimal places
Bug #30889:filesort and order by with float/numeric crashes server
Fixed by never changing the number of shown digits for DECIMAL when
used with a nonconstant number of decimal digits.

1 files changed, 14 insertions, 2 deletions

diff --git a/strings/decimal.c b/strings/decimal.c
index cbea0e340c6..3176cf6afa7 100644
--- a/strings/decimal.c
+++ b/strings/decimal.c
@@ -1595,9 +1595,21 @@ decimal_round(decimal_t *from, decimal_t *to, int scale,
       x+=10;
     *buf1=powers10[pos]*(x-y);
   }
-  if (frac0 < 0)
+  /*
+    In case we're rounding e.g. 1.5e9 to 2.0e9, the decimal_digit_t's inside
+    the buffer are as follows.
+
+    Before <1, 5e8>
+    After  <2, 5e8>
+
+    Hence we need to set the 2nd field to 0.
+    The same holds if we round 1.5e-9 to 2e-9.
+   */
+  if (frac0 < frac1)
   {
-    dec1 *end=to->buf+intg0, *buf=buf1+1;
+    dec1 *buf= to->buf + ((scale == 0 && intg0 == 0) ? 1 : intg0 + frac0);
+    dec1 *end= to->buf + len;
+
     while (buf < end)
       *buf++=0;
   }