MDEV-23415 Server crash or Assertion `dec_length <= str_length' failed in Item_func_format::val_str_ascii

Problem:

The crash happened in FORMAT(double, dec>=31, 'de_DE').

The patch for MDEV-23118 (commit 0041dacc1b8e85e1958355d1cfdc36055b05a884)
did not take into account that String::set_real() has a limit of 31
(FLOATING_POINT_DECIMALS) fractional digits. So for the range of 31..38
digits, set_real() switches to use:
- my_fcvt() - decimal point notation, e.g. 1.9999999999
- my_gcvt() - scientific notation,    e.g. 1e22

my_gcvt() returned a shorter string than Item_func_format::val_str_ascii()
expected to get after the my_fcvt() call, so it crashed on assert.

Solution:

We cannot extend set_real() to use the my_fcvt() mode for the range of
31..38 fractional digits, because set_real() is used in a lot of places
and such a change will break everything.

Introducing String::set_fcvt() which always prints using my_fcvt()
for the whole range of decimals 0..38, supported by the FORMAT() function.

1 files changed, 11 insertions, 0 deletions

diff --git a/sql/sql_string.cc b/sql/sql_string.cc
index 483eb4fcbec..4a94087a6ba 100644
--- a/sql/sql_string.cc
+++ b/sql/sql_string.cc
@@ -183,6 +183,17 @@ bool Binary_string::set_hex(const char *str, uint32 len)
 }
 
 
+bool Binary_string::set_fcvt(double num, uint decimals)
+{
+  // Assert that `decimals` is small enough to fit into FLOATING_POINT_BUFFER
+  DBUG_ASSERT(decimals < DECIMAL_NOT_SPECIFIED);
+  if (alloc(FLOATING_POINT_BUFFER))
+    return true;
+  length(my_fcvt(num, decimals, Ptr, NULL));
+  return false;
+}
+
+
 bool String::set_real(double num,uint decimals, CHARSET_INFO *cs)
 {
   char buff[FLOATING_POINT_BUFFER];