Fix for bug#57272: crash in rpad() when using utf8

Problem: if multibyte and binary string arguments passed to 
RPAD(), LPAD() or INSERT() functions, they might return 
wrong results or even lead to a server crash due to missed
character set convertion.

Fix: perform the convertion if necessary.


mysql-test/r/ctype_utf8.result:
  Fix for bug#57272: crash in rpad() when using utf8
    - test result.
mysql-test/t/ctype_utf8.test:
  Fix for bug#57272: crash in rpad() when using utf8
    - test case.
sql/item_strfunc.cc:
  Fix for bug#57272: crash in rpad() when using utf8
    - convert multibyte argument's character set to binary in case of
  FUNCTION(MULTIBYTE_ARG, .., BINARY_ARG,..) for RPAD(), LPAD() and 
  INSERT() functions.

1 files changed, 42 insertions, 0 deletions

diff --git a/sql/item_strfunc.cc b/sql/item_strfunc.cc
index 5d56b0a621a..9f06a4b5c9f 100644
--- a/sql/item_strfunc.cc
+++ b/sql/item_strfunc.cc
@@ -1013,6 +1013,20 @@ String *Item_func_insert::val_str(String *str)
   if ((length < 0) || (length > res->length()))
     length= res->length();
 
+  /*
+    There is one exception not handled (intentionaly) by the character set
+    aggregation code. If one string is strong side and is binary, and
+    another one is weak side and is a multi-byte character string,
+    then we need to operate on the second string in terms on bytes when
+    calling ::numchars() and ::charpos(), rather than in terms of characters.
+    Lets substitute its character set to binary.
+  */
+  if (collation.collation == &my_charset_bin)
+  {
+    res->set_charset(&my_charset_bin);
+    res2->set_charset(&my_charset_bin);
+  }
+
   /* start and length are now sufficiently valid to pass to charpos function */
    start= res->charpos((int) start);
    length= res->charpos((int) length, (uint32) start);
@@ -2514,6 +2528,20 @@ String *Item_func_rpad::val_str(String *str)
   /* Set here so that rest of code sees out-of-bound value as such. */
   if ((ulonglong) count > INT_MAX32)
     count= INT_MAX32;
+  /*
+    There is one exception not handled (intentionaly) by the character set
+    aggregation code. If one string is strong side and is binary, and
+    another one is weak side and is a multi-byte character string,
+    then we need to operate on the second string in terms on bytes when
+    calling ::numchars() and ::charpos(), rather than in terms of characters.
+    Lets substitute its character set to binary.
+  */
+  if (collation.collation == &my_charset_bin)
+  {
+    res->set_charset(&my_charset_bin);
+    rpad->set_charset(&my_charset_bin);
+  }
+
   if (count <= (res_char_length= res->numchars()))
   {						// String to pad is big enough
     res->length(res->charpos((int) count));	// Shorten result if longer
@@ -2616,6 +2644,20 @@ String *Item_func_lpad::val_str(String *str)
   if ((ulonglong) count > INT_MAX32)
     count= INT_MAX32;
 
+  /*
+    There is one exception not handled (intentionaly) by the character set
+    aggregation code. If one string is strong side and is binary, and
+    another one is weak side and is a multi-byte character string,
+    then we need to operate on the second string in terms on bytes when
+    calling ::numchars() and ::charpos(), rather than in terms of characters.
+    Lets substitute its character set to binary.
+  */
+  if (collation.collation == &my_charset_bin)
+  {
+    res->set_charset(&my_charset_bin);
+    pad->set_charset(&my_charset_bin);
+  }
+
   res_char_length= res->numchars();
 
   if (count <= res_char_length)