Fixed bug #35993: memory corruption and crash with multibyte conversion.

Grouping or ordering of long values in not indexed BLOB/TEXT columns
with GBK or BIG5 charsets crashes the server.

MySQL server uses sorting (the filesort procedure) in the temporary
table to evaluate the GROUP BY clause in case of lack of suitable index.
That procedure takes into account only first @max_sort_length bytes
(system variable, usually 1024) of TEXT/BLOB sorting key string.
The my_strnxfrm_gbk and my_strnxfrm_big5 fill temporary keys
with data of whole blob length instead of @max_sort_length bytes
length. That buffer overrun has been fixed.


mysql-test/r/ctype_gbk.result:
  Added test case for bug #35993.
mysql-test/t/ctype_gbk.test:
  Added test case for bug #35993.
strings/ctype-big5.c:
  Fixed bug #35993: memory corruption and crash with multibyte conversion.
  
  Buffer overrun has been fixed in the my_strnxfrm_big5 function.
strings/ctype-gbk.c:
  Fixed bug #35993: memory corruption and crash with multibyte conversion.
  
  Buffer overrun has been fixed in the my_strnxfrm_gbk function.

2 files changed, 8 insertions, 4 deletions

diff --git a/strings/ctype-big5.c b/strings/ctype-big5.c
index 44b9951657d..c73247db404 100644
--- a/strings/ctype-big5.c
+++ b/strings/ctype-big5.c
@@ -307,15 +307,17 @@ static int my_strnxfrm_big5(CHARSET_INFO *cs __attribute__((unused)),
 {
   uint16 e;
   uint dstlen= len;
+  uchar *dest_end= dest + dstlen;
 
   len = srclen;
-  while (len--)
+  while (len-- && dest < dest_end)
   {
     if ((len > 0) && isbig5code(*src, *(src+1)))
     {
       e = big5strokexfrm((uint16) big5code(*src, *(src+1)));
       *dest++ = big5head(e);
-      *dest++ = big5tail(e);
+      if (dest < dest_end)
+        *dest++ = big5tail(e);
       src +=2;
       len--;
     } else
diff --git a/strings/ctype-gbk.c b/strings/ctype-gbk.c
index 8ac7d62c9da..d0ba33aa3cc 100644
--- a/strings/ctype-gbk.c
+++ b/strings/ctype-gbk.c
@@ -2668,15 +2668,17 @@ static int my_strnxfrm_gbk(CHARSET_INFO *cs __attribute__((unused)),
 {
   uint16 e;
   uint dstlen= len;
+  uchar *dest_end= dest + dstlen;
 
   len = srclen;
-  while (len--)
+  while (len-- && dest < dest_end)
   {
     if ((len > 0) && isgbkcode(*src, *(src+1)))
     {
       e = gbksortorder((uint16) gbkcode(*src, *(src+1)));
       *dest++ = gbkhead(e);
-      *dest++ = gbktail(e);
+      if (dest < dest_end)
+        *dest++ = gbktail(e);
       src+=2;
       len--;
     } else