Bug#13580775 ASSERTION FAILED: RECORD_LENGTH == M_RECORD_LENGTH

Bug#13011410 CRASH IN FILESORT CODE WITH GROUP BY/ROLLUP

The assert in 13580775 is visible in 5.6 only, 
but shows that all versions are vulnerable.
13011410 crashes in all versions.

filesort tries to re-use the sort buffer between invocations in order to save
malloc/free overhead.
The fix for Bug 11748783 - 37359: FILESORT CAN BE MORE EFFICIENT.
added an assert that buffer properties (num_records, record_length) are
consistent between invocations. Indeed, they are not necessarily consistent.
  
Fix: re-allocate the sort buffer if properties change.


mysql-test/r/partition.result:
  New tests.
mysql-test/t/partition.test:
  New tests.
sql/filesort.cc:
  If we already have allocated a sort buffer in a previous execution,
  then verify that it is big enough for the current one.
sql/table.h:
  Add sort_keys_size; Number of bytes allocated for the sort_keys buffer.

1 files changed, 40 insertions, 0 deletions

diff --git a/mysql-test/r/partition.result b/mysql-test/r/partition.result
index 379f7499e11..49fe208d9bd 100644
--- a/mysql-test/r/partition.result
+++ b/mysql-test/r/partition.result
@@ -2206,4 +2206,44 @@ TRUNCATE TABLE t1;
 INSERT INTO t1 VALUES(0);
 DROP TABLE t1;
 SET GLOBAL myisam_use_mmap=default;
+#
+# Bug#13580775 ASSERTION FAILED: RECORD_LENGTH == M_RECORD_LENGTH,
+# FILE FILESORT_UTILS.CC
+#
+CREATE TABLE t1 (
+a INT PRIMARY KEY,
+b INT,
+c CHAR(1),
+d INT,
+KEY (c,d)
+) PARTITION BY KEY () PARTITIONS 1;
+INSERT INTO t1 VALUES (1,1,'a',1), (2,2,'a',1);
+SELECT 1 FROM t1 WHERE 1 IN
+(SELECT  group_concat(b)
+FROM t1
+WHERE c > geomfromtext('point(1 1)')
+GROUP BY b
+);
+1
+1
+1
+DROP TABLE t1;
+#
+# Bug#13011410 CRASH IN FILESORT CODE WITH GROUP BY/ROLLUP
+#
+CREATE TABLE t1 (
+a INT,
+b MEDIUMINT,
+c VARCHAR(300) CHARACTER SET hp8 COLLATE hp8_bin,
+PRIMARY KEY (a,c(299))) 
+ENGINE=myisam
+PARTITION BY LINEAR KEY () PARTITIONS 2;
+INSERT INTO t1 VALUES (1,2,'test'), (2,3,'hi'), (4,5,'bye');
+SELECT 1 FROM t1 WHERE b < SOME
+( SELECT 1 FROM t1 WHERE a >= 1
+GROUP BY b WITH ROLLUP
+HAVING b > geomfromtext("")
+);
+1
+DROP TABLE t1;
 End of 5.1 tests