Bug#14521864: MYSQL 5.1 TO 5.5 BUGS PARTITIONING

Due to an internal change in the server code in between 5.1 and 5.5
(wl#2649) the hash function used in KEY partitioning changed
for numeric and date/time columns (from binary hash calculation
to character based hash calculation).

Also enum/set changed from latin1 ci based hash calculation to
binary hash between 5.1 and 5.5. (bug#11759782).

These changes makes KEY [sub]partitioned tables on any of
the affected column types incompatible with 5.5 and above,
since the calculation of partition id differs.

Also since InnoDB asserts that a deleted row was previously
read (positioned), the server asserts on delete of a row that
is in the wrong partition.

The solution for this situation is:

1) The partitioning engine will check that delete/update will go to the
partition the row was read from and give an error otherwise, consisting
of the rows partitioning fields. This will avoid asserts in InnoDB and
also alert the user that there is a misplaced row. A detailed error
message will be given, including an entry to the error log consisting
of both table name, partition and row content (PK if exists, otherwise
all partitioning columns).


2) A new optional syntax for KEY () partitioning in 5.5 is allowed:
[SUB]PARTITION BY KEY [ALGORITHM = N] (list_of_cols)
Where N = 1 uses the same hashing as 5.1 (Numeric/date/time fields uses
binary hashing, ENUM/SET uses charset hashing) N = 2 uses the same
hashing as 5.5 (Numeric/date/time fields uses charset hashing,
ENUM/SET uses binary hashing). If not set on CREATE/ALTER it will
default to 2.

This new syntax should probably be ignored by NDB.


3) Since there is a demand for avoiding scanning through the full
table, during upgrade the ALTER TABLE t PARTITION BY ... command is
considered a no-op (only .frm change) if everything except ALGORITHM
is the same and ALGORITHM was not set before, which allows manually
upgrading such table by something like:
ALTER TABLE t PARTITION BY KEY ALGORITHM = 1 () or
ALTER TABLE t PARTITION BY KEY ALGORITHM = 2 ()


4) Enhanced partitioning with CHECK/REPAIR to also check for/repair
misplaced rows. (Also works for ALTER TABLE t CHECK/REPAIR PARTITION)

CHECK FOR UPGRADE:
If the .frm version is < 5.5.3
and uses KEY [sub]partitioning
and an affected column type
then it will fail with an message:
KEY () partitioning changed, please run:
ALTER TABLE `test`.`t1`  PARTITION BY KEY ALGORITHM = 1 (a)
PARTITIONS 12
(i.e. current partitioning clause, with the addition of
ALGORITHM = 1)

CHECK without FOR UPGRADE:
if MEDIUM (default) or EXTENDED options are given:
Scan all rows and verify that it is in the correct partition.
Fail for the first misplaced row.

REPAIR:
if default or EXTENDED (i.e. not QUICK/USE_FRM):
Scan all rows and every misplaced row is moved into its correct
partitions.


5) Updated mysqlcheck (called by mysql_upgrade) to handle the
new output from CHECK FOR UPGRADE, to run the ALTER statement
instead of running REPAIR.

This will allow mysql_upgrade (or CHECK TABLE t FOR UPGRADE) to upgrade
a KEY [sub]partitioned table that has any affected field type
and a .frm version < 5.5.3 to ALGORITHM = 1 without rebuild.


Also notice that if the .frm has a version of >= 5.5.3 and ALGORITHM
is not set, it is not possible to know if it consists of rows from
5.1 or 5.5! In these cases I suggest that the user does:
(optional)
LOCK TABLE t WRITE;
SHOW CREATE TABLE t;
(verify that it has no ALGORITHM = N, and to be safe, I would suggest
backing up the .frm file, to be used if one need to change to another
ALGORITHM = N, without needing to rebuild/repair)
ALTER TABLE t <old partitioning clause, but with ALGORITHM = N>;
which should set the ALGORITHM to N (if the table has rows from
5.1 I would suggest N = 1, otherwise N = 2)
CHECK TABLE t;
(here one could use the backed up .frm instead and change to a new N
and run CHECK again and see if it passes)
and if there are misplaced rows:
REPAIR TABLE t;
(optional)
UNLOCK TABLES;

6 files changed, 38 insertions, 38 deletions

diff --git a/mysql-test/r/partition.result b/mysql-test/r/partition.result
index b64b0abef0c..dbc212a0222 100644
--- a/mysql-test/r/partition.result
+++ b/mysql-test/r/partition.result
@@ -81,7 +81,7 @@ t1	CREATE TABLE `t1` (
   `a` int(11) NOT NULL,
   `b` int(11) NOT NULL
 ) ENGINE=MyISAM DEFAULT CHARSET=latin1
-/*!50100 PARTITION BY KEY (a)
+/*!50100 PARTITION BY KEY ALGORITHM = 2 (a)
 PARTITIONS 2 */
 SELECT * FROM t1;
 a	b
@@ -175,7 +175,7 @@ t1	CREATE TABLE `t1` (
   PRIMARY KEY (`ID`,`aaaa,aaaaa`,`ddddddddd`)
 ) ENGINE=MyISAM DEFAULT CHARSET=latin1
 /*!50100 PARTITION BY RANGE (ID)
-SUBPARTITION BY LINEAR KEY (ID,`aaaa,aaaaa`)
+SUBPARTITION BY LINEAR KEY ALGORITHM = 2 (ID,`aaaa,aaaaa`)
 SUBPARTITIONS 2
 (PARTITION p01 VALUES LESS THAN (100) ENGINE = MyISAM,
  PARTITION p11 VALUES LESS THAN (200) ENGINE = MyISAM,
@@ -722,7 +722,7 @@ t1	CREATE TABLE `t1` (
   `a` int(11) DEFAULT NULL
 ) ENGINE=MyISAM DEFAULT CHARSET=latin1
 /*!50100 PARTITION BY RANGE (a)
-SUBPARTITION BY KEY (a)
+SUBPARTITION BY KEY ALGORITHM = 2 (a)
 (PARTITION p0 VALUES LESS THAN (1) ENGINE = MyISAM,
  PARTITION p1 VALUES LESS THAN (2) ENGINE = MyISAM) */
 alter table t1 reorganize partition p1 into (partition p1 values less than (3));
@@ -732,7 +732,7 @@ t1	CREATE TABLE `t1` (
   `a` int(11) DEFAULT NULL
 ) ENGINE=MyISAM DEFAULT CHARSET=latin1
 /*!50100 PARTITION BY RANGE (a)
-SUBPARTITION BY KEY (a)
+SUBPARTITION BY KEY ALGORITHM = 2 (a)
 (PARTITION p0 VALUES LESS THAN (1) ENGINE = MyISAM,
  PARTITION p1 VALUES LESS THAN (3) ENGINE = MyISAM) */
 drop table t1;
@@ -753,7 +753,7 @@ t1	CREATE TABLE `t1` (
   `c` int(11) NOT NULL,
   PRIMARY KEY (`a`,`b`)
 ) ENGINE=MyISAM DEFAULT CHARSET=latin1
-/*!50100 PARTITION BY KEY (a) */
+/*!50100 PARTITION BY KEY ALGORITHM = 2 (a) */
 drop table t1;
 CREATE TABLE t1 (
 a int not null,
@@ -1074,7 +1074,7 @@ Table	Create Table
 t1	CREATE TABLE `t1` (
   `a` int(11) DEFAULT NULL
 ) ENGINE=MyISAM DEFAULT CHARSET=latin1
-/*!50100 PARTITION BY KEY (a)
+/*!50100 PARTITION BY KEY ALGORITHM = 2 (a)
 (PARTITION p0 ENGINE = MyISAM,
  PARTITION p1 ENGINE = MyISAM) */
 alter table t1;
@@ -1083,7 +1083,7 @@ Table	Create Table
 t1	CREATE TABLE `t1` (
   `a` int(11) DEFAULT NULL
 ) ENGINE=MyISAM DEFAULT CHARSET=latin1
-/*!50100 PARTITION BY KEY (a)
+/*!50100 PARTITION BY KEY ALGORITHM = 2 (a)
 (PARTITION p0 ENGINE = MyISAM,
  PARTITION p1 ENGINE = MyISAM) */
 alter table t1 engine=myisam;
@@ -1092,7 +1092,7 @@ Table	Create Table
 t1	CREATE TABLE `t1` (
   `a` int(11) DEFAULT NULL
 ) ENGINE=MyISAM DEFAULT CHARSET=latin1
-/*!50100 PARTITION BY KEY (a)
+/*!50100 PARTITION BY KEY ALGORITHM = 2 (a)
 (PARTITION p0 ENGINE = MyISAM,
  PARTITION p1 ENGINE = MyISAM) */
 alter table t1 engine=heap;
@@ -1101,7 +1101,7 @@ Table	Create Table
 t1	CREATE TABLE `t1` (
   `a` int(11) DEFAULT NULL
 ) ENGINE=MEMORY DEFAULT CHARSET=latin1
-/*!50100 PARTITION BY KEY (a)
+/*!50100 PARTITION BY KEY ALGORITHM = 2 (a)
 (PARTITION p0 ENGINE = MEMORY,
  PARTITION p1 ENGINE = MEMORY) */
 alter table t1 remove partitioning;
@@ -1121,7 +1121,7 @@ Table	Create Table
 t1	CREATE TABLE `t1` (
   `a` int(11) DEFAULT NULL
 ) ENGINE=MyISAM DEFAULT CHARSET=latin1
-/*!50100 PARTITION BY KEY (a)
+/*!50100 PARTITION BY KEY ALGORITHM = 2 (a)
 (PARTITION p0 ENGINE = MyISAM,
  PARTITION p1 ENGINE = MyISAM) */
 alter table t1 add column b int remove partitioning;
@@ -1141,7 +1141,7 @@ t1	CREATE TABLE `t1` (
   `a` int(11) DEFAULT NULL,
   `b` int(11) DEFAULT NULL
 ) ENGINE=MyISAM DEFAULT CHARSET=latin1
-/*!50100 PARTITION BY KEY (a)
+/*!50100 PARTITION BY KEY ALGORITHM = 2 (a)
 (PARTITION p0 ENGINE = MyISAM,
  PARTITION p1 ENGINE = MyISAM) */
 alter table t1
@@ -1154,7 +1154,7 @@ t1	CREATE TABLE `t1` (
   `a` int(11) DEFAULT NULL,
   `b` int(11) DEFAULT NULL
 ) ENGINE=MEMORY DEFAULT CHARSET=latin1
-/*!50100 PARTITION BY KEY (a)
+/*!50100 PARTITION BY KEY ALGORITHM = 2 (a)
 (PARTITION p0 ENGINE = MEMORY,
  PARTITION p1 ENGINE = MEMORY) */
 alter table t1 engine=myisam, add column c int remove partitioning;
@@ -1176,7 +1176,7 @@ t1	CREATE TABLE `t1` (
   `b` int(11) DEFAULT NULL,
   `c` int(11) DEFAULT NULL
 ) ENGINE=MEMORY DEFAULT CHARSET=latin1
-/*!50100 PARTITION BY KEY (a)
+/*!50100 PARTITION BY KEY ALGORITHM = 2 (a)
 (PARTITION p0 ENGINE = MEMORY,
  PARTITION p1 ENGINE = MEMORY) */
 alter table t1
@@ -1189,7 +1189,7 @@ t1	CREATE TABLE `t1` (
   `b` int(11) DEFAULT NULL,
   `c` int(11) DEFAULT NULL
 ) ENGINE=MEMORY DEFAULT CHARSET=latin1
-/*!50100 PARTITION BY KEY (a)
+/*!50100 PARTITION BY KEY ALGORITHM = 2 (a)
 (PARTITION p0 ENGINE = MEMORY,
  PARTITION p1 ENGINE = MEMORY) */
 alter table t1
@@ -1203,7 +1203,7 @@ t1	CREATE TABLE `t1` (
   `b` int(11) DEFAULT NULL,
   `c` int(11) DEFAULT NULL
 ) ENGINE=MEMORY DEFAULT CHARSET=latin1
-/*!50100 PARTITION BY KEY (a)
+/*!50100 PARTITION BY KEY ALGORITHM = 2 (a)
 (PARTITION p0 ENGINE = MEMORY,
  PARTITION p1 ENGINE = MEMORY) */
 alter table t1
@@ -1373,14 +1373,14 @@ Table	Create Table
 t1	CREATE TABLE `t1` (
   `a` int(11) DEFAULT NULL
 ) ENGINE=MyISAM DEFAULT CHARSET=latin1
-/*!50100 PARTITION BY KEY (a) */
+/*!50100 PARTITION BY KEY ALGORITHM = 2 (a) */
 alter table t1 add partition (partition p1);
 show create table t1;
 Table	Create Table
 t1	CREATE TABLE `t1` (
   `a` int(11) DEFAULT NULL
 ) ENGINE=MyISAM DEFAULT CHARSET=latin1
-/*!50100 PARTITION BY KEY (a)
+/*!50100 PARTITION BY KEY ALGORITHM = 2 (a)
 (PARTITION p0 ENGINE = MyISAM,
  PARTITION p1 ENGINE = MyISAM) */
 drop table t1;
@@ -1480,7 +1480,7 @@ t2	CREATE TABLE `t2` (
   `c` char(10) DEFAULT NULL,
   PRIMARY KEY (`a`)
 ) ENGINE=MyISAM DEFAULT CHARSET=latin1 COMMENT='no comment'
-/*!50100 PARTITION BY KEY (a) */
+/*!50100 PARTITION BY KEY ALGORITHM = 2 (a) */
 drop table t2;
 create table t1 (f1 int) partition by hash (f1) as select 1;
 drop table t1;
@@ -1715,7 +1715,7 @@ Table	Create Table
 t1	CREATE TABLE `t1` (
   `a` int(11) DEFAULT NULL
 )
-/*!50100 PARTITION BY KEY (a)
+/*!50100 PARTITION BY KEY ALGORITHM = 2 (a)
 (PARTITION p0) */
 set session sql_mode='';
 drop table t1;
@@ -1731,7 +1731,7 @@ Table	Create Table
 t1	CREATE TABLE `t1` (
   `a` varchar(1) DEFAULT NULL
 ) ENGINE=MyISAM DEFAULT CHARSET=latin1
-/*!50100 PARTITION BY KEY (a) */
+/*!50100 PARTITION BY KEY ALGORITHM = 2 (a) */
 drop table t1;
 CREATE TABLE t1 (a int) ENGINE = MYISAM PARTITION BY KEY(a);
 INSERT into t1 values (1), (2);
@@ -1758,13 +1758,13 @@ engine=MEMORY
 partition by key (a);
 REPAIR TABLE t1;
 Table	Op	Msg_type	Msg_text
-test.t1	repair	note	The storage engine for the table doesn't support repair
+test.t1	repair	status	OK
 OPTIMIZE TABLE t1;
 Table	Op	Msg_type	Msg_text
 test.t1	optimize	note	The storage engine for the table doesn't support optimize
 CHECK TABLE t1;
 Table	Op	Msg_type	Msg_text
-test.t1	check	note	The storage engine for the table doesn't support check
+test.t1	check	status	OK
 ANALYZE TABLE t1;
 Table	Op	Msg_type	Msg_text
 test.t1	analyze	note	The storage engine for the table doesn't support analyze
@@ -1804,7 +1804,7 @@ t1	CREATE TABLE `t1` (
   `a` bigint(20) unsigned NOT NULL,
   PRIMARY KEY (`a`)
 ) ENGINE=MyISAM DEFAULT CHARSET=latin1
-/*!50100 PARTITION BY KEY (a)
+/*!50100 PARTITION BY KEY ALGORITHM = 2 (a)
 PARTITIONS 10 */
 insert into t1 values (18446744073709551615), (0xFFFFFFFFFFFFFFFE),
 (18446744073709551613), (18446744073709551612);
diff --git a/mysql-test/r/partition_column.result b/mysql-test/r/partition_column.result
index a494656a6a6..d7cd0beb35f 100644
--- a/mysql-test/r/partition_column.result
+++ b/mysql-test/r/partition_column.result
@@ -454,7 +454,7 @@ t1	CREATE TABLE `t1` (
   `d` int(11) DEFAULT NULL
 ) ENGINE=MyISAM DEFAULT CHARSET=latin1
 /*!50500 PARTITION BY RANGE  COLUMNS(a,b,c)
-SUBPARTITION BY KEY (c,d)
+SUBPARTITION BY KEY ALGORITHM = 2 (c,d)
 SUBPARTITIONS 3
 (PARTITION p0 VALUES LESS THAN (1,'abc','abc') ENGINE = MyISAM,
  PARTITION p1 VALUES LESS THAN (2,'abc','abc') ENGINE = MyISAM,
diff --git a/mysql-test/r/partition_innodb.result b/mysql-test/r/partition_innodb.result
index 516d824b343..aca7060bba4 100644
--- a/mysql-test/r/partition_innodb.result
+++ b/mysql-test/r/partition_innodb.result
@@ -425,7 +425,7 @@ Table	Create Table
 t1	CREATE TABLE `t1` (
   `a` int(11) DEFAULT NULL
 ) ENGINE=InnoDB DEFAULT CHARSET=latin1
-/*!50100 PARTITION BY KEY (a) */
+/*!50100 PARTITION BY KEY ALGORITHM = 2 (a) */
 drop table t1;
 create table t1 (a int)
 engine = innodb
@@ -518,7 +518,7 @@ t1	CREATE TABLE `t1` (
   `char_column` char(5) DEFAULT NULL
 ) ENGINE=MyISAM DEFAULT CHARSET=latin1
 /*!50100 PARTITION BY RANGE (int_column)
-SUBPARTITION BY KEY (char_column)
+SUBPARTITION BY KEY ALGORITHM = 2 (char_column)
 SUBPARTITIONS 2
 (PARTITION p1 VALUES LESS THAN (5) ENGINE = MyISAM) */
 drop table t1;
@@ -565,7 +565,7 @@ t1	CREATE TABLE `t1` (
   `b` int(11) NOT NULL,
   PRIMARY KEY (`a`)
 ) ENGINE=InnoDB DEFAULT CHARSET=latin1
-/*!50100 PARTITION BY KEY (a)
+/*!50100 PARTITION BY KEY ALGORITHM = 2 (a)
 PARTITIONS 2 */
 SELECT * FROM t1;
 a	b
diff --git a/mysql-test/r/partition_mgm_err.result b/mysql-test/r/partition_mgm_err.result
index a13278d724e..50f8f39cc71 100644
--- a/mysql-test/r/partition_mgm_err.result
+++ b/mysql-test/r/partition_mgm_err.result
@@ -145,7 +145,7 @@ Table	Create Table
 t1	CREATE TABLE `t1` (
   `a` int(11) DEFAULT NULL
 ) ENGINE=MyISAM DEFAULT CHARSET=latin1
-/*!50100 PARTITION BY KEY (a)
+/*!50100 PARTITION BY KEY ALGORITHM = 2 (a)
 PARTITIONS 2 */
 DROP TABLE t1;
 CREATE TABLE t1 (a INT) PARTITION BY HASH(a);
diff --git a/mysql-test/r/partition_myisam.result b/mysql-test/r/partition_myisam.result
index 97bcc11495c..6b85bac3908 100644
--- a/mysql-test/r/partition_myisam.result
+++ b/mysql-test/r/partition_myisam.result
@@ -150,7 +150,7 @@ t1	CREATE TABLE `t1` (
   `a` int(11) NOT NULL AUTO_INCREMENT,
   PRIMARY KEY (`a`)
 ) ENGINE=MyISAM AUTO_INCREMENT=14 DEFAULT CHARSET=latin1
-/*!50100 PARTITION BY KEY (a) */
+/*!50100 PARTITION BY KEY ALGORITHM = 2 (a) */
 INSERT INTO t1 VALUES (NULL);
 SELECT * FROM t1;
 a
diff --git a/mysql-test/r/show_check.result b/mysql-test/r/show_check.result
index 1d4202efc18..6d46f0560ce 100644
--- a/mysql-test/r/show_check.result
+++ b/mysql-test/r/show_check.result
@@ -15,7 +15,7 @@ Catalog	Database	Table	Table_alias	Column	Column_alias	Type	Length	Max length	Is
 def					Table	253	128	7	Y	0	31	8
 def					Op	253	10	5	Y	0	31	8
 def					Msg_type	253	10	6	Y	0	31	8
-def					Msg_text	253	255	27	Y	0	31	8
+def					Msg_text	250	393216	27	Y	0	31	8
 Table	Op	Msg_type	Msg_text
 test.t1	check	status	Table is already up to date
 check table t1 fast;
@@ -23,7 +23,7 @@ Catalog	Database	Table	Table_alias	Column	Column_alias	Type	Length	Max length	Is
 def					Table	253	128	7	Y	0	31	8
 def					Op	253	10	5	Y	0	31	8
 def					Msg_type	253	10	6	Y	0	31	8
-def					Msg_text	253	255	27	Y	0	31	8
+def					Msg_text	250	393216	27	Y	0	31	8
 Table	Op	Msg_type	Msg_text
 test.t1	check	status	Table is already up to date
 check table t1 changed;
@@ -31,7 +31,7 @@ Catalog	Database	Table	Table_alias	Column	Column_alias	Type	Length	Max length	Is
 def					Table	253	128	7	Y	0	31	8
 def					Op	253	10	5	Y	0	31	8
 def					Msg_type	253	10	6	Y	0	31	8
-def					Msg_text	253	255	2	Y	0	31	8
+def					Msg_text	250	393216	2	Y	0	31	8
 Table	Op	Msg_type	Msg_text
 test.t1	check	status	OK
 insert into t1 values (5,5,5);
@@ -40,7 +40,7 @@ Catalog	Database	Table	Table_alias	Column	Column_alias	Type	Length	Max length	Is
 def					Table	253	128	7	Y	0	31	8
 def					Op	253	10	5	Y	0	31	8
 def					Msg_type	253	10	6	Y	0	31	8
-def					Msg_text	253	255	2	Y	0	31	8
+def					Msg_text	250	393216	2	Y	0	31	8
 Table	Op	Msg_type	Msg_text
 test.t1	check	status	OK
 check table t1 medium;
@@ -48,7 +48,7 @@ Catalog	Database	Table	Table_alias	Column	Column_alias	Type	Length	Max length	Is
 def					Table	253	128	7	Y	0	31	8
 def					Op	253	10	5	Y	0	31	8
 def					Msg_type	253	10	6	Y	0	31	8
-def					Msg_text	253	255	2	Y	0	31	8
+def					Msg_text	250	393216	2	Y	0	31	8
 Table	Op	Msg_type	Msg_text
 test.t1	check	status	OK
 check table t1 extended;
@@ -56,7 +56,7 @@ Catalog	Database	Table	Table_alias	Column	Column_alias	Type	Length	Max length	Is
 def					Table	253	128	7	Y	0	31	8
 def					Op	253	10	5	Y	0	31	8
 def					Msg_type	253	10	6	Y	0	31	8
-def					Msg_text	253	255	2	Y	0	31	8
+def					Msg_text	250	393216	2	Y	0	31	8
 Table	Op	Msg_type	Msg_text
 test.t1	check	status	OK
 show index from t1;
@@ -88,7 +88,7 @@ Catalog	Database	Table	Table_alias	Column	Column_alias	Type	Length	Max length	Is
 def					Table	253	128	7	Y	0	31	8
 def					Op	253	10	8	Y	0	31	8
 def					Msg_type	253	10	6	Y	0	31	8
-def					Msg_text	253	255	2	Y	0	31	8
+def					Msg_text	250	393216	2	Y	0	31	8
 Table	Op	Msg_type	Msg_text
 test.t1	optimize	status	OK
 optimize table t1;
@@ -160,7 +160,7 @@ Catalog	Database	Table	Table_alias	Column	Column_alias	Type	Length	Max length	Is
 def					Table	253	128	7	Y	0	31	8
 def					Op	253	10	7	Y	0	31	8
 def					Msg_type	253	10	6	Y	0	31	8
-def					Msg_text	253	255	2	Y	0	31	8
+def					Msg_text	250	393216	2	Y	0	31	8
 Table	Op	Msg_type	Msg_text
 test.t1	analyze	status	OK
 show index from t1;
@@ -177,7 +177,7 @@ Catalog	Database	Table	Table_alias	Column	Column_alias	Type	Length	Max length	Is
 def					Table	253	128	7	Y	0	31	8
 def					Op	253	10	6	Y	0	31	8
 def					Msg_type	253	10	6	Y	0	31	8
-def					Msg_text	253	255	2	Y	0	31	8
+def					Msg_text	250	393216	2	Y	0	31	8
 Table	Op	Msg_type	Msg_text
 test.t1	repair	status	OK
 show index from t1;