Bug#14521864: MYSQL 5.1 TO 5.5 BUGS PARTITIONING

Due to an internal change in the server code in between 5.1 and 5.5
(wl#2649) the hash function used in KEY partitioning changed
for numeric and date/time columns (from binary hash calculation
to character based hash calculation).

Also enum/set changed from latin1 ci based hash calculation to
binary hash between 5.1 and 5.5. (bug#11759782).

These changes makes KEY [sub]partitioned tables on any of
the affected column types incompatible with 5.5 and above,
since the calculation of partition id differs.

Also since InnoDB asserts that a deleted row was previously
read (positioned), the server asserts on delete of a row that
is in the wrong partition.

The solution for this situation is:

1) The partitioning engine will check that delete/update will go to the
partition the row was read from and give an error otherwise, consisting
of the rows partitioning fields. This will avoid asserts in InnoDB and
also alert the user that there is a misplaced row. A detailed error
message will be given, including an entry to the error log consisting
of both table name, partition and row content (PK if exists, otherwise
all partitioning columns).


2) A new optional syntax for KEY () partitioning in 5.5 is allowed:
[SUB]PARTITION BY KEY [ALGORITHM = N] (list_of_cols)
Where N = 1 uses the same hashing as 5.1 (Numeric/date/time fields uses
binary hashing, ENUM/SET uses charset hashing) N = 2 uses the same
hashing as 5.5 (Numeric/date/time fields uses charset hashing,
ENUM/SET uses binary hashing). If not set on CREATE/ALTER it will
default to 2.

This new syntax should probably be ignored by NDB.


3) Since there is a demand for avoiding scanning through the full
table, during upgrade the ALTER TABLE t PARTITION BY ... command is
considered a no-op (only .frm change) if everything except ALGORITHM
is the same and ALGORITHM was not set before, which allows manually
upgrading such table by something like:
ALTER TABLE t PARTITION BY KEY ALGORITHM = 1 () or
ALTER TABLE t PARTITION BY KEY ALGORITHM = 2 ()


4) Enhanced partitioning with CHECK/REPAIR to also check for/repair
misplaced rows. (Also works for ALTER TABLE t CHECK/REPAIR PARTITION)

CHECK FOR UPGRADE:
If the .frm version is < 5.5.3
and uses KEY [sub]partitioning
and an affected column type
then it will fail with an message:
KEY () partitioning changed, please run:
ALTER TABLE `test`.`t1`  PARTITION BY KEY ALGORITHM = 1 (a)
PARTITIONS 12
(i.e. current partitioning clause, with the addition of
ALGORITHM = 1)

CHECK without FOR UPGRADE:
if MEDIUM (default) or EXTENDED options are given:
Scan all rows and verify that it is in the correct partition.
Fail for the first misplaced row.

REPAIR:
if default or EXTENDED (i.e. not QUICK/USE_FRM):
Scan all rows and every misplaced row is moved into its correct
partitions.


5) Updated mysqlcheck (called by mysql_upgrade) to handle the
new output from CHECK FOR UPGRADE, to run the ALTER statement
instead of running REPAIR.

This will allow mysql_upgrade (or CHECK TABLE t FOR UPGRADE) to upgrade
a KEY [sub]partitioned table that has any affected field type
and a .frm version < 5.5.3 to ALGORITHM = 1 without rebuild.


Also notice that if the .frm has a version of >= 5.5.3 and ALGORITHM
is not set, it is not possible to know if it consists of rows from
5.1 or 5.5! In these cases I suggest that the user does:
(optional)
LOCK TABLE t WRITE;
SHOW CREATE TABLE t;
(verify that it has no ALGORITHM = N, and to be safe, I would suggest
backing up the .frm file, to be used if one need to change to another
ALGORITHM = N, without needing to rebuild/repair)
ALTER TABLE t <old partitioning clause, but with ALGORITHM = N>;
which should set the ALGORITHM to N (if the table has rows from
5.1 I would suggest N = 1, otherwise N = 2)
CHECK TABLE t;
(here one could use the backed up .frm instead and change to a new N
and run CHECK again and see if it passes)
and if there are misplaced rows:
REPAIR TABLE t;
(optional)
UNLOCK TABLES;

1 files changed, 278 insertions, 2 deletions

diff --git a/sql/partition_info.cc b/sql/partition_info.cc
index cbf22d18c87..5a7d0bf0c43 100644
--- a/sql/partition_info.cc
+++ b/sql/partition_info.cc
@@ -1,4 +1,4 @@
-/* Copyright (c) 2006, 2010, Oracle and/or its affiliates. All rights reserved.
+/* Copyright (c) 2006, 2013, Oracle and/or its affiliates. All rights reserved.
 
    This program is free software; you can redistribute it and/or modify
    it under the terms of the GNU General Public License as published by
@@ -2180,9 +2180,36 @@ int partition_info::fix_parser_data(THD *thd)
   if (!(part_type == RANGE_PARTITION ||
         part_type == LIST_PARTITION))
   {
-    /* Nothing to do for HASH/KEY partitioning */
+    if (part_type == HASH_PARTITION && list_of_part_fields)
+    {
+      /* KEY partitioning, check ALGORITHM = N. Should not pass the parser! */
+      if (key_algorithm > KEY_ALGORITHM_55)
+      {
+        my_error(ER_PARTITION_FUNCTION_IS_NOT_ALLOWED, MYF(0));
+        DBUG_RETURN(true);
+      }
+      /* If not set, use DEFAULT = 2 for CREATE and ALTER! */
+      if ((thd_sql_command(thd) == SQLCOM_CREATE_TABLE ||
+           thd_sql_command(thd) == SQLCOM_ALTER_TABLE) &&
+          key_algorithm == KEY_ALGORITHM_NONE)
+        key_algorithm= KEY_ALGORITHM_55;
+    }
     DBUG_RETURN(FALSE);
   }
+  if (is_sub_partitioned() && list_of_subpart_fields)
+  {
+    /* KEY subpartitioning, check ALGORITHM = N. Should not pass the parser! */
+    if (key_algorithm > KEY_ALGORITHM_55)
+    {
+      my_error(ER_PARTITION_FUNCTION_IS_NOT_ALLOWED, MYF(0));
+      DBUG_RETURN(true);
+    }
+    /* If not set, use DEFAULT = 2 for CREATE and ALTER! */
+    if ((thd_sql_command(thd) == SQLCOM_CREATE_TABLE ||
+         thd_sql_command(thd) == SQLCOM_ALTER_TABLE) &&
+        key_algorithm == KEY_ALGORITHM_NONE)
+      key_algorithm= KEY_ALGORITHM_55;
+  }
   do
   {
     part_elem= it++;
@@ -2231,6 +2258,255 @@ int partition_info::fix_parser_data(THD *thd)
   DBUG_RETURN(FALSE);
 }
 
+
+/**
+  helper function to compare strings that can also be
+  a NULL pointer.
+
+  @param a  char pointer (can be NULL).
+  @param b  char pointer (can be NULL).
+
+  @return false if equal
+    @retval true  strings differs
+    @retval false strings is equal
+*/
+
+static bool strcmp_null(const char *a, const char *b)
+{
+  if (!a && !b)
+    return false;
+  if (a && b && !strcmp(a, b))
+    return false;
+  return true;
+}
+
+
+/**
+  Check if the new part_info has the same partitioning.
+
+  @param new_part_info  New partition definition to compare with.
+
+  @return True if not considered to have changed the partitioning.
+    @retval true  Allowed change (only .frm change, compatible distribution).
+    @retval false Different partitioning, will need redistribution of rows.
+
+  @note Currently only used to allow changing from non-set key_algorithm
+  to a specified key_algorithm, to avoid rebuild when upgrading from 5.1 of
+  such partitioned tables using numeric colums in the partitioning expression.
+  For more info see bug#14521864.
+  Does not check if columns etc has changed, i.e. only for
+  alter_info->flags == ALTER_PARTITION.
+*/
+
+bool partition_info::has_same_partitioning(partition_info *new_part_info)
+{
+  DBUG_ENTER("partition_info::has_same_partitioning");
+
+  if (!new_part_info ||
+      part_type != new_part_info->part_type ||
+      num_parts != new_part_info->num_parts ||
+      use_default_partitions != new_part_info->use_default_partitions ||
+      new_part_info->is_sub_partitioned() != is_sub_partitioned())
+    DBUG_RETURN(false);
+
+  if (part_type != HASH_PARTITION)
+  {
+    /*
+      RANGE or LIST partitioning, check if KEY subpartitioned.
+      Also COLUMNS partitioning was added in 5.5, so treat that as different.
+    */
+    if (!is_sub_partitioned() ||
+        !new_part_info->is_sub_partitioned() ||
+        column_list ||
+        new_part_info->column_list ||
+        !list_of_subpart_fields ||
+        !new_part_info->list_of_subpart_fields ||
+        new_part_info->num_subparts != num_subparts ||
+        new_part_info->subpart_field_list.elements !=
+          subpart_field_list.elements ||
+        new_part_info->use_default_subpartitions !=
+          use_default_subpartitions)
+      DBUG_RETURN(false);
+  }
+  else
+  {
+    /* Check if KEY partitioned. */
+    if (!new_part_info->list_of_part_fields ||
+        !list_of_part_fields ||
+        new_part_info->part_field_list.elements != part_field_list.elements)
+      DBUG_RETURN(false);
+  }
+
+  /* Check that it will use the same fields in KEY (fields) list. */
+  List_iterator<char> old_field_name_it(part_field_list);
+  List_iterator<char> new_field_name_it(new_part_info->part_field_list);
+  char *old_name, *new_name;
+  while ((old_name= old_field_name_it++))
+  {
+    new_name= new_field_name_it++;
+    if (!new_name || my_strcasecmp(system_charset_info,
+                                   new_name,
+                                   old_name))
+      DBUG_RETURN(false);
+  }
+
+  if (is_sub_partitioned())
+  {
+    /* Check that it will use the same fields in KEY subpart fields list. */
+    List_iterator<char> old_field_name_it(subpart_field_list);
+    List_iterator<char> new_field_name_it(new_part_info->subpart_field_list);
+    char *old_name, *new_name;
+    while ((old_name= old_field_name_it++))
+    {
+      new_name= new_field_name_it++;
+      if (!new_name || my_strcasecmp(system_charset_info,
+                                     new_name,
+                                     old_name))
+        DBUG_RETURN(false);
+    }
+  }
+
+  if (!use_default_partitions)
+  {
+    /*
+      Loop over partitions/subpartition to verify that they are
+      the same, including state and name.
+    */
+    List_iterator<partition_element> part_it(partitions);
+    List_iterator<partition_element> new_part_it(new_part_info->partitions);
+    uint i= 0;
+    do
+    {
+      partition_element *part_elem= part_it++;
+      partition_element *new_part_elem= new_part_it++;
+      /*
+        The following must match:
+        partition_name, tablespace_name, data_file_name, index_file_name,
+        engine_type, part_max_rows, part_min_rows, nodegroup_id.
+        (max_value, signed_flag, has_null_value only on partition level,
+        RANGE/LIST)
+        The following can differ:
+          - part_comment
+        part_state must be PART_NORMAL!
+      */
+      if (!part_elem || !new_part_elem ||
+          strcmp(part_elem->partition_name,
+                 new_part_elem->partition_name) ||
+          part_elem->part_state != PART_NORMAL ||
+          new_part_elem->part_state != PART_NORMAL ||
+          part_elem->max_value != new_part_elem->max_value ||
+          part_elem->signed_flag != new_part_elem->signed_flag ||
+          part_elem->has_null_value != new_part_elem->has_null_value)
+        DBUG_RETURN(false);
+
+      /* new_part_elem may not have engine_type set! */
+      if (new_part_elem->engine_type &&
+          part_elem->engine_type != new_part_elem->engine_type)
+        DBUG_RETURN(false);
+
+      if (is_sub_partitioned())
+      {
+        /*
+          Check that both old and new partition has the same definition
+          (VALUES IN/VALUES LESS THAN) (No COLUMNS partitioning, see above)
+        */
+        if (part_type == LIST_PARTITION)
+        {
+          List_iterator<part_elem_value> list_vals(part_elem->list_val_list);
+          List_iterator<part_elem_value>
+            new_list_vals(new_part_elem->list_val_list);
+          part_elem_value *val;
+          part_elem_value *new_val;
+          while ((val= list_vals++))
+          {
+            new_val= new_list_vals++;
+            if (!new_val)
+              DBUG_RETURN(false);
+            if ((!val->null_value && !new_val->null_value) &&
+                val->value != new_val->value)
+              DBUG_RETURN(false);
+          }
+          if (new_list_vals++)
+            DBUG_RETURN(false);
+        }
+        else
+        {
+          DBUG_ASSERT(part_type == RANGE_PARTITION);
+          if (new_part_elem->range_value != part_elem->range_value)
+            DBUG_RETURN(false);
+        }
+
+        if (!use_default_subpartitions)
+        {
+          List_iterator<partition_element>
+            sub_part_it(part_elem->subpartitions);
+          List_iterator<partition_element>
+            new_sub_part_it(new_part_elem->subpartitions);
+          uint j= 0;
+          do
+          {
+            partition_element *sub_part_elem= sub_part_it++;
+            partition_element *new_sub_part_elem= new_sub_part_it++;
+            /* new_part_elem may not have engine_type set! */
+            if (new_sub_part_elem->engine_type &&
+                sub_part_elem->engine_type != new_part_elem->engine_type)
+              DBUG_RETURN(false);
+
+            if (strcmp(sub_part_elem->partition_name,
+                       new_sub_part_elem->partition_name) ||
+                sub_part_elem->part_state != PART_NORMAL ||
+                new_sub_part_elem->part_state != PART_NORMAL ||
+                sub_part_elem->part_min_rows !=
+                  new_sub_part_elem->part_min_rows ||
+                sub_part_elem->part_max_rows !=
+                  new_sub_part_elem->part_max_rows ||
+                sub_part_elem->nodegroup_id !=
+                  new_sub_part_elem->nodegroup_id)
+              DBUG_RETURN(false);
+  
+            if (strcmp_null(sub_part_elem->data_file_name,
+                            new_sub_part_elem->data_file_name) ||
+                strcmp_null(sub_part_elem->index_file_name,
+                            new_sub_part_elem->index_file_name) ||
+                strcmp_null(sub_part_elem->tablespace_name,
+                            new_sub_part_elem->tablespace_name))
+              DBUG_RETURN(false);
+
+          } while (++j < num_subparts);
+        }
+      }
+      else
+      {
+        if (part_elem->part_min_rows != new_part_elem->part_min_rows ||
+            part_elem->part_max_rows != new_part_elem->part_max_rows ||
+            part_elem->nodegroup_id != new_part_elem->nodegroup_id)
+          DBUG_RETURN(false);
+
+        if (strcmp_null(part_elem->data_file_name,
+                        new_part_elem->data_file_name) ||
+            strcmp_null(part_elem->index_file_name,
+                        new_part_elem->index_file_name) ||
+            strcmp_null(part_elem->tablespace_name,
+                        new_part_elem->tablespace_name))
+          DBUG_RETURN(false);
+      }
+    } while (++i < num_parts);
+  }
+
+  /*
+    Only if key_algorithm was not specified before and it is now set,
+    consider this as nothing was changed!
+    But if already set, consider it as a change, and force rebuild!
+  */
+  DBUG_ASSERT(new_part_info->key_algorithm !=
+                partition_info::KEY_ALGORITHM_NONE);
+  if (key_algorithm != partition_info::KEY_ALGORITHM_NONE)
+    DBUG_RETURN(false);
+
+  DBUG_RETURN(true);
+}
+
+
 void partition_info::print_debug(const char *str, uint *value)
 {
   DBUG_ENTER("print_debug");