WL #2604: Partition Management

Optimised version of ADD/DROP/REORGANIZE partitions for
non-NDB storage engines.
New syntax to handle REBUILD/OPTIMIZE/ANALYZE/CHECK/REPAIR partitions
Quite a few bug fixes


include/thr_lock.h:
  New method to downgrade locks from TL_WRITE_ONLY
  Possibility to upgrade lock while aborting locks
mysql-test/r/ndb_autodiscover.result:
  Fix for lowercase and that all NDB tables are now partitioned
mysql-test/r/ndb_bitfield.result:
  Fix for lowercase and that all NDB tables are now partitioned
mysql-test/r/ndb_gis.result:
  Fix for lowercase and that all NDB tables are now partitioned
mysql-test/r/ndb_partition_key.result:
  New test case
mysql-test/r/partition.result:
  New test case
mysql-test/r/partition_error.result:
  New test case
mysql-test/r/partition_mgm_err.result:
  Fix of test case results
mysql-test/t/disabled.def:
  partition_03ndb still has bug
mysql-test/t/ndb_partition_key.test:
  New test cases for new functionality and bugs
mysql-test/t/partition.test:
  New test cases for new functionality and bugs
mysql-test/t/partition_error.test:
  New test cases for new functionality and bugs
mysql-test/t/partition_mgm_err.test:
  New test cases for new functionality and bugs
mysys/thr_lock.c:
  New method to downgrade TL_WRITE_ONLY locks
  Possibility to specify if locks are to be upgraded at abort locks
sql/ha_archive.cc:
  New handlerton methods
sql/ha_berkeley.cc:
  New handlerton methods
sql/ha_blackhole.cc:
  New handlerton methods
sql/ha_federated.cc:
  New handlerton methods
sql/ha_heap.cc:
  New handlerton methods
sql/ha_innodb.cc:
  New handlerton methods
sql/ha_myisam.cc:
  New handlerton methods
sql/ha_myisammrg.cc:
  New handlerton methods
sql/ha_ndbcluster.cc:
  New handlerton methods
  Moved out packfrm and unpackfrm methods
  Adapted many parts to use table_share instead of table->s
  Ensured that .ndb file uses filename and not tablename
  according to new encoding of names (WL 1324)
  All NDB tables are partitioned and set up partition info
  Fixed such that tablenames use tablenames and not filenames in NDB
  NDB uses auto partitioning for ENGINE=NDB tables
  Warning for very large tables
  Set RANGE data
  Set LIST data
  New method to set-up partition info
  Set Default number of partitions flag
  Set linear hash flag
  Set node group array
  Set number of fragments
  Set max rows
  Set tablespace names
  New method to get number of partitions of table to use at open table
sql/ha_ndbcluster.h:
  Removed partition_flags and alter_table_flags from handler class
  A couple of new and changed method headers
sql/ha_ndbcluster_binlog.cc:
  Use new method headers
sql/ha_partition.cc:
  New handlerton methods
  Lots of new function headers
  Use #P# as separator between table name and partition name and
  #SP# as separator between partition name and subpartition name
  Use filename encoding for files both of table name part and of
  partition name parts
  New method to drop partitions based on partition state
  New method to rename partitions based on partition state
  New methods to optimize, analyze, check and repair partitions
  New methods to optimize, analyze, check and repair table
  Helper method to create new partition, open it and external lock
  it, not needed to lock it internally since no one else knows about
  it yet.
  Cleanup method at error for new partitions
  New methods to perform bulk of work at ADD/REORGANIZE partitions
  (change_partitions, copy_partitions)
sql/ha_partition.h:
  New methods and variables
  A few dropped ones and a few changed ones
sql/handler.cc:
  Handlerton interface changes
  New flag to open_table_from_share
sql/handler.h:
  New alter_table_flags
  New partition flags
  New partition states
  More states for default handling
  Lots of new, dropped and changed interfaces
sql/lex.h:
  Added REBUILD and changed name of REORGANISE to REORGANIZE
sql/lock.cc:
  Method to downgrade locks
  Able to specify if locks upgraded on abort locks
sql/log.cc:
  New handlerton methods
sql/mysql_priv.h:
  Lots of new interfaces
sql/share/errmsg.txt:
  Lots of new, dropped and changed error messages
sql/sql_base.cc:
  Adapted to new method headers
  New method to abort and upgrade lock
  New method to close open tables and downgrade lock
  New method to wait for completed table
sql/sql_lex.h:
  New flags
sql/sql_partition.cc:
  Return int instead of bool in get_partition_id
  More defaults handling
  Make use of new mem_alloc_error method
  More work on function headers
  Changes to generate partition syntax to cater for intermediate
  partition states
  Lots of new code with large comments describing new features for
  Partition Management:
  ADD/DROP/REORGANIZE/OPTIMIZE/ANALYZE/CHECK/REPAIR partitions
sql/sql_show.cc:
  Minors
sql/sql_table.cc:
  Moved a couple of methods
  New methods to copy create lists and key lists
  for use with mysql_prepare_table
  New method to write frm file
  New handling of handlers with auto partitioning
  Fix CREATE TABLE LIKE
  Moved code for ADD/DROP/REORGANIZE partitions
  Use handlerton method for alter_table_flags
sql/sql_yacc.yy:
  More memory alloc error checks
  New syntax for REBUILD, ANALYZE, CHECK, OPTIMIZE, REPAIR partitions
sql/table.cc:
  Fix length of extra part to be 4 bytes
  Partition state introduced in frm file
sql/table.h:
  Partition state introduced
sql/unireg.cc:
  Partition state introduced
  Default partition
storage/csv/ha_tina.cc:
  New handlerton methods
storage/example/ha_example.cc:
  New handlerton methods
storage/ndb/include/kernel/ndb_limits.h:
  RANGE DATA
storage/ndb/include/kernel/signaldata/AlterTable.hpp:
  New interfaces in ALTER TABLE towards NDB kernel
storage/ndb/include/kernel/signaldata/DiAddTab.hpp:
  New section
storage/ndb/include/kernel/signaldata/DictTabInfo.hpp:
  Lots of new parts of table description
storage/ndb/include/kernel/signaldata/LqhFrag.hpp:
  tablespace id specified in LQHFRAGREQ
storage/ndb/include/ndbapi/NdbDictionary.hpp:
  Lots of new methods in NDB dictionary
storage/ndb/src/common/debugger/signaldata/DictTabInfo.cpp:
  Lots of new variables in table description
storage/ndb/src/kernel/blocks/dbdict/Dbdict.cpp:
  Lots of new variables in table description
storage/ndb/src/kernel/blocks/dbdict/Dbdict.hpp:
  Lots of new variables in table description
storage/ndb/src/kernel/blocks/dblqh/DblqhMain.cpp:
  New error insertion
storage/ndb/src/kernel/blocks/dbtup/DbtupMeta.cpp:
  a few extra jam's
storage/ndb/src/ndbapi/NdbBlob.cpp:
  Changes to definition of blob tables
storage/ndb/src/ndbapi/NdbDictionary.cpp:
  Lots of new stuff in NDB dictionary
storage/ndb/src/ndbapi/NdbDictionaryImpl.cpp:
  Lots of new stuff in NDB dictionary
storage/ndb/src/ndbapi/NdbDictionaryImpl.hpp:
  Lots of new stuff in NDB dictionary
storage/ndb/test/ndbapi/test_event.cpp:
  removed use of methods no longer in existence
storage/ndb/tools/restore/Restore.cpp:
  Renamed variable

1 files changed, 116 insertions, 47 deletions

diff --git a/sql/handler.h b/sql/handler.h
index e766797133d..9b870be4505 100644
--- a/sql/handler.h
+++ b/sql/handler.h
@@ -99,6 +99,7 @@
 #define HA_CAN_PARTITION       (1 << 0) /* Partition support */
 #define HA_CAN_UPDATE_PARTITION_KEY (1 << 1)
 #define HA_CAN_PARTITION_UNIQUE (1 << 2)
+#define HA_USE_AUTO_PARTITION (1 << 3)
 
 
 /* bits in index_flags(index_number) for what you can do with index */
@@ -109,30 +110,58 @@
 #define HA_ONLY_WHOLE_INDEX	16	/* Can't use part key searches */
 #define HA_KEYREAD_ONLY         64	/* Support HA_EXTRA_KEYREAD */
 
-/* bits in alter_table_flags */
-#define HA_ONLINE_ADD_EMPTY_PARTITION           0x00000001
-#define HA_ONLINE_DROP_PARTITION                0x00000002
+/*
+  bits in alter_table_flags:
+*/
 /*
   These bits are set if different kinds of indexes can be created
   off-line without re-create of the table (but with a table lock).
 */
-#define HA_ONLINE_ADD_INDEX_NO_WRITES           0x00000004 /*add index w/lock*/
-#define HA_ONLINE_DROP_INDEX_NO_WRITES          0x00000008 /*drop index w/lock*/
-#define HA_ONLINE_ADD_UNIQUE_INDEX_NO_WRITES    0x00000010 /*add unique w/lock*/
-#define HA_ONLINE_DROP_UNIQUE_INDEX_NO_WRITES   0x00000020 /*drop uniq. w/lock*/
-#define HA_ONLINE_ADD_PK_INDEX_NO_WRITES        0x00000040 /*add prim. w/lock*/
-#define HA_ONLINE_DROP_PK_INDEX_NO_WRITES       0x00000080 /*drop prim. w/lock*/
+#define HA_ONLINE_ADD_INDEX_NO_WRITES           (1L << 0) /*add index w/lock*/
+#define HA_ONLINE_DROP_INDEX_NO_WRITES          (1L << 1) /*drop index w/lock*/
+#define HA_ONLINE_ADD_UNIQUE_INDEX_NO_WRITES    (1L << 2) /*add unique w/lock*/
+#define HA_ONLINE_DROP_UNIQUE_INDEX_NO_WRITES   (1L << 3) /*drop uniq. w/lock*/
+#define HA_ONLINE_ADD_PK_INDEX_NO_WRITES        (1L << 4) /*add prim. w/lock*/
+#define HA_ONLINE_DROP_PK_INDEX_NO_WRITES       (1L << 5) /*drop prim. w/lock*/
 /*
   These are set if different kinds of indexes can be created on-line
   (without a table lock). If a handler is capable of one or more of
   these, it should also set the corresponding *_NO_WRITES bit(s).
 */
-#define HA_ONLINE_ADD_INDEX                     0x00000100 /*add index online*/
-#define HA_ONLINE_DROP_INDEX                    0x00000200 /*drop index online*/
-#define HA_ONLINE_ADD_UNIQUE_INDEX              0x00000400 /*add unique online*/
-#define HA_ONLINE_DROP_UNIQUE_INDEX             0x00000800 /*drop uniq. online*/
-#define HA_ONLINE_ADD_PK_INDEX                  0x00001000 /*add prim. online*/
-#define HA_ONLINE_DROP_PK_INDEX                 0x00002000 /*drop prim. online*/
+#define HA_ONLINE_ADD_INDEX                     (1L << 6) /*add index online*/
+#define HA_ONLINE_DROP_INDEX                    (1L << 7) /*drop index online*/
+#define HA_ONLINE_ADD_UNIQUE_INDEX              (1L << 8) /*add unique online*/
+#define HA_ONLINE_DROP_UNIQUE_INDEX             (1L << 9) /*drop uniq. online*/
+#define HA_ONLINE_ADD_PK_INDEX                  (1L << 10)/*add prim. online*/
+#define HA_ONLINE_DROP_PK_INDEX                 (1L << 11)/*drop prim. online*/
+/*
+  HA_PARTITION_FUNCTION_SUPPORTED indicates that the function is
+  supported at all.
+  HA_FAST_CHANGE_PARTITION means that optimised variants of the changes
+  exists but they are not necessarily done online.
+
+  HA_ONLINE_DOUBLE_WRITE means that the handler supports writing to both
+  the new partition and to the old partitions when updating through the
+  old partitioning schema while performing a change of the partitioning.
+  This means that we can support updating of the table while performing
+  the copy phase of the change. For no lock at all also a double write
+  from new to old must exist and this is not required when this flag is
+  set.
+  This is actually removed even before it was introduced the first time.
+  The new idea is that handlers will handle the lock level already in
+  store_lock for ALTER TABLE partitions.
+
+  HA_PARTITION_ONE_PHASE is a flag that can be set by handlers that take
+  care of changing the partitions online and in one phase. Thus all phases
+  needed to handle the change are implemented inside the storage engine.
+  The storage engine must also support auto-discovery since the frm file
+  is changed as part of the change and this change must be controlled by
+  the storage engine. A typical engine to support this is NDB (through
+  WL #2498).
+*/
+#define HA_PARTITION_FUNCTION_SUPPORTED         (1L << 12)
+#define HA_FAST_CHANGE_PARTITION                (1L << 13)
+#define HA_PARTITION_ONE_PHASE                  (1L << 14)
 
 /*
   Index scan will not return records in rowid order. Not guaranteed to be
@@ -140,7 +169,6 @@
 */
 #define HA_KEY_SCAN_NOT_ROR     128 
 
-
 /* operations for disable/enable indexes */
 #define HA_KEY_SWITCH_NONUNIQ      0
 #define HA_KEY_SWITCH_ALL          1
@@ -540,6 +568,8 @@ typedef struct
    int (*start_consistent_snapshot)(THD *thd);
    bool (*flush_logs)();
    bool (*show_status)(THD *thd, stat_print_fn *print, enum ha_stat_type stat);
+   uint (*partition_flags)();
+   uint (*alter_table_flags)(uint flags);
    int (*alter_tablespace)(THD *thd, st_alter_tablespace *ts_info);
    uint32 flags;                                /* global handler flags */
    /* 
@@ -604,10 +634,12 @@ enum partition_state {
   PART_NORMAL= 0,
   PART_IS_DROPPED= 1,
   PART_TO_BE_DROPPED= 2,
-  PART_DROPPING= 3,
-  PART_IS_ADDED= 4,
-  PART_ADDING= 5,
-  PART_ADDED= 6
+  PART_TO_BE_ADDED= 3,
+  PART_TO_BE_REORGED= 4,
+  PART_REORGED_DROPPED= 5,
+  PART_CHANGED= 6,
+  PART_IS_CHANGED= 7,
+  PART_IS_ADDED= 8
 };
 
 typedef struct {
@@ -657,12 +689,12 @@ public:
 
 typedef struct {
   longlong list_value;
-  uint partition_id;
+  uint32 partition_id;
 } LIST_PART_ENTRY;
 
 class partition_info;
 
-typedef bool (*get_part_id_func)(partition_info *part_info,
+typedef int (*get_part_id_func)(partition_info *part_info,
                                  uint32 *part_id);
 typedef uint32 (*get_subpart_id_func)(partition_info *part_info);
 
@@ -732,6 +764,8 @@ public:
   char *part_func_string;
   char *subpart_func_string;
 
+  uchar *part_state;
+
   partition_element *curr_part_elem;
   partition_element *current_partition;
   /*
@@ -748,12 +782,12 @@ public:
   partition_type subpart_type;
 
   uint part_info_len;
+  uint part_state_len;
   uint part_func_len;
   uint subpart_func_len;
 
   uint no_parts;
   uint no_subparts;
-  uint count_curr_parts;
   uint count_curr_subparts;
 
   uint part_error_code;
@@ -764,14 +798,24 @@ public:
   uint no_subpart_fields;
   uint no_full_part_fields;
 
+  /*
+    This variable is used to calculate the partition id when using
+    LINEAR KEY/HASH. This functionality is kept in the MySQL Server
+    but mainly of use to handlers supporting partitioning.
+  */
   uint16 linear_hash_mask;
 
   bool use_default_partitions;
+  bool use_default_no_partitions;
   bool use_default_subpartitions;
+  bool use_default_no_subpartitions;
+  bool default_partitions_setup;
   bool defined_max_value;
   bool list_of_part_fields;
   bool list_of_subpart_fields;
   bool linear_hash_ind;
+  bool fixed;
+  bool from_openfrm;
 
   partition_info()
   : get_partition_id(NULL), get_part_partition_id(NULL),
@@ -782,19 +826,27 @@ public:
     list_array(NULL),
     part_info_string(NULL),
     part_func_string(NULL), subpart_func_string(NULL),
+    part_state(NULL),
     curr_part_elem(NULL), current_partition(NULL),
     default_engine_type(NULL),
     part_result_type(INT_RESULT),
     part_type(NOT_A_PARTITION), subpart_type(NOT_A_PARTITION),
-    part_info_len(0), part_func_len(0), subpart_func_len(0),
+    part_info_len(0), part_state_len(0),
+    part_func_len(0), subpart_func_len(0),
     no_parts(0), no_subparts(0),
-    count_curr_parts(0), count_curr_subparts(0), part_error_code(0),
+    count_curr_subparts(0), part_error_code(0),
     no_list_values(0), no_part_fields(0), no_subpart_fields(0),
     no_full_part_fields(0), linear_hash_mask(0),
     use_default_partitions(TRUE),
-    use_default_subpartitions(TRUE), defined_max_value(FALSE),
+    use_default_no_partitions(TRUE),
+    use_default_subpartitions(TRUE),
+    use_default_no_subpartitions(TRUE),
+    default_partitions_setup(FALSE),
+    defined_max_value(FALSE),
     list_of_part_fields(FALSE), list_of_subpart_fields(FALSE),
-    linear_hash_ind(FALSE)
+    linear_hash_ind(FALSE),
+    fixed(FALSE),
+    from_openfrm(FALSE)
   {
     all_fields_in_PF.clear_all();
     all_fields_in_PPF.clear_all();
@@ -842,6 +894,8 @@ uint get_tot_partitions(partition_info *part_info)
   return part_info->no_parts *
          (is_sub_partitioned(part_info) ? part_info->no_subparts : 1);
 }
+
+
 #endif
 
 typedef struct st_ha_create_information
@@ -891,8 +945,8 @@ typedef struct st_ha_check_opt
 
 #ifdef WITH_PARTITION_STORAGE_ENGINE
 bool is_partition_in_list(char *part_name, List<char> list_part_names);
-bool is_partitions_in_table(partition_info *new_part_info,
-                            partition_info *old_part_info);
+char *are_partitions_in_table(partition_info *new_part_info,
+                              partition_info *old_part_info);
 bool check_reorganise_list(partition_info *new_part_info,
                            partition_info *old_part_info,
                            List<char> list_part_names);
@@ -906,12 +960,13 @@ int get_parts_for_update(const byte *old_data, byte *new_data,
                          uint32 *old_part_id, uint32 *new_part_id);
 int get_part_for_delete(const byte *buf, const byte *rec0,
                         partition_info *part_info, uint32 *part_id);
-bool check_partition_info(partition_info *part_info,handlerton *eng_type,
+bool check_partition_info(partition_info *part_info,handlerton **eng_type,
                           handler *file, ulonglong max_rows);
-bool fix_partition_func(THD *thd, const char *name, TABLE *table);
+bool fix_partition_func(THD *thd, const char *name, TABLE *table,
+                        bool create_table_ind);
 char *generate_partition_syntax(partition_info *part_info,
                                 uint *buf_length, bool use_sql_alloc,
-                                bool add_default_info);
+                                bool write_all);
 bool partition_key_modified(TABLE *table, List<Item> &fields);
 void get_partition_set(const TABLE *table, byte *buf, const uint index,
                        const key_range *key_spec,
@@ -921,7 +976,9 @@ void get_full_part_id_from_key(const TABLE *table, byte *buf,
                                const key_range *key_spec,
                                part_id_range *part_spec);
 bool mysql_unpack_partition(THD *thd, const uchar *part_buf,
-                            uint part_info_len, TABLE *table,
+                            uint part_info_len,
+                            uchar *part_state, uint part_state_len,
+                            TABLE *table, bool is_create_table_ind,
                             handlerton *default_db_type);
 void make_used_partitions_str(partition_info *part_info, String *parts_str);
 uint32 get_list_array_idx_for_endpoint(partition_info *part_info,
@@ -1480,11 +1537,16 @@ public:
   virtual const char *table_type() const =0;
   virtual const char **bas_ext() const =0;
   virtual ulong table_flags(void) const =0;
-  virtual ulong alter_table_flags(void) const { return 0; }
 #ifdef WITH_PARTITION_STORAGE_ENGINE
-  virtual ulong partition_flags(void) const { return 0;}
   virtual int get_default_no_partitions(ulonglong max_rows) { return 1;}
-  virtual void set_part_info(partition_info *part_info) { return; }
+  virtual void set_auto_partitions(partition_info *part_info) { return; }
+  virtual bool get_no_parts(const char *name,
+                            uint *no_parts)
+  {
+    *no_parts= 0;
+    return 0;
+  }
+  virtual void set_part_info(partition_info *part_info) {return;}
 #endif
   virtual ulong index_flags(uint idx, uint part, bool all_parts) const =0;
 
@@ -1530,19 +1592,26 @@ public:
   virtual int create(const char *name, TABLE *form, HA_CREATE_INFO *info)=0;
   virtual int create_handler_files(const char *name) { return FALSE;}
 
-  /*
-    SYNOPSIS
-      drop_partitions()
-      path                        Complete path of db and table name
-    RETURN VALUE
-      TRUE                        Failure
-      FALSE                       Success
-    DESCRIPTION
-      Drop a partition, during this operation no other activity is ongoing
-      in this server on the table.
-  */
+  virtual int change_partitions(HA_CREATE_INFO *create_info,
+                                const char *path,
+                                ulonglong *copied,
+                                ulonglong *deleted,
+                                const void *pack_frm_data,
+                                uint pack_frm_len)
+  { return HA_ERR_WRONG_COMMAND; }
   virtual int drop_partitions(const char *path)
   { return HA_ERR_WRONG_COMMAND; }
+  virtual int rename_partitions(const char *path)
+  { return HA_ERR_WRONG_COMMAND; }
+  virtual int optimize_partitions(THD *thd)
+  { return HA_ERR_WRONG_COMMAND; }
+  virtual int analyze_partitions(THD *thd)
+  { return HA_ERR_WRONG_COMMAND; }
+  virtual int check_partitions(THD *thd)
+  { return HA_ERR_WRONG_COMMAND; }
+  virtual int repair_partitions(THD *thd)
+  { return HA_ERR_WRONG_COMMAND; }
+
   /* lock_count() can be more than one if the table is a MERGE */
   virtual uint lock_count(void) const { return 1; }
   virtual THR_LOCK_DATA **store_lock(THD *thd,