WL #2604: Partition Management

Optimised version of ADD/DROP/REORGANIZE partitions for
non-NDB storage engines.
New syntax to handle REBUILD/OPTIMIZE/ANALYZE/CHECK/REPAIR partitions
Quite a few bug fixes


include/thr_lock.h:
  New method to downgrade locks from TL_WRITE_ONLY
  Possibility to upgrade lock while aborting locks
mysql-test/r/ndb_autodiscover.result:
  Fix for lowercase and that all NDB tables are now partitioned
mysql-test/r/ndb_bitfield.result:
  Fix for lowercase and that all NDB tables are now partitioned
mysql-test/r/ndb_gis.result:
  Fix for lowercase and that all NDB tables are now partitioned
mysql-test/r/ndb_partition_key.result:
  New test case
mysql-test/r/partition.result:
  New test case
mysql-test/r/partition_error.result:
  New test case
mysql-test/r/partition_mgm_err.result:
  Fix of test case results
mysql-test/t/disabled.def:
  partition_03ndb still has bug
mysql-test/t/ndb_partition_key.test:
  New test cases for new functionality and bugs
mysql-test/t/partition.test:
  New test cases for new functionality and bugs
mysql-test/t/partition_error.test:
  New test cases for new functionality and bugs
mysql-test/t/partition_mgm_err.test:
  New test cases for new functionality and bugs
mysys/thr_lock.c:
  New method to downgrade TL_WRITE_ONLY locks
  Possibility to specify if locks are to be upgraded at abort locks
sql/ha_archive.cc:
  New handlerton methods
sql/ha_berkeley.cc:
  New handlerton methods
sql/ha_blackhole.cc:
  New handlerton methods
sql/ha_federated.cc:
  New handlerton methods
sql/ha_heap.cc:
  New handlerton methods
sql/ha_innodb.cc:
  New handlerton methods
sql/ha_myisam.cc:
  New handlerton methods
sql/ha_myisammrg.cc:
  New handlerton methods
sql/ha_ndbcluster.cc:
  New handlerton methods
  Moved out packfrm and unpackfrm methods
  Adapted many parts to use table_share instead of table->s
  Ensured that .ndb file uses filename and not tablename
  according to new encoding of names (WL 1324)
  All NDB tables are partitioned and set up partition info
  Fixed such that tablenames use tablenames and not filenames in NDB
  NDB uses auto partitioning for ENGINE=NDB tables
  Warning for very large tables
  Set RANGE data
  Set LIST data
  New method to set-up partition info
  Set Default number of partitions flag
  Set linear hash flag
  Set node group array
  Set number of fragments
  Set max rows
  Set tablespace names
  New method to get number of partitions of table to use at open table
sql/ha_ndbcluster.h:
  Removed partition_flags and alter_table_flags from handler class
  A couple of new and changed method headers
sql/ha_ndbcluster_binlog.cc:
  Use new method headers
sql/ha_partition.cc:
  New handlerton methods
  Lots of new function headers
  Use #P# as separator between table name and partition name and
  #SP# as separator between partition name and subpartition name
  Use filename encoding for files both of table name part and of
  partition name parts
  New method to drop partitions based on partition state
  New method to rename partitions based on partition state
  New methods to optimize, analyze, check and repair partitions
  New methods to optimize, analyze, check and repair table
  Helper method to create new partition, open it and external lock
  it, not needed to lock it internally since no one else knows about
  it yet.
  Cleanup method at error for new partitions
  New methods to perform bulk of work at ADD/REORGANIZE partitions
  (change_partitions, copy_partitions)
sql/ha_partition.h:
  New methods and variables
  A few dropped ones and a few changed ones
sql/handler.cc:
  Handlerton interface changes
  New flag to open_table_from_share
sql/handler.h:
  New alter_table_flags
  New partition flags
  New partition states
  More states for default handling
  Lots of new, dropped and changed interfaces
sql/lex.h:
  Added REBUILD and changed name of REORGANISE to REORGANIZE
sql/lock.cc:
  Method to downgrade locks
  Able to specify if locks upgraded on abort locks
sql/log.cc:
  New handlerton methods
sql/mysql_priv.h:
  Lots of new interfaces
sql/share/errmsg.txt:
  Lots of new, dropped and changed error messages
sql/sql_base.cc:
  Adapted to new method headers
  New method to abort and upgrade lock
  New method to close open tables and downgrade lock
  New method to wait for completed table
sql/sql_lex.h:
  New flags
sql/sql_partition.cc:
  Return int instead of bool in get_partition_id
  More defaults handling
  Make use of new mem_alloc_error method
  More work on function headers
  Changes to generate partition syntax to cater for intermediate
  partition states
  Lots of new code with large comments describing new features for
  Partition Management:
  ADD/DROP/REORGANIZE/OPTIMIZE/ANALYZE/CHECK/REPAIR partitions
sql/sql_show.cc:
  Minors
sql/sql_table.cc:
  Moved a couple of methods
  New methods to copy create lists and key lists
  for use with mysql_prepare_table
  New method to write frm file
  New handling of handlers with auto partitioning
  Fix CREATE TABLE LIKE
  Moved code for ADD/DROP/REORGANIZE partitions
  Use handlerton method for alter_table_flags
sql/sql_yacc.yy:
  More memory alloc error checks
  New syntax for REBUILD, ANALYZE, CHECK, OPTIMIZE, REPAIR partitions
sql/table.cc:
  Fix length of extra part to be 4 bytes
  Partition state introduced in frm file
sql/table.h:
  Partition state introduced
sql/unireg.cc:
  Partition state introduced
  Default partition
storage/csv/ha_tina.cc:
  New handlerton methods
storage/example/ha_example.cc:
  New handlerton methods
storage/ndb/include/kernel/ndb_limits.h:
  RANGE DATA
storage/ndb/include/kernel/signaldata/AlterTable.hpp:
  New interfaces in ALTER TABLE towards NDB kernel
storage/ndb/include/kernel/signaldata/DiAddTab.hpp:
  New section
storage/ndb/include/kernel/signaldata/DictTabInfo.hpp:
  Lots of new parts of table description
storage/ndb/include/kernel/signaldata/LqhFrag.hpp:
  tablespace id specified in LQHFRAGREQ
storage/ndb/include/ndbapi/NdbDictionary.hpp:
  Lots of new methods in NDB dictionary
storage/ndb/src/common/debugger/signaldata/DictTabInfo.cpp:
  Lots of new variables in table description
storage/ndb/src/kernel/blocks/dbdict/Dbdict.cpp:
  Lots of new variables in table description
storage/ndb/src/kernel/blocks/dbdict/Dbdict.hpp:
  Lots of new variables in table description
storage/ndb/src/kernel/blocks/dblqh/DblqhMain.cpp:
  New error insertion
storage/ndb/src/kernel/blocks/dbtup/DbtupMeta.cpp:
  a few extra jam's
storage/ndb/src/ndbapi/NdbBlob.cpp:
  Changes to definition of blob tables
storage/ndb/src/ndbapi/NdbDictionary.cpp:
  Lots of new stuff in NDB dictionary
storage/ndb/src/ndbapi/NdbDictionaryImpl.cpp:
  Lots of new stuff in NDB dictionary
storage/ndb/src/ndbapi/NdbDictionaryImpl.hpp:
  Lots of new stuff in NDB dictionary
storage/ndb/test/ndbapi/test_event.cpp:
  removed use of methods no longer in existence
storage/ndb/tools/restore/Restore.cpp:
  Renamed variable

1 files changed, 55 insertions, 33 deletions

diff --git a/sql/ha_partition.h b/sql/ha_partition.h
index 03acf217419..60f6a5ca15b 100644
--- a/sql/ha_partition.h
+++ b/sql/ha_partition.h
@@ -49,10 +49,15 @@ private:
     partition_no_index_scan= 3
   };
   /* Data for the partition handler */
+  int  m_mode;                          // Open mode
+  uint m_open_test_lock;                // Open test_if_locked
   char *m_file_buffer;                  // Buffer with names
   char *m_name_buffer_ptr;		// Pointer to first partition name
   handlerton **m_engine_array;          // Array of types of the handlers
   handler **m_file;                     // Array of references to handler inst.
+  handler **m_new_file;                 // Array of references to new handlers
+  handler **m_reorged_file;             // Reorganised partitions
+  handler **m_added_file;               // Added parts kept for errors
   partition_info *m_part_info;          // local reference to partition
   byte *m_start_key_ref;                // Reference of start key in current
                                         // index scan info
@@ -60,7 +65,7 @@ private:
   byte *m_ordered_rec_buffer;           // Row and key buffer for ord. idx scan
   KEY *m_curr_key_info;                 // Current index
   byte *m_rec0;                         // table->record[0]
-  QUEUE queue;                          // Prio queue used by sorted read
+  QUEUE m_queue;                        // Prio queue used by sorted read
   /*
     Since the partition handler is a handler on top of other handlers, it
     is necessary to keep information about what the underlying handler
@@ -71,6 +76,7 @@ private:
   u_long m_table_flags;
   u_long m_low_byte_first;
 
+  uint m_reorged_parts;                  // Number of reorganised parts
   uint m_tot_parts;                      // Total number of partitions;
   uint m_no_locks;                        // For engines like ha_blackhole, which needs no locks
   uint m_last_part;                      // Last file that we update,write
@@ -172,21 +178,38 @@ public:
   */
   virtual int delete_table(const char *from);
   virtual int rename_table(const char *from, const char *to);
-  virtual int create(const char *name, TABLE * form,
-		     HA_CREATE_INFO * create_info);
+  virtual int create(const char *name, TABLE *form,
+		     HA_CREATE_INFO *create_info);
   virtual int create_handler_files(const char *name);
-  virtual void update_create_info(HA_CREATE_INFO * create_info);
+  virtual void update_create_info(HA_CREATE_INFO *create_info);
   virtual char *update_table_comment(const char *comment);
+  virtual int change_partitions(HA_CREATE_INFO *create_info,
+                                const char *path,
+                                ulonglong *copied,
+                                ulonglong *deleted,
+                                const void *pack_frm_data,
+                                uint pack_frm_len);
   virtual int drop_partitions(const char *path);
+  virtual int rename_partitions(const char *path);
+  bool get_no_parts(const char *name, uint *no_parts)
+  {
+    DBUG_ENTER("ha_partition::get_no_parts");
+    *no_parts= m_tot_parts;
+    DBUG_RETURN(0);
+  }
 private:
+  int copy_partitions(ulonglong *copied, ulonglong *deleted);
+  void cleanup_new_partition(uint part_count);
+  int prepare_new_partition(TABLE *table, HA_CREATE_INFO *create_info,
+                            handler *file, const char *part_name);
   /*
     delete_table, rename_table and create uses very similar logic which
     is packed into this routine.
   */
     uint del_ren_cre_table(const char *from,
 			   const char *to= NULL,
-			   TABLE * table_arg= NULL,
-			   HA_CREATE_INFO * create_info= NULL);
+			   TABLE *table_arg= NULL,
+			   HA_CREATE_INFO *create_info= NULL);
   /*
     One method to create the table_name.par file containing the names of the
     underlying partitions, their engine and the number of partitions.
@@ -647,30 +670,8 @@ public:
     index scan module.
     (NDB)
   */
-  virtual ulong alter_table_flags(void) const
-  {
-    //return HA_ONLINE_ADD_EMPTY_PARTITION + HA_ONLINE_DROP_PARTITION;
-    return HA_ONLINE_DROP_PARTITION;
-  }
   virtual ulong table_flags() const
   { return m_table_flags; }
-  /*
-    HA_CAN_PARTITION:
-    Used by storage engines that can handle partitioning without this
-    partition handler
-    (Partition, NDB)
-
-    HA_CAN_UPDATE_PARTITION_KEY:
-    Set if the handler can update fields that are part of the partition
-    function.
-
-    HA_CAN_PARTITION_UNIQUE:
-    Set if the handler can handle unique indexes where the fields of the
-    unique key are not part of the fields of the partition function. Thus
-    a unique key can be set on all fields.
-  */
-  virtual ulong partition_flags() const
-  { return HA_CAN_PARTITION; }
 
   /*
     This is a bitmap of flags that says how the storage engine
@@ -834,6 +835,8 @@ public:
     description of how the CREATE TABLE part to define FOREIGN KEY's is done.
     free_foreign_key_create_info is used to free the memory area that provided
     this description.
+    can_switch_engines checks if it is ok to switch to a new engine based on
+    the foreign key info in the table.
     -------------------------------------------------------------------------
 
     virtual char* get_foreign_key_create_info()
@@ -843,7 +846,7 @@ public:
     List<FOREIGN_KEY_INFO> *f_key_list)
     virtual uint referenced_by_foreign_key()
   */
-
+    virtual bool can_switch_engines();
   /*
     -------------------------------------------------------------------------
     MODULE fulltext index
@@ -892,16 +895,35 @@ public:
     -------------------------------------------------------------------------
     MODULE admin MyISAM
     -------------------------------------------------------------------------
+
+    -------------------------------------------------------------------------
+      OPTIMIZE TABLE, CHECK TABLE, ANALYZE TABLE and REPAIR TABLE are
+      mapped to a routine that handles looping over a given set of
+      partitions and those routines send a flag indicating to execute on
+      all partitions.
+    -------------------------------------------------------------------------
+  */
+    virtual int optimize(THD* thd, HA_CHECK_OPT *check_opt);
+    virtual int analyze(THD* thd, HA_CHECK_OPT *check_opt);
+    virtual int check(THD* thd, HA_CHECK_OPT *check_opt);
+    virtual int repair(THD* thd, HA_CHECK_OPT *check_opt);
+    virtual int optimize_partitions(THD *thd);
+    virtual int analyze_partitions(THD *thd);
+    virtual int check_partitions(THD *thd);
+    virtual int repair_partitions(THD *thd);
+
+    private:
+    int handle_opt_partitions(THD *thd, HA_CHECK_OPT *check_opt,
+                              uint flags, bool all_parts);
+    public:
+  /*
+    -------------------------------------------------------------------------
     Admin commands not supported currently (almost purely MyISAM routines)
     This means that the following methods are not implemented:
     -------------------------------------------------------------------------
 
-    virtual int check(THD* thd, HA_CHECK_OPT *check_opt);
     virtual int backup(TD* thd, HA_CHECK_OPT *check_opt);
     virtual int restore(THD* thd, HA_CHECK_OPT *check_opt);
-    virtual int repair(THD* thd, HA_CHECK_OPT *check_opt);
-    virtual int optimize(THD* thd, HA_CHECK_OPT *check_opt);
-    virtual int analyze(THD* thd, HA_CHECK_OPT *check_opt);
     virtual int assign_to_keycache(THD* thd, HA_CHECK_OPT *check_opt);
     virtual int preload_keys(THD *thd, HA_CHECK_OPT *check_opt);
     virtual bool check_and_repair(THD *thd);