WL #2604: Partition Management

Optimised version of ADD/DROP/REORGANIZE partitions for
non-NDB storage engines.
New syntax to handle REBUILD/OPTIMIZE/ANALYZE/CHECK/REPAIR partitions
Quite a few bug fixes


include/thr_lock.h:
  New method to downgrade locks from TL_WRITE_ONLY
  Possibility to upgrade lock while aborting locks
mysql-test/r/ndb_autodiscover.result:
  Fix for lowercase and that all NDB tables are now partitioned
mysql-test/r/ndb_bitfield.result:
  Fix for lowercase and that all NDB tables are now partitioned
mysql-test/r/ndb_gis.result:
  Fix for lowercase and that all NDB tables are now partitioned
mysql-test/r/ndb_partition_key.result:
  New test case
mysql-test/r/partition.result:
  New test case
mysql-test/r/partition_error.result:
  New test case
mysql-test/r/partition_mgm_err.result:
  Fix of test case results
mysql-test/t/disabled.def:
  partition_03ndb still has bug
mysql-test/t/ndb_partition_key.test:
  New test cases for new functionality and bugs
mysql-test/t/partition.test:
  New test cases for new functionality and bugs
mysql-test/t/partition_error.test:
  New test cases for new functionality and bugs
mysql-test/t/partition_mgm_err.test:
  New test cases for new functionality and bugs
mysys/thr_lock.c:
  New method to downgrade TL_WRITE_ONLY locks
  Possibility to specify if locks are to be upgraded at abort locks
sql/ha_archive.cc:
  New handlerton methods
sql/ha_berkeley.cc:
  New handlerton methods
sql/ha_blackhole.cc:
  New handlerton methods
sql/ha_federated.cc:
  New handlerton methods
sql/ha_heap.cc:
  New handlerton methods
sql/ha_innodb.cc:
  New handlerton methods
sql/ha_myisam.cc:
  New handlerton methods
sql/ha_myisammrg.cc:
  New handlerton methods
sql/ha_ndbcluster.cc:
  New handlerton methods
  Moved out packfrm and unpackfrm methods
  Adapted many parts to use table_share instead of table->s
  Ensured that .ndb file uses filename and not tablename
  according to new encoding of names (WL 1324)
  All NDB tables are partitioned and set up partition info
  Fixed such that tablenames use tablenames and not filenames in NDB
  NDB uses auto partitioning for ENGINE=NDB tables
  Warning for very large tables
  Set RANGE data
  Set LIST data
  New method to set-up partition info
  Set Default number of partitions flag
  Set linear hash flag
  Set node group array
  Set number of fragments
  Set max rows
  Set tablespace names
  New method to get number of partitions of table to use at open table
sql/ha_ndbcluster.h:
  Removed partition_flags and alter_table_flags from handler class
  A couple of new and changed method headers
sql/ha_ndbcluster_binlog.cc:
  Use new method headers
sql/ha_partition.cc:
  New handlerton methods
  Lots of new function headers
  Use #P# as separator between table name and partition name and
  #SP# as separator between partition name and subpartition name
  Use filename encoding for files both of table name part and of
  partition name parts
  New method to drop partitions based on partition state
  New method to rename partitions based on partition state
  New methods to optimize, analyze, check and repair partitions
  New methods to optimize, analyze, check and repair table
  Helper method to create new partition, open it and external lock
  it, not needed to lock it internally since no one else knows about
  it yet.
  Cleanup method at error for new partitions
  New methods to perform bulk of work at ADD/REORGANIZE partitions
  (change_partitions, copy_partitions)
sql/ha_partition.h:
  New methods and variables
  A few dropped ones and a few changed ones
sql/handler.cc:
  Handlerton interface changes
  New flag to open_table_from_share
sql/handler.h:
  New alter_table_flags
  New partition flags
  New partition states
  More states for default handling
  Lots of new, dropped and changed interfaces
sql/lex.h:
  Added REBUILD and changed name of REORGANISE to REORGANIZE
sql/lock.cc:
  Method to downgrade locks
  Able to specify if locks upgraded on abort locks
sql/log.cc:
  New handlerton methods
sql/mysql_priv.h:
  Lots of new interfaces
sql/share/errmsg.txt:
  Lots of new, dropped and changed error messages
sql/sql_base.cc:
  Adapted to new method headers
  New method to abort and upgrade lock
  New method to close open tables and downgrade lock
  New method to wait for completed table
sql/sql_lex.h:
  New flags
sql/sql_partition.cc:
  Return int instead of bool in get_partition_id
  More defaults handling
  Make use of new mem_alloc_error method
  More work on function headers
  Changes to generate partition syntax to cater for intermediate
  partition states
  Lots of new code with large comments describing new features for
  Partition Management:
  ADD/DROP/REORGANIZE/OPTIMIZE/ANALYZE/CHECK/REPAIR partitions
sql/sql_show.cc:
  Minors
sql/sql_table.cc:
  Moved a couple of methods
  New methods to copy create lists and key lists
  for use with mysql_prepare_table
  New method to write frm file
  New handling of handlers with auto partitioning
  Fix CREATE TABLE LIKE
  Moved code for ADD/DROP/REORGANIZE partitions
  Use handlerton method for alter_table_flags
sql/sql_yacc.yy:
  More memory alloc error checks
  New syntax for REBUILD, ANALYZE, CHECK, OPTIMIZE, REPAIR partitions
sql/table.cc:
  Fix length of extra part to be 4 bytes
  Partition state introduced in frm file
sql/table.h:
  Partition state introduced
sql/unireg.cc:
  Partition state introduced
  Default partition
storage/csv/ha_tina.cc:
  New handlerton methods
storage/example/ha_example.cc:
  New handlerton methods
storage/ndb/include/kernel/ndb_limits.h:
  RANGE DATA
storage/ndb/include/kernel/signaldata/AlterTable.hpp:
  New interfaces in ALTER TABLE towards NDB kernel
storage/ndb/include/kernel/signaldata/DiAddTab.hpp:
  New section
storage/ndb/include/kernel/signaldata/DictTabInfo.hpp:
  Lots of new parts of table description
storage/ndb/include/kernel/signaldata/LqhFrag.hpp:
  tablespace id specified in LQHFRAGREQ
storage/ndb/include/ndbapi/NdbDictionary.hpp:
  Lots of new methods in NDB dictionary
storage/ndb/src/common/debugger/signaldata/DictTabInfo.cpp:
  Lots of new variables in table description
storage/ndb/src/kernel/blocks/dbdict/Dbdict.cpp:
  Lots of new variables in table description
storage/ndb/src/kernel/blocks/dbdict/Dbdict.hpp:
  Lots of new variables in table description
storage/ndb/src/kernel/blocks/dblqh/DblqhMain.cpp:
  New error insertion
storage/ndb/src/kernel/blocks/dbtup/DbtupMeta.cpp:
  a few extra jam's
storage/ndb/src/ndbapi/NdbBlob.cpp:
  Changes to definition of blob tables
storage/ndb/src/ndbapi/NdbDictionary.cpp:
  Lots of new stuff in NDB dictionary
storage/ndb/src/ndbapi/NdbDictionaryImpl.cpp:
  Lots of new stuff in NDB dictionary
storage/ndb/src/ndbapi/NdbDictionaryImpl.hpp:
  Lots of new stuff in NDB dictionary
storage/ndb/test/ndbapi/test_event.cpp:
  removed use of methods no longer in existence
storage/ndb/tools/restore/Restore.cpp:
  Renamed variable

1 files changed, 209 insertions, 2 deletions

diff --git a/mysys/thr_lock.c b/mysys/thr_lock.c
index f5a8b618949..4b3e03750c8 100644
--- a/mysys/thr_lock.c
+++ b/mysys/thr_lock.c
@@ -1009,7 +1009,7 @@ void thr_multi_unlock(THR_LOCK_DATA **data,uint count)
   TL_WRITE_ONLY to abort any new accesses to the lock
 */
 
-void thr_abort_locks(THR_LOCK *lock)
+void thr_abort_locks(THR_LOCK *lock, bool upgrade_lock)
 {
   THR_LOCK_DATA *data;
   DBUG_ENTER("thr_abort_locks");
@@ -1031,7 +1031,7 @@ void thr_abort_locks(THR_LOCK *lock)
   lock->read_wait.last= &lock->read_wait.data;
   lock->write_wait.last= &lock->write_wait.data;
   lock->read_wait.data=lock->write_wait.data=0;
-  if (lock->write.data)
+  if (upgrade_lock && lock->write.data)
     lock->write.data->type=TL_WRITE_ONLY;
   pthread_mutex_unlock(&lock->mutex);
   DBUG_VOID_RETURN;
@@ -1089,6 +1089,213 @@ my_bool thr_abort_locks_for_thread(THR_LOCK *lock, pthread_t thread)
 }
 
 
+/*
+  Downgrade a WRITE_* to a lower WRITE level
+  SYNOPSIS
+    thr_downgrade_write_lock()
+    in_data                   Lock data of thread downgrading its lock
+    new_lock_type             New write lock type
+  RETURN VALUE
+    NONE
+  DESCRIPTION
+    This can be used to downgrade a lock already owned. When the downgrade
+    occurs also other waiters, both readers and writers can be allowed to
+    start.
+    The previous lock is often TL_WRITE_ONLY but can also be
+    TL_WRITE and TL_WRITE_ALLOW_READ. The normal downgrade variants are
+    TL_WRITE_ONLY => TL_WRITE_ALLOW_READ After a short exclusive lock
+    TL_WRITE_ALLOW_READ => TL_WRITE_ALLOW_WRITE After discovering that the
+    operation didn't need such a high lock.
+    TL_WRITE_ONLY => TL_WRITE after a short exclusive lock while holding a
+    write table lock
+    TL_WRITE_ONLY => TL_WRITE_ALLOW_WRITE After a short exclusive lock after
+    already earlier having dongraded lock to TL_WRITE_ALLOW_WRITE
+    The implementation is conservative and rather don't start rather than
+    go on unknown paths to start, the common cases are handled.
+
+    NOTE:
+    In its current implementation it is only allowed to downgrade from
+    TL_WRITE_ONLY. In this case there are no waiters. Thus no wake up
+    logic is required.
+*/
+
+void thr_downgrade_write_lock(THR_LOCK_DATA *in_data,
+                              enum thr_lock_type new_lock_type)
+{
+  THR_LOCK *lock=in_data->lock;
+  THR_LOCK_DATA *data, *next;
+  enum thr_lock_type old_lock_type= in_data->type;
+  bool start_writers= FALSE;
+  bool start_readers= FALSE;
+  DBUG_ENTER("thr_downgrade_write_only_lock");
+
+  pthread_mutex_lock(&lock->mutex);
+  DBUG_ASSERT(old_lock_type == TL_WRITE_ONLY);
+  DBUG_ASSERT(old_lock_type > new_lock_type);
+  in_data->type= new_lock_type;
+  check_locks(lock,"after downgrading lock",0);
+#if 0
+  switch (old_lock_type)
+  {
+    case TL_WRITE_ONLY:
+    case TL_WRITE:
+    case TL_WRITE_LOW_PRIORITY:
+    /*
+      Previous lock was exclusive we are now ready to start up most waiting
+      threads.
+    */
+      switch (new_lock_type)
+      {
+        case TL_WRITE_ALLOW_READ:
+        /* Still cannot start WRITE operations. Can only start readers.  */
+          start_readers= TRUE;
+          break;
+        case TL_WRITE:
+        case TL_WRITE_LOW_PRIORITY:
+        /*
+           Still cannot start anything, but new requests are no longer
+           aborted.
+        */
+          break;
+        case TL_WRITE_ALLOW_WRITE:
+        /*
+          We can start both writers and readers.
+        */
+          start_writers= TRUE;
+          start_readers= TRUE;
+          break;
+        case TL_WRITE_CONCURRENT_INSERT:
+        case TL_WRITE_DELAYED:
+        /*
+          This routine is not designed for those. Lock will be downgraded
+          but no start of waiters will occur. This is not the optimal but
+          should be a correct behaviour.
+        */
+          break;
+        default:
+          DBUG_ASSERT(0);
+      }
+      break;
+    case TL_WRITE_DELAYED:
+    case TL_WRITE_CONCURRENT_INSERT:
+    /*
+      This routine is not designed for those. Lock will be downgraded
+      but no start of waiters will occur. This is not the optimal but
+      should be a correct behaviour.
+    */
+      break;
+    case TL_WRITE_ALLOW_READ:
+      DBUG_ASSERT(new_lock_type == TL_WRITE_ALLOW_WRITE);
+      /*
+        Previously writers were not allowed to start, now it is ok to
+        start them again. Readers are already allowed so no reason to
+        handle them.
+      */
+      start_writers= TRUE;
+      break;
+    default:
+      DBUG_ASSERT(0);
+      break;
+  }
+  if (start_writers)
+  {
+    /*
+      At this time the only active writer can be ourselves. Thus we need
+      not worry about that there are other concurrent write operations
+      active on the table. Thus we only need to worry about starting
+      waiting operations.
+      We also only come here with TL_WRITE_ALLOW_WRITE as the new
+      lock type, thus we can start other writers also of the same type.
+      If we find a lock at exclusive level >= TL_WRITE_LOW_PRIORITY we
+      don't start any more operations that would be mean those operations
+      will have to wait for things started afterwards.
+    */
+    DBUG_ASSERT(new_lock_type == TL_WRITE_ALLOW_WRITE);
+    for (data=lock->write_wait.data; data ; data= next)
+    {
+      /*
+        All WRITE requests compatible with new lock type are also
+        started
+      */
+      next= data->next;
+      if (start_writers && data->type == new_lock_type)
+      {
+        pthread_cond_t *cond= data->cond;
+        /*
+          It is ok to start this waiter.
+          Move from being first in wait queue to be last in write queue.
+        */
+        if (((*data->prev)= data->next))
+          data->next->prev= data->prev;
+        else
+          lock->write_wait.last= data->prev;
+        data->prev= lock->write.last;
+        lock->write.last= &data->next;
+        data->next= 0;
+        check_locks(lock, "Started write lock after downgrade",0);
+        data->cond= 0;
+        pthread_cond_signal(cond);
+      }
+      else
+      {
+        /*
+          We found an incompatible lock, we won't start any more write
+          requests to avoid letting writers pass other writers in the
+          queue.
+        */
+        start_writers= FALSE;
+        if (data->type >= TL_WRITE_LOW_PRIORITY)
+        {
+          /*
+            We have an exclusive writer in the queue so we won't start
+            readers either.
+          */
+          start_readers= FALSE;
+        }
+      }
+    }
+  }
+  if (start_readers)
+  {
+    DBUG_ASSERT(new_lock_type == TL_WRITE_ALLOW_WRITE ||
+                new_lock_type == TL_WRITE_ALLOW_READ);
+    /*
+      When we come here we know that the write locks are
+      TL_WRITE_ALLOW_WRITE or TL_WRITE_ALLOW_READ. This means that reads
+      are ok
+    */
+    for (data=lock->read_wait.data; data ; data=next)
+    {
+      next= data->next;
+      /*
+        All reads are ok to start now except TL_READ_NO_INSERT when
+        write lock is TL_WRITE_ALLOW_READ.
+      */
+      if (new_lock_type != TL_WRITE_ALLOW_READ ||
+          data->type != TL_READ_NO_INSERT)
+      {
+        pthread_cond_t *cond= data->cond;
+        if (((*data->prev)= data->next))
+          data->next->prev= data->prev;
+        else
+          lock->read_wait.last= data->prev;
+        data->prev= lock->read.last;
+        lock->read.last= &data->next;
+        data->next= 0;
+
+        if (data->type == TL_READ_NO_INSERT)
+          lock->read_no_write_count++;
+        check_locks(lock, "Started read lock after downgrade",0);
+        data->cond= 0;
+        pthread_cond_signal(cond);
+      }
+    }
+  }
+  check_locks(lock,"after starting waiters after downgrading lock",0);
+#endif
+  pthread_mutex_unlock(&lock->mutex);
+  DBUG_VOID_RETURN;
+}
 
 /* Upgrade a WRITE_DELAY lock to a WRITE_LOCK */