Bug#38804: Query deadlock causes all tables to be inaccessible.

Problem was a mutex added in bug n 27405 for solving a problem
with auto_increment in partitioned innodb tables.
(in ha_partition::write_row over partitions file->ha_write_row)

Solution is to use the patch for bug#33479, which refines the
usage of mutexes for auto_increment.

Backport of bug-33479 from 6.0:

Bug-33479: auto_increment failures in partitioning

Several problems with auto_increment in partitioning
(with MyISAM, InnoDB. Locking issues, not handling
multi-row INSERTs properly etc.)

Changed the auto_increment handling for partitioning:
Added a ha_data variable in table_share for storage engine specific data
such as auto_increment value handling in partitioning, also see WL 4305
and using the ha_data->mutex to lock around read + update.

The idea is this:
Store the table's reserved auto_increment value in
the TABLE_SHARE and use a mutex to, lock it for reading and updating it
and unlocking it, in one block. Only accessing all partitions
when it is not initialized.
Also allow reservations of ranges, and if no one has done a reservation
afterwards, lower the reservation to what was actually used after
the statement is done (via release_auto_increment from WL 3146).
The lock is kept from the first reservation if it is statement based
replication and a multi-row INSERT statement where the number of
candidate rows to insert is not known in advance (like INSERT SELECT,
LOAD DATA, unlike INSERT VALUES (row1), (row2),,(rowN)).

This should also lead to better concurrancy (no need to have a mutex
protection around write_row in all cases)
and work with any local storage engine.

mysql-test/suite/parts/inc/partition_auto_increment.inc:
  Bug#38804: Query deadlock causes all tables to be inaccessible.
  Backporting from 6.0 of:
  Bug-33479: auto_increment failures in partitioning
  
  Test source file for testing auto_increment
mysql-test/suite/parts/r/partition_auto_increment_archive.result:
  Bug#38804: Query deadlock causes all tables to be inaccessible.
  Backporting from 6.0 of:
  Bug-33479: auto_increment failures in partitioning
  
  result file for testing auto_increment
mysql-test/suite/parts/r/partition_auto_increment_blackhole.result:
  Bug#38804: Query deadlock causes all tables to be inaccessible.
  Backporting from 6.0 of:
  Bug-33479: auto_increment failures in partitioning
  
  result file for testing auto_increment
mysql-test/suite/parts/r/partition_auto_increment_innodb.result:
  Bug#38804: Query deadlock causes all tables to be inaccessible.
  Backporting from 6.0 of:
  Bug-33479: auto_increment failures in partitioning
  
  result file for testing auto_increment
mysql-test/suite/parts/r/partition_auto_increment_memory.result:
  Bug#38804: Query deadlock causes all tables to be inaccessible.
  Backporting from 6.0 of:
  Bug-33479: auto_increment failures in partitioning
  
  result file for testing auto_increment
mysql-test/suite/parts/r/partition_auto_increment_myisam.result:
  Bug#38804: Query deadlock causes all tables to be inaccessible.
  Backporting from 6.0 of:
  Bug-33479: auto_increment failures in partitioning
  
  result file for testing auto_increment
mysql-test/suite/parts/r/partition_auto_increment_ndb.result:
  Bug#38804: Query deadlock causes all tables to be inaccessible.
  Backporting from 6.0 of:
  Bug-33479: auto_increment failures in partitioning
  
  result file for testing auto_increment
mysql-test/suite/parts/t/partition_auto_increment_archive.test:
  Bug#38804: Query deadlock causes all tables to be inaccessible.
  Backporting from 6.0 of:
  Bug-33479: auto_increment failures in partitioning
  
  test file for testing auto_increment
mysql-test/suite/parts/t/partition_auto_increment_blackhole.test:
  Bug#38804: Query deadlock causes all tables to be inaccessible.
  Backporting from 6.0 of:
  Bug-33479: auto_increment failures in partitioning
  
  test file for testing auto_increment
mysql-test/suite/parts/t/partition_auto_increment_innodb.test:
  Bug#38804: Query deadlock causes all tables to be inaccessible.
  Backporting from 6.0 of:
  Bug-33479: auto_increment failures in partitioning
  
  test file for testing auto_increment
mysql-test/suite/parts/t/partition_auto_increment_memory.test:
  Bug#38804: Query deadlock causes all tables to be inaccessible.
  Backporting from 6.0 of:
  Bug-33479: auto_increment failures in partitioning
  
  test file for testing auto_increment
mysql-test/suite/parts/t/partition_auto_increment_myisam.test:
  Bug#38804: Query deadlock causes all tables to be inaccessible.
  Backporting from 6.0 of:
  Bug-33479: auto_increment failures in partitioning
  
  test file for testing auto_increment
mysql-test/suite/parts/t/partition_auto_increment_ndb.test:
  Bug#38804: Query deadlock causes all tables to be inaccessible.
  Backporting from 6.0 of:
  Bug-33479: auto_increment failures in partitioning
  
  test file for testing auto_increment
sql/ha_partition.cc:
  Bug#38804: Query deadlock causes all tables to be inaccessible.
  Backporting from 6.0 of:
  Bug-33479: Failures using auto_increment and partitioning
  
  Changed ha_partition::get_auto_increment from file->get_auto_increment
  to file->info(HA_AUTO_STATUS), since it is works better with InnoDB
  (InnoDB can have issues with partitioning and auto_increment,
  where get_auto_increment sometimes can return a non updated value.)
  
  Using the new table_share->ha_data for keeping the auto_increment
  value, shared by all instances of the same table.
  It is read+updated when holding a auto_increment specific mutex.
  Also added release_auto_increment to decrease gaps if possible.
  And a lock for multi-row INSERT statements where the number of candidate
  rows to insert is not known in advance (like INSERT SELECT, LOAD DATA;
  Unlike INSERT INTO (row1),(row2),,(rowN)).
  Fixed a small bug, copied++ to (*copied)++ and the same for deleted.
  Changed from current_thd, to ha_thd()
sql/ha_partition.h:
  Bug#38804: Query deadlock causes all tables to be inaccessible.
  Backporting from 6.0 of:
  Bug-33479: Failures using auto_increment and partitioning
  
  Added a new struct HA_DATA_PARTITION to be used in table_share->ha_data
  Added a private function to set auto_increment values if needed
  Removed the restore_auto_increment (the hander version is better)
  Added lock/unlock functions for auto_increment handling.
  Changed copied/deleted to const.
sql/handler.h:
  Bug#38804: Query deadlock causes all tables to be inaccessible.
  Backporting from 6.0 of:
  Bug-33479: auto_increment failures in partitioning
  
  Added const for changed_partitions
  Added comments about SQLCOM_TRUNCATE for delete_all_rows
sql/table.h:
  Bug#38804: Query deadlock causes all tables to be inaccessible.
  Backporting from 6.0 of:
  Bug-33479: Failures using auto_increment and partitioning
  
  Added a variable in table_share: ha_data for storage of storage engine
  specific data (such as auto_increment handling in partitioning).

1 files changed, 58 insertions, 4 deletions

diff --git a/sql/ha_partition.h b/sql/ha_partition.h
index 97f5624608f..908248e1dda 100644
--- a/sql/ha_partition.h
+++ b/sql/ha_partition.h
@@ -37,6 +37,15 @@ typedef struct st_partition_share
 } PARTITION_SHARE;
 #endif
 
+/**
+  Partition specific ha_data struct.
+  @todo: move all partition specific data from TABLE_SHARE here.
+*/
+typedef struct st_ha_data_partition
+{
+  ulonglong next_auto_inc_val;                 /**< first non reserved value */
+  bool auto_inc_initialized;
+} HA_DATA_PARTITION;
 
 #define PARTITION_BYTES_IN_POS 2
 class ha_partition :public handler
@@ -141,6 +150,12 @@ private:
     "own" the m_part_info structure.
   */
   bool is_clone;
+  bool auto_increment_lock;             /**< lock reading/updating auto_inc */
+  /**
+    Flag to keep the auto_increment lock through out the statement.
+    This to ensure it will work with statement based replication.
+  */
+  bool auto_increment_safe_stmt_log_lock;
 public:
   handler *clone(MEM_ROOT *mem_root);
   virtual void set_part_info(partition_info *part_info)
@@ -197,8 +212,8 @@ public:
   virtual char *update_table_comment(const char *comment);
   virtual int change_partitions(HA_CREATE_INFO *create_info,
                                 const char *path,
-                                ulonglong *copied,
-                                ulonglong *deleted,
+                                ulonglong * const copied,
+                                ulonglong * const deleted,
                                 const uchar *pack_frm_data,
                                 size_t pack_frm_len);
   virtual int drop_partitions(const char *path);
@@ -212,7 +227,7 @@ public:
   virtual void change_table_ptr(TABLE *table_arg, TABLE_SHARE *share);
 private:
   int prepare_for_rename();
-  int copy_partitions(ulonglong *copied, ulonglong *deleted);
+  int copy_partitions(ulonglong * const copied, ulonglong * const deleted);
   void cleanup_new_partition(uint part_count);
   int prepare_new_partition(TABLE *table, HA_CREATE_INFO *create_info,
                             handler *file, const char *part_name,
@@ -829,12 +844,51 @@ public:
     auto_increment_column_changed
      -------------------------------------------------------------------------
   */
-  virtual void restore_auto_increment(ulonglong prev_insert_id);
   virtual void get_auto_increment(ulonglong offset, ulonglong increment,
                                   ulonglong nb_desired_values,
                                   ulonglong *first_value,
                                   ulonglong *nb_reserved_values);
   virtual void release_auto_increment();
+private:
+  virtual int reset_auto_increment(ulonglong value);
+  virtual void lock_auto_increment()
+  {
+    /* lock already taken */
+    if (auto_increment_safe_stmt_log_lock)
+      return;
+    DBUG_ASSERT(table_share->ha_data && !auto_increment_lock);
+    if(table_share->tmp_table == NO_TMP_TABLE)
+    {
+      auto_increment_lock= TRUE;
+      pthread_mutex_lock(&table_share->mutex);
+    }
+  }
+  virtual void unlock_auto_increment()
+  {
+    DBUG_ASSERT(table_share->ha_data);
+    /*
+      If auto_increment_safe_stmt_log_lock is true, we have to keep the lock.
+      It will be set to false and thus unlocked at the end of the statement by
+      ha_partition::release_auto_increment.
+    */
+    if(auto_increment_lock && !auto_increment_safe_stmt_log_lock)
+    {
+      pthread_mutex_unlock(&table_share->mutex);
+      auto_increment_lock= FALSE;
+    }
+  }
+  virtual void set_auto_increment_if_higher(const ulonglong nr)
+  {
+    HA_DATA_PARTITION *ha_data= (HA_DATA_PARTITION*) table_share->ha_data;
+    lock_auto_increment();
+    /* must check when the mutex is taken */
+    if (nr >= ha_data->next_auto_inc_val)
+      ha_data->next_auto_inc_val= nr + 1;
+    ha_data->auto_inc_initialized= TRUE;
+    unlock_auto_increment();
+  }
+
+public:
 
   /*
      -------------------------------------------------------------------------