WL#3262 add mutex lock order checking to safemutex (also called safe_mutex_deadlock_detector)

This writes a warning on stderr if one uses mutex in different order,
like if one in one case would lock mutex in the order A,B and in another case
would lock mutex in the order B,A
This is inspired by and loosely based on the LOCKDEP patch by Jonas
Wrong mutex order is either fixed or mutex are marked with MYF_NO_DEADLOCK_DETECTION
if used inconsistently (need to be fixed by server team)

KNOWN_BUGS.txt:
  Added information that one need to dump and restore Maria tables
include/hash.h:
  Added prototype function for walking over all elements in a hash
include/my_pthread.h:
  Added my_pthread_mutex_init() and my_pthread_mutex_lock(); These should be used if one wants to disable mutex order checking.
  Changed names of the nonposix mutex_init functions to not conflict with my_phread_mutex_init()
  Added and extended structures for mutex deadlock detection.
  New arguments to sage_mutex_init() and safe_mutex_lock() to allow one to disable mutex order checking.
  Added variable 'safe_mutex_deadlock_detector' to enable/disable deadlock detection for all pthread_mutex_init()
mysys/Makefile.am:
  Added cleaning of test files
  Added test_thr_mutex
mysys/hash.c:
  Added hash_iterate() to iterate over all elements in a hash
  More comments
mysys/my_init.c:
  Added calls to destory all mutex uses by mysys()
  Added waiting for threads to end before calling TERMINATE() to list not freed memory
mysys/my_pthread.c:
  Changed names to free my_pthread_mutex_init() for mutex-lock-order-checking
mysys/my_sleep.c:
  Fixed too long wait if using 1000000L as argument
mysys/my_thr_init.c:
  Mark THR_LOCK_threads and THR_LOCK_malloc to not have mutex deadlock detection.
  (We can't have it enabled for this as these are internal mutex used by the detector  
  Call my_thread_init() early as we need thread specific variables enabled for the following pthread_mutex_init()
  Move code to wait for threads to end to my_wait_for_other_threads_to_die()
  Don't destroy mutex and conditions unless all threads have died
  Added my_thread_destroy_mutex() to destroy all mutex used by the mysys thread system
  Name the thread specific mutex as "mysys_var->mutex"
  Added my_thread_var_mutex_in_use() to return pointer to mutex in use or 0 if thread variables are not initialized
mysys/mysys_priv.h:
  Added prototypes for functions used internally with mutex-wrong-usage detection
mysys/thr_mutex.c:
  Added runtime detection of mutex used in conflicting order
  See WL#3262 or test_thr_mutex.c for examples
  The base idea is for each mutex have two hashes:
  - mutex->locked_mutex points to all mutex used after this one
  - mutex->used_mutex points to all mutex which has this mutex in it's mutex->locked_mutex
  There is a wrong mutex order if any mutex currently locked before this mutex is in the mutex->locked_mutex hash
sql/event_queue.cc:
  Mark mutex used inconsistently (need to be fixed by server team)
sql/event_scheduler.cc:
  Declare the right order to take the mutex
sql/events.cc:
  Mark mutex used inconsistently (need to be fixed by server team)
sql/ha_ndbcluster_binlog.cc:
  Mark mutex used inconsistently (need to be fixed by server team)
sql/log.cc:
  Mark mutex used inconsistently (need to be fixed by server team)
sql/mysqld.cc:
  Use pthread_mutex_trylock instead of pthread_mutex_unlock() when sending kill signal to thread
  This is needed to avoid wrong mutex order as normally one takes 'current_mutex' before mysys_var->mutex.
  Added call to free sp cache.
  Add destruction of LOCK_server_started and COND_server_started.
  Added register_mutex_order() function to register in which order mutex should be taken
  (to initiailize mutex_deadlock_detector).
  Added option to turn off safe_mutex_deadlock_detector
sql/protocol.cc:
  Fixed wrong argument to DBUG_PRINT (found by valgrind)
sql/rpl_mi.cc:
  Mark mutex used inconsistently (need to be fixed by server team)
sql/set_var.cc:
  Remove wrong locking of LOCK_global_system_variables when reading and setting log variables
  (would cause inconsistent mutex order).
  Update global variables outside of logger.unlock() as LOCK_global_system_variables has to be taken before logger locks
  Reviewed by gluh
sql/sp_cache.cc:
  Added function to destroy mutex used by sp cache
sql/sp_cache.h:
  Added function to destroy mutex used by sp cache
sql/sql_class.cc:
  Use pthread_mutex_trylock instead of pthread_mutex_unlock() when sending kill signal to thread
  This is needed to avoid wrong mutex order as normally one takes 'current_mutex' before mysys_var->mutex.
  Register order in which LOCK_delete and mysys_var->mutex is taken
sql/sql_insert.cc:
  Give a name for Delayed_insert::mutex
  Mark mutex used inconsistently (need to be fixed by server team)
  Move closing of tables outside of di->mutex (to avoid wrong mutex order)
sql/sql_show.cc:
  Don't keep LOCK_global_system_variables locked over value->show_type() as this leads to wrong mutex order
storage/innobase/handler/ha_innodb.cc:
  Disable safe_muted_deadlock_detector for innobase intern mutex (to speed up page cache initialization)
storage/maria/ha_maria.cc:
  Added flag to ha_maria::info() to signal if we need to lock table share or not.
  This is needed to avoid locking mutex in wrong order
storage/maria/ha_maria.h:
  Added flag to ha_maria::info() to signal if we need to lock table share or not.
storage/maria/ma_close.c:
  Destroy key_del_lock
  Simplify freeing ftparser_param
storage/maria/ma_key.c:
  Better comment
storage/maria/ma_loghandler.c:
  Mark mutex used inconsistently (need to be fixed by sanja)
storage/maria/ma_state.c:
  More comments
storage/maria/ma_test1.c:
  Ensure that safe_mutex_deadlock_detector is always on (should be, this is just for safety)
storage/maria/ma_test2.c:
  Ensure that safe_mutex_deadlock_detector is always on (should be, this is just for safety)

14 files changed, 167 insertions, 37 deletions

diff --git a/sql/event_queue.cc b/sql/event_queue.cc
index 04d4f858b43..d68dc8ef479 100644
--- a/sql/event_queue.cc
+++ b/sql/event_queue.cc
@@ -94,7 +94,12 @@ Event_queue::Event_queue()
    mutex_queue_data_attempting_lock(FALSE),
    waiting_on_cond(FALSE)
 {
-  pthread_mutex_init(&LOCK_event_queue, MY_MUTEX_INIT_FAST);
+  /*
+    Inconsisent usage between LOCK_event_queue and LOCK_scheduler_state and
+    LOCK_open
+  */
+  my_pthread_mutex_init(&LOCK_event_queue, MY_MUTEX_INIT_FAST,
+                        "LOCK_event_queue", MYF_NO_DEADLOCK_DETECTION);
   pthread_cond_init(&COND_queue_state, NULL);
 }
 
diff --git a/sql/event_scheduler.cc b/sql/event_scheduler.cc
index 5655a8acc99..d3e51d40822 100644
--- a/sql/event_scheduler.cc
+++ b/sql/event_scheduler.cc
@@ -350,6 +350,14 @@ Event_scheduler::Event_scheduler(Event_queue *queue_arg)
 {
   pthread_mutex_init(&LOCK_scheduler_state, MY_MUTEX_INIT_FAST);
   pthread_cond_init(&COND_state, NULL);
+
+#ifdef SAFE_MUTEX
+  /* Ensure right mutex order */
+  pthread_mutex_lock(&LOCK_scheduler_state);
+  pthread_mutex_lock(&LOCK_global_system_variables);
+  pthread_mutex_unlock(&LOCK_global_system_variables);
+  pthread_mutex_unlock(&LOCK_scheduler_state);
+#endif
 }
 
 
diff --git a/sql/events.cc b/sql/events.cc
index df49b7db773..73c76688eb7 100644
--- a/sql/events.cc
+++ b/sql/events.cc
@@ -995,7 +995,12 @@ Events::deinit()
 void
 Events::init_mutexes()
 {
-  pthread_mutex_init(&LOCK_event_metadata, MY_MUTEX_INIT_FAST);
+  /*
+    Inconsisent usage between LOCK_event_metadata and LOCK_scheduler_state
+    and LOCK_open
+  */
+  my_pthread_mutex_init(&LOCK_event_metadata, MY_MUTEX_INIT_FAST,
+                        "LOCK_event_metadata", MYF_NO_DEADLOCK_DETECTION);
 }
 
 
diff --git a/sql/ha_ndbcluster_binlog.cc b/sql/ha_ndbcluster_binlog.cc
index 4fd5ee1b402..1788fed26b1 100644
--- a/sql/ha_ndbcluster_binlog.cc
+++ b/sql/ha_ndbcluster_binlog.cc
@@ -1550,7 +1550,9 @@ end:
       dict->forceGCPWait();
 
     int max_timeout= opt_ndb_sync_timeout;
-    (void) pthread_mutex_lock(&ndb_schema_object->mutex);
+    /* Inconsistent usage of ndb_schema_object->mutex and LOCK_open */
+    (void) my_pthread_mutex_lock(&ndb_schema_object->mutex,
+                                 MYF_NO_DEADLOCK_DETECTION);
     if (have_lock_open)
     {
       safe_mutex_assert_owner(&LOCK_open);
diff --git a/sql/log.cc b/sql/log.cc
index b6fe1196835..4c5acf83bc4 100644
--- a/sql/log.cc
+++ b/sql/log.cc
@@ -2403,7 +2403,12 @@ void MYSQL_BIN_LOG::init_pthread_objects()
   DBUG_ASSERT(inited == 0);
   inited= 1;
   (void) pthread_mutex_init(&LOCK_log, MY_MUTEX_INIT_SLOW);
-  (void) pthread_mutex_init(&LOCK_index, MY_MUTEX_INIT_SLOW);
+  /*
+    LOCK_index and LOCK_log are taken in wrong order
+    Can be seen with 'mysql-test-run ndb.ndb_binlog_basic'
+  */ 
+  (void) my_pthread_mutex_init(&LOCK_index, MY_MUTEX_INIT_SLOW, "LOCK_index",
+                               MYF_NO_DEADLOCK_DETECTION);
   (void) pthread_cond_init(&update_cond, 0);
 }
 
diff --git a/sql/mysqld.cc b/sql/mysqld.cc
index b1ec3eb1e3a..8415342b3df 100644
--- a/sql/mysqld.cc
+++ b/sql/mysqld.cc
@@ -757,6 +757,7 @@ static ulong find_bit_type_or_exit(const char *x, TYPELIB *bit_lib,
                                    const char *option);
 static void clean_up(bool print_message);
 static int test_if_case_insensitive(const char *dir_name);
+static void register_mutex_order();
 
 #ifndef EMBEDDED_LIBRARY
 static void usage(void);
@@ -902,9 +903,19 @@ static void close_connections(void)
       pthread_mutex_lock(&tmp->mysys_var->mutex);
       if (tmp->mysys_var->current_cond)
       {
-	pthread_mutex_lock(tmp->mysys_var->current_mutex);
-	pthread_cond_broadcast(tmp->mysys_var->current_cond);
-	pthread_mutex_unlock(tmp->mysys_var->current_mutex);
+        uint i;
+        for (i=0; i < 2; i++)
+        {
+          int ret= pthread_mutex_trylock(tmp->mysys_var->current_mutex);
+          pthread_cond_broadcast(tmp->mysys_var->current_cond);
+          if (!ret)
+          {
+            /* Thread has surely got the signal, unlock and abort */
+            pthread_mutex_unlock(tmp->mysys_var->current_mutex);
+            break;
+          }
+          sleep(1);
+        }
       }
       pthread_mutex_unlock(&tmp->mysys_var->mutex);
     }
@@ -1244,6 +1255,7 @@ void clean_up(bool print_message)
   wt_end();
   delete_elements(&key_caches, (void (*)(const char*, uchar*)) free_key_cache);
   multi_keycache_free();
+  sp_cache_end();
   free_status_vars();
   end_thr_alarm(1);			/* Free allocated memory */
   my_free_open_file_info();
@@ -1336,6 +1348,7 @@ static void wait_for_signal_thread_to_end()
 
 static void clean_up_mutexes()
 {
+  DBUG_ENTER("clean_up_mutexes");
   (void) pthread_mutex_destroy(&LOCK_mysql_create_db);
   (void) pthread_mutex_destroy(&LOCK_lock_db);
   (void) pthread_mutex_destroy(&LOCK_Acl);
@@ -1367,6 +1380,8 @@ static void clean_up_mutexes()
   (void) pthread_mutex_destroy(&LOCK_rpl_status);
   (void) pthread_cond_destroy(&COND_rpl_status);
 #endif
+  (void) pthread_mutex_destroy(&LOCK_server_started);
+  (void) pthread_cond_destroy(&COND_server_started);
   (void) pthread_mutex_destroy(&LOCK_active_mi);
   (void) rwlock_destroy(&LOCK_sys_init_connect);
   (void) rwlock_destroy(&LOCK_sys_init_slave);
@@ -1381,11 +1396,35 @@ static void clean_up_mutexes()
   (void) pthread_cond_destroy(&COND_thread_cache);
   (void) pthread_cond_destroy(&COND_flush_thread_cache);
   (void) pthread_cond_destroy(&COND_manager);
+  DBUG_VOID_RETURN;
 }
 
 #endif /*EMBEDDED_LIBRARY*/
 
 
+/**
+   Register order of mutex for wrong mutex deadlock detector
+
+   By aquiring all mutex in order here, the mutex order detector in
+   mysys/thr_mutex.c, will give a warning on first wrong mutex usage!
+*/
+
+static void register_mutex_order()
+{
+#ifdef SAFE_MUTEX
+  /*
+    We must have LOCK_open before LOCK_global_system_variables because
+    LOCK_open is hold while sql_plugin.c::intern_sys_var_ptr() is called.
+  */
+  pthread_mutex_lock(&LOCK_open);
+  pthread_mutex_lock(&LOCK_global_system_variables);
+
+  pthread_mutex_unlock(&LOCK_global_system_variables);
+  pthread_mutex_unlock(&LOCK_open);
+#endif
+}
+
+
 /****************************************************************************
 ** Init IP and UNIX socket
 ****************************************************************************/
@@ -3549,6 +3588,7 @@ static int init_thread_environment()
     sql_print_error("Can't create thread-keys");
     return 1;
   }
+  register_mutex_order();
   return 0;
 }
 
@@ -5464,7 +5504,7 @@ enum options_mysqld
   OPT_NDB_REPORT_THRESH_BINLOG_EPOCH_SLIP,
   OPT_NDB_REPORT_THRESH_BINLOG_MEM_USAGE,
   OPT_NDB_USE_COPYING_ALTER_TABLE,
-  OPT_SKIP_SAFEMALLOC,
+  OPT_SKIP_SAFEMALLOC, OPT_MUTEX_DEADLOCK_DETECTOR,
   OPT_TEMP_POOL, OPT_TX_ISOLATION, OPT_COMPLETION_TYPE,
   OPT_SKIP_STACK_TRACE, OPT_SKIP_SYMLINKS,
   OPT_MAX_BINLOG_DUMP_EVENTS, OPT_SPORADIC_BINLOG_DUMP_FAIL,
@@ -6002,6 +6042,13 @@ master-ssl",
 #endif /* HAVE_REPLICATION */
   {"memlock", OPT_MEMLOCK, "Lock mysqld in memory.", (uchar**) &locked_in_memory,
    (uchar**) &locked_in_memory, 0, GET_BOOL, NO_ARG, 0, 0, 0, 0, 0, 0},
+#ifdef SAFE_MUTEX
+  {"mutex-deadlock-detector", OPT_MUTEX_DEADLOCK_DETECTOR,
+   "Enable checking of wrong mutex usage.",
+   (uchar**) &safe_mutex_deadlock_detector,
+   (uchar**) &safe_mutex_deadlock_detector,
+   0, GET_BOOL, NO_ARG, 1, 0, 0, 0, 0, 0},
+#endif
   {"myisam-recover", OPT_MYISAM_RECOVER,
    "Syntax: myisam-recover[=option[,option...]], where option can be DEFAULT, BACKUP, FORCE or QUICK.",
    (uchar**) &myisam_recover_options_str, (uchar**) &myisam_recover_options_str, 0,
diff --git a/sql/protocol.cc b/sql/protocol.cc
index 3eccc6632ce..bb69cdb36af 100644
--- a/sql/protocol.cc
+++ b/sql/protocol.cc
@@ -790,8 +790,8 @@ bool Protocol_text::store(const char *from, size_t length,
 {
   CHARSET_INFO *tocs= this->thd->variables.character_set_results;
 #ifndef DBUG_OFF
-  DBUG_PRINT("info", ("Protocol_text::store field %u (%u): %s", field_pos,
-                      field_count, from));
+  DBUG_PRINT("info", ("Protocol_text::store field %u (%u): %*s", field_pos,
+                      field_count, (int) length, from));
   DBUG_ASSERT(field_pos < field_count);
   DBUG_ASSERT(field_types == 0 ||
 	      field_types[field_pos] == MYSQL_TYPE_DECIMAL ||
diff --git a/sql/rpl_mi.cc b/sql/rpl_mi.cc
index 5e46837e948..cb8b0e02ef9 100644
--- a/sql/rpl_mi.cc
+++ b/sql/rpl_mi.cc
@@ -38,11 +38,26 @@ Master_info::Master_info()
   ssl_cipher[0]= 0; ssl_key[0]= 0;
 
   bzero((char*) &file, sizeof(file));
-  pthread_mutex_init(&run_lock, MY_MUTEX_INIT_FAST);
-  pthread_mutex_init(&data_lock, MY_MUTEX_INIT_FAST);
+  /*
+    We have to use MYF_NO_DEADLOCK_DETECTION because mysqld doesn't
+    lock run_lock and data_lock consistently.
+    Should be fixed as this can easily lead to deadlocks
+  */
+  my_pthread_mutex_init(&run_lock, MY_MUTEX_INIT_FAST, 
+                        "Master_info::run_lock", MYF_NO_DEADLOCK_DETECTION);
+  my_pthread_mutex_init(&data_lock, MY_MUTEX_INIT_FAST,
+                        "Master_info::data_lock", MYF_NO_DEADLOCK_DETECTION);
   pthread_cond_init(&data_cond, NULL);
   pthread_cond_init(&start_cond, NULL);
   pthread_cond_init(&stop_cond, NULL);
+
+#ifdef SAFE_MUTEX
+  /* Define mutex order for locks to find wrong lock usage */
+  pthread_mutex_lock(&data_lock);
+  pthread_mutex_lock(&run_lock);
+  pthread_mutex_unlock(&run_lock);
+  pthread_mutex_unlock(&data_lock);
+#endif
 }
 
 Master_info::~Master_info()
diff --git a/sql/set_var.cc b/sql/set_var.cc
index df1badebc50..074835107d9 100644
--- a/sql/set_var.cc
+++ b/sql/set_var.cc
@@ -2403,7 +2403,6 @@ end:
 bool sys_var_log_state::update(THD *thd, set_var *var)
 {
   bool res;
-  pthread_mutex_lock(&LOCK_global_system_variables);
   if (!var->save_result.ulong_value)
   {
     logger.deactivate_log_handler(thd, log_type);
@@ -2411,15 +2410,12 @@ bool sys_var_log_state::update(THD *thd, set_var *var)
   }
   else
     res= logger.activate_log_handler(thd, log_type);
-  pthread_mutex_unlock(&LOCK_global_system_variables);
   return res;
 }
 
 void sys_var_log_state::set_default(THD *thd, enum_var_type type)
 {
-  pthread_mutex_lock(&LOCK_global_system_variables);
   logger.deactivate_log_handler(thd, log_type);
-  pthread_mutex_unlock(&LOCK_global_system_variables);
 }
 
 
@@ -2515,23 +2511,18 @@ bool update_sys_var_str_path(THD *thd, sys_var_str *var_str,
     goto err;
   }
 
-  pthread_mutex_lock(&LOCK_global_system_variables);
   logger.lock_exclusive();
 
   if (file_log && log_state)
     file_log->close(0);
-  old_value= var_str->value;
-  var_str->value= res;
-  var_str->value_length= str_length;
-  my_free(old_value, MYF(MY_ALLOW_ZERO_PTR));
   if (file_log && log_state)
   {
     switch (log_type) {
     case QUERY_LOG_SLOW:
-      file_log->open_slow_log(sys_var_slow_log_path.value);
+      file_log->open_slow_log(res);
       break;
     case QUERY_LOG_GENERAL:
-      file_log->open_query_log(sys_var_general_log_path.value);
+      file_log->open_query_log(res);
       break;
     default:
       DBUG_ASSERT(0);
@@ -2539,6 +2530,13 @@ bool update_sys_var_str_path(THD *thd, sys_var_str *var_str,
   }
 
   logger.unlock();
+
+  /* update global variable */
+  pthread_mutex_lock(&LOCK_global_system_variables);
+  old_value= var_str->value;
+  var_str->value= res;
+  var_str->value_length= str_length;
+  my_free(old_value, MYF(MY_ALLOW_ZERO_PTR));
   pthread_mutex_unlock(&LOCK_global_system_variables);
 
 err:
@@ -2578,26 +2576,22 @@ static void sys_default_slow_log_path(THD *thd, enum_var_type type)
 
 bool sys_var_log_output::update(THD *thd, set_var *var)
 {
-  pthread_mutex_lock(&LOCK_global_system_variables);
   logger.lock_exclusive();
   logger.init_slow_log(var->save_result.ulong_value);
   logger.init_general_log(var->save_result.ulong_value);
   *value= var->save_result.ulong_value;
   logger.unlock();
-  pthread_mutex_unlock(&LOCK_global_system_variables);
   return 0;
 }
 
 
 void sys_var_log_output::set_default(THD *thd, enum_var_type type)
 {
-  pthread_mutex_lock(&LOCK_global_system_variables);
   logger.lock_exclusive();
   logger.init_slow_log(LOG_FILE);
   logger.init_general_log(LOG_FILE);
   *value= LOG_FILE;
   logger.unlock();
-  pthread_mutex_unlock(&LOCK_global_system_variables);
 }
 
 
diff --git a/sql/sp_cache.cc b/sql/sp_cache.cc
index 64898915b7e..13b3e771a91 100644
--- a/sql/sp_cache.cc
+++ b/sql/sp_cache.cc
@@ -106,6 +106,12 @@ void sp_cache_clear(sp_cache **cp)
 }
 
 
+void sp_cache_end()
+{
+  pthread_mutex_destroy(&Cversion_lock);
+}
+
+
 /*
   Insert a routine into the cache.
 
diff --git a/sql/sp_cache.h b/sql/sp_cache.h
index f4d44a1f29f..efb61d76719 100644
--- a/sql/sp_cache.h
+++ b/sql/sp_cache.h
@@ -53,6 +53,7 @@ class sp_cache;
 */
 
 void sp_cache_init();
+void sp_cache_end();
 void sp_cache_clear(sp_cache **cp);
 void sp_cache_insert(sp_cache **cp, sp_head *sp);
 sp_head *sp_cache_lookup(sp_cache **cp, sp_name *name);
diff --git a/sql/sql_class.cc b/sql/sql_class.cc
index 9a450f245be..3504b4fdea5 100644
--- a/sql/sql_class.cc
+++ b/sql/sql_class.cc
@@ -978,6 +978,14 @@ void add_diff_to_status(STATUS_VAR *to_var, STATUS_VAR *from_var,
     *(to++)+= *(from++) - *(dec++);
 }
 
+#define SECONDS_TO_WAIT_FOR_KILL 2
+#if !defined(__WIN__) && defined(HAVE_SELECT)
+/* my_sleep() can wait for sub second times */
+#define WAIT_FOR_KILL_TRY_TIMES 20
+#else
+#define WAIT_FOR_KILL_TRY_TIMES 2
+#endif
+
 
 void THD::awake(THD::killed_state state_to_set)
 {
@@ -1032,12 +1040,35 @@ void THD::awake(THD::killed_state state_to_set)
       we issue a second KILL or the status it's waiting for happens).
       It's true that we have set its thd->killed but it may not
       see it immediately and so may have time to reach the cond_wait().
+
+      We have to do the loop with trylock, because if we would use
+      pthread_mutex_lock(), we can cause a deadlock as we are here locking
+      the mysys_var->mutex and mysys_var->current_mutex in a different order
+      than in the thread we are trying to kill.
+      We only sleep for 2 seconds as we don't want to have LOCK_delete
+      locked too long time.
+
+      There is a small change we may not succeed in aborting a thread that
+      is not yet waiting for a mutex, but as this happens only for a
+      thread that was doing something else when the kill was issued and
+      which should detect the kill flag before it starts to wait, this
+      should be good enough.
     */
     if (mysys_var->current_cond && mysys_var->current_mutex)
     {
-      pthread_mutex_lock(mysys_var->current_mutex);
-      pthread_cond_broadcast(mysys_var->current_cond);
-      pthread_mutex_unlock(mysys_var->current_mutex);
+      uint i;
+      for (i= 0; i < WAIT_FOR_KILL_TRY_TIMES * SECONDS_TO_WAIT_FOR_KILL; i++)
+      {
+        int ret= pthread_mutex_trylock(mysys_var->current_mutex);
+        pthread_cond_broadcast(mysys_var->current_cond);
+        if (!ret)
+        {
+          /* Signal is sure to get through */
+          pthread_mutex_unlock(mysys_var->current_mutex);
+          break;
+        }
+      }
+      my_sleep(1000000L / WAIT_FOR_KILL_TRY_TIMES);
     }
     pthread_mutex_unlock(&mysys_var->mutex);
   }
@@ -1073,6 +1104,15 @@ bool THD::store_globals()
     created in another thread
   */
   thr_lock_info_init(&lock_info);
+
+#ifdef SAFE_MUTEX
+  /* Register order of mutex for wrong mutex deadlock detector */
+  pthread_mutex_lock(&LOCK_delete);
+  pthread_mutex_lock(&mysys_var->mutex);
+
+  pthread_mutex_unlock(&mysys_var->mutex);
+  pthread_mutex_unlock(&LOCK_delete);
+#endif
   return 0;
 }
 
diff --git a/sql/sql_insert.cc b/sql/sql_insert.cc
index a275c680cb5..d8838b3b03d 100644
--- a/sql/sql_insert.cc
+++ b/sql/sql_insert.cc
@@ -1719,7 +1719,8 @@ public:
     thd.system_thread= SYSTEM_THREAD_DELAYED_INSERT;
     thd.security_ctx->host_or_ip= "";
     bzero((char*) &info,sizeof(info));
-    pthread_mutex_init(&mutex,MY_MUTEX_INIT_FAST);
+    my_pthread_mutex_init(&mutex, MY_MUTEX_INIT_FAST, "Delayed_insert::mutex",
+                          0);
     pthread_cond_init(&cond,NULL);
     pthread_cond_init(&cond_client,NULL);
     VOID(pthread_mutex_lock(&LOCK_thread_count));
@@ -2224,7 +2225,8 @@ void kill_delayed_threads(void)
 	  in handle_delayed_insert()
 	*/
 	if (&di->mutex != di->thd.mysys_var->current_mutex)
-	  pthread_mutex_lock(di->thd.mysys_var->current_mutex);
+	  my_pthread_mutex_lock(di->thd.mysys_var->current_mutex, 
+                                MYF_NO_DEADLOCK_DETECTION);
 	pthread_cond_broadcast(di->thd.mysys_var->current_cond);
 	if (&di->mutex != di->thd.mysys_var->current_mutex)
 	  pthread_mutex_unlock(di->thd.mysys_var->current_mutex);
@@ -2470,13 +2472,14 @@ end:
     clients
   */
 
-  close_thread_tables(thd);			// Free the table
   di->table=0;
   di->dead= 1;                                  // If error
   thd->killed= THD::KILL_CONNECTION;	        // If error
-  pthread_cond_broadcast(&di->cond_client);	// Safety
   pthread_mutex_unlock(&di->mutex);
 
+  close_thread_tables(thd);			// Free the table
+  pthread_cond_broadcast(&di->cond_client);	// Safety
+
   pthread_mutex_lock(&LOCK_delayed_create);	// Because of delayed_get_table
   pthread_mutex_lock(&LOCK_delayed_insert);	
   delete di;
diff --git a/sql/sql_show.cc b/sql/sql_show.cc
index b5ab6484d12..6e321af9292 100644
--- a/sql/sql_show.cc
+++ b/sql/sql_show.cc
@@ -2102,7 +2102,6 @@ static bool show_status_array(THD *thd, const char *wild,
         const char *pos, *end;                  // We assign a lot of const's
 
         pthread_mutex_lock(&LOCK_global_system_variables);
-
         if (show_type == SHOW_SYS)
         {
           show_type= ((sys_var*) value)->show_type();
@@ -2183,14 +2182,14 @@ static bool show_status_array(THD *thd, const char *wild,
           DBUG_ASSERT(0);
           break;
         }
+        pthread_mutex_unlock(&LOCK_global_system_variables);
+
         restore_record(table, s->default_values);
         table->field[0]->store(name_buffer, strlen(name_buffer),
                                system_charset_info);
         table->field[1]->store(pos, (uint32) (end - pos), system_charset_info);
         table->field[1]->set_notnull();
 
-        pthread_mutex_unlock(&LOCK_global_system_variables);
-
         if (schema_table_store_record(thd, table))
           DBUG_RETURN(TRUE);
       }