WL#3262 add mutex lock order checking to safemutex (also called safe_mutex_deadlock_detector)

This writes a warning on stderr if one uses mutex in different order,
like if one in one case would lock mutex in the order A,B and in another case
would lock mutex in the order B,A
This is inspired by and loosely based on the LOCKDEP patch by Jonas
Wrong mutex order is either fixed or mutex are marked with MYF_NO_DEADLOCK_DETECTION
if used inconsistently (need to be fixed by server team)

KNOWN_BUGS.txt:
  Added information that one need to dump and restore Maria tables
include/hash.h:
  Added prototype function for walking over all elements in a hash
include/my_pthread.h:
  Added my_pthread_mutex_init() and my_pthread_mutex_lock(); These should be used if one wants to disable mutex order checking.
  Changed names of the nonposix mutex_init functions to not conflict with my_phread_mutex_init()
  Added and extended structures for mutex deadlock detection.
  New arguments to sage_mutex_init() and safe_mutex_lock() to allow one to disable mutex order checking.
  Added variable 'safe_mutex_deadlock_detector' to enable/disable deadlock detection for all pthread_mutex_init()
mysys/Makefile.am:
  Added cleaning of test files
  Added test_thr_mutex
mysys/hash.c:
  Added hash_iterate() to iterate over all elements in a hash
  More comments
mysys/my_init.c:
  Added calls to destory all mutex uses by mysys()
  Added waiting for threads to end before calling TERMINATE() to list not freed memory
mysys/my_pthread.c:
  Changed names to free my_pthread_mutex_init() for mutex-lock-order-checking
mysys/my_sleep.c:
  Fixed too long wait if using 1000000L as argument
mysys/my_thr_init.c:
  Mark THR_LOCK_threads and THR_LOCK_malloc to not have mutex deadlock detection.
  (We can't have it enabled for this as these are internal mutex used by the detector  
  Call my_thread_init() early as we need thread specific variables enabled for the following pthread_mutex_init()
  Move code to wait for threads to end to my_wait_for_other_threads_to_die()
  Don't destroy mutex and conditions unless all threads have died
  Added my_thread_destroy_mutex() to destroy all mutex used by the mysys thread system
  Name the thread specific mutex as "mysys_var->mutex"
  Added my_thread_var_mutex_in_use() to return pointer to mutex in use or 0 if thread variables are not initialized
mysys/mysys_priv.h:
  Added prototypes for functions used internally with mutex-wrong-usage detection
mysys/thr_mutex.c:
  Added runtime detection of mutex used in conflicting order
  See WL#3262 or test_thr_mutex.c for examples
  The base idea is for each mutex have two hashes:
  - mutex->locked_mutex points to all mutex used after this one
  - mutex->used_mutex points to all mutex which has this mutex in it's mutex->locked_mutex
  There is a wrong mutex order if any mutex currently locked before this mutex is in the mutex->locked_mutex hash
sql/event_queue.cc:
  Mark mutex used inconsistently (need to be fixed by server team)
sql/event_scheduler.cc:
  Declare the right order to take the mutex
sql/events.cc:
  Mark mutex used inconsistently (need to be fixed by server team)
sql/ha_ndbcluster_binlog.cc:
  Mark mutex used inconsistently (need to be fixed by server team)
sql/log.cc:
  Mark mutex used inconsistently (need to be fixed by server team)
sql/mysqld.cc:
  Use pthread_mutex_trylock instead of pthread_mutex_unlock() when sending kill signal to thread
  This is needed to avoid wrong mutex order as normally one takes 'current_mutex' before mysys_var->mutex.
  Added call to free sp cache.
  Add destruction of LOCK_server_started and COND_server_started.
  Added register_mutex_order() function to register in which order mutex should be taken
  (to initiailize mutex_deadlock_detector).
  Added option to turn off safe_mutex_deadlock_detector
sql/protocol.cc:
  Fixed wrong argument to DBUG_PRINT (found by valgrind)
sql/rpl_mi.cc:
  Mark mutex used inconsistently (need to be fixed by server team)
sql/set_var.cc:
  Remove wrong locking of LOCK_global_system_variables when reading and setting log variables
  (would cause inconsistent mutex order).
  Update global variables outside of logger.unlock() as LOCK_global_system_variables has to be taken before logger locks
  Reviewed by gluh
sql/sp_cache.cc:
  Added function to destroy mutex used by sp cache
sql/sp_cache.h:
  Added function to destroy mutex used by sp cache
sql/sql_class.cc:
  Use pthread_mutex_trylock instead of pthread_mutex_unlock() when sending kill signal to thread
  This is needed to avoid wrong mutex order as normally one takes 'current_mutex' before mysys_var->mutex.
  Register order in which LOCK_delete and mysys_var->mutex is taken
sql/sql_insert.cc:
  Give a name for Delayed_insert::mutex
  Mark mutex used inconsistently (need to be fixed by server team)
  Move closing of tables outside of di->mutex (to avoid wrong mutex order)
sql/sql_show.cc:
  Don't keep LOCK_global_system_variables locked over value->show_type() as this leads to wrong mutex order
storage/innobase/handler/ha_innodb.cc:
  Disable safe_muted_deadlock_detector for innobase intern mutex (to speed up page cache initialization)
storage/maria/ha_maria.cc:
  Added flag to ha_maria::info() to signal if we need to lock table share or not.
  This is needed to avoid locking mutex in wrong order
storage/maria/ha_maria.h:
  Added flag to ha_maria::info() to signal if we need to lock table share or not.
storage/maria/ma_close.c:
  Destroy key_del_lock
  Simplify freeing ftparser_param
storage/maria/ma_key.c:
  Better comment
storage/maria/ma_loghandler.c:
  Mark mutex used inconsistently (need to be fixed by sanja)
storage/maria/ma_state.c:
  More comments
storage/maria/ma_test1.c:
  Ensure that safe_mutex_deadlock_detector is always on (should be, this is just for safety)
storage/maria/ma_test2.c:
  Ensure that safe_mutex_deadlock_detector is always on (should be, this is just for safety)

1 files changed, 387 insertions, 39 deletions

diff --git a/mysys/thr_mutex.c b/mysys/thr_mutex.c
index aa46021a938..ddbe613cdae 100644
--- a/mysys/thr_mutex.c
+++ b/mysys/thr_mutex.c
@@ -1,4 +1,4 @@
-/* Copyright (C) 2000-2003 MySQL AB
+/* Copyright (C) 2000-2008 MySQL AB, 2008 Sun Microsystems, Inc
 
    This program is free software; you can redistribute it and/or modify
    it under the terms of the GNU General Public License as published by
@@ -24,6 +24,7 @@
 #include "mysys_priv.h"
 #include "my_static.h"
 #include <m_string.h>
+#include <hash.h>
 
 #ifndef DO_NOT_REMOVE_THREAD_WRAPPERS
 /* Remove wrappers */
@@ -34,28 +35,68 @@
 #undef pthread_mutex_destroy
 #undef pthread_cond_wait
 #undef pthread_cond_timedwait
+#undef safe_mutex_free_deadlock_data
 #ifdef HAVE_NONPOSIX_PTHREAD_MUTEX_INIT
-#define pthread_mutex_init(a,b) my_pthread_mutex_init((a),(b))
+#define pthread_mutex_init(a,b) my_pthread_noposix_mutex_init((a),(b))
 #endif
 #endif /* DO_NOT_REMOVE_THREAD_WRAPPERS */
 
 static pthread_mutex_t THR_LOCK_mutex;
 static ulong safe_mutex_count= 0;		/* Number of mutexes created */
+static ulong safe_mutex_id= 0;
+my_bool safe_mutex_deadlock_detector= 1;        /* On by default */
+
 #ifdef SAFE_MUTEX_DETECT_DESTROY
-static struct st_safe_mutex_info_t *safe_mutex_root= NULL;
+static struct st_safe_mutex_create_info_t *safe_mutex_create_root= NULL;
 #endif
 
+static my_bool add_used_to_locked_mutex(safe_mutex_t *used_mutex,
+                                        safe_mutex_deadlock_t *locked_mutex);
+static my_bool add_to_locked_mutex(safe_mutex_deadlock_t *locked_mutex,
+                                   safe_mutex_t *current_mutex);
+static my_bool remove_from_locked_mutex(safe_mutex_t *mp,
+                                        safe_mutex_t *delete_mutex);
+static my_bool remove_from_used_mutex(safe_mutex_deadlock_t *locked_mutex,
+                                      safe_mutex_t *mutex);
+static void print_deadlock_warning(safe_mutex_t *new_mutex,
+                                   safe_mutex_t *conflicting_mutex);
+
 void safe_mutex_global_init(void)
 {
   pthread_mutex_init(&THR_LOCK_mutex,MY_MUTEX_INIT_FAST);
+  safe_mutex_id= safe_mutex_count= 0;
+  safe_mutex_deadlock_detector= 1;
+
+#ifdef SAFE_MUTEX_DETECT_DESTROY
+  safe_mutex_create_root= 0;
+#endif
+}
+
+static inline void remove_from_active_list(safe_mutex_t *mp)
+{
+  if (!(mp->active_flags & (MYF_NO_DEADLOCK_DETECTION | MYF_TRY_LOCK)))
+  {
+    /* Remove mutex from active mutex linked list */
+    if (mp->next)
+      mp->next->prev= mp->prev;
+    if (mp->prev)
+      mp->prev->next= mp->next;
+    else
+      *my_thread_var_mutex_in_use()= mp->next;
+  }
+  mp->prev= mp->next= 0;
 }
 
 
 int safe_mutex_init(safe_mutex_t *mp,
 		    const pthread_mutexattr_t *attr __attribute__((unused)),
+                    const char *name,
+                    myf my_flags,
 		    const char *file,
-		    uint line, const char *name)
+		    uint line)
 {
+  DBUG_ENTER("safe_mutex_init");
+  DBUG_PRINT("enter",("mutex: 0x%lx  name: %s", (ulong) mp, name));
   bzero((char*) mp,sizeof(*mp));
   pthread_mutex_init(&mp->global,MY_MUTEX_INIT_ERRCHK);
   pthread_mutex_init(&mp->mutex,attr);
@@ -65,6 +106,36 @@ int safe_mutex_init(safe_mutex_t *mp,
   /* Skip the very common '&' prefix from the autogenerated name */
   mp->name= name[0] == '&' ? name + 1 : name;
 
+  if (safe_mutex_deadlock_detector && !( my_flags & MYF_NO_DEADLOCK_DETECTION))
+  {
+    if (!my_multi_malloc(MY_FAE | MY_WME,
+                         &mp->locked_mutex, sizeof(*mp->locked_mutex),
+                         &mp->used_mutex, sizeof(*mp->used_mutex), NullS))
+    {
+      /* Disable deadlock handling for this mutex */
+      my_flags|= MYF_NO_DEADLOCK_DETECTION;
+    }
+    else
+    {
+      pthread_mutex_lock(&THR_LOCK_mutex);
+      mp->id= ++safe_mutex_id;
+      pthread_mutex_unlock(&THR_LOCK_mutex);
+      hash_init(mp->locked_mutex, &my_charset_bin,
+                1000,
+                offsetof(safe_mutex_deadlock_t, id),
+                sizeof(mp->id),
+                0, 0, HASH_UNIQUE);
+      hash_init(mp->used_mutex, &my_charset_bin,
+                1000,
+                offsetof(safe_mutex_t, id),
+                sizeof(mp->id),
+                0, 0, HASH_UNIQUE);
+    }
+  }
+  else
+    my_flags|= MYF_NO_DEADLOCK_DETECTION;
+  mp->create_flags= my_flags;
+
 #ifdef SAFE_MUTEX_DETECT_DESTROY
   /*
     Monitor the freeing of mutexes.  This code depends on single thread init
@@ -72,7 +143,7 @@ int safe_mutex_init(safe_mutex_t *mp,
   */
   if ((mp->info= (safe_mutex_info_t *) malloc(sizeof(safe_mutex_info_t))))
   {
-    struct st_safe_mutex_info_t *info =mp->info;
+    struct st_safe_mutex_info_t *info= mp->info;
 
     info->init_file= file;
     info->init_line= line;
@@ -80,20 +151,21 @@ int safe_mutex_init(safe_mutex_t *mp,
     info->next= NULL;
 
     pthread_mutex_lock(&THR_LOCK_mutex);
-    if ((info->next= safe_mutex_root))
-      safe_mutex_root->prev= info;
-    safe_mutex_root= info;
+    if ((info->next= safe_mutex_create_root))
+      safe_mutex_create_root->prev= info;
+    safe_mutex_create_root= info;
     safe_mutex_count++;
     pthread_mutex_unlock(&THR_LOCK_mutex);
   }
 #else
   thread_safe_increment(safe_mutex_count, &THR_LOCK_mutex);
 #endif /* SAFE_MUTEX_DETECT_DESTROY */
-  return 0;
+  DBUG_RETURN(0);
 }
 
 
-int safe_mutex_lock(safe_mutex_t *mp, my_bool try_lock, const char *file, uint line)
+int safe_mutex_lock(safe_mutex_t *mp, myf my_flags, const char *file,
+                    uint line)
 {
   int error;
   DBUG_PRINT("mutex", ("%s (0x%lx) locking", mp->name ? mp->name : "Null",
@@ -110,12 +182,13 @@ int safe_mutex_lock(safe_mutex_t *mp, my_bool try_lock, const char *file, uint l
   pthread_mutex_lock(&mp->global);
   if (mp->count > 0)
   {
-    if (try_lock)
-    {
-      pthread_mutex_unlock(&mp->global);
-      return EBUSY;
-    }
-    else if (pthread_equal(pthread_self(),mp->thread))
+    /*
+      Check that we are not trying to lock mutex twice. This is an error
+      even if we are using 'try_lock' as it's not portably what happens
+      if you lock the mutex many times and this is in any case bad
+      behaviour that should not be encouraged
+    */
+    if (pthread_equal(pthread_self(),mp->thread))
     {
       fprintf(stderr,
               "safe_mutex: Trying to lock mutex at %s, line %d, when the"
@@ -143,7 +216,7 @@ int safe_mutex_lock(safe_mutex_t *mp, my_bool try_lock, const char *file, uint l
       instead just return EBUSY, since this is the expected behaviour
       of trylock().
    */
-  if (try_lock)
+  if (my_flags & MYF_TRY_LOCK)
   {
     error= pthread_mutex_trylock(&mp->mutex);
     if (error == EBUSY)
@@ -169,7 +242,93 @@ int safe_mutex_lock(safe_mutex_t *mp, my_bool try_lock, const char *file, uint l
   }
   mp->file= file;
   mp->line= line;
+  mp->active_flags= mp->create_flags | my_flags;
   pthread_mutex_unlock(&mp->global);
+
+  /* Deadlock detection */
+
+  mp->prev= mp->next= 0;
+  if (!(mp->active_flags & (MYF_TRY_LOCK | MYF_NO_DEADLOCK_DETECTION)))
+  {
+    safe_mutex_t **mutex_in_use= my_thread_var_mutex_in_use();
+
+    if (!mutex_in_use)
+    {
+      /* thread has not called my_thread_init() */
+      mp->active_flags|= MYF_NO_DEADLOCK_DETECTION;
+    }
+    else
+    {
+      safe_mutex_t *mutex_root;
+      if ((mutex_root= *mutex_in_use))   /* If not first locked */
+      {
+        /*
+          Protect locked_mutex against changes if a mutex is deleted
+        */
+        pthread_mutex_lock(&THR_LOCK_mutex);
+
+        if (!hash_search(mutex_root->locked_mutex, (uchar*) &mp->id, 0))
+        {
+          safe_mutex_deadlock_t *deadlock;
+          safe_mutex_t *mutex;
+
+          /* Create object to store mutex info */
+          if (!(deadlock= my_malloc(sizeof(*deadlock),
+                                    MYF(MY_ZEROFILL | MY_WME | MY_FAE))))
+            goto abort_loop;
+          deadlock->name= mp->name;
+          deadlock->id= mp->id;
+          deadlock->mutex= mp;
+          /* The following is useful for debugging wrong mutex usage */
+          deadlock->file= file;
+          deadlock->line= line;
+
+          /* Check if potential deadlock */
+          mutex= mutex_root;
+          do
+          {
+            if (hash_search(mp->locked_mutex, (uchar*) &mutex->id, 0))
+            {
+              print_deadlock_warning(mp, mutex);
+              /* Mark wrong usage to avoid future warnings for same error */
+              deadlock->warning_only= 1;
+              add_to_locked_mutex(deadlock, mutex_root);
+              DBUG_ASSERT(deadlock->count > 0);
+              goto abort_loop;
+            }
+          }
+          while ((mutex= mutex->next));
+
+          /*
+            Copy current mutex and all mutex that has been locked
+            after current mutex (mp->locked_mutex) to all mutex that
+            was locked before previous mutex (mutex_root->used_mutex)
+
+            For example if A->B would have been done before and we
+            are now locking (C) in B->C, then we would add C into
+            B->locked_mutex and A->locked_mutex
+          */
+          hash_iterate(mutex_root->used_mutex,
+                       (hash_walk_action) add_used_to_locked_mutex,
+                       deadlock);
+
+          /*
+            Copy all current mutex and all mutex locked after current one
+            into the prev mutex
+          */
+          add_used_to_locked_mutex(mutex_root, deadlock);
+          DBUG_ASSERT(deadlock->count > 0);
+        }
+  abort_loop:
+        pthread_mutex_unlock(&THR_LOCK_mutex);
+      }
+      /* Link mutex into mutex_in_use list */
+      if ((mp->next= *mutex_in_use))
+        (*mutex_in_use)->prev= mp;
+      *mutex_in_use= mp;
+    }
+  }
+
   DBUG_PRINT("mutex", ("%s (0x%lx) locked", mp->name, (ulong) mp));
   return error;
 }
@@ -182,7 +341,9 @@ int safe_mutex_unlock(safe_mutex_t *mp,const char *file, uint line)
   pthread_mutex_lock(&mp->global);
   if (mp->count == 0)
   {
-    fprintf(stderr,"safe_mutex: Trying to unlock mutex %s that wasn't locked at %s, line %d\n"
+    fprintf(stderr,
+            "safe_mutex: Trying to unlock mutex %s that wasn't locked at "
+            "%s, line %d\n"
             "Last used at %s, line: %d\n",
 	    mp->name ? mp->name : "Null", file, line,
             mp->file ? mp->file : "Null", mp->line);
@@ -191,7 +352,9 @@ int safe_mutex_unlock(safe_mutex_t *mp,const char *file, uint line)
   }
   if (!pthread_equal(pthread_self(),mp->thread))
   {
-    fprintf(stderr,"safe_mutex: Trying to unlock mutex %s at %s, line %d that was locked by "
+    fprintf(stderr,
+            "safe_mutex: Trying to unlock mutex %s at %s, line %d that was "
+            "locked by "
             "another thread at: %s, line: %d\n",
 	    mp->name, file, line, mp->file, mp->line);
     fflush(stderr);
@@ -199,6 +362,9 @@ int safe_mutex_unlock(safe_mutex_t *mp,const char *file, uint line)
   }
   mp->thread= 0;
   mp->count--;
+
+  remove_from_active_list(mp);
+
 #ifdef __WIN__
   pthread_mutex_unlock(&mp->mutex);
   error=0;
@@ -206,8 +372,9 @@ int safe_mutex_unlock(safe_mutex_t *mp,const char *file, uint line)
   error=pthread_mutex_unlock(&mp->mutex);
   if (error)
   {
-    fprintf(stderr,"safe_mutex: Got error: %d (%d) when trying to unlock mutex %s at %s, "
-            "line %d\n", error, errno, mp->name, file, line);
+    fprintf(stderr,
+            "safe_mutex: Got error: %d (%d) when trying to unlock mutex "
+            "%s at %s, line %d\n", error, errno, mp->name, file, line);
     fflush(stderr);
     abort();
   }
@@ -221,18 +388,23 @@ int safe_cond_wait(pthread_cond_t *cond, safe_mutex_t *mp, const char *file,
 		   uint line)
 {
   int error;
+  safe_mutex_t save_state;
+
   pthread_mutex_lock(&mp->global);
   if (mp->count == 0)
   {
-    fprintf(stderr,"safe_mutex: Trying to cond_wait on a unlocked mutex %s at %s, line %d\n",
+    fprintf(stderr,
+            "safe_mutex: Trying to cond_wait on a unlocked mutex %s at %s, "
+            "line %d\n",
             mp->name ? mp->name : "Null", file, line);
     fflush(stderr);
     abort();
   }
   if (!pthread_equal(pthread_self(),mp->thread))
   {
-    fprintf(stderr,"safe_mutex: Trying to cond_wait on a mutex %s at %s, line %d that was "
-            "locked by another thread at: %s, line: %d\n",
+    fprintf(stderr,
+            "safe_mutex: Trying to cond_wait on a mutex %s at %s, line %d "
+            "that was locked by another thread at: %s, line: %d\n",
 	    mp->name, file, line, mp->file, mp->line);
     fflush(stderr);
     abort();
@@ -240,26 +412,37 @@ int safe_cond_wait(pthread_cond_t *cond, safe_mutex_t *mp, const char *file,
 
   if (mp->count-- != 1)
   {
-    fprintf(stderr,"safe_mutex:  Count was %d on locked mutex %s at %s, line %d\n",
+    fprintf(stderr,
+            "safe_mutex:  Count was %d on locked mutex %s at %s, line %d\n",
 	    mp->count+1, mp->name, file, line);
     fflush(stderr);
     abort();
   }
+  save_state= *mp;
+  remove_from_active_list(mp);
   pthread_mutex_unlock(&mp->global);
   error=pthread_cond_wait(cond,&mp->mutex);
   pthread_mutex_lock(&mp->global);
+
   if (error)
   {
-    fprintf(stderr,"safe_mutex: Got error: %d (%d) when doing a safe_mutex_wait on %s at %s, "
-            "line %d\n", error, errno, mp->name, file, line);
+    fprintf(stderr,
+            "safe_mutex: Got error: %d (%d) when doing a safe_mutex_wait on "
+            "%s at %s, line %d\n", error, errno, mp->name, file, line);
     fflush(stderr);
     abort();
   }
-  mp->thread=pthread_self();
+  /* Restore state as it was before */
+  mp->thread=       save_state.thread;
+  mp->active_flags= save_state.active_flags;
+  mp->next=         save_state.next;
+  mp->prev=         save_state.prev;
+
   if (mp->count++)
   {
     fprintf(stderr,
-	    "safe_mutex:  Count was %d in thread 0x%lx when locking mutex %s at %s, line %d\n",
+	    "safe_mutex:  Count was %d in thread 0x%lx when locking mutex %s "
+            "at %s, line %d\n",
 	    mp->count-1, my_thread_dbug_id(), mp->name, file, line);
     fflush(stderr);
     abort();
@@ -276,33 +459,44 @@ int safe_cond_timedwait(pthread_cond_t *cond, safe_mutex_t *mp,
 			const char *file, uint line)
 {
   int error;
+  safe_mutex_t save_state;
+
   pthread_mutex_lock(&mp->global);
   if (mp->count != 1 || !pthread_equal(pthread_self(),mp->thread))
   {
-    fprintf(stderr,"safe_mutex: Trying to cond_wait at %s, line %d on a not hold mutex %s\n",
+    fprintf(stderr,
+            "safe_mutex: Trying to cond_wait at %s, line %d on a not hold "
+            "mutex %s\n",
             file, line, mp->name ? mp->name : "Null");
     fflush(stderr);
     abort();
   }
   mp->count--;					/* Mutex will be released */
+  save_state= *mp;
+  remove_from_active_list(mp);
   pthread_mutex_unlock(&mp->global);
   error=pthread_cond_timedwait(cond,&mp->mutex,abstime);
 #ifdef EXTRA_DEBUG
   if (error && (error != EINTR && error != ETIMEDOUT && error != ETIME))
   {
     fprintf(stderr,
-            "safe_mutex: Got error: %d (%d) when doing a safe_mutex_timedwait on %s at %s, "
-            "line %d\n",
+            "safe_mutex: Got error: %d (%d) when doing a safe_mutex_timedwait "
+            "on %s at %s, line %d\n",
             error, errno, mp->name, file, line);
   }
 #endif
   pthread_mutex_lock(&mp->global);
-  mp->thread=pthread_self();
+  /* Restore state as it was before */
+  mp->thread=       save_state.thread;
+  mp->active_flags= save_state.active_flags;
+  mp->next=         save_state.next;
+  mp->prev=         save_state.prev;
+
   if (mp->count++)
   {
     fprintf(stderr,
-	    "safe_mutex:  Count was %d in thread 0x%lx when locking mutex %s at %s, line %d "
-            "(error: %d (%d))\n",
+	    "safe_mutex:  Count was %d in thread 0x%lx when locking mutex "
+            "%s at %s, line %d (error: %d (%d))\n",
 	    mp->count-1, my_thread_dbug_id(), mp->name, file, line,
             error, error);
     fflush(stderr);
@@ -318,6 +512,8 @@ int safe_cond_timedwait(pthread_cond_t *cond, safe_mutex_t *mp,
 int safe_mutex_destroy(safe_mutex_t *mp, const char *file, uint line)
 {
   int error=0;
+  DBUG_ENTER("safe_mutex_destroy");
+  DBUG_PRINT("enter", ("mutex: 0x%lx  name: %s", (ulong) mp, mp->name));
   if (!mp->file)
   {
     fprintf(stderr,
@@ -328,12 +524,17 @@ int safe_mutex_destroy(safe_mutex_t *mp, const char *file, uint line)
   }
   if (mp->count != 0)
   {
-    fprintf(stderr,"safe_mutex: Trying to destroy a mutex %s that was locked at %s, "
+    fprintf(stderr,
+            "safe_mutex: Trying to destroy a mutex %s that was locked at %s, "
             "line %d at %s, line %d\n",
 	    mp->name, mp->file, mp->line, file, line);
     fflush(stderr);
     abort();
   }
+
+  /* Free all entries that points to this one */
+  safe_mutex_free_deadlock_data(mp);
+
 #ifdef __WIN__ 
   pthread_mutex_destroy(&mp->global);
   pthread_mutex_destroy(&mp->mutex);
@@ -354,7 +555,7 @@ int safe_mutex_destroy(safe_mutex_t *mp, const char *file, uint line)
     if (info->prev)
       info->prev->next = info->next;
     else
-      safe_mutex_root = info->next;
+      safe_mutex_create_root = info->next;
     if (info->next)
       info->next->prev = info->prev;
     safe_mutex_count--;
@@ -366,10 +567,36 @@ int safe_mutex_destroy(safe_mutex_t *mp, const char *file, uint line)
 #else
   thread_safe_sub(safe_mutex_count, 1, &THR_LOCK_mutex);
 #endif /* SAFE_MUTEX_DETECT_DESTROY */
-  return error;
+  DBUG_RETURN(error);
 }
 
 
+/**
+  Free all data related to deadlock detection
+
+  This is also useful together with safemalloc when you don't want to
+  have reports of not freed memory for mysys mutexes.
+*/
+
+void safe_mutex_free_deadlock_data(safe_mutex_t *mp)
+{
+  /* Free all entries that points to this one */
+  if (!(mp->create_flags & MYF_NO_DEADLOCK_DETECTION))
+  {
+    pthread_mutex_lock(&THR_LOCK_mutex);
+    hash_iterate(mp->used_mutex, (hash_walk_action) remove_from_locked_mutex,
+                 mp);
+    hash_iterate(mp->locked_mutex, (hash_walk_action) remove_from_used_mutex,
+                 mp);
+    pthread_mutex_unlock(&THR_LOCK_mutex);
+
+    hash_free(mp->used_mutex);
+    hash_free(mp->locked_mutex);
+    my_free(mp->locked_mutex, 0);
+    mp->create_flags|= MYF_NO_DEADLOCK_DETECTION;
+  }
+}
+
 /*
   Free global resources and check that all mutex has been destroyed
 
@@ -400,7 +627,7 @@ void safe_mutex_end(FILE *file __attribute__((unused)))
   }
   {
     struct st_safe_mutex_info_t *ptr;
-    for (ptr= safe_mutex_root ; ptr ; ptr= ptr->next)
+    for (ptr= safe_mutex_create_root ; ptr ; ptr= ptr->next)
     {
       fprintf(file, "\tMutex %s initiated at line %4u in '%s'\n",
 	      ptr->name, ptr->init_line, ptr->init_file);
@@ -410,6 +637,127 @@ void safe_mutex_end(FILE *file __attribute__((unused)))
 #endif /* SAFE_MUTEX_DETECT_DESTROY */
 }
 
+
+static my_bool add_used_to_locked_mutex(safe_mutex_t *used_mutex,
+                                        safe_mutex_deadlock_t *locked_mutex)
+{
+  /* Add mutex to all parent of the current mutex */
+  if (!locked_mutex->warning_only)
+  {
+    (void) hash_iterate(locked_mutex->mutex->locked_mutex,
+                        (hash_walk_action) add_to_locked_mutex,
+                        used_mutex);
+    /* mark that locked_mutex is locked after used_mutex */
+    (void) add_to_locked_mutex(locked_mutex, used_mutex);
+  }
+  return 0;
+}
+
+
+/**
+   register that locked_mutex was locked after current_mutex
+*/
+
+static my_bool add_to_locked_mutex(safe_mutex_deadlock_t *locked_mutex,
+                                   safe_mutex_t *current_mutex)
+{
+  DBUG_ENTER("add_to_locked_mutex");
+  DBUG_PRINT("info", ("inserting 0x%lx  into  0x%lx  (id: %lu -> %lu)",
+                      (ulong) locked_mutex, (long) current_mutex,
+                      locked_mutex->id, current_mutex->id));
+  if (my_hash_insert(current_mutex->locked_mutex, (uchar*) locked_mutex))
+  {
+    /* Got mutex through two paths; ignore */
+    DBUG_RETURN(0);
+  }
+  locked_mutex->count++;
+  if (my_hash_insert(locked_mutex->mutex->used_mutex,
+                     (uchar*) current_mutex))
+  {
+    DBUG_ASSERT(0);
+  }
+  DBUG_RETURN(0);
+}
+
+
+/**
+  Remove mutex from the locked mutex hash
+  @fn    remove_from_used_mutex()
+  @param mp            Mutex that has delete_mutex in it's locked_mutex hash
+  @param delete_mutex  Mutex should be removed from the hash
+
+  @notes
+    safe_mutex_deadlock_t entries in the locked hash are shared.
+    When counter goes to 0, we delete the safe_mutex_deadlock_t entry.
+*/
+
+static my_bool remove_from_locked_mutex(safe_mutex_t *mp,
+                                        safe_mutex_t *delete_mutex)
+{
+  safe_mutex_deadlock_t *found;
+  DBUG_ENTER("remove_from_locked_mutex");
+  DBUG_PRINT("enter", ("delete_mutex: 0x%lx  mutex: 0x%lx  (id: %lu <- %lu)",
+                       (ulong) delete_mutex, (ulong) mp, 
+                       delete_mutex->id, mp->id));
+
+  found= (safe_mutex_deadlock_t *) hash_search(mp->locked_mutex,
+                                               (uchar*) &delete_mutex->id, 0);
+  DBUG_ASSERT(found);
+  if (found)
+  {
+    if (hash_delete(mp->locked_mutex, (uchar*) found))
+    {
+      DBUG_ASSERT(0);
+    }
+    if (!--found->count)
+      my_free(found, MYF(0));
+  }
+  DBUG_RETURN(0);
+}
+
+static my_bool remove_from_used_mutex(safe_mutex_deadlock_t *locked_mutex,
+                                      safe_mutex_t *mutex)
+{
+  DBUG_ENTER("remove_from_used_mutex");
+  DBUG_PRINT("enter", ("delete_mutex: 0x%lx  mutex: 0x%lx  (id: %lu <- %lu)",
+                       (ulong) mutex, (ulong) locked_mutex, 
+                       mutex->id, locked_mutex->id));
+  if (hash_delete(locked_mutex->mutex->used_mutex, (uchar*) mutex))
+  {
+    DBUG_ASSERT(0);
+  }
+  if (!--locked_mutex->count)
+    my_free(locked_mutex, MYF(0));
+  DBUG_RETURN(0);
+}
+
+
+static void print_deadlock_warning(safe_mutex_t *new_mutex,
+                                   safe_mutex_t *parent_mutex)
+{
+  safe_mutex_t *mutex_root;
+  DBUG_ENTER("print_deadlock_warning");
+  DBUG_PRINT("enter", ("mutex: %s  parent: %s",
+                       new_mutex->name, parent_mutex->name));
+
+  fprintf(stderr, "safe_mutex: Found wrong usage of mutex "
+          "'%s' and '%s'\n",
+          parent_mutex->name, new_mutex->name);
+  fprintf(stderr, "Mutex currently locked (in reverse order):\n");
+  fprintf(stderr, "%-32.32s  %s  line %u\n", new_mutex->name, new_mutex->file,
+          new_mutex->line);
+  for (mutex_root= *my_thread_var_mutex_in_use() ;
+       mutex_root;
+       mutex_root= mutex_root->next)
+  {
+    fprintf(stderr, "%-32.32s  %s  line %u\n", mutex_root->name,
+            mutex_root->file, mutex_root->line);
+  }
+  fflush(stderr);
+  DBUG_VOID_RETURN;
+}
+
+
 #endif /* THREAD && SAFE_MUTEX */
 
 #if defined(THREAD) && defined(MY_PTHREAD_FASTMUTEX) && !defined(SAFE_MUTEX)