WL#3262 add mutex lock order checking to safemutex (also called safe_mutex_deadlock_detector)

This writes a warning on stderr if one uses mutex in different order,
like if one in one case would lock mutex in the order A,B and in another case
would lock mutex in the order B,A
This is inspired by and loosely based on the LOCKDEP patch by Jonas
Wrong mutex order is either fixed or mutex are marked with MYF_NO_DEADLOCK_DETECTION
if used inconsistently (need to be fixed by server team)

KNOWN_BUGS.txt:
  Added information that one need to dump and restore Maria tables
include/hash.h:
  Added prototype function for walking over all elements in a hash
include/my_pthread.h:
  Added my_pthread_mutex_init() and my_pthread_mutex_lock(); These should be used if one wants to disable mutex order checking.
  Changed names of the nonposix mutex_init functions to not conflict with my_phread_mutex_init()
  Added and extended structures for mutex deadlock detection.
  New arguments to sage_mutex_init() and safe_mutex_lock() to allow one to disable mutex order checking.
  Added variable 'safe_mutex_deadlock_detector' to enable/disable deadlock detection for all pthread_mutex_init()
mysys/Makefile.am:
  Added cleaning of test files
  Added test_thr_mutex
mysys/hash.c:
  Added hash_iterate() to iterate over all elements in a hash
  More comments
mysys/my_init.c:
  Added calls to destory all mutex uses by mysys()
  Added waiting for threads to end before calling TERMINATE() to list not freed memory
mysys/my_pthread.c:
  Changed names to free my_pthread_mutex_init() for mutex-lock-order-checking
mysys/my_sleep.c:
  Fixed too long wait if using 1000000L as argument
mysys/my_thr_init.c:
  Mark THR_LOCK_threads and THR_LOCK_malloc to not have mutex deadlock detection.
  (We can't have it enabled for this as these are internal mutex used by the detector  
  Call my_thread_init() early as we need thread specific variables enabled for the following pthread_mutex_init()
  Move code to wait for threads to end to my_wait_for_other_threads_to_die()
  Don't destroy mutex and conditions unless all threads have died
  Added my_thread_destroy_mutex() to destroy all mutex used by the mysys thread system
  Name the thread specific mutex as "mysys_var->mutex"
  Added my_thread_var_mutex_in_use() to return pointer to mutex in use or 0 if thread variables are not initialized
mysys/mysys_priv.h:
  Added prototypes for functions used internally with mutex-wrong-usage detection
mysys/thr_mutex.c:
  Added runtime detection of mutex used in conflicting order
  See WL#3262 or test_thr_mutex.c for examples
  The base idea is for each mutex have two hashes:
  - mutex->locked_mutex points to all mutex used after this one
  - mutex->used_mutex points to all mutex which has this mutex in it's mutex->locked_mutex
  There is a wrong mutex order if any mutex currently locked before this mutex is in the mutex->locked_mutex hash
sql/event_queue.cc:
  Mark mutex used inconsistently (need to be fixed by server team)
sql/event_scheduler.cc:
  Declare the right order to take the mutex
sql/events.cc:
  Mark mutex used inconsistently (need to be fixed by server team)
sql/ha_ndbcluster_binlog.cc:
  Mark mutex used inconsistently (need to be fixed by server team)
sql/log.cc:
  Mark mutex used inconsistently (need to be fixed by server team)
sql/mysqld.cc:
  Use pthread_mutex_trylock instead of pthread_mutex_unlock() when sending kill signal to thread
  This is needed to avoid wrong mutex order as normally one takes 'current_mutex' before mysys_var->mutex.
  Added call to free sp cache.
  Add destruction of LOCK_server_started and COND_server_started.
  Added register_mutex_order() function to register in which order mutex should be taken
  (to initiailize mutex_deadlock_detector).
  Added option to turn off safe_mutex_deadlock_detector
sql/protocol.cc:
  Fixed wrong argument to DBUG_PRINT (found by valgrind)
sql/rpl_mi.cc:
  Mark mutex used inconsistently (need to be fixed by server team)
sql/set_var.cc:
  Remove wrong locking of LOCK_global_system_variables when reading and setting log variables
  (would cause inconsistent mutex order).
  Update global variables outside of logger.unlock() as LOCK_global_system_variables has to be taken before logger locks
  Reviewed by gluh
sql/sp_cache.cc:
  Added function to destroy mutex used by sp cache
sql/sp_cache.h:
  Added function to destroy mutex used by sp cache
sql/sql_class.cc:
  Use pthread_mutex_trylock instead of pthread_mutex_unlock() when sending kill signal to thread
  This is needed to avoid wrong mutex order as normally one takes 'current_mutex' before mysys_var->mutex.
  Register order in which LOCK_delete and mysys_var->mutex is taken
sql/sql_insert.cc:
  Give a name for Delayed_insert::mutex
  Mark mutex used inconsistently (need to be fixed by server team)
  Move closing of tables outside of di->mutex (to avoid wrong mutex order)
sql/sql_show.cc:
  Don't keep LOCK_global_system_variables locked over value->show_type() as this leads to wrong mutex order
storage/innobase/handler/ha_innodb.cc:
  Disable safe_muted_deadlock_detector for innobase intern mutex (to speed up page cache initialization)
storage/maria/ha_maria.cc:
  Added flag to ha_maria::info() to signal if we need to lock table share or not.
  This is needed to avoid locking mutex in wrong order
storage/maria/ha_maria.h:
  Added flag to ha_maria::info() to signal if we need to lock table share or not.
storage/maria/ma_close.c:
  Destroy key_del_lock
  Simplify freeing ftparser_param
storage/maria/ma_key.c:
  Better comment
storage/maria/ma_loghandler.c:
  Mark mutex used inconsistently (need to be fixed by sanja)
storage/maria/ma_state.c:
  More comments
storage/maria/ma_test1.c:
  Ensure that safe_mutex_deadlock_detector is always on (should be, this is just for safety)
storage/maria/ma_test2.c:
  Ensure that safe_mutex_deadlock_detector is always on (should be, this is just for safety)

34 files changed, 832 insertions, 146 deletions

diff --git a/KNOWN_BUGS.txt b/KNOWN_BUGS.txt
index 12f9c2fa123..189c7dcd613 100644
--- a/KNOWN_BUGS.txt
+++ b/KNOWN_BUGS.txt
@@ -20,14 +20,29 @@ If you have found a bug that is not listed here, please add it to
 http://bugs.mysql.com/ so that we can either fix it for next release
 or in the worst case add it here for others to know!
 
+IMPORTANT:
+
+If you have been using a MySQL-5.1-Maria-alpha build and upgrading to
+MySQL-5.1-Maria-beta you MUST run maria_chk --recover on all your
+Maria tables.  This is because we made an incompatible change of how
+transaction id is stored and old transaction id's must be reset!
+
+cd mysql-data-directory
+maria_chk --recover */*.MAI
+
+As the Maria-1.5 engine is now in beta we will do our best to not
+introduce any incompatible changes in the data format for the Maria
+tables; If this would be ever be needed, we will, if possible, support
+both the old and the new version to make upgrades as easy as possible.
 
 Known bugs that we are working on and will be fixed shortly
 ===========================================================
 
-- We have some instabilities in log writing that is under investigatation
+- We have some time ago some instabilities in log writing that is was 
+  under investigation but we haven't been able to repeat in a while.
   This causes mainly assert to triggers in the code and sometimes
   the log handler doesn't start up after restart.
-  Most of this should now be fixed...
+  Most of this should now be fixed.
 
 - INSERT on a duplicate key against a key inserted by another connection
   that has not yet ended will give a duplicate key error instead of
diff --git a/include/hash.h b/include/hash.h
index 4ca8dc0e8bf..5a338bb63f2 100644
--- a/include/hash.h
+++ b/include/hash.h
@@ -32,6 +32,7 @@ extern "C" {
 
 typedef uchar *(*hash_get_key)(const uchar *,size_t*,my_bool);
 typedef void (*hash_free_key)(void *);
+typedef my_bool (*hash_walk_action)(void *,void *);
 
 typedef struct st_hash {
   size_t key_offset,key_length;		/* Length of key if const length */
@@ -66,6 +67,7 @@ my_bool hash_delete(HASH *hash,uchar *record);
 my_bool hash_update(HASH *hash,uchar *record,uchar *old_key,size_t old_key_length);
 void hash_replace(HASH *hash, HASH_SEARCH_STATE *state, uchar *new_row);
 my_bool hash_check(HASH *hash);			/* Only in debug library */
+my_bool hash_iterate(HASH *hash, hash_walk_action action, void *argument);
 
 #define hash_clear(H) bzero((char*) (H),sizeof(*(H)))
 #define hash_inited(H) ((H)->array.buffer != 0)
diff --git a/include/my_pthread.h b/include/my_pthread.h
index 9df8a9e6a65..2df664006e9 100644
--- a/include/my_pthread.h
+++ b/include/my_pthread.h
@@ -239,13 +239,13 @@ int my_sigwait(const sigset_t *set,int *sig);
 
 #ifdef HAVE_NONPOSIX_PTHREAD_MUTEX_INIT
 #ifndef SAFE_MUTEX
-#define pthread_mutex_init(a,b) my_pthread_mutex_init((a),(b))
-extern int my_pthread_mutex_init(pthread_mutex_t *mp,
-				 const pthread_mutexattr_t *attr);
+#define pthread_mutex_init(a,b) my_pthread_mutex_noposix_init((a),(b))
+extern int my_pthread_mutex_noposix_init(pthread_mutex_t *mp,
+                                         const pthread_mutexattr_t *attr);
 #endif /* SAFE_MUTEX */
-#define pthread_cond_init(a,b) my_pthread_cond_init((a),(b))
-extern int my_pthread_cond_init(pthread_cond_t *mp,
-				const pthread_condattr_t *attr);
+#define pthread_cond_init(a,b) my_pthread_cond_noposix_init((a),(b))
+extern int my_pthread_cond_noposix_init(pthread_cond_t *mp,
+                                        const pthread_condattr_t *attr);
 #endif /* HAVE_NONPOSIX_PTHREAD_MUTEX_INIT */
 
 #if defined(HAVE_SIGTHREADMASK) && !defined(HAVE_PTHREAD_SIGMASK)
@@ -449,18 +449,33 @@ int my_pthread_mutex_trylock(pthread_mutex_t *mutex);
 #if defined(__NETWARE__) && !defined(SAFE_MUTEX_DETECT_DESTROY)
 #define SAFE_MUTEX_DETECT_DESTROY
 #endif
+struct st_hash;
 
 typedef struct st_safe_mutex_t
 {
   pthread_mutex_t global,mutex;
   const char *file, *name;
   uint line,count;
+  myf create_flags, active_flags;
+  ulong id;
   pthread_t thread;
+  struct st_hash *locked_mutex, *used_mutex;
+  struct st_safe_mutex_t *prev, *next;
 #ifdef SAFE_MUTEX_DETECT_DESTROY
   struct st_safe_mutex_info_t *info;	/* to track destroying of mutexes */
 #endif
 } safe_mutex_t;
 
+typedef struct st_safe_mutex_deadlock_t
+{
+  const char *file, *name;
+  safe_mutex_t *mutex;
+  uint line;
+  ulong count;
+  ulong id;
+  my_bool warning_only;
+} safe_mutex_deadlock_t;
+
 #ifdef SAFE_MUTEX_DETECT_DESTROY
 /*
   Used to track the destroying of mutexes. This needs to be a seperate
@@ -478,8 +493,10 @@ typedef struct st_safe_mutex_info_t
 #endif /* SAFE_MUTEX_DETECT_DESTROY */
 
 int safe_mutex_init(safe_mutex_t *mp, const pthread_mutexattr_t *attr,
-                    const char *file, uint line, const char *name);
-int safe_mutex_lock(safe_mutex_t *mp, my_bool try_lock, const char *file, uint line);
+                    const char *name, myf my_flags,
+                    const char *file, uint line);
+int safe_mutex_lock(safe_mutex_t *mp, myf my_flags, const char *file,
+                    uint line);
 int safe_mutex_unlock(safe_mutex_t *mp,const char *file, uint line);
 int safe_mutex_destroy(safe_mutex_t *mp,const char *file, uint line);
 int safe_cond_wait(pthread_cond_t *cond, safe_mutex_t *mp,const char *file,
@@ -488,8 +505,12 @@ int safe_cond_timedwait(pthread_cond_t *cond, safe_mutex_t *mp,
 			struct timespec *abstime, const char *file, uint line);
 void safe_mutex_global_init(void);
 void safe_mutex_end(FILE *file);
+void safe_mutex_free_deadlock_data(safe_mutex_t *mp);
 
 	/* Wrappers if safe mutex is actually used */
+#define MYF_TRY_LOCK              1
+#define MYF_NO_DEADLOCK_DETECTION 2
+
 #ifdef SAFE_MUTEX
 #undef pthread_mutex_init
 #undef pthread_mutex_lock
@@ -501,13 +522,15 @@ void safe_mutex_end(FILE *file);
 #undef pthread_cond_wait
 #undef pthread_cond_timedwait
 #undef pthread_mutex_trylock
-#define pthread_mutex_init(A,B) safe_mutex_init((A),(B),__FILE__,__LINE__,#A)
-#define pthread_mutex_lock(A) safe_mutex_lock((A), FALSE, __FILE__, __LINE__)
+#define my_pthread_mutex_init(A,B,C,D) safe_mutex_init((A),(B),(C),(D),__FILE__,__LINE__)
+#define pthread_mutex_init(A,B) safe_mutex_init((A),(B),#A,0,__FILE__,__LINE__)
+#define pthread_mutex_lock(A) safe_mutex_lock((A), 0, __FILE__, __LINE__)
+#define my_pthread_mutex_lock(A,B) safe_mutex_lock((A), (B), __FILE__, __LINE__)
 #define pthread_mutex_unlock(A) safe_mutex_unlock((A),__FILE__,__LINE__)
 #define pthread_mutex_destroy(A) safe_mutex_destroy((A),__FILE__,__LINE__)
 #define pthread_cond_wait(A,B) safe_cond_wait((A),(B),__FILE__,__LINE__)
 #define pthread_cond_timedwait(A,B,C) safe_cond_timedwait((A),(B),(C),__FILE__,__LINE__)
-#define pthread_mutex_trylock(A) safe_mutex_lock((A), TRUE, __FILE__, __LINE__)
+#define pthread_mutex_trylock(A) safe_mutex_lock((A), MYF_TRY_LOCK, __FILE__, __LINE__)
 #define pthread_mutex_t safe_mutex_t
 #define safe_mutex_assert_owner(mp) \
           DBUG_ASSERT((mp)->count > 0 && \
@@ -516,8 +539,11 @@ void safe_mutex_end(FILE *file);
           DBUG_ASSERT(! (mp)->count || \
                       ! pthread_equal(pthread_self(), (mp)->thread))
 #else
+#define my_pthread_mutex_init(A,B,C,D) pthread_mutex_init((A),(B))
+#define my_pthread_mutex_lock(A,B) pthread_mutex_lock(A)
 #define safe_mutex_assert_owner(mp)
 #define safe_mutex_assert_not_owner(mp)
+#define safe_mutex_free_deadlock_data(mp)
 #endif /* SAFE_MUTEX */
 
 #if defined(MY_PTHREAD_FASTMUTEX) && !defined(SAFE_MUTEX)
@@ -685,6 +711,7 @@ struct st_my_thread_var
   void *opt_info;
   uint  lock_type; /* used by conditional release the queue */
   void  *stack_ends_here;
+  safe_mutex_t *mutex_in_use;
 #ifndef DBUG_OFF
   void *dbug;
   char name[THREAD_NAME_SIZE+1];
@@ -693,7 +720,9 @@ struct st_my_thread_var
 
 extern struct st_my_thread_var *_my_thread_var(void) __attribute__ ((const));
 extern void **my_thread_var_dbug();
+extern safe_mutex_t **my_thread_var_mutex_in_use();
 extern uint my_thread_end_wait_time;
+extern my_bool safe_mutex_deadlock_detector;
 #define my_thread_var (_my_thread_var())
 #define my_errno my_thread_var->thr_errno
 /*
diff --git a/mysys/Makefile.am b/mysys/Makefile.am
index e5e7539ece6..6efdd0d75e7 100644
--- a/mysys/Makefile.am
+++ b/mysys/Makefile.am
@@ -79,6 +79,13 @@ DEFS =			-DDEFAULT_BASEDIR=\"$(prefix)\" \
 # I hope this always does the right thing. Otherwise this is only test programs
 FLAGS=$(DEFS) $(INCLUDES) $(CPPFLAGS) $(CFLAGS) @NOINST_LDFLAGS@
 
+CLEANFILES =		test_bitmap$(EXEEXT) test_priority_queue$(EXEEXT) \
+			test_thr_alarm$(EXEEXT) test_thr_lock$(EXEEXT) \
+			test_vsnprintf$(EXEEXT) test_io_cache$(EXEEXT) \
+			test_dir$(EXEEXT) test_charset$(EXEEXT) \
+			testhash$(EXEEXT) test_gethwaddr$(EXEEXT) \
+			test_base64$(EXEEXT) test_thr_mutex$(EXEEXT)
+
 #
 # The CP .. RM stuff is to avoid problems with some compilers (like alpha ccc)
 # which automaticly removes the object files you use to compile a final program
@@ -129,6 +136,9 @@ test_base64$(EXEEXT): base64.c $(LIBRARIES)
 	$(LINK) $(FLAGS) -DMAIN  ./test_base64.c $(LDADD) $(LIBS)
 	$(RM) -f ./test_base64.c
 
+test_thr_mutex$(EXEEXT): test_thr_mutex.c $(LIBRARIES)
+	$(LINK) $(FLAGS) $(srcdir)/test_thr_mutex.c $(LDADD) $(LIBS)
+
 
 # Don't update the files from bitkeeper
 %::SCCS/s.%
diff --git a/mysys/hash.c b/mysys/hash.c
index 3a9f05a3e0b..0d3f79bc40f 100644
--- a/mysys/hash.c
+++ b/mysys/hash.c
@@ -304,7 +304,13 @@ static int hashcmp(const HASH *hash, HASH_LINK *pos, const uchar *key,
 }
 
 
-	/* Write a hash-key to the hash-index */
+/**
+   Write a hash-key to the hash-index
+
+   @return
+   @retval  0  ok
+   @retval  1  Duplicate key or out of memory
+*/
 
 my_bool my_hash_insert(HASH *info,const uchar *record)
 {
@@ -443,11 +449,21 @@ my_bool my_hash_insert(HASH *info,const uchar *record)
 }
 
 
-/******************************************************************************
-** Remove one record from hash-table. The record with the same record
-** ptr is removed.
-** if there is a free-function it's called for record if found
-******************************************************************************/
+/**
+   Remove one record from hash-table.
+
+   @fn    hash_delete()
+   @param hash		Hash tree
+   @param record	Row to be deleted
+
+   @notes
+   The record with the same record ptr is removed.
+   If there is a free-function it's called if record was found.
+
+   @return
+   @retval  0  ok
+   @retval  1 Record not found
+*/
 
 my_bool hash_delete(HASH *hash,uchar *record)
 {
@@ -656,6 +672,37 @@ void hash_replace(HASH *hash, HASH_SEARCH_STATE *current_record, uchar *new_row)
 }
 
 
+/**
+   Iterate over all elements in hash and call function with the element
+
+   @param hash     hash array
+   @param action   function to call for each argument
+   @param argument second argument for call to action
+
+   @notes
+   If one of functions calls returns 1 then the iteration aborts
+
+   @retval 0  ok
+   @retval 1  iteration aborted becasue action returned 1
+*/
+
+my_bool hash_iterate(HASH *hash, hash_walk_action action, void *argument)
+{
+  uint records, i;
+  HASH_LINK *data;
+
+  records= hash->records;
+  data= dynamic_element(&hash->array,0,HASH_LINK*);
+
+  for (i= 0 ; i < records ; i++)
+  {
+    if ((*action)(data[i].data, argument))
+      return 1;
+  }
+  return 0;
+}
+
+
 #ifndef DBUG_OFF
 
 my_bool hash_check(HASH *hash)
diff --git a/mysys/my_init.c b/mysys/my_init.c
index a153275f87e..453e62b19bb 100644
--- a/mysys/my_init.c
+++ b/mysys/my_init.c
@@ -165,6 +165,9 @@ void my_end(int infoflag)
   free_charsets();
   my_error_unregister_all();
   my_once_free();
+#ifdef THREAD
+  my_thread_destroy_mutex();
+#endif
 
   if ((infoflag & MY_GIVE_INFO) || print_info)
   {
@@ -195,6 +198,8 @@ Voluntary context switches %ld, Involuntary context switches %ld\n",
     fprintf(info_file,"\nRun time: %.1f\n",(double) clock()/CLOCKS_PER_SEC);
 #endif
 #if defined(SAFEMALLOC)
+    /* Wait for other threads to free mysys_var */
+    (void) my_wait_for_other_threads_to_die(1);
     TERMINATE(stderr, (infoflag & MY_GIVE_INFO) != 0);
 #elif defined(__WIN__) && defined(_MSC_VER)
    _CrtSetReportMode( _CRT_WARN, _CRTDBG_MODE_FILE );
diff --git a/mysys/my_pthread.c b/mysys/my_pthread.c
index aba3e47d754..e97bbe89be0 100644
--- a/mysys/my_pthread.c
+++ b/mysys/my_pthread.c
@@ -429,7 +429,8 @@ int sigwait(sigset_t *setp, int *sigp)
 
 #include <netdb.h>
 
-int my_pthread_mutex_init(pthread_mutex_t *mp, const pthread_mutexattr_t *attr)
+int my_pthread_mutex_noposix_init(pthread_mutex_t *mp,
+                                  const pthread_mutexattr_t *attr)
 {
   int error;
   if (!attr)
@@ -439,7 +440,8 @@ int my_pthread_mutex_init(pthread_mutex_t *mp, const pthread_mutexattr_t *attr)
   return error;
 }
 
-int my_pthread_cond_init(pthread_cond_t *mp, const pthread_condattr_t *attr)
+int my_pthread_cond_noposix_init(pthread_cond_t *mp,
+                                 const pthread_condattr_t *attr)
 {
   int error;
   if (!attr)
diff --git a/mysys/my_sleep.c b/mysys/my_sleep.c
index 87170e4af41..cb21c15a925 100644
--- a/mysys/my_sleep.c
+++ b/mysys/my_sleep.c
@@ -30,7 +30,7 @@ void my_sleep(ulong m_seconds)
   t.tv_usec= m_seconds % 1000000L;
   select(0,0,0,0,&t); /* sleep */
 #else
-  uint sec=    (uint) (m_seconds / 1000000L);
+  uint sec=    (uint) ((m_seconds + 999999L) / 1000000L);
   ulong start= (ulong) time((time_t*) 0);
   while ((ulong) time((time_t*) 0) < start+sec);
 #endif
diff --git a/mysys/my_thr_init.c b/mysys/my_thr_init.c
index 1d03577ce34..6ebd1f512f1 100644
--- a/mysys/my_thr_init.c
+++ b/mysys/my_thr_init.c
@@ -115,6 +115,15 @@ my_bool my_thread_global_init(void)
   }
 #endif /* TARGET_OS_LINUX */
 
+  /* Mutex used by my_thread_init() and after my_thread_destroy_mutex() */
+  my_pthread_mutex_init(&THR_LOCK_threads, MY_MUTEX_INIT_FAST, 
+                        "THR_LOCK_threads", MYF_NO_DEADLOCK_DETECTION);
+  my_pthread_mutex_init(&THR_LOCK_malloc, MY_MUTEX_INIT_FAST,
+                        "THR_LOCK_malloc", MYF_NO_DEADLOCK_DETECTION);
+
+  if (my_thread_init())
+    return 1;
+
 #ifdef PTHREAD_ADAPTIVE_MUTEX_INITIALIZER_NP
   /*
     Set mutex type to "fast" a.k.a "adaptive"
@@ -138,7 +147,7 @@ my_bool my_thread_global_init(void)
                             PTHREAD_MUTEX_ERRORCHECK);
 #endif
 
-  pthread_mutex_init(&THR_LOCK_malloc,MY_MUTEX_INIT_FAST);
+  /* Mutex uses by mysys */
   pthread_mutex_init(&THR_LOCK_open,MY_MUTEX_INIT_FAST);
   pthread_mutex_init(&THR_LOCK_lock,MY_MUTEX_INIT_FAST);
   pthread_mutex_init(&THR_LOCK_isam,MY_MUTEX_INIT_SLOW);
@@ -146,7 +155,6 @@ my_bool my_thread_global_init(void)
   pthread_mutex_init(&THR_LOCK_heap,MY_MUTEX_INIT_FAST);
   pthread_mutex_init(&THR_LOCK_net,MY_MUTEX_INIT_FAST);
   pthread_mutex_init(&THR_LOCK_charset,MY_MUTEX_INIT_FAST);
-  pthread_mutex_init(&THR_LOCK_threads,MY_MUTEX_INIT_FAST);
   pthread_mutex_init(&THR_LOCK_time,MY_MUTEX_INIT_FAST);
   pthread_cond_init(&THR_COND_threads, NULL);
 #if defined( __WIN__) || defined(OS2)
@@ -158,44 +166,64 @@ my_bool my_thread_global_init(void)
 #ifndef HAVE_GETHOSTBYNAME_R
   pthread_mutex_init(&LOCK_gethostbyname_r,MY_MUTEX_INIT_SLOW);
 #endif
-  if (my_thread_init())
-  {
-    my_thread_global_end();			/* Clean up */
-    return 1;
-  }
   return 0;
 }
 
 
-void my_thread_global_end(void)
+/**
+   Wait for all threads in system to die
+   @fn    my_wait_for_other_threads_to_die()
+   @param number_of_threads  Wait until this number of threads
+
+   @retval  0  Less or equal to number_of_threads left
+   @retval  1  Wait failed
+*/
+
+my_bool my_wait_for_other_threads_to_die(uint number_of_threads)
 {
   struct timespec abstime;
   my_bool all_threads_killed= 1;
 
   set_timespec(abstime, my_thread_end_wait_time);
   pthread_mutex_lock(&THR_LOCK_threads);
-  while (THR_thread_count > 0)
+  while (THR_thread_count > number_of_threads)
   {
     int error= pthread_cond_timedwait(&THR_COND_threads, &THR_LOCK_threads,
                                       &abstime);
     if (error == ETIMEDOUT || error == ETIME)
     {
-#ifdef HAVE_PTHREAD_KILL
-      /*
-        We shouldn't give an error here, because if we don't have
-        pthread_kill(), programs like mysqld can't ensure that all threads
-        are killed when we enter here.
-      */
-      if (THR_thread_count)
-        fprintf(stderr,
-                "Error in my_thread_global_end(): %d threads didn't exit\n",
-                THR_thread_count);
-#endif
       all_threads_killed= 0;
       break;
     }
   }
   pthread_mutex_unlock(&THR_LOCK_threads);
+  return all_threads_killed;
+}
+
+
+/**
+   End the mysys thread system. Called when ending the last thread
+*/
+
+
+void my_thread_global_end(void)
+{
+  my_bool all_threads_killed;
+
+  if (!(all_threads_killed= my_wait_for_other_threads_to_die(0)))
+  {
+#ifdef HAVE_PTHREAD_KILL
+    /*
+      We shouldn't give an error here, because if we don't have
+      pthread_kill(), programs like mysqld can't ensure that all threads
+      are killed when we enter here.
+    */
+    if (THR_thread_count)
+      fprintf(stderr,
+              "Error in my_thread_global_end(): %d threads didn't exit\n",
+              THR_thread_count);
+#endif
+  }
 
   pthread_key_delete(THR_KEY_mysys);
 #ifdef PTHREAD_ADAPTIVE_MUTEX_INITIALIZER_NP
@@ -204,7 +232,25 @@ void my_thread_global_end(void)
 #ifdef PTHREAD_ERRORCHECK_MUTEX_INITIALIZER_NP
   pthread_mutexattr_destroy(&my_errorcheck_mutexattr);
 #endif
-  pthread_mutex_destroy(&THR_LOCK_malloc);
+  if (all_threads_killed)
+  {
+    pthread_mutex_destroy(&THR_LOCK_threads);
+    pthread_cond_destroy(&THR_COND_threads);
+    pthread_mutex_destroy(&THR_LOCK_malloc);
+  }
+}
+
+/* Free all mutex used by mysys */
+
+void my_thread_destroy_mutex(void)
+{
+  struct st_my_thread_var *tmp;
+  tmp= my_pthread_getspecific(struct st_my_thread_var*,THR_KEY_mysys);
+  if (tmp)
+  {
+    safe_mutex_free_deadlock_data(&tmp->mutex);
+  }  
+
   pthread_mutex_destroy(&THR_LOCK_open);
   pthread_mutex_destroy(&THR_LOCK_lock);
   pthread_mutex_destroy(&THR_LOCK_isam);
@@ -213,11 +259,6 @@ void my_thread_global_end(void)
   pthread_mutex_destroy(&THR_LOCK_net);
   pthread_mutex_destroy(&THR_LOCK_time);
   pthread_mutex_destroy(&THR_LOCK_charset);
-  if (all_threads_killed)
-  {
-    pthread_mutex_destroy(&THR_LOCK_threads);
-    pthread_cond_destroy(&THR_COND_threads);
-  }
 #if !defined(HAVE_LOCALTIME_R) || !defined(HAVE_GMTIME_R)
   pthread_mutex_destroy(&LOCK_localtime_r);
 #endif
@@ -287,7 +328,8 @@ my_bool my_thread_init(void)
 #else
   tmp->pthread_self= pthread_self();
 #endif
-  pthread_mutex_init(&tmp->mutex,MY_MUTEX_INIT_FAST);
+  my_pthread_mutex_init(&tmp->mutex, MY_MUTEX_INIT_FAST, "mysys_var->mutex",
+                        0);
   pthread_cond_init(&tmp->suspend, NULL);
 
   tmp->stack_ends_here= &tmp + STACK_DIRECTION * my_thread_stack_size;
@@ -330,6 +372,13 @@ void my_thread_end(void)
 #endif
   if (tmp && tmp->init)
   {
+
+#if !defined(__bsdi__) && !defined(__OpenBSD__)
+ /* bsdi and openbsd 3.5 dumps core here */
+    pthread_cond_destroy(&tmp->suspend);
+#endif
+    pthread_mutex_destroy(&tmp->mutex);
+
 #if !defined(DBUG_OFF)
     /* tmp->dbug is allocated inside DBUG library */
     if (tmp->dbug)
@@ -339,12 +388,11 @@ void my_thread_end(void)
       tmp->dbug=0;
     }
 #endif
-#if !defined(__bsdi__) && !defined(__OpenBSD__)
- /* bsdi and openbsd 3.5 dumps core here */
-    pthread_cond_destroy(&tmp->suspend);
-#endif
-    pthread_mutex_destroy(&tmp->mutex);
 #if !defined(__WIN__) || defined(USE_TLS)
+#ifndef DBUG_OFF
+    /* To find bugs when accessing unallocated data */
+    bfill(tmp, sizeof(tmp), 0x8F);
+#endif
     free(tmp);
 #else
     tmp->init= 0;
@@ -399,6 +447,15 @@ extern void **my_thread_var_dbug()
 }
 #endif
 
+/* Return pointer to mutex_in_use */
+
+safe_mutex_t **my_thread_var_mutex_in_use()
+{
+  struct st_my_thread_var *tmp=
+    my_pthread_getspecific(struct st_my_thread_var*,THR_KEY_mysys);
+  return tmp ? &tmp->mutex_in_use : 0;
+}
+
 /****************************************************************************
   Get name of current thread.
 ****************************************************************************/
diff --git a/mysys/mysys_priv.h b/mysys/mysys_priv.h
index 6e0959ae08c..113b64005f2 100644
--- a/mysys/mysys_priv.h
+++ b/mysys/mysys_priv.h
@@ -33,6 +33,7 @@ extern pthread_mutex_t THR_LOCK_charset, THR_LOCK_time;
 #include <my_no_pthread.h>
 #endif
 
+
 /*
   EDQUOT is used only in 3 C files only in mysys/. If it does not exist on
   system, we set it to some value which can never happen.
@@ -42,3 +43,5 @@ extern pthread_mutex_t THR_LOCK_charset, THR_LOCK_time;
 #endif
 
 void my_error_unregister_all(void);
+void my_thread_destroy_mutex(void);
+my_bool my_wait_for_other_threads_to_die(uint number_of_threads);
diff --git a/mysys/thr_mutex.c b/mysys/thr_mutex.c
index aa46021a938..ddbe613cdae 100644
--- a/mysys/thr_mutex.c
+++ b/mysys/thr_mutex.c
@@ -1,4 +1,4 @@
-/* Copyright (C) 2000-2003 MySQL AB
+/* Copyright (C) 2000-2008 MySQL AB, 2008 Sun Microsystems, Inc
 
    This program is free software; you can redistribute it and/or modify
    it under the terms of the GNU General Public License as published by
@@ -24,6 +24,7 @@
 #include "mysys_priv.h"
 #include "my_static.h"
 #include <m_string.h>
+#include <hash.h>
 
 #ifndef DO_NOT_REMOVE_THREAD_WRAPPERS
 /* Remove wrappers */
@@ -34,28 +35,68 @@
 #undef pthread_mutex_destroy
 #undef pthread_cond_wait
 #undef pthread_cond_timedwait
+#undef safe_mutex_free_deadlock_data
 #ifdef HAVE_NONPOSIX_PTHREAD_MUTEX_INIT
-#define pthread_mutex_init(a,b) my_pthread_mutex_init((a),(b))
+#define pthread_mutex_init(a,b) my_pthread_noposix_mutex_init((a),(b))
 #endif
 #endif /* DO_NOT_REMOVE_THREAD_WRAPPERS */
 
 static pthread_mutex_t THR_LOCK_mutex;
 static ulong safe_mutex_count= 0;		/* Number of mutexes created */
+static ulong safe_mutex_id= 0;
+my_bool safe_mutex_deadlock_detector= 1;        /* On by default */
+
 #ifdef SAFE_MUTEX_DETECT_DESTROY
-static struct st_safe_mutex_info_t *safe_mutex_root= NULL;
+static struct st_safe_mutex_create_info_t *safe_mutex_create_root= NULL;
 #endif
 
+static my_bool add_used_to_locked_mutex(safe_mutex_t *used_mutex,
+                                        safe_mutex_deadlock_t *locked_mutex);
+static my_bool add_to_locked_mutex(safe_mutex_deadlock_t *locked_mutex,
+                                   safe_mutex_t *current_mutex);
+static my_bool remove_from_locked_mutex(safe_mutex_t *mp,
+                                        safe_mutex_t *delete_mutex);
+static my_bool remove_from_used_mutex(safe_mutex_deadlock_t *locked_mutex,
+                                      safe_mutex_t *mutex);
+static void print_deadlock_warning(safe_mutex_t *new_mutex,
+                                   safe_mutex_t *conflicting_mutex);
+
 void safe_mutex_global_init(void)
 {
   pthread_mutex_init(&THR_LOCK_mutex,MY_MUTEX_INIT_FAST);
+  safe_mutex_id= safe_mutex_count= 0;
+  safe_mutex_deadlock_detector= 1;
+
+#ifdef SAFE_MUTEX_DETECT_DESTROY
+  safe_mutex_create_root= 0;
+#endif
+}
+
+static inline void remove_from_active_list(safe_mutex_t *mp)
+{
+  if (!(mp->active_flags & (MYF_NO_DEADLOCK_DETECTION | MYF_TRY_LOCK)))
+  {
+    /* Remove mutex from active mutex linked list */
+    if (mp->next)
+      mp->next->prev= mp->prev;
+    if (mp->prev)
+      mp->prev->next= mp->next;
+    else
+      *my_thread_var_mutex_in_use()= mp->next;
+  }
+  mp->prev= mp->next= 0;
 }
 
 
 int safe_mutex_init(safe_mutex_t *mp,
 		    const pthread_mutexattr_t *attr __attribute__((unused)),
+                    const char *name,
+                    myf my_flags,
 		    const char *file,
-		    uint line, const char *name)
+		    uint line)
 {
+  DBUG_ENTER("safe_mutex_init");
+  DBUG_PRINT("enter",("mutex: 0x%lx  name: %s", (ulong) mp, name));
   bzero((char*) mp,sizeof(*mp));
   pthread_mutex_init(&mp->global,MY_MUTEX_INIT_ERRCHK);
   pthread_mutex_init(&mp->mutex,attr);
@@ -65,6 +106,36 @@ int safe_mutex_init(safe_mutex_t *mp,
   /* Skip the very common '&' prefix from the autogenerated name */
   mp->name= name[0] == '&' ? name + 1 : name;
 
+  if (safe_mutex_deadlock_detector && !( my_flags & MYF_NO_DEADLOCK_DETECTION))
+  {
+    if (!my_multi_malloc(MY_FAE | MY_WME,
+                         &mp->locked_mutex, sizeof(*mp->locked_mutex),
+                         &mp->used_mutex, sizeof(*mp->used_mutex), NullS))
+    {
+      /* Disable deadlock handling for this mutex */
+      my_flags|= MYF_NO_DEADLOCK_DETECTION;
+    }
+    else
+    {
+      pthread_mutex_lock(&THR_LOCK_mutex);
+      mp->id= ++safe_mutex_id;
+      pthread_mutex_unlock(&THR_LOCK_mutex);
+      hash_init(mp->locked_mutex, &my_charset_bin,
+                1000,
+                offsetof(safe_mutex_deadlock_t, id),
+                sizeof(mp->id),
+                0, 0, HASH_UNIQUE);
+      hash_init(mp->used_mutex, &my_charset_bin,
+                1000,
+                offsetof(safe_mutex_t, id),
+                sizeof(mp->id),
+                0, 0, HASH_UNIQUE);
+    }
+  }
+  else
+    my_flags|= MYF_NO_DEADLOCK_DETECTION;
+  mp->create_flags= my_flags;
+
 #ifdef SAFE_MUTEX_DETECT_DESTROY
   /*
     Monitor the freeing of mutexes.  This code depends on single thread init
@@ -72,7 +143,7 @@ int safe_mutex_init(safe_mutex_t *mp,
   */
   if ((mp->info= (safe_mutex_info_t *) malloc(sizeof(safe_mutex_info_t))))
   {
-    struct st_safe_mutex_info_t *info =mp->info;
+    struct st_safe_mutex_info_t *info= mp->info;
 
     info->init_file= file;
     info->init_line= line;
@@ -80,20 +151,21 @@ int safe_mutex_init(safe_mutex_t *mp,
     info->next= NULL;
 
     pthread_mutex_lock(&THR_LOCK_mutex);
-    if ((info->next= safe_mutex_root))
-      safe_mutex_root->prev= info;
-    safe_mutex_root= info;
+    if ((info->next= safe_mutex_create_root))
+      safe_mutex_create_root->prev= info;
+    safe_mutex_create_root= info;
     safe_mutex_count++;
     pthread_mutex_unlock(&THR_LOCK_mutex);
   }
 #else
   thread_safe_increment(safe_mutex_count, &THR_LOCK_mutex);
 #endif /* SAFE_MUTEX_DETECT_DESTROY */
-  return 0;
+  DBUG_RETURN(0);
 }
 
 
-int safe_mutex_lock(safe_mutex_t *mp, my_bool try_lock, const char *file, uint line)
+int safe_mutex_lock(safe_mutex_t *mp, myf my_flags, const char *file,
+                    uint line)
 {
   int error;
   DBUG_PRINT("mutex", ("%s (0x%lx) locking", mp->name ? mp->name : "Null",
@@ -110,12 +182,13 @@ int safe_mutex_lock(safe_mutex_t *mp, my_bool try_lock, const char *file, uint l
   pthread_mutex_lock(&mp->global);
   if (mp->count > 0)
   {
-    if (try_lock)
-    {
-      pthread_mutex_unlock(&mp->global);
-      return EBUSY;
-    }
-    else if (pthread_equal(pthread_self(),mp->thread))
+    /*
+      Check that we are not trying to lock mutex twice. This is an error
+      even if we are using 'try_lock' as it's not portably what happens
+      if you lock the mutex many times and this is in any case bad
+      behaviour that should not be encouraged
+    */
+    if (pthread_equal(pthread_self(),mp->thread))
     {
       fprintf(stderr,
               "safe_mutex: Trying to lock mutex at %s, line %d, when the"
@@ -143,7 +216,7 @@ int safe_mutex_lock(safe_mutex_t *mp, my_bool try_lock, const char *file, uint l
       instead just return EBUSY, since this is the expected behaviour
       of trylock().
    */
-  if (try_lock)
+  if (my_flags & MYF_TRY_LOCK)
   {
     error= pthread_mutex_trylock(&mp->mutex);
     if (error == EBUSY)
@@ -169,7 +242,93 @@ int safe_mutex_lock(safe_mutex_t *mp, my_bool try_lock, const char *file, uint l
   }
   mp->file= file;
   mp->line= line;
+  mp->active_flags= mp->create_flags | my_flags;
   pthread_mutex_unlock(&mp->global);
+
+  /* Deadlock detection */
+
+  mp->prev= mp->next= 0;
+  if (!(mp->active_flags & (MYF_TRY_LOCK | MYF_NO_DEADLOCK_DETECTION)))
+  {
+    safe_mutex_t **mutex_in_use= my_thread_var_mutex_in_use();
+
+    if (!mutex_in_use)
+    {
+      /* thread has not called my_thread_init() */
+      mp->active_flags|= MYF_NO_DEADLOCK_DETECTION;
+    }
+    else
+    {
+      safe_mutex_t *mutex_root;
+      if ((mutex_root= *mutex_in_use))   /* If not first locked */
+      {
+        /*
+          Protect locked_mutex against changes if a mutex is deleted
+        */
+        pthread_mutex_lock(&THR_LOCK_mutex);
+
+        if (!hash_search(mutex_root->locked_mutex, (uchar*) &mp->id, 0))
+        {
+          safe_mutex_deadlock_t *deadlock;
+          safe_mutex_t *mutex;
+
+          /* Create object to store mutex info */
+          if (!(deadlock= my_malloc(sizeof(*deadlock),
+                                    MYF(MY_ZEROFILL | MY_WME | MY_FAE))))
+            goto abort_loop;
+          deadlock->name= mp->name;
+          deadlock->id= mp->id;
+          deadlock->mutex= mp;
+          /* The following is useful for debugging wrong mutex usage */
+          deadlock->file= file;
+          deadlock->line= line;
+
+          /* Check if potential deadlock */
+          mutex= mutex_root;
+          do
+          {
+            if (hash_search(mp->locked_mutex, (uchar*) &mutex->id, 0))
+            {
+              print_deadlock_warning(mp, mutex);
+              /* Mark wrong usage to avoid future warnings for same error */
+              deadlock->warning_only= 1;
+              add_to_locked_mutex(deadlock, mutex_root);
+              DBUG_ASSERT(deadlock->count > 0);
+              goto abort_loop;
+            }
+          }
+          while ((mutex= mutex->next));
+
+          /*
+            Copy current mutex and all mutex that has been locked
+            after current mutex (mp->locked_mutex) to all mutex that
+            was locked before previous mutex (mutex_root->used_mutex)
+
+            For example if A->B would have been done before and we
+            are now locking (C) in B->C, then we would add C into
+            B->locked_mutex and A->locked_mutex
+          */
+          hash_iterate(mutex_root->used_mutex,
+                       (hash_walk_action) add_used_to_locked_mutex,
+                       deadlock);
+
+          /*
+            Copy all current mutex and all mutex locked after current one
+            into the prev mutex
+          */
+          add_used_to_locked_mutex(mutex_root, deadlock);
+          DBUG_ASSERT(deadlock->count > 0);
+        }
+  abort_loop:
+        pthread_mutex_unlock(&THR_LOCK_mutex);
+      }
+      /* Link mutex into mutex_in_use list */
+      if ((mp->next= *mutex_in_use))
+        (*mutex_in_use)->prev= mp;
+      *mutex_in_use= mp;
+    }
+  }
+
   DBUG_PRINT("mutex", ("%s (0x%lx) locked", mp->name, (ulong) mp));
   return error;
 }
@@ -182,7 +341,9 @@ int safe_mutex_unlock(safe_mutex_t *mp,const char *file, uint line)
   pthread_mutex_lock(&mp->global);
   if (mp->count == 0)
   {
-    fprintf(stderr,"safe_mutex: Trying to unlock mutex %s that wasn't locked at %s, line %d\n"
+    fprintf(stderr,
+            "safe_mutex: Trying to unlock mutex %s that wasn't locked at "
+            "%s, line %d\n"
             "Last used at %s, line: %d\n",
 	    mp->name ? mp->name : "Null", file, line,
             mp->file ? mp->file : "Null", mp->line);
@@ -191,7 +352,9 @@ int safe_mutex_unlock(safe_mutex_t *mp,const char *file, uint line)
   }
   if (!pthread_equal(pthread_self(),mp->thread))
   {
-    fprintf(stderr,"safe_mutex: Trying to unlock mutex %s at %s, line %d that was locked by "
+    fprintf(stderr,
+            "safe_mutex: Trying to unlock mutex %s at %s, line %d that was "
+            "locked by "
             "another thread at: %s, line: %d\n",
 	    mp->name, file, line, mp->file, mp->line);
     fflush(stderr);
@@ -199,6 +362,9 @@ int safe_mutex_unlock(safe_mutex_t *mp,const char *file, uint line)
   }
   mp->thread= 0;
   mp->count--;
+
+  remove_from_active_list(mp);
+
 #ifdef __WIN__
   pthread_mutex_unlock(&mp->mutex);
   error=0;
@@ -206,8 +372,9 @@ int safe_mutex_unlock(safe_mutex_t *mp,const char *file, uint line)
   error=pthread_mutex_unlock(&mp->mutex);
   if (error)
   {
-    fprintf(stderr,"safe_mutex: Got error: %d (%d) when trying to unlock mutex %s at %s, "
-            "line %d\n", error, errno, mp->name, file, line);
+    fprintf(stderr,
+            "safe_mutex: Got error: %d (%d) when trying to unlock mutex "
+            "%s at %s, line %d\n", error, errno, mp->name, file, line);
     fflush(stderr);
     abort();
   }
@@ -221,18 +388,23 @@ int safe_cond_wait(pthread_cond_t *cond, safe_mutex_t *mp, const char *file,
 		   uint line)
 {
   int error;
+  safe_mutex_t save_state;
+
   pthread_mutex_lock(&mp->global);
   if (mp->count == 0)
   {
-    fprintf(stderr,"safe_mutex: Trying to cond_wait on a unlocked mutex %s at %s, line %d\n",
+    fprintf(stderr,
+            "safe_mutex: Trying to cond_wait on a unlocked mutex %s at %s, "
+            "line %d\n",
             mp->name ? mp->name : "Null", file, line);
     fflush(stderr);
     abort();
   }
   if (!pthread_equal(pthread_self(),mp->thread))
   {
-    fprintf(stderr,"safe_mutex: Trying to cond_wait on a mutex %s at %s, line %d that was "
-            "locked by another thread at: %s, line: %d\n",
+    fprintf(stderr,
+            "safe_mutex: Trying to cond_wait on a mutex %s at %s, line %d "
+            "that was locked by another thread at: %s, line: %d\n",
 	    mp->name, file, line, mp->file, mp->line);
     fflush(stderr);
     abort();
@@ -240,26 +412,37 @@ int safe_cond_wait(pthread_cond_t *cond, safe_mutex_t *mp, const char *file,
 
   if (mp->count-- != 1)
   {
-    fprintf(stderr,"safe_mutex:  Count was %d on locked mutex %s at %s, line %d\n",
+    fprintf(stderr,
+            "safe_mutex:  Count was %d on locked mutex %s at %s, line %d\n",
 	    mp->count+1, mp->name, file, line);
     fflush(stderr);
     abort();
   }
+  save_state= *mp;
+  remove_from_active_list(mp);
   pthread_mutex_unlock(&mp->global);
   error=pthread_cond_wait(cond,&mp->mutex);
   pthread_mutex_lock(&mp->global);
+
   if (error)
   {
-    fprintf(stderr,"safe_mutex: Got error: %d (%d) when doing a safe_mutex_wait on %s at %s, "
-            "line %d\n", error, errno, mp->name, file, line);
+    fprintf(stderr,
+            "safe_mutex: Got error: %d (%d) when doing a safe_mutex_wait on "
+            "%s at %s, line %d\n", error, errno, mp->name, file, line);
     fflush(stderr);
     abort();
   }
-  mp->thread=pthread_self();
+  /* Restore state as it was before */
+  mp->thread=       save_state.thread;
+  mp->active_flags= save_state.active_flags;
+  mp->next=         save_state.next;
+  mp->prev=         save_state.prev;
+
   if (mp->count++)
   {
     fprintf(stderr,
-	    "safe_mutex:  Count was %d in thread 0x%lx when locking mutex %s at %s, line %d\n",
+	    "safe_mutex:  Count was %d in thread 0x%lx when locking mutex %s "
+            "at %s, line %d\n",
 	    mp->count-1, my_thread_dbug_id(), mp->name, file, line);
     fflush(stderr);
     abort();
@@ -276,33 +459,44 @@ int safe_cond_timedwait(pthread_cond_t *cond, safe_mutex_t *mp,
 			const char *file, uint line)
 {
   int error;
+  safe_mutex_t save_state;
+
   pthread_mutex_lock(&mp->global);
   if (mp->count != 1 || !pthread_equal(pthread_self(),mp->thread))
   {
-    fprintf(stderr,"safe_mutex: Trying to cond_wait at %s, line %d on a not hold mutex %s\n",
+    fprintf(stderr,
+            "safe_mutex: Trying to cond_wait at %s, line %d on a not hold "
+            "mutex %s\n",
             file, line, mp->name ? mp->name : "Null");
     fflush(stderr);
     abort();
   }
   mp->count--;					/* Mutex will be released */
+  save_state= *mp;
+  remove_from_active_list(mp);
   pthread_mutex_unlock(&mp->global);
   error=pthread_cond_timedwait(cond,&mp->mutex,abstime);
 #ifdef EXTRA_DEBUG
   if (error && (error != EINTR && error != ETIMEDOUT && error != ETIME))
   {
     fprintf(stderr,
-            "safe_mutex: Got error: %d (%d) when doing a safe_mutex_timedwait on %s at %s, "
-            "line %d\n",
+            "safe_mutex: Got error: %d (%d) when doing a safe_mutex_timedwait "
+            "on %s at %s, line %d\n",
             error, errno, mp->name, file, line);
   }
 #endif
   pthread_mutex_lock(&mp->global);
-  mp->thread=pthread_self();
+  /* Restore state as it was before */
+  mp->thread=       save_state.thread;
+  mp->active_flags= save_state.active_flags;
+  mp->next=         save_state.next;
+  mp->prev=         save_state.prev;
+
   if (mp->count++)
   {
     fprintf(stderr,
-	    "safe_mutex:  Count was %d in thread 0x%lx when locking mutex %s at %s, line %d "
-            "(error: %d (%d))\n",
+	    "safe_mutex:  Count was %d in thread 0x%lx when locking mutex "
+            "%s at %s, line %d (error: %d (%d))\n",
 	    mp->count-1, my_thread_dbug_id(), mp->name, file, line,
             error, error);
     fflush(stderr);
@@ -318,6 +512,8 @@ int safe_cond_timedwait(pthread_cond_t *cond, safe_mutex_t *mp,
 int safe_mutex_destroy(safe_mutex_t *mp, const char *file, uint line)
 {
   int error=0;
+  DBUG_ENTER("safe_mutex_destroy");
+  DBUG_PRINT("enter", ("mutex: 0x%lx  name: %s", (ulong) mp, mp->name));
   if (!mp->file)
   {
     fprintf(stderr,
@@ -328,12 +524,17 @@ int safe_mutex_destroy(safe_mutex_t *mp, const char *file, uint line)
   }
   if (mp->count != 0)
   {
-    fprintf(stderr,"safe_mutex: Trying to destroy a mutex %s that was locked at %s, "
+    fprintf(stderr,
+            "safe_mutex: Trying to destroy a mutex %s that was locked at %s, "
             "line %d at %s, line %d\n",
 	    mp->name, mp->file, mp->line, file, line);
     fflush(stderr);
     abort();
   }
+
+  /* Free all entries that points to this one */
+  safe_mutex_free_deadlock_data(mp);
+
 #ifdef __WIN__ 
   pthread_mutex_destroy(&mp->global);
   pthread_mutex_destroy(&mp->mutex);
@@ -354,7 +555,7 @@ int safe_mutex_destroy(safe_mutex_t *mp, const char *file, uint line)
     if (info->prev)
       info->prev->next = info->next;
     else
-      safe_mutex_root = info->next;
+      safe_mutex_create_root = info->next;
     if (info->next)
       info->next->prev = info->prev;
     safe_mutex_count--;
@@ -366,10 +567,36 @@ int safe_mutex_destroy(safe_mutex_t *mp, const char *file, uint line)
 #else
   thread_safe_sub(safe_mutex_count, 1, &THR_LOCK_mutex);
 #endif /* SAFE_MUTEX_DETECT_DESTROY */
-  return error;
+  DBUG_RETURN(error);
 }
 
 
+/**
+  Free all data related to deadlock detection
+
+  This is also useful together with safemalloc when you don't want to
+  have reports of not freed memory for mysys mutexes.
+*/
+
+void safe_mutex_free_deadlock_data(safe_mutex_t *mp)
+{
+  /* Free all entries that points to this one */
+  if (!(mp->create_flags & MYF_NO_DEADLOCK_DETECTION))
+  {
+    pthread_mutex_lock(&THR_LOCK_mutex);
+    hash_iterate(mp->used_mutex, (hash_walk_action) remove_from_locked_mutex,
+                 mp);
+    hash_iterate(mp->locked_mutex, (hash_walk_action) remove_from_used_mutex,
+                 mp);
+    pthread_mutex_unlock(&THR_LOCK_mutex);
+
+    hash_free(mp->used_mutex);
+    hash_free(mp->locked_mutex);
+    my_free(mp->locked_mutex, 0);
+    mp->create_flags|= MYF_NO_DEADLOCK_DETECTION;
+  }
+}
+
 /*
   Free global resources and check that all mutex has been destroyed
 
@@ -400,7 +627,7 @@ void safe_mutex_end(FILE *file __attribute__((unused)))
   }
   {
     struct st_safe_mutex_info_t *ptr;
-    for (ptr= safe_mutex_root ; ptr ; ptr= ptr->next)
+    for (ptr= safe_mutex_create_root ; ptr ; ptr= ptr->next)
     {
       fprintf(file, "\tMutex %s initiated at line %4u in '%s'\n",
 	      ptr->name, ptr->init_line, ptr->init_file);
@@ -410,6 +637,127 @@ void safe_mutex_end(FILE *file __attribute__((unused)))
 #endif /* SAFE_MUTEX_DETECT_DESTROY */
 }
 
+
+static my_bool add_used_to_locked_mutex(safe_mutex_t *used_mutex,
+                                        safe_mutex_deadlock_t *locked_mutex)
+{
+  /* Add mutex to all parent of the current mutex */
+  if (!locked_mutex->warning_only)
+  {
+    (void) hash_iterate(locked_mutex->mutex->locked_mutex,
+                        (hash_walk_action) add_to_locked_mutex,
+                        used_mutex);
+    /* mark that locked_mutex is locked after used_mutex */
+    (void) add_to_locked_mutex(locked_mutex, used_mutex);
+  }
+  return 0;
+}
+
+
+/**
+   register that locked_mutex was locked after current_mutex
+*/
+
+static my_bool add_to_locked_mutex(safe_mutex_deadlock_t *locked_mutex,
+                                   safe_mutex_t *current_mutex)
+{
+  DBUG_ENTER("add_to_locked_mutex");
+  DBUG_PRINT("info", ("inserting 0x%lx  into  0x%lx  (id: %lu -> %lu)",
+                      (ulong) locked_mutex, (long) current_mutex,
+                      locked_mutex->id, current_mutex->id));
+  if (my_hash_insert(current_mutex->locked_mutex, (uchar*) locked_mutex))
+  {
+    /* Got mutex through two paths; ignore */
+    DBUG_RETURN(0);
+  }
+  locked_mutex->count++;
+  if (my_hash_insert(locked_mutex->mutex->used_mutex,
+                     (uchar*) current_mutex))
+  {
+    DBUG_ASSERT(0);
+  }
+  DBUG_RETURN(0);
+}
+
+
+/**
+  Remove mutex from the locked mutex hash
+  @fn    remove_from_used_mutex()
+  @param mp            Mutex that has delete_mutex in it's locked_mutex hash
+  @param delete_mutex  Mutex should be removed from the hash
+
+  @notes
+    safe_mutex_deadlock_t entries in the locked hash are shared.
+    When counter goes to 0, we delete the safe_mutex_deadlock_t entry.
+*/
+
+static my_bool remove_from_locked_mutex(safe_mutex_t *mp,
+                                        safe_mutex_t *delete_mutex)
+{
+  safe_mutex_deadlock_t *found;
+  DBUG_ENTER("remove_from_locked_mutex");
+  DBUG_PRINT("enter", ("delete_mutex: 0x%lx  mutex: 0x%lx  (id: %lu <- %lu)",
+                       (ulong) delete_mutex, (ulong) mp, 
+                       delete_mutex->id, mp->id));
+
+  found= (safe_mutex_deadlock_t *) hash_search(mp->locked_mutex,
+                                               (uchar*) &delete_mutex->id, 0);
+  DBUG_ASSERT(found);
+  if (found)
+  {
+    if (hash_delete(mp->locked_mutex, (uchar*) found))
+    {
+      DBUG_ASSERT(0);
+    }
+    if (!--found->count)
+      my_free(found, MYF(0));
+  }
+  DBUG_RETURN(0);
+}
+
+static my_bool remove_from_used_mutex(safe_mutex_deadlock_t *locked_mutex,
+                                      safe_mutex_t *mutex)
+{
+  DBUG_ENTER("remove_from_used_mutex");
+  DBUG_PRINT("enter", ("delete_mutex: 0x%lx  mutex: 0x%lx  (id: %lu <- %lu)",
+                       (ulong) mutex, (ulong) locked_mutex, 
+                       mutex->id, locked_mutex->id));
+  if (hash_delete(locked_mutex->mutex->used_mutex, (uchar*) mutex))
+  {
+    DBUG_ASSERT(0);
+  }
+  if (!--locked_mutex->count)
+    my_free(locked_mutex, MYF(0));
+  DBUG_RETURN(0);
+}
+
+
+static void print_deadlock_warning(safe_mutex_t *new_mutex,
+                                   safe_mutex_t *parent_mutex)
+{
+  safe_mutex_t *mutex_root;
+  DBUG_ENTER("print_deadlock_warning");
+  DBUG_PRINT("enter", ("mutex: %s  parent: %s",
+                       new_mutex->name, parent_mutex->name));
+
+  fprintf(stderr, "safe_mutex: Found wrong usage of mutex "
+          "'%s' and '%s'\n",
+          parent_mutex->name, new_mutex->name);
+  fprintf(stderr, "Mutex currently locked (in reverse order):\n");
+  fprintf(stderr, "%-32.32s  %s  line %u\n", new_mutex->name, new_mutex->file,
+          new_mutex->line);
+  for (mutex_root= *my_thread_var_mutex_in_use() ;
+       mutex_root;
+       mutex_root= mutex_root->next)
+  {
+    fprintf(stderr, "%-32.32s  %s  line %u\n", mutex_root->name,
+            mutex_root->file, mutex_root->line);
+  }
+  fflush(stderr);
+  DBUG_VOID_RETURN;
+}
+
+
 #endif /* THREAD && SAFE_MUTEX */
 
 #if defined(THREAD) && defined(MY_PTHREAD_FASTMUTEX) && !defined(SAFE_MUTEX)
diff --git a/sql/event_queue.cc b/sql/event_queue.cc
index 04d4f858b43..d68dc8ef479 100644
--- a/sql/event_queue.cc
+++ b/sql/event_queue.cc
@@ -94,7 +94,12 @@ Event_queue::Event_queue()
    mutex_queue_data_attempting_lock(FALSE),
    waiting_on_cond(FALSE)
 {
-  pthread_mutex_init(&LOCK_event_queue, MY_MUTEX_INIT_FAST);
+  /*
+    Inconsisent usage between LOCK_event_queue and LOCK_scheduler_state and
+    LOCK_open
+  */
+  my_pthread_mutex_init(&LOCK_event_queue, MY_MUTEX_INIT_FAST,
+                        "LOCK_event_queue", MYF_NO_DEADLOCK_DETECTION);
   pthread_cond_init(&COND_queue_state, NULL);
 }
 
diff --git a/sql/event_scheduler.cc b/sql/event_scheduler.cc
index 5655a8acc99..d3e51d40822 100644
--- a/sql/event_scheduler.cc
+++ b/sql/event_scheduler.cc
@@ -350,6 +350,14 @@ Event_scheduler::Event_scheduler(Event_queue *queue_arg)
 {
   pthread_mutex_init(&LOCK_scheduler_state, MY_MUTEX_INIT_FAST);
   pthread_cond_init(&COND_state, NULL);
+
+#ifdef SAFE_MUTEX
+  /* Ensure right mutex order */
+  pthread_mutex_lock(&LOCK_scheduler_state);
+  pthread_mutex_lock(&LOCK_global_system_variables);
+  pthread_mutex_unlock(&LOCK_global_system_variables);
+  pthread_mutex_unlock(&LOCK_scheduler_state);
+#endif
 }
 
 
diff --git a/sql/events.cc b/sql/events.cc
index df49b7db773..73c76688eb7 100644
--- a/sql/events.cc
+++ b/sql/events.cc
@@ -995,7 +995,12 @@ Events::deinit()
 void
 Events::init_mutexes()
 {
-  pthread_mutex_init(&LOCK_event_metadata, MY_MUTEX_INIT_FAST);
+  /*
+    Inconsisent usage between LOCK_event_metadata and LOCK_scheduler_state
+    and LOCK_open
+  */
+  my_pthread_mutex_init(&LOCK_event_metadata, MY_MUTEX_INIT_FAST,
+                        "LOCK_event_metadata", MYF_NO_DEADLOCK_DETECTION);
 }
 
 
diff --git a/sql/ha_ndbcluster_binlog.cc b/sql/ha_ndbcluster_binlog.cc
index 4fd5ee1b402..1788fed26b1 100644
--- a/sql/ha_ndbcluster_binlog.cc
+++ b/sql/ha_ndbcluster_binlog.cc
@@ -1550,7 +1550,9 @@ end:
       dict->forceGCPWait();
 
     int max_timeout= opt_ndb_sync_timeout;
-    (void) pthread_mutex_lock(&ndb_schema_object->mutex);
+    /* Inconsistent usage of ndb_schema_object->mutex and LOCK_open */
+    (void) my_pthread_mutex_lock(&ndb_schema_object->mutex,
+                                 MYF_NO_DEADLOCK_DETECTION);
     if (have_lock_open)
     {
       safe_mutex_assert_owner(&LOCK_open);
diff --git a/sql/log.cc b/sql/log.cc
index b6fe1196835..4c5acf83bc4 100644
--- a/sql/log.cc
+++ b/sql/log.cc
@@ -2403,7 +2403,12 @@ void MYSQL_BIN_LOG::init_pthread_objects()
   DBUG_ASSERT(inited == 0);
   inited= 1;
   (void) pthread_mutex_init(&LOCK_log, MY_MUTEX_INIT_SLOW);
-  (void) pthread_mutex_init(&LOCK_index, MY_MUTEX_INIT_SLOW);
+  /*
+    LOCK_index and LOCK_log are taken in wrong order
+    Can be seen with 'mysql-test-run ndb.ndb_binlog_basic'
+  */ 
+  (void) my_pthread_mutex_init(&LOCK_index, MY_MUTEX_INIT_SLOW, "LOCK_index",
+                               MYF_NO_DEADLOCK_DETECTION);
   (void) pthread_cond_init(&update_cond, 0);
 }
 
diff --git a/sql/mysqld.cc b/sql/mysqld.cc
index b1ec3eb1e3a..8415342b3df 100644
--- a/sql/mysqld.cc
+++ b/sql/mysqld.cc
@@ -757,6 +757,7 @@ static ulong find_bit_type_or_exit(const char *x, TYPELIB *bit_lib,
                                    const char *option);
 static void clean_up(bool print_message);
 static int test_if_case_insensitive(const char *dir_name);
+static void register_mutex_order();
 
 #ifndef EMBEDDED_LIBRARY
 static void usage(void);
@@ -902,9 +903,19 @@ static void close_connections(void)
       pthread_mutex_lock(&tmp->mysys_var->mutex);
       if (tmp->mysys_var->current_cond)
       {
-	pthread_mutex_lock(tmp->mysys_var->current_mutex);
-	pthread_cond_broadcast(tmp->mysys_var->current_cond);
-	pthread_mutex_unlock(tmp->mysys_var->current_mutex);
+        uint i;
+        for (i=0; i < 2; i++)
+        {
+          int ret= pthread_mutex_trylock(tmp->mysys_var->current_mutex);
+          pthread_cond_broadcast(tmp->mysys_var->current_cond);
+          if (!ret)
+          {
+            /* Thread has surely got the signal, unlock and abort */
+            pthread_mutex_unlock(tmp->mysys_var->current_mutex);
+            break;
+          }
+          sleep(1);
+        }
       }
       pthread_mutex_unlock(&tmp->mysys_var->mutex);
     }
@@ -1244,6 +1255,7 @@ void clean_up(bool print_message)
   wt_end();
   delete_elements(&key_caches, (void (*)(const char*, uchar*)) free_key_cache);
   multi_keycache_free();
+  sp_cache_end();
   free_status_vars();
   end_thr_alarm(1);			/* Free allocated memory */
   my_free_open_file_info();
@@ -1336,6 +1348,7 @@ static void wait_for_signal_thread_to_end()
 
 static void clean_up_mutexes()
 {
+  DBUG_ENTER("clean_up_mutexes");
   (void) pthread_mutex_destroy(&LOCK_mysql_create_db);
   (void) pthread_mutex_destroy(&LOCK_lock_db);
   (void) pthread_mutex_destroy(&LOCK_Acl);
@@ -1367,6 +1380,8 @@ static void clean_up_mutexes()
   (void) pthread_mutex_destroy(&LOCK_rpl_status);
   (void) pthread_cond_destroy(&COND_rpl_status);
 #endif
+  (void) pthread_mutex_destroy(&LOCK_server_started);
+  (void) pthread_cond_destroy(&COND_server_started);
   (void) pthread_mutex_destroy(&LOCK_active_mi);
   (void) rwlock_destroy(&LOCK_sys_init_connect);
   (void) rwlock_destroy(&LOCK_sys_init_slave);
@@ -1381,11 +1396,35 @@ static void clean_up_mutexes()
   (void) pthread_cond_destroy(&COND_thread_cache);
   (void) pthread_cond_destroy(&COND_flush_thread_cache);
   (void) pthread_cond_destroy(&COND_manager);
+  DBUG_VOID_RETURN;
 }
 
 #endif /*EMBEDDED_LIBRARY*/
 
 
+/**
+   Register order of mutex for wrong mutex deadlock detector
+
+   By aquiring all mutex in order here, the mutex order detector in
+   mysys/thr_mutex.c, will give a warning on first wrong mutex usage!
+*/
+
+static void register_mutex_order()
+{
+#ifdef SAFE_MUTEX
+  /*
+    We must have LOCK_open before LOCK_global_system_variables because
+    LOCK_open is hold while sql_plugin.c::intern_sys_var_ptr() is called.
+  */
+  pthread_mutex_lock(&LOCK_open);
+  pthread_mutex_lock(&LOCK_global_system_variables);
+
+  pthread_mutex_unlock(&LOCK_global_system_variables);
+  pthread_mutex_unlock(&LOCK_open);
+#endif
+}
+
+
 /****************************************************************************
 ** Init IP and UNIX socket
 ****************************************************************************/
@@ -3549,6 +3588,7 @@ static int init_thread_environment()
     sql_print_error("Can't create thread-keys");
     return 1;
   }
+  register_mutex_order();
   return 0;
 }
 
@@ -5464,7 +5504,7 @@ enum options_mysqld
   OPT_NDB_REPORT_THRESH_BINLOG_EPOCH_SLIP,
   OPT_NDB_REPORT_THRESH_BINLOG_MEM_USAGE,
   OPT_NDB_USE_COPYING_ALTER_TABLE,
-  OPT_SKIP_SAFEMALLOC,
+  OPT_SKIP_SAFEMALLOC, OPT_MUTEX_DEADLOCK_DETECTOR,
   OPT_TEMP_POOL, OPT_TX_ISOLATION, OPT_COMPLETION_TYPE,
   OPT_SKIP_STACK_TRACE, OPT_SKIP_SYMLINKS,
   OPT_MAX_BINLOG_DUMP_EVENTS, OPT_SPORADIC_BINLOG_DUMP_FAIL,
@@ -6002,6 +6042,13 @@ master-ssl",
 #endif /* HAVE_REPLICATION */
   {"memlock", OPT_MEMLOCK, "Lock mysqld in memory.", (uchar**) &locked_in_memory,
    (uchar**) &locked_in_memory, 0, GET_BOOL, NO_ARG, 0, 0, 0, 0, 0, 0},
+#ifdef SAFE_MUTEX
+  {"mutex-deadlock-detector", OPT_MUTEX_DEADLOCK_DETECTOR,
+   "Enable checking of wrong mutex usage.",
+   (uchar**) &safe_mutex_deadlock_detector,
+   (uchar**) &safe_mutex_deadlock_detector,
+   0, GET_BOOL, NO_ARG, 1, 0, 0, 0, 0, 0},
+#endif
   {"myisam-recover", OPT_MYISAM_RECOVER,
    "Syntax: myisam-recover[=option[,option...]], where option can be DEFAULT, BACKUP, FORCE or QUICK.",
    (uchar**) &myisam_recover_options_str, (uchar**) &myisam_recover_options_str, 0,
diff --git a/sql/protocol.cc b/sql/protocol.cc
index 3eccc6632ce..bb69cdb36af 100644
--- a/sql/protocol.cc
+++ b/sql/protocol.cc
@@ -790,8 +790,8 @@ bool Protocol_text::store(const char *from, size_t length,
 {
   CHARSET_INFO *tocs= this->thd->variables.character_set_results;
 #ifndef DBUG_OFF
-  DBUG_PRINT("info", ("Protocol_text::store field %u (%u): %s", field_pos,
-                      field_count, from));
+  DBUG_PRINT("info", ("Protocol_text::store field %u (%u): %*s", field_pos,
+                      field_count, (int) length, from));
   DBUG_ASSERT(field_pos < field_count);
   DBUG_ASSERT(field_types == 0 ||
 	      field_types[field_pos] == MYSQL_TYPE_DECIMAL ||
diff --git a/sql/rpl_mi.cc b/sql/rpl_mi.cc
index 5e46837e948..cb8b0e02ef9 100644
--- a/sql/rpl_mi.cc
+++ b/sql/rpl_mi.cc
@@ -38,11 +38,26 @@ Master_info::Master_info()
   ssl_cipher[0]= 0; ssl_key[0]= 0;
 
   bzero((char*) &file, sizeof(file));
-  pthread_mutex_init(&run_lock, MY_MUTEX_INIT_FAST);
-  pthread_mutex_init(&data_lock, MY_MUTEX_INIT_FAST);
+  /*
+    We have to use MYF_NO_DEADLOCK_DETECTION because mysqld doesn't
+    lock run_lock and data_lock consistently.
+    Should be fixed as this can easily lead to deadlocks
+  */
+  my_pthread_mutex_init(&run_lock, MY_MUTEX_INIT_FAST, 
+                        "Master_info::run_lock", MYF_NO_DEADLOCK_DETECTION);
+  my_pthread_mutex_init(&data_lock, MY_MUTEX_INIT_FAST,
+                        "Master_info::data_lock", MYF_NO_DEADLOCK_DETECTION);
   pthread_cond_init(&data_cond, NULL);
   pthread_cond_init(&start_cond, NULL);
   pthread_cond_init(&stop_cond, NULL);
+
+#ifdef SAFE_MUTEX
+  /* Define mutex order for locks to find wrong lock usage */
+  pthread_mutex_lock(&data_lock);
+  pthread_mutex_lock(&run_lock);
+  pthread_mutex_unlock(&run_lock);
+  pthread_mutex_unlock(&data_lock);
+#endif
 }
 
 Master_info::~Master_info()
diff --git a/sql/set_var.cc b/sql/set_var.cc
index df1badebc50..074835107d9 100644
--- a/sql/set_var.cc
+++ b/sql/set_var.cc
@@ -2403,7 +2403,6 @@ end:
 bool sys_var_log_state::update(THD *thd, set_var *var)
 {
   bool res;
-  pthread_mutex_lock(&LOCK_global_system_variables);
   if (!var->save_result.ulong_value)
   {
     logger.deactivate_log_handler(thd, log_type);
@@ -2411,15 +2410,12 @@ bool sys_var_log_state::update(THD *thd, set_var *var)
   }
   else
     res= logger.activate_log_handler(thd, log_type);
-  pthread_mutex_unlock(&LOCK_global_system_variables);
   return res;
 }
 
 void sys_var_log_state::set_default(THD *thd, enum_var_type type)
 {
-  pthread_mutex_lock(&LOCK_global_system_variables);
   logger.deactivate_log_handler(thd, log_type);
-  pthread_mutex_unlock(&LOCK_global_system_variables);
 }
 
 
@@ -2515,23 +2511,18 @@ bool update_sys_var_str_path(THD *thd, sys_var_str *var_str,
     goto err;
   }
 
-  pthread_mutex_lock(&LOCK_global_system_variables);
   logger.lock_exclusive();
 
   if (file_log && log_state)
     file_log->close(0);
-  old_value= var_str->value;
-  var_str->value= res;
-  var_str->value_length= str_length;
-  my_free(old_value, MYF(MY_ALLOW_ZERO_PTR));
   if (file_log && log_state)
   {
     switch (log_type) {
     case QUERY_LOG_SLOW:
-      file_log->open_slow_log(sys_var_slow_log_path.value);
+      file_log->open_slow_log(res);
       break;
     case QUERY_LOG_GENERAL:
-      file_log->open_query_log(sys_var_general_log_path.value);
+      file_log->open_query_log(res);
       break;
     default:
       DBUG_ASSERT(0);
@@ -2539,6 +2530,13 @@ bool update_sys_var_str_path(THD *thd, sys_var_str *var_str,
   }
 
   logger.unlock();
+
+  /* update global variable */
+  pthread_mutex_lock(&LOCK_global_system_variables);
+  old_value= var_str->value;
+  var_str->value= res;
+  var_str->value_length= str_length;
+  my_free(old_value, MYF(MY_ALLOW_ZERO_PTR));
   pthread_mutex_unlock(&LOCK_global_system_variables);
 
 err:
@@ -2578,26 +2576,22 @@ static void sys_default_slow_log_path(THD *thd, enum_var_type type)
 
 bool sys_var_log_output::update(THD *thd, set_var *var)
 {
-  pthread_mutex_lock(&LOCK_global_system_variables);
   logger.lock_exclusive();
   logger.init_slow_log(var->save_result.ulong_value);
   logger.init_general_log(var->save_result.ulong_value);
   *value= var->save_result.ulong_value;
   logger.unlock();
-  pthread_mutex_unlock(&LOCK_global_system_variables);
   return 0;
 }
 
 
 void sys_var_log_output::set_default(THD *thd, enum_var_type type)
 {
-  pthread_mutex_lock(&LOCK_global_system_variables);
   logger.lock_exclusive();
   logger.init_slow_log(LOG_FILE);
   logger.init_general_log(LOG_FILE);
   *value= LOG_FILE;
   logger.unlock();
-  pthread_mutex_unlock(&LOCK_global_system_variables);
 }
 
 
diff --git a/sql/sp_cache.cc b/sql/sp_cache.cc
index 64898915b7e..13b3e771a91 100644
--- a/sql/sp_cache.cc
+++ b/sql/sp_cache.cc
@@ -106,6 +106,12 @@ void sp_cache_clear(sp_cache **cp)
 }
 
 
+void sp_cache_end()
+{
+  pthread_mutex_destroy(&Cversion_lock);
+}
+
+
 /*
   Insert a routine into the cache.
 
diff --git a/sql/sp_cache.h b/sql/sp_cache.h
index f4d44a1f29f..efb61d76719 100644
--- a/sql/sp_cache.h
+++ b/sql/sp_cache.h
@@ -53,6 +53,7 @@ class sp_cache;
 */
 
 void sp_cache_init();
+void sp_cache_end();
 void sp_cache_clear(sp_cache **cp);
 void sp_cache_insert(sp_cache **cp, sp_head *sp);
 sp_head *sp_cache_lookup(sp_cache **cp, sp_name *name);
diff --git a/sql/sql_class.cc b/sql/sql_class.cc
index 9a450f245be..3504b4fdea5 100644
--- a/sql/sql_class.cc
+++ b/sql/sql_class.cc
@@ -978,6 +978,14 @@ void add_diff_to_status(STATUS_VAR *to_var, STATUS_VAR *from_var,
     *(to++)+= *(from++) - *(dec++);
 }
 
+#define SECONDS_TO_WAIT_FOR_KILL 2
+#if !defined(__WIN__) && defined(HAVE_SELECT)
+/* my_sleep() can wait for sub second times */
+#define WAIT_FOR_KILL_TRY_TIMES 20
+#else
+#define WAIT_FOR_KILL_TRY_TIMES 2
+#endif
+
 
 void THD::awake(THD::killed_state state_to_set)
 {
@@ -1032,12 +1040,35 @@ void THD::awake(THD::killed_state state_to_set)
       we issue a second KILL or the status it's waiting for happens).
       It's true that we have set its thd->killed but it may not
       see it immediately and so may have time to reach the cond_wait().
+
+      We have to do the loop with trylock, because if we would use
+      pthread_mutex_lock(), we can cause a deadlock as we are here locking
+      the mysys_var->mutex and mysys_var->current_mutex in a different order
+      than in the thread we are trying to kill.
+      We only sleep for 2 seconds as we don't want to have LOCK_delete
+      locked too long time.
+
+      There is a small change we may not succeed in aborting a thread that
+      is not yet waiting for a mutex, but as this happens only for a
+      thread that was doing something else when the kill was issued and
+      which should detect the kill flag before it starts to wait, this
+      should be good enough.
     */
     if (mysys_var->current_cond && mysys_var->current_mutex)
     {
-      pthread_mutex_lock(mysys_var->current_mutex);
-      pthread_cond_broadcast(mysys_var->current_cond);
-      pthread_mutex_unlock(mysys_var->current_mutex);
+      uint i;
+      for (i= 0; i < WAIT_FOR_KILL_TRY_TIMES * SECONDS_TO_WAIT_FOR_KILL; i++)
+      {
+        int ret= pthread_mutex_trylock(mysys_var->current_mutex);
+        pthread_cond_broadcast(mysys_var->current_cond);
+        if (!ret)
+        {
+          /* Signal is sure to get through */
+          pthread_mutex_unlock(mysys_var->current_mutex);
+          break;
+        }
+      }
+      my_sleep(1000000L / WAIT_FOR_KILL_TRY_TIMES);
     }
     pthread_mutex_unlock(&mysys_var->mutex);
   }
@@ -1073,6 +1104,15 @@ bool THD::store_globals()
     created in another thread
   */
   thr_lock_info_init(&lock_info);
+
+#ifdef SAFE_MUTEX
+  /* Register order of mutex for wrong mutex deadlock detector */
+  pthread_mutex_lock(&LOCK_delete);
+  pthread_mutex_lock(&mysys_var->mutex);
+
+  pthread_mutex_unlock(&mysys_var->mutex);
+  pthread_mutex_unlock(&LOCK_delete);
+#endif
   return 0;
 }
 
diff --git a/sql/sql_insert.cc b/sql/sql_insert.cc
index a275c680cb5..d8838b3b03d 100644
--- a/sql/sql_insert.cc
+++ b/sql/sql_insert.cc
@@ -1719,7 +1719,8 @@ public:
     thd.system_thread= SYSTEM_THREAD_DELAYED_INSERT;
     thd.security_ctx->host_or_ip= "";
     bzero((char*) &info,sizeof(info));
-    pthread_mutex_init(&mutex,MY_MUTEX_INIT_FAST);
+    my_pthread_mutex_init(&mutex, MY_MUTEX_INIT_FAST, "Delayed_insert::mutex",
+                          0);
     pthread_cond_init(&cond,NULL);
     pthread_cond_init(&cond_client,NULL);
     VOID(pthread_mutex_lock(&LOCK_thread_count));
@@ -2224,7 +2225,8 @@ void kill_delayed_threads(void)
 	  in handle_delayed_insert()
 	*/
 	if (&di->mutex != di->thd.mysys_var->current_mutex)
-	  pthread_mutex_lock(di->thd.mysys_var->current_mutex);
+	  my_pthread_mutex_lock(di->thd.mysys_var->current_mutex, 
+                                MYF_NO_DEADLOCK_DETECTION);
 	pthread_cond_broadcast(di->thd.mysys_var->current_cond);
 	if (&di->mutex != di->thd.mysys_var->current_mutex)
 	  pthread_mutex_unlock(di->thd.mysys_var->current_mutex);
@@ -2470,13 +2472,14 @@ end:
     clients
   */
 
-  close_thread_tables(thd);			// Free the table
   di->table=0;
   di->dead= 1;                                  // If error
   thd->killed= THD::KILL_CONNECTION;	        // If error
-  pthread_cond_broadcast(&di->cond_client);	// Safety
   pthread_mutex_unlock(&di->mutex);
 
+  close_thread_tables(thd);			// Free the table
+  pthread_cond_broadcast(&di->cond_client);	// Safety
+
   pthread_mutex_lock(&LOCK_delayed_create);	// Because of delayed_get_table
   pthread_mutex_lock(&LOCK_delayed_insert);	
   delete di;
diff --git a/sql/sql_show.cc b/sql/sql_show.cc
index b5ab6484d12..6e321af9292 100644
--- a/sql/sql_show.cc
+++ b/sql/sql_show.cc
@@ -2102,7 +2102,6 @@ static bool show_status_array(THD *thd, const char *wild,
         const char *pos, *end;                  // We assign a lot of const's
 
         pthread_mutex_lock(&LOCK_global_system_variables);
-
         if (show_type == SHOW_SYS)
         {
           show_type= ((sys_var*) value)->show_type();
@@ -2183,14 +2182,14 @@ static bool show_status_array(THD *thd, const char *wild,
           DBUG_ASSERT(0);
           break;
         }
+        pthread_mutex_unlock(&LOCK_global_system_variables);
+
         restore_record(table, s->default_values);
         table->field[0]->store(name_buffer, strlen(name_buffer),
                                system_charset_info);
         table->field[1]->store(pos, (uint32) (end - pos), system_charset_info);
         table->field[1]->set_notnull();
 
-        pthread_mutex_unlock(&LOCK_global_system_variables);
-
         if (schema_table_store_record(thd, table))
           DBUG_RETURN(TRUE);
       }
diff --git a/storage/innobase/handler/ha_innodb.cc b/storage/innobase/handler/ha_innodb.cc
index 887acacbd1f..9c2b71094b6 100644
--- a/storage/innobase/handler/ha_innodb.cc
+++ b/storage/innobase/handler/ha_innodb.cc
@@ -1659,8 +1659,15 @@ innobase_init(
 
 	srv_sizeof_trx_t_in_ha_innodb_cc = sizeof(trx_t);
 
+#ifdef SAFE_MUTEX
+        /* Disable deadlock detection as it's very slow for the buffer pool */
+        my_bool old_safe_mutex_deadlock_detector;
+        safe_mutex_deadlock_detector= 0;
+#endif
 	err = innobase_start_or_create_for_mysql();
-
+#ifdef SAFE_MUTEX
+        safe_mutex_deadlock_detector= old_safe_mutex_deadlock_detector;
+#endif
 	if (err != DB_SUCCESS) {
 		my_free(internal_innobase_data_file_path,
 						MYF(MY_ALLOW_ZERO_PTR));
diff --git a/storage/maria/ha_maria.cc b/storage/maria/ha_maria.cc
index fa2edb24791..3a32ef8cce2 100644
--- a/storage/maria/ha_maria.cc
+++ b/storage/maria/ha_maria.cc
@@ -1460,7 +1460,7 @@ int ha_maria::repair(THD *thd, HA_CHECK *param, bool do_optimize)
                                      (local_testflag &
                                       T_STATISTICS ? UPDATE_STAT : 0));
     info(HA_STATUS_NO_LOCK | HA_STATUS_TIME | HA_STATUS_VARIABLE |
-         HA_STATUS_CONST);
+         HA_STATUS_CONST, 0);
     if (rows != file->state->records && !(param->testflag & T_VERY_SILENT))
     {
       char llbuff[22], llbuff2[22];
@@ -2148,6 +2148,11 @@ void ha_maria::position(const uchar *record)
 
 int ha_maria::info(uint flag)
 {
+  return info(flag, table->s->tmp_table == NO_TMP_TABLE);
+}
+
+int ha_maria::info(uint flag, my_bool lock_table_share)
+{
   MARIA_INFO maria_info;
   char name_buff[FN_REFLEN];
 
@@ -2173,7 +2178,7 @@ int ha_maria::info(uint flag)
     stats.block_size= maria_block_size;
 
     /* Update share */
-    if (share->tmp_table == NO_TMP_TABLE)
+    if (lock_table_share)
       pthread_mutex_lock(&share->mutex);
     share->keys_in_use.set_prefix(share->keys);
     share->keys_in_use.intersect_extended(maria_info.key_map);
@@ -2186,7 +2191,7 @@ int ha_maria::info(uint flag)
       for (end= to+ share->key_parts ; to < end ; to++, from++)
         *to= (ulong) (*from + 0.5);
     }
-    if (share->tmp_table == NO_TMP_TABLE)
+    if (lock_table_share)
       pthread_mutex_unlock(&share->mutex);
 
     /*
diff --git a/storage/maria/ha_maria.h b/storage/maria/ha_maria.h
index ba3ea39b92a..23fb74f1c27 100644
--- a/storage/maria/ha_maria.h
+++ b/storage/maria/ha_maria.h
@@ -112,6 +112,7 @@ public:
   int restart_rnd_next(uchar * buf);
   void position(const uchar * record);
   int info(uint);
+  int info(uint, my_bool);
   int extra(enum ha_extra_function operation);
   int extra_opt(enum ha_extra_function operation, ulong cache_size);
   int reset(void);
diff --git a/storage/maria/ma_close.c b/storage/maria/ma_close.c
index 9463ad8078d..aa4a2f28d9f 100644
--- a/storage/maria/ma_close.c
+++ b/storage/maria/ma_close.c
@@ -114,6 +114,7 @@ int maria_close(register MARIA_HA *info)
     }
 #ifdef THREAD
     thr_lock_delete(&share->lock);
+    (void) pthread_mutex_destroy(&share->key_del_lock);
     {
       int i,keys;
       keys = share->state.header.keys;
@@ -162,14 +163,10 @@ int maria_close(register MARIA_HA *info)
   pthread_mutex_unlock(&share->intern_lock);
   if (share_can_be_freed)
   {
-    VOID(pthread_mutex_destroy(&share->intern_lock));
+    (void) pthread_mutex_destroy(&share->intern_lock);
     my_free((uchar *)share, MYF(0));
   }
-  if (info->ftparser_param)
-  {
-    my_free((uchar*)info->ftparser_param, MYF(0));
-    info->ftparser_param= 0;
-  }
+  my_free(info->ftparser_param, MYF(MY_ALLOW_ZERO_PTR));
   if (info->dfile.file >= 0)
   {
     /*
diff --git a/storage/maria/ma_key.c b/storage/maria/ma_key.c
index 729d3cbc6de..08702a9109e 100644
--- a/storage/maria/ma_key.c
+++ b/storage/maria/ma_key.c
@@ -67,14 +67,16 @@ static int _ma_put_key_in_record(MARIA_HA *info,uint keynr,uchar *record);
   Prefix bytes 244 to 249 are reserved for negative transid, that can be used
   when we pack transid relative to each other on a key block.
 
-  We have to store transid in high-byte-first order to be able to do a
-  fast byte-per-byte comparision of them without packing them up.
+  We have to store transid in high-byte-first order so that we can compare
+  them unpacked byte per byte and as soon we find a difference we know
+  which is smaller.
 
   For example, assuming we the following data:
 
   key_data:               1                (4 byte integer)
   pointer_to_row:         2 << 8 + 3 = 515 (page 2, row 3)
-  table_create_transid    1000             Defined at create table time
+  table_create_transid    1000             Defined at create table time and
+                                           stored in table definition
   transid                 1010	           Transaction that created row
   delete_transid          2011             Transaction that deleted row
 
@@ -86,8 +88,9 @@ static int _ma_put_key_in_record(MARIA_HA *info,uint keynr,uchar *record);
 
   00 00 00 01     Key data (1 stored high byte first)
   00 00 00 47	  (515 << 1) + 1         ;  The last 1 is marker that key cont.
-  15              ((1000-1010) << 1) + 1 ;  The last 1 is marker that key cont.
-  FB 07 E6        length byte and  ((2011 - 1000) << 1) = 07 E6
+  15              ((1010-1000) << 1) + 1 ;  The last 1 is marker that key cont.
+  FB 07 E6        Length byte (FE = 249 + 2 means 2 bytes) and 
+                  ((2011 - 1000) << 1) = 07 E6
 */
 
 uint transid_store_packed(MARIA_HA *info, uchar *to, ulonglong trid)
diff --git a/storage/maria/ma_loghandler.c b/storage/maria/ma_loghandler.c
index e314363b1d2..cc8c7898d84 100644
--- a/storage/maria/ma_loghandler.c
+++ b/storage/maria/ma_loghandler.c
@@ -1431,7 +1431,9 @@ static my_bool translog_buffer_init(struct st_translog_buffer *buffer)
   /* list of waiting buffer ready threads */
   buffer->waiting_flush= 0;
   /* lock for the buffer. Current buffer also lock the handler */
-  if (pthread_mutex_init(&buffer->mutex, MY_MUTEX_INIT_FAST) ||
+  if (my_pthread_mutex_init(&buffer->mutex, MY_MUTEX_INIT_FAST,
+                            "translog_buffer->mutex",
+                            MYF_NO_DEADLOCK_DETECTION) ||
       pthread_cond_init(&buffer->prev_sent_to_disk_cond, 0))
     DBUG_RETURN(1);
   buffer->is_closing_buffer= 0;
diff --git a/storage/maria/ma_state.c b/storage/maria/ma_state.c
index 785f1689a37..a4b002dec64 100644
--- a/storage/maria/ma_state.c
+++ b/storage/maria/ma_state.c
@@ -366,6 +366,15 @@ my_bool _ma_check_status(void *param)
 
 /**
    @brief write hook at end of trans to store status for all used table
+
+   @Notes
+   This function must be called under trnman_lock in trnman_end_trn()
+   because of the following reasons:
+   - After trnman_end_trn() is called, the current transaction will be
+   regarded as committed and all used tables state_history will be
+   visible to other transactions.  To do this, we loop over all used
+   tables and create/update a history entries that contains the correct
+   state_history for them.
 */
 
 my_bool _ma_trnman_end_trans_hook(TRN *trn, my_bool commit,
@@ -390,6 +399,13 @@ my_bool _ma_trnman_end_trans_hook(TRN *trn, my_bool commit,
           trnman_exists_active_transactions(share->state_history->trid,
                                             trn->commit_trid, 1))
       {
+        /*
+          There exist transactions that are still using the current
+          share->state_history.  Create a new history item for this
+          commit and add it first in the state_history list. This
+          ensures that all history items are stored in the list in
+          decresing trid order.
+        */
         if (!(history= my_malloc(sizeof(*history), MYF(MY_WME))))
         {
           /* purecov: begin inspected */
diff --git a/storage/maria/ma_test1.c b/storage/maria/ma_test1.c
index b0993f45524..affa3a71634 100644
--- a/storage/maria/ma_test1.c
+++ b/storage/maria/ma_test1.c
@@ -70,6 +70,9 @@ extern int _ma_flush_table_files(MARIA_HA *info, uint flush_data_or_index,
 
 int main(int argc,char *argv[])
 {
+#if defined(SAFE_MUTEX) && defined(THREAD)
+  safe_mutex_deadlock_detector= 1;
+#endif
   MY_INIT(argv[0]);
   get_options(argc,argv);
   maria_data_root= (char *)".";
diff --git a/storage/maria/ma_test2.c b/storage/maria/ma_test2.c
index 08d3ae486cf..07d67b4fce4 100644
--- a/storage/maria/ma_test2.c
+++ b/storage/maria/ma_test2.c
@@ -72,6 +72,10 @@ int main(int argc, char *argv[])
   const char *filename;
   char *blob_buffer;
   MARIA_CREATE_INFO create_info;
+
+#if defined(SAFE_MUTEX) && defined(THREAD)
+  safe_mutex_deadlock_detector= 1;
+#endif
   MY_INIT(argv[0]);
 
   filename= "test2";