Race leads to deadlock during shutdown (sigterm/sigusr1)

We've noticed in some rare cases, that sending a sigterm to a memcached process
could lead to a race condition which leads to a deadlock of the process.

The worker threads are stuck on 'worker_hang_lock' because the shutdown
process hasn't completed yet.

The shudown process hasn't completed since the logger thread signals
'logger_stack_cond' without locking its corresponding 'logger_stack_lock'.
This could cause the signal to get lost, causing the logger thread to
wait indefinitely on 'logger_stack_cond' in logger_thread().

1 files changed, 4 insertions, 0 deletions

diff --git a/logger.c b/logger.c
index 0ef822c..89dfa0b 100644
--- a/logger.c
+++ b/logger.c
@@ -826,8 +826,12 @@ static int start_logger_thread(void) {
 }
 
 static int stop_logger_thread(void) {
+    // Guarantees that the logger thread is waiting on 'logger_stack_cond'
+    // before we signal it.
+    pthread_mutex_lock(&logger_stack_lock);
     do_run_logger_thread = 0;
     pthread_cond_signal(&logger_stack_cond);
+    pthread_mutex_unlock(&logger_stack_lock);
     pthread_join(logger_tid, NULL);
     return 0;
 }