diff options
author | Dmitry Lenev <dlenev@mysql.com> | 2009-12-30 20:53:30 +0300 |
---|---|---|
committer | Dmitry Lenev <dlenev@mysql.com> | 2009-12-30 20:53:30 +0300 |
commit | 0228c9893622b1a60a3128b74c35b06d3a63c72b (patch) | |
tree | 8db2608b4c8fb2a523f143d1b99effb2fd74b32e /sql/mdl.cc | |
parent | 1c327209e1512174d3353af0e079a45b9bb38e1e (diff) | |
download | mariadb-git-0228c9893622b1a60a3128b74c35b06d3a63c72b.tar.gz |
Implementation of simple deadlock detection for metadata locks.
This change is supposed to reduce number of ER_LOCK_DEADLOCK
errors which occur when multi-statement transaction encounters
conflicting metadata lock in cases when waiting is possible.
The idea is not to fail ER_LOCK_DEADLOCK error immediately when
we encounter conflicting metadata lock. Instead we release all
metadata locks acquired by current statement and start to wait
until conflicting lock go away. To avoid deadlocks we use simple
empiric which aborts waiting with ER_LOCK_DEADLOCK error if it
turns out that somebody is waiting for metadata locks owned by
this transaction.
This patch also fixes bug #46273 "MySQL 5.4.4 new MDL: Bug#989
is not fully fixed in case of ALTER".
The bug was that concurrent execution of UPDATE or MULTI-UPDATE
statement as a part of multi-statement transaction that already
has used table being updated and ALTER TABLE statement might have
resulted of loss of isolation between this transaction and ALTER
TABLE statement, which manifested itself as changes performed by
ALTER TABLE becoming visible in transaction and wrong binary log
order as a consequence.
This problem occurred when UPDATE or MULTI-UPDATE's wait in
mysql_lock_tables() call was aborted due to metadata lock
upgrade performed by concurrent ALTER TABLE. After such abort all
metadata locks held by transaction were released but transaction
silently continued to be executed as if nothing has happened.
We solve this problem by changing our code not to release all
locks in such case. Instead we release only locks which were
acquired by current statement and then try to reacquire them
by restarting open/lock tables process. We piggyback on simple
deadlock detector implementation since this change has to be
done anyway for it.
mysql-test/include/handler.inc:
After introduction of basic deadlock detector for metadata locks
it became necessary to change parts of test for HANDLER statements
which covered some of scenarios in which ER_LOCK_DEADLOCK error
was detected in absence of real deadlock (with new deadlock detector
this no longer happens).
Also adjusted test to the fact that HANDLER READ for the table no
longer will be blocked by ALTER TABLE for the same table which awaits
for metadata lock upgrade (this is due to removal of mysql_lock_abort()
from wait_while_table_is_used()).
mysql-test/r/handler_innodb.result:
After introduction of basic deadlock detector for metadata locks
it became necessary to change parts of test for HANDLER statements
which covered some of scenarios in which ER_LOCK_DEADLOCK error
was detected in absence of real deadlock (with new deadlock detector
this no longer happens).
Also adjusted test to the fact that HANDLER READ for the table no
longer will be blocked by ALTER TABLE for the same table which awaits
for metadata lock upgrade (this is due to removal of mysql_lock_abort()
from wait_while_table_is_used()).
mysql-test/r/handler_myisam.result:
After introduction of basic deadlock detector for metadata locks
it became necessary to change parts of test for HANDLER statements
which covered some of scenarios in which ER_LOCK_DEADLOCK error
was detected in absence of real deadlock (with new deadlock detector
this no longer happens).
Also adjusted test to the fact that HANDLER READ for the table no
longer will be blocked by ALTER TABLE for the same table which awaits
for metadata lock upgrade (this is due to removal of mysql_lock_abort()
from wait_while_table_is_used()).
mysql-test/r/mdl_sync.result:
Added test coverage for basic deadlock detection in metadata
locking subsystem and for bug #46273 "MySQL 5.4.4 new MDL:
Bug#989 is not fully fixed in case of ALTER".
mysql-test/r/sp-lock.result:
Adjusted test coverage for metadata locking for stored routines
since after introduction of basic deadlock detector for metadata
locks number of scenarios in which ER_LOCK_DEADLOCK error in
absence of deadlock has decreased.
mysql-test/t/mdl_sync.test:
Added test coverage for basic deadlock detection in metadata
locking subsystem and for bug #46273 "MySQL 5.4.4 new MDL:
Bug#989 is not fully fixed in case of ALTER".
mysql-test/t/sp-lock.test:
Adjusted test coverage for metadata locking for stored routines
since after introduction of basic deadlock detector for metadata
locks number of scenarios in which ER_LOCK_DEADLOCK error in
absence of deadlock has decreased.
sql/log_event_old.cc:
close_tables_for_reopen() now takes one more argument which
specifies at which point it should stop releasing metadata
locks acquired by this connection.
sql/mdl.cc:
Changed metadata locking subsystem to support basic deadlock detection
with a help of the following simple empiric -- we assume that there is
a deadlock if there is a connection which has to wait for a metadata
lock which is currently acquired by some connection which is itself
waiting to be able to acquire some shared metadata lock.
To implement this change:
- Added MDL_context::can_wait_lead_to_deadlock()/_impl() methods
which allow to find out if there is someone waiting for metadata
lock which is held by the connection and therefore deadlocks are
possible if this connection is going to wait for some metadata lock.
To do this added version of MDL_ticket::has_pending_conflicting_lock()
method which assumes that its caller already owns LOCK_mdl mutex.
- Changed MDL_context::wait_for_locks() to use one of the above methods
to check if somebody is waiting for metadata lock owned by this
context (and therefore deadlock is possible) and emit ER_LOCK_DEADLOCK
error in this case. Also now we mark context of connections waiting
inside of this method by setting MDL_context::m_is_waiting_in_mdl
member. Thanks to this such connection could be waken up if some
other connection starts waiting for one of its metadata locks and
so a deadlock can occur.
- Adjusted notify_shared_lock() to wake up connections which wait inside
MDL_context::wait_for_locks() while holding shared metadata lock.
- Changed MDL_ticket::upgrade_shared_lock_to_exclusive() to add
temporary ticket for exclusive lock to MDL_lock::waiting queue, so
request for metadata lock upgrade can be properly detected by our
empiric.
Also now this method invokes a callback which forces transactions
holding shared metadata lock on the table to call MDL_context::
can_wait_lead_to_deadlock() method even if they don't need any new
metadata locks. Thanks to this such transactions can detect deadlocks/
livelocks between MDL and table-level locks.
Also reduced timeouts between calls to notify_shared_lock()
in MDL_ticket::upgrade_shared_lock_to_exclusive() and
MDL_context::acquire_exclusive_locks(). This was necessary
to get rid of call to mysql_lock_abort() in wait_while_table_is_used().
(Now we instead rely on notify_shared_lock() timely calling
mysql_lock_abort_for_thread() for the table on which lock
is being upgraded/acquired).
sql/mdl.h:
- Added a version of MDL_ticket::has_pending_conflicting_lock() method
to be used in situations when caller already has acquired LOCK_mdl
mutex.
- Added MDL_context::can_wait_lead_to_deadlock()/_impl() methods
which allow to find out if there is someone waiting for metadata lock
which is held by this connection and thus deadlocks are possible if
this connections will start waiting for some metadata lock.
- Added MDL_context::m_is_waiting_in_mdl member to mark connections
waiting in MDL_context::wait_for_locks() method of metadata locking
subsystem. Added getter method for this private member to make it
accessible in notify_shared_lock() auxiliary so we can wake-up such
connections if they hold shared metadata locks.
- Finally, added mysql_abort_transactions_with_shared_lock() callback
to be able force transactions which don't need any new metadata
locks still call MDL_context::can_wait_lead_to_deadlock() and detect
some of deadlocks between metadata locks and table-level locks.
sql/mysql_priv.h:
close_tables_for_reopen() now takes one more argument which
specifies at which point it should stop releasing metadata
locks acquired by this connection.
sql/sql_base.cc:
Changed approach to metadata locking for multi-statement transactions.
We no longer fail ER_LOCK_DEADLOCK error immediately when we encounter
conflicting metadata lock. Instead we release all metadata locks
acquired by current statement and start to wait until conflicting
locks to go away by calling MDL_context::wait_for_locks() method.
To avoid deadlocks the latter implements simple empiric which aborts
waiting with ER_LOCK_DEADLOCK error if it turns out that somebody
is waiting for metadata locks owned by this transaction.
To implement the change described above:
- Introduced Open_table_context::m_start_of_statement_svp member to
store state of metadata locks at the start of the statement.
- Changed Open_table_context::request_backoff_action() not to
fail with ER_LOCK_DEADLOCK immediately if back-off is requested
due to conflicting metadata lock.
- Added new argument for close_tables_for_reopen() procedure which
allows to specify subset of metadata locks to be released.
- Changed open_tables() not to release all metadata locks acquired
by current transaction when metadata lock conflict is discovered.
Instead we release only locks acquired by current statement.
- Changed open_ltable() and open_and_lock_tables_derived() not to emit
ER_LOCK_DEADLOCK error when mysql_lock_tables() is aborted in
multi-statement transaction when somebody tries to acquire exclusive
metadata lock on the table. Instead we release metadata locks acquired
by current statement and try to wait until they can be re-acquired.
- Adjusted tdc_wait_for_old_versions() to check if there is someone
waiting for one of metadata locks held by this connection and run
deadlock detection in order to avoid deadlocks in some
situations.
- Added mysql_abort_transactions_with_shared_lock() callback which
allows to force transactions holding shared metadata lock on the
table to call MDL_context::can_wait_lead_to_deadlock() even if they
don't need any new metadata locks so they can detect potential
deadlocks between metadata locking subsystem and table-level locks.
- Adjusted wait_while_table_is_used() not to set TABLE::version to
0 as it is now done only when necessary by the above-mentioned
callback. Also removed unnecessary call to mysql_lock_abort().
Instead we rely on code performing metadata lock upgrade aborting
waits on the table-level lock for this table by calling
mysql_lock_abort_for_thread() (invoked by
mysql_notify_thread_having_shared_lock()). In future this should
allow to reduce number of scenarios in which we produce
ER_LOCK_DEADLOCK error even though no real deadlock exists.
sql/sql_class.h:
Introduced Open_table_context::m_start_of_statement_svp member to
store state of metadata locks at the start of the statement.
Replaced Open_table_context::m_can_deadlock member with m_has_locks
member to reflect the fact that we no longer unconditionally emit
ER_LOCK_DEADLOCK error for transaction having some metadata locks
when conflicting metadata lock is discovered.
sql/sql_insert.cc:
close_tables_for_reopen() now takes one more argument which
specifies at which point it should stop releasing metadata
locks acquired by this connection.
sql/sql_plist.h:
Made I_P_List_iterator<T, B> usable with const lists.
sql/sql_show.cc:
close_tables_for_reopen() now takes one more argument which
specifies at which point it should stop releasing metadata
locks acquired by this connection.
sql/sql_update.cc:
Changed UPDATE and MULTI-UPDATE code not to release all metadata
locks when calls to mysql_lock_tables() are aborted. Instead we
release only locks which are acquired by this statement and then
try to reacquire them by calling open_tables(). This solves
bug #46273 "MySQL 5.4.4 new MDL: Bug#989 is not fully fixed in
case of ALTER".
Diffstat (limited to 'sql/mdl.cc')
-rw-r--r-- | sql/mdl.cc | 172 |
1 files changed, 163 insertions, 9 deletions
diff --git a/sql/mdl.cc b/sql/mdl.cc index 40074879e21..af7f310e598 100644 --- a/sql/mdl.cc +++ b/sql/mdl.cc @@ -196,6 +196,7 @@ void MDL_context::init(THD *thd_arg) to empty the list. */ m_tickets.empty(); + m_is_waiting_in_mdl= FALSE; } @@ -803,14 +804,28 @@ MDL_context::clone_ticket(MDL_request *mdl_request) @retval FALSE Lock is not a shared one or no thread was woken up */ -static bool notify_shared_lock(THD *thd, MDL_ticket *conflicting_ticket) +bool notify_shared_lock(THD *thd, MDL_ticket *conflicting_ticket) { bool woke= FALSE; if (conflicting_ticket->is_shared()) { THD *conflicting_thd= conflicting_ticket->get_ctx()->get_thd(); DBUG_ASSERT(thd != conflicting_thd); /* Self-deadlock */ - woke= mysql_notify_thread_having_shared_lock(thd, conflicting_thd); + + /* + If the thread that holds the conflicting lock is waiting + on an MDL lock, wake it up by broadcasting on COND_mdl. + Otherwise it must be waiting on a table-level lock + or some other non-MDL resource, so delegate its waking up + to an external call. + */ + if (conflicting_ticket->get_ctx()->is_waiting_in_mdl()) + { + pthread_cond_broadcast(&COND_mdl); + woke= TRUE; + } + else + woke= mysql_notify_thread_having_shared_lock(thd, conflicting_thd); } return woke; } @@ -957,7 +972,7 @@ bool MDL_context::acquire_exclusive_locks(MDL_request_list *mdl_requests) to abort this thread once again. */ struct timespec abstime; - set_timespec(abstime, 10); + set_timespec(abstime, 1); pthread_cond_timedwait(&COND_mdl, &LOCK_mdl, &abstime); } if (mysys_var->abort) @@ -1032,6 +1047,7 @@ MDL_ticket::upgrade_shared_lock_to_exclusive() const char *old_msg; st_my_thread_var *mysys_var= my_thread_var; THD *thd= m_ctx->get_thd(); + MDL_ticket *pending_ticket; DBUG_ENTER("MDL_ticket::upgrade_shared_lock_to_exclusive"); DEBUG_SYNC(thd, "mdl_upgrade_shared_lock_to_exclusive"); @@ -1045,8 +1061,22 @@ MDL_ticket::upgrade_shared_lock_to_exclusive() /* Only allow upgrades from MDL_SHARED_UPGRADABLE */ DBUG_ASSERT(m_type == MDL_SHARED_UPGRADABLE); + /* + Create an auxiliary ticket to represent a pending exclusive + lock and add it to the 'waiting' queue for the duration + of upgrade. During upgrade we abort waits of connections + that own conflicting locks. A pending request is used + to signal such connections that upon waking up they + must back off, rather than fall into sleep again. + */ + if (! (pending_ticket= MDL_ticket::create(m_ctx, MDL_EXCLUSIVE))) + DBUG_RETURN(TRUE); + pthread_mutex_lock(&LOCK_mdl); + pending_ticket->m_lock= m_lock; + m_lock->waiting.push_front(pending_ticket); + old_msg= MDL_ENTER_COND(thd, mysys_var); /* @@ -1088,6 +1118,30 @@ MDL_ticket::upgrade_shared_lock_to_exclusive() MDL_ticket *conflicting_ticket; MDL_lock::Ticket_iterator it(m_lock->granted); + /* + A temporary work-around to avoid deadlocks/livelocks in + a situation when in one connection ALTER TABLE tries to + upgrade its metadata lock and in another connection + the active transaction already got this lock in some + of its earlier statements. + In such case this transaction always succeeds with getting + a metadata lock on the table -- it already has one. + But later on it may block on the table level lock, since ALTER + got TL_WRITE_ALLOW_READ, and subsequently get aborted + by notify_shared_lock(). + An abort will lead to a back off, and a second attempt to + get an MDL lock (successful), and a table lock (-> livelock). + + The call below breaks this loop by forcing transactions to call + tdc_wait_for_old_versions() (even if the transaction doesn't need + any new metadata locks), which in turn will check if someone + is waiting on the owned MDL lock, and produce ER_LOCK_DEADLOCK. + + TODO: Long-term such deadlocks/livelock will be resolved within + MDL subsystem and thus this call will become unnecessary. + */ + mysql_abort_transactions_with_shared_lock(&m_lock->key); + while ((conflicting_ticket= it++)) { if (conflicting_ticket->m_ctx != m_ctx) @@ -1108,12 +1162,15 @@ MDL_ticket::upgrade_shared_lock_to_exclusive() to abort this thread once again. */ struct timespec abstime; - set_timespec(abstime, 10); + set_timespec(abstime, 1); DBUG_PRINT("info", ("Failed to wake-up from table-level lock ... sleeping")); pthread_cond_timedwait(&COND_mdl, &LOCK_mdl, &abstime); } if (mysys_var->abort) { + /* Remove and destroy the auxiliary pending ticket. */ + m_lock->waiting.remove(pending_ticket); + MDL_ticket::destroy(pending_ticket); /* Pending requests for shared locks can be satisfied now. */ pthread_cond_broadcast(&COND_mdl); MDL_EXIT_COND(thd, mysys_var, old_msg); @@ -1124,6 +1181,11 @@ MDL_ticket::upgrade_shared_lock_to_exclusive() m_lock->type= MDL_lock::MDL_LOCK_EXCLUSIVE; /* Set the new type of lock in the ticket. */ m_type= MDL_EXCLUSIVE; + + /* Remove and destroy the auxiliary pending ticket. */ + m_lock->waiting.remove(pending_ticket); + MDL_ticket::destroy(pending_ticket); + if (m_lock->cached_object) (*m_lock->cached_object_release_hook)(m_lock->cached_object); m_lock->cached_object= 0; @@ -1240,6 +1302,59 @@ bool MDL_context::acquire_global_shared_lock() /** + Check if there are any pending exclusive locks which conflict + with shared locks held by this thread. + + @pre The caller already has acquired LOCK_mdl. + + @return TRUE If there are any pending conflicting locks. + FALSE Otherwise. +*/ + +bool MDL_context::can_wait_lead_to_deadlock_impl() const +{ + Ticket_iterator ticket_it(m_tickets); + MDL_ticket *ticket; + + while ((ticket= ticket_it++)) + { + /* + In MySQL we never call this method while holding exclusive or + upgradeable shared metadata locks. + Otherwise we would also have to check for the presence of pending + requests for conflicting types of global lock. + In addition MDL_ticket::has_pending_conflicting_lock_impl() + won't work properly for exclusive type of lock. + */ + DBUG_ASSERT(! ticket->is_upgradable_or_exclusive()); + + if (ticket->has_pending_conflicting_lock_impl()) + return TRUE; + } + return FALSE; +} + + +/** + Implement a simple deadlock detection heuristic: check if there + are any pending exclusive locks which conflict with shared locks + held by this thread. In that case waiting can be circular, + i.e. lead to a deadlock. + + @return TRUE if there are any conflicting locks, FALSE otherwise. +*/ + +bool MDL_context::can_wait_lead_to_deadlock() const +{ + bool result; + pthread_mutex_lock(&LOCK_mdl); + result= can_wait_lead_to_deadlock_impl(); + pthread_mutex_unlock(&LOCK_mdl); + return result; +} + + +/** Wait until there will be no locks that conflict with lock requests in the given list. @@ -1249,7 +1364,7 @@ bool MDL_context::acquire_global_shared_lock() Does not acquire the locks! @retval FALSE Success. One can try to obtain metadata locks. - @retval TRUE Failure (thread was killed) + @retval TRUE Failure (thread was killed or deadlock is possible). */ bool @@ -1278,6 +1393,26 @@ MDL_context::wait_for_locks(MDL_request_list *mdl_requests) mysql_ha_flush(m_thd); pthread_mutex_lock(&LOCK_mdl); old_msg= MDL_ENTER_COND(m_thd, mysys_var); + + /* + In cases when we wait while still holding some metadata + locks deadlocks are possible. + To avoid them we use the following simple empiric - don't + wait for new lock request to be satisfied if for one of the + locks which are already held by this connection there is + a conflicting request (i.e. this connection should not wait + if someone waits for it). + This empiric should work well (e.g. give low number of false + negatives) in situations when conflicts are rare (in our + case this is true since DDL statements should be rare). + */ + if (can_wait_lead_to_deadlock_impl()) + { + MDL_EXIT_COND(m_thd, mysys_var, old_msg); + my_error(ER_LOCK_DEADLOCK, MYF(0)); + return TRUE; + } + it.rewind(); while ((mdl_request= it++)) { @@ -1301,7 +1436,9 @@ MDL_context::wait_for_locks(MDL_request_list *mdl_requests) MDL_EXIT_COND(m_thd, mysys_var, old_msg); break; } + m_is_waiting_in_mdl= TRUE; pthread_cond_wait(&COND_mdl, &LOCK_mdl); + m_is_waiting_in_mdl= FALSE; /* As a side-effect MDL_EXIT_COND() unlocks LOCK_mdl. */ MDL_EXIT_COND(m_thd, mysys_var, old_msg); } @@ -1550,21 +1687,38 @@ MDL_context::is_lock_owner(MDL_key::enum_mdl_namespace mdl_namespace, existing shared lock. @pre The ticket must match an acquired lock. + @pre The caller already has acquired LOCK_mdl. - @param ticket Shared lock against which check should be performed. + @return TRUE if there is a conflicting lock request, FALSE otherwise. +*/ - @return TRUE if there are any conflicting locks, FALSE otherwise. +bool MDL_ticket::has_pending_conflicting_lock_impl() const +{ + DBUG_ASSERT(is_shared()); + safe_mutex_assert_owner(&LOCK_mdl); + + return !m_lock->waiting.is_empty(); +} + + +/** + Check if we have any pending exclusive locks which conflict with + existing shared lock. + + @pre The ticket must match an acquired lock. + + @return TRUE if there is a pending conflicting lock request, + FALSE otherwise. */ bool MDL_ticket::has_pending_conflicting_lock() const { bool result; - DBUG_ASSERT(is_shared()); safe_mutex_assert_not_owner(&LOCK_open); pthread_mutex_lock(&LOCK_mdl); - result= !m_lock->waiting.is_empty(); + result= has_pending_conflicting_lock_impl(); pthread_mutex_unlock(&LOCK_mdl); return result; } |