A prerequisite patch for the fix for Bug#46224

"HANDLER statements within a transaction might lead to deadlocks".
Introduce a notion of a sentinel to MDL_context. A sentinel
is a ticket that separates all tickets in the context into two
groups: before and after it. Currently we can have (and need) only
one designated sentinel -- it separates all locks taken by LOCK
TABLE or HANDLER statement, which must survive COMMIT and ROLLBACK
and all other locks, which must be released at COMMIT or ROLLBACK.
The tricky part is maintaining the sentinel up to date when
someone release its corresponding ticket. This can happen, e.g.
if someone issues DROP TABLE under LOCK TABLES (generally,
see all calls to release_all_locks_for_name()).
MDL_context::release_ticket() is modified to take care of it.

******
A fix and a test case for Bug#46224 "HANDLER statements within a
transaction might lead to deadlocks".

An attempt to mix HANDLER SQL statements, which are transaction-
agnostic, an open multi-statement transaction,
and DDL against the involved tables (in a concurrent connection) 
could lead to a deadlock. The deadlock would occur when
HANDLER OPEN or HANDLER READ would have to wait on a conflicting
metadata lock. If the connection that issued HANDLER statement
also had other metadata locks (say, acquired in scope of a 
transaction), a classical deadlock situation of mutual wait
could occur.

Incompatible change: entering LOCK TABLES mode automatically
closes all open HANDLERs in the current connection.

Incompatible change: previously an attempt to wait on a lock
in a connection that has an open HANDLER statement could wait
indefinitely/deadlock. After this patch, an error ER_LOCK_DEADLOCK
is produced.

The idea of the fix is to merge thd->handler_mdl_context
with the main mdl_context of the connection, used for transactional
locks. This makes deadlock detection possible, since all waits
with locks are "visible" and available to analysis in a single
MDL context of the connection.

Since HANDLER locks and transactional locks have a different life
cycle -- HANDLERs are explicitly open and closed, and so
are HANDLER locks, explicitly acquired and released, whereas
transactional locks "accumulate" till the end of a transaction
and are released only with COMMIT, ROLLBACK and ROLLBACK TO SAVEPOINT,
a concept of "sentinel" was introduced to MDL_context.
All locks, HANDLER and others, reside in the same linked list.
However, a selected element of the list separates locks with
different life cycle. HANDLER locks always reside at the
end of the list, after the sentinel. Transactional locks are
prepended to the beginning of the list, before the sentinel.
Thus, ROLLBACK, COMMIT or ROLLBACK TO SAVEPOINT, only
release those locks that reside before the sentinel. HANDLER locks
must be released explicitly as part of HANDLER CLOSE statement,
or an implicit close. 
The same approach with sentinel
is also employed for LOCK TABLES locks. Since HANDLER and LOCK TABLES
statement has never worked together, the implementation is
made simple and only maintains one sentinel, which is used either
for HANDLER locks, or for LOCK TABLES locks.


mysql-test/include/handler.inc:
  Add test coverage for Bug#46224 "HANDLER statements within a
  transaction might lead to deadlocks".
  Extended HANDLER coverage to cover a mix of HANDLER, transactions
  and DDL statements.
mysql-test/r/handler_innodb.result:
  Update results (Bug#46224).
mysql-test/r/handler_myisam.result:
  Update results (Bug#46224).
sql/lock.cc:
  Remove thd->some_tables_deleted, it's never used.
sql/log_event.cc:
  No need to check for thd->locked_tables_mode, 
  it's done inside release_transactional_locks().
sql/mdl.cc:
  Implement the concept of HANDLER and LOCK TABLES "sentinel".
  Implement a method to clone an acquired ticket.
  Do not return tickets beyond the sentinel when acquiring
  locks, create a copy.
  Remove methods to merge and backup MDL_context, they are now
  not used (Hurra!). This opens a path to a proper constructor
  and destructor of class MDL_context (to be done in a separate
  patch).
  Modify find_ticket() to provide information about where
  the ticket position is with regard to the sentinel.
sql/mdl.h:
  Add declarations necessary for the implementation of the concept
  of "sentinel", a dedicated ticket separating transactional and
  non-transactional locks.
sql/mysql_priv.h:
  Add mark_tmp_table_for_reuse() declaration, 
  a function to "close" a single session (temporary) table.
sql/sql_base.cc:
  Remove thd->some_tables_deleted.
  Modify deadlock-prevention asserts and deadlock detection
  heuristics to take into account that from now on HANDLER locks
  reside in the same locking context.
  Add broadcast_refresh() to mysql_notify_thread_having_shared_lock():
  this is necessary for the case when a thread having a shared lock
  is asleep in tdc_wait_for_old_versions(). This situation is only
  possible with HANDLER t1 OPEN; FLUSH TABLE (since all over code paths
  that lead to tdc_wait_for_old_versions() always have an
  empty MDL_context). Previously the server would simply deadlock
  in this situation.
sql/sql_class.cc:
  Remove now unused member "THD::some_tables_deleted". 
  Move mysql_ha_cleanup() a few lines above in THD::cleanup() 
  to make sure that all handlers are closed when it's time to 
  destroy the MDL_context of this connection.
  Remove handler_mdl_context and handler_tables.
sql/sql_class.h:
  Remove THD::handler_tables, THD::handler_mdl_context,
  THD::some_tables_deleted.
sql/sql_handler.cc:
  Remove thd->handler_tables.
  Remove thd->handler_mdl_context.
  Rewrite mysql_ha_open() to have no special provision for MERGE
  tables, now that we don't have to manipulate with thd->handler_tables
  it's easy to do.
  Remove dead code.
  Fix a bug in mysql_ha_flush() when we would always flush
  a temporary HANDLER when mysql_ha_flush() is called (actually
  mysql_ha_flush() never needs to flush temporary tables).
sql/sql_insert.cc:
  Update a comment, no more thd->some_tables_deleted.
sql/sql_parse.cc:
  Implement an incompatible change: entering LOCK TABLES closes
  active HANDLERs, if any.
  Now that we have a sentinel, we don't need to check
  for thd->locked_tables_mode when releasing metadata locks in
  COMMIT/ROLLBACK.
sql/sql_plist.h:
  Add new (now necessary) methods to the list class.
sql/sql_prepare.cc:
  Make sure we don't release HANDLER locks when rollback to a
  savepoint, set to not keep locks taken at PREPARE.
sql/sql_servers.cc:
  Update to a new signature of MDL_context::release_all_locks().
sql/sql_table.cc:
  Remove thd->some_tables_deleted.
sql/transaction.cc:
  Add comments. 
  Make sure rollback to (MDL) savepoint works under LOCK TABLES and
  with HANDLER tables.

1 files changed, 38 insertions, 8 deletions

diff --git a/sql/mdl.h b/sql/mdl.h
index 2758bd3a8e6..e85f1232ff9 100644
--- a/sql/mdl.h
+++ b/sql/mdl.h
@@ -327,19 +327,15 @@ public:
   void init(THD *thd);
   void destroy();
 
-  void backup_and_reset(MDL_context *backup);
-  void restore_from_backup(MDL_context *backup);
-  void merge(MDL_context *source);
-
   bool try_acquire_shared_lock(MDL_request *mdl_request);
   bool acquire_exclusive_lock(MDL_request *mdl_request);
   bool acquire_exclusive_locks(MDL_request_list *requests);
   bool try_acquire_exclusive_lock(MDL_request *mdl_request);
   bool acquire_global_shared_lock();
+  bool clone_ticket(MDL_request *mdl_request);
 
   bool wait_for_locks(MDL_request_list *requests);
 
-  void release_all_locks();
   void release_all_locks_for_name(MDL_ticket *ticket);
   void release_lock(MDL_ticket *ticket);
   void release_global_shared_lock();
@@ -350,26 +346,60 @@ public:
   bool is_lock_owner(MDL_key::enum_mdl_namespace mdl_namespace,
                      const char *db, const char *name);
 
+
+  bool has_lock(MDL_ticket *mdl_savepoint, MDL_ticket *mdl_ticket);
+
   inline bool has_locks() const
   {
     return !m_tickets.is_empty();
   }
 
-  inline MDL_ticket *mdl_savepoint()
+  MDL_ticket *mdl_savepoint()
+  {
+    /*
+      NULL savepoint represents the start of the transaction.
+      Checking for m_lt_or_ha_sentinel also makes sure we never
+      return a pointer to HANDLER ticket as a savepoint.
+    */
+    return m_tickets.front() == m_lt_or_ha_sentinel ? NULL : m_tickets.front();
+  }
+
+  void set_lt_or_ha_sentinel()
   {
-    return m_tickets.head();
+    DBUG_ASSERT(m_lt_or_ha_sentinel == NULL);
+    m_lt_or_ha_sentinel= mdl_savepoint();
   }
+  MDL_ticket *lt_or_ha_sentinel() const { return m_lt_or_ha_sentinel; }
 
+  void clear_lt_or_ha_sentinel()
+  {
+    m_lt_or_ha_sentinel= NULL;
+  }
+  void move_ticket_after_lt_or_ha_sentinel(MDL_ticket *mdl_ticket);
+
+  void release_transactional_locks();
   void rollback_to_savepoint(MDL_ticket *mdl_savepoint);
 
   inline THD *get_thd() const { return m_thd; }
 private:
   Ticket_list m_tickets;
   bool m_has_global_shared_lock;
+  /**
+    This member has two uses:
+    1) When entering LOCK TABLES mode, remember the last taken
+    metadata lock. COMMIT/ROLLBACK must preserve these metadata
+    locks.
+    2) When we have an open HANDLER tables, store the position
+    in the list beyond which we keep locks for HANDLER tables.
+    COMMIT/ROLLBACK must, again, preserve HANDLER metadata locks.
+  */
+  MDL_ticket *m_lt_or_ha_sentinel;
   THD *m_thd;
 private:
   void release_ticket(MDL_ticket *ticket);
-  MDL_ticket *find_ticket(MDL_request *mdl_req);
+  MDL_ticket *find_ticket(MDL_request *mdl_req,
+                          bool *is_lt_or_ha);
+  void release_locks_stored_before(MDL_ticket *sentinel);
 };