BUG#51894 Replication failure with SBR on DROP TEMPORARY TABLE inside a

          transaction
BUG#52616 Temp table prevents switch binlog format from STATEMENT to ROW

Before the WL#2687 and BUG#46364, every non-transactional change that happened
after a transactional change was written to trx-cache and flushed upon
committing the transaction. WL#2687 and BUG#46364 changed this behavior and
non-transactional changes are now written to the binary log upon committing
the statement.

A binary log event is identified as transactional or non-transactional through
a flag in the Log_event which is set taking into account the underlie storage
engine on what it is stems from. In the current bug, this flag was not being
set properly when the DROP TEMPORARY TABLE was executed.

However, while fixing this bug we figured out that changes to temporary tables
should be always written to the trx-cache if there is an on-going transaction.
Otherwise, binlog events in the reversed order would be produced.

Regarding concurrency, keeping changes to temporary tables in the trx-cache is
also safe as temporary tables are only visible to the owner connection.

In this patch, we classify the following statements as unsafe:
   1 - INSERT INTO t_myisam SELECT * FROM t_myisam_temp

   2 - INSERT INTO t_myisam_temp SELECT * FROM t_myisam

   3 - CREATE TEMPORARY TABLE t_myisam_temp SELECT * FROM t_myisam

On the other hand, the following statements are classified as safe:

   1 - INSERT INTO t_innodb SELECT * FROM t_myisam_temp

   2 - INSERT INTO t_myisam_temp SELECT * FROM t_innodb

The patch also guarantees that transactions that have a DROP TEMPORARY are
always written to the binary log regardless of the mode and the outcome:
commit or rollback. In particular, the DROP TEMPORARY is extended with the
IF EXISTS clause when the current statement logging format is set to row.

Finally, the patch allows to switch from STATEMENT to MIXED/ROW when there
are temporary tables but the contrary is not possible.

mysql-test/extra/rpl_tests/rpl_binlog_max_cache_size.test:
  Updated the test case because 
     CREATE TEMPORARY TABLE t_innodb_temp SELECT * FROM t_myisam is not unsafe.
mysql-test/extra/rpl_tests/rpl_implicit_commit_binlog.test:
  Updated the test case due to the new rules: changes to
  temporary tables are written to the binary log in the
  boundaries of a transaction if there is any.
mysql-test/extra/rpl_tests/rpl_innodb.test:
  Removed comments from the test case that became false after the patch.
mysql-test/extra/rpl_tests/rpl_loaddata.test:
  Suppressed warning messages due to the following cases:
  
     1 - INSERT INTO t_myisam SELECT * FROM t_myisam_temp
  
     2 - INSERT INTO t_myisam_temp SELECT * FROM t_myisam
mysql-test/include/ctype_utf8_table.inc:
  Suppressed warning messages due to the following cases:
  
     1 - INSERT INTO t_myisam SELECT * FROM t_myisam_temp
  
     2 - INSERT INTO t_myisam_temp SELECT * FROM t_myisam
mysql-test/r/ctype_cp932_binlog_stm.result:
  Updated the test case due to the new rules: changes to
  temporary tables are written to the binary log in the
  boundaries of a transaction if there is any.
mysql-test/suite/binlog/r/binlog_database.result:
  Updated the test case due to the new rules: changes to
  temporary tables are written to the binary log in the
  boundaries of a transaction if there is any.
mysql-test/suite/binlog/r/binlog_innodb_row.result:
  Updated the result file.
mysql-test/suite/binlog/r/binlog_multi_engine.result:
  Updated the unsafe message.
mysql-test/suite/binlog/r/binlog_row_binlog.result:
  Updated the test case due to the new rules: changes to
  temporary tables are written to the binary log in the
  boundaries of a transaction if there is any.
mysql-test/suite/binlog/r/binlog_row_drop_tmp_tbl.result:
  Updated the test case due to the new rules: changes to
  temporary tables are written to the binary log in the
  boundaries of a transaction if there is any.
mysql-test/suite/binlog/r/binlog_row_mix_innodb_myisam.result:
  Updated the test case due to the new rules: changes to
  temporary tables are written to the binary log in the
  boundaries of a transaction if there is any.
mysql-test/suite/binlog/r/binlog_stm_binlog.result:
  Updated the result file.
mysql-test/suite/binlog/r/binlog_stm_mix_innodb_myisam.result:
  Updated the test case due to the new rules: changes to
  temporary tables are written to the binary log in the
  boundaries of a transaction if there is any.
mysql-test/suite/binlog/t/binlog_tmp_table.test:
  Suppressed warning messages due to the following cases:
  
     1 - INSERT INTO t_myisam SELECT * FROM t_myisam_temp
  
     2 - INSERT INTO t_myisam_temp SELECT * FROM t_myisam
mysql-test/suite/ndb/r/ndb_binlog_format.result:
  Updated the unsafe message.
mysql-test/suite/rpl/r/rpl_concurrency_error.result:
  Updated the unsafe message.
mysql-test/suite/rpl/r/rpl_mixed_binlog_max_cache_size.result:
  Updated the result file because 
     CREATE TEMPORARY TABLE t_innodb_temp SELECT * FROM t_myisam is not unsafe.
mysql-test/suite/rpl/r/rpl_mixed_implicit_commit_binlog.result:
  Updated the test case due to the new rules: changes to
  temporary tables are written to the binary log in the
  boundaries of a transaction if there is any.
mysql-test/suite/rpl/r/rpl_mixed_mixing_engines.result:
  Updated the test case due to the new rules: changes to
  temporary tables are written to the binary log in the
  boundaries of a transaction if there is any.
mysql-test/suite/rpl/r/rpl_mixed_row_innodb.result:
  Added some comments to ease the understanding of the result file.
mysql-test/suite/rpl/r/rpl_non_direct_mixed_mixing_engines.result:
  Updated the test case due to the new rules: changes to
  temporary tables are written to the binary log in the
  boundaries of a transaction if there is any.
mysql-test/suite/rpl/r/rpl_non_direct_row_mixing_engines.result:
  Updated the test case due to the new rules: changes to
  temporary tables are written to the binary log in the
  boundaries of a transaction if there is any.
mysql-test/suite/rpl/r/rpl_non_direct_stm_mixing_engines.result:
  Updated the test case due to the new rules: changes to
  temporary tables are written to the binary log in the
  boundaries of a transaction if there is any.
mysql-test/suite/rpl/r/rpl_row_drop.result:
  Updated the test case due to the new rules: changes to
  temporary tables are written to the binary log in the
  boundaries of a transaction if there is any.
mysql-test/suite/rpl/r/rpl_row_implicit_commit_binlog.result:
  Updated the test case due to the new rules: changes to
  temporary tables are written to the binary log in the
  boundaries of a transaction if there is any.
mysql-test/suite/rpl/r/rpl_row_mixing_engines.result:
  Updated the test case due to the new rules: changes to
  temporary tables are written to the binary log in the
  boundaries of a transaction if there is any.
mysql-test/suite/rpl/r/rpl_stm_binlog_max_cache_size.result:
  Updated the result file because 
     CREATE TEMPORARY TABLE t_innodb_temp SELECT * FROM t_myisam is not unsafe.
mysql-test/suite/rpl/r/rpl_stm_implicit_commit_binlog.result:
  Updated the test case due to the new rules: changes to
  temporary tables are written to the binary log in the
  boundaries of a transaction if there is any.
mysql-test/suite/rpl/r/rpl_stm_innodb.result:
  Added some comments to ease the understanding of the result file.
mysql-test/suite/rpl/r/rpl_stm_mixing_engines.result:
  Updated the test case due to the new rules: changes to
  temporary tables are written to the binary log in the
  boundaries of a transaction if there is any.
mysql-test/suite/rpl/r/rpl_stm_stop_middle_group.result:
  Updated the unsafe message.
mysql-test/suite/rpl/r/rpl_temp_temporary.result:
  Added a test case.
mysql-test/suite/rpl/t/rpl000013.test:
  Suppressed warning messages due to the following cases:
  
     1 - INSERT INTO t_myisam SELECT * FROM t_myisam_temp
  
     2 - INSERT INTO t_myisam_temp SELECT * FROM t_myisam
  
     3 - CREATE TEMPORARY TABLE t_myisam_temp SELECT * FROM t_myisam
mysql-test/suite/rpl/t/rpl_misc_functions.test:
  Suppressed warning messages.
mysql-test/suite/rpl/t/rpl_temp_table.test:
  Suppressed warning messages due to the following cases:
  
     1 - INSERT INTO t_myisam SELECT * FROM t_myisam_temp
  
     2 - INSERT INTO t_myisam_temp SELECT * FROM t_myisam
  
     3 - CREATE TEMPORARY TABLE t_myisam_temp SELECT * FROM t_myisam
mysql-test/suite/rpl/t/rpl_temp_temporary.test:
  Added a test case.
mysql-test/suite/rpl/t/rpl_temporary.test:
  Suppressed warning messages due to the following cases:
  
     1 - INSERT INTO t_myisam SELECT * FROM t_myisam_temp
  
     2 - INSERT INTO t_myisam_temp SELECT * FROM t_myisam
  
     3 - CREATE TEMPORARY TABLE t_myisam_temp SELECT * FROM t_myisam
mysql-test/suite/rpl_ndb/r/rpl_ndb_row_implicit_commit_binlog.result:
  Updated the test case due to the new rules: changes to
  temporary tables are written to the binary log in the
  boundaries of a transaction if there is any.
mysql-test/suite/rpl_ndb/r/rpl_truncate_7ndb.result:
  Updated the test case to remove references to positions
  in the binary log.
mysql-test/suite/rpl_ndb/t/rpl_truncate_7ndb.test:
  Updated the test case to remove references to positions
  in the binary log.
mysql-test/t/create_select_tmp.test:
  Suppressed warning messages due to the following cases:
  
     1 - INSERT INTO t_myisam SELECT * FROM t_myisam_temp
  
     2 - INSERT INTO t_myisam_temp SELECT * FROM t_myisam
  
     3 - CREATE TEMPORARY TABLE t_myisam_temp SELECT * FROM t_myisam
mysql-test/t/ctype_cp932_binlog_stm.test:
  Suppressed warning messages due to the following cases:
  
     1 - INSERT INTO t_myisam SELECT * FROM t_myisam_temp
  
     2 - INSERT INTO t_myisam_temp SELECT * FROM t_myisam
  
     3 - CREATE TEMPORARY TABLE t_myisam_temp SELECT * FROM t_myisam
mysql-test/t/mysqlbinlog.test:
  Suppressed warning messages due to the following cases:
  
     1 - INSERT INTO t_myisam SELECT * FROM t_myisam_temp
  
     2 - INSERT INTO t_myisam_temp SELECT * FROM t_myisam
  
     3 - CREATE TEMPORARY TABLE t_myisam_temp SELECT * FROM t_myisam
sql/log.cc:
  Improved the code by creating several functions to hide decision
  on type of engine changed, commit/abort, etc:  
  
  . stmt_has_updated_non_trans_table
  
  . trans_has_updated_non_trans_table
  
  . ending_trans
  
  Updated the binlog_rollback function and the use of the 
  OPTION_KEEP_LOG which indincates when a temporary table was
  either created or dropped and as such the command must be 
  logged if not in MIXED mode and even while rolling back the
  transaction.
sql/log.h:
  Improved the code by creating several functions to hide decision
  on type of engine changed, commit/abort, etc.
sql/log_event.cc:
  Removed the setting of the OPTION_KEEP_LOG as it is related to CREATE
  TEMPORARY and DROP TEMPORARY and not to the type of engine (i.e.
  transactional or non-transactional).
sql/log_event_old.cc:
  Removed the setting of the OPTION_KEEP_LOG as it is related to CREATE
  TEMPORARY and DROP TEMPORARY and not to the type of engine (i.e.
  transactional or non-transactional).
sql/share/errmsg-utf8.txt:
  Updated the unsafe message.
sql/sql_class.cc:
  Classifies the following statements as unsafe:
     1 - INSERT INTO t_myisam SELECT * FROM t_myisam_temp
  
     2 - INSERT INTO t_myisam_temp SELECT * FROM t_myisam
  
  On the other hand, the following statements are classified as safe:
  
     1 - INSERT INTO t_innodb SELECT * FROM t_myisam_temp
  
     2 - INSERT INTO t_myisam_temp SELECT * FROM t_innodb
sql/sql_class.h:
  It allows to switch from STATEMENT to MIXED/ROW when there are temporary
  tables but the contrary is not possible.
sql/sql_table.cc:
  Fixed the case that a DROP/DROP TEMPORARY that affects a temporary table in MIXED
  mode is written as a DROP TEMPORARY TABLE IF EXISTS because the table may not exist in the slave and due to the IF EXISTS token an error will never happen
  while processing the statement in the slave.
  
  Removed a function that was not being used.

1 files changed, 63 insertions, 22 deletions

diff --git a/sql/log.cc b/sql/log.cc
index 92ffcc58961..18f812d96bf 100644
--- a/sql/log.cc
+++ b/sql/log.cc
@@ -1576,7 +1576,7 @@ binlog_truncate_trx_cache(THD *thd, binlog_cache_mngr *cache_mngr, bool all)
     inside a transaction, we reset the transaction cache.
   */
   thd->binlog_remove_pending_rows_event(TRUE, is_transactional);
-  if (all || !thd->in_multi_stmt_transaction())
+  if (ending_trans(thd, all))
   {
     if (cache_mngr->trx_cache.has_incident())
       error= mysql_bin_log.write_incident(thd, TRUE);
@@ -1682,12 +1682,11 @@ static int binlog_commit(handlerton *hton, THD *thd, bool all)
   DBUG_ENTER("binlog_commit");
   binlog_cache_mngr *const cache_mngr=
     (binlog_cache_mngr*) thd_get_ha_data(thd, binlog_hton);
-  bool const in_transaction= thd->in_multi_stmt_transaction();
 
   DBUG_PRINT("debug",
              ("all: %d, in_transaction: %s, all.modified_non_trans_table: %s, stmt.modified_non_trans_table: %s",
               all,
-              YESNO(in_transaction),
+              YESNO(thd->in_multi_stmt_transaction()),
               YESNO(thd->transaction.all.modified_non_trans_table),
               YESNO(thd->transaction.stmt.modified_non_trans_table)));
 
@@ -1711,7 +1710,7 @@ static int binlog_commit(handlerton *hton, THD *thd, bool all)
      - We are in a transaction and a full transaction is committed.
     Otherwise, we accumulate the changes.
   */
-  if (!in_transaction || all)
+  if (ending_trans(thd, all))
   {
     Query_log_event qev(thd, STRING_WITH_LEN("COMMIT"), TRUE, FALSE, TRUE, 0);
     error= binlog_flush_trx_cache(thd, cache_mngr, &qev);
@@ -1789,29 +1788,28 @@ static int binlog_rollback(handlerton *hton, THD *thd, bool all)
   {  
     /*
       We flush the cache wrapped in a beging/rollback if:
-        . aborting a transcation that modified a non-transactional table or;
-        . aborting a statement that modified both transactional and
-         non-transctional tables but which is not in the boundaries of any
-         transaction;
+        . aborting a single or multi-statement transaction and;
+        . the format is STMT and non-trans engines were updated or;
         . the OPTION_KEEP_LOG is activate.
     */
-    if (thd->variables.binlog_format == BINLOG_FORMAT_STMT &&
-        ((all && thd->transaction.all.modified_non_trans_table) ||
-        (!all && thd->transaction.stmt.modified_non_trans_table &&
-        !thd->in_multi_stmt_transaction()) ||
-        (thd->variables.option_bits & OPTION_KEEP_LOG)))
+    if (ending_trans(thd, all) &&
+        ((thd->variables.option_bits & OPTION_KEEP_LOG) ||
+         (trans_has_updated_non_trans_table(thd) &&
+          thd->variables.binlog_format == BINLOG_FORMAT_STMT)))
     {
       Query_log_event qev(thd, STRING_WITH_LEN("ROLLBACK"), TRUE, FALSE, TRUE, 0);
       error= binlog_flush_trx_cache(thd, cache_mngr, &qev);
     }
     /*
-      Otherwise, we simply truncate the cache as there is no change on
-      non-transactional tables as follows.
+      Truncate the cache if:
+        . aborting a single or multi-statement transaction or;
+        . the OPTION_KEEP_LOG is not activate and;
+        . the format is not STMT or no non-trans were updated.
     */
-    else if (all || (!all &&
-                     (!thd->transaction.stmt.modified_non_trans_table ||
-                      !thd->in_multi_stmt_transaction() || 
-                      thd->variables.binlog_format != BINLOG_FORMAT_STMT)))
+    else if (ending_trans(thd, all) ||
+             (!(thd->variables.option_bits & OPTION_KEEP_LOG) &&
+              ((!stmt_has_updated_non_trans_table(thd) ||
+               thd->variables.binlog_format != BINLOG_FORMAT_STMT))))
       error= binlog_truncate_trx_cache(thd, cache_mngr, all);
   }
 
@@ -1909,7 +1907,7 @@ static int binlog_savepoint_rollback(handlerton *hton, THD *thd, void *sv)
     non-transactional table. Otherwise, truncate the binlog cache starting
     from the SAVEPOINT command.
   */
-  if (unlikely(thd->transaction.all.modified_non_trans_table ||
+  if (unlikely(trans_has_updated_non_trans_table(thd) ||
                (thd->variables.option_bits & OPTION_KEEP_LOG)))
   {
     int errcode= query_error_code(thd, thd->killed == THD::NOT_KILLED);
@@ -4257,6 +4255,49 @@ bool use_trans_cache(const THD* thd, bool is_transactional)
      (is_transactional || !cache_mngr->trx_cache.empty()));
 }
 
+/**
+  This function checks if a transaction, either a multi-statement
+  or a single statement transaction is about to commit or not.
+
+  @param thd The client thread that executed the current statement.
+  @param all Committing a transaction (i.e. TRUE) or a statement
+             (i.e. FALSE).
+  @return
+    @c true if committing a transaction, otherwise @c false.
+*/
+bool ending_trans(THD* thd, const bool all)
+{
+  return (all || (!all && !thd->in_multi_stmt_transaction()));
+}
+
+/**
+  This function checks if a non-transactional table was updated by
+  the current transaction.
+
+  @param thd The client thread that executed the current statement.
+  @return
+    @c true if a non-transactional table was updated, @c false
+    otherwise.
+*/
+bool trans_has_updated_non_trans_table(const THD* thd)
+{
+  return (thd->transaction.all.modified_non_trans_table ||
+          thd->transaction.stmt.modified_non_trans_table);
+}
+
+/**
+  This function checks if a non-transactional table was updated by the
+  current statement.
+
+  @param thd The client thread that executed the current statement.
+  @return
+    @c true if a non-transactional table was updated, @c false otherwise.
+*/
+bool stmt_has_updated_non_trans_table(const THD* thd)
+{
+  return (thd->transaction.stmt.modified_non_trans_table);
+}
+
 /*
   These functions are placed in this file since they need access to
   binlog_hton, which has internal linkage.
@@ -4539,7 +4580,7 @@ MYSQL_BIN_LOG::flush_and_set_pending_rows_event(THD *thd,
     {
       set_write_error(thd);
       if (check_write_error(thd) && cache_data &&
-          thd->transaction.stmt.modified_non_trans_table)
+          stmt_has_updated_non_trans_table(thd))
         cache_data->set_incident();
       DBUG_RETURN(1);
     }
@@ -4740,7 +4781,7 @@ unlock:
     {
       set_write_error(thd);
       if (check_write_error(thd) && cache_data &&
-        thd->transaction.stmt.modified_non_trans_table)
+          stmt_has_updated_non_trans_table(thd))
         cache_data->set_incident();
     }
   }