BUG#51291 Unfortunate effect around variable binlog_direct_non_transactional_updates

BUG#46364 introduced the flag binlog_direct_non_transactional_updates which
would make N-changes to be written to the binary log upon committing the
statement when "ON". On the other hand, when "OFF" the option was supposed
to mimic the behavior in 5.1. However, the implementation was not mimicking
the behavior correctly and the following bugs popped up:

  Case #1: N-changes executed within a transaction would go into
           the S-cache. When later in the same transaction a
           T-change occurs, N-changes following it were written
           to the T-cache instead of the S-cache. In some cases,
           this raises problems. For example, a
           Table_map_log_event being written initially into the
           S-cache, together with the initial N-changes, would be
           absent from the T-cache. This would log N-changes
           orphaned from a Table_map_log_event (thence discarded
           at the slave). (MIXED and ROW)

   Case #2: When rolling back a transaction, the N-changes that
            might be in the T-cache were disregarded and
            truncated along with the T-changes. (MIXED and ROW)

   Case #3: When a MIXED statement (TN) is ahead of any other
            T-changes in the transaction and it fails, it is kept
            in the T-cache until the transaction ends. This is
            not the case in 5.1 or Betony (5.5.2). In these, the
            failed TN statement would be written to the binlog at
            the same instant it had failed and not deferred until
            transaction end. (SBR)

To fix these problems, we have decided to do what follows:

   For Case #1 and #2, we circumvent them:

      1. by not letting binlog_direct_non_transactional_updates
         affect MIXED and RBR. These modes will keep the behavior
         provided by WL#2687. Although this will make Celosia to
         behave differently from 5.1, an execution will be always
         safe under such modes in the sense that slaves will never
         go out sync. In 5.1, using either MIXED or ROW while
         mixing N-statements and T-statements was not safe.

   For Case #3, we don't actually fix it. We:

      1. keep it and make all MIXED statements whether they end
         up failing or not or whether they are up front in the
         transaction or after some transactional change to always
         be stored in the T-cache. This means that it is written
         to the binary log on transaction commit/rollback only.

      2. We make the warning message even more specific about the
         MIXED statement and SBR.

mysql-test/extra/rpl_tests/rpl_mixing_engines.test:
  Updated the test case to avoid checking inconsistencies between the master and slave
  when session.binlog_direct_non_transactional_updates is ON and the format is statement.
  
  In this scenario, they will diverge because a counter (within a triger) is incremented
  and associated to the issued statement. However, an n-statement is logged ahead of
  the transaction and thus is not executed by the same order in the slave and thus gets
  a different value from the counter.
mysql-test/suite/binlog/r/binlog_multi_engine.result:
  Updated the test case with the new error ER_BINLOG_UNSAFE_MIXED_STATEMENT.
mysql-test/suite/binlog/r/binlog_stm_mix_innodb_myisam.result:
  Updated the test case with the new error ER_BINLOG_UNSAFE_MIXED_STATEMENT.
mysql-test/suite/ndb/r/ndb_binlog_format.result:
  Updated the test case with the new error ER_BINLOG_UNSAFE_MIXED_STATEMENT.
mysql-test/suite/rpl/r/rpl_concurrency_error.result:
  Updated the test case with the new error ER_BINLOG_UNSAFE_MIXED_STATEMENT.
mysql-test/suite/rpl/r/rpl_stm_binlog_max_cache_size.result:
  Updated the test case with the new error ER_BINLOG_UNSAFE_MIXED_STATEMENT.
mysql-test/suite/rpl/r/rpl_stm_mixing_engines.result:
  Updated the test case with the new error ER_BINLOG_UNSAFE_MIXED_STATEMENT.
mysql-test/suite/rpl/r/rpl_stm_stop_middle_group.result:
  Updated the test case with the new error ER_BINLOG_UNSAFE_MIXED_STATEMENT.
sql/log.cc:
  Checked if either a trx-cache or a non-trx-cache should be used. 
  
  If bin_log_direct_non_trans_update is active or the format is either
  MIXED or ROW, the cache to be used depends on the flag is_transactional.
  
  When the format is STMT, the non-trx-cache should be used if the statement
  is non-transactional and the trx-cache is empty, i.e. if any transactional
  statement has not committed yet. Otherwise, the trx-cache should be used.
sql/share/errmsg-utf8.txt:
  Added the new unsafe error ER_BINLOG_UNSAFE_MIXED_STATEMENT.
sql/sql_class.cc:
  Started printing ER_BINLOG_UNSAFE_MIXED_STATEMENT, when there
  is a mixed-statement.
  
  Organized the names of the variables and added comments.
sql/sql_lex.cc:
  Added the new unsafe error ER_BINLOG_UNSAFE_MIXED_STATEMENT.
sql/sql_lex.h:
  Added the new unsafe error ER_BINLOG_UNSAFE_MIXED_STATEMENT.

1 files changed, 3 insertions, 0 deletions

diff --git a/sql/share/errmsg-utf8.txt b/sql/share/errmsg-utf8.txt
index 30d12e89ca0..60bf27499be 100644
--- a/sql/share/errmsg-utf8.txt
+++ b/sql/share/errmsg-utf8.txt
@@ -6324,3 +6324,6 @@ ER_TOO_LONG_INDEX_COMMENT
 
 ER_BINLOG_UNSAFE_MULTIPLE_ENGINES_AND_SELF_LOGGING_ENGINE
   eng "Mixing self-logging and non-self-logging engines in a statement is unsafe."
+
+ER_BINLOG_UNSAFE_MIXED_STATEMENT
+  eng "Statements that read from both transactional and non-transactional tables and write to any of them are unsafe."