Bug#12352846 - TRANS_XA_START(THD*):

               ASSERTION THD->TRANSACTION.XID_STATE.XID.IS_NULL()
               FAILED

The triggered assert checks that the previous XA transaction has
done proper cleanup before a new XA transaction is started.
The bug that caused it to be triggered was that XA COMMIT did not
clean up error state if XA COMMIT discovered that the current XA
transaction had to be rolled back.

This patch fixes the problem by resetting the XA error state
before XA COMMIT calls ha_rollback_trans(). This allows following
XA transactions to be started without triggering the assert.

Test case added to xa.test.

1 files changed, 29 insertions, 0 deletions

diff --git a/mysql-test/r/xa.result b/mysql-test/r/xa.result
index 2122212c528..7b580abb19f 100644
--- a/mysql-test/r/xa.result
+++ b/mysql-test/r/xa.result
@@ -200,3 +200,32 @@ SELECT * FROM t1;
 a
 1
 DROP TABLE t1;
+#
+# Bug#12352846 - TRANS_XA_START(THD*):
+#                ASSERTION THD->TRANSACTION.XID_STATE.XID.IS_NULL()
+#                FAILED 
+#
+DROP TABLE IF EXISTS t1, t2;
+CREATE TABLE t1 (a INT);
+CREATE TABLE t2 (a INT);
+START TRANSACTION;
+INSERT INTO t1 VALUES (1);
+# Connection con2
+XA START 'xid1';
+# Sending:
+INSERT INTO t2 SELECT a FROM t1;
+# Connection default
+# Waiting until INSERT ... is blocked
+DELETE FROM t1;
+COMMIT;
+# Connection con2
+# Reaping: INSERT INTO t2 SELECT a FROM t1
+ERROR 40001: Deadlock found when trying to get lock; try restarting transaction
+XA COMMIT 'xid1';
+ERROR XA102: XA_RBDEADLOCK: Transaction branch was rolled back: deadlock was detected
+XA START 'xid1';
+XA END 'xid1';
+XA PREPARE 'xid1';
+XA ROLLBACK 'xid1';
+# Connection default
+DROP TABLE t1, t2;