Bug#28323: Server crashed in xid cache operations

The problem was that the server did not robustly handle a
unilateral roll back issued by the Resource Manager (RM)
due to a resource deadlock within the transaction branch.
By not acknowledging the roll back, the server (TM) would
eventually corrupt the XA transaction state and crash.

The solution is to mark the transaction as rollback-only
if the RM indicates that it rolled back its branch of the
transaction.

mysql-test/r/xa.result:
  Add test case result for Bug#28323
mysql-test/t/xa.test:
  Add test case for Bug#28323
sql/handler.cc:
  Reset XID only at the end of the global transaction.
sql/share/errmsg.txt:
  Add new error codes.
sql/sql_class.h:
  Remember the error reported by the Resource Manager.
sql/sql_parse.cc:
  Rollback the transaction if the Resource Manager reported
  a error and rolled back its branch of the transaction.

1 files changed, 44 insertions, 0 deletions

diff --git a/mysql-test/t/xa.test b/mysql-test/t/xa.test
index 0d564727fe3..8f408fb1eda 100644
--- a/mysql-test/t/xa.test
+++ b/mysql-test/t/xa.test
@@ -74,3 +74,47 @@ xa start 'zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz';
 select * from t1;
 drop table t1;
 
+disconnect con1;
+
+#
+# Bug#28323: Server crashed in xid cache operations
+#
+
+--disable_warnings
+drop table if exists t1;
+--enable_warnings
+
+create table t1(a int, b int, c varchar(20), primary key(a)) engine = innodb;
+insert into t1 values(1, 1, 'a');
+insert into t1 values(2, 2, 'b');
+
+connect (con1,localhost,root,,);
+connect (con2,localhost,root,,);
+
+--connection con1
+xa start 'a','b';
+update t1 set c = 'aa' where a = 1;
+--connection con2
+xa start 'a','c';
+update t1 set c = 'bb' where a = 2;
+--connection con1
+--send update t1 set c = 'bb' where a = 2
+--connection con2
+--sleep 1
+--error ER_LOCK_DEADLOCK
+update t1 set c = 'aa' where a = 1;
+select count(*) from t1;
+--error ER_XA_RBDEADLOCK
+xa end 'a','c';
+xa rollback 'a','c';
+--disconnect con2
+
+connect (con3,localhost,root,,);
+--connection con3
+xa start 'a','c';
+
+--disconnect con1
+--disconnect con3
+--connection default
+
+--echo End of 5.0 tests