Bug#28323: Server crashed in xid cache operations

The problem was that the server did not robustly handle a
unilateral roll back issued by the Resource Manager (RM)
due to a resource deadlock within the transaction branch.
By not acknowledging the roll back, the server (TM) would
eventually corrupt the XA transaction state and crash.

The solution is to mark the transaction as rollback-only
if the RM indicates that it rolled back its branch of the
transaction.

mysql-test/r/xa.result:
  Add test case result for Bug#28323
mysql-test/t/xa.test:
  Add test case for Bug#28323
sql/handler.cc:
  Reset XID only at the end of the global transaction.
sql/share/errmsg.txt:
  Add new error codes.
sql/sql_class.h:
  Remember the error reported by the Resource Manager.
sql/sql_parse.cc:
  Rollback the transaction if the Resource Manager reported
  a error and rolled back its branch of the transaction.

1 files changed, 19 insertions, 0 deletions

diff --git a/mysql-test/r/xa.result b/mysql-test/r/xa.result
index 5fb03d2378e..25d09f59247 100644
--- a/mysql-test/r/xa.result
+++ b/mysql-test/r/xa.result
@@ -55,3 +55,22 @@ select * from t1;
 a
 20
 drop table t1;
+drop table if exists t1;
+create table t1(a int, b int, c varchar(20), primary key(a)) engine = innodb;
+insert into t1 values(1, 1, 'a');
+insert into t1 values(2, 2, 'b');
+xa start 'a','b';
+update t1 set c = 'aa' where a = 1;
+xa start 'a','c';
+update t1 set c = 'bb' where a = 2;
+update t1 set c = 'bb' where a = 2;
+update t1 set c = 'aa' where a = 1;
+ERROR 40001: Deadlock found when trying to get lock; try restarting transaction
+select count(*) from t1;
+count(*)
+2
+xa end 'a','c';
+ERROR XA102: XA_RBDEADLOCK: Transaction branch was rolled back: deadlock was detected
+xa rollback 'a','c';
+xa start 'a','c';
+End of 5.0 tests