Fixes for Aria transaction handling with lock tables

MDEV-10130 Assertion `share->in_trans == 0' failed in storage/maria/ma_close.c
MDEV-10378 Assertion `trn' failed in virtual int ha_maria::start_stmt

The problem was that maria_handler->trn was not properly reset
at commit/rollback and ha_maria::exernal_lock() could get confused
because.

There was some old code in ha_maria::implicit_commit() that tried
to take care of this, but it was not bullet proof.

Fixed by adding list of all tables that is part of the maria transaction to
TRN.

A nice side effect was of the fix is that loops in
ha_maria::implict_commit() got to be much simpler.

Other things:
- Fixed a bug in mysql_admin_table() where argument open_for_modify
  was wrongly reset for the next table in the chain
- rollback admin command also in case of fatal error.
- Split _ma_set_trn_for_table() to three version to simplify code
  and debugging.
- Several new asserts to detect the original problem (that file was
  not properly removed from trn before calling ma_close())

2 files changed, 22 insertions, 0 deletions

diff --git a/mysql-test/suite/maria/lock.result b/mysql-test/suite/maria/lock.result
index 90250568ef5..1c9d6b18100 100644
--- a/mysql-test/suite/maria/lock.result
+++ b/mysql-test/suite/maria/lock.result
@@ -99,3 +99,13 @@ f2
 3
 unlock tables;
 DROP TABLE t1,t2,tmp;
+#
+# MDEV-10378 Assertion `trn' failed in virtual int ha_maria::start_stmt
+#
+CREATE TABLE t1 (f1 VARCHAR(3), f2 INT, pk INT, PRIMARY KEY (pk)) ENGINE=Aria;
+INSERT INTO t1 VALUES  ('foo',10,1), ('foo',1,2);
+LOCK TABLE t1 WRITE;
+ALTER TABLE t1 ADD UNIQUE KEY (f1);
+ERROR 23000: Duplicate entry 'foo' for key 'f1'
+ALTER TABLE t1 ADD KEY (f2);
+DROP TABLE t1;
diff --git a/mysql-test/suite/maria/lock.test b/mysql-test/suite/maria/lock.test
index 57447a18c55..4f3d4e8065e 100644
--- a/mysql-test/suite/maria/lock.test
+++ b/mysql-test/suite/maria/lock.test
@@ -105,3 +105,15 @@ INSERT INTO t2 (f2) SELECT f3 FROM tmp AS tmp_alias;
 select * from t2;
 unlock tables;
 DROP TABLE t1,t2,tmp;
+
+--echo #
+--echo # MDEV-10378 Assertion `trn' failed in virtual int ha_maria::start_stmt
+--echo #
+
+CREATE TABLE t1 (f1 VARCHAR(3), f2 INT, pk INT, PRIMARY KEY (pk)) ENGINE=Aria;
+INSERT INTO t1 VALUES  ('foo',10,1), ('foo',1,2);
+LOCK TABLE t1 WRITE;
+--error ER_DUP_ENTRY
+ALTER TABLE t1 ADD UNIQUE KEY (f1);
+ALTER TABLE t1 ADD KEY (f2);
+DROP TABLE t1;