MDEV-21577 MDL BF-BF conflict

Some DDL statements appear to acquire MDL locks for a table referenced by
foreign key constraint from the actual affected table of the DDL statement.
OPTIMIZE, REPAIR and ALTER TABLE belong to this class of DDL statements.

Earlier MariaDB version did not take this in consideration, and appended
only affected table in the certification key list in write set.
Because of missing certification information, it could happen that e.g.
OPTIMIZE table for FK child table could be allowed to apply in parallel
with DML operating on the foreign key parent table, and this could lead to
unhandled MDL lock conflicts between two high priority appliers (BF).

The fix in this patch, changes the TOI replication for OPTIMIZE, REPAIR and
ALTER TABLE statements so that before the execution of respective DDL
statement, there is foreign key parent search round. This FK parent search
contains following steps:
* open and lock the affected table (with permissive shared locks)
* iterate over foreign key contstraints and collect and array of Fk parent
  table names
* close all tables open for the THD and release MDL locks
* do the actual TOI replication with the affected table and FK parent
  table names as key values

The patch contains also new mtr test for verifying that the above mentioned
DDL statements replicate without problems when operating on FK child table.
The mtr test scenario #1, which can be used to check if some other DDL
(on top of OPTIMIZE, REPAIR and ALTER) could cause similar excessive FK
parent table locking.

Reviewed-by: Aleksey Midenkov <aleksey.midenkov@mariadb.com>
Reviewed-by: Jan Lindström <jan.lindstrom@mariadb.com>

2 files changed, 139 insertions, 0 deletions

diff --git a/mysql-test/suite/galera/t/galera_ddl_fk_conflict.inc b/mysql-test/suite/galera/t/galera_ddl_fk_conflict.inc
new file mode 100644
index 00000000000..18271d0b641
--- /dev/null
+++ b/mysql-test/suite/galera/t/galera_ddl_fk_conflict.inc
@@ -0,0 +1,114 @@
+#
+# Test for MDL BF-BF lock conflict
+# There are some DDL statements, which take extensive MDL lock for
+# a table referenced by foreign key constraint from the actual affetec table.
+# This extensive MDL lock may cause MDL BF-BF confclict situations, if the
+# FK parent table is not listed as certification key in the replication write set.
+# i.e. if replication allows such DDL to apply in parallel with regular DML operating
+# on the FK parent table.
+#
+# This test has two scenarios, where DML modifies FK parent table in node 1,
+# and offending DDL for FK child table is sent from node 2.
+#
+# param: $table_admin_command
+#        DDL table command to test, script will build full SQL statement:
+#        $table_admin_command TABLE c;
+#
+# param: $table_admin_command_end
+#        Optional additional SQL syntax to end the SQL statement, if any
+#        $table_admin_command TABLE c $table_admin_command_end;
+#
+# scenario 1, can be used to test if a DDL statement causes such MDL locking vulnerability.
+# call this test script with some table DDL command in $table_admin_command
+# if scenario 1 passes (especially COMMIT does fail for ER_LOCK_DEADLOCK),
+# then this particular DDL is vulnerable. scenraio 2 should fail for this DDL
+# unless code has not been fixed to append parent table certification keys for it.
+#
+
+--echo ######################################################################
+--echo # Test for $table_admin_command $table_admin_command_end
+--echo ######################################################################
+
+
+--echo ######################################################################
+--echo #
+--echo # Scenario #1: DML working on FK parent table BF aborted by DDL
+--echo #              over child table
+--echo #
+--echo ######################################################################
+
+--connection node_1
+SET SESSION wsrep_sync_wait=0;
+
+CREATE TABLE p (pk INTEGER PRIMARY KEY, f2 CHAR(30));
+INSERT INTO p VALUES (1, 'INITIAL VALUE');
+INSERT INTO p VALUES (2, 'INITIAL VALUE');
+
+CREATE TABLE c (pk INTEGER PRIMARY KEY, fk INTEGER, FOREIGN KEY (fk) REFERENCES p(pk));
+INSERT INTO c VALUES (1,1), (2,2);
+
+--connection node_1
+SET AUTOCOMMIT=ON;
+START TRANSACTION;
+
+UPDATE p SET f2 = 'TO DEADLOCK' WHERE pk = 1;
+
+--connection node_2
+SET SESSION wsrep_sync_wait=0;
+--eval $table_admin_command TABLE c $table_admin_command_end
+
+--connection node_1
+--error ER_LOCK_DEADLOCK
+COMMIT;
+
+SELECT COUNT(*) AS EXPECT_2 FROM p WHERE f2 = 'INITIAL VALUE';
+
+--connection node_2
+SELECT COUNT(*) AS EXPECT_2 FROM p WHERE f2 = 'INITIAL VALUE';
+
+--echo ######################################################################
+--echo #
+--echo # Scenario #2: DML working on FK parent table tries to replicate, but
+--echo #              fails in certification for earlier DDL on child table
+--echo #
+--echo ######################################################################
+
+--connection node_1
+BEGIN;
+
+# Block the applier on node #1 and issue DDL on node 2
+--let $galera_sync_point = apply_monitor_slave_enter_sync
+--source include/galera_set_sync_point.inc
+
+--connection node_2
+--eval $table_admin_command TABLE c $table_admin_command_end
+
+--connection node_1a
+--source include/galera_wait_sync_point.inc
+--source include/galera_clear_sync_point.inc
+--let $expected_cert_failures = `SELECT VARIABLE_VALUE+1 FROM information_schema.global_status WHERE VARIABLE_NAME = 'wsrep_local_cert_failures'`
+
+--connection node_1
+UPDATE p SET f2 = 'TO DEADLOCK' WHERE pk = 1;
+--send COMMIT
+
+--connection node_1a
+--let $galera_sync_point = apply_monitor_slave_enter_sync
+--source include/galera_signal_sync_point.inc
+
+--let $wait_condition = SELECT VARIABLE_VALUE = $expected_cert_failures FROM information_schema.global_status WHERE VARIABLE_NAME = 'wsrep_local_cert_failures'
+--source include/wait_condition.inc
+
+--connection node_1
+--error ER_LOCK_DEADLOCK
+--reap
+
+SELECT 'I deadlocked';
+
+SELECT COUNT(*) AS EXPECT_2 FROM p WHERE f2 = 'INITIAL VALUE';
+
+--connection node_2
+SELECT COUNT(*) AS EXPECT_2 FROM p WHERE f2 = 'INITIAL VALUE';
+
+DROP TABLE c;
+DROP TABLE p;
diff --git a/mysql-test/suite/galera/t/galera_ddl_fk_conflict.test b/mysql-test/suite/galera/t/galera_ddl_fk_conflict.test
new file mode 100644
index 00000000000..a07006a8e49
--- /dev/null
+++ b/mysql-test/suite/galera/t/galera_ddl_fk_conflict.test
@@ -0,0 +1,25 @@
+#
+# MDL BF-BF lock conflict
+#
+
+--source include/galera_cluster.inc
+--source include/have_innodb.inc
+--source include/have_debug_sync.inc
+--source include/galera_have_debug_sync.inc
+
+--connect node_1a, 127.0.0.1, root, , test, $NODE_MYPORT_1
+--connection node_1a
+SET SESSION wsrep_sync_wait=0;
+
+--let $table_admin_command = OPTIMIZE
+--source galera_ddl_fk_conflict.inc
+
+--let $table_admin_command = REPAIR
+--source galera_ddl_fk_conflict.inc
+
+--let $table_admin_command = ALTER
+--let $table_admin_command_end = ENGINE=INNODB
+--source galera_ddl_fk_conflict.inc
+
+# CHECK and ANALYZE are not affected
+