MDEV-24962: Galera SST innobackupex-move ignores Environment settings

After switching to the new mariabackup interface (instead of
the outdated innobackupex interface, which is supported for
compatibility), we need to explicitly pass a path to the datadir
directory as a parameter, since in the new interface the value
of this option is not automatically set in such a way that it
always matches the SST/IST logic. This commit adds passing this
option as an explicit parameter to mariabackup. This commit also
removed unnecessary options that are not used and not supported
by mariabackup.

Also, numerous flaws in the common wsrep_sst_common script have
been fixed:

 1) There are many bash-specific constructs in the script that
    may not be supported by other interpreters, which can lead
    to the most unexpected errors during SST, because failures
    in the interpretation of bash-specific constructs lead to
    incorrect parsing of arguments;
 2) There is parse_cnf() function which is often called by other
    scripts for the "mysqld" or "--mysqld" group, but it does not
    take into account the default group suffix, which leads to
    reading values only from the default group, which then leads
    to errors due to reading the default values instead of the
    values for a specific group;
 3) Some options such as --user, --innodb-data-home-dir or --datadir
    are not removed from the --mysqld-args list, although they are
    processed inside scripts (and passing of these options funther
    may cause problems for mariabackup);
 4) If an argument that the script understands is present in
    the --mysqld-args list twice, then this causes SST to fail,
    instead of reading the most recent value;
 5) The "--host" parameter is technically still supported among
    the arguments of the SST scripts, but in reality scripts do not
    work with it as expected, especially if it has an IPv6 address;
 6) If the port number is absent in the --address parameter value,
    but the port number is explicitly passed through the --port
    argument, then the scripts for mariabackup and xtrabackup-v2
    fail;
 7) If a new address interface is used (with the --address parameter),
    then automatic default port substitution is not performed, although
    it is supported for the legacy --host/--port interface.
 8) If there are spaces in the parameter values after --mysqld_args,
    then their further transfer does not occur correctly, which
    causes mariabackup to fail during SST - the space splits
    the argument in such a way that it breaks the parsing of the
    following parameters;
 9) If most of the parameters that are names or paths to the files
    or directories contain spaces, then SST scripts fail in an
    unpredictable way due to incorrect variable substitutions;
10) If the --log-bin option is passed among the arguments of myqlds
    (--mysqld-args) without a parameter, and the --binlog option
    is not specified, then the script cannot substitute the default
    name for binlog and cannot construct binlog name using the
    --log-basename argument (which is against server specifications);
11) Tail slashes are not removed from the directory names, which,
    upon further substitution, leads to the appearance of a double
    slash in the file paths;
12) The explicit --binlog parameter (which is now always transmitted
    from the server side) and the "hidden" --log-bin parameter in the
    list of arguments after --mysqld-args are perceived as two different
    parameters in different parts of the scripts, and if they are do not
    match for some reason, this will lead to failures during SST;

Also, all new changes from the 10.6 branch have been migrated here,
including the latest pull requests for authentication (only the part
that concerns SST scripts).

It also fixes dozens of other bugs in all SST scripts.

9 files changed, 48 insertions, 27 deletions

diff --git a/mysql-test/suite/galera/disabled.def b/mysql-test/suite/galera/disabled.def
index d9483864844..176b83d56b2 100644
--- a/mysql-test/suite/galera/disabled.def
+++ b/mysql-test/suite/galera/disabled.def
@@ -5,7 +5,7 @@
 #  Separate the test case name and the comment with ':'.
 #
 #    <testcasename> : MDEV-<xxxx> <comment>
-# 
+#
 #  Do not use any TAB characters for whitespace.
 #
 ##############################################################################
@@ -17,7 +17,7 @@ galera_as_slave_replication_bundle : MDEV-15785 OPTION_GTID_BEGIN is set in Gtid
 galera_concurrent_ctas : MDEV-24842 Galera test failure on galera_concurrent_ctas
 galera_gcache_recover_manytrx : MDEV-18834 Galera test failure
 galera_mdl_race : MDEV-21524: galera.galera_mdl_race MTR failed: query 'reap' succeeded - should have failed with errno 1213
-galera_parallel_simple : MDEV-20318 galera.galera_parallel_simple fails	
+galera_parallel_simple : MDEV-20318 galera.galera_parallel_simple fails
 galera_partition : MDEV-21806: galera.galera_partition MTR failed: failed to recover from DONOR state
 galera_shutdown_nonprim : MDEV-21493 galera.galera_shutdown_nonprim
 galera_var_node_address : MDEV-20485 Galera test failure
@@ -28,3 +28,6 @@ sql_log_bin : MDEV-21491 galera.sql_log_bin
 versioning_trx_id: MDEV-18590: galera.versioning_trx_id: Test failure: mysqltest: Result content mismatch
 galera_wsrep_provider_unset_set: wsrep_provider is read-only for security reasons
 pxc-421: wsrep_provider is read-only for security reasons
+galera_sst_xtrabackup-v2: Test fails due to innodb issues
+galera_sst_xtrabackup-v2_encrypt_with_key: Test fails due to innodb issues
+galera_sst_xtrabackup-v2_data_dir: Test fails due to innodb issues
diff --git a/mysql-test/suite/galera/r/galera_ist_innodb_flush_logs,debug.rdiff b/mysql-test/suite/galera/r/galera_ist_innodb_flush_logs,debug.rdiff
index fa1b67e7ef3..55f1150c930 100644
--- a/mysql-test/suite/galera/r/galera_ist_innodb_flush_logs,debug.rdiff
+++ b/mysql-test/suite/galera/r/galera_ist_innodb_flush_logs,debug.rdiff
@@ -1,11 +1,12 @@
 --- r/galera_ist_innodb_flush_logs.result	2018-09-05 10:34:36.192439933 +0300
 +++ r/galera_ist_innodb_flush_logs.reject	2018-09-17 10:20:06.039150838 +0300
-@@ -86,3 +86,100 @@
+@@ -86,3 +86,111 @@
  DROP TABLE t1;
  COMMIT;
  SET AUTOCOMMIT=ON;
 +Performing State Transfer on a server that has been killed and restarted
 +while a DDL was in progress on it
++connection node_1;
 +CREATE TABLE t1 (f1 CHAR(255)) ENGINE=InnoDB;
 +SET AUTOCOMMIT=OFF;
 +START TRANSACTION;
@@ -14,6 +15,7 @@
 +INSERT INTO t1 VALUES ('node1_committed_before');
 +INSERT INTO t1 VALUES ('node1_committed_before');
 +INSERT INTO t1 VALUES ('node1_committed_before');
++connection node_2;
 +START TRANSACTION;
 +INSERT INTO t1 VALUES ('node2_committed_before');
 +INSERT INTO t1 VALUES ('node2_committed_before');
@@ -22,9 +24,12 @@
 +INSERT INTO t1 VALUES ('node2_committed_before');
 +COMMIT;
 +SET GLOBAL debug_dbug = 'd,sync.alter_opened_table';
++connection node_1;
 +ALTER TABLE t1 ADD COLUMN f2 INTEGER;
++connection node_2;
 +SET wsrep_sync_wait = 0;
 +Killing server ...
++connection node_1;
 +SET AUTOCOMMIT=OFF;
 +START TRANSACTION;
 +INSERT INTO t1 (f1) VALUES ('node1_committed_during');
@@ -39,6 +44,7 @@
 +INSERT INTO t1 (f1) VALUES ('node1_to_be_committed_after');
 +INSERT INTO t1 (f1) VALUES ('node1_to_be_committed_after');
 +INSERT INTO t1 (f1) VALUES ('node1_to_be_committed_after');
++connect node_1a_galera_st_kill_slave_ddl, 127.0.0.1, root, , test, $NODE_MYPORT_1;
 +SET AUTOCOMMIT=OFF;
 +START TRANSACTION;
 +INSERT INTO t1 (f1) VALUES ('node1_to_be_rollbacked_after');
@@ -46,7 +52,9 @@
 +INSERT INTO t1 (f1) VALUES ('node1_to_be_rollbacked_after');
 +INSERT INTO t1 (f1) VALUES ('node1_to_be_rollbacked_after');
 +INSERT INTO t1 (f1) VALUES ('node1_to_be_rollbacked_after');
++connection node_2;
 +Performing --wsrep-recover ...
++connection node_2;
 +Starting server ...
 +Using --wsrep-start-position when starting mysqld ...
 +SET AUTOCOMMIT=OFF;
@@ -57,6 +65,7 @@
 +INSERT INTO t1 (f1) VALUES ('node2_committed_after');
 +INSERT INTO t1 (f1) VALUES ('node2_committed_after');
 +COMMIT;
++connection node_1;
 +INSERT INTO t1 (f1) VALUES ('node1_to_be_committed_after');
 +INSERT INTO t1 (f1) VALUES ('node1_to_be_committed_after');
 +INSERT INTO t1 (f1) VALUES ('node1_to_be_committed_after');
@@ -71,6 +80,7 @@
 +INSERT INTO t1 (f1) VALUES ('node1_committed_after');
 +INSERT INTO t1 (f1) VALUES ('node1_committed_after');
 +COMMIT;
++connection node_1a_galera_st_kill_slave_ddl;
 +INSERT INTO t1 (f1) VALUES ('node1_to_be_rollbacked_after');
 +INSERT INTO t1 (f1) VALUES ('node1_to_be_rollbacked_after');
 +INSERT INTO t1 (f1) VALUES ('node1_to_be_rollbacked_after');
@@ -88,6 +98,7 @@
 +1
 +COMMIT;
 +SET AUTOCOMMIT=ON;
++connection node_1;
 +SELECT COUNT(*) = 2 FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_NAME = 't1';
 +COUNT(*) = 2
 +1
diff --git a/mysql-test/suite/galera/r/galera_ist_xtrabackup-v2.result b/mysql-test/suite/galera/r/galera_ist_xtrabackup-v2.result
index 8a7c02ab1b6..80a28d349ba 100644
--- a/mysql-test/suite/galera/r/galera_ist_xtrabackup-v2.result
+++ b/mysql-test/suite/galera/r/galera_ist_xtrabackup-v2.result
@@ -21,7 +21,7 @@ INSERT INTO t1 VALUES ('node2_committed_before');
 INSERT INTO t1 VALUES ('node2_committed_before');
 COMMIT;
 Unloading wsrep provider ...
-SET GLOBAL wsrep_provider = 'none';
+SET GLOBAL wsrep_cluster_address = '';
 connection node_1;
 SET AUTOCOMMIT=OFF;
 START TRANSACTION;
diff --git a/mysql-test/suite/galera/r/galera_ssl_upgrade.result b/mysql-test/suite/galera/r/galera_ssl_upgrade.result
index 8aab135c6a2..1443e34d041 100644
--- a/mysql-test/suite/galera/r/galera_ssl_upgrade.result
+++ b/mysql-test/suite/galera/r/galera_ssl_upgrade.result
@@ -1,10 +1,3 @@
-call mtr.add_suppression("WSREP: write_handler(): protocol is shutdown");
-SELECT VARIABLE_VALUE = 'Synced' FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME = 'wsrep_local_state_comment';
-VARIABLE_VALUE = 'Synced'
-1
-SELECT VARIABLE_VALUE = 2 FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME = 'wsrep_cluster_size';
-VARIABLE_VALUE = 2
-1
 connection node_1;
 call mtr.add_suppression("WSREP: write_handler(): protocol is shutdown.*");
 connection node_2;
@@ -12,6 +5,14 @@ call mtr.add_suppression("WSREP: write_handler(): protocol is shutdown.*");
 connection node_1;
 connection node_2;
 connection node_1;
+connection node_2;
+SELECT VARIABLE_VALUE = 'Synced' FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME = 'wsrep_local_state_comment';
+VARIABLE_VALUE = 'Synced'
+1
+SELECT VARIABLE_VALUE = 2 FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME = 'wsrep_cluster_size';
+VARIABLE_VALUE = 2
+1
+connection node_1;
 SELECT VARIABLE_VALUE = 2 FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME = 'wsrep_cluster_size';
 VARIABLE_VALUE = 2
 1
@@ -23,3 +24,5 @@ connection node_1;
 SELECT VARIABLE_VALUE = 2 FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME = 'wsrep_cluster_size';
 VARIABLE_VALUE = 2
 1
+disconnect node_2;
+disconnect node_1;
diff --git a/mysql-test/suite/galera/t/galera_ist_innodb_flush_logs.cnf b/mysql-test/suite/galera/t/galera_ist_innodb_flush_logs.cnf
index b91c897e66e..a93c291b47c 100644
--- a/mysql-test/suite/galera/t/galera_ist_innodb_flush_logs.cnf
+++ b/mysql-test/suite/galera/t/galera_ist_innodb_flush_logs.cnf
@@ -1,7 +1,7 @@
 !include ../galera_2nodes.cnf
 
 [mysqld]
-wsrep_sst_method=xtrabackup-v2
+wsrep_sst_method=mariabackup
 wsrep_sst_auth=root:
 innodb_safe_truncate=OFF
 
diff --git a/mysql-test/suite/galera/t/galera_ist_innodb_flush_logs.test b/mysql-test/suite/galera/t/galera_ist_innodb_flush_logs.test
index 839383ebfc1..fd362a26840 100644
--- a/mysql-test/suite/galera/t/galera_ist_innodb_flush_logs.test
+++ b/mysql-test/suite/galera/t/galera_ist_innodb_flush_logs.test
@@ -7,7 +7,7 @@
 --source include/big_test.inc
 --source include/galera_cluster.inc
 --source include/have_innodb.inc
---source include/have_xtrabackup.inc
+--source include/have_mariabackup.inc
 
 --source suite/galera/include/galera_st_kill_slave.inc
 --source suite/galera/include/galera_st_kill_slave_ddl.inc
diff --git a/mysql-test/suite/galera/t/galera_ssl_upgrade.test b/mysql-test/suite/galera/t/galera_ssl_upgrade.test
index 33c5a43df9d..4d6b9159c78 100644
--- a/mysql-test/suite/galera/t/galera_ssl_upgrade.test
+++ b/mysql-test/suite/galera/t/galera_ssl_upgrade.test
@@ -8,16 +8,16 @@
 --source include/have_innodb.inc
 --source include/have_ssl_communication.inc
 
-call mtr.add_suppression("WSREP: write_handler(): protocol is shutdown");
-
-SELECT VARIABLE_VALUE = 'Synced' FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME = 'wsrep_local_state_comment';
-SELECT VARIABLE_VALUE = 2 FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME = 'wsrep_cluster_size';
-
 --connection node_1
 call mtr.add_suppression("WSREP: write_handler(): protocol is shutdown.*");
 --connection node_2
 call mtr.add_suppression("WSREP: write_handler(): protocol is shutdown.*");
 
+# Save original auto_increment_offset values.
+--let $node_1=node_1
+--let $node_2=node_2
+--source include/auto_increment_offset_save.inc
+
 # Setup galera ports
 --connection node_1
 --source suite/galera/include/galera_base_port.inc
@@ -27,6 +27,9 @@ call mtr.add_suppression("WSREP: write_handler(): protocol is shutdown.*");
 --source suite/galera/include/galera_base_port.inc
 --let $NODE_GALERAPORT_2 = $_NODE_GALERAPORT
 
+SELECT VARIABLE_VALUE = 'Synced' FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME = 'wsrep_local_state_comment';
+SELECT VARIABLE_VALUE = 2 FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME = 'wsrep_cluster_size';
+
 # 2. Restart node #1 with a socket.ssl_ca that includes both the new and the old certificate
 
 --connection node_1
@@ -40,7 +43,7 @@ call mtr.add_suppression("WSREP: write_handler(): protocol is shutdown.*");
 --source include/wait_condition.inc
 SELECT VARIABLE_VALUE = 2 FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME = 'wsrep_cluster_size';
 
-# 3. Restart node #2 with the new socket.ssl_ca , socket.ssl_cert and socket.ssl_key 
+# 3. Restart node #2 with the new socket.ssl_ca , socket.ssl_cert and socket.ssl_key
 
 --connection node_2
 --source include/shutdown_mysqld.inc
@@ -52,7 +55,7 @@ SELECT VARIABLE_VALUE = 2 FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_N
 --source include/wait_condition.inc
 SELECT VARIABLE_VALUE = 2 FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME = 'wsrep_cluster_size';
 
-# 4. Restart node #1 with the new socket.ssl_ca , socket.ssl_cert and socket.ssl_key 
+# 4. Restart node #1 with the new socket.ssl_ca , socket.ssl_cert and socket.ssl_key
 
 --connection node_1
 --source include/shutdown_mysqld.inc
@@ -65,3 +68,8 @@ SELECT VARIABLE_VALUE = 2 FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_N
 SELECT VARIABLE_VALUE = 2 FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME = 'wsrep_cluster_size';
 
 # Upgrade complete. Both nodes now use the new key and certificate
+
+# Restore original auto_increment_offset values.
+--source include/auto_increment_offset_restore.inc
+
+--source include/galera_end.inc
diff --git a/mysql-test/suite/galera/t/galera_sst_xtrabackup-v2_encrypt_with_key.cnf b/mysql-test/suite/galera/t/galera_sst_xtrabackup-v2_encrypt_with_key.cnf
index 792b0d15ac4..7a98fd2aa58 100644
--- a/mysql-test/suite/galera/t/galera_sst_xtrabackup-v2_encrypt_with_key.cnf
+++ b/mysql-test/suite/galera/t/galera_sst_xtrabackup-v2_encrypt_with_key.cnf
@@ -7,7 +7,7 @@ wsrep_debug=ON
 innodb_safe_truncate=OFF
 
 [SST]
-tkey=@ENV.MYSQL_TEST_DIR/std_data/galera-key.pem
-tcert=@ENV.MYSQL_TEST_DIR/std_data/galera-cert.pem
+tkey=@ENV.MYSQL_TEST_DIR/std_data/cakey.pem
+tcert=@ENV.MYSQL_TEST_DIR/std_data/cacert.pem
 encrypt=3
 transferfmt=@ENV.MTR_GALERA_TFMT
diff --git a/mysql-test/suite/galera_3nodes/disabled.def b/mysql-test/suite/galera_3nodes/disabled.def
index 87eca207d5a..7c4d16f98f5 100644
--- a/mysql-test/suite/galera_3nodes/disabled.def
+++ b/mysql-test/suite/galera_3nodes/disabled.def
@@ -5,17 +5,13 @@
 #  Separate the test case name and the comment with ':'.
 #
 #    <testcasename> : MDEV-<xxxx> <comment>
-# 
+#
 #  Do not use any TAB characters for whitespace.
 #
 ##############################################################################
 
 GAL-501 : MDEV-24645 galera_3nodes.GAL-501 MTR failed: failed to open gcomm backend connection: 110
 galera_gtid_2_cluster : MDEV-23775 Galera test failure on galera_3nodes.galera_gtid_2_cluster
-galera_ipv6_mariabackup : MDEV-24440: galera_3nodes.galera_ipv6_mariabackup MTR fails sporadically: Failed to read from: wsrep_sst_mariabackup --role 'donor' --address '[::1]:16028/xtrabackup_sst//1'
-galera_ipv6_mariabackup_section : MDEV-22195: galera_3nodes.galera_ipv6_mariabackup_section MTR failed: assert_grep.inc failed
-galera_ipv6_mysqldump : MDEV-24036: galera_3nodes.galera_ipv6_mysqldump: rare random crashes during shutdown
-galera_ipv6_rsync_section : MDEV-23580: galera_3nodes.galera_ipv6_rsync_section MTR failed: WSREP_SST: [ERROR] rsync daemon port '16008' has been taken
 galera_ist_gcache_rollover : MDEV-23578 WSREP: exception caused by message: {v=0,t=1,ut=255,o=4,s=0,sr=0,as=1,f=6,src=50524cfe,srcvid=view_id(REG,50524cfe,4),insvid=view_id(UNKNOWN,00000000,0),ru=00000000,r=[-1,-1],fs=75,nl=(}
 galera_load_data_ist :  MDEV-24639 galera_3nodes.galera_load_data_ist MTR failed with SIGABRT: query 'reap' failed: 2013: Lost connection to MySQL server during query
 galera_load_data_ist :  MDEV-24639 galera_3nodes.galera_load_data_ist MTR failed with SIGABRT: query 'reap' failed: 2013: Lost connection to MySQL server during query