MDEV-25236 Online log apply fails for ROW_FORMAT=REDUNDANT tables

In other ROW_FORMAT than REDUNDANT, the InnoDB record header
size calculation depends on dict_index_t::n_core_null_bytes.

In ROW_FORMAT=REDUNDANT, the record header always is 6 bytes
plus n_fields or 2*n_fields bytes, depending on the maximum
record size. But, during online ALTER TABLE, the log records
in the temporary file always use a format similar to
ROW_FORMAT=DYNAMIC, even omitting the 5-byte fixed-length part
of the header.

While creating a temporary file record for a ROW_FORMAT=REDUNDANT
table, InnoDB must refer to dict_index_t::n_nullable.
The field dict_index_t::n_core_null_bytes is only valid for
other than ROW_FORMAT=REDUNDANT tables.

The bug does not affect MariaDB 10.3, because only
commit 7a27db778e3e5a04271568a94c75157bb6fb48f1 (MDEV-15563)
allowed an ALGORITHM=INSTANT change of a NOT NULL column to
NULL in a ROW_FORMAT=REDUNDANT table.

The fix was developed by Thirunarayanan Balathandayuthapani
and tested by Matthias Leich. The test case was simplified by me.

2 files changed, 48 insertions, 1 deletions

diff --git a/mysql-test/suite/innodb/r/instant_alter_debug.result b/mysql-test/suite/innodb/r/instant_alter_debug.result
index 9a681b77c76..359eb1fc384 100644
--- a/mysql-test/suite/innodb/r/instant_alter_debug.result
+++ b/mysql-test/suite/innodb/r/instant_alter_debug.result
@@ -443,10 +443,31 @@ SET GLOBAL innodb_limit_optimistic_insert_debug=@save_limit;
 SELECT * FROM t1;
 c2	c
 DROP TABLE t1;
+#
+# MDEV-25236 Online log apply fails for ROW_FORMAT=REDUNDANT tables
+#
+CREATE TABLE t1
+(a INT NOT NULL, b INT, c INT, d INT, e INT, f INT, g INT, h INT, i TEXT)
+ENGINE=InnoDB;
+ALTER TABLE t1 MODIFY a INT NULL;
+SET DEBUG_SYNC='innodb_inplace_alter_table_enter SIGNAL alter WAIT_FOR go';
+ALTER TABLE t1 ADD PRIMARY KEY (a);
+connect con1,localhost,root,,;
+set DEBUG_SYNC='now WAIT_FOR alter';
+BEGIN;
+INSERT INTO t1 SET a=0, i=REPEAT('1', 10000);
+ROLLBACK;
+set DEBUG_SYNC='now SIGNAL go';
+connection default;
+disconnect con1;
+SELECT * FROM t1;
+a	b	c	d	e	f	g	h	i
+DROP TABLE t1;
+SET DEBUG_SYNC=RESET;
 # End of 10.4 tests
 SET GLOBAL innodb_purge_rseg_truncate_frequency = @save_frequency;
 SELECT variable_value-@old_instant instants
 FROM information_schema.global_status
 WHERE variable_name = 'innodb_instant_alter_column';
 instants
-32
+33
diff --git a/mysql-test/suite/innodb/t/instant_alter_debug.test b/mysql-test/suite/innodb/t/instant_alter_debug.test
index b93b9dd8f1b..10f7546cc36 100644
--- a/mysql-test/suite/innodb/t/instant_alter_debug.test
+++ b/mysql-test/suite/innodb/t/instant_alter_debug.test
@@ -512,6 +512,32 @@ SET GLOBAL innodb_limit_optimistic_insert_debug=@save_limit;
 SELECT * FROM t1;
 DROP TABLE t1;
 
+--echo #
+--echo # MDEV-25236 Online log apply fails for ROW_FORMAT=REDUNDANT tables
+--echo #
+
+CREATE TABLE t1
+(a INT NOT NULL, b INT, c INT, d INT, e INT, f INT, g INT, h INT, i TEXT)
+ENGINE=InnoDB;
+
+ALTER TABLE t1 MODIFY a INT NULL;
+
+SET DEBUG_SYNC='innodb_inplace_alter_table_enter SIGNAL alter WAIT_FOR go';
+send ALTER TABLE t1 ADD PRIMARY KEY (a);
+connect(con1,localhost,root,,);
+set DEBUG_SYNC='now WAIT_FOR alter';
+BEGIN;
+INSERT INTO t1 SET a=0, i=REPEAT('1', 10000);
+ROLLBACK;
+set DEBUG_SYNC='now SIGNAL go';
+connection default;
+reap;
+
+disconnect con1;
+SELECT * FROM t1;
+DROP TABLE t1;
+SET DEBUG_SYNC=RESET;
+
 --echo # End of 10.4 tests
 
 SET GLOBAL innodb_purge_rseg_truncate_frequency = @save_frequency;