MDEV-9500 Bug after upgrade to 10.1.10 (and 10.1.11)

Case: table with a NOT NULL field, BEFORE UPDATE trigger,
and UPDATE with a subquery that uses GROUP BY on that
NOT NULL field, and needs a temporary table for it.

Because of the BEFORE trigger, the field becomes nullable
temporarily. But its Item_field (used in GROUP BY) doesn't.
When working with the temptable some code looked at
item->maybe_null, some - at field->null_ptr.
The fix: make Item_field nullable when its field is.

This triggers an assert. The group key size is calculated
before the item is made nullable, so the group key doesn't
have a null byte. The fix: make fields/items nullable
before the group key size is calculated.

1 files changed, 12 insertions, 0 deletions

diff --git a/mysql-test/r/trigger_null-8605.result b/mysql-test/r/trigger_null-8605.result
index 3b630166663..ff1cb563f39 100644
--- a/mysql-test/r/trigger_null-8605.result
+++ b/mysql-test/r/trigger_null-8605.result
@@ -322,3 +322,15 @@ select * from t1;
 id
 0
 drop table t1;
+create table t1 (a int not null, b int);
+create trigger trgi before update on t1 for each row do 1;
+insert t1 values (1,1),(2,2),(3,3),(1,4);
+create table t2 select a as c, b as d from t1;
+update t1 set a=(select count(c) from t2 where c+1=a+1 group by a);
+select * from t1;
+a	b
+2	1
+1	2
+1	3
+2	4
+drop table t1, t2;