Fix for bug#20390 "SELECT FOR UPDATE does not release locks

of untouched rows in full table scans".

SELECT ... FOR UPDATE/LOCK IN SHARE MODE statements as well as
UPDATE/DELETE statements which were executed using full table
scan were not releasing locks on rows which didn't satisfy
WHERE condition.
This bug surfaced in 5.0 and affected NDB tables. (InnoDB tables
intentionally don't support such unlocking in default mode).

This problem occured because code implementing join didn't call
handler::unlock_row() for rows which didn't satisfy part of condition
attached to this particular table/level of nested loop. So we solve
the problem adding this call.
Note that we already had this call in place in 4.1 but it was lost
(actually not quite correctly placed) when we have introduced nested 
joins.

Also note that additional QA should be requested once this patch is
pushed as interaction between handler::unlock_row() and many recent
MySQL features such as subqueries, unions, views is not tested enough.

1 files changed, 43 insertions, 6 deletions

diff --git a/mysql-test/t/ndb_lock.test b/mysql-test/t/ndb_lock.test
index 48a8b77dcd7..8dc02dc20cb 100644
--- a/mysql-test/t/ndb_lock.test
+++ b/mysql-test/t/ndb_lock.test
@@ -102,16 +102,36 @@ connection con1;
 commit;
 
 # table scan
+#
+# Note that there are two distinct execution paths in which we unlock
+# non-matching rows inspected during table scan - one that is used in
+# case of filesort and one that used in rest of cases. Below we cover
+# the latter (Bug #20390 "SELECT FOR UPDATE does not release locks of
+# untouched rows in full table scans").
 connection con1;
 begin;
-select * from t1 where y = 'one' or y = 'three' order by x for update;
+# We can't use "order by x" here as it will cause filesort
+--replace_column 1 # 2 # 3 #
+select * from t1 where y = 'one' or y = 'three' for update;
 
 connection con2;
 begin;
 # Have to check with pk access here since scans take locks on
 # all rows and then release them in chunks
-# Bug #20390 SELECT FOR UPDATE does not release locks of untouched rows in full table scans
-#select * from t1 where x = 2 for update;
+select * from t1 where x = 2 for update;
+--error 1205
+select * from t1 where x = 1 for update;
+rollback;
+
+connection con1;
+commit;
+
+# And now the test for case with filesort
+begin;
+select * from t1 where y = 'one' or y = 'three' order by x for update;
+connection con2;
+begin;
+select * from t1 where x = 2 for update;
 --error 1205
 select * from t1 where x = 1 for update;
 rollback;
@@ -157,15 +177,32 @@ commit;
 # table scan
 connection con1;
 begin;
-select * from t1 where y = 'one' or y = 'three' order by x lock in share mode;
+# We can't use "order by x" here as it will cause filesort
+--replace_column 1 # 2 # 3 #
+select * from t1 where y = 'one' or y = 'three' lock in share mode;
 
 connection con2;
 begin;
 select * from t1 where y = 'one' lock in share mode;
 # Have to check with pk access here since scans take locks on
 # all rows and then release them in chunks
-# Bug #20390 SELECT FOR UPDATE does not release locks of untouched rows in full table scans
-#select * from t1 where x = 2 for update;
+select * from t1 where x = 2 for update;
+--error 1205
+select * from t1 where x = 1 for update;
+rollback;
+
+connection con1;
+commit;
+
+# And the same test for case with filesort
+connection con1;
+begin;
+select * from t1 where y = 'one' or y = 'three' order by x lock in share mode;
+
+connection con2;
+begin;
+select * from t1 where y = 'one' lock in share mode;
+select * from t1 where x = 2 for update;
 --error 1205
 select * from t1 where x = 1 for update;
 rollback;