MDEV-27262 Unexpected index intersection with full index scan for an index

If when extracting a range condition for an index from the WHERE condition
Range Optimizer sees that the range condition covers the whole index then
such condition should be discarded because it cannot be used in any range
scan. In some cases Range Optimizer really does it, but there remained some
conditions for which it was not done. As a result the optimizer could
produce index merge plans with the full index scan for one of the indexes
participating in the index merge.
This could be observed in one of the test cases from index_merge1.inc
where a plan with index_merge_sort_union was produced and in the test case
reported for this bug where a plan with index_merge_sort_intersect was
produced. In both cases one of two index scans participating in index merge
ran over the whole index.
The patch slightly changes the original above mentioned test case from
index_merge1.inc to be able to produce an intended plan employing
index_merge_sort_union. The original query was left to show that index
merge is not used for it anymore.
It should be noted that for the plan with index_merge_sort_intersect could
be chosen for execution only due to a defect in the InnoDB code that
returns wrong estimates for the cardinality of big ranges.

This bug led to serious problems in 10.4+ where the optimization using
Rowid filters is employed (see mdev-26446).

Approved by Sergey Petrunia <sergey@mariadb.com>

1 files changed, 12 insertions, 2 deletions

diff --git a/mysql-test/include/index_merge1.inc b/mysql-test/include/index_merge1.inc
index b168a767d7c..ebadb5077ae 100644
--- a/mysql-test/include/index_merge1.inc
+++ b/mysql-test/include/index_merge1.inc
@@ -150,12 +150,22 @@ explain select * from t0 where
     (((key3 <7 and key7 < 6) or key5 < 2) and (key5 < 5 or key6 < 6));
 
 explain select * from t0 where
-    ((key3 <5 or key5 < 4) and (key1 < 4 or key2 < 4))
+    ((key3 < 4 or key5 < 4) and (key1 < 4 or key2 < 4))
   or
     ((key3 >=5 or key5 < 2) and (key5 < 5 or key6 < 6));
 
 explain select * from t0 force index(i1, i2, i3, i4, i5, i6 ) where
-    ((key3 <5 or key5 < 4) and (key1 < 4 or key2 < 4))
+    ((key3 < 4 or key5 < 4) and (key1 < 4 or key2 < 4))
+  or
+    ((key3 >=5 or key5 < 2) and (key5 < 5 or key6 < 6));
+
+explain select * from t0 force index(i1, i2, i3, i4, i5, i6 ) where
+    ((key3 < 5 or key5 < 4) and (key1 < 4 or key2 < 4))
+  or
+    ((key3 >=5 or key5 < 2) and (key5 < 5 or key6 < 6));
+
+explain select * from t0 force index(i1, i2, i3, i4, i5, i6 ) where
+    ((key3 < 10 or key5 < 4) and (key1 < 4 or key2 < 4))
   or
     ((key3 >=5 or key5 < 2) and (key5 < 5 or key6 < 6));