MDEV-20109: Optimizer ignores distinct key created for materialized...

(Backported to 10.3, addressed review input)

Sj_materialization_picker::check_qep(): fix error in cost/fanout
calculations:
- for each join prefix, add #prefix_rows / TIME_FOR_COMPARE to the cost,
  like best_extension_by_limited_search does
- Remove the fanout produced by the subquery tables.
- Also take into account join condition selectivity

optimize_wo_join_buffering() (used by LooseScan and FirstMatch)
- also add #prefix_rows / TIME_FOR_COMPARE to the cost of each prefix.
- Also take into account join condition selectivity

1 files changed, 12 insertions, 12 deletions

diff --git a/mysql-test/main/cte_nonrecursive.result b/mysql-test/main/cte_nonrecursive.result
index d80d34ecc7f..69494d0c602 100644
--- a/mysql-test/main/cte_nonrecursive.result
+++ b/mysql-test/main/cte_nonrecursive.result
@@ -1159,9 +1159,9 @@ with cte as
 union
 (select a from t1 where a < 2);
 a
-7
-5
 4
+5
+7
 1
 prepare stmt from "with cte as
 (select a from t1 where a between 4 and 7 group by a)
@@ -1170,15 +1170,15 @@ union
 (select a from t1 where a < 2)";
 execute stmt;
 a
-7
-5
 4
+5
+7
 1
 execute stmt;
 a
-7
-5
 4
+5
+7
 1
 deallocate prepare stmt;
 with cte as
@@ -1188,9 +1188,9 @@ union
 (select a from cte where exists( select a from t1 where cte.a=t1.a ));
 a
 1
-7
-5
 4
+5
+7
 prepare stmt from "with cte as
 (select a from t1 where a between 4 and 7 group by a)
 (select a from t1 where a < 2)
@@ -1199,15 +1199,15 @@ union
 execute stmt;
 a
 1
-7
-5
 4
+5
+7
 execute stmt;
 a
 1
-7
-5
 4
+5
+7
 deallocate prepare stmt;
 with cte as
 (select a from t1 where a between 4 and 7)