Fixed bug #18360: Type aggregation for IN and CASE may lead to a wrong

result

The IN function aggregates result types of all expressions. It uses that 
type in comparison of left expression and expressions in right part. 
This approach works in most cases. But let's consider the case when the
right part contains both strings and integers. In that case this approach may
cause wrong results because all strings which do not start with a digit are
evaluated as 0.
CASE uses the same approach when a CASE expression is given thus it's also
affected.

The idea behind this fix is to make IN function to compare expressions with
different result types differently. For example a string in the left
part will be compared as string with strings specified in right part and
will be converted to real for comparison to int or real items in the right
part.

A new function called collect_cmp_types() is added. It collects different
result types for comparison of first item in the provided list with each 
other item in the list. 

The Item_func_in class now can refer up to 5 cmp_item objects: 1 for each
result type for comparison purposes. cmp_item objects are allocated according
to found result types. The comparison of the left expression with any
right part expression is now based only on result types of these expressions.

The Item_func_case class is modified in the similar way when a CASE
expression is specified. Now it can allocate up to 5 cmp_item objects
to compare CASE expression with WHEN expressions of different types.
The comparison of the CASE expression with any WHEN expression now based only 
on result types of these expressions.

1 files changed, 24 insertions, 0 deletions

diff --git a/mysql-test/t/func_in.test b/mysql-test/t/func_in.test
index 8ddf1fbe314..906747c2f78 100644
--- a/mysql-test/t/func_in.test
+++ b/mysql-test/t/func_in.test
@@ -232,3 +232,27 @@ select some_id from t1 where some_id not in(2,-1);
 select some_id from t1 where some_id not in(-4,-1,-4);
 select some_id from t1 where some_id not in(-4,-1,3423534,2342342);
 drop table t1;
+
+#
+# Bug#18360: Type aggregation for IN and CASE may lead to a wrong result 
+#
+create table t1(f1 char(1));
+insert into t1 values ('a'),('b'),('1');
+select f1 from t1 where f1 in ('a',1);
+select f1, case f1 when 'a' then '+' when 1 then '-' end from t1;
+create index t1f1_idx on t1(f1);
+select f1 from t1 where f1 in ('a',1);
+explain select f1 from t1 where f1 in ('a',1);
+select f1 from t1 where f1 in ('a','b');
+explain select f1 from t1 where f1 in ('a','b');
+select f1 from t1 where f1 in (2,1);
+explain select f1 from t1 where f1 in (2,1);
+create table t2(f2 int, index t2f2(f2));
+insert into t2 values(0),(1),(2);
+select f2 from t2 where f2 in ('a',2);
+explain select f2 from t2 where f2 in ('a',2);
+select f2 from t2 where f2 in ('a','b');
+explain select f2 from t2 where f2 in ('a','b');
+select f2 from t2 where f2 in (1,'b');
+explain select f2 from t2 where f2 in (1,'b');
+drop table t1, t2;