Bug#15972635: Incorrect results returned in 32 table join with HAVING

The problem is a shift operation that is not 64-bit safe.
The consequence is that used tables information for a join with 32 tables
or more will be incorrect.

Fixed by adding a type cast in Item_sum::update_used_tables().

Also used the opportunity to fix some other potential bugs by adding an
explicit type-cast to an integer in a left-shift operation.
Some of them were quite harmless, but was fixed in order to get the same
signed-ness as the other operand of the operation it was used in.

sql/item_cmpfunc.cc
  Adjusted signed-ness for some integers in left-shift.

sql/item_subselect.cc
  Added type-cast to nesting_map (which is a 32/64 bit type, so
  potential bug for deeply nested queries).

sql/item_sum.cc
  Added type-cast to nesting_map (32/64-bit type) and table_map
  (64-bit type).

sql/opt_range.cc
  Added type-cast to ulonglong (which is a 64-bit type).

sql/sql_base.cc
  Added type-cast to nesting_map (which is a 32/64-bit type).

sql/sql_select.cc
  Added type-cast to nesting_map (32/64-bit type) and key_part_map
  (64-bit type).

sql/strfunc.cc
  Changed type-cast from longlong to ulonglong, to preserve signed-ness.

1 files changed, 9 insertions, 6 deletions

diff --git a/sql/item_sum.cc b/sql/item_sum.cc
index d9e634f8c16..75aec7918fa 100644
--- a/sql/item_sum.cc
+++ b/sql/item_sum.cc
@@ -151,9 +151,10 @@ bool Item_sum::check_sum_func(THD *thd, Item **ref)
       If it is there under a construct where it is not allowed 
       we report an error. 
     */ 
-    invalid= !(allow_sum_func & (1 << max_arg_level));
+    invalid= !(allow_sum_func & ((nesting_map)1 << max_arg_level));
   }
-  else if (max_arg_level >= 0 || !(allow_sum_func & (1 << nest_level)))
+  else if (max_arg_level >= 0 ||
+           !(allow_sum_func & ((nesting_map)1 << nest_level)))
   {
     /*
       The set function can be aggregated only in outer subqueries.
@@ -162,7 +163,8 @@ bool Item_sum::check_sum_func(THD *thd, Item **ref)
     */
     if (register_sum_func(thd, ref))
       return TRUE;
-    invalid= aggr_level < 0 && !(allow_sum_func & (1 << nest_level));
+    invalid= aggr_level < 0 &&
+             !(allow_sum_func & ((nesting_map)1 << nest_level));
     if (!invalid && thd->variables.sql_mode & MODE_ANSI)
       invalid= aggr_level < 0 && max_arg_level < nest_level;
   }
@@ -310,14 +312,15 @@ bool Item_sum::register_sum_func(THD *thd, Item **ref)
        sl && sl->nest_level > max_arg_level;
        sl= sl->master_unit()->outer_select() )
   {
-    if (aggr_level < 0 && (allow_sum_func & (1 << sl->nest_level)))
+    if (aggr_level < 0 &&
+        (allow_sum_func & ((nesting_map)1 << sl->nest_level)))
     {
       /* Found the most nested subquery where the function can be aggregated */
       aggr_level= sl->nest_level;
       aggr_sel= sl;
     }
   }
-  if (sl && (allow_sum_func & (1 << sl->nest_level)))
+  if (sl && (allow_sum_func & ((nesting_map)1 << sl->nest_level)))
   {
     /* 
       We reached the subquery of level max_arg_level and checked
@@ -542,7 +545,7 @@ void Item_sum::update_used_tables ()
     used_tables_cache&= PSEUDO_TABLE_BITS;
 
     /* the aggregate function is aggregated into its local context */
-    used_tables_cache |=  (1 << aggr_sel->join->tables) - 1;
+    used_tables_cache|= ((table_map)1 << aggr_sel->join->tables) - 1;
   }
 }