Fixed bug #28076: inconsistent binary/varbinary comparison.

After adding an index the <VARBINARY> IN (SELECT <BINARY> ...)
clause returned a wrong result: the VARBINARY value was illegally padded
with zero bytes to the length of the BINARY column for the index search.
(<VARBINARY>, ...) IN (SELECT <BINARY>, ... ) clauses are affected too.


sql/item.cc:
  Fixed bug #28076.
  The Item_cache_str::save_in_field method has been overloaded
  to check cached values for an illegal padding before the saving
  into a field.
sql/item.h:
  Fixed bug #28076.
  The Item_cache_str::is_varbinary flag has been added and the
  Item_cache_str::save_in_field method has been overloaded to prevent
  cached values from an illegal padding when saving in fields.
  The signature of the Item_cache::get_cache method has been
  changed to accept pointers to Item instead of Item_result
  values.
sql/item_cmpfunc.cc:
  Fixed bug #28076.
  The Item_in_optimizer::fix_left method has been modified to
  to call Item_cache::get_cache in a new manner.
sql/item_subselect.cc:
  Fixed bug #28076.
  The subselect_indexsubquery_engine::exec method has been
  modified to take into account field conversion errors
  (copy&paste from subselect_uniquesubquery_engine::exec).
sql/sp_rcontext.cc:
  Fixed bug #28076.
  The sp_rcontext::create_case_expr_holder method has been
  modified to call Item_cache::get_cache in a new manner.
sql/sp_rcontext.h:
  Fixed bug #28076.
  The sp_rcontext::create_case_expr_holder method signature
  has been modified to pass Item pointers to the
  Item_cache::get_cache method.
sql/sql_class.cc:
  Fixed bug #28076.
  The select_max_min_finder_subselect::send_data method has been
  modified to call Item_cache::get_cache in a new manner.
mysql-test/t/subselect.test:
  Added test case for bug #28076.
mysql-test/r/subselect.result:
  Added test case for bug #28076.

1 files changed, 12 insertions, 1 deletions

diff --git a/sql/item_subselect.cc b/sql/item_subselect.cc
index 0020dd35c61..57c3b391507 100644
--- a/sql/item_subselect.cc
+++ b/sql/item_subselect.cc
@@ -1717,7 +1717,7 @@ void subselect_engine::set_row(List<Item> &item_list, Item_cache **row)
     item->decimals= sel_item->decimals;
     item->unsigned_flag= sel_item->unsigned_flag;
     maybe_null= sel_item->maybe_null;
-    if (!(row[i]= Item_cache::get_cache(res_type)))
+    if (!(row[i]= Item_cache::get_cache(sel_item)))
       return;
     row[i]->setup(sel_item);
   }
@@ -2178,6 +2178,7 @@ int subselect_indexsubquery_engine::exec()
   ((Item_in_subselect *) item)->value= 0;
   empty_result_set= TRUE;
   null_keypart= 0;
+  table->status= 0;
 
   if (check_null)
   {
@@ -2190,6 +2191,16 @@ int subselect_indexsubquery_engine::exec()
   if (copy_ref_key())
     DBUG_RETURN(1);
 
+  if (table->status)
+  {
+    /* 
+      We know that there will be no rows even if we scan. 
+      Can be set in copy_ref_key.
+    */
+    ((Item_in_subselect *) item)->value= 0;
+    DBUG_RETURN(0);
+  }
+
   if (null_keypart)
     DBUG_RETURN(scan_table());