MDEV-7122: Assertion `0' failed in subselect_hash_sj_engine::init

The select mentioned in the bug attempted to create a temporary table
using the maria storage engine. The table needs to have primary keys such that
duplicates can be removed. Unfortunately this use case has a longer
than allowed key and the tmp table got created without a temporary key.
We must not allow materialization for the subquery if the total key
length and key parts is greater than what the storage engine supports.

1 files changed, 10 insertions, 0 deletions

diff --git a/sql/opt_subselect.cc b/sql/opt_subselect.cc
index 8378c3feb89..4c7925e3e71 100644
--- a/sql/opt_subselect.cc
+++ b/sql/opt_subselect.cc
@@ -830,12 +830,14 @@ bool subquery_types_allow_materialization(Item_in_subselect *in_subs)
   in_subs->sjm_scan_allowed= FALSE;
   
   bool all_are_fields= TRUE;
+  uint32 total_key_length = 0;
   for (uint i= 0; i < elements; i++)
   {
     Item *outer= in_subs->left_expr->element_index(i);
     Item *inner= it++;
     all_are_fields &= (outer->real_item()->type() == Item::FIELD_ITEM && 
                        inner->real_item()->type() == Item::FIELD_ITEM);
+    total_key_length += inner->max_length;
     if (outer->cmp_type() != inner->cmp_type())
       DBUG_RETURN(FALSE);
     switch (outer->cmp_type()) {
@@ -866,6 +868,14 @@ bool subquery_types_allow_materialization(Item_in_subselect *in_subs)
     }
   }
 
+  /*
+     Make sure that create_tmp_table will not fail due to too long keys.
+     See MDEV-7122. This check is performed inside create_tmp_table also and
+     we must do it so that we know the table has keys created.
+  */
+  if (total_key_length > HA_MAX_KEY_LENGTH || elements > HA_MAX_KEY_SEG)
+    DBUG_RETURN(FALSE);
+
   in_subs->types_allow_materialization= TRUE;
   in_subs->sjm_scan_allowed= all_are_fields;
   DBUG_PRINT("info",("subquery_types_allow_materialization: ok, allowed"));