BUG#23443: user-defined variables can consume too much memory in the

           server

The problem was that when memory was exhausted HEAP engine could crash
(GROUP BY uses HEAP TABLE).  Alternatively, if SET was used, it could
report an error "You may only use constant expressions with SET" instead
of "Out of memory (Needed NNNNNN bytes)".

The solution is:
 - pass MY_WME to (some) calls to my_malloc() to get correct message.
 - fix heap_write() so that the first key is skipped during cleanup
   on ENOMEM because it wasn't inserted and doesn't have to be
   deleted.

No test case is provided because we can't test out-of-memory behaviour
in our current test framework.

2 files changed, 13 insertions, 4 deletions

diff --git a/heap/hp_block.c b/heap/hp_block.c
index 5c052218e58..8afcaf8d945 100644
--- a/heap/hp_block.c
+++ b/heap/hp_block.c
@@ -47,7 +47,7 @@ int _hp_get_new_block(HP_BLOCK *block, ulong *alloc_length)
       break;
 
   *alloc_length=sizeof(HP_PTRS)*i+block->records_in_block* block->recbuffer;
-  if (!(root=(HP_PTRS*) my_malloc(*alloc_length,MYF(0))))
+  if (!(root=(HP_PTRS*) my_malloc(*alloc_length,MYF(MY_WME))))
     return 1;
 
   if (i == 0)
diff --git a/heap/hp_write.c b/heap/hp_write.c
index 18fa95e7760..b5349d74691 100644
--- a/heap/hp_write.c
+++ b/heap/hp_write.c
@@ -66,13 +66,22 @@ int heap_write(HP_INFO *info, const byte *record)
   DBUG_RETURN(0);
 
 err:
-  DBUG_PRINT("info",("Duplicate key: %d",key));
+  if (my_errno == HA_ERR_FOUND_DUPP_KEY)
+    DBUG_PRINT("info",("Duplicate key: %d",key));
   info->errkey= key;
-  do
+  /*
+    Because 'key' is unsigned, we increase it before the loop, unless
+    we have to skip the key that wasn't inserted yet due to OOM.  In
+    the loop we test 'key' before decreasing it as the protection
+    against value wraparound.
+  */
+  if (my_errno != ENOMEM)
+    key++;
+  while (key-- > 0)
   {
     if (_hp_delete_key(info,share->keydef+key,record,pos,0))
       break;
-  } while (key-- > 0);
+  }
 
   share->deleted++;
   *((byte**) pos)=share->del_link;