BUG#21206: memory corruption when too many cursors are opened at once

Too many cursors (more than 1024) could lead to memory corruption.
This affects both, stored routines and C API cursors, and the
threshold is per-server, not per-connection.  Similarly, the
corruption could happen when the server was under heavy load
(executing more than 1024 simultaneous complex queries), and this is
the reason why this bug is fixed in 4.1, which doesn't support
cursors.

The corruption was caused by a bug in the temporary tables code, when
an attempt to create a table could lead to a write beyond allocated
space.  Note, that only internal tables were affected (the tables
created internally by the server to resolve the query), not tables
created with CREATE TEMPORARY TABLE.  Another pre-condition for the
bug is TRUE value of --temp-pool startup option, which, however, is a
default.

The cause of a bug was that random memory was overwritten in
bitmap_set_next() due to out-of-bound memory access.

2 files changed, 9 insertions, 5 deletions

diff --git a/mysys/my_bitmap.c b/mysys/my_bitmap.c
index f0d3339535d..2af4edbf1a5 100644
--- a/mysys/my_bitmap.c
+++ b/mysys/my_bitmap.c
@@ -110,7 +110,7 @@ uint bitmap_set_next(MY_BITMAP *map)
 {
   uchar *bitmap=map->bitmap;
   uint bit_found = MY_BIT_NONE;
-  uint bitmap_size=map->bitmap_size*8;
+  uint bitmap_size=map->bitmap_size;
   uint i;
 
   DBUG_ASSERT(map->bitmap);
diff --git a/sql/sql_select.cc b/sql/sql_select.cc
index fd8a5149edd..9e3883e87e0 100644
--- a/sql/sql_select.cc
+++ b/sql/sql_select.cc
@@ -5240,12 +5240,14 @@ create_tmp_table(THD *thd,TMP_TABLE_PARAM *param,List<Item> &fields,
 		       param->group_length : 0,
 		       NullS))
   {
-    bitmap_clear_bit(&temp_pool, temp_pool_slot);
+    if (temp_pool_slot != MY_BIT_NONE)
+      bitmap_clear_bit(&temp_pool, temp_pool_slot);
     DBUG_RETURN(NULL);				/* purecov: inspected */
   }
   if (!(param->copy_field=copy=new Copy_field[field_count]))
   {
-    bitmap_clear_bit(&temp_pool, temp_pool_slot);
+    if (temp_pool_slot != MY_BIT_NONE)
+      bitmap_clear_bit(&temp_pool, temp_pool_slot);
     my_free((gptr) table,MYF(0));		/* purecov: inspected */
     DBUG_RETURN(NULL);				/* purecov: inspected */
   }
@@ -5668,7 +5670,8 @@ create_tmp_table(THD *thd,TMP_TABLE_PARAM *param,List<Item> &fields,
   */
   *table->blob_field= 0;
   free_tmp_table(thd,table);                    /* purecov: inspected */
-  bitmap_clear_bit(&temp_pool, temp_pool_slot);
+  if (temp_pool_slot != MY_BIT_NONE)
+    bitmap_clear_bit(&temp_pool, temp_pool_slot);
   DBUG_RETURN(NULL);				/* purecov: inspected */
 }
 
@@ -5831,7 +5834,8 @@ free_tmp_table(THD *thd, TABLE *entry)
   my_free((gptr) entry->record[0],MYF(0));
   free_io_cache(entry);
 
-  bitmap_clear_bit(&temp_pool, entry->temp_pool_slot);
+  if (entry->temp_pool_slot != MY_BIT_NONE)
+    bitmap_clear_bit(&temp_pool, entry->temp_pool_slot);
 
   my_free((gptr) entry,MYF(0));
   thd->proc_info=save_proc_info;