Bug#37958 - test main.plugin crash on Mac OS X when selecting from EXAMPLE engine.

This patch contains fixes for two problems:

1. As originally reported, the server crashed on Mac OS X when trying to access
   an EXAMPLE table after the EXAMPLE plugin was installed.

   It turned out that the dynamically loaded EXAMPLE plugin called the
   function hash_earch() from a Mac OS X system library, instead of
   hash_earch() from MySQL's mysys library. Makefile.am in storage/example
   does not include libmysys. So the Mac OS X linker arranged the hash_search()
   function to be linked to the system library when the shared object is
   loaded.

   One possible solution would be to include libmysys into the linkage of
   dynamic plugins. But then we must have a libmysys.so, which must be
   used by the server too. This could have a minimal performance impact,
   but foremost the change seems to bee too risky at the current state of
   MySQL 5.1.

   The selected solution is to rename MySQL's hash_search() to my_hash_search()
   like it has been done before with hash_insert() and hash_reset().

   Since this is the third time, we need to rename a hash_*() function,
   I did renamed all hash_*() functions to my_hash_*().

   To avoid changing a zillion calls to these functions, and announcing
   this to hundreds of developers, I added defines that map the old names
   to the new names.

   This change is in hash.h and hash.c.

2. The other problem was improper implementation of the handlerton-to-plugin
   mapping. We use a fixed-size array to hold a plugin reference for each
   handlerton. On every install of a handler plugin, we allocated a new slot
   of the array. On uninstall we did not free it. After some uninstall/install
   cycles the array overflowed. We did not check for overflow.

   One fix is to check for overflow to stop the crashes.

   Another fix is to free the array slot at uninstall and search for a free slot
   at plugin install.

   This change is in handler.cc.



include/hash.h:
  Bug#37958 - test main.plugin crash on Mac OS X when selecting from EXAMPLE engine.
  Renamed hash_*() functions to my_hash_*().
  Added defines that map old names to new names.
mysys/hash.c:
  Bug#37958 - test main.plugin crash on Mac OS X when selecting from EXAMPLE engine.
  Renamed hash_*() functions to my_hash_*().
sql/handler.cc:
  Bug#37958 - test main.plugin crash on Mac OS X when selecting from EXAMPLE engine.
  Protect against a failing ha_initialize_handlerton() in ha_finalize_handlerton().
  Free hton2plugin slot on uninstall of a handler plugin.
  Reuse freed slost of the hton2plugin array.
  Protect against array overrun.

1 files changed, 38 insertions, 1 deletions

diff --git a/sql/handler.cc b/sql/handler.cc
index f26e84bdfdf..8765a7f86bf 100644
--- a/sql/handler.cc
+++ b/sql/handler.cc
@@ -373,6 +373,10 @@ int ha_finalize_handlerton(st_plugin_int *plugin)
   handlerton *hton= (handlerton *)plugin->data;
   DBUG_ENTER("ha_finalize_handlerton");
 
+  /* hton can be NULL here, if ha_initialize_handlerton() failed. */
+  if (!hton)
+    goto end;
+
   switch (hton->state)
   {
   case SHOW_OPTION_NO:
@@ -401,8 +405,16 @@ int ha_finalize_handlerton(st_plugin_int *plugin)
     }
   }
 
+  /*
+    In case a plugin is uninstalled and re-installed later, it should
+    reuse an array slot. Otherwise the number of uninstall/install
+    cycles would be limited.
+  */
+  hton2plugin[hton->slot]= NULL;
+
   my_free((uchar*)hton, MYF(0));
 
+ end:
   DBUG_RETURN(0);
 }
 
@@ -437,6 +449,7 @@ int ha_initialize_handlerton(st_plugin_int *plugin)
   case SHOW_OPTION_YES:
     {
       uint tmp;
+      ulong fslot;
       /* now check the db_type for conflict */
       if (hton->db_type <= DB_TYPE_UNKNOWN ||
           hton->db_type >= DB_TYPE_DEFAULT ||
@@ -461,7 +474,31 @@ int ha_initialize_handlerton(st_plugin_int *plugin)
       tmp= hton->savepoint_offset;
       hton->savepoint_offset= savepoint_alloc_size;
       savepoint_alloc_size+= tmp;
-      hton->slot= total_ha++;
+
+      /*
+        In case a plugin is uninstalled and re-installed later, it should
+        reuse an array slot. Otherwise the number of uninstall/install
+        cycles would be limited. So look for a free slot.
+      */
+      DBUG_PRINT("plugin", ("total_ha: %lu", total_ha));
+      for (fslot= 0; fslot < total_ha; fslot++)
+      {
+        if (!hton2plugin[fslot])
+          break;
+      }
+      if (fslot < total_ha)
+        hton->slot= fslot;
+      else
+      {
+        if (total_ha >= MAX_HA)
+        {
+          sql_print_error("Too many plugins loaded. Limit is %lu. "
+                          "Failed on '%s'", (ulong) MAX_HA, plugin->name.str);
+          goto err;
+        }
+        hton->slot= total_ha++;
+      }
+
       hton2plugin[hton->slot]=plugin;
       if (hton->prepare)
         total_ha_2pc++;