Table definition cache, part 2

The table opening process now works the following way:
- Create common TABLE_SHARE object
- Read the .frm file and unpack it into the TABLE_SHARE object
- Create a TABLE object based on the information in the TABLE_SHARE
  object and open a handler to the table object

Other noteworthy changes:
- In TABLE_SHARE the most common strings are now LEX_STRING's
- Better error message when table is not found
- Variable table_cache is now renamed 'table_open_cache'
- New variable 'table_definition_cache' that is the number of table defintions that will be cached
- strxnmov() calls are now fixed to avoid overflows
- strxnmov() will now always add one end \0 to result
- engine objects are now created with a TABLE_SHARE object instead of a TABLE object.
- After creating a field object one must call field->init(table) before using it

- For a busy system this change will give you:
 - Less memory usage for table object
 - Faster opening of tables (if it's has been in use or is in table definition cache)
 - Allow you to cache many table definitions objects
 - Faster drop of table


mysql-test/mysql-test-run.sh:
  Fixed some problems with --gdb option
  Test both with socket and tcp/ip port that all old servers are killed
mysql-test/r/flush_table.result:
  More tests with lock table with 2 threads + flush table
mysql-test/r/information_schema.result:
  Removed old (now wrong) result
mysql-test/r/innodb.result:
  Better error messages (thanks to TDC patch)
mysql-test/r/merge.result:
  Extra flush table test
mysql-test/r/ndb_bitfield.result:
  Better error messages (thanks to TDC patch)
mysql-test/r/ndb_partition_error.result:
  Better error messages (thanks to TDC patch)
mysql-test/r/query_cache.result:
  Remove tables left from old tests
mysql-test/r/temp_table.result:
  Test truncate with temporary tables
mysql-test/r/variables.result:
  Table_cache -> Table_open_cache
mysql-test/t/flush_table.test:
  More tests with lock table with 2 threads + flush table
mysql-test/t/merge.test:
  Extra flush table test
mysql-test/t/multi_update.test:
  Added 'sleep' to make test predictable
mysql-test/t/query_cache.test:
  Remove tables left from old tests
mysql-test/t/temp_table.test:
  Test truncate with temporary tables
mysql-test/t/variables.test:
  Table_cache -> Table_open_cache
mysql-test/valgrind.supp:
  Remove warning that may happens becasue threads dies in different order
mysys/hash.c:
  Fixed wrong DBUG_PRINT
mysys/mf_dirname.c:
  More DBUG
mysys/mf_pack.c:
  Better comment
mysys/mf_tempdir.c:
  More DBUG
  Ensure that we call cleanup_dirname() on all temporary directory paths.
  
  If we don't do this, we will get a failure when comparing temporary table
  names as in some cases the temporary table name is run through convert_dirname())
mysys/my_alloc.c:
  Indentation fix
sql/examples/ha_example.cc:
  We new use TABLE_SHARE instead of TABLE when creating engine handlers
sql/examples/ha_example.h:
  We new use TABLE_SHARE instead of TABLE when creating engine handlers
sql/examples/ha_tina.cc:
  We new use TABLE_SHARE instead of TABLE when creating engine handlers
sql/examples/ha_tina.h:
  We new use TABLE_SHARE instead of TABLE when creating engine handlers
sql/field.cc:
  Update for table definition cache:
  - Field creation now takes TABLE_SHARE instead of TABLE as argument
    (This is becasue field definitions are now cached in TABLE_SHARE)
    When a field is created, one now must call field->init(TABLE) before using it
  - Use s->db instead of s->table_cache_key
  - Added Field::clone() to create a field in TABLE from a field in TABLE_SHARE
  - make_field() takes TABLE_SHARE as argument instead of TABLE
  - move_field() -> move_field_offset()
sql/field.h:
  Update for table definition cache:
  - Field creation now takes TABLE_SHARE instead of TABLE as argument
    (This is becasue field definitions are now cached in TABLE_SHARE)
    When a field is created, one now must call field->init(TABLE) before using it
  - Added Field::clone() to create a field in TABLE from a field in TABLE_SHARE
  - make_field() takes TABLE_SHARE as argument instead of TABLE
  - move_field() -> move_field_offset()
sql/ha_archive.cc:
  We new use TABLE_SHARE instead of TABLE when creating engine handlers
sql/ha_archive.h:
  We new use TABLE_SHARE instead of TABLE when creating engine handlers
sql/ha_berkeley.cc:
  We new use TABLE_SHARE instead of TABLE when creating engine handlers
  Changed name of argument create() to not hide internal 'table' variable.
  table->s  -> table_share
sql/ha_berkeley.h:
  We new use TABLE_SHARE instead of TABLE when creating engine handlers
sql/ha_blackhole.cc:
  We new use TABLE_SHARE instead of TABLE when creating engine handlers
sql/ha_blackhole.h:
  We new use TABLE_SHARE instead of TABLE when creating engine handlers
sql/ha_federated.cc:
  We new use TABLE_SHARE instead of TABLE when creating engine handlers
  Fixed comments
  Remove index variable and replace with pointers (simple optimization)
  move_field() -> move_field_offset()
  Removed some strlen() calls
sql/ha_federated.h:
  We new use TABLE_SHARE instead of TABLE when creating engine handlers
sql/ha_heap.cc:
  We new use TABLE_SHARE instead of TABLE when creating engine handlers
  Simplify delete_table() and create() as the given file names are now without extension
sql/ha_heap.h:
  We new use TABLE_SHARE instead of TABLE when creating engine handlers
sql/ha_innodb.cc:
  We new use TABLE_SHARE instead of TABLE when creating engine handlers
sql/ha_innodb.h:
  We new use TABLE_SHARE instead of TABLE when creating engine handlers
sql/ha_myisam.cc:
  We new use TABLE_SHARE instead of TABLE when creating engine handlers
  Remove not needed fn_format()
  Fixed for new table->s structure
sql/ha_myisam.h:
  We new use TABLE_SHARE instead of TABLE when creating engine handlers
sql/ha_myisammrg.cc:
  We new use TABLE_SHARE instead of TABLE when creating engine handlers
  Don't set 'is_view' for MERGE tables
  Use new interface to find_temporary_table()
sql/ha_myisammrg.h:
  We new use TABLE_SHARE instead of TABLE when creating engine handlers
  Added flag HA_NO_COPY_ON_ALTER
sql/ha_ndbcluster.cc:
  We new use TABLE_SHARE instead of TABLE when creating engine handlers
  Fixed wrong calls to strxnmov()
  Give error HA_ERR_TABLE_DEF_CHANGED if table definition has changed
  drop_table -> intern_drop_table()
  table->s -> table_share
  Move part_info to TABLE
  Fixed comments & DBUG print's
  New arguments to print_error()
sql/ha_ndbcluster.h:
  We new use TABLE_SHARE instead of TABLE when creating engine handlers
sql/ha_partition.cc:
  We new use TABLE_SHARE instead of TABLE when creating engine handlers
  We can't set up or use part_info when creating handler as there is not yet any table object
  New ha_intialise() to work with TDC (Done by Mikael)
sql/ha_partition.h:
  We new use TABLE_SHARE instead of TABLE when creating engine handlers
  Got set_part_info() from Mikael
sql/handler.cc:
  We new use TABLE_SHARE instead of TABLE when creating engine handlers
  ha_delete_table() now also takes database as an argument
  handler::ha_open() now takes TABLE as argument
  ha_open() now calls ha_allocate_read_write_set()
  Simplify ha_allocate_read_write_set()
  Remove ha_deallocate_read_write_set()
  Use table_share (Cached by table definition cache)
sql/handler.h:
  New table flag: HA_NO_COPY_ON_ALTER (used by merge tables)
  Remove ha_deallocate_read_write_set()
  get_new_handler() now takes TABLE_SHARE as argument
  ha_delete_table() now gets database as argument
sql/item.cc:
  table_name and db are now LEX_STRING objects
  When creating fields, we have now have to call field->init(table)
  move_field -> move_field_offset()
sql/item.h:
  tmp_table_field_from_field_type() now takes an extra paramenter 'fixed_length' to allow one to force usage of CHAR
   instead of BLOB
sql/item_cmpfunc.cc:
  Fixed call to tmp_table_field_from_field_type()
sql/item_create.cc:
  Assert if new not handled cast type
sql/item_func.cc:
  When creating fields, we have now have to call field->init(table)
  dummy_table used by 'sp' now needs a TABLE_SHARE object
sql/item_subselect.cc:
  Trivial code cleanups
sql/item_sum.cc:
  When creating fields, we have now have to call field->init(table)
sql/item_timefunc.cc:
  Item_func_str_to_date::tmp_table_field() now replaced by call to
   tmp_table_field_from_field_type() (see item_timefunc.h)
sql/item_timefunc.h:
  Simply tmp_table_field()
sql/item_uniq.cc:
  When creating fields, we have now have to call field->init(table)
sql/key.cc:
  Added 'KEY' argument to 'find_ref_key' to simplify code
sql/lock.cc:
  More debugging
  Use create_table_def_key() to create key for table cache
  Allocate TABLE_SHARE properly when creating name lock
  Fix that locked_table_name doesn't test same table twice
sql/mysql_priv.h:
  New functions for table definition cache
  New interfaces to a lot of functions.
  New faster interface to find_temporary_table() and close_temporary_table()
sql/mysqld.cc:
  Added support for table definition cache of size 'table_def_size'
  Fixed som calls to strnmov()
  Changed name of 'table_cache' to 'table_open_cache'
sql/opt_range.cc:
  Use new interfaces
  Fixed warnings from valgrind
sql/parse_file.cc:
  Safer calls to strxnmov()
  Fixed typo
sql/set_var.cc:
  Added variable 'table_definition_cache'
  Variable table_cache renamed to 'table_open_cache'
sql/slave.cc:
  Use new interface
sql/sp.cc:
  Proper use of TABLE_SHARE
sql/sp_head.cc:
  Remove compiler warnings
  We have now to call field->init(table)
sql/sp_head.h:
  Pointers to parsed strings are now const
sql/sql_acl.cc:
  table_name is now a LEX_STRING
sql/sql_base.cc:
  Main implementation of table definition cache
  (The #ifdef's are there for the future when table definition cache will replace open table cache)
  Now table definitions are cached indepndent of open tables, which will speed up things when a table is in use at once from several places
  Views are not yet cached; For the moment we only cache if a table is a view or not.
  
  Faster implementation of find_temorary_table()
  Replace 'wait_for_refresh()' with the more general function 'wait_for_condition()'
  Drop table is slightly faster as we can use the table definition cache to know the type of the table
sql/sql_cache.cc:
  table_cache_key and table_name are now LEX_STRING
  'sDBUG print fixes
sql/sql_class.cc:
  table_cache_key is now a LEX_STRING
  safer strxnmov()
sql/sql_class.h:
  Added number of open table shares (table definitions)
sql/sql_db.cc:
  safer strxnmov()
sql/sql_delete.cc:
  Use new interface to find_temporary_table()
sql/sql_derived.cc:
  table_name is now a LEX_STRING
sql/sql_handler.cc:
  TABLE_SHARE->db and TABLE_SHARE->table_name are now LEX_STRING's
sql/sql_insert.cc:
  TABLE_SHARE->db and TABLE_SHARE->table_name are now LEX_STRING's
sql/sql_lex.cc:
  Make parsed string a const (to quickly find out if anything is trying to change the query string)
sql/sql_lex.h:
  Make parsed string a const (to quickly find out if anything is trying to change the query string)
sql/sql_load.cc:
  Safer strxnmov()
sql/sql_parse.cc:
  Better error if wrong DB name
sql/sql_partition.cc:
  part_info moved to TABLE from TABLE_SHARE
  Indentation changes
sql/sql_select.cc:
  Indentation fixes
  Call field->init(TABLE) for new created fields
  Update create_tmp_table() to use TABLE_SHARE properly
sql/sql_select.h:
  Call field->init(TABLE) for new created fields
sql/sql_show.cc:
  table_name is now a LEX_STRING
  part_info moved to TABLE
sql/sql_table.cc:
  Use table definition cache to speed up delete of tables
  Fixed calls to functions with new interfaces
  Don't use 'share_not_to_be_used'
  Instead of doing openfrm() when doing repair, we now have to call
  get_table_share() followed by open_table_from_share().
  Replace some fn_format() with faster unpack_filename().
  Safer strxnmov()
  part_info is now in TABLE
  Added Mikaels patch for partition and ALTER TABLE
  Instead of using 'TABLE_SHARE->is_view' use 'table_flags() & HA_NO_COPY_ON_ALTER
sql/sql_test.cc:
  table_name and table_cache_key are now LEX_STRING's
sql/sql_trigger.cc:
  TABLE_SHARE->db and TABLE_SHARE->table_name are now LEX_STRING's
  safer strxnmov()
  Removed compiler warnings
sql/sql_update.cc:
  Call field->init(TABLE) after field is created
sql/sql_view.cc:
  safer strxnmov()
  Create common TABLE_SHARE object for views to allow us to cache if table is a view
sql/structs.h:
  Added SHOW_TABLE_DEFINITIONS
sql/table.cc:
  Creation and destruct of TABLE_SHARE objects that are common for many TABLE objects
  
  The table opening process now works the following way:
  - Create common TABLE_SHARE object
  - Read the .frm file and unpack it into the TABLE_SHARE object
  - Create a TABLE object based on the information in the TABLE_SHARE
    object and open a handler to the table object
  
  open_table_def() is written in such a way that it should be trival to add parsing of the .frm files in new formats
sql/table.h:
  TABLE objects for the same database table now share a common TABLE_SHARE object
  In TABLE_SHARE the most common strings are now LEX_STRING's
sql/unireg.cc:
  Changed arguments to rea_create_table() to have same order as other functions
  Call field->init(table) for new created fields
sql/unireg.h:
  Added OPEN_VIEW
strings/strxnmov.c:
  Change strxnmov() to always add end \0
  This makes usage of strxnmov() safer as most of MySQL code assumes that strxnmov() will create a null terminated string

1 files changed, 43 insertions, 31 deletions

diff --git a/sql/sql_partition.cc b/sql/sql_partition.cc
index 202c747cf8c..2792bda1365 100644
--- a/sql/sql_partition.cc
+++ b/sql/sql_partition.cc
@@ -39,6 +39,7 @@
 #include "md5.h"
 
 #ifdef WITH_PARTITION_STORAGE_ENGINE
+#include <ha_partition.h>
 /*
   Partition related functions declarations and some static constants;
 */
@@ -901,7 +902,7 @@ static bool set_up_field_array(TABLE *table,
 {
   Field **ptr, *field, **field_array;
   uint no_fields= 0, size_field_array, i= 0;
-  partition_info *part_info= table->s->part_info;
+  partition_info *part_info= table->part_info;
   int result= FALSE;
   DBUG_ENTER("set_up_field_array");
 
@@ -1276,7 +1277,7 @@ static bool check_primary_key(TABLE *table)
   if (primary_key < MAX_KEY)
   {
     set_indicator_in_key_fields(table->key_info+primary_key);
-    check_fields_in_PF(table->s->part_info->full_part_field_array,
+    check_fields_in_PF(table->part_info->full_part_field_array,
                         &all_fields, &some_fields);
     clear_indicator_in_key_fields(table->key_info+primary_key);
     if (unlikely(!all_fields))
@@ -1314,7 +1315,7 @@ static bool check_unique_keys(TABLE *table)
     if (table->key_info[i].flags & HA_NOSAME) //Unique index
     {
       set_indicator_in_key_fields(table->key_info+i);
-      check_fields_in_PF(table->s->part_info->full_part_field_array,
+      check_fields_in_PF(table->part_info->full_part_field_array,
                          &all_fields, &some_fields);
       clear_indicator_in_key_fields(table->key_info+i);
       if (unlikely(!all_fields))
@@ -1596,26 +1597,32 @@ static uint32 get_part_id_from_linear_hash(longlong hash_value, uint mask,
 }
 
 /*
-  This function is called as part of opening the table by opening the .frm
-  file. It is a part of CREATE TABLE to do this so it is quite permissible
-  that errors due to erroneus syntax isn't found until we come here.
-  If the user has used a non-existing field in the table is one such example
-  of an error that is not discovered until here.
+  fix partition functions
+
   SYNOPSIS
     fix_partition_func()
     thd                  The thread object
     name                 The name of the partitioned table
     table                TABLE object for which partition fields are set-up
+
   RETURN VALUE
     TRUE
     FALSE
+
   DESCRIPTION
     The name parameter contains the full table name and is used to get the
     database name of the table which is used to set-up a correct
     TABLE_LIST object for use in fix_fields.
+
+NOTES
+    This function is called as part of opening the table by opening the .frm
+    file. It is a part of CREATE TABLE to do this so it is quite permissible
+    that errors due to erroneus syntax isn't found until we come here.
+    If the user has used a non-existing field in the table is one such example
+    of an error that is not discovered until here.
 */
 
-bool fix_partition_func(THD *thd, const char* name, TABLE *table)
+bool fix_partition_func(THD *thd, const char *name, TABLE *table)
 {
   bool result= TRUE;
   uint dir_length, home_dir_length;
@@ -1623,19 +1630,19 @@ bool fix_partition_func(THD *thd, const char* name, TABLE *table)
   TABLE_SHARE *share= table->s;
   char db_name_string[FN_REFLEN];
   char* db_name;
-  partition_info *part_info= share->part_info;
+  partition_info *part_info= table->part_info;
   ulong save_set_query_id= thd->set_query_id;
   DBUG_ENTER("fix_partition_func");
 
   thd->set_query_id= 0;
   /*
-  Set-up the TABLE_LIST object to be a list with a single table
-  Set the object to zero to create NULL pointers and set alias
-  and real name to table name and get database name from file name.
+    Set-up the TABLE_LIST object to be a list with a single table
+    Set the object to zero to create NULL pointers and set alias
+    and real name to table name and get database name from file name.
   */
 
   bzero((void*)&tables, sizeof(TABLE_LIST));
-  tables.alias= tables.table_name= (char*)share->table_name;
+  tables.alias= tables.table_name= (char*) share->table_name.str;
   tables.table= table;
   tables.next_local= 0;
   tables.next_name_resolution_table= 0;
@@ -1650,8 +1657,8 @@ bool fix_partition_func(THD *thd, const char* name, TABLE *table)
   {
     DBUG_ASSERT(part_info->subpart_type == HASH_PARTITION);
     /*
-     Subpartition is defined. We need to verify that subpartitioning
-     function is correct.
+      Subpartition is defined. We need to verify that subpartitioning
+      function is correct.
     */
     if (part_info->linear_hash_ind)
       set_linear_hash_mask(part_info, part_info->no_subparts);
@@ -1664,7 +1671,8 @@ bool fix_partition_func(THD *thd, const char* name, TABLE *table)
     else
     {
       if (unlikely(fix_fields_part_func(thd, &tables,
-                   part_info->subpart_expr, part_info, TRUE)))
+                                        part_info->subpart_expr, part_info,
+                                        TRUE)))
         goto end;
       if (unlikely(part_info->subpart_expr->result_type() != INT_RESULT))
       {
@@ -1676,8 +1684,8 @@ bool fix_partition_func(THD *thd, const char* name, TABLE *table)
   }
   DBUG_ASSERT(part_info->part_type != NOT_A_PARTITION);
   /*
-   Partition is defined. We need to verify that partitioning
-   function is correct.
+    Partition is defined. We need to verify that partitioning
+    function is correct.
   */
   if (part_info->part_type == HASH_PARTITION)
   {
@@ -1999,6 +2007,7 @@ char *generate_partition_syntax(partition_info *part_info,
   File fptr;
   char *buf= NULL; //Return buffer
   const char *file_name;
+
   sprintf(path, "%s_%lx_%lx", "part_syntax", current_pid,
           current_thd->thread_id);
   fn_format(path,path,mysql_tmpdir,".psy", MY_REPLACE_EXT);
@@ -2160,7 +2169,7 @@ close_file:
 bool partition_key_modified(TABLE *table, List<Item> &fields)
 {
   List_iterator_fast<Item> f(fields);
-  partition_info *part_info= table->s->part_info;
+  partition_info *part_info= table->part_info;
   Item_field *item_field;
   DBUG_ENTER("partition_key_modified");
   if (!part_info)
@@ -2770,7 +2779,7 @@ static uint32 get_sub_part_id_from_key(const TABLE *table,byte *buf,
                                        const key_range *key_spec)
 {
   byte *rec0= table->record[0];
-  partition_info *part_info= table->s->part_info;
+  partition_info *part_info= table->part_info;
   uint32 part_id;
   DBUG_ENTER("get_sub_part_id_from_key");
 
@@ -2809,7 +2818,7 @@ bool get_part_id_from_key(const TABLE *table, byte *buf, KEY *key_info,
 {
   bool result;
   byte *rec0= table->record[0];
-  partition_info *part_info= table->s->part_info;
+  partition_info *part_info= table->part_info;
   DBUG_ENTER("get_part_id_from_key");
 
   key_restore(buf, (byte*)key_spec->key, key_info, key_spec->length);
@@ -2849,7 +2858,7 @@ void get_full_part_id_from_key(const TABLE *table, byte *buf,
                                part_id_range *part_spec)
 {
   bool result;
-  partition_info *part_info= table->s->part_info;
+  partition_info *part_info= table->part_info;
   byte *rec0= table->record[0];
   DBUG_ENTER("get_full_part_id_from_key");
 
@@ -2894,7 +2903,7 @@ void get_full_part_id_from_key(const TABLE *table, byte *buf,
 void get_partition_set(const TABLE *table, byte *buf, const uint index,
                        const key_range *key_spec, part_id_range *part_spec)
 {
-  partition_info *part_info= table->s->part_info;
+  partition_info *part_info= table->part_info;
   uint no_parts= get_tot_partitions(part_info), i, part_id;
   uint sub_part= no_parts;
   uint32 part_part= no_parts;
@@ -3081,14 +3090,16 @@ void get_partition_set(const TABLE *table, byte *buf, const uint index,
      possible to retrace this given an item tree.
 */
 
-bool mysql_unpack_partition(THD *thd, uchar *part_buf, uint part_info_len,
-                            TABLE* table, enum db_type default_db_type)
+bool mysql_unpack_partition(THD *thd, const uchar *part_buf,
+                            uint part_info_len, TABLE* table,
+                            enum db_type default_db_type)
 {
   Item *thd_free_list= thd->free_list;
   bool result= TRUE;
   partition_info *part_info;
   LEX *old_lex= thd->lex, lex;
   DBUG_ENTER("mysql_unpack_partition");
+
   thd->lex= &lex;
   lex_start(thd, part_buf, part_info_len);
   /*
@@ -3116,7 +3127,8 @@ bool mysql_unpack_partition(THD *thd, uchar *part_buf, uint part_info_len,
     goto end;
   }
   part_info= lex.part_info;
-  table->s->part_info= part_info;
+  table->part_info= part_info;
+  ((ha_partition*)table->file)->set_part_info(part_info);
   if (part_info->default_engine_type == DB_TYPE_UNKNOWN)
     part_info->default_engine_type= default_db_type;
   else
@@ -3138,9 +3150,9 @@ bool mysql_unpack_partition(THD *thd, uchar *part_buf, uint part_info_len,
     uint part_func_len= part_info->part_func_len;
     uint subpart_func_len= part_info->subpart_func_len; 
     char *part_func_string, *subpart_func_string= NULL;
-    if (!((part_func_string= sql_alloc(part_func_len))) ||
+    if (!((part_func_string= thd->alloc(part_func_len))) ||
         (subpart_func_len &&
-        !((subpart_func_string= sql_alloc(subpart_func_len)))))
+        !((subpart_func_string= thd->alloc(subpart_func_len)))))
     {
       my_error(ER_OUTOFMEMORY, MYF(0), part_func_len);
       free_items(thd->free_list);
@@ -3187,7 +3199,7 @@ void set_field_ptr(Field **ptr, const byte *new_buf,
 
   do
   {
-    (*ptr)->move_field(diff);
+    (*ptr)->move_field_offset(diff);
   } while (*(++ptr));
   DBUG_VOID_RETURN;
 }
@@ -3221,7 +3233,7 @@ void set_key_field_ptr(KEY *key_info, const byte *new_buf,
 
   do
   {
-    key_part->field->move_field(diff);
+    key_part->field->move_field_offset(diff);
     key_part++;
   } while (++i < key_parts);
   DBUG_VOID_RETURN;