Table definition cache, part 2

The table opening process now works the following way:
- Create common TABLE_SHARE object
- Read the .frm file and unpack it into the TABLE_SHARE object
- Create a TABLE object based on the information in the TABLE_SHARE
  object and open a handler to the table object

Other noteworthy changes:
- In TABLE_SHARE the most common strings are now LEX_STRING's
- Better error message when table is not found
- Variable table_cache is now renamed 'table_open_cache'
- New variable 'table_definition_cache' that is the number of table defintions that will be cached
- strxnmov() calls are now fixed to avoid overflows
- strxnmov() will now always add one end \0 to result
- engine objects are now created with a TABLE_SHARE object instead of a TABLE object.
- After creating a field object one must call field->init(table) before using it

- For a busy system this change will give you:
 - Less memory usage for table object
 - Faster opening of tables (if it's has been in use or is in table definition cache)
 - Allow you to cache many table definitions objects
 - Faster drop of table


mysql-test/mysql-test-run.sh:
  Fixed some problems with --gdb option
  Test both with socket and tcp/ip port that all old servers are killed
mysql-test/r/flush_table.result:
  More tests with lock table with 2 threads + flush table
mysql-test/r/information_schema.result:
  Removed old (now wrong) result
mysql-test/r/innodb.result:
  Better error messages (thanks to TDC patch)
mysql-test/r/merge.result:
  Extra flush table test
mysql-test/r/ndb_bitfield.result:
  Better error messages (thanks to TDC patch)
mysql-test/r/ndb_partition_error.result:
  Better error messages (thanks to TDC patch)
mysql-test/r/query_cache.result:
  Remove tables left from old tests
mysql-test/r/temp_table.result:
  Test truncate with temporary tables
mysql-test/r/variables.result:
  Table_cache -> Table_open_cache
mysql-test/t/flush_table.test:
  More tests with lock table with 2 threads + flush table
mysql-test/t/merge.test:
  Extra flush table test
mysql-test/t/multi_update.test:
  Added 'sleep' to make test predictable
mysql-test/t/query_cache.test:
  Remove tables left from old tests
mysql-test/t/temp_table.test:
  Test truncate with temporary tables
mysql-test/t/variables.test:
  Table_cache -> Table_open_cache
mysql-test/valgrind.supp:
  Remove warning that may happens becasue threads dies in different order
mysys/hash.c:
  Fixed wrong DBUG_PRINT
mysys/mf_dirname.c:
  More DBUG
mysys/mf_pack.c:
  Better comment
mysys/mf_tempdir.c:
  More DBUG
  Ensure that we call cleanup_dirname() on all temporary directory paths.
  
  If we don't do this, we will get a failure when comparing temporary table
  names as in some cases the temporary table name is run through convert_dirname())
mysys/my_alloc.c:
  Indentation fix
sql/examples/ha_example.cc:
  We new use TABLE_SHARE instead of TABLE when creating engine handlers
sql/examples/ha_example.h:
  We new use TABLE_SHARE instead of TABLE when creating engine handlers
sql/examples/ha_tina.cc:
  We new use TABLE_SHARE instead of TABLE when creating engine handlers
sql/examples/ha_tina.h:
  We new use TABLE_SHARE instead of TABLE when creating engine handlers
sql/field.cc:
  Update for table definition cache:
  - Field creation now takes TABLE_SHARE instead of TABLE as argument
    (This is becasue field definitions are now cached in TABLE_SHARE)
    When a field is created, one now must call field->init(TABLE) before using it
  - Use s->db instead of s->table_cache_key
  - Added Field::clone() to create a field in TABLE from a field in TABLE_SHARE
  - make_field() takes TABLE_SHARE as argument instead of TABLE
  - move_field() -> move_field_offset()
sql/field.h:
  Update for table definition cache:
  - Field creation now takes TABLE_SHARE instead of TABLE as argument
    (This is becasue field definitions are now cached in TABLE_SHARE)
    When a field is created, one now must call field->init(TABLE) before using it
  - Added Field::clone() to create a field in TABLE from a field in TABLE_SHARE
  - make_field() takes TABLE_SHARE as argument instead of TABLE
  - move_field() -> move_field_offset()
sql/ha_archive.cc:
  We new use TABLE_SHARE instead of TABLE when creating engine handlers
sql/ha_archive.h:
  We new use TABLE_SHARE instead of TABLE when creating engine handlers
sql/ha_berkeley.cc:
  We new use TABLE_SHARE instead of TABLE when creating engine handlers
  Changed name of argument create() to not hide internal 'table' variable.
  table->s  -> table_share
sql/ha_berkeley.h:
  We new use TABLE_SHARE instead of TABLE when creating engine handlers
sql/ha_blackhole.cc:
  We new use TABLE_SHARE instead of TABLE when creating engine handlers
sql/ha_blackhole.h:
  We new use TABLE_SHARE instead of TABLE when creating engine handlers
sql/ha_federated.cc:
  We new use TABLE_SHARE instead of TABLE when creating engine handlers
  Fixed comments
  Remove index variable and replace with pointers (simple optimization)
  move_field() -> move_field_offset()
  Removed some strlen() calls
sql/ha_federated.h:
  We new use TABLE_SHARE instead of TABLE when creating engine handlers
sql/ha_heap.cc:
  We new use TABLE_SHARE instead of TABLE when creating engine handlers
  Simplify delete_table() and create() as the given file names are now without extension
sql/ha_heap.h:
  We new use TABLE_SHARE instead of TABLE when creating engine handlers
sql/ha_innodb.cc:
  We new use TABLE_SHARE instead of TABLE when creating engine handlers
sql/ha_innodb.h:
  We new use TABLE_SHARE instead of TABLE when creating engine handlers
sql/ha_myisam.cc:
  We new use TABLE_SHARE instead of TABLE when creating engine handlers
  Remove not needed fn_format()
  Fixed for new table->s structure
sql/ha_myisam.h:
  We new use TABLE_SHARE instead of TABLE when creating engine handlers
sql/ha_myisammrg.cc:
  We new use TABLE_SHARE instead of TABLE when creating engine handlers
  Don't set 'is_view' for MERGE tables
  Use new interface to find_temporary_table()
sql/ha_myisammrg.h:
  We new use TABLE_SHARE instead of TABLE when creating engine handlers
  Added flag HA_NO_COPY_ON_ALTER
sql/ha_ndbcluster.cc:
  We new use TABLE_SHARE instead of TABLE when creating engine handlers
  Fixed wrong calls to strxnmov()
  Give error HA_ERR_TABLE_DEF_CHANGED if table definition has changed
  drop_table -> intern_drop_table()
  table->s -> table_share
  Move part_info to TABLE
  Fixed comments & DBUG print's
  New arguments to print_error()
sql/ha_ndbcluster.h:
  We new use TABLE_SHARE instead of TABLE when creating engine handlers
sql/ha_partition.cc:
  We new use TABLE_SHARE instead of TABLE when creating engine handlers
  We can't set up or use part_info when creating handler as there is not yet any table object
  New ha_intialise() to work with TDC (Done by Mikael)
sql/ha_partition.h:
  We new use TABLE_SHARE instead of TABLE when creating engine handlers
  Got set_part_info() from Mikael
sql/handler.cc:
  We new use TABLE_SHARE instead of TABLE when creating engine handlers
  ha_delete_table() now also takes database as an argument
  handler::ha_open() now takes TABLE as argument
  ha_open() now calls ha_allocate_read_write_set()
  Simplify ha_allocate_read_write_set()
  Remove ha_deallocate_read_write_set()
  Use table_share (Cached by table definition cache)
sql/handler.h:
  New table flag: HA_NO_COPY_ON_ALTER (used by merge tables)
  Remove ha_deallocate_read_write_set()
  get_new_handler() now takes TABLE_SHARE as argument
  ha_delete_table() now gets database as argument
sql/item.cc:
  table_name and db are now LEX_STRING objects
  When creating fields, we have now have to call field->init(table)
  move_field -> move_field_offset()
sql/item.h:
  tmp_table_field_from_field_type() now takes an extra paramenter 'fixed_length' to allow one to force usage of CHAR
   instead of BLOB
sql/item_cmpfunc.cc:
  Fixed call to tmp_table_field_from_field_type()
sql/item_create.cc:
  Assert if new not handled cast type
sql/item_func.cc:
  When creating fields, we have now have to call field->init(table)
  dummy_table used by 'sp' now needs a TABLE_SHARE object
sql/item_subselect.cc:
  Trivial code cleanups
sql/item_sum.cc:
  When creating fields, we have now have to call field->init(table)
sql/item_timefunc.cc:
  Item_func_str_to_date::tmp_table_field() now replaced by call to
   tmp_table_field_from_field_type() (see item_timefunc.h)
sql/item_timefunc.h:
  Simply tmp_table_field()
sql/item_uniq.cc:
  When creating fields, we have now have to call field->init(table)
sql/key.cc:
  Added 'KEY' argument to 'find_ref_key' to simplify code
sql/lock.cc:
  More debugging
  Use create_table_def_key() to create key for table cache
  Allocate TABLE_SHARE properly when creating name lock
  Fix that locked_table_name doesn't test same table twice
sql/mysql_priv.h:
  New functions for table definition cache
  New interfaces to a lot of functions.
  New faster interface to find_temporary_table() and close_temporary_table()
sql/mysqld.cc:
  Added support for table definition cache of size 'table_def_size'
  Fixed som calls to strnmov()
  Changed name of 'table_cache' to 'table_open_cache'
sql/opt_range.cc:
  Use new interfaces
  Fixed warnings from valgrind
sql/parse_file.cc:
  Safer calls to strxnmov()
  Fixed typo
sql/set_var.cc:
  Added variable 'table_definition_cache'
  Variable table_cache renamed to 'table_open_cache'
sql/slave.cc:
  Use new interface
sql/sp.cc:
  Proper use of TABLE_SHARE
sql/sp_head.cc:
  Remove compiler warnings
  We have now to call field->init(table)
sql/sp_head.h:
  Pointers to parsed strings are now const
sql/sql_acl.cc:
  table_name is now a LEX_STRING
sql/sql_base.cc:
  Main implementation of table definition cache
  (The #ifdef's are there for the future when table definition cache will replace open table cache)
  Now table definitions are cached indepndent of open tables, which will speed up things when a table is in use at once from several places
  Views are not yet cached; For the moment we only cache if a table is a view or not.
  
  Faster implementation of find_temorary_table()
  Replace 'wait_for_refresh()' with the more general function 'wait_for_condition()'
  Drop table is slightly faster as we can use the table definition cache to know the type of the table
sql/sql_cache.cc:
  table_cache_key and table_name are now LEX_STRING
  'sDBUG print fixes
sql/sql_class.cc:
  table_cache_key is now a LEX_STRING
  safer strxnmov()
sql/sql_class.h:
  Added number of open table shares (table definitions)
sql/sql_db.cc:
  safer strxnmov()
sql/sql_delete.cc:
  Use new interface to find_temporary_table()
sql/sql_derived.cc:
  table_name is now a LEX_STRING
sql/sql_handler.cc:
  TABLE_SHARE->db and TABLE_SHARE->table_name are now LEX_STRING's
sql/sql_insert.cc:
  TABLE_SHARE->db and TABLE_SHARE->table_name are now LEX_STRING's
sql/sql_lex.cc:
  Make parsed string a const (to quickly find out if anything is trying to change the query string)
sql/sql_lex.h:
  Make parsed string a const (to quickly find out if anything is trying to change the query string)
sql/sql_load.cc:
  Safer strxnmov()
sql/sql_parse.cc:
  Better error if wrong DB name
sql/sql_partition.cc:
  part_info moved to TABLE from TABLE_SHARE
  Indentation changes
sql/sql_select.cc:
  Indentation fixes
  Call field->init(TABLE) for new created fields
  Update create_tmp_table() to use TABLE_SHARE properly
sql/sql_select.h:
  Call field->init(TABLE) for new created fields
sql/sql_show.cc:
  table_name is now a LEX_STRING
  part_info moved to TABLE
sql/sql_table.cc:
  Use table definition cache to speed up delete of tables
  Fixed calls to functions with new interfaces
  Don't use 'share_not_to_be_used'
  Instead of doing openfrm() when doing repair, we now have to call
  get_table_share() followed by open_table_from_share().
  Replace some fn_format() with faster unpack_filename().
  Safer strxnmov()
  part_info is now in TABLE
  Added Mikaels patch for partition and ALTER TABLE
  Instead of using 'TABLE_SHARE->is_view' use 'table_flags() & HA_NO_COPY_ON_ALTER
sql/sql_test.cc:
  table_name and table_cache_key are now LEX_STRING's
sql/sql_trigger.cc:
  TABLE_SHARE->db and TABLE_SHARE->table_name are now LEX_STRING's
  safer strxnmov()
  Removed compiler warnings
sql/sql_update.cc:
  Call field->init(TABLE) after field is created
sql/sql_view.cc:
  safer strxnmov()
  Create common TABLE_SHARE object for views to allow us to cache if table is a view
sql/structs.h:
  Added SHOW_TABLE_DEFINITIONS
sql/table.cc:
  Creation and destruct of TABLE_SHARE objects that are common for many TABLE objects
  
  The table opening process now works the following way:
  - Create common TABLE_SHARE object
  - Read the .frm file and unpack it into the TABLE_SHARE object
  - Create a TABLE object based on the information in the TABLE_SHARE
    object and open a handler to the table object
  
  open_table_def() is written in such a way that it should be trival to add parsing of the .frm files in new formats
sql/table.h:
  TABLE objects for the same database table now share a common TABLE_SHARE object
  In TABLE_SHARE the most common strings are now LEX_STRING's
sql/unireg.cc:
  Changed arguments to rea_create_table() to have same order as other functions
  Call field->init(table) for new created fields
sql/unireg.h:
  Added OPEN_VIEW
strings/strxnmov.c:
  Change strxnmov() to always add end \0
  This makes usage of strxnmov() safer as most of MySQL code assumes that strxnmov() will create a null terminated string

1 files changed, 46 insertions, 50 deletions

diff --git a/sql/ha_federated.cc b/sql/ha_federated.cc
index 57dc51edb90..e50961a5700 100644
--- a/sql/ha_federated.cc
+++ b/sql/ha_federated.cc
@@ -355,15 +355,12 @@
 
 #include "m_string.h"
 /* Variables for federated share methods */
-static HASH federated_open_tables;              // Hash used to track open
-                                                // tables
-pthread_mutex_t federated_mutex;                // This is the mutex we use to
-                                                // init the hash
-static int federated_init= FALSE;               // Variable for checking the
-                                                // init state of hash
+static HASH federated_open_tables;              // To track open tables
+pthread_mutex_t federated_mutex;                // To init the hash
+static int federated_init= FALSE;               // Checking the state of hash
 
 /* Static declaration for handerton */
-static handler *federated_create_handler(TABLE *table);
+static handler *federated_create_handler(TABLE_SHARE *table);
 static int federated_commit(THD *thd, bool all);
 static int federated_rollback(THD *thd, bool all);
 
@@ -403,13 +400,13 @@ handlerton federated_hton= {
 };
 
 
-static handler *federated_create_handler(TABLE *table)
+static handler *federated_create_handler(TABLE_SHARE *table)
 {
   return new ha_federated(table);
 }
 
 
-/* Function we use in the creation of our hash to get key.  */
+/* Function we use in the creation of our hash to get key */
 
 static byte *federated_get_key(FEDERATED_SHARE *share, uint *length,
                                my_bool not_used __attribute__ ((unused)))
@@ -438,13 +435,11 @@ bool federated_db_init()
   if (hash_init(&federated_open_tables, system_charset_info, 32, 0, 0,
                     (hash_get_key) federated_get_key, 0, 0))
   {
-    VOID(pthread_mutex_destroy(&federated_mutex));
-  }
-  else
-  {
     federated_init= TRUE;
     DBUG_RETURN(FALSE);
   }
+
+  VOID(pthread_mutex_destroy(&federated_mutex));
 error:
   have_federated_db= SHOW_OPTION_DISABLED;	// If we couldn't use handler
   DBUG_RETURN(TRUE);
@@ -456,7 +451,6 @@ error:
 
   SYNOPSIS
     federated_db_end()
-    void
 
   RETURN
     FALSE       OK
@@ -473,6 +467,7 @@ int federated_db_end(ha_panic_function type)
   return 0;
 }
 
+
 /*
  Check (in create) whether the tables exists, and that it can be connected to
 
@@ -605,12 +600,12 @@ static int parse_url_error(FEDERATED_SHARE *share, TABLE *table, int error_num)
 
   SYNOPSIS
     parse_url()
-      share               pointer to FEDERATED share
-      table               pointer to current TABLE class
-      table_create_flag   determines what error to throw
+    share               pointer to FEDERATED share
+    table               pointer to current TABLE class
+    table_create_flag   determines what error to throw
 
   DESCRIPTION
-    populates the share with information about the connection
+    Populates the share with information about the connection
     to the foreign database that will serve as the data source.
     This string must be specified (currently) in the "comment" field,
     listed in the CREATE TABLE statement.
@@ -629,7 +624,7 @@ static int parse_url_error(FEDERATED_SHARE *share, TABLE *table, int error_num)
   ***IMPORTANT***
   Currently, only "mysql://" is supported.
 
-    'password' and 'port' are both optional.
+  'password' and 'port' are both optional.
 
   RETURN VALUE
     0           success
@@ -739,8 +734,8 @@ static int parse_url(FEDERATED_SHARE *share, TABLE *table,
   }
 
   DBUG_PRINT("info",
-             ("scheme %s username %s password %s \
-              hostname %s port %d database %s tablename %s",
+             ("scheme: %s  username: %s  password: %s \
+               hostname: %s  port: %d  database: %s  tablename: %s",
               share->scheme, share->username, share->password,
               share->hostname, share->port, share->database,
               share->table_name));
@@ -756,7 +751,7 @@ error:
 ** FEDERATED tables
 *****************************************************************************/
 
-ha_federated::ha_federated(TABLE *table_arg)
+ha_federated::ha_federated(TABLE_SHARE *table_arg)
   :handler(&federated_hton, table_arg),
   mysql(0), stored_result(0), scan_flag(0),
   ref_length(sizeof(MYSQL_ROW_OFFSET)), current_position(0)
@@ -770,8 +765,8 @@ ha_federated::ha_federated(TABLE *table_arg)
 
   SYNOPSIS
     convert_row_to_internal_format()
-      record    Byte pointer to record
-      row       MySQL result set row from fetchrow()
+    record    Byte pointer to record
+    row       MySQL result set row from fetchrow()
 
   DESCRIPTION
     This method simply iterates through a row returned via fetchrow with
@@ -782,7 +777,7 @@ ha_federated::ha_federated(TABLE *table_arg)
 
   RETURN VALUE
     0   After fields have had field values stored from record
- */
+*/
 
 uint ha_federated::convert_row_to_internal_format(byte *record, MYSQL_ROW row)
 {
@@ -793,24 +788,23 @@ uint ha_federated::convert_row_to_internal_format(byte *record, MYSQL_ROW row)
   lengths= mysql_fetch_lengths(stored_result);
   memset(record, 0, table->s->null_bytes);
 
-  for (field= table->field; *field; field++)
+  for (field= table->field; *field; field++, row++, lengths++)
   {
     /*
       index variable to move us through the row at the
       same iterative step as the field
     */
-    int x= field - table->field;
     my_ptrdiff_t old_ptr;
     old_ptr= (my_ptrdiff_t) (record - table->record[0]);
-    (*field)->move_field(old_ptr);
-    if (!row[x])
+    (*field)->move_field_offset(old_ptr);
+    if (!*row)
       (*field)->set_null();
     else
     {
       (*field)->set_notnull();
-      (*field)->store(row[x], lengths[x], &my_charset_bin);
+      (*field)->store(*row, *lengths, &my_charset_bin);
     }
-    (*field)->move_field(-old_ptr);
+    (*field)->move_field_offset(-old_ptr);
   }
 
   DBUG_RETURN(0);
@@ -1215,8 +1209,8 @@ bool ha_federated::create_where_from_key(String *to,
             DBUG_RETURN(1);
         }
         else
-          /* LIKE */
         {
+          /* LIKE */
           if (emit_key_part_name(&tmp, key_part) ||
               tmp.append(FEDERATED_LIKE) ||
               emit_key_part_element(&tmp, key_part, needs_quotes, 1, ptr,
@@ -1328,16 +1322,16 @@ static FEDERATED_SHARE *get_share(const char *table_name, TABLE *table)
   /*
     In order to use this string, we must first zero it's length,
     or it will contain garbage
-   */
+  */
   query.length(0);
 
   pthread_mutex_lock(&federated_mutex);
-  tmp_table_name= (char *)table->s->table_name;
-  tmp_table_name_length= (uint) strlen(tmp_table_name);
+  tmp_table_name=        table->s->table_name.str;
+  tmp_table_name_length= table->s->table_name.length;
 
   if (!(share= (FEDERATED_SHARE *) hash_search(&federated_open_tables,
                                                (byte*) table_name,
-                                               strlen(table_name))))
+                                               tmp_table_name_length)))
   {
     query.set_charset(system_charset_info);
     query.append(FEDERATED_SELECT);
@@ -1348,7 +1342,7 @@ static FEDERATED_SHARE *get_share(const char *table_name, TABLE *table)
       query.append(FEDERATED_BTICK);
       query.append(FEDERATED_COMMA);
     }
-    query.length(query.length()- strlen(FEDERATED_COMMA));
+    query.length(query.length()- FEDERATED_COMMA_LEN);
     query.append(FEDERATED_FROM);
     query.append(FEDERATED_BTICK);
 
@@ -1372,7 +1366,6 @@ static FEDERATED_SHARE *get_share(const char *table_name, TABLE *table)
     share->select_query= select_query;
     strmov(share->select_query, query.ptr());
     share->use_count= 0;
-    share->table_name_length= strlen(share->table_name);
     DBUG_PRINT("info",
                ("share->select_query %s", share->select_query));
 
@@ -1486,8 +1479,8 @@ int ha_federated::open(const char *name, int mode, uint test_if_locked)
   }
   /*
     Since we do not support transactions at this version, we can let the client
-    API silently reconnect. For future versions, we will need more logic to deal
-    with transactions
+    API silently reconnect. For future versions, we will need more logic to
+    deal with transactions
   */
   mysql->reconnect= 1;
 
@@ -1563,6 +1556,7 @@ inline uint field_in_record_is_null(TABLE *table,
   DBUG_RETURN(0);
 }
 
+
 /*
   write_row() inserts a row. No extra() hint is given currently if a bulk load
   is happeneding. buf() is a byte array of data. You can use the field
@@ -1819,15 +1813,15 @@ int ha_federated::update_row(const byte *old_data, byte *new_data)
   update_string.append(FEDERATED_BTICK);
   update_string.append(FEDERATED_SET);
 
-/*
-  In this loop, we want to match column names to values being inserted
-  (while building INSERT statement).
+  /*
+    In this loop, we want to match column names to values being inserted
+    (while building INSERT statement).
 
-  Iterate through table->field (new data) and share->old_filed (old_data)
-  using the same index to created an SQL UPDATE statement, new data is
-  used to create SET field=value and old data is used to create WHERE
-  field=oldvalue
- */
+    Iterate through table->field (new data) and share->old_filed (old_data)
+    using the same index to created an SQL UPDATE statement, new data is
+    used to create SET field=value and old data is used to create WHERE
+    field=oldvalue
+  */
 
   for (Field **field= table->field; *field; field++)
   {
@@ -2048,7 +2042,7 @@ int ha_federated::index_read_idx(byte *buf, uint index, const byte *key,
    This basically says that the record in table->record[0] is legal,
    and that it is ok to use this record, for whatever reason, such
    as with a join (without it, joins will not work)
- */
+  */
   table->status= 0;
 
   retval= rnd_next(buf);
@@ -2070,7 +2064,7 @@ int ha_federated::index_init(uint keynr, bool sorted)
 {
   DBUG_ENTER("ha_federated::index_init");
   DBUG_PRINT("info",
-             ("table: '%s'  key: %d", table->s->table_name, keynr));
+             ("table: '%s'  key: %d", table->s->table_name.str, keynr));
   active_index= keynr;
   DBUG_RETURN(0);
 }
@@ -2251,6 +2245,7 @@ int ha_federated::rnd_end()
   DBUG_RETURN(retval);
 }
 
+
 int ha_federated::index_end(void)
 {
   DBUG_ENTER("ha_federated::index_end");
@@ -2258,6 +2253,7 @@ int ha_federated::index_end(void)
   DBUG_RETURN(0);
 }
 
+
 /*
   This is called for each row of the table scan. When you run out of records
   you should return HA_ERR_END_OF_FILE. Fill buff up with the row information.