WL#5151: Conversion between different types when replicating

Row-based replication requires the types of columns on the
master and slave to be approximately the same (some safe
conversions between strings are allowed), but does not
allow safe conversions between fields of similar types such
as TINYINT and INT.

This patch implement type conversions between similar fields
on the master and slave.

The conversions are controlled using a new variable
SLAVE_TYPE_CONVERSIONS of type SET('ALL_LOSSY','ALL_NON_LOSSY').

Non-lossy conversions are any conversions that do not run the
risk of losing any information, while lossy conversions can
potentially truncate the value. The column definitions are
checked to decide if the conversion is acceptable.

If neither conversion is enabled, it is required that the
definitions of the columns are identical on master and slave.

Conversion is done by creating an internal conversion table,
unpacking the master data into it, and then copy the data to
the real table on the slave.

.bzrignore:
  New files added
client/Makefile.am:
  New files added
client/mysqlbinlog.cc:
  Functions in rpl_utility.cc is now needed by mysqlbinlog.cc.
libmysqld/Makefile.am:
  New files added
mysql-test/extra/rpl_tests/check_type.inc:
  Test include file to check a single type conversion.
mysql-test/extra/rpl_tests/rpl_extraSlave_Col.test:
  Switching to use INT instead of TEXT for column that should not have matching types.
mysql-test/extra/rpl_tests/rpl_row_basic.test:
  Adding code to enable type conversions for BIT tests since InnoDB
  cannot handle them properly due to incorrect information stored as
  metadata.
mysql-test/extra/rpl_tests/type_conversions.test:
  Test file to check a set of type conversions
  with current settings of slave_type_conversions.
mysql-test/suite/rpl/t/rpl_typeconv.test:
  Test file to test conversions from master to slave with
  all possible values for slave_type_conversions.
  
  The test also checks that the slave_type_conversions
  variable works as expected.
sql/field.cc:
  Changing definition of compatible_field_size to both check if 
  two field with identical base types are compatible and give an
  order between them if they are compatible.
  
  This only implement checking on the slave, so it will not affect
  replication from an old master to a new slave.
sql/field.h:
  Changing prototypes for functions:
  - compatible_field_size()
  - init_for_tmp_table()
  - row_pack_length()
sql/log_event.cc:
  Changing compability checks to build a conversion table if the fields
  are compatible, but does not have the same base type.
sql/log_event_old.cc:
  Changing compability checks to build a conversion table if the fields
  are compatible, but does not have the same base type.
sql/mysql_priv.h:
  Adding global option variable for SLAVE_TYPE_CONVERSIONS
sql/mysqld.cc:
  Adding SLAVE_TYPE_CONVERSIONS global server variable.
sql/rpl_record.cc:
  Changing unpack_row to use the conversion table if present.
sql/rpl_rli.h:
  Removing function get_tabledef and replacing it with get_table_data().
  This function retrieve data for table opened for replication, not just
  table definition.
sql/rpl_utility.cc:
  Function table_def::compatible_with is changed to compare table on master
  and slave for compatibility and generate a conversions table if they are
  compatible.
  
  Computing real type of fields from metadata for ENUM and SET types.
  Computing pack_length correctly for ENUM, SET, and BLOB types.
  
  Adding optimization to not check compatibility if no
  slave type conversions are enabled.
sql/rpl_utility.h:
  Changing prototypes since implementation has changed.
  
  Modifying table_def::type() to return real type instead of stored type.
sql/set_var.cc:
  Adding SLAVE_TYPE_CONVERSIONS variable.
sql/set_var.h:
  Adding SLAVE_TYPE_CONVERSIONS variable.
sql/share/errmsg.txt:
  Adding error messages for slave type conversions.
sql/sql_class.h:
  Adding SLAVE_TYPE_CONVERSIONS variable.
sql/sql_select.cc:
  Correcting create_virtual_tmp_table() to compute null bit positions
  correctly in the presence of bit fields.

1 files changed, 78 insertions, 108 deletions

diff --git a/sql/rpl_utility.h b/sql/rpl_utility.h
index 1f4ca246ff1..97ee668c82d 100644
--- a/sql/rpl_utility.h
+++ b/sql/rpl_utility.h
@@ -21,6 +21,7 @@
 #endif
 
 #include "mysql_priv.h"
+#include "mysql_com.h"
 
 class Relay_log_info;
 
@@ -44,115 +45,18 @@ class table_def
 {
 public:
   /**
-    Convenience declaration of the type of the field type data in a
-    table map event.
-  */
-  typedef unsigned char field_type;
-
-  /**
     Constructor.
 
-    @param types Array of types
+    @param types Array of types, each stored as a byte
     @param size  Number of elements in array 'types'
     @param field_metadata Array of extra information about fields
     @param metadata_size Size of the field_metadata array
     @param null_bitmap The bitmap of fields that can be null
    */
-  table_def(field_type *types, ulong size, uchar *field_metadata, 
-      int metadata_size, uchar *null_bitmap)
-    : m_size(size), m_type(0), m_field_metadata_size(metadata_size),
-      m_field_metadata(0), m_null_bits(0), m_memory(NULL)
-  {
-    m_memory= (uchar *)my_multi_malloc(MYF(MY_WME),
-                                       &m_type, size,
-                                       &m_field_metadata,
-                                       size * sizeof(uint16),
-                                       &m_null_bits, (size + 7) / 8,
-                                       NULL);
+  table_def(unsigned char *types, ulong size, uchar *field_metadata,
+            int metadata_size, uchar *null_bitmap);
 
-    bzero(m_field_metadata, size * sizeof(uint16));
-
-    if (m_type)
-      memcpy(m_type, types, size);
-    else
-      m_size= 0;
-    /*
-      Extract the data from the table map into the field metadata array
-      iff there is field metadata. The variable metadata_size will be
-      0 if we are replicating from an older version server since no field
-      metadata was written to the table map. This can also happen if 
-      there were no fields in the master that needed extra metadata.
-    */
-    if (m_size && metadata_size)
-    { 
-      int index= 0;
-      for (unsigned int i= 0; i < m_size; i++)
-      {
-        switch (m_type[i]) {
-        case MYSQL_TYPE_TINY_BLOB:
-        case MYSQL_TYPE_BLOB:
-        case MYSQL_TYPE_MEDIUM_BLOB:
-        case MYSQL_TYPE_LONG_BLOB:
-        case MYSQL_TYPE_DOUBLE:
-        case MYSQL_TYPE_FLOAT:
-        {
-          /*
-            These types store a single byte.
-          */
-          m_field_metadata[i]= field_metadata[index];
-          index++;
-          break;
-        }
-        case MYSQL_TYPE_SET:
-        case MYSQL_TYPE_ENUM:
-        case MYSQL_TYPE_STRING:
-        {
-          uint16 x= field_metadata[index++] << 8U; // real_type
-          x+= field_metadata[index++];            // pack or field length
-          m_field_metadata[i]= x;
-          break;
-        }
-        case MYSQL_TYPE_BIT:
-        {
-          uint16 x= field_metadata[index++]; 
-          x = x + (field_metadata[index++] << 8U);
-          m_field_metadata[i]= x;
-          break;
-        }
-        case MYSQL_TYPE_VARCHAR:
-        {
-          /*
-            These types store two bytes.
-          */
-          char *ptr= (char *)&field_metadata[index];
-          m_field_metadata[i]= uint2korr(ptr);
-          index= index + 2;
-          break;
-        }
-        case MYSQL_TYPE_NEWDECIMAL:
-        {
-          uint16 x= field_metadata[index++] << 8U; // precision
-          x+= field_metadata[index++];            // decimals
-          m_field_metadata[i]= x;
-          break;
-        }
-        default:
-          m_field_metadata[i]= 0;
-          break;
-        }
-      }
-    }
-    if (m_size && null_bitmap)
-       memcpy(m_null_bits, null_bitmap, (m_size + 7) / 8);
-  }
-
-  ~table_def() {
-    my_free(m_memory, MYF(0));
-#ifndef DBUG_OFF
-    m_type= 0;
-    m_size= 0;
-#endif
-  }
+  ~table_def();
 
   /**
     Return the number of fields there is type data for.
@@ -171,10 +75,40 @@ public:
     <code>index</code>. Currently, only the type identifier is
     returned.
    */
-  field_type type(ulong index) const
+  enum_field_types type(ulong index) const
   {
     DBUG_ASSERT(index < m_size);
-    return m_type[index];
+    /*
+      If the source type is MYSQL_TYPE_STRING, it can in reality be
+      either MYSQL_TYPE_STRING, MYSQL_TYPE_ENUM, or MYSQL_TYPE_SET, so
+      we might need to modify the type to get the real type.
+    */
+    enum_field_types source_type= static_cast<enum_field_types>(m_type[index]);
+    uint16 source_metadata= m_field_metadata[index];
+    switch (source_type)
+    {
+    case MYSQL_TYPE_STRING:
+    {
+      int real_type= source_metadata >> 8;
+      if (real_type == MYSQL_TYPE_ENUM || real_type == MYSQL_TYPE_SET)
+        source_type= static_cast<enum_field_types>(real_type);
+      break;
+    }
+
+    /*
+      This type has not been used since before row-based replication,
+      so we can safely assume that it really is MYSQL_TYPE_NEWDATE.
+    */
+    case MYSQL_TYPE_DATE:
+      source_type= MYSQL_TYPE_NEWDATE;
+      break;
+
+    default:
+      /* Do nothing */
+      break;
+    }
+
+    return source_type;
   }
 
 
@@ -226,23 +160,58 @@ public:
     with it.
 
     A table definition is compatible with a table if:
-      - the columns types of the table definition is a (not
-        necessarily proper) prefix of the column type of the table, or
-      - the other way around
+      - The columns types of the table definition is a (not
+        necessarily proper) prefix of the column type of the table.
 
+      - The other way around.
+
+      - Each column on the master that also exists on the slave can be
+        converted according to the current settings of @c
+        SLAVE_TYPE_CONVERSIONS.
+
+    @param thd
     @param rli   Pointer to relay log info
     @param table Pointer to table to compare with.
 
+    @param[out] tmp_table_var Pointer to temporary table for holding
+    conversion table.
+
     @retval 1  if the table definition is not compatible with @c table
     @retval 0  if the table definition is compatible with @c table
   */
 #ifndef MYSQL_CLIENT
-  int compatible_with(Relay_log_info const *rli, TABLE *table) const;
+  bool compatible_with(THD *thd, Relay_log_info *rli, TABLE *table,
+                      TABLE **conv_table_var) const;
+
+  /**
+   Create a virtual in-memory temporary table structure.
+
+   The table structure has records and field array so that a row can
+   be unpacked into the record for further processing.
+
+   In the virtual table, each field that requires conversion will
+   have a non-NULL value, while fields that do not require
+   conversion will have a NULL value.
+
+   Some information that is missing in the events, such as the
+   character set for string types, are taken from the table that the
+   field is going to be pushed into, so the target table that the data
+   eventually need to be pushed into need to be supplied.
+
+   @param thd Thread to allocate memory from.
+   @param rli Relay log info structure, for error reporting.
+   @param target_table Target table for fields.
+
+   @return A pointer to a temporary table with memory allocated in the
+   thread's memroot, NULL if the table could not be created
+   */
+  TABLE *create_conversion_table(THD *thd, Relay_log_info *rli, TABLE *target_table) const;
 #endif
 
+
 private:
   ulong m_size;           // Number of elements in the types array
-  field_type *m_type;                     // Array of type descriptors
+  unsigned char *m_type;  // Array of type descriptors
   uint m_field_metadata_size;
   uint16 *m_field_metadata;
   uchar *m_null_bits;
@@ -260,6 +229,7 @@ struct RPL_TABLE_LIST
 {
   bool m_tabledef_valid;
   table_def m_tabledef;
+  TABLE *m_conv_table;
 };