Applied innodb snapshot ss792 and ss854.

This ChangeSet will be null-merged to mysql-5.1.

Bugs fixed:
- Bug #21638:  InnoDB: possible segfault in page_copy_rec_list_end_no_locks
  If prefix_len is specified, write it to the insert buffer instead of type->len
- bug #10746:  InnoDB: Error: stored_select_lock_type is 0 inside ::start_stmt()!
  INSERT ... SELECT uses LOCK_NONE in innodb_locks_unsafe_for_binlog mode, so do
  not check for LOCK_X or LOCK_S in start_stmt()


innobase/ibuf/ibuf0ibuf.c:
  Applied innodb snapshot ss792 and ss854.
  
  ibuf_entry_build(): Write prefix_len to the insert buffer instead of type->len
  when prefix_len is specified.  Otherwise, btr_page_reorganize() during
  insert buffer merge would fail on ROW_FORMAT=COMPACT tables.  (Bug #21638)
innobase/include/data0type.h:
  Applied innodb snapshot ss792 and ss854.
  
  ibuf_entry_build(): Write prefix_len to the insert buffer instead of type->len
  when prefix_len is specified.  Otherwise, btr_page_reorganize() during
  insert buffer merge would fail on ROW_FORMAT=COMPACT tables.  (Bug #21638)
innobase/include/data0type.ic:
  Applied innodb snapshot ss792 and ss854.
  
  ibuf_entry_build(): Write prefix_len to the insert buffer instead of type->len
  when prefix_len is specified.  Otherwise, btr_page_reorganize() during
  insert buffer merge would fail on ROW_FORMAT=COMPACT tables.  (Bug #21638)
sql/ha_innodb.cc:
  Applied innodb snapshot ss792 and ss854.
  
  ha_innobase::start_stmt(): Remove the check for
  prebuilt->stored_select_lock_type being LOCK_X or LOCK_S.
  This would cause false alarms with INSERT ... SELECT, which would use
  LOCK_NONE in innodb_locks_unsafe_for_binlog mode.  (Bug #10746)

2 files changed, 10 insertions, 3 deletions

diff --git a/innobase/include/data0type.h b/innobase/include/data0type.h
index 30c0f732c34..abfbc24e336 100644
--- a/innobase/include/data0type.h
+++ b/innobase/include/data0type.h
@@ -366,7 +366,9 @@ dtype_new_store_for_order_and_null_size(
 	byte*		buf,	/* in: buffer for
 				DATA_NEW_ORDER_NULL_TYPE_BUF_SIZE
 				bytes where we store the info */
-	dtype_t*	type);	/* in: type struct */
+	dtype_t*	type,	/* in: type struct */
+	ulint		prefix_len);/* in: prefix length to
+				replace type->len, or 0 */
 /**************************************************************************
 Reads to a type the stored information which determines its alphabetical
 ordering and the storage size of an SQL NULL value. This is the 4.1.x storage
diff --git a/innobase/include/data0type.ic b/innobase/include/data0type.ic
index ad0f02fa63d..032c1d7efd7 100644
--- a/innobase/include/data0type.ic
+++ b/innobase/include/data0type.ic
@@ -230,11 +230,14 @@ dtype_new_store_for_order_and_null_size(
 	byte*		buf,	/* in: buffer for
 				DATA_NEW_ORDER_NULL_TYPE_BUF_SIZE
 				bytes where we store the info */
-	dtype_t*	type)	/* in: type struct */
+	dtype_t*	type,	/* in: type struct */
+	ulint		prefix_len)/* in: prefix length to
+				replace type->len, or 0 */
 {
 #if 6 != DATA_NEW_ORDER_NULL_TYPE_BUF_SIZE
 #error "6 != DATA_NEW_ORDER_NULL_TYPE_BUF_SIZE"
 #endif
+	ulint	len;
 
 	buf[0] = (byte)(type->mtype & 0xFFUL);
 
@@ -249,7 +252,9 @@ dtype_new_store_for_order_and_null_size(
 
 	buf[1] = (byte)(type->prtype & 0xFFUL);
 
-	mach_write_to_2(buf + 2, type->len & 0xFFFFUL);
+	len = prefix_len ? prefix_len : type->len;
+
+	mach_write_to_2(buf + 2, len & 0xFFFFUL);
 
 	ut_ad(dtype_get_charset_coll(type->prtype) < 256);
 	mach_write_to_2(buf + 4, dtype_get_charset_coll(type->prtype));