Added some fixes that should make MyISAM & Aria REPAIR work with more than 4G records

- If one specifies --force twice to myisamchk and aria_chk, then we will try to finnish the repair even if sort_buffer would be too small.
  This was done by dynamically allocate buffer handler objects as long as memory lasts.
- New option for myisamchk and aria_chk: create-missing-keys
- Changed default size of myisam_sort_buffer_size from 8M to 128M.
- Changed default size of sort_buffer_size in aria_chk from 128M to 256M.
- Increased information in error message about 'sort_buffer_size' beeing to small.
- Print also to 'show warnings' if repair was retried.
- Increased size of internal sort-buffer-readers from 16K to 128K
- Changed printing of 'number of records' to use %ll instead of casting to long
- Changed buffer sizes for myisam and aria to use MY_ALIGN_DOWN() to get same number of bytes allocated on different machines.





include/my_global.h:
  Added MY_ALIGN_DOWN() to get previous alignment (for big memory areas)
include/myisam.h:
  Increased size of types to be able to handle more records
include/myisamchk.h:
  Increased size of types to be able to handle more records
  Added T_FORCE_SORT_MEMORY to force repair to work even if sort_buffer would not be big enough
mysql-test/r/myisam.result:
  Updated result
mysql-test/r/mysqld--help.result:
  Updated result
mysql-test/r/repair.result:
  Updated result
mysql-test/suite/maria/maria.result:
  Added test cases for some fixed bugs in MyISAM to verify that Aria doesn't have them too.
mysql-test/suite/maria/maria.test:
  Added test cases for some fixed bugs in MyISAM to verify that Aria doesn't have them too.
mysql-test/suite/maria/maria3.result:
  Updated result after sort buffer size increase
mysql-test/suite/sys_vars/r/aria_sort_buffer_size_basic.result:
  Updated result after sort buffer size increase
mysql-test/suite/sys_vars/r/myisam_sort_buffer_size_basic.result:
  Updated result after sort buffer size increase
mysql-test/suite/sys_vars/t/myisam_sort_buffer_size_basic.test:
  Updated result after sort buffer size increase
mysql-test/t/myisam.test:
  Fixed error messages to not print system specific data
mysql-test/t/repair.test:
  Fixed error messages to not print system specific data
storage/maria/ha_maria.cc:
  Print also to 'show warnings' if repair was retried
  Changed default size of sort_buffer_size from 128M to 256M (same as in mysqld)
storage/maria/ma_check.c:
  Renamed USE_BUFFER_INIT -> PAGE_BUFFER_INIT
storage/maria/ma_sort.c:
  Increased size of internal sort-buffer-readers from 16K to 128K
  Increased size of types to be able to handle more records
  Added support for T_FORCE_SORT_MEMORY
  Don't allocate too many extra BUFFPEK at a time (they are probably not needed)
  Improved error message for "sort_buffer_size is too small"
  Changed printing of 'number of records' to use %ll instead of casting to long
  Fixed bug where maria_update_key_parts() was called too early.
  Fixed bug in detecting result from read_to_buffer().
  Added 'out of memory' checking when calling 'alloc_dynamic()'.
storage/maria/maria_chk.c:
  Added --create-missing-keys
  If one specifies --force twice then we will try to finnish the repair even if sort_buffer would be too small.
  check_param.sort_buffer_length varialble was used with wrong type.
storage/maria/maria_def.h:
  Increased size of types to be able to handle more records
  Use MY_ALIGN_DOWN() to get same number of bytes allocated on different machines
  Renamed USE_BUFFER_INIT -> PAGE_BUFFER_INIT
storage/maria/maria_ftdump.c:
  Renamed USE_BUFFER_INIT -> PAGE_BUFFER_INIT
storage/maria/maria_read_log.c:
  Use PAGE_BUFFER_INIT for page cache
storage/myisam/ha_myisam.cc:
  Changed default size of myisam_sort_buffer_size from 8M to 128M
storage/myisam/mi_check.c:
  Renamed USE_BUFFER_INIT -> KEY_BUFFER_INIT
storage/myisam/myisam_ftdump.c:
  Renamed USE_BUFFER_INIT -> KEY_BUFFER_INIT
storage/myisam/myisamchk.c:
  Added --create-missing-keys
  If one specifies --force twice then we will try to finnish the repair even if sort_buffer would be too small.
  check_param.sort_buffer_length varialble was used with wrong type.
  Renamed USE_BUFFER_INIT -> KEY_BUFFER_INIT
storage/myisam/myisamdef.h:
  Increased SORT_BUFFER_INIT to 64M
  (speeds up repair a lot and most machines have nowadays a lot of memory)
  Use MY_ALIGN_DOWN() to get same number of bytes allocated on different machines
  Renamed USE_BUFFER_INIT -> KEY_BUFFER_INIT
storage/myisam/sort.c:
  Increased size of internal sort-buffer-readers from 16K to 128K
  Increased size of types to be able to handle more records
  Added support for T_FORCE_SORT_MEMORY
  Don't allocate too many extra BUFFPEK at a time (they are probably not needed)
  Improved error message for "sort_buffer_size is too small"
  Changed printing of 'number of records' to use %ll instead of casting to long
  Fixed bug in detecting result from read_to_buffer().
  Added 'out of memory' checking when calling 'alloc_dynamic()'.

11 files changed, 143 insertions, 27 deletions

diff --git a/mysql-test/r/myisam.result b/mysql-test/r/myisam.result
index 2530182ea67..267110be487 100644
--- a/mysql-test/r/myisam.result
+++ b/mysql-test/r/myisam.result
@@ -2355,7 +2355,7 @@ Warnings:
 Warning	1292	Truncated incorrect myisam_sort_buffer_size value: '4'
 REPAIR TABLE t1;
 Table	Op	Msg_type	Msg_text
-test.t1	repair	error	myisam_sort_buffer_size is too small
+test.t1	repair	error	myisam_sort_buffer_size is too small. X
 test.t1	repair	warning	Number of rows changed from 0 to 7168
 test.t1	repair	status	OK
 SET myisam_repair_threads=2;
@@ -2424,7 +2424,7 @@ INSERT INTO t1 VALUES
 ('0'),('0'),('0'),('0'),('0'),('0'),('0'),('0'),('0'),('0'),
 ('0'),('0'),('0'),('0'),('0'),('0'),('0');
 Warnings:
-Error	1034	myisam_sort_buffer_size is too small
+Error	1034	myisam_sort_buffer_size is too small. X
 Error	1034	Number of rows changed from 0 to 157
 SET myisam_sort_buffer_size=@@global.myisam_sort_buffer_size;
 INSERT INTO t1 VALUES('1');
diff --git a/mysql-test/r/mysqld--help.result b/mysql-test/r/mysqld--help.result
index dfd6998b9f3..70b93e7025f 100644
--- a/mysql-test/r/mysqld--help.result
+++ b/mysql-test/r/mysqld--help.result
@@ -1045,7 +1045,7 @@ myisam-max-sort-file-size 9223372036853727232
 myisam-mmap-size 18446744073709551615
 myisam-recover-options DEFAULT
 myisam-repair-threads 1
-myisam-sort-buffer-size 8388608
+myisam-sort-buffer-size 134216704
 myisam-stats-method nulls_unequal
 myisam-use-mmap FALSE
 net-buffer-length 16384
diff --git a/mysql-test/r/repair.result b/mysql-test/r/repair.result
index 1a98f2f0f36..241c32b4b40 100644
--- a/mysql-test/r/repair.result
+++ b/mysql-test/r/repair.result
@@ -77,12 +77,12 @@ INSERT INTO t1 VALUES
 ('0'),('0'),('0'),('0'),('0'),('0'),('0'),('0'),('0'),('0'),
 ('0'),('0'),('0'),('0'),('0'),('0'),('0');
 Warnings:
-Error	1034	myisam_sort_buffer_size is too small
+Error	1034	myisam_sort_buffer_size is too small. X
 Error	1034	Number of rows changed from 0 to 157
 SET myisam_repair_threads=2;
 REPAIR TABLE t1;
 Table	Op	Msg_type	Msg_text
-test.t1	repair	error	myisam_sort_buffer_size is too small
+test.t1	repair	error	myisam_sort_buffer_size is too small. X
 test.t1	repair	warning	Number of rows changed from 0 to 157
 test.t1	repair	status	OK
 SET myisam_repair_threads=@@global.myisam_repair_threads;
diff --git a/mysql-test/suite/maria/maria.result b/mysql-test/suite/maria/maria.result
index ddd0adade57..914c9568ea4 100644
--- a/mysql-test/suite/maria/maria.result
+++ b/mysql-test/suite/maria/maria.result
@@ -2679,3 +2679,65 @@ select count(*) from t1;
 count(*)
 13
 drop table t1;
+#
+# BUG#47444 - --myisam_repair_threads > 1 can result in all index
+#             cardinalities=1
+#
+SET aria_repair_threads=2;
+SET aria_sort_buffer_size=8192;
+CREATE TABLE t1(a CHAR(255), KEY(a), KEY(a), KEY(a));
+INSERT INTO t1 VALUES (0),(1),(2),(3),(4),(5),(6),(7),(8),(9),(0),(1),(2),(3);
+REPAIR TABLE t1;
+Table	Op	Msg_type	Msg_text
+test.t1	repair	status	OK
+SELECT CARDINALITY FROM INFORMATION_SCHEMA.STATISTICS WHERE TABLE_SCHEMA='test' AND TABLE_NAME='t1';
+CARDINALITY
+14
+14
+14
+CHECK TABLE t1;
+Table	Op	Msg_type	Msg_text
+test.t1	check	status	OK
+DROP TABLE t1;
+SET aria_sort_buffer_size=@@global.aria_sort_buffer_size;
+SET aria_repair_threads=@@global.aria_repair_threads;
+#
+# BUG#47073 - valgrind errs, corruption,failed repair of partition,
+#             low myisam_sort_buffer_size
+#
+CREATE TABLE t1(a INT, b CHAR(10), KEY(a), KEY(b));
+INSERT INTO t1 VALUES(1,'0'),(2,'0'),(3,'0'),(4,'0'),(5,'0'),
+(6,'0'),(7,'0');
+INSERT INTO t1 SELECT a+10,b FROM t1;
+INSERT INTO t1 SELECT a+20,b FROM t1;
+INSERT INTO t1 SELECT a+40,b FROM t1;
+INSERT INTO t1 SELECT a+80,b FROM t1;
+INSERT INTO t1 SELECT a+160,b FROM t1;
+INSERT INTO t1 SELECT a+320,b FROM t1;
+INSERT INTO t1 SELECT a+640,b FROM t1;
+INSERT INTO t1 SELECT a+1280,b FROM t1;
+INSERT INTO t1 SELECT a+2560,b FROM t1;
+INSERT INTO t1 SELECT a+5120,b FROM t1;
+SET aria_sort_buffer_size=4096;
+REPAIR TABLE t1;
+Table	Op	Msg_type	Msg_text
+test.t1	repair	error	aria_sort_buffer_size is too small. X
+test.t1	repair	error	Create index by sort failed
+test.t1	repair	info	Retrying repair with keycache
+test.t1	repair	status	OK
+CHECK TABLE t1;
+Table	Op	Msg_type	Msg_text
+test.t1	check	status	OK
+SET aria_repair_threads=2;
+REPAIR TABLE t1;
+Table	Op	Msg_type	Msg_text
+test.t1	repair	error	aria_sort_buffer_size is too small. X
+test.t1	repair	error	Create index by sort failed
+test.t1	repair	info	Retrying repair with keycache
+test.t1	repair	status	OK
+CHECK TABLE t1;
+Table	Op	Msg_type	Msg_text
+test.t1	check	status	OK
+SET aria_repair_threads=@@global.aria_repair_threads;
+SET aria_sort_buffer_size=@@global.aria_sort_buffer_size;
+DROP TABLE t1;
diff --git a/mysql-test/suite/maria/maria.test b/mysql-test/suite/maria/maria.test
index 3fa7d755fe4..7ecf1e33061 100644
--- a/mysql-test/suite/maria/maria.test
+++ b/mysql-test/suite/maria/maria.test
@@ -1963,6 +1963,52 @@ unlock tables;
 select count(*) from t1;
 drop table t1;
 
+--echo #
+--echo # BUG#47444 - --myisam_repair_threads > 1 can result in all index
+--echo #             cardinalities=1
+--echo #
+SET aria_repair_threads=2;
+SET aria_sort_buffer_size=8192;
+CREATE TABLE t1(a CHAR(255), KEY(a), KEY(a), KEY(a));
+INSERT INTO t1 VALUES (0),(1),(2),(3),(4),(5),(6),(7),(8),(9),(0),(1),(2),(3);
+--replace_regex /Current aria_sort_buffer_size.*/X/
+REPAIR TABLE t1;
+SELECT CARDINALITY FROM INFORMATION_SCHEMA.STATISTICS WHERE TABLE_SCHEMA='test' AND TABLE_NAME='t1';
+CHECK TABLE t1;
+DROP TABLE t1;
+SET aria_sort_buffer_size=@@global.aria_sort_buffer_size;
+SET aria_repair_threads=@@global.aria_repair_threads;
+
+--echo #
+--echo # BUG#47073 - valgrind errs, corruption,failed repair of partition,
+--echo #             low myisam_sort_buffer_size
+--echo #
+CREATE TABLE t1(a INT, b CHAR(10), KEY(a), KEY(b));
+INSERT INTO t1 VALUES(1,'0'),(2,'0'),(3,'0'),(4,'0'),(5,'0'),
+                     (6,'0'),(7,'0');
+INSERT INTO t1 SELECT a+10,b FROM t1;
+INSERT INTO t1 SELECT a+20,b FROM t1;
+INSERT INTO t1 SELECT a+40,b FROM t1;
+INSERT INTO t1 SELECT a+80,b FROM t1;
+INSERT INTO t1 SELECT a+160,b FROM t1;
+INSERT INTO t1 SELECT a+320,b FROM t1;
+INSERT INTO t1 SELECT a+640,b FROM t1;
+INSERT INTO t1 SELECT a+1280,b FROM t1;
+INSERT INTO t1 SELECT a+2560,b FROM t1;
+INSERT INTO t1 SELECT a+5120,b FROM t1;
+SET aria_sort_buffer_size=4096;
+--replace_regex /Current aria_sort_buffer_size.*/X/
+REPAIR TABLE t1;
+CHECK TABLE t1;
+SET aria_repair_threads=2;
+# May report different values depending on threads activity.
+--replace_regex /Current aria_sort_buffer_size.*/X/
+REPAIR TABLE t1;
+CHECK TABLE t1;
+SET aria_repair_threads=@@global.aria_repair_threads;
+SET aria_sort_buffer_size=@@global.aria_sort_buffer_size;
+DROP TABLE t1;
+
 #
 # End of test
 #
diff --git a/mysql-test/suite/maria/maria3.result b/mysql-test/suite/maria/maria3.result
index b502d71e772..189329fe5c8 100644
--- a/mysql-test/suite/maria/maria3.result
+++ b/mysql-test/suite/maria/maria3.result
@@ -317,7 +317,7 @@ aria_pagecache_division_limit	100
 aria_page_checksum	OFF
 aria_recover	NORMAL
 aria_repair_threads	1
-aria_sort_buffer_size	134217728
+aria_sort_buffer_size	268434432
 aria_stats_method	nulls_unequal
 aria_sync_log_dir	NEWFILE
 show status like 'aria%';
diff --git a/mysql-test/suite/sys_vars/r/aria_sort_buffer_size_basic.result b/mysql-test/suite/sys_vars/r/aria_sort_buffer_size_basic.result
index cf067c7e7b6..56522566ec9 100644
--- a/mysql-test/suite/sys_vars/r/aria_sort_buffer_size_basic.result
+++ b/mysql-test/suite/sys_vars/r/aria_sort_buffer_size_basic.result
@@ -1,30 +1,34 @@
 SET @start_global_value = @@global.aria_sort_buffer_size;
 select @@global.aria_sort_buffer_size;
 @@global.aria_sort_buffer_size
-134217728
+268434432
 select @@session.aria_sort_buffer_size;
 @@session.aria_sort_buffer_size
-134217728
+268434432
 show global variables like 'aria_sort_buffer_size';
 Variable_name	Value
-aria_sort_buffer_size	134217728
+aria_sort_buffer_size	268434432
 show session variables like 'aria_sort_buffer_size';
 Variable_name	Value
-aria_sort_buffer_size	134217728
+aria_sort_buffer_size	268434432
 select * from information_schema.global_variables where variable_name='aria_sort_buffer_size';
 VARIABLE_NAME	VARIABLE_VALUE
-ARIA_SORT_BUFFER_SIZE	134217728
+ARIA_SORT_BUFFER_SIZE	268434432
 select * from information_schema.session_variables where variable_name='aria_sort_buffer_size';
 VARIABLE_NAME	VARIABLE_VALUE
-ARIA_SORT_BUFFER_SIZE	134217728
+ARIA_SORT_BUFFER_SIZE	268434432
 set global aria_sort_buffer_size=10;
+Warnings:
+Warning	1292	Truncated incorrect aria_sort_buffer_size value: '10'
 select @@global.aria_sort_buffer_size;
 @@global.aria_sort_buffer_size
-10
+4096
 set session aria_sort_buffer_size=10;
+Warnings:
+Warning	1292	Truncated incorrect aria_sort_buffer_size value: '10'
 select @@session.aria_sort_buffer_size;
 @@session.aria_sort_buffer_size
-10
+4096
 set global aria_sort_buffer_size=1.1;
 ERROR 42000: Incorrect argument type to variable 'aria_sort_buffer_size'
 set session aria_sort_buffer_size=1e1;
@@ -36,7 +40,7 @@ Warnings:
 Warning	1292	Truncated incorrect aria_sort_buffer_size value: '0'
 select @@global.aria_sort_buffer_size;
 @@global.aria_sort_buffer_size
-4
+4096
 set session aria_sort_buffer_size=cast(-1 as unsigned int);
 select @@session.aria_sort_buffer_size;
 @@session.aria_sort_buffer_size
diff --git a/mysql-test/suite/sys_vars/r/myisam_sort_buffer_size_basic.result b/mysql-test/suite/sys_vars/r/myisam_sort_buffer_size_basic.result
index 85482be9ad4..932d06668fb 100644
--- a/mysql-test/suite/sys_vars/r/myisam_sort_buffer_size_basic.result
+++ b/mysql-test/suite/sys_vars/r/myisam_sort_buffer_size_basic.result
@@ -1,11 +1,11 @@
 SET @start_global_value = @@global.myisam_sort_buffer_size ;
 SELECT @start_global_value;
 @start_global_value
-8388608
+134216704
 SET @start_session_value = @@session.myisam_sort_buffer_size ;
 SELECT @start_session_value;
 @start_session_value
-8388608
+134216704
 '#--------------------FN_DYNVARS_005_01-------------------------#'
 SET @@global.myisam_sort_buffer_size   = 100;
 Warnings:
@@ -13,22 +13,22 @@ Warning	1292	Truncated incorrect myisam_sort_buffer_size value: '100'
 SET @@global.myisam_sort_buffer_size   = DEFAULT;
 SELECT @@global.myisam_sort_buffer_size  ;
 @@global.myisam_sort_buffer_size
-8388608
+134216704
 SET @@session.myisam_sort_buffer_size   = 200;
 Warnings:
 Warning	1292	Truncated incorrect myisam_sort_buffer_size value: '200'
 SET @@session.myisam_sort_buffer_size   = DEFAULT;
 SELECT @@session.myisam_sort_buffer_size  ;
 @@session.myisam_sort_buffer_size
-8388608
+134216704
 '#--------------------FN_DYNVARS_005_02-------------------------#'
 SET @@global.myisam_sort_buffer_size   = DEFAULT;
-SELECT @@global.myisam_sort_buffer_size   = 8388608;
-@@global.myisam_sort_buffer_size   = 8388608
+SELECT @@global.myisam_sort_buffer_size   = 134216704;
+@@global.myisam_sort_buffer_size   = 134216704
 1
 SET @@session.myisam_sort_buffer_size   = DEFAULT;
-SELECT @@session.myisam_sort_buffer_size   = 8388608;
-@@session.myisam_sort_buffer_size   = 8388608
+SELECT @@session.myisam_sort_buffer_size   = 134216704;
+@@session.myisam_sort_buffer_size   = 134216704
 1
 '#--------------------FN_DYNVARS_005_03-------------------------#'
 SET @@global.myisam_sort_buffer_size   = 4;
@@ -187,8 +187,8 @@ ERROR 42S22: Unknown column 'myisam_sort_buffer_size' in 'field list'
 SET @@global.myisam_sort_buffer_size   = @start_global_value;
 SELECT @@global.myisam_sort_buffer_size  ;
 @@global.myisam_sort_buffer_size
-8388608
+134216704
 SET @@session.myisam_sort_buffer_size   = @start_session_value;
 SELECT @@session.myisam_sort_buffer_size  ;
 @@session.myisam_sort_buffer_size
-8388608
+134216704
diff --git a/mysql-test/suite/sys_vars/t/myisam_sort_buffer_size_basic.test b/mysql-test/suite/sys_vars/t/myisam_sort_buffer_size_basic.test
index c35949ac5f3..1aa25fa2c8e 100644
--- a/mysql-test/suite/sys_vars/t/myisam_sort_buffer_size_basic.test
+++ b/mysql-test/suite/sys_vars/t/myisam_sort_buffer_size_basic.test
@@ -60,10 +60,10 @@ SELECT @@session.myisam_sort_buffer_size  ;
 ########################################################################
 
 SET @@global.myisam_sort_buffer_size   = DEFAULT;
-SELECT @@global.myisam_sort_buffer_size   = 8388608;
+SELECT @@global.myisam_sort_buffer_size   = 134216704;
 
 SET @@session.myisam_sort_buffer_size   = DEFAULT;
-SELECT @@session.myisam_sort_buffer_size   = 8388608;
+SELECT @@session.myisam_sort_buffer_size   = 134216704;
 
 
 --echo '#--------------------FN_DYNVARS_005_03-------------------------#'
diff --git a/mysql-test/t/myisam.test b/mysql-test/t/myisam.test
index 8323890b028..6c9371eed25 100644
--- a/mysql-test/t/myisam.test
+++ b/mysql-test/t/myisam.test
@@ -1600,6 +1600,7 @@ INSERT INTO t1 SELECT a+1280,b FROM t1;
 INSERT INTO t1 SELECT a+2560,b FROM t1;
 INSERT INTO t1 SELECT a+5120,b FROM t1;
 SET myisam_sort_buffer_size=4;
+--replace_regex /Current myisam_sort_buffer_size.*/X/
 REPAIR TABLE t1;
 
 SET myisam_repair_threads=2;
@@ -1648,6 +1649,7 @@ DROP TABLE t1, t2, t3;
 CREATE TABLE t1(a CHAR(255), KEY(a));
 SELECT * FROM t1, t1 AS a1;
 SET myisam_sort_buffer_size=4;
+--replace_regex /Current myisam_sort_buffer_size.*/X/
 INSERT INTO t1 VALUES
 ('0'),('0'),('0'),('0'),('0'),('0'),('0'),('0'),('0'),('0'),
 ('0'),('0'),('0'),('0'),('0'),('0'),('0'),('0'),('0'),('0'),
diff --git a/mysql-test/t/repair.test b/mysql-test/t/repair.test
index 3c55f06ff4c..6536c052019 100644
--- a/mysql-test/t/repair.test
+++ b/mysql-test/t/repair.test
@@ -61,6 +61,7 @@ DROP TABLE t1;
 #
 CREATE TABLE t1(a CHAR(255), KEY(a));
 SET myisam_sort_buffer_size=4096;
+--replace_regex /Current myisam_sort_buffer_size.*/X/
 INSERT INTO t1 VALUES
 ('0'),('0'),('0'),('0'),('0'),('0'),('0'),('0'),('0'),('0'),
 ('0'),('0'),('0'),('0'),('0'),('0'),('0'),('0'),('0'),('0'),
@@ -79,6 +80,7 @@ INSERT INTO t1 VALUES
 ('0'),('0'),('0'),('0'),('0'),('0'),('0'),('0'),('0'),('0'),
 ('0'),('0'),('0'),('0'),('0'),('0'),('0');
 SET myisam_repair_threads=2;
+--replace_regex /Current myisam_sort_buffer_size.*/X/
 REPAIR TABLE t1;
 SET myisam_repair_threads=@@global.myisam_repair_threads;
 SET myisam_sort_buffer_size=@@global.myisam_sort_buffer_size;