BUG#51866 - crash with repair by sort and fulltext keys

Repairing MyISAM table with fulltext indexes and low
myisam_sort_buffer_size may crash the server.

Estimation of number of index entries was done incorrectly,
causing further assertion failure or server crash.

Docs note: min value for myisam_sort_buffer_size has been
changed from 4 to 4096.

mysql-test/r/fulltext.result:
  A test case for BUG#51866.
mysql-test/r/myisam.result:
  Min value for myisam_sort_buffer_size is 4096.
mysql-test/r/variables.result:
  Min value for myisam_sort_buffer_size is 4096.
mysql-test/suite/sys_vars/r/myisam_sort_buffer_size_basic_32.result:
  Min value for myisam_sort_buffer_size is 4096.
mysql-test/t/fulltext.test:
  A test case for BUG#51866.
sql/mysqld.cc:
  Min value for myisam_sort_buffer_size is 4096.
storage/myisam/mi_check.c:
  When estimating number of index entries for external
  fulltext parser, take into account that key_length may
  be bigger than myisam_sort_buffer_size. Reuse logic
  from _create_index_by_sort(): force MIN_SORT_BUFFER to
  be min value for myisam_sort_buffer_size.
  
  Another problem is that ftkey_nr has no other meaning
  than serial number of fulltext index starting with 1.
  We can't say if this key using built-in or external
  parser basing on it's value. In other words we always
  entered if-branch for external parser. At this point,
  the only way to check if we use default parser is to
  compare keyinfo::parser with &ft_default_parser.
storage/myisam/sort.c:
  Get rid of MIN_SORT_MEMORY, use MIN_SORT_BUFFER instead
  (defined in myisamdef.h, has the same value and purpose).

1 files changed, 4 insertions, 5 deletions

diff --git a/storage/myisam/mi_check.c b/storage/myisam/mi_check.c
index c10cfdd0c9b..8e3864d1c44 100644
--- a/storage/myisam/mi_check.c
+++ b/storage/myisam/mi_check.c
@@ -2396,10 +2396,8 @@ int mi_repair_by_sort(MI_CHECK *param, register MI_INFO *info,
       /*
         fulltext indexes may have much more entries than the
         number of rows in the table. We estimate the number here.
-
-        Note, built-in parser is always nr. 0 - see ftparser_call_initializer()
       */
-      if (sort_param.keyinfo->ftkey_nr == 0)
+      if (sort_param.keyinfo->parser == &ft_default_parser)
       {
         /*
           for built-in parser the number of generated index entries
@@ -2416,8 +2414,9 @@ int mi_repair_by_sort(MI_CHECK *param, register MI_INFO *info,
           so, we'll use all the sort memory and start from ~10 buffpeks.
           (see _create_index_by_sort)
         */
-        sort_info.max_records=
-          10*param->sort_buffer_length/sort_param.key_length;
+        sort_info.max_records= 10 *
+                               max(param->sort_buffer_length, MIN_SORT_BUFFER) /
+                               sort_param.key_length;
       }
 
       sort_param.key_read=sort_ft_key_read;