Bug #23578: Corruption prevents Optimize table from working properly with a

spatial index
 While executing OPTIMIZE TABLE on MyISAM tables the server re-creates the
 index file(s) in order to sort them physically by the key. This cannot be 
 done for R-tree indexes as it makes no sense.
 The server was not checking the type of the index and was accessing an 
 R-tree index as if it was a B-tree.
 Fixed by preventing sorting the index file if it contains an R-tree index.  
 


myisam/mi_check.c:
  Bug #23578: Corruption prevents Optimize table from working properly with a
  spatial index
   - disable sorting the index file if it contains an R-tree index.
mysql-test/r/gis-rtree.result:
  Bug #23578: Corruption prevents Optimize table from working properly with a
  spatial index
   - test case
mysql-test/t/gis-rtree.test:
  Bug #23578: Corruption prevents Optimize table from working properly with a
  spatial index
   - test case

1 files changed, 8 insertions, 0 deletions

diff --git a/myisam/mi_check.c b/myisam/mi_check.c
index 06fdd3bff4c..68e5860a6d1 100644
--- a/myisam/mi_check.c
+++ b/myisam/mi_check.c
@@ -1758,6 +1758,12 @@ int mi_sort_index(MI_CHECK *param, register MI_INFO *info, my_string name)
   MI_STATE_INFO old_state;
   DBUG_ENTER("mi_sort_index");
 
+  /* cannot sort index files with R-tree indexes */
+  for (key= 0,keyinfo= &share->keyinfo[0]; key < share->base.keys ;
+       key++,keyinfo++)
+    if (keyinfo->key_alg == HA_KEY_ALG_RTREE)
+      return 0;
+
   if (!(param->testflag & T_SILENT))
     printf("- Sorting index for MyISAM-table '%s'\n",name);
 
@@ -1850,6 +1856,8 @@ static int sort_one_index(MI_CHECK *param, MI_INFO *info, MI_KEYDEF *keyinfo,
   char llbuff[22];
   DBUG_ENTER("sort_one_index");
 
+  /* cannot walk over R-tree indices */
+  DBUG_ASSERT(keyinfo->key_alg != HA_KEY_ALG_RTREE);
   new_page_pos=param->new_file_pos;
   param->new_file_pos+=keyinfo->block_length;