Bug#11764487: myisam corruption with insert ignore and invalid spatial data

Problem: in case of wrong data insert into indexed GEOMETRY fields 
(e.g. NULL value for a not NULL field) MyISAM reported 
"ERROR 126 (HY000): Incorrect key file for table; try to repair it"
due to misuse of the key deletion function.

Fix: always use R-tree key functions for R-tree based indexes
and B-tree key functions for B-tree based indexes.


mysql-test/r/gis-rtree.result:
  Bug#11764487: myisam corruption with insert ignore and invalid spatial data
    - test result.
mysql-test/t/gis-rtree.test:
  Bug#11764487: myisam corruption with insert ignore and invalid spatial data
    - test case.
storage/myisam/mi_update.c:
  Bug#11764487: myisam corruption with insert ignore and invalid spatial data
    - handling update errors check for HA_ERR_NULL_IN_SPATIAL as well to be 
  consistent with mi_write();
    - always use keyinfo->ck_delete()/ck_insert() instead of _mi_ck_delete()/_mi_ck_write()
  to handle index properly, as it may be of B-tree or R-tree type.
storage/myisam/mi_write.c:
  Bug#11764487: myisam corruption with insert ignore and invalid spatial data
    - always use keyinfo->ck_delete() instead of _mi_ck_delete() to handle
  index properly, as it may be of B-tree or R-tree type.

2 files changed, 6 insertions, 5 deletions

diff --git a/storage/myisam/mi_update.c b/storage/myisam/mi_update.c
index a18bb5f1443..c767fa44ed5 100644
--- a/storage/myisam/mi_update.c
+++ b/storage/myisam/mi_update.c
@@ -193,8 +193,8 @@ err:
   save_errno=my_errno;
   if (changed)
     key_changed|= HA_STATE_CHANGED;
-  if (my_errno == HA_ERR_FOUND_DUPP_KEY || my_errno == HA_ERR_OUT_OF_MEM ||
-      my_errno == HA_ERR_RECORD_FILE_FULL)
+  if (my_errno == HA_ERR_FOUND_DUPP_KEY || my_errno == HA_ERR_RECORD_FILE_FULL ||
+      my_errno == HA_ERR_NULL_IN_SPATIAL || my_errno == HA_ERR_OUT_OF_MEM)
   {
     info->errkey= (int) i;
     flag=0;
@@ -212,8 +212,9 @@ err:
 	{
 	  uint new_length=_mi_make_key(info,i,new_key,newrec,pos);
 	  uint old_length= _mi_make_key(info,i,old_key,oldrec,pos);
-	  if ((flag++ && _mi_ck_delete(info,i,new_key,new_length)) ||
-	      _mi_ck_write(info,i,old_key,old_length))
+	  if ((flag++ && 
+	       share->keyinfo[i].ck_delete(info, i, new_key, new_length)) ||
+	      share->keyinfo[i].ck_insert(info, i, old_key, old_length))
 	    break;
 	}
       }
diff --git a/storage/myisam/mi_write.c b/storage/myisam/mi_write.c
index 3c8ebe5dbd8..51ae06b117d 100644
--- a/storage/myisam/mi_write.c
+++ b/storage/myisam/mi_write.c
@@ -211,7 +211,7 @@ err:
         else
 	{
 	  uint key_length=_mi_make_key(info,i,buff,record,filepos);
-	  if (_mi_ck_delete(info,i,buff,key_length))
+	  if (share->keyinfo[i].ck_delete(info, i, buff, key_length))
 	  {
 	    if (local_lock_tree)
 	      rw_unlock(&share->key_root_lock[i]);