Bug#8283 - OPTIMIZE TABLE causes data loss

OPTIMIZE TABLE with myisam_repair_threads > 1 performs a non-quick 
parallel repair. This means that it does not only rebuild all 
indexes, but also the data file.

Non-quick parallel repair works so that there is one thread per 
index. The first of the threads rebuilds also the new data file.

The problem was that all threads shared the read io cache on the
old data file. If there were holes (deleted records) in the table,
the first thread skipped them, writing only contiguous, non-deleted
records to the new data file. Then it built the new index so that
its entries pointed to the correct record positions. But the other
threads didn't know the new record positions, but put the positions
from the old data file into the index.

The new design is so that there is a shared io cache which is filled
by the first thread (the data file writer) with the new contiguous
records and read by the other threads. Now they know the new record
positions.

Another problem was that for the parallel repair of compressed
tables a common bit_buff and rec_buff was used. I changed it so
that thread specific buffers are used for parallel repair.

A similar problem existed for checksum calculation. I made this
multi-thread safe too.


include/my_sys.h:
  Bug#8283 - OPTIMIZE TABLE causes data loss
  Redesign of io_cache_share.
include/myisam.h:
  Bug#8283 - OPTIMIZE TABLE causes data loss
  Redesign of checksum calculation in mi_check.c.
  'calc_checksum' is now in myisamdef.h:st_mi_sort_param.
myisam/mi_check.c:
  Bug#8283 - OPTIMIZE TABLE causes data loss
  Implemented a new parallel repair design.
  Using a synchronized shared read/write cache.
  Allowed for thread specific bit_buff, rec_buff, and calc_checksum.
myisam/mi_open.c:
  Bug#8283 - OPTIMIZE TABLE causes data loss
  Added DBUG output.
myisam/mi_packrec.c:
  Bug#8283 - OPTIMIZE TABLE causes data loss
  Allowed for thread specific bit_buff and rec_buff.
myisam/myisamdef.h:
  Bug#8283 - OPTIMIZE TABLE causes data loss
  Commented on checksum calculation variables.
  Allowed for thread specific bit_buff.
  Added DBUG output for better table crash detection.
myisam/sort.c:
  Bug#8283 - OPTIMIZE TABLE causes data loss
  Added implications of the new parallel repair design.
  Renamed 'info' -> 'sort_param'.
  Added DBUG output.
mysql-test/r/myisam.result:
  Bug#8283 - OPTIMIZE TABLE causes data loss
  Added test results.
mysql-test/t/myisam.test:
  Bug#8283 - OPTIMIZE TABLE causes data loss
  Added test cases.
mysys/mf_iocache.c:
  Bug#8283 - OPTIMIZE TABLE causes data loss
  Redesign of io_cache_share.
  We do now allow a writer to synchronize himself with the
  readers of a shared cache. When all threads join in the lock,
  the writer copies the data from his write buffer to the shared
  read buffer.

1 files changed, 14 insertions, 8 deletions

diff --git a/include/my_sys.h b/include/my_sys.h
index 02ea188a18e..92f3758a846 100644
--- a/include/my_sys.h
+++ b/include/my_sys.h
@@ -325,12 +325,18 @@ typedef int (*IO_CACHE_CALLBACK)(struct st_io_cache*);
 #ifdef THREAD
 typedef struct st_io_cache_share
 {
-  /* to sync on reads into buffer */
-  pthread_mutex_t mutex;
-  pthread_cond_t  cond;
-  int             count, total;
-  /* actual IO_CACHE that filled the buffer */
-  struct st_io_cache *active;
+  pthread_mutex_t       mutex;           /* To sync on reads into buffer. */
+  pthread_cond_t        cond;            /* To wait for signals. */
+  pthread_cond_t        cond_writer;     /* For a synchronized writer. */
+  /* Offset in file corresponding to the first byte of buffer. */
+  my_off_t              pos_in_file;
+  /* If a synchronized write cache is the source of the data. */
+  struct st_io_cache    *source_cache;
+  byte                  *buffer;         /* The read buffer. */
+  byte                  *read_end;       /* Behind last valid byte of buffer. */
+  int                   running_threads; /* threads not in lock. */
+  int                   total_threads;   /* threads sharing the cache. */
+  int                   error;           /* Last error. */
 #ifdef NOT_YET_IMPLEMENTED
   /* whether the structure should be free'd */
   my_bool alloced;
@@ -672,8 +678,8 @@ extern void setup_io_cache(IO_CACHE* info);
 extern int _my_b_read(IO_CACHE *info,byte *Buffer,uint Count);
 #ifdef THREAD
 extern int _my_b_read_r(IO_CACHE *info,byte *Buffer,uint Count);
-extern void init_io_cache_share(IO_CACHE *info,
-				IO_CACHE_SHARE *s, uint num_threads);
+extern void init_io_cache_share(IO_CACHE *read_cache, IO_CACHE_SHARE *cshare,
+                                IO_CACHE *write_cache, uint num_threads);
 extern void remove_io_thread(IO_CACHE *info);
 #endif
 extern int _my_b_seq_read(IO_CACHE *info,byte *Buffer,uint Count);