UNDO of rows now puts back all part of the row on their original pages and positions

Added variable _dbug_on_ to speed up execution when DBUG is not going to be used
Added --debug-on option to mysqld (to be able to turn of DBUG with --debug-on=0)
Fixed some bugs with 'non_flushable' marking of bitmap pages
Don't use 'non_flushable' marking of bitmap pages for not transactional tables
SHOW CREATE TABLE now shows if table was created with page checksums
Fixed a lot of bugs with BLOB handling in case of update/REDO and UNDO
More tests (especially for blobs) and DBUG_ASSERTS()
More readable output from maria_read_log and maria_chk
Fixed wrong shift that caused Maria to crash on files > 4G
Mark tables as crashed of REDO fails


dbug/dbug.c:
  Changed to use my_bool (allowed me to remove some windows specific code)
  Added variable _dbug_on_ to speed up execution when DBUG is not going to be used
  Removed initialization of variables if not needed
include/my_dbug.h:
  Use my_bool for some functions that was defined as BOOLEAN in dbug.c code
  Added DBUGGER_ON/DEBUGGER_OFF to speed up execution when DBUG is not used
include/my_global.h:
  Define my_bool early
  Increase MY_HOW_OFTEN_TO_WRITE as computers are now faster than 10 years ago
mysql-test/mysql-test-run.pl:
  Added debug-on=0 to speed up tests
mysql-test/r/maria-recovery.result:
  Added new test by Guilhem to test if UNDO_ROW_DELETE preserves rowid
mysql-test/r/maria.result:
  Added testing of page checksums
mysql-test/t/crash_commit_before-master.opt:
  Added --debug-on as test require DBUG to work
mysql-test/t/maria-recovery-bitmap-master.opt:
  Added --debug-on as test require DBUG to work
mysql-test/t/maria-recovery-master.opt:
  Added --debug-on as test require DBUG to work
mysql-test/t/maria-recovery.test:
  Added new test by Guilhem to test if UNDO_ROW_DELETE preserves rowid
mysql-test/t/maria.test:
  Added testing of page checksums
sql/mysqld.cc:
  Added --debug-on option (to be able to turn of DBUG with --debug-on=0)
  Indentation fixes
  Removed end spaces
sql/sql_show.cc:
  Allow update_create_info() to inform MySQL if PACK_KEYS, NO_PACK_KEYS, CHECKSUM, PAGE_CHECKSUM or DELAY_KEY_WRITE is used
storage/maria/Makefile.am:
  Added ma_test_big.sh
storage/maria/ha_maria.cc:
  Store in create_info if page checksums are used (For SHOW CREATE TABLE)
storage/maria/ma_bitmap.c:
  Added _ma_bitmap_wait_or_flush() to cause reader of bitmap pages to wait with reading until bitmap is flushed.
  Use TAIL_PAGE_COUNT_MARKER for tail pages
  Set 'sub_blocks' for and only for the head page or for the first extent of a blob. This is needed for store_extent_info() to be able to set START_EXTENT_BIT's
  Don't allocate more than 0x3ffff pages in one extent (We need bit 0x4000 as a START_EXTENT_BIT)
  Increase the calculated 'head_length' with the number of bytes used for extents.
  Update row->space_on_head_page also in _ma_bitmap_find_new_place()
  Make _ma_bitmap_get_page_bits() global. (Needed for UNDO handling)
  Changed _ma_bitmap_flushable() to take MARIA_HA instead of MARIA_SHARE.
  This was needed to be able to mark the handler if we had a 'non_flushable' call pending or not.
  Don't use 'non_flushable' marking of bitmap pages for not transactional tables.
  Added BLOCKUSED_USE_ORG_BITMAP handling also for tail pages.
  Added more DBUG_ASSERT() to find possible errors in other code
  Some code simplications by adding new local variables
storage/maria/ma_blockrec.c:
  UNDO of rows now puts back all part of the row on their original pages and positions.
  Changed UNDO of DELETE and UNDO of UPDATE to contain information about the original length of data on head block and also extent information
  This changes a lot of logic as now an insert of a row on a page may happen to any position (and not just to the first or next free)
  Use PAGE_COUNT to mark if an extent is the start of of a blob. (Needed for extent_to_bitmap_blocks())
  Added check_directory() for checking that directroy entries are correct.
  Added checking of row checksums when reading rows (with EXTRA_DEBUG)
  Added make_space_for_directory() and extend_directory() for doing expansion of directory
  Added get_rowpos_in_head_or_tail_page() to be able to store head/tail on original position in UNDO
  Added extent_to_bitmap_blocks() to be able to generate original bitmap blocks from UNDO entry
  Added _ma_update_at_original_place() for UNDO of DELETES
  Added row->min_length to hold minmum required space needed on head page
  Changed find_free_position() to use make_space_for_directory()
  Changed make_empty_page() to allow optional creation of directory entry
  Changed delete_head_or_tail() and _ma_apply_undo_row_isnert() to not copy pagecache block (speed optimization)
  Changed _ma_apply_redo_insert_row_head_or_tail() to be able to insert new row at any position on 'new' page
  Changed _ma_apply_undo_row_delete() and _ma_apply_undo_row_update() to put row in it's original position
  Ensure allocation of tail blocks are of at least MIN_TAIL_SIZE.
  Ensure we store pages in pinned pages even if read failed. (If not we will have pages pinned forever in page cache)
  Write original extent information in UNDO entry, not compacted ones (we need position to tails!)
  When setting BLOCKUSED_USED, don't clear other bits (we have to preserve BLOCKUSED_USE_ORG_BITMAP)
  Fixed som bugs in directory handling
  Fixed bug where we wrote wrong lsn to blob pages
  Added separate blob_buffer for fixing bug when updating row that had char/varchar that spanned several pages and also had blobs
  Ensure we call _ma_bitmap_flushable() also in case of errors
  When doing an update, first delete old entries, then search in bitmap for where to put new information
  Info->s -> share
  Rowid -> rowid
  More DBUG_ASSERT()
storage/maria/ma_blockrec.h:
  Added START_EXTENT_BIT and TAIL_PAGE_COUNT_MARKER
  Added _ma_bitmap_wait_or_flush() and _ma_bitmap_get_page_bits()
storage/maria/ma_check.c:
  Don't write extra empty line if there is no deleted blocks
  Ignore START_EXTENT_BIT's in page count
  Call _ma_fast_unlock_key_del() to free key_del link
storage/maria/ma_close.c:
  Ensure that used_key_del is 0. (If not, someone forgot to call _ma_unlock_key_del())
storage/maria/ma_create.c:
  Changed constant to macro
storage/maria/ma_delete.c:
  For deleted keys, log also position to row
storage/maria/ma_extra.c:
  Release blob buffer at maria_reset() if bigger than MARIA_SMALL_BLOB_BUFFER
storage/maria/ma_key_recover.c:
  Added bzero() of LSN that confused paged cache in case of uninitialized block
  Mark file crashed if applying of index changes fails
  Added calls to _ma_fast_unlock_key_del() for protection of shared key_del link.
storage/maria/ma_locking.c:
  Added usage of MARIA_FILE_OPEN_COUNT_OFFSET
  Added _ma_mark_file_crashed()
storage/maria/ma_loghandler.c:
  Fixed bug where we logged uninitialized memory
storage/maria/ma_open.c:
  Moved state->changed to be at start of state info on disk to allow one to easly mark files as crashed
storage/maria/ma_page.c:
  Disable 'dummy' checksumming of pages as this gave false warnings.
  (Need to investigate if this is ever needed)
storage/maria/ma_pagecache.c:
  Fixed wrong shift that caused Maria to crash on files > 4G
storage/maria/ma_recovery.c:
  In case of errors, start writing on new line if we where in %## %## printing mode (Made errors more readable)
  Changed global variable name from warnings -> recovery_warnings
  Use MARIA_FILE_CREATE_RENAME_LSN_OFFSET instead of constant
  Removed special handling of row position for deleted keys. Keys now always includes row positions
  _ma_apply_undo_row_delete() now gets page and row position
  Added check that we don't loop forever when handling undo's (in case of bug in undo chain)
  Print name of failed REDO/UNDO
storage/maria/ma_recovery.h:
  Removed old comment
storage/maria/ma_static.c:
  Chaned version number of Maria files to not accidently use old ones (becasue of change of ordering of status variables)
storage/maria/ma_test2.c:
  Added option -u to specify number of rows to update
  Changed old option -u to be -A, as for ma_test1
  Fixed bug in update of rows with blobs (before blobs was always reset to empty on update)
  First created blob is now of max blob length to ensure we have at least one big blob in the table
storage/maria/ma_test_all.sh:
  More tests
storage/maria/ma_test_recovery.expected:
  Updated results
storage/maria/ma_test_recovery:
  Changed tests to use bigger blobs (not just 1K)
  Added new tests that tests recovery of update with blobs
  Removed comparision of .MAD file as it's not guranteed that recovery from scratch gives identical data file as original update
  (compact_page() may be called at different times during normal execution and during REDO)
storage/maria/ma_update.c:
  Simplify code (changed * to if)
storage/maria/maria_chk.c:
  Make output more readable
storage/maria/maria_def.h:
  Changed 'changed' to int to prepare for more bits
  Added 2 more bytes to status information
  Added 'st_mara_row->min_length' for storing min length needed on head page
  Added 'st_mara_handler->blob_buff & blob_buff_size' for storing blobs
  Moved all tunning parameters into one block
  Added MARIA_SMALL_BLOB_BUFFER
  Added _ma_mark_file_crashed()
storage/myisam/mi_test2.c:
  Fixed bug in update of rows with blobs (before blobs was always reset to empty on update)
storage/maria/ma_test_big.sh:
  Testing of insert, update, delete, recovery and undo of rows with blobs
  Thanks to the random-ness of ma_test2 this is likely to find most bugs in the row handling

1 files changed, 48 insertions, 33 deletions

diff --git a/dbug/dbug.c b/dbug/dbug.c
index 7bb7396ad5b..b2a70fd7ebe 100644
--- a/dbug/dbug.c
+++ b/dbug/dbug.c
@@ -123,11 +123,7 @@
  *      Typedefs to make things more obvious.
  */
 
-#ifndef __WIN__
-typedef int BOOLEAN;
-#else
-#define BOOLEAN BOOL
-#endif
+#define BOOLEAN my_bool
 
 /*
  *      Make it easy to change storage classes if necessary.
@@ -216,6 +212,7 @@ struct settings {
 static BOOLEAN init_done= FALSE; /* Set to TRUE when initialization done */
 static struct settings init_settings;
 static const char *db_process= 0;/* Pointer to process name; argv[0] */
+my_bool _dbug_on_= TRUE;	 /* FALSE if no debugging at all */
 
 typedef struct _db_code_state_ {
   const char *process;          /* Pointer to process name; usually argv[0] */
@@ -248,7 +245,8 @@ typedef struct _db_code_state_ {
   The test below is so we could call functions with DBUG_ENTER before
   my_thread_init().
 */
-#define get_code_state_or_return if (!cs && !((cs=code_state()))) return
+#define get_code_state_if_not_set_or_return if (!cs && !((cs=code_state()))) return
+#define get_code_state_or_return if (!((cs=code_state()))) return
 
         /* Handling lists */
 static struct link *ListAdd(struct link *, const char *, const char *);
@@ -332,13 +330,20 @@ static CODE_STATE *code_state(void)
 {
   CODE_STATE *cs, **cs_ptr;
 
+  /*
+    _dbug_on_ is reset if we don't plan to use any debug commands at all and
+    we want to run on maximum speed
+   */
+  if (!_dbug_on_)
+    return 0;
+
   if (!init_done)
   {
+    init_done=TRUE;
     pthread_mutex_init(&THR_LOCK_dbug,MY_MUTEX_INIT_FAST);
     bzero(&init_settings, sizeof(init_settings));
     init_settings.out_file=stderr;
     init_settings.flags=OPEN_APPEND;
-    init_done=TRUE;
   }
 
   if (!(cs_ptr= (CODE_STATE**) my_thread_var_dbug()))
@@ -405,7 +410,7 @@ static CODE_STATE *code_state(void)
 
 void _db_process_(const char *name)
 {
-  CODE_STATE *cs=0;
+  CODE_STATE *cs;
 
   if (!db_process)
     db_process= name;
@@ -449,10 +454,10 @@ void _db_process_(const char *name)
 void _db_set_(CODE_STATE *cs, const char *control)
 {
   const char *end;
-  int rel=0;
+  int rel;
   struct settings *stack;
 
-  get_code_state_or_return;
+  get_code_state_if_not_set_or_return;
   stack= cs->stack;
 
   if (control[0] == '-' && control[1] == '#')
@@ -693,7 +698,7 @@ void _db_set_(CODE_STATE *cs, const char *control)
 
 void _db_push_(const char *control)
 {
-  CODE_STATE *cs=0;
+  CODE_STATE *cs;
   get_code_state_or_return;
   PushState(cs);
   _db_set_(cs, control);
@@ -742,7 +747,7 @@ void _db_set_init_(const char *control)
 void _db_pop_()
 {
   struct settings *discard;
-  CODE_STATE *cs=0;
+  CODE_STATE *cs;
 
   get_code_state_or_return;
 
@@ -836,7 +841,7 @@ int _db_explain_ (CODE_STATE *cs, char *buf, size_t len)
 {
   char *start=buf, *end=buf+len-4;
 
-  get_code_state_or_return *buf=0;
+  get_code_state_if_not_set_or_return *buf=0;
 
   op_list_to_buf('d', cs->stack->keywords, DEBUGGING);
   op_int_to_buf ('D', cs->stack->delay, 0);
@@ -939,9 +944,15 @@ void _db_enter_(const char *_func_, const char *_file_,
                 uint _line_, const char **_sfunc_, const char **_sfile_,
                 uint *_slevel_, char ***_sframep_ __attribute__((unused)))
 {
-  int save_errno=errno;
-  CODE_STATE *cs=0;
+  int save_errno;
+  CODE_STATE *cs;
+  if (!((cs=code_state())))
+  {
+    *_slevel_= 0; /* Set to avoid valgrind warnings if dbug is enabled later */
+    return;
+  }
   get_code_state_or_return;
+  save_errno= errno;
 
   *_sfunc_= cs->func;
   *_sfile_= cs->file;
@@ -1015,7 +1026,7 @@ void _db_return_(uint _line_, const char **_sfunc_,
                  const char **_sfile_, uint *_slevel_)
 {
   int save_errno=errno;
-  CODE_STATE *cs=0;
+  CODE_STATE *cs;
   get_code_state_or_return;
 
   if (cs->level != (int) *_slevel_)
@@ -1049,7 +1060,11 @@ void _db_return_(uint _line_, const char **_sfunc_,
       dbug_flush(cs);
     }
   }
-  cs->level= *_slevel_-1;
+  /*
+    Check to not set level < 0. This can happen if DBUG was disabled when
+    function was entered and enabled in function.
+  */
+  cs->level= *_slevel_ != 0 ? *_slevel_-1 : 0;
   cs->func= *_sfunc_;
   cs->file= *_sfile_;
 #ifndef THREAD
@@ -1082,7 +1097,7 @@ void _db_return_(uint _line_, const char **_sfunc_,
 
 void _db_pargs_(uint _line_, const char *keyword)
 {
-  CODE_STATE *cs=0;
+  CODE_STATE *cs;
   get_code_state_or_return;
   cs->u_line= _line_;
   cs->u_keyword= keyword;
@@ -1118,8 +1133,7 @@ void _db_pargs_(uint _line_, const char *keyword)
 void _db_doprnt_(const char *format,...)
 {
   va_list args;
-
-  CODE_STATE *cs=0;
+  CODE_STATE *cs;
   get_code_state_or_return;
 
   va_start(args,format);
@@ -1167,8 +1181,7 @@ void _db_dump_(uint _line_, const char *keyword,
 {
   int pos;
   char dbuff[90];
-
-  CODE_STATE *cs=0;
+  CODE_STATE *cs;
   get_code_state_or_return;
 
   if (_db_keyword_(cs, keyword))
@@ -1480,8 +1493,12 @@ void _db_end_()
 {
   struct settings *discard;
   static struct settings tmp;
-  CODE_STATE *cs=0;
-
+  CODE_STATE *cs;
+  /*
+    Set _dbug_on_ to be able to do full reset even when DEBUGGER_OFF was
+    called after dbug was initialized
+  */
+  _dbug_on_= 1;
   get_code_state_or_return;
 
   while ((discard= cs->stack))
@@ -1568,7 +1585,7 @@ static BOOLEAN DoProfile(CODE_STATE *cs)
 
 FILE *_db_fp_(void)
 {
-  CODE_STATE *cs=0;
+  CODE_STATE *cs;
   get_code_state_or_return NULL;
   return cs->stack->out_file;
 }
@@ -1596,7 +1613,7 @@ FILE *_db_fp_(void)
 
 BOOLEAN _db_strict_keyword_(const char *keyword)
 {
-  CODE_STATE *cs=0;
+  CODE_STATE *cs;
   get_code_state_or_return FALSE;
   if (!DEBUGGING || cs->stack->keywords == NULL)
     return FALSE;
@@ -1630,7 +1647,7 @@ BOOLEAN _db_strict_keyword_(const char *keyword)
 
 BOOLEAN _db_keyword_(CODE_STATE *cs, const char *keyword)
 {
-  get_code_state_or_return FALSE;
+  get_code_state_if_not_set_or_return FALSE;
 
   return (DEBUGGING &&
           (!TRACING || cs->level <= cs->stack->maxdepth) &&
@@ -2149,7 +2166,7 @@ static void ChangeOwner(CODE_STATE *cs, char *pathname)
 
 EXPORT void _db_setjmp_()
 {
-  CODE_STATE *cs=0;
+  CODE_STATE *cs;
   get_code_state_or_return;
 
   cs->jmplevel= cs->level;
@@ -2176,7 +2193,7 @@ EXPORT void _db_setjmp_()
 
 EXPORT void _db_longjmp_()
 {
-  CODE_STATE *cs=0;
+  CODE_STATE *cs;
   get_code_state_or_return;
 
   cs->level= cs->jmplevel;
@@ -2229,9 +2246,7 @@ char *s;
 
 static void dbug_flush(CODE_STATE *cs)
 {
-#ifndef THREAD
   if (cs->stack->flags & FLUSH_ON_WRITE)
-#endif
   {
 #if defined(MSDOS) || defined(__WIN__)
     if (cs->stack->out_file != stdout && cs->stack->out_file != stderr)
@@ -2258,7 +2273,7 @@ static void dbug_flush(CODE_STATE *cs)
 
 void _db_lock_file_()
 {
-  CODE_STATE *cs=0;
+  CODE_STATE *cs;
   get_code_state_or_return;
   pthread_mutex_lock(&THR_LOCK_dbug);
   cs->locked=1;
@@ -2266,7 +2281,7 @@ void _db_lock_file_()
 
 void _db_unlock_file_()
 {
-  CODE_STATE *cs=0;
+  CODE_STATE *cs;
   get_code_state_or_return;
   cs->locked=0;
   pthread_mutex_unlock(&THR_LOCK_dbug);