diff options
| author | unknown <monty@mysql.com/narttu.mysql.fi> | 2007-12-30 22:40:03 +0200 |
|---|---|---|
| committer | unknown <monty@mysql.com/narttu.mysql.fi> | 2007-12-30 22:40:03 +0200 |
| commit | bfd5c273c4fb16396279a8a7b396a6558e5a59b0 (patch) | |
| tree | bd6a19210e59542cdcb83b7320588d66e527e83f | |
| parent | 630169c6d896bd33b65866d91f2bd8894b79bfa1 (diff) | |
| download | mariadb-git-bfd5c273c4fb16396279a8a7b396a6558e5a59b0.tar.gz | |
UNDO of rows now puts back all part of the row on their original pages and positions
Added variable _dbug_on_ to speed up execution when DBUG is not going to be used
Added --debug-on option to mysqld (to be able to turn of DBUG with --debug-on=0)
Fixed some bugs with 'non_flushable' marking of bitmap pages
Don't use 'non_flushable' marking of bitmap pages for not transactional tables
SHOW CREATE TABLE now shows if table was created with page checksums
Fixed a lot of bugs with BLOB handling in case of update/REDO and UNDO
More tests (especially for blobs) and DBUG_ASSERTS()
More readable output from maria_read_log and maria_chk
Fixed wrong shift that caused Maria to crash on files > 4G
Mark tables as crashed of REDO fails
dbug/dbug.c:
Changed to use my_bool (allowed me to remove some windows specific code)
Added variable _dbug_on_ to speed up execution when DBUG is not going to be used
Removed initialization of variables if not needed
include/my_dbug.h:
Use my_bool for some functions that was defined as BOOLEAN in dbug.c code
Added DBUGGER_ON/DEBUGGER_OFF to speed up execution when DBUG is not used
include/my_global.h:
Define my_bool early
Increase MY_HOW_OFTEN_TO_WRITE as computers are now faster than 10 years ago
mysql-test/mysql-test-run.pl:
Added debug-on=0 to speed up tests
mysql-test/r/maria-recovery.result:
Added new test by Guilhem to test if UNDO_ROW_DELETE preserves rowid
mysql-test/r/maria.result:
Added testing of page checksums
mysql-test/t/crash_commit_before-master.opt:
Added --debug-on as test require DBUG to work
mysql-test/t/maria-recovery-bitmap-master.opt:
Added --debug-on as test require DBUG to work
mysql-test/t/maria-recovery-master.opt:
Added --debug-on as test require DBUG to work
mysql-test/t/maria-recovery.test:
Added new test by Guilhem to test if UNDO_ROW_DELETE preserves rowid
mysql-test/t/maria.test:
Added testing of page checksums
sql/mysqld.cc:
Added --debug-on option (to be able to turn of DBUG with --debug-on=0)
Indentation fixes
Removed end spaces
sql/sql_show.cc:
Allow update_create_info() to inform MySQL if PACK_KEYS, NO_PACK_KEYS, CHECKSUM, PAGE_CHECKSUM or DELAY_KEY_WRITE is used
storage/maria/Makefile.am:
Added ma_test_big.sh
storage/maria/ha_maria.cc:
Store in create_info if page checksums are used (For SHOW CREATE TABLE)
storage/maria/ma_bitmap.c:
Added _ma_bitmap_wait_or_flush() to cause reader of bitmap pages to wait with reading until bitmap is flushed.
Use TAIL_PAGE_COUNT_MARKER for tail pages
Set 'sub_blocks' for and only for the head page or for the first extent of a blob. This is needed for store_extent_info() to be able to set START_EXTENT_BIT's
Don't allocate more than 0x3ffff pages in one extent (We need bit 0x4000 as a START_EXTENT_BIT)
Increase the calculated 'head_length' with the number of bytes used for extents.
Update row->space_on_head_page also in _ma_bitmap_find_new_place()
Make _ma_bitmap_get_page_bits() global. (Needed for UNDO handling)
Changed _ma_bitmap_flushable() to take MARIA_HA instead of MARIA_SHARE.
This was needed to be able to mark the handler if we had a 'non_flushable' call pending or not.
Don't use 'non_flushable' marking of bitmap pages for not transactional tables.
Added BLOCKUSED_USE_ORG_BITMAP handling also for tail pages.
Added more DBUG_ASSERT() to find possible errors in other code
Some code simplications by adding new local variables
storage/maria/ma_blockrec.c:
UNDO of rows now puts back all part of the row on their original pages and positions.
Changed UNDO of DELETE and UNDO of UPDATE to contain information about the original length of data on head block and also extent information
This changes a lot of logic as now an insert of a row on a page may happen to any position (and not just to the first or next free)
Use PAGE_COUNT to mark if an extent is the start of of a blob. (Needed for extent_to_bitmap_blocks())
Added check_directory() for checking that directroy entries are correct.
Added checking of row checksums when reading rows (with EXTRA_DEBUG)
Added make_space_for_directory() and extend_directory() for doing expansion of directory
Added get_rowpos_in_head_or_tail_page() to be able to store head/tail on original position in UNDO
Added extent_to_bitmap_blocks() to be able to generate original bitmap blocks from UNDO entry
Added _ma_update_at_original_place() for UNDO of DELETES
Added row->min_length to hold minmum required space needed on head page
Changed find_free_position() to use make_space_for_directory()
Changed make_empty_page() to allow optional creation of directory entry
Changed delete_head_or_tail() and _ma_apply_undo_row_isnert() to not copy pagecache block (speed optimization)
Changed _ma_apply_redo_insert_row_head_or_tail() to be able to insert new row at any position on 'new' page
Changed _ma_apply_undo_row_delete() and _ma_apply_undo_row_update() to put row in it's original position
Ensure allocation of tail blocks are of at least MIN_TAIL_SIZE.
Ensure we store pages in pinned pages even if read failed. (If not we will have pages pinned forever in page cache)
Write original extent information in UNDO entry, not compacted ones (we need position to tails!)
When setting BLOCKUSED_USED, don't clear other bits (we have to preserve BLOCKUSED_USE_ORG_BITMAP)
Fixed som bugs in directory handling
Fixed bug where we wrote wrong lsn to blob pages
Added separate blob_buffer for fixing bug when updating row that had char/varchar that spanned several pages and also had blobs
Ensure we call _ma_bitmap_flushable() also in case of errors
When doing an update, first delete old entries, then search in bitmap for where to put new information
Info->s -> share
Rowid -> rowid
More DBUG_ASSERT()
storage/maria/ma_blockrec.h:
Added START_EXTENT_BIT and TAIL_PAGE_COUNT_MARKER
Added _ma_bitmap_wait_or_flush() and _ma_bitmap_get_page_bits()
storage/maria/ma_check.c:
Don't write extra empty line if there is no deleted blocks
Ignore START_EXTENT_BIT's in page count
Call _ma_fast_unlock_key_del() to free key_del link
storage/maria/ma_close.c:
Ensure that used_key_del is 0. (If not, someone forgot to call _ma_unlock_key_del())
storage/maria/ma_create.c:
Changed constant to macro
storage/maria/ma_delete.c:
For deleted keys, log also position to row
storage/maria/ma_extra.c:
Release blob buffer at maria_reset() if bigger than MARIA_SMALL_BLOB_BUFFER
storage/maria/ma_key_recover.c:
Added bzero() of LSN that confused paged cache in case of uninitialized block
Mark file crashed if applying of index changes fails
Added calls to _ma_fast_unlock_key_del() for protection of shared key_del link.
storage/maria/ma_locking.c:
Added usage of MARIA_FILE_OPEN_COUNT_OFFSET
Added _ma_mark_file_crashed()
storage/maria/ma_loghandler.c:
Fixed bug where we logged uninitialized memory
storage/maria/ma_open.c:
Moved state->changed to be at start of state info on disk to allow one to easly mark files as crashed
storage/maria/ma_page.c:
Disable 'dummy' checksumming of pages as this gave false warnings.
(Need to investigate if this is ever needed)
storage/maria/ma_pagecache.c:
Fixed wrong shift that caused Maria to crash on files > 4G
storage/maria/ma_recovery.c:
In case of errors, start writing on new line if we where in %## %## printing mode (Made errors more readable)
Changed global variable name from warnings -> recovery_warnings
Use MARIA_FILE_CREATE_RENAME_LSN_OFFSET instead of constant
Removed special handling of row position for deleted keys. Keys now always includes row positions
_ma_apply_undo_row_delete() now gets page and row position
Added check that we don't loop forever when handling undo's (in case of bug in undo chain)
Print name of failed REDO/UNDO
storage/maria/ma_recovery.h:
Removed old comment
storage/maria/ma_static.c:
Chaned version number of Maria files to not accidently use old ones (becasue of change of ordering of status variables)
storage/maria/ma_test2.c:
Added option -u to specify number of rows to update
Changed old option -u to be -A, as for ma_test1
Fixed bug in update of rows with blobs (before blobs was always reset to empty on update)
First created blob is now of max blob length to ensure we have at least one big blob in the table
storage/maria/ma_test_all.sh:
More tests
storage/maria/ma_test_recovery.expected:
Updated results
storage/maria/ma_test_recovery:
Changed tests to use bigger blobs (not just 1K)
Added new tests that tests recovery of update with blobs
Removed comparision of .MAD file as it's not guranteed that recovery from scratch gives identical data file as original update
(compact_page() may be called at different times during normal execution and during REDO)
storage/maria/ma_update.c:
Simplify code (changed * to if)
storage/maria/maria_chk.c:
Make output more readable
storage/maria/maria_def.h:
Changed 'changed' to int to prepare for more bits
Added 2 more bytes to status information
Added 'st_mara_row->min_length' for storing min length needed on head page
Added 'st_mara_handler->blob_buff & blob_buff_size' for storing blobs
Moved all tunning parameters into one block
Added MARIA_SMALL_BLOB_BUFFER
Added _ma_mark_file_crashed()
storage/myisam/mi_test2.c:
Fixed bug in update of rows with blobs (before blobs was always reset to empty on update)
storage/maria/ma_test_big.sh:
Testing of insert, update, delete, recovery and undo of rows with blobs
Thanks to the random-ness of ma_test2 this is likely to find most bugs in the row handling
41 files changed, 2287 insertions, 856 deletions
diff --git a/dbug/dbug.c b/dbug/dbug.c index 7bb7396ad5b..b2a70fd7ebe 100644 --- a/dbug/dbug.c +++ b/dbug/dbug.c @@ -123,11 +123,7 @@ * Typedefs to make things more obvious. */ -#ifndef __WIN__ -typedef int BOOLEAN; -#else -#define BOOLEAN BOOL -#endif +#define BOOLEAN my_bool /* * Make it easy to change storage classes if necessary. @@ -216,6 +212,7 @@ struct settings { static BOOLEAN init_done= FALSE; /* Set to TRUE when initialization done */ static struct settings init_settings; static const char *db_process= 0;/* Pointer to process name; argv[0] */ +my_bool _dbug_on_= TRUE; /* FALSE if no debugging at all */ typedef struct _db_code_state_ { const char *process; /* Pointer to process name; usually argv[0] */ @@ -248,7 +245,8 @@ typedef struct _db_code_state_ { The test below is so we could call functions with DBUG_ENTER before my_thread_init(). */ -#define get_code_state_or_return if (!cs && !((cs=code_state()))) return +#define get_code_state_if_not_set_or_return if (!cs && !((cs=code_state()))) return +#define get_code_state_or_return if (!((cs=code_state()))) return /* Handling lists */ static struct link *ListAdd(struct link *, const char *, const char *); @@ -332,13 +330,20 @@ static CODE_STATE *code_state(void) { CODE_STATE *cs, **cs_ptr; + /* + _dbug_on_ is reset if we don't plan to use any debug commands at all and + we want to run on maximum speed + */ + if (!_dbug_on_) + return 0; + if (!init_done) { + init_done=TRUE; pthread_mutex_init(&THR_LOCK_dbug,MY_MUTEX_INIT_FAST); bzero(&init_settings, sizeof(init_settings)); init_settings.out_file=stderr; init_settings.flags=OPEN_APPEND; - init_done=TRUE; } if (!(cs_ptr= (CODE_STATE**) my_thread_var_dbug())) @@ -405,7 +410,7 @@ static CODE_STATE *code_state(void) void _db_process_(const char *name) { - CODE_STATE *cs=0; + CODE_STATE *cs; if (!db_process) db_process= name; @@ -449,10 +454,10 @@ void _db_process_(const char *name) void _db_set_(CODE_STATE *cs, const char *control) { const char *end; - int rel=0; + int rel; struct settings *stack; - get_code_state_or_return; + get_code_state_if_not_set_or_return; stack= cs->stack; if (control[0] == '-' && control[1] == '#') @@ -693,7 +698,7 @@ void _db_set_(CODE_STATE *cs, const char *control) void _db_push_(const char *control) { - CODE_STATE *cs=0; + CODE_STATE *cs; get_code_state_or_return; PushState(cs); _db_set_(cs, control); @@ -742,7 +747,7 @@ void _db_set_init_(const char *control) void _db_pop_() { struct settings *discard; - CODE_STATE *cs=0; + CODE_STATE *cs; get_code_state_or_return; @@ -836,7 +841,7 @@ int _db_explain_ (CODE_STATE *cs, char *buf, size_t len) { char *start=buf, *end=buf+len-4; - get_code_state_or_return *buf=0; + get_code_state_if_not_set_or_return *buf=0; op_list_to_buf('d', cs->stack->keywords, DEBUGGING); op_int_to_buf ('D', cs->stack->delay, 0); @@ -939,9 +944,15 @@ void _db_enter_(const char *_func_, const char *_file_, uint _line_, const char **_sfunc_, const char **_sfile_, uint *_slevel_, char ***_sframep_ __attribute__((unused))) { - int save_errno=errno; - CODE_STATE *cs=0; + int save_errno; + CODE_STATE *cs; + if (!((cs=code_state()))) + { + *_slevel_= 0; /* Set to avoid valgrind warnings if dbug is enabled later */ + return; + } get_code_state_or_return; + save_errno= errno; *_sfunc_= cs->func; *_sfile_= cs->file; @@ -1015,7 +1026,7 @@ void _db_return_(uint _line_, const char **_sfunc_, const char **_sfile_, uint *_slevel_) { int save_errno=errno; - CODE_STATE *cs=0; + CODE_STATE *cs; get_code_state_or_return; if (cs->level != (int) *_slevel_) @@ -1049,7 +1060,11 @@ void _db_return_(uint _line_, const char **_sfunc_, dbug_flush(cs); } } - cs->level= *_slevel_-1; + /* + Check to not set level < 0. This can happen if DBUG was disabled when + function was entered and enabled in function. + */ + cs->level= *_slevel_ != 0 ? *_slevel_-1 : 0; cs->func= *_sfunc_; cs->file= *_sfile_; #ifndef THREAD @@ -1082,7 +1097,7 @@ void _db_return_(uint _line_, const char **_sfunc_, void _db_pargs_(uint _line_, const char *keyword) { - CODE_STATE *cs=0; + CODE_STATE *cs; get_code_state_or_return; cs->u_line= _line_; cs->u_keyword= keyword; @@ -1118,8 +1133,7 @@ void _db_pargs_(uint _line_, const char *keyword) void _db_doprnt_(const char *format,...) { va_list args; - - CODE_STATE *cs=0; + CODE_STATE *cs; get_code_state_or_return; va_start(args,format); @@ -1167,8 +1181,7 @@ void _db_dump_(uint _line_, const char *keyword, { int pos; char dbuff[90]; - - CODE_STATE *cs=0; + CODE_STATE *cs; get_code_state_or_return; if (_db_keyword_(cs, keyword)) @@ -1480,8 +1493,12 @@ void _db_end_() { struct settings *discard; static struct settings tmp; - CODE_STATE *cs=0; - + CODE_STATE *cs; + /* + Set _dbug_on_ to be able to do full reset even when DEBUGGER_OFF was + called after dbug was initialized + */ + _dbug_on_= 1; get_code_state_or_return; while ((discard= cs->stack)) @@ -1568,7 +1585,7 @@ static BOOLEAN DoProfile(CODE_STATE *cs) FILE *_db_fp_(void) { - CODE_STATE *cs=0; + CODE_STATE *cs; get_code_state_or_return NULL; return cs->stack->out_file; } @@ -1596,7 +1613,7 @@ FILE *_db_fp_(void) BOOLEAN _db_strict_keyword_(const char *keyword) { - CODE_STATE *cs=0; + CODE_STATE *cs; get_code_state_or_return FALSE; if (!DEBUGGING || cs->stack->keywords == NULL) return FALSE; @@ -1630,7 +1647,7 @@ BOOLEAN _db_strict_keyword_(const char *keyword) BOOLEAN _db_keyword_(CODE_STATE *cs, const char *keyword) { - get_code_state_or_return FALSE; + get_code_state_if_not_set_or_return FALSE; return (DEBUGGING && (!TRACING || cs->level <= cs->stack->maxdepth) && @@ -2149,7 +2166,7 @@ static void ChangeOwner(CODE_STATE *cs, char *pathname) EXPORT void _db_setjmp_() { - CODE_STATE *cs=0; + CODE_STATE *cs; get_code_state_or_return; cs->jmplevel= cs->level; @@ -2176,7 +2193,7 @@ EXPORT void _db_setjmp_() EXPORT void _db_longjmp_() { - CODE_STATE *cs=0; + CODE_STATE *cs; get_code_state_or_return; cs->level= cs->jmplevel; @@ -2229,9 +2246,7 @@ char *s; static void dbug_flush(CODE_STATE *cs) { -#ifndef THREAD if (cs->stack->flags & FLUSH_ON_WRITE) -#endif { #if defined(MSDOS) || defined(__WIN__) if (cs->stack->out_file != stdout && cs->stack->out_file != stderr) @@ -2258,7 +2273,7 @@ static void dbug_flush(CODE_STATE *cs) void _db_lock_file_() { - CODE_STATE *cs=0; + CODE_STATE *cs; get_code_state_or_return; pthread_mutex_lock(&THR_LOCK_dbug); cs->locked=1; @@ -2266,7 +2281,7 @@ void _db_lock_file_() void _db_unlock_file_() { - CODE_STATE *cs=0; + CODE_STATE *cs; get_code_state_or_return; cs->locked=0; pthread_mutex_unlock(&THR_LOCK_dbug); diff --git a/include/my_dbug.h b/include/my_dbug.h index a77e439b5db..8c0df771ff2 100644 --- a/include/my_dbug.h +++ b/include/my_dbug.h @@ -20,9 +20,10 @@ extern "C" { #endif #if !defined(DBUG_OFF) && !defined(_lint) -struct _db_code_state_; -extern int _db_keyword_(struct _db_code_state_ *cs, const char *keyword); -extern int _db_strict_keyword_(const char *keyword); +struct _db_code_state_; +extern my_bool _dbug_on_; +extern my_bool _db_keyword_(struct _db_code_state_ *cs, const char *keyword); +extern my_bool _db_strict_keyword_(const char *keyword); extern int _db_explain_(struct _db_code_state_ *cs, char *buf, size_t len); extern int _db_explain_init_(char *buf, size_t len); extern void _db_setjmp_(void); @@ -45,7 +46,7 @@ extern void _db_dump_(uint _line_,const char *keyword, extern void _db_end_(void); extern void _db_lock_file_(void); extern void _db_unlock_file_(void); -extern FILE *_db_fp_(void); +extern FILE *_db_fp_(void); #define DBUG_ENTER(a) const char *_db_func_, *_db_file_; uint _db_level_; \ char **_db_framep_; \ @@ -80,6 +81,8 @@ extern FILE *_db_fp_(void); #define DBUG_ASSERT(A) assert(A) #define DBUG_EXPLAIN(buf,len) _db_explain_(0, (buf),(len)) #define DBUG_EXPLAIN_INITIAL(buf,len) _db_explain_init_((buf),(len)) +#define DEBUGGER_OFF _dbug_on_= 0 +#define DEBUGGER_ON _dbug_on_= 1 #define IF_DBUG(A) A #else /* No debugger */ @@ -104,6 +107,8 @@ extern FILE *_db_fp_(void); #define DBUG_ASSERT(A) do { } while(0) #define DBUG_LOCK_FILE #define DBUG_FILE (stderr) +#define DEBUGGER_OFF +#define DEBUGGER_ON #define DBUG_UNLOCK_FILE #define DBUG_EXPLAIN(buf,len) #define DBUG_EXPLAIN_INITIAL(buf,len) diff --git a/include/my_global.h b/include/my_global.h index 56a94927a1e..f63a8a0d1d0 100644 --- a/include/my_global.h +++ b/include/my_global.h @@ -642,6 +642,7 @@ C_MODE_END # endif #endif +typedef char my_bool; /* Small bool */ #include <my_dbug.h> #define MIN_ARRAY_SIZE 0 /* Zero or One. Gcc allows zero*/ @@ -1072,7 +1073,6 @@ typedef off_t os_off_t; typedef uint8 int7; /* Most effective integer 0 <= x <= 127 */ typedef short int15; /* Most effective integer 0 <= x <= 32767 */ typedef int myf; /* Type of MyFlags in my_funcs */ -typedef char my_bool; /* Small bool */ #if !defined(bool) && !defined(bool_defined) && (!defined(HAVE_BOOL) || !defined(__cplusplus)) typedef char bool; /* Ordinary boolean values 0 1 */ #endif @@ -1136,9 +1136,7 @@ typedef char bool; /* Ordinary boolean values 0 1 */ #define SCALE_SEC 100 #define SCALE_USEC 10000 #define MY_HOW_OFTEN_TO_ALARM 2 /* How often we want info on screen */ -#define MY_HOW_OFTEN_TO_WRITE 1000 /* How often we want info on screen */ - - +#define MY_HOW_OFTEN_TO_WRITE 10000 /* How often we want info on screen */ /* Define-funktions for reading and storing in machine independent format diff --git a/mysql-test/mysql-test-run.pl b/mysql-test/mysql-test-run.pl index a4fa415caf6..3ceccdc9565 100755 --- a/mysql-test/mysql-test-run.pl +++ b/mysql-test/mysql-test-run.pl @@ -3039,6 +3039,7 @@ sub install_db ($$) { mtr_add_arg($args, "--loose-skip-innodb"); mtr_add_arg($args, "--loose-skip-ndbcluster"); mtr_add_arg($args, "--sync-frm=0"); + mtr_add_arg($args, "--loose-debug-on=0"); mtr_add_arg($args, "--tmpdir=."); mtr_add_arg($args, "--core-file"); @@ -3780,6 +3781,7 @@ sub mysqld_arguments ($$$$) { $mysqld->{'path_myddir'}); mtr_add_arg($args, "--sync-frm=0"); # Faster test + mtr_add_arg($args, "--loose-debug-on=0"); if ( $mysql_version_id >= 50106 ) { diff --git a/mysql-test/r/maria-recovery.result b/mysql-test/r/maria-recovery.result index 2d4b91c890d..1be16ec821f 100644 --- a/mysql-test/r/maria-recovery.result +++ b/mysql-test/r/maria-recovery.result @@ -186,7 +186,7 @@ t1 CREATE TABLE `t1` ( `c` varchar(6) DEFAULT NULL, PRIMARY KEY (`i`), KEY `c` (`c`) -) ENGINE=MARIA AUTO_INCREMENT=5 DEFAULT CHARSET=latin1 +) ENGINE=MARIA AUTO_INCREMENT=5 DEFAULT CHARSET=latin1 PAGE_CHECKSUM=1 * TEST of UPDATE vs state.auto_increment * copied t1 for feeding_recovery update t1 set i=15 where c="a"; @@ -212,7 +212,28 @@ t1 CREATE TABLE `t1` ( `c` varchar(6) DEFAULT NULL, PRIMARY KEY (`i`), KEY `c` (`c`) -) ENGINE=MARIA AUTO_INCREMENT=16 DEFAULT CHARSET=latin1 +) ENGINE=MARIA AUTO_INCREMENT=16 DEFAULT CHARSET=latin1 PAGE_CHECKSUM=1 +drop table t1; +* TEST of UNDO_ROW_DELETE preserving rowid +create table t1(a int) engine=maria; +insert into t1 values(1),(2); +flush table t1; +* copied t1 for comparison +lock tables t1 write; +insert into t1 values(3); +delete from t1 where a in (1,2,3); +SET SESSION debug="+d,maria_flush_whole_log,maria_crash"; +* crashing mysqld intentionally +set global maria_checkpoint_interval=1; +ERROR HY000: Lost connection to MySQL server during query +* recovery happens +check table t1 extended; +Table Op Msg_type Msg_text +mysqltest.t1 check status OK +* testing that checksum after recovery is as expected +Checksum-check +ok +use mysqltest; drop table t1; drop database mysqltest_for_feeding_recovery; drop database mysqltest_for_comparison; diff --git a/mysql-test/r/maria.result b/mysql-test/r/maria.result index 8c462d6206c..8ed544e730b 100644 --- a/mysql-test/r/maria.result +++ b/mysql-test/r/maria.result @@ -1,5 +1,6 @@ set global storage_engine=maria; set session storage_engine=maria; +set global maria_page_checksum=0; set global maria_log_file_size=4294967296; drop table if exists t1,t2; SET SQL_WARNINGS=1; @@ -2052,7 +2053,7 @@ maria_checkpoint_interval 30 maria_log_file_size 4294959104 maria_log_purge_type immediate maria_max_sort_file_size 9223372036853727232 -maria_page_checksum ON +maria_page_checksum OFF maria_pagecache_age_threshold 300 maria_pagecache_buffer_size 8384512 maria_pagecache_division_limit 100 @@ -2069,6 +2070,14 @@ Maria_pagecache_read_requests # Maria_pagecache_reads # Maria_pagecache_write_requests # Maria_pagecache_writes # +set global maria_page_checksum=1; +create table t1 (a int); +show create table t1; +Table Create Table +t1 CREATE TABLE `t1` ( + `a` int(11) DEFAULT NULL +) ENGINE=MARIA DEFAULT CHARSET=latin1 PAGE_CHECKSUM=1 +drop table t1; create table t1 (s varchar(25), fulltext(s)) TRANSACTIONAL= 1; ERROR HY000: Maria can't yet handle SPATIAL or FULLTEXT keys in transactional mode. For now use TRANSACTIONAL=0 drop table if exists t1; diff --git a/mysql-test/t/crash_commit_before-master.opt b/mysql-test/t/crash_commit_before-master.opt index a745693594e..a5d4a0275f3 100644 --- a/mysql-test/t/crash_commit_before-master.opt +++ b/mysql-test/t/crash_commit_before-master.opt @@ -1,2 +1 @@ ---skip-stack-trace --skip-core-file - +--skip-stack-trace --skip-core-file --loose-debug-on=1 diff --git a/mysql-test/t/maria-recovery-bitmap-master.opt b/mysql-test/t/maria-recovery-bitmap-master.opt index a745693594e..d09f57dc35d 100644 --- a/mysql-test/t/maria-recovery-bitmap-master.opt +++ b/mysql-test/t/maria-recovery-bitmap-master.opt @@ -1,2 +1,2 @@ ---skip-stack-trace --skip-core-file +--skip-stack-trace --skip-core-file --loose-debug-on=1 diff --git a/mysql-test/t/maria-recovery-master.opt b/mysql-test/t/maria-recovery-master.opt index a745693594e..d09f57dc35d 100644 --- a/mysql-test/t/maria-recovery-master.opt +++ b/mysql-test/t/maria-recovery-master.opt @@ -1,2 +1,2 @@ ---skip-stack-trace --skip-core-file +--skip-stack-trace --skip-core-file --loose-debug-on=1 diff --git a/mysql-test/t/maria-recovery.test b/mysql-test/t/maria-recovery.test index 0b70c8702d9..cccde1495f5 100644 --- a/mysql-test/t/maria-recovery.test +++ b/mysql-test/t/maria-recovery.test @@ -179,6 +179,22 @@ let $mvr_crash_statement= set global maria_checkpoint_interval=1; show create table t1; drop table t1; +--echo * TEST of UNDO_ROW_DELETE preserving rowid +# we want recovery to use the tables as they were at time of crash +let $mvr_restore_old_snapshot=0; +# UNDO phase prevents physical comparison, normally, +# so we'll only use checksums to compare. +let $mms_compare_physically=0; +let $mvr_crash_statement= set global maria_checkpoint_interval=1; +create table t1(a int) engine=maria; +insert into t1 values(1),(2); +-- source include/maria_make_snapshot_for_comparison.inc +lock tables t1 write; +insert into t1 values(3); +delete from t1 where a in (1,2,3); +-- source include/maria_verify_recovery.inc +drop table t1; + # clean up everything let $mms_purpose=feeding_recovery; eval drop database mysqltest_for_$mms_purpose; diff --git a/mysql-test/t/maria.test b/mysql-test/t/maria.test index dad107ad474..6913a6e16d7 100644 --- a/mysql-test/t/maria.test +++ b/mysql-test/t/maria.test @@ -5,10 +5,11 @@ -- source include/have_maria.inc -let $default=`select @@global.storage_engine`; +let $default_engine=`select @@global.storage_engine`; +let $default_checksum=`select @@global.maria_page_checksum`; set global storage_engine=maria; set session storage_engine=maria; - +set global maria_page_checksum=0; set global maria_log_file_size=4294967296; # Initialise @@ -1307,6 +1308,14 @@ show variables like 'maria%'; show status like 'maria%'; # +# Show that page_checksum is remembered +# +set global maria_page_checksum=1; +create table t1 (a int); +show create table t1; +drop table t1; + +# # Show that we can't yet create fulltext or spatial index with Maria # --error 138 @@ -1322,6 +1331,6 @@ drop table if exists t1; --disable_result_log --disable_query_log -eval set global storage_engine=$default; +eval set global storage_engine=$default_engine, maria_page_checksum=$default_checksum; --enable_result_log --enable_query_log diff --git a/sql/mysqld.cc b/sql/mysqld.cc index e0cc3253989..ef61c01e264 100644 --- a/sql/mysqld.cc +++ b/sql/mysqld.cc @@ -159,7 +159,7 @@ static void registerwithneb(); static void getvolumename(); static void getvolumeID(BYTE *volumeName); #endif /* __NETWARE__ */ - + #ifdef _AIX41 int initgroups(const char *,unsigned int); @@ -312,7 +312,7 @@ arg_cmp_func Arg_comparator::comparator_matrix[5][2] = const char *log_output_names[] = { "NONE", "FILE", "TABLE", NullS}; static const unsigned int log_output_names_len[]= { 4, 4, 5, 0 }; TYPELIB log_output_typelib= {array_elements(log_output_names)-1,"", - log_output_names, + log_output_names, (unsigned int *) log_output_names_len}; /* static variables */ @@ -335,7 +335,7 @@ static char *default_character_set_name; static char *character_set_filesystem_name; static char *lc_time_names_name; static char *my_bind_addr_str; -static char *default_collation_name; +static char *default_collation_name; static char *default_storage_engine_str; static char compiled_default_collation_name[]= MYSQL_DEFAULT_COLLATION_NAME; static I_List<THD> thread_cache; @@ -363,7 +363,7 @@ bool volatile shutdown_in_progress; @brief 'grant_option' is used to indicate if privileges needs to be checked, in which case the lock, LOCK_grant, is used to protect access to the grant table. - @note This flag is dropped in 5.1 + @note This flag is dropped in 5.1 @see grant_init() */ bool volatile grant_option; @@ -375,7 +375,7 @@ my_bool opt_local_infile, opt_slave_compressed_protocol; my_bool opt_safe_user_create = 0, opt_no_mix_types = 0; my_bool opt_show_slave_auth_info, opt_sql_bin_update = 0; my_bool opt_log_slave_updates= 0; -bool slave_warning_issued = false; +bool slave_warning_issued = false; /* Legacy global handlerton. These will be removed (please do not add more). @@ -1052,18 +1052,18 @@ static void __cdecl kill_server(int sig_ptr) else sql_print_error(ER(ER_GOT_SIGNAL),my_progname,sig); /* purecov: inspected */ -#if defined(HAVE_SMEM) && defined(__WIN__) - /* - Send event to smem_event_connect_request for aborting - */ - if (!SetEvent(smem_event_connect_request)) - { - DBUG_PRINT("error", - ("Got error: %ld from SetEvent of smem_event_connect_request", - GetLastError())); - } -#endif - +#if defined(HAVE_SMEM) && defined(__WIN__) + /* + Send event to smem_event_connect_request for aborting + */ + if (!SetEvent(smem_event_connect_request)) + { + DBUG_PRINT("error", + ("Got error: %ld from SetEvent of smem_event_connect_request", + GetLastError())); + } +#endif + close_connections(); if (sig != MYSQL_KILL_SIGNAL && #ifdef __WIN__ @@ -1589,7 +1589,7 @@ static void network_init(void) if (Service.IsNT() && mysqld_unix_port[0] && !opt_bootstrap && opt_enable_named_pipe) { - + pipe_name[sizeof(pipe_name)-1]= 0; /* Safety if too long string */ strxnmov(pipe_name, sizeof(pipe_name)-1, "\\\\.\\pipe\\", mysqld_unix_port, NullS); @@ -1985,7 +1985,7 @@ static void registerwithneb() { ConsumerRegistrationInfo reg_info; - + /* Clear NEB registration structure */ bzero((char*) ®_info, sizeof(struct ConsumerRegistrationInfo)); @@ -2001,7 +2001,7 @@ static void registerwithneb() reg_info.CRIOwnerID= (LoadDefinitionStructure *)getnlmhandle(); reg_info.CRIConsumerESR= NULL; // No consumer ESR required reg_info.CRISecurityToken= 0; // No security token for the event - reg_info.CRIConsumerFlags= 0; // SMP_ENABLED_BIT; + reg_info.CRIConsumerFlags= 0; // SMP_ENABLED_BIT; reg_info.CRIFilterName= 0; // No event filtering reg_info.CRIFilterDataLength= 0; // No filtering data reg_info.CRIFilterData= 0; // No filtering data @@ -2026,7 +2026,7 @@ static void registerwithneb() Get the NSS volume ID of the MySQL Data volume. Volume ID is stored in a global variable */ - getvolumeID((BYTE*) datavolname); + getvolumeID((BYTE*) datavolname); } @@ -2091,7 +2091,7 @@ static void getvolumeID(BYTE *volumeName) strxmov(path, (const char *) ADMIN_VOL_PATH, (const char *) volumeName, NullS); - if ((status= zOpen(rootKey, zNSS_TASK, zNSPACE_LONG|zMODE_UTF8, + if ((status= zOpen(rootKey, zNSS_TASK, zNSPACE_LONG|zMODE_UTF8, (BYTE *) path, zRR_READ_ACCESS, &fileKey)) != zOK) { consoleprintf("\nGetNSSVolumeProperties - Failed to get file, status: %d\n.", (int) status); @@ -2099,7 +2099,7 @@ static void getvolumeID(BYTE *volumeName) } getInfoMask= zGET_IDS | zGET_VOLUME_INFO ; - if ((status= zGetInfo(fileKey, getInfoMask, sizeof(info), + if ((status= zGetInfo(fileKey, getInfoMask, sizeof(info), zINFO_VERSION_A, &info)) != zOK) { consoleprintf("\nGetNSSVolumeProperties - Failed in zGetInfo, status: %d\n.", (int) status); @@ -2262,7 +2262,7 @@ You should either build a dynamically-linked binary, or force LinuxThreads\n\ to be used with the LD_ASSUME_KERNEL environment variable. Please consult\n\ the documentation for your distribution on how to do that.\n"); #endif - + if (locked_in_memory) { fprintf(stderr, "\n\ @@ -2749,7 +2749,7 @@ static int init_common_variables(const char *conf_file_name, int argc, server_start_time= my_time(0); rpl_filter= new Rpl_filter; binlog_filter= new Rpl_filter; - if (!rpl_filter || !binlog_filter) + if (!rpl_filter || !binlog_filter) { sql_perror("Could not allocate replication and binlog filters"); exit(1); @@ -2769,13 +2769,13 @@ static int init_common_variables(const char *conf_file_name, int argc, } #endif /* - We set SYSTEM time zone as reasonable default and + We set SYSTEM time zone as reasonable default and also for failure of my_tz_init() and bootstrap mode. If user explicitly set time zone with --default-time-zone option we will change this value in my_tz_init(). */ global_system_variables.time_zone= my_tz_SYSTEM; - + /* Init mutexes for the global MYSQL_BIN_LOG objects. As safe_mutex depends on what MY_INIT() does, we can't init the mutexes of @@ -2860,7 +2860,7 @@ static int init_common_variables(const char *conf_file_name, int argc, */ table_cache_size= (ulong) min(max((files-10-max_connections)/2, TABLE_OPEN_CACHE_MIN), - table_cache_size); + table_cache_size); DBUG_PRINT("warning", ("Changed limits: max_open_files: %u max_connections: %ld table_cache: %ld", files, max_connections, table_cache_size)); @@ -2944,7 +2944,7 @@ static int init_common_variables(const char *conf_file_name, int argc, global_system_variables.character_set_client= default_charset_info; global_system_variables.collation_connection= default_charset_info; - if (!(character_set_filesystem= + if (!(character_set_filesystem= get_charset_by_csname(character_set_filesystem_name, MY_CS_PRIMARY, MYF(MY_WME)))) return 1; @@ -2957,7 +2957,7 @@ static int init_common_variables(const char *conf_file_name, int argc, return 1; } global_system_variables.lc_time_names= my_default_lc_time_names; - + sys_init_connect.value_length= 0; if ((sys_init_connect.value= opt_init_connect)) sys_init_connect.value_length= strlen(opt_init_connect); @@ -3077,7 +3077,7 @@ static int init_thread_environment() openssl_stdlocks= (openssl_lock_t*) OPENSSL_malloc(CRYPTO_num_locks() * sizeof(openssl_lock_t)); for (int i= 0; i < CRYPTO_num_locks(); ++i) - (void) my_rwlock_init(&openssl_stdlocks[i].lock, NULL); + (void) my_rwlock_init(&openssl_stdlocks[i].lock, NULL); CRYPTO_set_dynlock_create_callback(openssl_dynlock_create); CRYPTO_set_dynlock_destroy_callback(openssl_dynlock_destroy); CRYPTO_set_dynlock_lock_callback(openssl_lock); @@ -3122,20 +3122,20 @@ static int init_thread_environment() #if defined(HAVE_OPENSSL) && !defined(HAVE_YASSL) static unsigned long openssl_id_function() -{ +{ return (unsigned long) pthread_self(); -} +} static openssl_lock_t *openssl_dynlock_create(const char *file, int line) -{ +{ openssl_lock_t *lock= new openssl_lock_t; my_rwlock_init(&lock->lock, NULL); return lock; } -static void openssl_dynlock_destroy(openssl_lock_t *lock, const char *file, +static void openssl_dynlock_destroy(openssl_lock_t *lock, const char *file, int line) { rwlock_destroy(&lock->lock); @@ -3155,7 +3155,7 @@ static void openssl_lock_function(int mode, int n, const char *file, int line) } -static void openssl_lock(int mode, openssl_lock_t *lock, const char *file, +static void openssl_lock(int mode, openssl_lock_t *lock, const char *file, int line) { int err; @@ -3180,7 +3180,7 @@ static void openssl_lock(int mode, openssl_lock_t *lock, const char *file, sql_print_error("Fatal: OpenSSL interface problem (mode=0x%x)", mode); abort(); } - if (err) + if (err) { sql_print_error("Fatal: can't %s OpenSSL lock", what); abort(); @@ -3363,7 +3363,7 @@ with --log-bin instead."); if (opt_binlog_format_id == BINLOG_FORMAT_UNSPEC) global_system_variables.binlog_format= BINLOG_FORMAT_MIXED; else - { + { DBUG_ASSERT(global_system_variables.binlog_format != BINLOG_FORMAT_UNSPEC); } @@ -3526,7 +3526,7 @@ server."); strlen(default_storage_engine_str) }; plugin_ref plugin; handlerton *hton; - + if ((plugin= ha_resolve_by_name(0, &name))) hton= plugin_data(plugin, handlerton*); else @@ -3548,7 +3548,7 @@ server."); else { /* - Need to unlock as global_system_variables.table_plugin + Need to unlock as global_system_variables.table_plugin was acquired during plugin_init() */ plugin_unlock(0, global_system_variables.table_plugin); @@ -3701,7 +3701,7 @@ static void handle_connections_methods() handler_count--; } } -#endif +#endif while (handler_count > 0) pthread_cond_wait(&COND_handler_count,&LOCK_thread_count); @@ -3714,7 +3714,7 @@ void decrement_handler_count() pthread_mutex_lock(&LOCK_thread_count); handler_count--; pthread_cond_signal(&COND_handler_count); - pthread_mutex_unlock(&LOCK_thread_count); + pthread_mutex_unlock(&LOCK_thread_count); my_thread_end(); } #else @@ -3878,7 +3878,7 @@ we force server id to 2, but this MySQL server will not act as a slave."); } else { - /* Don't show error dialog box when on foreground: it stops the server */ + /* Don't show error dialog box when on foreground: it stops the server */ SetErrorMode(SEM_NOOPENFILEERRORBOX | SEM_FAILCRITICALERRORS); } #endif @@ -3984,7 +3984,7 @@ we force server id to 2, but this MySQL server will not act as a slave."); #endif /* __NT__ */ /* (void) pthread_attr_destroy(&connection_attrib); */ - + DBUG_PRINT("quit",("Exiting main thread")); #ifndef __WIN__ @@ -4288,7 +4288,7 @@ static bool read_init_file(char *file_name) When we enter this function, LOCK_thread_count is hold! */ - + void handle_connection_in_main_thread(THD *thd) { safe_mutex_assert_owner(&LOCK_thread_count); @@ -4510,7 +4510,7 @@ pthread_handler_t handle_connections_sockets(void *arg __attribute__((unused))) size_socket length=sizeof(struct sockaddr_in); new_sock = accept(sock, my_reinterpret_cast(struct sockaddr *) (&cAddr), &length); -#ifdef __NETWARE__ +#ifdef __NETWARE__ // TODO: temporary fix, waiting for TCP/IP fix - DEFECT000303149 if ((new_sock == INVALID_SOCKET) && (socket_errno == EINVAL)) { @@ -4911,7 +4911,7 @@ errorconn: NullS); sql_perror(buff); } - if (handle_client_file_map) + if (handle_client_file_map) CloseHandle(handle_client_file_map); if (handle_client_map) UnmapViewOfFile(handle_client_map); @@ -4959,8 +4959,8 @@ error: enum options_mysqld { - OPT_ISAM_LOG=256, OPT_SKIP_NEW, - OPT_SKIP_GRANT, OPT_SKIP_LOCK, + OPT_ISAM_LOG=256, OPT_SKIP_NEW, + OPT_SKIP_GRANT, OPT_SKIP_LOCK, OPT_ENABLE_LOCK, OPT_USE_LOCKING, OPT_SOCKET, OPT_UPDATE_LOG, OPT_BIN_LOG, OPT_SKIP_RESOLVE, @@ -4989,18 +4989,18 @@ enum options_mysqld #ifndef DBUG_OFF OPT_BINLOG_SHOW_XID, #endif - OPT_BINLOG_ROWS_EVENT_MAX_SIZE, + OPT_BINLOG_ROWS_EVENT_MAX_SIZE, OPT_WANT_CORE, OPT_CONCURRENT_INSERT, OPT_MEMLOCK, OPT_MYISAM_RECOVER, OPT_REPLICATE_REWRITE_DB, OPT_SERVER_ID, - OPT_SKIP_SLAVE_START, OPT_SAFE_SHOW_DB, + OPT_SKIP_SLAVE_START, OPT_SAFE_SHOW_DB, OPT_SAFEMALLOC_MEM_LIMIT, OPT_REPLICATE_DO_TABLE, OPT_REPLICATE_IGNORE_TABLE, OPT_REPLICATE_WILD_DO_TABLE, OPT_REPLICATE_WILD_IGNORE_TABLE, OPT_REPLICATE_SAME_SERVER_ID, OPT_DISCONNECT_SLAVE_EVENT_COUNT, OPT_TC_HEURISTIC_RECOVER, OPT_ABORT_SLAVE_EVENT_COUNT, OPT_LOG_BIN_TRUST_FUNCTION_CREATORS, - OPT_ENGINE_CONDITION_PUSHDOWN, OPT_NDB_CONNECTSTRING, + OPT_ENGINE_CONDITION_PUSHDOWN, OPT_NDB_CONNECTSTRING, OPT_NDB_USE_EXACT_COUNT, OPT_NDB_USE_TRANSACTIONS, OPT_NDB_FORCE_SEND, OPT_NDB_AUTOINCREMENT_PREFETCH_SZ, OPT_NDB_SHM, OPT_NDB_OPTIMIZED_NODE_SELECTION, OPT_NDB_CACHE_CHECK_TIME, @@ -5119,7 +5119,7 @@ enum options_mysqld OPT_SECURE_FILE_PRIV, OPT_MIN_EXAMINED_ROW_LIMIT, OPT_LOG_SLOW_SLAVE_STATEMENTS, - OPT_DEBUG_CRC, OPT_OLD_MODE + OPT_DEBUG_CRC, OPT_DEBUG_ON, OPT_OLD_MODE }; @@ -5127,7 +5127,7 @@ enum options_mysqld struct my_option my_long_options[] = { - {"help", '?', "Display this help and exit.", + {"help", '?', "Display this help and exit.", (uchar**) &opt_help, (uchar**) &opt_help, 0, GET_BOOL, NO_ARG, 0, 0, 0, 0, 0, 0}, #ifdef HAVE_REPLICATION @@ -5193,11 +5193,11 @@ struct my_option my_long_options[] = "The maximum size of a row-based binary log event in bytes. Rows will be " "grouped into events smaller than this size if possible. " "The value has to be a multiple of 256.", - (uchar**) &opt_binlog_rows_event_max_size, - (uchar**) &opt_binlog_rows_event_max_size, 0, - GET_ULONG, REQUIRED_ARG, - /* def_value */ 1024, /* min_value */ 256, /* max_value */ ULONG_MAX, - /* sub_size */ 0, /* block_size */ 256, + (uchar**) &opt_binlog_rows_event_max_size, + (uchar**) &opt_binlog_rows_event_max_size, 0, + GET_ULONG, REQUIRED_ARG, + /* def_value */ 1024, /* min_value */ 256, /* max_value */ ULONG_MAX, + /* sub_size */ 0, /* block_size */ 256, /* app_type */ 0 }, #ifndef DISABLE_GRANT_OPTIONS @@ -5248,6 +5248,11 @@ struct my_option my_long_options[] = "Call my_debug_put_break_here() if crc matches this number (for debug).", (uchar**) &opt_my_crc_dbug_check, (uchar**) &opt_my_crc_dbug_check, 0, GET_ULONG, REQUIRED_ARG, 0, 0, ~(ulong) 0L, 0, 0, 0}, + {"debug-on", OPT_DEBUG_ON, + "Enable all DBUG commands. Needed if you want to use DBUG_EXECUTE without " + "starting mysqld with --debug", + (uchar**) &_dbug_on_, (uchar**) 0, 0, GET_BOOL, NO_ARG, 1, + 0, 0, 0, 0, 0}, #endif {"default-character-set", 'C', "Set the default character set (deprecated option, use --character-set-server instead).", (uchar**) &default_character_set_name, (uchar**) &default_character_set_name, @@ -5398,7 +5403,7 @@ Disable with --skip-large-pages.", (uchar**) &myisam_log_filename, (uchar**) &myisam_log_filename, 0, GET_STR, OPT_ARG, 0, 0, 0, 0, 0, 0}, {"log-long-format", '0', - "Log some extra information to update log. Please note that this option is deprecated; see --log-short-format option.", + "Log some extra information to update log. Please note that this option is deprecated; see --log-short-format option.", 0, 0, 0, GET_NO_ARG, NO_ARG, 0, 0, 0, 0, 0, 0}, #ifdef WITH_CSV_STORAGE_ENGINE {"log-output", OPT_LOG_OUTPUT, @@ -5630,7 +5635,7 @@ master-ssl", "Force ndbcluster to always copy tables at alter table (should only be used if on-line alter table fails).", (uchar**) &global_system_variables.ndb_use_copying_alter_table, (uchar**) &global_system_variables.ndb_use_copying_alter_table, - 0, GET_BOOL, NO_ARG, 0, 0, 0, 0, 0, 0}, + 0, GET_BOOL, NO_ARG, 0, 0, 0, 0, 0, 0}, {"new", 'n', "Use very new possible 'unsafe' functions.", (uchar**) &global_system_variables.new_mode, (uchar**) &max_system_variables.new_mode, @@ -5869,7 +5874,7 @@ log and this option does nothing anymore.", 0, 0, 0, 0, 0}, {"timed_mutexes", OPT_TIMED_MUTEXES, "Specify whether to time mutexes (only InnoDB mutexes are currently supported)", - (uchar**) &timed_mutexes, (uchar**) &timed_mutexes, 0, GET_BOOL, NO_ARG, 0, + (uchar**) &timed_mutexes, (uchar**) &timed_mutexes, 0, GET_BOOL, NO_ARG, 0, 0, 0, 0, 0, 0}, {"tmpdir", 't', "Path for temporary files. Several paths may be specified, separated by a " @@ -6009,7 +6014,7 @@ log and this option does nothing anymore.", "This characterizes the number of hits a hot block has to be untouched until it is considered aged enough to be downgraded to a warm block. This specifies the percentage ratio of that number of hits to the total number of blocks in key cache", (uchar**) &dflt_key_cache_var.param_age_threshold, (uchar**) 0, - 0, (GET_ULONG | GET_ASK_ADDR), REQUIRED_ARG, + 0, (GET_ULONG | GET_ASK_ADDR), REQUIRED_ARG, 300, 100, ~0L, 0, 100, 0}, {"key_cache_block_size", OPT_KEY_CACHE_BLOCK_SIZE, "The default size of key cache blocks", @@ -6165,7 +6170,7 @@ The minimum value for this variable is 4096.", {"myisam_use_mmap", OPT_MYISAM_USE_MMAP, "Use memory mapping for reading and writing MyISAM tables", (uchar**) &opt_myisam_use_mmap, - (uchar**) &opt_myisam_use_mmap, 0, GET_BOOL, NO_ARG, 0, + (uchar**) &opt_myisam_use_mmap, 0, GET_BOOL, NO_ARG, 0, 0, 0, 0, 0, 0}, {"myisam_stats_method", OPT_MYISAM_STATS_METHOD, "Specifies how MyISAM index statistics collection code should threat NULLs. " @@ -6193,9 +6198,9 @@ The minimum value for this variable is 4096.", (uchar**) &global_system_variables.net_write_timeout, (uchar**) &max_system_variables.net_write_timeout, 0, GET_ULONG, REQUIRED_ARG, NET_WRITE_TIMEOUT, 1, LONG_TIMEOUT, 0, 1, 0}, - {"old", OPT_OLD_MODE, "Use compatible behavior.", + {"old", OPT_OLD_MODE, "Use compatible behavior.", (uchar**) &global_system_variables.old_mode, - (uchar**) &max_system_variables.old_mode, 0, GET_BOOL, NO_ARG, + (uchar**) &max_system_variables.old_mode, 0, GET_BOOL, NO_ARG, 0, 0, 0, 0, 0, 0}, {"open_files_limit", OPT_OPEN_FILES_LIMIT, "If this is not 0, then mysqld will use this value to reserve file descriptors to use with setrlimit(). If this value is 0 then mysqld will reserve max_connections*5 or max_connections + table_cache*2 (whichever is larger) number of files.", @@ -6662,7 +6667,7 @@ static int show_ssl_ctx_get_session_cache_mode(THD *thd, SHOW_VAR *var, char *bu } /* - Functions relying on SSL + Functions relying on SSL Note: In the show_ssl_* functions, we need to check if we have a valid vio-object since this isn't always true, specifically when session_status or global_status is requested from @@ -7295,6 +7300,7 @@ mysqld_get_one_option(int optid, #ifndef DBUG_OFF DBUG_SET_INITIAL(argument ? argument : default_dbug_option); #endif + DEBUGGER_ON; opt_endinfo=1; /* unireg: memory allocation */ break; case 'a': @@ -7589,16 +7595,16 @@ mysqld_get_one_option(int optid, case OPT_MASTER_PASSWORD: case OPT_MASTER_PORT: case OPT_MASTER_CONNECT_RETRY: - case OPT_MASTER_SSL: + case OPT_MASTER_SSL: case OPT_MASTER_SSL_KEY: - case OPT_MASTER_SSL_CERT: + case OPT_MASTER_SSL_CERT: case OPT_MASTER_SSL_CAPATH: case OPT_MASTER_SSL_CIPHER: case OPT_MASTER_SSL_CA: if (!slave_warning_issued) //only show the warning once { - slave_warning_issued = true; - WARN_DEPRECATED(NULL, "5.2", "for replication startup options", + slave_warning_issued = true; + WARN_DEPRECATED(NULL, "5.2", "for replication startup options", "'CHANGE MASTER'"); } break; @@ -8055,7 +8061,7 @@ static ulong find_bit_type_or_exit(const char *x, TYPELIB *bit_lib, ulong res; const char **ptr; - + if ((res= find_bit_type(x, bit_lib)) == ~(ulong) 0) { ptr= bit_lib->type_names; diff --git a/sql/sql_show.cc b/sql/sql_show.cc index 4280722607b..32a1fab6a43 100644 --- a/sql/sql_show.cc +++ b/sql/sql_show.cc @@ -1185,8 +1185,10 @@ int store_create_info(THD *thd, TABLE_LIST *table_list, String *packet, key_info= table->key_info; bzero((char*) &create_info, sizeof(create_info)); - /* Allow update_create_info to update row type */ + /* Allow update_create_info to update row type, page checksums and options */ create_info.row_type= share->row_type; + create_info.page_checksum= share->page_checksum; + create_info.options= share->db_create_options; file->update_create_info(&create_info); primary_key= share->primary_key; @@ -1367,19 +1369,19 @@ int store_create_info(THD *thd, TABLE_LIST *table_list, String *packet, packet->append(buff, (uint) (end - buff)); } - if (share->db_create_options & HA_OPTION_PACK_KEYS) + if (create_info.options & HA_OPTION_PACK_KEYS) packet->append(STRING_WITH_LEN(" PACK_KEYS=1")); - if (share->db_create_options & HA_OPTION_NO_PACK_KEYS) + if (create_info.options & HA_OPTION_NO_PACK_KEYS) packet->append(STRING_WITH_LEN(" PACK_KEYS=0")); /* We use CHECKSUM, instead of TABLE_CHECKSUM, for backward compability */ - if (share->db_create_options & HA_OPTION_CHECKSUM) + if (create_info.options & HA_OPTION_CHECKSUM) packet->append(STRING_WITH_LEN(" CHECKSUM=1")); - if (share->page_checksum != HA_CHOICE_UNDEF) + if (create_info.page_checksum != HA_CHOICE_UNDEF) { packet->append(STRING_WITH_LEN(" PAGE_CHECKSUM=")); - packet->append(ha_choice_values[(uint) share->page_checksum], 1); + packet->append(ha_choice_values[create_info.page_checksum], 1); } - if (share->db_create_options & HA_OPTION_DELAY_KEY_WRITE) + if (create_info.options & HA_OPTION_DELAY_KEY_WRITE) packet->append(STRING_WITH_LEN(" DELAY_KEY_WRITE=1")); if (create_info.row_type != ROW_TYPE_DEFAULT) { diff --git a/storage/maria/Makefile.am b/storage/maria/Makefile.am index 07b59526eba..3a5eabd539d 100644 --- a/storage/maria/Makefile.am +++ b/storage/maria/Makefile.am @@ -30,7 +30,8 @@ DEFS = @DEFS@ # "." is needed first because tests in unittest need libmaria SUBDIRS = . unittest -EXTRA_DIST = ma_test_all.sh ma_test_all.res ma_ft_stem.c CMakeLists.txt plug.in ma_test_recovery +EXTRA_DIST = ma_test_all.sh ma_test_all.res ma_test_big.sh \ + ma_ft_stem.c CMakeLists.txt plug.in ma_test_recovery pkgdata_DATA = ma_test_all ma_test_all.res ma_test_recovery pkglib_LIBRARIES = libmaria.a bin_PROGRAMS = maria_chk maria_pack maria_ftdump maria_read_log diff --git a/storage/maria/ha_maria.cc b/storage/maria/ha_maria.cc index cd13f19d646..06b3d7d136a 100644 --- a/storage/maria/ha_maria.cc +++ b/storage/maria/ha_maria.cc @@ -2169,6 +2169,12 @@ void ha_maria::update_create_info(HA_CREATE_INFO *create_info) if (create_info->row_type != ROW_TYPE_DEFAULT && !(create_info->used_fields & HA_CREATE_USED_ROW_FORMAT)) create_info->row_type= get_row_type(); + /* + Show always page checksums, as this can be forced with + maria_page_checksums variable + */ + if (file->s->options & HA_OPTION_PAGE_CHECKSUM) + create_info->page_checksum= HA_CHOICE_YES; } @@ -2267,7 +2273,8 @@ int ha_maria::create(const char *name, register TABLE *table_arg, create_flags|= HA_CREATE_CHECKSUM; if (options & HA_OPTION_DELAY_KEY_WRITE) create_flags|= HA_CREATE_DELAY_KEY_WRITE; - if ((ha_create_info->page_checksum == HA_CHOICE_UNDEF && maria_page_checksums) || + if ((ha_create_info->page_checksum == HA_CHOICE_UNDEF && + maria_page_checksums) || ha_create_info->page_checksum == HA_CHOICE_YES) create_flags|= HA_CREATE_PAGE_CHECKSUM; diff --git a/storage/maria/ma_bitmap.c b/storage/maria/ma_bitmap.c index 447e0de9ad7..4effd443328 100644 --- a/storage/maria/ma_bitmap.c +++ b/storage/maria/ma_bitmap.c @@ -319,8 +319,8 @@ my_bool _ma_bitmap_flush(MARIA_SHARE *share) pthread_mutex_lock(&share->bitmap.bitmap_lock); if (share->bitmap.changed) { - res= write_changed_bitmap(share, &share->bitmap); share->bitmap.changed= 0; + res= write_changed_bitmap(share, &share->bitmap); } pthread_mutex_unlock(&share->bitmap.bitmap_lock); } @@ -328,6 +328,38 @@ my_bool _ma_bitmap_flush(MARIA_SHARE *share) } +/* + @brief Send updated bitmap to the page cache if bitmap is free + + @note + This is used by reader threads which don't unpin things +*/ + +my_bool _ma_bitmap_wait_or_flush(MARIA_SHARE *share) +{ + my_bool res= 0; + MARIA_FILE_BITMAP *bitmap= &share->bitmap; + DBUG_ENTER("_ma_bitmap_flush"); + if (bitmap->changed) + { + pthread_mutex_lock(&bitmap->bitmap_lock); + while (bitmap->non_flushable && bitmap->changed) + { + DBUG_PRINT("info", ("waiting for bitmap to be flushable")); + pthread_cond_wait(&bitmap->bitmap_cond, &bitmap->bitmap_lock); + } + if (bitmap->changed) + { + bitmap->changed= 0; + res= write_changed_bitmap(share, bitmap); + } + pthread_mutex_unlock(&bitmap->bitmap_lock); + } + DBUG_RETURN(res); +} + + + /** Dirty-page filtering criteria for bitmap pages @@ -910,9 +942,9 @@ static void fill_block(MARIA_FILE_BITMAP *bitmap, /* For each 6 bytes we have 6*8/3= 16 patterns */ page= (best_data - bitmap->map) / 6 * 16 + best_pos; block->page= bitmap->page + 1 + page; - block->page_count= 1 + TAIL_BIT; + block->page_count= TAIL_PAGE_COUNT_MARKER; block->empty_space= pattern_to_size(bitmap, best_bits); - block->sub_blocks= 1; + block->sub_blocks= 0; block->org_bitmap_value= best_bits; block->used= BLOCKUSED_TAIL; /* See _ma_bitmap_release_unused() */ @@ -1244,7 +1276,7 @@ static ulong allocate_full_pages(MARIA_FILE_BITMAP *bitmap, block->page= bitmap->page + 1 + page; block->page_count= best_area_size; block->empty_space= 0; - block->sub_blocks= 1; + block->sub_blocks= 0; block->org_bitmap_value= 0; block->used= 0; DBUG_PRINT("info", ("page: %lu page_count: %u", @@ -1432,6 +1464,7 @@ static my_bool find_blob(MARIA_HA *info, ulong length) rest_length= 0; } + first_block_pos= info->bitmap_blocks.elements; if (pages) { MARIA_BITMAP_BLOCK *block; @@ -1439,15 +1472,17 @@ static my_bool find_blob(MARIA_HA *info, ulong length) info->bitmap_blocks.elements + pages / BLOB_SEGMENT_MIN_SIZE + 2)) DBUG_RETURN(1); - first_block_pos= info->bitmap_blocks.elements; block= dynamic_element(&info->bitmap_blocks, info->bitmap_blocks.elements, MARIA_BITMAP_BLOCK*); - first_block= block; do { + /* + We use 0x3fff here as the two upmost bits are reserved for + TAIL_BIT and START_EXTENT_BIT + */ used= allocate_full_pages(bitmap, - (pages >= 65535 ? 65535 : (uint) pages), block, - 0); + (pages >= 0x3fff ? 0x3fff : (uint) pages), + block, 0); if (!used) { if (move_to_next_bitmap(info, bitmap)) @@ -1464,8 +1499,9 @@ static my_bool find_blob(MARIA_HA *info, ulong length) if (rest_length && find_tail(info, rest_length, info->bitmap_blocks.elements++)) DBUG_RETURN(1); - if (first_block) - first_block->sub_blocks= info->bitmap_blocks.elements - first_block_pos; + first_block= dynamic_element(&info->bitmap_blocks, first_block_pos, + MARIA_BITMAP_BLOCK*); + first_block->sub_blocks= info->bitmap_blocks.elements - first_block_pos; DBUG_RETURN(0); } @@ -1535,7 +1571,6 @@ static void use_head(MARIA_HA *info, ulonglong page, uint size, block->page= page; block->page_count= 1 + TAIL_BIT; block->empty_space= size; - block->sub_blocks= 1; block->used= BLOCKUSED_TAIL; /* @@ -1572,9 +1607,16 @@ static void use_head(MARIA_HA *info, ulonglong page, uint size, static uint find_where_to_split_row(MARIA_SHARE *share, MARIA_ROW *row, uint extents_length, uint split_size) { - uint row_length= row->base_length; uint *lengths, *lengths_end; - + /* + Ensure we have the minimum required space on head page: + - Header + length of field lengths (row->min_length) + - Number of extents + - One extent + */ + uint row_length= (row->min_length + + size_to_store_key_length(extents_length) + + ROW_EXTENT_SIZE); DBUG_ASSERT(row_length < split_size); /* Store first in all_field_lengths the different parts that are written @@ -1716,14 +1758,19 @@ my_bool _ma_bitmap_find_place(MARIA_HA *info, MARIA_ROW *row, } /* - First allocate all blobs (so that we can find out the needed size for + First allocate all blobs so that we can find out the needed size for the main block. */ if (row->blob_length && allocate_blobs(info, row)) goto abort; extents_length= row->extents_count * ROW_EXTENT_SIZE; - if ((head_length= (row->head_length + extents_length)) <= max_page_size) + /* + The + 3 here is space to be able to store the number of segments + in the row header. + */ + if ((head_length= (row->head_length + extents_length + 3)) <= + max_page_size) { /* Main row part fits into one page */ position= ELEMENTS_RESERVED_FOR_MAIN_PART - 1; @@ -1747,6 +1794,7 @@ my_bool _ma_bitmap_find_place(MARIA_HA *info, MARIA_ROW *row, if (find_head(info, row_length, position)) goto abort; row->space_on_head_page= row_length; + rest_length= head_length - row_length; if (write_rest_of_head(info, position, rest_length)) goto abort; @@ -1820,11 +1868,12 @@ my_bool _ma_bitmap_find_new_place(MARIA_HA *info, MARIA_ROW *row, goto abort; extents_length= row->extents_count * ROW_EXTENT_SIZE; - if ((head_length= (row->head_length + extents_length)) <= free_size) + if ((head_length= (row->head_length + extents_length + 3)) <= free_size) { /* Main row part fits into one page */ position= ELEMENTS_RESERVED_FOR_MAIN_PART - 1; use_head(info, page, head_length, position); + row->space_on_head_page= head_length; goto end; } @@ -1838,8 +1887,9 @@ my_bool _ma_bitmap_find_new_place(MARIA_HA *info, MARIA_ROW *row, if (head_length - row_length < MAX_TAIL_SIZE(share->block_size)) position= ELEMENTS_RESERVED_FOR_MAIN_PART -2; /* Only head and tail */ use_head(info, page, row_length, position); - rest_length= head_length - row_length; + row->space_on_head_page= row_length; + rest_length= head_length - row_length; if (write_rest_of_head(info, position, rest_length)) goto abort; @@ -1936,13 +1986,13 @@ static my_bool set_page_bits(MARIA_HA *info, MARIA_FILE_BITMAP *bitmap, ~0 Error (couldn't read page) */ -static uint get_page_bits(MARIA_HA *info, MARIA_FILE_BITMAP *bitmap, - ulonglong page) +uint _ma_bitmap_get_page_bits(MARIA_HA *info, MARIA_FILE_BITMAP *bitmap, + ulonglong page) { ulonglong bitmap_page; uint offset_page, offset, tmp; uchar *data; - DBUG_ENTER("get_page_bits"); + DBUG_ENTER("_ma_bitmap_get_page_bits"); bitmap_page= page - page % bitmap->pages_covered; if (bitmap_page != bitmap->page && @@ -1994,6 +2044,8 @@ my_bool _ma_bitmap_reset_full_page_bits(MARIA_HA *info, safe_mutex_assert_owner(&info->s->bitmap.bitmap_lock); bitmap_page= page - page % bitmap->pages_covered; + DBUG_ASSERT(page != bitmap_page); + if (bitmap_page != bitmap->page && _ma_change_bitmap_page(info, bitmap, bitmap_page)) DBUG_RETURN(1); @@ -2116,6 +2168,7 @@ my_bool _ma_bitmap_set_full_page_bits(MARIA_HA *info, /** + @brief Make a transition of MARIA_FILE_BITMAP::non_flushable. If the bitmap becomes flushable, which requires that REDO-UNDO has been logged and all bitmap pages touched by the thread have a correct @@ -2125,29 +2178,34 @@ my_bool _ma_bitmap_set_full_page_bits(MARIA_HA *info, unless a concurrent _ma_bitmap_flush_all() is happening, in which case the function first waits for the flush to be done. + @note + info->non_flushable_state is set to 1 if we have incremented + bitmap->info->non_flushable and not yet decremented it. + @param share Table's share @param non_flushable_inc Increment of MARIA_FILE_BITMAP::non_flushable (-1 or +1). */ -void _ma_bitmap_flushable(MARIA_SHARE *share, int non_flushable_inc) +void _ma_bitmap_flushable(MARIA_HA *info, int non_flushable_inc) { + MARIA_SHARE *share= info->s; MARIA_FILE_BITMAP *bitmap; /* Not transactional tables are never automaticly flushed and needs no protection */ -#ifndef EXTRA_DEBUG if (!share->now_transactional) return; -#endif bitmap= &share->bitmap; if (non_flushable_inc == -1) { pthread_mutex_lock(&bitmap->bitmap_lock); - DBUG_ASSERT(bitmap->non_flushable > 0); + DBUG_ASSERT((int) bitmap->non_flushable > 0 && + info->non_flushable_state == 1); + info->non_flushable_state= 0; if (--bitmap->non_flushable == 0) { _ma_bitmap_unpin_all(share); @@ -2161,7 +2219,7 @@ void _ma_bitmap_flushable(MARIA_SHARE *share, int non_flushable_inc) pthread_mutex_unlock(&bitmap->bitmap_lock); return; } - DBUG_ASSERT(non_flushable_inc == 1); + DBUG_ASSERT(non_flushable_inc == 1 && info->non_flushable_state == 0); /* It is a read without mutex because only an optimization */ if (unlikely(bitmap->flush_all_requested)) { @@ -2189,6 +2247,7 @@ void _ma_bitmap_flushable(MARIA_SHARE *share, int non_flushable_inc) touch it we will take the mutex. */ bitmap->non_flushable++; + info->non_flushable_state= 1; DBUG_PRINT("info", ("bitmap->non_flushable: %u", bitmap->non_flushable)); } @@ -2240,17 +2299,24 @@ my_bool _ma_bitmap_release_unused(MARIA_HA *info, MARIA_BITMAP_BLOCKS *blocks) /* First handle head block */ if (block->used & BLOCKUSED_USED) { - DBUG_PRINT("info", ("head empty_space: %u", block->empty_space)); + DBUG_PRINT("info", ("head page: %lu empty_space: %u", + (ulong) block->page, block->empty_space)); bits= _ma_free_size_to_head_pattern(bitmap, block->empty_space); if (block->used & BLOCKUSED_USE_ORG_BITMAP) current_bitmap_value= block->org_bitmap_value; } else bits= block->org_bitmap_value; - if (bits != current_bitmap_value && - set_page_bits(info, bitmap, block->page, bits)) - goto err; - + if (bits != current_bitmap_value) + { + if (set_page_bits(info, bitmap, block->page, bits)) + goto err; + } + else + { + DBUG_ASSERT(current_bitmap_value == + _ma_bitmap_get_page_bits(info, bitmap, block->page)); + } /* Handle all full pages and tail pages (for head page and blob) */ for (block++; block < end; block++) @@ -2262,12 +2328,16 @@ my_bool _ma_bitmap_release_unused(MARIA_HA *info, MARIA_BITMAP_BLOCKS *blocks) page_count= block->page_count; if (block->used & BLOCKUSED_TAIL) { + current_bitmap_value= FULL_TAIL_PAGE; /* The bitmap page is only one page */ page_count= 1; if (block->used & BLOCKUSED_USED) { - DBUG_PRINT("info", ("tail empty_space: %u", block->empty_space)); + DBUG_PRINT("info", ("tail page: %lu empty_space: %u", + (ulong) block->page, block->empty_space)); bits= free_size_to_tail_pattern(bitmap, block->empty_space); + if (block->used & BLOCKUSED_USE_ORG_BITMAP) + current_bitmap_value= block->org_bitmap_value; } else bits= block->org_bitmap_value; @@ -2276,7 +2346,7 @@ my_bool _ma_bitmap_release_unused(MARIA_HA *info, MARIA_BITMAP_BLOCKS *blocks) The page has all bits set; The following test is an optimization to not set the bits to the same value as before. */ - if (bits != FULL_TAIL_PAGE && + if (bits != current_bitmap_value && set_page_bits(info, bitmap, block->page, bits)) goto err; } @@ -2286,13 +2356,19 @@ my_bool _ma_bitmap_release_unused(MARIA_HA *info, MARIA_BITMAP_BLOCKS *blocks) goto err; } - if (--bitmap->non_flushable == 0) + if (info->s->now_transactional) { - _ma_bitmap_unpin_all(info->s); - if (unlikely(bitmap->flush_all_requested)) + DBUG_ASSERT((int) bitmap->non_flushable >= 0 && + info->non_flushable_state); + info->non_flushable_state= 0; + if (--bitmap->non_flushable == 0) { - DBUG_PRINT("info", ("bitmap flushable waking up flusher")); - pthread_cond_broadcast(&bitmap->bitmap_cond); + _ma_bitmap_unpin_all(info->s); + if (unlikely(bitmap->flush_all_requested)) + { + DBUG_PRINT("info", ("bitmap flushable waking up flusher")); + pthread_cond_broadcast(&bitmap->bitmap_cond); + } } } DBUG_PRINT("info", ("bitmap->non_flushable: %u", bitmap->non_flushable)); @@ -2327,28 +2403,29 @@ err: my_bool _ma_bitmap_free_full_pages(MARIA_HA *info, const uchar *extents, uint count) { + MARIA_FILE_BITMAP *bitmap= &info->s->bitmap; DBUG_ENTER("_ma_bitmap_free_full_pages"); - pthread_mutex_lock(&info->s->bitmap.bitmap_lock); + pthread_mutex_lock(&bitmap->bitmap_lock); for (; count--; extents+= ROW_EXTENT_SIZE) { ulonglong page= uint5korr(extents); - uint page_count= uint2korr(extents + ROW_EXTENT_PAGE_SIZE); + uint page_count= (uint2korr(extents + ROW_EXTENT_PAGE_SIZE) & + ~START_EXTENT_BIT); if (!(page_count & TAIL_BIT)) { if (page == 0 && page_count == 0) continue; /* Not used extent */ if (pagecache_delete_pages(info->s->pagecache, &info->dfile, page, page_count, PAGECACHE_LOCK_WRITE, 1) || - _ma_bitmap_reset_full_page_bits(info, &info->s->bitmap, page, - page_count)) + _ma_bitmap_reset_full_page_bits(info, bitmap, page, page_count)) { - pthread_mutex_unlock(&info->s->bitmap.bitmap_lock); + pthread_mutex_unlock(&bitmap->bitmap_lock); DBUG_RETURN(1); } } } - pthread_mutex_unlock(&info->s->bitmap.bitmap_lock); + pthread_mutex_unlock(&bitmap->bitmap_lock); DBUG_RETURN(0); } @@ -2429,8 +2506,8 @@ my_bool _ma_check_bitmap_data(MARIA_HA *info, bits= 0; /* to satisfy compiler */ DBUG_ASSERT(0); } - return (*bitmap_pattern= get_page_bits(info, &info->s->bitmap, page)) != - bits; + return ((*bitmap_pattern= _ma_bitmap_get_page_bits(info, &info->s->bitmap, + page)) != bits); } @@ -2458,7 +2535,8 @@ my_bool _ma_check_if_right_bitmap_type(MARIA_HA *info, ulonglong page, uint *bitmap_pattern) { - if ((*bitmap_pattern= get_page_bits(info, &info->s->bitmap, page)) > 7) + if ((*bitmap_pattern= _ma_bitmap_get_page_bits(info, &info->s->bitmap, + page)) > 7) return 1; /* Couldn't read page */ switch (page_type) { case HEAD_PAGE: diff --git a/storage/maria/ma_blockrec.c b/storage/maria/ma_blockrec.c index c31238368de..6615aa660ef 100644 --- a/storage/maria/ma_blockrec.c +++ b/storage/maria/ma_blockrec.c @@ -220,8 +220,10 @@ A ROW EXTENT is range of pages. One ROW_EXTENT is coded as: START_PAGE 5 bytes - PAGE_COUNT 2 bytes. High bit is used to indicate tail page/ - end of blob + PAGE_COUNT 2 bytes. Bit 16 is set if this is a tail page. + Bit 15 is to set if this is start of a new + blob extent. + With 8K pages, we can cover 256M in one extent. This coding gives us a maximum file size of 2^40*8192 = 8192 tera @@ -456,36 +458,37 @@ my_bool _ma_once_end_block_record(MARIA_SHARE *share) my_bool _ma_init_block_record(MARIA_HA *info) { MARIA_ROW *row= &info->cur_row, *new_row= &info->new_row; + MARIA_SHARE *share= info->s; uint default_extents; DBUG_ENTER("_ma_init_block_record"); if (!my_multi_malloc(MY_WME, - &row->empty_bits, info->s->base.pack_bytes, + &row->empty_bits, share->base.pack_bytes, &row->field_lengths, - info->s->base.max_field_lengths + 2, - &row->blob_lengths, sizeof(ulong) * info->s->base.blobs, + share->base.max_field_lengths + 2, + &row->blob_lengths, sizeof(ulong) * share->base.blobs, &row->null_field_lengths, (sizeof(uint) * - (info->s->base.fields - - info->s->base.blobs + + (share->base.fields - + share->base.blobs + EXTRA_LENGTH_FIELDS)), &row->tail_positions, (sizeof(MARIA_RECORD_POS) * - (info->s->base.blobs + 2)), - &new_row->empty_bits, info->s->base.pack_bytes, + (share->base.blobs + 2)), + &new_row->empty_bits, share->base.pack_bytes, &new_row->field_lengths, - info->s->base.max_field_lengths + 2, + share->base.max_field_lengths + 2, &new_row->blob_lengths, - sizeof(ulong) * info->s->base.blobs, + sizeof(ulong) * share->base.blobs, &new_row->null_field_lengths, (sizeof(uint) * - (info->s->base.fields - - info->s->base.blobs + + (share->base.fields - + share->base.blobs + EXTRA_LENGTH_FIELDS)), &info->log_row_parts, sizeof(*info->log_row_parts) * - (TRANSLOG_INTERNAL_PARTS + 2 + - info->s->base.fields + 3), + (TRANSLOG_INTERNAL_PARTS + 3 + + share->base.fields + 3), &info->update_field_data, - (info->s->base.fields * 4 + - info->s->base.max_field_lengths + 1 + 4), + (share->base.fields * 4 + + share->base.max_field_lengths + 1 + 4), NullS, 0)) DBUG_RETURN(1); /* Skip over bytes used to store length of field length for logging */ @@ -495,7 +498,7 @@ my_bool _ma_init_block_record(MARIA_HA *info) /* Reserve some initial space to avoid mallocs during execution */ default_extents= (ELEMENTS_RESERVED_FOR_MAIN_PART + 1 + (AVERAGE_BLOB_SIZE / - FULL_PAGE_SIZE(info->s->block_size) / + FULL_PAGE_SIZE(share->block_size) / BLOB_SEGMENT_MIN_SIZE)); if (my_init_dynamic_array(&info->bitmap_blocks, @@ -507,7 +510,7 @@ my_bool _ma_init_block_record(MARIA_HA *info) MYF(MY_WME)))) goto err; - row->base_length= new_row->base_length= info->s->base_length; + row->base_length= new_row->base_length= share->base_length; /* We need to reserve 'EXTRA_LENGTH_FIELDS' number of parts in @@ -599,6 +602,86 @@ static inline uint end_of_previous_entry(uchar *dir, uchar *end) } +#ifndef DBUG_OFF + +static void _ma_print_directory(uchar *buff, uint block_size) +{ + uint max_entry= (uint) ((uchar *) buff)[DIR_COUNT_OFFSET], row= 0; + uint end_of_prev_row= PAGE_HEADER_SIZE; + uchar *dir, *end; + + dir= dir_entry_pos(buff, block_size, max_entry-1); + end= dir_entry_pos(buff, block_size, 0); + + DBUG_LOCK_FILE; + fprintf(DBUG_FILE,"Directory dump (pos:length):\n"); + + for (row= 1; dir <= end ; end-= DIR_ENTRY_SIZE, row++) + { + uint offset= uint2korr(end); + uint length= uint2korr(end+2); + fprintf(DBUG_FILE, " %4u:%4u", offset, offset ? length : 0); + if (!(row % (80/12))) + fputc('\n', DBUG_FILE); + if (offset) + { + DBUG_ASSERT(offset >= end_of_prev_row); + end_of_prev_row= offset + length; + } + } + fputc('\n', DBUG_FILE); + fflush(DBUG_FILE); + DBUG_UNLOCK_FILE; +} + + +static void check_directory(uchar *buff, uint block_size) +{ + uchar *dir, *end; + uint max_entry= (uint) buff[DIR_COUNT_OFFSET]; + uint start_of_dir, deleted; + uchar free_entry, prev_free_entry; + uint end_of_prev_row= PAGE_HEADER_SIZE; + + dir= dir_entry_pos(buff, block_size, max_entry-1); + start_of_dir= (uint) (dir - buff); + end= dir_entry_pos(buff, block_size, 0); + deleted= 0; + + /* Ensure that all rows are in increasing order and no overlaps */ + for (; dir <= end ; end-= DIR_ENTRY_SIZE) + { + uint offset= uint2korr(end); + uint length= uint2korr(end+2); + if (offset) + { + DBUG_ASSERT(offset >= end_of_prev_row); + end_of_prev_row= offset + length; + } + else + deleted++; + } + DBUG_ASSERT(end_of_prev_row <= start_of_dir); + + /* check free links */ + free_entry= buff[DIR_FREE_OFFSET]; + prev_free_entry= END_OF_DIR_FREE_LIST; + while (free_entry != END_OF_DIR_FREE_LIST) + { + uchar *dir= dir_entry_pos(buff, block_size, free_entry); + DBUG_ASSERT(dir[0] == 0 && dir[1] == 0); + DBUG_ASSERT(dir[2] == prev_free_entry); + prev_free_entry= free_entry; + free_entry= dir[3]; + deleted--; + } + DBUG_ASSERT(deleted == 0); +} +#else +#define check_directory(A,B) +#endif /* DBUG_OFF */ + + /** @brief Extend a record area to fit a given size block @@ -609,21 +692,28 @@ static inline uint end_of_previous_entry(uchar *dir, uchar *end) @param block_size Block size of buffer @param request_length How much data we want to put at [dir] @param empty_space Total empty space in buffer + This is updated with length after dir + is allocated and current block freed - IMPLEMENTATION + @implementation The logic is as follows (same as in _ma_update_block_record()) - If new data fits in old block, use old block. - Extend block with empty space before block. If enough, use it. - Extend block with empty space after block. If enough, use it. - Use compact_page() to get all empty space at dir. - RETURN - @retval 0 ok - @retval ret_offset Pointer to store offset to found area - @retval ret_length Pointer to store length of found area - @retval [dir] rec_offset is store here too + @note + The given directory entry is set to rec length. + empty_space doesn't include the new directory entry + + + @return + @retval 0 ok + @retval ret_offset Pointer to store offset to found area + @retval ret_length Pointer to store length of found area + @retval [dir] rec_offset is store here too - @retval 1 error (wrong info in block) + @retval 1 error (wrong info in block) */ static my_bool extend_area_on_page(uchar *buff, uchar *dir, @@ -678,11 +768,12 @@ static my_bool extend_area_on_page(uchar *buff, uchar *dir, rec_offset= end_of_previous_entry(dir, buff + block_size - PAGE_SUFFIX_SIZE); length+= (uint) (old_rec_offset - rec_offset); + DBUG_ASSERT(old_rec_offset); /* - old_rec_offset is 0 if we are doing an insert into a not allocated block. - This can only happen during REDO of INSERT + 'length' is 0 if we are doing an insert into a not allocated block. + This can only happen during "REDO of INSERT" or "UNDO of DELETE." */ - if (!old_rec_offset || length < request_length) + if (length < request_length) { /* Did not fit in current block + empty space. Extend with @@ -696,11 +787,13 @@ static my_bool extend_area_on_page(uchar *buff, uchar *dir, } else length= start_of_next_entry(dir) - rec_offset; - DBUG_ASSERT((int) length > 0); + DBUG_ASSERT((int) length >= 0); if (length < request_length) { - /* Not enough continues space, compact page to get more */ + /* Not enough continuous space, compact page to get more */ int2store(dir, rec_offset); + /* Reset length, as this may be a deleted block */ + int2store(dir+2, 0); compact_page(buff, block_size, rownr, 1); rec_offset= uint2korr(dir); length= uint2korr(dir+2); @@ -717,8 +810,10 @@ static my_bool extend_area_on_page(uchar *buff, uchar *dir, } } int2store(dir, rec_offset); + int2store(dir + 2, length); *ret_offset= rec_offset; *ret_length= length; + check_directory(buff, block_size); DBUG_RETURN(0); } @@ -800,6 +895,73 @@ static uint empty_space_on_page(uchar *buff, uint block_size) } #endif + +/* + @brief Ensure we have space for new directory entries + + @fn make_space_for_directory() + @param buff Page buffer + @param block_size Block size for pages + @param max_entry Number of current entries in directory + @param count Number of new entries to be added to directory + @param first_dir First directory entry on page + @param empty_space Total empty space in buffer. It's updated + to reflect the new empty space + @param first_pos Store position to last data byte on page here + + @note + This function is inline as the argument passing is the biggest + part of the function + + @return + @retval 0 ok + @retval 1 error (No data on page, fatal error) +*/ + +static inline my_bool +make_space_for_directory(uchar *buff, uint block_size, uint max_entry, + uint count, uchar *first_dir, uint *empty_space, + uint *first_pos) +{ + uint length_needed= DIR_ENTRY_SIZE * count; + + /* + The following is not true only in the case and UNDO is used to reinsert + a row on a previously not used page + */ + if (likely(max_entry)) + { + /* Check if there is place for the directory entry on the page */ + *first_pos= uint2korr(first_dir) + uint2korr(first_dir + 2); + + if ((uint) (first_dir - buff) < *first_pos + length_needed) + { + /* Create place for directory */ + compact_page(buff, block_size, max_entry - 1, 0); + *first_pos= (uint2korr(first_dir) + uint2korr(first_dir + 2)); + *empty_space= uint2korr(buff + EMPTY_SPACE_OFFSET); + if (*empty_space < length_needed) + { + /* + We should always have space, as we only come here for + UNDO of DELETE (in which case we know the row was on the + page before) or if the bitmap told us there was space on page + */ + DBUG_ASSERT(0); + return(1); + } + } + } + else + *first_pos= PAGE_HEADER_SIZE; + + /* Reduce directory entry size from free space size */ + (*empty_space)-= length_needed; + buff[DIR_COUNT_OFFSET]= (uchar) (max_entry + count); + return(0); +} + + /* Find free position in directory @@ -855,7 +1017,7 @@ static uchar *find_free_position(uchar *buff, uint block_size, uint *res_rownr, if (free_entry != END_OF_DIR_FREE_LIST) { if (free_entry >= max_entry) - DBUG_RETURN(0); + DBUG_RETURN(0); /* Consistency error */ dir= dir_entry_pos(buff, block_size, free_entry); DBUG_ASSERT(uint2korr(dir) == 0 && dir[2] == END_OF_DIR_FREE_LIST); /* Relink free list */ @@ -874,39 +1036,108 @@ static uchar *find_free_position(uchar *buff, uint block_size, uint *res_rownr, int2store(dir + 2, length); *res_rownr= free_entry; *res_length= length; + + check_directory(buff, block_size); DBUG_RETURN(dir); } /* No free places in dir; create a new one */ - /* Check if there is place for the directory entry */ if (max_entry == MAX_ROWS_PER_PAGE) DBUG_RETURN(0); + + if (make_space_for_directory(buff, block_size, max_entry, 1, + first_dir, empty_space, &first_pos)) + DBUG_RETURN(0); + dir= first_dir - DIR_ENTRY_SIZE; - /* Last used place on page */ - first_pos= uint2korr(first_dir) + uint2korr(first_dir + 2); - /* Check if there is place for the directory entry on the page */ - if ((uint) (dir - buff) < first_pos) - { - /* Create place for directory */ - compact_page(buff, block_size, max_entry-1, 0); - first_pos= (uint2korr(first_dir) + uint2korr(first_dir + 2)); - *empty_space= uint2korr(buff + EMPTY_SPACE_OFFSET); - DBUG_ASSERT(*empty_space > DIR_ENTRY_SIZE); - } - buff[DIR_COUNT_OFFSET]= (uchar) max_entry+1; length= (uint) (dir - buff - first_pos); - DBUG_ASSERT(length <= *empty_space - DIR_ENTRY_SIZE); + DBUG_ASSERT(length <= *empty_space); int2store(dir, first_pos); int2store(dir+2, length); /* Max length of region */ *res_rownr= max_entry; *res_length= length; - /* Reduce directory entry size from free space size */ - (*empty_space)-= DIR_ENTRY_SIZE; DBUG_RETURN(dir); } +/** + @brief Enlarge page directory to hold more entries + + @fn extend_directory() + @param buff Page buffer + @param block_size Block size + @param max_entry Number of directory entries on page + @param new_entry Position for new entry + @param empty_space Total empty space in buffer. It's updated + to reflect the new empty space + + @note + This is only called on UNDO when we want to expand the directory + to be able to re-insert row in a given position + + The new directory entry will be set to cover the maximum possible space + + @return + @retval 0 ok + @retval 1 error (No data on page, fatal error) +*/ + +static my_bool extend_directory(uchar *buff, uint block_size, uint max_entry, + uint new_entry, uint *empty_space) +{ + uint length, first_pos; + uchar *dir, *first_dir; + DBUG_ENTER("extend_directory"); + + /* + Note that in if max_entry is 0, then first_dir will point to + an illegal directory entry. This is ok, as in this case we will + not access anything through first_dir. + */ + first_dir= dir_entry_pos(buff, block_size, max_entry) + DIR_ENTRY_SIZE; + + if (make_space_for_directory(buff, block_size, max_entry, + new_entry - max_entry + 1, + first_dir, empty_space, &first_pos)) + DBUG_RETURN(1); + + /* Set the new directory entry to cover the max possible length */ + dir= first_dir - DIR_ENTRY_SIZE * (new_entry - max_entry + 1); + length= (uint) (dir - buff - first_pos); + int2store(dir, first_pos); + int2store(dir+2, length); + *empty_space-= length; + + if (new_entry-- > max_entry) + { + /* Link all row entries between new_entry and max_entry into free list */ + uint free_entry= (uint) buff[DIR_FREE_OFFSET]; + uint prev_entry= END_OF_DIR_FREE_LIST; + buff[DIR_FREE_OFFSET]= new_entry; + do + { + dir+= DIR_ENTRY_SIZE; + dir[0]= dir[1]= 0; + dir[2]= (uchar) prev_entry; + dir[3]= (uchar) new_entry-1; + prev_entry= new_entry; + } while (new_entry-- > max_entry); + if ((dir[3]= free_entry) != END_OF_DIR_FREE_LIST) + { + /* Relink next entry to point to newly freed entry */ + uchar *next_entry= dir_entry_pos(buff, block_size, (uint) dir[3]); + DBUG_ASSERT(uint2korr(next_entry) == 0 && + next_entry[2] == END_OF_DIR_FREE_LIST); + next_entry[2]= max_entry; + } + } + + check_directory(buff, block_size); + DBUG_RETURN(0); +} + + /**************************************************************************** Updating records ****************************************************************************/ @@ -1054,10 +1285,11 @@ static void calc_record_size(MARIA_HA *info, const uchar *record, } } row->field_lengths_length= (uint) (field_length_data - row->field_lengths); - row->head_length= (row->base_length + + row->min_length= (row->base_length + + size_to_store_key_length(row->field_lengths_length)); + row->head_length= (row->min_length + share->base.fixed_not_null_fields_length + row->field_lengths_length + - size_to_store_key_length(row->field_lengths_length) + row->normal_length + row->char_length + row->varchar_length); row->total_length= (row->head_length + row->blob_length); @@ -1092,7 +1324,7 @@ static void calc_record_size(MARIA_HA *info, const uchar *record, static void compact_page(uchar *buff, uint block_size, uint rownr, my_bool extend_block) { - uint max_entry= (uint) ((uchar *) buff)[DIR_COUNT_OFFSET]; + uint max_entry= (uint) buff[DIR_COUNT_OFFSET]; uint page_pos, next_free_pos, start_of_found_block, diff, end_of_found_block; uchar *dir, *end; DBUG_ENTER("compact_page"); @@ -1214,15 +1446,17 @@ static void compact_page(uchar *buff, uint block_size, uint rownr, SYNOPSIS make_empty_page() - buff Page buffer - block_size Block size - page_type HEAD_PAGE or TAIL_PAGE + buff Page buffer + block_size Block size + page_type HEAD_PAGE or TAIL_PAGE + create_dir_entry TRUE of we should create a directory entry NOTES EMPTY_SPACE is not updated */ -static void make_empty_page(MARIA_HA *info, uchar *buff, uint page_type) +static void make_empty_page(MARIA_HA *info, uchar *buff, uint page_type, + my_bool create_dir_entry) { uint block_size= info->s->block_size; DBUG_ENTER("make_empty_page"); @@ -1239,11 +1473,15 @@ static void make_empty_page(MARIA_HA *info, uchar *buff, uint page_type) bzero(buff+ PAGE_HEADER_SIZE, block_size - PAGE_HEADER_SIZE); #endif buff[PAGE_TYPE_OFFSET]= (uchar) page_type; - buff[DIR_COUNT_OFFSET]= 1; + buff[DIR_COUNT_OFFSET]= (int) create_dir_entry; buff[DIR_FREE_OFFSET]= END_OF_DIR_FREE_LIST; - int2store(buff + block_size - PAGE_SUFFIX_SIZE - DIR_ENTRY_SIZE, - PAGE_HEADER_SIZE); - + if (create_dir_entry) + { + /* Create directory entry to point to start of page with size 0 */ + buff+= block_size - PAGE_SUFFIX_SIZE - DIR_ENTRY_SIZE; + int2store(buff, PAGE_HEADER_SIZE); + int2store(buff+2, 0); + } DBUG_VOID_RETURN; } @@ -1264,6 +1502,8 @@ static void make_empty_page(MARIA_HA *info, uchar *buff, uint page_type) We don't decremented buff[EMPTY_SPACE_OFFSET] with the allocated data as we don't know how much data the caller will actually use. + res->empty_space is set to length of empty space + RETURN 0 ok All slots in 'res' are updated 1 error my_errno is set @@ -1296,7 +1536,7 @@ static my_bool get_head_or_tail_page(MARIA_HA *info, if (block->org_bitmap_value == 0) /* Empty block */ { /* New page */ - make_empty_page(info, buff, page_type); + make_empty_page(info, buff, page_type, 1); res->buff= buff; res->empty_space= res->length= (block_size - PAGE_OVERHEAD_SIZE); res->data= (buff + PAGE_HEADER_SIZE); @@ -1308,16 +1548,14 @@ static my_bool get_head_or_tail_page(MARIA_HA *info, { uchar *dir; /* Read old page */ - DBUG_ASSERT(share->pagecache->block_size == block_size); - if (!(res->buff= pagecache_read(share->pagecache, - &info->dfile, - (my_off_t) block->page, 0, - buff, share->page_type, - lock, &page_link.link))) - DBUG_RETURN(1); page_link.unlock= PAGECACHE_LOCK_WRITE_UNLOCK; - page_link.changed= 1; + res->buff= pagecache_read(share->pagecache, &info->dfile, + block->page, 0, buff, share->page_type, + lock, &page_link.link); + page_link.changed= res->buff == 0; push_dynamic(&info->pinned_pages, (void*) &page_link); + if (page_link.changed) + goto crashed; DBUG_ASSERT((res->buff[PAGE_TYPE_OFFSET] & PAGE_TYPE_MASK) == page_type); if (!(dir= find_free_position(res->buff, block_size, &res->rownr, @@ -1352,6 +1590,103 @@ crashed: /* + @brief Create room for a head or tail row on a given page at given position + + @fn get_rowpos_in_head_or_tail_page() + @param info Maria handler + @param block Block to read + @param buff Suggest this buffer to key cache + @param length Minimum space needed + @param page_type HEAD_PAGE || TAIL_PAGE + @param rownr Rownr to use + @param res Store result position here + + @note + This is essential same as get_head_or_tail_page, with the difference + that the caller species at what position the row should be put. + This is used when restoring a row to it's original position as + part of UNDO DELETE or UNDO UPDATE + + @return + @retval 0 ok All slots in 'res' are updated + @retval 1 error my_errno is set +*/ + +static my_bool get_rowpos_in_head_or_tail_page(MARIA_HA *info, + MARIA_BITMAP_BLOCK *block, + uchar *buff, uint length, + uint page_type, + enum pagecache_page_lock lock, + uint rownr, + struct st_row_pos_info *res) +{ + MARIA_PINNED_PAGE page_link; + MARIA_SHARE *share= info->s; + uchar *dir; + uint block_size= share->block_size; + uint max_entry, max_length, rec_offset; + DBUG_ENTER("get_rowpos_in_head_or_tail_page"); + + if (block->org_bitmap_value == 0) /* Empty block */ + { + /* New page */ + make_empty_page(info, buff, page_type, 0); + res->empty_space= block_size - PAGE_HEADER_SIZE - PAGE_SUFFIX_SIZE; + } + else + { + page_link.unlock= PAGECACHE_LOCK_WRITE_UNLOCK; + res->buff= pagecache_read(share->pagecache, &info->dfile, + block->page, 0, buff, share->page_type, + lock, &page_link.link); + page_link.changed= res->buff != 0; + push_dynamic(&info->pinned_pages, (void*) &page_link); + if (!page_link.changed) /* Read error */ + goto err; + DBUG_ASSERT((buff[PAGE_TYPE_OFFSET] & PAGE_TYPE_MASK) == + (uchar) page_type); + if ((buff[PAGE_TYPE_OFFSET] & PAGE_TYPE_MASK) != (uchar) page_type) + goto err; + res->empty_space= uint2korr(buff + EMPTY_SPACE_OFFSET); + } + + max_entry= (uint) buff[DIR_COUNT_OFFSET]; + if (max_entry <= rownr) + { + if (extend_directory(buff, block_size, max_entry, rownr, + &res->empty_space)) + goto err; + } + + dir= dir_entry_pos(buff, block_size, rownr); +#ifdef SANITY_CHECKS + /* Tail's should always be unused */ + if (page_type == TAIL_PAGE && max_entry > rownr && + (dir[0] != 0 || dir[1] != 0)) + { + DBUG_ASSERT(0); + goto err; + } +#endif + + if (extend_area_on_page(buff, dir, rownr, block_size, length, + &res->empty_space, &rec_offset, &max_length)) + goto err; + + res->buff= buff; + res->rownr= rownr; + res->dir= dir; + res->data= buff + rec_offset; + res->length= length; + DBUG_RETURN(0); + +err: + my_errno= HA_ERR_WRONG_IN_RECORD; /* File crashed */ + DBUG_RETURN(1); +} + + +/* Write tail for head data or blob SYNOPSIS @@ -1374,11 +1709,11 @@ crashed: static my_bool write_tail(MARIA_HA *info, MARIA_BITMAP_BLOCK *block, - uchar *row_part, uint length) + uchar *row_part, uint org_length) { MARIA_SHARE *share= info->s; MARIA_PINNED_PAGE page_link; - uint block_size= share->block_size, empty_space; + uint block_size= share->block_size, empty_space, length= org_length; struct st_row_pos_info row_pos; my_off_t position; my_bool res, block_is_read; @@ -1387,15 +1722,41 @@ static my_bool write_tail(MARIA_HA *info, (ulong) block->page, length)); info->keyread_buff_used= 1; + /* + Don't allocate smaller block than MIN_TAIL_SIZE (we want to give rows + some place to grow in the future) + */ + if (length < MIN_TAIL_SIZE) + length= MIN_TAIL_SIZE; + + if (block->page_count == TAIL_PAGE_COUNT_MARKER) + { + /* + Create new tail + page will be pinned & locked by get_head_or_tail_page + */ + if (get_head_or_tail_page(info, block, info->keyread_buff, length, + TAIL_PAGE, PAGECACHE_LOCK_WRITE, + &row_pos)) + DBUG_RETURN(1); + } + else + { + /* Write tail on predefined row position */ + if (get_rowpos_in_head_or_tail_page(info, block, info->keyread_buff, + length, TAIL_PAGE, + PAGECACHE_LOCK_WRITE, + block->page_count & ~TAIL_BIT, + &row_pos)) + DBUG_RETURN(1); + } + DBUG_PRINT("info", ("tailid: %lu (%lu:%u)", + (ulong) ma_recordpos(block->page, row_pos.rownr), + (ulong) block->page, row_pos.rownr)); - /* page will be pinned & locked by get_head_or_tail_page */ - if (get_head_or_tail_page(info, block, info->keyread_buff, length, - TAIL_PAGE, PAGECACHE_LOCK_WRITE, - &row_pos)) - DBUG_RETURN(1); block_is_read= block->org_bitmap_value != 0; - memcpy(row_pos.data, row_part, length); + memcpy(row_pos.data, row_part, org_length); if (share->now_transactional) { @@ -1404,7 +1765,12 @@ static my_bool write_tail(MARIA_HA *info, LEX_STRING log_array[TRANSLOG_INTERNAL_PARTS + 2]; LSN lsn; - /* Log REDO changes of tail page */ + /* + Log REDO changes of tail page + Note that we have to log length, not org_length, to be sure that + REDO, which doesn't use write_tail, also creates a block of at least + MIN_TAIL_SIZE + */ page_store(log_data + FILEID_STORE_SIZE, block->page); dirpos_store(log_data + FILEID_STORE_SIZE + PAGE_STORE_SIZE, row_pos.rownr); @@ -1419,12 +1785,6 @@ static my_bool write_tail(MARIA_HA *info, DBUG_RETURN(1); } - /* - Don't allocate smaller block than MIN_TAIL_SIZE (we want to give rows - some place to grow in the future) - */ - if (length < MIN_TAIL_SIZE) - length= MIN_TAIL_SIZE; int2store(row_pos.dir + 2, length); empty_space= row_pos.empty_space - length; int2store(row_pos.buff + EMPTY_SPACE_OFFSET, empty_space); @@ -1509,7 +1869,7 @@ static my_bool write_full_pages(MARIA_HA *info, uint block_size= share->block_size; uint data_size= FULL_PAGE_SIZE(block_size); uchar *buff= info->keyread_buff; - uint page_count; + uint page_count, sub_blocks; my_off_t position; DBUG_ENTER("write_full_pages"); DBUG_PRINT("enter", ("length: %lu page: %lu page_count: %lu", @@ -1520,6 +1880,7 @@ static my_bool write_full_pages(MARIA_HA *info, info->keyread_buff_used= 1; page= block->page; page_count= block->page_count; + sub_blocks= block->sub_blocks; position= (my_off_t) (page + page_count) * block_size; if (info->state->data_file_length < position) @@ -1532,6 +1893,13 @@ static my_bool write_full_pages(MARIA_HA *info, uint copy_length; if (!page_count--) { + if (!--sub_blocks) + { + DBUG_ASSERT(0); /* Wrong in bitmap or UNDO */ + my_errno= HA_ERR_WRONG_IN_RECORD; /* File crashed */ + DBUG_RETURN(1); + } + block++; page= block->page; page_count= block->page_count - 1; @@ -1650,6 +2018,11 @@ static uchar *store_page_range(uchar *to, MARIA_BITMAP_BLOCK *block, NOTES We don't have to store the position for the head block + + We have to set the START_EXTENT_BIT for every extent where the + blob will be stored on a page of it's own. We need this in the + UNDO phase to generate MARIA_BITMAP_BLOCK's for undo-delete and + undo-update. */ static void store_extent_info(uchar *to, @@ -1667,9 +2040,18 @@ static void store_extent_info(uchar *to, /* The following is only false for marker blocks */ if (likely(block->used & BLOCKUSED_USED)) { - DBUG_ASSERT(block->page_count != 0); + uint page_count= block->page_count; + DBUG_ASSERT(page_count != 0); page_store(to, block->page); - pagerange_store(to + PAGE_STORE_SIZE, block->page_count); + if (block->sub_blocks) + { + /* + Set a bit so that we later know that this was the first block + for a blob + */ + page_count|= START_EXTENT_BIT; + } + pagerange_store(to + PAGE_STORE_SIZE, page_count); to+= ROW_EXTENT_SIZE; if (!first_found) { @@ -1687,6 +2069,88 @@ static void store_extent_info(uchar *to, } +/** + @brief + Convert extent info read from file to MARIA_BITMAP_BLOCKS suitable + for write_block_record + + @note + In case of blobs, this function marks all the blob pages in the bitmap + as full pages. The bitmap bits for other pages will be marked + when write_block_record() calls _ma_bitmap_release_unused(). + + This function will be removed in Maria 2.0 when we instead of delete rows + mark them as deleted and only remove them after commit. + + @return + @retval 0 ok + @retval 1 Error (out of memory or disk error changing bitmap) +*/ + +static my_bool extent_to_bitmap_blocks(MARIA_HA *info, + MARIA_BITMAP_BLOCKS *blocks, + ulonglong head_page, + uint extent_count, + const uchar *extent_info) +{ + MARIA_BITMAP_BLOCK *block, *start_block; + MARIA_SHARE *share= info->s; + uint i; + DBUG_ENTER("extent_to_bitmap_blocks"); + + if (allocate_dynamic(&info->bitmap_blocks, extent_count + 2)) + DBUG_RETURN(1); + block= blocks->block= dynamic_element(&info->bitmap_blocks, 0, + MARIA_BITMAP_BLOCK*); + blocks->count= extent_count + 1; + blocks->tail_page_skipped= blocks->page_skipped= 0; + block->page= head_page; + block->page_count= 1; + block->used= BLOCKUSED_USED | BLOCKUSED_USE_ORG_BITMAP; + /* Impossible value, will force storage of real value */ + block->org_bitmap_value= 255; + + start_block= block++; + for (i=0 ; + i++ < extent_count ; + block++, extent_info+= ROW_EXTENT_SIZE) + { + uint page_count= uint2korr(extent_info + ROW_EXTENT_PAGE_SIZE); + if (page_count & START_EXTENT_BIT) + { + page_count&= ~START_EXTENT_BIT; + start_block->sub_blocks= (uint) (block - start_block); + start_block= block; + + } + block->page= page_korr(extent_info); + block->page_count= page_count; + block->sub_blocks= 0; + + if (page_count & TAIL_BIT) + { + block->org_bitmap_value= _ma_bitmap_get_page_bits(info, &share->bitmap, + block->page); + block->used= (BLOCKUSED_TAIL | BLOCKUSED_USED | + BLOCKUSED_USE_ORG_BITMAP); + } + else + { + my_bool res; + pthread_mutex_lock(&share->bitmap.bitmap_lock); + res= _ma_bitmap_set_full_page_bits(info, &share->bitmap, + block->page, block->page_count); + pthread_mutex_unlock(&share->bitmap.bitmap_lock); + if (res) + DBUG_RETURN(1); + block->used= BLOCKUSED_USED; + } + } + start_block->sub_blocks= (uint) (block - start_block); + DBUG_RETURN(0); +} + + /* Free regions of pages with logging @@ -1711,15 +2175,22 @@ static my_bool free_full_pages(MARIA_HA *info, MARIA_ROW *row) if (info->s->now_transactional) { /* Compact events by removing filler and tail events */ - uchar *start= extents; uchar *new_block= 0; - uchar *end; + uchar *end, *to, *compact_extent_info; + my_bool res; + uint extents_count; + + if (!(compact_extent_info= my_alloca(row->extents_count * + ROW_EXTENT_SIZE))) + DBUG_RETURN(1); + to= compact_extent_info; for (end= extents + row->extents_count * ROW_EXTENT_SIZE ; extents < end ; extents+= ROW_EXTENT_SIZE) { uint page_count= uint2korr(extents + ROW_EXTENT_PAGE_SIZE); + page_count&= ~START_EXTENT_BIT; if (! (page_count & TAIL_BIT) && page_count != 0) { /* Found correct extent */ @@ -1727,53 +2198,43 @@ static my_bool free_full_pages(MARIA_HA *info, MARIA_ROW *row) new_block= extents; /* First extent in range */ continue; } - /* Found extent to remove, move everything found up */ + /* Found extent to remove, copy everything found so far */ if (new_block) { - if (new_block == start) - start= extents; - else - { - size_t length= (size_t) (extents - new_block); - memmove(start, new_block, length); - start+= length; - } + size_t length= (size_t) (extents - new_block); + memcpy(to, new_block, length); + to+= length; + new_block= 0; } - new_block= 0; } if (new_block) { - if (new_block == start) - start= extents; /* Nothing to delete */ - else - { - /* Move rest down */ - size_t length= (size_t) (extents - new_block); - memmove(start, new_block, length); - start+= length; - } + size_t length= (size_t) (extents - new_block); + memcpy(to, new_block, length); + to+= length; } - if (!unlikely(extents_length= (start - row->extents))) + if (!unlikely(extents_length= (uint) (to - compact_extent_info))) { /* No ranges. This happens in the rear case when we have a allocated place for a blob on a tail page but it did fit into the main page. */ + my_afree(compact_extent_info); DBUG_RETURN(0); } - row->extents_count= extents_length / ROW_EXTENT_SIZE; - - pagerange_store(log_data + FILEID_STORE_SIZE, - row->extents_count); + extents_count= extents_length / ROW_EXTENT_SIZE; + pagerange_store(log_data + FILEID_STORE_SIZE, extents_count); log_array[TRANSLOG_INTERNAL_PARTS + 0].str= (char*) log_data; log_array[TRANSLOG_INTERNAL_PARTS + 0].length= sizeof(log_data); - log_array[TRANSLOG_INTERNAL_PARTS + 1].str= row->extents; + log_array[TRANSLOG_INTERNAL_PARTS + 1].str= compact_extent_info; log_array[TRANSLOG_INTERNAL_PARTS + 1].length= extents_length; - if (translog_write_record(&lsn, LOGREC_REDO_FREE_BLOCKS, info->trn, - info, sizeof(log_data) + extents_length, - TRANSLOG_INTERNAL_PARTS + 2, log_array, - log_data, NULL)) + res= translog_write_record(&lsn, LOGREC_REDO_FREE_BLOCKS, info->trn, + info, sizeof(log_data) + extents_length, + TRANSLOG_INTERNAL_PARTS + 2, log_array, + log_data, NULL); + my_afree(compact_extent_info); + if (res) DBUG_RETURN(1); } @@ -1796,13 +2257,14 @@ static my_bool free_full_pages(MARIA_HA *info, MARIA_ROW *row) static my_bool free_full_page_range(MARIA_HA *info, ulonglong page, uint count) { my_bool res= 0; + MARIA_SHARE *share= info->s; DBUG_ENTER("free_full_page_range"); - if (pagecache_delete_pages(info->s->pagecache, &info->dfile, + if (pagecache_delete_pages(share->pagecache, &info->dfile, page, count, PAGECACHE_LOCK_WRITE, 0)) res= 1; - if (info->s->now_transactional) + if (share->now_transactional) { LSN lsn; /** @todo unify log_data's shape with delete_head_or_tail() */ @@ -1824,10 +2286,10 @@ static my_bool free_full_page_range(MARIA_HA *info, ulonglong page, uint count) log_data, NULL)) res= 1; } - pthread_mutex_lock(&info->s->bitmap.bitmap_lock); - if (_ma_bitmap_reset_full_page_bits(info, &info->s->bitmap, page, count)) + pthread_mutex_lock(&share->bitmap.bitmap_lock); + if (_ma_bitmap_reset_full_page_bits(info, &share->bitmap, page, count)) res= 1; - pthread_mutex_unlock(&info->s->bitmap.bitmap_lock); + pthread_mutex_unlock(&share->bitmap.bitmap_lock); DBUG_RETURN(res); } @@ -1853,13 +2315,17 @@ static my_bool free_full_page_range(MARIA_HA *info, ulonglong page, uint count) @note On return all pinned pages are released. + [page_buff + EMPTY_SPACE_OFFSET] is set to + row_pos->empty_space - head_length + @return Operation status - @retval 0 OK - @retval 1 Error + @retval 0 OK + @retval 1 Error */ static my_bool write_block_record(MARIA_HA *info, - const uchar *old_record, const uchar *record, + const uchar *old_record, + const uchar *record, MARIA_ROW *row, MARIA_BITMAP_BLOCKS *bitmap_blocks, my_bool head_block_is_read, @@ -1875,7 +2341,7 @@ static my_bool write_block_record(MARIA_HA *info, MARIA_SHARE *share= info->s; MARIA_COLUMNDEF *column, *end_column; MARIA_PINNED_PAGE page_link; - uint block_size, flag; + uint block_size, flag, head_length; ulong *blob_lengths; my_bool row_extents_in_use, blob_full_pages_exists; LSN lsn; @@ -1899,6 +2365,7 @@ static my_bool write_block_record(MARIA_HA *info, if (unlikely(row->total_length > row_pos->length)) { /* Need extent */ + DBUG_ASSERT(bitmap_blocks->count > 1); if (bitmap_blocks->count <= 1) goto crashed; /* Wrong in bitmap */ flag|= ROW_FLAG_EXTENTS; @@ -2075,29 +2542,27 @@ static my_bool write_block_record(MARIA_HA *info, else data= tmp_data_used; /* Get last used on page */ + /* Update page directory */ + head_length= (uint) (data - row_pos->data); + DBUG_PRINT("info", ("Used head length on page: %u", head_length)); + DBUG_ASSERT(data <= end_of_data); + if (head_length < share->base.min_block_length) { - /* Update page directory */ - uint length= (uint) (data - row_pos->data); - DBUG_PRINT("info", ("Used head length on page: %u", length)); - DBUG_ASSERT(data <= end_of_data); - if (length < info->s->base.min_block_length) - { - /* Extend row to be of size min_block_length */ - uint diff_length= info->s->base.min_block_length - length; - bzero(data, diff_length); - data+= diff_length; - length= info->s->base.min_block_length; - } - int2store(row_pos->dir + 2, length); - /* update empty space at start of block */ - row_pos->empty_space-= length; - int2store(page_buff + EMPTY_SPACE_OFFSET, row_pos->empty_space); - /* Mark in bitmaps how the current page was actually used */ - head_block->empty_space= row_pos->empty_space; - if (page_buff[DIR_COUNT_OFFSET] == MAX_ROWS_PER_PAGE) - head_block->empty_space= 0; /* Page is full */ - head_block->used= BLOCKUSED_USED; + /* Extend row to be of size min_block_length */ + uint diff_length= share->base.min_block_length - head_length; + bzero(data, diff_length); + data+= diff_length; + head_length= share->base.min_block_length; } + int2store(row_pos->dir + 2, head_length); + /* update empty space at start of block */ + row_pos->empty_space-= head_length; + int2store(page_buff + EMPTY_SPACE_OFFSET, row_pos->empty_space); + /* Mark in bitmaps how the current page was actually used */ + head_block->empty_space= row_pos->empty_space; + if (page_buff[DIR_COUNT_OFFSET] == MAX_ROWS_PER_PAGE) + head_block->empty_space= 0; /* Page is full */ + head_block->used|= BLOCKUSED_USED; /* Now we have to write tail pages, as we need to store the position @@ -2166,6 +2631,7 @@ static my_bool write_block_record(MARIA_HA *info, ulong data_length= (tmp_data - info->rec_buff); #ifdef SANITY_CHECKS + DBUG_ASSERT(head_block->sub_blocks != 1); if (head_block->sub_blocks == 1) goto crashed; /* no reserved full or tails */ #endif @@ -2183,10 +2649,10 @@ static my_bool write_block_record(MARIA_HA *info, The reserved pages in bitmap_blocks for the main page has one of the following allocations: - Full pages, with following blocks: - # * full pages - empty page ; To be used if we change last full to tail page. This - has 'count' = 0. - tail page (optional, if last full page was part full) + # * full pages + empty page ; To be used if we change last full to tail page. This + has 'count' = 0. + tail page (optional, if last full page was part full) - One tail page */ @@ -2201,11 +2667,13 @@ static my_bool write_block_record(MARIA_HA *info, cur_block->page_count) { #ifdef SANITY_CHECKS + DBUG_ASSERT(!((cur_block == end_block) || + (cur_block->used & BLOCKUSED_USED))); if ((cur_block == end_block) || (cur_block->used & BLOCKUSED_USED)) goto crashed; #endif data_length-= length; - (cur_block++)->used= BLOCKUSED_USED; + (cur_block++)->used|= BLOCKUSED_USED; } last_head_block= cur_block; if (data_length) @@ -2216,6 +2684,7 @@ static my_bool write_block_record(MARIA_HA *info, cur_block++; } #ifdef SANITY_CHECKS + DBUG_ASSERT(!(cur_block >= end_block)); if ((cur_block >= end_block)) goto crashed; #endif @@ -2223,13 +2692,13 @@ static my_bool write_block_record(MARIA_HA *info, { DBUG_ASSERT(data_length < MAX_TAIL_SIZE(block_size)); /* tail written to full tail page */ - cur_block->used= BLOCKUSED_USED; + cur_block->used|= BLOCKUSED_USED; head_tail_block= cur_block; } else if (data_length > length - MAX_TAIL_SIZE(block_size)) { /* tail written to full page */ - cur_block->used= BLOCKUSED_USED; + cur_block->used|= BLOCKUSED_USED; if ((cur_block != end_block - 1) && (end_block[-1].used & BLOCKUSED_TAIL)) bitmap_blocks->tail_page_skipped= 1; @@ -2249,7 +2718,7 @@ static my_bool write_block_record(MARIA_HA *info, if (cur_block->page_count == 1) { /* convert full block to tail block */ - cur_block->used= BLOCKUSED_USED | BLOCKUSED_TAIL; + cur_block->used|= BLOCKUSED_USED | BLOCKUSED_TAIL; head_tail_block= cur_block; } else @@ -2260,7 +2729,7 @@ static my_bool write_block_record(MARIA_HA *info, cur_block[1].page_count= 1; /* Avoid DBUG_ASSERT */ cur_block[1].used= BLOCKUSED_USED | BLOCKUSED_TAIL; cur_block->page_count--; - cur_block->used= BLOCKUSED_USED; + cur_block->used|= BLOCKUSED_USED; last_head_block= head_tail_block= cur_block+1; } if (end_block[-1].used & BLOCKUSED_TAIL) @@ -2336,7 +2805,6 @@ static my_bool write_block_record(MARIA_HA *info, { uchar log_data[FILEID_STORE_SIZE + PAGE_STORE_SIZE + DIRPOS_STORE_SIZE]; LEX_STRING log_array[TRANSLOG_INTERNAL_PARTS + 2]; - size_t block_length= (size_t) (data - row_pos->data); /* Log REDO changes of head page */ page_store(log_data + FILEID_STORE_SIZE, head_block->page); @@ -2345,9 +2813,9 @@ static my_bool write_block_record(MARIA_HA *info, log_array[TRANSLOG_INTERNAL_PARTS + 0].str= (char*) log_data; log_array[TRANSLOG_INTERNAL_PARTS + 0].length= sizeof(log_data); log_array[TRANSLOG_INTERNAL_PARTS + 1].str= (char*) row_pos->data; - log_array[TRANSLOG_INTERNAL_PARTS + 1].length= block_length; + log_array[TRANSLOG_INTERNAL_PARTS + 1].length= head_length; if (translog_write_record(&lsn, LOGREC_REDO_INSERT_ROW_HEAD, info->trn, - info, sizeof(log_data) + block_length, + info, sizeof(log_data) + head_length, TRANSLOG_INTERNAL_PARTS + 2, log_array, log_data, NULL)) goto disk_err; @@ -2532,7 +3000,8 @@ static my_bool write_block_record(MARIA_HA *info, { uchar log_data[LSN_STORE_SIZE + FILEID_STORE_SIZE + PAGE_STORE_SIZE + DIRPOS_STORE_SIZE + - HA_CHECKSUM_STORE_SIZE]; + HA_CHECKSUM_STORE_SIZE + 2 + PAGERANGE_STORE_SIZE + + ROW_EXTENT_SIZE]; ha_checksum checksum_delta; /* LOGREC_UNDO_ROW_INSERT & LOGREC_UNDO_ROW_UPDATE share same header */ @@ -2545,7 +3014,8 @@ static my_bool write_block_record(MARIA_HA *info, log_array[TRANSLOG_INTERNAL_PARTS + 0].str= (char*) log_data; log_array[TRANSLOG_INTERNAL_PARTS + 0].length= - sizeof(log_data) - HA_CHECKSUM_STORE_SIZE; + (LSN_STORE_SIZE + FILEID_STORE_SIZE + PAGE_STORE_SIZE + + DIRPOS_STORE_SIZE); store_checksum_in_rec(share, checksum_delta, row->checksum - old_record_checksum, log_data + LSN_STORE_SIZE + FILEID_STORE_SIZE + @@ -2569,18 +3039,38 @@ static my_bool write_block_record(MARIA_HA *info, else { /* Write UNDO log record for the UPDATE */ - size_t row_length; + uchar *log_pos= (log_data + + log_array[TRANSLOG_INTERNAL_PARTS + 0].length); + size_t row_length, extents_length; uint row_parts_count; + + /* + Write head length and extents of the original row so that we + during UNDO can put it back in the original position + */ + int2store(log_pos, info->cur_row.head_length); + pagerange_store(log_pos + 2, info->cur_row.extents_count); + log_pos+= 2 + PAGERANGE_STORE_SIZE; + log_array[TRANSLOG_INTERNAL_PARTS + 0].length+= (2 + + PAGERANGE_STORE_SIZE); + info->log_row_parts[TRANSLOG_INTERNAL_PARTS+1].str= + info->cur_row.extents; + info->log_row_parts[TRANSLOG_INTERNAL_PARTS+1].length= + extents_length= info->cur_row.extents_count * ROW_EXTENT_SIZE; + row_length= fill_update_undo_parts(info, old_record, record, log_array + - TRANSLOG_INTERNAL_PARTS + 1, + TRANSLOG_INTERNAL_PARTS + 2, &row_parts_count); if (translog_write_record(&lsn, LOGREC_UNDO_ROW_UPDATE, info->trn, - info, log_array[TRANSLOG_INTERNAL_PARTS + - 0].length + row_length, - TRANSLOG_INTERNAL_PARTS + 1 + - row_parts_count, log_array, - log_data + LSN_STORE_SIZE, &checksum_delta)) + info, + log_array[TRANSLOG_INTERNAL_PARTS + + 0].length + extents_length + + row_length, + TRANSLOG_INTERNAL_PARTS + 2 + + row_parts_count, + log_array, log_data + LSN_STORE_SIZE, + &checksum_delta)) goto disk_err; } } @@ -2597,7 +3087,7 @@ static my_bool write_block_record(MARIA_HA *info, This is the char/varchar data that didn't fit into the head page. */ DBUG_ASSERT(bitmap_blocks->count != 0); - if (write_full_pages(info, info->trn->undo_lsn, head_block + 1, + if (write_full_pages(info, lsn, head_block + 1, info->rec_buff, (ulong) (tmp_data - info->rec_buff))) goto disk_err; } @@ -2618,7 +3108,7 @@ static my_bool write_block_record(MARIA_HA *info, if (block[block->sub_blocks - 1].used & BLOCKUSED_TAIL) blob_length-= (blob_length % FULL_PAGE_SIZE(block_size)); - if (blob_length && write_full_pages(info, info->trn->undo_lsn, block, + if (blob_length && write_full_pages(info, lsn, block, blob_pos, blob_length)) goto disk_err; block+= block->sub_blocks; @@ -2688,7 +3178,7 @@ static my_bool allocate_and_write_block_record(MARIA_HA *info, MARIA_BITMAP_BLOCKS *blocks= &row->insert_blocks; DBUG_ENTER("allocate_and_write_block_record"); - _ma_bitmap_flushable(info->s, 1); + _ma_bitmap_flushable(info, 1); if (_ma_bitmap_find_place(info, row, blocks)) goto err; /* Error reading bitmap */ @@ -2718,14 +3208,15 @@ static my_bool allocate_and_write_block_record(MARIA_HA *info, blocks, blocks->block->org_bitmap_value != 0, &row_pos, undo_lsn, 0)) goto err; /* Error reading bitmap */ - DBUG_PRINT("exit", ("Rowid: %lu (%lu:%u)", (ulong) row->lastpos, + DBUG_PRINT("exit", ("rowid: %lu (%lu:%u)", (ulong) row->lastpos, (ulong) ma_recordpos_to_page(row->lastpos), ma_recordpos_to_dir_entry(row->lastpos))); /* Now let checkpoint happen but don't commit */ DBUG_EXECUTE_IF("maria_over_alloc_bitmap", sleep(1000);); DBUG_RETURN(0); err: - _ma_bitmap_flushable(info->s, -1); + if (info->non_flushable_state) + _ma_bitmap_flushable(info, -1); _ma_unpin_all_pages_and_finalize_row(info, LSN_IMPOSSIBLE); DBUG_RETURN(1); } @@ -2797,7 +3288,7 @@ my_bool _ma_write_abort_block_record(MARIA_HA *info) MARIA_SHARE *share= info->s; DBUG_ENTER("_ma_write_abort_block_record"); - _ma_bitmap_flushable(share, 1); + _ma_bitmap_flushable(info, 1); if (delete_head_or_tail(info, ma_recordpos_to_page(info->cur_row.lastpos), ma_recordpos_to_dir_entry(info->cur_row.lastpos), 1, @@ -2832,7 +3323,7 @@ my_bool _ma_write_abort_block_record(MARIA_HA *info) &lsn, (void*) 0)) res= 1; } - _ma_bitmap_flushable(share, -1); + _ma_bitmap_flushable(info, -1); _ma_unpin_all_pages_and_finalize_row(info, lsn); DBUG_RETURN(res); } @@ -2882,16 +3373,16 @@ static my_bool _ma_update_block_record2(MARIA_HA *info, calc_record_size(info, record, new_row); page= ma_recordpos_to_page(record_pos); - _ma_bitmap_flushable(share, 1); - DBUG_ASSERT(share->pagecache->block_size == block_size); - if (!(buff= pagecache_read(share->pagecache, - &info->dfile, (pgcache_page_no_t) page, 0, - info->buff, share->page_type, - PAGECACHE_LOCK_WRITE, &page_link.link))) - goto err; + _ma_bitmap_flushable(info, 1); + buff= pagecache_read(share->pagecache, + &info->dfile, (pgcache_page_no_t) page, 0, + info->buff, share->page_type, + PAGECACHE_LOCK_WRITE, &page_link.link); page_link.unlock= PAGECACHE_LOCK_WRITE_UNLOCK; - page_link.changed= 1; + page_link.changed= buff != 0; push_dynamic(&info->pinned_pages, (void*) &page_link); + if (!buff) + goto err; org_empty_size= uint2korr(buff + EMPTY_SPACE_OFFSET); rownr= ma_recordpos_to_dir_entry(record_pos); @@ -2908,8 +3399,7 @@ static my_bool _ma_update_block_record2(MARIA_HA *info, */ block.org_bitmap_value= _ma_free_size_to_head_pattern(&share->bitmap, org_empty_size); - - if (extend_area_on_page(buff, dir, rownr, share->block_size, + if (extend_area_on_page(buff, dir, rownr, block_size, new_row->total_length, &org_empty_size, &rec_offset, &length)) goto err; @@ -2926,8 +3416,6 @@ static my_bool _ma_update_block_record2(MARIA_HA *info, block.sub_blocks= 1; block.used= BLOCKUSED_USED | BLOCKUSED_USE_ORG_BITMAP; block.empty_space= row_pos.empty_space; - /* Update cur_row, if someone calls update at once again */ - cur_row->head_length= new_row->total_length; if (*cur_row->tail_positions && delete_tails(info, cur_row->tail_positions)) @@ -2936,45 +3424,173 @@ static my_bool _ma_update_block_record2(MARIA_HA *info, goto err; res= write_block_record(info, oldrec, record, new_row, blocks, 1, &row_pos, undo_lsn, old_checksum); + /* We can't update or delete this without re-reading it again */ + info->update&= ~HA_STATE_AKTIV; DBUG_RETURN(res); } + /* Delete old row */ + if (*cur_row->tail_positions && + delete_tails(info, cur_row->tail_positions)) + goto err; + if (cur_row->extents_count && free_full_pages(info, cur_row)) + goto err; + + head_length= uint2korr(dir + 2); + if (_ma_bitmap_find_new_place(info, new_row, page, head_length + + org_empty_size, blocks)) + goto err; + /* Allocate all size in block for record TODO: Need to improve this to do compact if we can fit one more blob into the head page */ - head_length= uint2korr(dir + 2); - if ((buff[PAGE_TYPE_OFFSET] & PAGE_CAN_BE_COMPACTED) && org_empty_size && - (head_length < new_row->head_length || + if ((head_length < new_row->space_on_head_page || (new_row->total_length <= head_length && org_empty_size + head_length >= new_row->total_length))) { - compact_page(buff, share->block_size, rownr, 1); + compact_page(buff, block_size, rownr, 1); org_empty_size= 0; head_length= uint2korr(dir + 2); } - /* Delete old row */ - if (*cur_row->tail_positions && delete_tails(info, cur_row->tail_positions)) + row_pos.buff= buff; + row_pos.rownr= rownr; + row_pos.empty_space= org_empty_size + head_length; + row_pos.dir= dir; + row_pos.data= buff + uint2korr(dir); + row_pos.length= head_length; + if ((res= write_block_record(info, oldrec, record, new_row, blocks, 1, + &row_pos, undo_lsn, old_checksum))) goto err; - if (cur_row->extents_count && free_full_pages(info, cur_row)) + DBUG_RETURN(0); + +err: + if (info->non_flushable_state) + _ma_bitmap_flushable(info, -1); + _ma_unpin_all_pages_and_finalize_row(info, LSN_IMPOSSIBLE); + DBUG_RETURN(1); +} + + +/* + @brief Store new row on it's original position + + @note + This is basicly a copy of _ma_update_block_record2 + When we have a purge thread for deleted row, we can remove this function + and use _ma_update_block_record2 instead. + + This is the main reason we don't make a lot of subfunctions that are + common between _ma_update_block_record2() and this function. +*/ + +static my_bool _ma_update_at_original_place(MARIA_HA *info, + ulonglong page, + uint rownr, + uint length_on_head_page, + uint extent_count, + const uchar *extent_info, + const uchar *oldrec, + const uchar *record, + LSN undo_lsn) +{ + MARIA_BITMAP_BLOCKS *blocks; + MARIA_BITMAP_BLOCK *block; + MARIA_ROW *cur_row= &info->cur_row, *new_row= &info->new_row; + MARIA_PINNED_PAGE page_link; + MARIA_SHARE *share= info->s; + ha_checksum old_checksum; + uint org_empty_size, empty_size; + uint block_size= info->s->block_size; + uchar *dir, *buff; + struct st_row_pos_info row_pos; + my_bool res; + uint rec_offset, length; + DBUG_ENTER("_ma_update_at_original_place"); + +#ifdef ENABLE_IF_PROBLEM_WITH_UPDATE + DBUG_DUMP("oldrec", oldrec, share->base.reclength); + DBUG_DUMP("newrec", record, share->base.reclength); +#endif + + /* + Checksums of new and old rows were computed by callers already; new + row's was put into cur_row, old row's was put into new_row. + */ + old_checksum= new_row->checksum; + new_row->checksum= cur_row->checksum; + calc_record_size(info, record, new_row); + + _ma_bitmap_flushable(info, 1); + buff= pagecache_read(share->pagecache, + &info->dfile, (pgcache_page_no_t) page, 0, + info->buff, share->page_type, + PAGECACHE_LOCK_WRITE, &page_link.link); + page_link.unlock= PAGECACHE_LOCK_WRITE_UNLOCK; + page_link.changed= buff != 0; + push_dynamic(&info->pinned_pages, (void*) &page_link); + if (!buff) + goto err; + + org_empty_size= uint2korr(buff + EMPTY_SPACE_OFFSET); + dir= dir_entry_pos(buff, block_size, rownr); + + if ((org_empty_size + cur_row->head_length) < length_on_head_page) + { + my_errno= HA_ERR_WRONG_IN_RECORD; goto err; - if (_ma_bitmap_find_new_place(info, new_row, page, head_length, blocks)) + } + + /* + We can fit the new row in the same page as the original head part + of the row + */ + empty_size= org_empty_size; + if (extend_area_on_page(buff, dir, rownr, block_size, + length_on_head_page, &empty_size, + &rec_offset, &length)) goto err; row_pos.buff= buff; row_pos.rownr= rownr; - row_pos.empty_space= org_empty_size + head_length; + row_pos.empty_space= empty_size; row_pos.dir= dir; - row_pos.data= buff + uint2korr(dir); - row_pos.length= head_length; - res= write_block_record(info, oldrec, record, new_row, blocks, 1, - &row_pos, undo_lsn, old_checksum); - DBUG_RETURN(res); + row_pos.data= buff + rec_offset; + row_pos.length= length_on_head_page; + + /* Delete old row */ + if (*cur_row->tail_positions && + delete_tails(info, cur_row->tail_positions)) + goto err; + if (cur_row->extents_count && free_full_pages(info, cur_row)) + goto err; + + /* Change extent information to be usable by write_block_record() */ + blocks= &cur_row->insert_blocks; + if (extent_to_bitmap_blocks(info, blocks, page, extent_count, extent_info)) + goto err; + block= blocks->block; + block->empty_space= row_pos.empty_space; + block->org_bitmap_value= _ma_free_size_to_head_pattern(&share->bitmap, + org_empty_size); + DBUG_ASSERT(block->org_bitmap_value == + _ma_bitmap_get_page_bits(info, &info->s->bitmap, page)); + block->used|= BLOCKUSED_USE_ORG_BITMAP; + + DBUG_ASSERT(blocks->count > 1 || + max(new_row->total_length, share->base.min_block_length) == + length_on_head_page); + + if ((res= write_block_record(info, oldrec, record, new_row, blocks, + 1, &row_pos, undo_lsn, old_checksum))) + goto err; + DBUG_RETURN(0); err: - _ma_bitmap_flushable(share, -1); + if (info->non_flushable_state) + _ma_bitmap_flushable(info, -1); _ma_unpin_all_pages_and_finalize_row(info, LSN_IMPOSSIBLE); DBUG_RETURN(1); } @@ -3026,6 +3642,7 @@ static int delete_dir_entry(uchar *buff, uint block_size, uint record_number, } #endif + check_directory(buff, block_size); empty_space= uint2korr(buff + EMPTY_SPACE_OFFSET); dir= dir_entry_pos(buff, block_size, record_number); length= uint2korr(dir + 2); @@ -3099,6 +3716,8 @@ static int delete_dir_entry(uchar *buff, uint block_size, uint record_number, buff[PAGE_TYPE_OFFSET]|= (uchar) PAGE_CAN_BE_COMPACTED; *empty_space_res= empty_space; + + check_directory(buff, block_size); DBUG_RETURN(0); } @@ -3136,18 +3755,23 @@ static my_bool delete_head_or_tail(MARIA_HA *info, int res; enum pagecache_page_lock lock_at_write, lock_at_unpin; DBUG_ENTER("delete_head_or_tail"); + DBUG_PRINT("enter", ("id: %lu (%lu:%u)", + (ulong) ma_recordpos(page, record_number), + (ulong) page, record_number)); - info->keyread_buff_used= 1; - DBUG_ASSERT(info->s->pagecache->block_size == block_size); - if (!(buff= pagecache_read(share->pagecache, - &info->dfile, page, 0, - info->keyread_buff, - info->s->page_type, - PAGECACHE_LOCK_WRITE, &page_link.link))) - DBUG_RETURN(1); + DBUG_ASSERT(share->pagecache->block_size == block_size); + buff= pagecache_read(share->pagecache, + &info->dfile, page, 0, + 0, + share->page_type, + PAGECACHE_LOCK_WRITE, &page_link.link); page_link.unlock= PAGECACHE_LOCK_WRITE_UNLOCK; - page_link.changed= 1; + page_link.changed= buff != 0; push_dynamic(&info->pinned_pages, (void*) &page_link); + if (!buff) + DBUG_RETURN(1); + DBUG_ASSERT((buff[PAGE_TYPE_OFFSET] & PAGE_TYPE_MASK) == + (head ? HEAD_PAGE : TAIL_PAGE)); if (from_update) { @@ -3167,7 +3791,7 @@ static my_bool delete_head_or_tail(MARIA_HA *info, { uchar log_data[FILEID_STORE_SIZE + PAGE_STORE_SIZE + DIRPOS_STORE_SIZE]; LEX_STRING log_array[TRANSLOG_INTERNAL_PARTS + 1]; - if (info->s->now_transactional) + if (share->now_transactional) { /* Log REDO data */ page_store(log_data + FILEID_STORE_SIZE, page); @@ -3194,7 +3818,7 @@ static my_bool delete_head_or_tail(MARIA_HA *info, } else /* page is now empty */ { - if (info->s->now_transactional) + if (share->now_transactional) { uchar log_data[FILEID_STORE_SIZE + PAGE_STORE_SIZE]; LEX_STRING log_array[TRANSLOG_INTERNAL_PARTS + 1]; @@ -3217,11 +3841,10 @@ static my_bool delete_head_or_tail(MARIA_HA *info, LSN_IMPOSSIBLE)) DBUG_RETURN(1); - DBUG_ASSERT(empty_space >= info->s->bitmap.sizes[0]); + DBUG_ASSERT(empty_space >= share->bitmap.sizes[0]); } /* The page is pinned with a read lock */ page_link.unlock= lock_at_unpin; - page_link.changed= 1; set_dynamic(&info->pinned_pages, (void*) &page_link, info->pinned_pages.elements-1); @@ -3280,10 +3903,10 @@ my_bool _ma_delete_block_record(MARIA_HA *info, const uchar *record) page= ma_recordpos_to_page(info->cur_row.lastpos); record_number= ma_recordpos_to_dir_entry(info->cur_row.lastpos); - DBUG_PRINT("enter", ("Rowid: %lu (%lu:%u)", (ulong) info->cur_row.lastpos, + DBUG_PRINT("enter", ("rowid: %lu (%lu:%u)", (ulong) info->cur_row.lastpos, (ulong) page, record_number)); - _ma_bitmap_flushable(share, 1); + _ma_bitmap_flushable(info, 1); if (delete_head_or_tail(info, page, record_number, 1, 0) || delete_tails(info, info->cur_row.tail_positions)) goto err; @@ -3294,48 +3917,57 @@ my_bool _ma_delete_block_record(MARIA_HA *info, const uchar *record) if (share->now_transactional) { uchar log_data[LSN_STORE_SIZE + FILEID_STORE_SIZE + PAGE_STORE_SIZE + - DIRPOS_STORE_SIZE + HA_CHECKSUM_STORE_SIZE]; + DIRPOS_STORE_SIZE + 2 + PAGERANGE_STORE_SIZE + + HA_CHECKSUM_STORE_SIZE]; + uchar *log_pos; size_t row_length; - uint row_parts_count; + uint row_parts_count, extents_length; ha_checksum checksum_delta; /* Write UNDO record */ lsn_store(log_data, info->trn->undo_lsn); page_store(log_data + LSN_STORE_SIZE + FILEID_STORE_SIZE, page); - dirpos_store(log_data + LSN_STORE_SIZE + FILEID_STORE_SIZE + - PAGE_STORE_SIZE, record_number); + log_pos= log_data + LSN_STORE_SIZE + FILEID_STORE_SIZE + PAGE_STORE_SIZE; + dirpos_store(log_pos, record_number); + log_pos+= DIRPOS_STORE_SIZE; + int2store(log_pos, info->cur_row.head_length); + log_pos+= 2; + pagerange_store(log_pos, info->cur_row.extents_count); + log_pos+= PAGERANGE_STORE_SIZE; info->log_row_parts[TRANSLOG_INTERNAL_PARTS].str= (char*) log_data; info->log_row_parts[TRANSLOG_INTERNAL_PARTS].length= sizeof(log_data) - HA_CHECKSUM_STORE_SIZE; store_checksum_in_rec(share, checksum_delta, - - info->cur_row.checksum, - log_data + LSN_STORE_SIZE + FILEID_STORE_SIZE + - PAGE_STORE_SIZE + DIRPOS_STORE_SIZE, + - info->cur_row.checksum, log_pos, info->log_row_parts[TRANSLOG_INTERNAL_PARTS + 0].length); + info->log_row_parts[TRANSLOG_INTERNAL_PARTS+1].str= + info->cur_row.extents; + info->log_row_parts[TRANSLOG_INTERNAL_PARTS+1].length= + extents_length= info->cur_row.extents_count * ROW_EXTENT_SIZE; row_length= fill_insert_undo_parts(info, record, info->log_row_parts + - TRANSLOG_INTERNAL_PARTS + 1, + TRANSLOG_INTERNAL_PARTS + 2, &row_parts_count); if (translog_write_record(&lsn, LOGREC_UNDO_ROW_DELETE, info->trn, info, - info->log_row_parts[TRANSLOG_INTERNAL_PARTS + - 0].length + row_length, - TRANSLOG_INTERNAL_PARTS + 1 + row_parts_count, + (info->log_row_parts[TRANSLOG_INTERNAL_PARTS + + 0].length + row_length + + extents_length), + TRANSLOG_INTERNAL_PARTS + 2 + row_parts_count, info->log_row_parts, log_data + LSN_STORE_SIZE, &checksum_delta)) goto err; - } - _ma_bitmap_flushable(share, -1); + _ma_bitmap_flushable(info, -1); _ma_unpin_all_pages_and_finalize_row(info, lsn); DBUG_RETURN(0); err: - _ma_bitmap_flushable(share, -1); + _ma_bitmap_flushable(info, -1); _ma_unpin_all_pages_and_finalize_row(info, LSN_IMPOSSIBLE); DBUG_RETURN(1); } @@ -3417,7 +4049,8 @@ static void init_extent(MARIA_EXTENT_CURSOR *extent, uchar *extent_info, extent->extent= extent_info; extent->extent_count= extents; extent->page= page_korr(extent_info); /* First extent */ - page_count= uint2korr(extent_info + ROW_EXTENT_PAGE_SIZE); + page_count= (uint2korr(extent_info + ROW_EXTENT_PAGE_SIZE) & + ~START_EXTENT_BIT); extent->tail= page_count & TAIL_BIT; if (extent->tail) { @@ -3466,7 +4099,8 @@ static uchar *read_next_extent(MARIA_HA *info, MARIA_EXTENT_CURSOR *extent, goto crashed; extent->extent+= ROW_EXTENT_SIZE; extent->page= page_korr(extent->extent); - page_count= uint2korr(extent->extent+ROW_EXTENT_PAGE_SIZE); + page_count= (uint2korr(extent->extent+ROW_EXTENT_PAGE_SIZE) & + ~START_EXTENT_BIT); if (!page_count) goto crashed; extent->tail= page_count & TAIL_BIT; @@ -3485,10 +4119,18 @@ static uchar *read_next_extent(MARIA_HA *info, MARIA_EXTENT_CURSOR *extent, lock= extent->lock_for_tail_pages; DBUG_ASSERT(share->pagecache->block_size == share->block_size); - if (!(buff= pagecache_read(share->pagecache, - &info->dfile, extent->page, 0, - info->buff, share->page_type, - lock, &page_link.link))) + buff= pagecache_read(share->pagecache, + &info->dfile, extent->page, 0, + info->buff, share->page_type, + lock, &page_link.link); + if (lock != PAGECACHE_LOCK_LEFT_UNLOCKED) + { + /* Read during UNDO */ + page_link.unlock= PAGECACHE_LOCK_WRITE_UNLOCK; + page_link.changed= buff != 0; + push_dynamic(&info->pinned_pages, (void*) &page_link); + } + if (!buff) { /* check if we tried to read over end of file (ie: bad data in record) */ if ((extent->page + 1) * share->block_size > info->state->data_file_length) @@ -3509,14 +4151,6 @@ static uchar *read_next_extent(MARIA_HA *info, MARIA_EXTENT_CURSOR *extent, } /* Found tail */ - if (lock != PAGECACHE_LOCK_LEFT_UNLOCKED) - { - /* Read during redo */ - page_link.unlock= PAGECACHE_LOCK_WRITE_UNLOCK; - page_link.changed= 1; - push_dynamic(&info->pinned_pages, (void*) &page_link); - } - if ((buff[PAGE_TYPE_OFFSET] & PAGE_TYPE_MASK) != TAIL_PAGE) goto crashed; *(extent->tail_positions++)= ma_recordpos(extent->page, @@ -3850,10 +4484,10 @@ int _ma_read_block_record2(MARIA_HA *info, uchar *record, } cur_row->blob_length= blob_lengths; DBUG_PRINT("info", ("Total blob length: %lu", blob_lengths)); - if (_ma_alloc_buffer(&info->rec_buff, &info->rec_buff_size, + if (_ma_alloc_buffer(&info->blob_buff, &info->blob_buff_size, blob_lengths)) DBUG_RETURN(my_errno); - blob_buffer= info->rec_buff; + blob_buffer= info->blob_buff; } memcpy(field_pos, field_length_data, column_size_length); @@ -3910,10 +4544,17 @@ int _ma_read_block_record2(MARIA_HA *info, uchar *record, for allowing the row to expand. */ if (data != end_of_data && (uint) (end_of_data - start_of_data) > - info->s->base.min_block_length) + share->base.min_block_length) goto err; } - +#ifdef EXTRA_DEBUG + if (share->calc_checksum) + { + /* Esnure that row checksum is correct */ + DBUG_ASSERT(((share->calc_checksum)(info, record) & 255) == + cur_row->checksum); + } +#endif info->update|= HA_STATE_AKTIV; /* We have an active record */ DBUG_RETURN(0); @@ -3942,9 +4583,11 @@ static my_bool read_row_extent_info(MARIA_HA *info, uchar *buff, uint record_number) { MARIA_SHARE *share= info->s; - uchar *data, *end_of_data; - uint flag, row_extents, field_lengths; MARIA_EXTENT_CURSOR extent; + MARIA_RECORD_POS *tail_pos; + uchar *data, *end_of_data; + uint flag, row_extents, row_extents_size, field_lengths; + uchar *extents, *end; DBUG_ENTER("read_row_extent_info"); if (!(data= get_record_position(buff, share->block_size, @@ -3956,19 +4599,19 @@ static my_bool read_row_extent_info(MARIA_HA *info, uchar *buff, data+= total_header_size[(flag & PRECALC_HEADER_BITMASK)]; row_extents= 0; + row_extents_size= 0; if (flag & ROW_FLAG_EXTENTS) { - uint row_extent_size; /* Record is split over many data pages. Get number of extents and first extent */ get_key_length(row_extents, data); - row_extent_size= row_extents * ROW_EXTENT_SIZE; - if (info->cur_row.extents_buffer_length < row_extent_size && + row_extents_size= row_extents * ROW_EXTENT_SIZE; + if (info->cur_row.extents_buffer_length < row_extents_size && _ma_alloc_buffer(&info->cur_row.extents, &info->cur_row.extents_buffer_length, - row_extent_size)) + row_extents_size)) DBUG_RETURN(1); memcpy(info->cur_row.extents, data, ROW_EXTENT_SIZE); data+= ROW_EXTENT_SIZE; @@ -3976,8 +4619,6 @@ static my_bool read_row_extent_info(MARIA_HA *info, uchar *buff, info->cur_row.tail_positions); extent.first_extent= 1; } - else - (*info->cur_row.tail_positions)= 0; info->cur_row.extents_count= row_extents; if (share->base.max_field_lengths) @@ -3987,9 +4628,6 @@ static my_bool read_row_extent_info(MARIA_HA *info, uchar *buff, info->cur_row.checksum= (uint) (uchar) *data++; if (row_extents > 1) { - MARIA_RECORD_POS *tail_pos; - uchar *extents, *end; - data+= share->base.null_bytes; data+= share->base.pack_bytes; data+= share->base.field_offsets * FIELD_OFFSET_SIZE; @@ -4001,28 +4639,28 @@ static my_bool read_row_extent_info(MARIA_HA *info, uchar *buff, */ extent.lock_for_tail_pages= PAGECACHE_LOCK_LEFT_WRITELOCKED; if (read_long_data(info, info->cur_row.extents + ROW_EXTENT_SIZE, - (row_extents - 1) * ROW_EXTENT_SIZE, + row_extents_size - ROW_EXTENT_SIZE, &extent, &data, &end_of_data)) DBUG_RETURN(1); + } - /* Update tail_positions with pointer to tails */ - tail_pos= info->cur_row.tail_positions; - for (extents= info->cur_row.extents, end= extents+ row_extents; - extents < end; - extents += ROW_EXTENT_SIZE) - { - ulonglong page= uint5korr(extents); - uint page_count= uint2korr(extents + ROW_EXTENT_PAGE_SIZE); - if (page_count & TAIL_BIT) - *(tail_pos++)= ma_recordpos(page, (page_count & ~TAIL_BIT)); - } - *tail_pos= 0; /* End marker */ + /* Update tail_positions with pointer to tails */ + tail_pos= info->cur_row.tail_positions; + for (extents= info->cur_row.extents, end= extents + row_extents_size; + extents < end; + extents+= ROW_EXTENT_SIZE) + { + ulonglong page= uint5korr(extents); + uint page_count= uint2korr(extents + ROW_EXTENT_PAGE_SIZE); + if (page_count & TAIL_BIT) + *(tail_pos++)= ma_recordpos(page, (page_count & ~ (TAIL_BIT | + START_EXTENT_BIT))); } + *tail_pos= 0; /* End marker */ DBUG_RETURN(0); } - /* Read a record based on record position @@ -4039,18 +4677,19 @@ static my_bool read_row_extent_info(MARIA_HA *info, uchar *buff, int _ma_read_block_record(MARIA_HA *info, uchar *record, MARIA_RECORD_POS record_pos) { + MARIA_SHARE *share= info->s; uchar *data, *end_of_data, *buff; uint offset; - uint block_size= info->s->block_size; + uint block_size= share->block_size; DBUG_ENTER("_ma_read_block_record"); DBUG_PRINT("enter", ("rowid: %lu", (long) record_pos)); offset= ma_recordpos_to_dir_entry(record_pos); - DBUG_ASSERT(info->s->pagecache->block_size == block_size); - if (!(buff= pagecache_read(info->s->pagecache, + DBUG_ASSERT(share->pagecache->block_size == block_size); + if (!(buff= pagecache_read(share->pagecache, &info->dfile, ma_recordpos_to_page(record_pos), 0, - info->buff, info->s->page_type, + info->buff, share->page_type, PAGECACHE_LOCK_LEFT_UNLOCKED, 0))) DBUG_RETURN(my_errno); DBUG_ASSERT((buff[PAGE_TYPE_OFFSET] & PAGE_TYPE_MASK) == HEAD_PAGE); @@ -4122,6 +4761,7 @@ my_bool _ma_cmp_block_unique(MARIA_HA *info, MARIA_UNIQUEDEF *def, my_bool _ma_scan_init_block_record(MARIA_HA *info) { + MARIA_SHARE *share= info->s; DBUG_ENTER("_ma_scan_init_block_record"); /* bitmap_buff may already be allocated if this is the second call to @@ -4129,20 +4769,20 @@ my_bool _ma_scan_init_block_record(MARIA_HA *info) */ if (!(info->scan.bitmap_buff || ((info->scan.bitmap_buff= - (uchar *) my_malloc(info->s->block_size * 2, MYF(MY_WME)))))) + (uchar *) my_malloc(share->block_size * 2, MYF(MY_WME)))))) DBUG_RETURN(1); - info->scan.page_buff= info->scan.bitmap_buff + info->s->block_size; - info->scan.bitmap_end= info->scan.bitmap_buff + info->s->bitmap.total_size; + info->scan.page_buff= info->scan.bitmap_buff + share->block_size; + info->scan.bitmap_end= info->scan.bitmap_buff + share->bitmap.total_size; /* Set scan variables to get _ma_scan_block() to start with reading bitmap */ info->scan.number_of_rows= 0; info->scan.bitmap_pos= info->scan.bitmap_end; - info->scan.bitmap_page= (ulong) - (long) info->s->bitmap.pages_covered; + info->scan.bitmap_page= (ulong) - (long) share->bitmap.pages_covered; /* We have to flush bitmap as we will read the bitmap from the page cache while scanning rows */ - DBUG_RETURN(_ma_bitmap_flush(info->s)); + DBUG_RETURN(_ma_bitmap_wait_or_flush(info->s)); } @@ -4420,40 +5060,6 @@ my_bool _ma_compare_block_record(MARIA_HA *info __attribute__ ((unused)), } -#ifndef DBUG_OFF - -static void _ma_print_directory(uchar *buff, uint block_size) -{ - uint max_entry= (uint) ((uchar *) buff)[DIR_COUNT_OFFSET], row= 0; - uint end_of_prev_row= PAGE_HEADER_SIZE; - uchar *dir, *end; - - dir= dir_entry_pos(buff, block_size, max_entry-1); - end= dir_entry_pos(buff, block_size, 0); - - DBUG_LOCK_FILE; - fprintf(DBUG_FILE,"Directory dump (pos:length):\n"); - - for (row= 1; dir <= end ; end-= DIR_ENTRY_SIZE, row++) - { - uint offset= uint2korr(end); - uint length= uint2korr(end+2); - fprintf(DBUG_FILE, " %4u:%4u", offset, offset ? length : 0); - if (!(row % (80/12))) - fputc('\n', DBUG_FILE); - if (offset) - { - DBUG_ASSERT(offset >= end_of_prev_row); - end_of_prev_row= offset + length; - } - } - fputc('\n', DBUG_FILE); - fflush(DBUG_FILE); - DBUG_UNLOCK_FILE; -} -#endif /* DBUG_OFF */ - - /* Store an integer with simple packing @@ -4612,7 +5218,9 @@ static size_t fill_insert_undo_parts(MARIA_HA *info, const uchar *record, if (share->base.blobs) { - /* Store total blob length to make buffer allocation easier during undo */ + /* + Store total blob length to make buffer allocation easier during UNDO + */ log_parts->str= info->length_buff; log_parts->length= (uint) (ma_store_length(log_parts->str, info->cur_row.blob_length) - @@ -5130,6 +5738,7 @@ uint _ma_apply_redo_insert_row_head_or_tail(MARIA_HA *info, LSN lsn, enum pagecache_page_lock unlock_method; enum pagecache_page_pin unpin_method; my_off_t end_of_page; + uint error; DBUG_ENTER("_ma_apply_redo_insert_row_head_or_tail"); page= page_korr(header); @@ -5139,7 +5748,7 @@ uint _ma_apply_redo_insert_row_head_or_tail(MARIA_HA *info, LSN lsn, (ulong) ma_recordpos(page, rownr), (ulong) page, rownr, (uint) data_length)); - end_of_page= (page + 1) * info->s->block_size; + end_of_page= (page + 1) * share->block_size; if (end_of_page > info->state->data_file_length) { DBUG_PRINT("info", ("Enlarging data file from %lu to %lu", @@ -5152,21 +5761,25 @@ uint _ma_apply_redo_insert_row_head_or_tail(MARIA_HA *info, LSN lsn, fill it entirely with zeroes, then the REDO will put correct data on it. */ - DBUG_ASSERT(rownr == 0); - if (rownr != 0) - goto err; unlock_method= PAGECACHE_LOCK_WRITE; unpin_method= PAGECACHE_PIN; + DBUG_ASSERT(rownr == 0); + if (rownr != 0) + goto crashed_file; + buff= info->keyread_buff; info->keyread_buff_used= 1; - make_empty_page(info, buff, page_type); + make_empty_page(info, buff, page_type, 1); empty_space= (block_size - PAGE_OVERHEAD_SIZE); rec_offset= PAGE_HEADER_SIZE; dir= buff+ block_size - PAGE_SUFFIX_SIZE - DIR_ENTRY_SIZE; } else { + unlock_method= PAGECACHE_LOCK_LEFT_WRITELOCKED; + unpin_method= PAGECACHE_PIN_LEFT_PINNED; + share->pagecache->readwrite_flags&= ~MY_WME; buff= pagecache_read(share->pagecache, &info->dfile, page, 0, 0, @@ -5178,32 +5791,23 @@ uint _ma_apply_redo_insert_row_head_or_tail(MARIA_HA *info, LSN lsn, /* Skip errors when reading outside of file and uninitialized pages */ if (my_errno != HA_ERR_FILE_TOO_SHORT && my_errno != HA_ERR_WRONG_CRC) - { - /* Fatal disk error when reading page */ - pagecache_unlock_by_link(share->pagecache, page_link.link, - PAGECACHE_LOCK_WRITE_UNLOCK, - PAGECACHE_UNPIN, LSN_IMPOSSIBLE, - LSN_IMPOSSIBLE, 0); - DBUG_RETURN(my_errno); - } + goto err; /* Create new page */ buff= pagecache_block_link_to_buffer(page_link.link); buff[PAGE_TYPE_OFFSET]= UNALLOCATED_PAGE; } else if (lsn_korr(buff) >= lsn) /* Test if already applied */ { + /* Fix bitmap, just in case */ + empty_space= uint2korr(buff + EMPTY_SPACE_OFFSET); + if (_ma_bitmap_set(info, page, page_type == HEAD_PAGE, empty_space)) + goto err; pagecache_unlock_by_link(share->pagecache, page_link.link, PAGECACHE_LOCK_WRITE_UNLOCK, PAGECACHE_UNPIN, LSN_IMPOSSIBLE, LSN_IMPOSSIBLE, 0); - /* Fix bitmap, just in case */ - empty_space= uint2korr(buff + EMPTY_SPACE_OFFSET); - if (_ma_bitmap_set(info, page, page_type == HEAD_PAGE, empty_space)) - DBUG_RETURN(my_errno); DBUG_RETURN(0); } - unlock_method= PAGECACHE_LOCK_LEFT_WRITELOCKED; - unpin_method= PAGECACHE_PIN_LEFT_PINNED; if (((buff[PAGE_TYPE_OFFSET] & PAGE_TYPE_MASK) != page_type)) { @@ -5211,19 +5815,21 @@ uint _ma_apply_redo_insert_row_head_or_tail(MARIA_HA *info, LSN lsn, This is a page that has been freed before and now should be changed to new type. */ - DBUG_ASSERT(rownr == 0); if (((buff[PAGE_TYPE_OFFSET] & PAGE_TYPE_MASK) != BLOB_PAGE && - (buff[PAGE_TYPE_OFFSET] & PAGE_TYPE_MASK) != UNALLOCATED_PAGE) || - rownr != 0) - goto err; - make_empty_page(info, buff, page_type); - empty_space= (block_size - PAGE_OVERHEAD_SIZE); + (buff[PAGE_TYPE_OFFSET] & PAGE_TYPE_MASK) != UNALLOCATED_PAGE)) + goto crashed_file; + make_empty_page(info, buff, page_type, 0); + empty_space= block_size - PAGE_HEADER_SIZE - PAGE_SUFFIX_SIZE; + (void) extend_directory(buff, block_size, 0, rownr, + &empty_space); rec_offset= PAGE_HEADER_SIZE; - dir= buff+ block_size - PAGE_SUFFIX_SIZE - DIR_ENTRY_SIZE; + dir= dir_entry_pos(buff, block_size, rownr); + empty_space+= uint2korr(dir+2); } else { uint max_entry= (uint) buff[DIR_COUNT_OFFSET]; + uint length; dir= dir_entry_pos(buff, block_size, rownr); empty_space= uint2korr(buff + EMPTY_SPACE_OFFSET); @@ -5231,38 +5837,14 @@ uint _ma_apply_redo_insert_row_head_or_tail(MARIA_HA *info, LSN lsn, if (max_entry <= rownr) { /* Add directory entry first in directory and data last on page */ - DBUG_ASSERT(max_entry == rownr); - if (max_entry != rownr) - goto err; - rec_offset= (uint2korr(dir + DIR_ENTRY_SIZE) + - uint2korr(dir + DIR_ENTRY_SIZE +2)); - if ((uint) (dir - buff) < rec_offset + data_length) - { - /* Create place for directory & data */ - compact_page(buff, block_size, max_entry - 1, 0); - rec_offset= (uint2korr(dir + DIR_ENTRY_SIZE) + - uint2korr(dir + DIR_ENTRY_SIZE + 2)); - empty_space= uint2korr(buff + EMPTY_SPACE_OFFSET); - DBUG_ASSERT(!((uint) (dir - buff) < rec_offset + data_length)); - if ((uint) (dir - buff) < rec_offset + data_length) - goto err; - } - buff[DIR_COUNT_OFFSET]= (uchar) max_entry+1; - int2store(dir, rec_offset); - empty_space-= DIR_ENTRY_SIZE; - } - else - { - uint length; - /* - Reuse old entry. This is empty if the command was an insert and - possible used if the command was an update. - */ - if (extend_area_on_page(buff, dir, rownr, block_size, - data_length, &empty_space, - &rec_offset, &length)) - goto err; + if (extend_directory(buff, block_size, max_entry, rownr, + &empty_space)) + goto crashed_file; } + if (extend_area_on_page(buff, dir, rownr, block_size, + data_length, &empty_space, + &rec_offset, &length)) + goto crashed_file; } } /* Copy data */ @@ -5288,14 +5870,14 @@ uint _ma_apply_redo_insert_row_head_or_tail(MARIA_HA *info, LSN lsn, LSN_IMPOSSIBLE)) result= my_errno; + /* Fix bitmap */ + if (_ma_bitmap_set(info, page, page_type == HEAD_PAGE, empty_space)) + goto err; + page_link.unlock= PAGECACHE_LOCK_WRITE_UNLOCK; page_link.changed= 1; push_dynamic(&info->pinned_pages, (void*) &page_link); - /* Fix bitmap */ - if (_ma_bitmap_set(info, page, page_type == HEAD_PAGE, empty_space)) - result= my_errno; - /* Data page and bitmap page are in place, we can update data_file_length in case we extended the file. We could not do it earlier: bitmap code tests @@ -5304,13 +5886,17 @@ uint _ma_apply_redo_insert_row_head_or_tail(MARIA_HA *info, LSN lsn, set_if_bigger(info->state->data_file_length, end_of_page); DBUG_RETURN(result); +crashed_file: + my_errno= HA_ERR_WRONG_IN_RECORD; err: + error= my_errno; if (unlock_method == PAGECACHE_LOCK_LEFT_WRITELOCKED) pagecache_unlock_by_link(share->pagecache, page_link.link, PAGECACHE_LOCK_WRITE_UNLOCK, PAGECACHE_UNPIN, LSN_IMPOSSIBLE, LSN_IMPOSSIBLE, 0); - DBUG_RETURN(HA_ERR_WRONG_IN_RECORD); + _ma_mark_file_crashed(share); + DBUG_RETURN((my_errno= error)); } @@ -5342,6 +5928,7 @@ uint _ma_apply_redo_purge_row_head_or_tail(MARIA_HA *info, LSN lsn, uint block_size= share->block_size; uchar *buff= info->keyread_buff; int result; + uint error; MARIA_PINNED_PAGE page_link; DBUG_ENTER("_ma_apply_redo_purge_row_head_or_tail"); @@ -5357,13 +5944,7 @@ uint _ma_apply_redo_purge_row_head_or_tail(MARIA_HA *info, LSN lsn, page, 0, 0, PAGECACHE_PLAIN_PAGE, PAGECACHE_LOCK_WRITE, &page_link.link))) - { - pagecache_unlock_by_link(share->pagecache, page_link.link, - PAGECACHE_LOCK_WRITE_UNLOCK, - PAGECACHE_UNPIN, LSN_IMPOSSIBLE, - LSN_IMPOSSIBLE, 0); - DBUG_RETURN(my_errno); - } + goto err; if (lsn_korr(buff) >= lsn) { @@ -5372,25 +5953,27 @@ uint _ma_apply_redo_purge_row_head_or_tail(MARIA_HA *info, LSN lsn, Note that in case the page is not anymore a head or tail page a future redo will fix the bitmap. */ - pagecache_unlock_by_link(share->pagecache, page_link.link, - PAGECACHE_LOCK_WRITE_UNLOCK, - PAGECACHE_UNPIN, LSN_IMPOSSIBLE, - LSN_IMPOSSIBLE, 0); - if ((buff[PAGE_TYPE_OFFSET] & PAGE_TYPE_MASK) == page_type) { empty_space= uint2korr(buff+EMPTY_SPACE_OFFSET); if (_ma_bitmap_set(info, page, page_type == HEAD_PAGE, empty_space)) - DBUG_RETURN(my_errno); + goto err; } + pagecache_unlock_by_link(share->pagecache, page_link.link, + PAGECACHE_LOCK_WRITE_UNLOCK, + PAGECACHE_UNPIN, LSN_IMPOSSIBLE, + LSN_IMPOSSIBLE, 0); DBUG_RETURN(0); } DBUG_ASSERT((buff[PAGE_TYPE_OFFSET] & PAGE_TYPE_MASK) == (uchar) page_type); if (delete_dir_entry(buff, block_size, rownr, &empty_space) < 0) + { + my_errno= HA_ERR_WRONG_IN_RECORD; goto err; + } page_link.unlock= PAGECACHE_LOCK_WRITE_UNLOCK; page_link.changed= 1; @@ -5404,11 +5987,13 @@ uint _ma_apply_redo_purge_row_head_or_tail(MARIA_HA *info, LSN lsn, DBUG_RETURN(result); err: + error= my_errno; pagecache_unlock_by_link(share->pagecache, page_link.link, PAGECACHE_LOCK_WRITE_UNLOCK, PAGECACHE_UNPIN, LSN_IMPOSSIBLE, LSN_IMPOSSIBLE, 0); - DBUG_RETURN(HA_ERR_WRONG_IN_RECORD); + _ma_mark_file_crashed(share); + DBUG_RETURN((my_errno= error)); } @@ -5447,13 +6032,12 @@ uint _ma_apply_redo_free_blocks(MARIA_HA *info, start_page= page= page_korr(header); header+= PAGE_STORE_SIZE; /* Page range may have this bit set to indicate a tail page */ - page_range= pagerange_korr(header) & ~TAIL_BIT; + page_range= pagerange_korr(header) & ~(TAIL_BIT | START_EXTENT_BIT); DBUG_ASSERT(page_range > 0); header+= PAGERANGE_STORE_SIZE; DBUG_PRINT("info", ("page: %lu pages: %u", (long) page, page_range)); - DBUG_ASSERT((page_range & TAIL_BIT) == 0); /** @todo leave bitmap lock to the bitmap code... */ pthread_mutex_lock(&share->bitmap.bitmap_lock); @@ -5461,7 +6045,10 @@ uint _ma_apply_redo_free_blocks(MARIA_HA *info, page_range); pthread_mutex_unlock(&share->bitmap.bitmap_lock); if (res) + { + _ma_mark_file_crashed(share); DBUG_RETURN(res); + } } DBUG_RETURN(0); } @@ -5503,7 +6090,7 @@ uint _ma_apply_redo_free_head_or_tail(MARIA_HA *info, LSN lsn, PAGECACHE_LOCK_WRITE_UNLOCK, PAGECACHE_UNPIN, LSN_IMPOSSIBLE, LSN_IMPOSSIBLE, 0); - DBUG_RETURN(1); + goto err; } if (lsn_korr(buff) >= lsn) { @@ -5519,7 +6106,7 @@ uint _ma_apply_redo_free_head_or_tail(MARIA_HA *info, LSN lsn, #ifdef IDENTICAL_PAGES_AFTER_RECOVERY { uint number_of_records= (uint) buff[DIR_COUNT_OFFSET]; - uchar *dir= dir_entry_pos(buff, info->s->block_size, + uchar *dir= dir_entry_pos(buff, share->block_size, number_of_records-1); buff[DIR_FREE_OFFSET]= END_OF_DIR_FREE_LIST; bzero(dir, number_of_records * DIR_ENTRY_SIZE); @@ -5535,8 +6122,12 @@ uint _ma_apply_redo_free_head_or_tail(MARIA_HA *info, LSN lsn, res= _ma_bitmap_reset_full_page_bits(info, &share->bitmap, page, 1); pthread_mutex_unlock(&share->bitmap.bitmap_lock); if (res) - DBUG_RETURN(res); + goto err; DBUG_RETURN(0); + +err: + _ma_mark_file_crashed(share); + DBUG_RETURN(1); } @@ -5558,7 +6149,7 @@ uint _ma_apply_redo_insert_row_blobs(MARIA_HA *info, { MARIA_SHARE *share= info->s; const uchar *data; - uint data_size= FULL_PAGE_SIZE(info->s->block_size); + uint data_size= FULL_PAGE_SIZE(share->block_size); uint blob_count, ranges; DBUG_ENTER("_ma_apply_redo_insert_row_blobs"); @@ -5601,17 +6192,17 @@ uint _ma_apply_redo_insert_row_blobs(MARIA_HA *info, enum pagecache_page_pin unpin_method; uint length; - if (((page + 1) * info->s->block_size) > + if (((page + 1) * share->block_size) > info->state->data_file_length) { /* New page or half written page at end of file */ DBUG_PRINT("info", ("Enlarging data file from %lu to %lu", (ulong) info->state->data_file_length, - (ulong) ((page + 1 ) * info->s->block_size))); - info->state->data_file_length= (page + 1) * info->s->block_size; + (ulong) ((page + 1 ) * share->block_size))); + info->state->data_file_length= (page + 1) * share->block_size; buff= info->keyread_buff; info->keyread_buff_used= 1; - make_empty_page(info, buff, BLOB_PAGE); + make_empty_page(info, buff, BLOB_PAGE, 0); unlock_method= PAGECACHE_LOCK_LEFT_UNLOCKED; unpin_method= PAGECACHE_PIN_LEFT_UNPINNED; } @@ -5635,7 +6226,7 @@ uint _ma_apply_redo_insert_row_blobs(MARIA_HA *info, PAGECACHE_LOCK_WRITE_UNLOCK, PAGECACHE_UNPIN, LSN_IMPOSSIBLE, LSN_IMPOSSIBLE, 0); - DBUG_RETURN(my_errno); + goto err; } /* Physical file was too short, create new page. It can be that @@ -5644,7 +6235,7 @@ uint _ma_apply_redo_insert_row_blobs(MARIA_HA *info, length), now reads page N+1: the read fails. */ buff= pagecache_block_link_to_buffer(page_link.link); - make_empty_page(info, buff, BLOB_PAGE); + make_empty_page(info, buff, BLOB_PAGE, 0); } else { @@ -5675,7 +6266,7 @@ uint _ma_apply_redo_insert_row_blobs(MARIA_HA *info, /* Last page may be only partly filled. */ length-= empty_space; #ifdef IDENTICAL_PAGES_AFTER_RECOVERY - bzero(buff + info->s->block_size - PAGE_SUFFIX_SIZE - empty_space, + bzero(buff + share->block_size - PAGE_SUFFIX_SIZE - empty_space, empty_space); #endif } @@ -5686,7 +6277,7 @@ uint _ma_apply_redo_insert_row_blobs(MARIA_HA *info, buff, PAGECACHE_PLAIN_PAGE, unlock_method, unpin_method, PAGECACHE_WRITE_DELAY, 0, LSN_IMPOSSIBLE)) - DBUG_RETURN(my_errno); + goto err; } /** @todo leave bitmap lock to the bitmap code... */ pthread_mutex_lock(&share->bitmap.bitmap_lock); @@ -5694,10 +6285,14 @@ uint _ma_apply_redo_insert_row_blobs(MARIA_HA *info, page_range); pthread_mutex_unlock(&share->bitmap.bitmap_lock); if (res) - DBUG_RETURN(res); + goto err; } } DBUG_RETURN(0); + +err: + _ma_mark_file_crashed(share); + DBUG_RETURN(1); } @@ -5705,6 +6300,8 @@ uint _ma_apply_redo_insert_row_blobs(MARIA_HA *info, Applying of UNDO entries ****************************************************************************/ +/* Execute undo of a row insert (delete the inserted row) */ + my_bool _ma_apply_undo_row_insert(MARIA_HA *info, LSN undo_lsn, const uchar *header) { @@ -5722,23 +6319,25 @@ my_bool _ma_apply_undo_row_insert(MARIA_HA *info, LSN undo_lsn, header+= PAGE_STORE_SIZE; rownr= dirpos_korr(header); header+= DIRPOS_STORE_SIZE; - DBUG_PRINT("enter", ("Page: %lu rownr: %u", (ulong) page, rownr)); - - if (!(buff= pagecache_read(share->pagecache, - &info->dfile, page, 0, - info->buff, share->page_type, - PAGECACHE_LOCK_WRITE, - &page_link.link))) - DBUG_RETURN(1); + DBUG_PRINT("enter", ("rowid: %lu page: %lu rownr: %u", + (ulong) ma_recordpos(page, rownr), + (ulong) page, rownr)); + buff= pagecache_read(share->pagecache, + &info->dfile, page, 0, + 0, share->page_type, + PAGECACHE_LOCK_WRITE, + &page_link.link); page_link.unlock= PAGECACHE_LOCK_WRITE_UNLOCK; - page_link.changed= 1; + page_link.changed= buff != 0; push_dynamic(&info->pinned_pages, (void*) &page_link); + if (!buff) + goto err; if (read_row_extent_info(info, buff, rownr)) - DBUG_RETURN(1); + goto err; - _ma_bitmap_flushable(share, 1); + _ma_bitmap_flushable(info, 1); if (delete_head_or_tail(info, page, rownr, 1, 1) || delete_tails(info, info->cur_row.tail_positions)) goto err; @@ -5755,26 +6354,30 @@ my_bool _ma_apply_undo_row_insert(MARIA_HA *info, LSN undo_lsn, res= 0; err: - _ma_bitmap_flushable(share, -1); + if (info->non_flushable_state) + _ma_bitmap_flushable(info, -1); _ma_unpin_all_pages_and_finalize_row(info, lsn); DBUG_RETURN(res); } -/* Execute undo of a row delete (insert the row back somewhere) */ +/* Execute undo of a row delete (insert the row back where it was) */ my_bool _ma_apply_undo_row_delete(MARIA_HA *info, LSN undo_lsn, const uchar *header, - size_t header_length __attribute__((unused))) + size_t header_length + __attribute__((unused))) { - uchar *record; - const uchar *null_bits, *field_length_data; MARIA_SHARE *share= info->s; MARIA_ROW row; - uint *null_field_lengths; - ulong *blob_lengths; MARIA_COLUMNDEF *column, *end_column; - my_bool res; + MARIA_BITMAP_BLOCKS *blocks; + struct st_row_pos_info row_pos; + uchar *record; + const uchar *null_bits, *field_length_data, *extent_info; + ulonglong page; + ulong *blob_lengths; + uint *null_field_lengths, extent_count, rownr, length_on_head_page; DBUG_ENTER("_ma_apply_undo_row_delete"); /* @@ -5782,6 +6385,19 @@ my_bool _ma_apply_undo_row_delete(MARIA_HA *info, LSN undo_lsn, some buffers to point directly to 'header' */ memcpy(&row, &info->cur_row, sizeof(row)); + + page= page_korr(header); + header+= PAGE_STORE_SIZE; + rownr= dirpos_korr(header); + header+= DIRPOS_STORE_SIZE; + length_on_head_page= uint2korr(header); + header+= 2; + extent_count= pagerange_korr(header); + header+= PAGERANGE_STORE_SIZE; + DBUG_PRINT("enter", ("rowid: %lu page: %lu rownr: %u", + (ulong) ma_recordpos(page, rownr), + (ulong) page, rownr)); + if (share->calc_checksum) { /* @@ -5791,6 +6407,8 @@ my_bool _ma_apply_undo_row_delete(MARIA_HA *info, LSN undo_lsn, row.checksum= - ha_checksum_korr(header); header+= HA_CHECKSUM_STORE_SIZE; } + extent_info= header; + header+= extent_count * ROW_EXTENT_SIZE; null_field_lengths= row.null_field_lengths; blob_lengths= row.blob_lengths; @@ -5919,7 +6537,6 @@ my_bool _ma_apply_undo_row_delete(MARIA_HA *info, LSN undo_lsn, memcpy(field_pos + size_length, &header, sizeof(&header)); header+= blob_length; *blob_lengths++= blob_length; - row.blob_length+= blob_length; break; } default: @@ -5936,12 +6553,47 @@ my_bool _ma_apply_undo_row_delete(MARIA_HA *info, LSN undo_lsn, if (row.total_length < share->base.min_block_length) row.total_length= share->base.min_block_length; - /* Row is now up to date. Time to insert the record */ + /* + Row is now generated. Now we need to insert record on the original + pages with original size on each page. + */ + + _ma_bitmap_flushable(info, 1); + /* Change extent information to be usable by write_block_record() */ + blocks= &row.insert_blocks; + if (extent_to_bitmap_blocks(info, blocks, page, extent_count, extent_info)) + goto err; + blocks->block->org_bitmap_value= _ma_bitmap_get_page_bits(info, + &share->bitmap, + page); + blocks->block->used|= BLOCKUSED_USE_ORG_BITMAP; + + /* Read head page and allocate data for rowid */ + if (get_rowpos_in_head_or_tail_page(info, blocks->block, + info->buff, + length_on_head_page, + HEAD_PAGE, PAGECACHE_LOCK_WRITE, + rownr, &row_pos)) + goto err; + + if (share->calc_checksum) + { + DBUG_ASSERT(row.checksum == (share->calc_checksum)(info, record)); + } + if (write_block_record(info, (uchar*) 0, record, &row, + blocks, blocks->block->org_bitmap_value != 0, + &row_pos, undo_lsn, 0)) + goto err; + + my_free(record, MYF(0)); + DBUG_RETURN(0); - res= allocate_and_write_block_record(info, record, &row, undo_lsn); - info->cur_row.lastpos= row.lastpos; +err: + if (info->non_flushable_state) + _ma_bitmap_flushable(info, -1); + _ma_unpin_all_pages_and_finalize_row(info, LSN_IMPOSSIBLE); my_free(record, MYF(0)); - DBUG_RETURN(res); + DBUG_RETURN(1); } @@ -5957,16 +6609,17 @@ my_bool _ma_apply_undo_row_delete(MARIA_HA *info, LSN undo_lsn, my_bool _ma_apply_undo_row_update(MARIA_HA *info, LSN undo_lsn, const uchar *header, - size_t header_length __attribute__((unused))) + size_t header_length + __attribute__((unused))) { - ulonglong page; - uint rownr, field_length_header; MARIA_SHARE *share= info->s; - const uchar *field_length_data, *field_length_data_end; - uchar *current_record, *orig_record; - int error= 1; MARIA_RECORD_POS record_pos; + const uchar *field_length_data, *field_length_data_end, *extent_info; + uchar *current_record, *orig_record; + ulonglong page; ha_checksum checksum_delta; + uint rownr, field_length_header, extent_count, length_on_head_page; + int error= 1; DBUG_ENTER("_ma_apply_undo_row_update"); LINT_INIT(checksum_delta); @@ -5975,14 +6628,21 @@ my_bool _ma_apply_undo_row_update(MARIA_HA *info, LSN undo_lsn, rownr= dirpos_korr(header); header+= DIRPOS_STORE_SIZE; record_pos= ma_recordpos(page, rownr); - info->cur_row.lastpos= record_pos; /* For key insert */ - DBUG_PRINT("enter", ("Page: %lu rownr: %u", (ulong) page, rownr)); + DBUG_PRINT("enter", ("rowid: %lu page: %lu rownr: %u", + (ulong) record_pos, (ulong) page, rownr)); if (share->calc_checksum) { checksum_delta= ha_checksum_korr(header); header+= HA_CHECKSUM_STORE_SIZE; } + length_on_head_page= uint2korr(header); + header+= 2; + extent_count= pagerange_korr(header); + header+= PAGERANGE_STORE_SIZE; + extent_info= header; + header+= extent_count * ROW_EXTENT_SIZE; + /* Set header to point to old field values, generated by fill_update_undo_parts() @@ -6089,8 +6749,9 @@ my_bool _ma_apply_undo_row_update(MARIA_HA *info, LSN undo_lsn, } /* Now records are up to date, execute the update to original values */ - if (_ma_update_block_record2(info, record_pos, current_record, orig_record, - undo_lsn)) + if (_ma_update_at_original_place(info, page, rownr, length_on_head_page, + extent_count, extent_info, + current_record, orig_record, undo_lsn)) goto err; error= 0; diff --git a/storage/maria/ma_blockrec.h b/storage/maria/ma_blockrec.h index a834b4788df..b55d676fc8a 100644 --- a/storage/maria/ma_blockrec.h +++ b/storage/maria/ma_blockrec.h @@ -38,6 +38,9 @@ #define BLOCK_FILLER_SIZE 2 #define ROW_EXTENT_SIZE (ROW_EXTENT_PAGE_SIZE + ROW_EXTENT_COUNT_SIZE) #define TAIL_BIT 0x8000 /* Bit in page_count to signify tail */ +#define START_EXTENT_BIT 0x4000 /* Bit in page_count to signify start*/ +/* page_count set by bitmap code for tail pages */ +#define TAIL_PAGE_COUNT_MARKER 0xffff /* Number of extents reserved MARIA_BITMAP_BLOCKS to store head part */ #define ELEMENTS_RESERVED_FOR_MAIN_PART 4 /* This is just used to prealloc a dynamic array */ @@ -175,6 +178,7 @@ my_bool _ma_compare_block_record(register MARIA_HA *info, my_bool _ma_bitmap_init(MARIA_SHARE *share, File file); my_bool _ma_bitmap_end(MARIA_SHARE *share); my_bool _ma_bitmap_flush(MARIA_SHARE *share); +my_bool _ma_bitmap_wait_or_flush(MARIA_SHARE *share); my_bool _ma_bitmap_flush_all(MARIA_SHARE *share); void _ma_bitmap_reset_cache(MARIA_SHARE *share); my_bool _ma_bitmap_find_place(MARIA_HA *info, MARIA_ROW *row, @@ -201,9 +205,11 @@ my_bool _ma_check_if_right_bitmap_type(MARIA_HA *info, enum en_page_type page_type, ulonglong page, uint *bitmap_pattern); +uint _ma_bitmap_get_page_bits(MARIA_HA *info, MARIA_FILE_BITMAP *bitmap, + ulonglong page); void _ma_bitmap_delete_all(MARIA_SHARE *share); int _ma_bitmap_create_first(MARIA_SHARE *share); -void _ma_bitmap_flushable(MARIA_SHARE *share, int non_flushable_inc); +void _ma_bitmap_flushable(MARIA_HA *info, int non_flushable_inc); #ifndef DBUG_OFF void _ma_print_bitmap(MARIA_FILE_BITMAP *bitmap, uchar *data, ulonglong page); diff --git a/storage/maria/ma_check.c b/storage/maria/ma_check.c index c122ea6e7ba..7c1c5b82d46 100644 --- a/storage/maria/ma_check.c +++ b/storage/maria/ma_check.c @@ -41,7 +41,11 @@ */ #include "ma_ftdefs.h" -#include <myisamchk.h> +#include "ma_rt_index.h" +#include "ma_blockrec.h" +#include "trnman.h" +#include "ma_key_recover.h" + #include <stdarg.h> #include <my_getopt.h> #ifdef HAVE_SYS_VADVISE_H @@ -50,9 +54,6 @@ #ifdef HAVE_SYS_MMAN_H #include <sys/mman.h> #endif -#include "ma_rt_index.h" -#include "ma_blockrec.h" -#include "trnman_public.h" /* Functions defined in this file */ @@ -226,7 +227,7 @@ int maria_chk_del(HA_CHECK *param, register MARIA_HA *info, uint test_flag) empty+=share->base.pack_reclength; } } - if (test_flag & T_VERBOSE) + if (info->state->del && (test_flag & T_VERBOSE)) puts("\n"); if (empty != info->state->empty) { @@ -255,7 +256,8 @@ int maria_chk_del(HA_CHECK *param, register MARIA_HA *info, uint test_flag) wrong: param->testflag|=T_RETRY_WITHOUT_QUICK; - if (test_flag & T_VERBOSE) puts(""); + if (test_flag & T_VERBOSE) + puts(""); _ma_check_print_error(param,"record delete-link-chain corrupted"); DBUG_RETURN(1); } /* maria_chk_del */ @@ -273,6 +275,9 @@ static int check_k_link(HA_CHECK *param, register MARIA_HA *info, uchar *buff; DBUG_ENTER("check_k_link"); + if (next_link == HA_OFFSET_ERROR) + DBUG_RETURN(0); /* Avoid printing empty line */ + records= (ha_rows) (info->state->key_file_length / block_size); while (next_link != HA_OFFSET_ERROR && records > 0) { @@ -446,7 +451,8 @@ int maria_chk_key(HA_CHECK *param, register MARIA_HA *info) DBUG_RETURN(-1); } - if (!(param->testflag & T_SILENT)) puts("- check index reference"); + if (!(param->testflag & T_SILENT)) + puts("- check index reference"); all_keydata=all_totaldata=key_totlength=0; init_checksum=param->record_checksum; @@ -1627,7 +1633,7 @@ static my_bool check_head_page(HA_CHECK *param, MARIA_HA *info, uchar *record, { uint page, page_count, page_type; page= uint5korr(extents); - page_count= uint2korr(extents+5); + page_count= uint2korr(extents+5) & ~START_EXTENT_BIT; extents+= ROW_EXTENT_SIZE; page_type= BLOB_PAGE; if (page_count & TAIL_BIT) @@ -1635,6 +1641,11 @@ static my_bool check_head_page(HA_CHECK *param, MARIA_HA *info, uchar *record, page_count= 1; page_type= TAIL_PAGE; } + /* + TODO OPTIMIZE: + Check the whole extent with one test and only do the loop if + something is wrong (for exact error reporting) + */ for ( ; page_count--; page++) { uint bitmap_pattern; @@ -2638,8 +2649,8 @@ int maria_sort_index(HA_CHECK *param, register MARIA_HA *info, char *name) int old_lock; MARIA_SHARE *share= info->s; MARIA_STATE_INFO old_state; - myf sync_dir= (share->now_transactional && !share->temporary) ? - MY_SYNC_DIR : 0; + myf sync_dir= ((share->now_transactional && !share->temporary) ? + MY_SYNC_DIR : 0); DBUG_ENTER("maria_sort_index"); /* cannot sort index files with R-tree indexes */ @@ -3388,8 +3399,8 @@ int maria_repair_parallel(HA_CHECK *param, register MARIA_HA *info, MARIA_SORT_INFO sort_info; ulonglong key_map=share->state.key_map; pthread_attr_t thr_attr; - myf sync_dir= (share->now_transactional && !share->temporary) ? - MY_SYNC_DIR : 0; + myf sync_dir= ((share->now_transactional && !share->temporary) ? + MY_SYNC_DIR : 0); DBUG_ENTER("maria_repair_parallel"); got_error= 1; @@ -4874,6 +4885,7 @@ static int sort_insert_key(MARIA_SORT_PARAM *sort_param, key_file_length=info->state->key_file_length; if ((filepos= _ma_new(info, DFLT_INIT_HITS, &page_link)) == HA_OFFSET_ERROR) DBUG_RETURN(1); + _ma_fast_unlock_key_del(info); /* If we read the page from the key cache, we have to write it back to it */ if (page_link->changed) @@ -4997,7 +5009,7 @@ int _ma_flush_pending_blocks(MARIA_SORT_PARAM *sort_param) bzero(key_block->buff+length, keyinfo->block_length-length); if ((filepos= _ma_new(info, DFLT_INIT_HITS, &page_link)) == HA_OFFSET_ERROR) - DBUG_RETURN(1); + goto err; /* If we read the page from the key cache, we have to write it back */ if (page_link->changed) @@ -5006,20 +5018,25 @@ int _ma_flush_pending_blocks(MARIA_SORT_PARAM *sort_param) if (_ma_write_keypage(info, keyinfo, filepos, PAGECACHE_LOCK_WRITE_UNLOCK, DFLT_INIT_HITS, key_block->buff)) - DBUG_RETURN(1); + goto err; } else { put_crc(key_block->buff, filepos, info->s); if (my_pwrite(info->s->kfile.file, key_block->buff, (uint) keyinfo->block_length,filepos, myf_rw)) - DBUG_RETURN(1); + goto err; } DBUG_DUMP("buff",key_block->buff,length); nod_flag=1; } info->s->state.key_root[sort_param->key]=filepos; /* Last is root for tree */ + _ma_fast_unlock_key_del(info); DBUG_RETURN(0); + +err: + _ma_fast_unlock_key_del(info); + DBUG_RETURN(1); } /* _ma_flush_pending_blocks */ /* alloc space and pointers for key_blocks */ diff --git a/storage/maria/ma_close.c b/storage/maria/ma_close.c index f058754c0ad..1f8b5880115 100644 --- a/storage/maria/ma_close.c +++ b/storage/maria/ma_close.c @@ -32,6 +32,9 @@ int maria_close(register MARIA_HA *info) (long) info, (uint) share->reopen, (uint) share->tot_locks)); + /* Check that we have unlocked key delete-links properly */ + DBUG_ASSERT(info->used_key_del == 0); + pthread_mutex_lock(&THR_LOCK_maria); if (info->lock_type == F_EXTRA_LCK) info->lock_type=F_UNLCK; /* HA_EXTRA_NO_USER_CHANGE */ diff --git a/storage/maria/ma_create.c b/storage/maria/ma_create.c index 767242ec027..83457847d5b 100644 --- a/storage/maria/ma_create.c +++ b/storage/maria/ma_create.c @@ -1339,6 +1339,7 @@ int _ma_update_create_rename_lsn_sub(MARIA_SHARE *share, translog_deassign_id_from_share(share); } return my_pwrite(file, buf, sizeof(buf), - sizeof(share->state.header) + 2, MYF(MY_NABP)) || + sizeof(share->state.header) + + MARIA_FILE_CREATE_RENAME_LSN_OFFSET, MYF(MY_NABP)) || (do_sync && my_sync(file, MYF(0))); } diff --git a/storage/maria/ma_delete.c b/storage/maria/ma_delete.c index 67d3d8d7092..bbe56432e87 100644 --- a/storage/maria/ma_delete.c +++ b/storage/maria/ma_delete.c @@ -192,6 +192,9 @@ int _ma_ck_delete(register MARIA_HA *info, uint keynr, uchar *key, log_type= LOGREC_UNDO_KEY_DELETE_WITH_ROOT; } + /* Log also position to row */ + key_length+= share->rec_reflength; + /* Note that for delete key, we don't log the reference to the record. This is because the row may be inserted at a different place when @@ -652,12 +655,12 @@ static int del(register MARIA_HA *info, MARIA_KEYDEF *keyinfo, new_leaf_length)) goto err; + leaf_page_link->changed= 1; /* Safety */ if (new_leaf_length <= (info->quick_mode ? MARIA_MIN_KEYBLOCK_LENGTH : (uint) keyinfo->underflow_block_length)) { /* Underflow, leaf_page will be written by caller */ ret_value= 1; - leaf_page_link->changed= 1; /* Safety */ } else { diff --git a/storage/maria/ma_extra.c b/storage/maria/ma_extra.c index 26e129245d6..ee1439e752a 100644 --- a/storage/maria/ma_extra.c +++ b/storage/maria/ma_extra.c @@ -380,13 +380,6 @@ int maria_extra(MARIA_HA *info, enum ha_extra_function function, maria_mark_crashed(info); /* Fatal error found */ } } - if (share->base.blobs && info->rec_buff_size > - share->base.default_rec_buff_size) - { - info->rec_buff_size= 1; /* Force realloc */ - _ma_alloc_buffer(&info->rec_buff, &info->rec_buff_size, - share->base.default_rec_buff_size); - } break; case HA_EXTRA_NORMAL: /* Theese isn't in use */ info->quick_mode= 0; @@ -489,13 +482,22 @@ int maria_reset(MARIA_HA *info) info->opt_flag&= ~(READ_CACHE_USED | WRITE_CACHE_USED); error= end_io_cache(&info->rec_cache); } - if (share->base.blobs && info->rec_buff_size > - share->base.default_rec_buff_size) + /* Free memory used for keeping blobs */ + if (share->base.blobs) + { + if (info->rec_buff_size > share->base.default_rec_buff_size) { info->rec_buff_size= 1; /* Force realloc */ _ma_alloc_buffer(&info->rec_buff, &info->rec_buff_size, share->base.default_rec_buff_size); } + if (info->blob_buff_size > MARIA_SMALL_BLOB_BUFFER) + { + info->blob_buff_size= 1; /* Force realloc */ + _ma_alloc_buffer(&info->blob_buff, &info->blob_buff_size, + MARIA_SMALL_BLOB_BUFFER); + } + } #if defined(HAVE_MMAP) && defined(HAVE_MADVISE) if (info->opt_flag & MEMMAP_USED) madvise((char*) share->file_map, share->state.state.data_file_length, diff --git a/storage/maria/ma_key_recover.c b/storage/maria/ma_key_recover.c index 071c49661ef..c46dae81853 100644 --- a/storage/maria/ma_key_recover.c +++ b/storage/maria/ma_key_recover.c @@ -33,8 +33,8 @@ undo (like on duplicate key errors) @note - We unpin pages in the reverse order as they where pinned; This may not - be strictly necessary but may simplify things in the future. + We unpin pages in the reverse order as they where pinned; This is not + necessary now, but may simplify things in the future. @return @retval 0 ok @@ -54,8 +54,15 @@ void _ma_unpin_all_pages(MARIA_HA *info, LSN undo_lsn) while (pinned_page-- != page_link) { + /* + Note this assert fails if we got a disk error or the record file + is corrupted, which means we should have this enabled only in debug + builds. + */ +#ifdef EXTRA_DEBUG DBUG_ASSERT(!pinned_page->changed || undo_lsn != LSN_IMPOSSIBLE || !info->s->now_transactional); +#endif pagecache_unlock_by_link(info->s->pagecache, pinned_page->link, pinned_page->unlock, PAGECACHE_UNPIN, info->trn->rec_lsn, undo_lsn, @@ -595,6 +602,7 @@ uint _ma_apply_redo_index_new_page(MARIA_HA *info, LSN lsn, else if (lsn_korr(buff) >= lsn) { /* Already applied */ + DBUG_PRINT("info", ("Page is up to date, skipping redo")); result= 0; goto err; } @@ -603,6 +611,7 @@ uint _ma_apply_redo_index_new_page(MARIA_HA *info, LSN lsn, } /* Write modified page */ + bzero(buff, LSN_STORE_SIZE); memcpy(buff + LSN_STORE_SIZE, header, length); bzero(buff + LSN_STORE_SIZE + length, share->block_size - LSN_STORE_SIZE - KEYPAGE_CHECKSUM_SIZE - length); @@ -772,6 +781,7 @@ uint _ma_apply_redo_index(MARIA_HA *info, if (lsn_korr(buff) >= lsn) { /* Already applied */ + DBUG_PRINT("info", ("Page is up to date, skipping redo")); result= 0; goto err; } @@ -905,6 +915,8 @@ err: PAGECACHE_LOCK_WRITE_UNLOCK, PAGECACHE_UNPIN, LSN_IMPOSSIBLE, LSN_IMPOSSIBLE, 0); + if (result) + _ma_mark_file_crashed(share); DBUG_RETURN(result); } @@ -941,7 +953,8 @@ my_bool _ma_apply_undo_key_insert(MARIA_HA *info, LSN undo_lsn, new_root= share->state.key_root[keynr]; res= _ma_ck_real_delete(info, share->keyinfo+keynr, key, length - share->rec_reflength, &new_root); - + if (res) + _ma_mark_file_crashed(share); msg.root= &share->state.key_root[keynr]; msg.value= new_root; msg.keynr= keynr; @@ -951,6 +964,7 @@ my_bool _ma_apply_undo_key_insert(MARIA_HA *info, LSN undo_lsn, 0, 0, &lsn, (void*) &msg)) res= 1; + _ma_fast_unlock_key_del(info); _ma_unpin_all_pages_and_finalize_row(info, lsn); DBUG_RETURN(res); } @@ -979,14 +993,15 @@ my_bool _ma_apply_undo_key_delete(MARIA_HA *info, LSN undo_lsn, /* We have to copy key as _ma_ck_real_write_btree() may change it */ memcpy(key, header + KEY_NR_STORE_SIZE, length); - _ma_dpointer(info, key + length, info->cur_row.lastpos); - DBUG_DUMP("key", key, length + share->rec_reflength); + DBUG_DUMP("key", key, length); new_root= share->state.key_root[keynr]; res= _ma_ck_real_write_btree(info, share->keyinfo+keynr, key, - length, + length - share->rec_reflength, &new_root, share->keyinfo[keynr].write_comp_flag); + if (res) + _ma_mark_file_crashed(share); msg.root= &share->state.key_root[keynr]; msg.value= new_root; @@ -998,6 +1013,7 @@ my_bool _ma_apply_undo_key_delete(MARIA_HA *info, LSN undo_lsn, (void*) &msg)) res= 1; + _ma_fast_unlock_key_del(info); _ma_unpin_all_pages_and_finalize_row(info, lsn); DBUG_RETURN(res); } diff --git a/storage/maria/ma_locking.c b/storage/maria/ma_locking.c index a25820b81fb..6a904e36ea2 100644 --- a/storage/maria/ma_locking.c +++ b/storage/maria/ma_locking.c @@ -497,9 +497,7 @@ int _ma_test_if_changed(register MARIA_HA *info) /* Put a mark in the .MYI file that someone is updating the table - DOCUMENTATION - state.open_count in the .MYI file is used the following way: - For the first change of the .MYI file in this process open_count is incremented by _ma_mark_file_changed(). (We have a write lock on the file @@ -540,7 +538,8 @@ int _ma_mark_file_changed(MARIA_HA *info) mi_int2store(buff,share->state.open_count); buff[2]=1; /* Mark that it's changed */ DBUG_RETURN(my_pwrite(share->kfile.file, buff, sizeof(buff), - sizeof(share->state.header), + sizeof(share->state.header) + + MARIA_FILE_OPEN_COUNT_OFFSET, MYF(MY_NABP))); } } @@ -571,8 +570,9 @@ int _ma_decrement_open_count(MARIA_HA *info) { mi_int2store(buff,share->state.open_count); write_error= my_pwrite(share->kfile.file, buff, sizeof(buff), - sizeof(share->state.header), - MYF(MY_NABP)); + sizeof(share->state.header) + + MARIA_FILE_OPEN_COUNT_OFFSET, + MYF(MY_NABP)); } } if (!lock_error) @@ -580,3 +580,19 @@ int _ma_decrement_open_count(MARIA_HA *info) } return test(lock_error || write_error); } + + +/** @brief mark file as crashed */ + +int _ma_mark_file_crashed(MARIA_SHARE *share) +{ + uchar buff[2]; + DBUG_ENTER("_ma_mark_file_crashed"); + + share->state.changed|= STATE_CRASHED; + mi_int2store(buff, share->state.changed); + DBUG_RETURN(my_pwrite(share->kfile.file, buff, sizeof(buff), + sizeof(share->state.header) + + MARIA_FILE_CHANGED_OFFSET, + MYF(MY_NABP))); +} diff --git a/storage/maria/ma_loghandler.c b/storage/maria/ma_loghandler.c index 2a176d14454..9030deb4b73 100644 --- a/storage/maria/ma_loghandler.c +++ b/storage/maria/ma_loghandler.c @@ -7347,7 +7347,7 @@ LSN translog_first_lsn_in_log() uint16 chunk_offset; uchar *page; DBUG_ENTER("translog_first_lsn_in_log"); - DBUG_PRINT("info", ("Horizon: (%lu,0x%lx)", LSN_IN_PARTS(addr))); + DBUG_PRINT("info", ("Horizon: (%lu,0x%lx)", LSN_IN_PARTS(horizon))); DBUG_ASSERT(translog_status == TRANSLOG_OK || translog_status == TRANSLOG_READONLY); diff --git a/storage/maria/ma_open.c b/storage/maria/ma_open.c index 54f23da4ec9..cdf3848035c 100644 --- a/storage/maria/ma_open.c +++ b/storage/maria/ma_open.c @@ -1124,6 +1124,9 @@ uint _ma_state_info_write_sub(File file, MARIA_STATE_INFO *state, uint pWrite) /* open_count must be first because of _ma_mark_file_changed ! */ mi_int2store(ptr,state->open_count); ptr+= 2; + /* changed must be second, because of _ma_mark_file_crashed */ + mi_int2store(ptr,state->changed); ptr+= 2; + /* if you change the offset of create_rename_lsn/is_of_horizon inside the index file's header, fix ma_create + ma_rename + ma_delete_all + @@ -1131,8 +1134,6 @@ uint _ma_state_info_write_sub(File file, MARIA_STATE_INFO *state, uint pWrite) */ lsn_store(ptr, state->create_rename_lsn); ptr+= LSN_STORE_SIZE; lsn_store(ptr, state->is_of_horizon); ptr+= LSN_STORE_SIZE; - *ptr++= (uchar)state->changed; - *ptr++= state->sortkey; mi_rowstore(ptr,state->state.records); ptr+= 8; mi_rowstore(ptr,state->state.del); ptr+= 8; mi_rowstore(ptr,state->split); ptr+= 8; @@ -1148,7 +1149,8 @@ uint _ma_state_info_write_sub(File file, MARIA_STATE_INFO *state, uint pWrite) mi_int4store(ptr,state->unique); ptr+= 4; mi_int4store(ptr,state->status); ptr+= 4; mi_int4store(ptr,state->update_count); ptr+= 4; - + *ptr++= state->sortkey; + *ptr++= 0; /* Reserved */ ptr+= state->state_diff_length; for (i=0; i < keys; i++) @@ -1194,10 +1196,9 @@ static uchar *_ma_state_info_read(uchar *ptr, MARIA_STATE_INFO *state) key_parts= mi_uint2korr(state->header.key_parts); state->open_count = mi_uint2korr(ptr); ptr+= 2; + state->changed= mi_uint2korr(ptr); ptr+= 2; state->create_rename_lsn= lsn_korr(ptr); ptr+= LSN_STORE_SIZE; state->is_of_horizon= lsn_korr(ptr); ptr+= LSN_STORE_SIZE; - state->changed= (my_bool) *ptr++; - state->sortkey= (uint) *ptr++; state->state.records= mi_rowkorr(ptr); ptr+= 8; state->state.del = mi_rowkorr(ptr); ptr+= 8; state->split = mi_rowkorr(ptr); ptr+= 8; @@ -1215,6 +1216,8 @@ static uchar *_ma_state_info_read(uchar *ptr, MARIA_STATE_INFO *state) state->unique = mi_uint4korr(ptr); ptr+= 4; state->status = mi_uint4korr(ptr); ptr+= 4; state->update_count=mi_uint4korr(ptr); ptr+= 4; + state->sortkey= (uint) *ptr++; + ptr++; /* reserved */ ptr+= state->state_diff_length; diff --git a/storage/maria/ma_page.c b/storage/maria/ma_page.c index 863f3eede3f..cbd6ddcae76 100644 --- a/storage/maria/ma_page.c +++ b/storage/maria/ma_page.c @@ -121,10 +121,11 @@ int _ma_write_keypage(register MARIA_HA *info, /* Verify that keynr is correct */ DBUG_ASSERT(_ma_get_keynr(share, buff) == keyinfo->key_nr); -#if defined(EXTRA_DEBUG) && defined(HAVE_purify) +#if defined(EXTRA_DEBUG) && defined(HAVE_purify) && defined(NOT_ANYMORE) { /* This is here to catch uninitialized bytes */ - ulong crc= my_checksum(0, buff, block_size - KEYPAGE_CHECKSUM_SIZE); + uint length= _ma_get_page_used(share, buff); + ulong crc= my_checksum(0, buff, length); int4store(buff + block_size - KEYPAGE_CHECKSUM_SIZE, crc); } #endif @@ -309,10 +310,6 @@ my_off_t _ma_new(register MARIA_HA *info, int level, else { uchar *buff; - /* - TODO: replace PAGECACHE_PLAIN_PAGE with PAGECACHE_LSN_PAGE when - LSN on the pages will be implemented - */ pos= share->current_key_del; /* Protected */ DBUG_ASSERT(share->pagecache->block_size == block_size); if (!(buff= pagecache_read(share->pagecache, diff --git a/storage/maria/ma_pagecache.c b/storage/maria/ma_pagecache.c index 1a0f466c532..614763fe6ef 100755 --- a/storage/maria/ma_pagecache.c +++ b/storage/maria/ma_pagecache.c @@ -625,7 +625,7 @@ static uint pagecache_fwrite(PAGECACHE *pagecache, } DBUG_RETURN(my_pwrite(filedesc->file, buffer, pagecache->block_size, - (pageno)<<(pagecache->shift), flags)); + ((my_off_t) pageno << pagecache->shift), flags)); } @@ -642,7 +642,7 @@ static uint pagecache_fwrite(PAGECACHE *pagecache, */ #define pagecache_fread(pagecache, filedesc, buffer, pageno, flags) \ my_pread((filedesc)->file, buffer, pagecache->block_size, \ - (pageno)<<(pagecache->shift), flags) + ((my_off_t) pageno << pagecache->shift), flags) /** diff --git a/storage/maria/ma_recovery.c b/storage/maria/ma_recovery.c index 03a0cca5bf3..64517b14caa 100644 --- a/storage/maria/ma_recovery.c +++ b/storage/maria/ma_recovery.c @@ -60,7 +60,8 @@ static my_bool skip_DDLs; /**< if REDO phase should skip DDL records */ static my_bool checkpoint_useful; static my_bool procent_printed; static ulonglong now; /**< for tracking execution time of phases */ -uint warnings; /**< count of warnings */ +static int (*save_error_handler_hook)(uint, const char *,myf); +static uint recovery_warnings; /**< count of warnings */ #define prototype_redo_exec_hook(R) \ static int exec_REDO_LOGREC_ ## R(const TRANSLOG_HEADER_BUFFER *rec) @@ -194,6 +195,20 @@ void eprint(FILE *trace_file __attribute__ ((unused)), } +/* Hook to ensure we get nicer output if we get an error */ + +int maria_recover_error_handler_hook(uint error, const char *str, + myf flags) +{ + if (procent_printed) + { + procent_printed= 0; + fputc('\n', stderr); + fflush(stderr); + } + return (*save_error_handler_hook)(error, str, flags); +} + #define ALERT_USER() DBUG_ASSERT(0) static void print_preamble() @@ -289,7 +304,7 @@ int maria_apply_log(LSN from_lsn, enum maria_apply_log_way apply, DBUG_ASSERT(apply == MARIA_LOG_APPLY || !should_run_undo_phase); DBUG_ASSERT(!maria_multi_threaded); - warnings= 0; + recovery_warnings= 0; /* checkpoints can happen only if TRNs have been built */ DBUG_ASSERT(should_run_undo_phase || !take_checkpoints); all_active_trans= (struct st_trn_for_recovery *) @@ -298,6 +313,10 @@ int maria_apply_log(LSN from_lsn, enum maria_apply_log_way apply, all_tables= (struct st_table_for_recovery *) my_malloc((SHARE_ID_MAX + 1) * sizeof(struct st_table_for_recovery), MYF(MY_ZEROFILL)); + + save_error_handler_hook= error_handler_hook; + error_handler_hook= maria_recover_error_handler_hook; + if (!all_active_trans || !all_tables) goto err; @@ -390,9 +409,9 @@ int maria_apply_log(LSN from_lsn, enum maria_apply_log_way apply, } else if (uncommitted_trans > 0) { - tprint(tracef, "***WARNING: %u uncommitted transactions; some tables may" + eprint(tracef, "***WARNING: %u uncommitted transactions; some tables may" " be left inconsistent!***\n", uncommitted_trans); - warnings++; + recovery_warnings++; } old_now= now; @@ -435,6 +454,7 @@ err: error= 1; tprint(tracef, "\nRecovery of tables with transaction logs FAILED\n"); end: + error_handler_hook= save_error_handler_hook; hash_free(&all_dirty_pages); bzero(&all_dirty_pages, sizeof(all_dirty_pages)); my_free(dirty_pages_pool, MYF(MY_ALLOW_ZERO_PTR)); @@ -447,10 +467,14 @@ end: log_record_buffer.str= NULL; log_record_buffer.length= 0; ma_checkpoint_end(); - *warnings_count= warnings; + *warnings_count= recovery_warnings; if (recovery_message_printed != REC_MSG_NONE) { - fprintf(stderr, "\n"); + if (procent_printed) + { + procent_printed= 0; + fprintf(stderr, "\n"); + } if (!error) ma_message_no_user(ME_JUST_INFO, "recovery done"); } @@ -492,7 +516,8 @@ static int display_and_apply_record(const LOG_DESC *log_desc, return 1; } if ((error= (*log_desc->record_execute_in_redo_phase)(rec))) - eprint(tracef, "Got error %d when executing record\n", my_errno); + eprint(tracef, "Got error %d when executing record %s\n", + my_errno, log_desc->name); return error; } @@ -602,7 +627,7 @@ prototype_redo_exec_hook(INCOMPLETE_LOG) " about insertion of data by ALTER TABLE and CREATE SELECT," " as they are not necessary for recovery;" " present applying of log records may well not work.***\n"); - warnings++; + recovery_warnings++; return 0; } @@ -704,9 +729,11 @@ prototype_redo_exec_hook(REDO_CREATE_TABLE) kfile_header= (uchar *)ptr; ptr+= kfile_size_before_extension; /* set create_rename_lsn (for maria_read_log to be idempotent) */ - lsn_store(kfile_header + sizeof(info->s->state.header) + 2, rec->lsn); + lsn_store(kfile_header + sizeof(info->s->state.header) + + MARIA_FILE_CREATE_RENAME_LSN_OFFSET, rec->lsn); /* we also set is_of_horizon, like maria_create() does */ - lsn_store(kfile_header + sizeof(info->s->state.header) + 2 + LSN_STORE_SIZE, + lsn_store(kfile_header + sizeof(info->s->state.header) + + MARIA_FILE_CREATE_RENAME_LSN_OFFSET + LSN_STORE_SIZE, rec->lsn); data_file_name= ptr; ptr+= strlen(data_file_name) + 1; @@ -725,7 +752,7 @@ prototype_redo_exec_hook(REDO_CREATE_TABLE) MY_APPEND_EXT); linkname_ptr= NULL; create_flag= MY_DELETE_OLD; - tprint(tracef, "Table '%s' creating as '%s'", name, filename); + tprint(tracef, "Table '%s' creating as '%s'\n", name, filename); if ((kfile= my_create_with_symlink(linkname_ptr, filename, 0, create_mode, MYF(MY_WME|create_flag))) < 0) { @@ -768,7 +795,6 @@ prototype_redo_exec_hook(REDO_CREATE_TABLE) } error= 0; end: - tprint(tracef, "\n"); if (kfile >= 0) error|= my_close(kfile, MYF(MY_WME)); if (info != NULL) @@ -1615,7 +1641,8 @@ prototype_redo_exec_hook(UNDO_ROW_DELETE) { uchar buff[HA_CHECKSUM_STORE_SIZE]; if (translog_read_record(rec->lsn, LSN_STORE_SIZE + FILEID_STORE_SIZE + - PAGE_STORE_SIZE + DIRPOS_STORE_SIZE, + PAGE_STORE_SIZE + DIRPOS_STORE_SIZE + 2 + + PAGERANGE_STORE_SIZE, HA_CHECKSUM_STORE_SIZE, buff, NULL) != HA_CHECKSUM_STORE_SIZE) { @@ -1780,25 +1807,6 @@ prototype_redo_exec_hook(COMMIT) return 0; } - -/* - Set position for next active record that will have key inserted -*/ - -static void set_lastpos(MARIA_HA *info, uchar *pos) -{ - ulonglong page; - uint dir_entry; - - /* If we have checksum, it's before rowid */ - if (info->s->calc_checksum) - pos+= HA_CHECKSUM_STORE_SIZE; - page= page_korr(pos); - dir_entry= dirpos_korr(pos + PAGE_STORE_SIZE); - info->cur_row.lastpos= ma_recordpos(page, dir_entry); -} - - prototype_redo_exec_hook(CLR_END) { MARIA_HA *info= get_MARIA_HA_from_UNDO_record(rec); @@ -1841,7 +1849,6 @@ prototype_redo_exec_hook(CLR_END) case LOGREC_UNDO_ROW_DELETE: row_entry= 1; share->state.state.records++; - set_lastpos(info, logpos); break; case LOGREC_UNDO_ROW_INSERT: share->state.state.records--; @@ -1850,7 +1857,6 @@ prototype_redo_exec_hook(CLR_END) break; case LOGREC_UNDO_ROW_UPDATE: row_entry= 1; - set_lastpos(info, logpos); break; case LOGREC_UNDO_KEY_INSERT: case LOGREC_UNDO_KEY_DELETE: @@ -1874,18 +1880,6 @@ prototype_redo_exec_hook(CLR_END) share->state.state.checksum+= ha_checksum_korr(logpos); share->state.changed|= STATE_CHANGED | STATE_NOT_ANALYZED; } - else - { - /* We must set lastpos for upcoming undo delete keys */ - switch (undone_record_type) { - case LOGREC_UNDO_ROW_DELETE: - case LOGREC_UNDO_ROW_UPDATE: - set_lastpos(info, logpos); - break; - default: - break; - } - } if (row_entry) tprint(tracef, " rows' count %lu\n", (ulong)share->state.state.records); _ma_unpin_all_pages(info, rec->lsn); @@ -1971,17 +1965,11 @@ prototype_undo_exec_hook(UNDO_ROW_DELETE) } info->trn= trn; - /* - For now we skip the page and directory entry. This is to be used - later when we mark rows as deleted. - */ error= _ma_apply_undo_row_delete(info, previous_undo_lsn, log_record_buffer.str + LSN_STORE_SIZE + - FILEID_STORE_SIZE + PAGE_STORE_SIZE + - DIRPOS_STORE_SIZE, + FILEID_STORE_SIZE, rec->record_length - - (LSN_STORE_SIZE + FILEID_STORE_SIZE + - PAGE_STORE_SIZE + DIRPOS_STORE_SIZE)); + (LSN_STORE_SIZE + FILEID_STORE_SIZE)); info->trn= 0; tprint(tracef, " rows' count %lu\n undo_lsn now LSN (%lu,0x%lx)\n", (ulong)share->state.state.records, LSN_IN_PARTS(previous_undo_lsn)); @@ -2478,6 +2466,7 @@ static uint end_of_redo_phase(my_bool prepare_for_undo_phase) static int run_undo_phase(uint uncommitted) { + LSN last_undo; DBUG_ENTER("run_undo_phase"); if (uncommitted > 0) @@ -2491,6 +2480,7 @@ static int run_undo_phase(uint uncommitted) recovery_message_printed= REC_MSG_UNDO; } tprint(tracef, "%u transactions will be rolled back\n", uncommitted); + procent_printed= 1; for( ; ; ) { char llbuf[22]; @@ -2503,12 +2493,16 @@ static int run_undo_phase(uint uncommitted) DBUG_ASSERT(trn != NULL); llstr(trn->trid, llbuf); tprint(tracef, "Rolling back transaction of long id %s\n", llbuf); + last_undo= trn->undo_lsn + 1; /* Execute all undo entries */ while (trn->undo_lsn) { TRANSLOG_HEADER_BUFFER rec; LOG_DESC *log_desc; + DBUG_ASSERT(trn->undo_lsn < last_undo); + last_undo= trn->undo_lsn; + if (translog_read_record_header(trn->undo_lsn, &rec) == RECHEADER_READ_ERROR) DBUG_RETURN(1); @@ -2516,7 +2510,8 @@ static int run_undo_phase(uint uncommitted) display_record_position(log_desc, &rec, 0); if (log_desc->record_execute_in_undo_phase(&rec, trn)) { - tprint(tracef, "Got error %d when executing undo\n", my_errno); + eprint(tracef, "Got error %d when executing undo %s\n", my_errno, + log_desc->name); DBUG_RETURN(1); } } @@ -2527,6 +2522,7 @@ static int run_undo_phase(uint uncommitted) /* In the future, we want to have this phase *online* */ } } + procent_printed= 0; DBUG_RETURN(0); } diff --git a/storage/maria/ma_recovery.h b/storage/maria/ma_recovery.h index f44891a36df..56d75f16dde 100644 --- a/storage/maria/ma_recovery.h +++ b/storage/maria/ma_recovery.h @@ -16,7 +16,6 @@ /* WL#3072 Maria recovery First version written by Guilhem Bichot on 2006-04-27. - Does not compile yet. */ /* This is the interface of this module. */ diff --git a/storage/maria/ma_static.c b/storage/maria/ma_static.c index 33f6e9f9fbe..169dd7f1348 100644 --- a/storage/maria/ma_static.c +++ b/storage/maria/ma_static.c @@ -25,7 +25,7 @@ LIST *maria_open_list=0; uchar maria_file_magic[]= -{ (uchar) 254, (uchar) 254, (uchar) 9, '\001', }; +{ (uchar) 254, (uchar) 254, (uchar) 9, '\002', }; uchar maria_pack_file_magic[]= { (uchar) 254, (uchar) 254, (uchar) 10, '\001', }; /* Unique number for this maria instance */ diff --git a/storage/maria/ma_test2.c b/storage/maria/ma_test2.c index b196455e950..6b6a1e8079d 100644 --- a/storage/maria/ma_test2.c +++ b/storage/maria/ma_test2.c @@ -44,18 +44,17 @@ static void put_blob_in_record(uchar *blob_pos,char **blob_buffer, ulong *length); static void copy_key(MARIA_HA *info, uint inx, uchar *record, uchar *key); -static int verbose=0,testflag=0, - first_key=0,async_io=0,pagecacheing=0,write_cacheing=0,locking=0, - rec_pointer_size=0,pack_fields=1,silent=0, - opt_quick_mode=0, transactional= 0, skip_update= 0, - die_in_middle_of_transaction= 0; -static int pack_seg=HA_SPACE_PACK,pack_type=HA_PACK_KEY,remove_count=-1; +static int verbose= 0, testflag= 0, first_key= 0, async_io= 0, pagecacheing= 0; +static int write_cacheing= 0, locking= 0, rec_pointer_size= 0, pack_fields= 1; +static int silent= 0, opt_quick_mode= 0, transactional= 0, skip_update= 0; +static int die_in_middle_of_transaction= 0; +static int pack_seg= HA_SPACE_PACK, pack_type= HA_PACK_KEY, remove_count= -1; static int create_flag= 0, srand_arg= 0, checkpoint= 0; +static uint use_blob= 0, update_count= 0; static ulong pagecache_size=8192*32; static enum data_file_type record_type= DYNAMIC_RECORD; static uint keys=MARIA_KEYS,recant=1000; -static uint use_blob=0; static uint16 key1[1001],key3[5000]; static uchar record[300],record2[300],key[100],key2[100]; static uchar read_record[300],read_record2[300],read_record3[300]; @@ -357,7 +356,10 @@ int main(int argc, char *argv[]) printf("- Update\n"); if (srand_arg) srand(srand_arg); - for (i=0 ; i<recant/10 ; i++) + if (!update_count) + update_count= recant/10; + + for (i=0 ; i < update_count ; i++) { n1=rnd(1000); n2=rnd(100); n3=rnd(5000); sprintf((char*) record2,"%6d:%4d:%8d:XXX: %4d ",n1,n2,n3,update); @@ -385,9 +387,9 @@ int main(int argc, char *argv[]) { ulong blob_length; if (i & 1) - put_blob_in_record(record+blob_pos,&blob_buffer, &blob_length); + put_blob_in_record(record2+blob_pos,&blob_buffer, &blob_length); else - bmove(record+blob_pos,read_record+blob_pos,8); + bmove(record2+blob_pos, read_record+blob_pos, 4 + sizeof(char*)); } if (skip_update) continue; @@ -874,7 +876,7 @@ int main(int argc, char *argv[]) } if (maria_delete(file,read_record)) { - printf("can't delete record: %6.6s, delete_count: %d\n", + printf("can't delete record: %6.6s, delete_count: %d\n", read_record, opt_delete); maria_scan_end(file); goto err; @@ -1036,7 +1038,7 @@ static void get_options(int argc, char **argv) case 'L': locking=1; break; - case 'A': /* use asyncron io */ + case 'a': /* use asyncron io */ async_io=1; if (*++pos) my_default_record_cache_size=atoi(pos); @@ -1098,9 +1100,14 @@ static void get_options(int argc, char **argv) case 'T': transactional= 1; break; - case 'u': + case 'A': die_in_middle_of_transaction= atoi(++pos); break; + case 'u': + update_count=atoi(++pos); + if (!update_count) + skip_update= 1; + break; case 'q': opt_quick_mode=1; break; @@ -1116,8 +1123,8 @@ static void get_options(int argc, char **argv) case '?': case 'I': case 'V': - printf("%s Ver 1.0 for %s at %s\n",progname,SYSTEM_TYPE,MACHINE_TYPE); - puts("By Monty, for your professional use\n"); + printf("%s Ver 1.1 for %s at %s\n",progname,SYSTEM_TYPE,MACHINE_TYPE); + puts("By Monty, for testing Maria\n"); printf("Usage: %s [-?AbBcDIKLPRqSsTVWltv] [-k#] [-f#] [-m#] [-e#] [-E#] [-t#]\n", progname); exit(0); @@ -1151,7 +1158,9 @@ static void fix_length(uchar *rec, uint length) } /* fix_length */ - /* Put maybe a blob in record */ +/* Put maybe a blob in record */ + +static int first_entry; static void put_blob_in_record(uchar *blob_pos, char **blob_buffer, ulong *blob_length) @@ -1160,15 +1169,21 @@ static void put_blob_in_record(uchar *blob_pos, char **blob_buffer, *blob_length= 0; if (use_blob) { + if (! *blob_buffer && + !(*blob_buffer=my_malloc((uint) use_blob,MYF(MY_WME)))) + { + use_blob= 0; + return; + } if (rnd(10) == 0) { - if (! *blob_buffer && - !(*blob_buffer=my_malloc((uint) use_blob,MYF(MY_WME)))) + if (first_entry++ == 0) { - use_blob=0; - return; + /* Ensure we have at least one blob of max length in file */ + length= use_blob; } - length=rnd(use_blob); + else + length=rnd(use_blob); for (i=0 ; i < length ; i++) (*blob_buffer)[i]=(char) (length+i); int4store(blob_pos,length); diff --git a/storage/maria/ma_test_all.sh b/storage/maria/ma_test_all.sh index 0f3b23f09a7..b81c9ef0045 100755 --- a/storage/maria/ma_test_all.sh +++ b/storage/maria/ma_test_all.sh @@ -107,6 +107,11 @@ run_tests() $maria_path/maria_chk$suffix -sm test2 $maria_path/ma_test2$suffix $silent -L -K -W -P -A $row_type $maria_path/maria_chk$suffix -sm test2 + $maria_path/ma_test2$suffix $silent -L -K -W -P -b32768 $row_type + $maria_path/maria_chk$suffix -sm test2 + $maria_path/ma_test2 -s -L -K -W -P -M -T -c -b32768 -t4 -m300 + $maria_path/maria_chk$suffix -sm test2 + $maria_path/ma_test2$suffix $silent -L -K -P -R3 -m50 -b1000000 $row_type $maria_path/maria_chk$suffix -sm test2 $maria_path/ma_test2$suffix $silent -L -B $row_type @@ -239,13 +244,13 @@ $maria_path/maria_chk$suffix -ssm test2 # /bin/sh $maria_path/ma_test_recovery -# +#x1 # Extra tests that has caused failures in the past # # Problem with re-executing CLR's rm -f maria_log.* maria_log_control -$maria_path/ma_test2 -s -L -K -W -P -M -T -c -b -t2 -u1 +$maria_path/ma_test2 -s -L -K -W -P -M -T -c -b -t2 -A1 cp maria_log_control tmp $maria_path/maria_read_log -a -s $maria_path/maria_chk -s -e test2 @@ -253,15 +258,29 @@ cp tmp/maria_log_control . rm test2.MA? $maria_path/maria_read_log -a -s $maria_path/maria_chk -s -e test2 +rm test2.MA? # Problem with re-executing CLR's rm -f maria_log.* maria_log_control -$maria_path/ma_test2 -s -L -K -W -P -M -T -c -b -t2 -u1 +$maria_path/ma_test2 -s -L -K -W -P -M -T -c -b -t2 -A1 $maria_path/maria_read_log -a -s $maria_path/maria_chk -s -e test2 rm test2.MA? $maria_path/maria_read_log -a -s $maria_path/maria_chk -e -s test2 +rm test2.MA? + +# Problem with re-executing clrs: +rm -f maria_log.* maria_log_control +ma_test2 -s -L -K -W -P -M -T -c -b32768 -t4 -A1 +maria_read_log -a -s +maria_chk -es test2 +maria_read_log -a -s +maria_chk -es test2 +rm test2.MA? +maria_read_log -a -s +maria_chk -es test2 +rm test2.MA? # # Some timing tests diff --git a/storage/maria/ma_test_big.sh b/storage/maria/ma_test_big.sh new file mode 100644 index 00000000000..6419d05e3a4 --- /dev/null +++ b/storage/maria/ma_test_big.sh @@ -0,0 +1,22 @@ +#!/bin/sh +# +# This tests is good to find bugs in the redo/undo handling and in +# finding bugs in blob handling +# + +set -e +a=15 +while test $a -le 5000 +do + echo $a + rm -f maria_log* + ma_test2 -s -L -K -W -P -M -T -c -b32768 -t4 -A1 -m$a > /dev/null + maria_read_log -a -s >& /dev/null + maria_chk -es test2 + maria_read_log -a -s >& /dev/null + maria_chk -es test2 + rm test2.MA? + maria_read_log -a -s >& /dev/null + maria_chk -es test2 + a=$((a+1)) +done diff --git a/storage/maria/ma_test_recovery b/storage/maria/ma_test_recovery index ee1d8c5366f..1794f245dfc 100755 --- a/storage/maria/ma_test_recovery +++ b/storage/maria/ma_test_recovery @@ -1,5 +1,7 @@ #!/bin/sh +# Remove comment from next line if this script fails and you need more +# information of what's going on #set -x -v set -e silent="-s" @@ -122,12 +124,13 @@ echo "Testing the REDO AND UNDO PHASE" for take_checkpoint in "no" "yes" do -for blobs in "" "-b" # we test table without blobs and then table with blobs +# we test table without blobs and then table with blobs +for blobs in "" "-b32768" do for test_undo in 1 2 3 4 do # first iteration tests rollback of insert, second tests rollback of delete - set -- "ma_test1 $silent -M -T -c -N $blobs -H1" "--testflag=1" "--testflag=2 --test-undo=" "ma_test1 $silent -M -T -c -N $blobs -H2" "--testflag=3" "--testflag=4 --test-undo=" "ma_test1 $silent -M -T -c -N $blobs -H2 " "--testflag=2" "--testflag=3 --test-undo=" "ma_test2 $silent -L -K -W -P -M -T -c $blobs -H1" "-t1" "-t2 -u" + set -- "ma_test1 $silent -M -T -c -N $blobs -H1" "--testflag=1" "--testflag=2 --test-undo=" "ma_test1 $silent -M -T -c -N $blobs -H2" "--testflag=3" "--testflag=4 --test-undo=" "ma_test1 $silent -M -T -c -N $blobs -H2 " "--testflag=2" "--testflag=3 --test-undo=" "ma_test2 $silent -L -K -W -P -M -T -c $blobs -H1" "-t1" "-t2 -A" "ma_test2 $silent -L -K -W -P -M -T -c $blobs -H1" "-t1" "-t4 -A" # -N (create NULL fields) is needed because --test-undo adds it anyway while [ $# != 0 ] do @@ -178,18 +181,21 @@ do check_table_is_same echo "testing idempotency" apply_log "shouldnotchangelog" - cmp $table.MAD $tmp/$table.MAD.after_undo + # We can't do a binary compary as there may have been different number + # of calls to compact_page. We can enable this if we first call + # maria-check to generate identically compacted pages. +# cmp $table.MAD $tmp/$table.MAD.after_undo # can't do this, creation time differs at least; enable it if you # have a "cmp" which ignores the header. -# cmp $table.MAI $tmp/$table.MAI.after_undo + cmp $table.MAI $tmp/$table.MAI.after_undo check_table_is_same echo "testing applying of CLRs to recreate table" rm $table.MA? # cp $tmp/maria_log* $maria_path #unneeded apply_log "shouldnotchangelog" - cmp $table.MAD $tmp/$table.MAD.after_undo +# cmp $table.MAD $tmp/$table.MAD.after_undo # can't do this, creation time differs at least -# cmp $table.MAI $tmp/$table.MAI.after_undo + cmp $table.MAI $tmp/$table.MAI.after_undo check_table_is_same shift 3 done @@ -200,6 +206,12 @@ done ) 2>&1 > $tmp/ma_test_recovery.output +if [ "$?" != 0 ] +then + echo "Some test failed" + exit 1 +fi + # also note that maria_chk -dvv shows differences for ma_test2 in UNDO phase, # this is normal: removing records does not shrink the data/key file, # does not put back the "analyzed,optimized keys"(etc) index state. diff --git a/storage/maria/ma_test_recovery.expected b/storage/maria/ma_test_recovery.expected index 814b0cc2eb8..d16bd52d9a6 100644 --- a/storage/maria/ma_test_recovery.expected +++ b/storage/maria/ma_test_recovery.expected @@ -42,7 +42,36 @@ applying log testing applying of CLRs to recreate table applying log TEST WITH ma_test2 -s -L -K -W -P -M -T -c -t1 (commit at end) -TEST WITH ma_test2 -s -L -K -W -P -M -T -c -t2 -u1 (additional aborted work) +TEST WITH ma_test2 -s -L -K -W -P -M -T -c -t2 -A1 (additional aborted work) +Dying on request without maria_commit()/maria_close() +applying log +Differences in maria_chk -dvv, recovery not yet perfect ! +========DIFF START======= +6c6 +< Status: checked,analyzed,optimized keys,sorted index pages +--- +> Status: changed +========DIFF END======= +testing idempotency +applying log +Differences in maria_chk -dvv, recovery not yet perfect ! +========DIFF START======= +6c6 +< Status: checked,analyzed,optimized keys,sorted index pages +--- +> Status: changed +========DIFF END======= +testing applying of CLRs to recreate table +applying log +Differences in maria_chk -dvv, recovery not yet perfect ! +========DIFF START======= +6c6 +< Status: checked,analyzed,optimized keys,sorted index pages +--- +> Status: changed +========DIFF END======= +TEST WITH ma_test2 -s -L -K -W -P -M -T -c -t1 (commit at end) +TEST WITH ma_test2 -s -L -K -W -P -M -T -c -t4 -A1 (additional aborted work) Dying on request without maria_commit()/maria_close() applying log Differences in maria_chk -dvv, recovery not yet perfect ! @@ -100,7 +129,36 @@ applying log testing applying of CLRs to recreate table applying log TEST WITH ma_test2 -s -L -K -W -P -M -T -c -t1 (commit at end) -TEST WITH ma_test2 -s -L -K -W -P -M -T -c -t2 -u2 (additional aborted work) +TEST WITH ma_test2 -s -L -K -W -P -M -T -c -t2 -A2 (additional aborted work) +Dying on request without maria_commit()/maria_close() +applying log +Differences in maria_chk -dvv, recovery not yet perfect ! +========DIFF START======= +6c6 +< Status: checked,analyzed,optimized keys,sorted index pages +--- +> Status: changed +========DIFF END======= +testing idempotency +applying log +Differences in maria_chk -dvv, recovery not yet perfect ! +========DIFF START======= +6c6 +< Status: checked,analyzed,optimized keys,sorted index pages +--- +> Status: changed +========DIFF END======= +testing applying of CLRs to recreate table +applying log +Differences in maria_chk -dvv, recovery not yet perfect ! +========DIFF START======= +6c6 +< Status: checked,analyzed,optimized keys,sorted index pages +--- +> Status: changed +========DIFF END======= +TEST WITH ma_test2 -s -L -K -W -P -M -T -c -t1 (commit at end) +TEST WITH ma_test2 -s -L -K -W -P -M -T -c -t4 -A2 (additional aborted work) Dying on request without maria_commit()/maria_close() applying log Differences in maria_chk -dvv, recovery not yet perfect ! @@ -158,7 +216,36 @@ applying log testing applying of CLRs to recreate table applying log TEST WITH ma_test2 -s -L -K -W -P -M -T -c -t1 (commit at end) -TEST WITH ma_test2 -s -L -K -W -P -M -T -c -t2 -u3 (additional aborted work) +TEST WITH ma_test2 -s -L -K -W -P -M -T -c -t2 -A3 (additional aborted work) +Dying on request without maria_commit()/maria_close() +applying log +Differences in maria_chk -dvv, recovery not yet perfect ! +========DIFF START======= +6c6 +< Status: checked,analyzed,optimized keys,sorted index pages +--- +> Status: changed +========DIFF END======= +testing idempotency +applying log +Differences in maria_chk -dvv, recovery not yet perfect ! +========DIFF START======= +6c6 +< Status: checked,analyzed,optimized keys,sorted index pages +--- +> Status: changed +========DIFF END======= +testing applying of CLRs to recreate table +applying log +Differences in maria_chk -dvv, recovery not yet perfect ! +========DIFF START======= +6c6 +< Status: checked,analyzed,optimized keys,sorted index pages +--- +> Status: changed +========DIFF END======= +TEST WITH ma_test2 -s -L -K -W -P -M -T -c -t1 (commit at end) +TEST WITH ma_test2 -s -L -K -W -P -M -T -c -t4 -A3 (additional aborted work) Dying on request without maria_commit()/maria_close() applying log Differences in maria_chk -dvv, recovery not yet perfect ! @@ -216,7 +303,36 @@ applying log testing applying of CLRs to recreate table applying log TEST WITH ma_test2 -s -L -K -W -P -M -T -c -t1 (commit at end) -TEST WITH ma_test2 -s -L -K -W -P -M -T -c -t2 -u4 (additional aborted work) +TEST WITH ma_test2 -s -L -K -W -P -M -T -c -t2 -A4 (additional aborted work) +Dying on request without maria_commit()/maria_close() +applying log +Differences in maria_chk -dvv, recovery not yet perfect ! +========DIFF START======= +6c6 +< Status: checked,analyzed,optimized keys,sorted index pages +--- +> Status: changed +========DIFF END======= +testing idempotency +applying log +Differences in maria_chk -dvv, recovery not yet perfect ! +========DIFF START======= +6c6 +< Status: checked,analyzed,optimized keys,sorted index pages +--- +> Status: changed +========DIFF END======= +testing applying of CLRs to recreate table +applying log +Differences in maria_chk -dvv, recovery not yet perfect ! +========DIFF START======= +6c6 +< Status: checked,analyzed,optimized keys,sorted index pages +--- +> Status: changed +========DIFF END======= +TEST WITH ma_test2 -s -L -K -W -P -M -T -c -t1 (commit at end) +TEST WITH ma_test2 -s -L -K -W -P -M -T -c -t4 -A4 (additional aborted work) Dying on request without maria_commit()/maria_close() applying log Differences in maria_chk -dvv, recovery not yet perfect ! @@ -244,8 +360,8 @@ Differences in maria_chk -dvv, recovery not yet perfect ! --- > Status: changed ========DIFF END======= -TEST WITH ma_test1 -s -M -T -c -N -b --testflag=1 (commit at end) -TEST WITH ma_test1 -s -M -T -c -N -b --testflag=2 --test-undo=1 (additional aborted work) +TEST WITH ma_test1 -s -M -T -c -N -b32768 --testflag=1 (commit at end) +TEST WITH ma_test1 -s -M -T -c -N -b32768 --testflag=2 --test-undo=1 (additional aborted work) Terminating after inserts Dying on request without maria_commit()/maria_close() applying log @@ -253,9 +369,9 @@ testing idempotency applying log testing applying of CLRs to recreate table applying log -TEST WITH ma_test1 -s -M -T -c -N -b --testflag=3 (commit at end) +TEST WITH ma_test1 -s -M -T -c -N -b32768 --testflag=3 (commit at end) Terminating after updates -TEST WITH ma_test1 -s -M -T -c -N -b --testflag=4 --test-undo=1 (additional aborted work) +TEST WITH ma_test1 -s -M -T -c -N -b32768 --testflag=4 --test-undo=1 (additional aborted work) Terminating after deletes Dying on request without maria_commit()/maria_close() applying log @@ -263,9 +379,9 @@ testing idempotency applying log testing applying of CLRs to recreate table applying log -TEST WITH ma_test1 -s -M -T -c -N -b --testflag=2 (commit at end) +TEST WITH ma_test1 -s -M -T -c -N -b32768 --testflag=2 (commit at end) Terminating after inserts -TEST WITH ma_test1 -s -M -T -c -N -b --testflag=3 --test-undo=1 (additional aborted work) +TEST WITH ma_test1 -s -M -T -c -N -b32768 --testflag=3 --test-undo=1 (additional aborted work) Terminating after updates Dying on request without maria_commit()/maria_close() applying log @@ -273,8 +389,8 @@ testing idempotency applying log testing applying of CLRs to recreate table applying log -TEST WITH ma_test2 -s -L -K -W -P -M -T -c -b -t1 (commit at end) -TEST WITH ma_test2 -s -L -K -W -P -M -T -c -b -t2 -u1 (additional aborted work) +TEST WITH ma_test2 -s -L -K -W -P -M -T -c -b32768 -t1 (commit at end) +TEST WITH ma_test2 -s -L -K -W -P -M -T -c -b32768 -t2 -A1 (additional aborted work) Dying on request without maria_commit()/maria_close() applying log Differences in maria_chk -dvv, recovery not yet perfect ! @@ -302,8 +418,37 @@ Differences in maria_chk -dvv, recovery not yet perfect ! --- > Status: changed ========DIFF END======= -TEST WITH ma_test1 -s -M -T -c -N -b --testflag=1 (commit at end) -TEST WITH ma_test1 -s -M -T -c -N -b --testflag=2 --test-undo=2 (additional aborted work) +TEST WITH ma_test2 -s -L -K -W -P -M -T -c -b32768 -t1 (commit at end) +TEST WITH ma_test2 -s -L -K -W -P -M -T -c -b32768 -t4 -A1 (additional aborted work) +Dying on request without maria_commit()/maria_close() +applying log +Differences in maria_chk -dvv, recovery not yet perfect ! +========DIFF START======= +6c6 +< Status: checked,analyzed,optimized keys,sorted index pages +--- +> Status: changed +========DIFF END======= +testing idempotency +applying log +Differences in maria_chk -dvv, recovery not yet perfect ! +========DIFF START======= +6c6 +< Status: checked,analyzed,optimized keys,sorted index pages +--- +> Status: changed +========DIFF END======= +testing applying of CLRs to recreate table +applying log +Differences in maria_chk -dvv, recovery not yet perfect ! +========DIFF START======= +6c6 +< Status: checked,analyzed,optimized keys,sorted index pages +--- +> Status: changed +========DIFF END======= +TEST WITH ma_test1 -s -M -T -c -N -b32768 --testflag=1 (commit at end) +TEST WITH ma_test1 -s -M -T -c -N -b32768 --testflag=2 --test-undo=2 (additional aborted work) Terminating after inserts Dying on request without maria_commit()/maria_close() applying log @@ -311,9 +456,9 @@ testing idempotency applying log testing applying of CLRs to recreate table applying log -TEST WITH ma_test1 -s -M -T -c -N -b --testflag=3 (commit at end) +TEST WITH ma_test1 -s -M -T -c -N -b32768 --testflag=3 (commit at end) Terminating after updates -TEST WITH ma_test1 -s -M -T -c -N -b --testflag=4 --test-undo=2 (additional aborted work) +TEST WITH ma_test1 -s -M -T -c -N -b32768 --testflag=4 --test-undo=2 (additional aborted work) Terminating after deletes Dying on request without maria_commit()/maria_close() applying log @@ -321,9 +466,9 @@ testing idempotency applying log testing applying of CLRs to recreate table applying log -TEST WITH ma_test1 -s -M -T -c -N -b --testflag=2 (commit at end) +TEST WITH ma_test1 -s -M -T -c -N -b32768 --testflag=2 (commit at end) Terminating after inserts -TEST WITH ma_test1 -s -M -T -c -N -b --testflag=3 --test-undo=2 (additional aborted work) +TEST WITH ma_test1 -s -M -T -c -N -b32768 --testflag=3 --test-undo=2 (additional aborted work) Terminating after updates Dying on request without maria_commit()/maria_close() applying log @@ -331,8 +476,37 @@ testing idempotency applying log testing applying of CLRs to recreate table applying log -TEST WITH ma_test2 -s -L -K -W -P -M -T -c -b -t1 (commit at end) -TEST WITH ma_test2 -s -L -K -W -P -M -T -c -b -t2 -u2 (additional aborted work) +TEST WITH ma_test2 -s -L -K -W -P -M -T -c -b32768 -t1 (commit at end) +TEST WITH ma_test2 -s -L -K -W -P -M -T -c -b32768 -t2 -A2 (additional aborted work) +Dying on request without maria_commit()/maria_close() +applying log +Differences in maria_chk -dvv, recovery not yet perfect ! +========DIFF START======= +6c6 +< Status: checked,analyzed,optimized keys,sorted index pages +--- +> Status: changed +========DIFF END======= +testing idempotency +applying log +Differences in maria_chk -dvv, recovery not yet perfect ! +========DIFF START======= +6c6 +< Status: checked,analyzed,optimized keys,sorted index pages +--- +> Status: changed +========DIFF END======= +testing applying of CLRs to recreate table +applying log +Differences in maria_chk -dvv, recovery not yet perfect ! +========DIFF START======= +6c6 +< Status: checked,analyzed,optimized keys,sorted index pages +--- +> Status: changed +========DIFF END======= +TEST WITH ma_test2 -s -L -K -W -P -M -T -c -b32768 -t1 (commit at end) +TEST WITH ma_test2 -s -L -K -W -P -M -T -c -b32768 -t4 -A2 (additional aborted work) Dying on request without maria_commit()/maria_close() applying log Differences in maria_chk -dvv, recovery not yet perfect ! @@ -360,8 +534,8 @@ Differences in maria_chk -dvv, recovery not yet perfect ! --- > Status: changed ========DIFF END======= -TEST WITH ma_test1 -s -M -T -c -N -b --testflag=1 (commit at end) -TEST WITH ma_test1 -s -M -T -c -N -b --testflag=2 --test-undo=3 (additional aborted work) +TEST WITH ma_test1 -s -M -T -c -N -b32768 --testflag=1 (commit at end) +TEST WITH ma_test1 -s -M -T -c -N -b32768 --testflag=2 --test-undo=3 (additional aborted work) Terminating after inserts Dying on request without maria_commit()/maria_close() applying log @@ -369,9 +543,9 @@ testing idempotency applying log testing applying of CLRs to recreate table applying log -TEST WITH ma_test1 -s -M -T -c -N -b --testflag=3 (commit at end) +TEST WITH ma_test1 -s -M -T -c -N -b32768 --testflag=3 (commit at end) Terminating after updates -TEST WITH ma_test1 -s -M -T -c -N -b --testflag=4 --test-undo=3 (additional aborted work) +TEST WITH ma_test1 -s -M -T -c -N -b32768 --testflag=4 --test-undo=3 (additional aborted work) Terminating after deletes Dying on request without maria_commit()/maria_close() applying log @@ -379,9 +553,9 @@ testing idempotency applying log testing applying of CLRs to recreate table applying log -TEST WITH ma_test1 -s -M -T -c -N -b --testflag=2 (commit at end) +TEST WITH ma_test1 -s -M -T -c -N -b32768 --testflag=2 (commit at end) Terminating after inserts -TEST WITH ma_test1 -s -M -T -c -N -b --testflag=3 --test-undo=3 (additional aborted work) +TEST WITH ma_test1 -s -M -T -c -N -b32768 --testflag=3 --test-undo=3 (additional aborted work) Terminating after updates Dying on request without maria_commit()/maria_close() applying log @@ -389,8 +563,37 @@ testing idempotency applying log testing applying of CLRs to recreate table applying log -TEST WITH ma_test2 -s -L -K -W -P -M -T -c -b -t1 (commit at end) -TEST WITH ma_test2 -s -L -K -W -P -M -T -c -b -t2 -u3 (additional aborted work) +TEST WITH ma_test2 -s -L -K -W -P -M -T -c -b32768 -t1 (commit at end) +TEST WITH ma_test2 -s -L -K -W -P -M -T -c -b32768 -t2 -A3 (additional aborted work) +Dying on request without maria_commit()/maria_close() +applying log +Differences in maria_chk -dvv, recovery not yet perfect ! +========DIFF START======= +6c6 +< Status: checked,analyzed,optimized keys,sorted index pages +--- +> Status: changed +========DIFF END======= +testing idempotency +applying log +Differences in maria_chk -dvv, recovery not yet perfect ! +========DIFF START======= +6c6 +< Status: checked,analyzed,optimized keys,sorted index pages +--- +> Status: changed +========DIFF END======= +testing applying of CLRs to recreate table +applying log +Differences in maria_chk -dvv, recovery not yet perfect ! +========DIFF START======= +6c6 +< Status: checked,analyzed,optimized keys,sorted index pages +--- +> Status: changed +========DIFF END======= +TEST WITH ma_test2 -s -L -K -W -P -M -T -c -b32768 -t1 (commit at end) +TEST WITH ma_test2 -s -L -K -W -P -M -T -c -b32768 -t4 -A3 (additional aborted work) Dying on request without maria_commit()/maria_close() applying log Differences in maria_chk -dvv, recovery not yet perfect ! @@ -418,8 +621,8 @@ Differences in maria_chk -dvv, recovery not yet perfect ! --- > Status: changed ========DIFF END======= -TEST WITH ma_test1 -s -M -T -c -N -b --testflag=1 (commit at end) -TEST WITH ma_test1 -s -M -T -c -N -b --testflag=2 --test-undo=4 (additional aborted work) +TEST WITH ma_test1 -s -M -T -c -N -b32768 --testflag=1 (commit at end) +TEST WITH ma_test1 -s -M -T -c -N -b32768 --testflag=2 --test-undo=4 (additional aborted work) Terminating after inserts Dying on request without maria_commit()/maria_close() applying log @@ -427,9 +630,9 @@ testing idempotency applying log testing applying of CLRs to recreate table applying log -TEST WITH ma_test1 -s -M -T -c -N -b --testflag=3 (commit at end) +TEST WITH ma_test1 -s -M -T -c -N -b32768 --testflag=3 (commit at end) Terminating after updates -TEST WITH ma_test1 -s -M -T -c -N -b --testflag=4 --test-undo=4 (additional aborted work) +TEST WITH ma_test1 -s -M -T -c -N -b32768 --testflag=4 --test-undo=4 (additional aborted work) Terminating after deletes Dying on request without maria_commit()/maria_close() applying log @@ -437,9 +640,9 @@ testing idempotency applying log testing applying of CLRs to recreate table applying log -TEST WITH ma_test1 -s -M -T -c -N -b --testflag=2 (commit at end) +TEST WITH ma_test1 -s -M -T -c -N -b32768 --testflag=2 (commit at end) Terminating after inserts -TEST WITH ma_test1 -s -M -T -c -N -b --testflag=3 --test-undo=4 (additional aborted work) +TEST WITH ma_test1 -s -M -T -c -N -b32768 --testflag=3 --test-undo=4 (additional aborted work) Terminating after updates Dying on request without maria_commit()/maria_close() applying log @@ -447,8 +650,37 @@ testing idempotency applying log testing applying of CLRs to recreate table applying log -TEST WITH ma_test2 -s -L -K -W -P -M -T -c -b -t1 (commit at end) -TEST WITH ma_test2 -s -L -K -W -P -M -T -c -b -t2 -u4 (additional aborted work) +TEST WITH ma_test2 -s -L -K -W -P -M -T -c -b32768 -t1 (commit at end) +TEST WITH ma_test2 -s -L -K -W -P -M -T -c -b32768 -t2 -A4 (additional aborted work) +Dying on request without maria_commit()/maria_close() +applying log +Differences in maria_chk -dvv, recovery not yet perfect ! +========DIFF START======= +6c6 +< Status: checked,analyzed,optimized keys,sorted index pages +--- +> Status: changed +========DIFF END======= +testing idempotency +applying log +Differences in maria_chk -dvv, recovery not yet perfect ! +========DIFF START======= +6c6 +< Status: checked,analyzed,optimized keys,sorted index pages +--- +> Status: changed +========DIFF END======= +testing applying of CLRs to recreate table +applying log +Differences in maria_chk -dvv, recovery not yet perfect ! +========DIFF START======= +6c6 +< Status: checked,analyzed,optimized keys,sorted index pages +--- +> Status: changed +========DIFF END======= +TEST WITH ma_test2 -s -L -K -W -P -M -T -c -b32768 -t1 (commit at end) +TEST WITH ma_test2 -s -L -K -W -P -M -T -c -b32768 -t4 -A4 (additional aborted work) Dying on request without maria_commit()/maria_close() applying log Differences in maria_chk -dvv, recovery not yet perfect ! @@ -506,7 +738,36 @@ applying log testing applying of CLRs to recreate table applying log TEST WITH ma_test2 -s -L -K -W -P -M -T -c -H1 -t1 (commit at end) -TEST WITH ma_test2 -s -L -K -W -P -M -T -c -H1 -t2 -u1 (additional aborted work) +TEST WITH ma_test2 -s -L -K -W -P -M -T -c -H1 -t2 -A1 (additional aborted work) +Dying on request without maria_commit()/maria_close() +applying log +Differences in maria_chk -dvv, recovery not yet perfect ! +========DIFF START======= +6c6 +< Status: checked,analyzed,optimized keys,sorted index pages +--- +> Status: changed +========DIFF END======= +testing idempotency +applying log +Differences in maria_chk -dvv, recovery not yet perfect ! +========DIFF START======= +6c6 +< Status: checked,analyzed,optimized keys,sorted index pages +--- +> Status: changed +========DIFF END======= +testing applying of CLRs to recreate table +applying log +Differences in maria_chk -dvv, recovery not yet perfect ! +========DIFF START======= +6c6 +< Status: checked,analyzed,optimized keys,sorted index pages +--- +> Status: changed +========DIFF END======= +TEST WITH ma_test2 -s -L -K -W -P -M -T -c -H1 -t1 (commit at end) +TEST WITH ma_test2 -s -L -K -W -P -M -T -c -H1 -t4 -A1 (additional aborted work) Dying on request without maria_commit()/maria_close() applying log Differences in maria_chk -dvv, recovery not yet perfect ! @@ -564,7 +825,36 @@ applying log testing applying of CLRs to recreate table applying log TEST WITH ma_test2 -s -L -K -W -P -M -T -c -H1 -t1 (commit at end) -TEST WITH ma_test2 -s -L -K -W -P -M -T -c -H1 -t2 -u2 (additional aborted work) +TEST WITH ma_test2 -s -L -K -W -P -M -T -c -H1 -t2 -A2 (additional aborted work) +Dying on request without maria_commit()/maria_close() +applying log +Differences in maria_chk -dvv, recovery not yet perfect ! +========DIFF START======= +6c6 +< Status: checked,analyzed,optimized keys,sorted index pages +--- +> Status: changed +========DIFF END======= +testing idempotency +applying log +Differences in maria_chk -dvv, recovery not yet perfect ! +========DIFF START======= +6c6 +< Status: checked,analyzed,optimized keys,sorted index pages +--- +> Status: changed +========DIFF END======= +testing applying of CLRs to recreate table +applying log +Differences in maria_chk -dvv, recovery not yet perfect ! +========DIFF START======= +6c6 +< Status: checked,analyzed,optimized keys,sorted index pages +--- +> Status: changed +========DIFF END======= +TEST WITH ma_test2 -s -L -K -W -P -M -T -c -H1 -t1 (commit at end) +TEST WITH ma_test2 -s -L -K -W -P -M -T -c -H1 -t4 -A2 (additional aborted work) Dying on request without maria_commit()/maria_close() applying log Differences in maria_chk -dvv, recovery not yet perfect ! @@ -622,7 +912,36 @@ applying log testing applying of CLRs to recreate table applying log TEST WITH ma_test2 -s -L -K -W -P -M -T -c -H1 -t1 (commit at end) -TEST WITH ma_test2 -s -L -K -W -P -M -T -c -H1 -t2 -u3 (additional aborted work) +TEST WITH ma_test2 -s -L -K -W -P -M -T -c -H1 -t2 -A3 (additional aborted work) +Dying on request without maria_commit()/maria_close() +applying log +Differences in maria_chk -dvv, recovery not yet perfect ! +========DIFF START======= +6c6 +< Status: checked,analyzed,optimized keys,sorted index pages +--- +> Status: changed +========DIFF END======= +testing idempotency +applying log +Differences in maria_chk -dvv, recovery not yet perfect ! +========DIFF START======= +6c6 +< Status: checked,analyzed,optimized keys,sorted index pages +--- +> Status: changed +========DIFF END======= +testing applying of CLRs to recreate table +applying log +Differences in maria_chk -dvv, recovery not yet perfect ! +========DIFF START======= +6c6 +< Status: checked,analyzed,optimized keys,sorted index pages +--- +> Status: changed +========DIFF END======= +TEST WITH ma_test2 -s -L -K -W -P -M -T -c -H1 -t1 (commit at end) +TEST WITH ma_test2 -s -L -K -W -P -M -T -c -H1 -t4 -A3 (additional aborted work) Dying on request without maria_commit()/maria_close() applying log Differences in maria_chk -dvv, recovery not yet perfect ! @@ -680,7 +999,36 @@ applying log testing applying of CLRs to recreate table applying log TEST WITH ma_test2 -s -L -K -W -P -M -T -c -H1 -t1 (commit at end) -TEST WITH ma_test2 -s -L -K -W -P -M -T -c -H1 -t2 -u4 (additional aborted work) +TEST WITH ma_test2 -s -L -K -W -P -M -T -c -H1 -t2 -A4 (additional aborted work) +Dying on request without maria_commit()/maria_close() +applying log +Differences in maria_chk -dvv, recovery not yet perfect ! +========DIFF START======= +6c6 +< Status: checked,analyzed,optimized keys,sorted index pages +--- +> Status: changed +========DIFF END======= +testing idempotency +applying log +Differences in maria_chk -dvv, recovery not yet perfect ! +========DIFF START======= +6c6 +< Status: checked,analyzed,optimized keys,sorted index pages +--- +> Status: changed +========DIFF END======= +testing applying of CLRs to recreate table +applying log +Differences in maria_chk -dvv, recovery not yet perfect ! +========DIFF START======= +6c6 +< Status: checked,analyzed,optimized keys,sorted index pages +--- +> Status: changed +========DIFF END======= +TEST WITH ma_test2 -s -L -K -W -P -M -T -c -H1 -t1 (commit at end) +TEST WITH ma_test2 -s -L -K -W -P -M -T -c -H1 -t4 -A4 (additional aborted work) Dying on request without maria_commit()/maria_close() applying log Differences in maria_chk -dvv, recovery not yet perfect ! @@ -708,8 +1056,8 @@ Differences in maria_chk -dvv, recovery not yet perfect ! --- > Status: changed ========DIFF END======= -TEST WITH ma_test1 -s -M -T -c -N -b -H1 --testflag=1 (commit at end) -TEST WITH ma_test1 -s -M -T -c -N -b -H1 --testflag=2 --test-undo=1 (additional aborted work) +TEST WITH ma_test1 -s -M -T -c -N -b32768 -H1 --testflag=1 (commit at end) +TEST WITH ma_test1 -s -M -T -c -N -b32768 -H1 --testflag=2 --test-undo=1 (additional aborted work) Terminating after inserts Dying on request without maria_commit()/maria_close() applying log @@ -717,9 +1065,9 @@ testing idempotency applying log testing applying of CLRs to recreate table applying log -TEST WITH ma_test1 -s -M -T -c -N -b -H2 --testflag=3 (commit at end) +TEST WITH ma_test1 -s -M -T -c -N -b32768 -H2 --testflag=3 (commit at end) Terminating after updates -TEST WITH ma_test1 -s -M -T -c -N -b -H2 --testflag=4 --test-undo=1 (additional aborted work) +TEST WITH ma_test1 -s -M -T -c -N -b32768 -H2 --testflag=4 --test-undo=1 (additional aborted work) Terminating after deletes Dying on request without maria_commit()/maria_close() applying log @@ -727,9 +1075,9 @@ testing idempotency applying log testing applying of CLRs to recreate table applying log -TEST WITH ma_test1 -s -M -T -c -N -b -H2 --testflag=2 (commit at end) +TEST WITH ma_test1 -s -M -T -c -N -b32768 -H2 --testflag=2 (commit at end) Terminating after inserts -TEST WITH ma_test1 -s -M -T -c -N -b -H2 --testflag=3 --test-undo=1 (additional aborted work) +TEST WITH ma_test1 -s -M -T -c -N -b32768 -H2 --testflag=3 --test-undo=1 (additional aborted work) Terminating after updates Dying on request without maria_commit()/maria_close() applying log @@ -737,8 +1085,8 @@ testing idempotency applying log testing applying of CLRs to recreate table applying log -TEST WITH ma_test2 -s -L -K -W -P -M -T -c -b -H1 -t1 (commit at end) -TEST WITH ma_test2 -s -L -K -W -P -M -T -c -b -H1 -t2 -u1 (additional aborted work) +TEST WITH ma_test2 -s -L -K -W -P -M -T -c -b32768 -H1 -t1 (commit at end) +TEST WITH ma_test2 -s -L -K -W -P -M -T -c -b32768 -H1 -t2 -A1 (additional aborted work) Dying on request without maria_commit()/maria_close() applying log Differences in maria_chk -dvv, recovery not yet perfect ! @@ -766,8 +1114,37 @@ Differences in maria_chk -dvv, recovery not yet perfect ! --- > Status: changed ========DIFF END======= -TEST WITH ma_test1 -s -M -T -c -N -b -H1 --testflag=1 (commit at end) -TEST WITH ma_test1 -s -M -T -c -N -b -H1 --testflag=2 --test-undo=2 (additional aborted work) +TEST WITH ma_test2 -s -L -K -W -P -M -T -c -b32768 -H1 -t1 (commit at end) +TEST WITH ma_test2 -s -L -K -W -P -M -T -c -b32768 -H1 -t4 -A1 (additional aborted work) +Dying on request without maria_commit()/maria_close() +applying log +Differences in maria_chk -dvv, recovery not yet perfect ! +========DIFF START======= +6c6 +< Status: checked,analyzed,optimized keys,sorted index pages +--- +> Status: changed +========DIFF END======= +testing idempotency +applying log +Differences in maria_chk -dvv, recovery not yet perfect ! +========DIFF START======= +6c6 +< Status: checked,analyzed,optimized keys,sorted index pages +--- +> Status: changed +========DIFF END======= +testing applying of CLRs to recreate table +applying log +Differences in maria_chk -dvv, recovery not yet perfect ! +========DIFF START======= +6c6 +< Status: checked,analyzed,optimized keys,sorted index pages +--- +> Status: changed +========DIFF END======= +TEST WITH ma_test1 -s -M -T -c -N -b32768 -H1 --testflag=1 (commit at end) +TEST WITH ma_test1 -s -M -T -c -N -b32768 -H1 --testflag=2 --test-undo=2 (additional aborted work) Terminating after inserts Dying on request without maria_commit()/maria_close() applying log @@ -775,9 +1152,9 @@ testing idempotency applying log testing applying of CLRs to recreate table applying log -TEST WITH ma_test1 -s -M -T -c -N -b -H2 --testflag=3 (commit at end) +TEST WITH ma_test1 -s -M -T -c -N -b32768 -H2 --testflag=3 (commit at end) Terminating after updates -TEST WITH ma_test1 -s -M -T -c -N -b -H2 --testflag=4 --test-undo=2 (additional aborted work) +TEST WITH ma_test1 -s -M -T -c -N -b32768 -H2 --testflag=4 --test-undo=2 (additional aborted work) Terminating after deletes Dying on request without maria_commit()/maria_close() applying log @@ -785,9 +1162,9 @@ testing idempotency applying log testing applying of CLRs to recreate table applying log -TEST WITH ma_test1 -s -M -T -c -N -b -H2 --testflag=2 (commit at end) +TEST WITH ma_test1 -s -M -T -c -N -b32768 -H2 --testflag=2 (commit at end) Terminating after inserts -TEST WITH ma_test1 -s -M -T -c -N -b -H2 --testflag=3 --test-undo=2 (additional aborted work) +TEST WITH ma_test1 -s -M -T -c -N -b32768 -H2 --testflag=3 --test-undo=2 (additional aborted work) Terminating after updates Dying on request without maria_commit()/maria_close() applying log @@ -795,8 +1172,37 @@ testing idempotency applying log testing applying of CLRs to recreate table applying log -TEST WITH ma_test2 -s -L -K -W -P -M -T -c -b -H1 -t1 (commit at end) -TEST WITH ma_test2 -s -L -K -W -P -M -T -c -b -H1 -t2 -u2 (additional aborted work) +TEST WITH ma_test2 -s -L -K -W -P -M -T -c -b32768 -H1 -t1 (commit at end) +TEST WITH ma_test2 -s -L -K -W -P -M -T -c -b32768 -H1 -t2 -A2 (additional aborted work) +Dying on request without maria_commit()/maria_close() +applying log +Differences in maria_chk -dvv, recovery not yet perfect ! +========DIFF START======= +6c6 +< Status: checked,analyzed,optimized keys,sorted index pages +--- +> Status: changed +========DIFF END======= +testing idempotency +applying log +Differences in maria_chk -dvv, recovery not yet perfect ! +========DIFF START======= +6c6 +< Status: checked,analyzed,optimized keys,sorted index pages +--- +> Status: changed +========DIFF END======= +testing applying of CLRs to recreate table +applying log +Differences in maria_chk -dvv, recovery not yet perfect ! +========DIFF START======= +6c6 +< Status: checked,analyzed,optimized keys,sorted index pages +--- +> Status: changed +========DIFF END======= +TEST WITH ma_test2 -s -L -K -W -P -M -T -c -b32768 -H1 -t1 (commit at end) +TEST WITH ma_test2 -s -L -K -W -P -M -T -c -b32768 -H1 -t4 -A2 (additional aborted work) Dying on request without maria_commit()/maria_close() applying log Differences in maria_chk -dvv, recovery not yet perfect ! @@ -824,8 +1230,8 @@ Differences in maria_chk -dvv, recovery not yet perfect ! --- > Status: changed ========DIFF END======= -TEST WITH ma_test1 -s -M -T -c -N -b -H1 --testflag=1 (commit at end) -TEST WITH ma_test1 -s -M -T -c -N -b -H1 --testflag=2 --test-undo=3 (additional aborted work) +TEST WITH ma_test1 -s -M -T -c -N -b32768 -H1 --testflag=1 (commit at end) +TEST WITH ma_test1 -s -M -T -c -N -b32768 -H1 --testflag=2 --test-undo=3 (additional aborted work) Terminating after inserts Dying on request without maria_commit()/maria_close() applying log @@ -833,9 +1239,9 @@ testing idempotency applying log testing applying of CLRs to recreate table applying log -TEST WITH ma_test1 -s -M -T -c -N -b -H2 --testflag=3 (commit at end) +TEST WITH ma_test1 -s -M -T -c -N -b32768 -H2 --testflag=3 (commit at end) Terminating after updates -TEST WITH ma_test1 -s -M -T -c -N -b -H2 --testflag=4 --test-undo=3 (additional aborted work) +TEST WITH ma_test1 -s -M -T -c -N -b32768 -H2 --testflag=4 --test-undo=3 (additional aborted work) Terminating after deletes Dying on request without maria_commit()/maria_close() applying log @@ -843,9 +1249,9 @@ testing idempotency applying log testing applying of CLRs to recreate table applying log -TEST WITH ma_test1 -s -M -T -c -N -b -H2 --testflag=2 (commit at end) +TEST WITH ma_test1 -s -M -T -c -N -b32768 -H2 --testflag=2 (commit at end) Terminating after inserts -TEST WITH ma_test1 -s -M -T -c -N -b -H2 --testflag=3 --test-undo=3 (additional aborted work) +TEST WITH ma_test1 -s -M -T -c -N -b32768 -H2 --testflag=3 --test-undo=3 (additional aborted work) Terminating after updates Dying on request without maria_commit()/maria_close() applying log @@ -853,8 +1259,37 @@ testing idempotency applying log testing applying of CLRs to recreate table applying log -TEST WITH ma_test2 -s -L -K -W -P -M -T -c -b -H1 -t1 (commit at end) -TEST WITH ma_test2 -s -L -K -W -P -M -T -c -b -H1 -t2 -u3 (additional aborted work) +TEST WITH ma_test2 -s -L -K -W -P -M -T -c -b32768 -H1 -t1 (commit at end) +TEST WITH ma_test2 -s -L -K -W -P -M -T -c -b32768 -H1 -t2 -A3 (additional aborted work) +Dying on request without maria_commit()/maria_close() +applying log +Differences in maria_chk -dvv, recovery not yet perfect ! +========DIFF START======= +6c6 +< Status: checked,analyzed,optimized keys,sorted index pages +--- +> Status: changed +========DIFF END======= +testing idempotency +applying log +Differences in maria_chk -dvv, recovery not yet perfect ! +========DIFF START======= +6c6 +< Status: checked,analyzed,optimized keys,sorted index pages +--- +> Status: changed +========DIFF END======= +testing applying of CLRs to recreate table +applying log +Differences in maria_chk -dvv, recovery not yet perfect ! +========DIFF START======= +6c6 +< Status: checked,analyzed,optimized keys,sorted index pages +--- +> Status: changed +========DIFF END======= +TEST WITH ma_test2 -s -L -K -W -P -M -T -c -b32768 -H1 -t1 (commit at end) +TEST WITH ma_test2 -s -L -K -W -P -M -T -c -b32768 -H1 -t4 -A3 (additional aborted work) Dying on request without maria_commit()/maria_close() applying log Differences in maria_chk -dvv, recovery not yet perfect ! @@ -882,8 +1317,8 @@ Differences in maria_chk -dvv, recovery not yet perfect ! --- > Status: changed ========DIFF END======= -TEST WITH ma_test1 -s -M -T -c -N -b -H1 --testflag=1 (commit at end) -TEST WITH ma_test1 -s -M -T -c -N -b -H1 --testflag=2 --test-undo=4 (additional aborted work) +TEST WITH ma_test1 -s -M -T -c -N -b32768 -H1 --testflag=1 (commit at end) +TEST WITH ma_test1 -s -M -T -c -N -b32768 -H1 --testflag=2 --test-undo=4 (additional aborted work) Terminating after inserts Dying on request without maria_commit()/maria_close() applying log @@ -891,9 +1326,9 @@ testing idempotency applying log testing applying of CLRs to recreate table applying log -TEST WITH ma_test1 -s -M -T -c -N -b -H2 --testflag=3 (commit at end) +TEST WITH ma_test1 -s -M -T -c -N -b32768 -H2 --testflag=3 (commit at end) Terminating after updates -TEST WITH ma_test1 -s -M -T -c -N -b -H2 --testflag=4 --test-undo=4 (additional aborted work) +TEST WITH ma_test1 -s -M -T -c -N -b32768 -H2 --testflag=4 --test-undo=4 (additional aborted work) Terminating after deletes Dying on request without maria_commit()/maria_close() applying log @@ -901,9 +1336,9 @@ testing idempotency applying log testing applying of CLRs to recreate table applying log -TEST WITH ma_test1 -s -M -T -c -N -b -H2 --testflag=2 (commit at end) +TEST WITH ma_test1 -s -M -T -c -N -b32768 -H2 --testflag=2 (commit at end) Terminating after inserts -TEST WITH ma_test1 -s -M -T -c -N -b -H2 --testflag=3 --test-undo=4 (additional aborted work) +TEST WITH ma_test1 -s -M -T -c -N -b32768 -H2 --testflag=3 --test-undo=4 (additional aborted work) Terminating after updates Dying on request without maria_commit()/maria_close() applying log @@ -911,8 +1346,37 @@ testing idempotency applying log testing applying of CLRs to recreate table applying log -TEST WITH ma_test2 -s -L -K -W -P -M -T -c -b -H1 -t1 (commit at end) -TEST WITH ma_test2 -s -L -K -W -P -M -T -c -b -H1 -t2 -u4 (additional aborted work) +TEST WITH ma_test2 -s -L -K -W -P -M -T -c -b32768 -H1 -t1 (commit at end) +TEST WITH ma_test2 -s -L -K -W -P -M -T -c -b32768 -H1 -t2 -A4 (additional aborted work) +Dying on request without maria_commit()/maria_close() +applying log +Differences in maria_chk -dvv, recovery not yet perfect ! +========DIFF START======= +6c6 +< Status: checked,analyzed,optimized keys,sorted index pages +--- +> Status: changed +========DIFF END======= +testing idempotency +applying log +Differences in maria_chk -dvv, recovery not yet perfect ! +========DIFF START======= +6c6 +< Status: checked,analyzed,optimized keys,sorted index pages +--- +> Status: changed +========DIFF END======= +testing applying of CLRs to recreate table +applying log +Differences in maria_chk -dvv, recovery not yet perfect ! +========DIFF START======= +6c6 +< Status: checked,analyzed,optimized keys,sorted index pages +--- +> Status: changed +========DIFF END======= +TEST WITH ma_test2 -s -L -K -W -P -M -T -c -b32768 -H1 -t1 (commit at end) +TEST WITH ma_test2 -s -L -K -W -P -M -T -c -b32768 -H1 -t4 -A4 (additional aborted work) Dying on request without maria_commit()/maria_close() applying log Differences in maria_chk -dvv, recovery not yet perfect ! diff --git a/storage/maria/ma_update.c b/storage/maria/ma_update.c index 3d670e0d966..a1c003ea069 100644 --- a/storage/maria/ma_update.c +++ b/storage/maria/ma_update.c @@ -139,9 +139,10 @@ int maria_update(register MARIA_HA *info, const uchar *oldrec, uchar *newrec) If (*update_record)() fails, table will be marked corrupted so no need to revert the live checksum change. */ - info->state->checksum+= !share->now_transactional * - ((info->cur_row.checksum= (*share->calc_checksum)(info, newrec)) - - (info->new_row.checksum= (*share->calc_checksum)(info, oldrec))); + info->cur_row.checksum= (*share->calc_checksum)(info, newrec); + info->new_row.checksum= (*share->calc_checksum)(info, oldrec); + if (!share->now_transactional) + info->state->checksum+= info->cur_row.checksum - info->new_row.checksum; } { /* diff --git a/storage/maria/maria_chk.c b/storage/maria/maria_chk.c index 8eb9f135a36..40f4a3da03d 100644 --- a/storage/maria/maria_chk.c +++ b/storage/maria/maria_chk.c @@ -1275,7 +1275,7 @@ static void descript(HA_CHECK *param, register MARIA_HA *info, char *name) enum en_fieldtype type; MARIA_SHARE *share= info->s; char llbuff[22],llbuff2[22]; - DBUG_ENTER("describe"); + DBUG_ENTER("descript"); if (param->testflag & T_VERY_SILENT) { @@ -1329,12 +1329,6 @@ static void descript(HA_CHECK *param, register MARIA_HA *info, char *name) pos[-1]=0; /* Remove extra ',' */ } printf("Status: %s\n",buff); - if (share->base.auto_key) - { - printf("Auto increment key: %16d Last value: %18s\n", - share->base.auto_key, - llstr(share->state.auto_increment,llbuff)); - } if (share->options & (HA_OPTION_CHECKSUM | HA_OPTION_COMPRESS_RECORD)) printf("Checksum: %26s\n",llstr(info->state->checksum,llbuff)); ; @@ -1343,6 +1337,12 @@ static void descript(HA_CHECK *param, register MARIA_HA *info, char *name) if (share->options & HA_OPTION_PAGE_CHECKSUM) printf("Page checksums are used\n"); + if (share->base.auto_key) + { + printf("Auto increment key: %16d Last value: %18s\n", + share->base.auto_key, + llstr(share->state.auto_increment,llbuff)); + } } printf("Data records: %16s Deleted blocks: %18s\n", llstr(info->state->records,llbuff),llstr(info->state->del,llbuff2)); diff --git a/storage/maria/maria_def.h b/storage/maria/maria_def.h index 46564420deb..fac22ccf50b 100644 --- a/storage/maria/maria_def.h +++ b/storage/maria/maria_def.h @@ -40,7 +40,6 @@ #define MAX_NONMAPPED_INSERTS 1000 #define MARIA_MAX_TREE_LEVELS 32 -#define MARIA_MAX_CONTROL_FILE_LOCK_RETRY 30 /* Retry this many times */ struct st_transaction; @@ -106,7 +105,7 @@ typedef struct st_maria_state_info time_t check_time; /* Time for last check */ uint sortkey; /* sorted by this key (not used) */ uint open_count; - uint8 changed; /* Changed since mariachk */ + uint changed; /* Changed since maria_chk */ LSN create_rename_lsn; /**< LSN when table was last created/renamed */ /** @brief Log horizon when state was last updated on disk */ TRANSLOG_ADDRESS is_of_horizon; @@ -119,7 +118,11 @@ typedef struct st_maria_state_info #define MARIA_STATE_INFO_SIZE \ - (24 + LSN_STORE_SIZE*2 + 4 + 11*8 + 4*4 + 8 + 3*4 + 5*8) + (24 + 2 + LSN_STORE_SIZE*2 + 4 + 11*8 + 4*4 + 8 + 3*4 + 5*8) +#define MARIA_FILE_OPEN_COUNT_OFFSET 0 +#define MARIA_FILE_CHANGED_OFFSET 2 +#define MARIA_FILE_CREATE_RENAME_LSN_OFFSET 4 + #define MARIA_STATE_KEY_SIZE (8 + 4) #define MARIA_STATE_KEYBLOCK_SIZE 8 #define MARIA_STATE_KEYSEG_SIZE 12 @@ -405,8 +408,8 @@ typedef struct st_maria_row uchar *empty_bits, *field_lengths; uint *null_field_lengths; /* All null field lengths */ ulong *blob_lengths; /* Length for each blob */ - ulong base_length, normal_length, char_length, varchar_length, blob_length; - ulong head_length, total_length; + ulong base_length, min_length, normal_length, char_length, varchar_length; + ulong blob_length, head_length, total_length; size_t extents_buffer_length; /* Size of 'extents' buffer */ uint field_lengths_length; /* Length of data in field_lengths */ uint extents_count; /* number of extents in 'extents' */ @@ -451,6 +454,7 @@ struct st_maria_handler uchar *lastkey, *lastkey2; /* Last used search key */ uchar *first_mbr_key; /* Searhed spatial key */ uchar *rec_buff; /* Temp buffer for recordpack */ + uchar *blob_buff; /* Temp buffer for blobs */ uchar *int_keypos, /* Save position for next/previous */ *int_maxpos; /* -""- */ uchar *update_field_data; /* Used by update in rows-in-block */ @@ -473,7 +477,7 @@ struct st_maria_handler as they are not compatible with parallel repair */ ulong packed_length, blob_length; /* Length of found, packed record */ - size_t rec_buff_size; + size_t rec_buff_size, blob_buff_size; PAGECACHE_FILE dfile; /* The datafile */ IO_CACHE rec_cache; /* When cacheing records */ LIST open_list; @@ -483,6 +487,7 @@ struct st_maria_handler int lastinx; /* Last used index */ uint lastkey_length; /* Length of key in lastkey */ uint last_rkey_length; /* Last length in maria_rkey() */ + uint non_flushable_state; enum ha_rkey_function last_key_func; /* CONTAIN, OVERLAP, etc */ uint save_lastkey_length; uint pack_key_length; /* For MARIAMRG */ @@ -658,12 +663,6 @@ struct st_maria_handler #define MARIA_BLOCK_SIZE(key_length,data_pointer,key_pointer,block_size) (((((key_length)+(data_pointer)+(key_pointer))*4+(key_pointer)+2)/(block_size)+1)*(block_size)) #define MARIA_MAX_KEYPTR_SIZE 5 /* For calculating block lengths */ -#define MARIA_MIN_KEYBLOCK_LENGTH 50 /* When to split delete blocks */ - -#define MARIA_MIN_SIZE_BULK_INSERT_TREE 16384 /* this is per key */ -#define MARIA_MIN_ROWS_TO_USE_BULK_INSERT 100 -#define MARIA_MIN_ROWS_TO_DISABLE_INDEXES 100 -#define MARIA_MIN_ROWS_TO_USE_WRITE_CACHE 10 /* Marker for impossible delete link */ #define IMPOSSIBLE_PAGE_NO LL(0xFFFFFFFFFF) @@ -682,6 +681,16 @@ extern pthread_mutex_t THR_LOCK_maria; #define rw_unlock(A) {} #endif +/* Some tuning parameters */ +#define MARIA_MIN_KEYBLOCK_LENGTH 50 /* When to split delete blocks */ +#define MARIA_MIN_SIZE_BULK_INSERT_TREE 16384 /* this is per key */ +#define MARIA_MIN_ROWS_TO_USE_BULK_INSERT 100 +#define MARIA_MIN_ROWS_TO_DISABLE_INDEXES 100 +#define MARIA_MIN_ROWS_TO_USE_WRITE_CACHE 10 +/* Keep a small buffer for tables only using small blobs */ +#define MARIA_SMALL_BLOB_BUFFER 1024 +#define MARIA_MAX_CONTROL_FILE_LOCK_RETRY 30 /* Retry this many times */ + /* Some extern variables */ extern LIST *maria_open_list; @@ -799,6 +808,7 @@ extern int _ma_readinfo(MARIA_HA *info, int lock_flag, int check_keybuffer); extern int _ma_writeinfo(MARIA_HA *info, uint options); extern int _ma_test_if_changed(MARIA_HA *info); extern int _ma_mark_file_changed(MARIA_HA *info); +extern int _ma_mark_file_crashed(MARIA_SHARE *share); extern int _ma_decrement_open_count(MARIA_HA *info); extern int _ma_check_index(MARIA_HA *info, int inx); extern int _ma_search(MARIA_HA *info, MARIA_KEYDEF *keyinfo, uchar *key, diff --git a/storage/myisam/mi_test2.c b/storage/myisam/mi_test2.c index 6fb71feb1e7..e75a09702ce 100644 --- a/storage/myisam/mi_test2.c +++ b/storage/myisam/mi_test2.c @@ -334,9 +334,9 @@ int main(int argc, char *argv[]) if (use_blob) { if (i & 1) - put_blob_in_record(record+blob_pos,&blob_buffer); + put_blob_in_record(record2+blob_pos,&blob_buffer); else - bmove(record+blob_pos,read_record+blob_pos,8); + bmove(record2+blob_pos,read_record+blob_pos,8); } if (mi_update(file,read_record,record2)) { |