diff options
| author | Luke Chen <luke.chen@mongodb.com> | 2020-03-15 19:44:15 +1100 |
|---|---|---|
| committer | Evergreen Agent <no-reply@evergreen.mongodb.com> | 2020-03-15 09:41:33 +0000 |
| commit | 00502b11804516673266087d8d10292edb94a70a (patch) | |
| tree | 306f979dd5b14026a01855f6fb6a19ff1a16dddd /src/third_party | |
| parent | 8f6fa99d2c9eb037db6bb9522cf0fb4ca1d938e0 (diff) | |
| download | mongo-00502b11804516673266087d8d10292edb94a70a.tar.gz | |
Import wiredtiger: 187983a50c696eb217a780bb6b29e4bd3433c13b from branch mongodb-4.4
ref: 7e595e4a3a..187983a50c
for: 4.4.0-rc0
WT-3821 Python test case fails due to pinned timestamp warning
WT-4458 Only sweep btree handles that have been evicted from cache
WT-4671 Remove Limbo pages
WT-4674 Instantiate birthmark records using in-memory metadata
WT-4676 Search LAS for the history and compare to in-memory update
WT-4677 Re-initialize prepared updates
WT-4678 Write the test workload for relevant history in cache
WT-4681 Check if expired content is being swept from lookaside
WT-4683 Reverse the modifies when writing to LAS
WT-4685 Add new statistics for relevant history in cache
WT-4868 Aggregate btree write gen from leaf pages in salvage
WT-4954 Document duplicate backup cursors
WT-5001 Migrate Jenkins “wiredtiger-test-format-stress-sanitizer-ppc” job to Evergreen
WT-5045 Add more statistics tracking where checkpoint spends time
WT-5052 Enhance lookaside search functionality to handle WT_MODIFY records
WT-5058 Stop initializing a page with updates from lookaside
WT-5066 Extend test_las06 to support column store.
WT-5072 Modify __wt_rec_upd_select() to pull an update from lookaside
WT-5073 Review update chain traversals
WT-5089 test_las02.py fails: Unable to locate the correct record
WT-5092 Modify lookaside schema to add timestamp and transaction id to the key
WT-5098 Modify __wt_update_alloc() to accept NULL WT_ITEM values
WT-5141 Handle multiple modifies for the same key on lookaside eviction
WT-5146 Returning wrong record value for variable length columnar table
WT-5158 Remove assertion in __wt_find_lookaside_upd for cursor comparison
WT-5167 Fixing the missing pieces of page instantiation from las
WT-5186 Handle reading of LAS modifies on top of a birthmark record
WT-5191 Avoid LAS sweep server remove records that don't have disk image with proper version
WT-5203 lookaside history should no longer be preferentially evicted
WT-5212 Backup data validation tests
WT-5214 Verify potential incremental failures
WT-5215 Stress testing of incremental backup
WT-5222 Make checkpoint always choose skew newest
WT-5225 Create persistent file for History store
WT-5227 Enable python test cursor04 on relevant history feature branch
WT-5228 Several tests fail with incorrect or missing record after reconciliation changes
WT-5233 Prepared update not visible in las06
WT-5236 Enhance workgen to support draft lookaside stress workload
WT-5246 Update WiredTiger backup documentation
WT-5253 Add History store file to checkpoint target list.
WT-5254 Add aggregate timestamp info to the WT_REF.
WT-5256 Fix relevant history build after develop merged brought LIMBO reference
WT-5264 Add a delete record for every insert record into the WT history store to represent stop time pair
WT-5267 Enable storing start/stop time pair for each record in the history store
WT-5271 Remove WT_PAGE_LOOKASIDE structure.
WT-5279 Disable failing tests in the preferred way
WT-5280 Write all other versions to history store during checkpoint.
WT-5281 Assert that we're no longer writing tombstones to the history store
WT-5283 Eviction to create disk image with newest committed version.
WT-5284 Test to check eviction is writing the newest version to data store
WT-5286 Enable the data format to store time pair
WT-5289 Remove the code that does garbage collection from sweep server
WT-5290 Add new verbose flag and verbose logs for checkpoint garbage collection
WT-5291 Additional statistics for checkpoint garbage collection
WT-5292 Test the garbage collection by checkpoint
WT-5295 Disable remaining failing tests in durable history branch
WT-5298 Add a condition to verify the timestamp of a page during checkpoint of history store
WT-5300 Traverse the history store internal page for obsolete child pages by checkpoint
WT-5302 Begin writing validity window for new data format
WT-5303 Update checkpoint statistics for the new history store file.
WT-5308 Remove WT --enable-page-version-ts build configuration option
WT-5317 Eviction writes uncommitted metadata changes to disk
WT-5320 Save update restore to reuse lookaside eviction.
WT-5321 Update the stats and counters that represent lookaside inserts according to durable history
WT-5328 Evict obsolete history store page
WT-5333 Fix search not found issue due to skipped birthmark and rework txn_read
WT-5335 Read the start and stop time pairs back from cell when we reconstruct it on update chain
WT-5341 Clang sanitiser was triggered by checkpoint writing to the history store change
WT-5343 Show history store in the wt list
WT-5345 Show aggregated timestamp information for internal pages and leaf pages
WT-5347 Add ability for wt util to print historical versions of a key
WT-5353 Verify data continuity between history and data store
WT-5354 Verify history store key is not missing from the data store
WT-5355 Add the ability to dump a given btree as of a timestamp
WT-5361 Fix selecting aborted txn to write to disk
WT-5369 Don't use transaction ids from a page with a previous startup generation
WT-5385 Use WT_UPDATE_HISTORY_STORE to avoid repeatedly inserting updates to lookaside
WT-5386 Re-enable test_inmem01
WT-5388 Fix write squash statistic caused by change in reverse delta
WT-5392 Fall out of in-memory obsolete history page eviction
WT-5414 Change the usage from durable timestamp to commit timestamp during eviction.
WT-5416 Make sure that rollback to stable works on all objects in WT
WT-5417 Skip the clean pages of each tree with aggregated stop timestamp is less than stable timestamp
WT-5419 Create a TOMBSTONE update to represent the on-disk value state
WT-5420 Reconcile should consider TOMBSTONE update to replace or remove the on-disk value
WT-5421 Search the history store to find the value that can be replaced the on-disk value
WT-5422 Remove the history updates for reconciled data pages
WT-5423 Convert a history store modified value into a full update
WT-5424 Add statistics for rollback to stable operations
WT-5425 Add a new verbose flag to print the rollback to stable operation
WT-5426 Ensure rollback to stable is executed from recovery
WT-5427 Implement tests for rollback to stable
WT-5431 Merge develop branch to durable history branch
WT-5434 Remove develop-timestamp and add test for 4.0 and 4.2 pair to compatibility_test_for_mongodb_releases.sh after merge durable history to develop
WT-5441 Fix known visibility issues
WT-5446 Fix not writing to lookaside
WT-5448 Reconciliation wrongly overwrites the cell with default time pairs
WT-5451 Update reverse modify logic to always insert a modify into the history store
WT-5453 Fix cell packing for globally visible values.
WT-5454 Rename cache_overflow configuration option.
WT-5455 Fix test_hs03.py
WT-5457 single block reconciliation with saved updates can leak blocks
WT-5461 Search should not return onpage value if nothing is found by transaction read
WT-5462 Rebalance the Evergreen Python test buckets
WT-5464 Open reading history cursors on the user session
WT-5467 Make history store cursor part of each session, get rid of the cursor pool
WT-5469 Support mixing timestamped and non-timestamped updates
WT-5476 Continue writing tombstones when the tombstone is globally visable.
WT-5478 Insert records directly into history store without a txn
WT-5482 Increment cache usage when appending on-disk value to update list
WT-5484 Check visibility before saving updates for in memory reconciliation
WT-5489 page-read can race with threads locking in-memory page structures
WT-5491 Add an option to wt verify to confirm that no data exists after the stable timestamp
WT-5493 Re-enabling test_bug008
WT-5495 WT-5495 For column store, check the on-page value and history store even if the key is in the insert list
WT-5496 upd should be reset to NULL if all upd on the chain are not visible in txn read
WT-5500 Implement new history store format
WT-5501 Do not use default session to create a history store cursor when configuring
WT-5502 Re-add changes reverted by merge commit
WT-5503 We can only free updates inserted into history store after a full update.
WT-5508 Some of the txnids not cleared after restart
WT-5509 Infinite loop when reading from history store at early specific timestamp in test_util01
WT-5510 Fix test_hs01.py, test_hs06.py (test_hs_prepare_reads)
WT-5513 Don't consider TOMBSTONE/Stop time pair for history store reads
WT-5515 Enable test_hs06
WT-5516 Backup starts with base_write_gen 1
WT-5518 Split-parent code can race with other threads when checking the WT_REF.state
WT-5519 Apply version from datastore after finding a modify when possible.
WT-5522 Remove update free logic in hs_insert_updates.
WT-5523 Inserting history store needs to handle modify based on a tombstone
WT-5525 Free up 3B in the WT_REF structure
WT-5529 Improve usage of upd in txn.i
WT-5540 Call cursor disable bulk insert on first insert to history store.
WT-5541 Use snapshot isolation whenever we use history store cursors in verification
WT-5542 History store not using the on disk value as the base value for modify when key boundary crossed
WT-5547 Disable all the skipped rollback to stable tests for column store types
WT-5549 Fix the recovery rollback to stable and enable the passing tests for both row and variable columnar types
WT-5551 Fix the history store insert statistics
WT-5552 Checkpoint reconciliation and page splits free the WT_REF.addr field without locking
WT-5555 Update base write gen to the maximum write gen we have seen in recover
WT-5556 Verify of a file should verify its history store content too
WT-5558 Use durable timestamp from the update instead of start
WT-5563 Transactions ID are getting wiped which causes errors in WT Verify
WT-5565 Core dump is generated when running test_random_abort or test_random_direcio run on durable history branch
WT-5566 Update rollback to stable tests to use new statistics
WT-5567 Fix an assert in txn_read always be true
WT-5569 Update WiredTiger source code to include 2020 copyright notices for durable history
WT-5570 Refactor the __upd_alloc_tombstone() according to the new use in durable history
WT-5574 Rolling back a prepared transaction with `cursor_copy` results in a use-after-free
WT-5575 Fix the test_durable_ts01 test to expect older data after recovery
WT-5576 Temporarily add lookaside score stat and cache_overflow config option
WT-5579 Fix Evergreen memory sanitizer test failure
WT-5581 Address sanitizer test failure running bloom filter testing
WT-5582 Long unit testing sweeping cursors failing
WT-5583 Applying operations in recovery encounters unexpected operation type
WT-5587 Limit how many checkpoints are dropped by a subsequent checkpoint
WT-5589 force_stop on duplicate cursor open not returning error
WT-5590 Fix spellings so s_string passes
WT-5593 test/format assertion failure addr->size != 0
WT-5595 test/format data mismatch errors
WT-5596 Increase format stress testing scope.
WT-5597 Fix the history store file access during recovery
WT-5598 __verify_timestamp_to_pretty_string uses local buffer which is freed before access
WT-5602 Rollback transaction core dumped with upd->start_ts >= unpack->stop_ts
WT-5603 test/format assertion failure while discarding in-memory page
WT-5605 Update test checkpoint to no longer use checkpoint cursors
WT-5607 Successful 'verify -h' calls are returning not found
WT-5610 Fix assertion for reconciling fixed length column store
WT-5611 Don't write updates with different commit and durable timestamps to data store
WT-5612 Remove history store values for non-timestamp tables
WT-5613 Remove birthmark update type
WT-5615 Coverity: Read of uninitialised value
WT-5618 Skip timestamp range overlap check if start timestamp is zero
WT-5620 Skip the history store TOMBSTONE only for rollback to stable operation
WT-5624 Incremental unit test should use offset/length ranges
WT-5625 corruption detected during validation - root page's aggregated timestamp incorrect
WT-5626 Remove assert which checks for newer updates in the history store
WT-5628 rollback_to_stable failed with no such file
WT-5631 Recovery rollback to stable for timestamped logged tables
WT-5632 Don't write stop_ts of 1 for non-timestamped delete
WT-5633 Fix another assertion for reconciling fixed length column store
WT-5636 prefix compression is slow in the history-store access pattern
WT-5638 Ignore checking visibility of history store updates as they are implicitly committed
WT-5640 test_wt2323_join_visibility fail when processing consecutive tombstones in __wt_hs_insert_updates
WT-5641 Clear history store content when deleting a key due to a globally visible tombstone
WT-5644 Appending onpage value to an aborted update triggers an assert
WT-5647 replace the WT_REF structure's WT_REF_READING state with a flag
WT-5648 Add a leaf or internal page type flag to the WT_REF structure
WT-5649 Refactor WT_REF locking, review all WT_REF.addr reads for locking issues
WT-5650 Fix a race condition between reading the WT_PAGE.modify field and the page being dirtied.
WT-5651 Fix the RTS assert to consider error scenarios of search
WT-5654 Add version information to the history store key format
WT-5658 Fix heap-use-after-free in parent split code
WT-5665 Data mismatch bug when running new version of test_checkpoint with timestamps
WT-5666 Deleting a chunk of the namespace changes the WT_REF type
WT-5667 Remove usage of checkpoint cursor in test/format
WT-5668 Prepare support with durable history: implement data format changes
WT-5678 Fix infinite looping behaviour in history store cursor positioning
WT-5680 segfault when dereferencing NULL addr while reconciling
WT-5682 Ensure that we can't apply modifies on top of tombstones
WT-5684 overflow values must be discarded when there is no update for a key
WT-5685 Set aspell dictionary to en_US
WT-5688 Memory leak detected during page overflow read
WT-5689 reduce work required for the cursor-pinned test.
WT-5690 Rollback to stable assertion failure regarding update visibility
WT-5692 Revert a test change to fix a Python hang
WT-5695 Fixed incremental backup example to use O_CREAT in the backup range case
WT-5696 test_timestamp_abort fails with data mismatch
WT-5698 Disabling bulk cursor changes broke a Jenkins compile
WT-5699 Refactor incremental backup RANGE code
WT-5700 Add smoke test script for incremental backup stress test
WT-5701 If an out-of-order update masks an on-disk value, don't append it
WT-5704 Incremental backup smoke test core dumped
WT-5706 Fix csuite-incr-backup-test calculation of value sizes
WT-5707 Reduce the test load for checkpoint stress test
WT-5712 Ensure WT command line utility treats history store consistently
WT-5713 Fix failures so test_durable_rollback_stable.py can be enabled
WT-5719 Incremental backup metadata should quote the ID string
WT-5722 Incremental backup should do a name check on identifiers
WT-5745 Don't copy a value into tombstones
WT-5747 Cope with updates out of timestamp order
WT-5753 Fix divide by zero error in test/csuite/test_incr_backup
WT-5756 heap-use-after-free in __wt_row_modify
WT-5762 Make test_hs10 more robust
WT-5767 Fix search_near invocations for history store
WT-5771 make-check-msan-test failed with use-of-uninitialized-value error on RHEL 8.0
WT-5774 Move stress test tasks into a separate build variant
WT-5775 Fix leak of updates from history store
WT-5777 Add statistic for tracking history store deletions due to key removal
WT-5780 Fix timestamp_index_build.js in noPassthrough suite.
WT-5781 Fix basic.js in parallel suite
WT-5786 Detect if a file is too small to read a descriptor block
WT-5792 Dump and verify can't see the history store
WT-5795 Disable assertion that inserts to the history store are unique
WT-5798 Check that the history store file exists before performing rollback to stable
WT-5799 Dont assume we have ordered timestamps when doing rec append original value
WT-5800 Temporarily disable history store verify
WT-5806 Perform rollback to stable on a clean shutdown
WT-5809 Invariant failure: stable timestamp does not equal appliedThrough timestamp
WT-5820 Change format.sh to forcibly quit if a test runs out of disk space.
WT-5822 Don't evict metadata updates from a running checkpoint
WT-5827 Enable test_schema_abort
WT-5830 Enable two c tests in evergreen
Diffstat (limited to 'src/third_party')
225 files changed, 14066 insertions, 7738 deletions
diff --git a/src/third_party/wiredtiger/NEWS b/src/third_party/wiredtiger/NEWS index 1dc9dc6eac6..46fc48ffc5c 100644 --- a/src/third_party/wiredtiger/NEWS +++ b/src/third_party/wiredtiger/NEWS @@ -2460,7 +2460,7 @@ below: now it will be used if the shared_cache configuration option is included. * Add the ability to specify a per-connection reserved size for cache - pools. Ensure cache pool reconfiguration is honoured quickly. + pools. Ensure cache pool reconfiguration is honored quickly. * Rework hazard pointer coupling during cursor walks to be more efficient. diff --git a/src/third_party/wiredtiger/SConstruct b/src/third_party/wiredtiger/SConstruct index ab5f3ab49cc..fd04adfbc39 100644 --- a/src/third_party/wiredtiger/SConstruct +++ b/src/third_party/wiredtiger/SConstruct @@ -407,7 +407,8 @@ shim = env.Library("window_shim", examples = [ "ex_access", - "ex_all", + # Temporarily disabled + # "ex_all", "ex_async", "ex_call_center", "ex_config_parse", diff --git a/src/third_party/wiredtiger/bench/workgen/runner/evict-btree-lookaside.py b/src/third_party/wiredtiger/bench/workgen/runner/evict-btree-hs.py index fd9cfd51fb6..d516074b2bb 100755 --- a/src/third_party/wiredtiger/bench/workgen/runner/evict-btree-lookaside.py +++ b/src/third_party/wiredtiger/bench/workgen/runner/evict-btree-hs.py @@ -26,7 +26,7 @@ # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. -# This benchmark is designed to stress disk access to the LAS/History Store file. +# This benchmark is designed to stress disk access to the history store file. # This is achieved through: # - Long running transactions consisting of read and update operations. # - Low cache size (~20%) for a reasonably sized WT table with large documents. diff --git a/src/third_party/wiredtiger/build_posix/aclocal/options.m4 b/src/third_party/wiredtiger/build_posix/aclocal/options.m4 index 132a0cb609f..2a20716a9bc 100644 --- a/src/third_party/wiredtiger/build_posix/aclocal/options.m4 +++ b/src/third_party/wiredtiger/build_posix/aclocal/options.m4 @@ -185,20 +185,6 @@ pthread_adaptive|pthreads_adaptive) esac AC_MSG_RESULT($with_spinlock) -AH_TEMPLATE(HAVE_PAGE_VERSION_TS, - [Define to 1 to enable writing timestamp version page formats.]) -AC_MSG_CHECKING(if --enable-page-version-ts option specified) -AC_ARG_ENABLE(page-version-ts, - [AS_HELP_STRING([--enable-page-version-ts], - [Configure for timestamp version page formats])], - r=$enableval, r=no) -case "$r" in -no) wt_cv_enable_page_version_ts=no;; -*) AC_DEFINE(HAVE_PAGE_VERSION_TS) - wt_cv_enable_page_version_ts=yes;; -esac -AC_MSG_RESULT($wt_cv_enable_page_version_ts) - AC_MSG_CHECKING(if --enable-strict option specified) AC_ARG_ENABLE(strict, [AS_HELP_STRING([--enable-strict], diff --git a/src/third_party/wiredtiger/build_win/wiredtiger_config.h b/src/third_party/wiredtiger/build_win/wiredtiger_config.h index c5c0dfda580..8c4e32875a4 100644 --- a/src/third_party/wiredtiger/build_win/wiredtiger_config.h +++ b/src/third_party/wiredtiger/build_win/wiredtiger_config.h @@ -73,9 +73,6 @@ /* Define to 1 to disable any crc32 hardware support. */ /* #undef HAVE_NO_CRC32_HARDWARE */ -/* Define to 1 to enable writing timestamp version page formats. */ -/* #undef HAVE_PAGE_VERSION_TS */ - /* Define to 1 if pthread condition variables support monotonic clocks. */ /* #undef HAVE_PTHREAD_COND_MONOTONIC */ diff --git a/src/third_party/wiredtiger/dist/api_data.py b/src/third_party/wiredtiger/dist/api_data.py index 45cca0ef829..1f4d42d3197 100644 --- a/src/third_party/wiredtiger/dist/api_data.py +++ b/src/third_party/wiredtiger/dist/api_data.py @@ -461,7 +461,19 @@ connection_runtime_config = [ this size, a panic will be triggered. The default value means that the cache overflow file is unbounded and may use as much space as the filesystem will accommodate. The minimum non-zero setting is - 100MB.''', # !!! Must match WT_LAS_FILE_MIN + 100MB.''', # !!! TODO: WT-5585 To be removed when we switch to history_store config + min='0') + ]), + Config('history_store', '', r''' + history store configuration options''', + type='category', subconfig=[ + Config('file_max', '0', r''' + The maximum number of bytes that WiredTiger is allowed to use for + its history store mechanism. If the history store file exceeds + this size, a panic will be triggered. The default value means that + the history store file is unbounded and may use as much space as + the filesystem will accommodate. The minimum non-zero setting is + 100MB.''', # !!! Must match WT_HS_FILE_MIN min='0') ]), Config('cache_overhead', '8', r''' @@ -506,7 +518,7 @@ connection_runtime_config = [ type='boolean'), Config('eviction', 'false', r''' if true, modify internal algorithms to change skew to force - lookaside eviction to happen more aggressively. This includes but + history store eviction to happen more aggressively. This includes but is not limited to not skewing newest, not favoring leaf pages, and modifying the eviction score mechanism.''', type='boolean'), @@ -687,7 +699,7 @@ connection_runtime_config = [ intended for use with internal stress testing of WiredTiger.''', type='list', undoc=True, choices=[ - 'aggressive_sweep', 'checkpoint_slow', 'lookaside_sweep_race', + 'aggressive_sweep', 'checkpoint_slow', 'history_store_sweep_race', 'split_1', 'split_2', 'split_3', 'split_4', 'split_5', 'split_6', 'split_7', 'split_8']), Config('verbose', '', r''' @@ -698,6 +710,7 @@ connection_runtime_config = [ 'backup', 'block', 'checkpoint', + 'checkpoint_gc', 'checkpoint_progress', 'compact', 'compact_progress', @@ -708,8 +721,8 @@ connection_runtime_config = [ 'fileops', 'handleops', 'log', - 'lookaside', - 'lookaside_activity', + 'history_store', + 'history_store_activity', 'lsm', 'lsm_manager', 'metadata', @@ -720,6 +733,7 @@ connection_runtime_config = [ 'reconcile', 'recovery', 'recovery_progress', + 'rts', 'salvage', 'shared_cache', 'split', @@ -1346,13 +1360,19 @@ methods = { 'WT_SESSION.upgrade' : Method([]), 'WT_SESSION.verify' : Method([ Config('dump_address', 'false', r''' - Display addresses and page types as pages are verified, - using the application's message handler, intended for debugging''', + Display page addresses, start and stop time pairs and page types as + pages are verified, using the application's message handler, + intended for debugging''', type='boolean'), Config('dump_blocks', 'false', r''' Display the contents of on-disk blocks as they are verified, using the application's message handler, intended for debugging''', type='boolean'), + Config('dump_history', 'false', r''' + Display a key's values along with its start and stop time pairs as + they are verified against the history store, using the application's + message handler, intended for debugging''', + type='boolean'), Config('dump_layout', 'false', r''' Display the layout of the files as they are verified, using the application's message handler, intended for debugging; requires @@ -1366,11 +1386,18 @@ methods = { Display the contents of in-memory pages as they are verified, using the application's message handler, intended for debugging''', type='boolean'), + Config('history_store', 'false', r''' + Verify the history store.''', + type='boolean'), + Config('stable_timestamp', 'false', r''' + Ensure that no data has a start timestamp after the stable timestamp, + to be run after rollback_to_stable.''', + type='boolean'), Config('strict', 'false', r''' Treat any verification problem as an error; by default, verify will warn, but not fail, in the case of errors that won't affect future behavior (for example, a leaked block)''', - type='boolean') + type='boolean'), ]), 'WT_SESSION.begin_transaction' : Method([ @@ -1421,9 +1448,6 @@ methods = { read timestamp will be rounded up to the oldest timestamp''', type='boolean'), ]), - Config('snapshot', '', r''' - use a named, in-memory snapshot, see - @ref transaction_named_snapshots'''), Config('sync', '', r''' whether to sync log records when the transaction commits, inherited from ::wiredtiger_open \c transaction_sync''', @@ -1513,28 +1537,6 @@ methods = { type='boolean'), ]), -'WT_SESSION.snapshot' : Method([ - Config('drop', '', r''' - if non-empty, specifies which snapshots to drop. Where a group - of snapshots are being dropped, the order is based on snapshot - creation order not alphanumeric name order''', - type='category', subconfig=[ - Config('all', 'false', r''' - drop all named snapshots''', type='boolean'), - Config('before', '', r''' - drop all snapshots up to but not including the specified name'''), - Config('names', '', r''' - drop specific named snapshots''', type='list'), - Config('to', '', r''' - drop all snapshots up to and including the specified name'''), - ]), - Config('include_updates', 'false', r''' - make updates from the current transaction visible to users of the - named snapshot. Transactions started with such a named snapshot are - restricted to being read-only''', type='boolean'), - Config('name', '', r'''specify a name for the snapshot'''), -]), - 'WT_CONNECTION.add_collator' : Method([]), 'WT_CONNECTION.add_compressor' : Method([]), 'WT_CONNECTION.add_data_source' : Method([]), diff --git a/src/third_party/wiredtiger/dist/filelist b/src/third_party/wiredtiger/dist/filelist index 6cf9369fecd..118c3e114a0 100644 --- a/src/third_party/wiredtiger/dist/filelist +++ b/src/third_party/wiredtiger/dist/filelist @@ -49,7 +49,6 @@ src/btree/col_srch.c src/btree/row_key.c src/btree/row_modify.c src/btree/row_srch.c -src/cache/cache_las.c src/checksum/arm64/crc32-arm64.c ARM64_HOST src/checksum/power8/crc32.sx POWERPC_HOST src/checksum/power8/crc32_wrapper.c POWERPC_HOST @@ -98,6 +97,7 @@ src/evict/evict_file.c src/evict/evict_lru.c src/evict/evict_page.c src/evict/evict_stat.c +src/history/hs.c src/log/log.c src/log/log_auto.c src/log/log_slot.c @@ -209,7 +209,6 @@ src/txn/txn.c src/txn/txn_ckpt.c src/txn/txn_ext.c src/txn/txn_log.c -src/txn/txn_nsnap.c src/txn/txn_recover.c src/txn/txn_rollback_to_stable.c src/txn/txn_timestamp.c diff --git a/src/third_party/wiredtiger/dist/s_clang-scan.diff b/src/third_party/wiredtiger/dist/s_clang-scan.diff index a78a7cf1bee..fc5ce463cd9 100644 --- a/src/third_party/wiredtiger/dist/s_clang-scan.diff +++ b/src/third_party/wiredtiger/dist/s_clang-scan.diff @@ -1,33 +1,27 @@ In file included from src/block/block_write.c:9: -In file included from ./src/include/wt_internal.h:412: -./src/include/intpack.i:194:4: warning: Assigned value is garbage or undefined - p = *pp; - ^ ~~~ +In file included from ./src/include/wt_internal.h:418: +./src/include/intpack.i:193:7: warning: Assigned value is garbage or undefined + p = *pp; + ^ ~~~ 1 warning generated. -src/conn/conn_capacity.c:310:2: warning: Value stored to 'capacity' is never read - capacity = steal_capacity = 0; - ^ ~~~~~~~~~~~~~~~~~~ +In file included from src/btree/col_modify.c:9: +In file included from ./src/include/wt_internal.h:423: +./src/include/mutex.i:158:13: warning: Null pointer passed as an argument to a 'nonnull' parameter + return (pthread_mutex_trylock(&t->lock)); + ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +1 warning generated. +src/conn/conn_capacity.c:291:5: warning: Value stored to 'capacity' is never read + capacity = steal_capacity = 0; + ^ ~~~~~~~~~~~~~~~~~~ +1 warning generated. +src/reconcile/rec_col.c:1079:25: warning: Null pointer passed as an argument to a 'nonnull' parameter + memcmp(last.value->data, data, size) == 0))) { + ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1 warning generated. -src/reconcile/rec_col.c:700:2: warning: Value stored to 'start_ts' is never read - start_ts = WT_TS_MAX; - ^ ~~~~~~~~~ -src/reconcile/rec_col.c:701:2: warning: Value stored to 'start_txn' is never read - start_txn = WT_TXN_MAX; - ^ ~~~~~~~~~~ -src/reconcile/rec_col.c:702:2: warning: Value stored to 'stop_ts' is never read - stop_ts = WT_TS_NONE; - ^ ~~~~~~~~~~ -src/reconcile/rec_col.c:703:2: warning: Value stored to 'stop_txn' is never read - stop_txn = WT_TS_NONE; - ^ ~~~~~~~~~~ -src/reconcile/rec_col.c:1199:9: warning: Null pointer passed as an argument to a 'nonnull' parameter - memcmp( - ^~~~~~~ -5 warnings generated. In file included from src/reconcile/rec_write.c:9: -In file included from ./src/include/wt_internal.h:417: -./src/include/mutex.i:187:13: warning: Null pointer passed as an argument to a 'nonnull' parameter - if ((ret = pthread_mutex_unlock(&t->lock)) != 0) - ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +In file included from ./src/include/wt_internal.h:423: +./src/include/mutex.i:184:16: warning: Null pointer passed as an argument to a 'nonnull' parameter + if ((ret = pthread_mutex_unlock(&t->lock)) != 0) + ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1 warning generated. ar: `u' modifier ignored since `D' is the default (see `U') diff --git a/src/third_party/wiredtiger/dist/s_define.list b/src/third_party/wiredtiger/dist/s_define.list index 01fa0a7846b..5a5c524c777 100644 --- a/src/third_party/wiredtiger/dist/s_define.list +++ b/src/third_party/wiredtiger/dist/s_define.list @@ -35,6 +35,7 @@ WT_ERR_ERROR_OK WT_EXT_FOREACH_OFF WT_HANDLE_CLOSED WT_HANDLE_NULLABLE +WT_HS_COMPRESSOR WT_LOG_SLOT_ACTIVE WT_LOG_SLOT_BITS WT_LOG_SLOT_JOIN_MASK @@ -44,7 +45,6 @@ WT_LOG_SLOT_MAXBITS WT_LOG_SLOT_UNBUFFERED_ISSET WT_LOG_V3_MAJOR WT_LOG_V3_MINOR -WT_LOOKASIDE_COMPRESSOR WT_OPTRACK_BUFSIZE WT_OPTRACK_MAXRECS WT_PACKED_STRUCT_BEGIN @@ -82,6 +82,7 @@ WT_TRACK_OP WT_TRACK_OP_END WT_TRACK_OP_INIT WT_TRET_ERROR_OK +WT_TXN_UPDATE WT_UPDATE_SIZE WT_USE_OPENAT WT_WITH_LOCK_NOWAIT diff --git a/src/third_party/wiredtiger/dist/s_funcs.list b/src/third_party/wiredtiger/dist/s_funcs.list index 0218937fffc..fe19e596bf4 100644 --- a/src/third_party/wiredtiger/dist/s_funcs.list +++ b/src/third_party/wiredtiger/dist/s_funcs.list @@ -16,8 +16,8 @@ __wt_config_getone __wt_cursor_get_raw_value __wt_debug_addr __wt_debug_addr_print -__wt_debug_cursor_las __wt_debug_cursor_page +__wt_debug_cursor_tree_hs __wt_debug_offset __wt_debug_set_verbose __wt_debug_tree diff --git a/src/third_party/wiredtiger/dist/s_string b/src/third_party/wiredtiger/dist/s_string index b7b0a4bbdba..417d99a20cd 100755 --- a/src/third_party/wiredtiger/dist/s_string +++ b/src/third_party/wiredtiger/dist/s_string @@ -22,7 +22,7 @@ type aspell > /dev/null 2>&1 || { # catalogs and generating a shorter list on any single system will break other # systems. replace() { - aspell --mode=ccpp --lang=en list < ../$1 | + aspell --mode=ccpp --lang=en_US list < ../$1 | sort -u | comm -12 /dev/stdin s_string.ok } @@ -33,7 +33,7 @@ check() { # Strip out git hashes, which are seven character hex strings. # Strip out double quote char literals ('"'), they confuse aspell. sed -e 's/ [0-9a-f]\{7\} / /g' -e "s/'\"'//g" ../$2 | - aspell --lang=en $1 list | + aspell --lang=en_US $1 list | sort -u | comm -23 /dev/stdin s_string.ok > $t test -s $t && { diff --git a/src/third_party/wiredtiger/dist/s_string.ok b/src/third_party/wiredtiger/dist/s_string.ok index a1d98a7af82..7a841259fec 100644 --- a/src/third_party/wiredtiger/dist/s_string.ok +++ b/src/third_party/wiredtiger/dist/s_string.ok @@ -174,6 +174,7 @@ HHHHLL HHHLL HILQr HOTBACKUP +HSdump Hendrik HyperLevelDB ID's @@ -210,7 +211,6 @@ Kanowski's Kounavis LANGID LAS -LASdump LF LLLLLL LLLLLLL @@ -267,6 +267,7 @@ Mutex MySecret NEEDKEY NEEDVALUE +NNN NOLINT NOLINTNEXTLINE NOLL @@ -313,6 +314,7 @@ PowerPC Pre Preload Prepend +Prev Qsort RCS RDNOLOCK @@ -363,6 +365,7 @@ Sevii SiH Skiplist SleepConditionVariableCS +SmallVector Solaris Spinlock Spinlocks @@ -441,6 +444,7 @@ WiredTiger WiredTiger's WiredTigerCheckpoint WiredTigerException +WiredTigerHS WiredTigerInit WiredTigerLAS WiredTigerLog @@ -793,6 +797,7 @@ func funcid fvisibility fwrite +gc gcc gdb ge @@ -825,6 +830,7 @@ hhh highjack hilq hotbackup +hs hselasky html huffman @@ -931,6 +937,7 @@ libwiredtiger linkers llll llu +llvm loadtext localTime localkey @@ -961,6 +968,7 @@ lsnappy lt lu lwsync +lx lz lzo mT @@ -1111,6 +1119,7 @@ prepended prepending presize presync +prev prevlsn primary's printf @@ -1166,6 +1175,7 @@ regionp reinitialization relocked repl +resizable resize resizing ret @@ -1180,6 +1190,7 @@ rotN rotn rp rpc +rts ru run's runtime @@ -1201,6 +1212,7 @@ setstr setv setvbuf sfence +signalled sii sizeof sizep @@ -1264,6 +1276,8 @@ syscall sysinfo sz t's +tK +tM tV tablename tcbench diff --git a/src/third_party/wiredtiger/dist/s_void b/src/third_party/wiredtiger/dist/s_void index 5591e90a298..cbff017ebf9 100755 --- a/src/third_party/wiredtiger/dist/s_void +++ b/src/third_party/wiredtiger/dist/s_void @@ -51,6 +51,7 @@ func_ok() -e '/int __tree_walk_skip_count_callback$/d' \ -e '/int __txn_rollback_to_stable_custom_skip$/d' \ -e '/int __win_terminate$/d' \ + -e '/int wiredtiger_calc_modify/d' \ -e '/int __wt_block_compact_end$/d' \ -e '/int __wt_block_compact_start$/d' \ -e '/int __wt_block_manager_size$/d' \ @@ -117,7 +118,7 @@ func_ok() -e '/int snappy_pre_size$/d' \ -e '/int snappy_terminate$/d' \ -e '/int subtest_error_handler$/d' \ - -e '/int test_las_workload$/d' \ + -e '/int test_hs_workload$/d' \ -e '/int uri2name$/d' \ -e '/int usage$/d' \ -e '/int util_err$/d' \ diff --git a/src/third_party/wiredtiger/dist/stat_data.py b/src/third_party/wiredtiger/dist/stat_data.py index aa9ed2cd3fa..0bdc6ddce84 100644 --- a/src/third_party/wiredtiger/dist/stat_data.py +++ b/src/third_party/wiredtiger/dist/stat_data.py @@ -72,6 +72,10 @@ class DhandleStat(Stat): prefix = 'data-handle' def __init__(self, name, desc, flags=''): Stat.__init__(self, name, DhandleStat.prefix, desc, flags) +class HistoryStat(Stat): + prefix = 'history' + def __init__(self, name, desc, flags=''): + Stat.__init__(self, name, HistoryStat.prefix, desc, flags) class JoinStat(Stat): prefix = '' # prefix is inserted dynamically def __init__(self, name, desc, flags=''): @@ -205,7 +209,7 @@ connection_stats = [ CacheStat('cache_bytes_internal', 'tracked bytes belonging to internal pages in the cache', 'no_clear,no_scale,size'), CacheStat('cache_bytes_inuse', 'bytes currently in the cache', 'no_clear,no_scale,size'), CacheStat('cache_bytes_leaf', 'tracked bytes belonging to leaf pages in the cache', 'no_clear,no_scale,size'), - CacheStat('cache_bytes_lookaside', 'bytes belonging to the cache overflow table in the cache', 'no_clear,no_scale,size'), + CacheStat('cache_bytes_hs', 'bytes belonging to the history store table in the cache', 'no_clear,no_scale,size'), CacheStat('cache_bytes_max', 'maximum bytes configured', 'no_clear,no_scale,size'), CacheStat('cache_bytes_other', 'bytes not belonging to page images in the cache', 'no_clear,no_scale,size'), CacheStat('cache_bytes_read', 'bytes read into cache', 'size'), @@ -283,15 +287,19 @@ connection_stats = [ CacheStat('cache_hazard_checks', 'hazard pointer check calls'), CacheStat('cache_hazard_max', 'hazard pointer maximum array length', 'max_aggregate,no_scale'), CacheStat('cache_hazard_walks', 'hazard pointer check entries walked'), + CacheStat('cache_hs_insert', 'history store table insert calls'), + CacheStat('cache_hs_key_truncate_onpage_removal', 'history store key truncation due to the key being removed from the data page'), + CacheStat('cache_hs_key_truncate_mix_ts', 'history store key truncation due to mixed timestamps'), + CacheStat('cache_hs_ondisk', 'history store table on-disk size', 'no_clear,no_scale,size'), + CacheStat('cache_hs_ondisk_max', 'history store table max on-disk size', 'no_clear,no_scale,size'), + CacheStat('cache_hs_read', 'history store table reads'), + CacheStat('cache_hs_read_miss', 'history store table reads missed'), + CacheStat('cache_hs_read_squash', 'history store table reads requiring squashed modifies'), + CacheStat('cache_hs_remove_key_truncate', 'history store table remove calls due to key truncation'), + CacheStat('cache_hs_score', 'history store score', 'no_clear,no_scale'), + CacheStat('cache_hs_write_squash', 'history store table writes requiring squashed modifies'), CacheStat('cache_inmem_split', 'in-memory page splits'), CacheStat('cache_inmem_splittable', 'in-memory page passed criteria to be split'), - CacheStat('cache_lookaside_cursor_wait_application', 'cache overflow cursor application thread wait time (usecs)'), - CacheStat('cache_lookaside_cursor_wait_internal', 'cache overflow cursor internal thread wait time (usecs)'), - CacheStat('cache_lookaside_entries', 'cache overflow table entries', 'no_clear,no_scale'), - CacheStat('cache_lookaside_insert', 'cache overflow table insert calls'), - CacheStat('cache_lookaside_ondisk', 'cache overflow table on-disk size', 'no_clear,no_scale,size'), - CacheStat('cache_lookaside_ondisk_max', 'cache overflow table max on-disk size', 'no_clear,no_scale,size'), - CacheStat('cache_lookaside_remove', 'cache overflow table remove calls'), CacheStat('cache_lookaside_score', 'cache overflow score', 'no_clear,no_scale'), CacheStat('cache_overhead', 'percentage overhead', 'no_clear,no_scale'), CacheStat('cache_pages_dirty', 'tracked dirty pages in the cache', 'no_clear,no_scale'), @@ -302,17 +310,12 @@ connection_stats = [ CacheStat('cache_read_app_time', 'application threads page read from disk to cache time (usecs)'), CacheStat('cache_read_deleted', 'pages read into cache after truncate'), CacheStat('cache_read_deleted_prepared', 'pages read into cache after truncate in prepare state'), - CacheStat('cache_read_lookaside', 'pages read into cache requiring cache overflow entries'), - CacheStat('cache_read_lookaside_checkpoint', 'pages read into cache requiring cache overflow for checkpoint'), - CacheStat('cache_read_lookaside_delay', 'pages read into cache with skipped cache overflow entries needed later'), - CacheStat('cache_read_lookaside_delay_checkpoint', 'pages read into cache with skipped cache overflow entries needed later by checkpoint'), - CacheStat('cache_read_lookaside_skipped', 'pages read into cache skipping older cache overflow entries'), CacheStat('cache_read_overflow', 'overflow pages read into cache'), CacheStat('cache_timed_out_ops', 'operations timed out waiting for space in cache'), CacheStat('cache_write', 'pages written from cache'), CacheStat('cache_write_app_count', 'application threads page write from cache to disk count'), CacheStat('cache_write_app_time', 'application threads page write from cache to disk time (usecs)'), - CacheStat('cache_write_lookaside', 'page written requiring cache overflow records'), + CacheStat('cache_write_hs', 'page written requiring history store records'), CacheStat('cache_write_restore', 'pages written requiring in-memory restoration'), ########################################## @@ -383,6 +386,13 @@ connection_stats = [ DhandleStat('dh_sweeps', 'connection sweeps'), ########################################## + # History statistics + ########################################## + HistoryStat('hs_gc_pages_evict', 'history pages added for eviction during garbage collection'), + HistoryStat('hs_gc_pages_removed', 'history pages removed for garbage collection'), + HistoryStat('hs_gc_pages_visited', 'history pages visited for garbage collection'), + + ########################################## # Locking statistics ########################################## LockStat('lock_checkpoint_count', 'checkpoint lock acquisitions'), @@ -556,7 +566,13 @@ connection_stats = [ TxnStat('txn_checkpoint', 'transaction checkpoints'), TxnStat('txn_checkpoint_fsync_post', 'transaction fsync calls for checkpoint after allocating the transaction ID'), TxnStat('txn_checkpoint_fsync_post_duration', 'transaction fsync duration for checkpoint after allocating the transaction ID (usecs)', 'no_clear,no_scale'), + TxnStat('txn_hs_ckpt_duration', 'transaction checkpoint history store file duration (usecs)'), TxnStat('txn_checkpoint_generation', 'transaction checkpoint generation', 'no_clear,no_scale'), + TxnStat('txn_checkpoint_prep_max', 'transaction checkpoint prepare max time (msecs)', 'no_clear,no_scale'), + TxnStat('txn_checkpoint_prep_min', 'transaction checkpoint prepare min time (msecs)', 'no_clear,no_scale'), + TxnStat('txn_checkpoint_prep_recent', 'transaction checkpoint prepare most recent time (msecs)', 'no_clear,no_scale'), + TxnStat('txn_checkpoint_prep_running', 'transaction checkpoint prepare currently running', 'no_clear,no_scale'), + TxnStat('txn_checkpoint_prep_total', 'transaction checkpoint prepare total time (msecs)', 'no_clear,no_scale'), TxnStat('txn_checkpoint_running', 'transaction checkpoint currently running', 'no_clear,no_scale'), TxnStat('txn_checkpoint_scrub_target', 'transaction checkpoint scrub dirty target', 'no_clear,no_scale'), TxnStat('txn_checkpoint_scrub_time', 'transaction checkpoint scrub time (msecs)', 'no_clear,no_scale'), @@ -571,10 +587,9 @@ connection_stats = [ TxnStat('txn_durable_queue_inserts', 'durable timestamp queue inserts total'), TxnStat('txn_durable_queue_len', 'durable timestamp queue length'), TxnStat('txn_durable_queue_walked', 'durable timestamp queue entries walked'), - TxnStat('txn_fail_cache', 'transaction failures due to cache overflow'), + TxnStat('txn_fail_cache', 'transaction failures due to history store'), TxnStat('txn_pinned_checkpoint_range', 'transaction range of IDs currently pinned by a checkpoint', 'no_clear,no_scale'), TxnStat('txn_pinned_range', 'transaction range of IDs currently pinned', 'no_clear,no_scale'), - TxnStat('txn_pinned_snapshot_range', 'transaction range of IDs currently pinned by named snapshots', 'no_clear,no_scale'), TxnStat('txn_pinned_timestamp', 'transaction range of timestamps currently pinned', 'no_clear,no_scale'), TxnStat('txn_pinned_timestamp_checkpoint', 'transaction range of timestamps pinned by a checkpoint', 'no_clear,no_scale'), TxnStat('txn_pinned_timestamp_oldest', 'transaction range of timestamps pinned by the oldest timestamp', 'no_clear,no_scale'), @@ -584,7 +599,6 @@ connection_stats = [ TxnStat('txn_prepare_commit', 'prepared transactions committed'), TxnStat('txn_prepare_rollback', 'prepared transactions rolled back'), TxnStat('txn_prepared_updates_count', 'Number of prepared updates'), - TxnStat('txn_prepared_updates_lookaside_inserts', 'Number of prepared updates added to cache overflow'), TxnStat('txn_query_ts', 'query timestamp calls'), TxnStat('txn_read_queue_empty', 'read timestamp queue insert to empty'), TxnStat('txn_read_queue_head', 'read timestamp queue inserts to head'), @@ -592,9 +606,12 @@ connection_stats = [ TxnStat('txn_read_queue_len', 'read timestamp queue length'), TxnStat('txn_read_queue_walked', 'read timestamp queue entries walked'), TxnStat('txn_rollback', 'transactions rolled back'), - TxnStat('txn_rollback_las_removed', 'rollback to stable updates removed from cache overflow'), - TxnStat('txn_rollback_to_stable', 'rollback to stable calls'), - TxnStat('txn_rollback_upd_aborted', 'rollback to stable updates aborted'), + TxnStat('txn_rts', 'rollback to stable calls'), + TxnStat('txn_rts_hs_removed', 'rollback to stable updates removed from history store'), + TxnStat('txn_rts_keys_removed', 'rollback to stable keys removed'), + TxnStat('txn_rts_keys_restored', 'rollback to stable keys restored'), + TxnStat('txn_rts_pages_visited', 'rollback to stable pages visited'), + TxnStat('txn_rts_upd_aborted', 'rollback to stable updates aborted'), TxnStat('txn_set_ts', 'set timestamp calls'), TxnStat('txn_set_ts_durable', 'set timestamp durable calls'), TxnStat('txn_set_ts_durable_upd', 'set timestamp durable updates'), @@ -602,8 +619,6 @@ connection_stats = [ TxnStat('txn_set_ts_oldest_upd', 'set timestamp oldest updates'), TxnStat('txn_set_ts_stable', 'set timestamp stable calls'), TxnStat('txn_set_ts_stable_upd', 'set timestamp stable updates'), - TxnStat('txn_snapshots_created', 'number of named snapshots created'), - TxnStat('txn_snapshots_dropped', 'number of named snapshots dropped'), TxnStat('txn_sync', 'transaction sync calls'), TxnStat('txn_timestamp_oldest_active_read', 'transaction read timestamp of the oldest active reader', 'no_clear,no_scale'), TxnStat('txn_update_conflict', 'update conflicts'), @@ -702,16 +717,16 @@ dsrc_stats = [ CacheStat('cache_eviction_walks_gave_up_no_targets', 'eviction walks gave up because they saw too many pages and found no candidates'), CacheStat('cache_eviction_walks_gave_up_ratio', 'eviction walks gave up because they saw too many pages and found too few candidates'), CacheStat('cache_eviction_walks_stopped', 'eviction walks gave up because they restarted their walk twice'), + CacheStat('cache_hs_read', 'history store table reads'), CacheStat('cache_inmem_split', 'in-memory page splits'), CacheStat('cache_inmem_splittable', 'in-memory page passed criteria to be split'), CacheStat('cache_pages_requested', 'pages requested from the cache'), CacheStat('cache_read', 'pages read into cache'), CacheStat('cache_read_deleted', 'pages read into cache after truncate'), CacheStat('cache_read_deleted_prepared', 'pages read into cache after truncate in prepare state'), - CacheStat('cache_read_lookaside', 'pages read into cache requiring cache overflow entries'), CacheStat('cache_read_overflow', 'overflow pages read into cache'), CacheStat('cache_write', 'pages written from cache'), - CacheStat('cache_write_lookaside', 'page written requiring cache overflow records'), + CacheStat('cache_write_hs', 'page written requiring history store records'), CacheStat('cache_write_restore', 'pages written requiring in-memory restoration'), ########################################## diff --git a/src/third_party/wiredtiger/examples/c/ex_all.c b/src/third_party/wiredtiger/examples/c/ex_all.c index 5cd014493dd..64d3c7da5a4 100644 --- a/src/third_party/wiredtiger/examples/c/ex_all.c +++ b/src/third_party/wiredtiger/examples/c/ex_all.c @@ -44,7 +44,6 @@ static void connection_ops(WT_CONNECTION *conn); static int cursor_ops(WT_SESSION *session); static void cursor_search_near(WT_CURSOR *cursor); static void cursor_statistics(WT_SESSION *session); -static void named_snapshot_ops(WT_SESSION *session); static void pack_ops(WT_SESSION *session); static void session_ops(WT_SESSION *session); static void transaction_ops(WT_SESSION *session); @@ -554,23 +553,6 @@ cursor_statistics(WT_SESSION *session) } static void -named_snapshot_ops(WT_SESSION *session) -{ - /*! [Snapshot examples] */ - /* Create a named snapshot */ - error_check(session->snapshot(session, "name=June01")); - - /* Open a transaction at a given snapshot */ - error_check(session->begin_transaction(session, "snapshot=June01")); - - /* Drop all named snapshots */ - error_check(session->snapshot(session, "drop=(all)")); - /*! [Snapshot examples] */ - - error_check(session->rollback_transaction(session, NULL)); -} - -static void session_ops_create(WT_SESSION *session) { /*! [Create a table] */ @@ -791,7 +773,6 @@ session_ops(WT_SESSION *session) checkpoint_ops(session); error_check(cursor_ops(session)); cursor_statistics(session); - named_snapshot_ops(session); pack_ops(session); transaction_ops(session); @@ -1102,6 +1083,7 @@ backup(WT_SESSION *session) { char buf[1024]; + WT_CURSOR *dup_cursor; /*! [backup]*/ WT_CURSOR *cursor; const char *filename; @@ -1125,12 +1107,26 @@ backup(WT_SESSION *session) error_check(cursor->close(cursor)); /*! [backup]*/ + /*! [backup log duplicate]*/ + /* Open the backup data source. */ + error_check(session->open_cursor(session, "backup:", NULL, NULL, &cursor)); + /* Open a duplicate cursor for additional log files. */ + error_check(session->open_cursor(session, NULL, cursor, "target=(\"log:\")", &dup_cursor)); + /*! [backup log duplicate]*/ + /*! [incremental backup]*/ - /* Open the backup data source for incremental backup. */ + /* Open the backup data source for log-based incremental backup. */ error_check(session->open_cursor(session, "backup:", NULL, "target=(\"log:\")", &cursor)); /*! [incremental backup]*/ error_check(cursor->close(cursor)); + /*! [incremental block backup]*/ + /* Open the backup data source for block-based incremental backup. */ + error_check(session->open_cursor( + session, "backup:", NULL, "incremental=(enabled,src_id=ID0,this_id=ID1)", &cursor)); + /*! [incremental block backup]*/ + error_check(cursor->close(cursor)); + /*! [backup of a checkpoint]*/ error_check(session->checkpoint(session, "drop=(from=June01),name=June01")); /*! [backup of a checkpoint]*/ diff --git a/src/third_party/wiredtiger/examples/c/ex_backup_block.c b/src/third_party/wiredtiger/examples/c/ex_backup_block.c index f374424d442..24ec718af53 100644 --- a/src/third_party/wiredtiger/examples/c/ex_backup_block.c +++ b/src/third_party/wiredtiger/examples/c/ex_backup_block.c @@ -269,7 +269,7 @@ take_full_backup(WT_SESSION *session, int i) hdir = home_incr; if (i == 0) { (void)snprintf( - buf, sizeof(buf), "incremental=(granularity=1M,enabled=true,this_id=ID%d)", i); + buf, sizeof(buf), "incremental=(granularity=1M,enabled=true,this_id=\"ID%d\")", i); error_check(session->open_cursor(session, "backup:", NULL, buf, &cursor)); } else error_check(session->open_cursor(session, "backup:", NULL, NULL, &cursor)); @@ -330,12 +330,10 @@ take_incr_backup(WT_SESSION *session, int i) const char *filename; bool first; - /*! [incremental backup using block transfer]*/ - tmp = NULL; tmp_sz = 0; /* Open the backup data source for incremental backup. */ - (void)snprintf(buf, sizeof(buf), "incremental=(src_id=ID%d,this_id=ID%d)", i - 1, i); + (void)snprintf(buf, sizeof(buf), "incremental=(src_id=\"ID%d\",this_id=\"ID%d\")", i - 1, i); error_check(session->open_cursor(session, "backup:", NULL, buf, &backup_cur)); rfd = wfd = -1; count = 0; @@ -385,7 +383,7 @@ take_incr_backup(WT_SESSION *session, int i) error_sys_check(rfd = open(buf, O_RDONLY, 0)); (void)snprintf(h, sizeof(h), "%s.%d", home_incr, i); (void)snprintf(buf, sizeof(buf), "%s/%s", h, filename); - error_sys_check(wfd = open(buf, O_WRONLY, 0)); + error_sys_check(wfd = open(buf, O_WRONLY | O_CREAT, 0)); first = false; } @@ -439,7 +437,6 @@ take_incr_backup(WT_SESSION *session, int i) error_check(backup_cur->close(backup_cur)); error_check(finalize_files(flist, count)); free(tmp); - /*! [incremental backup using block transfer]*/ } int @@ -506,7 +503,8 @@ main(int argc, char *argv[]) /* * We should have an entry for i-1 and i-2. Use the older one. */ - (void)snprintf(cmd_buf, sizeof(cmd_buf), "incremental=(src_id=ID%d,this_id=ID%d)", i - 2, i); + (void)snprintf( + cmd_buf, sizeof(cmd_buf), "incremental=(src_id=\"ID%d\",this_id=\"ID%d\")", i - 2, i); error_check(session->open_cursor(session, "backup:", NULL, cmd_buf, &backup_cur)); error_check(backup_cur->close(backup_cur)); @@ -540,7 +538,8 @@ main(int argc, char *argv[]) /* * We should not have any information. */ - (void)snprintf(cmd_buf, sizeof(cmd_buf), "incremental=(src_id=ID%d,this_id=ID%d)", i - 2, i); + (void)snprintf( + cmd_buf, sizeof(cmd_buf), "incremental=(src_id=\"ID%d\",this_id=\"ID%d\")", i - 2, i); testutil_assert(session->open_cursor(session, "backup:", NULL, cmd_buf, &backup_cur) == ENOENT); error_check(wt_conn->close(wt_conn, NULL)); diff --git a/src/third_party/wiredtiger/examples/java/com/wiredtiger/examples/ex_all.java b/src/third_party/wiredtiger/examples/java/com/wiredtiger/examples/ex_all.java index 1258e195929..50130663462 100644 --- a/src/third_party/wiredtiger/examples/java/com/wiredtiger/examples/ex_all.java +++ b/src/third_party/wiredtiger/examples/java/com/wiredtiger/examples/ex_all.java @@ -845,6 +845,7 @@ backup(Session session) { char buf[] = new char[1024]; + Cursor dup_cursor; /*! [backup]*/ Cursor cursor; String filename; @@ -890,6 +891,21 @@ backup(Session session) } /*! [backup]*/ try { + /*! [backup log duplicate]*/ + /* Open the backup data source. */ + cursor = session.open_cursor("backup:", null, null); + /* Open a duplicate cursor for additional log files. */ + dup_cursor = session.open_cursor(null, cursor, "target=(\"log:\")"); + /*! [backup log duplicate]*/ + + ret = dup_cursor.close(); + ret = cursor.close(); + } + catch (Exception ex) { + System.err.println(progname + + ": duplicate log backup failed: " + ex.toString()); + } + try { /*! [incremental backup]*/ /* Open the backup data source for incremental backup. */ cursor = session.open_cursor("backup:", null, "target=(\"log:\")"); @@ -902,6 +918,19 @@ backup(Session session) ": incremental backup failed: " + ex.toString()); } + try { + /*! [incremental block backup]*/ + /* Open the backup data source for incremental backup. */ + cursor = session.open_cursor("backup:", null, "incremental=(enabled,src_id=ID0,this_id=ID1)"); + /*! [incremental block backup]*/ + + ret = cursor.close(); + } + catch (Exception ex) { + System.err.println(progname + + ": incremental backup failed: " + ex.toString()); + } + /*! [backup of a checkpoint]*/ ret = session.checkpoint("drop=(from=June01),name=June01"); /*! [backup of a checkpoint]*/ diff --git a/src/third_party/wiredtiger/import.data b/src/third_party/wiredtiger/import.data index 283621c29c6..5883a331087 100644 --- a/src/third_party/wiredtiger/import.data +++ b/src/third_party/wiredtiger/import.data @@ -2,5 +2,5 @@ "vendor": "wiredtiger", "github": "wiredtiger/wiredtiger.git", "branch": "mongodb-4.4", - "commit": "7e595e4a3a9c30c9db0eb33f7da72c97526a2b99" + "commit": "187983a50c696eb217a780bb6b29e4bd3433c13b" } diff --git a/src/third_party/wiredtiger/lang/java/java_doc.i b/src/third_party/wiredtiger/lang/java/java_doc.i index fa6a4c883c4..f5fb035c863 100644 --- a/src/third_party/wiredtiger/lang/java/java_doc.i +++ b/src/third_party/wiredtiger/lang/java/java_doc.i @@ -56,7 +56,6 @@ COPYDOC(__wt_session, WT_SESSION, rollback_transaction) COPYDOC(__wt_session, WT_SESSION, timestamp_transaction) COPYDOC(__wt_session, WT_SESSION, query_timestamp) COPYDOC(__wt_session, WT_SESSION, checkpoint) -COPYDOC(__wt_session, WT_SESSION, snapshot) COPYDOC(__wt_session, WT_SESSION, transaction_pinned_range) COPYDOC(__wt_session, WT_SESSION, transaction_sync) COPYDOC(__wt_session, WT_SESSION, breakpoint) diff --git a/src/third_party/wiredtiger/src/block/block_ckpt_scan.c b/src/third_party/wiredtiger/src/block/block_ckpt_scan.c index f781015f09c..4d47c4301b2 100644 --- a/src/third_party/wiredtiger/src/block/block_ckpt_scan.c +++ b/src/third_party/wiredtiger/src/block/block_ckpt_scan.c @@ -9,40 +9,29 @@ #include "wt_internal.h" /* - * It wasn't possible to open standalone files in historic WiredTiger databases, - * you're done if you lose the file's associated metadata. That was a mistake - * and this code is the workaround. What we need to crack a file is database - * metadata plus a list of active checkpoints as of the file's clean shutdown - * (normally stored in the database metadata). The last write done in a block - * manager's checkpoint is the avail list. If current metadata and checkpoint - * information is included in that write, we're close. We can open the file, - * read the blocks, scan until we find the avail list, and read the metadata - * and checkpoint information from there. - * Two problems remain: first, the checkpoint information isn't correct - * until we write the avail list and the checkpoint information has to include - * the avail list address plus the final file size after the write. Fortunately, - * when scanning the file for the avail lists, we're figuring out exactly the - * information needed to fix up the checkpoint information we wrote, that is, - * the avail list's offset, size and checksum triplet. As for the final file - * size, we allocate all space in the file before we calculate block checksums, - * so we can do that space allocation, then fill in the final file size before - * calculating the checksum and writing the actual block. - * The second problem is we have to be able to find the avail lists that - * include checkpoint information (ignoring previous files created by previous - * releases, and, of course, making upgrade/downgrade work seamlessly). Extent - * lists are written to their own pages, and we could version this change using - * the page header version. Extent lists have WT_PAGE_BLOCK_MANAGER page types, - * we could version this change using the upcoming WT_PAGE_VERSION_TS upgrade. - * However, that requires waiting a release (we would have to first release a - * version that ignores those new page header versions so downgrade works), and - * we're not planning a release that writes WT_PAGE_VERSION_TS page headers for - * awhile. Happily, historic WiredTiger releases have a bug. Extent lists - * consist of a set of offset/size pairs, with magic offset/size pairs at the - * beginning and end of the list. Historic releases only verified the offset of - * the special pair at the end of the list, ignoring the size. To detect avail - * lists that include appended metadata and checkpoint information, this change - * adds a version to the extent list: if size is WT_BLOCK_EXTLIST_VERSION_CKPT, - * then metadata/checkpoint information follows. + * It wasn't possible to open standalone files in historic WiredTiger databases, you're done if you + * lose the file's associated metadata. That was a mistake and this code is the workaround. What we + * need to crack a file is database metadata plus a list of active checkpoints as of the file's + * clean shutdown (normally stored in the database metadata). The last write done in a block + * manager's checkpoint is the avail list. If current metadata and checkpoint information is + * included in that write, we're close. We can open the file, read the blocks, scan until we find + * the avail list, and read the metadata and checkpoint information from there. + * Two problems remain: first, the checkpoint information isn't correct until we write the + * avail list and the checkpoint information has to include the avail list address plus the final + * file size after the write. Fortunately, when scanning the file for the avail lists, we're + * figuring out exactly the information needed to fix up the checkpoint information we wrote, that + * is, the avail list's offset, size and checksum triplet. As for the final file size, we allocate + * all space in the file before we calculate block checksums, so we can do that space allocation, + * then fill in the final file size before calculating the checksum and writing the actual block. + * The second problem is we have to be able to find the avail lists that include checkpoint + * information (ignoring previous files created by previous releases, and, of course, making + * upgrade/downgrade work seamlessly). Extent lists are written to their own pages, and we could + * version this change using the page header version. Happily, historic WiredTiger releases have a + * bug. Extent lists consist of a set of offset/size pairs, with magic offset/size pairs at the + * beginning and end of the list. Historic releases only verified the offset of the special pair at + * the end of the list, ignoring the size. To detect avail lists that include appended metadata and + * checkpoint information, this change adds a version to the extent list: if size is + * WT_BLOCK_EXTLIST_VERSION_CKPT, then metadata/checkpoint information follows. */ /* diff --git a/src/third_party/wiredtiger/src/block/block_ext.c b/src/third_party/wiredtiger/src/block/block_ext.c index e8d100d6df7..3f11cbe5496 100644 --- a/src/third_party/wiredtiger/src/block/block_ext.c +++ b/src/third_party/wiredtiger/src/block/block_ext.c @@ -1185,7 +1185,6 @@ __wt_block_extlist_write( dsk = tmp->mem; memset(dsk, 0, WT_BLOCK_HEADER_BYTE_SIZE); dsk->type = WT_PAGE_BLOCK_MANAGER; - dsk->version = WT_PAGE_VERSION_TS; /* Fill the page's data. */ p = WT_BLOCK_HEADER_BYTE(dsk); diff --git a/src/third_party/wiredtiger/src/block/block_open.c b/src/third_party/wiredtiger/src/block/block_open.c index 32b40d56128..45229528905 100644 --- a/src/third_party/wiredtiger/src/block/block_open.c +++ b/src/third_party/wiredtiger/src/block/block_open.c @@ -316,6 +316,16 @@ __desc_read(WT_SESSION_IMPL *session, uint32_t allocsize, WT_BLOCK *block) if (F_ISSET(S2C(session), WT_CONN_IN_MEMORY)) return (0); + /* + * If a data file is smaller than the allocation size, we're not going to be able to read the + * descriptor block. We should treat this as if the file has been deleted; that is, to log an + * error but continue on. + */ + if (block->size < allocsize) + WT_RET_MSG(session, ENOENT, + "File %s is smaller than allocation size; file size=%" PRId64 ", alloc size=%" PRIu32, + block->name, block->size, allocsize); + /* Use a scratch buffer to get correct alignment for direct I/O. */ WT_RET(__wt_scr_alloc(session, allocsize, &buf)); diff --git a/src/third_party/wiredtiger/src/btree/bt_compact.c b/src/third_party/wiredtiger/src/btree/bt_compact.c index c68ff7cbbd7..1f657fdd931 100644 --- a/src/third_party/wiredtiger/src/btree/bt_compact.c +++ b/src/third_party/wiredtiger/src/btree/bt_compact.c @@ -15,6 +15,7 @@ static int __compact_rewrite(WT_SESSION_IMPL *session, WT_REF *ref, bool *skipp) { + WT_ADDR_COPY addr; WT_BM *bm; WT_MULTI *multi; WT_PAGE_MODIFY *mod; @@ -24,13 +25,15 @@ __compact_rewrite(WT_SESSION_IMPL *session, WT_REF *ref, bool *skipp) bm = S2BT(session)->bm; + /* If the page is clean, test the original addresses. */ + if (__wt_page_evict_clean(ref->page)) + return (__wt_ref_addr_copy(session, ref, &addr) ? + bm->compact_page_skip(bm, session, addr.addr, addr.size, skipp) : + 0); + /* * If the page is a replacement, test the replacement addresses. Ignore empty pages, they get * merged into the parent. - * - * Page-modify variable initialization done here because the page could be modified while we're - * looking at it, so the page modified structure may appear at any time (but cannot disappear). - * We've confirmed there is a page modify structure, it's OK to look at it. */ mod = ref->page->modify; if (mod->rec_result == WT_PM_REC_REPLACE) @@ -56,38 +59,19 @@ __compact_rewrite(WT_SESSION_IMPL *session, WT_REF *ref, bool *skipp) static int __compact_rewrite_lock(WT_SESSION_IMPL *session, WT_REF *ref, bool *skipp) { - WT_BM *bm; WT_BTREE *btree; WT_DECL_RET; - size_t addr_size; - const uint8_t *addr; - - *skipp = true; /* Default skip. */ btree = S2BT(session); - bm = btree->bm; - - /* - * If the page is clean, test the original addresses. We're holding a hazard pointer on the - * page, so we're safe from eviction, no additional locking is required. - */ - if (__wt_page_evict_clean(ref->page)) { - __wt_ref_info(session, ref, &addr, &addr_size, NULL); - if (addr == NULL) - return (0); - return (bm->compact_page_skip(bm, session, addr, addr_size, skipp)); - } /* * Reviewing in-memory pages requires looking at page reconciliation results, because we care * about where the page is stored now, not where the page was stored when we first read it into * the cache. We need to ensure we don't race with page reconciliation as it's writing the page - * modify information. - * - * There are two ways we call reconciliation: checkpoints and eviction. Get the tree's flush - * lock which blocks threads writing pages for checkpoints. If checkpoint is holding the lock, - * quit working this file, we'll visit it again in our next pass. As noted above, we're holding - * a hazard pointer on the page, we're safe from eviction. + * modify information. There are two ways we call reconciliation: checkpoints and eviction. We + * are holding a hazard pointer that blocks eviction, but there's nothing blocking a checkpoint. + * Get the tree's flush lock which blocks threads writing pages for checkpoints. If checkpoint + * is holding the lock, quit working this file, we'll visit it again in our next pass. */ WT_RET(__wt_spin_trylock(session, &btree->flush_lock)); @@ -194,7 +178,11 @@ __wt_compact(WT_SESSION_IMPL *session) * Ignore the root: it may not have a replacement address, and besides, if anything else * gets written, so will it. * - * Ignore dirty pages, checkpoint writes them regardless. + * Ignore dirty pages, checkpoint will likely write them. There are cases where checkpoint + * can skip dirty pages: to avoid that, we could alter the transactional information of the + * page, which is what checkpoint reviews to decide if a page can be skipped. Not doing that + * for now, the repeated checkpoints that compaction requires are more than likely to pick + * up all dirty pages at some point. */ if (__wt_ref_is_root(ref)) continue; @@ -227,15 +215,19 @@ err: int __wt_compact_page_skip(WT_SESSION_IMPL *session, WT_REF *ref, void *context, bool *skipp) { + WT_ADDR_COPY addr; WT_BM *bm; - size_t addr_size; - uint8_t addr[WT_BTREE_MAX_ADDR_COOKIE]; - bool is_leaf; + uint8_t previous_state; + bool diskaddr; WT_UNUSED(context); *skipp = false; /* Default to reading */ + /* Internal pages must be read to walk the tree. */ + if (F_ISSET(ref, WT_REF_FLAG_INTERNAL)) + return (0); + /* * Skip deleted pages, rewriting them doesn't seem useful; in a better world we'd write the * parent to delete the page. @@ -247,27 +239,23 @@ __wt_compact_page_skip(WT_SESSION_IMPL *session, WT_REF *ref, void *context, boo /* * If the page is in-memory, we want to look at it (it may have been modified and written, and - * the current location is the interesting one in terms of compaction, not the original - * location). - * - * This test could be combined with the next one, but this is a cheap test and the next one is - * expensive. + * the current location is the interesting one in terms of compaction, not the original). */ if (ref->state != WT_REF_DISK) return (0); /* - * Internal pages must be read to walk the tree; ask the block-manager if it's useful to rewrite - * leaf pages, don't do the I/O if a rewrite won't help. - * - * There can be NULL WT_REF.addr values, where the underlying call won't return a valid address. - * The "it's a leaf page" return is enough to confirm we have a valid address for a leaf page. + * Lock the WT_REF and if it's still on-disk, get a copy of the address. This is safe because + * it's an on-disk page and we're holding the WT_REF locked, so nobody can read the page giving + * either checkpoint or eviction a chance to modify the address. */ - __wt_ref_info_lock(session, ref, addr, &addr_size, &is_leaf); - if (is_leaf) { - bm = S2BT(session)->bm; - return (bm->compact_page_skip(bm, session, addr, addr_size, skipp)); - } + WT_REF_LOCK(session, ref, &previous_state); + diskaddr = previous_state == WT_REF_DISK && __wt_ref_addr_copy(session, ref, &addr); + WT_REF_UNLOCK(ref, previous_state); + if (!diskaddr) + return (0); - return (0); + /* Ask the block-manager if it's useful to rewrite the page. */ + bm = S2BT(session)->bm; + return (bm->compact_page_skip(bm, session, addr.addr, addr.size, skipp)); } diff --git a/src/third_party/wiredtiger/src/btree/bt_curnext.c b/src/third_party/wiredtiger/src/btree/bt_curnext.c index 23967190d7d..6fd6cb2e89c 100644 --- a/src/third_party/wiredtiger/src/btree/bt_curnext.c +++ b/src/third_party/wiredtiger/src/btree/bt_curnext.c @@ -58,12 +58,15 @@ __cursor_fix_append_next(WT_CURSOR_BTREE *cbt, bool newpage, bool restart) cbt->iface.value.data = &cbt->v; } else { restart_read: - WT_RET(__wt_txn_read(session, cbt->ins->upd, &upd)); + WT_RET(__wt_txn_read_upd_list(session, cbt->ins->upd, &upd)); if (upd == NULL) { cbt->v = 0; cbt->iface.value.data = &cbt->v; - } else + } else { + if (F_ISSET(upd, WT_UPDATE_RESTORED_FROM_DISK) && upd->type != WT_UPDATE_TOMBSTONE) + return (__wt_value_return(cbt, upd)); cbt->iface.value.data = upd->data; + } } cbt->iface.value.size = 1; return (0); @@ -110,14 +113,21 @@ new_page: cbt->ins = __col_insert_search(cbt->ins_head, cbt->ins_stack, cbt->next_stack, cbt->recno); if (cbt->ins != NULL && cbt->recno != WT_INSERT_RECNO(cbt->ins)) cbt->ins = NULL; + /* + * FIXME-PM-1523: Now we only do transaction read if we have an update chain and it doesn't work + * in durable history. Review this when we have a plan for fixed-length column store. + */ if (cbt->ins != NULL) restart_read: - WT_RET(__wt_txn_read(session, cbt->ins->upd, &upd)); + WT_RET(__wt_txn_read(session, cbt, cbt->ins->upd, NULL, &upd)); if (upd == NULL) { cbt->v = __bit_getv_recno(cbt->ref, cbt->recno, btree->bitcnt); cbt->iface.value.data = &cbt->v; - } else + } else { + if (F_ISSET(upd, WT_UPDATE_RESTORED_FROM_DISK) && upd->type != WT_UPDATE_TOMBSTONE) + return (__wt_value_return(cbt, upd)); cbt->iface.value.data = upd->data; + } cbt->iface.value.size = 1; return (0); } @@ -151,12 +161,15 @@ new_page: __cursor_set_recno(cbt, WT_INSERT_RECNO(cbt->ins)); restart_read: - WT_RET(__wt_txn_read(session, cbt->ins->upd, &upd)); + WT_RET(__wt_txn_read_upd_list(session, cbt->ins->upd, &upd)); + if (upd == NULL) continue; if (upd->type == WT_UPDATE_TOMBSTONE) { if (upd->txnid != WT_TXN_NONE && __wt_txn_upd_visible_all(session, upd)) ++cbt->page_deleted_count; + if (F_ISSET(upd, WT_UPDATE_RESTORED_FROM_DISK)) + __wt_free_update_list(session, &upd); continue; } return (__wt_value_return(cbt, upd)); @@ -221,11 +234,13 @@ restart_read: cbt->ins = __col_insert_search_match(cbt->ins_head, cbt->recno); upd = NULL; if (cbt->ins != NULL) - WT_RET(__wt_txn_read(session, cbt->ins->upd, &upd)); + WT_RET(__wt_txn_read_upd_list(session, cbt->ins->upd, &upd)); if (upd != NULL) { if (upd->type == WT_UPDATE_TOMBSTONE) { if (upd->txnid != WT_TXN_NONE && __wt_txn_upd_visible_all(session, upd)) ++cbt->page_deleted_count; + if (F_ISSET(upd, WT_UPDATE_RESTORED_FROM_DISK)) + __wt_free_update_list(session, &upd); continue; } return (__wt_value_return(cbt, upd)); @@ -269,9 +284,11 @@ restart_read: --cbt->recno; continue; } - WT_RET(__wt_page_cell_data_ref(session, page, &unpack, cbt->tmp)); - cbt->cip_saved = cip; + WT_RET(__wt_bt_col_var_cursor_walk_txn_read(session, cbt, page, &unpack, cip, &upd)); + if (upd == NULL) + continue; + return (0); } cbt->iface.value.data = cbt->tmp->data; cbt->iface.value.size = cbt->tmp->size; @@ -287,12 +304,14 @@ restart_read: static inline int __cursor_row_next(WT_CURSOR_BTREE *cbt, bool newpage, bool restart) { + WT_CELL_UNPACK kpack; WT_INSERT *ins; WT_ITEM *key; WT_PAGE *page; WT_ROW *rip; WT_SESSION_IMPL *session; WT_UPDATE *upd; + bool kpack_used; session = (WT_SESSION_IMPL *)cbt->iface.session; page = cbt->ref->page; @@ -341,16 +360,18 @@ new_insert: cbt->iter_retry = WT_CBT_RETRY_INSERT; restart_read_insert: if ((ins = cbt->ins) != NULL) { - WT_RET(__wt_txn_read(session, ins->upd, &upd)); + key->data = WT_INSERT_KEY(ins); + key->size = WT_INSERT_KEY_SIZE(ins); + WT_RET(__wt_txn_read_upd_list(session, ins->upd, &upd)); if (upd == NULL) continue; if (upd->type == WT_UPDATE_TOMBSTONE) { if (upd->txnid != WT_TXN_NONE && __wt_txn_upd_visible_all(session, upd)) ++cbt->page_deleted_count; + if (F_ISSET(upd, WT_UPDATE_RESTORED_FROM_DISK)) + __wt_free_update_list(session, &upd); continue; } - key->data = WT_INSERT_KEY(ins); - key->size = WT_INSERT_KEY_SIZE(ins); return (__wt_value_return(cbt, upd)); } @@ -375,13 +396,18 @@ restart_read_insert: cbt->slot = cbt->row_iteration_slot / 2 - 1; restart_read_page: rip = &page->pg_row[cbt->slot]; - WT_RET(__wt_txn_read(session, WT_ROW_UPDATE(page, rip), &upd)); + WT_RET(__cursor_row_slot_key_return(cbt, rip, &kpack, &kpack_used)); + WT_RET(__wt_txn_read(session, cbt, WT_ROW_UPDATE(page, rip), NULL, &upd)); + if (upd == NULL) + continue; if (upd != NULL && upd->type == WT_UPDATE_TOMBSTONE) { if (upd->txnid != WT_TXN_NONE && __wt_txn_upd_visible_all(session, upd)) ++cbt->page_deleted_count; + if (F_ISSET(upd, WT_UPDATE_RESTORED_FROM_DISK)) + __wt_free_update_list(session, &upd); continue; } - return (__cursor_row_slot_return(cbt, rip, upd)); + return (__wt_value_return(cbt, upd)); } /* NOTREACHED */ } diff --git a/src/third_party/wiredtiger/src/btree/bt_curprev.c b/src/third_party/wiredtiger/src/btree/bt_curprev.c index 087df4b1a8a..c84fc94e686 100644 --- a/src/third_party/wiredtiger/src/btree/bt_curprev.c +++ b/src/third_party/wiredtiger/src/btree/bt_curprev.c @@ -197,14 +197,16 @@ __cursor_fix_append_prev(WT_CURSOR_BTREE *cbt, bool newpage, bool restart) cbt->v = 0; cbt->iface.value.data = &cbt->v; } else { - upd = NULL; restart_read: - WT_RET(__wt_txn_read(session, cbt->ins->upd, &upd)); + WT_RET(__wt_txn_read_upd_list(session, cbt->ins->upd, &upd)); if (upd == NULL) { cbt->v = 0; cbt->iface.value.data = &cbt->v; - } else + } else { + if (F_ISSET(upd, WT_UPDATE_RESTORED_FROM_DISK) && upd->type != WT_UPDATE_TOMBSTONE) + return (__wt_value_return(cbt, upd)); cbt->iface.value.data = upd->data; + } } cbt->iface.value.size = 1; return (0); @@ -251,14 +253,21 @@ new_page: if (cbt->ins != NULL && cbt->recno != WT_INSERT_RECNO(cbt->ins)) cbt->ins = NULL; upd = NULL; + /* + * FIXME-PM-1523: Now we only do transaction read if we have an update chain and it doesn't work + * in durable history. Review this when we have a plan for fixed-length column store. + */ if (cbt->ins != NULL) restart_read: - WT_RET(__wt_txn_read(session, cbt->ins->upd, &upd)); + WT_RET(__wt_txn_read(session, cbt, cbt->ins->upd, NULL, &upd)); if (upd == NULL) { cbt->v = __bit_getv_recno(cbt->ref, cbt->recno, btree->bitcnt); cbt->iface.value.data = &cbt->v; - } else + } else { + if (F_ISSET(upd, WT_UPDATE_RESTORED_FROM_DISK) && upd->type != WT_UPDATE_TOMBSTONE) + return (__wt_value_return(cbt, upd)); cbt->iface.value.data = upd->data; + } cbt->iface.value.size = 1; return (0); } @@ -292,12 +301,14 @@ new_page: __cursor_set_recno(cbt, WT_INSERT_RECNO(cbt->ins)); restart_read: - WT_RET(__wt_txn_read(session, cbt->ins->upd, &upd)); + WT_RET(__wt_txn_read_upd_list(session, cbt->ins->upd, &upd)); if (upd == NULL) continue; if (upd->type == WT_UPDATE_TOMBSTONE) { if (upd->txnid != WT_TXN_NONE && __wt_txn_upd_visible_all(session, upd)) ++cbt->page_deleted_count; + if (F_ISSET(upd, WT_UPDATE_RESTORED_FROM_DISK) && upd->type != WT_UPDATE_TOMBSTONE) + __wt_free_update_list(session, &upd); continue; } return (__wt_value_return(cbt, upd)); @@ -363,11 +374,13 @@ restart_read: cbt->ins = __col_insert_search_match(cbt->ins_head, cbt->recno); upd = NULL; if (cbt->ins != NULL) - WT_RET(__wt_txn_read(session, cbt->ins->upd, &upd)); + WT_RET(__wt_txn_read_upd_list(session, cbt->ins->upd, &upd)); if (upd != NULL) { if (upd->type == WT_UPDATE_TOMBSTONE) { if (upd->txnid != WT_TXN_NONE && __wt_txn_upd_visible_all(session, upd)) ++cbt->page_deleted_count; + if (F_ISSET(upd, WT_UPDATE_RESTORED_FROM_DISK)) + __wt_free_update_list(session, &upd); continue; } return (__wt_value_return(cbt, upd)); @@ -386,6 +399,7 @@ restart_read: if (unpack.type == WT_CELL_DEL) { if (__wt_cell_rle(&unpack) == 1) continue; + /* * There can be huge gaps in the variable-length column-store name space appearing * as deleted records. If more than one deleted record, do the work of finding the @@ -410,9 +424,11 @@ restart_read: ++cbt->recno; continue; } - WT_RET(__wt_page_cell_data_ref(session, page, &unpack, cbt->tmp)); - cbt->cip_saved = cip; + WT_RET(__wt_bt_col_var_cursor_walk_txn_read(session, cbt, page, &unpack, cip, &upd)); + if (upd == NULL) + continue; + return (0); } cbt->iface.value.data = cbt->tmp->data; cbt->iface.value.size = cbt->tmp->size; @@ -428,12 +444,14 @@ restart_read: static inline int __cursor_row_prev(WT_CURSOR_BTREE *cbt, bool newpage, bool restart) { + WT_CELL_UNPACK kpack; WT_INSERT *ins; WT_ITEM *key; WT_PAGE *page; WT_ROW *rip; WT_SESSION_IMPL *session; WT_UPDATE *upd; + bool kpack_used; session = (WT_SESSION_IMPL *)cbt->iface.session; page = cbt->ref->page; @@ -492,16 +510,18 @@ new_insert: cbt->iter_retry = WT_CBT_RETRY_INSERT; restart_read_insert: if ((ins = cbt->ins) != NULL) { - WT_RET(__wt_txn_read(session, ins->upd, &upd)); + key->data = WT_INSERT_KEY(ins); + key->size = WT_INSERT_KEY_SIZE(ins); + WT_RET(__wt_txn_read_upd_list(session, ins->upd, &upd)); if (upd == NULL) continue; if (upd->type == WT_UPDATE_TOMBSTONE) { if (upd->txnid != WT_TXN_NONE && __wt_txn_upd_visible_all(session, upd)) ++cbt->page_deleted_count; + if (F_ISSET(upd, WT_UPDATE_RESTORED_FROM_DISK)) + __wt_free_update_list(session, &upd); continue; } - key->data = WT_INSERT_KEY(ins); - key->size = WT_INSERT_KEY_SIZE(ins); return (__wt_value_return(cbt, upd)); } @@ -528,13 +548,18 @@ restart_read_insert: cbt->slot = cbt->row_iteration_slot / 2 - 1; restart_read_page: rip = &page->pg_row[cbt->slot]; - WT_RET(__wt_txn_read(session, WT_ROW_UPDATE(page, rip), &upd)); + WT_RET(__cursor_row_slot_key_return(cbt, rip, &kpack, &kpack_used)); + WT_RET(__wt_txn_read(session, cbt, WT_ROW_UPDATE(page, rip), NULL, &upd)); + if (upd == NULL) + continue; if (upd != NULL && upd->type == WT_UPDATE_TOMBSTONE) { if (upd->txnid != WT_TXN_NONE && __wt_txn_upd_visible_all(session, upd)) ++cbt->page_deleted_count; + if (F_ISSET(upd, WT_UPDATE_RESTORED_FROM_DISK)) + __wt_free_update_list(session, &upd); continue; } - return (__cursor_row_slot_return(cbt, rip, upd)); + return (__wt_value_return(cbt, upd)); } /* NOTREACHED */ } diff --git a/src/third_party/wiredtiger/src/btree/bt_cursor.c b/src/third_party/wiredtiger/src/btree/bt_cursor.c index 9996f5fcd7a..1877b2100de 100644 --- a/src/third_party/wiredtiger/src/btree/bt_cursor.c +++ b/src/third_party/wiredtiger/src/btree/bt_cursor.c @@ -58,7 +58,6 @@ __cursor_page_pinned(WT_CURSOR_BTREE *cbt, bool search_operation) { WT_CURSOR *cursor; WT_SESSION_IMPL *session; - uint32_t current_state; cursor = &cbt->iface; session = (WT_SESSION_IMPL *)cursor->session; @@ -100,17 +99,7 @@ __cursor_page_pinned(WT_CURSOR_BTREE *cbt, bool search_operation) if (cbt->ref->page->read_gen == WT_READGEN_OLDEST) return (false); - /* - * We need a page with history: updates need complete update lists and a read might be based on - * a different timestamp than the one that brought the page into memory. Release the page and - * read it again with history if required. Eviction may be locking the page, wait until we see a - * "normal" state and then test against that state (eviction may have already locked the page - * again). - */ - while ((current_state = cbt->ref->state) == WT_REF_LOCKED) - __wt_yield(); - WT_ASSERT(session, current_state == WT_REF_LIMBO || current_state == WT_REF_MEM); - return (current_state == WT_REF_MEM); + return (true); } /* @@ -159,34 +148,6 @@ __cursor_size_chk(WT_SESSION_IMPL *session, WT_ITEM *kv) } /* - * __cursor_disable_bulk -- - * Disable bulk loads into a tree. - */ -static inline void -__cursor_disable_bulk(WT_SESSION_IMPL *session, WT_BTREE *btree) -{ - /* - * Once a tree (other than the LSM primary) is no longer empty, eviction should pay attention to - * it, and it's no longer possible to bulk-load into it. - */ - if (!btree->original) - return; - if (btree->lsm_primary) { - btree->original = 0; /* Make the next test faster. */ - return; - } - - /* - * We use a compare-and-swap here to avoid races among the first inserts into a tree. Eviction - * is disabled when an empty tree is opened, and it must only be enabled once. - */ - if (__wt_atomic_cas8(&btree->original, 1, 0)) { - btree->evict_disabled_open = false; - __wt_evict_file_exclusive_off(session); - } -} - -/* * __cursor_fix_implicit -- * Return if search went past the end of the tree. */ @@ -271,12 +232,16 @@ __wt_cursor_valid(WT_CURSOR_BTREE *cbt, WT_UPDATE **updp, bool *valid) * update that's been deleted is not a valid key/value pair). */ if (cbt->ins != NULL) { - WT_RET(__wt_txn_read(session, cbt->ins->upd, &upd)); + WT_RET(__wt_txn_read_upd_list(session, cbt->ins->upd, &upd)); if (upd != NULL) { - if (upd->type == WT_UPDATE_TOMBSTONE) + if (upd->type == WT_UPDATE_TOMBSTONE) { + WT_ASSERT(session, !F_ISSET(upd, WT_UPDATE_RESTORED_FROM_DISK)); return (0); + } if (updp != NULL) *updp = upd; + else if (F_ISSET(upd, WT_UPDATE_RESTORED_FROM_DISK)) + __wt_free_update_list(session, &upd); *valid = true; return (0); } @@ -298,6 +263,7 @@ __wt_cursor_valid(WT_CURSOR_BTREE *cbt, WT_UPDATE **updp, bool *valid) if (cbt->recno >= cbt->ref->ref_recno + page->entries) return (0); + *valid = true; /* * An update would have appeared as an "insert" object; no further checks to do. */ @@ -328,6 +294,24 @@ __wt_cursor_valid(WT_CURSOR_BTREE *cbt, WT_UPDATE **updp, bool *valid) cell = WT_COL_PTR(page, cip); if (__wt_cell_type(cell) == WT_CELL_DEL) return (0); + + /* + * Check for an update ondisk or in the history store. For column store, an insert object + * can have the same key as an on-page or history store object. + */ + WT_RET(__wt_txn_read(session, cbt, NULL, NULL, &upd)); + if (upd != NULL) { + if (upd->type == WT_UPDATE_TOMBSTONE) { + if (F_ISSET(upd, WT_UPDATE_RESTORED_FROM_DISK)) + __wt_free_update_list(session, &upd); + return (0); + } + if (updp != NULL) + *updp = upd; + else if (F_ISSET(upd, WT_UPDATE_RESTORED_FROM_DISK)) + __wt_free_update_list(session, &upd); + *valid = true; + } break; case BTREE_ROW: /* The search function doesn't check for empty pages. */ @@ -347,18 +331,25 @@ __wt_cursor_valid(WT_CURSOR_BTREE *cbt, WT_UPDATE **updp, bool *valid) return (0); /* Check for an update. */ - if (page->modify != NULL && page->modify->mod_row_update != NULL) { - WT_RET(__wt_txn_read(session, page->modify->mod_row_update[cbt->slot], &upd)); - if (upd != NULL) { - if (upd->type == WT_UPDATE_TOMBSTONE) - return (0); - if (updp != NULL) - *updp = upd; + WT_RET(__wt_txn_read(session, cbt, + (page->modify != NULL && page->modify->mod_row_update != NULL) ? + page->modify->mod_row_update[cbt->slot] : + NULL, + NULL, &upd)); + if (upd != NULL) { + if (upd->type == WT_UPDATE_TOMBSTONE) { + if (F_ISSET(upd, WT_UPDATE_RESTORED_FROM_DISK)) + __wt_free_update_list(session, &upd); + return (0); } + if (updp != NULL) + *updp = upd; + else if (F_ISSET(upd, WT_UPDATE_RESTORED_FROM_DISK)) + __wt_free_update_list(session, &upd); + *valid = true; } break; } - *valid = true; return (0); } @@ -477,14 +468,12 @@ __wt_btcur_search_uncommitted(WT_CURSOR *cursor, WT_UPDATE **updp) { WT_BTREE *btree; WT_CURSOR_BTREE *cbt; - WT_SESSION_IMPL *session; WT_UPDATE *upd; *updp = NULL; cbt = (WT_CURSOR_BTREE *)cursor; btree = cbt->btree; - session = (WT_SESSION_IMPL *)cursor->session; upd = NULL; /* -Wuninitialized */ /* @@ -499,24 +488,43 @@ __wt_btcur_search_uncommitted(WT_CURSOR *cursor, WT_UPDATE **updp) __cursor_col_search(cbt, NULL, NULL)); /* - * Ideally exact match should be found, as this transaction has searched for updates done by - * itself. But, we cannot be sure of finding one, as pre processing of this prepared transaction - * updates could have happened as part of resolving earlier transaction operations. + * Ideally an exact match will be found, as this transaction is searching for updates done by + * itself. But, we cannot be sure of finding one, as pre-processing of the updates could have + * happened as part of resolving earlier transaction operations. */ if (cbt->compare != 0) return (0); + /* Get any uncommitted update from the in-memory page. */ + switch (cbt->btree->type) { + case BTREE_ROW: + /* + * Any update must be either in the insert list, in which case search will have returned a + * pointer for us, or as an update in a particular key's update list, in which case the slot + * will be returned to us. In either case, we want the most recent update (any update + * attempted after the prepare would have failed). + */ + if (cbt->ins != NULL) + upd = cbt->ins->upd; + else if (cbt->ref->page->modify != NULL && cbt->ref->page->modify->mod_row_update != NULL) + upd = cbt->ref->page->modify->mod_row_update[cbt->slot]; + break; + case BTREE_COL_FIX: + case BTREE_COL_VAR: + /* + * Any update must be in the insert list and we want the most recent update (any update + * attempted after the prepare would have failed). + */ + if (cbt->ins != NULL) + upd = cbt->ins->upd; + break; + } + /* - * Get the uncommitted update from the cursor. For column store there will be always a insert - * structure for updates irrespective of fixed length or variable length. + * Like regular uncommitted updates, pages with prepared updates are pinned to the cache and can + * never be written to the history store. Therefore, there is no need to do a search here for + * uncommitted updates. */ - if (cbt->ins != NULL) - upd = cbt->ins->upd; - else if (cbt->btree->type == BTREE_ROW) { - WT_ASSERT(session, cbt->btree->type == BTREE_ROW && cbt->ref->page->modify != NULL && - cbt->ref->page->modify->mod_row_update != NULL); - upd = cbt->ref->page->modify->mod_row_update[cbt->slot]; - } *updp = upd; return (0); @@ -795,8 +803,11 @@ __wt_btcur_insert(WT_CURSOR_BTREE *cbt) WT_RET(__cursor_size_chk(session, &cursor->key)); WT_RET(__cursor_size_chk(session, &cursor->value)); + WT_RET_ASSERT( + session, S2BT(session) == btree, WT_PANIC, "btree differs unexpectedly from session's btree"); + /* It's no longer possible to bulk-load into the tree. */ - __cursor_disable_bulk(session, btree); + __wt_cursor_disable_bulk(session); /* * Insert a new record if WT_CURSTD_APPEND configured, (ignoring any application set record @@ -924,20 +935,27 @@ static int __curfile_update_check(WT_CURSOR_BTREE *cbt) { WT_BTREE *btree; + WT_PAGE *page; WT_SESSION_IMPL *session; + WT_UPDATE *upd; btree = cbt->btree; + page = cbt->ref->page; session = (WT_SESSION_IMPL *)cbt->iface.session; + upd = NULL; if (cbt->compare != 0) return (0); + if (cbt->ins != NULL) - return (__wt_txn_update_check(session, cbt->ins->upd)); + upd = cbt->ins->upd; + else if (btree->type == BTREE_ROW && page->modify != NULL && + page->modify->mod_row_update != NULL) + upd = page->modify->mod_row_update[cbt->slot]; + else if (btree->type != BTREE_COL_VAR) + return (0); - if (btree->type == BTREE_ROW && cbt->ref->page->modify != NULL && - cbt->ref->page->modify->mod_row_update != NULL) - return (__wt_txn_update_check(session, cbt->ref->page->modify->mod_row_update[cbt->slot])); - return (0); + return (__wt_txn_update_check(session, cbt, upd)); } /* @@ -1199,8 +1217,11 @@ __btcur_update(WT_CURSOR_BTREE *cbt, WT_ITEM *value, u_int modify_type) session = (WT_SESSION_IMPL *)cursor->session; yield_count = sleep_usecs = 0; + WT_RET_ASSERT( + session, S2BT(session) == btree, WT_PANIC, "btree differs unexpectedly from session's btree"); + /* It's no longer possible to bulk-load into the tree. */ - __cursor_disable_bulk(session, btree); + __wt_cursor_disable_bulk(session); /* Save the cursor state. */ __cursor_state_save(cursor, &state); @@ -1335,7 +1356,6 @@ done: */ ret = __wt_key_return(cbt); break; - case WT_UPDATE_BIRTHMARK: case WT_UPDATE_TOMBSTONE: default: return (__wt_illegal_value(session, modify_type)); @@ -1453,7 +1473,7 @@ __wt_btcur_modify(WT_CURSOR_BTREE *cbt, WT_MODIFY *entries, int nentries) if (!F_ISSET(cursor, WT_CURSTD_KEY_INT) || !F_ISSET(cursor, WT_CURSTD_VALUE_INT)) WT_ERR(__wt_btcur_search(cbt)); - WT_ERR(__wt_modify_pack(cursor, &modify, entries, nentries)); + WT_ERR(__wt_modify_pack(cursor, entries, nentries, &modify)); orig = cursor->value.size; WT_ERR(__wt_modify_apply(cursor, modify->data)); @@ -1863,9 +1883,9 @@ __wt_btcur_close(WT_CURSOR_BTREE *cbt, bool lowlevel) session = (WT_SESSION_IMPL *)cbt->iface.session; /* - * The in-memory split and lookaside table code creates low-level btree cursors to search/modify - * leaf pages. Those cursors don't hold hazard pointers, nor are they counted in the session - * handle's cursor count. Skip the usual cursor tear-down in that case. + * The in-memory split and history store table code creates low-level btree cursors to + * search/modify leaf pages. Those cursors don't hold hazard pointers, nor are they counted in + * the session handle's cursor count. Skip the usual cursor tear-down in that case. */ if (!lowlevel) ret = __cursor_reset(cbt); diff --git a/src/third_party/wiredtiger/src/btree/bt_debug.c b/src/third_party/wiredtiger/src/btree/bt_debug.c index ad1805dbb7d..63f8113eceb 100644 --- a/src/third_party/wiredtiger/src/btree/bt_debug.c +++ b/src/third_party/wiredtiger/src/btree/bt_debug.c @@ -41,6 +41,7 @@ static int __debug_col_skip(WT_DBG *, WT_INSERT_HEAD *, const char *, bool); static int __debug_config(WT_SESSION_IMPL *, WT_DBG *, const char *); static int __debug_dsk_cell(WT_DBG *, const WT_PAGE_HEADER *); static int __debug_dsk_col_fix(WT_DBG *, const WT_PAGE_HEADER *); +static int __debug_modify(WT_DBG *, WT_UPDATE *, const char *); static int __debug_page(WT_DBG *, WT_REF *, uint32_t); static int __debug_page_col_fix(WT_DBG *, WT_REF *); static int __debug_page_col_int(WT_DBG *, WT_PAGE *, uint32_t); @@ -124,16 +125,8 @@ __debug_item_key(WT_DBG *ds, const char *tag, const void *data_arg, size_t size) session = ds->session; - /* - * If the format is 'S', it's a string and our version of it may not yet be nul-terminated. - */ - if (WT_STREQ(ds->key_format, "S") && ((char *)data_arg)[size - 1] != '\0') { - WT_RET(__wt_buf_fmt(session, ds->t2, "%.*s", (int)size, (char *)data_arg)); - data_arg = ds->t2->data; - size = ds->t2->size + 1; - } return (ds->f(ds, "\t%s%s{%s}\n", tag == NULL ? "" : tag, tag == NULL ? "" : " ", - __wt_buf_set_printable_format(session, data_arg, size, ds->key_format, ds->t1))); + __wt_key_string(session, data_arg, size, ds->key_format, ds->t1))); } /* @@ -163,6 +156,21 @@ __debug_item_value(WT_DBG *ds, const char *tag, const void *data_arg, size_t siz } /* + * __debug_time_pairs -- + * Dump a set of start and stop time pairs, with an optional tag. + */ +static inline int +__debug_time_pairs(WT_DBG *ds, const char *tag, wt_timestamp_t start_ts, uint64_t start_txn, + wt_timestamp_t stop_ts, uint64_t stop_txn) +{ + char tp_string[2][WT_TP_STRING_SIZE]; + + return (ds->f(ds, "\t%s%s%s,%s\n", tag == NULL ? "" : tag, tag == NULL ? "" : " ", + __wt_time_pair_to_string(start_ts, start_txn, tp_string[0]), + __wt_time_pair_to_string(stop_ts, stop_txn, tp_string[1]))); +} + +/* * __dmsg_event -- * Send a debug message to the event handler. */ @@ -444,8 +452,6 @@ __wt_debug_disk(WT_SESSION_IMPL *session, const WT_PAGE_HEADER *dsk, const char WT_ERR(ds->f(ds, ", empty-all")); if (F_ISSET(dsk, WT_PAGE_EMPTY_V_NONE)) WT_ERR(ds->f(ds, ", empty-none")); - if (F_ISSET(dsk, WT_PAGE_LAS_UPDATE)) - WT_ERR(ds->f(ds, ", LAS-update")); WT_ERR(ds->f(ds, ", generation %" PRIu64 "\n", dsk->write_gen)); @@ -560,7 +566,7 @@ __debug_tree_shape_worker(WT_DBG *ds, WT_REF *ref, int level) session = ds->session; - if (WT_PAGE_IS_INTERNAL(ref->page)) { + if (F_ISSET(ref, WT_REF_FLAG_INTERNAL)) { WT_RET(ds->f(ds, "%*s" "I" @@ -698,25 +704,116 @@ __wt_debug_cursor_page(void *cursor_arg, const char *ofile) } /* - * __wt_debug_cursor_las -- - * Dump the LAS tree given a user cursor. + * __wt_debug_cursor_tree_hs -- + * Dump the history store tree given a user cursor. */ int -__wt_debug_cursor_las(void *cursor_arg, const char *ofile) +__wt_debug_cursor_tree_hs(void *cursor_arg, const char *ofile) WT_GCC_FUNC_ATTRIBUTE((visibility("default"))) { - WT_CONNECTION_IMPL *conn; WT_CURSOR *cursor; WT_CURSOR_BTREE *cbt; - WT_SESSION_IMPL *las_session; + WT_DECL_RET; + WT_SESSION_IMPL *session; + uint32_t session_flags; + bool is_owner; cursor = cursor_arg; - conn = S2C((WT_SESSION_IMPL *)cursor->session); - las_session = conn->cache->las_session[0]; - if (las_session == NULL) - return (0); - cbt = (WT_CURSOR_BTREE *)las_session->las_cursor; - return (__wt_debug_tree_all(las_session, cbt->btree, NULL, ofile)); + session = (WT_SESSION_IMPL *)cursor->session; + session_flags = 0; /* [-Werror=maybe-uninitialized] */ + + WT_RET(__wt_hs_cursor(session, &session_flags, &is_owner)); + cbt = (WT_CURSOR_BTREE *)session->hs_cursor; + ret = __wt_debug_tree_all(session, cbt->btree, NULL, ofile); + WT_TRET(__wt_hs_cursor_close(session, session_flags, is_owner)); + + return (ret); +} + +/* + * __wt_debug_cursor_hs -- + * Dump information pointed to by a single history store cursor. + */ +int +__wt_debug_cursor_hs(WT_SESSION_IMPL *session, WT_CURSOR *hs_cursor) +{ + WT_DBG *ds, _ds; + WT_DECL_ITEM(hs_key); + WT_DECL_ITEM(hs_value); + WT_DECL_RET; + WT_TIME_PAIR start, stop; + WT_UPDATE *upd; + wt_timestamp_t hs_durable_ts; + size_t notused; + uint64_t hs_upd_type_full; + uint32_t hs_btree_id; + uint8_t hs_prep_state, hs_upd_type; + + ds = &_ds; + notused = 0; + + WT_ERR(__wt_scr_alloc(session, 0, &hs_key)); + WT_ERR(__wt_scr_alloc(session, 0, &hs_value)); + WT_ERR(__debug_config(session, ds, NULL)); + + WT_ERR(hs_cursor->get_key(hs_cursor, &hs_btree_id, hs_key, &start.timestamp, &start.txnid, + &stop.timestamp, &stop.txnid)); + + WT_ERR(__debug_time_pairs(ds, "T", start.timestamp, start.txnid, stop.timestamp, stop.txnid)); + + WT_ERR( + hs_cursor->get_value(hs_cursor, &hs_durable_ts, &hs_prep_state, &hs_upd_type_full, hs_value)); + hs_upd_type = (uint8_t)hs_upd_type_full; + switch (hs_upd_type) { + case WT_UPDATE_MODIFY: + WT_ERR(__wt_update_alloc(session, hs_value, &upd, ¬used, hs_upd_type)); + WT_ERR(__debug_modify(ds, upd, "\tM ")); + break; + case WT_UPDATE_STANDARD: + WT_ERR(__debug_item_value(ds, "V", hs_value->data, hs_value->size)); + break; + default: + /* + * Currently, we expect only modifies or full values to be exposed by hs_cursors. This means + * we can ignore other types for now. + */ + WT_ASSERT(session, hs_upd_type == WT_UPDATE_MODIFY || hs_upd_type == WT_UPDATE_STANDARD); + break; + } + +err: + __wt_scr_free(session, &hs_key); + __wt_scr_free(session, &hs_value); + WT_RET(__debug_wrapup(ds)); + + return (ret); +} + +/* + * __wt_debug_key_value -- + * Dump information about a key and/or value. + */ +int +__wt_debug_key_value(WT_SESSION_IMPL *session, WT_ITEM *key, WT_CELL_UNPACK *value) +{ + WT_DBG *ds, _ds; + WT_DECL_RET; + + ds = &_ds; + + WT_ERR(__debug_config(session, ds, NULL)); + + if (key != NULL) + WT_ERR(__debug_item_key(ds, "K", key->data, key->size)); + if (value != NULL) { + WT_ERR(__debug_time_pairs( + ds, "T", value->start_ts, value->start_txn, value->stop_ts, value->stop_txn)); + WT_ERR(__debug_cell_data(ds, NULL, value != NULL ? value->type : 0, "V", value)); + } + +err: + WT_RET(__debug_wrapup(ds)); + return (ret); } /* @@ -801,8 +898,8 @@ __debug_page_metadata(WT_DBG *ds, WT_REF *ref) uint64_t split_gen; uint32_t entries; - session = ds->session; page = ref->page; + session = ds->session; mod = page->modify; split_gen = 0; @@ -836,6 +933,8 @@ __debug_page_metadata(WT_DBG *ds, WT_REF *ref) } WT_RET(ds->f(ds, ": %s\n", __wt_page_type_string(page->type))); + WT_RET(__debug_ref(ds, ref)); + WT_RET(ds->f(ds, "\t" "disk %p", @@ -881,9 +980,7 @@ __debug_page_metadata(WT_DBG *ds, WT_REF *ref) if (mod != NULL) WT_RET(ds->f(ds, ", page-state=%" PRIu32, mod->page_state)); WT_RET(ds->f(ds, ", memory-size %" WT_SIZET_FMT, page->memory_footprint)); - WT_RET(ds->f(ds, "\n")); - - return (0); + return (ds->f(ds, "\n")); } /* @@ -1125,7 +1222,7 @@ __debug_row_skip(WT_DBG *ds, WT_INSERT_HEAD *head) * Dump a modify update. */ static int -__debug_modify(WT_DBG *ds, WT_UPDATE *upd) +__debug_modify(WT_DBG *ds, WT_UPDATE *upd, const char *tag) { size_t nentries, data_size, offset, size; const size_t *p; @@ -1135,7 +1232,7 @@ __debug_modify(WT_DBG *ds, WT_UPDATE *upd) memcpy(&nentries, p++, sizeof(size_t)); data = upd->data + sizeof(size_t) + (nentries * 3 * sizeof(size_t)); - WT_RET(ds->f(ds, "%" WT_SIZET_FMT ": ", nentries)); + WT_RET(ds->f(ds, "%s%" WT_SIZET_FMT ": ", tag != NULL ? tag : "", nentries)); for (; nentries-- > 0; data += data_size) { memcpy(&data_size, p++, sizeof(size_t)); memcpy(&offset, p++, sizeof(size_t)); @@ -1164,12 +1261,9 @@ __debug_update(WT_DBG *ds, WT_UPDATE *upd, bool hexbyte) case WT_UPDATE_INVALID: WT_RET(ds->f(ds, "\tvalue {invalid}\n")); break; - case WT_UPDATE_BIRTHMARK: - WT_RET(ds->f(ds, "\tvalue {birthmark}\n")); - break; case WT_UPDATE_MODIFY: WT_RET(ds->f(ds, "\tvalue {modify: ")); - WT_RET(__debug_modify(ds, upd)); + WT_RET(__debug_modify(ds, upd, NULL)); WT_RET(ds->f(ds, "}\n")); break; case WT_UPDATE_RESERVE: @@ -1226,51 +1320,60 @@ __debug_update(WT_DBG *ds, WT_UPDATE *upd, bool hexbyte) } /* - * __debug_ref -- - * Dump a WT_REF structure. + * __debug_ref_state -- + * Return a string representing the WT_REF state. */ -static int -__debug_ref(WT_DBG *ds, WT_REF *ref) +static const char * +__debug_ref_state(u_int state) { - WT_SESSION_IMPL *session; - size_t addr_size; - const uint8_t *addr; - const char *state; - - session = ds->session; - - switch (ref->state) { + switch (state) { case WT_REF_DISK: - state = "disk"; - break; + return ("disk"); case WT_REF_DELETED: - state = "deleted"; - break; + return ("deleted"); case WT_REF_LOCKED: - state = "locked"; - break; - case WT_REF_LOOKASIDE: - state = "lookaside"; - break; + return ("locked"); case WT_REF_MEM: - state = "memory"; - break; - case WT_REF_READING: - state = "reading"; - break; + return ("memory"); case WT_REF_SPLIT: - state = "split"; - break; + return ("split"); default: - state = "INVALID"; - break; + return ("INVALID"); } + /* NOTREACHED */ +} - __wt_ref_info(session, ref, &addr, &addr_size, NULL); - return (ds->f(ds, - "\t" - "%p %s %s\n", - (void *)ref, state, __wt_addr_string(session, addr, addr_size, ds->t1))); +/* + * __debug_ref -- + * Dump a WT_REF structure. + */ +static int +__debug_ref(WT_DBG *ds, WT_REF *ref) +{ + WT_ADDR_COPY addr; + WT_SESSION_IMPL *session; + char tp_string[2][WT_TP_STRING_SIZE]; + char ts_string[2][WT_TS_INT_STRING_SIZE]; + + session = ds->session; + + WT_RET(ds->f(ds, "\t%p, ", (void *)ref)); + WT_RET(ds->f(ds, "%s", __debug_ref_state(ref->state))); + if (F_ISSET(ref, WT_REF_FLAG_INTERNAL)) + WT_RET(ds->f(ds, ", %s", "internal")); + if (F_ISSET(ref, WT_REF_FLAG_LEAF)) + WT_RET(ds->f(ds, ", %s", "leaf")); + if (F_ISSET(ref, WT_REF_FLAG_READING)) + WT_RET(ds->f(ds, ", %s", "reading")); + + if (__wt_ref_addr_copy(session, ref, &addr)) + WT_RET(ds->f(ds, ", start/stop durable ts %s,%s, start/stop ts/txn %s,%s, %s", + __wt_timestamp_to_string(addr.start_durable_ts, ts_string[0]), + __wt_timestamp_to_string(addr.stop_durable_ts, ts_string[1]), + __wt_time_pair_to_string(addr.oldest_start_ts, addr.oldest_start_txn, tp_string[0]), + __wt_time_pair_to_string(addr.newest_stop_ts, addr.newest_stop_txn, tp_string[1]), + __wt_addr_string(session, addr.addr, addr.size, ds->t1))); + return (ds->f(ds, "\n")); } /* @@ -1283,7 +1386,8 @@ __debug_cell(WT_DBG *ds, const WT_PAGE_HEADER *dsk, WT_CELL_UNPACK *unpack) WT_DECL_ITEM(buf); WT_DECL_RET; WT_SESSION_IMPL *session; - char ts_string[3][WT_TS_INT_STRING_SIZE]; + char tp_string[2][WT_TP_STRING_SIZE]; + char ts_string[2][WT_TS_INT_STRING_SIZE]; session = ds->session; @@ -1325,10 +1429,11 @@ __debug_cell(WT_DBG *ds, const WT_PAGE_HEADER *dsk, WT_CELL_UNPACK *unpack) case WT_CELL_ADDR_INT: case WT_CELL_ADDR_LEAF: case WT_CELL_ADDR_LEAF_NO: - WT_RET(ds->f(ds, ", ts/txn %s,%s/%" PRIu64 ",%s/%" PRIu64, - __wt_timestamp_to_string(unpack->newest_durable_ts, ts_string[0]), - __wt_timestamp_to_string(unpack->oldest_start_ts, ts_string[1]), unpack->oldest_start_txn, - __wt_timestamp_to_string(unpack->newest_stop_ts, ts_string[2]), unpack->newest_stop_txn)); + WT_RET(ds->f(ds, ", ts/txn %s,%s,%s,%s", + __wt_timestamp_to_string(unpack->newest_start_durable_ts, ts_string[0]), + __wt_timestamp_to_string(unpack->newest_stop_durable_ts, ts_string[1]), + __wt_time_pair_to_string(unpack->oldest_start_ts, unpack->oldest_start_txn, tp_string[0]), + __wt_time_pair_to_string(unpack->newest_stop_ts, unpack->newest_stop_txn, tp_string[1]))); break; case WT_CELL_DEL: case WT_CELL_VALUE: @@ -1336,9 +1441,9 @@ __debug_cell(WT_DBG *ds, const WT_PAGE_HEADER *dsk, WT_CELL_UNPACK *unpack) case WT_CELL_VALUE_OVFL: case WT_CELL_VALUE_OVFL_RM: case WT_CELL_VALUE_SHORT: - WT_RET(ds->f(ds, ", ts/txn %s/%" PRIu64 ",%s/%" PRIu64, - __wt_timestamp_to_string(unpack->start_ts, ts_string[0]), unpack->start_txn, - __wt_timestamp_to_string(unpack->stop_ts, ts_string[1]), unpack->stop_txn)); + WT_RET(ds->f(ds, ", ts/txn %s,%s", + __wt_time_pair_to_string(unpack->start_ts, unpack->start_txn, tp_string[0]), + __wt_time_pair_to_string(unpack->stop_ts, unpack->stop_txn, tp_string[1]))); break; } diff --git a/src/third_party/wiredtiger/src/btree/bt_delete.c b/src/third_party/wiredtiger/src/btree/bt_delete.c index 87f1e4f3d6f..69651d9b5ca 100644 --- a/src/third_party/wiredtiger/src/btree/bt_delete.c +++ b/src/third_party/wiredtiger/src/btree/bt_delete.c @@ -18,9 +18,9 @@ * pages of the truncate range, then walks the tree with a flag so the tree walk code skips reading * eligible pages within the range and instead just marks them as deleted, by changing their WT_REF * state to WT_REF_DELETED. Pages ineligible for this fast path include pages already in the cache, - * having overflow items, or requiring lookaside records. Ineligible pages are read and have their - * rows updated/deleted individually. The transaction for the delete operation is stored in memory - * referenced by the WT_REF.page_del field. + * having overflow items, or requiring history store records. Ineligible pages are read and have + * their rows updated/deleted individually. The transaction for the delete operation is stored in + * memory referenced by the WT_REF.page_del field. * * Future cursor walks of the tree will skip the deleted page based on the transaction stored for * the delete, but it gets more complicated if a read is done using a random key, or a cursor walk @@ -56,15 +56,15 @@ int __wt_delete_page(WT_SESSION_IMPL *session, WT_REF *ref, bool *skipp) { - WT_ADDR *ref_addr; + WT_ADDR_COPY addr; WT_DECL_RET; - uint32_t previous_state; + uint8_t previous_state; *skipp = false; /* If we have a clean page in memory, attempt to evict it. */ previous_state = ref->state; - if ((previous_state == WT_REF_MEM || previous_state == WT_REF_LIMBO) && + if (previous_state == WT_REF_MEM && WT_REF_CAS_STATE(session, ref, previous_state, WT_REF_LOCKED)) { if (__wt_page_is_modified(ref->page)) { WT_REF_SET_STATE(ref, previous_state); @@ -84,7 +84,6 @@ __wt_delete_page(WT_SESSION_IMPL *session, WT_REF *ref, bool *skipp) previous_state = ref->state; switch (previous_state) { case WT_REF_DISK: - case WT_REF_LOOKASIDE: break; default: return (0); @@ -109,16 +108,14 @@ __wt_delete_page(WT_SESSION_IMPL *session, WT_REF *ref, bool *skipp) * discarded. The way we figure that out is to check the page's cell type, cells for leaf pages * without overflow items are special. * - * To look at an on-page cell, we need to look at the parent page, and that's dangerous, our - * parent page could change without warning if the parent page were to split, deepening the - * tree. We can look at the parent page itself because the page can't change underneath us. - * However, if the parent page splits, our reference address can change; we don't care what - * version of it we read, as long as we don't read it twice. + * Additionally, if the aggregated start time pair on the page is not visible to us then we + * cannot truncate the page. */ - WT_ORDERED_READ(ref_addr, ref->addr); - if (ref_addr != NULL && (__wt_off_page(ref->home, ref_addr) ? - ref_addr->type != WT_ADDR_LEAF_NO : - __wt_cell_type_raw((WT_CELL *)ref_addr) != WT_CELL_ADDR_LEAF_NO)) + if (!__wt_ref_addr_copy(session, ref, &addr)) + goto err; + if (addr.type != WT_ADDR_LEAF_NO) + goto err; + if (!__wt_txn_visible(session, addr.oldest_start_txn, addr.oldest_start_ts)) goto err; /* @@ -158,7 +155,7 @@ __wt_delete_page_rollback(WT_SESSION_IMPL *session, WT_REF *ref) { WT_UPDATE **updp; uint64_t sleep_usecs, yield_count; - uint32_t current_state; + uint8_t current_state; bool locked; /* Lock the reference. We cannot access ref->page_del except when locked. */ @@ -173,9 +170,6 @@ __wt_delete_page_rollback(WT_SESSION_IMPL *session, WT_REF *ref) locked = true; break; case WT_REF_DISK: - case WT_REF_LIMBO: - case WT_REF_LOOKASIDE: - case WT_REF_READING: default: return (__wt_illegal_value(session, current_state)); } @@ -238,13 +232,10 @@ __wt_delete_page_skip(WT_SESSION_IMPL *session, WT_REF *ref, bool visible_all) * being read into memory right now, though, and the page could switch to an in-memory state at * any time. Lock down the structure, just to be safe. */ - if (ref->page_del == NULL && ref->page_las == NULL) - return (true); - if (!WT_REF_CAS_STATE(session, ref, WT_REF_DELETED, WT_REF_LOCKED)) return (false); - skip = !__wt_page_del_active(session, ref, visible_all) && !__wt_page_las_active(session, ref); + skip = !__wt_page_del_active(session, ref, visible_all); /* * The page_del structure can be freed as soon as the delete is stable: it is only read when the @@ -301,6 +292,7 @@ __wt_delete_page_instantiate(WT_SESSION_IMPL *session, WT_REF *ref) WT_PAGE *page; WT_PAGE_DELETED *page_del; WT_ROW *rip; + WT_TIME_PAIR start, stop; WT_UPDATE **upd_array, *upd; size_t size; uint32_t count, i; @@ -349,7 +341,7 @@ __wt_delete_page_instantiate(WT_SESSION_IMPL *session, WT_REF *ref) /* * Allocate the per-page update array if one doesn't already exist. (It might already exist - * because deletes are instantiated after lookaside table updates.) + * because deletes are instantiated after the history store table updates.) */ if (page->entries != 0 && page->modify->mod_row_update == NULL) WT_RET(__wt_calloc_def(session, page->entries, &page->modify->mod_row_update)); @@ -387,22 +379,29 @@ __wt_delete_page_instantiate(WT_SESSION_IMPL *session, WT_REF *ref) page_del->update_list[count++] = upd; } WT_ROW_FOREACH (page, rip, i) { - WT_ERR(__tombstone_update_alloc(session, page_del, &upd, &size)); - upd->next = upd_array[WT_ROW_SLOT(page, rip)]; - upd_array[WT_ROW_SLOT(page, rip)] = upd; - - if (page_del != NULL) - page_del->update_list[count++] = upd; - - if ((insert = WT_ROW_INSERT(page, rip)) != NULL) - WT_SKIP_FOREACH (ins, insert) { - WT_ERR(__tombstone_update_alloc(session, page_del, &upd, &size)); - upd->next = ins->upd; - ins->upd = upd; - - if (page_del != NULL) - page_del->update_list[count++] = upd; - } + /* + * Retrieve the stop time pair from the page's row. If we find an existing stop time pair we + * don't need to append a tombstone. + */ + __wt_read_row_time_pairs(session, page, rip, &start, &stop); + if (stop.timestamp == WT_TS_MAX && stop.txnid == WT_TXN_MAX) { + WT_ERR(__tombstone_update_alloc(session, page_del, &upd, &size)); + upd->next = upd_array[WT_ROW_SLOT(page, rip)]; + upd_array[WT_ROW_SLOT(page, rip)] = upd; + + if (page_del != NULL) + page_del->update_list[count++] = upd; + + if ((insert = WT_ROW_INSERT(page, rip)) != NULL) + WT_SKIP_FOREACH (ins, insert) { + WT_ERR(__tombstone_update_alloc(session, page_del, &upd, &size)); + upd->next = ins->upd; + ins->upd = upd; + + if (page_del != NULL) + page_del->update_list[count++] = upd; + } + } } __wt_cache_page_inmem_incr(session, page, size); diff --git a/src/third_party/wiredtiger/src/btree/bt_discard.c b/src/third_party/wiredtiger/src/btree/bt_discard.c index e3f1413c7d4..2878391fd53 100644 --- a/src/third_party/wiredtiger/src/btree/bt_discard.c +++ b/src/third_party/wiredtiger/src/btree/bt_discard.c @@ -215,6 +215,23 @@ __free_page_modify(WT_SESSION_IMPL *session, WT_PAGE *page) } /* + * __wt_ref_addr_free -- + * Free the address in a reference, if necessary. + */ +void +__wt_ref_addr_free(WT_SESSION_IMPL *session, WT_REF *ref) +{ + if (ref->addr == NULL) + return; + + if (ref->home == NULL || __wt_off_page(ref->home, ref->addr)) { + __wt_free(session, ((WT_ADDR *)ref->addr)->addr); + __wt_free(session, ref->addr); + } + ref->addr = NULL; +} + +/* * __wt_free_ref -- * Discard the contents of a WT_REF structure (optionally including the pages it references). */ @@ -227,6 +244,12 @@ __wt_free_ref(WT_SESSION_IMPL *session, WT_REF *ref, int page_type, bool free_pa return; /* + * We create WT_REFs in many places, assert a WT_REF has been configured as either an internal + * page or a leaf page, to catch any we've missed. + */ + WT_ASSERT(session, F_ISSET(ref, WT_REF_FLAG_INTERNAL) || F_ISSET(ref, WT_REF_FLAG_LEAF)); + + /* * Optionally free the referenced pages. (The path to free referenced page is used for error * cleanup, no instantiated and then discarded page should have WT_REF entries with real pages. * The page may have been marked dirty as well; page discard checks for that, so we mark it @@ -256,8 +279,7 @@ __wt_free_ref(WT_SESSION_IMPL *session, WT_REF *ref, int page_type, bool free_pa /* Free any address allocation. */ __wt_ref_addr_free(session, ref); - /* Free any lookaside or page-deleted information. */ - __wt_free(session, ref->page_las); + /* Free any page-deleted information. */ if (ref->page_del != NULL) { __wt_free(session, ref->page_del->update_list); __wt_free(session, ref->page_del); @@ -381,7 +403,7 @@ __free_skip_list(WT_SESSION_IMPL *session, WT_INSERT *ins, bool update_ignore) for (; ins != NULL; ins = next) { if (!update_ignore) - __wt_free_update_list(session, ins->upd); + __wt_free_update_list(session, &ins->upd); next = WT_SKIP_NEXT(ins); __wt_free(session, ins); } @@ -403,8 +425,7 @@ __free_update( */ if (!update_ignore) for (updp = update_head; entries > 0; --entries, ++updp) - if (*updp != NULL) - __wt_free_update_list(session, *updp); + __wt_free_update_list(session, updp); /* Free the update array. */ __wt_free(session, update_head); @@ -416,12 +437,13 @@ __free_update( * structure and its associated data. */ void -__wt_free_update_list(WT_SESSION_IMPL *session, WT_UPDATE *upd) +__wt_free_update_list(WT_SESSION_IMPL *session, WT_UPDATE **updp) { - WT_UPDATE *next; + WT_UPDATE *next, *upd; - for (; upd != NULL; upd = next) { + for (upd = *updp; upd != NULL; upd = next) { next = upd->next; __wt_free(session, upd); } + *updp = NULL; } diff --git a/src/third_party/wiredtiger/src/btree/bt_handle.c b/src/third_party/wiredtiger/src/btree/bt_handle.c index d07b7b859ba..c0bf610e984 100644 --- a/src/third_party/wiredtiger/src/btree/bt_handle.c +++ b/src/third_party/wiredtiger/src/btree/bt_handle.c @@ -15,32 +15,6 @@ static int __btree_preload(WT_SESSION_IMPL *); static int __btree_tree_open_empty(WT_SESSION_IMPL *, bool); /* - * __wt_btree_page_version_config -- - * Select a Btree page format. - */ -void -__wt_btree_page_version_config(WT_SESSION_IMPL *session) -{ - WT_CONNECTION_IMPL *conn; - - conn = S2C(session); - -/* - * Write timestamp format pages if at the right version or if configured at build-time. - * - * WiredTiger version where timestamp page format is written. This is a future release, and the - * values may require update when the release is named. - */ -#define WT_VERSION_TS_MAJOR 3 -#define WT_VERSION_TS_MINOR 3 - __wt_process.page_version_ts = - conn->compat_major >= WT_VERSION_TS_MAJOR && conn->compat_minor >= WT_VERSION_TS_MINOR; -#if defined(HAVE_PAGE_VERSION_TS) - __wt_process.page_version_ts = true; -#endif -} - -/* * __btree_clear -- * Clear a Btree, either on handle discard or re-open. */ @@ -136,12 +110,6 @@ __wt_btree_open(WT_SESSION_IMPL *session, const char *op_cfg[]) /* Initialize and configure the WT_BTREE structure. */ WT_ERR(__btree_conf(session, &ckpt)); - /* - * We could be a re-open of a table that was put in the lookaside dropped list. Remove our id - * from that list. - */ - __wt_las_remove_dropped(session); - /* Connect to the underlying block manager. */ filename = dhandle->name; if (!WT_PREFIX_SKIP(filename, "file:")) @@ -251,12 +219,11 @@ __wt_btree_close(WT_SESSION_IMPL *session) F_SET(btree, WT_BTREE_CLOSED); /* - * If closing a tree let sweep drop lookaside entries for it. + * Verify the history store state. If the history store is open and this btree has history store + * entries, it can't be a metadata file, nor can it be the history store file. */ - if (F_ISSET(S2C(session), WT_CONN_LOOKASIDE_OPEN) && btree->lookaside_entries) { - WT_ASSERT(session, !WT_IS_METADATA(btree->dhandle) && !F_ISSET(btree, WT_BTREE_LOOKASIDE)); - WT_TRET(__wt_las_save_dropped(session)); - } + WT_ASSERT(session, !F_ISSET(S2C(session), WT_CONN_HS_OPEN) || !btree->hs_entries || + (!WT_IS_METADATA(btree->dhandle) && !WT_IS_HS(btree))); /* * If we turned eviction off and never turned it back on, do that now, otherwise the counter @@ -543,6 +510,12 @@ __btree_conf(WT_SESSION_IMPL *session, WT_CKPT *ckpt) } } + /* Set special flags for the history store table. */ + if (strcmp(session->dhandle->name, WT_HS_URI) == 0) { + F_SET(btree, WT_BTREE_HS); + F_SET(btree, WT_BTREE_NO_LOGGING); + } + /* Configure encryption. */ WT_RET(__wt_btree_config_encryptor(session, cfg, &btree->kencryptor)); @@ -572,6 +545,7 @@ __wt_root_ref_init(WT_SESSION_IMPL *session, WT_REF *root_ref, WT_PAGE *root, bo memset(root_ref, 0, sizeof(*root_ref)); root_ref->page = root; + F_SET(root_ref, WT_REF_FLAG_INTERNAL); WT_REF_SET_STATE(root_ref, WT_REF_MEM); root_ref->ref_recno = is_recno ? 1 : WT_RECNO_OOB; @@ -644,7 +618,7 @@ __wt_btree_tree_open(WT_SESSION_IMPL *session, const uint8_t *addr, size_t addr_ * the disk image on return, the in-memory object steals it. */ WT_ERR(__wt_page_inmem(session, NULL, dsk.data, - WT_DATA_IN_ITEM(&dsk) ? WT_PAGE_DISK_ALLOC : WT_PAGE_DISK_MAPPED, true, &page)); + WT_DATA_IN_ITEM(&dsk) ? WT_PAGE_DISK_ALLOC : WT_PAGE_DISK_MAPPED, &page)); dsk.mem = NULL; /* Finish initializing the root, root reference links. */ @@ -666,12 +640,12 @@ __btree_tree_open_empty(WT_SESSION_IMPL *session, bool creation) { WT_BTREE *btree; WT_DECL_RET; - WT_PAGE *leaf, *root; + WT_PAGE *root; WT_PAGE_INDEX *pindex; WT_REF *ref; btree = S2BT(session); - root = leaf = NULL; + root = NULL; ref = NULL; /* @@ -682,15 +656,13 @@ __btree_tree_open_empty(WT_SESSION_IMPL *session, bool creation) btree->original = 1; /* - * A note about empty trees: the initial tree is a single root page. - * It has a single reference to a leaf page, marked deleted. The leaf - * page will be created by the first update. If the root is evicted - * without being modified, that's OK, nothing is ever written. + * A note about empty trees: the initial tree is a single root page. It has a single reference + * to a leaf page, marked deleted. The leaf page will be created by the first update. If the + * root is evicted without being modified, that's OK, nothing is ever written. * * !!! - * Be cautious about changing the order of updates in this code: to call - * __wt_page_out on error, we require a correct page setup at each point - * where we might fail. + * Be cautious about changing the order of updates in this code: to call __wt_page_out on error, + * we require a correct page setup at each point where we might fail. */ switch (btree->type) { case BTREE_COL_FIX: @@ -703,6 +675,7 @@ __btree_tree_open_empty(WT_SESSION_IMPL *session, bool creation) ref->home = root; ref->page = NULL; ref->addr = NULL; + F_SET(ref, WT_REF_FLAG_LEAF); WT_REF_SET_STATE(ref, WT_REF_DELETED); ref->ref_recno = 1; break; @@ -715,6 +688,7 @@ __btree_tree_open_empty(WT_SESSION_IMPL *session, bool creation) ref->home = root; ref->page = NULL; ref->addr = NULL; + F_SET(ref, WT_REF_FLAG_LEAF); WT_REF_SET_STATE(ref, WT_REF_DELETED); WT_ERR(__wt_row_ikey_incr(session, root, 0, "", 1, ref)); break; @@ -722,11 +696,11 @@ __btree_tree_open_empty(WT_SESSION_IMPL *session, bool creation) /* Bulk loads require a leaf page for reconciliation: create it now. */ if (F_ISSET(btree, WT_BTREE_BULK)) { - WT_ERR(__wt_btree_new_leaf_page(session, &leaf)); - ref->page = leaf; + WT_ERR(__wt_btree_new_leaf_page(session, ref)); + F_SET(ref, WT_REF_FLAG_LEAF); WT_REF_SET_STATE(ref, WT_REF_MEM); - WT_ERR(__wt_page_modify_init(session, leaf)); - __wt_page_only_modify_set(session, leaf); + WT_ERR(__wt_page_modify_init(session, ref->page)); + __wt_page_only_modify_set(session, ref->page); } /* Finish initializing the root, root reference links. */ @@ -735,8 +709,8 @@ __btree_tree_open_empty(WT_SESSION_IMPL *session, bool creation) return (0); err: - if (leaf != NULL) - __wt_page_out(session, &leaf); + if (ref->page != NULL) + __wt_page_out(session, &ref->page); if (root != NULL) __wt_page_out(session, &root); return (ret); @@ -747,7 +721,7 @@ err: * Create an empty leaf page. */ int -__wt_btree_new_leaf_page(WT_SESSION_IMPL *session, WT_PAGE **pagep) +__wt_btree_new_leaf_page(WT_SESSION_IMPL *session, WT_REF *ref) { WT_BTREE *btree; @@ -755,15 +729,24 @@ __wt_btree_new_leaf_page(WT_SESSION_IMPL *session, WT_PAGE **pagep) switch (btree->type) { case BTREE_COL_FIX: - WT_RET(__wt_page_alloc(session, WT_PAGE_COL_FIX, 0, false, pagep)); + WT_RET(__wt_page_alloc(session, WT_PAGE_COL_FIX, 0, false, &ref->page)); break; case BTREE_COL_VAR: - WT_RET(__wt_page_alloc(session, WT_PAGE_COL_VAR, 0, false, pagep)); + WT_RET(__wt_page_alloc(session, WT_PAGE_COL_VAR, 0, false, &ref->page)); break; case BTREE_ROW: - WT_RET(__wt_page_alloc(session, WT_PAGE_ROW_LEAF, 0, false, pagep)); + WT_RET(__wt_page_alloc(session, WT_PAGE_ROW_LEAF, 0, false, &ref->page)); break; } + + /* + * When deleting a chunk of the name-space, we can delete internal pages. However, if we are + * ever forced to re-instantiate that piece of the namespace, it comes back as a leaf page. + * Reset the WT_REF type as it's possible that it has changed. + */ + F_CLR(ref, WT_REF_FLAG_INTERNAL); + F_SET(ref, WT_REF_FLAG_LEAF); + return (0); } @@ -774,21 +757,18 @@ __wt_btree_new_leaf_page(WT_SESSION_IMPL *session, WT_PAGE **pagep) static int __btree_preload(WT_SESSION_IMPL *session) { + WT_ADDR_COPY addr; WT_BM *bm; WT_BTREE *btree; WT_REF *ref; - size_t addr_size; - const uint8_t *addr; btree = S2BT(session); bm = btree->bm; /* Pre-load the second-level internal pages. */ - WT_INTL_FOREACH_BEGIN (session, btree->root.page, ref) { - __wt_ref_info(session, ref, &addr, &addr_size, NULL); - if (addr != NULL) - WT_RET(bm->preload(bm, session, addr, addr_size)); - } + WT_INTL_FOREACH_BEGIN (session, btree->root.page, ref) + if (__wt_ref_addr_copy(session, ref, &addr)) + WT_RET(bm->preload(bm, session, addr.addr, addr.size)); WT_INTL_FOREACH_END; return (0); } diff --git a/src/third_party/wiredtiger/src/btree/bt_misc.c b/src/third_party/wiredtiger/src/btree/bt_misc.c index d5ce78e23aa..edf6e4bad05 100644 --- a/src/third_party/wiredtiger/src/btree/bt_misc.c +++ b/src/third_party/wiredtiger/src/btree/bt_misc.c @@ -9,33 +9,26 @@ #include "wt_internal.h" /* - * __wt_page_type_string -- - * Return a string representing the page type. + * __wt_addr_string -- + * Load a buffer with a printable, nul-terminated representation of an address. */ const char * -__wt_page_type_string(u_int type) WT_GCC_FUNC_ATTRIBUTE((visibility("default"))) +__wt_addr_string(WT_SESSION_IMPL *session, const uint8_t *addr, size_t addr_size, WT_ITEM *buf) { - switch (type) { - case WT_PAGE_INVALID: - return ("invalid"); - case WT_PAGE_BLOCK_MANAGER: - return ("block manager"); - case WT_PAGE_COL_FIX: - return ("column-store fixed-length leaf"); - case WT_PAGE_COL_INT: - return ("column-store internal"); - case WT_PAGE_COL_VAR: - return ("column-store variable-length leaf"); - case WT_PAGE_OVFL: - return ("overflow"); - case WT_PAGE_ROW_INT: - return ("row-store internal"); - case WT_PAGE_ROW_LEAF: - return ("row-store leaf"); - default: - return ("unknown"); + WT_BM *bm; + WT_BTREE *btree; + + btree = S2BT_SAFE(session); + + if (addr == NULL || addr_size == 0) { + buf->data = WT_NO_ADDR_STRING; + buf->size = strlen(WT_NO_ADDR_STRING); + } else if (btree == NULL || (bm = btree->bm) == NULL || + bm->addr_string(bm, session, buf, addr, addr_size) != 0) { + buf->data = WT_ERR_STRING; + buf->size = strlen(WT_ERR_STRING); } - /* NOTREACHED */ + return (buf->data); } /* @@ -85,24 +78,56 @@ __wt_cell_type_string(uint8_t type) } /* - * __wt_addr_string -- - * Load a buffer with a printable, nul-terminated representation of an address. + * __wt_key_string -- + * Load a buffer with a printable, nul-terminated representation of a key. */ const char * -__wt_addr_string(WT_SESSION_IMPL *session, const uint8_t *addr, size_t addr_size, WT_ITEM *buf) +__wt_key_string( + WT_SESSION_IMPL *session, const void *data_arg, size_t size, const char *key_format, WT_ITEM *buf) { - WT_BM *bm; - WT_BTREE *btree; - - btree = S2BT_SAFE(session); + WT_ITEM tmp; + /* + * If the format is 'S', it's a string and our version of it may not yet be nul-terminated. + */ + if (WT_STREQ(key_format, "S") && ((char *)data_arg)[size - 1] != '\0') { + WT_CLEAR(tmp); + if (__wt_buf_fmt(session, &tmp, "%.*s", (int)size, (char *)data_arg) == 0) { + data_arg = tmp.data; + size = tmp.size + 1; + } else { + data_arg = WT_ERR_STRING; + size = sizeof(WT_ERR_STRING); + } + } + return __wt_buf_set_printable_format(session, data_arg, size, key_format, buf); +} - if (addr == NULL) { - buf->data = "[NoAddr]"; - buf->size = strlen("[NoAddr]"); - } else if (btree == NULL || (bm = btree->bm) == NULL || - bm->addr_string(bm, session, buf, addr, addr_size) != 0) { - buf->data = "[Error]"; - buf->size = strlen("[Error]"); +/* + * __wt_page_type_string -- + * Return a string representing the page type. + */ +const char * +__wt_page_type_string(u_int type) WT_GCC_FUNC_ATTRIBUTE((visibility("default"))) +{ + switch (type) { + case WT_PAGE_INVALID: + return ("invalid"); + case WT_PAGE_BLOCK_MANAGER: + return ("block manager"); + case WT_PAGE_COL_FIX: + return ("column-store fixed-length leaf"); + case WT_PAGE_COL_INT: + return ("column-store internal"); + case WT_PAGE_COL_VAR: + return ("column-store variable-length leaf"); + case WT_PAGE_OVFL: + return ("overflow"); + case WT_PAGE_ROW_INT: + return ("row-store internal"); + case WT_PAGE_ROW_LEAF: + return ("row-store leaf"); + default: + return ("unknown"); } - return (buf->data); + /* NOTREACHED */ } diff --git a/src/third_party/wiredtiger/src/btree/bt_page.c b/src/third_party/wiredtiger/src/btree/bt_page.c index 5cdf6993777..ca776d91918 100644 --- a/src/third_party/wiredtiger/src/btree/bt_page.c +++ b/src/third_party/wiredtiger/src/btree/bt_page.c @@ -10,9 +10,9 @@ static void __inmem_col_fix(WT_SESSION_IMPL *, WT_PAGE *); static void __inmem_col_int(WT_SESSION_IMPL *, WT_PAGE *); -static int __inmem_col_var(WT_SESSION_IMPL *, WT_PAGE *, uint64_t, size_t *, bool); +static int __inmem_col_var(WT_SESSION_IMPL *, WT_PAGE *, uint64_t, size_t *); static int __inmem_row_int(WT_SESSION_IMPL *, WT_PAGE *, size_t *); -static int __inmem_row_leaf(WT_SESSION_IMPL *, WT_PAGE *, bool); +static int __inmem_row_leaf(WT_SESSION_IMPL *, WT_PAGE *); static int __inmem_row_leaf_entries(WT_SESSION_IMPL *, const WT_PAGE_HEADER *, uint32_t *); /* @@ -125,8 +125,8 @@ err: * Build in-memory page information. */ int -__wt_page_inmem(WT_SESSION_IMPL *session, WT_REF *ref, const void *image, uint32_t flags, - bool check_unstable, WT_PAGE **pagep) +__wt_page_inmem( + WT_SESSION_IMPL *session, WT_REF *ref, const void *image, uint32_t flags, WT_PAGE **pagep) { WT_DECL_RET; WT_PAGE *page; @@ -206,13 +206,13 @@ __wt_page_inmem(WT_SESSION_IMPL *session, WT_REF *ref, const void *image, uint32 __inmem_col_int(session, page); break; case WT_PAGE_COL_VAR: - WT_ERR(__inmem_col_var(session, page, dsk->recno, &size, check_unstable)); + WT_ERR(__inmem_col_var(session, page, dsk->recno, &size)); break; case WT_PAGE_ROW_INT: WT_ERR(__inmem_row_int(session, page, &size)); break; case WT_PAGE_ROW_LEAF: - WT_ERR(__inmem_row_leaf(session, page, check_unstable)); + WT_ERR(__inmem_row_leaf(session, page)); break; default: WT_ERR(__wt_illegal_value(session, page->type)); @@ -286,6 +286,8 @@ __inmem_col_int(WT_SESSION_IMPL *session, WT_PAGE *page) ref->pindex_hint = hint++; ref->addr = unpack.cell; ref->ref_recno = unpack.v; + + F_SET(ref, unpack.type == WT_CELL_ADDR_INT ? WT_REF_FLAG_INTERNAL : WT_REF_FLAG_LEAF); } WT_CELL_FOREACH_END; } @@ -313,27 +315,11 @@ __inmem_col_var_repeats(WT_SESSION_IMPL *session, WT_PAGE *page, uint32_t *np) } /* - * __unstable_skip -- - * Optionally skip unstable entries - */ -static inline bool -__unstable_skip(WT_SESSION_IMPL *session, const WT_PAGE_HEADER *dsk, WT_CELL_UNPACK *unpack) -{ - /* - * Skip unstable entries after downgrade to releases without validity windows and from previous - * wiredtiger_open connections. - */ - return ((unpack->stop_ts != WT_TS_MAX || unpack->stop_txn != WT_TXN_MAX) && - (S2C(session)->base_write_gen > dsk->write_gen || !__wt_process.page_version_ts)); -} - -/* * __inmem_col_var -- * Build in-memory index for variable-length, data-only leaf pages in column-store trees. */ static int -__inmem_col_var( - WT_SESSION_IMPL *session, WT_PAGE *page, uint64_t recno, size_t *sizep, bool check_unstable) +__inmem_col_var(WT_SESSION_IMPL *session, WT_PAGE *page, uint64_t recno, size_t *sizep) { WT_BTREE *btree; WT_CELL_UNPACK unpack; @@ -357,12 +343,6 @@ __inmem_col_var( indx = 0; cip = page->pg_var; WT_CELL_FOREACH_BEGIN (session, btree, page->dsk, unpack) { - /* Optionally skip unstable values */ - if (check_unstable && __unstable_skip(session, page->dsk, &unpack)) { - --page->entries; - continue; - } - WT_COL_PTR_SET(cip, WT_PAGE_DISK_OFFSET(page, unpack.cell)); cip++; @@ -429,6 +409,17 @@ __inmem_row_int(WT_SESSION_IMPL *session, WT_PAGE *page, size_t *sizep) ref->pindex_hint = hint++; switch (unpack.type) { + case WT_CELL_ADDR_INT: + F_SET(ref, WT_REF_FLAG_INTERNAL); + break; + case WT_CELL_ADDR_DEL: + case WT_CELL_ADDR_LEAF: + case WT_CELL_ADDR_LEAF_NO: + F_SET(ref, WT_REF_FLAG_LEAF); + break; + } + + switch (unpack.type) { case WT_CELL_KEY: /* * Note: we don't Huffman encode internal page keys, there's no decoding work to do. @@ -549,13 +540,15 @@ __inmem_row_leaf_entries(WT_SESSION_IMPL *session, const WT_PAGE_HEADER *dsk, ui * Build in-memory index for row-store leaf pages. */ static int -__inmem_row_leaf(WT_SESSION_IMPL *session, WT_PAGE *page, bool check_unstable) +__inmem_row_leaf(WT_SESSION_IMPL *session, WT_PAGE *page) { WT_BTREE *btree; WT_CELL_UNPACK unpack; WT_ROW *rip; + WT_TXN_GLOBAL *txn_global; btree = S2BT(session); + txn_global = &S2C(session)->txn_global; /* Walk the page, building indices. */ rip = page->pg_row; @@ -577,25 +570,19 @@ __inmem_row_leaf(WT_SESSION_IMPL *session, WT_PAGE *page, bool check_unstable) ++rip; break; case WT_CELL_VALUE: - /* Optionally skip unstable values */ - if (check_unstable && __unstable_skip(session, page->dsk, &unpack)) { - --rip; - --page->entries; - } - /* * Simple values without compression can be directly referenced on the page to avoid * repeatedly unpacking their cells. + * + * The visibility information is not referenced on the page so we need to ensure that + * the value is globally visible at the point in time where we read the page into cache. */ - if (!btree->huffman_value) + if (!btree->huffman_value && unpack.stop_txn == WT_TXN_MAX && + unpack.stop_ts == WT_TS_MAX && txn_global->has_oldest_timestamp && + unpack.start_ts <= txn_global->oldest_timestamp) __wt_row_leaf_value_set(page, rip - 1, &unpack); break; case WT_CELL_VALUE_OVFL: - /* Optionally skip unstable values */ - if (check_unstable && __unstable_skip(session, page->dsk, &unpack)) { - --rip; - --page->entries; - } break; default: return (__wt_illegal_value(session, unpack.type)); diff --git a/src/third_party/wiredtiger/src/btree/bt_random.c b/src/third_party/wiredtiger/src/btree/bt_random.c index 78f073a653e..11b090faa55 100644 --- a/src/third_party/wiredtiger/src/btree/bt_random.c +++ b/src/third_party/wiredtiger/src/btree/bt_random.c @@ -395,10 +395,10 @@ restart: /* Search the internal pages of the tree. */ current = &btree->root; for (;;) { - page = current->page; - if (!WT_PAGE_IS_INTERNAL(page)) + if (F_ISSET(current, WT_REF_FLAG_LEAF)) break; + page = current->page; WT_INTL_INDEX_GET(session, page, pindex); entries = pindex->entries; @@ -420,15 +420,13 @@ restart: descent = NULL; for (i = 0; i < entries; ++i) { descent = pindex->index[__wt_random(&session->rnd) % entries]; - if (descent->state == WT_REF_DISK || descent->state == WT_REF_LIMBO || - descent->state == WT_REF_LOOKASIDE || descent->state == WT_REF_MEM) + if (descent->state == WT_REF_DISK || descent->state == WT_REF_MEM) break; } if (i == entries) for (i = 0; i < entries; ++i) { descent = pindex->index[i]; - if (descent->state == WT_REF_DISK || descent->state == WT_REF_LIMBO || - descent->state == WT_REF_LOOKASIDE || descent->state == WT_REF_MEM) + if (descent->state == WT_REF_DISK || descent->state == WT_REF_MEM) break; } if (i == entries || descent == NULL) { diff --git a/src/third_party/wiredtiger/src/btree/bt_read.c b/src/third_party/wiredtiger/src/btree/bt_read.c index 2af497c066e..554c4e047d6 100644 --- a/src/third_party/wiredtiger/src/btree/bt_read.c +++ b/src/third_party/wiredtiger/src/btree/bt_read.c @@ -9,298 +9,6 @@ #include "wt_internal.h" /* - * __col_instantiate -- - * Update a column-store page entry based on a lookaside table update list. - */ -static int -__col_instantiate( - WT_SESSION_IMPL *session, uint64_t recno, WT_REF *ref, WT_CURSOR_BTREE *cbt, WT_UPDATE *updlist) -{ - WT_PAGE *page; - WT_UPDATE *upd; - - page = ref->page; - - /* - * Discard any of the updates we don't need. - * - * Just free the memory: it hasn't been accounted for on the page yet. - */ - if (updlist->next != NULL && - (upd = __wt_update_obsolete_check(session, page, updlist, false)) != NULL) - __wt_free_update_list(session, upd); - - /* Search the page and add updates. */ - WT_RET(__wt_col_search(cbt, recno, ref, true, NULL)); - WT_RET(__wt_col_modify(cbt, recno, NULL, updlist, WT_UPDATE_INVALID, false)); - return (0); -} - -/* - * __row_instantiate -- - * Update a row-store page entry based on a lookaside table update list. - */ -static int -__row_instantiate( - WT_SESSION_IMPL *session, WT_ITEM *key, WT_REF *ref, WT_CURSOR_BTREE *cbt, WT_UPDATE *updlist) -{ - WT_PAGE *page; - WT_UPDATE *upd; - - page = ref->page; - - /* - * Discard any of the updates we don't need. - * - * Just free the memory: it hasn't been accounted for on the page yet. - */ - if (updlist->next != NULL && - (upd = __wt_update_obsolete_check(session, page, updlist, false)) != NULL) - __wt_free_update_list(session, upd); - - /* Search the page and add updates. */ - WT_RET(__wt_row_search(cbt, key, true, ref, true, NULL)); - WT_RET(__wt_row_modify(cbt, key, NULL, updlist, WT_UPDATE_INVALID, false)); - return (0); -} - -/* - * __las_page_instantiate_verbose -- - * Create a verbose message to display at most once per checkpoint when performing a lookaside - * table read. - */ -static void -__las_page_instantiate_verbose(WT_SESSION_IMPL *session, uint64_t las_pageid) -{ - WT_CACHE *cache; - uint64_t ckpt_gen_current, ckpt_gen_last; - - if (!WT_VERBOSE_ISSET(session, WT_VERB_LOOKASIDE | WT_VERB_LOOKASIDE_ACTIVITY)) - return; - - cache = S2C(session)->cache; - ckpt_gen_current = __wt_gen(session, WT_GEN_CHECKPOINT); - ckpt_gen_last = cache->las_verb_gen_read; - - /* - * This message is throttled to one per checkpoint. To do this we track the generation of the - * last checkpoint for which the message was printed and check against the current checkpoint - * generation. - */ - if (WT_VERBOSE_ISSET(session, WT_VERB_LOOKASIDE) || ckpt_gen_current > ckpt_gen_last) { - /* - * Attempt to atomically replace the last checkpoint generation for which this message was - * printed. If the atomic swap fails we have raced and the winning thread will print the - * message. - */ - if (__wt_atomic_casv64(&cache->las_verb_gen_read, ckpt_gen_last, ckpt_gen_current)) { - __wt_verbose(session, WT_VERB_LOOKASIDE | WT_VERB_LOOKASIDE_ACTIVITY, - "Read from lookaside file triggered for " - "file ID %" PRIu32 ", page ID %" PRIu64, - S2BT(session)->id, las_pageid); - } - } -} - -/* - * __las_page_instantiate -- - * Instantiate lookaside update records in a recently read page. - */ -static int -__las_page_instantiate(WT_SESSION_IMPL *session, WT_REF *ref) -{ - WT_CACHE *cache; - WT_CURSOR *cursor; - WT_CURSOR_BTREE cbt; - WT_DECL_ITEM(current_key); - WT_DECL_RET; - WT_ITEM las_key, las_value; - WT_PAGE *page; - WT_PAGE_LOOKASIDE *page_las; - WT_UPDATE *first_upd, *last_upd, *upd; - wt_timestamp_t durable_timestamp, las_timestamp; - size_t incr, total_incr; - uint64_t current_recno, las_counter, las_pageid, las_txnid, recno; - uint32_t las_id, session_flags; - uint8_t prepare_state, upd_type; - const uint8_t *p; - bool locked; - - cursor = NULL; - page = ref->page; - first_upd = last_upd = upd = NULL; - locked = false; - total_incr = 0; - current_recno = recno = WT_RECNO_OOB; - page_las = ref->page_las; - las_pageid = page_las->las_pageid; - session_flags = 0; /* [-Werror=maybe-uninitialized] */ - WT_CLEAR(las_key); - - cache = S2C(session)->cache; - __las_page_instantiate_verbose(session, las_pageid); - WT_STAT_CONN_INCR(session, cache_read_lookaside); - WT_STAT_DATA_INCR(session, cache_read_lookaside); - if (WT_SESSION_IS_CHECKPOINT(session)) - WT_STAT_CONN_INCR(session, cache_read_lookaside_checkpoint); - - __wt_btcur_init(session, &cbt); - __wt_btcur_open(&cbt); - - WT_ERR(__wt_scr_alloc(session, 0, ¤t_key)); - - /* Open a lookaside table cursor. */ - __wt_las_cursor(session, &cursor, &session_flags); - - /* - * The lookaside records are in key and update order, that is, there will be a set of in-order - * updates for a key, then another set of in-order updates for a subsequent key. We process all - * of the updates for a key and then insert those updates into the page, then all the updates - * for the next key, and so on. - */ - WT_PUBLISH(cache->las_reader, true); - __wt_readlock(session, &cache->las_sweepwalk_lock); - WT_PUBLISH(cache->las_reader, false); - locked = true; - for (ret = __wt_las_cursor_position(cursor, las_pageid); ret == 0; ret = cursor->next(cursor)) { - WT_ERR(cursor->get_key(cursor, &las_pageid, &las_id, &las_counter, &las_key)); - - /* - * Confirm the search using the unique prefix; if not a match, we're done searching for - * records for this page. - */ - if (las_pageid != page_las->las_pageid) - break; - - /* Allocate the WT_UPDATE structure. */ - WT_ERR(cursor->get_value(cursor, &las_txnid, &las_timestamp, &durable_timestamp, - &prepare_state, &upd_type, &las_value)); - WT_ERR(__wt_update_alloc(session, &las_value, &upd, &incr, upd_type)); - total_incr += incr; - upd->txnid = las_txnid; - upd->durable_ts = durable_timestamp; - upd->start_ts = las_timestamp; - upd->prepare_state = prepare_state; - - switch (page->type) { - case WT_PAGE_COL_FIX: - case WT_PAGE_COL_VAR: - p = las_key.data; - WT_ERR(__wt_vunpack_uint(&p, 0, &recno)); - if (current_recno == recno) - break; - WT_ASSERT(session, current_recno < recno); - - if (first_upd != NULL) { - WT_ERR(__col_instantiate(session, current_recno, ref, &cbt, first_upd)); - first_upd = NULL; - } - current_recno = recno; - break; - case WT_PAGE_ROW_LEAF: - if (current_key->size == las_key.size && - memcmp(current_key->data, las_key.data, las_key.size) == 0) - break; - - if (first_upd != NULL) { - WT_ERR(__row_instantiate(session, current_key, ref, &cbt, first_upd)); - first_upd = NULL; - } - WT_ERR(__wt_buf_set(session, current_key, las_key.data, las_key.size)); - break; - default: - WT_ERR(__wt_illegal_value(session, page->type)); - } - - /* Append the latest update to the list. */ - if (first_upd == NULL) - first_upd = last_upd = upd; - else { - last_upd->next = upd; - last_upd = upd; - } - upd = NULL; - } - __wt_readunlock(session, &cache->las_sweepwalk_lock); - locked = false; - WT_ERR_NOTFOUND_OK(ret); - - /* Insert the last set of updates, if any. */ - if (first_upd != NULL) { - WT_ASSERT(session, __wt_count_birthmarks(first_upd) <= 1); - switch (page->type) { - case WT_PAGE_COL_FIX: - case WT_PAGE_COL_VAR: - WT_ERR(__col_instantiate(session, current_recno, ref, &cbt, first_upd)); - first_upd = NULL; - break; - case WT_PAGE_ROW_LEAF: - WT_ERR(__row_instantiate(session, current_key, ref, &cbt, first_upd)); - first_upd = NULL; - break; - default: - WT_ERR(__wt_illegal_value(session, page->type)); - } - } - - /* Discard the cursor. */ - WT_ERR(__wt_las_cursor_close(session, &cursor, session_flags)); - - if (total_incr != 0) { - __wt_cache_page_inmem_incr(session, page, total_incr); - - /* - * If the updates in lookaside are newer than the versions on - * the page, it must be included in the next checkpoint. - * - * Otherwise, the page image contained the newest versions of - * data so the updates are all older and we could consider - * marking it clean (i.e., the next checkpoint can use the - * version already on disk). - * - * This needs care because (a) it creates pages with history - * that can't be evicted until they are marked dirty again, and - * (b) checkpoints may need to visit these pages to resolve - * changes evicted while a checkpoint is running. - */ - page->modify->first_dirty_txn = WT_TXN_FIRST; - - FLD_SET(page->modify->restore_state, WT_PAGE_RS_LOOKASIDE); - - if (page_las->min_skipped_ts == WT_TS_MAX && !page_las->has_prepares && - !S2C(session)->txn_global.has_stable_timestamp && - __wt_txn_visible_all(session, page_las->max_txn, page_las->max_ondisk_ts)) { - page->modify->rec_max_txn = page_las->max_txn; - page->modify->rec_max_timestamp = page_las->max_ondisk_ts; - __wt_page_modify_clear(session, page); - } - } - - /* - * Now the lookaside history has been read into cache there is no further need to maintain a - * reference to it. - */ - page_las->eviction_to_lookaside = false; - page_las->resolved = true; - -err: - if (locked) - __wt_readunlock(session, &cache->las_sweepwalk_lock); - WT_TRET(__wt_las_cursor_close(session, &cursor, session_flags)); - WT_TRET(__wt_btcur_close(&cbt, true)); - - /* - * On error, upd points to a single unlinked WT_UPDATE structure, first_upd points to a list. - */ - __wt_free(session, upd); - __wt_free_update_list(session, first_upd); - - __wt_scr_free(session, ¤t_key); - - return (ret); -} - -/* * __evict_force_check -- * Check if a page matches the criteria for forced eviction. */ @@ -315,7 +23,7 @@ __evict_force_check(WT_SESSION_IMPL *session, WT_REF *ref) page = ref->page; /* Leaf pages only. */ - if (WT_PAGE_IS_INTERNAL(page)) + if (F_ISSET(ref, WT_REF_FLAG_INTERNAL)) return (false); /* @@ -373,52 +81,19 @@ __evict_force_check(WT_SESSION_IMPL *session, WT_REF *ref) } /* - * __page_read_lookaside -- - * Figure out whether to instantiate content from lookaside on page access. - */ -static inline int -__page_read_lookaside( - WT_SESSION_IMPL *session, WT_REF *ref, uint32_t previous_state, uint32_t *final_statep) -{ - /* - * Reading a lookaside ref for the first time, and not requiring the history triggers a - * transition to WT_REF_LIMBO, if we are already in limbo and still don't need the history - we - * are done. - */ - if (__wt_las_page_skip_locked(session, ref)) { - if (previous_state == WT_REF_LOOKASIDE) { - WT_STAT_CONN_INCR(session, cache_read_lookaside_skipped); - ref->page_las->eviction_to_lookaside = true; - } - *final_statep = WT_REF_LIMBO; - return (0); - } - - /* Instantiate updates from the database's lookaside table. */ - if (previous_state == WT_REF_LIMBO) { - WT_STAT_CONN_INCR(session, cache_read_lookaside_delay); - if (WT_SESSION_IS_CHECKPOINT(session)) - WT_STAT_CONN_INCR(session, cache_read_lookaside_delay_checkpoint); - } - - WT_RET(__las_page_instantiate(session, ref)); - return (0); -} - -/* * __page_read -- * Read a page from the file. */ static int __page_read(WT_SESSION_IMPL *session, WT_REF *ref, uint32_t flags) { + WT_ADDR_COPY addr; WT_DECL_RET; WT_ITEM tmp; WT_PAGE *notused; - size_t addr_size; uint64_t time_diff, time_start, time_stop; - uint32_t page_flags, final_state, new_state, previous_state; - const uint8_t *addr; + uint32_t page_flags; + uint8_t previous_state; bool timer; time_start = time_stop = 0; @@ -429,55 +104,41 @@ __page_read(WT_SESSION_IMPL *session, WT_REF *ref, uint32_t flags) */ WT_CLEAR(tmp); - /* - * Attempt to set the state to WT_REF_READING for normal reads, or WT_REF_LOCKED, for deleted - * pages or pages with lookaside entries. The difference is that checkpoints can skip over clean - * pages that are being read into cache, but need to wait for deletes or lookaside updates to be - * resolved (in order for checkpoint to write the correct version of the page). - * - * If successful, we've won the race, read the page. - */ + /* Lock the WT_REF. */ switch (previous_state = ref->state) { case WT_REF_DISK: - new_state = WT_REF_READING; - break; case WT_REF_DELETED: - case WT_REF_LIMBO: - case WT_REF_LOOKASIDE: - new_state = WT_REF_LOCKED; - break; + if (WT_REF_CAS_STATE(session, ref, previous_state, WT_REF_LOCKED)) + break; + return (0); default: return (0); } - if (!WT_REF_CAS_STATE(session, ref, previous_state, new_state)) - return (0); - - final_state = WT_REF_MEM; - /* If we already have the page image, just instantiate the history. */ - if (previous_state == WT_REF_LIMBO) - goto skip_read; + /* + * Set the WT_REF_FLAG_READING flag for normal reads. Checkpoints can skip over clean pages + * being read into cache, but need to wait for deletes to be resolved (in order for checkpoint + * to write the correct version of the page). + */ + if (previous_state == WT_REF_DISK) + F_SET(ref, WT_REF_FLAG_READING); /* - * Get the address: if there is no address, the page was deleted or had only lookaside entries, - * and a subsequent search or insert is forcing re-creation of the name space. + * Get the address: if there is no address, the page was deleted and a subsequent search or + * insert is forcing re-creation of the name space. */ - __wt_ref_info(session, ref, &addr, &addr_size, NULL); - if (addr == NULL) { - WT_ASSERT(session, previous_state != WT_REF_DISK); + if (!__wt_ref_addr_copy(session, ref, &addr)) { + WT_ASSERT(session, previous_state == WT_REF_DELETED); - WT_ERR(__wt_btree_new_leaf_page(session, &ref->page)); + WT_ERR(__wt_btree_new_leaf_page(session, ref)); goto skip_read; } - /* - * There's an address, read or map the backing disk page and build an in-memory version of the - * page. - */ + /* There's an address, read the backing disk page and build an in-memory version of the page. */ timer = !F_ISSET(session, WT_SESSION_INTERNAL); if (timer) time_start = __wt_clock(session); - WT_ERR(__wt_bt_read(session, &tmp, addr, addr_size)); + WT_ERR(__wt_bt_read(session, &tmp, addr.addr, addr.size)); if (timer) { time_stop = __wt_clock(session); time_diff = WT_CLOCKDIFF_US(time_stop, time_start); @@ -498,52 +159,19 @@ __page_read(WT_SESSION_IMPL *session, WT_REF *ref, uint32_t flags) page_flags = WT_DATA_IN_ITEM(&tmp) ? WT_PAGE_DISK_ALLOC : WT_PAGE_DISK_MAPPED; if (LF_ISSET(WT_READ_IGNORE_CACHE_SIZE)) FLD_SET(page_flags, WT_PAGE_EVICT_NO_PROGRESS); - WT_ERR(__wt_page_inmem(session, ref, tmp.data, page_flags, true, ¬used)); + WT_ERR(__wt_page_inmem(session, ref, tmp.data, page_flags, ¬used)); tmp.mem = NULL; - /* - * The WT_REF lookaside state should match the page-header state of any page we read. - */ - WT_ASSERT(session, (previous_state != WT_REF_LIMBO && previous_state != WT_REF_LOOKASIDE) || - ref->page->dsk == NULL || F_ISSET(ref->page->dsk, WT_PAGE_LAS_UPDATE)); - skip_read: switch (previous_state) { case WT_REF_DELETED: - /* - * A truncated page may also have lookaside information. The delete happened after page - * eviction (writing the lookaside information), first update based on the lookaside table - * and then apply the delete. - */ - if (ref->page_las != NULL) - WT_ERR(__las_page_instantiate(session, ref)); - /* Move all records to a deleted state. */ WT_ERR(__wt_delete_page_instantiate(session, ref)); break; - case WT_REF_LIMBO: - case WT_REF_LOOKASIDE: - WT_ERR(__page_read_lookaside(session, ref, previous_state, &final_state)); - break; } - /* - * Once the page is instantiated, we no longer need the history in - * lookaside. We leave the lookaside sweep thread to do most cleanup, - * but it can only remove committed updates and keys that skew newest - * (if there are entries in the lookaside newer than the page, they need - * to be read back into cache or they will be lost). - * - * Prepared updates can not be removed by the lookaside sweep, remove - * them as we read the page back in memory. - * - * Don't free WT_REF.page_las, there may be concurrent readers. - */ - if (final_state == WT_REF_MEM && ref->page_las != NULL && - (ref->page_las->min_skipped_ts != WT_TS_MAX || ref->page_las->has_prepares)) - WT_ERR(__wt_las_remove_block(session, ref->page_las->las_pageid)); - - WT_REF_SET_STATE(ref, final_state); + F_CLR(ref, WT_REF_FLAG_READING); + WT_REF_SET_STATE(ref, WT_REF_MEM); WT_ASSERT(session, ret == 0); return (0); @@ -553,8 +181,10 @@ err: * If the function building an in-memory version of the page failed, it discarded the page, but * not the disk image. Discard the page and separately discard the disk image in all cases. */ - if (ref->page != NULL && previous_state != WT_REF_LIMBO) + if (ref->page != NULL) __wt_ref_out(session, ref); + + F_CLR(ref, WT_REF_FLAG_READING); WT_REF_SET_STATE(ref, previous_state); __wt_buf_free(session, &tmp); @@ -579,7 +209,7 @@ __wt_page_in_func(WT_SESSION_IMPL *session, WT_REF *ref, uint32_t flags WT_DECL_RET; WT_PAGE *page; uint64_t sleep_usecs, yield_cnt; - uint32_t current_state; + uint8_t current_state; int force_attempts; bool busy, cache_work, evict_skip, stalled, wont_need; @@ -589,15 +219,15 @@ __wt_page_in_func(WT_SESSION_IMPL *session, WT_REF *ref, uint32_t flags LF_SET(WT_READ_IGNORE_CACHE_SIZE); /* Sanity check flag combinations. */ - WT_ASSERT(session, !LF_ISSET(WT_READ_DELETED_SKIP | WT_READ_NO_WAIT | WT_READ_LOOKASIDE) || - LF_ISSET(WT_READ_CACHE)); + WT_ASSERT(session, !LF_ISSET(WT_READ_DELETED_SKIP | WT_READ_NO_WAIT) || + LF_ISSET(WT_READ_CACHE | WT_READ_CACHE_LEAF)); WT_ASSERT(session, !LF_ISSET(WT_READ_DELETED_CHECK) || !LF_ISSET(WT_READ_DELETED_SKIP)); /* * Ignore reads of pages already known to be in cache, otherwise the eviction server can * dominate these statistics. */ - if (!LF_ISSET(WT_READ_CACHE)) { + if (!LF_ISSET(WT_READ_CACHE | WT_READ_CACHE_LEAF)) { WT_STAT_CONN_INCR(session, cache_pages_requested); WT_STAT_DATA_INCR(session, cache_pages_requested); } @@ -611,24 +241,15 @@ __wt_page_in_func(WT_SESSION_IMPL *session, WT_REF *ref, uint32_t flags if (LF_ISSET(WT_READ_DELETED_CHECK) && __wt_delete_page_skip(session, ref, false)) return (WT_NOTFOUND); goto read; - case WT_REF_LOOKASIDE: - if (LF_ISSET(WT_READ_CACHE)) { - if (!LF_ISSET(WT_READ_LOOKASIDE)) - return (WT_NOTFOUND); - /* - * If we skip a lookaside page, the tree cannot be left clean: lookaside entries - * must be resolved before the tree can be discarded. - */ - if (__wt_las_page_skip(session, ref)) { - __wt_tree_modify_set(session); - return (WT_NOTFOUND); - } - } - goto read; case WT_REF_DISK: + /* Optionally limit reads to cache-only. */ if (LF_ISSET(WT_READ_CACHE)) return (WT_NOTFOUND); + /* Optionally limit reads to internal pages only. */ + if (LF_ISSET(WT_READ_CACHE_LEAF) && F_ISSET(ref, WT_REF_FLAG_LEAF)) + return (WT_NOTFOUND); + read: /* * The page isn't in memory, read it. If this thread respects the cache size, check for @@ -652,27 +273,24 @@ read: F_ISSET(session, WT_SESSION_READ_WONT_NEED) || F_ISSET(S2C(session)->cache, WT_CACHE_EVICT_NOKEEP); continue; - case WT_REF_READING: - if (LF_ISSET(WT_READ_CACHE)) - return (WT_NOTFOUND); - if (LF_ISSET(WT_READ_NO_WAIT)) - return (WT_NOTFOUND); - - /* Waiting on another thread's read, stall. */ - WT_STAT_CONN_INCR(session, page_read_blocked); - stalled = true; - break; case WT_REF_LOCKED: if (LF_ISSET(WT_READ_NO_WAIT)) return (WT_NOTFOUND); - /* Waiting on eviction, stall. */ - WT_STAT_CONN_INCR(session, page_locked_blocked); + if (F_ISSET(ref, WT_REF_FLAG_READING)) { + if (LF_ISSET(WT_READ_CACHE)) + return (WT_NOTFOUND); + + /* Waiting on another thread's read, stall. */ + WT_STAT_CONN_INCR(session, page_read_blocked); + } else + /* Waiting on eviction, stall. */ + WT_STAT_CONN_INCR(session, page_locked_blocked); + stalled = true; break; case WT_REF_SPLIT: return (WT_RESTART); - case WT_REF_LIMBO: case WT_REF_MEM: /* * The page is in memory. @@ -688,27 +306,14 @@ read: * try again. */ #ifdef HAVE_DIAGNOSTIC - WT_RET(__wt_hazard_set(session, ref, &busy, func, line)); + WT_RET(__wt_hazard_set_func(session, ref, &busy, func, line)); #else - WT_RET(__wt_hazard_set(session, ref, &busy)); + WT_RET(__wt_hazard_set_func(session, ref, &busy)); #endif if (busy) { WT_STAT_CONN_INCR(session, page_busy_blocked); break; } - /* - * If we are a limbo page check whether we need to instantiate the history. By having a - * hazard pointer we can use the locked version. - */ - if (current_state == WT_REF_LIMBO && - ((!LF_ISSET(WT_READ_CACHE) || LF_ISSET(WT_READ_LOOKASIDE)) && - !__wt_las_page_skip_locked(session, ref))) { - WT_RET(__wt_hazard_clear(session, ref)); - goto read; - } - if (current_state == WT_REF_LIMBO && LF_ISSET(WT_READ_CACHE) && - LF_ISSET(WT_READ_LOOKASIDE)) - __wt_tree_modify_set(session); /* * If a page has grown too large, we'll try and forcibly evict it before making it @@ -721,8 +326,8 @@ read: goto skip_evict; /* - * If reconciliation is disabled (e.g., when inserting into the lookaside table), skip - * forced eviction if the page can't split. + * If reconciliation is disabled (e.g., when inserting into the history store table), + * skip forced eviction if the page can't split. */ if (F_ISSET(session, WT_SESSION_NO_RECONCILE) && !__wt_leaf_page_can_split(session, ref->page)) @@ -779,9 +384,9 @@ skip_evict: * * The logic here is a little weird: some code paths do a blanket ban on checking the * cache size in sessions, but still require a transaction (e.g., when updating metadata - * or lookaside). If WT_READ_IGNORE_CACHE_SIZE was passed in explicitly, we're done. If - * we set WT_READ_IGNORE_CACHE_SIZE because it was set in the session then make sure we - * start a transaction. + * or the history store). If WT_READ_IGNORE_CACHE_SIZE was passed in explicitly, we're + * done. If we set WT_READ_IGNORE_CACHE_SIZE because it was set in the session then make + * sure we start a transaction. */ return (LF_ISSET(WT_READ_IGNORE_CACHE_SIZE) && !F_ISSET(session, WT_SESSION_IGNORE_CACHE_SIZE) ? diff --git a/src/third_party/wiredtiger/src/btree/bt_rebalance.c b/src/third_party/wiredtiger/src/btree/bt_rebalance.c index 4a66db142f3..eae6857dca3 100644 --- a/src/third_party/wiredtiger/src/btree/bt_rebalance.c +++ b/src/third_party/wiredtiger/src/btree/bt_rebalance.c @@ -71,11 +71,13 @@ __rebalance_leaf_append(WT_SESSION_IMPL *session, wt_timestamp_t durable_ts, con WT_RET(__wt_calloc_one(session, ©)); rs->leaf[rs->leaf_next++] = copy; + F_SET(copy, WT_REF_FLAG_LEAF); copy->state = WT_REF_DISK; WT_RET(__wt_calloc_one(session, ©_addr)); copy->addr = copy_addr; - copy_addr->newest_durable_ts = durable_ts; + /* FIXME-prepare-support: use durable timestamps from unpack struct */ + copy_addr->stop_durable_ts = durable_ts; copy_addr->oldest_start_ts = unpack->oldest_start_ts; copy_addr->oldest_start_txn = unpack->oldest_start_txn; copy_addr->newest_stop_ts = unpack->newest_stop_ts; @@ -211,7 +213,7 @@ __rebalance_col_walk(WT_SESSION_IMPL *session, wt_timestamp_t durable_ts, const case WT_CELL_ADDR_INT: /* An internal page: read it and recursively walk it. */ WT_ERR(__wt_bt_read(session, buf, unpack.data, unpack.size)); - WT_ERR(__rebalance_col_walk(session, unpack.newest_durable_ts, buf->data, rs)); + WT_ERR(__rebalance_col_walk(session, unpack.newest_stop_durable_ts, buf->data, rs)); __wt_verbose(session, WT_VERB_REBALANCE, "free-list append internal page: %s", __wt_addr_string(session, unpack.data, unpack.size, rs->tmp1)); WT_ERR(__rebalance_fl_append(session, unpack.data, unpack.size, rs)); @@ -251,7 +253,7 @@ __rebalance_row_leaf_key(WT_SESSION_IMPL *session, const uint8_t *addr, size_t a * we don't want page discard to free it. */ WT_RET(__wt_bt_read(session, rs->tmp1, addr, addr_len)); - WT_RET(__wt_page_inmem(session, NULL, rs->tmp1->data, 0, false, &page)); + WT_RET(__wt_page_inmem(session, NULL, rs->tmp1->data, 0, &page)); ret = __wt_row_leaf_key_copy(session, page, &page->pg_row[0], key); __wt_page_out(session, &page); return (ret); @@ -325,7 +327,7 @@ __rebalance_row_walk(WT_SESSION_IMPL *session, wt_timestamp_t durable_ts, const /* Read and recursively walk the page. */ WT_ERR(__wt_bt_read(session, buf, unpack.data, unpack.size)); - WT_ERR(__rebalance_row_walk(session, unpack.newest_durable_ts, buf->data, rs)); + WT_ERR(__rebalance_row_walk(session, unpack.newest_stop_durable_ts, buf->data, rs)); break; case WT_CELL_ADDR_LEAF: case WT_CELL_ADDR_LEAF_NO: diff --git a/src/third_party/wiredtiger/src/btree/bt_ret.c b/src/third_party/wiredtiger/src/btree/bt_ret.c index 04393a228ac..2061d561a7a 100644 --- a/src/third_party/wiredtiger/src/btree/bt_ret.c +++ b/src/third_party/wiredtiger/src/btree/bt_ret.c @@ -70,11 +70,102 @@ __key_return(WT_CURSOR_BTREE *cbt) } /* - * __value_return -- - * Change the cursor to reference an internal original-page return value. + * __time_pairs_init -- + * Initialize the time pairs to globally visible. */ -static inline int -__value_return(WT_CURSOR_BTREE *cbt) +static inline void +__time_pairs_init(WT_TIME_PAIR *start, WT_TIME_PAIR *stop) +{ + start->txnid = WT_TXN_NONE; + start->timestamp = WT_TS_NONE; + stop->txnid = WT_TXN_MAX; + stop->timestamp = WT_TS_MAX; +} + +/* + * __time_pairs_set -- + * Set the time pairs. + */ +static inline void +__time_pairs_set(WT_TIME_PAIR *start, WT_TIME_PAIR *stop, WT_CELL_UNPACK *unpack) +{ + start->timestamp = unpack->start_ts; + start->txnid = unpack->start_txn; + stop->timestamp = unpack->stop_ts; + stop->txnid = unpack->stop_txn; +} + +/* + * __wt_read_cell_time_pairs -- + * Read the time pairs from the cell. + */ +void +__wt_read_cell_time_pairs( + WT_CURSOR_BTREE *cbt, WT_REF *ref, WT_TIME_PAIR *start, WT_TIME_PAIR *stop) +{ + WT_PAGE *page; + WT_SESSION_IMPL *session; + + session = (WT_SESSION_IMPL *)cbt->iface.session; + page = ref->page; + + WT_ASSERT(session, start != NULL && stop != NULL); + + /* Take the value from the original page cell. */ + if (page->type == WT_PAGE_ROW_LEAF) { + __wt_read_row_time_pairs(session, page, &page->pg_row[cbt->slot], start, stop); + } else if (page->type == WT_PAGE_COL_VAR) { + __wt_read_col_time_pairs( + session, page, WT_COL_PTR(page, &page->pg_var[cbt->slot]), start, stop); + } else { + /* WT_PAGE_COL_FIX: return the default time pairs. */ + __time_pairs_init(start, stop); + } +} + +/* + * __wt_read_col_time_pairs -- + * Retrieve the time pairs from a column store cell. + */ +void +__wt_read_col_time_pairs( + WT_SESSION_IMPL *session, WT_PAGE *page, WT_CELL *cell, WT_TIME_PAIR *start, WT_TIME_PAIR *stop) +{ + WT_CELL_UNPACK unpack; + + __wt_cell_unpack(session, page, cell, &unpack); + __time_pairs_set(start, stop, &unpack); +} + +/* + * __wt_read_row_time_pairs -- + * Retrieve the time pairs from a row. + */ +void +__wt_read_row_time_pairs( + WT_SESSION_IMPL *session, WT_PAGE *page, WT_ROW *rip, WT_TIME_PAIR *start, WT_TIME_PAIR *stop) +{ + WT_CELL_UNPACK unpack; + + __time_pairs_init(start, stop); + /* + * If a value is simple and is globally visible at the time of reading a page into cache, we set + * the time pairs as globally visible. + */ + if (__wt_row_leaf_value_exists(rip)) + return; + + __wt_row_leaf_value_cell(session, page, rip, NULL, &unpack); + __time_pairs_set(start, stop, &unpack); +} + +/* + * __wt_value_return_buf -- + * Change a buffer to reference an internal original-page return value. + */ +int +__wt_value_return_buf( + WT_CURSOR_BTREE *cbt, WT_REF *ref, WT_ITEM *buf, WT_TIME_PAIR *start, WT_TIME_PAIR *stop) { WT_BTREE *btree; WT_CELL *cell; @@ -88,39 +179,61 @@ __value_return(WT_CURSOR_BTREE *cbt) session = (WT_SESSION_IMPL *)cbt->iface.session; btree = S2BT(session); - page = cbt->ref->page; + page = ref->page; cursor = &cbt->iface; + if (start != NULL && stop != NULL) + __time_pairs_init(start, stop); + + /* Must provide either both start and stop as output parameters or neither. */ + WT_ASSERT(session, (start != NULL && stop != NULL) || (start == NULL && stop == NULL)); + if (page->type == WT_PAGE_ROW_LEAF) { rip = &page->pg_row[cbt->slot]; - /* Simple values have their location encoded in the WT_ROW. */ - if (__wt_row_leaf_value(page, rip, &cursor->value)) + /* + * If a value is simple and is globally visible at the time of reading a page into cache, we + * encode its location into the WT_ROW. + */ + if (__wt_row_leaf_value(page, rip, buf)) return (0); /* Take the value from the original page cell. */ __wt_row_leaf_value_cell(session, page, rip, NULL, &unpack); - return (__wt_page_cell_data_ref(session, page, &unpack, &cursor->value)); + if (start != NULL && stop != NULL) + __time_pairs_set(start, stop, &unpack); + + return (__wt_page_cell_data_ref(session, page, &unpack, buf)); } if (page->type == WT_PAGE_COL_VAR) { /* Take the value from the original page cell. */ cell = WT_COL_PTR(page, &page->pg_var[cbt->slot]); __wt_cell_unpack(session, page, cell, &unpack); - return (__wt_page_cell_data_ref(session, page, &unpack, &cursor->value)); + if (start != NULL && stop != NULL) + __time_pairs_set(start, stop, &unpack); + + return (__wt_page_cell_data_ref(session, page, &unpack, buf)); } - /* WT_PAGE_COL_FIX: Take the value from the original page. */ - v = __bit_getv_recno(cbt->ref, cursor->recno, btree->bitcnt); - return (__wt_buf_set(session, &cursor->value, &v, 1)); + /* + * WT_PAGE_COL_FIX: Take the value from the original page. + * + * FIXME-PM-1523: Should also check visibility here + */ + v = __bit_getv_recno(ref, cursor->recno, btree->bitcnt); + return (__wt_buf_set(session, buf, &v, 1)); } /* - * When threads race modifying a record, we can end up with more than the usual maximum number of - * modifications in an update list. We'd prefer not to allocate memory in a return path, so add a - * few additional slots to the array we use to build up a list of modify records to apply. + * __value_return -- + * Change the cursor to reference an internal original-page return value. */ -#define WT_MODIFY_ARRAY_SIZE (WT_MAX_MODIFY_UPDATE + 10) +static inline int +__value_return(WT_CURSOR_BTREE *cbt) +{ + return (__wt_value_return_buf(cbt, cbt->ref, &cbt->iface.value, NULL, NULL)); +} /* * __wt_value_return_upd -- @@ -131,14 +244,13 @@ __wt_value_return_upd(WT_CURSOR_BTREE *cbt, WT_UPDATE *upd) { WT_CURSOR *cursor; WT_DECL_RET; + WT_MODIFY_VECTOR modifies; WT_SESSION_IMPL *session; - WT_UPDATE **listp, *list[WT_MODIFY_ARRAY_SIZE]; - size_t allocated_bytes; - u_int i; + WT_TIME_PAIR start, stop; cursor = &cbt->iface; session = (WT_SESSION_IMPL *)cbt->iface.session; - allocated_bytes = 0; + __wt_modify_vector_init(session, &modifies); /* * We're passed a "standard" or "modified" update that's visible to us. Our caller should have @@ -147,8 +259,14 @@ __wt_value_return_upd(WT_CURSOR_BTREE *cbt, WT_UPDATE *upd) * Fast path if it's a standard item, assert our caller's behavior. */ if (upd->type == WT_UPDATE_STANDARD) { - cursor->value.data = upd->data; - cursor->value.size = upd->size; + if (F_ISSET(upd, WT_UPDATE_RESTORED_FROM_DISK)) { + /* Copy an external update, and delete after using it */ + WT_RET(__wt_buf_set(session, &cursor->value, upd->data, upd->size)); + __wt_free_update_list(session, &upd); + } else { + cursor->value.data = upd->data; + cursor->value.size = upd->size; + } return (0); } WT_ASSERT(session, upd->type == WT_UPDATE_MODIFY); @@ -156,33 +274,15 @@ __wt_value_return_upd(WT_CURSOR_BTREE *cbt, WT_UPDATE *upd) /* * Find a complete update. */ - for (i = 0, listp = list; upd != NULL; upd = upd->next) { + for (; upd != NULL; upd = upd->next) { if (upd->txnid == WT_TXN_ABORTED) continue; - if (upd->type == WT_UPDATE_BIRTHMARK) { - upd = NULL; - break; - } - if (WT_UPDATE_DATA_VALUE(upd)) break; - if (upd->type == WT_UPDATE_MODIFY) { - /* - * Update lists are expected to be short, but it's not guaranteed. There's sufficient - * room on the stack to avoid memory allocation in normal cases, but we have to handle - * the edge cases too. - */ - if (i >= WT_MODIFY_ARRAY_SIZE) { - if (i == WT_MODIFY_ARRAY_SIZE) - listp = NULL; - WT_ERR(__wt_realloc_def(session, &allocated_bytes, i + 1, &listp)); - if (i == WT_MODIFY_ARRAY_SIZE) - memcpy(listp, list, sizeof(list)); - } - listp[i++] = upd; - } + if (upd->type == WT_UPDATE_MODIFY) + WT_ERR(__wt_modify_vector_push(&modifies, upd)); } /* @@ -198,21 +298,28 @@ __wt_value_return_upd(WT_CURSOR_BTREE *cbt, WT_UPDATE *upd) */ WT_ASSERT(session, cbt->slot != UINT32_MAX); - WT_ERR(__value_return(cbt)); - } else if (upd->type == WT_UPDATE_TOMBSTONE) - WT_ERR(__wt_buf_set(session, &cursor->value, "", 0)); - else + WT_ERR(__wt_value_return_buf(cbt, cbt->ref, &cbt->iface.value, &start, &stop)); + /* + * Applying modifies on top of a tombstone is invalid. So if we're using the onpage value, + * the stop time pair should be unset. + */ + WT_ASSERT(session, stop.txnid == WT_TXN_MAX && stop.timestamp == WT_TS_MAX); + } else { + /* The base update must not be a tombstone. */ + WT_ASSERT(session, upd->type == WT_UPDATE_STANDARD); WT_ERR(__wt_buf_set(session, &cursor->value, upd->data, upd->size)); + } /* * Once we have a base item, roll forward through any visible modify updates. */ - while (i > 0) - WT_ERR(__wt_modify_apply(cursor, listp[--i]->data)); + while (modifies.size > 0) { + __wt_modify_vector_pop(&modifies, &upd); + WT_ERR(__wt_modify_apply(cursor, upd->data)); + } err: - if (allocated_bytes != 0) - __wt_free(session, listp); + __wt_modify_vector_free(&modifies); return (ret); } diff --git a/src/third_party/wiredtiger/src/btree/bt_slvg.c b/src/third_party/wiredtiger/src/btree/bt_slvg.c index 894bbbe2bfc..cc934937681 100644 --- a/src/third_party/wiredtiger/src/btree/bt_slvg.c +++ b/src/third_party/wiredtiger/src/btree/bt_slvg.c @@ -191,6 +191,7 @@ __slvg_checkpoint(WT_SESSION_IMPL *session, WT_REF *root) ckptbase->oldest_start_txn = WT_TXN_NONE; ckptbase->newest_stop_ts = WT_TS_MAX; ckptbase->newest_stop_txn = WT_TXN_MAX; + ckptbase->write_gen = btree->write_gen; F_SET(ckptbase, WT_CKPT_ADD); /* @@ -344,15 +345,22 @@ __wt_salvage(WT_SESSION_IMPL *session, const char *cfg[]) /* * !!! (Don't format the comment.) * Step 7: + * Track the maximum write gen of the leaf pages and set that as the btree write gen. * Build an internal page that references all of the leaf pages, and write it, as well as any * merged pages, to the file. * + * In the case of metadata, we will bump the connection base write gen to the metadata write gen + * after metadata salvage completes. + * * Count how many leaf pages we have (we could track this during the array shuffling/splitting, * but that's a lot harder). */ for (leaf_cnt = i = 0; i < ss->pages_next; ++i) - if (ss->pages[i] != NULL) + if (ss->pages[i] != NULL) { ++leaf_cnt; + btree->write_gen = WT_MAX(btree->write_gen, ss->pages[i]->shared->gen); + } + if (leaf_cnt != 0) switch (ss->page_type) { case WT_PAGE_COL_FIX: @@ -624,7 +632,7 @@ __slvg_trk_leaf(WT_SESSION_IMPL *session, const WT_PAGE_HEADER *dsk, uint8_t *ad * Page flags are 0 because we aren't releasing the memory used to read the page into memory * and we don't want page discard to free it. */ - WT_ERR(__wt_page_inmem(session, NULL, dsk, 0, false, &page)); + WT_ERR(__wt_page_inmem(session, NULL, dsk, 0, &page)); WT_ERR(__wt_row_leaf_key_copy(session, page, &page->pg_row[0], &trk->row_start)); WT_ERR( __wt_row_leaf_key_copy(session, page, &page->pg_row[page->entries - 1], &trk->row_stop)); @@ -688,7 +696,7 @@ __slvg_trk_leaf_ovfl(WT_SESSION_IMPL *session, const WT_PAGE_HEADER *dsk, WT_TRA /* Count page overflow items. */ ovfl_cnt = 0; WT_CELL_FOREACH_BEGIN (session, btree, dsk, unpack) { - if (unpack.ovfl) + if (FLD_ISSET(unpack.flags, WT_CELL_UNPACK_OVERFLOW)) ++ovfl_cnt; } WT_CELL_FOREACH_END; @@ -703,7 +711,7 @@ __slvg_trk_leaf_ovfl(WT_SESSION_IMPL *session, const WT_PAGE_HEADER *dsk, WT_TRA ovfl_cnt = 0; WT_CELL_FOREACH_BEGIN (session, btree, dsk, unpack) { - if (unpack.ovfl) { + if (FLD_ISSET(unpack.flags, WT_CELL_UNPACK_OVERFLOW)) { WT_RET( __wt_memdup(session, unpack.data, unpack.size, &trk->trk_ovfl_addr[ovfl_cnt].addr)); trk->trk_ovfl_addr[ovfl_cnt].size = (uint8_t)unpack.size; @@ -1165,7 +1173,7 @@ __slvg_col_build_internal(WT_SESSION_IMPL *session, uint32_t leaf_cnt, WT_STUFF * regardless of a value's timestamps or transaction IDs. */ WT_ERR(__wt_calloc_one(session, &addr)); - addr->newest_durable_ts = addr->oldest_start_ts = WT_TS_NONE; + addr->start_durable_ts = addr->stop_durable_ts = addr->oldest_start_ts = WT_TS_NONE; addr->oldest_start_txn = WT_TXN_NONE; addr->newest_stop_ts = WT_TS_MAX; addr->newest_stop_txn = WT_TXN_MAX; @@ -1176,6 +1184,7 @@ __slvg_col_build_internal(WT_SESSION_IMPL *session, uint32_t leaf_cnt, WT_STUFF addr = NULL; ref->ref_recno = trk->col_start; + F_SET(ref, WT_REF_FLAG_LEAF); WT_REF_SET_STATE(ref, WT_REF_DISK); /* @@ -1272,7 +1281,7 @@ __slvg_col_build_leaf(WT_SESSION_IMPL *session, WT_TRACK *trk, WT_REF *ref) /* Write the new version of the leaf page to disk. */ WT_ERR(__slvg_modify_init(session, page)); - WT_ERR(__wt_reconcile(session, ref, cookie, WT_REC_VISIBILITY_ERR, NULL)); + WT_ERR(__wt_reconcile(session, ref, cookie, WT_REC_VISIBILITY_ERR)); /* Reset the page. */ page->pg_var = save_col_var; @@ -1685,7 +1694,7 @@ __slvg_row_trk_update_start(WT_SESSION_IMPL *session, WT_ITEM *stop, uint32_t sl */ WT_RET(__wt_scr_alloc(session, trk->trk_size, &dsk)); WT_ERR(__wt_bt_read(session, dsk, trk->trk_addr, trk->trk_addr_size)); - WT_ERR(__wt_page_inmem(session, NULL, dsk->data, 0, false, &page)); + WT_ERR(__wt_page_inmem(session, NULL, dsk->data, 0, &page)); /* * Walk the page, looking for a key sorting greater than the specified stop key -- that's our @@ -1772,7 +1781,7 @@ __slvg_row_build_internal(WT_SESSION_IMPL *session, uint32_t leaf_cnt, WT_STUFF * regardless of a value's timestamps or transaction IDs. */ WT_ERR(__wt_calloc_one(session, &addr)); - addr->newest_durable_ts = addr->oldest_start_ts = WT_TS_NONE; + addr->start_durable_ts = addr->stop_durable_ts = addr->oldest_start_ts = WT_TS_NONE; addr->oldest_start_txn = WT_TXN_NONE; addr->newest_stop_ts = WT_TS_MAX; addr->newest_stop_txn = WT_TXN_MAX; @@ -1783,6 +1792,7 @@ __slvg_row_build_internal(WT_SESSION_IMPL *session, uint32_t leaf_cnt, WT_STUFF addr = NULL; __wt_ref_key_clear(ref); + F_SET(ref, WT_REF_FLAG_LEAF); WT_REF_SET_STATE(ref, WT_REF_DISK); /* @@ -1940,7 +1950,7 @@ __slvg_row_build_leaf(WT_SESSION_IMPL *session, WT_TRACK *trk, WT_REF *ref, WT_S /* Write the new version of the leaf page to disk. */ WT_ERR(__slvg_modify_init(session, page)); - WT_ERR(__wt_reconcile(session, ref, cookie, WT_REC_VISIBILITY_ERR, NULL)); + WT_ERR(__wt_reconcile(session, ref, cookie, WT_REC_VISIBILITY_ERR)); /* Reset the page. */ page->entries += skip_stop; diff --git a/src/third_party/wiredtiger/src/btree/bt_split.c b/src/third_party/wiredtiger/src/btree/bt_split.c index 3cc0d4018f0..fa58bc17caf 100644 --- a/src/third_party/wiredtiger/src/btree/bt_split.c +++ b/src/third_party/wiredtiger/src/btree/bt_split.c @@ -119,6 +119,10 @@ __split_verify_root(WT_SESSION_IMPL *session, WT_PAGE *page) WT_REF *ref; uint32_t read_flags; + /* + * Ignore pages not in-memory (deleted, on-disk, being read), there's no in-memory structure to + * check. + */ read_flags = WT_READ_CACHE | WT_READ_NO_EVICT; /* The split is complete and live, verify all of the pages involved. */ @@ -126,14 +130,8 @@ __split_verify_root(WT_SESSION_IMPL *session, WT_PAGE *page) WT_INTL_FOREACH_BEGIN (session, page, ref) { /* - * An eviction thread might be attempting to evict the page - * (the WT_REF may be WT_REF_LOCKED), or it may be a disk based - * page (the WT_REF may be WT_REF_READING), or it may be in - * some other state. Acquire a hazard pointer for any - * in-memory pages so we know the state of the page. - * - * Ignore pages not in-memory (deleted, on-disk, being read), - * there's no in-memory structure to check. + * The page might be in transition, being read or evicted or something else. Acquire a + * hazard pointer for the page so we know its state. */ if ((ret = __wt_page_in(session, ref, read_flags)) == WT_NOTFOUND) continue; @@ -184,7 +182,7 @@ __split_ovfl_key_cleanup(WT_SESSION_IMPL *session, WT_PAGE *page, WT_REF *ref) cell = WT_PAGE_REF_OFFSET(page, cell_offset); __wt_cell_unpack(session, page, cell, &kpack); - if (kpack.ovfl && kpack.raw != WT_CELL_KEY_OVFL_RM) + if (FLD_ISSET(kpack.flags, WT_CELL_UNPACK_OVERFLOW) && kpack.raw != WT_CELL_KEY_OVFL_RM) WT_RET(__wt_ovfl_discard(session, page, cell)); return (0); @@ -251,7 +249,8 @@ __split_ref_move(WT_SESSION_IMPL *session, WT_PAGE *from_home, WT_REF **from_ref if (ref_addr != NULL && !__wt_off_page(from_home, ref_addr)) { __wt_cell_unpack(session, from_home, (WT_CELL *)ref_addr, &unpack); WT_RET(__wt_calloc_one(session, &addr)); - addr->newest_durable_ts = unpack.newest_durable_ts; + addr->start_durable_ts = unpack.newest_start_durable_ts; + addr->stop_durable_ts = unpack.newest_stop_durable_ts; addr->oldest_start_ts = unpack.oldest_start_ts; addr->oldest_start_txn = unpack.oldest_start_txn; addr->newest_stop_ts = unpack.newest_stop_ts; @@ -478,6 +477,7 @@ __split_root(WT_SESSION_IMPL *session, WT_PAGE *root) root_incr += sizeof(WT_IKEY) + size; } else ref->ref_recno = (*root_refp)->ref_recno; + F_SET(ref, WT_REF_FLAG_INTERNAL); WT_REF_SET_STATE(ref, WT_REF_MEM); /* @@ -590,6 +590,60 @@ err: } /* + * __split_parent_discard_ref -- + * Worker routine to discard WT_REFs for the split-parent function. + */ +static int +__split_parent_discard_ref(WT_SESSION_IMPL *session, WT_REF *ref, WT_PAGE *parent, size_t *decrp, + uint64_t split_gen, bool exclusive) +{ + WT_DECL_RET; + WT_IKEY *ikey; + size_t size; + + /* + * Row-store trees where the old version of the page is being discarded: the previous parent + * page's key for this child page may have been an on-page overflow key. In that case, if the + * key hasn't been deleted, delete it now, including its backing blocks. We are exchanging the + * WT_REF that referenced it for the split page WT_REFs and their keys, and there's no longer + * any reference to it. Done after completing the split (if we failed, we'd leak the underlying + * blocks, but the parent page would be unaffected). + */ + if (parent->type == WT_PAGE_ROW_INT) { + WT_TRET(__split_ovfl_key_cleanup(session, parent, ref)); + ikey = __wt_ref_key_instantiated(ref); + if (ikey != NULL) { + size = sizeof(WT_IKEY) + ikey->size; + WT_TRET(__split_safe_free(session, split_gen, exclusive, ikey, size)); + *decrp += size; + } + } + + /* + * The page-delete and history store memory weren't added to the parent's footprint, ignore it + * here. + */ + if (ref->page_del != NULL) { + __wt_free(session, ref->page_del->update_list); + __wt_free(session, ref->page_del); + } + + /* Free the backing block and address. */ + WT_TRET(__wt_ref_block_free(session, ref)); + + /* + * Set the WT_REF state. It may be possible to immediately free the WT_REF, so this is our last + * chance. + */ + WT_REF_SET_STATE(ref, WT_REF_SPLIT); + + WT_TRET(__split_safe_free(session, split_gen, exclusive, ref, sizeof(WT_REF))); + *decrp += sizeof(WT_REF); + + return (ret); +} + +/* * __split_parent -- * Resolve a multi-page split, inserting new information into the parent. */ @@ -600,7 +654,6 @@ __split_parent(WT_SESSION_IMPL *session, WT_REF *ref, WT_REF **ref_new, uint32_t WT_BTREE *btree; WT_DECL_ITEM(scr); WT_DECL_RET; - WT_IKEY *ikey; WT_PAGE *parent; WT_PAGE_INDEX *alloc_index, *pindex; WT_REF **alloc_refp, *next_ref; @@ -617,6 +670,7 @@ __split_parent(WT_SESSION_IMPL *session, WT_REF *ref, WT_REF **ref_new, uint32_t alloc_index = pindex = NULL; parent_decr = 0; + deleted_refs = NULL; empty_parent = false; complete = WT_ERR_RETURN; @@ -633,34 +687,35 @@ __split_parent(WT_SESSION_IMPL *session, WT_REF *ref, WT_REF **ref_new, uint32_t /* * Remove any refs to deleted pages while we are splitting, we have the internal page locked - * down, and are copying the refs into a new array anyway. Switch them to the special split - * state, so that any reading thread will restart. + * down and are copying the refs into a new page-index array anyway. * * We can't do this if there is a sync running in the tree in another session: removing the refs * frees the blocks for the deleted pages, which can corrupt the free list calculated by the * sync. */ - WT_ERR(__wt_scr_alloc(session, 10 * sizeof(uint32_t), &scr)); - for (deleted_entries = 0, i = 0; i < parent_entries; ++i) { - next_ref = pindex->index[i]; - WT_ASSERT(session, next_ref->state != WT_REF_SPLIT); - if ((discard && next_ref == ref) || - ((!WT_BTREE_SYNCING(btree) || WT_SESSION_BTREE_SYNC(session)) && - next_ref->state == WT_REF_DELETED && __wt_delete_page_skip(session, next_ref, true) && - WT_REF_CAS_STATE(session, next_ref, WT_REF_DELETED, WT_REF_SPLIT))) { - WT_ERR(__wt_buf_grow(session, scr, (deleted_entries + 1) * sizeof(uint32_t))); - deleted_refs = scr->mem; - deleted_refs[deleted_entries++] = i; + deleted_entries = 0; + if (!WT_BTREE_SYNCING(btree) || WT_SESSION_BTREE_SYNC(session)) + for (i = 0; i < parent_entries; ++i) { + next_ref = pindex->index[i]; + WT_ASSERT(session, next_ref->state != WT_REF_SPLIT); + + /* Protect against including the replaced WT_REF in the list of deleted items. */ + if (next_ref != ref && next_ref->state == WT_REF_DELETED && + __wt_delete_page_skip(session, next_ref, true) && + WT_REF_CAS_STATE(session, next_ref, WT_REF_DELETED, WT_REF_LOCKED)) { + if (scr == NULL) + WT_ERR(__wt_scr_alloc(session, 10 * sizeof(uint32_t), &scr)); + WT_ERR(__wt_buf_grow(session, scr, (deleted_entries + 1) * sizeof(uint32_t))); + deleted_refs = scr->mem; + deleted_refs[deleted_entries++] = i; + } } - } /* - * The final entry count consists of the original count, plus any new pages, less any WT_REFs - * we're removing (deleted entries plus the entry we're replacing). + * The final entry count is the original count, where one entry will be replaced by some number + * of new entries, and some number will be deleted. */ - result_entries = (parent_entries + new_entries) - deleted_entries; - if (!discard) - --result_entries; + result_entries = (parent_entries + (new_entries - 1)) - deleted_entries; /* * If there are no remaining entries on the parent, give up, we can't leave an empty internal @@ -688,20 +743,29 @@ __split_parent(WT_SESSION_IMPL *session, WT_REF *ref, WT_REF **ref_new, uint32_t alloc_index->entries = result_entries; for (alloc_refp = alloc_index->index, hint = i = 0; i < parent_entries; ++i) { next_ref = pindex->index[i]; - if (next_ref == ref) + if (next_ref == ref) { for (j = 0; j < new_entries; ++j) { ref_new[j]->home = parent; ref_new[j]->pindex_hint = hint++; *alloc_refp++ = ref_new[j]; } - else if (next_ref->state != WT_REF_SPLIT) { - /* Skip refs we have marked for deletion. */ - next_ref->pindex_hint = hint++; - *alloc_refp++ = next_ref; + continue; } + + /* Skip refs we have marked for deletion. */ + if (deleted_entries != 0) { + for (j = 0; j < deleted_entries; ++j) + if (deleted_refs[j] == i) + break; + if (j < deleted_entries) + continue; + } + + next_ref->pindex_hint = hint++; + *alloc_refp++ = next_ref; } - /* Check that we filled in all the entries. */ + /* Check we filled in the expected number of entries. */ WT_ASSERT(session, alloc_refp - alloc_index->index == (ptrdiff_t)result_entries); /* Start making real changes to the tree, errors are fatal. */ @@ -730,26 +794,6 @@ __split_parent(WT_SESSION_IMPL *session, WT_REF *ref, WT_REF **ref_new, uint32_t split_gen = __wt_gen_next(session, WT_GEN_SPLIT); parent->pg_intl_split_gen = split_gen; - /* - * If discarding the page's original WT_REF field, reset it to split. Threads cursoring through - * the tree were blocked because that WT_REF state was set to locked. Changing the locked state - * to split unblocks those threads and causes them to re-calculate their position based on the - * just-updated parent page's index. - */ - if (discard) { - /* - * Set the discarded WT_REF state to split, ensuring we don't race with any discard of the - * WT_REF deleted fields. - */ - WT_REF_SET_STATE(ref, WT_REF_SPLIT); - - /* - * Push out the change: not required for correctness, but stops threads spinning on - * incorrect page references. - */ - WT_FULL_BARRIER(); - } - #ifdef HAVE_DIAGNOSTIC WT_WITH_PAGE_INDEX(session, __split_verify_intl_key_order(session, parent)); #endif @@ -758,72 +802,24 @@ __split_parent(WT_SESSION_IMPL *session, WT_REF *ref, WT_REF **ref_new, uint32_t complete = WT_ERR_IGNORE; /* - * !!! - * Swapping in the new page index released the page for eviction, we can - * no longer look inside the page. - */ - if (ref->page == NULL) - __wt_verbose(session, WT_VERB_SPLIT, - "%p: reverse split into parent %p, %" PRIu32 " -> %" PRIu32 " (-%" PRIu32 ")", - (void *)ref->page, (void *)parent, parent_entries, result_entries, - parent_entries - result_entries); - else - __wt_verbose(session, WT_VERB_SPLIT, - "%p: split into parent %p, %" PRIu32 " -> %" PRIu32 " (+%" PRIu32 ")", (void *)ref->page, - (void *)parent, parent_entries, result_entries, result_entries - parent_entries); - - /* - * The new page index is in place, free the WT_REF we were splitting and any deleted WT_REFs we - * found, modulo the usual safe free semantics. + * The new page index is in place. Threads cursoring in the tree are blocked because the WT_REF + * being discarded (if any), and deleted WT_REFs (if any) are in a locked state. Changing the + * locked state to split unblocks those threads and causes them to re-calculate their position + * based on the just-updated parent page's index. The split state doesn't lock the WT_REF.addr + * information which is read by cursor threads in some tree-walk cases: free the WT_REF we were + * splitting and any deleted WT_REFs we found, modulo the usual safe free semantics, then reset + * the WT_REF state. */ - for (i = 0, deleted_refs = scr->mem; i < deleted_entries; ++i) { + if (discard) { + WT_ASSERT(session, exclusive || ref->state == WT_REF_LOCKED); + WT_TRET( + __split_parent_discard_ref(session, ref, parent, &parent_decr, split_gen, exclusive)); + } + for (i = 0; i < deleted_entries; ++i) { next_ref = pindex->index[deleted_refs[i]]; -#ifdef HAVE_DIAGNOSTIC - { - uint32_t ref_state; - WT_ORDERED_READ(ref_state, next_ref->state); - WT_ASSERT(session, ref_state == WT_REF_LOCKED || ref_state == WT_REF_SPLIT); - } -#endif - - /* - * We set the WT_REF to split, discard it, freeing any resources it holds. - * - * Row-store trees where the old version of the page is being discarded: the previous parent - * page's key for this child page may have been an on-page overflow key. In that case, if - * the key hasn't been deleted, delete it now, including its backing blocks. We are - * exchanging the WT_REF that referenced it for the split page WT_REFs and their keys, and - * there's no longer any reference to it. Done after completing the split (if we failed, - * we'd leak the underlying blocks, but the parent page would be unaffected). - */ - if (parent->type == WT_PAGE_ROW_INT) { - WT_TRET(__split_ovfl_key_cleanup(session, parent, next_ref)); - ikey = __wt_ref_key_instantiated(next_ref); - if (ikey != NULL) { - size = sizeof(WT_IKEY) + ikey->size; - WT_TRET(__split_safe_free(session, split_gen, exclusive, ikey, size)); - parent_decr += size; - } - } - - /* Check that we are not discarding active history. */ - WT_ASSERT(session, !__wt_page_las_active(session, next_ref)); - - /* - * The page-delete and lookaside memory weren't added to the parent's footprint, ignore it - * here. - */ - if (next_ref->page_del != NULL) { - __wt_free(session, next_ref->page_del->update_list); - __wt_free(session, next_ref->page_del); - } - __wt_free(session, next_ref->page_las); - - /* Free the backing block and address. */ - WT_TRET(__wt_ref_block_free(session, next_ref)); - - WT_TRET(__split_safe_free(session, split_gen, exclusive, next_ref, sizeof(WT_REF))); - parent_decr += sizeof(WT_REF); + WT_ASSERT(session, next_ref->state == WT_REF_LOCKED); + WT_TRET(__split_parent_discard_ref( + session, next_ref, parent, &parent_decr, split_gen, exclusive)); } /* @@ -832,6 +828,12 @@ __split_parent(WT_SESSION_IMPL *session, WT_REF *ref, WT_REF **ref_new, uint32_t */ /* + * Don't cache the change: not required for correctness, but stops threads spinning on incorrect + * page references. + */ + WT_FULL_BARRIER(); + + /* * We can't free the previous page index, there may be threads using it. Add it to the session * discard list, to be freed when it's safe. */ @@ -843,6 +845,14 @@ __split_parent(WT_SESSION_IMPL *session, WT_REF *ref, WT_REF **ref_new, uint32_t __wt_cache_page_inmem_incr(session, parent, parent_incr); __wt_cache_page_inmem_decr(session, parent, parent_decr); + /* + * We've discarded the WT_REFs and swapping in a new page index released the page for eviction; + * we can no longer look inside the WT_REF or the page, be careful logging the results. + */ + __wt_verbose(session, WT_VERB_SPLIT, + "%p: split into parent, %" PRIu32 "->%" PRIu32 ", %" PRIu32 " deleted", (void *)ref, + parent_entries, result_entries, deleted_entries); + err: __wt_scr_free(session, &scr); /* @@ -851,10 +861,11 @@ err: */ switch (complete) { case WT_ERR_RETURN: - for (i = 0; i < parent_entries; ++i) { - next_ref = pindex->index[i]; - if (next_ref->state == WT_REF_SPLIT) - WT_REF_SET_STATE(next_ref, WT_REF_DELETED); + /* Unlock WT_REFs locked because they were in a deleted state. */ + for (i = 0; i < deleted_entries; ++i) { + next_ref = pindex->index[deleted_refs[i]]; + WT_ASSERT(session, next_ref->state == WT_REF_LOCKED); + WT_REF_SET_STATE(next_ref, WT_REF_DELETED); } __wt_free_ref_index(session, NULL, alloc_index, false); @@ -1003,6 +1014,7 @@ __split_internal(WT_SESSION_IMPL *session, WT_PAGE *parent, WT_PAGE *page) parent_incr += sizeof(WT_IKEY) + size; } else ref->ref_recno = (*page_refp)->ref_recno; + F_SET(ref, WT_REF_FLAG_INTERNAL); WT_REF_SET_STATE(ref, WT_REF_MEM); /* @@ -1363,24 +1375,6 @@ err: return (ret); } -#ifdef HAVE_DIAGNOSTIC -/* - * __wt_count_birthmarks -- - * Sanity check an update list. In particular, make sure there no birthmarks. - */ -int -__wt_count_birthmarks(WT_UPDATE *upd) -{ - int birthmark_count; - - for (birthmark_count = 0; upd != NULL; upd = upd->next) - if (upd->type == WT_UPDATE_BIRTHMARK) - ++birthmark_count; - - return (birthmark_count); -} -#endif - /* * __split_multi_inmem -- * Instantiate a page from a disk image. @@ -1398,8 +1392,6 @@ __split_multi_inmem(WT_SESSION_IMPL *session, WT_PAGE *orig, WT_MULTI *multi, WT uint64_t recno; uint32_t i, slot; - WT_ASSERT(session, multi->page_las.las_pageid == 0); - /* * In 04/2016, we removed column-store record numbers from the WT_PAGE structure, leading to * hard-to-debug problems because we corrupt the page if we search it using the wrong initial @@ -1420,7 +1412,7 @@ __split_multi_inmem(WT_SESSION_IMPL *session, WT_PAGE *orig, WT_MULTI *multi, WT * our caller will not discard the disk image when discarding the original page, and our caller * will discard the allocated page on error, when discarding the allocated WT_REF. */ - WT_RET(__wt_page_inmem(session, ref, multi->disk_image, WT_PAGE_DISK_ALLOC, false, &page)); + WT_RET(__wt_page_inmem(session, ref, multi->disk_image, WT_PAGE_DISK_ALLOC, &page)); multi->disk_image = NULL; /* @@ -1470,14 +1462,9 @@ __split_multi_inmem(WT_SESSION_IMPL *session, WT_PAGE *orig, WT_MULTI *multi, WT key->size = WT_INSERT_KEY_SIZE(supd->ins); } - WT_ASSERT(session, __wt_count_birthmarks(upd) <= 1); - /* Search the page. */ WT_ERR(__wt_row_search(&cbt, key, true, ref, true, NULL)); - /* Birthmarks should only be applied to on-page values. */ - WT_ASSERT(session, cbt.compare == 0 || upd->type != WT_UPDATE_BIRTHMARK); - /* Apply the modification. */ WT_ERR(__wt_row_modify(&cbt, key, NULL, upd, WT_UPDATE_INVALID, true)); break; @@ -1495,17 +1482,17 @@ __split_multi_inmem(WT_SESSION_IMPL *session, WT_PAGE *orig, WT_MULTI *multi, WT mod->first_dirty_txn = WT_TXN_FIRST; /* - * If the new page is modified, save the eviction generation to avoid repeatedly attempting - * eviction on the same page. + * Restore the previous page's modify state to avoid repeatedly attempting eviction on the same + * page. */ mod->last_evict_pass_gen = orig->modify->last_evict_pass_gen; mod->last_eviction_id = orig->modify->last_eviction_id; mod->last_eviction_timestamp = orig->modify->last_eviction_timestamp; - - /* Add the update/restore flag to any previous state. */ - mod->last_stable_timestamp = orig->modify->last_stable_timestamp; mod->rec_max_txn = orig->modify->rec_max_txn; mod->rec_max_timestamp = orig->modify->rec_max_timestamp; + mod->last_stable_timestamp = orig->modify->last_stable_timestamp; + + /* Add the update/restore flag to any previous state. */ mod->restore_state = orig->modify->restore_state; FLD_SET(mod->restore_state, WT_PAGE_RS_RESTORED); @@ -1583,6 +1570,20 @@ __wt_multi_to_ref(WT_SESSION_IMPL *session, WT_PAGE *page, WT_MULTI *multi, WT_R WT_IKEY *ikey; WT_REF *ref; + /* There can be an address or a disk image or both. */ + WT_ASSERT(session, multi->addr.addr != NULL || multi->disk_image != NULL); + + /* If closing the file, there better be an address. */ + WT_ASSERT(session, !closing || multi->addr.addr != NULL); + + /* If closing the file, there better not be any saved updates. */ + WT_ASSERT(session, !closing || multi->supd == NULL); + + /* Verify any disk image we have. */ + WT_ASSERT(session, multi->disk_image == NULL || + __wt_verify_dsk_image( + session, "[page instantiate]", multi->disk_image, 0, &multi->addr, true) == 0); + /* Allocate an underlying WT_REF. */ WT_RET(__wt_calloc_one(session, refp)); ref = *refp; @@ -1606,26 +1607,15 @@ __wt_multi_to_ref(WT_SESSION_IMPL *session, WT_PAGE *page, WT_MULTI *multi, WT_R break; } - /* - * There can be an address or a disk image or both, but if there is neither, there must be a - * backing lookaside page. - */ - WT_ASSERT(session, - multi->page_las.las_pageid != 0 || multi->addr.addr != NULL || multi->disk_image != NULL); - - /* If closing the file, there better be an address. */ - WT_ASSERT(session, !closing || multi->addr.addr != NULL); - - /* If closing the file, there better not be any saved updates. */ - WT_ASSERT(session, !closing || multi->supd == NULL); - - /* If there are saved updates, there better be a disk image. */ - WT_ASSERT(session, multi->supd == NULL || multi->disk_image != NULL); - - /* Verify any disk image we have. */ - WT_ASSERT(session, multi->disk_image == NULL || - __wt_verify_dsk_image( - session, "[page instantiate]", multi->disk_image, 0, &multi->addr, true) == 0); + switch (page->type) { + case WT_PAGE_COL_INT: + case WT_PAGE_ROW_INT: + F_SET(ref, WT_REF_FLAG_INTERNAL); + break; + default: + F_SET(ref, WT_REF_FLAG_LEAF); + break; + } /* * If there's an address, the page was written, set it. @@ -1637,7 +1627,8 @@ __wt_multi_to_ref(WT_SESSION_IMPL *session, WT_PAGE *page, WT_MULTI *multi, WT_R if (multi->addr.addr != NULL) { WT_RET(__wt_calloc_one(session, &addr)); ref->addr = addr; - addr->newest_durable_ts = multi->addr.newest_durable_ts; + addr->start_durable_ts = multi->addr.start_durable_ts; + addr->stop_durable_ts = multi->addr.stop_durable_ts; addr->oldest_start_ts = multi->addr.oldest_start_ts; addr->oldest_start_txn = multi->addr.oldest_start_txn; addr->newest_stop_ts = multi->addr.newest_stop_ts; @@ -1650,22 +1641,6 @@ __wt_multi_to_ref(WT_SESSION_IMPL *session, WT_PAGE *page, WT_MULTI *multi, WT_R } /* - * Copy any associated lookaside reference, potentially resetting WT_REF.state. Regardless of a - * backing address, WT_REF_LOOKASIDE overrides WT_REF_DISK. - */ - if (multi->page_las.las_pageid != 0) { - /* - * We should not have a disk image if we did lookaside eviction. - */ - WT_ASSERT(session, multi->disk_image == NULL); - - WT_RET(__wt_calloc_one(session, &ref->page_las)); - *ref->page_las = multi->page_las; - WT_ASSERT(session, ref->page_las->max_txn != WT_TXN_NONE); - WT_REF_SET_STATE(ref, WT_REF_LOOKASIDE); - } - - /* * If we have a disk image and we're not closing the file, re-instantiate the page. * * Discard any page image we don't use. @@ -1739,8 +1714,12 @@ __split_insert(WT_SESSION_IMPL *session, WT_REF *ref) child->page = ref->page; child->home = ref->home; child->pindex_hint = ref->pindex_hint; - child->state = WT_REF_MEM; child->addr = ref->addr; + F_SET(child, WT_REF_FLAG_LEAF); + child->state = WT_REF_MEM; + + WT_ERR_ASSERT(session, ref->page_del == NULL, WT_PANIC, + "unexpected page-delete structure when splitting a page"); /* * The address has moved to the replacement WT_REF. Make sure it isn't freed when the original @@ -1799,6 +1778,7 @@ __split_insert(WT_SESSION_IMPL *session, WT_REF *ref) parent_incr += sizeof(WT_REF); child = split_ref[1]; child->page = right; + F_SET(child, WT_REF_FLAG_LEAF); child->state = WT_REF_MEM; if (type == WT_PAGE_ROW_LEAF) { diff --git a/src/third_party/wiredtiger/src/btree/bt_stat.c b/src/third_party/wiredtiger/src/btree/bt_stat.c index 20add448fed..2005c279771 100644 --- a/src/third_party/wiredtiger/src/btree/bt_stat.c +++ b/src/third_party/wiredtiger/src/btree/bt_stat.c @@ -163,7 +163,7 @@ __stat_page_col_var(WT_SESSION_IMPL *session, WT_PAGE *page, WT_DSRC_STATS **sta entry_cnt += __wt_cell_rle(unpack); } rle_cnt += __wt_cell_rle(unpack) - 1; - if (unpack->ovfl) + if (F_ISSET(unpack, WT_CELL_UNPACK_OVERFLOW)) ++ovfl_cnt; /* diff --git a/src/third_party/wiredtiger/src/btree/bt_sync.c b/src/third_party/wiredtiger/src/btree/bt_sync.c index 8778496cd7a..ed0fdde6839 100644 --- a/src/third_party/wiredtiger/src/btree/bt_sync.c +++ b/src/third_party/wiredtiger/src/btree/bt_sync.c @@ -9,31 +9,47 @@ #include "wt_internal.h" /* + * A list of WT_REF's. + */ +typedef struct { + WT_REF **list; + size_t entry; /* next entry available in list */ + size_t max_entry; /* how many allocated in list */ +} WT_REF_LIST; + +/* * __sync_checkpoint_can_skip -- * There are limited conditions under which we can skip writing a dirty page during checkpoint. */ static inline bool -__sync_checkpoint_can_skip(WT_SESSION_IMPL *session, WT_PAGE *page) +__sync_checkpoint_can_skip(WT_SESSION_IMPL *session, WT_REF *ref) { WT_MULTI *multi; WT_PAGE_MODIFY *mod; WT_TXN *txn; u_int i; - mod = page->modify; + mod = ref->page->modify; txn = &session->txn; /* * We can skip some dirty pages during a checkpoint. The requirements: * - * 1. they must be leaf pages, - * 2. there is a snapshot transaction active (which is the case in - * ordinary application checkpoints but not all internal cases), - * 3. the first dirty update on the page is sufficiently recent the - * checkpoint transaction would skip them, - * 4. there's already an address for every disk block involved. + * 1. Not a history btree. As part of the checkpointing the data store, we will move the older + * values into the history store without using any transactions. This led to representation + * of all the modifications on the history store page with a transaction that is maximum than + * the checkpoint snapshot. But these modifications are done by the checkpoint itself, so we + * shouldn't ignore them for consistency. + * 2. they must be leaf pages, + * 3. there is a snapshot transaction active (which is the case in ordinary application + * checkpoints but not all internal cases), + * 4. the first dirty update on the page is sufficiently recent the checkpoint transaction would + * skip them, + * 5. there's already an address for every disk block involved. */ - if (WT_PAGE_IS_INTERNAL(page)) + if (WT_IS_HS(S2BT(session))) + return (false); + if (F_ISSET(ref, WT_REF_FLAG_INTERNAL)) return (false); if (!F_ISSET(txn, WT_TXN_HAS_SNAPSHOT)) return (false); @@ -59,6 +75,30 @@ __sync_checkpoint_can_skip(WT_SESSION_IMPL *session, WT_PAGE *page) } /* + * __sync_dup_hazard_pointer -- + * Get a duplicate hazard pointer. + */ +static inline int +__sync_dup_hazard_pointer(WT_SESSION_IMPL *session, WT_REF *walk) +{ + bool busy; + + /* Get a duplicate hazard pointer. */ + for (;;) { + /* + * We already have a hazard pointer, we should generally be able to get another one. We can + * get spurious busy errors (e.g., if eviction is attempting to lock the page). Keep trying: + * we have one hazard pointer so we should be able to get another one. + */ + WT_RET(__wt_hazard_set(session, walk, &busy)); + if (!busy) + break; + __wt_yield(); + } + return (0); +} + +/* * __sync_dup_walk -- * Duplicate a tree walk point. */ @@ -66,7 +106,6 @@ static inline int __sync_dup_walk(WT_SESSION_IMPL *session, WT_REF *walk, uint32_t flags, WT_REF **dupp) { WT_REF *old; - bool busy; if ((old = *dupp) != NULL) { *dupp = NULL; @@ -79,24 +118,224 @@ __sync_dup_walk(WT_SESSION_IMPL *session, WT_REF *walk, uint32_t flags, WT_REF * return (0); } - /* Get a duplicate hazard pointer. */ - for (;;) { -#ifdef HAVE_DIAGNOSTIC - WT_RET(__wt_hazard_set(session, walk, &busy, __func__, __LINE__)); -#else - WT_RET(__wt_hazard_set(session, walk, &busy)); -#endif + WT_RET(__sync_dup_hazard_pointer(session, walk)); + *dupp = walk; + return (0); +} + +/* + * __sync_ref_list_add -- + * Add an obsolete history store ref to the list. + */ +static int +__sync_ref_list_add(WT_SESSION_IMPL *session, WT_REF_LIST *rlp, WT_REF *ref) +{ + WT_RET(__wt_realloc_def(session, &rlp->max_entry, rlp->entry + 1, &rlp->list)); + rlp->list[rlp->entry++] = ref; + return (0); +} + +/* + * __sync_ref_list_pop -- + * Add the stored ref to urgent eviction queue and free the list. + */ +static int +__sync_ref_list_pop(WT_SESSION_IMPL *session, WT_REF_LIST *rlp, uint32_t flags) +{ + WT_DECL_RET; + size_t i; + + for (i = 0; i < rlp->entry; i++) { /* - * We already have a hazard pointer, we should generally be able to get another one. We can - * get spurious busy errors (e.g., if eviction is attempting to lock the page. Keep trying: - * we have one hazard pointer so we should be able to get another one. + * Ignore the failure from urgent eviction. The failed refs are taken care in the next + * checkpoint. */ - if (!busy) - break; - __wt_yield(); + WT_IGNORE_RET_BOOL(__wt_page_evict_urgent(session, rlp->list[i])); + + /* Accumulate errors but continue till all the refs are processed. */ + WT_TRET(__wt_page_release(session, rlp->list[i], flags)); + WT_STAT_CONN_INCR(session, hs_gc_pages_evict); + __wt_verbose(session, WT_VERB_CHECKPOINT_GC, + "%p: is an in-memory obsolete page, added to urgent eviction queue.", + (void *)rlp->list[i]); } - *dupp = walk; + __wt_free(session, rlp->list); + rlp->entry = 0; + rlp->max_entry = 0; + + return (ret); +} + +/* + * __sync_ref_obsolete_check -- + * Check whether the ref is obsolete according to the newest stop time pair and handle the + * obsolete page. + */ +static int +__sync_ref_obsolete_check(WT_SESSION_IMPL *session, WT_REF *ref, WT_REF_LIST *rlp) +{ + WT_ADDR_COPY addr; + WT_DECL_RET; + WT_MULTI *multi; + WT_PAGE_MODIFY *mod; + wt_timestamp_t newest_stop_ts; + uint64_t newest_stop_txn; + uint32_t i; + uint8_t previous_state; + char tp_string[WT_TP_STRING_SIZE]; + const char *tag; + bool busy, hazard, obsolete; + + /* Ignore root pages as they can never be deleted. */ + if (__wt_ref_is_root(ref)) { + __wt_verbose(session, WT_VERB_CHECKPOINT_GC, "%p: skipping root page", (void *)ref); + return (0); + } + + /* Ignore internal pages, these are taken care of during reconciliation. */ + if (F_ISSET(ref, WT_REF_FLAG_INTERNAL)) { + __wt_verbose(session, WT_VERB_CHECKPOINT_GC, "%p: skipping internal page with parent: %p", + (void *)ref, (void *)ref->home); + return (0); + } + + /* Fast-check, ignore deleted pages. */ + if (ref->state == WT_REF_DELETED) { + __wt_verbose(session, WT_VERB_CHECKPOINT_GC, "%p: skipping deleted page", (void *)ref); + return (0); + } + + /* Lock the WT_REF. */ + WT_REF_LOCK(session, ref, &previous_state); + + /* + * If the page is on-disk and obsolete, mark the page as deleted and also set the parent page as + * dirty. This is to ensure the parent is written during the checkpoint and the child page + * discarded. + */ + newest_stop_ts = WT_TS_NONE; + newest_stop_txn = WT_TXN_NONE; + obsolete = false; + if (previous_state == WT_REF_DISK) { + /* There should be an address, but simply skip any page where we don't find one. */ + if (__wt_ref_addr_copy(session, ref, &addr)) { + newest_stop_ts = addr.newest_stop_ts; + newest_stop_txn = addr.newest_stop_txn; + obsolete = __wt_txn_visible_all(session, newest_stop_txn, newest_stop_ts); + } + + if (obsolete) { + WT_REF_UNLOCK(ref, WT_REF_DELETED); + WT_STAT_CONN_INCR(session, hs_gc_pages_removed); + + WT_RET(__wt_page_parent_modify_set(session, ref, true)); + } else + WT_REF_UNLOCK(ref, previous_state); + + __wt_verbose(session, WT_VERB_CHECKPOINT_GC, + "%p on-disk page obsolete check: %s" + "obsolete, stop ts/txn %s", + (void *)ref, obsolete ? "" : "not ", + __wt_time_pair_to_string(newest_stop_ts, newest_stop_txn, tp_string)); + return (0); + } + WT_REF_UNLOCK(ref, previous_state); + + /* + * Ignore pages that aren't in-memory for some reason other than they're on-disk, for example, + * they might have split or been deleted while we were locking the WT_REF. + */ + if (previous_state != WT_REF_MEM) { + __wt_verbose(session, WT_VERB_CHECKPOINT_GC, "%p: skipping page", (void *)ref); + return (0); + } + + /* + * Reviewing in-memory pages requires looking at page reconciliation results and we must ensure + * we don't race with page reconciliation as it's writing the page modify information. There are + * two ways we call reconciliation: checkpoints and eviction. We are the checkpoint thread so + * that's not a problem, acquire a hazard pointer to prevent page eviction. If the page is in + * transition or switches state (we've already released our lock), just walk away, we'll deal + * with it next time. + */ + WT_RET(__wt_hazard_set(session, ref, &busy)); + if (busy) + return (0); + hazard = true; + + mod = ref->page == NULL ? NULL : ref->page->modify; + if (mod != NULL && mod->rec_result == WT_PM_REC_EMPTY) { + tag = "reconciled empty"; + + obsolete = true; + } else if (mod != NULL && mod->rec_result == WT_PM_REC_MULTIBLOCK) { + tag = "reconciled multi-block"; + + /* Calculate the max stop time pair by traversing all multi addresses. */ + for (multi = mod->mod_multi, i = 0; i < mod->mod_multi_entries; ++multi, ++i) { + newest_stop_txn = WT_MAX(newest_stop_txn, multi->addr.newest_stop_txn); + newest_stop_ts = WT_MAX(newest_stop_ts, multi->addr.newest_stop_ts); + } + obsolete = __wt_txn_visible_all(session, newest_stop_txn, newest_stop_ts); + } else if (mod != NULL && mod->rec_result == WT_PM_REC_REPLACE) { + tag = "reconciled replacement block"; + + newest_stop_txn = mod->mod_replace.newest_stop_txn; + newest_stop_ts = mod->mod_replace.newest_stop_ts; + obsolete = __wt_txn_visible_all(session, newest_stop_txn, newest_stop_ts); + } else if (__wt_ref_addr_copy(session, ref, &addr)) { + tag = "WT_REF address"; + + newest_stop_txn = addr.newest_stop_txn; + newest_stop_ts = addr.newest_stop_ts; + obsolete = __wt_txn_visible_all(session, newest_stop_txn, newest_stop_ts); + } else + tag = "unexpected page state"; + + /* If the page is obsolete, add it to the list of pages to be evicted. */ + if (obsolete) { + WT_ERR(__sync_ref_list_add(session, rlp, ref)); + hazard = false; + } + + __wt_verbose(session, WT_VERB_CHECKPOINT_GC, + "%p in-memory page obsolete check: %s %s" + "obsolete, stop ts/txn %s", + (void *)ref, tag, obsolete ? "" : "not ", + __wt_time_pair_to_string(newest_stop_ts, newest_stop_txn, tp_string)); + +err: + if (hazard) + WT_TRET(__wt_hazard_clear(session, ref)); + return (ret); +} + +/* + * __sync_ref_int_obsolete_cleanup -- + * Traverse the internal page and identify the leaf pages that are obsolete and mark them as + * deleted. + */ +static int +__sync_ref_int_obsolete_cleanup(WT_SESSION_IMPL *session, WT_REF *parent, WT_REF_LIST *rlp) +{ + WT_PAGE_INDEX *pindex; + WT_REF *ref; + uint32_t slot; + + __wt_verbose(session, WT_VERB_CHECKPOINT_GC, + "%p: traversing the internal page %p for obsolete child pages", (void *)parent, + (void *)parent->page); + + WT_INTL_INDEX_GET(session, parent->page, pindex); + for (slot = 0; slot < pindex->entries; slot++) { + ref = pindex->index[slot]; + + WT_RET(__sync_ref_obsolete_check(session, ref, rlp)); + } + + WT_STAT_CONN_INCRV(session, hs_gc_pages_visited, pindex->entries); + return (0); } @@ -113,11 +352,12 @@ __wt_sync_file(WT_SESSION_IMPL *session, WT_CACHE_OP syncop) WT_PAGE *page; WT_PAGE_MODIFY *mod; WT_REF *prev, *walk; + WT_REF_LIST ref_list; WT_TXN *txn; uint64_t internal_bytes, internal_pages, leaf_bytes, leaf_pages; uint64_t oldest_id, saved_pinned_id, time_start, time_stop; - uint32_t flags; - bool timer, tried_eviction; + uint32_t flags, rec_flags; + bool dirty, is_hs, timer, tried_eviction; conn = S2C(session); btree = S2BT(session); @@ -125,6 +365,7 @@ __wt_sync_file(WT_SESSION_IMPL *session, WT_CACHE_OP syncop) txn = &session->txn; tried_eviction = false; time_start = time_stop = 0; + is_hs = false; /* Only visit pages in cache and don't bump page read generations. */ flags = WT_READ_CACHE | WT_READ_NO_GEN; @@ -186,7 +427,7 @@ __wt_sync_file(WT_SESSION_IMPL *session, WT_CACHE_OP syncop) __wt_txn_get_snapshot(session); leaf_bytes += page->memory_footprint; ++leaf_pages; - WT_ERR(__wt_reconcile(session, walk, NULL, WT_REC_CHECKPOINT, NULL)); + WT_ERR(__wt_reconcile(session, walk, NULL, WT_REC_CHECKPOINT)); } } break; @@ -226,49 +467,79 @@ __wt_sync_file(WT_SESSION_IMPL *session, WT_CACHE_OP syncop) btree->syncing = WT_BTREE_SYNC_WAIT; __wt_gen_next_drain(session, WT_GEN_EVICT); btree->syncing = WT_BTREE_SYNC_RUNNING; + is_hs = WT_IS_HS(btree); + + /* + * Add in history store reconciliation for standard files. + * + * FIXME-PM-1521: Remove the history store check, and assert that no updates from the + * history store are copied to the history store recursively. + */ + rec_flags = WT_REC_CHECKPOINT; + if (!is_hs && !WT_IS_METADATA(btree->dhandle)) + rec_flags |= WT_REC_HS; /* Write all dirty in-cache pages. */ LF_SET(WT_READ_NO_EVICT); - /* Read pages with lookaside entries and evict them asap. */ - LF_SET(WT_READ_LOOKASIDE | WT_READ_WONT_NEED); + /* Read pages with history store entries and evict them asap. */ + LF_SET(WT_READ_WONT_NEED); + + /* Read internal pages if it is history store */ + if (is_hs) { + LF_CLR(WT_READ_CACHE); + LF_SET(WT_READ_CACHE_LEAF); + memset(&ref_list, 0, sizeof(ref_list)); + } for (;;) { WT_ERR(__sync_dup_walk(session, walk, flags, &prev)); WT_ERR(__wt_tree_walk(session, &walk, flags)); - if (walk == NULL) + if (walk == NULL) { + if (is_hs) + WT_ERR(__sync_ref_list_pop(session, &ref_list, flags)); break; + } + + /* Traverse through the internal page for obsolete child pages. */ + if (is_hs && F_ISSET(walk, WT_REF_FLAG_INTERNAL)) { + WT_WITH_PAGE_INDEX( + session, ret = __sync_ref_int_obsolete_cleanup(session, walk, &ref_list)); + WT_ERR(ret); + } + + page = walk->page; /* - * Skip clean pages, but need to make sure maximum transaction ID is always updated. + * Check if the page is dirty. Add a barrier between the check and taking a reference to + * any page modify structure. (It needs to be ordered else a page could be dirtied after + * taking the local reference.) */ - if (!__wt_page_is_modified(walk->page)) { - if (((mod = walk->page->modify) != NULL) && mod->rec_max_txn > btree->rec_max_txn) + WT_ORDERED_READ(dirty, __wt_page_is_modified(page)); + + /* Skip clean pages, but always update the maximum transaction ID. */ + if (!dirty) { + mod = page->modify; + if (mod != NULL && mod->rec_max_txn > btree->rec_max_txn) btree->rec_max_txn = mod->rec_max_txn; if (mod != NULL && btree->rec_max_timestamp < mod->rec_max_timestamp) btree->rec_max_timestamp = mod->rec_max_timestamp; + continue; } /* - * Take a local reference to the page modify structure now that we know the page is - * dirty. It needs to be done in this order otherwise the page modify structure could - * have been created between taking the reference and checking modified. - */ - page = walk->page; - - /* * Write dirty pages, if we can't skip them. If we skip a page, mark the tree dirty. The * checkpoint marked it clean and we can't skip future checkpoints until this page is * written. */ - if (__sync_checkpoint_can_skip(session, page)) { + if (__sync_checkpoint_can_skip(session, walk)) { __wt_tree_modify_set(session); continue; } - if (WT_PAGE_IS_INTERNAL(page)) { + if (F_ISSET(walk, WT_REF_FLAG_INTERNAL)) { internal_bytes += page->memory_footprint; ++internal_pages; /* Slow down checkpoints. */ @@ -295,7 +566,7 @@ __wt_sync_file(WT_SESSION_IMPL *session, WT_CACHE_OP syncop) * Once the transaction has given up it's snapshot it is no longer safe to reconcile * pages. That happens prior to the final metadata checkpoint. */ - if (!WT_PAGE_IS_INTERNAL(page) && page->read_gen == WT_READGEN_WONT_NEED && + if (F_ISSET(walk, WT_REF_FLAG_LEAF) && page->read_gen == WT_READGEN_WONT_NEED && !tried_eviction && F_ISSET(&session->txn, WT_TXN_HAS_SNAPSHOT)) { ret = __wt_page_release_evict(session, walk, 0); walk = NULL; @@ -308,7 +579,7 @@ __wt_sync_file(WT_SESSION_IMPL *session, WT_CACHE_OP syncop) } tried_eviction = false; - WT_ERR(__wt_reconcile(session, walk, NULL, WT_REC_CHECKPOINT, NULL)); + WT_ERR(__wt_reconcile(session, walk, NULL, rec_flags)); /* * Update checkpoint IO tracking data if configured to log verbose progress messages. @@ -343,6 +614,10 @@ err: WT_TRET(__wt_page_release(session, walk, flags)); WT_TRET(__wt_page_release(session, prev, flags)); + /* On error, Process the refs that are saved and free the list. */ + if (is_hs) + WT_TRET(__sync_ref_list_pop(session, &ref_list, flags)); + /* * If we got a snapshot in order to write pages, and there was no snapshot active when we * started, release it. diff --git a/src/third_party/wiredtiger/src/btree/bt_vrfy.c b/src/third_party/wiredtiger/src/btree/bt_vrfy.c index ea134802bdd..4c0962d180b 100644 --- a/src/third_party/wiredtiger/src/btree/bt_vrfy.c +++ b/src/third_party/wiredtiger/src/btree/bt_vrfy.c @@ -20,24 +20,35 @@ typedef struct { uint64_t fcnt; /* Progress counter */ + /* Configuration options passed in. */ + wt_timestamp_t stable_timestamp; /* Stable timestamp to verify against if desired */ #define WT_VRFY_DUMP(vs) \ ((vs)->dump_address || (vs)->dump_blocks || (vs)->dump_layout || (vs)->dump_pages) bool dump_address; /* Configure: dump special */ bool dump_blocks; + bool dump_history; bool dump_layout; bool dump_pages; - /* Page layout information */ + bool hs_verify; + + /* Page layout information. */ uint64_t depth, depth_internal[100], depth_leaf[100]; WT_ITEM *tmp1, *tmp2, *tmp3, *tmp4; /* Temporary buffers */ } WT_VSTUFF; static void __verify_checkpoint_reset(WT_VSTUFF *); +static int __verify_col_var_page_hs(WT_SESSION_IMPL *, WT_REF *, WT_VSTUFF *); +static int __verify_key_hs(WT_SESSION_IMPL *, WT_ITEM *, WT_CELL_UNPACK *, WT_VSTUFF *); static int __verify_page_cell(WT_SESSION_IMPL *, WT_REF *, WT_CELL_UNPACK *, WT_VSTUFF *); static int __verify_row_int_key_order( WT_SESSION_IMPL *, WT_PAGE *, WT_REF *, uint32_t, WT_VSTUFF *); static int __verify_row_leaf_key_order(WT_SESSION_IMPL *, WT_REF *, WT_VSTUFF *); +static int __verify_row_leaf_page_hs(WT_SESSION_IMPL *, WT_REF *, WT_VSTUFF *); +static const char *__verify_timestamp_to_pretty_string(wt_timestamp_t, char *ts_string); static int __verify_tree(WT_SESSION_IMPL *, WT_REF *, WT_CELL_UNPACK *, WT_VSTUFF *); +static int __verify_ts_stable_cmp( + WT_SESSION_IMPL *, WT_ITEM *, WT_REF *, uint32_t, wt_timestamp_t, wt_timestamp_t, WT_VSTUFF *); /* * __verify_config -- @@ -47,6 +58,9 @@ static int __verify_config(WT_SESSION_IMPL *session, const char *cfg[], WT_VSTUFF *vs) { WT_CONFIG_ITEM cval; + WT_TXN_GLOBAL *txn_global; + + txn_global = &S2C(session)->txn_global; WT_RET(__wt_config_gets(session, cfg, "dump_address", &cval)); vs->dump_address = cval.val != 0; @@ -54,16 +68,29 @@ __verify_config(WT_SESSION_IMPL *session, const char *cfg[], WT_VSTUFF *vs) WT_RET(__wt_config_gets(session, cfg, "dump_blocks", &cval)); vs->dump_blocks = cval.val != 0; + WT_RET(__wt_config_gets(session, cfg, "dump_history", &cval)); + vs->dump_history = cval.val != 0; + WT_RET(__wt_config_gets(session, cfg, "dump_layout", &cval)); vs->dump_layout = cval.val != 0; WT_RET(__wt_config_gets(session, cfg, "dump_pages", &cval)); vs->dump_pages = cval.val != 0; + WT_RET(__wt_config_gets(session, cfg, "stable_timestamp", &cval)); + vs->stable_timestamp = WT_TS_NONE; /* Ignored unless a value has been set */ + if (cval.val != 0) { + if (!txn_global->has_stable_timestamp) + WT_RET_MSG(session, ENOTSUP, + "cannot verify against the stable timestamp if it has not been set"); + vs->stable_timestamp = txn_global->stable_timestamp; + } + #if !defined(HAVE_DIAGNOSTIC) - if (vs->dump_blocks || vs->dump_pages) + if (vs->dump_blocks || vs->dump_pages || vs->dump_history) WT_RET_MSG(session, ENOTSUP, "the WiredTiger library was not built in diagnostic mode"); #endif + return (0); } @@ -103,11 +130,11 @@ __verify_config_offsets(WT_SESSION_IMPL *session, const char *cfg[], bool *quitp } /* - * __verify_layout -- + * __dump_layout -- * Dump the tree shape. */ static int -__verify_layout(WT_SESSION_IMPL *session, WT_VSTUFF *vs) +__dump_layout(WT_SESSION_IMPL *session, WT_VSTUFF *vs) { size_t i; uint64_t total; @@ -133,6 +160,191 @@ __verify_layout(WT_SESSION_IMPL *session, WT_VSTUFF *vs) } /* + * __verify_col_var_page_hs -- + * Verify a page against the history store. + */ +static int +__verify_col_var_page_hs(WT_SESSION_IMPL *session, WT_REF *ref, WT_VSTUFF *vs) +{ + WT_CELL *cell; + WT_CELL_UNPACK *unpack, _unpack; + WT_COL *cip; + WT_DECL_ITEM(key); + WT_DECL_RET; + WT_PAGE *page; + uint64_t recno, rle; + uint32_t i; + uint8_t *p; + + page = ref->page; + recno = ref->ref_recno; + unpack = &_unpack; + + /* Ensure enough room for a column-store key without checking. */ + WT_ERR(__wt_scr_alloc(session, WT_INTPACK64_MAXSIZE, &key)); + + WT_COL_FOREACH (page, cip, i) { + p = key->mem; + WT_ERR(__wt_vpack_uint(&p, 0, recno)); + key->size = WT_PTRDIFF(p, key->data); + + cell = WT_COL_PTR(page, cip); + __wt_cell_unpack(session, page, cell, unpack); + rle = __wt_cell_rle(unpack); + +#ifdef HAVE_DIAGNOSTIC + /* Optionally dump historical time pairs and values in debug mode. */ + if (vs->dump_history) { + WT_ERR(__wt_msg(session, "\tK {%" PRIu64 " %" PRIu64 "}", recno, rle)); + WT_ERR(__wt_debug_key_value(session, NULL, unpack)); + } +#endif + + WT_ERR(__verify_key_hs(session, key, unpack, vs)); + recno += rle; + } + +err: + __wt_scr_free(session, &key); + + return (ret); +} + +/* + * __verify_row_leaf_page_hs -- + * Verify a page against the history store. + */ +static int +__verify_row_leaf_page_hs(WT_SESSION_IMPL *session, WT_REF *ref, WT_VSTUFF *vs) +{ + WT_CELL_UNPACK *unpack, _unpack; + WT_DECL_ITEM(key); + WT_DECL_RET; + WT_PAGE *page; + WT_ROW *rip; + uint32_t i; + + page = ref->page; + unpack = &_unpack; + + WT_RET(__wt_scr_alloc(session, 256, &key)); + + WT_ROW_FOREACH (page, rip, i) { + WT_ERR(__wt_row_leaf_key(session, page, rip, key, false)); + __wt_row_leaf_value_cell(session, page, rip, NULL, unpack); + +#ifdef HAVE_DIAGNOSTIC + /* Optionally dump historical time pairs and values in debug mode. */ + if (vs->dump_history) + WT_ERR(__wt_debug_key_value(session, key, unpack)); +#endif + + WT_ERR(__verify_key_hs(session, key, unpack, vs)); + } + +err: + __wt_scr_free(session, &key); + return (ret); +} + +/* + * __verify_key_hs -- + * Verify a key against the history store. The unpack denotes the data store's timestamp range + * information and is used for verifying timestamp range overlaps. + */ +static int +__verify_key_hs(WT_SESSION_IMPL *session, WT_ITEM *key, WT_CELL_UNPACK *unpack, WT_VSTUFF *vs) +{ + WT_BTREE *btree; + WT_CURSOR *hs_cursor; + WT_DECL_ITEM(hs_key); + WT_DECL_RET; + wt_timestamp_t newer_start_ts, older_start_ts, older_stop_ts; + uint64_t hs_counter; + uint32_t hs_btree_id, session_flags; + int cmp, exact; + char ts_string[2][WT_TS_INT_STRING_SIZE]; + bool is_owner; + + btree = S2BT(session); + hs_cursor = NULL; + hs_btree_id = btree->id; + /* + * Set the data store timestamp and transactions to initiate timestamp range verification. Since + * transaction-ids are wiped out on start, we could possibly have a start txn-id of WT_TXN_NONE, + * in which case we initialize our newest with the max txn-id. + */ + newer_start_ts = unpack->start_ts; + session_flags = 0; + older_stop_ts = 0; + is_owner = false; + + WT_ERR(__wt_scr_alloc(session, 0, &hs_key)); + + /* + * Open a history store cursor positioned at the end of the data store key (the newest record) + * and iterate backwards until we reach a different key or btree. + */ + WT_ERR(__wt_hs_cursor(session, &session_flags, &is_owner)); + hs_cursor = session->hs_cursor; + hs_cursor->set_key(hs_cursor, hs_btree_id, key, WT_TS_MAX, UINT64_MAX); + WT_ERR(hs_cursor->search_near(hs_cursor, &exact)); + + /* If we jumped to the next key, go back to the previous key. */ + if (exact > 0) + WT_ERR(hs_cursor->prev(hs_cursor)); + + for (; ret == 0; ret = hs_cursor->prev(hs_cursor)) { + WT_ERR(hs_cursor->get_key(hs_cursor, &hs_btree_id, hs_key, &older_start_ts, &hs_counter)); + + if (hs_btree_id != btree->id) + break; + + WT_ERR(__wt_compare(session, NULL, hs_key, key, &cmp)); + if (cmp != 0) + break; + +#ifdef HAVE_DIAGNOSTIC + /* Optionally dump historical time pairs and values in debug mode. */ + if (vs->dump_history) + WT_ERR(__wt_debug_cursor_hs(session, hs_cursor)); +#else + WT_UNUSED(vs); +#endif + + /* Verify that the newer record's start is later than the older record's stop. */ + if (newer_start_ts < older_stop_ts) { + WT_ERR_MSG(session, WT_ERROR, + "In the Btree %" PRIu32 + ", Key %s has a overlap of " + "timestamp ranges between history store stop timestamp %s being " + "newer than a more recent timestamp range having start timestamp %s", + hs_btree_id, __wt_buf_set_printable(session, hs_key->data, hs_key->size, vs->tmp1), + __verify_timestamp_to_pretty_string(older_stop_ts, ts_string[0]), + __verify_timestamp_to_pretty_string(newer_start_ts, ts_string[1])); + } + /* + * Since we are iterating from newer to older, the current older record becomes the newer + * for the next round of verification. + */ + newer_start_ts = older_start_ts; + + WT_ERR(__verify_ts_stable_cmp(session, key, NULL, 0, older_start_ts, older_stop_ts, vs)); + } + WT_ERR_NOTFOUND_OK(ret); + +err: + /* It is okay to have not found the key. */ + if (ret == WT_NOTFOUND) + ret = 0; + + __wt_scr_free(session, &hs_key); + WT_TRET(__wt_hs_cursor_close(session, session_flags, is_owner)); + + return (ret); +} + +/* * __wt_verify -- * Verify a file. */ @@ -223,7 +435,7 @@ __wt_verify(WT_SESSION_IMPL *session, const char *cfg[]) * Create a fake, unpacked parent cell for the tree based on the checkpoint information. */ memset(&addr_unpack, 0, sizeof(addr_unpack)); - addr_unpack.newest_durable_ts = ckpt->newest_durable_ts; + addr_unpack.newest_stop_durable_ts = ckpt->newest_durable_ts; addr_unpack.oldest_start_ts = ckpt->oldest_start_ts; addr_unpack.oldest_start_txn = ckpt->oldest_start_txn; addr_unpack.newest_stop_ts = ckpt->newest_stop_ts; @@ -259,7 +471,7 @@ __wt_verify(WT_SESSION_IMPL *session, const char *cfg[]) /* Display the tree shape. */ if (vs->dump_layout) - WT_ERR(__verify_layout(session, vs)); + WT_ERR(__dump_layout(session, vs)); } done: @@ -310,17 +522,23 @@ __verify_checkpoint_reset(WT_VSTUFF *vs) static const char * __verify_addr_string(WT_SESSION_IMPL *session, WT_REF *ref, WT_ITEM *buf) { - size_t addr_size; - const uint8_t *addr; + WT_ADDR_COPY addr; + WT_DECL_ITEM(tmp); + WT_DECL_RET; + char tp_string[2][WT_TP_STRING_SIZE]; - if (__wt_ref_is_root(ref)) { - buf->data = "[Root]"; - buf->size = strlen("[Root]"); - return (buf->data); - } + if (__wt_ref_addr_copy(session, ref, &addr)) { + WT_ERR(__wt_scr_alloc(session, 0, &tmp)); + WT_ERR(__wt_buf_fmt(session, buf, "%s %s,%s", + __wt_addr_string(session, addr.addr, addr.size, tmp), + __wt_time_pair_to_string(addr.oldest_start_ts, addr.oldest_start_txn, tp_string[0]), + __wt_time_pair_to_string(addr.newest_stop_ts, addr.newest_stop_txn, tp_string[1]))); + } else + WT_ERR(__wt_buf_fmt(session, buf, "%s -/-,-/-", __wt_addr_string(session, NULL, 0, tmp))); - __wt_ref_info(session, ref, &addr, &addr_size, NULL); - return (__wt_addr_string(session, addr, addr_size, buf)); +err: + __wt_scr_free(session, &tmp); + return (buf->data); } /* @@ -332,7 +550,7 @@ __verify_addr_ts(WT_SESSION_IMPL *session, WT_REF *ref, WT_CELL_UNPACK *unpack, { char ts_string[2][WT_TS_INT_STRING_SIZE]; - if (unpack->newest_stop_ts == WT_TS_NONE) + if (unpack->oldest_start_ts != WT_TS_NONE && unpack->newest_stop_ts == WT_TS_NONE) WT_RET_MSG(session, WT_ERROR, "internal page reference at %s has a newest stop " "timestamp of 0", @@ -344,11 +562,6 @@ __verify_addr_ts(WT_SESSION_IMPL *session, WT_REF *ref, WT_CELL_UNPACK *unpack, __verify_addr_string(session, ref, vs->tmp1), __wt_timestamp_to_string(unpack->oldest_start_ts, ts_string[0]), __wt_timestamp_to_string(unpack->newest_stop_ts, ts_string[1])); - if (unpack->newest_stop_txn == WT_TXN_NONE) - WT_RET_MSG(session, WT_ERROR, - "internal page reference at %s has a newest stop " - "transaction of 0", - __verify_addr_string(session, ref, vs->tmp1)); if (unpack->oldest_start_txn > unpack->newest_stop_txn) WT_RET_MSG(session, WT_ERROR, "internal page reference at %s has an oldest start " @@ -361,6 +574,125 @@ __verify_addr_ts(WT_SESSION_IMPL *session, WT_REF *ref, WT_CELL_UNPACK *unpack, } /* + * __wt_verify_history_store_tree -- + * Verify the history store. There can't be an entry in the history store without having the + * latest value for the respective key in the data store. If given a uri, limit the verification + * to the corresponding btree. + */ +int +__wt_verify_history_store_tree(WT_SESSION_IMPL *session, const char *uri) +{ + WT_CURSOR *cursor, *data_cursor; + WT_DECL_ITEM(tmp); + WT_DECL_RET; + WT_ITEM hs_key, prev_hs_key; + wt_timestamp_t hs_start_ts; + uint64_t hs_counter; + uint32_t btree_id, btree_id_given_uri, session_flags, prev_btree_id; + int exact, cmp; + char *uri_itr; + bool is_owner; + + session_flags = 0; + data_cursor = NULL; + WT_CLEAR(prev_hs_key); + WT_CLEAR(hs_key); + btree_id_given_uri = 0; /* [-Wconditional-uninitialized] */ + prev_btree_id = 0; /* [-Wconditional-uninitialized] */ + uri_itr = NULL; + + WT_RET(__wt_hs_cursor(session, &session_flags, &is_owner)); + cursor = session->hs_cursor; + + /* + * If a uri has been provided, limit verification to the corresponding btree by jumping to the + * first record for that btree in the history store. Otherwise scan the whole history store. + */ + if (uri != NULL) { + ret = __wt_metadata_uri_to_btree_id(session, uri, &btree_id_given_uri); + if (ret != 0) + WT_ERR_MSG(session, ret, "Unable to locate the URI %s in the metadata file", uri); + + /* + * Position the cursor at the first record of the specified btree, or one after. It is + * possible there are no records in the history store for this btree. + */ + cursor->set_key(cursor, btree_id_given_uri, &hs_key, 0, 0, 0, 0); + ret = cursor->search_near(cursor, &exact); + if (ret == 0 && exact < 0) + ret = cursor->next(cursor); + } else + ret = cursor->next(cursor); + + /* We have the history store cursor positioned at the first record that we want to verify. */ + for (; ret == 0; ret = cursor->next(cursor)) { + WT_ERR(cursor->get_key(cursor, &btree_id, &hs_key, &hs_start_ts, &hs_counter)); + + /* When limiting our verification to a uri, bail out if the btree-id doesn't match. */ + if (uri != NULL && btree_id != btree_id_given_uri) + break; + + /* + * Keep track of the previous comparison. The history store is stored in order, so we can + * avoid redundant comparisons. Previous btree ID isn't set, until data cursor is open. + */ + if (data_cursor == NULL || (prev_btree_id != btree_id)) { + /* + * Check whether this btree-id exists in the metadata. We do that by finding what uri + * this btree belongs to. Using this URI, verify the history store key with the data + * store. + */ + if (data_cursor != NULL) { + WT_ERR(data_cursor->close(data_cursor)); + /* Setting data_cursor to null, to avoid double free */ + data_cursor = NULL; + } + /* + * Using the btree-id find the metadata entry and extract the URI for this btree. Don't + * forget to free the copy of the URI returned. + */ + __wt_free(session, uri_itr); + ret = __wt_metadata_btree_id_to_uri(session, btree_id, &uri_itr); + if (ret != 0) { + WT_ERR(__wt_scr_alloc(session, 0, &tmp)); + WT_ERR_MSG(session, ret, "Unable to find btree-id %" PRIu32 + " in the metadata file for the associated " + "history store key %s", + btree_id, __wt_buf_set_printable(session, hs_key.data, hs_key.size, tmp)); + } + + WT_ERR(__wt_open_cursor(session, uri_itr, NULL, NULL, &data_cursor)); + F_SET(data_cursor, WT_CURSOR_RAW_OK); + } else { + WT_ERR(__wt_compare(session, NULL, &hs_key, &prev_hs_key, &cmp)); + if (cmp == 0) + continue; + } + WT_ERR(__wt_buf_set(session, &prev_hs_key, hs_key.data, hs_key.size)); + prev_btree_id = btree_id; + + data_cursor->set_key(data_cursor, &hs_key); + ret = data_cursor->search(data_cursor); + if (ret == WT_NOTFOUND) { + WT_ERR(__wt_scr_alloc(session, 0, &tmp)); + WT_ERR_MSG(session, WT_NOTFOUND, + "In the URI %s, the associated history store key %s cannot be found in the data " + "store", + uri_itr, __wt_buf_set_printable(session, hs_key.data, hs_key.size, tmp)); + } + WT_ERR(ret); + } + WT_ERR_NOTFOUND_OK(ret); +err: + if (data_cursor != NULL) + WT_TRET(data_cursor->close(data_cursor)); + WT_TRET(__wt_hs_cursor_close(session, session_flags, is_owner)); + __wt_scr_free(session, &tmp); + __wt_free(session, uri_itr); + return (ret); +} + +/* * __verify_tree -- * Verify a tree, recursively descending through it in depth-first fashion. The page argument * was physically verified (so we know it's correctly formed), and the in-memory version built. @@ -378,22 +710,26 @@ __verify_tree(WT_SESSION_IMPL *session, WT_REF *ref, WT_CELL_UNPACK *addr_unpack WT_REF *child_ref; uint64_t recno; uint32_t entry, i; + bool enable_hs_verify; bm = S2BT(session)->bm; page = ref->page; + /* Temporarily disable as MongoDB tests are timing out. Re-enable with WT-5796. */ + enable_hs_verify = false; + unpack = &_unpack; __wt_verbose(session, WT_VERB_VERIFY, "%s %s", __verify_addr_string(session, ref, vs->tmp1), __wt_page_type_string(page->type)); - /* Optionally dump the address. */ + /* Optionally dump address information. */ if (vs->dump_address) WT_RET(__wt_msg(session, "%s %s", __verify_addr_string(session, ref, vs->tmp1), __wt_page_type_string(page->type))); /* Track the shape of the tree. */ - if (WT_PAGE_IS_INTERNAL(page)) + if (F_ISSET(ref, WT_REF_FLAG_INTERNAL)) ++vs->depth_internal[WT_MIN(vs->depth, WT_ELEMENTS(vs->depth_internal) - 1)]; else ++vs->depth_leaf[WT_MIN(vs->depth, WT_ELEMENTS(vs->depth_internal) - 1)]; @@ -476,6 +812,23 @@ recno_chk: break; } + /* + * History store checks. Ensure continuity between the data store and history store based on + * keys in leaf/var pages. + * + * Temporarily disable as MongoDB tests are timing out. Re-enable with WT-5796. + */ + if (enable_hs_verify) { + switch (page->type) { + case WT_PAGE_ROW_LEAF: + WT_RET(__verify_row_leaf_page_hs(session, ref, vs)); + break; + case WT_PAGE_COL_VAR: + WT_RET(__verify_col_var_page_hs(session, ref, vs)); + break; + } + } + /* Compare the address type against the page type. */ switch (page->type) { case WT_PAGE_COL_FIX: @@ -717,40 +1070,66 @@ __verify_ts_addr_cmp(WT_SESSION_IMPL *session, WT_REF *ref, uint32_t cell_num, c wt_timestamp_t ts1, const char *ts2_name, wt_timestamp_t ts2, bool gt, WT_VSTUFF *vs) { char ts_string[2][WT_TS_INT_STRING_SIZE]; - const char *ts1_bp, *ts2_bp; if (gt && ts1 >= ts2) return (0); if (!gt && ts1 <= ts2) return (0); - switch (ts1) { - case WT_TS_MAX: - ts1_bp = "WT_TS_MAX"; - break; - case WT_TS_NONE: - ts1_bp = "WT_TS_NONE"; - break; - default: - ts1_bp = __wt_timestamp_to_string(ts1, ts_string[0]); - break; - } - switch (ts2) { - case WT_TS_MAX: - ts2_bp = "WT_TS_MAX"; - break; - case WT_TS_NONE: - ts2_bp = "WT_TS_NONE"; - break; - default: - ts2_bp = __wt_timestamp_to_string(ts2, ts_string[1]); - break; - } WT_RET_MSG(session, WT_ERROR, "cell %" PRIu32 " on page at %s failed verification with %s " "timestamp of %s, %s the parent's %s timestamp of %s", - cell_num, __verify_addr_string(session, ref, vs->tmp1), ts1_name, ts1_bp, - gt ? "less than" : "greater than", ts2_name, ts2_bp); + cell_num, __verify_addr_string(session, ref, vs->tmp1), ts1_name, + __verify_timestamp_to_pretty_string(ts1, ts_string[0]), gt ? "less than" : "greater than", + ts2_name, __verify_timestamp_to_pretty_string(ts2, ts_string[1])); +} + +/* + * __verify_ts_stable_cmp -- + * Verify that a pair of start and stop timestamps are valid against the global stable + * timestamp. Takes in either a key for history store timestamps or a ref and cell number. + */ +static int +__verify_ts_stable_cmp(WT_SESSION_IMPL *session, WT_ITEM *key, WT_REF *ref, uint32_t cell_num, + wt_timestamp_t start_ts, wt_timestamp_t stop_ts, WT_VSTUFF *vs) +{ + WT_BTREE *btree; + WT_DECL_RET; + char tp_string[2][WT_TP_STRING_SIZE]; + bool start; + + btree = S2BT(session); + start = true; + + /* Only verify if -S option is specified. */ + if (vs->stable_timestamp == WT_TS_NONE) + return (0); + + if (start_ts != WT_TS_NONE && start_ts > vs->stable_timestamp) + goto msg; + + if (stop_ts != WT_TS_MAX && stop_ts > vs->stable_timestamp) { + start = false; + goto msg; + } + + return (ret); + +msg: + WT_ASSERT(session, ref != NULL || key != NULL); + if (ref != NULL) + WT_RET(__wt_buf_fmt(session, vs->tmp1, "cell %" PRIu32 " on page at %s", cell_num, + __verify_addr_string(session, ref, vs->tmp2))); + else if (key != NULL) + WT_RET(__wt_buf_fmt(session, vs->tmp1, "Value in history store for key {%s}", + __wt_key_string(session, key->data, key->size, btree->key_format, vs->tmp2))); + + WT_RET_MSG(session, WT_ERROR, + "%s has failed verification with a %s" + " timestamp of %s greater than the stable_timestamp of %s", + (char *)vs->tmp1->data, start ? "start" : "stop", + __wt_timestamp_to_string(start ? start_ts : stop_ts, tp_string[0]), + __wt_timestamp_to_string(vs->stable_timestamp, tp_string[1])); } /* @@ -760,12 +1139,18 @@ __verify_ts_addr_cmp(WT_SESSION_IMPL *session, WT_REF *ref, uint32_t cell_num, c static int __verify_txn_addr_cmp(WT_SESSION_IMPL *session, WT_REF *ref, uint32_t cell_num, const char *txn1_name, uint64_t txn1, const char *txn2_name, uint64_t txn2, bool gt, - WT_VSTUFF *vs) + const WT_PAGE_HEADER *dsk, WT_VSTUFF *vs) { if (gt && txn1 >= txn2) return (0); if (!gt && txn1 <= txn2) return (0); + /* + * If we unpack a value that was written as part of a previous startup generation, we set start + * id to "none" and the stop id to "max" so we need an exception here. + */ + if (dsk->write_gen <= S2C(session)->base_write_gen) + return (0); WT_RET_MSG(session, WT_ERROR, "cell %" PRIu32 " on page at %s failed verification with %s " @@ -777,6 +1162,29 @@ __verify_txn_addr_cmp(WT_SESSION_IMPL *session, WT_REF *ref, uint32_t cell_num, } /* + * __verify_timestamp_to_pretty_string -- + * Convert a timestamp to a pretty string, utilizes existing timestamp to string function. + */ +static const char * +__verify_timestamp_to_pretty_string(wt_timestamp_t ts, char *ts_string) +{ + const char *ts_bp; + + switch (ts) { + case WT_TS_MAX: + ts_bp = "WT_TS_MAX"; + break; + case WT_TS_NONE: + ts_bp = "WT_TS_NONE"; + break; + default: + ts_bp = __wt_timestamp_to_string(ts, ts_string); + break; + } + return ts_bp; +} + +/* * __verify_page_cell -- * Verify the cells on the page. */ @@ -829,16 +1237,11 @@ __verify_page_cell( case WT_CELL_ADDR_INT: case WT_CELL_ADDR_LEAF: case WT_CELL_ADDR_LEAF_NO: - if (unpack.newest_stop_ts == WT_TS_NONE) + if (unpack.oldest_start_ts != WT_TS_NONE && unpack.newest_stop_ts == WT_TS_NONE) WT_RET_MSG(session, WT_ERROR, "cell %" PRIu32 " on page at %s has a " "newest stop timestamp of 0", cell_num - 1, __verify_addr_string(session, ref, vs->tmp1)); - if (unpack.newest_stop_txn == WT_TXN_NONE) - WT_RET_MSG(session, WT_ERROR, "cell %" PRIu32 - " on page at %s has a " - "newest stop transaction of 0", - cell_num - 1, __verify_addr_string(session, ref, vs->tmp1)); if (unpack.oldest_start_ts > unpack.newest_stop_ts) WT_RET_MSG(session, WT_ERROR, "cell %" PRIu32 " on page at %s has an " @@ -858,24 +1261,28 @@ __verify_page_cell( unpack.oldest_start_txn, unpack.newest_stop_txn); } + /* FIXME-prepare-support: check newest start durable timestamp as well. */ WT_RET(__verify_ts_addr_cmp(session, ref, cell_num - 1, "newest durable", - unpack.newest_durable_ts, "newest durable", addr_unpack->newest_durable_ts, false, - vs)); + unpack.newest_stop_durable_ts, "newest durable", addr_unpack->newest_stop_durable_ts, + false, vs)); WT_RET(__verify_ts_addr_cmp(session, ref, cell_num - 1, "oldest start", unpack.oldest_start_ts, "oldest start", addr_unpack->oldest_start_ts, true, vs)); WT_RET(__verify_txn_addr_cmp(session, ref, cell_num - 1, "oldest start", - unpack.oldest_start_txn, "oldest start", addr_unpack->oldest_start_txn, true, vs)); + unpack.oldest_start_txn, "oldest start", addr_unpack->oldest_start_txn, true, dsk, + vs)); WT_RET(__verify_ts_addr_cmp(session, ref, cell_num - 1, "newest stop", unpack.newest_stop_ts, "newest stop", addr_unpack->newest_stop_ts, false, vs)); WT_RET(__verify_txn_addr_cmp(session, ref, cell_num - 1, "newest stop", - unpack.newest_stop_txn, "newest stop", addr_unpack->newest_stop_txn, false, vs)); + unpack.newest_stop_txn, "newest stop", addr_unpack->newest_stop_txn, false, dsk, vs)); + WT_RET(__verify_ts_stable_cmp( + session, NULL, ref, cell_num - 1, addr_unpack->start_ts, addr_unpack->stop_ts, vs)); break; case WT_CELL_DEL: case WT_CELL_VALUE: case WT_CELL_VALUE_COPY: case WT_CELL_VALUE_OVFL: case WT_CELL_VALUE_SHORT: - if (unpack.stop_ts == WT_TS_NONE) + if (unpack.start_ts != WT_TS_NONE && unpack.stop_ts == WT_TS_NONE) WT_RET_MSG(session, WT_ERROR, "cell %" PRIu32 " on page at %s has a stop " "timestamp of 0", @@ -888,11 +1295,6 @@ __verify_page_cell( cell_num - 1, __verify_addr_string(session, ref, vs->tmp1), __wt_timestamp_to_string(unpack.start_ts, ts_string[0]), __wt_timestamp_to_string(unpack.stop_ts, ts_string[1])); - if (unpack.stop_txn == WT_TXN_NONE) - WT_RET_MSG(session, WT_ERROR, "cell %" PRIu32 - " on page at %s has a stop " - "transaction of 0", - cell_num - 1, __verify_addr_string(session, ref, vs->tmp1)); if (unpack.start_txn > unpack.stop_txn) WT_RET_MSG(session, WT_ERROR, "cell %" PRIu32 " on page at %s has a " @@ -905,11 +1307,13 @@ __verify_page_cell( WT_RET(__verify_ts_addr_cmp(session, ref, cell_num - 1, "start", unpack.start_ts, "oldest start", addr_unpack->oldest_start_ts, true, vs)); WT_RET(__verify_txn_addr_cmp(session, ref, cell_num - 1, "start", unpack.start_txn, - "oldest start", addr_unpack->oldest_start_txn, true, vs)); + "oldest start", addr_unpack->oldest_start_txn, true, dsk, vs)); WT_RET(__verify_ts_addr_cmp(session, ref, cell_num - 1, "stop", unpack.stop_ts, "newest stop", addr_unpack->newest_stop_ts, false, vs)); WT_RET(__verify_txn_addr_cmp(session, ref, cell_num - 1, "stop", unpack.stop_txn, - "newest stop", addr_unpack->newest_stop_txn, false, vs)); + "newest stop", addr_unpack->newest_stop_txn, false, dsk, vs)); + WT_RET(__verify_ts_stable_cmp( + session, NULL, ref, cell_num - 1, unpack.start_ts, unpack.stop_ts, vs)); break; } } diff --git a/src/third_party/wiredtiger/src/btree/bt_vrfy_dsk.c b/src/third_party/wiredtiger/src/btree/bt_vrfy_dsk.c index 55d7b77778e..6313bfa45f5 100644 --- a/src/third_party/wiredtiger/src/btree/bt_vrfy_dsk.c +++ b/src/third_party/wiredtiger/src/btree/bt_vrfy_dsk.c @@ -110,8 +110,6 @@ __wt_verify_dsk_image(WT_SESSION_IMPL *session, const char *tag, const WT_PAGE_H } if (LF_ISSET(WT_PAGE_ENCRYPTED)) LF_CLR(WT_PAGE_ENCRYPTED); - if (LF_ISSET(WT_PAGE_LAS_UPDATE)) - LF_CLR(WT_PAGE_LAS_UPDATE); if (flags != 0) WT_RET_VRFY(session, "page at %s has invalid flags set: 0x%" PRIx8, tag, flags); @@ -119,15 +117,6 @@ __wt_verify_dsk_image(WT_SESSION_IMPL *session, const char *tag, const WT_PAGE_H if (dsk->unused != 0) WT_RET_VRFY(session, "page at %s has non-zero unused page header bytes", tag); - /* Check the page version. */ - switch (dsk->version) { - case WT_PAGE_VERSION_ORIG: - case WT_PAGE_VERSION_TS: - break; - default: - WT_RET_VRFY(session, "page at %s has an invalid version of %" PRIu8, tag, dsk->version); - } - /* * Any bytes after the data chunk should be nul bytes; ignore if the size is 0, that allows easy * checking of disk images where we don't have the size. @@ -238,12 +227,19 @@ __verify_dsk_ts_addr_cmp(WT_SESSION_IMPL *session, uint32_t cell_num, const char */ static int __verify_dsk_txn_addr_cmp(WT_SESSION_IMPL *session, uint32_t cell_num, const char *txn1_name, - uint64_t txn1, const char *txn2_name, uint64_t txn2, bool gt, const char *tag) + uint64_t txn1, const char *txn2_name, uint64_t txn2, bool gt, const char *tag, + const WT_PAGE_HEADER *dsk) { if (gt && txn1 >= txn2) return (0); if (!gt && txn1 <= txn2) return (0); + /* + * If we unpack a value that was written as part of a previous startup generation, it may have a + * later stop time pair than its parent. + */ + if (dsk->write_gen <= S2C(session)->base_write_gen) + return (0); WT_RET_MSG(session, WT_ERROR, "cell %" PRIu32 " on page at %s failed verification with %s " @@ -259,7 +255,7 @@ __verify_dsk_txn_addr_cmp(WT_SESSION_IMPL *session, uint32_t cell_num, const cha */ static int __verify_dsk_validity(WT_SESSION_IMPL *session, WT_CELL_UNPACK *unpack, uint32_t cell_num, - WT_ADDR *addr, const char *tag) + WT_ADDR *addr, const char *tag, const WT_PAGE_HEADER *dsk) { char ts_string[2][WT_TS_INT_STRING_SIZE]; @@ -277,16 +273,11 @@ __verify_dsk_validity(WT_SESSION_IMPL *session, WT_CELL_UNPACK *unpack, uint32_t case WT_CELL_ADDR_INT: case WT_CELL_ADDR_LEAF: case WT_CELL_ADDR_LEAF_NO: - if (unpack->newest_stop_ts == WT_TS_NONE) + if (unpack->oldest_start_ts != WT_TS_NONE && unpack->newest_stop_ts == WT_TS_NONE) WT_RET_VRFY(session, "cell %" PRIu32 " on page at %s has a newest stop " "timestamp of 0", cell_num - 1, tag); - if (unpack->newest_stop_txn == WT_TXN_NONE) - WT_RET_VRFY(session, "cell %" PRIu32 - " on page at %s has a newest stop " - "transaction of 0", - cell_num - 1, tag); if (unpack->oldest_start_ts > unpack->newest_stop_ts) WT_RET_VRFY(session, "cell %" PRIu32 " on page at %s has an oldest " @@ -305,16 +296,17 @@ __verify_dsk_validity(WT_SESSION_IMPL *session, WT_CELL_UNPACK *unpack, uint32_t if (addr == NULL) break; + /* FIXME-prepare-support: check newest start durable timestamp as well. */ WT_RET(__verify_dsk_ts_addr_cmp(session, cell_num - 1, "newest durable", - unpack->newest_durable_ts, "newest durable", addr->newest_durable_ts, false, tag)); + unpack->newest_stop_durable_ts, "newest durable", addr->stop_durable_ts, false, tag)); WT_RET(__verify_dsk_ts_addr_cmp(session, cell_num - 1, "oldest start", unpack->oldest_start_ts, "oldest start", addr->oldest_start_ts, true, tag)); WT_RET(__verify_dsk_txn_addr_cmp(session, cell_num - 1, "oldest start", - unpack->oldest_start_txn, "oldest start", addr->oldest_start_txn, true, tag)); + unpack->oldest_start_txn, "oldest start", addr->oldest_start_txn, true, tag, dsk)); WT_RET(__verify_dsk_ts_addr_cmp(session, cell_num - 1, "newest stop", unpack->newest_stop_ts, "newest stop", addr->newest_stop_ts, false, tag)); WT_RET(__verify_dsk_txn_addr_cmp(session, cell_num - 1, "newest stop", - unpack->newest_stop_txn, "newest stop", addr->newest_stop_txn, false, tag)); + unpack->newest_stop_txn, "newest stop", addr->newest_stop_txn, false, tag, dsk)); break; case WT_CELL_DEL: case WT_CELL_VALUE: @@ -322,7 +314,7 @@ __verify_dsk_validity(WT_SESSION_IMPL *session, WT_CELL_UNPACK *unpack, uint32_t case WT_CELL_VALUE_OVFL: case WT_CELL_VALUE_OVFL_RM: case WT_CELL_VALUE_SHORT: - if (unpack->stop_ts == WT_TS_NONE) + if (unpack->start_ts != WT_TS_NONE && unpack->stop_ts == WT_TS_NONE) WT_RET_VRFY(session, "cell %" PRIu32 " on page at %s has a stop " "timestamp of 0", @@ -333,11 +325,6 @@ __verify_dsk_validity(WT_SESSION_IMPL *session, WT_CELL_UNPACK *unpack, uint32_t "timestamp %s newer than its stop timestamp %s", cell_num - 1, tag, __wt_timestamp_to_string(unpack->start_ts, ts_string[0]), __wt_timestamp_to_string(unpack->stop_ts, ts_string[1])); - if (unpack->stop_txn == WT_TXN_NONE) - WT_RET_VRFY(session, "cell %" PRIu32 - " on page at %s has a stop " - "transaction of 0", - cell_num - 1, tag); if (unpack->start_txn > unpack->stop_txn) WT_RET_VRFY(session, "cell %" PRIu32 " on page at %s has a start " @@ -352,11 +339,11 @@ __verify_dsk_validity(WT_SESSION_IMPL *session, WT_CELL_UNPACK *unpack, uint32_t WT_RET(__verify_dsk_ts_addr_cmp(session, cell_num - 1, "start", unpack->start_ts, "oldest start", addr->oldest_start_ts, true, tag)); WT_RET(__verify_dsk_txn_addr_cmp(session, cell_num - 1, "start", unpack->start_txn, - "oldest start", addr->oldest_start_txn, true, tag)); + "oldest start", addr->oldest_start_txn, true, tag, dsk)); WT_RET(__verify_dsk_ts_addr_cmp(session, cell_num - 1, "stop", unpack->stop_ts, "newest stop", addr->newest_stop_ts, false, tag)); WT_RET(__verify_dsk_txn_addr_cmp(session, cell_num - 1, "stop", unpack->stop_txn, - "newest stop", addr->newest_stop_txn, false, tag)); + "newest stop", addr->newest_stop_txn, false, tag, dsk)); break; } @@ -471,7 +458,7 @@ __verify_dsk_row( } /* Check the validity window. */ - WT_ERR(__verify_dsk_validity(session, unpack, cell_num, addr, tag)); + WT_ERR(__verify_dsk_validity(session, unpack, cell_num, addr, tag, dsk)); /* Check if any referenced item has an invalid address. */ switch (cell_type) { @@ -666,7 +653,7 @@ __verify_dsk_col_int( WT_RET(__err_cell_type(session, cell_num, tag, unpack->type, dsk->type)); /* Check the validity window. */ - WT_RET(__verify_dsk_validity(session, unpack, cell_num, addr, tag)); + WT_RET(__verify_dsk_validity(session, unpack, cell_num, addr, tag, dsk)); /* Check if any referenced item is entirely in the file. */ ret = bm->addr_invalid(bm, session, unpack->data, unpack->size); @@ -748,7 +735,7 @@ __verify_dsk_col_var( cell_type = unpack->type; /* Check the validity window. */ - WT_RET(__verify_dsk_validity(session, unpack, cell_num, addr, tag)); + WT_RET(__verify_dsk_validity(session, unpack, cell_num, addr, tag, dsk)); /* Check if any referenced item is entirely in the file. */ if (cell_type == WT_CELL_VALUE_OVFL) { diff --git a/src/third_party/wiredtiger/src/btree/bt_walk.c b/src/third_party/wiredtiger/src/btree/bt_walk.c index 67e9e3e3b82..d5935712c9f 100644 --- a/src/third_party/wiredtiger/src/btree/bt_walk.c +++ b/src/third_party/wiredtiger/src/btree/bt_walk.c @@ -73,19 +73,6 @@ found: } /* - * __ref_is_leaf -- - * Check if a reference is for a leaf page. - */ -static inline bool -__ref_is_leaf(WT_SESSION_IMPL *session, WT_REF *ref) -{ - bool is_leaf; - - __wt_ref_info_lock(session, ref, NULL, NULL, &is_leaf); - return (is_leaf); -} - -/* * __ref_ascend -- * Ascend the tree one level. */ @@ -258,7 +245,8 @@ __tree_walk_internal(WT_SESSION_IMPL *session, WT_REF **refp, uint64_t *walkcntp WT_PAGE_INDEX *pindex; WT_REF *couple, *ref, *ref_orig; uint64_t restart_sleep, restart_yield; - uint32_t current_state, slot; + uint32_t slot; + uint8_t current_state; bool empty_internal, prev, skip; btree = S2BT(session); @@ -443,12 +431,7 @@ descend: /* * Only look at unlocked pages in memory: fast-path some common cases. */ - if (LF_ISSET(WT_READ_NO_WAIT) && current_state != WT_REF_MEM && - current_state != WT_REF_LIMBO) - break; - - /* Skip lookaside pages if not requested. */ - if (current_state == WT_REF_LOOKASIDE && !LF_ISSET(WT_READ_LOOKASIDE)) + if (LF_ISSET(WT_READ_NO_WAIT) && current_state != WT_REF_MEM) break; } else if (LF_ISSET(WT_READ_TRUNCATE)) { /* @@ -482,7 +465,7 @@ descend: couple = NULL; /* Return leaf pages to our caller. */ - if (!WT_PAGE_IS_INTERNAL(ref->page)) { + if (F_ISSET(ref, WT_REF_FLAG_LEAF)) { *refp = ref; goto done; } @@ -589,7 +572,7 @@ __tree_walk_skip_count_callback(WT_SESSION_IMPL *session, WT_REF *ref, void *con */ if (ref->state == WT_REF_DELETED && __wt_delete_page_skip(session, ref, false)) *skipp = true; - else if (*skipleafcntp > 0 && __ref_is_leaf(session, ref)) { + else if (*skipleafcntp > 0 && F_ISSET(ref, WT_REF_FLAG_LEAF)) { --*skipleafcntp; *skipp = true; } else diff --git a/src/third_party/wiredtiger/src/btree/col_modify.c b/src/third_party/wiredtiger/src/btree/col_modify.c index 8262c0eb227..687013be783 100644 --- a/src/third_party/wiredtiger/src/btree/col_modify.c +++ b/src/third_party/wiredtiger/src/btree/col_modify.c @@ -38,6 +38,14 @@ __wt_col_modify(WT_CURSOR_BTREE *cbt, uint64_t recno, const WT_ITEM *value, WT_U upd = upd_arg; append = logged = false; + /* + * We should have EITHER: + * - A full update list to instantiate with. + * - An update to append the existing update list with. + * - A key/value pair to create an update with and append to the update list. + */ + WT_ASSERT(session, (value == NULL && upd_arg != NULL) || (value != NULL && upd_arg == NULL)); + if (upd_arg == NULL) { if (modify_type == WT_UPDATE_RESERVE || modify_type == WT_UPDATE_TOMBSTONE) { /* @@ -66,9 +74,6 @@ __wt_col_modify(WT_CURSOR_BTREE *cbt, uint64_t recno, const WT_ITEM *value, WT_U } } - /* We're going to modify the page, we should have loaded history. */ - WT_ASSERT(session, cbt->ref->state != WT_REF_LIMBO); - /* If we don't yet have a modify structure, we'll need one. */ WT_RET(__wt_page_modify_init(session, page)); mod = page->modify; @@ -118,19 +123,19 @@ __wt_col_modify(WT_CURSOR_BTREE *cbt, uint64_t recno, const WT_ITEM *value, WT_U * it into place. */ if (cbt->compare == 0 && cbt->ins != NULL) { - /* - * If we are restoring updates that couldn't be evicted, the key must not exist on the new - * page. - */ - WT_ASSERT(session, upd_arg == NULL); - - /* Make sure the update can proceed. */ - WT_ERR(__wt_txn_update_check(session, old_upd = cbt->ins->upd)); + old_upd = cbt->ins->upd; + if (upd_arg == NULL) { + /* Make sure the update can proceed. */ + WT_ERR(__wt_txn_update_check(session, cbt, old_upd)); - /* Allocate a WT_UPDATE structure and transaction ID. */ - WT_ERR(__wt_update_alloc(session, value, &upd, &upd_size, modify_type)); - WT_ERR(__wt_txn_modify(session, upd)); - logged = true; + /* Allocate a WT_UPDATE structure and transaction ID. */ + WT_ERR(__wt_update_alloc(session, value, &upd, &upd_size, modify_type)); + WT_ERR(__wt_txn_modify(session, upd)); + logged = true; + } else { + upd = upd_arg; + upd_size = WT_UPDATE_MEMSIZE(upd); + } /* Avoid a data copy in WT_CURSOR.update. */ cbt->modify_update = upd; @@ -142,7 +147,7 @@ __wt_col_modify(WT_CURSOR_BTREE *cbt, uint64_t recno, const WT_ITEM *value, WT_U upd->next = old_upd; /* Serialize the update. */ - WT_ERR(__wt_update_serial(session, page, &cbt->ins->upd, &upd, upd_size, false)); + WT_ERR(__wt_update_serial(session, cbt, page, &cbt->ins->upd, &upd, upd_size, false)); } else { /* Allocate the append/update list reference as necessary. */ if (append) { diff --git a/src/third_party/wiredtiger/src/btree/row_key.c b/src/third_party/wiredtiger/src/btree/row_key.c index b2893fc2075..aec356f1a90 100644 --- a/src/third_party/wiredtiger/src/btree/row_key.c +++ b/src/third_party/wiredtiger/src/btree/row_key.c @@ -316,13 +316,6 @@ switch_and_jump: */ if (unpack->prefix == 0) { /* - * The only reason to be here is a Huffman encoded key, a non-encoded key with no prefix - * compression should have been directly referenced, and we should not have needed to - * unpack its cell. - */ - WT_ASSERT(session, btree->huffman_key != NULL); - - /* * If this is the key we originally wanted, we don't * care if we're rolling forward or backward, it's * what we want. Take a copy and wrap up. diff --git a/src/third_party/wiredtiger/src/btree/row_modify.c b/src/third_party/wiredtiger/src/btree/row_modify.c index ca03d888c73..d90412eab65 100644 --- a/src/third_party/wiredtiger/src/btree/row_modify.c +++ b/src/third_party/wiredtiger/src/btree/row_modify.c @@ -62,14 +62,22 @@ __wt_row_modify(WT_CURSOR_BTREE *cbt, const WT_ITEM *key, const WT_ITEM *value, upd = upd_arg; logged = false; - /* We're going to modify the page, we should have loaded history. */ - WT_ASSERT(session, cbt->ref->state != WT_REF_LIMBO); - /* If we don't yet have a modify structure, we'll need one. */ WT_RET(__wt_page_modify_init(session, page)); mod = page->modify; /* + * We should have EITHER: + * - A full update list to instantiate with. + * - An update to append the existing update list with. + * - A key/value pair to create an update with and append to the update list. + * + * A full update list is distinguished from an update by checking whether it has any "next" + * update. + */ + WT_ASSERT(session, (value == NULL && upd_arg != NULL) || (value != NULL && upd_arg == NULL)); + + /* * Modify: allocate an update array as necessary, build a WT_UPDATE structure, and call a * serialized function to insert the WT_UPDATE structure. * @@ -88,7 +96,7 @@ __wt_row_modify(WT_CURSOR_BTREE *cbt, const WT_ITEM *key, const WT_ITEM *value, if (upd_arg == NULL) { /* Make sure the update can proceed. */ - WT_ERR(__wt_txn_update_check(session, old_upd = *upd_entry)); + WT_ERR(__wt_txn_update_check(session, cbt, old_upd = *upd_entry)); /* Allocate a WT_UPDATE structure and transaction ID. */ WT_ERR(__wt_update_alloc(session, value, &upd, &upd_size, modify_type)); @@ -101,16 +109,15 @@ __wt_row_modify(WT_CURSOR_BTREE *cbt, const WT_ITEM *key, const WT_ITEM *value, upd_size = __wt_update_list_memsize(upd); /* - * We are restoring updates that couldn't be evicted, there should only be one update - * list per key. - */ - WT_ASSERT(session, *upd_entry == NULL); - - /* + * If it's a full update list, we're trying to instantiate the row. Otherwise, it's just + * a single update that we'd like to append to the update list. + * * Set the "old" entry to the second update in the list so that the serialization * function succeeds in swapping the first update into place. */ - old_upd = *upd_entry = upd->next; + if (upd->next != NULL) + *upd_entry = upd->next; + old_upd = *upd_entry; } /* @@ -120,7 +127,7 @@ __wt_row_modify(WT_CURSOR_BTREE *cbt, const WT_ITEM *key, const WT_ITEM *value, upd->next = old_upd; /* Serialize the update. */ - WT_ERR(__wt_update_serial(session, page, upd_entry, &upd, upd_size, exclusive)); + WT_ERR(__wt_update_serial(session, cbt, page, upd_entry, &upd, upd_size, exclusive)); } else { /* * Allocate the insert array as necessary. @@ -259,18 +266,14 @@ __wt_update_alloc(WT_SESSION_IMPL *session, const WT_ITEM *value, WT_UPDATE **up */ WT_ASSERT(session, modify_type != WT_UPDATE_INVALID); - /* - * Allocate the WT_UPDATE structure and room for the value, then copy the value into place. - */ - if (modify_type == WT_UPDATE_BIRTHMARK || modify_type == WT_UPDATE_RESERVE || - modify_type == WT_UPDATE_TOMBSTONE) - WT_RET(__wt_calloc(session, 1, WT_UPDATE_SIZE, &upd)); - else { - WT_RET(__wt_calloc(session, 1, WT_UPDATE_SIZE + value->size, &upd)); - if (value->size != 0) { - upd->size = WT_STORE_SIZE(value->size); - memcpy(upd->data, value->data, value->size); - } + if (modify_type == WT_UPDATE_TOMBSTONE || modify_type == WT_UPDATE_RESERVE) + value = NULL; + + /* Allocate the WT_UPDATE structure and room for the value, then copy any value into place. */ + WT_RET(__wt_calloc(session, 1, WT_UPDATE_SIZE + (value == NULL ? 0 : value->size), &upd)); + if (value != NULL && value->size != 0) { + upd->size = WT_STORE_SIZE(value->size); + memcpy(upd->data, value->data, value->size); } upd->type = (uint8_t)modify_type; @@ -288,11 +291,10 @@ __wt_update_obsolete_check( WT_SESSION_IMPL *session, WT_PAGE *page, WT_UPDATE *upd, bool update_accounting) { WT_TXN_GLOBAL *txn_global; - WT_UPDATE *first, *next, *prev; + WT_UPDATE *first, *next; size_t size; uint64_t oldest, stable; u_int count, upd_seen, upd_unstable; - bool upd_visible_all_seen; txn_global = &S2C(session)->txn_global; @@ -308,20 +310,16 @@ __wt_update_obsolete_check( * * Only updates with globally visible, self-contained data can terminate update chains. * - * Birthmarks are a special case: once a birthmark becomes obsolete, it can be discarded if - * there is a globally visible update before it and subsequent reads will see the on-page value - * (as expected). Inserting updates into the lookaside table relies on this behavior to avoid - * creating update chains with multiple birthmarks. We cannot discard the birthmark if it's the - * first globally visible update as the previous updates can be aborted and be freed causing the - * entire update chain being removed. */ - for (first = prev = NULL, upd_visible_all_seen = false, count = 0; upd != NULL; - prev = upd, upd = upd->next, count++) { + for (first = NULL, count = 0; upd != NULL; upd = upd->next, count++) { if (upd->txnid == WT_TXN_ABORTED) continue; ++upd_seen; - if (!__wt_txn_upd_visible_all(session, upd)) { + if (__wt_txn_upd_visible_all(session, upd)) { + if (first == NULL && WT_UPDATE_DATA_VALUE(upd)) + first = upd; + } else { first = NULL; /* * While we're here, also check for the update being kept only for timestamp history to @@ -329,27 +327,10 @@ __wt_update_obsolete_check( */ if (upd->start_ts != WT_TS_NONE && upd->start_ts >= oldest && upd->start_ts < stable) ++upd_unstable; - } else { - if (first == NULL) { - /* - * If we have seen a globally visible update before the birthmark, the birthmark can - * be discarded. - */ - if (upd_visible_all_seen && upd->type == WT_UPDATE_BIRTHMARK) - first = prev; - /* - * We cannot discard the birthmark if it is the first globally visible update as the - * previous updates can be aborted resulting the entire update chain being removed. - */ - else if (upd->type == WT_UPDATE_BIRTHMARK || WT_UPDATE_DATA_VALUE(upd)) - first = upd; - } - - upd_visible_all_seen = true; } } - __wt_cache_update_lookaside_score(session, upd_seen, upd_unstable); + __wt_cache_update_hs_score(session, upd_seen, upd_unstable); /* * We cannot discard this WT_UPDATE structure, we can only discard WT_UPDATE structures diff --git a/src/third_party/wiredtiger/src/cache/cache_las.c b/src/third_party/wiredtiger/src/cache/cache_las.c deleted file mode 100644 index 29546148688..00000000000 --- a/src/third_party/wiredtiger/src/cache/cache_las.c +++ /dev/null @@ -1,1239 +0,0 @@ -/*- - * Copyright (c) 2014-2020 MongoDB, Inc. - * Copyright (c) 2008-2014 WiredTiger, Inc. - * All rights reserved. - * - * See the file LICENSE for redistribution information. - */ - -#include "wt_internal.h" - -/* - * When an operation is accessing the lookaside table, it should ignore the cache size (since the - * cache is already full), any pages it reads should be evicted before application data, and the - * operation can't reenter reconciliation. - */ -#define WT_LAS_SESSION_FLAGS \ - (WT_SESSION_IGNORE_CACHE_SIZE | WT_SESSION_READ_WONT_NEED | WT_SESSION_NO_RECONCILE) - -/* - * __las_set_isolation -- - * Switch to read-uncommitted. - */ -static void -__las_set_isolation(WT_SESSION_IMPL *session, WT_TXN_ISOLATION *saved_isolationp) -{ - *saved_isolationp = session->txn.isolation; - session->txn.isolation = WT_ISO_READ_UNCOMMITTED; -} - -/* - * __las_restore_isolation -- - * Restore isolation. - */ -static void -__las_restore_isolation(WT_SESSION_IMPL *session, WT_TXN_ISOLATION saved_isolation) -{ - session->txn.isolation = saved_isolation; -} - -/* - * __las_entry_count -- - * Return when there are entries in the lookaside table. - */ -static uint64_t -__las_entry_count(WT_CACHE *cache) -{ - uint64_t insert_cnt, remove_cnt; - - insert_cnt = cache->las_insert_count; - WT_ORDERED_READ(remove_cnt, cache->las_remove_count); - - return (insert_cnt > remove_cnt ? insert_cnt - remove_cnt : 0); -} - -/* - * __wt_las_config -- - * Configure the lookaside table. - */ -int -__wt_las_config(WT_SESSION_IMPL *session, const char **cfg) -{ - WT_CONFIG_ITEM cval; - WT_CURSOR_BTREE *las_cursor; - WT_SESSION_IMPL *las_session; - - WT_RET(__wt_config_gets(session, cfg, "cache_overflow.file_max", &cval)); - - if (cval.val != 0 && cval.val < WT_LAS_FILE_MIN) - WT_RET_MSG(session, EINVAL, "max cache overflow size %" PRId64 " below minimum %d", - cval.val, WT_LAS_FILE_MIN); - - /* This is expected for in-memory configurations. */ - las_session = S2C(session)->cache->las_session[0]; - WT_ASSERT(session, las_session != NULL || F_ISSET(S2C(session), WT_CONN_IN_MEMORY)); - - if (las_session == NULL) - return (0); - - /* - * We need to set file_max on the btree associated with one of the lookaside sessions. - */ - las_cursor = (WT_CURSOR_BTREE *)las_session->las_cursor; - las_cursor->btree->file_max = (uint64_t)cval.val; - - WT_STAT_CONN_SET(session, cache_lookaside_ondisk_max, las_cursor->btree->file_max); - - return (0); -} - -/* - * __wt_las_empty -- - * Return when there are entries in the lookaside table. - */ -bool -__wt_las_empty(WT_SESSION_IMPL *session) -{ - return (__las_entry_count(S2C(session)->cache) == 0); -} - -/* - * __wt_las_stats_update -- - * Update the lookaside table statistics for return to the application. - */ -void -__wt_las_stats_update(WT_SESSION_IMPL *session) -{ - WT_CACHE *cache; - WT_CONNECTION_IMPL *conn; - WT_CONNECTION_STATS **cstats; - WT_DSRC_STATS **dstats; - int64_t v; - - conn = S2C(session); - cache = conn->cache; - - /* - * Lookaside table statistics are copied from the underlying lookaside table data-source - * statistics. If there's no lookaside table, values remain 0. - */ - if (!F_ISSET(conn, WT_CONN_LOOKASIDE_OPEN)) - return; - - /* Set the connection-wide statistics. */ - cstats = conn->stats; - - WT_STAT_SET(session, cstats, cache_lookaside_entries, __las_entry_count(cache)); - - /* - * We have a cursor, and we need the underlying data handle; we can get to it by way of the - * underlying btree handle, but it's a little ugly. - */ - dstats = ((WT_CURSOR_BTREE *)cache->las_session[0]->las_cursor)->btree->dhandle->stats; - - v = WT_STAT_READ(dstats, cursor_update); - WT_STAT_SET(session, cstats, cache_lookaside_insert, v); - v = WT_STAT_READ(dstats, cursor_remove); - WT_STAT_SET(session, cstats, cache_lookaside_remove, v); - - /* - * If we're clearing stats we need to clear the cursor values we just read. This does not clear - * the rest of the statistics in the lookaside data source stat cursor, but we own that - * namespace so we don't have to worry about users seeing inconsistent data source information. - */ - if (FLD_ISSET(conn->stat_flags, WT_STAT_CLEAR)) { - WT_STAT_SET(session, dstats, cursor_update, 0); - WT_STAT_SET(session, dstats, cursor_remove, 0); - } -} - -/* - * __wt_las_create -- - * Initialize the database's lookaside store. - */ -int -__wt_las_create(WT_SESSION_IMPL *session, const char **cfg) -{ - WT_CACHE *cache; - WT_CONNECTION_IMPL *conn; - WT_DECL_RET; - int i; - const char *drop_cfg[] = {WT_CONFIG_BASE(session, WT_SESSION_drop), "force=true", NULL}; - - conn = S2C(session); - cache = conn->cache; - - /* Read-only and in-memory configurations don't need the LAS table. */ - if (F_ISSET(conn, WT_CONN_IN_MEMORY | WT_CONN_READONLY)) - return (0); - - /* - * Done at startup: we cannot do it on demand because we require the schema lock to create and - * drop the table, and it may not always be available. - * - * Discard any previous incarnation of the table. - */ - WT_WITH_SCHEMA_LOCK(session, ret = __wt_schema_drop(session, WT_LAS_URI, drop_cfg)); - WT_RET(ret); - - /* Re-create the table. */ - WT_RET(__wt_session_create(session, WT_LAS_URI, WT_LAS_CONFIG)); - - /* - * Open a shared internal session and cursor used for the lookaside table. This session should - * never perform reconciliation. - */ - for (i = 0; i < WT_LAS_NUM_SESSIONS; i++) { - WT_RET(__wt_open_internal_session( - conn, "lookaside table", true, WT_LAS_SESSION_FLAGS, &cache->las_session[i])); - WT_RET(__wt_las_cursor_open(cache->las_session[i])); - } - - WT_RET(__wt_las_config(session, cfg)); - - /* The statistics server is already running, make sure we don't race. */ - WT_WRITE_BARRIER(); - F_SET(conn, WT_CONN_LOOKASIDE_OPEN); - - return (0); -} - -/* - * __wt_las_destroy -- - * Destroy the database's lookaside store. - */ -int -__wt_las_destroy(WT_SESSION_IMPL *session) -{ - WT_CACHE *cache; - WT_CONNECTION_IMPL *conn; - WT_DECL_RET; - WT_SESSION *wt_session; - int i; - - conn = S2C(session); - cache = conn->cache; - - F_CLR(conn, WT_CONN_LOOKASIDE_OPEN); - if (cache == NULL) - return (0); - - for (i = 0; i < WT_LAS_NUM_SESSIONS; i++) { - if (cache->las_session[i] == NULL) - continue; - - wt_session = &cache->las_session[i]->iface; - WT_TRET(wt_session->close(wt_session, NULL)); - cache->las_session[i] = NULL; - } - - __wt_buf_free(session, &cache->las_sweep_key); - __wt_free(session, cache->las_dropped); - __wt_free(session, cache->las_sweep_dropmap); - - return (ret); -} - -/* - * __wt_las_cursor_open -- - * Open a new lookaside table cursor. - */ -int -__wt_las_cursor_open(WT_SESSION_IMPL *session) -{ - WT_BTREE *btree; - WT_CURSOR *cursor; - WT_DECL_RET; - const char *open_cursor_cfg[] = {WT_CONFIG_BASE(session, WT_SESSION_open_cursor), NULL}; - - WT_WITHOUT_DHANDLE( - session, ret = __wt_open_cursor(session, WT_LAS_URI, NULL, open_cursor_cfg, &cursor)); - WT_RET(ret); - - /* - * Retrieve the btree from the cursor, rather than the session because we don't always switch - * the LAS handle in to the session before entering this function. - */ - btree = ((WT_CURSOR_BTREE *)cursor)->btree; - - /* Track the lookaside file ID. */ - if (S2C(session)->cache->las_fileid == 0) - S2C(session)->cache->las_fileid = btree->id; - - /* - * Set special flags for the lookaside table: the lookaside flag (used, for example, to avoid - * writing records during reconciliation), also turn off checkpoints and logging. - * - * Test flags before setting them so updates can't race in subsequent opens (the first update is - * safe because it's single-threaded from wiredtiger_open). - */ - if (!F_ISSET(btree, WT_BTREE_LOOKASIDE)) - F_SET(btree, WT_BTREE_LOOKASIDE); - if (!F_ISSET(btree, WT_BTREE_NO_CHECKPOINT)) - F_SET(btree, WT_BTREE_NO_CHECKPOINT); - if (!F_ISSET(btree, WT_BTREE_NO_LOGGING)) - F_SET(btree, WT_BTREE_NO_LOGGING); - - session->las_cursor = cursor; - F_SET(session, WT_SESSION_LOOKASIDE_CURSOR); - - return (0); -} - -/* - * __wt_las_cursor -- - * Return a lookaside cursor. - */ -void -__wt_las_cursor(WT_SESSION_IMPL *session, WT_CURSOR **cursorp, uint32_t *session_flags) -{ - WT_CACHE *cache; - int i; - - *cursorp = NULL; - - /* - * We don't want to get tapped for eviction after we start using the lookaside cursor; save a - * copy of the current eviction state, we'll turn eviction off before we return. - * - * Don't cache lookaside table pages, we're here because of eviction problems and there's no - * reason to believe lookaside pages will be useful more than once. - */ - *session_flags = F_MASK(session, WT_LAS_SESSION_FLAGS); - - cache = S2C(session)->cache; - - /* - * Some threads have their own lookaside table cursors, else lock the shared lookaside cursor. - */ - if (F_ISSET(session, WT_SESSION_LOOKASIDE_CURSOR)) - *cursorp = session->las_cursor; - else { - for (;;) { - __wt_spin_lock(session, &cache->las_lock); - for (i = 0; i < WT_LAS_NUM_SESSIONS; i++) { - if (!cache->las_session_inuse[i]) { - *cursorp = cache->las_session[i]->las_cursor; - cache->las_session_inuse[i] = true; - break; - } - } - __wt_spin_unlock(session, &cache->las_lock); - if (*cursorp != NULL) - break; - /* - * If all the lookaside sessions are busy, stall. - * - * XXX better as a condition variable. - */ - __wt_sleep(0, WT_THOUSAND); - if (F_ISSET(session, WT_SESSION_INTERNAL)) - WT_STAT_CONN_INCRV(session, cache_lookaside_cursor_wait_internal, WT_THOUSAND); - else - WT_STAT_CONN_INCRV(session, cache_lookaside_cursor_wait_application, WT_THOUSAND); - } - } - - /* Configure session to access the lookaside table. */ - F_SET(session, WT_LAS_SESSION_FLAGS); -} - -/* - * __wt_las_cursor_close -- - * Discard a lookaside cursor. - */ -int -__wt_las_cursor_close(WT_SESSION_IMPL *session, WT_CURSOR **cursorp, uint32_t session_flags) -{ - WT_CACHE *cache; - WT_CURSOR *cursor; - WT_DECL_RET; - int i; - - cache = S2C(session)->cache; - - if ((cursor = *cursorp) == NULL) - return (0); - *cursorp = NULL; - - /* Reset the cursor. */ - ret = cursor->reset(cursor); - - /* - * We turned off caching and eviction while the lookaside cursor was in use, restore the - * session's flags. - */ - F_CLR(session, WT_LAS_SESSION_FLAGS); - F_SET(session, session_flags); - - /* - * Some threads have their own lookaside table cursors, else unlock the shared lookaside cursor. - */ - if (!F_ISSET(session, WT_SESSION_LOOKASIDE_CURSOR)) { - __wt_spin_lock(session, &cache->las_lock); - for (i = 0; i < WT_LAS_NUM_SESSIONS; i++) - if (cursor->session == &cache->las_session[i]->iface) { - cache->las_session_inuse[i] = false; - break; - } - __wt_spin_unlock(session, &cache->las_lock); - WT_ASSERT(session, i != WT_LAS_NUM_SESSIONS); - } - - return (ret); -} - -/* - * __wt_las_page_skip_locked -- - * Check if we can skip reading a page with lookaside entries, where the page is already locked. - */ -bool -__wt_las_page_skip_locked(WT_SESSION_IMPL *session, WT_REF *ref) -{ - WT_TXN *txn; - - txn = &session->txn; - - /* - * Skip lookaside pages if reading without a timestamp and all the updates in lookaside are in - * the past. - * - * Lookaside eviction preferentially chooses the newest updates when creating page images with - * no stable timestamp. If a stable timestamp has been set, we have to visit the page because - * eviction chooses old version of records in that case. - * - * One case where we may need to visit the page is if lookaside eviction is active in tree 2 - * when a checkpoint has started and is working its way through tree 1. In that case, lookaside - * may have created a page image with updates in the future of the checkpoint. - * - * We also need to instantiate a lookaside page if this is an update operation in progress or - * transaction is in prepared state. - */ - if (F_ISSET(txn, WT_TXN_PREPARE | WT_TXN_UPDATE)) - return (false); - - if (!F_ISSET(txn, WT_TXN_HAS_SNAPSHOT)) - return (false); - - /* - * If some of the page's history overlaps with the reader's snapshot then we have to read it. - */ - if (WT_TXNID_LE(txn->snap_min, ref->page_las->max_txn)) - return (false); - - /* - * Otherwise, if not reading at a timestamp, the page's history is in the past, so the page - * image is correct if it contains the most recent versions of everything and nothing was - * prepared. - */ - if (!F_ISSET(txn, WT_TXN_HAS_TS_READ)) - return (!ref->page_las->has_prepares && ref->page_las->min_skipped_ts == WT_TS_MAX); - - /* - * Skip lookaside history if reading as of a timestamp, we evicted new versions of data and all - * the updates are in the past. This is not possible for prepared updates, because the commit - * timestamp was not known when the page was evicted. - * - * Otherwise, skip reading lookaside history if everything on the page is older than the read - * timestamp, and the oldest update in lookaside newer than the page is in the future of the - * reader. This seems unlikely, but is exactly what eviction tries to do when a checkpoint is - * running. - */ - if (!ref->page_las->has_prepares && ref->page_las->min_skipped_ts == WT_TS_MAX && - txn->read_timestamp >= ref->page_las->max_ondisk_ts) - return (true); - - if (txn->read_timestamp >= ref->page_las->max_ondisk_ts && - txn->read_timestamp < ref->page_las->min_skipped_ts) - return (true); - - return (false); -} - -/* - * __wt_las_page_skip -- - * Check if we can skip reading a page with lookaside entries, where the page needs to be locked - * before checking. - */ -bool -__wt_las_page_skip(WT_SESSION_IMPL *session, WT_REF *ref) -{ - uint32_t previous_state; - bool skip; - - if ((previous_state = ref->state) != WT_REF_LIMBO && previous_state != WT_REF_LOOKASIDE) - return (false); - - if (!WT_REF_CAS_STATE(session, ref, previous_state, WT_REF_LOCKED)) - return (false); - - skip = __wt_las_page_skip_locked(session, ref); - - /* Restore the state and push the change. */ - WT_REF_SET_STATE(ref, previous_state); - WT_FULL_BARRIER(); - - return (skip); -} - -/* - * __las_remove_block -- - * Remove all records for a given page from the lookaside store. - */ -static int -__las_remove_block(WT_CURSOR *cursor, uint64_t pageid, bool lock_wait, uint64_t *remove_cntp) -{ - WT_CONNECTION_IMPL *conn; - WT_DECL_RET; - WT_ITEM las_key; - WT_SESSION_IMPL *session; - WT_TXN_ISOLATION saved_isolation; - uint64_t las_counter, las_pageid; - uint32_t las_id; - bool local_txn; - - *remove_cntp = 0; - saved_isolation = 0; /*[-Wconditional-uninitialized] */ - - session = (WT_SESSION_IMPL *)cursor->session; - conn = S2C(session); - local_txn = false; - - /* Prevent the sweep thread from removing the block. */ - if (lock_wait) - __wt_writelock(session, &conn->cache->las_sweepwalk_lock); - else - WT_RET(__wt_try_writelock(session, &conn->cache->las_sweepwalk_lock)); - - WT_ERR(__wt_txn_begin(session, NULL)); - __las_set_isolation(session, &saved_isolation); - local_txn = true; - - /* - * Search for the block's unique btree ID and page ID prefix and step through all matching - * records, removing them. - */ - for (ret = __wt_las_cursor_position(cursor, pageid); ret == 0; ret = cursor->next(cursor)) { - WT_ERR(cursor->get_key(cursor, &las_pageid, &las_id, &las_counter, &las_key)); - - /* Confirm that we have a matching record. */ - if (las_pageid != pageid) - break; - - WT_ERR(cursor->remove(cursor)); - ++*remove_cntp; - } - WT_ERR_NOTFOUND_OK(ret); - -err: - if (local_txn) { - if (ret == 0) - ret = __wt_txn_commit(session, NULL); - else - WT_TRET(__wt_txn_rollback(session, NULL)); - __las_restore_isolation(session, saved_isolation); - } - - __wt_writeunlock(session, &conn->cache->las_sweepwalk_lock); - return (ret); -} - -/* - * __las_insert_block_verbose -- - * Display a verbose message once per checkpoint with details about the cache state when - * performing a lookaside table write. - */ -static void -__las_insert_block_verbose(WT_SESSION_IMPL *session, WT_BTREE *btree, WT_MULTI *multi) -{ - WT_CACHE *cache; - WT_CONNECTION_IMPL *conn; - double pct_dirty, pct_full; - uint64_t ckpt_gen_current, ckpt_gen_last; - uint32_t btree_id; - char ts_string[2][WT_TS_INT_STRING_SIZE]; - - btree_id = btree->id; - - if (!WT_VERBOSE_ISSET(session, WT_VERB_LOOKASIDE | WT_VERB_LOOKASIDE_ACTIVITY)) - return; - - conn = S2C(session); - cache = conn->cache; - ckpt_gen_current = __wt_gen(session, WT_GEN_CHECKPOINT); - ckpt_gen_last = cache->las_verb_gen_write; - - /* - * Print a message if verbose lookaside, or once per checkpoint if only reporting activity. - * Avoid an expensive atomic operation as often as possible when the message rate is limited. - */ - if (WT_VERBOSE_ISSET(session, WT_VERB_LOOKASIDE) || - (ckpt_gen_current > ckpt_gen_last && - __wt_atomic_casv64(&cache->las_verb_gen_write, ckpt_gen_last, ckpt_gen_current))) { - WT_IGNORE_RET_BOOL(__wt_eviction_clean_needed(session, &pct_full)); - WT_IGNORE_RET_BOOL(__wt_eviction_dirty_needed(session, &pct_dirty)); - - __wt_verbose(session, WT_VERB_LOOKASIDE | WT_VERB_LOOKASIDE_ACTIVITY, - "Page reconciliation triggered lookaside write " - "file ID %" PRIu32 ", page ID %" PRIu64 - ". " - "Max txn ID %" PRIu64 - ", max ondisk timestamp %s, " - "first skipped ts %s. " - "Entries now in lookaside file: %" PRId64 - ", " - "cache dirty: %2.3f%% , " - "cache use: %2.3f%%", - btree_id, multi->page_las.las_pageid, multi->page_las.max_txn, - __wt_timestamp_to_string(multi->page_las.max_ondisk_ts, ts_string[0]), - __wt_timestamp_to_string(multi->page_las.min_skipped_ts, ts_string[1]), - WT_STAT_READ(conn->stats, cache_lookaside_entries), pct_dirty, pct_full); - } - - /* Never skip updating the tracked generation */ - if (WT_VERBOSE_ISSET(session, WT_VERB_LOOKASIDE)) - cache->las_verb_gen_write = ckpt_gen_current; -} - -/* - * __wt_las_insert_block -- - * Copy one set of saved updates into the database's lookaside table. - */ -int -__wt_las_insert_block( - WT_CURSOR *cursor, WT_BTREE *btree, WT_PAGE *page, WT_MULTI *multi, WT_ITEM *key) -{ - WT_CONNECTION_IMPL *conn; - WT_DECL_RET; - WT_ITEM las_value; - WT_SAVE_UPD *list; - WT_SESSION_IMPL *session; - WT_TXN_ISOLATION saved_isolation; - WT_UPDATE *first_upd, *upd; - wt_off_t las_size; - uint64_t insert_cnt, las_counter, las_pageid, max_las_size; - uint64_t prepared_insert_cnt; - uint32_t btree_id, i, slot; - uint8_t *p; - bool local_txn; - - session = (WT_SESSION_IMPL *)cursor->session; - conn = S2C(session); - WT_CLEAR(las_value); - saved_isolation = 0; /*[-Wconditional-uninitialized] */ - insert_cnt = prepared_insert_cnt = 0; - btree_id = btree->id; - local_txn = false; - - las_pageid = __wt_atomic_add64(&conn->cache->las_pageid, 1); - - if (!btree->lookaside_entries) - btree->lookaside_entries = true; - -#ifdef HAVE_DIAGNOSTIC - { - uint64_t remove_cnt; - /* - * There should never be any entries with the page ID we are about to use. - */ - WT_RET_BUSY_OK(__las_remove_block(cursor, las_pageid, false, &remove_cnt)); - WT_ASSERT(session, remove_cnt == 0); - } -#endif - - /* Wrap all the updates in a transaction. */ - WT_ERR(__wt_txn_begin(session, NULL)); - __las_set_isolation(session, &saved_isolation); - local_txn = true; - - /* Inserts should be on the same page absent a split, search any pinned leaf page. */ - F_SET(cursor, WT_CURSTD_UPDATE_LOCAL); - - /* Enter each update in the boundary's list into the lookaside store. */ - for (las_counter = 0, i = 0, list = multi->supd; i < multi->supd_entries; ++i, ++list) { - /* Lookaside table key component: source key. */ - switch (page->type) { - case WT_PAGE_COL_FIX: - case WT_PAGE_COL_VAR: - p = key->mem; - WT_ERR(__wt_vpack_uint(&p, 0, WT_INSERT_RECNO(list->ins))); - key->size = WT_PTRDIFF(p, key->data); - break; - case WT_PAGE_ROW_LEAF: - if (list->ins == NULL) { - WT_WITH_BTREE( - session, btree, ret = __wt_row_leaf_key(session, page, list->ripcip, key, false)); - WT_ERR(ret); - } else { - key->data = WT_INSERT_KEY(list->ins); - key->size = WT_INSERT_KEY_SIZE(list->ins); - } - break; - default: - WT_ERR(__wt_illegal_value(session, page->type)); - } - - /* - * Lookaside table value component: update reference. Updates come from the row-store insert - * list (an inserted item), or update array (an update to an original on-page item), or from - * a column-store insert list (column-store format has no update array, the insert list - * contains both inserted items and updates to original on-page items). When rolling forward - * a modify update from an original on-page item, we need an on-page slot so we can find the - * original on-page item. When rolling forward from an inserted item, no on-page slot is - * possible. - */ - slot = UINT32_MAX; /* Impossible slot */ - if (list->ripcip != NULL) - slot = page->type == WT_PAGE_ROW_LEAF ? WT_ROW_SLOT(page, list->ripcip) : - WT_COL_SLOT(page, list->ripcip); - first_upd = list->ins == NULL ? page->modify->mod_row_update[slot] : list->ins->upd; - - /* - * Trim any updates before writing to lookaside. This saves wasted work, but is also - * necessary because the reconciliation only resolves existing birthmarks if they aren't - * obsolete. - */ - WT_WITH_BTREE( - session, btree, upd = __wt_update_obsolete_check(session, page, first_upd, true)); - if (upd != NULL) - __wt_free_update_list(session, upd); - upd = first_upd; - - /* - * It's not OK for the update list to contain a birthmark on entry - we will generate one - * below if necessary. - */ - WT_ASSERT(session, __wt_count_birthmarks(first_upd) == 0); - - /* - * Walk the list of updates, storing each key/value pair into the lookaside table. Skip - * aborted items (there's no point to restoring them), and assert we never see a reserved - * item. - */ - do { - if (upd->txnid == WT_TXN_ABORTED) - continue; - - switch (upd->type) { - case WT_UPDATE_MODIFY: - case WT_UPDATE_STANDARD: - las_value.data = upd->data; - las_value.size = upd->size; - break; - case WT_UPDATE_TOMBSTONE: - las_value.size = 0; - break; - default: - /* - * It is never OK to see a birthmark here - it would be referring to the wrong page - * image. - */ - WT_ERR(__wt_illegal_value(session, upd->type)); - } - - cursor->set_key(cursor, las_pageid, btree_id, ++las_counter, key); - - /* - * If saving a non-zero length value on the page, save a birthmark instead of - * duplicating it in the lookaside table. (We check the length because row-store doesn't - * write zero-length data items.) - */ - if (upd == list->onpage_upd && upd->size > 0 && - (upd->type == WT_UPDATE_STANDARD || upd->type == WT_UPDATE_MODIFY)) { - las_value.size = 0; - cursor->set_value(cursor, upd->txnid, upd->start_ts, upd->durable_ts, - upd->prepare_state, WT_UPDATE_BIRTHMARK, &las_value); - } else - cursor->set_value(cursor, upd->txnid, upd->start_ts, upd->durable_ts, - upd->prepare_state, upd->type, &las_value); - - /* - * Using update instead of insert so the page stays pinned and can be searched before - * the tree. - */ - WT_ERR(cursor->update(cursor)); - ++insert_cnt; - if (upd->prepare_state == WT_PREPARE_INPROGRESS) - ++prepared_insert_cnt; - } while ((upd = upd->next) != NULL); - } - - WT_ERR(__wt_block_manager_named_size(session, WT_LAS_FILE, &las_size)); - WT_STAT_CONN_SET(session, cache_lookaside_ondisk, las_size); - max_las_size = ((WT_CURSOR_BTREE *)cursor)->btree->file_max; - if (max_las_size != 0 && (uint64_t)las_size > max_las_size) - WT_PANIC_MSG(session, WT_PANIC, "WiredTigerLAS: file size of %" PRIu64 - " exceeds maximum " - "size %" PRIu64, - (uint64_t)las_size, max_las_size); - -err: - /* Resolve the transaction. */ - if (local_txn) { - if (ret == 0) - ret = __wt_txn_commit(session, NULL); - else - WT_TRET(__wt_txn_rollback(session, NULL)); - __las_restore_isolation(session, saved_isolation); - F_CLR(cursor, WT_CURSTD_UPDATE_LOCAL); - - /* Adjust the entry count. */ - if (ret == 0) { - (void)__wt_atomic_add64(&conn->cache->las_insert_count, insert_cnt); - WT_STAT_CONN_INCRV( - session, txn_prepared_updates_lookaside_inserts, prepared_insert_cnt); - } - } - - if (ret == 0 && insert_cnt > 0) { - multi->page_las.las_pageid = las_pageid; - multi->page_las.has_prepares = prepared_insert_cnt > 0; - __las_insert_block_verbose(session, btree, multi); - } - - WT_UNUSED(first_upd); - return (ret); -} - -/* - * __wt_las_cursor_position -- - * Position a lookaside cursor at the beginning of a block. There may be no block of lookaside - * entries if they have been removed by WT_CONNECTION::rollback_to_stable. - */ -int -__wt_las_cursor_position(WT_CURSOR *cursor, uint64_t pageid) -{ - WT_ITEM las_key; - uint64_t las_counter, las_pageid; - uint32_t las_id; - int exact; - - /* - * When scanning for all pages, start at the beginning of the lookaside table. - */ - if (pageid == 0) { - WT_RET(cursor->reset(cursor)); - return (cursor->next(cursor)); - } - - /* - * Because of the special visibility rules for lookaside, a new block can appear in between our - * search and the block of interest. Keep trying until we find it. - */ - for (;;) { - WT_CLEAR(las_key); - cursor->set_key(cursor, pageid, (uint32_t)0, (uint64_t)0, &las_key); - WT_RET(cursor->search_near(cursor, &exact)); - if (exact < 0) - WT_RET(cursor->next(cursor)); - - /* - * Because of the special visibility rules for lookaside, a new block can appear in between - * our search and the block of interest. Keep trying while we have a key lower than we - * expect. - * - * There may be no block of lookaside entries if they have been removed by - * WT_CONNECTION::rollback_to_stable. - */ - WT_RET(cursor->get_key(cursor, &las_pageid, &las_id, &las_counter, &las_key)); - if (las_pageid >= pageid) - return (0); - } - - /* NOTREACHED */ -} - -/* - * __wt_las_remove_block -- - * Remove all records for a given page from the lookaside table. - */ -int -__wt_las_remove_block(WT_SESSION_IMPL *session, uint64_t pageid) -{ - WT_CONNECTION_IMPL *conn; - WT_CURSOR *cursor; - WT_DECL_RET; - uint64_t remove_cnt; - uint32_t session_flags; - - conn = S2C(session); - session_flags = 0; /* [-Wconditional-uninitialized] */ - - /* - * This is an external API for removing records from the lookaside table, first acquiring a - * lookaside table cursor and enclosing transaction, then calling an underlying function to do - * the work. - */ - __wt_las_cursor(session, &cursor, &session_flags); - - if ((ret = __las_remove_block(cursor, pageid, true, &remove_cnt)) == 0) - (void)__wt_atomic_add64(&conn->cache->las_remove_count, remove_cnt); - - WT_TRET(__wt_las_cursor_close(session, &cursor, session_flags)); - return (ret); -} - -/* - * __wt_las_remove_dropped -- - * Remove an opened btree ID if it is in the dropped table. - */ -void -__wt_las_remove_dropped(WT_SESSION_IMPL *session) -{ - WT_BTREE *btree; - WT_CACHE *cache; - u_int i, j; - - btree = S2BT(session); - cache = S2C(session)->cache; - - __wt_spin_lock(session, &cache->las_sweep_lock); - for (i = 0; i < cache->las_dropped_next && cache->las_dropped[i] != btree->id; i++) - ; - - if (i < cache->las_dropped_next) { - cache->las_dropped_next--; - for (j = i; j < cache->las_dropped_next; j++) - cache->las_dropped[j] = cache->las_dropped[j + 1]; - } - __wt_spin_unlock(session, &cache->las_sweep_lock); -} - -/* - * __wt_las_save_dropped -- - * Save a dropped btree ID to be swept from the lookaside table. - */ -int -__wt_las_save_dropped(WT_SESSION_IMPL *session) -{ - WT_BTREE *btree; - WT_CACHE *cache; - WT_DECL_RET; - - btree = S2BT(session); - cache = S2C(session)->cache; - - __wt_spin_lock(session, &cache->las_sweep_lock); - WT_ERR(__wt_realloc_def( - session, &cache->las_dropped_alloc, cache->las_dropped_next + 1, &cache->las_dropped)); - cache->las_dropped[cache->las_dropped_next++] = btree->id; -err: - __wt_spin_unlock(session, &cache->las_sweep_lock); - return (ret); -} - -/* - * __las_sweep_count -- - * Calculate how many records to examine per sweep step. - */ -static inline uint64_t -__las_sweep_count(WT_CACHE *cache) -{ - uint64_t las_entry_count; - - /* - * The sweep server is a slow moving thread. Try to review the entire lookaside table once every - * 5 minutes. - * - * The reason is because the lookaside table exists because we're seeing cache/eviction pressure - * (it allows us to trade performance and disk space for cache space), and it's likely lookaside - * blocks are being evicted, and reading them back in doesn't help things. A trickier, but - * possibly better, alternative might be to review all lookaside blocks in the cache in order to - * get rid of them, and slowly review lookaside blocks that have already been evicted. - * - * Put upper and lower bounds on the calculation: since reads of pages with lookaside entries - * are blocked during sweep, make sure we do some work but don't block reads for too long. - */ - las_entry_count = __las_entry_count(cache); - return ( - (uint64_t)WT_MAX(WT_LAS_SWEEP_ENTRIES, las_entry_count / (5 * WT_MINUTE / WT_LAS_SWEEP_SEC))); -} - -/* - * __las_sweep_init -- - * Prepare to start a lookaside sweep. - */ -static int -__las_sweep_init(WT_SESSION_IMPL *session) -{ - WT_CACHE *cache; - WT_DECL_RET; - u_int i; - - cache = S2C(session)->cache; - - __wt_spin_lock(session, &cache->las_sweep_lock); - - /* - * If no files have been dropped and the lookaside file is empty, there's nothing to do. - */ - if (cache->las_dropped_next == 0 && __wt_las_empty(session)) - WT_ERR(WT_NOTFOUND); - - /* - * Record the current page ID: sweep will stop after this point. - * - * Since the btree IDs we're scanning are closed, any eviction must have already completed, so - * we won't miss anything with this approach. - * - * Also, if a tree is reopened and there is lookaside activity before this sweep completes, it - * will have a higher page ID and should not be removed. - */ - cache->las_sweep_max_pageid = cache->las_pageid; - - /* Scan the btree IDs to find min/max. */ - cache->las_sweep_dropmin = UINT32_MAX; - cache->las_sweep_dropmax = 0; - for (i = 0; i < cache->las_dropped_next; i++) { - cache->las_sweep_dropmin = WT_MIN(cache->las_sweep_dropmin, cache->las_dropped[i]); - cache->las_sweep_dropmax = WT_MAX(cache->las_sweep_dropmax, cache->las_dropped[i]); - } - - /* Initialize the bitmap. */ - __wt_free(session, cache->las_sweep_dropmap); - WT_ERR(__bit_alloc( - session, 1 + cache->las_sweep_dropmax - cache->las_sweep_dropmin, &cache->las_sweep_dropmap)); - for (i = 0; i < cache->las_dropped_next; i++) - __bit_set(cache->las_sweep_dropmap, cache->las_dropped[i] - cache->las_sweep_dropmin); - - /* Clear the list of btree IDs. */ - cache->las_dropped_next = 0; - -err: - __wt_spin_unlock(session, &cache->las_sweep_lock); - return (ret); -} - -/* - * __wt_las_sweep -- - * Sweep the lookaside table. - */ -int -__wt_las_sweep(WT_SESSION_IMPL *session) -{ - WT_CACHE *cache; - WT_CURSOR *cursor; - WT_DECL_ITEM(saved_key); - WT_DECL_RET; - WT_ITEM las_key, las_value; - WT_ITEM *sweep_key; - wt_timestamp_t durable_timestamp, las_timestamp; - uint64_t cnt, remove_cnt, las_pageid, saved_pageid, visit_cnt; - uint64_t las_counter, las_txnid; - uint32_t las_id, session_flags; - uint8_t prepare_state, upd_type; - int notused; - bool local_txn, locked, removing_key_block; - - cache = S2C(session)->cache; - cursor = NULL; - sweep_key = &cache->las_sweep_key; - remove_cnt = 0; - session_flags = 0; /* [-Werror=maybe-uninitialized] */ - local_txn = locked = removing_key_block = false; - - WT_RET(__wt_scr_alloc(session, 0, &saved_key)); - saved_pageid = 0; - - /* - * Prevent other threads removing entries from underneath the sweep. - */ - __wt_writelock(session, &cache->las_sweepwalk_lock); - locked = true; - - /* - * Allocate a cursor and wrap all the updates in a transaction. We should have our own lookaside - * cursor. - */ - __wt_las_cursor(session, &cursor, &session_flags); - WT_ASSERT(session, cursor->session == &session->iface); - WT_ERR(__wt_txn_begin(session, NULL)); - local_txn = true; - - /* Encourage a race */ - __wt_timing_stress(session, WT_TIMING_STRESS_LOOKASIDE_SWEEP); - - /* - * When continuing a sweep, position the cursor using the key from the last call (we don't care - * if we're before or after the key, either side is fine). - * - * Otherwise, we're starting a new sweep, gather the list of trees to sweep. - */ - if (sweep_key->size != 0) { - __wt_cursor_set_raw_key(cursor, sweep_key); - ret = cursor->search_near(cursor, ¬used); - - /* - * Don't search for the same key twice; if we don't set a new key below, it's because we've - * reached the end of the table and we want the next pass to start at the beginning of the - * table. Searching for the same key could leave us stuck at the end of the table, - * repeatedly checking the same rows. - */ - __wt_buf_free(session, sweep_key); - } else - ret = __las_sweep_init(session); - if (ret != 0) - goto srch_notfound; - - cnt = __las_sweep_count(cache); - visit_cnt = 0; - - /* Walk the file. */ - while ((ret = cursor->next(cursor)) == 0) { - WT_ERR(cursor->get_key(cursor, &las_pageid, &las_id, &las_counter, &las_key)); - - __wt_verbose(session, WT_VERB_LOOKASIDE_ACTIVITY, - "Sweep reviewing lookaside entry with lookaside " - "page ID %" PRIu64 " btree ID %" PRIu32 " saved key size: %" WT_SIZET_FMT, - las_pageid, las_id, saved_key->size); - - /* - * Signal to stop if the cache is stuck: we are ignoring the cache size while scanning the - * lookaside table, so we're making things worse. - */ - if (__wt_cache_stuck(session)) - cnt = 0; - - /* - * Don't go past the end of lookaside from when sweep started. If a file is reopened, its ID - * may be reused past this point so the bitmap we're using is not valid. - */ - if (las_pageid > cache->las_sweep_max_pageid) { - __wt_buf_free(session, sweep_key); - ret = WT_NOTFOUND; - break; - } - - /* - * We only want to break between key blocks. Stop if we've processed enough entries either - * all we wanted or enough and there is a reader waiting and we're on a key boundary. - */ - ++visit_cnt; - if (!removing_key_block && - (cnt == 0 || (visit_cnt > WT_LAS_SWEEP_ENTRIES && cache->las_reader))) - break; - if (cnt > 0) - --cnt; - - /* - * If the entry belongs to a dropped tree, discard it. - * - * Cursor opened overwrite=true: won't return WT_NOTFOUND should another thread remove the - * record before we do (not expected for dropped trees), and the cursor remains positioned - * in that case. - */ - if (las_id >= cache->las_sweep_dropmin && las_id <= cache->las_sweep_dropmax && - __bit_test(cache->las_sweep_dropmap, las_id - cache->las_sweep_dropmin)) { - WT_ERR(cursor->remove(cursor)); - ++remove_cnt; - saved_key->size = 0; - /* - * Allow sweep to break while removing entries from a dead file. - */ - removing_key_block = false; - continue; - } - - /* - * Remove all entries for a key once they have aged out and are no longer needed. - */ - WT_ERR(cursor->get_value(cursor, &las_txnid, &las_timestamp, &durable_timestamp, - &prepare_state, &upd_type, &las_value)); - - /* - * Check to see if the page or key has changed this iteration, and if they have, setup - * context for safely removing obsolete updates. - * - * It's important to check for page boundaries explicitly because it is possible for the - * same key to be at the start of the next block. See WT-3982 for details. - */ - if (las_pageid != saved_pageid || saved_key->size != las_key.size || - memcmp(saved_key->data, las_key.data, las_key.size) != 0) { - /* If we've examined enough entries, give up. */ - if (cnt == 0) - break; - - saved_pageid = las_pageid; - WT_ERR(__wt_buf_set(session, saved_key, las_key.data, las_key.size)); - - /* - * Expect an update entry with: - * 1. not in a prepare locked state - * 2. durable timestamp as not max timestamp. - * 3. for an in-progress prepared update, durable - * timestamp should be zero. - * 4. no restriction on durable timestamp value - * for other updates. - */ - WT_ASSERT(session, prepare_state != WT_PREPARE_LOCKED && - durable_timestamp != WT_TS_MAX && - (prepare_state != WT_PREPARE_INPROGRESS || durable_timestamp == 0)); - - WT_ASSERT(session, - (prepare_state == WT_PREPARE_INPROGRESS || durable_timestamp >= las_timestamp)); - - /* - * There are several conditions that need to be met - * before we choose to remove a key block: - * * The entries were written with skew newest. - * Indicated by the first entry being a birthmark. - * * The first entry is globally visible. - * * The entry wasn't from a prepared transaction. - */ - if (upd_type == WT_UPDATE_BIRTHMARK && - __wt_txn_visible_all(session, las_txnid, durable_timestamp) && - prepare_state != WT_PREPARE_INPROGRESS) - removing_key_block = true; - else - removing_key_block = false; - } - - if (!removing_key_block) - continue; - - __wt_verbose(session, WT_VERB_LOOKASIDE_ACTIVITY, - "Sweep removing lookaside entry with " - "page ID: %" PRIu64 " btree ID: %" PRIu32 " saved key size: %" WT_SIZET_FMT - ", record type: %" PRIu8 " transaction ID: %" PRIu64, - las_pageid, las_id, saved_key->size, upd_type, las_txnid); - WT_ERR(cursor->remove(cursor)); - ++remove_cnt; - } - - /* - * If the loop terminates after completing a work unit, we will continue the table sweep next - * time. Get a local copy of the sweep key, we're going to reset the cursor; do so before - * calling cursor.remove, cursor.remove can discard our hazard pointer and the page could be - * evicted from underneath us. - */ - if (ret == 0) { - WT_ERR(__wt_cursor_get_raw_key(cursor, sweep_key)); - if (!WT_DATA_IN_ITEM(sweep_key)) - WT_ERR(__wt_buf_set(session, sweep_key, sweep_key->data, sweep_key->size)); - } - -srch_notfound: - WT_ERR_NOTFOUND_OK(ret); - - if (0) { -err: - __wt_buf_free(session, sweep_key); - } - if (local_txn) { - if (ret == 0) - ret = __wt_txn_commit(session, NULL); - else - WT_TRET(__wt_txn_rollback(session, NULL)); - if (ret == 0) - (void)__wt_atomic_add64(&cache->las_remove_count, remove_cnt); - } - - WT_TRET(__wt_las_cursor_close(session, &cursor, session_flags)); - - if (locked) - __wt_writeunlock(session, &cache->las_sweepwalk_lock); - - __wt_scr_free(session, &saved_key); - - return (ret); -} diff --git a/src/third_party/wiredtiger/src/config/config_def.c b/src/third_party/wiredtiger/src/config/config_def.c index bba4a9b914b..e42c9255c41 100644 --- a/src/third_party/wiredtiger/src/config/config_def.c +++ b/src/third_party/wiredtiger/src/config/config_def.c @@ -70,6 +70,9 @@ static const WT_CONFIG_CHECK confchk_wiredtiger_open_file_manager_subconfigs[] = {"close_scan_interval", "int", NULL, "min=1,max=100000", NULL, 0}, {NULL, NULL, NULL, NULL, NULL, 0}}; +static const WT_CONFIG_CHECK confchk_wiredtiger_open_history_store_subconfigs[] = { + {"file_max", "int", NULL, "min=0", NULL, 0}, {NULL, NULL, NULL, NULL, NULL, 0}}; + static const WT_CONFIG_CHECK confchk_wiredtiger_open_io_capacity_subconfigs[] = { {"total", "int", NULL, "min=0,max=1TB", NULL, 0}, {NULL, NULL, NULL, NULL, NULL, 0}}; @@ -115,6 +118,7 @@ static const WT_CONFIG_CHECK confchk_WT_CONNECTION_reconfigure[] = { {"eviction_target", "int", NULL, "min=10,max=10TB", NULL, 0}, {"eviction_trigger", "int", NULL, "min=10,max=10TB", NULL, 0}, {"file_manager", "category", NULL, NULL, confchk_wiredtiger_open_file_manager_subconfigs, 3}, + {"history_store", "category", NULL, NULL, confchk_wiredtiger_open_history_store_subconfigs, 1}, {"io_capacity", "category", NULL, NULL, confchk_wiredtiger_open_io_capacity_subconfigs, 1}, {"log", "category", NULL, NULL, confchk_WT_CONNECTION_reconfigure_log_subconfigs, 4}, {"lsm_manager", "category", NULL, NULL, confchk_wiredtiger_open_lsm_manager_subconfigs, 2}, @@ -130,19 +134,20 @@ static const WT_CONFIG_CHECK confchk_WT_CONNECTION_reconfigure[] = { confchk_WT_CONNECTION_reconfigure_statistics_log_subconfigs, 5}, {"timing_stress_for_test", "list", NULL, "choices=[\"aggressive_sweep\",\"checkpoint_slow\"," - "\"lookaside_sweep_race\",\"split_1\",\"split_2\",\"split_3\"," - "\"split_4\",\"split_5\",\"split_6\",\"split_7\",\"split_8\"]", + "\"history_store_sweep_race\",\"split_1\",\"split_2\",\"split_3\"" + ",\"split_4\",\"split_5\",\"split_6\",\"split_7\",\"split_8\"]", NULL, 0}, {"verbose", "list", NULL, "choices=[\"api\",\"backup\",\"block\",\"checkpoint\"," - "\"checkpoint_progress\",\"compact\",\"compact_progress\"," - "\"error_returns\",\"evict\",\"evict_stuck\",\"evictserver\"," - "\"fileops\",\"handleops\",\"log\",\"lookaside\"," - "\"lookaside_activity\",\"lsm\",\"lsm_manager\",\"metadata\"," - "\"mutex\",\"overflow\",\"read\",\"rebalance\",\"reconcile\"," - "\"recovery\",\"recovery_progress\",\"salvage\",\"shared_cache\"," - "\"split\",\"temporary\",\"thread_group\",\"timestamp\"," - "\"transaction\",\"verify\",\"version\",\"write\"]", + "\"checkpoint_gc\",\"checkpoint_progress\",\"compact\"," + "\"compact_progress\",\"error_returns\",\"evict\",\"evict_stuck\"" + ",\"evictserver\",\"fileops\",\"handleops\",\"log\"," + "\"history_store\",\"history_store_activity\",\"lsm\"," + "\"lsm_manager\",\"metadata\",\"mutex\",\"overflow\",\"read\"," + "\"rebalance\",\"reconcile\",\"recovery\",\"recovery_progress\"," + "\"rts\",\"salvage\",\"shared_cache\",\"split\",\"temporary\"," + "\"thread_group\",\"timestamp\",\"transaction\",\"verify\"," + "\"version\",\"write\"]", NULL, 0}, {NULL, NULL, NULL, NULL, NULL, 0}}; @@ -196,8 +201,7 @@ static const WT_CONFIG_CHECK confchk_WT_SESSION_begin_transaction[] = { {"read_timestamp", "string", NULL, NULL, NULL, 0}, {"roundup_timestamps", "category", NULL, NULL, confchk_WT_SESSION_begin_transaction_roundup_timestamps_subconfigs, 2}, - {"snapshot", "string", NULL, NULL, NULL, 0}, {"sync", "boolean", NULL, NULL, NULL, 0}, - {NULL, NULL, NULL, NULL, NULL, 0}}; + {"sync", "boolean", NULL, NULL, NULL, 0}, {NULL, NULL, NULL, NULL, NULL, 0}}; static const WT_CONFIG_CHECK confchk_WT_SESSION_checkpoint[] = { {"drop", "list", NULL, NULL, NULL, 0}, {"force", "boolean", NULL, NULL, NULL, 0}, @@ -340,16 +344,6 @@ static const WT_CONFIG_CHECK confchk_WT_SESSION_reconfigure[] = { static const WT_CONFIG_CHECK confchk_WT_SESSION_salvage[] = { {"force", "boolean", NULL, NULL, NULL, 0}, {NULL, NULL, NULL, NULL, NULL, 0}}; -static const WT_CONFIG_CHECK confchk_WT_SESSION_snapshot_drop_subconfigs[] = { - {"all", "boolean", NULL, NULL, NULL, 0}, {"before", "string", NULL, NULL, NULL, 0}, - {"names", "list", NULL, NULL, NULL, 0}, {"to", "string", NULL, NULL, NULL, 0}, - {NULL, NULL, NULL, NULL, NULL, 0}}; - -static const WT_CONFIG_CHECK confchk_WT_SESSION_snapshot[] = { - {"drop", "category", NULL, NULL, confchk_WT_SESSION_snapshot_drop_subconfigs, 4}, - {"include_updates", "boolean", NULL, NULL, NULL, 0}, {"name", "string", NULL, NULL, NULL, 0}, - {NULL, NULL, NULL, NULL, NULL, 0}}; - static const WT_CONFIG_CHECK confchk_WT_SESSION_timestamp_transaction[] = { {"commit_timestamp", "string", NULL, NULL, NULL, 0}, {"durable_timestamp", "string", NULL, NULL, NULL, 0}, @@ -361,8 +355,10 @@ static const WT_CONFIG_CHECK confchk_WT_SESSION_transaction_sync[] = { static const WT_CONFIG_CHECK confchk_WT_SESSION_verify[] = { {"dump_address", "boolean", NULL, NULL, NULL, 0}, {"dump_blocks", "boolean", NULL, NULL, NULL, 0}, - {"dump_layout", "boolean", NULL, NULL, NULL, 0}, {"dump_offsets", "list", NULL, NULL, NULL, 0}, - {"dump_pages", "boolean", NULL, NULL, NULL, 0}, {"strict", "boolean", NULL, NULL, NULL, 0}, + {"dump_history", "boolean", NULL, NULL, NULL, 0}, {"dump_layout", "boolean", NULL, NULL, NULL, 0}, + {"dump_offsets", "list", NULL, NULL, NULL, 0}, {"dump_pages", "boolean", NULL, NULL, NULL, 0}, + {"history_store", "boolean", NULL, NULL, NULL, 0}, + {"stable_timestamp", "boolean", NULL, NULL, NULL, 0}, {"strict", "boolean", NULL, NULL, NULL, 0}, {NULL, NULL, NULL, NULL, NULL, 0}}; static const WT_CONFIG_CHECK confchk_colgroup_meta[] = { @@ -556,7 +552,9 @@ static const WT_CONFIG_CHECK confchk_wiredtiger_open[] = { {"exclusive", "boolean", NULL, NULL, NULL, 0}, {"extensions", "list", NULL, NULL, NULL, 0}, {"file_extend", "list", NULL, "choices=[\"data\",\"log\"]", NULL, 0}, {"file_manager", "category", NULL, NULL, confchk_wiredtiger_open_file_manager_subconfigs, 3}, - {"hazard_max", "int", NULL, "min=15", NULL, 0}, {"in_memory", "boolean", NULL, NULL, NULL, 0}, + {"hazard_max", "int", NULL, "min=15", NULL, 0}, + {"history_store", "category", NULL, NULL, confchk_wiredtiger_open_history_store_subconfigs, 1}, + {"in_memory", "boolean", NULL, NULL, NULL, 0}, {"io_capacity", "category", NULL, NULL, confchk_wiredtiger_open_io_capacity_subconfigs, 1}, {"log", "category", NULL, NULL, confchk_wiredtiger_open_log_subconfigs, 9}, {"lsm_manager", "category", NULL, NULL, confchk_wiredtiger_open_lsm_manager_subconfigs, 2}, @@ -576,8 +574,8 @@ static const WT_CONFIG_CHECK confchk_wiredtiger_open[] = { {"statistics_log", "category", NULL, NULL, confchk_wiredtiger_open_statistics_log_subconfigs, 6}, {"timing_stress_for_test", "list", NULL, "choices=[\"aggressive_sweep\",\"checkpoint_slow\"," - "\"lookaside_sweep_race\",\"split_1\",\"split_2\",\"split_3\"," - "\"split_4\",\"split_5\",\"split_6\",\"split_7\",\"split_8\"]", + "\"history_store_sweep_race\",\"split_1\",\"split_2\",\"split_3\"" + ",\"split_4\",\"split_5\",\"split_6\",\"split_7\",\"split_8\"]", NULL, 0}, {"transaction_sync", "category", NULL, NULL, confchk_wiredtiger_open_transaction_sync_subconfigs, 2}, @@ -585,14 +583,15 @@ static const WT_CONFIG_CHECK confchk_wiredtiger_open[] = { {"use_environment_priv", "boolean", NULL, NULL, NULL, 0}, {"verbose", "list", NULL, "choices=[\"api\",\"backup\",\"block\",\"checkpoint\"," - "\"checkpoint_progress\",\"compact\",\"compact_progress\"," - "\"error_returns\",\"evict\",\"evict_stuck\",\"evictserver\"," - "\"fileops\",\"handleops\",\"log\",\"lookaside\"," - "\"lookaside_activity\",\"lsm\",\"lsm_manager\",\"metadata\"," - "\"mutex\",\"overflow\",\"read\",\"rebalance\",\"reconcile\"," - "\"recovery\",\"recovery_progress\",\"salvage\",\"shared_cache\"," - "\"split\",\"temporary\",\"thread_group\",\"timestamp\"," - "\"transaction\",\"verify\",\"version\",\"write\"]", + "\"checkpoint_gc\",\"checkpoint_progress\",\"compact\"," + "\"compact_progress\",\"error_returns\",\"evict\",\"evict_stuck\"" + ",\"evictserver\",\"fileops\",\"handleops\",\"log\"," + "\"history_store\",\"history_store_activity\",\"lsm\"," + "\"lsm_manager\",\"metadata\",\"mutex\",\"overflow\",\"read\"," + "\"rebalance\",\"reconcile\",\"recovery\",\"recovery_progress\"," + "\"rts\",\"salvage\",\"shared_cache\",\"split\",\"temporary\"," + "\"thread_group\",\"timestamp\",\"transaction\",\"verify\"," + "\"version\",\"write\"]", NULL, 0}, {"write_through", "list", NULL, "choices=[\"data\",\"log\"]", NULL, 0}, {NULL, NULL, NULL, NULL, NULL, 0}}; @@ -623,7 +622,9 @@ static const WT_CONFIG_CHECK confchk_wiredtiger_open_all[] = { {"exclusive", "boolean", NULL, NULL, NULL, 0}, {"extensions", "list", NULL, NULL, NULL, 0}, {"file_extend", "list", NULL, "choices=[\"data\",\"log\"]", NULL, 0}, {"file_manager", "category", NULL, NULL, confchk_wiredtiger_open_file_manager_subconfigs, 3}, - {"hazard_max", "int", NULL, "min=15", NULL, 0}, {"in_memory", "boolean", NULL, NULL, NULL, 0}, + {"hazard_max", "int", NULL, "min=15", NULL, 0}, + {"history_store", "category", NULL, NULL, confchk_wiredtiger_open_history_store_subconfigs, 1}, + {"in_memory", "boolean", NULL, NULL, NULL, 0}, {"io_capacity", "category", NULL, NULL, confchk_wiredtiger_open_io_capacity_subconfigs, 1}, {"log", "category", NULL, NULL, confchk_wiredtiger_open_log_subconfigs, 9}, {"lsm_manager", "category", NULL, NULL, confchk_wiredtiger_open_lsm_manager_subconfigs, 2}, @@ -643,8 +644,8 @@ static const WT_CONFIG_CHECK confchk_wiredtiger_open_all[] = { {"statistics_log", "category", NULL, NULL, confchk_wiredtiger_open_statistics_log_subconfigs, 6}, {"timing_stress_for_test", "list", NULL, "choices=[\"aggressive_sweep\",\"checkpoint_slow\"," - "\"lookaside_sweep_race\",\"split_1\",\"split_2\",\"split_3\"," - "\"split_4\",\"split_5\",\"split_6\",\"split_7\",\"split_8\"]", + "\"history_store_sweep_race\",\"split_1\",\"split_2\",\"split_3\"" + ",\"split_4\",\"split_5\",\"split_6\",\"split_7\",\"split_8\"]", NULL, 0}, {"transaction_sync", "category", NULL, NULL, confchk_wiredtiger_open_transaction_sync_subconfigs, 2}, @@ -652,14 +653,15 @@ static const WT_CONFIG_CHECK confchk_wiredtiger_open_all[] = { {"use_environment_priv", "boolean", NULL, NULL, NULL, 0}, {"verbose", "list", NULL, "choices=[\"api\",\"backup\",\"block\",\"checkpoint\"," - "\"checkpoint_progress\",\"compact\",\"compact_progress\"," - "\"error_returns\",\"evict\",\"evict_stuck\",\"evictserver\"," - "\"fileops\",\"handleops\",\"log\",\"lookaside\"," - "\"lookaside_activity\",\"lsm\",\"lsm_manager\",\"metadata\"," - "\"mutex\",\"overflow\",\"read\",\"rebalance\",\"reconcile\"," - "\"recovery\",\"recovery_progress\",\"salvage\",\"shared_cache\"," - "\"split\",\"temporary\",\"thread_group\",\"timestamp\"," - "\"transaction\",\"verify\",\"version\",\"write\"]", + "\"checkpoint_gc\",\"checkpoint_progress\",\"compact\"," + "\"compact_progress\",\"error_returns\",\"evict\",\"evict_stuck\"" + ",\"evictserver\",\"fileops\",\"handleops\",\"log\"," + "\"history_store\",\"history_store_activity\",\"lsm\"," + "\"lsm_manager\",\"metadata\",\"mutex\",\"overflow\",\"read\"," + "\"rebalance\",\"reconcile\",\"recovery\",\"recovery_progress\"," + "\"rts\",\"salvage\",\"shared_cache\",\"split\",\"temporary\"," + "\"thread_group\",\"timestamp\",\"transaction\",\"verify\"," + "\"version\",\"write\"]", NULL, 0}, {"version", "string", NULL, NULL, NULL, 0}, {"write_through", "list", NULL, "choices=[\"data\",\"log\"]", NULL, 0}, @@ -691,6 +693,7 @@ static const WT_CONFIG_CHECK confchk_wiredtiger_open_basecfg[] = { {"file_extend", "list", NULL, "choices=[\"data\",\"log\"]", NULL, 0}, {"file_manager", "category", NULL, NULL, confchk_wiredtiger_open_file_manager_subconfigs, 3}, {"hazard_max", "int", NULL, "min=15", NULL, 0}, + {"history_store", "category", NULL, NULL, confchk_wiredtiger_open_history_store_subconfigs, 1}, {"io_capacity", "category", NULL, NULL, confchk_wiredtiger_open_io_capacity_subconfigs, 1}, {"log", "category", NULL, NULL, confchk_wiredtiger_open_log_subconfigs, 9}, {"lsm_manager", "category", NULL, NULL, confchk_wiredtiger_open_lsm_manager_subconfigs, 2}, @@ -710,21 +713,22 @@ static const WT_CONFIG_CHECK confchk_wiredtiger_open_basecfg[] = { {"statistics_log", "category", NULL, NULL, confchk_wiredtiger_open_statistics_log_subconfigs, 6}, {"timing_stress_for_test", "list", NULL, "choices=[\"aggressive_sweep\",\"checkpoint_slow\"," - "\"lookaside_sweep_race\",\"split_1\",\"split_2\",\"split_3\"," - "\"split_4\",\"split_5\",\"split_6\",\"split_7\",\"split_8\"]", + "\"history_store_sweep_race\",\"split_1\",\"split_2\",\"split_3\"" + ",\"split_4\",\"split_5\",\"split_6\",\"split_7\",\"split_8\"]", NULL, 0}, {"transaction_sync", "category", NULL, NULL, confchk_wiredtiger_open_transaction_sync_subconfigs, 2}, {"verbose", "list", NULL, "choices=[\"api\",\"backup\",\"block\",\"checkpoint\"," - "\"checkpoint_progress\",\"compact\",\"compact_progress\"," - "\"error_returns\",\"evict\",\"evict_stuck\",\"evictserver\"," - "\"fileops\",\"handleops\",\"log\",\"lookaside\"," - "\"lookaside_activity\",\"lsm\",\"lsm_manager\",\"metadata\"," - "\"mutex\",\"overflow\",\"read\",\"rebalance\",\"reconcile\"," - "\"recovery\",\"recovery_progress\",\"salvage\",\"shared_cache\"," - "\"split\",\"temporary\",\"thread_group\",\"timestamp\"," - "\"transaction\",\"verify\",\"version\",\"write\"]", + "\"checkpoint_gc\",\"checkpoint_progress\",\"compact\"," + "\"compact_progress\",\"error_returns\",\"evict\",\"evict_stuck\"" + ",\"evictserver\",\"fileops\",\"handleops\",\"log\"," + "\"history_store\",\"history_store_activity\",\"lsm\"," + "\"lsm_manager\",\"metadata\",\"mutex\",\"overflow\",\"read\"," + "\"rebalance\",\"reconcile\",\"recovery\",\"recovery_progress\"," + "\"rts\",\"salvage\",\"shared_cache\",\"split\",\"temporary\"," + "\"thread_group\",\"timestamp\",\"transaction\",\"verify\"," + "\"version\",\"write\"]", NULL, 0}, {"version", "string", NULL, NULL, NULL, 0}, {"write_through", "list", NULL, "choices=[\"data\",\"log\"]", NULL, 0}, @@ -756,6 +760,7 @@ static const WT_CONFIG_CHECK confchk_wiredtiger_open_usercfg[] = { {"file_extend", "list", NULL, "choices=[\"data\",\"log\"]", NULL, 0}, {"file_manager", "category", NULL, NULL, confchk_wiredtiger_open_file_manager_subconfigs, 3}, {"hazard_max", "int", NULL, "min=15", NULL, 0}, + {"history_store", "category", NULL, NULL, confchk_wiredtiger_open_history_store_subconfigs, 1}, {"io_capacity", "category", NULL, NULL, confchk_wiredtiger_open_io_capacity_subconfigs, 1}, {"log", "category", NULL, NULL, confchk_wiredtiger_open_log_subconfigs, 9}, {"lsm_manager", "category", NULL, NULL, confchk_wiredtiger_open_lsm_manager_subconfigs, 2}, @@ -775,21 +780,22 @@ static const WT_CONFIG_CHECK confchk_wiredtiger_open_usercfg[] = { {"statistics_log", "category", NULL, NULL, confchk_wiredtiger_open_statistics_log_subconfigs, 6}, {"timing_stress_for_test", "list", NULL, "choices=[\"aggressive_sweep\",\"checkpoint_slow\"," - "\"lookaside_sweep_race\",\"split_1\",\"split_2\",\"split_3\"," - "\"split_4\",\"split_5\",\"split_6\",\"split_7\",\"split_8\"]", + "\"history_store_sweep_race\",\"split_1\",\"split_2\",\"split_3\"" + ",\"split_4\",\"split_5\",\"split_6\",\"split_7\",\"split_8\"]", NULL, 0}, {"transaction_sync", "category", NULL, NULL, confchk_wiredtiger_open_transaction_sync_subconfigs, 2}, {"verbose", "list", NULL, "choices=[\"api\",\"backup\",\"block\",\"checkpoint\"," - "\"checkpoint_progress\",\"compact\",\"compact_progress\"," - "\"error_returns\",\"evict\",\"evict_stuck\",\"evictserver\"," - "\"fileops\",\"handleops\",\"log\",\"lookaside\"," - "\"lookaside_activity\",\"lsm\",\"lsm_manager\",\"metadata\"," - "\"mutex\",\"overflow\",\"read\",\"rebalance\",\"reconcile\"," - "\"recovery\",\"recovery_progress\",\"salvage\",\"shared_cache\"," - "\"split\",\"temporary\",\"thread_group\",\"timestamp\"," - "\"transaction\",\"verify\",\"version\",\"write\"]", + "\"checkpoint_gc\",\"checkpoint_progress\",\"compact\"," + "\"compact_progress\",\"error_returns\",\"evict\",\"evict_stuck\"" + ",\"evictserver\",\"fileops\",\"handleops\",\"log\"," + "\"history_store\",\"history_store_activity\",\"lsm\"," + "\"lsm_manager\",\"metadata\",\"mutex\",\"overflow\",\"read\"," + "\"rebalance\",\"reconcile\",\"recovery\",\"recovery_progress\"," + "\"rts\",\"salvage\",\"shared_cache\",\"split\",\"temporary\"," + "\"thread_group\",\"timestamp\",\"transaction\",\"verify\"," + "\"version\",\"write\"]", NULL, 0}, {"write_through", "list", NULL, "choices=[\"data\",\"log\"]", NULL, 0}, {NULL, NULL, NULL, NULL, NULL, 0}}; @@ -824,15 +830,16 @@ static const WT_CONFIG_ENTRY config_entries[] = {{"WT_CONNECTION.add_collator", "eviction_checkpoint_target=1,eviction_dirty_target=5," "eviction_dirty_trigger=20,eviction_target=80,eviction_trigger=95" ",file_manager=(close_handle_minimum=250,close_idle_time=30," - "close_scan_interval=10),io_capacity=(total=0),log=(archive=true," - "os_cache_dirty_pct=0,prealloc=true,zero_fill=false)," - "lsm_manager=(merge=true,worker_thread_max=4)," - "operation_timeout_ms=0,operation_tracking=(enabled=false," - "path=\".\"),shared_cache=(chunk=10MB,name=,quota=0,reserve=0," - "size=500MB),statistics=none,statistics_log=(json=false," - "on_close=false,sources=,timestamp=\"%b %d %H:%M:%S\",wait=0)," + "close_scan_interval=10),history_store=(file_max=0)," + "io_capacity=(total=0),log=(archive=true,os_cache_dirty_pct=0," + "prealloc=true,zero_fill=false),lsm_manager=(merge=true," + "worker_thread_max=4),operation_timeout_ms=0," + "operation_tracking=(enabled=false,path=\".\")," + "shared_cache=(chunk=10MB,name=,quota=0,reserve=0,size=500MB)," + "statistics=none,statistics_log=(json=false,on_close=false," + "sources=,timestamp=\"%b %d %H:%M:%S\",wait=0)," "timing_stress_for_test=,verbose=", - confchk_WT_CONNECTION_reconfigure, 26}, + confchk_WT_CONNECTION_reconfigure, 27}, {"WT_CONNECTION.rollback_to_stable", "", NULL, 0}, {"WT_CONNECTION.set_file_system", "", NULL, 0}, {"WT_CONNECTION.set_timestamp", "commit_timestamp=,durable_timestamp=,force=false," @@ -850,8 +857,8 @@ static const WT_CONFIG_ENTRY config_entries[] = {{"WT_CONNECTION.add_collator", {"WT_SESSION.begin_transaction", "ignore_prepare=false,isolation=,name=,operation_timeout_ms=0," "priority=0,read_timestamp=,roundup_timestamps=(prepared=false," - "read=false),snapshot=,sync=", - confchk_WT_SESSION_begin_transaction, 9}, + "read=false),sync=", + confchk_WT_SESSION_begin_transaction, 8}, {"WT_SESSION.checkpoint", "drop=,force=false,name=,target=,use_timestamp=true", confchk_WT_SESSION_checkpoint, 5}, {"WT_SESSION.close", "", NULL, 0}, @@ -909,8 +916,6 @@ static const WT_CONFIG_ENTRY config_entries[] = {{"WT_CONNECTION.add_collator", {"WT_SESSION.rename", "", NULL, 0}, {"WT_SESSION.reset", "", NULL, 0}, {"WT_SESSION.rollback_transaction", "", NULL, 0}, {"WT_SESSION.salvage", "force=false", confchk_WT_SESSION_salvage, 1}, - {"WT_SESSION.snapshot", "drop=(all=false,before=,names=,to=),include_updates=false,name=", - confchk_WT_SESSION_snapshot, 3}, {"WT_SESSION.strerror", "", NULL, 0}, {"WT_SESSION.timestamp_transaction", "commit_timestamp=,durable_timestamp=,prepare_timestamp=," "read_timestamp=", @@ -918,9 +923,10 @@ static const WT_CONFIG_ENTRY config_entries[] = {{"WT_CONNECTION.add_collator", {"WT_SESSION.transaction_sync", "timeout_ms=1200000", confchk_WT_SESSION_transaction_sync, 1}, {"WT_SESSION.truncate", "", NULL, 0}, {"WT_SESSION.upgrade", "", NULL, 0}, {"WT_SESSION.verify", - "dump_address=false,dump_blocks=false,dump_layout=false," - "dump_offsets=,dump_pages=false,strict=false", - confchk_WT_SESSION_verify, 6}, + "dump_address=false,dump_blocks=false,dump_history=false," + "dump_layout=false,dump_offsets=,dump_pages=false," + "history_store=false,stable_timestamp=false,strict=false", + confchk_WT_SESSION_verify, 9}, {"colgroup.meta", "app_metadata=,collator=,columns=,source=,type=file", confchk_colgroup_meta, 5}, {"file.config", "access_pattern_hint=none,allocation_size=4KB,app_metadata=," @@ -1000,12 +1006,12 @@ static const WT_CONFIG_ENTRY config_entries[] = {{"WT_CONNECTION.add_collator", "eviction_dirty_trigger=20,eviction_target=80,eviction_trigger=95" ",exclusive=false,extensions=,file_extend=," "file_manager=(close_handle_minimum=250,close_idle_time=30," - "close_scan_interval=10),hazard_max=1000,in_memory=false," - "io_capacity=(total=0),log=(archive=true,compressor=," - "enabled=false,file_max=100MB,os_cache_dirty_pct=0,path=\".\"," - "prealloc=true,recover=on,zero_fill=false)," - "lsm_manager=(merge=true,worker_thread_max=4),mmap=true," - "multiprocess=false,operation_timeout_ms=0," + "close_scan_interval=10),hazard_max=1000," + "history_store=(file_max=0),in_memory=false,io_capacity=(total=0)" + ",log=(archive=true,compressor=,enabled=false,file_max=100MB," + "os_cache_dirty_pct=0,path=\".\",prealloc=true,recover=on," + "zero_fill=false),lsm_manager=(merge=true,worker_thread_max=4)," + "mmap=true,multiprocess=false,operation_timeout_ms=0," "operation_tracking=(enabled=false,path=\".\"),readonly=false," "salvage=false,session_max=100,session_scratch_max=2MB," "session_table_cache=true,shared_cache=(chunk=10MB,name=,quota=0," @@ -1014,7 +1020,7 @@ static const WT_CONFIG_ENTRY config_entries[] = {{"WT_CONNECTION.add_collator", ",wait=0),timing_stress_for_test=,transaction_sync=(enabled=false" ",method=fsync),use_environment=true,use_environment_priv=false," "verbose=,write_through=", - confchk_wiredtiger_open, 50}, + confchk_wiredtiger_open, 51}, {"wiredtiger_open_all", "async=(enabled=false,ops_max=1024,threads=2),buffer_alignment=-1" ",builtin_extension_config=,cache_cursors=true," @@ -1031,12 +1037,12 @@ static const WT_CONFIG_ENTRY config_entries[] = {{"WT_CONNECTION.add_collator", "eviction_dirty_trigger=20,eviction_target=80,eviction_trigger=95" ",exclusive=false,extensions=,file_extend=," "file_manager=(close_handle_minimum=250,close_idle_time=30," - "close_scan_interval=10),hazard_max=1000,in_memory=false," - "io_capacity=(total=0),log=(archive=true,compressor=," - "enabled=false,file_max=100MB,os_cache_dirty_pct=0,path=\".\"," - "prealloc=true,recover=on,zero_fill=false)," - "lsm_manager=(merge=true,worker_thread_max=4),mmap=true," - "multiprocess=false,operation_timeout_ms=0," + "close_scan_interval=10),hazard_max=1000," + "history_store=(file_max=0),in_memory=false,io_capacity=(total=0)" + ",log=(archive=true,compressor=,enabled=false,file_max=100MB," + "os_cache_dirty_pct=0,path=\".\",prealloc=true,recover=on," + "zero_fill=false),lsm_manager=(merge=true,worker_thread_max=4)," + "mmap=true,multiprocess=false,operation_timeout_ms=0," "operation_tracking=(enabled=false,path=\".\"),readonly=false," "salvage=false,session_max=100,session_scratch_max=2MB," "session_table_cache=true,shared_cache=(chunk=10MB,name=,quota=0," @@ -1045,7 +1051,7 @@ static const WT_CONFIG_ENTRY config_entries[] = {{"WT_CONNECTION.add_collator", ",wait=0),timing_stress_for_test=,transaction_sync=(enabled=false" ",method=fsync),use_environment=true,use_environment_priv=false," "verbose=,version=(major=0,minor=0),write_through=", - confchk_wiredtiger_open_all, 51}, + confchk_wiredtiger_open_all, 52}, {"wiredtiger_open_basecfg", "async=(enabled=false,ops_max=1024,threads=2),buffer_alignment=-1" ",builtin_extension_config=,cache_cursors=true," @@ -1061,11 +1067,11 @@ static const WT_CONFIG_ENTRY config_entries[] = {{"WT_CONNECTION.add_collator", "eviction_dirty_trigger=20,eviction_target=80,eviction_trigger=95" ",extensions=,file_extend=,file_manager=(close_handle_minimum=250" ",close_idle_time=30,close_scan_interval=10),hazard_max=1000," - "io_capacity=(total=0),log=(archive=true,compressor=," - "enabled=false,file_max=100MB,os_cache_dirty_pct=0,path=\".\"," - "prealloc=true,recover=on,zero_fill=false)," - "lsm_manager=(merge=true,worker_thread_max=4),mmap=true," - "multiprocess=false,operation_timeout_ms=0," + "history_store=(file_max=0),io_capacity=(total=0)," + "log=(archive=true,compressor=,enabled=false,file_max=100MB," + "os_cache_dirty_pct=0,path=\".\",prealloc=true,recover=on," + "zero_fill=false),lsm_manager=(merge=true,worker_thread_max=4)," + "mmap=true,multiprocess=false,operation_timeout_ms=0," "operation_tracking=(enabled=false,path=\".\"),readonly=false," "salvage=false,session_max=100,session_scratch_max=2MB," "session_table_cache=true,shared_cache=(chunk=10MB,name=,quota=0," @@ -1073,7 +1079,7 @@ static const WT_CONFIG_ENTRY config_entries[] = {{"WT_CONNECTION.add_collator", ",on_close=false,path=\".\",sources=,timestamp=\"%b %d %H:%M:%S\"" ",wait=0),timing_stress_for_test=,transaction_sync=(enabled=false" ",method=fsync),verbose=,version=(major=0,minor=0),write_through=", - confchk_wiredtiger_open_basecfg, 45}, + confchk_wiredtiger_open_basecfg, 46}, {"wiredtiger_open_usercfg", "async=(enabled=false,ops_max=1024,threads=2),buffer_alignment=-1" ",builtin_extension_config=,cache_cursors=true," @@ -1089,11 +1095,11 @@ static const WT_CONFIG_ENTRY config_entries[] = {{"WT_CONNECTION.add_collator", "eviction_dirty_trigger=20,eviction_target=80,eviction_trigger=95" ",extensions=,file_extend=,file_manager=(close_handle_minimum=250" ",close_idle_time=30,close_scan_interval=10),hazard_max=1000," - "io_capacity=(total=0),log=(archive=true,compressor=," - "enabled=false,file_max=100MB,os_cache_dirty_pct=0,path=\".\"," - "prealloc=true,recover=on,zero_fill=false)," - "lsm_manager=(merge=true,worker_thread_max=4),mmap=true," - "multiprocess=false,operation_timeout_ms=0," + "history_store=(file_max=0),io_capacity=(total=0)," + "log=(archive=true,compressor=,enabled=false,file_max=100MB," + "os_cache_dirty_pct=0,path=\".\",prealloc=true,recover=on," + "zero_fill=false),lsm_manager=(merge=true,worker_thread_max=4)," + "mmap=true,multiprocess=false,operation_timeout_ms=0," "operation_tracking=(enabled=false,path=\".\"),readonly=false," "salvage=false,session_max=100,session_scratch_max=2MB," "session_table_cache=true,shared_cache=(chunk=10MB,name=,quota=0," @@ -1101,7 +1107,7 @@ static const WT_CONFIG_ENTRY config_entries[] = {{"WT_CONNECTION.add_collator", ",on_close=false,path=\".\",sources=,timestamp=\"%b %d %H:%M:%S\"" ",wait=0),timing_stress_for_test=,transaction_sync=(enabled=false" ",method=fsync),verbose=,write_through=", - confchk_wiredtiger_open_usercfg, 44}, + confchk_wiredtiger_open_usercfg, 45}, {NULL, NULL, NULL, 0}}; int diff --git a/src/third_party/wiredtiger/src/conn/api_calc_modify.c b/src/third_party/wiredtiger/src/conn/api_calc_modify.c index b22ea055c5e..86912dfbd79 100644 --- a/src/third_party/wiredtiger/src/conn/api_calc_modify.c +++ b/src/third_party/wiredtiger/src/conn/api_calc_modify.c @@ -105,11 +105,11 @@ __cm_fingerprint(const uint8_t *p) } /* - * wiredtiger_calc_modify -- + * __wt_calc_modify -- * Calculate a set of WT_MODIFY operations to represent an update. */ int -wiredtiger_calc_modify(WT_SESSION *wt_session, const WT_ITEM *oldv, const WT_ITEM *newv, +__wt_calc_modify(WT_SESSION_IMPL *wt_session, const WT_ITEM *oldv, const WT_ITEM *newv, size_t maxdiff, WT_MODIFY *entries, int *nentriesp) { WT_CM_MATCH match; @@ -191,3 +191,14 @@ end: return (0); } + +/* + * wiredtiger_calc_modify -- + * Calculate a set of WT_MODIFY operations to represent an update. + */ +int +wiredtiger_calc_modify(WT_SESSION *wt_session, const WT_ITEM *oldv, const WT_ITEM *newv, + size_t maxdiff, WT_MODIFY *entries, int *nentriesp) +{ + return __wt_calc_modify((WT_SESSION_IMPL *)wt_session, oldv, newv, maxdiff, entries, nentriesp); +} diff --git a/src/third_party/wiredtiger/src/conn/conn_api.c b/src/third_party/wiredtiger/src/conn/conn_api.c index 65580de7a7b..c904f4702b3 100644 --- a/src/third_party/wiredtiger/src/conn/conn_api.c +++ b/src/third_party/wiredtiger/src/conn/conn_api.c @@ -1038,9 +1038,6 @@ err: WT_TRET(wt_session->rollback_transaction(wt_session, NULL)); } - /* Release all named snapshots. */ - __wt_txn_named_snapshot_destroy(session); - /* Close open, external sessions. */ for (s = conn->sessions, i = 0; i < conn->session_cnt; ++s, ++i) if (s->active && !F_ISSET(s, WT_SESSION_INTERNAL)) { @@ -1058,12 +1055,6 @@ err: WT_TRET(__wt_txn_activity_drain(session)); /* - * Disable lookaside eviction: it doesn't help us shut down and can lead to pages being marked - * dirty, causing spurious assertions to fire. - */ - F_SET(conn, WT_CONN_EVICTION_NO_LOOKASIDE); - - /* * Clear any pending async operations and shut down the async worker threads and system before * closing LSM. */ @@ -1246,7 +1237,8 @@ __conn_rollback_to_stable(WT_CONNECTION *wt_conn, const char *config) conn = (WT_CONNECTION_IMPL *)wt_conn; CONNECTION_API_CALL(conn, session, rollback_to_stable, config, cfg); - WT_TRET(__wt_txn_rollback_to_stable(session, cfg)); + WT_STAT_CONN_INCR(session, txn_rts); + WT_TRET(__wt_rollback_to_stable(session, cfg, false)); err: API_END_RET(session, ret); } @@ -1841,17 +1833,18 @@ __wt_verbose_config(WT_SESSION_IMPL *session, const char *cfg[]) { static const WT_NAME_FLAG verbtypes[] = {{"api", WT_VERB_API}, {"backup", WT_VERB_BACKUP}, {"block", WT_VERB_BLOCK}, {"checkpoint", WT_VERB_CHECKPOINT}, + {"checkpoint_gc", WT_VERB_CHECKPOINT_GC}, {"checkpoint_progress", WT_VERB_CHECKPOINT_PROGRESS}, {"compact", WT_VERB_COMPACT}, {"compact_progress", WT_VERB_COMPACT_PROGRESS}, {"error_returns", WT_VERB_ERROR_RETURNS}, {"evict", WT_VERB_EVICT}, {"evict_stuck", WT_VERB_EVICT_STUCK}, {"evictserver", WT_VERB_EVICTSERVER}, {"fileops", WT_VERB_FILEOPS}, - {"handleops", WT_VERB_HANDLEOPS}, {"log", WT_VERB_LOG}, {"lookaside", WT_VERB_LOOKASIDE}, - {"lookaside_activity", WT_VERB_LOOKASIDE_ACTIVITY}, {"lsm", WT_VERB_LSM}, + {"handleops", WT_VERB_HANDLEOPS}, {"log", WT_VERB_LOG}, {"hs", WT_VERB_HS}, + {"history_store_activity", WT_VERB_HS_ACTIVITY}, {"lsm", WT_VERB_LSM}, {"lsm_manager", WT_VERB_LSM_MANAGER}, {"metadata", WT_VERB_METADATA}, {"mutex", WT_VERB_MUTEX}, {"overflow", WT_VERB_OVERFLOW}, {"read", WT_VERB_READ}, {"rebalance", WT_VERB_REBALANCE}, {"reconcile", WT_VERB_RECONCILE}, {"recovery", WT_VERB_RECOVERY}, {"recovery_progress", WT_VERB_RECOVERY_PROGRESS}, - {"salvage", WT_VERB_SALVAGE}, {"shared_cache", WT_VERB_SHARED_CACHE}, + {"rts", WT_VERB_RTS}, {"salvage", WT_VERB_SALVAGE}, {"shared_cache", WT_VERB_SHARED_CACHE}, {"split", WT_VERB_SPLIT}, {"temporary", WT_VERB_TEMPORARY}, {"thread_group", WT_VERB_THREAD_GROUP}, {"timestamp", WT_VERB_TIMESTAMP}, {"transaction", WT_VERB_TRANSACTION}, {"verify", WT_VERB_VERIFY}, @@ -1981,7 +1974,7 @@ __wt_timing_stress_config(WT_SESSION_IMPL *session, const char *cfg[]) static const WT_NAME_FLAG stress_types[] = { {"aggressive_sweep", WT_TIMING_STRESS_AGGRESSIVE_SWEEP}, {"checkpoint_slow", WT_TIMING_STRESS_CHECKPOINT_SLOW}, - {"lookaside_sweep_race", WT_TIMING_STRESS_LOOKASIDE_SWEEP}, + {"history_store_sweep_race", WT_TIMING_STRESS_HS_SWEEP}, {"split_1", WT_TIMING_STRESS_SPLIT_1}, {"split_2", WT_TIMING_STRESS_SPLIT_2}, {"split_3", WT_TIMING_STRESS_SPLIT_3}, {"split_4", WT_TIMING_STRESS_SPLIT_4}, {"split_5", WT_TIMING_STRESS_SPLIT_5}, {"split_6", WT_TIMING_STRESS_SPLIT_6}, @@ -2491,7 +2484,6 @@ wiredtiger_open(const char *home, WT_EVENT_HANDLER *event_handler, const char *c } WT_ERR(__wt_verbose_config(session, cfg)); WT_ERR(__wt_timing_stress_config(session, cfg)); - __wt_btree_page_version_config(session); /* Set up operation tracking if configured. */ WT_ERR(__wt_conn_optrack_setup(session, cfg, false)); @@ -2666,8 +2658,8 @@ wiredtiger_open(const char *home, WT_EVENT_HANDLER *event_handler, const char *c if (F_ISSET(conn, WT_CONN_SALVAGE)) WT_ERR(__wt_metadata_salvage(session)); - /* Set the connection's base write generation. */ - WT_ERR(__wt_metadata_set_base_write_gen(session)); + /* Initialize the connection's base write generation. */ + WT_ERR(__wt_metadata_init_base_write_gen(session)); WT_ERR(__wt_metadata_cursor(session, NULL)); diff --git a/src/third_party/wiredtiger/src/conn/conn_cache.c b/src/third_party/wiredtiger/src/conn/conn_cache.c index c4a5d5d145e..0512d0f1b17 100644 --- a/src/third_party/wiredtiger/src/conn/conn_cache.c +++ b/src/third_party/wiredtiger/src/conn/conn_cache.c @@ -189,8 +189,7 @@ __wt_cache_config(WT_SESSION_IMPL *session, bool reconfigure, const char *cfg[]) */ if (reconfigure) WT_RET(__wt_thread_group_resize(session, &conn->evict_threads, conn->evict_threads_min, - conn->evict_threads_max, - WT_THREAD_CAN_WAIT | WT_THREAD_LOOKASIDE | WT_THREAD_PANIC_FAIL)); + conn->evict_threads_max, WT_THREAD_CAN_WAIT | WT_THREAD_HS | WT_THREAD_PANIC_FAIL)); return (0); } @@ -239,10 +238,6 @@ __wt_cache_create(WT_SESSION_IMPL *session, const char *cfg[]) conn, "evict pass", false, WT_SESSION_NO_DATA_HANDLES, &cache->walk_session)) != 0) WT_RET_MSG(NULL, ret, "Failed to create session for eviction walks"); - WT_RET(__wt_rwlock_init(session, &cache->las_sweepwalk_lock)); - WT_RET(__wt_spin_init(session, &cache->las_lock, "lookaside table")); - WT_RET(__wt_spin_init(session, &cache->las_sweep_lock, "lookaside sweep")); - /* Allocate the LRU eviction queue. */ cache->evict_slots = WT_EVICT_WALK_BASE + WT_EVICT_WALK_INCR; for (i = 0; i < WT_EVICT_QUEUE_MAX; ++i) { @@ -296,9 +291,9 @@ __wt_cache_stats_update(WT_SESSION_IMPL *session) WT_STAT_SET(session, stats, cache_pages_inuse, __wt_cache_pages_inuse(cache)); WT_STAT_SET(session, stats, cache_bytes_internal, cache->bytes_internal); WT_STAT_SET(session, stats, cache_bytes_leaf, leaf); - if (F_ISSET(conn, WT_CONN_LOOKASIDE_OPEN)) { - WT_STAT_SET(session, stats, cache_bytes_lookaside, - __wt_cache_bytes_plus_overhead(cache, cache->bytes_lookaside)); + if (F_ISSET(conn, WT_CONN_HS_OPEN)) { + WT_STAT_SET( + session, stats, cache_bytes_hs, __wt_cache_bytes_plus_overhead(cache, cache->bytes_hs)); } WT_STAT_SET(session, stats, cache_bytes_other, __wt_cache_bytes_other(cache)); @@ -309,7 +304,7 @@ __wt_cache_stats_update(WT_SESSION_IMPL *session) WT_STAT_SET(session, stats, cache_eviction_state, cache->flags); WT_STAT_SET(session, stats, cache_eviction_aggressive_set, cache->evict_aggressive_score); WT_STAT_SET(session, stats, cache_eviction_empty_score, cache->evict_empty_score); - WT_STAT_SET(session, stats, cache_lookaside_score, __wt_cache_lookaside_score(cache)); + WT_STAT_SET(session, stats, cache_hs_score, __wt_cache_hs_score(cache)); WT_STAT_SET(session, stats, cache_eviction_active_workers, conn->evict_threads.current_threads); WT_STAT_SET( @@ -321,6 +316,9 @@ __wt_cache_stats_update(WT_SESSION_IMPL *session) */ if (conn->evict_server_running) WT_STAT_SET(session, stats, cache_eviction_walks_active, cache->walk_session->nhazard); + + /* TODO: WT-5585 Remove lookaside score statistic after MongoDB switches to an alternative. */ + WT_STAT_SET(session, stats, cache_lookaside_score, 0); } /* @@ -367,9 +365,6 @@ __wt_cache_destroy(WT_SESSION_IMPL *session) __wt_spin_destroy(session, &cache->evict_pass_lock); __wt_spin_destroy(session, &cache->evict_queue_lock); __wt_spin_destroy(session, &cache->evict_walk_lock); - __wt_spin_destroy(session, &cache->las_lock); - __wt_spin_destroy(session, &cache->las_sweep_lock); - __wt_rwlock_destroy(session, &cache->las_sweepwalk_lock); wt_session = &cache->walk_session->iface; if (wt_session != NULL) WT_TRET(wt_session->close(wt_session, NULL)); diff --git a/src/third_party/wiredtiger/src/conn/conn_dhandle.c b/src/third_party/wiredtiger/src/conn/conn_dhandle.c index 427eaa2d9a4..1c391200cb1 100644 --- a/src/third_party/wiredtiger/src/conn/conn_dhandle.c +++ b/src/third_party/wiredtiger/src/conn/conn_dhandle.c @@ -426,7 +426,8 @@ __wt_conn_dhandle_open(WT_SESSION_IMPL *session, const char *cfg[], uint32_t fla WT_ASSERT(session, F_ISSET(dhandle, WT_DHANDLE_EXCLUSIVE) && !LF_ISSET(WT_DHANDLE_LOCK_ONLY)); - WT_ASSERT(session, !F_ISSET(S2C(session), WT_CONN_CLOSING_NO_MORE_OPENS)); + WT_ASSERT(session, F_ISSET(session, WT_SESSION_ROLLBACK_TO_STABLE_FLAGS) || + !F_ISSET(S2C(session), WT_CONN_CLOSING_NO_MORE_OPENS)); /* Turn off eviction. */ if (dhandle->type == WT_DHANDLE_TYPE_BTREE) @@ -768,12 +769,13 @@ __wt_conn_dhandle_discard(WT_SESSION_IMPL *session) __wt_session_close_cache(session); /* - * Close open data handles: first, everything apart from metadata and lookaside (as closing a normal - * file may write metadata and read lookaside entries). Then close whatever is left open. + * Close open data handles: first, everything apart from metadata and the history store (as closing + * a normal file may write metadata and read history store entries). Then close whatever is left + * open. */ restart: TAILQ_FOREACH (dhandle, &conn->dhqh, q) { - if (WT_IS_METADATA(dhandle) || strcmp(dhandle->name, WT_LAS_URI) == 0 || + if (WT_IS_METADATA(dhandle) || strcmp(dhandle->name, WT_HS_URI) == 0 || WT_PREFIX_MATCH(dhandle->name, WT_SYSTEM_PREFIX)) continue; @@ -782,8 +784,8 @@ restart: goto restart; } - /* Shut down the lookaside table after all eviction is complete. */ - WT_TRET(__wt_las_destroy(session)); + /* Shut down the history store table after all eviction is complete. */ + __wt_hs_destroy(session); /* * Closing the files may have resulted in entries on our default session's list of open data diff --git a/src/third_party/wiredtiger/src/conn/conn_open.c b/src/third_party/wiredtiger/src/conn/conn_open.c index d0dff5995b1..f59ce5d25d8 100644 --- a/src/third_party/wiredtiger/src/conn/conn_open.c +++ b/src/third_party/wiredtiger/src/conn/conn_open.c @@ -204,7 +204,7 @@ __wt_connection_workers(WT_SESSION_IMPL *session, const char *cfg[]) /* * Run recovery. * NOTE: This call will start (and stop) eviction if recovery is - * required. Recovery must run before the lookaside table is created + * required. Recovery must run before the history store table is created * (because recovery will update the metadata), and before eviction is * started for real. */ @@ -220,11 +220,12 @@ __wt_connection_workers(WT_SESSION_IMPL *session, const char *cfg[]) /* Initialize metadata tracking, required before creating tables. */ WT_RET(__wt_meta_track_init(session)); - /* Create the lookaside table. */ - WT_RET(__wt_las_create(session, cfg)); + /* Create the history store table. */ + WT_RET(__wt_hs_create(session, cfg)); /* - * Start eviction threads. NOTE: Eviction must be started after the lookaside table is created. + * Start eviction threads. NOTE: Eviction must be started after the history store table is + * created. */ WT_RET(__wt_evict_create(session)); diff --git a/src/third_party/wiredtiger/src/conn/conn_reconfig.c b/src/third_party/wiredtiger/src/conn/conn_reconfig.c index 7e36ce04724..043cd19b661 100644 --- a/src/third_party/wiredtiger/src/conn/conn_reconfig.c +++ b/src/third_party/wiredtiger/src/conn/conn_reconfig.c @@ -428,7 +428,7 @@ __wt_conn_reconfig(WT_SESSION_IMPL *session, const char **cfg) WT_ERR(__wt_capacity_server_create(session, cfg)); WT_ERR(__wt_checkpoint_server_create(session, cfg)); WT_ERR(__wt_debug_mode_config(session, cfg)); - WT_ERR(__wt_las_config(session, cfg)); + WT_ERR(__wt_hs_config(session, cfg)); WT_ERR(__wt_logmgr_reconfig(session, cfg)); WT_ERR(__wt_lsm_manager_reconfig(session, cfg)); WT_ERR(__wt_statlog_create(session, cfg)); diff --git a/src/third_party/wiredtiger/src/conn/conn_stat.c b/src/third_party/wiredtiger/src/conn/conn_stat.c index c572ff9fe80..92ebd55a6fc 100644 --- a/src/third_party/wiredtiger/src/conn/conn_stat.c +++ b/src/third_party/wiredtiger/src/conn/conn_stat.c @@ -75,7 +75,6 @@ __wt_conn_stat_init(WT_SESSION_IMPL *session) __wt_async_stats_update(session); __wt_cache_stats_update(session); - __wt_las_stats_update(session); __wt_txn_stats_update(session); WT_STAT_SET(session, stats, file_open, conn->open_file_count); diff --git a/src/third_party/wiredtiger/src/conn/conn_sweep.c b/src/third_party/wiredtiger/src/conn/conn_sweep.c index 5403f69f992..8ab0a51a401 100644 --- a/src/third_party/wiredtiger/src/conn/conn_sweep.c +++ b/src/third_party/wiredtiger/src/conn/conn_sweep.c @@ -266,14 +266,12 @@ __sweep_server(void *arg) WT_DECL_RET; WT_SESSION_IMPL *session; uint64_t last, now; - uint64_t last_las_sweep_id, min_sleep, oldest_id, sweep_interval; + uint64_t sweep_interval; u_int dead_handles; bool cv_signalled; session = arg; conn = S2C(session); - last_las_sweep_id = WT_TXN_NONE; - min_sleep = WT_MIN(WT_LAS_SWEEP_SEC, conn->sweep_interval); if (FLD_ISSET(conn->timing_stress_flags, WT_TIMING_STRESS_AGGRESSIVE_SWEEP)) sweep_interval = conn->sweep_interval / 10; else @@ -286,10 +284,10 @@ __sweep_server(void *arg) for (;;) { /* Wait until the next event. */ if (FLD_ISSET(conn->timing_stress_flags, WT_TIMING_STRESS_AGGRESSIVE_SWEEP)) - __wt_cond_wait_signal(session, conn->sweep_cond, min_sleep * 100 * WT_THOUSAND, - __sweep_server_run_chk, &cv_signalled); + __wt_cond_wait_signal(session, conn->sweep_cond, + conn->sweep_interval * 100 * WT_THOUSAND, __sweep_server_run_chk, &cv_signalled); else - __wt_cond_wait_signal(session, conn->sweep_cond, min_sleep * WT_MILLION, + __wt_cond_wait_signal(session, conn->sweep_cond, conn->sweep_interval * WT_MILLION, __sweep_server_run_chk, &cv_signalled); /* Check if we're quitting or being reconfigured. */ @@ -299,27 +297,8 @@ __sweep_server(void *arg) __wt_seconds(session, &now); /* - * Sweep the lookaside table. If the lookaside table hasn't yet been written, there's no - * work to do. - * - * Don't sweep the lookaside table if the cache is stuck full. The sweep uses the cache and - * can exacerbate the problem. If we try to sweep when the cache is full or we aren't making - * progress in eviction, sweeping can wind up constantly bringing in and evicting pages from - * the lookaside table, which will stop the cache from moving into the stuck state. - */ - if ((FLD_ISSET(conn->timing_stress_flags, WT_TIMING_STRESS_AGGRESSIVE_SWEEP) || - now - last >= WT_LAS_SWEEP_SEC) && - !__wt_las_empty(session) && !__wt_cache_stuck(session)) { - oldest_id = __wt_txn_oldest_id(session); - if (WT_TXNID_LT(last_las_sweep_id, oldest_id)) { - WT_ERR(__wt_las_sweep(session)); - last_las_sweep_id = oldest_id; - } - } - - /* * See if it is time to sweep the data handles. Those are swept less frequently than the - * lookaside table by default and the frequency is controlled by a user setting. + * history store table by default and the frequency is controlled by a user setting. */ if (!cv_signalled && (now - last < sweep_interval)) continue; @@ -410,13 +389,6 @@ __wt_sweep_create(WT_SESSION_IMPL *session) __wt_open_internal_session(conn, "sweep-server", true, session_flags, &conn->sweep_session)); session = conn->sweep_session; - /* - * Sweep should have it's own lookaside cursor to avoid blocking reads and eviction when - * processing drops. - */ - if (F_ISSET(conn, WT_CONN_LOOKASIDE_OPEN)) - WT_RET(__wt_las_cursor_open(session)); - WT_RET(__wt_cond_alloc(session, "handle sweep server", &conn->sweep_cond)); WT_RET(__wt_thread_create(session, &conn->sweep_tid, __sweep_server, session)); diff --git a/src/third_party/wiredtiger/src/cursor/cur_backup.c b/src/third_party/wiredtiger/src/cursor/cur_backup.c index af6cd9d0dc4..c209bca7301 100644 --- a/src/third_party/wiredtiger/src/cursor/cur_backup.c +++ b/src/third_party/wiredtiger/src/cursor/cur_backup.c @@ -351,8 +351,7 @@ __backup_add_id(WT_SESSION_IMPL *session, WT_CONFIG_ITEM *cval) return (0); err: - if (blk != NULL) - __wt_free(session, blk->id_str); + __wt_free(session, blk->id_str); return (ret); } @@ -368,6 +367,7 @@ __backup_find_id(WT_SESSION_IMPL *session, WT_CONFIG_ITEM *cval, WT_BLKINCR **in u_int i; conn = S2C(session); + WT_RET(__wt_name_check(session, cval->str, cval->len, false)); for (i = 0; i < WT_BLKINCR_MAX; ++i) { blk = &conn->incr_backups[i]; /* If it isn't valid, skip it. */ @@ -416,6 +416,10 @@ err: /* * __backup_config -- * Backup configuration. + * + * NOTE: this function handles all of the backup configuration except for the incremental use of + * force_stop. That is handled at the beginning of __backup_start because we want to deal with + * that setting without any of the other cursor setup. */ static int __backup_config(WT_SESSION_IMPL *session, WT_CURSOR_BACKUP *cb, const char *cfg[], @@ -439,19 +443,6 @@ __backup_config(WT_SESSION_IMPL *session, WT_CURSOR_BACKUP *cb, const char *cfg[ * Per-file offset incremental hot backup configurations take a starting checkpoint and optional * maximum transfer size, and the subsequent duplicate cursors take a file object. */ - WT_RET_NOTFOUND_OK(__wt_config_gets(session, cfg, "incremental.force_stop", &cval)); - if (cval.val) { - /* - * If we're force stopping incremental backup, set the flag. The resources involved in - * incremental backup will be released on cursor close and that is the only expected usage - * for this cursor. - */ - if (is_dup) - WT_RET_MSG(session, EINVAL, - "Incremental force stop can only be specified on a primary backup cursor"); - F_SET(cb, WT_CURBACKUP_FORCE_STOP); - return (0); - } WT_RET_NOTFOUND_OK(__wt_config_gets(session, cfg, "incremental.enabled", &cval)); if (cval.val) { if (!F_ISSET(conn, WT_CONN_INCR_BACKUP)) { @@ -504,8 +495,10 @@ __backup_config(WT_SESSION_IMPL *session, WT_CURSOR_BACKUP *cb, const char *cfg[ WT_ERR_MSG(session, EINVAL, "Incremental identifier can only be specified on a primary backup cursor"); ret = __backup_find_id(session, &cval, NULL); - if (ret != WT_NOTFOUND) + if (ret == 0) WT_ERR_MSG(session, EINVAL, "Incremental identifier already exists"); + if (ret != WT_NOTFOUND) + WT_ERR(ret); WT_ERR(__backup_add_id(session, &cval)); incremental_config = true; @@ -637,6 +630,9 @@ __backup_start( * incremental backup will be released on cursor close and that is the only expected usage * for this cursor. */ + if (is_dup) + WT_RET_MSG(session, EINVAL, + "Incremental force stop can only be specified on a primary backup cursor"); F_SET(cb, WT_CURBACKUP_FORCE_STOP); return (0); } @@ -821,10 +817,6 @@ __backup_list_uri_append(WT_SESSION_IMPL *session, const char *name, bool *skip) !WT_PREFIX_MATCH(name, WT_SYSTEM_PREFIX) && !WT_PREFIX_MATCH(name, "table:")) WT_RET_MSG(session, ENOTSUP, "hot backup is not supported for objects of type %s", name); - /* Ignore the lookaside table or system info. */ - if (strcmp(name, WT_LAS_URI) == 0) - return (0); - /* Add the metadata entry to the backup file. */ WT_RET(__wt_metadata_search(session, name, &value)); ret = __wt_fprintf(session, cb->bfs, "%s\n%s\n", name, value); diff --git a/src/third_party/wiredtiger/src/cursor/cur_backup_incr.c b/src/third_party/wiredtiger/src/cursor/cur_backup_incr.c index d44070a2160..be1c265a97b 100644 --- a/src/third_party/wiredtiger/src/cursor/cur_backup_incr.c +++ b/src/third_party/wiredtiger/src/cursor/cur_backup_incr.c @@ -101,22 +101,11 @@ __curbackup_incr_next(WT_CURSOR *cursor) CURSOR_API_CALL(cursor, session, get_value, btree); F_CLR(cursor, WT_CURSTD_RAW); - if (cb->incr_init) { - /* Look for the next chunk that had modifications. */ - while (cb->bit_offset < cb->nbits) - if (__bit_test(cb->bitstring.mem, cb->bit_offset)) - break; - else - ++cb->bit_offset; - - /* We either have this object's incremental information or we're done. */ - if (cb->bit_offset >= cb->nbits) - WT_ERR(WT_NOTFOUND); - __wt_cursor_set_key(cursor, cb->offset + cb->granularity * cb->bit_offset++, - cb->granularity, WT_BACKUP_RANGE); - } else if (btree == NULL || F_ISSET(cb, WT_CURBACKUP_FORCE_FULL)) { - /* We don't have this object's incremental information, and it's a full file copy. */ - /* If this is a log file, use the full pathname that may include the log path. */ + if (!cb->incr_init && (btree == NULL || F_ISSET(cb, WT_CURBACKUP_FORCE_FULL))) { + /* + * We don't have this object's incremental information or it's a forced file copy. If this + * is a log file, use the full pathname that may include the log path. + */ file = cb->incr_file; if (WT_PREFIX_MATCH(file, WT_LOG_FILENAME)) { WT_ERR(__wt_scr_alloc(session, 0, &buf)); @@ -128,22 +117,39 @@ __curbackup_incr_next(WT_CURSOR *cursor) cb->nbits = 0; cb->offset = 0; cb->bit_offset = 0; + /* + * By setting this to true, the next call will detect we're done in the code for the + * incremental cursor below and return WT_NOTFOUND. + */ cb->incr_init = true; __wt_cursor_set_key(cursor, 0, size, WT_BACKUP_FILE); } else { - /* - * We don't have this object's incremental information, and it's not a full file copy. Get a - * list of the block modifications for the file. The block modifications are from the - * incremental identifier starting point. Walk the list looking for one with a source of our - * id. - */ - WT_ERR(__curbackup_incr_blkmod(session, btree, cb)); - /* - * If there is no block modification information for this file, there is no information to - * return to the user. - */ - if (cb->bitstring.mem == NULL) - WT_ERR(WT_NOTFOUND); + if (cb->incr_init) { + /* Look for the next chunk that had modifications. */ + while (cb->bit_offset < cb->nbits) + if (__bit_test(cb->bitstring.mem, cb->bit_offset)) + break; + else + ++cb->bit_offset; + + /* We either have this object's incremental information or we're done. */ + if (cb->bit_offset >= cb->nbits) + WT_ERR(WT_NOTFOUND); + } else { + /* + * We don't have this object's incremental information, and it's not a full file copy. + * Get a list of the block modifications for the file. The block modifications are from + * the incremental identifier starting point. Walk the list looking for one with a + * source of our id. + */ + WT_ERR(__curbackup_incr_blkmod(session, btree, cb)); + /* + * If there is no block modification information for this file, there is no information + * to return to the user. + */ + if (cb->bitstring.mem == NULL) + WT_ERR(WT_NOTFOUND); + } __wt_cursor_set_key(cursor, cb->offset + cb->granularity * cb->bit_offset++, cb->granularity, WT_BACKUP_RANGE); } @@ -197,8 +203,8 @@ __wt_curbackup_open_incr(WT_SESSION_IMPL *session, const char *uri, WT_CURSOR *o /* All WiredTiger owned files are full file copies. */ if (F_ISSET(other_cb->incr_src, WT_BLKINCR_FULL) || WT_PREFIX_MATCH(cb->incr_file, "WiredTiger")) { - __wt_verbose(session, WT_VERB_BACKUP, "Forcing full file copies for id %s", - other_cb->incr_src->id_str); + __wt_verbose(session, WT_VERB_BACKUP, "Forcing full file copies for %s for id %s", + cb->incr_file, other_cb->incr_src->id_str); F_SET(cb, WT_CURBACKUP_FORCE_FULL); } /* diff --git a/src/third_party/wiredtiger/src/cursor/cur_file.c b/src/third_party/wiredtiger/src/cursor/cur_file.c index 62854711b0b..acb513ebcc6 100644 --- a/src/third_party/wiredtiger/src/cursor/cur_file.c +++ b/src/third_party/wiredtiger/src/cursor/cur_file.c @@ -718,9 +718,7 @@ __curfile_create(WT_SESSION_IMPL *session, WT_CURSOR *owner, const char *cfg[], S2C(session)->compat_major >= WT_LOG_V2_MAJOR) cursor->modify = __curfile_modify; - /* - * WiredTiger.wt should not be cached, doing so interferes with named checkpoints. - */ + /* Cursors on metadata should not be cached, doing so interferes with named checkpoints. */ if (cacheable && strcmp(WT_METAFILE_URI, cursor->internal_uri) != 0) F_SET(cursor, WT_CURSTD_CACHEABLE); diff --git a/src/third_party/wiredtiger/src/cursor/cur_std.c b/src/third_party/wiredtiger/src/cursor/cur_std.c index 9ff94ee9aa6..8a347b63353 100644 --- a/src/third_party/wiredtiger/src/cursor/cur_std.c +++ b/src/third_party/wiredtiger/src/cursor/cur_std.c @@ -240,6 +240,10 @@ __wt_cursor_copy_release_item(WT_CURSOR *cursor, WT_ITEM *item) WT_GCC_FUNC_ATTR session = (WT_SESSION_IMPL *)cursor->session; + /* Bail out if the item has been cleared. */ + if (item->data == NULL) + return (0); + /* * Whether or not we own the memory for the item, make a copy of the data and use that instead. * That allows us to overwrite and free memory owned by the item, potentially uncovering diff --git a/src/third_party/wiredtiger/src/docs/backup.dox b/src/third_party/wiredtiger/src/docs/backup.dox index b59d099175f..610033d05cf 100644 --- a/src/third_party/wiredtiger/src/docs/backup.dox +++ b/src/third_party/wiredtiger/src/docs/backup.dox @@ -35,7 +35,7 @@ continue to read and write the databases while a snapshot is taken. files have been copied. The directory into which the files are copied may subsequently be -specified as an directory to the ::wiredtiger_open function and +specified as a directory to the ::wiredtiger_open function and accessed as a WiredTiger database home. Copying the database files for a backup does not require any special @@ -57,7 +57,7 @@ arguments to a file archiver such as the system tar utility. During the period the backup cursor is open, database checkpoints can be created, but no checkpoints can be deleted. This may result in -significant file growth. Additionally while the backup cursor is open +significant file growth. Additionally while the backup cursor is open automatic log file archiving, even if enabled, will not reclaim any log files. @@ -72,6 +72,24 @@ The following is a programmatic example of creating a backup: @snippet ex_all.c backup +When logging is enabled, opening the backup cursor forces a log file switch. +The reason is so that only data that was committed and visible at the time of +the backup is available in the backup when that log file is included in the +list of files. WiredTiger offers a mechanism to gather additional log files that +may be created during the backup. + +Since backups can take a long time, it may be desirable to catch up at the +end of a backup with the log files so that operations that occurred during +backup can be recovered. WiredTiger provides the ability to open a duplicate +backup cursor with the configuration \c target=log:. This secondary backup +cursor will return the file names of all log files via \c dup_cursor->get_key(). +There will be overlap with log file names returned in the original cursor. The user +only needs to copy file names that are new but there is no error copying all +log file names returned. This secondary cursor must be closed explicitly prior +to closing the parent backup cursor. + +@snippet ex_all.c backup log duplicate + In cases where the backup is desired for a checkpoint other than the most recent, applications can discard all checkpoints subsequent to the checkpoint they want using the WT_SESSION::checkpoint method. For @@ -89,7 +107,76 @@ rm -rf /path/database.backup && wt -h /path/database.source backup /path/database.backup @endcode -@section backup_incremental Incremental backup +@section backup_incremental-block Block-based Incremental backup + +Once a full backup has been done, it can be rolled forward incrementally by +copying only modified blocks and new files to the backup copy directory. +The application is responsible for removing files that +are no longer part of the backup when later incremental backups no longer +return their name. This is especially important for WiredTiger log files +that are no longer needed and must be removed before recovery is run. + +@copydoc doc_bulk_durability + +The following is the procedure for incrementally backing up a database +using block modifications: + +1. Perform a full backup of the database (as described above), with the +additional configuration \c incremental=(enabled=true,this_id=”ID1”). +The identifier specified in \c this_id starts block tracking and that +identifier can be used in the future as the source of an incremental +backup. + +2. Begin the incremental backup by opening a backup cursor with the +\c backup: URI and config string of \c incremental=(src_id="ID1",this_id="ID2"). +Call this \c backup_cursor. Like a normal full backup cursor, +this cursor will return the filename as the key. There is no associated +value. The information returned will be based on blocks tracked since the time of +the previous backup designated with "ID1". New block tracking will be started as +"ID2" as well. WiredTiger will maintain modifications from two IDs, the current +and the most recent completed one. Note that all backup identifiers are subject to +the same naming restrictions as other configuration naming. See @ref config_intro +for details. + +3. For each file returned by \c backup_cursor->next(), open a duplicate +backup cursor to do the incremental backup on that file. The list +returned will also include log files (prefixed by \c WiredTigerLog) that need to +be copied. Configure that duplicate cursor with \c incremental=(file=name). +The \c name comes from the string returned from \c backup_cursor->get_key(). +Call this incr_cursor. + +4. The key format for the duplicate backup cursor, \c incr_cursor, is +\c qqq, representing a file offset and size pair plus a type indicator +for the range given. There is no associated value. The type indicator +will be one of \c WT_BACKUP_FILE or \c WT_BACKUP_RANGE. For \c WT_BACKUP_RANGE, +read the block from the source database file indicated by the file offset and +size pair and write the block to the same offset in the +backup database file, replacing the portion of the file represented by +the offset/size pair. It is not an error for an offset/size pair to extend past +the current end of the source file, and any missing file data should be ignored. +For \c WT_BACKUP_FILE, the user can choose to copy the entire file in +any way they choose, or to use the offset/size pair which will +indicate the expected size WiredTiger knew at the time of the call. + +5. Close the duplicate backup cursor, \c incr_cursor. + +6. Repeat steps 3-5 as many times as necessary while \c backup_cursor->next() +returns files to copy. + +7. Close the backup cursor, \c backup_cursor. + +8. Repeat steps 2-7 as often as desired. + +Full and incremental backups may be repeated as long as the backup +database directory has not been opened and recovery run. Once recovery +has run in a backup directory, you can no longer back up to that +database directory. + +An example of opening the backup data source for block-based incremental backup: + +@snippet ex_all.c incremental block backup + +@section backup_incremental Log-based Incremental backup Once a backup has been done, it can be rolled forward incrementally by adding log files to the backup copy. Adding log files to the copy @@ -139,7 +226,7 @@ database directory has not been opened and recovery run. Once recovery has run in a backup directory, you can no longer back up to that database directory. -An example of opening the backup data source for an incremental backup: +An example of opening the backup data source for log-based incremental backup: @snippet ex_all.c incremental backup diff --git a/src/third_party/wiredtiger/src/docs/command-line.dox b/src/third_party/wiredtiger/src/docs/command-line.dox index de845ae9f6f..bdad2a06016 100644 --- a/src/third_party/wiredtiger/src/docs/command-line.dox +++ b/src/third_party/wiredtiger/src/docs/command-line.dox @@ -148,7 +148,7 @@ which can be re-loaded into a new table using the \c load command. See @subpage dump_formats for details of the dump file formats. @subsection util_dump_synopsis Synopsis -<code>wt [-RrVv] [-C config] [-E secretkey ] [-h directory] dump [-jrx] [-c checkpoint] [-f output] uri</code> +<code>wt [-RrVv] [-C config] [-E secretkey ] [-h directory] dump [-jrx] [-c checkpoint] [-f output] [-t timestamp] uri</code> @subsection util_dump_options Options The following are command-specific options for the \c dump command: @@ -173,6 +173,11 @@ Dump in reverse order, from largest key to smallest. Dump all characters in a hexadecimal encoding (the default is to leave printable characters unencoded). +@par <code>-t</code> +By default, the \c dump command reads the most recent timestamp versions of +the data source; the \c -t option changes the \c dump command to read at a +specific timestamp. + <hr> @section util_list wt list List the tables in the database. @@ -413,10 +418,23 @@ The \c verify command verifies the specified table, exiting success if the data source is correct, and failure if the data source is corrupted. @subsection util_verify_synopsis Synopsis -<code>wt [-RrVv] [-C config] [-E secretkey ] [-h directory] verify uri</code> +<code>wt [-RrVv] [-C config] [-E secretkey ] [-h directory] verify [-d dump_address | dump_blocks | dump_history | dump_layout | dump_offsets=#,# | dump_pages ] [-s] -a|uri</code> @subsection util_verify_options Options -The \c verify command has no command-specific options. +The following are command-specific options for the \c verify command: + +<code>-d [config]</code> +This option allows you to specify values which you want to be displayed +when verification is run. See the WT_SESSION::verify configuration options. + +<code>-s [config]</code> +This option allows you to verify against the stable timestamp, valid only after a +rollback-to-stable operation. See the WT_SESSION::verify configuration options. + +<code>-a [config]</code> +This options allows you to verify the history store against the current state +of the data store. A uri is not passed along with this option. See the WT_SESSION::verify +configuration options. <hr> @section util_write wt write diff --git a/src/third_party/wiredtiger/src/docs/programming.dox b/src/third_party/wiredtiger/src/docs/programming.dox index 722b67fbebf..106d1bdd1c5 100644 --- a/src/third_party/wiredtiger/src/docs/programming.dox +++ b/src/third_party/wiredtiger/src/docs/programming.dox @@ -45,7 +45,6 @@ each of which is ordered by one or more columns. - @subpage cursor_join - @subpage cursor_log - @subpage operation_tracking -- @ref transaction_named_snapshots - @subpage rebalance - @subpage shared_cache - @subpage statistics diff --git a/src/third_party/wiredtiger/src/docs/spell.ok b/src/third_party/wiredtiger/src/docs/spell.ok index 1f93969ca69..95fb2756f7c 100644 --- a/src/third_party/wiredtiger/src/docs/spell.ok +++ b/src/third_party/wiredtiger/src/docs/spell.ok @@ -283,6 +283,7 @@ hugepage icount ie iflag +incr indices init insn @@ -428,6 +429,7 @@ putValue putValueString py qnx +qqq rVv rdbms rdlock diff --git a/src/third_party/wiredtiger/src/docs/transactions.dox b/src/third_party/wiredtiger/src/docs/transactions.dox index 3d656675381..870e4735f54 100644 --- a/src/third_party/wiredtiger/src/docs/transactions.dox +++ b/src/third_party/wiredtiger/src/docs/transactions.dox @@ -129,28 +129,6 @@ re-configured on a per-session basis: @snippet ex_all.c session isolation re-configuration -@section transaction_named_snapshots Named Snapshots - -Applications can create named snapshots by calling WT_SESSION::snapshot -with a configuration that includes <code>"name=foo"</code>. -This configuration creates a new named snapshot, as if a snapshot isolation -transaction were started at the time of the WT_SESSION::snapshot call. - -Subsequent transactions can be started "as of" that snapshot by calling -WT_SESSION::begin_transaction with a configuration that includes -<code>snapshot=foo</code>. That transaction will run at snapshot isolation -as if the transaction started at the time of the WT_SESSION::snapshot -call that created the snapshot. - -Named snapshots keep data pinned in cache as if a real transaction were -running for the time that the named snapshot is active. The resources -associated with named snapshots should be released by calling -WT_SESSION::snapshot with a configuration that includes -<code>"drop="</code>. See WT_SESSION::snapshot documentation for details of -the semantics supported by the drop configuration. - -Named snapshots are not durable: they do not survive WT_CONNECTION::close. - @section transaction_timestamps Application-specified Transaction Timestamps @subsection timestamp_overview Timestamp overview diff --git a/src/third_party/wiredtiger/src/docs/upgrading.dox b/src/third_party/wiredtiger/src/docs/upgrading.dox index ffab7237f6d..feacbea9f9c 100644 --- a/src/third_party/wiredtiger/src/docs/upgrading.dox +++ b/src/third_party/wiredtiger/src/docs/upgrading.dox @@ -1,6 +1,18 @@ /*! @page upgrading Upgrading WiredTiger applications </dl><hr> +@section version_322 Upgrading to Version 3.2.2 +<dl> + +<dt>Named snapshots</dt> +<dd> +Named snapshot functionality has been removed from WiredTiger as timestamps offer a better solution +to the general problem of applications wanting fine-grained control over sequencing reads and writes +across sessions. The WT_SESSION.begin_transaction method's \c snapshot configuration and the +WT_SESSION::snapshot method have been removed and are no longer available. +</dd> + +</dl><hr> @section version_321 Upgrading to Version 3.2.1 <dl> diff --git a/src/third_party/wiredtiger/src/evict/evict_file.c b/src/third_party/wiredtiger/src/evict/evict_file.c index b96331e85d1..d84304622c1 100644 --- a/src/third_party/wiredtiger/src/evict/evict_file.c +++ b/src/third_party/wiredtiger/src/evict/evict_file.c @@ -41,8 +41,7 @@ __wt_evict_file(WT_SESSION_IMPL *session, WT_CACHE_OP syncop) WT_RET(__wt_txn_update_oldest(session, WT_TXN_OLDEST_STRICT | WT_TXN_OLDEST_WAIT)); /* Walk the tree, discarding pages. */ - walk_flags = - WT_READ_CACHE | WT_READ_NO_EVICT | (syncop == WT_SYNC_CLOSE ? WT_READ_LOOKASIDE : 0); + walk_flags = WT_READ_CACHE | WT_READ_NO_EVICT; next_ref = NULL; WT_ERR(__wt_tree_walk(session, &next_ref, walk_flags)); while ((ref = next_ref) != NULL) { @@ -60,12 +59,12 @@ __wt_evict_file(WT_SESSION_IMPL *session, WT_CACHE_OP syncop) * reconciliation of other page types changes, and there's no advantage to doing so. * * Eviction can also fail because an update cannot be written. If sessions have disjoint - * sets of files open, updates in a no-longer-referenced file may not yet be globally - * visible, and the write will fail with EBUSY. Our caller handles that error, retrying - * later. + * sets of files open, updates in a no-longer-referenced file may not yet be visible, and + * the write will fail with EBUSY. Our caller handles that error, retrying later. */ if (syncop == WT_SYNC_CLOSE && __wt_page_is_modified(page)) - WT_ERR(__wt_reconcile(session, ref, NULL, WT_REC_EVICT | WT_REC_VISIBLE_ALL, NULL)); + WT_ERR(__wt_reconcile(session, ref, NULL, + WT_REC_EVICT | WT_REC_HS | WT_REC_CLEAN_AFTER_REC | WT_REC_VISIBLE_ALL)); /* * We can't evict the page just returned to us (it marks our place in the tree), so move the diff --git a/src/third_party/wiredtiger/src/evict/evict_lru.c b/src/third_party/wiredtiger/src/evict/evict_lru.c index 59a8cb97671..c342fb232ca 100644 --- a/src/third_party/wiredtiger/src/evict/evict_lru.c +++ b/src/third_party/wiredtiger/src/evict/evict_lru.c @@ -96,7 +96,7 @@ __evict_entry_priority(WT_SESSION_IMPL *session, WT_REF *ref) read_gen += btree->evict_priority; #define WT_EVICT_INTL_SKEW 1000 - if (WT_PAGE_IS_INTERNAL(page)) + if (F_ISSET(ref, WT_REF_FLAG_INTERNAL)) read_gen += WT_EVICT_INTL_SKEW; return (read_gen); @@ -473,7 +473,7 @@ __wt_evict_create(WT_SESSION_IMPL *session) /* * Create the eviction thread group. Set the group size to the maximum allowed sessions. */ - session_flags = WT_THREAD_CAN_WAIT | WT_THREAD_LOOKASIDE | WT_THREAD_PANIC_FAIL; + session_flags = WT_THREAD_CAN_WAIT | WT_THREAD_HS | WT_THREAD_PANIC_FAIL; WT_RET(__wt_thread_group_create(session, &conn->evict_threads, "eviction-server", conn->evict_threads_min, conn->evict_threads_max, session_flags, __wt_evict_thread_chk, __wt_evict_thread_run, __wt_evict_thread_stop)); @@ -537,7 +537,7 @@ __wt_evict_destroy(WT_SESSION_IMPL *session) static bool __evict_update_work(WT_SESSION_IMPL *session) { - WT_BTREE *las_tree; + WT_BTREE *hs_tree; WT_CACHE *cache; WT_CONNECTION_IMPL *conn; double dirty_target, dirty_trigger, target, trigger; @@ -563,12 +563,8 @@ __evict_update_work(WT_SESSION_IMPL *session) if (!__evict_queue_empty(cache->evict_urgent_queue, false)) LF_SET(WT_CACHE_EVICT_URGENT); - if (F_ISSET(conn, WT_CONN_LOOKASIDE_OPEN)) { - WT_ASSERT(session, F_ISSET(session, WT_SESSION_LOOKASIDE_CURSOR)); - - las_tree = ((WT_CURSOR_BTREE *)session->las_cursor)->btree; - cache->bytes_lookaside = las_tree->bytes_inmem; - } + if (F_ISSET(conn, WT_CONN_HS_OPEN) && __wt_hs_get_btree(session, &hs_tree) == 0) + cache->bytes_hs = hs_tree->bytes_inmem; /* * If we need space in the cache, try to find clean pages to evict. @@ -606,16 +602,16 @@ __evict_update_work(WT_SESSION_IMPL *session) LF_SET(WT_CACHE_EVICT_NOKEEP); /* - * Try lookaside evict when: + * Try history store evict when: * (1) the cache is stuck; OR - * (2) the lookaside score goes over 80; and + * (2) the history store score goes over 80; and * (3) the cache is more than half way from the dirty target to the * dirty trigger. */ if (__wt_cache_stuck(session) || - (__wt_cache_lookaside_score(cache) > 80 && + (__wt_cache_hs_score(cache) > 80 && dirty_inuse > (uint64_t)((dirty_target + dirty_trigger) * bytes_max) / 200)) - LF_SET(WT_CACHE_EVICT_LOOKASIDE); + LF_SET(WT_CACHE_EVICT_HS); /* * With an in-memory cache, we only do dirty eviction in order to scrub pages. @@ -643,6 +639,7 @@ __evict_pass(WT_SESSION_IMPL *session) { WT_CACHE *cache; WT_CONNECTION_IMPL *conn; + WT_DECL_RET; WT_TXN_GLOBAL *txn_global; uint64_t eviction_progress, oldest_id, prev_oldest_id; uint64_t time_now, time_prev; @@ -703,8 +700,23 @@ __evict_pass(WT_SESSION_IMPL *session) */ if (!WT_EVICT_HAS_WORKERS(session) && (cache->evict_empty_score < WT_EVICT_SCORE_CUTOFF || - !__evict_queue_empty(cache->evict_urgent_queue, false))) - WT_RET(__evict_lru_pages(session, true)); + !__evict_queue_empty(cache->evict_urgent_queue, false))) { + /* + * Release the evict pass lock as the thread is going to evict the pages from the queue. + * Otherwise, it can lead to a deadlock while evicting the page in the flow of clearing + * the eviction walk. + * + * As there is only one eviction thread that is active currently, there couldn't be any + * race conditions that other threads can enter into the flow of evict server when there + * is already another server is running. + */ + F_CLR(session, WT_SESSION_LOCKED_PASS); + __wt_spin_unlock(session, &cache->evict_pass_lock); + ret = __evict_lru_pages(session, true); + __wt_spin_lock(session, &cache->evict_pass_lock); + F_SET(session, WT_SESSION_LOCKED_PASS); + WT_RET(ret); + } if (cache->pass_intr != 0) break; @@ -716,7 +728,7 @@ __evict_pass(WT_SESSION_IMPL *session) * We check for progress every 20ms, the idea being that the aggressive score will reach 10 * after 200ms if we aren't making progress and eviction will start considering more pages. * If there is still no progress after 2s, we will treat the cache as stuck and start - * rolling back transactions and writing updates to the lookaside table. + * rolling back transactions and writing updates to the history store table. */ if (eviction_progress == cache->eviction_progress) { if (WT_CLOCKDIFF_MS(time_now, time_prev) >= 20 && @@ -1783,8 +1795,7 @@ __evict_walk_tree(WT_SESSION_IMPL *session, WT_EVICT_QUEUE *queue, u_int max_ent * create "deserts" in trees where no good eviction candidates can be found. Abandon the * walk if we get into that situation. */ - give_up = !__wt_cache_aggressive(session) && !F_ISSET(btree, WT_BTREE_LOOKASIDE) && - pages_seen > min_pages && + give_up = !__wt_cache_aggressive(session) && !WT_IS_HS(btree) && pages_seen > min_pages && (pages_queued == 0 || (pages_seen / pages_queued) > (min_pages / target_pages)); if (give_up) { /* @@ -1842,16 +1853,14 @@ __evict_walk_tree(WT_SESSION_IMPL *session, WT_EVICT_QUEUE *queue, u_int max_ent modified = __wt_page_is_modified(page); page->evict_pass_gen = cache->evict_pass_gen; - /* count internal pages seen. */ - if (WT_PAGE_IS_INTERNAL(page)) + /* Count internal pages seen. */ + if (F_ISSET(ref, WT_REF_FLAG_INTERNAL)) internal_pages_seen++; - /* - * Use the EVICT_LRU flag to avoid putting pages onto the list multiple times. - */ + /* Use the EVICT_LRU flag to avoid putting pages onto the list multiple times. */ if (F_ISSET_ATOMIC(page, WT_PAGE_EVICT_LRU)) { pages_already_queued++; - if (WT_PAGE_IS_INTERNAL(page)) + if (F_ISSET(ref, WT_REF_FLAG_INTERNAL)) internal_pages_already_queued++; continue; } @@ -1880,21 +1889,20 @@ __evict_walk_tree(WT_SESSION_IMPL *session, WT_EVICT_QUEUE *queue, u_int max_ent /* * Pages that are empty or from dead trees are fast-tracked. * - * Also evict lookaside table pages without further filtering: the cache is under pressure - * by definition and we want to free space. + * Also evict the history store table pages without further filtering: the cache is under + * pressure by definition and we want to free space. */ if (__wt_page_is_empty(page) || F_ISSET(session->dhandle, WT_DHANDLE_DEAD) || - F_ISSET(btree, WT_BTREE_LOOKASIDE)) + WT_IS_HS(btree)) goto fast; /* * If application threads are blocked on eviction of clean pages, and the only thing * preventing a clean leaf page from being evicted is it contains historical data, mark it - * dirty so we can do lookaside eviction. We also mark the tree dirty to avoid an assertion - * that we don't discard dirty pages from a clean tree. + * dirty so we can do history store eviction. We also mark the tree dirty to avoid an + * assertion that we don't discard dirty pages from a clean tree. */ - if (F_ISSET(cache, WT_CACHE_EVICT_CLEAN_HARD) && - !F_ISSET(conn, WT_CONN_EVICTION_NO_LOOKASIDE) && !WT_PAGE_IS_INTERNAL(page) && + if (F_ISSET(cache, WT_CACHE_EVICT_CLEAN_HARD) && F_ISSET(ref, WT_REF_FLAG_LEAF) && !modified && page->modify != NULL && !__wt_txn_visible_all( session, page->modify->rec_max_txn, page->modify->rec_max_timestamp)) { @@ -1918,7 +1926,7 @@ __evict_walk_tree(WT_SESSION_IMPL *session, WT_EVICT_QUEUE *queue, u_int max_ent * being skipped for walks), or we are in eviction debug mode. The goal here is that if * trees become completely idle, we eventually push them out of cache completely. */ - if (!F_ISSET(cache, WT_CACHE_EVICT_DEBUG_MODE) && WT_PAGE_IS_INTERNAL(page)) { + if (!F_ISSET(cache, WT_CACHE_EVICT_DEBUG_MODE) && F_ISSET(ref, WT_REF_FLAG_INTERNAL)) { if (page == last_parent) continue; if (btree->evict_walk_period == 0 && !__wt_cache_aggressive(session)) @@ -1951,8 +1959,8 @@ fast: ++pages_queued; ++btree->evict_walk_progress; - /* count internal pages queued. */ - if (WT_PAGE_IS_INTERNAL(page)) + /* Count internal pages queued. */ + if (F_ISSET(ref, WT_REF_FLAG_INTERNAL)) internal_pages_queued++; __wt_verbose(session, WT_VERB_EVICTSERVER, "select: %p, size %" WT_SIZET_FMT, (void *)page, @@ -1985,9 +1993,9 @@ fast: * Likewise if we found no new candidates during the walk: there is no point keeping a page * pinned, since it may be the only candidate in an idle tree. * - * If we land on a page requiring forced eviction, or that isn't an ordinary in-memory page - * (e.g., WT_REF_LIMBO), move until we find an ordinary page: we should not prevent exclusive - * access to the page until the next walk. + * If we land on a page requiring forced eviction, or that isn't an ordinary in-memory page, + * move until we find an ordinary page: we should not prevent exclusive access to the page until + * the next walk. */ if (ref != NULL) { if (__wt_ref_is_root(ref) || evict == start || give_up || @@ -2023,12 +2031,13 @@ fast: */ static int __evict_get_ref(WT_SESSION_IMPL *session, bool is_server, WT_BTREE **btreep, WT_REF **refp, - uint32_t *previous_statep) + uint8_t *previous_statep) { WT_CACHE *cache; WT_EVICT_ENTRY *evict; WT_EVICT_QUEUE *queue, *other_queue, *urgent_queue; - uint32_t candidates, previous_state; + uint32_t candidates; + uint8_t previous_state; bool is_app, server_only, urgent_ok; *btreep = NULL; @@ -2146,8 +2155,7 @@ __evict_get_ref(WT_SESSION_IMPL *session, bool is_server, WT_BTREE **btreep, WT_ * Lock the page while holding the eviction mutex to prevent multiple attempts to evict it. * For pages that are already being evicted, this operation will fail and we will move on. */ - if (((previous_state = evict->ref->state) != WT_REF_MEM && - previous_state != WT_REF_LIMBO) || + if ((previous_state = evict->ref->state) != WT_REF_MEM || !WT_REF_CAS_STATE(session, evict->ref, previous_state, WT_REF_LOCKED)) { __evict_list_clear(session, evict); continue; @@ -2193,7 +2201,7 @@ __evict_page(WT_SESSION_IMPL *session, bool is_server) WT_REF *ref; WT_TRACK_OP_DECL; uint64_t time_start, time_stop; - uint32_t previous_state; + uint8_t previous_state; bool app_timer; WT_TRACK_OP_INIT(session); @@ -2488,7 +2496,7 @@ __verbose_dump_cache_single( page = next_walk->page; size = page->memory_footprint; - if (WT_PAGE_IS_INTERNAL(page)) { + if (F_ISSET(next_walk, WT_REF_FLAG_INTERNAL)) { ++intl_pages; intl_bytes += size; intl_bytes_max = WT_MAX(intl_bytes_max, size); diff --git a/src/third_party/wiredtiger/src/evict/evict_page.c b/src/third_party/wiredtiger/src/evict/evict_page.c index 83e7aac7669..9fe4677da49 100644 --- a/src/third_party/wiredtiger/src/evict/evict_page.c +++ b/src/third_party/wiredtiger/src/evict/evict_page.c @@ -17,7 +17,7 @@ static int __evict_review(WT_SESSION_IMPL *, WT_REF *, uint32_t, bool *); * Release exclusive access to a page. */ static inline void -__evict_exclusive_clear(WT_SESSION_IMPL *session, WT_REF *ref, uint32_t previous_state) +__evict_exclusive_clear(WT_SESSION_IMPL *session, WT_REF *ref, uint8_t previous_state) { WT_ASSERT(session, ref->state == WT_REF_LOCKED && ref->page != NULL); @@ -54,7 +54,8 @@ __wt_page_release_evict(WT_SESSION_IMPL *session, WT_REF *ref, uint32_t flags) { WT_BTREE *btree; WT_DECL_RET; - uint32_t evict_flags, previous_state; + uint32_t evict_flags; + uint8_t previous_state; bool locked; btree = S2BT(session); @@ -65,8 +66,8 @@ __wt_page_release_evict(WT_SESSION_IMPL *session, WT_REF *ref, uint32_t flags) * hazard pointer without first locking the page, it could be evicted in between. */ previous_state = ref->state; - locked = (previous_state == WT_REF_MEM || previous_state == WT_REF_LIMBO) && - WT_REF_CAS_STATE(session, ref, previous_state, WT_REF_LOCKED); + locked = + previous_state == WT_REF_MEM && WT_REF_CAS_STATE(session, ref, previous_state, WT_REF_LOCKED); if ((ret = __wt_hazard_clear(session, ref)) != 0 || !locked) { if (locked) WT_REF_SET_STATE(ref, previous_state); @@ -88,7 +89,7 @@ __wt_page_release_evict(WT_SESSION_IMPL *session, WT_REF *ref, uint32_t flags) * Evict a page. */ int -__wt_evict(WT_SESSION_IMPL *session, WT_REF *ref, uint32_t previous_state, uint32_t flags) +__wt_evict(WT_SESSION_IMPL *session, WT_REF *ref, uint8_t previous_state, uint32_t flags) { WT_CONNECTION_IMPL *conn; WT_DECL_RET; @@ -156,7 +157,7 @@ __wt_evict(WT_SESSION_IMPL *session, WT_REF *ref, uint32_t previous_state, uint3 goto done; /* Count evictions of internal pages during normal operation. */ - if (!closing && WT_PAGE_IS_INTERNAL(page)) { + if (!closing && F_ISSET(ref, WT_REF_FLAG_INTERNAL)) { WT_STAT_CONN_INCR(session, cache_eviction_internal); WT_STAT_DATA_INCR(session, cache_eviction_internal); } @@ -292,32 +293,23 @@ static int __evict_page_clean_update(WT_SESSION_IMPL *session, WT_REF *ref, uint32_t flags) { WT_DECL_RET; - bool closing; - - closing = LF_ISSET(WT_EVICT_CALL_CLOSING); /* * Before discarding a page, assert that all updates are globally visible unless the tree is - * closing, dead, or we're evicting with history in lookaside. + * closing or dead. */ - WT_ASSERT(session, closing || ref->page->modify == NULL || + WT_ASSERT(session, LF_ISSET(WT_EVICT_CALL_CLOSING) || ref->page->modify == NULL || F_ISSET(session->dhandle, WT_DHANDLE_DEAD) || - (ref->page_las != NULL && ref->page_las->eviction_to_lookaside) || __wt_txn_visible_all(session, ref->page->modify->rec_max_txn, ref->page->modify->rec_max_timestamp)); /* - * Discard the page and update the reference structure. If evicting a WT_REF_LIMBO page with - * active history, transition back to WT_REF_LOOKASIDE. Otherwise, a page with a disk address is - * an on-disk page, and a page without a disk address is a re-instantiated deleted page (for - * example, by searching), that was never subsequently written. + * Discard the page and update the reference structure. A page with a disk address is an on-disk + * page, and a page without a disk address is a re-instantiated deleted page (for example, by + * searching), that was never subsequently written. */ __wt_ref_out(session, ref); - if (!closing && ref->page_las != NULL && ref->page_las->eviction_to_lookaside && - __wt_page_las_active(session, ref)) { - ref->page_las->eviction_to_lookaside = false; - WT_REF_SET_STATE(ref, WT_REF_LOOKASIDE); - } else if (ref->addr == NULL) { + if (ref->addr == NULL) { WT_WITH_PAGE_INDEX(session, ret = __evict_delete_ref(session, ref, flags)); WT_RET_BUSY_OK(ret); } else @@ -345,74 +337,60 @@ __evict_page_dirty_update(WT_SESSION_IMPL *session, WT_REF *ref, uint32_t evict_ WT_ASSERT(session, ref->addr == NULL); switch (mod->rec_result) { - case WT_PM_REC_EMPTY: /* Page is empty */ - /* - * Update the parent to reference a deleted page. Reconciliation left the - * page "empty", so there's no older transaction in the system that might - * need to see an earlier version of the page. There's no backing address, - * if we're forced to "read" into that namespace, we instantiate a new - * page instead of trying to read from the backing store. - */ + case WT_PM_REC_EMPTY: + /* + * Page is empty: Update the parent to reference a deleted page. Reconciliation left the + * page "empty", so there's no older transaction in the system that might need to see an + * earlier version of the page. There's no backing address, if we're forced to "read" into + * that namespace, we instantiate a new page instead of trying to read from the backing + * store. + */ __wt_ref_out(session, ref); WT_WITH_PAGE_INDEX(session, ret = __evict_delete_ref(session, ref, evict_flags)); WT_RET_BUSY_OK(ret); break; - case WT_PM_REC_MULTIBLOCK: /* Multiple blocks */ - /* - * Either a split where we reconciled a page and it turned into a lot - * of pages or an in-memory page that got too large, we forcibly - * evicted it, and there wasn't anything to write. - * - * The latter is a special case of forced eviction. Imagine a thread - * updating a small set keys on a leaf page. The page is too large or - * has too many deleted items, so we try and evict it, but after - * reconciliation there's only a small amount of live data (so it's a - * single page we can't split), and if there's an older reader - * somewhere, there's data on the page we can't write (so the page - * can't be evicted). In that case, we end up here with a single - * block that we can't write. Take advantage of the fact we have - * exclusive access to the page and rewrite it in memory. - */ + case WT_PM_REC_MULTIBLOCK: + /* + * Multiple blocks: Either a split where we reconciled a page and it turned into a lot of + * pages or an in-memory page that got too large, we forcibly evicted it, and there wasn't + * anything to write. + * + * The latter is a special case of forced eviction. Imagine a thread updating a small set + * keys on a leaf page. The page is too large or has too many deleted items, so we try and + * evict it, but after reconciliation there's only a small amount of live data (so it's a + * single page we can't split), and if there's an older reader somewhere, there's data on + * the page we can't write (so the page can't be evicted). In that case, we end up here with + * a single block that we can't write. Take advantage of the fact we have exclusive access + * to the page and rewrite it in memory. + */ if (mod->mod_multi_entries == 1) { WT_ASSERT(session, closing == false); WT_RET(__wt_split_rewrite(session, ref, &mod->mod_multi[0])); } else WT_RET(__wt_split_multi(session, ref, closing)); break; - case WT_PM_REC_REPLACE: /* 1-for-1 page swap */ - /* - * Update the parent to reference the replacement page. - * - * A page evicted with lookaside entries may not have an address, if no - * updates were visible to reconciliation. - * - * Publish: a barrier to ensure the structure fields are set before the - * state change makes the page available to readers. - */ - if (mod->mod_replace.addr != NULL) { - WT_RET(__wt_calloc_one(session, &addr)); - *addr = mod->mod_replace; - mod->mod_replace.addr = NULL; - mod->mod_replace.size = 0; - ref->addr = addr; - } + case WT_PM_REC_REPLACE: + /* + * 1-for-1 page swap: Update the parent to reference the replacement page. + * + * Publish: a barrier to ensure the structure fields are set before the state change makes + * the page available to readers. + */ + WT_ASSERT(session, mod->mod_replace.addr != NULL); + WT_RET(__wt_calloc_one(session, &addr)); + *addr = mod->mod_replace; + mod->mod_replace.addr = NULL; + mod->mod_replace.size = 0; + ref->addr = addr; /* * Eviction wants to keep this page if we have a disk image, re-instantiate the page in * memory, else discard the page. */ - __wt_free(session, ref->page_las); if (mod->mod_disk_image == NULL) { - if (mod->mod_page_las.las_pageid != 0) { - WT_RET(__wt_calloc_one(session, &ref->page_las)); - *ref->page_las = mod->mod_page_las; - __wt_page_modify_clear(session, ref->page); - __wt_ref_out(session, ref); - WT_REF_SET_STATE(ref, WT_REF_LOOKASIDE); - } else { - __wt_ref_out(session, ref); - WT_REF_SET_STATE(ref, WT_REF_DISK); - } + __wt_page_modify_clear(session, ref->page); + __wt_ref_out(session, ref); + WT_REF_SET_STATE(ref, WT_REF_DISK); } else { /* * The split code works with WT_MULTI structures, build one for the disk image. @@ -451,9 +429,8 @@ __evict_child_check(WT_SESSION_IMPL *session, WT_REF *parent) */ WT_INTL_FOREACH_BEGIN (session, parent->page, child) { switch (child->state) { - case WT_REF_DISK: /* On-disk */ - case WT_REF_DELETED: /* On-disk, deleted */ - case WT_REF_LOOKASIDE: /* On-disk, lookaside */ + case WT_REF_DISK: /* On-disk */ + case WT_REF_DELETED: /* On-disk, deleted */ break; default: return (__wt_set_return(session, EBUSY)); @@ -463,9 +440,8 @@ __evict_child_check(WT_SESSION_IMPL *session, WT_REF *parent) WT_INTL_FOREACH_REVERSE_BEGIN(session, parent->page, child) { switch (child->state) { - case WT_REF_DISK: /* On-disk */ - case WT_REF_DELETED: /* On-disk, deleted */ - case WT_REF_LOOKASIDE: /* On-disk, lookaside */ + case WT_REF_DISK: /* On-disk */ + case WT_REF_DELETED: /* On-disk, deleted */ break; default: return (__wt_set_return(session, EBUSY)); @@ -494,21 +470,13 @@ __evict_child_check(WT_SESSION_IMPL *session, WT_REF *parent) * this check safe: if that fails, we have raced with a read and should * give up on evicting the parent. */ - if (!__wt_atomic_casv32(&child->state, WT_REF_DELETED, WT_REF_LOCKED)) + if (!__wt_atomic_casv8(&child->state, WT_REF_DELETED, WT_REF_LOCKED)) return (__wt_set_return(session, EBUSY)); active = __wt_page_del_active(session, child, true); child->state = WT_REF_DELETED; if (active) return (__wt_set_return(session, EBUSY)); break; - case WT_REF_LOOKASIDE: /* On-disk, lookaside */ - /* - * If the lookaside history is obsolete, the reference can be - * ignored. - */ - if (__wt_page_las_active(session, child)) - return (__wt_set_return(session, EBUSY)); - break; default: return (__wt_set_return(session, EBUSY)); } @@ -531,7 +499,7 @@ __evict_review(WT_SESSION_IMPL *session, WT_REF *ref, uint32_t evict_flags, bool WT_DECL_RET; WT_PAGE *page; uint32_t flags; - bool closing, lookaside_retry, *lookaside_retryp, modified; + bool closing, modified; *inmem_splitp = false; @@ -547,7 +515,7 @@ __evict_review(WT_SESSION_IMPL *session, WT_REF *ref, uint32_t evict_flags, bool * necessary but shouldn't fire much: the eviction code is biased for leaf pages, an internal * page shouldn't be selected for eviction until all children have been evicted. */ - if (WT_PAGE_IS_INTERNAL(page)) { + if (F_ISSET(ref, WT_REF_FLAG_INTERNAL)) { WT_WITH_PAGE_INDEX(session, ret = __evict_child_check(session, ref)); if (ret != 0) WT_STAT_CONN_INCR(session, cache_eviction_fail_active_children_on_an_internal_page); @@ -600,8 +568,8 @@ __evict_review(WT_SESSION_IMPL *session, WT_REF *ref, uint32_t evict_flags, bool return (0); /* - * If reconciliation is disabled for this thread (e.g., during an eviction that writes to - * lookaside), give up. + * If reconciliation is disabled for this thread (e.g., during an eviction that writes to the + * history store), give up. */ if (F_ISSET(session, WT_SESSION_NO_RECONCILE)) return (__wt_set_return(session, EBUSY)); @@ -613,7 +581,8 @@ __evict_review(WT_SESSION_IMPL *session, WT_REF *ref, uint32_t evict_flags, bool * cannot read. * * Don't set any other flags for internal pages: there are no update lists to be saved and - * restored, changes can't be written into the lookaside table, nor can we re-create internal + * restored, changes can't be written into the history store table, nor can we re-create + * internal * pages in memory. * * For leaf pages: @@ -634,19 +603,17 @@ __evict_review(WT_SESSION_IMPL *session, WT_REF *ref, uint32_t evict_flags, bool * memory. */ cache = conn->cache; - lookaside_retry = false; - lookaside_retryp = NULL; if (closing) LF_SET(WT_REC_VISIBILITY_ERR); - else if (WT_PAGE_IS_INTERNAL(page) || F_ISSET(S2BT(session), WT_BTREE_LOOKASIDE)) + else if (F_ISSET(ref, WT_REF_FLAG_INTERNAL) || WT_IS_HS(S2BT(session))) ; else if (WT_SESSION_BTREE_SYNC(session)) - LF_SET(WT_REC_LOOKASIDE); + LF_SET(WT_REC_HS); else if (F_ISSET(conn, WT_CONN_IN_MEMORY)) - LF_SET(WT_REC_IN_MEMORY | WT_REC_SCRUB | WT_REC_UPDATE_RESTORE); + LF_SET(WT_REC_IN_MEMORY | WT_REC_SCRUB); else { - LF_SET(WT_REC_UPDATE_RESTORE); + LF_SET(WT_REC_HS); /* * Scrub if we're supposed to or toss it in sometimes if we are in debugging mode. @@ -654,36 +621,10 @@ __evict_review(WT_SESSION_IMPL *session, WT_REF *ref, uint32_t evict_flags, bool if (F_ISSET(cache, WT_CACHE_EVICT_SCRUB) || (F_ISSET(cache, WT_CACHE_EVICT_DEBUG_MODE) && __wt_random(&session->rnd) % 3 == 0)) LF_SET(WT_REC_SCRUB); - - /* - * If the cache is under pressure with many updates that can't be evicted, check if - * reconciliation suggests trying the lookaside table. - */ - if (!WT_IS_METADATA(session->dhandle) && F_ISSET(cache, WT_CACHE_EVICT_LOOKASIDE) && - !F_ISSET(conn, WT_CONN_EVICTION_NO_LOOKASIDE)) { - if (F_ISSET(cache, WT_CACHE_EVICT_DEBUG_MODE) && __wt_random(&session->rnd) % 10 == 0) { - LF_CLR(WT_REC_SCRUB | WT_REC_UPDATE_RESTORE); - LF_SET(WT_REC_LOOKASIDE); - } - lookaside_retryp = &lookaside_retry; - } } /* Reconcile the page. */ - ret = __wt_reconcile(session, ref, NULL, flags, lookaside_retryp); - WT_ASSERT(session, __wt_page_is_modified(page) || - __wt_txn_visible_all(session, page->modify->rec_max_txn, page->modify->rec_max_timestamp)); - - /* - * If reconciliation fails but reports it might succeed if we use the lookaside table, try again - * with the lookaside table, allowing the eviction of pages we'd otherwise have to retain in - * cache to support older readers. - */ - if (ret == EBUSY && lookaside_retry) { - LF_CLR(WT_REC_SCRUB | WT_REC_UPDATE_RESTORE); - LF_SET(WT_REC_LOOKASIDE); - ret = __wt_reconcile(session, ref, NULL, flags, NULL); - } + ret = __wt_reconcile(session, ref, NULL, flags); if (ret != 0) WT_STAT_CONN_INCR(session, cache_eviction_fail_in_reconciliation); @@ -693,8 +634,8 @@ __evict_review(WT_SESSION_IMPL *session, WT_REF *ref, uint32_t evict_flags, bool /* * Give up on eviction during a checkpoint if the page splits. * - * We get here if checkpoint reads a page with lookaside entries: if more of those entries are - * visible now than when the original eviction happened, the page could split. In most + * We get here if checkpoint reads a page with history store entries: if more of those entries + * are visible now than when the original eviction happened, the page could split. In most * workloads, this is very unlikely. However, since checkpoint is partway through reconciling * the parent page, a split can corrupt the checkpoint. */ @@ -704,8 +645,7 @@ __evict_review(WT_SESSION_IMPL *session, WT_REF *ref, uint32_t evict_flags, bool /* * Success: assert that the page is clean or reconciliation was configured to save updates. */ - WT_ASSERT( - session, !__wt_page_is_modified(page) || LF_ISSET(WT_REC_LOOKASIDE | WT_REC_UPDATE_RESTORE)); + WT_ASSERT(session, !__wt_page_is_modified(page) || LF_ISSET(WT_REC_HS | WT_REC_IN_MEMORY)); return (0); } diff --git a/src/third_party/wiredtiger/src/evict/evict_stat.c b/src/third_party/wiredtiger/src/evict/evict_stat.c index 5c8700029eb..9d42ac577c4 100644 --- a/src/third_party/wiredtiger/src/evict/evict_stat.c +++ b/src/third_party/wiredtiger/src/evict/evict_stat.c @@ -28,7 +28,6 @@ __evict_stat_walk(WT_SESSION_IMPL *session) btree = S2BT(session); cache = S2C(session)->cache; - next_walk = NULL; gen_gap_max = gen_gap_sum = max_pagesize = 0; num_memory = num_not_queueable = num_queued = 0; num_smaller_allocsz = pages_clean = pages_dirty = pages_internal = 0; @@ -37,6 +36,7 @@ __evict_stat_walk(WT_SESSION_IMPL *session) walk_count = written_size_cnt = written_size_sum = 0; min_written_size = UINT64_MAX; + next_walk = NULL; while (__wt_tree_walk_count(session, &next_walk, &walk_count, WT_READ_CACHE | WT_READ_NO_EVICT | WT_READ_NO_GEN | WT_READ_NO_WAIT) == 0 && next_walk != NULL) { @@ -69,7 +69,7 @@ __evict_stat_walk(WT_SESSION_IMPL *session) } else ++num_memory; - if (WT_PAGE_IS_INTERNAL(page)) + if (F_ISSET(next_walk, WT_REF_FLAG_INTERNAL)) ++pages_internal; else ++pages_leaf; diff --git a/src/third_party/wiredtiger/src/history/hs.c b/src/third_party/wiredtiger/src/history/hs.c new file mode 100644 index 00000000000..861ddb5f996 --- /dev/null +++ b/src/third_party/wiredtiger/src/history/hs.c @@ -0,0 +1,1236 @@ +/*- + * Copyright (c) 2014-2020 MongoDB, Inc. + * Copyright (c) 2008-2014 WiredTiger, Inc. + * All rights reserved. + * + * See the file LICENSE for redistribution information. + */ + +#include "wt_internal.h" + +/* + * When an operation is accessing the history store table, it should ignore the cache size (since + * the cache is already full), and the operation can't reenter reconciliation. + */ +#define WT_HS_SESSION_FLAGS (WT_SESSION_IGNORE_CACHE_SIZE | WT_SESSION_NO_RECONCILE) + +static int __hs_delete_key_from_pos( + WT_SESSION_IMPL *session, WT_CURSOR *hs_cursor, uint32_t btree_id, const WT_ITEM *key); + +/* + * __hs_start_internal_session -- + * Create a temporary internal session to retrieve history store. + */ +static int +__hs_start_internal_session(WT_SESSION_IMPL *session, WT_SESSION_IMPL **int_sessionp) +{ + WT_ASSERT(session, !F_ISSET(session, WT_CONN_HS_OPEN)); + return (__wt_open_internal_session(S2C(session), "hs_access", true, 0, int_sessionp)); +} + +/* + * __hs_release_internal_session -- + * Release the temporary internal session started to retrieve history store. + */ +static int +__hs_release_internal_session(WT_SESSION_IMPL *int_session) +{ + WT_SESSION *wt_session; + + wt_session = &int_session->iface; + return (wt_session->close(wt_session, NULL)); +} + +/* + * __wt_hs_get_btree -- + * Get the history store btree. Open a history store cursor if needed to get the btree. + */ +int +__wt_hs_get_btree(WT_SESSION_IMPL *session, WT_BTREE **hs_btreep) +{ + WT_DECL_RET; + uint32_t session_flags; + bool is_owner; + + *hs_btreep = NULL; + session_flags = 0; /* [-Werror=maybe-uninitialized] */ + + WT_RET(__wt_hs_cursor(session, &session_flags, &is_owner)); + + *hs_btreep = ((WT_CURSOR_BTREE *)session->hs_cursor)->btree; + WT_ASSERT(session, *hs_btreep != NULL); + + WT_TRET(__wt_hs_cursor_close(session, session_flags, is_owner)); + + return (ret); +} + +/* + * __wt_hs_config -- + * Configure the history store table. + */ +int +__wt_hs_config(WT_SESSION_IMPL *session, const char **cfg) +{ + WT_BTREE *btree; + WT_CONFIG_ITEM cval; + WT_CONNECTION_IMPL *conn; + WT_DECL_RET; + WT_SESSION_IMPL *tmp_setup_session; + + conn = S2C(session); + tmp_setup_session = NULL; + + WT_ERR(__wt_config_gets(session, cfg, "history_store.file_max", &cval)); + if (cval.val != 0 && cval.val < WT_HS_FILE_MIN) + WT_ERR_MSG(session, EINVAL, "max history store size %" PRId64 " below minimum %d", cval.val, + WT_HS_FILE_MIN); + + /* TODO: WT-5585 Remove after we switch to using history_store config in MongoDB. */ + if (cval.val == 0) { + WT_ERR(__wt_config_gets(session, cfg, "cache_overflow.file_max", &cval)); + if (cval.val != 0 && cval.val < WT_HS_FILE_MIN) + WT_ERR_MSG(session, EINVAL, "max history store size %" PRId64 " below minimum %d", + cval.val, WT_HS_FILE_MIN); + } + + /* in-memory or readonly configurations do not have a history store. */ + if (F_ISSET(conn, WT_CONN_IN_MEMORY | WT_CONN_READONLY)) + return (0); + + WT_ERR(__hs_start_internal_session(session, &tmp_setup_session)); + + /* + * Retrieve the btree from the history store cursor. + */ + WT_ERR(__wt_hs_get_btree(tmp_setup_session, &btree)); + + /* Track the history store file ID. */ + if (conn->cache->hs_fileid == 0) + conn->cache->hs_fileid = btree->id; + + /* + * Set special flags for the history store table: the history store flag (used, for example, to + * avoid writing records during reconciliation), also turn off checkpoints and logging. + * + * Test flags before setting them so updates can't race in subsequent opens (the first update is + * safe because it's single-threaded from wiredtiger_open). + */ + if (!F_ISSET(btree, WT_BTREE_HS)) + F_SET(btree, WT_BTREE_HS); + if (!F_ISSET(btree, WT_BTREE_NO_LOGGING)) + F_SET(btree, WT_BTREE_NO_LOGGING); + + /* + * We need to set file_max on the btree associated with one of the history store sessions. + */ + btree->file_max = (uint64_t)cval.val; + WT_STAT_CONN_SET(session, cache_hs_ondisk_max, btree->file_max); + +err: + if (tmp_setup_session != NULL) + WT_TRET(__hs_release_internal_session(tmp_setup_session)); + return (ret); +} + +/* + * __wt_hs_create -- + * Initialize the database's history store. + */ +int +__wt_hs_create(WT_SESSION_IMPL *session, const char **cfg) +{ + WT_CONNECTION_IMPL *conn; + + conn = S2C(session); + + /* Read-only and in-memory configurations don't need the history store table. */ + if (F_ISSET(conn, WT_CONN_IN_MEMORY | WT_CONN_READONLY)) + return (0); + + /* Re-create the table. */ + WT_RET(__wt_session_create(session, WT_HS_URI, WT_HS_CONFIG)); + + WT_RET(__wt_hs_config(session, cfg)); + + /* The statistics server is already running, make sure we don't race. */ + WT_WRITE_BARRIER(); + F_SET(conn, WT_CONN_HS_OPEN); + + return (0); +} + +/* + * __wt_hs_destroy -- + * Destroy the database's history store. + */ +void +__wt_hs_destroy(WT_SESSION_IMPL *session) +{ + F_CLR(S2C(session), WT_CONN_HS_OPEN); +} + +/* + * __wt_hs_cursor_open -- + * Open a new history store table cursor. + */ +int +__wt_hs_cursor_open(WT_SESSION_IMPL *session) +{ + WT_CURSOR *cursor; + WT_DECL_RET; + const char *open_cursor_cfg[] = {WT_CONFIG_BASE(session, WT_SESSION_open_cursor), NULL}; + + WT_WITHOUT_DHANDLE( + session, ret = __wt_open_cursor(session, WT_HS_URI, NULL, open_cursor_cfg, &cursor)); + WT_RET(ret); + + session->hs_cursor = cursor; + F_SET(session, WT_SESSION_HS_CURSOR); + + return (0); +} + +/* + * __wt_hs_cursor -- + * Return a history store cursor, open one if not already open. + */ +int +__wt_hs_cursor(WT_SESSION_IMPL *session, uint32_t *session_flags, bool *is_owner) +{ + /* We should never reach here if working in context of the default session. */ + WT_ASSERT(session, S2C(session)->default_session != session); + + /* + * We don't want to get tapped for eviction after we start using the history store cursor; save + * a copy of the current eviction state, we'll turn eviction off before we return. + * + * Don't cache history store table pages, we're here because of eviction problems and there's no + * reason to believe history store pages will be useful more than once. + */ + *session_flags = F_MASK(session, WT_HS_SESSION_FLAGS); + *is_owner = false; + + /* Open a cursor if this session doesn't already have one. */ + if (!F_ISSET(session, WT_SESSION_HS_CURSOR)) { + /* The caller is responsible for closing this cursor. */ + *is_owner = true; + WT_RET(__wt_hs_cursor_open(session)); + } + + WT_ASSERT(session, session->hs_cursor != NULL); + + /* Configure session to access the history store table. */ + F_SET(session, WT_HS_SESSION_FLAGS); + + return (0); +} + +/* + * __wt_hs_cursor_close -- + * Discard a history store cursor. + */ +int +__wt_hs_cursor_close(WT_SESSION_IMPL *session, uint32_t session_flags, bool is_owner) +{ + /* Nothing to do if the session doesn't have a HS cursor opened. */ + if (!F_ISSET(session, WT_SESSION_HS_CURSOR)) { + WT_ASSERT(session, session->hs_cursor == NULL); + return (0); + } + WT_ASSERT(session, session->hs_cursor != NULL); + + /* + * If we're not the owner, we're not responsible for closing this cursor. Reset the cursor to + * avoid pinning the page in cache. + */ + if (!is_owner) + return (session->hs_cursor->reset(session->hs_cursor)); + + /* + * We turned off caching and eviction while the history store cursor was in use, restore the + * session's flags. + */ + F_CLR(session, WT_HS_SESSION_FLAGS); + F_SET(session, session_flags); + + WT_RET(session->hs_cursor->close(session->hs_cursor)); + session->hs_cursor = NULL; + F_CLR(session, WT_SESSION_HS_CURSOR); + + return (0); +} + +/* + * __wt_hs_modify -- + * Make an update to the history store. + * + * History store updates don't use transactions as those updates should be immediately visible and + * don't follow normal transaction semantics. For this reason, history store updates are + * directly modified using the low level api instead of the ordinary cursor api. + */ +int +__wt_hs_modify(WT_CURSOR_BTREE *hs_cbt, WT_UPDATE *hs_upd) +{ + WT_DECL_RET; + WT_PAGE_MODIFY *mod; + WT_SESSION_IMPL *session; + WT_UPDATE *last_upd; + + session = (WT_SESSION_IMPL *)hs_cbt->iface.session; + + /* If there are existing updates, append them after the new updates. */ + if (hs_cbt->compare == 0) { + for (last_upd = hs_upd; last_upd->next != NULL; last_upd = last_upd->next) + ; + if (hs_cbt->ins != NULL) + last_upd->next = hs_cbt->ins->upd; + else if ((mod = hs_cbt->ref->page->modify) != NULL && mod->mod_row_update != NULL) + last_upd->next = mod->mod_row_update[hs_cbt->slot]; + } + + WT_WITH_BTREE(session, hs_cbt->btree, + ret = __wt_row_modify(hs_cbt, &hs_cbt->iface.key, NULL, hs_upd, WT_UPDATE_INVALID, true)); + return (ret); +} + +/* + * __hs_insert_updates_verbose -- + * Display a verbose message once per checkpoint with details about the cache state when + * performing a history store table write. + */ +static void +__hs_insert_updates_verbose(WT_SESSION_IMPL *session, WT_BTREE *btree) +{ + WT_CACHE *cache; + WT_CONNECTION_IMPL *conn; + double pct_dirty, pct_full; + uint64_t ckpt_gen_current, ckpt_gen_last; + uint32_t btree_id; + + btree_id = btree->id; + + if (!WT_VERBOSE_ISSET(session, WT_VERB_HS | WT_VERB_HS_ACTIVITY)) + return; + + conn = S2C(session); + cache = conn->cache; + ckpt_gen_current = __wt_gen(session, WT_GEN_CHECKPOINT); + ckpt_gen_last = cache->hs_verb_gen_write; + + /* + * Print a message if verbose history store, or once per checkpoint if only reporting activity. + * Avoid an expensive atomic operation as often as possible when the message rate is limited. + */ + if (WT_VERBOSE_ISSET(session, WT_VERB_HS) || + (ckpt_gen_current > ckpt_gen_last && + __wt_atomic_casv64(&cache->hs_verb_gen_write, ckpt_gen_last, ckpt_gen_current))) { + WT_IGNORE_RET_BOOL(__wt_eviction_clean_needed(session, &pct_full)); + WT_IGNORE_RET_BOOL(__wt_eviction_dirty_needed(session, &pct_dirty)); + + __wt_verbose(session, WT_VERB_HS | WT_VERB_HS_ACTIVITY, + "Page reconciliation triggered history store write: file ID %" PRIu32 + ". " + "Current history store file size: %" PRId64 + ", " + "cache dirty: %2.3f%% , " + "cache use: %2.3f%%", + btree_id, WT_STAT_READ(conn->stats, cache_hs_ondisk), pct_dirty, pct_full); + } + + /* Never skip updating the tracked generation */ + if (WT_VERBOSE_ISSET(session, WT_VERB_HS)) + cache->hs_verb_gen_write = ckpt_gen_current; +} + +/* + * __hs_insert_record_with_btree_int -- + * Internal helper for inserting history store records. + */ +static int +__hs_insert_record_with_btree_int(WT_SESSION_IMPL *session, WT_CURSOR *cursor, WT_BTREE *btree, + const WT_ITEM *key, const WT_UPDATE *upd, const uint8_t type, const WT_ITEM *hs_value, + WT_TIME_PAIR stop_ts_pair) +{ + WT_CURSOR_BTREE *cbt; + WT_DECL_RET; + WT_UPDATE *hs_upd; + size_t notused; + uint32_t session_flags; + + cbt = (WT_CURSOR_BTREE *)cursor; + hs_upd = NULL; + + /* + * Use WT_CURSOR.set_key and WT_CURSOR.set_value to create key and value items, then use them to + * create an update chain for a direct insertion onto the history store page. + */ + cursor->set_key( + cursor, btree->id, key, upd->start_ts, __wt_atomic_add64(&btree->hs_counter, 1)); + cursor->set_value(cursor, stop_ts_pair.timestamp, upd->durable_ts, (uint64_t)type, hs_value); + + /* + * Insert a delete record to represent stop time pair for the actual record to be inserted. Set + * the stop time pair as the commit time pair of the history store delete record. + */ + WT_ERR(__wt_update_alloc(session, NULL, &hs_upd, ¬used, WT_UPDATE_TOMBSTONE)); + hs_upd->start_ts = stop_ts_pair.timestamp; + hs_upd->txnid = stop_ts_pair.txnid; + + /* + * Append to the delete record, the actual record to be inserted into the history store. Set the + * current update start time pair as the commit time pair to the history store record. + */ + WT_ERR(__wt_update_alloc(session, &cursor->value, &hs_upd->next, ¬used, WT_UPDATE_STANDARD)); + hs_upd->next->start_ts = upd->start_ts; + hs_upd->next->txnid = upd->txnid; + + /* + * Search the page and insert the updates. We expect there will be no existing data: assert that + * we don't find a matching key. + */ + WT_WITH_PAGE_INDEX(session, ret = __wt_row_search(cbt, &cursor->key, true, NULL, false, NULL)); + WT_ERR(ret); + WT_ERR(__wt_hs_modify(cbt, hs_upd)); + + /* + * Since the two updates (tombstone and the standard) will reconcile into a single entry, we are + * incrementing the history store insert statistic by one. + */ + WT_STAT_CONN_INCR(session, cache_hs_insert); + +err: + if (ret != 0) + __wt_free_update_list(session, &hs_upd); + /* + * If we inserted an update with no timestamp, we need to delete all history records for that + * key that are further in the history table than us (the key is lexicographically greater). For + * timestamped tables that are occasionally getting a non-timestamped update, that means that + * all timestamped updates should get removed. In the case of non-timestamped tables, that means + * that all updates with higher transaction ids will get removed (which could happen at some + * more relaxed isolation levels). + */ + if (ret == 0 && upd->start_ts == WT_TS_NONE) { +#ifdef HAVE_DIAGNOSTIC + /* + * We need to initialize the last searched key so that we can do key comparisons when we + * begin iterating over the history store. This needs to be done otherwise the subsequent + * "next" calls will blow up. + */ + WT_TRET(__wt_cursor_key_order_init(cbt)); +#endif + session_flags = session->flags; + F_SET(session, WT_SESSION_IGNORE_HS_TOMBSTONE); + /* We're pointing at the newly inserted update. Iterate once more to avoid deleting it. */ + ret = cursor->next(cursor); + if (ret == WT_NOTFOUND) + ret = 0; + else if (ret == 0) { + WT_TRET(__hs_delete_key_from_pos(session, cursor, btree->id, key)); + WT_STAT_CONN_INCR(session, cache_hs_key_truncate_mix_ts); + } + if (!FLD_ISSET(session_flags, WT_SESSION_IGNORE_HS_TOMBSTONE)) + F_CLR(session, WT_SESSION_IGNORE_HS_TOMBSTONE); + } + /* We did a row search, release the cursor so that the page doesn't continue being held. */ + cursor->reset(cursor); + + return (ret); +} + +/* + * __hs_insert_record_with_btree -- + * A helper function to insert the record into the history store including stop time pair. + * Should be called with session's btree switched to the history store. + */ +static int +__hs_insert_record_with_btree(WT_SESSION_IMPL *session, WT_CURSOR *cursor, WT_BTREE *btree, + const WT_ITEM *key, const WT_UPDATE *upd, const uint8_t type, const WT_ITEM *hs_value, + WT_TIME_PAIR stop_ts_pair) +{ + WT_DECL_RET; + + /* + * The session should be pointing at the history store btree since this is the one that we'll be + * inserting into. The btree parameter that we're passing in should is the btree that the + * history store content is associated with (this is where the btree id part of the history + * store key comes from). + */ + WT_ASSERT(session, WT_IS_HS(S2BT(session))); + WT_ASSERT(session, !WT_IS_HS(btree)); + + /* + * Disable bulk loads into history store. This would normally occur when updating a record with + * a cursor however the history store doesn't use cursor update, so we do it here. + */ + __wt_cursor_disable_bulk(session); + + /* + * Only deltas or full updates should be written to the history store. More specifically, we + * should NOT be writing tombstone records in the history store table. + */ + WT_ASSERT(session, type == WT_UPDATE_STANDARD || type == WT_UPDATE_MODIFY); + + /* + * If the time pairs are out of order (which can happen if the application performs updates with + * out-of-order timestamps), so this value can never be seen, don't bother inserting it. + */ + if (stop_ts_pair.timestamp < upd->start_ts || + (stop_ts_pair.timestamp == upd->start_ts && stop_ts_pair.txnid <= upd->txnid)) { + char ts_string[2][WT_TS_INT_STRING_SIZE]; + __wt_verbose(session, WT_VERB_TIMESTAMP, + "Warning: fixing out-of-order timestamps %s earlier than previous update %s", + __wt_timestamp_to_string(stop_ts_pair.timestamp, ts_string[0]), + __wt_timestamp_to_string(upd->start_ts, ts_string[1])); + return (0); + } + + /* The tree structure can change while we try to insert the mod list, retry if that happens. */ + while ((ret = __hs_insert_record_with_btree_int( + session, cursor, btree, key, upd, type, hs_value, stop_ts_pair)) == WT_RESTART) + ; + + return (ret); +} + +/* + * __hs_insert_record -- + * Temporarily switches to history store btree and calls the helper routine to insert records. + */ +static int +__hs_insert_record(WT_SESSION_IMPL *session, WT_CURSOR *cursor, WT_BTREE *btree, const WT_ITEM *key, + const WT_UPDATE *upd, const uint8_t type, const WT_ITEM *hs_value, WT_TIME_PAIR stop_ts_pair) +{ + WT_CURSOR_BTREE *cbt; + WT_DECL_RET; + + cbt = (WT_CURSOR_BTREE *)cursor; + WT_WITH_BTREE(session, cbt->btree, ret = __hs_insert_record_with_btree(session, cursor, btree, + key, upd, type, hs_value, stop_ts_pair)); + return (ret); +} + +/* + * __hs_calculate_full_value -- + * Calculate the full value of an update. + */ +static inline int +__hs_calculate_full_value(WT_SESSION_IMPL *session, WT_ITEM *full_value, WT_UPDATE *upd, + const void *base_full_value, size_t size) +{ + if (upd->type == WT_UPDATE_MODIFY) { + WT_RET(__wt_buf_set(session, full_value, base_full_value, size)); + WT_RET(__wt_modify_apply_item(session, full_value, upd->data, false)); + } else { + WT_ASSERT(session, upd->type == WT_UPDATE_STANDARD); + full_value->data = upd->data; + full_value->size = upd->size; + } + + return (0); +} + +/* + * __wt_hs_insert_updates -- + * Copy one set of saved updates into the database's history store table. + */ +int +__wt_hs_insert_updates(WT_CURSOR *cursor, WT_BTREE *btree, WT_PAGE *page, WT_MULTI *multi) +{ + WT_DECL_ITEM(full_value); + WT_DECL_ITEM(key); + WT_DECL_ITEM(modify_value); + WT_DECL_ITEM(prev_full_value); + WT_DECL_ITEM(tmp); + WT_DECL_RET; +/* If the limit is exceeded, we will insert a full update to the history store */ +#define MAX_REVERSE_MODIFY_NUM 16 + WT_MODIFY entries[MAX_REVERSE_MODIFY_NUM]; + WT_MODIFY_VECTOR modifies; + WT_SAVE_UPD *list; + WT_SESSION_IMPL *session; + WT_UPDATE *prev_upd, *upd; + WT_TIME_PAIR stop_ts_pair; + wt_off_t hs_size; + uint64_t insert_cnt, max_hs_size; + uint32_t i; + uint8_t *p; + int nentries; + bool squashed; + + prev_upd = NULL; + session = (WT_SESSION_IMPL *)cursor->session; + insert_cnt = 0; + __wt_modify_vector_init(session, &modifies); + + if (!btree->hs_entries) + btree->hs_entries = true; + + /* Ensure enough room for a column-store key without checking. */ + WT_ERR(__wt_scr_alloc(session, WT_INTPACK64_MAXSIZE, &key)); + + WT_ERR(__wt_scr_alloc(session, 0, &full_value)); + + WT_ERR(__wt_scr_alloc(session, 0, &prev_full_value)); + + /* Enter each update in the boundary's list into the history store. */ + for (i = 0, list = multi->supd; i < multi->supd_entries; ++i, ++list) { + /* If no onpage_upd is selected, we don't need to insert anything into the history store. */ + if (list->onpage_upd == NULL) + continue; + + /* onpage_upd now is always from the update chain */ + WT_ASSERT(session, !F_ISSET(list->onpage_upd, WT_UPDATE_RESTORED_FROM_DISK)); + + /* History store table key component: source key. */ + switch (page->type) { + case WT_PAGE_COL_FIX: + case WT_PAGE_COL_VAR: + p = key->mem; + WT_ERR(__wt_vpack_uint(&p, 0, WT_INSERT_RECNO(list->ins))); + key->size = WT_PTRDIFF(p, key->data); + break; + case WT_PAGE_ROW_LEAF: + if (list->ins == NULL) { + WT_WITH_BTREE( + session, btree, ret = __wt_row_leaf_key(session, page, list->ripcip, key, false)); + WT_ERR(ret); + } else { + key->data = WT_INSERT_KEY(list->ins); + key->size = WT_INSERT_KEY_SIZE(list->ins); + } + break; + default: + WT_ERR(__wt_illegal_value(session, page->type)); + } + + /* + * Trim any updates before writing to history store. This saves wasted work. + */ + WT_WITH_BTREE( + session, btree, upd = __wt_update_obsolete_check(session, page, list->onpage_upd, true)); + __wt_free_update_list(session, &upd); + upd = list->onpage_upd; + + /* + * The algorithm assumes the oldest update on the update chain in memory is either a full + * update or a tombstone. + * + * This is guaranteed by __wt_rec_upd_select appends the original onpage value at the end of + * the chain. It also assumes the onpage_upd selected cannot be a TOMBSTONE and the update + * newer than a TOMBSTONE must be a full update. + * + * The algorithm walks from the oldest update, or the most recently inserted into history + * store update. To the newest update and build full updates along the way. It sets the stop + * time pair of the update to the start time pair of the next update, squashes the updates + * that are from the same transaction and of the same start timestamp, calculates reverse + * modification if prev_upd is a MODIFY, and inserts the update to the history store. + * + * It deals with the following scenarios: + * 1) We only have full updates on the chain and we only insert full updates to + * the history store. + * 2) We have modifies on the chain, e.g., U (selected onpage value) -> M -> M ->U. We + * reverse the modifies and insert the reversed modifies to the history store if it is not + * the newest update written to the history store and the reverse operation is successful. + * With regard to the example, we insert U -> RM -> U to the history store. + * 3) We have tombstones in the middle of the chain, e.g., + * U (selected onpage value) -> U -> T -> M -> U. + * We write the stop time pair of M with the start time pair of the tombstone and skip the + * tombstone. + * 4) We have a single tombstone on the chain, it is simply ignored. + */ + for (; upd != NULL; upd = upd->next) { + if (upd->txnid == WT_TXN_ABORTED) + continue; + WT_ERR(__wt_modify_vector_push(&modifies, upd)); + /* + * If we've reached a full update and its in the history store we don't need to continue + * as anything beyond this point won't help with calculating deltas. + */ + if (upd->type == WT_UPDATE_STANDARD && F_ISSET(upd, WT_UPDATE_HS)) + break; + } + + upd = NULL; + + /* Construct the oldest full update. */ + WT_ASSERT(session, modifies.size > 0); + __wt_modify_vector_pop(&modifies, &upd); + + WT_ASSERT(session, upd->type == WT_UPDATE_STANDARD || upd->type == WT_UPDATE_TOMBSTONE); + /* Skip TOMBSTONE at the end of the update chain. */ + if (upd->type == WT_UPDATE_TOMBSTONE) { + if (modifies.size > 0) { + if (upd->start_ts == WT_TS_NONE) { + WT_ERR(__wt_hs_delete_key(session, btree->id, key)); + WT_STAT_CONN_INCR(session, cache_hs_key_truncate_mix_ts); + } + __wt_modify_vector_pop(&modifies, &upd); + } else + continue; + } + + WT_ASSERT(session, upd->type == WT_UPDATE_STANDARD); + full_value->data = upd->data; + full_value->size = upd->size; + + squashed = false; + + /* + * Flush the updates on stack. Stopping once we run out or we reach the onpage upd start + * time pair, we can squash modifies with the same start time pair as the onpage upd away. + */ + for (; modifies.size > 0 && + !(upd->txnid == list->onpage_upd->txnid && + upd->start_ts == list->onpage_upd->start_ts); + tmp = full_value, full_value = prev_full_value, prev_full_value = tmp, + upd = prev_upd) { + WT_ASSERT(session, upd->type == WT_UPDATE_STANDARD || upd->type == WT_UPDATE_MODIFY); + + __wt_modify_vector_pop(&modifies, &prev_upd); + + /* + * Set the stop timestamp from durable timestamp instead of commit timestamp. The + * Garbage collection of history store removes the history values once the stop + * timestamp is globally visible. i.e. durable timestamp of data store version. + */ + WT_ASSERT(session, prev_upd->start_ts <= prev_upd->durable_ts); + stop_ts_pair.timestamp = prev_upd->durable_ts; + stop_ts_pair.txnid = prev_upd->txnid; + + if (prev_upd->type == WT_UPDATE_TOMBSTONE) { + WT_ASSERT(session, modifies.size > 0); + if (prev_upd->start_ts == WT_TS_NONE) { + WT_ERR(__wt_hs_delete_key(session, btree->id, key)); + WT_STAT_CONN_INCR(session, cache_hs_key_truncate_mix_ts); + } + __wt_modify_vector_pop(&modifies, &prev_upd); + WT_ASSERT(session, prev_upd->type == WT_UPDATE_STANDARD); + prev_full_value->data = prev_upd->data; + prev_full_value->size = prev_upd->size; + } else + WT_ERR(__hs_calculate_full_value( + session, prev_full_value, prev_upd, full_value->data, full_value->size)); + + /* + * Skip the updates have the same start timestamp and transaction id + * + * Modifies that have the same start time pair as the onpage_upd can be squashed away. + */ + if (upd->start_ts != prev_upd->start_ts || upd->txnid != prev_upd->txnid) { + /* + * Calculate reverse delta. Insert full update for the newest historical record even + * it's a MODIFY. + * + * It is not correct to check prev_upd == list->onpage_upd as we may have aborted + * updates in the middle. + */ + nentries = MAX_REVERSE_MODIFY_NUM; + if (!F_ISSET(upd, WT_UPDATE_HS)) { + if (upd->type == WT_UPDATE_MODIFY && + __wt_calc_modify(session, prev_full_value, full_value, + prev_full_value->size / 10, entries, &nentries) == 0) { + WT_ERR(__wt_modify_pack(cursor, entries, nentries, &modify_value)); + WT_ERR(__hs_insert_record(session, cursor, btree, key, upd, + WT_UPDATE_MODIFY, modify_value, stop_ts_pair)); + __wt_scr_free(session, &modify_value); + } else + WT_ERR(__hs_insert_record(session, cursor, btree, key, upd, + WT_UPDATE_STANDARD, full_value, stop_ts_pair)); + + /* Flag the update as now in the history store. */ + F_SET(upd, WT_UPDATE_HS); + ++insert_cnt; + if (squashed) { + WT_STAT_CONN_INCR(session, cache_hs_write_squash); + squashed = false; + } + } + } else + squashed = true; + } + + if (modifies.size > 0) + WT_STAT_CONN_INCR(session, cache_hs_write_squash); + } + + WT_ERR(__wt_block_manager_named_size(session, WT_HS_FILE, &hs_size)); + WT_STAT_CONN_SET(session, cache_hs_ondisk, hs_size); + max_hs_size = ((WT_CURSOR_BTREE *)cursor)->btree->file_max; + if (max_hs_size != 0 && (uint64_t)hs_size > max_hs_size) + WT_PANIC_ERR(session, WT_PANIC, "WiredTigerHS: file size of %" PRIu64 + " exceeds maximum " + "size %" PRIu64, + (uint64_t)hs_size, max_hs_size); + +err: + if (ret == 0 && insert_cnt > 0) + __hs_insert_updates_verbose(session, btree); + + __wt_scr_free(session, &key); + /* modify_value is allocated in __wt_modify_pack. Free it if it is allocated. */ + if (modify_value != NULL) + __wt_scr_free(session, &modify_value); + __wt_modify_vector_free(&modifies); + __wt_scr_free(session, &full_value); + __wt_scr_free(session, &prev_full_value); + return (ret); +} + +/* + * __wt_hs_cursor_position -- + * Position a history store cursor at the end of a set of updates for a given btree id, record + * key and timestamp. There may be no history store entries for the given btree id and record + * key if they have been removed by WT_CONNECTION::rollback_to_stable. + */ +int +__wt_hs_cursor_position(WT_SESSION_IMPL *session, WT_CURSOR *cursor, uint32_t btree_id, + WT_ITEM *key, wt_timestamp_t timestamp) +{ + WT_DECL_ITEM(srch_key); + WT_DECL_RET; + int cmp, exact; + + WT_RET(__wt_scr_alloc(session, 0, &srch_key)); + + /* + * Because of the special visibility rules for the history store, a new key can appear in + * between our search and the set of updates that we're interested in. Keep trying until we find + * it. + * + * There may be no history store entries for the given btree id and record key if they have been + * removed by WT_CONNECTION::rollback_to_stable. + * + * Note that we need to compare the raw key off the cursor to determine where we are in the + * history store as opposed to comparing the embedded data store key since the ordering is not + * guaranteed to be the same. + * + * FIXME: We should be repeatedly moving the cursor backwards within the loop instead of doing a + * search near operation each time as it is cheaper. + */ + cursor->set_key( + cursor, btree_id, key, timestamp != WT_TS_NONE ? timestamp : WT_TS_MAX, UINT64_MAX); + /* Copy the raw key before searching as a basis for comparison. */ + WT_ERR(__wt_buf_set(session, srch_key, cursor->key.data, cursor->key.size)); + WT_ERR(cursor->search_near(cursor, &exact)); + if (exact > 0) { + /* + * It's possible that we may race with a history store insert for another key. So we may be + * more than one record away the end of our target key/timestamp range. Keep iterating + * backwards until we land on our key. + */ + while ((ret = cursor->prev(cursor)) == 0) { + WT_ERR(__wt_compare(session, NULL, &cursor->key, srch_key, &cmp)); + if (cmp <= 0) + break; + } + } +#ifdef HAVE_DIAGNOSTIC + if (ret == 0) { + WT_ERR(__wt_compare(session, NULL, &cursor->key, srch_key, &cmp)); + WT_ASSERT(session, cmp <= 0); + } +#endif +err: + __wt_scr_free(session, &srch_key); + return (ret); +} + +/* + * __hs_save_read_timestamp -- + * Save the currently running transaction's read timestamp into a variable. + */ +static void +__hs_save_read_timestamp(WT_SESSION_IMPL *session, wt_timestamp_t *saved_timestamp) +{ + *saved_timestamp = session->txn.read_timestamp; +} + +/* + * __hs_restore_read_timestamp -- + * Reset the currently running transaction's read timestamp with a previously saved one. + */ +static void +__hs_restore_read_timestamp(WT_SESSION_IMPL *session, wt_timestamp_t saved_timestamp) +{ + session->txn.read_timestamp = saved_timestamp; +} + +/* + * __wt_find_hs_upd -- + * Scan the history store for a record the btree cursor wants to position on. Create an update + * for the record and return to the caller. The caller may choose to optionally allow prepared + * updates to be returned regardless of whether prepare is being ignored globally. Otherwise, a + * prepare conflict will be returned upon reading a prepared update. + */ +int +__wt_find_hs_upd(WT_SESSION_IMPL *session, WT_CURSOR_BTREE *cbt, WT_UPDATE **updp, + bool allow_prepare, WT_ITEM *on_disk_buf) +{ + WT_CURSOR *hs_cursor; + WT_DECL_ITEM(hs_key); + WT_DECL_ITEM(hs_value); + WT_DECL_ITEM(orig_hs_value_buf); + WT_DECL_RET; + WT_ITEM *key, _key; + WT_MODIFY_VECTOR modifies; + WT_TXN *txn; + WT_UPDATE *mod_upd, *upd; + wt_timestamp_t durable_timestamp, durable_timestamp_tmp, hs_start_ts, hs_start_ts_tmp; + wt_timestamp_t hs_stop_ts, hs_stop_ts_tmp, read_timestamp, saved_timestamp; + size_t notused, size; + uint64_t hs_counter, hs_counter_tmp, upd_type_full; + uint32_t hs_btree_id, session_flags; + uint8_t *p, recno_key[WT_INTPACK64_MAXSIZE], upd_type; + int cmp; + bool is_owner, modify; + + *updp = NULL; + hs_cursor = NULL; + key = NULL; + mod_upd = upd = NULL; + orig_hs_value_buf = NULL; + __wt_modify_vector_init(session, &modifies); + txn = &session->txn; + __hs_save_read_timestamp(session, &saved_timestamp); + notused = size = 0; + hs_btree_id = S2BT(session)->id; + session_flags = 0; /* [-Werror=maybe-uninitialized] */ + WT_NOT_READ(modify, false); + is_owner = false; + + /* Row-store has the key available, create the column-store key on demand. */ + switch (cbt->btree->type) { + case BTREE_ROW: + key = &cbt->iface.key; + break; + case BTREE_COL_FIX: + case BTREE_COL_VAR: + p = recno_key; + WT_RET(__wt_vpack_uint(&p, 0, cbt->recno)); + WT_CLEAR(_key); + _key.data = recno_key; + _key.size = WT_PTRDIFF(p, recno_key); + key = &_key; + } + + /* Allocate buffers for the history store key/value. */ + WT_ERR(__wt_scr_alloc(session, 0, &hs_key)); + WT_ERR(__wt_scr_alloc(session, 0, &hs_value)); + + /* Open a history store table cursor. */ + WT_ERR(__wt_hs_cursor(session, &session_flags, &is_owner)); + hs_cursor = session->hs_cursor; + + /* + * After positioning our cursor, we're stepping backwards to find the correct update. Since the + * timestamp is part of the key, our cursor needs to go from the newest record (further in the + * las) to the oldest (earlier in the las) for a given key. + */ + read_timestamp = allow_prepare ? txn->prepare_timestamp : txn->read_timestamp; + ret = __wt_hs_cursor_position(session, hs_cursor, hs_btree_id, key, read_timestamp); + if (ret == WT_NOTFOUND) { + ret = 0; + goto done; + } + WT_ERR(ret); + WT_ERR(hs_cursor->get_key(hs_cursor, &hs_btree_id, hs_key, &hs_start_ts, &hs_counter)); + + /* Stop before crossing over to the next btree */ + if (hs_btree_id != S2BT(session)->id) + goto done; + + /* + * Keys are sorted in an order, skip the ones before the desired key, and bail out if we have + * crossed over the desired key and not found the record we are looking for. + */ + WT_ERR(__wt_compare(session, NULL, hs_key, key, &cmp)); + if (cmp != 0) + goto done; + + WT_ERR( + hs_cursor->get_value(hs_cursor, &hs_stop_ts, &durable_timestamp, &upd_type_full, hs_value)); + upd_type = (uint8_t)upd_type_full; + + /* We do not have tombstones in the history store anymore. */ + WT_ASSERT(session, upd_type != WT_UPDATE_TOMBSTONE); + + /* + * Keep walking until we get a non-modify update. Once we get to that point, squash the updates + * together. + */ + if (upd_type == WT_UPDATE_MODIFY) { + WT_NOT_READ(modify, true); + /* Store this so that we don't have to make a special case for the first modify. */ + hs_stop_ts_tmp = hs_stop_ts; + while (upd_type == WT_UPDATE_MODIFY) { + WT_ERR(__wt_update_alloc(session, hs_value, &mod_upd, ¬used, upd_type)); + WT_ERR(__wt_modify_vector_push(&modifies, mod_upd)); + mod_upd = NULL; + + /* + * Each entry in the lookaside is written with the actual start and stop time pair + * embedded in the key. In order to traverse a sequence of modifies, we're going to have + * to manipulate our read timestamp to see records we wouldn't otherwise be able to see. + * + * In this case, we want to read the next update in the chain meaning that its start + * timestamp should be equivalent to the stop timestamp of the record that we're + * currently on. + */ + session->txn.read_timestamp = hs_stop_ts_tmp; + + /* + * Find the base update to apply the reverse deltas. If our cursor next fails to find an + * update here we fall back to the datastore version. If its timestamp doesn't match our + * timestamp then we return not found. + */ + if ((ret = hs_cursor->next(hs_cursor)) == WT_NOTFOUND) { + /* Fallback to the onpage value as the base value. */ + orig_hs_value_buf = hs_value; + hs_value = on_disk_buf; + upd_type = WT_UPDATE_STANDARD; + break; + } + hs_start_ts_tmp = WT_TS_NONE; + /* + * Make sure we use the temporary variants of these variables. We need to retain the + * timestamps of the original modify we saw. + * + * We keep looking back into history store until we find a base update to apply the + * reverse deltas on top of. + */ + WT_ERR(hs_cursor->get_key( + hs_cursor, &hs_btree_id, hs_key, &hs_start_ts_tmp, &hs_counter_tmp)); + + WT_ERR(__wt_compare(session, NULL, hs_key, key, &cmp)); + + if (cmp != 0) { + /* Fallback to the onpage value as the base value. */ + orig_hs_value_buf = hs_value; + hs_value = on_disk_buf; + upd_type = WT_UPDATE_STANDARD; + break; + } + + WT_ERR(hs_cursor->get_value( + hs_cursor, &hs_stop_ts_tmp, &durable_timestamp_tmp, &upd_type_full, hs_value)); + upd_type = (uint8_t)upd_type_full; + } + + WT_ASSERT(session, upd_type == WT_UPDATE_STANDARD); + while (modifies.size > 0) { + __wt_modify_vector_pop(&modifies, &mod_upd); + WT_ERR(__wt_modify_apply_item(session, hs_value, mod_upd->data, false)); + __wt_free_update_list(session, &mod_upd); + mod_upd = NULL; + } + /* After we're done looping over modifies, reset the read timestamp. */ + __hs_restore_read_timestamp(session, saved_timestamp); + WT_STAT_CONN_INCR(session, cache_hs_read_squash); + } + + /* Allocate an update structure for the record found. */ + WT_ERR(__wt_update_alloc(session, hs_value, &upd, &size, upd_type)); + upd->txnid = WT_TXN_NONE; + upd->durable_ts = durable_timestamp; + upd->start_ts = hs_start_ts; + upd->prepare_state = upd->start_ts == upd->durable_ts ? WT_PREPARE_INIT : WT_PREPARE_RESOLVED; + + /* + * We're not keeping this in our update list as we want to get rid of it after the read has been + * dealt with. Mark this update as external and to be discarded when not needed. + */ + F_SET(upd, WT_UPDATE_RESTORED_FROM_DISK); + *updp = upd; + +done: +err: + if (orig_hs_value_buf != NULL) + __wt_scr_free(session, &orig_hs_value_buf); + else + __wt_scr_free(session, &hs_value); + __wt_scr_free(session, &hs_key); + + /* + * Restore the read timestamp if we encountered an error while processing a modify. There's no + * harm in doing this multiple times. + */ + __hs_restore_read_timestamp(session, saved_timestamp); + WT_TRET(__wt_hs_cursor_close(session, session_flags, is_owner)); + + __wt_free_update_list(session, &mod_upd); + while (modifies.size > 0) { + __wt_modify_vector_pop(&modifies, &upd); + __wt_free_update_list(session, &upd); + } + __wt_modify_vector_free(&modifies); + + if (ret == 0) { + /* Couldn't find a record. */ + if (upd == NULL) { + ret = WT_NOTFOUND; + WT_STAT_CONN_INCR(session, cache_hs_read_miss); + } else { + WT_STAT_CONN_INCR(session, cache_hs_read); + WT_STAT_DATA_INCR(session, cache_hs_read); + } + } + + WT_ASSERT(session, upd != NULL || ret != 0); + + return (ret); +} + +/* + * __hs_delete_key_int -- + * Internal helper for deleting history store content for a given key. + */ +static int +__hs_delete_key_int(WT_SESSION_IMPL *session, uint32_t btree_id, const WT_ITEM *key) +{ + WT_CURSOR *hs_cursor; + WT_DECL_ITEM(srch_key); + WT_DECL_RET; + WT_ITEM hs_key; + wt_timestamp_t hs_start_ts; + uint64_t hs_counter; + uint32_t hs_btree_id; + int cmp, exact; + + hs_cursor = session->hs_cursor; + WT_RET(__wt_scr_alloc(session, 0, &srch_key)); + + hs_cursor->set_key(hs_cursor, btree_id, key, WT_TS_NONE, (uint64_t)0); + WT_ERR(__wt_buf_set(session, srch_key, hs_cursor->key.data, hs_cursor->key.size)); + ret = hs_cursor->search_near(hs_cursor, &exact); + /* Empty history store is fine. */ + if (ret == WT_NOTFOUND) + goto done; + WT_ERR(ret); + /* + * If we raced with a history store insert, we may be two or more records away from our target. + * Keep iterating forwards until we are on or past our target key. + * + * We can't use the cursor positioning helper that we use for regular reads since that will + * place us at the end of a particular key/timestamp range whereas we want to be placed at the + * beginning. + */ + if (exact < 0) { + while ((ret = hs_cursor->next(hs_cursor)) == 0) { + WT_ERR(__wt_compare(session, NULL, &hs_cursor->key, srch_key, &cmp)); + if (cmp >= 0) + break; + } + /* No entries greater than or equal to the key we searched for. */ + if (ret == WT_NOTFOUND) + goto done; + WT_ERR(ret); + } + /* Bailing out here also means we have no history store records for our key. */ + WT_ERR(hs_cursor->get_key(hs_cursor, &hs_btree_id, &hs_key, &hs_start_ts, &hs_counter)); + if (hs_btree_id != btree_id) + goto done; + WT_ERR(__wt_compare(session, NULL, &hs_key, key, &cmp)); + if (cmp != 0) + goto done; + WT_ERR(__hs_delete_key_from_pos(session, hs_cursor, btree_id, key)); +done: + ret = 0; +err: + __wt_scr_free(session, &srch_key); + return (ret); +} + +/* + * __wt_hs_delete_key -- + * Delete an entire key's worth of data in the history store. + */ +int +__wt_hs_delete_key(WT_SESSION_IMPL *session, uint32_t btree_id, const WT_ITEM *key) +{ + WT_DECL_RET; + uint32_t session_flags; + bool is_owner; + + session_flags = session->flags; + + /* + * Some code paths such as schema removal involve deleting keys in metadata and assert that we + * shouldn't be opening new dhandles. We won't ever need to blow away history store content in + * these cases so let's just return early here. + */ + if (F_ISSET(session, WT_SESSION_NO_DATA_HANDLES)) + return (0); + + WT_RET(__wt_hs_cursor(session, &session_flags, &is_owner)); + /* + * In order to delete a key range, we need to be able to inspect all history store records + * regardless of their stop time pairs. + */ + F_SET(session, WT_SESSION_IGNORE_HS_TOMBSTONE); + /* The tree structure can change while we try to insert the mod list, retry if that happens. */ + while ((ret = __hs_delete_key_int(session, btree_id, key)) == WT_RESTART) + ; + + if (!FLD_ISSET(session_flags, WT_SESSION_IGNORE_HS_TOMBSTONE)) + F_CLR(session, WT_SESSION_IGNORE_HS_TOMBSTONE); + WT_TRET(__wt_hs_cursor_close(session, session_flags, is_owner)); + return (ret); +} + +/* + * __hs_delete_key_from_pos -- + * Delete an entire key's worth of data in the history store assuming that the input cursor is + * positioned at the beginning of the key range. + */ +static int +__hs_delete_key_from_pos( + WT_SESSION_IMPL *session, WT_CURSOR *hs_cursor, uint32_t btree_id, const WT_ITEM *key) +{ + WT_CURSOR_BTREE *hs_cbt; + WT_DECL_RET; + WT_ITEM hs_key; + WT_UPDATE *upd; + wt_timestamp_t hs_start_ts; + size_t size; + uint64_t hs_counter; + uint32_t hs_btree_id; + int cmp; + + hs_cbt = (WT_CURSOR_BTREE *)hs_cursor; + upd = NULL; + + /* If there is nothing else in history store, we're done here. */ + for (; ret == 0; ret = hs_cursor->next(hs_cursor)) { + WT_RET(hs_cursor->get_key(hs_cursor, &hs_btree_id, &hs_key, &hs_start_ts, &hs_counter)); + /* + * If the btree id or key isn't ours, that means that we've hit the end of the key range and + * that there is no more history store content for this key. + */ + if (hs_btree_id != btree_id) + break; + WT_RET(__wt_compare(session, NULL, &hs_key, key, &cmp)); + if (cmp != 0) + break; + /* + * Since we're using internal functions to modify the row structure, we need to manually set + * the comparison to an exact match. + */ + hs_cbt->compare = 0; + /* + * Append a globally visible tombstone to the update list. This will effectively make the + * value invisible and the key itself will eventually get removed during reconciliation. + */ + WT_RET(__wt_update_alloc(session, NULL, &upd, &size, WT_UPDATE_TOMBSTONE)); + upd->txnid = WT_TXN_NONE; + upd->start_ts = upd->durable_ts = WT_TS_NONE; + WT_ERR(__wt_hs_modify(hs_cbt, upd)); + upd = NULL; + WT_STAT_CONN_INCR(session, cache_hs_remove_key_truncate); + } + if (ret == WT_NOTFOUND) + return (0); +err: + __wt_free(session, upd); + return (ret); +} diff --git a/src/third_party/wiredtiger/src/include/api.h b/src/third_party/wiredtiger/src/include/api.h index 484331e2752..81118e421d2 100644 --- a/src/third_party/wiredtiger/src/include/api.h +++ b/src/third_party/wiredtiger/src/include/api.h @@ -200,7 +200,8 @@ #define CURSOR_API_CALL(cur, s, n, bt) \ (s) = (WT_SESSION_IMPL *)(cur)->session; \ - SESSION_API_PREPARE_CHECK(s, WT_CURSOR, n); \ + if (!F_ISSET(s, WT_SESSION_HS_CURSOR)) \ + SESSION_API_PREPARE_CHECK(s, WT_CURSOR, n); \ API_CALL_NOCONF(s, WT_CURSOR, n, ((bt) == NULL) ? NULL : ((WT_BTREE *)(bt))->dhandle); \ if (F_ISSET(cur, WT_CURSTD_CACHED)) \ WT_ERR(__wt_cursor_cached(cur)) diff --git a/src/third_party/wiredtiger/src/include/btmem.h b/src/third_party/wiredtiger/src/include/btmem.h index 1fcfb9c2033..e9f728e3ef9 100644 --- a/src/third_party/wiredtiger/src/include/btmem.h +++ b/src/third_party/wiredtiger/src/include/btmem.h @@ -10,10 +10,10 @@ /* AUTOMATIC FLAG VALUE GENERATION START */ #define WT_READ_CACHE 0x0001u -#define WT_READ_DELETED_CHECK 0x0002u -#define WT_READ_DELETED_SKIP 0x0004u -#define WT_READ_IGNORE_CACHE_SIZE 0x0008u -#define WT_READ_LOOKASIDE 0x0010u +#define WT_READ_CACHE_LEAF 0x0002u +#define WT_READ_DELETED_CHECK 0x0004u +#define WT_READ_DELETED_SKIP 0x0008u +#define WT_READ_IGNORE_CACHE_SIZE 0x0010u #define WT_READ_NOTFOUND_OK 0x0020u #define WT_READ_NO_GEN 0x0040u #define WT_READ_NO_SPLIT 0x0080u @@ -27,11 +27,11 @@ /* AUTOMATIC FLAG VALUE GENERATION START */ #define WT_REC_CHECKPOINT 0x01u -#define WT_REC_EVICT 0x02u -#define WT_REC_IN_MEMORY 0x04u -#define WT_REC_LOOKASIDE 0x08u -#define WT_REC_SCRUB 0x10u -#define WT_REC_UPDATE_RESTORE 0x20u +#define WT_REC_CLEAN_AFTER_REC 0x02u +#define WT_REC_EVICT 0x04u +#define WT_REC_HS 0x08u +#define WT_REC_IN_MEMORY 0x10u +#define WT_REC_SCRUB 0x20u #define WT_REC_VISIBILITY_ERR 0x40u #define WT_REC_VISIBLE_ALL 0x80u /* AUTOMATIC FLAG VALUE GENERATION STOP */ @@ -74,15 +74,10 @@ struct __wt_page_header { #define WT_PAGE_EMPTY_V_ALL 0x02u /* Page has all zero-length values */ #define WT_PAGE_EMPTY_V_NONE 0x04u /* Page has no zero-length values */ #define WT_PAGE_ENCRYPTED 0x08u /* Page is encrypted on disk */ -#define WT_PAGE_LAS_UPDATE 0x10u /* Page updates in lookaside store */ uint8_t flags; /* 25: flags */ /* A byte of padding, positioned to be added to the flags. */ uint8_t unused; /* 26: unused padding */ - -#define WT_PAGE_VERSION_ORIG 0 /* Original version */ -#define WT_PAGE_VERSION_TS 1 /* Timestamps added */ - uint8_t version; /* 27: version */ }; /* * WT_PAGE_HEADER_SIZE is the number of bytes we allocate for the structure: if the compiler inserts @@ -127,11 +122,12 @@ __wt_page_header_byteswap(WT_PAGE_HEADER *dsk) */ struct __wt_addr { /* Validity window */ - wt_timestamp_t newest_durable_ts; wt_timestamp_t oldest_start_ts; uint64_t oldest_start_txn; + wt_timestamp_t start_durable_ts; wt_timestamp_t newest_stop_ts; uint64_t newest_stop_txn; + wt_timestamp_t stop_durable_ts; uint8_t *addr; /* Block-manager's cookie */ uint8_t size; /* Block-manager's cookie length */ @@ -152,6 +148,26 @@ struct __wt_addr { }; /* + * WT_ADDR_COPY -- + * We have to lock the WT_REF to look at a WT_ADDR: a structure we can use to quickly get a + * copy of the WT_REF address information. + */ +struct __wt_addr_copy { + /* Validity window */ + wt_timestamp_t oldest_start_ts; + uint64_t oldest_start_txn; + wt_timestamp_t start_durable_ts; + wt_timestamp_t newest_stop_ts; + uint64_t newest_stop_txn; + wt_timestamp_t stop_durable_ts; + + uint8_t type; + + uint8_t addr[255 /* WT_BTREE_MAX_ADDR_COOKIE */]; + uint8_t size; +}; + +/* * Overflow tracking for reuse: When a page is reconciled, we write new K/V overflow items. If pages * are reconciled multiple times, we need to know if we've already written a particular overflow * record (so we don't write it again), as well as if we've modified an overflow record previously @@ -191,70 +207,44 @@ struct __wt_ovfl_reuse { }; /* - * Lookaside table support: when a page is being reconciled for eviction and has - * updates that might be required by earlier readers in the system, the updates - * are written into a lookaside table, and restored as necessary if the page is - * read. + * History store table support: when a page is being reconciled for eviction and has updates that + * might be required by earlier readers in the system, the updates are written into the history + * store table, and restored as necessary if the page is read. * - * The key is a unique marker for the page (a page ID plus a file ID, ordered - * this way so that overall the lookaside table is append-mostly), a counter - * (used to ensure the update records remain in the original order), and the - * record's key (byte-string for row-store, record number for column-store). + * The first part of the key is comprised of a file ID, record key (byte-string for row-store, + * record number for column-store) and timestamp. This allows us to search efficiently for a given + * record key and read timestamp combination. The last part of the key is a monotonically increasing + * counter to keep the key unique in the case where we have multiple transactions committing at the + * same timestamp. * The value is the WT_UPDATE structure's: - * - transaction ID - * - timestamp + * - stop timestamp * - durable timestamp - * - update's prepare state * - update type * - value. * - * As the key for the lookaside table is different for row- and column-store, we - * store both key types in a WT_ITEM, building/parsing them in the code, because - * otherwise we'd need two lookaside files with different key formats. We could - * make the lookaside table's key standard by moving the source key into the - * lookaside table value, but that doesn't make the coding any simpler, and it - * makes the lookaside table's value more likely to overflow the page size when - * the row-store key is relatively large. + * As the key for the history store table is different for row- and column-store, we store both key + * types in a WT_ITEM, building/parsing them in the code, because otherwise we'd need two + * history store files with different key formats. We could make the history store table's key + * standard by moving the source key into the history store table value, but that doesn't make the + * coding any simpler, and it makes the history store table's value more likely to overflow the page + * size when the row-store key is relatively large. + * + * Note that we deliberately store the update type as larger than necessary (8 bytes vs 1 byte). + * We've done this to leave room in case we need to store extra bit flags in this value at a later + * point. If we need to store more information, we can potentially tack extra information at the end + * of the "value" buffer and then use bit flags within the update type to determine how to interpret + * it. */ #ifdef HAVE_BUILTIN_EXTENSION_SNAPPY -#define WT_LOOKASIDE_COMPRESSOR "snappy" +#define WT_HS_COMPRESSOR "snappy" #else -#define WT_LOOKASIDE_COMPRESSOR "none" +#define WT_HS_COMPRESSOR "none" #endif -#define WT_LAS_CONFIG \ - "key_format=" WT_UNCHECKED_STRING(QIQu) ",value_format=" WT_UNCHECKED_STRING( \ - QQQBBu) ",block_compressor=" WT_LOOKASIDE_COMPRESSOR \ - ",leaf_value_max=64MB" \ - ",prefix_compression=true" - -/* - * WT_PAGE_LOOKASIDE -- - * Information for on-disk pages with lookaside entries. - * - * This information is used to decide whether history evicted to lookaside is - * needed for a read, and when it is no longer needed at all. We track the - * newest update written to the disk image in `max_ondisk_ts`, and the oldest - * update skipped to choose the on-disk version in `min_skipped_ts`. If no - * updates were skipped, then the disk image contains the newest versions of - * all updates and `min_skipped_ts == WT_TS_MAX`. - * - * For reads without a timestamp, we check that there are no skipped updates - * and that the reader's snapshot can see everything on disk. - * - * For readers with a timestamp, it is safe to ignore lookaside if either - * (a) there are no skipped updates and everything on disk is visible, or - * (b) everything on disk is visible, and the minimum skipped update is in - * the future of the reader. - */ -struct __wt_page_lookaside { - uint64_t las_pageid; /* Page ID in lookaside */ - uint64_t max_txn; /* Maximum transaction ID */ - wt_timestamp_t max_ondisk_ts; /* Maximum timestamp on disk */ - wt_timestamp_t min_skipped_ts; /* Skipped in favor of disk version */ - bool eviction_to_lookaside; /* Revert to lookaside on eviction */ - bool has_prepares; /* One or more updates are prepared */ - bool resolved; /* History has been read into cache */ -}; +#define WT_HS_CONFIG \ + "key_format=" WT_UNCHECKED_STRING(IuQQ) ",value_format=" WT_UNCHECKED_STRING( \ + QQQu) ",block_compressor=" WT_HS_COMPRESSOR \ + ",leaf_value_max=64MB" \ + ",prefix_compression=false" /* * WT_PAGE_MODIFY -- @@ -313,16 +303,11 @@ struct __wt_page_modify { * in memory. */ void *disk_image; - - /* The page has lookaside entries. */ - WT_PAGE_LOOKASIDE page_las; } r; #undef mod_replace #define mod_replace u1.r.replace #undef mod_disk_image #define mod_disk_image u1.r.disk_image -#undef mod_page_las -#define mod_page_las u1.r.page_las struct { /* Multiple replacement blocks */ struct __wt_multi { @@ -343,7 +328,7 @@ struct __wt_page_modify { /* * List of unresolved updates. Updates are either a row-store insert or update list, - * or column-store insert list. When creating lookaside records, there is an + * or column-store insert list. When creating history store records, there is an * additional value, the committed item's transaction information. * * If there are unresolved updates, the block wasn't written and there will always @@ -365,8 +350,6 @@ struct __wt_page_modify { WT_ADDR addr; uint32_t size; uint32_t checksum; - - WT_PAGE_LOOKASIDE page_las; } * multi; uint32_t multi_entries; /* Multiple blocks element count */ } m; @@ -493,7 +476,7 @@ struct __wt_page_modify { #define WT_PM_REC_REPLACE 3 /* Reconciliation: single block */ uint8_t rec_result; /* Reconciliation state */ -#define WT_PAGE_RS_LOOKASIDE 0x1 +#define WT_PAGE_RS_HS 0x1 #define WT_PAGE_RS_RESTORED 0x2 uint8_t restore_state; /* Created by restoring updates */ }; @@ -781,10 +764,6 @@ struct __wt_page { * row-store leaf pages without reading them if they don't reference * overflow items. * - * WT_REF_LIMBO: - * The page image has been loaded into memory but there is additional - * history in the lookaside table that has not been applied. - * * WT_REF_LOCKED: * Locked for exclusive access. In eviction, this page or a parent has * been selected for eviction; once hazard pointers are checked, the page @@ -793,19 +772,10 @@ struct __wt_page { * thread that set the page to WT_REF_LOCKED has exclusive access, no * other thread may use the WT_REF until the state is changed. * - * WT_REF_LOOKASIDE: - * The page is on disk (as per WT_REF_DISK) and has entries in the - * lookaside table that must be applied before the page can be read. - * * WT_REF_MEM: * Set by a reading thread once the page has been read from disk; the page * is in the cache and the page reference is OK. * - * WT_REF_READING: - * Set by a reading thread before reading an ordinary page from disk; - * other readers of the page wait until the read completes. Sync can - * safely skip over such pages: they are clean by definition. - * * WT_REF_SPLIT: * Set when the page is split; the WT_REF is dead and can no longer be * used. @@ -845,15 +815,23 @@ struct __wt_page_deleted { */ volatile uint8_t prepare_state; /* Prepare state. */ - uint32_t previous_state; /* Previous state */ + uint8_t previous_state; /* Previous state */ WT_UPDATE **update_list; /* List of updates for abort */ }; /* + * WT_TIME_PAIR -- + * A pair containing a timestamp and transaction id. + */ +struct __wt_time_pair { + wt_timestamp_t timestamp; + uint64_t txnid; +}; + +/* * WT_REF -- - * A single in-memory page and the state information used to determine if - * it's OK to dereference the pointer to the page. + * A single in-memory page and state information. */ struct __wt_ref { WT_PAGE *page; /* Page */ @@ -865,15 +843,27 @@ struct __wt_ref { WT_PAGE *volatile home; /* Reference page */ volatile uint32_t pindex_hint; /* Reference page index hint */ -#define WT_REF_DISK 0 /* Page is on disk */ -#define WT_REF_DELETED 1 /* Page is on disk, but deleted */ -#define WT_REF_LIMBO 2 /* Page is in cache without history */ -#define WT_REF_LOCKED 3 /* Page locked for exclusive access */ -#define WT_REF_LOOKASIDE 4 /* Page is on disk with lookaside */ -#define WT_REF_MEM 5 /* Page is in cache and valid */ -#define WT_REF_READING 6 /* Page being read */ -#define WT_REF_SPLIT 7 /* Parent page split (WT_REF dead) */ - volatile uint32_t state; /* Page state */ + uint8_t unused[2]; /* Padding: before the flags field so flags can be easily expanded. */ + +/* + * Define both internal- and leaf-page flags for now: we only need one, but it provides an easy way + * to assert a page-type flag is always set (we allocate WT_REFs in lots of places and it's easy to + * miss one). If we run out of bits in the flags field, remove the internal flag and rewrite tests + * depending on it to be "!leaf" instead. + */ +/* AUTOMATIC FLAG VALUE GENERATION START */ +#define WT_REF_FLAG_INTERNAL 0x1u /* Page is an internal page */ +#define WT_REF_FLAG_LEAF 0x2u /* Page is a leaf page */ +#define WT_REF_FLAG_READING 0x4u /* Page is being read in */ + /* AUTOMATIC FLAG VALUE GENERATION STOP */ + uint8_t flags; + +#define WT_REF_DISK 0 /* Page is on disk */ +#define WT_REF_DELETED 1 /* Page is on disk, but deleted */ +#define WT_REF_LOCKED 2 /* Page locked for exclusive access */ +#define WT_REF_MEM 3 /* Page is in cache and valid */ +#define WT_REF_SPLIT 4 /* Parent page split (WT_REF dead) */ + volatile uint8_t state; /* Page state */ /* * Address: on-page cell if read from backing block, off-page WT_ADDR if instantiated in-memory, @@ -894,8 +884,7 @@ struct __wt_ref { #undef ref_ikey #define ref_ikey key.ikey - WT_PAGE_DELETED *page_del; /* Deleted page information */ - WT_PAGE_LOOKASIDE *page_las; /* Lookaside information */ + WT_PAGE_DELETED *page_del; /* Deleted page information */ /* * In DIAGNOSTIC mode we overwrite the WT_REF on free to force failures. Don't clear the history in @@ -933,21 +922,36 @@ struct __wt_ref { #else #define WT_REF_SET_STATE(ref, s) WT_PUBLISH((ref)->state, s) #endif - -/* A macro wrapper allowing us to remember the callers code location */ -#define WT_REF_CAS_STATE(session, ref, old_state, new_state) \ - __wt_ref_cas_state_int(session, ref, old_state, new_state, __func__, __LINE__) }; + /* * WT_REF_SIZE is the expected structure size -- we verify the build to ensure the compiler hasn't * inserted padding which would break the world. */ #ifdef HAVE_DIAGNOSTIC -#define WT_REF_SIZE (56 + WT_REF_SAVE_STATE_MAX * sizeof(WT_REF_HIST) + 8) +#define WT_REF_SIZE (48 + WT_REF_SAVE_STATE_MAX * sizeof(WT_REF_HIST) + 8) #else -#define WT_REF_SIZE 56 +#define WT_REF_SIZE 48 #endif +/* A macro wrapper allowing us to remember the callers code location */ +#define WT_REF_CAS_STATE(session, ref, old_state, new_state) \ + __wt_ref_cas_state_int(session, ref, old_state, new_state, __func__, __LINE__) + +#define WT_REF_LOCK(session, ref, previous_statep) \ + do { \ + uint8_t __previous_state; \ + for (;; __wt_yield()) { \ + __previous_state = (ref)->state; \ + if (__previous_state != WT_REF_LOCKED && \ + WT_REF_CAS_STATE(session, ref, __previous_state, WT_REF_LOCKED)) \ + break; \ + } \ + *(previous_statep) = __previous_state; \ + } while (0) + +#define WT_REF_UNLOCK(ref, state) WT_REF_SET_STATE(ref, state) + /* * WT_ROW -- * Each in-memory page row-store leaf page has an array of WT_ROW structures: @@ -1079,11 +1083,10 @@ struct __wt_update { uint32_t size; /* data length */ #define WT_UPDATE_INVALID 0 /* diagnostic check */ -#define WT_UPDATE_BIRTHMARK 1 /* transaction for on-page value */ -#define WT_UPDATE_MODIFY 2 /* partial-update modify value */ -#define WT_UPDATE_RESERVE 3 /* reserved */ -#define WT_UPDATE_STANDARD 4 /* complete value */ -#define WT_UPDATE_TOMBSTONE 5 /* deleted */ +#define WT_UPDATE_MODIFY 1 /* partial-update modify value */ +#define WT_UPDATE_RESERVE 2 /* reserved */ +#define WT_UPDATE_STANDARD 3 /* complete value */ +#define WT_UPDATE_TOMBSTONE 4 /* deleted */ uint8_t type; /* type (one byte to conserve memory) */ /* If the update includes a complete value. */ @@ -1096,6 +1099,13 @@ struct __wt_update { */ volatile uint8_t prepare_state; /* prepare state */ +/* AUTOMATIC FLAG VALUE GENERATION START */ +#define WT_UPDATE_HS 0x1u /* Update has been written to history store. */ +#define WT_UPDATE_RESTORED_FOR_ROLLBACK 0x2u /* Update restored for rollback to stable. */ +#define WT_UPDATE_RESTORED_FROM_DISK 0x4u /* Update is temporary retrieved from disk. */ + /* AUTOMATIC FLAG VALUE GENERATION STOP */ + uint8_t flags; + /* * Zero or more bytes of value (the payload) immediately follows the WT_UPDATE structure. We use * a C99 flexible array member which has the semantics we want. @@ -1107,7 +1117,7 @@ struct __wt_update { * WT_UPDATE_SIZE is the expected structure size excluding the payload data -- we verify the build * to ensure the compiler hasn't inserted padding. */ -#define WT_UPDATE_SIZE 38 +#define WT_UPDATE_SIZE 39 /* * The memory size of an update: include some padding because this is such a common case that @@ -1116,16 +1126,34 @@ struct __wt_update { #define WT_UPDATE_MEMSIZE(upd) WT_ALIGN(WT_UPDATE_SIZE + (upd)->size, 32) /* - * WT_MAX_MODIFY_UPDATE -- - * Limit update chains value to avoid penalizing reads and - * permit truncation. Having a smaller value will penalize the cases - * when history has to be maintained, resulting in multiplying cache - * pressure. + * WT_MAX_MODIFY_UPDATE, WT_MODIFY_VECTOR_STACK_SIZE + * Limit update chains value to avoid penalizing reads and permit truncation. Having a smaller + * value will penalize the cases when history has to be maintained, resulting in multiplying cache + * pressure. + * + * When threads race modifying a record, we can end up with more than the usual maximum number of + * modifications in an update list. We use small vectors of modify updates in a couple of places to + * avoid heap allocation, add a few additional slots to that array. */ #define WT_MAX_MODIFY_UPDATE 10 +#define WT_MODIFY_VECTOR_STACK_SIZE (WT_MAX_MODIFY_UPDATE + 10) + +/* + * WT_MODIFY_VECTOR -- + * A resizable array for storing modify updates. The allocation strategy is similar to that of + * llvm::SmallVector<T> where we keep space on the stack for the regular case but fall back to + * dynamic allocation as needed. + */ +struct __wt_modify_vector { + WT_SESSION_IMPL *session; + WT_UPDATE *list[WT_MODIFY_VECTOR_STACK_SIZE]; + WT_UPDATE **listp; + size_t allocated_bytes; + size_t size; +}; /* - * WT_MODIFY_MEM_FACTOR -- + * WT_MODIFY_MEM_FRACTION * Limit update chains to a fraction of the base document size. */ #define WT_MODIFY_MEM_FRACTION 10 diff --git a/src/third_party/wiredtiger/src/include/btree.h b/src/third_party/wiredtiger/src/include/btree.h index b50e9438337..53bd608efb0 100644 --- a/src/third_party/wiredtiger/src/include/btree.h +++ b/src/third_party/wiredtiger/src/include/btree.h @@ -155,8 +155,8 @@ struct __wt_btree { uint8_t original; /* Newly created: bulk-load possible (want a bool but needs atomic cas) */ - bool lookaside_entries; /* Has entries in the lookaside table */ - bool lsm_primary; /* Handle is/was the LSM primary */ + bool hs_entries; /* Has entries in the history store table */ + bool lsm_primary; /* Handle is/was the LSM primary */ WT_BM *bm; /* Block manager reference */ u_int block_header; /* WT_PAGE_HEADER_BYTE_SIZE */ @@ -164,6 +164,7 @@ struct __wt_btree { uint64_t write_gen; /* Write generation */ uint64_t rec_max_txn; /* Maximum txn seen (clean trees) */ wt_timestamp_t rec_max_timestamp; + uint64_t hs_counter; /* History store counter */ uint64_t checkpoint_gen; /* Checkpoint generation */ WT_SESSION_IMPL *sync_session; /* Syncing session */ @@ -192,7 +193,7 @@ struct __wt_btree { /* * The maximum bytes allowed to be used for the table on disk. This is currently only used for - * the lookaside table. + * the history store table. */ uint64_t file_max; @@ -238,7 +239,7 @@ struct __wt_btree { #define WT_BTREE_CLOSED 0x000400u /* Handle closed */ #define WT_BTREE_IGNORE_CACHE 0x000800u /* Cache-resident object */ #define WT_BTREE_IN_MEMORY 0x001000u /* Cache-resident object */ -#define WT_BTREE_LOOKASIDE 0x002000u /* Look-aside table */ +#define WT_BTREE_HS 0x002000u /* History store table */ #define WT_BTREE_NO_CHECKPOINT 0x004000u /* Disable checkpoints */ #define WT_BTREE_NO_LOGGING 0x008000u /* Disable logging */ #define WT_BTREE_READONLY 0x010000u /* Handle is readonly */ diff --git a/src/third_party/wiredtiger/src/include/btree.i b/src/third_party/wiredtiger/src/include/btree.i index f39a87d53c9..133277a4b3e 100644 --- a/src/third_party/wiredtiger/src/include/btree.i +++ b/src/third_party/wiredtiger/src/include/btree.i @@ -17,12 +17,45 @@ __wt_ref_is_root(WT_REF *ref) } /* + * __wt_ref_cas_state_int -- + * Try to do a compare and swap, if successful update the ref history in diagnostic mode. + */ +static inline bool +__wt_ref_cas_state_int(WT_SESSION_IMPL *session, WT_REF *ref, uint8_t old_state, uint8_t new_state, + const char *func, int line) +{ + bool cas_result; + + /* Parameters that are used in a macro for diagnostic builds */ + WT_UNUSED(session); + WT_UNUSED(func); + WT_UNUSED(line); + + cas_result = __wt_atomic_casv8(&ref->state, old_state, new_state); + +#ifdef HAVE_DIAGNOSTIC + /* + * The history update here has potential to race; if the state gets updated again after the CAS + * above but before the history has been updated. + */ + if (cas_result) + WT_REF_SAVE_STATE(ref, new_state, func, line); +#endif + return (cas_result); +} + +/* * __wt_page_is_empty -- * Return if the page is empty. */ static inline bool __wt_page_is_empty(WT_PAGE *page) { + /* + * Be cautious modifying this function: it's reading fields set by checkpoint reconciliation, + * and we're not blocking checkpoints (although we must block eviction as it might clear and + * free these structures). + */ return (page->modify != NULL && page->modify->rec_result == WT_PM_REC_EMPTY); } @@ -33,6 +66,11 @@ __wt_page_is_empty(WT_PAGE *page) static inline bool __wt_page_evict_clean(WT_PAGE *page) { + /* + * Be cautious modifying this function: it's reading fields set by checkpoint reconciliation, + * and we're not blocking checkpoints (although we must block eviction as it might clear and + * free these structures). + */ return (page->modify == NULL || (page->modify->page_state == WT_PAGE_CLEAN && page->modify->rec_result == 0)); } @@ -44,6 +82,11 @@ __wt_page_evict_clean(WT_PAGE *page) static inline bool __wt_page_is_modified(WT_PAGE *page) { + /* + * Be cautious modifying this function: it's reading fields set by checkpoint reconciliation, + * and we're not blocking checkpoints (although we must block eviction as it might clear and + * free these structures). + */ return (page->modify != NULL && page->modify->page_state != WT_PAGE_CLEAN); } @@ -149,6 +192,9 @@ __wt_cache_page_inmem_incr(WT_SESSION_IMPL *session, WT_PAGE *page, size_t size) btree = S2BT(session); cache = S2C(session)->cache; + if (size == 0) + return; + (void)__wt_atomic_add64(&btree->bytes_inmem, size); (void)__wt_atomic_add64(&cache->bytes_inmem, size); (void)__wt_atomic_addsize(&page->memory_footprint, size); @@ -469,8 +515,6 @@ __wt_page_only_modify_set(WT_SESSION_IMPL *session, WT_PAGE *page) { uint64_t last_running; - WT_ASSERT(session, !F_ISSET(session->dhandle, WT_DHANDLE_DEAD)); - last_running = 0; if (page->modify->page_state == WT_PAGE_CLEAN) last_running = S2C(session)->txn_global.last_running; @@ -632,23 +676,6 @@ __wt_off_page(WT_PAGE *page, const void *p) } /* - * __wt_ref_addr_free -- - * Free the address in a reference, if necessary. - */ -static inline void -__wt_ref_addr_free(WT_SESSION_IMPL *session, WT_REF *ref) -{ - if (ref->addr == NULL) - return; - - if (ref->home == NULL || __wt_off_page(ref->home, ref->addr)) { - __wt_free(session, ((WT_ADDR *)ref->addr)->addr); - __wt_free(session, ref->addr); - } - ref->addr = NULL; -} - -/* * __wt_ref_key -- * Return a reference to a row-store internal page key as cheaply as possible. */ @@ -1025,6 +1052,16 @@ __wt_row_leaf_value_cell(WT_SESSION_IMPL *session, WT_PAGE *page, WT_ROW *rip, } /* + * __wt_row_leaf_value_exists -- + * Check if the value for a row-store leaf page encoded key/value pair exists. + */ +static inline bool +__wt_row_leaf_value_exists(WT_ROW *rip) +{ + return (((uintptr_t)WT_ROW_KEY_COPY(rip) & 0x03) == WT_KV_FLAG); +} + +/* * __wt_row_leaf_value -- * Return the value for a row-store leaf page encoded key/value pair. */ @@ -1048,85 +1085,67 @@ __wt_row_leaf_value(WT_PAGE *page, WT_ROW *rip, WT_ITEM *value) } /* - * __wt_ref_info -- - * Return the addr/size and type triplet for a reference. + * __wt_ref_addr_copy -- + * Return a copy of the WT_REF address information. */ -static inline void -__wt_ref_info( - WT_SESSION_IMPL *session, WT_REF *ref, const uint8_t **addrp, size_t *sizep, bool *is_leafp) +static inline bool +__wt_ref_addr_copy(WT_SESSION_IMPL *session, WT_REF *ref, WT_ADDR_COPY *copy) { WT_ADDR *addr; WT_CELL_UNPACK *unpack, _unpack; WT_PAGE *page; - addr = ref->addr; unpack = &_unpack; page = ref->home; /* - * If NULL, there is no location. If off-page, the pointer references a WT_ADDR structure. If - * on-page, the pointer references a cell. - * - * The type is of a limited set: internal, leaf or no-overflow leaf. + * To look at an on-page cell, we need to look at the parent page's disk image, and that can be + * dangerous. The problem is if the parent page splits, deepening the tree. As part of that + * process, the WT_REF WT_ADDRs pointing into the parent's disk image are copied into off-page + * WT_ADDRs and swapped into place. The content of the two WT_ADDRs are identical, and we don't + * care which version we get as long as we don't mix-and-match the two. */ - if (addr == NULL) { - *addrp = NULL; - *sizep = 0; - if (is_leafp != NULL) - *is_leafp = false; - } else if (__wt_off_page(page, addr)) { - *addrp = addr->addr; - *sizep = addr->size; - if (is_leafp != NULL) - *is_leafp = addr->type != WT_ADDR_INT; - } else { - __wt_cell_unpack(session, page, (WT_CELL *)addr, unpack); - *addrp = unpack->data; - *sizep = unpack->size; + WT_ORDERED_READ(addr, ref->addr); - if (is_leafp != NULL) - *is_leafp = unpack->type != WT_CELL_ADDR_INT; - } -} - -/* - * __wt_ref_info_lock -- - * Lock the WT_REF and return the addr/size and type triplet for a reference. - */ -static inline void -__wt_ref_info_lock( - WT_SESSION_IMPL *session, WT_REF *ref, uint8_t *addr_buf, size_t *sizep, bool *is_leafp) -{ - size_t size; - uint32_t previous_state; - const uint8_t *addr; - bool is_leaf; + /* If NULL, there is no information. */ + if (addr == NULL) + return (false); - /* - * The WT_REF address references either an on-page cell or in-memory structure, and eviction - * frees both. If our caller is already blocking eviction (either because the WT_REF is locked - * or there's a hazard pointer on the page), no locking is required, and the caller should call - * the underlying function directly. Otherwise, our caller is not blocking eviction and we lock - * here, and copy out the address instead of returning a reference. - */ - for (;; __wt_yield()) { - previous_state = ref->state; - if (previous_state != WT_REF_LOCKED && previous_state != WT_REF_READING && - WT_REF_CAS_STATE(session, ref, previous_state, WT_REF_LOCKED)) - break; + /* If off-page, the pointer references a WT_ADDR structure. */ + if (__wt_off_page(page, addr)) { + copy->oldest_start_ts = addr->oldest_start_ts; + copy->oldest_start_txn = addr->oldest_start_txn; + copy->start_durable_ts = addr->start_durable_ts; + copy->newest_stop_ts = addr->newest_stop_ts; + copy->newest_stop_txn = addr->newest_stop_txn; + copy->stop_durable_ts = addr->stop_durable_ts; + copy->type = addr->type; + memcpy(copy->addr, addr->addr, copy->size = addr->size); + return (true); } - __wt_ref_info(session, ref, &addr, &size, &is_leaf); - - if (addr_buf != NULL) { - if (addr != NULL) - memcpy(addr_buf, addr, size); - *sizep = size; + /* If on-page, the pointer references a cell. */ + __wt_cell_unpack(session, page, (WT_CELL *)addr, unpack); + copy->oldest_start_ts = unpack->oldest_start_ts; + copy->oldest_start_txn = unpack->oldest_start_txn; + copy->start_durable_ts = unpack->newest_start_durable_ts; + copy->newest_stop_ts = unpack->newest_stop_ts; + copy->newest_stop_txn = unpack->newest_stop_txn; + copy->stop_durable_ts = unpack->newest_stop_durable_ts; + copy->type = 0; /* Avoid static analyzer uninitialized value complaints. */ + switch (unpack->raw) { + case WT_CELL_ADDR_INT: + copy->type = WT_ADDR_INT; + break; + case WT_CELL_ADDR_LEAF: + copy->type = WT_ADDR_LEAF; + break; + case WT_CELL_ADDR_LEAF_NO: + copy->type = WT_ADDR_LEAF_NO; + break; } - if (is_leafp != NULL) - *is_leafp = is_leaf; - - WT_REF_SET_STATE(ref, previous_state); + memcpy(copy->addr, unpack->data, copy->size = (uint8_t)unpack->size); + return (true); } /* @@ -1136,14 +1155,12 @@ __wt_ref_info_lock( static inline int __wt_ref_block_free(WT_SESSION_IMPL *session, WT_REF *ref) { - size_t addr_size; - const uint8_t *addr; + WT_ADDR_COPY addr; - if (ref->addr == NULL) + if (!__wt_ref_addr_copy(session, ref, &addr)) return (0); - __wt_ref_info(session, ref, &addr, &addr_size, NULL); - WT_RET(__wt_btree_block_free(session, addr, addr_size)); + WT_RET(__wt_btree_block_free(session, addr.addr, addr.size)); /* Clear the address (so we don't free it twice). */ __wt_ref_addr_free(session, ref); @@ -1172,25 +1189,6 @@ __wt_page_del_active(WT_SESSION_IMPL *session, WT_REF *ref, bool visible_all) } /* - * __wt_page_las_active -- - * Return if lookaside data for a page is still required. - */ -static inline bool -__wt_page_las_active(WT_SESSION_IMPL *session, WT_REF *ref) -{ - WT_PAGE_LOOKASIDE *page_las; - - if ((page_las = ref->page_las) == NULL) - return (false); - if (page_las->resolved) - return (false); - if (page_las->min_skipped_ts != WT_TS_MAX || page_las->has_prepares) - return (true); - - return (!__wt_txn_visible_all(session, page_las->max_txn, page_las->max_ondisk_ts)); -} - -/* * __wt_btree_can_evict_dirty -- * Check whether eviction of dirty pages or splits are permitted in the current tree. We cannot * evict dirty pages or split while a checkpoint is in progress, unless the checkpoint thread is @@ -1416,7 +1414,7 @@ __wt_page_can_evict(WT_SESSION_IMPL *session, WT_REF *ref, bool *inmem_splitp) * One special case where we know this is safe is if the handle is locked exclusive (e.g., when * the whole tree is being evicted). In that case, no readers can be looking at an old index. */ - if (WT_PAGE_IS_INTERNAL(page) && !F_ISSET(session->dhandle, WT_DHANDLE_EXCLUSIVE) && + if (F_ISSET(ref, WT_REF_FLAG_INTERNAL) && !F_ISSET(session->dhandle, WT_DHANDLE_EXCLUSIVE) && __wt_gen_active(session, WT_GEN_SPLIT, page->pg_intl_split_gen)) return (false); @@ -1699,3 +1697,33 @@ __wt_page_swap_func(WT_SESSION_IMPL *session, WT_REF *held, WT_REF *want, uint32 return (ret); } + +/* + * __wt_bt_col_var_cursor_walk_txn_read -- + * transactionally read the onpage value and the history store for col var cursor walk. + */ +static inline int +__wt_bt_col_var_cursor_walk_txn_read(WT_SESSION_IMPL *session, WT_CURSOR_BTREE *cbt, WT_PAGE *page, + WT_CELL_UNPACK *unpack, WT_COL *cip, WT_UPDATE **updp) +{ + WT_UPDATE *upd; + + upd = NULL; + *updp = NULL; + cbt->slot = WT_COL_SLOT(page, cip); + WT_RET(__wt_txn_read(session, cbt, NULL, unpack, &upd)); + if (upd == NULL) + return (0); + if (upd != NULL && upd->type == WT_UPDATE_TOMBSTONE) { + if (F_ISSET(upd, WT_UPDATE_RESTORED_FROM_DISK)) + __wt_free_update_list(session, &upd); + return (0); + } + + *updp = upd; + WT_RET(__wt_value_return(cbt, upd)); + cbt->tmp->data = cbt->iface.value.data; + cbt->tmp->size = cbt->iface.value.size; + cbt->cip_saved = cip; + return (0); +} diff --git a/src/third_party/wiredtiger/src/include/cache.h b/src/third_party/wiredtiger/src/include/cache.h index d3f2b3714c7..698cea9447c 100644 --- a/src/third_party/wiredtiger/src/include/cache.h +++ b/src/third_party/wiredtiger/src/include/cache.h @@ -54,10 +54,7 @@ typedef enum __wt_cache_op { WT_SYNC_WRITE_LEAVES } WT_CACHE_OP; -#define WT_LAS_FILE_MIN (100 * WT_MEGABYTE) -#define WT_LAS_NUM_SESSIONS 5 -#define WT_LAS_SWEEP_ENTRIES (20 * WT_THOUSAND) -#define WT_LAS_SWEEP_SEC 2 +#define WT_HS_FILE_MIN (100 * WT_MEGABYTE) /* * WiredTiger cache structure. @@ -83,7 +80,7 @@ struct __wt_cache { uint64_t bytes_read; /* Bytes read into memory */ uint64_t bytes_written; - uint64_t bytes_lookaside; /* Lookaside bytes inmem */ + uint64_t bytes_hs; /* History store bytes inmem */ volatile uint64_t eviction_progress; /* Eviction progress count */ uint64_t last_eviction_progress; /* Tracked eviction progress */ @@ -184,41 +181,16 @@ struct __wt_cache { * varies between 0, if reconciliation always sees updates that are globally visible and hence * can be discarded, to 100 if no updates are globally visible. */ - int32_t evict_lookaside_score; + int32_t evict_hs_score; - /* - * Shared lookaside lock, session and cursor, used by threads accessing the lookaside table - * (other than eviction server and worker threads and the sweep thread, all of which have their - * own lookaside cursors). - */ - WT_SPINLOCK las_lock; - WT_SESSION_IMPL *las_session[WT_LAS_NUM_SESSIONS]; - bool las_session_inuse[WT_LAS_NUM_SESSIONS]; - - uint32_t las_fileid; /* Lookaside table file ID */ - uint64_t las_insert_count; /* Count of inserts to lookaside */ - uint64_t las_remove_count; /* Count of removes from lookaside */ - uint64_t las_pageid; /* Lookaside table page ID counter */ - - bool las_reader; /* Indicate an LAS reader to sweep */ - WT_RWLOCK las_sweepwalk_lock; - WT_SPINLOCK las_sweep_lock; - WT_ITEM las_sweep_key; /* Track sweep position. */ - uint32_t las_sweep_dropmin; /* Minimum btree ID in current set. */ - uint8_t *las_sweep_dropmap; /* Bitmap of dropped btree IDs. */ - uint32_t las_sweep_dropmax; /* Maximum btree ID in current set. */ - uint64_t las_sweep_max_pageid; /* Maximum page ID for sweep. */ - - uint32_t *las_dropped; /* List of dropped btree IDs. */ - size_t las_dropped_next; /* Next index into drop list. */ - size_t las_dropped_alloc; /* Allocated size of drop list. */ + uint32_t hs_fileid; /* History store table file ID */ /* - * The "lookaside_activity" verbose messages are throttled to once per checkpoint. To accomplish + * The "history_activity" verbose messages are throttled to once per checkpoint. To accomplish * this we track the checkpoint generation for the most recent read and write verbose messages. */ - uint64_t las_verb_gen_read; - uint64_t las_verb_gen_write; + uint64_t hs_verb_gen_read; + uint64_t hs_verb_gen_write; /* * Cache pool information. @@ -249,7 +221,7 @@ struct __wt_cache { #define WT_CACHE_EVICT_DEBUG_MODE 0x004u /* Aggressive debugging mode */ #define WT_CACHE_EVICT_DIRTY 0x008u /* Evict dirty pages */ #define WT_CACHE_EVICT_DIRTY_HARD 0x010u /* Dirty % blocking app threads */ -#define WT_CACHE_EVICT_LOOKASIDE 0x020u /* Try lookaside eviction */ +#define WT_CACHE_EVICT_HS 0x020u /* Try history store eviction */ #define WT_CACHE_EVICT_NOKEEP 0x040u /* Don't add read pages to cache */ #define WT_CACHE_EVICT_SCRUB 0x080u /* Scrub dirty pages */ #define WT_CACHE_EVICT_URGENT 0x100u /* Pages are in the urgent queue */ @@ -288,6 +260,12 @@ struct __wt_cache_pool { uint8_t flags; }; +/* + * Optimize comparisons against the history store URI, flag handles that reference the history store + * file. + */ +#define WT_IS_HS(btree) F_ISSET(btree, WT_BTREE_HS) + /* Flags used with __wt_evict */ /* AUTOMATIC FLAG VALUE GENERATION START */ #define WT_EVICT_CALL_CLOSING 0x1u /* Closing connection or tree */ diff --git a/src/third_party/wiredtiger/src/include/cache.i b/src/third_party/wiredtiger/src/include/cache.i index aae98cdb58b..b96f079f5bd 100644 --- a/src/third_party/wiredtiger/src/include/cache.i +++ b/src/third_party/wiredtiger/src/include/cache.i @@ -183,25 +183,24 @@ __wt_cache_bytes_other(WT_CACHE *cache) } /* - * __wt_cache_lookaside_score -- - * Get the current lookaside score (between 0 and 100). + * __wt_cache_hs_score -- + * Get the current history store score (between 0 and 100). */ static inline uint32_t -__wt_cache_lookaside_score(WT_CACHE *cache) +__wt_cache_hs_score(WT_CACHE *cache) { int32_t global_score; - global_score = cache->evict_lookaside_score; + global_score = cache->evict_hs_score; return ((uint32_t)WT_MIN(WT_MAX(global_score, 0), 100)); } /* - * __wt_cache_update_lookaside_score -- - * Update the lookaside score based how many unstable updates are seen. + * __wt_cache_update_hs_score -- + * Update the history store score based how many unstable updates are seen. */ static inline void -__wt_cache_update_lookaside_score( - WT_SESSION_IMPL *session, u_int updates_seen, u_int updates_unstable) +__wt_cache_update_hs_score(WT_SESSION_IMPL *session, u_int updates_seen, u_int updates_unstable) { WT_CACHE *cache; int32_t global_score, score; @@ -211,12 +210,12 @@ __wt_cache_update_lookaside_score( cache = S2C(session)->cache; score = (int32_t)((100 * updates_unstable) / updates_seen); - global_score = cache->evict_lookaside_score; + global_score = cache->evict_hs_score; if (score > global_score && global_score < 100) - (void)__wt_atomic_addi32(&cache->evict_lookaside_score, 1); + (void)__wt_atomic_addi32(&cache->evict_hs_score, 1); else if (score < global_score && global_score > 0) - (void)__wt_atomic_subi32(&cache->evict_lookaside_score, 1); + (void)__wt_atomic_subi32(&cache->evict_hs_score, 1); } /* diff --git a/src/third_party/wiredtiger/src/include/cell.h b/src/third_party/wiredtiger/src/include/cell.h index 1973f31931c..b80449a8c18 100644 --- a/src/third_party/wiredtiger/src/include/cell.h +++ b/src/third_party/wiredtiger/src/include/cell.h @@ -61,8 +61,9 @@ * * Bit 4 marks a value with an additional descriptor byte. If this flag is set, * the next byte after the initial cell byte is an additional description byte. - * The bottom 4 bits describe a validity window of timestamp/transaction IDs. - * The top 4 bits are currently unused. + * The bottom bit in this additional byte indicates that the cell is part of a + * prepared, and not yet committed transaction. The next 6 bits describe a validity + * and durability window of timestamp/transaction IDs. The top bit is currently unused. * * Bits 5-8 are cell "types". */ @@ -77,11 +78,13 @@ #define WT_CELL_64V 0x04 /* Associated value */ #define WT_CELL_SECOND_DESC 0x08 /* Second descriptor byte */ -#define WT_CELL_TS_DURABLE 0x01 /* Newest-durable timestamp */ -#define WT_CELL_TS_START 0x02 /* Oldest-start timestamp */ -#define WT_CELL_TS_STOP 0x04 /* Newest-stop timestamp */ -#define WT_CELL_TXN_START 0x08 /* Oldest-start txn ID */ -#define WT_CELL_TXN_STOP 0x10 /* Newest-stop txn ID */ +#define WT_CELL_PREPARE 0x01 /* Part of prepared transaction */ +#define WT_CELL_TS_DURABLE_START 0x02 /* Start durable timestamp */ +#define WT_CELL_TS_DURABLE_STOP 0x04 /* Stop durable timestamp */ +#define WT_CELL_TS_START 0x08 /* Oldest-start timestamp */ +#define WT_CELL_TS_STOP 0x10 /* Newest-stop timestamp */ +#define WT_CELL_TXN_START 0x20 /* Oldest-start txn ID */ +#define WT_CELL_TXN_STOP 0x40 /* Newest-stop txn ID */ /* * WT_CELL_ADDR_INT is an internal block location, WT_CELL_ADDR_LEAF is a leaf block location, and @@ -125,11 +128,11 @@ */ struct __wt_cell { /* - * Maximum of 62 bytes: + * Maximum of 71 bytes: * 1: cell descriptor byte * 1: prefix compression count * 1: secondary descriptor byte - * 27: 3 timestamps (uint64_t encoding, max 9 bytes) + * 36: 4 timestamps (uint64_t encoding, max 9 bytes) * 18: 2 transaction IDs (uint64_t encoding, max 9 bytes) * 9: associated 64-bit value (uint64_t encoding, max 9 bytes) * 5: data length (uint32_t encoding, max 5 bytes) @@ -138,7 +141,7 @@ struct __wt_cell { * count and 64V value overlap, and the validity window, 64V value * and data length are all optional in some cases. */ - uint8_t __chunk[1 + 1 + 1 + 6 * WT_INTPACK64_MAXSIZE + WT_INTPACK32_MAXSIZE]; + uint8_t __chunk[1 + 1 + 1 + 7 * WT_INTPACK64_MAXSIZE + WT_INTPACK32_MAXSIZE]; }; /* @@ -150,17 +153,21 @@ struct __wt_cell_unpack { uint64_t v; /* RLE count or recno */ - wt_timestamp_t start_ts; /* Value validity window */ - uint64_t start_txn; - wt_timestamp_t stop_ts; - uint64_t stop_txn; + /* Value validity window */ + wt_timestamp_t start_ts; /* default value: WT_TS_NONE */ + uint64_t start_txn; /* default value: WT_TXN_NONE */ + wt_timestamp_t durable_start_ts; /* default value: WT_TS_NONE */ + wt_timestamp_t stop_ts; /* default value: WT_TS_MAX */ + uint64_t stop_txn; /* default value: WT_TXN_MAX */ + wt_timestamp_t durable_stop_ts; /* default value: WT_TS_NONE */ /* Address validity window */ - wt_timestamp_t newest_durable_ts; - wt_timestamp_t oldest_start_ts; - uint64_t oldest_start_txn; - wt_timestamp_t newest_stop_ts; - uint64_t newest_stop_txn; + wt_timestamp_t oldest_start_ts; /* default value: WT_TS_NONE */ + uint64_t oldest_start_txn; /* default value: WT_TXN_NONE */ + wt_timestamp_t newest_start_durable_ts; /* default value: WT_TS_NONE */ + wt_timestamp_t newest_stop_ts; /* default value: WT_TS_MAX */ + uint64_t newest_stop_txn; /* default value: WT_TXN_MAX */ + wt_timestamp_t newest_stop_durable_ts; /* default value: WT_TS_NONE */ /* * !!! @@ -177,5 +184,10 @@ struct __wt_cell_unpack { uint8_t raw; /* Raw cell type (include "shorts") */ uint8_t type; /* Cell type */ - uint8_t ovfl; /* boolean: cell is an overflow */ +/* AUTOMATIC FLAG VALUE GENERATION START */ +#define WT_CELL_UNPACK_OVERFLOW 0x1u /* cell is an overflow */ +#define WT_CELL_UNPACK_PREPARE 0x2u /* cell is part of a prepared transaction */ +#define WT_CELL_UNPACK_TIME_PAIRS_CLEARED 0x4u /* time pairs are cleared because of restart */ + /* AUTOMATIC FLAG VALUE GENERATION STOP */ + uint8_t flags; }; diff --git a/src/third_party/wiredtiger/src/include/cell.i b/src/third_party/wiredtiger/src/include/cell.i index da313396859..b3eb91efc78 100644 --- a/src/third_party/wiredtiger/src/include/cell.i +++ b/src/third_party/wiredtiger/src/include/cell.i @@ -11,13 +11,21 @@ * Check the value's validity window for sanity. */ static inline void -__cell_check_value_validity(WT_SESSION_IMPL *session, wt_timestamp_t start_ts, uint64_t start_txn, +__cell_check_value_validity(WT_SESSION_IMPL *session, wt_timestamp_t durable_start_ts, + wt_timestamp_t durable_stop_ts, wt_timestamp_t start_ts, uint64_t start_txn, wt_timestamp_t stop_ts, uint64_t stop_txn) { #ifdef HAVE_DIAGNOSTIC char ts_string[2][WT_TS_INT_STRING_SIZE]; - if (stop_ts == WT_TS_NONE) { + if (durable_start_ts > durable_stop_ts) { + __wt_errx(session, "a durable start timestamp %s newer than its durable stop timestamp %s", + __wt_timestamp_to_string(durable_start_ts, ts_string[0]), + __wt_timestamp_to_string(durable_stop_ts, ts_string[1])); + WT_ASSERT(session, durable_start_ts <= durable_stop_ts); + } + + if (start_ts != WT_TS_NONE && stop_ts == WT_TS_NONE) { __wt_errx(session, "stop timestamp of 0"); WT_ASSERT(session, stop_ts != WT_TS_NONE); } @@ -28,10 +36,6 @@ __cell_check_value_validity(WT_SESSION_IMPL *session, wt_timestamp_t start_ts, u WT_ASSERT(session, start_ts <= stop_ts); } - if (stop_txn == WT_TXN_NONE) { - __wt_errx(session, "stop transaction ID of 0"); - WT_ASSERT(session, stop_txn != WT_TXN_NONE); - } if (start_txn > stop_txn) { __wt_errx(session, "a start transaction ID %" PRIu64 " newer than its stop " @@ -41,6 +45,8 @@ __cell_check_value_validity(WT_SESSION_IMPL *session, wt_timestamp_t start_ts, u } #else WT_UNUSED(session); + WT_UNUSED(durable_start_ts); + WT_UNUSED(durable_stop_ts); WT_UNUSED(start_ts); WT_UNUSED(start_txn); WT_UNUSED(stop_ts); @@ -53,19 +59,18 @@ __cell_check_value_validity(WT_SESSION_IMPL *session, wt_timestamp_t start_ts, u * Pack the validity window for a value. */ static inline void -__cell_pack_value_validity(WT_SESSION_IMPL *session, uint8_t **pp, wt_timestamp_t start_ts, - uint64_t start_txn, wt_timestamp_t stop_ts, uint64_t stop_txn) +__cell_pack_value_validity(WT_SESSION_IMPL *session, uint8_t **pp, wt_timestamp_t durable_start_ts, + wt_timestamp_t durable_stop_ts, wt_timestamp_t start_ts, uint64_t start_txn, + wt_timestamp_t stop_ts, uint64_t stop_txn, bool prepare) { uint8_t flags, *flagsp; - __cell_check_value_validity(session, start_ts, start_txn, stop_ts, stop_txn); + __cell_check_value_validity( + session, durable_start_ts, durable_stop_ts, start_ts, start_txn, stop_ts, stop_txn); - /* - * Historic page versions and globally visible values have no associated validity window, else - * set a flag bit and store them. - */ - if (!__wt_process.page_version_ts || (start_ts == WT_TS_NONE && start_txn == WT_TXN_NONE && - stop_ts == WT_TS_MAX && stop_txn == WT_TXN_MAX)) + /* Globally visible values have no associated validity window, set a flag bit and store them. */ + if (start_ts == WT_TS_NONE && start_txn == WT_TXN_NONE && stop_ts == WT_TS_MAX && + stop_txn == WT_TXN_MAX) ++*pp; else { **pp |= WT_CELL_SECOND_DESC; @@ -82,6 +87,12 @@ __cell_pack_value_validity(WT_SESSION_IMPL *session, uint8_t **pp, wt_timestamp_ WT_IGNORE_RET(__wt_vpack_uint(pp, 0, start_txn)); LF_SET(WT_CELL_TXN_START); } + if (durable_start_ts != WT_TS_NONE) { + /* Store differences, not absolutes. */ + WT_ASSERT(session, start_ts != WT_TS_NONE && start_ts <= durable_start_ts); + WT_IGNORE_RET(__wt_vpack_uint(pp, 0, durable_start_ts - start_ts)); + LF_SET(WT_CELL_TS_DURABLE_START); + } if (stop_ts != WT_TS_MAX) { /* Store differences, not absolutes. */ WT_IGNORE_RET(__wt_vpack_uint(pp, 0, stop_ts - start_ts)); @@ -92,6 +103,14 @@ __cell_pack_value_validity(WT_SESSION_IMPL *session, uint8_t **pp, wt_timestamp_ WT_IGNORE_RET(__wt_vpack_uint(pp, 0, stop_txn - start_txn)); LF_SET(WT_CELL_TXN_STOP); } + if (durable_stop_ts != WT_TS_NONE) { + /* Store differences, not absolutes. */ + WT_ASSERT(session, stop_ts != WT_TS_MAX && stop_ts <= durable_stop_ts); + WT_IGNORE_RET(__wt_vpack_uint(pp, 0, durable_stop_ts - stop_ts)); + LF_SET(WT_CELL_TS_DURABLE_STOP); + } + if (prepare) + LF_SET(WT_CELL_PREPARE); *flagsp = flags; } } @@ -104,10 +123,11 @@ static inline void __wt_check_addr_validity(WT_SESSION_IMPL *session, wt_timestamp_t oldest_start_ts, uint64_t oldest_start_txn, wt_timestamp_t newest_stop_ts, uint64_t newest_stop_txn) { +/* FIXME-prepare-support: accept durable timestamps as args, and do checks on them. */ #ifdef HAVE_DIAGNOSTIC char ts_string[2][WT_TS_INT_STRING_SIZE]; - if (newest_stop_ts == WT_TS_NONE) { + if (oldest_start_ts != WT_TS_NONE && newest_stop_ts == WT_TS_NONE) { __wt_errx(session, "newest stop timestamp of 0"); WT_ASSERT(session, newest_stop_ts != WT_TS_NONE); } @@ -119,10 +139,6 @@ __wt_check_addr_validity(WT_SESSION_IMPL *session, wt_timestamp_t oldest_start_t __wt_timestamp_to_string(newest_stop_ts, ts_string[1])); WT_ASSERT(session, oldest_start_ts <= newest_stop_ts); } - if (newest_stop_txn == WT_TXN_NONE) { - __wt_errx(session, "newest stop transaction of 0"); - WT_ASSERT(session, newest_stop_txn != WT_TXN_NONE); - } if (oldest_start_txn > newest_stop_txn) { __wt_errx(session, "an oldest start transaction %" PRIu64 " newer than its " @@ -144,23 +160,20 @@ __wt_check_addr_validity(WT_SESSION_IMPL *session, wt_timestamp_t oldest_start_t * Pack the validity window for an address. */ static inline void -__cell_pack_addr_validity(WT_SESSION_IMPL *session, uint8_t **pp, wt_timestamp_t newest_durable_ts, - wt_timestamp_t oldest_start_ts, uint64_t oldest_start_txn, wt_timestamp_t newest_stop_ts, - uint64_t newest_stop_txn) +__cell_pack_addr_validity(WT_SESSION_IMPL *session, uint8_t **pp, wt_timestamp_t start_durable_ts, + wt_timestamp_t stop_durable_ts, wt_timestamp_t oldest_start_ts, uint64_t oldest_start_txn, + wt_timestamp_t newest_stop_ts, uint64_t newest_stop_txn) { uint8_t flags, *flagsp; + /* FIXME-prepare-support: Check validity of durable timestamps. */ __wt_check_addr_validity( session, oldest_start_ts, oldest_start_txn, newest_stop_ts, newest_stop_txn); - /* - * Historic page versions and globally visible values have no associated validity window, else - * set a flag bit and store them. - */ - if (!__wt_process.page_version_ts || - (newest_durable_ts == WT_TS_NONE && oldest_start_ts == WT_TS_NONE && - oldest_start_txn == WT_TXN_NONE && newest_stop_ts == WT_TS_MAX && - newest_stop_txn == WT_TXN_MAX)) + /* Globally visible values have no associated validity window, set a flag bit and store them. */ + if (start_durable_ts == WT_TS_NONE && stop_durable_ts == WT_TS_NONE && + oldest_start_ts == WT_TS_NONE && oldest_start_txn == WT_TXN_NONE && + newest_stop_ts == WT_TS_MAX && newest_stop_txn == WT_TXN_MAX) ++*pp; else { **pp |= WT_CELL_SECOND_DESC; @@ -169,10 +182,6 @@ __cell_pack_addr_validity(WT_SESSION_IMPL *session, uint8_t **pp, wt_timestamp_t ++*pp; flags = 0; - if (newest_durable_ts != WT_TS_NONE) { - WT_IGNORE_RET(__wt_vpack_uint(pp, 0, newest_durable_ts)); - LF_SET(WT_CELL_TS_DURABLE); - } if (oldest_start_ts != WT_TS_NONE) { WT_IGNORE_RET(__wt_vpack_uint(pp, 0, oldest_start_ts)); LF_SET(WT_CELL_TS_START); @@ -181,6 +190,13 @@ __cell_pack_addr_validity(WT_SESSION_IMPL *session, uint8_t **pp, wt_timestamp_t WT_IGNORE_RET(__wt_vpack_uint(pp, 0, oldest_start_txn)); LF_SET(WT_CELL_TXN_START); } + if (start_durable_ts != WT_TS_NONE) { + /* Store differences, not absolutes. */ + WT_ASSERT( + session, oldest_start_ts != WT_TS_NONE && oldest_start_ts <= start_durable_ts); + WT_IGNORE_RET(__wt_vpack_uint(pp, 0, start_durable_ts - oldest_start_ts)); + LF_SET(WT_CELL_TS_DURABLE_START); + } if (newest_stop_ts != WT_TS_MAX) { /* Store differences, not absolutes. */ WT_IGNORE_RET(__wt_vpack_uint(pp, 0, newest_stop_ts - oldest_start_ts)); @@ -191,6 +207,16 @@ __cell_pack_addr_validity(WT_SESSION_IMPL *session, uint8_t **pp, wt_timestamp_t WT_IGNORE_RET(__wt_vpack_uint(pp, 0, newest_stop_txn - oldest_start_txn)); LF_SET(WT_CELL_TXN_STOP); } + if (stop_durable_ts != WT_TS_NONE) { + /* Store differences, not absolutes. */ + /* + * FIXME-prepare-support: + * WT_ASSERT(session, + * newest_stop_ts != WT_TS_MAX && newest_stop_ts <= stop_durable__ts); + */ + WT_IGNORE_RET(__wt_vpack_uint(pp, 0, stop_durable_ts - newest_stop_ts)); + LF_SET(WT_CELL_TS_DURABLE_STOP); + } *flagsp = flags; } } @@ -201,17 +227,24 @@ __cell_pack_addr_validity(WT_SESSION_IMPL *session, uint8_t **pp, wt_timestamp_t */ static inline size_t __wt_cell_pack_addr(WT_SESSION_IMPL *session, WT_CELL *cell, u_int cell_type, uint64_t recno, - wt_timestamp_t newest_durable_ts, wt_timestamp_t oldest_start_ts, uint64_t oldest_start_txn, + wt_timestamp_t stop_durable_ts, wt_timestamp_t oldest_start_ts, uint64_t oldest_start_txn, wt_timestamp_t newest_stop_ts, uint64_t newest_stop_txn, size_t size) { + wt_timestamp_t start_durable_ts; uint8_t *p; + /* + * FIXME-prepare-support: This value should be passed in when support for prepared transactions + * with durable history is fully implemented. + */ + start_durable_ts = WT_TS_NONE; + /* Start building a cell: the descriptor byte starts zero. */ p = cell->__chunk; *p = '\0'; - __cell_pack_addr_validity(session, &p, newest_durable_ts, oldest_start_ts, oldest_start_txn, - newest_stop_ts, newest_stop_txn); + __cell_pack_addr_validity(session, &p, start_durable_ts, stop_durable_ts, oldest_start_ts, + oldest_start_txn, newest_stop_ts, newest_stop_txn); if (recno == WT_RECNO_OOB) cell->__chunk[0] |= (uint8_t)cell_type; /* Type */ @@ -233,14 +266,24 @@ static inline size_t __wt_cell_pack_value(WT_SESSION_IMPL *session, WT_CELL *cell, wt_timestamp_t start_ts, uint64_t start_txn, wt_timestamp_t stop_ts, uint64_t stop_txn, uint64_t rle, size_t size) { + wt_timestamp_t durable_start_ts, durable_stop_ts; uint8_t byte, *p; - bool validity; + bool prepare, validity; + + /* + * FIXME-prepare-support: These values should be passed in when support for prepared + * transactions with durable history is fully implemented. + */ + durable_start_ts = WT_TS_NONE; + durable_stop_ts = WT_TS_NONE; + prepare = false; /* Start building a cell: the descriptor byte starts zero. */ p = cell->__chunk; *p = '\0'; - __cell_pack_value_validity(session, &p, start_ts, start_txn, stop_ts, stop_txn); + __cell_pack_value_validity(session, &p, durable_start_ts, durable_stop_ts, start_ts, start_txn, + stop_ts, stop_txn, prepare); /* * Short data cells without a validity window or run-length encoding have 6 bits of data length @@ -304,6 +347,10 @@ __wt_cell_pack_value_match( if (validity) { /* Skip validity window */ flags = *a; ++a; + if (LF_ISSET(WT_CELL_TS_DURABLE_START)) + WT_RET(__wt_vunpack_uint(&a, 0, &v)); + if (LF_ISSET(WT_CELL_TS_DURABLE_STOP)) + WT_RET(__wt_vunpack_uint(&a, 0, &v)); if (LF_ISSET(WT_CELL_TS_START)) WT_RET(__wt_vunpack_uint(&a, 0, &v)); if (LF_ISSET(WT_CELL_TS_STOP)) @@ -329,6 +376,10 @@ __wt_cell_pack_value_match( if (validity) { /* Skip validity window */ flags = *b; ++b; + if (LF_ISSET(WT_CELL_TS_DURABLE_START)) + WT_RET(__wt_vunpack_uint(&b, 0, &v)); + if (LF_ISSET(WT_CELL_TS_DURABLE_STOP)) + WT_RET(__wt_vunpack_uint(&b, 0, &v)); if (LF_ISSET(WT_CELL_TS_START)) WT_RET(__wt_vunpack_uint(&b, 0, &v)); if (LF_ISSET(WT_CELL_TS_STOP)) @@ -357,13 +408,24 @@ static inline size_t __wt_cell_pack_copy(WT_SESSION_IMPL *session, WT_CELL *cell, wt_timestamp_t start_ts, uint64_t start_txn, wt_timestamp_t stop_ts, uint64_t stop_txn, uint64_t rle, uint64_t v) { + wt_timestamp_t durable_start_ts, durable_stop_ts; uint8_t *p; + bool prepare; + + /* + * FIXME-prepare-support: These values should be passed in when support for prepared + * transactions with durable history is fully implemented. + */ + durable_start_ts = WT_TS_NONE; + durable_stop_ts = WT_TS_NONE; + prepare = false; /* Start building a cell: the descriptor byte starts zero. */ p = cell->__chunk; *p = '\0'; - __cell_pack_value_validity(session, &p, start_ts, start_txn, stop_ts, stop_txn); + __cell_pack_value_validity(session, &p, durable_start_ts, durable_stop_ts, start_ts, start_txn, + stop_ts, stop_txn, prepare); if (rle < 2) cell->__chunk[0] |= WT_CELL_VALUE_COPY; /* Type */ @@ -392,7 +454,9 @@ __wt_cell_pack_del(WT_SESSION_IMPL *session, WT_CELL *cell, wt_timestamp_t start p = cell->__chunk; *p = '\0'; - __cell_pack_value_validity(session, &p, start_ts, start_txn, stop_ts, stop_txn); + /* FIXME-prepare-support: we should pass durable start and stop values. */ + __cell_pack_value_validity( + session, &p, WT_TS_NONE, WT_TS_NONE, start_ts, start_txn, stop_ts, stop_txn, false); if (rle < 2) cell->__chunk[0] |= WT_CELL_DEL; /* Type */ @@ -481,7 +545,14 @@ static inline size_t __wt_cell_pack_ovfl(WT_SESSION_IMPL *session, WT_CELL *cell, uint8_t type, wt_timestamp_t start_ts, uint64_t start_txn, wt_timestamp_t stop_ts, uint64_t stop_txn, uint64_t rle, size_t size) { + wt_timestamp_t durable_start_ts, durable_stop_ts; uint8_t *p; + bool prepare; + + /* FIXME-prepare-support: The durable timestamps should be passed in. */ + durable_start_ts = WT_TS_NONE; + durable_stop_ts = WT_TS_NONE; + prepare = false; /* Start building a cell: the descriptor byte starts zero. */ p = cell->__chunk; @@ -494,7 +565,8 @@ __wt_cell_pack_ovfl(WT_SESSION_IMPL *session, WT_CELL *cell, uint8_t type, wt_ti break; case WT_CELL_VALUE_OVFL: case WT_CELL_VALUE_OVFL_RM: - __cell_pack_value_validity(session, &p, start_ts, start_txn, stop_ts, stop_txn); + __cell_pack_value_validity(session, &p, durable_start_ts, durable_stop_ts, start_ts, + start_txn, stop_ts, stop_txn, prepare); break; } @@ -696,18 +768,21 @@ restart: * following switch. All validity windows default to durability. */ unpack->v = 0; + unpack->durable_start_ts = WT_TS_NONE; + unpack->durable_stop_ts = WT_TS_NONE; unpack->start_ts = WT_TS_NONE; unpack->start_txn = WT_TXN_NONE; unpack->stop_ts = WT_TS_MAX; unpack->stop_txn = WT_TXN_MAX; - unpack->newest_durable_ts = WT_TS_NONE; + unpack->newest_start_durable_ts = WT_TS_NONE; + unpack->newest_stop_durable_ts = WT_TS_NONE; unpack->oldest_start_ts = WT_TS_NONE; unpack->oldest_start_txn = WT_TXN_NONE; unpack->newest_stop_ts = WT_TS_MAX; unpack->newest_stop_txn = WT_TXN_MAX; unpack->raw = (uint8_t)__wt_cell_type_raw(cell); unpack->type = (uint8_t)__wt_cell_type(cell); - unpack->ovfl = 0; + unpack->flags = 0; /* * Handle cells with none of RLE counts, validity window or data length: short key/data cells @@ -756,15 +831,19 @@ restart: break; flags = *p++; /* skip second descriptor byte */ - if (LF_ISSET(WT_CELL_TS_DURABLE)) - WT_RET(__wt_vunpack_uint( - &p, end == NULL ? 0 : WT_PTRDIFF(end, p), &unpack->newest_durable_ts)); + if (LF_ISSET(WT_CELL_PREPARE)) + F_SET(unpack, WT_CELL_UNPACK_PREPARE); if (LF_ISSET(WT_CELL_TS_START)) WT_RET(__wt_vunpack_uint( &p, end == NULL ? 0 : WT_PTRDIFF(end, p), &unpack->oldest_start_ts)); if (LF_ISSET(WT_CELL_TXN_START)) WT_RET(__wt_vunpack_uint( &p, end == NULL ? 0 : WT_PTRDIFF(end, p), &unpack->oldest_start_txn)); + if (LF_ISSET(WT_CELL_TS_DURABLE_START)) { + WT_RET(__wt_vunpack_uint( + &p, end == NULL ? 0 : WT_PTRDIFF(end, p), &unpack->newest_start_durable_ts)); + unpack->newest_start_durable_ts += unpack->oldest_start_ts; + } if (LF_ISSET(WT_CELL_TS_STOP)) { WT_RET( __wt_vunpack_uint(&p, end == NULL ? 0 : WT_PTRDIFF(end, p), &unpack->newest_stop_ts)); @@ -775,6 +854,13 @@ restart: &p, end == NULL ? 0 : WT_PTRDIFF(end, p), &unpack->newest_stop_txn)); unpack->newest_stop_txn += unpack->oldest_start_txn; } + if (LF_ISSET(WT_CELL_TS_DURABLE_STOP)) { + WT_RET(__wt_vunpack_uint( + &p, end == NULL ? 0 : WT_PTRDIFF(end, p), &unpack->newest_stop_durable_ts)); + unpack->newest_stop_durable_ts += unpack->newest_stop_ts; + } + + /* FIXME-prepare-support: Check validity of durable timestamps. */ __wt_check_addr_validity(session, unpack->oldest_start_ts, unpack->oldest_start_txn, unpack->newest_stop_ts, unpack->newest_stop_txn); break; @@ -787,10 +873,17 @@ restart: break; flags = *p++; /* skip second descriptor byte */ + if (LF_ISSET(WT_CELL_PREPARE)) + F_SET(unpack, WT_CELL_UNPACK_PREPARE); if (LF_ISSET(WT_CELL_TS_START)) WT_RET(__wt_vunpack_uint(&p, end == NULL ? 0 : WT_PTRDIFF(end, p), &unpack->start_ts)); if (LF_ISSET(WT_CELL_TXN_START)) WT_RET(__wt_vunpack_uint(&p, end == NULL ? 0 : WT_PTRDIFF(end, p), &unpack->start_txn)); + if (LF_ISSET(WT_CELL_TS_DURABLE_START)) { + WT_RET(__wt_vunpack_uint( + &p, end == NULL ? 0 : WT_PTRDIFF(end, p), &unpack->durable_start_ts)); + unpack->durable_start_ts += unpack->start_ts; + } if (LF_ISSET(WT_CELL_TS_STOP)) { WT_RET(__wt_vunpack_uint(&p, end == NULL ? 0 : WT_PTRDIFF(end, p), &unpack->stop_ts)); unpack->stop_ts += unpack->start_ts; @@ -799,8 +892,13 @@ restart: WT_RET(__wt_vunpack_uint(&p, end == NULL ? 0 : WT_PTRDIFF(end, p), &unpack->stop_txn)); unpack->stop_txn += unpack->start_txn; } - __cell_check_value_validity( - session, unpack->start_ts, unpack->start_txn, unpack->stop_ts, unpack->stop_txn); + if (LF_ISSET(WT_CELL_TS_DURABLE_STOP)) { + WT_RET(__wt_vunpack_uint( + &p, end == NULL ? 0 : WT_PTRDIFF(end, p), &unpack->durable_stop_ts)); + unpack->durable_stop_ts += unpack->stop_ts; + } + __cell_check_value_validity(session, unpack->durable_start_ts, unpack->durable_stop_ts, + unpack->start_ts, unpack->start_txn, unpack->stop_ts, unpack->stop_txn); break; } @@ -839,7 +937,7 @@ restart: /* * Set overflow flag. */ - unpack->ovfl = 1; + F_SET(unpack, WT_CELL_UNPACK_OVERFLOW); /* FALLTHROUGH */ case WT_CELL_ADDR_DEL: @@ -912,11 +1010,14 @@ __wt_cell_unpack_dsk( * If there isn't any value validity window (which is what it will take to get to a * zero-length item), the value must be stable. */ + unpack->durable_start_ts = WT_TS_NONE; + unpack->durable_stop_ts = WT_TS_NONE; unpack->start_ts = WT_TS_NONE; unpack->start_txn = WT_TXN_NONE; unpack->stop_ts = WT_TS_MAX; unpack->stop_txn = WT_TXN_MAX; - unpack->newest_durable_ts = WT_TS_NONE; + unpack->newest_start_durable_ts = WT_TS_NONE; + unpack->newest_stop_durable_ts = WT_TS_NONE; unpack->oldest_start_ts = WT_TS_NONE; unpack->oldest_start_txn = WT_TXN_NONE; unpack->newest_stop_ts = WT_TS_MAX; @@ -926,11 +1027,53 @@ __wt_cell_unpack_dsk( unpack->__len = 0; unpack->prefix = 0; unpack->raw = unpack->type = WT_CELL_VALUE; - unpack->ovfl = 0; + unpack->flags = 0; return; } WT_IGNORE_RET(__wt_cell_unpack_safe(session, dsk, cell, unpack, NULL)); + + /* + * If the page came from a previous run, reset the transaction ids to "none" and timestamps to 0 + * as appropriate. Transaction ids shouldn't persist between runs so these are always set to + * "none". Timestamps should persist between runs however, the absence of a timestamp (in the + * case of a non-timestamped write) should default to WT_TS_NONE rather than "max" as usual. + * + * Note that it is still necessary to unpack each value above even if we end up overwriting them + * since values in a cell need to be unpacked sequentially. + * + * This is how the stop time pair should be interpreted for each type of delete: + * - + * Timestamp delete Non-timestamp delete No delete + * Current startup txnid=x, ts=y txnid=x, ts=WT_TS_NONE txnid=MAX, ts=MAX + * Previous startup txnid=0, ts=y txnid=0, ts=WT_TS_NONE txnid=MAX, ts=MAX + */ + if (dsk->write_gen > 0 && dsk->write_gen <= S2C(session)->base_write_gen) { + /* FIXME-prepare-support: deal with durable timestamps. */ + /* Tell reconciliation we cleared the transaction ids and the cell needs to be rebuilt. */ + if (unpack->start_txn != WT_TXN_NONE) { + unpack->start_txn = WT_TXN_NONE; + F_SET(unpack, WT_CELL_UNPACK_TIME_PAIRS_CLEARED); + } + if (unpack->stop_txn != WT_TXN_MAX) { + unpack->stop_txn = WT_TXN_NONE; + F_SET(unpack, WT_CELL_UNPACK_TIME_PAIRS_CLEARED); + if (unpack->stop_ts == WT_TS_MAX) + unpack->stop_ts = WT_TS_NONE; + } else + WT_ASSERT(session, unpack->stop_ts == WT_TS_MAX); + if (unpack->oldest_start_txn != WT_TXN_NONE) { + unpack->oldest_start_txn = WT_TXN_NONE; + F_SET(unpack, WT_CELL_UNPACK_TIME_PAIRS_CLEARED); + } + if (unpack->newest_stop_txn != WT_TXN_MAX) { + unpack->newest_stop_txn = WT_TXN_NONE; + F_SET(unpack, WT_CELL_UNPACK_TIME_PAIRS_CLEARED); + if (unpack->newest_stop_ts == WT_TS_MAX) + unpack->newest_stop_ts = WT_TS_NONE; + } else + WT_ASSERT(session, unpack->newest_stop_ts == WT_TS_MAX); + } } /* diff --git a/src/third_party/wiredtiger/src/include/config.h b/src/third_party/wiredtiger/src/include/config.h index 2ef90837754..0be38097dba 100644 --- a/src/third_party/wiredtiger/src/include/config.h +++ b/src/third_party/wiredtiger/src/include/config.h @@ -89,23 +89,22 @@ struct __wt_config_parser_impl { #define WT_CONFIG_ENTRY_WT_SESSION_reset 35 #define WT_CONFIG_ENTRY_WT_SESSION_rollback_transaction 36 #define WT_CONFIG_ENTRY_WT_SESSION_salvage 37 -#define WT_CONFIG_ENTRY_WT_SESSION_snapshot 38 -#define WT_CONFIG_ENTRY_WT_SESSION_strerror 39 -#define WT_CONFIG_ENTRY_WT_SESSION_timestamp_transaction 40 -#define WT_CONFIG_ENTRY_WT_SESSION_transaction_sync 41 -#define WT_CONFIG_ENTRY_WT_SESSION_truncate 42 -#define WT_CONFIG_ENTRY_WT_SESSION_upgrade 43 -#define WT_CONFIG_ENTRY_WT_SESSION_verify 44 -#define WT_CONFIG_ENTRY_colgroup_meta 45 -#define WT_CONFIG_ENTRY_file_config 46 -#define WT_CONFIG_ENTRY_file_meta 47 -#define WT_CONFIG_ENTRY_index_meta 48 -#define WT_CONFIG_ENTRY_lsm_meta 49 -#define WT_CONFIG_ENTRY_table_meta 50 -#define WT_CONFIG_ENTRY_wiredtiger_open 51 -#define WT_CONFIG_ENTRY_wiredtiger_open_all 52 -#define WT_CONFIG_ENTRY_wiredtiger_open_basecfg 53 -#define WT_CONFIG_ENTRY_wiredtiger_open_usercfg 54 +#define WT_CONFIG_ENTRY_WT_SESSION_strerror 38 +#define WT_CONFIG_ENTRY_WT_SESSION_timestamp_transaction 39 +#define WT_CONFIG_ENTRY_WT_SESSION_transaction_sync 40 +#define WT_CONFIG_ENTRY_WT_SESSION_truncate 41 +#define WT_CONFIG_ENTRY_WT_SESSION_upgrade 42 +#define WT_CONFIG_ENTRY_WT_SESSION_verify 43 +#define WT_CONFIG_ENTRY_colgroup_meta 44 +#define WT_CONFIG_ENTRY_file_config 45 +#define WT_CONFIG_ENTRY_file_meta 46 +#define WT_CONFIG_ENTRY_index_meta 47 +#define WT_CONFIG_ENTRY_lsm_meta 48 +#define WT_CONFIG_ENTRY_table_meta 49 +#define WT_CONFIG_ENTRY_wiredtiger_open 50 +#define WT_CONFIG_ENTRY_wiredtiger_open_all 51 +#define WT_CONFIG_ENTRY_wiredtiger_open_basecfg 52 +#define WT_CONFIG_ENTRY_wiredtiger_open_usercfg 53 /* * configuration section: END * DO NOT EDIT: automatically built by dist/flags.py. diff --git a/src/third_party/wiredtiger/src/include/connection.h b/src/third_party/wiredtiger/src/include/connection.h index 54d1213173a..285f760019e 100644 --- a/src/third_party/wiredtiger/src/include/connection.h +++ b/src/third_party/wiredtiger/src/include/connection.h @@ -284,12 +284,18 @@ struct __wt_connection_impl { bool ckpt_signalled; /* Checkpoint signalled */ uint64_t ckpt_usecs; /* Checkpoint timer */ + uint64_t ckpt_prep_max; /* Checkpoint prepare time min/max */ + uint64_t ckpt_prep_min; + uint64_t ckpt_prep_recent; /* Checkpoint prepare time recent/total */ + uint64_t ckpt_prep_total; uint64_t ckpt_time_max; /* Checkpoint time min/max */ uint64_t ckpt_time_min; uint64_t ckpt_time_recent; /* Checkpoint time recent/total */ uint64_t ckpt_time_total; /* Checkpoint stats and verbosity timers */ + struct timespec ckpt_prep_end; + struct timespec ckpt_prep_start; struct timespec ckpt_timer_start; struct timespec ckpt_timer_scrub_end; @@ -302,8 +308,7 @@ struct __wt_connection_impl { uint64_t incr_granularity; WT_BLKINCR incr_backups[WT_BLKINCR_MAX]; - /* Connection's maximum and base write generations. */ - uint64_t max_write_gen; + /* Connection's base write generation. */ uint64_t base_write_gen; uint32_t stat_flags; /* Options declared in flags.py */ @@ -399,8 +404,6 @@ struct __wt_connection_impl { uint64_t sweep_interval; /* Handle sweep interval */ uint64_t sweep_handles_min; /* Handle sweep minimum open */ - /* Set of btree IDs not being rolled back */ - uint8_t *stable_rollback_bitstring; uint32_t stable_rollback_maxfile; /* Locked: collator list */ @@ -444,42 +447,44 @@ struct __wt_connection_impl { int page_size; /* OS page size for mmap alignment */ /* AUTOMATIC FLAG VALUE GENERATION START */ -#define WT_VERB_API 0x000000001u -#define WT_VERB_BACKUP 0x000000002u -#define WT_VERB_BLOCK 0x000000004u -#define WT_VERB_CHECKPOINT 0x000000008u -#define WT_VERB_CHECKPOINT_PROGRESS 0x000000010u -#define WT_VERB_COMPACT 0x000000020u -#define WT_VERB_COMPACT_PROGRESS 0x000000040u -#define WT_VERB_ERROR_RETURNS 0x000000080u -#define WT_VERB_EVICT 0x000000100u -#define WT_VERB_EVICTSERVER 0x000000200u -#define WT_VERB_EVICT_STUCK 0x000000400u -#define WT_VERB_FILEOPS 0x000000800u -#define WT_VERB_HANDLEOPS 0x000001000u -#define WT_VERB_LOG 0x000002000u -#define WT_VERB_LOOKASIDE 0x000004000u -#define WT_VERB_LOOKASIDE_ACTIVITY 0x000008000u -#define WT_VERB_LSM 0x000010000u -#define WT_VERB_LSM_MANAGER 0x000020000u -#define WT_VERB_METADATA 0x000040000u -#define WT_VERB_MUTEX 0x000080000u -#define WT_VERB_OVERFLOW 0x000100000u -#define WT_VERB_READ 0x000200000u -#define WT_VERB_REBALANCE 0x000400000u -#define WT_VERB_RECONCILE 0x000800000u -#define WT_VERB_RECOVERY 0x001000000u -#define WT_VERB_RECOVERY_PROGRESS 0x002000000u -#define WT_VERB_SALVAGE 0x004000000u -#define WT_VERB_SHARED_CACHE 0x008000000u -#define WT_VERB_SPLIT 0x010000000u -#define WT_VERB_TEMPORARY 0x020000000u -#define WT_VERB_THREAD_GROUP 0x040000000u -#define WT_VERB_TIMESTAMP 0x080000000u -#define WT_VERB_TRANSACTION 0x100000000u -#define WT_VERB_VERIFY 0x200000000u -#define WT_VERB_VERSION 0x400000000u -#define WT_VERB_WRITE 0x800000000u +#define WT_VERB_API 0x0000000001u +#define WT_VERB_BACKUP 0x0000000002u +#define WT_VERB_BLOCK 0x0000000004u +#define WT_VERB_CHECKPOINT 0x0000000008u +#define WT_VERB_CHECKPOINT_GC 0x0000000010u +#define WT_VERB_CHECKPOINT_PROGRESS 0x0000000020u +#define WT_VERB_COMPACT 0x0000000040u +#define WT_VERB_COMPACT_PROGRESS 0x0000000080u +#define WT_VERB_ERROR_RETURNS 0x0000000100u +#define WT_VERB_EVICT 0x0000000200u +#define WT_VERB_EVICTSERVER 0x0000000400u +#define WT_VERB_EVICT_STUCK 0x0000000800u +#define WT_VERB_FILEOPS 0x0000001000u +#define WT_VERB_HANDLEOPS 0x0000002000u +#define WT_VERB_HS 0x0000004000u +#define WT_VERB_HS_ACTIVITY 0x0000008000u +#define WT_VERB_LOG 0x0000010000u +#define WT_VERB_LSM 0x0000020000u +#define WT_VERB_LSM_MANAGER 0x0000040000u +#define WT_VERB_METADATA 0x0000080000u +#define WT_VERB_MUTEX 0x0000100000u +#define WT_VERB_OVERFLOW 0x0000200000u +#define WT_VERB_READ 0x0000400000u +#define WT_VERB_REBALANCE 0x0000800000u +#define WT_VERB_RECONCILE 0x0001000000u +#define WT_VERB_RECOVERY 0x0002000000u +#define WT_VERB_RECOVERY_PROGRESS 0x0004000000u +#define WT_VERB_RTS 0x0008000000u +#define WT_VERB_SALVAGE 0x0010000000u +#define WT_VERB_SHARED_CACHE 0x0020000000u +#define WT_VERB_SPLIT 0x0040000000u +#define WT_VERB_TEMPORARY 0x0080000000u +#define WT_VERB_THREAD_GROUP 0x0100000000u +#define WT_VERB_TIMESTAMP 0x0200000000u +#define WT_VERB_TRANSACTION 0x0400000000u +#define WT_VERB_VERIFY 0x0800000000u +#define WT_VERB_VERSION 0x1000000000u +#define WT_VERB_WRITE 0x2000000000u /* AUTOMATIC FLAG VALUE GENERATION STOP */ uint64_t verbose; @@ -489,7 +494,7 @@ struct __wt_connection_impl { /* AUTOMATIC FLAG VALUE GENERATION START */ #define WT_TIMING_STRESS_AGGRESSIVE_SWEEP 0x001u #define WT_TIMING_STRESS_CHECKPOINT_SLOW 0x002u -#define WT_TIMING_STRESS_LOOKASIDE_SWEEP 0x004u +#define WT_TIMING_STRESS_HS_SWEEP 0x004u #define WT_TIMING_STRESS_SPLIT_1 0x008u #define WT_TIMING_STRESS_SPLIT_2 0x010u #define WT_TIMING_STRESS_SPLIT_3 0x020u @@ -522,27 +527,26 @@ struct __wt_connection_impl { #define WT_CONN_DEBUG_CURSOR_COPY 0x00000100u #define WT_CONN_DEBUG_REALLOC_EXACT 0x00000200u #define WT_CONN_DEBUG_SLOW_CKPT 0x00000400u -#define WT_CONN_EVICTION_NO_LOOKASIDE 0x00000800u -#define WT_CONN_EVICTION_RUN 0x00001000u +#define WT_CONN_EVICTION_RUN 0x00000800u +#define WT_CONN_HS_OPEN 0x00001000u #define WT_CONN_INCR_BACKUP 0x00002000u #define WT_CONN_IN_MEMORY 0x00004000u #define WT_CONN_LEAK_MEMORY 0x00008000u -#define WT_CONN_LOOKASIDE_OPEN 0x00010000u -#define WT_CONN_LSM_MERGE 0x00020000u -#define WT_CONN_OPTRACK 0x00040000u -#define WT_CONN_PANIC 0x00080000u -#define WT_CONN_READONLY 0x00100000u -#define WT_CONN_RECONFIGURING 0x00200000u -#define WT_CONN_RECOVERING 0x00400000u -#define WT_CONN_SALVAGE 0x00800000u -#define WT_CONN_SERVER_ASYNC 0x01000000u -#define WT_CONN_SERVER_CAPACITY 0x02000000u -#define WT_CONN_SERVER_CHECKPOINT 0x04000000u -#define WT_CONN_SERVER_LOG 0x08000000u -#define WT_CONN_SERVER_LSM 0x10000000u -#define WT_CONN_SERVER_STATISTICS 0x20000000u -#define WT_CONN_SERVER_SWEEP 0x40000000u -#define WT_CONN_WAS_BACKUP 0x80000000u +#define WT_CONN_LSM_MERGE 0x00010000u +#define WT_CONN_OPTRACK 0x00020000u +#define WT_CONN_PANIC 0x00040000u +#define WT_CONN_READONLY 0x00080000u +#define WT_CONN_RECONFIGURING 0x00100000u +#define WT_CONN_RECOVERING 0x00200000u +#define WT_CONN_SALVAGE 0x00400000u +#define WT_CONN_SERVER_ASYNC 0x00800000u +#define WT_CONN_SERVER_CAPACITY 0x01000000u +#define WT_CONN_SERVER_CHECKPOINT 0x02000000u +#define WT_CONN_SERVER_LOG 0x04000000u +#define WT_CONN_SERVER_LSM 0x08000000u +#define WT_CONN_SERVER_STATISTICS 0x10000000u +#define WT_CONN_SERVER_SWEEP 0x20000000u +#define WT_CONN_WAS_BACKUP 0x40000000u /* AUTOMATIC FLAG VALUE GENERATION STOP */ uint32_t flags; }; diff --git a/src/third_party/wiredtiger/src/include/cursor.i b/src/third_party/wiredtiger/src/include/cursor.i index ab5f4b64141..5c698f604c3 100644 --- a/src/third_party/wiredtiger/src/include/cursor.i +++ b/src/third_party/wiredtiger/src/include/cursor.i @@ -313,6 +313,38 @@ __wt_cursor_dhandle_decr_use(WT_SESSION_IMPL *session) } /* + * __wt_cursor_disable_bulk -- + * Disable bulk loads into a tree. + */ +static inline void +__wt_cursor_disable_bulk(WT_SESSION_IMPL *session) +{ + WT_BTREE *btree; + + btree = S2BT(session); + + /* + * Once a tree (other than the LSM primary) is no longer empty, eviction should pay attention to + * it, and it's no longer possible to bulk-load into it. + */ + if (!btree->original) + return; + if (btree->lsm_primary) { + btree->original = 0; /* Make the next test faster. */ + return; + } + + /* + * We use a compare-and-swap here to avoid races among the first inserts into a tree. Eviction + * is disabled when an empty tree is opened, and it must only be enabled once. + */ + if (__wt_atomic_cas8(&btree->original, 1, 0)) { + btree->evict_disabled_open = false; + __wt_evict_file_exclusive_off(session); + } +} + +/* * __cursor_kv_return -- * Return a page referenced key/value pair to the application. */ @@ -368,29 +400,27 @@ __cursor_func_init(WT_CURSOR_BTREE *cbt, bool reenter) } /* - * __cursor_row_slot_return -- - * Return a row-store leaf page slot's K/V pair. + * __cursor_row_slot_key_return -- + * Return a row-store leaf page slot's key. */ static inline int -__cursor_row_slot_return(WT_CURSOR_BTREE *cbt, WT_ROW *rip, WT_UPDATE *upd) +__cursor_row_slot_key_return( + WT_CURSOR_BTREE *cbt, WT_ROW *rip, WT_CELL_UNPACK *kpack, bool *kpack_used) { WT_BTREE *btree; WT_CELL *cell; - WT_CELL_UNPACK *kpack, _kpack, *vpack, _vpack; - WT_ITEM *kb, *vb; + WT_ITEM *kb; WT_PAGE *page; WT_SESSION_IMPL *session; void *copy; + *kpack_used = false; + session = (WT_SESSION_IMPL *)cbt->iface.session; btree = S2BT(session); page = cbt->ref->page; - kpack = NULL; - vpack = &_vpack; - kb = &cbt->iface.key; - vb = &cbt->iface.value; /* * The row-store key can change underfoot; explicitly take a copy. @@ -405,7 +435,7 @@ __cursor_row_slot_return(WT_CURSOR_BTREE *cbt, WT_ROW *rip, WT_UPDATE *upd) * First, check for an immediately available key. */ if (__wt_row_leaf_key_info(page, copy, NULL, &cell, &kb->data, &kb->size)) - goto value; + return (0); /* Huffman encoded keys are a slow path in all cases. */ if (btree->huffman_key != NULL) @@ -419,9 +449,9 @@ __cursor_row_slot_return(WT_CURSOR_BTREE *cbt, WT_ROW *rip, WT_UPDATE *upd) * do it in lots of other places), but disabling shared builds (--disable-shared) results in the * compiler complaining about uninitialized field use. */ - kpack = &_kpack; memset(kpack, 0, sizeof(*kpack)); __wt_cell_unpack(session, page, cell, kpack); + *kpack_used = true; if (kpack->type == WT_CELL_KEY && cbt->rip_saved != NULL && cbt->rip_saved == rip - 1) { WT_ASSERT(session, cbt->row_key->size >= kpack->prefix); @@ -447,21 +477,5 @@ slow: kb->data = cbt->row_key->data; kb->size = cbt->row_key->size; cbt->rip_saved = rip; - -value: - /* - * If the item was ever modified, use the WT_UPDATE data. Note the - * caller passes us the update: it has already resolved which one - * (if any) is visible. - */ - if (upd != NULL) - return (__wt_value_return(cbt, upd)); - - /* Else, simple values have their location encoded in the WT_ROW. */ - if (__wt_row_leaf_value(page, rip, vb)) - return (0); - - /* Else, take the value from the original page cell. */ - __wt_row_leaf_value_cell(session, page, rip, kpack, vpack); - return (__wt_page_cell_data_ref(session, cbt->ref->page, vpack, vb)); + return (0); } diff --git a/src/third_party/wiredtiger/src/include/extern.h b/src/third_party/wiredtiger/src/include/extern.h index 4ad93180572..940ab258eb8 100644 --- a/src/third_party/wiredtiger/src/include/extern.h +++ b/src/third_party/wiredtiger/src/include/extern.h @@ -26,12 +26,6 @@ extern bool __wt_handle_is_open(WT_SESSION_IMPL *session, const char *name) extern bool __wt_hazard_check_assert(WT_SESSION_IMPL *session, void *ref, bool waitfor) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern bool __wt_ispo2(uint32_t v) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); -extern bool __wt_las_empty(WT_SESSION_IMPL *session) - WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); -extern bool __wt_las_page_skip(WT_SESSION_IMPL *session, WT_REF *ref) - WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); -extern bool __wt_las_page_skip_locked(WT_SESSION_IMPL *session, WT_REF *ref) - WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern bool __wt_lsm_chunk_visible_all(WT_SESSION_IMPL *session, WT_LSM_CHUNK *chunk) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern bool __wt_modify_idempotent(const void *modify) @@ -40,6 +34,8 @@ extern bool __wt_page_evict_urgent(WT_SESSION_IMPL *session, WT_REF *ref) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern bool __wt_rwlock_islocked(WT_SESSION_IMPL *session, WT_RWLOCK *l) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); +extern char *__wt_time_pair_to_string(wt_timestamp_t timestamp, uint64_t txn_id, char *tp_string) + WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern char *__wt_timestamp_to_string(wt_timestamp_t ts, char *ts_string) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern const WT_CONFIG_ENTRY *__wt_conn_config_match(const char *method) @@ -58,6 +54,8 @@ extern const char *__wt_ext_strerror(WT_EXTENSION_API *wt_api, WT_SESSION *wt_se WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern const char *__wt_json_tokname(int toktype) WT_GCC_FUNC_DECL_ATTRIBUTE( (visibility("default"))) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); +extern const char *__wt_key_string(WT_SESSION_IMPL *session, const void *data_arg, size_t size, + const char *key_format, WT_ITEM *buf) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern const char *__wt_page_type_string(u_int type) WT_GCC_FUNC_DECL_ATTRIBUTE( (visibility("default"))) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern const char *__wt_session_strerror(WT_SESSION *wt_session, int error) @@ -293,7 +291,7 @@ extern int __wt_btree_discard(WT_SESSION_IMPL *session) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_btree_huffman_open(WT_SESSION_IMPL *session) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); -extern int __wt_btree_new_leaf_page(WT_SESSION_IMPL *session, WT_PAGE **pagep) +extern int __wt_btree_new_leaf_page(WT_SESSION_IMPL *session, WT_REF *ref) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_btree_open(WT_SESSION_IMPL *session, const char *op_cfg[]) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); @@ -334,6 +332,9 @@ extern int __wt_cache_eviction_worker(WT_SESSION_IMPL *session, bool busy, bool double pct_full) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_cache_pool_config(WT_SESSION_IMPL *session, const char **cfg) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); +extern int __wt_calc_modify(WT_SESSION_IMPL *wt_session, const WT_ITEM *oldv, const WT_ITEM *newv, + size_t maxdiff, WT_MODIFY *entries, int *nentriesp) + WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_calloc(WT_SESSION_IMPL *session, size_t number, size_t size, void *retp) WT_GCC_FUNC_DECL_ATTRIBUTE((visibility("default"))) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); @@ -468,7 +469,6 @@ extern int __wt_connection_workers(WT_SESSION_IMPL *session, const char *cfg[]) extern int __wt_copy_and_sync(WT_SESSION *wt_session, const char *from, const char *to) WT_GCC_FUNC_DECL_ATTRIBUTE((visibility("default"))) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); -extern int __wt_count_birthmarks(WT_UPDATE *upd) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_curbackup_open(WT_SESSION_IMPL *session, const char *uri, WT_CURSOR *other, const char *cfg[], WT_CURSOR **cursorp) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_curbackup_open_incr(WT_SESSION_IMPL *session, const char *uri, WT_CURSOR *other, @@ -583,12 +583,17 @@ extern int __wt_debug_addr(WT_SESSION_IMPL *session, const uint8_t *addr, size_t const char *ofile) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_debug_addr_print(WT_SESSION_IMPL *session, const uint8_t *addr, size_t addr_size) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); -extern int __wt_debug_cursor_las(void *cursor_arg, const char *ofile) WT_GCC_FUNC_DECL_ATTRIBUTE( - (visibility("default"))) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); +extern int __wt_debug_cursor_hs(WT_SESSION_IMPL *session, WT_CURSOR *hs_cursor) + WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_debug_cursor_page(void *cursor_arg, const char *ofile) WT_GCC_FUNC_DECL_ATTRIBUTE( (visibility("default"))) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); +extern int __wt_debug_cursor_tree_hs(void *cursor_arg, const char *ofile) + WT_GCC_FUNC_DECL_ATTRIBUTE((visibility("default"))) + WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_debug_disk(WT_SESSION_IMPL *session, const WT_PAGE_HEADER *dsk, const char *ofile) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); +extern int __wt_debug_key_value(WT_SESSION_IMPL *session, WT_ITEM *key, WT_CELL_UNPACK *value) + WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_debug_mode_config(WT_SESSION_IMPL *session, const char *cfg[]) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_debug_offset(WT_SESSION_IMPL *session, wt_off_t offset, uint32_t size, @@ -625,8 +630,8 @@ extern int __wt_encryptor_config(WT_SESSION_IMPL *session, WT_CONFIG_ITEM *cval, extern int __wt_errno(void) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_esc_hex_to_raw(WT_SESSION_IMPL *session, const char *from, WT_ITEM *to) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); -extern int __wt_evict(WT_SESSION_IMPL *session, WT_REF *ref, uint32_t previous_state, - uint32_t flags) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); +extern int __wt_evict(WT_SESSION_IMPL *session, WT_REF *ref, uint8_t previous_state, uint32_t flags) + WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_evict_create(WT_SESSION_IMPL *session) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_evict_destroy(WT_SESSION_IMPL *session) @@ -715,6 +720,8 @@ extern int __wt_filename(WT_SESSION_IMPL *session, const char *name, char **path extern int __wt_filename_construct(WT_SESSION_IMPL *session, const char *path, const char *file_prefix, uintmax_t id_1, uint32_t id_2, WT_ITEM *buf) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); +extern int __wt_find_hs_upd(WT_SESSION_IMPL *session, WT_CURSOR_BTREE *cbt, WT_UPDATE **updp, + bool allow_prepare, WT_ITEM *on_disk_buf) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_fopen(WT_SESSION_IMPL *session, const char *name, uint32_t open_flags, uint32_t flags, WT_FSTREAM **fstrp) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_fsync_background(WT_SESSION_IMPL *session) @@ -724,7 +731,7 @@ extern int __wt_getopt(const char *progname, int nargc, char *const *nargv, cons WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_hazard_clear(WT_SESSION_IMPL *session, WT_REF *ref) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); -extern int __wt_hazard_set(WT_SESSION_IMPL *session, WT_REF *ref, bool *busyp +extern int __wt_hazard_set_func(WT_SESSION_IMPL *session, WT_REF *ref, bool *busyp #ifdef HAVE_DIAGNOSTIC , const char *func, int line @@ -734,6 +741,26 @@ extern int __wt_hex2byte(const u_char *from, u_char *to) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_hex_to_raw(WT_SESSION_IMPL *session, const char *from, WT_ITEM *to) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); +extern int __wt_hs_config(WT_SESSION_IMPL *session, const char **cfg) + WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); +extern int __wt_hs_create(WT_SESSION_IMPL *session, const char **cfg) + WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); +extern int __wt_hs_cursor(WT_SESSION_IMPL *session, uint32_t *session_flags, bool *is_owner) + WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); +extern int __wt_hs_cursor_close(WT_SESSION_IMPL *session, uint32_t session_flags, bool is_owner) + WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); +extern int __wt_hs_cursor_open(WT_SESSION_IMPL *session) + WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); +extern int __wt_hs_cursor_position(WT_SESSION_IMPL *session, WT_CURSOR *cursor, uint32_t btree_id, + WT_ITEM *key, wt_timestamp_t timestamp) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); +extern int __wt_hs_delete_key(WT_SESSION_IMPL *session, uint32_t btree_id, const WT_ITEM *key) + WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); +extern int __wt_hs_get_btree(WT_SESSION_IMPL *session, WT_BTREE **hs_btreep) + WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); +extern int __wt_hs_insert_updates(WT_CURSOR *cursor, WT_BTREE *btree, WT_PAGE *page, + WT_MULTI *multi) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); +extern int __wt_hs_modify(WT_CURSOR_BTREE *hs_cbt, WT_UPDATE *hs_upd) + WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_huffman_decode(WT_SESSION_IMPL *session, void *huffman_arg, const uint8_t *from_arg, size_t from_len, WT_ITEM *to_buf) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_huffman_encode(WT_SESSION_IMPL *session, void *huffman_arg, const uint8_t *from_arg, @@ -762,26 +789,6 @@ extern int __wt_json_token(WT_SESSION *wt_session, const char *src, int *toktype const char **tokstart, size_t *toklen) WT_GCC_FUNC_DECL_ATTRIBUTE((visibility("default"))) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_key_return(WT_CURSOR_BTREE *cbt) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); -extern int __wt_las_config(WT_SESSION_IMPL *session, const char **cfg) - WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); -extern int __wt_las_create(WT_SESSION_IMPL *session, const char **cfg) - WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); -extern int __wt_las_cursor_close(WT_SESSION_IMPL *session, WT_CURSOR **cursorp, - uint32_t session_flags) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); -extern int __wt_las_cursor_open(WT_SESSION_IMPL *session) - WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); -extern int __wt_las_cursor_position(WT_CURSOR *cursor, uint64_t pageid) - WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); -extern int __wt_las_destroy(WT_SESSION_IMPL *session) - WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); -extern int __wt_las_insert_block(WT_CURSOR *cursor, WT_BTREE *btree, WT_PAGE *page, WT_MULTI *multi, - WT_ITEM *key) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); -extern int __wt_las_remove_block(WT_SESSION_IMPL *session, uint64_t pageid) - WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); -extern int __wt_las_save_dropped(WT_SESSION_IMPL *session) - WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); -extern int __wt_las_sweep(WT_SESSION_IMPL *session) - WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_library_init(void) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_log_acquire(WT_SESSION_IMPL *session, uint64_t recsize, WT_LOGSLOT *slot) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); @@ -1045,6 +1052,8 @@ extern int __wt_meta_track_sub_off(WT_SESSION_IMPL *session) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_meta_track_update(WT_SESSION_IMPL *session, const char *key) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); +extern int __wt_metadata_btree_id_to_uri(WT_SESSION_IMPL *session, uint32_t btree_id, char **uri) + WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_metadata_cursor(WT_SESSION_IMPL *session, WT_CURSOR **cursorp) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_metadata_cursor_open(WT_SESSION_IMPL *session, const char *config, @@ -1054,6 +1063,8 @@ extern int __wt_metadata_cursor_release(WT_SESSION_IMPL *session, WT_CURSOR **cu extern int __wt_metadata_get_ckptlist(WT_SESSION *session, const char *name, WT_CKPT **ckptbasep) WT_GCC_FUNC_DECL_ATTRIBUTE((visibility("default"))) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); +extern int __wt_metadata_init_base_write_gen(WT_SESSION_IMPL *session) + WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_metadata_insert(WT_SESSION_IMPL *session, const char *key, const char *value) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_metadata_remove(WT_SESSION_IMPL *session, const char *key) @@ -1062,25 +1073,31 @@ extern int __wt_metadata_salvage(WT_SESSION_IMPL *session) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_metadata_search(WT_SESSION_IMPL *session, const char *key, char **valuep) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); -extern int __wt_metadata_set_base_write_gen(WT_SESSION_IMPL *session) - WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_metadata_turtle_rewrite(WT_SESSION_IMPL *session) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_metadata_update(WT_SESSION_IMPL *session, const char *key, const char *value) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); +extern int __wt_metadata_update_base_write_gen(WT_SESSION_IMPL *session, const char *config) + WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); +extern int __wt_metadata_uri_to_btree_id(WT_SESSION_IMPL *session, const char *uri, + uint32_t *btree_id) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_modify_apply(WT_CURSOR *cursor, const void *modify) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_modify_apply_api(WT_CURSOR *cursor, WT_MODIFY *entries, int nentries) WT_GCC_FUNC_DECL_ATTRIBUTE((visibility("default"))) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); -extern int __wt_modify_pack(WT_CURSOR *cursor, WT_ITEM **modifyp, WT_MODIFY *entries, int nentries) +extern int __wt_modify_apply_item(WT_SESSION_IMPL *session, WT_ITEM *value, const void *modify, + bool sformat) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); +extern int __wt_modify_pack(WT_CURSOR *cursor, WT_MODIFY *entries, int nentries, WT_ITEM **modifyp) + WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); +extern int __wt_modify_vector_push(WT_MODIFY_VECTOR *modifies, WT_UPDATE *upd) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_msg(WT_SESSION_IMPL *session, const char *fmt, ...) WT_GCC_FUNC_DECL_ATTRIBUTE((cold)) WT_GCC_FUNC_DECL_ATTRIBUTE((format(printf, 2, 3))) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_multi_to_ref(WT_SESSION_IMPL *session, WT_PAGE *page, WT_MULTI *multi, WT_REF **refp, size_t *incrp, bool closing) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); -extern int __wt_name_check(WT_SESSION_IMPL *session, const char *str, size_t len) +extern int __wt_name_check(WT_SESSION_IMPL *session, const char *str, size_t len, bool check_uri) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_nfilename(WT_SESSION_IMPL *session, const char *name, size_t namelen, char **path) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); @@ -1129,7 +1146,7 @@ extern int __wt_page_in_func(WT_SESSION_IMPL *session, WT_REF *ref, uint32_t fla #endif ) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_page_inmem(WT_SESSION_IMPL *session, WT_REF *ref, const void *image, uint32_t flags, - bool check_unstable, WT_PAGE **pagep) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); + WT_PAGE **pagep) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_page_modify_alloc(WT_SESSION_IMPL *session, WT_PAGE *page) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_page_release_evict(WT_SESSION_IMPL *session, WT_REF *ref, uint32_t flags) @@ -1186,9 +1203,11 @@ extern int __wt_rec_upd_select(WT_SESSION_IMPL *session, WT_RECONCILE *r, WT_INS void *ripcip, WT_CELL_UNPACK *vpack, WT_UPDATE_SELECT *upd_select) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_reconcile(WT_SESSION_IMPL *session, WT_REF *ref, WT_SALVAGE_COOKIE *salvage, - uint32_t flags, bool *lookaside_retryp) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); + uint32_t flags) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_remove_if_exists(WT_SESSION_IMPL *session, const char *name, bool durable) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); +extern int __wt_rollback_to_stable(WT_SESSION_IMPL *session, const char *cfg[], bool no_ckpt) + WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_row_ikey(WT_SESSION_IMPL *session, uint32_t cell_offset, const void *key, size_t size, WT_REF *ref) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_row_ikey_alloc(WT_SESSION_IMPL *session, uint32_t cell_offset, const void *key, @@ -1443,14 +1462,6 @@ extern int __wt_txn_log_commit(WT_SESSION_IMPL *session, const char *cfg[]) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_txn_log_op(WT_SESSION_IMPL *session, WT_CURSOR_BTREE *cbt) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); -extern int __wt_txn_named_snapshot_begin(WT_SESSION_IMPL *session, const char *cfg[]) - WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); -extern int __wt_txn_named_snapshot_config(WT_SESSION_IMPL *session, const char *cfg[], - bool *has_create, bool *has_drops) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); -extern int __wt_txn_named_snapshot_drop(WT_SESSION_IMPL *session, const char *cfg[]) - WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); -extern int __wt_txn_named_snapshot_get(WT_SESSION_IMPL *session, WT_CONFIG_ITEM *nameval) - WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_txn_op_printlog(WT_SESSION_IMPL *session, const uint8_t **pp, const uint8_t *end, WT_TXN_PRINTLOG_ARGS *args) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_txn_parse_timestamp(WT_SESSION_IMPL *session, const char *name, @@ -1472,8 +1483,6 @@ extern int __wt_txn_rollback(WT_SESSION_IMPL *session, const char *cfg[]) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_txn_rollback_required(WT_SESSION_IMPL *session, const char *reason) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); -extern int __wt_txn_rollback_to_stable(WT_SESSION_IMPL *session, const char *cfg[]) - WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_txn_set_commit_timestamp(WT_SESSION_IMPL *session, wt_timestamp_t commit_ts) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_txn_set_durable_timestamp(WT_SESSION_IMPL *session, wt_timestamp_t durable_ts) @@ -1501,6 +1510,8 @@ extern int __wt_upgrade(WT_SESSION_IMPL *session, const char *cfg[]) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_value_return(WT_CURSOR_BTREE *cbt, WT_UPDATE *upd) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); +extern int __wt_value_return_buf(WT_CURSOR_BTREE *cbt, WT_REF *ref, WT_ITEM *buf, + WT_TIME_PAIR *start, WT_TIME_PAIR *stop) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_value_return_upd(WT_CURSOR_BTREE *cbt, WT_UPDATE *upd) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_verbose_config(WT_SESSION_IMPL *session, const char *cfg[]) @@ -1530,6 +1541,8 @@ extern int __wt_verify_dsk(WT_SESSION_IMPL *session, const char *tag, WT_ITEM *b extern int __wt_verify_dsk_image(WT_SESSION_IMPL *session, const char *tag, const WT_PAGE_HEADER *dsk, size_t size, WT_ADDR *addr, bool empty_page_ok) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); +extern int __wt_verify_history_store_tree(WT_SESSION_IMPL *session, const char *uri) + WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int64_t __wt_log_slot_release(WT_MYSLOT *myslot, int64_t size) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern size_t __wt_json_unpack_char(u_char ch, u_char *buf, size_t bufsz, bool force_unicode) @@ -1584,7 +1597,6 @@ extern void __wt_btcur_init(WT_SESSION_IMPL *session, WT_CURSOR_BTREE *cbt); extern void __wt_btcur_iterate_setup(WT_CURSOR_BTREE *cbt); extern void __wt_btcur_open(WT_CURSOR_BTREE *cbt); extern void __wt_btree_huffman_close(WT_SESSION_IMPL *session); -extern void __wt_btree_page_version_config(WT_SESSION_IMPL *session); extern void __wt_cache_stats_update(WT_SESSION_IMPL *session); extern void __wt_capacity_throttle(WT_SESSION_IMPL *session, uint64_t bytes, WT_THROTTLE_TYPE type); extern void __wt_checkpoint_progress(WT_SESSION_IMPL *session, bool closing); @@ -1644,16 +1656,14 @@ extern void __wt_free_int(WT_SESSION_IMPL *session, const void *p_arg) extern void __wt_free_ref(WT_SESSION_IMPL *session, WT_REF *ref, int page_type, bool free_pages); extern void __wt_free_ref_index( WT_SESSION_IMPL *session, WT_PAGE *page, WT_PAGE_INDEX *pindex, bool free_pages); -extern void __wt_free_update_list(WT_SESSION_IMPL *session, WT_UPDATE *upd); +extern void __wt_free_update_list(WT_SESSION_IMPL *session, WT_UPDATE **updp); extern void __wt_gen_drain(WT_SESSION_IMPL *session, int which, uint64_t generation); extern void __wt_gen_init(WT_SESSION_IMPL *session); extern void __wt_gen_next_drain(WT_SESSION_IMPL *session, int which); extern void __wt_hazard_close(WT_SESSION_IMPL *session); +extern void __wt_hs_destroy(WT_SESSION_IMPL *session); extern void __wt_huffman_close(WT_SESSION_IMPL *session, void *huffman_arg); extern void __wt_json_close(WT_SESSION_IMPL *session, WT_CURSOR *cursor); -extern void __wt_las_cursor(WT_SESSION_IMPL *session, WT_CURSOR **cursorp, uint32_t *session_flags); -extern void __wt_las_remove_dropped(WT_SESSION_IMPL *session); -extern void __wt_las_stats_update(WT_SESSION_IMPL *session); extern void __wt_log_background(WT_SESSION_IMPL *session, WT_LSN *lsn); extern void __wt_log_ckpt(WT_SESSION_IMPL *session, WT_LSN *ckpt_lsn); extern void __wt_log_slot_activate(WT_SESSION_IMPL *session, WT_LOGSLOT *slot); @@ -1678,6 +1688,9 @@ extern void __wt_meta_track_discard(WT_SESSION_IMPL *session); extern void __wt_meta_track_sub_on(WT_SESSION_IMPL *session); extern void __wt_metadata_free_ckptlist(WT_SESSION *session, WT_CKPT *ckptbase) WT_GCC_FUNC_DECL_ATTRIBUTE((visibility("default"))); +extern void __wt_modify_vector_free(WT_MODIFY_VECTOR *modifies); +extern void __wt_modify_vector_init(WT_SESSION_IMPL *session, WT_MODIFY_VECTOR *modifies); +extern void __wt_modify_vector_pop(WT_MODIFY_VECTOR *modifies, WT_UPDATE **updp); extern void __wt_optrack_flush_buffer(WT_SESSION_IMPL *s); extern void __wt_optrack_record_funcid( WT_SESSION_IMPL *session, const char *func, uint16_t *func_idp); @@ -1691,10 +1704,17 @@ extern void __wt_random_init(WT_RAND_STATE volatile *rnd_state) WT_GCC_FUNC_DECL_ATTRIBUTE((visibility("default"))); extern void __wt_random_init_seed(WT_SESSION_IMPL *session, WT_RAND_STATE volatile *rnd_state) WT_GCC_FUNC_DECL_ATTRIBUTE((visibility("default"))); +extern void __wt_read_cell_time_pairs( + WT_CURSOR_BTREE *cbt, WT_REF *ref, WT_TIME_PAIR *start, WT_TIME_PAIR *stop); +extern void __wt_read_col_time_pairs( + WT_SESSION_IMPL *session, WT_PAGE *page, WT_CELL *cell, WT_TIME_PAIR *start, WT_TIME_PAIR *stop); +extern void __wt_read_row_time_pairs( + WT_SESSION_IMPL *session, WT_PAGE *page, WT_ROW *rip, WT_TIME_PAIR *start, WT_TIME_PAIR *stop); extern void __wt_readlock(WT_SESSION_IMPL *session, WT_RWLOCK *l); extern void __wt_readunlock(WT_SESSION_IMPL *session, WT_RWLOCK *l); extern void __wt_rec_dictionary_free(WT_SESSION_IMPL *session, WT_RECONCILE *r); extern void __wt_rec_dictionary_reset(WT_RECONCILE *r); +extern void __wt_ref_addr_free(WT_SESSION_IMPL *session, WT_REF *ref); extern void __wt_ref_out(WT_SESSION_IMPL *session, WT_REF *ref); extern void __wt_root_ref_init( WT_SESSION_IMPL *session, WT_REF *root_ref, WT_PAGE *root, bool is_recno); @@ -1733,7 +1753,6 @@ extern void __wt_txn_clear_timestamp_queues(WT_SESSION_IMPL *session); extern void __wt_txn_destroy(WT_SESSION_IMPL *session); extern void __wt_txn_get_snapshot(WT_SESSION_IMPL *session); extern void __wt_txn_global_destroy(WT_SESSION_IMPL *session); -extern void __wt_txn_named_snapshot_destroy(WT_SESSION_IMPL *session); extern void __wt_txn_op_free(WT_SESSION_IMPL *session, WT_TXN_OP *op); extern void __wt_txn_publish_read_timestamp(WT_SESSION_IMPL *session); extern void __wt_txn_publish_timestamp(WT_SESSION_IMPL *session); @@ -1794,17 +1813,19 @@ static inline bool __wt_page_is_empty(WT_PAGE *page) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); static inline bool __wt_page_is_modified(WT_PAGE *page) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); -static inline bool __wt_page_las_active(WT_SESSION_IMPL *session, WT_REF *ref) - WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); static inline bool __wt_rec_need_split(WT_RECONCILE *r, size_t len) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); -static inline bool __wt_ref_cas_state_int(WT_SESSION_IMPL *session, WT_REF *ref, uint32_t old_state, - uint32_t new_state, const char *func, int line) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); +static inline bool __wt_ref_addr_copy(WT_SESSION_IMPL *session, WT_REF *ref, WT_ADDR_COPY *copy) + WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); +static inline bool __wt_ref_cas_state_int(WT_SESSION_IMPL *session, WT_REF *ref, uint8_t old_state, + uint8_t new_state, const char *func, int line) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); static inline bool __wt_ref_is_root(WT_REF *ref) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); static inline bool __wt_row_leaf_key_info(WT_PAGE *page, void *copy, WT_IKEY **ikeyp, WT_CELL **cellp, void *datap, size_t *sizep) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); static inline bool __wt_row_leaf_value(WT_PAGE *page, WT_ROW *rip, WT_ITEM *value) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); +static inline bool __wt_row_leaf_value_exists(WT_ROW *rip) + WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); static inline bool __wt_session_can_wait(WT_SESSION_IMPL *session) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); static inline bool __wt_split_descent_race(WT_SESSION_IMPL *session, WT_REF *ref, @@ -1819,6 +1840,9 @@ static inline bool __wt_txn_visible_all(WT_SESSION_IMPL *session, uint64_t id, wt_timestamp_t timestamp) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); static inline double __wt_eviction_dirty_target(WT_CACHE *cache) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); +static inline int __wt_bt_col_var_cursor_walk_txn_read(WT_SESSION_IMPL *session, + WT_CURSOR_BTREE *cbt, WT_PAGE *page, WT_CELL_UNPACK *unpack, WT_COL *cip, WT_UPDATE **updp) + WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); static inline int __wt_btree_block_free(WT_SESSION_IMPL *session, const uint8_t *addr, size_t addr_size) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); static inline int __wt_buf_extend(WT_SESSION_IMPL *session, WT_ITEM *buf, size_t size) @@ -1979,14 +2003,18 @@ static inline int __wt_txn_modify_page_delete(WT_SESSION_IMPL *session, WT_REF * WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); static inline int __wt_txn_op_set_key(WT_SESSION_IMPL *session, const WT_ITEM *key) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); -static inline int __wt_txn_read(WT_SESSION_IMPL *session, WT_UPDATE *upd, WT_UPDATE **updp) +static inline int __wt_txn_read(WT_SESSION_IMPL *session, WT_CURSOR_BTREE *cbt, WT_UPDATE *upd, + WT_CELL_UNPACK *vpack, WT_UPDATE **updp) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); +static inline int __wt_txn_read_upd_list(WT_SESSION_IMPL *session, WT_UPDATE *upd, WT_UPDATE **updp) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); static inline int __wt_txn_search_check(WT_SESSION_IMPL *session) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); -static inline int __wt_txn_update_check(WT_SESSION_IMPL *session, WT_UPDATE *upd) +static inline int __wt_txn_update_check(WT_SESSION_IMPL *session, WT_CURSOR_BTREE *cbt, + WT_UPDATE *upd) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); +static inline int __wt_upd_alloc_tombstone(WT_SESSION_IMPL *session, WT_UPDATE **updp) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); -static inline int __wt_update_serial(WT_SESSION_IMPL *session, WT_PAGE *page, WT_UPDATE **srch_upd, - WT_UPDATE **updp, size_t upd_size, bool exclusive) +static inline int __wt_update_serial(WT_SESSION_IMPL *session, WT_CURSOR_BTREE *cbt, WT_PAGE *page, + WT_UPDATE **srch_upd, WT_UPDATE **updp, size_t upd_size, bool exclusive) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); static inline int __wt_vfprintf(WT_SESSION_IMPL *session, WT_FSTREAM *fstr, const char *fmt, va_list ap) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); @@ -2013,7 +2041,7 @@ static inline int __wt_vunpack_uint(const uint8_t **pp, size_t maxlen, uint64_t static inline int __wt_write(WT_SESSION_IMPL *session, WT_FH *fh, wt_off_t offset, size_t len, const void *buf) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); static inline size_t __wt_cell_pack_addr(WT_SESSION_IMPL *session, WT_CELL *cell, u_int cell_type, - uint64_t recno, wt_timestamp_t newest_durable_ts, wt_timestamp_t oldest_start_ts, + uint64_t recno, wt_timestamp_t stop_durable_ts, wt_timestamp_t oldest_start_ts, uint64_t oldest_start_txn, wt_timestamp_t newest_stop_ts, uint64_t newest_stop_txn, size_t size) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); static inline size_t __wt_cell_pack_copy(WT_SESSION_IMPL *session, WT_CELL *cell, @@ -2049,7 +2077,7 @@ static inline u_int __wt_cell_type_raw(WT_CELL *cell) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); static inline u_int __wt_skip_choose_depth(WT_SESSION_IMPL *session) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); -static inline uint32_t __wt_cache_lookaside_score(WT_CACHE *cache) +static inline uint32_t __wt_cache_hs_score(WT_CACHE *cache) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); static inline uint64_t __wt_btree_bytes_evictable(WT_SESSION_IMPL *session) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); @@ -2103,7 +2131,7 @@ static inline void __wt_cache_page_inmem_incr(WT_SESSION_IMPL *session, WT_PAGE static inline void __wt_cache_read_gen_bump(WT_SESSION_IMPL *session, WT_PAGE *page); static inline void __wt_cache_read_gen_incr(WT_SESSION_IMPL *session); static inline void __wt_cache_read_gen_new(WT_SESSION_IMPL *session, WT_PAGE *page); -static inline void __wt_cache_update_lookaside_score( +static inline void __wt_cache_update_hs_score( WT_SESSION_IMPL *session, u_int updates_seen, u_int updates_unstable); static inline void __wt_cell_type_reset( WT_SESSION_IMPL *session, WT_CELL *cell, u_int old_type, u_int new_type); @@ -2118,6 +2146,7 @@ static inline void __wt_cond_wait( WT_SESSION_IMPL *session, WT_CONDVAR *cond, uint64_t usecs, bool (*run_func)(WT_SESSION_IMPL *)); static inline void __wt_cursor_dhandle_decr_use(WT_SESSION_IMPL *session); static inline void __wt_cursor_dhandle_incr_use(WT_SESSION_IMPL *session); +static inline void __wt_cursor_disable_bulk(WT_SESSION_IMPL *session); static inline void __wt_epoch(WT_SESSION_IMPL *session, struct timespec *tsp); static inline void __wt_op_timer_start(WT_SESSION_IMPL *session); static inline void __wt_op_timer_stop(WT_SESSION_IMPL *session); @@ -2131,16 +2160,11 @@ static inline void __wt_rec_addr_ts_init(WT_RECONCILE *r, wt_timestamp_t *newest static inline void __wt_rec_addr_ts_update(WT_RECONCILE *r, wt_timestamp_t newest_durable_ts, wt_timestamp_t oldest_start_ts, uint64_t oldest_start_txn, wt_timestamp_t newest_stop_ts, uint64_t newest_stop_txn); -static inline void __wt_rec_cell_build_addr( - WT_SESSION_IMPL *session, WT_RECONCILE *r, WT_ADDR *addr, bool proxy_cell, uint64_t recno); +static inline void __wt_rec_cell_build_addr(WT_SESSION_IMPL *session, WT_RECONCILE *r, + WT_ADDR *addr, WT_CELL_UNPACK *vpack, bool proxy_cell, uint64_t recno); static inline void __wt_rec_image_copy(WT_SESSION_IMPL *session, WT_RECONCILE *r, WT_REC_KV *kv); static inline void __wt_rec_incr( WT_SESSION_IMPL *session, WT_RECONCILE *r, uint32_t v, size_t size); -static inline void __wt_ref_addr_free(WT_SESSION_IMPL *session, WT_REF *ref); -static inline void __wt_ref_info( - WT_SESSION_IMPL *session, WT_REF *ref, const uint8_t **addrp, size_t *sizep, bool *is_leafp); -static inline void __wt_ref_info_lock( - WT_SESSION_IMPL *session, WT_REF *ref, uint8_t *addr_buf, size_t *sizep, bool *is_leafp); static inline void __wt_ref_key(WT_PAGE *page, WT_REF *ref, void *keyp, size_t *sizep); static inline void __wt_ref_key_clear(WT_REF *ref); static inline void __wt_ref_key_onpage_set(WT_PAGE *page, WT_REF *ref, WT_CELL_UNPACK *unpack); diff --git a/src/third_party/wiredtiger/src/include/gcc.h b/src/third_party/wiredtiger/src/include/gcc.h index acf88f97988..be4503f3492 100644 --- a/src/third_party/wiredtiger/src/include/gcc.h +++ b/src/third_party/wiredtiger/src/include/gcc.h @@ -106,6 +106,7 @@ return (WT_ATOMIC_CAS(vp, &old, new)); \ } WT_ATOMIC_CAS_FUNC(8, uint8_t *vp, uint8_t old, uint8_t new) +WT_ATOMIC_CAS_FUNC(v8, volatile uint8_t *vp, uint8_t old, volatile uint8_t new) WT_ATOMIC_CAS_FUNC(16, uint16_t *vp, uint16_t old, uint16_t new) WT_ATOMIC_CAS_FUNC(32, uint32_t *vp, uint32_t old, uint32_t new) WT_ATOMIC_CAS_FUNC(v32, volatile uint32_t *vp, uint32_t old, volatile uint32_t new) @@ -141,6 +142,7 @@ __wt_atomic_cas_ptr(void *vp, void *old, void *new) return (__atomic_sub_fetch(vp, v, __ATOMIC_SEQ_CST)); \ } WT_ATOMIC_FUNC(8, uint8_t, uint8_t *vp, uint8_t v) +WT_ATOMIC_FUNC(v8, uint8_t, volatile uint8_t *vp, volatile uint8_t v) WT_ATOMIC_FUNC(16, uint16_t, uint16_t *vp, uint16_t v) WT_ATOMIC_FUNC(32, uint32_t, uint32_t *vp, uint32_t v) WT_ATOMIC_FUNC(v32, uint32_t, volatile uint32_t *vp, volatile uint32_t v) diff --git a/src/third_party/wiredtiger/src/include/hardware.h b/src/third_party/wiredtiger/src/include/hardware.h index b4bc9d3c506..aa63293f07a 100644 --- a/src/third_party/wiredtiger/src/include/hardware.h +++ b/src/third_party/wiredtiger/src/include/hardware.h @@ -16,13 +16,6 @@ (v) = (val); \ } while (0) -/* Write after all previous stores are completed. */ -#define WT_ORDERED_WRITE(v, val) \ - do { \ - WT_WRITE_BARRIER(); \ - (v) = (val); \ - } while (0) - /* * Read a shared location and guarantee that subsequent reads do not see any earlier state. */ diff --git a/src/third_party/wiredtiger/src/include/meta.h b/src/third_party/wiredtiger/src/include/meta.h index 2a27e083b18..a92d7e88e9f 100644 --- a/src/third_party/wiredtiger/src/include/meta.h +++ b/src/third_party/wiredtiger/src/include/meta.h @@ -30,8 +30,8 @@ #define WT_METAFILE_SLVG "WiredTiger.wt.orig" /* Metadata copy */ #define WT_METAFILE_URI "file:WiredTiger.wt" /* Metadata table URI */ -#define WT_LAS_FILE "WiredTigerLAS.wt" /* Lookaside table */ -#define WT_LAS_URI "file:WiredTigerLAS.wt" /* Lookaside table URI*/ +#define WT_HS_FILE "WiredTigerHS.wt" /* History store table */ +#define WT_HS_URI "file:WiredTigerHS.wt" /* History store table URI */ #define WT_SYSTEM_PREFIX "system:" /* System URI prefix */ #define WT_SYSTEM_CKPT_URI "system:checkpoint" /* Checkpoint URI */ diff --git a/src/third_party/wiredtiger/src/include/misc.h b/src/third_party/wiredtiger/src/include/misc.h index ecbc406de1a..b2df8478dd7 100644 --- a/src/third_party/wiredtiger/src/include/misc.h +++ b/src/third_party/wiredtiger/src/include/misc.h @@ -53,6 +53,10 @@ #define WT_PETABYTE ((uint64_t)1125899906842624) #define WT_EXABYTE ((uint64_t)1152921504606846976) +/* Strings used for indicating failed string buffer construction. */ +#define WT_ERR_STRING "[Error]" +#define WT_NO_ADDR_STRING "[NoAddr]" + /* * Sizes that cannot be larger than 2**32 are stored in uint32_t fields in common structures to save * space. To minimize conversions from size_t to uint32_t through the code, we use the following @@ -290,12 +294,15 @@ * acquired. */ #ifdef HAVE_DIAGNOSTIC +#define __wt_hazard_set(session, walk, busyp) \ + __wt_hazard_set_func(session, walk, busyp, __func__, __LINE__) #define __wt_scr_alloc(session, size, scratchp) \ __wt_scr_alloc_func(session, size, scratchp, __func__, __LINE__) #define __wt_page_in(session, ref, flags) __wt_page_in_func(session, ref, flags, __func__, __LINE__) #define __wt_page_swap(session, held, want, flags) \ __wt_page_swap_func(session, held, want, flags, __func__, __LINE__) #else +#define __wt_hazard_set(session, walk, busyp) __wt_hazard_set_func(session, walk, busyp) #define __wt_scr_alloc(session, size, scratchp) __wt_scr_alloc_func(session, size, scratchp) #define __wt_page_in(session, ref, flags) __wt_page_in_func(session, ref, flags) #define __wt_page_swap(session, held, want, flags) __wt_page_swap_func(session, held, want, flags) diff --git a/src/third_party/wiredtiger/src/include/msvc.h b/src/third_party/wiredtiger/src/include/msvc.h index 9ead2ecdd7a..a834f201d28 100644 --- a/src/third_party/wiredtiger/src/include/msvc.h +++ b/src/third_party/wiredtiger/src/include/msvc.h @@ -51,6 +51,7 @@ } WT_ATOMIC_FUNC(8, uint8_t, uint8_t, 8, char) +WT_ATOMIC_FUNC(v8, uint8_t, volatile uint8_t, 8, char) WT_ATOMIC_FUNC(16, uint16_t, uint16_t, 16, short) WT_ATOMIC_FUNC(32, uint32_t, uint32_t, , long) WT_ATOMIC_FUNC(v32, uint32_t, volatile uint32_t, , long) diff --git a/src/third_party/wiredtiger/src/include/reconcile.h b/src/third_party/wiredtiger/src/include/reconcile.h index f72d30ab579..783345420c7 100644 --- a/src/third_party/wiredtiger/src/include/reconcile.h +++ b/src/third_party/wiredtiger/src/include/reconcile.h @@ -20,15 +20,15 @@ struct __wt_reconcile { uint32_t flags; /* Caller's configuration */ /* - * Track start/stop checkpoint generations to decide if lookaside table records are correct. + * Track start/stop checkpoint generations to decide if history store table records are correct. */ uint64_t orig_btree_checkpoint_gen; uint64_t orig_txn_checkpoint_gen; /* - * Track the oldest running transaction and whether to skew lookaside to the newest update. + * Track the oldest running transaction and whether to skew history store to the newest update. */ - bool las_skew_newest; + bool hs_skew_newest; uint64_t last_running; /* Track the page's min/maximum transactions. */ @@ -40,15 +40,9 @@ struct __wt_reconcile { u_int updates_seen; /* Count of updates seen. */ u_int updates_unstable; /* Count of updates not visible_all. */ - bool update_uncommitted; /* An update was uncommitted. */ - bool update_used; /* An update could be used. */ - - /* All the updates are with prepare in-progress state. */ - bool all_upd_prepare_in_prog; - /* - * When we can't mark the page clean (for example, checkpoint found some uncommitted updates), - * there's a leave-dirty flag. + * When we can't mark the page clean after reconciliation (for example, checkpoint or eviction + * found some uncommitted updates), there's a leave-dirty flag. */ bool leave_dirty; @@ -156,9 +150,9 @@ struct __wt_reconcile { size_t min_space_avail; /* Remaining space in this chunk to put a minimum size boundary */ /* - * Saved update list, supporting the WT_REC_UPDATE_RESTORE and WT_REC_LOOKASIDE configurations. - * While reviewing updates for each page, we save WT_UPDATE lists here, and then move them to - * per-block areas as the blocks are defined. + * Saved update list, supporting WT_REC_HS configurations. While reviewing updates for each + * page, we save WT_UPDATE lists here, and then move them to per-block areas as the blocks are + * defined. */ WT_SAVE_UPD *supd; /* Saved updates */ uint32_t supd_next; @@ -231,10 +225,10 @@ struct __wt_reconcile { WT_SALVAGE_COOKIE *salvage; /* If it's a salvage operation */ - bool cache_write_lookaside; /* Used the lookaside table */ - bool cache_write_restore; /* Used update/restoration */ + bool cache_write_hs; /* Used the history store table */ + bool cache_write_restore; /* Used update/restoration */ - uint32_t tested_ref_state; /* Debugging information */ + uint8_t tested_ref_state; /* Debugging information */ /* * XXX In the case of a modified update, we may need a copy of the current value as a set of @@ -252,9 +246,6 @@ typedef struct { uint64_t start_txn; wt_timestamp_t stop_ts; uint64_t stop_txn; - - bool upd_saved; /* Updates saved to list */ - } WT_UPDATE_SELECT; /* diff --git a/src/third_party/wiredtiger/src/include/reconcile.i b/src/third_party/wiredtiger/src/include/reconcile.i index 5f700abccd3..89416ed12ec 100644 --- a/src/third_party/wiredtiger/src/include/reconcile.i +++ b/src/third_party/wiredtiger/src/include/reconcile.i @@ -21,15 +21,15 @@ __wt_rec_need_split(WT_RECONCILE *r, size_t len) { /* * In the case of a row-store leaf page, trigger a split if a threshold number of saved updates - * is reached. This allows pages to split for update/restore and lookaside eviction when there - * is no visible data causing the disk image to grow. + * is reached. This allows pages to split for update/restore and history store eviction when + * there is no visible data causing the disk image to grow. * * In the case of small pages or large keys, we might try to split when a page has no updates or - * entries, which isn't possible. To consider update/restore or lookaside information, require - * either page entries or updates that will be attached to the image. The limit is one of - * either, but it doesn't make sense to create pages or images with few entries or updates, even - * where page sizes are small (especially as updates that will eventually become overflow items - * can throw off our calculations). Bound the combination at something reasonable. + * entries, which isn't possible. To consider update/restore or history store information, + * require either page entries or updates that will be attached to the image. The limit is one + * of either, but it doesn't make sense to create pages or images with few entries or updates, + * even where page sizes are small (especially as updates that will eventually become overflow + * items can throw off our calculations). Bound the combination at something reasonable. */ if (r->page->type == WT_PAGE_ROW_LEAF && r->entries + r->supd_next > 10) len += r->supd_memsize; @@ -48,17 +48,17 @@ __wt_rec_addr_ts_init(WT_RECONCILE *r, wt_timestamp_t *newest_durable_ts, uint64_t *newest_stop_txnp) { /* - * If the page format supports address timestamps (and not fixed-length column-store, where we - * don't maintain timestamps at all), set the oldest/newest timestamps to values at the end of - * their expected range so they're corrected as we process key/value items. Otherwise, set the - * oldest/newest timestamps to simple durability. + * If the page is not fixed-length column-store, where we don't maintain timestamps at all, set + * the oldest/newest timestamps to values at the end of their expected range so they're + * corrected as we process key/value items. Otherwise, set the oldest/newest timestamps to + * simple durability. */ *newest_durable_ts = WT_TS_NONE; *oldest_start_tsp = WT_TS_MAX; *oldest_start_txnp = WT_TXN_MAX; *newest_stop_tsp = WT_TS_NONE; *newest_stop_txnp = WT_TXN_NONE; - if (!__wt_process.page_version_ts || r->page->type == WT_PAGE_COL_FIX) { + if (r->page->type == WT_PAGE_COL_FIX) { *newest_durable_ts = WT_TS_NONE; *oldest_start_tsp = WT_TS_NONE; *oldest_start_txnp = WT_TXN_NONE; @@ -144,11 +144,11 @@ __wt_rec_image_copy(WT_SESSION_IMPL *session, WT_RECONCILE *r, WT_REC_KV *kv) /* * __wt_rec_cell_build_addr -- - * Process an address reference and return a cell structure to be stored on the page. + * Process an address or unpack reference and return a cell structure to be stored on the page. */ static inline void -__wt_rec_cell_build_addr( - WT_SESSION_IMPL *session, WT_RECONCILE *r, WT_ADDR *addr, bool proxy_cell, uint64_t recno) +__wt_rec_cell_build_addr(WT_SESSION_IMPL *session, WT_RECONCILE *r, WT_ADDR *addr, + WT_CELL_UNPACK *vpack, bool proxy_cell, uint64_t recno) { WT_REC_KV *val; u_int cell_type; @@ -161,6 +161,8 @@ __wt_rec_cell_build_addr( */ if (proxy_cell) cell_type = WT_CELL_ADDR_DEL; + else if (vpack != NULL) + cell_type = vpack->type; else { switch (addr->type) { case WT_ADDR_INT: @@ -188,11 +190,22 @@ __wt_rec_cell_build_addr( * We don't copy the data into the buffer, it's not necessary; just re-point the buffer's * data/length fields. */ - val->buf.data = addr->addr; - val->buf.size = addr->size; - val->cell_len = __wt_cell_pack_addr(session, &val->cell, cell_type, recno, - addr->newest_durable_ts, addr->oldest_start_ts, addr->oldest_start_txn, addr->newest_stop_ts, - addr->newest_stop_txn, val->buf.size); + if (vpack == NULL) { + WT_ASSERT(session, addr != NULL); + val->buf.data = addr->addr; + val->buf.size = addr->size; + val->cell_len = __wt_cell_pack_addr(session, &val->cell, cell_type, recno, + addr->stop_durable_ts, addr->oldest_start_ts, addr->oldest_start_txn, + addr->newest_stop_ts, addr->newest_stop_txn, val->buf.size); + } else { + WT_ASSERT(session, addr == NULL); + val->buf.data = vpack->data; + val->buf.size = vpack->size; + val->cell_len = __wt_cell_pack_addr(session, &val->cell, cell_type, recno, + vpack->newest_stop_durable_ts, vpack->oldest_start_ts, vpack->oldest_start_txn, + vpack->newest_stop_ts, vpack->newest_stop_txn, val->buf.size); + } + val->len = val->cell_len + val->buf.size; } @@ -209,12 +222,11 @@ __wt_rec_cell_build_val(WT_SESSION_IMPL *session, WT_RECONCILE *r, const void *d WT_REC_KV *val; btree = S2BT(session); - val = &r->v; /* - * We don't copy the data into the buffer, it's not necessary; just re-point the buffer's - * data/length fields. + * Unless necessary we don't copy the data into the buffer; start by just re-pointing the + * buffer's data/length fields. */ val->buf.data = data; val->buf.size = size; @@ -234,6 +246,7 @@ __wt_rec_cell_build_val(WT_SESSION_IMPL *session, WT_RECONCILE *r, const void *d session, r, val, WT_CELL_VALUE_OVFL, start_ts, start_txn, stop_ts, stop_txn, rle)); } } + val->cell_len = __wt_cell_pack_value( session, &val->cell, start_ts, start_txn, stop_ts, stop_txn, rle, val->buf.size); val->len = val->cell_len + val->buf.size; diff --git a/src/third_party/wiredtiger/src/include/serial.i b/src/third_party/wiredtiger/src/include/serial.i index 59a0839c8ac..11e49c9081a 100644 --- a/src/third_party/wiredtiger/src/include/serial.i +++ b/src/third_party/wiredtiger/src/include/serial.i @@ -217,8 +217,8 @@ __wt_insert_serial(WT_SESSION_IMPL *session, WT_PAGE *page, WT_INSERT_HEAD *ins_ * Update a row or column-store entry. */ static inline int -__wt_update_serial(WT_SESSION_IMPL *session, WT_PAGE *page, WT_UPDATE **srch_upd, WT_UPDATE **updp, - size_t upd_size, bool exclusive) +__wt_update_serial(WT_SESSION_IMPL *session, WT_CURSOR_BTREE *cbt, WT_PAGE *page, + WT_UPDATE **srch_upd, WT_UPDATE **updp, size_t upd_size, bool exclusive) { WT_DECL_RET; WT_UPDATE *obsolete, *upd; @@ -237,7 +237,7 @@ __wt_update_serial(WT_SESSION_IMPL *session, WT_PAGE *page, WT_UPDATE **srch_upd * Check if our update is still permitted. */ while (!__wt_atomic_cas_ptr(srch_upd, upd->next, upd)) { - if ((ret = __wt_txn_update_check(session, upd->next = *srch_upd)) != 0) { + if ((ret = __wt_txn_update_check(session, cbt, upd->next = *srch_upd)) != 0) { /* Free unused memory on error. */ __wt_free(session, upd); return (ret); @@ -284,8 +284,7 @@ __wt_update_serial(WT_SESSION_IMPL *session, WT_PAGE *page, WT_UPDATE **srch_upd WT_PAGE_UNLOCK(session, page); - if (obsolete != NULL) - __wt_free_update_list(session, obsolete); + __wt_free_update_list(session, &obsolete); return (0); } diff --git a/src/third_party/wiredtiger/src/include/session.h b/src/third_party/wiredtiger/src/include/session.h index a265ffda153..c92abe34c5c 100644 --- a/src/third_party/wiredtiger/src/include/session.h +++ b/src/third_party/wiredtiger/src/include/session.h @@ -93,7 +93,11 @@ struct __wt_session_impl { WT_COMPACT_STATE *compact; /* Compaction information */ enum { WT_COMPACT_NONE = 0, WT_COMPACT_RUNNING, WT_COMPACT_SUCCESS } compact_state; - WT_CURSOR *las_cursor; /* Lookaside table cursor */ + WT_CURSOR *hs_cursor; /* History store table cursor */ + + /* Original transaction time pair to use for the history store inserts */ + uint64_t orig_txnid_to_las; + wt_timestamp_t orig_timestamp_to_las; WT_CURSOR *meta_cursor; /* Metadata file */ void *meta_track; /* Metadata operation tracking */ @@ -170,31 +174,32 @@ struct __wt_session_impl { #define WT_SESSION_BACKUP_DUP 0x00000002u #define WT_SESSION_CACHE_CURSORS 0x00000004u #define WT_SESSION_CAN_WAIT 0x00000008u -#define WT_SESSION_IGNORE_CACHE_SIZE 0x00000010u -#define WT_SESSION_INTERNAL 0x00000020u -#define WT_SESSION_LOCKED_CHECKPOINT 0x00000040u -#define WT_SESSION_LOCKED_HANDLE_LIST_READ 0x00000080u -#define WT_SESSION_LOCKED_HANDLE_LIST_WRITE 0x00000100u -#define WT_SESSION_LOCKED_HOTBACKUP_READ 0x00000200u -#define WT_SESSION_LOCKED_HOTBACKUP_WRITE 0x00000400u -#define WT_SESSION_LOCKED_METADATA 0x00000800u -#define WT_SESSION_LOCKED_PASS 0x00001000u -#define WT_SESSION_LOCKED_SCHEMA 0x00002000u -#define WT_SESSION_LOCKED_SLOT 0x00004000u -#define WT_SESSION_LOCKED_TABLE_READ 0x00008000u -#define WT_SESSION_LOCKED_TABLE_WRITE 0x00010000u -#define WT_SESSION_LOCKED_TURTLE 0x00020000u -#define WT_SESSION_LOGGING_INMEM 0x00040000u -#define WT_SESSION_LOOKASIDE_CURSOR 0x00080000u -#define WT_SESSION_NO_DATA_HANDLES 0x00100000u -#define WT_SESSION_NO_LOGGING 0x00200000u -#define WT_SESSION_NO_RECONCILE 0x00400000u -#define WT_SESSION_NO_SCHEMA_LOCK 0x00800000u -#define WT_SESSION_QUIET_CORRUPT_FILE 0x01000000u -#define WT_SESSION_READ_WONT_NEED 0x02000000u -#define WT_SESSION_RESOLVING_TXN 0x04000000u -#define WT_SESSION_SCHEMA_TXN 0x08000000u -#define WT_SESSION_SERVER_ASYNC 0x10000000u +#define WT_SESSION_HS_CURSOR 0x00000010u +#define WT_SESSION_IGNORE_CACHE_SIZE 0x00000020u +#define WT_SESSION_IGNORE_HS_TOMBSTONE 0x00000040u +#define WT_SESSION_INTERNAL 0x00000080u +#define WT_SESSION_LOCKED_CHECKPOINT 0x00000100u +#define WT_SESSION_LOCKED_HANDLE_LIST_READ 0x00000200u +#define WT_SESSION_LOCKED_HANDLE_LIST_WRITE 0x00000400u +#define WT_SESSION_LOCKED_HOTBACKUP_READ 0x00000800u +#define WT_SESSION_LOCKED_HOTBACKUP_WRITE 0x00001000u +#define WT_SESSION_LOCKED_METADATA 0x00002000u +#define WT_SESSION_LOCKED_PASS 0x00004000u +#define WT_SESSION_LOCKED_SCHEMA 0x00008000u +#define WT_SESSION_LOCKED_SLOT 0x00010000u +#define WT_SESSION_LOCKED_TABLE_READ 0x00020000u +#define WT_SESSION_LOCKED_TABLE_WRITE 0x00040000u +#define WT_SESSION_LOCKED_TURTLE 0x00080000u +#define WT_SESSION_LOGGING_INMEM 0x00100000u +#define WT_SESSION_NO_DATA_HANDLES 0x00200000u +#define WT_SESSION_NO_LOGGING 0x00400000u +#define WT_SESSION_NO_RECONCILE 0x00800000u +#define WT_SESSION_NO_SCHEMA_LOCK 0x01000000u +#define WT_SESSION_QUIET_CORRUPT_FILE 0x02000000u +#define WT_SESSION_READ_WONT_NEED 0x04000000u +#define WT_SESSION_RESOLVING_TXN 0x08000000u +#define WT_SESSION_SCHEMA_TXN 0x10000000u +#define WT_SESSION_SERVER_ASYNC 0x20000000u /* AUTOMATIC FLAG VALUE GENERATION STOP */ uint32_t flags; @@ -274,3 +279,9 @@ struct __wt_session_impl { WT_SESSION_STATS stats; }; + +/* + * Rollback to stable should ignore tombstones in the history store since it needs to scan the + * entire table sequentially. + */ +#define WT_SESSION_ROLLBACK_TO_STABLE_FLAGS (WT_SESSION_IGNORE_HS_TOMBSTONE) diff --git a/src/third_party/wiredtiger/src/include/stat.h b/src/third_party/wiredtiger/src/include/stat.h index c1d024b73a0..23761319646 100644 --- a/src/third_party/wiredtiger/src/include/stat.h +++ b/src/third_party/wiredtiger/src/include/stat.h @@ -331,20 +331,13 @@ struct __wt_connection_stats { int64_t cache_write_app_count; int64_t cache_write_app_time; int64_t cache_bytes_image; - int64_t cache_bytes_lookaside; + int64_t cache_bytes_hs; int64_t cache_bytes_inuse; int64_t cache_bytes_dirty_total; int64_t cache_bytes_other; int64_t cache_bytes_read; int64_t cache_bytes_write; - int64_t cache_lookaside_cursor_wait_application; - int64_t cache_lookaside_cursor_wait_internal; int64_t cache_lookaside_score; - int64_t cache_lookaside_entries; - int64_t cache_lookaside_insert; - int64_t cache_lookaside_ondisk_max; - int64_t cache_lookaside_ondisk; - int64_t cache_lookaside_remove; int64_t cache_eviction_checkpoint; int64_t cache_eviction_get_ref; int64_t cache_eviction_get_ref_empty; @@ -394,6 +387,17 @@ struct __wt_connection_stats { int64_t cache_hazard_checks; int64_t cache_hazard_walks; int64_t cache_hazard_max; + int64_t cache_hs_key_truncate_mix_ts; + int64_t cache_hs_key_truncate_onpage_removal; + int64_t cache_hs_score; + int64_t cache_hs_insert; + int64_t cache_hs_ondisk_max; + int64_t cache_hs_ondisk; + int64_t cache_hs_read; + int64_t cache_hs_read_miss; + int64_t cache_hs_read_squash; + int64_t cache_hs_remove_key_truncate; + int64_t cache_hs_write_squash; int64_t cache_inmem_splittable; int64_t cache_inmem_split; int64_t cache_eviction_internal; @@ -409,7 +413,7 @@ struct __wt_connection_stats { int64_t cache_timed_out_ops; int64_t cache_read_overflow; int64_t cache_eviction_deepen; - int64_t cache_write_lookaside; + int64_t cache_write_hs; int64_t cache_pages_inuse; int64_t cache_eviction_app; int64_t cache_eviction_pages_queued; @@ -419,11 +423,6 @@ struct __wt_connection_stats { int64_t cache_read; int64_t cache_read_deleted; int64_t cache_read_deleted_prepared; - int64_t cache_read_lookaside; - int64_t cache_read_lookaside_checkpoint; - int64_t cache_read_lookaside_skipped; - int64_t cache_read_lookaside_delay; - int64_t cache_read_lookaside_delay_checkpoint; int64_t cache_pages_requested; int64_t cache_eviction_pages_seen; int64_t cache_eviction_pages_already_queued; @@ -505,6 +504,9 @@ struct __wt_connection_stats { int64_t dh_sweeps; int64_t dh_session_handles; int64_t dh_session_sweeps; + int64_t hs_gc_pages_evict; + int64_t hs_gc_pages_removed; + int64_t hs_gc_pages_visited; int64_t lock_checkpoint_count; int64_t lock_checkpoint_wait_application; int64_t lock_checkpoint_wait_internal; @@ -650,14 +652,11 @@ struct __wt_connection_stats { int64_t page_del_rollback_blocked; int64_t child_modify_blocked_page; int64_t txn_prepared_updates_count; - int64_t txn_prepared_updates_lookaside_inserts; int64_t txn_durable_queue_walked; int64_t txn_durable_queue_empty; int64_t txn_durable_queue_head; int64_t txn_durable_queue_inserts; int64_t txn_durable_queue_len; - int64_t txn_snapshots_created; - int64_t txn_snapshots_dropped; int64_t txn_prepare; int64_t txn_prepare_commit; int64_t txn_prepare_active; @@ -668,9 +667,12 @@ struct __wt_connection_stats { int64_t txn_read_queue_head; int64_t txn_read_queue_inserts; int64_t txn_read_queue_len; - int64_t txn_rollback_to_stable; - int64_t txn_rollback_upd_aborted; - int64_t txn_rollback_las_removed; + int64_t txn_rts; + int64_t txn_rts_keys_removed; + int64_t txn_rts_keys_restored; + int64_t txn_rts_pages_visited; + int64_t txn_rts_upd_aborted; + int64_t txn_rts_hs_removed; int64_t txn_set_ts; int64_t txn_set_ts_durable; int64_t txn_set_ts_durable_upd; @@ -681,9 +683,15 @@ struct __wt_connection_stats { int64_t txn_begin; int64_t txn_checkpoint_running; int64_t txn_checkpoint_generation; + int64_t txn_hs_ckpt_duration; int64_t txn_checkpoint_time_max; int64_t txn_checkpoint_time_min; int64_t txn_checkpoint_time_recent; + int64_t txn_checkpoint_prep_running; + int64_t txn_checkpoint_prep_max; + int64_t txn_checkpoint_prep_min; + int64_t txn_checkpoint_prep_recent; + int64_t txn_checkpoint_prep_total; int64_t txn_checkpoint_scrub_target; int64_t txn_checkpoint_scrub_time; int64_t txn_checkpoint_time_total; @@ -694,7 +702,6 @@ struct __wt_connection_stats { int64_t txn_checkpoint_fsync_post_duration; int64_t txn_pinned_range; int64_t txn_pinned_checkpoint_range; - int64_t txn_pinned_snapshot_range; int64_t txn_pinned_timestamp; int64_t txn_pinned_timestamp_checkpoint; int64_t txn_pinned_timestamp_reader; @@ -772,6 +779,7 @@ struct __wt_dsrc_stats { int64_t cache_eviction_walk_from_root; int64_t cache_eviction_walk_saved_pos; int64_t cache_eviction_hazard; + int64_t cache_hs_read; int64_t cache_inmem_splittable; int64_t cache_inmem_split; int64_t cache_eviction_internal; @@ -780,11 +788,10 @@ struct __wt_dsrc_stats { int64_t cache_eviction_dirty; int64_t cache_read_overflow; int64_t cache_eviction_deepen; - int64_t cache_write_lookaside; + int64_t cache_write_hs; int64_t cache_read; int64_t cache_read_deleted; int64_t cache_read_deleted_prepared; - int64_t cache_read_lookaside; int64_t cache_pages_requested; int64_t cache_eviction_pages_seen; int64_t cache_write; diff --git a/src/third_party/wiredtiger/src/include/thread_group.h b/src/third_party/wiredtiger/src/include/thread_group.h index d858881bcec..6a3908ba591 100644 --- a/src/third_party/wiredtiger/src/include/thread_group.h +++ b/src/third_party/wiredtiger/src/include/thread_group.h @@ -24,7 +24,7 @@ struct __wt_thread { /* AUTOMATIC FLAG VALUE GENERATION START */ #define WT_THREAD_ACTIVE 0x01u /* thread is active or paused */ #define WT_THREAD_CAN_WAIT 0x02u /* WT_SESSION_CAN_WAIT */ -#define WT_THREAD_LOOKASIDE 0x04u /* open lookaside cursor */ +#define WT_THREAD_HS 0x04u /* open history store cursor */ #define WT_THREAD_PANIC_FAIL 0x08u /* panic if the thread fails */ #define WT_THREAD_RUN 0x10u /* thread is running */ /* AUTOMATIC FLAG VALUE GENERATION STOP */ diff --git a/src/third_party/wiredtiger/src/include/txn.h b/src/third_party/wiredtiger/src/include/txn.h index bea0d63b753..588368599ad 100644 --- a/src/third_party/wiredtiger/src/include/txn.h +++ b/src/third_party/wiredtiger/src/include/txn.h @@ -57,12 +57,19 @@ typedef enum { * We format timestamps in a couple of ways, declare appropriate sized buffers. Hexadecimal is 2x * the size of the value. MongoDB format (high/low pairs of 4B unsigned integers, with surrounding * parenthesis and separating comma and space), is 2x the maximum digits from a 4B unsigned integer - * plus 4. Both sizes include a trailing nul byte as well. + * plus 4. Both sizes include a trailing null byte as well. */ #define WT_TS_HEX_STRING_SIZE (2 * sizeof(wt_timestamp_t) + 1) #define WT_TS_INT_STRING_SIZE (2 * 10 + 4 + 1) /* + * We need an appropriately sized buffer for formatted time pairs. This is for time pairs of the + * form (time_stamp, slash and transaction_id), which gives the max digits of a timestamp plus slash + * plus max digits of a 8 byte integer with a trailing null byte. + */ +#define WT_TP_STRING_SIZE (WT_TS_INT_STRING_SIZE + 1 + 20 + 1) + +/* * Perform an operation at the specified isolation level. * * This is fiddly: we can't cope with operations that begin transactions @@ -173,11 +180,6 @@ struct __wt_txn_global { uint64_t debug_rollback; /* Debug mode rollback */ volatile uint64_t metadata_pinned; /* Oldest ID for metadata */ - /* Named snapshot state. */ - WT_RWLOCK nsnap_rwlock; - volatile uint64_t nsnap_oldest_id; - TAILQ_HEAD(__wt_nsnap_qh, __wt_named_snapshot) nsnaph; - WT_TXN_STATE *states; /* Per-session transaction states */ }; @@ -337,31 +339,30 @@ struct __wt_txn { */ /* AUTOMATIC FLAG VALUE GENERATION START */ -#define WT_TXN_AUTOCOMMIT 0x0000001u -#define WT_TXN_ERROR 0x0000002u -#define WT_TXN_HAS_ID 0x0000004u -#define WT_TXN_HAS_SNAPSHOT 0x0000008u -#define WT_TXN_HAS_TS_COMMIT 0x0000010u -#define WT_TXN_HAS_TS_DURABLE 0x0000020u -#define WT_TXN_HAS_TS_PREPARE 0x0000040u -#define WT_TXN_HAS_TS_READ 0x0000080u -#define WT_TXN_IGNORE_PREPARE 0x0000100u -#define WT_TXN_NAMED_SNAPSHOT 0x0000200u -#define WT_TXN_PREPARE 0x0000400u -#define WT_TXN_PUBLIC_TS_READ 0x0000800u -#define WT_TXN_READONLY 0x0001000u -#define WT_TXN_RUNNING 0x0002000u -#define WT_TXN_SYNC_SET 0x0004000u -#define WT_TXN_TS_COMMIT_ALWAYS 0x0008000u -#define WT_TXN_TS_COMMIT_KEYS 0x0010000u -#define WT_TXN_TS_COMMIT_NEVER 0x0020000u -#define WT_TXN_TS_DURABLE_ALWAYS 0x0040000u -#define WT_TXN_TS_DURABLE_KEYS 0x0080000u -#define WT_TXN_TS_DURABLE_NEVER 0x0100000u -#define WT_TXN_TS_PUBLISHED 0x0200000u -#define WT_TXN_TS_ROUND_PREPARED 0x0400000u -#define WT_TXN_TS_ROUND_READ 0x0800000u -#define WT_TXN_UPDATE 0x1000000u +#define WT_TXN_AUTOCOMMIT 0x000001u +#define WT_TXN_ERROR 0x000002u +#define WT_TXN_HAS_ID 0x000004u +#define WT_TXN_HAS_SNAPSHOT 0x000008u +#define WT_TXN_HAS_TS_COMMIT 0x000010u +#define WT_TXN_HAS_TS_DURABLE 0x000020u +#define WT_TXN_HAS_TS_PREPARE 0x000040u +#define WT_TXN_HAS_TS_READ 0x000080u +#define WT_TXN_IGNORE_PREPARE 0x000100u +#define WT_TXN_PREPARE 0x000200u +#define WT_TXN_PUBLIC_TS_READ 0x000400u +#define WT_TXN_READONLY 0x000800u +#define WT_TXN_RUNNING 0x001000u +#define WT_TXN_SYNC_SET 0x002000u +#define WT_TXN_TS_COMMIT_ALWAYS 0x004000u +#define WT_TXN_TS_COMMIT_KEYS 0x008000u +#define WT_TXN_TS_COMMIT_NEVER 0x010000u +#define WT_TXN_TS_DURABLE_ALWAYS 0x020000u +#define WT_TXN_TS_DURABLE_KEYS 0x040000u +#define WT_TXN_TS_DURABLE_NEVER 0x080000u +#define WT_TXN_TS_PUBLISHED 0x100000u +#define WT_TXN_TS_ROUND_PREPARED 0x200000u +#define WT_TXN_TS_ROUND_READ 0x400000u +#define WT_TXN_UPDATE 0x800000u /* AUTOMATIC FLAG VALUE GENERATION STOP */ uint32_t flags; }; diff --git a/src/third_party/wiredtiger/src/include/txn.i b/src/third_party/wiredtiger/src/include/txn.i index 98631a61b1a..12114d9a55c 100644 --- a/src/third_party/wiredtiger/src/include/txn.i +++ b/src/third_party/wiredtiger/src/include/txn.i @@ -7,34 +7,6 @@ */ /* - * __wt_ref_cas_state_int -- - * Try to do a compare and swap, if successful update the ref history in diagnostic mode. - */ -static inline bool -__wt_ref_cas_state_int(WT_SESSION_IMPL *session, WT_REF *ref, uint32_t old_state, - uint32_t new_state, const char *func, int line) -{ - bool cas_result; - - /* Parameters that are used in a macro for diagnostic builds */ - WT_UNUSED(session); - WT_UNUSED(func); - WT_UNUSED(line); - - cas_result = __wt_atomic_casv32(&ref->state, old_state, new_state); - -#ifdef HAVE_DIAGNOSTIC - /* - * The history update here has potential to race; if the state gets updated again after the CAS - * above but before the history has been updated. - */ - if (cas_result) - WT_REF_SAVE_STATE(ref, new_state, func, line); -#endif - return (cas_result); -} - -/* * __wt_txn_context_prepare_check -- * Return an error if the current transaction is in the prepare state. */ @@ -134,15 +106,15 @@ __wt_txn_op_set_recno(WT_SESSION_IMPL *session, uint64_t recno) WT_ASSERT(session, txn->mod_count > 0 && recno != WT_RECNO_OOB); op = txn->mod + txn->mod_count - 1; - if (WT_SESSION_IS_CHECKPOINT(session) || F_ISSET(op->btree, WT_BTREE_LOOKASIDE) || + if (WT_SESSION_IS_CHECKPOINT(session) || WT_IS_HS(op->btree) || WT_IS_METADATA(op->btree->dhandle)) return; WT_ASSERT(session, op->type == WT_TXN_OP_BASIC_COL || op->type == WT_TXN_OP_INMEM_COL); /* - * Copy the recno into the transaction operation structure, so when update is evicted to - * lookaside, we have a chance of finding it again. Even though only prepared updates can be + * Copy the recno into the transaction operation structure, so when update is evicted to the + * history store, we have a chance of finding it again. Even though only prepared updates can be * evicted, at this stage we don't know whether this transaction will be prepared or not, hence * we are copying the key for all operations, so that we can use this key to fetch the update in * case this transaction is prepared. @@ -166,15 +138,15 @@ __wt_txn_op_set_key(WT_SESSION_IMPL *session, const WT_ITEM *key) op = txn->mod + txn->mod_count - 1; - if (WT_SESSION_IS_CHECKPOINT(session) || F_ISSET(op->btree, WT_BTREE_LOOKASIDE) || + if (WT_SESSION_IS_CHECKPOINT(session) || WT_IS_HS(op->btree) || WT_IS_METADATA(op->btree->dhandle)) return (0); WT_ASSERT(session, op->type == WT_TXN_OP_BASIC_ROW || op->type == WT_TXN_OP_INMEM_ROW); /* - * Copy the key into the transaction operation structure, so when update is evicted to - * lookaside, we have a chance of finding it again. Even though only prepared updates can be + * Copy the key into the transaction operation structure, so when update is evicted to the + * history store, we have a chance of finding it again. Even though only prepared updates can be * evicted, at this stage we don't know whether this transaction will be prepared or not, hence * we are copying the key for all operations, so that we can use this key to fetch the update in * case this transaction is prepared. @@ -267,8 +239,7 @@ __wt_txn_op_apply_prepare_state(WT_SESSION_IMPL *session, WT_REF *ref, bool comm WT_TXN *txn; WT_UPDATE **updp; wt_timestamp_t ts; - uint32_t previous_state; - uint8_t prepare_state; + uint8_t prepare_state, previous_state; txn = &session->txn; @@ -276,13 +247,7 @@ __wt_txn_op_apply_prepare_state(WT_SESSION_IMPL *session, WT_REF *ref, bool comm * Lock the ref to ensure we don't race with eviction freeing the page deleted update list or * with a page instantiate. */ - for (;; __wt_yield()) { - previous_state = ref->state; - WT_ASSERT(session, previous_state != WT_REF_READING); - if (previous_state != WT_REF_LOCKED && - WT_REF_CAS_STATE(session, ref, previous_state, WT_REF_LOCKED)) - break; - } + WT_REF_LOCK(session, ref, &previous_state); if (commit) { ts = txn->commit_timestamp; @@ -306,8 +271,7 @@ __wt_txn_op_apply_prepare_state(WT_SESSION_IMPL *session, WT_REF *ref, bool comm ref->page_del->durable_timestamp = txn->durable_timestamp; WT_PUBLISH(ref->page_del->prepare_state, prepare_state); - /* Unlock the page by setting it back to it's previous state */ - WT_REF_SET_STATE(ref, previous_state); + WT_REF_UNLOCK(ref, previous_state); } /* @@ -319,7 +283,7 @@ __wt_txn_op_delete_commit_apply_timestamps(WT_SESSION_IMPL *session, WT_REF *ref { WT_TXN *txn; WT_UPDATE **updp; - uint32_t previous_state; + uint8_t previous_state; txn = &session->txn; @@ -327,21 +291,14 @@ __wt_txn_op_delete_commit_apply_timestamps(WT_SESSION_IMPL *session, WT_REF *ref * Lock the ref to ensure we don't race with eviction freeing the page deleted update list or * with a page instantiate. */ - for (;; __wt_yield()) { - previous_state = ref->state; - WT_ASSERT(session, previous_state != WT_REF_READING); - if (previous_state != WT_REF_LOCKED && - WT_REF_CAS_STATE(session, ref, previous_state, WT_REF_LOCKED)) - break; - } + WT_REF_LOCK(session, ref, &previous_state); for (updp = ref->page_del->update_list; updp != NULL && *updp != NULL; ++updp) { (*updp)->start_ts = txn->commit_timestamp; (*updp)->durable_ts = txn->durable_timestamp; } - /* Unlock the page by setting it back to it's previous state */ - WT_REF_SET_STATE(ref, previous_state); + WT_REF_UNLOCK(ref, previous_state); } /* @@ -432,9 +389,16 @@ __wt_txn_modify(WT_SESSION_IMPL *session, WT_UPDATE *upd) op->type = WT_TXN_OP_BASIC_COL; } op->u.op_upd = upd; - upd->txnid = session->txn.id; - __wt_txn_op_set_timestamp(session, op); + /* Use the original transaction time pair for the history store inserts */ + if (WT_IS_HS(S2BT(session))) { + upd->txnid = session->orig_txnid_to_las; + upd->start_ts = session->orig_timestamp_to_las; + } else { + upd->txnid = session->txn.id; + __wt_txn_op_set_timestamp(session, op); + } + return (0); } @@ -491,9 +455,8 @@ __wt_txn_oldest_id(WT_SESSION_IMPL *session) * Take a local copy of these IDs in case they are updated while we are checking visibility. */ oldest_id = txn_global->oldest_id; - include_checkpoint_txn = - btree == NULL || (!F_ISSET(btree, WT_BTREE_LOOKASIDE) && - btree->checkpoint_gen != __wt_gen(session, WT_GEN_CHECKPOINT)); + include_checkpoint_txn = btree == NULL || + (!WT_IS_HS(btree) && btree->checkpoint_gen != __wt_gen(session, WT_GEN_CHECKPOINT)); if (!include_checkpoint_txn) return (oldest_id); @@ -544,9 +507,8 @@ __wt_txn_pinned_timestamp(WT_SESSION_IMPL *session, wt_timestamp_t *pinned_tsp) * If there is no active checkpoint or this handle is up to date with the active checkpoint then * it's safe to ignore the checkpoint ID in the visibility check. */ - include_checkpoint_txn = - btree == NULL || (!F_ISSET(btree, WT_BTREE_LOOKASIDE) && - btree->checkpoint_gen != __wt_gen(session, WT_GEN_CHECKPOINT)); + include_checkpoint_txn = btree == NULL || + (!WT_IS_HS(btree) && btree->checkpoint_gen != __wt_gen(session, WT_GEN_CHECKPOINT)); if (!include_checkpoint_txn) return; @@ -621,11 +583,7 @@ __wt_txn_upd_visible_all(WT_SESSION_IMPL *session, WT_UPDATE *upd) if (upd->prepare_state == WT_PREPARE_LOCKED || upd->prepare_state == WT_PREPARE_INPROGRESS) return (false); - /* - * This function is used to determine when an update is obsolete: that should take into account - * the durable timestamp which is greater than or equal to the start timestamp. - */ - return (__wt_txn_visible_all(session, upd->txnid, upd->durable_ts)); + return (__wt_txn_visible_all(session, upd->txnid, upd->start_ts)); } /* @@ -755,42 +713,156 @@ __wt_txn_upd_visible(WT_SESSION_IMPL *session, WT_UPDATE *upd) } /* - * __wt_txn_read -- + * __wt_upd_alloc_tombstone -- + * Allocate a tombstone update with default values. + */ +static inline int +__wt_upd_alloc_tombstone(WT_SESSION_IMPL *session, WT_UPDATE **updp) +{ + size_t notused; + + /* + * The underlying allocation code clears memory, which is the equivalent of setting: + * + * WT_UPDATE.txnid = WT_TXN_NONE; + * WT_UPDATE.durable_ts = WT_TS_NONE; + * WT_UPDATE.start_ts = WT_TS_NONE; + * WT_UPDATE.prepare_state = WT_PREPARE_INIT; + * WT_UPDATE.flags = 0; + */ + return (__wt_update_alloc(session, NULL, updp, ¬used, WT_UPDATE_TOMBSTONE)); +} + +/* + * __wt_txn_read_upd_list -- * Get the first visible update in a list (or NULL if none are visible). */ static inline int -__wt_txn_read(WT_SESSION_IMPL *session, WT_UPDATE *upd, WT_UPDATE **updp) +__wt_txn_read_upd_list(WT_SESSION_IMPL *session, WT_UPDATE *upd, WT_UPDATE **updp) { - static WT_UPDATE tombstone = {.txnid = WT_TXN_NONE, .type = WT_UPDATE_TOMBSTONE}; WT_VISIBLE_TYPE upd_visible; uint8_t type; - bool skipped_birthmark; *updp = NULL; - type = WT_UPDATE_INVALID; /* [-Wconditional-uninitialized] */ - for (skipped_birthmark = false; upd != NULL; upd = upd->next) { + for (; upd != NULL; upd = upd->next) { WT_ORDERED_READ(type, upd->type); - /* Skip reserved place-holders, they're never visible. */ - if (type != WT_UPDATE_RESERVE) { - upd_visible = __wt_txn_upd_visible_type(session, upd); - if (upd_visible == WT_VISIBLE_TRUE) - break; - if (upd_visible == WT_VISIBLE_PREPARE) - return (WT_PREPARE_CONFLICT); + if (type == WT_UPDATE_RESERVE) + continue; + upd_visible = __wt_txn_upd_visible_type(session, upd); + if (upd_visible == WT_VISIBLE_TRUE) { + /* Don't consider tombstone updates for the history store during rollback to stable. */ + if (type == WT_UPDATE_TOMBSTONE && WT_IS_HS(S2BT(session)) && + F_ISSET(session, WT_SESSION_IGNORE_HS_TOMBSTONE)) + continue; + *updp = upd; + return (0); + } + if (upd_visible == WT_VISIBLE_PREPARE) + return (WT_PREPARE_CONFLICT); + } + return (0); +} + +/* + * __wt_txn_read -- + * Get the first visible update in a chain. This function will first check the update list + * supplied as a function argument. If there is no visible update, it will check the onpage + * value for the given key. Finally, if the onpage value is not visible to the reader, the + * function will search the history store for a visible update. + */ +static inline int +__wt_txn_read(WT_SESSION_IMPL *session, WT_CURSOR_BTREE *cbt, WT_UPDATE *upd, WT_CELL_UNPACK *vpack, + WT_UPDATE **updp) +{ + WT_DECL_RET; + WT_ITEM buf; + WT_TIME_PAIR start, stop; + size_t size; + + *updp = NULL; + WT_RET(__wt_txn_read_upd_list(session, upd, updp)); + if (*updp != NULL) + return (0); + + /* If there is no ondisk value, there can't be anything in the history store either. */ + if (cbt->ref->page->dsk == NULL || cbt->slot == UINT32_MAX) + return (__wt_upd_alloc_tombstone(session, updp)); + + buf.data = NULL; + buf.size = 0; + buf.mem = NULL; + buf.memsize = 0; + buf.flags = 0; + + /* Check the ondisk value. */ + if (vpack == NULL) { + ret = __wt_value_return_buf(cbt, cbt->ref, &buf, &start, &stop); + if (ret != 0) { + __wt_buf_free(session, &buf); + return (ret); } - /* An invisible birthmark is equivalent to a tombstone. */ - if (type == WT_UPDATE_BIRTHMARK) - skipped_birthmark = true; + } else { + start.timestamp = vpack->start_ts; + start.txnid = vpack->start_txn; + stop.timestamp = vpack->stop_ts; + stop.txnid = vpack->start_txn; + buf.data = vpack->data; + buf.size = vpack->size; + } + + /* + * If the stop pair is set, that means that there is a tombstone at that time. If the stop time + * pair is visible to our txn then that means we've just spotted a tombstone and should return + * "not found", except for history store scan during rollback to stable. + */ + if (stop.txnid != WT_TXN_MAX && stop.timestamp != WT_TS_MAX && + (!WT_IS_HS(S2BT(session)) || !F_ISSET(session, WT_SESSION_IGNORE_HS_TOMBSTONE)) && + __wt_txn_visible(session, stop.txnid, stop.timestamp)) { + __wt_buf_free(session, &buf); + WT_RET(__wt_upd_alloc_tombstone(session, updp)); + (*updp)->txnid = stop.txnid; + /* FIXME: Reevaluate this as part of PM-1524. */ + (*updp)->durable_ts = (*updp)->start_ts = stop.timestamp; + F_SET(*updp, WT_UPDATE_RESTORED_FROM_DISK); + return (0); + } + + /* + * If the start time pair is visible then we need to return the ondisk value. + * + * FIXME-PM-1521: This should be probably be re-factored to return a buffer of bytes rather than + * an update. This allocation is expensive and doesn't serve a purpose other than to work within + * the current system. + */ + if (__wt_txn_visible(session, start.txnid, start.timestamp)) { + ret = __wt_update_alloc(session, &buf, updp, &size, WT_UPDATE_STANDARD); + __wt_buf_free(session, &buf); + WT_RET(ret); + (*updp)->txnid = start.txnid; + (*updp)->start_ts = start.timestamp; + F_SET((*updp), WT_UPDATE_RESTORED_FROM_DISK); + return (0); } - if (upd == NULL && skipped_birthmark) { - upd = &tombstone; - type = upd->type; + /* If there's no visible update in the update chain or ondisk, check the history store file. */ + if (F_ISSET(S2C(session), WT_CONN_HS_OPEN) && !F_ISSET(S2BT(session), WT_BTREE_HS)) { + ret = __wt_find_hs_upd(session, cbt, updp, false, &buf); + __wt_buf_free(session, &buf); + WT_RET_NOTFOUND_OK(ret); } - *updp = upd == NULL || type == WT_UPDATE_BIRTHMARK ? NULL : upd; + __wt_buf_free(session, &buf); + /* + * Return null not tombstone if nothing is found in history store. + */ + WT_ASSERT(session, (*updp) == NULL || (*updp)->type != WT_UPDATE_TOMBSTONE); + + /* + * FIXME-PM-1521: We call transaction read in a lot of places so we can't do this yet. When we + * re-factor this function to return a byte array, we should tackle this at the same time. + */ return (0); } @@ -807,13 +879,13 @@ __wt_txn_begin(WT_SESSION_IMPL *session, const char *cfg[]) txn->isolation = session->isolation; txn->txn_logsync = S2C(session)->txn_logsync; + WT_ASSERT(session, !F_ISSET(txn, WT_TXN_RUNNING)); + if (cfg != NULL) WT_RET(__wt_txn_config(session, cfg)); - /* - * Allocate a snapshot if required. Named snapshot transactions already have an ID setup. - */ - if (txn->isolation == WT_ISO_SNAPSHOT && !F_ISSET(txn, WT_TXN_NAMED_SNAPSHOT)) { + /* Allocate a snapshot if required. */ + if (txn->isolation == WT_ISO_SNAPSHOT) { if (session->ncursors > 0) WT_RET(__wt_session_copy_values(session)); @@ -990,12 +1062,15 @@ __wt_txn_search_check(WT_SESSION_IMPL *session) * Check if the current transaction can update an item. */ static inline int -__wt_txn_update_check(WT_SESSION_IMPL *session, WT_UPDATE *upd) +__wt_txn_update_check(WT_SESSION_IMPL *session, WT_CURSOR_BTREE *cbt, WT_UPDATE *upd) { + WT_DECL_RET; + WT_TIME_PAIR start, stop; WT_TXN *txn; WT_TXN_GLOBAL *txn_global; - bool ignore_prepare_set; + bool ignore_prepare_set, rollback; + rollback = false; txn = &session->txn; txn_global = &S2C(session)->txn_global; @@ -1013,17 +1088,36 @@ __wt_txn_update_check(WT_SESSION_IMPL *session, WT_UPDATE *upd) F_CLR(txn, WT_TXN_IGNORE_PREPARE); for (; upd != NULL && !__wt_txn_upd_visible(session, upd); upd = upd->next) { if (upd->txnid != WT_TXN_ABORTED) { - if (ignore_prepare_set) - F_SET(txn, WT_TXN_IGNORE_PREPARE); - WT_STAT_CONN_INCR(session, txn_update_conflict); - WT_STAT_DATA_INCR(session, txn_update_conflict); - return (__wt_txn_rollback_required(session, "conflict between concurrent operations")); + rollback = true; + break; } } + WT_ASSERT(session, upd != NULL || !rollback); + + /* + * Check conflict against the on page value if there is no update on the update chain except + * aborted updates. Otherwise, we would have either already detected a conflict if we saw an + * uncommitted update or determined that it would be safe to write if we saw a committed update. + */ + if (!rollback && upd == NULL && cbt != NULL && cbt->btree->type != BTREE_COL_FIX && + cbt->ins == NULL) { + __wt_read_cell_time_pairs(cbt, cbt->ref, &start, &stop); + if (stop.txnid != WT_TXN_MAX && stop.timestamp != WT_TS_MAX) + rollback = !__wt_txn_visible(session, stop.txnid, stop.timestamp); + else + rollback = !__wt_txn_visible(session, start.txnid, start.timestamp); + } + + if (rollback) { + WT_STAT_CONN_INCR(session, txn_update_conflict); + WT_STAT_DATA_INCR(session, txn_update_conflict); + ret = __wt_txn_rollback_required(session, "conflict between concurrent operations"); + } + if (ignore_prepare_set) F_SET(txn, WT_TXN_IGNORE_PREPARE); - return (0); + return (ret); } /* @@ -1102,7 +1196,7 @@ __wt_txn_activity_check(WT_SESSION_IMPL *session, bool *txn_active) /* * Default to true - callers shouldn't rely on this if an error is returned, but let's give them - * deterministic behaviour if they do. + * deterministic behavior if they do. */ *txn_active = true; diff --git a/src/third_party/wiredtiger/src/include/wiredtiger.in b/src/third_party/wiredtiger/src/include/wiredtiger.in index d0fb48c1c9a..63aac197478 100644 --- a/src/third_party/wiredtiger/src/include/wiredtiger.in +++ b/src/third_party/wiredtiger/src/include/wiredtiger.in @@ -1693,14 +1693,18 @@ struct __wt_session { * @exclusive * * @param session the session handle - * @param name the URI of the table or file to verify + * @param name the URI of the table or file to verify, optional if verifying the history + * store * @configstart{WT_SESSION.verify, see dist/api_data.py} - * @config{dump_address, Display addresses and page types as pages are verified\, using the - * application's message handler\, intended for debugging., a boolean flag; default \c - * false.} + * @config{dump_address, Display page addresses\, start and stop time pairs and page types + * as pages are verified\, using the application's message handler\, intended for + * debugging., a boolean flag; default \c false.} * @config{dump_blocks, Display the contents of on-disk blocks as they are verified\, using * the application's message handler\, intended for debugging., a boolean flag; default \c * false.} + * @config{dump_history, Display a key's values along with its start and stop time pairs as + * they are verified against the history store\, using the application's message handler\, + * intended for debugging., a boolean flag; default \c false.} * @config{dump_layout, Display the layout of the files as they are verified\, using the * application's message handler\, intended for debugging; requires optional support from * the block manager., a boolean flag; default \c false.} @@ -1710,6 +1714,9 @@ struct __wt_session { * @config{dump_pages, Display the contents of in-memory pages as they are verified\, using * the application's message handler\, intended for debugging., a boolean flag; default \c * false.} + * @config{history_store, Verify the history store., a boolean flag; default \c false.} + * @config{stable_timestamp, Ensure that no data has a start timestamp after the stable + * timestamp\, to be run after rollback_to_stable., a boolean flag; default \c false.} * @config{strict, Treat any verification problem as an error; by default\, verify will * warn\, but not fail\, in the case of errors that won't affect future behavior (for * example\, a leaked block)., a boolean flag; default \c false.} @@ -1776,8 +1783,6 @@ struct __wt_session { * oldest timestamp\, the read timestamp will be rounded up to the oldest timestamp., a * boolean flag; default \c false.} * @config{ ),,} - * @config{snapshot, use a named\, in-memory snapshot\, see @ref - * transaction_named_snapshots., a string; default empty.} * @config{sync, whether to sync log records when the transaction commits\, inherited from * ::wiredtiger_open \c transaction_sync., a boolean flag; default empty.} * @configend @@ -1957,36 +1962,6 @@ struct __wt_session { int __F(checkpoint)(WT_SESSION *session, const char *config); /*! - * Manage named snapshot transactions. Use this API to create and drop - * named snapshots. Named snapshot transactions can be accessed via - * WT_CURSOR::open. See @ref transaction_named_snapshots. - * - * @snippet ex_all.c Snapshot examples - * - * @param session the session handle - * @configstart{WT_SESSION.snapshot, see dist/api_data.py} - * @config{drop = (, if non-empty\, specifies which snapshots to drop. Where a group of - * snapshots are being dropped\, the order is based on snapshot creation order not - * alphanumeric name order., a set of related configuration options defined below.} - * @config{ all, drop all named snapshots., a boolean flag; default - * \c false.} - * @config{ before, drop all snapshots up to but not - * including the specified name., a string; default empty.} - * @config{ - * names, drop specific named snapshots., a list of strings; default empty.} - * @config{ to, drop all snapshots up to and including the specified - * name., a string; default empty.} - * @config{ ),,} - * @config{include_updates, make updates from the current transaction visible to users of - * the named snapshot. Transactions started with such a named snapshot are restricted to - * being read-only., a boolean flag; default \c false.} - * @config{name, specify a name for the snapshot., a string; default empty.} - * @configend - * @errors - */ - int __F(snapshot)(WT_SESSION *session, const char *config); - - /*! * Return the transaction ID range pinned by the session handle. * * The ID range is approximate and is calculated based on the oldest @@ -2208,9 +2183,9 @@ struct __wt_connection { * to detect inappropriate references to memory owned by cursors., a boolean flag; default * \c false.} * @config{ eviction, if true\, modify internal algorithms - * to change skew to force lookaside eviction to happen more aggressively. This includes - * but is not limited to not skewing newest\, not favoring leaf pages\, and modifying the - * eviction score mechanism., a boolean flag; default \c false.} + * to change skew to force history store eviction to happen more aggressively. This + * includes but is not limited to not skewing newest\, not favoring leaf pages\, and + * modifying the eviction score mechanism., a boolean flag; default \c false.} * @config{ realloc_exact, if true\, reallocation of memory will only * provide the exact amount requested. This will help with spotting memory allocation * issues more easily., a boolean flag; default \c false.} @@ -2271,6 +2246,15 @@ struct __wt_connection { * check for files that are inactive and close them., an integer between 1 and 100000; * default \c 10.} * @config{ ),,} + * @config{history_store = (, history store configuration options., a set of related + * configuration options defined below.} + * @config{ file_max, The + * maximum number of bytes that WiredTiger is allowed to use for its history store + * mechanism. If the history store file exceeds this size\, a panic will be triggered. The + * default value means that the history store file is unbounded and may use as much space as + * the filesystem will accommodate. The minimum non-zero setting is 100MB., an integer + * greater than or equal to 0; default \c 0.} + * @config{ ),,} * @config{io_capacity = (, control how many bytes per second are written and read. * Exceeding the capacity results in throttling., a set of related configuration options * defined below.} @@ -2373,13 +2357,13 @@ struct __wt_connection { * @config{verbose, enable messages for various events. Options are given as a list\, such * as <code>"verbose=[evictserver\,read]"</code>., a list\, with values chosen from the * following options: \c "api"\, \c "backup"\, \c "block"\, \c "checkpoint"\, \c - * "checkpoint_progress"\, \c "compact"\, \c "compact_progress"\, \c "error_returns"\, \c - * "evict"\, \c "evict_stuck"\, \c "evictserver"\, \c "fileops"\, \c "handleops"\, \c - * "log"\, \c "lookaside"\, \c "lookaside_activity"\, \c "lsm"\, \c "lsm_manager"\, \c - * "metadata"\, \c "mutex"\, \c "overflow"\, \c "read"\, \c "rebalance"\, \c "reconcile"\, - * \c "recovery"\, \c "recovery_progress"\, \c "salvage"\, \c "shared_cache"\, \c "split"\, - * \c "temporary"\, \c "thread_group"\, \c "timestamp"\, \c "transaction"\, \c "verify"\, \c - * "version"\, \c "write"; default empty.} + * "checkpoint_gc"\, \c "checkpoint_progress"\, \c "compact"\, \c "compact_progress"\, \c + * "error_returns"\, \c "evict"\, \c "evict_stuck"\, \c "evictserver"\, \c "fileops"\, \c + * "handleops"\, \c "log"\, \c "history_store"\, \c "history_store_activity"\, \c "lsm"\, \c + * "lsm_manager"\, \c "metadata"\, \c "mutex"\, \c "overflow"\, \c "read"\, \c "rebalance"\, + * \c "reconcile"\, \c "recovery"\, \c "recovery_progress"\, \c "rts"\, \c "salvage"\, \c + * "shared_cache"\, \c "split"\, \c "temporary"\, \c "thread_group"\, \c "timestamp"\, \c + * "transaction"\, \c "verify"\, \c "version"\, \c "write"; default empty.} * @configend * @errors */ @@ -2816,7 +2800,7 @@ struct __wt_connection { * on the next cursor operation. This allows memory sanitizers to detect inappropriate references * to memory owned by cursors., a boolean flag; default \c false.} * @config{ - * eviction, if true\, modify internal algorithms to change skew to force lookaside eviction to + * eviction, if true\, modify internal algorithms to change skew to force history store eviction to * happen more aggressively. This includes but is not limited to not skewing newest\, not favoring * leaf pages\, and modifying the eviction score mechanism., a boolean flag; default \c false.} * @config{ realloc_exact, if true\, reallocation of memory will only provide @@ -2918,6 +2902,14 @@ struct __wt_connection { * @config{ close_scan_interval, interval in seconds at which to check for * files that are inactive and close them., an integer between 1 and 100000; default \c 10.} * @config{ ),,} + * @config{history_store = (, history store configuration options., a set of related configuration + * options defined below.} + * @config{ file_max, The maximum number of bytes + * that WiredTiger is allowed to use for its history store mechanism. If the history store file + * exceeds this size\, a panic will be triggered. The default value means that the history store + * file is unbounded and may use as much space as the filesystem will accommodate. The minimum + * non-zero setting is 100MB., an integer greater than or equal to 0; default \c 0.} + * @config{ ),,} * @config{in_memory, keep data in-memory only. See @ref in_memory for more information., a boolean * flag; default \c false.} * @config{io_capacity = (, control how many bytes per second are written and read. Exceeding the @@ -3062,13 +3054,14 @@ struct __wt_connection { * information., a boolean flag; default \c false.} * @config{verbose, enable messages for various events. Options are given as a list\, such as * <code>"verbose=[evictserver\,read]"</code>., a list\, with values chosen from the following - * options: \c "api"\, \c "backup"\, \c "block"\, \c "checkpoint"\, \c "checkpoint_progress"\, \c - * "compact"\, \c "compact_progress"\, \c "error_returns"\, \c "evict"\, \c "evict_stuck"\, \c - * "evictserver"\, \c "fileops"\, \c "handleops"\, \c "log"\, \c "lookaside"\, \c - * "lookaside_activity"\, \c "lsm"\, \c "lsm_manager"\, \c "metadata"\, \c "mutex"\, \c "overflow"\, - * \c "read"\, \c "rebalance"\, \c "reconcile"\, \c "recovery"\, \c "recovery_progress"\, \c - * "salvage"\, \c "shared_cache"\, \c "split"\, \c "temporary"\, \c "thread_group"\, \c - * "timestamp"\, \c "transaction"\, \c "verify"\, \c "version"\, \c "write"; default empty.} + * options: \c "api"\, \c "backup"\, \c "block"\, \c "checkpoint"\, \c "checkpoint_gc"\, \c + * "checkpoint_progress"\, \c "compact"\, \c "compact_progress"\, \c "error_returns"\, \c "evict"\, + * \c "evict_stuck"\, \c "evictserver"\, \c "fileops"\, \c "handleops"\, \c "log"\, \c + * "history_store"\, \c "history_store_activity"\, \c "lsm"\, \c "lsm_manager"\, \c "metadata"\, \c + * "mutex"\, \c "overflow"\, \c "read"\, \c "rebalance"\, \c "reconcile"\, \c "recovery"\, \c + * "recovery_progress"\, \c "rts"\, \c "salvage"\, \c "shared_cache"\, \c "split"\, \c "temporary"\, + * \c "thread_group"\, \c "timestamp"\, \c "transaction"\, \c "verify"\, \c "version"\, \c "write"; + * default empty.} * @config{write_through, Use \c FILE_FLAG_WRITE_THROUGH on Windows to write to files. Ignored on * non-Windows systems. Options are given as a list\, such as <code>"write_through=[data]"</code>. * Configuring \c write_through requires care\, see @ref tuning_system_buffer_cache_direct_io for @@ -4984,8 +4977,8 @@ extern int wiredtiger_extension_terminate(WT_CONNECTION *connection); #define WT_STAT_CONN_CACHE_WRITE_APP_TIME 1034 /*! cache: bytes belonging to page images in the cache */ #define WT_STAT_CONN_CACHE_BYTES_IMAGE 1035 -/*! cache: bytes belonging to the cache overflow table in the cache */ -#define WT_STAT_CONN_CACHE_BYTES_LOOKASIDE 1036 +/*! cache: bytes belonging to the history store table in the cache */ +#define WT_STAT_CONN_CACHE_BYTES_HS 1036 /*! cache: bytes currently in the cache */ #define WT_STAT_CONN_CACHE_BYTES_INUSE 1037 /*! cache: bytes dirty in the cache cumulative */ @@ -4996,812 +4989,823 @@ extern int wiredtiger_extension_terminate(WT_CONNECTION *connection); #define WT_STAT_CONN_CACHE_BYTES_READ 1040 /*! cache: bytes written from cache */ #define WT_STAT_CONN_CACHE_BYTES_WRITE 1041 -/*! cache: cache overflow cursor application thread wait time (usecs) */ -#define WT_STAT_CONN_CACHE_LOOKASIDE_CURSOR_WAIT_APPLICATION 1042 -/*! cache: cache overflow cursor internal thread wait time (usecs) */ -#define WT_STAT_CONN_CACHE_LOOKASIDE_CURSOR_WAIT_INTERNAL 1043 /*! cache: cache overflow score */ -#define WT_STAT_CONN_CACHE_LOOKASIDE_SCORE 1044 -/*! cache: cache overflow table entries */ -#define WT_STAT_CONN_CACHE_LOOKASIDE_ENTRIES 1045 -/*! cache: cache overflow table insert calls */ -#define WT_STAT_CONN_CACHE_LOOKASIDE_INSERT 1046 -/*! cache: cache overflow table max on-disk size */ -#define WT_STAT_CONN_CACHE_LOOKASIDE_ONDISK_MAX 1047 -/*! cache: cache overflow table on-disk size */ -#define WT_STAT_CONN_CACHE_LOOKASIDE_ONDISK 1048 -/*! cache: cache overflow table remove calls */ -#define WT_STAT_CONN_CACHE_LOOKASIDE_REMOVE 1049 +#define WT_STAT_CONN_CACHE_LOOKASIDE_SCORE 1042 /*! cache: checkpoint blocked page eviction */ -#define WT_STAT_CONN_CACHE_EVICTION_CHECKPOINT 1050 +#define WT_STAT_CONN_CACHE_EVICTION_CHECKPOINT 1043 /*! cache: eviction calls to get a page */ -#define WT_STAT_CONN_CACHE_EVICTION_GET_REF 1051 +#define WT_STAT_CONN_CACHE_EVICTION_GET_REF 1044 /*! cache: eviction calls to get a page found queue empty */ -#define WT_STAT_CONN_CACHE_EVICTION_GET_REF_EMPTY 1052 +#define WT_STAT_CONN_CACHE_EVICTION_GET_REF_EMPTY 1045 /*! cache: eviction calls to get a page found queue empty after locking */ -#define WT_STAT_CONN_CACHE_EVICTION_GET_REF_EMPTY2 1053 +#define WT_STAT_CONN_CACHE_EVICTION_GET_REF_EMPTY2 1046 /*! cache: eviction currently operating in aggressive mode */ -#define WT_STAT_CONN_CACHE_EVICTION_AGGRESSIVE_SET 1054 +#define WT_STAT_CONN_CACHE_EVICTION_AGGRESSIVE_SET 1047 /*! cache: eviction empty score */ -#define WT_STAT_CONN_CACHE_EVICTION_EMPTY_SCORE 1055 +#define WT_STAT_CONN_CACHE_EVICTION_EMPTY_SCORE 1048 /*! cache: eviction passes of a file */ -#define WT_STAT_CONN_CACHE_EVICTION_WALK_PASSES 1056 +#define WT_STAT_CONN_CACHE_EVICTION_WALK_PASSES 1049 /*! cache: eviction server candidate queue empty when topping up */ -#define WT_STAT_CONN_CACHE_EVICTION_QUEUE_EMPTY 1057 +#define WT_STAT_CONN_CACHE_EVICTION_QUEUE_EMPTY 1050 /*! cache: eviction server candidate queue not empty when topping up */ -#define WT_STAT_CONN_CACHE_EVICTION_QUEUE_NOT_EMPTY 1058 +#define WT_STAT_CONN_CACHE_EVICTION_QUEUE_NOT_EMPTY 1051 /*! cache: eviction server evicting pages */ -#define WT_STAT_CONN_CACHE_EVICTION_SERVER_EVICTING 1059 +#define WT_STAT_CONN_CACHE_EVICTION_SERVER_EVICTING 1052 /*! * cache: eviction server slept, because we did not make progress with * eviction */ -#define WT_STAT_CONN_CACHE_EVICTION_SERVER_SLEPT 1060 +#define WT_STAT_CONN_CACHE_EVICTION_SERVER_SLEPT 1053 /*! cache: eviction server unable to reach eviction goal */ -#define WT_STAT_CONN_CACHE_EVICTION_SLOW 1061 +#define WT_STAT_CONN_CACHE_EVICTION_SLOW 1054 /*! cache: eviction server waiting for a leaf page */ -#define WT_STAT_CONN_CACHE_EVICTION_WALK_LEAF_NOTFOUND 1062 +#define WT_STAT_CONN_CACHE_EVICTION_WALK_LEAF_NOTFOUND 1055 /*! cache: eviction state */ -#define WT_STAT_CONN_CACHE_EVICTION_STATE 1063 +#define WT_STAT_CONN_CACHE_EVICTION_STATE 1056 /*! cache: eviction walk target pages histogram - 0-9 */ -#define WT_STAT_CONN_CACHE_EVICTION_TARGET_PAGE_LT10 1064 +#define WT_STAT_CONN_CACHE_EVICTION_TARGET_PAGE_LT10 1057 /*! cache: eviction walk target pages histogram - 10-31 */ -#define WT_STAT_CONN_CACHE_EVICTION_TARGET_PAGE_LT32 1065 +#define WT_STAT_CONN_CACHE_EVICTION_TARGET_PAGE_LT32 1058 /*! cache: eviction walk target pages histogram - 128 and higher */ -#define WT_STAT_CONN_CACHE_EVICTION_TARGET_PAGE_GE128 1066 +#define WT_STAT_CONN_CACHE_EVICTION_TARGET_PAGE_GE128 1059 /*! cache: eviction walk target pages histogram - 32-63 */ -#define WT_STAT_CONN_CACHE_EVICTION_TARGET_PAGE_LT64 1067 +#define WT_STAT_CONN_CACHE_EVICTION_TARGET_PAGE_LT64 1060 /*! cache: eviction walk target pages histogram - 64-128 */ -#define WT_STAT_CONN_CACHE_EVICTION_TARGET_PAGE_LT128 1068 +#define WT_STAT_CONN_CACHE_EVICTION_TARGET_PAGE_LT128 1061 /*! cache: eviction walk target strategy both clean and dirty pages */ -#define WT_STAT_CONN_CACHE_EVICTION_TARGET_STRATEGY_BOTH_CLEAN_AND_DIRTY 1069 +#define WT_STAT_CONN_CACHE_EVICTION_TARGET_STRATEGY_BOTH_CLEAN_AND_DIRTY 1062 /*! cache: eviction walk target strategy only clean pages */ -#define WT_STAT_CONN_CACHE_EVICTION_TARGET_STRATEGY_CLEAN 1070 +#define WT_STAT_CONN_CACHE_EVICTION_TARGET_STRATEGY_CLEAN 1063 /*! cache: eviction walk target strategy only dirty pages */ -#define WT_STAT_CONN_CACHE_EVICTION_TARGET_STRATEGY_DIRTY 1071 +#define WT_STAT_CONN_CACHE_EVICTION_TARGET_STRATEGY_DIRTY 1064 /*! cache: eviction walks abandoned */ -#define WT_STAT_CONN_CACHE_EVICTION_WALKS_ABANDONED 1072 +#define WT_STAT_CONN_CACHE_EVICTION_WALKS_ABANDONED 1065 /*! cache: eviction walks gave up because they restarted their walk twice */ -#define WT_STAT_CONN_CACHE_EVICTION_WALKS_STOPPED 1073 +#define WT_STAT_CONN_CACHE_EVICTION_WALKS_STOPPED 1066 /*! * cache: eviction walks gave up because they saw too many pages and * found no candidates */ -#define WT_STAT_CONN_CACHE_EVICTION_WALKS_GAVE_UP_NO_TARGETS 1074 +#define WT_STAT_CONN_CACHE_EVICTION_WALKS_GAVE_UP_NO_TARGETS 1067 /*! * cache: eviction walks gave up because they saw too many pages and * found too few candidates */ -#define WT_STAT_CONN_CACHE_EVICTION_WALKS_GAVE_UP_RATIO 1075 +#define WT_STAT_CONN_CACHE_EVICTION_WALKS_GAVE_UP_RATIO 1068 /*! cache: eviction walks reached end of tree */ -#define WT_STAT_CONN_CACHE_EVICTION_WALKS_ENDED 1076 +#define WT_STAT_CONN_CACHE_EVICTION_WALKS_ENDED 1069 /*! cache: eviction walks started from root of tree */ -#define WT_STAT_CONN_CACHE_EVICTION_WALK_FROM_ROOT 1077 +#define WT_STAT_CONN_CACHE_EVICTION_WALK_FROM_ROOT 1070 /*! cache: eviction walks started from saved location in tree */ -#define WT_STAT_CONN_CACHE_EVICTION_WALK_SAVED_POS 1078 +#define WT_STAT_CONN_CACHE_EVICTION_WALK_SAVED_POS 1071 /*! cache: eviction worker thread active */ -#define WT_STAT_CONN_CACHE_EVICTION_ACTIVE_WORKERS 1079 +#define WT_STAT_CONN_CACHE_EVICTION_ACTIVE_WORKERS 1072 /*! cache: eviction worker thread created */ -#define WT_STAT_CONN_CACHE_EVICTION_WORKER_CREATED 1080 +#define WT_STAT_CONN_CACHE_EVICTION_WORKER_CREATED 1073 /*! cache: eviction worker thread evicting pages */ -#define WT_STAT_CONN_CACHE_EVICTION_WORKER_EVICTING 1081 +#define WT_STAT_CONN_CACHE_EVICTION_WORKER_EVICTING 1074 /*! cache: eviction worker thread removed */ -#define WT_STAT_CONN_CACHE_EVICTION_WORKER_REMOVED 1082 +#define WT_STAT_CONN_CACHE_EVICTION_WORKER_REMOVED 1075 /*! cache: eviction worker thread stable number */ -#define WT_STAT_CONN_CACHE_EVICTION_STABLE_STATE_WORKERS 1083 +#define WT_STAT_CONN_CACHE_EVICTION_STABLE_STATE_WORKERS 1076 /*! cache: files with active eviction walks */ -#define WT_STAT_CONN_CACHE_EVICTION_WALKS_ACTIVE 1084 +#define WT_STAT_CONN_CACHE_EVICTION_WALKS_ACTIVE 1077 /*! cache: files with new eviction walks started */ -#define WT_STAT_CONN_CACHE_EVICTION_WALKS_STARTED 1085 +#define WT_STAT_CONN_CACHE_EVICTION_WALKS_STARTED 1078 /*! cache: force re-tuning of eviction workers once in a while */ -#define WT_STAT_CONN_CACHE_EVICTION_FORCE_RETUNE 1086 +#define WT_STAT_CONN_CACHE_EVICTION_FORCE_RETUNE 1079 /*! cache: forced eviction - pages evicted that were clean count */ -#define WT_STAT_CONN_CACHE_EVICTION_FORCE_CLEAN 1087 +#define WT_STAT_CONN_CACHE_EVICTION_FORCE_CLEAN 1080 /*! cache: forced eviction - pages evicted that were clean time (usecs) */ -#define WT_STAT_CONN_CACHE_EVICTION_FORCE_CLEAN_TIME 1088 +#define WT_STAT_CONN_CACHE_EVICTION_FORCE_CLEAN_TIME 1081 /*! cache: forced eviction - pages evicted that were dirty count */ -#define WT_STAT_CONN_CACHE_EVICTION_FORCE_DIRTY 1089 +#define WT_STAT_CONN_CACHE_EVICTION_FORCE_DIRTY 1082 /*! cache: forced eviction - pages evicted that were dirty time (usecs) */ -#define WT_STAT_CONN_CACHE_EVICTION_FORCE_DIRTY_TIME 1090 +#define WT_STAT_CONN_CACHE_EVICTION_FORCE_DIRTY_TIME 1083 /*! * cache: forced eviction - pages selected because of too many deleted * items count */ -#define WT_STAT_CONN_CACHE_EVICTION_FORCE_DELETE 1091 +#define WT_STAT_CONN_CACHE_EVICTION_FORCE_DELETE 1084 /*! cache: forced eviction - pages selected count */ -#define WT_STAT_CONN_CACHE_EVICTION_FORCE 1092 +#define WT_STAT_CONN_CACHE_EVICTION_FORCE 1085 /*! cache: forced eviction - pages selected unable to be evicted count */ -#define WT_STAT_CONN_CACHE_EVICTION_FORCE_FAIL 1093 +#define WT_STAT_CONN_CACHE_EVICTION_FORCE_FAIL 1086 /*! cache: forced eviction - pages selected unable to be evicted time */ -#define WT_STAT_CONN_CACHE_EVICTION_FORCE_FAIL_TIME 1094 +#define WT_STAT_CONN_CACHE_EVICTION_FORCE_FAIL_TIME 1087 /*! cache: hazard pointer blocked page eviction */ -#define WT_STAT_CONN_CACHE_EVICTION_HAZARD 1095 +#define WT_STAT_CONN_CACHE_EVICTION_HAZARD 1088 /*! cache: hazard pointer check calls */ -#define WT_STAT_CONN_CACHE_HAZARD_CHECKS 1096 +#define WT_STAT_CONN_CACHE_HAZARD_CHECKS 1089 /*! cache: hazard pointer check entries walked */ -#define WT_STAT_CONN_CACHE_HAZARD_WALKS 1097 +#define WT_STAT_CONN_CACHE_HAZARD_WALKS 1090 /*! cache: hazard pointer maximum array length */ -#define WT_STAT_CONN_CACHE_HAZARD_MAX 1098 +#define WT_STAT_CONN_CACHE_HAZARD_MAX 1091 +/*! cache: history store key truncation due to mixed timestamps */ +#define WT_STAT_CONN_CACHE_HS_KEY_TRUNCATE_MIX_TS 1092 +/*! + * cache: history store key truncation due to the key being removed from + * the data page + */ +#define WT_STAT_CONN_CACHE_HS_KEY_TRUNCATE_ONPAGE_REMOVAL 1093 +/*! cache: history store score */ +#define WT_STAT_CONN_CACHE_HS_SCORE 1094 +/*! cache: history store table insert calls */ +#define WT_STAT_CONN_CACHE_HS_INSERT 1095 +/*! cache: history store table max on-disk size */ +#define WT_STAT_CONN_CACHE_HS_ONDISK_MAX 1096 +/*! cache: history store table on-disk size */ +#define WT_STAT_CONN_CACHE_HS_ONDISK 1097 +/*! cache: history store table reads */ +#define WT_STAT_CONN_CACHE_HS_READ 1098 +/*! cache: history store table reads missed */ +#define WT_STAT_CONN_CACHE_HS_READ_MISS 1099 +/*! cache: history store table reads requiring squashed modifies */ +#define WT_STAT_CONN_CACHE_HS_READ_SQUASH 1100 +/*! cache: history store table remove calls due to key truncation */ +#define WT_STAT_CONN_CACHE_HS_REMOVE_KEY_TRUNCATE 1101 +/*! cache: history store table writes requiring squashed modifies */ +#define WT_STAT_CONN_CACHE_HS_WRITE_SQUASH 1102 /*! cache: in-memory page passed criteria to be split */ -#define WT_STAT_CONN_CACHE_INMEM_SPLITTABLE 1099 +#define WT_STAT_CONN_CACHE_INMEM_SPLITTABLE 1103 /*! cache: in-memory page splits */ -#define WT_STAT_CONN_CACHE_INMEM_SPLIT 1100 +#define WT_STAT_CONN_CACHE_INMEM_SPLIT 1104 /*! cache: internal pages evicted */ -#define WT_STAT_CONN_CACHE_EVICTION_INTERNAL 1101 +#define WT_STAT_CONN_CACHE_EVICTION_INTERNAL 1105 /*! cache: internal pages queued for eviction */ -#define WT_STAT_CONN_CACHE_EVICTION_INTERNAL_PAGES_QUEUED 1102 +#define WT_STAT_CONN_CACHE_EVICTION_INTERNAL_PAGES_QUEUED 1106 /*! cache: internal pages seen by eviction walk */ -#define WT_STAT_CONN_CACHE_EVICTION_INTERNAL_PAGES_SEEN 1103 +#define WT_STAT_CONN_CACHE_EVICTION_INTERNAL_PAGES_SEEN 1107 /*! cache: internal pages seen by eviction walk that are already queued */ -#define WT_STAT_CONN_CACHE_EVICTION_INTERNAL_PAGES_ALREADY_QUEUED 1104 +#define WT_STAT_CONN_CACHE_EVICTION_INTERNAL_PAGES_ALREADY_QUEUED 1108 /*! cache: internal pages split during eviction */ -#define WT_STAT_CONN_CACHE_EVICTION_SPLIT_INTERNAL 1105 +#define WT_STAT_CONN_CACHE_EVICTION_SPLIT_INTERNAL 1109 /*! cache: leaf pages split during eviction */ -#define WT_STAT_CONN_CACHE_EVICTION_SPLIT_LEAF 1106 +#define WT_STAT_CONN_CACHE_EVICTION_SPLIT_LEAF 1110 /*! cache: maximum bytes configured */ -#define WT_STAT_CONN_CACHE_BYTES_MAX 1107 +#define WT_STAT_CONN_CACHE_BYTES_MAX 1111 /*! cache: maximum page size at eviction */ -#define WT_STAT_CONN_CACHE_EVICTION_MAXIMUM_PAGE_SIZE 1108 +#define WT_STAT_CONN_CACHE_EVICTION_MAXIMUM_PAGE_SIZE 1112 /*! cache: modified pages evicted */ -#define WT_STAT_CONN_CACHE_EVICTION_DIRTY 1109 +#define WT_STAT_CONN_CACHE_EVICTION_DIRTY 1113 /*! cache: modified pages evicted by application threads */ -#define WT_STAT_CONN_CACHE_EVICTION_APP_DIRTY 1110 +#define WT_STAT_CONN_CACHE_EVICTION_APP_DIRTY 1114 /*! cache: operations timed out waiting for space in cache */ -#define WT_STAT_CONN_CACHE_TIMED_OUT_OPS 1111 +#define WT_STAT_CONN_CACHE_TIMED_OUT_OPS 1115 /*! cache: overflow pages read into cache */ -#define WT_STAT_CONN_CACHE_READ_OVERFLOW 1112 +#define WT_STAT_CONN_CACHE_READ_OVERFLOW 1116 /*! cache: page split during eviction deepened the tree */ -#define WT_STAT_CONN_CACHE_EVICTION_DEEPEN 1113 -/*! cache: page written requiring cache overflow records */ -#define WT_STAT_CONN_CACHE_WRITE_LOOKASIDE 1114 +#define WT_STAT_CONN_CACHE_EVICTION_DEEPEN 1117 +/*! cache: page written requiring history store records */ +#define WT_STAT_CONN_CACHE_WRITE_HS 1118 /*! cache: pages currently held in the cache */ -#define WT_STAT_CONN_CACHE_PAGES_INUSE 1115 +#define WT_STAT_CONN_CACHE_PAGES_INUSE 1119 /*! cache: pages evicted by application threads */ -#define WT_STAT_CONN_CACHE_EVICTION_APP 1116 +#define WT_STAT_CONN_CACHE_EVICTION_APP 1120 /*! cache: pages queued for eviction */ -#define WT_STAT_CONN_CACHE_EVICTION_PAGES_QUEUED 1117 +#define WT_STAT_CONN_CACHE_EVICTION_PAGES_QUEUED 1121 /*! cache: pages queued for eviction post lru sorting */ -#define WT_STAT_CONN_CACHE_EVICTION_PAGES_QUEUED_POST_LRU 1118 +#define WT_STAT_CONN_CACHE_EVICTION_PAGES_QUEUED_POST_LRU 1122 /*! cache: pages queued for urgent eviction */ -#define WT_STAT_CONN_CACHE_EVICTION_PAGES_QUEUED_URGENT 1119 +#define WT_STAT_CONN_CACHE_EVICTION_PAGES_QUEUED_URGENT 1123 /*! cache: pages queued for urgent eviction during walk */ -#define WT_STAT_CONN_CACHE_EVICTION_PAGES_QUEUED_OLDEST 1120 +#define WT_STAT_CONN_CACHE_EVICTION_PAGES_QUEUED_OLDEST 1124 /*! cache: pages read into cache */ -#define WT_STAT_CONN_CACHE_READ 1121 +#define WT_STAT_CONN_CACHE_READ 1125 /*! cache: pages read into cache after truncate */ -#define WT_STAT_CONN_CACHE_READ_DELETED 1122 +#define WT_STAT_CONN_CACHE_READ_DELETED 1126 /*! cache: pages read into cache after truncate in prepare state */ -#define WT_STAT_CONN_CACHE_READ_DELETED_PREPARED 1123 -/*! cache: pages read into cache requiring cache overflow entries */ -#define WT_STAT_CONN_CACHE_READ_LOOKASIDE 1124 -/*! cache: pages read into cache requiring cache overflow for checkpoint */ -#define WT_STAT_CONN_CACHE_READ_LOOKASIDE_CHECKPOINT 1125 -/*! cache: pages read into cache skipping older cache overflow entries */ -#define WT_STAT_CONN_CACHE_READ_LOOKASIDE_SKIPPED 1126 -/*! - * cache: pages read into cache with skipped cache overflow entries - * needed later - */ -#define WT_STAT_CONN_CACHE_READ_LOOKASIDE_DELAY 1127 -/*! - * cache: pages read into cache with skipped cache overflow entries - * needed later by checkpoint - */ -#define WT_STAT_CONN_CACHE_READ_LOOKASIDE_DELAY_CHECKPOINT 1128 +#define WT_STAT_CONN_CACHE_READ_DELETED_PREPARED 1127 /*! cache: pages requested from the cache */ -#define WT_STAT_CONN_CACHE_PAGES_REQUESTED 1129 +#define WT_STAT_CONN_CACHE_PAGES_REQUESTED 1128 /*! cache: pages seen by eviction walk */ -#define WT_STAT_CONN_CACHE_EVICTION_PAGES_SEEN 1130 +#define WT_STAT_CONN_CACHE_EVICTION_PAGES_SEEN 1129 /*! cache: pages seen by eviction walk that are already queued */ -#define WT_STAT_CONN_CACHE_EVICTION_PAGES_ALREADY_QUEUED 1131 +#define WT_STAT_CONN_CACHE_EVICTION_PAGES_ALREADY_QUEUED 1130 /*! cache: pages selected for eviction unable to be evicted */ -#define WT_STAT_CONN_CACHE_EVICTION_FAIL 1132 +#define WT_STAT_CONN_CACHE_EVICTION_FAIL 1131 /*! * cache: pages selected for eviction unable to be evicted as the parent * page has overflow items */ -#define WT_STAT_CONN_CACHE_EVICTION_FAIL_PARENT_HAS_OVERFLOW_ITEMS 1133 +#define WT_STAT_CONN_CACHE_EVICTION_FAIL_PARENT_HAS_OVERFLOW_ITEMS 1132 /*! * cache: pages selected for eviction unable to be evicted because of * active children on an internal page */ -#define WT_STAT_CONN_CACHE_EVICTION_FAIL_ACTIVE_CHILDREN_ON_AN_INTERNAL_PAGE 1134 +#define WT_STAT_CONN_CACHE_EVICTION_FAIL_ACTIVE_CHILDREN_ON_AN_INTERNAL_PAGE 1133 /*! * cache: pages selected for eviction unable to be evicted because of * failure in reconciliation */ -#define WT_STAT_CONN_CACHE_EVICTION_FAIL_IN_RECONCILIATION 1135 +#define WT_STAT_CONN_CACHE_EVICTION_FAIL_IN_RECONCILIATION 1134 /*! * cache: pages selected for eviction unable to be evicted due to newer * modifications on a clean page */ -#define WT_STAT_CONN_CACHE_EVICTION_FAIL_WITH_NEWER_MODIFICATIONS_ON_A_CLEAN_PAGE 1136 +#define WT_STAT_CONN_CACHE_EVICTION_FAIL_WITH_NEWER_MODIFICATIONS_ON_A_CLEAN_PAGE 1135 /*! cache: pages walked for eviction */ -#define WT_STAT_CONN_CACHE_EVICTION_WALK 1137 +#define WT_STAT_CONN_CACHE_EVICTION_WALK 1136 /*! cache: pages written from cache */ -#define WT_STAT_CONN_CACHE_WRITE 1138 +#define WT_STAT_CONN_CACHE_WRITE 1137 /*! cache: pages written requiring in-memory restoration */ -#define WT_STAT_CONN_CACHE_WRITE_RESTORE 1139 +#define WT_STAT_CONN_CACHE_WRITE_RESTORE 1138 /*! cache: percentage overhead */ -#define WT_STAT_CONN_CACHE_OVERHEAD 1140 +#define WT_STAT_CONN_CACHE_OVERHEAD 1139 /*! cache: tracked bytes belonging to internal pages in the cache */ -#define WT_STAT_CONN_CACHE_BYTES_INTERNAL 1141 +#define WT_STAT_CONN_CACHE_BYTES_INTERNAL 1140 /*! cache: tracked bytes belonging to leaf pages in the cache */ -#define WT_STAT_CONN_CACHE_BYTES_LEAF 1142 +#define WT_STAT_CONN_CACHE_BYTES_LEAF 1141 /*! cache: tracked dirty bytes in the cache */ -#define WT_STAT_CONN_CACHE_BYTES_DIRTY 1143 +#define WT_STAT_CONN_CACHE_BYTES_DIRTY 1142 /*! cache: tracked dirty pages in the cache */ -#define WT_STAT_CONN_CACHE_PAGES_DIRTY 1144 +#define WT_STAT_CONN_CACHE_PAGES_DIRTY 1143 /*! cache: unmodified pages evicted */ -#define WT_STAT_CONN_CACHE_EVICTION_CLEAN 1145 +#define WT_STAT_CONN_CACHE_EVICTION_CLEAN 1144 /*! capacity: background fsync file handles considered */ -#define WT_STAT_CONN_FSYNC_ALL_FH_TOTAL 1146 +#define WT_STAT_CONN_FSYNC_ALL_FH_TOTAL 1145 /*! capacity: background fsync file handles synced */ -#define WT_STAT_CONN_FSYNC_ALL_FH 1147 +#define WT_STAT_CONN_FSYNC_ALL_FH 1146 /*! capacity: background fsync time (msecs) */ -#define WT_STAT_CONN_FSYNC_ALL_TIME 1148 +#define WT_STAT_CONN_FSYNC_ALL_TIME 1147 /*! capacity: bytes read */ -#define WT_STAT_CONN_CAPACITY_BYTES_READ 1149 +#define WT_STAT_CONN_CAPACITY_BYTES_READ 1148 /*! capacity: bytes written for checkpoint */ -#define WT_STAT_CONN_CAPACITY_BYTES_CKPT 1150 +#define WT_STAT_CONN_CAPACITY_BYTES_CKPT 1149 /*! capacity: bytes written for eviction */ -#define WT_STAT_CONN_CAPACITY_BYTES_EVICT 1151 +#define WT_STAT_CONN_CAPACITY_BYTES_EVICT 1150 /*! capacity: bytes written for log */ -#define WT_STAT_CONN_CAPACITY_BYTES_LOG 1152 +#define WT_STAT_CONN_CAPACITY_BYTES_LOG 1151 /*! capacity: bytes written total */ -#define WT_STAT_CONN_CAPACITY_BYTES_WRITTEN 1153 +#define WT_STAT_CONN_CAPACITY_BYTES_WRITTEN 1152 /*! capacity: threshold to call fsync */ -#define WT_STAT_CONN_CAPACITY_THRESHOLD 1154 +#define WT_STAT_CONN_CAPACITY_THRESHOLD 1153 /*! capacity: time waiting due to total capacity (usecs) */ -#define WT_STAT_CONN_CAPACITY_TIME_TOTAL 1155 +#define WT_STAT_CONN_CAPACITY_TIME_TOTAL 1154 /*! capacity: time waiting during checkpoint (usecs) */ -#define WT_STAT_CONN_CAPACITY_TIME_CKPT 1156 +#define WT_STAT_CONN_CAPACITY_TIME_CKPT 1155 /*! capacity: time waiting during eviction (usecs) */ -#define WT_STAT_CONN_CAPACITY_TIME_EVICT 1157 +#define WT_STAT_CONN_CAPACITY_TIME_EVICT 1156 /*! capacity: time waiting during logging (usecs) */ -#define WT_STAT_CONN_CAPACITY_TIME_LOG 1158 +#define WT_STAT_CONN_CAPACITY_TIME_LOG 1157 /*! capacity: time waiting during read (usecs) */ -#define WT_STAT_CONN_CAPACITY_TIME_READ 1159 +#define WT_STAT_CONN_CAPACITY_TIME_READ 1158 /*! connection: auto adjusting condition resets */ -#define WT_STAT_CONN_COND_AUTO_WAIT_RESET 1160 +#define WT_STAT_CONN_COND_AUTO_WAIT_RESET 1159 /*! connection: auto adjusting condition wait calls */ -#define WT_STAT_CONN_COND_AUTO_WAIT 1161 +#define WT_STAT_CONN_COND_AUTO_WAIT 1160 /*! connection: detected system time went backwards */ -#define WT_STAT_CONN_TIME_TRAVEL 1162 +#define WT_STAT_CONN_TIME_TRAVEL 1161 /*! connection: files currently open */ -#define WT_STAT_CONN_FILE_OPEN 1163 +#define WT_STAT_CONN_FILE_OPEN 1162 /*! connection: memory allocations */ -#define WT_STAT_CONN_MEMORY_ALLOCATION 1164 +#define WT_STAT_CONN_MEMORY_ALLOCATION 1163 /*! connection: memory frees */ -#define WT_STAT_CONN_MEMORY_FREE 1165 +#define WT_STAT_CONN_MEMORY_FREE 1164 /*! connection: memory re-allocations */ -#define WT_STAT_CONN_MEMORY_GROW 1166 +#define WT_STAT_CONN_MEMORY_GROW 1165 /*! connection: pthread mutex condition wait calls */ -#define WT_STAT_CONN_COND_WAIT 1167 +#define WT_STAT_CONN_COND_WAIT 1166 /*! connection: pthread mutex shared lock read-lock calls */ -#define WT_STAT_CONN_RWLOCK_READ 1168 +#define WT_STAT_CONN_RWLOCK_READ 1167 /*! connection: pthread mutex shared lock write-lock calls */ -#define WT_STAT_CONN_RWLOCK_WRITE 1169 +#define WT_STAT_CONN_RWLOCK_WRITE 1168 /*! connection: total fsync I/Os */ -#define WT_STAT_CONN_FSYNC_IO 1170 +#define WT_STAT_CONN_FSYNC_IO 1169 /*! connection: total read I/Os */ -#define WT_STAT_CONN_READ_IO 1171 +#define WT_STAT_CONN_READ_IO 1170 /*! connection: total write I/Os */ -#define WT_STAT_CONN_WRITE_IO 1172 +#define WT_STAT_CONN_WRITE_IO 1171 /*! cursor: cached cursor count */ -#define WT_STAT_CONN_CURSOR_CACHED_COUNT 1173 +#define WT_STAT_CONN_CURSOR_CACHED_COUNT 1172 /*! cursor: cursor bulk loaded cursor insert calls */ -#define WT_STAT_CONN_CURSOR_INSERT_BULK 1174 +#define WT_STAT_CONN_CURSOR_INSERT_BULK 1173 /*! cursor: cursor close calls that result in cache */ -#define WT_STAT_CONN_CURSOR_CACHE 1175 +#define WT_STAT_CONN_CURSOR_CACHE 1174 /*! cursor: cursor create calls */ -#define WT_STAT_CONN_CURSOR_CREATE 1176 +#define WT_STAT_CONN_CURSOR_CREATE 1175 /*! cursor: cursor insert calls */ -#define WT_STAT_CONN_CURSOR_INSERT 1177 +#define WT_STAT_CONN_CURSOR_INSERT 1176 /*! cursor: cursor insert key and value bytes */ -#define WT_STAT_CONN_CURSOR_INSERT_BYTES 1178 +#define WT_STAT_CONN_CURSOR_INSERT_BYTES 1177 /*! cursor: cursor modify calls */ -#define WT_STAT_CONN_CURSOR_MODIFY 1179 +#define WT_STAT_CONN_CURSOR_MODIFY 1178 /*! cursor: cursor modify key and value bytes affected */ -#define WT_STAT_CONN_CURSOR_MODIFY_BYTES 1180 +#define WT_STAT_CONN_CURSOR_MODIFY_BYTES 1179 /*! cursor: cursor modify value bytes modified */ -#define WT_STAT_CONN_CURSOR_MODIFY_BYTES_TOUCH 1181 +#define WT_STAT_CONN_CURSOR_MODIFY_BYTES_TOUCH 1180 /*! cursor: cursor next calls */ -#define WT_STAT_CONN_CURSOR_NEXT 1182 +#define WT_STAT_CONN_CURSOR_NEXT 1181 /*! cursor: cursor operation restarted */ -#define WT_STAT_CONN_CURSOR_RESTART 1183 +#define WT_STAT_CONN_CURSOR_RESTART 1182 /*! cursor: cursor prev calls */ -#define WT_STAT_CONN_CURSOR_PREV 1184 +#define WT_STAT_CONN_CURSOR_PREV 1183 /*! cursor: cursor remove calls */ -#define WT_STAT_CONN_CURSOR_REMOVE 1185 +#define WT_STAT_CONN_CURSOR_REMOVE 1184 /*! cursor: cursor remove key bytes removed */ -#define WT_STAT_CONN_CURSOR_REMOVE_BYTES 1186 +#define WT_STAT_CONN_CURSOR_REMOVE_BYTES 1185 /*! cursor: cursor reserve calls */ -#define WT_STAT_CONN_CURSOR_RESERVE 1187 +#define WT_STAT_CONN_CURSOR_RESERVE 1186 /*! cursor: cursor reset calls */ -#define WT_STAT_CONN_CURSOR_RESET 1188 +#define WT_STAT_CONN_CURSOR_RESET 1187 /*! cursor: cursor search calls */ -#define WT_STAT_CONN_CURSOR_SEARCH 1189 +#define WT_STAT_CONN_CURSOR_SEARCH 1188 /*! cursor: cursor search near calls */ -#define WT_STAT_CONN_CURSOR_SEARCH_NEAR 1190 +#define WT_STAT_CONN_CURSOR_SEARCH_NEAR 1189 /*! cursor: cursor sweep buckets */ -#define WT_STAT_CONN_CURSOR_SWEEP_BUCKETS 1191 +#define WT_STAT_CONN_CURSOR_SWEEP_BUCKETS 1190 /*! cursor: cursor sweep cursors closed */ -#define WT_STAT_CONN_CURSOR_SWEEP_CLOSED 1192 +#define WT_STAT_CONN_CURSOR_SWEEP_CLOSED 1191 /*! cursor: cursor sweep cursors examined */ -#define WT_STAT_CONN_CURSOR_SWEEP_EXAMINED 1193 +#define WT_STAT_CONN_CURSOR_SWEEP_EXAMINED 1192 /*! cursor: cursor sweeps */ -#define WT_STAT_CONN_CURSOR_SWEEP 1194 +#define WT_STAT_CONN_CURSOR_SWEEP 1193 /*! cursor: cursor truncate calls */ -#define WT_STAT_CONN_CURSOR_TRUNCATE 1195 +#define WT_STAT_CONN_CURSOR_TRUNCATE 1194 /*! cursor: cursor update calls */ -#define WT_STAT_CONN_CURSOR_UPDATE 1196 +#define WT_STAT_CONN_CURSOR_UPDATE 1195 /*! cursor: cursor update key and value bytes */ -#define WT_STAT_CONN_CURSOR_UPDATE_BYTES 1197 +#define WT_STAT_CONN_CURSOR_UPDATE_BYTES 1196 /*! cursor: cursor update value size change */ -#define WT_STAT_CONN_CURSOR_UPDATE_BYTES_CHANGED 1198 +#define WT_STAT_CONN_CURSOR_UPDATE_BYTES_CHANGED 1197 /*! cursor: cursors reused from cache */ -#define WT_STAT_CONN_CURSOR_REOPEN 1199 +#define WT_STAT_CONN_CURSOR_REOPEN 1198 /*! cursor: open cursor count */ -#define WT_STAT_CONN_CURSOR_OPEN_COUNT 1200 +#define WT_STAT_CONN_CURSOR_OPEN_COUNT 1199 /*! data-handle: connection data handle size */ -#define WT_STAT_CONN_DH_CONN_HANDLE_SIZE 1201 +#define WT_STAT_CONN_DH_CONN_HANDLE_SIZE 1200 /*! data-handle: connection data handles currently active */ -#define WT_STAT_CONN_DH_CONN_HANDLE_COUNT 1202 +#define WT_STAT_CONN_DH_CONN_HANDLE_COUNT 1201 /*! data-handle: connection sweep candidate became referenced */ -#define WT_STAT_CONN_DH_SWEEP_REF 1203 +#define WT_STAT_CONN_DH_SWEEP_REF 1202 /*! data-handle: connection sweep dhandles closed */ -#define WT_STAT_CONN_DH_SWEEP_CLOSE 1204 +#define WT_STAT_CONN_DH_SWEEP_CLOSE 1203 /*! data-handle: connection sweep dhandles removed from hash list */ -#define WT_STAT_CONN_DH_SWEEP_REMOVE 1205 +#define WT_STAT_CONN_DH_SWEEP_REMOVE 1204 /*! data-handle: connection sweep time-of-death sets */ -#define WT_STAT_CONN_DH_SWEEP_TOD 1206 +#define WT_STAT_CONN_DH_SWEEP_TOD 1205 /*! data-handle: connection sweeps */ -#define WT_STAT_CONN_DH_SWEEPS 1207 +#define WT_STAT_CONN_DH_SWEEPS 1206 /*! data-handle: session dhandles swept */ -#define WT_STAT_CONN_DH_SESSION_HANDLES 1208 +#define WT_STAT_CONN_DH_SESSION_HANDLES 1207 /*! data-handle: session sweep attempts */ -#define WT_STAT_CONN_DH_SESSION_SWEEPS 1209 +#define WT_STAT_CONN_DH_SESSION_SWEEPS 1208 +/*! history: history pages added for eviction during garbage collection */ +#define WT_STAT_CONN_HS_GC_PAGES_EVICT 1209 +/*! history: history pages removed for garbage collection */ +#define WT_STAT_CONN_HS_GC_PAGES_REMOVED 1210 +/*! history: history pages visited for garbage collection */ +#define WT_STAT_CONN_HS_GC_PAGES_VISITED 1211 /*! lock: checkpoint lock acquisitions */ -#define WT_STAT_CONN_LOCK_CHECKPOINT_COUNT 1210 +#define WT_STAT_CONN_LOCK_CHECKPOINT_COUNT 1212 /*! lock: checkpoint lock application thread wait time (usecs) */ -#define WT_STAT_CONN_LOCK_CHECKPOINT_WAIT_APPLICATION 1211 +#define WT_STAT_CONN_LOCK_CHECKPOINT_WAIT_APPLICATION 1213 /*! lock: checkpoint lock internal thread wait time (usecs) */ -#define WT_STAT_CONN_LOCK_CHECKPOINT_WAIT_INTERNAL 1212 +#define WT_STAT_CONN_LOCK_CHECKPOINT_WAIT_INTERNAL 1214 /*! lock: dhandle lock application thread time waiting (usecs) */ -#define WT_STAT_CONN_LOCK_DHANDLE_WAIT_APPLICATION 1213 +#define WT_STAT_CONN_LOCK_DHANDLE_WAIT_APPLICATION 1215 /*! lock: dhandle lock internal thread time waiting (usecs) */ -#define WT_STAT_CONN_LOCK_DHANDLE_WAIT_INTERNAL 1214 +#define WT_STAT_CONN_LOCK_DHANDLE_WAIT_INTERNAL 1216 /*! lock: dhandle read lock acquisitions */ -#define WT_STAT_CONN_LOCK_DHANDLE_READ_COUNT 1215 +#define WT_STAT_CONN_LOCK_DHANDLE_READ_COUNT 1217 /*! lock: dhandle write lock acquisitions */ -#define WT_STAT_CONN_LOCK_DHANDLE_WRITE_COUNT 1216 +#define WT_STAT_CONN_LOCK_DHANDLE_WRITE_COUNT 1218 /*! * lock: durable timestamp queue lock application thread time waiting * (usecs) */ -#define WT_STAT_CONN_LOCK_DURABLE_TIMESTAMP_WAIT_APPLICATION 1217 +#define WT_STAT_CONN_LOCK_DURABLE_TIMESTAMP_WAIT_APPLICATION 1219 /*! * lock: durable timestamp queue lock internal thread time waiting * (usecs) */ -#define WT_STAT_CONN_LOCK_DURABLE_TIMESTAMP_WAIT_INTERNAL 1218 +#define WT_STAT_CONN_LOCK_DURABLE_TIMESTAMP_WAIT_INTERNAL 1220 /*! lock: durable timestamp queue read lock acquisitions */ -#define WT_STAT_CONN_LOCK_DURABLE_TIMESTAMP_READ_COUNT 1219 +#define WT_STAT_CONN_LOCK_DURABLE_TIMESTAMP_READ_COUNT 1221 /*! lock: durable timestamp queue write lock acquisitions */ -#define WT_STAT_CONN_LOCK_DURABLE_TIMESTAMP_WRITE_COUNT 1220 +#define WT_STAT_CONN_LOCK_DURABLE_TIMESTAMP_WRITE_COUNT 1222 /*! lock: metadata lock acquisitions */ -#define WT_STAT_CONN_LOCK_METADATA_COUNT 1221 +#define WT_STAT_CONN_LOCK_METADATA_COUNT 1223 /*! lock: metadata lock application thread wait time (usecs) */ -#define WT_STAT_CONN_LOCK_METADATA_WAIT_APPLICATION 1222 +#define WT_STAT_CONN_LOCK_METADATA_WAIT_APPLICATION 1224 /*! lock: metadata lock internal thread wait time (usecs) */ -#define WT_STAT_CONN_LOCK_METADATA_WAIT_INTERNAL 1223 +#define WT_STAT_CONN_LOCK_METADATA_WAIT_INTERNAL 1225 /*! * lock: read timestamp queue lock application thread time waiting * (usecs) */ -#define WT_STAT_CONN_LOCK_READ_TIMESTAMP_WAIT_APPLICATION 1224 +#define WT_STAT_CONN_LOCK_READ_TIMESTAMP_WAIT_APPLICATION 1226 /*! lock: read timestamp queue lock internal thread time waiting (usecs) */ -#define WT_STAT_CONN_LOCK_READ_TIMESTAMP_WAIT_INTERNAL 1225 +#define WT_STAT_CONN_LOCK_READ_TIMESTAMP_WAIT_INTERNAL 1227 /*! lock: read timestamp queue read lock acquisitions */ -#define WT_STAT_CONN_LOCK_READ_TIMESTAMP_READ_COUNT 1226 +#define WT_STAT_CONN_LOCK_READ_TIMESTAMP_READ_COUNT 1228 /*! lock: read timestamp queue write lock acquisitions */ -#define WT_STAT_CONN_LOCK_READ_TIMESTAMP_WRITE_COUNT 1227 +#define WT_STAT_CONN_LOCK_READ_TIMESTAMP_WRITE_COUNT 1229 /*! lock: schema lock acquisitions */ -#define WT_STAT_CONN_LOCK_SCHEMA_COUNT 1228 +#define WT_STAT_CONN_LOCK_SCHEMA_COUNT 1230 /*! lock: schema lock application thread wait time (usecs) */ -#define WT_STAT_CONN_LOCK_SCHEMA_WAIT_APPLICATION 1229 +#define WT_STAT_CONN_LOCK_SCHEMA_WAIT_APPLICATION 1231 /*! lock: schema lock internal thread wait time (usecs) */ -#define WT_STAT_CONN_LOCK_SCHEMA_WAIT_INTERNAL 1230 +#define WT_STAT_CONN_LOCK_SCHEMA_WAIT_INTERNAL 1232 /*! * lock: table lock application thread time waiting for the table lock * (usecs) */ -#define WT_STAT_CONN_LOCK_TABLE_WAIT_APPLICATION 1231 +#define WT_STAT_CONN_LOCK_TABLE_WAIT_APPLICATION 1233 /*! * lock: table lock internal thread time waiting for the table lock * (usecs) */ -#define WT_STAT_CONN_LOCK_TABLE_WAIT_INTERNAL 1232 +#define WT_STAT_CONN_LOCK_TABLE_WAIT_INTERNAL 1234 /*! lock: table read lock acquisitions */ -#define WT_STAT_CONN_LOCK_TABLE_READ_COUNT 1233 +#define WT_STAT_CONN_LOCK_TABLE_READ_COUNT 1235 /*! lock: table write lock acquisitions */ -#define WT_STAT_CONN_LOCK_TABLE_WRITE_COUNT 1234 +#define WT_STAT_CONN_LOCK_TABLE_WRITE_COUNT 1236 /*! lock: txn global lock application thread time waiting (usecs) */ -#define WT_STAT_CONN_LOCK_TXN_GLOBAL_WAIT_APPLICATION 1235 +#define WT_STAT_CONN_LOCK_TXN_GLOBAL_WAIT_APPLICATION 1237 /*! lock: txn global lock internal thread time waiting (usecs) */ -#define WT_STAT_CONN_LOCK_TXN_GLOBAL_WAIT_INTERNAL 1236 +#define WT_STAT_CONN_LOCK_TXN_GLOBAL_WAIT_INTERNAL 1238 /*! lock: txn global read lock acquisitions */ -#define WT_STAT_CONN_LOCK_TXN_GLOBAL_READ_COUNT 1237 +#define WT_STAT_CONN_LOCK_TXN_GLOBAL_READ_COUNT 1239 /*! lock: txn global write lock acquisitions */ -#define WT_STAT_CONN_LOCK_TXN_GLOBAL_WRITE_COUNT 1238 +#define WT_STAT_CONN_LOCK_TXN_GLOBAL_WRITE_COUNT 1240 /*! log: busy returns attempting to switch slots */ -#define WT_STAT_CONN_LOG_SLOT_SWITCH_BUSY 1239 +#define WT_STAT_CONN_LOG_SLOT_SWITCH_BUSY 1241 /*! log: force archive time sleeping (usecs) */ -#define WT_STAT_CONN_LOG_FORCE_ARCHIVE_SLEEP 1240 +#define WT_STAT_CONN_LOG_FORCE_ARCHIVE_SLEEP 1242 /*! log: log bytes of payload data */ -#define WT_STAT_CONN_LOG_BYTES_PAYLOAD 1241 +#define WT_STAT_CONN_LOG_BYTES_PAYLOAD 1243 /*! log: log bytes written */ -#define WT_STAT_CONN_LOG_BYTES_WRITTEN 1242 +#define WT_STAT_CONN_LOG_BYTES_WRITTEN 1244 /*! log: log files manually zero-filled */ -#define WT_STAT_CONN_LOG_ZERO_FILLS 1243 +#define WT_STAT_CONN_LOG_ZERO_FILLS 1245 /*! log: log flush operations */ -#define WT_STAT_CONN_LOG_FLUSH 1244 +#define WT_STAT_CONN_LOG_FLUSH 1246 /*! log: log force write operations */ -#define WT_STAT_CONN_LOG_FORCE_WRITE 1245 +#define WT_STAT_CONN_LOG_FORCE_WRITE 1247 /*! log: log force write operations skipped */ -#define WT_STAT_CONN_LOG_FORCE_WRITE_SKIP 1246 +#define WT_STAT_CONN_LOG_FORCE_WRITE_SKIP 1248 /*! log: log records compressed */ -#define WT_STAT_CONN_LOG_COMPRESS_WRITES 1247 +#define WT_STAT_CONN_LOG_COMPRESS_WRITES 1249 /*! log: log records not compressed */ -#define WT_STAT_CONN_LOG_COMPRESS_WRITE_FAILS 1248 +#define WT_STAT_CONN_LOG_COMPRESS_WRITE_FAILS 1250 /*! log: log records too small to compress */ -#define WT_STAT_CONN_LOG_COMPRESS_SMALL 1249 +#define WT_STAT_CONN_LOG_COMPRESS_SMALL 1251 /*! log: log release advances write LSN */ -#define WT_STAT_CONN_LOG_RELEASE_WRITE_LSN 1250 +#define WT_STAT_CONN_LOG_RELEASE_WRITE_LSN 1252 /*! log: log scan operations */ -#define WT_STAT_CONN_LOG_SCANS 1251 +#define WT_STAT_CONN_LOG_SCANS 1253 /*! log: log scan records requiring two reads */ -#define WT_STAT_CONN_LOG_SCAN_REREADS 1252 +#define WT_STAT_CONN_LOG_SCAN_REREADS 1254 /*! log: log server thread advances write LSN */ -#define WT_STAT_CONN_LOG_WRITE_LSN 1253 +#define WT_STAT_CONN_LOG_WRITE_LSN 1255 /*! log: log server thread write LSN walk skipped */ -#define WT_STAT_CONN_LOG_WRITE_LSN_SKIP 1254 +#define WT_STAT_CONN_LOG_WRITE_LSN_SKIP 1256 /*! log: log sync operations */ -#define WT_STAT_CONN_LOG_SYNC 1255 +#define WT_STAT_CONN_LOG_SYNC 1257 /*! log: log sync time duration (usecs) */ -#define WT_STAT_CONN_LOG_SYNC_DURATION 1256 +#define WT_STAT_CONN_LOG_SYNC_DURATION 1258 /*! log: log sync_dir operations */ -#define WT_STAT_CONN_LOG_SYNC_DIR 1257 +#define WT_STAT_CONN_LOG_SYNC_DIR 1259 /*! log: log sync_dir time duration (usecs) */ -#define WT_STAT_CONN_LOG_SYNC_DIR_DURATION 1258 +#define WT_STAT_CONN_LOG_SYNC_DIR_DURATION 1260 /*! log: log write operations */ -#define WT_STAT_CONN_LOG_WRITES 1259 +#define WT_STAT_CONN_LOG_WRITES 1261 /*! log: logging bytes consolidated */ -#define WT_STAT_CONN_LOG_SLOT_CONSOLIDATED 1260 +#define WT_STAT_CONN_LOG_SLOT_CONSOLIDATED 1262 /*! log: maximum log file size */ -#define WT_STAT_CONN_LOG_MAX_FILESIZE 1261 +#define WT_STAT_CONN_LOG_MAX_FILESIZE 1263 /*! log: number of pre-allocated log files to create */ -#define WT_STAT_CONN_LOG_PREALLOC_MAX 1262 +#define WT_STAT_CONN_LOG_PREALLOC_MAX 1264 /*! log: pre-allocated log files not ready and missed */ -#define WT_STAT_CONN_LOG_PREALLOC_MISSED 1263 +#define WT_STAT_CONN_LOG_PREALLOC_MISSED 1265 /*! log: pre-allocated log files prepared */ -#define WT_STAT_CONN_LOG_PREALLOC_FILES 1264 +#define WT_STAT_CONN_LOG_PREALLOC_FILES 1266 /*! log: pre-allocated log files used */ -#define WT_STAT_CONN_LOG_PREALLOC_USED 1265 +#define WT_STAT_CONN_LOG_PREALLOC_USED 1267 /*! log: records processed by log scan */ -#define WT_STAT_CONN_LOG_SCAN_RECORDS 1266 +#define WT_STAT_CONN_LOG_SCAN_RECORDS 1268 /*! log: slot close lost race */ -#define WT_STAT_CONN_LOG_SLOT_CLOSE_RACE 1267 +#define WT_STAT_CONN_LOG_SLOT_CLOSE_RACE 1269 /*! log: slot close unbuffered waits */ -#define WT_STAT_CONN_LOG_SLOT_CLOSE_UNBUF 1268 +#define WT_STAT_CONN_LOG_SLOT_CLOSE_UNBUF 1270 /*! log: slot closures */ -#define WT_STAT_CONN_LOG_SLOT_CLOSES 1269 +#define WT_STAT_CONN_LOG_SLOT_CLOSES 1271 /*! log: slot join atomic update races */ -#define WT_STAT_CONN_LOG_SLOT_RACES 1270 +#define WT_STAT_CONN_LOG_SLOT_RACES 1272 /*! log: slot join calls atomic updates raced */ -#define WT_STAT_CONN_LOG_SLOT_YIELD_RACE 1271 +#define WT_STAT_CONN_LOG_SLOT_YIELD_RACE 1273 /*! log: slot join calls did not yield */ -#define WT_STAT_CONN_LOG_SLOT_IMMEDIATE 1272 +#define WT_STAT_CONN_LOG_SLOT_IMMEDIATE 1274 /*! log: slot join calls found active slot closed */ -#define WT_STAT_CONN_LOG_SLOT_YIELD_CLOSE 1273 +#define WT_STAT_CONN_LOG_SLOT_YIELD_CLOSE 1275 /*! log: slot join calls slept */ -#define WT_STAT_CONN_LOG_SLOT_YIELD_SLEEP 1274 +#define WT_STAT_CONN_LOG_SLOT_YIELD_SLEEP 1276 /*! log: slot join calls yielded */ -#define WT_STAT_CONN_LOG_SLOT_YIELD 1275 +#define WT_STAT_CONN_LOG_SLOT_YIELD 1277 /*! log: slot join found active slot closed */ -#define WT_STAT_CONN_LOG_SLOT_ACTIVE_CLOSED 1276 +#define WT_STAT_CONN_LOG_SLOT_ACTIVE_CLOSED 1278 /*! log: slot joins yield time (usecs) */ -#define WT_STAT_CONN_LOG_SLOT_YIELD_DURATION 1277 +#define WT_STAT_CONN_LOG_SLOT_YIELD_DURATION 1279 /*! log: slot transitions unable to find free slot */ -#define WT_STAT_CONN_LOG_SLOT_NO_FREE_SLOTS 1278 +#define WT_STAT_CONN_LOG_SLOT_NO_FREE_SLOTS 1280 /*! log: slot unbuffered writes */ -#define WT_STAT_CONN_LOG_SLOT_UNBUFFERED 1279 +#define WT_STAT_CONN_LOG_SLOT_UNBUFFERED 1281 /*! log: total in-memory size of compressed records */ -#define WT_STAT_CONN_LOG_COMPRESS_MEM 1280 +#define WT_STAT_CONN_LOG_COMPRESS_MEM 1282 /*! log: total log buffer size */ -#define WT_STAT_CONN_LOG_BUFFER_SIZE 1281 +#define WT_STAT_CONN_LOG_BUFFER_SIZE 1283 /*! log: total size of compressed records */ -#define WT_STAT_CONN_LOG_COMPRESS_LEN 1282 +#define WT_STAT_CONN_LOG_COMPRESS_LEN 1284 /*! log: written slots coalesced */ -#define WT_STAT_CONN_LOG_SLOT_COALESCED 1283 +#define WT_STAT_CONN_LOG_SLOT_COALESCED 1285 /*! log: yields waiting for previous log file close */ -#define WT_STAT_CONN_LOG_CLOSE_YIELDS 1284 +#define WT_STAT_CONN_LOG_CLOSE_YIELDS 1286 /*! perf: file system read latency histogram (bucket 1) - 10-49ms */ -#define WT_STAT_CONN_PERF_HIST_FSREAD_LATENCY_LT50 1285 +#define WT_STAT_CONN_PERF_HIST_FSREAD_LATENCY_LT50 1287 /*! perf: file system read latency histogram (bucket 2) - 50-99ms */ -#define WT_STAT_CONN_PERF_HIST_FSREAD_LATENCY_LT100 1286 +#define WT_STAT_CONN_PERF_HIST_FSREAD_LATENCY_LT100 1288 /*! perf: file system read latency histogram (bucket 3) - 100-249ms */ -#define WT_STAT_CONN_PERF_HIST_FSREAD_LATENCY_LT250 1287 +#define WT_STAT_CONN_PERF_HIST_FSREAD_LATENCY_LT250 1289 /*! perf: file system read latency histogram (bucket 4) - 250-499ms */ -#define WT_STAT_CONN_PERF_HIST_FSREAD_LATENCY_LT500 1288 +#define WT_STAT_CONN_PERF_HIST_FSREAD_LATENCY_LT500 1290 /*! perf: file system read latency histogram (bucket 5) - 500-999ms */ -#define WT_STAT_CONN_PERF_HIST_FSREAD_LATENCY_LT1000 1289 +#define WT_STAT_CONN_PERF_HIST_FSREAD_LATENCY_LT1000 1291 /*! perf: file system read latency histogram (bucket 6) - 1000ms+ */ -#define WT_STAT_CONN_PERF_HIST_FSREAD_LATENCY_GT1000 1290 +#define WT_STAT_CONN_PERF_HIST_FSREAD_LATENCY_GT1000 1292 /*! perf: file system write latency histogram (bucket 1) - 10-49ms */ -#define WT_STAT_CONN_PERF_HIST_FSWRITE_LATENCY_LT50 1291 +#define WT_STAT_CONN_PERF_HIST_FSWRITE_LATENCY_LT50 1293 /*! perf: file system write latency histogram (bucket 2) - 50-99ms */ -#define WT_STAT_CONN_PERF_HIST_FSWRITE_LATENCY_LT100 1292 +#define WT_STAT_CONN_PERF_HIST_FSWRITE_LATENCY_LT100 1294 /*! perf: file system write latency histogram (bucket 3) - 100-249ms */ -#define WT_STAT_CONN_PERF_HIST_FSWRITE_LATENCY_LT250 1293 +#define WT_STAT_CONN_PERF_HIST_FSWRITE_LATENCY_LT250 1295 /*! perf: file system write latency histogram (bucket 4) - 250-499ms */ -#define WT_STAT_CONN_PERF_HIST_FSWRITE_LATENCY_LT500 1294 +#define WT_STAT_CONN_PERF_HIST_FSWRITE_LATENCY_LT500 1296 /*! perf: file system write latency histogram (bucket 5) - 500-999ms */ -#define WT_STAT_CONN_PERF_HIST_FSWRITE_LATENCY_LT1000 1295 +#define WT_STAT_CONN_PERF_HIST_FSWRITE_LATENCY_LT1000 1297 /*! perf: file system write latency histogram (bucket 6) - 1000ms+ */ -#define WT_STAT_CONN_PERF_HIST_FSWRITE_LATENCY_GT1000 1296 +#define WT_STAT_CONN_PERF_HIST_FSWRITE_LATENCY_GT1000 1298 /*! perf: operation read latency histogram (bucket 1) - 100-249us */ -#define WT_STAT_CONN_PERF_HIST_OPREAD_LATENCY_LT250 1297 +#define WT_STAT_CONN_PERF_HIST_OPREAD_LATENCY_LT250 1299 /*! perf: operation read latency histogram (bucket 2) - 250-499us */ -#define WT_STAT_CONN_PERF_HIST_OPREAD_LATENCY_LT500 1298 +#define WT_STAT_CONN_PERF_HIST_OPREAD_LATENCY_LT500 1300 /*! perf: operation read latency histogram (bucket 3) - 500-999us */ -#define WT_STAT_CONN_PERF_HIST_OPREAD_LATENCY_LT1000 1299 +#define WT_STAT_CONN_PERF_HIST_OPREAD_LATENCY_LT1000 1301 /*! perf: operation read latency histogram (bucket 4) - 1000-9999us */ -#define WT_STAT_CONN_PERF_HIST_OPREAD_LATENCY_LT10000 1300 +#define WT_STAT_CONN_PERF_HIST_OPREAD_LATENCY_LT10000 1302 /*! perf: operation read latency histogram (bucket 5) - 10000us+ */ -#define WT_STAT_CONN_PERF_HIST_OPREAD_LATENCY_GT10000 1301 +#define WT_STAT_CONN_PERF_HIST_OPREAD_LATENCY_GT10000 1303 /*! perf: operation write latency histogram (bucket 1) - 100-249us */ -#define WT_STAT_CONN_PERF_HIST_OPWRITE_LATENCY_LT250 1302 +#define WT_STAT_CONN_PERF_HIST_OPWRITE_LATENCY_LT250 1304 /*! perf: operation write latency histogram (bucket 2) - 250-499us */ -#define WT_STAT_CONN_PERF_HIST_OPWRITE_LATENCY_LT500 1303 +#define WT_STAT_CONN_PERF_HIST_OPWRITE_LATENCY_LT500 1305 /*! perf: operation write latency histogram (bucket 3) - 500-999us */ -#define WT_STAT_CONN_PERF_HIST_OPWRITE_LATENCY_LT1000 1304 +#define WT_STAT_CONN_PERF_HIST_OPWRITE_LATENCY_LT1000 1306 /*! perf: operation write latency histogram (bucket 4) - 1000-9999us */ -#define WT_STAT_CONN_PERF_HIST_OPWRITE_LATENCY_LT10000 1305 +#define WT_STAT_CONN_PERF_HIST_OPWRITE_LATENCY_LT10000 1307 /*! perf: operation write latency histogram (bucket 5) - 10000us+ */ -#define WT_STAT_CONN_PERF_HIST_OPWRITE_LATENCY_GT10000 1306 +#define WT_STAT_CONN_PERF_HIST_OPWRITE_LATENCY_GT10000 1308 /*! reconciliation: fast-path pages deleted */ -#define WT_STAT_CONN_REC_PAGE_DELETE_FAST 1307 +#define WT_STAT_CONN_REC_PAGE_DELETE_FAST 1309 /*! reconciliation: page reconciliation calls */ -#define WT_STAT_CONN_REC_PAGES 1308 +#define WT_STAT_CONN_REC_PAGES 1310 /*! reconciliation: page reconciliation calls for eviction */ -#define WT_STAT_CONN_REC_PAGES_EVICTION 1309 +#define WT_STAT_CONN_REC_PAGES_EVICTION 1311 /*! reconciliation: pages deleted */ -#define WT_STAT_CONN_REC_PAGE_DELETE 1310 +#define WT_STAT_CONN_REC_PAGE_DELETE 1312 /*! reconciliation: split bytes currently awaiting free */ -#define WT_STAT_CONN_REC_SPLIT_STASHED_BYTES 1311 +#define WT_STAT_CONN_REC_SPLIT_STASHED_BYTES 1313 /*! reconciliation: split objects currently awaiting free */ -#define WT_STAT_CONN_REC_SPLIT_STASHED_OBJECTS 1312 +#define WT_STAT_CONN_REC_SPLIT_STASHED_OBJECTS 1314 /*! session: open session count */ -#define WT_STAT_CONN_SESSION_OPEN 1313 +#define WT_STAT_CONN_SESSION_OPEN 1315 /*! session: session query timestamp calls */ -#define WT_STAT_CONN_SESSION_QUERY_TS 1314 +#define WT_STAT_CONN_SESSION_QUERY_TS 1316 /*! session: table alter failed calls */ -#define WT_STAT_CONN_SESSION_TABLE_ALTER_FAIL 1315 +#define WT_STAT_CONN_SESSION_TABLE_ALTER_FAIL 1317 /*! session: table alter successful calls */ -#define WT_STAT_CONN_SESSION_TABLE_ALTER_SUCCESS 1316 +#define WT_STAT_CONN_SESSION_TABLE_ALTER_SUCCESS 1318 /*! session: table alter unchanged and skipped */ -#define WT_STAT_CONN_SESSION_TABLE_ALTER_SKIP 1317 +#define WT_STAT_CONN_SESSION_TABLE_ALTER_SKIP 1319 /*! session: table compact failed calls */ -#define WT_STAT_CONN_SESSION_TABLE_COMPACT_FAIL 1318 +#define WT_STAT_CONN_SESSION_TABLE_COMPACT_FAIL 1320 /*! session: table compact successful calls */ -#define WT_STAT_CONN_SESSION_TABLE_COMPACT_SUCCESS 1319 +#define WT_STAT_CONN_SESSION_TABLE_COMPACT_SUCCESS 1321 /*! session: table create failed calls */ -#define WT_STAT_CONN_SESSION_TABLE_CREATE_FAIL 1320 +#define WT_STAT_CONN_SESSION_TABLE_CREATE_FAIL 1322 /*! session: table create successful calls */ -#define WT_STAT_CONN_SESSION_TABLE_CREATE_SUCCESS 1321 +#define WT_STAT_CONN_SESSION_TABLE_CREATE_SUCCESS 1323 /*! session: table drop failed calls */ -#define WT_STAT_CONN_SESSION_TABLE_DROP_FAIL 1322 +#define WT_STAT_CONN_SESSION_TABLE_DROP_FAIL 1324 /*! session: table drop successful calls */ -#define WT_STAT_CONN_SESSION_TABLE_DROP_SUCCESS 1323 +#define WT_STAT_CONN_SESSION_TABLE_DROP_SUCCESS 1325 /*! session: table import failed calls */ -#define WT_STAT_CONN_SESSION_TABLE_IMPORT_FAIL 1324 +#define WT_STAT_CONN_SESSION_TABLE_IMPORT_FAIL 1326 /*! session: table import successful calls */ -#define WT_STAT_CONN_SESSION_TABLE_IMPORT_SUCCESS 1325 +#define WT_STAT_CONN_SESSION_TABLE_IMPORT_SUCCESS 1327 /*! session: table rebalance failed calls */ -#define WT_STAT_CONN_SESSION_TABLE_REBALANCE_FAIL 1326 +#define WT_STAT_CONN_SESSION_TABLE_REBALANCE_FAIL 1328 /*! session: table rebalance successful calls */ -#define WT_STAT_CONN_SESSION_TABLE_REBALANCE_SUCCESS 1327 +#define WT_STAT_CONN_SESSION_TABLE_REBALANCE_SUCCESS 1329 /*! session: table rename failed calls */ -#define WT_STAT_CONN_SESSION_TABLE_RENAME_FAIL 1328 +#define WT_STAT_CONN_SESSION_TABLE_RENAME_FAIL 1330 /*! session: table rename successful calls */ -#define WT_STAT_CONN_SESSION_TABLE_RENAME_SUCCESS 1329 +#define WT_STAT_CONN_SESSION_TABLE_RENAME_SUCCESS 1331 /*! session: table salvage failed calls */ -#define WT_STAT_CONN_SESSION_TABLE_SALVAGE_FAIL 1330 +#define WT_STAT_CONN_SESSION_TABLE_SALVAGE_FAIL 1332 /*! session: table salvage successful calls */ -#define WT_STAT_CONN_SESSION_TABLE_SALVAGE_SUCCESS 1331 +#define WT_STAT_CONN_SESSION_TABLE_SALVAGE_SUCCESS 1333 /*! session: table truncate failed calls */ -#define WT_STAT_CONN_SESSION_TABLE_TRUNCATE_FAIL 1332 +#define WT_STAT_CONN_SESSION_TABLE_TRUNCATE_FAIL 1334 /*! session: table truncate successful calls */ -#define WT_STAT_CONN_SESSION_TABLE_TRUNCATE_SUCCESS 1333 +#define WT_STAT_CONN_SESSION_TABLE_TRUNCATE_SUCCESS 1335 /*! session: table verify failed calls */ -#define WT_STAT_CONN_SESSION_TABLE_VERIFY_FAIL 1334 +#define WT_STAT_CONN_SESSION_TABLE_VERIFY_FAIL 1336 /*! session: table verify successful calls */ -#define WT_STAT_CONN_SESSION_TABLE_VERIFY_SUCCESS 1335 +#define WT_STAT_CONN_SESSION_TABLE_VERIFY_SUCCESS 1337 /*! thread-state: active filesystem fsync calls */ -#define WT_STAT_CONN_THREAD_FSYNC_ACTIVE 1336 +#define WT_STAT_CONN_THREAD_FSYNC_ACTIVE 1338 /*! thread-state: active filesystem read calls */ -#define WT_STAT_CONN_THREAD_READ_ACTIVE 1337 +#define WT_STAT_CONN_THREAD_READ_ACTIVE 1339 /*! thread-state: active filesystem write calls */ -#define WT_STAT_CONN_THREAD_WRITE_ACTIVE 1338 +#define WT_STAT_CONN_THREAD_WRITE_ACTIVE 1340 /*! thread-yield: application thread time evicting (usecs) */ -#define WT_STAT_CONN_APPLICATION_EVICT_TIME 1339 +#define WT_STAT_CONN_APPLICATION_EVICT_TIME 1341 /*! thread-yield: application thread time waiting for cache (usecs) */ -#define WT_STAT_CONN_APPLICATION_CACHE_TIME 1340 +#define WT_STAT_CONN_APPLICATION_CACHE_TIME 1342 /*! * thread-yield: connection close blocked waiting for transaction state * stabilization */ -#define WT_STAT_CONN_TXN_RELEASE_BLOCKED 1341 +#define WT_STAT_CONN_TXN_RELEASE_BLOCKED 1343 /*! thread-yield: connection close yielded for lsm manager shutdown */ -#define WT_STAT_CONN_CONN_CLOSE_BLOCKED_LSM 1342 +#define WT_STAT_CONN_CONN_CLOSE_BLOCKED_LSM 1344 /*! thread-yield: data handle lock yielded */ -#define WT_STAT_CONN_DHANDLE_LOCK_BLOCKED 1343 +#define WT_STAT_CONN_DHANDLE_LOCK_BLOCKED 1345 /*! * thread-yield: get reference for page index and slot time sleeping * (usecs) */ -#define WT_STAT_CONN_PAGE_INDEX_SLOT_REF_BLOCKED 1344 +#define WT_STAT_CONN_PAGE_INDEX_SLOT_REF_BLOCKED 1346 /*! thread-yield: log server sync yielded for log write */ -#define WT_STAT_CONN_LOG_SERVER_SYNC_BLOCKED 1345 +#define WT_STAT_CONN_LOG_SERVER_SYNC_BLOCKED 1347 /*! thread-yield: page access yielded due to prepare state change */ -#define WT_STAT_CONN_PREPARED_TRANSITION_BLOCKED_PAGE 1346 +#define WT_STAT_CONN_PREPARED_TRANSITION_BLOCKED_PAGE 1348 /*! thread-yield: page acquire busy blocked */ -#define WT_STAT_CONN_PAGE_BUSY_BLOCKED 1347 +#define WT_STAT_CONN_PAGE_BUSY_BLOCKED 1349 /*! thread-yield: page acquire eviction blocked */ -#define WT_STAT_CONN_PAGE_FORCIBLE_EVICT_BLOCKED 1348 +#define WT_STAT_CONN_PAGE_FORCIBLE_EVICT_BLOCKED 1350 /*! thread-yield: page acquire locked blocked */ -#define WT_STAT_CONN_PAGE_LOCKED_BLOCKED 1349 +#define WT_STAT_CONN_PAGE_LOCKED_BLOCKED 1351 /*! thread-yield: page acquire read blocked */ -#define WT_STAT_CONN_PAGE_READ_BLOCKED 1350 +#define WT_STAT_CONN_PAGE_READ_BLOCKED 1352 /*! thread-yield: page acquire time sleeping (usecs) */ -#define WT_STAT_CONN_PAGE_SLEEP 1351 +#define WT_STAT_CONN_PAGE_SLEEP 1353 /*! * thread-yield: page delete rollback time sleeping for state change * (usecs) */ -#define WT_STAT_CONN_PAGE_DEL_ROLLBACK_BLOCKED 1352 +#define WT_STAT_CONN_PAGE_DEL_ROLLBACK_BLOCKED 1354 /*! thread-yield: page reconciliation yielded due to child modification */ -#define WT_STAT_CONN_CHILD_MODIFY_BLOCKED_PAGE 1353 +#define WT_STAT_CONN_CHILD_MODIFY_BLOCKED_PAGE 1355 /*! transaction: Number of prepared updates */ -#define WT_STAT_CONN_TXN_PREPARED_UPDATES_COUNT 1354 -/*! transaction: Number of prepared updates added to cache overflow */ -#define WT_STAT_CONN_TXN_PREPARED_UPDATES_LOOKASIDE_INSERTS 1355 +#define WT_STAT_CONN_TXN_PREPARED_UPDATES_COUNT 1356 /*! transaction: durable timestamp queue entries walked */ -#define WT_STAT_CONN_TXN_DURABLE_QUEUE_WALKED 1356 +#define WT_STAT_CONN_TXN_DURABLE_QUEUE_WALKED 1357 /*! transaction: durable timestamp queue insert to empty */ -#define WT_STAT_CONN_TXN_DURABLE_QUEUE_EMPTY 1357 +#define WT_STAT_CONN_TXN_DURABLE_QUEUE_EMPTY 1358 /*! transaction: durable timestamp queue inserts to head */ -#define WT_STAT_CONN_TXN_DURABLE_QUEUE_HEAD 1358 +#define WT_STAT_CONN_TXN_DURABLE_QUEUE_HEAD 1359 /*! transaction: durable timestamp queue inserts total */ -#define WT_STAT_CONN_TXN_DURABLE_QUEUE_INSERTS 1359 +#define WT_STAT_CONN_TXN_DURABLE_QUEUE_INSERTS 1360 /*! transaction: durable timestamp queue length */ -#define WT_STAT_CONN_TXN_DURABLE_QUEUE_LEN 1360 -/*! transaction: number of named snapshots created */ -#define WT_STAT_CONN_TXN_SNAPSHOTS_CREATED 1361 -/*! transaction: number of named snapshots dropped */ -#define WT_STAT_CONN_TXN_SNAPSHOTS_DROPPED 1362 +#define WT_STAT_CONN_TXN_DURABLE_QUEUE_LEN 1361 /*! transaction: prepared transactions */ -#define WT_STAT_CONN_TXN_PREPARE 1363 +#define WT_STAT_CONN_TXN_PREPARE 1362 /*! transaction: prepared transactions committed */ -#define WT_STAT_CONN_TXN_PREPARE_COMMIT 1364 +#define WT_STAT_CONN_TXN_PREPARE_COMMIT 1363 /*! transaction: prepared transactions currently active */ -#define WT_STAT_CONN_TXN_PREPARE_ACTIVE 1365 +#define WT_STAT_CONN_TXN_PREPARE_ACTIVE 1364 /*! transaction: prepared transactions rolled back */ -#define WT_STAT_CONN_TXN_PREPARE_ROLLBACK 1366 +#define WT_STAT_CONN_TXN_PREPARE_ROLLBACK 1365 /*! transaction: query timestamp calls */ -#define WT_STAT_CONN_TXN_QUERY_TS 1367 +#define WT_STAT_CONN_TXN_QUERY_TS 1366 /*! transaction: read timestamp queue entries walked */ -#define WT_STAT_CONN_TXN_READ_QUEUE_WALKED 1368 +#define WT_STAT_CONN_TXN_READ_QUEUE_WALKED 1367 /*! transaction: read timestamp queue insert to empty */ -#define WT_STAT_CONN_TXN_READ_QUEUE_EMPTY 1369 +#define WT_STAT_CONN_TXN_READ_QUEUE_EMPTY 1368 /*! transaction: read timestamp queue inserts to head */ -#define WT_STAT_CONN_TXN_READ_QUEUE_HEAD 1370 +#define WT_STAT_CONN_TXN_READ_QUEUE_HEAD 1369 /*! transaction: read timestamp queue inserts total */ -#define WT_STAT_CONN_TXN_READ_QUEUE_INSERTS 1371 +#define WT_STAT_CONN_TXN_READ_QUEUE_INSERTS 1370 /*! transaction: read timestamp queue length */ -#define WT_STAT_CONN_TXN_READ_QUEUE_LEN 1372 +#define WT_STAT_CONN_TXN_READ_QUEUE_LEN 1371 /*! transaction: rollback to stable calls */ -#define WT_STAT_CONN_TXN_ROLLBACK_TO_STABLE 1373 +#define WT_STAT_CONN_TXN_RTS 1372 +/*! transaction: rollback to stable keys removed */ +#define WT_STAT_CONN_TXN_RTS_KEYS_REMOVED 1373 +/*! transaction: rollback to stable keys restored */ +#define WT_STAT_CONN_TXN_RTS_KEYS_RESTORED 1374 +/*! transaction: rollback to stable pages visited */ +#define WT_STAT_CONN_TXN_RTS_PAGES_VISITED 1375 /*! transaction: rollback to stable updates aborted */ -#define WT_STAT_CONN_TXN_ROLLBACK_UPD_ABORTED 1374 -/*! transaction: rollback to stable updates removed from cache overflow */ -#define WT_STAT_CONN_TXN_ROLLBACK_LAS_REMOVED 1375 +#define WT_STAT_CONN_TXN_RTS_UPD_ABORTED 1376 +/*! transaction: rollback to stable updates removed from history store */ +#define WT_STAT_CONN_TXN_RTS_HS_REMOVED 1377 /*! transaction: set timestamp calls */ -#define WT_STAT_CONN_TXN_SET_TS 1376 +#define WT_STAT_CONN_TXN_SET_TS 1378 /*! transaction: set timestamp durable calls */ -#define WT_STAT_CONN_TXN_SET_TS_DURABLE 1377 +#define WT_STAT_CONN_TXN_SET_TS_DURABLE 1379 /*! transaction: set timestamp durable updates */ -#define WT_STAT_CONN_TXN_SET_TS_DURABLE_UPD 1378 +#define WT_STAT_CONN_TXN_SET_TS_DURABLE_UPD 1380 /*! transaction: set timestamp oldest calls */ -#define WT_STAT_CONN_TXN_SET_TS_OLDEST 1379 +#define WT_STAT_CONN_TXN_SET_TS_OLDEST 1381 /*! transaction: set timestamp oldest updates */ -#define WT_STAT_CONN_TXN_SET_TS_OLDEST_UPD 1380 +#define WT_STAT_CONN_TXN_SET_TS_OLDEST_UPD 1382 /*! transaction: set timestamp stable calls */ -#define WT_STAT_CONN_TXN_SET_TS_STABLE 1381 +#define WT_STAT_CONN_TXN_SET_TS_STABLE 1383 /*! transaction: set timestamp stable updates */ -#define WT_STAT_CONN_TXN_SET_TS_STABLE_UPD 1382 +#define WT_STAT_CONN_TXN_SET_TS_STABLE_UPD 1384 /*! transaction: transaction begins */ -#define WT_STAT_CONN_TXN_BEGIN 1383 +#define WT_STAT_CONN_TXN_BEGIN 1385 /*! transaction: transaction checkpoint currently running */ -#define WT_STAT_CONN_TXN_CHECKPOINT_RUNNING 1384 +#define WT_STAT_CONN_TXN_CHECKPOINT_RUNNING 1386 /*! transaction: transaction checkpoint generation */ -#define WT_STAT_CONN_TXN_CHECKPOINT_GENERATION 1385 +#define WT_STAT_CONN_TXN_CHECKPOINT_GENERATION 1387 +/*! + * transaction: transaction checkpoint history store file duration + * (usecs) + */ +#define WT_STAT_CONN_TXN_HS_CKPT_DURATION 1388 /*! transaction: transaction checkpoint max time (msecs) */ -#define WT_STAT_CONN_TXN_CHECKPOINT_TIME_MAX 1386 +#define WT_STAT_CONN_TXN_CHECKPOINT_TIME_MAX 1389 /*! transaction: transaction checkpoint min time (msecs) */ -#define WT_STAT_CONN_TXN_CHECKPOINT_TIME_MIN 1387 +#define WT_STAT_CONN_TXN_CHECKPOINT_TIME_MIN 1390 /*! transaction: transaction checkpoint most recent time (msecs) */ -#define WT_STAT_CONN_TXN_CHECKPOINT_TIME_RECENT 1388 +#define WT_STAT_CONN_TXN_CHECKPOINT_TIME_RECENT 1391 +/*! transaction: transaction checkpoint prepare currently running */ +#define WT_STAT_CONN_TXN_CHECKPOINT_PREP_RUNNING 1392 +/*! transaction: transaction checkpoint prepare max time (msecs) */ +#define WT_STAT_CONN_TXN_CHECKPOINT_PREP_MAX 1393 +/*! transaction: transaction checkpoint prepare min time (msecs) */ +#define WT_STAT_CONN_TXN_CHECKPOINT_PREP_MIN 1394 +/*! transaction: transaction checkpoint prepare most recent time (msecs) */ +#define WT_STAT_CONN_TXN_CHECKPOINT_PREP_RECENT 1395 +/*! transaction: transaction checkpoint prepare total time (msecs) */ +#define WT_STAT_CONN_TXN_CHECKPOINT_PREP_TOTAL 1396 /*! transaction: transaction checkpoint scrub dirty target */ -#define WT_STAT_CONN_TXN_CHECKPOINT_SCRUB_TARGET 1389 +#define WT_STAT_CONN_TXN_CHECKPOINT_SCRUB_TARGET 1397 /*! transaction: transaction checkpoint scrub time (msecs) */ -#define WT_STAT_CONN_TXN_CHECKPOINT_SCRUB_TIME 1390 +#define WT_STAT_CONN_TXN_CHECKPOINT_SCRUB_TIME 1398 /*! transaction: transaction checkpoint total time (msecs) */ -#define WT_STAT_CONN_TXN_CHECKPOINT_TIME_TOTAL 1391 +#define WT_STAT_CONN_TXN_CHECKPOINT_TIME_TOTAL 1399 /*! transaction: transaction checkpoints */ -#define WT_STAT_CONN_TXN_CHECKPOINT 1392 +#define WT_STAT_CONN_TXN_CHECKPOINT 1400 /*! * transaction: transaction checkpoints skipped because database was * clean */ -#define WT_STAT_CONN_TXN_CHECKPOINT_SKIPPED 1393 -/*! transaction: transaction failures due to cache overflow */ -#define WT_STAT_CONN_TXN_FAIL_CACHE 1394 +#define WT_STAT_CONN_TXN_CHECKPOINT_SKIPPED 1401 +/*! transaction: transaction failures due to history store */ +#define WT_STAT_CONN_TXN_FAIL_CACHE 1402 /*! * transaction: transaction fsync calls for checkpoint after allocating * the transaction ID */ -#define WT_STAT_CONN_TXN_CHECKPOINT_FSYNC_POST 1395 +#define WT_STAT_CONN_TXN_CHECKPOINT_FSYNC_POST 1403 /*! * transaction: transaction fsync duration for checkpoint after * allocating the transaction ID (usecs) */ -#define WT_STAT_CONN_TXN_CHECKPOINT_FSYNC_POST_DURATION 1396 +#define WT_STAT_CONN_TXN_CHECKPOINT_FSYNC_POST_DURATION 1404 /*! transaction: transaction range of IDs currently pinned */ -#define WT_STAT_CONN_TXN_PINNED_RANGE 1397 +#define WT_STAT_CONN_TXN_PINNED_RANGE 1405 /*! transaction: transaction range of IDs currently pinned by a checkpoint */ -#define WT_STAT_CONN_TXN_PINNED_CHECKPOINT_RANGE 1398 -/*! - * transaction: transaction range of IDs currently pinned by named - * snapshots - */ -#define WT_STAT_CONN_TXN_PINNED_SNAPSHOT_RANGE 1399 +#define WT_STAT_CONN_TXN_PINNED_CHECKPOINT_RANGE 1406 /*! transaction: transaction range of timestamps currently pinned */ -#define WT_STAT_CONN_TXN_PINNED_TIMESTAMP 1400 +#define WT_STAT_CONN_TXN_PINNED_TIMESTAMP 1407 /*! transaction: transaction range of timestamps pinned by a checkpoint */ -#define WT_STAT_CONN_TXN_PINNED_TIMESTAMP_CHECKPOINT 1401 +#define WT_STAT_CONN_TXN_PINNED_TIMESTAMP_CHECKPOINT 1408 /*! * transaction: transaction range of timestamps pinned by the oldest * active read timestamp */ -#define WT_STAT_CONN_TXN_PINNED_TIMESTAMP_READER 1402 +#define WT_STAT_CONN_TXN_PINNED_TIMESTAMP_READER 1409 /*! * transaction: transaction range of timestamps pinned by the oldest * timestamp */ -#define WT_STAT_CONN_TXN_PINNED_TIMESTAMP_OLDEST 1403 +#define WT_STAT_CONN_TXN_PINNED_TIMESTAMP_OLDEST 1410 /*! transaction: transaction read timestamp of the oldest active reader */ -#define WT_STAT_CONN_TXN_TIMESTAMP_OLDEST_ACTIVE_READ 1404 +#define WT_STAT_CONN_TXN_TIMESTAMP_OLDEST_ACTIVE_READ 1411 /*! transaction: transaction sync calls */ -#define WT_STAT_CONN_TXN_SYNC 1405 +#define WT_STAT_CONN_TXN_SYNC 1412 /*! transaction: transactions committed */ -#define WT_STAT_CONN_TXN_COMMIT 1406 +#define WT_STAT_CONN_TXN_COMMIT 1413 /*! transaction: transactions rolled back */ -#define WT_STAT_CONN_TXN_ROLLBACK 1407 +#define WT_STAT_CONN_TXN_ROLLBACK 1414 /*! transaction: update conflicts */ -#define WT_STAT_CONN_TXN_UPDATE_CONFLICT 1408 +#define WT_STAT_CONN_TXN_UPDATE_CONFLICT 1415 /*! * @} @@ -5970,32 +5974,32 @@ extern int wiredtiger_extension_terminate(WT_CONNECTION *connection); #define WT_STAT_DSRC_CACHE_EVICTION_WALK_SAVED_POS 2059 /*! cache: hazard pointer blocked page eviction */ #define WT_STAT_DSRC_CACHE_EVICTION_HAZARD 2060 +/*! cache: history store table reads */ +#define WT_STAT_DSRC_CACHE_HS_READ 2061 /*! cache: in-memory page passed criteria to be split */ -#define WT_STAT_DSRC_CACHE_INMEM_SPLITTABLE 2061 +#define WT_STAT_DSRC_CACHE_INMEM_SPLITTABLE 2062 /*! cache: in-memory page splits */ -#define WT_STAT_DSRC_CACHE_INMEM_SPLIT 2062 +#define WT_STAT_DSRC_CACHE_INMEM_SPLIT 2063 /*! cache: internal pages evicted */ -#define WT_STAT_DSRC_CACHE_EVICTION_INTERNAL 2063 +#define WT_STAT_DSRC_CACHE_EVICTION_INTERNAL 2064 /*! cache: internal pages split during eviction */ -#define WT_STAT_DSRC_CACHE_EVICTION_SPLIT_INTERNAL 2064 +#define WT_STAT_DSRC_CACHE_EVICTION_SPLIT_INTERNAL 2065 /*! cache: leaf pages split during eviction */ -#define WT_STAT_DSRC_CACHE_EVICTION_SPLIT_LEAF 2065 +#define WT_STAT_DSRC_CACHE_EVICTION_SPLIT_LEAF 2066 /*! cache: modified pages evicted */ -#define WT_STAT_DSRC_CACHE_EVICTION_DIRTY 2066 +#define WT_STAT_DSRC_CACHE_EVICTION_DIRTY 2067 /*! cache: overflow pages read into cache */ -#define WT_STAT_DSRC_CACHE_READ_OVERFLOW 2067 +#define WT_STAT_DSRC_CACHE_READ_OVERFLOW 2068 /*! cache: page split during eviction deepened the tree */ -#define WT_STAT_DSRC_CACHE_EVICTION_DEEPEN 2068 -/*! cache: page written requiring cache overflow records */ -#define WT_STAT_DSRC_CACHE_WRITE_LOOKASIDE 2069 +#define WT_STAT_DSRC_CACHE_EVICTION_DEEPEN 2069 +/*! cache: page written requiring history store records */ +#define WT_STAT_DSRC_CACHE_WRITE_HS 2070 /*! cache: pages read into cache */ -#define WT_STAT_DSRC_CACHE_READ 2070 +#define WT_STAT_DSRC_CACHE_READ 2071 /*! cache: pages read into cache after truncate */ -#define WT_STAT_DSRC_CACHE_READ_DELETED 2071 +#define WT_STAT_DSRC_CACHE_READ_DELETED 2072 /*! cache: pages read into cache after truncate in prepare state */ -#define WT_STAT_DSRC_CACHE_READ_DELETED_PREPARED 2072 -/*! cache: pages read into cache requiring cache overflow entries */ -#define WT_STAT_DSRC_CACHE_READ_LOOKASIDE 2073 +#define WT_STAT_DSRC_CACHE_READ_DELETED_PREPARED 2073 /*! cache: pages requested from the cache */ #define WT_STAT_DSRC_CACHE_PAGES_REQUESTED 2074 /*! cache: pages seen by eviction walk */ diff --git a/src/third_party/wiredtiger/src/include/wt_internal.h b/src/third_party/wiredtiger/src/include/wt_internal.h index e9f3c09c9a3..9b3bda96e7a 100644 --- a/src/third_party/wiredtiger/src/include/wt_internal.h +++ b/src/third_party/wiredtiger/src/include/wt_internal.h @@ -67,6 +67,8 @@ extern "C" { */ struct __wt_addr; typedef struct __wt_addr WT_ADDR; +struct __wt_addr_copy; +typedef struct __wt_addr_copy WT_ADDR_COPY; struct __wt_async; typedef struct __wt_async WT_ASYNC; struct __wt_async_cursor; @@ -237,6 +239,8 @@ struct __wt_lsm_worker_args; typedef struct __wt_lsm_worker_args WT_LSM_WORKER_ARGS; struct __wt_lsm_worker_cookie; typedef struct __wt_lsm_worker_cookie WT_LSM_WORKER_COOKIE; +struct __wt_modify_vector; +typedef struct __wt_modify_vector WT_MODIFY_VECTOR; struct __wt_multi; typedef struct __wt_multi WT_MULTI; struct __wt_myslot; @@ -269,8 +273,6 @@ struct __wt_page_header; typedef struct __wt_page_header WT_PAGE_HEADER; struct __wt_page_index; typedef struct __wt_page_index WT_PAGE_INDEX; -struct __wt_page_lookaside; -typedef struct __wt_page_lookaside WT_PAGE_LOOKASIDE; struct __wt_page_modify; typedef struct __wt_page_modify WT_PAGE_MODIFY; struct __wt_process; @@ -315,6 +317,8 @@ struct __wt_thread; typedef struct __wt_thread WT_THREAD; struct __wt_thread_group; typedef struct __wt_thread_group WT_THREAD_GROUP; +struct __wt_time_pair; +typedef struct __wt_time_pair WT_TIME_PAIR; struct __wt_txn; typedef struct __wt_txn WT_TXN; struct __wt_txn_global; diff --git a/src/third_party/wiredtiger/src/log/log.c b/src/third_party/wiredtiger/src/log/log.c index 4c1f6ada07d..b7b7ff66c58 100644 --- a/src/third_party/wiredtiger/src/log/log.c +++ b/src/third_party/wiredtiger/src/log/log.c @@ -1894,7 +1894,7 @@ __wt_log_release(WT_SESSION_IMPL *session, WT_LOGSLOT *slot, bool *freep) * After this point the worker thread owns the slot. There is nothing more to do but return. */ /* - * !!! Signalling the wrlsn_cond condition here results in + * !!! Signaling the wrlsn_cond condition here results in * worse performance because it causes more scheduling churn * and more walking of the slot pool for a very small number * of slots to process. Don't signal here. diff --git a/src/third_party/wiredtiger/src/meta/meta_ckpt.c b/src/third_party/wiredtiger/src/meta/meta_ckpt.c index 2f0caf17def..db846698277 100644 --- a/src/third_party/wiredtiger/src/meta/meta_ckpt.c +++ b/src/third_party/wiredtiger/src/meta/meta_ckpt.c @@ -617,59 +617,48 @@ format: } /* - * __wt_metadata_set_base_write_gen -- - * Set the connection's base write generation. + * __wt_metadata_update_base_write_gen -- + * Update the connection's base write generation. */ int -__wt_metadata_set_base_write_gen(WT_SESSION_IMPL *session) +__wt_metadata_update_base_write_gen(WT_SESSION_IMPL *session, const char *config) { WT_CKPT ckpt; + WT_CONNECTION_IMPL *conn; + WT_DECL_RET; - WT_RET(__wt_meta_checkpoint(session, WT_METAFILE_URI, NULL, &ckpt)); - - /* - * We track the maximum page generation we've ever seen, and I'm not interested in debugging - * off-by-ones. - */ - S2C(session)->base_write_gen = ckpt.write_gen + 1; + conn = S2C(session); + memset(&ckpt, 0, sizeof(ckpt)); - __wt_meta_checkpoint_free(session, &ckpt); + if ((ret = __ckpt_last(session, config, &ckpt)) == 0) { + conn->base_write_gen = WT_MAX(ckpt.write_gen + 1, conn->base_write_gen); + __wt_meta_checkpoint_free(session, &ckpt); + } else + WT_RET_NOTFOUND_OK(ret); return (0); } /* - * __ckptlist_review_write_gen -- - * Review the checkpoint's write generation. + * __wt_metadata_init_base_write_gen -- + * Initialize the connection's base write generation. */ -static void -__ckptlist_review_write_gen(WT_SESSION_IMPL *session, WT_CKPT *ckpt) +int +__wt_metadata_init_base_write_gen(WT_SESSION_IMPL *session) { - uint64_t v; + WT_DECL_RET; + char *config; - /* - * Every page written in a given wiredtiger_open() session needs to be in a single "generation", - * it's how we know to ignore transactional information found on pages written in previous - * generations. We make this work by writing the maximum write generation we've ever seen as the - * write-generation of the metadata file's checkpoint. When wiredtiger_open() is called, we copy - * that write generation into the connection's name space as the base write generation value. - * Then, whenever we open a file, if the file's write generation is less than the base value, we - * update the file's write generation so all writes will appear after the base value, and we - * ignore transactions on pages where the write generation is less than the base value. - * - * At every checkpoint, if the file's checkpoint write generation is larger than the - * connection's maximum write generation, update the connection. - */ - do { - WT_ORDERED_READ(v, S2C(session)->max_write_gen); - } while ( - ckpt->write_gen > v && !__wt_atomic_cas64(&S2C(session)->max_write_gen, v, ckpt->write_gen)); + /* Initialize the base write gen to 1 */ + S2C(session)->base_write_gen = 1; + /* Retrieve the metadata entry for the metadata file. */ + WT_ERR(__wt_metadata_search(session, WT_METAFILE_URI, &config)); + /* Update base write gen to the write gen of metadata. */ + WT_ERR(__wt_metadata_update_base_write_gen(session, config)); - /* - * If checkpointing the metadata file, update its write generation to be the maximum we've seen. - */ - if (session->dhandle != NULL && WT_IS_METADATA(session->dhandle) && ckpt->write_gen < v) - ckpt->write_gen = v; +err: + __wt_free(session, config); + return (ret); } /* @@ -760,7 +749,7 @@ __ckpt_blkmod_to_meta(WT_SESSION_IMPL *session, WT_ITEM *buf, WT_CKPT *ckpt) if (!F_ISSET(blk, WT_BLOCK_MODS_VALID)) continue; WT_RET(__wt_raw_to_hex(session, blk->bitstring.data, blk->bitstring.size, &bitstring)); - WT_RET(__wt_buf_catfmt(session, buf, "%s%s=(id=%" PRIu32 ",granularity=%" PRIu64 + WT_RET(__wt_buf_catfmt(session, buf, "%s\"%s\"=(id=%" PRIu32 ",granularity=%" PRIu64 ",nbits=%" PRIu64 ",offset=%" PRIu64 ",blocks=%.*s)", i == 0 ? "" : ",", blk->id_str, i, blk->granularity, blk->nbits, blk->offset, (int)bitstring.size, (char *)bitstring.data)); @@ -800,10 +789,6 @@ __wt_meta_ckptlist_set( WT_ERR(__ckpt_set(session, fname, buf->mem, has_lsn)); - /* Review the checkpoint's write generation. */ - WT_CKPT_FOREACH (ckptbase, ckpt) - __ckptlist_review_write_gen(session, ckpt); - err: __wt_scr_free(session, &buf); return (ret); diff --git a/src/third_party/wiredtiger/src/meta/meta_table.c b/src/third_party/wiredtiger/src/meta/meta_table.c index 1e573f5f4a0..7c1a8a80dc9 100644 --- a/src/third_party/wiredtiger/src/meta/meta_table.c +++ b/src/third_party/wiredtiger/src/meta/meta_table.c @@ -343,3 +343,61 @@ __wt_metadata_salvage(WT_SESSION_IMPL *session) WT_RET(wt_session->salvage(wt_session, WT_METAFILE_URI, NULL)); return (0); } + +/* + * __wt_metadata_uri_to_btree_id -- + * Given a uri, find the btree id from the metadata. WT_NOTFOUND is returned for a non-file uri. + */ +int +__wt_metadata_uri_to_btree_id(WT_SESSION_IMPL *session, const char *uri, uint32_t *btree_id) +{ + WT_CONFIG_ITEM id; + WT_DECL_RET; + char *value; + + value = NULL; + + if (!WT_PREFIX_MATCH(uri, "file:")) + return (WT_NOTFOUND); + + WT_ERR(__wt_metadata_search(session, uri, &value)); + WT_ERR(__wt_config_getones(session, value, "id", &id)); + *btree_id = (uint32_t)id.val; + +err: + __wt_free(session, value); + return (ret); +} + +/* + * __wt_metadata_btree_id_to_uri -- + * Given a btree id, find the matching entry in the metadata and return a copy of the uri. The + * caller has to free the returned uri. + */ +int +__wt_metadata_btree_id_to_uri(WT_SESSION_IMPL *session, uint32_t btree_id, char **uri) +{ + WT_CONFIG_ITEM id; + WT_CURSOR *cursor; + WT_DECL_RET; + char *key, *value; + + *uri = NULL; + key = NULL; + + WT_RET(__wt_metadata_cursor(session, &cursor)); + while ((ret = cursor->next(cursor)) == 0) { + WT_ERR(cursor->get_value(cursor, &value)); + if ((ret = __wt_config_getones(session, value, "id", &id)) == 0 && btree_id == id.val) { + WT_ERR(cursor->get_key(cursor, &key)); + /* Return a copy as the uri. */ + WT_ERR(__wt_strdup(session, key, uri)); + break; + } + WT_ERR_NOTFOUND_OK(ret); + } + +err: + WT_TRET(__wt_metadata_cursor_release(session, &cursor)); + return (ret); +} diff --git a/src/third_party/wiredtiger/src/reconcile/rec_child.c b/src/third_party/wiredtiger/src/reconcile/rec_child.c index f3e98f591fc..2d3f17a22af 100644 --- a/src/third_party/wiredtiger/src/reconcile/rec_child.c +++ b/src/third_party/wiredtiger/src/reconcile/rec_child.c @@ -33,9 +33,12 @@ __rec_child_deleted(WT_SESSION_IMPL *session, WT_RECONCILE *r, WT_REF *ref, WT_C * A visible update to be in READY state (i.e. not in LOCKED or PREPARED state), for truly * visible to others. */ - if (F_ISSET(r, WT_REC_VISIBILITY_ERR) && page_del != NULL && - __wt_page_del_active(session, ref, false)) - WT_PANIC_RET(session, EINVAL, "reconciliation illegally skipped an update"); + if (F_ISSET(r, WT_REC_CLEAN_AFTER_REC | WT_REC_VISIBILITY_ERR) && page_del != NULL && + __wt_page_del_active(session, ref, false)) { + if (F_ISSET(r, WT_REC_VISIBILITY_ERR)) + WT_PANIC_RET(session, EINVAL, "reconciliation illegally skipped an update"); + return (__wt_set_return(session, EBUSY)); + } /* * Deal with any underlying disk blocks. @@ -136,6 +139,7 @@ __wt_rec_child_modify( switch (r->tested_ref_state = ref->state) { case WT_REF_DISK: /* On disk, not modified by definition. */ + WT_ASSERT(session, ref->addr != NULL); goto done; case WT_REF_DELETED: @@ -159,43 +163,21 @@ __wt_rec_child_modify( * We should never be here during eviction, active child pages in an evicted page's * subtree fails the eviction attempt. */ - WT_ASSERT(session, !F_ISSET(r, WT_REC_EVICT)); - if (F_ISSET(r, WT_REC_EVICT)) - return (__wt_set_return(session, EBUSY)); - - /* - * If called during checkpoint, the child is being considered by the eviction server or - * the child is a truncated page being read. The eviction may have started before the - * checkpoint and so we must wait for the eviction to be resolved. I suspect we could - * handle reads of truncated pages, but we can't distinguish between the two and reads - * of truncated pages aren't expected to be common. - */ - break; + WT_RET_ASSERT(session, !F_ISSET(r, WT_REC_EVICT), EBUSY, + "unexpected WT_REF_LOCKED child state during eviction reconciliation"); - case WT_REF_LIMBO: - WT_ASSERT(session, !F_ISSET(r, WT_REC_EVICT)); - /* FALLTHROUGH */ - case WT_REF_LOOKASIDE: - /* - * On disk or in cache with lookaside updates. - * - * We should never be here during eviction: active child pages in an evicted page's - * subtree fails the eviction attempt. - */ - if (F_ISSET(r, WT_REC_EVICT) && __wt_page_las_active(session, ref)) { - WT_ASSERT(session, false); - return (__wt_set_return(session, EBUSY)); - } + /* If the page is being read from disk, it's not modified by definition. */ + if (F_ISSET(ref, WT_REF_FLAG_READING)) + goto done; /* - * A page evicted with lookaside entries may not have an address, if no updates were - * visible to reconciliation. Any child pages in that state should be ignored. + * Otherwise, the child is being considered by the eviction server or the child is a + * deleted page being read. The eviction may have started before the checkpoint and so + * we must wait for the eviction to be resolved. I suspect we could handle reads of + * deleted pages, but we can't distinguish between the two and reads of deleted pages + * aren't expected to be common. */ - if (ref->addr == NULL) { - *statep = WT_CHILD_IGNORE; - WT_CHILD_RELEASE(session, *hazardp, ref); - } - goto done; + break; case WT_REF_MEM: /* @@ -204,9 +186,8 @@ __wt_rec_child_modify( * We should never be here during eviction, active child pages in an evicted page's * subtree fails the eviction attempt. */ - WT_ASSERT(session, !F_ISSET(r, WT_REC_EVICT)); - if (F_ISSET(r, WT_REC_EVICT)) - return (__wt_set_return(session, EBUSY)); + WT_RET_ASSERT(session, !F_ISSET(r, WT_REC_EVICT), EBUSY, + "unexpected WT_REF_MEM child state during eviction reconciliation"); /* * If called during checkpoint, acquire a hazard pointer so the child isn't evicted, @@ -229,18 +210,6 @@ __wt_rec_child_modify( *hazardp = true; goto in_memory; - case WT_REF_READING: - /* - * Being read, not modified by definition. - * - * We should never be here during eviction, active child pages in an evicted page's - * subtree fails the eviction attempt. - */ - WT_ASSERT(session, !F_ISSET(r, WT_REC_EVICT)); - if (F_ISSET(r, WT_REC_EVICT)) - return (__wt_set_return(session, EBUSY)); - goto done; - case WT_REF_SPLIT: /* * The page was split out from under us. @@ -252,8 +221,10 @@ __wt_rec_child_modify( * checkpoint, all splits in process will have completed before we walk any pages for * checkpoint. */ - WT_ASSERT(session, WT_REF_SPLIT != WT_REF_SPLIT); - return (__wt_set_return(session, EBUSY)); + WT_RET_ASSERT( + session, false, EBUSY, "unexpected WT_REF_SPLIT child state during reconciliation"); + /* NOTREACHED */ + return (EBUSY); default: return (__wt_illegal_value(session, r->tested_ref_state)); diff --git a/src/third_party/wiredtiger/src/reconcile/rec_col.c b/src/third_party/wiredtiger/src/reconcile/rec_col.c index eea1fc58ee7..adf2db1a76d 100644 --- a/src/third_party/wiredtiger/src/reconcile/rec_col.c +++ b/src/third_party/wiredtiger/src/reconcile/rec_col.c @@ -170,7 +170,7 @@ __rec_col_merge(WT_SESSION_IMPL *session, WT_RECONCILE *r, WT_PAGE *page) /* Build the value cell. */ addr = &multi->addr; - __wt_rec_cell_build_addr(session, r, addr, false, r->recno); + __wt_rec_cell_build_addr(session, r, addr, NULL, false, r->recno); /* Boundary: split or write the page. */ if (__wt_rec_need_split(r, val->len)) @@ -178,7 +178,11 @@ __rec_col_merge(WT_SESSION_IMPL *session, WT_RECONCILE *r, WT_PAGE *page) /* Copy the value onto the page. */ __wt_rec_image_copy(session, r, val); - __wt_rec_addr_ts_update(r, addr->newest_durable_ts, addr->oldest_start_ts, + /* + * FIXME-prepare-support: audit the use of durable timestamps in this file, use both durable + * timestamps. + */ + __wt_rec_addr_ts_update(r, addr->start_durable_ts, addr->oldest_start_ts, addr->oldest_start_txn, addr->newest_stop_ts, addr->newest_stop_txn); } return (0); @@ -281,14 +285,14 @@ __wt_rec_col_int(WT_SESSION_IMPL *session, WT_RECONCILE *r, WT_REF *pageref) val->buf.size = __wt_cell_total_len(vpack); val->cell_len = 0; val->len = val->buf.size; - newest_durable_ts = vpack->newest_durable_ts; + newest_durable_ts = vpack->newest_stop_durable_ts; oldest_start_ts = vpack->oldest_start_ts; oldest_start_txn = vpack->oldest_start_txn; newest_stop_ts = vpack->newest_stop_ts; newest_stop_txn = vpack->newest_stop_txn; } else { - __wt_rec_cell_build_addr(session, r, addr, false, ref->ref_recno); - newest_durable_ts = addr->newest_durable_ts; + __wt_rec_cell_build_addr(session, r, addr, NULL, false, ref->ref_recno); + newest_durable_ts = addr->stop_durable_ts; oldest_start_ts = addr->oldest_start_ts; oldest_start_txn = addr->oldest_start_txn; newest_stop_ts = addr->newest_stop_ts; @@ -323,6 +327,7 @@ int __wt_rec_col_fix(WT_SESSION_IMPL *session, WT_RECONCILE *r, WT_REF *pageref) { WT_BTREE *btree; + WT_DECL_RET; WT_INSERT *ins; WT_PAGE *page; WT_UPDATE *upd; @@ -332,6 +337,7 @@ __wt_rec_col_fix(WT_SESSION_IMPL *session, WT_RECONCILE *r, WT_REF *pageref) btree = S2BT(session); page = pageref->page; + upd = NULL; WT_RET(__wt_rec_split_init(session, r, page, pageref->ref_recno, btree->maxleafpage)); @@ -342,9 +348,13 @@ __wt_rec_col_fix(WT_SESSION_IMPL *session, WT_RECONCILE *r, WT_REF *pageref) WT_SKIP_FOREACH (ins, WT_COL_UPDATE_SINGLE(page)) { WT_RET(__wt_rec_upd_select(session, r, ins, NULL, NULL, &upd_select)); upd = upd_select.upd; - if (upd != NULL) + if (upd != NULL) { __bit_setv( r->first_free, WT_INSERT_RECNO(ins) - pageref->ref_recno, btree->bitcnt, *upd->data); + /* Free the update if it is external. */ + if (F_ISSET(upd, WT_UPDATE_RESTORED_FROM_DISK)) + __wt_free_update_list(session, &upd); + } } /* Calculate the number of entries per page remainder. */ @@ -410,13 +420,17 @@ __wt_rec_col_fix(WT_SESSION_IMPL *session, WT_RECONCILE *r, WT_REF *pageref) * last, allowing it to grow in the future. */ __wt_rec_incr(session, r, entry, __bitstr_size((size_t)entry * btree->bitcnt)); - WT_RET(__wt_rec_split(session, r, 0, false)); + WT_ERR(__wt_rec_split(session, r, 0, false)); /* Calculate the number of entries per page. */ entry = 0; nrecs = WT_FIX_BYTES_TO_ENTRIES(btree, r->space_avail); } + /* Free the update if it is external. */ + if (upd != NULL && F_ISSET(upd, WT_UPDATE_RESTORED_FROM_DISK)) + __wt_free_update_list(session, &upd); + /* * Execute this loop once without an insert item to catch any missing records due to a * split, then quit. @@ -429,7 +443,14 @@ __wt_rec_col_fix(WT_SESSION_IMPL *session, WT_RECONCILE *r, WT_REF *pageref) __wt_rec_incr(session, r, entry, __bitstr_size((size_t)entry * btree->bitcnt)); /* Write the remnant page. */ - return (__wt_rec_split_finish(session, r)); + ret = __wt_rec_split_finish(session, r); + +err: + /* Free the update if it is external. */ + if (upd != NULL && F_ISSET(upd, WT_UPDATE_RESTORED_FROM_DISK)) + __wt_free_update_list(session, &upd); + + return (ret); } /* @@ -449,16 +470,13 @@ __wt_rec_col_fix_slvg( page = pageref->page; /* - * !!! - * It's vanishingly unlikely and probably impossible for fixed-length - * column-store files to have overlapping key ranges. It's possible - * for an entire key range to go missing (if a page is corrupted and - * lost), but because pages can't split, it shouldn't be possible to - * find pages where the key ranges overlap. That said, we check for - * it during salvage and clean up after it here because it doesn't - * cost much and future column-store formats or operations might allow - * for fixed-length format ranges to overlap during salvage, and I - * don't want to have to retrofit the code later. + * It's vanishingly unlikely and probably impossible for fixed-length column-store files to have + * overlapping key ranges. It's possible for an entire key range to go missing (if a page is + * corrupted and lost), but because pages can't split, it shouldn't be possible to find pages + * where the key ranges overlap. That said, we check for it during salvage and clean up after it + * here because it doesn't cost much and future column-store formats or operations might allow + * for fixed-length format ranges to overlap during salvage, and I don't want to have to + * retrofit the code later. */ WT_RET(__wt_rec_split_init(session, r, page, pageref->ref_recno, btree->maxleafpage)); @@ -621,10 +639,10 @@ __wt_rec_col_var( if ((addr = pageref->addr) == NULL) newest_durable_ts = WT_TS_NONE; else if (__wt_off_page(pageref->home, addr)) - newest_durable_ts = addr->newest_durable_ts; + newest_durable_ts = addr->stop_durable_ts; else { __wt_cell_unpack(session, pageref->home, pageref->addr, vpack); - newest_durable_ts = vpack->newest_durable_ts; + newest_durable_ts = vpack->newest_stop_durable_ts; } /* Set the "last" values to cause failure if they're not set. */ @@ -717,7 +735,7 @@ __wt_rec_col_var( * record, and in that case we'll do the comparisons, but we don't read overflow items just * to see if they match records on either side. */ - if (vpack->ovfl) { + if (F_ISSET(vpack, WT_CELL_UNPACK_OVERFLOW)) { ovfl_state = OVFL_UNUSED; goto record_loop; } @@ -737,40 +755,25 @@ record_loop: * record number. The WT_INSERT lists are in sorted order, so only need check the next one. */ for (n = 0; n < nrepeat; n += repeat_count, src_recno += repeat_count) { - durable_ts = newest_durable_ts; - start_ts = vpack->start_ts; - start_txn = vpack->start_txn; - stop_ts = vpack->stop_ts; - stop_txn = vpack->stop_txn; upd = NULL; if (ins != NULL && WT_INSERT_RECNO(ins) == src_recno) { WT_ERR(__wt_rec_upd_select(session, r, ins, cip, vpack, &upd_select)); upd = upd_select.upd; - if (upd == NULL) { - /* - * TIMESTAMP-FIXME I'm pretty sure this is wrong: a NULL update means an item - * was deleted, and I think that requires a tombstone on the page. - */ - durable_ts = WT_TS_NONE; - start_ts = WT_TS_NONE; - start_txn = WT_TXN_NONE; - stop_ts = WT_TS_MAX; - stop_txn = WT_TXN_MAX; - } else { - durable_ts = upd_select.durable_ts; - start_ts = upd_select.start_ts; - start_txn = upd_select.start_txn; - stop_ts = upd_select.stop_ts; - stop_txn = upd_select.stop_txn; - } ins = WT_SKIP_NEXT(ins); } - update_no_copy = true; /* No data copy */ - repeat_count = 1; /* Single record */ + update_no_copy = + upd == NULL || !F_ISSET(upd, WT_UPDATE_RESTORED_FROM_DISK); /* No data copy */ + repeat_count = 1; /* Single record */ deleted = false; if (upd != NULL) { + durable_ts = upd_select.durable_ts; + start_ts = upd_select.start_ts; + start_txn = upd_select.start_txn; + stop_ts = upd_select.stop_ts; + stop_txn = upd_select.stop_txn; + switch (upd->type) { case WT_UPDATE_MODIFY: cbt->slot = WT_COL_SLOT(page, cip); @@ -784,29 +787,16 @@ record_loop: size = upd->size; break; case WT_UPDATE_TOMBSTONE: + durable_ts = WT_TS_NONE; + start_ts = WT_TS_NONE; + start_txn = WT_TXN_NONE; + stop_ts = WT_TS_MAX; + stop_txn = WT_TXN_MAX; deleted = true; break; default: WT_ERR(__wt_illegal_value(session, upd->type)); } - } else if (vpack->raw == WT_CELL_VALUE_OVFL_RM) { - /* - * If doing an update save and restore, and the underlying value is a removed - * overflow value, we end up here. - * - * If necessary, when the overflow value was originally removed, reconciliation - * appended a globally visible copy of the value to the key's update list, meaning - * the on-page item isn't accessed after page re-instantiation. - * - * Assert the case. - */ - WT_ASSERT(session, F_ISSET(r, WT_REC_UPDATE_RESTORE)); - - /* - * The on-page value will never be accessed, write a placeholder record. - */ - data = "ovfl-unused"; - size = WT_STORE_SIZE(strlen("ovfl-unused")); } else { update_no_copy = false; /* Maybe data copy */ @@ -820,8 +810,28 @@ record_loop: repeat_count = WT_INSERT_RECNO(ins) - src_recno; deleted = orig_deleted; - if (deleted) + if (deleted) { + /* Set time pairs for the deleted key. */ + durable_ts = WT_TS_NONE; + start_ts = WT_TS_NONE; + start_txn = WT_TXN_NONE; + stop_ts = WT_TS_MAX; + stop_txn = WT_TXN_MAX; + goto compare; + } + + /* + * The key on the old disk image is unchanged. Use time pairs from the cell. + * + * FIXME-prepare-support: Currently, we don't store durable_ts in cell, which is a + * problem we need to solve for prepared transactions. + */ + durable_ts = newest_durable_ts; + start_ts = vpack->start_ts; + start_txn = vpack->start_txn; + stop_ts = vpack->stop_ts; + stop_txn = vpack->stop_txn; /* * If we are handling overflow items, use the overflow item itself exactly once, @@ -880,12 +890,20 @@ compare: * record number, we've been doing that all along. */ if (rle != 0) { - if ((!__wt_process.page_version_ts || - (last.start_ts == start_ts && last.start_txn == start_txn && - last.stop_ts == stop_ts && last.stop_txn == stop_txn)) && + if ((last.start_ts == start_ts && last.start_txn == start_txn && + last.stop_ts == stop_ts && last.stop_txn == stop_txn) && ((deleted && last.deleted) || (!deleted && !last.deleted && last.value->size == size && memcmp(last.value->data, data, size) == 0))) { + /* + * The start time pair for deleted keys must be (WT_TS_NONE, WT_TXN_NONE) and + * stop time pair must be (WT_TS_MAX, WT_TXN_MAX) since we no longer select + * tombstone to write to disk and the deletion of the keys must be globally + * visible. + */ + WT_ASSERT(session, (!deleted && !last.deleted) || + (last.start_ts == WT_TS_NONE && last.start_txn == WT_TXN_NONE && + last.stop_ts == WT_TS_MAX && last.stop_txn == WT_TXN_MAX)); rle += repeat_count; continue; } @@ -901,17 +919,12 @@ compare: */ if (!deleted) { /* - * We can't simply assign the data values into - * the last buffer because they may have come - * from a copy built from an encoded/overflow - * cell and creating the next record is going - * to overwrite that memory. Check, because - * encoded/overflow cells aren't that common - * and we'd like to avoid the copy. If data - * was taken from the current unpack structure - * (which points into the page), or was taken - * from an update structure, we can just use - * the pointers, they're not moving. + * We can't simply assign the data values into the last buffer because they may have + * come from a copy built from an encoded/overflow cell and creating the next record + * is going to overwrite that memory. Check, because encoded/overflow cells aren't + * that common and we'd like to avoid the copy. If data was taken from the current + * unpack structure (which points into the page), or was taken from an update + * structure, we can just use the pointers, they're not moving. */ if (data == vpack->data || update_no_copy) { last.value->data = data; @@ -919,6 +932,11 @@ compare: } else WT_ERR(__wt_buf_set(session, last.value, data, size)); } + + /* Free the update if it is external. */ + if (upd != NULL && F_ISSET(upd, WT_UPDATE_RESTORED_FROM_DISK)) + __wt_free_update_list(session, &upd); + last.start_ts = start_ts; last.start_txn = start_txn; last.stop_ts = stop_ts; @@ -967,26 +985,11 @@ compare: upd = upd_select.upd; n = WT_INSERT_RECNO(ins); } - if (upd == NULL) { - /* - * TIMESTAMP-FIXME I'm pretty sure this is wrong: a NULL update means an item was - * deleted, and I think that requires a tombstone on the page. - */ - durable_ts = WT_TS_NONE; - start_ts = WT_TS_NONE; - start_txn = WT_TXN_NONE; - stop_ts = WT_TS_MAX; - stop_txn = WT_TXN_MAX; - } else { - durable_ts = upd_select.durable_ts; - start_ts = upd_select.start_ts; - start_txn = upd_select.start_txn; - stop_ts = upd_select.stop_ts; - stop_txn = upd_select.stop_txn; - } + while (src_recno <= n) { + update_no_copy = + upd == NULL || !F_ISSET(upd, WT_UPDATE_RESTORED_FROM_DISK); /* No data copy */ deleted = false; - update_no_copy = true; /* * The application may have inserted records which left gaps in the name space, and @@ -994,9 +997,16 @@ compare: */ if (src_recno < n) { deleted = true; - if (last.deleted && (!__wt_process.page_version_ts || - (last.start_ts == start_ts && last.start_txn == start_txn && - last.stop_ts == stop_ts && last.stop_txn == stop_txn))) { + if (last.deleted) { + /* + * The start time pair for deleted keys must be (WT_TS_NONE, WT_TXN_NONE) and + * stop time pair must be (WT_TS_MAX, WT_TXN_MAX) since we no longer select + * tombstone to write to disk and the deletion of the keys must be globally + * visible. + */ + WT_ASSERT(session, last.start_ts == WT_TS_NONE && + last.start_txn == WT_TXN_NONE && last.stop_ts == WT_TS_MAX && + last.stop_txn == WT_TXN_MAX); /* * The record adjustment is decremented by one so we can naturally fall into the * RLE accounting below, where we increment rle by one, then continue in the @@ -1005,12 +1015,16 @@ compare: skip = (n - src_recno) - 1; rle += skip; src_recno += skip; + } else { + /* Set time pairs for the first deleted key in a deleted range. */ + durable_ts = WT_TS_NONE; + start_ts = WT_TS_NONE; + start_txn = WT_TXN_NONE; + stop_ts = WT_TS_MAX; + stop_txn = WT_TXN_MAX; } } else if (upd == NULL) { - /* - * TIMESTAMP-FIXME I'm pretty sure this is wrong: a NULL update means an item was - * deleted, and I think that requires a tombstone on the page. - */ + /* The updates on the key are all uncommitted so we write a deleted key to disk. */ durable_ts = WT_TS_NONE; start_ts = WT_TS_NONE; start_txn = WT_TXN_NONE; @@ -1019,6 +1033,7 @@ compare: deleted = true; } else { + /* Set time pairs for a key. */ durable_ts = upd_select.durable_ts; start_ts = upd_select.start_ts; start_txn = upd_select.start_txn; @@ -1041,6 +1056,11 @@ compare: size = upd->size; break; case WT_UPDATE_TOMBSTONE: + durable_ts = WT_TS_NONE; + start_ts = WT_TS_NONE; + start_txn = WT_TXN_NONE; + stop_ts = WT_TS_MAX; + stop_txn = WT_TXN_MAX; deleted = true; break; default: @@ -1053,12 +1073,23 @@ compare: * the same thing. */ if (rle != 0) { - if ((!__wt_process.page_version_ts || - (last.start_ts == start_ts && last.start_txn == start_txn && - last.stop_ts == stop_ts && last.stop_txn == stop_txn)) && + /* + * FIXME-PM-1521: Follow up issue with clang in WT-5341. + */ + if ((last.start_ts == start_ts && last.start_txn == start_txn && + last.stop_ts == stop_ts && last.stop_txn == stop_txn) && ((deleted && last.deleted) || (!deleted && !last.deleted && last.value->size == size && memcmp(last.value->data, data, size) == 0))) { + /* + * The start time pair for deleted keys must be (WT_TS_NONE, WT_TXN_NONE) and + * stop time pair must be (WT_TS_MAX, WT_TXN_MAX) since we no longer select + * tombstone to write to disk and the deletion of the keys must be globally + * visible. + */ + WT_ASSERT(session, (!deleted && !last.deleted) || + (last.start_ts == WT_TS_NONE && last.start_txn == WT_TXN_NONE && + last.stop_ts == WT_TS_MAX && last.stop_txn == WT_TXN_MAX)); ++rle; goto next; } @@ -1082,6 +1113,10 @@ compare: WT_ERR(__wt_buf_set(session, last.value, data, size)); } + /* Free the update if it is external. */ + if (upd != NULL && F_ISSET(upd, WT_UPDATE_RESTORED_FROM_DISK)) + __wt_free_update_list(session, &upd); + /* Ready for the next loop, reset the RLE counter. */ last.start_ts = start_ts; last.start_txn = start_txn; @@ -1117,6 +1152,10 @@ next: ret = __wt_rec_split_finish(session, r); err: + /* Free the update if it is external. */ + if (upd != NULL && F_ISSET(upd, WT_UPDATE_RESTORED_FROM_DISK)) + __wt_free_update_list(session, &upd); + __wt_scr_free(session, &orig); return (ret); } diff --git a/src/third_party/wiredtiger/src/reconcile/rec_row.c b/src/third_party/wiredtiger/src/reconcile/rec_row.c index bc24c661b27..443ac43e186 100644 --- a/src/third_party/wiredtiger/src/reconcile/rec_row.c +++ b/src/third_party/wiredtiger/src/reconcile/rec_row.c @@ -205,8 +205,8 @@ __wt_bulk_insert_row(WT_SESSION_IMPL *session, WT_CURSOR_BULK *cbulk) val = &r->v; WT_RET(__rec_cell_build_leaf_key(session, r, /* Build key cell */ cursor->key.data, cursor->key.size, &ovfl_key)); - WT_RET(__wt_rec_cell_build_val(session, r, /* Build value cell */ - cursor->value.data, cursor->value.size, WT_TS_NONE, WT_TXN_NONE, WT_TS_MAX, WT_TXN_MAX, 0)); + WT_RET(__wt_rec_cell_build_val(session, r, cursor->value.data, /* Build value cell */ + cursor->value.size, WT_TS_NONE, WT_TXN_NONE, WT_TS_MAX, WT_TXN_MAX, 0)); /* Boundary: split or write the page. */ if (WT_CROSSING_SPLIT_BND(r, key->len + val->len)) { @@ -268,7 +268,7 @@ __rec_row_merge(WT_SESSION_IMPL *session, WT_RECONCILE *r, WT_PAGE *page) r->cell_zero = false; addr = &multi->addr; - __wt_rec_cell_build_addr(session, r, addr, false, WT_RECNO_OOB); + __wt_rec_cell_build_addr(session, r, addr, NULL, false, WT_RECNO_OOB); /* Boundary: split or write the page. */ if (__wt_rec_need_split(r, key->len + val->len)) @@ -277,7 +277,11 @@ __rec_row_merge(WT_SESSION_IMPL *session, WT_RECONCILE *r, WT_PAGE *page) /* Copy the key and value onto the page. */ __wt_rec_image_copy(session, r, key); __wt_rec_image_copy(session, r, val); - __wt_rec_addr_ts_update(r, addr->newest_durable_ts, addr->oldest_start_ts, + /* + * FIXME-prepare-support: audit the use of durable timestamps in this file, use both durable + * timestamps. + */ + __wt_rec_addr_ts_update(r, addr->stop_durable_ts, addr->oldest_start_ts, addr->oldest_start_txn, addr->newest_stop_ts, addr->newest_stop_txn); /* Update compression state. */ @@ -357,7 +361,8 @@ __wt_rec_row_int(WT_SESSION_IMPL *session, WT_RECONCILE *r, WT_PAGE *page) if (ikey != NULL && ikey->cell_offset != 0) { cell = WT_PAGE_REF_OFFSET(page, ikey->cell_offset); __wt_cell_unpack(session, page, cell, kpack); - key_onpage_ovfl = kpack->ovfl && kpack->raw != WT_CELL_KEY_OVFL_RM; + key_onpage_ovfl = + F_ISSET(kpack, WT_CELL_UNPACK_OVERFLOW) && kpack->raw != WT_CELL_KEY_OVFL_RM; } WT_ERR(__wt_rec_child_modify(session, r, ref, &hazard, &state)); @@ -432,24 +437,33 @@ __wt_rec_row_int(WT_SESSION_IMPL *session, WT_RECONCILE *r, WT_PAGE *page) * requiring a proxy cell, otherwise use the information from the addr or original cell. */ if (__wt_off_page(page, addr)) { - __wt_rec_cell_build_addr(session, r, addr, state == WT_CHILD_PROXY, WT_RECNO_OOB); - newest_durable_ts = addr->newest_durable_ts; + __wt_rec_cell_build_addr(session, r, addr, NULL, state == WT_CHILD_PROXY, WT_RECNO_OOB); + newest_durable_ts = addr->stop_durable_ts; oldest_start_ts = addr->oldest_start_ts; oldest_start_txn = addr->oldest_start_txn; newest_stop_ts = addr->newest_stop_ts; newest_stop_txn = addr->newest_stop_txn; } else { __wt_cell_unpack(session, page, ref->addr, vpack); - if (state == WT_CHILD_PROXY) { + if (F_ISSET(vpack, WT_CELL_UNPACK_TIME_PAIRS_CLEARED)) { + /* + * The transaction ids are cleared after restart. Repack the cell with new validity + * to flush the cleared transaction ids. + */ + __wt_rec_cell_build_addr( + session, r, NULL, vpack, state == WT_CHILD_PROXY, WT_RECNO_OOB); + } else if (state == WT_CHILD_PROXY) { WT_ERR(__wt_buf_set(session, &val->buf, ref->addr, __wt_cell_total_len(vpack))); __wt_cell_type_reset(session, val->buf.mem, 0, WT_CELL_ADDR_DEL); + val->cell_len = 0; + val->len = val->buf.size; } else { val->buf.data = ref->addr; val->buf.size = __wt_cell_total_len(vpack); + val->cell_len = 0; + val->len = val->buf.size; } - val->cell_len = 0; - val->len = val->buf.size; - newest_durable_ts = vpack->newest_durable_ts; + newest_durable_ts = vpack->newest_stop_durable_ts; oldest_start_ts = vpack->oldest_start_ts; oldest_start_txn = vpack->oldest_start_txn; newest_stop_ts = vpack->newest_stop_ts; @@ -531,10 +545,6 @@ static bool __rec_row_zero_len(WT_SESSION_IMPL *session, wt_timestamp_t start_ts, uint64_t start_txn, wt_timestamp_t stop_ts, uint64_t stop_txn) { - /* Before timestamps were stored on pages, it was always possible. */ - if (!__wt_process.page_version_ts) - return (true); - /* * The item must be globally visible because we're not writing anything on the page. */ @@ -552,12 +562,13 @@ __rec_row_leaf_insert(WT_SESSION_IMPL *session, WT_RECONCILE *r, WT_INSERT *ins) { WT_BTREE *btree; WT_CURSOR_BTREE *cbt; + WT_DECL_RET; WT_REC_KV *key, *val; WT_UPDATE *upd; WT_UPDATE_SELECT upd_select; wt_timestamp_t durable_ts, start_ts, stop_ts; uint64_t start_txn, stop_txn; - bool ovfl_key, upd_saved; + bool ovfl_key; btree = S2BT(session); @@ -567,35 +578,18 @@ __rec_row_leaf_insert(WT_SESSION_IMPL *session, WT_RECONCILE *r, WT_INSERT *ins) key = &r->k; val = &r->v; + upd = NULL; + for (; ins != NULL; ins = WT_SKIP_NEXT(ins)) { WT_RET(__wt_rec_upd_select(session, r, ins, NULL, NULL, &upd_select)); - upd = upd_select.upd; + if ((upd = upd_select.upd) == NULL) + continue; + durable_ts = upd_select.durable_ts; start_ts = upd_select.start_ts; start_txn = upd_select.start_txn; stop_ts = upd_select.stop_ts; stop_txn = upd_select.stop_txn; - upd_saved = upd_select.upd_saved; - - if (upd == NULL) { - /* - * If no update is visible but some were saved, check for splits. - */ - if (!upd_saved) - continue; - if (!__wt_rec_need_split(r, WT_INSERT_KEY_SIZE(ins))) - continue; - - /* Copy the current key into place and then split. */ - WT_RET(__wt_buf_set(session, r->cur, WT_INSERT_KEY(ins), WT_INSERT_KEY_SIZE(ins))); - WT_RET(__wt_rec_split_crossing_bnd(session, r, WT_INSERT_KEY_SIZE(ins), false)); - - /* - * Turn off prefix and suffix compression until a full key is written into the new page. - */ - r->key_pfx_compress = r->key_sfx_compress = false; - continue; - } switch (upd->type) { case WT_UPDATE_MODIFY: @@ -609,17 +603,21 @@ __rec_row_leaf_insert(WT_SESSION_IMPL *session, WT_RECONCILE *r, WT_INSERT *ins) break; case WT_UPDATE_STANDARD: /* Take the value from the update. */ - WT_RET(__wt_rec_cell_build_val( + WT_ERR(__wt_rec_cell_build_val( session, r, upd->data, upd->size, start_ts, start_txn, stop_ts, stop_txn, 0)); break; case WT_UPDATE_TOMBSTONE: continue; default: - return (__wt_illegal_value(session, upd->type)); + ret = __wt_illegal_value(session, upd->type); + WT_ERR(ret); } + /* Free the update if it is external. */ + if (F_ISSET(upd, WT_UPDATE_RESTORED_FROM_DISK)) + __wt_free_update_list(session, &upd); /* Build key cell. */ - WT_RET(__rec_cell_build_leaf_key( + WT_ERR(__rec_cell_build_leaf_key( session, r, WT_INSERT_KEY(ins), WT_INSERT_KEY_SIZE(ins), &ovfl_key)); /* Boundary: split or write the page. */ @@ -631,10 +629,10 @@ __rec_row_leaf_insert(WT_SESSION_IMPL *session, WT_RECONCILE *r, WT_INSERT *ins) if (r->key_pfx_compress_conf) { r->key_pfx_compress = false; if (!ovfl_key) - WT_RET(__rec_cell_build_leaf_key(session, r, NULL, 0, &ovfl_key)); + WT_ERR(__rec_cell_build_leaf_key(session, r, NULL, 0, &ovfl_key)); } - WT_RET(__wt_rec_split_crossing_bnd(session, r, key->len + val->len, false)); + WT_ERR(__wt_rec_split_crossing_bnd(session, r, key->len + val->len, false)); } /* Copy the key/value pair onto the page. */ @@ -644,7 +642,7 @@ __rec_row_leaf_insert(WT_SESSION_IMPL *session, WT_RECONCILE *r, WT_INSERT *ins) else { r->all_empty_value = false; if (btree->dictionary) - WT_RET(__wt_rec_dict_replace( + WT_ERR(__wt_rec_dict_replace( session, r, start_ts, start_txn, stop_ts, stop_txn, 0, val)); __wt_rec_image_copy(session, r, val); } @@ -654,7 +652,44 @@ __rec_row_leaf_insert(WT_SESSION_IMPL *session, WT_RECONCILE *r, WT_INSERT *ins) __rec_key_state_update(r, ovfl_key); } - return (0); +err: + /* Free the update if it is external. */ + if (upd != NULL && F_ISSET(upd, WT_UPDATE_RESTORED_FROM_DISK)) + __wt_free_update_list(session, &upd); + + return (ret); +} + +/* + * __rec_cell_repack -- + * Repack a cell. + */ +static inline int +__rec_cell_repack(WT_SESSION_IMPL *session, WT_BTREE *btree, WT_RECONCILE *r, WT_CELL_UNPACK *vpack, + uint64_t start_txn, wt_timestamp_t start_ts, uint64_t stop_txn, wt_timestamp_t stop_ts) +{ + WT_DECL_ITEM(tmpval); + WT_DECL_RET; + size_t size; + const void *p; + + WT_ERR(__wt_scr_alloc(session, 0, &tmpval)); + + /* If the item is Huffman encoded, decode it. */ + if (btree->huffman_value == NULL) { + p = vpack->data; + size = vpack->size; + } else { + WT_ERR( + __wt_huffman_decode(session, btree->huffman_value, vpack->data, vpack->size, tmpval)); + p = tmpval->data; + size = tmpval->size; + } + WT_ERR(__wt_rec_cell_build_val(session, r, p, size, start_ts, start_txn, stop_ts, stop_txn, 0)); + +err: + __wt_scr_free(session, &tmpval); + return (ret); } /* @@ -665,13 +700,13 @@ int __wt_rec_row_leaf( WT_SESSION_IMPL *session, WT_RECONCILE *r, WT_REF *pageref, WT_SALVAGE_COOKIE *salvage) { + static WT_UPDATE upd_tombstone = {.txnid = WT_TXN_NONE, .type = WT_UPDATE_TOMBSTONE}; WT_ADDR *addr; WT_BTREE *btree; WT_CELL *cell; WT_CELL_UNPACK *kpack, _kpack, *vpack, _vpack; WT_CURSOR_BTREE *cbt; WT_DECL_ITEM(tmpkey); - WT_DECL_ITEM(tmpval); WT_DECL_RET; WT_IKEY *ikey; WT_INSERT *ins; @@ -681,12 +716,10 @@ __wt_rec_row_leaf( WT_UPDATE *upd; WT_UPDATE_SELECT upd_select; wt_timestamp_t durable_ts, newest_durable_ts, start_ts, stop_ts; - size_t size; uint64_t slvg_skip, start_txn, stop_txn; uint32_t i; bool dictionary, key_onpage_ovfl, ovfl_key; void *copy; - const void *p; btree = S2BT(session); page = pageref->page; @@ -699,6 +732,8 @@ __wt_rec_row_leaf( val = &r->v; vpack = &_vpack; + upd = NULL; + /* * Acquire the newest-durable timestamp for this page so we can roll it forward. If it exists, * it's in the WT_REF structure or the parent's disk image. @@ -706,10 +741,10 @@ __wt_rec_row_leaf( if ((addr = pageref->addr) == NULL) newest_durable_ts = WT_TS_NONE; else if (__wt_off_page(pageref->home, addr)) - newest_durable_ts = addr->newest_durable_ts; + newest_durable_ts = addr->stop_durable_ts; else { __wt_cell_unpack(session, pageref->home, pageref->addr, vpack); - newest_durable_ts = vpack->newest_durable_ts; + newest_durable_ts = vpack->newest_stop_durable_ts; } WT_RET(__wt_rec_split_init(session, r, page, 0, btree->maxleafpage_precomp)); @@ -724,7 +759,6 @@ __wt_rec_row_leaf( * Temporary buffers in which to instantiate any uninstantiated keys or value items we need. */ WT_ERR(__wt_scr_alloc(session, 0, &tmpkey)); - WT_ERR(__wt_scr_alloc(session, 0, &tmpval)); /* For each entry in the page... */ WT_ROW_FOREACH (page, rip, i) { @@ -739,6 +773,7 @@ __wt_rec_row_leaf( --slvg_skip; continue; } + dictionary = false; /* * Figure out the key: set any cell reference (and unpack it), set any instantiated key @@ -753,13 +788,8 @@ __wt_rec_row_leaf( __wt_cell_unpack(session, page, cell, kpack); } - /* Unpack the on-page value cell, set the default timestamps. */ + /* Unpack the on-page value cell. */ __wt_row_leaf_value_cell(session, page, rip, NULL, vpack); - durable_ts = newest_durable_ts; - start_ts = vpack->start_ts; - start_txn = vpack->start_txn; - stop_ts = vpack->stop_ts; - stop_txn = vpack->stop_txn; /* Look for an update. */ WT_ERR(__wt_rec_upd_select(session, r, NULL, rip, vpack, &upd_select)); @@ -769,10 +799,36 @@ __wt_rec_row_leaf( start_txn = upd_select.start_txn; stop_ts = upd_select.stop_ts; stop_txn = upd_select.stop_txn; + } else { + /* + * FIXME: Temporary fix until the value cell has the durable timestamp. Currently, value + * cell doesn't store the information of durable timestamp, so we lose the information + * of aggregated durable timestamp information when the page is reconciled without + * writing to the disk (in-memory page re-instantiate). As part of page re-instantiate + * scenarios, the calculated aggregated durable timestamp gets lost and when the same + * page gets reconciled again, we don't have any durable timestamp from the cell. Use + * commit timestamp from the cell also as the durable timestamp instead of setting it to + * zero until we store the durable timestamp in the cell. + */ + if (newest_durable_ts != WT_TS_NONE) + durable_ts = newest_durable_ts; + else + durable_ts = vpack->start_ts; + start_ts = vpack->start_ts; + start_txn = vpack->start_txn; + stop_ts = vpack->stop_ts; + stop_txn = vpack->stop_txn; } + /* + * If we reconcile an on disk key with a globally visible stop time pair and there are no + * new updates for that key, skip writing that key. + */ + if (upd == NULL && (vpack->stop_txn != WT_TXN_MAX || vpack->stop_ts != WT_TS_MAX) && + __wt_txn_visible_all(session, vpack->stop_txn, vpack->stop_ts)) + upd = &upd_tombstone; + /* Build value cell. */ - dictionary = false; if (upd == NULL) { /* * When the page was read into memory, there may not have been a value item. @@ -780,54 +836,34 @@ __wt_rec_row_leaf( * If there was a value item, check if it's a dictionary cell (a copy of another item on * the page). If it's a copy, we have to create a new value item as the old item might * have been discarded from the page. + * + * Repack the cell if we clear the transaction ids in the cell. */ if (vpack->raw == WT_CELL_VALUE_COPY) { - /* If the item is Huffman encoded, decode it. */ - if (btree->huffman_value == NULL) { - p = vpack->data; - size = vpack->size; - } else { - WT_ERR(__wt_huffman_decode( - session, btree->huffman_value, vpack->data, vpack->size, tmpval)); - p = tmpval->data; - size = tmpval->size; - } - WT_ERR(__wt_rec_cell_build_val( - session, r, p, size, start_ts, start_txn, stop_ts, stop_txn, 0)); + WT_ERR(__rec_cell_repack( + session, btree, r, vpack, start_txn, start_ts, stop_txn, stop_ts)); + dictionary = true; - } else if (vpack->raw == WT_CELL_VALUE_OVFL_RM) { + } else if (F_ISSET(vpack, WT_CELL_UNPACK_TIME_PAIRS_CLEARED)) { /* - * If doing an update save and restore, and the underlying value is a removed - * overflow value, we end up here. - * - * If necessary, when the overflow value was originally removed, reconciliation - * appended a globally visible copy of the value to the key's update list, meaning - * the on-page item isn't accessed after page re-instantiation. - * - * Assert the case. + * The transaction ids are cleared after restart. Repack the cell to flush the + * cleared transaction ids. */ - WT_ASSERT(session, F_ISSET(r, WT_REC_UPDATE_RESTORE)); + if (F_ISSET(vpack, WT_CELL_UNPACK_OVERFLOW)) { + r->ovfl_items = true; - /* - * If the key is also a removed overflow item, don't write anything at all. - * - * We don't have to write anything because the code re-instantiating the page gets - * the key to match the saved list of updates from the original page. By not putting - * the key on the page, we'll move the key/value set from a row-store leaf page slot - * to an insert list, but that shouldn't matter. - * - * The reason we bother with the test is because overflows are expensive to write. - * It's hard to imagine a real workload where this test is worth the effort, but - * it's a simple test. - */ - if (kpack != NULL && kpack->raw == WT_CELL_KEY_OVFL_RM) - goto leaf_insert; + val->buf.data = vpack->data; + val->buf.size = vpack->size; - /* - * The on-page value will never be accessed, write a placeholder record. - */ - WT_ERR(__wt_rec_cell_build_val(session, r, "ovfl-unused", strlen("ovfl-unused"), - start_ts, start_txn, stop_ts, stop_txn, 0)); + /* Rebuild the cell. */ + val->cell_len = __wt_cell_pack_ovfl(session, &val->cell, vpack->raw, start_ts, + start_txn, stop_ts, stop_txn, 0, val->buf.size); + val->len = val->cell_len + val->buf.size; + } else + WT_ERR(__rec_cell_repack( + session, btree, r, vpack, start_txn, start_ts, stop_txn, stop_ts)); + + dictionary = true; } else { val->buf.data = vpack->cell; val->buf.size = __wt_cell_total_len(vpack); @@ -835,15 +871,12 @@ __wt_rec_row_leaf( val->len = val->buf.size; /* Track if page has overflow items. */ - if (vpack->ovfl) + if (F_ISSET(vpack, WT_CELL_UNPACK_OVERFLOW)) r->ovfl_items = true; } } else { - /* - * The first time we find an overflow record we're not going to use, discard the - * underlying blocks. - */ - if (vpack->ovfl && vpack->raw != WT_CELL_VALUE_OVFL_RM) + /* The first time we find an overflow record, discard the underlying blocks. */ + if (F_ISSET(vpack, WT_CELL_UNPACK_OVERFLOW) && vpack->raw != WT_CELL_VALUE_OVFL_RM) WT_ERR(__wt_ovfl_remove(session, page, vpack, F_ISSET(r, WT_REC_EVICT))); switch (upd->type) { @@ -868,7 +901,8 @@ __wt_rec_row_leaf( * backing blocks. Don't worry about reuse, reusing keys from a row-store page * reconciliation seems unlikely enough to ignore. */ - if (kpack != NULL && kpack->ovfl && kpack->raw != WT_CELL_KEY_OVFL_RM) { + if (kpack != NULL && F_ISSET(kpack, WT_CELL_UNPACK_OVERFLOW) && + kpack->raw != WT_CELL_KEY_OVFL_RM) { /* * Keys are part of the name-space, we can't remove them from the in-memory * tree; if an overflow key was deleted without being instantiated (for example, @@ -881,8 +915,20 @@ __wt_rec_row_leaf( } /* - * We aren't actually creating the key so we can't use bytes from this key to - * provide prefix information for a subsequent key. + * If we're removing a key, also remove the history store contents associated with + * that key. Even if we fail reconciliation after this point, we're safe to do this. + * The history store content must be obsolete in order for us to consider removing + * the key. + */ + if (F_ISSET(S2C(session), WT_CONN_HS_OPEN) && !WT_IS_HS(btree)) { + WT_ERR(__wt_row_leaf_key(session, page, rip, tmpkey, true)); + WT_ERR(__wt_hs_delete_key(session, btree->id, tmpkey)); + WT_STAT_CONN_INCR(session, cache_hs_key_truncate_onpage_removal); + } + + /* + * We aren't creating a key so we can't use bytes from this key to provide prefix + * information for a subsequent key. */ tmpkey->size = 0; @@ -891,6 +937,9 @@ __wt_rec_row_leaf( default: WT_ERR(__wt_illegal_value(session, upd->type)); } + /* Free the update if it is external. */ + if (F_ISSET(upd, WT_UPDATE_RESTORED_FROM_DISK)) + __wt_free_update_list(session, &upd); } /* @@ -898,7 +947,8 @@ __wt_rec_row_leaf( * * If the key is an overflow key that hasn't been removed, use the original backing blocks. */ - key_onpage_ovfl = kpack != NULL && kpack->ovfl && kpack->raw != WT_CELL_KEY_OVFL_RM; + key_onpage_ovfl = kpack != NULL && F_ISSET(kpack, WT_CELL_UNPACK_OVERFLOW) && + kpack->raw != WT_CELL_KEY_OVFL_RM; if (key_onpage_ovfl) { key->buf.data = cell; key->buf.size = __wt_cell_total_len(kpack); @@ -925,13 +975,7 @@ __wt_rec_row_leaf( kpack = &_kpack; __wt_cell_unpack(session, page, cell, kpack); if (btree->huffman_key == NULL && kpack->type == WT_CELL_KEY && - tmpkey->size >= kpack->prefix) { - /* - * The previous clause checked for a prefix of zero, which means the temporary - * buffer must have a non-zero size, and it references a valid key. - */ - WT_ASSERT(session, tmpkey->size != 0); - + tmpkey->size >= kpack->prefix && tmpkey->size != 0) { /* * Grow the buffer as necessary, ensuring data data has been copied into local * buffer space, then append the suffix to the prefix already in the buffer. @@ -999,7 +1043,10 @@ leaf_insert: ret = __wt_rec_split_finish(session, r); err: + /* Free the update if it is external. */ + if (upd != NULL && F_ISSET(upd, WT_UPDATE_RESTORED_FROM_DISK)) + __wt_free_update_list(session, &upd); + __wt_scr_free(session, &tmpkey); - __wt_scr_free(session, &tmpval); return (ret); } diff --git a/src/third_party/wiredtiger/src/reconcile/rec_visibility.c b/src/third_party/wiredtiger/src/reconcile/rec_visibility.c index 8fb786822c7..8c065a5fc87 100644 --- a/src/third_party/wiredtiger/src/reconcile/rec_visibility.c +++ b/src/third_party/wiredtiger/src/reconcile/rec_visibility.c @@ -9,16 +9,16 @@ #include "wt_internal.h" /* - * __rec_update_durable -- - * Return whether an update is suitable for writing to a disk image. + * __rec_update_stable -- + * Return whether an update is stable or not. */ static bool -__rec_update_durable(WT_SESSION_IMPL *session, WT_RECONCILE *r, WT_UPDATE *upd) +__rec_update_stable(WT_SESSION_IMPL *session, WT_RECONCILE *r, WT_UPDATE *upd) { return (F_ISSET(r, WT_REC_VISIBLE_ALL) ? __wt_txn_upd_visible_all(session, upd) : __wt_txn_upd_visible_type(session, upd) == WT_VISIBLE_TRUE && - __wt_txn_visible(session, upd->txnid, upd->durable_ts)); + __wt_txn_visible(session, upd->txnid, upd->start_ts)); } /* @@ -29,10 +29,16 @@ static int __rec_update_save(WT_SESSION_IMPL *session, WT_RECONCILE *r, WT_INSERT *ins, void *ripcip, WT_UPDATE *onpage_upd, size_t upd_memsize) { + WT_SAVE_UPD *supd; + WT_RET(__wt_realloc_def(session, &r->supd_allocated, r->supd_next + 1, &r->supd)); - r->supd[r->supd_next].ins = ins; - r->supd[r->supd_next].ripcip = ripcip; - r->supd[r->supd_next].onpage_upd = onpage_upd; + supd = &r->supd[r->supd_next]; + supd->ins = ins; + supd->ripcip = ripcip; + WT_CLEAR(supd->onpage_upd); + if (onpage_upd != NULL && + (onpage_upd->type == WT_UPDATE_STANDARD || onpage_upd->type == WT_UPDATE_MODIFY)) + supd->onpage_upd = onpage_upd; ++r->supd_next; r->supd_memsize += upd_memsize; return (0); @@ -48,19 +54,24 @@ __rec_append_orig_value( { WT_DECL_ITEM(tmp); WT_DECL_RET; - WT_UPDATE *append; - size_t size; + WT_UPDATE *append, *tombstone; + size_t size, total_size; - /* Done if at least one self-contained update is globally visible. */ for (;; upd = upd->next) { + /* Done if at least one self-contained update is globally visible. */ if (WT_UPDATE_DATA_VALUE(upd) && __wt_txn_upd_visible_all(session, upd)) return (0); - /* Add the original value after birthmarks. */ - if (upd->type == WT_UPDATE_BIRTHMARK) { - WT_ASSERT(session, unpack != NULL && unpack->type != WT_CELL_DEL); - break; - } + /* + * If the update is restored from the history store for the rollback to stable operation we + * don't need the on-disk value anymore and we're done. + */ + if (F_ISSET(upd, WT_UPDATE_RESTORED_FOR_ROLLBACK)) + return (0); + + /* On page value already on chain */ + if (unpack != NULL && unpack->start_ts == upd->start_ts && unpack->start_txn == upd->txnid) + return (0); /* Leave reference at the last item in the chain. */ if (upd->next == NULL) @@ -73,40 +84,46 @@ __rec_append_orig_value( * * If we don't have a value cell, it's an insert/append list key/value pair which simply doesn't * exist for some reader; place a deleted record at the end of the update list. + * + * If the an update is out of order so it masks the value in the cell, don't append. */ - append = NULL; /* -Wconditional-uninitialized */ - size = 0; /* -Wconditional-uninitialized */ + append = tombstone = NULL; /* -Wconditional-uninitialized */ + total_size = size = 0; /* -Wconditional-uninitialized */ if (unpack == NULL || unpack->type == WT_CELL_DEL) WT_RET(__wt_update_alloc(session, NULL, &append, &size, WT_UPDATE_TOMBSTONE)); else { WT_RET(__wt_scr_alloc(session, 0, &tmp)); WT_ERR(__wt_page_cell_data_ref(session, page, unpack, tmp)); WT_ERR(__wt_update_alloc(session, tmp, &append, &size, WT_UPDATE_STANDARD)); - } + append->start_ts = append->durable_ts = unpack->start_ts; + append->txnid = unpack->start_txn; + total_size = size; - /* - * If we're saving the original value for a birthmark, transfer over the transaction ID and - * clear out the birthmark update. Else, set the entry's transaction information to the lowest - * possible value (as cleared memory matches the lowest possible transaction ID and timestamp, - * do nothing). - */ - if (upd->type == WT_UPDATE_BIRTHMARK) { - append->txnid = upd->txnid; - append->start_ts = upd->start_ts; - append->durable_ts = upd->durable_ts; - append->next = upd->next; + /* + * We need to append a TOMBSTONE before the onpage value if the onpage value has a valid + * stop pair. + * + * Imagine a case we insert and delete a value respectively at timestamp 0 and 10, and later + * insert it again at 20. We need the TOMBSTONE to tell us there is no value between 10 and + * 20. + */ + if (unpack->stop_ts != WT_TS_MAX || unpack->stop_txn != WT_TXN_MAX) { + WT_ERR(__wt_update_alloc(session, NULL, &tombstone, &size, WT_UPDATE_TOMBSTONE)); + tombstone->txnid = unpack->stop_txn; + tombstone->start_ts = unpack->stop_ts; + tombstone->durable_ts = unpack->stop_ts; + tombstone->next = append; + total_size += size; + } } + if (tombstone != NULL) + append = tombstone; + /* Append the new entry into the update list. */ WT_PUBLISH(upd->next, append); - /* Replace the birthmark with an aborted transaction. */ - if (upd->type == WT_UPDATE_BIRTHMARK) { - WT_ORDERED_WRITE(upd->txnid, WT_TXN_ABORTED); - WT_ORDERED_WRITE(upd->type, WT_UPDATE_STANDARD); - } - - __wt_cache_page_inmem_incr(session, page, size); + __wt_cache_page_inmem_incr(session, page, total_size); err: __wt_scr_free(session, &tmp); @@ -114,6 +131,33 @@ err: } /* + * __rec_need_save_upd -- + * Return if we need to save the update chain + */ +static bool +__rec_need_save_upd( + WT_SESSION_IMPL *session, WT_RECONCILE *r, WT_UPDATE_SELECT *upd_select, bool has_newer_updates) +{ + if (F_ISSET(r, WT_REC_EVICT) && has_newer_updates) + return (true); + + /* + * Save updates for any reconciliation that doesn't involve history store (in-memory database + * and fixed length column store), except when the maximum timestamp and txnid are globally + * visible. + */ + if (!F_ISSET(r, WT_REC_HS) && !F_ISSET(r, WT_REC_IN_MEMORY) && r->page->type != WT_PAGE_COL_FIX) + return (false); + + /* When in checkpoint, no need to save update if no onpage value is selected. */ + if (F_ISSET(r, WT_REC_CHECKPOINT) && upd_select->upd == NULL) + return (false); + + return (!__wt_txn_visible_all(session, upd_select->stop_txn, upd_select->stop_ts) && + !__wt_txn_visible_all(session, upd_select->start_txn, upd_select->start_ts)); +} + +/* * __wt_rec_upd_select -- * Return the update in a list that should be written (or NULL if none can be written). */ @@ -121,31 +165,40 @@ int __wt_rec_upd_select(WT_SESSION_IMPL *session, WT_RECONCILE *r, WT_INSERT *ins, void *ripcip, WT_CELL_UNPACK *vpack, WT_UPDATE_SELECT *upd_select) { + WT_DECL_ITEM(tmp); + WT_DECL_RET; WT_PAGE *page; - WT_UPDATE *first_txn_upd, *first_upd, *upd; - wt_timestamp_t max_ts; - size_t upd_memsize; + WT_UPDATE *first_txn_upd, *first_upd, *upd, *last_upd; + wt_timestamp_t checkpoint_timestamp, max_ts, tombstone_durable_ts; + size_t size, upd_memsize; uint64_t max_txn, txnid; - bool all_stable, list_prepared, list_uncommitted, skipped_birthmark; + bool has_newer_updates, is_hs_page, upd_saved; /* * The "saved updates" return value is used independently of returning an update we can write, * both must be initialized. */ upd_select->upd = NULL; - upd_select->upd_saved = false; + upd_select->durable_ts = WT_TS_NONE; + upd_select->start_ts = WT_TS_NONE; + upd_select->start_txn = WT_TXN_NONE; + upd_select->stop_ts = WT_TS_MAX; + upd_select->stop_txn = WT_TXN_MAX; page = r->page; - first_txn_upd = NULL; + first_txn_upd = upd = last_upd = NULL; upd_memsize = 0; + checkpoint_timestamp = S2C(session)->txn_global.checkpoint_timestamp; max_ts = WT_TS_NONE; + tombstone_durable_ts = WT_TS_MAX; max_txn = WT_TXN_NONE; - list_prepared = list_uncommitted = skipped_birthmark = false; + has_newer_updates = upd_saved = false; + is_hs_page = F_ISSET(S2BT(session), WT_BTREE_HS); /* - * If called with a WT_INSERT item, use its WT_UPDATE list (which must - * exist), otherwise check for an on-page row-store WT_UPDATE list - * (which may not exist). Return immediately if the item has no updates. + * If called with a WT_INSERT item, use its WT_UPDATE list (which must exist), otherwise check + * for an on-page row-store WT_UPDATE list (which may not exist). Return immediately if the item + * has no updates. */ if (ins != NULL) first_upd = ins->upd; @@ -168,60 +221,53 @@ __wt_rec_upd_select(WT_SESSION_IMPL *session, WT_RECONCILE *r, WT_INSERT *ins, v max_txn = txnid; /* - * Track if all the updates are not with in-progress prepare state. - */ - if (upd->prepare_state == WT_PREPARE_RESOLVED) - r->all_upd_prepare_in_prog = false; - - /* * Check whether the update was committed before reconciliation started. The global commit * point can move forward during reconciliation so we use a cached copy to avoid races when - * a concurrent transaction commits or rolls back while we are examining its updates. As + * a concurrent transaction commits or rolls back while we are examining its updates. This + * check is not required for history store updates as they are implicitly committed. As * prepared transaction IDs are globally visible, need to check the update state as well. */ - if (F_ISSET(r, WT_REC_EVICT)) { - if (F_ISSET(r, WT_REC_VISIBLE_ALL) ? WT_TXNID_LE(r->last_running, txnid) : - !__txn_visible_id(session, txnid)) { - r->update_uncommitted = list_uncommitted = true; - continue; - } - if (upd->prepare_state == WT_PREPARE_LOCKED || - upd->prepare_state == WT_PREPARE_INPROGRESS) { - list_prepared = true; - if (upd->start_ts > max_ts) - max_ts = upd->start_ts; - - /* - * Track the oldest update not on the page, used to decide whether reads can use the - * page image, hence using the start rather than the durable timestamp. - */ - if (upd->start_ts < r->min_skipped_ts) - r->min_skipped_ts = upd->start_ts; - continue; - } + if (!is_hs_page && (F_ISSET(r, WT_REC_VISIBLE_ALL) ? WT_TXNID_LE(r->last_running, txnid) : + !__txn_visible_id(session, txnid))) { + has_newer_updates = true; + continue; + } + if (upd->prepare_state == WT_PREPARE_LOCKED || + upd->prepare_state == WT_PREPARE_INPROGRESS) { + has_newer_updates = true; + if (upd->start_ts > max_ts) + max_ts = upd->start_ts; + + /* + * Track the oldest update not on the page, used to decide whether reads can use the + * page image, hence using the start rather than the durable timestamp. + */ + if (upd->start_ts < r->min_skipped_ts) + r->min_skipped_ts = upd->start_ts; + continue; } /* Track the first update with non-zero timestamp. */ - if (upd->durable_ts > max_ts) - max_ts = upd->durable_ts; + if (upd->start_ts > max_ts) + max_ts = upd->start_ts; /* - * Select the update to write to the disk image. - * - * Lookaside and update/restore eviction try to choose the same - * version as a subsequent checkpoint, so that checkpoint can - * skip over pages with lookaside entries. If the application - * has supplied a stable timestamp, we assume (a) that it is - * old, and (b) that the next checkpoint will use it, so we wait - * to see a stable update. If there is no stable timestamp, we - * assume the next checkpoint will write the most recent version - * (but we save enough information that checkpoint can fix - * things up if we choose an update that is too new). + * FIXME-prepare-support: A temporary solution for not storing durable timestamp in the + * cell. Properly fix this problem in PM-1524. It is currently not OK to write prepared + * updates with durable timestamp larger than checkpoint timestamp to data store as we don't + * store durable timestamp in the cell. However, it is OK to write them to the history store + * as we store the durable timestamp in the history store value. */ - if (upd_select->upd == NULL && r->las_skew_newest) + if (upd->durable_ts != upd->start_ts && upd->durable_ts > checkpoint_timestamp) { + has_newer_updates = true; + continue; + } + + /* Always select the newest committed update to write to disk */ + if (upd_select->upd == NULL) upd_select->upd = upd; - if (!__rec_update_durable(session, r, upd)) { + if (!__rec_update_stable(session, r, upd)) { if (F_ISSET(r, WT_REC_EVICT)) ++r->updates_unstable; @@ -231,32 +277,9 @@ __wt_rec_upd_select(WT_SESSION_IMPL *session, WT_RECONCILE *r, WT_INSERT *ins, v * to discard updates from the stable update and older for correctness and we can't * discard an uncommitted update. */ - if (F_ISSET(r, WT_REC_UPDATE_RESTORE) && upd_select->upd != NULL && - (list_prepared || list_uncommitted)) + if (upd_select->upd != NULL && has_newer_updates) return (__wt_set_return(session, EBUSY)); - - if (upd->type == WT_UPDATE_BIRTHMARK) - skipped_birthmark = true; - - /* - * Track the oldest update not on the page, used to decide whether reads can use the - * page image, hence using the start rather than the durable timestamp. - */ - if (upd_select->upd == NULL && upd->start_ts < r->min_skipped_ts) - r->min_skipped_ts = upd->start_ts; - - continue; - } - - /* - * Lookaside without stable timestamp was taken care of above - * (set to the first uncommitted transaction). All other - * reconciliation takes the first stable update. - */ - if (upd_select->upd == NULL) - upd_select->upd = upd; - - if (!F_ISSET(r, WT_REC_EVICT)) + } else if (!F_ISSET(r, WT_REC_EVICT)) break; } @@ -281,49 +304,133 @@ __wt_rec_upd_select(WT_SESSION_IMPL *session, WT_RECONCILE *r, WT_INSERT *ins, v return (0); } - /* If no updates were skipped, record that we're making progress. */ - if (upd == first_txn_upd) - r->update_used = true; + /* + * We expect the page to be clean after reconciliation. If there are invisible updates, abort + * eviction. + */ + if (has_newer_updates && F_ISSET(r, WT_REC_CLEAN_AFTER_REC | WT_REC_VISIBILITY_ERR)) { + if (F_ISSET(r, WT_REC_VISIBILITY_ERR)) + WT_PANIC_RET(session, EINVAL, "reconciliation error, update not visible"); + return (__wt_set_return(session, EBUSY)); + } - if (upd != NULL && upd->durable_ts > r->max_ondisk_ts) - r->max_ondisk_ts = upd->durable_ts; + if (upd != NULL && upd->start_ts > r->max_ondisk_ts) + r->max_ondisk_ts = upd->start_ts; /* - * TIMESTAMP-FIXME The start timestamp is determined by the commit timestamp when the key is - * first inserted (or last updated). The end timestamp is set when a key/value pair becomes - * invalid, either because of a remove or a modify/update operation on the same key. + * The start timestamp is determined by the commit timestamp when the key is first inserted (or + * last updated). The end timestamp is set when a key/value pair becomes invalid, either because + * of a remove or a modify/update operation on the same key. + * + * In the case of a tombstone where the previous update is the ondisk value, we'll allocate an + * update here to represent the ondisk value. Keep a pointer to the original update (the + * tombstone) since we do some pointer comparisons below to check whether or not all updates are + * stable. */ if (upd != NULL) { /* - * TIMESTAMP-FIXME This is waiting on the WT_UPDATE structure's start/stop - * timestamp/transaction work. For now, if we don't have a timestamp/transaction, just - * pretend it's durable. If we do have a timestamp/transaction, make the durable and start - * timestamps equal to the start timestamp and the start transaction equal to the - * transaction, and again, pretend it's durable. + * If the newest is a tombstone then select the update before it and set the end of the + * visibility window to its time pair as appropriate to indicate that we should return "not + * found" for reads after this point. + * + * Otherwise, leave the end of the visibility window at the maximum possible value to + * indicate that the value is visible to any timestamp/transaction id ahead of it. */ - upd_select->durable_ts = WT_TS_NONE; - upd_select->start_ts = WT_TS_NONE; - upd_select->start_txn = WT_TXN_NONE; - upd_select->stop_ts = WT_TS_MAX; - upd_select->stop_txn = WT_TXN_MAX; - if (upd_select->upd->start_ts != WT_TS_NONE) - upd_select->durable_ts = upd_select->start_ts = upd_select->upd->start_ts; - if (upd_select->upd->txnid != WT_TXN_NONE) - upd_select->start_txn = upd_select->upd->txnid; + if (upd->type == WT_UPDATE_TOMBSTONE) { + upd_select->stop_ts = upd->start_ts; + if (upd->txnid != WT_TXN_NONE) + upd_select->stop_txn = upd->txnid; + if (upd->durable_ts != WT_TS_NONE) + tombstone_durable_ts = upd->durable_ts; + + /* Find the update this tombstone applies to. */ + if (!__wt_txn_visible_all(session, upd->txnid, upd->start_ts)) { + while (upd->next != NULL && upd->next->txnid == WT_TXN_ABORTED) + upd = upd->next; + WT_ASSERT(session, upd->next == NULL || upd->next->txnid != WT_TXN_ABORTED); + if (upd->next == NULL) + last_upd = upd; + upd_select->upd = upd = upd->next; + } + } + if (upd != NULL) { + /* The beginning of the validity window is the selected update's time pair. */ + upd_select->durable_ts = upd_select->start_ts = upd->start_ts; + /* If durable timestamp is provided, use it. */ + if (upd->durable_ts != WT_TS_NONE) + upd_select->durable_ts = upd->durable_ts; + upd_select->start_txn = upd->txnid; + + /* Use the tombstone durable timestamp as the overall durable timestamp if it exists. */ + if (tombstone_durable_ts != WT_TS_MAX) + upd_select->durable_ts = tombstone_durable_ts; + } else if (upd_select->stop_ts != WT_TS_NONE || upd_select->stop_txn != WT_TXN_NONE) { + /* If we only have a tombstone in the update list, we must have an ondisk value. */ + WT_ASSERT(session, vpack != NULL); + /* + * It's possible to have a tombstone as the only update in the update list. If we + * reconciled before with only a single update and then read the page back into cache, + * we'll have an empty update list. And applying a delete on top of that will result in + * ONLY a tombstone in the update list. + * + * In this case, we should leave the selected update unset to indicate that we want to + * keep the same on-disk value but set the stop time pair to indicate that the validity + * window ends when this tombstone started. + */ + upd_select->durable_ts = upd_select->start_ts = vpack->start_ts; + upd_select->start_txn = vpack->start_txn; - /* - * Finalize the timestamps and transactions, checking if the update is globally visible and - * nothing needs to be written. - */ - if ((upd_select->stop_ts == WT_TS_MAX && upd_select->stop_txn == WT_TXN_MAX) && - ((upd_select->start_ts == WT_TS_NONE && upd_select->start_txn == WT_TXN_NONE) || - __wt_txn_visible_all(session, upd_select->start_txn, upd_select->start_ts))) { - upd_select->start_ts = WT_TS_NONE; - upd_select->start_txn = WT_TXN_NONE; - upd_select->stop_ts = WT_TS_MAX; - upd_select->stop_txn = WT_TXN_MAX; + /* Use the tombstone durable timestamp as the overall durable timestamp if it exists. */ + if (tombstone_durable_ts != WT_TS_MAX) + upd_select->durable_ts = tombstone_durable_ts; + + /* + * Leaving the update unset means that we can skip reconciling. If we've set the stop + * time pair because of a tombstone after the on-disk value, we still have work to do so + * that is NOT ok. Let's append the on-disk value to the chain. + */ + WT_ERR(__wt_scr_alloc(session, 0, &tmp)); + WT_ERR(__wt_page_cell_data_ref(session, page, vpack, tmp)); + WT_ERR(__wt_update_alloc(session, tmp, &upd, &size, WT_UPDATE_STANDARD)); + upd->start_ts = upd->durable_ts = vpack->start_ts; + upd->txnid = vpack->start_txn; + WT_PUBLISH(last_upd->next, upd); + /* This is going in our update list so it should be accounted for in cache usage. */ + __wt_cache_page_inmem_incr(session, page, size); + upd_select->upd = upd; } } + /* + * If we've set the stop to a zeroed pair, we intend to remove the key. Instead of selecting the + * onpage value and setting the stop a zeroed time pair which would trigger a rewrite of the + * cell with the new stop time pair, we should unset the selected update so the key itself gets + * omitted from the new page image. + */ + if (upd_select->stop_ts == WT_TS_NONE && upd_select->stop_txn == WT_TXN_NONE) + upd_select->upd = NULL; + + /* + * If we found a tombstone with a time pair earlier than the update it applies to, which can + * happen if the application performs operations with timestamps out-of-order, make it invisible + * by making the start time pair match the stop time pair of the tombstone. We don't guarantee + * that older readers will be able to continue reading content that has been made invisible by + * out-of-order updates. + * + * Note that we carefully don't take this path when the stop time pair is equal to the start + * time pair. While unusual, it is permitted for a single transaction to insert and then remove + * a record. We don't want to generate a warning in that case. + */ + if (upd_select->stop_ts < upd_select->start_ts || + (upd_select->stop_ts == upd_select->start_ts && + upd_select->stop_txn < upd_select->start_txn)) { + char ts_string[2][WT_TS_INT_STRING_SIZE]; + __wt_verbose(session, WT_VERB_TIMESTAMP, + "Warning: fixing out-of-order timestamps remove at %s earlier than value at %s", + __wt_timestamp_to_string(upd_select->stop_ts, ts_string[0]), + __wt_timestamp_to_string(upd_select->start_ts, ts_string[1])); + upd_select->start_ts = upd_select->stop_ts; + upd_select->start_txn = upd_select->stop_txn; + } /* * Track the most recent transaction in the page. We store this in the tree at the end of @@ -337,72 +444,29 @@ __wt_rec_upd_select(WT_SESSION_IMPL *session, WT_RECONCILE *r, WT_INSERT *ins, v if (max_ts > r->max_ts) r->max_ts = max_ts; - /* - * If the update we chose was a birthmark, or we are doing update-restore and we skipped a - * birthmark, the original on-page value must be retained. - */ - if (upd != NULL && (upd->type == WT_UPDATE_BIRTHMARK || - (F_ISSET(r, WT_REC_UPDATE_RESTORE) && skipped_birthmark))) { - /* - * Resolve the birthmark now regardless of whether the update being written to the data file - * is the same as it was the previous reconciliation. Otherwise lookaside can end up with - * two birthmark records in the same update chain. - */ - WT_RET(__rec_append_orig_value(session, page, first_upd, vpack)); - upd_select->upd = NULL; - } + /* Mark the page dirty after reconciliation. */ + if (has_newer_updates) + r->leave_dirty = true; /* - * Check if all updates on the page are visible, if not, it must stay dirty. - * - * Updates can be out of transaction ID order (but not out of timestamp order), so we track the - * maximum transaction ID and the newest update with a timestamp (if any). + * We should restore the update chains to the new disk image if there are newer updates in + * eviction. */ - all_stable = upd == first_txn_upd && !list_prepared && !list_uncommitted && - __wt_txn_visible_all(session, max_txn, max_ts); - - if (all_stable) - goto check_original_value; - - r->leave_dirty = true; - - if (F_ISSET(r, WT_REC_VISIBILITY_ERR)) - WT_PANIC_RET(session, EINVAL, "reconciliation error, update not visible"); - - /* If not trying to evict the page, we know what we'll write and we're done. */ - if (!F_ISSET(r, WT_REC_EVICT)) - goto check_original_value; + if (has_newer_updates && F_ISSET(r, WT_REC_EVICT)) + r->cache_write_restore = true; /* - * We are attempting eviction with changes that are not yet stable - * (i.e. globally visible). There are two ways to continue, the - * save/restore eviction path or the lookaside table eviction path. - * Both cannot be configured because the paths track different - * information. The update/restore path can handle uncommitted changes, - * by evicting most of the page and then creating a new, smaller page - * to which we re-attach those changes. Lookaside eviction writes - * changes into the lookaside table and restores them on demand if and - * when the page is read back into memory. + * The update doesn't have any further updates that need to be written to the history store, + * skip saving the update as saving the update will cause reconciliation to think there is work + * that needs to be done when there might not be. * - * Both paths are configured outside of reconciliation: the save/restore - * path is the WT_REC_UPDATE_RESTORE flag, the lookaside table path is - * the WT_REC_LOOKASIDE flag. - */ - if (!F_ISSET(r, WT_REC_LOOKASIDE | WT_REC_UPDATE_RESTORE)) - return (__wt_set_return(session, EBUSY)); - if (list_uncommitted && !F_ISSET(r, WT_REC_UPDATE_RESTORE)) - return (__wt_set_return(session, EBUSY)); - - WT_ASSERT(session, r->max_txn != WT_TXN_NONE); - - /* - * The order of the updates on the list matters, we can't move only the unresolved updates, move - * the entire update list. + * Additionally history store reconciliation is not set skip saving an update. */ - WT_RET(__rec_update_save(session, r, ins, ripcip, upd_select->upd, upd_memsize)); - upd_select->upd_saved = true; + if (__rec_need_save_upd(session, r, upd_select, has_newer_updates)) { + WT_ERR(__rec_update_save(session, r, ins, ripcip, upd_select->upd, upd_memsize)); + upd_saved = true; + } -check_original_value: /* * Paranoia: check that we didn't choose an update that has since been rolled back. */ @@ -410,15 +474,16 @@ check_original_value: /* * Returning an update means the original on-page value might be lost, and that's a problem if - * there's a reader that needs it. This call makes a copy of the on-page value and if there is a - * birthmark in the update list, replaces it. We do that any time there are saved updates and - * during reconciliation of a backing overflow record that will be physically removed once it's - * no longer needed + * there's a reader that needs it. This call makes a copy of the on-page value. We do that any + * time there are saved updates and during reconciliation of a backing overflow record that will + * be physically removed once it's no longer needed. */ if (upd_select->upd != NULL && - (upd_select->upd_saved || - (vpack != NULL && vpack->ovfl && vpack->raw != WT_CELL_VALUE_OVFL_RM))) - WT_RET(__rec_append_orig_value(session, page, first_upd, vpack)); + (upd_saved || (vpack != NULL && F_ISSET(vpack, WT_CELL_UNPACK_OVERFLOW) && + vpack->raw != WT_CELL_VALUE_OVFL_RM))) + WT_ERR(__rec_append_orig_value(session, page, upd_select->upd, vpack)); - return (0); +err: + __wt_scr_free(session, &tmp); + return (ret); } diff --git a/src/third_party/wiredtiger/src/reconcile/rec_write.c b/src/third_party/wiredtiger/src/reconcile/rec_write.c index a24cd11116b..eb78d49c924 100644 --- a/src/third_party/wiredtiger/src/reconcile/rec_write.c +++ b/src/third_party/wiredtiger/src/reconcile/rec_write.c @@ -12,55 +12,40 @@ static void __rec_cleanup(WT_SESSION_IMPL *, WT_RECONCILE *); static void __rec_destroy(WT_SESSION_IMPL *, void *); static int __rec_destroy_session(WT_SESSION_IMPL *); static int __rec_init(WT_SESSION_IMPL *, WT_REF *, uint32_t, WT_SALVAGE_COOKIE *, void *); -static int __rec_las_wrapup(WT_SESSION_IMPL *, WT_RECONCILE *); -static int __rec_las_wrapup_err(WT_SESSION_IMPL *, WT_RECONCILE *); +static int __rec_hs_wrapup(WT_SESSION_IMPL *, WT_RECONCILE *); static int __rec_root_write(WT_SESSION_IMPL *, WT_PAGE *, uint32_t); static int __rec_split_discard(WT_SESSION_IMPL *, WT_PAGE *); static int __rec_split_row_promote(WT_SESSION_IMPL *, WT_RECONCILE *, WT_ITEM *, uint8_t); static int __rec_split_write(WT_SESSION_IMPL *, WT_RECONCILE *, WT_REC_CHUNK *, WT_ITEM *, bool); -static int __rec_write_check_complete(WT_SESSION_IMPL *, WT_RECONCILE *, int, bool *); static void __rec_write_page_status(WT_SESSION_IMPL *, WT_RECONCILE *); static int __rec_write_wrapup(WT_SESSION_IMPL *, WT_RECONCILE *, WT_PAGE *); static int __rec_write_wrapup_err(WT_SESSION_IMPL *, WT_RECONCILE *, WT_PAGE *); -static int __reconcile(WT_SESSION_IMPL *, WT_REF *, WT_SALVAGE_COOKIE *, uint32_t, bool *, bool *); +static int __reconcile(WT_SESSION_IMPL *, WT_REF *, WT_SALVAGE_COOKIE *, uint32_t, bool *); /* * __wt_reconcile -- * Reconcile an in-memory page into its on-disk format, and write it. */ int -__wt_reconcile(WT_SESSION_IMPL *session, WT_REF *ref, WT_SALVAGE_COOKIE *salvage, uint32_t flags, - bool *lookaside_retryp) +__wt_reconcile(WT_SESSION_IMPL *session, WT_REF *ref, WT_SALVAGE_COOKIE *salvage, uint32_t flags) { WT_DECL_RET; WT_PAGE *page; bool no_reconcile_set, page_locked; - if (lookaside_retryp != NULL) - *lookaside_retryp = false; - page = ref->page; - __wt_verbose(session, WT_VERB_RECONCILE, "%p reconcile %s (%s%s%s)", (void *)ref, + __wt_verbose(session, WT_VERB_RECONCILE, "%p reconcile %s (%s%s)", (void *)ref, __wt_page_type_string(page->type), LF_ISSET(WT_REC_EVICT) ? "evict" : "checkpoint", - LF_ISSET(WT_REC_LOOKASIDE) ? ", lookaside" : "", - LF_ISSET(WT_REC_UPDATE_RESTORE) ? ", update/restore" : ""); + LF_ISSET(WT_REC_HS) ? ", history store" : ""); /* * Sanity check flags. * - * We can only do update/restore eviction when the version that ends up - * in the page image is the oldest one any reader could need. - * Otherwise we would need to keep updates in memory that go back older - * than the version in the disk image, and since modify operations - * aren't idempotent, that is problematic. - * * If we try to do eviction using transaction visibility, we had better * have a snapshot. This doesn't apply to checkpoints: there are * (rare) cases where we write data at read-uncommitted isolation. */ - WT_ASSERT(session, !LF_ISSET(WT_REC_LOOKASIDE) || !LF_ISSET(WT_REC_UPDATE_RESTORE)); - WT_ASSERT(session, !LF_ISSET(WT_REC_UPDATE_RESTORE) || LF_ISSET(WT_REC_VISIBLE_ALL)); WT_ASSERT(session, !LF_ISSET(WT_REC_EVICT) || LF_ISSET(WT_REC_VISIBLE_ALL) || F_ISSET(&session->txn, WT_TXN_HAS_SNAPSHOT)); @@ -99,7 +84,7 @@ __wt_reconcile(WT_SESSION_IMPL *session, WT_REF *ref, WT_SALVAGE_COOKIE *salvage * Reconcile the page. The reconciliation code unlocks the page as soon as possible, and returns * that information. */ - ret = __reconcile(session, ref, salvage, flags, lookaside_retryp, &page_locked); + ret = __reconcile(session, ref, salvage, flags, &page_locked); err: if (page_locked) @@ -151,7 +136,7 @@ __reconcile_save_evict_state(WT_SESSION_IMPL *session, WT_REF *ref, uint32_t fla */ static int __reconcile(WT_SESSION_IMPL *session, WT_REF *ref, WT_SALVAGE_COOKIE *salvage, uint32_t flags, - bool *lookaside_retryp, bool *page_lockedp) + bool *page_lockedp) { WT_BTREE *btree; WT_DECL_RET; @@ -196,14 +181,11 @@ __reconcile(WT_SESSION_IMPL *session, WT_REF *ref, WT_SALVAGE_COOKIE *salvage, u } /* - * Update the global lookaside score. Only use observations during eviction, not checkpoints and - * don't count eviction of the lookaside table itself. + * Update the global history store score. Only use observations during eviction, not checkpoints + * and don't count eviction of the history store table itself. */ - if (F_ISSET(r, WT_REC_EVICT) && !F_ISSET(btree, WT_BTREE_LOOKASIDE)) - __wt_cache_update_lookaside_score(session, r->updates_seen, r->updates_unstable); - - /* Check for a successful reconciliation. */ - WT_TRET(__rec_write_check_complete(session, r, ret, lookaside_retryp)); + if (F_ISSET(r, WT_REC_EVICT) && !WT_IS_HS(btree)) + __wt_cache_update_hs_score(session, r->updates_seen, r->updates_unstable); /* Wrap up the page reconciliation. */ if (ret == 0 && (ret = __rec_write_wrapup(session, r, page)) == 0) @@ -228,9 +210,9 @@ __reconcile(WT_SESSION_IMPL *session, WT_REF *ref, WT_SALVAGE_COOKIE *salvage, u WT_STAT_CONN_INCR(session, rec_pages_eviction); WT_STAT_DATA_INCR(session, rec_pages_eviction); } - if (r->cache_write_lookaside) { - WT_STAT_CONN_INCR(session, cache_write_lookaside); - WT_STAT_DATA_INCR(session, cache_write_lookaside); + if (r->cache_write_hs) { + WT_STAT_CONN_INCR(session, cache_write_hs); + WT_STAT_DATA_INCR(session, cache_write_hs); } if (r->cache_write_restore) { WT_STAT_CONN_INCR(session, cache_write_restore); @@ -288,68 +270,6 @@ __reconcile(WT_SESSION_IMPL *session, WT_REF *ref, WT_SALVAGE_COOKIE *salvage, u } /* - * __rec_write_check_complete -- - * Check that reconciliation should complete. - */ -static int -__rec_write_check_complete( - WT_SESSION_IMPL *session, WT_RECONCILE *r, int tret, bool *lookaside_retryp) -{ - /* - * Tests in this function are lookaside tests and tests to decide if rewriting a page in memory - * is worth doing. In-memory configurations can't use a lookaside table, and we ignore page - * rewrite desirability checks for in-memory eviction because a small cache can force us to - * rewrite every possible page. - */ - if (F_ISSET(r, WT_REC_IN_MEMORY)) - return (0); - - /* - * Fall back to lookaside eviction during checkpoints if a page can't be evicted. - */ - if (tret == EBUSY && lookaside_retryp != NULL && !F_ISSET(r, WT_REC_UPDATE_RESTORE) && - !r->update_uncommitted) - *lookaside_retryp = true; - - /* Don't continue if we have already given up. */ - WT_RET(tret); - - /* - * Check if this reconciliation attempt is making progress. If there's any sign of progress, - * don't fall back to the lookaside table. - * - * Check if the current reconciliation split, in which case we'll likely get to write at least - * one of the blocks. If we've created a page image for a page that previously didn't have one, - * or we had a page image and it is now empty, that's also progress. - */ - if (r->multi_next > 1) - return (0); - - /* - * We only suggest lookaside if currently in an evict/restore attempt and some updates were - * saved. Our caller sets the evict/restore flag based on various conditions (like if this is a - * leaf page), which is why we're testing that flag instead of a set of other conditions. If no - * updates were saved, eviction will succeed without needing to restore anything. - */ - if (!F_ISSET(r, WT_REC_UPDATE_RESTORE) || lookaside_retryp == NULL || - (r->multi_next == 1 && r->multi->supd_entries == 0)) - return (0); - - /* - * Check if the current reconciliation applied some updates, in which case evict/restore should - * gain us some space. - * - * Check if lookaside eviction is possible. If any of the updates we saw were uncommitted, the - * lookaside table cannot be used. - */ - if (r->update_uncommitted || r->update_used) - return (0); - - *lookaside_retryp = true; - return (__wt_set_return(session, EBUSY)); -} - -/* * __rec_write_page_status -- * Set the page status after reconciliation. */ @@ -382,10 +302,12 @@ __rec_write_page_status(WT_SESSION_IMPL *session, WT_RECONCILE *r) S2C(session)->modified = true; /* - * Eviction should only be here if following the save/restore eviction path. + * Eviction should only be here if allowing writes to history store or in the in-memory + * eviction case. Otherwise, we must be reconciling a fixed length column store page (which + * does not allow history store content). */ - WT_ASSERT(session, - !F_ISSET(r, WT_REC_EVICT) || F_ISSET(r, WT_REC_LOOKASIDE | WT_REC_UPDATE_RESTORE)); + WT_ASSERT(session, !F_ISSET(r, WT_REC_EVICT) || + (F_ISSET(r, WT_REC_HS | WT_REC_IN_MEMORY) || page->type == WT_PAGE_COL_FIX)); /* * We have written the page, but something prevents it from being evicted. If we wrote the @@ -393,7 +315,7 @@ __rec_write_page_status(WT_SESSION_IMPL *session, WT_RECONCILE *r) * checkpoint would write. Make sure that checkpoint visits the page and (if necessary) * fixes things up. */ - if (r->las_skew_newest) + if (r->hs_skew_newest) mod->first_dirty_txn = WT_TXN_FIRST; } else { /* @@ -513,7 +435,7 @@ __rec_root_write(WT_SESSION_IMPL *session, WT_PAGE *page, uint32_t flags) * Fake up a reference structure, and write the next root page. */ __wt_root_ref_init(session, &fake_ref, next, page->type == WT_PAGE_COL_INT); - return (__wt_reconcile(session, &fake_ref, NULL, flags, NULL)); + return (__wt_reconcile(session, &fake_ref, NULL, flags)); err: __wt_page_out(session, &next); @@ -533,6 +455,7 @@ __rec_init(WT_SESSION_IMPL *session, WT_REF *ref, uint32_t flags, WT_SALVAGE_COO WT_PAGE *page; WT_RECONCILE *r; WT_TXN_GLOBAL *txn_global; + uint64_t ckpt_txn; btree = S2BT(session); page = ref->page; @@ -590,44 +513,51 @@ __rec_init(WT_SESSION_IMPL *session, WT_REF *ref, uint32_t flags, WT_SALVAGE_COO WT_ORDERED_READ(r->last_running, txn_global->last_running); /* + * The checkpoint transaction doesn't pin the oldest txn id, therefore the global last_running + * can move beyond the checkpoint transaction id. When reconciling the metadata, we have to take + * checkpoints into account. + */ + if (WT_IS_METADATA(session->dhandle)) { + WT_ORDERED_READ(ckpt_txn, txn_global->checkpoint_id); + if (ckpt_txn != WT_TXN_NONE && WT_TXNID_LT(ckpt_txn, r->last_running)) + r->last_running = ckpt_txn; + } + + /* * Decide whether to skew on-page values towards newer or older versions. This is a heuristic * attempting to minimize the number of pages that need to be rewritten by future checkpoints. * * We usually prefer to skew to newer versions, the logic being that by the time the next * checkpoint runs, it is likely that all the updates we choose will be stable. However, if * checkpointing with a timestamp (indicated by a stable_timestamp being set), and there is a - * checkpoint already running, or this page was read with lookaside history, or the stable + * checkpoint already running, or this page was read with history store history, or the stable * timestamp hasn't changed since last time this page was successfully, skew oldest instead. */ if (F_ISSET(S2C(session)->cache, WT_CACHE_EVICT_DEBUG_MODE) && __wt_random(&session->rnd) % 3 == 0) - r->las_skew_newest = false; + r->hs_skew_newest = false; else - r->las_skew_newest = LF_ISSET(WT_REC_LOOKASIDE) && LF_ISSET(WT_REC_VISIBLE_ALL); + r->hs_skew_newest = LF_ISSET(WT_REC_HS) && LF_ISSET(WT_REC_VISIBLE_ALL); - if (r->las_skew_newest && !__wt_btree_immediately_durable(session) && + if (r->hs_skew_newest && !__wt_btree_immediately_durable(session) && txn_global->has_stable_timestamp && ((btree->checkpoint_gen != __wt_gen(session, WT_GEN_CHECKPOINT) && txn_global->stable_is_pinned) || - FLD_ISSET(page->modify->restore_state, WT_PAGE_RS_LOOKASIDE) || + FLD_ISSET(page->modify->restore_state, WT_PAGE_RS_HS) || page->modify->last_stable_timestamp == txn_global->stable_timestamp)) - r->las_skew_newest = false; + r->hs_skew_newest = false; - /* - * When operating on the lookaside table, we should never try update/restore or lookaside - * eviction. - */ - WT_ASSERT(session, - !F_ISSET(btree, WT_BTREE_LOOKASIDE) || !LF_ISSET(WT_REC_LOOKASIDE | WT_REC_UPDATE_RESTORE)); + /* When operating on the history store table, we should never try history store eviction. */ + WT_ASSERT(session, !F_ISSET(btree, WT_BTREE_HS) || !LF_ISSET(WT_REC_HS)); /* - * Lookaside table eviction is configured when eviction gets aggressive, + * History store table eviction is configured when eviction gets aggressive, * adjust the flags for cases we don't support. * * We don't yet support fixed-length column-store combined with the - * lookaside table. It's not hard to do, but the underlying function + * history store table. It's not hard to do, but the underlying function * that reviews which updates can be written to the evicted page and - * which updates need to be written to the lookaside table needs access + * which updates need to be written to the history store table needs access * to the original value from the page being evicted, and there's no * code path for that in the case of fixed-length column-store objects. * (Row-store and variable-width column-store objects provide a @@ -636,7 +566,7 @@ __rec_init(WT_SESSION_IMPL *session, WT_REF *ref, uint32_t flags, WT_SALVAGE_COO * now, turn it off. */ if (page->type == WT_PAGE_COL_FIX) - LF_CLR(WT_REC_LOOKASIDE); + LF_CLR(WT_REC_HS); r->flags = flags; @@ -647,10 +577,6 @@ __rec_init(WT_SESSION_IMPL *session, WT_REF *ref, uint32_t flags, WT_SALVAGE_COO /* Track if updates were used and/or uncommitted. */ r->updates_seen = r->updates_unstable = 0; - r->update_uncommitted = r->update_used = false; - - /* Track if all the updates are with prepare in-progress state. */ - r->all_upd_prepare_in_prog = true; /* Track if the page can be marked clean. */ r->leave_dirty = false; @@ -713,7 +639,7 @@ __rec_init(WT_SESSION_IMPL *session, WT_REF *ref, uint32_t flags, WT_SALVAGE_COO r->salvage = salvage; - r->cache_write_lookaside = r->cache_write_restore = false; + r->cache_write_hs = r->cache_write_restore = false; /* * The fake cursor used to figure out modified update values points to the enclosing WT_REF as a @@ -1136,11 +1062,12 @@ __rec_split_row_promote(WT_SESSION_IMPL *session, WT_RECONCILE *r, WT_ITEM *key, /* * Note #2: if we skipped updates, an update key may be larger than the last key stored in the - * previous block (probable for append-centric workloads). If there are skipped updates, check - * for one larger than the last key and smaller than the current key. + * previous block (probable for append-centric workloads). If there are skipped updates and we + * cannot evict the page, check for one larger than the last key and smaller than the current + * key. */ max = r->last; - if (F_ISSET(r, WT_REC_UPDATE_RESTORE)) + if (r->cache_write_restore) for (i = r->supd_next; i > 0; --i) { supd = &r->supd[i - 1]; if (supd->ins == NULL) @@ -1480,8 +1407,12 @@ __wt_rec_split_finish(WT_SESSION_IMPL *session, WT_RECONCILE *r) * * Pages with skipped or not-yet-globally visible updates aren't really empty; otherwise, the * page is truly empty and we will merge it into its parent during the parent's reconciliation. + * + * Checkpoint never writes uncommitted changes to disk and only saves the updates to move older + * updates to the history store. Thus it can consider the reconciliation done if there are no + * more entries left to write. This will also remove its reference entry from its parent. */ - if (r->entries == 0 && r->supd_next == 0) + if (r->entries == 0 && (r->supd_next == 0 || F_ISSET(r, WT_REC_CHECKPOINT))) return (0); /* Set the number of entries and size for the just finished chunk. */ @@ -1545,7 +1476,7 @@ __rec_split_write_supd( WT_RET(__rec_supd_move(session, multi, r->supd, r->supd_next)); r->supd_next = 0; r->supd_memsize = 0; - goto done; + return (ret); } /* @@ -1600,14 +1531,6 @@ __rec_split_write_supd( r->supd_next = j; } -done: - if (F_ISSET(r, WT_REC_LOOKASIDE)) { - /* Track the oldest lookaside timestamp seen so far. */ - multi->page_las.max_txn = r->max_txn; - multi->page_las.max_ondisk_ts = r->max_ondisk_ts; - multi->page_las.min_skipped_ts = r->min_skipped_ts; - } - err: __wt_scr_free(session, &key); return (ret); @@ -1645,17 +1568,8 @@ __rec_split_write_header(WT_SESSION_IMPL *session, WT_RECONCILE *r, WT_REC_CHUNK F_SET(dsk, WT_PAGE_EMPTY_V_NONE); } - /* - * Note in the page header if using the lookaside table eviction path and we found updates that - * weren't globally visible when reconciling this page. - */ - if (F_ISSET(r, WT_REC_LOOKASIDE) && multi->supd != NULL) - F_SET(dsk, WT_PAGE_LAS_UPDATE); - dsk->unused = 0; - dsk->version = __wt_process.page_version_ts ? WT_PAGE_VERSION_TS : WT_PAGE_VERSION_ORIG; - /* Clear the memory owned by the block manager. */ memset(WT_BLOCK_HEADER_REF(dsk), 0, btree->block_header); } @@ -1846,8 +1760,12 @@ __rec_split_write(WT_SESSION_IMPL *session, WT_RECONCILE *r, WT_REC_CHUNK *chunk WT_RET(__wt_realloc_def(session, &r->multi_allocated, r->multi_next + 1, &r->multi)); multi = &r->multi[r->multi_next++]; + /* + * FIXME-prepare-support: audit the use of durable timestamps in this file, use both durable + * timestamps. + */ /* Initialize the address (set the addr type for the parent). */ - multi->addr.newest_durable_ts = chunk->newest_durable_ts; + multi->addr.stop_durable_ts = chunk->newest_durable_ts; multi->addr.oldest_start_ts = chunk->oldest_start_ts; multi->addr.oldest_start_txn = chunk->oldest_start_txn; multi->addr.newest_stop_ts = chunk->newest_stop_ts; @@ -1913,36 +1831,22 @@ __rec_split_write(WT_SESSION_IMPL *session, WT_RECONCILE *r, WT_REC_CHUNK *chunk if (F_ISSET(r, WT_REC_IN_MEMORY)) goto copy_image; - /* - * If there are saved updates, either doing update/restore eviction or lookaside eviction. - */ - if (multi->supd != NULL) { + /* Check the eviction flag as checkpoint also saves updates. */ + if (F_ISSET(r, WT_REC_EVICT) && multi->supd != NULL) { /* * XXX If no entries were used, the page is empty and we can only restore eviction/restore - * or lookaside updates against empty row-store leaf pages, column-store modify attempts to - * allocate a zero-length array. + * or history store updates against empty row-store leaf pages, column-store modify attempts + * to allocate a zero-length array. */ if (r->page->type != WT_PAGE_ROW_LEAF && chunk->entries == 0) return (__wt_set_return(session, EBUSY)); - if (F_ISSET(r, WT_REC_LOOKASIDE)) { - r->cache_write_lookaside = true; - - /* - * Lookaside eviction writes disk images, but if no entries were used, there's no disk - * image to write. There's no more work to do in this case, lookaside eviction doesn't - * copy disk images. - */ - if (chunk->entries == 0) - return (0); - } else { - r->cache_write_restore = true; - - /* - * Update/restore never writes a disk image, but always copies a disk image. - */ + /* If we need to restore the page to memory, copy the disk image. */ + if (r->cache_write_restore) goto copy_image; - } + + if (chunk->entries == 0) + return (0); } /* @@ -1981,12 +1885,11 @@ copy_image: __wt_verify_dsk_image( session, "[reconcile-image]", chunk->image.data, 0, &multi->addr, true) == 0); #endif - /* * If re-instantiating this page in memory (either because eviction wants to, or because we * skipped updates to build the disk image), save a copy of the disk image. */ - if (F_ISSET(r, WT_REC_SCRUB) || (F_ISSET(r, WT_REC_UPDATE_RESTORE) && multi->supd != NULL)) + if (F_ISSET(r, WT_REC_SCRUB) || (r->cache_write_restore && multi->supd != NULL)) WT_RET(__wt_memdup(session, chunk->image.data, chunk->image.size, &multi->disk_image)); return (0); @@ -2198,6 +2101,7 @@ __rec_write_wrapup(WT_SESSION_IMPL *session, WT_RECONCILE *r, WT_PAGE *page) */ if (__wt_ref_is_root(ref)) break; + WT_RET(__wt_ref_block_free(session, ref)); break; case WT_PM_REC_EMPTY: /* Page deleted */ @@ -2231,11 +2135,11 @@ __rec_write_wrapup(WT_SESSION_IMPL *session, WT_RECONCILE *r, WT_PAGE *page) mod->rec_result = 0; /* - * If using the lookaside table eviction path and we found updates that weren't globally visible - * when reconciling this page, copy them into the database's lookaside store. + * If using the history store table eviction path and we found updates that weren't globally + * visible when reconciling this page, copy them into the database's history store. */ - if (F_ISSET(r, WT_REC_LOOKASIDE)) - WT_RET(__rec_las_wrapup(session, r)); + if (F_ISSET(r, WT_REC_HS)) + WT_RET(__rec_hs_wrapup(session, r)); /* * Wrap up overflow tracking. If we are about to create a checkpoint, the system must be @@ -2283,9 +2187,11 @@ __rec_write_wrapup(WT_SESSION_IMPL *session, WT_RECONCILE *r, WT_PAGE *page) * eviction has decided to retain the page in memory because the latter can't handle * update lists and splits can. */ - if (F_ISSET(r, WT_REC_IN_MEMORY) || - (F_ISSET(r, WT_REC_UPDATE_RESTORE) && r->multi->supd_entries != 0)) + if (F_ISSET(r, WT_REC_IN_MEMORY) || r->cache_write_restore) { + WT_ASSERT(session, F_ISSET(r, WT_REC_IN_MEMORY) || + (F_ISSET(r, WT_REC_EVICT) && r->leave_dirty && r->multi->supd_entries != 0)); goto split; + } /* * We may have a root page, create a sync point. (The write code ignores root page updates, @@ -2296,9 +2202,8 @@ __rec_write_wrapup(WT_SESSION_IMPL *session, WT_RECONCILE *r, WT_PAGE *page) r->multi->addr.addr = NULL; mod->mod_disk_image = r->multi->disk_image; r->multi->disk_image = NULL; - mod->mod_page_las = r->multi->page_las; } else { - __wt_checkpoint_tree_reconcile_update(session, r->multi->addr.newest_durable_ts, + __wt_checkpoint_tree_reconcile_update(session, r->multi->addr.stop_durable_ts, r->multi->addr.oldest_start_ts, r->multi->addr.oldest_start_txn, r->multi->addr.newest_stop_ts, r->multi->addr.newest_stop_txn); WT_RET(__wt_bt_write(session, r->wrapup_checkpoint, NULL, NULL, NULL, true, @@ -2324,6 +2229,7 @@ __rec_write_wrapup(WT_SESSION_IMPL *session, WT_RECONCILE *r, WT_PAGE *page) split: for (multi = r->multi, i = 0; i < r->multi_next; ++multi, ++i) multi->addr.reuse = 0; + mod->mod_multi = r->multi; mod->mod_multi_entries = r->multi_next; mod->rec_result = WT_PM_REC_MULTIBLOCK; @@ -2373,31 +2279,22 @@ __rec_write_wrapup_err(WT_SESSION_IMPL *session, WT_RECONCILE *r, WT_PAGE *page) WT_TRET(__wt_btree_block_free(session, multi->addr.addr, multi->addr.size)); } - /* - * If using the lookaside table eviction path and we found updates that weren't globally visible - * when reconciling this page, we might have already copied them into the database's lookaside - * store. Remove them. - */ - if (F_ISSET(r, WT_REC_LOOKASIDE)) - WT_TRET(__rec_las_wrapup_err(session, r)); - WT_TRET(__wt_ovfl_track_wrapup_err(session, page)); return (ret); } /* - * __rec_las_wrapup -- - * Copy all of the saved updates into the database's lookaside table. + * __rec_hs_wrapup -- + * Copy all of the saved updates into the database's history store table. */ static int -__rec_las_wrapup(WT_SESSION_IMPL *session, WT_RECONCILE *r) +__rec_hs_wrapup(WT_SESSION_IMPL *session, WT_RECONCILE *r) { - WT_CURSOR *cursor; - WT_DECL_ITEM(key); WT_DECL_RET; WT_MULTI *multi; uint32_t i, session_flags; + bool is_owner; /* Check if there's work to do. */ for (multi = r->multi, i = 0; i < r->multi_next; ++multi, ++i) @@ -2406,46 +2303,20 @@ __rec_las_wrapup(WT_SESSION_IMPL *session, WT_RECONCILE *r) if (i == r->multi_next) return (0); - /* Ensure enough room for a column-store key without checking. */ - WT_RET(__wt_scr_alloc(session, WT_INTPACK64_MAXSIZE, &key)); - - __wt_las_cursor(session, &cursor, &session_flags); + WT_RET(__wt_hs_cursor(session, &session_flags, &is_owner)); for (multi = r->multi, i = 0; i < r->multi_next; ++multi, ++i) if (multi->supd != NULL) { - WT_ERR(__wt_las_insert_block(cursor, S2BT(session), r->page, multi, key)); - - __wt_free(session, multi->supd); - multi->supd_entries = 0; + WT_ERR(__wt_hs_insert_updates(session->hs_cursor, S2BT(session), r->page, multi)); + r->cache_write_hs = true; + if (!r->cache_write_restore) { + __wt_free(session, multi->supd); + multi->supd_entries = 0; + } } err: - WT_TRET(__wt_las_cursor_close(session, &cursor, session_flags)); - - __wt_scr_free(session, &key); - return (ret); -} - -/* - * __rec_las_wrapup_err -- - * Discard any saved updates from the database's lookaside buffer. - */ -static int -__rec_las_wrapup_err(WT_SESSION_IMPL *session, WT_RECONCILE *r) -{ - WT_DECL_RET; - WT_MULTI *multi; - uint64_t las_pageid; - uint32_t i; - - /* - * Note the additional check for a non-zero lookaside page ID, that flags if lookaside table - * entries for this page have been written. - */ - for (multi = r->multi, i = 0; i < r->multi_next; ++multi, ++i) - if (multi->supd != NULL && (las_pageid = multi->page_las.las_pageid) != 0) - WT_TRET(__wt_las_remove_block(session, las_pageid)); - + WT_TRET(__wt_hs_cursor_close(session, session_flags, is_owner)); return (ret); } diff --git a/src/third_party/wiredtiger/src/schema/schema_create.c b/src/third_party/wiredtiger/src/schema/schema_create.c index 96b2512d6ce..c33fcb2be02 100644 --- a/src/third_party/wiredtiger/src/schema/schema_create.c +++ b/src/third_party/wiredtiger/src/schema/schema_create.c @@ -58,7 +58,7 @@ __create_file(WT_SESSION_IMPL *session, const char *uri, bool exclusive, const c *filecfg[] = {WT_CONFIG_BASE(session, file_meta), config, NULL, NULL}; char *fileconf; uint32_t allocsize; - bool is_metadata; + bool exists, is_metadata; fileconf = NULL; @@ -74,6 +74,20 @@ __create_file(WT_SESSION_IMPL *session, const char *uri, bool exclusive, const c goto err; } + exists = false; + /* + * At this moment the uri doesn't exist in the metadata. In scenarios like, the database folder + * is copied without a checkpoint into another location and trying to recover from it leads to + * that history store file exists on disk but not as part of metadata. As we recreate the + * history store file on every restart to ensure that history store file is present. Make sure + * to remove the already exist history store file in the directory. + */ + if (strcmp(uri, WT_HS_URI) == 0) { + WT_IGNORE_RET(__wt_fs_exist(session, filename, &exists)); + if (exists) + WT_IGNORE_RET(__wt_fs_remove(session, filename, true)); + } + /* Sanity check the allocation size. */ WT_ERR(__wt_direct_io_size_check(session, filecfg, "allocation_size", &allocsize)); diff --git a/src/third_party/wiredtiger/src/schema/schema_util.c b/src/third_party/wiredtiger/src/schema/schema_util.c index 2c65e4297db..25ef013648c 100644 --- a/src/third_party/wiredtiger/src/schema/schema_util.c +++ b/src/third_party/wiredtiger/src/schema/schema_util.c @@ -111,6 +111,33 @@ __wt_schema_session_release(WT_SESSION_IMPL *session, WT_SESSION_IMPL *int_sessi } /* + * __str_name_check -- + * Internal function to disallow any use of the WiredTiger name space. Can be called directly or + * after skipping the URI prefix. + */ +static int +__str_name_check(WT_SESSION_IMPL *session, const char *name, bool skip_wt) +{ + + if (!skip_wt && WT_PREFIX_MATCH(name, "WiredTiger")) + WT_RET_MSG(session, EINVAL, + "%s: the \"WiredTiger\" name space may not be " + "used by applications", + name); + + /* + * Disallow JSON quoting characters -- the config string parsing code supports quoted strings, + * but there's no good reason to use them in names and we're not going to do the testing. + */ + if (strpbrk(name, "{},:[]\\\"'") != NULL) + WT_RET_MSG(session, EINVAL, + "%s: WiredTiger objects should not include grouping " + "characters in their names", + name); + return (0); +} + +/* * __wt_str_name_check -- * Disallow any use of the WiredTiger name space. */ @@ -119,36 +146,24 @@ __wt_str_name_check(WT_SESSION_IMPL *session, const char *str) { int skipped; const char *name, *sep; + bool skip; /* * Check if name is somewhere in the WiredTiger name space: it would be * "bad" if the application truncated the metadata file. Skip any - * leading URI prefix, check and then skip over a table name. + * leading URI prefix if needed, check and then skip over a table name. */ name = str; + skip = false; for (skipped = 0; skipped < 2; skipped++) { - if ((sep = strchr(name, ':')) == NULL) + if ((sep = strchr(name, ':')) == NULL) { + skip = true; break; + } name = sep + 1; - if (WT_PREFIX_MATCH(name, "WiredTiger")) - WT_RET_MSG(session, EINVAL, - "%s: the \"WiredTiger\" name space may not be " - "used by applications", - name); } - - /* - * Disallow JSON quoting characters -- the config string parsing code supports quoted strings, - * but there's no good reason to use them in names and we're not going to do the testing. - */ - if (strpbrk(name, "{},:[]\\\"'") != NULL) - WT_RET_MSG(session, EINVAL, - "%s: WiredTiger objects should not include grouping " - "characters in their names", - name); - - return (0); + return (__str_name_check(session, name, skip)); } /* @@ -156,7 +171,7 @@ __wt_str_name_check(WT_SESSION_IMPL *session, const char *str) * Disallow any use of the WiredTiger name space. */ int -__wt_name_check(WT_SESSION_IMPL *session, const char *str, size_t len) +__wt_name_check(WT_SESSION_IMPL *session, const char *str, size_t len, bool check_uri) { WT_DECL_ITEM(tmp); WT_DECL_RET; @@ -165,7 +180,9 @@ __wt_name_check(WT_SESSION_IMPL *session, const char *str, size_t len) WT_ERR(__wt_buf_fmt(session, tmp, "%.*s", (int)len, str)); - ret = __wt_str_name_check(session, tmp->data); + /* If we want to skip the URI check call the internal function directly. */ + ret = check_uri ? __wt_str_name_check(session, tmp->data) : + __str_name_check(session, tmp->data, false); err: __wt_scr_free(session, &tmp); diff --git a/src/third_party/wiredtiger/src/session/session_api.c b/src/third_party/wiredtiger/src/session/session_api.c index a67c263d7d9..a3a5b9ef143 100644 --- a/src/third_party/wiredtiger/src/session/session_api.c +++ b/src/third_party/wiredtiger/src/session/session_api.c @@ -223,7 +223,7 @@ __session_close_cursors(WT_SESSION_IMPL *session, WT_CURSOR_LIST *cursors) */ WT_TRET_NOTFOUND_OK(cursor->reopen(cursor, false)); else if (session->event_handler->handle_close != NULL && - strcmp(cursor->internal_uri, WT_LAS_URI) != 0) + strcmp(cursor->internal_uri, WT_HS_URI) != 0) /* * Notify the user that we are closing the cursor handle via the registered close * callback. @@ -1566,6 +1566,7 @@ err: static int __session_verify(WT_SESSION *wt_session, const char *uri, const char *config) { + WT_CONFIG_ITEM cval; WT_DECL_RET; WT_SESSION_IMPL *session; @@ -1575,12 +1576,34 @@ __session_verify(WT_SESSION *wt_session, const char *uri, const char *config) WT_ERR(__wt_inmem_unsupported_op(session, NULL)); - /* Block out checkpoints to avoid spurious EBUSY errors. */ - WT_WITH_CHECKPOINT_LOCK( - session, WT_WITH_SCHEMA_LOCK(session, ret = __wt_schema_worker(session, uri, __wt_verify, - NULL, cfg, WT_DHANDLE_EXCLUSIVE | WT_BTREE_VERIFY))); + /* + * Even if we're not verifying the history store, we need to be able to iterate over the history + * store content for another table. In order to do this, we must ignore tombstones in the + * history store since every history store record is succeeded with a tombstone. + */ + F_SET(session, WT_SESSION_IGNORE_HS_TOMBSTONE); + /* Block out checkpoints to avoid spurious EBUSY errors. */ + WT_ERR(__wt_config_gets(session, cfg, "history_store", &cval)); + if (cval.val == true) { + /* Can't give a URI with history store verification. */ + if (uri != NULL) + WT_ERR_MSG(session, EINVAL, "URI not applicable when verifying the history store"); + + WT_WITH_CHECKPOINT_LOCK(session, + WT_WITH_SCHEMA_LOCK(session, ret = __wt_verify_history_store_tree(session, NULL))); + } else { + WT_WITH_CHECKPOINT_LOCK(session, + WT_WITH_SCHEMA_LOCK(session, ret = __wt_schema_worker(session, uri, __wt_verify, NULL, + cfg, WT_DHANDLE_EXCLUSIVE | WT_BTREE_VERIFY))); + WT_ERR(ret); + /* TODO: WT-5643 Add history store verification for non file URI */ + if (WT_PREFIX_MATCH(uri, "file:")) + WT_WITH_CHECKPOINT_LOCK(session, + WT_WITH_SCHEMA_LOCK(session, ret = __wt_verify_history_store_tree(session, uri))); + } err: + F_CLR(session, WT_SESSION_IGNORE_HS_TOMBSTONE); if (ret != 0) WT_STAT_CONN_INCR(session, session_table_verify_fail); else @@ -1986,42 +2009,6 @@ err: } /* - * __session_snapshot -- - * WT_SESSION->snapshot method. - */ -static int -__session_snapshot(WT_SESSION *wt_session, const char *config) -{ - WT_DECL_RET; - WT_SESSION_IMPL *session; - WT_TXN_GLOBAL *txn_global; - bool has_create, has_drop; - - has_create = has_drop = false; - session = (WT_SESSION_IMPL *)wt_session; - txn_global = &S2C(session)->txn_global; - - SESSION_API_CALL(session, snapshot, config, cfg); - - WT_ERR(__wt_txn_named_snapshot_config(session, cfg, &has_create, &has_drop)); - - __wt_writelock(session, &txn_global->nsnap_rwlock); - - /* Drop any snapshots to be removed first. */ - if (has_drop) - WT_ERR(__wt_txn_named_snapshot_drop(session, cfg)); - - /* Start the named snapshot if requested. */ - if (has_create) - WT_ERR(__wt_txn_named_snapshot_begin(session, cfg)); - -err: - __wt_writeunlock(session, &txn_global->nsnap_rwlock); - - API_END_RET_NOTFOUND_MAP(session, ret); -} - -/* * __wt_session_strerror -- * WT_SESSION->strerror method. */ @@ -2063,8 +2050,8 @@ __open_session(WT_CONNECTION_IMPL *conn, WT_EVENT_HANDLER *event_handler, const __session_salvage, __session_truncate, __session_upgrade, __session_verify, __session_begin_transaction, __session_commit_transaction, __session_prepare_transaction, __session_rollback_transaction, __session_timestamp_transaction, __session_query_timestamp, - __session_checkpoint, __session_snapshot, __session_transaction_pinned_range, - __session_transaction_sync, __wt_session_breakpoint}, + __session_checkpoint, __session_transaction_pinned_range, __session_transaction_sync, + __wt_session_breakpoint}, stds_readonly = {NULL, NULL, __session_close, __session_reconfigure, __wt_session_strerror, __session_open_cursor, __session_alter_readonly, __session_create_readonly, __session_import_readonly, __wt_session_compact_readonly, __session_drop_readonly, @@ -2074,7 +2061,7 @@ __open_session(WT_CONNECTION_IMPL *conn, WT_EVENT_HANDLER *event_handler, const __session_verify, __session_begin_transaction, __session_commit_transaction, __session_prepare_transaction_readonly, __session_rollback_transaction, __session_timestamp_transaction, __session_query_timestamp, __session_checkpoint_readonly, - __session_snapshot, __session_transaction_pinned_range, __session_transaction_sync_readonly, + __session_transaction_pinned_range, __session_transaction_sync_readonly, __wt_session_breakpoint}; WT_DECL_RET; WT_SESSION_IMPL *session, *session_ret; diff --git a/src/third_party/wiredtiger/src/session/session_dhandle.c b/src/third_party/wiredtiger/src/session/session_dhandle.c index 6582417331f..8ec514caf46 100644 --- a/src/third_party/wiredtiger/src/session/session_dhandle.c +++ b/src/third_party/wiredtiger/src/session/session_dhandle.c @@ -363,6 +363,7 @@ __session_dhandle_sweep(WT_SESSION_IMPL *session) WT_DATA_HANDLE *dhandle; WT_DATA_HANDLE_CACHE *dhandle_cache, *dhandle_cache_tmp; uint64_t now; + bool empty_btree; conn = S2C(session); @@ -379,9 +380,15 @@ __session_dhandle_sweep(WT_SESSION_IMPL *session) TAILQ_FOREACH_SAFE(dhandle_cache, &session->dhandles, q, dhandle_cache_tmp) { dhandle = dhandle_cache->dhandle; + empty_btree = false; + if (dhandle->type == WT_DHANDLE_TYPE_BTREE) + WT_WITH_DHANDLE( + session, dhandle, empty_btree = (__wt_btree_bytes_evictable(session) == 0)); + if (dhandle != session->dhandle && dhandle->session_inuse == 0 && (WT_DHANDLE_INACTIVE(dhandle) || - (dhandle->timeofdeath != 0 && now - dhandle->timeofdeath > conn->sweep_idle_time))) { + (dhandle->timeofdeath != 0 && now - dhandle->timeofdeath > conn->sweep_idle_time) || + empty_btree)) { WT_STAT_CONN_INCR(session, dh_session_handles); WT_ASSERT(session, !WT_IS_METADATA(dhandle)); __session_discard_dhandle(session, dhandle_cache); diff --git a/src/third_party/wiredtiger/src/support/hazard.c b/src/third_party/wiredtiger/src/support/hazard.c index 4770ef57e67..34101eb2242 100644 --- a/src/third_party/wiredtiger/src/support/hazard.c +++ b/src/third_party/wiredtiger/src/support/hazard.c @@ -57,11 +57,11 @@ hazard_grow(WT_SESSION_IMPL *session) } /* - * __wt_hazard_set -- + * __wt_hazard_set_func -- * Set a hazard pointer. */ int -__wt_hazard_set(WT_SESSION_IMPL *session, WT_REF *ref, bool *busyp +__wt_hazard_set_func(WT_SESSION_IMPL *session, WT_REF *ref, bool *busyp #ifdef HAVE_DIAGNOSTIC , const char *func, int line @@ -69,7 +69,7 @@ __wt_hazard_set(WT_SESSION_IMPL *session, WT_REF *ref, bool *busyp ) { WT_HAZARD *hp; - uint32_t current_state; + uint8_t current_state; *busyp = false; @@ -82,7 +82,7 @@ __wt_hazard_set(WT_SESSION_IMPL *session, WT_REF *ref, bool *busyp * re-check it after a barrier to make sure we have a valid reference. */ current_state = ref->state; - if (current_state != WT_REF_LIMBO && current_state != WT_REF_MEM) { + if (current_state != WT_REF_MEM) { *busyp = true; return (0); } @@ -124,8 +124,8 @@ __wt_hazard_set(WT_SESSION_IMPL *session, WT_REF *ref, bool *busyp /* * Do the dance: * - * The memory location which makes a page "real" is the WT_REF's state of WT_REF_LIMBO or - * WT_REF_MEM, which can be set to WT_REF_LOCKED at any time by the page eviction server. + * The memory location which makes a page "real" is the WT_REF's state of WT_REF_MEM, which can + * be set to WT_REF_LOCKED at any time by the page eviction server. * * Add the WT_REF reference to the session's hazard list and flush the write, then see if the * page's state is still valid. If so, we can use the page because the page eviction server will @@ -141,11 +141,10 @@ __wt_hazard_set(WT_SESSION_IMPL *session, WT_REF *ref, bool *busyp WT_FULL_BARRIER(); /* - * Check if the page state is still valid, where valid means a state of WT_REF_LIMBO or - * WT_REF_MEM. + * Check if the page state is still valid, where valid means a state of WT_REF_MEM. */ current_state = ref->state; - if (current_state == WT_REF_LIMBO || current_state == WT_REF_MEM) { + if (current_state == WT_REF_MEM) { ++session->nhazard; /* diff --git a/src/third_party/wiredtiger/src/support/modify.c b/src/third_party/wiredtiger/src/support/modify.c index b891d050745..caaaa7abfbb 100644 --- a/src/third_party/wiredtiger/src/support/modify.c +++ b/src/third_party/wiredtiger/src/support/modify.c @@ -73,7 +73,7 @@ __wt_modify_idempotent(const void *modify) * Pack a modify structure into a buffer. */ int -__wt_modify_pack(WT_CURSOR *cursor, WT_ITEM **modifyp, WT_MODIFY *entries, int nentries) +__wt_modify_pack(WT_CURSOR *cursor, WT_MODIFY *entries, int nentries, WT_ITEM **modifyp) { WT_ITEM *modify; WT_SESSION_IMPL *session; @@ -82,6 +82,7 @@ __wt_modify_pack(WT_CURSOR *cursor, WT_ITEM **modifyp, WT_MODIFY *entries, int n int i; session = (WT_SESSION_IMPL *)cursor->session; + *modifyp = NULL; /* * Build the in-memory modify value. It's the entries count, followed by the modify structure @@ -346,22 +347,32 @@ __modify_apply_no_overlap(WT_SESSION_IMPL *session, WT_ITEM *value, const size_t /* * __wt_modify_apply -- - * Apply a single set of WT_MODIFY changes to a buffer. + * Apply a single set of WT_MODIFY changes to a cursor buffer. */ int __wt_modify_apply(WT_CURSOR *cursor, const void *modify) { - WT_ITEM *value; - WT_MODIFY mod; WT_SESSION_IMPL *session; - size_t datasz, destsz, item_offset, tmp; - const size_t *p; - int napplied, nentries; - bool overlap, sformat; + bool sformat; session = (WT_SESSION_IMPL *)cursor->session; sformat = cursor->value_format[0] == 'S'; - value = &cursor->value; + + return (__wt_modify_apply_item(session, &cursor->value, modify, sformat)); +} + +/* + * __wt_modify_apply_item -- + * Apply a single set of WT_MODIFY changes to a WT_ITEM buffer. + */ +int +__wt_modify_apply_item(WT_SESSION_IMPL *session, WT_ITEM *value, const void *modify, bool sformat) +{ + WT_MODIFY mod; + size_t datasz, destsz, item_offset, tmp; + const size_t *p; + int napplied, nentries; + bool overlap; /* * Get the number of modify entries and set a second pointer to reference the replacement data. @@ -425,10 +436,90 @@ __wt_modify_apply_api(WT_CURSOR *cursor, WT_MODIFY *entries, int nentries) WT_DECL_ITEM(modify); WT_DECL_RET; - WT_ERR(__wt_modify_pack(cursor, &modify, entries, nentries)); + WT_ERR(__wt_modify_pack(cursor, entries, nentries, &modify)); WT_ERR(__wt_modify_apply(cursor, modify->data)); err: __wt_scr_free((WT_SESSION_IMPL *)cursor->session, &modify); return (ret); } + +/* + * __wt_modify_vector_init -- + * Initialize a modify vector. + */ +void +__wt_modify_vector_init(WT_SESSION_IMPL *session, WT_MODIFY_VECTOR *modifies) +{ + WT_CLEAR(*modifies); + modifies->session = session; + modifies->listp = modifies->list; +} + +/* + * __wt_modify_vector_push -- + * Push a modify update pointer to a modify vector. If we exceed the allowed stack space in the + * vector, we'll be doing malloc here. + */ +int +__wt_modify_vector_push(WT_MODIFY_VECTOR *modifies, WT_UPDATE *upd) +{ + WT_DECL_RET; + bool migrate_from_stack; + + migrate_from_stack = false; + + if (modifies->size >= WT_MODIFY_VECTOR_STACK_SIZE) { + if (modifies->allocated_bytes == 0 && modifies->size == WT_MODIFY_VECTOR_STACK_SIZE) { + migrate_from_stack = true; + modifies->listp = NULL; + } + WT_ERR(__wt_realloc_def( + modifies->session, &modifies->allocated_bytes, modifies->size + 1, &modifies->listp)); + if (migrate_from_stack) + memcpy(modifies->listp, modifies->list, sizeof(modifies->list)); + } + modifies->listp[modifies->size++] = upd; + return (0); + +err: + /* + * This only happens when we're migrating from the stack to the heap but failed to allocate. In + * that case, point back to the stack allocated memory and set the allocation to zero to + * indicate that we don't have heap memory to free. + * + * If we're already on the heap, we have nothing to do. The realloc call above won't touch the + * list pointer unless allocation is successful and we won't have incremented the size yet. + */ + if (modifies->listp == NULL) { + WT_ASSERT(modifies->session, modifies->size == WT_MODIFY_VECTOR_STACK_SIZE); + modifies->listp = modifies->list; + modifies->allocated_bytes = 0; + } + return (ret); +} + +/* + * __wt_modify_vector_pop -- + * Pop an update pointer off a modify vector. + */ +void +__wt_modify_vector_pop(WT_MODIFY_VECTOR *modifies, WT_UPDATE **updp) +{ + WT_ASSERT(modifies->session, modifies->size > 0); + + *updp = modifies->listp[--modifies->size]; +} + +/* + * __wt_modify_vector_free -- + * Free any resources associated with a modify vector. If we exceeded the allowed stack space on + * the vector and had to fallback to dynamic allocations, we'll be doing a free here. + */ +void +__wt_modify_vector_free(WT_MODIFY_VECTOR *modifies) +{ + if (modifies->allocated_bytes != 0) + __wt_free(modifies->session, modifies->listp); + __wt_modify_vector_init(modifies->session, modifies); +} diff --git a/src/third_party/wiredtiger/src/support/stat.c b/src/third_party/wiredtiger/src/support/stat.c index 0232ea1d2c6..dd9210ceecb 100644 --- a/src/third_party/wiredtiger/src/support/stat.c +++ b/src/third_party/wiredtiger/src/support/stat.c @@ -39,15 +39,14 @@ static const char *const __stats_dsrc_desc[] = { "cache: eviction walks gave up because they saw too many pages and found too few candidates", "cache: eviction walks reached end of tree", "cache: eviction walks started from root of tree", "cache: eviction walks started from saved location in tree", - "cache: hazard pointer blocked page eviction", + "cache: hazard pointer blocked page eviction", "cache: history store table reads", "cache: in-memory page passed criteria to be split", "cache: in-memory page splits", "cache: internal pages evicted", "cache: internal pages split during eviction", "cache: leaf pages split during eviction", "cache: modified pages evicted", "cache: overflow pages read into cache", "cache: page split during eviction deepened the tree", - "cache: page written requiring cache overflow records", "cache: pages read into cache", + "cache: page written requiring history store records", "cache: pages read into cache", "cache: pages read into cache after truncate", "cache: pages read into cache after truncate in prepare state", - "cache: pages read into cache requiring cache overflow entries", "cache: pages requested from the cache", "cache: pages seen by eviction walk", "cache: pages written from cache", "cache: pages written requiring in-memory restoration", "cache: tracked dirty bytes in the cache", "cache: unmodified pages evicted", @@ -192,6 +191,7 @@ __wt_stat_dsrc_clear_single(WT_DSRC_STATS *stats) stats->cache_eviction_walk_from_root = 0; stats->cache_eviction_walk_saved_pos = 0; stats->cache_eviction_hazard = 0; + stats->cache_hs_read = 0; stats->cache_inmem_splittable = 0; stats->cache_inmem_split = 0; stats->cache_eviction_internal = 0; @@ -200,11 +200,10 @@ __wt_stat_dsrc_clear_single(WT_DSRC_STATS *stats) stats->cache_eviction_dirty = 0; stats->cache_read_overflow = 0; stats->cache_eviction_deepen = 0; - stats->cache_write_lookaside = 0; + stats->cache_write_hs = 0; stats->cache_read = 0; stats->cache_read_deleted = 0; stats->cache_read_deleted_prepared = 0; - stats->cache_read_lookaside = 0; stats->cache_pages_requested = 0; stats->cache_eviction_pages_seen = 0; stats->cache_write = 0; @@ -364,6 +363,7 @@ __wt_stat_dsrc_aggregate_single(WT_DSRC_STATS *from, WT_DSRC_STATS *to) to->cache_eviction_walk_from_root += from->cache_eviction_walk_from_root; to->cache_eviction_walk_saved_pos += from->cache_eviction_walk_saved_pos; to->cache_eviction_hazard += from->cache_eviction_hazard; + to->cache_hs_read += from->cache_hs_read; to->cache_inmem_splittable += from->cache_inmem_splittable; to->cache_inmem_split += from->cache_inmem_split; to->cache_eviction_internal += from->cache_eviction_internal; @@ -372,11 +372,10 @@ __wt_stat_dsrc_aggregate_single(WT_DSRC_STATS *from, WT_DSRC_STATS *to) to->cache_eviction_dirty += from->cache_eviction_dirty; to->cache_read_overflow += from->cache_read_overflow; to->cache_eviction_deepen += from->cache_eviction_deepen; - to->cache_write_lookaside += from->cache_write_lookaside; + to->cache_write_hs += from->cache_write_hs; to->cache_read += from->cache_read; to->cache_read_deleted += from->cache_read_deleted; to->cache_read_deleted_prepared += from->cache_read_deleted_prepared; - to->cache_read_lookaside += from->cache_read_lookaside; to->cache_pages_requested += from->cache_pages_requested; to->cache_eviction_pages_seen += from->cache_eviction_pages_seen; to->cache_write += from->cache_write; @@ -532,6 +531,7 @@ __wt_stat_dsrc_aggregate(WT_DSRC_STATS **from, WT_DSRC_STATS *to) to->cache_eviction_walk_from_root += WT_STAT_READ(from, cache_eviction_walk_from_root); to->cache_eviction_walk_saved_pos += WT_STAT_READ(from, cache_eviction_walk_saved_pos); to->cache_eviction_hazard += WT_STAT_READ(from, cache_eviction_hazard); + to->cache_hs_read += WT_STAT_READ(from, cache_hs_read); to->cache_inmem_splittable += WT_STAT_READ(from, cache_inmem_splittable); to->cache_inmem_split += WT_STAT_READ(from, cache_inmem_split); to->cache_eviction_internal += WT_STAT_READ(from, cache_eviction_internal); @@ -540,11 +540,10 @@ __wt_stat_dsrc_aggregate(WT_DSRC_STATS **from, WT_DSRC_STATS *to) to->cache_eviction_dirty += WT_STAT_READ(from, cache_eviction_dirty); to->cache_read_overflow += WT_STAT_READ(from, cache_read_overflow); to->cache_eviction_deepen += WT_STAT_READ(from, cache_eviction_deepen); - to->cache_write_lookaside += WT_STAT_READ(from, cache_write_lookaside); + to->cache_write_hs += WT_STAT_READ(from, cache_write_hs); to->cache_read += WT_STAT_READ(from, cache_read); to->cache_read_deleted += WT_STAT_READ(from, cache_read_deleted); to->cache_read_deleted_prepared += WT_STAT_READ(from, cache_read_deleted_prepared); - to->cache_read_lookaside += WT_STAT_READ(from, cache_read_lookaside); to->cache_pages_requested += WT_STAT_READ(from, cache_pages_requested); to->cache_eviction_pages_seen += WT_STAT_READ(from, cache_eviction_pages_seen); to->cache_write += WT_STAT_READ(from, cache_write); @@ -643,16 +642,12 @@ static const char *const __stats_connection_desc[] = { "cache: application threads page write from cache to disk count", "cache: application threads page write from cache to disk time (usecs)", "cache: bytes belonging to page images in the cache", - "cache: bytes belonging to the cache overflow table in the cache", + "cache: bytes belonging to the history store table in the cache", "cache: bytes currently in the cache", "cache: bytes dirty in the cache cumulative", "cache: bytes not belonging to page images in the cache", "cache: bytes read into cache", - "cache: bytes written from cache", - "cache: cache overflow cursor application thread wait time (usecs)", - "cache: cache overflow cursor internal thread wait time (usecs)", "cache: cache overflow score", - "cache: cache overflow table entries", "cache: cache overflow table insert calls", - "cache: cache overflow table max on-disk size", "cache: cache overflow table on-disk size", - "cache: cache overflow table remove calls", "cache: checkpoint blocked page eviction", - "cache: eviction calls to get a page", "cache: eviction calls to get a page found queue empty", + "cache: bytes written from cache", "cache: cache overflow score", + "cache: checkpoint blocked page eviction", "cache: eviction calls to get a page", + "cache: eviction calls to get a page found queue empty", "cache: eviction calls to get a page found queue empty after locking", "cache: eviction currently operating in aggressive mode", "cache: eviction empty score", "cache: eviction passes of a file", @@ -690,6 +685,14 @@ static const char *const __stats_connection_desc[] = { "cache: forced eviction - pages selected unable to be evicted time", "cache: hazard pointer blocked page eviction", "cache: hazard pointer check calls", "cache: hazard pointer check entries walked", "cache: hazard pointer maximum array length", + "cache: history store key truncation due to mixed timestamps", + "cache: history store key truncation due to the key being removed from the data page", + "cache: history store score", "cache: history store table insert calls", + "cache: history store table max on-disk size", "cache: history store table on-disk size", + "cache: history store table reads", "cache: history store table reads missed", + "cache: history store table reads requiring squashed modifies", + "cache: history store table remove calls due to key truncation", + "cache: history store table writes requiring squashed modifies", "cache: in-memory page passed criteria to be split", "cache: in-memory page splits", "cache: internal pages evicted", "cache: internal pages queued for eviction", "cache: internal pages seen by eviction walk", @@ -699,17 +702,12 @@ static const char *const __stats_connection_desc[] = { "cache: modified pages evicted", "cache: modified pages evicted by application threads", "cache: operations timed out waiting for space in cache", "cache: overflow pages read into cache", "cache: page split during eviction deepened the tree", - "cache: page written requiring cache overflow records", - "cache: pages currently held in the cache", "cache: pages evicted by application threads", - "cache: pages queued for eviction", "cache: pages queued for eviction post lru sorting", - "cache: pages queued for urgent eviction", "cache: pages queued for urgent eviction during walk", - "cache: pages read into cache", "cache: pages read into cache after truncate", + "cache: page written requiring history store records", "cache: pages currently held in the cache", + "cache: pages evicted by application threads", "cache: pages queued for eviction", + "cache: pages queued for eviction post lru sorting", "cache: pages queued for urgent eviction", + "cache: pages queued for urgent eviction during walk", "cache: pages read into cache", + "cache: pages read into cache after truncate", "cache: pages read into cache after truncate in prepare state", - "cache: pages read into cache requiring cache overflow entries", - "cache: pages read into cache requiring cache overflow for checkpoint", - "cache: pages read into cache skipping older cache overflow entries", - "cache: pages read into cache with skipped cache overflow entries needed later", - "cache: pages read into cache with skipped cache overflow entries needed later by checkpoint", "cache: pages requested from the cache", "cache: pages seen by eviction walk", "cache: pages seen by eviction walk that are already queued", "cache: pages selected for eviction unable to be evicted", @@ -759,7 +757,9 @@ static const char *const __stats_connection_desc[] = { "data-handle: connection sweep dhandles removed from hash list", "data-handle: connection sweep time-of-death sets", "data-handle: connection sweeps", "data-handle: session dhandles swept", "data-handle: session sweep attempts", - "lock: checkpoint lock acquisitions", + "history: history pages added for eviction during garbage collection", + "history: history pages removed for garbage collection", + "history: history pages visited for garbage collection", "lock: checkpoint lock acquisitions", "lock: checkpoint lock application thread wait time (usecs)", "lock: checkpoint lock internal thread wait time (usecs)", "lock: dhandle lock application thread time waiting (usecs)", @@ -858,14 +858,11 @@ static const char *const __stats_connection_desc[] = { "thread-yield: page acquire time sleeping (usecs)", "thread-yield: page delete rollback time sleeping for state change (usecs)", "thread-yield: page reconciliation yielded due to child modification", - "transaction: Number of prepared updates", - "transaction: Number of prepared updates added to cache overflow", - "transaction: durable timestamp queue entries walked", + "transaction: Number of prepared updates", "transaction: durable timestamp queue entries walked", "transaction: durable timestamp queue insert to empty", "transaction: durable timestamp queue inserts to head", "transaction: durable timestamp queue inserts total", - "transaction: durable timestamp queue length", "transaction: number of named snapshots created", - "transaction: number of named snapshots dropped", "transaction: prepared transactions", + "transaction: durable timestamp queue length", "transaction: prepared transactions", "transaction: prepared transactions committed", "transaction: prepared transactions currently active", "transaction: prepared transactions rolled back", "transaction: query timestamp calls", @@ -873,28 +870,35 @@ static const char *const __stats_connection_desc[] = { "transaction: read timestamp queue insert to empty", "transaction: read timestamp queue inserts to head", "transaction: read timestamp queue inserts total", "transaction: read timestamp queue length", - "transaction: rollback to stable calls", "transaction: rollback to stable updates aborted", - "transaction: rollback to stable updates removed from cache overflow", + "transaction: rollback to stable calls", "transaction: rollback to stable keys removed", + "transaction: rollback to stable keys restored", "transaction: rollback to stable pages visited", + "transaction: rollback to stable updates aborted", + "transaction: rollback to stable updates removed from history store", "transaction: set timestamp calls", "transaction: set timestamp durable calls", "transaction: set timestamp durable updates", "transaction: set timestamp oldest calls", "transaction: set timestamp oldest updates", "transaction: set timestamp stable calls", "transaction: set timestamp stable updates", "transaction: transaction begins", "transaction: transaction checkpoint currently running", "transaction: transaction checkpoint generation", + "transaction: transaction checkpoint history store file duration (usecs)", "transaction: transaction checkpoint max time (msecs)", "transaction: transaction checkpoint min time (msecs)", "transaction: transaction checkpoint most recent time (msecs)", + "transaction: transaction checkpoint prepare currently running", + "transaction: transaction checkpoint prepare max time (msecs)", + "transaction: transaction checkpoint prepare min time (msecs)", + "transaction: transaction checkpoint prepare most recent time (msecs)", + "transaction: transaction checkpoint prepare total time (msecs)", "transaction: transaction checkpoint scrub dirty target", "transaction: transaction checkpoint scrub time (msecs)", "transaction: transaction checkpoint total time (msecs)", "transaction: transaction checkpoints", "transaction: transaction checkpoints skipped because database was clean", - "transaction: transaction failures due to cache overflow", + "transaction: transaction failures due to history store", "transaction: transaction fsync calls for checkpoint after allocating the transaction ID", "transaction: transaction fsync duration for checkpoint after allocating the transaction ID " "(usecs)", "transaction: transaction range of IDs currently pinned", "transaction: transaction range of IDs currently pinned by a checkpoint", - "transaction: transaction range of IDs currently pinned by named snapshots", "transaction: transaction range of timestamps currently pinned", "transaction: transaction range of timestamps pinned by a checkpoint", "transaction: transaction range of timestamps pinned by the oldest active read timestamp", @@ -978,20 +982,13 @@ __wt_stat_connection_clear_single(WT_CONNECTION_STATS *stats) stats->cache_write_app_count = 0; stats->cache_write_app_time = 0; /* not clearing cache_bytes_image */ - /* not clearing cache_bytes_lookaside */ + /* not clearing cache_bytes_hs */ /* not clearing cache_bytes_inuse */ /* not clearing cache_bytes_dirty_total */ /* not clearing cache_bytes_other */ stats->cache_bytes_read = 0; stats->cache_bytes_write = 0; - stats->cache_lookaside_cursor_wait_application = 0; - stats->cache_lookaside_cursor_wait_internal = 0; /* not clearing cache_lookaside_score */ - /* not clearing cache_lookaside_entries */ - stats->cache_lookaside_insert = 0; - /* not clearing cache_lookaside_ondisk_max */ - /* not clearing cache_lookaside_ondisk */ - stats->cache_lookaside_remove = 0; stats->cache_eviction_checkpoint = 0; stats->cache_eviction_get_ref = 0; stats->cache_eviction_get_ref_empty = 0; @@ -1041,6 +1038,17 @@ __wt_stat_connection_clear_single(WT_CONNECTION_STATS *stats) stats->cache_hazard_checks = 0; stats->cache_hazard_walks = 0; stats->cache_hazard_max = 0; + stats->cache_hs_key_truncate_mix_ts = 0; + stats->cache_hs_key_truncate_onpage_removal = 0; + /* not clearing cache_hs_score */ + stats->cache_hs_insert = 0; + /* not clearing cache_hs_ondisk_max */ + /* not clearing cache_hs_ondisk */ + stats->cache_hs_read = 0; + stats->cache_hs_read_miss = 0; + stats->cache_hs_read_squash = 0; + stats->cache_hs_remove_key_truncate = 0; + stats->cache_hs_write_squash = 0; stats->cache_inmem_splittable = 0; stats->cache_inmem_split = 0; stats->cache_eviction_internal = 0; @@ -1056,7 +1064,7 @@ __wt_stat_connection_clear_single(WT_CONNECTION_STATS *stats) stats->cache_timed_out_ops = 0; stats->cache_read_overflow = 0; stats->cache_eviction_deepen = 0; - stats->cache_write_lookaside = 0; + stats->cache_write_hs = 0; /* not clearing cache_pages_inuse */ stats->cache_eviction_app = 0; stats->cache_eviction_pages_queued = 0; @@ -1066,11 +1074,6 @@ __wt_stat_connection_clear_single(WT_CONNECTION_STATS *stats) stats->cache_read = 0; stats->cache_read_deleted = 0; stats->cache_read_deleted_prepared = 0; - stats->cache_read_lookaside = 0; - stats->cache_read_lookaside_checkpoint = 0; - stats->cache_read_lookaside_skipped = 0; - stats->cache_read_lookaside_delay = 0; - stats->cache_read_lookaside_delay_checkpoint = 0; stats->cache_pages_requested = 0; stats->cache_eviction_pages_seen = 0; stats->cache_eviction_pages_already_queued = 0; @@ -1152,6 +1155,9 @@ __wt_stat_connection_clear_single(WT_CONNECTION_STATS *stats) stats->dh_sweeps = 0; stats->dh_session_handles = 0; stats->dh_session_sweeps = 0; + stats->hs_gc_pages_evict = 0; + stats->hs_gc_pages_removed = 0; + stats->hs_gc_pages_visited = 0; stats->lock_checkpoint_count = 0; stats->lock_checkpoint_wait_application = 0; stats->lock_checkpoint_wait_internal = 0; @@ -1297,14 +1303,11 @@ __wt_stat_connection_clear_single(WT_CONNECTION_STATS *stats) stats->page_del_rollback_blocked = 0; stats->child_modify_blocked_page = 0; stats->txn_prepared_updates_count = 0; - stats->txn_prepared_updates_lookaside_inserts = 0; stats->txn_durable_queue_walked = 0; stats->txn_durable_queue_empty = 0; stats->txn_durable_queue_head = 0; stats->txn_durable_queue_inserts = 0; stats->txn_durable_queue_len = 0; - stats->txn_snapshots_created = 0; - stats->txn_snapshots_dropped = 0; stats->txn_prepare = 0; stats->txn_prepare_commit = 0; stats->txn_prepare_active = 0; @@ -1315,9 +1318,12 @@ __wt_stat_connection_clear_single(WT_CONNECTION_STATS *stats) stats->txn_read_queue_head = 0; stats->txn_read_queue_inserts = 0; stats->txn_read_queue_len = 0; - stats->txn_rollback_to_stable = 0; - stats->txn_rollback_upd_aborted = 0; - stats->txn_rollback_las_removed = 0; + stats->txn_rts = 0; + stats->txn_rts_keys_removed = 0; + stats->txn_rts_keys_restored = 0; + stats->txn_rts_pages_visited = 0; + stats->txn_rts_upd_aborted = 0; + stats->txn_rts_hs_removed = 0; stats->txn_set_ts = 0; stats->txn_set_ts_durable = 0; stats->txn_set_ts_durable_upd = 0; @@ -1328,9 +1334,15 @@ __wt_stat_connection_clear_single(WT_CONNECTION_STATS *stats) stats->txn_begin = 0; /* not clearing txn_checkpoint_running */ /* not clearing txn_checkpoint_generation */ + stats->txn_hs_ckpt_duration = 0; /* not clearing txn_checkpoint_time_max */ /* not clearing txn_checkpoint_time_min */ /* not clearing txn_checkpoint_time_recent */ + /* not clearing txn_checkpoint_prep_running */ + /* not clearing txn_checkpoint_prep_max */ + /* not clearing txn_checkpoint_prep_min */ + /* not clearing txn_checkpoint_prep_recent */ + /* not clearing txn_checkpoint_prep_total */ /* not clearing txn_checkpoint_scrub_target */ /* not clearing txn_checkpoint_scrub_time */ /* not clearing txn_checkpoint_time_total */ @@ -1341,7 +1353,6 @@ __wt_stat_connection_clear_single(WT_CONNECTION_STATS *stats) /* not clearing txn_checkpoint_fsync_post_duration */ /* not clearing txn_pinned_range */ /* not clearing txn_pinned_checkpoint_range */ - /* not clearing txn_pinned_snapshot_range */ /* not clearing txn_pinned_timestamp */ /* not clearing txn_pinned_timestamp_checkpoint */ /* not clearing txn_pinned_timestamp_reader */ @@ -1403,22 +1414,13 @@ __wt_stat_connection_aggregate(WT_CONNECTION_STATS **from, WT_CONNECTION_STATS * to->cache_write_app_count += WT_STAT_READ(from, cache_write_app_count); to->cache_write_app_time += WT_STAT_READ(from, cache_write_app_time); to->cache_bytes_image += WT_STAT_READ(from, cache_bytes_image); - to->cache_bytes_lookaside += WT_STAT_READ(from, cache_bytes_lookaside); + to->cache_bytes_hs += WT_STAT_READ(from, cache_bytes_hs); to->cache_bytes_inuse += WT_STAT_READ(from, cache_bytes_inuse); to->cache_bytes_dirty_total += WT_STAT_READ(from, cache_bytes_dirty_total); to->cache_bytes_other += WT_STAT_READ(from, cache_bytes_other); to->cache_bytes_read += WT_STAT_READ(from, cache_bytes_read); to->cache_bytes_write += WT_STAT_READ(from, cache_bytes_write); - to->cache_lookaside_cursor_wait_application += - WT_STAT_READ(from, cache_lookaside_cursor_wait_application); - to->cache_lookaside_cursor_wait_internal += - WT_STAT_READ(from, cache_lookaside_cursor_wait_internal); to->cache_lookaside_score += WT_STAT_READ(from, cache_lookaside_score); - to->cache_lookaside_entries += WT_STAT_READ(from, cache_lookaside_entries); - to->cache_lookaside_insert += WT_STAT_READ(from, cache_lookaside_insert); - to->cache_lookaside_ondisk_max += WT_STAT_READ(from, cache_lookaside_ondisk_max); - to->cache_lookaside_ondisk += WT_STAT_READ(from, cache_lookaside_ondisk); - to->cache_lookaside_remove += WT_STAT_READ(from, cache_lookaside_remove); to->cache_eviction_checkpoint += WT_STAT_READ(from, cache_eviction_checkpoint); to->cache_eviction_get_ref += WT_STAT_READ(from, cache_eviction_get_ref); to->cache_eviction_get_ref_empty += WT_STAT_READ(from, cache_eviction_get_ref_empty); @@ -1475,6 +1477,18 @@ __wt_stat_connection_aggregate(WT_CONNECTION_STATS **from, WT_CONNECTION_STATS * to->cache_hazard_walks += WT_STAT_READ(from, cache_hazard_walks); if ((v = WT_STAT_READ(from, cache_hazard_max)) > to->cache_hazard_max) to->cache_hazard_max = v; + to->cache_hs_key_truncate_mix_ts += WT_STAT_READ(from, cache_hs_key_truncate_mix_ts); + to->cache_hs_key_truncate_onpage_removal += + WT_STAT_READ(from, cache_hs_key_truncate_onpage_removal); + to->cache_hs_score += WT_STAT_READ(from, cache_hs_score); + to->cache_hs_insert += WT_STAT_READ(from, cache_hs_insert); + to->cache_hs_ondisk_max += WT_STAT_READ(from, cache_hs_ondisk_max); + to->cache_hs_ondisk += WT_STAT_READ(from, cache_hs_ondisk); + to->cache_hs_read += WT_STAT_READ(from, cache_hs_read); + to->cache_hs_read_miss += WT_STAT_READ(from, cache_hs_read_miss); + to->cache_hs_read_squash += WT_STAT_READ(from, cache_hs_read_squash); + to->cache_hs_remove_key_truncate += WT_STAT_READ(from, cache_hs_remove_key_truncate); + to->cache_hs_write_squash += WT_STAT_READ(from, cache_hs_write_squash); to->cache_inmem_splittable += WT_STAT_READ(from, cache_inmem_splittable); to->cache_inmem_split += WT_STAT_READ(from, cache_inmem_split); to->cache_eviction_internal += WT_STAT_READ(from, cache_eviction_internal); @@ -1493,7 +1507,7 @@ __wt_stat_connection_aggregate(WT_CONNECTION_STATS **from, WT_CONNECTION_STATS * to->cache_timed_out_ops += WT_STAT_READ(from, cache_timed_out_ops); to->cache_read_overflow += WT_STAT_READ(from, cache_read_overflow); to->cache_eviction_deepen += WT_STAT_READ(from, cache_eviction_deepen); - to->cache_write_lookaside += WT_STAT_READ(from, cache_write_lookaside); + to->cache_write_hs += WT_STAT_READ(from, cache_write_hs); to->cache_pages_inuse += WT_STAT_READ(from, cache_pages_inuse); to->cache_eviction_app += WT_STAT_READ(from, cache_eviction_app); to->cache_eviction_pages_queued += WT_STAT_READ(from, cache_eviction_pages_queued); @@ -1506,12 +1520,6 @@ __wt_stat_connection_aggregate(WT_CONNECTION_STATS **from, WT_CONNECTION_STATS * to->cache_read += WT_STAT_READ(from, cache_read); to->cache_read_deleted += WT_STAT_READ(from, cache_read_deleted); to->cache_read_deleted_prepared += WT_STAT_READ(from, cache_read_deleted_prepared); - to->cache_read_lookaside += WT_STAT_READ(from, cache_read_lookaside); - to->cache_read_lookaside_checkpoint += WT_STAT_READ(from, cache_read_lookaside_checkpoint); - to->cache_read_lookaside_skipped += WT_STAT_READ(from, cache_read_lookaside_skipped); - to->cache_read_lookaside_delay += WT_STAT_READ(from, cache_read_lookaside_delay); - to->cache_read_lookaside_delay_checkpoint += - WT_STAT_READ(from, cache_read_lookaside_delay_checkpoint); to->cache_pages_requested += WT_STAT_READ(from, cache_pages_requested); to->cache_eviction_pages_seen += WT_STAT_READ(from, cache_eviction_pages_seen); to->cache_eviction_pages_already_queued += @@ -1598,6 +1606,9 @@ __wt_stat_connection_aggregate(WT_CONNECTION_STATS **from, WT_CONNECTION_STATS * to->dh_sweeps += WT_STAT_READ(from, dh_sweeps); to->dh_session_handles += WT_STAT_READ(from, dh_session_handles); to->dh_session_sweeps += WT_STAT_READ(from, dh_session_sweeps); + to->hs_gc_pages_evict += WT_STAT_READ(from, hs_gc_pages_evict); + to->hs_gc_pages_removed += WT_STAT_READ(from, hs_gc_pages_removed); + to->hs_gc_pages_visited += WT_STAT_READ(from, hs_gc_pages_visited); to->lock_checkpoint_count += WT_STAT_READ(from, lock_checkpoint_count); to->lock_checkpoint_wait_application += WT_STAT_READ(from, lock_checkpoint_wait_application); to->lock_checkpoint_wait_internal += WT_STAT_READ(from, lock_checkpoint_wait_internal); @@ -1747,15 +1758,11 @@ __wt_stat_connection_aggregate(WT_CONNECTION_STATS **from, WT_CONNECTION_STATS * to->page_del_rollback_blocked += WT_STAT_READ(from, page_del_rollback_blocked); to->child_modify_blocked_page += WT_STAT_READ(from, child_modify_blocked_page); to->txn_prepared_updates_count += WT_STAT_READ(from, txn_prepared_updates_count); - to->txn_prepared_updates_lookaside_inserts += - WT_STAT_READ(from, txn_prepared_updates_lookaside_inserts); to->txn_durable_queue_walked += WT_STAT_READ(from, txn_durable_queue_walked); to->txn_durable_queue_empty += WT_STAT_READ(from, txn_durable_queue_empty); to->txn_durable_queue_head += WT_STAT_READ(from, txn_durable_queue_head); to->txn_durable_queue_inserts += WT_STAT_READ(from, txn_durable_queue_inserts); to->txn_durable_queue_len += WT_STAT_READ(from, txn_durable_queue_len); - to->txn_snapshots_created += WT_STAT_READ(from, txn_snapshots_created); - to->txn_snapshots_dropped += WT_STAT_READ(from, txn_snapshots_dropped); to->txn_prepare += WT_STAT_READ(from, txn_prepare); to->txn_prepare_commit += WT_STAT_READ(from, txn_prepare_commit); to->txn_prepare_active += WT_STAT_READ(from, txn_prepare_active); @@ -1766,9 +1773,12 @@ __wt_stat_connection_aggregate(WT_CONNECTION_STATS **from, WT_CONNECTION_STATS * to->txn_read_queue_head += WT_STAT_READ(from, txn_read_queue_head); to->txn_read_queue_inserts += WT_STAT_READ(from, txn_read_queue_inserts); to->txn_read_queue_len += WT_STAT_READ(from, txn_read_queue_len); - to->txn_rollback_to_stable += WT_STAT_READ(from, txn_rollback_to_stable); - to->txn_rollback_upd_aborted += WT_STAT_READ(from, txn_rollback_upd_aborted); - to->txn_rollback_las_removed += WT_STAT_READ(from, txn_rollback_las_removed); + to->txn_rts += WT_STAT_READ(from, txn_rts); + to->txn_rts_keys_removed += WT_STAT_READ(from, txn_rts_keys_removed); + to->txn_rts_keys_restored += WT_STAT_READ(from, txn_rts_keys_restored); + to->txn_rts_pages_visited += WT_STAT_READ(from, txn_rts_pages_visited); + to->txn_rts_upd_aborted += WT_STAT_READ(from, txn_rts_upd_aborted); + to->txn_rts_hs_removed += WT_STAT_READ(from, txn_rts_hs_removed); to->txn_set_ts += WT_STAT_READ(from, txn_set_ts); to->txn_set_ts_durable += WT_STAT_READ(from, txn_set_ts_durable); to->txn_set_ts_durable_upd += WT_STAT_READ(from, txn_set_ts_durable_upd); @@ -1779,9 +1789,15 @@ __wt_stat_connection_aggregate(WT_CONNECTION_STATS **from, WT_CONNECTION_STATS * to->txn_begin += WT_STAT_READ(from, txn_begin); to->txn_checkpoint_running += WT_STAT_READ(from, txn_checkpoint_running); to->txn_checkpoint_generation += WT_STAT_READ(from, txn_checkpoint_generation); + to->txn_hs_ckpt_duration += WT_STAT_READ(from, txn_hs_ckpt_duration); to->txn_checkpoint_time_max += WT_STAT_READ(from, txn_checkpoint_time_max); to->txn_checkpoint_time_min += WT_STAT_READ(from, txn_checkpoint_time_min); to->txn_checkpoint_time_recent += WT_STAT_READ(from, txn_checkpoint_time_recent); + to->txn_checkpoint_prep_running += WT_STAT_READ(from, txn_checkpoint_prep_running); + to->txn_checkpoint_prep_max += WT_STAT_READ(from, txn_checkpoint_prep_max); + to->txn_checkpoint_prep_min += WT_STAT_READ(from, txn_checkpoint_prep_min); + to->txn_checkpoint_prep_recent += WT_STAT_READ(from, txn_checkpoint_prep_recent); + to->txn_checkpoint_prep_total += WT_STAT_READ(from, txn_checkpoint_prep_total); to->txn_checkpoint_scrub_target += WT_STAT_READ(from, txn_checkpoint_scrub_target); to->txn_checkpoint_scrub_time += WT_STAT_READ(from, txn_checkpoint_scrub_time); to->txn_checkpoint_time_total += WT_STAT_READ(from, txn_checkpoint_time_total); @@ -1793,7 +1809,6 @@ __wt_stat_connection_aggregate(WT_CONNECTION_STATS **from, WT_CONNECTION_STATS * WT_STAT_READ(from, txn_checkpoint_fsync_post_duration); to->txn_pinned_range += WT_STAT_READ(from, txn_pinned_range); to->txn_pinned_checkpoint_range += WT_STAT_READ(from, txn_pinned_checkpoint_range); - to->txn_pinned_snapshot_range += WT_STAT_READ(from, txn_pinned_snapshot_range); to->txn_pinned_timestamp += WT_STAT_READ(from, txn_pinned_timestamp); to->txn_pinned_timestamp_checkpoint += WT_STAT_READ(from, txn_pinned_timestamp_checkpoint); to->txn_pinned_timestamp_reader += WT_STAT_READ(from, txn_pinned_timestamp_reader); diff --git a/src/third_party/wiredtiger/src/support/thread_group.c b/src/third_party/wiredtiger/src/support/thread_group.c index c01349016bf..b2da78afa8e 100644 --- a/src/third_party/wiredtiger/src/support/thread_group.c +++ b/src/third_party/wiredtiger/src/support/thread_group.c @@ -176,14 +176,14 @@ __thread_group_resize(WT_SESSION_IMPL *session, WT_THREAD_GROUP *group, uint32_t for (i = group->max; i < new_max; i++) { WT_ERR(__wt_calloc_one(session, &thread)); /* - * Threads get their own session and lookaside table cursor - * (if the lookaside table is open). + * Threads get their own session and hs table cursor + * (if the hs table is open). */ session_flags = LF_ISSET(WT_THREAD_CAN_WAIT) ? WT_SESSION_CAN_WAIT : 0; WT_ERR( __wt_open_internal_session(conn, group->name, false, session_flags, &thread->session)); - if (LF_ISSET(WT_THREAD_LOOKASIDE) && F_ISSET(conn, WT_CONN_LOOKASIDE_OPEN)) - WT_ERR(__wt_las_cursor_open(thread->session)); + if (LF_ISSET(WT_THREAD_HS) && F_ISSET(conn, WT_CONN_HS_OPEN)) + WT_ERR(__wt_hs_cursor_open(thread->session)); if (LF_ISSET(WT_THREAD_PANIC_FAIL)) F_SET(thread, WT_THREAD_PANIC_FAIL); thread->id = i; diff --git a/src/third_party/wiredtiger/src/txn/txn.c b/src/third_party/wiredtiger/src/txn/txn.c index 58f16081f8e..d180ed076b7 100644 --- a/src/third_party/wiredtiger/src/txn/txn.c +++ b/src/third_party/wiredtiger/src/txn/txn.c @@ -320,10 +320,6 @@ __txn_oldest_scan(WT_SESSION_IMPL *session, uint64_t *oldest_idp, uint64_t *last if (WT_TXNID_LT(last_running, oldest_id)) oldest_id = last_running; - /* The oldest ID can't move past any named snapshots. */ - if ((id = txn_global->nsnap_oldest_id) != WT_TXN_NONE && WT_TXNID_LT(id, oldest_id)) - oldest_id = id; - /* The metadata pinned ID can't move past the oldest ID. */ if (WT_TXNID_LT(oldest_id, metadata_pinned)) metadata_pinned = oldest_id; @@ -410,18 +406,6 @@ __wt_txn_update_oldest(WT_SESSION_IMPL *session, uint32_t flags) */ __txn_oldest_scan(session, &oldest_id, &last_running, &metadata_pinned, &oldest_session); -#ifdef HAVE_DIAGNOSTIC - { - /* - * Make sure the ID doesn't move past any named snapshots. - * - * Don't include the read/assignment in the assert statement. Coverity complains if there - * are assignments only done in diagnostic builds, and when the read is from a volatile. - */ - uint64_t id = txn_global->nsnap_oldest_id; - WT_ASSERT(session, id == WT_TXN_NONE || !WT_TXNID_LT(id, oldest_id)); - } -#endif /* Update the public IDs. */ if (WT_TXNID_LT(txn_global->metadata_pinned, metadata_pinned)) txn_global->metadata_pinned = metadata_pinned; @@ -492,15 +476,6 @@ __wt_txn_config(WT_SESSION_IMPL *session, const char *cfg[]) if (cval.val == 0) txn->txn_logsync = 0; - WT_RET(__wt_config_gets_def(session, cfg, "snapshot", 0, &cval)); - if (cval.len > 0) - /* - * The layering here isn't ideal - the named snapshot get function does both validation and - * setup. Otherwise we'd need to walk the list of named snapshots twice during transaction - * open. - */ - WT_RET(__wt_txn_named_snapshot_get(session, &cval)); - /* Check if prepared updates should be ignored during reads. */ WT_RET(__wt_config_gets_def(session, cfg, "ignore_prepare", 0, &cval)); if (cval.len > 0 && WT_STRING_MATCH("force", cval.str, cval.len)) @@ -1062,12 +1037,12 @@ __wt_txn_commit(WT_SESSION_IMPL *session, const char *cfg[]) } /* - * Writes to the lookaside file can be evicted as soon as they commit. + * Don't reset the timestamp of the history store records with history store + * transaction timestamp. Those records should already have the original time pair + * when they are inserted into the history store. */ - if (conn->cache->las_fileid != 0 && fileid == conn->cache->las_fileid) { - upd->txnid = WT_TXN_NONE; + if (conn->cache->hs_fileid != 0 && fileid == conn->cache->hs_fileid) break; - } __wt_txn_op_set_timestamp(session, op); } else { @@ -1089,6 +1064,9 @@ __wt_txn_commit(WT_SESSION_IMPL *session, const char *cfg[]) } __wt_txn_op_free(session, op); + /* If we used the cursor to resolve prepared updates, the key now has been freed. */ + if (cursor != NULL) + WT_CLEAR(cursor->key); } txn->mod_count = 0; @@ -1216,9 +1194,8 @@ __wt_txn_prepare(WT_SESSION_IMPL *session, const char *cfg[]) } for (i = 0, op = txn->mod; i < txn->mod_count; i++, op++) { - /* Assert it's not an update to the lookaside file. */ - WT_ASSERT( - session, S2C(session)->cache->las_fileid == 0 || !F_ISSET(op->btree, WT_BTREE_LOOKASIDE)); + /* Assert it's not an update to the history store file. */ + WT_ASSERT(session, S2C(session)->cache->hs_fileid == 0 || !WT_IS_HS(op->btree)); /* Metadata updates should never be prepared. */ WT_ASSERT(session, !WT_IS_METADATA(op->btree->dhandle)); @@ -1329,6 +1306,9 @@ __wt_txn_rollback(WT_SESSION_IMPL *session, const char *cfg[]) /* Rollback and free updates. */ for (i = 0, op = txn->mod; i < txn->mod_count; i++, op++) { + /* Assert it's not an update to the history store file. */ + WT_ASSERT(session, S2C(session)->cache->hs_fileid == 0 || !WT_IS_HS(op->btree)); + /* Metadata updates should never be rolled back. */ WT_ASSERT(session, !WT_IS_METADATA(op->btree->dhandle)); if (WT_IS_METADATA(op->btree->dhandle)) @@ -1344,6 +1324,9 @@ __wt_txn_rollback(WT_SESSION_IMPL *session, const char *cfg[]) upd = op->u.op_upd; if (!prepare) { + if (S2C(session)->cache->hs_fileid != 0 && + op->btree->id == S2C(session)->cache->hs_fileid) + break; WT_ASSERT(session, upd->txnid == txn->id || upd->txnid == WT_TXN_ABORTED); upd->txnid = WT_TXN_ABORTED; } else { @@ -1369,6 +1352,9 @@ __wt_txn_rollback(WT_SESSION_IMPL *session, const char *cfg[]) } __wt_txn_op_free(session, op); + /* If we used the cursor to resolve prepared updates, the key now has been freed. */ + if (cursor != NULL) + WT_CLEAR(cursor->key); } txn->mod_count = 0; @@ -1445,13 +1431,12 @@ __wt_txn_stats_update(WT_SESSION_IMPL *session) wt_timestamp_t durable_timestamp; wt_timestamp_t oldest_active_read_timestamp; wt_timestamp_t pinned_timestamp; - uint64_t checkpoint_pinned, snapshot_pinned; + uint64_t checkpoint_pinned; conn = S2C(session); txn_global = &conn->txn_global; stats = conn->stats; checkpoint_pinned = txn_global->checkpoint_state.pinned_id; - snapshot_pinned = txn_global->nsnap_oldest_id; WT_STAT_SET(session, stats, txn_pinned_range, txn_global->current - txn_global->oldest_id); @@ -1475,12 +1460,13 @@ __wt_txn_stats_update(WT_SESSION_IMPL *session) WT_STAT_SET(session, stats, txn_pinned_timestamp_reader, 0); } - WT_STAT_SET(session, stats, txn_pinned_snapshot_range, - snapshot_pinned == WT_TXN_NONE ? 0 : txn_global->current - snapshot_pinned); - WT_STAT_SET(session, stats, txn_pinned_checkpoint_range, checkpoint_pinned == WT_TXN_NONE ? 0 : txn_global->current - checkpoint_pinned); + WT_STAT_SET(session, stats, txn_checkpoint_prep_max, conn->ckpt_prep_max); + WT_STAT_SET(session, stats, txn_checkpoint_prep_min, conn->ckpt_prep_min); + WT_STAT_SET(session, stats, txn_checkpoint_prep_recent, conn->ckpt_prep_recent); + WT_STAT_SET(session, stats, txn_checkpoint_prep_total, conn->ckpt_prep_total); WT_STAT_SET(session, stats, txn_checkpoint_time_max, conn->ckpt_time_max); WT_STAT_SET(session, stats, txn_checkpoint_time_min, conn->ckpt_time_min); WT_STAT_SET(session, stats, txn_checkpoint_time_recent, conn->ckpt_time_recent); @@ -1546,10 +1532,6 @@ __wt_txn_global_init(WT_SESSION_IMPL *session, const char *cfg[]) WT_RWLOCK_INIT_TRACKED(session, &txn_global->read_timestamp_rwlock, read_timestamp); TAILQ_INIT(&txn_global->read_timestamph); - WT_RET(__wt_rwlock_init(session, &txn_global->nsnap_rwlock)); - txn_global->nsnap_oldest_id = WT_TXN_NONE; - TAILQ_INIT(&txn_global->nsnaph); - WT_RET(__wt_calloc_def(session, conn->session_size, &txn_global->states)); for (i = 0, s = txn_global->states; i < conn->session_size; i++, s++) @@ -1578,7 +1560,6 @@ __wt_txn_global_destroy(WT_SESSION_IMPL *session) __wt_rwlock_destroy(session, &txn_global->rwlock); __wt_rwlock_destroy(session, &txn_global->durable_timestamp_rwlock); __wt_rwlock_destroy(session, &txn_global->read_timestamp_rwlock); - __wt_rwlock_destroy(session, &txn_global->nsnap_rwlock); __wt_rwlock_destroy(session, &txn_global->visibility_rwlock); __wt_free(session, txn_global->states); } @@ -1639,6 +1620,13 @@ __wt_txn_global_shutdown(WT_SESSION_IMPL *session, const char *config, const cha F_SET(conn, WT_CONN_CLOSING_TIMESTAMP); } if (!F_ISSET(conn, WT_CONN_IN_MEMORY | WT_CONN_READONLY)) { + /* + * Perform rollback to stable to ensure that the stable version is written to disk on a + * clean shutdown. + */ + if (F_ISSET(conn, WT_CONN_CLOSING_TIMESTAMP)) + WT_TRET(__wt_rollback_to_stable(session, cfg, true)); + s = NULL; WT_TRET(__wt_open_internal_session(conn, "close_ckpt", true, 0, &s)); if (s != NULL) { @@ -1812,8 +1800,6 @@ __wt_verbose_dump_txn(WT_SESSION_IMPL *session) __wt_msg(session, "checkpoint pinned ID: %" PRIu64, txn_global->checkpoint_state.pinned_id)); WT_RET(__wt_msg(session, "checkpoint txn ID: %" PRIu64, txn_global->checkpoint_state.id)); - WT_RET(__wt_msg(session, "oldest named snapshot ID: %" PRIu64, txn_global->nsnap_oldest_id)); - WT_ORDERED_READ(session_cnt, conn->session_cnt); WT_RET(__wt_msg(session, "session count: %" PRIu32, session_cnt)); WT_RET(__wt_msg(session, "Transaction state of active sessions:")); @@ -1856,9 +1842,6 @@ __wt_verbose_dump_update(WT_SESSION_IMPL *session, WT_UPDATE *upd) case WT_UPDATE_INVALID: upd_type = "WT_UPDATE_INVALID"; break; - case WT_UPDATE_BIRTHMARK: - upd_type = "WT_UPDATE_BIRTHMARK"; - break; case WT_UPDATE_MODIFY: upd_type = "WT_UPDATE_MODIFY"; break; diff --git a/src/third_party/wiredtiger/src/txn/txn_ckpt.c b/src/third_party/wiredtiger/src/txn/txn_ckpt.c index 5f538f7fcda..00d68c614eb 100644 --- a/src/third_party/wiredtiger/src/txn/txn_ckpt.c +++ b/src/third_party/wiredtiger/src/txn/txn_ckpt.c @@ -22,7 +22,7 @@ static int __checkpoint_name_ok(WT_SESSION_IMPL *session, const char *name, size_t len) { /* Check for characters we don't want to see in a metadata file. */ - WT_RET(__wt_name_check(session, name, len)); + WT_RET(__wt_name_check(session, name, len, true)); /* * The internal checkpoint name is special, applications aren't allowed to use it. Be aggressive @@ -106,11 +106,11 @@ __checkpoint_update_generation(WT_SESSION_IMPL *session) } /* - * __checkpoint_apply_all -- - * Apply an operation to all files involved in a checkpoint. + * __checkpoint_apply_operation -- + * Apply a preliminary operation to all files involved in a checkpoint. */ static int -__checkpoint_apply_all( +__checkpoint_apply_operation( WT_SESSION_IMPL *session, const char *cfg[], int (*op)(WT_SESSION_IMPL *, const char *[])) { WT_CONFIG targetconf; @@ -182,11 +182,11 @@ err: } /* - * __checkpoint_apply -- + * __checkpoint_apply_to_dhandles -- * Apply an operation to all handles locked for a checkpoint. */ static int -__checkpoint_apply( +__checkpoint_apply_to_dhandles( WT_SESSION_IMPL *session, const char *cfg[], int (*op)(WT_SESSION_IMPL *, const char *[])) { WT_DECL_RET; @@ -261,8 +261,11 @@ __wt_checkpoint_get_handles(WT_SESSION_IMPL *session, const char *cfg[]) btree = S2BT(session); - /* Skip files that are never involved in a checkpoint. */ - if (F_ISSET(btree, WT_BTREE_NO_CHECKPOINT)) + /* + * Skip files that are never involved in a checkpoint. Skip the history store file as it is, + * checkpointed manually later. + */ + if (F_ISSET(btree, WT_BTREE_NO_CHECKPOINT) || WT_IS_HS(btree)) return (0); /* @@ -391,10 +394,10 @@ __checkpoint_reduce_dirty_cache(WT_SESSION_IMPL *session) break; /* - * Don't scrub when the lookaside table is in use: scrubbing is counter-productive in that - * case. + * Don't scrub when the history store table is in use: scrubbing is counter-productive in + * that case. */ - if (F_ISSET(cache, WT_CACHE_EVICT_LOOKASIDE)) + if (F_ISSET(cache, WT_CACHE_EVICT_HS)) break; /* @@ -452,10 +455,11 @@ __checkpoint_stats(WT_SESSION_IMPL *session) conn = S2C(session); - /* Output a verbose progress message for long running checkpoints */ + /* Output a verbose progress message for long running checkpoints. */ if (conn->ckpt_progress_msg_count > 0) __wt_checkpoint_progress(session, true); + /* Compute end-to-end timer statistics for checkpoint. */ __wt_epoch(session, &stop); msec = WT_TIMEDIFF_MS(stop, conn->ckpt_timer_scrub_end); @@ -465,6 +469,16 @@ __checkpoint_stats(WT_SESSION_IMPL *session) conn->ckpt_time_min = msec; conn->ckpt_time_recent = msec; conn->ckpt_time_total += msec; + + /* Compute timer statistics for the checkpoint prepare. */ + msec = WT_TIMEDIFF_MS(conn->ckpt_prep_end, conn->ckpt_prep_start); + + if (msec > conn->ckpt_prep_max) + conn->ckpt_prep_max = msec; + if (conn->ckpt_prep_min == 0 || msec < conn->ckpt_prep_min) + conn->ckpt_prep_min = msec; + conn->ckpt_prep_recent = msec; + conn->ckpt_prep_total += msec; } /* @@ -536,6 +550,8 @@ __checkpoint_prepare(WT_SESSION_IMPL *session, bool *trackingp, const char *cfg[ * Note: we don't go through the public API calls because they have side effects on cursors, * which applications can hold open across calls to checkpoint. */ + WT_STAT_CONN_SET(session, txn_checkpoint_prep_running, 1); + __wt_epoch(session, &conn->ckpt_prep_start); WT_RET(__wt_txn_begin(session, txn_cfg)); WT_DIAGNOSTIC_YIELD; @@ -601,30 +617,22 @@ __checkpoint_prepare(WT_SESSION_IMPL *session, bool *trackingp, const char *cfg[ * because recovery doesn't set the recovery timestamp until its checkpoint is complete. */ if (txn_global->has_stable_timestamp) { - txn->read_timestamp = txn_global->stable_timestamp; - txn_global->checkpoint_timestamp = txn->read_timestamp; - F_SET(txn, WT_TXN_HAS_TS_READ); + txn_global->checkpoint_timestamp = txn_global->stable_timestamp; if (!F_ISSET(conn, WT_CONN_RECOVERING)) - txn_global->meta_ckpt_timestamp = txn->read_timestamp; + txn_global->meta_ckpt_timestamp = txn_global->checkpoint_timestamp; } else if (!F_ISSET(conn, WT_CONN_RECOVERING)) txn_global->meta_ckpt_timestamp = txn_global->recovery_timestamp; - } else if (!F_ISSET(conn, WT_CONN_RECOVERING)) - txn_global->meta_ckpt_timestamp = WT_TS_NONE; + } else { + if (!F_ISSET(conn, WT_CONN_RECOVERING)) + txn_global->meta_ckpt_timestamp = WT_TS_NONE; + txn->read_timestamp = WT_TS_NONE; + } __wt_writeunlock(session, &txn_global->rwlock); - if (F_ISSET(txn, WT_TXN_HAS_TS_READ)) { + if (use_timestamp) __wt_verbose_timestamp( - session, txn->read_timestamp, "Checkpoint requested at stable timestamp"); - - /* - * The snapshot we established when the transaction started may be too early to match the - * timestamp we just read. - * - * Get a new one. - */ - __wt_txn_get_snapshot(session); - } + session, txn_global->checkpoint_timestamp, "Checkpoint requested at stable timestamp"); /* * Get a list of handles we want to flush; for named checkpoints this may pull closed objects @@ -635,7 +643,10 @@ __checkpoint_prepare(WT_SESSION_IMPL *session, bool *trackingp, const char *cfg[ */ WT_ASSERT(session, session->ckpt_handle_next == 0); WT_WITH_TABLE_READ_LOCK( - session, ret = __checkpoint_apply_all(session, cfg, __wt_checkpoint_get_handles)); + session, ret = __checkpoint_apply_operation(session, cfg, __wt_checkpoint_get_handles)); + + __wt_epoch(session, &conn->ckpt_prep_end); + WT_STAT_CONN_SET(session, txn_checkpoint_prep_running, 0); return (ret); } @@ -692,11 +703,17 @@ __txn_checkpoint_can_skip( return (0); /* - * It isn't currently safe to skip timestamp checkpoints - see WT-4958. We should fix this so we - * can skip timestamp checkpoints if they don't have new content. + * If the checkpoint is using timestamps, and the stable timestamp hasn't been updated since the + * last checkpoint there is nothing more that could be written. Except when a non timestamped + * file has been modified, as such if the connection has been modified it is currently unsafe to + * skip checkpoints. */ - if (use_timestamp) + if (!conn->modified && use_timestamp && txn_global->has_stable_timestamp && + txn_global->last_ckpt_timestamp != WT_TS_NONE && + txn_global->last_ckpt_timestamp == txn_global->stable_timestamp) { + *can_skipp = true; return (0); + } /* * Skip checkpointing the database if nothing has been dirtied since the last checkpoint. That @@ -704,22 +721,13 @@ __txn_checkpoint_can_skip( * be. We might skip a checkpoint in that short instance, which is okay because by the next time * we get to checkpoint, the connection would have been marked dirty and hence the checkpoint * will not be skipped again. + * + * If we are using timestamps then we shouldn't skip as the stable timestamp must have moved, + * and as such we still need to run checkpoint to update the checkpoint timestamp and the + * metadata. */ - if (!conn->modified) { - *can_skipp = true; - return (0); - } - - /* - * If the checkpoint is using timestamps, and the stable timestamp hasn't been updated since the - * last checkpoint there is nothing more that could be written. - */ - if (use_timestamp && txn_global->has_stable_timestamp && - txn_global->last_ckpt_timestamp != WT_TS_NONE && - txn_global->last_ckpt_timestamp == txn_global->stable_timestamp) { + if (!use_timestamp && !conn->modified) *can_skipp = true; - return (0); - } return (0); } @@ -733,18 +741,21 @@ __txn_checkpoint(WT_SESSION_IMPL *session, const char *cfg[]) { WT_CACHE *cache; WT_CONNECTION_IMPL *conn; + WT_DATA_HANDLE *hs_dhandle; WT_DECL_RET; WT_TXN *txn; WT_TXN_GLOBAL *txn_global; WT_TXN_ISOLATION saved_isolation; wt_timestamp_t ckpt_tmp_ts; - uint64_t fsync_duration_usecs, generation, time_start, time_stop; + uint64_t fsync_duration_usecs, generation, time_start_fsync, time_stop_fsync; + uint64_t time_start_hs, time_stop_hs, hs_ckpt_duration_usecs; u_int i; bool can_skip, failed, full, idle, logging, tracking, use_timestamp; void *saved_meta_next; conn = S2C(session); cache = conn->cache; + hs_dhandle = NULL; txn = &session->txn; txn_global = &conn->txn_global; saved_isolation = session->isolation; @@ -760,7 +771,7 @@ __txn_checkpoint(WT_SESSION_IMPL *session, const char *cfg[]) /* * Do a pass over the configuration arguments and figure out what kind of checkpoint this is. */ - WT_RET(__checkpoint_apply_all(session, cfg, NULL)); + WT_RET(__checkpoint_apply_operation(session, cfg, NULL)); logging = FLD_ISSET(conn->log_flags, WT_CONN_LOG_ENABLED); @@ -830,6 +841,12 @@ __txn_checkpoint(WT_SESSION_IMPL *session, const char *cfg[]) WT_WITH_SCHEMA_LOCK(session, ret = __checkpoint_prepare(session, &tracking, cfg)); WT_ERR(ret); + /* + * Save the checkpoint timestamp in a temporary variable, when we release our snapshot it'll be + * reset to zero. + */ + WT_ORDERED_READ(ckpt_tmp_ts, txn_global->checkpoint_timestamp); + WT_ASSERT(session, txn->isolation == WT_ISO_SNAPSHOT); /* @@ -844,8 +861,29 @@ __txn_checkpoint(WT_SESSION_IMPL *session, const char *cfg[]) WT_ERR(__wt_txn_checkpoint_log(session, full, WT_TXN_LOG_CKPT_START, NULL)); __checkpoint_timing_stress(session); + WT_ERR(__checkpoint_apply_to_dhandles(session, cfg, __checkpoint_tree_helper)); + + /* + * Get a history store dhandle. If the history store file is opened for a special operation this + * will return EBUSY which we treat as an error. In scenarios where the history store is not + * part of the metadata file (performing recovery on backup folder where no checkpoint + * occurred), this will return ENOENT which we ignore and continue. + */ + WT_ERR_ERROR_OK(__wt_session_get_dhandle(session, WT_HS_URI, NULL, NULL, 0), ENOENT); + hs_dhandle = session->dhandle; - WT_ERR(__checkpoint_apply(session, cfg, __checkpoint_tree_helper)); + /* + * It is possible that we don't have a history store file in certain recovery scenarios. As such + * we could get a dhandle that is not opened. + */ + if (F_ISSET(hs_dhandle, WT_DHANDLE_OPEN)) { + time_start_hs = __wt_clock(session); + WT_WITH_DHANDLE(session, hs_dhandle, ret = __wt_checkpoint(session, cfg)); + WT_ERR(ret); + time_stop_hs = __wt_clock(session); + hs_ckpt_duration_usecs = WT_CLOCKDIFF_US(time_stop_hs, time_start_hs); + WT_STAT_CONN_SET(session, txn_hs_ckpt_duration, hs_ckpt_duration_usecs); + } /* * Clear the dhandle so the visibility check doesn't get confused about the snap min. Don't @@ -853,25 +891,16 @@ __txn_checkpoint(WT_SESSION_IMPL *session, const char *cfg[]) * checkpoint. */ session->dhandle = NULL; - - /* - * Record the timestamp from the transaction if we were successful. Store it in a temp variable - * now because it will be invalidated during commit but we don't want to set it until we know - * the checkpoint is successful. We have to set the system information before we release the - * snapshot. - */ - ckpt_tmp_ts = 0; + /* We have to set the system information before we release the snapshot. */ if (full) { WT_ERR(__wt_meta_sysinfo_set(session)); - ckpt_tmp_ts = txn->read_timestamp; } /* Release the snapshot so we aren't pinning updates in cache. */ __wt_txn_release_snapshot(session); /* Mark all trees as open for business (particularly eviction). */ - WT_ERR(__checkpoint_apply(session, cfg, __checkpoint_presync)); - __wt_evict_server_wake(session); + WT_ERR(__checkpoint_apply_to_dhandles(session, cfg, __checkpoint_presync)); __checkpoint_verbose_track(session, "committing transaction"); @@ -879,10 +908,16 @@ __txn_checkpoint(WT_SESSION_IMPL *session, const char *cfg[]) * Checkpoints have to hit disk (it would be reasonable to configure for lazy checkpoints, but * we don't support them yet). */ - time_start = __wt_clock(session); - WT_ERR(__checkpoint_apply(session, cfg, __wt_checkpoint_sync)); - time_stop = __wt_clock(session); - fsync_duration_usecs = WT_CLOCKDIFF_US(time_stop, time_start); + time_start_fsync = __wt_clock(session); + + WT_ERR(__checkpoint_apply_to_dhandles(session, cfg, __wt_checkpoint_sync)); + + /* Sync the history store file. */ + if (F_ISSET(hs_dhandle, WT_DHANDLE_OPEN)) + WT_WITH_DHANDLE(session, hs_dhandle, ret = __wt_checkpoint_sync(session, NULL)); + + time_stop_fsync = __wt_clock(session); + fsync_duration_usecs = WT_CLOCKDIFF_US(time_stop_fsync, time_start_fsync); WT_STAT_CONN_INCR(session, txn_checkpoint_fsync_post); WT_STAT_CONN_SET(session, txn_checkpoint_fsync_post_duration, fsync_duration_usecs); @@ -939,7 +974,7 @@ __txn_checkpoint(WT_SESSION_IMPL *session, const char *cfg[]) * timestamp is WT_TS_NONE, set it to 1 so we can tell the difference. */ if (use_timestamp) { - conn->txn_global.last_ckpt_timestamp = use_timestamp ? ckpt_tmp_ts : WT_TS_NONE; + conn->txn_global.last_ckpt_timestamp = ckpt_tmp_ts; /* * MongoDB assumes the checkpoint timestamp will be initialized with WT_TS_NONE. In such * cases it queries the recovery timestamp to determine the last stable recovery @@ -1075,14 +1110,15 @@ __wt_txn_checkpoint(WT_SESSION_IMPL *session, const char *cfg[], bool waiting) * Application checkpoints wait until the checkpoint lock is available, compaction checkpoints * don't. * - * Checkpoints should always use a separate session for lookaside updates, otherwise those updates - * are pinned until the checkpoint commits. Also, there are unfortunate interactions between the - * special rules for lookaside eviction and the special handling of the checkpoint transaction. + * Checkpoints should always use a separate session for history store updates, otherwise those + * updates are pinned until the checkpoint commits. Also, there are unfortunate interactions between + * the special rules for history store eviction and the special handling of the checkpoint + * transaction. */ #undef WT_CHECKPOINT_SESSION_FLAGS #define WT_CHECKPOINT_SESSION_FLAGS (WT_SESSION_CAN_WAIT | WT_SESSION_IGNORE_CACHE_SIZE) #undef WT_CHECKPOINT_SESSION_FLAGS_OFF -#define WT_CHECKPOINT_SESSION_FLAGS_OFF (WT_SESSION_LOOKASIDE_CURSOR) +#define WT_CHECKPOINT_SESSION_FLAGS_OFF (WT_SESSION_HS_CURSOR) orig_flags = F_MASK(session, WT_CHECKPOINT_SESSION_FLAGS | WT_CHECKPOINT_SESSION_FLAGS_OFF); F_SET(session, WT_CHECKPOINT_SESSION_FLAGS); F_CLR(session, WT_CHECKPOINT_SESSION_FLAGS_OFF); diff --git a/src/third_party/wiredtiger/src/txn/txn_nsnap.c b/src/third_party/wiredtiger/src/txn/txn_nsnap.c deleted file mode 100644 index 5ac6e3b62b4..00000000000 --- a/src/third_party/wiredtiger/src/txn/txn_nsnap.c +++ /dev/null @@ -1,406 +0,0 @@ -/*- - * Copyright (c) 2014-2020 MongoDB, Inc. - * Copyright (c) 2008-2014 WiredTiger, Inc. - * All rights reserved. - * - * See the file LICENSE for redistribution information. - */ - -#include "wt_internal.h" - -/* - * __nsnap_destroy -- - * Destroy a named snapshot structure. - */ -static void -__nsnap_destroy(WT_SESSION_IMPL *session, WT_NAMED_SNAPSHOT *nsnap) -{ - __wt_free(session, nsnap->name); - __wt_free(session, nsnap->snapshot); - __wt_free(session, nsnap); -} - -/* - * __nsnap_drop_one -- - * Drop a single named snapshot. The named snapshot lock must be held write locked. - */ -static int -__nsnap_drop_one(WT_SESSION_IMPL *session, WT_CONFIG_ITEM *name) -{ - WT_NAMED_SNAPSHOT *found; - WT_TXN_GLOBAL *txn_global; - - txn_global = &S2C(session)->txn_global; - - TAILQ_FOREACH (found, &txn_global->nsnaph, q) - if (WT_STRING_MATCH(found->name, name->str, name->len)) - break; - - if (found == NULL) - return (WT_NOTFOUND); - - /* Bump the global ID if we are removing the first entry */ - if (found == TAILQ_FIRST(&txn_global->nsnaph)) { - WT_ASSERT(session, !__wt_txn_visible_all(session, txn_global->nsnap_oldest_id, WT_TS_NONE)); - txn_global->nsnap_oldest_id = - (TAILQ_NEXT(found, q) != NULL) ? TAILQ_NEXT(found, q)->pinned_id : WT_TXN_NONE; - WT_DIAGNOSTIC_YIELD; - WT_ASSERT(session, txn_global->nsnap_oldest_id == WT_TXN_NONE || - !__wt_txn_visible_all(session, txn_global->nsnap_oldest_id, WT_TS_NONE)); - } - TAILQ_REMOVE(&txn_global->nsnaph, found, q); - __nsnap_destroy(session, found); - WT_STAT_CONN_INCR(session, txn_snapshots_dropped); - - return (0); -} - -/* - * __nsnap_drop_to -- - * Drop named snapshots, if the name is NULL all snapshots will be dropped. The named snapshot - * lock must be held write locked. - */ -static int -__nsnap_drop_to(WT_SESSION_IMPL *session, WT_CONFIG_ITEM *name, bool inclusive) -{ - WT_NAMED_SNAPSHOT *last, *nsnap, *prev; - WT_TXN_GLOBAL *txn_global; - uint64_t new_nsnap_oldest; - - last = nsnap = prev = NULL; - txn_global = &S2C(session)->txn_global; - - if (TAILQ_EMPTY(&txn_global->nsnaph)) { - if (name == NULL) - return (0); - /* - * Dropping specific snapshots when there aren't any it's an error. - */ - WT_RET_MSG( - session, EINVAL, "Named snapshot '%.*s' for drop not found", (int)name->len, name->str); - } - - /* - * The new ID will be none if we are removing all named snapshots which is the default behavior - * of this loop. - */ - new_nsnap_oldest = WT_TXN_NONE; - if (name != NULL) { - TAILQ_FOREACH (last, &txn_global->nsnaph, q) { - if (WT_STRING_MATCH(last->name, name->str, name->len)) - break; - prev = last; - } - if (last == NULL) - WT_RET_MSG(session, EINVAL, "Named snapshot '%.*s' for drop not found", (int)name->len, - name->str); - - if (!inclusive) { - /* We are done if a drop before points to the head */ - if (prev == 0) - return (0); - last = prev; - } - - if (TAILQ_NEXT(last, q) != NULL) - new_nsnap_oldest = TAILQ_NEXT(last, q)->pinned_id; - } - - do { - nsnap = TAILQ_FIRST(&txn_global->nsnaph); - WT_ASSERT(session, nsnap != NULL); - TAILQ_REMOVE(&txn_global->nsnaph, nsnap, q); - __nsnap_destroy(session, nsnap); - WT_STAT_CONN_INCR(session, txn_snapshots_dropped); - /* Last will be NULL in the all case so it will never match */ - } while (nsnap != last && !TAILQ_EMPTY(&txn_global->nsnaph)); - - /* Now that the queue of named snapshots is updated, update the ID */ - WT_ASSERT(session, !__wt_txn_visible_all(session, txn_global->nsnap_oldest_id, WT_TS_NONE) && - (new_nsnap_oldest == WT_TXN_NONE || - WT_TXNID_LE(txn_global->nsnap_oldest_id, new_nsnap_oldest))); - txn_global->nsnap_oldest_id = new_nsnap_oldest; - WT_DIAGNOSTIC_YIELD; - WT_ASSERT(session, new_nsnap_oldest == WT_TXN_NONE || - !__wt_txn_visible_all(session, new_nsnap_oldest, WT_TS_NONE)); - - return (0); -} - -/* - * __wt_txn_named_snapshot_begin -- - * Begin an named in-memory snapshot. - */ -int -__wt_txn_named_snapshot_begin(WT_SESSION_IMPL *session, const char *cfg[]) -{ - WT_CONFIG_ITEM cval; - WT_DECL_RET; - WT_NAMED_SNAPSHOT *nsnap, *nsnap_new; - WT_TXN *txn; - WT_TXN_GLOBAL *txn_global; - const char *txn_cfg[] = { - WT_CONFIG_BASE(session, WT_SESSION_begin_transaction), "isolation=snapshot", NULL}; - bool include_updates, started_txn; - - started_txn = false; - nsnap_new = NULL; - txn_global = &S2C(session)->txn_global; - txn = &session->txn; - - WT_RET(__wt_config_gets_def(session, cfg, "include_updates", 0, &cval)); - include_updates = cval.val != 0; - - WT_RET(__wt_config_gets_def(session, cfg, "name", 0, &cval)); - WT_ASSERT(session, cval.len != 0); - - if (!F_ISSET(txn, WT_TXN_RUNNING)) { - if (include_updates) - WT_RET_MSG(session, EINVAL, - "A transaction must be " - "running to include updates in a named snapshot"); - - WT_RET(__wt_txn_begin(session, txn_cfg)); - started_txn = true; - } - if (!include_updates) - F_SET(txn, WT_TXN_READONLY); - - /* Save a copy of the transaction's snapshot. */ - WT_ERR(__wt_calloc_one(session, &nsnap_new)); - nsnap = nsnap_new; - WT_ERR(__wt_strndup(session, cval.str, cval.len, &nsnap->name)); - - /* - * To include updates from a writing transaction, make sure a transaction ID has been allocated. - */ - if (include_updates) { - WT_ERR(__wt_txn_id_check(session)); - WT_ASSERT(session, txn->id != WT_TXN_NONE); - nsnap->id = txn->id; - } else - nsnap->id = WT_TXN_NONE; - nsnap->pinned_id = WT_SESSION_TXN_STATE(session)->pinned_id; - nsnap->snap_min = txn->snap_min; - nsnap->snap_max = txn->snap_max; - if (txn->snapshot_count > 0) { - WT_ERR(__wt_calloc_def(session, txn->snapshot_count, &nsnap->snapshot)); - memcpy(nsnap->snapshot, txn->snapshot, txn->snapshot_count * sizeof(*nsnap->snapshot)); - } - nsnap->snapshot_count = txn->snapshot_count; - - /* Update the list. */ - - /* - * The semantic is that a new snapshot with the same name as an existing snapshot will replace - * the old one. - */ - WT_ERR_NOTFOUND_OK(__nsnap_drop_one(session, &cval)); - - if (TAILQ_EMPTY(&txn_global->nsnaph)) { - WT_ASSERT(session, txn_global->nsnap_oldest_id == WT_TXN_NONE && - !__wt_txn_visible_all(session, nsnap_new->pinned_id, WT_TS_NONE)); - __wt_readlock(session, &txn_global->rwlock); - txn_global->nsnap_oldest_id = nsnap_new->pinned_id; - __wt_readunlock(session, &txn_global->rwlock); - } - TAILQ_INSERT_TAIL(&txn_global->nsnaph, nsnap_new, q); - WT_STAT_CONN_INCR(session, txn_snapshots_created); - nsnap_new = NULL; - -err: - if (started_txn) { -#ifdef HAVE_DIAGNOSTIC - uint64_t pinned_id = WT_SESSION_TXN_STATE(session)->pinned_id; -#endif - WT_TRET(__wt_txn_rollback(session, NULL)); - WT_DIAGNOSTIC_YIELD; - WT_ASSERT(session, !__wt_txn_visible_all(session, pinned_id, WT_TS_NONE)); - } - - if (nsnap_new != NULL) - __nsnap_destroy(session, nsnap_new); - - return (ret); -} - -/* - * __wt_txn_named_snapshot_drop -- - * Drop named snapshots - */ -int -__wt_txn_named_snapshot_drop(WT_SESSION_IMPL *session, const char *cfg[]) -{ - WT_CONFIG objectconf; - WT_CONFIG_ITEM all_config, before_config, k, names_config, to_config, v; - WT_DECL_RET; - - WT_RET(__wt_config_gets_def(session, cfg, "drop.all", 0, &all_config)); - WT_RET(__wt_config_gets_def(session, cfg, "drop.names", 0, &names_config)); - WT_RET(__wt_config_gets_def(session, cfg, "drop.to", 0, &to_config)); - WT_RET(__wt_config_gets_def(session, cfg, "drop.before", 0, &before_config)); - - if (all_config.val != 0) - WT_RET(__nsnap_drop_to(session, NULL, true)); - else if (before_config.len != 0) - WT_RET(__nsnap_drop_to(session, &before_config, false)); - else if (to_config.len != 0) - WT_RET(__nsnap_drop_to(session, &to_config, true)); - - /* We are done if there are no named drops */ - - if (names_config.len != 0) { - __wt_config_subinit(session, &objectconf, &names_config); - while ((ret = __wt_config_next(&objectconf, &k, &v)) == 0) { - ret = __nsnap_drop_one(session, &k); - if (ret != 0) - WT_RET_MSG( - session, EINVAL, "Named snapshot '%.*s' for drop not found", (int)k.len, k.str); - } - if (ret == WT_NOTFOUND) - ret = 0; - } - - return (ret); -} - -/* - * __wt_txn_named_snapshot_get -- - * Lookup a named snapshot for a transaction. - */ -int -__wt_txn_named_snapshot_get(WT_SESSION_IMPL *session, WT_CONFIG_ITEM *nameval) -{ - WT_NAMED_SNAPSHOT *nsnap; - WT_TXN *txn; - WT_TXN_GLOBAL *txn_global; - WT_TXN_STATE *txn_state; - - txn = &session->txn; - txn_global = &S2C(session)->txn_global; - txn_state = WT_SESSION_TXN_STATE(session); - - txn->isolation = WT_ISO_SNAPSHOT; - if (session->ncursors > 0) - WT_RET(__wt_session_copy_values(session)); - - __wt_readlock(session, &txn_global->nsnap_rwlock); - TAILQ_FOREACH (nsnap, &txn_global->nsnaph, q) - if (WT_STRING_MATCH(nsnap->name, nameval->str, nameval->len)) { - /* - * Acquire the scan lock so the oldest ID can't move forward without seeing our pinned - * ID. - */ - __wt_readlock(session, &txn_global->rwlock); - txn_state->pinned_id = nsnap->pinned_id; - __wt_readunlock(session, &txn_global->rwlock); - - WT_ASSERT(session, !__wt_txn_visible_all(session, txn_state->pinned_id, WT_TS_NONE) && - txn_global->nsnap_oldest_id != WT_TXN_NONE && - WT_TXNID_LE(txn_global->nsnap_oldest_id, txn_state->pinned_id)); - txn->snap_min = nsnap->snap_min; - txn->snap_max = nsnap->snap_max; - if ((txn->snapshot_count = nsnap->snapshot_count) != 0) - memcpy( - txn->snapshot, nsnap->snapshot, nsnap->snapshot_count * sizeof(*nsnap->snapshot)); - if (nsnap->id != WT_TXN_NONE) { - WT_ASSERT(session, txn->id == WT_TXN_NONE); - txn->id = nsnap->id; - F_SET(txn, WT_TXN_READONLY); - } - F_SET(txn, WT_TXN_HAS_SNAPSHOT); - break; - } - __wt_readunlock(session, &txn_global->nsnap_rwlock); - - if (nsnap == NULL) - WT_RET_MSG( - session, EINVAL, "Named snapshot '%.*s' not found", (int)nameval->len, nameval->str); - - /* Flag that this transaction is opened on a named snapshot */ - F_SET(txn, WT_TXN_NAMED_SNAPSHOT); - - return (0); -} - -/* - * __wt_txn_named_snapshot_config -- - * Check the configuration for a named snapshot - */ -int -__wt_txn_named_snapshot_config( - WT_SESSION_IMPL *session, const char *cfg[], bool *has_create, bool *has_drops) -{ - WT_CONFIG_ITEM all_config, before_config, names_config, to_config; - WT_CONFIG_ITEM cval; - WT_TXN *txn; - - txn = &session->txn; - *has_create = *has_drops = false; - - /* Verify that the name is legal. */ - WT_RET(__wt_config_gets_def(session, cfg, "name", 0, &cval)); - if (cval.len != 0) { - if (WT_STRING_MATCH("all", cval.str, cval.len)) - WT_RET_MSG(session, EINVAL, "Can't create snapshot with reserved \"all\" name"); - - WT_RET(__wt_name_check(session, cval.str, cval.len)); - - if (F_ISSET(txn, WT_TXN_RUNNING) && txn->isolation != WT_ISO_SNAPSHOT) - WT_RET_MSG(session, EINVAL, - "Can't create a named snapshot from a running " - "transaction that isn't snapshot isolation"); - else if (F_ISSET(txn, WT_TXN_RUNNING) && txn->mod_count != 0) - WT_RET_MSG(session, EINVAL, - "Can't create a named snapshot from a running " - "transaction that has made updates"); - *has_create = true; - } - - /* Verify that the drop configuration is sane. */ - WT_RET(__wt_config_gets_def(session, cfg, "drop.all", 0, &all_config)); - WT_RET(__wt_config_gets_def(session, cfg, "drop.names", 0, &names_config)); - WT_RET(__wt_config_gets_def(session, cfg, "drop.to", 0, &to_config)); - WT_RET(__wt_config_gets_def(session, cfg, "drop.before", 0, &before_config)); - - /* Avoid more work if no drops are configured. */ - if (all_config.val != 0 || names_config.len != 0 || before_config.len != 0 || - to_config.len != 0) { - if (before_config.len != 0 && to_config.len != 0) - WT_RET_MSG(session, EINVAL, - "Illegal configuration; named snapshot drop can't " - "specify both before and to options"); - if (all_config.val != 0 && - (names_config.len != 0 || to_config.len != 0 || before_config.len != 0)) - WT_RET_MSG(session, EINVAL, - "Illegal configuration; named snapshot drop can't " - "specify all and any other options"); - *has_drops = true; - } - - if (!*has_create && !*has_drops) - WT_RET_MSG(session, EINVAL, - "WT_SESSION::snapshot API called without any drop or " - "name option"); - - return (0); -} - -/* - * __wt_txn_named_snapshot_destroy -- - * Destroy all named snapshots on connection close - */ -void -__wt_txn_named_snapshot_destroy(WT_SESSION_IMPL *session) -{ - WT_NAMED_SNAPSHOT *nsnap; - WT_TXN_GLOBAL *txn_global; - - txn_global = &S2C(session)->txn_global; - txn_global->nsnap_oldest_id = WT_TXN_NONE; - - while ((nsnap = TAILQ_FIRST(&txn_global->nsnaph)) != NULL) { - TAILQ_REMOVE(&txn_global->nsnaph, nsnap, q); - __nsnap_destroy(session, nsnap); - } -} diff --git a/src/third_party/wiredtiger/src/txn/txn_recover.c b/src/third_party/wiredtiger/src/txn/txn_recover.c index 598dbb3f9d7..e3ca72e3b43 100644 --- a/src/third_party/wiredtiger/src/txn/txn_recover.c +++ b/src/third_party/wiredtiger/src/txn/txn_recover.c @@ -402,7 +402,8 @@ err: /* * __recovery_setup_file -- - * Set up the recovery slot for a file. + * Set up the recovery slot for a file, track the largest file ID, and update the base write gen + * based on the file's configuration. */ static int __recovery_setup_file(WT_RECOVERY *r, const char *uri, const char *config) @@ -430,7 +431,7 @@ __recovery_setup_file(WT_RECOVERY *r, const char *uri, const char *config) uri, r->files[fileid].uri, fileid); WT_RET(__wt_strdup(r->session, uri, &r->files[fileid].uri)); WT_RET(__wt_config_getones(r->session, config, "checkpoint_lsn", &cval)); - /* If there is checkpoint logged for the file, apply everything. */ + /* If there is no checkpoint logged for the file, apply everything. */ if (cval.type != WT_CONFIG_ITEM_STRUCT) WT_INIT_LSN(&lsn); /* NOLINTNEXTLINE(cert-err34-c) */ @@ -449,7 +450,8 @@ __recovery_setup_file(WT_RECOVERY *r, const char *uri, const char *config) (WT_IS_MAX_LSN(&r->max_ckpt_lsn) || __wt_log_cmp(&lsn, &r->max_ckpt_lsn) > 0)) r->max_ckpt_lsn = lsn; - return (0); + /* Update the base write gen based on this file's configuration. */ + return (__wt_metadata_update_base_write_gen(r->session, config)); } /* @@ -521,13 +523,13 @@ __wt_txn_recover(WT_SESSION_IMPL *session) WT_RECOVERY r; WT_RECOVERY_FILE *metafile; char *config; - bool do_checkpoint, eviction_started, needs_rec, was_backup; + bool do_checkpoint, eviction_started, hs_exists, needs_rec, was_backup; conn = S2C(session); WT_CLEAR(r); WT_INIT_LSN(&r.ckpt_lsn); config = NULL; - do_checkpoint = true; + do_checkpoint = hs_exists = true; eviction_started = false; was_backup = F_ISSET(conn, WT_CONN_WAS_BACKUP); @@ -628,6 +630,17 @@ __wt_txn_recover(WT_SESSION_IMPL *session) WT_NOT_READ(metafile, NULL); /* + * While we have the metadata cursor open, we should check whether the history store file exists + * or not. If it does not, then we should not apply rollback to stable to each table. This might + * happen if we're upgrading from an older version. + */ + metac->set_key(metac, WT_HS_URI); + ret = metac->search(metac); + if (ret == WT_NOTFOUND) + hs_exists = false; + WT_ERR_NOTFOUND_OK(ret); + + /* * We no longer need the metadata cursor: close it to avoid pinning any resources that could * block eviction during recovery. */ @@ -661,7 +674,7 @@ __wt_txn_recover(WT_SESSION_IMPL *session) /* * Recovery can touch more data than fits in cache, so it relies on regular eviction to manage - * paging. Start eviction threads for recovery without LAS cursors. + * paging. Start eviction threads for recovery without history store cursors. */ WT_ERR(__wt_evict_create(session)); eviction_started = true; @@ -685,7 +698,61 @@ __wt_txn_recover(WT_SESSION_IMPL *session) done: WT_ERR(__recovery_set_checkpoint_timestamp(&r)); - if (do_checkpoint) + + /* + * Perform rollback to stable only when the following conditions met. + * 1. The connection is not read-only. A read-only connection expects that there shouldn't be + * any changes that need to be done on the database other than reading. + * 2. A valid recovery timestamp. The recovery timestamp is the stable timestamp retrieved + * from the metadata checkpoint information to indicate the stable timestamp when the + * checkpoint happened. Anything updates newer than this timestamp must rollback. + * 3. The history store file was found in the metadata. + */ + if (hs_exists && !F_ISSET(conn, WT_CONN_READONLY) && + conn->txn_global.recovery_timestamp != WT_TS_NONE) { + /* Start the eviction threads for rollback to stable if not already started. */ + if (!eviction_started) { + WT_ERR(__wt_evict_create(session)); + eviction_started = true; + } + + /* + * Currently, rollback to stable only needs to make changes to tables that use timestamps. + * That is because eviction does not run in parallel with a checkpoint, so content that is + * written never uses transaction IDs newer than the checkpoint's transaction ID and thus + * never needs to be rolled back. Once eviction is allowed while a checkpoint is active, it + * will be necessary to take the page write generation number into account during rollback + * to stable. For example, a page with write generation 10 and txnid 20 is written in one + * checkpoint, and in the next restart a new page with write generation 30 and txnid 20 is + * written. The rollback to stable operation should only rollback the latest page changes + * solely based on the write generation numbers. + */ + + WT_ASSERT(session, conn->txn_global.has_stable_timestamp == false && + conn->txn_global.stable_timestamp == WT_TS_NONE); + WT_ASSERT(session, conn->txn_global.has_oldest_timestamp == false && + conn->txn_global.oldest_timestamp == WT_TS_NONE); + + /* + * Set the stable timestamp from recovery timestamp and process the trees for rollback to + * stable. + */ + conn->txn_global.stable_timestamp = conn->txn_global.recovery_timestamp; + conn->txn_global.has_stable_timestamp = true; + + /* + * Set the oldest timestamp to WT_TS_NONE to make sure that we didn't remove any history + * window as part of rollback to stable operation. + */ + conn->txn_global.oldest_timestamp = WT_TS_NONE; + conn->txn_global.has_oldest_timestamp = true; + + WT_ERR(__wt_rollback_to_stable(session, NULL, false)); + + /* Reset the oldest timestamp. */ + conn->txn_global.oldest_timestamp = WT_TS_NONE; + conn->txn_global.has_oldest_timestamp = false; + } else if (do_checkpoint) /* * Forcibly log a checkpoint so the next open is fast and keep the metadata up to date with * the checkpoint LSN and archiving. @@ -712,7 +779,7 @@ err: /* * Destroy the eviction threads that were started in support of recovery. They will be restarted - * once the lookaside table is created. + * once the history store table is created. */ if (eviction_started) WT_TRET(__wt_evict_destroy(session)); diff --git a/src/third_party/wiredtiger/src/txn/txn_rollback_to_stable.c b/src/third_party/wiredtiger/src/txn/txn_rollback_to_stable.c index b0e0df6b724..11e72474b3a 100644 --- a/src/third_party/wiredtiger/src/txn/txn_rollback_to_stable.c +++ b/src/third_party/wiredtiger/src/txn/txn_rollback_to_stable.c @@ -9,87 +9,12 @@ #include "wt_internal.h" /* - * __txn_rollback_to_stable_lookaside_fixup -- - * Remove any updates that need to be rolled back from the lookaside file. - */ -static int -__txn_rollback_to_stable_lookaside_fixup(WT_SESSION_IMPL *session) -{ - WT_CONNECTION_IMPL *conn; - WT_CURSOR *cursor; - WT_DECL_RET; - WT_ITEM las_key, las_value; - WT_TXN_GLOBAL *txn_global; - wt_timestamp_t durable_timestamp, las_timestamp, rollback_timestamp; - uint64_t las_counter, las_pageid, las_total, las_txnid; - uint32_t las_id, session_flags; - uint8_t prepare_state, upd_type; - - conn = S2C(session); - cursor = NULL; - las_total = 0; - session_flags = 0; /* [-Werror=maybe-uninitialized] */ - - /* - * Copy the stable timestamp, otherwise we'd need to lock it each time it's accessed. Even - * though the stable timestamp isn't supposed to be updated while rolling back, accessing it - * without a lock would violate protocol. - */ - txn_global = &conn->txn_global; - WT_ORDERED_READ(rollback_timestamp, txn_global->stable_timestamp); - - __wt_las_cursor(session, &cursor, &session_flags); - - /* Discard pages we read as soon as we're done with them. */ - F_SET(session, WT_SESSION_READ_WONT_NEED); - - /* Walk the file. */ - __wt_writelock(session, &conn->cache->las_sweepwalk_lock); - while ((ret = cursor->next(cursor)) == 0) { - ++las_total; - WT_ERR(cursor->get_key(cursor, &las_pageid, &las_id, &las_counter, &las_key)); - - /* Check the file ID so we can skip durable tables */ - if (las_id >= conn->stable_rollback_maxfile) - WT_PANIC_RET(session, EINVAL, - "file ID %" PRIu32 " in lookaside table larger than max %" PRIu32, las_id, - conn->stable_rollback_maxfile); - if (__bit_test(conn->stable_rollback_bitstring, las_id)) - continue; - - WT_ERR(cursor->get_value(cursor, &las_txnid, &las_timestamp, &durable_timestamp, - &prepare_state, &upd_type, &las_value)); - - /* - * Entries with no timestamp will have a timestamp of zero, which will fail the following - * check and cause them to never be removed. - */ - if (rollback_timestamp < durable_timestamp) { - WT_ERR(cursor->remove(cursor)); - WT_STAT_CONN_INCR(session, txn_rollback_las_removed); - --las_total; - } - } - WT_ERR_NOTFOUND_OK(ret); -err: - if (ret == 0) { - conn->cache->las_insert_count = las_total; - conn->cache->las_remove_count = 0; - } - __wt_writeunlock(session, &conn->cache->las_sweepwalk_lock); - WT_TRET(__wt_las_cursor_close(session, &cursor, session_flags)); - - F_CLR(session, WT_SESSION_READ_WONT_NEED); - - return (ret); -} - -/* - * __txn_abort_newer_update -- - * Abort updates in an update change with timestamps newer than the rollback timestamp. + * __rollback_abort_newer_update -- + * Abort updates in an update change with timestamps newer than the rollback timestamp. Also, + * clear the history store flag for the first stable update in the update. */ static void -__txn_abort_newer_update( +__rollback_abort_newer_update( WT_SESSION_IMPL *session, WT_UPDATE *first_upd, wt_timestamp_t rollback_timestamp) { WT_UPDATE *upd; @@ -116,33 +41,321 @@ __txn_abort_newer_update( first_upd = upd->next; upd->txnid = WT_TXN_ABORTED; - WT_STAT_CONN_INCR(session, txn_rollback_upd_aborted); + WT_STAT_CONN_INCR(session, txn_rts_upd_aborted); upd->durable_ts = upd->start_ts = WT_TS_NONE; } } + + /* + * Clear the history store flag for the stable update to indicate that this update should not be + * written into the history store later, when all the aborted updates are removed from the + * history store. The next time when this update is moved into the history store, it will have a + * different stop time pair. + */ + if (first_upd != NULL) + F_CLR(first_upd, WT_UPDATE_HS); } /* - * __txn_abort_newer_insert -- - * Apply the update abort check to each entry in an insert skip list + * __rollback_abort_newer_insert -- + * Apply the update abort check to each entry in an insert skip list. */ static void -__txn_abort_newer_insert( +__rollback_abort_newer_insert( WT_SESSION_IMPL *session, WT_INSERT_HEAD *head, wt_timestamp_t rollback_timestamp) { WT_INSERT *ins; WT_SKIP_FOREACH (ins, head) - __txn_abort_newer_update(session, ins->upd, rollback_timestamp); + if (ins->upd != NULL) + __rollback_abort_newer_update(session, ins->upd, rollback_timestamp); +} + +/* + * __rollback_row_add_update -- + * Add the provided update to the head of the update list. + */ +static inline int +__rollback_row_add_update(WT_SESSION_IMPL *session, WT_PAGE *page, WT_ROW *rip, WT_UPDATE *upd) +{ + WT_DECL_RET; + WT_PAGE_MODIFY *mod; + WT_UPDATE *old_upd, **upd_entry; + size_t upd_size; + + /* If we don't yet have a modify structure, we'll need one. */ + WT_RET(__wt_page_modify_init(session, page)); + mod = page->modify; + + /* Allocate an update array as necessary. */ + WT_PAGE_ALLOC_AND_SWAP(session, page, mod->mod_row_update, upd_entry, page->entries); + + /* Set the WT_UPDATE array reference. */ + upd_entry = &mod->mod_row_update[WT_ROW_SLOT(page, rip)]; + upd_size = __wt_update_list_memsize(upd); + + /* + * If it's a full update list, we're trying to instantiate the row. Otherwise, it's just a + * single update that we'd like to append to the update list. + * + * Set the "old" entry to the second update in the list so that the serialization function + * succeeds in swapping the first update into place. + */ + if (upd->next != NULL) + *upd_entry = upd->next; + old_upd = *upd_entry; + + /* + * Point the new WT_UPDATE item to the next element in the list. The serialization function acts + * as our memory barrier to flush this write. + */ + upd->next = old_upd; + + /* + * Serialize the update. Rollback to stable doesn't need to check the visibility of the on page + * value to detect conflict. + */ + WT_ERR(__wt_update_serial(session, NULL, page, upd_entry, &upd, upd_size, true)); + +err: + return (ret); } /* - * __txn_abort_newer_col_var -- + * __rollback_row_ondisk_fixup_key -- + * Abort updates in the history store and replace the on-disk value with an update that + * satisfies the given timestamp. + */ +static int +__rollback_row_ondisk_fixup_key(WT_SESSION_IMPL *session, WT_PAGE *page, WT_ROW *rip, + wt_timestamp_t rollback_timestamp, bool replace) +{ + WT_CELL_UNPACK *unpack, _unpack; + WT_CURSOR *hs_cursor; + WT_CURSOR_BTREE *cbt; + WT_DECL_ITEM(hs_key); + WT_DECL_ITEM(hs_value); + WT_DECL_ITEM(key); + WT_DECL_RET; + WT_ITEM full_value; + WT_UPDATE *hs_upd, *upd; + wt_timestamp_t durable_ts, hs_start_ts, hs_stop_ts, newer_hs_ts; + size_t size; + uint64_t hs_counter, type_full; + uint32_t hs_btree_id, session_flags; + uint8_t type; + int cmp; + bool is_owner, valid_update_found; + + hs_cursor = NULL; + hs_upd = upd = NULL; + durable_ts = hs_start_ts = newer_hs_ts = WT_TS_NONE; + hs_btree_id = S2BT(session)->id; + session_flags = 0; + is_owner = valid_update_found = false; + + /* Allocate buffers for the data store and history store key. */ + WT_RET(__wt_scr_alloc(session, 0, &key)); + WT_ERR(__wt_scr_alloc(session, 0, &hs_key)); + WT_ERR(__wt_scr_alloc(session, 0, &hs_value)); + + WT_ERR(__wt_row_leaf_key(session, page, rip, key, false)); + + /* Get the full update value from the data store. */ + WT_CLEAR(full_value); + if (!__wt_row_leaf_value(page, rip, &full_value)) { + unpack = &_unpack; + __wt_row_leaf_value_cell(session, page, rip, NULL, unpack); + WT_ERR(__wt_page_cell_data_ref(session, page, unpack, &full_value)); + } + WT_ERR(__wt_buf_set(session, &full_value, full_value.data, full_value.size)); + + /* Open a history store table cursor. */ + WT_ERR(__wt_hs_cursor(session, &session_flags, &is_owner)); + hs_cursor = session->hs_cursor; + cbt = (WT_CURSOR_BTREE *)hs_cursor; + + /* + * Scan the history store for the given btree and key with maximum start and stop time pair to + * let the search point to the last version of the key and start traversing backwards to find + * out the satisfying record according the given timestamp. Any satisfying history store record + * is moved into data store and removed from history store. If none of the history store records + * satisfy the given timestamp, the key is removed from data store. + */ + ret = __wt_hs_cursor_position(session, hs_cursor, hs_btree_id, key, WT_TS_MAX); + for (; ret == 0; ret = hs_cursor->prev(hs_cursor)) { + WT_ERR(hs_cursor->get_key(hs_cursor, &hs_btree_id, hs_key, &hs_start_ts, &hs_counter)); + + /* Stop before crossing over to the next btree */ + if (hs_btree_id != S2BT(session)->id) + break; + + /* + * Keys are sorted in an order, skip the ones before the desired key, and bail out if we + * have crossed over the desired key and not found the record we are looking for. + */ + WT_ERR(__wt_compare(session, NULL, hs_key, key, &cmp)); + if (cmp != 0) + break; + + /* + * As part of the history store search, we never get an exact match based on our search + * criteria as we always search for a maximum record for that key. Make sure that we set the + * comparison result as an exact match to remove this key as part of rollback to stable. In + * case if we don't mark the comparison result as same, later the __wt_row_modify function + * will not properly remove the update from history store. + */ + cbt->compare = 0; + + /* Get current value and convert to full update if it is a modify. */ + WT_ERR(hs_cursor->get_value(hs_cursor, &hs_stop_ts, &durable_ts, &type_full, hs_value)); + type = (uint8_t)type_full; + if (type == WT_UPDATE_MODIFY) + WT_ERR(__wt_modify_apply_item(session, &full_value, hs_value->data, false)); + else { + WT_ASSERT(session, type == WT_UPDATE_STANDARD); + WT_ERR(__wt_buf_set(session, &full_value, hs_value->data, hs_value->size)); + } + + /* + * Verify the history store timestamps are in order. The start timestamp may be equal to the + * stop timestamp if the original update's commit timestamp is out of order. + */ + WT_ASSERT(session, + (newer_hs_ts == WT_TS_NONE || hs_stop_ts <= newer_hs_ts || hs_start_ts == hs_stop_ts)); + + /* + * Stop processing when we find the newer version value of this key is stable according to + * the current version stop timestamp. Also it confirms that history store doesn't contains + * any newer version than the current version for the key. + */ + if (hs_stop_ts <= rollback_timestamp) + break; + + /* Stop processing when we find a stable update according to the given timestamp. */ + if (durable_ts <= rollback_timestamp) { + valid_update_found = true; + break; + } + + newer_hs_ts = hs_start_ts; + WT_ERR(__wt_upd_alloc_tombstone(session, &hs_upd)); + WT_ERR(__wt_hs_modify(cbt, hs_upd)); + WT_STAT_CONN_INCR(session, txn_rts_hs_removed); + hs_upd = NULL; + } + + if (replace) { + /* + * If we found a history value that satisfied the given timestamp, add it to the update + * list. Otherwise remove the key by adding a tombstone. + */ + if (valid_update_found) { + WT_ERR(__wt_update_alloc(session, &full_value, &upd, &size, WT_UPDATE_STANDARD)); + + upd->txnid = WT_TXN_NONE; + upd->durable_ts = durable_ts; + upd->start_ts = hs_start_ts; + __wt_verbose(session, WT_VERB_RTS, "Update restored from history store (txnid: %" PRIu64 + ", start_ts: %" PRIu64 ", durable_ts: %" PRIu64 ")", + upd->txnid, upd->start_ts, upd->durable_ts); + + /* + * Set the flag to indicate that this update has been restored from history store for + * the rollback to stable operation. + */ + F_SET(upd, WT_UPDATE_RESTORED_FOR_ROLLBACK); + } else { + WT_ERR(__wt_upd_alloc_tombstone(session, &upd)); + WT_STAT_CONN_INCR(session, txn_rts_keys_removed); + __wt_verbose(session, WT_VERB_RTS, "%p: key removed", (void *)key); + } + + WT_ERR(__rollback_row_add_update(session, page, rip, upd)); + upd = NULL; + } + + /* Finally remove that update from history store. */ + if (valid_update_found) { + WT_ERR(__wt_upd_alloc_tombstone(session, &hs_upd)); + WT_ERR(__wt_hs_modify(cbt, hs_upd)); + WT_STAT_CONN_INCR(session, txn_rts_hs_removed); + hs_upd = NULL; + } + +err: + __wt_scr_free(session, &key); + __wt_scr_free(session, &hs_key); + __wt_scr_free(session, &hs_value); + __wt_buf_free(session, &full_value); + __wt_free(session, hs_upd); + __wt_free(session, upd); + WT_TRET(__wt_hs_cursor_close(session, session_flags, is_owner)); + + return (ret); +} + +/* + * __rollback_abort_row_ondisk_kv -- + * Fix the on-disk row K/V version according to the given timestamp. + */ +static int +__rollback_abort_row_ondisk_kv( + WT_SESSION_IMPL *session, WT_PAGE *page, WT_ROW *rip, wt_timestamp_t rollback_timestamp) +{ + WT_CELL_UNPACK *vpack, _vpack; + WT_DECL_RET; + WT_ITEM buf; + WT_UPDATE *upd; + size_t size; + + vpack = &_vpack; + upd = NULL; + __wt_row_leaf_value_cell(session, page, rip, NULL, vpack); + if (vpack->start_ts > rollback_timestamp) + return (__rollback_row_ondisk_fixup_key(session, page, rip, rollback_timestamp, true)); + else if (vpack->stop_ts != WT_TS_MAX && vpack->stop_ts > rollback_timestamp) { + /* + * Clear the remove operation from the key by inserting the original on-disk value as a + * standard update. + */ + WT_CLEAR(buf); + + /* + * If a value is simple(no compression), and is globally visible at the time of reading a + * page into cache, we encode its location into the WT_ROW. + */ + if (!__wt_row_leaf_value(page, rip, &buf)) + /* Take the value from the original page cell. */ + WT_RET(__wt_page_cell_data_ref(session, page, vpack, &buf)); + + WT_RET(__wt_update_alloc(session, &buf, &upd, &size, WT_UPDATE_STANDARD)); + upd->txnid = vpack->start_txn; + upd->durable_ts = vpack->start_ts; + upd->start_ts = vpack->start_ts; + WT_STAT_CONN_INCR(session, txn_rts_keys_restored); + __wt_verbose(session, WT_VERB_RTS, + "Key restored (txnid: %" PRIu64 ", start_ts: %" PRIu64 ", durable_ts: %" PRIu64 ")", + upd->txnid, upd->start_ts, upd->durable_ts); + } else + /* Stable version according to the timestamp. */ + return (0); + + WT_ERR(__rollback_row_add_update(session, page, rip, upd)); + return (0); + +err: + __wt_free(session, upd); + return (ret); +} + +/* + * __rollback_abort_newer_col_var -- * Abort updates on a variable length col leaf page with timestamps newer than the rollback * timestamp. */ static void -__txn_abort_newer_col_var( +__rollback_abort_newer_col_var( WT_SESSION_IMPL *session, WT_PAGE *page, wt_timestamp_t rollback_timestamp) { WT_COL *cip; @@ -152,39 +365,133 @@ __txn_abort_newer_col_var( /* Review the changes to the original on-page data items */ WT_COL_FOREACH (page, cip, i) if ((ins = WT_COL_UPDATE(page, cip)) != NULL) - __txn_abort_newer_insert(session, ins, rollback_timestamp); + __rollback_abort_newer_insert(session, ins, rollback_timestamp); /* Review the append list */ if ((ins = WT_COL_APPEND(page)) != NULL) - __txn_abort_newer_insert(session, ins, rollback_timestamp); + __rollback_abort_newer_insert(session, ins, rollback_timestamp); } /* - * __txn_abort_newer_col_fix -- + * __rollback_abort_newer_col_fix -- * Abort updates on a fixed length col leaf page with timestamps newer than the rollback * timestamp. */ static void -__txn_abort_newer_col_fix( +__rollback_abort_newer_col_fix( WT_SESSION_IMPL *session, WT_PAGE *page, wt_timestamp_t rollback_timestamp) { WT_INSERT_HEAD *ins; /* Review the changes to the original on-page data items */ if ((ins = WT_COL_UPDATE_SINGLE(page)) != NULL) - __txn_abort_newer_insert(session, ins, rollback_timestamp); + __rollback_abort_newer_insert(session, ins, rollback_timestamp); /* Review the append list */ if ((ins = WT_COL_APPEND(page)) != NULL) - __txn_abort_newer_insert(session, ins, rollback_timestamp); + __rollback_abort_newer_insert(session, ins, rollback_timestamp); +} + +/* + * __rollback_abort_row_reconciled_page_internal -- + * Abort updates on a history store using the in-memory build reconciled page of data store. + */ +static int +__rollback_abort_row_reconciled_page_internal(WT_SESSION_IMPL *session, const void *image, + const uint8_t *addr, size_t addr_size, wt_timestamp_t rollback_timestamp) +{ + WT_DECL_RET; + WT_ITEM tmp; + WT_PAGE *mod_page; + WT_ROW *rip; + uint32_t i, page_flags; + const void *image_local; + + /* + * Don't pass an allocated buffer to the underlying block read function, force allocation of new + * memory of the appropriate size. + */ + WT_CLEAR(tmp); + + mod_page = NULL; + image_local = image; + + if (image_local == NULL) { + WT_RET(__wt_bt_read(session, &tmp, addr, addr_size)); + image_local = tmp.data; + } + + page_flags = WT_DATA_IN_ITEM(&tmp) ? WT_PAGE_DISK_ALLOC : WT_PAGE_DISK_MAPPED; + WT_ERR(__wt_page_inmem(session, NULL, image_local, page_flags, &mod_page)); + tmp.mem = NULL; + WT_ROW_FOREACH (mod_page, rip, i) + WT_ERR_NOTFOUND_OK( + __rollback_row_ondisk_fixup_key(session, mod_page, rip, rollback_timestamp, false)); + +err: + if (mod_page != NULL) + __wt_page_out(session, &mod_page); + __wt_buf_free(session, &tmp); + + return (ret); +} + +/* + * __rollback_abort_row_reconciled_page -- + * Abort updates on a history store using the reconciled pages of data store. + */ +static int +__rollback_abort_row_reconciled_page( + WT_SESSION_IMPL *session, WT_PAGE *page, wt_timestamp_t rollback_timestamp) +{ + WT_MULTI *multi; + WT_PAGE_MODIFY *mod; + uint32_t multi_entry; + + if ((mod = page->modify) == NULL) + return (0); + + /* + * FIXME-prepare-support: audit the use of durable timestamps in this file, use both durable + * timestamps. + */ + if (mod->rec_result == WT_PM_REC_REPLACE && + mod->mod_replace.stop_durable_ts > rollback_timestamp) { + WT_RET(__rollback_abort_row_reconciled_page_internal(session, mod->u1.r.disk_image, + mod->u1.r.replace.addr, mod->u1.r.replace.size, rollback_timestamp)); + + /* + * As this page has newer aborts that are aborted, make sure to mark the page as dirty to + * let the reconciliation happens again on the page. Otherwise, the eviction may pick the + * already reconciled page to write to disk with newer updates. + */ + __wt_page_only_modify_set(session, page); + } else if (mod->rec_result == WT_PM_REC_MULTIBLOCK) { + for (multi = mod->mod_multi, multi_entry = 0; multi_entry < mod->mod_multi_entries; + ++multi, ++multi_entry) + if (multi->addr.stop_durable_ts > rollback_timestamp) { + WT_RET(__rollback_abort_row_reconciled_page_internal(session, multi->disk_image, + multi->addr.addr, multi->addr.size, rollback_timestamp)); + + /* + * As this page has newer aborts that are aborted, make sure to mark the page as + * dirty to let the reconciliation happens again on the page. Otherwise, the + * eviction may pick the already reconciled page to write to disk with newer + * updates. + */ + __wt_page_only_modify_set(session, page); + } + } + + return (0); } /* - * __txn_abort_newer_row_leaf -- + * __rollback_abort_newer_row_leaf -- * Abort updates on a row leaf page with timestamps newer than the rollback timestamp. */ -static void -__txn_abort_newer_row_leaf( +static int +__rollback_abort_newer_row_leaf( WT_SESSION_IMPL *session, WT_PAGE *page, wt_timestamp_t rollback_timestamp) { WT_INSERT_HEAD *insert; @@ -196,7 +503,7 @@ __txn_abort_newer_row_leaf( * Review the insert list for keys before the first entry on the disk page. */ if ((insert = WT_ROW_INSERT_SMALLEST(page)) != NULL) - __txn_abort_newer_insert(session, insert, rollback_timestamp); + __rollback_abort_newer_insert(session, insert, rollback_timestamp); /* * Review updates that belong to keys that are on the disk image, as well as for keys inserted @@ -204,82 +511,150 @@ __txn_abort_newer_row_leaf( */ WT_ROW_FOREACH (page, rip, i) { if ((upd = WT_ROW_UPDATE(page, rip)) != NULL) - __txn_abort_newer_update(session, upd, rollback_timestamp); + __rollback_abort_newer_update(session, upd, rollback_timestamp); if ((insert = WT_ROW_INSERT(page, rip)) != NULL) - __txn_abort_newer_insert(session, insert, rollback_timestamp); + __rollback_abort_newer_insert(session, insert, rollback_timestamp); + + /* If the configuration is not in-memory, abort any on-disk value. */ + if (!F_ISSET(S2C(session), WT_CONN_IN_MEMORY)) + WT_RET(__rollback_abort_row_ondisk_kv(session, page, rip, rollback_timestamp)); } + + /* + * If the configuration is not in-memory, abort history store updates from the reconciled pages + * of data store. + */ + if (!F_ISSET(S2C(session), WT_CONN_IN_MEMORY)) + WT_RET(__rollback_abort_row_reconciled_page(session, page, rollback_timestamp)); + return (0); } /* - * __txn_abort_newer_updates -- + * __rollback_page_needs_abort -- + * Check whether the page needs rollback. Return true if the page has modifications newer than + * the given timestamp Otherwise return false. + */ +static bool +__rollback_page_needs_abort( + WT_SESSION_IMPL *session, WT_REF *ref, wt_timestamp_t rollback_timestamp) +{ + WT_ADDR *addr; + WT_CELL_UNPACK vpack; + WT_MULTI *multi; + WT_PAGE_MODIFY *mod; + wt_timestamp_t multi_newest_durable_ts; + uint32_t i; + + addr = ref->addr; + mod = ref->page == NULL ? NULL : ref->page->modify; + + /* + * The rollback operation should be performed on this page when any one of the following is + * greater than the given timestamp: + * 1. The reconciled replace page max durable timestamp. + * 2. The reconciled multi page max durable timestamp. + * 3. The on page address max durable timestamp. + * 4. The off page address max durable timestamp. + */ + if (mod != NULL && mod->rec_result == WT_PM_REC_REPLACE) + return (mod->mod_replace.stop_durable_ts > rollback_timestamp); + else if (mod != NULL && mod->rec_result == WT_PM_REC_MULTIBLOCK) { + multi_newest_durable_ts = WT_TS_NONE; + /* Calculate the max durable timestamp by traversing all multi addresses. */ + for (multi = mod->mod_multi, i = 0; i < mod->mod_multi_entries; ++multi, ++i) + multi_newest_durable_ts = WT_MAX(multi_newest_durable_ts, multi->addr.stop_durable_ts); + return (multi_newest_durable_ts > rollback_timestamp); + } else if (!__wt_off_page(ref->home, addr)) { + /* Check if the page is obsolete using the page disk address. */ + __wt_cell_unpack(session, ref->home, (WT_CELL *)addr, &vpack); + return (vpack.newest_stop_durable_ts > rollback_timestamp); + } else if (addr != NULL) + return (addr->stop_durable_ts > rollback_timestamp); + + return (false); +} + +#ifdef HAVE_DIAGNOSTIC +/* + * __rollback_verify_ondisk_page -- + * Verify the on-disk page that it doesn't have updates newer than the timestamp. + */ +static void +__rollback_verify_ondisk_page( + WT_SESSION_IMPL *session, WT_PAGE *page, wt_timestamp_t rollback_timestamp) +{ + WT_CELL_UNPACK *vpack, _vpack; + WT_ROW *rip; + uint32_t i; + + vpack = &_vpack; + + /* Review updates that belong to keys that are on the disk image. */ + WT_ROW_FOREACH (page, rip, i) { + __wt_row_leaf_value_cell(session, page, rip, NULL, vpack); + WT_ASSERT(session, vpack->start_ts <= rollback_timestamp); + if (vpack->stop_ts != WT_TS_MAX) + WT_ASSERT(session, vpack->stop_ts <= rollback_timestamp); + } +} +#endif + +/* + * __rollback_abort_newer_updates -- * Abort updates on this page newer than the timestamp. */ static int -__txn_abort_newer_updates(WT_SESSION_IMPL *session, WT_REF *ref, wt_timestamp_t rollback_timestamp) +__rollback_abort_newer_updates( + WT_SESSION_IMPL *session, WT_REF *ref, wt_timestamp_t rollback_timestamp) { WT_DECL_RET; WT_PAGE *page; - WT_PAGE_LOOKASIDE *page_las; - uint32_t read_flags; bool local_read; - /* - * If we created a page image with updates that need to be rolled back, read the history into - * cache now and make sure the page is marked dirty. Otherwise, the history we need could be - * swept from the lookaside table before the page is read because the lookaside sweep code has - * no way to tell that the page image is invalid. - * - * So, if there is lookaside history for a page, first check if the history needs to be rolled - * back then ensure the history is loaded into cache. - * - * Also, we have separately discarded any lookaside history more recent than the rollback - * timestamp. For page_las structures in cache, reset any future timestamps back to the rollback - * timestamp. This allows those structures to be discarded once the rollback timestamp is stable - * (crucially for tests, they can be discarded if the connection is closed right after a - * rollback_to_stable call). - */ local_read = false; - read_flags = WT_READ_WONT_NEED; - if ((page_las = ref->page_las) != NULL) { - if (rollback_timestamp < page_las->max_ondisk_ts) { - /* - * Make sure we get back a page with history, not a limbo page. - */ - WT_ASSERT(session, !F_ISSET(&session->txn, WT_TXN_HAS_SNAPSHOT)); - WT_RET(__wt_page_in(session, ref, read_flags)); - WT_ASSERT(session, - ref->state != WT_REF_LIMBO && ref->page != NULL && __wt_page_is_modified(ref->page)); - local_read = true; - page_las->max_ondisk_ts = rollback_timestamp; - } - if (rollback_timestamp < page_las->min_skipped_ts) - page_las->min_skipped_ts = rollback_timestamp; - } - /* Review deleted page saved to the ref */ - if (ref->page_del != NULL && rollback_timestamp < ref->page_del->durable_timestamp) - WT_ERR(__wt_delete_page_rollback(session, ref)); + /* Review deleted page saved to the ref. */ + if (ref->page_del != NULL && rollback_timestamp < ref->page_del->durable_timestamp) { + __wt_verbose(session, WT_VERB_RTS, "%p: deleted page rolled back", (void *)ref); + WT_RET(__wt_delete_page_rollback(session, ref)); + } /* - * If we have a ref with no page, or the page is clean, there is nothing to roll back. - * - * This check for a clean page is partly an optimization (checkpoint only marks pages clean when - * they have no unwritten updates so there's no point visiting them again), but also covers a - * corner case of a checkpoint with use_timestamp=false. Such a checkpoint effectively moves the - * stable timestamp forward, because changes that are written in the checkpoint cannot be - * reliably rolled back. The actual stable timestamp doesn't change, though, so if we try to - * roll back clean pages the in-memory tree can get out of sync with the on-disk tree. + * If we have a ref with no page, or the page is clean, find out whether the page has any + * modifications that are newer than the given timestamp. As eviction writes the newest version + * to page, even a clean page may also contain modifications that need rollback. Such pages are + * read back into memory and processed like other modified pages. */ - if ((page = ref->page) == NULL || !__wt_page_is_modified(page)) - goto err; + if ((page = ref->page) == NULL || !__wt_page_is_modified(page)) { + if (!__rollback_page_needs_abort(session, ref, rollback_timestamp)) { + __wt_verbose(session, WT_VERB_RTS, "%p: page skipped", (void *)ref); +#ifdef HAVE_DIAGNOSTIC + if (ref->page == NULL && !F_ISSET(S2C(session), WT_CONN_IN_MEMORY)) { + WT_RET(__wt_page_in(session, ref, 0)); + __rollback_verify_ondisk_page(session, ref->page, rollback_timestamp); + WT_TRET(__wt_page_release(session, ref, 0)); + } +#endif + return (0); + } + + /* Page needs rollback, read it into cache. */ + if (page == NULL) { + WT_RET(__wt_page_in(session, ref, 0)); + local_read = true; + } + page = ref->page; + } + WT_STAT_CONN_INCR(session, txn_rts_pages_visited); + __wt_verbose(session, WT_VERB_RTS, "%p: page rolled back", (void *)ref); switch (page->type) { case WT_PAGE_COL_FIX: - __txn_abort_newer_col_fix(session, page, rollback_timestamp); + __rollback_abort_newer_col_fix(session, page, rollback_timestamp); break; case WT_PAGE_COL_VAR: - __txn_abort_newer_col_var(session, page, rollback_timestamp); + __rollback_abort_newer_col_var(session, page, rollback_timestamp); break; case WT_PAGE_COL_INT: case WT_PAGE_ROW_INT: @@ -290,7 +665,7 @@ __txn_abort_newer_updates(WT_SESSION_IMPL *session, WT_REF *ref, wt_timestamp_t */ break; case WT_PAGE_ROW_LEAF: - __txn_abort_newer_row_leaf(session, page, rollback_timestamp); + WT_ERR(__rollback_abort_newer_row_leaf(session, page, rollback_timestamp)); break; default: WT_ERR(__wt_illegal_value(session, page->type)); @@ -298,16 +673,16 @@ __txn_abort_newer_updates(WT_SESSION_IMPL *session, WT_REF *ref, wt_timestamp_t err: if (local_read) - WT_TRET(__wt_page_release(session, ref, read_flags)); + WT_TRET(__wt_page_release(session, ref, 0)); return (ret); } /* - * __txn_rollback_to_stable_btree_walk -- + * __rollback_to_stable_btree_walk -- * Called for each open handle - choose to either skip or wipe the commits */ static int -__txn_rollback_to_stable_btree_walk(WT_SESSION_IMPL *session, wt_timestamp_t rollback_timestamp) +__rollback_to_stable_btree_walk(WT_SESSION_IMPL *session, wt_timestamp_t rollback_timestamp) { WT_DECL_RET; WT_REF *child_ref, *ref; @@ -315,25 +690,24 @@ __txn_rollback_to_stable_btree_walk(WT_SESSION_IMPL *session, wt_timestamp_t rol /* Walk the tree, marking commits aborted where appropriate. */ ref = NULL; while ((ret = __wt_tree_walk( - session, &ref, WT_READ_CACHE | WT_READ_NO_EVICT | WT_READ_WONT_NEED)) == 0 && - ref != NULL) { - if (WT_PAGE_IS_INTERNAL(ref->page)) { + session, &ref, WT_READ_CACHE_LEAF | WT_READ_NO_EVICT | WT_READ_WONT_NEED)) == 0 && + ref != NULL) + if (F_ISSET(ref, WT_REF_FLAG_INTERNAL)) { WT_INTL_FOREACH_BEGIN (session, ref->page, child_ref) { - WT_RET(__txn_abort_newer_updates(session, child_ref, rollback_timestamp)); + WT_RET(__rollback_abort_newer_updates(session, child_ref, rollback_timestamp)); } WT_INTL_FOREACH_END; - } else - WT_RET(__txn_abort_newer_updates(session, ref, rollback_timestamp)); - } + } + return (ret); } /* - * __txn_rollback_eviction_drain -- + * __rollback_eviction_drain -- * Wait for eviction to drain from a tree. */ static int -__txn_rollback_eviction_drain(WT_SESSION_IMPL *session, const char *cfg[]) +__rollback_eviction_drain(WT_SESSION_IMPL *session, const char *cfg[]) { WT_UNUSED(cfg); @@ -343,23 +717,18 @@ __txn_rollback_eviction_drain(WT_SESSION_IMPL *session, const char *cfg[]) } /* - * __txn_rollback_to_stable_btree -- - * Called for each open handle - choose to either skip or wipe the commits + * __rollback_to_stable_btree -- + * Called for each object handle - choose to either skip or wipe the commits */ static int -__txn_rollback_to_stable_btree(WT_SESSION_IMPL *session, const char *cfg[]) +__rollback_to_stable_btree(WT_SESSION_IMPL *session, wt_timestamp_t rollback_timestamp) { WT_BTREE *btree; WT_CONNECTION_IMPL *conn; WT_DECL_RET; - WT_TXN_GLOBAL *txn_global; - wt_timestamp_t rollback_timestamp; - - WT_UNUSED(cfg); btree = S2BT(session); conn = S2C(session); - txn_global = &conn->txn_global; /* * Immediately durable files don't get their commits wiped. This case mostly exists to support @@ -369,14 +738,11 @@ __txn_rollback_to_stable_btree(WT_SESSION_IMPL *session, const char *cfg[]) * inconsistent. */ if (__wt_btree_immediately_durable(session)) { - /* - * Add the btree ID to the bitstring, so we can exclude any lookaside entries for this - * btree. - */ if (btree->id >= conn->stable_rollback_maxfile) WT_PANIC_RET(session, EINVAL, "btree file ID %" PRIu32 " larger than max %" PRIu32, btree->id, conn->stable_rollback_maxfile); - __bit_set(conn->stable_rollback_bitstring, btree->id); + __wt_verbose(session, WT_VERB_RTS, + "%s: Immediately durable btree skipped for rollback to stable", btree->dhandle->name); return (0); } @@ -387,33 +753,24 @@ __txn_rollback_to_stable_btree(WT_SESSION_IMPL *session, const char *cfg[]) /* There is nothing to do on an empty tree. */ if (btree->root.page == NULL) return (0); - - /* - * Copy the stable timestamp, otherwise we'd need to lock it each time it's accessed. Even - * though the stable timestamp isn't supposed to be updated while rolling back, accessing it - * without a lock would violate protocol. - */ - WT_ORDERED_READ(rollback_timestamp, txn_global->stable_timestamp); - /* * Ensure the eviction server is out of the file - we don't want it messing with us. This step * shouldn't be required, but it simplifies some of the reasoning about what state trees can be * in. */ WT_RET(__wt_evict_file_exclusive_on(session)); - WT_WITH_PAGE_INDEX( - session, ret = __txn_rollback_to_stable_btree_walk(session, rollback_timestamp)); + WT_WITH_PAGE_INDEX(session, ret = __rollback_to_stable_btree_walk(session, rollback_timestamp)); __wt_evict_file_exclusive_off(session); return (ret); } /* - * __txn_rollback_to_stable_check -- + * __rollback_to_stable_check -- * Ensure the rollback request is reasonable. */ static int -__txn_rollback_to_stable_check(WT_SESSION_IMPL *session) +__rollback_to_stable_check(WT_SESSION_IMPL *session) { WT_CONNECTION_IMPL *conn; WT_DECL_RET; @@ -429,15 +786,13 @@ __txn_rollback_to_stable_check(WT_SESSION_IMPL *session) /* * Help the user comply with the requirement that there are no concurrent operations. Protect * against spurious conflicts with the sweep server: we exclude it from running concurrent with - * rolling back the lookaside contents. + * rolling back the history store contents. */ - __wt_writelock(session, &conn->cache->las_sweepwalk_lock); ret = __wt_txn_activity_check(session, &txn_active); #ifdef HAVE_DIAGNOSTIC if (txn_active) WT_TRET(__wt_verbose_dump_txn(session)); #endif - __wt_writeunlock(session, &conn->cache->las_sweepwalk_lock); if (ret == 0 && txn_active) WT_RET_MSG(session, EINVAL, "rollback_to_stable illegal with active transactions"); @@ -446,63 +801,201 @@ __txn_rollback_to_stable_check(WT_SESSION_IMPL *session) } /* - * __txn_rollback_to_stable -- - * Rollback all in-memory state related to timestamps more recent than the passed in timestamp. + * __rollback_to_stable_btree_hs_cleanup -- + * Wipe all history store updates for the btree (non-timestamped tables) */ static int -__txn_rollback_to_stable(WT_SESSION_IMPL *session, const char *cfg[]) +__rollback_to_stable_btree_hs_cleanup(WT_SESSION_IMPL *session, uint32_t btree_id) { - WT_CONNECTION_IMPL *conn; + WT_CURSOR *hs_cursor; + WT_CURSOR_BTREE *cbt; + WT_DECL_ITEM(hs_key); WT_DECL_RET; + WT_ITEM key; + WT_UPDATE *hs_upd; + wt_timestamp_t hs_start_ts; + uint64_t hs_counter; + uint32_t hs_btree_id, session_flags; + int exact; + bool is_owner; + + hs_cursor = NULL; + WT_CLEAR(key); + hs_upd = NULL; + session_flags = 0; + is_owner = false; + + WT_ERR(__wt_scr_alloc(session, 0, &hs_key)); + + /* Open a history store table cursor. */ + WT_ERR(__wt_hs_cursor(session, &session_flags, &is_owner)); + hs_cursor = session->hs_cursor; + cbt = (WT_CURSOR_BTREE *)hs_cursor; + + /* Walk the history store for the given btree. */ + hs_cursor->set_key(hs_cursor, btree_id, &key, WT_TS_NONE, 0); + ret = hs_cursor->search_near(hs_cursor, &exact); - conn = S2C(session); + /* + * The search should always end up pointing to the start of the required btree or end of the + * previous btree on success. Move the cursor based on the result. + */ + WT_ASSERT(session, (ret != 0 || exact != 0)); + if (ret == 0 && exact < 0) + ret = hs_cursor->next(hs_cursor); + + for (; ret == 0; ret = hs_cursor->next(hs_cursor)) { + WT_ERR(hs_cursor->get_key(hs_cursor, &hs_btree_id, hs_key, &hs_start_ts, &hs_counter)); + + /* Stop crossing into the next btree boundary. */ + if (btree_id != hs_btree_id) + break; + + /* Set this comparison as exact match of the search for later use. */ + cbt->compare = 0; + + WT_ERR(__wt_upd_alloc_tombstone(session, &hs_upd)); + WT_ERR(__wt_hs_modify(cbt, hs_upd)); + WT_STAT_CONN_INCR(session, txn_rts_hs_removed); + hs_upd = NULL; + } + WT_ERR_NOTFOUND_OK(ret); + +err: + __wt_scr_free(session, &hs_key); + __wt_free(session, hs_upd); + WT_TRET(__wt_hs_cursor_close(session, session_flags, is_owner)); + + return (ret); +} + +/* + * __rollback_to_stable_btree_apply -- + * Perform rollback to stable to all files listed in the metadata, apart from the metadata and + * history store files. + */ +static int +__rollback_to_stable_btree_apply(WT_SESSION_IMPL *session) +{ + WT_CONFIG ckptconf; + WT_CONFIG_ITEM cval, durableval, key; + WT_CURSOR *cursor; + WT_DECL_RET; + WT_TXN_GLOBAL *txn_global; + wt_timestamp_t newest_durable_ts, rollback_timestamp; + const char *config, *uri; + bool durable_ts_found; + + txn_global = &S2C(session)->txn_global; - WT_STAT_CONN_INCR(session, txn_rollback_to_stable); /* - * Mark that a rollback operation is in progress and wait for eviction to drain. This is - * necessary because lookaside eviction uses transactions and causes the check for a quiescent - * system to fail. - * - * Configuring lookaside eviction off isn't atomic, safe because the flag is only otherwise set - * when closing down the database. Assert to avoid confusion in the future. + * Copy the stable timestamp, otherwise we'd need to lock it each time it's accessed. Even + * though the stable timestamp isn't supposed to be updated while rolling back, accessing it + * without a lock would violate protocol. */ - WT_ASSERT(session, !F_ISSET(conn, WT_CONN_EVICTION_NO_LOOKASIDE)); - F_SET(conn, WT_CONN_EVICTION_NO_LOOKASIDE); + WT_ORDERED_READ(rollback_timestamp, txn_global->stable_timestamp); + + WT_ASSERT(session, F_ISSET(session, WT_SESSION_LOCKED_SCHEMA)); + WT_RET(__wt_metadata_cursor(session, &cursor)); + + while ((ret = cursor->next(cursor)) == 0) { + WT_ERR(cursor->get_key(cursor, &uri)); + + /* Ignore metadata and history store files. */ + if (strcmp(uri, WT_METAFILE_URI) == 0 || strcmp(uri, WT_HS_URI) == 0) + continue; + + if (!WT_PREFIX_MATCH(uri, "file:")) + continue; + + WT_ERR(cursor->get_value(cursor, &config)); + + /* Find out the max durable timestamp of the object from checkpoint. */ + newest_durable_ts = WT_TS_NONE; + durable_ts_found = false; + WT_ERR(__wt_config_getones(session, config, "checkpoint", &cval)); + __wt_config_subinit(session, &ckptconf, &cval); + for (; __wt_config_next(&ckptconf, &key, &cval) == 0;) { + ret = __wt_config_subgets(session, &cval, "newest_durable_ts", &durableval); + if (ret == 0) { + newest_durable_ts = WT_MAX(newest_durable_ts, (wt_timestamp_t)durableval.val); + durable_ts_found = true; + } + WT_ERR_NOTFOUND_OK(ret); + } + + ret = __wt_session_get_dhandle(session, uri, NULL, NULL, 0); + /* Ignore performing rollback to stable on files that don't exist. */ + if (ret == ENOENT) { + __wt_verbose( + session, WT_VERB_RTS, "%s: rollback to stable ignored on non existing file", uri); + continue; + } + WT_ERR(ret); + + /* + * The rollback operation should be performed on this file based on the following: + * 1. The tree is modified. + * 2. The checkpoint durable timestamp is greater than the rollback timestamp. + * 3. There is no durable timestamp in any checkpoint. + */ + if (S2BT(session)->modified || newest_durable_ts > rollback_timestamp || + !durable_ts_found) { + __wt_verbose(session, WT_VERB_RTS, "%s: file rolled back", uri); + WT_TRET(__rollback_to_stable_btree(session, rollback_timestamp)); + } else + __wt_verbose(session, WT_VERB_RTS, "%s: file skipped", uri); + + /* Cleanup any history store entries for this non-timestamped table. */ + if (newest_durable_ts == WT_TS_NONE && !F_ISSET(S2C(session), WT_CONN_IN_MEMORY)) { + __wt_verbose( + session, WT_VERB_RTS, "%s: non-timestamped file history store cleanup", uri); + WT_TRET(__rollback_to_stable_btree_hs_cleanup(session, S2BT(session)->id)); + } + + WT_TRET(__wt_session_release_dhandle(session)); + WT_ERR(ret); + } + WT_ERR_NOTFOUND_OK(ret); + +err: + WT_TRET(__wt_metadata_cursor_release(session, &cursor)); + return (ret); +} - WT_ERR(__wt_conn_btree_apply(session, NULL, __txn_rollback_eviction_drain, NULL, cfg)); +/* + * __rollback_to_stable -- + * Rollback all modifications with timestamps more recent than the passed in timestamp. + */ +static int +__rollback_to_stable(WT_SESSION_IMPL *session, const char *cfg[]) +{ + WT_CONNECTION_IMPL *conn; + WT_DECL_RET; - WT_ERR(__txn_rollback_to_stable_check(session)); + conn = S2C(session); - F_CLR(conn, WT_CONN_EVICTION_NO_LOOKASIDE); + /* Mark that a rollback operation is in progress and wait for eviction to drain. */ + WT_RET(__wt_conn_btree_apply(session, NULL, __rollback_eviction_drain, NULL, cfg)); + + WT_RET(__rollback_to_stable_check(session)); /* * Allocate a non-durable btree bitstring. We increment the global value before using it, so the * current value is already in use, and hence we need to add one here. */ conn->stable_rollback_maxfile = conn->next_file_id + 1; - WT_ERR(__bit_alloc(session, conn->stable_rollback_maxfile, &conn->stable_rollback_bitstring)); - WT_ERR(__wt_conn_btree_apply(session, NULL, __txn_rollback_to_stable_btree, NULL, cfg)); + WT_WITH_SCHEMA_LOCK(session, ret = __rollback_to_stable_btree_apply(session)); - /* - * Clear any offending content from the lookaside file. This must be done after the in-memory - * application, since the process of walking trees in cache populates a list that is used to - * check which lookaside records should be removed. - */ - if (!F_ISSET(conn, WT_CONN_IN_MEMORY)) - WT_ERR(__txn_rollback_to_stable_lookaside_fixup(session)); - -err: - F_CLR(conn, WT_CONN_EVICTION_NO_LOOKASIDE); - __wt_free(session, conn->stable_rollback_bitstring); return (ret); } /* - * __wt_txn_rollback_to_stable -- - * Rollback all in-memory state related to timestamps more recent than the passed in timestamp. + * __wt_rollback_to_stable -- + * Rollback all modifications with timestamps more recent than the passed in timestamp. */ int -__wt_txn_rollback_to_stable(WT_SESSION_IMPL *session, const char *cfg[]) +__wt_rollback_to_stable(WT_SESSION_IMPL *session, const char *cfg[], bool no_ckpt) { WT_DECL_RET; @@ -512,7 +1005,18 @@ __wt_txn_rollback_to_stable(WT_SESSION_IMPL *session, const char *cfg[]) * concurrently. */ WT_RET(__wt_open_internal_session(S2C(session), "txn rollback_to_stable", true, 0, &session)); - ret = __txn_rollback_to_stable(session, cfg); + + F_SET(session, WT_SESSION_ROLLBACK_TO_STABLE_FLAGS); + ret = __rollback_to_stable(session, cfg); + F_CLR(session, WT_SESSION_ROLLBACK_TO_STABLE_FLAGS); + + /* + * If the configuration is not in-memory, forcibly log a checkpoint after rollback to stable to + * ensure that both in-memory and on-disk versions are the same unless caller requested for no + * checkpoint. + */ + if (!F_ISSET(S2C(session), WT_CONN_IN_MEMORY) && !no_ckpt) + WT_TRET(session->iface.checkpoint(&session->iface, "force=1")); WT_TRET(session->iface.close(&session->iface, NULL)); return (ret); diff --git a/src/third_party/wiredtiger/src/txn/txn_timestamp.c b/src/third_party/wiredtiger/src/txn/txn_timestamp.c index a0cd978529a..a10745aa411 100644 --- a/src/third_party/wiredtiger/src/txn/txn_timestamp.c +++ b/src/third_party/wiredtiger/src/txn/txn_timestamp.c @@ -9,6 +9,20 @@ #include "wt_internal.h" /* + * __wt_time_pair_to_string -- + * Converts a time pair to a standard string representation. + */ +char * +__wt_time_pair_to_string(wt_timestamp_t timestamp, uint64_t txn_id, char *tp_string) +{ + char ts_string[WT_TS_INT_STRING_SIZE]; + + WT_IGNORE_RET(__wt_snprintf(tp_string, WT_TP_STRING_SIZE, "%s/%" PRIu64, + __wt_timestamp_to_string(timestamp, ts_string), txn_id)); + return (tp_string); +} + +/* * __wt_timestamp_to_string -- * Convert a timestamp to the MongoDB string representation. */ diff --git a/src/third_party/wiredtiger/src/utilities/util_dump.c b/src/third_party/wiredtiger/src/utilities/util_dump.c index a4834cbd596..5bf06a01be3 100644..100755 --- a/src/third_party/wiredtiger/src/utilities/util_dump.c +++ b/src/third_party/wiredtiger/src/utilities/util_dump.c @@ -25,6 +25,7 @@ static int dump_table_config(WT_SESSION *, WT_CURSOR *, WT_CURSOR *, const char static int dump_table_parts_config(WT_SESSION *, WT_CURSOR *, const char *, const char *, bool); static int dup_json_string(const char *, char **); static int print_config(WT_SESSION *, const char *, const char *, bool, bool); +static int time_pair_to_timestamp(WT_SESSION_IMPL *, char *, WT_ITEM *); static int usage(void); static FILE *fp; @@ -37,15 +38,15 @@ util_dump(WT_SESSION *session, int argc, char *argv[]) WT_DECL_RET; WT_SESSION_IMPL *session_impl; int ch, i; - char *checkpoint, *ofile, *p, *simpleuri, *uri; + char *checkpoint, *ofile, *p, *simpleuri, *timestamp, *uri; bool hex, json, reverse; session_impl = (WT_SESSION_IMPL *)session; cursor = NULL; - checkpoint = ofile = simpleuri = uri = NULL; + checkpoint = ofile = simpleuri = uri = timestamp = NULL; hex = json = reverse = false; - while ((ch = __wt_getopt(progname, argc, argv, "c:f:jrx")) != EOF) + while ((ch = __wt_getopt(progname, argc, argv, "c:f:t:jrx")) != EOF) switch (ch) { case 'c': checkpoint = __wt_optarg; @@ -53,6 +54,9 @@ util_dump(WT_SESSION *session, int argc, char *argv[]) case 'f': ofile = __wt_optarg; break; + case 't': + timestamp = __wt_optarg; + break; case 'j': json = true; break; @@ -100,6 +104,16 @@ util_dump(WT_SESSION *session, int argc, char *argv[]) if ((uri = util_uri(session, argv[i], "table")) == NULL) goto err; + if (timestamp != NULL) { + WT_ERR(__wt_buf_set(session_impl, tmp, "", 0)); + WT_ERR(time_pair_to_timestamp(session_impl, timestamp, tmp)); + WT_ERR(__wt_buf_catfmt(session_impl, tmp, "isolation=snapshot,")); + if ((ret = session->begin_transaction(session, (char *)tmp->data)) != 0) { + fprintf(stderr, "%s: begin transaction failed: %s\n", progname, + session->strerror(session, ret)); + goto err; + } + } WT_ERR(__wt_buf_set(session_impl, tmp, "", 0)); if (checkpoint != NULL) WT_ERR(__wt_buf_catfmt(session_impl, tmp, "checkpoint=%s,", checkpoint)); @@ -117,6 +131,14 @@ util_dump(WT_SESSION *session, int argc, char *argv[]) } if ((p = strchr(simpleuri, '(')) != NULL) *p = '\0'; + /* + * If we're dumping the history store, we need to set this flag to ignore tombstones. Every + * record in the history store is succeeded by a tombstone so we need to do this otherwise + * nothing will be visible. The only exception is if we've supplied a timestamp in which + * case, we're specifically interested in what is visible at a given read timestamp. + */ + if (WT_STREQ(simpleuri, WT_HS_URI) && timestamp == NULL) + F_SET(session_impl, WT_SESSION_IGNORE_HS_TOMBSTONE); if (dump_config(session, simpleuri, cursor, hex, json) != 0) goto err; @@ -140,6 +162,7 @@ err: ret = 1; } + F_CLR(session_impl, WT_SESSION_IGNORE_HS_TOMBSTONE); if (cursor != NULL && (ret = cursor->close(cursor)) != 0) ret = util_err(session, ret, NULL); if (ofile != NULL && (ret = fclose(fp)) != 0) @@ -153,6 +176,27 @@ err: } /* + * time_pair_to_timestamp -- + * Convert a timestamp output format to timestamp representation. + */ +static int +time_pair_to_timestamp(WT_SESSION_IMPL *session_impl, char *ts_string, WT_ITEM *buf) +{ + wt_timestamp_t timestamp; + uint32_t first, second; + + if (ts_string[0] == '(') { + if (sscanf(ts_string, "(%" SCNu32 " ,%" SCNu32 ")", &first, &second) != 2) + return (EINVAL); + timestamp = ((wt_timestamp_t)first << 32) | second; + } else + timestamp = __wt_strtouq(ts_string, NULL, 10); + + WT_RET(__wt_buf_catfmt(session_impl, buf, "read_timestamp=%" PRIx64 ",", timestamp)); + return (0); +} + +/* * dump_config -- * Dump the config for the uri. */ @@ -626,7 +670,7 @@ usage(void) { (void)fprintf(stderr, "usage: %s %s " - "dump [-jrx] [-c checkpoint] [-f output-file] uri\n", + "dump [-jrx] [-c checkpoint] [-f output-file] [-t timestamp] uri\n", progname, usage_prefix); return (1); } diff --git a/src/third_party/wiredtiger/src/utilities/util_list.c b/src/third_party/wiredtiger/src/utilities/util_list.c index 76f9fc05206..1463ba23862 100644 --- a/src/third_party/wiredtiger/src/utilities/util_list.c +++ b/src/third_party/wiredtiger/src/utilities/util_list.c @@ -124,9 +124,7 @@ list_print(WT_SESSION *session, const char *uri, bool cflag, bool vflag) if ((ret = cursor->get_key(cursor, &key)) != 0) return (util_cerr(cursor, "get_key", ret)); - /* - * If a name is specified, only show objects that match. - */ + /* If a name is specified, only show objects that match. */ if (uri != NULL) { if (!WT_PREFIX_MATCH(key, uri)) continue; @@ -135,15 +133,13 @@ list_print(WT_SESSION *session, const char *uri, bool cflag, bool vflag) /* * !!! - * We don't normally say anything about the WiredTiger metadata - * and lookaside tables, they're not application/user "objects" - * in the database. I'm making an exception for the checkpoint - * and verbose options. However, skip over the metadata system - * information for anything except the verbose option. + * Don't report anything about the WiredTiger metadata and history store since they are not + * user created objects unless the verbose or checkpoint options are passed in. However, + * skip over the metadata system information for anything except the verbose option. */ if (!vflag && WT_PREFIX_MATCH(key, WT_SYSTEM_PREFIX)) continue; - if (cflag || vflag || (strcmp(key, WT_METADATA_URI) != 0 && strcmp(key, WT_LAS_URI) != 0)) + if (cflag || vflag || (strcmp(key, WT_METADATA_URI) != 0 && strcmp(key, WT_HS_URI) != 0)) printf("%s\n", key); if (!cflag && !vflag) diff --git a/src/third_party/wiredtiger/src/utilities/util_verify.c b/src/third_party/wiredtiger/src/utilities/util_verify.c index 29b777ea8b7..daf80e4bb1f 100644 --- a/src/third_party/wiredtiger/src/utilities/util_verify.c +++ b/src/third_party/wiredtiger/src/utilities/util_verify.c @@ -17,17 +17,24 @@ util_verify(WT_SESSION *session, int argc, char *argv[]) size_t size; int ch; char *config, *dump_offsets, *uri; - bool dump_address, dump_blocks, dump_layout, dump_pages; + bool dump_address, dump_blocks, dump_layout, dump_pages, dump_history, hs_verify, + stable_timestamp; - dump_address = dump_blocks = dump_layout = dump_pages = false; + dump_address = dump_blocks = dump_history = dump_layout = dump_pages = hs_verify = + stable_timestamp = false; config = dump_offsets = uri = NULL; - while ((ch = __wt_getopt(progname, argc, argv, "d:")) != EOF) + while ((ch = __wt_getopt(progname, argc, argv, "ad:s")) != EOF) switch (ch) { + case 'a': + hs_verify = true; + break; case 'd': if (strcmp(__wt_optarg, "dump_address") == 0) dump_address = true; else if (strcmp(__wt_optarg, "dump_blocks") == 0) dump_blocks = true; + else if (strcmp(__wt_optarg, "dump_history") == 0) + dump_history = true; else if (strcmp(__wt_optarg, "dump_layout") == 0) dump_layout = true; else if (WT_PREFIX_MATCH(__wt_optarg, "dump_offsets=")) { @@ -44,6 +51,9 @@ util_verify(WT_SESSION *session, int argc, char *argv[]) else return (usage()); break; + case 's': + stable_timestamp = true; + break; case '?': default: return (usage()); @@ -51,26 +61,41 @@ util_verify(WT_SESSION *session, int argc, char *argv[]) argc -= __wt_optind; argv += __wt_optind; - /* The remaining argument is the table name. */ - if (argc != 1) - return (usage()); - if ((uri = util_uri(session, *argv, "table")) == NULL) - return (1); + /* + * The remaining argument is the table name. If we are verifying the history store we do not + * accept a URI. Otherwise, we need a URI top operate on. + */ + if (hs_verify && argc != 0) + (void)util_err(session, 0, "-a can't be used along with a uri"); + if (!hs_verify) { + if (argc != 1) + return (usage()); + if ((uri = util_uri(session, *argv, "table")) == NULL) + return (1); + } + + if (hs_verify && (dump_address || dump_blocks || dump_layout || dump_offsets != NULL || + dump_pages || stable_timestamp)) { + (void)util_err(session, 0, "-a and -d are not supported together"); + } - /* Build the configuration string as necessary. */ - if (dump_address || dump_blocks || dump_layout || dump_offsets != NULL || dump_pages) { - size = strlen("dump_address,") + strlen("dump_blocks,") + strlen("dump_layout,") + - strlen("dump_pages,") + strlen("dump_offsets[],") + - (dump_offsets == NULL ? 0 : strlen(dump_offsets)) + 20; + if (dump_address || dump_blocks || dump_history || dump_layout || dump_offsets != NULL || + dump_pages || hs_verify || stable_timestamp) { + size = strlen("dump_address,") + strlen("dump_blocks,") + strlen("dump_history") + + strlen("dump_layout,") + strlen("dump_pages,") + strlen("dump_offsets[],") + + (dump_offsets == NULL ? 0 : strlen(dump_offsets)) + strlen("history_store") + + strlen("stable_timestamp,") + 20; if ((config = malloc(size)) == NULL) { ret = util_err(session, errno, NULL); goto err; } - if ((ret = __wt_snprintf(config, size, "%s%s%s%s%s%s%s", + if ((ret = __wt_snprintf(config, size, "%s%s%s%s%s%s%s%s%s%s", dump_address ? "dump_address," : "", dump_blocks ? "dump_blocks," : "", - dump_layout ? "dump_layout," : "", dump_offsets != NULL ? "dump_offsets=[" : "", + dump_history ? "dump_history," : "", dump_layout ? "dump_layout," : "", + dump_offsets != NULL ? "dump_offsets=[" : "", dump_offsets != NULL ? dump_offsets : "", dump_offsets != NULL ? "]," : "", - dump_pages ? "dump_pages," : "")) != 0) { + dump_pages ? "dump_pages," : "", hs_verify ? "history_store" : "", + stable_timestamp ? "stable_timestamp," : "")) != 0) { (void)util_err(session, ret, NULL); goto err; } @@ -98,7 +123,7 @@ usage(void) "usage: %s %s " "verify %s\n", progname, usage_prefix, - "[-d dump_address | dump_blocks | dump_layout | " - "dump_offsets=#,# | dump_pages] uri"); + "[-d dump_address | dump_blocks | dump_history | dump_layout | " + "dump_offsets=#,# | dump_pages] [-s] -a|uri"); return (1); } diff --git a/src/third_party/wiredtiger/test/checkpoint/Makefile.am b/src/third_party/wiredtiger/test/checkpoint/Makefile.am index da7b85ec0b2..f5a4a8decd1 100644 --- a/src/third_party/wiredtiger/test/checkpoint/Makefile.am +++ b/src/third_party/wiredtiger/test/checkpoint/Makefile.am @@ -9,7 +9,8 @@ t_LDADD = $(top_builddir)/test/utility/libtest_util.la t_LDADD +=$(top_builddir)/libwiredtiger.la t_LDFLAGS = -static -TESTS = smoke.sh +# Temporarily disabled +# TESTS = smoke.sh clean-local: rm -rf WT_TEST core.* *.core diff --git a/src/third_party/wiredtiger/test/checkpoint/checkpointer.c b/src/third_party/wiredtiger/test/checkpoint/checkpointer.c index 28a6f6202f7..5831bb5eee1 100644 --- a/src/third_party/wiredtiger/test/checkpoint/checkpointer.c +++ b/src/third_party/wiredtiger/test/checkpoint/checkpointer.c @@ -33,7 +33,7 @@ static WT_THREAD_RET clock_thread(void *); static int compare_cursors(WT_CURSOR *, const char *, WT_CURSOR *, const char *); static int diagnose_key_error(WT_CURSOR *, int, WT_CURSOR *, int); static int real_checkpointer(void); -static int verify_consistency(WT_SESSION *, bool); +static int verify_consistency(WT_SESSION *, char *); /* * start_checkpoints -- @@ -82,14 +82,13 @@ clock_thread(void *arg) testutil_check(g.conn->open_session(g.conn, NULL, NULL, &wt_session)); session = (WT_SESSION_IMPL *)wt_session; - g.ts = 0; + g.ts_stable = 0; while (g.running) { __wt_writelock(session, &g.clock_lock); - ++g.ts; - testutil_check( - __wt_snprintf(buf, sizeof(buf), "oldest_timestamp=%x,stable_timestamp=%x", g.ts, g.ts)); + ++g.ts_stable; + testutil_check(__wt_snprintf(buf, sizeof(buf), "stable_timestamp=%x", g.ts_stable)); testutil_check(g.conn->set_timestamp(g.conn, buf)); - if (g.ts % 997 == 0) { + if (g.ts_stable % 997 == 0) { /* * Random value between 6 and 10 seconds. */ @@ -139,7 +138,11 @@ real_checkpointer(void) WT_SESSION *session; uint64_t delay; int ret; - char buf[128], *checkpoint_config; + char buf[128], timestamp_buf[64]; + const char *checkpoint_config; + + checkpoint_config = "use_timestamp=false"; + g.ts_oldest = 0; if (g.running == 0) return (log_print_err("Checkpoint thread started stopped\n", EINVAL, 1)); @@ -151,18 +154,28 @@ real_checkpointer(void) if ((ret = g.conn->open_session(g.conn, NULL, NULL, &session)) != 0) return (log_print_err("conn.open_session", ret, 1)); - if (WT_PREFIX_MATCH(g.checkpoint_name, "WiredTigerCheckpoint")) - checkpoint_config = NULL; - else { - testutil_check(__wt_snprintf(buf, sizeof(buf), "name=%s", g.checkpoint_name)); + if (g.use_timestamps) + checkpoint_config = "use_timestamp=true"; + + if (!WT_PREFIX_MATCH(g.checkpoint_name, "WiredTigerCheckpoint")) { + testutil_check( + __wt_snprintf(buf, sizeof(buf), "name=%s,%s", g.checkpoint_name, checkpoint_config)); checkpoint_config = buf; } while (g.running) { - /* Check for consistency of online data */ - if ((ret = verify_consistency(session, false)) != 0) + /* + * Check for consistency of online data, here we don't expect to see the version at the + * checkpoint just a consistent view across all tables. + */ + if ((ret = verify_consistency(session, NULL)) != 0) return (log_print_err("verify_consistency (online)", ret, 1)); + if (g.use_timestamps) { + WT_ORDERED_READ(g.ts_oldest, g.ts_stable); + testutil_check(g.conn->query_timestamp(g.conn, timestamp_buf, "get=stable")); + } + /* Execute a checkpoint */ if ((ret = session->checkpoint(session, checkpoint_config)) != 0) return (log_print_err("session.checkpoint", ret, 1)); @@ -172,13 +185,21 @@ real_checkpointer(void) if (!g.running) goto done; - /* Verify the content of the checkpoint. */ - if ((ret = verify_consistency(session, true)) != 0) - return (log_print_err("verify_consistency (offline)", ret, 1)); - /* - * Random value between 4 and 8 seconds. + * Verify the content of the checkpoint at the stable timestamp. We can't verify checkpoints + * without timestamps as such we don't perform a verification here in the non-timestamped + * scenario. */ + if (g.use_timestamps && (ret = verify_consistency(session, timestamp_buf)) != 0) + return (log_print_err("verify_consistency (timestamps)", ret, 1)); + + /* Advance the oldest timestamp to the most recently set stable timestamp. */ + if (g.use_timestamps && g.ts_oldest != 0) { + testutil_check(__wt_snprintf( + timestamp_buf, sizeof(timestamp_buf), "oldest_timestamp=%x", g.ts_oldest)); + testutil_check(g.conn->set_timestamp(g.conn, timestamp_buf)); + } + /* Random value between 4 and 8 seconds. */ if (g.sweep_stress) { delay = __wt_random(&rnd) % 5; __wt_sleep(delay + 4, 0); @@ -198,13 +219,13 @@ done: * The key/values should match across all tables. */ static int -verify_consistency(WT_SESSION *session, bool use_checkpoint) +verify_consistency(WT_SESSION *session, char *stable_timestamp) { WT_CURSOR **cursors; uint64_t key_count; int i, ret, t_ret; - char ckpt_buf[128], next_uri[128]; - const char *ckpt, *type0, *typei; + char cfg_buf[128], next_uri[128]; + const char *type0, *typei; ret = t_ret = 0; key_count = 0; @@ -212,23 +233,17 @@ verify_consistency(WT_SESSION *session, bool use_checkpoint) if (cursors == NULL) return (log_print_err("verify_consistency", ENOMEM, 1)); - if (use_checkpoint) { - testutil_check( - __wt_snprintf(ckpt_buf, sizeof(ckpt_buf), "checkpoint=%s", g.checkpoint_name)); - ckpt = ckpt_buf; + if (stable_timestamp != NULL) { + testutil_check(__wt_snprintf( + cfg_buf, sizeof(cfg_buf), "isolation=snapshot,read_timestamp=%s", stable_timestamp)); } else { - ckpt = NULL; - testutil_check(session->begin_transaction(session, "isolation=snapshot")); + testutil_check(__wt_snprintf(cfg_buf, sizeof(cfg_buf), "isolation=snapshot")); } + testutil_check(session->begin_transaction(session, cfg_buf)); for (i = 0; i < g.ntables; i++) { - /* - * TODO: LSM doesn't currently support reading from checkpoints. - */ - if (use_checkpoint && g.cookies[i].type == LSM) - continue; testutil_check(__wt_snprintf(next_uri, sizeof(next_uri), "table:__wt%04d", i)); - if ((ret = session->open_cursor(session, next_uri, NULL, ckpt, &cursors[i])) != 0) { + if ((ret = session->open_cursor(session, next_uri, NULL, NULL, &cursors[i])) != 0) { (void)log_print_err("verify_consistency:session.open_cursor", ret, 1); goto err; } @@ -283,7 +298,7 @@ verify_consistency(WT_SESSION *session, bool use_checkpoint) } } printf("Finished verifying a %s with %d tables and %" PRIu64 " keys\n", - use_checkpoint ? "checkpoint" : "snapshot", g.ntables, key_count); + stable_timestamp != NULL ? "checkpoint" : "snapshot", g.ntables, key_count); fflush(stdout); err: @@ -291,8 +306,7 @@ err: if (cursors[i] != NULL && (ret = cursors[i]->close(cursors[i])) != 0) (void)log_print_err("verify_consistency:cursor close", ret, 1); } - if (!use_checkpoint) - testutil_check(session->commit_transaction(session, NULL)); + testutil_check(session->commit_transaction(session, NULL)); free(cursors); return (ret); } diff --git a/src/third_party/wiredtiger/test/checkpoint/smoke.sh b/src/third_party/wiredtiger/test/checkpoint/smoke.sh index e1bc3819c74..faae6f1ddea 100755 --- a/src/third_party/wiredtiger/test/checkpoint/smoke.sh +++ b/src/third_party/wiredtiger/test/checkpoint/smoke.sh @@ -24,7 +24,7 @@ $TEST_WRAPPER ./t -T 6 -t r echo "checkpoint: 6 row-store tables, named checkpoint" $TEST_WRAPPER ./t -c 'TeSt' -T 6 -t r -echo "checkpoint: row-store tables, stress LAS. Sweep and timestamps" +echo "checkpoint: row-store tables, stress history store. Sweep and timestamps" $TEST_WRAPPER ./t -t r -W 3 -r 2 -D -s -x -n 100000 -k 100000 -C cache_size=100MB echo "checkpoint: row-store tables, Sweep and timestamps" diff --git a/src/third_party/wiredtiger/test/checkpoint/test_checkpoint.c b/src/third_party/wiredtiger/test/checkpoint/test_checkpoint.c index e588eba913e..0166bddb45a 100644 --- a/src/third_party/wiredtiger/test/checkpoint/test_checkpoint.c +++ b/src/third_party/wiredtiger/test/checkpoint/test_checkpoint.c @@ -249,7 +249,8 @@ cleanup(bool remove_dir) { g.running = 0; g.ntables_created = 0; - g.ts = 0; + g.ts_oldest = 0; + g.ts_stable = 0; if (remove_dir) testutil_make_work_dir(g.home); @@ -350,6 +351,7 @@ usage(void) "\t-r set number of runs (0 for continuous)\n" "\t-T specify a table configuration\n" "\t-t set a file type ( col | mix | row | lsm )\n" - "\t-W set number of worker threads\n"); + "\t-W set number of worker threads\n" + "\t-x use timestamps\n"); return (EXIT_FAILURE); } diff --git a/src/third_party/wiredtiger/test/checkpoint/test_checkpoint.h b/src/third_party/wiredtiger/test/checkpoint/test_checkpoint.h index 61baaed235e..92dd47dda33 100644 --- a/src/third_party/wiredtiger/test/checkpoint/test_checkpoint.h +++ b/src/third_party/wiredtiger/test/checkpoint/test_checkpoint.h @@ -65,7 +65,8 @@ typedef struct { volatile int running; /* Whether to stop */ int status; /* Exit status */ bool sweep_stress; /* Sweep stress test */ - u_int ts; /* Current timestamp */ + u_int ts_oldest; /* Current oldest timestamp */ + u_int ts_stable; /* Current stable timestamp */ bool use_timestamps; /* Use txn timestamps */ COOKIE *cookies; /* Per-thread info */ WT_RWLOCK clock_lock; /* Clock synchronization */ diff --git a/src/third_party/wiredtiger/test/checkpoint/workers.c b/src/third_party/wiredtiger/test/checkpoint/workers.c index 0dd703a0beb..d8df3d49393 100644 --- a/src/third_party/wiredtiger/test/checkpoint/workers.c +++ b/src/third_party/wiredtiger/test/checkpoint/workers.c @@ -216,13 +216,15 @@ real_worker(void) WT_CURSOR **cursors; WT_RAND_STATE rnd; WT_SESSION *session; - u_int i, keyno; + u_int i, keyno, next_rnd; int j, ret, t_ret; char buf[128]; const char *begin_cfg; - bool has_cursors; + bool reopen_cursors, start_txn; ret = t_ret = 0; + reopen_cursors = false; + start_txn = true; if ((cursors = calloc((size_t)(g.ntables), sizeof(WT_CURSOR *))) == NULL) return (log_print_err("malloc", ENOMEM, 1)); @@ -232,6 +234,11 @@ real_worker(void) goto err; } + if (g.use_timestamps) + begin_cfg = "read_timestamp=1,roundup_timestamps=(read=true)"; + else + begin_cfg = NULL; + __wt_random_init_seed((WT_SESSION_IMPL *)session, &rnd); for (j = 0; j < g.ntables; j++) @@ -239,56 +246,67 @@ real_worker(void) (void)log_print_err("session.open_cursor", ret, 1); goto err; } - has_cursors = true; - - if (g.use_timestamps) - begin_cfg = "read_timestamp=1,roundup_timestamps=(read=true)"; - else - begin_cfg = NULL; for (i = 0; i < g.nops && g.running; ++i, __wt_yield()) { - if ((ret = session->begin_transaction(session, begin_cfg)) != 0) { - (void)log_print_err("real_worker:begin_transaction", ret, 1); - goto err; - } - keyno = __wt_random(&rnd) % g.nkeys + 1; - if (g.use_timestamps && i % 23 == 0) { - if (__wt_try_readlock((WT_SESSION_IMPL *)session, &g.clock_lock) != 0) { - testutil_check(session->commit_transaction(session, NULL)); - for (j = 0; j < g.ntables; j++) - testutil_check(cursors[j]->close(cursors[j])); - has_cursors = false; - __wt_readlock((WT_SESSION_IMPL *)session, &g.clock_lock); - testutil_check(session->begin_transaction(session, begin_cfg)); + if (start_txn) { + if ((ret = session->begin_transaction(session, begin_cfg)) != 0) { + (void)log_print_err("real_worker:begin_transaction", ret, 1); + goto err; } - testutil_check(__wt_snprintf(buf, sizeof(buf), "commit_timestamp=%x", g.ts + 1)); - testutil_check(session->timestamp_transaction(session, buf)); - __wt_readunlock((WT_SESSION_IMPL *)session, &g.clock_lock); - - for (j = 0; !has_cursors && j < g.ntables; j++) - if ((ret = session->open_cursor( - session, g.cookies[j].uri, NULL, NULL, &cursors[j])) != 0) { - (void)log_print_err("session.open_cursor", ret, 1); - goto err; - } - has_cursors = true; + start_txn = false; } - for (j = 0; ret == 0 && j < g.ntables; j++) { + keyno = __wt_random(&rnd) % g.nkeys + 1; + for (j = 0; ret == 0 && j < g.ntables; j++) ret = worker_op(cursors[j], keyno, i); - } if (ret != 0 && ret != WT_ROLLBACK) { (void)log_print_err("worker op failed", ret, 1); goto err; - } else if (ret == 0 && __wt_random(&rnd) % 7 != 0) { - if ((ret = session->commit_transaction(session, NULL)) != 0) { - (void)log_print_err("real_worker:commit_transaction", ret, 1); - goto err; + } else if (ret == 0) { + next_rnd = __wt_random(&rnd); + if (next_rnd % 7 != 0) { + if (g.use_timestamps) { + if (__wt_try_readlock((WT_SESSION_IMPL *)session, &g.clock_lock) == 0) { + testutil_check( + __wt_snprintf(buf, sizeof(buf), "commit_timestamp=%x", g.ts_stable + 1)); + __wt_readunlock((WT_SESSION_IMPL *)session, &g.clock_lock); + if ((ret = session->commit_transaction(session, buf)) != 0) { + (void)log_print_err("real_worker:commit_transaction", ret, 1); + goto err; + } + start_txn = true; + /* Occasionally reopen cursors after committing. */ + if (next_rnd % 13 == 0) { + reopen_cursors = true; + } + } + } else { + if ((ret = session->commit_transaction(session, NULL)) != 0) { + (void)log_print_err("real_worker:commit_transaction", ret, 1); + goto err; + } + start_txn = true; + } + } else if (next_rnd % 15 == 0) { + /* Occasionally reopen cursors during a running transaction. */ + reopen_cursors = true; } } else { if ((ret = session->rollback_transaction(session, NULL)) != 0) { (void)log_print_err("real_worker:rollback_transaction", ret, 1); goto err; } + start_txn = true; + } + if (reopen_cursors) { + for (j = 0; j < g.ntables; j++) { + testutil_check(cursors[j]->close(cursors[j])); + if ((ret = session->open_cursor( + session, g.cookies[j].uri, NULL, NULL, &cursors[j])) != 0) { + (void)log_print_err("session.open_cursor", ret, 1); + goto err; + } + } + reopen_cursors = false; } } diff --git a/src/third_party/wiredtiger/test/csuite/Makefile.am b/src/third_party/wiredtiger/test/csuite/Makefile.am index e2b7233f45b..94b8dc2bab6 100644 --- a/src/third_party/wiredtiger/test/csuite/Makefile.am +++ b/src/third_party/wiredtiger/test/csuite/Makefile.am @@ -8,7 +8,12 @@ all_TESTS= noinst_PROGRAMS= # The import test is only a shell script -all_TESTS += import/smoke.sh +# Temporarily disabled +# all_TESTS += import/smoke.sh + +test_incr_backup_SOURCES = incr_backup/main.c +noinst_PROGRAMS += test_incr_backup +all_TESTS += incr_backup/smoke.sh test_random_abort_SOURCES = random_abort/main.c noinst_PROGRAMS += test_random_abort @@ -116,7 +121,8 @@ all_TESTS += test_wt3874_pad_byte_collator test_wt4105_large_doc_small_upd_SOURCES = wt4105_large_doc_small_upd/main.c noinst_PROGRAMS += test_wt4105_large_doc_small_upd -all_TESTS += test_wt4105_large_doc_small_upd +# Temporarily disabled (WT-5579) +# all_TESTS += test_wt4105_large_doc_small_upd test_wt4117_checksum_SOURCES = wt4117_checksum/main.c noinst_PROGRAMS += test_wt4117_checksum @@ -128,15 +134,16 @@ all_TESTS += test_wt4156_metadata_salvage test_wt4333_handle_locks_SOURCES = wt4333_handle_locks/main.c noinst_PROGRAMS += test_wt4333_handle_locks -all_TESTS += test_wt4333_handle_locks +# Temporarily disabled +# all_TESTS += test_wt4333_handle_locks test_wt4699_json_SOURCES = wt4699_json/main.c noinst_PROGRAMS += test_wt4699_json all_TESTS += test_wt4699_json -test_wt4803_cache_overflow_abort_SOURCES = wt4803_cache_overflow_abort/main.c -noinst_PROGRAMS += test_wt4803_cache_overflow_abort -all_TESTS += test_wt4803_cache_overflow_abort +test_wt4803_history_store_abort_SOURCES = wt4803_history_store_abort/main.c +noinst_PROGRAMS += test_wt4803_history_store_abort +all_TESTS += test_wt4803_history_store_abort test_wt4891_meta_ckptlist_get_alloc_SOURCES=wt4891_meta_ckptlist_get_alloc/main.c noinst_PROGRAMS += test_wt4891_meta_ckptlist_get_alloc diff --git a/src/third_party/wiredtiger/test/csuite/incr_backup/main.c b/src/third_party/wiredtiger/test/csuite/incr_backup/main.c new file mode 100644 index 00000000000..3b9ed1319bb --- /dev/null +++ b/src/third_party/wiredtiger/test/csuite/incr_backup/main.c @@ -0,0 +1,891 @@ +/*- + * Public Domain 2014-2020 MongoDB, Inc. + * Public Domain 2008-2014 WiredTiger, Inc. + * + * This is free and unencumbered software released into the public domain. + * + * Anyone is free to copy, modify, publish, use, compile, sell, or + * distribute this software, either in source code form or as a compiled + * binary, for any purpose, commercial or non-commercial, and by any + * means. + * + * In jurisdictions that recognize copyright laws, the author or authors + * of this software dedicate any and all copyright interest in the + * software to the public domain. We make this dedication for the benefit + * of the public at large and to the detriment of our heirs and + * successors. We intend this dedication to be an overt act of + * relinquishment in perpetuity of all present and future rights to this + * software under copyright law. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. + * IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + */ + +/* + * This program tests incremental backup in a randomized way. The random seed used is reported and + * can be used in another run. + */ + +#include "test_util.h" + +#include <sys/wait.h> +#include <signal.h> + +#define ITERATIONS 10 +#define MAX_NTABLES 100 + +#define MAX_KEY_SIZE 100 +#define MAX_VALUE_SIZE 10000 +#define MAX_MODIFY_ENTRIES 10 +#define MAX_MODIFY_DIFF 500 + +#define URI_MAX_LEN 32 +#define URI_FORMAT "table:t%d-%d" +#define KEY_FORMAT "key-%d-%d" + +static int verbose_level = 0; +static uint64_t seed = 0; + +static void usage(void) WT_GCC_FUNC_DECL_ATTRIBUTE((noreturn)); + +/* + * Note: set this to true to copy incremental files completely. + */ +static bool slow_incremental = false; + +/* TODO: rename and drop are not currently working, they give resource busy. */ +static bool do_rename = false; +static bool do_drop = false; + +#define VERBOSE(level, fmt, ...) \ + do { \ + if (level <= verbose_level) \ + printf(fmt, __VA_ARGS__); \ + } while (0) + +/* + * We keep an array of tables, each one may or may not be in use. + * "In use" means it has been created, and will be updated from time to time. + */ +typedef struct { + char *name; /* non-null entries represent tables in use */ + uint32_t name_index; /* bumped when we rename or drop, so we get unique names. */ + uint64_t change_count; /* number of changes so far to the table */ + WT_RAND_STATE rand; + uint32_t max_value_size; +} TABLE; +#define TABLE_VALID(tablep) ((tablep)->name != NULL) + +/* + * The set of all tables in play, and other information used for this run. + */ +typedef struct { + TABLE *table; /* set of potential tables */ + uint32_t table_count; /* size of table array */ + uint32_t tables_in_use; /* count of tables that exist */ + uint32_t full_backup_number; + uint32_t incr_backup_number; +} TABLE_INFO; + +/* + * The set of active files in a backup. This is our "memory" of files that are used in each backup, + * so we can remove any that are not mentioned in the next backup. + */ +typedef struct { + char **names; + uint32_t count; +} ACTIVE_FILES; + +extern int __wt_optind; +extern char *__wt_optarg; + +/* + * The choices of operations we do to each table. + */ +typedef enum { INSERT, MODIFY, REMOVE, UPDATE, _OPERATION_TYPE_COUNT } OPERATION_TYPE; + +/* + * Cycle of changes to a table. + * + * When making changes to a table, the first KEYS_PER_TABLE changes are all inserts, the next + * KEYS_PER_TABLE are updates of the same records. The next KEYS_PER_TABLE are modifications of + * existing records, and the last KEYS_PER_TABLE will be removes. This defines one "cycle", and + * CHANGES_PER_CYCLE is the number of changes in a complete cycle. Thus at the end/beginning of each + * cycle, there are zero keys in the table. + * + * Having a predictable cycle makes it easy on the checking side (knowing how many total changes + * have been made) to check the state of the table. + */ +#define KEYS_PER_TABLE 10000 +#define CHANGES_PER_CYCLE (KEYS_PER_TABLE * _OPERATION_TYPE_COUNT) + +/* + * usage -- + * Print usage message and exit. + */ +static void +usage(void) +{ + fprintf(stderr, "usage: %s [-h dir] [-S seed] [-v verbose_level]\n", progname); + exit(EXIT_FAILURE); +} + +/* + * die -- + * Called when testutil_assert or testutil_check fails. + */ +static void +die(void) +{ + fprintf(stderr, + "**** FAILURE\n" + "To reproduce, please rerun with: %s -S %" PRIu64 "\n", + progname, seed); +} + +/* + * key_value -- + * Return the key, value and operation type for a given change to a table. See "Cycle of changes + * to a table" above. + * + * The keys generated are unique among the 10000, but we purposely don't make them sequential, so + * that insertions tend to be scattered among the pages in the B-tree. + * + * "key-0-0", "key-1-0", "key-2-0""... "key-99-0", "key-0-1", "key-1-1", ... + */ +static void +key_value(uint64_t change_count, char *key, size_t key_size, WT_ITEM *item, OPERATION_TYPE *typep) +{ + uint32_t key_num; + OPERATION_TYPE op_type; + size_t pos, value_size; + char *cp; + char ch; + + key_num = change_count % KEYS_PER_TABLE; + *typep = op_type = (OPERATION_TYPE)((change_count % CHANGES_PER_CYCLE) / KEYS_PER_TABLE); + + testutil_check( + __wt_snprintf(key, key_size, KEY_FORMAT, (int)(key_num % 100), (int)(key_num / 100))); + if (op_type == REMOVE) + return; /* remove needs no key */ + + /* The value sizes vary "predictably" up to the max value size for this table. */ + value_size = (change_count * 103) % (item->size + 1); + testutil_assert(value_size <= item->size); + + /* + * For a given key, a value is first inserted, then later updated, then modified. When a value + * is inserted, it is all the letter 'a'. When the value is updated, is it mostly 'b', with some + * 'c' mixed in. When the value is to modified, we'll end up with a value with mostly 'b' and + * 'M' mixed in, in different spots. Thus the modify operation will have both additions ('M') + * and + * subtractions ('c') from the previous version. + */ + if (op_type == INSERT) + ch = 'a'; + else + ch = 'b'; + + cp = (char *)item->data; + for (pos = 0; pos < value_size; pos++) { + cp[pos] = ch; + if (op_type == UPDATE && ((50 < pos && pos < 60) || (150 < pos && pos < 160))) + cp[pos] = 'c'; + else if (op_type == MODIFY && ((20 < pos && pos < 30) || (120 < pos && pos < 130))) + cp[pos] = 'M'; + } + item->size = value_size; +} + +/* + * active_files_init -- + * Initialize (clear) the active file struct. + */ +static void +active_files_init(ACTIVE_FILES *active) +{ + WT_CLEAR(*active); +} + +/* + * active_files_print -- + * Print the set of active files for debugging. + */ +static void +active_files_print(ACTIVE_FILES *active, const char *msg) +{ + uint32_t i; + + VERBOSE(6, "Active files: %s, %d entries\n", msg, (int)active->count); + for (i = 0; i < active->count; i++) + VERBOSE(6, " %s\n", active->names[i]); +} + +/* + * active_files_add -- + * Add a new name to the active file list. + */ +static void +active_files_add(ACTIVE_FILES *active, const char *name) +{ + uint32_t pos; + + pos = active->count++; + active->names = drealloc(active->names, sizeof(char *) * active->count); + active->names[pos] = strdup(name); +} + +/* + * active_files_sort_function -- + * Sort function for qsort. + */ +static int +active_files_sort_function(const void *left, const void *right) +{ + return (strcmp(*(const char **)left, *(const char **)right)); +} + +/* + * active_files_sort -- + * Sort the list of names in the active file list. + */ +static void +active_files_sort(ACTIVE_FILES *active) +{ + __wt_qsort(active->names, active->count, sizeof(char *), active_files_sort_function); +} + +/* + * active_files_remove_missing -- + * Files in the previous list that are missing from the current list are removed. + */ +static void +active_files_remove_missing(ACTIVE_FILES *prev, ACTIVE_FILES *cur, const char *dirname) +{ + uint32_t curpos, prevpos; + int cmp; + char filename[1024]; + + active_files_print(prev, "computing removals: previous list of active files"); + active_files_print(cur, "computing removals: current list of active files"); + curpos = 0; + /* + * Walk through the two lists looking for non-matches. + */ + for (prevpos = 0; prevpos < prev->count; prevpos++) { +again: + if (curpos >= cur->count) + cmp = -1; /* There are extra entries at the end of the prev list */ + else + cmp = strcmp(prev->names[prevpos], cur->names[curpos]); + + if (cmp == 0) + curpos++; + else if (cmp < 0) { + /* + * There is something in the prev list not in the current list. Remove it, and continue + * - don't advance the current list. + */ + testutil_check( + __wt_snprintf(filename, sizeof(filename), "%s/%s", dirname, prev->names[prevpos])); + VERBOSE(3, "Removing file from backup: %s\n", filename); + remove(filename); + } else { + /* + * There is something in the current list not in the prev list. Walk past it in the + * current list and try again. + */ + curpos++; + goto again; + } + } +} + +/* + * active_files_free -- + * Free the list of active files. + */ +static void +active_files_free(ACTIVE_FILES *active) +{ + uint32_t i; + + for (i = 0; i < active->count; i++) + free(active->names[i]); + free(active->names); + active_files_init(active); +} + +/* + * active_files_move -- + * Move an active file list to the destination list. + */ +static void +active_files_move(ACTIVE_FILES *dest, ACTIVE_FILES *src) +{ + active_files_free(dest); + *dest = *src; + WT_CLEAR(*src); +} + +/* + * table_changes -- + * Potentially make changes to a single table. + */ +static void +table_changes(WT_SESSION *session, TABLE *table) +{ + WT_CURSOR *cur; + WT_ITEM item, item2; + WT_MODIFY modify_entries[MAX_MODIFY_ENTRIES]; + OPERATION_TYPE op_type; + uint64_t change_count; + uint32_t i, nrecords; + int modify_count; + u_char *value, *value2; + char key[MAX_KEY_SIZE]; + + /* + * We change each table in use about half the time. + */ + if (__wt_random(&table->rand) % 2 == 0) { + value = dcalloc(1, table->max_value_size); + value2 = dcalloc(1, table->max_value_size); + nrecords = __wt_random(&table->rand) % 1000; + VERBOSE(4, "changing %d records in %s\n", (int)nrecords, table->name); + testutil_check(session->open_cursor(session, table->name, NULL, NULL, &cur)); + for (i = 0; i < nrecords; i++) { + change_count = table->change_count++; + item.data = value; + item.size = table->max_value_size; + key_value(change_count, key, sizeof(key), &item, &op_type); + cur->set_key(cur, key); + switch (op_type) { + case INSERT: + cur->set_value(cur, &item); + testutil_check(cur->insert(cur)); + break; + case MODIFY: + item2.data = value2; + item2.size = table->max_value_size; + key_value(change_count - KEYS_PER_TABLE, NULL, 0, &item2, &op_type); + modify_count = MAX_MODIFY_ENTRIES; + testutil_check(wiredtiger_calc_modify( + session, &item2, &item, MAX_MODIFY_DIFF, modify_entries, &modify_count)); + testutil_check(cur->modify(cur, modify_entries, modify_count)); + break; + case REMOVE: + testutil_check(cur->remove(cur)); + break; + case UPDATE: + cur->set_value(cur, &item); + testutil_check(cur->update(cur)); + break; + case _OPERATION_TYPE_COUNT: + testutil_assert(false); + break; + } + } + free(value); + free(value2); + testutil_check(cur->close(cur)); + } +} + +/* + * create_table -- + * Create a table for the given slot. + */ +static void +create_table(WT_SESSION *session, TABLE_INFO *tinfo, uint32_t slot) +{ + char *uri; + + testutil_assert(!TABLE_VALID(&tinfo->table[slot])); + uri = dcalloc(1, URI_MAX_LEN); + testutil_check( + __wt_snprintf(uri, URI_MAX_LEN, URI_FORMAT, (int)slot, (int)tinfo->table[slot].name_index++)); + + VERBOSE(3, "create %s\n", uri); + testutil_check(session->create(session, uri, "key_format=S,value_format=u")); + tinfo->table[slot].name = uri; + tinfo->tables_in_use++; +} + +static void +rename_table(WT_SESSION *session, TABLE_INFO *tinfo, uint32_t slot) +{ + char *olduri, *uri; + + testutil_assert(TABLE_VALID(&tinfo->table[slot])); + uri = dcalloc(1, URI_MAX_LEN); + testutil_check( + __wt_snprintf(uri, URI_MAX_LEN, URI_FORMAT, (int)slot, (int)tinfo->table[slot].name_index++)); + + olduri = tinfo->table[slot].name; + VERBOSE(3, "rename %s %s\n", olduri, uri); + testutil_check(session->rename(session, olduri, uri, NULL)); + free(olduri); + tinfo->table[slot].name = uri; +} + +static void +drop_table(WT_SESSION *session, TABLE_INFO *tinfo, uint32_t slot) +{ + char *uri; + + testutil_assert(TABLE_VALID(&tinfo->table[slot])); + uri = tinfo->table[slot].name; + + VERBOSE(3, "drop %s\n", uri); + testutil_check(session->drop(session, uri, NULL)); + free(uri); + tinfo->table[slot].name = NULL; + tinfo->table[slot].change_count = 0; + tinfo->tables_in_use--; +} + +/* + * tables_free -- + * Free the list of tables. + */ +static void +tables_free(TABLE_INFO *tinfo) +{ + uint32_t slot; + + for (slot = 0; slot < tinfo->table_count; slot++) { + if (tinfo->table[slot].name != NULL) { + free(tinfo->table[slot].name); + tinfo->table[slot].name = NULL; + } + } + free(tinfo->table); + tinfo->table = NULL; +} + +static void +base_backup(WT_CONNECTION *conn, WT_RAND_STATE *rand, const char *home, const char *backup_home, + TABLE_INFO *tinfo, ACTIVE_FILES *active) +{ + WT_CURSOR *cursor; + WT_SESSION *session; + uint32_t granularity; + int nfiles, ret; + char buf[4096]; + char *filename; + + nfiles = 0; + + VERBOSE(2, "BASE BACKUP: %s\n", backup_home); + active_files_free(active); + active_files_init(active); + testutil_check( + __wt_snprintf(buf, sizeof(buf), "rm -rf %s && mkdir %s", backup_home, backup_home)); + VERBOSE(3, " => %s\n", buf); + testutil_check(system(buf)); + + testutil_check(conn->open_session(conn, NULL, NULL, &session)); + tinfo->full_backup_number = tinfo->incr_backup_number++; + + /* Half of the runs with a low granularity: 1M */ + if (__wt_random(rand) % 2 == 0) + granularity = 1; + else + granularity = 1 + __wt_random(rand) % 20; + testutil_check(__wt_snprintf(buf, sizeof(buf), + "incremental=(granularity=%" PRIu32 "M,enabled=true,this_id=ID%d)", granularity, + (int)tinfo->full_backup_number)); + VERBOSE(3, "open_cursor(session, \"backup:\", NULL, \"%s\", &cursor)\n", buf); + testutil_check(session->open_cursor(session, "backup:", NULL, buf, &cursor)); + + while ((ret = cursor->next(cursor)) == 0) { + nfiles++; + testutil_check(cursor->get_key(cursor, &filename)); + active_files_add(active, filename); + testutil_check( + __wt_snprintf(buf, sizeof(buf), "cp %s/%s %s/%s", home, filename, backup_home, filename)); + VERBOSE(3, " => %s\n", buf); + testutil_check(system(buf)); + } + testutil_assert(ret == WT_NOTFOUND); + testutil_check(cursor->close(cursor)); + testutil_check(session->close(session, NULL)); + active_files_sort(active); + VERBOSE(2, " finished base backup: %d files\n", nfiles); +} + +/* + * Open a file if it isn't already open. The "memory" of the open file name is kept in the buffer + * passed in. + */ +static void +reopen_file(int *fdp, char *buf, size_t buflen, const char *filename, int oflag) +{ + /* Do we already have this file open? */ + if (strcmp(buf, filename) == 0 && *fdp != -1) + return; + if (*fdp != -1) + close(*fdp); + *fdp = open(filename, oflag, 0666); + strncpy(buf, filename, buflen); + testutil_assert(*fdp >= 0); +} + +/* + * Perform an incremental backup into an existing backup directory. + */ +static void +incr_backup(WT_CONNECTION *conn, const char *home, const char *backup_home, TABLE_INFO *tinfo, + ACTIVE_FILES *master_active) +{ + ACTIVE_FILES active; + WT_CURSOR *cursor, *file_cursor; + WT_SESSION *session; + void *tmp; + ssize_t rdsize; + uint64_t offset, size, type; + int rfd, ret, wfd, nfiles, nrange, ncopy; + char buf[4096], rbuf[4096], wbuf[4096]; + char *filename; + + VERBOSE(2, "INCREMENTAL BACKUP: %s\n", backup_home); + active_files_print(master_active, "master list before incremental backup"); + WT_CLEAR(rbuf); + WT_CLEAR(wbuf); + rfd = wfd = -1; + nfiles = nrange = ncopy = 0; + + active_files_init(&active); + testutil_check(conn->open_session(conn, NULL, NULL, &session)); + testutil_check(__wt_snprintf(buf, sizeof(buf), "incremental=(src_id=ID%d,this_id=ID%d)", + (int)tinfo->full_backup_number, (int)tinfo->incr_backup_number++)); + VERBOSE(3, "open_cursor(session, \"backup:\", NULL, \"%s\", &cursor)\n", buf); + testutil_check(session->open_cursor(session, "backup:", NULL, buf, &cursor)); + + while ((ret = cursor->next(cursor)) == 0) { + nfiles++; + testutil_check(cursor->get_key(cursor, &filename)); + active_files_add(&active, filename); + if (slow_incremental) { + /* + * The "slow" version of an incremental backup is to copy the entire file that was + * indicated to be changed. This may be useful for debugging problems that occur in + * backup. This path is typically disabled for the test program. + */ + testutil_check(__wt_snprintf( + buf, sizeof(buf), "cp %s/%s %s/%s", home, filename, backup_home, filename)); + VERBOSE(3, " => %s\n", buf); + testutil_check(system(buf)); + } else { + /* + * Here is the normal incremental backup. Now that we know what file has changed, we get + * the specific changes + */ + testutil_check(__wt_snprintf(buf, sizeof(buf), "incremental=(file=%s)", filename)); + testutil_check(session->open_cursor(session, NULL, cursor, buf, &file_cursor)); + VERBOSE(3, "open_cursor(session, NULL, cursor, \"%s\", &file_cursor)\n", buf); + while ((ret = file_cursor->next(file_cursor)) == 0) { + error_check(file_cursor->get_key(file_cursor, &offset, &size, &type)); + testutil_assert(type == WT_BACKUP_FILE || type == WT_BACKUP_RANGE); + if (type == WT_BACKUP_RANGE) { + nrange++; + tmp = dcalloc(1, size); + + testutil_check(__wt_snprintf(buf, sizeof(buf), "%s/%s", home, filename)); + VERBOSE(5, "Reopen read file: %s\n", buf); + reopen_file(&rfd, rbuf, sizeof(rbuf), buf, O_RDONLY); + rdsize = pread(rfd, tmp, (size_t)size, (wt_off_t)offset); + testutil_assert(rdsize >= 0); + + testutil_check(__wt_snprintf(buf, sizeof(buf), "%s/%s", backup_home, filename)); + VERBOSE(5, "Reopen write file: %s\n", buf); + reopen_file(&wfd, wbuf, sizeof(wbuf), buf, O_WRONLY | O_CREAT); + /* Use the read size since we may have read less than the granularity. */ + testutil_assert(pwrite(wfd, tmp, (size_t)rdsize, (wt_off_t)offset) == rdsize); + free(tmp); + } else { + ncopy++; + testutil_check(__wt_snprintf( + buf, sizeof(buf), "cp %s/%s %s/%s", home, filename, backup_home, filename)); + VERBOSE(3, " => %s\n", buf); + testutil_check(system(buf)); + } + } + testutil_assert(ret == WT_NOTFOUND); + testutil_check(file_cursor->close(file_cursor)); + } + } + testutil_assert(ret == WT_NOTFOUND); + if (rfd != -1) + testutil_check(close(rfd)); + if (wfd != -1) + testutil_check(close(wfd)); + testutil_check(cursor->close(cursor)); + testutil_check(session->close(session, NULL)); + VERBOSE(2, " finished incremental backup: %d files, %d range copy, %d file copy\n", nfiles, + nrange, ncopy); + active_files_sort(&active); + active_files_remove_missing(master_active, &active, backup_home); + + /* Move the current active list to the master list */ + active_files_move(master_active, &active); +} + +static void +check_table(WT_SESSION *session, TABLE *table) +{ + WT_CURSOR *cursor; + WT_ITEM item, got_value; + OPERATION_TYPE op_type; + uint64_t boundary, change_count, expect_records, got_records, total_changes; + int keylow, keyhigh, ret; + u_char *value; + char *got_key; + char key[MAX_KEY_SIZE]; + + expect_records = 0; + total_changes = table->change_count; + boundary = total_changes % KEYS_PER_TABLE; + op_type = (OPERATION_TYPE)(total_changes % CHANGES_PER_CYCLE) / KEYS_PER_TABLE; + value = dcalloc(1, table->max_value_size); + + VERBOSE(3, "Checking: %s\n", table->name); + switch (op_type) { + case INSERT: + expect_records = total_changes % KEYS_PER_TABLE; + break; + case MODIFY: + case UPDATE: + expect_records = KEYS_PER_TABLE; + break; + case REMOVE: + expect_records = KEYS_PER_TABLE - (total_changes % KEYS_PER_TABLE); + break; + case _OPERATION_TYPE_COUNT: + testutil_assert(false); + break; + } + + testutil_check(session->open_cursor(session, table->name, NULL, NULL, &cursor)); + got_records = 0; + while ((ret = cursor->next(cursor)) == 0) { + got_records++; + testutil_check(cursor->get_key(cursor, &got_key)); + testutil_check(cursor->get_value(cursor, &got_value)); + + /* + * Reconstruct the change number from the key. See key_value() for details on how the key is + * constructed. + */ + testutil_assert(sscanf(got_key, KEY_FORMAT, &keylow, &keyhigh) == 2); + change_count = (u_int)keyhigh * 100 + (u_int)keylow; + item.data = value; + item.size = table->max_value_size; + if (op_type == INSERT || (op_type == UPDATE && change_count < boundary)) + change_count += 0; + else if (op_type == UPDATE || (op_type == MODIFY && change_count < boundary)) + change_count += KEYS_PER_TABLE; + else if (op_type == MODIFY || (op_type == REMOVE && change_count < boundary)) + change_count += 20000; + else + testutil_assert(false); + key_value(change_count, key, sizeof(key), &item, &op_type); + testutil_assert(strcmp(key, got_key) == 0); + testutil_assert(got_value.size == item.size); + testutil_assert(memcmp(got_value.data, item.data, item.size) == 0); + } + testutil_assert(got_records == expect_records); + testutil_assert(ret == WT_NOTFOUND); + testutil_check(cursor->close(cursor)); + free(value); +} + +/* + * Verify the backup to make sure the proper tables exist and have the correct content. + */ +static void +check_backup(const char *backup_home, const char *backup_check, TABLE_INFO *tinfo) +{ + WT_CONNECTION *conn; + WT_SESSION *session; + uint32_t slot; + char buf[4096]; + + VERBOSE( + 2, "CHECK BACKUP: copy %s to %s, then check %s\n", backup_home, backup_check, backup_check); + + testutil_check(__wt_snprintf( + buf, sizeof(buf), "rm -rf %s && cp -r %s %s", backup_check, backup_home, backup_check)); + testutil_check(system(buf)); + + testutil_check(wiredtiger_open(backup_check, NULL, NULL, &conn)); + testutil_check(conn->open_session(conn, NULL, NULL, &session)); + + for (slot = 0; slot < tinfo->table_count; slot++) { + if (TABLE_VALID(&tinfo->table[slot])) + check_table(session, &tinfo->table[slot]); + } + + testutil_check(session->close(session, NULL)); + testutil_check(conn->close(conn, NULL)); +} + +int +main(int argc, char *argv[]) +{ + ACTIVE_FILES active; + TABLE_INFO tinfo; + WT_CONNECTION *conn; + WT_RAND_STATE rnd; + WT_SESSION *session; + uint32_t file_max, iter, max_value_size, next_checkpoint, rough_size, slot; + int ch, ncheckpoints, status; + const char *backup_verbose, *working_dir; + char conf[1024], home[1024], backup_check[1024], backup_dir[1024], command[4096]; + + ncheckpoints = 0; + (void)testutil_set_progname(argv); + custom_die = die; /* Set our own abort handler */ + WT_CLEAR(tinfo); + active_files_init(&active); + + working_dir = "WT_TEST.incr_backup"; + + while ((ch = __wt_getopt(progname, argc, argv, "h:S:v:")) != EOF) + switch (ch) { + case 'h': + working_dir = __wt_optarg; + break; + case 'S': + seed = (uint64_t)atoll(__wt_optarg); + break; + case 'v': + verbose_level = atoi(__wt_optarg); + break; + default: + usage(); + } + argc -= __wt_optind; + if (argc != 0) + usage(); + + if (seed == 0) { + __wt_random_init_seed(NULL, &rnd); + seed = rnd.v; + } else + rnd.v = seed; + + testutil_work_dir_from_path(home, sizeof(home), working_dir); + testutil_check(__wt_snprintf(backup_dir, sizeof(backup_dir), "%s.BACKUP", home)); + testutil_check(__wt_snprintf(backup_check, sizeof(backup_check), "%s.CHECK", home)); + fprintf(stderr, "Seed: %" PRIu64 "\n", seed); + + testutil_check( + __wt_snprintf(command, sizeof(command), "rm -rf %s %s; mkdir %s", home, backup_dir, home)); + if ((status = system(command)) < 0) + testutil_die(status, "system: %s", command); + + backup_verbose = (verbose_level >= 4) ? "verbose=(backup)" : ""; + + /* + * We create an overall max_value_size. From that, we'll set a random max_value_size per table. + * In addition, individual values put into each table vary randomly in size, up to the + * max_value_size of the table. + * This tends to make sure that 1) each table has a "personality" of size ranges within it + * 2) there are some runs that tend to have a lot more data than other runs. If we made every + * insert choose a uniform random size between 1 and MAX_VALUE_SIZE, once we did a bunch + * of inserts, each run would look very much the same with respect to value size. + */ + max_value_size = __wt_random(&rnd) % MAX_VALUE_SIZE; + + /* Compute a random value of file_max. */ + rough_size = __wt_random(&rnd) % 3; + if (rough_size == 0) + file_max = 100 + __wt_random(&rnd) % 100; /* small log files, min 100K */ + else if (rough_size == 1) + file_max = 200 + __wt_random(&rnd) % 1000; /* 200K to ~1M */ + else + file_max = 1000 + __wt_random(&rnd) % 20000; /* 1M to ~20M */ + testutil_check(__wt_snprintf(conf, sizeof(conf), + "create,%s,log=(enabled=true,file_max=%" PRIu32 "K)", backup_verbose, file_max)); + VERBOSE(2, "wiredtiger config: %s\n", conf); + testutil_check(wiredtiger_open(home, NULL, conf, &conn)); + testutil_check(conn->open_session(conn, NULL, NULL, &session)); + + tinfo.table_count = __wt_random(&rnd) % MAX_NTABLES + 1; + tinfo.table = dcalloc(tinfo.table_count, sizeof(tinfo.table[0])); + + /* + * Give each table its own random generator. This makes it easier to simplify a failing test to + * use fewer tables, but have those just tables behave the same. + */ + for (slot = 0; slot < tinfo.table_count; slot++) { + tinfo.table[slot].rand.v = seed + slot; + testutil_assert(!TABLE_VALID(&tinfo.table[slot])); + tinfo.table[slot].max_value_size = __wt_random(&rnd) % (max_value_size + 1); + } + + /* How many files should we update until next checkpoint. */ + next_checkpoint = __wt_random(&rnd) % tinfo.table_count; + + for (iter = 0; iter < ITERATIONS; iter++) { + VERBOSE(1, "**** iteration %d ****\n", (int)iter); + + /* + * We have schema changes during about half the iterations. The number of schema changes + * varies, averaging 10. + */ + if (tinfo.tables_in_use == 0 || __wt_random(&rnd) % 2 != 0) { + while (__wt_random(&rnd) % 10 != 0) { + /* + * For schema events, we choose to create, rename or drop tables. We pick a random + * slot, and if it is empty, create a table there. Otherwise, we rename or drop. + * That should give us a steady state with slots mostly filled. + */ + slot = __wt_random(&rnd) % tinfo.table_count; + if (!TABLE_VALID(&tinfo.table[slot])) + create_table(session, &tinfo, slot); + else if (__wt_random(&rnd) % 3 == 0 && do_rename) + rename_table(session, &tinfo, slot); + else if (do_drop) + drop_table(session, &tinfo, slot); + } + } + for (slot = 0; slot < tinfo.table_count; slot++) { + if (TABLE_VALID(&tinfo.table[slot])) + table_changes(session, &tinfo.table[slot]); + if (next_checkpoint-- == 0) { + VERBOSE(2, "Checkpoint %d\n", ncheckpoints); + testutil_check(session->checkpoint(session, NULL)); + next_checkpoint = __wt_random(&rnd) % tinfo.table_count; + ncheckpoints++; + } + } + + if (iter == 0) { + base_backup(conn, &rnd, home, backup_dir, &tinfo, &active); + check_backup(backup_dir, backup_check, &tinfo); + } else { + incr_backup(conn, home, backup_dir, &tinfo, &active); + check_backup(backup_dir, backup_check, &tinfo); + if (__wt_random(&rnd) % 10 == 0) { + base_backup(conn, &rnd, home, backup_dir, &tinfo, &active); + check_backup(backup_dir, backup_check, &tinfo); + } + } + } + testutil_check(session->close(session, NULL)); + testutil_check(conn->close(conn, NULL)); + active_files_free(&active); + tables_free(&tinfo); + + printf("Success.\n"); + return (0); +} diff --git a/src/third_party/wiredtiger/test/csuite/incr_backup/smoke.sh b/src/third_party/wiredtiger/test/csuite/incr_backup/smoke.sh new file mode 100755 index 00000000000..65727df015e --- /dev/null +++ b/src/third_party/wiredtiger/test/csuite/incr_backup/smoke.sh @@ -0,0 +1,12 @@ +#! /bin/sh + +set -e + +# Smoke-test incr-backup as part of running "make check". + +# If $top_builddir/$top_srcdir aren't set, default to building in build_posix +# and running in test/csuite. +top_builddir=${top_builddir:-../../build_posix} +top_srcdir=${top_srcdir:-../..} + +$TEST_WRAPPER $top_builddir/test/csuite/test_incr_backup -v 3 diff --git a/src/third_party/wiredtiger/test/csuite/wt4105_large_doc_small_upd/main.c b/src/third_party/wiredtiger/test/csuite/wt4105_large_doc_small_upd/main.c index ba91ffed0a9..3483b047fed 100644 --- a/src/third_party/wiredtiger/test/csuite/wt4105_large_doc_small_upd/main.c +++ b/src/third_party/wiredtiger/test/csuite/wt4105_large_doc_small_upd/main.c @@ -132,7 +132,8 @@ main(int argc, char *argv[]) modify_entry.data.size = strlen(modify_entry.data.data); modify_entry.offset = offset; modify_entry.size = modify_entry.data.size; - (void)alarm(1); + /* FIXME-PM-1521: extend timeout to pass the test */ + (void)alarm(7); testutil_check(c->modify(c, &modify_entry, 1)); (void)alarm(0); testutil_check(session2->commit_transaction(session2, NULL)); diff --git a/src/third_party/wiredtiger/test/csuite/wt4803_cache_overflow_abort/main.c b/src/third_party/wiredtiger/test/csuite/wt4803_history_store_abort/main.c index f2e4c302a40..692b0bac114 100644 --- a/src/third_party/wiredtiger/test/csuite/wt4803_cache_overflow_abort/main.c +++ b/src/third_party/wiredtiger/test/csuite/wt4803_history_store_abort/main.c @@ -32,15 +32,15 @@ /* * JIRA ticket reference: WT-4803 Test case description: This test is checking the functionality of - * the lookaside file_max configuration. When the size of the lookaside file exceeds this value, we - * expect to panic. Failure mode: If we receive a panic in the test cases we weren't expecting to - * and vice versa. + * the history store file_max configuration. When the size of the history store file exceeds this + * value, we expect to panic. Failure mode: If we receive a panic in the test cases we weren't + * expecting to and vice versa. */ #define NUM_KEYS 2000 /* - * This is a global flag that should be set before running test_las_workload. It lets the child + * This is a global flag that should be set before running test_hs_workload. It lets the child * process know whether it should be expecting a panic or not so that it can adjust its exit code as * needed. */ @@ -56,7 +56,7 @@ handle_message(WT_EVENT_HANDLER *handler, WT_SESSION *session, int error, const if (error == WT_PANIC && strstr(message, "exceeds maximum size") != NULL) { fprintf( - stderr, "Got cache overflow error (expect_panic=%s)\n", expect_panic ? "true" : "false"); + stderr, "Got history store error (expect_panic=%s)\n", expect_panic ? "true" : "false"); /* * If we're expecting a panic, exit with zero to indicate to the parent that this test was @@ -75,7 +75,7 @@ handle_message(WT_EVENT_HANDLER *handler, WT_SESSION *session, int error, const static WT_EVENT_HANDLER event_handler = {handle_message, NULL, NULL, NULL}; static void -las_workload(TEST_OPTS *opts, const char *las_file_max) +hs_workload(TEST_OPTS *opts, const char *hs_file_max) { WT_CURSOR *cursor; WT_SESSION *other_session, *session; @@ -83,7 +83,7 @@ las_workload(TEST_OPTS *opts, const char *las_file_max) char buf[WT_MEGABYTE], open_config[128]; testutil_check(__wt_snprintf(open_config, sizeof(open_config), - "create,cache_size=50MB,cache_overflow=(file_max=%s)", las_file_max)); + "create,cache_size=50MB,history_store=(file_max=%s)", hs_file_max)); testutil_check(wiredtiger_open(opts->home, &event_handler, open_config, &opts->conn)); testutil_check(opts->conn->open_session(opts->conn, NULL, NULL, &session)); @@ -104,7 +104,7 @@ las_workload(TEST_OPTS *opts, const char *las_file_max) * Open a snapshot isolation transaction in another session. This forces the cache to retain all * previous values. Then update all keys with a new value in the original session while keeping * that snapshot transaction open. With the large value buffer, small cache and lots of keys, - * this will force a lot of lookaside usage. + * this will force a lot of history store usage. * * When the file_max setting is small, the maximum size should easily be reached and we should * panic. When the maximum size is large or not set, then we should succeed. @@ -133,7 +133,7 @@ las_workload(TEST_OPTS *opts, const char *las_file_max) } static int -test_las_workload(TEST_OPTS *opts, const char *las_file_max) +test_hs_workload(TEST_OPTS *opts, const char *hs_file_max) { pid_t pid; int status; @@ -157,7 +157,7 @@ test_las_workload(TEST_OPTS *opts, const char *las_file_max) testutil_die(errno, "fork"); else if (pid == 0) { /* Child process from here. */ - las_workload(opts, las_file_max); + hs_workload(opts, hs_file_max); /* * If we're expecting a panic during the workload, we shouldn't get to this point. Exit with @@ -188,24 +188,25 @@ main(int argc, char **argv) testutil_check(testutil_parse_opts(argc, argv, &opts)); /* - * The lookaside is unbounded. We don't expect any failure since we can use as much as needed. + * The history store is unbounded. We don't expect any failure since we can use as much as + * needed. */ expect_panic = false; - testutil_check(test_las_workload(&opts, "0")); + testutil_check(test_hs_workload(&opts, "0")); /* - * The lookaside is limited to 5GB. This is more than enough for this workload so we don't + * The history store is limited to 5GB. This is more than enough for this workload so we don't * expect any failure. */ expect_panic = false; - testutil_check(test_las_workload(&opts, "5GB")); + testutil_check(test_hs_workload(&opts, "5GB")); /* - * The lookaside is limited to 100MB. This is insufficient for this workload so we're expecting - * a failure. + * The history store is limited to 100MB. This is insufficient for this workload so we're + * expecting a failure. */ expect_panic = true; - testutil_check(test_las_workload(&opts, "100MB")); + testutil_check(test_hs_workload(&opts, "100MB")); testutil_cleanup(&opts); diff --git a/src/third_party/wiredtiger/test/evergreen.yml b/src/third_party/wiredtiger/test/evergreen.yml index 628dc815785..1db2b29cb94 100755 --- a/src/third_party/wiredtiger/test/evergreen.yml +++ b/src/third_party/wiredtiger/test/evergreen.yml @@ -270,14 +270,15 @@ functions: rm -rf "wiredtiger" rm -rf "wiredtiger.tgz" - "checkpoint test": - command: shell.exec - params: - working_dir: "wiredtiger/build_posix/test/checkpoint" - script: | - set -o errexit - set -o verbose - ./t ${checkpoint_args} 2>&1 + # Temporarily disabled + # "checkpoint test": + # command: shell.exec + # params: + # working_dir: "wiredtiger/build_posix/test/checkpoint" + # script: | + # set -o errexit + # set -o verbose + # ./t ${checkpoint_args} 2>&1 "checkpoint stress test": command: shell.exec @@ -553,16 +554,17 @@ tasks: vars: directory: test/bloom - - name: checkpoint-test - tags: ["pull_request"] - depends_on: - - name: compile - commands: - - func: "fetch artifacts" - - func: "compile wiredtiger" - - func: "make check directory" - vars: - directory: test/checkpoint + # Temporarily disabled + # - name: checkpoint-test + # tags: ["pull_request"] + # depends_on: + # - name: compile + # commands: + # - func: "fetch artifacts" + # - func: "compile wiredtiger" + # - func: "make check directory" + # vars: + # directory: test/checkpoint - name: cursor-order-test tags: ["pull_request"] @@ -575,27 +577,29 @@ tasks: vars: directory: test/cursor_order - - name: fops-test - tags: ["pull_request"] - depends_on: - - name: compile - commands: - - func: "fetch artifacts" - - func: "compile wiredtiger" - - func: "make check directory" - vars: - directory: test/fops - - - name: format-test - tags: ["pull_request"] - depends_on: - - name: compile - commands: - - func: "fetch artifacts" - - func: "compile wiredtiger" - - func: "make check directory" - vars: - directory: test/format + # Temporarily disabled + # - name: fops-test + # tags: ["pull_request"] + # depends_on: + # - name: compile + # commands: + # - func: "fetch artifacts" + # - func: "compile wiredtiger" + # - func: "make check directory" + # vars: + # directory: test/fops + + # Temporarily disabled + # - name: format-test + # tags: ["pull_request"] + # depends_on: + # - name: compile + # commands: + # - func: "fetch artifacts" + # - func: "compile wiredtiger" + # - func: "make check directory" + # vars: + # directory: test/format - name: huge-test tags: ["pull_request"] @@ -695,7 +699,23 @@ tasks: # Start of csuite test tasks - - name: csuite-import-test + # Temporarily disabled + # - name: csuite-import-test + # tags: ["pull_request"] + # depends_on: + # - name: compile + # commands: + # - func: "fetch artifacts" + # - command: shell.exec + # params: + # working_dir: "wiredtiger/build_posix" + # script: | + # set -o errexit + # set -o verbose + + # ${test_env_vars|} $(pwd)/../test/csuite/import/smoke.sh 2>&1 + + - name: csuite-incr-backup-test tags: ["pull_request"] depends_on: - name: compile @@ -708,7 +728,7 @@ tasks: set -o errexit set -o verbose - ${test_env_vars|} $(pwd)/../test/csuite/import/smoke.sh 2>&1 + ${test_env_vars|} $(pwd)/test/csuite/test_incr_backup 2>&1 - name: csuite-random-abort-test tags: ["pull_request"] @@ -1033,7 +1053,7 @@ tasks: ${test_env_vars|} $(pwd)/test/csuite/test_wt4699_json 2>&1 - - name: csuite-wt4803-cache-overflow-abort-test + - name: csuite-wt4803-history-store-abort-test tags: ["pull_request"] depends_on: - name: compile @@ -1046,7 +1066,7 @@ tasks: set -o errexit set -o verbose - ${test_env_vars|} $(pwd)/test/csuite/test_wt4803_cache_overflow_abort 2>&1 + ${test_env_vars|} $(pwd)/test/csuite/test_wt4803_history_store_abort 2>&1 - name: csuite-wt4891-meta-ckptlist-get-alloc-test tags: ["pull_request"] @@ -1183,20 +1203,20 @@ tasks: ${test_env_vars|} $(pwd)/test/csuite/test_wt3338_partial_update 2>&1 - - name: csuite-wt4333-handle-locks-test - tags: ["pull_request"] - depends_on: - - name: compile - commands: - - func: "fetch artifacts" - - command: shell.exec - params: - working_dir: "wiredtiger/build_posix" - script: | - set -o errexit - set -o verbose - - ${test_env_vars|} $(pwd)/test/csuite/test_wt4333_handle_locks 2>&1 + # - name: csuite-wt4333-handle-locks-test + # tags: ["pull_request"] + # depends_on: + # - name: compile + # commands: + # - func: "fetch artifacts" + # - command: shell.exec + # params: + # working_dir: "wiredtiger/build_posix" + # script: | + # set -o errexit + # set -o verbose + + # ${test_env_vars|} $(pwd)/test/csuite/test_wt4333_handle_locks 2>&1 # End of csuite test tasks @@ -1295,8 +1315,7 @@ tasks: set -o errexit set -o verbose - ${test_env_vars|} ${python_binary|python} ../test/suite/run.py [defghijk] ${unit_test_args|-v 2} ${smp_command|} 2>&1 - + ${test_env_vars|} ${python_binary|python} ../test/suite/run.py [defg] ${unit_test_args|-v 2} ${smp_command|} 2>&1 - name: unit-test-bucket04 tags: ["pull_request", "unit_test"] depends_on: @@ -1310,7 +1329,9 @@ tasks: set -o errexit set -o verbose - ${test_env_vars|} ${python_binary|python} ../test/suite/run.py [lmnopq] ${unit_test_args|-v 2} ${smp_command|} 2>&1 + # Reserve this bucket only for history store tests, which take a long time to run + ${test_env_vars|} ${python_binary|python} ../test/suite/run.py hs ${unit_test_args|-v 2} ${smp_command|} 2>&1 + - name: unit-test-bucket05 tags: ["pull_request", "unit_test"] @@ -1325,7 +1346,10 @@ tasks: set -o errexit set -o verbose - ${test_env_vars|} ${python_binary|python} ../test/suite/run.py [rs] ${unit_test_args|-v 2} ${smp_command|} 2>&1 + # Non-history store tests in the 'h' family + non_ts_tests=$(ls ../test/suite/test_h*.py | xargs -n1 basename | grep -v _hs) + ${test_env_vars|} ${python_binary|python} ../test/suite/run.py $non_ts_tests ${unit_test_args|-v 2} ${smp_command|} 2>&1 + ${test_env_vars|} ${python_binary|python} ../test/suite/run.py [ijk] ${unit_test_args|-v 2} ${smp_command|} 2>&1 - name: unit-test-bucket06 tags: ["pull_request", "unit_test"] @@ -1340,10 +1364,40 @@ tasks: set -o errexit set -o verbose + ${test_env_vars|} ${python_binary|python} ../test/suite/run.py [lmnopq] ${unit_test_args|-v 2} ${smp_command|} 2>&1 + + - name: unit-test-bucket07 + tags: ["pull_request", "unit_test"] + depends_on: + - name: compile + commands: + - func: "fetch artifacts" + - command: shell.exec + params: + working_dir: "wiredtiger/build_posix" + script: | + set -o errexit + set -o verbose + + ${test_env_vars|} ${python_binary|python} ../test/suite/run.py [rs] ${unit_test_args|-v 2} ${smp_command|} 2>&1 + + - name: unit-test-bucket08 + tags: ["pull_request", "unit_test"] + depends_on: + - name: compile + commands: + - func: "fetch artifacts" + - command: shell.exec + params: + working_dir: "wiredtiger/build_posix" + script: | + set -o errexit + set -o verbose + # Reserve this bucket only for timestamp tests, which take a long time to run ${test_env_vars|} ${python_binary|python} ../test/suite/run.py timestamp ${unit_test_args|-v 2} ${smp_command|} 2>&1 - - name: unit-test-bucket07 + - name: unit-test-bucket09 tags: ["pull_request", "unit_test"] depends_on: - name: compile @@ -1359,6 +1413,20 @@ tasks: # Non-timestamp tests in the 't' family non_ts_tests=$(ls ../test/suite/test_t*.py | xargs -n1 basename | grep -v timestamp) ${test_env_vars|} ${python_binary|python} ../test/suite/run.py $non_ts_tests ${unit_test_args|-v 2} ${smp_command|} 2>&1 + + - name: unit-test-bucket10 + tags: ["pull_request", "unit_test"] + depends_on: + - name: compile + commands: + - func: "fetch artifacts" + - command: shell.exec + params: + working_dir: "wiredtiger/build_posix" + script: | + set -o errexit + set -o verbose + ${test_env_vars|} ${python_binary|python} ../test/suite/run.py [uvwxyz] ${unit_test_args|-v 2} ${smp_command|} 2>&1 # End of Python unit test tasks @@ -1389,7 +1457,7 @@ tasks: script: | set -o errexit set -o verbose - + ${test_env_vars|} ${python_binary|python} ../../test/wtperf/test_conf_dump.py 2>&1 - name: compile-windows-alt @@ -1424,38 +1492,40 @@ tasks: pip install scons==3.1.1 scons-3.1.1.bat ${smp_command|} check - - name: fops - tags: ["pull_request"] - depends_on: - - name: compile - commands: - - func: "fetch artifacts" - - command: shell.exec - params: - working_dir: "wiredtiger" - script: | - set -o errexit - set -o verbose - if [ "Windows_NT" = "$OS" ]; then - cmd.exe /c t_fops.exe - else - build_posix/test/fops/t - fi - - - name: format - tags: ["windows_only"] - depends_on: - - name: compile - commands: - - func: "fetch artifacts" - - command: shell.exec - params: - working_dir: "wiredtiger" - script: | - set -o errexit - set -o verbose - # format assumes we run it from the format directory - cmd.exe /c "cd test\\format && ..\\..\\t_format.exe reverse=0 encryption=none logging_compression=none runs=20" + # Temporarily disabled + # - name: fops + # tags: ["pull_request"] + # depends_on: + # - name: compile + # commands: + # - func: "fetch artifacts" + # - command: shell.exec + # params: + # working_dir: "wiredtiger" + # script: | + # set -o errexit + # set -o verbose + # if [ "Windows_NT" = "$OS" ]; then + # cmd.exe /c t_fops.exe + # else + # build_posix/test/fops/t + # fi + + # Temporarily disabled + # - name: format + # tags: ["windows_only"] + # depends_on: + # - name: compile + # commands: + # - func: "fetch artifacts" + # - command: shell.exec + # params: + # working_dir: "wiredtiger" + # script: | + # set -o errexit + # set -o verbose + # # format assumes we run it from the format directory + # cmd.exe /c "cd test\\format && ..\\..\\t_format.exe reverse=0 encryption=none logging_compression=none runs=20" - name: million-collection-test commands: @@ -1483,129 +1553,135 @@ tasks: set -o verbose test/evergreen/compatibility_test_for_mongodb_releases.sh - - name: generate-datafile-little-endian - depends_on: - - name: compile - commands: - - func: "fetch artifacts" - - func: "compile wiredtiger" - - func: "format test" - vars: - times: 10 - config: ../../../test/format/CONFIG.endian - extra_args: -h "WT_TEST.$i" - - command: shell.exec - params: - working_dir: "wiredtiger/build_posix/test/format" - shell: bash - script: | - set -o errexit - set -o verbose - # Archive the WT_TEST directories which include the generated wt data files - tar -zcvf WT_TEST.tgz WT_TEST* - - command: s3.put - params: - aws_secret: ${aws_secret} - aws_key: ${aws_key} - local_file: wiredtiger/build_posix/test/format/WT_TEST.tgz - bucket: build_external - permissions: public-read - content_type: application/tar - display_name: WT_TEST - remote_file: wiredtiger/little-endian/${revision}/artifacts/WT_TEST.tgz - - - name: verify-datafile-little-endian - depends_on: - - name: compile - - name: generate-datafile-little-endian - commands: - - func: "fetch artifacts" - - func: "fetch artifacts from little-endian" - - command: shell.exec - params: - working_dir: "wiredtiger" - script: | - set -o errexit - set -o verbose - ./test/evergreen/verify_wt_datafiles.sh 2>&1 - - - name: verify-datafile-from-little-endian - depends_on: - - name: compile - - name: generate-datafile-little-endian - variant: little-endian - commands: - - func: "fetch artifacts" - - func: "fetch artifacts from little-endian" - - command: shell.exec - params: - working_dir: "wiredtiger" - script: | - set -o errexit - set -o verbose - ./test/evergreen/verify_wt_datafiles.sh 2>&1 - - - name: generate-datafile-big-endian - depends_on: - - name: compile - commands: - - func: "fetch artifacts" - - func: "compile wiredtiger" - - func: "format test" - vars: - times: 10 - config: ../../../test/format/CONFIG.endian - extra_args: -h "WT_TEST.$i" - - command: shell.exec - params: - working_dir: "wiredtiger/build_posix/test/format" - shell: bash - script: | - set -o errexit - set -o verbose - # Archive the WT_TEST directories which include the generated wt data files - tar -zcvf WT_TEST.tgz WT_TEST* - - command: s3.put - params: - aws_secret: ${aws_secret} - aws_key: ${aws_key} - local_file: wiredtiger/build_posix/test/format/WT_TEST.tgz - bucket: build_external - permissions: public-read - content_type: application/tar - display_name: WT_TEST - remote_file: wiredtiger/big-endian/${revision}/artifacts/WT_TEST.tgz - - - name: verify-datafile-big-endian - depends_on: - - name: compile - - name: generate-datafile-big-endian - commands: - - func: "fetch artifacts" - - func: "fetch artifacts from big-endian" - - command: shell.exec - params: - working_dir: "wiredtiger" - script: | - set -o errexit - set -o verbose - ./test/evergreen/verify_wt_datafiles.sh 2>&1 - - - name: verify-datafile-from-big-endian - depends_on: - - name: compile - - name: generate-datafile-big-endian - variant: big-endian - commands: - - func: "fetch artifacts" - - func: "fetch artifacts from big-endian" - - command: shell.exec - params: - working_dir: "wiredtiger" - script: | - set -o errexit - set -o verbose - ./test/evergreen/verify_wt_datafiles.sh 2>&1 + # Temporarily disabled + # - name: generate-datafile-little-endian + # depends_on: + # - name: compile + # commands: + # - func: "fetch artifacts" + # - func: "compile wiredtiger" + # - func: "format test" + # vars: + # times: 10 + # config: ../../../test/format/CONFIG.endian + # extra_args: -h "WT_TEST.$i" + # - command: shell.exec + # params: + # working_dir: "wiredtiger/build_posix/test/format" + # shell: bash + # script: | + # set -o errexit + # set -o verbose + # # Archive the WT_TEST directories which include the generated wt data files + # tar -zcvf WT_TEST.tgz WT_TEST* + # - command: s3.put + # params: + # aws_secret: ${aws_secret} + # aws_key: ${aws_key} + # local_file: wiredtiger/build_posix/test/format/WT_TEST.tgz + # bucket: build_external + # permissions: public-read + # content_type: application/tar + # display_name: WT_TEST + # remote_file: wiredtiger/little-endian/${revision}/artifacts/WT_TEST.tgz + + # Temporarily disabled + # - name: verify-datafile-little-endian + # depends_on: + # - name: compile + # - name: generate-datafile-little-endian + # commands: + # - func: "fetch artifacts" + # - func: "fetch artifacts from little-endian" + # - command: shell.exec + # params: + # working_dir: "wiredtiger" + # script: | + # set -o errexit + # set -o verbose + # ./test/evergreen/verify_wt_datafiles.sh 2>&1 + + # Temporarily disabled + # - name: verify-datafile-from-little-endian + # depends_on: + # - name: compile + # - name: generate-datafile-little-endian + # variant: little-endian + # commands: + # - func: "fetch artifacts" + # - func: "fetch artifacts from little-endian" + # - command: shell.exec + # params: + # working_dir: "wiredtiger" + # script: | + # set -o errexit + # set -o verbose + # ./test/evergreen/verify_wt_datafiles.sh 2>&1 + + # Temporarily disabled + # - name: generate-datafile-big-endian + # depends_on: + # - name: compile + # commands: + # - func: "fetch artifacts" + # - func: "compile wiredtiger" + # - func: "format test" + # vars: + # times: 10 + # config: ../../../test/format/CONFIG.endian + # extra_args: -h "WT_TEST.$i" + # - command: shell.exec + # params: + # working_dir: "wiredtiger/build_posix/test/format" + # shell: bash + # script: | + # set -o errexit + # set -o verbose + # # Archive the WT_TEST directories which include the generated wt data files + # tar -zcvf WT_TEST.tgz WT_TEST* + # - command: s3.put + # params: + # aws_secret: ${aws_secret} + # aws_key: ${aws_key} + # local_file: wiredtiger/build_posix/test/format/WT_TEST.tgz + # bucket: build_external + # permissions: public-read + # content_type: application/tar + # display_name: WT_TEST + # remote_file: wiredtiger/big-endian/${revision}/artifacts/WT_TEST.tgz + + # Temporarily disabled + # - name: verify-datafile-big-endian + # depends_on: + # - name: compile + # - name: generate-datafile-big-endian + # commands: + # - func: "fetch artifacts" + # - func: "fetch artifacts from big-endian" + # - command: shell.exec + # params: + # working_dir: "wiredtiger" + # script: | + # set -o errexit + # set -o verbose + # ./test/evergreen/verify_wt_datafiles.sh 2>&1 + + # Temporarily disabled + # - name: verify-datafile-from-big-endian + # depends_on: + # - name: compile + # - name: generate-datafile-big-endian + # variant: big-endian + # commands: + # - func: "fetch artifacts" + # - func: "fetch artifacts from big-endian" + # - command: shell.exec + # params: + # working_dir: "wiredtiger" + # script: | + # set -o errexit + # set -o verbose + # ./test/evergreen/verify_wt_datafiles.sh 2>&1 - name: clang-analyzer tags: ["pull_request"] @@ -1653,17 +1729,31 @@ tasks: vars: format_test_script_args: -t 110 -j 4 direct_io=1 - - name: format-linux-no-ftruncate - depends_on: - - name: compile-linux-no-ftruncate - commands: - - func: "fetch artifacts" - vars: - dependent_task: compile-linux-no-ftruncate - - func: "compile wiredtiger no linux ftruncate" - - func: "format test" - vars: - times: 3 + # Temporarily disabled + # - name: linux-directio + # depends_on: + # - name: compile + # commands: + # - func: "fetch artifacts" + # - func: "compile wiredtiger" + # - func: "format test" + # vars: + # times: 3 + # config: ../../../test/format/CONFIG.stress + # extra_args: -C "direct_io=[data]" + + # Temporarily disabled + # - name: format-linux-no-ftruncate + # depends_on: + # - name: compile-linux-no-ftruncate + # commands: + # - func: "fetch artifacts" + # vars: + # dependent_task: compile-linux-no-ftruncate + # - func: "compile wiredtiger no linux ftruncate" + # - func: "format test" + # vars: + # times: 3 - name: package commands: @@ -1690,109 +1780,113 @@ tasks: set -o verbose ${python_binary|python} syscall.py --verbose --preserve - - name: checkpoint-filetypes-test - commands: - - func: "get project" - - func: "compile wiredtiger" - vars: - # Don't use diagnostic - this test looks for timing problems that are more likely to occur without it - posix_configure_flags: --enable-strict - - func: "checkpoint test" - vars: - checkpoint_args: -t m -n 1000000 -k 5000000 -C cache_size=100MB - - func: "checkpoint test" - vars: - checkpoint_args: -t r -n 1000000 -k 5000000 -C cache_size=100MB - - func: "checkpoint test" - vars: - checkpoint_args: -t c -n 1000000 -k 5000000 -C cache_size=100MB - - - name: coverage-report - commands: - - func: "get project" - - func: "compile wiredtiger" - vars: - configure_env_vars: CC=/opt/mongodbtoolchain/v3/bin/gcc CXX=/opt/mongodbtoolchain/v3/bin/g++ PATH=/opt/mongodbtoolchain/v3/bin:$PATH CFLAGS="--coverage -fPIC -ggdb" LDFLAGS=--coverage - posix_configure_flags: --enable-silent-rules --enable-diagnostic --enable-strict --enable-python --with-builtins=lz4,snappy,zlib - - func: "make check all" - - func: "unit test" - vars: - unit_test_args: -v 2 --long - - func: "format test" - vars: - extra_args: checkpoints=1 leak_memory=0 mmap=1 file_type=row compression=snappy logging=1 logging_compression=snappy logging_prealloc=1 - - func: "format test" - vars: - extra_args: checkpoints=1 leak_memory=0 mmap=1 file_type=row alter=1 backups=1 compaction=1 data_extend=1 prepare=1 rebalance=1 salvage=1 statistics=1 statistics_server=1 verify=1 - - func: "format test" - vars: - extra_args: checkpoints=1 leak_memory=0 mmap=1 file_type=row firstfit=1 internal_key_truncation=1 - - func: "format test" - vars: - extra_args: leak_memory=0 mmap=1 file_type=row checkpoints=0 in_memory=1 reverse=1 truncate=1 - - func: "format test" - vars: - extra_args: checkpoints=1 leak_memory=0 mmap=1 file_type=row compression=zlib huffman_key=1 huffman_value=1 - - func: "format test" - vars: - extra_args: checkpoints=1 leak_memory=0 mmap=1 file_type=row isolation=random transaction_timestamps=0 - - func: "format test" - vars: - extra_args: checkpoints=1 leak_memory=0 mmap=1 file_type=row data_source=lsm bloom=1 - - func: "format test" - vars: - extra_args: checkpoints=1 leak_memory=0 mmap=1 file_type=var compression=snappy checksum=uncompressed dictionary=1 repeat_data_pct=10 - - func: "format test" - vars: - extra_args: checkpoints=1 leak_memory=0 mmap=1 file_type=row compression=lz4 prefix_compression=1 leaf_page_max=9 internal_page_max=9 key_min=256 value_min=256 - - func: "format test" - vars: - extra_args: checkpoints=1 leak_memory=0 mmap=1 file_type=var leaf_page_max=9 internal_page_max=9 value_min=256 - - func: "format test" - vars: - extra_args: checkpoints=1 leak_memory=0 mmap=1 file_type=fix - - command: shell.exec - params: - working_dir: "wiredtiger/build_posix" - script: | - set -o errexit - set -o verbose - - GCOV=/opt/mongodbtoolchain/v3/bin/gcov gcovr -r .. -e '.*/bt_(debug|dump|misc|salvage|vrfy).*' -e '.*/(log|progress|verify_build|strerror|env_msg|err_file|cur_config|os_abort)\..*' -e '.*_stat\..*' --html -o ../coverage_report.html - - command: s3.put - params: - aws_secret: ${aws_secret} - aws_key: ${aws_key} - local_file: wiredtiger/coverage_report.html - bucket: build_external - permissions: public-read - content_type: text/html - display_name: Coverage report - remote_file: wiredtiger/${build_variant}/${revision}/coverage_report/coverage_report_${build_id}.html - - - name: spinlock-gcc-test - commands: - - func: "get project" - - func: "compile wiredtiger" - vars: - posix_configure_flags: --enable-python --with-spinlock=gcc --enable-strict - - func: "make check all" - - func: "format test" - vars: - times: 3 - - func: "unit test" - - - name: spinlock-pthread-adaptive-test - commands: - - func: "get project" - - func: "compile wiredtiger" - vars: - posix_configure_flags: --enable-python --with-spinlock=pthread_adaptive --enable-strict - - func: "make check all" - - func: "format test" - vars: - times: 3 - - func: "unit test" + # Temporarily disabled + # - name: checkpoint-filetypes-test + # commands: + # - func: "get project" + # - func: "compile wiredtiger" + # vars: + # # Don't use diagnostic - this test looks for timing problems that are more likely to occur without it + # posix_configure_flags: --enable-strict + # - func: "checkpoint test" + # vars: + # checkpoint_args: -t m -n 1000000 -k 5000000 -C cache_size=100MB + # - func: "checkpoint test" + # vars: + # checkpoint_args: -t r -n 1000000 -k 5000000 -C cache_size=100MB + # - func: "checkpoint test" + # vars: + # checkpoint_args: -t c -n 1000000 -k 5000000 -C cache_size=100MB + + # Temporarily disabled + # - name: coverage-report + # commands: + # - func: "get project" + # - func: "compile wiredtiger" + # vars: + # configure_env_vars: CC=/opt/mongodbtoolchain/v3/bin/gcc CXX=/opt/mongodbtoolchain/v3/bin/g++ PATH=/opt/mongodbtoolchain/v3/bin:$PATH CFLAGS="--coverage -fPIC -ggdb" LDFLAGS=--coverage + # posix_configure_flags: --enable-silent-rules --enable-diagnostic --enable-strict --enable-python --with-builtins=lz4,snappy,zlib + # - func: "make check all" + # - func: "unit test" + # vars: + # unit_test_args: -v 2 --long + # - func: "format test" + # vars: + # extra_args: checkpoints=1 leak_memory=0 mmap=1 file_type=row compression=snappy logging=1 logging_compression=snappy logging_prealloc=1 + # - func: "format test" + # vars: + # extra_args: checkpoints=1 leak_memory=0 mmap=1 file_type=row alter=1 backups=1 compaction=1 data_extend=1 prepare=1 rebalance=1 salvage=1 statistics=1 statistics_server=1 verify=1 + # - func: "format test" + # vars: + # extra_args: checkpoints=1 leak_memory=0 mmap=1 file_type=row firstfit=1 internal_key_truncation=1 + # - func: "format test" + # vars: + # extra_args: leak_memory=0 mmap=1 file_type=row checkpoints=0 in_memory=1 reverse=1 truncate=1 + # - func: "format test" + # vars: + # extra_args: checkpoints=1 leak_memory=0 mmap=1 file_type=row compression=zlib huffman_key=1 huffman_value=1 + # - func: "format test" + # vars: + # extra_args: checkpoints=1 leak_memory=0 mmap=1 file_type=row isolation=random transaction_timestamps=0 + # - func: "format test" + # vars: + # extra_args: checkpoints=1 leak_memory=0 mmap=1 file_type=row data_source=lsm bloom=1 + # - func: "format test" + # vars: + # extra_args: checkpoints=1 leak_memory=0 mmap=1 file_type=var compression=snappy checksum=uncompressed dictionary=1 repeat_data_pct=10 + # - func: "format test" + # vars: + # extra_args: checkpoints=1 leak_memory=0 mmap=1 file_type=row compression=lz4 prefix_compression=1 leaf_page_max=9 internal_page_max=9 key_min=256 value_min=256 + # - func: "format test" + # vars: + # extra_args: checkpoints=1 leak_memory=0 mmap=1 file_type=var leaf_page_max=9 internal_page_max=9 value_min=256 + # - func: "format test" + # vars: + # extra_args: checkpoints=1 leak_memory=0 mmap=1 file_type=fix + # - command: shell.exec + # params: + # working_dir: "wiredtiger/build_posix" + # script: | + # set -o errexit + # set -o verbose + + # GCOV=/opt/mongodbtoolchain/v3/bin/gcov gcovr -r .. -e '.*/bt_(debug|dump|misc|salvage|vrfy).*' -e '.*/(log|progress|verify_build|strerror|env_msg|err_file|cur_config|os_abort)\..*' -e '.*_stat\..*' --html -o ../coverage_report.html + # - command: s3.put + # params: + # aws_secret: ${aws_secret} + # aws_key: ${aws_key} + # local_file: wiredtiger/coverage_report.html + # bucket: build_external + # permissions: public-read + # content_type: text/html + # display_name: Coverage report + # remote_file: wiredtiger/${build_variant}/${revision}/coverage_report/coverage_report_${build_id}.html + + # Temporarily disabled + # - name: spinlock-gcc-test + # commands: + # - func: "get project" + # - func: "compile wiredtiger" + # vars: + # posix_configure_flags: --enable-python --with-spinlock=gcc --enable-strict + # - func: "make check all" + # - func: "format test" + # vars: + # times: 3 + # - func: "unit test" + + # Temporarily disabled + # - name: spinlock-pthread-adaptive-test + # commands: + # - func: "get project" + # - func: "compile wiredtiger" + # vars: + # posix_configure_flags: --enable-python --with-spinlock=pthread_adaptive --enable-strict + # - func: "make check all" + # - func: "format test" + # vars: + # times: 3 + # - func: "unit test" - name: wtperf-test depends_on: @@ -1816,7 +1910,7 @@ tasks: done - name: ftruncate-test - commands: + commands: - func: "get project" - func: "compile wiredtiger" vars: @@ -1827,7 +1921,7 @@ tasks: script: | set -o errexit set -o verbose - ${test_env_vars|} $(pwd)/../test/csuite/random_abort/smoke.sh 2>&1 + # ${test_env_vars|} $(pwd)/../test/csuite/random_abort/smoke.sh 2>&1 ${test_env_vars|} $(pwd)/../test/csuite/timestamp_abort/smoke.sh 2>&1 ${test_env_vars|} $(pwd)/test/csuite/test_truncated_log 2>&1 @@ -1839,7 +1933,7 @@ tasks: configure_env_vars: CC=/opt/mongodbtoolchain/v3/bin/gcc CXX=/opt/mongodbtoolchain/v3/bin/g++ PATH=/opt/mongodbtoolchain/v3/bin:$PATH CFLAGS="-g -Werror" posix_configure_flags: --enable-silent-rules --enable-diagnostic --disable-static - func: "make wiredtiger" - + # Run the long version of make check, that includes the full csuite tests - func: "make check all" vars: @@ -1852,7 +1946,7 @@ tasks: set -o verbose WT3363_CHECKPOINT_OP_RACES=1 test/csuite/./test_wt3363_checkpoint_op_races 2>&1 - + # Many dbs test - Run with: # 1. The defaults - func: "many dbs test" @@ -1868,27 +1962,27 @@ tasks: - func: "many dbs test" vars: many_db_args: -I -D 40 - + # extended test/thread runs - func: "thread test" - vars: + vars: thread_test_args: -t f - func: "thread test" - vars: + vars: thread_test_args: -S -F -n 100000 -t f - func: "thread test" - vars: + vars: thread_test_args: -t r - func: "thread test" - vars: + vars: thread_test_args: -S -F -n 100000 -t r - func: "thread test" - vars: + vars: thread_test_args: -t v - func: "thread test" vars: thread_test_args: -S -F -n 100000 -t v - + # random-abort - default (random time and number of threads) - func: "random abort test" # random-abort - minimum time, random number of threads @@ -1899,18 +1993,19 @@ tasks: - func: "random abort test" vars: random_abort_args: -t 40 - + # truncated-log - func: "truncated log test" - + # format test - - func: "format test" - vars: - extra_args: file_type=fix - - func: "format test" - vars: - extra_args: file_type=row - + # Temporarily disabled + # - func: "format test" + # vars: + # extra_args: file_type=fix + # - func: "format test" + # vars: + # extra_args: file_type=row + #FIXME: Add wtperf testing from Jenkin "wiredtiger-test-check-long" after fixing WT-5270 - name: time-shift-sensitivity-test @@ -1930,7 +2025,7 @@ tasks: set -o verbose ./time_shift_test.sh /usr/local/lib/faketime/libfaketimeMT.so.1 0-1 2>&1 - + - name: format-stress-sanitizer-test #set a 25 hours (25*60*60 = 90000 seconds) timeout exec_timeout_secs: 90000 @@ -1942,7 +2037,7 @@ tasks: test_env_vars: ASAN_OPTIONS="detect_leaks=1:abort_on_error=1:disable_coredump=0" ASAN_SYMBOLIZER_PATH=/opt/mongodbtoolchain/v3/bin/llvm-symbolizer # run for 24 hours ( 24 * 60 = 1440 minutes), skip known errors, don't stop at failed tests, use default config format_test_script_args: -E -t 1440 - + - name: format-stress-sanitizer-lsm-test commands: - func: "get project" @@ -1996,7 +2091,7 @@ tasks: # At the time of writing this script, one call to underlying scripts takes about ~15 mins to finish in worst case. # We are giving an extra ~20% room for vairance in execution time. times: 80 - + - name: split-stress-test commands: - func: "get project" @@ -2010,7 +2105,7 @@ tasks: set -o errexit set -o verbose for i in {1..10}; do ${python_binary|python} split_stress.py; done - + # Temporarily disabled - name: format-stress-test # Set 25 hours timeout exec_timeout_secs: 90000 @@ -2022,6 +2117,7 @@ tasks: #run for 24 hours ( 24 * 60 = 1440 minutes), use default config format_test_script_args: -b "SEGFAULT_SIGNALS=all catchsegv ./t" -t 1440 + # Temporarily disabled - name: format-stress-smoke-test # Set 7 hours timeout exec_timeout_secs: 25200 @@ -2047,7 +2143,7 @@ tasks: - func: "checkpoint stress test" vars: # No of times to run the loop - times: 2 + times: 1 # No of processes to run in the background no_of_procs: 10 @@ -2120,6 +2216,40 @@ tasks: working_dir: "wiredtiger" command: bash test/evergreen/compatibility_test_for_wiredtiger_releases.sh + - name: format-stress-sanitizer-ppc-test + # Set 2.5 hours timeout (60 * 60 * 2.5) + exec_timeout_secs: 9000 + commands: + - func: "get project" + - func: "compile wiredtiger" + vars: + # CC is set to the system default "clang" binary here as a workaround. Change it back to mongodbtoolchain "clang" binary after BUILD-10248 fix. + configure_env_vars: CCAS=/opt/mongodbtoolchain/v3/bin/gcc CC=/usr/bin/clang CXX=/opt/mongodbtoolchain/v3/bin/clang++ PATH=/opt/mongodbtoolchain/v3/bin:$PATH CFLAGS="-ggdb -fPIC -fsanitize=address -fno-omit-frame-pointer -I/opt/mongodbtoolchain/v3/lib/gcc/ppc64le-mongodb-linux/8.2.0/include" + posix_configure_flags: --enable-diagnostic --with-builtins=lz4,snappy,zlib + - func: "format test script" + vars: + test_env_vars: ASAN_OPTIONS="detect_leaks=1:abort_on_error=1:disable_coredump=0" ASAN_SYMBOLIZER_PATH=`ls /usr/bin/llvm-symbolizer* | tail -1` + # run for 2 hours ( 2 * 60 = 120 minutes), don't stop at failed tests, use default config + format_test_script_args: -t 120 + + - name: format-stress-sanitizer-smoke-ppc-test + # Set 7 hours timeout (60 * 60 * 7) + exec_timeout_secs: 25200 + commands: + - func: "get project" + - func: "compile wiredtiger" + vars: + # CC is set to the system default "clang" binary here as a workaround. Change it back to mongodbtoolchain "clang" binary after BUILD-10248 fix. + configure_env_vars: CCAS=/opt/mongodbtoolchain/v3/bin/gcc CC=/usr/bin/clang CXX=/opt/mongodbtoolchain/v3/bin/clang++ PATH=/opt/mongodbtoolchain/v3/bin:$PATH CFLAGS="-ggdb -fPIC -fsanitize=address -fno-omit-frame-pointer -I/opt/mongodbtoolchain/v3/lib/gcc/ppc64le-mongodb-linux/8.2.0/include" + posix_configure_flags: --enable-diagnostic --with-builtins=lz4,snappy,zlib + - func: "format test script" + # to emulate the original Jenkins job's test coverage, we are running the smoke test 16 times + # run smoke tests, don't stop at failed tests, use default config + vars: + test_env_vars: ASAN_OPTIONS="detect_leaks=1:abort_on_error=1:disable_coredump=0" ASAN_SYMBOLIZER_PATH=`ls /usr/bin/llvm-symbolizer* | tail -1` + format_test_script_args: -S + times: 16 + buildvariants: - name: ubuntu1804 display_name: Ubuntu 18.04 @@ -2127,7 +2257,7 @@ buildvariants: - ubuntu1804-test expansions: test_env_vars: LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libeatmydata.so PATH=/opt/mongodbtoolchain/v3/bin:$PATH LD_LIBRARY_PATH=$(pwd)/.libs top_srcdir=$(pwd)/.. top_builddir=$(pwd) - smp_command: -j $(grep -c ^processor /proc/cpuinfo) + smp_command: -j $(echo "`grep -c ^processor /proc/cpuinfo` * 2" | bc) posix_configure_flags: --enable-silent-rules --enable-diagnostic --enable-python --enable-zlib --enable-snappy --enable-strict --enable-static --prefix=$(pwd)/LOCAL_INSTALL make_command: PATH=/opt/mongodbtoolchain/v3/bin:$PATH make tasks: @@ -2136,29 +2266,23 @@ buildvariants: - name: make-check-msan-test - name: compile-ubsan - name: ubsan-test - - name: linux-directio - distros: ubuntu1804-build + # Temporarily disabled + # - name: linux-directio + # distros: ubuntu1804-build - name: syscall-linux - name: make-check-asan-test - name: configure-combinations - - name: checkpoint-filetypes-test - - name: coverage-report + # Temporarily disabled + # - name: checkpoint-filetypes-test + # - name: coverage-report - name: unit-test-long - - name: spinlock-gcc-test - - name: spinlock-pthread-adaptive-test + # Temporarily disabled + # - name: spinlock-gcc-test + # - name: spinlock-pthread-adaptive-test - name: compile-wtperf - name: wtperf-test - name: ftruncate-test - name: long-test - - name: recovery-stress-test - - name: format-stress-sanitizer-test - - name: format-stress-sanitizer-smoke-test - - name: format-stress-sanitizer-lsm-test - - name: split-stress-test - - name: format-stress-test - - name: format-stress-smoke-test - - name: race-condition-stress-sanitizer-test - - name: checkpoint-stress-test - name: static-wt-build-test - name: ubuntu1804-compilers @@ -2188,6 +2312,26 @@ buildvariants: - name: ".unit_test" - name: conf-dump-test +- name: ubuntu1804-stress-tests + display_name: Ubuntu 18.04 Stress tests + run_on: + - ubuntu1804-test + expansions: + test_env_vars: LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libeatmydata.so PATH=/opt/mongodbtoolchain/v3/bin:$PATH LD_LIBRARY_PATH=$(pwd)/.libs top_srcdir=$(pwd)/.. top_builddir=$(pwd) + smp_command: -j $(grep -c ^processor /proc/cpuinfo) + posix_configure_flags: --enable-silent-rules --enable-diagnostic --enable-python --enable-zlib --enable-snappy --enable-strict --enable-static --prefix=$(pwd)/LOCAL_INSTALL + make_command: PATH=/opt/mongodbtoolchain/v3/bin:$PATH make + tasks: + - name: recovery-stress-test + - name: format-stress-sanitizer-test + - name: format-stress-sanitizer-smoke-test + - name: format-stress-sanitizer-lsm-test + - name: split-stress-test + - name: format-stress-test + - name: format-stress-smoke-test + - name: race-condition-stress-sanitizer-test + - name: checkpoint-stress-test + - name: package display_name: Package batchtime: 1440 # 1 day @@ -2209,7 +2353,8 @@ buildvariants: - name: compile-linux-no-ftruncate - name: make-check-linux-no-ftruncate-test - name: unit-linux-no-ftruncate-test - - name: format-linux-no-ftruncate + # Temporarily disabled + # - name: format-linux-no-ftruncate - name: rhel80 display_name: RHEL 8.0 @@ -2223,30 +2368,35 @@ buildvariants: - name: compile - name: make-check-test - name: unit-test - - name: fops + # Temporarily disabled + # - name: fops - name: time-shift-sensitivity-test - name: compile-msan - name: make-check-msan-test - name: compile-ubsan - name: ubsan-test - - name: linux-directio - distros: rhel80-build + # Temporarily disabled + # - name: linux-directio + # distros: rhel80-build - name: syscall-linux - name: compile-asan - name: make-check-asan-test - - name: checkpoint-filetypes-test + # Temporarily disabled + # - name: checkpoint-filetypes-test - name: unit-test-long - - name: spinlock-gcc-test - - name: spinlock-pthread-adaptive-test + # Temporarily disabled + # - name: spinlock-gcc-test + # - name: spinlock-pthread-adaptive-test - name: compile-wtperf - name: wtperf-test - name: ftruncate-test - name: long-test - name: configure-combinations - - name: coverage-report + # Temporarily disabled + # - name: coverage-report -- name: large-scale-test - display_name: Large scale testing +- name: large-scale-tests + display_name: Large scale tests batchtime: 1440 # 1 day run_on: - rhel80-build @@ -2270,7 +2420,8 @@ buildvariants: - name: compile - name: ".windows_only" - name: ".unit_test" - - name: fops + # Temporarily disabled + # - name: fops - name: macos-1012 display_name: OS X 10.12 @@ -2286,37 +2437,40 @@ buildvariants: - name: compile - name: make-check-test - name: unit-test - - name: fops - -- name: little-endian - display_name: Little-endian (x86) - run_on: - - ubuntu1804-test - batchtime: 10080 # 7 days - expansions: - smp_command: -j $(grep -c ^processor /proc/cpuinfo) - test_env_vars: PATH=/opt/mongodbtoolchain/v3/bin:$PATH LD_LIBRARY_PATH=$(pwd)/.libs top_srcdir=$(pwd)/.. top_builddir=$(pwd) - tasks: - - name: compile - - name: generate-datafile-little-endian - - name: verify-datafile-little-endian - - name: verify-datafile-from-big-endian - -- name: big-endian - display_name: Big-endian (s390x/zSeries) - modules: - - enterprise - run_on: - - ubuntu1804-zseries-build - batchtime: 10080 # 7 days - expansions: - smp_command: -j $(grep -c ^processor /proc/cpuinfo) - test_env_vars: PATH=/opt/mongodbtoolchain/v3/bin:$PATH LD_LIBRARY_PATH=$(pwd)/.lib top_srcdir=$(pwd)/.. top_builddir=$(pwd) - tasks: - - name: compile - - name: generate-datafile-big-endian - - name: verify-datafile-big-endian - - name: verify-datafile-from-little-endian + # Temporarily disabled + # - name: fops + +# Temporarily disabled +# - name: little-endian +# display_name: Little-endian (x86) +# run_on: +# - ubuntu1804-test +# batchtime: 10080 # 7 days +# expansions: +# smp_command: -j $(grep -c ^processor /proc/cpuinfo) +# test_env_vars: PATH=/opt/mongodbtoolchain/v3/bin:$PATH LD_LIBRARY_PATH=$(pwd)/.libs top_srcdir=$(pwd)/.. top_builddir=$(pwd) +# tasks: +# - name: compile +# - name: generate-datafile-little-endian +# - name: verify-datafile-little-endian +# - name: verify-datafile-from-big-endian + +# Temporarily disabled +# - name: big-endian +# display_name: Big-endian (s390x/zSeries) +# modules: +# - enterprise +# run_on: +# - ubuntu1804-zseries-build +# batchtime: 10080 # 7 days +# expansions: +# smp_command: -j $(grep -c ^processor /proc/cpuinfo) +# test_env_vars: PATH=/opt/mongodbtoolchain/v3/bin:$PATH LD_LIBRARY_PATH=$(pwd)/.lib top_srcdir=$(pwd)/.. top_builddir=$(pwd) +# tasks: +# - name: compile +# - name: generate-datafile-big-endian +# - name: verify-datafile-big-endian +# - name: verify-datafile-from-little-endian - name: ubuntu1804-ppc display_name: Ubuntu 18.04 PPC @@ -2333,6 +2487,9 @@ buildvariants: - name: format-stress-ppc-zseries-test - name: format-stress-smoke-test - name: format-wtperf-test + # Replace the below two tests with format-stress-sanitizer-ppc-test and format-stress-sanitizer-smoke-test after BUILD-10248 fix. + - name: format-stress-sanitizer-ppc-test + - name: format-stress-sanitizer-smoke-ppc-test - name: ubuntu1804-zseries display_name: Ubuntu 18.04 zSeries diff --git a/src/third_party/wiredtiger/test/evergreen/compatibility_test_for_mongodb_releases.sh b/src/third_party/wiredtiger/test/evergreen/compatibility_test_for_mongodb_releases.sh index 1207c479c59..c115ddab829 100755 --- a/src/third_party/wiredtiger/test/evergreen/compatibility_test_for_mongodb_releases.sh +++ b/src/third_party/wiredtiger/test/evergreen/compatibility_test_for_mongodb_releases.sh @@ -30,17 +30,11 @@ build_rel() config="" config+="--enable-snappy " - case "$1" in - # Please note 'develop' here is planned as the future MongoDB release 4.2 - the only release that supports - # both enabling and disabling of timestamps in data format. Once 4.2 is released, we need to update this script. - "develop") - branch="develop";; - "develop-timestamps") + if [ $1 == "develop" ]; then branch="develop" - config+="--enable-page-version-ts";; - *) - branch=$(get_release "$1");; - esac + else + branch=$(get_release "$1") + fi git checkout --quiet -b $branch || return 1 @@ -114,15 +108,14 @@ run() (build_rel 3.4) || return 1 (build_rel 3.6) || return 1 (build_rel 4.0) || return 1 + (build_rel 4.2) || return 1 (build_rel develop) || return 1 - (build_rel develop-timestamps) || return 1 # Verify forward/backward compatibility. (verify 3.4 3.6) || return 1 (verify 3.6 4.0) || return 1 - (verify 4.0 develop) || return 1 - (verify 4.0 develop-timestamps) || return 1 - (verify develop develop-timestamps) || return 1 + (verify 4.0 4.2) || return 1 + (verify 4.2 develop) || return 1 return 0 } diff --git a/src/third_party/wiredtiger/test/fops/Makefile.am b/src/third_party/wiredtiger/test/fops/Makefile.am index 519f6315445..7a5920221ae 100644 --- a/src/third_party/wiredtiger/test/fops/Makefile.am +++ b/src/third_party/wiredtiger/test/fops/Makefile.am @@ -11,7 +11,8 @@ t_LDADD +=$(top_builddir)/libwiredtiger.la t_LDFLAGS = -static # Run this during a "make check" smoke test. -TESTS = $(noinst_PROGRAMS) +# Temporarily disabled +# TESTS = $(noinst_PROGRAMS) LOG_COMPILER = $(TEST_WRAPPER) clean-local: diff --git a/src/third_party/wiredtiger/test/format/CONFIG.stress b/src/third_party/wiredtiger/test/format/CONFIG.stress index 0b5251d7952..e5bfc026375 100644 --- a/src/third_party/wiredtiger/test/format/CONFIG.stress +++ b/src/third_party/wiredtiger/test/format/CONFIG.stress @@ -2,6 +2,7 @@ cache_minimum=20 huffman_key=0 huffman_value=0 -rows=1000000 +rows=1000000:5000000 runs=100 -timer=4 +threads=4:32 +timer=6:30 diff --git a/src/third_party/wiredtiger/test/format/Makefile.am b/src/third_party/wiredtiger/test/format/Makefile.am index bff2986f25e..a8f18731b1b 100644 --- a/src/third_party/wiredtiger/test/format/Makefile.am +++ b/src/third_party/wiredtiger/test/format/Makefile.am @@ -25,7 +25,8 @@ backup: refresh: rm -rf RUNDIR && cp -p -r BACKUP RUNDIR -TESTS = smoke.sh +# Temporarily disabled +# TESTS = smoke.sh clean-local: rm -rf RUNDIR s_dumpcmp core.* *.core diff --git a/src/third_party/wiredtiger/test/format/config.c b/src/third_party/wiredtiger/test/format/config.c index 1bfe87473c4..cad60906561 100644 --- a/src/third_party/wiredtiger/test/format/config.c +++ b/src/third_party/wiredtiger/test/format/config.c @@ -137,13 +137,10 @@ config_setup(void) } } - /* - * If data_source and file_type were both "permanent", we may still have a mismatch. - */ - if (DATASOURCE("lsm") && g.type != ROW) { - fprintf(stderr, "%s: lsm data_source is only compatible with row file_type\n", progname); - exit(EXIT_FAILURE); - } + /* If data_source and file_type were both "permanent", we may still have a mismatch. */ + if (DATASOURCE("lsm") && g.type != ROW) + testutil_die( + EINVAL, "%s: lsm data_source is only compatible with row file_type\n", progname); /* * Build the top-level object name: we're overloading data_source in our configuration, LSM @@ -500,7 +497,7 @@ config_encryption(void) static bool config_fix(void) { - /* Fixed-length column stores don't support the lookaside table, so no modify operations. */ + /* Fixed-length column stores don't support the history store table, so no modify operations. */ if (config_is_perm("modify_pct")) return (false); return (true); @@ -783,12 +780,24 @@ config_error(void) /* Display configuration names. */ fprintf(stderr, "\n"); + fprintf(stderr, "=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=\n"); + fprintf(stderr, "Configuration values:\n"); + fprintf(stderr, "%10s: %s\n", "off", "boolean off"); + fprintf(stderr, "%10s: %s\n", "on", "boolean on"); + fprintf(stderr, "%10s: %s\n", "0", "boolean off"); + fprintf(stderr, "%10s: %s\n", "1", "boolean on"); + fprintf(stderr, "%10s: %s\n", "NNN", "unsigned number"); + fprintf(stderr, "%10s: %s\n", "NNN-NNN", "number range, each number equally likely"); + fprintf(stderr, "%10s: %s\n", "NNN:NNN", "number range, lower numbers more likely"); + fprintf(stderr, "%10s: %s\n", "string", "configuration value"); + fprintf(stderr, "\n"); + fprintf(stderr, "=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=\n"); fprintf(stderr, "Configuration names:\n"); for (cp = c; cp->name != NULL; ++cp) - if (strlen(cp->name) > 17) - fprintf(stderr, "%s\n%17s: %s\n", cp->name, " ", cp->desc); + if (strlen(cp->name) > 25) + fprintf(stderr, "%s: %s\n", cp->name, cp->desc); else - fprintf(stderr, "%17s: %s\n", cp->name, cp->desc); + fprintf(stderr, "%25s: %s\n", cp->name, cp->desc); } /* @@ -920,40 +929,54 @@ config_find(const char *s, size_t len, bool fatal) if (strncmp(s, cp->name, len) == 0 && cp->name[len] == '\0') return (cp); - /* - * Optionally ignore unknown keywords, it makes it easier to run old CONFIG files. - */ - if (fatal) { - fprintf(stderr, "%s: %s: unknown required configuration keyword\n", progname, s); - exit(EXIT_FAILURE); - } + /* Optionally ignore unknown keywords, it makes it easier to run old CONFIG files. */ + if (fatal) + testutil_die(EINVAL, "%s: %s: unknown required configuration keyword\n", progname, s); + fprintf(stderr, "%s: %s: WARNING, ignoring unknown configuration keyword\n", progname, s); return (NULL); } /* + * config_value -- + * String to long helper function. + */ +static uint32_t +config_value(const char *config, const char *p, int match) +{ + long v; + char *endptr; + + errno = 0; + v = strtol(p, &endptr, 10); + if ((errno == ERANGE && (v == LONG_MAX || v == LONG_MIN)) || (errno != 0 && v == 0) || + *endptr != match || v < 0 || v > UINT32_MAX) + testutil_die( + EINVAL, "%s: %s: illegal numeric value or value out of range", progname, config); + return ((uint32_t)v); +} + +/* * config_single -- * Set a single configuration structure value. */ void config_single(const char *s, bool perm) { + enum { RANGE_FIXED, RANGE_NONE, RANGE_WEIGHTED } range; CONFIG *cp; - long vlong; - uint32_t v; - char *p; - const char *ep; - - if ((ep = strchr(s, '=')) == NULL) { - fprintf(stderr, "%s: %s: illegal configuration value\n", progname, s); - exit(EXIT_FAILURE); - } + uint32_t steps, v1, v2; + u_int i; + const char *equalp, *vp1, *vp2; + + if ((equalp = strchr(s, '=')) == NULL) + testutil_die(EINVAL, "%s: %s: illegal configuration value\n", progname, s); - if ((cp = config_find(s, (size_t)(ep - s), false)) == NULL) + if ((cp = config_find(s, (size_t)(equalp - s), false)) == NULL) return; F_SET(cp, perm ? C_PERM : C_TEMP); - ++ep; + ++equalp; if (F_ISSET(cp, C_STRING)) { /* @@ -965,64 +988,99 @@ config_single(const char *s, bool perm) } if (strncmp(s, "checkpoints", strlen("checkpoints")) == 0) { - config_map_checkpoint(ep, &g.c_checkpoint_flag); - *cp->vstr = dstrdup(ep); + config_map_checkpoint(equalp, &g.c_checkpoint_flag); + *cp->vstr = dstrdup(equalp); } else if (strncmp(s, "checksum", strlen("checksum")) == 0) { - config_map_checksum(ep, &g.c_checksum_flag); - *cp->vstr = dstrdup(ep); + config_map_checksum(equalp, &g.c_checksum_flag); + *cp->vstr = dstrdup(equalp); } else if (strncmp(s, "compression", strlen("compression")) == 0) { - config_map_compression(ep, &g.c_compression_flag); - *cp->vstr = dstrdup(ep); + config_map_compression(equalp, &g.c_compression_flag); + *cp->vstr = dstrdup(equalp); } else if (strncmp(s, "data_source", strlen("data_source")) == 0 && - strncmp("file", ep, strlen("file")) != 0 && strncmp("lsm", ep, strlen("lsm")) != 0 && - strncmp("table", ep, strlen("table")) != 0) { - fprintf(stderr, "Invalid data source option: %s\n", ep); - exit(EXIT_FAILURE); + strncmp("file", equalp, strlen("file")) != 0 && + strncmp("lsm", equalp, strlen("lsm")) != 0 && + strncmp("table", equalp, strlen("table")) != 0) { + testutil_die(EINVAL, "Invalid data source option: %s\n", equalp); } else if (strncmp(s, "encryption", strlen("encryption")) == 0) { - config_map_encryption(ep, &g.c_encryption_flag); - *cp->vstr = dstrdup(ep); + config_map_encryption(equalp, &g.c_encryption_flag); + *cp->vstr = dstrdup(equalp); } else if (strncmp(s, "file_type", strlen("file_type")) == 0) { - config_map_file_type(ep, &g.type); + config_map_file_type(equalp, &g.type); *cp->vstr = dstrdup(config_file_type(g.type)); } else if (strncmp(s, "isolation", strlen("isolation")) == 0) { - config_map_isolation(ep, &g.c_isolation_flag); - *cp->vstr = dstrdup(ep); + config_map_isolation(equalp, &g.c_isolation_flag); + *cp->vstr = dstrdup(equalp); } else if (strncmp(s, "logging_compression", strlen("logging_compression")) == 0) { - config_map_compression(ep, &g.c_logging_compression_flag); - *cp->vstr = dstrdup(ep); + config_map_compression(equalp, &g.c_logging_compression_flag); + *cp->vstr = dstrdup(equalp); } else - *cp->vstr = dstrdup(ep); + *cp->vstr = dstrdup(equalp); return; } - vlong = -1; if (F_ISSET(cp, C_BOOL)) { - if (strncmp(ep, "off", strlen("off")) == 0) - vlong = 0; - else if (strncmp(ep, "on", strlen("on")) == 0) - vlong = 1; - } - if (vlong == -1) { - vlong = strtol(ep, &p, 10); - if (*p != '\0') { - fprintf(stderr, "%s: %s: illegal numeric value\n", progname, s); - exit(EXIT_FAILURE); + if (strncmp(equalp, "off", strlen("off")) == 0) + v1 = 0; + else if (strncmp(equalp, "on", strlen("on")) == 0) + v1 = 1; + else { + v1 = config_value(s, equalp, '\0'); + if (v1 != 0 && v1 != 1) + testutil_die(EINVAL, "%s: %s: value of boolean not 0 or 1", progname, s); } + + *cp->v = v1; + return; } - v = (uint32_t)vlong; - if (F_ISSET(cp, C_BOOL)) { - if (v != 0 && v != 1) { - fprintf(stderr, "%s: %s: value of boolean not 0 or 1\n", progname, s); - exit(EXIT_FAILURE); - } - } else if (v < cp->min || v > cp->maxset) { - fprintf(stderr, "%s: %s: value outside min/max values of %" PRIu32 "-%" PRIu32 "\n", + + /* + * Three possible syntax elements: a number, two numbers separated by a dash, two numbers + * separated by an colon. The first is a fixed value, the second is a range where all values are + * equally possible, the third is a weighted range where lower values are more likely. + */ + vp1 = equalp; + range = RANGE_NONE; + if ((vp2 = strchr(vp1, '-')) != NULL) { + ++vp2; + range = RANGE_FIXED; + } else if ((vp2 = strchr(vp1, ':')) != NULL) { + ++vp2; + range = RANGE_WEIGHTED; + } + + v1 = config_value(s, vp1, range == RANGE_NONE ? '\0' : (range == RANGE_FIXED ? '-' : ':')); + if (v1 < cp->min || v1 > cp->maxset) + testutil_die(EINVAL, "%s: %s: value outside min/max values of %" PRIu32 "-%" PRIu32 "\n", progname, s, cp->min, cp->maxset); - exit(EXIT_FAILURE); + + if (range != RANGE_NONE) { + v2 = config_value(s, vp2, '\0'); + if (v2 < cp->min || v2 > cp->maxset) + testutil_die(EINVAL, + "%s: %s: value outside min/max values of %" PRIu32 "-%" PRIu32 "\n", progname, s, + cp->min, cp->maxset); + if (v1 > v2) + testutil_die(EINVAL, "%s: %s: illegal numeric range\n", progname, s); + + if (range == RANGE_FIXED) + v1 = mmrand(NULL, (u_int)v1, (u_int)v2); + else { + /* + * Roll dice, 50% chance of proceeding to the next larger value, and 5 steps to the + * maximum value. + */ + steps = ((v2 - v1) + 4) / 5; + if (steps == 0) + steps = 1; + for (i = 0; i < 5; ++i, v1 += steps) + if (mmrand(NULL, 0, 1) == 0) + break; + v1 = WT_MIN(v1, v2); + } } - *cp->v = v; + *cp->v = v1; } /* diff --git a/src/third_party/wiredtiger/test/format/config.h b/src/third_party/wiredtiger/test/format/config.h index 832c977df29..f7681cd2e2b 100644 --- a/src/third_party/wiredtiger/test/format/config.h +++ b/src/third_party/wiredtiger/test/format/config.h @@ -260,8 +260,8 @@ static CONFIG c[] = {{"abort", "if timed run should drop core", /* 0% */ {"timing_stress_checkpoint", "stress checkpoints", /* 2% */ C_BOOL, 2, 0, 0, &g.c_timing_stress_checkpoint, NULL}, - {"timing_stress_lookaside_sweep", "stress lookaside sweep", /* 2% */ - C_BOOL, 2, 0, 0, &g.c_timing_stress_lookaside_sweep, NULL}, + {"timing_stress_hs_sweep", "stress history store sweep", /* 2% */ + C_BOOL, 2, 0, 0, &g.c_timing_stress_hs_sweep, NULL}, {"timing_stress_split_1", "stress splits (#1)", /* 2% */ C_BOOL, 2, 0, 0, &g.c_timing_stress_split_1, NULL}, diff --git a/src/third_party/wiredtiger/test/format/format.h b/src/third_party/wiredtiger/test/format/format.h index 6407ee652d0..836e042a8ea 100644 --- a/src/third_party/wiredtiger/test/format/format.h +++ b/src/third_party/wiredtiger/test/format/format.h @@ -66,7 +66,7 @@ typedef struct { char *home_backup_init; /* Initialize backup command */ char *home_config; /* Run CONFIG file path */ char *home_init; /* Initialize home command */ - char *home_lasdump; /* LAS dump filename */ + char *home_hsdump; /* HS dump filename */ char *home_log; /* Operation log file path */ char *home_pagedump; /* Page dump filename */ char *home_rand; /* RNG log file path */ @@ -186,7 +186,7 @@ typedef struct { uint32_t c_timer; uint32_t c_timing_stress_aggressive_sweep; uint32_t c_timing_stress_checkpoint; - uint32_t c_timing_stress_lookaside_sweep; + uint32_t c_timing_stress_hs_sweep; uint32_t c_timing_stress_split_1; uint32_t c_timing_stress_split_2; uint32_t c_timing_stress_split_3; diff --git a/src/third_party/wiredtiger/test/format/format.i b/src/third_party/wiredtiger/test/format/format.i index 2b22eab069a..3f41173726c 100644 --- a/src/third_party/wiredtiger/test/format/format.i +++ b/src/third_party/wiredtiger/test/format/format.i @@ -76,7 +76,7 @@ rng(WT_RAND_STATE *rnd) * and replay because threaded operation order can't be replayed. Do that check inline so it's a * cheap call once thread performance starts to matter. */ - return (g.rand_log_stop ? __wt_random(rnd) : rng_slow(rnd)); + return (g.randfp == NULL || g.rand_log_stop ? __wt_random(rnd) : rng_slow(rnd)); } /* diff --git a/src/third_party/wiredtiger/test/format/format.sh b/src/third_party/wiredtiger/test/format/format.sh index 200aa5466c5..41c112ae7c4 100755 --- a/src/third_party/wiredtiger/test/format/format.sh +++ b/src/third_party/wiredtiger/test/format/format.sh @@ -248,8 +248,10 @@ report_failure() dir=$1 log="$dir.log" - skip_known_errors $log - skip_ret=$? + # DO NOT CURRENTLY SKIP ANY ERRORS. + skip_ret=0 + #skip_known_errors $log + #skip_ret=$? echo "$name: failure status reported" > $dir/$status [[ $skip_ret -ne 0 ]] && failure=$(($failure + 1)) @@ -318,6 +320,14 @@ resolve() continue } + # Check for Evergreen running out of disk space, and forcibly quit. + grep -E -i 'no space left on device' $log > /dev/null && { + rm -rf $dir $log + force_quit=1 + echo "$name: job in $dir ran out of disk space" + continue + } + # Test recovery on jobs configured for random abort. */ grep 'aborting to test recovery' $log > /dev/null && { cp -pr $dir $dir.RECOVER diff --git a/src/third_party/wiredtiger/test/format/ops.c b/src/third_party/wiredtiger/test/format/ops.c index 93d5fc204ac..d86c8ff29bd 100644 --- a/src/third_party/wiredtiger/test/format/ops.c +++ b/src/third_party/wiredtiger/test/format/ops.c @@ -532,10 +532,10 @@ prepare_transaction(TINFO *tinfo) * When in a transaction on the live table with snapshot isolation, track operations for later * repetition. */ -#define SNAP_TRACK(tinfo, op) \ - do { \ - if (intxn && !ckpt_handle && iso_config == ISOLATION_SNAPSHOT) \ - snap_track(tinfo, op); \ +#define SNAP_TRACK(tinfo, op) \ + do { \ + if (intxn && iso_config == ISOLATION_SNAPSHOT) \ + snap_track(tinfo, op); \ } while (0) /* @@ -543,7 +543,7 @@ prepare_transaction(TINFO *tinfo) * Create a new session/cursor pair for the thread. */ static void -ops_open_session(TINFO *tinfo, bool *ckpt_handlep) +ops_open_session(TINFO *tinfo) { WT_CONNECTION *conn; WT_CURSOR *cursor; @@ -559,38 +559,13 @@ ops_open_session(TINFO *tinfo, bool *ckpt_handlep) testutil_check(conn->open_session(conn, NULL, NULL, &session)); /* - * 10% of the time, perform some read-only operations from a checkpoint. - * Skip if we are using data-sources or LSM, they don't support reading - * from checkpoints. + * Configure "append", in the case of column stores, we append when inserting new rows. + * + * WT_SESSION.open_cursor can return EBUSY if concurrent with a metadata operation, retry. */ - cursor = NULL; - if (!DATASOURCE("lsm") && mmrand(&tinfo->rnd, 1, 10) == 1) { - /* - * WT_SESSION.open_cursor can return EBUSY if concurrent with a metadata operation, retry. - */ - while ((ret = session->open_cursor( - session, g.uri, NULL, "checkpoint=WiredTigerCheckpoint", &cursor)) == EBUSY) - __wt_yield(); - - /* - * If the checkpoint hasn't been created yet, ignore the error. - */ - if (ret != ENOENT) { - testutil_check(ret); - *ckpt_handlep = true; - } - } - if (cursor == NULL) { - /* - * Configure "append", in the case of column stores, we append when inserting new rows. - * - * WT_SESSION.open_cursor can return EBUSY if concurrent with a metadata operation, retry. - */ - while ((ret = session->open_cursor(session, g.uri, NULL, "append", &cursor)) == EBUSY) - __wt_yield(); - testutil_checkfmt(ret, "%s", g.uri); - *ckpt_handlep = false; - } + while ((ret = session->open_cursor(session, g.uri, NULL, "append", &cursor)) == EBUSY) + __wt_yield(); + testutil_checkfmt(ret, "%s", g.uri); tinfo->session = session; tinfo->cursor = cursor; @@ -611,12 +586,11 @@ ops(void *arg) uint64_t reset_op, session_op, truncate_op; uint32_t range, rnd; u_int i, j, iso_config; - bool ckpt_handle, greater_than, intxn, next, positioned, prepared; + bool greater_than, intxn, next, positioned, prepared; tinfo = arg; iso_config = ISOLATION_RANDOM; /* -Wconditional-uninitialized */ - ckpt_handle = false; /* -Wconditional-uninitialized */ /* Tracking of transactional snapshot isolation operations. */ tinfo->snap = tinfo->snap_first = tinfo->snap_list; @@ -651,7 +625,7 @@ ops(void *arg) intxn = false; } - ops_open_session(tinfo, &ckpt_handle); + ops_open_session(tinfo); /* Pick the next session/cursor close/open. */ session_op += mmrand(&tinfo->rnd, 100, 5000); @@ -676,7 +650,7 @@ ops(void *arg) * If not in a transaction, have a live handle and running in a timestamp world, * occasionally repeat a timestamped operation. */ - if (!intxn && !ckpt_handle && g.c_txn_timestamps && mmrand(&tinfo->rnd, 1, 15) == 1) { + if (!intxn && g.c_txn_timestamps && mmrand(&tinfo->rnd, 1, 15) == 1) { ++tinfo->search; snap_repeat_single(cursor, tinfo); } @@ -695,22 +669,20 @@ ops(void *arg) /* Select an operation. */ op = READ; - if (!ckpt_handle) { - i = mmrand(&tinfo->rnd, 1, 100); - if (i < g.c_delete_pct && tinfo->ops > truncate_op) { - op = TRUNCATE; - - /* Pick the next truncate operation. */ - truncate_op += mmrand(&tinfo->rnd, 20000, 100000); - } else if (i < g.c_delete_pct) - op = REMOVE; - else if (i < g.c_delete_pct + g.c_insert_pct) - op = INSERT; - else if (i < g.c_delete_pct + g.c_insert_pct + g.c_modify_pct) - op = MODIFY; - else if (i < g.c_delete_pct + g.c_insert_pct + g.c_modify_pct + g.c_write_pct) - op = UPDATE; - } + i = mmrand(&tinfo->rnd, 1, 100); + if (i < g.c_delete_pct && tinfo->ops > truncate_op) { + op = TRUNCATE; + + /* Pick the next truncate operation. */ + truncate_op += mmrand(&tinfo->rnd, 20000, 100000); + } else if (i < g.c_delete_pct) + op = REMOVE; + else if (i < g.c_delete_pct + g.c_insert_pct) + op = INSERT; + else if (i < g.c_delete_pct + g.c_insert_pct + g.c_modify_pct) + op = MODIFY; + else if (i < g.c_delete_pct + g.c_insert_pct + g.c_modify_pct + g.c_write_pct) + op = UPDATE; /* Select a row. */ tinfo->keyno = mmrand(&tinfo->rnd, 1, (u_int)g.rows); @@ -735,7 +707,7 @@ ops(void *arg) * Optionally reserve a row. Reserving a row before a read isn't all that sensible, but not * unexpected, either. */ - if (intxn && !ckpt_handle && mmrand(&tinfo->rnd, 0, 20) == 1) { + if (intxn && mmrand(&tinfo->rnd, 0, 20) == 1) { switch (g.type) { case ROW: ret = row_reserve(tinfo, cursor, positioned); @@ -955,7 +927,7 @@ update_instead_of_chosen_op: * Ending a transaction. If on a live handle and the transaction was configured for snapshot * isolation, repeat the operations and confirm the results are unchanged. */ - if (intxn && !ckpt_handle && iso_config == ISOLATION_SNAPSHOT) { + if (intxn && iso_config == ISOLATION_SNAPSHOT) { __wt_yield(); /* Encourage races */ ret = snap_repeat_txn(cursor, tinfo); diff --git a/src/third_party/wiredtiger/test/format/snap.c b/src/third_party/wiredtiger/test/format/snap.c index 96c23fd4afd..afea3b9ac33 100644 --- a/src/third_party/wiredtiger/test/format/snap.c +++ b/src/third_party/wiredtiger/test/format/snap.c @@ -231,18 +231,18 @@ snap_verify(WT_CURSOR *cursor, TINFO *tinfo, SNAP_OPS *snap) * We have a mismatch. Try to print out as much information as we can. In doing so, we are * calling into the debug code directly and that does not take locks, so it's possible we will * simply drop core. The most important information is the key/value mismatch information. Then - * try to dump out the other information. Right now we dump the entire lookaside table including - * what is on disk. That can potentially be very large. If it becomes a problem, this can be - * modified to just dump out the page this key is on. Write a failure message into the log file - * first so format.sh knows we failed, and turn off core dumps. + * try to dump out the other information. Right now we dump the entire history store table + * including what is on disk. That can potentially be very large. If it becomes a problem, this + * can be modified to just dump out the page this key is on. Write a failure message into the + * log file first so format.sh knows we failed, and turn off core dumps. */ fprintf(stderr, "\n%s: run FAILED\n", progname); set_core_off(); fprintf(stderr, "snapshot-isolation error: Dumping page to %s\n", g.home_pagedump); testutil_check(__wt_debug_cursor_page(cursor, g.home_pagedump)); - fprintf(stderr, "snapshot-isolation error: Dumping LAS to %s\n", g.home_lasdump); - testutil_check(__wt_debug_cursor_las(cursor, g.home_lasdump)); + fprintf(stderr, "snapshot-isolation error: Dumping HS to %s\n", g.home_hsdump); + testutil_check(__wt_debug_cursor_tree_hs(cursor, g.home_hsdump)); if (g.logging) testutil_check(cursor->session->log_flush(cursor->session, "sync=off")); #endif diff --git a/src/third_party/wiredtiger/test/format/util.c b/src/third_party/wiredtiger/test/format/util.c index 8a3bdb94693..380d3bdf691 100644 --- a/src/third_party/wiredtiger/test/format/util.c +++ b/src/third_party/wiredtiger/test/format/util.c @@ -326,10 +326,10 @@ path_setup(const char *home) g.home_log = dmalloc(len); testutil_check(__wt_snprintf(g.home_log, len, "%s/%s", g.home, "log")); - /* LAS dump file. */ - len = strlen(g.home) + strlen("LASdump") + 2; - g.home_lasdump = dmalloc(len); - testutil_check(__wt_snprintf(g.home_lasdump, len, "%s/%s", g.home, "LASdump")); + /* History store dump file. */ + len = strlen(g.home) + strlen("HSdump") + 2; + g.home_hsdump = dmalloc(len); + testutil_check(__wt_snprintf(g.home_hsdump, len, "%s/%s", g.home, "HSdump")); /* Page dump file. */ len = strlen(g.home) + strlen("pagedump") + 2; diff --git a/src/third_party/wiredtiger/test/format/wts.c b/src/third_party/wiredtiger/test/format/wts.c index b05893598ea..f783dcb2e63 100644 --- a/src/third_party/wiredtiger/test/format/wts.c +++ b/src/third_party/wiredtiger/test/format/wts.c @@ -228,8 +228,8 @@ wts_open(const char *home, bool set_api, WT_CONNECTION **connp) CONFIG_APPEND(p, ",aggressive_sweep"); if (g.c_timing_stress_checkpoint) CONFIG_APPEND(p, ",checkpoint_slow"); - if (g.c_timing_stress_lookaside_sweep) - CONFIG_APPEND(p, ",lookaside_sweep_race"); + if (g.c_timing_stress_hs_sweep) + CONFIG_APPEND(p, ",history_store_sweep_race"); if (g.c_timing_stress_split_1) CONFIG_APPEND(p, ",split_1"); if (g.c_timing_stress_split_2) diff --git a/src/third_party/wiredtiger/test/suite/test_backup08.py b/src/third_party/wiredtiger/test/suite/test_backup08.py index 6c4b04edd2a..f238a865514 100644 --- a/src/third_party/wiredtiger/test/suite/test_backup08.py +++ b/src/third_party/wiredtiger/test/suite/test_backup08.py @@ -31,7 +31,7 @@ # import os, shutil -import wiredtiger, wttest +import unittest, wiredtiger, wttest from wtscenario import make_scenarios def timestamp_str(t): diff --git a/src/third_party/wiredtiger/test/suite/test_backup11.py b/src/third_party/wiredtiger/test/suite/test_backup11.py index c5de361eb04..76fa70c4b2b 100644 --- a/src/third_party/wiredtiger/test/suite/test_backup11.py +++ b/src/third_party/wiredtiger/test/suite/test_backup11.py @@ -36,48 +36,28 @@ from wtscenario import make_scenarios # test_backup11.py # Test cursor backup with a duplicate backup cursor. class test_backup11(wttest.WiredTigerTestCase, suite_subprocess): + conn_config= 'cache_size=1G,log=(enabled,file_max=100K)' dir='backup.dir' # Backup directory name - logmax="100K" - uri="table:test" + mult=0 nops=100 - pfx = 'test_backup' - - # ('archiving', dict(archive='true')), - # ('not-archiving', dict(archive='false')), - scenarios = make_scenarios([ - ('archiving', dict(archive='true')), - ]) - - # Create a large cache, otherwise this test runs quite slowly. - def conn_config(self): - return 'cache_size=1G,log=(archive=%s,' % self.archive + \ - 'enabled,file_max=%s)' % self.logmax + uri="table:test" def add_data(self): - log2 = "WiredTigerLog.0000000002" - log3 = "WiredTigerLog.0000000003" - - self.session.create(self.uri, "key_format=S,value_format=S") - # Insert small amounts of data at a time stopping after we - # cross into log file 2. - loop = 0 c = self.session.open_cursor(self.uri) - while not os.path.exists(log2): - for i in range(0, self.nops): - num = i + (loop * self.nops) - key = 'key' + str(num) - val = 'value' + str(num) - c[key] = val - loop += 1 + for i in range(0, self.nops): + num = i + (self.mult * self.nops) + key = 'key' + str(num) + val = 'value' + str(num) + c[key] = val + self.mult += 1 self.session.checkpoint() c.close() - return loop def test_backup11(self): - - loop = self.add_data() + self.session.create(self.uri, "key_format=S,value_format=S") + self.add_data() # Open up the backup cursor. This causes a new log file to be created. # That log file is not part of the list returned. This is a full backup @@ -86,17 +66,8 @@ class test_backup11(wttest.WiredTigerTestCase, suite_subprocess): config = 'incremental=(enabled,this_id="ID1")' bkup_c = self.session.open_cursor('backup:', None, config) - # Add some data that will appear in log file 3. - c = self.session.open_cursor(self.uri) - for i in range(0, self.nops): - num = i + (loop * self.nops) - key = 'key' + str(num) - val = 'value' + str(num) - c[key] = val - loop += 1 - c.close() - self.session.log_flush('sync=on') - self.session.checkpoint() + # Add data while the backup cursor is open. + self.add_data() # Now copy the files returned by the backup cursor. orig_logs = [] @@ -136,37 +107,22 @@ class test_backup11(wttest.WiredTigerTestCase, suite_subprocess): bkup_c.close() # Add more data - c = self.session.open_cursor(self.uri) - for i in range(0, self.nops): - num = i + (loop * self.nops) - key = 'key' + str(num) - val = 'value' + str(num) - c[key] = val - loop += 1 - c.close() - self.session.log_flush('sync=on') - self.session.checkpoint() + self.add_data() - # Test a few error cases now. - # - Incremental filename must be on duplicate, not primary. - # - An incremental duplicate must have an incremental primary. - # - We cannot make multiple incremental duplcate backup cursors. - # - We cannot duplicate the duplicate backup cursor. - # - We cannot mix block incremental with a log target on the same duplicate. - # - Incremental ids must be on primary, not duplicate. - # - Incremental must be opened on a primary with a source identifier. - # - Force stop must be on primary, not duplicate. + # Test error cases now. # - Incremental filename must be on duplicate, not primary. # Test this first because we currently do not have a primary open. config = 'incremental=(file=test.wt)' msg = "/file name can only be specified on a duplicate/" + self.pr("Specify file on primary") self.assertRaisesWithMessage(wiredtiger.WiredTigerError, lambda:self.assertEquals(self.session.open_cursor('backup:', None, config), 0), msg) # Open a non-incremental full backup cursor. # - An incremental duplicate must have an incremental primary. + self.pr("Try to open an incremental on a non-incremental primary") bkup_c = self.session.open_cursor('backup:', None, None) msg = "/must have an incremental primary/" self.assertRaisesWithMessage(wiredtiger.WiredTigerError, @@ -229,13 +185,12 @@ class test_backup11(wttest.WiredTigerTestCase, suite_subprocess): bkup_c, config), 0), msg) # - Force stop must be on primary, not duplicate. - #self.pr("Test force stop") - #self.pr("=========") - #config = 'incremental=(force_stop=true)' - #print "config is " + config - #self.assertRaisesWithMessage(wiredtiger.WiredTigerError, - # lambda:self.assertEquals(self.session.open_cursor(None, - # bkup_c, config), 0), msg) + self.pr("Test force stop") + self.pr("=========") + config = 'incremental=(force_stop=true)' + self.assertRaisesWithMessage(wiredtiger.WiredTigerError, + lambda:self.assertEquals(self.session.open_cursor(None, + bkup_c, config), 0), msg) bkup_c.close() @@ -255,9 +210,61 @@ class test_backup11(wttest.WiredTigerTestCase, suite_subprocess): lambda: self.session.open_cursor(None, bkup_c, config), msg) bkup_c.close() + # - Test opening a primary backup with an unknown source id. + self.pr("Test incremental with unknown source identifier on primary") + self.pr("=========") + config = 'incremental=(enabled,src_id="ID_BAD",this_id="ID4")' + self.assertRaises(wiredtiger.WiredTigerError, + lambda: self.session.open_cursor('backup:', None, config)) + + # - Test opening a primary backup with an id in WiredTiger namespace. + self.pr("Test incremental with illegal src identifier using WiredTiger namespace") + self.pr("=========") + msg = '/name space may not/' + config = 'incremental=(enabled,src_id="WiredTiger.0")' + self.assertRaisesWithMessage(wiredtiger.WiredTigerError, + lambda: self.session.open_cursor('backup:', None, config), msg) + + # - Test opening a primary backup with an id in WiredTiger namespace. + self.pr("Test incremental with illegal this identifier using WiredTiger namespace") + self.pr("=========") + config = 'incremental=(enabled,this_id="WiredTiger.ID")' + self.assertRaisesWithMessage(wiredtiger.WiredTigerError, + lambda: self.session.open_cursor('backup:', None, config), msg) + + # - Test opening a primary backup with an id using illegal characters. + self.pr("Test incremental with illegal source identifier using illegal colon character") + self.pr("=========") + msg = '/grouping characters/' + config = 'incremental=(enabled,src_id="ID4:4.0")' + self.assertRaisesWithMessage(wiredtiger.WiredTigerError, + lambda: self.session.open_cursor('backup:', None, config), msg) + + # - Test opening a primary backup with an id using illegal characters. + self.pr("Test incremental with illegal this identifier using illegal colon character") + self.pr("=========") + config = 'incremental=(enabled,this_id="ID4:4.0")' + self.assertRaisesWithMessage(wiredtiger.WiredTigerError, + lambda: self.session.open_cursor('backup:', None, config), msg) + + # - Test opening a primary backup with the same source id and this id (new id). + self.pr("Test incremental with the same new source and this identifiers") + self.pr("=========") + config = 'incremental=(enabled,src_id="IDSAME",this_id="IDSAME")' + self.assertRaises(wiredtiger.WiredTigerError, + lambda: self.session.open_cursor('backup:', None, config)) + + # - Test opening a primary backup with the same source id and this id (reusing id). + self.pr("Test incremental with the same re-used source and this identifiers") + self.pr("=========") + msg = '/already in use/' + config = 'incremental=(enabled,src_id="ID2",this_id="ID2")' + self.assertRaisesWithMessage(wiredtiger.WiredTigerError, + lambda: self.session.open_cursor('backup:', None, config), msg) + # After the full backup, open and recover the backup database. - #backup_conn = self.wiredtiger_open(self.dir) - #backup_conn.close() + backup_conn = self.wiredtiger_open(self.dir) + backup_conn.close() if __name__ == '__main__': wttest.run() diff --git a/src/third_party/wiredtiger/test/suite/test_backup12.py b/src/third_party/wiredtiger/test/suite/test_backup12.py index 35948da5c44..f5fadcee393 100644 --- a/src/third_party/wiredtiger/test/suite/test_backup12.py +++ b/src/third_party/wiredtiger/test/suite/test_backup12.py @@ -36,48 +36,41 @@ from wtscenario import make_scenarios # test_backup12.py # Test cursor backup with a block-based incremental cursor. class test_backup12(wttest.WiredTigerTestCase, suite_subprocess): + conn_config='cache_size=1G,log=(enabled,file_max=100K)' dir='backup.dir' # Backup directory name logmax="100K" uri="table:test" - nops=100 + uri2="table:test2" + uri_rem="table:test_rem" + nops=1000 + mult=0 pfx = 'test_backup' + # Set the key and value big enough that we modify a few blocks. + bigkey = 'Key' * 100 + bigval = 'Value' * 100 - # ('archiving', dict(archive='true')), - # ('not-archiving', dict(archive='false')), - scenarios = make_scenarios([ - ('archiving', dict(archive='true')), - ]) + def add_data(self, uri): - # Create a large cache, otherwise this test runs quite slowly. - def conn_config(self): - return 'cache_size=1G,log=(archive=%s,' % self.archive + \ - 'enabled,file_max=%s)' % self.logmax - - def add_data(self): - log2 = "WiredTigerLog.0000000002" - log3 = "WiredTigerLog.0000000003" - - self.session.create(self.uri, "key_format=S,value_format=S") - - # Insert small amounts of data at a time stopping after we - # cross into log file 2. - loop = 0 - c = self.session.open_cursor(self.uri) - while not os.path.exists(log2): - for i in range(0, self.nops): - num = i + (loop * self.nops) - key = 'key' + str(num) - val = 'value' + str(num) - c[key] = val - loop += 1 + c = self.session.open_cursor(uri) + for i in range(0, self.nops): + num = i + (self.mult * self.nops) + key = self.bigkey + str(num) + val = self.bigval + str(num) + c[key] = val self.session.checkpoint() c.close() - return loop + # Increase the multiplier so that later calls insert unique items. + self.mult += 1 def test_backup12(self): - loop = self.add_data() + self.session.create(self.uri, "key_format=S,value_format=S") + self.session.create(self.uri2, "key_format=S,value_format=S") + self.session.create(self.uri_rem, "key_format=S,value_format=S") + self.add_data(self.uri) + self.add_data(self.uri2) + self.add_data(self.uri_rem) # Open up the backup cursor. This causes a new log file to be created. # That log file is not part of the list returned. This is a full backup @@ -86,23 +79,14 @@ class test_backup12(wttest.WiredTigerTestCase, suite_subprocess): # # Note, this first backup is actually done before a checkpoint is taken. # - config = 'incremental=(enabled,this_id="ID1")' + config = 'incremental=(enabled,granularity=1M,this_id="ID1")' bkup_c = self.session.open_cursor('backup:', None, config) - # Add some data that will appear in log file 3. - c = self.session.open_cursor(self.uri) - for i in range(0, self.nops): - num = i + (loop * self.nops) - key = 'key' + str(num) - val = 'value' + str(num) - c[key] = val - loop += 1 - c.close() - self.session.log_flush('sync=on') - self.session.checkpoint() + # Add more data while the backup cursor is open. + self.add_data(self.uri) # Now copy the files returned by the backup cursor. - orig_logs = [] + all_files = [] while True: ret = bkup_c.next() if ret != 0: @@ -111,8 +95,7 @@ class test_backup12(wttest.WiredTigerTestCase, suite_subprocess): sz = os.path.getsize(newfile) self.pr('Copy from: ' + newfile + ' (' + str(sz) + ') to ' + self.dir) shutil.copy(newfile, self.dir) - if "WiredTigerLog" in newfile: - orig_logs.append(newfile) + all_files.append(newfile) self.assertEqual(ret, wiredtiger.WT_NOTFOUND) # Now open a duplicate backup cursor. @@ -129,31 +112,28 @@ class test_backup12(wttest.WiredTigerTestCase, suite_subprocess): newfile = dupc.get_key() self.assertTrue("WiredTigerLog" in newfile) sz = os.path.getsize(newfile) - if (newfile not in orig_logs): + if (newfile not in all_files): self.pr('DUP: Copy from: ' + newfile + ' (' + str(sz) + ') to ' + self.dir) shutil.copy(newfile, self.dir) # Record all log files returned for later verification. dup_logs.append(newfile) + all_files.append(newfile) self.assertEqual(ret, wiredtiger.WT_NOTFOUND) dupc.close() bkup_c.close() # Add more data. - c = self.session.open_cursor(self.uri) - for i in range(0, self.nops): - num = i + (loop * self.nops) - key = 'key' + str(num) - val = 'value' + str(num) - c[key] = val - loop += 1 - c.close() - self.session.log_flush('sync=on') - self.session.checkpoint() + self.add_data(self.uri) + self.add_data(self.uri2) + + # Drop a table. + self.session.drop(self.uri_rem) # Now do an incremental backup. config = 'incremental=(src_id="ID1",this_id="ID2")' bkup_c = self.session.open_cursor('backup:', None, config) self.pr('Open backup cursor ID1') + bkup_files = [] while True: ret = bkup_c.next() if ret != 0: @@ -163,6 +143,8 @@ class test_backup12(wttest.WiredTigerTestCase, suite_subprocess): self.pr('Open incremental cursor with ' + config) dup_cnt = 0 dupc = self.session.open_cursor(None, bkup_c, config) + bkup_files.append(newfile) + all_files.append(newfile) while True: ret = dupc.next() if ret != 0: @@ -171,14 +153,34 @@ class test_backup12(wttest.WiredTigerTestCase, suite_subprocess): offset = incrlist[0] size = incrlist[1] curtype = incrlist[2] + # 1 is WT_BACKUP_FILE + # 2 is WT_BACKUP_RANGE self.assertTrue(curtype == 1 or curtype == 2) + if curtype == 1: + self.pr('Copy from: ' + newfile + ' (' + str(sz) + ') to ' + self.dir) + shutil.copy(newfile, self.dir) + else: + self.pr('Range copy file ' + newfile + ' offset ' + str(offset) + ' len ' + str(size)) + rfp = open(newfile, "r+b") + wfp = open(self.dir + '/' + newfile, "w+b") + rfp.seek(offset, 0) + wfp.seek(offset, 0) + buf = rfp.read(size) + wfp.write(buf) + rfp.close() + wfp.close() dup_cnt += 1 dupc.close() - self.pr('Copy from: ' + newfile + ' (' + str(sz) + ') to ' + self.dir) - shutil.copy(newfile, self.dir) self.assertEqual(ret, wiredtiger.WT_NOTFOUND) bkup_c.close() + # We need to remove files in the backup directory that are not in the current backup. + all_set = set(all_files) + bkup_set = set(bkup_files) + rem_files = list(all_set - bkup_set) + for l in rem_files: + self.pr('Remove file: ' + self.dir + '/' + l) + os.remove(self.dir + '/' + l) # After the full backup, open and recover the backup database. backup_conn = self.wiredtiger_open(self.dir) backup_conn.close() diff --git a/src/third_party/wiredtiger/test/suite/test_backup13.py b/src/third_party/wiredtiger/test/suite/test_backup13.py new file mode 100644 index 00000000000..445cbaa6dc1 --- /dev/null +++ b/src/third_party/wiredtiger/test/suite/test_backup13.py @@ -0,0 +1,168 @@ +#!/usr/bin/env python +# +# Public Domain 2014-2020 MongoDB, Inc. +# Public Domain 2008-2014 WiredTiger, Inc. +# +# This is free and unencumbered software released into the public domain. +# +# Anyone is free to copy, modify, publish, use, compile, sell, or +# distribute this software, either in source code form or as a compiled +# binary, for any purpose, commercial or non-commercial, and by any +# means. +# +# In jurisdictions that recognize copyright laws, the author or authors +# of this software dedicate any and all copyright interest in the +# software to the public domain. We make this dedication for the benefit +# of the public at large and to the detriment of our heirs and +# successors. We intend this dedication to be an overt act of +# relinquishment in perpetuity of all present and future rights to this +# software under copyright law. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +# IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. + +import wiredtiger, wttest +import os, shutil +from helper import compare_files +from suite_subprocess import suite_subprocess +from wtdataset import simple_key +from wtscenario import make_scenarios + +# test_backup13.py +# Test cursor backup with a block-based incremental cursor and force_stop. +class test_backup13(wttest.WiredTigerTestCase, suite_subprocess): + conn_config='cache_size=1G,log=(enabled,file_max=100K)' + dir='backup.dir' # Backup directory name + logmax="100K" + uri="table:test" + nops=1000 + mult=0 + + pfx = 'test_backup' + # Set the key and value big enough that we modify a few blocks. + bigkey = 'Key' * 100 + bigval = 'Value' * 100 + + def add_data(self, uri): + + c = self.session.open_cursor(uri) + for i in range(0, self.nops): + num = i + (self.mult * self.nops) + key = self.bigkey + str(num) + val = self.bigval + str(num) + c[key] = val + self.session.checkpoint() + c.close() + # Increase the multiplier so that later calls insert unique items. + self.mult += 1 + + def test_backup13(self): + + self.session.create(self.uri, "key_format=S,value_format=S") + self.add_data(self.uri) + + # Open up the backup cursor. This causes a new log file to be created. + # That log file is not part of the list returned. This is a full backup + # primary cursor with incremental configured. + os.mkdir(self.dir) + config = 'incremental=(enabled,granularity=1M,this_id="ID1")' + bkup_c = self.session.open_cursor('backup:', None, config) + + # Add more data while the backup cursor is open. + self.add_data(self.uri) + + # Now copy the files returned by the backup cursor. + all_files = [] + + # We cannot use 'for newfile in bkup_c:' usage because backup cursors don't have + # values and adding in get_values returns ENOTSUP and causes the usage to fail. + # If that changes then this, and the use of the duplicate below can change. + while True: + ret = bkup_c.next() + if ret != 0: + break + newfile = bkup_c.get_key() + sz = os.path.getsize(newfile) + self.pr('Copy from: ' + newfile + ' (' + str(sz) + ') to ' + self.dir) + shutil.copy(newfile, self.dir) + all_files.append(newfile) + self.assertEqual(ret, wiredtiger.WT_NOTFOUND) + bkup_c.close() + + # Add more data. + self.add_data(self.uri) + + # Now do an incremental backup. + config = 'incremental=(src_id="ID1",this_id="ID2")' + bkup_c = self.session.open_cursor('backup:', None, config) + self.pr('Open backup cursor ID1') + bkup_files = [] + while True: + ret = bkup_c.next() + if ret != 0: + break + newfile = bkup_c.get_key() + config = 'incremental=(file=' + newfile + ')' + self.pr('Open incremental cursor with ' + config) + dup_cnt = 0 + dupc = self.session.open_cursor(None, bkup_c, config) + bkup_files.append(newfile) + all_files.append(newfile) + while True: + ret = dupc.next() + if ret != 0: + break + incrlist = dupc.get_keys() + offset = incrlist[0] + size = incrlist[1] + curtype = incrlist[2] + self.assertTrue(curtype == wiredtiger.WT_BACKUP_FILE or curtype == wiredtiger.WT_BACKUP_RANGE) + if curtype == wiredtiger.WT_BACKUP_FILE: + self.pr('Copy from: ' + newfile + ' (' + str(sz) + ') to ' + self.dir) + shutil.copy(newfile, self.dir) + else: + self.pr('Range copy file ' + newfile + ' offset ' + str(offset) + ' len ' + str(size)) + rfp = open(newfile, "r+b") + wfp = open(self.dir + '/' + newfile, "w+b") + rfp.seek(offset, 0) + wfp.seek(offset, 0) + buf = rfp.read(size) + wfp.write(buf) + rfp.close() + wfp.close() + dup_cnt += 1 + dupc.close() + self.assertEqual(ret, wiredtiger.WT_NOTFOUND) + bkup_c.close() + + all_set = set(all_files) + bkup_set = set(bkup_files) + rem_files = list(all_set - bkup_set) + for l in rem_files: + self.pr('Remove file: ' + self.dir + '/' + l) + os.remove(self.dir + '/' + l) + # After the full backup, open and recover the backup database. + backup_conn = self.wiredtiger_open(self.dir) + backup_conn.close() + + # Do a force stop to release resources and reset the system. + config = 'incremental=(force_stop=true)' + bkup_c = self.session.open_cursor('backup:', None, config) + bkup_c.close() + + # Make sure after a force stop we cannot access old backup info. + config = 'incremental=(src_id="ID1",this_id="ID3")' + self.assertRaises(wiredtiger.WiredTigerError, + lambda: self.session.open_cursor('backup:', None, config)) + self.reopen_conn() + # Make sure after a restart we cannot access old backup info. + self.assertRaises(wiredtiger.WiredTigerError, + lambda: self.session.open_cursor('backup:', None, config)) + +if __name__ == '__main__': + wttest.run() diff --git a/src/third_party/wiredtiger/test/suite/test_backup14.py b/src/third_party/wiredtiger/test/suite/test_backup14.py new file mode 100644 index 00000000000..7a2ec4f427f --- /dev/null +++ b/src/third_party/wiredtiger/test/suite/test_backup14.py @@ -0,0 +1,367 @@ +#!/usr/bin/env python +# +# Public Domain 2014-2020 MongoDB, Inc. +# Public Domain 2008-2014 WiredTiger, Inc. +# +# This is free and unencumbered software released into the public domain. +# +# Anyone is free to copy, modify, publish, use, compile, sell, or +# distribute this software, either in source code form or as a compiled +# binary, for any purpose, commercial or non-commercial, and by any +# means. +# +# In jurisdictions that recognize copyright laws, the author or authors +# of this software dedicate any and all copyright interest in the +# software to the public domain. We make this dedication for the benefit +# of the public at large and to the detriment of our heirs and +# successors. We intend this dedication to be an overt act of +# relinquishment in perpetuity of all present and future rights to this +# software under copyright law. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +# IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. + +import wiredtiger, wttest +import os, shutil +from helper import compare_files +from suite_subprocess import suite_subprocess +from wtdataset import simple_key +from wtscenario import make_scenarios +import glob + +# test_backup14.py +# Test cursor backup with a block-based incremental cursor. +class test_backup14(wttest.WiredTigerTestCase, suite_subprocess): + conn_config='cache_size=1G,log=(enabled,file_max=100K)' + dir='backup.dir' # Backup directory name + logmax="100K" + uri="table:main" + uri2="table:extra" + uri_logged="table:logged_table" + uri_not_logged="table:not_logged_table" + full_out = "./backup_block_full" + incr_out = "./backup_block_incr" + bkp_home = "WT_BLOCK" + home_full = "WT_BLOCK_LOG_FULL" + home_incr = "WT_BLOCK_LOG_INCR" + logpath = "logpath" + nops=1000 + mult=0 + max_iteration=7 + counter=0 + new_table=False + initial_backup=False + + pfx = 'test_backup' + # Set the key and value big enough that we modify a few blocks. + bigkey = 'Key' * 100 + bigval = 'Value' * 100 + + # + # Set up all the directories needed for the test. We have a full backup directory for each + # iteration and an incremental backup for each iteration. That way we can compare the full and + # incremental each time through. + # + def setup_directories(self): + for i in range(0, self.max_iteration): + remove_dir = self.home_incr + '.' + str(i) + + create_dir = self.home_incr + '.' + str(i) + '/' + self.logpath + if os.path.exists(remove_dir): + os.remove(remove_dir) + os.makedirs(create_dir) + + if i == 0: + continue + remove_dir = self.home_full + '.' + str(i) + create_dir = self.home_full + '.' + str(i) + '/' + self.logpath + if os.path.exists(remove_dir): + os.remove(remove_dir) + os.makedirs(create_dir) + + def take_full_backup(self): + if self.counter != 0: + hdir = self.home_full + '.' + str(self.counter) + else: + hdir = self.home_incr + + # + # First time through we take a full backup into the incremental directories. Otherwise only + # into the appropriate full directory. + # + buf = None + if self.initial_backup == True: + buf = 'incremental=(granularity=1M,enabled=true,this_id=ID0)' + + cursor = self.session.open_cursor('backup:', None, buf) + while True: + ret = cursor.next() + if ret != 0: + break + newfile = cursor.get_key() + + if self.counter == 0: + # Take a full bakcup into each incremental directory + for i in range(0, self.max_iteration): + copy_from = newfile + # If it is log file, prepend the path. + if ("WiredTigerLog" in newfile): + copy_to = self.home_incr + '.' + str(i) + '/' + self.logpath + else: + copy_to = self.home_incr + '.' + str(i) + shutil.copy(copy_from, copy_to) + else: + copy_from = newfile + # If it is log file, prepend the path. + if ("WiredTigerLog" in newfile): + copy_to = hdir + '/' + self.logpath + else: + copy_to = hdir + + shutil.copy(copy_from, copy_to) + self.assertEqual(ret, wiredtiger.WT_NOTFOUND) + cursor.close() + + def take_incr_backup(self): + # Open the backup data source for incremental backup. + buf = 'incremental=(src_id="ID' + str(self.counter-1) + '",this_id="ID' + str(self.counter) + '")' + bkup_c = self.session.open_cursor('backup:', None, buf) + while True: + ret = bkup_c.next() + if ret != 0: + break + newfile = bkup_c.get_key() + h = self.home_incr + '.0' + copy_from = newfile + # If it is log file, prepend the path. + if ("WiredTigerLog" in newfile): + copy_to = h + '/' + self.logpath + else: + copy_to = h + + shutil.copy(copy_from, copy_to) + first = True + config = 'incremental=(file=' + newfile + ')' + dup_cnt = 0 + incr_c = self.session.open_cursor(None, bkup_c, config) + + # For each file listed, open a duplicate backup cursor and copy the blocks. + while True: + ret = incr_c.next() + if ret != 0: + break + incrlist = incr_c.get_keys() + offset = incrlist[0] + size = incrlist[1] + curtype = incrlist[2] + # 1 is WT_BACKUP_FILE + # 2 is WT_BACKUP_RANGE + self.assertTrue(curtype == 1 or curtype == 2) + if curtype == 1: + if first == True: + h = self.home_incr + '.' + str(self.counter) + first = False + + copy_from = newfile + if ("WiredTigerLog" in newfile): + copy_to = h + '/' + self.logpath + else: + copy_to = h + shutil.copy(copy_from, copy_to) + else: + self.pr('Range copy file ' + newfile + ' offset ' + str(offset) + ' len ' + str(size)) + write_from = newfile + write_to = self.home_incr + '.' + str(self.counter) + '/' + newfile + rfp = open(write_from, "r+b") + wfp = open(write_to, "w+b") + rfp.seek(offset, 0) + wfp.seek(offset, 0) + buf = rfp.read(size) + wfp.write(buf) + rfp.close() + wfp.close() + dup_cnt += 1 + self.assertEqual(ret, wiredtiger.WT_NOTFOUND) + incr_c.close() + + # For each file, we want to copy the file into each of the later incremental directories + for i in range(self.counter, self.max_iteration): + h = self.home_incr + '.' + str(i) + copy_from = newfile + if ("WiredTigerLog" in newfile): + copy_to = h + '/' + self.logpath + else: + copy_to = h + shutil.copy(copy_from, copy_to) + self.assertEqual(ret, wiredtiger.WT_NOTFOUND) + bkup_c.close() + + def compare_backups(self, t_uri): + # + # Run wt dump on full backup directory + # + full_backup_out = self.full_out + '.' + str(self.counter) + home_dir = self.home_full + '.' + str(self.counter) + if self.counter == 0: + home_dir = self.home + + self.runWt(['-R', '-h', home_dir, 'dump', t_uri], outfilename=full_backup_out) + # + # Run wt dump on incremental backup directory + # + incr_backup_out = self.incr_out + '.' + str(self.counter) + home_dir = self.home_incr + '.' + str(self.counter) + self.runWt(['-R', '-h', home_dir, 'dump', t_uri], outfilename=incr_backup_out) + + self.assertEqual(True, + compare_files(self, full_backup_out, incr_backup_out)) + + # + # Add data to the given uri. + # + def add_data(self, uri, bulk_option): + c = self.session.open_cursor(uri, None, bulk_option) + for i in range(0, self.nops): + num = i + (self.mult * self.nops) + key = self.bigkey + str(num) + val = self.bigval + str(num) + c[key] = val + c.close() + + # Increase the multiplier so that later calls insert unique items. + self.mult += 1 + # Increase the counter so that later backups have unique ids. + if self.initial_backup == False: + self.counter += 1 + + # + # Remove data from uri (table:main) + # + def remove_data(self): + c = self.session.open_cursor(self.uri) + # + # We run the outer loop until mult value to make sure we remove all the inserted records + # from the main table. + # + for i in range(0, self.mult): + for j in range(i, self.nops): + num = j + (i * self.nops) + key = self.bigkey + str(num) + c.set_key(key) + self.assertEquals(c.remove(), 0) + c.close() + # Increase the counter so that later backups have unique ids. + self.counter += 1 + + # + # This function will add records to the table (table:main), take incremental/full backups and + # validate the backups. + # + def add_data_validate_backups(self): + self.pr('Adding initial data') + self.initial_backup = True + self.add_data(self.uri, None) + self.take_full_backup() + self.initial_backup = False + self.session.checkpoint() + + self.add_data(self.uri, None) + self.take_full_backup() + self.take_incr_backup() + self.compare_backups(self.uri) + + # + # This function will remove all the records from table (table:main), take backup and validate the + # backup. + # + def remove_all_records_validate(self): + self.remove_data() + self.take_full_backup() + self.take_incr_backup() + self.compare_backups(self.uri) + + # + # This function will drop the existing table uri (table:main) that is part of the backups and + # create new table uri2 (table:extra), take incremental backup and validate. + # + def drop_old_add_new_table(self): + + # Drop main table. + self.session.drop(self.uri) + + # Create uri2 (table:extra) + self.session.create(self.uri2, "key_format=S,value_format=S") + + self.new_table = True + self.add_data(self.uri2, None) + self.take_incr_backup() + + table_list = 'tablelist.txt' + # Assert if the dropped table (table:main) exists in the incremental folder. + self.runWt(['-R', '-h', self.home, 'list'], outfilename=table_list) + ret = os.system("grep " + self.uri + " " + table_list) + self.assertNotEqual(ret, 0, self.uri + " dropped, but table exists in " + self.home) + + # + # This function will create previously dropped table uri (table:main) and add different content to + # it, take backups and validate the backups. + # + def create_dropped_table_add_new_content(self): + self.session.create(self.uri, "key_format=S,value_format=S") + self.add_data(self.uri, None) + self.take_full_backup() + self.take_incr_backup() + self.compare_backups(self.uri) + + # + # This function will insert bulk data in logged and not-logged table, take backups and validate the + # backups. + # + def insert_bulk_data(self): + # + # Insert bulk data into uri3 (table:logged_table). + # + self.session.create(self.uri_logged, "key_format=S,value_format=S") + self.add_data(self.uri_logged, 'bulk') + self.take_full_backup() + self.take_incr_backup() + self.compare_backups(self.uri_logged) + + # + # Insert bulk data into uri4 (table:not_logged_table). + # + self.session.create(self.uri_not_logged, "key_format=S,value_format=S,log=(enabled=false)") + self.add_data(self.uri_not_logged, 'bulk') + self.take_full_backup() + self.take_incr_backup() + self.compare_backups(self.uri_not_logged) + + def test_backup14(self): + os.mkdir(self.bkp_home) + self.home = self.bkp_home + self.session.create(self.uri, "key_format=S,value_format=S") + + self.setup_directories() + + self.pr('*** Add data, checkpoint, take backups and validate ***') + self.add_data_validate_backups() + + self.pr('*** Remove old records and validate ***') + self.remove_all_records_validate() + + self.pr('*** Drop old and add new table ***') + self.drop_old_add_new_table() + + self.pr('*** Create previously dropped table and add new content ***') + self.create_dropped_table_add_new_content() + + self.pr('*** Insert data into Logged and Not-Logged tables ***') + self.insert_bulk_data() + +if __name__ == '__main__': + wttest.run() diff --git a/src/third_party/wiredtiger/test/suite/test_bug008.py b/src/third_party/wiredtiger/test/suite/test_bug008.py index cb2987c234a..6af51e1eaba 100644 --- a/src/third_party/wiredtiger/test/suite/test_bug008.py +++ b/src/third_party/wiredtiger/test/suite/test_bug008.py @@ -29,7 +29,7 @@ # test_bug008.py # Regression tests. -import wiredtiger, wttest +import unittest, wiredtiger, wttest from wtdataset import SimpleDataSet from wtscenario import make_scenarios diff --git a/src/third_party/wiredtiger/test/suite/test_bug022.py b/src/third_party/wiredtiger/test/suite/test_bug022.py new file mode 100644 index 00000000000..b5b245f6ba7 --- /dev/null +++ b/src/third_party/wiredtiger/test/suite/test_bug022.py @@ -0,0 +1,72 @@ +#!/usr/bin/env python +# +# Public Domain 2014-2020 MongoDB, Inc. +# Public Domain 2008-2014 WiredTiger, Inc. +# +# This is free and unencumbered software released into the public domain. +# +# Anyone is free to copy, modify, publish, use, compile, sell, or +# distribute this software, either in source code form or as a compiled +# binary, for any purpose, commercial or non-commercial, and by any +# means. +# +# In jurisdictions that recognize copyright laws, the author or authors +# of this software dedicate any and all copyright interest in the +# software to the public domain. We make this dedication for the benefit +# of the public at large and to the detriment of our heirs and +# successors. We intend this dedication to be an overt act of +# relinquishment in perpetuity of all present and future rights to this +# software under copyright law. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +# IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. +# +# test_bug022.py +# Testing that we don't allow modifies on top of tombstone updates. + +import wiredtiger, wttest + +def timestamp_str(t): + return '%x' % t + +class test_bug022(wttest.WiredTigerTestCase): + uri = 'file:test_bug022' + conn_config = 'cache_size=50MB' + session_config = 'isolation=snapshot' + + def test_apply_modifies_on_onpage_tombstone(self): + self.session.create(self.uri, 'key_format=S,value_format=S') + self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(1)) + cursor = self.session.open_cursor(self.uri) + + value = 'a' * 500 + for i in range(1, 10000): + self.session.begin_transaction() + cursor[str(i)] = value + self.session.commit_transaction('commit_timestamp=' + timestamp_str(2)) + + # Apply tombstones for every key. + for i in range(1, 10000): + self.session.begin_transaction() + cursor.set_key(str(i)) + cursor.remove() + self.session.commit_transaction('commit_timestamp=' + timestamp_str(3)) + + self.session.checkpoint() + + # Now try to apply a modify on top of the tombstone at timestamp 3. + for i in range(1, 10000): + self.session.begin_transaction() + cursor.set_key(str(i)) + self.assertEqual(cursor.modify([wiredtiger.Modify('B', 0, 100)]), wiredtiger.WT_NOTFOUND) + self.session.rollback_transaction() + + # Check that the tombstone is visible. + for i in range(1, 10000): + cursor.set_key(str(i)) + self.assertEqual(cursor.search(), wiredtiger.WT_NOTFOUND) diff --git a/src/third_party/wiredtiger/test/suite/test_checkpoint03.py b/src/third_party/wiredtiger/test/suite/test_checkpoint03.py new file mode 100644 index 00000000000..9e2c299404f --- /dev/null +++ b/src/third_party/wiredtiger/test/suite/test_checkpoint03.py @@ -0,0 +1,106 @@ +#!/usr/bin/env python +# +# Public Domain 2014-2020 MongoDB, Inc. +# Public Domain 2008-2014 WiredTiger, Inc. +# +# This is free and unencumbered software released into the public domain. +# +# Anyone is free to copy, modify, publish, use, compile, sell, or +# distribute this software, either in source code form or as a compiled +# binary, for any purpose, commercial or non-commercial, and by any +# means. +# +# In jurisdictions that recognize copyright laws, the author or authors +# of this software dedicate any and all copyright interest in the +# software to the public domain. We make this dedication for the benefit +# of the public at large and to the detriment of our heirs and +# successors. We intend this dedication to be an overt act of +# relinquishment in perpetuity of all present and future rights to this +# software under copyright law. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +# IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. +# +# test_checkpoint03.py +# Test that checkpoint writes out updates to the history store file. +# + +from suite_subprocess import suite_subprocess +import wiredtiger, wttest +from wiredtiger import stat +from wtscenario import make_scenarios + +def timestamp_str(t): + return '%x' % t + +class test_checkpoint03(wttest.WiredTigerTestCase, suite_subprocess): + tablename = 'test_checkpoint03' + conn_config = 'statistics=(all)' + uri = 'table:' + tablename + session_config = 'isolation=snapshot, ' + + def get_stat(self, stat): + stat_cursor = self.session.open_cursor('statistics:') + val = stat_cursor[stat][2] + stat_cursor.close() + return val + + def test_checkpoint_writes_to_hs(self): + # Create a basic table. + self.session.create(self.uri, 'key_format=i,value_format=i') + self.session.begin_transaction() + self.conn.set_timestamp('oldest_timestamp=1') + + # Insert 3 updates in separate transactions. + cur1 = self.session.open_cursor(self.uri) + cur1[1] = 1 + self.session.commit_transaction('commit_timestamp=2') + + self.session.begin_transaction() + cur1[1] = 2 + self.session.commit_transaction('commit_timestamp=3') + + self.session.begin_transaction() + cur1[1] = 3 + self.session.commit_transaction('commit_timestamp=4') + + # Call checkpoint. + self.session.checkpoint() + + # Validate that we wrote to history store, note that the history store statistic is not + # counting how many writes we did, just that we did write. Hence for multiple writes it may + # only increment a single time. + hs_writes = self.get_stat(stat.conn.cache_write_hs) + self.assertGreaterEqual(hs_writes, 1) + + # Add a new update. + self.session.begin_transaction() + cur1[1] = 4 + self.session.commit_transaction('commit_timestamp=5') + self.session.checkpoint() + + # Check that we wrote something to the history store in the last checkpoint we ran, as we + # should've written the previous update. + hs_writes = self.get_stat(stat.conn.cache_write_hs) + self.assertGreaterEqual(hs_writes, 2) + + # Close the connection. + self.close_conn() + + # Open a new connection and validate that we see the latest update as part of the datafile. + conn2 = self.setUpConnectionOpen('.') + session2 = self.setUpSessionOpen(conn2) + session2.create(self.uri, 'key_format=i,value_format=i') + + cur2 = session2.open_cursor(self.uri) + cur2.set_key(1) + cur2.search() + self.assertEqual(cur2.get_value(), 4) + +if __name__ == '__main__': + wttest.run() diff --git a/src/third_party/wiredtiger/test/suite/test_checkpoint04.py b/src/third_party/wiredtiger/test/suite/test_checkpoint04.py new file mode 100644 index 00000000000..5ba0612c680 --- /dev/null +++ b/src/third_party/wiredtiger/test/suite/test_checkpoint04.py @@ -0,0 +1,107 @@ +#!/usr/bin/env python +# +# Public Domain 2014-2020 MongoDB, Inc. +# Public Domain 2008-2014 WiredTiger, Inc. +# +# This is free and unencumbered software released into the public domain. +# +# Anyone is free to copy, modify, publish, use, compile, sell, or +# distribute this software, either in source code form or as a compiled +# binary, for any purpose, commercial or non-commercial, and by any +# means. +# +# In jurisdictions that recognize copyright laws, the author or authors +# of this software dedicate any and all copyright interest in the +# software to the public domain. We make this dedication for the benefit +# of the public at large and to the detriment of our heirs and +# successors. We intend this dedication to be an overt act of +# relinquishment in perpetuity of all present and future rights to this +# software under copyright law. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +# IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. +# +# test_checkpoint04.py +# Test that the checkpoints timing statistics are populated as expected. + +import wiredtiger, wttest +from wiredtiger import stat +from wtdataset import SimpleDataSet + +def timestamp_str(t): + return '%x' % t + +class test_checkpoint04(wttest.WiredTigerTestCase): + conn_config = 'cache_size=50MB,log=(enabled),statistics=(all)' + session_config = 'isolation=snapshot' + + def create_tables(self, ntables): + tables = {} + for i in range(0, ntables): + uri = 'table:table' + str(i) + ds = SimpleDataSet( + self, uri, 0, key_format="i", value_format="S", config='log=(enabled=false)') + ds.populate() + tables[uri] = ds + return tables + + def add_updates(self, uri, ds, value, nrows, ts): + session = self.session + cursor = session.open_cursor(uri) + for i in range(0, nrows): + session.begin_transaction() + cursor[ds.key(i)] = value + session.commit_transaction('commit_timestamp=' + timestamp_str(ts)) + cursor.close() + + def get_stat(self, stat): + stat_cursor = self.session.open_cursor('statistics:') + val = stat_cursor[stat][2] + stat_cursor.close() + return val + + def test_checkpoint_stats(self): + nrows = 1000 + ntables = 50 + + self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(10) + + ',stable_timestamp=' + timestamp_str(10)) + + # Create many tables and perform many updates so our checkpoint stats are populated. + value = "wired" * 100 + tables = self.create_tables(ntables) + for uri, ds in tables.items(): + self.add_updates(uri, ds, value, nrows, 20) + + # Perform a checkpoint. + self.session.checkpoint() + + # Update the tables. + value = "tiger" * 100 + tables = self.create_tables(ntables) + for uri, ds in tables.items(): + self.add_updates(uri, ds, value, nrows, 30) + + # Perform a checkpoint. + self.session.checkpoint() + + # Check the statistics. + self.assertEqual(self.get_stat(stat.conn.txn_checkpoint), 2) + self.assertEqual(self.get_stat(stat.conn.txn_checkpoint_running), 0) + self.assertEqual(self.get_stat(stat.conn.txn_checkpoint_prep_running), 0) + self.assertLess(self.get_stat(stat.conn.txn_checkpoint_prep_min), + self.get_stat(stat.conn.txn_checkpoint_time_min)) + self.assertLess(self.get_stat(stat.conn.txn_checkpoint_prep_max), + self.get_stat(stat.conn.txn_checkpoint_time_max)) + self.assertLess(self.get_stat(stat.conn.txn_checkpoint_prep_recent), + self.get_stat(stat.conn.txn_checkpoint_time_recent)) + self.assertLess(self.get_stat(stat.conn.txn_checkpoint_prep_total), + self.get_stat(stat.conn.txn_checkpoint_time_total)) + +if __name__ == '__main__': + wttest.run() diff --git a/src/third_party/wiredtiger/test/suite/test_compact01.py b/src/third_party/wiredtiger/test/suite/test_compact01.py index 9e5d0e19c5e..0486429b972 100644 --- a/src/third_party/wiredtiger/test/suite/test_compact01.py +++ b/src/third_party/wiredtiger/test/suite/test_compact01.py @@ -26,7 +26,7 @@ # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. -import wiredtiger, wttest +import unittest, wiredtiger, wttest from suite_subprocess import suite_subprocess from wtdataset import SimpleDataSet, ComplexDataSet from wiredtiger import stat diff --git a/src/third_party/wiredtiger/test/suite/test_compact02.py b/src/third_party/wiredtiger/test/suite/test_compact02.py index c15fb5bc78b..ccbec469433 100644 --- a/src/third_party/wiredtiger/test/suite/test_compact02.py +++ b/src/third_party/wiredtiger/test/suite/test_compact02.py @@ -30,7 +30,7 @@ # Test that compact reduces the file size. # -import time, wiredtiger, wttest +import time, unittest, wiredtiger, wttest from wiredtiger import stat from wtscenario import make_scenarios @@ -109,6 +109,7 @@ class test_compact02(wttest.WiredTigerTestCase): self.session = self.conn.open_session(None) # Create a table, add keys with both big and small values. + @unittest.skip("Temporarily disabled") def test_compact02(self): self.ConnectionOpen(self.cacheSize) diff --git a/src/third_party/wiredtiger/test/suite/test_cursor13.py b/src/third_party/wiredtiger/test/suite/test_cursor13.py index c007462ddee..c4bb50c5bad 100755 --- a/src/third_party/wiredtiger/test/suite/test_cursor13.py +++ b/src/third_party/wiredtiger/test/suite/test_cursor13.py @@ -183,8 +183,7 @@ class test_cursor13_reopens(test_cursor13_base): # create operation above or if this is the second or later # time through the loop. c = session.open_cursor(self.uri) - self.assert_cursor_reopened(caching_enabled and \ - (opens != 0 or create)) + self.assert_cursor_reopened(caching_enabled and (opens != 0 or create)) # With one cursor for this URI already open, we'll only # get a reopened cursor if this is the second or later @@ -546,7 +545,6 @@ class test_cursor13_dup(test_cursor13_base): c1.next() for notused in range(0, 100): - self.session.breakpoint() c2 = self.session.open_cursor(None, c1, None) c2.close() stats = self.caching_stats() diff --git a/src/third_party/wiredtiger/test/suite/test_durable_rollback_to_stable.py b/src/third_party/wiredtiger/test/suite/test_durable_rollback_to_stable.py index a4ab4f3569f..7e434a53597 100644 --- a/src/third_party/wiredtiger/test/suite/test_durable_rollback_to_stable.py +++ b/src/third_party/wiredtiger/test/suite/test_durable_rollback_to_stable.py @@ -27,7 +27,8 @@ # OTHER DEALINGS IN THE SOFTWARE. from helper import copy_wiredtiger_home -import wiredtiger, wttest +import wiredtiger, wttest, unittest +from suite_subprocess import suite_subprocess from wtdataset import SimpleDataSet from wtscenario import make_scenarios @@ -37,13 +38,14 @@ def timestamp_str(t): # test_durable_rollback_to_stable.py # Checking visibility and durability of updates with durable_timestamp and # with rollback to stable. -class test_durable_rollback_to_stable(wttest.WiredTigerTestCase): +class test_durable_rollback_to_stable(wttest.WiredTigerTestCase, suite_subprocess): session_config = 'isolation=snapshot' keyfmt = [ ('row-string', dict(keyfmt='S')), ('row-int', dict(keyfmt='i')), - ('column-store', dict(keyfmt='r')), + # The commented columnar tests needs to be enabled once rollback to stable for columnar is fixed in (WT-5548). + # ('column-store', dict(keyfmt='r')), ] types = [ ('file', dict(uri='file', ds=SimpleDataSet)), @@ -111,7 +113,7 @@ class test_durable_rollback_to_stable(wttest.WiredTigerTestCase): self.assertEquals(cursor.next(), 0) session.commit_transaction() - # Check that latest value is same as first update value. + # Check that latest value is same as first update value. self.assertEquals(cursor.reset(), 0) session.begin_transaction() self.assertEquals(cursor.next(), 0) @@ -165,5 +167,11 @@ class test_durable_rollback_to_stable(wttest.WiredTigerTestCase): self.assertEquals(cursor.get_value(), ds.value(111)) self.assertEquals(cursor.next(), 0) + # Use util to verify that second updates values have been flushed. + errfilename = "verifyrollbackerr.out" + self.runWt(["verify", "-s", uri], + errfilename=errfilename, failure=False) + self.check_empty_file(errfilename) + if __name__ == '__main__': wttest.run() diff --git a/src/third_party/wiredtiger/test/suite/test_durable_ts03.py b/src/third_party/wiredtiger/test/suite/test_durable_ts03.py index 43e03431709..8fdb1f615ae 100755 --- a/src/third_party/wiredtiger/test/suite/test_durable_ts03.py +++ b/src/third_party/wiredtiger/test/suite/test_durable_ts03.py @@ -27,7 +27,7 @@ # OTHER DEALINGS IN THE SOFTWARE. from helper import copy_wiredtiger_home -import wiredtiger, wttest +import unittest, wiredtiger, wttest def timestamp_str(t): return '%x' %t @@ -38,6 +38,7 @@ class test_durable_ts03(wttest.WiredTigerTestCase): conn_config = 'cache_size=10MB' session_config = 'isolation=snapshot' + @unittest.skip("Temporarily disabled") def test_durable_ts03(self): # Create a table. uri = 'table:test_durable_ts03' diff --git a/src/third_party/wiredtiger/test/suite/test_gc01.py b/src/third_party/wiredtiger/test/suite/test_gc01.py new file mode 100755 index 00000000000..8f84c61b245 --- /dev/null +++ b/src/third_party/wiredtiger/test/suite/test_gc01.py @@ -0,0 +1,194 @@ +#!/usr/bin/env python +# +# Public Domain 2014-2020 MongoDB, Inc. +# Public Domain 2008-2014 WiredTiger, Inc. +# +# This is free and unencumbered software released into the public domain. +# +# Anyone is free to copy, modify, publish, use, compile, sell, or +# distribute this software, either in source code form or as a compiled +# binary, for any purpose, commercial or non-commercial, and by any +# means. +# +# In jurisdictions that recognize copyright laws, the author or authors +# of this software dedicate any and all copyright interest in the +# software to the public domain. We make this dedication for the benefit +# of the public at large and to the detriment of our heirs and +# successors. We intend this dedication to be an overt act of +# relinquishment in perpetuity of all present and future rights to this +# software under copyright law. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +# IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. + +import time +from helper import copy_wiredtiger_home +import unittest, wiredtiger, wttest +from wtdataset import SimpleDataSet +from wiredtiger import stat + +def timestamp_str(t): + return '%x' % t + +# test_gc01.py + +# Shared base class used by gc tests. +class test_gc_base(wttest.WiredTigerTestCase): + + def large_updates(self, uri, value, ds, nrows, commit_ts): + # Update a large number of records. + session = self.session + cursor = session.open_cursor(uri) + for i in range(0, nrows): + session.begin_transaction() + cursor[ds.key(i)] = value + session.commit_transaction('commit_timestamp=' + timestamp_str(commit_ts)) + cursor.close() + + def large_modifies(self, uri, value, ds, location, nbytes, nrows, commit_ts): + # Load a slight modification. + session = self.session + cursor = session.open_cursor(uri) + session.begin_transaction() + for i in range(0, nrows): + cursor.set_key(i) + mods = [wiredtiger.Modify(value, location, nbytes)] + self.assertEqual(cursor.modify(mods), 0) + session.commit_transaction('commit_timestamp=' + timestamp_str(commit_ts)) + cursor.close() + + def check(self, check_value, uri, nrows, read_ts): + session = self.session + session.begin_transaction('read_timestamp=' + timestamp_str(read_ts)) + cursor = session.open_cursor(uri) + count = 0 + for k, v in cursor: + self.assertEqual(v, check_value) + count += 1 + session.rollback_transaction() + self.assertEqual(count, nrows) + + def check_gc_stats(self): + c = self.session.open_cursor( 'statistics:') + self.assertGreaterEqual(c[stat.conn.hs_gc_pages_visited][2], 0) + self.assertGreaterEqual(c[stat.conn.hs_gc_pages_removed][2], 0) + c.close() + +# Test that checkpoint cleans the obsolete lookaside pages. +class test_gc01(test_gc_base): + # Force a small cache. + conn_config = 'cache_size=50MB,log=(enabled),statistics=(all)' + session_config = 'isolation=snapshot' + + def test_gc(self): + nrows = 10000 + + # Create a table without logging. + uri = "table:gc01" + ds = SimpleDataSet( + self, uri, 0, key_format="i", value_format="S", config='log=(enabled=false)') + ds.populate() + + # Pin oldest and stable to timestamp 1. + self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(1) + + ',stable_timestamp=' + timestamp_str(1)) + + bigvalue = "aaaaa" * 100 + bigvalue2 = "ddddd" * 100 + self.large_updates(uri, bigvalue, ds, nrows, 10) + + # Check that all updates are seen. + self.check(bigvalue, uri, nrows, 10) + + self.large_updates(uri, bigvalue2, ds, nrows, 100) + + # Check that the new updates are only seen after the update timestamp. + self.check(bigvalue2, uri, nrows, 100) + + # Check that old updates are seen. + self.check(bigvalue, uri, nrows, 10) + + # Pin oldest and stable to timestamp 100. + self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(100) + + ',stable_timestamp=' + timestamp_str(100)) + + # Checkpoint to ensure that the history store is cleaned. + self.session.checkpoint() + self.check_gc_stats() + + # Check that the new updates are only seen after the update timestamp. + self.check(bigvalue2, uri, nrows, 100) + + # Load a slight modification with a later timestamp. + self.large_modifies(uri, 'A', ds, 10, 1, nrows, 110) + self.large_modifies(uri, 'B', ds, 20, 1, nrows, 120) + self.large_modifies(uri, 'C', ds, 30, 1, nrows, 130) + + # Second set of update operations with increased timestamp. + self.large_updates(uri, bigvalue, ds, nrows, 200) + + # Check that the new updates are only seen after the update timestamp. + self.check(bigvalue, uri, nrows, 200) + + # Check that the modifies are seen. + bigvalue_modA = bigvalue2[0:10] + 'A' + bigvalue2[11:] + bigvalue_modB = bigvalue_modA[0:20] + 'B' + bigvalue_modA[21:] + bigvalue_modC = bigvalue_modB[0:30] + 'C' + bigvalue_modB[31:] + self.check(bigvalue_modA, uri, nrows, 110) + self.check(bigvalue_modB, uri, nrows, 120) + self.check(bigvalue_modC, uri, nrows, 130) + + # Check that old updates are seen. + self.check(bigvalue2, uri, nrows, 100) + + # Pin oldest and stable to timestamp 200. + self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(200) + + ',stable_timestamp=' + timestamp_str(200)) + + # Checkpoint to ensure that the history store is cleaned. + self.session.checkpoint() + self.check_gc_stats() + + # Check that the new updates are only seen after the update timestamp. + self.check(bigvalue, uri, nrows, 200) + + # Load a slight modification with a later timestamp. + self.large_modifies(uri, 'A', ds, 10, 1, nrows, 210) + self.large_modifies(uri, 'B', ds, 20, 1, nrows, 220) + self.large_modifies(uri, 'C', ds, 30, 1, nrows, 230) + + # Third set of update operations with increased timestamp. + self.large_updates(uri, bigvalue2, ds, nrows, 300) + + # Check that the new updates are only seen after the update timestamp. + self.check(bigvalue2, uri, nrows, 300) + + # Check that the modifies are seen. + bigvalue_modA = bigvalue[0:10] + 'A' + bigvalue[11:] + bigvalue_modB = bigvalue_modA[0:20] + 'B' + bigvalue_modA[21:] + bigvalue_modC = bigvalue_modB[0:30] + 'C' + bigvalue_modB[31:] + self.check(bigvalue_modA, uri, nrows, 210) + self.check(bigvalue_modB, uri, nrows, 220) + self.check(bigvalue_modC, uri, nrows, 230) + + # Check that old updates are seen. + self.check(bigvalue, uri, nrows, 200) + + # Pin oldest and stable to timestamp 300. + self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(300) + + ',stable_timestamp=' + timestamp_str(300)) + + # Checkpoint to ensure that the history store is cleaned. + self.session.checkpoint() + self.check_gc_stats() + + # Check that the new updates are only seen after the update timestamp. + self.check(bigvalue2, uri, nrows, 300) + +if __name__ == '__main__': + wttest.run() diff --git a/src/third_party/wiredtiger/test/suite/test_gc02.py b/src/third_party/wiredtiger/test/suite/test_gc02.py new file mode 100755 index 00000000000..d7acbcb8ffa --- /dev/null +++ b/src/third_party/wiredtiger/test/suite/test_gc02.py @@ -0,0 +1,128 @@ +#!/usr/bin/env python +# +# Public Domain 2014-2020 MongoDB, Inc. +# Public Domain 2008-2014 WiredTiger, Inc. +# +# This is free and unencumbered software released into the public domain. +# +# Anyone is free to copy, modify, publish, use, compile, sell, or +# distribute this software, either in source code form or as a compiled +# binary, for any purpose, commercial or non-commercial, and by any +# means. +# +# In jurisdictions that recognize copyright laws, the author or authors +# of this software dedicate any and all copyright interest in the +# software to the public domain. We make this dedication for the benefit +# of the public at large and to the detriment of our heirs and +# successors. We intend this dedication to be an overt act of +# relinquishment in perpetuity of all present and future rights to this +# software under copyright law. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +# IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. + +from test_gc01 import test_gc_base +from wiredtiger import stat +from wtdataset import SimpleDataSet + +def timestamp_str(t): + return '%x' % t + +# test_gc02.py +# Test that checkpoint cleans the obsolete history store internal pages. +class test_gc02(test_gc_base): + conn_config = 'cache_size=1GB,log=(enabled),statistics=(all)' + session_config = 'isolation=snapshot' + + def test_gc(self): + nrows = 100000 + + # Create a table without logging. + uri = "table:gc02" + ds = SimpleDataSet( + self, uri, 0, key_format="i", value_format="S", config='log=(enabled=false)') + ds.populate() + + # Pin oldest and stable to timestamp 1. + self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(1) + + ',stable_timestamp=' + timestamp_str(1)) + + bigvalue = "aaaaa" * 100 + bigvalue2 = "ddddd" * 100 + self.large_updates(uri, bigvalue, ds, nrows, 10) + + # Check that all updates are seen. + self.check(bigvalue, uri, nrows, 10) + + self.large_updates(uri, bigvalue2, ds, nrows, 100) + + # Check that the new updates are only seen after the update timestamp. + self.check(bigvalue2, uri, nrows, 100) + + # Check that old updates are seen. + self.check(bigvalue, uri, nrows, 10) + + # Checkpoint to ensure that the history store is checkpointed and not cleaned. + self.session.checkpoint() + c = self.session.open_cursor('statistics:') + self.assertEqual(c[stat.conn.hs_gc_pages_evict][2], 0) + self.assertEqual(c[stat.conn.hs_gc_pages_removed][2], 0) + self.assertGreater(c[stat.conn.hs_gc_pages_visited][2], 0) + c.close() + + # Pin oldest and stable to timestamp 100. + self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(100) + + ',stable_timestamp=' + timestamp_str(100)) + + # Check that the new updates are only seen after the update timestamp. + self.check(bigvalue2, uri, nrows, 100) + + # Load a slight modification with a later timestamp. + self.large_modifies(uri, 'A', ds, 10, 1, nrows, 110) + self.large_modifies(uri, 'B', ds, 20, 1, nrows, 120) + self.large_modifies(uri, 'C', ds, 30, 1, nrows, 130) + + # Set of update operations with increased timestamp. + self.large_updates(uri, bigvalue, ds, nrows, 150) + + # Set of update operations with increased timestamp. + self.large_updates(uri, bigvalue2, ds, nrows, 180) + + # Set of update operations with increased timestamp. + self.large_updates(uri, bigvalue, ds, nrows, 200) + + # Check that the modifies are seen. + bigvalue_modA = bigvalue2[0:10] + 'A' + bigvalue2[11:] + bigvalue_modB = bigvalue_modA[0:20] + 'B' + bigvalue_modA[21:] + bigvalue_modC = bigvalue_modB[0:30] + 'C' + bigvalue_modB[31:] + self.check(bigvalue_modA, uri, nrows, 110) + self.check(bigvalue_modB, uri, nrows, 120) + self.check(bigvalue_modC, uri, nrows, 130) + + # Check that the new updates are only seen after the update timestamp. + self.check(bigvalue, uri, nrows, 150) + + # Check that the new updates are only seen after the update timestamp. + self.check(bigvalue2, uri, nrows, 180) + + # Check that the new updates are only seen after the update timestamp. + self.check(bigvalue, uri, nrows, 200) + + # Pin oldest and stable to timestamp 200. + self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(200) + + ',stable_timestamp=' + timestamp_str(200)) + + # Checkpoint to ensure that the history store is cleaned. + self.session.checkpoint() + self.check_gc_stats() + + # Check that the new updates are only seen after the update timestamp. + self.check(bigvalue, uri, nrows, 200) + +if __name__ == '__main__': + wttest.run() diff --git a/src/third_party/wiredtiger/test/suite/test_gc03.py b/src/third_party/wiredtiger/test/suite/test_gc03.py new file mode 100755 index 00000000000..a6db467fe96 --- /dev/null +++ b/src/third_party/wiredtiger/test/suite/test_gc03.py @@ -0,0 +1,146 @@ +#!/usr/bin/env python +# +# Public Domain 2014-2020 MongoDB, Inc. +# Public Domain 2008-2014 WiredTiger, Inc. +# +# This is free and unencumbered software released into the public domain. +# +# Anyone is free to copy, modify, publish, use, compile, sell, or +# distribute this software, either in source code form or as a compiled +# binary, for any purpose, commercial or non-commercial, and by any +# means. +# +# In jurisdictions that recognize copyright laws, the author or authors +# of this software dedicate any and all copyright interest in the +# software to the public domain. We make this dedication for the benefit +# of the public at large and to the detriment of our heirs and +# successors. We intend this dedication to be an overt act of +# relinquishment in perpetuity of all present and future rights to this +# software under copyright law. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +# IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. + +from test_gc01 import test_gc_base +from wiredtiger import stat +from wtdataset import SimpleDataSet + +def timestamp_str(t): + return '%x' % t + +# test_gc03.py +# Test that checkpoint cleans the obsolete history store pages that are in-memory. +class test_gc03(test_gc_base): + conn_config = 'cache_size=4GB,log=(enabled),statistics=(all),statistics_log=(wait=0,on_close=true)' + session_config = 'isolation=snapshot' + + def get_stat(self, stat): + stat_cursor = self.session.open_cursor('statistics:') + val = stat_cursor[stat][2] + stat_cursor.close() + return val + + def test_gc(self): + nrows = 10000 + + # Create a table without logging. + uri = "table:gc03" + ds = SimpleDataSet( + self, uri, 0, key_format="i", value_format="S", config='log=(enabled=false)') + ds.populate() + + # Create an extra table without logging. + uri_extra = "table:gc03_extra" + ds_extra = SimpleDataSet( + self, uri_extra, 0, key_format="i", value_format="S", config='log=(enabled=false)') + ds_extra.populate() + + # Pin oldest and stable to timestamp 1. + self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(1) + + ',stable_timestamp=' + timestamp_str(1)) + + bigvalue = "aaaaa" * 100 + bigvalue2 = "ddddd" * 100 + self.large_updates(uri, bigvalue, ds, nrows, 10) + + # Check that all updates are seen. + self.check(bigvalue, uri, nrows, 10) + + self.large_updates(uri, bigvalue2, ds, nrows, 100) + + # Check that the new updates are only seen after the update timestamp. + self.check(bigvalue2, uri, nrows, 100) + + # Check that old updates are seen. + self.check(bigvalue, uri, nrows, 10) + + # Checkpoint to ensure that the history store is populated. + self.session.checkpoint() + self.assertGreater(self.get_stat(stat.conn.hs_gc_pages_visited), 0) + + # Pin oldest and stable to timestamp 100. + self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(100) + + ',stable_timestamp=' + timestamp_str(100)) + + # Check that the new updates are only seen after the update timestamp. + self.check(bigvalue2, uri, nrows, 100) + + # Load a slight modification with a later timestamp. + self.large_modifies(uri, 'A', ds, 10, 1, nrows, 110) + self.large_modifies(uri, 'B', ds, 20, 1, nrows, 120) + self.large_modifies(uri, 'C', ds, 30, 1, nrows, 130) + + # Second set of update operations with increased timestamp. + self.large_updates(uri, bigvalue, ds, nrows, 200) + + # Check that the new updates are only seen after the update timestamp. + self.check(bigvalue, uri, nrows, 200) + + # Check that the modifies are seen. + bigvalue_modA = bigvalue2[0:10] + 'A' + bigvalue2[11:] + bigvalue_modB = bigvalue_modA[0:20] + 'B' + bigvalue_modA[21:] + bigvalue_modC = bigvalue_modB[0:30] + 'C' + bigvalue_modB[31:] + self.check(bigvalue_modA, uri, nrows, 110) + self.check(bigvalue_modB, uri, nrows, 120) + self.check(bigvalue_modC, uri, nrows, 130) + + # Check that old updates are seen. + self.check(bigvalue2, uri, nrows, 100) + + # Pin oldest and stable to timestamp 200. + self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(200) + + ',stable_timestamp=' + timestamp_str(200)) + + # Update on extra table. + self.large_updates(uri_extra, bigvalue, ds_extra, 100, 210) + self.large_updates(uri_extra, bigvalue2, ds_extra, 100, 220) + + # Checkpoint to ensure that the history store is populated and added for eviction. + self.session.checkpoint() + self.assertGreater(self.get_stat(stat.conn.hs_gc_pages_evict), 0) + self.assertGreater(self.get_stat(stat.conn.hs_gc_pages_visited), 0) + + # Check that the new updates are only seen after the update timestamp. + self.check(bigvalue, uri, nrows, 200) + + # Pin oldest and stable to timestamp 300. + self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(300) + + ',stable_timestamp=' + timestamp_str(300)) + + self.large_updates(uri_extra, bigvalue, ds_extra, 100, 310) + self.large_updates(uri_extra, bigvalue2, ds_extra, 100, 320) + + # Checkpoint to ensure that the normal table history store gets cleaned. + self.session.checkpoint() + self.check_gc_stats() + + # Check that the new updates are only seen after the update timestamp. + self.check(bigvalue, uri, nrows, 300) + +if __name__ == '__main__': + wttest.run() diff --git a/src/third_party/wiredtiger/test/suite/test_gc04.py b/src/third_party/wiredtiger/test/suite/test_gc04.py new file mode 100755 index 00000000000..298ee932ed9 --- /dev/null +++ b/src/third_party/wiredtiger/test/suite/test_gc04.py @@ -0,0 +1,106 @@ +#!/usr/bin/env python +# +# Public Domain 2014-2020 MongoDB, Inc. +# Public Domain 2008-2014 WiredTiger, Inc. +# +# This is free and unencumbered software released into the public domain. +# +# Anyone is free to copy, modify, publish, use, compile, sell, or +# distribute this software, either in source code form or as a compiled +# binary, for any purpose, commercial or non-commercial, and by any +# means. +# +# In jurisdictions that recognize copyright laws, the author or authors +# of this software dedicate any and all copyright interest in the +# software to the public domain. We make this dedication for the benefit +# of the public at large and to the detriment of our heirs and +# successors. We intend this dedication to be an overt act of +# relinquishment in perpetuity of all present and future rights to this +# software under copyright law. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +# IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. + +from test_gc01 import test_gc_base +from wiredtiger import stat +from wtdataset import SimpleDataSet + +def timestamp_str(t): + return '%x' % t + +# test_gc04.py +# Test that checkpoint must not clean the pages that are not obsolete. +class test_gc04(test_gc_base): + conn_config = 'cache_size=50MB,log=(enabled),statistics=(all)' + session_config = 'isolation=snapshot' + + def get_stat(self, stat): + stat_cursor = self.session.open_cursor('statistics:') + val = stat_cursor[stat][2] + stat_cursor.close() + return val + + def test_gc(self): + nrows = 10000 + + # Create a table without logging. + uri = "table:gc04" + ds = SimpleDataSet( + self, uri, 0, key_format="i", value_format="S", config='log=(enabled=false)') + ds.populate() + + # Pin oldest and stable to timestamp 1. + self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(1) + + ',stable_timestamp=' + timestamp_str(1)) + + bigvalue = "aaaaa" * 100 + bigvalue2 = "ddddd" * 100 + self.large_updates(uri, bigvalue, ds, nrows, 10) + self.large_updates(uri, bigvalue2, ds, nrows, 20) + + # Checkpoint to ensure that the history store is populated. + self.session.checkpoint() + self.assertEqual(self.get_stat(stat.conn.hs_gc_pages_evict), 0) + self.assertEqual(self.get_stat(stat.conn.hs_gc_pages_removed), 0) + self.assertGreater(self.get_stat(stat.conn.hs_gc_pages_visited), 0) + + self.large_updates(uri, bigvalue, ds, nrows, 30) + + # Checkpoint to ensure that the history store is populated. + self.session.checkpoint() + self.assertEqual(self.get_stat(stat.conn.hs_gc_pages_evict), 0) + self.assertEqual(self.get_stat(stat.conn.hs_gc_pages_removed), 0) + self.assertGreater(self.get_stat(stat.conn.hs_gc_pages_visited), 0) + + self.large_updates(uri, bigvalue2, ds, nrows, 40) + + # Checkpoint to ensure that the history store is populated. + self.session.checkpoint() + self.assertEqual(self.get_stat(stat.conn.hs_gc_pages_evict), 0) + self.assertEqual(self.get_stat(stat.conn.hs_gc_pages_removed), 0) + self.assertGreater(self.get_stat(stat.conn.hs_gc_pages_visited), 0) + + self.large_updates(uri, bigvalue, ds, nrows, 50) + self.large_updates(uri, bigvalue2, ds, nrows, 60) + + # Checkpoint to ensure that the history store is populated. + self.session.checkpoint() + self.assertEqual(self.get_stat(stat.conn.hs_gc_pages_evict), 0) + self.assertEqual(self.get_stat(stat.conn.hs_gc_pages_removed), 0) + self.assertGreater(self.get_stat(stat.conn.hs_gc_pages_visited), 0) + + self.large_updates(uri, bigvalue, ds, nrows, 70) + + # Checkpoint to ensure that the history store is populated. + self.session.checkpoint() + self.assertEqual(self.get_stat(stat.conn.hs_gc_pages_evict), 0) + self.assertEqual(self.get_stat(stat.conn.hs_gc_pages_removed), 0) + self.assertGreater(self.get_stat(stat.conn.hs_gc_pages_visited), 0) + +if __name__ == '__main__': + wttest.run() diff --git a/src/third_party/wiredtiger/test/suite/test_gc05.py b/src/third_party/wiredtiger/test/suite/test_gc05.py new file mode 100755 index 00000000000..c4268696ea4 --- /dev/null +++ b/src/third_party/wiredtiger/test/suite/test_gc05.py @@ -0,0 +1,99 @@ +#!/usr/bin/env python +# +# Public Domain 2014-2020 MongoDB, Inc. +# Public Domain 2008-2014 WiredTiger, Inc. +# +# This is free and unencumbered software released into the public domain. +# +# Anyone is free to copy, modify, publish, use, compile, sell, or +# distribute this software, either in source code form or as a compiled +# binary, for any purpose, commercial or non-commercial, and by any +# means. +# +# In jurisdictions that recognize copyright laws, the author or authors +# of this software dedicate any and all copyright interest in the +# software to the public domain. We make this dedication for the benefit +# of the public at large and to the detriment of our heirs and +# successors. We intend this dedication to be an overt act of +# relinquishment in perpetuity of all present and future rights to this +# software under copyright law. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +# IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. + +from test_gc01 import test_gc_base +from wtdataset import SimpleDataSet + +def timestamp_str(t): + return '%x' % t + +# test_gc05.py +# Verify a locked checkpoint is not removed during garbage collection. +class test_gc05(test_gc_base): + conn_config = 'cache_size=50MB,log=(enabled),statistics=(all)' + session_config = 'isolation=snapshot' + + def test_gc(self): + uri = "table:gc05" + create_params = 'value_format=S,key_format=i' + self.session.create(uri, create_params) + + nrows = 10000 + value_x = "xxxxx" * 100 + value_y = "yyyyy" * 100 + value_z = "zzzzz" * 100 + ds = SimpleDataSet( + self, uri, 0, key_format="i", value_format="S", config='log=(enabled=false)') + ds.populate() + + # Set the oldest and stable timestamps to 10. + self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(10) + + ',stable_timestamp=' + timestamp_str(10)) + + # Insert values with varying timestamps. + self.large_updates(uri, value_x, ds, nrows, 20) + self.large_updates(uri, value_y, ds, nrows, 30) + self.large_updates(uri, value_z, ds, nrows, 40) + + # Perform a checkpoint. + self.session.checkpoint("name=checkpoint_one") + + # Check statistics. + self.check_gc_stats() + + # Open a cursor to the checkpoint just performed. + ckpt_cursor = self.session.open_cursor(uri, None, "checkpoint=checkpoint_one") + + # Move the oldest and stable timestamps to 40. + self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(40) + + ',stable_timestamp=' + timestamp_str(40)) + + # Insert values with varying timestamps. + self.large_updates(uri, value_z, ds, nrows, 50) + self.large_updates(uri, value_y, ds, nrows, 60) + self.large_updates(uri, value_x, ds, nrows, 70) + + # Move the oldest and stable timestamps to 70. + self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(70) + + ',stable_timestamp=' + timestamp_str(70)) + + # Perform a checkpoint. + self.session.checkpoint() + self.check_gc_stats() + + # Verify checkpoint_one still exists and contains the expected values. + for i in range(0, nrows): + ckpt_cursor.set_key(i) + ckpt_cursor.search() + self.assertEqual(value_z, ckpt_cursor.get_value()) + + # Close checkpoint cursor. + ckpt_cursor.close() + +if __name__ == '__main__': + wttest.run() diff --git a/src/third_party/wiredtiger/test/suite/test_las01.py b/src/third_party/wiredtiger/test/suite/test_hs01.py index 073e30ec8ad..3559acb0645 100755..100644 --- a/src/third_party/wiredtiger/test/suite/test_las01.py +++ b/src/third_party/wiredtiger/test/suite/test_hs01.py @@ -27,21 +27,21 @@ # OTHER DEALINGS IN THE SOFTWARE. from helper import copy_wiredtiger_home -import wiredtiger, wttest +import unittest, wiredtiger, wttest from wtdataset import SimpleDataSet def timestamp_str(t): return '%x' % t -# test_las01.py -# Smoke tests to ensure lookaside tables are working. -class test_las01(wttest.WiredTigerTestCase): +# test_hs01.py +# Smoke tests to ensure history store tables are working. +class test_hs01(wttest.WiredTigerTestCase): # Force a small cache. conn_config = 'cache_size=50MB' session_config = 'isolation=snapshot' def large_updates(self, session, uri, value, ds, nrows, timestamp=False): - # Update a large number of records, we'll hang if the lookaside table + # Update a large number of records, we'll hang if the history store table # isn't doing its thing. cursor = session.open_cursor(uri) for i in range(1, 10000): @@ -55,7 +55,7 @@ class test_las01(wttest.WiredTigerTestCase): cursor.close() def large_modifies(self, session, uri, offset, ds, nrows, timestamp=False): - # Modify a large number of records, we'll hang if the lookaside table + # Modify a large number of records, we'll hang if the history store table # isn't doing its thing. cursor = session.open_cursor(uri) for i in range(1, 10000): @@ -92,9 +92,9 @@ class test_las01(wttest.WiredTigerTestCase): session.close() conn.close() - def test_las(self): + def test_hs(self): # Create a small table. - uri = "table:test_las01" + uri = "table:test_hs01" nrows = 100 ds = SimpleDataSet(self, uri, nrows, key_format="S", value_format='u') ds.populate() @@ -110,17 +110,7 @@ class test_las01(wttest.WiredTigerTestCase): self.session.checkpoint() # Scenario: 1 - # Check to see if LAS is working with the old snapshot. - bigvalue1 = b"bbbbb" * 100 - self.session.snapshot("name=xxx") - # Update the values in different session after snapshot. - self.large_updates(self.session, uri, bigvalue1, ds, nrows) - # Check to see the value after recovery. - self.durable_check(bigvalue1, uri, ds, nrows) - self.session.snapshot("drop=(all)") - - # Scenario: 2 - # Check to see if LAS is working with the old reader. + # Check to see if the history store is working with the old reader. bigvalue2 = b"ccccc" * 100 session2 = self.conn.open_session() session2.begin_transaction('isolation=snapshot') @@ -130,8 +120,8 @@ class test_las01(wttest.WiredTigerTestCase): session2.rollback_transaction() session2.close() - # Scenario: 3 - # Check to see LAS working with modify operations. + # Scenario: 2 + # Check to see the history store working with modify operations. bigvalue3 = b"ccccc" * 100 bigvalue3 = b'AA' + bigvalue3[2:] session2 = self.conn.open_session() @@ -146,8 +136,8 @@ class test_las01(wttest.WiredTigerTestCase): session2.rollback_transaction() session2.close() - # Scenario: 4 - # Check to see if LAS is working with the old timestamp. + # Scenario: 3 + # Check to see if the history store is working with the old timestamp. bigvalue4 = b"ddddd" * 100 self.conn.set_timestamp('stable_timestamp=' + timestamp_str(1)) self.large_updates(self.session, uri, bigvalue4, ds, nrows, timestamp=True) diff --git a/src/third_party/wiredtiger/test/suite/test_las02.py b/src/third_party/wiredtiger/test/suite/test_hs02.py index 3df841f68ba..9019bc1c7e4 100644 --- a/src/third_party/wiredtiger/test/suite/test_las02.py +++ b/src/third_party/wiredtiger/test/suite/test_hs02.py @@ -33,15 +33,15 @@ from wtdataset import SimpleDataSet def timestamp_str(t): return '%x' % t -# test_las02.py -# Test that truncate with lookaside entries and timestamps gives expected results. -class test_las02(wttest.WiredTigerTestCase): +# test_hs02.py +# Test that truncate with history store entries and timestamps gives expected results. +class test_hs02(wttest.WiredTigerTestCase): # Force a small cache. conn_config = 'cache_size=50MB,log=(enabled)' session_config = 'isolation=snapshot' def large_updates(self, uri, value, ds, nrows, commit_ts): - # Update a large number of records, we'll hang if the lookaside table isn't working. + # Update a large number of records, we'll hang if the history store table isn't working. session = self.session cursor = session.open_cursor(uri) for i in range(1, nrows + 1): @@ -61,10 +61,11 @@ class test_las02(wttest.WiredTigerTestCase): session.rollback_transaction() self.assertEqual(count, nrows) - def test_las(self): + def test_hs(self): nrows = 10000 - # Create a table without logging to ensure we get "skew_newest" lookaside eviction behavior. + # Create a table without logging to ensure we get "skew_newest" history store eviction + # behavior. uri = "table:las02_main" ds = SimpleDataSet( self, uri, 0, key_format="S", value_format="S", config='log=(enabled=false)') @@ -84,7 +85,7 @@ class test_las02(wttest.WiredTigerTestCase): # Check that all updates are seen self.check(bigvalue, uri, nrows // 3, 1) - # Check to see lookaside working with old timestamp + # Check to see the history store working with old timestamp bigvalue2 = "ddddd" * 100 self.large_updates(uri, bigvalue2, ds, nrows, 100) diff --git a/src/third_party/wiredtiger/test/suite/test_las03.py b/src/third_party/wiredtiger/test/suite/test_hs03.py index 3d562eba05f..2172284ea02 100755..100644 --- a/src/third_party/wiredtiger/test/suite/test_las03.py +++ b/src/third_party/wiredtiger/test/suite/test_hs03.py @@ -27,16 +27,16 @@ # OTHER DEALINGS IN THE SOFTWARE. from helper import copy_wiredtiger_home -import wiredtiger, wttest +import unittest, wiredtiger, wttest from wiredtiger import stat from wtdataset import SimpleDataSet def timestamp_str(t): return '%x' % t -# test_las03.py -# Ensure checkpoints don't read too unnecessary lookaside entries. -class test_las03(wttest.WiredTigerTestCase): +# test_hs03.py +# Ensure checkpoints don't read too unnecessary history store entries. +class test_hs03(wttest.WiredTigerTestCase): # Force a small cache. conn_config = 'cache_size=50MB,statistics=(fast)' session_config = 'isolation=snapshot' @@ -48,7 +48,7 @@ class test_las03(wttest.WiredTigerTestCase): return val def large_updates(self, session, uri, value, ds, nrows, nops): - # Update a large number of records, we'll hang if the lookaside table + # Update a large number of records, we'll hang if the history store table # isn't doing its thing. cursor = session.open_cursor(uri) for i in range(nrows + 1, nrows + nops + 1): @@ -57,9 +57,9 @@ class test_las03(wttest.WiredTigerTestCase): session.commit_transaction('commit_timestamp=' + timestamp_str(i)) cursor.close() - def test_checkpoint_las_reads(self): + def test_checkpoint_hs_reads(self): # Create a small table. - uri = "table:test_las03" + uri = "table:test_hs03" nrows = 100 ds = SimpleDataSet(self, uri, nrows, key_format="S", value_format='u') ds.populate() @@ -72,16 +72,16 @@ class test_las03(wttest.WiredTigerTestCase): cursor.close() self.session.checkpoint() - # Check to see LAS working with old timestamp. + # Check to see the history store working with old timestamp. bigvalue2 = b"ddddd" * 100 self.conn.set_timestamp('stable_timestamp=' + timestamp_str(1)) - las_writes_start = self.get_stat(stat.conn.cache_write_lookaside) + hs_writes_start = self.get_stat(stat.conn.cache_write_hs) self.large_updates(self.session, uri, bigvalue2, ds, nrows, 10000) # If the test sizing is correct, the history will overflow the cache. self.session.checkpoint() - las_writes = self.get_stat(stat.conn.cache_write_lookaside) - las_writes_start - self.assertGreaterEqual(las_writes, 0) + hs_writes = self.get_stat(stat.conn.cache_write_hs) - hs_writes_start + self.assertGreaterEqual(hs_writes, 0) for ts in range(2, 4): self.conn.set_timestamp('stable_timestamp=' + timestamp_str(ts)) @@ -89,14 +89,14 @@ class test_las03(wttest.WiredTigerTestCase): # Now just update one record and checkpoint again. self.large_updates(self.session, uri, bigvalue2, ds, nrows, 1) - las_reads_start = self.get_stat(stat.conn.cache_read_lookaside) + hs_reads_start = self.get_stat(stat.conn.cache_hs_read) self.session.checkpoint() - las_reads = self.get_stat(stat.conn.cache_read_lookaside) - las_reads_start + hs_reads = self.get_stat(stat.conn.cache_hs_read) - hs_reads_start # Since we're dealing with eviction concurrent with checkpoints # and skewing is controlled by a heuristic, we can't put too tight # a bound on this. - self.assertLessEqual(las_reads, 200) + self.assertLessEqual(hs_reads, 200) if __name__ == '__main__': wttest.run() diff --git a/src/third_party/wiredtiger/test/suite/test_las04.py b/src/third_party/wiredtiger/test/suite/test_hs04.py index a9c27876958..84fc791c696 100644 --- a/src/third_party/wiredtiger/test/suite/test_las04.py +++ b/src/third_party/wiredtiger/test/suite/test_hs04.py @@ -26,8 +26,8 @@ # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. # -# test_las04.py -# Test file_max configuration and reconfiguration for the lookaside table. +# test_hs04.py +# Test file_max configuration and reconfiguration for the history store table. # import wiredtiger, wttest @@ -36,8 +36,8 @@ from wtscenario import make_scenarios # Taken from src/include/misc.h. WT_MB = 1048576 -class test_las04(wttest.WiredTigerTestCase): - uri = 'table:las_04' +class test_hs04(wttest.WiredTigerTestCase): + uri = 'table:hs_04' in_memory_values = [ ('false', dict(in_memory=False)), ('none', dict(in_memory=None)), @@ -60,7 +60,7 @@ class test_las04(wttest.WiredTigerTestCase): def conn_config(self): config = 'statistics=(fast)' if self.init_file_max is not None: - config += ',cache_overflow=(file_max={})'.format(self.init_file_max) + config += ',history_store=(file_max={})'.format(self.init_file_max) if self.in_memory is not None: config += ',in_memory=' + ('true' if self.in_memory else 'false') return config @@ -71,22 +71,22 @@ class test_las04(wttest.WiredTigerTestCase): stat_cursor.close() return val - def test_las(self): + def test_hs(self): self.session.create(self.uri, 'key_format=S,value_format=S') if self.in_memory: - # For in-memory configurations, we simply ignore any lookaside + # For in-memory configurations, we simply ignore any history store # related configuration. self.assertEqual( - self.get_stat(wiredtiger.stat.conn.cache_lookaside_ondisk_max), + self.get_stat(wiredtiger.stat.conn.cache_hs_ondisk_max), 0) else: self.assertEqual( - self.get_stat(wiredtiger.stat.conn.cache_lookaside_ondisk_max), + self.get_stat(wiredtiger.stat.conn.cache_hs_ondisk_max), self.init_stat_val) reconfigure = lambda: self.conn.reconfigure( - 'cache_overflow=(file_max={})'.format(self.reconfig_file_max)) + 'history_store=(file_max={})'.format(self.reconfig_file_max)) # We expect an error when the statistic value is None because the value # is out of range. @@ -99,11 +99,11 @@ class test_las04(wttest.WiredTigerTestCase): if self.in_memory: self.assertEqual( - self.get_stat(wiredtiger.stat.conn.cache_lookaside_ondisk_max), + self.get_stat(wiredtiger.stat.conn.cache_hs_ondisk_max), 0) else: self.assertEqual( - self.get_stat(wiredtiger.stat.conn.cache_lookaside_ondisk_max), + self.get_stat(wiredtiger.stat.conn.cache_hs_ondisk_max), self.reconfig_stat_val) if __name__ == '__main__': diff --git a/src/third_party/wiredtiger/test/suite/test_las05.py b/src/third_party/wiredtiger/test/suite/test_hs05.py index 0914c6e8a52..17c87109efd 100755..100644 --- a/src/third_party/wiredtiger/test/suite/test_las05.py +++ b/src/third_party/wiredtiger/test/suite/test_hs05.py @@ -34,10 +34,10 @@ from wtdataset import SimpleDataSet def timestamp_str(t): return '%x' % t -# test_las05.py -# Verify lookaside_score reflects cache pressure due to history -# even if we're not yet actively pushing into the lookaside file. -class test_las05(wttest.WiredTigerTestCase): +# test_hs05.py +# Verify hs_score reflects cache pressure due to history +# even if we're not yet actively pushing into the history store file. +class test_hs05(wttest.WiredTigerTestCase): # Force a small cache. conn_config = 'cache_size=50MB,statistics=(fast)' session_config = 'isolation=snapshot' @@ -50,24 +50,24 @@ class test_las05(wttest.WiredTigerTestCase): return val def large_updates(self, session, uri, value, ds, nrows, nops): - # Update a large number of records, we'll hang if the lookaside table + # Update a large number of records, we'll hang if the history store table # isn't doing its thing. cursor = session.open_cursor(uri) - score_start = self.get_stat(stat.conn.cache_lookaside_score) + score_start = self.get_stat(stat.conn.cache_hs_score) for i in range(nrows + 1, nrows + nops + 1): session.begin_transaction() cursor[ds.key(i)] = value session.commit_transaction('commit_timestamp=' + timestamp_str(self.stable + i)) cursor.close() - score_end = self.get_stat(stat.conn.cache_lookaside_score) + score_end = self.get_stat(stat.conn.cache_hs_score) score_diff = score_end - score_start self.pr("After large updates score start: " + str(score_start)) self.pr("After large updates score end: " + str(score_end)) - self.pr("After large updates lookaside score diff: " + str(score_diff)) + self.pr("After large updates hs score diff: " + str(score_diff)) - def test_checkpoint_las_reads(self): + def test_checkpoint_hs_reads(self): # Create a small table. - uri = "table:test_las05" + uri = "table:test_hs05" nrows = 100 ds = SimpleDataSet(self, uri, nrows, key_format="S", value_format='u') ds.populate() @@ -86,31 +86,31 @@ class test_las05(wttest.WiredTigerTestCase): # Pin the oldest timestamp so that all history has to stay. self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(1)) # Loop a couple times, partly filling the cache but not - # overfilling it to see the lookaside score value change - # even if lookaside is not yet in use. + # overfilling it to see the history store score value change + # even if the history store is not yet in use. # # Use smaller values, 50 bytes and fill 8 times, under full cache. valstr='abcdefghijklmnopqrstuvwxyz' - loop_start = self.get_stat(stat.conn.cache_lookaside_score) + loop_start = self.get_stat(stat.conn.cache_hs_score) for i in range(1, 9): bigvalue2 = valstr[i].encode() * 50 self.conn.set_timestamp('stable_timestamp=' + timestamp_str(self.stable)) - entries_start = self.get_stat(stat.conn.cache_lookaside_entries) - score_start = self.get_stat(stat.conn.cache_lookaside_score) + entries_start = self.get_stat(stat.conn.cache_hs_insert) + score_start = self.get_stat(stat.conn.cache_hs_score) self.pr("Update iteration: " + str(i) + " Value: " + str(bigvalue2)) self.pr("Update iteration: " + str(i) + " Score: " + str(score_start)) self.large_updates(self.session, uri, bigvalue2, ds, nrows, nrows) self.stable += nrows - score_end = self.get_stat(stat.conn.cache_lookaside_score) - entries_end = self.get_stat(stat.conn.cache_lookaside_entries) - # We expect to see the lookaside score increase but not writing - # any new entries to lookaside. + score_end = self.get_stat(stat.conn.cache_hs_score) + entries_end = self.get_stat(stat.conn.cache_hs_insert) + # We expect to see the history store score increase but not writing + # any new entries to the history store. self.assertGreaterEqual(score_end, score_start) self.assertEqual(entries_end, entries_start) # While each iteration may or may not increase the score, we expect the # score to have strictly increased from before the loop started. - loop_end = self.get_stat(stat.conn.cache_lookaside_score) + loop_end = self.get_stat(stat.conn.cache_hs_score) self.assertGreater(loop_end, loop_start) # Now move oldest timestamp forward and insert a couple large updates @@ -127,7 +127,7 @@ class test_las05(wttest.WiredTigerTestCase): self.conn.set_timestamp('stable_timestamp=' + timestamp_str(self.stable)) self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(self.stable)) self.stable += nrows - score_end = self.get_stat(stat.conn.cache_lookaside_score) + score_end = self.get_stat(stat.conn.cache_hs_score) self.assertLess(score_end, score_start) self.assertEqual(score_end, 0) diff --git a/src/third_party/wiredtiger/test/suite/test_hs06.py b/src/third_party/wiredtiger/test/suite/test_hs06.py new file mode 100644 index 00000000000..edef6368e9d --- /dev/null +++ b/src/third_party/wiredtiger/test/suite/test_hs06.py @@ -0,0 +1,530 @@ +#!/usr/bin/env python +# +# Public Domain 2014-2020 MongoDB, Inc. +# Public Domain 2008-2014 WiredTiger, Inc. +# +# This is free and unencumbered software released into the public domain. +# +# Anyone is free to copy, modify, publish, use, compile, sell, or +# distribute this software, either in source code form or as a compiled +# binary, for any purpose, commercial or non-commercial, and by any +# means. +# +# In jurisdictions that recognize copyright laws, the author or authors +# of this software dedicate any and all copyright interest in the +# software to the public domain. We make this dedication for the benefit +# of the public at large and to the detriment of our heirs and +# successors. We intend this dedication to be an overt act of +# relinquishment in perpetuity of all present and future rights to this +# software under copyright law. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +# IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. + +import wiredtiger, wttest +from wiredtiger import stat +from wtscenario import make_scenarios + +def timestamp_str(t): + return '%x' % t + +# test_hs06.py +# Verify that triggering history store usage does not cause a spike in memory usage +# to form an update chain from the history store contents. +# +# The required value should be fetched from the history store and then passed straight +# back to the user without putting together an update chain. +# +# TODO: Uncomment the checks after the main portion of the relevant history +# project work is complete. +class test_hs06(wttest.WiredTigerTestCase): + # Force a small cache. + conn_config = 'cache_size=50MB,statistics=(fast)' + session_config = 'isolation=snapshot' + key_format_values = [ + ('column', dict(key_format='r')), + ('integer', dict(key_format='i')), + ('string', dict(key_format='S')) + ] + scenarios = make_scenarios(key_format_values) + + def get_stat(self, stat): + stat_cursor = self.session.open_cursor('statistics:') + val = stat_cursor[stat][2] + stat_cursor.close() + return val + + def get_non_page_image_memory_usage(self): + return self.get_stat(stat.conn.cache_bytes_other) + + def create_key(self, i): + if self.key_format == 'S': + return str(i) + return i + + def test_hs_reads(self): + # Create a small table. + uri = "table:test_hs06" + create_params = 'key_format={},value_format=S'.format(self.key_format) + self.session.create(uri, create_params) + + value1 = 'a' * 500 + value2 = 'b' * 500 + + # Load 1Mb of data. + self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(1)) + cursor = self.session.open_cursor(uri) + self.session.begin_transaction() + for i in range(1, 2000): + cursor[self.create_key(i)] = value1 + self.session.commit_transaction('commit_timestamp=' + timestamp_str(2)) + + # Load another 1Mb of data with a later timestamp. + self.session.begin_transaction() + for i in range(1, 2000): + cursor[self.create_key(i)] = value2 + self.session.commit_transaction('commit_timestamp=' + timestamp_str(3)) + + # Write a version of the data to disk. + self.conn.set_timestamp('stable_timestamp=' + timestamp_str(2)) + self.session.checkpoint() + + # Check the checkpoint wrote the expected values. Todo: Fix checkpoint cursors WT-5492. + # cursor2 = self.session.open_cursor(uri, None, 'checkpoint=WiredTigerCheckpoint') + cursor2 = self.session.open_cursor(uri) + self.session.begin_transaction('read_timestamp=' + timestamp_str(2)) + for key, value in cursor2: + self.assertEqual(value, value1) + self.session.commit_transaction() + cursor2.close() + + start_usage = self.get_non_page_image_memory_usage() + + # Whenever we request something out of cache of timestamp 2, we should + # be reading it straight from the history store without initialising a full + # update chain of every version of the data. + self.session.begin_transaction('read_timestamp=' + timestamp_str(2)) + for i in range(1, 2000): + self.assertEqual(cursor[self.create_key(i)], value1) + self.session.rollback_transaction() + + end_usage = self.get_non_page_image_memory_usage() + + # Non-page related memory usage shouldn't spike significantly. + # + # Prior to this change, this type of workload would use a lot of memory + # to recreate update lists for each page. + # + # This check could be more aggressive but to avoid potential flakiness, + # lets just ensure that it hasn't doubled. + # + # TODO: Uncomment this once the project work is done. + # self.assertLessEqual(end_usage, (start_usage * 2)) + + # WT-5336 causing the read at timestamp 4 returning the value committed at timestamp 5 or 3 + def test_hs_modify_reads(self): + # Create a small table. + uri = "table:test_hs06" + create_params = 'key_format={},value_format=S'.format(self.key_format) + self.session.create(uri, create_params) + + # Create initial large values. + value1 = 'a' * 500 + value2 = 'd' * 500 + + # Load 1Mb of data. + self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(1)) + cursor = self.session.open_cursor(uri) + self.session.begin_transaction() + for i in range(1, 2000): + cursor[self.create_key(i)] = value1 + self.session.commit_transaction('commit_timestamp=' + timestamp_str(2)) + + # Load a slight modification with a later timestamp. + self.session.begin_transaction() + for i in range(1, 2000): + cursor.set_key(self.create_key(i)) + mods = [wiredtiger.Modify('B', 100, 1)] + self.assertEqual(cursor.modify(mods), 0) + self.session.commit_transaction('commit_timestamp=' + timestamp_str(3)) + + # And another. + self.session.begin_transaction() + for i in range(1, 2000): + cursor.set_key(self.create_key(i)) + mods = [wiredtiger.Modify('C', 200, 1)] + self.assertEqual(cursor.modify(mods), 0) + self.session.commit_transaction('commit_timestamp=' + timestamp_str(4)) + + # Now write something completely different. + self.session.begin_transaction() + for i in range(1, 2000): + cursor[self.create_key(i)] = value2 + self.session.commit_transaction('commit_timestamp=' + timestamp_str(5)) + + # Now the latest version will get written to the data file. + self.session.checkpoint() + + expected = list(value1) + expected[100] = 'B' + expected = str().join(expected) + + # Whenever we request something of timestamp 3, this should be a modify + # op. We should looking forwards in the history store until we find the + # newest whole update (timestamp 4). + # + # t5: value1 (full update on page) + # t4: full update in las + # t3: (reverse delta in las) <= We're querying for t4 so we begin here. + # t2: value2 (full update in las) + self.session.begin_transaction('read_timestamp=' + timestamp_str(3)) + for i in range(1, 2000): + self.assertEqual(cursor[self.create_key(i)], expected) + self.session.rollback_transaction() + + expected = list(expected) + expected[200] = 'C' + expected = str().join(expected) + + # Whenever we request something of timestamp 4, this should be a full + # update. We should get it from las directly. + # + # t5: value1 (full update) + # t4: full update in las <= We're querying for t4 and we return. + # t3: (reverse delta in las) + # t2: value2 (full update in las) + self.session.begin_transaction('read_timestamp=' + timestamp_str(4)) + for i in range(1, 2000): + self.assertEqual(cursor[self.create_key(i)], expected) + self.session.rollback_transaction() + + def test_hs_prepare_reads(self): + # Create a small table. + uri = "table:test_hs06" + create_params = 'key_format={},value_format=S'.format(self.key_format) + self.session.create(uri, create_params) + + value1 = 'a' * 500 + value2 = 'b' * 500 + + self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(1)) + cursor = self.session.open_cursor(uri) + for i in range(1, 2000): + self.session.begin_transaction() + cursor[self.create_key(i)] = value1 + self.session.commit_transaction('commit_timestamp=' + timestamp_str(2)) + + # Load prepared data and leave it in a prepared state. + prepare_session = self.conn.open_session(self.session_config) + prepare_cursor = prepare_session.open_cursor(uri) + prepare_session.begin_transaction() + for i in range(1, 11): + prepare_cursor[self.create_key(i)] = value2 + prepare_session.prepare_transaction( + 'prepare_timestamp=' + timestamp_str(3)) + + # Write some more to cause eviction of the prepared data. + for i in range(11, 2000): + self.session.begin_transaction() + cursor[self.create_key(i)] = value2 + self.session.commit_transaction('commit_timestamp=' + timestamp_str(4)) + + self.session.checkpoint() + + # Try to read every key of the prepared data again. + # Ensure that we read the history store to find the prepared update and + # return a prepare conflict as appropriate. + self.session.begin_transaction('read_timestamp=' + timestamp_str(3)) + for i in range(1, 11): + cursor.set_key(self.create_key(i)) + self.assertRaisesException( + wiredtiger.WiredTigerError, + lambda: cursor.search(), + '/conflict with a prepared update/') + self.session.rollback_transaction() + + prepare_session.commit_transaction( + 'commit_timestamp=' + timestamp_str(5) + ',durable_timestamp=' + timestamp_str(6)) + + self.session.begin_transaction('read_timestamp=' + timestamp_str(5)) + for i in range(1, 11): + self.assertEquals(value2, cursor[self.create_key(i)]) + self.session.rollback_transaction() + + def test_hs_multiple_updates(self): + # Create a small table. + uri = "table:test_hs06" + create_params = 'key_format={},value_format=S'.format(self.key_format) + self.session.create(uri, create_params) + + value1 = 'a' * 500 + value2 = 'b' * 500 + value3 = 'c' * 500 + value4 = 'd' * 500 + + # Load 1Mb of data. + self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(1)) + cursor = self.session.open_cursor(uri) + for i in range(1, 2000): + self.session.begin_transaction() + cursor[self.create_key(i)] = value1 + self.session.commit_transaction('commit_timestamp=' + timestamp_str(2)) + + # Do two different updates to the same key with the same timestamp. + # We want to make sure that the second value is the one that is visible even after eviction. + for i in range(1, 11): + self.session.begin_transaction() + cursor[self.create_key(i)] = value2 + cursor[self.create_key(i)] = value3 + self.session.commit_transaction('commit_timestamp=' + timestamp_str(3)) + + # Write a newer value on top. + for i in range(1, 2000): + self.session.begin_transaction() + cursor[self.create_key(i)] = value4 + self.session.commit_transaction('commit_timestamp=' + timestamp_str(4)) + + # Ensure that we see the last of the two updates that got applied. + self.session.begin_transaction('read_timestamp=' + timestamp_str(3)) + for i in range(1, 11): + self.assertEquals(cursor[self.create_key(i)], value3) + self.session.rollback_transaction() + + def test_hs_multiple_modifies(self): + # Create a small table. + uri = "table:test_hs06" + create_params = 'key_format={},value_format=S'.format(self.key_format) + self.session.create(uri, create_params) + + value1 = 'a' * 500 + value2 = 'b' * 500 + + # Load 1Mb of data. + self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(1)) + cursor = self.session.open_cursor(uri) + for i in range(1, 2000): + self.session.begin_transaction() + cursor[self.create_key(i)] = value1 + self.session.commit_transaction('commit_timestamp=' + timestamp_str(2)) + + # Apply three sets of modifies. + # They specifically need to be in separate modify calls. + for i in range(1, 11): + self.session.begin_transaction() + cursor.set_key(self.create_key(i)) + self.assertEqual(cursor.modify([wiredtiger.Modify('B', 100, 1)]), 0) + self.assertEqual(cursor.modify([wiredtiger.Modify('C', 200, 1)]), 0) + self.assertEqual(cursor.modify([wiredtiger.Modify('D', 300, 1)]), 0) + self.session.commit_transaction('commit_timestamp=' + timestamp_str(3)) + + expected = list(value1) + expected[100] = 'B' + expected[200] = 'C' + expected[300] = 'D' + expected = str().join(expected) + + # Write a newer value on top. + for i in range(1, 2000): + self.session.begin_transaction() + cursor[self.create_key(i)] = value2 + self.session.commit_transaction('commit_timestamp=' + timestamp_str(4)) + + # Go back and read. We should get the initial value with the 3 modifies applied on top. + self.session.begin_transaction('read_timestamp=' + timestamp_str(3)) + for i in range(1, 11): + self.assertEqual(cursor[self.create_key(i)], expected) + self.session.rollback_transaction() + + def test_hs_instantiated_modify(self): + # Create a small table. + uri = "table:test_hs06" + create_params = 'key_format={},value_format=S'.format(self.key_format) + self.session.create(uri, create_params) + + value1 = 'a' * 500 + value2 = 'b' * 500 + + # Load 5Mb of data. + self.conn.set_timestamp( + 'oldest_timestamp=' + timestamp_str(1) + ',stable_timestamp=' + timestamp_str(1)) + cursor = self.session.open_cursor(uri) + for i in range(1, 10000): + self.session.begin_transaction() + cursor[self.create_key(i)] = value1 + self.session.commit_transaction('commit_timestamp=' + timestamp_str(2)) + + # Apply three sets of modifies. + for i in range(1, 11): + self.session.begin_transaction() + cursor.set_key(self.create_key(i)) + self.assertEqual(cursor.modify([wiredtiger.Modify('B', 100, 1)]), 0) + self.session.commit_transaction('commit_timestamp=' + timestamp_str(3)) + + for i in range(1, 11): + self.session.begin_transaction() + cursor.set_key(self.create_key(i)) + self.assertEqual(cursor.modify([wiredtiger.Modify('C', 200, 1)]), 0) + self.session.commit_transaction('commit_timestamp=' + timestamp_str(4)) + + # Since the stable timestamp is still at 1, there will be no birthmark record. + # History store instantiation should choose this update since it is the most recent. + # We want to check that it gets converted into a standard update as appropriate. + for i in range(1, 11): + self.session.begin_transaction() + cursor.set_key(self.create_key(i)) + self.assertEqual(cursor.modify([wiredtiger.Modify('D', 300, 1)]), 0) + self.session.commit_transaction('commit_timestamp=' + timestamp_str(5)) + + # Make a bunch of updates to another table to flush everything out of cache. + uri2 = 'table:test_hs06_extra' + self.session.create(uri2, create_params) + cursor2 = self.session.open_cursor(uri2) + for i in range(1, 10000): + self.session.begin_transaction() + cursor2[self.create_key(i)] = value2 + self.session.commit_transaction('commit_timestamp=' + timestamp_str(6)) + + expected = list(value1) + expected[100] = 'B' + expected[200] = 'C' + expected[300] = 'D' + expected = str().join(expected) + + # Go back and read. We should get the initial value with the 3 modifies applied on top. + self.session.begin_transaction('read_timestamp=' + timestamp_str(5)) + for i in range(1, 11): + self.assertEqual(cursor[self.create_key(i)], expected) + self.session.rollback_transaction() + + def test_hs_modify_birthmark_is_base_update(self): + # Create a small table. + uri = "table:test_hs06" + create_params = 'key_format={},value_format=S'.format(self.key_format) + self.session.create(uri, create_params) + + value1 = 'a' * 500 + value2 = 'b' * 500 + + # Load 5Mb of data. + self.conn.set_timestamp( + 'oldest_timestamp=' + timestamp_str(1) + ',stable_timestamp=' + timestamp_str(1)) + + # The base update is at timestamp 1. + # When we history store evict these pages, the base update will be used as the birthmark since + # it's the only thing behind the stable timestamp. + cursor = self.session.open_cursor(uri) + for i in range(1, 10000): + self.session.begin_transaction() + cursor[self.create_key(i)] = value1 + self.session.commit_transaction('commit_timestamp=' + timestamp_str(1)) + + # Apply three sets of modifies. + for i in range(1, 11): + self.session.begin_transaction() + cursor.set_key(self.create_key(i)) + self.assertEqual(cursor.modify([wiredtiger.Modify('B', 100, 1)]), 0) + self.session.commit_transaction('commit_timestamp=' + timestamp_str(2)) + + for i in range(1, 11): + self.session.begin_transaction() + cursor.set_key(self.create_key(i)) + self.assertEqual(cursor.modify([wiredtiger.Modify('C', 200, 1)]), 0) + self.session.commit_transaction('commit_timestamp=' + timestamp_str(3)) + + for i in range(1, 11): + self.session.begin_transaction() + cursor.set_key(self.create_key(i)) + self.assertEqual(cursor.modify([wiredtiger.Modify('D', 300, 1)]), 0) + self.session.commit_transaction('commit_timestamp=' + timestamp_str(4)) + + # Make a bunch of updates to another table to flush everything out of cache. + uri2 = 'table:test_hs06_extra' + self.session.create(uri2, create_params) + cursor2 = self.session.open_cursor(uri2) + for i in range(1, 10000): + self.session.begin_transaction() + cursor2[self.create_key(i)] = value2 + self.session.commit_transaction('commit_timestamp=' + timestamp_str(5)) + + expected = list(value1) + expected[100] = 'B' + expected[200] = 'C' + expected[300] = 'D' + expected = str().join(expected) + + # Go back and read. We should get the initial value with the 3 modifies applied on top. + # Ensure that we're aware that the birthmark update could be the base update. + self.session.begin_transaction('read_timestamp=' + timestamp_str(4)) + for i in range(1, 11): + self.assertEqual(cursor[self.create_key(i)], expected) + self.session.rollback_transaction() + + def test_hs_rec_modify(self): + # Create a small table. + uri = "table:test_hs06" + create_params = 'key_format={},value_format=S'.format(self.key_format) + self.session.create(uri, create_params) + + value1 = 'a' * 500 + value2 = 'b' * 500 + + self.conn.set_timestamp( + 'oldest_timestamp=' + timestamp_str(1) + ',stable_timestamp=' + timestamp_str(1)) + cursor = self.session.open_cursor(uri) + + # Base update. + for i in range(1, 10000): + self.session.begin_transaction() + cursor[self.create_key(i)] = value1 + self.session.commit_transaction('commit_timestamp=' + timestamp_str(2)) + + # Apply three sets of modifies. + for i in range(1, 11): + self.session.begin_transaction() + cursor.set_key(self.create_key(i)) + self.assertEqual(cursor.modify([wiredtiger.Modify('B', 100, 1)]), 0) + self.session.commit_transaction('commit_timestamp=' + timestamp_str(3)) + + for i in range(1, 11): + self.session.begin_transaction() + cursor.set_key(self.create_key(i)) + self.assertEqual(cursor.modify([wiredtiger.Modify('C', 200, 1)]), 0) + self.session.commit_transaction('commit_timestamp=' + timestamp_str(4)) + + # This is the one we want to be selected by the checkpoint. + for i in range(1, 11): + self.session.begin_transaction() + cursor.set_key(self.create_key(i)) + self.assertEqual(cursor.modify([wiredtiger.Modify('D', 300, 1)]), 0) + self.session.commit_transaction('commit_timestamp=' + timestamp_str(5)) + + # Apply another update and evict the pages with the modifies out of cache. + for i in range(1, 10000): + self.session.begin_transaction() + cursor[self.create_key(i)] = value2 + self.session.commit_transaction('commit_timestamp=' + timestamp_str(6)) + + # Checkpoint such that the modifies will be selected. When we grab it from the history + # store, we'll need to unflatten it before using it for reconciliation. + self.conn.set_timestamp('stable_timestamp=' + timestamp_str(5)) + self.session.checkpoint() + + expected = list(value1) + expected[100] = 'B' + expected[200] = 'C' + expected[300] = 'D' + expected = str().join(expected) + + # Check that the correct value is visible after checkpoint. + self.session.begin_transaction('read_timestamp=' + timestamp_str(5)) + for i in range(1, 11): + self.assertEqual(cursor[self.create_key(i)], expected) + self.session.rollback_transaction() + +if __name__ == '__main__': + wttest.run() diff --git a/src/third_party/wiredtiger/test/suite/test_hs07.py b/src/third_party/wiredtiger/test/suite/test_hs07.py new file mode 100644 index 00000000000..e2242cd4a7e --- /dev/null +++ b/src/third_party/wiredtiger/test/suite/test_hs07.py @@ -0,0 +1,197 @@ +#!/usr/bin/env python +# +# Public Domain 2014-2020 MongoDB, Inc. +# Public Domain 2008-2014 WiredTiger, Inc. +# +# This is free and unencumbered software released into the public domain. +# +# Anyone is free to copy, modify, publish, use, compile, sell, or +# distribute this software, either in source code form or as a compiled +# binary, for any purpose, commercial or non-commercial, and by any +# means. +# +# In jurisdictions that recognize copyright laws, the author or authors +# of this software dedicate any and all copyright interest in the +# software to the public domain. We make this dedication for the benefit +# of the public at large and to the detriment of our heirs and +# successors. We intend this dedication to be an overt act of +# relinquishment in perpetuity of all present and future rights to this +# software under copyright law. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +# IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. + +import time +from helper import copy_wiredtiger_home +import unittest, wiredtiger, wttest +from wtdataset import SimpleDataSet + +def timestamp_str(t): + return '%x' % t + +# test_hs07.py +# Test that the history store sweep cleans the obsolete history store entries and gives expected results. +class test_hs07(wttest.WiredTigerTestCase): + # Force a small cache. + conn_config = 'cache_size=50MB,log=(enabled)' + session_config = 'isolation=snapshot' + + def large_updates(self, uri, value, ds, nrows, commit_ts): + # Update a large number of records, we'll hang if the history store table isn't working. + session = self.session + cursor = session.open_cursor(uri) + for i in range(1, nrows + 1): + session.begin_transaction() + cursor[ds.key(i)] = value + session.commit_transaction('commit_timestamp=' + timestamp_str(commit_ts)) + cursor.close() + + def check(self, check_value, uri, nrows, read_ts): + session = self.session + session.begin_transaction('read_timestamp=' + timestamp_str(read_ts)) + cursor = session.open_cursor(uri) + count = 0 + for k, v in cursor: + self.assertEqual(v, check_value) + count += 1 + session.rollback_transaction() + self.assertEqual(count, nrows) + + def test_hs(self): + nrows = 10000 + + # Create a table without logging to ensure we get "skew_newest" history store eviction + # behavior. + uri = "table:las07_main" + ds = SimpleDataSet( + self, uri, 0, key_format="i", value_format="S", config='log=(enabled=false)') + ds.populate() + + uri2 = "table:las07_extra" + ds2 = SimpleDataSet(self, uri2, 0, key_format="i", value_format="S") + ds2.populate() + + # Pin oldest and stable to timestamp 1. + self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(1) + + ',stable_timestamp=' + timestamp_str(1)) + + bigvalue = "aaaaa" * 100 + bigvalue2 = "ddddd" * 100 + self.large_updates(uri, bigvalue, ds, nrows, 1) + + # Check that all updates are seen + self.check(bigvalue, uri, nrows, 100) + + # Force out most of the pages by updating a different tree + self.large_updates(uri2, bigvalue, ds2, nrows, 100) + + # Check that the new updates are only seen after the update timestamp + self.check(bigvalue, uri, nrows, 100) + + # Pin oldest and stable to timestamp 100. + self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(100) + + ',stable_timestamp=' + timestamp_str(100)) + + # Sleep here to let that sweep server to trigger cleanup of obsolete entries. + time.sleep(10) + + # Check that the new updates are only seen after the update timestamp + self.check(bigvalue, uri, nrows, 100) + + # Load a slight modification with a later timestamp. + cursor = self.session.open_cursor(uri) + self.session.begin_transaction() + for i in range(1, nrows): + cursor.set_key(i) + mods = [wiredtiger.Modify('A', 10, 1)] + self.assertEqual(cursor.modify(mods), 0) + self.session.commit_transaction('commit_timestamp=' + timestamp_str(110)) + + # Load a slight modification with a later timestamp. + self.session.begin_transaction() + for i in range(1, nrows): + cursor.set_key(i) + mods = [wiredtiger.Modify('B', 20, 1)] + self.assertEqual(cursor.modify(mods), 0) + self.session.commit_transaction('commit_timestamp=' + timestamp_str(120)) + + # Load a slight modification with a later timestamp. + self.session.begin_transaction() + for i in range(1, nrows): + cursor.set_key(i) + mods = [wiredtiger.Modify('C', 30, 1)] + self.assertEqual(cursor.modify(mods), 0) + self.session.commit_transaction('commit_timestamp=' + timestamp_str(130)) + cursor.close() + + # Second set of update operations with increased timestamp + self.large_updates(uri, bigvalue2, ds, nrows, 200) + + # Force out most of the pages by updating a different tree + self.large_updates(uri2, bigvalue2, ds2, nrows, 200) + + # Check that the new updates are only seen after the update timestamp + self.check(bigvalue2, uri, nrows, 200) + + # Pin oldest and stable to timestamp 300. + self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(200) + + ',stable_timestamp=' + timestamp_str(200)) + + # Sleep here to let that sweep server to trigger cleanup of obsolete entries. + time.sleep(10) + + # Check that the new updates are only seen after the update timestamp + self.check(bigvalue2, uri, nrows, 200) + + # Load a slight modification with a later timestamp. + cursor = self.session.open_cursor(uri) + self.session.begin_transaction() + for i in range(1, nrows): + cursor.set_key(i) + mods = [wiredtiger.Modify('A', 10, 1)] + self.assertEqual(cursor.modify(mods), 0) + self.session.commit_transaction('commit_timestamp=' + timestamp_str(210)) + + # Load a slight modification with a later timestamp. + self.session.begin_transaction() + for i in range(1, nrows): + cursor.set_key(i) + mods = [wiredtiger.Modify('B', 20, 1)] + self.assertEqual(cursor.modify(mods), 0) + self.session.commit_transaction('commit_timestamp=' + timestamp_str(220)) + + # Load a slight modification with a later timestamp. + self.session.begin_transaction() + for i in range(1, nrows): + cursor.set_key(i) + mods = [wiredtiger.Modify('C', 30, 1)] + self.assertEqual(cursor.modify(mods), 0) + self.session.commit_transaction('commit_timestamp=' + timestamp_str(230)) + cursor.close() + + # Third set of update operations with increased timestamp + self.large_updates(uri, bigvalue, ds, nrows, 300) + + # Force out most of the pages by updating a different tree + self.large_updates(uri2, bigvalue, ds2, nrows, 300) + + # Check that the new updates are only seen after the update timestamp + self.check(bigvalue, uri, nrows, 300) + + # Pin oldest and stable to timestamp 400. + self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(300) + + ',stable_timestamp=' + timestamp_str(300)) + + # Sleep here to let that sweep server to trigger cleanup of obsolete entries. + time.sleep(10) + + # Check that the new updates are only seen after the update timestamp + self.check(bigvalue, uri, nrows, 300) + +if __name__ == '__main__': + wttest.run() diff --git a/src/third_party/wiredtiger/test/suite/test_hs08.py b/src/third_party/wiredtiger/test/suite/test_hs08.py new file mode 100644 index 00000000000..48afefef941 --- /dev/null +++ b/src/third_party/wiredtiger/test/suite/test_hs08.py @@ -0,0 +1,201 @@ +#!/usr/bin/env python +# +# Public Domain 2014-2020 MongoDB, Inc. +# Public Domain 2008-2014 WiredTiger, Inc. +# +# This is free and unencumbered software released into the public domain. +# +# Anyone is free to copy, modify, publish, use, compile, sell, or +# distribute this software, either in source code form or as a compiled +# binary, for any purpose, commercial or non-commercial, and by any +# means. +# +# In jurisdictions that recognize copyright laws, the author or authors +# of this software dedicate any and all copyright interest in the +# software to the public domain. We make this dedication for the benefit +# of the public at large and to the detriment of our heirs and +# successors. We intend this dedication to be an overt act of +# relinquishment in perpetuity of all present and future rights to this +# software under copyright law. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +# IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. + +import unittest, wiredtiger, wttest, time +from wiredtiger import stat +from wtscenario import make_scenarios + +def timestamp_str(t): + return '%x' % t + +# test_hs08.py +# Verify modify insert into history store logic. +class test_hs08(wttest.WiredTigerTestCase): + conn_config = 'cache_size=100MB,statistics=(all)' + session_config = 'isolation=snapshot' + + def get_stat(self, stat): + stat_cursor = self.session.open_cursor('statistics:') + val = stat_cursor[stat][2] + stat_cursor.close() + return val + + def test_modify_insert_to_las(self): + uri = "table:test_hs08" + create_params = 'value_format=S,key_format=i' + value1 = 'a' * 1000 + self.session.create(uri, create_params) + + # Insert a full value. + self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(1)) + cursor = self.session.open_cursor(uri) + self.session.begin_transaction() + cursor[1] = value1 + self.session.commit_transaction('commit_timestamp=' + timestamp_str(2)) + + # Insert 3 modifies in separate transactions. + self.session.begin_transaction() + cursor.set_key(1) + self.assertEqual(cursor.modify([wiredtiger.Modify('A', 1000, 1)]), 0) + self.session.commit_transaction('commit_timestamp=' + timestamp_str(3)) + + self.session.begin_transaction() + cursor.set_key(1) + self.assertEqual(cursor.modify([wiredtiger.Modify('B', 1001, 1)]), 0) + self.session.commit_transaction('commit_timestamp=' + timestamp_str(4)) + + self.session.begin_transaction() + cursor.set_key(1) + self.assertEqual(cursor.modify([wiredtiger.Modify('C', 1002, 1)]), 0) + self.session.commit_transaction('commit_timestamp=' + timestamp_str(5)) + + # Call checkpoint. + self.session.checkpoint('use_timestamp=true') + + # Validate that we did write at least once to the history store. + hs_writes = self.get_stat(stat.conn.cache_write_hs) + squashed_write = self.get_stat(stat.conn.cache_hs_write_squash) + self.assertGreaterEqual(hs_writes, 1) + self.assertEqual(squashed_write, 0) + + # Validate that we see the correct value at each of the timestamps. + self.session.begin_transaction('read_timestamp=' + timestamp_str(3)) + self.assertEqual(cursor[1], value1 + 'A') + self.session.commit_transaction() + + self.session.begin_transaction('read_timestamp=' + timestamp_str(4)) + self.assertEqual(cursor[1], value1 + 'AB') + self.session.commit_transaction() + + self.session.begin_transaction('read_timestamp=' + timestamp_str(5)) + self.assertEqual(cursor[1], value1 + 'ABC') + self.session.commit_transaction() + + # Insert another two modifies. When we call checkpoint the first modify + # will get written to the data store as a full value and the second will + # be written to the data store as a reverse delta. + self.session.begin_transaction() + cursor.set_key(1) + self.assertEqual(cursor.modify([wiredtiger.Modify('D', 1000, 1)]), 0) + self.session.commit_transaction('commit_timestamp=' + timestamp_str(7)) + + self.session.begin_transaction() + cursor.set_key(1) + self.assertEqual(cursor.modify([wiredtiger.Modify('E', 1001, 1)]), 0) + self.session.commit_transaction('commit_timestamp=' + timestamp_str(8)) + + # Call checkpoint again. + self.session.checkpoint('use_timestamp=true') + + # Validate that we wrote to the history store again. + hs_writes = self.get_stat(stat.conn.cache_write_hs) + squashed_write = self.get_stat(stat.conn.cache_hs_write_squash) + self.assertGreaterEqual(hs_writes, 2) + self.assertEqual(squashed_write, 0) + + # Validate that we see the expected value on the modifies, this + # scenario tests the logic that will retrieve a full value for + # a modify previously inserted into the history store. + self.session.begin_transaction('read_timestamp=' + timestamp_str(7)) + self.assertEqual(cursor[1], value1 + 'DBC') + self.session.commit_transaction() + + self.session.begin_transaction('read_timestamp=' + timestamp_str(8)) + self.assertEqual(cursor[1], value1 + 'DEC') + self.session.commit_transaction() + + # Insert multiple modifies in the same transaction the first two should be squashed. + self.session.begin_transaction() + cursor.set_key(1) + self.assertEqual(cursor.modify([wiredtiger.Modify('F', 1002, 1)]), 0) + self.assertEqual(cursor.modify([wiredtiger.Modify('G', 1003, 1)]), 0) + self.assertEqual(cursor.modify([wiredtiger.Modify('H', 1004, 1)]), 0) + self.session.commit_transaction('commit_timestamp=' + timestamp_str(9)) + + # Call checkpoint again. + self.session.checkpoint('use_timestamp=true') + + # Validate that we squashed two modifies. Note that we cant count the exact number + # we squashed, just that we did squash. + hs_writes = self.get_stat(stat.conn.cache_write_hs) + squashed_write = self.get_stat(stat.conn.cache_hs_write_squash) + self.assertGreaterEqual(hs_writes, 3) + self.assertEqual(squashed_write, 1) + + # Insert multiple modifies in two different transactions so we should squash two. + self.session.begin_transaction() + cursor.set_key(1) + self.assertEqual(cursor.modify([wiredtiger.Modify('F', 1002, 1)]), 0) + self.assertEqual(cursor.modify([wiredtiger.Modify('G', 1003, 1)]), 0) + self.session.commit_transaction('commit_timestamp=' + timestamp_str(10)) + + self.session.begin_transaction() + cursor.set_key(1) + self.assertEqual(cursor.modify([wiredtiger.Modify('F', 1002, 1)]), 0) + self.assertEqual(cursor.modify([wiredtiger.Modify('G', 1003, 1)]), 0) + self.session.commit_transaction('commit_timestamp=' + timestamp_str(11)) + + # Call checkpoint again. + self.session.checkpoint('use_timestamp=true') + + # Validate that we squashed two modifies. We also squashed a modify that was previously + # squashed hence the number actually goes up by three. + hs_writes = self.get_stat(stat.conn.cache_write_hs) + squashed_write = self.get_stat(stat.conn.cache_hs_write_squash) + self.assertGreaterEqual(hs_writes, 4) + self.assertEqual(squashed_write, 4) + + # Insert multiple modifies in different transactions with different timestamps on each + # modify to guarantee we squash zero modifies. + self.session.begin_transaction() + cursor.set_key(1) + self.session.timestamp_transaction('commit_timestamp=' + timestamp_str(12)) + self.assertEqual(cursor.modify([wiredtiger.Modify('F', 1002, 1)]), 0) + self.session.timestamp_transaction('commit_timestamp=' + timestamp_str(13)) + self.assertEqual(cursor.modify([wiredtiger.Modify('G', 1003, 1)]), 0) + self.session.commit_transaction() + + self.session.begin_transaction() + cursor.set_key(1) + self.session.timestamp_transaction('commit_timestamp=' + timestamp_str(14)) + self.assertEqual(cursor.modify([wiredtiger.Modify('F', 1002, 1)]), 0) + self.session.timestamp_transaction('commit_timestamp=' + timestamp_str(15)) + self.assertEqual(cursor.modify([wiredtiger.Modify('G', 1003, 1)]), 0) + self.session.commit_transaction() + + # Call checkpoint again. + self.session.checkpoint('use_timestamp=true') + + # Validate that we squashed zero modifies. + hs_writes = self.get_stat(stat.conn.cache_write_hs) + squashed_write = self.get_stat(stat.conn.cache_hs_write_squash) + self.assertGreaterEqual(hs_writes, 5) + self.assertEqual(squashed_write, 5) + +if __name__ == '__main__': + wttest.run() diff --git a/src/third_party/wiredtiger/test/suite/test_hs09.py b/src/third_party/wiredtiger/test/suite/test_hs09.py new file mode 100644 index 00000000000..a0d6790a87a --- /dev/null +++ b/src/third_party/wiredtiger/test/suite/test_hs09.py @@ -0,0 +1,199 @@ +#!/usr/bin/env python +# +# Public Domain 2014-2020 MongoDB, Inc. +# Public Domain 2008-2014 WiredTiger, Inc. +# +# This is free and unencumbered software released into the public domain. +# +# Anyone is free to copy, modify, publish, use, compile, sell, or +# distribute this software, either in source code form or as a compiled +# binary, for any purpose, commercial or non-commercial, and by any +# means. +# +# In jurisdictions that recognize copyright laws, the author or authors +# of this software dedicate any and all copyright interest in the +# software to the public domain. We make this dedication for the benefit +# of the public at large and to the detriment of our heirs and +# successors. We intend this dedication to be an overt act of +# relinquishment in perpetuity of all present and future rights to this +# software under copyright law. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +# IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. + +import unittest, wiredtiger, wttest +from wiredtiger import stat +from wtscenario import make_scenarios + +def timestamp_str(t): + return '%x' % t + +# test_hs09.py +# Verify that we write the newest committed version to data store and the +# second newest committed version to history store. +class test_hs09(wttest.WiredTigerTestCase): + # Force a small cache. + conn_config = 'cache_size=50MB,statistics=(fast)' + session_config = 'isolation=snapshot' + uri = "table:test_hs09" + key_format_values = [ + ('column', dict(key_format='r')), + ('integer', dict(key_format='i')), + ('string', dict(key_format='S')), + ] + scenarios = make_scenarios(key_format_values) + + def create_key(self, i): + if self.key_format == 'S': + return str(i) + return i + + def check_ckpt_hs(self, expected_data_value, expected_hs_value, expected_hs_start_ts, expected_hs_stop_ts): + session = self.conn.open_session(self.session_config) + session.checkpoint() + # Check the data file value + cursor = session.open_cursor(self.uri, None, 'checkpoint=WiredTigerCheckpoint') + for _, value in cursor: + self.assertEqual(value, expected_data_value) + cursor.close() + # Check the history store file value + cursor = session.open_cursor("file:WiredTigerHS.wt", None, 'checkpoint=WiredTigerCheckpoint') + for _, _, hs_start_ts, _, hs_stop_ts, _, _, _, type, value in cursor: + # No WT_UPDATE_TOMBSTONE in the history store + self.assertNotEqual(type, 5) + # No WT_UPDATE_BIRTHMARK in the history store + self.assertNotEqual(type, 1) + # WT_UPDATE_STANDARD + if (type == 4): + self.assertEqual(value.decode(), expected_hs_value + '\x00') + self.assertEqual(hs_start_ts, expected_hs_start_ts) + self.assertEqual(hs_stop_ts, expected_hs_stop_ts) + cursor.close() + session.close() + + def test_uncommitted_updates_not_written_to_hs(self): + # Create a small table. + create_params = 'key_format={},value_format=S'.format(self.key_format) + self.session.create(self.uri, create_params) + + value1 = 'a' * 500 + value2 = 'b' * 500 + value3 = 'c' * 500 + + # Load 500KB of data. + self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(1)) + cursor = self.session.open_cursor(self.uri) + self.session.begin_transaction() + for i in range(1, 1000): + cursor[self.create_key(i)] = value1 + self.session.commit_transaction('commit_timestamp=' + timestamp_str(2)) + + # Load another 500KB of data with a later timestamp. + self.session.begin_transaction() + for i in range(1, 1000): + cursor[self.create_key(i)] = value2 + self.session.commit_transaction('commit_timestamp=' + timestamp_str(3)) + + # Uncommitted changes + self.session.begin_transaction() + for i in range(1, 11): + cursor[self.create_key(i)] = value3 + + self.check_ckpt_hs(value2, value1, 2, 3) + + def test_prepared_updates_not_written_to_hs(self): + # Create a small table. + create_params = 'key_format={},value_format=S'.format(self.key_format) + self.session.create(self.uri, create_params) + + value1 = 'a' * 500 + value2 = 'b' * 500 + value3 = 'c' * 500 + + # Load 1MB of data. + self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(1)) + cursor = self.session.open_cursor(self.uri) + self.session.begin_transaction() + for i in range(1, 2000): + cursor[self.create_key(i)] = value1 + self.session.commit_transaction('commit_timestamp=' + timestamp_str(2)) + + # Load another 1MB of data with a later timestamp. + self.session.begin_transaction() + for i in range(1, 2000): + cursor[self.create_key(i)] = value2 + self.session.commit_transaction('commit_timestamp=' + timestamp_str(3)) + + # Prepare some updates + self.session.begin_transaction() + for i in range(1, 11): + cursor[self.create_key(i)] = value3 + self.session.prepare_transaction('prepare_timestamp=' + timestamp_str(4)) + + self.check_ckpt_hs(value2, value1, 2, 3) + self.session.commit_transaction('commit_timestamp=' + timestamp_str(5) + + ',durable_timestamp=' + timestamp_str(5)) + + def test_write_newest_version_to_data_store(self): + # Create a small table. + create_params = 'key_format={},value_format=S'.format(self.key_format) + self.session.create(self.uri, create_params) + + value1 = 'a' * 500 + value2 = 'b' * 500 + + # Load 500KB of data. + self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(1)) + cursor = self.session.open_cursor(self.uri) + self.session.begin_transaction() + for i in range(1, 1000): + cursor[self.create_key(i)] = value1 + self.session.commit_transaction('commit_timestamp=' + timestamp_str(2)) + + # Load another 500KB of data with a later timestamp. + self.session.begin_transaction() + for i in range(1, 1000): + cursor[self.create_key(i)] = value2 + self.session.commit_transaction('commit_timestamp=' + timestamp_str(3)) + + self.check_ckpt_hs(value2, value1, 2, 3) + + def test_write_deleted_version_to_data_store(self): + # Create a small table. + create_params = 'key_format={},value_format=S'.format(self.key_format) + self.session.create(self.uri, create_params) + + value1 = 'a' * 500 + value2 = 'b' * 500 + + # Load 500KB of data. + self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(1)) + cursor = self.session.open_cursor(self.uri) + self.session.begin_transaction() + for i in range(1, 1000): + cursor[self.create_key(i)] = value1 + self.session.commit_transaction('commit_timestamp=' + timestamp_str(2)) + + # Load another 500KB of data with a later timestamp. + self.session.begin_transaction() + for i in range(1, 1000): + cursor[self.create_key(i)] = value2 + self.session.commit_transaction('commit_timestamp=' + timestamp_str(3)) + + # Delete records. + self.session.begin_transaction() + for i in range(1, 1000): + cursor = self.session.open_cursor(self.uri) + cursor.set_key(self.create_key(i)) + self.assertEqual(cursor.remove(), 0) + self.session.commit_transaction('commit_timestamp=' + timestamp_str(4)) + + self.check_ckpt_hs(value2, value1, 2, 3) + +if __name__ == '__main__': + wttest.run() diff --git a/src/third_party/wiredtiger/test/suite/test_hs10.py b/src/third_party/wiredtiger/test/suite/test_hs10.py new file mode 100644 index 00000000000..4ba3b25c4f0 --- /dev/null +++ b/src/third_party/wiredtiger/test/suite/test_hs10.py @@ -0,0 +1,107 @@ +#!/usr/bin/env python +# +# Public Domain 2014-2020 MongoDB, Inc. +# Public Domain 2008-2014 WiredTiger, Inc. +# +# This is free and unencumbered software released into the public domain. +# +# Anyone is free to copy, modify, publish, use, compile, sell, or +# distribute this software, either in source code form or as a compiled +# binary, for any purpose, commercial or non-commercial, and by any +# means. +# +# In jurisdictions that recognize copyright laws, the author or authors +# of this software dedicate any and all copyright interest in the +# software to the public domain. We make this dedication for the benefit +# of the public at large and to the detriment of our heirs and +# successors. We intend this dedication to be an overt act of +# relinquishment in perpetuity of all present and future rights to this +# software under copyright law. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +# IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. + +import unittest, wiredtiger, wttest, time +from wiredtiger import stat +from wtscenario import make_scenarios + +def timestamp_str(t): + return '%x' % t + +# test_hs10.py +# Verify modify read after eviction. +class test_hs10(wttest.WiredTigerTestCase): + conn_config = 'cache_size=2MB,statistics=(all),eviction=(threads_max=1)' + session_config = 'isolation=snapshot' + + def get_stat(self, stat): + stat_cursor = self.session.open_cursor('statistics:') + val = stat_cursor[stat][2] + stat_cursor.close() + return val + + def test_modify_insert_to_las(self): + uri = "table:test_hs10" + uri2 = "table:test_hs10_otherdata" + create_params = 'value_format=S,key_format=i' + value1 = 'a' * 1000 + value2 = 'b' * 1000 + self.session.create(uri, create_params) + session2 = self.setUpSessionOpen(self.conn) + session2.create(uri2, create_params) + cursor2 = session2.open_cursor(uri2) + # Insert a full value. + self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(1)) + self.conn.set_timestamp('stable_timestamp=' + timestamp_str(1)) + cursor = self.session.open_cursor(uri) + self.session.begin_transaction() + cursor[1] = value1 + self.session.commit_transaction('commit_timestamp=' + timestamp_str(2)) + + # Insert 3 modifies in separate transactions. + self.session.begin_transaction() + cursor.set_key(1) + self.assertEqual(cursor.modify([wiredtiger.Modify('A', 1000, 1)]), 0) + self.session.commit_transaction('commit_timestamp=' + timestamp_str(3)) + + self.session.begin_transaction() + cursor.set_key(1) + self.assertEqual(cursor.modify([wiredtiger.Modify('B', 1001, 1)]), 0) + self.session.commit_transaction('commit_timestamp=' + timestamp_str(4)) + + self.session.begin_transaction() + cursor.set_key(1) + self.assertEqual(cursor.modify([wiredtiger.Modify('C', 1002, 1)]), 0) + self.session.commit_transaction('commit_timestamp=' + timestamp_str(5)) + self.session.checkpoint() + + # Insert a whole bunch of data into the other table to force wiredtiger to evict data + # from the previous table. + for i in range(1, 10000): + cursor2[i] = value2 + + # Validate that we see the correct value at each of the timestamps. + self.session.begin_transaction('read_timestamp=' + timestamp_str(3)) + cursor.set_key(1) + cursor.search() + self.assertEqual(cursor[1], value1 + 'A') + self.session.commit_transaction() + + cursor2 = self.session.open_cursor(uri) + self.session.begin_transaction('read_timestamp=' + timestamp_str(4)) + cursor2.set_key(1) + cursor2.search() + self.assertEqual(cursor2.get_value(), value1 + 'AB') + self.session.commit_transaction() + + self.session.begin_transaction('read_timestamp=' + timestamp_str(5)) + self.assertEqual(cursor[1], value1 + 'ABC') + self.session.commit_transaction() + +if __name__ == '__main__': + wttest.run() diff --git a/src/third_party/wiredtiger/test/suite/test_hs11.py b/src/third_party/wiredtiger/test/suite/test_hs11.py new file mode 100644 index 00000000000..efc9d02401c --- /dev/null +++ b/src/third_party/wiredtiger/test/suite/test_hs11.py @@ -0,0 +1,83 @@ +#!/usr/bin/env python +# +# Public Domain 2014-2020 MongoDB, Inc. +# Public Domain 2008-2014 WiredTiger, Inc. +# +# This is free and unencumbered software released into the public domain. +# +# Anyone is free to copy, modify, publish, use, compile, sell, or +# distribute this software, either in source code form or as a compiled +# binary, for any purpose, commercial or non-commercial, and by any +# means. +# +# In jurisdictions that recognize copyright laws, the author or authors +# of this software dedicate any and all copyright interest in the +# software to the public domain. We make this dedication for the benefit +# of the public at large and to the detriment of our heirs and +# successors. We intend this dedication to be an overt act of +# relinquishment in perpetuity of all present and future rights to this +# software under copyright law. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +# IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. + +import wiredtiger, wttest + +def timestamp_str(t): + return '%x' % t + +# test_hs11.py +# Ensure that when we delete a key due to a tombstone being globally visible, we delete its +# associated history store content. +class test_hs11(wttest.WiredTigerTestCase): + conn_config = 'cache_size=50MB' + session_config = 'isolation=snapshot' + + def test_key_deletion_clears_hs(self): + uri = 'table:test_hs11' + create_params = 'key_format=S,value_format=S' + self.session.create(uri, create_params) + + value1 = 'a' * 500 + value2 = 'b' * 500 + + # Apply a series of updates from timestamps 1-4. + self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(1)) + cursor = self.session.open_cursor(uri) + for ts in range(1, 5): + for i in range(1, 10000): + self.session.begin_transaction() + cursor[str(i)] = value1 + self.session.commit_transaction('commit_timestamp=' + timestamp_str(ts)) + + # Reconcile and flush versions 1-3 to the history store. + self.session.checkpoint() + + # Apply a non-timestamped tombstone. When the pages get evicted, the keys will get deleted + # since the tombstone is globally visible. + for i in range(1, 10000): + if i % 2 == 0: + cursor.set_key(str(i)) + cursor.remove() + + # Now apply an update at timestamp 10 to recreate each key. + for i in range(1, 10000): + self.session.begin_transaction() + cursor[str(i)] = value2 + self.session.commit_transaction('commit_timestamp=' + timestamp_str(10)) + + # Ensure that we blew away history store content. + for ts in range(1, 5): + self.session.begin_transaction('read_timestamp=' + timestamp_str(ts)) + for i in range(1, 10000): + if i % 2 == 0: + cursor.set_key(str(i)) + self.assertEqual(cursor.search(), wiredtiger.WT_NOTFOUND) + else: + self.assertEqual(cursor[str(i)], value1) + self.session.rollback_transaction() diff --git a/src/third_party/wiredtiger/test/suite/test_inmem01.py b/src/third_party/wiredtiger/test/suite/test_inmem01.py index 879872b9c07..2bc7df0d403 100644 --- a/src/third_party/wiredtiger/test/suite/test_inmem01.py +++ b/src/third_party/wiredtiger/test/suite/test_inmem01.py @@ -26,7 +26,7 @@ # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. -import wiredtiger, wttest +import unittest, wiredtiger, wttest from time import sleep from wtdataset import SimpleDataSet from wtscenario import make_scenarios diff --git a/src/third_party/wiredtiger/test/suite/test_intpack.py b/src/third_party/wiredtiger/test/suite/test_intpack.py index 1a1c5725792..5a90f309b0b 100644 --- a/src/third_party/wiredtiger/test/suite/test_intpack.py +++ b/src/third_party/wiredtiger/test/suite/test_intpack.py @@ -30,7 +30,7 @@ # Tests integer packing using public methods # -import wiredtiger, wttest +import unittest, wiredtiger, wttest from wtscenario import make_scenarios class PackTester: diff --git a/src/third_party/wiredtiger/test/suite/test_jsondump01.py b/src/third_party/wiredtiger/test/suite/test_jsondump01.py index 97af4764622..e5bc170c45c 100644 --- a/src/third_party/wiredtiger/test/suite/test_jsondump01.py +++ b/src/third_party/wiredtiger/test/suite/test_jsondump01.py @@ -26,7 +26,7 @@ # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. -import os, json +import os, json, unittest import wiredtiger, wttest from wtdataset import SimpleDataSet, SimpleLSMDataSet, SimpleIndexDataSet, \ ComplexDataSet, ComplexLSMDataSet diff --git a/src/third_party/wiredtiger/test/suite/test_jsondump02.py b/src/third_party/wiredtiger/test/suite/test_jsondump02.py index eec0e5e97fd..0fbe4da25db 100755 --- a/src/third_party/wiredtiger/test/suite/test_jsondump02.py +++ b/src/third_party/wiredtiger/test/suite/test_jsondump02.py @@ -26,7 +26,7 @@ # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. -import os, sys +import os, sys, unittest import wiredtiger, wttest from suite_subprocess import suite_subprocess diff --git a/src/third_party/wiredtiger/test/suite/test_nsnap01.py b/src/third_party/wiredtiger/test/suite/test_nsnap01.py deleted file mode 100644 index 2a741100cfd..00000000000 --- a/src/third_party/wiredtiger/test/suite/test_nsnap01.py +++ /dev/null @@ -1,87 +0,0 @@ -#!/usr/bin/env python -# -# Public Domain 2014-2020 MongoDB, Inc. -# Public Domain 2008-2014 WiredTiger, Inc. -# -# This is free and unencumbered software released into the public domain. -# -# Anyone is free to copy, modify, publish, use, compile, sell, or -# distribute this software, either in source code form or as a compiled -# binary, for any purpose, commercial or non-commercial, and by any -# means. -# -# In jurisdictions that recognize copyright laws, the author or authors -# of this software dedicate any and all copyright interest in the -# software to the public domain. We make this dedication for the benefit -# of the public at large and to the detriment of our heirs and -# successors. We intend this dedication to be an overt act of -# relinquishment in perpetuity of all present and future rights to this -# software under copyright law. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, -# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF -# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. -# IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR -# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, -# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR -# OTHER DEALINGS IN THE SOFTWARE. -# -# test_nsnap01.py -# Named snapshots: basic API - -from suite_subprocess import suite_subprocess -from wtdataset import SimpleDataSet -import wiredtiger, wttest - -class test_nsnap01(wttest.WiredTigerTestCase, suite_subprocess): - tablename = 'test_nsnap01' - uri = 'table:' + tablename - nrows = 300 - nrows_per_snap = 10 - nsnapshots = 10 - - def check_named_snapshot(self, c, snapshot, expected): - c.reset() - self.session.begin_transaction("snapshot=" + str(snapshot)) - count = 0 - for row in c: - count += 1 - self.session.commit_transaction() - # print "Checking snapshot %d, expect %d, found %d" % (snapshot, expected, count) - self.assertEqual(count, expected) - - def test_named_snapshots(self): - # Populate a table - end = start = 0 - SimpleDataSet(self, self.uri, 0, key_format='i').populate() - - # Now run a workload: - # every iteration: - # create a new named snapshot, N - # append 20 rows and delete the first 10 - # verify that every snapshot N contains the expected number of rows - # if there are more than 10 snapshots active, drop the first half - snapshots = [] - c = self.session.open_cursor(self.uri) - for n in range(self.nrows // self.nrows_per_snap): - if len(snapshots) > self.nsnapshots: - middle = len(snapshots) // 2 - dropcfg = ",drop=(to=%d)" % snapshots[middle][0] - snapshots = snapshots[middle + 1:] - else: - dropcfg = "" - - self.session.snapshot("name=%d%s" % (n, dropcfg)) - snapshots.append((n, end - start)) - for i in range(2 * self.nrows_per_snap): - c[end + i] = "some value" - end += 2 * self.nrows_per_snap - for i in range(self.nrows_per_snap): - del c[start + i] - start += self.nrows_per_snap - - for snapshot, expected in snapshots: - self.check_named_snapshot(c, snapshot, expected) - -if __name__ == '__main__': - wttest.run() diff --git a/src/third_party/wiredtiger/test/suite/test_nsnap02.py b/src/third_party/wiredtiger/test/suite/test_nsnap02.py deleted file mode 100644 index f615b0a55bb..00000000000 --- a/src/third_party/wiredtiger/test/suite/test_nsnap02.py +++ /dev/null @@ -1,250 +0,0 @@ -#!/usr/bin/env python -# -# Public Domain 2014-2020 MongoDB, Inc. -# Public Domain 2008-2014 WiredTiger, Inc. -# -# This is free and unencumbered software released into the public domain. -# -# Anyone is free to copy, modify, publish, use, compile, sell, or -# distribute this software, either in source code form or as a compiled -# binary, for any purpose, commercial or non-commercial, and by any -# means. -# -# In jurisdictions that recognize copyright laws, the author or authors -# of this software dedicate any and all copyright interest in the -# software to the public domain. We make this dedication for the benefit -# of the public at large and to the detriment of our heirs and -# successors. We intend this dedication to be an overt act of -# relinquishment in perpetuity of all present and future rights to this -# software under copyright law. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, -# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF -# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. -# IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR -# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, -# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR -# OTHER DEALINGS IN THE SOFTWARE. -# -# test_nsnap02.py -# Named snapshots: Combinations of dropping snapshots - -from suite_subprocess import suite_subprocess -from wtdataset import SimpleDataSet -import wiredtiger, wttest - -class test_nsnap02(wttest.WiredTigerTestCase, suite_subprocess): - tablename = 'test_nsnap02' - uri = 'table:' + tablename - nrows = 1000 - nrows_per_snap = 10 - nsnapshots = 10 - - def check_named_snapshot(self, c, snapshot, expected): - c.reset() - self.session.begin_transaction("snapshot=" + str(snapshot)) - count = 0 - for row in c: - count += 1 - self.session.commit_transaction() - # print "Checking snapshot %d, expect %d, found %d" % (snapshot, expected, count) - self.assertEqual(count, expected) - - def check_named_snapshots(self, snapshots): - c = self.session.open_cursor(self.uri) - for snap_name, expected, dropped in snapshots: - if dropped == 0: - self.check_named_snapshot(c, snap_name, expected) - else: - self.assertRaisesWithMessage(wiredtiger.WiredTigerError, lambda: - self.session.begin_transaction("snapshot=%d" % (snap_name)), - "/Invalid argument/") - - def create_snapshots(self): - # Populate a table - end = start = 0 - SimpleDataSet(self, self.uri, 0, key_format='i').populate() - - # Create a set of snapshots - # Each snapshot has a bunch of new data - # Each snapshot removes a (smaller) bunch of old data - snapshots = [] - c = self.session.open_cursor(self.uri) - for n in range(self.nsnapshots): - self.session.snapshot("name=%d" % (n)) - snapshots.append((n, end - start, 0)) - for i in range(2 * self.nrows_per_snap): - c[end + i] = "some value" - end += 2 * self.nrows_per_snap - for i in range(self.nrows_per_snap): - del c[start + i] - start += self.nrows_per_snap - return snapshots - - def test_drop_all_snapshots(self): - snapshots = self.create_snapshots() - - self.check_named_snapshots(snapshots) - - self.session.snapshot("drop=(all)") - - new_snapshots = [] - for snap_name, expected, dropped in snapshots: - new_snapshots.append((snap_name, expected, 1)) - - # Make sure all the snapshots are gone. - self.check_named_snapshots(new_snapshots) - - def test_drop_first_snapshot(self): - snapshots = self.create_snapshots() - - c = self.session.open_cursor(self.uri) - for snap_name, expected, dropped in snapshots: - self.check_named_snapshot(c, snap_name, expected) - - self.session.snapshot("drop=(names=[0])") - - # Construct a snapshot array matching the expected state. - new_snapshots = [] - for snap_name, expected, dropped in snapshots: - if snap_name == 0: - new_snapshots.append((snap_name, expected, 1)) - else: - new_snapshots.append((snap_name, expected, 0)) - - # Make sure all the snapshots are gone. - self.check_named_snapshots(new_snapshots) - - def test_drop_to_first_snapshot(self): - snapshots = self.create_snapshots() - - c = self.session.open_cursor(self.uri) - for snap_name, expected, dropped in snapshots: - self.check_named_snapshot(c, snap_name, expected) - - self.session.snapshot("drop=(to=0)") - - # Construct a snapshot array matching the expected state. - new_snapshots = [] - for snap_name, expected, dropped in snapshots: - if snap_name == 0: - new_snapshots.append((snap_name, expected, 1)) - else: - new_snapshots.append((snap_name, expected, 0)) - - # Make sure all the snapshots are gone. - self.check_named_snapshots(new_snapshots) - - def test_drop_before_first_snapshot(self): - snapshots = self.create_snapshots() - - c = self.session.open_cursor(self.uri) - for snap_name, expected, dropped in snapshots: - self.check_named_snapshot(c, snap_name, expected) - - self.session.snapshot("drop=(before=0)") - - # Make sure no snapshots are gone - self.check_named_snapshots(snapshots) - - def test_drop_to_third_snapshot(self): - snapshots = self.create_snapshots() - - c = self.session.open_cursor(self.uri) - for snap_name, expected, dropped in snapshots: - self.check_named_snapshot(c, snap_name, expected) - - self.session.snapshot("drop=(to=3)") - - # Construct a snapshot array matching the expected state. - new_snapshots = [] - for snap_name, expected, dropped in snapshots: - if snap_name <= 3: - new_snapshots.append((snap_name, expected, 1)) - else: - new_snapshots.append((snap_name, expected, 0)) - - # Make sure all the snapshots are gone. - self.check_named_snapshots(new_snapshots) - - def test_drop_before_third_snapshot(self): - snapshots = self.create_snapshots() - - c = self.session.open_cursor(self.uri) - for snap_name, expected, dropped in snapshots: - self.check_named_snapshot(c, snap_name, expected) - - self.session.snapshot("drop=(before=3)") - - # Construct a snapshot array matching the expected state. - new_snapshots = [] - for snap_name, expected, dropped in snapshots: - if snap_name < 3: - new_snapshots.append((snap_name, expected, 1)) - else: - new_snapshots.append((snap_name, expected, 0)) - - # Make sure all the snapshots are gone. - self.check_named_snapshots(new_snapshots) - - def test_drop_to_last_snapshot(self): - snapshots = self.create_snapshots() - - c = self.session.open_cursor(self.uri) - for snap_name, expected, dropped in snapshots: - self.check_named_snapshot(c, snap_name, expected) - - self.session.snapshot("drop=(to=%d)" % (self.nsnapshots - 1)) - - # Construct a snapshot array matching the expected state. - new_snapshots = [] - for snap_name, expected, dropped in snapshots: - new_snapshots.append((snap_name, expected, 1)) - - # Make sure all the snapshots are gone. - self.check_named_snapshots(new_snapshots) - - def test_drop_before_last_snapshot(self): - snapshots = self.create_snapshots() - - c = self.session.open_cursor(self.uri) - for snap_name, expected, dropped in snapshots: - self.check_named_snapshot(c, snap_name, expected) - - self.session.snapshot("drop=(before=%d)" % (self.nsnapshots - 1)) - - # Construct a snapshot array matching the expected state. - new_snapshots = [] - for snap_name, expected, dropped in snapshots: - if snap_name < self.nsnapshots - 1: - new_snapshots.append((snap_name, expected, 1)) - else: - new_snapshots.append((snap_name, expected, 0)) - - # Make sure all the snapshots are gone. - self.check_named_snapshots(new_snapshots) - - def test_drop_specific_snapshots1(self): - snapshots = self.create_snapshots() - - c = self.session.open_cursor(self.uri) - for snap_name, expected, dropped in snapshots: - self.check_named_snapshot(c, snap_name, expected) - - self.session.snapshot( - "drop=(names=[%d,%d,%d])" % (0, 3, self.nsnapshots - 1)) - - # Construct a snapshot array matching the expected state. - new_snapshots = [] - for snap_name, expected, dropped in snapshots: - if snap_name == 0 or snap_name == 3 or \ - snap_name == self.nsnapshots - 1: - new_snapshots.append((snap_name, expected, 1)) - else: - new_snapshots.append((snap_name, expected, 0)) - - # Make sure all the snapshots are gone. - self.check_named_snapshots(new_snapshots) - -if __name__ == '__main__': - wttest.run() diff --git a/src/third_party/wiredtiger/test/suite/test_nsnap03.py b/src/third_party/wiredtiger/test/suite/test_nsnap03.py deleted file mode 100644 index 88021a70b3b..00000000000 --- a/src/third_party/wiredtiger/test/suite/test_nsnap03.py +++ /dev/null @@ -1,95 +0,0 @@ -#!/usr/bin/env python -# -# Public Domain 2014-2020 MongoDB, Inc. -# Public Domain 2008-2014 WiredTiger, Inc. -# -# This is free and unencumbered software released into the public domain. -# -# Anyone is free to copy, modify, publish, use, compile, sell, or -# distribute this software, either in source code form or as a compiled -# binary, for any purpose, commercial or non-commercial, and by any -# means. -# -# In jurisdictions that recognize copyright laws, the author or authors -# of this software dedicate any and all copyright interest in the -# software to the public domain. We make this dedication for the benefit -# of the public at large and to the detriment of our heirs and -# successors. We intend this dedication to be an overt act of -# relinquishment in perpetuity of all present and future rights to this -# software under copyright law. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, -# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF -# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. -# IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR -# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, -# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR -# OTHER DEALINGS IN THE SOFTWARE. -# -# test_nsnap03.py -# Named snapshots: Access and create from multiple sessions - -from suite_subprocess import suite_subprocess -from wtdataset import SimpleDataSet -import wiredtiger, wttest - -class test_nsnap03(wttest.WiredTigerTestCase, suite_subprocess): - tablename = 'test_nsnap03' - uri = 'table:' + tablename - nrows = 300 - nrows_per_snap = 10 - nsnapshots = 10 - - def check_named_snapshot(self, snapshot, expected): - new_session = self.conn.open_session() - new_session.begin_transaction("snapshot=" + str(snapshot)) - c = new_session.open_cursor(self.uri) - count = 0 - for row in c: - count += 1 - new_session.commit_transaction() - # print "Checking snapshot %d, expect %d, found %d" % (snapshot, expected, count) - self.assertEqual(count, expected) - new_session.close() - - def test_named_snapshots(self): - # Populate a table - end = start = 0 - SimpleDataSet(self, self.uri, 0, key_format='i').populate() - - # Now run a workload: - # every iteration: - # create a new named snapshot, N - # append 20 rows and delete the first 10 - # verify that every snapshot N contains the expected number of rows - # if there are more than 10 snapshots active, drop the first half - snapshots = [] - c = self.session.open_cursor(self.uri) - for n in range(self.nrows // self.nrows_per_snap): - if len(snapshots) > self.nsnapshots: - middle = len(snapshots) // 2 - dropcfg = ",drop=(to=%d)" % snapshots[middle][0] - snapshots = snapshots[middle + 1:] - else: - dropcfg = "" - - # Close and start a new session every three snapshots - if n % 3 == 0: - self.session.close() - self.session = self.conn.open_session() - c = self.session.open_cursor(self.uri) - - self.session.snapshot("name=%d%s" % (n, dropcfg)) - snapshots.append((n, end - start)) - for i in range(2 * self.nrows_per_snap): - c[end + i] = "some value" - end += 2 * self.nrows_per_snap - for i in range(self.nrows_per_snap): - del c[start + i] - start += self.nrows_per_snap - - for snapshot, expected in snapshots: - self.check_named_snapshot(snapshot, expected) - -if __name__ == '__main__': - wttest.run() diff --git a/src/third_party/wiredtiger/test/suite/test_nsnap04.py b/src/third_party/wiredtiger/test/suite/test_nsnap04.py deleted file mode 100644 index b1ef5c03889..00000000000 --- a/src/third_party/wiredtiger/test/suite/test_nsnap04.py +++ /dev/null @@ -1,117 +0,0 @@ -#!/usr/bin/env python -# -# Public Domain 2014-2020 MongoDB, Inc. -# Public Domain 2008-2014 WiredTiger, Inc. -# -# This is free and unencumbered software released into the public domain. -# -# Anyone is free to copy, modify, publish, use, compile, sell, or -# distribute this software, either in source code form or as a compiled -# binary, for any purpose, commercial or non-commercial, and by any -# means. -# -# In jurisdictions that recognize copyright laws, the author or authors -# of this software dedicate any and all copyright interest in the -# software to the public domain. We make this dedication for the benefit -# of the public at large and to the detriment of our heirs and -# successors. We intend this dedication to be an overt act of -# relinquishment in perpetuity of all present and future rights to this -# software under copyright law. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, -# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF -# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. -# IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR -# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, -# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR -# OTHER DEALINGS IN THE SOFTWARE. -# -# test_nsnap04.py -# Named snapshots: Create snapshot from running transaction - -import wiredtiger, wttest -from suite_subprocess import suite_subprocess -from wtdataset import SimpleDataSet - -class test_nsnap04(wttest.WiredTigerTestCase, suite_subprocess): - tablename = 'test_nsnap04' - uri = 'table:' + tablename - nrows_per_itr = 10 - - def check_named_snapshot(self, snapshot, expected, skip_snapshot=False): - new_session = self.conn.open_session() - c = new_session.open_cursor(self.uri) - if skip_snapshot: - new_session.begin_transaction() - else: - new_session.begin_transaction("snapshot=" + str(snapshot)) - count = 0 - for row in c: - count += 1 - new_session.commit_transaction() - new_session.close() - # print "Checking snapshot %d, expect %d, found %d" % (snapshot, expected, count) - self.assertEqual(count, expected) - - def test_named_snapshots(self): - # Populate a table - end = start = 0 - SimpleDataSet(self, self.uri, 0, key_format='i').populate() - - snapshots = [] - c = self.session.open_cursor(self.uri) - for i in range(self.nrows_per_itr): - c[i] = "some value" - - # Start a new transaction in a different session - new_session = self.conn.open_session() - new_session.begin_transaction("isolation=snapshot") - new_c = new_session.open_cursor(self.uri) - count = 0 - for row in new_c: - count += 1 - new_session.snapshot("name=0") - - self.check_named_snapshot(0, self.nrows_per_itr) - - # Insert some more content using the original session. - for i in range(self.nrows_per_itr): - c[2 * self.nrows_per_itr + i] = "some value" - - self.check_named_snapshot(0, self.nrows_per_itr) - new_session.close() - # Update the named snapshot - self.session.snapshot("name=0") - self.check_named_snapshot(0, 2 * self.nrows_per_itr) - - def test_include_updates(self): - # Populate a table - end = start = 0 - SimpleDataSet(self, self.uri, 0, key_format='i').populate() - - snapshots = [] - c = self.session.open_cursor(self.uri) - for i in range(self.nrows_per_itr): - c[i] = "some value" - - self.session.begin_transaction("isolation=snapshot") - count = 0 - for row in c: - count += 1 - self.session.snapshot("name=0,include_updates=true") - - self.check_named_snapshot(0, self.nrows_per_itr) - - # Insert some more content using the active session. - for i in range(self.nrows_per_itr): - c[self.nrows_per_itr + i] = "some value" - - self.check_named_snapshot(0, 2 * self.nrows_per_itr) - # Ensure transactions not tracking the snapshot don't see the updates - self.check_named_snapshot(0, self.nrows_per_itr, skip_snapshot=True) - self.session.commit_transaction() - # Ensure content is visible to non-snapshot transactions after commit - self.check_named_snapshot(0, 2 * self.nrows_per_itr, skip_snapshot=True) - -if __name__ == '__main__': - wttest.run() diff --git a/src/third_party/wiredtiger/test/suite/test_prepare02.py b/src/third_party/wiredtiger/test/suite/test_prepare02.py index e876898350c..315cf02329f 100644 --- a/src/third_party/wiredtiger/test/suite/test_prepare02.py +++ b/src/third_party/wiredtiger/test/suite/test_prepare02.py @@ -95,8 +95,6 @@ class test_prepare02(wttest.WiredTigerTestCase, suite_subprocess): self.assertTimestampsEqual(self.session.query_timestamp('get=prepare'), '2a') self.assertRaisesWithMessage(wiredtiger.WiredTigerError, lambda:self.session.checkpoint(), msg) - self.assertRaisesWithMessage(wiredtiger.WiredTigerError, - lambda: self.session.snapshot("name=test"), msg) # WT_SESSION.transaction_pinned_range permitted, not supported in the Python API. self.assertRaisesWithMessage(wiredtiger.WiredTigerError, lambda:self.session.transaction_sync(), msg) diff --git a/src/third_party/wiredtiger/test/suite/test_prepare07.py b/src/third_party/wiredtiger/test/suite/test_prepare07.py index da42843e26d..441cef2b5bf 100644 --- a/src/third_party/wiredtiger/test/suite/test_prepare07.py +++ b/src/third_party/wiredtiger/test/suite/test_prepare07.py @@ -28,7 +28,7 @@ import fnmatch, os, shutil, time from helper import copy_wiredtiger_home -import wiredtiger, wttest +import unittest, wiredtiger, wttest from wtdataset import SimpleDataSet def timestamp_str(t): diff --git a/src/third_party/wiredtiger/test/suite/test_prepare_lookaside01.py b/src/third_party/wiredtiger/test/suite/test_prepare_hs01.py index 0fa5b462d73..aa1a8e875ee 100755..100644 --- a/src/third_party/wiredtiger/test/suite/test_prepare_lookaside01.py +++ b/src/third_party/wiredtiger/test/suite/test_prepare_hs01.py @@ -27,36 +27,36 @@ # OTHER DEALINGS IN THE SOFTWARE. from helper import copy_wiredtiger_home -import wiredtiger, wttest +import unittest, wiredtiger, wttest from wtdataset import SimpleDataSet def timestamp_str(t): return '%x' % t -# test_prepare_lookaside01.py -# test to ensure lookaside eviction is working for prepared transactions. -class test_prepare_lookaside01(wttest.WiredTigerTestCase): +# test_prepare_hs01.py +# test to ensure history store eviction is working for prepared transactions. +class test_prepare_hs01(wttest.WiredTigerTestCase): # Force a small cache. conn_config = 'cache_size=50MB' def prepare_updates(self, uri, ds, nrows, nsessions, nkeys): # Update a large number of records in their individual transactions. - # This will force eviction and start lookaside eviction of committed + # This will force eviction and start history store eviction of committed # updates. # # Follow this by updating a number of records in prepared transactions - # under multiple sessions. We'll hang if lookaside table isn't doing its + # under multiple sessions. We'll hang if the history store table isn't doing its # thing. If we do all updates in a single session, then hang will be due # to uncommitted updates, instead of prepared updates. # # Do another set of updates in that many transactions. This forces the - # pages that have been evicted to lookaside to be re-read and brought in - # memory. Hence testing if we can read prepared updates from lookaside. + # pages that have been evicted to the history store to be re-read and brought in + # memory. Hence testing if we can read prepared updates from the history store. # Start with setting a stable timestamp to pin history in cache self.conn.set_timestamp('stable_timestamp=' + timestamp_str(1)) - # Commit some updates to get eviction and lookaside fired up + # Commit some updates to get eviction and history store fired up bigvalue1 = b"bbbbb" * 100 cursor = self.session.open_cursor(uri) for i in range(1, nsessions * nkeys): @@ -67,7 +67,7 @@ class test_prepare_lookaside01(wttest.WiredTigerTestCase): self.session.commit_transaction('commit_timestamp=' + timestamp_str(1)) # Have prepared updates in multiple sessions. This should ensure writing - # prepared updates to the lookaside + # prepared updates to the history store sessions = [0] * nsessions cursors = [0] * nsessions bigvalue2 = b"ccccc" * 100 @@ -86,7 +86,7 @@ class test_prepare_lookaside01(wttest.WiredTigerTestCase): # Re-read the original versions of all the data. To do this, the pages # that were just evicted need to be read back. This ensures reading - # prepared updates from the lookaside + # prepared updates from the history store cursor = self.session.open_cursor(uri) self.session.begin_transaction('read_timestamp=' + timestamp_str(1)) for i in range(1, nsessions * nkeys): @@ -101,9 +101,10 @@ class test_prepare_lookaside01(wttest.WiredTigerTestCase): cursors[j].close() sessions[j].close() - def test_prepare_lookaside(self): + @unittest.skip("Temporarily disabled") + def test_prepare_hs(self): # Create a small table. - uri = "table:test_prepare_lookaside01" + uri = "table:test_prepare_hs01" nrows = 100 ds = SimpleDataSet(self, uri, nrows, key_format="S", value_format='u') ds.populate() @@ -118,7 +119,7 @@ class test_prepare_lookaside01(wttest.WiredTigerTestCase): cursor.close() self.session.checkpoint() - # Check if lookaside is working properly with prepare transactions. + # Check if the history store is working properly with prepare transactions. # We put prepared updates in multiple sessions so that we do not hang # because of cache being full with uncommitted updates. nsessions = 3 diff --git a/src/third_party/wiredtiger/test/suite/test_prepare_lookaside02.py b/src/third_party/wiredtiger/test/suite/test_prepare_hs02.py index 87eb49158ca..073cf0321e5 100644 --- a/src/third_party/wiredtiger/test/suite/test_prepare_lookaside02.py +++ b/src/third_party/wiredtiger/test/suite/test_prepare_hs02.py @@ -26,7 +26,7 @@ # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. # -# test_prepare_lookaside02.py +# test_prepare_hs02.py # Prepare updates can be resolved for both commit // rollback operations. # @@ -39,7 +39,7 @@ from wtscenario import make_scenarios def timestamp_str(t): return '%x' % t -class test_prepare_lookaside02(wttest.WiredTigerTestCase, suite_subprocess): +class test_prepare_hs02(wttest.WiredTigerTestCase, suite_subprocess): tablename = 'test_prepare_cursor' uri = 'table:' + tablename txn_config = 'isolation=snapshot' diff --git a/src/third_party/wiredtiger/test/suite/test_rollback_to_stable01.py b/src/third_party/wiredtiger/test/suite/test_rollback_to_stable01.py new file mode 100755 index 00000000000..d2891ef9c4c --- /dev/null +++ b/src/third_party/wiredtiger/test/suite/test_rollback_to_stable01.py @@ -0,0 +1,199 @@ +#!/usr/bin/env python +# +# Public Domain 2014-2020 MongoDB, Inc. +# Public Domain 2008-2014 WiredTiger, Inc. +# +# This is free and unencumbered software released into the public domain. +# +# Anyone is free to copy, modify, publish, use, compile, sell, or +# distribute this software, either in source code form or as a compiled +# binary, for any purpose, commercial or non-commercial, and by any +# means. +# +# In jurisdictions that recognize copyright laws, the author or authors +# of this software dedicate any and all copyright interest in the +# software to the public domain. We make this dedication for the benefit +# of the public at large and to the detriment of our heirs and +# successors. We intend this dedication to be an overt act of +# relinquishment in perpetuity of all present and future rights to this +# software under copyright law. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +# IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. + +import time +from helper import copy_wiredtiger_home +import unittest, wiredtiger, wttest +from wtdataset import SimpleDataSet +from wiredtiger import stat +from wtscenario import make_scenarios + +def timestamp_str(t): + return '%x' % t + +# test_rollback_to_stable01.py +# Shared base class used by gc tests. +class test_rollback_to_stable_base(wttest.WiredTigerTestCase): + def large_updates(self, uri, value, ds, nrows, commit_ts): + # Update a large number of records. + session = self.session + cursor = session.open_cursor(uri) + for i in range(0, nrows): + session.begin_transaction() + cursor[ds.key(i)] = value + if commit_ts == 0: + session.commit_transaction() + elif self.prepare: + session.prepare_transaction('prepare_timestamp=' + timestamp_str(commit_ts-1)) + session.timestamp_transaction('commit_timestamp=' + timestamp_str(commit_ts)) + session.timestamp_transaction('durable_timestamp=' + timestamp_str(commit_ts+1)) + session.commit_transaction() + else: + session.commit_transaction('commit_timestamp=' + timestamp_str(commit_ts)) + cursor.close() + + def large_modifies(self, uri, value, ds, location, nbytes, nrows, commit_ts): + # Load a slight modification. + session = self.session + cursor = session.open_cursor(uri) + session.begin_transaction() + for i in range(0, nrows): + cursor.set_key(i) + mods = [wiredtiger.Modify(value, location, nbytes)] + self.assertEqual(cursor.modify(mods), 0) + + if commit_ts == 0: + session.commit_transaction() + elif self.prepare: + session.prepare_transaction('prepare_timestamp=' + timestamp_str(commit_ts-1)) + session.timestamp_transaction('commit_timestamp=' + timestamp_str(commit_ts)) + session.timestamp_transaction('durable_timestamp=' + timestamp_str(commit_ts+1)) + session.commit_transaction() + else: + session.commit_transaction('commit_timestamp=' + timestamp_str(commit_ts)) + cursor.close() + + def large_removes(self, uri, ds, nrows, commit_ts): + # Remove a large number of records. + session = self.session + cursor = session.open_cursor(uri) + for i in range(0, nrows): + session.begin_transaction() + cursor.set_key(i) + cursor.remove() + if commit_ts == 0: + session.commit_transaction() + elif self.prepare: + session.prepare_transaction('prepare_timestamp=' + timestamp_str(commit_ts-1)) + session.timestamp_transaction('commit_timestamp=' + timestamp_str(commit_ts)) + session.timestamp_transaction('durable_timestamp=' + timestamp_str(commit_ts+1)) + session.commit_transaction() + else: + session.commit_transaction('commit_timestamp=' + timestamp_str(commit_ts)) + cursor.close() + + def check(self, check_value, uri, nrows, read_ts): + session = self.session + if read_ts == 0: + session.begin_transaction() + else: + session.begin_transaction('read_timestamp=' + timestamp_str(read_ts)) + cursor = session.open_cursor(uri) + count = 0 + for k, v in cursor: + self.assertEqual(v, check_value) + count += 1 + session.commit_transaction() + self.assertEqual(count, nrows) + +# Test that rollback to stable clears the remove operation. +class test_rollback_to_stable01(test_rollback_to_stable_base): + session_config = 'isolation=snapshot' + + in_memory_values = [ + ('no_inmem', dict(in_memory=False)), + ('inmem', dict(in_memory=True)) + ] + + prepare_values = [ + ('no_prepare', dict(prepare=False)), + ('prepare', dict(prepare=True)) + ] + + scenarios = make_scenarios(in_memory_values, prepare_values) + + def conn_config(self): + config = '' + # Temporarily solution to have good cache size until prepare updates are written to disk. + if self.prepare: + config += 'cache_size=100MB,statistics=(all)' + else: + config += 'cache_size=50MB,statistics=(all)' + if self.in_memory: + config += ',in_memory=true' + else: + config += ',log=(enabled),in_memory=false' + return config + + def test_rollback_to_stable(self): + nrows = 10000 + + # Create a table without logging. + uri = "table:rollback_to_stable01" + ds = SimpleDataSet( + self, uri, 0, key_format="i", value_format="S", config='log=(enabled=false)') + ds.populate() + + # Pin oldest and stable to timestamp 1. + self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(1) + + ',stable_timestamp=' + timestamp_str(1)) + + valuea = "aaaaa" * 100 + self.large_updates(uri, valuea, ds, nrows, 10) + # Check that all updates are seen. + self.check(valuea, uri, nrows, 10) + + # Remove all keys with newer timestamp. + self.large_removes(uri, ds, nrows, 20) + # Check that the no keys should be visible. + self.check(valuea, uri, 0, 20) + + # Pin stable to timestamp 20 if prepare otherwise 10. + if self.prepare: + self.conn.set_timestamp('stable_timestamp=' + timestamp_str(20)) + else: + self.conn.set_timestamp('stable_timestamp=' + timestamp_str(10)) + # Checkpoint to ensure that all the updates are flushed to disk. + if not self.in_memory: + self.session.checkpoint() + + self.conn.rollback_to_stable() + # Check that the new updates are only seen after the update timestamp. + self.check(valuea, uri, nrows, 20) + + stat_cursor = self.session.open_cursor('statistics:', None, None) + calls = stat_cursor[stat.conn.txn_rts][2] + hs_removed = stat_cursor[stat.conn.txn_rts_hs_removed][2] + keys_removed = stat_cursor[stat.conn.txn_rts_keys_removed][2] + keys_restored = stat_cursor[stat.conn.txn_rts_keys_restored][2] + pages_visited = stat_cursor[stat.conn.txn_rts_pages_visited][2] + upd_aborted = stat_cursor[stat.conn.txn_rts_upd_aborted][2] + stat_cursor.close() + + self.assertEqual(calls, 1) + self.assertEqual(hs_removed, 0) + self.assertEqual(keys_removed, 0) + if self.in_memory: + self.assertEqual(upd_aborted, nrows) + else: + self.assertEqual(upd_aborted + keys_restored, nrows) + self.assertGreaterEqual(keys_restored, 0) + self.assertGreater(pages_visited, 0) + +if __name__ == '__main__': + wttest.run() diff --git a/src/third_party/wiredtiger/test/suite/test_rollback_to_stable02.py b/src/third_party/wiredtiger/test/suite/test_rollback_to_stable02.py new file mode 100755 index 00000000000..fbdaaa9b2f9 --- /dev/null +++ b/src/third_party/wiredtiger/test/suite/test_rollback_to_stable02.py @@ -0,0 +1,134 @@ +#!/usr/bin/env python +# +# Public Domain 2014-2020 MongoDB, Inc. +# Public Domain 2008-2014 WiredTiger, Inc. +# +# This is free and unencumbered software released into the public domain. +# +# Anyone is free to copy, modify, publish, use, compile, sell, or +# distribute this software, either in source code form or as a compiled +# binary, for any purpose, commercial or non-commercial, and by any +# means. +# +# In jurisdictions that recognize copyright laws, the author or authors +# of this software dedicate any and all copyright interest in the +# software to the public domain. We make this dedication for the benefit +# of the public at large and to the detriment of our heirs and +# successors. We intend this dedication to be an overt act of +# relinquishment in perpetuity of all present and future rights to this +# software under copyright law. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +# IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. + +import time +from helper import copy_wiredtiger_home +import unittest, wiredtiger, wttest +from wtdataset import SimpleDataSet +from wiredtiger import stat +from wtscenario import make_scenarios +from test_rollback_to_stable01 import test_rollback_to_stable_base + +def timestamp_str(t): + return '%x' % t + +# test_rollback_to_stable02.py +# Test that rollback to stable brings back the history value to replace on-disk value. +class test_rollback_to_stable02(test_rollback_to_stable_base): + session_config = 'isolation=snapshot' + + in_memory_values = [ + ('no_inmem', dict(in_memory=False)), + ('inmem', dict(in_memory=True)) + ] + + prepare_values = [ + ('no_prepare', dict(prepare=False)), + ('prepare', dict(prepare=True)) + ] + + scenarios = make_scenarios(in_memory_values, prepare_values) + + def conn_config(self): + config = '' + # Temporarily solution to have good cache size until prepare updates are written to disk. + if self.prepare: + config += 'cache_size=250MB,statistics=(all)' + else: + config += 'cache_size=100MB,statistics=(all)' + if self.in_memory: + config += ',in_memory=true' + else: + config += ',log=(enabled),in_memory=false' + return config + + def test_rollback_to_stable(self): + nrows = 10000 + + # Create a table without logging. + uri = "table:rollback_to_stable02" + ds = SimpleDataSet( + self, uri, 0, key_format="i", value_format="S", config='log=(enabled=false)') + ds.populate() + + # Pin oldest and stable to timestamp 1. + self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(1) + + ',stable_timestamp=' + timestamp_str(1)) + + valuea = "aaaaa" * 100 + valueb = "bbbbb" * 100 + valuec = "ccccc" * 100 + valued = "ddddd" * 100 + self.large_updates(uri, valuea, ds, nrows, 10) + # Check that all updates are seen. + self.check(valuea, uri, nrows, 10) + + self.large_updates(uri, valueb, ds, nrows, 20) + # Check that the new updates are only seen after the update timestamp. + self.check(valueb, uri, nrows, 20) + + self.large_updates(uri, valuec, ds, nrows, 30) + # Check that the new updates are only seen after the update timestamp. + self.check(valuec, uri, nrows, 30) + + self.large_updates(uri, valued, ds, nrows, 40) + # Check that the new updates are only seen after the update timestamp. + self.check(valued, uri, nrows, 40) + + # Pin stable to timestamp 30 if prepare otherwise 20. + if self.prepare: + self.conn.set_timestamp('stable_timestamp=' + timestamp_str(30)) + else: + self.conn.set_timestamp('stable_timestamp=' + timestamp_str(20)) + # Checkpoint to ensure that all the data is flushed. + if not self.in_memory: + self.session.checkpoint() + + self.conn.rollback_to_stable() + # Check that the new updates are only seen after the update timestamp. + self.check(valueb, uri, nrows, 40) + self.check(valueb, uri, nrows, 20) + self.check(valuea, uri, nrows, 10) + + stat_cursor = self.session.open_cursor('statistics:', None, None) + calls = stat_cursor[stat.conn.txn_rts][2] + upd_aborted = (stat_cursor[stat.conn.txn_rts_upd_aborted][2] + + stat_cursor[stat.conn.txn_rts_hs_removed][2]) + keys_removed = stat_cursor[stat.conn.txn_rts_keys_removed][2] + keys_restored = stat_cursor[stat.conn.txn_rts_keys_restored][2] + pages_visited = stat_cursor[stat.conn.txn_rts_pages_visited][2] + stat_cursor.close() + + self.assertEqual(calls, 1) + self.assertEqual(keys_removed, 0) + self.assertEqual(keys_restored, 0) + self.assertGreater(pages_visited, 0) + self.assertGreaterEqual(upd_aborted, nrows * 2) + +if __name__ == '__main__': + wttest.run() diff --git a/src/third_party/wiredtiger/test/suite/test_rollback_to_stable03.py b/src/third_party/wiredtiger/test/suite/test_rollback_to_stable03.py new file mode 100755 index 00000000000..88bed706e7b --- /dev/null +++ b/src/third_party/wiredtiger/test/suite/test_rollback_to_stable03.py @@ -0,0 +1,127 @@ +#!/usr/bin/env python +# +# Public Domain 2014-2020 MongoDB, Inc. +# Public Domain 2008-2014 WiredTiger, Inc. +# +# This is free and unencumbered software released into the public domain. +# +# Anyone is free to copy, modify, publish, use, compile, sell, or +# distribute this software, either in source code form or as a compiled +# binary, for any purpose, commercial or non-commercial, and by any +# means. +# +# In jurisdictions that recognize copyright laws, the author or authors +# of this software dedicate any and all copyright interest in the +# software to the public domain. We make this dedication for the benefit +# of the public at large and to the detriment of our heirs and +# successors. We intend this dedication to be an overt act of +# relinquishment in perpetuity of all present and future rights to this +# software under copyright law. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +# IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. + +import time +from helper import copy_wiredtiger_home +import unittest, wiredtiger, wttest +from wtdataset import SimpleDataSet +from wiredtiger import stat +from wtscenario import make_scenarios +from test_rollback_to_stable01 import test_rollback_to_stable_base + +def timestamp_str(t): + return '%x' % t + +# test_rollback_to_stable03.py +# Test that rollback to stable clears the history store updates from reconciled pages. +class test_rollback_to_stable01(test_rollback_to_stable_base): + session_config = 'isolation=snapshot' + + in_memory_values = [ + ('no_inmem', dict(in_memory=False)), + ('inmem', dict(in_memory=True)) + ] + + prepare_values = [ + ('no_prepare', dict(prepare=False)), + ('prepare', dict(prepare=True)) + ] + + scenarios = make_scenarios(in_memory_values, prepare_values) + + def conn_config(self): + config = 'cache_size=4GB,statistics=(all)' + if self.in_memory: + config += ',in_memory=true' + else: + config += ',log=(enabled),in_memory=false' + return config + + def test_rollback_to_stable(self): + nrows = 1000 + + # Create a table without logging. + uri = "table:rollback_to_stable03" + ds = SimpleDataSet( + self, uri, 0, key_format="i", value_format="S", config='log=(enabled=false)') + ds.populate() + + # Pin oldest and stable to timestamp 1. + self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(1) + + ',stable_timestamp=' + timestamp_str(1)) + + valuea = "aaaaa" * 100 + valueb = "bbbbb" * 100 + valuec = "ccccc" * 100 + self.large_updates(uri, valuea, ds, nrows, 10) + # Check that all updates are seen. + self.check(valuea, uri, nrows, 10) + + self.large_updates(uri, valueb, ds, nrows, 20) + # Check that all updates are seen. + self.check(valueb, uri, nrows, 20) + + self.large_updates(uri, valuec, ds, nrows, 30) + # Check that all updates are seen. + self.check(valuec, uri, nrows, 30) + + # Pin stable to timestamp 30 if prepare otherwise 20. + if self.prepare: + self.conn.set_timestamp('stable_timestamp=' + timestamp_str(30)) + else: + self.conn.set_timestamp('stable_timestamp=' + timestamp_str(20)) + # Checkpoint to ensure that all the updates are flushed to disk. + if not self.in_memory: + self.session.checkpoint() + + self.conn.rollback_to_stable() + # Check that the old updates are only seen even with the update timestamp. + self.check(valueb, uri, nrows, 20) + self.check(valuea, uri, nrows, 10) + + stat_cursor = self.session.open_cursor('statistics:', None, None) + calls = stat_cursor[stat.conn.txn_rts][2] + hs_removed = stat_cursor[stat.conn.txn_rts_hs_removed][2] + keys_removed = stat_cursor[stat.conn.txn_rts_keys_removed][2] + keys_restored = stat_cursor[stat.conn.txn_rts_keys_restored][2] + pages_visited = stat_cursor[stat.conn.txn_rts_pages_visited][2] + upd_aborted = stat_cursor[stat.conn.txn_rts_upd_aborted][2] + stat_cursor.close() + + self.assertEqual(calls, 1) + self.assertEqual(keys_removed, 0) + self.assertEqual(keys_restored, 0) + if self.in_memory or self.prepare: + self.assertEqual(hs_removed, 0) + else: + self.assertEqual(hs_removed, nrows) + self.assertEqual(upd_aborted, nrows) + self.assertGreater(pages_visited, 0) + +if __name__ == '__main__': + wttest.run() diff --git a/src/third_party/wiredtiger/test/suite/test_rollback_to_stable04.py b/src/third_party/wiredtiger/test/suite/test_rollback_to_stable04.py new file mode 100755 index 00000000000..2522d7e7611 --- /dev/null +++ b/src/third_party/wiredtiger/test/suite/test_rollback_to_stable04.py @@ -0,0 +1,165 @@ +#!/usr/bin/env python +# +# Public Domain 2014-2020 MongoDB, Inc. +# Public Domain 2008-2014 WiredTiger, Inc. +# +# This is free and unencumbered software released into the public domain. +# +# Anyone is free to copy, modify, publish, use, compile, sell, or +# distribute this software, either in source code form or as a compiled +# binary, for any purpose, commercial or non-commercial, and by any +# means. +# +# In jurisdictions that recognize copyright laws, the author or authors +# of this software dedicate any and all copyright interest in the +# software to the public domain. We make this dedication for the benefit +# of the public at large and to the detriment of our heirs and +# successors. We intend this dedication to be an overt act of +# relinquishment in perpetuity of all present and future rights to this +# software under copyright law. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +# IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. + +from wiredtiger import stat +from wtdataset import SimpleDataSet +from wtscenario import make_scenarios +from test_rollback_to_stable01 import test_rollback_to_stable_base + +def timestamp_str(t): + return '%x' % t + +def mod_val(value, char, location, nbytes=1): + return value[0:location] + char + value[location+nbytes:] + +# test_rollback_to_stable04.py +# Test that rollback to stable always replaces the on-disk value with a full update +# from the history store. +class test_rollback_to_stable04(test_rollback_to_stable_base): + session_config = 'isolation=snapshot' + + in_memory_values = [ + ('no_inmem', dict(in_memory=False)), + ('inmem', dict(in_memory=True)) + ] + + prepare_values = [ + ('no_prepare', dict(prepare=False)), + ('prepare', dict(prepare=True)) + ] + + scenarios = make_scenarios(in_memory_values, prepare_values) + + def conn_config(self): + config = '' + # Temporarily solution to have good cache size until prepare updates are written to disk. + if self.prepare: + config += 'cache_size=2GB,statistics=(all)' + else: + config += 'cache_size=500MB,statistics=(all)' + if self.in_memory: + config += ',in_memory=true' + else: + config += ',log=(enabled),in_memory=false' + return config + + def test_rollback_to_stable(self): + nrows = 1000 + + # Create a table without logging. + uri = "table:rollback_to_stable04" + ds = SimpleDataSet( + self, uri, 0, key_format="i", value_format="S", config='log=(enabled=false)') + ds.populate() + + # Pin oldest and stable to timestamp 10. + self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(10) + + ',stable_timestamp=' + timestamp_str(10)) + + value_a = "aaaaa" * 100 + value_b = "bbbbb" * 100 + value_c = "ccccc" * 100 + value_d = "ddddd" * 100 + + value_modQ = mod_val(value_a, 'Q', 0) + value_modR = mod_val(value_modQ, 'R', 1) + value_modS = mod_val(value_modR, 'S', 2) + value_modT = mod_val(value_c, 'T', 3) + value_modW = mod_val(value_d, 'W', 4) + value_modX = mod_val(value_a, 'X', 5) + value_modY = mod_val(value_modX, 'Y', 6) + value_modZ = mod_val(value_modY, 'Z', 7) + + # Perform a combination of modifies and updates. + self.large_updates(uri, value_a, ds, nrows, 20) + self.large_modifies(uri, 'Q', ds, 0, 1, nrows, 30) + self.large_modifies(uri, 'R', ds, 1, 1, nrows, 40) + self.large_modifies(uri, 'S', ds, 2, 1, nrows, 50) + self.large_updates(uri, value_b, ds, nrows, 60) + self.large_updates(uri, value_c, ds, nrows, 70) + self.large_modifies(uri, 'T', ds, 3, 1, nrows, 80) + self.large_updates(uri, value_d, ds, nrows, 90) + self.large_modifies(uri, 'W', ds, 4, 1, nrows, 100) + self.large_updates(uri, value_a, ds, nrows, 110) + self.large_modifies(uri, 'X', ds, 5, 1, nrows, 120) + self.large_modifies(uri, 'Y', ds, 6, 1, nrows, 130) + self.large_modifies(uri, 'Z', ds, 7, 1, nrows, 140) + + # Verify data is visible and correct. + self.check(value_a, uri, nrows, 20) + self.check(value_modQ, uri, nrows, 30) + self.check(value_modR, uri, nrows, 40) + self.check(value_modS, uri, nrows, 50) + self.check(value_b, uri, nrows, 60) + self.check(value_c, uri, nrows, 70) + self.check(value_modT, uri, nrows, 80) + self.check(value_d, uri, nrows, 90) + self.check(value_modW, uri, nrows, 100) + self.check(value_a, uri, nrows, 110) + self.check(value_modX, uri, nrows, 120) + self.check(value_modY, uri, nrows, 130) + self.check(value_modZ, uri, nrows, 140) + + # Pin stable to timestamp 40 if prepare otherwise 30. + if self.prepare: + self.conn.set_timestamp('stable_timestamp=' + timestamp_str(40)) + else: + self.conn.set_timestamp('stable_timestamp=' + timestamp_str(30)) + + # Checkpoint to ensure the data is flushed, then rollback to the stable timestamp. + if not self.in_memory: + self.session.checkpoint() + self.conn.rollback_to_stable() + + # Check that the correct data is seen at and after the stable timestamp. + self.check(value_modQ, uri, nrows, 30) + self.check(value_modQ, uri, nrows, 150) + self.check(value_a, uri, nrows, 20) + + stat_cursor = self.session.open_cursor('statistics:', None, None) + calls = stat_cursor[stat.conn.txn_rts][2] + hs_removed = stat_cursor[stat.conn.txn_rts_hs_removed][2] + keys_removed = stat_cursor[stat.conn.txn_rts_keys_removed][2] + keys_restored = stat_cursor[stat.conn.txn_rts_keys_restored][2] + pages_visited = stat_cursor[stat.conn.txn_rts_pages_visited][2] + upd_aborted = stat_cursor[stat.conn.txn_rts_upd_aborted][2] + stat_cursor.close() + + self.assertEqual(calls, 1) + self.assertEqual(keys_removed, 0) + self.assertEqual(keys_restored, 0) + self.assertGreater(pages_visited, 0) + if self.in_memory or self.prepare: + self.assertGreaterEqual(upd_aborted, nrows * 11) + self.assertGreaterEqual(hs_removed, 0) + else: + self.assertGreaterEqual(upd_aborted, 0) + self.assertGreaterEqual(hs_removed, nrows * 11) + +if __name__ == '__main__': + wttest.run() diff --git a/src/third_party/wiredtiger/test/suite/test_rollback_to_stable05.py b/src/third_party/wiredtiger/test/suite/test_rollback_to_stable05.py new file mode 100755 index 00000000000..01733521fab --- /dev/null +++ b/src/third_party/wiredtiger/test/suite/test_rollback_to_stable05.py @@ -0,0 +1,161 @@ +#!/usr/bin/env python +# +# Public Domain 2014-2020 MongoDB, Inc. +# Public Domain 2008-2014 WiredTiger, Inc. +# +# This is free and unencumbered software released into the public domain. +# +# Anyone is free to copy, modify, publish, use, compile, sell, or +# distribute this software, either in source code form or as a compiled +# binary, for any purpose, commercial or non-commercial, and by any +# means. +# +# In jurisdictions that recognize copyright laws, the author or authors +# of this software dedicate any and all copyright interest in the +# software to the public domain. We make this dedication for the benefit +# of the public at large and to the detriment of our heirs and +# successors. We intend this dedication to be an overt act of +# relinquishment in perpetuity of all present and future rights to this +# software under copyright law. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +# IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. + +import time +from helper import copy_wiredtiger_home +import unittest, wiredtiger, wttest +from wtdataset import SimpleDataSet +from wiredtiger import stat +from wtscenario import make_scenarios +from test_rollback_to_stable01 import test_rollback_to_stable_base + +def timestamp_str(t): + return '%x' % t + +# test_rollback_to_stable05.py +# Test that rollback to stable cleans history store for non-timestamp tables. +class test_rollback_to_stable05(test_rollback_to_stable_base): + session_config = 'isolation=snapshot' + + in_memory_values = [ + ('no_inmem', dict(in_memory=False)), + ('inmem', dict(in_memory=True)) + ] + + prepare_values = [ + ('no_prepare', dict(prepare=False)), + ('prepare', dict(prepare=True)) + ] + + scenarios = make_scenarios(in_memory_values, prepare_values) + + def conn_config(self): + config = '' + # Temporarily solution to have good cache size until prepare updates are written to disk. + if self.prepare: + config += 'cache_size=250MB,statistics=(all)' + else: + config += 'cache_size=50MB,statistics=(all)' + if self.in_memory: + config += ',in_memory=true' + else: + config += ',log=(enabled),in_memory=false' + return config + + def test_rollback_to_stable(self): + nrows = 1000 + + # Create two tables without logging. + uri_1 = "table:rollback_to_stable05_1" + ds_1 = SimpleDataSet( + self, uri_1, 0, key_format="i", value_format="S", config='log=(enabled=false)') + ds_1.populate() + + uri_2 = "table:rollback_to_stable05_2" + ds_2 = SimpleDataSet( + self, uri_2, 0, key_format="i", value_format="S", config='log=(enabled=false)') + ds_2.populate() + + # Pin oldest and stable to timestamp 1. + self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(1) + + ',stable_timestamp=' + timestamp_str(1)) + + valuea = "aaaaa" * 100 + valueb = "bbbbb" * 100 + valuec = "ccccc" * 100 + valued = "ddddd" * 100 + self.large_updates(uri_1, valuea, ds_1, nrows, 0) + self.check(valuea, uri_1, nrows, 0) + + self.large_updates(uri_2, valuea, ds_2, nrows, 0) + self.check(valuea, uri_2, nrows, 0) + + # Start a long running transaction and keep it open. + session_2 = self.conn.open_session() + session_2.begin_transaction('isolation=snapshot') + + self.large_updates(uri_1, valueb, ds_1, nrows, 0) + self.check(valueb, uri_1, nrows, 0) + + self.large_updates(uri_1, valuec, ds_1, nrows, 0) + self.check(valuec, uri_1, nrows, 0) + + self.large_updates(uri_1, valued, ds_1, nrows, 0) + self.check(valued, uri_1, nrows, 0) + + # Add updates to the another table. + self.large_updates(uri_2, valueb, ds_2, nrows, 0) + self.check(valueb, uri_2, nrows, 0) + + self.large_updates(uri_2, valuec, ds_2, nrows, 0) + self.check(valuec, uri_2, nrows, 0) + + self.large_updates(uri_2, valued, ds_2, nrows, 0) + self.check(valued, uri_2, nrows, 0) + + # Pin stable to timestamp 20 if prepare otherwise 10. + if self.prepare: + self.conn.set_timestamp('stable_timestamp=' + timestamp_str(20)) + else: + self.conn.set_timestamp('stable_timestamp=' + timestamp_str(10)) + + # Checkpoint to ensure that all the data is flushed. + if not self.in_memory: + self.session.checkpoint() + + # Clear all running transactions before rollback to stable. + session_2.commit_transaction() + session_2.close() + + self.conn.rollback_to_stable() + self.check(valued, uri_1, nrows, 0) + self.check(valued, uri_2, nrows, 0) + + stat_cursor = self.session.open_cursor('statistics:', None, None) + calls = stat_cursor[stat.conn.txn_rts][2] + upd_aborted = stat_cursor[stat.conn.txn_rts_upd_aborted][2] + hs_removed = stat_cursor[stat.conn.txn_rts_hs_removed][2] + keys_removed = stat_cursor[stat.conn.txn_rts_keys_removed][2] + keys_restored = stat_cursor[stat.conn.txn_rts_keys_restored][2] + pages_visited = stat_cursor[stat.conn.txn_rts_pages_visited][2] + stat_cursor.close() + + self.assertEqual(calls, 1) + self.assertEqual(keys_removed, 0) + self.assertEqual(keys_restored, 0) + if self.in_memory: + self.assertGreaterEqual(pages_visited, 0) + self.assertEqual(upd_aborted, 0) + self.assertEqual(hs_removed, 0) + else: + self.assertEqual(pages_visited, 0) + self.assertEqual(upd_aborted, 0) + self.assertEqual(hs_removed, nrows * 3 * 2) + +if __name__ == '__main__': + wttest.run() diff --git a/src/third_party/wiredtiger/test/suite/test_rollback_to_stable06.py b/src/third_party/wiredtiger/test/suite/test_rollback_to_stable06.py new file mode 100755 index 00000000000..fd2d2871522 --- /dev/null +++ b/src/third_party/wiredtiger/test/suite/test_rollback_to_stable06.py @@ -0,0 +1,131 @@ +#!/usr/bin/env python +# +# Public Domain 2014-2020 MongoDB, Inc. +# Public Domain 2008-2014 WiredTiger, Inc. +# +# This is free and unencumbered software released into the public domain. +# +# Anyone is free to copy, modify, publish, use, compile, sell, or +# distribute this software, either in source code form or as a compiled +# binary, for any purpose, commercial or non-commercial, and by any +# means. +# +# In jurisdictions that recognize copyright laws, the author or authors +# of this software dedicate any and all copyright interest in the +# software to the public domain. We make this dedication for the benefit +# of the public at large and to the detriment of our heirs and +# successors. We intend this dedication to be an overt act of +# relinquishment in perpetuity of all present and future rights to this +# software under copyright law. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +# IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. + +from wiredtiger import stat +from wtdataset import SimpleDataSet +from wtscenario import make_scenarios +from test_rollback_to_stable01 import test_rollback_to_stable_base + +def timestamp_str(t): + return '%x' % t + +# test_rollback_to_stable06.py +# Test that rollback to stable removes all keys when the stable timestamp is earlier than +# all commit timestamps. +class test_rollback_to_stable06(test_rollback_to_stable_base): + session_config = 'isolation=snapshot' + + in_memory_values = [ + ('no_inmem', dict(in_memory=False)), + ('inmem', dict(in_memory=True)) + ] + + prepare_values = [ + ('no_prepare', dict(prepare=False)), + ('prepare', dict(prepare=True)) + ] + + scenarios = make_scenarios(in_memory_values, prepare_values) + + def conn_config(self): + config = '' + # Temporarily solution to have good cache size until prepare updates are written to disk. + if self.prepare: + config += 'cache_size=250MB,statistics=(all)' + else: + config += 'cache_size=50MB,statistics=(all)' + if self.in_memory: + config += ',in_memory=true' + else: + config += ',log=(enabled),in_memory=false' + return config + + def test_rollback_to_stable(self): + nrows = 1000 + + # Create a table without logging. + uri = "table:rollback_to_stable06" + ds = SimpleDataSet( + self, uri, 0, key_format="i", value_format="S", config='log=(enabled=false)') + ds.populate() + + # Pin oldest and stable to timestamp 10. + self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(10) + + ',stable_timestamp=' + timestamp_str(10)) + + value_a = "aaaaa" * 100 + value_b = "bbbbb" * 100 + value_c = "ccccc" * 100 + value_d = "ddddd" * 100 + + # Perform several updates. + self.large_updates(uri, value_a, ds, nrows, 20) + self.large_updates(uri, value_b, ds, nrows, 30) + self.large_updates(uri, value_c, ds, nrows, 40) + self.large_updates(uri, value_d, ds, nrows, 50) + + # Verify data is visible and correct. + self.check(value_a, uri, nrows, 20) + self.check(value_b, uri, nrows, 30) + self.check(value_c, uri, nrows, 40) + self.check(value_d, uri, nrows, 50) + + # Checkpoint to ensure the data is flushed, then rollback to the stable timestamp. + if not self.in_memory: + self.session.checkpoint() + self.conn.rollback_to_stable() + + # Check that all keys are removed. + self.check(value_a, uri, 0, 20) + self.check(value_b, uri, 0, 30) + self.check(value_c, uri, 0, 40) + self.check(value_d, uri, 0, 50) + + stat_cursor = self.session.open_cursor('statistics:', None, None) + calls = stat_cursor[stat.conn.txn_rts][2] + hs_removed = stat_cursor[stat.conn.txn_rts_hs_removed][2] + keys_removed = stat_cursor[stat.conn.txn_rts_keys_removed][2] + keys_restored = stat_cursor[stat.conn.txn_rts_keys_restored][2] + pages_visited = stat_cursor[stat.conn.txn_rts_pages_visited][2] + upd_aborted = stat_cursor[stat.conn.txn_rts_upd_aborted][2] + stat_cursor.close() + + self.assertEqual(calls, 1) + self.assertEqual(keys_restored, 0) + self.assertGreater(pages_visited, 0) + if self.in_memory or self.prepare: + self.assertEqual(keys_removed, 0) + self.assertEqual(upd_aborted, nrows * 4) + self.assertEqual(hs_removed, 0) + else: + self.assertGreaterEqual(keys_removed, 0) + self.assertGreaterEqual(upd_aborted, 0) + self.assertGreaterEqual(hs_removed, nrows * 3) + +if __name__ == '__main__': + wttest.run() diff --git a/src/third_party/wiredtiger/test/suite/test_rollback_to_stable07.py b/src/third_party/wiredtiger/test/suite/test_rollback_to_stable07.py new file mode 100755 index 00000000000..91f283391bf --- /dev/null +++ b/src/third_party/wiredtiger/test/suite/test_rollback_to_stable07.py @@ -0,0 +1,191 @@ +#!/usr/bin/env python +# +# Public Domain 2014-2020 MongoDB, Inc. +# Public Domain 2008-2014 WiredTiger, Inc. +# +# This is free and unencumbered software released into the public domain. +# +# Anyone is free to copy, modify, publish, use, compile, sell, or +# distribute this software, either in source code form or as a compiled +# binary, for any purpose, commercial or non-commercial, and by any +# means. +# +# In jurisdictions that recognize copyright laws, the author or authors +# of this software dedicate any and all copyright interest in the +# software to the public domain. We make this dedication for the benefit +# of the public at large and to the detriment of our heirs and +# successors. We intend this dedication to be an overt act of +# relinquishment in perpetuity of all present and future rights to this +# software under copyright law. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +# IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. + +import fnmatch, os, shutil, time +from helper import copy_wiredtiger_home +from test_rollback_to_stable01 import test_rollback_to_stable_base +from wiredtiger import stat +from wtdataset import SimpleDataSet +from wtscenario import make_scenarios + +def timestamp_str(t): + return '%x' % t + +# test_rollback_to_stable07.py +# Test the rollback to stable operation performs as expected following a server crash and +# recovery. Verify that +# (a) the on-disk value is replaced by the correct value from the history store, and +# (b) newer updates are removed. +class test_rollback_to_stable07(test_rollback_to_stable_base): + session_config = 'isolation=snapshot' + + prepare_values = [ + ('no_prepare', dict(prepare=False)), + ('prepare', dict(prepare=True)) + ] + + scenarios = make_scenarios(prepare_values) + + def conn_config(self): + config = '' + # Temporarily solution to have good cache size until prepare updates are written to disk. + if self.prepare: + config += 'cache_size=250MB,statistics=(all),log=(enabled=true)' + else: + config += 'cache_size=50MB,statistics=(all),log=(enabled=true)' + return config + + def simulate_crash_restart(self, uri, olddir, newdir): + ''' Simulate a crash from olddir and restart in newdir. ''' + # with the connection still open, copy files to new directory + shutil.rmtree(newdir, ignore_errors=True) + os.mkdir(newdir) + for fname in os.listdir(olddir): + fullname = os.path.join(olddir, fname) + # Skip lock file on Windows since it is locked + if os.path.isfile(fullname) and \ + "WiredTiger.lock" not in fullname and \ + "Tmplog" not in fullname and \ + "Preplog" not in fullname: + shutil.copy(fullname, newdir) + # + # close the original connection and open to new directory + # NOTE: This really cannot test the difference between the + # write-no-sync (off) version of log_flush and the sync + # version since we're not crashing the system itself. + # + self.close_conn() + self.conn = self.setUpConnectionOpen(newdir) + self.session = self.setUpSessionOpen(self.conn) + + def test_rollback_to_stable(self): + nrows = 1000 + + # Create a table without logging. + uri = "table:rollback_to_stable07" + ds = SimpleDataSet( + self, uri, 0, key_format="i", value_format="S", config='log=(enabled=false)') + ds.populate() + + # Pin oldest and stable to timestamp 10. + self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(10) + + ',stable_timestamp=' + timestamp_str(10)) + + value_a = "aaaaa" * 100 + value_b = "bbbbb" * 100 + value_c = "ccccc" * 100 + value_d = "ddddd" * 100 + + # Perform several updates. + self.large_updates(uri, value_d, ds, nrows, 20) + self.large_updates(uri, value_c, ds, nrows, 30) + self.large_updates(uri, value_b, ds, nrows, 40) + self.large_updates(uri, value_a, ds, nrows, 50) + + # Verify data is visible and correct. + self.check(value_d, uri, nrows, 20) + self.check(value_c, uri, nrows, 30) + self.check(value_b, uri, nrows, 40) + self.check(value_a, uri, nrows, 50) + + # Pin stable to timestamp 50 if prepare otherwise 40. + if self.prepare: + self.conn.set_timestamp('stable_timestamp=' + timestamp_str(50)) + else: + self.conn.set_timestamp('stable_timestamp=' + timestamp_str(40)) + + # Perform additional updates. + self.large_updates(uri, value_b, ds, nrows, 60) + self.large_updates(uri, value_c, ds, nrows, 70) + self.large_updates(uri, value_d, ds, nrows, 80) + + # Checkpoint to ensure the data is flushed to disk. + self.session.checkpoint() + + # Verify additional update data is visible and correct. + self.check(value_b, uri, nrows, 60) + self.check(value_c, uri, nrows, 70) + self.check(value_d, uri, nrows, 80) + + # Simulate a server crash and restart. + self.simulate_crash_restart(uri, ".", "RESTART") + + # Check that the correct data is seen at and after the stable timestamp. + self.check(value_b, uri, nrows, 40) + self.check(value_b, uri, nrows, 80) + self.check(value_c, uri, nrows, 30) + self.check(value_d, uri, nrows, 20) + + stat_cursor = self.session.open_cursor('statistics:', None, None) + calls = stat_cursor[stat.conn.txn_rts][2] + hs_removed = stat_cursor[stat.conn.txn_rts_hs_removed][2] + keys_removed = stat_cursor[stat.conn.txn_rts_keys_removed][2] + keys_restored = stat_cursor[stat.conn.txn_rts_keys_restored][2] + pages_visited = stat_cursor[stat.conn.txn_rts_pages_visited][2] + upd_aborted = stat_cursor[stat.conn.txn_rts_upd_aborted][2] + stat_cursor.close() + + self.assertEqual(calls, 0) + self.assertEqual(keys_removed, 0) + self.assertEqual(keys_restored, 0) + self.assertGreaterEqual(upd_aborted, 0) + if self.prepare: + # pages_visisted needs a fix once we started writing prepared updates to disk. + self.assertEqual(pages_visited, 0) + self.assertGreaterEqual(hs_removed, 0) + else: + self.assertGreater(pages_visited, 0) + self.assertGreaterEqual(hs_removed, nrows * 4) + + # Simulate another server crash and restart. + self.simulate_crash_restart(uri, "RESTART", "RESTART2") + + # Check that the correct data is seen at and after the stable timestamp. + self.check(value_b, uri, nrows, 40) + self.check(value_b, uri, nrows, 80) + self.check(value_c, uri, nrows, 30) + self.check(value_d, uri, nrows, 20) + + stat_cursor = self.session.open_cursor('statistics:', None, None) + calls = stat_cursor[stat.conn.txn_rts][2] + hs_removed = stat_cursor[stat.conn.txn_rts_hs_removed][2] + keys_removed = stat_cursor[stat.conn.txn_rts_keys_removed][2] + keys_restored = stat_cursor[stat.conn.txn_rts_keys_restored][2] + pages_visited = stat_cursor[stat.conn.txn_rts_pages_visited][2] + upd_aborted = stat_cursor[stat.conn.txn_rts_upd_aborted][2] + stat_cursor.close() + + self.assertEqual(calls, 0) + self.assertEqual(keys_removed, 0) + self.assertEqual(keys_restored, 0) + self.assertGreaterEqual(pages_visited, 0) + self.assertEqual(upd_aborted, 0) + self.assertEqual(hs_removed, 0) + +if __name__ == '__main__': + wttest.run() diff --git a/src/third_party/wiredtiger/test/suite/test_rollback_to_stable08.py b/src/third_party/wiredtiger/test/suite/test_rollback_to_stable08.py new file mode 100755 index 00000000000..5c59037e8c0 --- /dev/null +++ b/src/third_party/wiredtiger/test/suite/test_rollback_to_stable08.py @@ -0,0 +1,135 @@ +#!/usr/bin/env python +# +# Public Domain 2014-2020 MongoDB, Inc. +# Public Domain 2008-2014 WiredTiger, Inc. +# +# This is free and unencumbered software released into the public domain. +# +# Anyone is free to copy, modify, publish, use, compile, sell, or +# distribute this software, either in source code form or as a compiled +# binary, for any purpose, commercial or non-commercial, and by any +# means. +# +# In jurisdictions that recognize copyright laws, the author or authors +# of this software dedicate any and all copyright interest in the +# software to the public domain. We make this dedication for the benefit +# of the public at large and to the detriment of our heirs and +# successors. We intend this dedication to be an overt act of +# relinquishment in perpetuity of all present and future rights to this +# software under copyright law. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +# IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. + +from wiredtiger import stat +from wtdataset import SimpleDataSet +from wtscenario import make_scenarios +from test_rollback_to_stable01 import test_rollback_to_stable_base + +def timestamp_str(t): + return '%x' % t + +# test_rollback_to_stable08.py +# Test that rollback to stable does not abort updates when the stable timestamp is +# set to the latest commit. +class test_rollback_to_stable08(test_rollback_to_stable_base): + session_config = 'isolation=snapshot' + + in_memory_values = [ + ('no_inmem', dict(in_memory=False)), + ('inmem', dict(in_memory=True)) + ] + + prepare_values = [ + ('no_prepare', dict(prepare=False)), + ('prepare', dict(prepare=True)) + ] + + scenarios = make_scenarios(in_memory_values, prepare_values) + + def conn_config(self): + config = '' + # Temporarily solution to have good cache size until prepare updates are written to disk. + if self.prepare: + config += 'cache_size=250MB,statistics=(all)' + else: + config += 'cache_size=50MB,statistics=(all)' + if self.in_memory: + config += ',in_memory=true' + else: + config += ',log=(enabled),in_memory=false' + return config + + def test_rollback_to_stable(self): + nrows = 10000 + + # Create a table without logging. + uri = "table:rollback_to_stable08" + ds = SimpleDataSet( + self, uri, 0, key_format="i", value_format="S", config='log=(enabled=false)') + ds.populate() + + # Pin oldest and stable to timestamp 10. + self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(10) + + ',stable_timestamp=' + timestamp_str(10)) + + value_a = "aaaaa" * 100 + value_b = "bbbbb" * 100 + value_c = "ccccc" * 100 + value_d = "ddddd" * 100 + + # Perform several updates. + self.large_updates(uri, value_a, ds, nrows, 20) + self.large_updates(uri, value_b, ds, nrows, 30) + self.large_updates(uri, value_c, ds, nrows, 40) + self.large_updates(uri, value_d, ds, nrows, 50) + + # Verify data is visible and correct. + self.check(value_a, uri, nrows, 20) + self.check(value_b, uri, nrows, 30) + self.check(value_c, uri, nrows, 40) + self.check(value_d, uri, nrows, 50) + + # Pin stable to timestamp 60 if prepare otherwise 50. + if self.prepare: + self.conn.set_timestamp('stable_timestamp=' + timestamp_str(60)) + else: + self.conn.set_timestamp('stable_timestamp=' + timestamp_str(50)) + + # Checkpoint to ensure the data is flushed, then rollback to the stable timestamp. + if not self.in_memory: + self.session.checkpoint() + self.conn.rollback_to_stable() + + # Check that the correct data is seen. + self.check(value_a, uri, nrows, 20) + self.check(value_b, uri, nrows, 30) + self.check(value_c, uri, nrows, 40) + self.check(value_d, uri, nrows, 50) + + stat_cursor = self.session.open_cursor('statistics:', None, None) + calls = stat_cursor[stat.conn.txn_rts][2] + hs_removed = stat_cursor[stat.conn.txn_rts_hs_removed][2] + keys_removed = stat_cursor[stat.conn.txn_rts_keys_removed][2] + keys_restored = stat_cursor[stat.conn.txn_rts_keys_restored][2] + pages_visited = stat_cursor[stat.conn.txn_rts_pages_visited][2] + upd_aborted = stat_cursor[stat.conn.txn_rts_upd_aborted][2] + stat_cursor.close() + + self.assertEqual(calls, 1) + self.assertEqual(hs_removed, 0) + self.assertEqual(upd_aborted, 0) + self.assertEqual(keys_removed, 0) + self.assertEqual(keys_restored, 0) + if self.in_memory: + self.assertGreater(pages_visited, 0) + else: + self.assertEqual(pages_visited, 0) + +if __name__ == '__main__': + wttest.run() diff --git a/src/third_party/wiredtiger/test/suite/test_rollback_to_stable09.py b/src/third_party/wiredtiger/test/suite/test_rollback_to_stable09.py new file mode 100755 index 00000000000..31b9938b604 --- /dev/null +++ b/src/third_party/wiredtiger/test/suite/test_rollback_to_stable09.py @@ -0,0 +1,155 @@ +#!/usr/bin/env python +# +# Public Domain 2014-2020 MongoDB, Inc. +# Public Domain 2008-2014 WiredTiger, Inc. +# +# This is free and unencumbered software released into the public domain. +# +# Anyone is free to copy, modify, publish, use, compile, sell, or +# distribute this software, either in source code form or as a compiled +# binary, for any purpose, commercial or non-commercial, and by any +# means. +# +# In jurisdictions that recognize copyright laws, the author or authors +# of this software dedicate any and all copyright interest in the +# software to the public domain. We make this dedication for the benefit +# of the public at large and to the detriment of our heirs and +# successors. We intend this dedication to be an overt act of +# relinquishment in perpetuity of all present and future rights to this +# software under copyright law. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +# IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. + +import os +import wiredtiger +from wtdataset import SimpleDataSet +from wtscenario import make_scenarios +from test_rollback_to_stable01 import test_rollback_to_stable_base + +def timestamp_str(t): + return '%x' % t + +# test_rollback_to_stable09.py +# Test that rollback to stable does not abort schema operations that are done +# as they don't have transaction support +class test_rollback_to_stable09(test_rollback_to_stable_base): + session_config = 'isolation=snapshot' + + in_memory_values = [ + ('no_inmem', dict(in_memory=False)), + ('inmem', dict(in_memory=True)) + ] + + prepare_values = [ + ('no_prepare', dict(prepare=False)), + ('prepare', dict(prepare=True)) + ] + + tablename = "test_rollback_stable09" + uri = "table:" + tablename + index_uri = "index:test_rollback_stable09:country" + + scenarios = make_scenarios(in_memory_values, prepare_values) + + def conn_config(self): + config = 'cache_size=250MB,statistics=(all)' + if self.in_memory: + config += ',in_memory=true' + else: + config += ',log=(enabled),in_memory=false' + return config + + def create_table(self, commit_ts): + self.pr('create table') + session = self.session + session.begin_transaction() + session.create(self.uri, 'key_format=5s,value_format=HQ,' + + 'columns=(country,year,population),' ) + if commit_ts == 0: + session.commit_transaction() + elif self.prepare: + session.prepare_transaction('prepare_timestamp=' + timestamp_str(commit_ts-1)) + session.timestamp_transaction('commit_timestamp=' + timestamp_str(commit_ts)) + session.timestamp_transaction('durable_timestamp=' + timestamp_str(commit_ts+1)) + session.commit_transaction() + else: + session.commit_transaction('commit_timestamp=' + timestamp_str(commit_ts)) + + def drop_table(self, commit_ts): + self.pr('drop table') + session = self.session + session.begin_transaction() + session.drop(self.uri) + if commit_ts == 0: + session.commit_transaction() + elif self.prepare: + session.prepare_transaction('prepare_timestamp=' + timestamp_str(commit_ts-1)) + session.timestamp_transaction('commit_timestamp=' + timestamp_str(commit_ts)) + session.timestamp_transaction('durable_timestamp=' + timestamp_str(commit_ts+1)) + session.commit_transaction() + else: + session.commit_transaction('commit_timestamp=' + timestamp_str(commit_ts)) + + def create_index(self, commit_ts): + session = self.session + session.begin_transaction() + self.session.create(self.index_uri, "key_format=s,columns=(country)") + if commit_ts == 0: + session.commit_transaction() + elif self.prepare: + session.prepare_transaction('prepare_timestamp=' + timestamp_str(commit_ts-1)) + session.timestamp_transaction('commit_timestamp=' + timestamp_str(commit_ts)) + session.timestamp_transaction('durable_timestamp=' + timestamp_str(commit_ts+1)) + session.commit_transaction() + else: + session.commit_transaction('commit_timestamp=' + timestamp_str(commit_ts)) + + def test_rollback_to_stable(self): + # Pin oldest and stable to timestamp 10. + self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(10) + + ',stable_timestamp=' + timestamp_str(10)) + + # Create table and index at a later timestamp + self.create_table(20) + self.create_index(30) + + #perform rollback to stable, still the table and index must exist + self.conn.rollback_to_stable() + + if not self.in_memory: + self.assertTrue(os.path.exists(self.tablename + ".wt")) + self.assertTrue(os.path.exists(self.tablename + "_country.wti")) + + # Check that we are able to open cursor successfully on the table and index + c = self.session.open_cursor(self.uri, None, None) + self.assertTrue(c.next(), wiredtiger.WT_NOTFOUND) + self.assertEqual(c.close(), 0) + + c = self.session.open_cursor(self.index_uri, None, None) + self.assertTrue(c.next(), wiredtiger.WT_NOTFOUND) + self.assertEqual(c.close(), 0) + + # Drop the table + self.drop_table(40) + + #perform rollback to stable, the table and index must not exist + self.conn.rollback_to_stable() + + if not self.in_memory: + self.assertFalse(os.path.exists(self.tablename + ".wt")) + self.assertFalse(os.path.exists(self.tablename + "_country.wti")) + + # Check that we are unable to open cursor on the table and index + self.assertRaises(wiredtiger.WiredTigerError, lambda: + self.session.open_cursor(self.uri, None, None)) + self.assertRaises(wiredtiger.WiredTigerError, lambda: + self.session.open_cursor(self.index_uri, None, None)) + +if __name__ == '__main__': + wttest.run() diff --git a/src/third_party/wiredtiger/test/suite/test_schema08.py b/src/third_party/wiredtiger/test/suite/test_schema08.py index 91925ffc975..9a20b0bee6d 100644 --- a/src/third_party/wiredtiger/test/suite/test_schema08.py +++ b/src/third_party/wiredtiger/test/suite/test_schema08.py @@ -28,7 +28,7 @@ import fnmatch, os, shutil, sys from suite_subprocess import suite_subprocess -import wiredtiger, wttest +import unittest, wiredtiger, wttest from wtscenario import make_scenarios # test_schema08.py diff --git a/src/third_party/wiredtiger/test/suite/test_stat04.py b/src/third_party/wiredtiger/test/suite/test_stat04.py index a3d0d0eae65..b86e0187dd8 100644 --- a/src/third_party/wiredtiger/test/suite/test_stat04.py +++ b/src/third_party/wiredtiger/test/suite/test_stat04.py @@ -26,7 +26,7 @@ # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. -import os, struct +import os, struct, unittest from suite_subprocess import suite_subprocess from wtscenario import make_scenarios import wiredtiger, wttest diff --git a/src/third_party/wiredtiger/test/suite/test_sweep01.py b/src/third_party/wiredtiger/test/suite/test_sweep01.py index ce374d98db0..fa1b98f90c2 100644 --- a/src/third_party/wiredtiger/test/suite/test_sweep01.py +++ b/src/third_party/wiredtiger/test/suite/test_sweep01.py @@ -191,7 +191,7 @@ class test_sweep01(wttest.WiredTigerTestCase, suite_subprocess): print("ref1: " + str(ref1) + " ref2: " + str(ref2)) print("XX: nfile1: " + str(nfile1) + " nfile2: " + str(nfile2)) self.assertEqual(nfile2 < nfile1, True) - # The only files that should be left are the metadata, the lookaside + # The only files that should be left are the metadata, the history store # file, the lock file, and the active file. if (nfile2 != final_nfile): print("close1: " + str(close1) + " close2: " + str(close2)) diff --git a/src/third_party/wiredtiger/test/suite/test_timestamp03.py b/src/third_party/wiredtiger/test/suite/test_timestamp03.py index a2683d34e1a..767bfcc888f 100755 --- a/src/third_party/wiredtiger/test/suite/test_timestamp03.py +++ b/src/third_party/wiredtiger/test/suite/test_timestamp03.py @@ -33,7 +33,7 @@ from helper import copy_wiredtiger_home import random from suite_subprocess import suite_subprocess -import wiredtiger, wttest +import unittest, wiredtiger, wttest from wtscenario import make_scenarios def timestamp_str(t): @@ -325,6 +325,26 @@ class test_timestamp03(wttest.WiredTigerTestCase, suite_subprocess): if self.ckpt_ts == False: valcnt_ts_log = nkeys + # Take a checkpoint using the given configuration. Then verify + # whether value2 appears in a copy of that data or not. + valcnt_ts_log = valcnt_nots_log = valcnt_nots_nolog = nkeys + if self.ckpt_ts == False: + # if use_timestamp is false, then all updates will be checkpointed. + valcnt_ts_nolog = nkeys + else: + # Checkpoint will happen with stable_timestamp=100. + if self.using_log == True: + # only table_ts_nolog will have old values when logging is enabled + self.ckpt_backup(self.value, 0, nkeys, 0, 0) + else: + # Both table_ts_nolog and table_ts_log will have old values when + # logging is disabled. + self.ckpt_backup(self.value, nkeys, nkeys, 0, 0) + # table_ts_nolog will not have any new values (i.e. value2) + valcnt_ts_nolog = 0 + + if self.ckpt_ts == False: + valcnt_ts_log = nkeys else: # When log is enabled, table_ts_log will have all new values, else # none. diff --git a/src/third_party/wiredtiger/test/suite/test_timestamp04.py b/src/third_party/wiredtiger/test/suite/test_timestamp04.py index 0e154318014..d5dffd6a841 100644 --- a/src/third_party/wiredtiger/test/suite/test_timestamp04.py +++ b/src/third_party/wiredtiger/test/suite/test_timestamp04.py @@ -31,7 +31,7 @@ # from suite_subprocess import suite_subprocess -import wiredtiger, wttest +import unittest, wiredtiger, wttest from wiredtiger import stat from wtscenario import make_scenarios @@ -53,8 +53,9 @@ class test_timestamp04(wttest.WiredTigerTestCase, suite_subprocess): # Minimum cache_size requirement of lsm is 31MB. types = [ - ('col_fix', dict(empty=1, cacheSize='cache_size=20MB', extra_config=',key_format=r,value_format=8t')), - ('col_var', dict(empty=0, cacheSize='cache_size=20MB', extra_config=',key_format=r')), + # The commented columnar tests needs to be enabled once rollback to stable for columnar is fixed in (WT-5548). + # ('col_fix', dict(empty=1, cacheSize='cache_size=20MB', extra_config=',key_format=r,value_format=8t')), + # ('col_var', dict(empty=0, cacheSize='cache_size=20MB', extra_config=',key_format=r')), ('lsm', dict(empty=0, cacheSize='cache_size=31MB', extra_config=',type=lsm')), ('row', dict(empty=0, cacheSize='cache_size=20MB', extra_config='',)), ('row-smallcache', dict(empty=0, cacheSize='cache_size=2MB', extra_config='',)), @@ -169,9 +170,10 @@ class test_timestamp04(wttest.WiredTigerTestCase, suite_subprocess): self.conn.rollback_to_stable() stat_cursor = self.session.open_cursor('statistics:', None, None) - calls = stat_cursor[stat.conn.txn_rollback_to_stable][2] - upd_aborted = (stat_cursor[stat.conn.txn_rollback_upd_aborted][2] + - stat_cursor[stat.conn.txn_rollback_las_removed][2]) + calls = stat_cursor[stat.conn.txn_rts][2] + upd_aborted = (stat_cursor[stat.conn.txn_rts_upd_aborted][2] + + stat_cursor[stat.conn.txn_rts_hs_removed][2] + + stat_cursor[stat.conn.txn_rts_keys_removed][2]) stat_cursor.close() self.assertEqual(calls, 1) self.assertTrue(upd_aborted >= key_range/2) @@ -240,11 +242,13 @@ class test_timestamp04(wttest.WiredTigerTestCase, suite_subprocess): self.conn.set_timestamp('stable_timestamp=' + stable_ts) self.conn.rollback_to_stable() stat_cursor = self.session.open_cursor('statistics:', None, None) - calls = stat_cursor[stat.conn.txn_rollback_to_stable][2] - upd_aborted = (stat_cursor[stat.conn.txn_rollback_upd_aborted][2] + - stat_cursor[stat.conn.txn_rollback_las_removed][2]) + calls = stat_cursor[stat.conn.txn_rts][2] + upd_aborted = (stat_cursor[stat.conn.txn_rts_upd_aborted][2] + + stat_cursor[stat.conn.txn_rts_hs_removed][2] + + stat_cursor[stat.conn.txn_rts_keys_removed][2]) stat_cursor.close() self.assertEqual(calls, 2) + # # We rolled back half on the earlier call and now three-quarters on # this call, which is one and one quarter of all keys rolled back. diff --git a/src/third_party/wiredtiger/test/suite/test_timestamp06.py b/src/third_party/wiredtiger/test/suite/test_timestamp06.py index 26bbda36ad2..9129d2ddf24 100644 --- a/src/third_party/wiredtiger/test/suite/test_timestamp06.py +++ b/src/third_party/wiredtiger/test/suite/test_timestamp06.py @@ -33,7 +33,7 @@ from helper import copy_wiredtiger_home import random from suite_subprocess import suite_subprocess -import wiredtiger, wttest +import unittest, wiredtiger, wttest from wtscenario import make_scenarios def timestamp_str(t): diff --git a/src/third_party/wiredtiger/test/suite/test_timestamp07.py b/src/third_party/wiredtiger/test/suite/test_timestamp07.py index 6a5db08c73a..2b56e85b3ea 100755 --- a/src/third_party/wiredtiger/test/suite/test_timestamp07.py +++ b/src/third_party/wiredtiger/test/suite/test_timestamp07.py @@ -33,7 +33,7 @@ from helper import copy_wiredtiger_home import random from suite_subprocess import suite_subprocess -import wiredtiger, wttest +import unittest, wiredtiger, wttest from wtscenario import make_scenarios def timestamp_str(t): diff --git a/src/third_party/wiredtiger/test/suite/test_timestamp10.py b/src/third_party/wiredtiger/test/suite/test_timestamp10.py index 7f4e70d5013..46d42003885 100644 --- a/src/third_party/wiredtiger/test/suite/test_timestamp10.py +++ b/src/third_party/wiredtiger/test/suite/test_timestamp10.py @@ -31,7 +31,7 @@ # from suite_subprocess import suite_subprocess -import wiredtiger, wttest +import unittest, wiredtiger, wttest from wtscenario import make_scenarios def timestamp_str(t): @@ -119,7 +119,7 @@ class test_timestamp10(wttest.WiredTigerTestCase, suite_subprocess): # Run the wt command some number of times to get some runs in that do # not use timestamps. Make sure the recovery checkpoint is maintained. for i in range(0, self.run_wt): - self.runWt(['-h', '.', '-R', 'list', '-v'], outfilename="list.out") + self.runWt(['-C', 'config_base=false,create,log=(enabled)', '-h', '.', '-R', 'list', '-v'], outfilename="list.out") self.open_conn() q = self.conn.query_timestamp('get=recovery') diff --git a/src/third_party/wiredtiger/test/suite/test_timestamp11.py b/src/third_party/wiredtiger/test/suite/test_timestamp11.py index 3c56b66bf66..0b31abbccb9 100644 --- a/src/third_party/wiredtiger/test/suite/test_timestamp11.py +++ b/src/third_party/wiredtiger/test/suite/test_timestamp11.py @@ -31,7 +31,7 @@ # from suite_subprocess import suite_subprocess -import wiredtiger, wttest +import wiredtiger, wttest, unittest def timestamp_str(t): return '%x' % t diff --git a/src/third_party/wiredtiger/test/suite/test_timestamp12.py b/src/third_party/wiredtiger/test/suite/test_timestamp12.py index fcbaf5d01ef..cf258814a87 100644 --- a/src/third_party/wiredtiger/test/suite/test_timestamp12.py +++ b/src/third_party/wiredtiger/test/suite/test_timestamp12.py @@ -30,7 +30,7 @@ # Timestamps: Test the use_timestamp setting when closing the connection. # -import shutil, os, wiredtiger, wttest +import shutil, os, unittest, wiredtiger, wttest from wtscenario import make_scenarios def timestamp_str(t): diff --git a/src/third_party/wiredtiger/test/suite/test_timestamp14.py b/src/third_party/wiredtiger/test/suite/test_timestamp14.py index e5ae04ea022..c5add436882 100644 --- a/src/third_party/wiredtiger/test/suite/test_timestamp14.py +++ b/src/third_party/wiredtiger/test/suite/test_timestamp14.py @@ -259,7 +259,7 @@ class test_timestamp14(wttest.WiredTigerTestCase, suite_subprocess): # We have a running transaction with a lower commit_timestamp than we've # seen before. So all_durable should return (lowest commit timestamp - 1). session1.begin_transaction() - cur1[1] = 2 + cur1[2] = 2 session1.timestamp_transaction('commit_timestamp=2') self.assertTimestampsEqual( self.conn.query_timestamp('get=all_durable'), '1') @@ -272,7 +272,7 @@ class test_timestamp14(wttest.WiredTigerTestCase, suite_subprocess): # For prepared transactions, we take into account the durable timestamp # when calculating all_durable. session1.begin_transaction() - cur1[1] = 3 + cur1[3] = 3 session1.prepare_transaction('prepare_timestamp=6') # If we have a commit timestamp for a prepared transaction, then we @@ -290,7 +290,7 @@ class test_timestamp14(wttest.WiredTigerTestCase, suite_subprocess): # All durable moves back when we have a running prepared transaction # with a lower durable timestamp than has previously been committed. session1.begin_transaction() - cur1[1] = 4 + cur1[4] = 4 session1.prepare_transaction('prepare_timestamp=3') # If we have a commit timestamp for a prepared transaction, then we @@ -310,7 +310,7 @@ class test_timestamp14(wttest.WiredTigerTestCase, suite_subprocess): # Now test a scenario with multiple commit timestamps for a single txn. session1.begin_transaction() - cur1[1] = 5 + cur1[5] = 5 session1.timestamp_transaction('commit_timestamp=6') self.assertTimestampsEqual( self.conn.query_timestamp('get=all_durable'), '5') @@ -318,7 +318,7 @@ class test_timestamp14(wttest.WiredTigerTestCase, suite_subprocess): # Make more changes and set a new commit timestamp. # Our calculation should use the first commit timestamp so there should # be no observable difference to the all_durable value. - cur1[1] = 6 + cur1[6] = 6 session1.timestamp_transaction('commit_timestamp=7') self.assertTimestampsEqual( self.conn.query_timestamp('get=all_durable'), '5') @@ -345,7 +345,7 @@ class test_timestamp14(wttest.WiredTigerTestCase, suite_subprocess): cur1[1]=1 session1.commit_transaction('commit_timestamp=2') session1.begin_transaction() - cur1[1]=2 + cur1[2]=2 session1.commit_transaction('commit_timestamp=4') # Confirm all_durable is now 4. self.assertTimestampsEqual( @@ -360,7 +360,7 @@ class test_timestamp14(wttest.WiredTigerTestCase, suite_subprocess): self.assertEqual(cur1[1], 1) # Commit some data at timestamp 7. session2.begin_transaction() - cur2[2] = 2 + cur2[3] = 2 session2.commit_transaction('commit_timestamp=7') # All_durable should now be 7. self.assertTimestampsEqual( @@ -379,7 +379,7 @@ class test_timestamp14(wttest.WiredTigerTestCase, suite_subprocess): # to the oldest timestamp. session2.begin_transaction() session2.timestamp_transaction('commit_timestamp=6') - cur2[2] = 3 + cur2[4] = 3 # Confirm all_durable is now equal to oldest. self.assertTimestampsEqual( diff --git a/src/third_party/wiredtiger/test/suite/test_timestamp18.py b/src/third_party/wiredtiger/test/suite/test_timestamp18.py new file mode 100644 index 00000000000..913bd6c7649 --- /dev/null +++ b/src/third_party/wiredtiger/test/suite/test_timestamp18.py @@ -0,0 +1,107 @@ +#!/usr/bin/env python +# +# Public Domain 2014-2020 MongoDB, Inc. +# Public Domain 2008-2014 WiredTiger, Inc. +# +# This is free and unencumbered software released into the public domain. +# +# Anyone is free to copy, modify, publish, use, compile, sell, or +# distribute this software, either in source code form or as a compiled +# binary, for any purpose, commercial or non-commercial, and by any +# means. +# +# In jurisdictions that recognize copyright laws, the author or authors +# of this software dedicate any and all copyright interest in the +# software to the public domain. We make this dedication for the benefit +# of the public at large and to the detriment of our heirs and +# successors. We intend this dedication to be an overt act of +# relinquishment in perpetuity of all present and future rights to this +# software under copyright law. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +# IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. +# +# test_timestamp18.py +# Mixing timestamped and non-timestamped writes. +# + +import wiredtiger, wttest +from wtscenario import make_scenarios + +def timestamp_str(t): + return '%x' % t + +class test_timestamp18(wttest.WiredTigerTestCase): + conn_config = 'cache_size=50MB' + session_config = 'isolation=snapshot' + non_ts_writes = [ + ('insert', dict(delete=False)), + ('delete', dict(delete=True)), + ] + scenarios = make_scenarios(non_ts_writes) + + def test_ts_writes_with_non_ts_write(self): + uri = 'table:test_timestamp18' + self.session.create(uri, 'key_format=S,value_format=S') + self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(1)) + cursor = self.session.open_cursor(uri) + + value1 = 'a' * 500 + value2 = 'b' * 500 + value3 = 'c' * 500 + value4 = 'd' * 500 + + # A series of timestamped writes on each key. + for i in range(1, 10000): + self.session.begin_transaction() + cursor[str(i)] = value1 + self.session.commit_transaction('commit_timestamp=' + timestamp_str(2)) + + for i in range(1, 10000): + self.session.begin_transaction() + cursor[str(i)] = value2 + self.session.commit_transaction('commit_timestamp=' + timestamp_str(3)) + + for i in range(1, 10000): + self.session.begin_transaction() + cursor[str(i)] = value3 + self.session.commit_transaction('commit_timestamp=' + timestamp_str(4)) + + # Add a non-timestamped delete. + # Let's do every second key to ensure that we get the truncation right and don't + # accidentally destroy content from an adjacent key. + for i in range(1, 10000): + if i % 2 == 0: + if self.delete: + cursor.set_key(str(i)) + cursor.remove() + else: + cursor[str(i)] = value4 + + self.session.checkpoint() + + for ts in range(2, 4): + self.session.begin_transaction('read_timestamp=' + timestamp_str(ts)) + for i in range(1, 10000): + # The non-timestamped delete should cover all the previous writes and make them effectively + # invisible. + if i % 2 == 0: + if self.delete: + cursor.set_key(str(i)) + self.assertEqual(cursor.search(), wiredtiger.WT_NOTFOUND) + else: + self.assertEqual(cursor[str(i)], value4) + # Otherwise, expect one of the timestamped writes. + else: + if ts == 2: + self.assertEqual(cursor[str(i)], value1) + elif ts == 3: + self.assertEqual(cursor[str(i)], value2) + else: + self.assertEqual(cursor[str(i)], value3) + self.session.rollback_transaction() diff --git a/src/third_party/wiredtiger/test/suite/test_truncate02.py b/src/third_party/wiredtiger/test/suite/test_truncate02.py index c60698b2a15..f4e41e1aef2 100644 --- a/src/third_party/wiredtiger/test/suite/test_truncate02.py +++ b/src/third_party/wiredtiger/test/suite/test_truncate02.py @@ -30,7 +30,7 @@ # session level operations on tables # -import wiredtiger, wttest +import unittest, wiredtiger, wttest from wtdataset import SimpleDataSet from wtscenario import make_scenarios diff --git a/src/third_party/wiredtiger/test/suite/test_txn02.py b/src/third_party/wiredtiger/test/suite/test_txn02.py index ac6626a9b28..648e31b5374 100644 --- a/src/third_party/wiredtiger/test/suite/test_txn02.py +++ b/src/third_party/wiredtiger/test/suite/test_txn02.py @@ -30,7 +30,7 @@ # Transactions: commits and rollbacks # -import fnmatch, os, shutil, time +import fnmatch, os, shutil, time, unittest from suite_subprocess import suite_subprocess from wtscenario import make_scenarios import wttest diff --git a/src/third_party/wiredtiger/test/suite/test_txn04.py b/src/third_party/wiredtiger/test/suite/test_txn04.py index 5c45ba509e4..10c0f4ab3c5 100644 --- a/src/third_party/wiredtiger/test/suite/test_txn04.py +++ b/src/third_party/wiredtiger/test/suite/test_txn04.py @@ -33,7 +33,7 @@ import shutil, os from suite_subprocess import suite_subprocess from wtscenario import make_scenarios -import wttest +import unittest, wttest class test_txn04(wttest.WiredTigerTestCase, suite_subprocess): logmax = "100K" @@ -183,7 +183,6 @@ class test_txn04(wttest.WiredTigerTestCase, suite_subprocess): # Backup the target we modified and verify the data. # print 'Call hot_backup with ' + self.uri self.hot_backup(self.uri, committed) - def test_ops(self): self.backup_dir = os.path.join(self.home, "WT_BACKUP") self.session2 = self.conn.open_session() diff --git a/src/third_party/wiredtiger/test/suite/test_txn05.py b/src/third_party/wiredtiger/test/suite/test_txn05.py index 4ef8e8c9b83..53fc6496bae 100644 --- a/src/third_party/wiredtiger/test/suite/test_txn05.py +++ b/src/third_party/wiredtiger/test/suite/test_txn05.py @@ -30,7 +30,7 @@ # Transactions: commits and rollbacks # -import fnmatch, os, shutil, time +import fnmatch, os, shutil, time, unittest from suite_subprocess import suite_subprocess from wtscenario import make_scenarios import wttest diff --git a/src/third_party/wiredtiger/test/suite/test_txn06.py b/src/third_party/wiredtiger/test/suite/test_txn06.py index a4fb78ff7e6..24bb192e938 100755 --- a/src/third_party/wiredtiger/test/suite/test_txn06.py +++ b/src/third_party/wiredtiger/test/suite/test_txn06.py @@ -44,12 +44,17 @@ class test_txn06(wttest.WiredTigerTestCase, suite_subprocess): # Populate a table SimpleDataSet(self, self.source_uri, self.nrows).populate() - # Now scan the table and copy the rows into a new table + # Now scan the table and copy the rows into a new table. The cursor will keep the snapshot + # in self.session pinned while the inserts cause new IDs to be allocated. c_src = self.session.create(self.uri, "key_format=S,value_format=S") c_src = self.session.open_cursor(self.source_uri) - c = self.session.open_cursor(self.uri) + insert_session = self.conn.open_session() + c = insert_session.open_cursor(self.uri) for k, v in c_src: c[k] = v + # We were trying to generate a message matching this pattern. + self.captureout.checkAdditionalPattern(self, "old snapshot") + if __name__ == '__main__': wttest.run() diff --git a/src/third_party/wiredtiger/test/suite/test_txn07.py b/src/third_party/wiredtiger/test/suite/test_txn07.py index a7090b53703..0fe78cbb7ab 100644 --- a/src/third_party/wiredtiger/test/suite/test_txn07.py +++ b/src/third_party/wiredtiger/test/suite/test_txn07.py @@ -34,7 +34,7 @@ import fnmatch, os, shutil, run, time from suite_subprocess import suite_subprocess from wiredtiger import stat from wtscenario import make_scenarios -import wttest +import unittest, wttest class test_txn07(wttest.WiredTigerTestCase, suite_subprocess): logmax = "100K" @@ -51,10 +51,11 @@ class test_txn07(wttest.WiredTigerTestCase, suite_subprocess): types = [ ('row', dict(tabletype='row', create_params = 'key_format=i,value_format=S')), - ('var', dict(tabletype='var', - create_params = 'key_format=r,value_format=S')), - ('fix', dict(tabletype='fix', - create_params = 'key_format=r,value_format=8t')), + # The commented columnar tests needs to be enabled once rollback to stable for columnar is fixed in (WT-5548). + # ('var', dict(tabletype='var', + # create_params = 'key_format=r,value_format=S')), + # ('fix', dict(tabletype='fix', + # create_params = 'key_format=r,value_format=8t')), ] op1s = [ ('trunc-all', dict(op1=('all', 0))), @@ -200,14 +201,9 @@ class test_txn07(wttest.WiredTigerTestCase, suite_subprocess): # Check the state after each commit/rollback. self.check_all(current, committed) - # - # Run printlog and make sure it exits with zero status. This should be - # run as soon as we can after the crash to try and conflict with the - # journal file read. - # - - self.runWt(['-h', self.backup_dir, 'printlog'], outfilename='printlog.out') + # Gather statistics - this needs to be done before the connection is + # closed or statistics would be reset. stat_cursor = self.session.open_cursor('statistics:', None, None) clen = stat_cursor[stat.conn.log_compress_len][2] cmem = stat_cursor[stat.conn.log_compress_mem][2] @@ -229,5 +225,12 @@ class test_txn07(wttest.WiredTigerTestCase, suite_subprocess): self.assertEqual(cwrites > 0, True) self.assertEqual((cfails > 0 or csmall > 0), True) + # + # Run printlog and make sure it exits with zero status. This should be + # run as soon as we can after the crash to try and conflict with the + # journal file read. + # + self.runWt(['-h', self.backup_dir, 'printlog'], outfilename='printlog.out') + if __name__ == '__main__': wttest.run() diff --git a/src/third_party/wiredtiger/test/suite/test_txn09.py b/src/third_party/wiredtiger/test/suite/test_txn09.py index 98c7dbd9c48..a6ca3a64082 100644 --- a/src/third_party/wiredtiger/test/suite/test_txn09.py +++ b/src/third_party/wiredtiger/test/suite/test_txn09.py @@ -30,7 +30,7 @@ # Transactions: recovery toggling logging # -import fnmatch, os, shutil, time +import fnmatch, os, shutil, time, unittest from suite_subprocess import suite_subprocess from wtscenario import make_scenarios import wttest diff --git a/src/third_party/wiredtiger/test/suite/test_txn16.py b/src/third_party/wiredtiger/test/suite/test_txn16.py index 9e2829246e5..34a363a83e4 100644 --- a/src/third_party/wiredtiger/test/suite/test_txn16.py +++ b/src/third_party/wiredtiger/test/suite/test_txn16.py @@ -33,7 +33,7 @@ import fnmatch, os, shutil, time from suite_subprocess import suite_subprocess -import wttest +import unittest, wttest class test_txn16(wttest.WiredTigerTestCase, suite_subprocess): t1 = 'table:test_txn16_1' diff --git a/src/third_party/wiredtiger/test/suite/test_txn19.py b/src/third_party/wiredtiger/test/suite/test_txn19.py index 12484cd001a..417321bc97f 100755 --- a/src/third_party/wiredtiger/test/suite/test_txn19.py +++ b/src/third_party/wiredtiger/test/suite/test_txn19.py @@ -33,7 +33,7 @@ import fnmatch, os, shutil, time from wtscenario import make_scenarios from suite_subprocess import suite_subprocess -import wiredtiger, wttest +import unittest, wiredtiger, wttest # This test uses an artificially small log file limit, and creates # large records so two fit into a log file. This allows us to test @@ -384,7 +384,7 @@ class test_txn19_meta(wttest.WiredTigerTestCase, suite_subprocess): ('WiredTiger.basecfg', dict(filename='WiredTiger.basecfg')), ('WiredTiger.turtle', dict(filename='WiredTiger.turtle')), ('WiredTiger.wt', dict(filename='WiredTiger.wt')), - ('WiredTigerLAS.wt', dict(filename='WiredTigerLAS.wt')), + ('WiredTigerHS.wt', dict(filename='WiredTigerHS.wt')), ] # In many cases, wiredtiger_open without any salvage options will @@ -392,34 +392,34 @@ class test_txn19_meta(wttest.WiredTigerTestCase, suite_subprocess): openable = [ "removal:WiredTiger.basecfg", "removal:WiredTiger.turtle", - "removal:WiredTigerLAS.wt", + "removal:WiredTigerHS.wt", "truncate:WiredTiger", "truncate:WiredTiger.basecfg", - "truncate:WiredTigerLAS.wt", + "truncate:WiredTigerHS.wt", "truncate-middle:WiredTiger", "truncate-middle:WiredTiger.basecfg", "truncate-middle:WiredTiger.turtle", "truncate-middle:WiredTiger.wt", - "truncate-middle:WiredTigerLAS.wt", + "truncate-middle:WiredTigerHS.wt", "zero:WiredTiger", "zero:WiredTiger.basecfg", - "zero:WiredTigerLAS.wt", + "zero:WiredTigerHS.wt", "zero-end:WiredTiger", "zero-end:WiredTiger.basecfg", "zero-end:WiredTiger.turtle", "zero-end:WiredTiger.wt", - "zero-end:WiredTigerLAS.wt", + "zero-end:WiredTigerHS.wt", "garbage-begin:WiredTiger", - "garbage-begin:WiredTigerLAS.wt", + "garbage-begin:WiredTigerHS.wt", "garbage-middle:WiredTiger", "garbage-middle:WiredTiger.basecfg", "garbage-middle:WiredTiger.turtle", "garbage-middle:WiredTiger.wt", - "garbage-middle:WiredTigerLAS.wt", + "garbage-middle:WiredTigerHS.wt", "garbage-end:WiredTiger", "garbage-end:WiredTiger.turtle", "garbage-end:WiredTiger.wt", - "garbage-end:WiredTigerLAS.wt", + "garbage-end:WiredTigerHS.wt", ] # The cases for which salvage will not work, represented in the @@ -476,6 +476,7 @@ class test_txn19_meta(wttest.WiredTigerTestCase, suite_subprocess): key = self.kind + ':' + self.filename return key not in self.not_salvageable + @unittest.skip("Temporarily disabled") def test_corrupt_meta(self): errfile = 'list.err' outfile = 'list.out' diff --git a/src/third_party/wiredtiger/test/suite/test_util01.py b/src/third_party/wiredtiger/test/suite/test_util01.py index f78f408cfff..ba0e71caba0 100755 --- a/src/third_party/wiredtiger/test/suite/test_util01.py +++ b/src/third_party/wiredtiger/test/suite/test_util01.py @@ -26,12 +26,15 @@ # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. -import string, os, sys +import string, os, sys, random from suite_subprocess import suite_subprocess -import wiredtiger, wttest +import wiredtiger, wttest, unittest _python3 = (sys.version_info >= (3, 0, 0)) +def timestamp_str(t): + return '%x' % t + # test_util01.py # Utilities: wt dump, as well as the dump cursor class test_util01(wttest.WiredTigerTestCase, suite_subprocess): @@ -46,6 +49,7 @@ class test_util01(wttest.WiredTigerTestCase, suite_subprocess): tablename = 'test_util01.a' nentries = 1000 + session_config = 'isolation=snapshot' stringclass = ''.__class__ def compare_config(self, expected_cfg, actual_cfg): @@ -141,7 +145,24 @@ class test_util01(wttest.WiredTigerTestCase, suite_subprocess): # The output from dump is a 'u' format. return b.strip(b'\x00').decode() + '\n' - def dump(self, usingapi, hexoutput): + def write_entries(self, cursor, expectout, hexoutput, commit_timestamp, write_expected): + if commit_timestamp is not None: + self.session.begin_transaction() + for i in range(0, self.nentries): + key = self.get_key(i) + value = 0 + if write_expected: + value = self.get_value(i) + else: + value = self.get_value(i + random.randint(1, self.nentries)) + cursor[key] = value + if write_expected: + expectout.write(self.dumpstr(key, hexoutput)) + expectout.write(self.dumpstr(value, hexoutput)) + if commit_timestamp is not None: + self.session.commit_transaction('commit_timestamp=' + timestamp_str(commit_timestamp)) + + def dump(self, usingapi, hexoutput, commit_timestamp, read_timestamp): params = self.table_config() self.session.create('table:' + self.tablename, params) cursor = self.session.open_cursor('table:' + self.tablename, None, None) @@ -161,13 +182,19 @@ class test_util01(wttest.WiredTigerTestCase, suite_subprocess): expectout.write('table:' + self.tablename + '\n') expectout.write('colgroups=,columns=,' + params + '\n') expectout.write('Data\n') - for i in range(0, self.nentries): - key = self.get_key(i) - value = self.get_value(i) - cursor[key] = value - expectout.write(self.dumpstr(key, hexoutput)) - expectout.write(self.dumpstr(value, hexoutput)) - cursor.close() + if commit_timestamp is not None and read_timestamp is not None: + if commit_timestamp == read_timestamp: + self.write_entries(cursor, expectout, hexoutput, commit_timestamp, True) + self.write_entries(cursor, expectout, hexoutput, commit_timestamp + 1, False) + elif commit_timestamp < read_timestamp: + self.write_entries(cursor, expectout, hexoutput, commit_timestamp, False) + self.write_entries(cursor, expectout, hexoutput, commit_timestamp + 1, True) + else: + self.write_entries(cursor, expectout, hexoutput, commit_timestamp, False) + self.write_entries(cursor, expectout, hexoutput, commit_timestamp + 1, False) + else: + self.write_entries(cursor, expectout, hexoutput, commit_timestamp, True) + cursor.close() self.pr('calling dump') with open("dump.out", "w") as dumpout: @@ -186,22 +213,34 @@ class test_util01(wttest.WiredTigerTestCase, suite_subprocess): dumpargs = ["dump"] if hexoutput: dumpargs.append("-x") + if read_timestamp: + dumpargs.append("-t " + str(read_timestamp)) dumpargs.append(self.tablename) self.runWt(dumpargs, outfilename="dump.out") self.assertTrue(self.compare_files("expect.out", "dump.out")) def test_dump_process(self): - self.dump(False, False) + self.dump(False, False, None, None) def test_dump_process_hex(self): - self.dump(False, True) + self.dump(False, True, None, None) def test_dump_api(self): - self.dump(True, False) + self.dump(True, False, None, None) def test_dump_api_hex(self): - self.dump(True, True) + self.dump(True, True, None, None) + + @unittest.skip("Temporarily Disabled") + def test_dump_process_timestamp_old(self): + self.dump(False, False, 5, 5) + + def test_dump_process_timestamp_none(self): + self.dump(False, False, 5 , 3) + + def test_dump_process_timestamp_new(self): + self.dump(False, False, 5, 7) if __name__ == '__main__': wttest.run() diff --git a/src/third_party/wiredtiger/test/suite/test_util04.py b/src/third_party/wiredtiger/test/suite/test_util04.py index 518933e226c..69a76471741 100644 --- a/src/third_party/wiredtiger/test/suite/test_util04.py +++ b/src/third_party/wiredtiger/test/suite/test_util04.py @@ -26,7 +26,7 @@ # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. -import os +import os, unittest from suite_subprocess import suite_subprocess import wiredtiger, wttest diff --git a/src/third_party/wiredtiger/test/suite/test_util11.py b/src/third_party/wiredtiger/test/suite/test_util11.py index 92f3dbc5c75..aec700e51d2 100644 --- a/src/third_party/wiredtiger/test/suite/test_util11.py +++ b/src/third_party/wiredtiger/test/suite/test_util11.py @@ -26,7 +26,7 @@ # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. -import os, struct +import os, struct, unittest from suite_subprocess import suite_subprocess import wiredtiger, wttest diff --git a/src/third_party/wiredtiger/test/suite/test_util16.py b/src/third_party/wiredtiger/test/suite/test_util16.py index f2e50665c85..2d2db88e361 100644 --- a/src/third_party/wiredtiger/test/suite/test_util16.py +++ b/src/third_party/wiredtiger/test/suite/test_util16.py @@ -26,7 +26,7 @@ # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. -import os +import os, unittest from suite_subprocess import suite_subprocess import wiredtiger, wttest diff --git a/src/third_party/wiredtiger/test/syscall/wt2336_base/base.run b/src/third_party/wiredtiger/test/syscall/wt2336_base/base.run index 3f7174c3c9a..772bf939f40 100644 --- a/src/third_party/wiredtiger/test/syscall/wt2336_base/base.run +++ b/src/third_party/wiredtiger/test/syscall/wt2336_base/base.run @@ -112,7 +112,7 @@ rename("./WiredTiger.turtle.set", "./WiredTiger.turtle"); ... // There is a second open of turtle here, is it important? -fd = OPEN("./WiredTigerLAS.wt", O_RDWR|O_CREAT|O_EXCL|O_NOATIME|O_CLOEXEC, 0666); +fd = OPEN("./WiredTigerHS.wt", O_RDWR|O_CREAT|O_EXCL|O_NOATIME|O_CLOEXEC, 0666); #ifdef __linux__ dir = OPEN("./", O_RDONLY|O_CLOEXEC); @@ -127,7 +127,7 @@ fdatasync(fd); #endif /* __linux__ */ close(fd); -fd = OPEN_EXISTING("./WiredTigerLAS.wt", O_RDWR|O_NOATIME|O_CLOEXEC); +fd = OPEN_EXISTING("./WiredTigerHS.wt", O_RDWR|O_NOATIME|O_CLOEXEC); FTRUNCATE(fd, 0x1000); fd = OPEN_EXISTING("./WiredTiger.turtle", O_RDWR|O_CLOEXEC); close(fd); |