diff options
| author | Luke Chen <luke.chen@mongodb.com> | 2021-07-21 16:05:14 +1000 |
|---|---|---|
| committer | Evergreen Agent <no-reply@evergreen.mongodb.com> | 2021-07-21 06:24:58 +0000 |
| commit | 03c49a3d3b228c68a2a369bd7a4d8f5588c72d09 (patch) | |
| tree | c7d75b8281aa0d2702ba4de99b8860bcaa1f823e | |
| parent | 744675fe9ca246317dc8d68aaeb2e9070c06989c (diff) | |
| download | mongo-03c49a3d3b228c68a2a369bd7a4d8f5588c72d09.tar.gz | |
Import wiredtiger: 2b73914cd8912fab0e01ebd67cd0106de45442cd from branch mongodb-4.4
ref: 5444fd4334..2b73914cd8
for: 4.4.8
WT-6280 Fail eviction if out of order handling races with checkpoint
WT-6729 Quiesce eviction prior running rollback to stable's active transaction check
WT-6782 test_prepare_hs02 WT_ROLLBACK failure: conflict between concurrent operations
WT-7231 Add CMake build and tests into Evergreen
WT-7279 Allow multiple terminate calls for storage source extension
WT-7317 Improve the runtime_monitor component to track history store cleanup statistics
WT-7338 Copy the configuration directory when building the test framework
WT-7343 Write a script that executes many-collection-test.py
WT-7383 Add framework for new hs_cleanup test and refactor workload generator and database operation
WT-7447 Fix the assert fire because onpage out of order update is not popped from the stack
WT-7473 Resolve "TODO: tiered" comments in code
WT-7507 Update salvage for a history store and timestamp world
WT-7520 Add start and stop values to automatic flag generation code
WT-7524 Refactor functions to obtain checkpoint list; Clear delete on skipping checkpoints
WT-7539 Add a configuration option that allows the user to specify debug modes.
WT-7543 Pass in the correct test name when constructing the default config path
WT-7552 Add UBSAN to automated WT testing
WT-7553 Loosen the restrictions around evicting fast-truncate pages to avoid cache-stuck failures
WT-7556 Fix test_rollback_to_stable10 failure cache_hs_ondisk is 0
WT-7583 Coverity analysis defect 114074: Logically dead code (rework)
WT-7585 Fix cyclomatic-complexity test failure
WT-7589 Fix reopening connection after a flush_tier in tiered tests
WT-7591 Fixes to allow cursors to be open during flush_tier
WT-7603 Sort statistics to fix JSON output
WT-7605 Drop support for million-collection-test
WT-7609 Report on time taken to start and shutdown the database in many-coll-test
WT-7616 Create a pass or fail test based on many collection workgen workload
WT-7619 Add a new optimization to skip pages in cursor traversal when all entries on the page are deleted
WT-7626 We only ensure update restore eviction happened in test debug mode09
WT-7628 Return an error message when supplied invalid command line args in the test framework
WT-7629 Run clang format on .cxx files in the codebase.
WT-7632 Fix invalid argument in test_rollback_to_stable14
WT-7636 Fix leaked cursors by implementing scoped cursor and session types
WT-7639 Alter test_tiered02.py to collect more data before asserting for missing file
WT-7640 Fix test_backup02 failure where checkpoint tables differ due to checkpoint cursor not supported
WT-7644 Implement python hooks for tiered storage
WT-7646 Remove unneeded uses of WT_WITH_BUCKET_STORAGE macro
WT-7647 Change Zstandard wrapper to include context management
WT-7648 Improve error messaging in the test framework
WT-7649 Skip timestamp assert during recovery
WT-7659 Disallow rename on tiered table
WT-7660 Rename poc_test to base_test in the cpp test framework and add insert_operation logic
WT-7665 Apply op tracking inserts in the workload transaction
WT-7666 Add assertion to check whether duplicate history store inserts are modifies
WT-7667 Fix workgen JSON output
WT-7668 Overload the update method for the hs_cleanup test
WT-7670 Modify test tag format and tag additional python tests
WT-7672 Remove make-check-test from Windows CMake Evergreen build variant
WT-7674 reduce rollback-to-stable work for fast-truncate pages
WT-7675 Query last ckpt timestamp changes without taking checkpoint
WT-7676 Reformat wtperf backup to only read in files instead of wt_copy_and_sync
WT-7679 Create an evergreen test for many-dhandle-stress
WT-7680 refactor rollback-to-stable to support operating on individual files
WT-7683 Add python test hook to call flush_tier() during connection.close()
WT-7685 Fix invalid values and units for latencies in workgen
WT-7686 Fix config call, allowing "sync" option to work for flush_tier
WT-7687 Stop tiered manager thread before internal thread
WT-7689 Fix double free in `__curhs_insert`
WT-7690 Fix coverity error when enum is compared against 0 (incr_backup:table_changes)
WT-7692 fix make check test failure on osx10 14 cmake
WT-7696 Fix coverity error - Unused variable in _rollback_to_stable_btree_apply_all
WT-7698 Decrease max_latency value in many dhandles scenario for workgen
WT-7699 Fix RTS handling to abort an out of order prepared transaction
WT-7705 Add an assert to ensure that there are no updates on the new disk image in update restore
WT-7706 Use same transaction update when on-disk value is an aborted prepared update
WT-7707 Simplify insert splits to use the splitting WT_REFs key
WT-7708 Add an assert to ensure the durable timestamp is larger than stable timestamp at the end of prepared commit
WT-7710 Fix to use history store btree to initialise history store cursor
WT-7715 Fix uninitialized bool in txn_ckpt.c
WT-7717 Change macros in local_store.c to use WT namespace
WT-7719 Change default value of ENABLE_STRICT to "OFF" (CMake Usability Improvements)
WT-7720 Update POSIX CMAKE doxygen documentation (CMake Usability Improvements)
WT-7721 Update test-format to reopen an existing database with different config
WT-7723 Delete the updates in the history store if they are rolled back or is the first stable update on the update chain
WT-7724 Fix race when running concurrent checkpoint and flush_tier
WT-7725 Add missing brackets around parameter in macro definition
WT-7726 Separating out the validation portion from the database model
WT-7727 Fix null pointer passed to memcpy() during 'format' test.
WT-7729 Fix to write out correct tiered information on checkpoint
WT-7730 Shifting the oldest and stable timestamps to match the commit timestamp format
WT-7739 Switch back to using MacOS 10.14 for Evergreen compile task
WT-7741 Fix misaligned address in crc32-x86.c
WT-7742 Fix misaligned address in wt3184_dup_index_collator/main.c
WT-7743 Fix integer overflow within wt2999_join_extractor csuite test
WT-7744 Fix null pointer within wt3338_partial_update csuite
WT-7746 Improve directory syncing with CMake helper 'create_test_executable'
WT-7748 Fix CMake library probes for libraries not on the default linker path
WT-7749 Assorted fixes for (and from) building and testing on NetBSD
WT-7751 Add an assert to ensure we never select an update that has been written to the history store for data store
WT-7752 Update packing code according to documentation
WT-7754 Fix race when updating block manager file handle
WT-7755 YSCB: Add a native implementation of YCSB to WTPERF.
WT-7756 RTS to clear the HS flag of an update following tombstone
WT-7760 Support array parsing in CppSuite config handling
WT-7761 Improve debug_print to include timestamp, thread_id and reorder args in cppsuite.
WT-7762 Create stressful configs for the two tests add them to evergreen.
WT-7763 Coverity issues found in the stress testing framework
WT-7765 Fix signed integer overflow in intpack-test3
WT-7766 Fix null pointer passed to memset in test_wt3338_partial_update
WT-7767 Code cleanup for curhs_remove and __txn_fixup_prepared_update
WT-7770 Fix issue linking TCMalloc in CMake
WT-7776 Add a hard limit on the number of modify updates before we instantiate a complete update
WT-7778 Fix null dereferencing, and return of incorrect allocation size
WT-7780 Guarantee log message sequencing in the test framework.
WT-7781 Avoid skipping python tests for 3rd party ext libraries in CMake builds
WT-7782 Separate implementation from headers in cppsuite test framework
WT-7783 Fix RTS to restore tombstone when an on-disk update is out of order prepare update
WT-7784 Enable RTS to use checkpoint snapshot on timestamp tables
WT-7795 Fix CppSuite failure "expected ) before PRIxMAX"
WT-7796 Scan the tracking table and delete parts of it that are obsolete.
WT-7797 Disable postrun stats in CppSuite testing
WT-7799 Do not report failure of wiredtiger_open in python tests to output
WT-7802 Remove data store same transaction update squash logic
WT-7804 Fix test_hs24 committing mixed mode update from the wrong value
WT-7807 Remove unused arg in backup_config
WT-7811 Fix test_hs24 not commiting from 0 timestamp
WT-7813 Stop inserting to history store if we see a prepared update
WT-7815 Properly initialize prev_upd_ts for ordered timestamp assertion
WT-7825 Fix test_hs24 key order
WT-7828 Move many-coll-test to ubuntu1804-wt-large and update thresholds accordingly
WT-7831 Clear transaction ids from previous run when repack the cell
WT-7832 Add an encryptor extension that uses the libsodium cryptography library.
WT-7836 Fixing a number of small issues in the cppsuite test framework
WT-7837 Clear updates structure in wt_hs_insert_updates to avoid firing assert
WT-7841 add "only unencrypted" checksum configuration, switch checksum default to "on"
WT-7843 Add missing macros to define PRIxMAX
WT-7846 Disable test_tiered08
WT-7851 Fix illegal checksum configuration in compatibility-test-for-newer-releases
WT-7852 Don't release eviction server lock when evicting pages
WT-7856 Enable flush test with open cursor in test_tiered04
Reverted ticket(s):
WT-7443 Add error message when bulk cursor can't get exclusive access to dhandle
317 files changed, 13775 insertions, 6840 deletions
diff --git a/src/third_party/wiredtiger/bench/workgen/runner/many-dhandle-stress.py b/src/third_party/wiredtiger/bench/workgen/runner/many-dhandle-stress.py index 4fdd88dd88e..4c95d28b45d 100644 --- a/src/third_party/wiredtiger/bench/workgen/runner/many-dhandle-stress.py +++ b/src/third_party/wiredtiger/bench/workgen/runner/many-dhandle-stress.py @@ -115,7 +115,7 @@ checkpoint_thread = Thread(ops) workload = Workload(context, 10 * thread0 + 10 * thread1 + checkpoint_thread) workload.options.report_interval=5 workload.options.run_time=900 -workload.options.max_latency=1000 +workload.options.max_latency=60 workload.options.sample_rate=1 workload.options.sample_interval_ms = 5000 # Uncomment to fail instead of generating a warning diff --git a/src/third_party/wiredtiger/bench/workgen/runner/workgen_perf_check.sh b/src/third_party/wiredtiger/bench/workgen/runner/workgen_perf_check.sh new file mode 100755 index 00000000000..e9308435923 --- /dev/null +++ b/src/third_party/wiredtiger/bench/workgen/runner/workgen_perf_check.sh @@ -0,0 +1,93 @@ +#! /bin/bash +# +# Checks warnings and exceptions raised by a given workgen test as well as lantencies of specific operations. +# This script is used in evergreen to assess performance using workgen. +# + +usage () { + cat << EOF +Usage: $0 test_name output threshold_1 threshold_2 threshold_3 +Arguments: + test_name # Test to run + output # File output + threshold_1 # Maximum allowed warnings from DROP operations. + threshold_2 # Maximum time in seconds a DROP operation can take. + threshold_3 # Maximum allowed warnings from CREATE/INSERT/UPDATE operations. +EOF +} + +if [ $1 == "-h" ]; then + usage + exit +fi + +if [ "$#" -ne 5 ]; then + echo "Illegal number of parameters." + usage + echo FAILED + exit 1 +fi + +if [ ! -f $1 ]; then + echo "$1 does not exist." + echo FAILED + exit 1 +fi + +ERROR=0 +TEST=$1 +OUTPUT=$2 +DROP_WARNINGS_THRESHOLD=$3 +MAX_DROP_THRESHOLD=$4 +OP_WARNINGS_THRESHOLD=$5 + +echo "python3 $TEST 2>&1 | tee $OUTPUT" +python3 $TEST 2>&1 | tee $OUTPUT + +# Check exceptions +if grep -io "exception" $OUTPUT; then + echo ERROR + ERROR=1 +fi + +# Maximum number of DROP warnings +DROP_WARNINGS=$(grep -ic "cycling idle.*drop" $OUTPUT) +echo "Number of long drop operations: $DROP_WARNINGS" + +# Maximum number of READ/INSERT/UPDATE warnings +OP_WARNINGS=$(grep -ic "max latency exceeded" $OUTPUT) +echo "Number of long read/insert/update operations: $OP_WARNINGS" + +# Output the 5 worst DROP latencies +DROP_5=($(grep -i "cycling idle.*drop" $OUTPUT | awk '{print $9}' | sort -n | tail -5)) +echo -n "Five worst drop operations (s): " +for ((i = 0; i < ${#DROP_5[@]}; ++i)); do + echo -n "${DROP_5[$i]} " +done +echo + +# Check if too many DROP operations took too long +if [[ $DROP_WARNINGS -ge $DROP_WARNINGS_THRESHOLD ]]; then + echo "Too many long DROP operations: $DROP_WARNINGS (max allowed: $DROP_WARNINGS_THRESHOLD)" + ERROR=1 +fi + +# Check if a DROP operation took too long +MAX_DROP=$(grep -i "cycling idle.*drop" $OUTPUT | awk '{print $9}' | sort -n | tail -1) +if [[ $MAX_DROP -ge $MAX_DROP_THRESHOLD ]]; then + echo "A drop operation took too long: ${MAX_DROP}s (max allowed: ${MAX_DROP_THRESHOLD}s)" + ERROR=1 +fi + +# Check if too many READ/INSERT/UPDATE operations took too long +if [[ $OP_WARNINGS -ge $OP_WARNINGS_THRESHOLD ]]; then + echo "Too many long read/insert/update operations: $OP_WARNINGS (max allowed: $OP_WARNINGS_THRESHOLD)" + ERROR=1 +fi + +if [[ $ERROR -ne 0 ]]; then + echo FAILED + exit 1 +fi + +echo SUCCESS diff --git a/src/third_party/wiredtiger/bench/workgen/workgen.cxx b/src/third_party/wiredtiger/bench/workgen/workgen.cxx index cc2c93d3f6b..2f7c0c4f5c4 100644 --- a/src/third_party/wiredtiger/bench/workgen/workgen.cxx +++ b/src/third_party/wiredtiger/bench/workgen/workgen.cxx @@ -26,8 +26,8 @@ * OTHER DEALINGS IN THE SOFTWARE. */ -#define __STDC_LIMIT_MACROS // needed to get UINT64_MAX in C++ -#define __STDC_FORMAT_MACROS // needed to get PRIuXX macros in C++ +#define __STDC_LIMIT_MACROS // needed to get UINT64_MAX in C++ +#define __STDC_FORMAT_MACROS // needed to get PRIuXX macros in C++ #include <iomanip> #include <iostream> #include <fstream> @@ -52,55 +52,51 @@ extern "C" { #define LATENCY_MS_BUCKETS 1000 #define LATENCY_SEC_BUCKETS 100 -#define THROTTLE_PER_SEC 20 // times per sec we will throttle +#define THROTTLE_PER_SEC 20 // times per sec we will throttle -#define MIN(a, b) ((a) < (b) ? (a) : (b)) -#define MAX(a, b) ((a) < (b) ? (b) : (a)) -#define TIMESPEC_DOUBLE(ts) ((double)(ts).tv_sec + ts.tv_nsec * 0.000000001) -#define PCT(n, total) ((total) == 0 ? 0 : ((n) * 100) / (total)) -#define OPS_PER_SEC(ops, ts) (int) ((ts) == 0 ? 0.0 : \ - (ops) / TIMESPEC_DOUBLE(ts)) +#define MIN(a, b) ((a) < (b) ? (a) : (b)) +#define MAX(a, b) ((a) < (b) ? (b) : (a)) +#define TIMESPEC_DOUBLE(ts) ((double)(ts).tv_sec + ts.tv_nsec * 0.000000001) +#define PCT(n, total) ((total) == 0 ? 0 : ((n)*100) / (total)) +#define OPS_PER_SEC(ops, ts) (int)((ts) == 0 ? 0.0 : (ops) / TIMESPEC_DOUBLE(ts)) // Get the value of a STL container, even if it is not present -#define CONTAINER_VALUE(container, idx, dfault) \ +#define CONTAINER_VALUE(container, idx, dfault) \ (((container).count(idx) > 0) ? (container)[idx] : (dfault)) -#define CROSS_USAGE(a, b) \ - (((a & USAGE_READ) != 0 && (b & USAGE_WRITE) != 0) || \ - ((a & USAGE_WRITE) != 0 && (b & USAGE_READ) != 0)) - -#define ASSERT(cond) \ - do { \ - if (!(cond)) { \ - fprintf(stderr, "%s:%d: ASSERT failed: %s\n", \ - __FILE__, __LINE__, #cond); \ - abort(); \ - } \ - } while(0) - -#define THROW_ERRNO(e, args) \ - do { \ - std::stringstream __sstm; \ - __sstm << args; \ - WorkgenException __wge(e, __sstm.str().c_str()); \ - throw(__wge); \ - } while(0) - -#define THROW(args) THROW_ERRNO(0, args) - -#define VERBOSE(runner, args) \ - do { \ - if ((runner)._context->_verbose) \ - std::cout << args << std::endl; \ - } while(0) - -#define OP_HAS_VALUE(op) \ - ((op)->_optype == Operation::OP_INSERT || \ - (op)->_optype == Operation::OP_UPDATE) +#define CROSS_USAGE(a, b) \ + (((a & USAGE_READ) != 0 && (b & USAGE_WRITE) != 0) || \ + ((a & USAGE_WRITE) != 0 && (b & USAGE_READ) != 0)) + +#define ASSERT(cond) \ + do { \ + if (!(cond)) { \ + fprintf(stderr, "%s:%d: ASSERT failed: %s\n", __FILE__, __LINE__, #cond); \ + abort(); \ + } \ + } while (0) + +#define THROW_ERRNO(e, args) \ + do { \ + std::stringstream __sstm; \ + __sstm << args; \ + WorkgenException __wge(e, __sstm.str().c_str()); \ + throw(__wge); \ + } while (0) + +#define THROW(args) THROW_ERRNO(0, args) + +#define VERBOSE(runner, args) \ + do { \ + if ((runner)._context->_verbose) \ + std::cout << args << std::endl; \ + } while (0) + +#define OP_HAS_VALUE(op) \ + ((op)->_optype == Operation::OP_INSERT || (op)->_optype == Operation::OP_UPDATE) namespace workgen { - struct WorkloadRunnerConnection { WorkloadRunner *runner; WT_CONNECTION *connection; @@ -112,7 +108,9 @@ struct WorkloadRunnerConnection { // than one. static uint32_t context_count = 0; -static void *thread_runner_main(void *arg) { +static void * +thread_runner_main(void *arg) +{ ThreadRunner *runner = (ThreadRunner *)arg; try { runner->_errno = runner->run(); @@ -122,9 +120,11 @@ static void *thread_runner_main(void *arg) { return (NULL); } -static void *thread_workload(void *arg) { +static void * +thread_workload(void *arg) +{ - WorkloadRunnerConnection *runnerConnection = (WorkloadRunnerConnection *) arg; + WorkloadRunnerConnection *runnerConnection = (WorkloadRunnerConnection *)arg; WorkloadRunner *runner = runnerConnection->runner; WT_CONNECTION *connection = runnerConnection->connection; @@ -137,8 +137,10 @@ static void *thread_workload(void *arg) { return (NULL); } -static void *thread_idle_table_cycle_workload(void *arg) { - WorkloadRunnerConnection *runnerConnection = (WorkloadRunnerConnection *) arg; +static void * +thread_idle_table_cycle_workload(void *arg) +{ + WorkloadRunnerConnection *runnerConnection = (WorkloadRunnerConnection *)arg; WT_CONNECTION *connection = runnerConnection->connection; WorkloadRunner *runner = runnerConnection->runner; @@ -151,7 +153,9 @@ static void *thread_idle_table_cycle_workload(void *arg) { return (NULL); } -int WorkloadRunner::check_timing(const char *name, uint64_t last_interval) { +int +WorkloadRunner::check_timing(const char *name, uint64_t last_interval) +{ WorkloadOptions *options = &_workload->options; int msg_err; const char *str; @@ -165,12 +169,17 @@ int WorkloadRunner::check_timing(const char *name, uint64_t last_interval) { } else { str = "WARNING"; } - std::cerr << str << ": Cycling idle table failed because " << name << " took " << last_interval << " seconds which is longer than configured acceptable maximum of " << options->max_idle_table_cycle << std::endl; - } - return (msg_err); + std::cerr << str << ": Cycling idle table failed because " << name << " took " + << last_interval << " s which is longer than configured acceptable maximum of " + << options->max_idle_table_cycle << " s. Diff is " + << (last_interval - options->max_idle_table_cycle) << " s." << std::endl; + } + return (msg_err); } -int WorkloadRunner::start_table_idle_cycle(WT_CONNECTION *conn) { +int +WorkloadRunner::start_table_idle_cycle(WT_CONNECTION *conn) +{ WT_SESSION *session; WT_CURSOR *cursor; uint64_t start, stop, last_interval; @@ -182,7 +191,7 @@ int WorkloadRunner::start_table_idle_cycle(WT_CONNECTION *conn) { THROW("Error Opening a Session."); } - for (cycle_count = 0 ; !stopping ; ++cycle_count) { + for (cycle_count = 0; !stopping; ++cycle_count) { sprintf(uri, "table:test_cycle%04d", cycle_count); workgen_clock(&start); @@ -232,12 +241,13 @@ int WorkloadRunner::start_table_idle_cycle(WT_CONNECTION *conn) { * This function will sleep for "timestamp_advance" seconds, increment and set oldest_timestamp, * stable_timestamp with the specified lag until stopping is set to true */ -int WorkloadRunner::increment_timestamp(WT_CONNECTION *conn) { +int +WorkloadRunner::increment_timestamp(WT_CONNECTION *conn) +{ uint64_t time_us; char buf[BUF_SIZE]; - while (!stopping) - { + while (!stopping) { if (_workload->options.oldest_timestamp_lag > 0) { time_us = WorkgenTimeStamp::get_timestamp_lag(_workload->options.oldest_timestamp_lag); sprintf(buf, "oldest_timestamp=%" PRIu64, time_us); @@ -255,7 +265,9 @@ int WorkloadRunner::increment_timestamp(WT_CONNECTION *conn) { return 0; } -static void *monitor_main(void *arg) { +static void * +monitor_main(void *arg) +{ Monitor *monitor = (Monitor *)arg; try { monitor->_errno = monitor->run(); @@ -268,7 +280,9 @@ static void *monitor_main(void *arg) { // Exponentiate (like the pow function), except that it returns an exact // integral 64 bit value, and if it overflows, returns the maximum possible // value for the return type. -static uint64_t power64(int base, int exp) { +static uint64_t +power64(int base, int exp) +{ uint64_t last, result; result = 1; @@ -282,38 +296,42 @@ static uint64_t power64(int base, int exp) { } OptionsList::OptionsList() : _option_map() {} -OptionsList::OptionsList(const OptionsList &other) : - _option_map(other._option_map) {} +OptionsList::OptionsList(const OptionsList &other) : _option_map(other._option_map) {} -void OptionsList::add_option(const char *name, const std::string typestr, - const char *desc) { +void +OptionsList::add_option(const char *name, const std::string typestr, const char *desc) +{ TypeDescPair pair(typestr, desc); _option_map[name] = pair; } -void OptionsList::add_int(const char *name, int default_value, - const char *desc) { +void +OptionsList::add_int(const char *name, int default_value, const char *desc) +{ std::stringstream sstm; sstm << "int, default=" << default_value; add_option(name, sstm.str(), desc); } -void OptionsList::add_bool(const char *name, bool default_value, - const char *desc) { +void +OptionsList::add_bool(const char *name, bool default_value, const char *desc) +{ std::stringstream sstm; sstm << "boolean, default=" << (default_value ? "true" : "false"); add_option(name, sstm.str(), desc); } -void OptionsList::add_double(const char *name, double default_value, - const char *desc) { +void +OptionsList::add_double(const char *name, double default_value, const char *desc) +{ std::stringstream sstm; sstm << "double, default=" << default_value; add_option(name, sstm.str(), desc); } -void OptionsList::add_string(const char *name, - const std::string &default_value, const char *desc) { +void +OptionsList::add_string(const char *name, const std::string &default_value, const char *desc) +{ std::stringstream sstm; sstm << "string, default=\"" << default_value << "\""; add_option(name, sstm.str(), desc); @@ -329,7 +347,7 @@ pretty_print(const char *p, const char *indent, std::stringstream &sstm) break; for (t = p + 70; t > p && *t != ' '; --t) ; - if (t == p) /* No spaces? */ + if (t == p) /* No spaces? */ break; if (indent != NULL) sstm << indent; @@ -343,17 +361,21 @@ pretty_print(const char *p, const char *indent, std::stringstream &sstm) } } -std::string OptionsList::help() const { +std::string +OptionsList::help() const +{ std::stringstream sstm; - for (std::map<std::string, TypeDescPair>::const_iterator i = - _option_map.begin(); i != _option_map.end(); i++) { + for (std::map<std::string, TypeDescPair>::const_iterator i = _option_map.begin(); + i != _option_map.end(); i++) { sstm << i->first << " (" << i->second.first << ")" << std::endl; pretty_print(i->second.second.c_str(), "\t", sstm); } return sstm.str(); } -std::string OptionsList::help_description(const char *option_name) const { +std::string +OptionsList::help_description(const char *option_name) const +{ const std::string key(option_name); if (_option_map.count(key) == 0) return (std::string("")); @@ -361,7 +383,9 @@ std::string OptionsList::help_description(const char *option_name) const { return (_option_map.find(key)->second.second); } -std::string OptionsList::help_type(const char *option_name) const { +std::string +OptionsList::help_type(const char *option_name) const +{ const std::string key(option_name); if (_option_map.count(key) == 0) return std::string(""); @@ -370,28 +394,37 @@ std::string OptionsList::help_type(const char *option_name) const { } Context::Context() : _verbose(false), _internal(new ContextInternal()) {} -Context::~Context() { delete _internal; } -Context& Context::operator=(const Context &other) { +Context::~Context() +{ + delete _internal; +} +Context & +Context::operator=(const Context &other) +{ _verbose = other._verbose; *_internal = *other._internal; return (*this); } -ContextInternal::ContextInternal() : _tint(), _table_names(), - _table_runtime(NULL), _runtime_alloced(0), _tint_last(0), - _context_count(0) { +ContextInternal::ContextInternal() + : _tint(), _table_names(), _table_runtime(NULL), _runtime_alloced(0), _tint_last(0), + _context_count(0) +{ uint32_t count; if ((count = workgen_atomic_add32(&context_count, 1)) != 1) THROW("multiple Contexts not supported"); _context_count = count; } -ContextInternal::~ContextInternal() { +ContextInternal::~ContextInternal() +{ if (_table_runtime != NULL) delete _table_runtime; } -int ContextInternal::create_all() { +int +ContextInternal::create_all() +{ if (_runtime_alloced < _tint_last) { // The array references are 1-based, we'll waste one entry. TableRuntime *new_table_runtime = new TableRuntime[_tint_last + 1]; @@ -404,53 +437,45 @@ int ContextInternal::create_all() { return (0); } -Monitor::Monitor(WorkloadRunner &wrunner) : - _errno(0), _exception(), _wrunner(wrunner), _stop(false), _handle(), - _out(NULL), _json(NULL) {} +Monitor::Monitor(WorkloadRunner &wrunner) + : _errno(0), _exception(), _wrunner(wrunner), _stop(false), _handle(), _out(NULL), _json(NULL) +{ +} Monitor::~Monitor() {} -int Monitor::run() { +int +Monitor::run() +{ struct timespec t; struct tm *tm, _tm; - char time_buf[64], version[100]; + char version[100]; Stats prev_totals; WorkloadOptions *options = &_wrunner._workload->options; - uint64_t latency_max = (uint64_t)options->max_latency; - size_t buf_size; - bool first; + uint64_t latency_max = (uint64_t)options->max_latency * THOUSAND; + bool first_iteration; - (*_out) << "#time," - << "totalsec," - << "read ops per second," - << "insert ops per second," - << "update ops per second," - << "checkpoints," - << "read average latency(uS)," - << "read minimum latency(uS)," - << "read maximum latency(uS)," - << "insert average latency(uS)," - << "insert min latency(uS)," - << "insert maximum latency(uS)," - << "update average latency(uS)," - << "update min latency(uS)," - << "update maximum latency(uS)" - << std::endl; + // Format header of the table in _out stream. + if (_out != NULL) + _format_out_header(); - first = true; + first_iteration = true; workgen_version(version, sizeof(version)); Stats prev_interval; // The whole and fractional part of sample_interval are separated, // we don't want to sleep longer than a second. int sample_secs = ms_to_sec(options->sample_interval_ms); - useconds_t sample_usecs = - ms_to_us(options->sample_interval_ms) - sec_to_us(sample_secs); + useconds_t sample_usecs = ms_to_us(options->sample_interval_ms) - sec_to_us(sample_secs); + + // Format JSON prefix. + if (_json != NULL) + _format_json_prefix(version); while (!_stop) { int waitsecs; useconds_t waitusecs; - if (first && options->warmup > 0) { + if (first_iteration && options->warmup > 0) { waitsecs = options->warmup; waitusecs = 0; } else { @@ -468,136 +493,199 @@ int Monitor::run() { workgen_epoch(&t); tm = localtime_r(&t.tv_sec, &_tm); - (void)strftime(time_buf, sizeof(time_buf), "%b %d %H:%M:%S", tm); Stats new_totals(true); - for (std::vector<ThreadRunner>::iterator tr = - _wrunner._trunners.begin(); tr != _wrunner._trunners.end(); tr++) + for (std::vector<ThreadRunner>::iterator tr = _wrunner._trunners.begin(); + tr != _wrunner._trunners.end(); tr++) new_totals.add(tr->_stats, true); Stats interval(new_totals); interval.subtract(prev_totals); + bool checkpointing = + new_totals.checkpoint.ops_in_progress > 0 || interval.checkpoint.ops > 0; double interval_secs = options->sample_interval_ms / 1000.0; - uint64_t cur_reads = (uint64_t)(interval.read.ops / interval_secs); - uint64_t cur_inserts = (uint64_t)(interval.insert.ops / interval_secs); - uint64_t cur_updates = (uint64_t)(interval.update.ops / interval_secs); - bool checkpointing = new_totals.checkpoint.ops_in_progress > 0 || - interval.checkpoint.ops > 0; - - uint64_t totalsec = ts_sec(t - _wrunner._start); - (*_out) << time_buf - << "," << totalsec - << "," << cur_reads - << "," << cur_inserts - << "," << cur_updates - << "," << (checkpointing ? 'Y' : 'N') - << "," << interval.read.average_latency() - << "," << interval.read.min_latency - << "," << interval.read.max_latency - << "," << interval.insert.average_latency() - << "," << interval.insert.min_latency - << "," << interval.insert.max_latency - << "," << interval.update.average_latency() - << "," << interval.update.min_latency - << "," << interval.update.max_latency - << std::endl; - - if (_json != NULL) { -#define WORKGEN_TIMESTAMP_JSON "%Y-%m-%dT%H:%M:%S" - buf_size = strftime(time_buf, sizeof(time_buf), - WORKGEN_TIMESTAMP_JSON, tm); - ASSERT(buf_size <= sizeof(time_buf)); - snprintf(&time_buf[buf_size], sizeof(time_buf) - buf_size, - ".%3.3" PRIu64 "Z", (uint64_t)ns_to_ms(t.tv_nsec)); - - // Note: we could allow this to be configurable. - int percentiles[4] = {50, 95, 99, 0}; - -#define TRACK_JSON(f, name, t, percentiles, extra) \ - do { \ - int _i; \ - (f) << "\"" << (name) << "\":{" << extra \ - << "\"ops per sec\":" \ - << (uint64_t)((t).ops / interval_secs) \ - << ",\"rollbacks\":" << ((t).rollbacks) \ - << ",\"average latency\":" << (t).average_latency() \ - << ",\"min latency\":" << (t).min_latency \ - << ",\"max latency\":" << (t).max_latency; \ - for (_i = 0; (percentiles)[_i] != 0; _i++) \ - (f) << ",\"" << (percentiles)[_i] << "% latency\":" \ - << (t).percentile_latency(percentiles[_i]); \ - (f) << "}"; \ - } while(0) - - (*_json) << "{"; - if (first) { - (*_json) << "\"version\":\"" << version << "\","; - first = false; - } - (*_json) << "\"localTime\":\"" << time_buf - << "\",\"workgen\":{"; - TRACK_JSON(*_json, "read", interval.read, percentiles, ""); - (*_json) << ","; - TRACK_JSON(*_json, "insert", interval.insert, percentiles, ""); - (*_json) << ","; - TRACK_JSON(*_json, "update", interval.update, percentiles, ""); - (*_json) << ","; - TRACK_JSON(*_json, "checkpoint", interval.checkpoint, percentiles, - "\"active\":" << (checkpointing ? "1," : "0,")); - (*_json) << "}}" << std::endl; - } - uint64_t read_max = interval.read.max_latency; - uint64_t insert_max = interval.read.max_latency; - uint64_t update_max = interval.read.max_latency; - - if (latency_max != 0 && - (read_max > latency_max || insert_max > latency_max || - update_max > latency_max)) { - std::cerr << "WARNING: max latency exceeded:" - << " threshold " << latency_max - << " read max " << read_max - << " insert max " << insert_max - << " update max " << update_max << std::endl; - } + // Format entry into _out stream. + if (_out != NULL) + _format_out_entry(interval, interval_secs, t, checkpointing, *tm); + + // Format entry into _json stream. + if (_json != NULL) + _format_json_entry(*tm, t, first_iteration, interval, checkpointing, interval_secs); + + // Check latency threshold. Write warning into std::cerr in case read, insert or update + // exceeds latency_max. + _check_latency_threshold(interval, latency_max); prev_interval.assign(interval); prev_totals.assign(new_totals); + + first_iteration = false; } + + // Format JSON suffix. + if (_json != NULL) + _format_json_suffix(); + return (0); } +void +Monitor::_format_out_header() +{ + (*_out) << "#time," + << "totalsec," + << "read ops per second," + << "insert ops per second," + << "update ops per second," + << "checkpoints," + << "read average latency(uS)," + << "read minimum latency(uS)," + << "read maximum latency(uS)," + << "insert average latency(uS)," + << "insert min latency(uS)," + << "insert maximum latency(uS)," + << "update average latency(uS)," + << "update min latency(uS)," + << "update maximum latency(uS)" << std::endl; +} + +void +Monitor::_format_out_entry(const Stats &interval, double interval_secs, const timespec ×pec, + bool checkpointing, const tm &tm) +{ + char time_buf[64]; + uint64_t cur_reads = (uint64_t)(interval.read.ops / interval_secs); + uint64_t cur_inserts = (uint64_t)(interval.insert.ops / interval_secs); + uint64_t cur_updates = (uint64_t)(interval.update.ops / interval_secs); + uint64_t totalsec = ts_sec(timespec - _wrunner._start); + + (void)strftime(time_buf, sizeof(time_buf), "%b %d %H:%M:%S", &tm); + (*_out) << time_buf << "," << totalsec << "," << cur_reads << "," << cur_inserts << "," + << cur_updates << "," << (checkpointing ? 'Y' : 'N') << "," + << interval.read.average_latency() << "," << interval.read.min_latency << "," + << interval.read.max_latency << "," << interval.insert.average_latency() << "," + << interval.insert.min_latency << "," << interval.insert.max_latency << "," + << interval.update.average_latency() << "," << interval.update.min_latency << "," + << interval.update.max_latency << std::endl; +} + +void +Monitor::_format_json_prefix(const std::string &version) +{ + (*_json) << "{"; + (*_json) << "\"version\":\"" << version << "\","; + (*_json) << "\"workgen\":["; +} + +void +Monitor::_format_json_entry(const tm &tm, const timespec ×pec, bool first_iteration, + const Stats &interval, bool checkpointing, double interval_secs) +{ +#define WORKGEN_TIMESTAMP_JSON "%Y-%m-%dT%H:%M:%S" +#define TRACK_JSON(f, name, t, percentiles, extra) \ + do { \ + int _i; \ + (f) << "\"" << (name) << "\":{" << extra \ + << "\"ops per sec\":" << (uint64_t)((t).ops / interval_secs) \ + << ",\"rollbacks\":" << ((t).rollbacks) \ + << ",\"average latency\":" << (t).average_latency() \ + << ",\"min latency\":" << (t).min_latency << ",\"max latency\":" << (t).max_latency; \ + for (_i = 0; (percentiles)[_i] != 0; _i++) \ + (f) << ",\"" << (percentiles)[_i] \ + << "% latency\":" << (t).percentile_latency(percentiles[_i]); \ + (f) << "}"; \ + } while (0) + + // Note: we could allow this to be configurable. + int percentiles[4] = {50, 95, 99, 0}; + size_t buf_size; + char time_buf[64]; + + buf_size = strftime(time_buf, sizeof(time_buf), WORKGEN_TIMESTAMP_JSON, &tm); + ASSERT(buf_size <= sizeof(time_buf)); + snprintf(&time_buf[buf_size], sizeof(time_buf) - buf_size, ".%3.3" PRIu64 "Z", + (uint64_t)ns_to_ms(timespec.tv_nsec)); + + if (!first_iteration) + (*_json) << ","; + + (*_json) << "{"; + (*_json) << "\"localTime\":\"" << time_buf << "\","; + TRACK_JSON(*_json, "read", interval.read, percentiles, ""); + (*_json) << ","; + TRACK_JSON(*_json, "insert", interval.insert, percentiles, ""); + (*_json) << ","; + TRACK_JSON(*_json, "update", interval.update, percentiles, ""); + (*_json) << ","; + TRACK_JSON(*_json, "checkpoint", interval.checkpoint, percentiles, + "\"active\":" << (checkpointing ? "1," : "0,")); + (*_json) << "}" << std::endl; +} + +void +Monitor::_format_json_suffix() +{ + (*_json) << "]}" << std::endl; +} + +void +Monitor::_check_latency_threshold(const Stats &interval, uint64_t latency_max) +{ + uint64_t read_max = interval.read.max_latency; + uint64_t insert_max = interval.insert.max_latency; + uint64_t update_max = interval.update.max_latency; + + if (read_max > latency_max) + std::cerr << "WARNING: max latency exceeded for read operation. Threshold " << latency_max + << " us, recorded " << read_max << " us, diff " << (read_max - latency_max) + << " us." << std::endl; + if (insert_max > latency_max) + std::cerr << "WARNING: max latency exceeded for insert operation. Threshold " << latency_max + << " us, recorded " << insert_max << " us, diff " << (insert_max - latency_max) + << " us." << std::endl; + if (update_max > latency_max) + std::cerr << "WARNING: max latency exceeded for update operation. Threshold " << latency_max + << " us, recorded " << insert_max << " us, diff " << (update_max - latency_max) + << " us." << std::endl; +} + ParetoOptions ParetoOptions::DEFAULT; -ParetoOptions::ParetoOptions(int param_arg) : param(param_arg), range_low(0.0), - range_high(1.0), _options() { +ParetoOptions::ParetoOptions(int param_arg) + : param(param_arg), range_low(0.0), range_high(1.0), _options() +{ _options.add_int("param", param, "0 is disabled, otherwise a range from 1 (most aggressive) to " "100 (least aggressive)"); - _options.add_double("range_low", range_low, - "between 0.0 and 1.0, starting range of the pareto distribution"); - _options.add_double("range_high", range_high, - "between 0.0 and 1.0, ending range of the pareto distribution"); -} -ParetoOptions::ParetoOptions(const ParetoOptions &other) : - param(other.param), range_low(other.range_low), - range_high(other.range_high), _options(other._options) {} + _options.add_double( + "range_low", range_low, "between 0.0 and 1.0, starting range of the pareto distribution"); + _options.add_double( + "range_high", range_high, "between 0.0 and 1.0, ending range of the pareto distribution"); +} +ParetoOptions::ParetoOptions(const ParetoOptions &other) + : param(other.param), range_low(other.range_low), range_high(other.range_high), + _options(other._options) +{ +} ParetoOptions::~ParetoOptions() {} -ThreadRunner::ThreadRunner() : - _errno(0), _exception(), _thread(NULL), _context(NULL), _icontext(NULL), - _workload(NULL), _wrunner(NULL), _rand_state(NULL), - _throttle(NULL), _throttle_ops(0), _throttle_limit(0), - _in_transaction(false), _start_time_us(0), _op_time_us(0), - _number(0), _stats(false), _table_usage(), - _cursors(NULL), _stop(false), _session(NULL), _keybuf(NULL), - _valuebuf(NULL), _repeat(false) { +ThreadRunner::ThreadRunner() + : _errno(0), _exception(), _thread(NULL), _context(NULL), _icontext(NULL), _workload(NULL), + _wrunner(NULL), _rand_state(NULL), _throttle(NULL), _throttle_ops(0), _throttle_limit(0), + _in_transaction(false), _start_time_us(0), _op_time_us(0), _number(0), _stats(false), + _table_usage(), _cursors(NULL), _stop(false), _session(NULL), _keybuf(NULL), _valuebuf(NULL), + _repeat(false) +{ } -ThreadRunner::~ThreadRunner() { +ThreadRunner::~ThreadRunner() +{ free_all(); } -int ThreadRunner::create_all(WT_CONNECTION *conn) { +int +ThreadRunner::create_all(WT_CONNECTION *conn) +{ size_t keysize, valuesize; WT_RET(close_all()); @@ -621,23 +709,26 @@ int ThreadRunner::create_all(WT_CONNECTION *conn) { return (0); } -int ThreadRunner::open_all() { +int +ThreadRunner::open_all() +{ typedef WT_CURSOR *WT_CURSOR_PTR; if (_cursors != NULL) delete _cursors; _cursors = new WT_CURSOR_PTR[_icontext->_tint_last + 1]; - memset(_cursors, 0, sizeof (WT_CURSOR *) * (_icontext->_tint_last + 1)); - for (std::map<uint32_t, uint32_t>::iterator i = _table_usage.begin(); - i != _table_usage.end(); i++) { + memset(_cursors, 0, sizeof(WT_CURSOR *) * (_icontext->_tint_last + 1)); + for (std::map<uint32_t, uint32_t>::iterator i = _table_usage.begin(); i != _table_usage.end(); + i++) { uint32_t tindex = i->first; const char *uri = _icontext->_table_names[tindex].c_str(); - WT_RET(_session->open_cursor(_session, uri, NULL, NULL, - &_cursors[tindex])); + WT_RET(_session->open_cursor(_session, uri, NULL, NULL, &_cursors[tindex])); } return (0); } -int ThreadRunner::close_all() { +int +ThreadRunner::close_all() +{ if (_throttle != NULL) { delete _throttle; _throttle = NULL; @@ -650,7 +741,9 @@ int ThreadRunner::close_all() { return (0); } -void ThreadRunner::free_all() { +void +ThreadRunner::free_all() +{ if (_rand_state != NULL) { workgen_random_free(_rand_state); _rand_state = NULL; @@ -669,14 +762,15 @@ void ThreadRunner::free_all() { } } -int ThreadRunner::cross_check(std::vector<ThreadRunner> &runners) { +int +ThreadRunner::cross_check(std::vector<ThreadRunner> &runners) +{ std::map<uint32_t, uint32_t> usage; // Determine which tables have cross usage - for (std::vector<ThreadRunner>::iterator r = runners.begin(); - r != runners.end(); r++) { + for (std::vector<ThreadRunner>::iterator r = runners.begin(); r != runners.end(); r++) { for (std::map<uint32_t, uint32_t>::iterator i = r->_table_usage.begin(); - i != r->_table_usage.end(); i++) { + i != r->_table_usage.end(); i++) { uint32_t tindex = i->first; uint32_t thisusage = i->second; uint32_t curusage = CONTAINER_VALUE(usage, tindex, 0); @@ -685,11 +779,9 @@ int ThreadRunner::cross_check(std::vector<ThreadRunner> &runners) { usage[tindex] = curusage; } } - for (std::map<uint32_t, uint32_t>::iterator i = usage.begin(); - i != usage.end(); i++) { + for (std::map<uint32_t, uint32_t>::iterator i = usage.begin(); i != usage.end(); i++) { if ((i->second & USAGE_MIXED) != 0) { - for (std::vector<ThreadRunner>::iterator r = runners.begin(); - r != runners.end(); r++) { + for (std::vector<ThreadRunner>::iterator r = runners.begin(); r != runners.end(); r++) { r->_table_usage[i->first] |= USAGE_MIXED; } } @@ -697,7 +789,9 @@ int ThreadRunner::cross_check(std::vector<ThreadRunner> &runners) { return (0); } -int ThreadRunner::run() { +int +ThreadRunner::run() +{ WT_DECL_RET; ThreadOptions *options = &_thread->options; std::string name = options->name; @@ -709,20 +803,18 @@ int ThreadRunner::run() { VERBOSE(*this, "thread " << name << " running"); if (options->throttle != 0) { - _throttle = new Throttle(*this, options->throttle, - options->throttle_burst); + _throttle = new Throttle(*this, options->throttle, options->throttle_burst); } for (int cnt = 0; !_stop && (_repeat || cnt < 1) && ret == 0; cnt++) WT_ERR(op_run(&_thread->_op)); -err: +err : #ifdef _DEBUG - { - std::string messages = this->get_debug(); - if (!messages.empty()) - std::cerr << "DEBUG (thread " << name << "): " - << messages << std::endl; - } +{ + std::string messages = this->get_debug(); + if (!messages.empty()) + std::cerr << "DEBUG (thread " << name << "): " << messages << std::endl; +} #endif if (ret != 0) std::cerr << "thread " << name << " failed err=" << ret << std::endl; @@ -730,12 +822,15 @@ err: return (ret); } -void ThreadRunner::get_static_counts(Stats &stats) { +void +ThreadRunner::get_static_counts(Stats &stats) +{ _thread->_op.get_static_counts(stats, 1); } -void ThreadRunner::op_create_all(Operation *op, size_t &keysize, - size_t &valuesize) { +void +ThreadRunner::op_create_all(Operation *op, size_t &keysize, size_t &valuesize) +{ tint_t tint; op->create_all(); @@ -743,8 +838,7 @@ void ThreadRunner::op_create_all(Operation *op, size_t &keysize, op->kv_compute_max(true, false); if (OP_HAS_VALUE(op)) op->kv_compute_max(false, op->_table.options.random_value); - if (op->_key._keytype == Key::KEYGEN_PARETO && - op->_key._pareto.param == 0) + if (op->_key._keytype == Key::KEYGEN_PARETO && op->_key._pareto.param == 0) THROW("Key._pareto value must be set if KEYGEN_PARETO specified"); op->kv_size_buffer(true, keysize); op->kv_size_buffer(false, valuesize); @@ -768,8 +862,7 @@ void ThreadRunner::op_create_all(Operation *op, size_t &keysize, tint = _icontext->_tint[uri]; op->_table._internal->_tint = tint; } - uint32_t usage_flags = CONTAINER_VALUE(_table_usage, - op->_table._internal->_tint, 0); + uint32_t usage_flags = CONTAINER_VALUE(_table_usage, op->_table._internal->_tint, 0); if (op->_optype == Operation::OP_SEARCH) usage_flags |= ThreadRunner::USAGE_READ; else @@ -777,36 +870,33 @@ void ThreadRunner::op_create_all(Operation *op, size_t &keysize, _table_usage[op->_table._internal->_tint] = usage_flags; } if (op->_group != NULL) - for (std::vector<Operation>::iterator i = op->_group->begin(); - i != op->_group->end(); i++) + for (std::vector<Operation>::iterator i = op->_group->begin(); i != op->_group->end(); i++) op_create_all(&*i, keysize, valuesize); } - -#define PARETO_SHAPE 1.5 +#define PARETO_SHAPE 1.5 // Return a value within the interval [ 0, recno_max ) // that is weighted toward lower numbers with pareto_param at 0 (the minimum), // and more evenly distributed with pareto_param at 100 (the maximum). // static uint64_t -pareto_calculation(uint32_t randint, uint64_t recno_max, - ParetoOptions &pareto) { +pareto_calculation(uint32_t randint, uint64_t recno_max, ParetoOptions &pareto) +{ double S1, S2, U; uint32_t result; double r; r = (double)randint; if (pareto.range_high != 1.0 || pareto.range_low != 0.0) { - if (pareto.range_high <= pareto.range_low || - pareto.range_high > 1.0 || pareto.range_low < 0.0) + if (pareto.range_high <= pareto.range_low || pareto.range_high > 1.0 || + pareto.range_low < 0.0) THROW("Pareto illegal range"); - r = (pareto.range_low * (double)UINT32_MAX) + - r * (pareto.range_high - pareto.range_low); + r = (pareto.range_low * (double)UINT32_MAX) + r * (pareto.range_high - pareto.range_low); } S1 = (-1 / PARETO_SHAPE); S2 = recno_max * (pareto.param / 100.0) * (PARETO_SHAPE - 1); - U = 1 - r / (double)UINT32_MAX; // interval [0, 1) + U = 1 - r / (double)UINT32_MAX; // interval [0, 1) result = (uint64_t)((pow(U, S1) - 1) * S2); // This Pareto calculation chooses out of range values less than 20% @@ -825,8 +915,9 @@ pareto_calculation(uint32_t randint, uint64_t recno_max, return (result); } -uint64_t ThreadRunner::op_get_key_recno(Operation *op, uint64_t range, - tint_t tint) { +uint64_t +ThreadRunner::op_get_key_recno(Operation *op, uint64_t range, tint_t tint) +{ uint64_t recno_count; uint32_t rval; @@ -841,10 +932,12 @@ uint64_t ThreadRunner::op_get_key_recno(Operation *op, uint64_t range, rval = random_value(); if (op->_key._keytype == Key::KEYGEN_PARETO) rval = pareto_calculation(rval, recno_count, op->_key._pareto); - return (rval % recno_count + 1); // recnos are one-based. + return (rval % recno_count + 1); // recnos are one-based. } -int ThreadRunner::op_run(Operation *op) { +int +ThreadRunner::op_run(Operation *op) +{ Track *track; tint_t tint = op->_table._internal->_tint; WT_CURSOR *cursor; @@ -892,10 +985,8 @@ int ThreadRunner::op_run(Operation *op) { break; case Operation::OP_INSERT: track = &_stats.insert; - if (op->_key._keytype == Key::KEYGEN_APPEND || - op->_key._keytype == Key::KEYGEN_AUTO) - recno = workgen_atomic_add64( - &_icontext->_table_runtime[tint]._max_recno, 1); + if (op->_key._keytype == Key::KEYGEN_APPEND || op->_key._keytype == Key::KEYGEN_AUTO) + recno = workgen_atomic_add64(&_icontext->_table_runtime[tint]._max_recno, 1); else recno = op_get_key_recno(op, range, tint); break; @@ -921,14 +1012,12 @@ int ThreadRunner::op_run(Operation *op) { break; } if ((op->_internal->_flags & WORKGEN_OP_REOPEN) != 0) { - WT_ERR(_session->open_cursor(_session, op->_table._uri.c_str(), NULL, - NULL, &cursor)); + WT_ERR(_session->open_cursor(_session, op->_table._uri.c_str(), NULL, NULL, &cursor)); own_cursor = true; } else cursor = _cursors[tint]; - measure_latency = track != NULL && track->ops != 0 && - track->track_latency() && + measure_latency = track != NULL && track->ops != 0 && track->track_latency() && (track->ops % _workload->options.sample_rate == 0); VERBOSE(*this, "OP " << op->_optype << " " << op->_table._uri.c_str() << ", recno=" << recno); @@ -947,8 +1036,8 @@ int ThreadRunner::op_run(Operation *op) { op->kv_gen(this, true, 100, recno, _keybuf); cursor->set_key(cursor, _keybuf); if (OP_HAS_VALUE(op)) { - uint64_t compressibility = op->_table.options.random_value ? - 0 : op->_table.options.value_compressibility; + uint64_t compressibility = + op->_table.options.random_value ? 0 : op->_table.options.value_compressibility; op->kv_gen(this, false, compressibility, recno, _valuebuf); cursor->set_value(cursor, _valuebuf); } @@ -958,15 +1047,14 @@ int ThreadRunner::op_run(Operation *op) { if (op->transaction != NULL) { if (_in_transaction) THROW("nested transactions not supported"); - if (op->transaction->use_commit_timestamp && op->transaction->use_prepare_timestamp) - { + if (op->transaction->use_commit_timestamp && op->transaction->use_prepare_timestamp) { THROW("Either use_prepare_timestamp or use_commit_timestamp must be set."); } if (op->transaction->read_timestamp_lag > 0) { - uint64_t read = WorkgenTimeStamp::get_timestamp_lag(op->transaction->read_timestamp_lag); + uint64_t read = + WorkgenTimeStamp::get_timestamp_lag(op->transaction->read_timestamp_lag); sprintf(buf, "%s=%" PRIu64, op->transaction->_begin_config.c_str(), read); - } - else { + } else { sprintf(buf, "%s", op->transaction->_begin_config.c_str()); } WT_ERR(_session->begin_transaction(_session, buf)); @@ -1033,8 +1121,8 @@ int ThreadRunner::op_run(Operation *op) { if (op->_timed != 0.0) endtime = _op_time_us + secs_us(op->_timed); - VERBOSE(*this, "GROUP operation " << op->_timed << " secs, " - << op->_repeatgroup << "times"); + VERBOSE( + *this, "GROUP operation " << op->_timed << " secs, " << op->_repeatgroup << "times"); do { for (int count = 0; !_stop && count < op->_repeatgroup; count++) { @@ -1060,17 +1148,16 @@ err: time_us = WorkgenTimeStamp::get_timestamp(); sprintf(buf, "prepare_timestamp=%" PRIu64, time_us); ret = _session->prepare_transaction(_session, buf); - sprintf(buf, "commit_timestamp=%" PRIu64 ",durable_timestamp=%" PRIu64, time_us, time_us); + sprintf( + buf, "commit_timestamp=%" PRIu64 ",durable_timestamp=%" PRIu64, time_us, time_us); ret = _session->commit_transaction(_session, buf); - } - else if (op->transaction->use_commit_timestamp) { + } else if (op->transaction->use_commit_timestamp) { uint64_t commit_time_us = WorkgenTimeStamp::get_timestamp(); sprintf(buf, "commit_timestamp=%" PRIu64, commit_time_us); - ret = _session->commit_transaction(_session, buf); - } - else { - ret = _session->commit_transaction(_session, - op->transaction->_commit_config.c_str()); + ret = _session->commit_transaction(_session, buf); + } else { + ret = + _session->commit_transaction(_session, op->transaction->_commit_config.c_str()); } } _in_transaction = false; @@ -1079,28 +1166,34 @@ err: } #ifdef _DEBUG -std::string ThreadRunner::get_debug() { +std::string +ThreadRunner::get_debug() +{ return (_debug_messages.str()); } #endif -uint32_t ThreadRunner::random_value() { +uint32_t +ThreadRunner::random_value() +{ return (workgen_random(_rand_state)); } // Generate a random 32-bit value then return a float value equally distributed // between -1.0 and 1.0. -float ThreadRunner::random_signed() { +float +ThreadRunner::random_signed() +{ uint32_t r = random_value(); int sign = ((r & 0x1) == 0 ? 1 : -1); return (((float)r * sign) / UINT32_MAX); } -Throttle::Throttle(ThreadRunner &runner, double throttle, - double throttle_burst) : _runner(runner), _throttle(throttle), - _burst(throttle_burst), _next_div(), _ops_delta(0), _ops_prev(0), - _ops_per_div(0), _ms_per_div(0), _ops_left_this_second(throttle), - _div_pos(0), _started(false) { +Throttle::Throttle(ThreadRunner &runner, double throttle, double throttle_burst) + : _runner(runner), _throttle(throttle), _burst(throttle_burst), _next_div(), _ops_delta(0), + _ops_prev(0), _ops_per_div(0), _ms_per_div(0), _ops_left_this_second(throttle), _div_pos(0), + _started(false) +{ // Our throttling is done by dividing each second into THROTTLE_PER_SEC // parts (we call the parts divisions). In each division, we perform @@ -1109,7 +1202,7 @@ Throttle::Throttle(ThreadRunner &runner, double throttle, // a multiple of THROTTLE_PER_SEC, nor is it even necessarily an integer. // (That way we can have 1000 threads each inserting 0.5 a second). ts_clear(_next_div); - ASSERT(1000 % THROTTLE_PER_SEC == 0); // must evenly divide + ASSERT(1000 % THROTTLE_PER_SEC == 0); // must evenly divide _ms_per_div = 1000 / THROTTLE_PER_SEC; _ops_per_div = (uint64_t)ceill(_throttle / THROTTLE_PER_SEC); } @@ -1130,7 +1223,9 @@ Throttle::~Throttle() {} // greater). This has the effect of randomizing how much clumping happens, and // ensures that multiple threads aren't executing in lock step. // -int Throttle::throttle(uint64_t op_count, uint64_t *op_limit) { +int +Throttle::throttle(uint64_t op_count, uint64_t *op_limit) +{ uint64_t ops; int64_t sleep_ms; timespec now; @@ -1187,36 +1282,42 @@ int Throttle::throttle(uint64_t op_count, uint64_t *op_limit) { return (0); } -ThreadOptions::ThreadOptions() : name(), session_config(), throttle(0.0), throttle_burst(1.0), - synchronized(false), _options() { +ThreadOptions::ThreadOptions() + : name(), session_config(), throttle(0.0), throttle_burst(1.0), synchronized(false), _options() +{ _options.add_string("name", name, "name of the thread"); - _options.add_string("session_config", session_config, "session config which is passed to open_session"); - _options.add_double("throttle", throttle, - "Limit to this number of operations per second"); + _options.add_string( + "session_config", session_config, "session config which is passed to open_session"); + _options.add_double("throttle", throttle, "Limit to this number of operations per second"); _options.add_double("throttle_burst", throttle_burst, "Changes characteristic of throttling from smooth (0.0) " "to having large bursts with lulls (10.0 or larger)"); } -ThreadOptions::ThreadOptions(const ThreadOptions &other) : - name(other.name), session_config(other.session_config), throttle(other.throttle), - throttle_burst(other.throttle_burst), synchronized(other.synchronized), - _options(other._options) {} +ThreadOptions::ThreadOptions(const ThreadOptions &other) + : name(other.name), session_config(other.session_config), throttle(other.throttle), + throttle_burst(other.throttle_burst), synchronized(other.synchronized), + _options(other._options) +{ +} ThreadOptions::~ThreadOptions() {} void -ThreadListWrapper::extend(const ThreadListWrapper &other) { - for (std::vector<Thread>::const_iterator i = other._threads.begin(); - i != other._threads.end(); i++) +ThreadListWrapper::extend(const ThreadListWrapper &other) +{ + for (std::vector<Thread>::const_iterator i = other._threads.begin(); i != other._threads.end(); + i++) _threads.push_back(*i); } void -ThreadListWrapper::append(const Thread &t) { +ThreadListWrapper::append(const Thread &t) +{ _threads.push_back(t); } void -ThreadListWrapper::multiply(const int n) { +ThreadListWrapper::multiply(const int n) +{ if (n == 0) { _threads.clear(); } else { @@ -1226,74 +1327,80 @@ ThreadListWrapper::multiply(const int n) { } } -Thread::Thread() : options(), _op() { -} +Thread::Thread() : options(), _op() {} -Thread::Thread(const Operation &op) : options(), _op(op) { -} +Thread::Thread(const Operation &op) : options(), _op(op) {} -Thread::Thread(const Thread &other) : options(other.options), _op(other._op) { -} +Thread::Thread(const Thread &other) : options(other.options), _op(other._op) {} -Thread::~Thread() { -} +Thread::~Thread() {} -void Thread::describe(std::ostream &os) const { +void +Thread::describe(std::ostream &os) const +{ os << "Thread: [" << std::endl; - _op.describe(os); os << std::endl; + _op.describe(os); + os << std::endl; os << "]"; } -Operation::Operation() : - _optype(OP_NONE), _internal(NULL), _table(), _key(), _value(), _config(), - transaction(NULL), _group(NULL), _repeatgroup(0), _timed(0.0) { +Operation::Operation() + : _optype(OP_NONE), _internal(NULL), _table(), _key(), _value(), _config(), transaction(NULL), + _group(NULL), _repeatgroup(0), _timed(0.0) +{ init_internal(NULL); } -Operation::Operation(OpType optype, Table table, Key key, Value value) : - _optype(optype), _internal(NULL), _table(table), _key(key), _value(value), - _config(), transaction(NULL), _group(NULL), _repeatgroup(0), _timed(0.0) { +Operation::Operation(OpType optype, Table table, Key key, Value value) + : _optype(optype), _internal(NULL), _table(table), _key(key), _value(value), _config(), + transaction(NULL), _group(NULL), _repeatgroup(0), _timed(0.0) +{ init_internal(NULL); size_check(); } -Operation::Operation(OpType optype, Table table, Key key) : - _optype(optype), _internal(NULL), _table(table), _key(key), _value(), - _config(), transaction(NULL), _group(NULL), _repeatgroup(0), _timed(0.0) { +Operation::Operation(OpType optype, Table table, Key key) + : _optype(optype), _internal(NULL), _table(table), _key(key), _value(), _config(), + transaction(NULL), _group(NULL), _repeatgroup(0), _timed(0.0) +{ init_internal(NULL); size_check(); } -Operation::Operation(OpType optype, Table table) : - _optype(optype), _internal(NULL), _table(table), _key(), _value(), - _config(), transaction(NULL), _group(NULL), _repeatgroup(0), _timed(0.0) { +Operation::Operation(OpType optype, Table table) + : _optype(optype), _internal(NULL), _table(table), _key(), _value(), _config(), + transaction(NULL), _group(NULL), _repeatgroup(0), _timed(0.0) +{ init_internal(NULL); size_check(); } -Operation::Operation(const Operation &other) : - _optype(other._optype), _internal(NULL), _table(other._table), - _key(other._key), _value(other._value), _config(other._config), - transaction(other.transaction), _group(other._group), - _repeatgroup(other._repeatgroup), _timed(other._timed) { +Operation::Operation(const Operation &other) + : _optype(other._optype), _internal(NULL), _table(other._table), _key(other._key), + _value(other._value), _config(other._config), transaction(other.transaction), + _group(other._group), _repeatgroup(other._repeatgroup), _timed(other._timed) +{ // Creation and destruction of _group and transaction is managed // by Python. init_internal(other._internal); } -Operation::Operation(OpType optype, const char *config) : - _optype(optype), _internal(NULL), _table(), _key(), _value(), - _config(config), transaction(NULL), _group(NULL), _repeatgroup(0), - _timed(0.0) { +Operation::Operation(OpType optype, const char *config) + : _optype(optype), _internal(NULL), _table(), _key(), _value(), _config(config), + transaction(NULL), _group(NULL), _repeatgroup(0), _timed(0.0) +{ init_internal(NULL); } -Operation::~Operation() { +Operation::~Operation() +{ // Creation and destruction of _group, transaction is managed by Python. delete _internal; } -Operation& Operation::operator=(const Operation &other) { +Operation & +Operation::operator=(const Operation &other) +{ _optype = other._optype; _table = other._table; _key = other._key; @@ -1308,7 +1415,9 @@ Operation& Operation::operator=(const Operation &other) { return (*this); } -void Operation::init_internal(OperationInternal *other) { +void +Operation::init_internal(OperationInternal *other) +{ ASSERT(_internal == NULL); switch (_optype) { @@ -1316,8 +1425,7 @@ void Operation::init_internal(OperationInternal *other) { if (other == NULL) _internal = new CheckpointOperationInternal(); else - _internal = new CheckpointOperationInternal( - *(CheckpointOperationInternal *)other); + _internal = new CheckpointOperationInternal(*(CheckpointOperationInternal *)other); break; case OP_INSERT: case OP_REMOVE: @@ -1326,8 +1434,7 @@ void Operation::init_internal(OperationInternal *other) { if (other == NULL) _internal = new TableOperationInternal(); else - _internal = new TableOperationInternal( - *(TableOperationInternal *)other); + _internal = new TableOperationInternal(*(TableOperationInternal *)other); break; case OP_LOG_FLUSH: _internal = new LogFlushOperationInternal(); @@ -1343,32 +1450,40 @@ void Operation::init_internal(OperationInternal *other) { if (other == NULL) _internal = new SleepOperationInternal(); else - _internal = new SleepOperationInternal( - *(SleepOperationInternal *)other); + _internal = new SleepOperationInternal(*(SleepOperationInternal *)other); break; default: ASSERT(false); } } -bool Operation::combinable() const { - return (_group != NULL && _repeatgroup == 1 && _timed == 0.0 && - transaction == NULL && _config == ""); +bool +Operation::combinable() const +{ + return ( + _group != NULL && _repeatgroup == 1 && _timed == 0.0 && transaction == NULL && _config == ""); } -void Operation::create_all() { +void +Operation::create_all() +{ size_check(); _internal->_flags = 0; _internal->parse_config(_config); } -void Operation::describe(std::ostream &os) const { +void +Operation::describe(std::ostream &os) const +{ os << "Operation: " << _optype; if (is_table_op()) { - os << ", "; _table.describe(os); - os << ", "; _key.describe(os); - os << ", "; _value.describe(os); + os << ", "; + _table.describe(os); + os << ", "; + _key.describe(os); + os << ", "; + _value.describe(os); } if (!_config.empty()) os << ", '" << _config << "'"; @@ -1385,8 +1500,7 @@ void Operation::describe(std::ostream &os) const { os << "[repeat " << _repeatgroup << "]"; os << ": {"; bool first = true; - for (std::vector<Operation>::const_iterator i = _group->begin(); - i != _group->end(); i++) { + for (std::vector<Operation>::const_iterator i = _group->begin(); i != _group->end(); i++) { if (!first) os << "}, {"; i->describe(os); @@ -1396,7 +1510,9 @@ void Operation::describe(std::ostream &os) const { } } -void Operation::get_static_counts(Stats &stats, int multiplier) { +void +Operation::get_static_counts(Stats &stats, int multiplier) +{ if (is_table_op()) switch (_optype) { case OP_INSERT: @@ -1418,17 +1534,20 @@ void Operation::get_static_counts(Stats &stats, int multiplier) { stats.checkpoint.ops += multiplier; if (_group != NULL) - for (std::vector<Operation>::iterator i = _group->begin(); - i != _group->end(); i++) + for (std::vector<Operation>::iterator i = _group->begin(); i != _group->end(); i++) i->get_static_counts(stats, multiplier * _repeatgroup); } -bool Operation::is_table_op() const { - return (_optype == OP_INSERT || _optype == OP_REMOVE || - _optype == OP_SEARCH || _optype == OP_UPDATE); +bool +Operation::is_table_op() const +{ + return ( + _optype == OP_INSERT || _optype == OP_REMOVE || _optype == OP_SEARCH || _optype == OP_UPDATE); } -void Operation::kv_compute_max(bool iskey, bool has_random) { +void +Operation::kv_compute_max(bool iskey, bool has_random) +{ uint64_t max; int size; TableOperationInternal *internal; @@ -1463,7 +1582,9 @@ void Operation::kv_compute_max(bool iskey, bool has_random) { } } -void Operation::kv_size_buffer(bool iskey, size_t &maxsize) const { +void +Operation::kv_size_buffer(bool iskey, size_t &maxsize) const +{ TableOperationInternal *internal; ASSERT(is_table_op()); @@ -1478,8 +1599,10 @@ void Operation::kv_size_buffer(bool iskey, size_t &maxsize) const { } } -void Operation::kv_gen(ThreadRunner *runner, bool iskey, - uint64_t compressibility, uint64_t n, char *result) const { +void +Operation::kv_gen( + ThreadRunner *runner, bool iskey, uint64_t compressibility, uint64_t n, char *result) const +{ TableOperationInternal *internal; uint_t max; uint_t size; @@ -1490,14 +1613,13 @@ void Operation::kv_gen(ThreadRunner *runner, bool iskey, size = iskey ? internal->_keysize : internal->_valuesize; max = iskey ? internal->_keymax : internal->_valuemax; if (n > max) - THROW((iskey ? "Key" : "Value") << " (" << n - << ") too large for size (" << size << ")"); + THROW((iskey ? "Key" : "Value") << " (" << n << ") too large for size (" << size << ")"); /* Setup the buffer, defaulting to zero filled. */ workgen_u64_to_string_zf(n, result, size); /* - * Compressibility is a percentage, 100 is all zeroes, it applies to the - * proportion of the value that can't be used for the identifier. + * Compressibility is a percentage, 100 is all zeroes, it applies to the proportion of the value + * that can't be used for the identifier. */ if (size > 20 && compressibility < 100) { static const char alphanum[] = @@ -1520,50 +1642,54 @@ void Operation::kv_gen(ThreadRunner *runner, bool iskey, for (uint64_t i = 0; i < random_len; ++i) /* - * TODO: It'd be nice to use workgen_rand here, but this class - * is without the context of a runner thread, so it's not easy - * to get access to a state. + * TODO: It'd be nice to use workgen_rand here, but this class is without the context of + * a runner thread, so it's not easy to get access to a state. */ result[i] = alphanum[runner->random_value() % (sizeof(alphanum) - 1)]; } } -void Operation::size_check() const { +void +Operation::size_check() const +{ if (is_table_op()) { if (_key._size == 0 && _table.options.key_size == 0) THROW("operation requires a key size"); - if (OP_HAS_VALUE(this) && _value._size == 0 && - _table.options.value_size == 0) + if (OP_HAS_VALUE(this) && _value._size == 0 && _table.options.value_size == 0) THROW("operation requires a value size"); } } -void Operation::synchronized_check() const { +void +Operation::synchronized_check() const +{ if (_timed != 0.0) return; if (_optype != Operation::OP_NONE) { if (is_table_op() || _internal->sync_time_us() == 0) THROW("operation cannot be synchronized, needs to be timed()"); } else if (_group != NULL) { - for (std::vector<Operation>::iterator i = _group->begin(); - i != _group->end(); i++) + for (std::vector<Operation>::iterator i = _group->begin(); i != _group->end(); i++) i->synchronized_check(); } } -int CheckpointOperationInternal::run(ThreadRunner *runner, WT_SESSION *session) +int +CheckpointOperationInternal::run(ThreadRunner *runner, WT_SESSION *session) { - (void)runner; /* not used */ + (void)runner; /* not used */ return (session->checkpoint(session, NULL)); } -int LogFlushOperationInternal::run(ThreadRunner *runner, WT_SESSION *session) +int +LogFlushOperationInternal::run(ThreadRunner *runner, WT_SESSION *session) { - (void)runner; /* not used */ + (void)runner; /* not used */ return (session->log_flush(session, NULL)); } -void SleepOperationInternal::parse_config(const std::string &config) +void +SleepOperationInternal::parse_config(const std::string &config) { const char *configp; char *endp; @@ -1571,17 +1697,19 @@ void SleepOperationInternal::parse_config(const std::string &config) configp = config.c_str(); _sleepvalue = strtod(configp, &endp); if (configp == endp || *endp != '\0' || _sleepvalue < 0.0) - THROW("sleep operation requires a configuration string as " + THROW( + "sleep operation requires a configuration string as " "a non-negative float, e.g. '1.5'"); } -int SleepOperationInternal::run(ThreadRunner *runner, WT_SESSION *session) +int +SleepOperationInternal::run(ThreadRunner *runner, WT_SESSION *session) { uint64_t endtime; uint64_t now, now_us; - (void)runner; /* not used */ - (void)session; /* not used */ + (void)runner; /* not used */ + (void)session; /* not used */ workgen_clock(&now); now_us = ns_to_us(now); @@ -1605,12 +1733,14 @@ int SleepOperationInternal::run(ThreadRunner *runner, WT_SESSION *session) return (0); } -uint64_t SleepOperationInternal::sync_time_us() const +uint64_t +SleepOperationInternal::sync_time_us() const { return (secs_us(_sleepvalue)); } -void TableOperationInternal::parse_config(const std::string &config) +void +TableOperationInternal::parse_config(const std::string &config) { if (!config.empty()) { if (config == "reopen") @@ -1620,17 +1750,18 @@ void TableOperationInternal::parse_config(const std::string &config) } } -Track::Track(bool latency_tracking) : ops_in_progress(0), ops(0), rollbacks(0), - latency_ops(0), latency(0), bucket_ops(0), min_latency(0), max_latency(0), - us(NULL), ms(NULL), sec(NULL) { +Track::Track(bool latency_tracking) + : ops_in_progress(0), ops(0), rollbacks(0), latency_ops(0), latency(0), bucket_ops(0), + min_latency(0), max_latency(0), us(NULL), ms(NULL), sec(NULL) +{ track_latency(latency_tracking); } -Track::Track(const Track &other) : ops_in_progress(other.ops_in_progress), - ops(other.ops), rollbacks(other.rollbacks), - latency_ops(other.latency_ops), latency(other.latency), - bucket_ops(other.bucket_ops), min_latency(other.min_latency), - max_latency(other.max_latency), us(NULL), ms(NULL), sec(NULL) { +Track::Track(const Track &other) + : ops_in_progress(other.ops_in_progress), ops(other.ops), rollbacks(other.rollbacks), + latency_ops(other.latency_ops), latency(other.latency), bucket_ops(other.bucket_ops), + min_latency(other.min_latency), max_latency(other.max_latency), us(NULL), ms(NULL), sec(NULL) +{ if (other.us != NULL) { us = new uint32_t[LATENCY_US_BUCKETS]; ms = new uint32_t[LATENCY_MS_BUCKETS]; @@ -1641,7 +1772,8 @@ Track::Track(const Track &other) : ops_in_progress(other.ops_in_progress), } } -Track::~Track() { +Track::~Track() +{ if (us != NULL) { delete us; delete ms; @@ -1649,7 +1781,9 @@ Track::~Track() { } } -void Track::add(Track &other, bool reset) { +void +Track::add(Track &other, bool reset) +{ ops_in_progress += other.ops_in_progress; ops += other.ops; latency_ops += other.latency_ops; @@ -1672,7 +1806,9 @@ void Track::add(Track &other, bool reset) { } } -void Track::assign(const Track &other) { +void +Track::assign(const Track &other) +{ ops_in_progress = other.ops_in_progress; ops = other.ops; latency_ops = other.latency_ops; @@ -1687,8 +1823,7 @@ void Track::assign(const Track &other) { us = NULL; ms = NULL; sec = NULL; - } - else if (other.us != NULL && us == NULL) { + } else if (other.us != NULL && us == NULL) { us = new uint32_t[LATENCY_US_BUCKETS]; ms = new uint32_t[LATENCY_MS_BUCKETS]; sec = new uint32_t[LATENCY_SEC_BUCKETS]; @@ -1700,18 +1835,24 @@ void Track::assign(const Track &other) { } } -uint64_t Track::average_latency() const { +uint64_t +Track::average_latency() const +{ if (latency_ops == 0) return (0); else return (latency / latency_ops); } -void Track::begin() { +void +Track::begin() +{ ops_in_progress++; } -void Track::clear() { +void +Track::clear() +{ ops_in_progress = 0; ops = 0; rollbacks = 0; @@ -1727,12 +1868,16 @@ void Track::clear() { } } -void Track::complete() { +void +Track::complete() +{ --ops_in_progress; ops++; } -void Track::complete_with_latency(uint64_t usecs) { +void +Track::complete_with_latency(uint64_t usecs) +{ ASSERT(us != NULL); --ops_in_progress; @@ -1765,7 +1910,9 @@ void Track::complete_with_latency(uint64_t usecs) { // Return the latency for which the given percent is lower than it. // E.g. for percent == 95, returns the latency for which 95% of latencies // are faster (lower), and 5% are slower (higher). -uint64_t Track::percentile_latency(int percent) const { +uint64_t +Track::percentile_latency(int percent) const +{ // Get the total number of operations in the latency buckets. // We can't reliably use latency_ops, because this struct was // added up from Track structures that were being copied while @@ -1805,7 +1952,9 @@ uint64_t Track::percentile_latency(int percent) const { return (0); } -void Track::subtract(const Track &other) { +void +Track::subtract(const Track &other) +{ ops_in_progress -= other.ops_in_progress; ops -= other.ops; latency_ops -= other.latency_ops; @@ -1823,7 +1972,9 @@ void Track::subtract(const Track &other) { } } -void Track::track_latency(bool newval) { +void +Track::track_latency(bool newval) +{ if (newval) { if (us == NULL) { us = new uint32_t[LATENCY_US_BUCKETS]; @@ -1845,21 +1996,27 @@ void Track::track_latency(bool newval) { } } -void Track::_get_us(long *result) { +void +Track::_get_us(long *result) +{ if (us != NULL) { for (int i = 0; i < LATENCY_US_BUCKETS; i++) result[i] = (long)us[i]; } else memset(result, 0, sizeof(long) * LATENCY_US_BUCKETS); } -void Track::_get_ms(long *result) { +void +Track::_get_ms(long *result) +{ if (ms != NULL) { for (int i = 0; i < LATENCY_MS_BUCKETS; i++) result[i] = (long)ms[i]; } else memset(result, 0, sizeof(long) * LATENCY_MS_BUCKETS); } -void Track::_get_sec(long *result) { +void +Track::_get_sec(long *result) +{ if (sec != NULL) { for (int i = 0; i < LATENCY_SEC_BUCKETS; i++) result[i] = (long)sec[i]; @@ -1867,19 +2024,23 @@ void Track::_get_sec(long *result) { memset(result, 0, sizeof(long) * LATENCY_SEC_BUCKETS); } -Stats::Stats(bool latency) : checkpoint(latency), insert(latency), - not_found(latency), read(latency), remove(latency), update(latency), - truncate(latency) { +Stats::Stats(bool latency) + : checkpoint(latency), insert(latency), not_found(latency), read(latency), remove(latency), + update(latency), truncate(latency) +{ } -Stats::Stats(const Stats &other) : checkpoint(other.checkpoint), - insert(other.insert), not_found(other.not_found), read(other.read), - remove(other.remove), update(other.update), truncate(other.truncate) { +Stats::Stats(const Stats &other) + : checkpoint(other.checkpoint), insert(other.insert), not_found(other.not_found), + read(other.read), remove(other.remove), update(other.update), truncate(other.truncate) +{ } Stats::~Stats() {} -void Stats::add(Stats &other, bool reset) { +void +Stats::add(Stats &other, bool reset) +{ checkpoint.add(other.checkpoint, reset); insert.add(other.insert, reset); not_found.add(other.not_found, reset); @@ -1889,7 +2050,9 @@ void Stats::add(Stats &other, bool reset) { truncate.add(other.truncate, reset); } -void Stats::assign(const Stats &other) { +void +Stats::assign(const Stats &other) +{ checkpoint.assign(other.checkpoint); insert.assign(other.insert); not_found.assign(other.not_found); @@ -1899,7 +2062,9 @@ void Stats::assign(const Stats &other) { truncate.assign(other.truncate); } -void Stats::clear() { +void +Stats::clear() +{ checkpoint.clear(); insert.clear(); not_found.clear(); @@ -1909,7 +2074,9 @@ void Stats::clear() { truncate.clear(); } -void Stats::describe(std::ostream &os) const { +void +Stats::describe(std::ostream &os) const +{ os << "Stats: reads " << read.ops; if (not_found.ops > 0) { os << " (" << not_found.ops << " not found)"; @@ -1921,7 +2088,9 @@ void Stats::describe(std::ostream &os) const { os << ", checkpoints " << checkpoint.ops; } -void Stats::final_report(std::ostream &os, timespec &totalsecs) const { +void +Stats::final_report(std::ostream &os, timespec &totalsecs) const +{ uint64_t ops = 0; ops += checkpoint.ops; ops += read.ops; @@ -1931,10 +2100,9 @@ void Stats::final_report(std::ostream &os, timespec &totalsecs) const { ops += truncate.ops; ops += remove.ops; -#define FINAL_OUTPUT(os, field, singular, ops, totalsecs) \ - os << "Executed " << field << " " #singular " operations (" \ - << PCT(field, ops) << "%) " << OPS_PER_SEC(field, totalsecs) \ - << " ops/sec" << std::endl +#define FINAL_OUTPUT(os, field, singular, ops, totalsecs) \ + os << "Executed " << field << " " #singular " operations (" << PCT(field, ops) << "%) " \ + << OPS_PER_SEC(field, totalsecs) << " ops/sec" << std::endl FINAL_OUTPUT(os, read.ops, read, ops, totalsecs); FINAL_OUTPUT(os, not_found.ops, not found, ops, totalsecs); @@ -1945,7 +2113,9 @@ void Stats::final_report(std::ostream &os, timespec &totalsecs) const { FINAL_OUTPUT(os, checkpoint.ops, checkpoint, ops, totalsecs); } -void Stats::report(std::ostream &os) const { +void +Stats::report(std::ostream &os) const +{ os << read.ops << " reads"; if (not_found.ops > 0) { os << " (" << not_found.ops << " not found)"; @@ -1957,7 +2127,9 @@ void Stats::report(std::ostream &os) const { os << checkpoint.ops << " checkpoints"; } -void Stats::subtract(const Stats &other) { +void +Stats::subtract(const Stats &other) +{ checkpoint.subtract(other.checkpoint); insert.subtract(other.insert); not_found.subtract(other.not_found); @@ -1967,7 +2139,9 @@ void Stats::subtract(const Stats &other) { truncate.subtract(other.truncate); } -void Stats::track_latency(bool latency) { +void +Stats::track_latency(bool latency) +{ checkpoint.track_latency(latency); insert.track_latency(latency); not_found.track_latency(latency); @@ -1977,56 +2151,67 @@ void Stats::track_latency(bool latency) { truncate.track_latency(latency); } -TableOptions::TableOptions() : key_size(0), value_size(0), - value_compressibility(100), random_value(false), range(0), _options() { - _options.add_int("key_size", key_size, - "default size of the key, unless overridden by Key.size"); - _options.add_int("value_size", value_size, - "default size of the value, unless overridden by Value.size"); - _options.add_bool("random_value", random_value, - "generate random content for the value"); +TableOptions::TableOptions() + : key_size(0), value_size(0), value_compressibility(100), random_value(false), range(0), + _options() +{ + _options.add_int( + "key_size", key_size, "default size of the key, unless overridden by Key.size"); + _options.add_int( + "value_size", value_size, "default size of the value, unless overridden by Value.size"); + _options.add_bool("random_value", random_value, "generate random content for the value"); _options.add_bool("value_compressibility", value_compressibility, "How compressible the generated value should be"); _options.add_int("range", range, - "if zero, keys are inserted at the end and reads/updates are in the current range, if non-zero, inserts/reads/updates are at a random key between 0 and the given range"); + "if zero, keys are inserted at the end and reads/updates are in the current range, if " + "non-zero, inserts/reads/updates are at a random key between 0 and the given range"); +} +TableOptions::TableOptions(const TableOptions &other) + : key_size(other.key_size), value_size(other.value_size), + value_compressibility(other.value_compressibility), random_value(other.random_value), + range(other.range), _options(other._options) +{ } -TableOptions::TableOptions(const TableOptions &other) : - key_size(other.key_size), value_size(other.value_size), - value_compressibility(other.value_compressibility), - random_value(other.random_value), range(other.range), - _options(other._options) {} TableOptions::~TableOptions() {} -Table::Table() : options(), _uri(), _internal(new TableInternal()) { -} -Table::Table(const char *uri) : options(), _uri(uri), - _internal(new TableInternal()) { +Table::Table() : options(), _uri(), _internal(new TableInternal()) {} +Table::Table(const char *uri) : options(), _uri(uri), _internal(new TableInternal()) {} +Table::Table(const Table &other) + : options(other.options), _uri(other._uri), _internal(new TableInternal(*other._internal)) +{ } -Table::Table(const Table &other) : options(other.options), _uri(other._uri), - _internal(new TableInternal(*other._internal)) { +Table::~Table() +{ + delete _internal; } -Table::~Table() { delete _internal; } -Table& Table::operator=(const Table &other) { +Table & +Table::operator=(const Table &other) +{ options = other.options; _uri = other._uri; *_internal = *other._internal; return (*this); } -void Table::describe(std::ostream &os) const { +void +Table::describe(std::ostream &os) const +{ os << "Table: " << _uri; } TableInternal::TableInternal() : _tint(0), _context_count(0) {} -TableInternal::TableInternal(const TableInternal &other) : _tint(other._tint), - _context_count(other._context_count) {} +TableInternal::TableInternal(const TableInternal &other) + : _tint(other._tint), _context_count(other._context_count) +{ +} TableInternal::~TableInternal() {} -WorkloadOptions::WorkloadOptions() : max_latency(0), - report_file("workload.stat"), report_interval(0), run_time(0), - sample_file("monitor.json"), sample_interval_ms(0), max_idle_table_cycle(0), - sample_rate(1), warmup(0), oldest_timestamp_lag(0.0), stable_timestamp_lag(0.0), - timestamp_advance(0.0), max_idle_table_cycle_fatal(false), _options() { +WorkloadOptions::WorkloadOptions() + : max_latency(0), report_file("workload.stat"), report_interval(0), run_time(0), + sample_file("monitor.json"), sample_interval_ms(0), max_idle_table_cycle(0), sample_rate(1), + warmup(0), oldest_timestamp_lag(0.0), stable_timestamp_lag(0.0), timestamp_advance(0.0), + max_idle_table_cycle_fatal(false), _options() +{ _options.add_int("max_latency", max_latency, "prints warning if any latency measured exceeds this number of " "milliseconds. Requires sample_interval to be configured."); @@ -2051,8 +2236,8 @@ WorkloadOptions::WorkloadOptions() : max_latency(0), _options.add_int("sample_rate", sample_rate, "how often the latency of operations is measured. 1 for every operation, " "2 for every second operation, 3 for every third operation etc."); - _options.add_int("warmup", warmup, - "how long to run the workload phase before starting measurements"); + _options.add_int( + "warmup", warmup, "how long to run the workload phase before starting measurements"); _options.add_double("oldest_timestamp_lag", oldest_timestamp_lag, "how much lag to the oldest timestamp from epoch time"); _options.add_double("stable_timestamp_lag", stable_timestamp_lag, @@ -2064,31 +2249,38 @@ WorkloadOptions::WorkloadOptions() : max_latency(0), "print warning (false) or abort (true) of max_idle_table_cycle failure"); } -WorkloadOptions::WorkloadOptions(const WorkloadOptions &other) : - max_latency(other.max_latency), report_interval(other.report_interval), - run_time(other.run_time), sample_interval_ms(other.sample_interval_ms), - sample_rate(other.sample_rate), _options(other._options) {} +WorkloadOptions::WorkloadOptions(const WorkloadOptions &other) + : max_latency(other.max_latency), report_interval(other.report_interval), + run_time(other.run_time), sample_interval_ms(other.sample_interval_ms), + sample_rate(other.sample_rate), _options(other._options) +{ +} WorkloadOptions::~WorkloadOptions() {} -Workload::Workload(Context *context, const ThreadListWrapper &tlw) : - options(), stats(), _context(context), _threads(tlw._threads) { +Workload::Workload(Context *context, const ThreadListWrapper &tlw) + : options(), stats(), _context(context), _threads(tlw._threads) +{ if (context == NULL) THROW("Workload contructor requires a Context"); } -Workload::Workload(Context *context, const Thread &thread) : - options(), stats(), _context(context), _threads() { +Workload::Workload(Context *context, const Thread &thread) + : options(), stats(), _context(context), _threads() +{ if (context == NULL) THROW("Workload contructor requires a Context"); _threads.push_back(thread); } -Workload::Workload(const Workload &other) : - options(other.options), stats(other.stats), _context(other._context), - _threads(other._threads) {} +Workload::Workload(const Workload &other) + : options(other.options), stats(other.stats), _context(other._context), _threads(other._threads) +{ +} Workload::~Workload() {} -Workload& Workload::operator=(const Workload &other) { +Workload & +Workload::operator=(const Workload &other) +{ options = other.options; stats.assign(other.stats); *_context = *other._context; @@ -2096,53 +2288,63 @@ Workload& Workload::operator=(const Workload &other) { return (*this); } -int Workload::run(WT_CONNECTION *conn) { +int +Workload::run(WT_CONNECTION *conn) +{ WorkloadRunner runner(this); return (runner.run(conn)); } -WorkloadRunner::WorkloadRunner(Workload *workload) : - _workload(workload), _trunners(workload->_threads.size()), - _report_out(&std::cout), _start(), stopping(false) { +WorkloadRunner::WorkloadRunner(Workload *workload) + : _workload(workload), _trunners(workload->_threads.size()), _report_out(&std::cout), _start(), + stopping(false) +{ ts_clear(_start); } WorkloadRunner::~WorkloadRunner() {} -int WorkloadRunner::run(WT_CONNECTION *conn) { +int +WorkloadRunner::run(WT_CONNECTION *conn) +{ WT_DECL_RET; WorkloadOptions *options = &_workload->options; std::ofstream report_out; _wt_home = conn->get_home(conn); - if ( (options->oldest_timestamp_lag > 0 || options->stable_timestamp_lag > 0) && options->timestamp_advance < 0 ) - THROW("Workload.options.timestamp_advance must be positive if either Workload.options.oldest_timestamp_lag or Workload.options.stable_timestamp_lag is set"); + if ((options->oldest_timestamp_lag > 0 || options->stable_timestamp_lag > 0) && + options->timestamp_advance < 0) + THROW( + "Workload.options.timestamp_advance must be positive if either " + "Workload.options.oldest_timestamp_lag or Workload.options.stable_timestamp_lag is set"); if (options->sample_interval_ms > 0 && options->sample_rate <= 0) THROW("Workload.options.sample_rate must be positive"); if (!options->report_file.empty()) { - open_report_file(report_out, options->report_file.c_str(), - "Workload.options.report_file"); + open_report_file(report_out, options->report_file.c_str(), "Workload.options.report_file"); _report_out = &report_out; } WT_ERR(create_all(conn, _workload->_context)); WT_ERR(open_all()); WT_ERR(ThreadRunner::cross_check(_trunners)); WT_ERR(run_all(conn)); - err: - //TODO: (void)close_all(); +err: + // TODO: (void)close_all(); _report_out = &std::cout; return (ret); } -int WorkloadRunner::open_all() { +int +WorkloadRunner::open_all() +{ for (size_t i = 0; i < _trunners.size(); i++) { WT_RET(_trunners[i].open_all()); } return (0); } -void WorkloadRunner::open_report_file(std::ofstream &of, const char *filename, - const char *desc) { +void +WorkloadRunner::open_report_file(std::ofstream &of, const char *filename, const char *desc) +{ std::stringstream sstm; if (!_wt_home.empty()) @@ -2150,11 +2352,12 @@ void WorkloadRunner::open_report_file(std::ofstream &of, const char *filename, sstm << filename; of.open(sstm.str().c_str(), std::fstream::app); if (!of) - THROW_ERRNO(errno, desc << ": \"" << sstm.str() - << "\" could not be opened"); + THROW_ERRNO(errno, desc << ": \"" << sstm.str() << "\" could not be opened"); } -int WorkloadRunner::create_all(WT_CONNECTION *conn, Context *context) { +int +WorkloadRunner::create_all(WT_CONNECTION *conn, Context *context) +{ for (size_t i = 0; i < _trunners.size(); i++) { ThreadRunner *runner = &_trunners[i]; std::stringstream sstm; @@ -2176,20 +2379,25 @@ int WorkloadRunner::create_all(WT_CONNECTION *conn, Context *context) { return (0); } -int WorkloadRunner::close_all() { +int +WorkloadRunner::close_all() +{ for (size_t i = 0; i < _trunners.size(); i++) _trunners[i].close_all(); return (0); } -void WorkloadRunner::get_stats(Stats *result) { +void +WorkloadRunner::get_stats(Stats *result) +{ for (size_t i = 0; i < _trunners.size(); i++) result->add(_trunners[i]._stats); } -void WorkloadRunner::report(time_t interval, time_t totalsecs, - Stats *prev_totals) { +void +WorkloadRunner::report(time_t interval, time_t totalsecs, Stats *prev_totals) +{ std::ostream &out = *_report_out; Stats new_totals(prev_totals->track_latency()); @@ -2198,11 +2406,12 @@ void WorkloadRunner::report(time_t interval, time_t totalsecs, diff.subtract(*prev_totals); prev_totals->assign(new_totals); diff.report(out); - out << " in " << interval << " secs (" - << totalsecs << " total secs)" << std::endl; + out << " in " << interval << " secs (" << totalsecs << " total secs)" << std::endl; } -void WorkloadRunner::final_report(timespec &totalsecs) { +void +WorkloadRunner::final_report(timespec &totalsecs) +{ std::ostream &out = *_report_out; Stats *stats = &_workload->stats; @@ -2214,7 +2423,9 @@ void WorkloadRunner::final_report(timespec &totalsecs) { out << "Run completed: " << totalsecs << " seconds" << std::endl; } -int WorkloadRunner::run_all(WT_CONNECTION *conn) { +int +WorkloadRunner::run_all(WT_CONNECTION *conn) +{ void *status; std::vector<pthread_t> thread_handles; Stats counts(false); @@ -2242,13 +2453,11 @@ int WorkloadRunner::run_all(WT_CONNECTION *conn) { monitor._out = &monitor_out; if (!options->sample_file.empty()) { - open_report_file(monitor_json, options->sample_file.c_str(), - "sample JSON output file"); + open_report_file(monitor_json, options->sample_file.c_str(), "sample JSON output file"); monitor._json = &monitor_json; } - if ((ret = pthread_create(&monitor._handle, NULL, monitor_main, - &monitor)) != 0) { + if ((ret = pthread_create(&monitor._handle, NULL, monitor_main, &monitor)) != 0) { std::cerr << "monitor thread failed err=" << ret << std::endl; return (ret); } @@ -2259,8 +2468,7 @@ int WorkloadRunner::run_all(WT_CONNECTION *conn) { ThreadRunner *runner = &_trunners[i]; runner->_stop = false; runner->_repeat = (options->run_time != 0); - if ((ret = pthread_create(&thandle, NULL, thread_runner_main, - runner)) != 0) { + if ((ret = pthread_create(&thandle, NULL, thread_runner_main, runner)) != 0) { std::cerr << "pthread_create failed err=" << ret << std::endl; std::cerr << "Stopping all threads." << std::endl; for (size_t j = 0; j < thread_handles.size(); j++) { @@ -2280,8 +2488,7 @@ int WorkloadRunner::run_all(WT_CONNECTION *conn) { runnerConnection->runner = this; runnerConnection->connection = conn; - if ((ret = pthread_create(&time_thandle, NULL, thread_workload, - runnerConnection)) != 0) { + if ((ret = pthread_create(&time_thandle, NULL, thread_workload, runnerConnection)) != 0) { std::cerr << "pthread_create failed err=" << ret << std::endl; std::cerr << "Stopping Time threads." << std::endl; (void)pthread_join(time_thandle, &status); @@ -2299,7 +2506,7 @@ int WorkloadRunner::run_all(WT_CONNECTION *conn) { createDropTableCycle->connection = conn; if ((ret = pthread_create(&idle_table_thandle, NULL, thread_idle_table_cycle_workload, - createDropTableCycle)) != 0) { + createDropTableCycle)) != 0) { std::cerr << "pthread_create failed err=" << ret << std::endl; std::cerr << "Stopping Create Drop table idle cycle threads." << std::endl; (void)pthread_join(idle_table_thandle, &status); @@ -2312,8 +2519,7 @@ int WorkloadRunner::run_all(WT_CONNECTION *conn) { timespec now; /* Don't run the test if any of the above pthread_create fails. */ - if (!stopping && ret == 0) - { + if (!stopping && ret == 0) { // Treat warmup separately from report interval so that if we have a // warmup period we clear and ignore stats after it ends. if (options->warmup != 0) @@ -2344,7 +2550,7 @@ int WorkloadRunner::run_all(WT_CONNECTION *conn) { if (sleep_amt.tv_sec > 0) sleep((unsigned int)sleep_amt.tv_sec); else - usleep((useconds_t)((sleep_amt.tv_nsec + 999)/ 1000)); + usleep((useconds_t)((sleep_amt.tv_nsec + 999) / 1000)); workgen_epoch(&now); if (now >= next_report && now < end && options->report_interval != 0) { @@ -2370,8 +2576,7 @@ int WorkloadRunner::run_all(WT_CONNECTION *conn) { for (size_t i = 0; i < _trunners.size(); i++) { WT_TRET(pthread_join(thread_handles[i], &status)); if (_trunners[i]._errno != 0) - VERBOSE(_trunners[i], - "Thread " << i << " has errno " << _trunners[i]._errno); + VERBOSE(_trunners[i], "Thread " << i << " has errno " << _trunners[i]._errno); WT_TRET(_trunners[i]._errno); _trunners[i].close_all(); if (exception == NULL && !_trunners[i]._exception._str.empty()) @@ -2394,8 +2599,7 @@ int WorkloadRunner::run_all(WT_CONNECTION *conn) { if (options->sample_interval_ms > 0) { WT_TRET(pthread_join(monitor._handle, &status)); if (monitor._errno != 0) - std::cerr << "Monitor thread has errno " << monitor._errno - << std::endl; + std::cerr << "Monitor thread has errno " << monitor._errno << std::endl; if (exception == NULL && !monitor._exception._str.empty()) exception = &monitor._exception; @@ -2416,4 +2620,4 @@ int WorkloadRunner::run_all(WT_CONNECTION *conn) { return (ret); } -} +} // namespace workgen diff --git a/src/third_party/wiredtiger/bench/workgen/workgen_int.h b/src/third_party/wiredtiger/bench/workgen/workgen_int.h index 5e7982b3bea..8cbb44d454a 100644 --- a/src/third_party/wiredtiger/bench/workgen/workgen_int.h +++ b/src/third_party/wiredtiger/bench/workgen/workgen_int.h @@ -179,6 +179,16 @@ struct Monitor { Monitor(WorkloadRunner &wrunner); ~Monitor(); int run(); + +private: + void _format_out_header(); + void _format_out_entry(const Stats &interval, double interval_secs, const timespec ×pec, + bool checkpointing, const tm &tm); + void _format_json_prefix(const std::string &version); + void _format_json_entry(const tm &tm, const timespec ×pec, bool first_iteration, + const Stats &interval, bool checkpointing, double interval_secs); + void _format_json_suffix(); + void _check_latency_threshold(const Stats &interval, uint64_t latency_max); }; struct TableRuntime { diff --git a/src/third_party/wiredtiger/bench/wtperf/misc.c b/src/third_party/wiredtiger/bench/wtperf/misc.c index ac11d23fcea..948bb881095 100644 --- a/src/third_party/wiredtiger/bench/wtperf/misc.c +++ b/src/third_party/wiredtiger/bench/wtperf/misc.c @@ -28,6 +28,8 @@ #include "wtperf.h" +#define WT_BACKUP_COPY_SIZE (128 * 1024) + /* Setup the logging output mechanism. */ int setup_log_file(WTPERF *wtperf) @@ -102,3 +104,52 @@ lprintf(const WTPERF *wtperf, int err, uint32_t level, const char *fmt, ...) if (err == WT_PANIC) abort(); } + +/* + * backup_read -- + * Read in a file, used mainly to measure the impact of backup on a single machine. Backup is + * usually copied across different machines, therefore the write portion doesn't affect the + * machine backup is performing on. + */ +void +backup_read(WTPERF *wtperf, const char *filename) +{ + char *buf; + int rfd; + size_t len; + ssize_t rdsize; + struct stat st; + uint32_t buf_size, size, total; + + buf = NULL; + rfd = -1; + + /* Open the file handle. */ + len = strlen(wtperf->home) + strlen(filename) + 10; + buf = dmalloc(len); + testutil_check(__wt_snprintf(buf, len, "%s/%s", wtperf->home, filename)); + error_sys_check(rfd = open(buf, O_RDONLY, 0644)); + + /* Get the file's size. */ + testutil_check(stat(buf, &st)); + size = (uint32_t)st.st_size; + free(buf); + + buf = dmalloc(WT_BACKUP_COPY_SIZE); + total = 0; + buf_size = WT_MIN(size, WT_BACKUP_COPY_SIZE); + while (total < size) { + /* Use the read size since we may have read less than the granularity. */ + error_sys_check(rdsize = read(rfd, buf, buf_size)); + + /* If we get EOF, we're done. */ + if (rdsize == 0) + break; + total += (uint32_t)rdsize; + buf_size = WT_MIN(buf_size, size - total); + } + + if (rfd != -1) + testutil_check(close(rfd)); + free(buf); +} diff --git a/src/third_party/wiredtiger/bench/wtperf/runners/ycsb-a.wtperf b/src/third_party/wiredtiger/bench/wtperf/runners/ycsb-a.wtperf new file mode 100644 index 00000000000..38eac777200 --- /dev/null +++ b/src/third_party/wiredtiger/bench/wtperf/runners/ycsb-a.wtperf @@ -0,0 +1,12 @@ +conn_config="cache_size=40G,log=(enabled=true)" +pareto=20 +table_config="type=file" +key_sz=100 +value_sz=1024 +icount=120000000 +run_time=3600 +threads=((count=10,reads=1),(count=10,updates=1)) +warmup=120 +sample_interval=5 +populate_threads=8 +report_interval=5 diff --git a/src/third_party/wiredtiger/bench/wtperf/runners/ycsb-b.wtperf b/src/third_party/wiredtiger/bench/wtperf/runners/ycsb-b.wtperf new file mode 100644 index 00000000000..aa1d227fced --- /dev/null +++ b/src/third_party/wiredtiger/bench/wtperf/runners/ycsb-b.wtperf @@ -0,0 +1,12 @@ +conn_config="cache_size=40G,log=(enabled=true)" +pareto=20 +table_config="type=file" +key_sz=100 +value_sz=1024 +icount=90000000 +run_time=3600 +threads=((count=20,reads=95,updates=5)) +warmup=120 +sample_interval=5 +populate_threads=8 +report_interval=5 diff --git a/src/third_party/wiredtiger/bench/wtperf/runners/ycsb-c.wtperf b/src/third_party/wiredtiger/bench/wtperf/runners/ycsb-c.wtperf new file mode 100644 index 00000000000..603798f272b --- /dev/null +++ b/src/third_party/wiredtiger/bench/wtperf/runners/ycsb-c.wtperf @@ -0,0 +1,12 @@ +conn_config="cache_size=40G,log=(enabled=true)" +pareto=20 +table_config="type=file" +key_sz=100 +value_sz=1024 +icount=120000000 +run_time=3600 +threads=((count=20,reads=1)) +warmup=120 +sample_interval=5 +populate_threads=8 +report_interval=5 diff --git a/src/third_party/wiredtiger/bench/wtperf/runners/ycsb-d.wtperf b/src/third_party/wiredtiger/bench/wtperf/runners/ycsb-d.wtperf new file mode 100644 index 00000000000..4574ce0c517 --- /dev/null +++ b/src/third_party/wiredtiger/bench/wtperf/runners/ycsb-d.wtperf @@ -0,0 +1,12 @@ +conn_config="cache_size=40G,log=(enabled=true)" +table_config="type=file" +key_sz=100 +value_sz=1024 +icount=60000000 +run_time=3600 +select_latest=true +threads=((count=100,reads=95,inserts=5)) +warmup=120 +sample_interval=5 +populate_threads=8 +report_interval=5 diff --git a/src/third_party/wiredtiger/bench/wtperf/runners/ycsb-e.wtperf b/src/third_party/wiredtiger/bench/wtperf/runners/ycsb-e.wtperf new file mode 100644 index 00000000000..c91777ae8c3 --- /dev/null +++ b/src/third_party/wiredtiger/bench/wtperf/runners/ycsb-e.wtperf @@ -0,0 +1,14 @@ +conn_config="cache_size=40G,log=(enabled=true)" +pareto=20 +table_config="type=file" +key_sz=100 +value_sz=1024 +icount=90000000 +run_time=3600 +threads=((count=20,reads=95,inserts=5)) +read_range=100 +read_range_random=true +warmup=120 +sample_interval=5 +populate_threads=8 +report_interval=5 diff --git a/src/third_party/wiredtiger/bench/wtperf/runners/ycsb-f.wtperf b/src/third_party/wiredtiger/bench/wtperf/runners/ycsb-f.wtperf new file mode 100644 index 00000000000..c997b7c1231 --- /dev/null +++ b/src/third_party/wiredtiger/bench/wtperf/runners/ycsb-f.wtperf @@ -0,0 +1,12 @@ +conn_config="cache_size=40G,log=(enabled=true)" +pareto=20 +table_config="type=file" +key_sz=100 +value_sz=1024 +icount=120000000 +run_time=3600 +threads=((count=10,reads=1),(count=10,modifies=1)) +warmup=120 +sample_interval=5 +populate_threads=8 +report_interval=5 diff --git a/src/third_party/wiredtiger/bench/wtperf/wtperf.c b/src/third_party/wiredtiger/bench/wtperf/wtperf.c index 032acd855ac..6e16da0a0d7 100644 --- a/src/third_party/wiredtiger/bench/wtperf/wtperf.c +++ b/src/third_party/wiredtiger/bench/wtperf/wtperf.c @@ -243,14 +243,16 @@ op_name(uint8_t *op) * the keys we see are always in order. */ static int -do_range_reads(WTPERF *wtperf, WT_CURSOR *cursor, int64_t read_range) +do_range_reads(WTPERF_THREAD *thread, WT_CURSOR *cursor, int64_t read_range) { + WTPERF *wtperf; uint64_t next_val, prev_val; - int64_t range; + int64_t rand_range, range; char *range_key_buf; char buf[512]; int ret; + wtperf = thread->wtperf; ret = 0; if (read_range == 0) @@ -263,6 +265,17 @@ do_range_reads(WTPERF *wtperf, WT_CURSOR *cursor, int64_t read_range) testutil_check(cursor->get_key(cursor, &range_key_buf)); extract_key(range_key_buf, &next_val); + /* + * If an option tells us to randomly select the number of values to read in a range, we use the + * value of read_range as the upper bound on the number of values to read. YCSB-E stipulates + * that we use a uniform random distribution for the number of values, so we do not use the + * wtperf random routine, which may take us to Pareto. + */ + if (wtperf->opts->read_range_random) { + rand_range = __wt_random(&thread->rnd) % read_range; + read_range = rand_range; + } + for (range = 0; range < read_range; ++range) { prev_val = next_val; ret = cursor->next(cursor); @@ -487,7 +500,7 @@ worker(void *arg) * If we want to read a range, then call next for several operations, confirming * that the next key is in the correct order. */ - ret = do_range_reads(wtperf, cursor, workload->read_range); + ret = do_range_reads(thread, cursor, workload->read_range); } if (ret == 0 || ret == WT_NOTFOUND) @@ -806,6 +819,8 @@ run_mix_schedule(WTPERF *wtperf, WORKLOAD *workp) opts = wtperf->opts; + workp->read_range = opts->read_range; + if (workp->truncate != 0) { if (workp->insert != 0 || workp->modify != 0 || workp->read != 0 || workp->update != 0) { lprintf(wtperf, EINVAL, 0, "Can't configure truncate in a mixed workload"); @@ -1252,8 +1267,6 @@ backup_worker(void *arg) break; wtperf->backup = true; - /* Cleanup the data from the previous backup and create the backup directories. */ - testutil_create_backup_directory(wtperf->home); /* * open_cursor can return EBUSY if concurrent with a metadata operation, retry in that case. @@ -1266,7 +1279,7 @@ backup_worker(void *arg) while ((ret = backup_cursor->next(backup_cursor)) == 0) { testutil_check(backup_cursor->get_key(backup_cursor, &key)); - testutil_copy_file(session, key); + backup_read(wtperf, key); } if (ret != WT_NOTFOUND) { @@ -2653,7 +2666,10 @@ wtperf_rand(WTPERF_THREAD *thread) WTPERF *wtperf; double S1, S2, U; uint64_t end_range, range, rval, start_range; - int ret; +#ifdef __SIZEOF_INT128__ + unsigned __int128 rval128; +#endif + int i, ret; char *key_buf; wtperf = thread->wtperf; @@ -2705,6 +2721,59 @@ wtperf_rand(WTPERF_THREAD *thread) if (rval > end_range) rval = 0; } + + /* + * A distribution that selects the record with a higher key with higher probability. This was + * added to support the YCSB-D workload, which calls for "read latest" selection for records + * that are read. + */ + if (opts->select_latest) { + /* + * If we have 128-bit integers, we can use a fancy method described below. If not, we use a + * simple one. + */ +#ifdef __SIZEOF_INT128__ + /* + * We generate a random number that is in the range between 0 and largest * largest, where + * largest is the last inserted key. Then we take a square root of that random number -- + * this is our target selection. With this formula, larger keys are more likely to get + * selected than smaller keys, and the probability of selection is proportional to the + * value of the key, which is what we want. + * + * First we need a 128-bit random number, and the WiredTiger random number function gives us + * only a 32-bit random value. With only a 32-bit value, the range of the random number + * will always be smaller than the square of the largest insert key for workloads with a + * large number of keys. So we need a longer random number for that. + * + * We get a 128-bit random number by concatenating four 32-bit numbers. We get less entropy + * this way than via a true 128-bit generator, but we are not defending against crypto + * attacks here, so this is good enough. + */ + rval128 = rval; + for (i = 0; i < 3; i++) { + rval = __wt_random(&thread->rnd); + rval128 = (rval128 << 32) | rval; + } + + /* + * Now we limit the random value to be within the range of square of the latest insert key + * and take a square root of that value. + */ + rval128 = (rval128 % (wtperf->insert_key * wtperf->insert_key)); + rval = (uint64_t)(double)sqrtl((long double)rval128); + +#else +#define SELECT_LATEST_RANGE 1000 + /* If we don't have 128-bit integers, we simply select a number from a fixed sized group of + * recently inserted records. + */ + (void)i; + range = + (SELECT_LATEST_RANGE < wtperf->insert_key) ? SELECT_LATEST_RANGE : wtperf->insert_key; + start_range = wtperf->insert_key - range; +#endif + } + /* * Wrap the key to within the expected range and avoid zero: we never insert that key. */ diff --git a/src/third_party/wiredtiger/bench/wtperf/wtperf.h b/src/third_party/wiredtiger/bench/wtperf/wtperf.h index 5cedb438507..14b5d55a2ed 100644 --- a/src/third_party/wiredtiger/bench/wtperf/wtperf.h +++ b/src/third_party/wiredtiger/bench/wtperf/wtperf.h @@ -261,6 +261,7 @@ struct __wtperf_thread { /* Per-thread structure */ TRACK update; /* Update operations */ }; +void backup_read(WTPERF *, const char *); void cleanup_truncate_config(WTPERF *); int config_opt_file(WTPERF *, const char *); void config_opt_cleanup(CONFIG_OPTS *); diff --git a/src/third_party/wiredtiger/bench/wtperf/wtperf_opt.i b/src/third_party/wiredtiger/bench/wtperf/wtperf_opt.i index d971f8e4bae..67d81458d69 100644 --- a/src/third_party/wiredtiger/bench/wtperf/wtperf_opt.i +++ b/src/third_party/wiredtiger/bench/wtperf/wtperf_opt.i @@ -140,6 +140,11 @@ DEF_OPT_AS_UINT32(random_range, 0, "if non zero choose a value from within this range as the key for insert operations") DEF_OPT_AS_BOOL(random_value, 0, "generate random content for the value") DEF_OPT_AS_BOOL(range_partition, 0, "partition data by range (vs hash)") +DEF_OPT_AS_UINT32(read_range, 0, + "read a sequential range of keys upon each read operation. This value tells us how many keys " + "to read each time, or an upper bound on the number of keys read if read_range_random is set.") +DEF_OPT_AS_BOOL(read_range_random, 0, "if doing range reads, select the number of keys to read " + "in a range uniformly at random.") DEF_OPT_AS_BOOL(readonly, 0, "reopen the connection between populate and workload phases in readonly mode. Requires " "reopen_connection turned on (default). Requires that read be the only workload specified") @@ -161,6 +166,8 @@ DEF_OPT_AS_UINT32( DEF_OPT_AS_UINT32(scan_table_count, 0, "number of separate tables to be used for scanning. Zero indicates that tables are shared with " "other operations") +DEF_OPT_AS_BOOL(select_latest, 0, "in workloads that involve inserts and another type of operation," + "select the recently inserted records with higher probability") DEF_OPT_AS_CONFIG_STRING(sess_config, "", "session configuration string") DEF_OPT_AS_UINT32(session_count_idle, 0, "number of idle sessions to create. Default 0.") /* The following table configuration is based on the configuration MongoDB uses for collections. */ diff --git a/src/third_party/wiredtiger/build_cmake/README.md b/src/third_party/wiredtiger/build_cmake/README.md index 827242f722d..caea84aec53 100644 --- a/src/third_party/wiredtiger/build_cmake/README.md +++ b/src/third_party/wiredtiger/build_cmake/README.md @@ -89,6 +89,7 @@ There are a number of additional configuration options you can pass to the CMake * `-DENABLE_SNAPPY=1` : Build the snappy compressor extension * `-DENABLE_ZLIB=1` : Build the zlib compressor extension * `-DENABLE_ZSTD=1` : Build the libzstd compressor extension +* `-DENABLE_SODIUM=1` : Build the libsodium encryptor extension * `-DHAVE_DIAGNOSTIC=1` : Enable WiredTiger diagnostics * `-DHAVE_ATTACH=1` : Enable to pause for debugger attach on failure * `-DENABLE_STRICT=1` : Compile with strict compiler warnings enabled diff --git a/src/third_party/wiredtiger/build_cmake/configs/auto.cmake b/src/third_party/wiredtiger/build_cmake/configs/auto.cmake index 01029fd092f..b9026d90d70 100644 --- a/src/third_party/wiredtiger/build_cmake/configs/auto.cmake +++ b/src/third_party/wiredtiger/build_cmake/configs/auto.cmake @@ -229,56 +229,59 @@ config_lib( HAVE_LIBPTHREAD "Pthread library exists." LIB "pthread" - FUNC "pthread_create" ) config_lib( HAVE_LIBRT "rt library exists." LIB "rt" - FUNC "timer_create" ) config_lib( HAVE_LIBDL "dl library exists." LIB "dl" - FUNC "dlopen" ) config_lib( HAVE_LIBLZ4 "lz4 library exists." LIB "lz4" - FUNC "LZ4_versionNumber" + HEADER "lz4.h" ) config_lib( HAVE_LIBSNAPPY "snappy library exists." LIB "snappy" - FUNC "snappy_compress" + HEADER "snappy.h" ) config_lib( HAVE_LIBZ "zlib library exists." LIB "z" - FUNC "zlibVersion" + HEADER "zlib.h" ) config_lib( HAVE_LIBZSTD "zstd library exists." LIB "zstd" - FUNC "ZSTD_versionString" + HEADER "zstd.h" +) + +config_lib( + HAVE_LIBSODIUM + "sodium library exists." + LIB "sodium" + HEADER "sodium.h" ) config_lib( HAVE_LIBTCMALLOC "tcmalloc library exists." LIB "tcmalloc" - FUNC "tc_malloc" ) config_compile( diff --git a/src/third_party/wiredtiger/build_cmake/configs/base.cmake b/src/third_party/wiredtiger/build_cmake/configs/base.cmake index 95919f70f5e..2b44a7c85cb 100644 --- a/src/third_party/wiredtiger/build_cmake/configs/base.cmake +++ b/src/third_party/wiredtiger/build_cmake/configs/base.cmake @@ -63,7 +63,7 @@ config_bool( config_bool( ENABLE_STRICT "Compile with strict compiler warnings enabled" - DEFAULT ON + DEFAULT OFF ) config_bool( @@ -142,6 +142,16 @@ config_bool( ) config_bool( + ENABLE_SODIUM + "Build the libsodium encryption extension" + DEFAULT OFF + DEPENDS "HAVE_LIBSODIUM" + # Specifically throw a fatal error if a user tries to enable the libsodium encryptor without + # actually having the library available (as opposed to silently defaulting to OFF). + DEPENDS_ERROR ON "Failed to find sodium library" +) + +config_bool( ENABLE_TCMALLOC "Use TCMalloc as the backend allocator" DEFAULT OFF diff --git a/src/third_party/wiredtiger/build_cmake/configs/wiredtiger_config.h.in b/src/third_party/wiredtiger/build_cmake/configs/wiredtiger_config.h.in index 50937d6b95d..5cdf5d357b0 100644 --- a/src/third_party/wiredtiger/build_cmake/configs/wiredtiger_config.h.in +++ b/src/third_party/wiredtiger/build_cmake/configs/wiredtiger_config.h.in @@ -24,6 +24,9 @@ /* ZSTD support automatically loaded. */ #cmakedefine HAVE_BUILTIN_EXTENSION_ZSTD 1 +/* libsodium support automatically loaded. */ +#cmakedefine HAVE_BUILTIN_EXTENSION_SODIUM 1 + /* Define to 1 if you have the `clock_gettime' function. */ #cmakedefine HAVE_CLOCK_GETTIME 1 @@ -63,8 +66,19 @@ /* Define to 1 if you have the `snappy' library (-lsnappy). */ #cmakedefine HAVE_LIBSNAPPY 1 -/* Define to 1 if you have the `tcmalloc' library (-ltcmalloc). */ -#cmakedefine HAVE_LIBTCMALLOC 1 +/* Define to 1 if the user has explictly enabled TCMalloc builds. */ +#cmakedefine ENABLE_TCMALLOC 1 + +/* + * To remain compatible with autoconf & scons builds, we + * define HAVE_LIBTCMALLOC for configuring our sources to actually + * include the tcmalloc headers, as opposed to the sources + * using ENABLE_TCMALLOC. + */ +#if defined ENABLE_TCMALLOC + /* Define to 1 if you have the `tcmalloc' library (-ltcmalloc). */ + #cmakedefine HAVE_LIBTCMALLOC 1 +#endif /* Define to 1 if you have the `z' library (-lz). */ #cmakedefine HAVE_LIBZ 1 @@ -72,6 +86,9 @@ /* Define to 1 if you have the `zstd' library (-lzstd). */ #cmakedefine HAVE_LIBZSTD 1 +/* Define to 1 if you have the `sodium' library (-lsodium). */ +#cmakedefine HAVE_LIBSODIUM 1 + /* Define to 1 if you have the <memory.h> header file. */ #cmakedefine HAVE_MEMORY_H 1 diff --git a/src/third_party/wiredtiger/build_cmake/configs/x86/netbsd/config.cmake b/src/third_party/wiredtiger/build_cmake/configs/x86/netbsd/config.cmake new file mode 100644 index 00000000000..f307d3b0110 --- /dev/null +++ b/src/third_party/wiredtiger/build_cmake/configs/x86/netbsd/config.cmake @@ -0,0 +1,14 @@ +# +# Public Domain 2014-present MongoDB, Inc. +# Public Domain 2008-2014 WiredTiger, Inc. +# All rights reserved. +# +# See the file LICENSE for redistribution information +# + +set(WT_ARCH "x86" CACHE STRING "") +set(WT_OS "netbsd" CACHE STRING "") +set(WT_POSIX ON CACHE BOOL "") + +# NetBSD requires buffers aligned to 512-byte (DEV_BSIZE) boundaries for O_DIRECT to work. +set(WT_BUFFER_ALIGNMENT_DEFAULT "512" CACHE STRING "") diff --git a/src/third_party/wiredtiger/build_cmake/helpers.cmake b/src/third_party/wiredtiger/build_cmake/helpers.cmake index 2b65f537afa..5593018284a 100644 --- a/src/third_party/wiredtiger/build_cmake/helpers.cmake +++ b/src/third_party/wiredtiger/build_cmake/helpers.cmake @@ -343,7 +343,7 @@ function(config_include config_name description) ) if (NOT "${CONFIG_INCLUDE_UNPARSED_ARGUMENTS}" STREQUAL "") - message(FATAL_ERROR "Unknown arguments to config_func: ${CONFIG_INCLUDE_UNPARSED_ARGUMENTS}") + message(FATAL_ERROR "Unknown arguments to config_include: ${CONFIG_INCLUDE_UNPARSED_ARGUMENTS}") endif() # We require a include header (not optional). if ("${CONFIG_INCLUDE_FILE}" STREQUAL "") @@ -386,7 +386,7 @@ function(config_include config_name description) endif() endfunction() -# config_lib(config_name description LIB <library> FUNC <function-symbol> [DEPENDS <deps>]) +# config_lib(config_name description LIB <library> FUNC <function-symbol> [DEPENDS <deps>] [HEADER <file>]) # Defines a boolean (0/1) configuration option based on whether a given library exists. # The configuration option is stored in the cmake cache and can be exported to the wiredtiger config header. # config_name - name of the configuration option. @@ -401,7 +401,7 @@ function(config_lib config_name description) 2 "CONFIG_LIB" "" - "LIB;FUNC;DEPENDS" + "LIB;DEPENDS;HEADER" "" ) @@ -412,25 +412,36 @@ function(config_lib config_name description) if ("${CONFIG_LIB_LIB}" STREQUAL "") message(FATAL_ERROR "No library passed") endif() - # We require a function within the library (not optional). - if ("${CONFIG_LIB_FUNC}" STREQUAL "") - message(FATAL_ERROR "No library function passed") - endif() # Check that the configs dependencies are enabled before setting it to a visible enabled state. eval_dependency("${CONFIG_LIB_DEPENDS}" enabled) if(enabled) + message("-- Looking for library ${CONFIG_LIB_LIB}") # 'check_library_exists' won't use our current cache when test compiling the library. # To get around this we need to ensure we manually forward WT_ARCH and WT_OS as a minimum. This is particularly # needed if 'check_library_exists' will leverage one of our toolchain files. if((NOT "${WT_ARCH}" STREQUAL "") AND (NOT "${WT_ARCH}" STREQUAL "")) set(CMAKE_REQUIRED_FLAGS "-DWT_ARCH=${WT_ARCH} -DWT_OS=${WT_OS}") endif() - check_library_exists(${CONFIG_LIB_LIB} ${CONFIG_LIB_FUNC} "" has_lib_${config_name}) + find_library(has_lib_${config_name} ${CONFIG_LIB_LIB}) set(CMAKE_REQUIRED_FLAGS) set(has_lib "0") if(has_lib_${config_name}) set(has_lib ${has_lib_${config_name}}) + if (CONFIG_LIB_HEADER) + find_path(include_path_${config_name} ${CONFIG_LIB_HEADER}) + if (include_path_${config_name}) + message("-- Looking for library ${CONFIG_LIB_LIB}: found ${has_lib_${config_name}}, include path ${include_path_${config_name}}") + include_directories(${include_path_${config_name}}) + else() + message("-- Looking for library ${CONFIG_LIB_LIB}: found ${has_lib$_{config_name}}") + endif() + unset(include_path_${config_name} CACHE) + else() + message("-- Looking for library ${CONFIG_LIB_LIB}: found ${has_lib_${config_name}}") + endif() + else() + message("-- Looking for library ${CONFIG_LIB_LIB}: NOT found") endif() # Set an internal cache variable "${config_name}_DISABLED" to capture its enabled/disabled state. # We want to ensure we capture a transition from a disabled to enabled state when dependencies are met. @@ -444,6 +455,7 @@ function(config_lib config_name description) # configuration runs. unset(has_lib_${config_name} CACHE) else() + message("-- Not looking for library ${CONFIG_LIB_LIB}: disabled") set(${config_name} 0 CACHE INTERNAL "" FORCE) set(${config_name}_DISABLED ON CACHE INTERNAL "" FORCE) endif() diff --git a/src/third_party/wiredtiger/build_cmake/strict/clang_strict.cmake b/src/third_party/wiredtiger/build_cmake/strict/clang_strict.cmake index b00dd1de6dd..f87c988fe02 100644 --- a/src/third_party/wiredtiger/build_cmake/strict/clang_strict.cmake +++ b/src/third_party/wiredtiger/build_cmake/strict/clang_strict.cmake @@ -53,5 +53,13 @@ if(CMAKE_C_COMPILER_VERSION VERSION_GREATER_EQUAL 10) list(APPEND clang_base_c_flags "-Wno-implicit-int-float-conversion") endif() +if(WT_DARWIN AND NOT CMAKE_CROSSCOMPILING) + # If we are not cross-compiling, we can safely disable this diagnostic. + # Its incompatible with strict diagnostics when including external + # libraries that are not in the default linker path + # e.g. linking zlib/snappy/... from /usr/local/. + list(APPEND clang_base_c_flags "-Wno-poison-system-directories") +endif() + # Set our base clang flags to ensure it propogates to the rest of our build. set(COMPILER_DIAGNOSTIC_FLAGS "${COMPILER_DIAGNOSTIC_FLAGS};${clang_base_c_flags}" CACHE INTERNAL "" FORCE) diff --git a/src/third_party/wiredtiger/build_cmake/toolchains/cl.cmake b/src/third_party/wiredtiger/build_cmake/toolchains/cl.cmake new file mode 100644 index 00000000000..d6467238ce3 --- /dev/null +++ b/src/third_party/wiredtiger/build_cmake/toolchains/cl.cmake @@ -0,0 +1,13 @@ +# +# Public Domain 2014-present MongoDB, Inc. +# Public Domain 2008-2014 WiredTiger, Inc. +# All rights reserved. +# +# See the file LICENSE for redistribution information. +# + +cmake_minimum_required(VERSION 3.10.0) + +set(CMAKE_C_COMPILER "cl.exe") +set(CMAKE_CXX_COMPILER "cl.exe") +set(CMAKE_ASM_COMPILER "cl.exe") diff --git a/src/third_party/wiredtiger/build_cmake/toolchains/mongodbtoolchain_v3_clang.cmake b/src/third_party/wiredtiger/build_cmake/toolchains/mongodbtoolchain_v3_clang.cmake new file mode 100644 index 00000000000..2353ba1b7f7 --- /dev/null +++ b/src/third_party/wiredtiger/build_cmake/toolchains/mongodbtoolchain_v3_clang.cmake @@ -0,0 +1,17 @@ +# +# Public Domain 2014-present MongoDB, Inc. +# Public Domain 2008-2014 WiredTiger, Inc. +# All rights reserved. +# +# See the file LICENSE for redistribution information. +# + +cmake_minimum_required(VERSION 3.10.0) + +if(NOT TOOLCHAIN_ROOT) + set(TOOLCHAIN_ROOT "/opt/mongodbtoolchain/v3") +endif() + +set(CMAKE_C_COMPILER "${TOOLCHAIN_ROOT}/bin/clang") +set(CMAKE_CXX_COMPILER "${TOOLCHAIN_ROOT}/bin/clang++") +set(CMAKE_ASM_COMPILER "${TOOLCHAIN_ROOT}/bin/clang") diff --git a/src/third_party/wiredtiger/build_cmake/toolchains/mongodbtoolchain_v3_gcc.cmake b/src/third_party/wiredtiger/build_cmake/toolchains/mongodbtoolchain_v3_gcc.cmake new file mode 100644 index 00000000000..136d4a33f4f --- /dev/null +++ b/src/third_party/wiredtiger/build_cmake/toolchains/mongodbtoolchain_v3_gcc.cmake @@ -0,0 +1,17 @@ +# +# Public Domain 2014-present MongoDB, Inc. +# Public Domain 2008-2014 WiredTiger, Inc. +# All rights reserved. +# +# See the file LICENSE for redistribution information. +# + +cmake_minimum_required(VERSION 3.10.0) + +if(NOT TOOLCHAIN_ROOT) + set(TOOLCHAIN_ROOT "/opt/mongodbtoolchain/v3") +endif() + +set(CMAKE_C_COMPILER "${TOOLCHAIN_ROOT}/bin/gcc") +set(CMAKE_CXX_COMPILER "${TOOLCHAIN_ROOT}/bin/g++") +set(CMAKE_ASM_COMPILER "${TOOLCHAIN_ROOT}/bin/gcc") diff --git a/src/third_party/wiredtiger/build_posix/Make.base b/src/third_party/wiredtiger/build_posix/Make.base index 043120ba943..2ca49f1ada8 100644 --- a/src/third_party/wiredtiger/build_posix/Make.base +++ b/src/third_party/wiredtiger/build_posix/Make.base @@ -64,6 +64,9 @@ endif if HAVE_BUILTIN_EXTENSION_ZSTD libwiredtiger_la_LIBADD += ext/compressors/zstd/libwiredtiger_zstd.la endif +if HAVE_BUILTIN_EXTENSION_SODIUM +libwiredtiger_la_LIBADD += ext/encryptors/sodium/libwiredtiger_sodium.la +endif clean-local: rm -rf WT_TEST diff --git a/src/third_party/wiredtiger/build_posix/Make.subdirs b/src/third_party/wiredtiger/build_posix/Make.subdirs index cab55cd27fb..fa1063fc7ca 100644 --- a/src/third_party/wiredtiger/build_posix/Make.subdirs +++ b/src/third_party/wiredtiger/build_posix/Make.subdirs @@ -15,6 +15,7 @@ ext/compressors/zlib ZLIB ext/compressors/zstd ZSTD ext/encryptors/nop ext/encryptors/rotn +ext/encryptors/sodium SODIUM ext/extractors/csv ext/storage_sources/local_store POSIX_HOST ext/test/fail_fs diff --git a/src/third_party/wiredtiger/build_posix/aclocal/options.m4 b/src/third_party/wiredtiger/build_posix/aclocal/options.m4 index 06821d988dc..651b777fee1 100644 --- a/src/third_party/wiredtiger/build_posix/aclocal/options.m4 +++ b/src/third_party/wiredtiger/build_posix/aclocal/options.m4 @@ -21,10 +21,12 @@ AH_TEMPLATE(HAVE_BUILTIN_EXTENSION_ZLIB, [Zlib support automatically loaded.]) AH_TEMPLATE(HAVE_BUILTIN_EXTENSION_ZSTD, [ZSTD support automatically loaded.]) +AH_TEMPLATE(HAVE_BUILTIN_EXTENSION_SODIUM, + [Sodium support automatically loaded.]) AC_MSG_CHECKING(if --with-builtins option specified) AC_ARG_WITH(builtins, [AS_HELP_STRING([--with-builtins], - [builtin extension names (lz4, snappy, zlib, zstd).])], + [builtin extension names (lz4, snappy, zlib, zstd, sodium).])], [with_builtins=$withval], [with_builtins=]) @@ -40,6 +42,8 @@ for builtin_i in $builtin_list; do wt_cv_with_builtin_extension_zlib=yes;; zstd) AC_DEFINE(HAVE_BUILTIN_EXTENSION_ZSTD) wt_cv_with_builtin_extension_zstd=yes;; + sodium) AC_DEFINE(HAVE_BUILTIN_EXTENSION_SODIUM) + wt_cv_with_builtin_extension_sodium=yes;; *) AC_MSG_ERROR([Unknown builtin extension "$builtin_i"]);; esac done @@ -51,6 +55,8 @@ AM_CONDITIONAL([HAVE_BUILTIN_EXTENSION_ZLIB], [test "$wt_cv_with_builtin_extension_zlib" = "yes"]) AM_CONDITIONAL([HAVE_BUILTIN_EXTENSION_ZSTD], [test "$wt_cv_with_builtin_extension_zstd" = "yes"]) +AM_CONDITIONAL([HAVE_BUILTIN_EXTENSION_SODIUM], + [test "$wt_cv_with_builtin_extension_sodium" = "yes"]) AC_MSG_RESULT($with_builtins) AH_TEMPLATE(HAVE_DIAGNOSTIC, [Define to 1 for diagnostic tests.]) @@ -137,6 +143,32 @@ if test "$wt_cv_enable_lz4" = "yes"; then fi AM_CONDITIONAL([LZ4], [test "$wt_cv_enable_lz4" = "yes"]) +AC_MSG_CHECKING(if --enable-sodium option specified) +AC_ARG_ENABLE(sodium, + [AS_HELP_STRING([--enable-sodium], + [Build the libsodium encryptor extension.])], r=$enableval, r=no) +case "$r" in +no) if test "$wt_cv_with_builtin_extension_sodium" = "yes"; then + wt_cv_enable_sodium=yes + else + wt_cv_enable_sodium=no + fi + ;; +*) if test "$wt_cv_with_builtin_extension_sodium" = "yes"; then + AC_MSG_ERROR( + [Only one of --enable-sodium --with-builtins=sodium allowed]) + fi + wt_cv_enable_sodium=yes;; +esac +AC_MSG_RESULT($wt_cv_enable_sodium) +if test "$wt_cv_enable_sodium" = "yes"; then + AC_CHECK_HEADER(sodium.h,, + [AC_MSG_ERROR([--enable-sodium requires sodium.h])]) + AC_CHECK_LIB(sodium, sodium_init,, + [AC_MSG_ERROR([--enable-sodium requires sodium library])]) +fi +AM_CONDITIONAL([SODIUM], [test "$wt_cv_enable_sodium" = "yes"]) + AC_MSG_CHECKING(if --enable-tcmalloc option specified) AC_ARG_ENABLE(tcmalloc, [AS_HELP_STRING([--enable-tcmalloc], diff --git a/src/third_party/wiredtiger/build_win/wiredtiger_config.h b/src/third_party/wiredtiger/build_win/wiredtiger_config.h index 51a6af73856..a68d64a58bf 100644 --- a/src/third_party/wiredtiger/build_win/wiredtiger_config.h +++ b/src/third_party/wiredtiger/build_win/wiredtiger_config.h @@ -19,6 +19,9 @@ /* ZSTD support automatically loaded. */ /* #undef HAVE_BUILTIN_EXTENSION_ZSTD */ +/* libsodium support automatically loaded. */ +/* #undef HAVE_BUILTIN_EXTENSION_SODIUM */ + /* Define to 1 if you have the `clock_gettime' function. */ /* #undef HAVE_CLOCK_GETTIME */ @@ -67,6 +70,9 @@ /* Define to 1 if you have the `zstd' library (-lzstd). */ /* #undef HAVE_LIBZSTD */ +/* Define to 1 if you have the `sodium' library (-lsodium). */ +/* #undef HAVE_LIBSODIUM */ + /* Define to 1 if you have the <memory.h> header file. */ /* #undef HAVE_MEMORY_H */ diff --git a/src/third_party/wiredtiger/dist/api_data.py b/src/third_party/wiredtiger/dist/api_data.py index b23448940e1..c560a28eb11 100644 --- a/src/third_party/wiredtiger/dist/api_data.py +++ b/src/third_party/wiredtiger/dist/api_data.py @@ -297,14 +297,14 @@ file_config = format_meta + file_runtime_config + tiered_config + [ WT_CONNECTION::add_compressor. If WiredTiger has builtin support for \c "lz4", \c "snappy", \c "zlib" or \c "zstd" compression, these names are also available. See @ref compression for more information'''), - Config('checksum', 'uncompressed', r''' - configure block checksums; permitted values are <code>on</code> - (checksum all blocks), <code>off</code> (checksum no blocks) and - <code>uncompresssed</code> (checksum only blocks which are not - compressed for any reason). The \c uncompressed setting is for - applications which can rely on decompression to fail if a block - has been corrupted''', - choices=['on', 'off', 'uncompressed']), + Config('checksum', 'on', r''' + configure block checksums; the permitted values are \c on, \c off, \c uncompressed and + \c unencrypted. The default is \c on, in which case all block writes include a checksum + subsequently verified when the block is read. The \c off setting does no checksums, + the \c uncompressed setting only checksums blocks that are not compressed, and the + \c unencrypted setting only checksums blocks that are not encrypted. See @ref + tune_checksum for more information.''', + choices=['on', 'off', 'uncompressed', 'unencrypted']), Config('dictionary', '0', r''' the maximum number of unique values remembered in the Btree row-store leaf page value dictionary; see @@ -455,7 +455,7 @@ lsm_meta = file_config + lsm_config + [ obsolete chunks in the LSM tree'''), ] -tiered_meta = file_config + tiered_config + [ +tiered_meta = file_meta + tiered_config + [ Config('last', '0', r''' the last allocated object ID'''), Config('tiers', '', r''' diff --git a/src/third_party/wiredtiger/dist/extlist b/src/third_party/wiredtiger/dist/extlist index f7f7c460035..f6711186ad7 100644 --- a/src/third_party/wiredtiger/dist/extlist +++ b/src/third_party/wiredtiger/dist/extlist @@ -7,4 +7,5 @@ ext/compressors/snappy/snappy_compress.c ext/compressors/zlib/zlib_compress.c ext/encryptors/nop/nop_encrypt.c ext/encryptors/rotn/rotn_encrypt.c +ext/encryptors/sodium/sodium_encrypt.c ext/extractors/csv/csv_extractor.c diff --git a/src/third_party/wiredtiger/dist/filelist b/src/third_party/wiredtiger/dist/filelist index b1fe227cdfc..d2a2d93cff3 100644 --- a/src/third_party/wiredtiger/dist/filelist +++ b/src/third_party/wiredtiger/dist/filelist @@ -201,6 +201,7 @@ src/support/hash_fnv.c src/support/hazard.c src/support/hex.c src/support/huffman.c +src/support/lock_ext.c src/support/modify.c src/support/mtx_rw.c src/support/pow.c diff --git a/src/third_party/wiredtiger/dist/flags.py b/src/third_party/wiredtiger/dist/flags.py index 0f960e3f8b7..9b0c87a6806 100755 --- a/src/third_party/wiredtiger/dist/flags.py +++ b/src/third_party/wiredtiger/dist/flags.py @@ -5,7 +5,8 @@ import re, sys from dist import all_c_files, all_h_files, compare_srcfile # Automatically build flags values: read through all of the header files, and -# for each group of flags, sort them and give them a unique value. +# for each group of flags, sort them, check the start and stop boundaries on +# the flags and give them a unique value. # # To add a new flag declare it at the top of the flags list as: # #define WT_NEW_FLAG_NAME 0x0u @@ -18,30 +19,48 @@ def flag_declare(name): lcnt = 0 parsing = False + start = 0 for line in f: lcnt = lcnt + 1 if line.find('AUTOMATIC FLAG VALUE GENERATION START') != -1: + m = re.search("\d+", line) + if m == None: + print(name + ": automatic flag generation start at line " + + str(lcnt) + " needs start value e.g. AUTOMATIC FLAG VALUE" + + " GENERATION START 0", file=sys.stderr) + sys.exit(1) + start = int(m.group(0)) header = line defines = [] parsing = True elif line.find('AUTOMATIC FLAG VALUE GENERATION STOP') != -1: - # We only support 64 bits. - if len(defines) > 64: + m = re.search("\d+", line) + if m == None: + print(name + ": automatic flag generation stop at line " + + str(lcnt) + " needs stop value e.g. AUTOMATIC FLAG VALUE" + + " GENERATION STOP 32", file=sys.stderr) + sys.exit(1) + end = int(m.group(0)) + # Compare the number of flags defined and against the number + # of flags allowed + if len(defines) > end - start: print(name + ": line " + str(lcnt) +\ - ": exceeds maximum 64 bit flags", file=sys.stderr) + ": exceeds maximum {0} limit bit flags".format(end), file=sys.stderr) sys.exit(1) # Calculate number of hex bytes, create format string - fmt = "0x%%0%dxu" % ((len(defines) + 3) / 4) + fmt = "0x%%0%dxu" % ((start + len(defines) + 3) / 4) + # Generate the flags starting from an offset set from the start value. tfile.write(header) - v = 1 + v = 1 << start for d in sorted(defines): tfile.write(re.sub("0x[01248u]*", fmt % v, d)) v = v * 2 tfile.write(line) parsing = False + start = 0 elif parsing and line.find('#define') == -1: print(name + ": line " + str(lcnt) +\ ": unexpected flag line, no #define", file=sys.stderr) diff --git a/src/third_party/wiredtiger/dist/s_clang-format b/src/third_party/wiredtiger/dist/s_clang-format index a08a457b28d..a12101f2d53 100755 --- a/src/third_party/wiredtiger/dist/s_clang-format +++ b/src/third_party/wiredtiger/dist/s_clang-format @@ -42,7 +42,7 @@ fi case $# in 0) # Get all source files that aren't in s_clang-format.list. - search=`find bench examples ext src test -name '*.[ch]'` + search=`find bench examples ext src test -name '*.[ch]' -o -name '*.cxx'` for f in `cat dist/s_clang-format.list`; do search=`echo "$search" | sed "\#$f#d"` done;; @@ -59,7 +59,7 @@ for f in $search; do cat "$f" | \ clang-format --fallback-style=none | \ python dist/s_comment.py > "$t" || exit 1 - cmp --silent "$f" "$t" + cmp -s "$f" "$t" if test $? -ne 0; then if test $# -eq 0 ; then echo "Modifying $f" diff --git a/src/third_party/wiredtiger/dist/s_clang-format.list b/src/third_party/wiredtiger/dist/s_clang-format.list index d94677f1fc1..51c807aa2a4 100644 --- a/src/third_party/wiredtiger/dist/s_clang-format.list +++ b/src/third_party/wiredtiger/dist/s_clang-format.list @@ -1,6 +1,7 @@ bench/workgen/workgen.h bench/workgen/workgen_int.h bench/workgen/workgen_time.h +bench/workgen/workgen_wrap.cxx src/config/config_def.c src/conn/api_strerror.c src/include/bitstring_inline.h diff --git a/src/third_party/wiredtiger/dist/s_define.list b/src/third_party/wiredtiger/dist/s_define.list index 34e0a9a8aa2..f3607a74127 100644 --- a/src/third_party/wiredtiger/dist/s_define.list +++ b/src/third_party/wiredtiger/dist/s_define.list @@ -32,8 +32,6 @@ WT_CLOCKDIFF_NS WT_CONN_CHECK_PANIC WT_DEADLOCK WT_DEBUG_BYTE -WT_DHANDLE_MAX_FLAG -WT_DHANDLE_ZZZ_ENDFLAG WT_ERR_ASSERT WT_ERR_ERROR_OK WT_EXT_FOREACH_OFF @@ -100,5 +98,4 @@ __F __WIREDTIGER_EXT_H_ __WIREDTIGER_H_ __WT_INTERNAL_H -__func__ __wt_bswap16 diff --git a/src/third_party/wiredtiger/dist/s_export b/src/third_party/wiredtiger/dist/s_export index b3322ffe64e..0df4f16b3ef 100755 --- a/src/third_party/wiredtiger/dist/s_export +++ b/src/third_party/wiredtiger/dist/s_export @@ -33,26 +33,39 @@ check() test -s $t && { echo "=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=" - echo 'unexpected external symbols in the WiredTiger library' + echo 'unexpected external symbols in the WiredTiger library '"$f" echo "=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=" cat $t exit 1 } - - exit 0 } # This check would normally be done after the library is built, but this way # we don't forget about a symbol during development. We usually build in the # top-level or build_posix directories, check the previously built library, # if it exists. And, allow this script to be run from the top-level directory -# as well as locally. -for d in .. ../build_posix; do - for ext in a so dylib; do - f="$d/.libs/libwiredtiger.$ext" - test -f $f && check $f - done +# as well as locally. (??? this last wasn't true before my changes and still +# isn't) +# +# Check all library images that appear; typically they will be one of +# ../build_posix/.libs/libwiredtiger.$ext (autotools build) +# ../.libs/libwiredtiger.$ext (autotools build at top level) +# ../build/libwiredtiger.$ext (cmake build) +# ../build_cmake/libwiredtiger.$ext (cmake build) + +found=0 +for f in $(find .. -name "libwiredtiger.*" -print); do + case "$f" in + *.a|*.so|*.dylib) + check "$f" + found=1 + ;; + *) + ;; + esac done -echo "$0 skipped: libwiredtiger.[a|so|dylib] not found" +if [ $found = 0 ]; then + echo "$0 skipped: libwiredtiger.[a|so|dylib] not found" +fi exit 0 diff --git a/src/third_party/wiredtiger/dist/s_string.ok b/src/third_party/wiredtiger/dist/s_string.ok index e2603e2013d..eb797977de4 100644 --- a/src/third_party/wiredtiger/dist/s_string.ok +++ b/src/third_party/wiredtiger/dist/s_string.ok @@ -62,6 +62,7 @@ CHECKKEY CKPT CLOEXEC CMP +CMake CONCAT CONFIG COVERITY @@ -147,6 +148,7 @@ FOREACH FS FTRUNCATE FULLFSYNC +FUNCSIG Facebook FindClose FindFirstFile @@ -323,6 +325,7 @@ Phong PlatformSDK Posix PostgreSQL +Postrun PowerPC Pre Preload @@ -394,6 +397,7 @@ Syscall TAILQ TCMalloc TESTUTIL +TID TIMESTAMP TIMESTAMPS TMP @@ -423,6 +427,7 @@ URIs UTF UltraSparc Unbuffered +Unencrypted UnixLib UnlockFile Unmap @@ -472,11 +477,13 @@ WiredTigerTxn WithSeeds Wpedantic WriteFile +Wsign Wuninitialized Wunused XP Xcode Xcode/ +YCSB Yann ZSTD Zlib @@ -582,6 +589,7 @@ celsius centric cfg cfko +chacha change's changelog chdir @@ -599,6 +607,7 @@ chk chmod chongo cip +ciphertext cjoin ckpt ckptfrag @@ -640,6 +649,8 @@ create's createCStream crypto cryptobad +cryptographic +cryptographically cstring csuite csv @@ -704,6 +715,7 @@ designator dest destSize destructor +destructors dev dh dhandle @@ -854,6 +866,7 @@ gobare goesc gostring gostruct +goto goutf gt handleops @@ -952,6 +965,7 @@ kb kbits keycmp keyid +keyids keylen keyv kmsid @@ -976,6 +990,7 @@ lf lfence libdatasource libs +libsodium libtool libwiredtiger linkers @@ -1095,6 +1110,7 @@ nocase noclear nocrypto nolock +nonces nonliteral noop nop @@ -1161,6 +1177,7 @@ portably pos posint posix +postrun postsize powerpc pragmas @@ -1183,6 +1200,7 @@ printlog priv prog progname +programmatically ps psp pthread @@ -1202,6 +1220,7 @@ qsort quartile queueable quiesce +quiesced qup rN rS @@ -1258,6 +1277,7 @@ sT sanitizer sanitizers scalability +scalable sched scr sd @@ -1330,6 +1350,7 @@ subtree sunique supd superset +suseconds sw sx sy @@ -1354,6 +1375,7 @@ timedwait timestamp timestamped timestamps +timeval tinfo tlb tmp @@ -1364,6 +1386,7 @@ tokname tokstart toktype tolower +toolchain toplevel totalsec toupper @@ -1376,6 +1399,7 @@ trunc trylock tsc tupdate +tv tvalue tw txn @@ -1392,14 +1416,16 @@ uint uintmax unbare unbuffered +uncompressed uncompressing -uncompresssed +uncustomized undef underflowed unencrypted unesc unescape unescaped +unhandled unicode uninstantiated unistd diff --git a/src/third_party/wiredtiger/dist/s_typedef b/src/third_party/wiredtiger/dist/s_typedef index d3987618ab8..2da60035911 100755 --- a/src/third_party/wiredtiger/dist/s_typedef +++ b/src/third_party/wiredtiger/dist/s_typedef @@ -21,7 +21,7 @@ build() { l=`ls ../src/include/*.h ../src/include/*.in | sed -e '/wiredtiger.*/d' -e '/queue.h/d'` egrep -h \ - '^\s*(((struct|union)\s.*__wt_.*{)|WT_PACKED_STRUCT_BEGIN)' \ + '^[[:space:]]*(((struct|union)[[:space:]].*__wt_.*{)|WT_PACKED_STRUCT_BEGIN)' \ $l | sed -e 's/WT_PACKED_STRUCT_BEGIN(\(.*\))/struct \1 {/' \ -e 's/WT_COMPILER_TYPE_ALIGN(.*)[ ]*//' \ diff --git a/src/third_party/wiredtiger/dist/s_void b/src/third_party/wiredtiger/dist/s_void index c679f5d3b0a..da6e7b77710 100755 --- a/src/third_party/wiredtiger/dist/s_void +++ b/src/third_party/wiredtiger/dist/s_void @@ -79,7 +79,6 @@ func_ok() -e '/int __wt_stat_dsrc_desc$/d' \ -e '/int __wt_stat_join_desc$/d' \ -e '/int __wt_stat_session_desc/d' \ - -e '/int __wt_tiered_close/d' \ -e '/int __wt_txn_read_upd_list$/d' \ -e '/int __wt_txn_rollback_required$/d' \ -e '/int __wt_win_directory_list_free$/d' \ @@ -130,6 +129,10 @@ func_ok() -e '/int rotn_terminate$/d' \ -e '/int snappy_pre_size$/d' \ -e '/int snappy_terminate$/d' \ + -e '/int sodium_configure$/d' \ + -e '/int sodium_error$/d' \ + -e '/int sodium_sizing$/d' \ + -e '/int sodium_terminate$/d' \ -e '/int subtest_error_handler$/d' \ -e '/int test_hs_workload$/d' \ -e '/int uri2name$/d' \ diff --git a/src/third_party/wiredtiger/dist/stat.py b/src/third_party/wiredtiger/dist/stat.py index 2e0cb0a867e..73f8ced77be 100644 --- a/src/third_party/wiredtiger/dist/stat.py +++ b/src/third_party/wiredtiger/dist/stat.py @@ -3,15 +3,19 @@ import re, string, sys, textwrap from dist import compare_srcfile, format_srcfile +from operator import attrgetter # Read the source files. -from stat_data import groups, dsrc_stats, connection_stats, conn_dsrc_stats, join_stats, \ +from stat_data import groups, dsrc_stats, conn_stats, conn_dsrc_stats, join_stats, \ session_stats -connection_statistics = connection_stats -connection_statistics.extend(conn_dsrc_stats) -dsrc_statistics = dsrc_stats -dsrc_statistics.extend(conn_dsrc_stats) +# Statistic categories need to be sorted in order to generate a valid statistics JSON file. +sorted_conn_stats = conn_stats +sorted_conn_stats.extend(conn_dsrc_stats) +sorted_conn_stats = sorted(sorted_conn_stats, key=attrgetter('desc')) +sorted_dsrc_statistics = dsrc_stats +sorted_dsrc_statistics.extend(conn_dsrc_stats) +sorted_dsrc_statistics = sorted(sorted_dsrc_statistics, key=attrgetter('desc')) def print_struct(title, name, base, stats): '''Print the structures for the stat.h file.''' @@ -38,9 +42,8 @@ for line in open('../src/include/stat.h', 'r'): elif line.count('Statistics section: BEGIN'): f.write('\n') skip = 1 - print_struct( - 'connections', 'connection', 1000, connection_statistics) - print_struct('data sources', 'dsrc', 2000, dsrc_statistics) + print_struct('connections', 'connection', 1000, sorted_conn_stats) + print_struct('data sources', 'dsrc', 2000, sorted_dsrc_statistics) print_struct('join cursors', 'join', 3000, join_stats) print_struct('session', 'session', 4000, session_stats) f.close() @@ -79,7 +82,7 @@ def print_defines(): * @{ */ ''') - print_defines_one('CONN', 1000, connection_stats) + print_defines_one('CONN', 1000, sorted_conn_stats) f.write(''' /*! * @} @@ -88,7 +91,7 @@ def print_defines(): * @{ */ ''') - print_defines_one('DSRC', 2000, dsrc_stats) + print_defines_one('DSRC', 2000, sorted_dsrc_statistics) f.write(''' /*! * @} @@ -257,8 +260,8 @@ f = open(tmp_file, 'w') f.write('/* DO NOT EDIT: automatically built by dist/stat.py. */\n\n') f.write('#include "wt_internal.h"\n') -print_func('dsrc', 'WT_DATA_HANDLE', dsrc_statistics) -print_func('connection', 'WT_CONNECTION_IMPL', connection_statistics) +print_func('dsrc', 'WT_DATA_HANDLE', sorted_dsrc_statistics) +print_func('connection', 'WT_CONNECTION_IMPL', sorted_conn_stats) print_func('join', None, join_stats) print_func('session', None, session_stats) f.close() diff --git a/src/third_party/wiredtiger/dist/stat_data.py b/src/third_party/wiredtiger/dist/stat_data.py index de5cde1f494..aca6c6b2298 100644 --- a/src/third_party/wiredtiger/dist/stat_data.py +++ b/src/third_party/wiredtiger/dist/stat_data.py @@ -153,7 +153,7 @@ groups['system'] = [ ########################################## # CONNECTION statistics ########################################## -connection_stats = [ +conn_stats = [ ########################################## # System statistics ########################################## @@ -209,6 +209,7 @@ connection_stats = [ CacheStat('cache_eviction_empty_score', 'eviction empty score', 'no_clear,no_scale'), CacheStat('cache_eviction_fail', 'pages selected for eviction unable to be evicted'), CacheStat('cache_eviction_fail_active_children_on_an_internal_page', 'pages selected for eviction unable to be evicted because of active children on an internal page'), + CacheStat('cache_eviction_fail_checkpoint_out_of_order_ts', 'pages selected for eviction unable to be evicted because of race between checkpoint and out of order timestamps handling'), CacheStat('cache_eviction_fail_in_reconciliation', 'pages selected for eviction unable to be evicted because of failure in reconciliation'), CacheStat('cache_eviction_fail_parent_has_overflow_items', 'pages selected for eviction unable to be evicted as the parent page has overflow items'), CacheStat('cache_eviction_force', 'forced eviction - pages selected count'), @@ -503,7 +504,6 @@ connection_stats = [ ########################################## StorageStat('flush_state_races', 'flush state races'), StorageStat('flush_tier', 'flush_tier operation calls'), - StorageStat('flush_tier_busy', 'flush_tier busy retries'), ########################################## # Thread Count statistics @@ -596,7 +596,7 @@ connection_stats = [ YieldStat('txn_release_blocked', 'connection close blocked waiting for transaction state stabilization'), ] -connection_stats = sorted(connection_stats, key=attrgetter('desc')) +conn_stats = sorted(conn_stats, key=attrgetter('desc')) ########################################## # Data source statistics @@ -812,11 +812,13 @@ conn_dsrc_stats = [ CursorStat('cursor_next_hs_tombstone', 'cursor next calls that skip due to a globally visible history store tombstone'), CursorStat('cursor_next_skip_ge_100', 'cursor next calls that skip greater than or equal to 100 entries'), CursorStat('cursor_next_skip_lt_100', 'cursor next calls that skip less than 100 entries'), + CursorStat('cursor_next_skip_page_count', 'Total number of pages skipped without reading by cursor next calls'), CursorStat('cursor_next_skip_total', 'Total number of entries skipped by cursor next calls'), CursorStat('cursor_open_count', 'open cursor count', 'no_clear,no_scale'), CursorStat('cursor_prev_hs_tombstone', 'cursor prev calls that skip due to a globally visible history store tombstone'), CursorStat('cursor_prev_skip_ge_100', 'cursor prev calls that skip greater than or equal to 100 entries'), CursorStat('cursor_prev_skip_lt_100', 'cursor prev calls that skip less than 100 entries'), + CursorStat('cursor_prev_skip_page_count', 'Total number of pages skipped without reading by cursor prev calls'), CursorStat('cursor_prev_skip_total', 'Total number of entries skipped by cursor prev calls'), CursorStat('cursor_search_near_prefix_fast_paths', 'Total number of times a search near has exited due to prefix config'), CursorStat('cursor_skip_hs_cur_position', 'Total number of entries skipped to position the history store cursor'), diff --git a/src/third_party/wiredtiger/dist/test_data.py b/src/third_party/wiredtiger/dist/test_data.py index 48f51675514..62789154113 100644 --- a/src/third_party/wiredtiger/dist/test_data.py +++ b/src/third_party/wiredtiger/dist/test_data.py @@ -66,8 +66,10 @@ record_config = [ populate_config = record_config + [ Config('collection_count', 1, r''' The number of collections the workload generator operates over''', min=0, max=200000), - Config('key_count', 0, r''' + Config('key_count_per_collection', 0, r''' The number of keys to be operated on per collection''', min=0, max=1000000), + Config('thread_count', 1, r''' + The number of worker threads to use while populating the database.''') ] # @@ -110,8 +112,8 @@ transaction_config = [ ] thread_count = [ - Config('thread_count', 1, r''' - Specifies the number of threads that will be used to perform a certain function.''') + Config('thread_count', 0, r''' + Specifies the number of threads that will be used to perform a certain function.''', min=0) ] read_thread_config = thread_count + throttle_config + transaction_config @@ -132,7 +134,11 @@ runtime_monitor = enabled_config_true + component_config + [ type='category', subconfig=limit_stat), Config('stat_db_size', '', ''' The maximum on-disk database size in bytes that can be hit while running.''', - type='category', subconfig=limit_stat) + type='category', subconfig=limit_stat), + Config('postrun_statistics', '[]', ''' + A list of statistics to be checked after the workload has completed. Each element of the + list should be formatted as "stat_name:min_limit:max_limit".''', + type='list') ] # @@ -153,7 +159,10 @@ workload_tracking = enabled_config_true + component_config # # Configuration that applies to the workload_generator component. # -workload_generator = enabled_config_true + component_config + populate_config + [ +workload_generator = enabled_config_true + component_config + [ + Config('populate_config', '', r''' + Config that specifies how the database will be populated initially.''', + type='category', subconfig=populate_config), Config('read_config', '', r''' Config that specifies the number of read threads and their behaviour.''', type='category', subconfig=read_thread_config), @@ -186,13 +195,26 @@ test_config = [ # Non component top level configuration. Config('cache_size_mb', 0, r''' The cache size that wiredtiger will be configured to run with''', min=0, max=100000000000), + Config('compression_enabled', 'false', r''' + Whether the database files will use snappy compression or not.''', type='boolean'), Config('duration_seconds', 0, r''' The duration that the test run will last''', min=0, max=1000000), Config('enable_logging', 'false', r''' Enables write ahead logs''', type='boolean'), + Config('statistics_config', '', r''' + Statistic configuration that is passed into wiredtiger on open.''', + type='category', subconfig=[ + Config('type', 'all', r''' + The configuration that will get passed to wiredtiger to determine the style of + statistics gathering'''), + Config('enable_logging', 'true', r''' + Configuration enabling or disabling statistics logging in the form of json logging.''', + type='boolean') + ]), ] methods = { 'example_test' : Method(test_config), - 'poc_test' : Method(test_config), + 'hs_cleanup' : Method(test_config), + 'base_test' : Method(test_config), } diff --git a/src/third_party/wiredtiger/dist/test_tag.py b/src/third_party/wiredtiger/dist/test_tag.py index 98b4f976aef..f36fa6613e1 100644 --- a/src/third_party/wiredtiger/dist/test_tag.py +++ b/src/third_party/wiredtiger/dist/test_tag.py @@ -1,15 +1,8 @@ import os import sys +from dist import compare_srcfile ##### LOCAL VARIABLES AND CONSTANTS ##### -component = "" -testing_area = "" -test_type = "" - -is_end = False -is_file_ignored = False -is_file_tagged = False -is_start = False show_info = False show_missing_files = False @@ -21,14 +14,35 @@ sorted_tags = [] test_files = [] tagged_files = {} -valid_tags = {} +valid_tags = [] +test_type_tags = ['data_correctness', 'functional_correctness'] +C_FILE_TYPE = 0 +PY_FILE_TYPE = 1 END_TAG = "[END_TAGS]" IGNORE_FILE = "ignored_file" -NB_TAG_ARGS = 3 START_TAG = "[TEST_TAGS]" ##### +##### FUNCTIONS ##### +def validate_tag(tag, filename): + split_tag = tag.split(":") + # Ensure the array isn't too long. + if (len(split_tag) > 3): + print("Tag contains too many sub tags: " + tag + " filename: " + filename); + exit(1) + + # Walk the pieces of the tag and ensure they exist in test_tags.ok. + for sub_tag in split_tag: + if not sub_tag in valid_tags: + print( + "Invalid sub tag found: " + sub_tag + " in tag: " + tag + " filename: " + filename) + exit(1) + +def format_tag(tag): + return tag.replace("_", " ").title() +##### + ##### PROCESS ARGS ##### for arg in sys.argv: if arg == "-h": @@ -36,7 +50,7 @@ for arg in sys.argv: print("Options:") print("\t-i\tShow info") print("\t-p\tShow files with no test tags") - exit() + exit(1) elif arg == "-i": show_info = True elif arg == "-p": @@ -55,16 +69,17 @@ for root, dirs, files in os.walk("../test/"): ##### RETRIEVE VALID TAGS ##### validation_file = open("test_tags.ok", "r") -# The file has the following pattern -# <COMPONENT>:<TESTING_TYPE>:<TESTING_AREA>:<DESCRIPTION> -# A tag is made of the three first values: COMPONENT, TEST_TYPE and TESTING_AREA +# The validation file contains an alphabetized list of valid tag words, with one word per line. tags = validation_file.readlines() tags = [tag.replace('\n', '') for tag in tags] for tag in tags: - current_line = tag.split(':') - # Createa key value pair <TAG>:<DESCRIPTION> - valid_tags[':'.join(current_line[:NB_TAG_ARGS])] = ':'.join(current_line[NB_TAG_ARGS:]) + valid_tags.append(tag) + +# Check that the validation file is ordered. +if not all(tags[i] <= tags[i+1] for i in range(len(tags)-1)): + print("Error: test_tags.ok is not alphabetically ordered!") + exit(1) validation_file.close() ##### @@ -74,8 +89,7 @@ for filename in test_files: input_file = open(filename, "r") lines = input_file.readlines() - is_start = False - is_end = False + in_tag_block = False is_file_ignored = False is_file_tagged = False @@ -89,23 +103,22 @@ for filename in test_files: # Check if line is valid if not line: # Check if invalid line after START_TAG - if is_start == True: - print("Error syntax in file " + filename) - exit() + if in_tag_block == True: + print("Syntax error in file: " + filename) + exit(1) else: continue # Check if end of test tag if END_TAG in line: # END_TAG should not be before START_TAG - if is_start == False: - print("Error syntax in file " + filename + ". Unexpected tag: " + END_TAG) - exit() + if in_tag_block == False: + print("Syntax error in file: " + filename + ". Unexpected tag: " + END_TAG) + exit(1) # END_TAG should not be met before a test tag if is_file_ignored == False and is_file_tagged == False: - print("Error syntax in file " + filename + ". Missing test tag.") - exit() - is_end = True + print("Syntax error in file: " + filename + ". Missing test tag.") + exit(1) nb_valid_files = nb_valid_files + 1 # Go to next file break @@ -113,32 +126,45 @@ for filename in test_files: # Check if start of test tag if START_TAG in line: # Only one START_TAG is allowed - if is_start == True: - print("Error syntax in file " + filename + ". Unexpected tag: " + START_TAG) - exit() - is_start = True + if in_tag_block == True: + print("Syntax error in file: " + filename + ". Unexpected tag: " + START_TAG) + exit(1) + in_tag_block = True continue - if is_start == True: + if in_tag_block == True: + tag = line # Check if file is ignored if is_file_ignored == True: print("Unexpected value in ignored file: " + filename) - exit() - if line == IGNORE_FILE: + exit(1) + if tag == IGNORE_FILE: nb_ignored_files = nb_ignored_files + 1 is_file_ignored = True continue - # Check if current tag is valid - if not line in valid_tags: - print("Tag is not valid ! Add the new tag to test_tags.ok:\n" + line) - exit() - else: - is_file_tagged = True + + # Validate the tag's correctness. + validate_tag(tag, filename) + is_file_tagged = True + + skip_adding_test_type = False + # Add the test type to the tag if it wasn't already. + for test_type_tag in test_type_tags: + if (tag.startswith(test_type_tag)): + skip_adding_test_type = True + break + + if not skip_adding_test_type: + if filename.endswith(".c"): + tag = test_type_tags[C_FILE_TYPE] + ":" + tag + if filename.endswith(".py"): + tag = test_type_tags[PY_FILE_TYPE] + ":" + tag + # Check if current tag has already matched test files - if line in tagged_files: - tagged_files[line].append(filename) + if tag in tagged_files: + tagged_files[tag].append(filename) else: - tagged_files[line] = [filename] + tagged_files[tag] = [filename] if is_file_ignored == False and is_file_tagged == False: nb_missing_files = nb_missing_files + 1 @@ -148,46 +174,47 @@ for filename in test_files: input_file.close() ##### -##### GENERATE OUTPUT ##### -output_file = open("../test/test_coverage.md", "w") - -# Table headers -output_file.write("|Component|Test Type|Testing Area|Description|Existing tests|" + '\n') -output_file.write("|---|---|---|---|---|" + '\n') +##### GENERATE TEST COVERAGE MD ##### +tmp_filename = '__tmp' +tfile = open('__tmp', 'w') # Sort tags sorted_tags = list(tagged_files.keys()) sorted_tags.sort() -for tag in sorted_tags: - # Split line - current_line = tag.split(":") - - # Parse tag - component = current_line[0] - test_type = current_line[1] - testing_area = current_line[2] - - # Format output - component = component.replace("_", " ").title() - test_type = test_type.replace("_", " ").title() - testing_area = testing_area.replace("_", " ").title() - - # Relative path to test files - link = "" - # Sort the filenames associated to the current tag - tagged_files[tag].sort() - for name in tagged_files[tag]: - link += "[" + name + "](" + name + "), " - # Remove the extra ", " at the end - link = link[:-2] - - # Write to output - output_file.write('|' + component + '|' + test_type + '|' + \ - testing_area + '|' + valid_tags[tag] + '|' \ - + link + '\n') - -output_file.close() +for test_type_tag in test_type_tags: + tfile.write("## " + format_tag(test_type_tag) + " tests:\n\n") + tfile.write("|Component|Sub-component|Existing tests|\n") + tfile.write("|---|---|---|\n") + + for tag in sorted_tags: + # Split the tag. + current_tag = tag.split(":") + if (current_tag[0] != test_type_tag): + continue + + # Parse and format tag. + component = format_tag(current_tag[1]) + functionality = "" + + # The end tag is optional. + if (len(current_tag) == 3): + functionality = format_tag(current_tag[2]) + + # Relative path to test files + link = "" + # Sort the filenames associated to the current tag + tagged_files[tag].sort() + for name in tagged_files[tag]: + link += "[" + os.path.basename(name) + "](" + name + "), " + # Remove the extra ", " at the end + link = link[:-2] + + # Write to output + tfile.write('|' + component + '|' + \ + functionality + '|' + link + '\n') +tfile.close() +compare_srcfile(tmp_filename, "../test/test_coverage.md") ##### ##### STATS ##### diff --git a/src/third_party/wiredtiger/dist/test_tags.ok b/src/third_party/wiredtiger/dist/test_tags.ok index 017af50ee15..fca8a14d7e8 100644 --- a/src/third_party/wiredtiger/dist/test_tags.ok +++ b/src/third_party/wiredtiger/dist/test_tags.ok @@ -1,13 +1,43 @@ -backup:correctness:full_backup:Full backup contains correct data -checkpoints:correctness:checkpoint_data:On system with a complex, concurrent workload the correct versions of data appear in checkpoints -checkpoints:correctness:creation_and_deletion:Deleting arbitrary checkpoints doesn’t corrupt data held in other checkpoints -checkpoints:correctness:cross_table:Ensure that checkpoints across many tables are correct. -checkpoints:fault_injection:data_correctness:After a fault-induced failure, all data for completed checkpoints is properly preserved -checkpoints:liveness:liveness:Identify bugs and race conditions related to checkpoints that can cause deadlocks or livelocks -checkpoints:scalability:many_checkpoints:Ensure that we can take as many checkpoints as there is storage space for -caching_eviction:fault_injection:cache_corruption:If dirty or clean data is corrupted in cache, what happens? -caching_eviction:correctness:written_data:Ensure that the data written out by eviction is correct after reading -caching_eviction:liveness:cache_stuck:Identify bugs and bad workloads that can cause the WiredTiger cache to live lock (a.k.a., “cache stuck” errors). -caching_eviction:scalability:large_caches:Test that WT behaves correctly with very large cache sizes. -caching_eviction:scalability:large_dataset:Test workloads constructed to never/rarely hit in the cache. WT may perform slowly, but should be correct. -caching_eviction:scalability:large_metadata:Test workloads constructed so that the metadata is larger than the cache (i.e., very many files and small cache). +aggregated_time_windows +backup +checkpoint +compression +config_api +connection_api +cursors +data_correctness +encryption +eviction +eviction_checkpoint_interaction +flush_tier +functional_correctness +garbage_collection +history_store +huffman_encoding +indexes +isolation_levels +log_files +metadata +mixed_mode_timestamps +obsolete_data +out_of_order_timestamps +overflow_keys +prepare +reconciliation +reconfigure +recovery +rollback_to_stable +salvage +schema_api +search +search_near +session_api +statistics +tiered_storage +timestamps +transactions +truncate +turtle_file +verify +wiredtiger_open +wt_util diff --git a/src/third_party/wiredtiger/examples/c/ex_all.c b/src/third_party/wiredtiger/examples/c/ex_all.c index 36de04005c0..9742a4f59a0 100644 --- a/src/third_party/wiredtiger/examples/c/ex_all.c +++ b/src/third_party/wiredtiger/examples/c/ex_all.c @@ -1195,6 +1195,18 @@ main(int argc, char *argv[]) /*! [Configure zstd extension with compression level] */ error_check(conn->close(conn, NULL)); + /* this is outside the example snippet on purpose; don't encourage compiling in keys */ + const char *secretkey = "abcdef"; + /*! [Configure sodium extension] */ + char conf[1024]; + snprintf(conf, sizeof(conf), + "create,extensions=[/usr/local/lib/libwiredtiger_sodium.so]," + "encryption=(name=sodium,secretkey=%s)", + secretkey); + error_check(wiredtiger_open(home, NULL, conf, &conn)); + /*! [Configure sodium extension] */ + error_check(conn->close(conn, NULL)); + /* * This example code gets run, and direct I/O might not be available, causing the open to fail. * The documentation requires code snippets, use #ifdef's to avoid running it. diff --git a/src/third_party/wiredtiger/examples/c/ex_file_system.c b/src/third_party/wiredtiger/examples/c/ex_file_system.c index 01e71595f3d..ad43cfe1cc0 100644 --- a/src/third_party/wiredtiger/examples/c/ex_file_system.c +++ b/src/third_party/wiredtiger/examples/c/ex_file_system.c @@ -536,9 +536,12 @@ demo_fs_size(WT_FILE_SYSTEM *file_system, WT_SESSION *session, const char *name, ret = ENOENT; lock_file_system(&demo_fs->lock); - if ((demo_fh = demo_handle_search(file_system, name)) != NULL) + if ((demo_fh = demo_handle_search(file_system, name)) != NULL) { + unlock_file_system(&demo_fs->lock); ret = demo_file_size((WT_FILE_HANDLE *)demo_fh, session, sizep); - unlock_file_system(&demo_fs->lock); + } else { + unlock_file_system(&demo_fs->lock); + } return (ret); } diff --git a/src/third_party/wiredtiger/ext/compressors/zstd/zstd_compress.c b/src/third_party/wiredtiger/ext/compressors/zstd/zstd_compress.c index 6eb55332258..e1725142982 100644 --- a/src/third_party/wiredtiger/ext/compressors/zstd/zstd_compress.c +++ b/src/third_party/wiredtiger/ext/compressors/zstd/zstd_compress.c @@ -44,6 +44,26 @@ #define inline __inline #endif +/* Default context pool size. */ +#define CONTEXT_POOL_SIZE 50 + +struct ZSTD_Context; +typedef struct ZSTD_Context ZSTD_CONTEXT; +struct ZSTD_Context { + void *ctx; /* Either a compression context or a decompression context. */ + ZSTD_CONTEXT *next; +}; + +struct ZSTD_Context_Pool; +typedef struct ZSTD_Context_Pool ZSTD_CONTEXT_POOL; +struct ZSTD_Context_Pool { + int count; /* Pool size */ + WT_EXTENSION_SPINLOCK list_lock; /* Spinlock */ + ZSTD_CONTEXT *free_ctx_list; +}; + +typedef enum { CONTEXT_TYPE_COMPRESS, CONTEXT_TYPE_DECOMPRESS } CONTEXT_TYPE; + /* Local compressor structure. */ typedef struct { WT_COMPRESSOR compressor; /* Must come first */ @@ -51,6 +71,9 @@ typedef struct { WT_EXTENSION_API *wt_api; /* Extension API */ int compression_level; /* compression level */ + + ZSTD_CONTEXT_POOL *cctx_pool; /* Compression context pool. */ + ZSTD_CONTEXT_POOL *dctx_pool; /* Decompression context pool. */ } ZSTD_COMPRESSOR; /* @@ -90,6 +113,68 @@ zstd_error(WT_COMPRESSOR *compressor, WT_SESSION *session, const char *call, siz } /* + * zstd_get_context -- + * WiredTiger Zstd get a context from context pool. + */ +static void +zstd_get_context( + ZSTD_COMPRESSOR *zcompressor, WT_SESSION *session, CONTEXT_TYPE ctx_type, ZSTD_CONTEXT **contextp) +{ + WT_EXTENSION_API *wt_api; + ZSTD_CONTEXT_POOL *ctx_pool; + + wt_api = zcompressor->wt_api; + + /* Based on the type, decide the context pool from which the context to be allocated. */ + if (ctx_type == CONTEXT_TYPE_COMPRESS) + ctx_pool = zcompressor->cctx_pool; + else + ctx_pool = zcompressor->dctx_pool; + + *contextp = NULL; + if (ctx_pool->free_ctx_list == NULL) + return; + + wt_api->spin_lock(wt_api, session, &(ctx_pool->list_lock)); + *contextp = ctx_pool->free_ctx_list; + ctx_pool->free_ctx_list = (*contextp)->next; + wt_api->spin_unlock(wt_api, session, &(ctx_pool->list_lock)); + (*contextp)->next = NULL; + + return; +} + +/* + * zstd_release_context -- + * WiredTiger Zstd release a context back to context pool. + */ +static void +zstd_release_context( + ZSTD_COMPRESSOR *zcompressor, WT_SESSION *session, CONTEXT_TYPE ctx_type, ZSTD_CONTEXT *context) +{ + WT_EXTENSION_API *wt_api; + ZSTD_CONTEXT_POOL *ctx_pool; + + if (context == NULL) + return; + + wt_api = zcompressor->wt_api; + + /* Based on the type, decide the context pool to which the context to be released back. */ + if (ctx_type == CONTEXT_TYPE_COMPRESS) + ctx_pool = zcompressor->cctx_pool; + else + ctx_pool = zcompressor->dctx_pool; + + wt_api->spin_lock(wt_api, session, &(ctx_pool->list_lock)); + context->next = ctx_pool->free_ctx_list; + ctx_pool->free_ctx_list = context; + wt_api->spin_unlock(wt_api, session, &(ctx_pool->list_lock)); + + return; +} + +/* * zstd_compress -- * WiredTiger Zstd compression. */ @@ -98,15 +183,24 @@ zstd_compress(WT_COMPRESSOR *compressor, WT_SESSION *session, uint8_t *src, size uint8_t *dst, size_t dst_len, size_t *result_lenp, int *compression_failed) { ZSTD_COMPRESSOR *zcompressor; + ZSTD_CONTEXT *context = NULL; size_t zstd_ret; uint64_t zstd_len; zcompressor = (ZSTD_COMPRESSOR *)compressor; + zstd_get_context(zcompressor, session, CONTEXT_TYPE_COMPRESS, &context); + /* Compress, starting past the prefix bytes. */ - zstd_ret = ZSTD_compress( - dst + ZSTD_PREFIX, dst_len - ZSTD_PREFIX, src, src_len, zcompressor->compression_level); + if (context != NULL) { + zstd_ret = ZSTD_compressCCtx((ZSTD_CCtx *)context->ctx, dst + ZSTD_PREFIX, + dst_len - ZSTD_PREFIX, src, src_len, zcompressor->compression_level); + } else { + zstd_ret = ZSTD_compress( + dst + ZSTD_PREFIX, dst_len - ZSTD_PREFIX, src, src_len, zcompressor->compression_level); + } + zstd_release_context(zcompressor, session, CONTEXT_TYPE_COMPRESS, context); /* * If compression succeeded and the compressed length is smaller than the original size, return * success. @@ -144,10 +238,13 @@ zstd_decompress(WT_COMPRESSOR *compressor, WT_SESSION *session, uint8_t *src, si uint8_t *dst, size_t dst_len, size_t *result_lenp) { WT_EXTENSION_API *wt_api; + ZSTD_COMPRESSOR *zcompressor; + ZSTD_CONTEXT *context = NULL; size_t zstd_ret; uint64_t zstd_len; wt_api = ((ZSTD_COMPRESSOR *)compressor)->wt_api; + zcompressor = (ZSTD_COMPRESSOR *)compressor; /* * Retrieve the saved length, handling little- to big-endian conversion as necessary. @@ -162,8 +259,20 @@ zstd_decompress(WT_COMPRESSOR *compressor, WT_SESSION *session, uint8_t *src, si return (WT_ERROR); } - zstd_ret = ZSTD_decompress(dst, dst_len, src + ZSTD_PREFIX, (size_t)zstd_len); + /* + * This type of context management is useful to avoid repeated context allocation overhead. This + * is typically for block compression, for streaming compression, context could be reused over + * and over again for performance gains. + */ + zstd_get_context(zcompressor, session, CONTEXT_TYPE_DECOMPRESS, &context); + if (context != NULL) { + zstd_ret = ZSTD_decompressDCtx( + (ZSTD_DCtx *)context->ctx, dst, dst_len, src + ZSTD_PREFIX, (size_t)zstd_len); + } else { + zstd_ret = ZSTD_decompress(dst, dst_len, src + ZSTD_PREFIX, (size_t)zstd_len); + } + zstd_release_context(zcompressor, session, CONTEXT_TYPE_DECOMPRESS, context); if (!ZSTD_isError(zstd_ret)) { *result_lenp = zstd_ret; return (0); @@ -193,14 +302,108 @@ zstd_pre_size( } /* + * zstd_init_context_pool -- + * Initialize a given type of context pool. + */ +static int +zstd_init_context_pool( + ZSTD_COMPRESSOR *zcompressor, CONTEXT_TYPE ctx_type, int count, ZSTD_CONTEXT_POOL **context_poolp) +{ + WT_EXTENSION_API *wt_api; + ZSTD_CONTEXT *context; + ZSTD_CONTEXT_POOL *context_pool; + int i, ret; + + wt_api = zcompressor->wt_api; + + /* Allocate and initialize both the context pools. */ + if ((context_pool = calloc(1, sizeof(ZSTD_CONTEXT_POOL))) == NULL) + return (errno); + + if ((ret = wt_api->spin_init(wt_api, &(context_pool->list_lock), "zstd context")) != 0) { + (void)wt_api->err_printf( + wt_api, NULL, "zstd_init_context_pool: %s", wt_api->strerror(wt_api, NULL, ret)); + return (ret); + } + context_pool->count = 0; + context_pool->free_ctx_list = NULL; + + for (i = 0; i < count; i++) { + context = NULL; + if ((context = calloc(1, sizeof(ZSTD_CONTEXT))) == NULL) { + (void)wt_api->err_printf( + wt_api, NULL, "zstd_init_context_pool: context calloc failure"); + return (errno); + } + + if (ctx_type == CONTEXT_TYPE_COMPRESS) + context->ctx = (void *)ZSTD_createCCtx(); + else + context->ctx = (void *)ZSTD_createDCtx(); + + if (context->ctx == NULL) { + (void)wt_api->err_printf( + wt_api, NULL, "zstd_init_context_pool: context create failure"); + return (errno); + } + context->next = context_pool->free_ctx_list; + context_pool->free_ctx_list = context; + context_pool->count++; + } + + *context_poolp = context_pool; + return (0); +} + +/* + * zstd_terminate_context_pool -- + * Terminate the given context pool. + */ +static void +zstd_terminate_context_pool( + WT_COMPRESSOR *compressor, CONTEXT_TYPE context_type, ZSTD_CONTEXT_POOL **context_poolp) +{ + WT_EXTENSION_API *wt_api; + ZSTD_CONTEXT *context; + ZSTD_CONTEXT_POOL *context_pool; + int i; + + wt_api = ((ZSTD_COMPRESSOR *)compressor)->wt_api; + context_pool = *context_poolp; + + for (i = 0; i < context_pool->count; i++) { + context = context_pool->free_ctx_list; + context_pool->free_ctx_list = context->next; + if (context_type == CONTEXT_TYPE_COMPRESS) + ZSTD_freeCCtx((ZSTD_CCtx *)context->ctx); + else + ZSTD_freeDCtx((ZSTD_DCtx *)context->ctx); + free(context); + context = NULL; + } + + wt_api->spin_destroy(wt_api, &(context_pool->list_lock)); + context_pool->count = 0; + free(context_pool); + *context_poolp = NULL; + return; +} + +/* * zstd_terminate -- * WiredTiger Zstd compression termination. */ static int zstd_terminate(WT_COMPRESSOR *compressor, WT_SESSION *session) { - (void)session; /* Unused parameters */ + ZSTD_COMPRESSOR *zcompressor; + + zcompressor = (ZSTD_COMPRESSOR *)compressor; + + (void)session; /* Unused parameters. */ + zstd_terminate_context_pool(compressor, CONTEXT_TYPE_COMPRESS, &(zcompressor->cctx_pool)); + zstd_terminate_context_pool(compressor, CONTEXT_TYPE_DECOMPRESS, &(zcompressor->dctx_pool)); free(compressor); return (0); } @@ -219,7 +422,6 @@ zstd_init_config(WT_CONNECTION *connection, WT_CONFIG_ARG *config, int *compress /* If configured as a built-in, there's no configuration argument. */ if (config == NULL) return (0); - /* * Zstd compression engine allows applications to specify a compression level; review the * configuration. @@ -289,6 +491,11 @@ zstd_extension_init(WT_CONNECTION *connection, WT_CONFIG_ARG *config) zstd_compressor->compression_level = compression_level; + zstd_init_context_pool( + zstd_compressor, CONTEXT_TYPE_COMPRESS, CONTEXT_POOL_SIZE, &(zstd_compressor->cctx_pool)); + zstd_init_context_pool( + zstd_compressor, CONTEXT_TYPE_DECOMPRESS, CONTEXT_POOL_SIZE, &(zstd_compressor->dctx_pool)); + /* Load the compressor */ if ((ret = connection->add_compressor( connection, "zstd", (WT_COMPRESSOR *)zstd_compressor, NULL)) == 0) diff --git a/src/third_party/wiredtiger/ext/encryptors/nop/nop_encrypt.c b/src/third_party/wiredtiger/ext/encryptors/nop/nop_encrypt.c index 1e1c2892309..2c368ebebbb 100644 --- a/src/third_party/wiredtiger/ext/encryptors/nop/nop_encrypt.c +++ b/src/third_party/wiredtiger/ext/encryptors/nop/nop_encrypt.c @@ -130,6 +130,90 @@ nop_sizing(WT_ENCRYPTOR *encryptor, WT_SESSION *session, size_t *expansion_const } /*! [WT_ENCRYPTOR sizing] */ +/*! [WT_ENCRYPTOR customize] */ +/* + * nop_customize -- + * The customize function creates a customized encryptor. + */ +static int +nop_customize(WT_ENCRYPTOR *encryptor, WT_SESSION *session, WT_CONFIG_ARG *encrypt_config, + WT_ENCRYPTOR **customp) +{ + /* + * This is how keys are set: the extension is first loaded, and then for every distinct key used + * a copy is made by calling the customize method. The original uncustomized WT_ENCRYPTOR is + * ordinarily never used to encrypt or decrypt anything. + * + * The copy, with the key installed into it, should be returned to the caller via the customp + * argument. If the customize method succeeds but sets *customp to NULL, the original encryptor + * is used for that key. + * + * The customize method need not be provided, but in that case key configuration is not + * performed, the original encryptor is used for all encryption, and it must have some other + * means to get the key or keys it should use. + */ + + const NOP_ENCRYPTOR *orig; + NOP_ENCRYPTOR *new; + WT_CONFIG_ITEM keyid, secretkey; + WT_EXTENSION_API *wt_api; + int ret; + + orig = (const NOP_ENCRYPTOR *)encryptor; + wt_api = orig->wt_api; + + /* Allocate and initialize the new encryptor. */ + if ((new = calloc(1, sizeof(*new))) == NULL) + return (errno); + *new = *orig; + + /* Get the keyid, if any. */ + ret = wt_api->config_get(wt_api, session, encrypt_config, "keyid", &keyid); + if (ret != 0) + keyid.len = 0; + + /* Get the explicit secret key, if any. */ + ret = wt_api->config_get(wt_api, session, encrypt_config, "secretkey", &secretkey); + if (ret != 0) + secretkey.len = 0; + + /* Providing both a keyid and a secretkey is an error. */ + if (keyid.len != 0 && secretkey.len != 0) { + ret = nop_error( + new, NULL, EINVAL, "nop_customize: keys specified with both keyid= and secretkey="); + goto err; + } + + /* + * Providing neither is also normally an error. Allow it here for the benefit of the test suite. + */ + if (keyid.len == 0 && secretkey.len == 0) + (void)keyid; /* do nothing */ + + if (keyid.len != 0) + /* + * Here one would contact a key manager to get the key, then install it. + */ + (void)keyid.str; /* do nothing; add code here */ + + if (secretkey.len != 0) + /* + * Here one would install the explicit secret key, probably after base64- or hex-decoding + * it. If it's a passphrase rather than a key, one might hash it first. Other + * transformations might be needed or wanted as well. + */ + (void)secretkey.str; /* do nothing; add code here */ + + /* Return the new encryptor. */ + *customp = (WT_ENCRYPTOR *)new; + return (0); + +err: + free(new); + return (ret); +} +/*! [WT_ENCRYPTOR customize] */ + /*! [WT_ENCRYPTOR terminate] */ /* * nop_terminate -- @@ -176,6 +260,7 @@ wiredtiger_extension_init(WT_CONNECTION *connection, WT_CONFIG_ARG *config) nop_encryptor->encryptor.encrypt = nop_encrypt; nop_encryptor->encryptor.decrypt = nop_decrypt; nop_encryptor->encryptor.sizing = nop_sizing; + nop_encryptor->encryptor.customize = nop_customize; nop_encryptor->encryptor.terminate = nop_terminate; nop_encryptor->wt_api = connection->get_extension_api(connection); diff --git a/src/third_party/wiredtiger/ext/encryptors/rotn/rotn_encrypt.c b/src/third_party/wiredtiger/ext/encryptors/rotn/rotn_encrypt.c index a6d5e88b529..ccf87c642b5 100644 --- a/src/third_party/wiredtiger/ext/encryptors/rotn/rotn_encrypt.c +++ b/src/third_party/wiredtiger/ext/encryptors/rotn/rotn_encrypt.c @@ -333,6 +333,10 @@ rotn_customize(WT_ENCRYPTOR *encryptor, WT_SESSION *session, WT_CONFIG_ARG *encr /* * In this demonstration, the secret key must be alphabetic characters. We stash the secret key * from the configuration string and build some shift bytes to make encryption/decryption easy. + * + * We allow specifying both a keyid and an explicit secret key (which overrides the keyid) for + * testing purposes. Under ordinary circumstances being asked to use two keys at once would be + * an error. */ if ((ret = wt_api->config_get(wt_api, session, encrypt_config, "secretkey", &secret)) == 0 && secret.len != 0) { diff --git a/src/third_party/wiredtiger/ext/encryptors/sodium/Makefile.am b/src/third_party/wiredtiger/ext/encryptors/sodium/Makefile.am new file mode 100644 index 00000000000..d46be6eb4f5 --- /dev/null +++ b/src/third_party/wiredtiger/ext/encryptors/sodium/Makefile.am @@ -0,0 +1,10 @@ +AM_CPPFLAGS = -I$(top_builddir) -I$(top_srcdir)/src/include + +if HAVE_BUILTIN_EXTENSION_SODIUM +noinst_LTLIBRARIES = libwiredtiger_sodium.la +else +lib_LTLIBRARIES = libwiredtiger_sodium.la +libwiredtiger_sodium_la_LDFLAGS = -avoid-version -module +endif +libwiredtiger_sodium_la_SOURCES = sodium_encrypt.c +libwiredtiger_sodium_la_LIBADD = -lsodium diff --git a/src/third_party/wiredtiger/ext/encryptors/sodium/sodium_encrypt.c b/src/third_party/wiredtiger/ext/encryptors/sodium/sodium_encrypt.c new file mode 100644 index 00000000000..5cbc46196fe --- /dev/null +++ b/src/third_party/wiredtiger/ext/encryptors/sodium/sodium_encrypt.c @@ -0,0 +1,411 @@ +/*- + * Public Domain 2014-present MongoDB, Inc. + * Public Domain 2008-2014 WiredTiger, Inc. + * + * This is free and unencumbered software released into the public domain. + * + * Anyone is free to copy, modify, publish, use, compile, sell, or + * distribute this software, either in source code form or as a compiled + * binary, for any purpose, commercial or non-commercial, and by any + * means. + * + * In jurisdictions that recognize copyright laws, the author or authors + * of this software dedicate any and all copyright interest in the + * software to the public domain. We make this dedication for the benefit + * of the public at large and to the detriment of our heirs and + * successors. We intend this dedication to be an overt act of + * relinquishment in perpetuity of all present and future rights to this + * software under copyright law. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. + * IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + */ + +/* + * Encryption extension using libsodium for cryptography. + * + * Note: we recommend that all applications relying on encryption for security audit their + * application and storage toolchain code, including this implementation and the underlying + * cryptographic libraries. + * + * The only threat model this extension is intended to protect against is: your database was on your + * laptop, and your laptop was stolen while the database was at rest (shut down). + * + * Because the key is necessarily kept in memory while the database is running, it is important to + * make sure it does not get written to disk by OS mechanisms; for example, core dumps and kernel + * crash dumps should be disabled, and swapping should be either disabled or set up to be encrypted. + * Also note that a still-running database on a laptop that is suspended is not at rest. + * + * This code does not, for the moment, support any form of external key server or key management + * service. Keys must be configured with "secretkey=" at database open time, and not with "keyid=". + * This is workable, but less than optimal. To add support for your favorite key service, copy this + * file, edit where indicated below, and install as your own custom extension. (See the other + * encryption examples for further information about how to do this.) + * + * The secretkey= configured at database open time is expected to be a 256-bit chacha20 key, printed + * as hex (with no leading 0x), thus 64 characters long. If you want to use a passphrase instead, + * use the recommended tools in libsodium to generate a key from a passphrase and pass the results + * as the secretkey= configuration. + */ + +#include <errno.h> + +#include <wiredtiger.h> +#include <wiredtiger_ext.h> + +#include <sodium.h> + +/* + * Lengths of the pieces involved: the secret key, the non-secret but unique nonce, and the + * cryptographic checksum added as part of the encryption. + */ +#define CHECK_LEN crypto_aead_xchacha20poly1305_ietf_ABYTES +#define KEY_LEN crypto_aead_xchacha20poly1305_ietf_KEYBYTES +#define NONCE_LEN crypto_aead_xchacha20poly1305_ietf_NPUBBYTES + +/* + * We write a header on each block that records the expected format version and the cryptographic + * construction used. This is incorporated into the block's cryptographic checksum and thus + * protected from interference. The header is 4 bytes long to keep the following data aligned, but + * we only use two of them. Currently we only support the one construction, but that could + * conceivably change and in any event it's a good idea to make persistent formats extensible. + */ +#define HEADER_BYTE_FORMATVERSION 0 +#define HEADER_BYTE_CONSTRUCTION 1 +#define HEADER_BYTE_ZERO_0 2 +#define HEADER_BYTE_ZERO_1 3 +#define HEADER_LEN 4 + +/* Constants for the on-disk (output) format. */ +#define ONDISK_CIPHER_XCHACHA20POLY1305 1 +#define ONDISK_VERSION_CURRENT 1 + +/* + * Each output block contains, in order: + * - the header + * - the nonce + * - the cryptographic output (ciphertext and checksum) + * The header and nonce are not secret but are covered by the checksum. + */ +#define HEADER_LOCATION 0 +#define NONCE_LOCATION HEADER_LEN +#define CIPHERTEXT_LOCATION (HEADER_LEN + NONCE_LEN) + +/* + * Data for this extension. Note that the secret key has to be kept in memory for use. + */ +typedef struct { + WT_ENCRYPTOR encryptor; /* Must come first */ + WT_EXTENSION_API *wt_api; /* Extension API */ + + uint8_t *secretkey; /* Secret key. (bytes) */ +} SODIUM_ENCRYPTOR; + +/* + * sodium_error -- + * Display an error from this module in a standard way. + */ +static int +sodium_error(SODIUM_ENCRYPTOR *encryptor, WT_SESSION *session, int err, const char *msg) +{ + WT_EXTENSION_API *wt_api; + + wt_api = encryptor->wt_api; + (void)wt_api->err_printf( + wt_api, session, "sodium encryption: %s: %s", msg, wt_api->strerror(wt_api, NULL, err)); + return (err); +} + +/* + * create_nonce -- + * Generate a random nonce. + */ +static void +create_nonce(uint8_t *dst, size_t len) +{ + /* + * It would be tidier to use incrementing nonces, but currently doing so would require sharing + * the current nonce between all threads and then doing global locking to use it, which is + * probably not going to work out that well. There isn't a convenient way to store per-thread + * extension state. + */ + randombytes_buf(dst, len); +} + +/* + * sodium_encrypt -- + * Encrypt using libsodium. + * + * Note that the encryption does not require that the input be padded to any particular alignment. + */ +static int +sodium_encrypt(WT_ENCRYPTOR *encryptor, WT_SESSION *session, uint8_t *cleartext, size_t clear_len, + uint8_t *ciphertext, size_t cipher_maxlen, size_t *result_lenp) +{ + SODIUM_ENCRYPTOR *sodium_encryptor = (SODIUM_ENCRYPTOR *)encryptor; + unsigned long long cipher_len; + int ret; + + (void)session; /* Unused */ + + /* Make sure it is big enough. */ + if (cipher_maxlen < HEADER_LEN + NONCE_LEN + clear_len + CHECK_LEN) + return (sodium_error(sodium_encryptor, session, ENOMEM, "encrypt buffer not big enough")); + + /* Write the header. */ + ciphertext[HEADER_LOCATION + HEADER_BYTE_FORMATVERSION] = ONDISK_VERSION_CURRENT; + ciphertext[HEADER_LOCATION + HEADER_BYTE_CONSTRUCTION] = ONDISK_CIPHER_XCHACHA20POLY1305; + ciphertext[HEADER_LOCATION + HEADER_BYTE_ZERO_0] = 0; + ciphertext[HEADER_LOCATION + HEADER_BYTE_ZERO_1] = 0; + + /* Make a nonce. */ + create_nonce(&ciphertext[NONCE_LOCATION], NONCE_LEN); + + /* Encrypt and checksum into the ciphertext part of the output. */ + ret = crypto_aead_xchacha20poly1305_ietf_encrypt(&ciphertext[CIPHERTEXT_LOCATION], &cipher_len, + cleartext, clear_len, &ciphertext[HEADER_LOCATION], HEADER_LEN, NULL, + &ciphertext[NONCE_LOCATION], sodium_encryptor->secretkey); + if (ret < 0) + return (sodium_error(sodium_encryptor, session, WT_ERROR, "encryption failed")); + + *result_lenp = HEADER_LEN + NONCE_LEN + cipher_len; + return (0); +} + +/* + * sodium_decrypt -- + * Decrypt using libsodium. + */ +static int +sodium_decrypt(WT_ENCRYPTOR *encryptor, WT_SESSION *session, uint8_t *ciphertext, size_t cipher_len, + uint8_t *cleartext, size_t clear_maxlen, size_t *result_lenp) +{ + SODIUM_ENCRYPTOR *sodium_encryptor = (SODIUM_ENCRYPTOR *)encryptor; + size_t cipher_check_only_len; + unsigned long long clear_len; + int ret; + + /* Make sure it is big enough. */ + if (HEADER_LEN + NONCE_LEN + clear_maxlen + CHECK_LEN < cipher_len) + return (sodium_error(sodium_encryptor, session, ENOMEM, "decrypt buffer not big enough")); + + /* This is the length of just the ciphertext/checksum part. */ + cipher_check_only_len = cipher_len - HEADER_LEN - NONCE_LEN; + + /* Decrypt and verify the checksum. */ + ret = crypto_aead_xchacha20poly1305_ietf_decrypt(cleartext, &clear_len, NULL, + &ciphertext[CIPHERTEXT_LOCATION], cipher_check_only_len, &ciphertext[HEADER_LOCATION], + HEADER_LEN, &ciphertext[NONCE_LOCATION], sodium_encryptor->secretkey); + if (ret < 0) + return (sodium_error(sodium_encryptor, session, WT_ERROR, "decryption failed")); + + *result_lenp = clear_len; + return (0); +} + +/* + * sodium_sizing -- + * Report how much extra space we need in the output buffer. + */ +static int +sodium_sizing(WT_ENCRYPTOR *encryptor, WT_SESSION *session, size_t *expansion_constantp) +{ + /* + * Note that the interface assumes the expansion is always a constant; for the construction + * we're using that's true, but for one based on a block cipher it might need to be rounded up + * to allow for the ciphertext part of the output always being an integer multiple of the cipher + * block size. + */ + (void)encryptor; /* Unused parameters */ + (void)session; /* Unused parameters */ + + /* Expand by the header, the nonce, and the checksum. */ + *expansion_constantp = HEADER_LEN + NONCE_LEN + CHECK_LEN; + return (0); +} + +/* + * sodium_customize -- + * The customize function creates a customized encryptor. + */ +static int +sodium_customize(WT_ENCRYPTOR *encryptor, WT_SESSION *session, WT_CONFIG_ARG *encrypt_config, + WT_ENCRYPTOR **customp) +{ + /* + * This is how keys are set: the extension is first loaded, and then for every distinct key used + * a copy is made by calling the customize method. The original uncustomized WT_ENCRYPTOR is + * never used to encrypt or decrypt anything. + */ + const SODIUM_ENCRYPTOR *orig; + SODIUM_ENCRYPTOR *new; + WT_CONFIG_ITEM keyid, secretkey; + WT_EXTENSION_API *wt_api; + size_t keylen; + int ret; + + orig = (const SODIUM_ENCRYPTOR *)encryptor; + wt_api = orig->wt_api; + + /* Allocate and initialize the new encryptor. */ + if ((new = calloc(1, sizeof(*new))) == NULL) + return (errno); + *new = *orig; + new->secretkey = NULL; + + /* Get the keyid, if any. */ + ret = wt_api->config_get(wt_api, session, encrypt_config, "keyid", &keyid); + if (ret != 0) + keyid.len = 0; + + /* Get the explicit secret key, if any. */ + ret = wt_api->config_get(wt_api, session, encrypt_config, "secretkey", &secretkey); + if (ret != 0) + secretkey.len = 0; + + /* Providing both a keyid and a secretkey is an error. */ + if (keyid.len != 0 && secretkey.len != 0) { + ret = sodium_error( + new, NULL, EINVAL, "sodium_customize: keys specified with both keyid= and secretkey="); + goto err; + } + + /* Providing neither is also an error. */ + if (keyid.len == 0 && secretkey.len == 0) { + ret = sodium_error( + new, NULL, EINVAL, "sodium_customize: no key given with either keyid= or secretkey="); + goto err; + } + + /* Use sodium_malloc, which takes assorted precautions for working with secrets. */ + new->secretkey = sodium_malloc(KEY_LEN); + if (new->secretkey == NULL) { + ret = errno; + goto err; + } + + /* + * We don't support any key lookup services. Yet. To add support for your own, change the code + * here to fetch the key associated with the configured keyid string, and put it in ->secretkey. + * If the keyid is invalid, print an error and goto err. Note that there's no need to remember + * the keyid once the key is installed. + */ + if (keyid.len != 0) { + ret = sodium_error(new, NULL, EINVAL, "sodium_customize: keyids not supported yet"); + goto err; + } + + /* + * Load the key specified with secretkey=. This should be passed as a 64-character hex string + * (with no leading 0x). It is not a passphrase; passphrases should be converted to keys using + * appropriate tools (e.g. those provided by libsodium). + */ + if (secretkey.len != 0) { + /* Explicit keys are passed as hex strings of the proper length. */ + ret = sodium_hex2bin( + new->secretkey, KEY_LEN, secretkey.str, secretkey.len, NULL, &keylen, NULL); + if (ret < 0) { + ret = sodium_error(new, NULL, EINVAL, "sodium_customize: secret key not hex"); + goto err; + } + if (keylen != KEY_LEN) { + ret = sodium_error(new, NULL, EINVAL, "sodium_customize: wrong secret key length"); + goto err; + } + } + + *customp = (WT_ENCRYPTOR *)new; + return (0); + +err: + sodium_free(new->secretkey); + free(new); + return (ret); +} + +/* + * sodium_terminate -- + * Shut down the extension and avoid leaking memory. + */ +static int +sodium_terminate(WT_ENCRYPTOR *encryptor, WT_SESSION *session) +{ + SODIUM_ENCRYPTOR *sodium_encryptor = (SODIUM_ENCRYPTOR *)encryptor; + + (void)session; /* Unused parameters */ + + /* Free the allocated memory. */ + sodium_free(sodium_encryptor->secretkey); + free(sodium_encryptor); + return (0); +} + +/* + * sodium_configure -- + * Configure the extension. + * + * This is for static config, not where keys are loaded, so nothing happens. + */ +static int +sodium_configure(SODIUM_ENCRYPTOR *sodium_encryptor, WT_CONFIG_ARG *config) +{ + WT_EXTENSION_API *wt_api; /* Extension API */ + + wt_api = sodium_encryptor->wt_api; + + (void)config; + (void)wt_api; + + return (0); +} + +/* + * wiredtiger_extension_init -- + * Called to load and initialize the extension. + */ +int +wiredtiger_extension_init(WT_CONNECTION *connection, WT_CONFIG_ARG *config) +{ + SODIUM_ENCRYPTOR *sodium_encryptor; + int ret; + + if ((sodium_encryptor = calloc(1, sizeof(SODIUM_ENCRYPTOR))) == NULL) + return (errno); + + /* + * Allocate a local encryptor structure, with a WT_ENCRYPTOR structure as the first field, + * allowing us to treat references to either type of structure as a reference to the other type. + * + * Heap memory (not static), because it can support multiple databases. + */ + sodium_encryptor->encryptor.encrypt = sodium_encrypt; + sodium_encryptor->encryptor.decrypt = sodium_decrypt; + sodium_encryptor->encryptor.sizing = sodium_sizing; + sodium_encryptor->encryptor.customize = sodium_customize; + sodium_encryptor->encryptor.terminate = sodium_terminate; + sodium_encryptor->wt_api = connection->get_extension_api(connection); + + /* Initialize the crypto library. */ + if (sodium_init() < 0) + return (WT_ERROR); + + /* Configure the extension. */ + if ((ret = sodium_configure(sodium_encryptor, config)) != 0) { + free(sodium_encryptor); + return (ret); + } + + /* Attach the extension. */ + if ((ret = connection->add_encryptor( + connection, "sodium", (WT_ENCRYPTOR *)sodium_encryptor, NULL)) != 0) { + free(sodium_encryptor); + return (ret); + } + + return (0); +} diff --git a/src/third_party/wiredtiger/ext/storage_sources/local_store/local_store.c b/src/third_party/wiredtiger/ext/storage_sources/local_store/local_store.c index 0f6a7cfe473..eeca2a83b99 100644 --- a/src/third_party/wiredtiger/ext/storage_sources/local_store/local_store.c +++ b/src/third_party/wiredtiger/ext/storage_sources/local_store/local_store.c @@ -68,6 +68,11 @@ typedef struct { pthread_rwlock_t file_handle_lock; /* + * Keep the number of references to this storage source. + */ + uint32_t reference_count; + + /* * Configuration values are set at startup. */ uint32_t delay_ms; /* Average length of delay when simulated */ @@ -129,7 +134,7 @@ static int local_writeable(LOCAL_STORAGE *, const char *name, bool *writeable); /* * Forward function declarations for storage source API implementation */ -static int local_exist(WT_FILE_SYSTEM *, WT_SESSION *, const char *, bool *); +static int local_add_reference(WT_STORAGE_SOURCE *); static int local_customize_file_system( WT_STORAGE_SOURCE *, WT_SESSION *, const char *, const char *, const char *, WT_FILE_SYSTEM **); static int local_flush( @@ -149,6 +154,7 @@ static int local_directory_list_internal( static int local_directory_list_single( WT_FILE_SYSTEM *, WT_SESSION *, const char *, const char *, char ***, uint32_t *); static int local_directory_list_free(WT_FILE_SYSTEM *, WT_SESSION *, char **, uint32_t); +static int local_exist(WT_FILE_SYSTEM *, WT_SESSION *, const char *, bool *); static int local_fs_terminate(WT_FILE_SYSTEM *, WT_SESSION *); static int local_open(WT_FILE_SYSTEM *, WT_SESSION *, const char *, WT_FS_OPEN_FILE_TYPE file_type, uint32_t, WT_FILE_HANDLE **); @@ -171,14 +177,14 @@ static int local_file_write(WT_FILE_HANDLE *, WT_SESSION *, wt_off_t, size_t, co * Report an error for a file operation. Note that local_err returns its third argument, and this * macro will too. */ -#define FS2LOCAL(fs) (((LOCAL_FILE_SYSTEM *)(fs))->local_storage) +#define WT_FS2LOCAL(fs) (((LOCAL_FILE_SYSTEM *)(fs))->local_storage) -#define VERBOSE(local, ...) \ +#define WT_VERBOSE_LS(local, ...) \ do { \ if ((local)->verbose > 0) \ fprintf(stderr, __VA_ARGS__); \ } while (0); -#define SHOW_STRING(s) (((s) == NULL) ? "<null>" : (s)) +#define WT_SHOW_STRING(s) (((s) == NULL) ? "<null>" : (s)) /* * local_configure @@ -238,15 +244,21 @@ local_delay(LOCAL_STORAGE *local) ret = 0; if (local->force_delay != 0 && local->object_flushes % local->force_delay == 0) { - VERBOSE(local, + WT_VERBOSE_LS(local, "Artificial delay %" PRIu32 " milliseconds after %" PRIu64 " object flushes\n", local->delay_ms, local->object_flushes); + /* + * tv_usec has type suseconds_t, which is signed (hence the s), but ->delay_ms is unsigned. + * In both gcc8 and gcc10 with -Wsign-conversion enabled (as we do) this causes a spurious + * warning about the implicit conversion possibly changing the value. Hence the explicit + * cast. (both struct timeval and suseconds_t are POSIX) + */ tv.tv_sec = local->delay_ms / 1000; - tv.tv_usec = (local->delay_ms % 1000) * 1000; + tv.tv_usec = (suseconds_t)(local->delay_ms % 1000) * 1000; (void)select(0, NULL, NULL, NULL, &tv); } if (local->force_error != 0 && local->object_flushes % local->force_error == 0) { - VERBOSE(local, "Artificial error returned after %" PRIu64 " object flushes\n", + WT_VERBOSE_LS(local, "Artificial error returned after %" PRIu64 " object flushes\n", local->object_flushes); ret = ENETUNREACH; } @@ -369,13 +381,34 @@ local_path(WT_FILE_SYSTEM *file_system, const char *dir, const char *name, char } len = strlen(dir) + strlen(name) + 2; if ((p = malloc(len)) == NULL) - return (local_err(FS2LOCAL(file_system), NULL, ENOMEM, "local_path")); + return (local_err(WT_FS2LOCAL(file_system), NULL, ENOMEM, "local_path")); snprintf(p, len, "%s/%s", dir, name); *pathp = p; return (ret); } /* + * local_add_reference -- + * Add a reference to the storage source so we can reference count to know when to really + * terminate. + */ +static int +local_add_reference(WT_STORAGE_SOURCE *storage_source) +{ + LOCAL_STORAGE *local; + + local = (LOCAL_STORAGE *)storage_source; + + /* + * Missing reference or overflow? + */ + if (local->reference_count == 0 || local->reference_count + 1 == 0) + return (EINVAL); + ++local->reference_count; + return (0); +} + +/* * local_customize_file_system -- * Return a customized file system to access the local storage source objects. */ @@ -486,7 +519,7 @@ local_exist(WT_FILE_SYSTEM *file_system, WT_SESSION *session, const char *name, int ret; char *path; - local = FS2LOCAL(file_system); + local = WT_FS2LOCAL(file_system); path = NULL; /* If the file exists directly in the file system, it's not yet flushed, and we're done. */ @@ -655,7 +688,7 @@ static int local_directory_list(WT_FILE_SYSTEM *file_system, WT_SESSION *session, const char *directory, const char *prefix, char ***dirlistp, uint32_t *countp) { - FS2LOCAL(file_system)->op_count++; + WT_FS2LOCAL(file_system)->op_count++; return ( local_directory_list_internal(file_system, session, directory, prefix, 0, dirlistp, countp)); } @@ -668,7 +701,7 @@ static int local_directory_list_single(WT_FILE_SYSTEM *file_system, WT_SESSION *session, const char *directory, const char *prefix, char ***dirlistp, uint32_t *countp) { - FS2LOCAL(file_system)->op_count++; + WT_FS2LOCAL(file_system)->op_count++; return ( local_directory_list_internal(file_system, session, directory, prefix, 1, dirlistp, countp)); } @@ -683,7 +716,7 @@ local_directory_list_free( { (void)session; - FS2LOCAL(file_system)->op_count++; + WT_FS2LOCAL(file_system)->op_count++; if (dirlist != NULL) { while (count > 0) free(dirlist[--count]); @@ -811,7 +844,7 @@ local_fs_terminate(WT_FILE_SYSTEM *file_system, WT_SESSION *session) (void)session; /* unused */ local_fs = (LOCAL_FILE_SYSTEM *)file_system; - FS2LOCAL(file_system)->op_count++; + WT_FS2LOCAL(file_system)->op_count++; free(local_fs->auth_token); free(local_fs->bucket_dir); free(local_fs->cache_dir); @@ -948,8 +981,8 @@ local_open(WT_FILE_SYSTEM *file_system, WT_SESSION *session, const char *name, *file_handlep = file_handle; - VERBOSE( - local, "File opened: %s final path=%s\n", SHOW_STRING(name), SHOW_STRING(local_fh->fh->name)); + WT_VERBOSE_LS(local, "File opened: %s final path=%s\n", WT_SHOW_STRING(name), + WT_SHOW_STRING(local_fh->fh->name)); err: free(alloced_path); @@ -977,7 +1010,7 @@ local_rename(WT_FILE_SYSTEM *file_system, WT_SESSION *session, const char *from, int ret; bool writeable; - local = FS2LOCAL(file_system); + local = WT_FS2LOCAL(file_system); local_fs = (LOCAL_FILE_SYSTEM *)file_system; wt_fs = local_fs->wt_fs; @@ -1013,7 +1046,7 @@ local_remove(WT_FILE_SYSTEM *file_system, WT_SESSION *session, const char *name, (void)flags; /* Unused */ - local = FS2LOCAL(file_system); + local = WT_FS2LOCAL(file_system); local->op_count++; if ((ret = local_writeable(local, name, &writeable)) != 0) @@ -1045,7 +1078,7 @@ local_size(WT_FILE_SYSTEM *file_system, WT_SESSION *session, const char *name, w int ret; char *path; - local = FS2LOCAL(file_system); + local = WT_FS2LOCAL(file_system); path = NULL; local->op_count++; @@ -1085,6 +1118,9 @@ local_terminate(WT_STORAGE_SOURCE *storage, WT_SESSION *session) ret = 0; local = (LOCAL_STORAGE *)storage; + if (--local->reference_count != 0) + return (0); + local->op_count++; /* @@ -1264,11 +1300,17 @@ wiredtiger_extension_init(WT_CONNECTION *connection, WT_CONFIG_ARG *config) * Allocate a local storage structure, with a WT_STORAGE structure as the first field, allowing * us to treat references to either type of structure as a reference to the other type. */ + local->storage_source.ss_add_reference = local_add_reference; local->storage_source.ss_customize_file_system = local_customize_file_system; local->storage_source.ss_flush = local_flush; local->storage_source.ss_flush_finish = local_flush_finish; local->storage_source.terminate = local_terminate; + /* + * The first reference is implied by the call to add_storage_source. + */ + local->reference_count = 1; + if ((ret = local_configure(local, config)) != 0) { free(local); return (ret); diff --git a/src/third_party/wiredtiger/ext/test/fail_fs/fail_fs.c b/src/third_party/wiredtiger/ext/test/fail_fs/fail_fs.c index 8a18e6b5a80..f209a33dec9 100644 --- a/src/third_party/wiredtiger/ext/test/fail_fs/fail_fs.c +++ b/src/third_party/wiredtiger/ext/test/fail_fs/fail_fs.c @@ -648,7 +648,7 @@ fail_fs_simulate_fail( { FAIL_FILE_SYSTEM *fail_fs; WT_EXTENSION_API *wtext; -#ifdef __FreeBSD__ +#if defined(__FreeBSD__) || defined(__NetBSD__) || defined(__OpenBSD__) || defined(__DragonFly__) size_t btret, i; #else int btret, i; @@ -662,7 +662,7 @@ fail_fs_simulate_fail( (void)wtext->msg_printf(wtext, session, "fail_fs: %s: simulated failure after %" PRId64 " %s operations", fail_fh->iface.name, nops, opkind); -#ifdef __FreeBSD__ +#if defined(__FreeBSD__) || defined(__NetBSD__) || defined(__OpenBSD__) || defined(__DragonFly__) btret = backtrace(bt, sizeof(bt) / sizeof(bt[0])); #else btret = backtrace(bt, (int)(sizeof(bt) / sizeof(bt[0]))); diff --git a/src/third_party/wiredtiger/import.data b/src/third_party/wiredtiger/import.data index 4686a9ef10a..3a9e0ea16cc 100644 --- a/src/third_party/wiredtiger/import.data +++ b/src/third_party/wiredtiger/import.data @@ -2,5 +2,5 @@ "vendor": "wiredtiger", "github": "wiredtiger/wiredtiger.git", "branch": "mongodb-4.4", - "commit": "5444fd4334f3b124158bfeaba88391dac1182e7d" + "commit": "2b73914cd8912fab0e01ebd67cd0106de45442cd" } diff --git a/src/third_party/wiredtiger/lang/python/wiredtiger/packing.py b/src/third_party/wiredtiger/lang/python/wiredtiger/packing.py index f2ec68c6a08..fd051b2d6c7 100755 --- a/src/third_party/wiredtiger/lang/python/wiredtiger/packing.py +++ b/src/third_party/wiredtiger/lang/python/wiredtiger/packing.py @@ -46,7 +46,7 @@ Format Python Notes s str fixed-length string S str NUL-terminated string t int fixed-length bit field - u str raw byte array + u bytes raw byte array """ from wiredtiger.packutil import _chr, _is_string, _ord, _string_result, \ diff --git a/src/third_party/wiredtiger/src/block/block_addr.c b/src/third_party/wiredtiger/src/block/block_addr.c index 181ed8ad77f..abaa0280330 100644 --- a/src/third_party/wiredtiger/src/block/block_addr.c +++ b/src/third_party/wiredtiger/src/block/block_addr.c @@ -116,8 +116,8 @@ __wt_block_addr_invalid( * In diagnostic mode, verify the address isn't on the available list, or for live systems, the * discard list. */ - WT_RET( - __wt_block_misplaced(session, block, "addr-valid", offset, size, live, __func__, __LINE__)); + WT_RET(__wt_block_misplaced( + session, block, "addr-valid", offset, size, live, __PRETTY_FUNCTION__, __LINE__)); #endif /* Check if the address is past the end of the file. */ diff --git a/src/third_party/wiredtiger/src/block/block_ckpt.c b/src/third_party/wiredtiger/src/block/block_ckpt.c index f8c06b84714..54a1f98942e 100644 --- a/src/third_party/wiredtiger/src/block/block_ckpt.c +++ b/src/third_party/wiredtiger/src/block/block_ckpt.c @@ -97,8 +97,6 @@ __wt_block_checkpoint_load(WT_SESSION_IMPL *session, WT_BLOCK *block, const uint WT_ERR(__wt_block_addr_to_buffer( block, &endp, ci->root_objectid, ci->root_offset, ci->root_size, ci->root_checksum)); *root_addr_sizep = WT_PTRDIFF(endp, root_addr); - - WT_ERR(__wt_block_tiered_load(session, block, ci)); } /* diff --git a/src/third_party/wiredtiger/src/block/block_ckpt_scan.c b/src/third_party/wiredtiger/src/block/block_ckpt_scan.c index 8b87fb5ab62..b419a42ed73 100644 --- a/src/third_party/wiredtiger/src/block/block_ckpt_scan.c +++ b/src/third_party/wiredtiger/src/block/block_ckpt_scan.c @@ -227,7 +227,7 @@ __wt_block_checkpoint_last(WT_SESSION_IMPL *session, WT_BLOCK *block, char **met *metadatap = *checkpoint_listp = NULL; WT_RET(__wt_buf_init(session, checkpoint, WT_BLOCK_CHECKPOINT_BUFFER)); - /* TODO: tiered: scan all object IDs. */ + /* Tiered tables aren't supported yet. */ objectid = 0; /* diff --git a/src/third_party/wiredtiger/src/block/block_ext.c b/src/third_party/wiredtiger/src/block/block_ext.c index 2d95ed973cd..70903985d57 100644 --- a/src/third_party/wiredtiger/src/block/block_ext.c +++ b/src/third_party/wiredtiger/src/block/block_ext.c @@ -575,15 +575,14 @@ __wt_block_free(WT_SESSION_IMPL *session, WT_BLOCK *block, const uint8_t *addr, (intmax_t)offset, (intmax_t)size); #ifdef HAVE_DIAGNOSTIC - WT_RET(__wt_block_misplaced(session, block, "free", offset, size, true, __func__, __LINE__)); + WT_RET(__wt_block_misplaced( + session, block, "free", offset, size, true, __PRETTY_FUNCTION__, __LINE__)); #endif if (objectid == block->objectid) { WT_RET(__wt_block_ext_prealloc(session, 5)); __wt_spin_lock(session, &block->live_lock); ret = __wt_block_off_free(session, block, objectid, offset, (wt_off_t)size); __wt_spin_unlock(session, &block->live_lock); - } else { - /* TODO: update stats about older files to drive garbage collection. */ } return (ret); @@ -602,7 +601,7 @@ __wt_block_off_free( /* If a sync is running, no other sessions can free blocks. */ WT_ASSERT(session, WT_SESSION_BTREE_SYNC_SAFE(session, S2BT(session))); - /* TODO: track stats for old files to drive garbage collection. */ + /* We can't reuse free space in an object. */ if (objectid != block->objectid) return (0); @@ -1164,7 +1163,7 @@ __wt_block_extlist_write( WT_EXT *ext; WT_PAGE_HEADER *dsk; size_t size; - uint32_t objectid, entries; + uint32_t entries; uint8_t *p; WT_RET(__block_extlist_dump(session, block, el, "write")); @@ -1222,8 +1221,7 @@ __wt_block_extlist_write( /* Write the extent list to disk. */ WT_ERR(__wt_block_write_off( - session, block, tmp, &objectid, &el->offset, &el->size, &el->checksum, true, true, true)); - WT_UNUSED(objectid); /* TODO: tiered: check */ + session, block, tmp, &el->objectid, &el->offset, &el->size, &el->checksum, true, true, true)); /* * Remove the allocated blocks from the system's allocation list, extent blocks never appear on diff --git a/src/third_party/wiredtiger/src/block/block_read.c b/src/third_party/wiredtiger/src/block/block_read.c index d5c1d90718a..796d071b305 100644 --- a/src/third_party/wiredtiger/src/block/block_read.c +++ b/src/third_party/wiredtiger/src/block/block_read.c @@ -95,8 +95,8 @@ __wt_bm_read( * In diagnostic mode, verify the block we're about to read isn't on the available list, or for * live systems, the discard list. */ - WT_RET( - __wt_block_misplaced(session, block, "read", offset, size, bm->is_live, __func__, __LINE__)); + WT_RET(__wt_block_misplaced( + session, block, "read", offset, size, bm->is_live, __PRETTY_FUNCTION__, __LINE__)); #endif /* Read the block. */ __wt_capacity_throttle(session, size, WT_THROTTLE_READ); diff --git a/src/third_party/wiredtiger/src/block/block_slvg.c b/src/third_party/wiredtiger/src/block/block_slvg.c index 280d520d1dd..46a87e76f10 100644 --- a/src/third_party/wiredtiger/src/block/block_slvg.c +++ b/src/third_party/wiredtiger/src/block/block_slvg.c @@ -109,7 +109,7 @@ __wt_block_salvage_next( *eofp = 0; - /* TODO: tiered: salvage across all objects in a tiered tree. */ + /* Salvage isn't implemented (yet) for tiered trees. */ objectid = 0; fh = block->fh; diff --git a/src/third_party/wiredtiger/src/block/block_tiered.c b/src/third_party/wiredtiger/src/block/block_tiered.c index ffc3a35a147..2733e9a1386 100644 --- a/src/third_party/wiredtiger/src/block/block_tiered.c +++ b/src/third_party/wiredtiger/src/block/block_tiered.c @@ -16,14 +16,21 @@ static int __block_switch_writeable(WT_SESSION_IMPL *session, WT_BLOCK *block, uint32_t object_id) { WT_DECL_RET; - - WT_ERR(__wt_close(session, &block->fh)); + WT_FH *new_fh, *old_fh; /* * FIXME-WT-7470: write lock while opening a new write handle. + * + * The block manager must always have valid file handle since other threads may have concurrent + * requests in flight. */ + old_fh = block->fh; WT_ERR(block->opener->open( - block->opener, session, object_id, WT_FS_OPEN_FILE_TYPE_DATA, block->file_flags, &block->fh)); + block->opener, session, object_id, WT_FS_OPEN_FILE_TYPE_DATA, block->file_flags, &new_fh)); + block->fh = new_fh; + block->objectid = object_id; + + WT_ERR(__wt_close(session, &old_fh)); err: return (ret); @@ -87,22 +94,3 @@ __wt_block_switch_object( */ return (__block_switch_writeable(session, block, object_id)); } - -/* - * __wt_block_tiered_load -- - * Set up object file processing when loading a new root page. - */ -int -__wt_block_tiered_load(WT_SESSION_IMPL *session, WT_BLOCK *block, WT_BLOCK_CKPT *ci) -{ - WT_UNUSED(session); - - if (block->has_objects) - block->objectid = ci->root_objectid; - - /* - * FIXME-WT-7589: There is probably more work here, perhaps in switching the current file, and - * setting the live checkpoint to the argument checkpoint. - */ - return (0); -} diff --git a/src/third_party/wiredtiger/src/block/block_write.c b/src/third_party/wiredtiger/src/block/block_write.c index ecbc3d02553..72381c5492d 100644 --- a/src/third_party/wiredtiger/src/block/block_write.c +++ b/src/third_party/wiredtiger/src/block/block_write.c @@ -298,15 +298,15 @@ __block_write_off(WT_SESSION_IMPL *session, WT_BLOCK *block, WT_ITEM *buf, uint3 blk->disk_size = WT_STORE_SIZE(align_size); /* - * Update the block's checksum: if our caller specifies, checksum the complete data, otherwise - * checksum the leading WT_BLOCK_COMPRESS_SKIP bytes. The assumption is applications with good - * compression support turn off checksums and assume corrupted blocks won't decompress - * correctly. However, if compression failed to shrink the block, the block wasn't compressed, - * in which case our caller will tell us to checksum the data to detect corruption. If - * compression succeeded, we still need to checksum the first WT_BLOCK_COMPRESS_SKIP bytes - * because they're not compressed, both to give salvage a quick test of whether a block is - * useful and to give us a test so we don't lose the first WT_BLOCK_COMPRESS_SKIP bytes without - * noticing. + * Update the block's checksum: checksum the complete data if our caller specifies, otherwise + * checksum the leading WT_BLOCK_COMPRESS_SKIP bytes. Applications with a compression or + * encryption engine that includes checksums won't need a separate checksum. However, if the + * block was too small for compression, or compression failed to shrink the block, the block + * wasn't compressed, in which case our caller will tell us to checksum the data. If skipping + * checksums because of compression or encryption, we still need to checksum the first + * WT_BLOCK_COMPRESS_SKIP bytes because they're not compressed or encrypted, both to give + * salvage a quick test of whether a block is useful and to give us a test so we don't lose the + * first WT_BLOCK_COMPRESS_SKIP bytes without noticing. * * Checksum a little-endian version of the header, and write everything in little-endian format. * The checksum is (potentially) returned in a big-endian format, swap it into place in a diff --git a/src/third_party/wiredtiger/src/btree/bt_curnext.c b/src/third_party/wiredtiger/src/btree/bt_curnext.c index b4ce5d86b4c..96ad1432c87 100644 --- a/src/third_party/wiredtiger/src/btree/bt_curnext.c +++ b/src/third_party/wiredtiger/src/btree/bt_curnext.c @@ -644,12 +644,13 @@ __wt_btcur_next_prefix(WT_CURSOR_BTREE *cbt, WT_ITEM *prefix, bool truncating) WT_DECL_RET; WT_PAGE *page; WT_SESSION_IMPL *session; - size_t total_skipped, skipped; + size_t pages_skipped_count, total_skipped, skipped; uint32_t flags; bool newpage, restart; cursor = &cbt->iface; session = CUR2S(cbt); + pages_skipped_count = 0; total_skipped = 0; WT_STAT_CONN_DATA_INCR(session, cursor_next); @@ -677,6 +678,16 @@ __wt_btcur_next_prefix(WT_CURSOR_BTREE *cbt, WT_ITEM *prefix, bool truncating) for (newpage = false;; newpage = true, restart = false) { page = cbt->ref == NULL ? NULL : cbt->ref->page; + /* + * Determine if all records on the page have been deleted and all the tombstones are visible + * to our transaction. If so, we can avoid reading the records on the page and move to the + * next page. + */ + if (__wt_btcur_skip_page(cbt)) { + pages_skipped_count++; + goto skip_page; + } + if (F_ISSET(cbt, WT_CBT_ITERATE_APPEND)) { /* The page cannot be NULL if the above flag is set. */ WT_ASSERT(session, page != NULL); @@ -730,7 +741,6 @@ __wt_btcur_next_prefix(WT_CURSOR_BTREE *cbt, WT_ITEM *prefix, bool truncating) continue; } } - /* * If we saw a lot of deleted records on this page, or we went all the way through a page * and only saw deleted records, try to evict the page when we release it. Otherwise @@ -749,7 +759,7 @@ __wt_btcur_next_prefix(WT_CURSOR_BTREE *cbt, WT_ITEM *prefix, bool truncating) WT_STAT_CONN_INCR(session, cache_eviction_force_delete); } cbt->page_deleted_count = 0; - +skip_page: if (F_ISSET(cbt, WT_CBT_READ_ONCE)) LF_SET(WT_READ_WONT_NEED); WT_ERR(__wt_tree_walk(session, &cbt->ref, flags)); @@ -757,6 +767,8 @@ __wt_btcur_next_prefix(WT_CURSOR_BTREE *cbt, WT_ITEM *prefix, bool truncating) } err: + WT_STAT_CONN_DATA_INCRV(session, cursor_next_skip_page_count, pages_skipped_count); + if (total_skipped < 100) WT_STAT_CONN_DATA_INCR(session, cursor_next_skip_lt_100); else diff --git a/src/third_party/wiredtiger/src/btree/bt_curprev.c b/src/third_party/wiredtiger/src/btree/bt_curprev.c index b1bdfebffe1..5eb2dbdbc25 100644 --- a/src/third_party/wiredtiger/src/btree/bt_curprev.c +++ b/src/third_party/wiredtiger/src/btree/bt_curprev.c @@ -590,12 +590,13 @@ __wt_btcur_prev_prefix(WT_CURSOR_BTREE *cbt, WT_ITEM *prefix, bool truncating) WT_DECL_RET; WT_PAGE *page; WT_SESSION_IMPL *session; - size_t total_skipped, skipped; + size_t pages_skipped_count, total_skipped, skipped; uint32_t flags; bool newpage, restart; cursor = &cbt->iface; session = CUR2S(cbt); + pages_skipped_count = 0; total_skipped = 0; WT_STAT_CONN_DATA_INCR(session, cursor_prev); @@ -625,6 +626,16 @@ __wt_btcur_prev_prefix(WT_CURSOR_BTREE *cbt, WT_ITEM *prefix, bool truncating) page = cbt->ref == NULL ? NULL : cbt->ref->page; /* + * Determine if all records on the page have been deleted and all the tombstones are visible + * to our transaction. If so, we can avoid reading the records on the page and move to the + * next page. + */ + if (__wt_btcur_skip_page(cbt)) { + pages_skipped_count++; + goto skip_page; + } + + /* * Column-store pages may have appended entries. Handle it separately from the usual cursor * code, it's in a simple format. */ @@ -678,7 +689,6 @@ __wt_btcur_prev_prefix(WT_CURSOR_BTREE *cbt, WT_ITEM *prefix, bool truncating) if (ret != WT_NOTFOUND) break; } - /* * If we saw a lot of deleted records on this page, or we went all the way through a page * and only saw deleted records, try to evict the page when we release it. Otherwise @@ -697,7 +707,7 @@ __wt_btcur_prev_prefix(WT_CURSOR_BTREE *cbt, WT_ITEM *prefix, bool truncating) WT_STAT_CONN_INCR(session, cache_eviction_force_delete); } cbt->page_deleted_count = 0; - +skip_page: if (F_ISSET(cbt, WT_CBT_READ_ONCE)) LF_SET(WT_READ_WONT_NEED); WT_ERR(__wt_tree_walk(session, &cbt->ref, flags)); @@ -705,6 +715,8 @@ __wt_btcur_prev_prefix(WT_CURSOR_BTREE *cbt, WT_ITEM *prefix, bool truncating) } err: + WT_STAT_CONN_DATA_INCRV(session, cursor_prev_skip_page_count, pages_skipped_count); + if (total_skipped < 100) WT_STAT_CONN_DATA_INCR(session, cursor_prev_skip_lt_100); else diff --git a/src/third_party/wiredtiger/src/btree/bt_cursor.c b/src/third_party/wiredtiger/src/btree/bt_cursor.c index 3b489da2b89..46da608e096 100644 --- a/src/third_party/wiredtiger/src/btree/bt_cursor.c +++ b/src/third_party/wiredtiger/src/btree/bt_cursor.c @@ -394,7 +394,11 @@ __cursor_row_search(WT_CURSOR_BTREE *cbt, bool insert, WT_REF *leaf, bool *leaf_ static inline int __cursor_col_modify(WT_CURSOR_BTREE *cbt, WT_ITEM *value, u_int modify_type) { +#ifdef HAVE_DIAGNOSTIC + return (__wt_col_modify(cbt, cbt->iface.recno, value, NULL, modify_type, false, false)); +#else return (__wt_col_modify(cbt, cbt->iface.recno, value, NULL, modify_type, false)); +#endif } /* @@ -404,7 +408,11 @@ __cursor_col_modify(WT_CURSOR_BTREE *cbt, WT_ITEM *value, u_int modify_type) static inline int __cursor_row_modify(WT_CURSOR_BTREE *cbt, WT_ITEM *value, u_int modify_type) { +#ifdef HAVE_DIAGNOSTIC + return (__wt_row_modify(cbt, &cbt->iface.key, value, NULL, modify_type, false, false)); +#else return (__wt_row_modify(cbt, &cbt->iface.key, value, NULL, modify_type, false)); +#endif } /* @@ -1372,17 +1380,19 @@ __cursor_chain_exceeded(WT_CURSOR_BTREE *cbt) * of 1, the total size in memory of a set of modify updates is limited to double the size of * the modifies. * - * Otherwise, limit the length of the update chain to a fixed size to bound the cost of - * rebuilding the value during reads. When history has to be maintained, creating extra copies - * of large documents multiplies cache pressure because the old ones cannot be freed, so allow - * the modify chain to grow. + * Otherwise, limit the length of the update chain to bound the cost of rebuilding the value + * during reads. When history has to be maintained, creating extra copies of large documents + * multiplies cache pressure because the old ones cannot be freed, so allow the modify chain to + * grow. */ for (i = 0, upd_size = 0; upd != NULL && upd->type == WT_UPDATE_MODIFY; ++i, upd = upd->next) { + if (i >= WT_MODIFY_UPDATE_MAX) + return (true); upd_size += WT_UPDATE_MEMSIZE(upd); - if (i >= WT_MAX_MODIFY_UPDATE && upd_size * WT_MODIFY_MEM_FRACTION >= cursor->value.size) + if (i >= WT_MODIFY_UPDATE_MIN && upd_size * WT_MODIFY_MEM_FRACTION >= cursor->value.size) return (true); } - if (i >= WT_MAX_MODIFY_UPDATE && upd != NULL && upd->type == WT_UPDATE_STANDARD && + if (i >= WT_MODIFY_UPDATE_MIN && upd != NULL && upd->type == WT_UPDATE_STANDARD && __wt_txn_upd_visible_all(session, upd)) return (true); return (false); diff --git a/src/third_party/wiredtiger/src/btree/bt_debug.c b/src/third_party/wiredtiger/src/btree/bt_debug.c index 1093ed26b57..0e8daa06e34 100644 --- a/src/third_party/wiredtiger/src/btree/bt_debug.c +++ b/src/third_party/wiredtiger/src/btree/bt_debug.c @@ -849,10 +849,10 @@ __wt_debug_tree_shape(WT_SESSION_IMPL *session, WT_REF *ref, const char *ofile) return (ret); } -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_DEBUG_TREE_LEAF 0x1u /* Debug leaf pages */ #define WT_DEBUG_TREE_WALK 0x2u /* Descend the tree */ - /* AUTOMATIC FLAG VALUE GENERATION STOP */ +/* AUTOMATIC FLAG VALUE GENERATION STOP 32 */ /* * __wt_debug_tree_all -- diff --git a/src/third_party/wiredtiger/src/btree/bt_delete.c b/src/third_party/wiredtiger/src/btree/bt_delete.c index a4d82c3d904..acb89293002 100644 --- a/src/third_party/wiredtiger/src/btree/bt_delete.c +++ b/src/third_party/wiredtiger/src/btree/bt_delete.c @@ -20,16 +20,16 @@ * state to WT_REF_DELETED. Pages ineligible for this fast path include pages already in the cache, * having overflow items, or requiring history store records. Ineligible pages are read and have * their rows updated/deleted individually. The transaction for the delete operation is stored in - * memory referenced by the WT_REF.page_del field. + * memory referenced by the WT_REF.ft_info.del field. * * Future cursor walks of the tree will skip the deleted page based on the transaction stored for * the delete, but it gets more complicated if a read is done using a random key, or a cursor walk * is done with a transaction where the delete is not visible. In those cases, we read the original * contents of the page. The page-read code notices a deleted page is being read, and as part of the - * read instantiates the contents of the page, creating a WT_UPDATE with a deleted operation, in the - * same transaction as deleted the page. In other words, the read process makes it appear as if the - * page was read and each individual row deleted, exactly as would have happened if the page had - * been in the cache all along. + * read instantiates the contents of the page, creating a WT_UPDATE with a tombstone, in the same + * transaction as deleted the page. In other words, the read process makes it appear as if the page + * was read and each individual row deleted, exactly as would have happened if the page had been in + * the cache all along. * * There's an additional complication to support rollback of the page delete. When the page was * marked deleted, a pointer to the WT_REF was saved in the deleting session's transaction list and @@ -39,14 +39,14 @@ * saved/restored during reconciliation and appear on multiple pages, and the WT_REF stored in the * deleting session's transaction list is no longer useful. For this reason, when the page is * instantiated by a read, a list of the WT_UPDATE structures on the page is stored in the - * WT_REF.page_del field, with the transaction ID, that way the session committing/unrolling the - * delete can find all WT_UPDATE structures that require update. + * WT_REF.ft_info.update field, that way the session resolving the delete can find all WT_UPDATE + * structures that require update. * * One final note: pages can also be marked deleted if emptied and evicted. In that case, the WT_REF - * state will be set to WT_REF_DELETED but there will not be any associated WT_REF.page_del field. - * These pages are always skipped during cursor traversal (the page could not have been evicted if - * there were updates that weren't globally visible), and if read is forced to instantiate such a - * page, it simply creates an empty page from scratch. + * state will be set to WT_REF_DELETED but there will not be any associated WT_REF.ft_info.del + * field. These pages are always skipped during cursor traversal (the page could not have been + * evicted if there were updates that weren't globally visible), and if read is forced to + * instantiate such a page, it simply creates an empty page from scratch. */ /* @@ -93,16 +93,12 @@ __wt_delete_page(WT_SESSION_IMPL *session, WT_REF *ref, bool *skipp) return (0); /* - * If this WT_REF was previously part of a truncate operation, there may be existing page-delete - * information. The structure is only read while the state is locked, free the previous version. - * - * Note: changes have been made, we must publish any state change from this point on. + * There should be no previous page-delete information: if the previous fast-truncate didn't + * instantiate the page, then we'd never get here to do another delete; if the previous fast- + * truncate did instantiate the page, then any fast-truncate information was removed at that + * point and/or when the fast-truncate transaction was resolved. */ - if (ref->page_del != NULL) { - WT_ASSERT(session, ref->page_del->txnid == WT_TXN_ABORTED); - __wt_free(session, ref->page_del->update_list); - __wt_free(session, ref->page_del); - } + WT_ASSERT(session, ref->ft_info.del == NULL); /* * We cannot truncate pages that have overflow key/value items as the overflow blocks have to be @@ -128,8 +124,8 @@ __wt_delete_page(WT_SESSION_IMPL *session, WT_REF *ref, bool *skipp) WT_ERR(__wt_page_parent_modify_set(session, ref, false)); /* Allocate and initialize the page-deleted structure. */ - WT_ERR(__wt_calloc_one(session, &ref->page_del)); - ref->page_del->previous_state = previous_state; + WT_ERR(__wt_calloc_one(session, &ref->ft_info.del)); + ref->ft_info.del->previous_state = previous_state; WT_ERR(__wt_txn_modify_page_delete(session, ref)); @@ -141,7 +137,7 @@ __wt_delete_page(WT_SESSION_IMPL *session, WT_REF *ref, bool *skipp) return (0); err: - __wt_free(session, ref->page_del); + __wt_free(session, ref->ft_info.del); /* Publish the page to its previous state, ensuring visibility. */ WT_REF_SET_STATE(ref, previous_state); @@ -150,7 +146,7 @@ err: /* * __wt_delete_page_rollback -- - * Abort pages that were deleted without being instantiated. + * Abort fast-truncate operations. */ int __wt_delete_page_rollback(WT_SESSION_IMPL *session, WT_REF *ref) @@ -160,7 +156,7 @@ __wt_delete_page_rollback(WT_SESSION_IMPL *session, WT_REF *ref) uint8_t current_state; bool locked; - /* Lock the reference. We cannot access ref->page_del except when locked. */ + /* Lock the reference. We cannot access ref->ft_info.del except when locked. */ for (locked = false, sleep_usecs = yield_count = 0;;) { switch (current_state = ref->state) { case WT_REF_LOCKED: @@ -188,23 +184,25 @@ __wt_delete_page_rollback(WT_SESSION_IMPL *session, WT_REF *ref) } /* - * If the page is still "deleted", it's as we left it, all we have to do is reset the state. - * - * We can't use the normal read path to get a copy of the page because the session may have - * closed the cursor, we no longer have the reference to the tree required for a hazard pointer. - * We're safe because with unresolved transactions, the page isn't going anywhere. - * - * The page is in an in-memory state, which means it was instantiated at some point. Walk any - * list of update structures and abort them. + * If the page is still "deleted", it's as we left it, simply reset the state. Otherwise, the + * page is in an in-memory state, which means it was instantiated at some point. Walk any list + * of update structures and abort them. We can't use the normal read path to get the pages with + * updates (the original page may have split, so there many be more than one page), because the + * session may have closed the cursor, we no longer have the reference to the tree required for + * a hazard pointer. We're safe since pages with unresolved transactions aren't going anywhere. */ if (current_state == WT_REF_DELETED) - current_state = ref->page_del->previous_state; - else if ((updp = ref->page_del->update_list) != NULL) + current_state = ref->ft_info.del->previous_state; + else if ((updp = ref->ft_info.update) != NULL) for (; *updp != NULL; ++updp) (*updp)->txnid = WT_TXN_ABORTED; - /* Finally mark the truncate aborted */ - ref->page_del->txnid = WT_TXN_ABORTED; + /* + * We didn't set the WT_PAGE_DELETED transaction ID to aborted or discard any WT_UPDATE list, + * instead, we discard both structures entirely, it has the same effect. It's a single call, + * they're a union of two pointers. + */ + __wt_free(session, ref->ft_info.del); WT_REF_SET_STATE(ref, current_state); return (0); @@ -240,16 +238,14 @@ __wt_delete_page_skip(WT_SESSION_IMPL *session, WT_REF *ref, bool visible_all) skip = !__wt_page_del_active(session, ref, visible_all); /* - * The page_del structure can be freed as soon as the delete is stable: it is only read when the - * ref state is locked. It is worth checking every time we come through because once this is - * freed, we no longer need synchronization to check the ref. + * The fast-truncate structure can be freed as soon as the delete is stable: it is only read + * when the ref state is locked. It is worth checking every time we come through because once + * this is freed, we no longer need synchronization to check the ref. */ - if (skip && ref->page_del != NULL && + if (skip && ref->ft_info.del != NULL && (visible_all || - __wt_txn_visible_all(session, ref->page_del->txnid, ref->page_del->timestamp))) { - __wt_free(session, ref->page_del->update_list); - __wt_free(session, ref->page_del); - } + __wt_txn_visible_all(session, ref->ft_info.del->txnid, ref->ft_info.del->timestamp))) + __wt_overwrite_and_free(session, ref->ft_info.del); WT_REF_SET_STATE(ref, WT_REF_DELETED); return (skip); @@ -296,12 +292,14 @@ __wt_delete_page_instantiate(WT_SESSION_IMPL *session, WT_REF *ref) WT_PAGE_DELETED *page_del; WT_ROW *rip; WT_TIME_WINDOW tw; - WT_UPDATE **upd_array, *upd; + WT_UPDATE **upd_array, **update_list, *upd; size_t size, total_size; uint32_t count, i; btree = S2BT(session); page = ref->page; + page_del = NULL; + update_list = NULL; WT_STAT_CONN_DATA_INCR(session, cache_read_deleted); @@ -315,7 +313,15 @@ __wt_delete_page_instantiate(WT_SESSION_IMPL *session, WT_REF *ref) if (!F_ISSET(btree, WT_BTREE_READONLY)) __wt_page_modify_set(session, page); - if (ref->page_del != NULL && ref->page_del->prepare_state != WT_PREPARE_INIT) + /* + * Allocate the per-page update array if one doesn't already exist. (It might already exist + * because deletes are instantiated after the history store table updates.) + */ + if (page->entries != 0 && page->modify->mod_row_update == NULL) + WT_PAGE_ALLOC_AND_SWAP( + session, page, page->modify->mod_row_update, upd_array, page->entries); + + if (ref->ft_info.del != NULL && ref->ft_info.del->prepare_state != WT_PREPARE_INIT) WT_STAT_CONN_DATA_INCR(session, cache_read_deleted_prepared); /* @@ -332,26 +338,16 @@ __wt_delete_page_instantiate(WT_SESSION_IMPL *session, WT_REF *ref) * in the system forced us to keep the old version of the page around, then we crashed and * recovered or we're running inside a checkpoint, and now we're being forced to read that page. * - * Expect a page-deleted structure if there's a running transaction that needs to be resolved, - * otherwise, there may not be one (and, if the transaction has resolved, we can ignore the - * page-deleted structure). - */ - page_del = __wt_page_del_active(session, ref, true) ? ref->page_del : NULL; - - /* - * Allocate the per-page update array if one doesn't already exist. (It might already exist - * because deletes are instantiated after the history store table updates.) - */ - if (page->entries != 0 && page->modify->mod_row_update == NULL) - WT_PAGE_ALLOC_AND_SWAP( - session, page, page->modify->mod_row_update, upd_array, page->entries); - - /* - * Allocate the per-reference update array; in the case of instantiating a page deleted in a - * running transaction, we need a list of the update structures for the eventual commit or - * abort. + * If there's a page-deleted structure that's not yet globally visible, get a reference and + * migrate transaction ID and timestamp information to the updates (globally visible means the + * updates don't require that information). + * + * If the truncate operation is not yet resolved, link updates in the page-deleted structure so + * they can be found when the transaction is aborted or committed, even if they have moved to + * other pages. */ - if (page_del != NULL) { + page_del = __wt_page_del_active(session, ref, true) ? ref->ft_info.del : NULL; + if (page_del != NULL && page_del->committed == 0) { count = 0; if ((insert = WT_ROW_INSERT_SMALLEST(page)) != NULL) WT_SKIP_FOREACH (ins, insert) @@ -362,12 +358,11 @@ __wt_delete_page_instantiate(WT_SESSION_IMPL *session, WT_REF *ref) WT_SKIP_FOREACH (ins, insert) ++count; } - WT_RET(__wt_calloc_def(session, count + 1, &page_del->update_list)); - __wt_cache_page_inmem_incr(session, page, (count + 1) * sizeof(page_del->update_list)); + WT_RET(__wt_calloc_def(session, count + 1, &update_list)); } /* Walk the page entries, giving each one a tombstone. */ - size = total_size = 0; + total_size = size = 0; count = 0; upd_array = page->modify->mod_row_update; if ((insert = WT_ROW_INSERT_SMALLEST(page)) != NULL) @@ -377,8 +372,8 @@ __wt_delete_page_instantiate(WT_SESSION_IMPL *session, WT_REF *ref) upd->next = ins->upd; ins->upd = upd; - if (page_del != NULL) - page_del->update_list[count++] = upd; + if (update_list != NULL) + update_list[count++] = upd; } WT_ROW_FOREACH (page, rip, i) { /* @@ -392,8 +387,8 @@ __wt_delete_page_instantiate(WT_SESSION_IMPL *session, WT_REF *ref) upd->next = upd_array[WT_ROW_SLOT(page, rip)]; upd_array[WT_ROW_SLOT(page, rip)] = upd; - if (page_del != NULL) - page_del->update_list[count++] = upd; + if (update_list != NULL) + update_list[count++] = upd; if ((insert = WT_ROW_INSERT(page, rip)) != NULL) WT_SKIP_FOREACH (ins, insert) { @@ -402,23 +397,24 @@ __wt_delete_page_instantiate(WT_SESSION_IMPL *session, WT_REF *ref) upd->next = ins->upd; ins->upd = upd; - if (page_del != NULL) - page_del->update_list[count++] = upd; + if (update_list != NULL) + update_list[count++] = upd; } } } - __wt_cache_page_inmem_incr(session, page, total_size); + /* + * We no longer need the WT_PAGE_DELETED structure, all of its information should have been + * transferred to the list of WT_UPDATE structures (if any). + */ + __wt_overwrite_and_free(session, ref->ft_info.del); + if (update_list != NULL) + ref->ft_info.update = update_list; + return (0); err: - /* - * The page-delete update structure may have existed before we were called, and presumably might - * be in use by a running transaction. The list of update structures cannot have been created - * before we were called, and should not exist if we exit with an error. - */ - if (page_del != NULL) - __wt_free(session, page_del->update_list); + __wt_free(session, update_list); return (ret); } diff --git a/src/third_party/wiredtiger/src/btree/bt_discard.c b/src/third_party/wiredtiger/src/btree/bt_discard.c index 02bd970e0c6..2e8e333f9e6 100644 --- a/src/third_party/wiredtiger/src/btree/bt_discard.c +++ b/src/third_party/wiredtiger/src/btree/bt_discard.c @@ -38,6 +38,7 @@ __wt_ref_out(WT_SESSION_IMPL *session, WT_REF *ref) */ WT_ASSERT(session, __wt_hazard_check_assert(session, ref, true)); + /* Check we are not evicting an accessible internal page with an active split generation. */ WT_ASSERT(session, !F_ISSET(ref, WT_REF_FLAG_INTERNAL) || F_ISSET(session->dhandle, WT_DHANDLE_DEAD | WT_DHANDLE_EXCLUSIVE) || @@ -290,11 +291,8 @@ __wt_free_ref(WT_SESSION_IMPL *session, WT_REF *ref, int page_type, bool free_pa /* Free any address allocation. */ __wt_ref_addr_free(session, ref); - /* Free any page-deleted information. */ - if (ref->page_del != NULL) { - __wt_free(session, ref->page_del->update_list); - __wt_free(session, ref->page_del); - } + /* Free any backing fast-truncate memory. */ + __wt_free(session, ref->ft_info.del); __wt_overwrite_and_free_len(session, ref, WT_REF_CLEAR_SIZE); } diff --git a/src/third_party/wiredtiger/src/btree/bt_handle.c b/src/third_party/wiredtiger/src/btree/bt_handle.c index 81a580e9829..fdbb192774c 100644 --- a/src/third_party/wiredtiger/src/btree/bt_handle.c +++ b/src/third_party/wiredtiger/src/btree/bt_handle.c @@ -416,8 +416,10 @@ __btree_conf(WT_SESSION_IMPL *session, WT_CKPT *ckpt) btree->checksum = CKSUM_ON; else if (WT_STRING_MATCH("off", cval.str, cval.len)) btree->checksum = CKSUM_OFF; - else + else if (WT_STRING_MATCH("uncompressed", cval.str, cval.len)) btree->checksum = CKSUM_UNCOMPRESSED; + else + btree->checksum = CKSUM_UNENCRYPTED; /* Huffman encoding */ WT_RET(__wt_btree_huffman_open(session)); diff --git a/src/third_party/wiredtiger/src/btree/bt_io.c b/src/third_party/wiredtiger/src/btree/bt_io.c index 3b763a68172..385968a8a2c 100644 --- a/src/third_party/wiredtiger/src/btree/bt_io.c +++ b/src/third_party/wiredtiger/src/btree/bt_io.c @@ -35,16 +35,12 @@ __wt_bt_read(WT_SESSION_IMPL *session, WT_ITEM *buf, const uint8_t *addr, size_t * into the caller's buffer. Else, read directly into the caller's buffer. */ if (btree->compressor == NULL && btree->kencryptor == NULL) { - WT_WITH_BUCKET_STORAGE( - btree->bstorage, session, { ret = bm->read(bm, session, buf, addr, addr_size); }); - WT_RET(ret); + WT_RET(bm->read(bm, session, buf, addr, addr_size)); dsk = buf->data; ip = NULL; } else { WT_RET(__wt_scr_alloc(session, 0, &tmp)); - WT_WITH_BUCKET_STORAGE( - btree->bstorage, session, { ret = bm->read(bm, session, tmp, addr, addr_size); }); - WT_ERR(ret); + WT_ERR(bm->read(bm, session, tmp, addr, addr_size)); dsk = tmp->data; ip = tmp; } @@ -60,6 +56,10 @@ __wt_bt_read(WT_SESSION_IMPL *session, WT_ITEM *buf, const uint8_t *addr, size_t goto corrupt; } + /* + * If checksums were turned off because we're depending on decryption to fail on any + * corrupted data, we'll end up here on corrupted data. + */ WT_ERR(__wt_scr_alloc(session, 0, &etmp)); if ((ret = __wt_decrypt(session, encryptor, WT_BLOCK_ENCRYPT_SKIP, ip, etmp)) != 0) { fail_msg = "block decryption failed"; @@ -98,9 +98,8 @@ __wt_bt_read(WT_SESSION_IMPL *session, WT_ITEM *buf, const uint8_t *addr, size_t &result_len); /* - * If checksums were turned off because we're depending on the decompression to fail on any - * corrupted data, we'll end up here after corruption happens. If we're salvaging the file, - * it's OK, otherwise it's really, really bad. + * If checksums were turned off because we're depending on decompression to fail on any + * corrupted data, we'll end up here on corrupted data. */ if (ret != 0 || result_len != dsk->mem_size - WT_BLOCK_COMPRESS_SKIP) { fail_msg = "block decompression failed"; @@ -314,9 +313,7 @@ __wt_bt_write(WT_SESSION_IMPL *session, WT_ITEM *buf, uint8_t *addr, size_t *add */ WT_ASSERT(session, dsk->write_gen != 0); - /* - * Checksum the data if the buffer isn't compressed or checksums are configured. - */ + /* Determine if the data requires a checksum. */ WT_NOT_READ(data_checksum, true); switch (btree->checksum) { case CKSUM_ON: @@ -328,18 +325,17 @@ __wt_bt_write(WT_SESSION_IMPL *session, WT_ITEM *buf, uint8_t *addr, size_t *add case CKSUM_UNCOMPRESSED: data_checksum = !compressed; break; + case CKSUM_UNENCRYPTED: + data_checksum = !encrypted; + break; } timer = !F_ISSET(session, WT_SESSION_INTERNAL); if (timer) time_start = __wt_clock(session); - WT_WITH_BUCKET_STORAGE(btree->bstorage, session, { - /* Call the block manager to write the block. */ - ret = - (checkpoint ? bm->checkpoint(bm, session, ip, btree->ckpt, data_checksum) : + /* Call the block manager to write the block. */ + WT_ERR(checkpoint ? bm->checkpoint(bm, session, ip, btree->ckpt, data_checksum) : bm->write(bm, session, ip, addr, addr_sizep, data_checksum, checkpoint_io)); - }); - WT_ERR(ret); /* Update some statistics now that the write is done */ if (timer) { diff --git a/src/third_party/wiredtiger/src/btree/bt_slvg.c b/src/third_party/wiredtiger/src/btree/bt_slvg.c index 5608242a5dd..7db2b39d486 100644 --- a/src/third_party/wiredtiger/src/btree/bt_slvg.c +++ b/src/third_party/wiredtiger/src/btree/bt_slvg.c @@ -110,13 +110,13 @@ struct __wt_track { } col; } u; -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_TRACK_CHECK_START 0x1u /* Row: initial key updated */ #define WT_TRACK_CHECK_STOP 0x2u /* Row: last key updated */ #define WT_TRACK_MERGE 0x4u /* Page requires merging */ #define WT_TRACK_OVFL_REFD 0x8u /* Overflow page referenced */ - /* AUTOMATIC FLAG VALUE GENERATION STOP */ - u_int flags; + /* AUTOMATIC FLAG VALUE GENERATION STOP 32 */ + uint32_t flags; }; static int __slvg_cleanup(WT_SESSION_IMPL *, WT_STUFF *); @@ -134,6 +134,7 @@ static int __slvg_ovfl_reconcile(WT_SESSION_IMPL *, WT_STUFF *); static int __slvg_ovfl_ref(WT_SESSION_IMPL *, WT_TRACK *, bool); static int __slvg_ovfl_ref_all(WT_SESSION_IMPL *, WT_TRACK *); static int __slvg_read(WT_SESSION_IMPL *, WT_STUFF *); +static int __slvg_reconcile_free(WT_BM *, WT_SESSION_IMPL *, const uint8_t *, size_t); static int __slvg_row_build_internal(WT_SESSION_IMPL *, uint32_t, WT_STUFF *); static int __slvg_row_build_leaf(WT_SESSION_IMPL *, WT_TRACK *, WT_REF *, WT_STUFF *); static int __slvg_row_ovfl(WT_SESSION_IMPL *, WT_TRACK *, WT_PAGE *, uint32_t, uint32_t); @@ -214,7 +215,7 @@ __slvg_checkpoint(WT_SESSION_IMPL *session, WT_REF *root) if (ckptbase->raw.data == NULL) WT_TRET(__wt_meta_checkpoint_clear(session, dhandle->name)); else - WT_ERR(__wt_meta_ckptlist_set(session, dhandle->name, ckptbase, NULL)); + WT_ERR(__wt_meta_ckptlist_set(session, dhandle, ckptbase, NULL)); err: __wt_meta_ckptlist_free(session, &ckptbase); @@ -590,6 +591,7 @@ __slvg_trk_leaf(WT_SESSION_IMPL *session, const WT_PAGE_HEADER *dsk, uint8_t *ad * Column-store fixed-sized format: start and stop keys can be taken from the block's * header, and doesn't contain overflow items. */ + WT_TIME_AGGREGATE_INIT(&trk->trk_ta); trk->col_start = dsk->recno; trk->col_stop = dsk->recno + (dsk->u.entries - 1); @@ -598,12 +600,11 @@ __slvg_trk_leaf(WT_SESSION_IMPL *session, const WT_PAGE_HEADER *dsk, uint8_t *ad trk->col_stop); break; case WT_PAGE_COL_VAR: - WT_TIME_AGGREGATE_INIT_MERGE(&trk->trk_ta); - /* * Column-store variable-length format: the start key can be taken from the block's header, * stop key requires walking the page. */ + WT_TIME_AGGREGATE_INIT_MERGE(&trk->trk_ta); stop_recno = dsk->recno; WT_CELL_FOREACH_KV (session, dsk, unpack) { stop_recno += __wt_cell_rle(&unpack); @@ -1247,7 +1248,7 @@ __slvg_col_build_leaf(WT_SESSION_IMPL *session, WT_TRACK *trk, WT_REF *ref) __wt_addr_string(session, trk->trk_addr, trk->trk_addr_size, trk->ss->tmp1), skip, take); /* Set the referenced flag on overflow pages we're using. */ - if (page->type == WT_PAGE_COL_VAR && trk->trk_ovfl_cnt != 0) + if (trk->trk_ovfl_cnt != 0) WT_ERR(__slvg_col_ovfl(session, trk, page, recno, skip, take)); /* @@ -1282,6 +1283,7 @@ __slvg_col_build_leaf(WT_SESSION_IMPL *session, WT_TRACK *trk, WT_REF *ref) page->pg_var = save_col_var; page->entries = save_entries; + /* Discard our hazard pointer and evict the page, updating the parent's reference. */ ret = __wt_page_release(session, ref, 0); if (ret == 0) ret = __wt_evict(session, ref, WT_REF_MEM, WT_EVICT_CALL_CLOSING); @@ -1836,10 +1838,11 @@ __slvg_row_build_leaf(WT_SESSION_IMPL *session, WT_TRACK *trk, WT_REF *ref, WT_S WT_ROW *rip; WT_SALVAGE_COOKIE *cookie, _cookie; uint32_t i, skip_start, skip_stop; - int cmp; + int cmp, (*saved_free)(WT_BM *, WT_SESSION_IMPL *, const uint8_t *, size_t); btree = S2BT(session); page = NULL; + saved_free = NULL; cookie = &_cookie; WT_CLEAR(*cookie); @@ -1926,6 +1929,18 @@ __slvg_row_build_leaf(WT_SESSION_IMPL *session, WT_TRACK *trk, WT_REF *ref, WT_S */ __wt_ref_addr_free(session, ref); + /* + * Reconciliation may skip a key/value pair (based on timestamps), and in that case, if the + * key/value is an overflow item, reconciliation will free the underlying object's backing + * blocks. That's a problem when merging pages if the key is an overflow item: if we're + * processing a page multiple times to handle overlapping ranges, and if the first build and + * reconcile removes the overflow key, the second build/reconcile will fail when it can't read + * the key. Intercept any attempt by reconciliation to free blocks. + */ + saved_free = btree->bm->free; + btree->bm->free = __slvg_reconcile_free; + session->salvage_track = trk; + /* Write the new version of the leaf page to disk. */ WT_ERR(__slvg_modify_init(session, page)); WT_ERR(__wt_reconcile(session, ref, cookie, WT_REC_VISIBILITY_ERR)); @@ -1933,9 +1948,7 @@ __slvg_row_build_leaf(WT_SESSION_IMPL *session, WT_TRACK *trk, WT_REF *ref, WT_S /* Reset the page. */ page->entries += skip_stop; - /* - * Discard our hazard pointer and evict the page, updating the parent's reference. - */ + /* Discard our hazard pointer and evict the page, updating the parent's reference. */ ret = __wt_page_release(session, ref, 0); if (ret == 0) ret = __wt_evict(session, ref, WT_REF_MEM, WT_EVICT_CALL_CLOSING); @@ -1944,6 +1957,10 @@ __slvg_row_build_leaf(WT_SESSION_IMPL *session, WT_TRACK *trk, WT_REF *ref, WT_S err: WT_TRET(__wt_page_release(session, ref, 0)); } + if (saved_free != NULL) { + btree->bm->free = saved_free; + session->salvage_track = NULL; + } __wt_scr_free(session, &key); return (ret); @@ -2006,6 +2023,39 @@ __slvg_row_ovfl( } /* + * __slvg_reconcile_free -- + * Block manager replacement to update blocks reconciliation wants removed. + */ +static int +__slvg_reconcile_free(WT_BM *bm, WT_SESSION_IMPL *session, const uint8_t *addr, size_t addr_size) +{ + WT_TRACK *ovfl, *trk; + uint32_t i; + + WT_UNUSED(bm); + trk = session->salvage_track; + + /* + * Search the list of overflow records for this page -- we should find exactly one referenced + * match, and we clear that reference. + */ + for (i = 0; i < trk->trk_ovfl_cnt; ++i) { + ovfl = trk->ss->ovfl[trk->trk_ovfl_slot[i]]; + if (addr_size == ovfl->trk_addr_size && memcmp(addr, ovfl->trk_addr, addr_size) == 0) { + if (!F_ISSET(ovfl, WT_TRACK_OVFL_REFD)) + break; + + F_CLR(ovfl, WT_TRACK_OVFL_REFD); + return (0); + } + } + + WT_RET_PANIC(session, EINVAL, + "overflow record discarded by reconciliation during row-store page merge not %s", + i == trk->trk_ovfl_cnt ? "referenced" : "found"); +} + +/* * __slvg_trk_compare_addr -- * Compare two WT_TRACK array entries by address cookie. */ diff --git a/src/third_party/wiredtiger/src/btree/bt_split.c b/src/third_party/wiredtiger/src/btree/bt_split.c index b5b997054ef..c3af3cbb2e4 100644 --- a/src/third_party/wiredtiger/src/btree/bt_split.c +++ b/src/third_party/wiredtiger/src/btree/bt_split.c @@ -608,14 +608,8 @@ __split_parent_discard_ref(WT_SESSION_IMPL *session, WT_REF *ref, WT_PAGE *paren } } - /* - * The page-delete and history store memory weren't added to the parent's footprint, ignore it - * here. - */ - if (ref->page_del != NULL) { - __wt_free(session, ref->page_del->update_list); - __wt_free(session, ref->page_del); - } + /* Free any backing fast-truncate memory. */ + __wt_free(session, ref->ft_info.del); /* Free the backing block and address. */ WT_TRET(__wt_ref_block_free(session, ref)); @@ -1472,7 +1466,11 @@ __split_multi_inmem(WT_SESSION_IMPL *session, WT_PAGE *orig, WT_MULTI *multi, WT WT_ERR(__wt_col_search(&cbt, recno, ref, true, NULL)); /* Apply the modification. */ +#ifdef HAVE_DIAGNOSTIC + WT_ERR(__wt_col_modify(&cbt, recno, NULL, upd, WT_UPDATE_INVALID, true, true)); +#else WT_ERR(__wt_col_modify(&cbt, recno, NULL, upd, WT_UPDATE_INVALID, true)); +#endif break; case WT_PAGE_ROW_LEAF: /* Build a key. */ @@ -1487,7 +1485,11 @@ __split_multi_inmem(WT_SESSION_IMPL *session, WT_PAGE *orig, WT_MULTI *multi, WT WT_ERR(__wt_row_search(&cbt, key, true, ref, true, NULL)); /* Apply the modification. */ +#ifdef HAVE_DIAGNOSTIC + WT_ERR(__wt_row_modify(&cbt, key, NULL, upd, WT_UPDATE_INVALID, true, true)); +#else WT_ERR(__wt_row_modify(&cbt, key, NULL, upd, WT_UPDATE_INVALID, true)); +#endif break; default: WT_ERR(__wt_illegal_value(session, orig->type)); @@ -1717,15 +1719,15 @@ __wt_multi_to_ref(WT_SESSION_IMPL *session, WT_PAGE *page, WT_MULTI *multi, WT_R static int __split_insert(WT_SESSION_IMPL *session, WT_REF *ref) { - WT_DECL_ITEM(key); WT_DECL_RET; WT_INSERT *ins, **insp, *moved_ins, *prev_ins; WT_INSERT_HEAD *ins_head, *tmp_ins_head; WT_PAGE *page, *right; WT_REF *child, *split_ref[2] = {NULL, NULL}; - size_t page_decr, parent_incr, right_incr; + size_t key_size, page_decr, parent_incr, right_incr; uint8_t type; int i; + void *key; WT_STAT_CONN_DATA_INCR(session, cache_inmem_split); @@ -1738,13 +1740,12 @@ __split_insert(WT_SESSION_IMPL *session, WT_REF *ref) * Assert splitting makes sense; specifically assert the page is dirty, we depend on that, * otherwise the page might be evicted based on its last reconciliation which no longer matches * reality after the split. - * - * Note this page has already been through an in-memory split. */ WT_ASSERT(session, __wt_leaf_page_can_split(session, page)); WT_ASSERT(session, __wt_page_is_modified(page)); - WT_ASSERT(session, __wt_page_del_active(session, ref, true) == false); - F_SET_ATOMIC(page, WT_PAGE_SPLIT_INSERT); + WT_ASSERT(session, ref->ft_info.del == NULL); + + F_SET_ATOMIC(page, WT_PAGE_SPLIT_INSERT); /* Only split in-memory once. */ /* Find the last item on the page. */ if (type == WT_PAGE_ROW_LEAF) @@ -1755,14 +1756,8 @@ __split_insert(WT_SESSION_IMPL *session, WT_REF *ref) moved_ins = WT_SKIP_LAST(ins_head); /* - * The first page in the split is the current page, but we still have to create a replacement - * WT_REF, the original WT_REF will be set to split status and eventually freed. - * - * The new WT_REF is not quite identical: we have to instantiate a key, and the new reference is - * visible to readers once the split completes. - * - * Don't copy any deleted page state: we may be splitting a page that was instantiated after a - * truncate and that history should not be carried onto these new child pages. + * The first page in the split is almost identical to the current page, but we have to create a + * replacement WT_REF, the original WT_REF will be set to split status and eventually freed. */ WT_ERR(__wt_calloc_one(session, &split_ref[0])); parent_incr += sizeof(WT_REF); @@ -1770,9 +1765,15 @@ __split_insert(WT_SESSION_IMPL *session, WT_REF *ref) child->page = ref->page; child->home = ref->home; child->pindex_hint = ref->pindex_hint; - child->addr = ref->addr; F_SET(child, WT_REF_FLAG_LEAF); - child->state = WT_REF_MEM; + child->state = WT_REF_MEM; /* Visible as soon as the split completes. */ + child->addr = ref->addr; + if (type == WT_PAGE_ROW_LEAF) { + __wt_ref_key(ref->home, ref, &key, &key_size); + WT_ERR(__wt_row_ikey(session, 0, key, key_size, child)); + parent_incr += sizeof(WT_IKEY) + key_size; + } else + child->ref_recno = ref->ref_recno; /* * The address has moved to the replacement WT_REF. Make sure it isn't freed when the original @@ -1780,32 +1781,7 @@ __split_insert(WT_SESSION_IMPL *session, WT_REF *ref) */ ref->addr = NULL; - if (type == WT_PAGE_ROW_LEAF) { - /* - * Copy the first key from the original page into first ref in the new parent. Pages created - * in memory always have a "smallest" insert list, so look there first. If we don't find - * one, get the first key from the disk image. - * - * We can't just use the key from the original ref: it may have been suffix-compressed, and - * after the split the truncated key may not be valid. - */ - WT_ERR(__wt_scr_alloc(session, 0, &key)); - if ((ins = WT_SKIP_FIRST(WT_ROW_INSERT_SMALLEST(page))) != NULL) { - key->data = WT_INSERT_KEY(ins); - key->size = WT_INSERT_KEY_SIZE(ins); - } else { - WT_ASSERT(session, page->entries > 0); - WT_ERR(__wt_row_leaf_key(session, page, &page->pg_row[0], key, true)); - } - WT_ERR(__wt_row_ikey(session, 0, key->data, key->size, child)); - parent_incr += sizeof(WT_IKEY) + key->size; - __wt_scr_free(session, &key); - } else - child->ref_recno = ref->ref_recno; - - /* - * The second page in the split is a new WT_REF/page pair. - */ + /* The second page in the split is a new WT_REF/page pair. */ WT_ERR(__wt_page_alloc(session, type, 0, false, &right)); /* @@ -1830,8 +1806,7 @@ __split_insert(WT_SESSION_IMPL *session, WT_REF *ref) child = split_ref[1]; child->page = right; F_SET(child, WT_REF_FLAG_LEAF); - child->state = WT_REF_MEM; - + child->state = WT_REF_MEM; /* Visible as soon as the split completes. */ if (type == WT_PAGE_ROW_LEAF) { WT_ERR(__wt_row_ikey( session, 0, WT_INSERT_KEY(moved_ins), WT_INSERT_KEY_SIZE(moved_ins), child)); @@ -2025,7 +2000,6 @@ err: __wt_page_modify_clear(session, right); __wt_page_out(session, &right); } - __wt_scr_free(session, &key); return (ret); } diff --git a/src/third_party/wiredtiger/src/btree/bt_vrfy.c b/src/third_party/wiredtiger/src/btree/bt_vrfy.c index 039a9e7e823..764cec570c2 100644 --- a/src/third_party/wiredtiger/src/btree/bt_vrfy.c +++ b/src/third_party/wiredtiger/src/btree/bt_vrfy.c @@ -291,7 +291,7 @@ __wt_verify(WT_SESSION_IMPL *session, const char *cfg[]) * We have an exclusive lock on the handle, but we're swapping root pages in-and-out of * that handle, and there's a race with eviction entering the tree and seeing an invalid * root page. Eviction must work on trees being verified (else we'd have to do our own - * eviction), lock eviction out whenever we're loading a new root page. This loops works + * eviction), lock eviction out whenever we're loading a new root page. This loop works * because we are called with eviction locked out, so we release the lock at the top of * the loop and re-acquire it here. */ diff --git a/src/third_party/wiredtiger/src/btree/bt_vrfy_dsk.c b/src/third_party/wiredtiger/src/btree/bt_vrfy_dsk.c index 1ae069a4f19..994e35806f2 100644 --- a/src/third_party/wiredtiger/src/btree/bt_vrfy_dsk.c +++ b/src/third_party/wiredtiger/src/btree/bt_vrfy_dsk.c @@ -305,7 +305,7 @@ __verify_dsk_row_int( WT_ERR(__err_cell_type(session, cell_num, tag, unpack->type, dsk->type)); cell_type = unpack->type; - /* Internal row-store cells should not have prefix compression or recno/rle fields. */ + /* Internal row-store cells should not have prefix compression or recno/rle fields. */ if (unpack->prefix != 0) WT_ERR_VRFY( session, "the %" PRIu32 " cell on page at %s has a non-zero prefix", cell_num, tag); @@ -383,7 +383,7 @@ __verify_dsk_row_int( */ switch (cell_type) { case WT_CELL_KEY: - /* Get the cell's data/length and make sure we have enough buffer space. */ + /* Get the cell's data/length and make sure we have enough buffer space. */ WT_ERR(__wt_buf_init(session, current, unpack->size)); /* Copy the data into place. */ @@ -485,7 +485,7 @@ __verify_dsk_row_leaf( WT_ERR(__err_cell_type(session, cell_num, tag, unpack->type, dsk->type)); cell_type = unpack->type; - /* Leaf row-store cells should not have recno/rle fields. */ + /* Leaf row-store cells should not have recno/rle fields. */ if (unpack->v != 0) WT_ERR_VRFY(session, "the %" PRIu32 " cell on page at %s has a non-zero rle/recno field", cell_num, tag); diff --git a/src/third_party/wiredtiger/src/btree/col_modify.c b/src/third_party/wiredtiger/src/btree/col_modify.c index d0f9ff4670a..ec7b66b1012 100644 --- a/src/third_party/wiredtiger/src/btree/col_modify.c +++ b/src/third_party/wiredtiger/src/btree/col_modify.c @@ -16,7 +16,12 @@ static int __col_insert_alloc(WT_SESSION_IMPL *, uint64_t, u_int, WT_INSERT **, */ int __wt_col_modify(WT_CURSOR_BTREE *cbt, uint64_t recno, const WT_ITEM *value, WT_UPDATE *upd_arg, - u_int modify_type, bool exclusive) + u_int modify_type, bool exclusive +#ifdef HAVE_DIAGNOSTIC + , + bool restore +#endif +) { static const WT_ITEM col_fix_remove = {"", 1, NULL, 0, 0}; WT_BTREE *btree; @@ -145,6 +150,12 @@ __wt_col_modify(WT_CURSOR_BTREE *cbt, uint64_t recno, const WT_ITEM *value, WT_U __wt_upd_value_assign(cbt->modify_update, upd); /* + * If we restore an update chain in update restore eviction, there should be no update on + * the existing update chain. + */ + WT_ASSERT(session, !restore || old_upd == NULL); + + /* * Point the new WT_UPDATE item to the next element in the list. If we get it right, the * serialization function lock acts as our memory barrier to flush this write. */ diff --git a/src/third_party/wiredtiger/src/btree/row_modify.c b/src/third_party/wiredtiger/src/btree/row_modify.c index 958c15951aa..7c13e88e2be 100644 --- a/src/third_party/wiredtiger/src/btree/row_modify.c +++ b/src/third_party/wiredtiger/src/btree/row_modify.c @@ -42,7 +42,12 @@ err: */ int __wt_row_modify(WT_CURSOR_BTREE *cbt, const WT_ITEM *key, const WT_ITEM *value, WT_UPDATE *upd_arg, - u_int modify_type, bool exclusive) + u_int modify_type, bool exclusive +#ifdef HAVE_DIAGNOSTIC + , + bool restore +#endif +) { WT_DECL_RET; WT_INSERT *ins; @@ -142,6 +147,12 @@ __wt_row_modify(WT_CURSOR_BTREE *cbt, const WT_ITEM *key, const WT_ITEM *value, last_upd->next = *upd_entry; /* + * If we restore an update chain in update restore eviction, there should be no update + * on the existing update chain. + */ + WT_ASSERT(session, !restore || *upd_entry == NULL); + + /* * We can either put multiple new updates or a single update on the update chain. * * Set the "old" entry to the second update in the list so that the serialization diff --git a/src/third_party/wiredtiger/src/checksum/x86/crc32-x86.c b/src/third_party/wiredtiger/src/checksum/x86/crc32-x86.c index 9ab7e143431..266f56d4cc3 100644 --- a/src/third_party/wiredtiger/src/checksum/x86/crc32-x86.c +++ b/src/third_party/wiredtiger/src/checksum/x86/crc32-x86.c @@ -49,8 +49,8 @@ __wt_checksum_hw(const void *chunk, size_t len) crc = 0xffffffff; - /* Checksum one byte at a time to the first 4B boundary. */ - for (p = chunk; ((uintptr_t)p & (sizeof(uint32_t) - 1)) != 0 && len > 0; ++p, --len) { + /* Checksum one byte at a time to the first 8B boundary. */ + for (p = chunk; ((uintptr_t)p & (sizeof(uint64_t) - 1)) != 0 && len > 0; ++p, --len) { __asm__ __volatile__(".byte 0xF2, 0x0F, 0x38, 0xF0, 0xF1" : "=S"(crc) : "0"(crc), "c"(*p)); } diff --git a/src/third_party/wiredtiger/src/config/config_def.c b/src/third_party/wiredtiger/src/config/config_def.c index b9914861015..75f2db226f2 100644 --- a/src/third_party/wiredtiger/src/config/config_def.c +++ b/src/third_party/wiredtiger/src/config/config_def.c @@ -275,7 +275,10 @@ static const WT_CONFIG_CHECK confchk_WT_SESSION_create[] = { {"block_allocation", "string", NULL, "choices=[\"best\",\"first\"]", NULL, 0}, {"block_compressor", "string", NULL, NULL, NULL, 0}, {"cache_resident", "boolean", NULL, NULL, NULL, 0}, - {"checksum", "string", NULL, "choices=[\"on\",\"off\",\"uncompressed\"]", NULL, 0}, + {"checksum", "string", NULL, + "choices=[\"on\",\"off\",\"uncompressed\"," + "\"unencrypted\"]", + NULL, 0}, {"colgroups", "list", NULL, NULL, NULL, 0}, {"collator", "string", NULL, NULL, NULL, 0}, {"columns", "list", NULL, NULL, NULL, 0}, {"dictionary", "int", NULL, "min=0", NULL, 0}, {"encryption", "category", NULL, NULL, confchk_WT_SESSION_create_encryption_subconfigs, 2}, @@ -431,7 +434,10 @@ static const WT_CONFIG_CHECK confchk_file_config[] = { {"block_allocation", "string", NULL, "choices=[\"best\",\"first\"]", NULL, 0}, {"block_compressor", "string", NULL, NULL, NULL, 0}, {"cache_resident", "boolean", NULL, NULL, NULL, 0}, - {"checksum", "string", NULL, "choices=[\"on\",\"off\",\"uncompressed\"]", NULL, 0}, + {"checksum", "string", NULL, + "choices=[\"on\",\"off\",\"uncompressed\"," + "\"unencrypted\"]", + NULL, 0}, {"collator", "string", NULL, NULL, NULL, 0}, {"columns", "list", NULL, NULL, NULL, 0}, {"dictionary", "int", NULL, "min=0", NULL, 0}, {"encryption", "category", NULL, NULL, confchk_WT_SESSION_create_encryption_subconfigs, 2}, @@ -479,7 +485,10 @@ static const WT_CONFIG_CHECK confchk_file_meta[] = { {"cache_resident", "boolean", NULL, NULL, NULL, 0}, {"checkpoint", "string", NULL, NULL, NULL, 0}, {"checkpoint_backup_info", "string", NULL, NULL, NULL, 0}, {"checkpoint_lsn", "string", NULL, NULL, NULL, 0}, - {"checksum", "string", NULL, "choices=[\"on\",\"off\",\"uncompressed\"]", NULL, 0}, + {"checksum", "string", NULL, + "choices=[\"on\",\"off\",\"uncompressed\"," + "\"unencrypted\"]", + NULL, 0}, {"collator", "string", NULL, NULL, NULL, 0}, {"columns", "list", NULL, NULL, NULL, 0}, {"dictionary", "int", NULL, "min=0", NULL, 0}, {"encryption", "category", NULL, NULL, confchk_WT_SESSION_create_encryption_subconfigs, 2}, @@ -543,7 +552,10 @@ static const WT_CONFIG_CHECK confchk_lsm_meta[] = { {"block_allocation", "string", NULL, "choices=[\"best\",\"first\"]", NULL, 0}, {"block_compressor", "string", NULL, NULL, NULL, 0}, {"cache_resident", "boolean", NULL, NULL, NULL, 0}, - {"checksum", "string", NULL, "choices=[\"on\",\"off\",\"uncompressed\"]", NULL, 0}, + {"checksum", "string", NULL, + "choices=[\"on\",\"off\",\"uncompressed\"," + "\"unencrypted\"]", + NULL, 0}, {"chunks", "string", NULL, NULL, NULL, 0}, {"collator", "string", NULL, NULL, NULL, 0}, {"columns", "list", NULL, NULL, NULL, 0}, {"dictionary", "int", NULL, "min=0", NULL, 0}, {"encryption", "category", NULL, NULL, confchk_WT_SESSION_create_encryption_subconfigs, 2}, @@ -593,7 +605,10 @@ static const WT_CONFIG_CHECK confchk_object_meta[] = { {"cache_resident", "boolean", NULL, NULL, NULL, 0}, {"checkpoint", "string", NULL, NULL, NULL, 0}, {"checkpoint_backup_info", "string", NULL, NULL, NULL, 0}, {"checkpoint_lsn", "string", NULL, NULL, NULL, 0}, - {"checksum", "string", NULL, "choices=[\"on\",\"off\",\"uncompressed\"]", NULL, 0}, + {"checksum", "string", NULL, + "choices=[\"on\",\"off\",\"uncompressed\"," + "\"unencrypted\"]", + NULL, 0}, {"collator", "string", NULL, NULL, NULL, 0}, {"columns", "list", NULL, NULL, NULL, 0}, {"dictionary", "int", NULL, "min=0", NULL, 0}, {"encryption", "category", NULL, NULL, confchk_WT_SESSION_create_encryption_subconfigs, 2}, @@ -659,7 +674,10 @@ static const WT_CONFIG_CHECK confchk_tier_meta[] = { {"cache_resident", "boolean", NULL, NULL, NULL, 0}, {"checkpoint", "string", NULL, NULL, NULL, 0}, {"checkpoint_backup_info", "string", NULL, NULL, NULL, 0}, {"checkpoint_lsn", "string", NULL, NULL, NULL, 0}, - {"checksum", "string", NULL, "choices=[\"on\",\"off\",\"uncompressed\"]", NULL, 0}, + {"checksum", "string", NULL, + "choices=[\"on\",\"off\",\"uncompressed\"," + "\"unencrypted\"]", + NULL, 0}, {"collator", "string", NULL, NULL, NULL, 0}, {"columns", "list", NULL, NULL, NULL, 0}, {"dictionary", "int", NULL, "min=0", NULL, 0}, {"encryption", "category", NULL, NULL, confchk_WT_SESSION_create_encryption_subconfigs, 2}, @@ -706,13 +724,19 @@ static const WT_CONFIG_CHECK confchk_tiered_meta[] = { {"assert", "category", NULL, NULL, confchk_assert_subconfigs, 4}, {"block_allocation", "string", NULL, "choices=[\"best\",\"first\"]", NULL, 0}, {"block_compressor", "string", NULL, NULL, NULL, 0}, - {"cache_resident", "boolean", NULL, NULL, NULL, 0}, - {"checksum", "string", NULL, "choices=[\"on\",\"off\",\"uncompressed\"]", NULL, 0}, + {"cache_resident", "boolean", NULL, NULL, NULL, 0}, {"checkpoint", "string", NULL, NULL, NULL, 0}, + {"checkpoint_backup_info", "string", NULL, NULL, NULL, 0}, + {"checkpoint_lsn", "string", NULL, NULL, NULL, 0}, + {"checksum", "string", NULL, + "choices=[\"on\",\"off\",\"uncompressed\"," + "\"unencrypted\"]", + NULL, 0}, {"collator", "string", NULL, NULL, NULL, 0}, {"columns", "list", NULL, NULL, NULL, 0}, {"dictionary", "int", NULL, "min=0", NULL, 0}, {"encryption", "category", NULL, NULL, confchk_WT_SESSION_create_encryption_subconfigs, 2}, {"format", "string", NULL, "choices=[\"btree\"]", NULL, 0}, {"huffman_key", "string", NULL, NULL, NULL, 0}, {"huffman_value", "string", NULL, NULL, NULL, 0}, + {"id", "string", NULL, NULL, NULL, 0}, {"ignore_in_memory_cache_size", "boolean", NULL, NULL, NULL, 0}, {"internal_item_max", "int", NULL, "min=0", NULL, 0}, {"internal_key_max", "int", NULL, "min=0", NULL, 0}, @@ -740,6 +764,7 @@ static const WT_CONFIG_CHECK confchk_tiered_meta[] = { {"tiers", "list", NULL, NULL, NULL, 0}, {"value_format", "format", __wt_struct_confchk, NULL, NULL, 0}, {"verbose", "list", NULL, "choices=[\"write_timestamp\"]", NULL, 0}, + {"version", "string", NULL, NULL, NULL, 0}, {"write_timestamp_usage", "string", NULL, "choices=[\"always\",\"key_consistent\",\"mixed_mode\"," "\"never\",\"none\",\"ordered\"]", @@ -1161,9 +1186,9 @@ static const WT_CONFIG_ENTRY config_entries[] = {{"WT_CONNECTION.add_collator", "access_pattern_hint=none,allocation_size=4KB,app_metadata=," "assert=(commit_timestamp=none,durable_timestamp=none," "read_timestamp=none,write_timestamp=off),block_allocation=best," - "block_compressor=,cache_resident=false,checksum=uncompressed," - "colgroups=,collator=,columns=,dictionary=0,encryption=(keyid=," - "name=),exclusive=false,extractor=,format=btree,huffman_key=," + "block_compressor=,cache_resident=false,checksum=on,colgroups=," + "collator=,columns=,dictionary=0,encryption=(keyid=,name=)," + "exclusive=false,extractor=,format=btree,huffman_key=," "huffman_value=,ignore_in_memory_cache_size=false,immutable=false" ",import=(enabled=false,file_metadata=,repair=false)," "internal_item_max=0,internal_key_max=0," @@ -1235,29 +1260,28 @@ static const WT_CONFIG_ENTRY config_entries[] = {{"WT_CONNECTION.add_collator", "access_pattern_hint=none,allocation_size=4KB,app_metadata=," "assert=(commit_timestamp=none,durable_timestamp=none," "read_timestamp=none,write_timestamp=off),block_allocation=best," - "block_compressor=,cache_resident=false,checksum=uncompressed," - "collator=,columns=,dictionary=0,encryption=(keyid=,name=)," - "format=btree,huffman_key=,huffman_value=," - "ignore_in_memory_cache_size=false,internal_item_max=0," - "internal_key_max=0,internal_key_truncate=true," - "internal_page_max=4KB,key_format=u,key_gap=10,leaf_item_max=0," - "leaf_key_max=0,leaf_page_max=32KB,leaf_value_max=0," - "log=(enabled=true),memory_page_image_max=0,memory_page_max=5MB," - "os_cache_dirty_max=0,os_cache_max=0,prefix_compression=false," - "prefix_compression_min=4,readonly=false,split_deepen_min_child=0" - ",split_deepen_per_child=0,split_pct=90,tiered_object=false," - "tiered_storage=(auth_token=,bucket=,bucket_prefix=," - "local_retention=300,name=,object_target_size=10M),value_format=u" - ",verbose=[],write_timestamp_usage=none", + "block_compressor=,cache_resident=false,checksum=on,collator=," + "columns=,dictionary=0,encryption=(keyid=,name=),format=btree," + "huffman_key=,huffman_value=,ignore_in_memory_cache_size=false," + "internal_item_max=0,internal_key_max=0," + "internal_key_truncate=true,internal_page_max=4KB,key_format=u," + "key_gap=10,leaf_item_max=0,leaf_key_max=0,leaf_page_max=32KB," + "leaf_value_max=0,log=(enabled=true),memory_page_image_max=0," + "memory_page_max=5MB,os_cache_dirty_max=0,os_cache_max=0," + "prefix_compression=false,prefix_compression_min=4,readonly=false" + ",split_deepen_min_child=0,split_deepen_per_child=0,split_pct=90," + "tiered_object=false,tiered_storage=(auth_token=,bucket=," + "bucket_prefix=,local_retention=300,name=,object_target_size=10M)" + ",value_format=u,verbose=[],write_timestamp_usage=none", confchk_file_config, 42}, {"file.meta", "access_pattern_hint=none,allocation_size=4KB,app_metadata=," "assert=(commit_timestamp=none,durable_timestamp=none," "read_timestamp=none,write_timestamp=off),block_allocation=best," "block_compressor=,cache_resident=false,checkpoint=," - "checkpoint_backup_info=,checkpoint_lsn=,checksum=uncompressed," - "collator=,columns=,dictionary=0,encryption=(keyid=,name=)," - "format=btree,huffman_key=,huffman_value=,id=," + "checkpoint_backup_info=,checkpoint_lsn=,checksum=on,collator=," + "columns=,dictionary=0,encryption=(keyid=,name=),format=btree," + "huffman_key=,huffman_value=,id=," "ignore_in_memory_cache_size=false,internal_item_max=0," "internal_key_max=0,internal_key_truncate=true," "internal_page_max=4KB,key_format=u,key_gap=10,leaf_item_max=0," @@ -1281,9 +1305,9 @@ static const WT_CONFIG_ENTRY config_entries[] = {{"WT_CONNECTION.add_collator", "access_pattern_hint=none,allocation_size=4KB,app_metadata=," "assert=(commit_timestamp=none,durable_timestamp=none," "read_timestamp=none,write_timestamp=off),block_allocation=best," - "block_compressor=,cache_resident=false,checksum=uncompressed," - "chunks=,collator=,columns=,dictionary=0,encryption=(keyid=," - "name=),format=btree,huffman_key=,huffman_value=," + "block_compressor=,cache_resident=false,checksum=on,chunks=," + "collator=,columns=,dictionary=0,encryption=(keyid=,name=)," + "format=btree,huffman_key=,huffman_value=," "ignore_in_memory_cache_size=false,internal_item_max=0," "internal_key_max=0,internal_key_truncate=true," "internal_page_max=4KB,key_format=u,key_gap=10,last=0," @@ -1306,9 +1330,9 @@ static const WT_CONFIG_ENTRY config_entries[] = {{"WT_CONNECTION.add_collator", "assert=(commit_timestamp=none,durable_timestamp=none," "read_timestamp=none,write_timestamp=off),block_allocation=best," "block_compressor=,cache_resident=false,checkpoint=," - "checkpoint_backup_info=,checkpoint_lsn=,checksum=uncompressed," - "collator=,columns=,dictionary=0,encryption=(keyid=,name=)," - "flush=0,format=btree,huffman_key=,huffman_value=,id=," + "checkpoint_backup_info=,checkpoint_lsn=,checksum=on,collator=," + "columns=,dictionary=0,encryption=(keyid=,name=),flush=0," + "format=btree,huffman_key=,huffman_value=,id=," "ignore_in_memory_cache_size=false,internal_item_max=0," "internal_key_max=0,internal_key_truncate=true," "internal_page_max=4KB,key_format=u,key_gap=10,leaf_item_max=0," @@ -1332,29 +1356,29 @@ static const WT_CONFIG_ENTRY config_entries[] = {{"WT_CONNECTION.add_collator", "assert=(commit_timestamp=none,durable_timestamp=none," "read_timestamp=none,write_timestamp=off),block_allocation=best," "block_compressor=,bucket=,bucket_prefix=,cache_resident=false," - "checkpoint=,checkpoint_backup_info=,checkpoint_lsn=," - "checksum=uncompressed,collator=,columns=,dictionary=0," - "encryption=(keyid=,name=),format=btree,huffman_key=," - "huffman_value=,id=,ignore_in_memory_cache_size=false," - "internal_item_max=0,internal_key_max=0," - "internal_key_truncate=true,internal_page_max=4KB,key_format=u," - "key_gap=10,leaf_item_max=0,leaf_key_max=0,leaf_page_max=32KB," - "leaf_value_max=0,log=(enabled=true),memory_page_image_max=0," - "memory_page_max=5MB,os_cache_dirty_max=0,os_cache_max=0," - "prefix_compression=false,prefix_compression_min=4,readonly=false" - ",split_deepen_min_child=0,split_deepen_per_child=0,split_pct=90," - "tiered_object=false,tiered_storage=(auth_token=,bucket=," - "bucket_prefix=,local_retention=300,name=,object_target_size=10M)" - ",value_format=u,verbose=[],version=(major=0,minor=0)," - "write_timestamp_usage=none", + "checkpoint=,checkpoint_backup_info=,checkpoint_lsn=,checksum=on," + "collator=,columns=,dictionary=0,encryption=(keyid=,name=)," + "format=btree,huffman_key=,huffman_value=,id=," + "ignore_in_memory_cache_size=false,internal_item_max=0," + "internal_key_max=0,internal_key_truncate=true," + "internal_page_max=4KB,key_format=u,key_gap=10,leaf_item_max=0," + "leaf_key_max=0,leaf_page_max=32KB,leaf_value_max=0," + "log=(enabled=true),memory_page_image_max=0,memory_page_max=5MB," + "os_cache_dirty_max=0,os_cache_max=0,prefix_compression=false," + "prefix_compression_min=4,readonly=false,split_deepen_min_child=0" + ",split_deepen_per_child=0,split_pct=90,tiered_object=false," + "tiered_storage=(auth_token=,bucket=,bucket_prefix=," + "local_retention=300,name=,object_target_size=10M),value_format=u" + ",verbose=[],version=(major=0,minor=0),write_timestamp_usage=none", confchk_tier_meta, 49}, {"tiered.meta", "access_pattern_hint=none,allocation_size=4KB,app_metadata=," "assert=(commit_timestamp=none,durable_timestamp=none," "read_timestamp=none,write_timestamp=off),block_allocation=best," - "block_compressor=,cache_resident=false,checksum=uncompressed," - "collator=,columns=,dictionary=0,encryption=(keyid=,name=)," - "format=btree,huffman_key=,huffman_value=," + "block_compressor=,cache_resident=false,checkpoint=," + "checkpoint_backup_info=,checkpoint_lsn=,checksum=on,collator=," + "columns=,dictionary=0,encryption=(keyid=,name=),format=btree," + "huffman_key=,huffman_value=,id=," "ignore_in_memory_cache_size=false,internal_item_max=0," "internal_key_max=0,internal_key_truncate=true," "internal_page_max=4KB,key_format=u,key_gap=10,last=0," @@ -1365,8 +1389,9 @@ static const WT_CONFIG_ENTRY config_entries[] = {{"WT_CONNECTION.add_collator", ",split_deepen_min_child=0,split_deepen_per_child=0,split_pct=90," "tiered_object=false,tiered_storage=(auth_token=,bucket=," "bucket_prefix=,local_retention=300,name=,object_target_size=10M)" - ",tiers=,value_format=u,verbose=[],write_timestamp_usage=none", - confchk_tiered_meta, 44}, + ",tiers=,value_format=u,verbose=[],version=(major=0,minor=0)," + "write_timestamp_usage=none", + confchk_tiered_meta, 49}, {"wiredtiger_open", "buffer_alignment=-1,builtin_extension_config=,cache_cursors=true" ",cache_max_wait_ms=0,cache_overhead=8,cache_size=100MB," diff --git a/src/third_party/wiredtiger/src/config/test_config.c b/src/third_party/wiredtiger/src/config/test_config.c index 7108eccbc20..135cf6f01a2 100644 --- a/src/third_party/wiredtiger/src/config/test_config.c +++ b/src/third_party/wiredtiger/src/config/test_config.c @@ -16,10 +16,15 @@ static const WT_CONFIG_CHECK confchk_stat_db_size_subconfigs[] = { static const WT_CONFIG_CHECK confchk_runtime_monitor_subconfigs[] = { {"enabled", "boolean", NULL, NULL, NULL, 0}, {"op_rate", "string", NULL, NULL, NULL, 0}, + {"postrun_statistics", "list", NULL, NULL, NULL, 0}, {"stat_cache_size", "category", NULL, NULL, confchk_stat_cache_size_subconfigs, 2}, {"stat_db_size", "category", NULL, NULL, confchk_stat_db_size_subconfigs, 2}, {NULL, NULL, NULL, NULL, NULL, 0}}; +static const WT_CONFIG_CHECK confchk_statistics_config_subconfigs[] = { + {"enable_logging", "boolean", NULL, NULL, NULL, 0}, {"type", "string", NULL, NULL, NULL, 0}, + {NULL, NULL, NULL, NULL, NULL, 0}}; + static const WT_CONFIG_CHECK confchk_timestamp_manager_subconfigs[] = { {"enabled", "boolean", NULL, NULL, NULL, 0}, {"oldest_lag", "int", NULL, "min=0,max=1000000", NULL, 0}, @@ -33,89 +38,134 @@ static const WT_CONFIG_CHECK confchk_ops_per_transaction_subconfigs[] = { static const WT_CONFIG_CHECK confchk_insert_config_subconfigs[] = { {"key_size", "int", NULL, "min=0,max=10000", NULL, 0}, {"op_rate", "string", NULL, NULL, NULL, 0}, {"ops_per_transaction", "category", NULL, NULL, confchk_ops_per_transaction_subconfigs, 2}, + {"thread_count", "int", NULL, "min=0", NULL, 0}, + {"value_size", "int", NULL, "min=0,max=1000000000", NULL, 0}, {NULL, NULL, NULL, NULL, NULL, 0}}; + +static const WT_CONFIG_CHECK confchk_populate_config_subconfigs[] = { + {"collection_count", "int", NULL, "min=0,max=200000", NULL, 0}, + {"key_count_per_collection", "int", NULL, "min=0,max=1000000", NULL, 0}, + {"key_size", "int", NULL, "min=0,max=10000", NULL, 0}, {"thread_count", "string", NULL, NULL, NULL, 0}, {"value_size", "int", NULL, "min=0,max=1000000000", NULL, 0}, {NULL, NULL, NULL, NULL, NULL, 0}}; static const WT_CONFIG_CHECK confchk_read_config_subconfigs[] = { {"op_rate", "string", NULL, NULL, NULL, 0}, {"ops_per_transaction", "category", NULL, NULL, confchk_ops_per_transaction_subconfigs, 2}, - {"thread_count", "string", NULL, NULL, NULL, 0}, {NULL, NULL, NULL, NULL, NULL, 0}}; + {"thread_count", "int", NULL, "min=0", NULL, 0}, {NULL, NULL, NULL, NULL, NULL, 0}}; static const WT_CONFIG_CHECK confchk_update_config_subconfigs[] = { {"key_size", "int", NULL, "min=0,max=10000", NULL, 0}, {"op_rate", "string", NULL, NULL, NULL, 0}, {"ops_per_transaction", "category", NULL, NULL, confchk_ops_per_transaction_subconfigs, 2}, - {"thread_count", "string", NULL, NULL, NULL, 0}, + {"thread_count", "int", NULL, "min=0", NULL, 0}, {"value_size", "int", NULL, "min=0,max=1000000000", NULL, 0}, {NULL, NULL, NULL, NULL, NULL, 0}}; static const WT_CONFIG_CHECK confchk_workload_generator_subconfigs[] = { - {"collection_count", "int", NULL, "min=0,max=200000", NULL, 0}, {"enabled", "boolean", NULL, NULL, NULL, 0}, {"insert_config", "category", NULL, NULL, confchk_insert_config_subconfigs, 5}, - {"key_count", "int", NULL, "min=0,max=1000000", NULL, 0}, - {"key_size", "int", NULL, "min=0,max=10000", NULL, 0}, {"op_rate", "string", NULL, NULL, NULL, 0}, + {"op_rate", "string", NULL, NULL, NULL, 0}, + {"populate_config", "category", NULL, NULL, confchk_populate_config_subconfigs, 5}, {"read_config", "category", NULL, NULL, confchk_read_config_subconfigs, 3}, {"update_config", "category", NULL, NULL, confchk_update_config_subconfigs, 5}, - {"value_size", "int", NULL, "min=0,max=1000000000", NULL, 0}, {NULL, NULL, NULL, NULL, NULL, 0}}; + {NULL, NULL, NULL, NULL, NULL, 0}}; static const WT_CONFIG_CHECK confchk_workload_tracking_subconfigs[] = { {"enabled", "boolean", NULL, NULL, NULL, 0}, {"op_rate", "string", NULL, NULL, NULL, 0}, {NULL, NULL, NULL, NULL, NULL, 0}}; +static const WT_CONFIG_CHECK confchk_base_test[] = { + {"cache_size_mb", "int", NULL, "min=0,max=100000000000", NULL, 0}, + {"checkpoint_manager", "category", NULL, NULL, confchk_checkpoint_manager_subconfigs, 2}, + {"compression_enabled", "boolean", NULL, NULL, NULL, 0}, + {"duration_seconds", "int", NULL, "min=0,max=1000000", NULL, 0}, + {"enable_logging", "boolean", NULL, NULL, NULL, 0}, + {"runtime_monitor", "category", NULL, NULL, confchk_runtime_monitor_subconfigs, 5}, + {"statistics_config", "category", NULL, NULL, confchk_statistics_config_subconfigs, 2}, + {"timestamp_manager", "category", NULL, NULL, confchk_timestamp_manager_subconfigs, 4}, + {"workload_generator", "category", NULL, NULL, confchk_workload_generator_subconfigs, 6}, + {"workload_tracking", "category", NULL, NULL, confchk_workload_tracking_subconfigs, 2}, + {NULL, NULL, NULL, NULL, NULL, 0}}; + static const WT_CONFIG_CHECK confchk_example_test[] = { {"cache_size_mb", "int", NULL, "min=0,max=100000000000", NULL, 0}, {"checkpoint_manager", "category", NULL, NULL, confchk_checkpoint_manager_subconfigs, 2}, + {"compression_enabled", "boolean", NULL, NULL, NULL, 0}, {"duration_seconds", "int", NULL, "min=0,max=1000000", NULL, 0}, {"enable_logging", "boolean", NULL, NULL, NULL, 0}, - {"runtime_monitor", "category", NULL, NULL, confchk_runtime_monitor_subconfigs, 4}, + {"runtime_monitor", "category", NULL, NULL, confchk_runtime_monitor_subconfigs, 5}, + {"statistics_config", "category", NULL, NULL, confchk_statistics_config_subconfigs, 2}, {"timestamp_manager", "category", NULL, NULL, confchk_timestamp_manager_subconfigs, 4}, - {"workload_generator", "category", NULL, NULL, confchk_workload_generator_subconfigs, 9}, + {"workload_generator", "category", NULL, NULL, confchk_workload_generator_subconfigs, 6}, {"workload_tracking", "category", NULL, NULL, confchk_workload_tracking_subconfigs, 2}, {NULL, NULL, NULL, NULL, NULL, 0}}; -static const WT_CONFIG_CHECK confchk_poc_test[] = { +static const WT_CONFIG_CHECK confchk_hs_cleanup[] = { {"cache_size_mb", "int", NULL, "min=0,max=100000000000", NULL, 0}, {"checkpoint_manager", "category", NULL, NULL, confchk_checkpoint_manager_subconfigs, 2}, + {"compression_enabled", "boolean", NULL, NULL, NULL, 0}, {"duration_seconds", "int", NULL, "min=0,max=1000000", NULL, 0}, {"enable_logging", "boolean", NULL, NULL, NULL, 0}, - {"runtime_monitor", "category", NULL, NULL, confchk_runtime_monitor_subconfigs, 4}, + {"runtime_monitor", "category", NULL, NULL, confchk_runtime_monitor_subconfigs, 5}, + {"statistics_config", "category", NULL, NULL, confchk_statistics_config_subconfigs, 2}, {"timestamp_manager", "category", NULL, NULL, confchk_timestamp_manager_subconfigs, 4}, - {"workload_generator", "category", NULL, NULL, confchk_workload_generator_subconfigs, 9}, + {"workload_generator", "category", NULL, NULL, confchk_workload_generator_subconfigs, 6}, {"workload_tracking", "category", NULL, NULL, confchk_workload_tracking_subconfigs, 2}, {NULL, NULL, NULL, NULL, NULL, 0}}; static const WT_CONFIG_ENTRY config_entries[] = { + {"base_test", + "cache_size_mb=0,checkpoint_manager=(enabled=false,op_rate=1s)," + "compression_enabled=false,duration_seconds=0," + "enable_logging=false,runtime_monitor=(enabled=true,op_rate=1s," + "postrun_statistics=[],stat_cache_size=(enabled=false,limit=0)," + "stat_db_size=(enabled=false,limit=0))," + "statistics_config=(enable_logging=true,type=all)," + "timestamp_manager=(enabled=true,oldest_lag=1,op_rate=1s," + "stable_lag=1),workload_generator=(enabled=true," + "insert_config=(key_size=5,op_rate=1s,ops_per_transaction=(max=1," + "min=0),thread_count=0,value_size=5),op_rate=1s," + "populate_config=(collection_count=1,key_count_per_collection=0," + "key_size=5,thread_count=1,value_size=5),read_config=(op_rate=1s," + "ops_per_transaction=(max=1,min=0),thread_count=0)," + "update_config=(key_size=5,op_rate=1s,ops_per_transaction=(max=1," + "min=0),thread_count=0,value_size=5))," + "workload_tracking=(enabled=true,op_rate=1s)", + confchk_base_test, 10}, {"example_test", "cache_size_mb=0,checkpoint_manager=(enabled=false,op_rate=1s)," - "duration_seconds=0,enable_logging=false," - "runtime_monitor=(enabled=true,op_rate=1s," - "stat_cache_size=(enabled=false,limit=0)," + "compression_enabled=false,duration_seconds=0," + "enable_logging=false,runtime_monitor=(enabled=true,op_rate=1s," + "postrun_statistics=[],stat_cache_size=(enabled=false,limit=0)," "stat_db_size=(enabled=false,limit=0))," + "statistics_config=(enable_logging=true,type=all)," "timestamp_manager=(enabled=true,oldest_lag=1,op_rate=1s," - "stable_lag=1),workload_generator=(collection_count=1," - "enabled=true,insert_config=(key_size=5,op_rate=1s," - "ops_per_transaction=(max=1,min=0),thread_count=1,value_size=5)," - "key_count=0,key_size=5,op_rate=1s,read_config=(op_rate=1s," - "ops_per_transaction=(max=1,min=0),thread_count=1)," + "stable_lag=1),workload_generator=(enabled=true," + "insert_config=(key_size=5,op_rate=1s,ops_per_transaction=(max=1," + "min=0),thread_count=0,value_size=5),op_rate=1s," + "populate_config=(collection_count=1,key_count_per_collection=0," + "key_size=5,thread_count=1,value_size=5),read_config=(op_rate=1s," + "ops_per_transaction=(max=1,min=0),thread_count=0)," "update_config=(key_size=5,op_rate=1s,ops_per_transaction=(max=1," - "min=0),thread_count=1,value_size=5),value_size=5)," + "min=0),thread_count=0,value_size=5))," "workload_tracking=(enabled=true,op_rate=1s)", - confchk_example_test, 8}, - {"poc_test", + confchk_example_test, 10}, + {"hs_cleanup", "cache_size_mb=0,checkpoint_manager=(enabled=false,op_rate=1s)," - "duration_seconds=0,enable_logging=false," - "runtime_monitor=(enabled=true,op_rate=1s," - "stat_cache_size=(enabled=false,limit=0)," + "compression_enabled=false,duration_seconds=0," + "enable_logging=false,runtime_monitor=(enabled=true,op_rate=1s," + "postrun_statistics=[],stat_cache_size=(enabled=false,limit=0)," "stat_db_size=(enabled=false,limit=0))," + "statistics_config=(enable_logging=true,type=all)," "timestamp_manager=(enabled=true,oldest_lag=1,op_rate=1s," - "stable_lag=1),workload_generator=(collection_count=1," - "enabled=true,insert_config=(key_size=5,op_rate=1s," - "ops_per_transaction=(max=1,min=0),thread_count=1,value_size=5)," - "key_count=0,key_size=5,op_rate=1s,read_config=(op_rate=1s," - "ops_per_transaction=(max=1,min=0),thread_count=1)," + "stable_lag=1),workload_generator=(enabled=true," + "insert_config=(key_size=5,op_rate=1s,ops_per_transaction=(max=1," + "min=0),thread_count=0,value_size=5),op_rate=1s," + "populate_config=(collection_count=1,key_count_per_collection=0," + "key_size=5,thread_count=1,value_size=5),read_config=(op_rate=1s," + "ops_per_transaction=(max=1,min=0),thread_count=0)," "update_config=(key_size=5,op_rate=1s,ops_per_transaction=(max=1," - "min=0),thread_count=1,value_size=5),value_size=5)," + "min=0),thread_count=0,value_size=5))," "workload_tracking=(enabled=true,op_rate=1s)", - confchk_poc_test, 8}, + confchk_hs_cleanup, 10}, {NULL, NULL, NULL, 0}}; /* diff --git a/src/third_party/wiredtiger/src/conn/conn_api.c b/src/third_party/wiredtiger/src/conn/conn_api.c index b35d198c3aa..a05b4b1c85d 100644 --- a/src/third_party/wiredtiger/src/conn/conn_api.c +++ b/src/third_party/wiredtiger/src/conn/conn_api.c @@ -716,6 +716,7 @@ __conn_get_storage_source( WT_CONNECTION_IMPL *conn; WT_DECL_RET; WT_NAMED_STORAGE_SOURCE *nstorage_source; + WT_STORAGE_SOURCE *storage_source; conn = (WT_CONNECTION_IMPL *)wt_conn; *storage_sourcep = NULL; @@ -723,7 +724,9 @@ __conn_get_storage_source( ret = EINVAL; TAILQ_FOREACH (nstorage_source, &conn->storagesrcqh, q) if (WT_STREQ(nstorage_source->name, name)) { - *storage_sourcep = nstorage_source->storage_source; + storage_source = nstorage_source->storage_source; + WT_RET(storage_source->ss_add_reference(storage_source)); + *storage_sourcep = storage_source; ret = 0; break; } @@ -824,6 +827,10 @@ __conn_get_extension_api(WT_CONNECTION *wt_conn) conn->extension_api.struct_pack = __wt_ext_struct_pack; conn->extension_api.struct_size = __wt_ext_struct_size; conn->extension_api.struct_unpack = __wt_ext_struct_unpack; + conn->extension_api.spin_init = __wt_ext_spin_init; + conn->extension_api.spin_lock = __wt_ext_spin_lock; + conn->extension_api.spin_unlock = __wt_ext_spin_unlock; + conn->extension_api.spin_destroy = __wt_ext_spin_destroy; conn->extension_api.transaction_id = __wt_ext_transaction_id; conn->extension_api.transaction_isolation_level = __wt_ext_transaction_isolation_level; conn->extension_api.transaction_notify = __wt_ext_transaction_notify; @@ -2858,7 +2865,7 @@ wiredtiger_open(const char *home, WT_EVENT_HANDLER *event_handler, const char *c WT_ERR(__wt_config_gets(session, cfg, "verify_metadata", &cval)); verify_meta = cval.val; - /* Only verify the metadata file first. We will verify the history store table later. */ + /* Only verify the metadata file first. We will verify the history store table later. */ if (verify_meta) { wt_session = &session->iface; ret = wt_session->verify(wt_session, WT_METAFILE_URI, NULL); diff --git a/src/third_party/wiredtiger/src/conn/conn_dhandle.c b/src/third_party/wiredtiger/src/conn/conn_dhandle.c index 41654780528..3abeeb91152 100644 --- a/src/third_party/wiredtiger/src/conn/conn_dhandle.c +++ b/src/third_party/wiredtiger/src/conn/conn_dhandle.c @@ -41,7 +41,7 @@ __conn_dhandle_config_set(WT_SESSION_IMPL *session) WT_DATA_HANDLE *dhandle; WT_DECL_RET; char *metaconf, *tmp; - const char *base, *cfg[4]; + const char *base, *cfg[4], *strip; dhandle = session->dhandle; base = NULL; @@ -99,8 +99,11 @@ __conn_dhandle_config_set(WT_SESSION_IMPL *session) */ cfg[0] = tmp; cfg[1] = NULL; - WT_ERR(__wt_config_merge( - session, cfg, "checkpoint=,checkpoint_backup_info=,checkpoint_lsn=", &base)); + if (dhandle->type == WT_DHANDLE_TYPE_TIERED) + strip = "checkpoint=,checkpoint_backup_info=,checkpoint_lsn=,last=,tiers=()"; + else + strip = "checkpoint=,checkpoint_backup_info=,checkpoint_lsn="; + WT_ERR(__wt_config_merge(session, cfg, strip, &base)); __wt_free(session, tmp); break; case WT_DHANDLE_TYPE_TABLE: @@ -146,8 +149,7 @@ __conn_dhandle_destroy(WT_SESSION_IMPL *session, WT_DATA_HANDLE *dhandle) ret = __wt_schema_close_table(session, (WT_TABLE *)dhandle); break; case WT_DHANDLE_TYPE_TIERED: - WT_WITH_DHANDLE(session, dhandle, ret = __wt_btree_discard(session)); - ret = __wt_tiered_close(session, (WT_TIERED *)dhandle); + WT_WITH_DHANDLE(session, dhandle, ret = __wt_tiered_discard(session, (WT_TIERED *)dhandle)); break; case WT_DHANDLE_TYPE_TIERED_TREE: ret = __wt_tiered_tree_close(session, (WT_TIERED_TREE *)dhandle); @@ -400,9 +402,8 @@ __wt_conn_dhandle_close(WT_SESSION_IMPL *session, bool final, bool mark_dead) WT_TRET(__wt_schema_close_table(session, (WT_TABLE *)dhandle)); break; case WT_DHANDLE_TYPE_TIERED: - WT_TRET(__wt_btree_close(session)); + WT_TRET(__wt_tiered_close(session)); F_CLR(btree, WT_BTREE_SPECIAL_FLAGS); - WT_TRET(__wt_tiered_close(session, (WT_TIERED *)dhandle)); break; case WT_DHANDLE_TYPE_TIERED_TREE: WT_TRET(__wt_tiered_tree_close(session, (WT_TIERED_TREE *)dhandle)); @@ -605,9 +606,17 @@ err: F_CLR(btree, WT_BTREE_SPECIAL_FLAGS); } - if (WT_DHANDLE_BTREE(dhandle)) + if (WT_DHANDLE_BTREE(dhandle) && session->dhandle != NULL) { __wt_evict_file_exclusive_off(session); + /* + * We want to close the Btree for an object that lives in the local directory. It will + * actually be part of the corresponding tiered Btree. + */ + if (dhandle->type == WT_DHANDLE_TYPE_BTREE && WT_SUFFIX_MATCH(dhandle->name, ".wtobj")) + WT_TRET(__wt_btree_close(session)); + } + if (ret == ENOENT && F_ISSET(dhandle, WT_DHANDLE_IS_METADATA)) { F_SET(S2C(session), WT_CONN_DATA_CORRUPTION); return (WT_ERROR); diff --git a/src/third_party/wiredtiger/src/conn/conn_tiered.c b/src/third_party/wiredtiger/src/conn/conn_tiered.c index a443802197c..39524e0758b 100644 --- a/src/third_party/wiredtiger/src/conn/conn_tiered.c +++ b/src/third_party/wiredtiger/src/conn/conn_tiered.c @@ -42,10 +42,6 @@ __flush_tier_wait(WT_SESSION_IMPL *session) * It may be worthwhile looking at the add and decrement values and make choices of whether to * yield or wait based on how much of the workload has been performed. Flushing operations could * take a long time so yielding may not be effective. - * - * TODO: We should consider a maximum wait value as a configuration setting. If we add one, then - * this function returns an int and this loop would check how much time we've waited and break - * out with EBUSY. */ while (!WT_FLUSH_STATE_DONE(conn->flush_state)) { if (++yield_count < WT_THOUSAND) @@ -80,7 +76,8 @@ __flush_tier_once(WT_SESSION_IMPL *session, uint32_t flags) S2C(session)->flush_state = 0; /* - * XXX: Is it sufficient to walk the metadata cursor? If it is, why doesn't checkpoint do that? + * Walk the metadata cursor to find tiered tables to flush. This should be optimized to avoid + * flushing tables that haven't changed. */ WT_RET(__wt_metadata_cursor(session, &cursor)); while (cursor->next(cursor) == 0) { @@ -90,7 +87,7 @@ __flush_tier_once(WT_SESSION_IMPL *session, uint32_t flags) if (WT_PREFIX_MATCH(key, "tiered:")) { __wt_verbose(session, WT_VERB_TIERED, "FLUSH_TIER_ONCE: %s %s", key, value); /* Is this instantiating every handle even if it is not opened or in use? */ - WT_ERR(__wt_session_get_dhandle(session, key, NULL, NULL, WT_DHANDLE_EXCLUSIVE)); + WT_ERR(__wt_session_get_dhandle(session, key, NULL, NULL, 0)); /* * When we call wt_tiered_switch the session->dhandle points to the tiered: entry and * the arg is the config string that is currently in the metadata. @@ -188,7 +185,7 @@ __tier_flush_meta( WT_ERR(__wt_meta_track_on(session)); tracking = true; - WT_ERR(__wt_session_get_dhandle(session, dhandle->name, NULL, NULL, WT_DHANDLE_EXCLUSIVE)); + WT_ERR(__wt_session_get_dhandle(session, dhandle->name, NULL, NULL, 0)); release = true; /* * Once the flush call succeeds we want to first remove the file: entry from the metadata and @@ -226,7 +223,6 @@ __wt_tier_do_flush( WT_DECL_RET; WT_FILE_SYSTEM *bucket_fs; WT_STORAGE_SOURCE *storage_source; - uint32_t msec, retry; const char *local_name, *obj_name; storage_source = tiered->bstorage->storage_source; @@ -241,21 +237,8 @@ __wt_tier_do_flush( WT_RET(storage_source->ss_flush( storage_source, &session->iface, bucket_fs, local_name, obj_name, NULL)); - /* - * Flushing the metadata grabs the data handle with exclusive access, and the data handle may be - * held by the thread that queues the flush tier work item. As a result, the handle may be busy, - * so retry as needed, up to a few seconds. - */ - for (msec = 10, retry = 0; msec < 3000; msec *= 2, retry++) { - if (retry != 0) - __wt_sleep(0, msec * WT_THOUSAND); - WT_WITH_CHECKPOINT_LOCK(session, - WT_WITH_SCHEMA_LOCK( - session, ret = __tier_flush_meta(session, tiered, local_uri, obj_uri))); - if (ret != EBUSY) - break; - WT_STAT_CONN_INCR(session, flush_tier_busy); - } + WT_WITH_CHECKPOINT_LOCK(session, + WT_WITH_SCHEMA_LOCK(session, ret = __tier_flush_meta(session, tiered, local_uri, obj_uri))); WT_RET(ret); /* @@ -369,7 +352,7 @@ __wt_flush_tier(WT_SESSION_IMPL *session, const char *config) WT_RET(__wt_config_gets(session, cfg, "force", &cval)); if (cval.val) LF_SET(WT_FLUSH_TIER_FORCE); - WT_RET(__wt_config_gets_def(session, cfg, "sync", 0, &cval)); + WT_RET(__wt_config_gets(session, cfg, "sync", &cval)); if (WT_STRING_MATCH("off", cval.str, cval.len)) LF_SET(WT_FLUSH_TIER_OFF); else if (WT_STRING_MATCH("on", cval.str, cval.len)) @@ -398,9 +381,11 @@ __wt_flush_tier(WT_SESSION_IMPL *session, const char *config) */ __flush_tier_wait(session); if (wait) - WT_WITH_SCHEMA_LOCK(session, ret = __flush_tier_once(session, flags)); + WT_WITH_CHECKPOINT_LOCK( + session, WT_WITH_SCHEMA_LOCK(session, ret = __flush_tier_once(session, flags))); else - WT_WITH_SCHEMA_LOCK_NOWAIT(session, ret, ret = __flush_tier_once(session, flags)); + WT_WITH_CHECKPOINT_LOCK_NOWAIT(session, ret, + WT_WITH_SCHEMA_LOCK_NOWAIT(session, ret, ret = __flush_tier_once(session, flags))); __wt_spin_unlock(session, &conn->flush_tier_lock); if (ret == 0 && LF_ISSET(WT_FLUSH_TIER_ON)) @@ -644,10 +629,23 @@ __wt_tiered_storage_destroy(WT_SESSION_IMPL *session) conn = S2C(session); + FLD_CLR(conn->server_flags, WT_CONN_SERVER_TIERED_MGR); + /* + * Stop the storage manager thread. This must be stopped before the internal thread because it + * could be adding work for the internal thread. So stop it first and the internal thread will + * have the opportunity to drain all work. + */ + if (conn->tiered_mgr_tid_set) { + WT_ASSERT(session, conn->tiered_mgr_cond != NULL); + __wt_cond_signal(session, conn->tiered_mgr_cond); + WT_TRET(__wt_thread_join(session, &conn->tiered_mgr_tid)); + conn->tiered_mgr_tid_set = false; + } + /* Stop the internal server thread. */ if (conn->flush_cond != NULL) __wt_cond_signal(session, conn->flush_cond); - FLD_CLR(conn->server_flags, WT_CONN_SERVER_TIERED | WT_CONN_SERVER_TIERED_MGR); + FLD_CLR(conn->server_flags, WT_CONN_SERVER_TIERED); if (conn->tiered_tid_set) { WT_ASSERT(session, conn->tiered_cond != NULL); __wt_cond_signal(session, conn->tiered_cond); @@ -663,17 +661,10 @@ __wt_tiered_storage_destroy(WT_SESSION_IMPL *session) conn->tiered_session = NULL; } - /* Stop the storage manager thread. */ - if (conn->tiered_mgr_tid_set) { - WT_ASSERT(session, conn->tiered_mgr_cond != NULL); - __wt_cond_signal(session, conn->tiered_mgr_cond); - WT_TRET(__wt_thread_join(session, &conn->tiered_mgr_tid)); - conn->tiered_mgr_tid_set = false; - } /* Destroy all condition variables after threads have stopped. */ __wt_cond_destroy(session, &conn->tiered_cond); __wt_cond_destroy(session, &conn->tiered_mgr_cond); - /* The flush condition variable must be last because any internal thread could be using it. */ + /* The flush condition variable must be last because any internal thread could be using it. */ __wt_cond_destroy(session, &conn->flush_cond); if (conn->tiered_mgr_session != NULL) { diff --git a/src/third_party/wiredtiger/src/cursor/cur_backup.c b/src/third_party/wiredtiger/src/cursor/cur_backup.c index 2d7be4f4af7..fd416d3c452 100644 --- a/src/third_party/wiredtiger/src/cursor/cur_backup.c +++ b/src/third_party/wiredtiger/src/cursor/cur_backup.c @@ -348,7 +348,7 @@ __backup_add_id(WT_SESSION_IMPL *session, WT_CONFIG_ITEM *cval) if (i == WT_BLKINCR_MAX) WT_RET_PANIC(session, WT_NOTFOUND, "Could not find an incremental backup slot to use"); - /* Use the slot. */ + /* Use the slot. */ if (blk->id_str != NULL) __wt_verbose( session, WT_VERB_BACKUP, "Freeing and reusing backup slot with old id %s", blk->id_str); @@ -451,7 +451,7 @@ err: */ static int __backup_config(WT_SESSION_IMPL *session, WT_CURSOR_BACKUP *cb, const char *cfg[], - WT_CURSOR_BACKUP *othercb, bool *foundp, bool *log_only, bool *incr_only) + WT_CURSOR_BACKUP *othercb, bool *foundp, bool *log_only) { WT_CONFIG targetconf; WT_CONFIG_ITEM cval, k, v; @@ -461,7 +461,7 @@ __backup_config(WT_SESSION_IMPL *session, WT_CURSOR_BACKUP *cb, const char *cfg[ const char *uri; bool consolidate, incremental_config, is_dup, log_config, target_list; - *foundp = *incr_only = *log_only = false; + *foundp = *log_only = false; conn = S2C(session); incremental_config = log_config = false; @@ -665,7 +665,7 @@ __backup_start( WT_DECL_RET; WT_FSTREAM *srcfs; const char *dest; - bool exist, is_dup, is_incr, log_only, target_list; + bool exist, is_dup, log_only, target_list; conn = S2C(session); srcfs = NULL; @@ -748,7 +748,7 @@ __backup_start( * database objects and log files to the list. */ target_list = false; - WT_ERR(__backup_config(session, cb, cfg, othercb, &target_list, &log_only, &is_incr)); + WT_ERR(__backup_config(session, cb, cfg, othercb, &target_list, &log_only)); /* * For a duplicate cursor, all the work is done in backup_config. */ diff --git a/src/third_party/wiredtiger/src/cursor/cur_backup_incr.c b/src/third_party/wiredtiger/src/cursor/cur_backup_incr.c index 0f1cd879d72..341c604b52a 100644 --- a/src/third_party/wiredtiger/src/cursor/cur_backup_incr.c +++ b/src/third_party/wiredtiger/src/cursor/cur_backup_incr.c @@ -198,7 +198,7 @@ __curbackup_incr_next(WT_CURSOR *cursor) found = false; /* The bit offset can be less than or equal to but never greater than the number of bits. */ WT_ASSERT(session, cb->bit_offset <= cb->nbits); - /* Look for the next chunk that had modifications. */ + /* Look for the next chunk that had modifications. */ while (cb->bit_offset < cb->nbits) if (__bit_test(cb->bitstring.mem, cb->bit_offset)) { found = true; diff --git a/src/third_party/wiredtiger/src/cursor/cur_hs.c b/src/third_party/wiredtiger/src/cursor/cur_hs.c index aecc01ade8b..2a2657b828b 100644 --- a/src/third_party/wiredtiger/src/cursor/cur_hs.c +++ b/src/third_party/wiredtiger/src/cursor/cur_hs.c @@ -939,8 +939,7 @@ __curhs_insert(WT_CURSOR *cursor) /* Insert doesn't maintain a position across calls, clear resources. */ err: - __wt_free(session, hs_tombstone); - __wt_free(session, hs_upd); + __wt_free_update_list(session, &hs_upd); WT_TRET(cursor->reset(cursor)); API_END_RET(session, ret); } @@ -969,7 +968,7 @@ __curhs_remove(WT_CURSOR *cursor) cbt = (WT_CURSOR_BTREE *)file_cursor; hs_tombstone = NULL; - CURSOR_API_CALL_PREPARE_ALLOWED(cursor, session, insert, CUR2BT(file_cursor)); + CURSOR_API_CALL_PREPARE_ALLOWED(cursor, session, remove, CUR2BT(file_cursor)); /* Remove must be called with cursor positioned. */ WT_ASSERT(session, F_ISSET(cursor, WT_CURSTD_KEY_INT)); diff --git a/src/third_party/wiredtiger/src/docs/Doxyfile b/src/third_party/wiredtiger/src/docs/Doxyfile index 97487950586..8bd05ca0bfa 100644 --- a/src/third_party/wiredtiger/src/docs/Doxyfile +++ b/src/third_party/wiredtiger/src/docs/Doxyfile @@ -299,12 +299,6 @@ ALIASES = "arch_page_table{2}=<div class= arch_head><table><tr><t "row{9}=<tr><td>\1</td><td>\2</td><td>\3</td><td>\4</td><td>\5</td><td>\6</td><td>\7</td><td>\8</td><td>\9</td></tr>" \ "subpage_single=@subpage" -# This tag can be used to specify a number of word-keyword mappings (TCL only). -# A mapping has the form "name=value". For example adding "class=itcl::class" -# will allow you to use the command class in the itcl::class meaning. - -TCL_SUBST = - # Set the OPTIMIZE_OUTPUT_FOR_C tag to YES if your project consists of C sources # only. Doxygen will then generate output that is more tailored for C. For # instance, some of the names that are used will be different. The list of all @@ -982,6 +976,7 @@ EXAMPLE_PATH = ../../examples/c \ ../../ext/compressors/nop \ ../../ext/encryptors/nop \ ../../ext/encryptors/rotn \ + ../../ext/encryptors/sodium \ ../../examples/python \ ../../test/fuzz @@ -1828,7 +1823,7 @@ COMPACT_LATEX = NO # The default value is: a4. # This tag requires that the tag GENERATE_LATEX is set to YES. -PAPER_TYPE = a4wide +PAPER_TYPE = a4 # The EXTRA_PACKAGES tag can be used to specify one or more LaTeX package names # that should be included in the LaTeX output. The package can be specified just diff --git a/src/third_party/wiredtiger/src/docs/arch-index.dox b/src/third_party/wiredtiger/src/docs/arch-index.dox index c91a53129f2..75d5a8abf5c 100644 --- a/src/third_party/wiredtiger/src/docs/arch-index.dox +++ b/src/third_party/wiredtiger/src/docs/arch-index.dox @@ -6,7 +6,7 @@ Here is an overview of the software components in WiredTiger and how they are or An arrow indicates the "from" component uses "to" component. <div class="arch_diagram"> -@plantuml_start{wt_diagram.png} +@plantuml_start{wt_diagram.png } @startuml{wt_diagram.png} ' We add spacing to the diagram in places to influence the layout. diff --git a/src/third_party/wiredtiger/src/docs/arch-schema-ops.dox b/src/third_party/wiredtiger/src/docs/arch-schema-ops.dox index 8428208c6d9..5d647a05205 100644 --- a/src/third_party/wiredtiger/src/docs/arch-schema-ops.dox +++ b/src/third_party/wiredtiger/src/docs/arch-schema-ops.dox @@ -7,7 +7,7 @@ Schema operations cause an update to the metadata and are performed under a schema lock to avoid concurrent operations on the same object. The following sequence of steps define a generic schema operation: -@plantuml_start{schema_generic.png} +@plantuml_start{schema_generic.png } @startuml{schema_generic.png} :A schema operation; partition with-schema-lock { @@ -26,7 +26,7 @@ object on the filesystem with the right parameters and then creating an entry for this new object into the metadata. The sequence of operations involved in a create for various schema types are as follows: -@plantuml_start{schema_create.png} +@plantuml_start{schema_create.png } @startuml{schema_create.png} :WT_SESSION->create(.,name,.) (__session_create()); @@ -128,7 +128,7 @@ The rename schema operation is responsible for renaming the underlying data object on the filesystem and updating the metadata accordingly. The sequence of operations involved in a rename for various schema types are as follows: -@plantuml_start{schema_rename.png} +@plantuml_start{schema_rename.png } @startuml{schema_rename.png} :WT_SESSION->rename(old-uri, new-uri, .) (__session_rename()); diff --git a/src/third_party/wiredtiger/src/docs/arch-transaction.dox b/src/third_party/wiredtiger/src/docs/arch-transaction.dox index bb26829f509..c1be0f99394 100644 --- a/src/third_party/wiredtiger/src/docs/arch-transaction.dox +++ b/src/third_party/wiredtiger/src/docs/arch-transaction.dox @@ -17,7 +17,7 @@ run at a time per session, and that transaction must complete before another tra started. Since every session is singly-threaded, all the operations in the transaction are executed on the same thread. -@plantuml_start{transaction_lifecycle.png} +@plantuml_start{transaction_lifecycle.png } @startuml{transaction_lifecycle.png} :Transaction Lifecycle; diff --git a/src/third_party/wiredtiger/src/docs/backup.dox b/src/third_party/wiredtiger/src/docs/backup.dox index 5f333eca14d..72188570706 100644 --- a/src/third_party/wiredtiger/src/docs/backup.dox +++ b/src/third_party/wiredtiger/src/docs/backup.dox @@ -133,7 +133,7 @@ this cursor will return the filename as the key. There is no associated value. The information returned will be based on blocks tracked since the time of the previous backup designated with "ID1". New block tracking will be started as "ID2" as well. WiredTiger will maintain modifications from two IDs, the current -and the most recent completed one. Note that all backup identifiers are subject to +and the most recently completed one. Note that all backup identifiers are subject to the same naming restrictions as other configuration naming. See @ref config_intro for details. diff --git a/src/third_party/wiredtiger/src/docs/build-posix.dox b/src/third_party/wiredtiger/src/docs/build-posix.dox index 4f04cc54dd3..eb3528e249a 100644 --- a/src/third_party/wiredtiger/src/docs/build-posix.dox +++ b/src/third_party/wiredtiger/src/docs/build-posix.dox @@ -161,6 +161,34 @@ unless manually specified through the \c -DCC_OPTIMIZE_LEVEL configuration optio a different level of optimization: @code -cmake -DCC_OPTIMIZE_LEVEL=-01 -G Ninja ../. +cmake -DCC_OPTIMIZE_LEVEL=-O1 -G Ninja ../. +@endcode + +@section posix_ctest Running WiredTiger C/C++ Tests + +The WiredTiger CMake build makes available a suite of C/C++ based tests. +To run the available tests you can use our smoke test alias (\c check). Ensure you're in the build directory and execute: + +@code +ctest -j$(nproc) -L check +@endcode + +Alternatively to just run all the tests available: + +@code +ctest -j$(nproc) +@endcode + +In addition, to get verbose output with your test run you can use the \c -VV flag: + +@code +ctest -j$(nproc) -VV +@endcode + +If you want to focus on running a specific test (i.e. run a test that may be failing) you can use the \c -R flag: + +@code +# Note: -R specifies a regex, where any matching test will be run +ctest -j$(nproc) -R test_name @endcode */ diff --git a/src/third_party/wiredtiger/src/docs/encryption.dox b/src/third_party/wiredtiger/src/docs/encryption.dox index a3234e88d1a..69dcc4ad116 100644 --- a/src/third_party/wiredtiger/src/docs/encryption.dox +++ b/src/third_party/wiredtiger/src/docs/encryption.dox @@ -1,124 +1,189 @@ /*! @page encryption Encryptors @section encryption_overview Overview of Encryption in WiredTiger -@ref encryption_custom "Custom encryption engines" may be used to -extend WiredTiger. WiredTiger does not currently offer builtin -support for any particular encryption algorithm. -@ref encryption_examples "Example encryption code" is provided -to demonstrate how encryption extensions are created. - -\warning The encryption infrastructure included in WiredTiger, when used with a -strong encryption algorithm, is intended to protect data stored in files -(that is, <em>encryption at rest</em>). The table content (keys, values), -the metadata pertaining to data (table, index, column names, and other -configuration information) as well as the database log files are encrypted -on disk. Decryption occurs when the data is read into memory; thus an -attacker having the ability to directly read system memory will have access -to unencrypted data. Many systems may also page memory to a backing disk -under load. Access to any such \em paging or \em swap devices must be -considered when planning the security of a system. - -The encryption extension must be loaded in the ::wiredtiger_open call. -See @subpage_single extensions for details on how extensions are loaded. -Also, encryption is specified using \c encryption= in the configuration -for the ::wiredtiger_open call. This configuration establishes -the encryption algorithm and keys to be used for database log files and a subset -of the WiredTiger metadata files. By default, this same encryption -is also used for all data files. We call this the <em>system</em> encryption. - -It is also possible to use different encryption options when individual -data files are first created, using the \c encryption= configuration in -the WT_SESSION::create call. Such options override the default -(<em>system</em>) encryption that was indicated in the ::wiredtiger_open -call for the individual data file. It is possible to turn encryption off -for individual files, to use a different encryptor, or to specify a -different \c keyid. - -Overriding the system encryption for a table does not override -the system encryption for indices on that table, nor does it override -the system encryption for column groups specified on that table. -Encryption for column groups and indices must be specified when they -are created, if they are to be different than the system encryption. - -It is an error to specify encryption in a WT_SESSION::create call when it -was not specified in the ::wiredtiger_open call. This prevents accidental -exposure of the file's data in log files, which would be written in the -clear in such a scenario. - -\warning When using separate keys for individual data files or tables, the -key used for the \em system encryption continues to have fundamental -importance. The database log, protected by the \em system encryption, -contains a shared stream of changes to all data files. Thus, if the \em -system key is exposed, even when per-file keys are not exposed, an attacker -can read database log files, and hence has access to data in individual -files. - -@section encryption_parameters Encryption keyid and secretkey - -Two parameters, \c keyid and \c secretkey, may be specified -when configuring encryption for ::wiredtiger_open to allow the possibility -of varying the algorithm according to different keys. - -The configuration parameter <code>encryption=(keyid=<em>identifier</em>)</code> -may be used in ::wiredtiger_open or WT_SESSION::create calls. This is intended -to reference a key stored using a Key Management Solution (KMS). -The \c keyid given to ::wiredtiger_open is stored in the clear in -WiredTiger configuration files; it should never contain sensitive -information. As an example, with a \c keyid of \c "customerABC", -the encryptor would consult the KMS to return a key previously stored -for \c "customerABC". The encryptor will use the returned key when -applying the encryption. To effectively use the \c keyid, a custom -encryptor must implement the WT_ENCRYPTOR::customize callback. -It is during \c customize that the encryptor has an opportunity to use -the \c keyid to fetch the actual key. The \c customize function is -called on the first use of a \c keyid, and the same \em customized encryptor -will be used with each use of the same \c keyid. - -The configuration parameter <code>encryption=(secretkey)</code> is used -only in the ::wiredtiger_open call. The value of the secretkey -is never stored on disk in any form, so it must always be provided -when WiredTiger is reopened (again, with the ::wiredtiger_open call). -The secretkey is available to the encryptor during the -WT_ENCRYPTOR::customize callback, during which the encryptor may -be \em customized to keep the secretkey or a transformation of it -for use during the WT_ENCRYPTOR::encrypt and WT_ENCRYPTOR::decrypt callbacks. - -If a \c secretkey is used, it must be provided using the \c -E option -when using the \c wt utility. Specifying \c keyid is not needed with the \c wt -utility, as the \c keyid is stored in the clear on disk by WiredTiger. -Any additional \c keyid values needed to decrypt data files -are stored in WiredTiger metadata using the system encryptor. - -@section encryption_custom Custom encryption engines - -WiredTiger may be extended by adding custom encryption engines -that we call \em encryptors. Custom encryptors must be coded in the C language. -Once packaged, they can be used in any language. - -See @subpage_single extensions for general details on extending WiredTiger, -and see WT_ENCRYPTOR for the encryptor interface. - -Custom encryptors are registered by calling WT_CONNECTION::add_encryptor, -this creates an encryptor name that may be referenced using the -<code>encryption=(name=...</code> configuration string in the ::wiredtiger_open -or WT_SESSION::create call. - -@section encryption_examples Encryption examples - -There are two kinds of example code with overlapping functionality. -A simple, self contained encryption example is in @ex_ref{ex_encrypt.c}. -This example includes a small encryptor that rotates letters in the -alphabet by a fixed amount, based on the value of \c keyid. This example -also shows how encryption is configured within an application. -The second set of examples are in \c ext/encryptors. These are -encryptors only (no application level code), showing how encryptors -might be packaged in a loadable shared library. -@ex_ref{nop_encrypt.c} merely copies its input to its output. -@ex_ref{rotn_encrypt.c} is an extended version of the example that -rotates the alphabet. It adds a twist in that a \c secretkey can -be specified, changing the rotation per letter. The Python test suite -uses the rotn encryptor to help test the encryption framework. - -Please note that these samples are for demonstration use only. -They do not provide any security. +This section explains how to configure and work with encryption in WiredTiger. + +WiredTiger now provides built-in support for encryption with an extension using the +open-source <a href=https://libsodium.org>libsodium</a> cryptography library. Application +developers may also write their own @ref encryption_custom "custom encryption extensions". +Three example extensions are provided for expository and testing purposes. + +\warning The encryption infrastructure included in WiredTiger, when used with an extension +providing a strong encryption algorithm, is intended only to protect data stored in files +(that is, it provides <em>encryption at rest</em>). The table content (keys, values), the +metadata pertaining to data (table, index, column names, and other configuration +information) as well as the database log files are encrypted on disk. However, it does not +protect data in memory while the database is running. Decryption occurs when the data is +read into memory; thus an attacker having the ability to directly read system memory will +have access to unencrypted data. Many systems may also page memory to a backing disk under +load. The security of any such \em paging or \em swap devices must be considered when +planning the security of a system; in general, they should be disabled or themselves +encrypted using operating system or storage system facilities. + +@section encryption_use Using an encryption extension +Enabling encryption involves two steps: first, loading the encryptor extension, and +second, enabling and configuring it. Extensions must be loaded in the ::wiredtiger_open +call. See @subpage_single extensions for details about how extensions are loaded. + +Encryption must be enabled first for the overall database (the <em>system</em> encryption) +by passing the encryption configuration (described shortly) to the ::wiredtiger_open call. +This establishes the encryption algorithm and keys to be used for database log files and a +subset of the WiredTiger metadata files. By default, this same encryption is also used for +all data files. + +Encryption may also be enabled separately for individual tables, column groups, and +indices, by passing a different encryption configuration to the WT_SESSION::create call. +This configuration overrides the system encryption for the data file so created. It may +use a different encryptor and/or different key (except as noted below). It is also +possible to disable encryption for individual tables; this is of course usually +undesirable, but possibly a reasonable option for high-churn low-value data. + +Each table, column group, and index that is to have encryption different from the system +encryption must be configured explicitly; encryption configuration is not inherited by +column groups or indices from the table they apply to. Such encryption must be specified +when the data file is created; otherwise the system encryption is used. + +It is an error to specify encryption in a WT_SESSION::create call when it was not +specified in the ::wiredtiger_open call. This prevents accidental exposure of the file's +data in log files, which would be written in the clear in such a scenario. + +\warning When using separate keys for individual data files or tables, the key used for +the system encryption continues to have fundamental importance. The database log, +protected by the system encryption, contains a shared stream of changes to all data +files. Thus, if the system key is exposed, even when per-file keys are not exposed, an +attacker can read database log files, and hence has access to data in individual files. + +Note that because every encryptor providing strong encryption should also include a +cryptographically strong checksum, WiredTiger's own block checksums can safely be disabled. +See WT_SESSION::create. + +@section encryption_config Encryption configuration +Encryption is configured using the \c encryption= configuration parameter. As noted above +this must be passed to ::wiredtiger_open and may also be passed to WT_SESSION::create. The +value of \c encryption= is a nested configuration string with up to three parameters. The +first should be \c name= to select the encryption extension to use. This is the name of +the encryptor and should be the string the encryptor's initialization method passed to +WT_CONNECTION::add_encryptor. For the built-in libsodium extension, this name is "sodium". +The other parameters, \c keyid and \c secretkey, are used to specify the encryption key. +To disable encryption for an individual table the reserved value \c name=none may be +passed when creating it. (As is noted above, this is usually not desirable.) + +@section encryption_keyid Keys via key ID +The configuration parameter <code>encryption=(keyid=<em>identifier</em>)</code> may be +used in ::wiredtiger_open or WT_SESSION::create calls. This is intended to reference a key +stored using a Key Management Solution (KMS). The \c keyid given to ::wiredtiger_open is +stored in the clear in WiredTiger configuration files; it should never contain sensitive +information. Consequently, however, it need only be specified when the database is +created; it is remembered thereafter. Additional \c keyid values may be used freely for +different database files. These are stored in WiredTiger metadata using the system +encryption. + +As an example, with a \c keyid of \c "customerABC", the encryptor would consult the KMS to +return a key previously stored for \c "customerABC". The encryptor would then use the +returned key when encrypting. + +@section encryption_secretkey Keys via explicit secret key +The configuration parameter <code>encryption=(secretkey=<em>key</em>)</code> is used only +in the ::wiredtiger_open call. The value of the secret key is never stored on disk in any +form, so it must always be provided when WiredTiger is reopened (again, with the +::wiredtiger_open call). It may <em>not</em> be passed to WT_SESSION::create; consequently +if no KMS is available different keys cannot be used for individual tables. + +If a \c secretkey is used, it must be provided using the \c -E command-line option when +using the \c wt utility. Specifying \c keyid is not needed with the \c wt utility, as the +\c keyid is stored in the clear on disk by WiredTiger. Note that while the \c wt utility +takes some steps to prevent it, on most systems it is possible to see command-line +arguments using operating system facilities. This creates some risk of exposure that must +be considered. Using a KMS is probably preferable where possible. Alternatively, one might +avoid using the \c wt utility entirely when using encryption, as it is not necessary. + +Both \c keyid and \c secretkey values are not interpreted by the configuration framework; +their exact interpretation and the sets of possible values are determined by the encryptor +in use. + +@section encryption_custom Custom encryption extensions + +Applications may provide their own encryption extensions. Custom encryptors must be coded +in the C language. Once packaged, they can be used in any language. For further +information, see: @ref extensions "general information on extending WiredTiger", and the +encryptor interface WT_ENCRYPTOR. + +A simple example @ref ex_encrypt.c "application with its own encryptor" is provided to +demonstrate how this may be done. It shows both how to install a custom extension and +then how to configure it. (Note, however, that it uses a trivial "encryption" that +provides no security.) + +Three additional encryptor extensions are included with WiredTiger. These are built as +shared libraries to show how that can be done. + +@section encryption_sodium The libsodium encryption extension +The "sodium" encryptor uses the open source <a href=https://libsodium.org>libsodium</a> +cryptography library. It currently has no support for key management and thus does not +support \c keyid configurations, only \c secretkey. The \c secretkey is expected to be a +256-bit XChaCha20 key encoded in hex (with no leading <tt>0x</tt>). This is not directly +suitable for applications that wish to use manually-entered passwords or passphrases. +Applications for which these restrictions are unsuitable can copy the @ref +sodium_encrypt.c "extension code" and extend it as needed, using additional material from +libsodium or other libraries. + +The sodium encryptor is intended to protect against disclosure of data, such as might +otherwise happen with a database on a stolen laptop. It may provide some protection +against malicious corruption of a database by an adversary with write access to the +database files, but is not intended for this purpose and should not be relied upon in this +context. (Note that on most operating systems, and in most situations, an adversary with +write access to the database files is highly likely to also be able to read data out of +memory while the database is running, and change it as well, so this limitation is not +necessarily important.) + +To use the encryptor, first install libsodium; then you can compile WiredTiger with the +extension by passing the \c --enable-sodium option to configure, or the argument \c +-DENABLE_SODIUM=1 to cmake. + +If libsodium is installed in a location not normally searched by the compiler toolchain, +you'll need to modify the \c CPPFLAGS and \c LDFLAGS to indicate these locations. For +example, with the libsodium includes and libraries installed in \c /usr/local/include and +\c /usr/local/lib, you would run configure with the following additional arguments: + +@code +--enable-sodium CPPFLAGS="-I/usr/local/include" LDFLAGS="-L/usr/local/include" +@endcode + +Alternatively you would run cmake with the following additional arguments: + +@code +-DENABLE_SODIUM=1 -DCMAKE_INCLUDE_PATH=/usr/local/include -DCMAKE_LIBRARY_PATH=/usr/local/lib +@endcode + +When opening the WiredTiger database, load the libsodium plugin library as an extension. +For example, with the WiredTiger library installed in \c /usr/local/lib, you would use the +following code: + +@snippet ex_all.c Configure sodium extension + +where \c secretkey comes from some suitable place. (Never compile in cryptographic keys.) + +Because this encryptor includes a cryptographically strong checksum, it is safe to +disable WiredTiger's own block checksums. + +@section encryption_nop The nop encryption extension +The "nop" encryptor is a skeleton extension into which suitable cryptographic code may be +inserted. It is @ref nop_encrypt.c "provided as a framework" for developers writing +custom encryptors to copy as a means of getting started. It does not do anything and +should obviously not be used in production. + +@section encryption_rotn The rotn encryption extension +The "rotn" encryptor is a test extension that performs either of two forms of trivial +"encryption". It is used for testing the encryption framework. It provides no security +and should never be used in production. + +@section encryption_ex The ex_encrypt example +The @ex_ref{ex_encrypt.c} example (mentioned above) provides a runnable example of a +simple application that loads a custom encryptor compiled into the application (rather +than as a plugin library). The "encryption" it uses is trivial and should never be used +in production. + +\warning Developers writing custom encryptor extensions should be sure to consult +appropriate references to ensure that what they do is cryptographically sound and +addresses the threat model they have in mind. Proper use of cryptographic primitives and +constructions is a complex topic far beyond the scope of this documentation. */ diff --git a/src/third_party/wiredtiger/src/docs/examples.dox b/src/third_party/wiredtiger/src/docs/examples.dox index d5a5102be61..40e2670838a 100644 --- a/src/third_party/wiredtiger/src/docs/examples.dox +++ b/src/third_party/wiredtiger/src/docs/examples.dox @@ -61,4 +61,7 @@ Shows the basic framework for building an encryptor as a plug in library. @example rotn_encrypt.c Shows a simple encryptor as a plug in library. +@example sodium_encrypt.c +An encryptor using the <a href=https://libsodium.org>libsodium</a> cryptography library. + */ diff --git a/src/third_party/wiredtiger/src/docs/schema.dox b/src/third_party/wiredtiger/src/docs/schema.dox index 872fdf395b8..57f2801884f 100644 --- a/src/third_party/wiredtiger/src/docs/schema.dox +++ b/src/third_party/wiredtiger/src/docs/schema.dox @@ -66,24 +66,23 @@ struct module to describe the types of columns in a table: http://docs.python.org/library/struct <table> -@hrow{Format, C Type, Java type, Python type, Notes} +@hrow{Format, C Type, Python type, Notes} @row{\c x, N/A, N/A, N/A, pad byte\, no associated value} -@row{\c b, \c int8_t, \c byte, \c int, signed byte} -@row{\c B, \c uint8_t, \c byte, \c int, unsigned byte} -@row{\c h, \c int16_t, \c short, \c int, signed 16-bit} -@row{\c H, \c uint16_t, \c short, \c int, unsigned 16-bit} -@row{\c i, \c int32_t, \c int, \c int, signed 32-bit} -@row{\c I, \c uint32_t, \c int, \c int, unsigned 32-bit} -@row{\c l, \c int32_t, \c int, \c int, signed 32-bit} -@row{\c L, \c uint32_t, \c int, \c int, unsigned 32-bit} -@row{\c q, \c int64_t, \c long, \c int, signed 64-bit} -@row{\c Q, \c uint64_t, \c long, \c int, unsigned 64-bit} -@row{\c r, \c uint64_t, \c long, \c int, record number} -@row{\c s, \c char[], \c String, \c str, fixed-length string} -@row{\c S, \c char[], \c String, \c str, NUL-terminated string} -@row{\c t, \c uint8_t, \c byte, \c int, fixed-length bit field} -@row{\c u, <code>WT_ITEM *</code>, <code>byte[]</code>, \c str, - raw byte array} +@row{\c b, \c int8_t, \c int, signed byte} +@row{\c B, \c uint8_t, \c int, unsigned byte} +@row{\c h, \c int16_t, \c int, signed 16-bit} +@row{\c H, \c uint16_t, \c int, unsigned 16-bit} +@row{\c i, \c int32_t, \c int, signed 32-bit} +@row{\c I, \c uint32_t, \c int, unsigned 32-bit} +@row{\c l, \c int32_t, \c int, signed 32-bit} +@row{\c L, \c uint32_t, \c int, unsigned 32-bit} +@row{\c q, \c int64_t, \c int, signed 64-bit} +@row{\c Q, \c uint64_t, \c int, unsigned 64-bit} +@row{\c r, \c uint64_t, \c int, record number} +@row{\c s, \c char[], \c str, fixed-length string} +@row{\c S, \c char[], \c str, NUL-terminated string} +@row{\c t, \c uint8_t, \c int, fixed-length bit field} +@row{\c u, <code>WT_ITEM *</code>, \c bytes, raw byte array} </table> The \c 'r' type is used for record number keys in column stores. It is diff --git a/src/third_party/wiredtiger/src/docs/spell.ok b/src/third_party/wiredtiger/src/docs/spell.ok index 69103d28d1c..d4b22545abe 100644 --- a/src/third_party/wiredtiger/src/docs/spell.ok +++ b/src/third_party/wiredtiger/src/docs/spell.ok @@ -5,12 +5,9 @@ ASAN ActiveState Adler's Atomicity -autoconf BLOBs BLRrVv -ccache CFLAGS -cmake CMake COV CPPFLAGS @@ -29,19 +26,19 @@ Coverity's Cyclomatic DB's DBTs +DCC +DCMAKE +DENABLE DHANDLE +DHAVE DONTNEED +DSPINLOCK +DWT Datastore DbCursor DbEnv DbMultiple -DCC -DCMAKE -DENABLE -DHAVE -DSPINLOCK Durations -DWT EAGAIN EB EBUSY @@ -118,6 +115,7 @@ Roelofs RrVv Rrx SCons +SYSV Sanitizer Seward's SiH @@ -126,6 +124,7 @@ TXT UBSAN UNC URIs +VV Valgrind WILLNEED WiredTiger @@ -141,6 +140,7 @@ WiredTigerRollbackException WiredTigerStat WiredTigerTestCase WiredTigerTmplog +XChaCha XRay Yann Za @@ -187,6 +187,7 @@ cacheable cachesize calc callbk +ccache cd cdb cds @@ -197,6 +198,7 @@ checksums ckp ckpt clickable +cmake colgroup colgroups combinatorial @@ -217,7 +219,10 @@ cppsuite cpu crashless crc +cryptographic +cryptographically csuite +ctest curfile cursortype curtable @@ -269,6 +274,7 @@ dups encrypt encryptor encryptors +encryptor's endcode endcond endian @@ -380,6 +386,7 @@ libdir libfuzzer libhe libkvs +libsodium libtool libwiredtiger lifecycle @@ -456,6 +463,7 @@ nosync notgranted notyet nowait +nproc ns nul num @@ -578,7 +586,6 @@ superset svg sys syscalls -SYSV sz tRuE tablename @@ -599,6 +606,7 @@ touchpad tpc tpcb trackpad +tradeoffs transactional transactionally tt diff --git a/src/third_party/wiredtiger/src/docs/tune-checksum.dox b/src/third_party/wiredtiger/src/docs/tune-checksum.dox index bd7b2fb6ecc..822512c4f3b 100644 --- a/src/third_party/wiredtiger/src/docs/tune-checksum.dox +++ b/src/third_party/wiredtiger/src/docs/tune-checksum.dox @@ -1,22 +1,27 @@ /*! @page tune_checksum Checksums -WiredTiger optionally checksums file reads and writes to detect storage -failures. In read-only applications, or when file compression provides -any necessary checksum functionality, or when using backing storage -systems where blocks require no validation, performance can be increased -by turning off checksum support when calling the WT_SESSION::create +WiredTiger optionally checksums file reads and writes to detect storage failures. In read-only +applications, or when configured compression or encryption provides sufficient checksum +functionality, or when using backing storage systems where blocks require no validation, +performance can be increased by turning off checksum support when calling the WT_SESSION::create method. -Checksums can be configured to be "off" or "on", as well as "uncompressed". -The "uncompressed" configuration checksums blocks not otherwise protected -by compression. For example, in a system where the compression algorithms -provide sufficient protection against corruption, and when writing a block -which is too small to be usefully compressed, setting the checksum -configuration value to "uncompressed" causes WiredTiger to checksum the -blocks which are not compressed: +Checksums can be configured to be "off", "on", "uncompressed" or "unencrypted". The default is +"on", in which case all block writes include a checksum subsequently verified when the block is +read. The "off" setting does no checksums, intended for read-only applications or applications +with storage systems that detect block corruption. The "uncompressed" setting only checksums +blocks that are not compressed; the "unencrypted" setting only checksums blocks that are not +encrypted. The "uncompressed" and "unencrypted" settings are intended for applications with +compression or encryption engines that detect block corruption. Corruption detection is often +the case for encryption engines (including the sodium engine included in WiredTiger). Corruption +detection is often not the case for compression engines, specifically, applications configuring +the lz4, snappy, zlib and zstd compression engines included in WiredTiger are encouraged to +consider enabling separate checksum support for increased reliability. Of course, the failure +profile of the underlying storage layer and the cost of an undetected file corruption are +tradeoffs against the performance improvement of turning off checksums. -@snippet ex_all.c Configure checksums to uncompressed +The default WiredTiger checksum configuration is "on". -The default WiredTiger configuration is \c uncompressed. +@snippet ex_all.c Configure checksums to uncompressed */ diff --git a/src/third_party/wiredtiger/src/docs/wtperf.dox b/src/third_party/wiredtiger/src/docs/wtperf.dox index 916755ade32..739a7e96e15 100644 --- a/src/third_party/wiredtiger/src/docs/wtperf.dox +++ b/src/third_party/wiredtiger/src/docs/wtperf.dox @@ -190,6 +190,10 @@ if non zero choose a value from within this range as the key for insert operatio generate random content for the value @par range_partition (boolean, default=false) partition data by range (vs hash) +@par read_range (unsigned int, default=0) +read a sequential range of keys upon each read operation. This value tells us how many keys to read each time, or an upper bound on the number of keys read if read_range_random is set. +@par read_range_random (boolean, default=false) +if doing range reads, select the number of keys to read in a range uniformly at random. @par readonly (boolean, default=false) reopen the connection between populate and workload phases in readonly mode. Requires reopen_connection turned on (default). Requires that read be the only workload specified @par reopen_connection (boolean, default=true) @@ -212,6 +216,8 @@ scan tables every interval seconds during the workload phase, 0 to disable percentage of entire data set scanned, if scan_interval is enabled @par scan_table_count (unsigned int, default=0) number of separate tables to be used for scanning. Zero indicates that tables are shared with other operations +@par select_latest (boolean, default=false) +in workloads that involve inserts and another type of operation, select the recently inserted records with higher probability @par sess_config (string, default="") session configuration string @par session_count_idle (unsigned int, default=0) diff --git a/src/third_party/wiredtiger/src/evict/evict_file.c b/src/third_party/wiredtiger/src/evict/evict_file.c index 1e83f0a5643..f6091a4436e 100644 --- a/src/third_party/wiredtiger/src/evict/evict_file.c +++ b/src/third_party/wiredtiger/src/evict/evict_file.c @@ -85,24 +85,17 @@ __wt_evict_file(WT_SESSION_IMPL *session, WT_CACHE_OP syncop) switch (syncop) { case WT_SYNC_CLOSE: - /* - * Evict the page. - * - * Ensure the ref state is restored to the previous value if eviction fails. - */ + /* Evict the page. */ WT_ERR(__wt_evict(session, ref, ref->state, WT_EVICT_CALL_CLOSING)); break; case WT_SYNC_DISCARD: /* - * Discard the page regardless of whether it is dirty. - * - * If the page has a page deleted structure, we are discarding the page that is cleaned - * by a checkpoint. + * Discard the page whether it is dirty or not. The check if the page can be evicted is + * not exhaustive, but provides basic checking on the page's status. */ WT_ASSERT(session, F_ISSET(dhandle, WT_DHANDLE_DEAD) || F_ISSET(S2C(session), WT_CONN_CLOSING) || - __wt_page_can_evict(session, ref, NULL) || - (ref->page_del != NULL && page->modify->page_state == WT_PAGE_CLEAN)); + __wt_page_can_evict(session, ref, NULL)); __wt_ref_out(session, ref); break; case WT_SYNC_CHECKPOINT: diff --git a/src/third_party/wiredtiger/src/evict/evict_lru.c b/src/third_party/wiredtiger/src/evict/evict_lru.c index 26f572ce517..b0cada52261 100644 --- a/src/third_party/wiredtiger/src/evict/evict_lru.c +++ b/src/third_party/wiredtiger/src/evict/evict_lru.c @@ -667,7 +667,6 @@ __evict_pass(WT_SESSION_IMPL *session) { WT_CACHE *cache; WT_CONNECTION_IMPL *conn; - WT_DECL_RET; WT_TXN_GLOBAL *txn_global; uint64_t eviction_progress, oldest_id, prev_oldest_id; uint64_t time_now, time_prev; @@ -728,23 +727,8 @@ __evict_pass(WT_SESSION_IMPL *session) */ if (!WT_EVICT_HAS_WORKERS(session) && (cache->evict_empty_score < WT_EVICT_SCORE_CUTOFF || - !__evict_queue_empty(cache->evict_urgent_queue, false))) { - /* - * Release the evict pass lock as the thread is going to evict the pages from the queue. - * Otherwise, it can lead to a deadlock while evicting the page in the flow of clearing - * the eviction walk. - * - * As there is only one eviction thread that is active currently, there couldn't be any - * race conditions that other threads can enter into the flow of evict server when there - * is already another server is running. - */ - FLD_CLR(session->lock_flags, WT_SESSION_LOCKED_PASS); - __wt_spin_unlock(session, &cache->evict_pass_lock); - ret = __evict_lru_pages(session, true); - __wt_spin_lock(session, &cache->evict_pass_lock); - FLD_SET(session->lock_flags, WT_SESSION_LOCKED_PASS); - WT_RET(ret); - } + !__evict_queue_empty(cache->evict_urgent_queue, false))) + WT_RET(__evict_lru_pages(session, true)); if (cache->pass_intr != 0) break; @@ -1689,7 +1673,7 @@ __evict_walk_target(WT_SESSION_IMPL *session) if (target_pages < MIN_PAGES_PER_TREE) target_pages = MIN_PAGES_PER_TREE; - /* If the tree is dead, take a lot of pages. */ + /* If the tree is dead, take a lot of pages. */ if (F_ISSET(session->dhandle, WT_DHANDLE_DEAD)) target_pages *= 10; diff --git a/src/third_party/wiredtiger/src/evict/evict_page.c b/src/third_party/wiredtiger/src/evict/evict_page.c index 26b38dc5996..de249ec4a4b 100644 --- a/src/third_party/wiredtiger/src/evict/evict_page.c +++ b/src/third_party/wiredtiger/src/evict/evict_page.c @@ -165,9 +165,11 @@ __wt_evict(WT_SESSION_IMPL *session, WT_REF *ref, uint8_t previous_state, uint32 if (inmem_split) goto done; - /* Check that we are not about to evict an internal page with an active split generation. */ - if (F_ISSET(ref, WT_REF_FLAG_INTERNAL) && !closing) - WT_ASSERT(session, !__wt_gen_active(session, WT_GEN_SPLIT, page->pg_intl_split_gen)); + /* Check we are not evicting an accessible internal page with an active split generation. */ + WT_ASSERT(session, + closing || !F_ISSET(ref, WT_REF_FLAG_INTERNAL) || + F_ISSET(session->dhandle, WT_DHANDLE_DEAD | WT_DHANDLE_EXCLUSIVE) || + !__wt_gen_active(session, WT_GEN_SPLIT, page->pg_intl_split_gen)); /* Count evictions of internal pages during normal operation. */ if (!closing && F_ISSET(ref, WT_REF_FLAG_INTERNAL)) @@ -184,15 +186,6 @@ __wt_evict(WT_SESSION_IMPL *session, WT_REF *ref, uint8_t previous_state, uint32 /* Figure out whether reconciliation was done on the page */ clean_page = __wt_page_evict_clean(page); - /* - * Discard all page-deleted information. If a truncate call deleted this page, there's memory - * associated with it we no longer need, eviction will have built a new version of the page. - */ - if (ref->page_del != NULL) { - __wt_free(session, ref->page_del->update_list); - __wt_free(session, ref->page_del); - } - /* Update the reference and discard the page. */ if (__wt_ref_is_root(ref)) __wt_ref_out(session, ref); @@ -507,7 +500,7 @@ __evict_review(WT_SESSION_IMPL *session, WT_REF *ref, uint32_t evict_flags, bool WT_PAGE *page; uint32_t flags; bool closing, modified, snapshot_acquired; - bool use_snapshot_for_app_thread, is_eviction_thread; + bool is_eviction_thread, use_snapshot_for_app_thread; *inmem_splitp = false; @@ -530,9 +523,7 @@ __evict_review(WT_SESSION_IMPL *session, WT_REF *ref, uint32_t evict_flags, bool WT_RET(ret); } - /* - * It is always OK to evict pages from dead trees if they don't have children. - */ + /* It is always OK to evict pages from dead trees if they don't have children. */ if (F_ISSET(session->dhandle, WT_DHANDLE_DEAD)) return (0); @@ -674,34 +665,74 @@ __evict_review(WT_SESSION_IMPL *session, WT_REF *ref, uint32_t evict_flags, bool /* Make sure that both conditions above are not true at the same time. */ WT_ASSERT(session, !use_snapshot_for_app_thread || !is_eviction_thread); - if (!conn->txn_global.checkpoint_running && !WT_IS_HS(btree->dhandle) && - (use_snapshot_for_app_thread || is_eviction_thread)) { + /* + * History store data is always evictable. + * + * If checkpoint is running concurrently, set the checkpoint running flag and we will abort the + * eviction if we detect out of order timestamp updates. + */ + if (WT_IS_HS(btree->dhandle)) { + /* + * FIX-ME-WT-5316: remove this when we have removed history store score and + * __rec_update_stable. No need to set visibility flag for history store. + */ + if (!WT_SESSION_BTREE_SYNC(session)) + LF_SET(WT_REC_VISIBLE_ALL); + } else { if (is_eviction_thread) { - /* - * Eviction threads do not need to pin anything in the cache. We have a exclusive lock - * for the page being evicted so we are sure that the page will always be there while it - * is being processed. Therefore, we use snapshot API that doesn't publish shared IDs to - * the outside world. - */ - __wt_txn_bump_snapshot(session); - snapshot_acquired = true; - - /* - * Make sure once more that there is no checkpoint running. A new checkpoint might have - * started between previous check and acquiring snapshot. If there is a checkpoint - * running, release the snapshot and fallback to global visibility checks. - */ - if (conn->txn_global.checkpoint_running) { - __wt_txn_release_snapshot(session); - snapshot_acquired = false; + /* Eviction thread doing eviction. */ + if (!conn->txn_global.checkpoint_running) { + /* + * Eviction threads do not need to pin anything in the cache. We have an exclusive + * lock for the page being evicted so we are sure that the page will always be there + * while it is being processed. Therefore, we use snapshot API that doesn't publish + * shared IDs to the outside world. + */ + __wt_txn_bump_snapshot(session); + snapshot_acquired = true; + + /* + * Make sure once more that there is no checkpoint running. A new checkpoint might + * have started between previous check and acquiring snapshot. If there is a + * checkpoint running, release the snapshot and fallback to global visibility + * checks. + * + * There should be a read barrier here otherwise the second read may be optimized + * away by the compiler. However, we should have already met a barrier when we bump + * the snapshot so the barrier is omitted here. + */ + if (conn->txn_global.checkpoint_running) { + __wt_txn_release_snapshot(session); + snapshot_acquired = false; + LF_SET(WT_REC_VISIBLE_ALL | WT_REC_CHECKPOINT_RUNNING); + } + } else { + if (!WT_SESSION_BTREE_SYNC(session)) + LF_SET(WT_REC_VISIBLE_ALL); + LF_SET(WT_REC_CHECKPOINT_RUNNING); + } + } else if (use_snapshot_for_app_thread) { + /* Application thread that has a snapshot doing eviction. */ + if (!conn->txn_global.checkpoint_running) + /* + * Use application snapshot for eviction only when checkpoint is not running. + * + * Checkpoint may start concurrently at this point but that is OK as it should have + * obtained a newer snapshot than the application thread. + */ + LF_SET(WT_REC_APP_EVICTION_SNAPSHOT); + else { + if (!WT_SESSION_BTREE_SYNC(session)) + LF_SET(WT_REC_VISIBLE_ALL); + LF_SET(WT_REC_CHECKPOINT_RUNNING); } + } else { + if (conn->txn_global.checkpoint_running) + LF_SET(WT_REC_CHECKPOINT_RUNNING); + if (!WT_SESSION_BTREE_SYNC(session)) + LF_SET(WT_REC_VISIBLE_ALL); } - if (is_eviction_thread && !snapshot_acquired) - LF_SET(WT_REC_VISIBLE_ALL); - if (use_snapshot_for_app_thread) - LF_SET(WT_REC_APP_EVICTION_SNAPSHOT); - } else if (!WT_SESSION_BTREE_SYNC(session)) - LF_SET(WT_REC_VISIBLE_ALL); + } WT_ASSERT(session, LF_ISSET(WT_REC_VISIBLE_ALL) || F_ISSET(session->txn, WT_TXN_HAS_SNAPSHOT)); diff --git a/src/third_party/wiredtiger/src/history/hs_cursor.c b/src/third_party/wiredtiger/src/history/hs_cursor.c index 4f613e7d042..f9155063fe1 100644 --- a/src/third_party/wiredtiger/src/history/hs_cursor.c +++ b/src/third_party/wiredtiger/src/history/hs_cursor.c @@ -25,8 +25,14 @@ __wt_hs_modify(WT_CURSOR_BTREE *hs_cbt, WT_UPDATE *hs_upd) * We don't have exclusive access to the history store page so we need to pass "false" here to * ensure that we're locking when inserting new keys to an insert list. */ +#ifdef HAVE_DIAGNOSTIC + WT_WITH_BTREE(CUR2S(hs_cbt), CUR2BT(hs_cbt), + ret = + __wt_row_modify(hs_cbt, &hs_cbt->iface.key, NULL, hs_upd, WT_UPDATE_INVALID, false, false)); +#else WT_WITH_BTREE(CUR2S(hs_cbt), CUR2BT(hs_cbt), ret = __wt_row_modify(hs_cbt, &hs_cbt->iface.key, NULL, hs_upd, WT_UPDATE_INVALID, false)); +#endif return (ret); } diff --git a/src/third_party/wiredtiger/src/history/hs_rec.c b/src/third_party/wiredtiger/src/history/hs_rec.c index 43fc87989e4..8375de8ffdf 100644 --- a/src/third_party/wiredtiger/src/history/hs_rec.c +++ b/src/third_party/wiredtiger/src/history/hs_rec.c @@ -63,7 +63,7 @@ __hs_verbose_cache_stats(WT_SESSION_IMPL *session, WT_BTREE *btree) */ static int __hs_insert_record(WT_SESSION_IMPL *session, WT_CURSOR *cursor, WT_BTREE *btree, const WT_ITEM *key, - const uint8_t type, const WT_ITEM *hs_value, WT_TIME_WINDOW *tw) + const uint8_t type, const WT_ITEM *hs_value, WT_TIME_WINDOW *tw, bool checkpoint_running) { #ifdef HAVE_DIAGNOSTIC WT_CURSOR_BTREE *hs_cbt; @@ -150,16 +150,28 @@ __hs_insert_record(WT_SESSION_IMPL *session, WT_CURSOR *cursor, WT_BTREE *btree, &upd_type_full_diag, existing_val)); WT_ERR(__wt_compare(session, NULL, existing_val, hs_value, &cmp)); /* - * Same value should not be inserted again unless 1. previous entry is already - * deleted(i.e. the stop timestamp is globally visible), 2. from a different - * transaction 3. with a different timestamp if from the same transaction. + * The same value should not be inserted again unless: + * 1. the previous entry is already deleted (i.e. the stop timestamp is globally + * visible) + * 2. it came from a different transaction + * 3. it came from the same transaction but with a different timestamp */ - if (cmp == 0) - WT_ASSERT(session, - __wt_txn_tw_stop_visible_all(session, &hs_cbt->upd_value->tw) || - tw->start_txn == WT_TXN_NONE || - tw->start_txn != hs_cbt->upd_value->tw.start_txn || - tw->start_ts != hs_cbt->upd_value->tw.start_ts); + if (cmp == 0) { + if (!__wt_txn_tw_stop_visible_all(session, &hs_cbt->upd_value->tw) && + tw->start_txn != WT_TXN_NONE && + tw->start_txn == hs_cbt->upd_value->tw.start_txn && + tw->start_ts == hs_cbt->upd_value->tw.start_ts) { + /* + * If we have issues with duplicate history store records, we want to be able to + * distinguish between modifies and full updates. Since modifies are not + * idempotent, having them inserted multiple times can cause invalid values to + * be read. + */ + WT_ASSERT(session, + type != WT_UPDATE_MODIFY && (uint8_t)upd_type_full_diag != WT_UPDATE_MODIFY); + WT_ASSERT(session, false && "Duplicate values inserted into history store"); + } + } counter = hs_counter + 1; } #else @@ -185,9 +197,20 @@ __hs_insert_record(WT_SESSION_IMPL *session, WT_CURSOR *cursor, WT_BTREE *btree, if (!hs_read_all_flag) F_CLR(cursor, WT_CURSTD_HS_READ_ALL); } - if (ret == 0) + if (ret == 0) { + /* + * Fail the eviction if we detect out of order timestamp when checkpoint is running. We + * cannot modify the history store to fix the out of order timestamp updates as it may make + * the history store checkpoint inconsistent. + */ + if (checkpoint_running) { + ret = EBUSY; + WT_STAT_CONN_INCR(session, cache_eviction_fail_checkpoint_out_of_order_ts); + goto err; + } WT_ERR(__hs_delete_reinsert_from_pos( session, cursor, btree->id, key, tw->start_ts + 1, true, &counter)); + } #ifdef HAVE_DIAGNOSTIC /* @@ -263,8 +286,8 @@ __hs_next_upd_full_value(WT_SESSION_IMPL *session, WT_UPDATE_VECTOR *updates, * fails or succeeds, if there is a successful write to history, cache_write_hs is set to true. */ int -__wt_hs_insert_updates( - WT_SESSION_IMPL *session, WT_PAGE *page, WT_MULTI *multi, bool *cache_write_hs) +__wt_hs_insert_updates(WT_SESSION_IMPL *session, WT_PAGE *page, WT_MULTI *multi, + bool *cache_write_hs, bool checkpoint_running) { WT_BTREE *btree, *hs_btree; WT_CURSOR *hs_cursor; @@ -411,6 +434,17 @@ __wt_hs_insert_updates( if (min_ts_upd != NULL && min_ts_upd->start_ts < upd->start_ts && out_of_order_ts_upd != min_ts_upd) { /* + * Fail the eviction if we detect out of order timestamp when checkpoint is running. + * We cannot modify the history store to fix the out of order timestamp updates as + * it may make the history store checkpoint inconsistent. + */ + if (checkpoint_running) { + ret = EBUSY; + WT_STAT_CONN_INCR(session, cache_eviction_fail_checkpoint_out_of_order_ts); + goto err; + } + + /* * Always insert full update to the history store if we detect out of order * timestamp update. */ @@ -477,10 +511,9 @@ __wt_hs_insert_updates( fix_ts_upd = oldest_upd; if (!F_ISSET(fix_ts_upd, WT_UPDATE_FIXED_HS)) { - /* Delete and reinsert any update of the key with a higher timestamp. - */ - WT_ERR(__wt_hs_delete_key_from_ts( - session, hs_cursor, btree->id, key, fix_ts_upd->start_ts + 1, true)); + /* Delete and reinsert any update of the key with a higher timestamp. */ + WT_ERR(__wt_hs_delete_key_from_ts(session, hs_cursor, btree->id, key, + fix_ts_upd->start_ts + 1, true, checkpoint_running)); F_SET(fix_ts_upd, WT_UPDATE_FIXED_HS); } } @@ -490,13 +523,13 @@ __wt_hs_insert_updates( hs_inserted = squashed = false; /* - * Flush the updates on stack. Stopping once we run out or we reach the onpage upd start - * time point, we can squash updates with the same start time point as the onpage update - * away. + * Flush the updates on stack. Stopping once we run out or we reach the onpage update or we + * encounter a prepared update. */ modify_cnt = 0; - for (; updates.size > 0; tmp = full_value, full_value = prev_full_value, - prev_full_value = tmp, upd = prev_upd) { + for (; updates.size > 0 && upd->prepare_state != WT_PREPARE_INPROGRESS; + tmp = full_value, full_value = prev_full_value, prev_full_value = tmp, + upd = prev_upd) { /* We should never insert the onpage value to the history store. */ WT_ASSERT(session, upd != list->onpage_upd); WT_ASSERT(session, upd->type == WT_UPDATE_STANDARD || upd->type == WT_UPDATE_MODIFY); @@ -631,14 +664,14 @@ __wt_hs_insert_updates( __wt_calc_modify(session, prev_full_value, full_value, prev_full_value->size / 10, entries, &nentries) == 0) { WT_ERR(__wt_modify_pack(hs_cursor, entries, nentries, &modify_value)); - ret = __hs_insert_record( - session, hs_cursor, btree, key, WT_UPDATE_MODIFY, modify_value, &tw); + ret = __hs_insert_record(session, hs_cursor, btree, key, WT_UPDATE_MODIFY, + modify_value, &tw, checkpoint_running); __wt_scr_free(session, &modify_value); ++modify_cnt; } else { modify_cnt = 0; - ret = __hs_insert_record( - session, hs_cursor, btree, key, WT_UPDATE_STANDARD, full_value, &tw); + ret = __hs_insert_record(session, hs_cursor, btree, key, WT_UPDATE_STANDARD, + full_value, &tw, checkpoint_running); } /* @@ -681,10 +714,10 @@ __wt_hs_insert_updates( } #endif __wt_update_vector_clear(&out_of_order_ts_updates); + __wt_update_vector_clear(&updates); } WT_ERR(__wt_block_manager_named_size(session, WT_HS_FILE, &hs_size)); - WT_STAT_CONN_SET(session, cache_hs_ondisk, hs_size); hs_btree = __wt_curhs_get_btree(hs_cursor); max_hs_size = hs_btree->file_max; if (max_hs_size != 0 && (uint64_t)hs_size > max_hs_size) @@ -720,7 +753,7 @@ err: */ int __wt_hs_delete_key_from_ts(WT_SESSION_IMPL *session, WT_CURSOR *hs_cursor, uint32_t btree_id, - const WT_ITEM *key, wt_timestamp_t ts, bool reinsert) + const WT_ITEM *key, wt_timestamp_t ts, bool reinsert, bool checkpoint_running) { WT_DECL_RET; WT_ITEM hs_key; @@ -749,6 +782,17 @@ __wt_hs_delete_key_from_ts(WT_SESSION_IMPL *session, WT_CURSOR *hs_cursor, uint3 ++hs_counter; } + /* + * Fail the eviction if we detect out of order timestamp when checkpoint is running. We cannot + * modify the history store to fix the out of order timestamp updates as it may make the history + * store checkpoint inconsistent. + */ + if (checkpoint_running) { + ret = EBUSY; + WT_STAT_CONN_INCR(session, cache_eviction_fail_checkpoint_out_of_order_ts); + goto err; + } + WT_ERR( __hs_delete_reinsert_from_pos(session, hs_cursor, btree_id, key, ts, reinsert, &hs_counter)); done: diff --git a/src/third_party/wiredtiger/src/include/btmem.h b/src/third_party/wiredtiger/src/include/btmem.h index f86adf08d53..211f2f220b0 100644 --- a/src/third_party/wiredtiger/src/include/btmem.h +++ b/src/third_party/wiredtiger/src/include/btmem.h @@ -8,7 +8,7 @@ #define WT_RECNO_OOB 0 /* Illegal record number */ -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_READ_CACHE 0x0001u #define WT_READ_DELETED_CHECK 0x0002u #define WT_READ_DELETED_SKIP 0x0004u @@ -22,20 +22,21 @@ #define WT_READ_SKIP_INTL 0x0400u #define WT_READ_TRUNCATE 0x0800u #define WT_READ_WONT_NEED 0x1000u -/* AUTOMATIC FLAG VALUE GENERATION STOP */ +/* AUTOMATIC FLAG VALUE GENERATION STOP 32 */ -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_REC_APP_EVICTION_SNAPSHOT 0x001u #define WT_REC_CALL_URGENT 0x002u #define WT_REC_CHECKPOINT 0x004u -#define WT_REC_CLEAN_AFTER_REC 0x008u -#define WT_REC_EVICT 0x010u -#define WT_REC_HS 0x020u -#define WT_REC_IN_MEMORY 0x040u -#define WT_REC_SCRUB 0x080u -#define WT_REC_VISIBILITY_ERR 0x100u -#define WT_REC_VISIBLE_ALL 0x200u -/* AUTOMATIC FLAG VALUE GENERATION STOP */ +#define WT_REC_CHECKPOINT_RUNNING 0x008u +#define WT_REC_CLEAN_AFTER_REC 0x010u +#define WT_REC_EVICT 0x020u +#define WT_REC_HS 0x040u +#define WT_REC_IN_MEMORY 0x080u +#define WT_REC_SCRUB 0x100u +#define WT_REC_VISIBILITY_ERR 0x200u +#define WT_REC_VISIBLE_ALL 0x400u +/* AUTOMATIC FLAG VALUE GENERATION STOP 32 */ /* * WT_PAGE_HEADER -- @@ -184,10 +185,10 @@ struct __wt_ovfl_reuse { * reconciliation fails for any reason, discard the newly added skiplist entries, along with their * underlying blocks. */ -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_OVFL_REUSE_INUSE 0x1u #define WT_OVFL_REUSE_JUST_ADDED 0x2u - /* AUTOMATIC FLAG VALUE GENERATION STOP */ + /* AUTOMATIC FLAG VALUE GENERATION STOP 8 */ uint8_t flags; /* @@ -658,7 +659,7 @@ struct __wt_page { #define WT_PAGE_ROW_LEAF 7 /* Row-store leaf page */ uint8_t type; /* Page type */ -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_PAGE_BUILD_KEYS 0x01u /* Keys have been built in memory */ #define WT_PAGE_DISK_ALLOC 0x02u /* Disk image in allocated memory */ #define WT_PAGE_DISK_MAPPED 0x04u /* Disk image in mapped memory */ @@ -667,7 +668,7 @@ struct __wt_page { #define WT_PAGE_OVERFLOW_KEYS 0x20u /* Page has overflow keys */ #define WT_PAGE_SPLIT_INSERT 0x40u /* A leaf page was split for append */ #define WT_PAGE_UPDATE_IGNORE 0x80u /* Ignore updates on page discard */ - /* AUTOMATIC FLAG VALUE GENERATION STOP */ + /* AUTOMATIC FLAG VALUE GENERATION STOP 8 */ uint8_t flags_atomic; /* Atomic flags, use F_*_ATOMIC */ uint8_t unused[2]; /* Unused padding */ @@ -835,7 +836,7 @@ struct __wt_page_deleted { uint8_t previous_state; /* Previous state */ - WT_UPDATE **update_list; /* List of updates for abort */ + uint8_t committed; /* Committed */ }; /* @@ -873,11 +874,11 @@ struct __wt_ref { * miss one). If we run out of bits in the flags field, remove the internal flag and rewrite tests * depending on it to be "!leaf" instead. */ -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_REF_FLAG_INTERNAL 0x1u /* Page is an internal page */ #define WT_REF_FLAG_LEAF 0x2u /* Page is a leaf page */ #define WT_REF_FLAG_READING 0x4u /* Page is being read in */ - /* AUTOMATIC FLAG VALUE GENERATION STOP */ + /* AUTOMATIC FLAG VALUE GENERATION STOP 8 */ uint8_t flags; #define WT_REF_DISK 0 /* Page is on disk */ @@ -906,7 +907,21 @@ struct __wt_ref { #undef ref_ikey #define ref_ikey key.ikey - WT_PAGE_DELETED *page_del; /* Deleted page information */ + /* + * Fast-truncate information. When a WT_REF is included in a fast-truncate operation, WT_REF.del + * is allocated and initialized. If the page must be instantiated before the truncate becomes + * globally visible, WT_UPDATE structures are created for the page entries, the transaction + * information from WT_REF.del is migrated to those WT_UPDATE structures, and the WT_REF.del + * field is freed and replaced by the WT_REF.update array (needed for subsequent transaction + * commit/abort). Doing anything other than testing if WT_REF.del/update is non-NULL (which + * eviction does), requires the WT_REF be locked. If the locked WT_REF's previous state was + * WT_REF_DELETED, WT_REF.del is valid, if the WT_REF's previous state was an in-memory state, + * then WT_REF.update is valid. + */ + union { + WT_PAGE_DELETED *del; /* Page not instantiated, page-deleted structure */ + WT_UPDATE **update; /* Page instantiated, update list for subsequent commit/abort */ + } ft_info; /* * In DIAGNOSTIC mode we overwrite the WT_REF on free to force failures. Don't clear the history in @@ -929,10 +944,10 @@ struct __wt_ref { (ref)->hist[(ref)->histoff].state = (uint16_t)(s); \ (ref)->histoff = ((ref)->histoff + 1) % WT_ELEMENTS((ref)->hist); \ } while (0) -#define WT_REF_SET_STATE(ref, s) \ - do { \ - WT_REF_SAVE_STATE(ref, s, __func__, __LINE__); \ - WT_PUBLISH((ref)->state, s); \ +#define WT_REF_SET_STATE(ref, s) \ + do { \ + WT_REF_SAVE_STATE(ref, s, __PRETTY_FUNCTION__, __LINE__); \ + WT_PUBLISH((ref)->state, s); \ } while (0) #else #define WT_REF_SET_STATE(ref, s) WT_PUBLISH((ref)->state, s) @@ -951,7 +966,7 @@ struct __wt_ref { /* A macro wrapper allowing us to remember the callers code location */ #define WT_REF_CAS_STATE(session, ref, old_state, new_state) \ - __wt_ref_cas_state_int(session, ref, old_state, new_state, __func__, __LINE__) + __wt_ref_cas_state_int(session, ref, old_state, new_state, __PRETTY_FUNCTION__, __LINE__) #define WT_REF_LOCK(session, ref, previous_statep) \ do { \ @@ -1116,7 +1131,7 @@ struct __wt_update { */ volatile uint8_t prepare_state; /* prepare state */ -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_UPDATE_DS 0x01u /* Update has been written to the data store. */ #define WT_UPDATE_FIXED_HS 0x02u /* Update that fixed the history store. */ #define WT_UPDATE_HS 0x04u /* Update has been written to history store. */ @@ -1124,7 +1139,7 @@ struct __wt_update { #define WT_UPDATE_RESTORED_FAST_TRUNCATE 0x10u /* Fast truncate instantiation */ #define WT_UPDATE_RESTORED_FROM_DS 0x20u /* Update restored from data store. */ #define WT_UPDATE_RESTORED_FROM_HS 0x40u /* Update restored from history store. */ - /* AUTOMATIC FLAG VALUE GENERATION STOP */ + /* AUTOMATIC FLAG VALUE GENERATION STOP 8 */ uint8_t flags; /* @@ -1182,7 +1197,7 @@ struct __wt_update_value { } while (0) /* - * WT_MAX_MODIFY_UPDATE, WT_MODIFY_VECTOR_STACK_SIZE + * WT_MODIFY_UPDATE_MIN/MAX, WT_MODIFY_VECTOR_STACK_SIZE * Limit update chains value to avoid penalizing reads and permit truncation. Having a smaller * value will penalize the cases when history has to be maintained, resulting in multiplying cache * pressure. @@ -1191,8 +1206,9 @@ struct __wt_update_value { * modifications in an update list. We use small vectors of modify updates in a couple of places to * avoid heap allocation, add a few additional slots to that array. */ -#define WT_MAX_MODIFY_UPDATE 10 -#define WT_UPDATE_VECTOR_STACK_SIZE 20 +#define WT_MODIFY_UPDATE_MIN 10 /* Update count before we bother checking anything else */ +#define WT_MODIFY_UPDATE_MAX 200 /* Update count hard limit */ +#define WT_UPDATE_VECTOR_STACK_SIZE (WT_MODIFY_UPDATE_MIN + 10) /* * WT_UPDATE_VECTOR -- diff --git a/src/third_party/wiredtiger/src/include/btree.h b/src/third_party/wiredtiger/src/include/btree.h index 6f3a6e086b3..72fe0ec5594 100644 --- a/src/third_party/wiredtiger/src/include/btree.h +++ b/src/third_party/wiredtiger/src/include/btree.h @@ -78,9 +78,10 @@ typedef enum __wt_btree_sync { } WT_BTREE_SYNC; typedef enum { - CKSUM_ON = 1, /* On */ - CKSUM_OFF = 2, /* Off */ - CKSUM_UNCOMPRESSED = 3 /* Uncompressed blocks only */ + CKSUM_ON = 1, /* On */ + CKSUM_OFF = 2, /* Off */ + CKSUM_UNCOMPRESSED = 3, /* Uncompressed blocks only */ + CKSUM_UNENCRYPTED = 4 /* Unencrypted blocks only */ } WT_BTREE_CHECKSUM; typedef enum { /* Start position for eviction walk */ @@ -257,10 +258,8 @@ struct __wt_btree { /* * Flag values up to 0xfff are reserved for WT_DHANDLE_XXX. See comment with dhandle flags for an * explanation. - * - * We don't automatically generate these flag values for this reason; there's no way to start at an - * offset. */ +/* AUTOMATIC FLAG VALUE GENERATION START 12 */ #define WT_BTREE_ALTER 0x0001000u /* Handle is for alter */ #define WT_BTREE_BULK 0x0002000u /* Bulk-load handle */ #define WT_BTREE_CLOSED 0x0004000u /* Handle closed */ @@ -274,6 +273,7 @@ struct __wt_btree { #define WT_BTREE_SKIP_CKPT 0x0400000u /* Handle skipped checkpoint */ #define WT_BTREE_UPGRADE 0x0800000u /* Handle is for upgrade */ #define WT_BTREE_VERIFY 0x1000000u /* Handle is for verify */ + /* AUTOMATIC FLAG VALUE GENERATION STOP 32 */ uint32_t flags; }; diff --git a/src/third_party/wiredtiger/src/include/btree_inline.h b/src/third_party/wiredtiger/src/include/btree_inline.h index 9c0e0ce784e..6e2add93605 100644 --- a/src/third_party/wiredtiger/src/include/btree_inline.h +++ b/src/third_party/wiredtiger/src/include/btree_inline.h @@ -1464,7 +1464,9 @@ __wt_page_del_active(WT_SESSION_IMPL *session, WT_REF *ref, bool visible_all) WT_PAGE_DELETED *page_del; uint8_t prepare_state; - if ((page_del = ref->page_del) == NULL) + WT_ASSERT(session, ref->state == WT_REF_LOCKED); + + if ((page_del = ref->ft_info.del) == NULL) return (false); if (page_del->txnid == WT_TXN_ABORTED) return (false); @@ -1651,15 +1653,20 @@ __wt_page_can_evict(WT_SESSION_IMPL *session, WT_REF *ref, bool *inmem_splitp) page = ref->page; mod = page->modify; - /* A truncated page can't be evicted until the truncate completes. */ - if (__wt_page_del_active(session, ref, true)) - return (false); - - /* Otherwise, never modified pages can always be evicted. */ + /* Never modified pages can always be evicted. */ if (mod == NULL) return (true); /* + * If a fast-truncate page is subsequently instantiated, it can become an eviction candidate. If + * the fast-truncate itself has not resolved when the page is instantiated, a list of updates is + * created, which will be discarded as part of transaction resolution. Don't attempt to evict a + * fast-truncate page until any update list has been removed. + */ + if (ref->ft_info.update != NULL) + return (false); + + /* * We can't split or evict multiblock row-store pages where the parent's key for the page is an * overflow item, because the split into the parent frees the backing blocks for any * no-longer-used overflow keys, which will corrupt the checkpoint's block management. @@ -1693,15 +1700,18 @@ __wt_page_can_evict(WT_SESSION_IMPL *session, WT_REF *ref, bool *inmem_splitp) } /* + * Check we are not evicting an accessible internal page with an active split generation. + * * If a split created new internal pages, those newly created internal pages cannot be evicted * until all threads are known to have exited the original parent page's index, because evicting * an internal page discards its WT_REF array, and a thread traversing the original parent page * index might see a freed WT_REF. * - * One special case where we know this is safe is if the handle is locked exclusive (e.g., when - * the whole tree is being evicted). In that case, no readers can be looking at an old index. + * One special case where we know this is safe is if the handle is dead or locked exclusively, + * that is, no readers can be looking at an old index. */ - if (F_ISSET(ref, WT_REF_FLAG_INTERNAL) && !F_ISSET(session->dhandle, WT_DHANDLE_EXCLUSIVE) && + if (F_ISSET(ref, WT_REF_FLAG_INTERNAL) && + !F_ISSET(session->dhandle, WT_DHANDLE_DEAD | WT_DHANDLE_EXCLUSIVE) && __wt_gen_active(session, WT_GEN_SPLIT, page->pg_intl_split_gen)) return (false); @@ -2008,3 +2018,56 @@ __wt_bt_col_var_cursor_walk_txn_read(WT_SESSION_IMPL *session, WT_CURSOR_BTREE * cbt->cip_saved = cip; return (0); } + +/* + * __wt_btcur_skip_page -- + * Return if the cursor is pointing to a page with deleted records and can be skipped for cursor + * traversal. + */ +static inline bool +__wt_btcur_skip_page(WT_CURSOR_BTREE *cbt) +{ + WT_ADDR_COPY addr; + WT_PAGE *page; + WT_REF *ref; + WT_SESSION_IMPL *session; + uint8_t previous_state; + bool can_skip; + + session = CUR2S(cbt); + ref = cbt->ref; + page = cbt->ref == NULL ? NULL : cbt->ref->page; + + if (page == NULL) + return false; + + previous_state = ref->state; + can_skip = false; + + /* + * Determine if all records on the page have been deleted and all the tombstones are visible to + * our transaction. If so, we can avoid reading the records on the page and move to the next + * page. We base this decision on the aggregate stop point added to the page during the last + * reconciliation. We can skip this test if the page has been modified since it was reconciled + * or the underlying cursor is configured to ignore tombstones. + * + * We are making these decisions while holding a lock for the page as checkpoint or eviction can + * make changes to the data structures (i.e., aggregate timestamps) we are reading. + */ + if (session->txn->isolation == WT_ISO_SNAPSHOT && !__wt_page_is_modified(page) && + !F_ISSET(&cbt->iface, WT_CURSTD_IGNORE_TOMBSTONE) && previous_state == WT_REF_MEM) { + + /* We only try to lock the page once. */ + if (!WT_REF_CAS_STATE(session, ref, previous_state, WT_REF_LOCKED)) + return false; + + if (__wt_ref_addr_copy(session, ref, &addr) && + __wt_txn_visible(session, addr.ta.newest_stop_txn, addr.ta.newest_stop_ts) && + __wt_txn_visible(session, addr.ta.newest_stop_txn, addr.ta.newest_stop_durable_ts)) + can_skip = true; + + WT_REF_SET_STATE(ref, previous_state); + } + + return (can_skip); +} diff --git a/src/third_party/wiredtiger/src/include/cache.h b/src/third_party/wiredtiger/src/include/cache.h index 61f1e9f6f1b..2bdadc9522f 100644 --- a/src/third_party/wiredtiger/src/include/cache.h +++ b/src/third_party/wiredtiger/src/include/cache.h @@ -220,13 +220,13 @@ struct __wt_cache { /* * Flags. */ -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_CACHE_POOL_MANAGER 0x1u /* The active cache pool manager */ #define WT_CACHE_POOL_RUN 0x2u /* Cache pool thread running */ - /* AUTOMATIC FLAG VALUE GENERATION STOP */ + /* AUTOMATIC FLAG VALUE GENERATION STOP 32 */ uint32_t pool_flags; /* Cache pool flags */ -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_CACHE_EVICT_CLEAN 0x001u /* Evict clean pages */ #define WT_CACHE_EVICT_CLEAN_HARD 0x002u /* Clean % blocking app threads */ #define WT_CACHE_EVICT_DEBUG_MODE 0x004u /* Aggressive debugging mode */ @@ -237,7 +237,7 @@ struct __wt_cache { #define WT_CACHE_EVICT_UPDATES 0x080u /* Evict pages with updates */ #define WT_CACHE_EVICT_UPDATES_HARD 0x100u /* Update % blocking app threads */ #define WT_CACHE_EVICT_URGENT 0x200u /* Pages are in the urgent queue */ -/* AUTOMATIC FLAG VALUE GENERATION STOP */ +/* AUTOMATIC FLAG VALUE GENERATION STOP 32 */ #define WT_CACHE_EVICT_ALL (WT_CACHE_EVICT_CLEAN | WT_CACHE_EVICT_DIRTY | WT_CACHE_EVICT_UPDATES) #define WT_CACHE_EVICT_HARD \ (WT_CACHE_EVICT_CLEAN_HARD | WT_CACHE_EVICT_DIRTY_HARD | WT_CACHE_EVICT_UPDATES_HARD) @@ -246,7 +246,6 @@ struct __wt_cache { #define WT_WITH_PASS_LOCK(session, op) \ do { \ - WT_ASSERT(session, !FLD_ISSET(session->lock_flags, WT_SESSION_LOCKED_PASS)); \ WT_WITH_LOCK_WAIT(session, &cache->evict_pass_lock, WT_SESSION_LOCKED_PASS, op); \ } while (0) @@ -268,9 +267,9 @@ struct __wt_cache_pool { uint8_t pool_managed; /* Cache pool has a manager thread */ -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_CACHE_POOL_ACTIVE 0x1u /* Cache pool is active */ - /* AUTOMATIC FLAG VALUE GENERATION STOP */ + /* AUTOMATIC FLAG VALUE GENERATION STOP 8 */ uint8_t flags; }; @@ -281,8 +280,8 @@ struct __wt_cache_pool { #define WT_IS_HS(dh) F_ISSET(dh, WT_DHANDLE_HS) /* Flags used with __wt_evict */ -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_EVICT_CALL_CLOSING 0x1u /* Closing connection or tree */ #define WT_EVICT_CALL_NO_SPLIT 0x2u /* Splits not allowed */ #define WT_EVICT_CALL_URGENT 0x4u /* Urgent eviction */ -/* AUTOMATIC FLAG VALUE GENERATION STOP */ +/* AUTOMATIC FLAG VALUE GENERATION STOP 32 */ diff --git a/src/third_party/wiredtiger/src/include/cell.h b/src/third_party/wiredtiger/src/include/cell.h index 5b9db7a0173..b46f7918485 100644 --- a/src/third_party/wiredtiger/src/include/cell.h +++ b/src/third_party/wiredtiger/src/include/cell.h @@ -144,10 +144,10 @@ struct __wt_cell { uint8_t __chunk[1 + 1 + 1 + 7 * WT_INTPACK64_MAXSIZE + WT_INTPACK32_MAXSIZE]; }; -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_CELL_UNPACK_OVERFLOW 0x1u /* cell is an overflow */ #define WT_CELL_UNPACK_TIME_WINDOW_CLEARED 0x2u /* time window cleared because of restart */ - /* AUTOMATIC FLAG VALUE GENERATION STOP */ +/* AUTOMATIC FLAG VALUE GENERATION STOP 8 */ /* * We have two "unpacked cell" structures: one holding holds unpacked cells from internal nodes diff --git a/src/third_party/wiredtiger/src/include/cell_inline.h b/src/third_party/wiredtiger/src/include/cell_inline.h index d3e0f85359d..2d91b9a8ee6 100644 --- a/src/third_party/wiredtiger/src/include/cell_inline.h +++ b/src/third_party/wiredtiger/src/include/cell_inline.h @@ -941,38 +941,13 @@ done: } /* - * __cell_unpack_window_cleanup -- - * Clean up cells loaded from a previous run. + * __cell_addr_window_cleanup -- + * Clean up addr cells loaded from a previous run. */ static inline void -__cell_unpack_window_cleanup(WT_SESSION_IMPL *session, const WT_PAGE_HEADER *dsk, - WT_CELL_UNPACK_ADDR *unpack_addr, WT_CELL_UNPACK_KV *unpack_kv) +__cell_addr_window_cleanup(WT_SESSION_IMPL *session, WT_CELL_UNPACK_ADDR *unpack_addr) { WT_TIME_AGGREGATE *ta; - WT_TIME_WINDOW *tw; - - /* - * If the page came from a previous run, reset the transaction ids to "none" and timestamps to 0 - * as appropriate. Transaction ids shouldn't persist between runs so these are always set to - * "none". Timestamps should persist between runs however, the absence of a timestamp (in the - * case of a non-timestamped write) should default to WT_TS_NONE rather than "max" as usual. - * - * Note that it is still necessary to unpack each value above even if we end up overwriting them - * since values in a cell need to be unpacked sequentially. - * - * This is how the stop time point should be interpreted for each type of delete: - * - - * Current startup Previous startup - * Timestamp delete txnid=x, ts=y, txnid=0, ts=y, - * durable_ts=z durable_ts=z - * Non-timestamp delete txnid=x, ts=NONE, txnid=0, ts=NONE, - * durable_ts=NONE durable_ts=NONE - * No delete txnid=MAX, ts=MAX, txnid=MAX, ts=MAX, - * durable_ts=NONE durable_ts=NONE - */ - WT_ASSERT(session, dsk->write_gen != 0); - if (dsk->write_gen > S2BT(session)->base_write_gen) - return; /* Tell reconciliation we cleared the transaction ids and the cell needs to be rebuilt. */ if (unpack_addr != NULL) { @@ -998,6 +973,17 @@ __cell_unpack_window_cleanup(WT_SESSION_IMPL *session, const WT_PAGE_HEADER *dsk } else WT_ASSERT(session, ta->newest_stop_ts == WT_TS_MAX); } +} + +/* + * __cell_kv_window_cleanup -- + * Clean up kv cells loaded from a previous run. + */ +static inline void +__cell_kv_window_cleanup(WT_SESSION_IMPL *session, WT_CELL_UNPACK_KV *unpack_kv) +{ + WT_TIME_WINDOW *tw; + if (unpack_kv != NULL) { tw = &unpack_kv->tw; if (tw->start_txn != WT_TXN_NONE) { @@ -1024,6 +1010,59 @@ __cell_unpack_window_cleanup(WT_SESSION_IMPL *session, const WT_PAGE_HEADER *dsk } /* + * __cell_unpack_window_cleanup -- + * Clean up cells loaded from a previous run. + */ +static inline void +__cell_unpack_window_cleanup(WT_SESSION_IMPL *session, const WT_PAGE_HEADER *dsk, + WT_CELL_UNPACK_ADDR *unpack_addr, WT_CELL_UNPACK_KV *unpack_kv) +{ + /* + * If the page came from a previous run, reset the transaction ids to "none" and timestamps to 0 + * as appropriate. Transaction ids shouldn't persist between runs so these are always set to + * "none". Timestamps should persist between runs however, the absence of a timestamp (in the + * case of a non-timestamped write) should default to WT_TS_NONE rather than "max" as usual. + * + * Note that it is still necessary to unpack each value above even if we end up overwriting them + * since values in a cell need to be unpacked sequentially. + * + * This is how the stop time point should be interpreted for each type of delete: + * - + * Current startup Previous startup + * Timestamp delete txnid=x, ts=y, txnid=0, ts=y, + * durable_ts=z durable_ts=z + * Non-timestamp delete txnid=x, ts=NONE, txnid=0, ts=NONE, + * durable_ts=NONE durable_ts=NONE + * No delete txnid=MAX, ts=MAX, txnid=MAX, ts=MAX, + * durable_ts=NONE durable_ts=NONE + */ + WT_ASSERT(session, dsk->write_gen != 0); + if (dsk->write_gen > S2BT(session)->base_write_gen) + return; + + __cell_addr_window_cleanup(session, unpack_addr); + __cell_kv_window_cleanup(session, unpack_kv); +} + +/* + * __cell_pack_kv_window_cleanup -- + * Clean up cells loaded from a previous run while writing to disk. + */ +static inline void +__cell_pack_kv_window_cleanup( + WT_SESSION_IMPL *session, const WT_PAGE_HEADER *dsk, WT_CELL_UNPACK_KV *unpack_kv) +{ + /* + * If the page came from a previous run, reset the transaction ids to "none" and timestamps to 0 + * as appropriate when the cell information is used for packing the new cell. + */ + if (F_ISSET(S2C(session), WT_CONN_RECOVERING) && + dsk->write_gen > S2BT(session)->base_write_gen && + dsk->write_gen < S2BT(session)->run_write_gen) + __cell_kv_window_cleanup(session, unpack_kv); +} + +/* * __wt_cell_unpack_addr -- * Unpack an address WT_CELL into a structure. */ diff --git a/src/third_party/wiredtiger/src/include/connection.h b/src/third_party/wiredtiger/src/include/connection.h index a087086b200..68a03cad84b 100644 --- a/src/third_party/wiredtiger/src/include/connection.h +++ b/src/third_party/wiredtiger/src/include/connection.h @@ -48,9 +48,9 @@ struct __wt_bucket_storage { TAILQ_ENTRY(__wt_bucket_storage) hashq; TAILQ_ENTRY(__wt_bucket_storage) q; -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_BUCKET_FREE 0x1u - /* AUTOMATIC FLAG VALUE GENERATION STOP */ + /* AUTOMATIC FLAG VALUE GENERATION STOP 32 */ uint32_t flags; }; @@ -435,7 +435,7 @@ struct __wt_connection_impl { uint32_t tiered_threads_max; /* Max tiered threads */ uint32_t tiered_threads_min; /* Min tiered threads */ -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_CONN_LOG_ARCHIVE 0x001u /* Archive is enabled */ #define WT_CONN_LOG_CONFIG_ENABLED 0x002u /* Logging is configured */ #define WT_CONN_LOG_DEBUG_MODE 0x004u /* Debug-mode logging enabled */ @@ -448,7 +448,7 @@ struct __wt_connection_impl { #define WT_CONN_LOG_RECOVER_ERR 0x200u /* Error if recovery required */ #define WT_CONN_LOG_RECOVER_FAILED 0x400u /* Recovery failed */ #define WT_CONN_LOG_ZERO_FILL 0x800u /* Manually zero files */ - /* AUTOMATIC FLAG VALUE GENERATION STOP */ + /* AUTOMATIC FLAG VALUE GENERATION STOP 32 */ uint32_t log_flags; /* Global logging configuration */ WT_CONDVAR *log_cond; /* Log server wait mutex */ WT_SESSION_IMPL *log_session; /* Log server session */ @@ -488,8 +488,6 @@ struct __wt_connection_impl { uint64_t sweep_interval; /* Handle sweep interval */ uint64_t sweep_handles_min; /* Handle sweep minimum open */ - uint32_t stable_rollback_maxfile; - /* Locked: collator list */ TAILQ_HEAD(__wt_coll_qh, __wt_named_collator) collqh; @@ -523,11 +521,11 @@ struct __wt_connection_impl { wt_off_t data_extend_len; /* file_extend data length */ wt_off_t log_extend_len; /* file_extend log length */ -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_DIRECT_IO_CHECKPOINT 0x1u /* Checkpoints */ #define WT_DIRECT_IO_DATA 0x2u /* Data files */ #define WT_DIRECT_IO_LOG 0x4u /* Log files */ - /* AUTOMATIC FLAG VALUE GENERATION STOP */ + /* AUTOMATIC FLAG VALUE GENERATION STOP 64 */ uint64_t direct_io; /* O_DIRECT, FILE_FLAG_NO_BUFFERING */ uint64_t write_through; /* FILE_FLAG_WRITE_THROUGH */ @@ -540,17 +538,17 @@ struct __wt_connection_impl { uint32_t debug_ckpt_cnt; /* Checkpoint retention number. */ uint32_t debug_log_cnt; /* Log file retention count */ -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_CONN_DEBUG_CKPT_RETAIN 0x01u #define WT_CONN_DEBUG_CORRUPTION_ABORT 0x02u #define WT_CONN_DEBUG_CURSOR_COPY 0x04u #define WT_CONN_DEBUG_REALLOC_EXACT 0x08u #define WT_CONN_DEBUG_SLOW_CKPT 0x10u #define WT_CONN_DEBUG_UPDATE_RESTORE_EVICT 0x20u - /* AUTOMATIC FLAG VALUE GENERATION STOP */ + /* AUTOMATIC FLAG VALUE GENERATION STOP 64 */ uint64_t debug_flags; -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_VERB_API 0x0000000001u #define WT_VERB_BACKUP 0x0000000002u #define WT_VERB_BLOCK 0x0000000004u @@ -589,13 +587,13 @@ struct __wt_connection_impl { #define WT_VERB_VERIFY 0x0800000000u #define WT_VERB_VERSION 0x1000000000u #define WT_VERB_WRITE 0x2000000000u - /* AUTOMATIC FLAG VALUE GENERATION STOP */ + /* AUTOMATIC FLAG VALUE GENERATION STOP 64 */ uint64_t verbose; /* * Variable with flags for which subsystems the diagnostic stress timing delays have been requested. */ -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_TIMING_STRESS_AGGRESSIVE_SWEEP 0x0001u #define WT_TIMING_STRESS_BACKUP_RENAME 0x0002u #define WT_TIMING_STRESS_CHECKPOINT_SLOW 0x0004u @@ -611,7 +609,7 @@ struct __wt_connection_impl { #define WT_TIMING_STRESS_SPLIT_6 0x1000u #define WT_TIMING_STRESS_SPLIT_7 0x2000u #define WT_TIMING_STRESS_SPLIT_8 0x4000u - /* AUTOMATIC FLAG VALUE GENERATION STOP */ + /* AUTOMATIC FLAG VALUE GENERATION STOP 64 */ uint64_t timing_stress_flags; #define WT_STDERR(s) (&S2C(s)->wt_stderr) @@ -626,7 +624,7 @@ struct __wt_connection_impl { /* * Server subsystem flags. */ -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_CONN_SERVER_CAPACITY 0x01u #define WT_CONN_SERVER_CHECKPOINT 0x02u #define WT_CONN_SERVER_LOG 0x04u @@ -635,10 +633,10 @@ struct __wt_connection_impl { #define WT_CONN_SERVER_SWEEP 0x20u #define WT_CONN_SERVER_TIERED 0x40u #define WT_CONN_SERVER_TIERED_MGR 0x80u - /* AUTOMATIC FLAG VALUE GENERATION STOP */ + /* AUTOMATIC FLAG VALUE GENERATION STOP 32 */ uint32_t server_flags; -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_CONN_CACHE_CURSORS 0x000001u #define WT_CONN_CACHE_POOL 0x000002u #define WT_CONN_CKPT_GATHER 0x000004u @@ -662,6 +660,6 @@ struct __wt_connection_impl { #define WT_CONN_RECOVERING 0x100000u #define WT_CONN_SALVAGE 0x200000u #define WT_CONN_WAS_BACKUP 0x400000u - /* AUTOMATIC FLAG VALUE GENERATION STOP */ + /* AUTOMATIC FLAG VALUE GENERATION STOP 32 */ uint32_t flags; }; diff --git a/src/third_party/wiredtiger/src/include/cursor.h b/src/third_party/wiredtiger/src/include/cursor.h index ffe960de07f..9425292ad8a 100644 --- a/src/third_party/wiredtiger/src/include/cursor.h +++ b/src/third_party/wiredtiger/src/include/cursor.h @@ -60,7 +60,7 @@ struct __wt_cursor_backup { uint64_t bit_offset; /* Current offset */ uint64_t granularity; /* Length, transfer size */ -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_CURBACKUP_CKPT_FAKE 0x001u /* Object has fake checkpoint */ #define WT_CURBACKUP_CONSOLIDATE 0x002u /* Consolidate returned info on this object */ #define WT_CURBACKUP_DUP 0x004u /* Duplicated backup cursor */ @@ -72,7 +72,7 @@ struct __wt_cursor_backup { #define WT_CURBACKUP_LOCKER 0x100u /* Hot-backup started */ #define WT_CURBACKUP_QUERYID 0x200u /* Backup cursor for incremental ids */ #define WT_CURBACKUP_RENAME 0x400u /* Object had a rename */ - /* AUTOMATIC FLAG VALUE GENERATION STOP */ + /* AUTOMATIC FLAG VALUE GENERATION STOP 32 */ uint32_t flags; }; @@ -212,7 +212,7 @@ struct __wt_cursor_btree { uint64_t lastrecno; #endif -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_CBT_ACTIVE 0x001u /* Active in the tree */ #define WT_CBT_ITERATE_APPEND 0x002u /* Col-store: iterating append list */ #define WT_CBT_ITERATE_NEXT 0x004u /* Next iteration configuration */ @@ -224,7 +224,7 @@ struct __wt_cursor_btree { #define WT_CBT_READ_ONCE 0x100u /* Page in with WT_READ_WONT_NEED */ #define WT_CBT_SEARCH_SMALLEST 0x200u /* Row-store: small-key insert list */ #define WT_CBT_VAR_ONPAGE_MATCH 0x400u /* Var-store: on-page recno match */ - /* AUTOMATIC FLAG VALUE GENERATION STOP */ + /* AUTOMATIC FLAG VALUE GENERATION STOP 32 */ #define WT_CBT_POSITION_MASK /* Flags associated with position */ \ (WT_CBT_ITERATE_APPEND | WT_CBT_ITERATE_NEXT | WT_CBT_ITERATE_PREV | \ @@ -292,12 +292,12 @@ struct __wt_cursor_hs { bool insert_success; - /* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_HS_CUR_BTREE_ID_SET 0x1u #define WT_HS_CUR_COUNTER_SET 0x2u #define WT_HS_CUR_KEY_SET 0x4u #define WT_HS_CUR_TS_SET 0x8u - /* AUTOMATIC FLAG VALUE GENERATION STOP */ + /* AUTOMATIC FLAG VALUE GENERATION STOP 8 */ uint8_t flags; }; @@ -357,12 +357,12 @@ struct __wt_cursor_join_endpoint { uint8_t recno_buf[10]; /* holds packed recno */ WT_CURSOR *cursor; -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_CURJOIN_END_EQ 0x1u /* include values == cursor */ #define WT_CURJOIN_END_GT 0x2u /* include values > cursor */ #define WT_CURJOIN_END_LT 0x4u /* include values < cursor */ #define WT_CURJOIN_END_OWN_CURSOR 0x8u /* must close cursor */ -/* AUTOMATIC FLAG VALUE GENERATION STOP */ +/* AUTOMATIC FLAG VALUE GENERATION STOP 8 */ #define WT_CURJOIN_END_GE (WT_CURJOIN_END_GT | WT_CURJOIN_END_EQ) #define WT_CURJOIN_END_LE (WT_CURJOIN_END_LT | WT_CURJOIN_END_EQ) uint8_t flags; /* range for this endpoint */ @@ -386,12 +386,12 @@ struct __wt_cursor_join_entry { uint32_t bloom_hash_count; /* hash functions in bloom */ uint64_t count; /* approx number of matches */ -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_CURJOIN_ENTRY_BLOOM 0x1u /* use a bloom filter */ #define WT_CURJOIN_ENTRY_DISJUNCTION 0x2u /* endpoints are or-ed */ #define WT_CURJOIN_ENTRY_FALSE_POSITIVES 0x4u /* don't filter false pos */ #define WT_CURJOIN_ENTRY_OWN_BLOOM 0x8u /* this entry owns the bloom */ - /* AUTOMATIC FLAG VALUE GENERATION STOP */ + /* AUTOMATIC FLAG VALUE GENERATION STOP 8 */ uint8_t flags; WT_CURSOR_JOIN_ENDPOINT *ends; /* reference endpoints */ @@ -414,11 +414,11 @@ struct __wt_cursor_join { u_int entries_next; uint8_t recno_buf[10]; /* holds packed recno */ -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_CURJOIN_DISJUNCTION 0x1u /* Entries are or-ed */ #define WT_CURJOIN_ERROR 0x2u /* Error in initialization */ #define WT_CURJOIN_INITIALIZED 0x4u /* Successful initialization */ - /* AUTOMATIC FLAG VALUE GENERATION STOP */ + /* AUTOMATIC FLAG VALUE GENERATION STOP 8 */ uint8_t flags; }; @@ -443,9 +443,9 @@ struct __wt_cursor_log { uint32_t rectype; /* Record type */ uint64_t txnid; /* Record txnid */ -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_CURLOG_ARCHIVE_LOCK 0x1u /* Archive lock held */ - /* AUTOMATIC FLAG VALUE GENERATION STOP */ + /* AUTOMATIC FLAG VALUE GENERATION STOP 8 */ uint8_t flags; }; @@ -455,11 +455,11 @@ struct __wt_cursor_metadata { WT_CURSOR *file_cursor; /* Queries of regular metadata */ WT_CURSOR *create_cursor; /* Extra cursor for create option */ -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_MDC_CREATEONLY 0x1u #define WT_MDC_ONMETADATA 0x2u #define WT_MDC_POSITIONED 0x4u - /* AUTOMATIC FLAG VALUE GENERATION STOP */ + /* AUTOMATIC FLAG VALUE GENERATION STOP 8 */ uint8_t flags; }; diff --git a/src/third_party/wiredtiger/src/include/dhandle.h b/src/third_party/wiredtiger/src/include/dhandle.h index c17970f4760..133ab688544 100644 --- a/src/third_party/wiredtiger/src/include/dhandle.h +++ b/src/third_party/wiredtiger/src/include/dhandle.h @@ -113,12 +113,8 @@ struct __wt_data_handle { /* * Flags values over 0xfff are reserved for WT_BTREE_*. This lets us combine the dhandle and btree * flags when we need, for example, to pass both sets in a function call. - * - * To help avoid accidental overrun of the flag values, we add a special flag value that should - * always be the last and highest. We use this value to assert that the dhandle flags haven't run - * into the space reserved for btree flags. */ -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_DHANDLE_DEAD 0x001u /* Dead, awaiting discard */ #define WT_DHANDLE_DISCARD 0x002u /* Close on release */ #define WT_DHANDLE_DISCARD_KILL 0x004u /* Mark dead on release */ @@ -128,15 +124,10 @@ struct __wt_data_handle { #define WT_DHANDLE_IS_METADATA 0x040u /* Metadata handle */ #define WT_DHANDLE_LOCK_ONLY 0x080u /* Handle only used as a lock */ #define WT_DHANDLE_OPEN 0x100u /* Handle is open */ -#define WT_DHANDLE_ZZZ_ENDFLAG 0x200u /* One past highest flag value */ - /* AUTOMATIC FLAG VALUE GENERATION STOP */ + /* AUTOMATIC FLAG VALUE GENERATION STOP 12 */ uint32_t flags; -#define WT_DHANDLE_MAX_FLAG 0x1000u /* Used to ensure we don't overflow legal flag values */ -#if WT_DHANDLE_ZZZ_ENDFLAG > WT_DHANDLE_MAX_FLAG -#error "Too many dhandle flags" -#endif -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_DHANDLE_ASSERT_TS_READ_ALWAYS 0x001u /* Assert read always checking. */ #define WT_DHANDLE_ASSERT_TS_READ_NEVER 0x002u /* Assert read never checking. */ #define WT_DHANDLE_ASSERT_TS_WRITE 0x004u /* Assert write checking. */ @@ -146,6 +137,6 @@ struct __wt_data_handle { #define WT_DHANDLE_TS_NEVER 0x040u /* Handle never using timestamps checking. */ #define WT_DHANDLE_TS_ORDERED 0x080u /* Handle using ordered timestamps checking. */ #define WT_DHANDLE_VERB_TS_WRITE 0x100u /* Handle verbose logging for timestamps usage. */ - /* AUTOMATIC FLAG VALUE GENERATION STOP */ + /* AUTOMATIC FLAG VALUE GENERATION STOP 32 */ uint32_t ts_flags; }; diff --git a/src/third_party/wiredtiger/src/include/error.h b/src/third_party/wiredtiger/src/include/error.h index dde7bcf9710..ef3019405da 100644 --- a/src/third_party/wiredtiger/src/include/error.h +++ b/src/third_party/wiredtiger/src/include/error.h @@ -20,11 +20,13 @@ #define WT_DIAGNOSTIC_YIELD #endif -#define __wt_err(session, error, ...) __wt_err_func(session, error, __func__, __LINE__, __VA_ARGS__) -#define __wt_errx(session, ...) __wt_errx_func(session, __func__, __LINE__, __VA_ARGS__) +#define __wt_err(session, error, ...) \ + __wt_err_func(session, error, __PRETTY_FUNCTION__, __LINE__, __VA_ARGS__) +#define __wt_errx(session, ...) __wt_errx_func(session, __PRETTY_FUNCTION__, __LINE__, __VA_ARGS__) #define __wt_panic(session, error, ...) \ - __wt_panic_func(session, error, __func__, __LINE__, __VA_ARGS__) -#define __wt_set_return(session, error) __wt_set_return_func(session, __func__, __LINE__, error) + __wt_panic_func(session, error, __PRETTY_FUNCTION__, __LINE__, __VA_ARGS__) +#define __wt_set_return(session, error) \ + __wt_set_return_func(session, __PRETTY_FUNCTION__, __LINE__, error) /* Set "ret" and branch-to-err-label tests. */ #define WT_ERR(a) \ diff --git a/src/third_party/wiredtiger/src/include/extern.h b/src/third_party/wiredtiger/src/include/extern.h index 9d6499f31a6..38d8f72a052 100644 --- a/src/third_party/wiredtiger/src/include/extern.h +++ b/src/third_party/wiredtiger/src/include/extern.h @@ -204,8 +204,6 @@ extern int __wt_block_switch_object(WT_SESSION_IMPL *session, WT_BLOCK *block, u uint32_t flags) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_block_tiered_fh(WT_SESSION_IMPL *session, WT_BLOCK *block, uint32_t object_id, WT_FH **fhp) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); -extern int __wt_block_tiered_load(WT_SESSION_IMPL *session, WT_BLOCK *block, WT_BLOCK_CKPT *ci) - WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_block_truncate(WT_SESSION_IMPL *session, WT_BLOCK *block, wt_off_t len) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_block_unmap(WT_SESSION_IMPL *session, WT_BLOCK *block, void *mapped_region, @@ -381,8 +379,12 @@ extern int __wt_clsm_open_bulk(WT_CURSOR_LSM *clsm, const char *cfg[]) extern int __wt_clsm_request_switch(WT_CURSOR_LSM *clsm) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_col_modify(WT_CURSOR_BTREE *cbt, uint64_t recno, const WT_ITEM *value, - WT_UPDATE *upd_arg, u_int modify_type, bool exclusive) - WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); + WT_UPDATE *upd_arg, u_int modify_type, bool exclusive +#ifdef HAVE_DIAGNOSTIC + , + bool restore +#endif + ) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_col_search(WT_CURSOR_BTREE *cbt, uint64_t search_recno, WT_REF *leaf, bool leaf_safe, bool *leaf_foundp) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_collator_config(WT_SESSION_IMPL *session, const char *uri, WT_CONFIG_ITEM *cname, @@ -711,6 +713,8 @@ extern int __wt_ext_pack_str(WT_EXTENSION_API *wt_api, WT_PACK_STREAM *ps, const WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_ext_pack_uint(WT_EXTENSION_API *wt_api, WT_PACK_STREAM *ps, uint64_t u) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); +extern int __wt_ext_spin_init(WT_EXTENSION_API *wt_api, WT_EXTENSION_SPINLOCK *ext_spinlock, + const char *name) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_ext_struct_pack(WT_EXTENSION_API *wt_api, WT_SESSION *wt_session, void *buffer, size_t len, const char *fmt, ...) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_ext_struct_size(WT_EXTENSION_API *wt_api, WT_SESSION *wt_session, size_t *lenp, @@ -768,7 +772,7 @@ extern int __wt_hex_to_raw(WT_SESSION_IMPL *session, const char *from, WT_ITEM * extern int __wt_hs_config(WT_SESSION_IMPL *session, const char **cfg) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_hs_delete_key_from_ts(WT_SESSION_IMPL *session, WT_CURSOR *hs_cursor, - uint32_t btree_id, const WT_ITEM *key, wt_timestamp_t ts, bool reinsert) + uint32_t btree_id, const WT_ITEM *key, wt_timestamp_t ts, bool reinsert, bool checkpoint_running) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_hs_find_upd(WT_SESSION_IMPL *session, uint32_t btree_id, WT_ITEM *key, const char *value_format, uint64_t recno, WT_UPDATE_VALUE *upd_value, WT_ITEM *base_value_buf) @@ -776,7 +780,7 @@ extern int __wt_hs_find_upd(WT_SESSION_IMPL *session, uint32_t btree_id, WT_ITEM extern int __wt_hs_get_btree(WT_SESSION_IMPL *session, WT_BTREE **hs_btreep) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_hs_insert_updates(WT_SESSION_IMPL *session, WT_PAGE *page, WT_MULTI *multi, - bool *cache_write_hs) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); + bool *cache_write_hs, bool checkpoint_running) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_hs_modify(WT_CURSOR_BTREE *hs_cbt, WT_UPDATE *hs_upd) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_hs_open(WT_SESSION_IMPL *session, const char **cfg) @@ -1054,14 +1058,12 @@ extern int __wt_meta_ckptlist_get(WT_SESSION_IMPL *session, const char *fname, b extern int __wt_meta_ckptlist_get_from_config(WT_SESSION_IMPL *session, bool update, WT_CKPT **ckptbasep, size_t *allocatedp, const char *config) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); -extern int __wt_meta_ckptlist_set(WT_SESSION_IMPL *session, const char *fname, WT_CKPT *ckptbase, - WT_LSN *ckptlsn) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); +extern int __wt_meta_ckptlist_set(WT_SESSION_IMPL *session, WT_DATA_HANDLE *dhandle, + WT_CKPT *ckptbase, WT_LSN *ckptlsn) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_meta_ckptlist_to_meta(WT_SESSION_IMPL *session, WT_CKPT *ckptbase, WT_ITEM *buf) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_meta_ckptlist_update_config(WT_SESSION_IMPL *session, WT_CKPT *ckptbase, const char *oldcfg, char **newcfgp) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); -extern int __wt_meta_saved_ckptlist_get(WT_SESSION_IMPL *session, const char *fname, - WT_CKPT **ckptbasep) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_meta_sysinfo_set(WT_SESSION_IMPL *session) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_meta_track_checkpoint(WT_SESSION_IMPL *session) @@ -1241,6 +1243,8 @@ extern int __wt_reset_blkmod(WT_SESSION_IMPL *session, const char *orig_config, WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_rollback_to_stable(WT_SESSION_IMPL *session, const char *cfg[], bool no_ckpt) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); +extern int __wt_rollback_to_stable_one(WT_SESSION_IMPL *session, const char *uri, bool *skipp) + WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_row_ikey(WT_SESSION_IMPL *session, uint32_t cell_offset, const void *key, size_t size, WT_REF *ref) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_row_ikey_alloc(WT_SESSION_IMPL *session, uint32_t cell_offset, const void *key, @@ -1254,8 +1258,12 @@ extern int __wt_row_leaf_key_copy(WT_SESSION_IMPL *session, WT_PAGE *page, WT_RO extern int __wt_row_leaf_key_work(WT_SESSION_IMPL *session, WT_PAGE *page, WT_ROW *rip_arg, WT_ITEM *keyb, bool instantiate) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_row_modify(WT_CURSOR_BTREE *cbt, const WT_ITEM *key, const WT_ITEM *value, - WT_UPDATE *upd_arg, u_int modify_type, bool exclusive) - WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); + WT_UPDATE *upd_arg, u_int modify_type, bool exclusive +#ifdef HAVE_DIAGNOSTIC + , + bool restore +#endif + ) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_row_search(WT_CURSOR_BTREE *cbt, WT_ITEM *srch_key, bool insert, WT_REF *leaf, bool leaf_safe, bool *leaf_foundp) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_rts_page_skip(WT_SESSION_IMPL *session, WT_REF *ref, void *context, bool *skipp) @@ -1460,10 +1468,12 @@ extern int __wt_tier_flush(WT_SESSION_IMPL *session, WT_TIERED *tiered, uint32_t WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_tiered_bucket_config(WT_SESSION_IMPL *session, const char *cfg[], WT_BUCKET_STORAGE **bstoragep) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); -extern int __wt_tiered_close(WT_SESSION_IMPL *session, WT_TIERED *tiered) +extern int __wt_tiered_close(WT_SESSION_IMPL *session) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_tiered_conn_config(WT_SESSION_IMPL *session, const char **cfg, bool reconfig) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); +extern int __wt_tiered_discard(WT_SESSION_IMPL *session, WT_TIERED *tiered) + WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_tiered_name(WT_SESSION_IMPL *session, WT_DATA_HANDLE *dhandle, uint32_t id, uint32_t flags, const char **retp) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_tiered_open(WT_SESSION_IMPL *session, const char *cfg[]) @@ -1477,6 +1487,8 @@ extern int __wt_tiered_put_drop_shared(WT_SESSION_IMPL *session, WT_TIERED *tier WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_tiered_put_flush(WT_SESSION_IMPL *session, WT_TIERED *tiered) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); +extern int __wt_tiered_set_metadata(WT_SESSION_IMPL *session, WT_TIERED *tiered, WT_ITEM *buf) + WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_tiered_storage_create(WT_SESSION_IMPL *session, const char *cfg[]) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_tiered_storage_destroy(WT_SESSION_IMPL *session) @@ -1729,6 +1741,11 @@ extern void __wt_evict_priority_clear(WT_SESSION_IMPL *session); extern void __wt_evict_priority_set(WT_SESSION_IMPL *session, uint64_t v); extern void __wt_evict_server_wake(WT_SESSION_IMPL *session); extern void __wt_ext_scr_free(WT_EXTENSION_API *wt_api, WT_SESSION *wt_session, void *p); +extern void __wt_ext_spin_destroy(WT_EXTENSION_API *wt_api, WT_EXTENSION_SPINLOCK *ext_spinlock); +extern void __wt_ext_spin_lock( + WT_EXTENSION_API *wt_api, WT_SESSION *session, WT_EXTENSION_SPINLOCK *ext_spinlock); +extern void __wt_ext_spin_unlock( + WT_EXTENSION_API *wt_api, WT_SESSION *session, WT_EXTENSION_SPINLOCK *ext_spinlock); extern void __wt_fill_hex( const uint8_t *src, size_t src_max, uint8_t *dest, size_t dest_max, size_t *lenp); extern void __wt_free_int(WT_SESSION_IMPL *session, const void *p_arg) @@ -1867,6 +1884,8 @@ static inline WT_IKEY *__wt_ref_key_instantiated(WT_REF *ref) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); static inline WT_VISIBLE_TYPE __wt_txn_upd_visible_type(WT_SESSION_IMPL *session, WT_UPDATE *upd) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); +static inline bool __wt_btcur_skip_page(WT_CURSOR_BTREE *cbt) + WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); static inline bool __wt_btree_can_evict_dirty(WT_SESSION_IMPL *session) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); static inline bool __wt_btree_dominating_cache(WT_SESSION_IMPL *session, WT_BTREE *btree) diff --git a/src/third_party/wiredtiger/src/include/log.h b/src/third_party/wiredtiger/src/include/log.h index 4f23f98b463..132a1512b37 100644 --- a/src/third_party/wiredtiger/src/include/log.h +++ b/src/third_party/wiredtiger/src/include/log.h @@ -6,21 +6,21 @@ * See the file LICENSE for redistribution information. */ -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_LOGSCAN_FIRST 0x01u #define WT_LOGSCAN_FROM_CKP 0x02u #define WT_LOGSCAN_ONE 0x04u #define WT_LOGSCAN_RECOVER 0x08u #define WT_LOGSCAN_RECOVER_METADATA 0x10u -/* AUTOMATIC FLAG VALUE GENERATION STOP */ +/* AUTOMATIC FLAG VALUE GENERATION STOP 32 */ -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_LOG_BACKGROUND 0x01u #define WT_LOG_DSYNC 0x02u #define WT_LOG_FLUSH 0x04u #define WT_LOG_FSYNC 0x08u #define WT_LOG_SYNC_ENABLED 0x10u -/* AUTOMATIC FLAG VALUE GENERATION STOP */ +/* AUTOMATIC FLAG VALUE GENERATION STOP 32 */ #define WT_LOGOP_IGNORE 0x80000000 #define WT_LOGOP_IS_IGNORED(val) ((val)&WT_LOGOP_IGNORE) @@ -199,13 +199,13 @@ struct __wt_logslot { WT_FH *slot_fh; /* File handle for this group */ WT_ITEM slot_buf; /* Buffer for grouped writes */ -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_SLOT_CLOSEFH 0x01u /* Close old fh on release */ #define WT_SLOT_FLUSH 0x02u /* Wait for write */ #define WT_SLOT_SYNC 0x04u /* Needs sync on release */ #define WT_SLOT_SYNC_DIR 0x08u /* Directory sync on release */ #define WT_SLOT_SYNC_DIRTY 0x10u /* Sync system buffers on release */ - /* AUTOMATIC FLAG VALUE GENERATION STOP */ + /* AUTOMATIC FLAG VALUE GENERATION STOP 32 */ uint32_t flags; WT_CACHE_LINE_PAD_END }; @@ -225,11 +225,11 @@ struct __wt_myslot { wt_off_t end_offset; /* My end offset in buffer */ wt_off_t offset; /* Slot buffer offset */ -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_MYSLOT_CLOSE 0x1u /* This thread is closing the slot */ #define WT_MYSLOT_NEEDS_RELEASE 0x2u /* This thread is releasing the slot */ #define WT_MYSLOT_UNBUFFERED 0x4u /* Write directly */ - /* AUTOMATIC FLAG VALUE GENERATION STOP */ + /* AUTOMATIC FLAG VALUE GENERATION STOP 32 */ uint32_t flags; }; @@ -299,11 +299,11 @@ struct __wt_log { uint64_t write_calls; /* Calls to log_write */ #endif -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_LOG_FORCE_NEWFILE 0x1u /* Force switch to new log file */ #define WT_LOG_OPENED 0x2u /* Log subsystem successfully open */ #define WT_LOG_TRUNCATE_NOTSUP 0x4u /* File system truncate not supported */ - /* AUTOMATIC FLAG VALUE GENERATION STOP */ + /* AUTOMATIC FLAG VALUE GENERATION STOP 32 */ uint32_t flags; }; @@ -399,11 +399,11 @@ __wt_log_desc_byteswap(WT_LOG_DESC *desc) struct __wt_txn_printlog_args { WT_FSTREAM *fs; -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_TXN_PRINTLOG_HEX 0x1u /* Add hex output */ #define WT_TXN_PRINTLOG_MSG 0x2u /* Messages only */ #define WT_TXN_PRINTLOG_UNREDACT 0x4u /* Don't redact user data from output */ - /* AUTOMATIC FLAG VALUE GENERATION STOP */ + /* AUTOMATIC FLAG VALUE GENERATION STOP 32 */ uint32_t flags; }; diff --git a/src/third_party/wiredtiger/src/include/lsm.h b/src/third_party/wiredtiger/src/include/lsm.h index 0503a5c69f0..ef56793152f 100644 --- a/src/third_party/wiredtiger/src/include/lsm.h +++ b/src/third_party/wiredtiger/src/include/lsm.h @@ -66,7 +66,7 @@ struct __wt_cursor_lsm { u_int update_count; /* Updates performed. */ -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_CLSM_ACTIVE 0x001u /* Incremented the session count */ #define WT_CLSM_BULK 0x002u /* Open for snapshot isolation */ #define WT_CLSM_ITERATE_NEXT 0x004u /* Forward iteration */ @@ -76,7 +76,7 @@ struct __wt_cursor_lsm { #define WT_CLSM_MULTIPLE 0x040u /* Multiple cursors have values */ #define WT_CLSM_OPEN_READ 0x080u /* Open for reads */ #define WT_CLSM_OPEN_SNAPSHOT 0x100u /* Open for snapshot isolation */ - /* AUTOMATIC FLAG VALUE GENERATION STOP */ + /* AUTOMATIC FLAG VALUE GENERATION STOP 32 */ uint32_t flags; }; @@ -114,13 +114,13 @@ struct __wt_lsm_chunk { int8_t evicted; /* 1/0: in-memory chunk was evicted */ uint8_t flushing; /* 1/0: chunk flush in progress */ -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_LSM_CHUNK_BLOOM 0x01u #define WT_LSM_CHUNK_HAS_TIMESTAMP 0x02u #define WT_LSM_CHUNK_MERGING 0x04u #define WT_LSM_CHUNK_ONDISK 0x08u #define WT_LSM_CHUNK_STABLE 0x10u - /* AUTOMATIC FLAG VALUE GENERATION STOP */ + /* AUTOMATIC FLAG VALUE GENERATION STOP 32 */ uint32_t flags; }; @@ -128,14 +128,14 @@ struct __wt_lsm_chunk { * Different types of work units. Used by LSM worker threads to choose which type of work they will * execute, and by work units to define which action is required. */ -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_LSM_WORK_BLOOM 0x01u /* Create a bloom filter */ #define WT_LSM_WORK_DROP 0x02u /* Drop unused chunks */ #define WT_LSM_WORK_ENABLE_EVICT 0x04u /* Allow eviction of pinned chunk */ #define WT_LSM_WORK_FLUSH 0x08u /* Flush a chunk to disk */ #define WT_LSM_WORK_MERGE 0x10u /* Look for a tree merge */ #define WT_LSM_WORK_SWITCH 0x20u /* Switch the in-memory chunk */ -/* AUTOMATIC FLAG VALUE GENERATION STOP */ +/* AUTOMATIC FLAG VALUE GENERATION STOP 32 */ /* Work units that are serviced by general worker threads. */ #define WT_LSM_WORK_GENERAL_OPS \ @@ -149,9 +149,9 @@ struct __wt_lsm_chunk { struct __wt_lsm_work_unit { TAILQ_ENTRY(__wt_lsm_work_unit) q; /* Worker unit queue */ uint32_t type; /* Type of operation */ -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_LSM_WORK_FORCE 0x1u /* Force operation */ - /* AUTOMATIC FLAG VALUE GENERATION STOP */ + /* AUTOMATIC FLAG VALUE GENERATION STOP 32 */ uint32_t flags; /* Flags for operation */ WT_LSM_TREE *lsm_tree; }; @@ -185,9 +185,9 @@ struct __wt_lsm_manager { #define WT_LSM_MIN_WORKERS 3 WT_LSM_WORKER_ARGS lsm_worker_cookies[WT_LSM_MAX_WORKERS]; -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_LSM_MANAGER_SHUTDOWN 0x1u /* Manager has shut down */ - /* AUTOMATIC FLAG VALUE GENERATION STOP */ + /* AUTOMATIC FLAG VALUE GENERATION STOP 32 */ uint32_t flags; }; @@ -252,11 +252,11 @@ struct __wt_lsm_tree { uint64_t chunk_max; /* Maximum chunk a merge creates */ u_int merge_min, merge_max; -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_LSM_BLOOM_MERGED 0x1u #define WT_LSM_BLOOM_OFF 0x2u #define WT_LSM_BLOOM_OLDEST 0x4u - /* AUTOMATIC FLAG VALUE GENERATION STOP */ + /* AUTOMATIC FLAG VALUE GENERATION STOP 32 */ uint32_t bloom; /* Bloom creation policy */ WT_LSM_CHUNK **chunk; /* Array of active LSM chunks */ @@ -303,12 +303,12 @@ struct __wt_lsm_tree { * flags here are not protected for concurrent access, don't put anything here that is susceptible * to races. */ -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_LSM_TREE_COMPACTING 0x1u /* Tree being compacted */ #define WT_LSM_TREE_MERGES 0x2u /* Tree should run merges */ #define WT_LSM_TREE_OPEN 0x4u /* The tree is open */ #define WT_LSM_TREE_THROTTLE 0x8u /* Throttle updates */ - /* AUTOMATIC FLAG VALUE GENERATION STOP */ + /* AUTOMATIC FLAG VALUE GENERATION STOP 32 */ uint32_t flags; }; diff --git a/src/third_party/wiredtiger/src/include/meta.h b/src/third_party/wiredtiger/src/include/meta.h index 924aacc54db..f103e4f56c8 100644 --- a/src/third_party/wiredtiger/src/include/meta.h +++ b/src/third_party/wiredtiger/src/include/meta.h @@ -81,11 +81,11 @@ struct __wt_blkincr { const char *id_str; /* User's name for this backup. */ uint64_t granularity; /* Granularity of this backup. */ -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_BLKINCR_FULL 0x1u /* There is no checkpoint, always do full file */ #define WT_BLKINCR_INUSE 0x2u /* This entry is active */ #define WT_BLKINCR_VALID 0x4u /* This entry is valid */ - /* AUTOMATIC FLAG VALUE GENERATION STOP */ + /* AUTOMATIC FLAG VALUE GENERATION STOP 64 */ uint64_t flags; }; @@ -104,10 +104,10 @@ struct __wt_block_mods { uint64_t offset; /* Zero bit offset for bitstring */ uint64_t granularity; -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_BLOCK_MODS_RENAME 0x1u /* Entry is from a rename */ #define WT_BLOCK_MODS_VALID 0x2u /* Entry is valid */ - /* AUTOMATIC FLAG VALUE GENERATION STOP */ + /* AUTOMATIC FLAG VALUE GENERATION STOP 32 */ uint32_t flags; }; @@ -154,12 +154,12 @@ struct __wt_ckpt { void *bpriv; /* Block manager private */ -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_CKPT_ADD 0x01u /* Checkpoint to be added */ #define WT_CKPT_BLOCK_MODS 0x02u /* Return list of modified blocks */ #define WT_CKPT_DELETE 0x04u /* Checkpoint to be deleted */ #define WT_CKPT_FAKE 0x08u /* Checkpoint is a fake */ #define WT_CKPT_UPDATE 0x10u /* Checkpoint requires update */ - /* AUTOMATIC FLAG VALUE GENERATION STOP */ + /* AUTOMATIC FLAG VALUE GENERATION STOP 32 */ uint32_t flags; }; diff --git a/src/third_party/wiredtiger/src/include/misc.h b/src/third_party/wiredtiger/src/include/misc.h index 4451d2aa638..02395d8fa0b 100644 --- a/src/third_party/wiredtiger/src/include/misc.h +++ b/src/third_party/wiredtiger/src/include/misc.h @@ -118,11 +118,11 @@ addr)) /* - * Our internal free function clears the underlying address atomically so there is a smaller chance - * of racing threads seeing intermediate results while a structure is being free'd. (That would be a - * bug, of course, but I'd rather not drop core, just the same.) That's a non-standard "free" API, - * and the resulting bug is a mother to find -- make sure we get it right, don't make the caller - * remember to put the & operator on the pointer. + * Our internal free function clears the underlying address so there is a smaller chance of racing + * threads seeing intermediate results while a structure is being free'd. (That would be a bug, of + * course, but I'd rather not drop core, just the same.) That's a non-standard "free" API, and the + * resulting bug is non-trivial to find -- make sure we get it right, don't make the caller remember + * to put the & operator on the pointer. */ #define __wt_free(session, p) \ do { \ @@ -134,15 +134,21 @@ /* Overwrite whether or not this is a diagnostic build. */ #define __wt_explicit_overwrite(p, size) memset(p, WT_DEBUG_BYTE, size) #ifdef HAVE_DIAGNOSTIC -#define __wt_overwrite_and_free(session, p) \ - do { \ - __wt_explicit_overwrite(p, sizeof(*(p))); \ - __wt_free(session, p); \ +#define __wt_overwrite_and_free(session, p) \ + do { \ + void *__p = &(p); \ + if (*(void **)__p != NULL) { \ + __wt_explicit_overwrite(p, sizeof(*(p))); \ + __wt_free_int(session, __p); \ + } \ } while (0) #define __wt_overwrite_and_free_len(session, p, len) \ do { \ - __wt_explicit_overwrite(p, len); \ - __wt_free(session, p); \ + void *__p = &(p); \ + if (*(void **)__p != NULL) { \ + __wt_explicit_overwrite(p, len); \ + __wt_free_int(session, __p); \ + } \ } while (0) #else #define __wt_overwrite_and_free(session, p) __wt_free(session, p) @@ -236,6 +242,10 @@ #define WT_PREFIX_MATCH(str, pfx) \ (((const char *)(str))[0] == ((const char *)(pfx))[0] && strncmp(str, pfx, strlen(pfx)) == 0) +/* Check if a string matches a suffix. */ +#define WT_SUFFIX_MATCH(str, sfx) \ + (strlen(str) >= strlen(sfx) && strcmp(&str[strlen(str) - strlen(sfx)], sfx) == 0) + /* Check if a string matches a prefix, and move past it. */ #define WT_PREFIX_SKIP(str, pfx) (WT_PREFIX_MATCH(str, pfx) ? ((str) += strlen(pfx), 1) : 0) @@ -287,12 +297,13 @@ */ #ifdef HAVE_DIAGNOSTIC #define __wt_hazard_set(session, walk, busyp) \ - __wt_hazard_set_func(session, walk, busyp, __func__, __LINE__) + __wt_hazard_set_func(session, walk, busyp, __PRETTY_FUNCTION__, __LINE__) #define __wt_scr_alloc(session, size, scratchp) \ - __wt_scr_alloc_func(session, size, scratchp, __func__, __LINE__) -#define __wt_page_in(session, ref, flags) __wt_page_in_func(session, ref, flags, __func__, __LINE__) + __wt_scr_alloc_func(session, size, scratchp, __PRETTY_FUNCTION__, __LINE__) +#define __wt_page_in(session, ref, flags) \ + __wt_page_in_func(session, ref, flags, __PRETTY_FUNCTION__, __LINE__) #define __wt_page_swap(session, held, want, flags) \ - __wt_page_swap_func(session, held, want, flags, __func__, __LINE__) + __wt_page_swap_func(session, held, want, flags, __PRETTY_FUNCTION__, __LINE__) #else #define __wt_hazard_set(session, walk, busyp) __wt_hazard_set_func(session, walk, busyp) #define __wt_scr_alloc(session, size, scratchp) __wt_scr_alloc_func(session, size, scratchp) diff --git a/src/third_party/wiredtiger/src/include/msvc.h b/src/third_party/wiredtiger/src/include/msvc.h index ccffa827e4c..9823313035c 100644 --- a/src/third_party/wiredtiger/src/include/msvc.h +++ b/src/third_party/wiredtiger/src/include/msvc.h @@ -13,9 +13,9 @@ #define inline __inline -/* MSVC Doesn't provide __func__, it has __FUNCTION__ */ +/* MSVC Doesn't provide __PRETTY_FUNCTION__, it has __FUNCSIG__ */ #ifdef _MSC_VER -#define __func__ __FUNCTION__ +#define __PRETTY_FUNCTION__ __FUNCSIG__ #endif #define WT_PTRDIFFT_FMT "Id" /* ptrdiff_t format string */ diff --git a/src/third_party/wiredtiger/src/include/optrack.h b/src/third_party/wiredtiger/src/include/optrack.h index e18c56ac418..370715a1032 100644 --- a/src/third_party/wiredtiger/src/include/optrack.h +++ b/src/third_party/wiredtiger/src/include/optrack.h @@ -73,11 +73,11 @@ struct __wt_optrack_record { * is also used in error paths during failed open calls. */ #define WT_TRACK_OP_DECL static uint16_t __func_id = 0 -#define WT_TRACK_OP_INIT(s) \ - if (F_ISSET(S2C(s), WT_CONN_OPTRACK) && (s)->id != 0) { \ - if (__func_id == 0) \ - __wt_optrack_record_funcid(s, __func__, &__func_id); \ - WT_TRACK_OP(s, 0); \ +#define WT_TRACK_OP_INIT(s) \ + if (F_ISSET(S2C(s), WT_CONN_OPTRACK) && (s)->id != 0) { \ + if (__func_id == 0) \ + __wt_optrack_record_funcid(s, __PRETTY_FUNCTION__, &__func_id); \ + WT_TRACK_OP(s, 0); \ } #define WT_TRACK_OP_END(s) \ diff --git a/src/third_party/wiredtiger/src/include/os.h b/src/third_party/wiredtiger/src/include/os.h index 2b23882f660..f3d6602fbbc 100644 --- a/src/third_party/wiredtiger/src/include/os.h +++ b/src/third_party/wiredtiger/src/include/os.h @@ -173,11 +173,11 @@ struct __wt_fstream { wt_off_t size; /* File size */ WT_ITEM buf; /* Data */ -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_STREAM_APPEND 0x1u /* Open a stream for append */ #define WT_STREAM_READ 0x2u /* Open a stream for read */ #define WT_STREAM_WRITE 0x4u /* Open a stream for write */ - /* AUTOMATIC FLAG VALUE GENERATION STOP */ + /* AUTOMATIC FLAG VALUE GENERATION STOP 32 */ uint32_t flags; int (*close)(WT_SESSION_IMPL *, WT_FSTREAM *); diff --git a/src/third_party/wiredtiger/src/include/packing_inline.h b/src/third_party/wiredtiger/src/include/packing_inline.h index 84390ba7266..fd73ef8764c 100644 --- a/src/third_party/wiredtiger/src/include/packing_inline.h +++ b/src/third_party/wiredtiger/src/include/packing_inline.h @@ -212,51 +212,55 @@ next: } } -#define WT_PACK_GET(session, pv, ap) \ - do { \ - WT_ITEM *__item; \ - switch ((pv).type) { \ - case 'x': \ - break; \ - case 's': \ - case 'S': \ - (pv).u.s = va_arg(ap, const char *); \ - break; \ - case 'U': \ - case 'u': \ - __item = va_arg(ap, WT_ITEM *); \ - (pv).u.item.data = __item->data; \ - (pv).u.item.size = __item->size; \ - break; \ - case 'b': \ - case 'h': \ - case 'i': \ - (pv).u.i = va_arg(ap, int); \ - break; \ - case 'B': \ - case 'H': \ - case 'I': \ - case 't': \ - (pv).u.u = va_arg(ap, unsigned int); \ - break; \ - case 'l': \ - (pv).u.i = va_arg(ap, long); \ - break; \ - case 'L': \ - (pv).u.u = va_arg(ap, unsigned long); \ - break; \ - case 'q': \ - (pv).u.i = va_arg(ap, int64_t); \ - break; \ - case 'Q': \ - case 'r': \ - case 'R': \ - (pv).u.u = va_arg(ap, uint64_t); \ - break; \ - default: \ - /* User format strings have already been validated. */ \ - return (__wt_illegal_value(session, (pv).type)); \ - } \ +#define WT_PACK_GET(session, pv, ap) \ + do { \ + WT_ITEM *__item; \ + switch ((pv).type) { \ + case 'x': \ + break; \ + case 's': \ + case 'S': \ + (pv).u.s = va_arg(ap, const char *); \ + break; \ + case 'U': \ + case 'u': \ + __item = va_arg(ap, WT_ITEM *); \ + (pv).u.item.data = __item->data; \ + (pv).u.item.size = __item->size; \ + break; \ + case 'b': \ + case 'h': \ + case 'i': \ + case 'l': \ + /* Use the int type as compilers promote smaller sizes to int for variadic \ + * arguments. \ + * Note: 'l' accommodates 4 bytes \ + */ \ + (pv).u.i = va_arg(ap, int); \ + break; \ + case 'B': \ + case 'H': \ + case 'I': \ + case 'L': \ + case 't': \ + /* Use the int type as compilers promote smaller sizes to int for variadic \ + * arguments. \ + * Note: 'L' accommodates 4 bytes \ + */ \ + (pv).u.u = va_arg(ap, unsigned int); \ + break; \ + case 'q': \ + (pv).u.i = va_arg(ap, int64_t); \ + break; \ + case 'Q': \ + case 'r': \ + case 'R': \ + (pv).u.u = va_arg(ap, uint64_t); \ + break; \ + default: \ + /* User format strings have already been validated. */ \ + return (__wt_illegal_value(session, (pv).type)); \ + } \ } while (0) /* diff --git a/src/third_party/wiredtiger/src/include/reconcile.h b/src/third_party/wiredtiger/src/include/reconcile.h index 00057870a0c..ca94b8a1478 100644 --- a/src/third_party/wiredtiger/src/include/reconcile.h +++ b/src/third_party/wiredtiger/src/include/reconcile.h @@ -196,7 +196,7 @@ struct __wt_reconcile { uint32_t count_stop_txn; uint32_t count_prepare; -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_REC_TIME_NEWEST_START_DURABLE_TS 0x01u #define WT_REC_TIME_NEWEST_STOP_DURABLE_TS 0x02u #define WT_REC_TIME_NEWEST_STOP_TS 0x04u @@ -204,7 +204,7 @@ struct __wt_reconcile { #define WT_REC_TIME_NEWEST_TXN 0x10u #define WT_REC_TIME_OLDEST_START_TS 0x20u #define WT_REC_TIME_PREPARE 0x40u - /* AUTOMATIC FLAG VALUE GENERATION STOP */ + /* AUTOMATIC FLAG VALUE GENERATION STOP 16 */ uint16_t ts_usage_flags; /* diff --git a/src/third_party/wiredtiger/src/include/schema.h b/src/third_party/wiredtiger/src/include/schema.h index 9d2487798b5..e5e1f96704e 100644 --- a/src/third_party/wiredtiger/src/include/schema.h +++ b/src/third_party/wiredtiger/src/include/schema.h @@ -41,9 +41,9 @@ struct __wt_index { const char *idxkey_format; /* Index key format (hides primary) */ const char *exkey_format; /* Key format for custom extractors */ -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_INDEX_IMMUTABLE 0x1u - /* AUTOMATIC FLAG VALUE GENERATION STOP */ + /* AUTOMATIC FLAG VALUE GENERATION STOP 32 */ uint32_t flags; /* Index configuration flags */ }; diff --git a/src/third_party/wiredtiger/src/include/serial_inline.h b/src/third_party/wiredtiger/src/include/serial_inline.h index c98eac159ad..1fb7e31496e 100644 --- a/src/third_party/wiredtiger/src/include/serial_inline.h +++ b/src/third_party/wiredtiger/src/include/serial_inline.h @@ -229,6 +229,9 @@ __wt_update_serial(WT_SESSION_IMPL *session, WT_CURSOR_BTREE *cbt, WT_PAGE *page upd = *updp; *updp = NULL; prev_upd_ts = WT_TS_NONE; +#ifdef HAVE_DIAGNOSTIC + prev_upd_ts = upd->prev_durable_ts; +#endif /* * All structure setup must be flushed before the structure is entered into the list. We need a diff --git a/src/third_party/wiredtiger/src/include/session.h b/src/third_party/wiredtiger/src/include/session.h index f7ec0464a29..1bddc6a193e 100644 --- a/src/third_party/wiredtiger/src/include/session.h +++ b/src/third_party/wiredtiger/src/include/session.h @@ -164,6 +164,9 @@ struct __wt_session_impl { void *reconcile; /* Reconciliation support */ int (*reconcile_cleanup)(WT_SESSION_IMPL *); + /* Salvage support. */ + void *salvage_track; + /* Sessions have an associated statistics bucket based on its ID. */ u_int stat_bucket; /* Statistics bucket offset */ @@ -171,7 +174,7 @@ struct __wt_session_impl { uint8_t dump_raw; /* Configure debugging page dump */ #endif -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_SESSION_LOCKED_CHECKPOINT 0x0001u #define WT_SESSION_LOCKED_HANDLE_LIST_READ 0x0002u #define WT_SESSION_LOCKED_HANDLE_LIST_WRITE 0x0004u @@ -185,10 +188,10 @@ struct __wt_session_impl { #define WT_SESSION_LOCKED_TABLE_WRITE 0x0400u #define WT_SESSION_LOCKED_TURTLE 0x0800u #define WT_SESSION_NO_SCHEMA_LOCK 0x1000u - /* AUTOMATIC FLAG VALUE GENERATION STOP */ + /*AUTOMATIC FLAG VALUE GENERATION STOP 32 */ uint32_t lock_flags; -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_SESSION_BACKUP_CURSOR 0x00001u #define WT_SESSION_BACKUP_DUP 0x00002u #define WT_SESSION_CACHE_CURSORS 0x00004u @@ -208,7 +211,7 @@ struct __wt_session_impl { #define WT_SESSION_RESOLVING_TXN 0x10000u #define WT_SESSION_ROLLBACK_TO_STABLE 0x20000u #define WT_SESSION_SCHEMA_TXN 0x40000u - /* AUTOMATIC FLAG VALUE GENERATION STOP */ + /* AUTOMATIC FLAG VALUE GENERATION STOP 32 */ uint32_t flags; /* diff --git a/src/third_party/wiredtiger/src/include/stat.h b/src/third_party/wiredtiger/src/include/stat.h index 5b14e4bc80e..b0b161b876d 100644 --- a/src/third_party/wiredtiger/src/include/stat.h +++ b/src/third_party/wiredtiger/src/include/stat.h @@ -77,7 +77,7 @@ #define WT_SESSION_STATS_FIELD_TO_OFFSET(stats, fld) (int)(&(stats)->fld - (int64_t *)(stats)) -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_STAT_CLEAR 0x01u #define WT_STAT_JSON 0x02u #define WT_STAT_ON_CLOSE 0x04u @@ -86,7 +86,7 @@ #define WT_STAT_TYPE_FAST 0x20u #define WT_STAT_TYPE_SIZE 0x40u #define WT_STAT_TYPE_TREE_WALK 0x80u -/* AUTOMATIC FLAG VALUE GENERATION STOP */ +/* AUTOMATIC FLAG VALUE GENERATION STOP 32 */ /* * Sum the values from all structures in the array. @@ -316,6 +316,8 @@ struct __wt_connection_stats { int64_t lsm_work_queue_app; int64_t lsm_work_queue_manager; int64_t lsm_rows_merged; + int64_t lsm_checkpoint_throttle; + int64_t lsm_merge_throttle; int64_t lsm_work_queue_switch; int64_t lsm_work_units_discarded; int64_t lsm_work_units_done; @@ -342,8 +344,14 @@ struct __wt_connection_stats { int64_t cache_bytes_updates; int64_t cache_bytes_image; int64_t cache_bytes_hs; + int64_t cache_bytes_inuse; + int64_t cache_bytes_dirty_total; int64_t cache_bytes_other; + int64_t cache_bytes_read; + int64_t cache_bytes_write; int64_t cache_lookaside_score; + int64_t cache_eviction_checkpoint; + int64_t cache_eviction_blocked_checkpoint_hs; int64_t cache_eviction_get_ref; int64_t cache_eviction_get_ref_empty; int64_t cache_eviction_get_ref_empty2; @@ -357,9 +365,23 @@ struct __wt_connection_stats { int64_t cache_eviction_slow; int64_t cache_eviction_walk_leaf_notfound; int64_t cache_eviction_state; + int64_t cache_eviction_target_page_lt10; + int64_t cache_eviction_target_page_lt32; + int64_t cache_eviction_target_page_ge128; + int64_t cache_eviction_target_page_lt64; + int64_t cache_eviction_target_page_lt128; + int64_t cache_eviction_target_page_reduced; int64_t cache_eviction_target_strategy_both_clean_and_dirty; int64_t cache_eviction_target_strategy_clean; int64_t cache_eviction_target_strategy_dirty; + int64_t cache_eviction_walks_abandoned; + int64_t cache_eviction_walks_stopped; + int64_t cache_eviction_walks_gave_up_no_targets; + int64_t cache_eviction_walks_gave_up_ratio; + int64_t cache_eviction_walks_ended; + int64_t cache_eviction_walk_restart; + int64_t cache_eviction_walk_from_root; + int64_t cache_eviction_walk_saved_pos; int64_t cache_eviction_active_workers; int64_t cache_eviction_worker_created; int64_t cache_eviction_worker_evicting; @@ -379,19 +401,42 @@ struct __wt_connection_stats { int64_t cache_eviction_force; int64_t cache_eviction_force_fail; int64_t cache_eviction_force_fail_time; + int64_t cache_eviction_hazard; int64_t cache_hazard_checks; int64_t cache_hazard_walks; int64_t cache_hazard_max; int64_t cache_hs_score; + int64_t cache_hs_insert; + int64_t cache_hs_insert_restart; int64_t cache_hs_ondisk_max; int64_t cache_hs_ondisk; + int64_t cache_hs_order_lose_durable_timestamp; + int64_t cache_hs_order_reinsert; + int64_t cache_hs_read; + int64_t cache_hs_read_miss; + int64_t cache_hs_read_squash; + int64_t cache_hs_key_truncate_rts_unstable; + int64_t cache_hs_key_truncate_rts; + int64_t cache_hs_key_truncate; + int64_t cache_hs_key_truncate_onpage_removal; + int64_t cache_hs_order_remove; + int64_t cache_hs_write_squash; + int64_t cache_inmem_splittable; + int64_t cache_inmem_split; + int64_t cache_eviction_internal; int64_t cache_eviction_internal_pages_queued; int64_t cache_eviction_internal_pages_seen; int64_t cache_eviction_internal_pages_already_queued; + int64_t cache_eviction_split_internal; + int64_t cache_eviction_split_leaf; int64_t cache_bytes_max; int64_t cache_eviction_maximum_page_size; + int64_t cache_eviction_dirty; int64_t cache_eviction_app_dirty; int64_t cache_timed_out_ops; + int64_t cache_read_overflow; + int64_t cache_eviction_deepen; + int64_t cache_write_hs; int64_t cache_pages_inuse; int64_t cache_eviction_app; int64_t cache_eviction_pages_in_parallel_with_checkpoint; @@ -400,16 +445,26 @@ struct __wt_connection_stats { int64_t cache_eviction_pages_queued_urgent; int64_t cache_eviction_pages_queued_oldest; int64_t cache_eviction_pages_queued_urgent_hs_dirty; + int64_t cache_read; + int64_t cache_read_deleted; + int64_t cache_read_deleted_prepared; + int64_t cache_pages_requested; + int64_t cache_eviction_pages_seen; int64_t cache_eviction_pages_already_queued; int64_t cache_eviction_fail; int64_t cache_eviction_fail_parent_has_overflow_items; int64_t cache_eviction_fail_active_children_on_an_internal_page; int64_t cache_eviction_fail_in_reconciliation; + int64_t cache_eviction_fail_checkpoint_out_of_order_ts; int64_t cache_eviction_walk; + int64_t cache_write; + int64_t cache_write_restore; int64_t cache_overhead; int64_t cache_bytes_internal; int64_t cache_bytes_leaf; + int64_t cache_bytes_dirty; int64_t cache_pages_dirty; + int64_t cache_eviction_clean; int64_t fsync_all_fh_total; int64_t fsync_all_fh; int64_t fsync_all_time; @@ -424,6 +479,10 @@ struct __wt_connection_stats { int64_t capacity_time_evict; int64_t capacity_time_log; int64_t capacity_time_read; + int64_t cc_pages_evict; + int64_t cc_pages_removed; + int64_t cc_pages_walk_skipped; + int64_t cc_pages_visited; int64_t cond_auto_wait_reset; int64_t cond_auto_wait; int64_t cond_auto_wait_skipped; @@ -440,6 +499,12 @@ struct __wt_connection_stats { int64_t fsync_io; int64_t read_io; int64_t write_io; + int64_t cursor_next_skip_total; + int64_t cursor_prev_skip_total; + int64_t cursor_skip_hs_cur_position; + int64_t cursor_next_skip_page_count; + int64_t cursor_prev_skip_page_count; + int64_t cursor_search_near_prefix_fast_paths; int64_t cursor_cached_count; int64_t cursor_insert_bulk; int64_t cursor_cache; @@ -450,8 +515,14 @@ struct __wt_connection_stats { int64_t cursor_modify_bytes; int64_t cursor_modify_bytes_touch; int64_t cursor_next; + int64_t cursor_next_hs_tombstone; + int64_t cursor_next_skip_ge_100; + int64_t cursor_next_skip_lt_100; int64_t cursor_restart; int64_t cursor_prev; + int64_t cursor_prev_hs_tombstone; + int64_t cursor_prev_skip_ge_100; + int64_t cursor_prev_skip_lt_100; int64_t cursor_remove; int64_t cursor_remove_bytes; int64_t cursor_reserve; @@ -468,6 +539,7 @@ struct __wt_connection_stats { int64_t cursor_update_bytes; int64_t cursor_update_bytes_changed; int64_t cursor_reopen; + int64_t cursor_open_count; int64_t dh_conn_handle_size; int64_t dh_conn_handle_count; int64_t dh_sweep_ref; @@ -575,19 +647,42 @@ struct __wt_connection_stats { int64_t perf_hist_opwrite_latency_lt1000; int64_t perf_hist_opwrite_latency_lt10000; int64_t perf_hist_opwrite_latency_gt10000; + int64_t rec_time_window_bytes_ts; + int64_t rec_time_window_bytes_txn; + int64_t rec_page_delete_fast; int64_t rec_overflow_key_internal; int64_t rec_overflow_key_leaf; int64_t rec_maximum_seconds; + int64_t rec_pages; + int64_t rec_pages_eviction; int64_t rec_pages_with_prepare; int64_t rec_pages_with_ts; int64_t rec_pages_with_txn; + int64_t rec_page_delete; + int64_t rec_time_aggr_newest_start_durable_ts; + int64_t rec_time_aggr_newest_stop_durable_ts; + int64_t rec_time_aggr_newest_stop_ts; + int64_t rec_time_aggr_newest_stop_txn; + int64_t rec_time_aggr_newest_txn; + int64_t rec_time_aggr_oldest_start_ts; + int64_t rec_time_aggr_prepared; int64_t rec_time_window_pages_prepared; + int64_t rec_time_window_pages_durable_start_ts; int64_t rec_time_window_pages_start_ts; + int64_t rec_time_window_pages_start_txn; + int64_t rec_time_window_pages_durable_stop_ts; + int64_t rec_time_window_pages_stop_ts; + int64_t rec_time_window_pages_stop_txn; int64_t rec_time_window_prepared; + int64_t rec_time_window_durable_start_ts; + int64_t rec_time_window_start_ts; + int64_t rec_time_window_start_txn; + int64_t rec_time_window_durable_stop_ts; + int64_t rec_time_window_stop_ts; + int64_t rec_time_window_stop_txn; int64_t rec_split_stashed_bytes; int64_t rec_split_stashed_objects; int64_t flush_state_races; - int64_t flush_tier_busy; int64_t flush_tier; int64_t session_open; int64_t session_query_ts; @@ -608,6 +703,10 @@ struct __wt_connection_stats { int64_t session_table_truncate_success; int64_t session_table_verify_fail; int64_t session_table_verify_success; + int64_t tiered_work_units_dequeued; + int64_t tiered_work_units_created; + int64_t tiered_retention; + int64_t tiered_object_size; int64_t thread_fsync_active; int64_t thread_read_active; int64_t thread_write_active; @@ -635,10 +734,19 @@ struct __wt_connection_stats { int64_t txn_prepare_active; int64_t txn_prepare_rollback; int64_t txn_query_ts; + int64_t txn_read_race_prepare_update; int64_t txn_rts; + int64_t txn_rts_hs_stop_older_than_newer_start; + int64_t txn_rts_inconsistent_ckpt; + int64_t txn_rts_keys_removed; + int64_t txn_rts_keys_restored; int64_t txn_rts_pages_visited; + int64_t txn_rts_hs_restore_tombstones; + int64_t txn_rts_hs_restore_updates; + int64_t txn_rts_sweep_hs_keys; int64_t txn_rts_tree_walk_skip_pages; int64_t txn_rts_upd_aborted; + int64_t txn_rts_hs_removed; int64_t txn_sessions_walked; int64_t txn_set_ts; int64_t txn_set_ts_durable; @@ -670,6 +778,7 @@ struct __wt_connection_stats { int64_t txn_checkpoint_scrub_time; int64_t txn_checkpoint_time_total; int64_t txn_checkpoint; + int64_t txn_checkpoint_obsolete_applied; int64_t txn_checkpoint_skipped; int64_t txn_fail_cache; int64_t txn_checkpoint_fsync_post; @@ -686,113 +795,6 @@ struct __wt_connection_stats { int64_t txn_walk_sessions; int64_t txn_commit; int64_t txn_rollback; - int64_t lsm_checkpoint_throttle; - int64_t lsm_merge_throttle; - int64_t cache_bytes_inuse; - int64_t cache_bytes_dirty_total; - int64_t cache_bytes_read; - int64_t cache_bytes_write; - int64_t cache_eviction_checkpoint; - int64_t cache_eviction_blocked_checkpoint_hs; - int64_t cache_eviction_target_page_lt10; - int64_t cache_eviction_target_page_lt32; - int64_t cache_eviction_target_page_ge128; - int64_t cache_eviction_target_page_lt64; - int64_t cache_eviction_target_page_lt128; - int64_t cache_eviction_target_page_reduced; - int64_t cache_eviction_walks_abandoned; - int64_t cache_eviction_walks_stopped; - int64_t cache_eviction_walks_gave_up_no_targets; - int64_t cache_eviction_walks_gave_up_ratio; - int64_t cache_eviction_walks_ended; - int64_t cache_eviction_walk_restart; - int64_t cache_eviction_walk_from_root; - int64_t cache_eviction_walk_saved_pos; - int64_t cache_eviction_hazard; - int64_t cache_hs_insert; - int64_t cache_hs_insert_restart; - int64_t cache_hs_order_lose_durable_timestamp; - int64_t cache_hs_order_reinsert; - int64_t cache_hs_read; - int64_t cache_hs_read_miss; - int64_t cache_hs_read_squash; - int64_t cache_hs_key_truncate_rts_unstable; - int64_t cache_hs_key_truncate_rts; - int64_t cache_hs_key_truncate; - int64_t cache_hs_key_truncate_onpage_removal; - int64_t cache_hs_order_remove; - int64_t cache_hs_write_squash; - int64_t cache_inmem_splittable; - int64_t cache_inmem_split; - int64_t cache_eviction_internal; - int64_t cache_eviction_split_internal; - int64_t cache_eviction_split_leaf; - int64_t cache_eviction_dirty; - int64_t cache_read_overflow; - int64_t cache_eviction_deepen; - int64_t cache_write_hs; - int64_t cache_read; - int64_t cache_read_deleted; - int64_t cache_read_deleted_prepared; - int64_t cache_pages_requested; - int64_t cache_eviction_pages_seen; - int64_t cache_write; - int64_t cache_write_restore; - int64_t cache_bytes_dirty; - int64_t cache_eviction_clean; - int64_t cc_pages_evict; - int64_t cc_pages_removed; - int64_t cc_pages_walk_skipped; - int64_t cc_pages_visited; - int64_t cursor_next_skip_total; - int64_t cursor_prev_skip_total; - int64_t cursor_skip_hs_cur_position; - int64_t cursor_search_near_prefix_fast_paths; - int64_t cursor_next_hs_tombstone; - int64_t cursor_next_skip_ge_100; - int64_t cursor_next_skip_lt_100; - int64_t cursor_prev_hs_tombstone; - int64_t cursor_prev_skip_ge_100; - int64_t cursor_prev_skip_lt_100; - int64_t cursor_open_count; - int64_t rec_time_window_bytes_ts; - int64_t rec_time_window_bytes_txn; - int64_t rec_page_delete_fast; - int64_t rec_pages; - int64_t rec_pages_eviction; - int64_t rec_page_delete; - int64_t rec_time_aggr_newest_start_durable_ts; - int64_t rec_time_aggr_newest_stop_durable_ts; - int64_t rec_time_aggr_newest_stop_ts; - int64_t rec_time_aggr_newest_stop_txn; - int64_t rec_time_aggr_newest_txn; - int64_t rec_time_aggr_oldest_start_ts; - int64_t rec_time_aggr_prepared; - int64_t rec_time_window_pages_durable_start_ts; - int64_t rec_time_window_pages_start_txn; - int64_t rec_time_window_pages_durable_stop_ts; - int64_t rec_time_window_pages_stop_ts; - int64_t rec_time_window_pages_stop_txn; - int64_t rec_time_window_durable_start_ts; - int64_t rec_time_window_start_ts; - int64_t rec_time_window_start_txn; - int64_t rec_time_window_durable_stop_ts; - int64_t rec_time_window_stop_ts; - int64_t rec_time_window_stop_txn; - int64_t tiered_work_units_dequeued; - int64_t tiered_work_units_created; - int64_t tiered_retention; - int64_t tiered_object_size; - int64_t txn_read_race_prepare_update; - int64_t txn_rts_hs_stop_older_than_newer_start; - int64_t txn_rts_inconsistent_ckpt; - int64_t txn_rts_keys_removed; - int64_t txn_rts_keys_restored; - int64_t txn_rts_hs_restore_tombstones; - int64_t txn_rts_hs_restore_updates; - int64_t txn_rts_sweep_hs_keys; - int64_t txn_rts_hs_removed; - int64_t txn_checkpoint_obsolete_applied; int64_t txn_update_conflict; }; @@ -810,6 +812,8 @@ struct __wt_dsrc_stats { int64_t lsm_chunk_count; int64_t lsm_generation_max; int64_t lsm_lookup_no_bloom; + int64_t lsm_checkpoint_throttle; + int64_t lsm_merge_throttle; int64_t bloom_size; int64_t block_extension; int64_t block_alloc; @@ -841,80 +845,14 @@ struct __wt_dsrc_stats { int64_t btree_row_empty_values; int64_t btree_row_internal; int64_t btree_row_leaf; - int64_t cache_eviction_fail; - int64_t cache_eviction_walk_passes; - int64_t cache_state_gen_avg_gap; - int64_t cache_state_avg_written_size; - int64_t cache_state_avg_visited_age; - int64_t cache_state_avg_unvisited_age; - int64_t cache_state_pages_clean; - int64_t cache_state_gen_current; - int64_t cache_state_pages_dirty; - int64_t cache_state_root_entries; - int64_t cache_state_pages_internal; - int64_t cache_state_pages_leaf; - int64_t cache_state_gen_max_gap; - int64_t cache_state_max_pagesize; - int64_t cache_state_min_written_size; - int64_t cache_state_unvisited_count; - int64_t cache_state_smaller_alloc_size; - int64_t cache_state_memory; - int64_t cache_state_queued; - int64_t cache_state_not_queueable; - int64_t cache_state_refs_skipped; - int64_t cache_state_root_size; - int64_t cache_state_pages; - int64_t compress_precomp_intl_max_page_size; - int64_t compress_precomp_leaf_max_page_size; - int64_t compress_read; - int64_t compress_write; - int64_t compress_write_fail; - int64_t compress_write_too_small; - int64_t cursor_insert_bulk; - int64_t cursor_reopen; - int64_t cursor_cache; - int64_t cursor_create; - int64_t cursor_insert; - int64_t cursor_insert_bytes; - int64_t cursor_modify; - int64_t cursor_modify_bytes; - int64_t cursor_modify_bytes_touch; - int64_t cursor_next; - int64_t cursor_restart; - int64_t cursor_prev; - int64_t cursor_remove; - int64_t cursor_remove_bytes; - int64_t cursor_reserve; - int64_t cursor_reset; - int64_t cursor_search; - int64_t cursor_search_hs; - int64_t cursor_search_near; - int64_t cursor_truncate; - int64_t cursor_update; - int64_t cursor_update_bytes; - int64_t cursor_update_bytes_changed; - int64_t rec_dictionary; - int64_t rec_suffix_compression; - int64_t rec_multiblock_internal; - int64_t rec_overflow_key_internal; - int64_t rec_prefix_compression; - int64_t rec_multiblock_leaf; - int64_t rec_overflow_key_leaf; - int64_t rec_multiblock_max; - int64_t rec_overflow_value; - int64_t rec_page_match; - int64_t rec_time_window_pages_prepared; - int64_t rec_time_window_pages_start_ts; - int64_t rec_time_window_prepared; - int64_t session_compact; - int64_t lsm_checkpoint_throttle; - int64_t lsm_merge_throttle; int64_t cache_bytes_inuse; int64_t cache_bytes_dirty_total; int64_t cache_bytes_read; int64_t cache_bytes_write; int64_t cache_eviction_checkpoint; int64_t cache_eviction_blocked_checkpoint_hs; + int64_t cache_eviction_fail; + int64_t cache_eviction_walk_passes; int64_t cache_eviction_target_page_lt10; int64_t cache_eviction_target_page_lt32; int64_t cache_eviction_target_page_ge128; @@ -961,24 +899,86 @@ struct __wt_dsrc_stats { int64_t cache_write_restore; int64_t cache_bytes_dirty; int64_t cache_eviction_clean; + int64_t cache_state_gen_avg_gap; + int64_t cache_state_avg_written_size; + int64_t cache_state_avg_visited_age; + int64_t cache_state_avg_unvisited_age; + int64_t cache_state_pages_clean; + int64_t cache_state_gen_current; + int64_t cache_state_pages_dirty; + int64_t cache_state_root_entries; + int64_t cache_state_pages_internal; + int64_t cache_state_pages_leaf; + int64_t cache_state_gen_max_gap; + int64_t cache_state_max_pagesize; + int64_t cache_state_min_written_size; + int64_t cache_state_unvisited_count; + int64_t cache_state_smaller_alloc_size; + int64_t cache_state_memory; + int64_t cache_state_queued; + int64_t cache_state_not_queueable; + int64_t cache_state_refs_skipped; + int64_t cache_state_root_size; + int64_t cache_state_pages; int64_t cc_pages_evict; int64_t cc_pages_removed; int64_t cc_pages_walk_skipped; int64_t cc_pages_visited; + int64_t compress_precomp_intl_max_page_size; + int64_t compress_precomp_leaf_max_page_size; + int64_t compress_read; + int64_t compress_write; + int64_t compress_write_fail; + int64_t compress_write_too_small; int64_t cursor_next_skip_total; int64_t cursor_prev_skip_total; int64_t cursor_skip_hs_cur_position; + int64_t cursor_next_skip_page_count; + int64_t cursor_prev_skip_page_count; int64_t cursor_search_near_prefix_fast_paths; + int64_t cursor_insert_bulk; + int64_t cursor_reopen; + int64_t cursor_cache; + int64_t cursor_create; int64_t cursor_next_hs_tombstone; int64_t cursor_next_skip_ge_100; int64_t cursor_next_skip_lt_100; int64_t cursor_prev_hs_tombstone; int64_t cursor_prev_skip_ge_100; int64_t cursor_prev_skip_lt_100; + int64_t cursor_insert; + int64_t cursor_insert_bytes; + int64_t cursor_modify; + int64_t cursor_modify_bytes; + int64_t cursor_modify_bytes_touch; + int64_t cursor_next; int64_t cursor_open_count; + int64_t cursor_restart; + int64_t cursor_prev; + int64_t cursor_remove; + int64_t cursor_remove_bytes; + int64_t cursor_reserve; + int64_t cursor_reset; + int64_t cursor_search; + int64_t cursor_search_hs; + int64_t cursor_search_near; + int64_t cursor_truncate; + int64_t cursor_update; + int64_t cursor_update_bytes; + int64_t cursor_update_bytes_changed; int64_t rec_time_window_bytes_ts; int64_t rec_time_window_bytes_txn; + int64_t rec_dictionary; int64_t rec_page_delete_fast; + int64_t rec_suffix_compression; + int64_t rec_multiblock_internal; + int64_t rec_overflow_key_internal; + int64_t rec_prefix_compression; + int64_t rec_multiblock_leaf; + int64_t rec_overflow_key_leaf; + int64_t rec_multiblock_max; + int64_t rec_overflow_value; + int64_t rec_page_match; int64_t rec_pages; int64_t rec_pages_eviction; int64_t rec_page_delete; @@ -989,17 +989,21 @@ struct __wt_dsrc_stats { int64_t rec_time_aggr_newest_txn; int64_t rec_time_aggr_oldest_start_ts; int64_t rec_time_aggr_prepared; + int64_t rec_time_window_pages_prepared; int64_t rec_time_window_pages_durable_start_ts; + int64_t rec_time_window_pages_start_ts; int64_t rec_time_window_pages_start_txn; int64_t rec_time_window_pages_durable_stop_ts; int64_t rec_time_window_pages_stop_ts; int64_t rec_time_window_pages_stop_txn; + int64_t rec_time_window_prepared; int64_t rec_time_window_durable_start_ts; int64_t rec_time_window_start_ts; int64_t rec_time_window_start_txn; int64_t rec_time_window_durable_stop_ts; int64_t rec_time_window_stop_ts; int64_t rec_time_window_stop_txn; + int64_t session_compact; int64_t tiered_work_units_dequeued; int64_t tiered_work_units_created; int64_t tiered_retention; diff --git a/src/third_party/wiredtiger/src/include/thread_group.h b/src/third_party/wiredtiger/src/include/thread_group.h index b65313292e4..f5edf6d858a 100644 --- a/src/third_party/wiredtiger/src/include/thread_group.h +++ b/src/third_party/wiredtiger/src/include/thread_group.h @@ -21,12 +21,12 @@ struct __wt_thread { * WT_THREAD and thread-group function flags, merged because WT_THREAD_PANIC_FAIL appears in both * groups. */ -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_THREAD_ACTIVE 0x1u /* Thread is active or paused */ #define WT_THREAD_CAN_WAIT 0x2u /* WT_SESSION_CAN_WAIT */ #define WT_THREAD_PANIC_FAIL 0x4u /* Panic if the thread fails */ #define WT_THREAD_RUN 0x8u /* Thread is running */ - /* AUTOMATIC FLAG VALUE GENERATION STOP */ + /* AUTOMATIC FLAG VALUE GENERATION STOP 32 */ uint32_t flags; /* diff --git a/src/third_party/wiredtiger/src/include/tiered.h b/src/third_party/wiredtiger/src/include/tiered.h index 12ffc5f7b49..9758042daad 100644 --- a/src/third_party/wiredtiger/src/include/tiered.h +++ b/src/third_party/wiredtiger/src/include/tiered.h @@ -20,9 +20,9 @@ struct __wt_tiered_manager { #define WT_TIERED_MAX_WORKERS 20 #define WT_TIERED_MIN_WORKERS 1 -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_TIERED_MANAGER_SHUTDOWN 0x1u /* Manager has shut down */ - /* AUTOMATIC FLAG VALUE GENERATION STOP */ + /* AUTOMATIC FLAG VALUE GENERATION STOP 32 */ uint32_t flags; }; @@ -39,19 +39,19 @@ struct __wt_tiered_manager { #define WT_TIERED_MAX_TIERS 4 /* Object name types */ -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_TIERED_NAME_LOCAL 0x1u #define WT_TIERED_NAME_OBJECT 0x2u #define WT_TIERED_NAME_PREFIX 0x4u #define WT_TIERED_NAME_SHARED 0x8u -/* AUTOMATIC FLAG VALUE GENERATION STOP */ +/* AUTOMATIC FLAG VALUE GENERATION STOP 32 */ /* Flush tier flags */ -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_FLUSH_TIER_FORCE 0x1u #define WT_FLUSH_TIER_OFF 0x2u #define WT_FLUSH_TIER_ON 0x4u -/* AUTOMATIC FLAG VALUE GENERATION STOP */ +/* AUTOMATIC FLAG VALUE GENERATION STOP 32 */ /* * The flush state is a simple counter we manipulate atomically. @@ -61,11 +61,11 @@ struct __wt_tiered_manager { /* * Different types of work units for tiered trees. */ -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_TIERED_WORK_DROP_LOCAL 0x1u /* Drop object from local storage. */ #define WT_TIERED_WORK_DROP_SHARED 0x2u /* Drop object from tier. */ #define WT_TIERED_WORK_FLUSH 0x4u /* Flush object to tier. */ -/* AUTOMATIC FLAG VALUE GENERATION STOP */ +/* AUTOMATIC FLAG VALUE GENERATION STOP 32 */ /* * WT_TIERED_WORK_UNIT -- @@ -77,10 +77,10 @@ struct __wt_tiered_work_unit { uint64_t op_val; /* A value for the operation */ WT_TIERED *tiered; /* Tiered tree */ uint32_t id; /* Id of the object */ -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_TIERED_WORK_FORCE 0x1u /* Force operation */ #define WT_TIERED_WORK_FREE 0x2u /* Free data after operation */ - /* AUTOMATIC FLAG VALUE GENERATION STOP */ + /* AUTOMATIC FLAG VALUE GENERATION STOP 32 */ uint32_t flags; /* Flags for operation */ }; @@ -94,11 +94,11 @@ struct __wt_tiered_work_unit { struct __wt_tiered_tiers { WT_DATA_HANDLE *tier; /* Data handle for this tier */ const char *name; /* Tier's metadata name */ -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_TIERS_OP_FLUSH 0x1u #define WT_TIERS_OP_READ 0x2u #define WT_TIERS_OP_WRITE 0x4u - /* AUTOMATIC FLAG VALUE GENERATION STOP */ + /* AUTOMATIC FLAG VALUE GENERATION STOP 32 */ uint32_t flags; /* Flags including operations */ }; @@ -128,12 +128,9 @@ struct __wt_tiered { uint32_t current_id; /* Current object id number */ uint32_t next_id; /* Next object number */ - WT_COLLATOR *collator; /* TODO: handle custom collation */ - /* TODO: What about compression, encryption, etc? Do we need to worry about that here? */ - -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_TIERED_FLAG_UNUSED 0x1u - /* AUTOMATIC FLAG VALUE GENERATION STOP */ + /* AUTOMATIC FLAG VALUE GENERATION STOP 32 */ uint32_t flags; }; @@ -156,9 +153,9 @@ struct __wt_tiered_object { uint32_t generation; /* Do we need this?? */ uint32_t refcnt; /* Number of references */ -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_TIERED_OBJ_LOCAL 0x1u /* Local resident also */ - /* AUTOMATIC FLAG VALUE GENERATION STOP */ + /* AUTOMATIC FLAG VALUE GENERATION STOP 32 */ uint32_t flags; }; @@ -172,8 +169,8 @@ struct __wt_tiered_tree { const char *name, *config; const char *key_format, *value_format; -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_TIERED_TREE_UNUSED 0x1u - /* AUTOMATIC FLAG VALUE GENERATION STOP */ + /* AUTOMATIC FLAG VALUE GENERATION STOP 32 */ uint32_t flags; }; diff --git a/src/third_party/wiredtiger/src/include/timestamp_inline.h b/src/third_party/wiredtiger/src/include/timestamp_inline.h index 0d36eb85ea8..43f8c59cdf4 100644 --- a/src/third_party/wiredtiger/src/include/timestamp_inline.h +++ b/src/third_party/wiredtiger/src/include/timestamp_inline.h @@ -18,7 +18,7 @@ (tw)->prepare = 0; \ } while (0) -/* Copy the values from one time window structure to another. */ +/* Copy the values from one time window structure to another. */ #define WT_TIME_WINDOW_COPY(dest, source) (*(dest) = *(source)) /* Return true if the time window is equivalent to the default time window. */ diff --git a/src/third_party/wiredtiger/src/include/txn.h b/src/third_party/wiredtiger/src/include/txn.h index 07f1599aca8..3258cafb29f 100644 --- a/src/third_party/wiredtiger/src/include/txn.h +++ b/src/third_party/wiredtiger/src/include/txn.h @@ -14,24 +14,24 @@ #define WT_TS_NONE 0 /* Beginning of time */ #define WT_TS_MAX UINT64_MAX /* End of time */ -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_TXN_LOG_CKPT_CLEANUP 0x01u #define WT_TXN_LOG_CKPT_PREPARE 0x02u #define WT_TXN_LOG_CKPT_START 0x04u #define WT_TXN_LOG_CKPT_STOP 0x08u #define WT_TXN_LOG_CKPT_SYNC 0x10u -/* AUTOMATIC FLAG VALUE GENERATION STOP */ +/* AUTOMATIC FLAG VALUE GENERATION STOP 32 */ -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_TXN_OLDEST_STRICT 0x1u #define WT_TXN_OLDEST_WAIT 0x2u -/* AUTOMATIC FLAG VALUE GENERATION STOP */ +/* AUTOMATIC FLAG VALUE GENERATION STOP 32 */ -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_TXN_TS_ALREADY_LOCKED 0x1u #define WT_TXN_TS_INCLUDE_CKPT 0x2u #define WT_TXN_TS_INCLUDE_OLDEST 0x4u -/* AUTOMATIC FLAG VALUE GENERATION STOP */ +/* AUTOMATIC FLAG VALUE GENERATION STOP 32 */ typedef enum { WT_VISIBLE_FALSE = 0, /* Not a visible update */ @@ -225,9 +225,9 @@ struct __wt_txn_op { } truncate_row; } u; -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_TXN_OP_KEY_REPEATED 0x1u - /* AUTOMATIC FLAG VALUE GENERATION STOP */ + /* AUTOMATIC FLAG VALUE GENERATION STOP 32 */ uint32_t flags; }; @@ -314,7 +314,7 @@ struct __wt_txn { * clearing. */ -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_TXN_AUTOCOMMIT 0x000001u #define WT_TXN_ERROR 0x000002u #define WT_TXN_HAS_ID 0x000004u @@ -339,7 +339,7 @@ struct __wt_txn { #define WT_TXN_TS_WRITE_ORDERED 0x200000u #define WT_TXN_UPDATE 0x400000u #define WT_TXN_VERB_TS_WRITE 0x800000u - /* AUTOMATIC FLAG VALUE GENERATION STOP */ + /* AUTOMATIC FLAG VALUE GENERATION STOP 32 */ uint32_t flags; /* diff --git a/src/third_party/wiredtiger/src/include/txn_inline.h b/src/third_party/wiredtiger/src/include/txn_inline.h index 0deaf77a532..97fea21a14c 100644 --- a/src/third_party/wiredtiger/src/include/txn_inline.h +++ b/src/third_party/wiredtiger/src/include/txn_inline.h @@ -251,10 +251,7 @@ __wt_txn_op_apply_prepare_state(WT_SESSION_IMPL *session, WT_REF *ref, bool comm txn = session->txn; - /* - * Lock the ref to ensure we don't race with eviction freeing the page deleted update list or - * with a page instantiate. - */ + /* Lock the ref to ensure we don't race with page instantiation. */ WT_REF_LOCK(session, ref, &previous_state); if (commit) { @@ -264,20 +261,27 @@ __wt_txn_op_apply_prepare_state(WT_SESSION_IMPL *session, WT_REF *ref, bool comm ts = txn->prepare_timestamp; prepare_state = WT_PREPARE_INPROGRESS; } - for (updp = ref->page_del->update_list; updp != NULL && *updp != NULL; ++updp) { - (*updp)->start_ts = ts; - /* - * Holding the ref locked means we have exclusive access, so if we are committing we don't - * need to use the prepare locked transition state. - */ - (*updp)->prepare_state = prepare_state; + + /* + * Timestamps and prepare state are in the page deleted structure for truncates, or in the + * updates in the case of instantiated pages. + */ + if (previous_state == WT_REF_DELETED) { + ref->ft_info.del->timestamp = ts; if (commit) - (*updp)->durable_ts = txn->durable_timestamp; - } - ref->page_del->timestamp = ts; - if (commit) - ref->page_del->durable_timestamp = txn->durable_timestamp; - WT_PUBLISH(ref->page_del->prepare_state, prepare_state); + ref->ft_info.del->durable_timestamp = txn->durable_timestamp; + WT_PUBLISH(ref->ft_info.del->prepare_state, prepare_state); + } else if ((updp = ref->ft_info.update) != NULL) + for (; *updp != NULL; ++updp) { + (*updp)->start_ts = ts; + /* + * Holding the ref locked means we have exclusive access, so if we are committing we + * don't need to use the prepare locked transition state. + */ + (*updp)->prepare_state = prepare_state; + if (commit) + (*updp)->durable_ts = txn->durable_timestamp; + } WT_REF_UNLOCK(ref, previous_state); } @@ -295,16 +299,23 @@ __wt_txn_op_delete_commit_apply_timestamps(WT_SESSION_IMPL *session, WT_REF *ref txn = session->txn; - /* - * Lock the ref to ensure we don't race with eviction freeing the page deleted update list or - * with a page instantiate. - */ + /* Lock the ref to ensure we don't race with page instantiation. */ WT_REF_LOCK(session, ref, &previous_state); - for (updp = ref->page_del->update_list; updp != NULL && *updp != NULL; ++updp) { - (*updp)->start_ts = txn->commit_timestamp; - (*updp)->durable_ts = txn->durable_timestamp; - } + /* + * Timestamps are in the page deleted structure for truncates, or in the updates in the case of + * instantiated pages. Both commit and durable timestamps need to be updated. + */ + if (previous_state == WT_REF_DELETED) { + if (ref->ft_info.del->timestamp == WT_TS_NONE) { + ref->ft_info.del->timestamp = txn->commit_timestamp; + ref->ft_info.del->durable_timestamp = txn->durable_timestamp; + } + } else if ((updp = ref->ft_info.update) != NULL) + for (; *updp != NULL; ++updp) { + (*updp)->start_ts = txn->commit_timestamp; + (*updp)->durable_ts = txn->durable_timestamp; + } WT_REF_UNLOCK(ref, previous_state); } @@ -320,7 +331,6 @@ __wt_txn_op_set_timestamp(WT_SESSION_IMPL *session, WT_TXN_OP *op) { WT_TXN *txn; WT_UPDATE *upd; - wt_timestamp_t *timestamp; txn = session->txn; @@ -345,22 +355,19 @@ __wt_txn_op_set_timestamp(WT_SESSION_IMPL *session, WT_TXN_OP *op) __txn_resolve_prepared_update(session, upd); } } else { - /* - * The timestamp is in the page deleted structure for truncates, or in the update for other - * operations. Both commit and durable timestamps need to be updated. - */ - timestamp = op->type == WT_TXN_OP_REF_DELETE ? &op->u.ref->page_del->timestamp : - &op->u.op_upd->start_ts; - if (*timestamp == WT_TS_NONE) { - *timestamp = txn->commit_timestamp; - - timestamp = op->type == WT_TXN_OP_REF_DELETE ? &op->u.ref->page_del->durable_timestamp : - &op->u.op_upd->durable_ts; - *timestamp = txn->durable_timestamp; - } - if (op->type == WT_TXN_OP_REF_DELETE) __wt_txn_op_delete_commit_apply_timestamps(session, op->u.ref); + else { + /* + * The timestamp is in the update for operations other than truncate. Both commit and + * durable timestamps need to be updated. + */ + upd = op->u.op_upd; + if (upd->start_ts == WT_TS_NONE) { + upd->start_ts = txn->commit_timestamp; + upd->durable_ts = txn->durable_timestamp; + } + } } } @@ -421,9 +428,10 @@ __wt_txn_modify_page_delete(WT_SESSION_IMPL *session, WT_REF *ref) WT_RET(__txn_next_op(session, &op)); op->type = WT_TXN_OP_REF_DELETE; - op->u.ref = ref; - ref->page_del->txnid = txn->id; + + /* This access to the WT_PAGE_DELETED structure is safe, caller has the WT_REF locked. */ + ref->ft_info.del->txnid = txn->id; __wt_txn_op_set_timestamp(session, op); WT_ERR(__wt_txn_log_op(session, NULL)); diff --git a/src/third_party/wiredtiger/src/include/wiredtiger.in b/src/third_party/wiredtiger/src/include/wiredtiger.in index 75ac38e56d6..028bcddbadb 100644 --- a/src/third_party/wiredtiger/src/include/wiredtiger.in +++ b/src/third_party/wiredtiger/src/include/wiredtiger.in @@ -127,10 +127,10 @@ struct __wt_item { size_t memsize; /*! Object flags (internal use). */ -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_ITEM_ALIGNED 0x1u #define WT_ITEM_INUSE 0x2u -/* AUTOMATIC FLAG VALUE GENERATION STOP */ +/* AUTOMATIC FLAG VALUE GENERATION STOP 32 */ uint32_t flags; #endif }; @@ -503,15 +503,14 @@ struct __wt_cursor { * \c S), or raw byte arrays accessed using a WT_ITEM structure (value * format type \c u). * - * The WT_CURSOR::modify method stores a change record in cache and - * writes a change record to the log instead of the usual complete - * values. Note that WT_CURSOR::modify is generally slower than the - * WT_CURSOR::update method, and can result in slower reads because - * the complete value must be assembled during retrieval. The - * WT_CURSOR::modify method is intended for applications modifying - * large records where there is cache or I/O pressure, that is, - * applications that will benefit when data updates require less cache - * and they write less logging information. + * The WT_CURSOR::modify method stores a change record in cache and writes a change record + * to the log instead of the usual complete values. Using WT_CURSOR::modify will result in + * slower reads, and slower writes than the WT_CURSOR::insert or WT_CURSOR::update methods, + * because of the need to assemble the complete value in both the read and write paths. The + * WT_CURSOR::modify method is intended for applications where memory and log amplification + * are issues (in other words, applications where there is cache or I/O pressure and the + * application wants to trade performance for a smaller working set in cache and smaller + * log records). * * @snippet ex_all.c Modify an existing record * @@ -696,7 +695,7 @@ struct __wt_cursor { */ const char *internal_uri; -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_CURSTD_APPEND 0x0000001u #define WT_CURSTD_BULK 0x0000002u #define WT_CURSTD_CACHEABLE 0x0000004u @@ -723,7 +722,7 @@ struct __wt_cursor { #define WT_CURSTD_RAW_SEARCH 0x0800000u #define WT_CURSTD_VALUE_EXT 0x1000000u /* Value points out of tree. */ #define WT_CURSTD_VALUE_INT 0x2000000u /* Value points into tree. */ -/* AUTOMATIC FLAG VALUE GENERATION STOP */ +/* AUTOMATIC FLAG VALUE GENERATION STOP 32 */ #define WT_CURSTD_KEY_SET (WT_CURSTD_KEY_EXT | WT_CURSTD_KEY_INT) #define WT_CURSTD_VALUE_SET (WT_CURSTD_VALUE_EXT | WT_CURSTD_VALUE_INT) uint32_t flags; @@ -1086,12 +1085,13 @@ struct __wt_session { * @config{cache_resident, do not ever evict the object's pages from cache. Not compatible * with LSM tables; see @ref tuning_cache_resident for more information., a boolean flag; * default \c false.} - * @config{checksum, configure block checksums; permitted values are <code>on</code> - * (checksum all blocks)\, <code>off</code> (checksum no blocks) and - * <code>uncompresssed</code> (checksum only blocks which are not compressed for any - * reason). The \c uncompressed setting is for applications which can rely on decompression - * to fail if a block has been corrupted., a string\, chosen from the following options: \c - * "on"\, \c "off"\, \c "uncompressed"; default \c uncompressed.} + * @config{checksum, configure block checksums; the permitted values are \c on\, \c off\, \c + * uncompressed and \c unencrypted. The default is \c on\, in which case all block writes + * include a checksum subsequently verified when the block is read. The \c off setting does + * no checksums\, the \c uncompressed setting only checksums blocks that are not + * compressed\, and the \c unencrypted setting only checksums blocks that are not encrypted. + * See @ref tune_checksum for more information., a string\, chosen from the following + * options: \c "on"\, \c "off"\, \c "uncompressed"\, \c "unencrypted"; default \c on.} * @config{colgroups, comma-separated list of names of column groups. Each column group is * stored separately\, keyed by the primary key of the table. If no column groups are * specified\, all columns are stored together in a single file. All value columns in the @@ -2437,7 +2437,8 @@ struct __wt_connection { * to have completed. Any updates to checkpoint durable tables that are more recent than * the stable timestamp are removed. * - * This method requires that there are no active operations for the duration of the call. + * This method requires that there are no open cursors or active operations for the duration + * of the call. * * Any updates made to logged tables will not be rolled back. Any updates made without an * associated timestamp will not be rolled back. See @ref transaction_timestamps. @@ -2447,6 +2448,8 @@ struct __wt_connection { * @param connection the connection handle * @configempty{WT_CONNECTION.rollback_to_stable, see dist/api_data.py} * @errors + * An error should occur only in the case of a system problem, and an application typically + * will retry WT_CONNECTION::rollback_to_stable on error, or fail outright. */ int __F(rollback_to_stable)( WT_CONNECTION *connection, const char *config); @@ -2620,7 +2623,8 @@ struct __wt_connection { /*! * Get a storage source implementation. * - * Look up a storage source by name. + * Look up a storage source by name and return it. The returned storage source + * must be released by calling WT_STORAGE_SOURCE::terminate. * * @snippet ex_storage_source.c WT_STORAGE_SOURCE register * @@ -4049,17 +4053,15 @@ struct __wt_data_source { /*! * The interface implemented by applications to provide custom encryption. * - * Encryptors must implement the WT_ENCRYPTOR interface: the - * WT_ENCRYPTOR::encrypt, WT_ENCRYPTOR::decrypt and WT_ENCRYPTOR::sizing - * callbacks must be specified, WT_ENCRYPTOR::customize and - * WT_ENCRYPTOR::terminate are optional. To build your own encryptor, use - * one of the encryptors in \c ext/encryptors as a template: - * \c ext/encryptors/nop_encrypt is a simple encryptor that passes through - * data unchanged, and is a reasonable starting point; - * \c ext/encryptors/rotn_encrypt is an encryptor implementing - * a simple rotation cipher, it shows the use of \c keyid, \c secretkey, - * and implements the WT_ENCRYPTOR::customize and - * WT_ENCRYPTOR::terminate callbacks. + * Encryptors must implement the WT_ENCRYPTOR interface: the WT_ENCRYPTOR::encrypt, + * WT_ENCRYPTOR::decrypt and WT_ENCRYPTOR::sizing callbacks must be specified, + * WT_ENCRYPTOR::customize and WT_ENCRYPTOR::terminate are optional. To build your own + * encryptor, use one of the encryptors in \c ext/encryptors as a template: \c + * ext/encryptors/sodium_encrypt uses the open-source libsodium cryptographic library, and + * \c ext/encryptors/nop_encrypt is a simple template that passes through data unchanged, + * and is a reasonable starting point. \c ext/encryptors/rotn_encrypt is an encryptor + * implementing a simple (insecure) rotation cipher meant for testing. See @ref + * encryption "the encryptors page" for further information. * * Applications register their implementation with WiredTiger by calling * WT_CONNECTION::add_encryptor. @@ -4071,18 +4073,19 @@ struct __wt_encryptor { /*! * Callback to encrypt a chunk of data. * - * WT_ENCRYPTOR::encrypt takes a source buffer and a destination - * buffer. The callback encrypts the source buffer (plain text) - * into the destination buffer. + * WT_ENCRYPTOR::encrypt takes a source buffer and a destination buffer. The + * callback encrypts the source buffer (plain text) into the destination buffer. * - * On entry, \c src will point to memory, with the length of the memory - * in \c src_len. After successful completion, the callback should - * return \c 0 and set \c result_lenp to the number of bytes required - * for the encrypted representation. + * On entry, \c src will point to a block of memory to encrypt, with the length of + * the block in \c src_len. * - * On entry, \c dst points to the destination buffer with a length - * of \c dst_len. The destination buffer will be at least src_len - * plus the size returned by that WT_ENCRYPT::sizing. + * On entry, \c dst points to the destination buffer with a length of \c dst_len. + * The destination buffer will be at least src_len plus the size returned by that + * WT_ENCRYPT::sizing. + * + * After successful completion, the callback should return \c 0 and set \c + * result_lenp to the number of bytes required for the encrypted representation, + * which should be less than or equal to \c dst_len. * * This callback cannot be NULL. * @@ -4103,20 +4106,20 @@ struct __wt_encryptor { /*! * Callback to decrypt a chunk of data. * - * WT_ENCRYPTOR::decrypt takes a source buffer and a destination - * buffer. The contents are switched from \c encrypt: the - * source buffer is the encrypted value, and the destination buffer is - * sized to be the original size. If the callback successfully - * decrypts the source buffer to the destination buffer, it returns - * 0. If an error occurs, it returns an errno or WiredTiger error code. + * WT_ENCRYPTOR::decrypt takes a source buffer and a destination buffer. The + * contents are switched from \c encrypt: the source buffer is the encrypted + * value, and the destination buffer is sized to be the original size of the + * decrypted data. If the callback successfully decrypts the source buffer to the + * destination buffer, it returns 0. If an error occurs, it returns an errno or + * WiredTiger error code. * - * On entry, \c src will point to memory, with the length of the memory - * in \c src_len. After successful completion, the callback should - * return \c 0 and set \c result_lenp to the number of bytes required - * for the decrypted representation. + * On entry, \c src will point to memory, with the length of the memory in \c + * src_len. After successful completion, the callback should return \c 0 and set + * \c result_lenp to the number of bytes required for the decrypted + * representation. * - * If the \c dst buffer is not big enough to hold the decrypted - * data, the callback should return an error. + * If the \c dst buffer is not big enough to hold the decrypted data, the callback + * should return an error. * * This callback cannot be NULL. * @@ -4137,24 +4140,26 @@ struct __wt_encryptor { /*! * Callback to size a destination buffer for encryption. * - * WT_ENCRYPTOR::sizing is an callback that returns the number - * of additional bytes that is needed when encrypting a - * text buffer. This is always necessary, since encryptors - * typically generate encrypted text that is larger than the - * plain text input. Without such a call, WiredTiger would - * have no way to know the worst case for the encrypted buffer size. - * The WiredTiger encryption infrastructure assumes that - * buffer sizing is not dependent on the number of bytes - * of input, that there is a one to one relationship in number - * of bytes needed between input and output. + * WT_ENCRYPTOR::sizing is an callback that returns the number of additional bytes + * that is needed when encrypting a data block. This is always necessary, since + * encryptors should always generate some sort of cryptographic checksum as well + * as the ciphertext. Without such a call, WiredTiger would have no way to know + * the worst case for the encrypted buffer size. + * + * The WiredTiger encryption infrastructure assumes that buffer sizing is not + * dependent on the number of bytes of input, that there is a one-to-one + * relationship in number of bytes needed between input and output. This means + * that if the encryption uses a block cipher in such a way that the input size + * needs to be padded to the cipher block size, the sizing method should return + * the worst case to ensure enough space is available. * * This callback cannot be NULL. * - * The callback should set \c expansion_constantp to the additional - * number of bytes needed. + * The callback should set \c expansion_constantp to the additional number of + * bytes needed. * - * @param[out] expansion_constantp the additional number of bytes needed - * when encrypting. + * @param[out] expansion_constantp the additional number of bytes needed when + * encrypting. * @returns zero for success, non-zero to indicate an error. * * @snippet nop_encrypt.c WT_ENCRYPTOR sizing @@ -4163,17 +4168,36 @@ struct __wt_encryptor { size_t *expansion_constantp); /*! - * If non-NULL, this callback is called to customize the encryptor. - * The customize function is called whenever a keyid is used for the - * first time with this encryptor, whether it be in - * the ::wiredtiger_open call or the WT_SESSION::create - * call. This gives the algorithm an - * opportunity to retrieve and save keys in a customized encryptor. - * If the callback returns a non-NULL encryptor, that instance - * is used instead of this one for any callbacks. - * - * @param[in] encrypt_config the "encryption" portion of the - * configuration from the wiredtiger_open or WT_SESSION::create call + * If non-NULL, this callback is called to load keys into the encryptor. (That + * is, "customize" it for a given key.) The customize function is called whenever + * a new keyid is used for the first time with this encryptor, whether it be in + * the ::wiredtiger_open call or the WT_SESSION::create call. This should create a + * new encryptor instance and insert the requested key in it. + * + * The key may be specified either via \c keyid or \c secretkey in the \c + * encrypt_config parameter. In the former case, the encryptor should look up the + * requested key ID with whatever key management service is in use and install it + * in the new encryptor. In the latter case, the encryptor should save the + * provided secret key (or some transformation of it) in the new + * encryptor. Further encryption with the same \c keyid will use this new + * encryptor instance. (In the case of \c secretkey, only one key can be + * configured, for the system encryption, and the new encryptor will be used for + * all encryption involving it.) See @ref encryption for more information. + * + * This callback may return NULL as the new encryptor, in which case the original + * encryptor will be used for further operations on the selected key. Unless this + * happens, the original encryptor structure created during extension + * initialization will never be used for encryption or decryption. + * + * This callback may itself be NULL, in which case it is not called, but in that + * case there is no way to configure a key. This may be suitable for an + * environment where a key management service returns a single key under a + * well-known name that can be compiled in, but in a more general environment is + * not a useful approach. One should of course never compile in actual keys! + * + * @param[in] encrypt_config the "encryption" portion of the configuration from + * the wiredtiger_open or WT_SESSION::create call, containing the \c keyid or + * \c secretkey setting. * @param[out] customp the new modified encryptor, or NULL. * @returns zero for success, non-zero to indicate an error. */ @@ -4181,13 +4205,12 @@ struct __wt_encryptor { WT_CONFIG_ARG *encrypt_config, WT_ENCRYPTOR **customp); /*! - * If non-NULL, a callback performed when the database is closed. - * It is called for each encryptor that was added using - * WT_CONNECTION::add_encryptor or returned by the - * WT_ENCRYPTOR::customize callback. + * If non-NULL, a callback performed when the database is closed. It is called for + * each encryptor that was added using WT_CONNECTION::add_encryptor or returned by + * the WT_ENCRYPTOR::customize callback. * - * The WT_ENCRYPTOR::terminate callback is intended to allow cleanup, - * the handle will not be subsequently accessed by WiredTiger. + * The WT_ENCRYPTOR::terminate callback is intended to allow cleanup; the handle + * will not be subsequently accessed by WiredTiger. * * @snippet nop_encrypt.c WT_ENCRYPTOR terminate */ @@ -4281,7 +4304,7 @@ typedef enum { */ #define WT_FS_DURABLE 0x0 #else -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_FS_OPEN_ACCESS_RAND 0x01u #define WT_FS_OPEN_ACCESS_SEQ 0x02u #define WT_FS_OPEN_CREATE 0x04u @@ -4290,11 +4313,11 @@ typedef enum { #define WT_FS_OPEN_EXCLUSIVE 0x20u #define WT_FS_OPEN_FIXED 0x40u /* Path not home relative (internal) */ #define WT_FS_OPEN_READONLY 0x80u -/* AUTOMATIC FLAG VALUE GENERATION STOP */ +/* AUTOMATIC FLAG VALUE GENERATION STOP 32 */ -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_FS_DURABLE 0x1u -/* AUTOMATIC FLAG VALUE GENERATION STOP */ +/* AUTOMATIC FLAG VALUE GENERATION STOP 32 */ #endif /*! @@ -4765,6 +4788,17 @@ struct __wt_file_handle { */ struct __wt_storage_source { /*! + * A reference is added to the storage source. The reference is released by a + * call to WT_STORAGE_SOURCE::terminate. A reference is added as a side effect + * of calling WT_CONNECTION::get_storage_source. + * + * @errors + * + * @param storage_source the WT_STORAGE_SOURCE + */ + int (*ss_add_reference)(WT_STORAGE_SOURCE *storage_source); + + /*! * Create a customized file system to access the storage source * objects. * @@ -4849,8 +4883,11 @@ struct __wt_storage_source { const char *config); /*! - * A callback performed when the storage source is closed and will no - * longer be accessed by the WiredTiger database. + * A callback performed when the storage source or reference is closed + * and will no longer be used. The initial creation of the storage source + * counts as a reference, and each call to WT_STORAGE_SOURCE::add_reference + * increase the number of references. When all references are released, the + * storage source and any resources associated with it are released. * * This method is not required and should be set to NULL when not * required by the storage source implementation. @@ -4973,1145 +5010,1158 @@ extern int wiredtiger_extension_terminate(WT_CONNECTION *connection); #define WT_STAT_CONN_LSM_WORK_QUEUE_MANAGER 1001 /*! LSM: rows merged in an LSM tree */ #define WT_STAT_CONN_LSM_ROWS_MERGED 1002 +/*! LSM: sleep for LSM checkpoint throttle */ +#define WT_STAT_CONN_LSM_CHECKPOINT_THROTTLE 1003 +/*! LSM: sleep for LSM merge throttle */ +#define WT_STAT_CONN_LSM_MERGE_THROTTLE 1004 /*! LSM: switch work units currently queued */ -#define WT_STAT_CONN_LSM_WORK_QUEUE_SWITCH 1003 +#define WT_STAT_CONN_LSM_WORK_QUEUE_SWITCH 1005 /*! LSM: tree maintenance operations discarded */ -#define WT_STAT_CONN_LSM_WORK_UNITS_DISCARDED 1004 +#define WT_STAT_CONN_LSM_WORK_UNITS_DISCARDED 1006 /*! LSM: tree maintenance operations executed */ -#define WT_STAT_CONN_LSM_WORK_UNITS_DONE 1005 +#define WT_STAT_CONN_LSM_WORK_UNITS_DONE 1007 /*! LSM: tree maintenance operations scheduled */ -#define WT_STAT_CONN_LSM_WORK_UNITS_CREATED 1006 +#define WT_STAT_CONN_LSM_WORK_UNITS_CREATED 1008 /*! LSM: tree queue hit maximum */ -#define WT_STAT_CONN_LSM_WORK_QUEUE_MAX 1007 +#define WT_STAT_CONN_LSM_WORK_QUEUE_MAX 1009 /*! block-manager: blocks pre-loaded */ -#define WT_STAT_CONN_BLOCK_PRELOAD 1008 +#define WT_STAT_CONN_BLOCK_PRELOAD 1010 /*! block-manager: blocks read */ -#define WT_STAT_CONN_BLOCK_READ 1009 +#define WT_STAT_CONN_BLOCK_READ 1011 /*! block-manager: blocks written */ -#define WT_STAT_CONN_BLOCK_WRITE 1010 +#define WT_STAT_CONN_BLOCK_WRITE 1012 /*! block-manager: bytes read */ -#define WT_STAT_CONN_BLOCK_BYTE_READ 1011 +#define WT_STAT_CONN_BLOCK_BYTE_READ 1013 /*! block-manager: bytes read via memory map API */ -#define WT_STAT_CONN_BLOCK_BYTE_READ_MMAP 1012 +#define WT_STAT_CONN_BLOCK_BYTE_READ_MMAP 1014 /*! block-manager: bytes read via system call API */ -#define WT_STAT_CONN_BLOCK_BYTE_READ_SYSCALL 1013 +#define WT_STAT_CONN_BLOCK_BYTE_READ_SYSCALL 1015 /*! block-manager: bytes written */ -#define WT_STAT_CONN_BLOCK_BYTE_WRITE 1014 +#define WT_STAT_CONN_BLOCK_BYTE_WRITE 1016 /*! block-manager: bytes written for checkpoint */ -#define WT_STAT_CONN_BLOCK_BYTE_WRITE_CHECKPOINT 1015 +#define WT_STAT_CONN_BLOCK_BYTE_WRITE_CHECKPOINT 1017 /*! block-manager: bytes written via memory map API */ -#define WT_STAT_CONN_BLOCK_BYTE_WRITE_MMAP 1016 +#define WT_STAT_CONN_BLOCK_BYTE_WRITE_MMAP 1018 /*! block-manager: bytes written via system call API */ -#define WT_STAT_CONN_BLOCK_BYTE_WRITE_SYSCALL 1017 +#define WT_STAT_CONN_BLOCK_BYTE_WRITE_SYSCALL 1019 /*! block-manager: mapped blocks read */ -#define WT_STAT_CONN_BLOCK_MAP_READ 1018 +#define WT_STAT_CONN_BLOCK_MAP_READ 1020 /*! block-manager: mapped bytes read */ -#define WT_STAT_CONN_BLOCK_BYTE_MAP_READ 1019 +#define WT_STAT_CONN_BLOCK_BYTE_MAP_READ 1021 /*! * block-manager: number of times the file was remapped because it * changed size via fallocate or truncate */ -#define WT_STAT_CONN_BLOCK_REMAP_FILE_RESIZE 1020 +#define WT_STAT_CONN_BLOCK_REMAP_FILE_RESIZE 1022 /*! block-manager: number of times the region was remapped via write */ -#define WT_STAT_CONN_BLOCK_REMAP_FILE_WRITE 1021 +#define WT_STAT_CONN_BLOCK_REMAP_FILE_WRITE 1023 /*! cache: application threads page read from disk to cache count */ -#define WT_STAT_CONN_CACHE_READ_APP_COUNT 1022 +#define WT_STAT_CONN_CACHE_READ_APP_COUNT 1024 /*! cache: application threads page read from disk to cache time (usecs) */ -#define WT_STAT_CONN_CACHE_READ_APP_TIME 1023 +#define WT_STAT_CONN_CACHE_READ_APP_TIME 1025 /*! cache: application threads page write from cache to disk count */ -#define WT_STAT_CONN_CACHE_WRITE_APP_COUNT 1024 +#define WT_STAT_CONN_CACHE_WRITE_APP_COUNT 1026 /*! cache: application threads page write from cache to disk time (usecs) */ -#define WT_STAT_CONN_CACHE_WRITE_APP_TIME 1025 +#define WT_STAT_CONN_CACHE_WRITE_APP_TIME 1027 /*! cache: bytes allocated for updates */ -#define WT_STAT_CONN_CACHE_BYTES_UPDATES 1026 +#define WT_STAT_CONN_CACHE_BYTES_UPDATES 1028 /*! cache: bytes belonging to page images in the cache */ -#define WT_STAT_CONN_CACHE_BYTES_IMAGE 1027 +#define WT_STAT_CONN_CACHE_BYTES_IMAGE 1029 /*! cache: bytes belonging to the history store table in the cache */ -#define WT_STAT_CONN_CACHE_BYTES_HS 1028 +#define WT_STAT_CONN_CACHE_BYTES_HS 1030 +/*! cache: bytes currently in the cache */ +#define WT_STAT_CONN_CACHE_BYTES_INUSE 1031 +/*! cache: bytes dirty in the cache cumulative */ +#define WT_STAT_CONN_CACHE_BYTES_DIRTY_TOTAL 1032 /*! cache: bytes not belonging to page images in the cache */ -#define WT_STAT_CONN_CACHE_BYTES_OTHER 1029 +#define WT_STAT_CONN_CACHE_BYTES_OTHER 1033 +/*! cache: bytes read into cache */ +#define WT_STAT_CONN_CACHE_BYTES_READ 1034 +/*! cache: bytes written from cache */ +#define WT_STAT_CONN_CACHE_BYTES_WRITE 1035 /*! cache: cache overflow score */ -#define WT_STAT_CONN_CACHE_LOOKASIDE_SCORE 1030 +#define WT_STAT_CONN_CACHE_LOOKASIDE_SCORE 1036 +/*! cache: checkpoint blocked page eviction */ +#define WT_STAT_CONN_CACHE_EVICTION_CHECKPOINT 1037 +/*! + * cache: checkpoint of history store file blocked non-history store page + * eviction + */ +#define WT_STAT_CONN_CACHE_EVICTION_BLOCKED_CHECKPOINT_HS 1038 /*! cache: eviction calls to get a page */ -#define WT_STAT_CONN_CACHE_EVICTION_GET_REF 1031 +#define WT_STAT_CONN_CACHE_EVICTION_GET_REF 1039 /*! cache: eviction calls to get a page found queue empty */ -#define WT_STAT_CONN_CACHE_EVICTION_GET_REF_EMPTY 1032 +#define WT_STAT_CONN_CACHE_EVICTION_GET_REF_EMPTY 1040 /*! cache: eviction calls to get a page found queue empty after locking */ -#define WT_STAT_CONN_CACHE_EVICTION_GET_REF_EMPTY2 1033 +#define WT_STAT_CONN_CACHE_EVICTION_GET_REF_EMPTY2 1041 /*! cache: eviction currently operating in aggressive mode */ -#define WT_STAT_CONN_CACHE_EVICTION_AGGRESSIVE_SET 1034 +#define WT_STAT_CONN_CACHE_EVICTION_AGGRESSIVE_SET 1042 /*! cache: eviction empty score */ -#define WT_STAT_CONN_CACHE_EVICTION_EMPTY_SCORE 1035 +#define WT_STAT_CONN_CACHE_EVICTION_EMPTY_SCORE 1043 /*! cache: eviction passes of a file */ -#define WT_STAT_CONN_CACHE_EVICTION_WALK_PASSES 1036 +#define WT_STAT_CONN_CACHE_EVICTION_WALK_PASSES 1044 /*! cache: eviction server candidate queue empty when topping up */ -#define WT_STAT_CONN_CACHE_EVICTION_QUEUE_EMPTY 1037 +#define WT_STAT_CONN_CACHE_EVICTION_QUEUE_EMPTY 1045 /*! cache: eviction server candidate queue not empty when topping up */ -#define WT_STAT_CONN_CACHE_EVICTION_QUEUE_NOT_EMPTY 1038 +#define WT_STAT_CONN_CACHE_EVICTION_QUEUE_NOT_EMPTY 1046 /*! cache: eviction server evicting pages */ -#define WT_STAT_CONN_CACHE_EVICTION_SERVER_EVICTING 1039 +#define WT_STAT_CONN_CACHE_EVICTION_SERVER_EVICTING 1047 /*! * cache: eviction server slept, because we did not make progress with * eviction */ -#define WT_STAT_CONN_CACHE_EVICTION_SERVER_SLEPT 1040 +#define WT_STAT_CONN_CACHE_EVICTION_SERVER_SLEPT 1048 /*! cache: eviction server unable to reach eviction goal */ -#define WT_STAT_CONN_CACHE_EVICTION_SLOW 1041 +#define WT_STAT_CONN_CACHE_EVICTION_SLOW 1049 /*! cache: eviction server waiting for a leaf page */ -#define WT_STAT_CONN_CACHE_EVICTION_WALK_LEAF_NOTFOUND 1042 +#define WT_STAT_CONN_CACHE_EVICTION_WALK_LEAF_NOTFOUND 1050 /*! cache: eviction state */ -#define WT_STAT_CONN_CACHE_EVICTION_STATE 1043 +#define WT_STAT_CONN_CACHE_EVICTION_STATE 1051 +/*! cache: eviction walk target pages histogram - 0-9 */ +#define WT_STAT_CONN_CACHE_EVICTION_TARGET_PAGE_LT10 1052 +/*! cache: eviction walk target pages histogram - 10-31 */ +#define WT_STAT_CONN_CACHE_EVICTION_TARGET_PAGE_LT32 1053 +/*! cache: eviction walk target pages histogram - 128 and higher */ +#define WT_STAT_CONN_CACHE_EVICTION_TARGET_PAGE_GE128 1054 +/*! cache: eviction walk target pages histogram - 32-63 */ +#define WT_STAT_CONN_CACHE_EVICTION_TARGET_PAGE_LT64 1055 +/*! cache: eviction walk target pages histogram - 64-128 */ +#define WT_STAT_CONN_CACHE_EVICTION_TARGET_PAGE_LT128 1056 +/*! + * cache: eviction walk target pages reduced due to history store cache + * pressure + */ +#define WT_STAT_CONN_CACHE_EVICTION_TARGET_PAGE_REDUCED 1057 /*! cache: eviction walk target strategy both clean and dirty pages */ -#define WT_STAT_CONN_CACHE_EVICTION_TARGET_STRATEGY_BOTH_CLEAN_AND_DIRTY 1044 +#define WT_STAT_CONN_CACHE_EVICTION_TARGET_STRATEGY_BOTH_CLEAN_AND_DIRTY 1058 /*! cache: eviction walk target strategy only clean pages */ -#define WT_STAT_CONN_CACHE_EVICTION_TARGET_STRATEGY_CLEAN 1045 +#define WT_STAT_CONN_CACHE_EVICTION_TARGET_STRATEGY_CLEAN 1059 /*! cache: eviction walk target strategy only dirty pages */ -#define WT_STAT_CONN_CACHE_EVICTION_TARGET_STRATEGY_DIRTY 1046 +#define WT_STAT_CONN_CACHE_EVICTION_TARGET_STRATEGY_DIRTY 1060 +/*! cache: eviction walks abandoned */ +#define WT_STAT_CONN_CACHE_EVICTION_WALKS_ABANDONED 1061 +/*! cache: eviction walks gave up because they restarted their walk twice */ +#define WT_STAT_CONN_CACHE_EVICTION_WALKS_STOPPED 1062 +/*! + * cache: eviction walks gave up because they saw too many pages and + * found no candidates + */ +#define WT_STAT_CONN_CACHE_EVICTION_WALKS_GAVE_UP_NO_TARGETS 1063 +/*! + * cache: eviction walks gave up because they saw too many pages and + * found too few candidates + */ +#define WT_STAT_CONN_CACHE_EVICTION_WALKS_GAVE_UP_RATIO 1064 +/*! cache: eviction walks reached end of tree */ +#define WT_STAT_CONN_CACHE_EVICTION_WALKS_ENDED 1065 +/*! cache: eviction walks restarted */ +#define WT_STAT_CONN_CACHE_EVICTION_WALK_RESTART 1066 +/*! cache: eviction walks started from root of tree */ +#define WT_STAT_CONN_CACHE_EVICTION_WALK_FROM_ROOT 1067 +/*! cache: eviction walks started from saved location in tree */ +#define WT_STAT_CONN_CACHE_EVICTION_WALK_SAVED_POS 1068 /*! cache: eviction worker thread active */ -#define WT_STAT_CONN_CACHE_EVICTION_ACTIVE_WORKERS 1047 +#define WT_STAT_CONN_CACHE_EVICTION_ACTIVE_WORKERS 1069 /*! cache: eviction worker thread created */ -#define WT_STAT_CONN_CACHE_EVICTION_WORKER_CREATED 1048 +#define WT_STAT_CONN_CACHE_EVICTION_WORKER_CREATED 1070 /*! cache: eviction worker thread evicting pages */ -#define WT_STAT_CONN_CACHE_EVICTION_WORKER_EVICTING 1049 +#define WT_STAT_CONN_CACHE_EVICTION_WORKER_EVICTING 1071 /*! cache: eviction worker thread removed */ -#define WT_STAT_CONN_CACHE_EVICTION_WORKER_REMOVED 1050 +#define WT_STAT_CONN_CACHE_EVICTION_WORKER_REMOVED 1072 /*! cache: eviction worker thread stable number */ -#define WT_STAT_CONN_CACHE_EVICTION_STABLE_STATE_WORKERS 1051 +#define WT_STAT_CONN_CACHE_EVICTION_STABLE_STATE_WORKERS 1073 /*! cache: files with active eviction walks */ -#define WT_STAT_CONN_CACHE_EVICTION_WALKS_ACTIVE 1052 +#define WT_STAT_CONN_CACHE_EVICTION_WALKS_ACTIVE 1074 /*! cache: files with new eviction walks started */ -#define WT_STAT_CONN_CACHE_EVICTION_WALKS_STARTED 1053 +#define WT_STAT_CONN_CACHE_EVICTION_WALKS_STARTED 1075 /*! cache: force re-tuning of eviction workers once in a while */ -#define WT_STAT_CONN_CACHE_EVICTION_FORCE_RETUNE 1054 +#define WT_STAT_CONN_CACHE_EVICTION_FORCE_RETUNE 1076 /*! * cache: forced eviction - history store pages failed to evict while * session has history store cursor open */ -#define WT_STAT_CONN_CACHE_EVICTION_FORCE_HS_FAIL 1055 +#define WT_STAT_CONN_CACHE_EVICTION_FORCE_HS_FAIL 1077 /*! * cache: forced eviction - history store pages selected while session * has history store cursor open */ -#define WT_STAT_CONN_CACHE_EVICTION_FORCE_HS 1056 +#define WT_STAT_CONN_CACHE_EVICTION_FORCE_HS 1078 /*! * cache: forced eviction - history store pages successfully evicted * while session has history store cursor open */ -#define WT_STAT_CONN_CACHE_EVICTION_FORCE_HS_SUCCESS 1057 +#define WT_STAT_CONN_CACHE_EVICTION_FORCE_HS_SUCCESS 1079 /*! cache: forced eviction - pages evicted that were clean count */ -#define WT_STAT_CONN_CACHE_EVICTION_FORCE_CLEAN 1058 +#define WT_STAT_CONN_CACHE_EVICTION_FORCE_CLEAN 1080 /*! cache: forced eviction - pages evicted that were clean time (usecs) */ -#define WT_STAT_CONN_CACHE_EVICTION_FORCE_CLEAN_TIME 1059 +#define WT_STAT_CONN_CACHE_EVICTION_FORCE_CLEAN_TIME 1081 /*! cache: forced eviction - pages evicted that were dirty count */ -#define WT_STAT_CONN_CACHE_EVICTION_FORCE_DIRTY 1060 +#define WT_STAT_CONN_CACHE_EVICTION_FORCE_DIRTY 1082 /*! cache: forced eviction - pages evicted that were dirty time (usecs) */ -#define WT_STAT_CONN_CACHE_EVICTION_FORCE_DIRTY_TIME 1061 +#define WT_STAT_CONN_CACHE_EVICTION_FORCE_DIRTY_TIME 1083 /*! * cache: forced eviction - pages selected because of too many deleted * items count */ -#define WT_STAT_CONN_CACHE_EVICTION_FORCE_DELETE 1062 +#define WT_STAT_CONN_CACHE_EVICTION_FORCE_DELETE 1084 /*! cache: forced eviction - pages selected count */ -#define WT_STAT_CONN_CACHE_EVICTION_FORCE 1063 +#define WT_STAT_CONN_CACHE_EVICTION_FORCE 1085 /*! cache: forced eviction - pages selected unable to be evicted count */ -#define WT_STAT_CONN_CACHE_EVICTION_FORCE_FAIL 1064 +#define WT_STAT_CONN_CACHE_EVICTION_FORCE_FAIL 1086 /*! cache: forced eviction - pages selected unable to be evicted time */ -#define WT_STAT_CONN_CACHE_EVICTION_FORCE_FAIL_TIME 1065 +#define WT_STAT_CONN_CACHE_EVICTION_FORCE_FAIL_TIME 1087 +/*! cache: hazard pointer blocked page eviction */ +#define WT_STAT_CONN_CACHE_EVICTION_HAZARD 1088 /*! cache: hazard pointer check calls */ -#define WT_STAT_CONN_CACHE_HAZARD_CHECKS 1066 +#define WT_STAT_CONN_CACHE_HAZARD_CHECKS 1089 /*! cache: hazard pointer check entries walked */ -#define WT_STAT_CONN_CACHE_HAZARD_WALKS 1067 +#define WT_STAT_CONN_CACHE_HAZARD_WALKS 1090 /*! cache: hazard pointer maximum array length */ -#define WT_STAT_CONN_CACHE_HAZARD_MAX 1068 +#define WT_STAT_CONN_CACHE_HAZARD_MAX 1091 /*! cache: history store score */ -#define WT_STAT_CONN_CACHE_HS_SCORE 1069 +#define WT_STAT_CONN_CACHE_HS_SCORE 1092 +/*! cache: history store table insert calls */ +#define WT_STAT_CONN_CACHE_HS_INSERT 1093 +/*! cache: history store table insert calls that returned restart */ +#define WT_STAT_CONN_CACHE_HS_INSERT_RESTART 1094 /*! cache: history store table max on-disk size */ -#define WT_STAT_CONN_CACHE_HS_ONDISK_MAX 1070 +#define WT_STAT_CONN_CACHE_HS_ONDISK_MAX 1095 /*! cache: history store table on-disk size */ -#define WT_STAT_CONN_CACHE_HS_ONDISK 1071 +#define WT_STAT_CONN_CACHE_HS_ONDISK 1096 +/*! + * cache: history store table out-of-order resolved updates that lose + * their durable timestamp + */ +#define WT_STAT_CONN_CACHE_HS_ORDER_LOSE_DURABLE_TIMESTAMP 1097 +/*! + * cache: history store table out-of-order updates that were fixed up by + * reinserting with the fixed timestamp + */ +#define WT_STAT_CONN_CACHE_HS_ORDER_REINSERT 1098 +/*! cache: history store table reads */ +#define WT_STAT_CONN_CACHE_HS_READ 1099 +/*! cache: history store table reads missed */ +#define WT_STAT_CONN_CACHE_HS_READ_MISS 1100 +/*! cache: history store table reads requiring squashed modifies */ +#define WT_STAT_CONN_CACHE_HS_READ_SQUASH 1101 +/*! + * cache: history store table truncation by rollback to stable to remove + * an unstable update + */ +#define WT_STAT_CONN_CACHE_HS_KEY_TRUNCATE_RTS_UNSTABLE 1102 +/*! + * cache: history store table truncation by rollback to stable to remove + * an update + */ +#define WT_STAT_CONN_CACHE_HS_KEY_TRUNCATE_RTS 1103 +/*! cache: history store table truncation to remove an update */ +#define WT_STAT_CONN_CACHE_HS_KEY_TRUNCATE 1104 +/*! + * cache: history store table truncation to remove range of updates due + * to key being removed from the data page during reconciliation + */ +#define WT_STAT_CONN_CACHE_HS_KEY_TRUNCATE_ONPAGE_REMOVAL 1105 +/*! + * cache: history store table truncation to remove range of updates due + * to out-of-order timestamp update on data page + */ +#define WT_STAT_CONN_CACHE_HS_ORDER_REMOVE 1106 +/*! cache: history store table writes requiring squashed modifies */ +#define WT_STAT_CONN_CACHE_HS_WRITE_SQUASH 1107 +/*! cache: in-memory page passed criteria to be split */ +#define WT_STAT_CONN_CACHE_INMEM_SPLITTABLE 1108 +/*! cache: in-memory page splits */ +#define WT_STAT_CONN_CACHE_INMEM_SPLIT 1109 +/*! cache: internal pages evicted */ +#define WT_STAT_CONN_CACHE_EVICTION_INTERNAL 1110 /*! cache: internal pages queued for eviction */ -#define WT_STAT_CONN_CACHE_EVICTION_INTERNAL_PAGES_QUEUED 1072 +#define WT_STAT_CONN_CACHE_EVICTION_INTERNAL_PAGES_QUEUED 1111 /*! cache: internal pages seen by eviction walk */ -#define WT_STAT_CONN_CACHE_EVICTION_INTERNAL_PAGES_SEEN 1073 +#define WT_STAT_CONN_CACHE_EVICTION_INTERNAL_PAGES_SEEN 1112 /*! cache: internal pages seen by eviction walk that are already queued */ -#define WT_STAT_CONN_CACHE_EVICTION_INTERNAL_PAGES_ALREADY_QUEUED 1074 +#define WT_STAT_CONN_CACHE_EVICTION_INTERNAL_PAGES_ALREADY_QUEUED 1113 +/*! cache: internal pages split during eviction */ +#define WT_STAT_CONN_CACHE_EVICTION_SPLIT_INTERNAL 1114 +/*! cache: leaf pages split during eviction */ +#define WT_STAT_CONN_CACHE_EVICTION_SPLIT_LEAF 1115 /*! cache: maximum bytes configured */ -#define WT_STAT_CONN_CACHE_BYTES_MAX 1075 +#define WT_STAT_CONN_CACHE_BYTES_MAX 1116 /*! cache: maximum page size at eviction */ -#define WT_STAT_CONN_CACHE_EVICTION_MAXIMUM_PAGE_SIZE 1076 +#define WT_STAT_CONN_CACHE_EVICTION_MAXIMUM_PAGE_SIZE 1117 +/*! cache: modified pages evicted */ +#define WT_STAT_CONN_CACHE_EVICTION_DIRTY 1118 /*! cache: modified pages evicted by application threads */ -#define WT_STAT_CONN_CACHE_EVICTION_APP_DIRTY 1077 +#define WT_STAT_CONN_CACHE_EVICTION_APP_DIRTY 1119 /*! cache: operations timed out waiting for space in cache */ -#define WT_STAT_CONN_CACHE_TIMED_OUT_OPS 1078 +#define WT_STAT_CONN_CACHE_TIMED_OUT_OPS 1120 +/*! cache: overflow pages read into cache */ +#define WT_STAT_CONN_CACHE_READ_OVERFLOW 1121 +/*! cache: page split during eviction deepened the tree */ +#define WT_STAT_CONN_CACHE_EVICTION_DEEPEN 1122 +/*! cache: page written requiring history store records */ +#define WT_STAT_CONN_CACHE_WRITE_HS 1123 /*! cache: pages currently held in the cache */ -#define WT_STAT_CONN_CACHE_PAGES_INUSE 1079 +#define WT_STAT_CONN_CACHE_PAGES_INUSE 1124 /*! cache: pages evicted by application threads */ -#define WT_STAT_CONN_CACHE_EVICTION_APP 1080 +#define WT_STAT_CONN_CACHE_EVICTION_APP 1125 /*! cache: pages evicted in parallel with checkpoint */ -#define WT_STAT_CONN_CACHE_EVICTION_PAGES_IN_PARALLEL_WITH_CHECKPOINT 1081 +#define WT_STAT_CONN_CACHE_EVICTION_PAGES_IN_PARALLEL_WITH_CHECKPOINT 1126 /*! cache: pages queued for eviction */ -#define WT_STAT_CONN_CACHE_EVICTION_PAGES_QUEUED 1082 +#define WT_STAT_CONN_CACHE_EVICTION_PAGES_QUEUED 1127 /*! cache: pages queued for eviction post lru sorting */ -#define WT_STAT_CONN_CACHE_EVICTION_PAGES_QUEUED_POST_LRU 1083 +#define WT_STAT_CONN_CACHE_EVICTION_PAGES_QUEUED_POST_LRU 1128 /*! cache: pages queued for urgent eviction */ -#define WT_STAT_CONN_CACHE_EVICTION_PAGES_QUEUED_URGENT 1084 +#define WT_STAT_CONN_CACHE_EVICTION_PAGES_QUEUED_URGENT 1129 /*! cache: pages queued for urgent eviction during walk */ -#define WT_STAT_CONN_CACHE_EVICTION_PAGES_QUEUED_OLDEST 1085 +#define WT_STAT_CONN_CACHE_EVICTION_PAGES_QUEUED_OLDEST 1130 /*! * cache: pages queued for urgent eviction from history store due to high * dirty content */ -#define WT_STAT_CONN_CACHE_EVICTION_PAGES_QUEUED_URGENT_HS_DIRTY 1086 +#define WT_STAT_CONN_CACHE_EVICTION_PAGES_QUEUED_URGENT_HS_DIRTY 1131 +/*! cache: pages read into cache */ +#define WT_STAT_CONN_CACHE_READ 1132 +/*! cache: pages read into cache after truncate */ +#define WT_STAT_CONN_CACHE_READ_DELETED 1133 +/*! cache: pages read into cache after truncate in prepare state */ +#define WT_STAT_CONN_CACHE_READ_DELETED_PREPARED 1134 +/*! cache: pages requested from the cache */ +#define WT_STAT_CONN_CACHE_PAGES_REQUESTED 1135 +/*! cache: pages seen by eviction walk */ +#define WT_STAT_CONN_CACHE_EVICTION_PAGES_SEEN 1136 /*! cache: pages seen by eviction walk that are already queued */ -#define WT_STAT_CONN_CACHE_EVICTION_PAGES_ALREADY_QUEUED 1087 +#define WT_STAT_CONN_CACHE_EVICTION_PAGES_ALREADY_QUEUED 1137 /*! cache: pages selected for eviction unable to be evicted */ -#define WT_STAT_CONN_CACHE_EVICTION_FAIL 1088 +#define WT_STAT_CONN_CACHE_EVICTION_FAIL 1138 /*! * cache: pages selected for eviction unable to be evicted as the parent * page has overflow items */ -#define WT_STAT_CONN_CACHE_EVICTION_FAIL_PARENT_HAS_OVERFLOW_ITEMS 1089 +#define WT_STAT_CONN_CACHE_EVICTION_FAIL_PARENT_HAS_OVERFLOW_ITEMS 1139 /*! * cache: pages selected for eviction unable to be evicted because of * active children on an internal page */ -#define WT_STAT_CONN_CACHE_EVICTION_FAIL_ACTIVE_CHILDREN_ON_AN_INTERNAL_PAGE 1090 +#define WT_STAT_CONN_CACHE_EVICTION_FAIL_ACTIVE_CHILDREN_ON_AN_INTERNAL_PAGE 1140 /*! * cache: pages selected for eviction unable to be evicted because of * failure in reconciliation */ -#define WT_STAT_CONN_CACHE_EVICTION_FAIL_IN_RECONCILIATION 1091 +#define WT_STAT_CONN_CACHE_EVICTION_FAIL_IN_RECONCILIATION 1141 +/*! + * cache: pages selected for eviction unable to be evicted because of + * race between checkpoint and out of order timestamps handling + */ +#define WT_STAT_CONN_CACHE_EVICTION_FAIL_CHECKPOINT_OUT_OF_ORDER_TS 1142 /*! cache: pages walked for eviction */ -#define WT_STAT_CONN_CACHE_EVICTION_WALK 1092 +#define WT_STAT_CONN_CACHE_EVICTION_WALK 1143 +/*! cache: pages written from cache */ +#define WT_STAT_CONN_CACHE_WRITE 1144 +/*! cache: pages written requiring in-memory restoration */ +#define WT_STAT_CONN_CACHE_WRITE_RESTORE 1145 /*! cache: percentage overhead */ -#define WT_STAT_CONN_CACHE_OVERHEAD 1093 +#define WT_STAT_CONN_CACHE_OVERHEAD 1146 /*! cache: tracked bytes belonging to internal pages in the cache */ -#define WT_STAT_CONN_CACHE_BYTES_INTERNAL 1094 +#define WT_STAT_CONN_CACHE_BYTES_INTERNAL 1147 /*! cache: tracked bytes belonging to leaf pages in the cache */ -#define WT_STAT_CONN_CACHE_BYTES_LEAF 1095 +#define WT_STAT_CONN_CACHE_BYTES_LEAF 1148 +/*! cache: tracked dirty bytes in the cache */ +#define WT_STAT_CONN_CACHE_BYTES_DIRTY 1149 /*! cache: tracked dirty pages in the cache */ -#define WT_STAT_CONN_CACHE_PAGES_DIRTY 1096 +#define WT_STAT_CONN_CACHE_PAGES_DIRTY 1150 +/*! cache: unmodified pages evicted */ +#define WT_STAT_CONN_CACHE_EVICTION_CLEAN 1151 /*! capacity: background fsync file handles considered */ -#define WT_STAT_CONN_FSYNC_ALL_FH_TOTAL 1097 +#define WT_STAT_CONN_FSYNC_ALL_FH_TOTAL 1152 /*! capacity: background fsync file handles synced */ -#define WT_STAT_CONN_FSYNC_ALL_FH 1098 +#define WT_STAT_CONN_FSYNC_ALL_FH 1153 /*! capacity: background fsync time (msecs) */ -#define WT_STAT_CONN_FSYNC_ALL_TIME 1099 +#define WT_STAT_CONN_FSYNC_ALL_TIME 1154 /*! capacity: bytes read */ -#define WT_STAT_CONN_CAPACITY_BYTES_READ 1100 +#define WT_STAT_CONN_CAPACITY_BYTES_READ 1155 /*! capacity: bytes written for checkpoint */ -#define WT_STAT_CONN_CAPACITY_BYTES_CKPT 1101 +#define WT_STAT_CONN_CAPACITY_BYTES_CKPT 1156 /*! capacity: bytes written for eviction */ -#define WT_STAT_CONN_CAPACITY_BYTES_EVICT 1102 +#define WT_STAT_CONN_CAPACITY_BYTES_EVICT 1157 /*! capacity: bytes written for log */ -#define WT_STAT_CONN_CAPACITY_BYTES_LOG 1103 +#define WT_STAT_CONN_CAPACITY_BYTES_LOG 1158 /*! capacity: bytes written total */ -#define WT_STAT_CONN_CAPACITY_BYTES_WRITTEN 1104 +#define WT_STAT_CONN_CAPACITY_BYTES_WRITTEN 1159 /*! capacity: threshold to call fsync */ -#define WT_STAT_CONN_CAPACITY_THRESHOLD 1105 +#define WT_STAT_CONN_CAPACITY_THRESHOLD 1160 /*! capacity: time waiting due to total capacity (usecs) */ -#define WT_STAT_CONN_CAPACITY_TIME_TOTAL 1106 +#define WT_STAT_CONN_CAPACITY_TIME_TOTAL 1161 /*! capacity: time waiting during checkpoint (usecs) */ -#define WT_STAT_CONN_CAPACITY_TIME_CKPT 1107 +#define WT_STAT_CONN_CAPACITY_TIME_CKPT 1162 /*! capacity: time waiting during eviction (usecs) */ -#define WT_STAT_CONN_CAPACITY_TIME_EVICT 1108 +#define WT_STAT_CONN_CAPACITY_TIME_EVICT 1163 /*! capacity: time waiting during logging (usecs) */ -#define WT_STAT_CONN_CAPACITY_TIME_LOG 1109 +#define WT_STAT_CONN_CAPACITY_TIME_LOG 1164 /*! capacity: time waiting during read (usecs) */ -#define WT_STAT_CONN_CAPACITY_TIME_READ 1110 +#define WT_STAT_CONN_CAPACITY_TIME_READ 1165 +/*! checkpoint-cleanup: pages added for eviction */ +#define WT_STAT_CONN_CC_PAGES_EVICT 1166 +/*! checkpoint-cleanup: pages removed */ +#define WT_STAT_CONN_CC_PAGES_REMOVED 1167 +/*! checkpoint-cleanup: pages skipped during tree walk */ +#define WT_STAT_CONN_CC_PAGES_WALK_SKIPPED 1168 +/*! checkpoint-cleanup: pages visited */ +#define WT_STAT_CONN_CC_PAGES_VISITED 1169 /*! connection: auto adjusting condition resets */ -#define WT_STAT_CONN_COND_AUTO_WAIT_RESET 1111 +#define WT_STAT_CONN_COND_AUTO_WAIT_RESET 1170 /*! connection: auto adjusting condition wait calls */ -#define WT_STAT_CONN_COND_AUTO_WAIT 1112 +#define WT_STAT_CONN_COND_AUTO_WAIT 1171 /*! * connection: auto adjusting condition wait raced to update timeout and * skipped updating */ -#define WT_STAT_CONN_COND_AUTO_WAIT_SKIPPED 1113 +#define WT_STAT_CONN_COND_AUTO_WAIT_SKIPPED 1172 /*! connection: detected system time went backwards */ -#define WT_STAT_CONN_TIME_TRAVEL 1114 +#define WT_STAT_CONN_TIME_TRAVEL 1173 /*! connection: files currently open */ -#define WT_STAT_CONN_FILE_OPEN 1115 +#define WT_STAT_CONN_FILE_OPEN 1174 /*! connection: hash bucket array size for data handles */ -#define WT_STAT_CONN_BUCKETS_DH 1116 +#define WT_STAT_CONN_BUCKETS_DH 1175 /*! connection: hash bucket array size general */ -#define WT_STAT_CONN_BUCKETS 1117 +#define WT_STAT_CONN_BUCKETS 1176 /*! connection: memory allocations */ -#define WT_STAT_CONN_MEMORY_ALLOCATION 1118 +#define WT_STAT_CONN_MEMORY_ALLOCATION 1177 /*! connection: memory frees */ -#define WT_STAT_CONN_MEMORY_FREE 1119 +#define WT_STAT_CONN_MEMORY_FREE 1178 /*! connection: memory re-allocations */ -#define WT_STAT_CONN_MEMORY_GROW 1120 +#define WT_STAT_CONN_MEMORY_GROW 1179 /*! connection: pthread mutex condition wait calls */ -#define WT_STAT_CONN_COND_WAIT 1121 +#define WT_STAT_CONN_COND_WAIT 1180 /*! connection: pthread mutex shared lock read-lock calls */ -#define WT_STAT_CONN_RWLOCK_READ 1122 +#define WT_STAT_CONN_RWLOCK_READ 1181 /*! connection: pthread mutex shared lock write-lock calls */ -#define WT_STAT_CONN_RWLOCK_WRITE 1123 +#define WT_STAT_CONN_RWLOCK_WRITE 1182 /*! connection: total fsync I/Os */ -#define WT_STAT_CONN_FSYNC_IO 1124 +#define WT_STAT_CONN_FSYNC_IO 1183 /*! connection: total read I/Os */ -#define WT_STAT_CONN_READ_IO 1125 +#define WT_STAT_CONN_READ_IO 1184 /*! connection: total write I/Os */ -#define WT_STAT_CONN_WRITE_IO 1126 +#define WT_STAT_CONN_WRITE_IO 1185 +/*! cursor: Total number of entries skipped by cursor next calls */ +#define WT_STAT_CONN_CURSOR_NEXT_SKIP_TOTAL 1186 +/*! cursor: Total number of entries skipped by cursor prev calls */ +#define WT_STAT_CONN_CURSOR_PREV_SKIP_TOTAL 1187 +/*! + * cursor: Total number of entries skipped to position the history store + * cursor + */ +#define WT_STAT_CONN_CURSOR_SKIP_HS_CUR_POSITION 1188 +/*! + * cursor: Total number of pages skipped without reading by cursor next + * calls + */ +#define WT_STAT_CONN_CURSOR_NEXT_SKIP_PAGE_COUNT 1189 +/*! + * cursor: Total number of pages skipped without reading by cursor prev + * calls + */ +#define WT_STAT_CONN_CURSOR_PREV_SKIP_PAGE_COUNT 1190 +/*! + * cursor: Total number of times a search near has exited due to prefix + * config + */ +#define WT_STAT_CONN_CURSOR_SEARCH_NEAR_PREFIX_FAST_PATHS 1191 /*! cursor: cached cursor count */ -#define WT_STAT_CONN_CURSOR_CACHED_COUNT 1127 +#define WT_STAT_CONN_CURSOR_CACHED_COUNT 1192 /*! cursor: cursor bulk loaded cursor insert calls */ -#define WT_STAT_CONN_CURSOR_INSERT_BULK 1128 +#define WT_STAT_CONN_CURSOR_INSERT_BULK 1193 /*! cursor: cursor close calls that result in cache */ -#define WT_STAT_CONN_CURSOR_CACHE 1129 +#define WT_STAT_CONN_CURSOR_CACHE 1194 /*! cursor: cursor create calls */ -#define WT_STAT_CONN_CURSOR_CREATE 1130 +#define WT_STAT_CONN_CURSOR_CREATE 1195 /*! cursor: cursor insert calls */ -#define WT_STAT_CONN_CURSOR_INSERT 1131 +#define WT_STAT_CONN_CURSOR_INSERT 1196 /*! cursor: cursor insert key and value bytes */ -#define WT_STAT_CONN_CURSOR_INSERT_BYTES 1132 +#define WT_STAT_CONN_CURSOR_INSERT_BYTES 1197 /*! cursor: cursor modify calls */ -#define WT_STAT_CONN_CURSOR_MODIFY 1133 +#define WT_STAT_CONN_CURSOR_MODIFY 1198 /*! cursor: cursor modify key and value bytes affected */ -#define WT_STAT_CONN_CURSOR_MODIFY_BYTES 1134 +#define WT_STAT_CONN_CURSOR_MODIFY_BYTES 1199 /*! cursor: cursor modify value bytes modified */ -#define WT_STAT_CONN_CURSOR_MODIFY_BYTES_TOUCH 1135 +#define WT_STAT_CONN_CURSOR_MODIFY_BYTES_TOUCH 1200 /*! cursor: cursor next calls */ -#define WT_STAT_CONN_CURSOR_NEXT 1136 +#define WT_STAT_CONN_CURSOR_NEXT 1201 +/*! + * cursor: cursor next calls that skip due to a globally visible history + * store tombstone + */ +#define WT_STAT_CONN_CURSOR_NEXT_HS_TOMBSTONE 1202 +/*! + * cursor: cursor next calls that skip greater than or equal to 100 + * entries + */ +#define WT_STAT_CONN_CURSOR_NEXT_SKIP_GE_100 1203 +/*! cursor: cursor next calls that skip less than 100 entries */ +#define WT_STAT_CONN_CURSOR_NEXT_SKIP_LT_100 1204 /*! cursor: cursor operation restarted */ -#define WT_STAT_CONN_CURSOR_RESTART 1137 +#define WT_STAT_CONN_CURSOR_RESTART 1205 /*! cursor: cursor prev calls */ -#define WT_STAT_CONN_CURSOR_PREV 1138 +#define WT_STAT_CONN_CURSOR_PREV 1206 +/*! + * cursor: cursor prev calls that skip due to a globally visible history + * store tombstone + */ +#define WT_STAT_CONN_CURSOR_PREV_HS_TOMBSTONE 1207 +/*! + * cursor: cursor prev calls that skip greater than or equal to 100 + * entries + */ +#define WT_STAT_CONN_CURSOR_PREV_SKIP_GE_100 1208 +/*! cursor: cursor prev calls that skip less than 100 entries */ +#define WT_STAT_CONN_CURSOR_PREV_SKIP_LT_100 1209 /*! cursor: cursor remove calls */ -#define WT_STAT_CONN_CURSOR_REMOVE 1139 +#define WT_STAT_CONN_CURSOR_REMOVE 1210 /*! cursor: cursor remove key bytes removed */ -#define WT_STAT_CONN_CURSOR_REMOVE_BYTES 1140 +#define WT_STAT_CONN_CURSOR_REMOVE_BYTES 1211 /*! cursor: cursor reserve calls */ -#define WT_STAT_CONN_CURSOR_RESERVE 1141 +#define WT_STAT_CONN_CURSOR_RESERVE 1212 /*! cursor: cursor reset calls */ -#define WT_STAT_CONN_CURSOR_RESET 1142 +#define WT_STAT_CONN_CURSOR_RESET 1213 /*! cursor: cursor search calls */ -#define WT_STAT_CONN_CURSOR_SEARCH 1143 +#define WT_STAT_CONN_CURSOR_SEARCH 1214 /*! cursor: cursor search history store calls */ -#define WT_STAT_CONN_CURSOR_SEARCH_HS 1144 +#define WT_STAT_CONN_CURSOR_SEARCH_HS 1215 /*! cursor: cursor search near calls */ -#define WT_STAT_CONN_CURSOR_SEARCH_NEAR 1145 +#define WT_STAT_CONN_CURSOR_SEARCH_NEAR 1216 /*! cursor: cursor sweep buckets */ -#define WT_STAT_CONN_CURSOR_SWEEP_BUCKETS 1146 +#define WT_STAT_CONN_CURSOR_SWEEP_BUCKETS 1217 /*! cursor: cursor sweep cursors closed */ -#define WT_STAT_CONN_CURSOR_SWEEP_CLOSED 1147 +#define WT_STAT_CONN_CURSOR_SWEEP_CLOSED 1218 /*! cursor: cursor sweep cursors examined */ -#define WT_STAT_CONN_CURSOR_SWEEP_EXAMINED 1148 +#define WT_STAT_CONN_CURSOR_SWEEP_EXAMINED 1219 /*! cursor: cursor sweeps */ -#define WT_STAT_CONN_CURSOR_SWEEP 1149 +#define WT_STAT_CONN_CURSOR_SWEEP 1220 /*! cursor: cursor truncate calls */ -#define WT_STAT_CONN_CURSOR_TRUNCATE 1150 +#define WT_STAT_CONN_CURSOR_TRUNCATE 1221 /*! cursor: cursor update calls */ -#define WT_STAT_CONN_CURSOR_UPDATE 1151 +#define WT_STAT_CONN_CURSOR_UPDATE 1222 /*! cursor: cursor update key and value bytes */ -#define WT_STAT_CONN_CURSOR_UPDATE_BYTES 1152 +#define WT_STAT_CONN_CURSOR_UPDATE_BYTES 1223 /*! cursor: cursor update value size change */ -#define WT_STAT_CONN_CURSOR_UPDATE_BYTES_CHANGED 1153 +#define WT_STAT_CONN_CURSOR_UPDATE_BYTES_CHANGED 1224 /*! cursor: cursors reused from cache */ -#define WT_STAT_CONN_CURSOR_REOPEN 1154 +#define WT_STAT_CONN_CURSOR_REOPEN 1225 +/*! cursor: open cursor count */ +#define WT_STAT_CONN_CURSOR_OPEN_COUNT 1226 /*! data-handle: connection data handle size */ -#define WT_STAT_CONN_DH_CONN_HANDLE_SIZE 1155 +#define WT_STAT_CONN_DH_CONN_HANDLE_SIZE 1227 /*! data-handle: connection data handles currently active */ -#define WT_STAT_CONN_DH_CONN_HANDLE_COUNT 1156 +#define WT_STAT_CONN_DH_CONN_HANDLE_COUNT 1228 /*! data-handle: connection sweep candidate became referenced */ -#define WT_STAT_CONN_DH_SWEEP_REF 1157 +#define WT_STAT_CONN_DH_SWEEP_REF 1229 /*! data-handle: connection sweep dhandles closed */ -#define WT_STAT_CONN_DH_SWEEP_CLOSE 1158 +#define WT_STAT_CONN_DH_SWEEP_CLOSE 1230 /*! data-handle: connection sweep dhandles removed from hash list */ -#define WT_STAT_CONN_DH_SWEEP_REMOVE 1159 +#define WT_STAT_CONN_DH_SWEEP_REMOVE 1231 /*! data-handle: connection sweep time-of-death sets */ -#define WT_STAT_CONN_DH_SWEEP_TOD 1160 +#define WT_STAT_CONN_DH_SWEEP_TOD 1232 /*! data-handle: connection sweeps */ -#define WT_STAT_CONN_DH_SWEEPS 1161 +#define WT_STAT_CONN_DH_SWEEPS 1233 /*! * data-handle: connection sweeps skipped due to checkpoint gathering * handles */ -#define WT_STAT_CONN_DH_SWEEP_SKIP_CKPT 1162 +#define WT_STAT_CONN_DH_SWEEP_SKIP_CKPT 1234 /*! data-handle: session dhandles swept */ -#define WT_STAT_CONN_DH_SESSION_HANDLES 1163 +#define WT_STAT_CONN_DH_SESSION_HANDLES 1235 /*! data-handle: session sweep attempts */ -#define WT_STAT_CONN_DH_SESSION_SWEEPS 1164 +#define WT_STAT_CONN_DH_SESSION_SWEEPS 1236 /*! lock: checkpoint lock acquisitions */ -#define WT_STAT_CONN_LOCK_CHECKPOINT_COUNT 1165 +#define WT_STAT_CONN_LOCK_CHECKPOINT_COUNT 1237 /*! lock: checkpoint lock application thread wait time (usecs) */ -#define WT_STAT_CONN_LOCK_CHECKPOINT_WAIT_APPLICATION 1166 +#define WT_STAT_CONN_LOCK_CHECKPOINT_WAIT_APPLICATION 1238 /*! lock: checkpoint lock internal thread wait time (usecs) */ -#define WT_STAT_CONN_LOCK_CHECKPOINT_WAIT_INTERNAL 1167 +#define WT_STAT_CONN_LOCK_CHECKPOINT_WAIT_INTERNAL 1239 /*! lock: dhandle lock application thread time waiting (usecs) */ -#define WT_STAT_CONN_LOCK_DHANDLE_WAIT_APPLICATION 1168 +#define WT_STAT_CONN_LOCK_DHANDLE_WAIT_APPLICATION 1240 /*! lock: dhandle lock internal thread time waiting (usecs) */ -#define WT_STAT_CONN_LOCK_DHANDLE_WAIT_INTERNAL 1169 +#define WT_STAT_CONN_LOCK_DHANDLE_WAIT_INTERNAL 1241 /*! lock: dhandle read lock acquisitions */ -#define WT_STAT_CONN_LOCK_DHANDLE_READ_COUNT 1170 +#define WT_STAT_CONN_LOCK_DHANDLE_READ_COUNT 1242 /*! lock: dhandle write lock acquisitions */ -#define WT_STAT_CONN_LOCK_DHANDLE_WRITE_COUNT 1171 +#define WT_STAT_CONN_LOCK_DHANDLE_WRITE_COUNT 1243 /*! * lock: durable timestamp queue lock application thread time waiting * (usecs) */ -#define WT_STAT_CONN_LOCK_DURABLE_TIMESTAMP_WAIT_APPLICATION 1172 +#define WT_STAT_CONN_LOCK_DURABLE_TIMESTAMP_WAIT_APPLICATION 1244 /*! * lock: durable timestamp queue lock internal thread time waiting * (usecs) */ -#define WT_STAT_CONN_LOCK_DURABLE_TIMESTAMP_WAIT_INTERNAL 1173 +#define WT_STAT_CONN_LOCK_DURABLE_TIMESTAMP_WAIT_INTERNAL 1245 /*! lock: durable timestamp queue read lock acquisitions */ -#define WT_STAT_CONN_LOCK_DURABLE_TIMESTAMP_READ_COUNT 1174 +#define WT_STAT_CONN_LOCK_DURABLE_TIMESTAMP_READ_COUNT 1246 /*! lock: durable timestamp queue write lock acquisitions */ -#define WT_STAT_CONN_LOCK_DURABLE_TIMESTAMP_WRITE_COUNT 1175 +#define WT_STAT_CONN_LOCK_DURABLE_TIMESTAMP_WRITE_COUNT 1247 /*! lock: metadata lock acquisitions */ -#define WT_STAT_CONN_LOCK_METADATA_COUNT 1176 +#define WT_STAT_CONN_LOCK_METADATA_COUNT 1248 /*! lock: metadata lock application thread wait time (usecs) */ -#define WT_STAT_CONN_LOCK_METADATA_WAIT_APPLICATION 1177 +#define WT_STAT_CONN_LOCK_METADATA_WAIT_APPLICATION 1249 /*! lock: metadata lock internal thread wait time (usecs) */ -#define WT_STAT_CONN_LOCK_METADATA_WAIT_INTERNAL 1178 +#define WT_STAT_CONN_LOCK_METADATA_WAIT_INTERNAL 1250 /*! * lock: read timestamp queue lock application thread time waiting * (usecs) */ -#define WT_STAT_CONN_LOCK_READ_TIMESTAMP_WAIT_APPLICATION 1179 +#define WT_STAT_CONN_LOCK_READ_TIMESTAMP_WAIT_APPLICATION 1251 /*! lock: read timestamp queue lock internal thread time waiting (usecs) */ -#define WT_STAT_CONN_LOCK_READ_TIMESTAMP_WAIT_INTERNAL 1180 +#define WT_STAT_CONN_LOCK_READ_TIMESTAMP_WAIT_INTERNAL 1252 /*! lock: read timestamp queue read lock acquisitions */ -#define WT_STAT_CONN_LOCK_READ_TIMESTAMP_READ_COUNT 1181 +#define WT_STAT_CONN_LOCK_READ_TIMESTAMP_READ_COUNT 1253 /*! lock: read timestamp queue write lock acquisitions */ -#define WT_STAT_CONN_LOCK_READ_TIMESTAMP_WRITE_COUNT 1182 +#define WT_STAT_CONN_LOCK_READ_TIMESTAMP_WRITE_COUNT 1254 /*! lock: schema lock acquisitions */ -#define WT_STAT_CONN_LOCK_SCHEMA_COUNT 1183 +#define WT_STAT_CONN_LOCK_SCHEMA_COUNT 1255 /*! lock: schema lock application thread wait time (usecs) */ -#define WT_STAT_CONN_LOCK_SCHEMA_WAIT_APPLICATION 1184 +#define WT_STAT_CONN_LOCK_SCHEMA_WAIT_APPLICATION 1256 /*! lock: schema lock internal thread wait time (usecs) */ -#define WT_STAT_CONN_LOCK_SCHEMA_WAIT_INTERNAL 1185 +#define WT_STAT_CONN_LOCK_SCHEMA_WAIT_INTERNAL 1257 /*! * lock: table lock application thread time waiting for the table lock * (usecs) */ -#define WT_STAT_CONN_LOCK_TABLE_WAIT_APPLICATION 1186 +#define WT_STAT_CONN_LOCK_TABLE_WAIT_APPLICATION 1258 /*! * lock: table lock internal thread time waiting for the table lock * (usecs) */ -#define WT_STAT_CONN_LOCK_TABLE_WAIT_INTERNAL 1187 +#define WT_STAT_CONN_LOCK_TABLE_WAIT_INTERNAL 1259 /*! lock: table read lock acquisitions */ -#define WT_STAT_CONN_LOCK_TABLE_READ_COUNT 1188 +#define WT_STAT_CONN_LOCK_TABLE_READ_COUNT 1260 /*! lock: table write lock acquisitions */ -#define WT_STAT_CONN_LOCK_TABLE_WRITE_COUNT 1189 +#define WT_STAT_CONN_LOCK_TABLE_WRITE_COUNT 1261 /*! lock: txn global lock application thread time waiting (usecs) */ -#define WT_STAT_CONN_LOCK_TXN_GLOBAL_WAIT_APPLICATION 1190 +#define WT_STAT_CONN_LOCK_TXN_GLOBAL_WAIT_APPLICATION 1262 /*! lock: txn global lock internal thread time waiting (usecs) */ -#define WT_STAT_CONN_LOCK_TXN_GLOBAL_WAIT_INTERNAL 1191 +#define WT_STAT_CONN_LOCK_TXN_GLOBAL_WAIT_INTERNAL 1263 /*! lock: txn global read lock acquisitions */ -#define WT_STAT_CONN_LOCK_TXN_GLOBAL_READ_COUNT 1192 +#define WT_STAT_CONN_LOCK_TXN_GLOBAL_READ_COUNT 1264 /*! lock: txn global write lock acquisitions */ -#define WT_STAT_CONN_LOCK_TXN_GLOBAL_WRITE_COUNT 1193 +#define WT_STAT_CONN_LOCK_TXN_GLOBAL_WRITE_COUNT 1265 /*! log: busy returns attempting to switch slots */ -#define WT_STAT_CONN_LOG_SLOT_SWITCH_BUSY 1194 +#define WT_STAT_CONN_LOG_SLOT_SWITCH_BUSY 1266 /*! log: force archive time sleeping (usecs) */ -#define WT_STAT_CONN_LOG_FORCE_ARCHIVE_SLEEP 1195 +#define WT_STAT_CONN_LOG_FORCE_ARCHIVE_SLEEP 1267 /*! log: log bytes of payload data */ -#define WT_STAT_CONN_LOG_BYTES_PAYLOAD 1196 +#define WT_STAT_CONN_LOG_BYTES_PAYLOAD 1268 /*! log: log bytes written */ -#define WT_STAT_CONN_LOG_BYTES_WRITTEN 1197 +#define WT_STAT_CONN_LOG_BYTES_WRITTEN 1269 /*! log: log files manually zero-filled */ -#define WT_STAT_CONN_LOG_ZERO_FILLS 1198 +#define WT_STAT_CONN_LOG_ZERO_FILLS 1270 /*! log: log flush operations */ -#define WT_STAT_CONN_LOG_FLUSH 1199 +#define WT_STAT_CONN_LOG_FLUSH 1271 /*! log: log force write operations */ -#define WT_STAT_CONN_LOG_FORCE_WRITE 1200 +#define WT_STAT_CONN_LOG_FORCE_WRITE 1272 /*! log: log force write operations skipped */ -#define WT_STAT_CONN_LOG_FORCE_WRITE_SKIP 1201 +#define WT_STAT_CONN_LOG_FORCE_WRITE_SKIP 1273 /*! log: log records compressed */ -#define WT_STAT_CONN_LOG_COMPRESS_WRITES 1202 +#define WT_STAT_CONN_LOG_COMPRESS_WRITES 1274 /*! log: log records not compressed */ -#define WT_STAT_CONN_LOG_COMPRESS_WRITE_FAILS 1203 +#define WT_STAT_CONN_LOG_COMPRESS_WRITE_FAILS 1275 /*! log: log records too small to compress */ -#define WT_STAT_CONN_LOG_COMPRESS_SMALL 1204 +#define WT_STAT_CONN_LOG_COMPRESS_SMALL 1276 /*! log: log release advances write LSN */ -#define WT_STAT_CONN_LOG_RELEASE_WRITE_LSN 1205 +#define WT_STAT_CONN_LOG_RELEASE_WRITE_LSN 1277 /*! log: log scan operations */ -#define WT_STAT_CONN_LOG_SCANS 1206 +#define WT_STAT_CONN_LOG_SCANS 1278 /*! log: log scan records requiring two reads */ -#define WT_STAT_CONN_LOG_SCAN_REREADS 1207 +#define WT_STAT_CONN_LOG_SCAN_REREADS 1279 /*! log: log server thread advances write LSN */ -#define WT_STAT_CONN_LOG_WRITE_LSN 1208 +#define WT_STAT_CONN_LOG_WRITE_LSN 1280 /*! log: log server thread write LSN walk skipped */ -#define WT_STAT_CONN_LOG_WRITE_LSN_SKIP 1209 +#define WT_STAT_CONN_LOG_WRITE_LSN_SKIP 1281 /*! log: log sync operations */ -#define WT_STAT_CONN_LOG_SYNC 1210 +#define WT_STAT_CONN_LOG_SYNC 1282 /*! log: log sync time duration (usecs) */ -#define WT_STAT_CONN_LOG_SYNC_DURATION 1211 +#define WT_STAT_CONN_LOG_SYNC_DURATION 1283 /*! log: log sync_dir operations */ -#define WT_STAT_CONN_LOG_SYNC_DIR 1212 +#define WT_STAT_CONN_LOG_SYNC_DIR 1284 /*! log: log sync_dir time duration (usecs) */ -#define WT_STAT_CONN_LOG_SYNC_DIR_DURATION 1213 +#define WT_STAT_CONN_LOG_SYNC_DIR_DURATION 1285 /*! log: log write operations */ -#define WT_STAT_CONN_LOG_WRITES 1214 +#define WT_STAT_CONN_LOG_WRITES 1286 /*! log: logging bytes consolidated */ -#define WT_STAT_CONN_LOG_SLOT_CONSOLIDATED 1215 +#define WT_STAT_CONN_LOG_SLOT_CONSOLIDATED 1287 /*! log: maximum log file size */ -#define WT_STAT_CONN_LOG_MAX_FILESIZE 1216 +#define WT_STAT_CONN_LOG_MAX_FILESIZE 1288 /*! log: number of pre-allocated log files to create */ -#define WT_STAT_CONN_LOG_PREALLOC_MAX 1217 +#define WT_STAT_CONN_LOG_PREALLOC_MAX 1289 /*! log: pre-allocated log files not ready and missed */ -#define WT_STAT_CONN_LOG_PREALLOC_MISSED 1218 +#define WT_STAT_CONN_LOG_PREALLOC_MISSED 1290 /*! log: pre-allocated log files prepared */ -#define WT_STAT_CONN_LOG_PREALLOC_FILES 1219 +#define WT_STAT_CONN_LOG_PREALLOC_FILES 1291 /*! log: pre-allocated log files used */ -#define WT_STAT_CONN_LOG_PREALLOC_USED 1220 +#define WT_STAT_CONN_LOG_PREALLOC_USED 1292 /*! log: records processed by log scan */ -#define WT_STAT_CONN_LOG_SCAN_RECORDS 1221 +#define WT_STAT_CONN_LOG_SCAN_RECORDS 1293 /*! log: slot close lost race */ -#define WT_STAT_CONN_LOG_SLOT_CLOSE_RACE 1222 +#define WT_STAT_CONN_LOG_SLOT_CLOSE_RACE 1294 /*! log: slot close unbuffered waits */ -#define WT_STAT_CONN_LOG_SLOT_CLOSE_UNBUF 1223 +#define WT_STAT_CONN_LOG_SLOT_CLOSE_UNBUF 1295 /*! log: slot closures */ -#define WT_STAT_CONN_LOG_SLOT_CLOSES 1224 +#define WT_STAT_CONN_LOG_SLOT_CLOSES 1296 /*! log: slot join atomic update races */ -#define WT_STAT_CONN_LOG_SLOT_RACES 1225 +#define WT_STAT_CONN_LOG_SLOT_RACES 1297 /*! log: slot join calls atomic updates raced */ -#define WT_STAT_CONN_LOG_SLOT_YIELD_RACE 1226 +#define WT_STAT_CONN_LOG_SLOT_YIELD_RACE 1298 /*! log: slot join calls did not yield */ -#define WT_STAT_CONN_LOG_SLOT_IMMEDIATE 1227 +#define WT_STAT_CONN_LOG_SLOT_IMMEDIATE 1299 /*! log: slot join calls found active slot closed */ -#define WT_STAT_CONN_LOG_SLOT_YIELD_CLOSE 1228 +#define WT_STAT_CONN_LOG_SLOT_YIELD_CLOSE 1300 /*! log: slot join calls slept */ -#define WT_STAT_CONN_LOG_SLOT_YIELD_SLEEP 1229 +#define WT_STAT_CONN_LOG_SLOT_YIELD_SLEEP 1301 /*! log: slot join calls yielded */ -#define WT_STAT_CONN_LOG_SLOT_YIELD 1230 +#define WT_STAT_CONN_LOG_SLOT_YIELD 1302 /*! log: slot join found active slot closed */ -#define WT_STAT_CONN_LOG_SLOT_ACTIVE_CLOSED 1231 +#define WT_STAT_CONN_LOG_SLOT_ACTIVE_CLOSED 1303 /*! log: slot joins yield time (usecs) */ -#define WT_STAT_CONN_LOG_SLOT_YIELD_DURATION 1232 +#define WT_STAT_CONN_LOG_SLOT_YIELD_DURATION 1304 /*! log: slot transitions unable to find free slot */ -#define WT_STAT_CONN_LOG_SLOT_NO_FREE_SLOTS 1233 +#define WT_STAT_CONN_LOG_SLOT_NO_FREE_SLOTS 1305 /*! log: slot unbuffered writes */ -#define WT_STAT_CONN_LOG_SLOT_UNBUFFERED 1234 +#define WT_STAT_CONN_LOG_SLOT_UNBUFFERED 1306 /*! log: total in-memory size of compressed records */ -#define WT_STAT_CONN_LOG_COMPRESS_MEM 1235 +#define WT_STAT_CONN_LOG_COMPRESS_MEM 1307 /*! log: total log buffer size */ -#define WT_STAT_CONN_LOG_BUFFER_SIZE 1236 +#define WT_STAT_CONN_LOG_BUFFER_SIZE 1308 /*! log: total size of compressed records */ -#define WT_STAT_CONN_LOG_COMPRESS_LEN 1237 +#define WT_STAT_CONN_LOG_COMPRESS_LEN 1309 /*! log: written slots coalesced */ -#define WT_STAT_CONN_LOG_SLOT_COALESCED 1238 +#define WT_STAT_CONN_LOG_SLOT_COALESCED 1310 /*! log: yields waiting for previous log file close */ -#define WT_STAT_CONN_LOG_CLOSE_YIELDS 1239 +#define WT_STAT_CONN_LOG_CLOSE_YIELDS 1311 /*! perf: file system read latency histogram (bucket 1) - 10-49ms */ -#define WT_STAT_CONN_PERF_HIST_FSREAD_LATENCY_LT50 1240 +#define WT_STAT_CONN_PERF_HIST_FSREAD_LATENCY_LT50 1312 /*! perf: file system read latency histogram (bucket 2) - 50-99ms */ -#define WT_STAT_CONN_PERF_HIST_FSREAD_LATENCY_LT100 1241 +#define WT_STAT_CONN_PERF_HIST_FSREAD_LATENCY_LT100 1313 /*! perf: file system read latency histogram (bucket 3) - 100-249ms */ -#define WT_STAT_CONN_PERF_HIST_FSREAD_LATENCY_LT250 1242 +#define WT_STAT_CONN_PERF_HIST_FSREAD_LATENCY_LT250 1314 /*! perf: file system read latency histogram (bucket 4) - 250-499ms */ -#define WT_STAT_CONN_PERF_HIST_FSREAD_LATENCY_LT500 1243 +#define WT_STAT_CONN_PERF_HIST_FSREAD_LATENCY_LT500 1315 /*! perf: file system read latency histogram (bucket 5) - 500-999ms */ -#define WT_STAT_CONN_PERF_HIST_FSREAD_LATENCY_LT1000 1244 +#define WT_STAT_CONN_PERF_HIST_FSREAD_LATENCY_LT1000 1316 /*! perf: file system read latency histogram (bucket 6) - 1000ms+ */ -#define WT_STAT_CONN_PERF_HIST_FSREAD_LATENCY_GT1000 1245 +#define WT_STAT_CONN_PERF_HIST_FSREAD_LATENCY_GT1000 1317 /*! perf: file system write latency histogram (bucket 1) - 10-49ms */ -#define WT_STAT_CONN_PERF_HIST_FSWRITE_LATENCY_LT50 1246 +#define WT_STAT_CONN_PERF_HIST_FSWRITE_LATENCY_LT50 1318 /*! perf: file system write latency histogram (bucket 2) - 50-99ms */ -#define WT_STAT_CONN_PERF_HIST_FSWRITE_LATENCY_LT100 1247 +#define WT_STAT_CONN_PERF_HIST_FSWRITE_LATENCY_LT100 1319 /*! perf: file system write latency histogram (bucket 3) - 100-249ms */ -#define WT_STAT_CONN_PERF_HIST_FSWRITE_LATENCY_LT250 1248 +#define WT_STAT_CONN_PERF_HIST_FSWRITE_LATENCY_LT250 1320 /*! perf: file system write latency histogram (bucket 4) - 250-499ms */ -#define WT_STAT_CONN_PERF_HIST_FSWRITE_LATENCY_LT500 1249 +#define WT_STAT_CONN_PERF_HIST_FSWRITE_LATENCY_LT500 1321 /*! perf: file system write latency histogram (bucket 5) - 500-999ms */ -#define WT_STAT_CONN_PERF_HIST_FSWRITE_LATENCY_LT1000 1250 +#define WT_STAT_CONN_PERF_HIST_FSWRITE_LATENCY_LT1000 1322 /*! perf: file system write latency histogram (bucket 6) - 1000ms+ */ -#define WT_STAT_CONN_PERF_HIST_FSWRITE_LATENCY_GT1000 1251 +#define WT_STAT_CONN_PERF_HIST_FSWRITE_LATENCY_GT1000 1323 /*! perf: operation read latency histogram (bucket 1) - 100-249us */ -#define WT_STAT_CONN_PERF_HIST_OPREAD_LATENCY_LT250 1252 +#define WT_STAT_CONN_PERF_HIST_OPREAD_LATENCY_LT250 1324 /*! perf: operation read latency histogram (bucket 2) - 250-499us */ -#define WT_STAT_CONN_PERF_HIST_OPREAD_LATENCY_LT500 1253 +#define WT_STAT_CONN_PERF_HIST_OPREAD_LATENCY_LT500 1325 /*! perf: operation read latency histogram (bucket 3) - 500-999us */ -#define WT_STAT_CONN_PERF_HIST_OPREAD_LATENCY_LT1000 1254 +#define WT_STAT_CONN_PERF_HIST_OPREAD_LATENCY_LT1000 1326 /*! perf: operation read latency histogram (bucket 4) - 1000-9999us */ -#define WT_STAT_CONN_PERF_HIST_OPREAD_LATENCY_LT10000 1255 +#define WT_STAT_CONN_PERF_HIST_OPREAD_LATENCY_LT10000 1327 /*! perf: operation read latency histogram (bucket 5) - 10000us+ */ -#define WT_STAT_CONN_PERF_HIST_OPREAD_LATENCY_GT10000 1256 +#define WT_STAT_CONN_PERF_HIST_OPREAD_LATENCY_GT10000 1328 /*! perf: operation write latency histogram (bucket 1) - 100-249us */ -#define WT_STAT_CONN_PERF_HIST_OPWRITE_LATENCY_LT250 1257 +#define WT_STAT_CONN_PERF_HIST_OPWRITE_LATENCY_LT250 1329 /*! perf: operation write latency histogram (bucket 2) - 250-499us */ -#define WT_STAT_CONN_PERF_HIST_OPWRITE_LATENCY_LT500 1258 +#define WT_STAT_CONN_PERF_HIST_OPWRITE_LATENCY_LT500 1330 /*! perf: operation write latency histogram (bucket 3) - 500-999us */ -#define WT_STAT_CONN_PERF_HIST_OPWRITE_LATENCY_LT1000 1259 +#define WT_STAT_CONN_PERF_HIST_OPWRITE_LATENCY_LT1000 1331 /*! perf: operation write latency histogram (bucket 4) - 1000-9999us */ -#define WT_STAT_CONN_PERF_HIST_OPWRITE_LATENCY_LT10000 1260 +#define WT_STAT_CONN_PERF_HIST_OPWRITE_LATENCY_LT10000 1332 /*! perf: operation write latency histogram (bucket 5) - 10000us+ */ -#define WT_STAT_CONN_PERF_HIST_OPWRITE_LATENCY_GT10000 1261 +#define WT_STAT_CONN_PERF_HIST_OPWRITE_LATENCY_GT10000 1333 +/*! reconciliation: approximate byte size of timestamps in pages written */ +#define WT_STAT_CONN_REC_TIME_WINDOW_BYTES_TS 1334 +/*! + * reconciliation: approximate byte size of transaction IDs in pages + * written + */ +#define WT_STAT_CONN_REC_TIME_WINDOW_BYTES_TXN 1335 +/*! reconciliation: fast-path pages deleted */ +#define WT_STAT_CONN_REC_PAGE_DELETE_FAST 1336 /*! reconciliation: internal-page overflow keys */ -#define WT_STAT_CONN_REC_OVERFLOW_KEY_INTERNAL 1262 +#define WT_STAT_CONN_REC_OVERFLOW_KEY_INTERNAL 1337 /*! reconciliation: leaf-page overflow keys */ -#define WT_STAT_CONN_REC_OVERFLOW_KEY_LEAF 1263 +#define WT_STAT_CONN_REC_OVERFLOW_KEY_LEAF 1338 /*! reconciliation: maximum seconds spent in a reconciliation call */ -#define WT_STAT_CONN_REC_MAXIMUM_SECONDS 1264 +#define WT_STAT_CONN_REC_MAXIMUM_SECONDS 1339 +/*! reconciliation: page reconciliation calls */ +#define WT_STAT_CONN_REC_PAGES 1340 +/*! reconciliation: page reconciliation calls for eviction */ +#define WT_STAT_CONN_REC_PAGES_EVICTION 1341 /*! * reconciliation: page reconciliation calls that resulted in values with * prepared transaction metadata */ -#define WT_STAT_CONN_REC_PAGES_WITH_PREPARE 1265 +#define WT_STAT_CONN_REC_PAGES_WITH_PREPARE 1342 /*! * reconciliation: page reconciliation calls that resulted in values with * timestamps */ -#define WT_STAT_CONN_REC_PAGES_WITH_TS 1266 +#define WT_STAT_CONN_REC_PAGES_WITH_TS 1343 /*! * reconciliation: page reconciliation calls that resulted in values with * transaction ids */ -#define WT_STAT_CONN_REC_PAGES_WITH_TXN 1267 +#define WT_STAT_CONN_REC_PAGES_WITH_TXN 1344 +/*! reconciliation: pages deleted */ +#define WT_STAT_CONN_REC_PAGE_DELETE 1345 +/*! + * reconciliation: pages written including an aggregated newest start + * durable timestamp + */ +#define WT_STAT_CONN_REC_TIME_AGGR_NEWEST_START_DURABLE_TS 1346 +/*! + * reconciliation: pages written including an aggregated newest stop + * durable timestamp + */ +#define WT_STAT_CONN_REC_TIME_AGGR_NEWEST_STOP_DURABLE_TS 1347 +/*! + * reconciliation: pages written including an aggregated newest stop + * timestamp + */ +#define WT_STAT_CONN_REC_TIME_AGGR_NEWEST_STOP_TS 1348 +/*! + * reconciliation: pages written including an aggregated newest stop + * transaction ID + */ +#define WT_STAT_CONN_REC_TIME_AGGR_NEWEST_STOP_TXN 1349 +/*! + * reconciliation: pages written including an aggregated newest + * transaction ID + */ +#define WT_STAT_CONN_REC_TIME_AGGR_NEWEST_TXN 1350 +/*! + * reconciliation: pages written including an aggregated oldest start + * timestamp + */ +#define WT_STAT_CONN_REC_TIME_AGGR_OLDEST_START_TS 1351 +/*! reconciliation: pages written including an aggregated prepare */ +#define WT_STAT_CONN_REC_TIME_AGGR_PREPARED 1352 /*! reconciliation: pages written including at least one prepare state */ -#define WT_STAT_CONN_REC_TIME_WINDOW_PAGES_PREPARED 1268 +#define WT_STAT_CONN_REC_TIME_WINDOW_PAGES_PREPARED 1353 +/*! + * reconciliation: pages written including at least one start durable + * timestamp + */ +#define WT_STAT_CONN_REC_TIME_WINDOW_PAGES_DURABLE_START_TS 1354 /*! reconciliation: pages written including at least one start timestamp */ -#define WT_STAT_CONN_REC_TIME_WINDOW_PAGES_START_TS 1269 +#define WT_STAT_CONN_REC_TIME_WINDOW_PAGES_START_TS 1355 +/*! + * reconciliation: pages written including at least one start transaction + * ID + */ +#define WT_STAT_CONN_REC_TIME_WINDOW_PAGES_START_TXN 1356 +/*! + * reconciliation: pages written including at least one stop durable + * timestamp + */ +#define WT_STAT_CONN_REC_TIME_WINDOW_PAGES_DURABLE_STOP_TS 1357 +/*! reconciliation: pages written including at least one stop timestamp */ +#define WT_STAT_CONN_REC_TIME_WINDOW_PAGES_STOP_TS 1358 +/*! + * reconciliation: pages written including at least one stop transaction + * ID + */ +#define WT_STAT_CONN_REC_TIME_WINDOW_PAGES_STOP_TXN 1359 /*! reconciliation: records written including a prepare state */ -#define WT_STAT_CONN_REC_TIME_WINDOW_PREPARED 1270 +#define WT_STAT_CONN_REC_TIME_WINDOW_PREPARED 1360 +/*! reconciliation: records written including a start durable timestamp */ +#define WT_STAT_CONN_REC_TIME_WINDOW_DURABLE_START_TS 1361 +/*! reconciliation: records written including a start timestamp */ +#define WT_STAT_CONN_REC_TIME_WINDOW_START_TS 1362 +/*! reconciliation: records written including a start transaction ID */ +#define WT_STAT_CONN_REC_TIME_WINDOW_START_TXN 1363 +/*! reconciliation: records written including a stop durable timestamp */ +#define WT_STAT_CONN_REC_TIME_WINDOW_DURABLE_STOP_TS 1364 +/*! reconciliation: records written including a stop timestamp */ +#define WT_STAT_CONN_REC_TIME_WINDOW_STOP_TS 1365 +/*! reconciliation: records written including a stop transaction ID */ +#define WT_STAT_CONN_REC_TIME_WINDOW_STOP_TXN 1366 /*! reconciliation: split bytes currently awaiting free */ -#define WT_STAT_CONN_REC_SPLIT_STASHED_BYTES 1271 +#define WT_STAT_CONN_REC_SPLIT_STASHED_BYTES 1367 /*! reconciliation: split objects currently awaiting free */ -#define WT_STAT_CONN_REC_SPLIT_STASHED_OBJECTS 1272 +#define WT_STAT_CONN_REC_SPLIT_STASHED_OBJECTS 1368 /*! session: flush state races */ -#define WT_STAT_CONN_FLUSH_STATE_RACES 1273 -/*! session: flush_tier busy retries */ -#define WT_STAT_CONN_FLUSH_TIER_BUSY 1274 +#define WT_STAT_CONN_FLUSH_STATE_RACES 1369 /*! session: flush_tier operation calls */ -#define WT_STAT_CONN_FLUSH_TIER 1275 +#define WT_STAT_CONN_FLUSH_TIER 1370 /*! session: open session count */ -#define WT_STAT_CONN_SESSION_OPEN 1276 +#define WT_STAT_CONN_SESSION_OPEN 1371 /*! session: session query timestamp calls */ -#define WT_STAT_CONN_SESSION_QUERY_TS 1277 +#define WT_STAT_CONN_SESSION_QUERY_TS 1372 /*! session: table alter failed calls */ -#define WT_STAT_CONN_SESSION_TABLE_ALTER_FAIL 1278 +#define WT_STAT_CONN_SESSION_TABLE_ALTER_FAIL 1373 /*! session: table alter successful calls */ -#define WT_STAT_CONN_SESSION_TABLE_ALTER_SUCCESS 1279 +#define WT_STAT_CONN_SESSION_TABLE_ALTER_SUCCESS 1374 /*! session: table alter unchanged and skipped */ -#define WT_STAT_CONN_SESSION_TABLE_ALTER_SKIP 1280 +#define WT_STAT_CONN_SESSION_TABLE_ALTER_SKIP 1375 /*! session: table compact failed calls */ -#define WT_STAT_CONN_SESSION_TABLE_COMPACT_FAIL 1281 +#define WT_STAT_CONN_SESSION_TABLE_COMPACT_FAIL 1376 /*! session: table compact successful calls */ -#define WT_STAT_CONN_SESSION_TABLE_COMPACT_SUCCESS 1282 +#define WT_STAT_CONN_SESSION_TABLE_COMPACT_SUCCESS 1377 /*! session: table create failed calls */ -#define WT_STAT_CONN_SESSION_TABLE_CREATE_FAIL 1283 +#define WT_STAT_CONN_SESSION_TABLE_CREATE_FAIL 1378 /*! session: table create successful calls */ -#define WT_STAT_CONN_SESSION_TABLE_CREATE_SUCCESS 1284 +#define WT_STAT_CONN_SESSION_TABLE_CREATE_SUCCESS 1379 /*! session: table drop failed calls */ -#define WT_STAT_CONN_SESSION_TABLE_DROP_FAIL 1285 +#define WT_STAT_CONN_SESSION_TABLE_DROP_FAIL 1380 /*! session: table drop successful calls */ -#define WT_STAT_CONN_SESSION_TABLE_DROP_SUCCESS 1286 +#define WT_STAT_CONN_SESSION_TABLE_DROP_SUCCESS 1381 /*! session: table rename failed calls */ -#define WT_STAT_CONN_SESSION_TABLE_RENAME_FAIL 1287 +#define WT_STAT_CONN_SESSION_TABLE_RENAME_FAIL 1382 /*! session: table rename successful calls */ -#define WT_STAT_CONN_SESSION_TABLE_RENAME_SUCCESS 1288 +#define WT_STAT_CONN_SESSION_TABLE_RENAME_SUCCESS 1383 /*! session: table salvage failed calls */ -#define WT_STAT_CONN_SESSION_TABLE_SALVAGE_FAIL 1289 +#define WT_STAT_CONN_SESSION_TABLE_SALVAGE_FAIL 1384 /*! session: table salvage successful calls */ -#define WT_STAT_CONN_SESSION_TABLE_SALVAGE_SUCCESS 1290 +#define WT_STAT_CONN_SESSION_TABLE_SALVAGE_SUCCESS 1385 /*! session: table truncate failed calls */ -#define WT_STAT_CONN_SESSION_TABLE_TRUNCATE_FAIL 1291 +#define WT_STAT_CONN_SESSION_TABLE_TRUNCATE_FAIL 1386 /*! session: table truncate successful calls */ -#define WT_STAT_CONN_SESSION_TABLE_TRUNCATE_SUCCESS 1292 +#define WT_STAT_CONN_SESSION_TABLE_TRUNCATE_SUCCESS 1387 /*! session: table verify failed calls */ -#define WT_STAT_CONN_SESSION_TABLE_VERIFY_FAIL 1293 +#define WT_STAT_CONN_SESSION_TABLE_VERIFY_FAIL 1388 /*! session: table verify successful calls */ -#define WT_STAT_CONN_SESSION_TABLE_VERIFY_SUCCESS 1294 +#define WT_STAT_CONN_SESSION_TABLE_VERIFY_SUCCESS 1389 +/*! session: tiered operations dequeued and processed */ +#define WT_STAT_CONN_TIERED_WORK_UNITS_DEQUEUED 1390 +/*! session: tiered operations scheduled */ +#define WT_STAT_CONN_TIERED_WORK_UNITS_CREATED 1391 +/*! session: tiered storage local retention time (secs) */ +#define WT_STAT_CONN_TIERED_RETENTION 1392 +/*! session: tiered storage object size */ +#define WT_STAT_CONN_TIERED_OBJECT_SIZE 1393 /*! thread-state: active filesystem fsync calls */ -#define WT_STAT_CONN_THREAD_FSYNC_ACTIVE 1295 +#define WT_STAT_CONN_THREAD_FSYNC_ACTIVE 1394 /*! thread-state: active filesystem read calls */ -#define WT_STAT_CONN_THREAD_READ_ACTIVE 1296 +#define WT_STAT_CONN_THREAD_READ_ACTIVE 1395 /*! thread-state: active filesystem write calls */ -#define WT_STAT_CONN_THREAD_WRITE_ACTIVE 1297 +#define WT_STAT_CONN_THREAD_WRITE_ACTIVE 1396 /*! thread-yield: application thread time evicting (usecs) */ -#define WT_STAT_CONN_APPLICATION_EVICT_TIME 1298 +#define WT_STAT_CONN_APPLICATION_EVICT_TIME 1397 /*! thread-yield: application thread time waiting for cache (usecs) */ -#define WT_STAT_CONN_APPLICATION_CACHE_TIME 1299 +#define WT_STAT_CONN_APPLICATION_CACHE_TIME 1398 /*! * thread-yield: connection close blocked waiting for transaction state * stabilization */ -#define WT_STAT_CONN_TXN_RELEASE_BLOCKED 1300 +#define WT_STAT_CONN_TXN_RELEASE_BLOCKED 1399 /*! thread-yield: connection close yielded for lsm manager shutdown */ -#define WT_STAT_CONN_CONN_CLOSE_BLOCKED_LSM 1301 +#define WT_STAT_CONN_CONN_CLOSE_BLOCKED_LSM 1400 /*! thread-yield: data handle lock yielded */ -#define WT_STAT_CONN_DHANDLE_LOCK_BLOCKED 1302 +#define WT_STAT_CONN_DHANDLE_LOCK_BLOCKED 1401 /*! * thread-yield: get reference for page index and slot time sleeping * (usecs) */ -#define WT_STAT_CONN_PAGE_INDEX_SLOT_REF_BLOCKED 1303 +#define WT_STAT_CONN_PAGE_INDEX_SLOT_REF_BLOCKED 1402 /*! thread-yield: log server sync yielded for log write */ -#define WT_STAT_CONN_LOG_SERVER_SYNC_BLOCKED 1304 +#define WT_STAT_CONN_LOG_SERVER_SYNC_BLOCKED 1403 /*! thread-yield: page access yielded due to prepare state change */ -#define WT_STAT_CONN_PREPARED_TRANSITION_BLOCKED_PAGE 1305 +#define WT_STAT_CONN_PREPARED_TRANSITION_BLOCKED_PAGE 1404 /*! thread-yield: page acquire busy blocked */ -#define WT_STAT_CONN_PAGE_BUSY_BLOCKED 1306 +#define WT_STAT_CONN_PAGE_BUSY_BLOCKED 1405 /*! thread-yield: page acquire eviction blocked */ -#define WT_STAT_CONN_PAGE_FORCIBLE_EVICT_BLOCKED 1307 +#define WT_STAT_CONN_PAGE_FORCIBLE_EVICT_BLOCKED 1406 /*! thread-yield: page acquire locked blocked */ -#define WT_STAT_CONN_PAGE_LOCKED_BLOCKED 1308 +#define WT_STAT_CONN_PAGE_LOCKED_BLOCKED 1407 /*! thread-yield: page acquire read blocked */ -#define WT_STAT_CONN_PAGE_READ_BLOCKED 1309 +#define WT_STAT_CONN_PAGE_READ_BLOCKED 1408 /*! thread-yield: page acquire time sleeping (usecs) */ -#define WT_STAT_CONN_PAGE_SLEEP 1310 +#define WT_STAT_CONN_PAGE_SLEEP 1409 /*! * thread-yield: page delete rollback time sleeping for state change * (usecs) */ -#define WT_STAT_CONN_PAGE_DEL_ROLLBACK_BLOCKED 1311 +#define WT_STAT_CONN_PAGE_DEL_ROLLBACK_BLOCKED 1410 /*! thread-yield: page reconciliation yielded due to child modification */ -#define WT_STAT_CONN_CHILD_MODIFY_BLOCKED_PAGE 1312 +#define WT_STAT_CONN_CHILD_MODIFY_BLOCKED_PAGE 1411 /*! transaction: Number of prepared updates */ -#define WT_STAT_CONN_TXN_PREPARED_UPDATES 1313 +#define WT_STAT_CONN_TXN_PREPARED_UPDATES 1412 /*! transaction: Number of prepared updates committed */ -#define WT_STAT_CONN_TXN_PREPARED_UPDATES_COMMITTED 1314 +#define WT_STAT_CONN_TXN_PREPARED_UPDATES_COMMITTED 1413 /*! transaction: Number of prepared updates repeated on the same key */ -#define WT_STAT_CONN_TXN_PREPARED_UPDATES_KEY_REPEATED 1315 +#define WT_STAT_CONN_TXN_PREPARED_UPDATES_KEY_REPEATED 1414 /*! transaction: Number of prepared updates rolled back */ -#define WT_STAT_CONN_TXN_PREPARED_UPDATES_ROLLEDBACK 1316 +#define WT_STAT_CONN_TXN_PREPARED_UPDATES_ROLLEDBACK 1415 /*! transaction: prepared transactions */ -#define WT_STAT_CONN_TXN_PREPARE 1317 +#define WT_STAT_CONN_TXN_PREPARE 1416 /*! transaction: prepared transactions committed */ -#define WT_STAT_CONN_TXN_PREPARE_COMMIT 1318 +#define WT_STAT_CONN_TXN_PREPARE_COMMIT 1417 /*! transaction: prepared transactions currently active */ -#define WT_STAT_CONN_TXN_PREPARE_ACTIVE 1319 +#define WT_STAT_CONN_TXN_PREPARE_ACTIVE 1418 /*! transaction: prepared transactions rolled back */ -#define WT_STAT_CONN_TXN_PREPARE_ROLLBACK 1320 +#define WT_STAT_CONN_TXN_PREPARE_ROLLBACK 1419 /*! transaction: query timestamp calls */ -#define WT_STAT_CONN_TXN_QUERY_TS 1321 +#define WT_STAT_CONN_TXN_QUERY_TS 1420 +/*! transaction: race to read prepared update retry */ +#define WT_STAT_CONN_TXN_READ_RACE_PREPARE_UPDATE 1421 /*! transaction: rollback to stable calls */ -#define WT_STAT_CONN_TXN_RTS 1322 +#define WT_STAT_CONN_TXN_RTS 1422 +/*! + * transaction: rollback to stable history store records with stop + * timestamps older than newer records + */ +#define WT_STAT_CONN_TXN_RTS_HS_STOP_OLDER_THAN_NEWER_START 1423 +/*! transaction: rollback to stable inconsistent checkpoint */ +#define WT_STAT_CONN_TXN_RTS_INCONSISTENT_CKPT 1424 +/*! transaction: rollback to stable keys removed */ +#define WT_STAT_CONN_TXN_RTS_KEYS_REMOVED 1425 +/*! transaction: rollback to stable keys restored */ +#define WT_STAT_CONN_TXN_RTS_KEYS_RESTORED 1426 /*! transaction: rollback to stable pages visited */ -#define WT_STAT_CONN_TXN_RTS_PAGES_VISITED 1323 +#define WT_STAT_CONN_TXN_RTS_PAGES_VISITED 1427 +/*! transaction: rollback to stable restored tombstones from history store */ +#define WT_STAT_CONN_TXN_RTS_HS_RESTORE_TOMBSTONES 1428 +/*! transaction: rollback to stable restored updates from history store */ +#define WT_STAT_CONN_TXN_RTS_HS_RESTORE_UPDATES 1429 +/*! transaction: rollback to stable sweeping history store keys */ +#define WT_STAT_CONN_TXN_RTS_SWEEP_HS_KEYS 1430 /*! transaction: rollback to stable tree walk skipping pages */ -#define WT_STAT_CONN_TXN_RTS_TREE_WALK_SKIP_PAGES 1324 +#define WT_STAT_CONN_TXN_RTS_TREE_WALK_SKIP_PAGES 1431 /*! transaction: rollback to stable updates aborted */ -#define WT_STAT_CONN_TXN_RTS_UPD_ABORTED 1325 +#define WT_STAT_CONN_TXN_RTS_UPD_ABORTED 1432 +/*! transaction: rollback to stable updates removed from history store */ +#define WT_STAT_CONN_TXN_RTS_HS_REMOVED 1433 /*! transaction: sessions scanned in each walk of concurrent sessions */ -#define WT_STAT_CONN_TXN_SESSIONS_WALKED 1326 +#define WT_STAT_CONN_TXN_SESSIONS_WALKED 1434 /*! transaction: set timestamp calls */ -#define WT_STAT_CONN_TXN_SET_TS 1327 +#define WT_STAT_CONN_TXN_SET_TS 1435 /*! transaction: set timestamp durable calls */ -#define WT_STAT_CONN_TXN_SET_TS_DURABLE 1328 +#define WT_STAT_CONN_TXN_SET_TS_DURABLE 1436 /*! transaction: set timestamp durable updates */ -#define WT_STAT_CONN_TXN_SET_TS_DURABLE_UPD 1329 +#define WT_STAT_CONN_TXN_SET_TS_DURABLE_UPD 1437 /*! transaction: set timestamp oldest calls */ -#define WT_STAT_CONN_TXN_SET_TS_OLDEST 1330 +#define WT_STAT_CONN_TXN_SET_TS_OLDEST 1438 /*! transaction: set timestamp oldest updates */ -#define WT_STAT_CONN_TXN_SET_TS_OLDEST_UPD 1331 +#define WT_STAT_CONN_TXN_SET_TS_OLDEST_UPD 1439 /*! transaction: set timestamp stable calls */ -#define WT_STAT_CONN_TXN_SET_TS_STABLE 1332 +#define WT_STAT_CONN_TXN_SET_TS_STABLE 1440 /*! transaction: set timestamp stable updates */ -#define WT_STAT_CONN_TXN_SET_TS_STABLE_UPD 1333 +#define WT_STAT_CONN_TXN_SET_TS_STABLE_UPD 1441 /*! transaction: transaction begins */ -#define WT_STAT_CONN_TXN_BEGIN 1334 +#define WT_STAT_CONN_TXN_BEGIN 1442 /*! transaction: transaction checkpoint currently running */ -#define WT_STAT_CONN_TXN_CHECKPOINT_RUNNING 1335 +#define WT_STAT_CONN_TXN_CHECKPOINT_RUNNING 1443 /*! * transaction: transaction checkpoint currently running for history * store file */ -#define WT_STAT_CONN_TXN_CHECKPOINT_RUNNING_HS 1336 +#define WT_STAT_CONN_TXN_CHECKPOINT_RUNNING_HS 1444 /*! transaction: transaction checkpoint generation */ -#define WT_STAT_CONN_TXN_CHECKPOINT_GENERATION 1337 +#define WT_STAT_CONN_TXN_CHECKPOINT_GENERATION 1445 /*! * transaction: transaction checkpoint history store file duration * (usecs) */ -#define WT_STAT_CONN_TXN_HS_CKPT_DURATION 1338 +#define WT_STAT_CONN_TXN_HS_CKPT_DURATION 1446 /*! transaction: transaction checkpoint max time (msecs) */ -#define WT_STAT_CONN_TXN_CHECKPOINT_TIME_MAX 1339 +#define WT_STAT_CONN_TXN_CHECKPOINT_TIME_MAX 1447 /*! transaction: transaction checkpoint min time (msecs) */ -#define WT_STAT_CONN_TXN_CHECKPOINT_TIME_MIN 1340 +#define WT_STAT_CONN_TXN_CHECKPOINT_TIME_MIN 1448 /*! * transaction: transaction checkpoint most recent duration for gathering * all handles (usecs) */ -#define WT_STAT_CONN_TXN_CHECKPOINT_HANDLE_DURATION 1341 +#define WT_STAT_CONN_TXN_CHECKPOINT_HANDLE_DURATION 1449 /*! * transaction: transaction checkpoint most recent duration for gathering * applied handles (usecs) */ -#define WT_STAT_CONN_TXN_CHECKPOINT_HANDLE_DURATION_APPLY 1342 +#define WT_STAT_CONN_TXN_CHECKPOINT_HANDLE_DURATION_APPLY 1450 /*! * transaction: transaction checkpoint most recent duration for gathering * skipped handles (usecs) */ -#define WT_STAT_CONN_TXN_CHECKPOINT_HANDLE_DURATION_SKIP 1343 +#define WT_STAT_CONN_TXN_CHECKPOINT_HANDLE_DURATION_SKIP 1451 /*! transaction: transaction checkpoint most recent handles applied */ -#define WT_STAT_CONN_TXN_CHECKPOINT_HANDLE_APPLIED 1344 +#define WT_STAT_CONN_TXN_CHECKPOINT_HANDLE_APPLIED 1452 /*! transaction: transaction checkpoint most recent handles skipped */ -#define WT_STAT_CONN_TXN_CHECKPOINT_HANDLE_SKIPPED 1345 +#define WT_STAT_CONN_TXN_CHECKPOINT_HANDLE_SKIPPED 1453 /*! transaction: transaction checkpoint most recent handles walked */ -#define WT_STAT_CONN_TXN_CHECKPOINT_HANDLE_WALKED 1346 +#define WT_STAT_CONN_TXN_CHECKPOINT_HANDLE_WALKED 1454 /*! transaction: transaction checkpoint most recent time (msecs) */ -#define WT_STAT_CONN_TXN_CHECKPOINT_TIME_RECENT 1347 +#define WT_STAT_CONN_TXN_CHECKPOINT_TIME_RECENT 1455 /*! transaction: transaction checkpoint prepare currently running */ -#define WT_STAT_CONN_TXN_CHECKPOINT_PREP_RUNNING 1348 +#define WT_STAT_CONN_TXN_CHECKPOINT_PREP_RUNNING 1456 /*! transaction: transaction checkpoint prepare max time (msecs) */ -#define WT_STAT_CONN_TXN_CHECKPOINT_PREP_MAX 1349 +#define WT_STAT_CONN_TXN_CHECKPOINT_PREP_MAX 1457 /*! transaction: transaction checkpoint prepare min time (msecs) */ -#define WT_STAT_CONN_TXN_CHECKPOINT_PREP_MIN 1350 +#define WT_STAT_CONN_TXN_CHECKPOINT_PREP_MIN 1458 /*! transaction: transaction checkpoint prepare most recent time (msecs) */ -#define WT_STAT_CONN_TXN_CHECKPOINT_PREP_RECENT 1351 +#define WT_STAT_CONN_TXN_CHECKPOINT_PREP_RECENT 1459 /*! transaction: transaction checkpoint prepare total time (msecs) */ -#define WT_STAT_CONN_TXN_CHECKPOINT_PREP_TOTAL 1352 +#define WT_STAT_CONN_TXN_CHECKPOINT_PREP_TOTAL 1460 /*! transaction: transaction checkpoint scrub dirty target */ -#define WT_STAT_CONN_TXN_CHECKPOINT_SCRUB_TARGET 1353 +#define WT_STAT_CONN_TXN_CHECKPOINT_SCRUB_TARGET 1461 /*! transaction: transaction checkpoint scrub time (msecs) */ -#define WT_STAT_CONN_TXN_CHECKPOINT_SCRUB_TIME 1354 +#define WT_STAT_CONN_TXN_CHECKPOINT_SCRUB_TIME 1462 /*! transaction: transaction checkpoint total time (msecs) */ -#define WT_STAT_CONN_TXN_CHECKPOINT_TIME_TOTAL 1355 +#define WT_STAT_CONN_TXN_CHECKPOINT_TIME_TOTAL 1463 /*! transaction: transaction checkpoints */ -#define WT_STAT_CONN_TXN_CHECKPOINT 1356 +#define WT_STAT_CONN_TXN_CHECKPOINT 1464 +/*! transaction: transaction checkpoints due to obsolete pages */ +#define WT_STAT_CONN_TXN_CHECKPOINT_OBSOLETE_APPLIED 1465 /*! * transaction: transaction checkpoints skipped because database was * clean */ -#define WT_STAT_CONN_TXN_CHECKPOINT_SKIPPED 1357 +#define WT_STAT_CONN_TXN_CHECKPOINT_SKIPPED 1466 /*! transaction: transaction failures due to history store */ -#define WT_STAT_CONN_TXN_FAIL_CACHE 1358 +#define WT_STAT_CONN_TXN_FAIL_CACHE 1467 /*! * transaction: transaction fsync calls for checkpoint after allocating * the transaction ID */ -#define WT_STAT_CONN_TXN_CHECKPOINT_FSYNC_POST 1359 +#define WT_STAT_CONN_TXN_CHECKPOINT_FSYNC_POST 1468 /*! * transaction: transaction fsync duration for checkpoint after * allocating the transaction ID (usecs) */ -#define WT_STAT_CONN_TXN_CHECKPOINT_FSYNC_POST_DURATION 1360 +#define WT_STAT_CONN_TXN_CHECKPOINT_FSYNC_POST_DURATION 1469 /*! transaction: transaction range of IDs currently pinned */ -#define WT_STAT_CONN_TXN_PINNED_RANGE 1361 +#define WT_STAT_CONN_TXN_PINNED_RANGE 1470 /*! transaction: transaction range of IDs currently pinned by a checkpoint */ -#define WT_STAT_CONN_TXN_PINNED_CHECKPOINT_RANGE 1362 +#define WT_STAT_CONN_TXN_PINNED_CHECKPOINT_RANGE 1471 /*! transaction: transaction range of timestamps currently pinned */ -#define WT_STAT_CONN_TXN_PINNED_TIMESTAMP 1363 +#define WT_STAT_CONN_TXN_PINNED_TIMESTAMP 1472 /*! transaction: transaction range of timestamps pinned by a checkpoint */ -#define WT_STAT_CONN_TXN_PINNED_TIMESTAMP_CHECKPOINT 1364 +#define WT_STAT_CONN_TXN_PINNED_TIMESTAMP_CHECKPOINT 1473 /*! * transaction: transaction range of timestamps pinned by the oldest * active read timestamp */ -#define WT_STAT_CONN_TXN_PINNED_TIMESTAMP_READER 1365 +#define WT_STAT_CONN_TXN_PINNED_TIMESTAMP_READER 1474 /*! * transaction: transaction range of timestamps pinned by the oldest * timestamp */ -#define WT_STAT_CONN_TXN_PINNED_TIMESTAMP_OLDEST 1366 +#define WT_STAT_CONN_TXN_PINNED_TIMESTAMP_OLDEST 1475 /*! transaction: transaction read timestamp of the oldest active reader */ -#define WT_STAT_CONN_TXN_TIMESTAMP_OLDEST_ACTIVE_READ 1367 +#define WT_STAT_CONN_TXN_TIMESTAMP_OLDEST_ACTIVE_READ 1476 /*! transaction: transaction rollback to stable currently running */ -#define WT_STAT_CONN_TXN_ROLLBACK_TO_STABLE_RUNNING 1368 +#define WT_STAT_CONN_TXN_ROLLBACK_TO_STABLE_RUNNING 1477 /*! transaction: transaction sync calls */ -#define WT_STAT_CONN_TXN_SYNC 1369 +#define WT_STAT_CONN_TXN_SYNC 1478 /*! transaction: transaction walk of concurrent sessions */ -#define WT_STAT_CONN_TXN_WALK_SESSIONS 1370 +#define WT_STAT_CONN_TXN_WALK_SESSIONS 1479 /*! transaction: transactions committed */ -#define WT_STAT_CONN_TXN_COMMIT 1371 +#define WT_STAT_CONN_TXN_COMMIT 1480 /*! transaction: transactions rolled back */ -#define WT_STAT_CONN_TXN_ROLLBACK 1372 -/*! LSM: sleep for LSM checkpoint throttle */ -#define WT_STAT_CONN_LSM_CHECKPOINT_THROTTLE 1373 -/*! LSM: sleep for LSM merge throttle */ -#define WT_STAT_CONN_LSM_MERGE_THROTTLE 1374 -/*! cache: bytes currently in the cache */ -#define WT_STAT_CONN_CACHE_BYTES_INUSE 1375 -/*! cache: bytes dirty in the cache cumulative */ -#define WT_STAT_CONN_CACHE_BYTES_DIRTY_TOTAL 1376 -/*! cache: bytes read into cache */ -#define WT_STAT_CONN_CACHE_BYTES_READ 1377 -/*! cache: bytes written from cache */ -#define WT_STAT_CONN_CACHE_BYTES_WRITE 1378 -/*! cache: checkpoint blocked page eviction */ -#define WT_STAT_CONN_CACHE_EVICTION_CHECKPOINT 1379 -/*! - * cache: checkpoint of history store file blocked non-history store page - * eviction - */ -#define WT_STAT_CONN_CACHE_EVICTION_BLOCKED_CHECKPOINT_HS 1380 -/*! cache: eviction walk target pages histogram - 0-9 */ -#define WT_STAT_CONN_CACHE_EVICTION_TARGET_PAGE_LT10 1381 -/*! cache: eviction walk target pages histogram - 10-31 */ -#define WT_STAT_CONN_CACHE_EVICTION_TARGET_PAGE_LT32 1382 -/*! cache: eviction walk target pages histogram - 128 and higher */ -#define WT_STAT_CONN_CACHE_EVICTION_TARGET_PAGE_GE128 1383 -/*! cache: eviction walk target pages histogram - 32-63 */ -#define WT_STAT_CONN_CACHE_EVICTION_TARGET_PAGE_LT64 1384 -/*! cache: eviction walk target pages histogram - 64-128 */ -#define WT_STAT_CONN_CACHE_EVICTION_TARGET_PAGE_LT128 1385 -/*! - * cache: eviction walk target pages reduced due to history store cache - * pressure - */ -#define WT_STAT_CONN_CACHE_EVICTION_TARGET_PAGE_REDUCED 1386 -/*! cache: eviction walks abandoned */ -#define WT_STAT_CONN_CACHE_EVICTION_WALKS_ABANDONED 1387 -/*! cache: eviction walks gave up because they restarted their walk twice */ -#define WT_STAT_CONN_CACHE_EVICTION_WALKS_STOPPED 1388 -/*! - * cache: eviction walks gave up because they saw too many pages and - * found no candidates - */ -#define WT_STAT_CONN_CACHE_EVICTION_WALKS_GAVE_UP_NO_TARGETS 1389 -/*! - * cache: eviction walks gave up because they saw too many pages and - * found too few candidates - */ -#define WT_STAT_CONN_CACHE_EVICTION_WALKS_GAVE_UP_RATIO 1390 -/*! cache: eviction walks reached end of tree */ -#define WT_STAT_CONN_CACHE_EVICTION_WALKS_ENDED 1391 -/*! cache: eviction walks restarted */ -#define WT_STAT_CONN_CACHE_EVICTION_WALK_RESTART 1392 -/*! cache: eviction walks started from root of tree */ -#define WT_STAT_CONN_CACHE_EVICTION_WALK_FROM_ROOT 1393 -/*! cache: eviction walks started from saved location in tree */ -#define WT_STAT_CONN_CACHE_EVICTION_WALK_SAVED_POS 1394 -/*! cache: hazard pointer blocked page eviction */ -#define WT_STAT_CONN_CACHE_EVICTION_HAZARD 1395 -/*! cache: history store table insert calls */ -#define WT_STAT_CONN_CACHE_HS_INSERT 1396 -/*! cache: history store table insert calls that returned restart */ -#define WT_STAT_CONN_CACHE_HS_INSERT_RESTART 1397 -/*! - * cache: history store table out-of-order resolved updates that lose - * their durable timestamp - */ -#define WT_STAT_CONN_CACHE_HS_ORDER_LOSE_DURABLE_TIMESTAMP 1398 -/*! - * cache: history store table out-of-order updates that were fixed up by - * reinserting with the fixed timestamp - */ -#define WT_STAT_CONN_CACHE_HS_ORDER_REINSERT 1399 -/*! cache: history store table reads */ -#define WT_STAT_CONN_CACHE_HS_READ 1400 -/*! cache: history store table reads missed */ -#define WT_STAT_CONN_CACHE_HS_READ_MISS 1401 -/*! cache: history store table reads requiring squashed modifies */ -#define WT_STAT_CONN_CACHE_HS_READ_SQUASH 1402 -/*! - * cache: history store table truncation by rollback to stable to remove - * an unstable update - */ -#define WT_STAT_CONN_CACHE_HS_KEY_TRUNCATE_RTS_UNSTABLE 1403 -/*! - * cache: history store table truncation by rollback to stable to remove - * an update - */ -#define WT_STAT_CONN_CACHE_HS_KEY_TRUNCATE_RTS 1404 -/*! cache: history store table truncation to remove an update */ -#define WT_STAT_CONN_CACHE_HS_KEY_TRUNCATE 1405 -/*! - * cache: history store table truncation to remove range of updates due - * to key being removed from the data page during reconciliation - */ -#define WT_STAT_CONN_CACHE_HS_KEY_TRUNCATE_ONPAGE_REMOVAL 1406 -/*! - * cache: history store table truncation to remove range of updates due - * to out-of-order timestamp update on data page - */ -#define WT_STAT_CONN_CACHE_HS_ORDER_REMOVE 1407 -/*! cache: history store table writes requiring squashed modifies */ -#define WT_STAT_CONN_CACHE_HS_WRITE_SQUASH 1408 -/*! cache: in-memory page passed criteria to be split */ -#define WT_STAT_CONN_CACHE_INMEM_SPLITTABLE 1409 -/*! cache: in-memory page splits */ -#define WT_STAT_CONN_CACHE_INMEM_SPLIT 1410 -/*! cache: internal pages evicted */ -#define WT_STAT_CONN_CACHE_EVICTION_INTERNAL 1411 -/*! cache: internal pages split during eviction */ -#define WT_STAT_CONN_CACHE_EVICTION_SPLIT_INTERNAL 1412 -/*! cache: leaf pages split during eviction */ -#define WT_STAT_CONN_CACHE_EVICTION_SPLIT_LEAF 1413 -/*! cache: modified pages evicted */ -#define WT_STAT_CONN_CACHE_EVICTION_DIRTY 1414 -/*! cache: overflow pages read into cache */ -#define WT_STAT_CONN_CACHE_READ_OVERFLOW 1415 -/*! cache: page split during eviction deepened the tree */ -#define WT_STAT_CONN_CACHE_EVICTION_DEEPEN 1416 -/*! cache: page written requiring history store records */ -#define WT_STAT_CONN_CACHE_WRITE_HS 1417 -/*! cache: pages read into cache */ -#define WT_STAT_CONN_CACHE_READ 1418 -/*! cache: pages read into cache after truncate */ -#define WT_STAT_CONN_CACHE_READ_DELETED 1419 -/*! cache: pages read into cache after truncate in prepare state */ -#define WT_STAT_CONN_CACHE_READ_DELETED_PREPARED 1420 -/*! cache: pages requested from the cache */ -#define WT_STAT_CONN_CACHE_PAGES_REQUESTED 1421 -/*! cache: pages seen by eviction walk */ -#define WT_STAT_CONN_CACHE_EVICTION_PAGES_SEEN 1422 -/*! cache: pages written from cache */ -#define WT_STAT_CONN_CACHE_WRITE 1423 -/*! cache: pages written requiring in-memory restoration */ -#define WT_STAT_CONN_CACHE_WRITE_RESTORE 1424 -/*! cache: tracked dirty bytes in the cache */ -#define WT_STAT_CONN_CACHE_BYTES_DIRTY 1425 -/*! cache: unmodified pages evicted */ -#define WT_STAT_CONN_CACHE_EVICTION_CLEAN 1426 -/*! checkpoint-cleanup: pages added for eviction */ -#define WT_STAT_CONN_CC_PAGES_EVICT 1427 -/*! checkpoint-cleanup: pages removed */ -#define WT_STAT_CONN_CC_PAGES_REMOVED 1428 -/*! checkpoint-cleanup: pages skipped during tree walk */ -#define WT_STAT_CONN_CC_PAGES_WALK_SKIPPED 1429 -/*! checkpoint-cleanup: pages visited */ -#define WT_STAT_CONN_CC_PAGES_VISITED 1430 -/*! cursor: Total number of entries skipped by cursor next calls */ -#define WT_STAT_CONN_CURSOR_NEXT_SKIP_TOTAL 1431 -/*! cursor: Total number of entries skipped by cursor prev calls */ -#define WT_STAT_CONN_CURSOR_PREV_SKIP_TOTAL 1432 -/*! - * cursor: Total number of entries skipped to position the history store - * cursor - */ -#define WT_STAT_CONN_CURSOR_SKIP_HS_CUR_POSITION 1433 -/*! - * cursor: Total number of times a search near has exited due to prefix - * config - */ -#define WT_STAT_CONN_CURSOR_SEARCH_NEAR_PREFIX_FAST_PATHS 1434 -/*! - * cursor: cursor next calls that skip due to a globally visible history - * store tombstone - */ -#define WT_STAT_CONN_CURSOR_NEXT_HS_TOMBSTONE 1435 -/*! - * cursor: cursor next calls that skip greater than or equal to 100 - * entries - */ -#define WT_STAT_CONN_CURSOR_NEXT_SKIP_GE_100 1436 -/*! cursor: cursor next calls that skip less than 100 entries */ -#define WT_STAT_CONN_CURSOR_NEXT_SKIP_LT_100 1437 -/*! - * cursor: cursor prev calls that skip due to a globally visible history - * store tombstone - */ -#define WT_STAT_CONN_CURSOR_PREV_HS_TOMBSTONE 1438 -/*! - * cursor: cursor prev calls that skip greater than or equal to 100 - * entries - */ -#define WT_STAT_CONN_CURSOR_PREV_SKIP_GE_100 1439 -/*! cursor: cursor prev calls that skip less than 100 entries */ -#define WT_STAT_CONN_CURSOR_PREV_SKIP_LT_100 1440 -/*! cursor: open cursor count */ -#define WT_STAT_CONN_CURSOR_OPEN_COUNT 1441 -/*! reconciliation: approximate byte size of timestamps in pages written */ -#define WT_STAT_CONN_REC_TIME_WINDOW_BYTES_TS 1442 -/*! - * reconciliation: approximate byte size of transaction IDs in pages - * written - */ -#define WT_STAT_CONN_REC_TIME_WINDOW_BYTES_TXN 1443 -/*! reconciliation: fast-path pages deleted */ -#define WT_STAT_CONN_REC_PAGE_DELETE_FAST 1444 -/*! reconciliation: page reconciliation calls */ -#define WT_STAT_CONN_REC_PAGES 1445 -/*! reconciliation: page reconciliation calls for eviction */ -#define WT_STAT_CONN_REC_PAGES_EVICTION 1446 -/*! reconciliation: pages deleted */ -#define WT_STAT_CONN_REC_PAGE_DELETE 1447 -/*! - * reconciliation: pages written including an aggregated newest start - * durable timestamp - */ -#define WT_STAT_CONN_REC_TIME_AGGR_NEWEST_START_DURABLE_TS 1448 -/*! - * reconciliation: pages written including an aggregated newest stop - * durable timestamp - */ -#define WT_STAT_CONN_REC_TIME_AGGR_NEWEST_STOP_DURABLE_TS 1449 -/*! - * reconciliation: pages written including an aggregated newest stop - * timestamp - */ -#define WT_STAT_CONN_REC_TIME_AGGR_NEWEST_STOP_TS 1450 -/*! - * reconciliation: pages written including an aggregated newest stop - * transaction ID - */ -#define WT_STAT_CONN_REC_TIME_AGGR_NEWEST_STOP_TXN 1451 -/*! - * reconciliation: pages written including an aggregated newest - * transaction ID - */ -#define WT_STAT_CONN_REC_TIME_AGGR_NEWEST_TXN 1452 -/*! - * reconciliation: pages written including an aggregated oldest start - * timestamp - */ -#define WT_STAT_CONN_REC_TIME_AGGR_OLDEST_START_TS 1453 -/*! reconciliation: pages written including an aggregated prepare */ -#define WT_STAT_CONN_REC_TIME_AGGR_PREPARED 1454 -/*! - * reconciliation: pages written including at least one start durable - * timestamp - */ -#define WT_STAT_CONN_REC_TIME_WINDOW_PAGES_DURABLE_START_TS 1455 -/*! - * reconciliation: pages written including at least one start transaction - * ID - */ -#define WT_STAT_CONN_REC_TIME_WINDOW_PAGES_START_TXN 1456 -/*! - * reconciliation: pages written including at least one stop durable - * timestamp - */ -#define WT_STAT_CONN_REC_TIME_WINDOW_PAGES_DURABLE_STOP_TS 1457 -/*! reconciliation: pages written including at least one stop timestamp */ -#define WT_STAT_CONN_REC_TIME_WINDOW_PAGES_STOP_TS 1458 -/*! - * reconciliation: pages written including at least one stop transaction - * ID - */ -#define WT_STAT_CONN_REC_TIME_WINDOW_PAGES_STOP_TXN 1459 -/*! reconciliation: records written including a start durable timestamp */ -#define WT_STAT_CONN_REC_TIME_WINDOW_DURABLE_START_TS 1460 -/*! reconciliation: records written including a start timestamp */ -#define WT_STAT_CONN_REC_TIME_WINDOW_START_TS 1461 -/*! reconciliation: records written including a start transaction ID */ -#define WT_STAT_CONN_REC_TIME_WINDOW_START_TXN 1462 -/*! reconciliation: records written including a stop durable timestamp */ -#define WT_STAT_CONN_REC_TIME_WINDOW_DURABLE_STOP_TS 1463 -/*! reconciliation: records written including a stop timestamp */ -#define WT_STAT_CONN_REC_TIME_WINDOW_STOP_TS 1464 -/*! reconciliation: records written including a stop transaction ID */ -#define WT_STAT_CONN_REC_TIME_WINDOW_STOP_TXN 1465 -/*! session: tiered operations dequeued and processed */ -#define WT_STAT_CONN_TIERED_WORK_UNITS_DEQUEUED 1466 -/*! session: tiered operations scheduled */ -#define WT_STAT_CONN_TIERED_WORK_UNITS_CREATED 1467 -/*! session: tiered storage local retention time (secs) */ -#define WT_STAT_CONN_TIERED_RETENTION 1468 -/*! session: tiered storage object size */ -#define WT_STAT_CONN_TIERED_OBJECT_SIZE 1469 -/*! transaction: race to read prepared update retry */ -#define WT_STAT_CONN_TXN_READ_RACE_PREPARE_UPDATE 1470 -/*! - * transaction: rollback to stable history store records with stop - * timestamps older than newer records - */ -#define WT_STAT_CONN_TXN_RTS_HS_STOP_OLDER_THAN_NEWER_START 1471 -/*! transaction: rollback to stable inconsistent checkpoint */ -#define WT_STAT_CONN_TXN_RTS_INCONSISTENT_CKPT 1472 -/*! transaction: rollback to stable keys removed */ -#define WT_STAT_CONN_TXN_RTS_KEYS_REMOVED 1473 -/*! transaction: rollback to stable keys restored */ -#define WT_STAT_CONN_TXN_RTS_KEYS_RESTORED 1474 -/*! transaction: rollback to stable restored tombstones from history store */ -#define WT_STAT_CONN_TXN_RTS_HS_RESTORE_TOMBSTONES 1475 -/*! transaction: rollback to stable restored updates from history store */ -#define WT_STAT_CONN_TXN_RTS_HS_RESTORE_UPDATES 1476 -/*! transaction: rollback to stable sweeping history store keys */ -#define WT_STAT_CONN_TXN_RTS_SWEEP_HS_KEYS 1477 -/*! transaction: rollback to stable updates removed from history store */ -#define WT_STAT_CONN_TXN_RTS_HS_REMOVED 1478 -/*! transaction: transaction checkpoints due to obsolete pages */ -#define WT_STAT_CONN_TXN_CHECKPOINT_OBSOLETE_APPLIED 1479 +#define WT_STAT_CONN_TXN_ROLLBACK 1481 /*! transaction: update conflicts */ -#define WT_STAT_CONN_TXN_UPDATE_CONFLICT 1480 +#define WT_STAT_CONN_TXN_UPDATE_CONFLICT 1482 /*! * @} @@ -6140,606 +6190,616 @@ extern int wiredtiger_extension_terminate(WT_CONNECTION *connection); * not exist */ #define WT_STAT_DSRC_LSM_LOOKUP_NO_BLOOM 2008 +/*! LSM: sleep for LSM checkpoint throttle */ +#define WT_STAT_DSRC_LSM_CHECKPOINT_THROTTLE 2009 +/*! LSM: sleep for LSM merge throttle */ +#define WT_STAT_DSRC_LSM_MERGE_THROTTLE 2010 /*! LSM: total size of bloom filters */ -#define WT_STAT_DSRC_BLOOM_SIZE 2009 +#define WT_STAT_DSRC_BLOOM_SIZE 2011 /*! block-manager: allocations requiring file extension */ -#define WT_STAT_DSRC_BLOCK_EXTENSION 2010 +#define WT_STAT_DSRC_BLOCK_EXTENSION 2012 /*! block-manager: blocks allocated */ -#define WT_STAT_DSRC_BLOCK_ALLOC 2011 +#define WT_STAT_DSRC_BLOCK_ALLOC 2013 /*! block-manager: blocks freed */ -#define WT_STAT_DSRC_BLOCK_FREE 2012 +#define WT_STAT_DSRC_BLOCK_FREE 2014 /*! block-manager: checkpoint size */ -#define WT_STAT_DSRC_BLOCK_CHECKPOINT_SIZE 2013 +#define WT_STAT_DSRC_BLOCK_CHECKPOINT_SIZE 2015 /*! block-manager: file allocation unit size */ -#define WT_STAT_DSRC_ALLOCATION_SIZE 2014 +#define WT_STAT_DSRC_ALLOCATION_SIZE 2016 /*! block-manager: file bytes available for reuse */ -#define WT_STAT_DSRC_BLOCK_REUSE_BYTES 2015 +#define WT_STAT_DSRC_BLOCK_REUSE_BYTES 2017 /*! block-manager: file magic number */ -#define WT_STAT_DSRC_BLOCK_MAGIC 2016 +#define WT_STAT_DSRC_BLOCK_MAGIC 2018 /*! block-manager: file major version number */ -#define WT_STAT_DSRC_BLOCK_MAJOR 2017 +#define WT_STAT_DSRC_BLOCK_MAJOR 2019 /*! block-manager: file size in bytes */ -#define WT_STAT_DSRC_BLOCK_SIZE 2018 +#define WT_STAT_DSRC_BLOCK_SIZE 2020 /*! block-manager: minor version number */ -#define WT_STAT_DSRC_BLOCK_MINOR 2019 +#define WT_STAT_DSRC_BLOCK_MINOR 2021 /*! btree: btree checkpoint generation */ -#define WT_STAT_DSRC_BTREE_CHECKPOINT_GENERATION 2020 +#define WT_STAT_DSRC_BTREE_CHECKPOINT_GENERATION 2022 /*! btree: btree clean tree checkpoint expiration time */ -#define WT_STAT_DSRC_BTREE_CLEAN_CHECKPOINT_TIMER 2021 +#define WT_STAT_DSRC_BTREE_CLEAN_CHECKPOINT_TIMER 2023 /*! * btree: column-store fixed-size leaf pages, only reported if tree_walk * or all statistics are enabled */ -#define WT_STAT_DSRC_BTREE_COLUMN_FIX 2022 +#define WT_STAT_DSRC_BTREE_COLUMN_FIX 2024 /*! * btree: column-store internal pages, only reported if tree_walk or all * statistics are enabled */ -#define WT_STAT_DSRC_BTREE_COLUMN_INTERNAL 2023 +#define WT_STAT_DSRC_BTREE_COLUMN_INTERNAL 2025 /*! * btree: column-store variable-size RLE encoded values, only reported if * tree_walk or all statistics are enabled */ -#define WT_STAT_DSRC_BTREE_COLUMN_RLE 2024 +#define WT_STAT_DSRC_BTREE_COLUMN_RLE 2026 /*! * btree: column-store variable-size deleted values, only reported if * tree_walk or all statistics are enabled */ -#define WT_STAT_DSRC_BTREE_COLUMN_DELETED 2025 +#define WT_STAT_DSRC_BTREE_COLUMN_DELETED 2027 /*! * btree: column-store variable-size leaf pages, only reported if * tree_walk or all statistics are enabled */ -#define WT_STAT_DSRC_BTREE_COLUMN_VARIABLE 2026 +#define WT_STAT_DSRC_BTREE_COLUMN_VARIABLE 2028 /*! btree: fixed-record size */ -#define WT_STAT_DSRC_BTREE_FIXED_LEN 2027 +#define WT_STAT_DSRC_BTREE_FIXED_LEN 2029 /*! btree: maximum internal page key size */ -#define WT_STAT_DSRC_BTREE_MAXINTLKEY 2028 +#define WT_STAT_DSRC_BTREE_MAXINTLKEY 2030 /*! btree: maximum internal page size */ -#define WT_STAT_DSRC_BTREE_MAXINTLPAGE 2029 +#define WT_STAT_DSRC_BTREE_MAXINTLPAGE 2031 /*! btree: maximum leaf page key size */ -#define WT_STAT_DSRC_BTREE_MAXLEAFKEY 2030 +#define WT_STAT_DSRC_BTREE_MAXLEAFKEY 2032 /*! btree: maximum leaf page size */ -#define WT_STAT_DSRC_BTREE_MAXLEAFPAGE 2031 +#define WT_STAT_DSRC_BTREE_MAXLEAFPAGE 2033 /*! btree: maximum leaf page value size */ -#define WT_STAT_DSRC_BTREE_MAXLEAFVALUE 2032 +#define WT_STAT_DSRC_BTREE_MAXLEAFVALUE 2034 /*! btree: maximum tree depth */ -#define WT_STAT_DSRC_BTREE_MAXIMUM_DEPTH 2033 +#define WT_STAT_DSRC_BTREE_MAXIMUM_DEPTH 2035 /*! * btree: number of key/value pairs, only reported if tree_walk or all * statistics are enabled */ -#define WT_STAT_DSRC_BTREE_ENTRIES 2034 +#define WT_STAT_DSRC_BTREE_ENTRIES 2036 /*! * btree: overflow pages, only reported if tree_walk or all statistics * are enabled */ -#define WT_STAT_DSRC_BTREE_OVERFLOW 2035 +#define WT_STAT_DSRC_BTREE_OVERFLOW 2037 /*! btree: pages rewritten by compaction */ -#define WT_STAT_DSRC_BTREE_COMPACT_REWRITE 2036 +#define WT_STAT_DSRC_BTREE_COMPACT_REWRITE 2038 /*! * btree: row-store empty values, only reported if tree_walk or all * statistics are enabled */ -#define WT_STAT_DSRC_BTREE_ROW_EMPTY_VALUES 2037 +#define WT_STAT_DSRC_BTREE_ROW_EMPTY_VALUES 2039 /*! * btree: row-store internal pages, only reported if tree_walk or all * statistics are enabled */ -#define WT_STAT_DSRC_BTREE_ROW_INTERNAL 2038 +#define WT_STAT_DSRC_BTREE_ROW_INTERNAL 2040 /*! * btree: row-store leaf pages, only reported if tree_walk or all * statistics are enabled */ -#define WT_STAT_DSRC_BTREE_ROW_LEAF 2039 +#define WT_STAT_DSRC_BTREE_ROW_LEAF 2041 +/*! cache: bytes currently in the cache */ +#define WT_STAT_DSRC_CACHE_BYTES_INUSE 2042 +/*! cache: bytes dirty in the cache cumulative */ +#define WT_STAT_DSRC_CACHE_BYTES_DIRTY_TOTAL 2043 +/*! cache: bytes read into cache */ +#define WT_STAT_DSRC_CACHE_BYTES_READ 2044 +/*! cache: bytes written from cache */ +#define WT_STAT_DSRC_CACHE_BYTES_WRITE 2045 +/*! cache: checkpoint blocked page eviction */ +#define WT_STAT_DSRC_CACHE_EVICTION_CHECKPOINT 2046 +/*! + * cache: checkpoint of history store file blocked non-history store page + * eviction + */ +#define WT_STAT_DSRC_CACHE_EVICTION_BLOCKED_CHECKPOINT_HS 2047 /*! cache: data source pages selected for eviction unable to be evicted */ -#define WT_STAT_DSRC_CACHE_EVICTION_FAIL 2040 +#define WT_STAT_DSRC_CACHE_EVICTION_FAIL 2048 /*! cache: eviction walk passes of a file */ -#define WT_STAT_DSRC_CACHE_EVICTION_WALK_PASSES 2041 +#define WT_STAT_DSRC_CACHE_EVICTION_WALK_PASSES 2049 +/*! cache: eviction walk target pages histogram - 0-9 */ +#define WT_STAT_DSRC_CACHE_EVICTION_TARGET_PAGE_LT10 2050 +/*! cache: eviction walk target pages histogram - 10-31 */ +#define WT_STAT_DSRC_CACHE_EVICTION_TARGET_PAGE_LT32 2051 +/*! cache: eviction walk target pages histogram - 128 and higher */ +#define WT_STAT_DSRC_CACHE_EVICTION_TARGET_PAGE_GE128 2052 +/*! cache: eviction walk target pages histogram - 32-63 */ +#define WT_STAT_DSRC_CACHE_EVICTION_TARGET_PAGE_LT64 2053 +/*! cache: eviction walk target pages histogram - 64-128 */ +#define WT_STAT_DSRC_CACHE_EVICTION_TARGET_PAGE_LT128 2054 +/*! + * cache: eviction walk target pages reduced due to history store cache + * pressure + */ +#define WT_STAT_DSRC_CACHE_EVICTION_TARGET_PAGE_REDUCED 2055 +/*! cache: eviction walks abandoned */ +#define WT_STAT_DSRC_CACHE_EVICTION_WALKS_ABANDONED 2056 +/*! cache: eviction walks gave up because they restarted their walk twice */ +#define WT_STAT_DSRC_CACHE_EVICTION_WALKS_STOPPED 2057 +/*! + * cache: eviction walks gave up because they saw too many pages and + * found no candidates + */ +#define WT_STAT_DSRC_CACHE_EVICTION_WALKS_GAVE_UP_NO_TARGETS 2058 +/*! + * cache: eviction walks gave up because they saw too many pages and + * found too few candidates + */ +#define WT_STAT_DSRC_CACHE_EVICTION_WALKS_GAVE_UP_RATIO 2059 +/*! cache: eviction walks reached end of tree */ +#define WT_STAT_DSRC_CACHE_EVICTION_WALKS_ENDED 2060 +/*! cache: eviction walks restarted */ +#define WT_STAT_DSRC_CACHE_EVICTION_WALK_RESTART 2061 +/*! cache: eviction walks started from root of tree */ +#define WT_STAT_DSRC_CACHE_EVICTION_WALK_FROM_ROOT 2062 +/*! cache: eviction walks started from saved location in tree */ +#define WT_STAT_DSRC_CACHE_EVICTION_WALK_SAVED_POS 2063 +/*! cache: hazard pointer blocked page eviction */ +#define WT_STAT_DSRC_CACHE_EVICTION_HAZARD 2064 +/*! cache: history store table insert calls */ +#define WT_STAT_DSRC_CACHE_HS_INSERT 2065 +/*! cache: history store table insert calls that returned restart */ +#define WT_STAT_DSRC_CACHE_HS_INSERT_RESTART 2066 +/*! + * cache: history store table out-of-order resolved updates that lose + * their durable timestamp + */ +#define WT_STAT_DSRC_CACHE_HS_ORDER_LOSE_DURABLE_TIMESTAMP 2067 +/*! + * cache: history store table out-of-order updates that were fixed up by + * reinserting with the fixed timestamp + */ +#define WT_STAT_DSRC_CACHE_HS_ORDER_REINSERT 2068 +/*! cache: history store table reads */ +#define WT_STAT_DSRC_CACHE_HS_READ 2069 +/*! cache: history store table reads missed */ +#define WT_STAT_DSRC_CACHE_HS_READ_MISS 2070 +/*! cache: history store table reads requiring squashed modifies */ +#define WT_STAT_DSRC_CACHE_HS_READ_SQUASH 2071 +/*! + * cache: history store table truncation by rollback to stable to remove + * an unstable update + */ +#define WT_STAT_DSRC_CACHE_HS_KEY_TRUNCATE_RTS_UNSTABLE 2072 +/*! + * cache: history store table truncation by rollback to stable to remove + * an update + */ +#define WT_STAT_DSRC_CACHE_HS_KEY_TRUNCATE_RTS 2073 +/*! cache: history store table truncation to remove an update */ +#define WT_STAT_DSRC_CACHE_HS_KEY_TRUNCATE 2074 +/*! + * cache: history store table truncation to remove range of updates due + * to key being removed from the data page during reconciliation + */ +#define WT_STAT_DSRC_CACHE_HS_KEY_TRUNCATE_ONPAGE_REMOVAL 2075 +/*! + * cache: history store table truncation to remove range of updates due + * to out-of-order timestamp update on data page + */ +#define WT_STAT_DSRC_CACHE_HS_ORDER_REMOVE 2076 +/*! cache: history store table writes requiring squashed modifies */ +#define WT_STAT_DSRC_CACHE_HS_WRITE_SQUASH 2077 +/*! cache: in-memory page passed criteria to be split */ +#define WT_STAT_DSRC_CACHE_INMEM_SPLITTABLE 2078 +/*! cache: in-memory page splits */ +#define WT_STAT_DSRC_CACHE_INMEM_SPLIT 2079 +/*! cache: internal pages evicted */ +#define WT_STAT_DSRC_CACHE_EVICTION_INTERNAL 2080 +/*! cache: internal pages split during eviction */ +#define WT_STAT_DSRC_CACHE_EVICTION_SPLIT_INTERNAL 2081 +/*! cache: leaf pages split during eviction */ +#define WT_STAT_DSRC_CACHE_EVICTION_SPLIT_LEAF 2082 +/*! cache: modified pages evicted */ +#define WT_STAT_DSRC_CACHE_EVICTION_DIRTY 2083 +/*! cache: overflow pages read into cache */ +#define WT_STAT_DSRC_CACHE_READ_OVERFLOW 2084 +/*! cache: page split during eviction deepened the tree */ +#define WT_STAT_DSRC_CACHE_EVICTION_DEEPEN 2085 +/*! cache: page written requiring history store records */ +#define WT_STAT_DSRC_CACHE_WRITE_HS 2086 +/*! cache: pages read into cache */ +#define WT_STAT_DSRC_CACHE_READ 2087 +/*! cache: pages read into cache after truncate */ +#define WT_STAT_DSRC_CACHE_READ_DELETED 2088 +/*! cache: pages read into cache after truncate in prepare state */ +#define WT_STAT_DSRC_CACHE_READ_DELETED_PREPARED 2089 +/*! cache: pages requested from the cache */ +#define WT_STAT_DSRC_CACHE_PAGES_REQUESTED 2090 +/*! cache: pages seen by eviction walk */ +#define WT_STAT_DSRC_CACHE_EVICTION_PAGES_SEEN 2091 +/*! cache: pages written from cache */ +#define WT_STAT_DSRC_CACHE_WRITE 2092 +/*! cache: pages written requiring in-memory restoration */ +#define WT_STAT_DSRC_CACHE_WRITE_RESTORE 2093 +/*! cache: tracked dirty bytes in the cache */ +#define WT_STAT_DSRC_CACHE_BYTES_DIRTY 2094 +/*! cache: unmodified pages evicted */ +#define WT_STAT_DSRC_CACHE_EVICTION_CLEAN 2095 /*! * cache_walk: Average difference between current eviction generation * when the page was last considered, only reported if cache_walk or all * statistics are enabled */ -#define WT_STAT_DSRC_CACHE_STATE_GEN_AVG_GAP 2042 +#define WT_STAT_DSRC_CACHE_STATE_GEN_AVG_GAP 2096 /*! * cache_walk: Average on-disk page image size seen, only reported if * cache_walk or all statistics are enabled */ -#define WT_STAT_DSRC_CACHE_STATE_AVG_WRITTEN_SIZE 2043 +#define WT_STAT_DSRC_CACHE_STATE_AVG_WRITTEN_SIZE 2097 /*! * cache_walk: Average time in cache for pages that have been visited by * the eviction server, only reported if cache_walk or all statistics are * enabled */ -#define WT_STAT_DSRC_CACHE_STATE_AVG_VISITED_AGE 2044 +#define WT_STAT_DSRC_CACHE_STATE_AVG_VISITED_AGE 2098 /*! * cache_walk: Average time in cache for pages that have not been visited * by the eviction server, only reported if cache_walk or all statistics * are enabled */ -#define WT_STAT_DSRC_CACHE_STATE_AVG_UNVISITED_AGE 2045 +#define WT_STAT_DSRC_CACHE_STATE_AVG_UNVISITED_AGE 2099 /*! * cache_walk: Clean pages currently in cache, only reported if * cache_walk or all statistics are enabled */ -#define WT_STAT_DSRC_CACHE_STATE_PAGES_CLEAN 2046 +#define WT_STAT_DSRC_CACHE_STATE_PAGES_CLEAN 2100 /*! * cache_walk: Current eviction generation, only reported if cache_walk * or all statistics are enabled */ -#define WT_STAT_DSRC_CACHE_STATE_GEN_CURRENT 2047 +#define WT_STAT_DSRC_CACHE_STATE_GEN_CURRENT 2101 /*! * cache_walk: Dirty pages currently in cache, only reported if * cache_walk or all statistics are enabled */ -#define WT_STAT_DSRC_CACHE_STATE_PAGES_DIRTY 2048 +#define WT_STAT_DSRC_CACHE_STATE_PAGES_DIRTY 2102 /*! * cache_walk: Entries in the root page, only reported if cache_walk or * all statistics are enabled */ -#define WT_STAT_DSRC_CACHE_STATE_ROOT_ENTRIES 2049 +#define WT_STAT_DSRC_CACHE_STATE_ROOT_ENTRIES 2103 /*! * cache_walk: Internal pages currently in cache, only reported if * cache_walk or all statistics are enabled */ -#define WT_STAT_DSRC_CACHE_STATE_PAGES_INTERNAL 2050 +#define WT_STAT_DSRC_CACHE_STATE_PAGES_INTERNAL 2104 /*! * cache_walk: Leaf pages currently in cache, only reported if cache_walk * or all statistics are enabled */ -#define WT_STAT_DSRC_CACHE_STATE_PAGES_LEAF 2051 +#define WT_STAT_DSRC_CACHE_STATE_PAGES_LEAF 2105 /*! * cache_walk: Maximum difference between current eviction generation * when the page was last considered, only reported if cache_walk or all * statistics are enabled */ -#define WT_STAT_DSRC_CACHE_STATE_GEN_MAX_GAP 2052 +#define WT_STAT_DSRC_CACHE_STATE_GEN_MAX_GAP 2106 /*! * cache_walk: Maximum page size seen, only reported if cache_walk or all * statistics are enabled */ -#define WT_STAT_DSRC_CACHE_STATE_MAX_PAGESIZE 2053 +#define WT_STAT_DSRC_CACHE_STATE_MAX_PAGESIZE 2107 /*! * cache_walk: Minimum on-disk page image size seen, only reported if * cache_walk or all statistics are enabled */ -#define WT_STAT_DSRC_CACHE_STATE_MIN_WRITTEN_SIZE 2054 +#define WT_STAT_DSRC_CACHE_STATE_MIN_WRITTEN_SIZE 2108 /*! * cache_walk: Number of pages never visited by eviction server, only * reported if cache_walk or all statistics are enabled */ -#define WT_STAT_DSRC_CACHE_STATE_UNVISITED_COUNT 2055 +#define WT_STAT_DSRC_CACHE_STATE_UNVISITED_COUNT 2109 /*! * cache_walk: On-disk page image sizes smaller than a single allocation * unit, only reported if cache_walk or all statistics are enabled */ -#define WT_STAT_DSRC_CACHE_STATE_SMALLER_ALLOC_SIZE 2056 +#define WT_STAT_DSRC_CACHE_STATE_SMALLER_ALLOC_SIZE 2110 /*! * cache_walk: Pages created in memory and never written, only reported * if cache_walk or all statistics are enabled */ -#define WT_STAT_DSRC_CACHE_STATE_MEMORY 2057 +#define WT_STAT_DSRC_CACHE_STATE_MEMORY 2111 /*! * cache_walk: Pages currently queued for eviction, only reported if * cache_walk or all statistics are enabled */ -#define WT_STAT_DSRC_CACHE_STATE_QUEUED 2058 +#define WT_STAT_DSRC_CACHE_STATE_QUEUED 2112 /*! * cache_walk: Pages that could not be queued for eviction, only reported * if cache_walk or all statistics are enabled */ -#define WT_STAT_DSRC_CACHE_STATE_NOT_QUEUEABLE 2059 +#define WT_STAT_DSRC_CACHE_STATE_NOT_QUEUEABLE 2113 /*! * cache_walk: Refs skipped during cache traversal, only reported if * cache_walk or all statistics are enabled */ -#define WT_STAT_DSRC_CACHE_STATE_REFS_SKIPPED 2060 +#define WT_STAT_DSRC_CACHE_STATE_REFS_SKIPPED 2114 /*! * cache_walk: Size of the root page, only reported if cache_walk or all * statistics are enabled */ -#define WT_STAT_DSRC_CACHE_STATE_ROOT_SIZE 2061 +#define WT_STAT_DSRC_CACHE_STATE_ROOT_SIZE 2115 /*! * cache_walk: Total number of pages currently in cache, only reported if * cache_walk or all statistics are enabled */ -#define WT_STAT_DSRC_CACHE_STATE_PAGES 2062 +#define WT_STAT_DSRC_CACHE_STATE_PAGES 2116 +/*! checkpoint-cleanup: pages added for eviction */ +#define WT_STAT_DSRC_CC_PAGES_EVICT 2117 +/*! checkpoint-cleanup: pages removed */ +#define WT_STAT_DSRC_CC_PAGES_REMOVED 2118 +/*! checkpoint-cleanup: pages skipped during tree walk */ +#define WT_STAT_DSRC_CC_PAGES_WALK_SKIPPED 2119 +/*! checkpoint-cleanup: pages visited */ +#define WT_STAT_DSRC_CC_PAGES_VISITED 2120 /*! * compression: compressed page maximum internal page size prior to * compression */ -#define WT_STAT_DSRC_COMPRESS_PRECOMP_INTL_MAX_PAGE_SIZE 2063 +#define WT_STAT_DSRC_COMPRESS_PRECOMP_INTL_MAX_PAGE_SIZE 2121 /*! * compression: compressed page maximum leaf page size prior to * compression */ -#define WT_STAT_DSRC_COMPRESS_PRECOMP_LEAF_MAX_PAGE_SIZE 2064 +#define WT_STAT_DSRC_COMPRESS_PRECOMP_LEAF_MAX_PAGE_SIZE 2122 /*! compression: compressed pages read */ -#define WT_STAT_DSRC_COMPRESS_READ 2065 +#define WT_STAT_DSRC_COMPRESS_READ 2123 /*! compression: compressed pages written */ -#define WT_STAT_DSRC_COMPRESS_WRITE 2066 +#define WT_STAT_DSRC_COMPRESS_WRITE 2124 /*! compression: page written failed to compress */ -#define WT_STAT_DSRC_COMPRESS_WRITE_FAIL 2067 +#define WT_STAT_DSRC_COMPRESS_WRITE_FAIL 2125 /*! compression: page written was too small to compress */ -#define WT_STAT_DSRC_COMPRESS_WRITE_TOO_SMALL 2068 +#define WT_STAT_DSRC_COMPRESS_WRITE_TOO_SMALL 2126 +/*! cursor: Total number of entries skipped by cursor next calls */ +#define WT_STAT_DSRC_CURSOR_NEXT_SKIP_TOTAL 2127 +/*! cursor: Total number of entries skipped by cursor prev calls */ +#define WT_STAT_DSRC_CURSOR_PREV_SKIP_TOTAL 2128 +/*! + * cursor: Total number of entries skipped to position the history store + * cursor + */ +#define WT_STAT_DSRC_CURSOR_SKIP_HS_CUR_POSITION 2129 +/*! + * cursor: Total number of pages skipped without reading by cursor next + * calls + */ +#define WT_STAT_DSRC_CURSOR_NEXT_SKIP_PAGE_COUNT 2130 +/*! + * cursor: Total number of pages skipped without reading by cursor prev + * calls + */ +#define WT_STAT_DSRC_CURSOR_PREV_SKIP_PAGE_COUNT 2131 +/*! + * cursor: Total number of times a search near has exited due to prefix + * config + */ +#define WT_STAT_DSRC_CURSOR_SEARCH_NEAR_PREFIX_FAST_PATHS 2132 /*! cursor: bulk loaded cursor insert calls */ -#define WT_STAT_DSRC_CURSOR_INSERT_BULK 2069 +#define WT_STAT_DSRC_CURSOR_INSERT_BULK 2133 /*! cursor: cache cursors reuse count */ -#define WT_STAT_DSRC_CURSOR_REOPEN 2070 +#define WT_STAT_DSRC_CURSOR_REOPEN 2134 /*! cursor: close calls that result in cache */ -#define WT_STAT_DSRC_CURSOR_CACHE 2071 +#define WT_STAT_DSRC_CURSOR_CACHE 2135 /*! cursor: create calls */ -#define WT_STAT_DSRC_CURSOR_CREATE 2072 +#define WT_STAT_DSRC_CURSOR_CREATE 2136 +/*! + * cursor: cursor next calls that skip due to a globally visible history + * store tombstone + */ +#define WT_STAT_DSRC_CURSOR_NEXT_HS_TOMBSTONE 2137 +/*! + * cursor: cursor next calls that skip greater than or equal to 100 + * entries + */ +#define WT_STAT_DSRC_CURSOR_NEXT_SKIP_GE_100 2138 +/*! cursor: cursor next calls that skip less than 100 entries */ +#define WT_STAT_DSRC_CURSOR_NEXT_SKIP_LT_100 2139 +/*! + * cursor: cursor prev calls that skip due to a globally visible history + * store tombstone + */ +#define WT_STAT_DSRC_CURSOR_PREV_HS_TOMBSTONE 2140 +/*! + * cursor: cursor prev calls that skip greater than or equal to 100 + * entries + */ +#define WT_STAT_DSRC_CURSOR_PREV_SKIP_GE_100 2141 +/*! cursor: cursor prev calls that skip less than 100 entries */ +#define WT_STAT_DSRC_CURSOR_PREV_SKIP_LT_100 2142 /*! cursor: insert calls */ -#define WT_STAT_DSRC_CURSOR_INSERT 2073 +#define WT_STAT_DSRC_CURSOR_INSERT 2143 /*! cursor: insert key and value bytes */ -#define WT_STAT_DSRC_CURSOR_INSERT_BYTES 2074 +#define WT_STAT_DSRC_CURSOR_INSERT_BYTES 2144 /*! cursor: modify */ -#define WT_STAT_DSRC_CURSOR_MODIFY 2075 +#define WT_STAT_DSRC_CURSOR_MODIFY 2145 /*! cursor: modify key and value bytes affected */ -#define WT_STAT_DSRC_CURSOR_MODIFY_BYTES 2076 +#define WT_STAT_DSRC_CURSOR_MODIFY_BYTES 2146 /*! cursor: modify value bytes modified */ -#define WT_STAT_DSRC_CURSOR_MODIFY_BYTES_TOUCH 2077 +#define WT_STAT_DSRC_CURSOR_MODIFY_BYTES_TOUCH 2147 /*! cursor: next calls */ -#define WT_STAT_DSRC_CURSOR_NEXT 2078 +#define WT_STAT_DSRC_CURSOR_NEXT 2148 +/*! cursor: open cursor count */ +#define WT_STAT_DSRC_CURSOR_OPEN_COUNT 2149 /*! cursor: operation restarted */ -#define WT_STAT_DSRC_CURSOR_RESTART 2079 +#define WT_STAT_DSRC_CURSOR_RESTART 2150 /*! cursor: prev calls */ -#define WT_STAT_DSRC_CURSOR_PREV 2080 +#define WT_STAT_DSRC_CURSOR_PREV 2151 /*! cursor: remove calls */ -#define WT_STAT_DSRC_CURSOR_REMOVE 2081 +#define WT_STAT_DSRC_CURSOR_REMOVE 2152 /*! cursor: remove key bytes removed */ -#define WT_STAT_DSRC_CURSOR_REMOVE_BYTES 2082 +#define WT_STAT_DSRC_CURSOR_REMOVE_BYTES 2153 /*! cursor: reserve calls */ -#define WT_STAT_DSRC_CURSOR_RESERVE 2083 +#define WT_STAT_DSRC_CURSOR_RESERVE 2154 /*! cursor: reset calls */ -#define WT_STAT_DSRC_CURSOR_RESET 2084 +#define WT_STAT_DSRC_CURSOR_RESET 2155 /*! cursor: search calls */ -#define WT_STAT_DSRC_CURSOR_SEARCH 2085 +#define WT_STAT_DSRC_CURSOR_SEARCH 2156 /*! cursor: search history store calls */ -#define WT_STAT_DSRC_CURSOR_SEARCH_HS 2086 +#define WT_STAT_DSRC_CURSOR_SEARCH_HS 2157 /*! cursor: search near calls */ -#define WT_STAT_DSRC_CURSOR_SEARCH_NEAR 2087 +#define WT_STAT_DSRC_CURSOR_SEARCH_NEAR 2158 /*! cursor: truncate calls */ -#define WT_STAT_DSRC_CURSOR_TRUNCATE 2088 +#define WT_STAT_DSRC_CURSOR_TRUNCATE 2159 /*! cursor: update calls */ -#define WT_STAT_DSRC_CURSOR_UPDATE 2089 +#define WT_STAT_DSRC_CURSOR_UPDATE 2160 /*! cursor: update key and value bytes */ -#define WT_STAT_DSRC_CURSOR_UPDATE_BYTES 2090 +#define WT_STAT_DSRC_CURSOR_UPDATE_BYTES 2161 /*! cursor: update value size change */ -#define WT_STAT_DSRC_CURSOR_UPDATE_BYTES_CHANGED 2091 +#define WT_STAT_DSRC_CURSOR_UPDATE_BYTES_CHANGED 2162 +/*! reconciliation: approximate byte size of timestamps in pages written */ +#define WT_STAT_DSRC_REC_TIME_WINDOW_BYTES_TS 2163 +/*! + * reconciliation: approximate byte size of transaction IDs in pages + * written + */ +#define WT_STAT_DSRC_REC_TIME_WINDOW_BYTES_TXN 2164 /*! reconciliation: dictionary matches */ -#define WT_STAT_DSRC_REC_DICTIONARY 2092 +#define WT_STAT_DSRC_REC_DICTIONARY 2165 +/*! reconciliation: fast-path pages deleted */ +#define WT_STAT_DSRC_REC_PAGE_DELETE_FAST 2166 /*! * reconciliation: internal page key bytes discarded using suffix * compression */ -#define WT_STAT_DSRC_REC_SUFFIX_COMPRESSION 2093 +#define WT_STAT_DSRC_REC_SUFFIX_COMPRESSION 2167 /*! reconciliation: internal page multi-block writes */ -#define WT_STAT_DSRC_REC_MULTIBLOCK_INTERNAL 2094 +#define WT_STAT_DSRC_REC_MULTIBLOCK_INTERNAL 2168 /*! reconciliation: internal-page overflow keys */ -#define WT_STAT_DSRC_REC_OVERFLOW_KEY_INTERNAL 2095 +#define WT_STAT_DSRC_REC_OVERFLOW_KEY_INTERNAL 2169 /*! reconciliation: leaf page key bytes discarded using prefix compression */ -#define WT_STAT_DSRC_REC_PREFIX_COMPRESSION 2096 +#define WT_STAT_DSRC_REC_PREFIX_COMPRESSION 2170 /*! reconciliation: leaf page multi-block writes */ -#define WT_STAT_DSRC_REC_MULTIBLOCK_LEAF 2097 +#define WT_STAT_DSRC_REC_MULTIBLOCK_LEAF 2171 /*! reconciliation: leaf-page overflow keys */ -#define WT_STAT_DSRC_REC_OVERFLOW_KEY_LEAF 2098 +#define WT_STAT_DSRC_REC_OVERFLOW_KEY_LEAF 2172 /*! reconciliation: maximum blocks required for a page */ -#define WT_STAT_DSRC_REC_MULTIBLOCK_MAX 2099 +#define WT_STAT_DSRC_REC_MULTIBLOCK_MAX 2173 /*! reconciliation: overflow values written */ -#define WT_STAT_DSRC_REC_OVERFLOW_VALUE 2100 +#define WT_STAT_DSRC_REC_OVERFLOW_VALUE 2174 /*! reconciliation: page checksum matches */ -#define WT_STAT_DSRC_REC_PAGE_MATCH 2101 -/*! reconciliation: pages written including at least one prepare */ -#define WT_STAT_DSRC_REC_TIME_WINDOW_PAGES_PREPARED 2102 -/*! reconciliation: pages written including at least one start timestamp */ -#define WT_STAT_DSRC_REC_TIME_WINDOW_PAGES_START_TS 2103 -/*! reconciliation: records written including a prepare */ -#define WT_STAT_DSRC_REC_TIME_WINDOW_PREPARED 2104 -/*! session: object compaction */ -#define WT_STAT_DSRC_SESSION_COMPACT 2105 -/*! LSM: sleep for LSM checkpoint throttle */ -#define WT_STAT_DSRC_LSM_CHECKPOINT_THROTTLE 2106 -/*! LSM: sleep for LSM merge throttle */ -#define WT_STAT_DSRC_LSM_MERGE_THROTTLE 2107 -/*! cache: bytes currently in the cache */ -#define WT_STAT_DSRC_CACHE_BYTES_INUSE 2108 -/*! cache: bytes dirty in the cache cumulative */ -#define WT_STAT_DSRC_CACHE_BYTES_DIRTY_TOTAL 2109 -/*! cache: bytes read into cache */ -#define WT_STAT_DSRC_CACHE_BYTES_READ 2110 -/*! cache: bytes written from cache */ -#define WT_STAT_DSRC_CACHE_BYTES_WRITE 2111 -/*! cache: checkpoint blocked page eviction */ -#define WT_STAT_DSRC_CACHE_EVICTION_CHECKPOINT 2112 -/*! - * cache: checkpoint of history store file blocked non-history store page - * eviction - */ -#define WT_STAT_DSRC_CACHE_EVICTION_BLOCKED_CHECKPOINT_HS 2113 -/*! cache: eviction walk target pages histogram - 0-9 */ -#define WT_STAT_DSRC_CACHE_EVICTION_TARGET_PAGE_LT10 2114 -/*! cache: eviction walk target pages histogram - 10-31 */ -#define WT_STAT_DSRC_CACHE_EVICTION_TARGET_PAGE_LT32 2115 -/*! cache: eviction walk target pages histogram - 128 and higher */ -#define WT_STAT_DSRC_CACHE_EVICTION_TARGET_PAGE_GE128 2116 -/*! cache: eviction walk target pages histogram - 32-63 */ -#define WT_STAT_DSRC_CACHE_EVICTION_TARGET_PAGE_LT64 2117 -/*! cache: eviction walk target pages histogram - 64-128 */ -#define WT_STAT_DSRC_CACHE_EVICTION_TARGET_PAGE_LT128 2118 -/*! - * cache: eviction walk target pages reduced due to history store cache - * pressure - */ -#define WT_STAT_DSRC_CACHE_EVICTION_TARGET_PAGE_REDUCED 2119 -/*! cache: eviction walks abandoned */ -#define WT_STAT_DSRC_CACHE_EVICTION_WALKS_ABANDONED 2120 -/*! cache: eviction walks gave up because they restarted their walk twice */ -#define WT_STAT_DSRC_CACHE_EVICTION_WALKS_STOPPED 2121 -/*! - * cache: eviction walks gave up because they saw too many pages and - * found no candidates - */ -#define WT_STAT_DSRC_CACHE_EVICTION_WALKS_GAVE_UP_NO_TARGETS 2122 -/*! - * cache: eviction walks gave up because they saw too many pages and - * found too few candidates - */ -#define WT_STAT_DSRC_CACHE_EVICTION_WALKS_GAVE_UP_RATIO 2123 -/*! cache: eviction walks reached end of tree */ -#define WT_STAT_DSRC_CACHE_EVICTION_WALKS_ENDED 2124 -/*! cache: eviction walks restarted */ -#define WT_STAT_DSRC_CACHE_EVICTION_WALK_RESTART 2125 -/*! cache: eviction walks started from root of tree */ -#define WT_STAT_DSRC_CACHE_EVICTION_WALK_FROM_ROOT 2126 -/*! cache: eviction walks started from saved location in tree */ -#define WT_STAT_DSRC_CACHE_EVICTION_WALK_SAVED_POS 2127 -/*! cache: hazard pointer blocked page eviction */ -#define WT_STAT_DSRC_CACHE_EVICTION_HAZARD 2128 -/*! cache: history store table insert calls */ -#define WT_STAT_DSRC_CACHE_HS_INSERT 2129 -/*! cache: history store table insert calls that returned restart */ -#define WT_STAT_DSRC_CACHE_HS_INSERT_RESTART 2130 -/*! - * cache: history store table out-of-order resolved updates that lose - * their durable timestamp - */ -#define WT_STAT_DSRC_CACHE_HS_ORDER_LOSE_DURABLE_TIMESTAMP 2131 -/*! - * cache: history store table out-of-order updates that were fixed up by - * reinserting with the fixed timestamp - */ -#define WT_STAT_DSRC_CACHE_HS_ORDER_REINSERT 2132 -/*! cache: history store table reads */ -#define WT_STAT_DSRC_CACHE_HS_READ 2133 -/*! cache: history store table reads missed */ -#define WT_STAT_DSRC_CACHE_HS_READ_MISS 2134 -/*! cache: history store table reads requiring squashed modifies */ -#define WT_STAT_DSRC_CACHE_HS_READ_SQUASH 2135 -/*! - * cache: history store table truncation by rollback to stable to remove - * an unstable update - */ -#define WT_STAT_DSRC_CACHE_HS_KEY_TRUNCATE_RTS_UNSTABLE 2136 -/*! - * cache: history store table truncation by rollback to stable to remove - * an update - */ -#define WT_STAT_DSRC_CACHE_HS_KEY_TRUNCATE_RTS 2137 -/*! cache: history store table truncation to remove an update */ -#define WT_STAT_DSRC_CACHE_HS_KEY_TRUNCATE 2138 -/*! - * cache: history store table truncation to remove range of updates due - * to key being removed from the data page during reconciliation - */ -#define WT_STAT_DSRC_CACHE_HS_KEY_TRUNCATE_ONPAGE_REMOVAL 2139 -/*! - * cache: history store table truncation to remove range of updates due - * to out-of-order timestamp update on data page - */ -#define WT_STAT_DSRC_CACHE_HS_ORDER_REMOVE 2140 -/*! cache: history store table writes requiring squashed modifies */ -#define WT_STAT_DSRC_CACHE_HS_WRITE_SQUASH 2141 -/*! cache: in-memory page passed criteria to be split */ -#define WT_STAT_DSRC_CACHE_INMEM_SPLITTABLE 2142 -/*! cache: in-memory page splits */ -#define WT_STAT_DSRC_CACHE_INMEM_SPLIT 2143 -/*! cache: internal pages evicted */ -#define WT_STAT_DSRC_CACHE_EVICTION_INTERNAL 2144 -/*! cache: internal pages split during eviction */ -#define WT_STAT_DSRC_CACHE_EVICTION_SPLIT_INTERNAL 2145 -/*! cache: leaf pages split during eviction */ -#define WT_STAT_DSRC_CACHE_EVICTION_SPLIT_LEAF 2146 -/*! cache: modified pages evicted */ -#define WT_STAT_DSRC_CACHE_EVICTION_DIRTY 2147 -/*! cache: overflow pages read into cache */ -#define WT_STAT_DSRC_CACHE_READ_OVERFLOW 2148 -/*! cache: page split during eviction deepened the tree */ -#define WT_STAT_DSRC_CACHE_EVICTION_DEEPEN 2149 -/*! cache: page written requiring history store records */ -#define WT_STAT_DSRC_CACHE_WRITE_HS 2150 -/*! cache: pages read into cache */ -#define WT_STAT_DSRC_CACHE_READ 2151 -/*! cache: pages read into cache after truncate */ -#define WT_STAT_DSRC_CACHE_READ_DELETED 2152 -/*! cache: pages read into cache after truncate in prepare state */ -#define WT_STAT_DSRC_CACHE_READ_DELETED_PREPARED 2153 -/*! cache: pages requested from the cache */ -#define WT_STAT_DSRC_CACHE_PAGES_REQUESTED 2154 -/*! cache: pages seen by eviction walk */ -#define WT_STAT_DSRC_CACHE_EVICTION_PAGES_SEEN 2155 -/*! cache: pages written from cache */ -#define WT_STAT_DSRC_CACHE_WRITE 2156 -/*! cache: pages written requiring in-memory restoration */ -#define WT_STAT_DSRC_CACHE_WRITE_RESTORE 2157 -/*! cache: tracked dirty bytes in the cache */ -#define WT_STAT_DSRC_CACHE_BYTES_DIRTY 2158 -/*! cache: unmodified pages evicted */ -#define WT_STAT_DSRC_CACHE_EVICTION_CLEAN 2159 -/*! checkpoint-cleanup: pages added for eviction */ -#define WT_STAT_DSRC_CC_PAGES_EVICT 2160 -/*! checkpoint-cleanup: pages removed */ -#define WT_STAT_DSRC_CC_PAGES_REMOVED 2161 -/*! checkpoint-cleanup: pages skipped during tree walk */ -#define WT_STAT_DSRC_CC_PAGES_WALK_SKIPPED 2162 -/*! checkpoint-cleanup: pages visited */ -#define WT_STAT_DSRC_CC_PAGES_VISITED 2163 -/*! cursor: Total number of entries skipped by cursor next calls */ -#define WT_STAT_DSRC_CURSOR_NEXT_SKIP_TOTAL 2164 -/*! cursor: Total number of entries skipped by cursor prev calls */ -#define WT_STAT_DSRC_CURSOR_PREV_SKIP_TOTAL 2165 -/*! - * cursor: Total number of entries skipped to position the history store - * cursor - */ -#define WT_STAT_DSRC_CURSOR_SKIP_HS_CUR_POSITION 2166 -/*! - * cursor: Total number of times a search near has exited due to prefix - * config - */ -#define WT_STAT_DSRC_CURSOR_SEARCH_NEAR_PREFIX_FAST_PATHS 2167 -/*! - * cursor: cursor next calls that skip due to a globally visible history - * store tombstone - */ -#define WT_STAT_DSRC_CURSOR_NEXT_HS_TOMBSTONE 2168 -/*! - * cursor: cursor next calls that skip greater than or equal to 100 - * entries - */ -#define WT_STAT_DSRC_CURSOR_NEXT_SKIP_GE_100 2169 -/*! cursor: cursor next calls that skip less than 100 entries */ -#define WT_STAT_DSRC_CURSOR_NEXT_SKIP_LT_100 2170 -/*! - * cursor: cursor prev calls that skip due to a globally visible history - * store tombstone - */ -#define WT_STAT_DSRC_CURSOR_PREV_HS_TOMBSTONE 2171 -/*! - * cursor: cursor prev calls that skip greater than or equal to 100 - * entries - */ -#define WT_STAT_DSRC_CURSOR_PREV_SKIP_GE_100 2172 -/*! cursor: cursor prev calls that skip less than 100 entries */ -#define WT_STAT_DSRC_CURSOR_PREV_SKIP_LT_100 2173 -/*! cursor: open cursor count */ -#define WT_STAT_DSRC_CURSOR_OPEN_COUNT 2174 -/*! reconciliation: approximate byte size of timestamps in pages written */ -#define WT_STAT_DSRC_REC_TIME_WINDOW_BYTES_TS 2175 -/*! - * reconciliation: approximate byte size of transaction IDs in pages - * written - */ -#define WT_STAT_DSRC_REC_TIME_WINDOW_BYTES_TXN 2176 -/*! reconciliation: fast-path pages deleted */ -#define WT_STAT_DSRC_REC_PAGE_DELETE_FAST 2177 +#define WT_STAT_DSRC_REC_PAGE_MATCH 2175 /*! reconciliation: page reconciliation calls */ -#define WT_STAT_DSRC_REC_PAGES 2178 +#define WT_STAT_DSRC_REC_PAGES 2176 /*! reconciliation: page reconciliation calls for eviction */ -#define WT_STAT_DSRC_REC_PAGES_EVICTION 2179 +#define WT_STAT_DSRC_REC_PAGES_EVICTION 2177 /*! reconciliation: pages deleted */ -#define WT_STAT_DSRC_REC_PAGE_DELETE 2180 +#define WT_STAT_DSRC_REC_PAGE_DELETE 2178 /*! * reconciliation: pages written including an aggregated newest start * durable timestamp */ -#define WT_STAT_DSRC_REC_TIME_AGGR_NEWEST_START_DURABLE_TS 2181 +#define WT_STAT_DSRC_REC_TIME_AGGR_NEWEST_START_DURABLE_TS 2179 /*! * reconciliation: pages written including an aggregated newest stop * durable timestamp */ -#define WT_STAT_DSRC_REC_TIME_AGGR_NEWEST_STOP_DURABLE_TS 2182 +#define WT_STAT_DSRC_REC_TIME_AGGR_NEWEST_STOP_DURABLE_TS 2180 /*! * reconciliation: pages written including an aggregated newest stop * timestamp */ -#define WT_STAT_DSRC_REC_TIME_AGGR_NEWEST_STOP_TS 2183 +#define WT_STAT_DSRC_REC_TIME_AGGR_NEWEST_STOP_TS 2181 /*! * reconciliation: pages written including an aggregated newest stop * transaction ID */ -#define WT_STAT_DSRC_REC_TIME_AGGR_NEWEST_STOP_TXN 2184 +#define WT_STAT_DSRC_REC_TIME_AGGR_NEWEST_STOP_TXN 2182 /*! * reconciliation: pages written including an aggregated newest * transaction ID */ -#define WT_STAT_DSRC_REC_TIME_AGGR_NEWEST_TXN 2185 +#define WT_STAT_DSRC_REC_TIME_AGGR_NEWEST_TXN 2183 /*! * reconciliation: pages written including an aggregated oldest start * timestamp */ -#define WT_STAT_DSRC_REC_TIME_AGGR_OLDEST_START_TS 2186 +#define WT_STAT_DSRC_REC_TIME_AGGR_OLDEST_START_TS 2184 /*! reconciliation: pages written including an aggregated prepare */ -#define WT_STAT_DSRC_REC_TIME_AGGR_PREPARED 2187 +#define WT_STAT_DSRC_REC_TIME_AGGR_PREPARED 2185 +/*! reconciliation: pages written including at least one prepare */ +#define WT_STAT_DSRC_REC_TIME_WINDOW_PAGES_PREPARED 2186 /*! * reconciliation: pages written including at least one start durable * timestamp */ -#define WT_STAT_DSRC_REC_TIME_WINDOW_PAGES_DURABLE_START_TS 2188 +#define WT_STAT_DSRC_REC_TIME_WINDOW_PAGES_DURABLE_START_TS 2187 +/*! reconciliation: pages written including at least one start timestamp */ +#define WT_STAT_DSRC_REC_TIME_WINDOW_PAGES_START_TS 2188 /*! * reconciliation: pages written including at least one start transaction * ID */ -#define WT_STAT_DSRC_REC_TIME_WINDOW_PAGES_START_TXN 2189 +#define WT_STAT_DSRC_REC_TIME_WINDOW_PAGES_START_TXN 2187 /*! * reconciliation: pages written including at least one stop durable * timestamp */ -#define WT_STAT_DSRC_REC_TIME_WINDOW_PAGES_DURABLE_STOP_TS 2190 +#define WT_STAT_DSRC_REC_TIME_WINDOW_PAGES_DURABLE_STOP_TS 2188 /*! reconciliation: pages written including at least one stop timestamp */ -#define WT_STAT_DSRC_REC_TIME_WINDOW_PAGES_STOP_TS 2191 +#define WT_STAT_DSRC_REC_TIME_WINDOW_PAGES_STOP_TS 2189 /*! * reconciliation: pages written including at least one stop transaction * ID */ #define WT_STAT_DSRC_REC_TIME_WINDOW_PAGES_STOP_TXN 2192 +/*! reconciliation: records written including a prepare */ +#define WT_STAT_DSRC_REC_TIME_WINDOW_PREPARED 2193 /*! reconciliation: records written including a start durable timestamp */ -#define WT_STAT_DSRC_REC_TIME_WINDOW_DURABLE_START_TS 2193 +#define WT_STAT_DSRC_REC_TIME_WINDOW_DURABLE_START_TS 2194 /*! reconciliation: records written including a start timestamp */ -#define WT_STAT_DSRC_REC_TIME_WINDOW_START_TS 2194 +#define WT_STAT_DSRC_REC_TIME_WINDOW_START_TS 2195 /*! reconciliation: records written including a start transaction ID */ -#define WT_STAT_DSRC_REC_TIME_WINDOW_START_TXN 2195 +#define WT_STAT_DSRC_REC_TIME_WINDOW_START_TXN 2196 /*! reconciliation: records written including a stop durable timestamp */ -#define WT_STAT_DSRC_REC_TIME_WINDOW_DURABLE_STOP_TS 2196 +#define WT_STAT_DSRC_REC_TIME_WINDOW_DURABLE_STOP_TS 2197 /*! reconciliation: records written including a stop timestamp */ -#define WT_STAT_DSRC_REC_TIME_WINDOW_STOP_TS 2197 +#define WT_STAT_DSRC_REC_TIME_WINDOW_STOP_TS 2198 /*! reconciliation: records written including a stop transaction ID */ -#define WT_STAT_DSRC_REC_TIME_WINDOW_STOP_TXN 2198 +#define WT_STAT_DSRC_REC_TIME_WINDOW_STOP_TXN 2199 +/*! session: object compaction */ +#define WT_STAT_DSRC_SESSION_COMPACT 2200 /*! session: tiered operations dequeued and processed */ -#define WT_STAT_DSRC_TIERED_WORK_UNITS_DEQUEUED 2199 +#define WT_STAT_DSRC_TIERED_WORK_UNITS_DEQUEUED 2201 /*! session: tiered operations scheduled */ -#define WT_STAT_DSRC_TIERED_WORK_UNITS_CREATED 2200 +#define WT_STAT_DSRC_TIERED_WORK_UNITS_CREATED 2202 /*! session: tiered storage local retention time (secs) */ -#define WT_STAT_DSRC_TIERED_RETENTION 2201 +#define WT_STAT_DSRC_TIERED_RETENTION 2203 /*! session: tiered storage object size */ -#define WT_STAT_DSRC_TIERED_OBJECT_SIZE 2202 +#define WT_STAT_DSRC_TIERED_OBJECT_SIZE 2204 /*! transaction: race to read prepared update retry */ -#define WT_STAT_DSRC_TXN_READ_RACE_PREPARE_UPDATE 2203 +#define WT_STAT_DSRC_TXN_READ_RACE_PREPARE_UPDATE 2205 /*! * transaction: rollback to stable history store records with stop * timestamps older than newer records */ -#define WT_STAT_DSRC_TXN_RTS_HS_STOP_OLDER_THAN_NEWER_START 2204 +#define WT_STAT_DSRC_TXN_RTS_HS_STOP_OLDER_THAN_NEWER_START 2206 /*! transaction: rollback to stable inconsistent checkpoint */ -#define WT_STAT_DSRC_TXN_RTS_INCONSISTENT_CKPT 2205 +#define WT_STAT_DSRC_TXN_RTS_INCONSISTENT_CKPT 2207 /*! transaction: rollback to stable keys removed */ -#define WT_STAT_DSRC_TXN_RTS_KEYS_REMOVED 2206 +#define WT_STAT_DSRC_TXN_RTS_KEYS_REMOVED 2208 /*! transaction: rollback to stable keys restored */ -#define WT_STAT_DSRC_TXN_RTS_KEYS_RESTORED 2207 +#define WT_STAT_DSRC_TXN_RTS_KEYS_RESTORED 2209 /*! transaction: rollback to stable restored tombstones from history store */ -#define WT_STAT_DSRC_TXN_RTS_HS_RESTORE_TOMBSTONES 2208 +#define WT_STAT_DSRC_TXN_RTS_HS_RESTORE_TOMBSTONES 2210 /*! transaction: rollback to stable restored updates from history store */ -#define WT_STAT_DSRC_TXN_RTS_HS_RESTORE_UPDATES 2209 +#define WT_STAT_DSRC_TXN_RTS_HS_RESTORE_UPDATES 2211 /*! transaction: rollback to stable sweeping history store keys */ -#define WT_STAT_DSRC_TXN_RTS_SWEEP_HS_KEYS 2210 +#define WT_STAT_DSRC_TXN_RTS_SWEEP_HS_KEYS 2212 /*! transaction: rollback to stable updates removed from history store */ -#define WT_STAT_DSRC_TXN_RTS_HS_REMOVED 2211 +#define WT_STAT_DSRC_TXN_RTS_HS_REMOVED 2213 /*! transaction: transaction checkpoints due to obsolete pages */ -#define WT_STAT_DSRC_TXN_CHECKPOINT_OBSOLETE_APPLIED 2212 +#define WT_STAT_DSRC_TXN_CHECKPOINT_OBSOLETE_APPLIED 2214 /*! transaction: update conflicts */ -#define WT_STAT_DSRC_TXN_UPDATE_CONFLICT 2213 +#define WT_STAT_DSRC_TXN_UPDATE_CONFLICT 2215 /*! * @} diff --git a/src/third_party/wiredtiger/src/include/wiredtiger_ext.h b/src/third_party/wiredtiger/src/include/wiredtiger_ext.h index 0efdc3bfefc..22d798647d6 100644 --- a/src/third_party/wiredtiger/src/include/wiredtiger_ext.h +++ b/src/third_party/wiredtiger/src/include/wiredtiger_ext.h @@ -57,6 +57,15 @@ struct __wt_txn_notify { int (*notify)(WT_TXN_NOTIFY *notify, WT_SESSION *session, uint64_t txnid, int committed); }; +typedef struct __wt_extension_spinlock WT_EXTENSION_SPINLOCK; +/*! + * A placeholder data structure that allows for using the WiredTiger + * spinlock implementation from within extensions. + */ +struct __wt_extension_spinlock { + void *spinlock; /* Represents actual WiredTiger spinlock. */ +}; + /*! * Table of WiredTiger extension methods. * @@ -536,6 +545,49 @@ struct __wt_extension_api { * @copydoc wiredtiger_version */ const char *(*version)(int *majorp, int *minorp, int *patchp); + + /*! + * Initialize a spinlock + * + * @param wt_api the extension handle + * @param session the session handle + * @param spinlock the extension spinlock + * @param name the name for the spinlock + * + */ + int (*spin_init)(WT_EXTENSION_API *wt_api, WT_EXTENSION_SPINLOCK *spinlock, const char *name); + + /*! + * Destroy a spinlock + * + * @param wt_api the extension handle + * @param session the session handle + * @param spinlock the extension spinlock + * + */ + void (*spin_destroy)(WT_EXTENSION_API *wt_api, WT_EXTENSION_SPINLOCK *spinlock); + + /*! + * Spin until the lock is acquired. + * + * @param wt_api the extension handle + * @param session the session handle + * @param spinlock the extension spinlock + * + */ + void (*spin_lock)( + WT_EXTENSION_API *wt_api, WT_SESSION *session, WT_EXTENSION_SPINLOCK *spinlock); + + /*! + * Release the spinlock. + * + * @param wt_api the extension handle + * @param session the session handle + * @param spinlock the extension spinlock + * + */ + void (*spin_unlock)( + WT_EXTENSION_API *wt_api, WT_SESSION *session, WT_EXTENSION_SPINLOCK *spinlock); }; /*! diff --git a/src/third_party/wiredtiger/src/log/log.c b/src/third_party/wiredtiger/src/log/log.c index e95dde65807..bfa252ce4b8 100644 --- a/src/third_party/wiredtiger/src/log/log.c +++ b/src/third_party/wiredtiger/src/log/log.c @@ -16,9 +16,9 @@ static int __log_write_internal(WT_SESSION_IMPL *, WT_ITEM *, WT_LSN *, uint32_t #define WT_LOG_COMPRESS_SKIP (offsetof(WT_LOG_RECORD, record)) #define WT_LOG_ENCRYPT_SKIP (offsetof(WT_LOG_RECORD, record)) -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define WT_LOG_OPEN_CREATE_OK 0x1u /* Flag to __log_openfile() */ - /* AUTOMATIC FLAG VALUE GENERATION STOP */ +/* AUTOMATIC FLAG VALUE GENERATION STOP 32 */ /* * __wt_log_printf -- diff --git a/src/third_party/wiredtiger/src/meta/meta_apply.c b/src/third_party/wiredtiger/src/meta/meta_apply.c index a729ce1d69b..668b24cff07 100644 --- a/src/third_party/wiredtiger/src/meta/meta_apply.c +++ b/src/third_party/wiredtiger/src/meta/meta_apply.c @@ -37,7 +37,8 @@ __meta_btree_apply(WT_SESSION_IMPL *session, WT_CURSOR *cursor, continue; } - if (file_func == NULL || skip || !WT_PREFIX_MATCH(uri, "file:")) + if (file_func == NULL || skip || + (!WT_PREFIX_MATCH(uri, "file:") && !WT_PREFIX_MATCH(uri, "tiered:"))) continue; /* diff --git a/src/third_party/wiredtiger/src/meta/meta_ckpt.c b/src/third_party/wiredtiger/src/meta/meta_ckpt.c index d4c8f316e5f..b4c7c933c62 100644 --- a/src/third_party/wiredtiger/src/meta/meta_ckpt.c +++ b/src/third_party/wiredtiger/src/meta/meta_ckpt.c @@ -454,7 +454,7 @@ __ckpt_valid_blk_mods(WT_SESSION_IMPL *session, WT_CKPT *ckpt, bool rename) * resources and then set up our entry. */ - /* Check if the global entry is valid at our index. */ + /* Check if the global entry is valid at our index. */ if (!F_ISSET(blk, WT_BLKINCR_VALID)) { free = true; setup = false; @@ -473,7 +473,7 @@ __ckpt_valid_blk_mods(WT_SESSION_IMPL *session, WT_CKPT *ckpt, bool rename) if (rename && (!free || setup)) F_SET(blk_mod, WT_BLOCK_MODS_RENAME); - /* Free any old information if we need to do so. */ + /* Free any old information if we need to do so. */ if (free && F_ISSET(blk_mod, WT_BLOCK_MODS_VALID)) { __wt_free(session, blk_mod->id_str); __wt_buf_free(session, &blk_mod->bitstring); @@ -483,7 +483,7 @@ __ckpt_valid_blk_mods(WT_SESSION_IMPL *session, WT_CKPT *ckpt, bool rename) F_CLR(blk_mod, WT_BLOCK_MODS_VALID); } - /* Set up the block list to point to the current information. */ + /* Set up the block list to point to the current information. */ if (setup) { WT_RET(__wt_strdup(session, blk->id_str, &blk_mod->id_str)); WT_CLEAR(blk_mod->bitstring); @@ -557,24 +557,66 @@ __meta_blk_mods_load( } /* - * __wt_meta_ckptlist_get -- - * Load all available checkpoint information for a file. + * __meta_ckptlist_allocate_new_ckpt -- + * Provided a checkpoint list, allocate a new checkpoint. Either use the last checkpoint in the + * list or the file metadata to initialize this new checkpoint. */ -int -__wt_meta_ckptlist_get( - WT_SESSION_IMPL *session, const char *fname, bool update, WT_CKPT **ckptbasep, size_t *allocated) +static int +__meta_ckptlist_allocate_new_ckpt( + WT_SESSION_IMPL *session, WT_CKPT **ckptbasep, size_t *allocated, const char *config) { - WT_DECL_RET; - char *config; + WT_CKPT *ckptbase, *ckpt; + WT_CONNECTION_IMPL *conn; + size_t slot; + uint64_t most_recent; - config = NULL; + ckptbase = *ckptbasep; + conn = S2C(session); + slot = 0; - WT_ERR(__wt_metadata_search(session, fname, &config)); - WT_ERR(__wt_meta_ckptlist_get_from_config(session, update, ckptbasep, allocated, config)); + if (ckptbase != NULL) + WT_CKPT_FOREACH (ckptbase, ckpt) + slot++; -err: - __wt_free(session, config); - return (ret); + /* + * Either we have a configuration or an existing checkpoint to initialize with. Also, If we are + * using an existing checkpoint, we must have the associated metadata. + */ + WT_ASSERT(session, config != NULL || (slot != 0 && ckptbase[slot - 1].block_metadata != NULL)); + + /* + * This isn't clean, but there's necessary cooperation between the schema layer (that maintains + * the list of checkpoints), the btree layer (that knows when the root page is written, creating + * a new checkpoint), and the block manager (which actually creates the checkpoint). All of that + * cooperation is handled in the array of checkpoint structures referenced from the WT_BTREE + * structure. + * + * Allocate a slot for a new value, plus a slot to mark the end. + */ + WT_RET(__wt_realloc_def(session, allocated, slot + 2, &ckptbase)); + *ckptbasep = ckptbase; + + ckpt = &ckptbase[slot]; + ckpt->order = (slot == 0) ? 1 : ckptbase[slot - 1].order + 1; + __wt_seconds(session, &ckpt->sec); + /* + * Update time value for most recent checkpoint, not letting it move backwards. It is possible + * to race here, so use atomic CAS. This code relies on the fact that anyone we race with will + * only increase (never decrease) the most recent checkpoint time value. + */ + for (;;) { + WT_ORDERED_READ(most_recent, conn->ckpt_most_recent); + if (ckpt->sec <= most_recent || + __wt_atomic_cas64(&conn->ckpt_most_recent, most_recent, ckpt->sec)) + break; + } + + /* Either load block mods from the config, or from the previous checkpoint. */ + WT_RET( + __meta_blk_mods_load(session, config, (slot == 0 ? NULL : &ckptbase[slot - 1]), ckpt, false)); + WT_ASSERT(session, ckpt->block_metadata != NULL); + + return (0); } #ifdef HAVE_DIAGNOSTIC @@ -646,117 +688,62 @@ __assert_checkpoint_list_matches(WT_SESSION_IMPL *session, WT_CKPT *saved_list, #endif /* - * __meta_ckptlist_allocate_new_ckpt -- - * Provided a checkpoint list, allocate a new checkpoint. Either use the last checkpoint in the - * list or the file metadata to initialize this new checkpoint. - */ -static int -__meta_ckptlist_allocate_new_ckpt( - WT_SESSION_IMPL *session, WT_CKPT **ckptbasep, size_t *allocated, const char *config) -{ - WT_CKPT *ckptbase, *ckpt; - WT_CONNECTION_IMPL *conn; - size_t slot; - uint64_t most_recent; - - ckptbase = *ckptbasep; - conn = S2C(session); - slot = 0; - - if (ckptbase != NULL) - WT_CKPT_FOREACH (ckptbase, ckpt) - slot++; - - /* Either we have a configuration or an existing checkpoint to initialize with. */ - WT_ASSERT(session, config != NULL || slot != 0); - - /* - * If we are using an existing checkpoint, we must have the associated metadata. Otherwise we - * will have to go slow path and read the metadata. - */ - if (config == NULL && ckptbase[slot - 1].block_metadata == NULL) - return (WT_NOTFOUND); - - /* - * This isn't clean, but there's necessary cooperation between the schema layer (that maintains - * the list of checkpoints), the btree layer (that knows when the root page is written, creating - * a new checkpoint), and the block manager (which actually creates the checkpoint). All of that - * cooperation is handled in the array of checkpoint structures referenced from the WT_BTREE - * structure. - * - * Allocate a slot for a new value, plus a slot to mark the end. - */ - WT_RET(__wt_realloc_def(session, allocated, slot + 2, &ckptbase)); - *ckptbasep = ckptbase; - - ckpt = &ckptbase[slot]; - ckpt->order = (slot == 0) ? 1 : ckptbase[slot - 1].order + 1; - __wt_seconds(session, &ckpt->sec); - /* - * Update time value for most recent checkpoint, not letting it move backwards. It is possible - * to race here, so use atomic CAS. This code relies on the fact that anyone we race with will - * only increase (never decrease) the most recent checkpoint time value. - */ - for (;;) { - WT_ORDERED_READ(most_recent, conn->ckpt_most_recent); - if (ckpt->sec <= most_recent || - __wt_atomic_cas64(&conn->ckpt_most_recent, most_recent, ckpt->sec)) - break; - } - - /* Either load block mods from the config, or from the previous checkpoint. */ - WT_RET( - __meta_blk_mods_load(session, config, (slot == 0 ? NULL : &ckptbase[slot - 1]), ckpt, false)); - WT_ASSERT(session, ckpt->block_metadata != NULL); - - return (0); -} - -/* - * __wt_meta_saved_ckptlist_get -- - * Append the ckptlist with a new checkpoint to be added. + * __wt_meta_ckptlist_get -- + * Load all available checkpoint information for a file. Either use a cached copy of the + * checkpoints or rebuild from the metadata. */ int -__wt_meta_saved_ckptlist_get(WT_SESSION_IMPL *session, const char *fname, WT_CKPT **ckptbasep) +__wt_meta_ckptlist_get( + WT_SESSION_IMPL *session, const char *fname, bool update, WT_CKPT **ckptbasep, size_t *allocated) { WT_BTREE *btree; #ifdef HAVE_DIAGNOSTIC WT_CKPT *ckptbase_comp; #endif WT_DECL_RET; + char *config; *ckptbasep = NULL; - btree = S2BT(session); - - /* If we do not have a saved ckptlist, return not found. */ - if (btree->ckpt == NULL) - return (WT_NOTFOUND); + if (allocated != NULL) + *allocated = 0; - WT_ERR( - __meta_ckptlist_allocate_new_ckpt(session, &btree->ckpt, &btree->ckpt_bytes_allocated, NULL)); + btree = S2BT(session); + config = NULL; -#ifdef HAVE_DIAGNOSTIC /* - * Sanity check: Let's compare to a list generated from metadata. There should be no - * differences. + * Get the list of checkpoints for this file: We try to cache the ckptlist between each rebuild + * from the metadata. But there might not be one, as there are operations that can invalidate a + * ckptlist. So, use a cached ckptlist if there is one. Otherwise go through slow path of + * re-generating the ckptlist by reading the metadata. Also, we avoid using a cached checkpoint + * list for metadata itself. */ - if ((ret = __wt_meta_ckptlist_get(session, fname, true, &ckptbase_comp, NULL)) == 0) - __assert_checkpoint_list_matches(session, btree->ckpt, ckptbase_comp); - __wt_meta_ckptlist_free(session, &ckptbase_comp); - WT_ERR(ret); -#else - WT_UNUSED(fname); + if (!WT_IS_METADATA(session->dhandle) && btree->ckpt != NULL) { + *ckptbasep = btree->ckpt; + if (update) + WT_ERR(__meta_ckptlist_allocate_new_ckpt( + session, ckptbasep, &btree->ckpt_bytes_allocated, NULL)); + if (allocated != NULL) + *allocated = btree->ckpt_bytes_allocated; +#ifdef HAVE_DIAGNOSTIC + /* + * Sanity check: Let's compare to a list generated from metadata. There should be no + * differences. + */ + WT_ERR(__wt_metadata_search(session, fname, &config)); + if ((ret = __wt_meta_ckptlist_get_from_config( + session, update, &ckptbase_comp, NULL, config)) == 0) + __assert_checkpoint_list_matches(session, *ckptbasep, ckptbase_comp); + __wt_meta_ckptlist_free(session, &ckptbase_comp); + WT_ERR(ret); #endif - - /* Return the array to our caller. */ - *ckptbasep = btree->ckpt; - - if (0) { -err: - __wt_meta_saved_ckptlist_free(session); + } else { + WT_ERR(__wt_metadata_search(session, fname, &config)); + WT_ERR(__wt_meta_ckptlist_get_from_config(session, update, ckptbasep, allocated, config)); } +err: + __wt_free(session, config); return (ret); } @@ -792,6 +779,7 @@ __wt_meta_ckptlist_get_from_config(WT_SESSION_IMPL *session, bool update, WT_CKP ckpt = &ckptbase[slot]; WT_ERR(__ckpt_load(session, &k, &v, ckpt)); + WT_ERR(__wt_meta_block_metadata(session, config, ckpt)); } } WT_ERR_NOTFOUND_OK(ret, false); @@ -1141,13 +1129,15 @@ err: */ int __wt_meta_ckptlist_set( - WT_SESSION_IMPL *session, const char *fname, WT_CKPT *ckptbase, WT_LSN *ckptlsn) + WT_SESSION_IMPL *session, WT_DATA_HANDLE *dhandle, WT_CKPT *ckptbase, WT_LSN *ckptlsn) { WT_CKPT *ckpt; WT_DECL_ITEM(buf); WT_DECL_RET; + const char *fname; bool has_lsn; + fname = dhandle->name; WT_RET(__wt_scr_alloc(session, 1024, &buf)); WT_ERR(__wt_meta_ckptlist_to_meta(session, ckptbase, buf)); /* Add backup block modifications for any added checkpoint. */ @@ -1160,6 +1150,8 @@ __wt_meta_ckptlist_set( WT_ERR(__wt_buf_catfmt(session, buf, ",checkpoint_lsn=(%" PRIu32 ",%" PRIuMAX ")", ckptlsn->l.file, (uintmax_t)ckptlsn->l.offset)); + if (dhandle->type == WT_DHANDLE_TYPE_TIERED) + WT_ERR(__wt_tiered_set_metadata(session, (WT_TIERED *)dhandle, buf)); WT_ERR(__ckpt_set(session, fname, buf->mem, has_lsn)); err: @@ -1306,10 +1298,13 @@ __wt_meta_sysinfo_set(WT_SESSION_IMPL *session) __wt_verbose(session, WT_VERB_CHECKPOINT_PROGRESS, "saving checkpoint snapshot min: %" PRIu64 ", snapshot max: %" PRIu64 - " snapshot count: %" PRIu32 ", oldest timestamp: %s , meta checkpoint timestamp: %s", + " snapshot count: %" PRIu32 + ", oldest timestamp: %s , meta checkpoint timestamp: %s" + " base write gen: %" PRIu64, txn->snap_min, txn->snap_max, txn->snapshot_count, __wt_timestamp_to_string(txn_global->oldest_timestamp, ts_string[0]), - __wt_timestamp_to_string(txn_global->meta_ckpt_timestamp, ts_string[1])); + __wt_timestamp_to_string(txn_global->meta_ckpt_timestamp, ts_string[1]), + S2C(session)->base_write_gen); /* Record the base write gen in metadata as part of checkpoint */ WT_ERR(__wt_buf_fmt( diff --git a/src/third_party/wiredtiger/src/meta/meta_turtle.c b/src/third_party/wiredtiger/src/meta/meta_turtle.c index 6c0b432a067..aa2e93623c9 100644 --- a/src/third_party/wiredtiger/src/meta/meta_turtle.c +++ b/src/third_party/wiredtiger/src/meta/meta_turtle.c @@ -129,9 +129,7 @@ __metadata_load_bulk(WT_SESSION_IMPL *session) WT_ERR(cursor->get_value(cursor, &value)); filecfg[1] = value; WT_ERR(__wt_direct_io_size_check(session, filecfg, "allocation_size", &allocsize)); - WT_WITH_BUCKET_STORAGE( - NULL, session, ret = __wt_block_manager_create(session, key, allocsize)); - WT_ERR(ret); + WT_ERR(__wt_block_manager_create(session, key, allocsize)); } WT_ERR_NOTFOUND_OK(ret, false); diff --git a/src/third_party/wiredtiger/src/packing/pack_impl.c b/src/third_party/wiredtiger/src/packing/pack_impl.c index 3958c273737..650f5307fdc 100644 --- a/src/third_party/wiredtiger/src/packing/pack_impl.c +++ b/src/third_party/wiredtiger/src/packing/pack_impl.c @@ -29,13 +29,13 @@ __wt_struct_check( if (fixedp != NULL && fixed_lenp != NULL) { if (fields == 0) { - *fixedp = 1; + *fixedp = true; *fixed_lenp = 0; } else if (fields == 1 && pv.type == 't') { - *fixedp = 1; + *fixedp = true; *fixed_lenp = pv.size; } else - *fixedp = 0; + *fixedp = false; } return (0); diff --git a/src/third_party/wiredtiger/src/reconcile/rec_child.c b/src/third_party/wiredtiger/src/reconcile/rec_child.c index 56104639e53..6423eb3347d 100644 --- a/src/third_party/wiredtiger/src/reconcile/rec_child.c +++ b/src/third_party/wiredtiger/src/reconcile/rec_child.c @@ -17,7 +17,7 @@ __rec_child_deleted(WT_SESSION_IMPL *session, WT_RECONCILE *r, WT_REF *ref, WT_C { WT_PAGE_DELETED *page_del; - page_del = ref->page_del; + page_del = ref->ft_info.del; /* * Internal pages with child leaf pages in the WT_REF_DELETED state are a special case during @@ -61,18 +61,10 @@ __rec_child_deleted(WT_SESSION_IMPL *session, WT_RECONCILE *r, WT_REF *ref, WT_C * function instantiates an entirely new page.) */ if (ref->addr != NULL && !__wt_page_del_active(session, ref, true)) { - /* - * Minor memory cleanup: if a truncate call deleted this page and we were ever forced to - * instantiate the page in memory, we would have built a list of updates in the page - * reference in order to be able to commit/rollback the truncate. We just passed a - * visibility test, discard the update list. - */ - if (page_del != NULL) { - __wt_free(session, ref->page_del->update_list); - __wt_free(session, ref->page_del); - } - WT_RET(__wt_ref_block_free(session, ref)); + + /* Any fast-truncate information can be freed as soon as the delete is stable. */ + __wt_overwrite_and_free(session, ref->ft_info.del); } /* diff --git a/src/third_party/wiredtiger/src/reconcile/rec_col.c b/src/third_party/wiredtiger/src/reconcile/rec_col.c index 8c98134a6ce..417bafebb50 100644 --- a/src/third_party/wiredtiger/src/reconcile/rec_col.c +++ b/src/third_party/wiredtiger/src/reconcile/rec_col.c @@ -578,7 +578,7 @@ __wt_rec_col_var( WT_DECL_RET; WT_INSERT *ins; WT_PAGE *page; - WT_TIME_WINDOW tw, default_tw; + WT_TIME_WINDOW clear_tw, *twp; WT_UPDATE *upd; WT_UPDATE_SELECT upd_select; uint64_t n, nrepeat, repeat_count, rle, skip, src_recno; @@ -589,10 +589,11 @@ __wt_rec_col_var( btree = S2BT(session); vpack = &_vpack; page = pageref->page; + WT_TIME_WINDOW_INIT(&clear_tw); + twp = NULL; upd = NULL; size = 0; data = NULL; - WT_TIME_WINDOW_INIT(&default_tw); cbt = &r->update_modify_cbt; cbt->iface.session = (WT_SESSION *)session; @@ -602,13 +603,6 @@ __wt_rec_col_var( WT_TIME_WINDOW_INIT(&last.tw); last.deleted = false; - /* - * Set the start/stop values to cause failure if they're not set. - * [-Werror=maybe-uninitialized] - */ - /* NOLINTNEXTLINE(clang-analyzer-deadcode.DeadStores) */ - WT_TIME_WINDOW_INIT(&tw); - WT_RET(__wt_rec_split_init(session, r, page, pageref->ref_recno, btree->maxleafpage_precomp)); WT_RET(__wt_scr_alloc(session, 0, &orig)); @@ -626,7 +620,6 @@ __wt_rec_col_var( if (salvage != NULL && salvage->missing != 0) { if (salvage->skip == 0) { rle = salvage->missing; - WT_TIME_WINDOW_INIT(&last.tw); last.deleted = true; /* @@ -636,7 +629,7 @@ __wt_rec_col_var( salvage->take += salvage->missing; } else WT_ERR(__rec_col_var_helper( - session, r, NULL, NULL, &default_tw, salvage->missing, true, false)); + session, r, NULL, NULL, &clear_tw, salvage->missing, true, false)); } /* @@ -721,18 +714,16 @@ record_loop: repeat_count = WT_INSERT_RECNO(ins) - src_recno; /* - * The key on the old disk image is unchanged. If it is a deleted record or we are - * salvaging the file, clear the time window information, else take the time window - * from the cell. + * The key on the old disk image is unchanged. Clear the time window information if + * it's a deleted record, else take the time window from the cell. */ deleted = orig_deleted; - if (deleted || salvage) { - WT_TIME_WINDOW_INIT(&tw); - - if (deleted) - goto compare; - } else - WT_TIME_WINDOW_COPY(&tw, &vpack->tw); + if (deleted) { + twp = &clear_tw; + goto compare; + } + __cell_pack_kv_window_cleanup(session, page->dsk, vpack); + twp = &vpack->tw; /* * If we are handling overflow items, use the overflow item itself exactly once, @@ -755,7 +746,7 @@ record_loop: last.value->data = vpack->data; last.value->size = vpack->size; WT_ERR(__rec_col_var_helper( - session, r, salvage, last.value, &tw, repeat_count, false, true)); + session, r, salvage, last.value, twp, repeat_count, false, true)); /* Track if page has overflow items. */ r->ovfl_items = true; @@ -781,7 +772,7 @@ record_loop: break; } } else { - WT_TIME_WINDOW_COPY(&tw, &upd_select.tw); + twp = &upd_select.tw; switch (upd->type) { case WT_UPDATE_MODIFY: @@ -798,7 +789,7 @@ record_loop: size = upd->size; break; case WT_UPDATE_TOMBSTONE: - WT_TIME_WINDOW_INIT(&tw); + twp = &clear_tw; deleted = true; break; default: @@ -814,7 +805,7 @@ compare: * record number, we've been doing that all along. */ if (rle != 0) { - if (WT_TIME_WINDOWS_EQUAL(&tw, &last.tw) && + if (WT_TIME_WINDOWS_EQUAL(twp, &last.tw) && ((deleted && last.deleted) || (!deleted && !last.deleted && last.value->size == size && memcmp(last.value->data, data, size) == 0))) { @@ -849,7 +840,7 @@ compare: WT_ERR(__wt_buf_set(session, last.value, data, size)); } - WT_TIME_WINDOW_COPY(&last.tw, &tw); + WT_TIME_WINDOW_COPY(&last.tw, twp); last.deleted = deleted; rle = repeat_count; } @@ -918,14 +909,14 @@ compare: src_recno += skip; } else /* Set time window for the first deleted key in a deleted range. */ - WT_TIME_WINDOW_INIT(&tw); + twp = &clear_tw; } else if (upd == NULL) { /* The updates on the key are all uncommitted so we write a deleted key to disk. */ - WT_TIME_WINDOW_INIT(&tw); + twp = &clear_tw; deleted = true; } else { /* Set time window for the key. */ - WT_TIME_WINDOW_COPY(&tw, &upd_select.tw); + twp = &upd_select.tw; switch (upd->type) { case WT_UPDATE_MODIFY: @@ -945,7 +936,7 @@ compare: size = upd->size; break; case WT_UPDATE_TOMBSTONE: - WT_TIME_WINDOW_INIT(&tw); + twp = &clear_tw; deleted = true; break; default: @@ -958,7 +949,7 @@ compare: * the same thing. */ if (rle != 0) { - if (WT_TIME_WINDOWS_EQUAL(&last.tw, &tw) && + if (WT_TIME_WINDOWS_EQUAL(&last.tw, twp) && ((deleted && last.deleted) || (!deleted && !last.deleted && size != 0 && last.value->size == size && memcmp(last.value->data, data, size) == 0))) { @@ -990,7 +981,7 @@ compare: } /* Ready for the next loop, reset the RLE counter. */ - WT_TIME_WINDOW_COPY(&last.tw, &tw); + WT_TIME_WINDOW_COPY(&last.tw, twp); last.deleted = deleted; rle = 1; diff --git a/src/third_party/wiredtiger/src/reconcile/rec_row.c b/src/third_party/wiredtiger/src/reconcile/rec_row.c index bc4174e084a..a72bc170245 100644 --- a/src/third_party/wiredtiger/src/reconcile/rec_row.c +++ b/src/third_party/wiredtiger/src/reconcile/rec_row.c @@ -719,31 +719,46 @@ __wt_rec_row_leaf( WT_PAGE *page; WT_REC_KV *key, *val; WT_ROW *rip; - WT_TIME_WINDOW tw; + WT_TIME_WINDOW *twp; WT_UPDATE *upd; WT_UPDATE_SELECT upd_select; size_t key_size; uint64_t slvg_skip; uint32_t i; uint8_t key_prefix; - bool dictionary, key_onpage_ovfl, ovfl_key; + bool dictionary, hs_clear, key_onpage_ovfl, ovfl_key; void *copy; const void *key_data; btree = S2BT(session); - hs_cursor = NULL; page = pageref->page; + twp = NULL; + upd = NULL; slvg_skip = salvage == NULL ? 0 : salvage->skip; - WT_TIME_WINDOW_INIT(&tw); - - cbt = &r->update_modify_cbt; - cbt->iface.session = (WT_SESSION *)session; key = &r->k; val = &r->v; vpack = &_vpack; - upd = NULL; + cbt = &r->update_modify_cbt; + cbt->iface.session = (WT_SESSION *)session; + + /* + * When removing a key due to a tombstone with a durable timestamp of "none", also remove the + * history store contents associated with that key. It's safe to do even if we fail + * reconciliation after the removal, the history store content must be obsolete in order for us + * to consider removing the key. + * + * Ignore if this is metadata, as metadata doesn't have any history. + * + * Some code paths, such as schema removal, involve deleting keys in metadata and assert that + * they shouldn't open new dhandles. In those cases we won't ever need to blow away history + * store content, so we can skip this. + */ + hs_cursor = NULL; + hs_clear = F_ISSET(S2C(session), WT_CONN_HS_OPEN) && + !F_ISSET(session, WT_SESSION_NO_DATA_HANDLES) && !WT_IS_HS(btree->dhandle) && + !WT_IS_METADATA(btree->dhandle); WT_RET(__wt_rec_split_init(session, r, page, 0, btree->maxleafpage_precomp)); @@ -792,23 +807,18 @@ __wt_rec_row_leaf( WT_ERR(__wt_rec_upd_select(session, r, NULL, rip, vpack, &upd_select)); upd = upd_select.upd; - /* - * Figure out the timestamps. If there's no update and salvaging the file, clear the time - * pair information, else take the time window from the cell. - */ + /* Take the timestamp from the update or the cell. */ if (upd == NULL) { - if (!salvage) - WT_TIME_WINDOW_COPY(&tw, &vpack->tw); - else - WT_TIME_WINDOW_INIT(&tw); + __cell_pack_kv_window_cleanup(session, page->dsk, vpack); + twp = &vpack->tw; } else - WT_TIME_WINDOW_COPY(&tw, &upd_select.tw); + twp = &upd_select.tw; /* * If we reconcile an on disk key with a globally visible stop time point and there are no * new updates for that key, skip writing that key. */ - if (upd == NULL && __wt_txn_tw_stop_visible_all(session, &tw)) + if (upd == NULL && __wt_txn_tw_stop_visible_all(session, twp)) upd = &upd_tombstone; /* Build value cell. */ @@ -823,7 +833,7 @@ __wt_rec_row_leaf( * Repack the cell if we clear the transaction ids in the cell. */ if (vpack->raw == WT_CELL_VALUE_COPY) { - WT_ERR(__rec_cell_repack(session, btree, r, vpack, &tw)); + WT_ERR(__rec_cell_repack(session, btree, r, vpack, twp)); dictionary = true; } else if (F_ISSET(vpack, WT_CELL_UNPACK_TIME_WINDOW_CLEARED)) { @@ -839,10 +849,10 @@ __wt_rec_row_leaf( /* Rebuild the cell. */ val->cell_len = - __wt_cell_pack_ovfl(session, &val->cell, vpack->raw, &tw, 0, val->buf.size); + __wt_cell_pack_ovfl(session, &val->cell, vpack->raw, twp, 0, val->buf.size); val->len = val->cell_len + val->buf.size; } else - WT_ERR(__rec_cell_repack(session, btree, r, vpack, &tw)); + WT_ERR(__rec_cell_repack(session, btree, r, vpack, twp)); dictionary = true; } else { @@ -866,7 +876,7 @@ __wt_rec_row_leaf( */ WT_ASSERT(session, F_ISSET(upd, WT_UPDATE_DS) || !F_ISSET(r, WT_REC_HS) || - __wt_txn_tw_start_visible_all(session, &upd_select.tw)); + __wt_txn_tw_start_visible_all(session, twp)); /* The first time we find an overflow record, discard the underlying blocks. */ if (F_ISSET(vpack, WT_CELL_UNPACK_OVERFLOW) && vpack->raw != WT_CELL_VALUE_OVFL_RM) @@ -878,12 +888,12 @@ __wt_rec_row_leaf( WT_ERR(__wt_modify_reconstruct_from_upd_list(session, cbt, upd, cbt->upd_value)); WT_ERR(__wt_value_return(cbt, cbt->upd_value)); WT_ERR(__wt_rec_cell_build_val( - session, r, cbt->iface.value.data, cbt->iface.value.size, &tw, 0)); + session, r, cbt->iface.value.data, cbt->iface.value.size, twp, 0)); dictionary = true; break; case WT_UPDATE_STANDARD: /* Take the value from the update. */ - WT_ERR(__wt_rec_cell_build_val(session, r, upd->data, upd->size, &tw, 0)); + WT_ERR(__wt_rec_cell_build_val(session, r, upd->data, upd->size, twp, 0)); dictionary = true; break; case WT_UPDATE_TOMBSTONE: @@ -908,37 +918,22 @@ __wt_rec_row_leaf( } /* - * If we're removing a key due to a tombstone with a durable timestamp of "none", - * also remove the history store contents associated with that key. Even if we fail - * reconciliation after this point, we're safe to do this. The history store content - * must be obsolete in order for us to consider removing the key. Ignore if this is - * metadata, as metadata doesn't have any history. + * When removing a key due to a tombstone with a durable timestamp of "none", also + * remove the history store contents associated with that key. */ - if (tw.durable_stop_ts == WT_TS_NONE && F_ISSET(S2C(session), WT_CONN_HS_OPEN) && - !WT_IS_HS(btree->dhandle) && !WT_IS_METADATA(btree->dhandle)) { + if (twp->durable_stop_ts == WT_TS_NONE && hs_clear) { WT_ERR(__wt_row_leaf_key(session, page, rip, tmpkey, true)); - /* - * Start from WT_TS_NONE to delete all the history store content of the key. - * - * Some code paths, such as schema removal, involve deleting keys in metadata - * and assert that they shouldn't open new dhandles. In those cases we won't - * ever need to blow away history store content, so we can skip this. - */ - if (!F_ISSET(session, WT_SESSION_NO_DATA_HANDLES)) { - /* - * FIXME-WT-7053: we will hit the dhandle deadlock if we open multiple - * history store cursors in reconciliation. Once it is fixed, we can move - * the open and close of the history store cursor inside the delete key - * function. - */ + + /* Open a history store cursor if we don't yet have one. */ + if (hs_cursor == NULL) WT_ERR(__wt_curhs_open(session, NULL, &hs_cursor)); - WT_ERR(__wt_hs_delete_key_from_ts( - session, hs_cursor, btree->id, tmpkey, WT_TS_NONE, false)); - WT_ERR(hs_cursor->close(hs_cursor)); - hs_cursor = NULL; - WT_STAT_CONN_INCR(session, cache_hs_key_truncate_onpage_removal); - WT_STAT_DATA_INCR(session, cache_hs_key_truncate_onpage_removal); - } + + /* From WT_TS_NONE to delete all the history store content of the key. */ + WT_ERR(__wt_hs_delete_key_from_ts(session, hs_cursor, btree->id, tmpkey, + WT_TS_NONE, false, F_ISSET(r, WT_REC_CHECKPOINT_RUNNING))); + + WT_STAT_CONN_INCR(session, cache_hs_key_truncate_onpage_removal); + WT_STAT_DATA_INCR(session, cache_hs_key_truncate_onpage_removal); } /* @@ -1047,15 +1042,15 @@ build: /* Copy the key/value pair onto the page. */ __wt_rec_image_copy(session, r, key); - if (val->len == 0 && __rec_row_zero_len(session, &tw)) + if (val->len == 0 && __rec_row_zero_len(session, twp)) r->any_empty_value = true; else { r->all_empty_value = false; if (dictionary && btree->dictionary) - WT_ERR(__wt_rec_dict_replace(session, r, &tw, 0, val)); + WT_ERR(__wt_rec_dict_replace(session, r, twp, 0, val)); __wt_rec_image_copy(session, r, val); } - WT_TIME_AGGREGATE_UPDATE(session, &r->cur_ptr->ta, &tw); + WT_TIME_AGGREGATE_UPDATE(session, &r->cur_ptr->ta, twp); /* Update compression state. */ __rec_key_state_update(r, ovfl_key); diff --git a/src/third_party/wiredtiger/src/reconcile/rec_visibility.c b/src/third_party/wiredtiger/src/reconcile/rec_visibility.c index dc5748b184f..8caeac903b4 100644 --- a/src/third_party/wiredtiger/src/reconcile/rec_visibility.c +++ b/src/third_party/wiredtiger/src/reconcile/rec_visibility.c @@ -223,6 +223,32 @@ __rec_need_save_upd( } /* + * __timestamp_out_of_order_fix -- + * If we found a tombstone with a time point earlier than the update it applies to, which can + * happen if the application performs operations with timestamps out-of-order, make it invisible + * by making the start time point match the stop time point of the tombstone. We don't guarantee + * that older readers will be able to continue reading content that has been made invisible by + * out-of-order updates. Note that we carefully don't take this path when the stop time point is + * equal to the start time point. While unusual, it is permitted for a single transaction to + * insert and then remove a record. We don't want to generate a warning in that case. + */ +static inline void +__timestamp_out_of_order_fix(WT_SESSION_IMPL *session, WT_TIME_WINDOW *select_tw) +{ + char time_string[WT_TIME_STRING_SIZE]; + + if (select_tw->stop_ts < select_tw->start_ts || + (select_tw->stop_ts == select_tw->start_ts && select_tw->stop_txn < select_tw->start_txn)) { + __wt_verbose(session, WT_VERB_TIMESTAMP, + "Warning: fixing out-of-order timestamps remove earlier than value; time window %s", + __wt_time_window_to_string(select_tw, time_string)); + + select_tw->durable_start_ts = select_tw->durable_stop_ts; + select_tw->start_ts = select_tw->stop_ts; + } +} + +/* * __wt_rec_upd_select -- * Return the update in a list that should be written (or NULL if none can be written). */ @@ -234,11 +260,10 @@ __wt_rec_upd_select(WT_SESSION_IMPL *session, WT_RECONCILE *r, WT_INSERT *ins, v WT_DECL_RET; WT_PAGE *page; WT_TIME_WINDOW *select_tw; - WT_UPDATE *first_txn_upd, *first_upd, *upd, *last_upd, *same_txn_valid_upd, *tombstone; + WT_UPDATE *first_txn_upd, *first_upd, *upd, *last_upd, *tombstone; wt_timestamp_t max_ts; size_t upd_memsize; uint64_t max_txn, session_txnid, txnid; - char time_string[WT_TIME_STRING_SIZE]; bool has_newer_updates, is_hs_page, supd_restore, upd_saved; /* @@ -251,7 +276,7 @@ __wt_rec_upd_select(WT_SESSION_IMPL *session, WT_RECONCILE *r, WT_INSERT *ins, v WT_TIME_WINDOW_INIT(select_tw); page = r->page; - first_txn_upd = upd = last_upd = same_txn_valid_upd = tombstone = NULL; + first_txn_upd = upd = last_upd = tombstone = NULL; upd_memsize = 0; max_ts = WT_TS_NONE; max_txn = WT_TXN_NONE; @@ -439,71 +464,19 @@ __wt_rec_upd_select(WT_SESSION_IMPL *session, WT_RECONCILE *r, WT_INSERT *ins, v /* Find the update this tombstone applies to. */ if (!__wt_txn_upd_visible_all(session, upd)) { - /* - * Loop until a valid update from a different transaction is found in the update - * list. - */ - while (upd->next != NULL) { - if (upd->next->txnid == WT_TXN_ABORTED) - upd = upd->next; - else if (upd->next->txnid != WT_TXN_NONE && - tombstone->txnid == upd->next->txnid) { - upd = upd->next; - /* Save the latest update from the same transaction. */ - if (same_txn_valid_upd == NULL) - same_txn_valid_upd = upd; - } else - break; - } + while (upd->next != NULL && upd->next->txnid == WT_TXN_ABORTED) + upd = upd->next; WT_ASSERT(session, upd->next == NULL || upd->next->txnid != WT_TXN_ABORTED); upd_select->upd = upd = upd->next; - - /* - * If there is no on-disk update and any valid update from a different transaction - * is not found in the update list, write the same transaction update itself to disk - * to avoid blocking the eviction. - */ - if (vpack == NULL && upd == NULL) - upd_select->upd = upd = same_txn_valid_upd; - else if (upd != NULL && upd->type == WT_UPDATE_TOMBSTONE) { - /* - * The selected update from a different transaction is also a tombstone, use the - * update from the same transaction as the selected update. - */ - WT_ASSERT(session, - same_txn_valid_upd != NULL && - same_txn_valid_upd->type != WT_UPDATE_TOMBSTONE); - upd_select->upd = upd = same_txn_valid_upd; - } else if (same_txn_valid_upd != NULL && vpack != NULL && - WT_TIME_WINDOW_HAS_STOP(&vpack->tw)) { - /* - * The on-disk version has a valid stop timestamp, use the update from the same - * transaction as the selected update. - */ - WT_ASSERT(session, same_txn_valid_upd->type != WT_UPDATE_TOMBSTONE); - upd_select->upd = upd = same_txn_valid_upd; - - } else if (same_txn_valid_upd != NULL && vpack != NULL && vpack->tw.prepare) { - /* - * The on-disk version is from an aborted prepare transaction. Therefore, use - * the update from the same transaction as the selected update. We are sure that - * the on-disk prepared update has been aborted because otherwise we would have - * chosen it as an update this tombstone can be applied to. - */ - WT_ASSERT(session, same_txn_valid_upd->type != WT_UPDATE_TOMBSTONE); - upd_select->upd = upd = same_txn_valid_upd; - } } } + if (upd != NULL) /* The beginning of the validity window is the selected update's time point. */ WT_TIME_WINDOW_SET_START(select_tw, upd); else if (select_tw->stop_ts != WT_TS_NONE || select_tw->stop_txn != WT_TXN_NONE) { - /* - * We only have a tombstone on the update list or all the updates are from the same - * transaction. - */ + /* We only have a tombstone on the update list. */ WT_ASSERT(session, tombstone != NULL); /* We must have an ondisk value and it can't be a prepared update. */ @@ -511,14 +484,8 @@ __wt_rec_upd_select(WT_SESSION_IMPL *session, WT_RECONCILE *r, WT_INSERT *ins, v /* Move the pointer to the last update on the update chain. */ for (last_upd = tombstone; last_upd->next != NULL; last_upd = last_upd->next) - /* - * Tombstone is the only non-aborted update on the update chain or all the updates - * are from the same transaction. - */ - WT_ASSERT(session, - last_upd->next->txnid == WT_TXN_ABORTED || - (last_upd->next->txnid == tombstone->txnid && - last_upd->next->start_ts == tombstone->start_ts)); + /* Tombstone is the only non-aborted update on the update chain. */ + WT_ASSERT(session, last_upd->next->txnid == WT_TXN_ABORTED); /* * It's possible to have a tombstone as the only update in the update list. If we @@ -572,26 +539,7 @@ __wt_rec_upd_select(WT_SESSION_IMPL *session, WT_RECONCILE *r, WT_INSERT *ins, v } } - /* - * If we found a tombstone with a time point earlier than the update it applies to, which can - * happen if the application performs operations with timestamps out-of-order, make it invisible - * by making the start time point match the stop time point of the tombstone. We don't guarantee - * that older readers will be able to continue reading content that has been made invisible by - * out-of-order updates. - * - * Note that we carefully don't take this path when the stop time point is equal to the start - * time point. While unusual, it is permitted for a single transaction to insert and then remove - * a record. We don't want to generate a warning in that case. - */ - if (select_tw->stop_ts < select_tw->start_ts || - (select_tw->stop_ts == select_tw->start_ts && select_tw->stop_txn < select_tw->start_txn)) { - __wt_verbose(session, WT_VERB_TIMESTAMP, - "Warning: fixing out-of-order timestamps remove earlier than value; time window %s", - __wt_time_window_to_string(select_tw, time_string)); - - select_tw->durable_start_ts = select_tw->durable_stop_ts; - select_tw->start_ts = select_tw->stop_ts; - } + __timestamp_out_of_order_fix(session, select_tw); /* * Track the most recent transaction in the page. We store this in the tree at the end of @@ -652,6 +600,9 @@ __wt_rec_upd_select(WT_SESSION_IMPL *session, WT_RECONCILE *r, WT_INSERT *ins, v * Paranoia: check that we didn't choose an update that has since been rolled back. */ WT_ASSERT(session, upd_select->upd == NULL || upd_select->upd->txnid != WT_TXN_ABORTED); + /* We should never select an update that has been written to the history store. */ + WT_ASSERT(session, upd_select->upd == NULL || !F_ISSET(upd_select->upd, WT_UPDATE_HS)); + WT_ASSERT(session, tombstone == NULL || !F_ISSET(tombstone, WT_UPDATE_HS)); /* * Returning an update means the original on-page value might be lost, and that's a problem if diff --git a/src/third_party/wiredtiger/src/reconcile/rec_write.c b/src/third_party/wiredtiger/src/reconcile/rec_write.c index 24fe6bb252d..667e8c119b7 100644 --- a/src/third_party/wiredtiger/src/reconcile/rec_write.c +++ b/src/third_party/wiredtiger/src/reconcile/rec_write.c @@ -2316,7 +2316,8 @@ __rec_hs_wrapup(WT_SESSION_IMPL *session, WT_RECONCILE *r) for (multi = r->multi, i = 0; i < r->multi_next; ++multi, ++i) if (multi->supd != NULL) { - WT_ERR(__wt_hs_insert_updates(session, r->page, multi, &r->cache_write_hs)); + WT_ERR(__wt_hs_insert_updates( + session, r->page, multi, &r->cache_write_hs, F_ISSET(r, WT_REC_CHECKPOINT_RUNNING))); if (!multi->supd_restore) { __wt_free(session, multi->supd); multi->supd_entries = 0; diff --git a/src/third_party/wiredtiger/src/schema/schema_create.c b/src/third_party/wiredtiger/src/schema/schema_create.c index c814a32c935..be3f993f10d 100644 --- a/src/third_party/wiredtiger/src/schema/schema_create.c +++ b/src/third_party/wiredtiger/src/schema/schema_create.c @@ -864,8 +864,11 @@ __create_tiered(WT_SESSION_IMPL *session, const char *uri, bool exclusive, const * By default use the connection level bucket and prefix. Then we add in any user * configuration that may override the system one. */ - WT_ERR(__wt_buf_fmt(session, tmp, ",tiered_storage=(bucket=%s,bucket_prefix=%s)", - conn->bstorage->bucket, conn->bstorage->bucket_prefix)); + WT_ERR(__wt_buf_fmt(session, tmp, + ",tiered_storage=(bucket=%s,bucket_prefix=%s)" + ",id=%" PRIu32 ",version=(major=%d,minor=%d),checkpoint_lsn=", + conn->bstorage->bucket, conn->bstorage->bucket_prefix, ++conn->next_file_id, + WT_BTREE_MAJOR_VERSION_MAX, WT_BTREE_MINOR_VERSION_MAX)); cfg[1] = tmp->data; cfg[2] = config; cfg[3] = "tiers=()"; diff --git a/src/third_party/wiredtiger/src/schema/schema_rename.c b/src/third_party/wiredtiger/src/schema/schema_rename.c index b0ce8fc3ed7..73794a40f7b 100644 --- a/src/third_party/wiredtiger/src/schema/schema_rename.c +++ b/src/third_party/wiredtiger/src/schema/schema_rename.c @@ -259,23 +259,11 @@ err: static int __rename_tiered(WT_SESSION_IMPL *session, const char *olduri, const char *newuri, const char *cfg[]) { - WT_DECL_RET; - WT_TIERED *tiered; - - /* Get the tiered data handle. */ - WT_RET(__wt_session_get_dhandle(session, olduri, NULL, NULL, WT_DHANDLE_EXCLUSIVE)); - tiered = (WT_TIERED *)session->dhandle; - - /* TODO */ WT_UNUSED(olduri); WT_UNUSED(newuri); WT_UNUSED(cfg); - WT_UNUSED(tiered); - - F_SET(session->dhandle, WT_DHANDLE_DISCARD); - WT_TRET(__wt_session_release_dhandle(session)); - - return (ret); + /* We do not allow renaming a tiered table. */ + WT_RET_MSG(session, EINVAL, "rename of tiered table is not supported"); } /* diff --git a/src/third_party/wiredtiger/src/session/session_api.c b/src/third_party/wiredtiger/src/session/session_api.c index d47cfa46af8..513b466d0a9 100644 --- a/src/third_party/wiredtiger/src/session/session_api.c +++ b/src/third_party/wiredtiger/src/session/session_api.c @@ -1174,6 +1174,19 @@ err: } /* + * __session_salvage_worker -- + * Wrapper function for salvage processing. + */ +static int +__session_salvage_worker(WT_SESSION_IMPL *session, const char *uri, const char *cfg[]) +{ + WT_RET(__wt_schema_worker( + session, uri, __wt_salvage, NULL, cfg, WT_DHANDLE_EXCLUSIVE | WT_BTREE_SALVAGE)); + WT_RET(__wt_schema_worker(session, uri, NULL, __wt_rollback_to_stable_one, cfg, 0)); + return (0); +} + +/* * __session_salvage -- * WT_SESSION->salvage method. */ @@ -1189,11 +1202,17 @@ __session_salvage(WT_SESSION *wt_session, const char *uri, const char *config) WT_ERR(__wt_inmem_unsupported_op(session, NULL)); - /* Block out checkpoints to avoid spurious EBUSY errors. */ - WT_WITH_CHECKPOINT_LOCK(session, - WT_WITH_SCHEMA_LOCK(session, - ret = __wt_schema_worker( - session, uri, __wt_salvage, NULL, cfg, WT_DHANDLE_EXCLUSIVE | WT_BTREE_SALVAGE))); + /* + * Run salvage and then rollback-to-stable (to bring the object into compliance with database + * timestamps). + * + * Block out checkpoints to avoid spurious EBUSY errors. + * + * Hold the schema lock across both salvage and rollback-to-stable to avoid races where another + * thread opens the handle before rollback-to-stable completes. + */ + WT_WITH_CHECKPOINT_LOCK( + session, WT_WITH_SCHEMA_LOCK(session, ret = __session_salvage_worker(session, uri, cfg))); err: if (ret != 0) diff --git a/src/third_party/wiredtiger/src/support/lock_ext.c b/src/third_party/wiredtiger/src/support/lock_ext.c new file mode 100644 index 00000000000..38d338892c1 --- /dev/null +++ b/src/third_party/wiredtiger/src/support/lock_ext.c @@ -0,0 +1,84 @@ +/*- + * Copyright (c) 2014-present MongoDB, Inc. + * Copyright (c) 2008-2014 WiredTiger, Inc. + * All rights reserved. + * + * See the file LICENSE for redistribution information. + */ + +#include "wt_internal.h" + +/* + * __wt_ext_spin_init -- + * Allocate and initialize a spinlock. + */ +int +__wt_ext_spin_init(WT_EXTENSION_API *wt_api, WT_EXTENSION_SPINLOCK *ext_spinlock, const char *name) +{ + WT_DECL_RET; + WT_SESSION_IMPL *default_session; + WT_SPINLOCK *lock; + + ext_spinlock->spinlock = NULL; + default_session = ((WT_CONNECTION_IMPL *)wt_api->conn)->default_session; + if ((ret = __wt_calloc_one(default_session, &lock)) != 0) + return ret; + if ((ret = __wt_spin_init(default_session, lock, name)) != 0) { + __wt_free(default_session, lock); + return ret; + } + ext_spinlock->spinlock = lock; + return (0); +} + +/* + * __wt_ext_spin_lock -- + * Lock the spinlock. + */ +void +__wt_ext_spin_lock( + WT_EXTENSION_API *wt_api, WT_SESSION *session, WT_EXTENSION_SPINLOCK *ext_spinlock) +{ + WT_SPINLOCK *lock; + + WT_UNUSED(wt_api); /* Unused parameters */ + lock = ((WT_SPINLOCK *)ext_spinlock->spinlock); + __wt_spin_lock((WT_SESSION_IMPL *)session, lock); + return; +} + +/* + * __wt_ext_spin_unlock -- + * Unlock the spinlock. + */ +void +__wt_ext_spin_unlock( + WT_EXTENSION_API *wt_api, WT_SESSION *session, WT_EXTENSION_SPINLOCK *ext_spinlock) +{ + WT_SPINLOCK *lock; + + WT_UNUSED(wt_api); /* Unused parameters */ + lock = ((WT_SPINLOCK *)ext_spinlock->spinlock); + __wt_spin_unlock((WT_SESSION_IMPL *)session, lock); + return; +} + +/* + * __wt_ext_spin_destroy -- + * Destroy the spinlock. + */ +void +__wt_ext_spin_destroy(WT_EXTENSION_API *wt_api, WT_EXTENSION_SPINLOCK *ext_spinlock) +{ + WT_SESSION_IMPL *default_session; + WT_SPINLOCK *lock; + + lock = ((WT_SPINLOCK *)ext_spinlock->spinlock); + + /* Default session is used to comply with the lock initialization. */ + default_session = ((WT_CONNECTION_IMPL *)wt_api->conn)->default_session; + __wt_spin_destroy(default_session, lock); + __wt_free(default_session, lock); + ext_spinlock->spinlock = NULL; + return; +} diff --git a/src/third_party/wiredtiger/src/support/stat.c b/src/third_party/wiredtiger/src/support/stat.c index 912e00945f5..a7f7c70f9c6 100644 --- a/src/third_party/wiredtiger/src/support/stat.c +++ b/src/third_party/wiredtiger/src/support/stat.c @@ -12,6 +12,8 @@ static const char *const __stats_dsrc_desc[] = { "LSM: chunks in the LSM tree", "LSM: highest merge generation in the LSM tree", "LSM: queries that could have benefited from a Bloom filter that did not exist", + "LSM: sleep for LSM checkpoint throttle", + "LSM: sleep for LSM merge throttle", "LSM: total size of bloom filters", "block-manager: allocations requiring file extension", "block-manager: blocks allocated", @@ -43,82 +45,14 @@ static const char *const __stats_dsrc_desc[] = { "btree: row-store empty values", "btree: row-store internal pages", "btree: row-store leaf pages", - "cache: data source pages selected for eviction unable to be evicted", - "cache: eviction walk passes of a file", - "cache_walk: Average difference between current eviction generation when the page was last " - "considered", - "cache_walk: Average on-disk page image size seen", - "cache_walk: Average time in cache for pages that have been visited by the eviction server", - "cache_walk: Average time in cache for pages that have not been visited by the eviction server", - "cache_walk: Clean pages currently in cache", - "cache_walk: Current eviction generation", - "cache_walk: Dirty pages currently in cache", - "cache_walk: Entries in the root page", - "cache_walk: Internal pages currently in cache", - "cache_walk: Leaf pages currently in cache", - "cache_walk: Maximum difference between current eviction generation when the page was last " - "considered", - "cache_walk: Maximum page size seen", - "cache_walk: Minimum on-disk page image size seen", - "cache_walk: Number of pages never visited by eviction server", - "cache_walk: On-disk page image sizes smaller than a single allocation unit", - "cache_walk: Pages created in memory and never written", - "cache_walk: Pages currently queued for eviction", - "cache_walk: Pages that could not be queued for eviction", - "cache_walk: Refs skipped during cache traversal", - "cache_walk: Size of the root page", - "cache_walk: Total number of pages currently in cache", - "compression: compressed page maximum internal page size prior to compression", - "compression: compressed page maximum leaf page size prior to compression ", - "compression: compressed pages read", - "compression: compressed pages written", - "compression: page written failed to compress", - "compression: page written was too small to compress", - "cursor: bulk loaded cursor insert calls", - "cursor: cache cursors reuse count", - "cursor: close calls that result in cache", - "cursor: create calls", - "cursor: insert calls", - "cursor: insert key and value bytes", - "cursor: modify", - "cursor: modify key and value bytes affected", - "cursor: modify value bytes modified", - "cursor: next calls", - "cursor: operation restarted", - "cursor: prev calls", - "cursor: remove calls", - "cursor: remove key bytes removed", - "cursor: reserve calls", - "cursor: reset calls", - "cursor: search calls", - "cursor: search history store calls", - "cursor: search near calls", - "cursor: truncate calls", - "cursor: update calls", - "cursor: update key and value bytes", - "cursor: update value size change", - "reconciliation: dictionary matches", - "reconciliation: internal page key bytes discarded using suffix compression", - "reconciliation: internal page multi-block writes", - "reconciliation: internal-page overflow keys", - "reconciliation: leaf page key bytes discarded using prefix compression", - "reconciliation: leaf page multi-block writes", - "reconciliation: leaf-page overflow keys", - "reconciliation: maximum blocks required for a page", - "reconciliation: overflow values written", - "reconciliation: page checksum matches", - "reconciliation: pages written including at least one prepare", - "reconciliation: pages written including at least one start timestamp", - "reconciliation: records written including a prepare", - "session: object compaction", - "LSM: sleep for LSM checkpoint throttle", - "LSM: sleep for LSM merge throttle", "cache: bytes currently in the cache", "cache: bytes dirty in the cache cumulative", "cache: bytes read into cache", "cache: bytes written from cache", "cache: checkpoint blocked page eviction", "cache: checkpoint of history store file blocked non-history store page eviction", + "cache: data source pages selected for eviction unable to be evicted", + "cache: eviction walk passes of a file", "cache: eviction walk target pages histogram - 0-9", "cache: eviction walk target pages histogram - 10-31", "cache: eviction walk target pages histogram - 128 and higher", @@ -168,24 +102,88 @@ static const char *const __stats_dsrc_desc[] = { "cache: pages written requiring in-memory restoration", "cache: tracked dirty bytes in the cache", "cache: unmodified pages evicted", + "cache_walk: Average difference between current eviction generation when the page was last " + "considered", + "cache_walk: Average on-disk page image size seen", + "cache_walk: Average time in cache for pages that have been visited by the eviction server", + "cache_walk: Average time in cache for pages that have not been visited by the eviction server", + "cache_walk: Clean pages currently in cache", + "cache_walk: Current eviction generation", + "cache_walk: Dirty pages currently in cache", + "cache_walk: Entries in the root page", + "cache_walk: Internal pages currently in cache", + "cache_walk: Leaf pages currently in cache", + "cache_walk: Maximum difference between current eviction generation when the page was last " + "considered", + "cache_walk: Maximum page size seen", + "cache_walk: Minimum on-disk page image size seen", + "cache_walk: Number of pages never visited by eviction server", + "cache_walk: On-disk page image sizes smaller than a single allocation unit", + "cache_walk: Pages created in memory and never written", + "cache_walk: Pages currently queued for eviction", + "cache_walk: Pages that could not be queued for eviction", + "cache_walk: Refs skipped during cache traversal", + "cache_walk: Size of the root page", + "cache_walk: Total number of pages currently in cache", "checkpoint-cleanup: pages added for eviction", "checkpoint-cleanup: pages removed", "checkpoint-cleanup: pages skipped during tree walk", "checkpoint-cleanup: pages visited", + "compression: compressed page maximum internal page size prior to compression", + "compression: compressed page maximum leaf page size prior to compression ", + "compression: compressed pages read", + "compression: compressed pages written", + "compression: page written failed to compress", + "compression: page written was too small to compress", "cursor: Total number of entries skipped by cursor next calls", "cursor: Total number of entries skipped by cursor prev calls", "cursor: Total number of entries skipped to position the history store cursor", + "cursor: Total number of pages skipped without reading by cursor next calls", + "cursor: Total number of pages skipped without reading by cursor prev calls", "cursor: Total number of times a search near has exited due to prefix config", + "cursor: bulk loaded cursor insert calls", + "cursor: cache cursors reuse count", + "cursor: close calls that result in cache", + "cursor: create calls", "cursor: cursor next calls that skip due to a globally visible history store tombstone", "cursor: cursor next calls that skip greater than or equal to 100 entries", "cursor: cursor next calls that skip less than 100 entries", "cursor: cursor prev calls that skip due to a globally visible history store tombstone", "cursor: cursor prev calls that skip greater than or equal to 100 entries", "cursor: cursor prev calls that skip less than 100 entries", + "cursor: insert calls", + "cursor: insert key and value bytes", + "cursor: modify", + "cursor: modify key and value bytes affected", + "cursor: modify value bytes modified", + "cursor: next calls", "cursor: open cursor count", + "cursor: operation restarted", + "cursor: prev calls", + "cursor: remove calls", + "cursor: remove key bytes removed", + "cursor: reserve calls", + "cursor: reset calls", + "cursor: search calls", + "cursor: search history store calls", + "cursor: search near calls", + "cursor: truncate calls", + "cursor: update calls", + "cursor: update key and value bytes", + "cursor: update value size change", "reconciliation: approximate byte size of timestamps in pages written", "reconciliation: approximate byte size of transaction IDs in pages written", + "reconciliation: dictionary matches", "reconciliation: fast-path pages deleted", + "reconciliation: internal page key bytes discarded using suffix compression", + "reconciliation: internal page multi-block writes", + "reconciliation: internal-page overflow keys", + "reconciliation: leaf page key bytes discarded using prefix compression", + "reconciliation: leaf page multi-block writes", + "reconciliation: leaf-page overflow keys", + "reconciliation: maximum blocks required for a page", + "reconciliation: overflow values written", + "reconciliation: page checksum matches", "reconciliation: page reconciliation calls", "reconciliation: page reconciliation calls for eviction", "reconciliation: pages deleted", @@ -196,17 +194,21 @@ static const char *const __stats_dsrc_desc[] = { "reconciliation: pages written including an aggregated newest transaction ID ", "reconciliation: pages written including an aggregated oldest start timestamp ", "reconciliation: pages written including an aggregated prepare", + "reconciliation: pages written including at least one prepare", "reconciliation: pages written including at least one start durable timestamp", + "reconciliation: pages written including at least one start timestamp", "reconciliation: pages written including at least one start transaction ID", "reconciliation: pages written including at least one stop durable timestamp", "reconciliation: pages written including at least one stop timestamp", "reconciliation: pages written including at least one stop transaction ID", + "reconciliation: records written including a prepare", "reconciliation: records written including a start durable timestamp", "reconciliation: records written including a start timestamp", "reconciliation: records written including a start transaction ID", "reconciliation: records written including a stop durable timestamp", "reconciliation: records written including a stop timestamp", "reconciliation: records written including a stop transaction ID", + "session: object compaction", "session: tiered operations dequeued and processed", "session: tiered operations scheduled", "session: tiered storage local retention time (secs)", @@ -272,6 +274,8 @@ __wt_stat_dsrc_clear_single(WT_DSRC_STATS *stats) stats->lsm_chunk_count = 0; stats->lsm_generation_max = 0; stats->lsm_lookup_no_bloom = 0; + stats->lsm_checkpoint_throttle = 0; + stats->lsm_merge_throttle = 0; stats->bloom_size = 0; stats->block_extension = 0; stats->block_alloc = 0; @@ -303,80 +307,14 @@ __wt_stat_dsrc_clear_single(WT_DSRC_STATS *stats) stats->btree_row_empty_values = 0; stats->btree_row_internal = 0; stats->btree_row_leaf = 0; - stats->cache_eviction_fail = 0; - stats->cache_eviction_walk_passes = 0; - /* not clearing cache_state_gen_avg_gap */ - /* not clearing cache_state_avg_written_size */ - /* not clearing cache_state_avg_visited_age */ - /* not clearing cache_state_avg_unvisited_age */ - /* not clearing cache_state_pages_clean */ - /* not clearing cache_state_gen_current */ - /* not clearing cache_state_pages_dirty */ - /* not clearing cache_state_root_entries */ - /* not clearing cache_state_pages_internal */ - /* not clearing cache_state_pages_leaf */ - /* not clearing cache_state_gen_max_gap */ - /* not clearing cache_state_max_pagesize */ - /* not clearing cache_state_min_written_size */ - /* not clearing cache_state_unvisited_count */ - /* not clearing cache_state_smaller_alloc_size */ - /* not clearing cache_state_memory */ - /* not clearing cache_state_queued */ - /* not clearing cache_state_not_queueable */ - /* not clearing cache_state_refs_skipped */ - /* not clearing cache_state_root_size */ - /* not clearing cache_state_pages */ - /* not clearing compress_precomp_intl_max_page_size */ - /* not clearing compress_precomp_leaf_max_page_size */ - stats->compress_read = 0; - stats->compress_write = 0; - stats->compress_write_fail = 0; - stats->compress_write_too_small = 0; - stats->cursor_insert_bulk = 0; - stats->cursor_reopen = 0; - stats->cursor_cache = 0; - stats->cursor_create = 0; - stats->cursor_insert = 0; - stats->cursor_insert_bytes = 0; - stats->cursor_modify = 0; - stats->cursor_modify_bytes = 0; - stats->cursor_modify_bytes_touch = 0; - stats->cursor_next = 0; - stats->cursor_restart = 0; - stats->cursor_prev = 0; - stats->cursor_remove = 0; - stats->cursor_remove_bytes = 0; - stats->cursor_reserve = 0; - stats->cursor_reset = 0; - stats->cursor_search = 0; - stats->cursor_search_hs = 0; - stats->cursor_search_near = 0; - stats->cursor_truncate = 0; - stats->cursor_update = 0; - stats->cursor_update_bytes = 0; - stats->cursor_update_bytes_changed = 0; - stats->rec_dictionary = 0; - stats->rec_suffix_compression = 0; - stats->rec_multiblock_internal = 0; - stats->rec_overflow_key_internal = 0; - stats->rec_prefix_compression = 0; - stats->rec_multiblock_leaf = 0; - stats->rec_overflow_key_leaf = 0; - stats->rec_multiblock_max = 0; - stats->rec_overflow_value = 0; - stats->rec_page_match = 0; - stats->rec_time_window_pages_prepared = 0; - stats->rec_time_window_pages_start_ts = 0; - stats->rec_time_window_prepared = 0; - stats->session_compact = 0; - stats->lsm_checkpoint_throttle = 0; - stats->lsm_merge_throttle = 0; /* not clearing cache_bytes_inuse */ /* not clearing cache_bytes_dirty_total */ stats->cache_bytes_read = 0; stats->cache_bytes_write = 0; stats->cache_eviction_checkpoint = 0; stats->cache_eviction_blocked_checkpoint_hs = 0; + stats->cache_eviction_fail = 0; + stats->cache_eviction_walk_passes = 0; stats->cache_eviction_target_page_lt10 = 0; stats->cache_eviction_target_page_lt32 = 0; stats->cache_eviction_target_page_ge128 = 0; @@ -423,24 +361,86 @@ __wt_stat_dsrc_clear_single(WT_DSRC_STATS *stats) stats->cache_write_restore = 0; /* not clearing cache_bytes_dirty */ stats->cache_eviction_clean = 0; + /* not clearing cache_state_gen_avg_gap */ + /* not clearing cache_state_avg_written_size */ + /* not clearing cache_state_avg_visited_age */ + /* not clearing cache_state_avg_unvisited_age */ + /* not clearing cache_state_pages_clean */ + /* not clearing cache_state_gen_current */ + /* not clearing cache_state_pages_dirty */ + /* not clearing cache_state_root_entries */ + /* not clearing cache_state_pages_internal */ + /* not clearing cache_state_pages_leaf */ + /* not clearing cache_state_gen_max_gap */ + /* not clearing cache_state_max_pagesize */ + /* not clearing cache_state_min_written_size */ + /* not clearing cache_state_unvisited_count */ + /* not clearing cache_state_smaller_alloc_size */ + /* not clearing cache_state_memory */ + /* not clearing cache_state_queued */ + /* not clearing cache_state_not_queueable */ + /* not clearing cache_state_refs_skipped */ + /* not clearing cache_state_root_size */ + /* not clearing cache_state_pages */ stats->cc_pages_evict = 0; stats->cc_pages_removed = 0; stats->cc_pages_walk_skipped = 0; stats->cc_pages_visited = 0; + /* not clearing compress_precomp_intl_max_page_size */ + /* not clearing compress_precomp_leaf_max_page_size */ + stats->compress_read = 0; + stats->compress_write = 0; + stats->compress_write_fail = 0; + stats->compress_write_too_small = 0; stats->cursor_next_skip_total = 0; stats->cursor_prev_skip_total = 0; stats->cursor_skip_hs_cur_position = 0; + stats->cursor_next_skip_page_count = 0; + stats->cursor_prev_skip_page_count = 0; stats->cursor_search_near_prefix_fast_paths = 0; + stats->cursor_insert_bulk = 0; + stats->cursor_reopen = 0; + stats->cursor_cache = 0; + stats->cursor_create = 0; stats->cursor_next_hs_tombstone = 0; stats->cursor_next_skip_ge_100 = 0; stats->cursor_next_skip_lt_100 = 0; stats->cursor_prev_hs_tombstone = 0; stats->cursor_prev_skip_ge_100 = 0; stats->cursor_prev_skip_lt_100 = 0; + stats->cursor_insert = 0; + stats->cursor_insert_bytes = 0; + stats->cursor_modify = 0; + stats->cursor_modify_bytes = 0; + stats->cursor_modify_bytes_touch = 0; + stats->cursor_next = 0; /* not clearing cursor_open_count */ + stats->cursor_restart = 0; + stats->cursor_prev = 0; + stats->cursor_remove = 0; + stats->cursor_remove_bytes = 0; + stats->cursor_reserve = 0; + stats->cursor_reset = 0; + stats->cursor_search = 0; + stats->cursor_search_hs = 0; + stats->cursor_search_near = 0; + stats->cursor_truncate = 0; + stats->cursor_update = 0; + stats->cursor_update_bytes = 0; + stats->cursor_update_bytes_changed = 0; stats->rec_time_window_bytes_ts = 0; stats->rec_time_window_bytes_txn = 0; + stats->rec_dictionary = 0; stats->rec_page_delete_fast = 0; + stats->rec_suffix_compression = 0; + stats->rec_multiblock_internal = 0; + stats->rec_overflow_key_internal = 0; + stats->rec_prefix_compression = 0; + stats->rec_multiblock_leaf = 0; + stats->rec_overflow_key_leaf = 0; + stats->rec_multiblock_max = 0; + stats->rec_overflow_value = 0; + stats->rec_page_match = 0; stats->rec_pages = 0; stats->rec_pages_eviction = 0; stats->rec_page_delete = 0; @@ -451,17 +451,21 @@ __wt_stat_dsrc_clear_single(WT_DSRC_STATS *stats) stats->rec_time_aggr_newest_txn = 0; stats->rec_time_aggr_oldest_start_ts = 0; stats->rec_time_aggr_prepared = 0; + stats->rec_time_window_pages_prepared = 0; stats->rec_time_window_pages_durable_start_ts = 0; + stats->rec_time_window_pages_start_ts = 0; stats->rec_time_window_pages_start_txn = 0; stats->rec_time_window_pages_durable_stop_ts = 0; stats->rec_time_window_pages_stop_ts = 0; stats->rec_time_window_pages_stop_txn = 0; + stats->rec_time_window_prepared = 0; stats->rec_time_window_durable_start_ts = 0; stats->rec_time_window_start_ts = 0; stats->rec_time_window_start_txn = 0; stats->rec_time_window_durable_stop_ts = 0; stats->rec_time_window_stop_ts = 0; stats->rec_time_window_stop_txn = 0; + stats->session_compact = 0; stats->tiered_work_units_dequeued = 0; stats->tiered_work_units_created = 0; /* not clearing tiered_retention */ @@ -501,6 +505,8 @@ __wt_stat_dsrc_aggregate_single(WT_DSRC_STATS *from, WT_DSRC_STATS *to) if (from->lsm_generation_max > to->lsm_generation_max) to->lsm_generation_max = from->lsm_generation_max; to->lsm_lookup_no_bloom += from->lsm_lookup_no_bloom; + to->lsm_checkpoint_throttle += from->lsm_checkpoint_throttle; + to->lsm_merge_throttle += from->lsm_merge_throttle; to->bloom_size += from->bloom_size; to->block_extension += from->block_extension; to->block_alloc += from->block_alloc; @@ -543,81 +549,14 @@ __wt_stat_dsrc_aggregate_single(WT_DSRC_STATS *from, WT_DSRC_STATS *to) to->btree_row_empty_values += from->btree_row_empty_values; to->btree_row_internal += from->btree_row_internal; to->btree_row_leaf += from->btree_row_leaf; - to->cache_eviction_fail += from->cache_eviction_fail; - to->cache_eviction_walk_passes += from->cache_eviction_walk_passes; - to->cache_state_gen_avg_gap += from->cache_state_gen_avg_gap; - to->cache_state_avg_written_size += from->cache_state_avg_written_size; - to->cache_state_avg_visited_age += from->cache_state_avg_visited_age; - to->cache_state_avg_unvisited_age += from->cache_state_avg_unvisited_age; - to->cache_state_pages_clean += from->cache_state_pages_clean; - to->cache_state_gen_current += from->cache_state_gen_current; - to->cache_state_pages_dirty += from->cache_state_pages_dirty; - to->cache_state_root_entries += from->cache_state_root_entries; - to->cache_state_pages_internal += from->cache_state_pages_internal; - to->cache_state_pages_leaf += from->cache_state_pages_leaf; - to->cache_state_gen_max_gap += from->cache_state_gen_max_gap; - to->cache_state_max_pagesize += from->cache_state_max_pagesize; - to->cache_state_min_written_size += from->cache_state_min_written_size; - to->cache_state_unvisited_count += from->cache_state_unvisited_count; - to->cache_state_smaller_alloc_size += from->cache_state_smaller_alloc_size; - to->cache_state_memory += from->cache_state_memory; - to->cache_state_queued += from->cache_state_queued; - to->cache_state_not_queueable += from->cache_state_not_queueable; - to->cache_state_refs_skipped += from->cache_state_refs_skipped; - to->cache_state_root_size += from->cache_state_root_size; - to->cache_state_pages += from->cache_state_pages; - to->compress_precomp_intl_max_page_size += from->compress_precomp_intl_max_page_size; - to->compress_precomp_leaf_max_page_size += from->compress_precomp_leaf_max_page_size; - to->compress_read += from->compress_read; - to->compress_write += from->compress_write; - to->compress_write_fail += from->compress_write_fail; - to->compress_write_too_small += from->compress_write_too_small; - to->cursor_insert_bulk += from->cursor_insert_bulk; - to->cursor_reopen += from->cursor_reopen; - to->cursor_cache += from->cursor_cache; - to->cursor_create += from->cursor_create; - to->cursor_insert += from->cursor_insert; - to->cursor_insert_bytes += from->cursor_insert_bytes; - to->cursor_modify += from->cursor_modify; - to->cursor_modify_bytes += from->cursor_modify_bytes; - to->cursor_modify_bytes_touch += from->cursor_modify_bytes_touch; - to->cursor_next += from->cursor_next; - to->cursor_restart += from->cursor_restart; - to->cursor_prev += from->cursor_prev; - to->cursor_remove += from->cursor_remove; - to->cursor_remove_bytes += from->cursor_remove_bytes; - to->cursor_reserve += from->cursor_reserve; - to->cursor_reset += from->cursor_reset; - to->cursor_search += from->cursor_search; - to->cursor_search_hs += from->cursor_search_hs; - to->cursor_search_near += from->cursor_search_near; - to->cursor_truncate += from->cursor_truncate; - to->cursor_update += from->cursor_update; - to->cursor_update_bytes += from->cursor_update_bytes; - to->cursor_update_bytes_changed += from->cursor_update_bytes_changed; - to->rec_dictionary += from->rec_dictionary; - to->rec_suffix_compression += from->rec_suffix_compression; - to->rec_multiblock_internal += from->rec_multiblock_internal; - to->rec_overflow_key_internal += from->rec_overflow_key_internal; - to->rec_prefix_compression += from->rec_prefix_compression; - to->rec_multiblock_leaf += from->rec_multiblock_leaf; - to->rec_overflow_key_leaf += from->rec_overflow_key_leaf; - if (from->rec_multiblock_max > to->rec_multiblock_max) - to->rec_multiblock_max = from->rec_multiblock_max; - to->rec_overflow_value += from->rec_overflow_value; - to->rec_page_match += from->rec_page_match; - to->rec_time_window_pages_prepared += from->rec_time_window_pages_prepared; - to->rec_time_window_pages_start_ts += from->rec_time_window_pages_start_ts; - to->rec_time_window_prepared += from->rec_time_window_prepared; - to->session_compact += from->session_compact; - to->lsm_checkpoint_throttle += from->lsm_checkpoint_throttle; - to->lsm_merge_throttle += from->lsm_merge_throttle; to->cache_bytes_inuse += from->cache_bytes_inuse; to->cache_bytes_dirty_total += from->cache_bytes_dirty_total; to->cache_bytes_read += from->cache_bytes_read; to->cache_bytes_write += from->cache_bytes_write; to->cache_eviction_checkpoint += from->cache_eviction_checkpoint; to->cache_eviction_blocked_checkpoint_hs += from->cache_eviction_blocked_checkpoint_hs; + to->cache_eviction_fail += from->cache_eviction_fail; + to->cache_eviction_walk_passes += from->cache_eviction_walk_passes; to->cache_eviction_target_page_lt10 += from->cache_eviction_target_page_lt10; to->cache_eviction_target_page_lt32 += from->cache_eviction_target_page_lt32; to->cache_eviction_target_page_ge128 += from->cache_eviction_target_page_ge128; @@ -664,24 +603,87 @@ __wt_stat_dsrc_aggregate_single(WT_DSRC_STATS *from, WT_DSRC_STATS *to) to->cache_write_restore += from->cache_write_restore; to->cache_bytes_dirty += from->cache_bytes_dirty; to->cache_eviction_clean += from->cache_eviction_clean; + to->cache_state_gen_avg_gap += from->cache_state_gen_avg_gap; + to->cache_state_avg_written_size += from->cache_state_avg_written_size; + to->cache_state_avg_visited_age += from->cache_state_avg_visited_age; + to->cache_state_avg_unvisited_age += from->cache_state_avg_unvisited_age; + to->cache_state_pages_clean += from->cache_state_pages_clean; + to->cache_state_gen_current += from->cache_state_gen_current; + to->cache_state_pages_dirty += from->cache_state_pages_dirty; + to->cache_state_root_entries += from->cache_state_root_entries; + to->cache_state_pages_internal += from->cache_state_pages_internal; + to->cache_state_pages_leaf += from->cache_state_pages_leaf; + to->cache_state_gen_max_gap += from->cache_state_gen_max_gap; + to->cache_state_max_pagesize += from->cache_state_max_pagesize; + to->cache_state_min_written_size += from->cache_state_min_written_size; + to->cache_state_unvisited_count += from->cache_state_unvisited_count; + to->cache_state_smaller_alloc_size += from->cache_state_smaller_alloc_size; + to->cache_state_memory += from->cache_state_memory; + to->cache_state_queued += from->cache_state_queued; + to->cache_state_not_queueable += from->cache_state_not_queueable; + to->cache_state_refs_skipped += from->cache_state_refs_skipped; + to->cache_state_root_size += from->cache_state_root_size; + to->cache_state_pages += from->cache_state_pages; to->cc_pages_evict += from->cc_pages_evict; to->cc_pages_removed += from->cc_pages_removed; to->cc_pages_walk_skipped += from->cc_pages_walk_skipped; to->cc_pages_visited += from->cc_pages_visited; + to->compress_precomp_intl_max_page_size += from->compress_precomp_intl_max_page_size; + to->compress_precomp_leaf_max_page_size += from->compress_precomp_leaf_max_page_size; + to->compress_read += from->compress_read; + to->compress_write += from->compress_write; + to->compress_write_fail += from->compress_write_fail; + to->compress_write_too_small += from->compress_write_too_small; to->cursor_next_skip_total += from->cursor_next_skip_total; to->cursor_prev_skip_total += from->cursor_prev_skip_total; to->cursor_skip_hs_cur_position += from->cursor_skip_hs_cur_position; + to->cursor_next_skip_page_count += from->cursor_next_skip_page_count; + to->cursor_prev_skip_page_count += from->cursor_prev_skip_page_count; to->cursor_search_near_prefix_fast_paths += from->cursor_search_near_prefix_fast_paths; + to->cursor_insert_bulk += from->cursor_insert_bulk; + to->cursor_reopen += from->cursor_reopen; + to->cursor_cache += from->cursor_cache; + to->cursor_create += from->cursor_create; to->cursor_next_hs_tombstone += from->cursor_next_hs_tombstone; to->cursor_next_skip_ge_100 += from->cursor_next_skip_ge_100; to->cursor_next_skip_lt_100 += from->cursor_next_skip_lt_100; to->cursor_prev_hs_tombstone += from->cursor_prev_hs_tombstone; to->cursor_prev_skip_ge_100 += from->cursor_prev_skip_ge_100; to->cursor_prev_skip_lt_100 += from->cursor_prev_skip_lt_100; + to->cursor_insert += from->cursor_insert; + to->cursor_insert_bytes += from->cursor_insert_bytes; + to->cursor_modify += from->cursor_modify; + to->cursor_modify_bytes += from->cursor_modify_bytes; + to->cursor_modify_bytes_touch += from->cursor_modify_bytes_touch; + to->cursor_next += from->cursor_next; to->cursor_open_count += from->cursor_open_count; + to->cursor_restart += from->cursor_restart; + to->cursor_prev += from->cursor_prev; + to->cursor_remove += from->cursor_remove; + to->cursor_remove_bytes += from->cursor_remove_bytes; + to->cursor_reserve += from->cursor_reserve; + to->cursor_reset += from->cursor_reset; + to->cursor_search += from->cursor_search; + to->cursor_search_hs += from->cursor_search_hs; + to->cursor_search_near += from->cursor_search_near; + to->cursor_truncate += from->cursor_truncate; + to->cursor_update += from->cursor_update; + to->cursor_update_bytes += from->cursor_update_bytes; + to->cursor_update_bytes_changed += from->cursor_update_bytes_changed; to->rec_time_window_bytes_ts += from->rec_time_window_bytes_ts; to->rec_time_window_bytes_txn += from->rec_time_window_bytes_txn; + to->rec_dictionary += from->rec_dictionary; to->rec_page_delete_fast += from->rec_page_delete_fast; + to->rec_suffix_compression += from->rec_suffix_compression; + to->rec_multiblock_internal += from->rec_multiblock_internal; + to->rec_overflow_key_internal += from->rec_overflow_key_internal; + to->rec_prefix_compression += from->rec_prefix_compression; + to->rec_multiblock_leaf += from->rec_multiblock_leaf; + to->rec_overflow_key_leaf += from->rec_overflow_key_leaf; + if (from->rec_multiblock_max > to->rec_multiblock_max) + to->rec_multiblock_max = from->rec_multiblock_max; + to->rec_overflow_value += from->rec_overflow_value; + to->rec_page_match += from->rec_page_match; to->rec_pages += from->rec_pages; to->rec_pages_eviction += from->rec_pages_eviction; to->rec_page_delete += from->rec_page_delete; @@ -692,17 +694,21 @@ __wt_stat_dsrc_aggregate_single(WT_DSRC_STATS *from, WT_DSRC_STATS *to) to->rec_time_aggr_newest_txn += from->rec_time_aggr_newest_txn; to->rec_time_aggr_oldest_start_ts += from->rec_time_aggr_oldest_start_ts; to->rec_time_aggr_prepared += from->rec_time_aggr_prepared; + to->rec_time_window_pages_prepared += from->rec_time_window_pages_prepared; to->rec_time_window_pages_durable_start_ts += from->rec_time_window_pages_durable_start_ts; + to->rec_time_window_pages_start_ts += from->rec_time_window_pages_start_ts; to->rec_time_window_pages_start_txn += from->rec_time_window_pages_start_txn; to->rec_time_window_pages_durable_stop_ts += from->rec_time_window_pages_durable_stop_ts; to->rec_time_window_pages_stop_ts += from->rec_time_window_pages_stop_ts; to->rec_time_window_pages_stop_txn += from->rec_time_window_pages_stop_txn; + to->rec_time_window_prepared += from->rec_time_window_prepared; to->rec_time_window_durable_start_ts += from->rec_time_window_durable_start_ts; to->rec_time_window_start_ts += from->rec_time_window_start_ts; to->rec_time_window_start_txn += from->rec_time_window_start_txn; to->rec_time_window_durable_stop_ts += from->rec_time_window_durable_stop_ts; to->rec_time_window_stop_ts += from->rec_time_window_stop_ts; to->rec_time_window_stop_txn += from->rec_time_window_stop_txn; + to->session_compact += from->session_compact; to->tiered_work_units_dequeued += from->tiered_work_units_dequeued; to->tiered_work_units_created += from->tiered_work_units_created; to->tiered_retention += from->tiered_retention; @@ -735,6 +741,8 @@ __wt_stat_dsrc_aggregate(WT_DSRC_STATS **from, WT_DSRC_STATS *to) if ((v = WT_STAT_READ(from, lsm_generation_max)) > to->lsm_generation_max) to->lsm_generation_max = v; to->lsm_lookup_no_bloom += WT_STAT_READ(from, lsm_lookup_no_bloom); + to->lsm_checkpoint_throttle += WT_STAT_READ(from, lsm_checkpoint_throttle); + to->lsm_merge_throttle += WT_STAT_READ(from, lsm_merge_throttle); to->bloom_size += WT_STAT_READ(from, bloom_size); to->block_extension += WT_STAT_READ(from, block_extension); to->block_alloc += WT_STAT_READ(from, block_alloc); @@ -777,77 +785,6 @@ __wt_stat_dsrc_aggregate(WT_DSRC_STATS **from, WT_DSRC_STATS *to) to->btree_row_empty_values += WT_STAT_READ(from, btree_row_empty_values); to->btree_row_internal += WT_STAT_READ(from, btree_row_internal); to->btree_row_leaf += WT_STAT_READ(from, btree_row_leaf); - to->cache_eviction_fail += WT_STAT_READ(from, cache_eviction_fail); - to->cache_eviction_walk_passes += WT_STAT_READ(from, cache_eviction_walk_passes); - to->cache_state_gen_avg_gap += WT_STAT_READ(from, cache_state_gen_avg_gap); - to->cache_state_avg_written_size += WT_STAT_READ(from, cache_state_avg_written_size); - to->cache_state_avg_visited_age += WT_STAT_READ(from, cache_state_avg_visited_age); - to->cache_state_avg_unvisited_age += WT_STAT_READ(from, cache_state_avg_unvisited_age); - to->cache_state_pages_clean += WT_STAT_READ(from, cache_state_pages_clean); - to->cache_state_gen_current += WT_STAT_READ(from, cache_state_gen_current); - to->cache_state_pages_dirty += WT_STAT_READ(from, cache_state_pages_dirty); - to->cache_state_root_entries += WT_STAT_READ(from, cache_state_root_entries); - to->cache_state_pages_internal += WT_STAT_READ(from, cache_state_pages_internal); - to->cache_state_pages_leaf += WT_STAT_READ(from, cache_state_pages_leaf); - to->cache_state_gen_max_gap += WT_STAT_READ(from, cache_state_gen_max_gap); - to->cache_state_max_pagesize += WT_STAT_READ(from, cache_state_max_pagesize); - to->cache_state_min_written_size += WT_STAT_READ(from, cache_state_min_written_size); - to->cache_state_unvisited_count += WT_STAT_READ(from, cache_state_unvisited_count); - to->cache_state_smaller_alloc_size += WT_STAT_READ(from, cache_state_smaller_alloc_size); - to->cache_state_memory += WT_STAT_READ(from, cache_state_memory); - to->cache_state_queued += WT_STAT_READ(from, cache_state_queued); - to->cache_state_not_queueable += WT_STAT_READ(from, cache_state_not_queueable); - to->cache_state_refs_skipped += WT_STAT_READ(from, cache_state_refs_skipped); - to->cache_state_root_size += WT_STAT_READ(from, cache_state_root_size); - to->cache_state_pages += WT_STAT_READ(from, cache_state_pages); - to->compress_precomp_intl_max_page_size += - WT_STAT_READ(from, compress_precomp_intl_max_page_size); - to->compress_precomp_leaf_max_page_size += - WT_STAT_READ(from, compress_precomp_leaf_max_page_size); - to->compress_read += WT_STAT_READ(from, compress_read); - to->compress_write += WT_STAT_READ(from, compress_write); - to->compress_write_fail += WT_STAT_READ(from, compress_write_fail); - to->compress_write_too_small += WT_STAT_READ(from, compress_write_too_small); - to->cursor_insert_bulk += WT_STAT_READ(from, cursor_insert_bulk); - to->cursor_reopen += WT_STAT_READ(from, cursor_reopen); - to->cursor_cache += WT_STAT_READ(from, cursor_cache); - to->cursor_create += WT_STAT_READ(from, cursor_create); - to->cursor_insert += WT_STAT_READ(from, cursor_insert); - to->cursor_insert_bytes += WT_STAT_READ(from, cursor_insert_bytes); - to->cursor_modify += WT_STAT_READ(from, cursor_modify); - to->cursor_modify_bytes += WT_STAT_READ(from, cursor_modify_bytes); - to->cursor_modify_bytes_touch += WT_STAT_READ(from, cursor_modify_bytes_touch); - to->cursor_next += WT_STAT_READ(from, cursor_next); - to->cursor_restart += WT_STAT_READ(from, cursor_restart); - to->cursor_prev += WT_STAT_READ(from, cursor_prev); - to->cursor_remove += WT_STAT_READ(from, cursor_remove); - to->cursor_remove_bytes += WT_STAT_READ(from, cursor_remove_bytes); - to->cursor_reserve += WT_STAT_READ(from, cursor_reserve); - to->cursor_reset += WT_STAT_READ(from, cursor_reset); - to->cursor_search += WT_STAT_READ(from, cursor_search); - to->cursor_search_hs += WT_STAT_READ(from, cursor_search_hs); - to->cursor_search_near += WT_STAT_READ(from, cursor_search_near); - to->cursor_truncate += WT_STAT_READ(from, cursor_truncate); - to->cursor_update += WT_STAT_READ(from, cursor_update); - to->cursor_update_bytes += WT_STAT_READ(from, cursor_update_bytes); - to->cursor_update_bytes_changed += WT_STAT_READ(from, cursor_update_bytes_changed); - to->rec_dictionary += WT_STAT_READ(from, rec_dictionary); - to->rec_suffix_compression += WT_STAT_READ(from, rec_suffix_compression); - to->rec_multiblock_internal += WT_STAT_READ(from, rec_multiblock_internal); - to->rec_overflow_key_internal += WT_STAT_READ(from, rec_overflow_key_internal); - to->rec_prefix_compression += WT_STAT_READ(from, rec_prefix_compression); - to->rec_multiblock_leaf += WT_STAT_READ(from, rec_multiblock_leaf); - to->rec_overflow_key_leaf += WT_STAT_READ(from, rec_overflow_key_leaf); - if ((v = WT_STAT_READ(from, rec_multiblock_max)) > to->rec_multiblock_max) - to->rec_multiblock_max = v; - to->rec_overflow_value += WT_STAT_READ(from, rec_overflow_value); - to->rec_page_match += WT_STAT_READ(from, rec_page_match); - to->rec_time_window_pages_prepared += WT_STAT_READ(from, rec_time_window_pages_prepared); - to->rec_time_window_pages_start_ts += WT_STAT_READ(from, rec_time_window_pages_start_ts); - to->rec_time_window_prepared += WT_STAT_READ(from, rec_time_window_prepared); - to->session_compact += WT_STAT_READ(from, session_compact); - to->lsm_checkpoint_throttle += WT_STAT_READ(from, lsm_checkpoint_throttle); - to->lsm_merge_throttle += WT_STAT_READ(from, lsm_merge_throttle); to->cache_bytes_inuse += WT_STAT_READ(from, cache_bytes_inuse); to->cache_bytes_dirty_total += WT_STAT_READ(from, cache_bytes_dirty_total); to->cache_bytes_read += WT_STAT_READ(from, cache_bytes_read); @@ -855,6 +792,8 @@ __wt_stat_dsrc_aggregate(WT_DSRC_STATS **from, WT_DSRC_STATS *to) to->cache_eviction_checkpoint += WT_STAT_READ(from, cache_eviction_checkpoint); to->cache_eviction_blocked_checkpoint_hs += WT_STAT_READ(from, cache_eviction_blocked_checkpoint_hs); + to->cache_eviction_fail += WT_STAT_READ(from, cache_eviction_fail); + to->cache_eviction_walk_passes += WT_STAT_READ(from, cache_eviction_walk_passes); to->cache_eviction_target_page_lt10 += WT_STAT_READ(from, cache_eviction_target_page_lt10); to->cache_eviction_target_page_lt32 += WT_STAT_READ(from, cache_eviction_target_page_lt32); to->cache_eviction_target_page_ge128 += WT_STAT_READ(from, cache_eviction_target_page_ge128); @@ -907,25 +846,90 @@ __wt_stat_dsrc_aggregate(WT_DSRC_STATS **from, WT_DSRC_STATS *to) to->cache_write_restore += WT_STAT_READ(from, cache_write_restore); to->cache_bytes_dirty += WT_STAT_READ(from, cache_bytes_dirty); to->cache_eviction_clean += WT_STAT_READ(from, cache_eviction_clean); + to->cache_state_gen_avg_gap += WT_STAT_READ(from, cache_state_gen_avg_gap); + to->cache_state_avg_written_size += WT_STAT_READ(from, cache_state_avg_written_size); + to->cache_state_avg_visited_age += WT_STAT_READ(from, cache_state_avg_visited_age); + to->cache_state_avg_unvisited_age += WT_STAT_READ(from, cache_state_avg_unvisited_age); + to->cache_state_pages_clean += WT_STAT_READ(from, cache_state_pages_clean); + to->cache_state_gen_current += WT_STAT_READ(from, cache_state_gen_current); + to->cache_state_pages_dirty += WT_STAT_READ(from, cache_state_pages_dirty); + to->cache_state_root_entries += WT_STAT_READ(from, cache_state_root_entries); + to->cache_state_pages_internal += WT_STAT_READ(from, cache_state_pages_internal); + to->cache_state_pages_leaf += WT_STAT_READ(from, cache_state_pages_leaf); + to->cache_state_gen_max_gap += WT_STAT_READ(from, cache_state_gen_max_gap); + to->cache_state_max_pagesize += WT_STAT_READ(from, cache_state_max_pagesize); + to->cache_state_min_written_size += WT_STAT_READ(from, cache_state_min_written_size); + to->cache_state_unvisited_count += WT_STAT_READ(from, cache_state_unvisited_count); + to->cache_state_smaller_alloc_size += WT_STAT_READ(from, cache_state_smaller_alloc_size); + to->cache_state_memory += WT_STAT_READ(from, cache_state_memory); + to->cache_state_queued += WT_STAT_READ(from, cache_state_queued); + to->cache_state_not_queueable += WT_STAT_READ(from, cache_state_not_queueable); + to->cache_state_refs_skipped += WT_STAT_READ(from, cache_state_refs_skipped); + to->cache_state_root_size += WT_STAT_READ(from, cache_state_root_size); + to->cache_state_pages += WT_STAT_READ(from, cache_state_pages); to->cc_pages_evict += WT_STAT_READ(from, cc_pages_evict); to->cc_pages_removed += WT_STAT_READ(from, cc_pages_removed); to->cc_pages_walk_skipped += WT_STAT_READ(from, cc_pages_walk_skipped); to->cc_pages_visited += WT_STAT_READ(from, cc_pages_visited); + to->compress_precomp_intl_max_page_size += + WT_STAT_READ(from, compress_precomp_intl_max_page_size); + to->compress_precomp_leaf_max_page_size += + WT_STAT_READ(from, compress_precomp_leaf_max_page_size); + to->compress_read += WT_STAT_READ(from, compress_read); + to->compress_write += WT_STAT_READ(from, compress_write); + to->compress_write_fail += WT_STAT_READ(from, compress_write_fail); + to->compress_write_too_small += WT_STAT_READ(from, compress_write_too_small); to->cursor_next_skip_total += WT_STAT_READ(from, cursor_next_skip_total); to->cursor_prev_skip_total += WT_STAT_READ(from, cursor_prev_skip_total); to->cursor_skip_hs_cur_position += WT_STAT_READ(from, cursor_skip_hs_cur_position); + to->cursor_next_skip_page_count += WT_STAT_READ(from, cursor_next_skip_page_count); + to->cursor_prev_skip_page_count += WT_STAT_READ(from, cursor_prev_skip_page_count); to->cursor_search_near_prefix_fast_paths += WT_STAT_READ(from, cursor_search_near_prefix_fast_paths); + to->cursor_insert_bulk += WT_STAT_READ(from, cursor_insert_bulk); + to->cursor_reopen += WT_STAT_READ(from, cursor_reopen); + to->cursor_cache += WT_STAT_READ(from, cursor_cache); + to->cursor_create += WT_STAT_READ(from, cursor_create); to->cursor_next_hs_tombstone += WT_STAT_READ(from, cursor_next_hs_tombstone); to->cursor_next_skip_ge_100 += WT_STAT_READ(from, cursor_next_skip_ge_100); to->cursor_next_skip_lt_100 += WT_STAT_READ(from, cursor_next_skip_lt_100); to->cursor_prev_hs_tombstone += WT_STAT_READ(from, cursor_prev_hs_tombstone); to->cursor_prev_skip_ge_100 += WT_STAT_READ(from, cursor_prev_skip_ge_100); to->cursor_prev_skip_lt_100 += WT_STAT_READ(from, cursor_prev_skip_lt_100); + to->cursor_insert += WT_STAT_READ(from, cursor_insert); + to->cursor_insert_bytes += WT_STAT_READ(from, cursor_insert_bytes); + to->cursor_modify += WT_STAT_READ(from, cursor_modify); + to->cursor_modify_bytes += WT_STAT_READ(from, cursor_modify_bytes); + to->cursor_modify_bytes_touch += WT_STAT_READ(from, cursor_modify_bytes_touch); + to->cursor_next += WT_STAT_READ(from, cursor_next); to->cursor_open_count += WT_STAT_READ(from, cursor_open_count); + to->cursor_restart += WT_STAT_READ(from, cursor_restart); + to->cursor_prev += WT_STAT_READ(from, cursor_prev); + to->cursor_remove += WT_STAT_READ(from, cursor_remove); + to->cursor_remove_bytes += WT_STAT_READ(from, cursor_remove_bytes); + to->cursor_reserve += WT_STAT_READ(from, cursor_reserve); + to->cursor_reset += WT_STAT_READ(from, cursor_reset); + to->cursor_search += WT_STAT_READ(from, cursor_search); + to->cursor_search_hs += WT_STAT_READ(from, cursor_search_hs); + to->cursor_search_near += WT_STAT_READ(from, cursor_search_near); + to->cursor_truncate += WT_STAT_READ(from, cursor_truncate); + to->cursor_update += WT_STAT_READ(from, cursor_update); + to->cursor_update_bytes += WT_STAT_READ(from, cursor_update_bytes); + to->cursor_update_bytes_changed += WT_STAT_READ(from, cursor_update_bytes_changed); to->rec_time_window_bytes_ts += WT_STAT_READ(from, rec_time_window_bytes_ts); to->rec_time_window_bytes_txn += WT_STAT_READ(from, rec_time_window_bytes_txn); + to->rec_dictionary += WT_STAT_READ(from, rec_dictionary); to->rec_page_delete_fast += WT_STAT_READ(from, rec_page_delete_fast); + to->rec_suffix_compression += WT_STAT_READ(from, rec_suffix_compression); + to->rec_multiblock_internal += WT_STAT_READ(from, rec_multiblock_internal); + to->rec_overflow_key_internal += WT_STAT_READ(from, rec_overflow_key_internal); + to->rec_prefix_compression += WT_STAT_READ(from, rec_prefix_compression); + to->rec_multiblock_leaf += WT_STAT_READ(from, rec_multiblock_leaf); + to->rec_overflow_key_leaf += WT_STAT_READ(from, rec_overflow_key_leaf); + if ((v = WT_STAT_READ(from, rec_multiblock_max)) > to->rec_multiblock_max) + to->rec_multiblock_max = v; + to->rec_overflow_value += WT_STAT_READ(from, rec_overflow_value); + to->rec_page_match += WT_STAT_READ(from, rec_page_match); to->rec_pages += WT_STAT_READ(from, rec_pages); to->rec_pages_eviction += WT_STAT_READ(from, rec_pages_eviction); to->rec_page_delete += WT_STAT_READ(from, rec_page_delete); @@ -938,19 +942,23 @@ __wt_stat_dsrc_aggregate(WT_DSRC_STATS **from, WT_DSRC_STATS *to) to->rec_time_aggr_newest_txn += WT_STAT_READ(from, rec_time_aggr_newest_txn); to->rec_time_aggr_oldest_start_ts += WT_STAT_READ(from, rec_time_aggr_oldest_start_ts); to->rec_time_aggr_prepared += WT_STAT_READ(from, rec_time_aggr_prepared); + to->rec_time_window_pages_prepared += WT_STAT_READ(from, rec_time_window_pages_prepared); to->rec_time_window_pages_durable_start_ts += WT_STAT_READ(from, rec_time_window_pages_durable_start_ts); + to->rec_time_window_pages_start_ts += WT_STAT_READ(from, rec_time_window_pages_start_ts); to->rec_time_window_pages_start_txn += WT_STAT_READ(from, rec_time_window_pages_start_txn); to->rec_time_window_pages_durable_stop_ts += WT_STAT_READ(from, rec_time_window_pages_durable_stop_ts); to->rec_time_window_pages_stop_ts += WT_STAT_READ(from, rec_time_window_pages_stop_ts); to->rec_time_window_pages_stop_txn += WT_STAT_READ(from, rec_time_window_pages_stop_txn); + to->rec_time_window_prepared += WT_STAT_READ(from, rec_time_window_prepared); to->rec_time_window_durable_start_ts += WT_STAT_READ(from, rec_time_window_durable_start_ts); to->rec_time_window_start_ts += WT_STAT_READ(from, rec_time_window_start_ts); to->rec_time_window_start_txn += WT_STAT_READ(from, rec_time_window_start_txn); to->rec_time_window_durable_stop_ts += WT_STAT_READ(from, rec_time_window_durable_stop_ts); to->rec_time_window_stop_ts += WT_STAT_READ(from, rec_time_window_stop_ts); to->rec_time_window_stop_txn += WT_STAT_READ(from, rec_time_window_stop_txn); + to->session_compact += WT_STAT_READ(from, session_compact); to->tiered_work_units_dequeued += WT_STAT_READ(from, tiered_work_units_dequeued); to->tiered_work_units_created += WT_STAT_READ(from, tiered_work_units_created); to->tiered_retention += WT_STAT_READ(from, tiered_retention); @@ -973,6 +981,8 @@ static const char *const __stats_connection_desc[] = { "LSM: application work units currently queued", "LSM: merge work units currently queued", "LSM: rows merged in an LSM tree", + "LSM: sleep for LSM checkpoint throttle", + "LSM: sleep for LSM merge throttle", "LSM: switch work units currently queued", "LSM: tree maintenance operations discarded", "LSM: tree maintenance operations executed", @@ -1000,8 +1010,14 @@ static const char *const __stats_connection_desc[] = { "cache: bytes allocated for updates", "cache: bytes belonging to page images in the cache", "cache: bytes belonging to the history store table in the cache", + "cache: bytes currently in the cache", + "cache: bytes dirty in the cache cumulative", "cache: bytes not belonging to page images in the cache", + "cache: bytes read into cache", + "cache: bytes written from cache", "cache: cache overflow score", + "cache: checkpoint blocked page eviction", + "cache: checkpoint of history store file blocked non-history store page eviction", "cache: eviction calls to get a page", "cache: eviction calls to get a page found queue empty", "cache: eviction calls to get a page found queue empty after locking", @@ -1015,9 +1031,23 @@ static const char *const __stats_connection_desc[] = { "cache: eviction server unable to reach eviction goal", "cache: eviction server waiting for a leaf page", "cache: eviction state", + "cache: eviction walk target pages histogram - 0-9", + "cache: eviction walk target pages histogram - 10-31", + "cache: eviction walk target pages histogram - 128 and higher", + "cache: eviction walk target pages histogram - 32-63", + "cache: eviction walk target pages histogram - 64-128", + "cache: eviction walk target pages reduced due to history store cache pressure", "cache: eviction walk target strategy both clean and dirty pages", "cache: eviction walk target strategy only clean pages", "cache: eviction walk target strategy only dirty pages", + "cache: eviction walks abandoned", + "cache: eviction walks gave up because they restarted their walk twice", + "cache: eviction walks gave up because they saw too many pages and found no candidates", + "cache: eviction walks gave up because they saw too many pages and found too few candidates", + "cache: eviction walks reached end of tree", + "cache: eviction walks restarted", + "cache: eviction walks started from root of tree", + "cache: eviction walks started from saved location in tree", "cache: eviction worker thread active", "cache: eviction worker thread created", "cache: eviction worker thread evicting pages", @@ -1040,19 +1070,45 @@ static const char *const __stats_connection_desc[] = { "cache: forced eviction - pages selected count", "cache: forced eviction - pages selected unable to be evicted count", "cache: forced eviction - pages selected unable to be evicted time", + "cache: hazard pointer blocked page eviction", "cache: hazard pointer check calls", "cache: hazard pointer check entries walked", "cache: hazard pointer maximum array length", "cache: history store score", + "cache: history store table insert calls", + "cache: history store table insert calls that returned restart", "cache: history store table max on-disk size", "cache: history store table on-disk size", + "cache: history store table out-of-order resolved updates that lose their durable timestamp", + "cache: history store table out-of-order updates that were fixed up by reinserting with the " + "fixed timestamp", + "cache: history store table reads", + "cache: history store table reads missed", + "cache: history store table reads requiring squashed modifies", + "cache: history store table truncation by rollback to stable to remove an unstable update", + "cache: history store table truncation by rollback to stable to remove an update", + "cache: history store table truncation to remove an update", + "cache: history store table truncation to remove range of updates due to key being removed from " + "the data page during reconciliation", + "cache: history store table truncation to remove range of updates due to out-of-order timestamp " + "update on data page", + "cache: history store table writes requiring squashed modifies", + "cache: in-memory page passed criteria to be split", + "cache: in-memory page splits", + "cache: internal pages evicted", "cache: internal pages queued for eviction", "cache: internal pages seen by eviction walk", "cache: internal pages seen by eviction walk that are already queued", + "cache: internal pages split during eviction", + "cache: leaf pages split during eviction", "cache: maximum bytes configured", "cache: maximum page size at eviction", + "cache: modified pages evicted", "cache: modified pages evicted by application threads", "cache: operations timed out waiting for space in cache", + "cache: overflow pages read into cache", + "cache: page split during eviction deepened the tree", + "cache: page written requiring history store records", "cache: pages currently held in the cache", "cache: pages evicted by application threads", "cache: pages evicted in parallel with checkpoint", @@ -1061,17 +1117,28 @@ static const char *const __stats_connection_desc[] = { "cache: pages queued for urgent eviction", "cache: pages queued for urgent eviction during walk", "cache: pages queued for urgent eviction from history store due to high dirty content", + "cache: pages read into cache", + "cache: pages read into cache after truncate", + "cache: pages read into cache after truncate in prepare state", + "cache: pages requested from the cache", + "cache: pages seen by eviction walk", "cache: pages seen by eviction walk that are already queued", "cache: pages selected for eviction unable to be evicted", "cache: pages selected for eviction unable to be evicted as the parent page has overflow items", "cache: pages selected for eviction unable to be evicted because of active children on an " "internal page", "cache: pages selected for eviction unable to be evicted because of failure in reconciliation", + "cache: pages selected for eviction unable to be evicted because of race between checkpoint and " + "out of order timestamps handling", "cache: pages walked for eviction", + "cache: pages written from cache", + "cache: pages written requiring in-memory restoration", "cache: percentage overhead", "cache: tracked bytes belonging to internal pages in the cache", "cache: tracked bytes belonging to leaf pages in the cache", + "cache: tracked dirty bytes in the cache", "cache: tracked dirty pages in the cache", + "cache: unmodified pages evicted", "capacity: background fsync file handles considered", "capacity: background fsync file handles synced", "capacity: background fsync time (msecs)", @@ -1086,6 +1153,10 @@ static const char *const __stats_connection_desc[] = { "capacity: time waiting during eviction (usecs)", "capacity: time waiting during logging (usecs)", "capacity: time waiting during read (usecs)", + "checkpoint-cleanup: pages added for eviction", + "checkpoint-cleanup: pages removed", + "checkpoint-cleanup: pages skipped during tree walk", + "checkpoint-cleanup: pages visited", "connection: auto adjusting condition resets", "connection: auto adjusting condition wait calls", "connection: auto adjusting condition wait raced to update timeout and skipped updating", @@ -1102,6 +1173,12 @@ static const char *const __stats_connection_desc[] = { "connection: total fsync I/Os", "connection: total read I/Os", "connection: total write I/Os", + "cursor: Total number of entries skipped by cursor next calls", + "cursor: Total number of entries skipped by cursor prev calls", + "cursor: Total number of entries skipped to position the history store cursor", + "cursor: Total number of pages skipped without reading by cursor next calls", + "cursor: Total number of pages skipped without reading by cursor prev calls", + "cursor: Total number of times a search near has exited due to prefix config", "cursor: cached cursor count", "cursor: cursor bulk loaded cursor insert calls", "cursor: cursor close calls that result in cache", @@ -1112,8 +1189,14 @@ static const char *const __stats_connection_desc[] = { "cursor: cursor modify key and value bytes affected", "cursor: cursor modify value bytes modified", "cursor: cursor next calls", + "cursor: cursor next calls that skip due to a globally visible history store tombstone", + "cursor: cursor next calls that skip greater than or equal to 100 entries", + "cursor: cursor next calls that skip less than 100 entries", "cursor: cursor operation restarted", "cursor: cursor prev calls", + "cursor: cursor prev calls that skip due to a globally visible history store tombstone", + "cursor: cursor prev calls that skip greater than or equal to 100 entries", + "cursor: cursor prev calls that skip less than 100 entries", "cursor: cursor remove calls", "cursor: cursor remove key bytes removed", "cursor: cursor reserve calls", @@ -1130,6 +1213,7 @@ static const char *const __stats_connection_desc[] = { "cursor: cursor update key and value bytes", "cursor: cursor update value size change", "cursor: cursors reused from cache", + "cursor: open cursor count", "data-handle: connection data handle size", "data-handle: connection data handles currently active", "data-handle: connection sweep candidate became referenced", @@ -1237,20 +1321,43 @@ static const char *const __stats_connection_desc[] = { "perf: operation write latency histogram (bucket 3) - 500-999us", "perf: operation write latency histogram (bucket 4) - 1000-9999us", "perf: operation write latency histogram (bucket 5) - 10000us+", + "reconciliation: approximate byte size of timestamps in pages written", + "reconciliation: approximate byte size of transaction IDs in pages written", + "reconciliation: fast-path pages deleted", "reconciliation: internal-page overflow keys", "reconciliation: leaf-page overflow keys", "reconciliation: maximum seconds spent in a reconciliation call", + "reconciliation: page reconciliation calls", + "reconciliation: page reconciliation calls for eviction", "reconciliation: page reconciliation calls that resulted in values with prepared transaction " "metadata", "reconciliation: page reconciliation calls that resulted in values with timestamps", "reconciliation: page reconciliation calls that resulted in values with transaction ids", + "reconciliation: pages deleted", + "reconciliation: pages written including an aggregated newest start durable timestamp ", + "reconciliation: pages written including an aggregated newest stop durable timestamp ", + "reconciliation: pages written including an aggregated newest stop timestamp ", + "reconciliation: pages written including an aggregated newest stop transaction ID", + "reconciliation: pages written including an aggregated newest transaction ID ", + "reconciliation: pages written including an aggregated oldest start timestamp ", + "reconciliation: pages written including an aggregated prepare", "reconciliation: pages written including at least one prepare state", + "reconciliation: pages written including at least one start durable timestamp", "reconciliation: pages written including at least one start timestamp", + "reconciliation: pages written including at least one start transaction ID", + "reconciliation: pages written including at least one stop durable timestamp", + "reconciliation: pages written including at least one stop timestamp", + "reconciliation: pages written including at least one stop transaction ID", "reconciliation: records written including a prepare state", + "reconciliation: records written including a start durable timestamp", + "reconciliation: records written including a start timestamp", + "reconciliation: records written including a start transaction ID", + "reconciliation: records written including a stop durable timestamp", + "reconciliation: records written including a stop timestamp", + "reconciliation: records written including a stop transaction ID", "reconciliation: split bytes currently awaiting free", "reconciliation: split objects currently awaiting free", "session: flush state races", - "session: flush_tier busy retries", "session: flush_tier operation calls", "session: open session count", "session: session query timestamp calls", @@ -1271,6 +1378,10 @@ static const char *const __stats_connection_desc[] = { "session: table truncate successful calls", "session: table verify failed calls", "session: table verify successful calls", + "session: tiered operations dequeued and processed", + "session: tiered operations scheduled", + "session: tiered storage local retention time (secs)", + "session: tiered storage object size", "thread-state: active filesystem fsync calls", "thread-state: active filesystem read calls", "thread-state: active filesystem write calls", @@ -1298,10 +1409,20 @@ static const char *const __stats_connection_desc[] = { "transaction: prepared transactions currently active", "transaction: prepared transactions rolled back", "transaction: query timestamp calls", + "transaction: race to read prepared update retry", "transaction: rollback to stable calls", + "transaction: rollback to stable history store records with stop timestamps older than newer " + "records", + "transaction: rollback to stable inconsistent checkpoint", + "transaction: rollback to stable keys removed", + "transaction: rollback to stable keys restored", "transaction: rollback to stable pages visited", + "transaction: rollback to stable restored tombstones from history store", + "transaction: rollback to stable restored updates from history store", + "transaction: rollback to stable sweeping history store keys", "transaction: rollback to stable tree walk skipping pages", "transaction: rollback to stable updates aborted", + "transaction: rollback to stable updates removed from history store", "transaction: sessions scanned in each walk of concurrent sessions", "transaction: set timestamp calls", "transaction: set timestamp durable calls", @@ -1333,6 +1454,7 @@ static const char *const __stats_connection_desc[] = { "transaction: transaction checkpoint scrub time (msecs)", "transaction: transaction checkpoint total time (msecs)", "transaction: transaction checkpoints", + "transaction: transaction checkpoints due to obsolete pages", "transaction: transaction checkpoints skipped because database was clean", "transaction: transaction failures due to history store", "transaction: transaction fsync calls for checkpoint after allocating the transaction ID", @@ -1350,117 +1472,6 @@ static const char *const __stats_connection_desc[] = { "transaction: transaction walk of concurrent sessions", "transaction: transactions committed", "transaction: transactions rolled back", - "LSM: sleep for LSM checkpoint throttle", - "LSM: sleep for LSM merge throttle", - "cache: bytes currently in the cache", - "cache: bytes dirty in the cache cumulative", - "cache: bytes read into cache", - "cache: bytes written from cache", - "cache: checkpoint blocked page eviction", - "cache: checkpoint of history store file blocked non-history store page eviction", - "cache: eviction walk target pages histogram - 0-9", - "cache: eviction walk target pages histogram - 10-31", - "cache: eviction walk target pages histogram - 128 and higher", - "cache: eviction walk target pages histogram - 32-63", - "cache: eviction walk target pages histogram - 64-128", - "cache: eviction walk target pages reduced due to history store cache pressure", - "cache: eviction walks abandoned", - "cache: eviction walks gave up because they restarted their walk twice", - "cache: eviction walks gave up because they saw too many pages and found no candidates", - "cache: eviction walks gave up because they saw too many pages and found too few candidates", - "cache: eviction walks reached end of tree", - "cache: eviction walks restarted", - "cache: eviction walks started from root of tree", - "cache: eviction walks started from saved location in tree", - "cache: hazard pointer blocked page eviction", - "cache: history store table insert calls", - "cache: history store table insert calls that returned restart", - "cache: history store table out-of-order resolved updates that lose their durable timestamp", - "cache: history store table out-of-order updates that were fixed up by reinserting with the " - "fixed timestamp", - "cache: history store table reads", - "cache: history store table reads missed", - "cache: history store table reads requiring squashed modifies", - "cache: history store table truncation by rollback to stable to remove an unstable update", - "cache: history store table truncation by rollback to stable to remove an update", - "cache: history store table truncation to remove an update", - "cache: history store table truncation to remove range of updates due to key being removed from " - "the data page during reconciliation", - "cache: history store table truncation to remove range of updates due to out-of-order timestamp " - "update on data page", - "cache: history store table writes requiring squashed modifies", - "cache: in-memory page passed criteria to be split", - "cache: in-memory page splits", - "cache: internal pages evicted", - "cache: internal pages split during eviction", - "cache: leaf pages split during eviction", - "cache: modified pages evicted", - "cache: overflow pages read into cache", - "cache: page split during eviction deepened the tree", - "cache: page written requiring history store records", - "cache: pages read into cache", - "cache: pages read into cache after truncate", - "cache: pages read into cache after truncate in prepare state", - "cache: pages requested from the cache", - "cache: pages seen by eviction walk", - "cache: pages written from cache", - "cache: pages written requiring in-memory restoration", - "cache: tracked dirty bytes in the cache", - "cache: unmodified pages evicted", - "checkpoint-cleanup: pages added for eviction", - "checkpoint-cleanup: pages removed", - "checkpoint-cleanup: pages skipped during tree walk", - "checkpoint-cleanup: pages visited", - "cursor: Total number of entries skipped by cursor next calls", - "cursor: Total number of entries skipped by cursor prev calls", - "cursor: Total number of entries skipped to position the history store cursor", - "cursor: Total number of times a search near has exited due to prefix config", - "cursor: cursor next calls that skip due to a globally visible history store tombstone", - "cursor: cursor next calls that skip greater than or equal to 100 entries", - "cursor: cursor next calls that skip less than 100 entries", - "cursor: cursor prev calls that skip due to a globally visible history store tombstone", - "cursor: cursor prev calls that skip greater than or equal to 100 entries", - "cursor: cursor prev calls that skip less than 100 entries", - "cursor: open cursor count", - "reconciliation: approximate byte size of timestamps in pages written", - "reconciliation: approximate byte size of transaction IDs in pages written", - "reconciliation: fast-path pages deleted", - "reconciliation: page reconciliation calls", - "reconciliation: page reconciliation calls for eviction", - "reconciliation: pages deleted", - "reconciliation: pages written including an aggregated newest start durable timestamp ", - "reconciliation: pages written including an aggregated newest stop durable timestamp ", - "reconciliation: pages written including an aggregated newest stop timestamp ", - "reconciliation: pages written including an aggregated newest stop transaction ID", - "reconciliation: pages written including an aggregated newest transaction ID ", - "reconciliation: pages written including an aggregated oldest start timestamp ", - "reconciliation: pages written including an aggregated prepare", - "reconciliation: pages written including at least one start durable timestamp", - "reconciliation: pages written including at least one start transaction ID", - "reconciliation: pages written including at least one stop durable timestamp", - "reconciliation: pages written including at least one stop timestamp", - "reconciliation: pages written including at least one stop transaction ID", - "reconciliation: records written including a start durable timestamp", - "reconciliation: records written including a start timestamp", - "reconciliation: records written including a start transaction ID", - "reconciliation: records written including a stop durable timestamp", - "reconciliation: records written including a stop timestamp", - "reconciliation: records written including a stop transaction ID", - "session: tiered operations dequeued and processed", - "session: tiered operations scheduled", - "session: tiered storage local retention time (secs)", - "session: tiered storage object size", - "transaction: race to read prepared update retry", - "transaction: rollback to stable history store records with stop timestamps older than newer " - "records", - "transaction: rollback to stable inconsistent checkpoint", - "transaction: rollback to stable keys removed", - "transaction: rollback to stable keys restored", - "transaction: rollback to stable restored tombstones from history store", - "transaction: rollback to stable restored updates from history store", - "transaction: rollback to stable sweeping history store keys", - "transaction: rollback to stable updates removed from history store", - "transaction: transaction checkpoints due to obsolete pages", "transaction: update conflicts", }; @@ -1505,6 +1516,8 @@ __wt_stat_connection_clear_single(WT_CONNECTION_STATS *stats) /* not clearing lsm_work_queue_app */ /* not clearing lsm_work_queue_manager */ stats->lsm_rows_merged = 0; + stats->lsm_checkpoint_throttle = 0; + stats->lsm_merge_throttle = 0; /* not clearing lsm_work_queue_switch */ stats->lsm_work_units_discarded = 0; stats->lsm_work_units_done = 0; @@ -1531,8 +1544,14 @@ __wt_stat_connection_clear_single(WT_CONNECTION_STATS *stats) /* not clearing cache_bytes_updates */ /* not clearing cache_bytes_image */ /* not clearing cache_bytes_hs */ + /* not clearing cache_bytes_inuse */ + /* not clearing cache_bytes_dirty_total */ /* not clearing cache_bytes_other */ + stats->cache_bytes_read = 0; + stats->cache_bytes_write = 0; /* not clearing cache_lookaside_score */ + stats->cache_eviction_checkpoint = 0; + stats->cache_eviction_blocked_checkpoint_hs = 0; stats->cache_eviction_get_ref = 0; stats->cache_eviction_get_ref_empty = 0; stats->cache_eviction_get_ref_empty2 = 0; @@ -1546,9 +1565,23 @@ __wt_stat_connection_clear_single(WT_CONNECTION_STATS *stats) stats->cache_eviction_slow = 0; stats->cache_eviction_walk_leaf_notfound = 0; /* not clearing cache_eviction_state */ + stats->cache_eviction_target_page_lt10 = 0; + stats->cache_eviction_target_page_lt32 = 0; + stats->cache_eviction_target_page_ge128 = 0; + stats->cache_eviction_target_page_lt64 = 0; + stats->cache_eviction_target_page_lt128 = 0; + stats->cache_eviction_target_page_reduced = 0; stats->cache_eviction_target_strategy_both_clean_and_dirty = 0; stats->cache_eviction_target_strategy_clean = 0; stats->cache_eviction_target_strategy_dirty = 0; + stats->cache_eviction_walks_abandoned = 0; + stats->cache_eviction_walks_stopped = 0; + stats->cache_eviction_walks_gave_up_no_targets = 0; + stats->cache_eviction_walks_gave_up_ratio = 0; + stats->cache_eviction_walks_ended = 0; + stats->cache_eviction_walk_restart = 0; + stats->cache_eviction_walk_from_root = 0; + stats->cache_eviction_walk_saved_pos = 0; /* not clearing cache_eviction_active_workers */ stats->cache_eviction_worker_created = 0; stats->cache_eviction_worker_evicting = 0; @@ -1568,19 +1601,42 @@ __wt_stat_connection_clear_single(WT_CONNECTION_STATS *stats) stats->cache_eviction_force = 0; stats->cache_eviction_force_fail = 0; stats->cache_eviction_force_fail_time = 0; + stats->cache_eviction_hazard = 0; stats->cache_hazard_checks = 0; stats->cache_hazard_walks = 0; stats->cache_hazard_max = 0; /* not clearing cache_hs_score */ + stats->cache_hs_insert = 0; + stats->cache_hs_insert_restart = 0; /* not clearing cache_hs_ondisk_max */ /* not clearing cache_hs_ondisk */ + stats->cache_hs_order_lose_durable_timestamp = 0; + stats->cache_hs_order_reinsert = 0; + stats->cache_hs_read = 0; + stats->cache_hs_read_miss = 0; + stats->cache_hs_read_squash = 0; + stats->cache_hs_key_truncate_rts_unstable = 0; + stats->cache_hs_key_truncate_rts = 0; + stats->cache_hs_key_truncate = 0; + stats->cache_hs_key_truncate_onpage_removal = 0; + stats->cache_hs_order_remove = 0; + stats->cache_hs_write_squash = 0; + stats->cache_inmem_splittable = 0; + stats->cache_inmem_split = 0; + stats->cache_eviction_internal = 0; stats->cache_eviction_internal_pages_queued = 0; stats->cache_eviction_internal_pages_seen = 0; stats->cache_eviction_internal_pages_already_queued = 0; + stats->cache_eviction_split_internal = 0; + stats->cache_eviction_split_leaf = 0; /* not clearing cache_bytes_max */ /* not clearing cache_eviction_maximum_page_size */ + stats->cache_eviction_dirty = 0; stats->cache_eviction_app_dirty = 0; stats->cache_timed_out_ops = 0; + stats->cache_read_overflow = 0; + stats->cache_eviction_deepen = 0; + stats->cache_write_hs = 0; /* not clearing cache_pages_inuse */ stats->cache_eviction_app = 0; stats->cache_eviction_pages_in_parallel_with_checkpoint = 0; @@ -1589,16 +1645,26 @@ __wt_stat_connection_clear_single(WT_CONNECTION_STATS *stats) stats->cache_eviction_pages_queued_urgent = 0; stats->cache_eviction_pages_queued_oldest = 0; stats->cache_eviction_pages_queued_urgent_hs_dirty = 0; + stats->cache_read = 0; + stats->cache_read_deleted = 0; + stats->cache_read_deleted_prepared = 0; + stats->cache_pages_requested = 0; + stats->cache_eviction_pages_seen = 0; stats->cache_eviction_pages_already_queued = 0; stats->cache_eviction_fail = 0; stats->cache_eviction_fail_parent_has_overflow_items = 0; stats->cache_eviction_fail_active_children_on_an_internal_page = 0; stats->cache_eviction_fail_in_reconciliation = 0; + stats->cache_eviction_fail_checkpoint_out_of_order_ts = 0; stats->cache_eviction_walk = 0; + stats->cache_write = 0; + stats->cache_write_restore = 0; /* not clearing cache_overhead */ /* not clearing cache_bytes_internal */ /* not clearing cache_bytes_leaf */ + /* not clearing cache_bytes_dirty */ /* not clearing cache_pages_dirty */ + stats->cache_eviction_clean = 0; stats->fsync_all_fh_total = 0; stats->fsync_all_fh = 0; /* not clearing fsync_all_time */ @@ -1613,6 +1679,10 @@ __wt_stat_connection_clear_single(WT_CONNECTION_STATS *stats) stats->capacity_time_evict = 0; stats->capacity_time_log = 0; stats->capacity_time_read = 0; + stats->cc_pages_evict = 0; + stats->cc_pages_removed = 0; + stats->cc_pages_walk_skipped = 0; + stats->cc_pages_visited = 0; stats->cond_auto_wait_reset = 0; stats->cond_auto_wait = 0; stats->cond_auto_wait_skipped = 0; @@ -1629,6 +1699,12 @@ __wt_stat_connection_clear_single(WT_CONNECTION_STATS *stats) stats->fsync_io = 0; stats->read_io = 0; stats->write_io = 0; + stats->cursor_next_skip_total = 0; + stats->cursor_prev_skip_total = 0; + stats->cursor_skip_hs_cur_position = 0; + stats->cursor_next_skip_page_count = 0; + stats->cursor_prev_skip_page_count = 0; + stats->cursor_search_near_prefix_fast_paths = 0; /* not clearing cursor_cached_count */ stats->cursor_insert_bulk = 0; stats->cursor_cache = 0; @@ -1639,8 +1715,14 @@ __wt_stat_connection_clear_single(WT_CONNECTION_STATS *stats) stats->cursor_modify_bytes = 0; stats->cursor_modify_bytes_touch = 0; stats->cursor_next = 0; + stats->cursor_next_hs_tombstone = 0; + stats->cursor_next_skip_ge_100 = 0; + stats->cursor_next_skip_lt_100 = 0; stats->cursor_restart = 0; stats->cursor_prev = 0; + stats->cursor_prev_hs_tombstone = 0; + stats->cursor_prev_skip_ge_100 = 0; + stats->cursor_prev_skip_lt_100 = 0; stats->cursor_remove = 0; stats->cursor_remove_bytes = 0; stats->cursor_reserve = 0; @@ -1657,6 +1739,7 @@ __wt_stat_connection_clear_single(WT_CONNECTION_STATS *stats) stats->cursor_update_bytes = 0; stats->cursor_update_bytes_changed = 0; stats->cursor_reopen = 0; + /* not clearing cursor_open_count */ /* not clearing dh_conn_handle_size */ /* not clearing dh_conn_handle_count */ stats->dh_sweep_ref = 0; @@ -1764,19 +1847,42 @@ __wt_stat_connection_clear_single(WT_CONNECTION_STATS *stats) stats->perf_hist_opwrite_latency_lt1000 = 0; stats->perf_hist_opwrite_latency_lt10000 = 0; stats->perf_hist_opwrite_latency_gt10000 = 0; + stats->rec_time_window_bytes_ts = 0; + stats->rec_time_window_bytes_txn = 0; + stats->rec_page_delete_fast = 0; stats->rec_overflow_key_internal = 0; stats->rec_overflow_key_leaf = 0; /* not clearing rec_maximum_seconds */ + stats->rec_pages = 0; + stats->rec_pages_eviction = 0; stats->rec_pages_with_prepare = 0; stats->rec_pages_with_ts = 0; stats->rec_pages_with_txn = 0; + stats->rec_page_delete = 0; + stats->rec_time_aggr_newest_start_durable_ts = 0; + stats->rec_time_aggr_newest_stop_durable_ts = 0; + stats->rec_time_aggr_newest_stop_ts = 0; + stats->rec_time_aggr_newest_stop_txn = 0; + stats->rec_time_aggr_newest_txn = 0; + stats->rec_time_aggr_oldest_start_ts = 0; + stats->rec_time_aggr_prepared = 0; stats->rec_time_window_pages_prepared = 0; + stats->rec_time_window_pages_durable_start_ts = 0; stats->rec_time_window_pages_start_ts = 0; + stats->rec_time_window_pages_start_txn = 0; + stats->rec_time_window_pages_durable_stop_ts = 0; + stats->rec_time_window_pages_stop_ts = 0; + stats->rec_time_window_pages_stop_txn = 0; stats->rec_time_window_prepared = 0; + stats->rec_time_window_durable_start_ts = 0; + stats->rec_time_window_start_ts = 0; + stats->rec_time_window_start_txn = 0; + stats->rec_time_window_durable_stop_ts = 0; + stats->rec_time_window_stop_ts = 0; + stats->rec_time_window_stop_txn = 0; /* not clearing rec_split_stashed_bytes */ /* not clearing rec_split_stashed_objects */ stats->flush_state_races = 0; - stats->flush_tier_busy = 0; stats->flush_tier = 0; /* not clearing session_open */ stats->session_query_ts = 0; @@ -1797,6 +1903,10 @@ __wt_stat_connection_clear_single(WT_CONNECTION_STATS *stats) /* not clearing session_table_truncate_success */ /* not clearing session_table_verify_fail */ /* not clearing session_table_verify_success */ + stats->tiered_work_units_dequeued = 0; + stats->tiered_work_units_created = 0; + /* not clearing tiered_retention */ + /* not clearing tiered_object_size */ /* not clearing thread_fsync_active */ /* not clearing thread_read_active */ /* not clearing thread_write_active */ @@ -1824,10 +1934,19 @@ __wt_stat_connection_clear_single(WT_CONNECTION_STATS *stats) stats->txn_prepare_active = 0; stats->txn_prepare_rollback = 0; stats->txn_query_ts = 0; + stats->txn_read_race_prepare_update = 0; stats->txn_rts = 0; + stats->txn_rts_hs_stop_older_than_newer_start = 0; + stats->txn_rts_inconsistent_ckpt = 0; + stats->txn_rts_keys_removed = 0; + stats->txn_rts_keys_restored = 0; stats->txn_rts_pages_visited = 0; + stats->txn_rts_hs_restore_tombstones = 0; + stats->txn_rts_hs_restore_updates = 0; + stats->txn_rts_sweep_hs_keys = 0; stats->txn_rts_tree_walk_skip_pages = 0; stats->txn_rts_upd_aborted = 0; + stats->txn_rts_hs_removed = 0; stats->txn_sessions_walked = 0; stats->txn_set_ts = 0; stats->txn_set_ts_durable = 0; @@ -1859,6 +1978,7 @@ __wt_stat_connection_clear_single(WT_CONNECTION_STATS *stats) /* not clearing txn_checkpoint_scrub_time */ /* not clearing txn_checkpoint_time_total */ stats->txn_checkpoint = 0; + stats->txn_checkpoint_obsolete_applied = 0; stats->txn_checkpoint_skipped = 0; stats->txn_fail_cache = 0; stats->txn_checkpoint_fsync_post = 0; @@ -1875,113 +1995,6 @@ __wt_stat_connection_clear_single(WT_CONNECTION_STATS *stats) stats->txn_walk_sessions = 0; stats->txn_commit = 0; stats->txn_rollback = 0; - stats->lsm_checkpoint_throttle = 0; - stats->lsm_merge_throttle = 0; - /* not clearing cache_bytes_inuse */ - /* not clearing cache_bytes_dirty_total */ - stats->cache_bytes_read = 0; - stats->cache_bytes_write = 0; - stats->cache_eviction_checkpoint = 0; - stats->cache_eviction_blocked_checkpoint_hs = 0; - stats->cache_eviction_target_page_lt10 = 0; - stats->cache_eviction_target_page_lt32 = 0; - stats->cache_eviction_target_page_ge128 = 0; - stats->cache_eviction_target_page_lt64 = 0; - stats->cache_eviction_target_page_lt128 = 0; - stats->cache_eviction_target_page_reduced = 0; - stats->cache_eviction_walks_abandoned = 0; - stats->cache_eviction_walks_stopped = 0; - stats->cache_eviction_walks_gave_up_no_targets = 0; - stats->cache_eviction_walks_gave_up_ratio = 0; - stats->cache_eviction_walks_ended = 0; - stats->cache_eviction_walk_restart = 0; - stats->cache_eviction_walk_from_root = 0; - stats->cache_eviction_walk_saved_pos = 0; - stats->cache_eviction_hazard = 0; - stats->cache_hs_insert = 0; - stats->cache_hs_insert_restart = 0; - stats->cache_hs_order_lose_durable_timestamp = 0; - stats->cache_hs_order_reinsert = 0; - stats->cache_hs_read = 0; - stats->cache_hs_read_miss = 0; - stats->cache_hs_read_squash = 0; - stats->cache_hs_key_truncate_rts_unstable = 0; - stats->cache_hs_key_truncate_rts = 0; - stats->cache_hs_key_truncate = 0; - stats->cache_hs_key_truncate_onpage_removal = 0; - stats->cache_hs_order_remove = 0; - stats->cache_hs_write_squash = 0; - stats->cache_inmem_splittable = 0; - stats->cache_inmem_split = 0; - stats->cache_eviction_internal = 0; - stats->cache_eviction_split_internal = 0; - stats->cache_eviction_split_leaf = 0; - stats->cache_eviction_dirty = 0; - stats->cache_read_overflow = 0; - stats->cache_eviction_deepen = 0; - stats->cache_write_hs = 0; - stats->cache_read = 0; - stats->cache_read_deleted = 0; - stats->cache_read_deleted_prepared = 0; - stats->cache_pages_requested = 0; - stats->cache_eviction_pages_seen = 0; - stats->cache_write = 0; - stats->cache_write_restore = 0; - /* not clearing cache_bytes_dirty */ - stats->cache_eviction_clean = 0; - stats->cc_pages_evict = 0; - stats->cc_pages_removed = 0; - stats->cc_pages_walk_skipped = 0; - stats->cc_pages_visited = 0; - stats->cursor_next_skip_total = 0; - stats->cursor_prev_skip_total = 0; - stats->cursor_skip_hs_cur_position = 0; - stats->cursor_search_near_prefix_fast_paths = 0; - stats->cursor_next_hs_tombstone = 0; - stats->cursor_next_skip_ge_100 = 0; - stats->cursor_next_skip_lt_100 = 0; - stats->cursor_prev_hs_tombstone = 0; - stats->cursor_prev_skip_ge_100 = 0; - stats->cursor_prev_skip_lt_100 = 0; - /* not clearing cursor_open_count */ - stats->rec_time_window_bytes_ts = 0; - stats->rec_time_window_bytes_txn = 0; - stats->rec_page_delete_fast = 0; - stats->rec_pages = 0; - stats->rec_pages_eviction = 0; - stats->rec_page_delete = 0; - stats->rec_time_aggr_newest_start_durable_ts = 0; - stats->rec_time_aggr_newest_stop_durable_ts = 0; - stats->rec_time_aggr_newest_stop_ts = 0; - stats->rec_time_aggr_newest_stop_txn = 0; - stats->rec_time_aggr_newest_txn = 0; - stats->rec_time_aggr_oldest_start_ts = 0; - stats->rec_time_aggr_prepared = 0; - stats->rec_time_window_pages_durable_start_ts = 0; - stats->rec_time_window_pages_start_txn = 0; - stats->rec_time_window_pages_durable_stop_ts = 0; - stats->rec_time_window_pages_stop_ts = 0; - stats->rec_time_window_pages_stop_txn = 0; - stats->rec_time_window_durable_start_ts = 0; - stats->rec_time_window_start_ts = 0; - stats->rec_time_window_start_txn = 0; - stats->rec_time_window_durable_stop_ts = 0; - stats->rec_time_window_stop_ts = 0; - stats->rec_time_window_stop_txn = 0; - stats->tiered_work_units_dequeued = 0; - stats->tiered_work_units_created = 0; - /* not clearing tiered_retention */ - /* not clearing tiered_object_size */ - stats->txn_read_race_prepare_update = 0; - stats->txn_rts_hs_stop_older_than_newer_start = 0; - stats->txn_rts_inconsistent_ckpt = 0; - stats->txn_rts_keys_removed = 0; - stats->txn_rts_keys_restored = 0; - stats->txn_rts_hs_restore_tombstones = 0; - stats->txn_rts_hs_restore_updates = 0; - stats->txn_rts_sweep_hs_keys = 0; - stats->txn_rts_hs_removed = 0; - stats->txn_checkpoint_obsolete_applied = 0; stats->txn_update_conflict = 0; } @@ -2002,6 +2015,8 @@ __wt_stat_connection_aggregate(WT_CONNECTION_STATS **from, WT_CONNECTION_STATS * to->lsm_work_queue_app += WT_STAT_READ(from, lsm_work_queue_app); to->lsm_work_queue_manager += WT_STAT_READ(from, lsm_work_queue_manager); to->lsm_rows_merged += WT_STAT_READ(from, lsm_rows_merged); + to->lsm_checkpoint_throttle += WT_STAT_READ(from, lsm_checkpoint_throttle); + to->lsm_merge_throttle += WT_STAT_READ(from, lsm_merge_throttle); to->lsm_work_queue_switch += WT_STAT_READ(from, lsm_work_queue_switch); to->lsm_work_units_discarded += WT_STAT_READ(from, lsm_work_units_discarded); to->lsm_work_units_done += WT_STAT_READ(from, lsm_work_units_done); @@ -2028,8 +2043,15 @@ __wt_stat_connection_aggregate(WT_CONNECTION_STATS **from, WT_CONNECTION_STATS * to->cache_bytes_updates += WT_STAT_READ(from, cache_bytes_updates); to->cache_bytes_image += WT_STAT_READ(from, cache_bytes_image); to->cache_bytes_hs += WT_STAT_READ(from, cache_bytes_hs); + to->cache_bytes_inuse += WT_STAT_READ(from, cache_bytes_inuse); + to->cache_bytes_dirty_total += WT_STAT_READ(from, cache_bytes_dirty_total); to->cache_bytes_other += WT_STAT_READ(from, cache_bytes_other); + to->cache_bytes_read += WT_STAT_READ(from, cache_bytes_read); + to->cache_bytes_write += WT_STAT_READ(from, cache_bytes_write); to->cache_lookaside_score += WT_STAT_READ(from, cache_lookaside_score); + to->cache_eviction_checkpoint += WT_STAT_READ(from, cache_eviction_checkpoint); + to->cache_eviction_blocked_checkpoint_hs += + WT_STAT_READ(from, cache_eviction_blocked_checkpoint_hs); to->cache_eviction_get_ref += WT_STAT_READ(from, cache_eviction_get_ref); to->cache_eviction_get_ref_empty += WT_STAT_READ(from, cache_eviction_get_ref_empty); to->cache_eviction_get_ref_empty2 += WT_STAT_READ(from, cache_eviction_get_ref_empty2); @@ -2043,12 +2065,29 @@ __wt_stat_connection_aggregate(WT_CONNECTION_STATS **from, WT_CONNECTION_STATS * to->cache_eviction_slow += WT_STAT_READ(from, cache_eviction_slow); to->cache_eviction_walk_leaf_notfound += WT_STAT_READ(from, cache_eviction_walk_leaf_notfound); to->cache_eviction_state += WT_STAT_READ(from, cache_eviction_state); + to->cache_eviction_target_page_lt10 += WT_STAT_READ(from, cache_eviction_target_page_lt10); + to->cache_eviction_target_page_lt32 += WT_STAT_READ(from, cache_eviction_target_page_lt32); + to->cache_eviction_target_page_ge128 += WT_STAT_READ(from, cache_eviction_target_page_ge128); + to->cache_eviction_target_page_lt64 += WT_STAT_READ(from, cache_eviction_target_page_lt64); + to->cache_eviction_target_page_lt128 += WT_STAT_READ(from, cache_eviction_target_page_lt128); + to->cache_eviction_target_page_reduced += + WT_STAT_READ(from, cache_eviction_target_page_reduced); to->cache_eviction_target_strategy_both_clean_and_dirty += WT_STAT_READ(from, cache_eviction_target_strategy_both_clean_and_dirty); to->cache_eviction_target_strategy_clean += WT_STAT_READ(from, cache_eviction_target_strategy_clean); to->cache_eviction_target_strategy_dirty += WT_STAT_READ(from, cache_eviction_target_strategy_dirty); + to->cache_eviction_walks_abandoned += WT_STAT_READ(from, cache_eviction_walks_abandoned); + to->cache_eviction_walks_stopped += WT_STAT_READ(from, cache_eviction_walks_stopped); + to->cache_eviction_walks_gave_up_no_targets += + WT_STAT_READ(from, cache_eviction_walks_gave_up_no_targets); + to->cache_eviction_walks_gave_up_ratio += + WT_STAT_READ(from, cache_eviction_walks_gave_up_ratio); + to->cache_eviction_walks_ended += WT_STAT_READ(from, cache_eviction_walks_ended); + to->cache_eviction_walk_restart += WT_STAT_READ(from, cache_eviction_walk_restart); + to->cache_eviction_walk_from_root += WT_STAT_READ(from, cache_eviction_walk_from_root); + to->cache_eviction_walk_saved_pos += WT_STAT_READ(from, cache_eviction_walk_saved_pos); to->cache_eviction_active_workers += WT_STAT_READ(from, cache_eviction_active_workers); to->cache_eviction_worker_created += WT_STAT_READ(from, cache_eviction_worker_created); to->cache_eviction_worker_evicting += WT_STAT_READ(from, cache_eviction_worker_evicting); @@ -2069,23 +2108,49 @@ __wt_stat_connection_aggregate(WT_CONNECTION_STATS **from, WT_CONNECTION_STATS * to->cache_eviction_force += WT_STAT_READ(from, cache_eviction_force); to->cache_eviction_force_fail += WT_STAT_READ(from, cache_eviction_force_fail); to->cache_eviction_force_fail_time += WT_STAT_READ(from, cache_eviction_force_fail_time); + to->cache_eviction_hazard += WT_STAT_READ(from, cache_eviction_hazard); to->cache_hazard_checks += WT_STAT_READ(from, cache_hazard_checks); to->cache_hazard_walks += WT_STAT_READ(from, cache_hazard_walks); if ((v = WT_STAT_READ(from, cache_hazard_max)) > to->cache_hazard_max) to->cache_hazard_max = v; to->cache_hs_score += WT_STAT_READ(from, cache_hs_score); + to->cache_hs_insert += WT_STAT_READ(from, cache_hs_insert); + to->cache_hs_insert_restart += WT_STAT_READ(from, cache_hs_insert_restart); to->cache_hs_ondisk_max += WT_STAT_READ(from, cache_hs_ondisk_max); to->cache_hs_ondisk += WT_STAT_READ(from, cache_hs_ondisk); + to->cache_hs_order_lose_durable_timestamp += + WT_STAT_READ(from, cache_hs_order_lose_durable_timestamp); + to->cache_hs_order_reinsert += WT_STAT_READ(from, cache_hs_order_reinsert); + to->cache_hs_read += WT_STAT_READ(from, cache_hs_read); + to->cache_hs_read_miss += WT_STAT_READ(from, cache_hs_read_miss); + to->cache_hs_read_squash += WT_STAT_READ(from, cache_hs_read_squash); + to->cache_hs_key_truncate_rts_unstable += + WT_STAT_READ(from, cache_hs_key_truncate_rts_unstable); + to->cache_hs_key_truncate_rts += WT_STAT_READ(from, cache_hs_key_truncate_rts); + to->cache_hs_key_truncate += WT_STAT_READ(from, cache_hs_key_truncate); + to->cache_hs_key_truncate_onpage_removal += + WT_STAT_READ(from, cache_hs_key_truncate_onpage_removal); + to->cache_hs_order_remove += WT_STAT_READ(from, cache_hs_order_remove); + to->cache_hs_write_squash += WT_STAT_READ(from, cache_hs_write_squash); + to->cache_inmem_splittable += WT_STAT_READ(from, cache_inmem_splittable); + to->cache_inmem_split += WT_STAT_READ(from, cache_inmem_split); + to->cache_eviction_internal += WT_STAT_READ(from, cache_eviction_internal); to->cache_eviction_internal_pages_queued += WT_STAT_READ(from, cache_eviction_internal_pages_queued); to->cache_eviction_internal_pages_seen += WT_STAT_READ(from, cache_eviction_internal_pages_seen); to->cache_eviction_internal_pages_already_queued += WT_STAT_READ(from, cache_eviction_internal_pages_already_queued); + to->cache_eviction_split_internal += WT_STAT_READ(from, cache_eviction_split_internal); + to->cache_eviction_split_leaf += WT_STAT_READ(from, cache_eviction_split_leaf); to->cache_bytes_max += WT_STAT_READ(from, cache_bytes_max); to->cache_eviction_maximum_page_size += WT_STAT_READ(from, cache_eviction_maximum_page_size); + to->cache_eviction_dirty += WT_STAT_READ(from, cache_eviction_dirty); to->cache_eviction_app_dirty += WT_STAT_READ(from, cache_eviction_app_dirty); to->cache_timed_out_ops += WT_STAT_READ(from, cache_timed_out_ops); + to->cache_read_overflow += WT_STAT_READ(from, cache_read_overflow); + to->cache_eviction_deepen += WT_STAT_READ(from, cache_eviction_deepen); + to->cache_write_hs += WT_STAT_READ(from, cache_write_hs); to->cache_pages_inuse += WT_STAT_READ(from, cache_pages_inuse); to->cache_eviction_app += WT_STAT_READ(from, cache_eviction_app); to->cache_eviction_pages_in_parallel_with_checkpoint += @@ -2099,6 +2164,11 @@ __wt_stat_connection_aggregate(WT_CONNECTION_STATS **from, WT_CONNECTION_STATS * WT_STAT_READ(from, cache_eviction_pages_queued_oldest); to->cache_eviction_pages_queued_urgent_hs_dirty += WT_STAT_READ(from, cache_eviction_pages_queued_urgent_hs_dirty); + to->cache_read += WT_STAT_READ(from, cache_read); + to->cache_read_deleted += WT_STAT_READ(from, cache_read_deleted); + to->cache_read_deleted_prepared += WT_STAT_READ(from, cache_read_deleted_prepared); + to->cache_pages_requested += WT_STAT_READ(from, cache_pages_requested); + to->cache_eviction_pages_seen += WT_STAT_READ(from, cache_eviction_pages_seen); to->cache_eviction_pages_already_queued += WT_STAT_READ(from, cache_eviction_pages_already_queued); to->cache_eviction_fail += WT_STAT_READ(from, cache_eviction_fail); @@ -2108,11 +2178,17 @@ __wt_stat_connection_aggregate(WT_CONNECTION_STATS **from, WT_CONNECTION_STATS * WT_STAT_READ(from, cache_eviction_fail_active_children_on_an_internal_page); to->cache_eviction_fail_in_reconciliation += WT_STAT_READ(from, cache_eviction_fail_in_reconciliation); + to->cache_eviction_fail_checkpoint_out_of_order_ts += + WT_STAT_READ(from, cache_eviction_fail_checkpoint_out_of_order_ts); to->cache_eviction_walk += WT_STAT_READ(from, cache_eviction_walk); + to->cache_write += WT_STAT_READ(from, cache_write); + to->cache_write_restore += WT_STAT_READ(from, cache_write_restore); to->cache_overhead += WT_STAT_READ(from, cache_overhead); to->cache_bytes_internal += WT_STAT_READ(from, cache_bytes_internal); to->cache_bytes_leaf += WT_STAT_READ(from, cache_bytes_leaf); + to->cache_bytes_dirty += WT_STAT_READ(from, cache_bytes_dirty); to->cache_pages_dirty += WT_STAT_READ(from, cache_pages_dirty); + to->cache_eviction_clean += WT_STAT_READ(from, cache_eviction_clean); to->fsync_all_fh_total += WT_STAT_READ(from, fsync_all_fh_total); to->fsync_all_fh += WT_STAT_READ(from, fsync_all_fh); to->fsync_all_time += WT_STAT_READ(from, fsync_all_time); @@ -2127,6 +2203,10 @@ __wt_stat_connection_aggregate(WT_CONNECTION_STATS **from, WT_CONNECTION_STATS * to->capacity_time_evict += WT_STAT_READ(from, capacity_time_evict); to->capacity_time_log += WT_STAT_READ(from, capacity_time_log); to->capacity_time_read += WT_STAT_READ(from, capacity_time_read); + to->cc_pages_evict += WT_STAT_READ(from, cc_pages_evict); + to->cc_pages_removed += WT_STAT_READ(from, cc_pages_removed); + to->cc_pages_walk_skipped += WT_STAT_READ(from, cc_pages_walk_skipped); + to->cc_pages_visited += WT_STAT_READ(from, cc_pages_visited); to->cond_auto_wait_reset += WT_STAT_READ(from, cond_auto_wait_reset); to->cond_auto_wait += WT_STAT_READ(from, cond_auto_wait); to->cond_auto_wait_skipped += WT_STAT_READ(from, cond_auto_wait_skipped); @@ -2143,6 +2223,13 @@ __wt_stat_connection_aggregate(WT_CONNECTION_STATS **from, WT_CONNECTION_STATS * to->fsync_io += WT_STAT_READ(from, fsync_io); to->read_io += WT_STAT_READ(from, read_io); to->write_io += WT_STAT_READ(from, write_io); + to->cursor_next_skip_total += WT_STAT_READ(from, cursor_next_skip_total); + to->cursor_prev_skip_total += WT_STAT_READ(from, cursor_prev_skip_total); + to->cursor_skip_hs_cur_position += WT_STAT_READ(from, cursor_skip_hs_cur_position); + to->cursor_next_skip_page_count += WT_STAT_READ(from, cursor_next_skip_page_count); + to->cursor_prev_skip_page_count += WT_STAT_READ(from, cursor_prev_skip_page_count); + to->cursor_search_near_prefix_fast_paths += + WT_STAT_READ(from, cursor_search_near_prefix_fast_paths); to->cursor_cached_count += WT_STAT_READ(from, cursor_cached_count); to->cursor_insert_bulk += WT_STAT_READ(from, cursor_insert_bulk); to->cursor_cache += WT_STAT_READ(from, cursor_cache); @@ -2153,8 +2240,14 @@ __wt_stat_connection_aggregate(WT_CONNECTION_STATS **from, WT_CONNECTION_STATS * to->cursor_modify_bytes += WT_STAT_READ(from, cursor_modify_bytes); to->cursor_modify_bytes_touch += WT_STAT_READ(from, cursor_modify_bytes_touch); to->cursor_next += WT_STAT_READ(from, cursor_next); + to->cursor_next_hs_tombstone += WT_STAT_READ(from, cursor_next_hs_tombstone); + to->cursor_next_skip_ge_100 += WT_STAT_READ(from, cursor_next_skip_ge_100); + to->cursor_next_skip_lt_100 += WT_STAT_READ(from, cursor_next_skip_lt_100); to->cursor_restart += WT_STAT_READ(from, cursor_restart); to->cursor_prev += WT_STAT_READ(from, cursor_prev); + to->cursor_prev_hs_tombstone += WT_STAT_READ(from, cursor_prev_hs_tombstone); + to->cursor_prev_skip_ge_100 += WT_STAT_READ(from, cursor_prev_skip_ge_100); + to->cursor_prev_skip_lt_100 += WT_STAT_READ(from, cursor_prev_skip_lt_100); to->cursor_remove += WT_STAT_READ(from, cursor_remove); to->cursor_remove_bytes += WT_STAT_READ(from, cursor_remove_bytes); to->cursor_reserve += WT_STAT_READ(from, cursor_reserve); @@ -2171,6 +2264,7 @@ __wt_stat_connection_aggregate(WT_CONNECTION_STATS **from, WT_CONNECTION_STATS * to->cursor_update_bytes += WT_STAT_READ(from, cursor_update_bytes); to->cursor_update_bytes_changed += WT_STAT_READ(from, cursor_update_bytes_changed); to->cursor_reopen += WT_STAT_READ(from, cursor_reopen); + to->cursor_open_count += WT_STAT_READ(from, cursor_open_count); to->dh_conn_handle_size += WT_STAT_READ(from, dh_conn_handle_size); to->dh_conn_handle_count += WT_STAT_READ(from, dh_conn_handle_count); to->dh_sweep_ref += WT_STAT_READ(from, dh_sweep_ref); @@ -2282,19 +2376,46 @@ __wt_stat_connection_aggregate(WT_CONNECTION_STATS **from, WT_CONNECTION_STATS * to->perf_hist_opwrite_latency_lt1000 += WT_STAT_READ(from, perf_hist_opwrite_latency_lt1000); to->perf_hist_opwrite_latency_lt10000 += WT_STAT_READ(from, perf_hist_opwrite_latency_lt10000); to->perf_hist_opwrite_latency_gt10000 += WT_STAT_READ(from, perf_hist_opwrite_latency_gt10000); + to->rec_time_window_bytes_ts += WT_STAT_READ(from, rec_time_window_bytes_ts); + to->rec_time_window_bytes_txn += WT_STAT_READ(from, rec_time_window_bytes_txn); + to->rec_page_delete_fast += WT_STAT_READ(from, rec_page_delete_fast); to->rec_overflow_key_internal += WT_STAT_READ(from, rec_overflow_key_internal); to->rec_overflow_key_leaf += WT_STAT_READ(from, rec_overflow_key_leaf); to->rec_maximum_seconds += WT_STAT_READ(from, rec_maximum_seconds); + to->rec_pages += WT_STAT_READ(from, rec_pages); + to->rec_pages_eviction += WT_STAT_READ(from, rec_pages_eviction); to->rec_pages_with_prepare += WT_STAT_READ(from, rec_pages_with_prepare); to->rec_pages_with_ts += WT_STAT_READ(from, rec_pages_with_ts); to->rec_pages_with_txn += WT_STAT_READ(from, rec_pages_with_txn); + to->rec_page_delete += WT_STAT_READ(from, rec_page_delete); + to->rec_time_aggr_newest_start_durable_ts += + WT_STAT_READ(from, rec_time_aggr_newest_start_durable_ts); + to->rec_time_aggr_newest_stop_durable_ts += + WT_STAT_READ(from, rec_time_aggr_newest_stop_durable_ts); + to->rec_time_aggr_newest_stop_ts += WT_STAT_READ(from, rec_time_aggr_newest_stop_ts); + to->rec_time_aggr_newest_stop_txn += WT_STAT_READ(from, rec_time_aggr_newest_stop_txn); + to->rec_time_aggr_newest_txn += WT_STAT_READ(from, rec_time_aggr_newest_txn); + to->rec_time_aggr_oldest_start_ts += WT_STAT_READ(from, rec_time_aggr_oldest_start_ts); + to->rec_time_aggr_prepared += WT_STAT_READ(from, rec_time_aggr_prepared); to->rec_time_window_pages_prepared += WT_STAT_READ(from, rec_time_window_pages_prepared); + to->rec_time_window_pages_durable_start_ts += + WT_STAT_READ(from, rec_time_window_pages_durable_start_ts); to->rec_time_window_pages_start_ts += WT_STAT_READ(from, rec_time_window_pages_start_ts); + to->rec_time_window_pages_start_txn += WT_STAT_READ(from, rec_time_window_pages_start_txn); + to->rec_time_window_pages_durable_stop_ts += + WT_STAT_READ(from, rec_time_window_pages_durable_stop_ts); + to->rec_time_window_pages_stop_ts += WT_STAT_READ(from, rec_time_window_pages_stop_ts); + to->rec_time_window_pages_stop_txn += WT_STAT_READ(from, rec_time_window_pages_stop_txn); to->rec_time_window_prepared += WT_STAT_READ(from, rec_time_window_prepared); + to->rec_time_window_durable_start_ts += WT_STAT_READ(from, rec_time_window_durable_start_ts); + to->rec_time_window_start_ts += WT_STAT_READ(from, rec_time_window_start_ts); + to->rec_time_window_start_txn += WT_STAT_READ(from, rec_time_window_start_txn); + to->rec_time_window_durable_stop_ts += WT_STAT_READ(from, rec_time_window_durable_stop_ts); + to->rec_time_window_stop_ts += WT_STAT_READ(from, rec_time_window_stop_ts); + to->rec_time_window_stop_txn += WT_STAT_READ(from, rec_time_window_stop_txn); to->rec_split_stashed_bytes += WT_STAT_READ(from, rec_split_stashed_bytes); to->rec_split_stashed_objects += WT_STAT_READ(from, rec_split_stashed_objects); to->flush_state_races += WT_STAT_READ(from, flush_state_races); - to->flush_tier_busy += WT_STAT_READ(from, flush_tier_busy); to->flush_tier += WT_STAT_READ(from, flush_tier); to->session_open += WT_STAT_READ(from, session_open); to->session_query_ts += WT_STAT_READ(from, session_query_ts); @@ -2315,6 +2436,10 @@ __wt_stat_connection_aggregate(WT_CONNECTION_STATS **from, WT_CONNECTION_STATS * to->session_table_truncate_success += WT_STAT_READ(from, session_table_truncate_success); to->session_table_verify_fail += WT_STAT_READ(from, session_table_verify_fail); to->session_table_verify_success += WT_STAT_READ(from, session_table_verify_success); + to->tiered_work_units_dequeued += WT_STAT_READ(from, tiered_work_units_dequeued); + to->tiered_work_units_created += WT_STAT_READ(from, tiered_work_units_created); + to->tiered_retention += WT_STAT_READ(from, tiered_retention); + to->tiered_object_size += WT_STAT_READ(from, tiered_object_size); to->thread_fsync_active += WT_STAT_READ(from, thread_fsync_active); to->thread_read_active += WT_STAT_READ(from, thread_read_active); to->thread_write_active += WT_STAT_READ(from, thread_write_active); @@ -2342,10 +2467,20 @@ __wt_stat_connection_aggregate(WT_CONNECTION_STATS **from, WT_CONNECTION_STATS * to->txn_prepare_active += WT_STAT_READ(from, txn_prepare_active); to->txn_prepare_rollback += WT_STAT_READ(from, txn_prepare_rollback); to->txn_query_ts += WT_STAT_READ(from, txn_query_ts); + to->txn_read_race_prepare_update += WT_STAT_READ(from, txn_read_race_prepare_update); to->txn_rts += WT_STAT_READ(from, txn_rts); + to->txn_rts_hs_stop_older_than_newer_start += + WT_STAT_READ(from, txn_rts_hs_stop_older_than_newer_start); + to->txn_rts_inconsistent_ckpt += WT_STAT_READ(from, txn_rts_inconsistent_ckpt); + to->txn_rts_keys_removed += WT_STAT_READ(from, txn_rts_keys_removed); + to->txn_rts_keys_restored += WT_STAT_READ(from, txn_rts_keys_restored); to->txn_rts_pages_visited += WT_STAT_READ(from, txn_rts_pages_visited); + to->txn_rts_hs_restore_tombstones += WT_STAT_READ(from, txn_rts_hs_restore_tombstones); + to->txn_rts_hs_restore_updates += WT_STAT_READ(from, txn_rts_hs_restore_updates); + to->txn_rts_sweep_hs_keys += WT_STAT_READ(from, txn_rts_sweep_hs_keys); to->txn_rts_tree_walk_skip_pages += WT_STAT_READ(from, txn_rts_tree_walk_skip_pages); to->txn_rts_upd_aborted += WT_STAT_READ(from, txn_rts_upd_aborted); + to->txn_rts_hs_removed += WT_STAT_READ(from, txn_rts_hs_removed); to->txn_sessions_walked += WT_STAT_READ(from, txn_sessions_walked); to->txn_set_ts += WT_STAT_READ(from, txn_set_ts); to->txn_set_ts_durable += WT_STAT_READ(from, txn_set_ts_durable); @@ -2379,6 +2514,7 @@ __wt_stat_connection_aggregate(WT_CONNECTION_STATS **from, WT_CONNECTION_STATS * to->txn_checkpoint_scrub_time += WT_STAT_READ(from, txn_checkpoint_scrub_time); to->txn_checkpoint_time_total += WT_STAT_READ(from, txn_checkpoint_time_total); to->txn_checkpoint += WT_STAT_READ(from, txn_checkpoint); + to->txn_checkpoint_obsolete_applied += WT_STAT_READ(from, txn_checkpoint_obsolete_applied); to->txn_checkpoint_skipped += WT_STAT_READ(from, txn_checkpoint_skipped); to->txn_fail_cache += WT_STAT_READ(from, txn_fail_cache); to->txn_checkpoint_fsync_post += WT_STAT_READ(from, txn_checkpoint_fsync_post); @@ -2396,126 +2532,6 @@ __wt_stat_connection_aggregate(WT_CONNECTION_STATS **from, WT_CONNECTION_STATS * to->txn_walk_sessions += WT_STAT_READ(from, txn_walk_sessions); to->txn_commit += WT_STAT_READ(from, txn_commit); to->txn_rollback += WT_STAT_READ(from, txn_rollback); - to->lsm_checkpoint_throttle += WT_STAT_READ(from, lsm_checkpoint_throttle); - to->lsm_merge_throttle += WT_STAT_READ(from, lsm_merge_throttle); - to->cache_bytes_inuse += WT_STAT_READ(from, cache_bytes_inuse); - to->cache_bytes_dirty_total += WT_STAT_READ(from, cache_bytes_dirty_total); - to->cache_bytes_read += WT_STAT_READ(from, cache_bytes_read); - to->cache_bytes_write += WT_STAT_READ(from, cache_bytes_write); - to->cache_eviction_checkpoint += WT_STAT_READ(from, cache_eviction_checkpoint); - to->cache_eviction_blocked_checkpoint_hs += - WT_STAT_READ(from, cache_eviction_blocked_checkpoint_hs); - to->cache_eviction_target_page_lt10 += WT_STAT_READ(from, cache_eviction_target_page_lt10); - to->cache_eviction_target_page_lt32 += WT_STAT_READ(from, cache_eviction_target_page_lt32); - to->cache_eviction_target_page_ge128 += WT_STAT_READ(from, cache_eviction_target_page_ge128); - to->cache_eviction_target_page_lt64 += WT_STAT_READ(from, cache_eviction_target_page_lt64); - to->cache_eviction_target_page_lt128 += WT_STAT_READ(from, cache_eviction_target_page_lt128); - to->cache_eviction_target_page_reduced += - WT_STAT_READ(from, cache_eviction_target_page_reduced); - to->cache_eviction_walks_abandoned += WT_STAT_READ(from, cache_eviction_walks_abandoned); - to->cache_eviction_walks_stopped += WT_STAT_READ(from, cache_eviction_walks_stopped); - to->cache_eviction_walks_gave_up_no_targets += - WT_STAT_READ(from, cache_eviction_walks_gave_up_no_targets); - to->cache_eviction_walks_gave_up_ratio += - WT_STAT_READ(from, cache_eviction_walks_gave_up_ratio); - to->cache_eviction_walks_ended += WT_STAT_READ(from, cache_eviction_walks_ended); - to->cache_eviction_walk_restart += WT_STAT_READ(from, cache_eviction_walk_restart); - to->cache_eviction_walk_from_root += WT_STAT_READ(from, cache_eviction_walk_from_root); - to->cache_eviction_walk_saved_pos += WT_STAT_READ(from, cache_eviction_walk_saved_pos); - to->cache_eviction_hazard += WT_STAT_READ(from, cache_eviction_hazard); - to->cache_hs_insert += WT_STAT_READ(from, cache_hs_insert); - to->cache_hs_insert_restart += WT_STAT_READ(from, cache_hs_insert_restart); - to->cache_hs_order_lose_durable_timestamp += - WT_STAT_READ(from, cache_hs_order_lose_durable_timestamp); - to->cache_hs_order_reinsert += WT_STAT_READ(from, cache_hs_order_reinsert); - to->cache_hs_read += WT_STAT_READ(from, cache_hs_read); - to->cache_hs_read_miss += WT_STAT_READ(from, cache_hs_read_miss); - to->cache_hs_read_squash += WT_STAT_READ(from, cache_hs_read_squash); - to->cache_hs_key_truncate_rts_unstable += - WT_STAT_READ(from, cache_hs_key_truncate_rts_unstable); - to->cache_hs_key_truncate_rts += WT_STAT_READ(from, cache_hs_key_truncate_rts); - to->cache_hs_key_truncate += WT_STAT_READ(from, cache_hs_key_truncate); - to->cache_hs_key_truncate_onpage_removal += - WT_STAT_READ(from, cache_hs_key_truncate_onpage_removal); - to->cache_hs_order_remove += WT_STAT_READ(from, cache_hs_order_remove); - to->cache_hs_write_squash += WT_STAT_READ(from, cache_hs_write_squash); - to->cache_inmem_splittable += WT_STAT_READ(from, cache_inmem_splittable); - to->cache_inmem_split += WT_STAT_READ(from, cache_inmem_split); - to->cache_eviction_internal += WT_STAT_READ(from, cache_eviction_internal); - to->cache_eviction_split_internal += WT_STAT_READ(from, cache_eviction_split_internal); - to->cache_eviction_split_leaf += WT_STAT_READ(from, cache_eviction_split_leaf); - to->cache_eviction_dirty += WT_STAT_READ(from, cache_eviction_dirty); - to->cache_read_overflow += WT_STAT_READ(from, cache_read_overflow); - to->cache_eviction_deepen += WT_STAT_READ(from, cache_eviction_deepen); - to->cache_write_hs += WT_STAT_READ(from, cache_write_hs); - to->cache_read += WT_STAT_READ(from, cache_read); - to->cache_read_deleted += WT_STAT_READ(from, cache_read_deleted); - to->cache_read_deleted_prepared += WT_STAT_READ(from, cache_read_deleted_prepared); - to->cache_pages_requested += WT_STAT_READ(from, cache_pages_requested); - to->cache_eviction_pages_seen += WT_STAT_READ(from, cache_eviction_pages_seen); - to->cache_write += WT_STAT_READ(from, cache_write); - to->cache_write_restore += WT_STAT_READ(from, cache_write_restore); - to->cache_bytes_dirty += WT_STAT_READ(from, cache_bytes_dirty); - to->cache_eviction_clean += WT_STAT_READ(from, cache_eviction_clean); - to->cc_pages_evict += WT_STAT_READ(from, cc_pages_evict); - to->cc_pages_removed += WT_STAT_READ(from, cc_pages_removed); - to->cc_pages_walk_skipped += WT_STAT_READ(from, cc_pages_walk_skipped); - to->cc_pages_visited += WT_STAT_READ(from, cc_pages_visited); - to->cursor_next_skip_total += WT_STAT_READ(from, cursor_next_skip_total); - to->cursor_prev_skip_total += WT_STAT_READ(from, cursor_prev_skip_total); - to->cursor_skip_hs_cur_position += WT_STAT_READ(from, cursor_skip_hs_cur_position); - to->cursor_search_near_prefix_fast_paths += - WT_STAT_READ(from, cursor_search_near_prefix_fast_paths); - to->cursor_next_hs_tombstone += WT_STAT_READ(from, cursor_next_hs_tombstone); - to->cursor_next_skip_ge_100 += WT_STAT_READ(from, cursor_next_skip_ge_100); - to->cursor_next_skip_lt_100 += WT_STAT_READ(from, cursor_next_skip_lt_100); - to->cursor_prev_hs_tombstone += WT_STAT_READ(from, cursor_prev_hs_tombstone); - to->cursor_prev_skip_ge_100 += WT_STAT_READ(from, cursor_prev_skip_ge_100); - to->cursor_prev_skip_lt_100 += WT_STAT_READ(from, cursor_prev_skip_lt_100); - to->cursor_open_count += WT_STAT_READ(from, cursor_open_count); - to->rec_time_window_bytes_ts += WT_STAT_READ(from, rec_time_window_bytes_ts); - to->rec_time_window_bytes_txn += WT_STAT_READ(from, rec_time_window_bytes_txn); - to->rec_page_delete_fast += WT_STAT_READ(from, rec_page_delete_fast); - to->rec_pages += WT_STAT_READ(from, rec_pages); - to->rec_pages_eviction += WT_STAT_READ(from, rec_pages_eviction); - to->rec_page_delete += WT_STAT_READ(from, rec_page_delete); - to->rec_time_aggr_newest_start_durable_ts += - WT_STAT_READ(from, rec_time_aggr_newest_start_durable_ts); - to->rec_time_aggr_newest_stop_durable_ts += - WT_STAT_READ(from, rec_time_aggr_newest_stop_durable_ts); - to->rec_time_aggr_newest_stop_ts += WT_STAT_READ(from, rec_time_aggr_newest_stop_ts); - to->rec_time_aggr_newest_stop_txn += WT_STAT_READ(from, rec_time_aggr_newest_stop_txn); - to->rec_time_aggr_newest_txn += WT_STAT_READ(from, rec_time_aggr_newest_txn); - to->rec_time_aggr_oldest_start_ts += WT_STAT_READ(from, rec_time_aggr_oldest_start_ts); - to->rec_time_aggr_prepared += WT_STAT_READ(from, rec_time_aggr_prepared); - to->rec_time_window_pages_durable_start_ts += - WT_STAT_READ(from, rec_time_window_pages_durable_start_ts); - to->rec_time_window_pages_start_txn += WT_STAT_READ(from, rec_time_window_pages_start_txn); - to->rec_time_window_pages_durable_stop_ts += - WT_STAT_READ(from, rec_time_window_pages_durable_stop_ts); - to->rec_time_window_pages_stop_ts += WT_STAT_READ(from, rec_time_window_pages_stop_ts); - to->rec_time_window_pages_stop_txn += WT_STAT_READ(from, rec_time_window_pages_stop_txn); - to->rec_time_window_durable_start_ts += WT_STAT_READ(from, rec_time_window_durable_start_ts); - to->rec_time_window_start_ts += WT_STAT_READ(from, rec_time_window_start_ts); - to->rec_time_window_start_txn += WT_STAT_READ(from, rec_time_window_start_txn); - to->rec_time_window_durable_stop_ts += WT_STAT_READ(from, rec_time_window_durable_stop_ts); - to->rec_time_window_stop_ts += WT_STAT_READ(from, rec_time_window_stop_ts); - to->rec_time_window_stop_txn += WT_STAT_READ(from, rec_time_window_stop_txn); - to->tiered_work_units_dequeued += WT_STAT_READ(from, tiered_work_units_dequeued); - to->tiered_work_units_created += WT_STAT_READ(from, tiered_work_units_created); - to->tiered_retention += WT_STAT_READ(from, tiered_retention); - to->tiered_object_size += WT_STAT_READ(from, tiered_object_size); - to->txn_read_race_prepare_update += WT_STAT_READ(from, txn_read_race_prepare_update); - to->txn_rts_hs_stop_older_than_newer_start += - WT_STAT_READ(from, txn_rts_hs_stop_older_than_newer_start); - to->txn_rts_inconsistent_ckpt += WT_STAT_READ(from, txn_rts_inconsistent_ckpt); - to->txn_rts_keys_removed += WT_STAT_READ(from, txn_rts_keys_removed); - to->txn_rts_keys_restored += WT_STAT_READ(from, txn_rts_keys_restored); - to->txn_rts_hs_restore_tombstones += WT_STAT_READ(from, txn_rts_hs_restore_tombstones); - to->txn_rts_hs_restore_updates += WT_STAT_READ(from, txn_rts_hs_restore_updates); - to->txn_rts_sweep_hs_keys += WT_STAT_READ(from, txn_rts_sweep_hs_keys); - to->txn_rts_hs_removed += WT_STAT_READ(from, txn_rts_hs_removed); - to->txn_checkpoint_obsolete_applied += WT_STAT_READ(from, txn_checkpoint_obsolete_applied); to->txn_update_conflict += WT_STAT_READ(from, txn_update_conflict); } diff --git a/src/third_party/wiredtiger/src/tiered/tiered_config.c b/src/third_party/wiredtiger/src/tiered/tiered_config.c index 8c0ec27333e..5c88ea1adaa 100644 --- a/src/third_party/wiredtiger/src/tiered/tiered_config.c +++ b/src/third_party/wiredtiger/src/tiered/tiered_config.c @@ -121,7 +121,7 @@ __wt_tiered_bucket_config( storage, &session->iface, new->bucket, new->auth_token, NULL, &new->file_system)); new->storage_source = storage; - /* If we're creating a new bucket storage, parse the other settings into it. */ + /* If we're creating a new bucket storage, parse the other settings into it. */ TAILQ_INSERT_HEAD(&nstorage->bucketqh, new, q); TAILQ_INSERT_HEAD(&nstorage->buckethashqh[hash_bucket], new, hashq); F_SET(new, WT_BUCKET_FREE); diff --git a/src/third_party/wiredtiger/src/tiered/tiered_handle.c b/src/third_party/wiredtiger/src/tiered/tiered_handle.c index 007f8f5de39..af7a81bf341 100644 --- a/src/third_party/wiredtiger/src/tiered/tiered_handle.c +++ b/src/third_party/wiredtiger/src/tiered/tiered_handle.c @@ -26,6 +26,12 @@ __tiered_dhandle_setup(WT_SESSION_IMPL *session, WT_TIERED *tiered, uint32_t i, id = WT_TIERED_INDEX_LOCAL; else if (type == WT_DHANDLE_TYPE_TIERED) id = WT_TIERED_INDEX_LOCAL; + else if (type == WT_DHANDLE_TYPE_TIERED_TREE) + /* + * FIXME-WT-7538: this type can be removed. For now, there is nothing to do for this + * type. + */ + goto err; else WT_ERR_MSG( session, EINVAL, "Unknown or unsupported tiered dhandle type %" PRIu32, type); @@ -57,6 +63,7 @@ __tiered_init_tiers(WT_SESSION_IMPL *session, WT_TIERED *tiered, WT_CONFIG_ITEM WT_CONFIG_ITEM ckey, cval; WT_DECL_ITEM(tmp); WT_DECL_RET; + WT_TIERED_TIERS *local_tier; WT_RET(__wt_scr_alloc(session, 0, &tmp)); __wt_config_subinit(session, &cparser, tierconf); @@ -65,8 +72,16 @@ __tiered_init_tiers(WT_SESSION_IMPL *session, WT_TIERED *tiered, WT_CONFIG_ITEM WT_ERR(__wt_buf_fmt(session, tmp, "%.*s", (int)ckey.len, ckey.str)); __wt_verbose( session, WT_VERB_TIERED, "INIT_TIERS: tiered URI dhandle %s", (char *)tmp->data); - WT_ERR(__tiered_dhandle_setup( - session, tiered, WT_TIERED_INDEX_INVALID, (const char *)tmp->data)); + WT_SAVE_DHANDLE(session, + ret = __tiered_dhandle_setup( + session, tiered, WT_TIERED_INDEX_INVALID, (const char *)tmp->data)); + WT_ERR(ret); + } + local_tier = &tiered->tiers[WT_TIERED_INDEX_LOCAL]; + if (local_tier->name == NULL) { + WT_ERR(__wt_tiered_name( + session, &tiered->iface, tiered->current_id, WT_TIERED_NAME_LOCAL, &local_tier->name)); + F_SET(local_tier, WT_TIERS_OP_READ | WT_TIERS_OP_WRITE); } WT_ERR_NOTFOUND_OK(ret, false); err: @@ -81,6 +96,9 @@ err: static int __tiered_create_local(WT_SESSION_IMPL *session, WT_TIERED *tiered) { + WT_CONFIG cparser; + WT_CONFIG_ITEM ck, cv; + WT_DECL_ITEM(build); WT_DECL_RET; WT_TIERED_TIERS *this_tier; const char *cfg[4] = {NULL, NULL, NULL, NULL}; @@ -101,6 +119,22 @@ __tiered_create_local(WT_SESSION_IMPL *session, WT_TIERED *tiered) __wt_verbose(session, WT_VERB_TIERED, "TIER_CREATE_LOCAL: obj_config: %s : %s", name, cfg[1]); WT_ASSERT(session, tiered->obj_config != NULL); WT_ERR(__wt_config_merge(session, cfg, NULL, (const char **)&config)); + + /* + * Remove any checkpoint entry from the configuration. The local file we are now creating is + * empty and does not have any checkpoints. + */ + WT_ERR(__wt_scr_alloc(session, 1024, &build)); + __wt_config_init(session, &cparser, config); + while ((ret = __wt_config_next(&cparser, &ck, &cv)) == 0) { + if (!WT_STRING_MATCH("checkpoint", ck.str, ck.len)) + /* Append the entry to the new buffer. */ + WT_ERR(__wt_buf_catfmt( + session, build, "%.*s=%.*s,", (int)ck.len, ck.str, (int)cv.len, cv.str)); + } + __wt_free(session, config); + WT_ERR(__wt_strndup(session, build->data, build->size, &config)); + /* * XXX Need to verify user doesn't create a table of the same name. What does LSM do? It * definitely has the same problem with chunks. @@ -119,6 +153,7 @@ __tiered_create_local(WT_SESSION_IMPL *session, WT_TIERED *tiered) WT_ERR(ret); err: + __wt_scr_free(session, &build); if (ret != 0) /* Only free name on error. */ __wt_free(session, name); @@ -258,6 +293,29 @@ err: } /* + * __wt_tiered_set_metadata -- + * Generate the tiered metadata information string into the given buffer. + */ +int +__wt_tiered_set_metadata(WT_SESSION_IMPL *session, WT_TIERED *tiered, WT_ITEM *buf) +{ + uint32_t i; + + WT_RET(__wt_buf_catfmt(session, buf, ",last=%" PRIu32 ",tiers=(", tiered->current_id)); + for (i = 0; i < WT_TIERED_MAX_TIERS; ++i) { + if (tiered->tiers[i].name == NULL) { + __wt_verbose(session, WT_VERB_TIERED, "TIER_SET_META: names[%" PRIu32 "] NULL", i); + continue; + } + __wt_verbose(session, WT_VERB_TIERED, "TIER_SET_META: names[%" PRIu32 "]: %s", i, + tiered->tiers[i].name); + WT_RET(__wt_buf_catfmt(session, buf, "%s\"%s\"", i == 0 ? "" : ",", tiered->tiers[i].name)); + } + WT_RET(__wt_buf_catfmt(session, buf, ")")); + return (0); +} + +/* * __tiered_update_metadata -- * Update the metadata for a tiered structure after object switching. */ @@ -267,7 +325,6 @@ __tiered_update_metadata(WT_SESSION_IMPL *session, WT_TIERED *tiered, const char WT_DATA_HANDLE *dhandle; WT_DECL_ITEM(tmp); WT_DECL_RET; - uint32_t i; const char *cfg[4] = {NULL, NULL, NULL, NULL}; const char *newconfig; @@ -275,17 +332,7 @@ __tiered_update_metadata(WT_SESSION_IMPL *session, WT_TIERED *tiered, const char newconfig = NULL; WT_RET(__wt_scr_alloc(session, 0, &tmp)); - WT_RET(__wt_buf_fmt(session, tmp, "last=%" PRIu32 ",tiers=(", tiered->current_id)); - for (i = 0; i < WT_TIERED_MAX_TIERS; ++i) { - if (tiered->tiers[i].name == NULL) { - __wt_verbose(session, WT_VERB_TIERED, "TIER_UPDATE_META: names[%" PRIu32 "] NULL", i); - continue; - } - __wt_verbose(session, WT_VERB_TIERED, "TIER_UPDATE_META: names[%" PRIu32 "]: %s", i, - tiered->tiers[i].name); - WT_RET(__wt_buf_catfmt(session, tmp, "%s\"%s\"", i == 0 ? "" : ",", tiered->tiers[i].name)); - } - WT_RET(__wt_buf_catfmt(session, tmp, ")")); + WT_ERR(__wt_tiered_set_metadata(session, tiered, tmp)); cfg[0] = WT_CONFIG_BASE(session, tiered_meta); cfg[1] = orig_config; @@ -357,11 +404,7 @@ __tiered_switch(WT_SESSION_IMPL *session, const char *config) /* Create the object: entry in the metadata. */ if (need_object) { WT_ERR(__tiered_create_object(session, tiered)); -#if 1 WT_ERR(__wt_tiered_put_flush(session, tiered)); -#else - WT_ERR(__wt_tier_flush(session, tiered, tiered->current_id)); -#endif } /* We always need to create a local object. */ @@ -455,24 +498,20 @@ static int __tiered_open(WT_SESSION_IMPL *session, const char *cfg[]) { WT_CONFIG_ITEM cval, tierconf; - WT_DATA_HANDLE *dhandle, *file_dhandle; + WT_DATA_HANDLE *dhandle; WT_DECL_ITEM(tmp); WT_DECL_RET; WT_TIERED *tiered; -#if 1 WT_TIERED_WORK_UNIT *entry; uint32_t unused; -#endif char *metaconf; - const char *newconfig; const char *obj_cfg[] = {WT_CONFIG_BASE(session, object_meta), NULL, NULL}; - const char *new_tiered_cfg[] = {NULL, NULL, NULL, NULL}; const char **tiered_cfg, *config; dhandle = session->dhandle; tiered = (WT_TIERED *)dhandle; tiered_cfg = dhandle->cfg; - config = newconfig = NULL; + config = NULL; metaconf = NULL; WT_RET(__wt_scr_alloc(session, 0, &tmp)); @@ -519,27 +558,6 @@ __tiered_open(WT_SESSION_IMPL *session, const char *cfg[]) __wt_verbose( session, WT_VERB_TIERED, "TIERED_OPEN: create %s config %s", dhandle->name, config); WT_ERR(__wt_tiered_switch(session, config)); - file_dhandle = tiered->tiers[WT_TIERED_INDEX_LOCAL].tier; - WT_ASSERT(session, file_dhandle != dhandle && file_dhandle->type == WT_DHANDLE_TYPE_BTREE); - - /* - * XXX brute force, need to figure out functions to use to do this properly. - * - * We are updating the tiered dhandle config entry to reflect the new tiers metadata. The - * tiered dhandle must look almost exactly like the local file dhandle. The difference is - * that the local file dhandle is marked as readonly and also tagged as a tiered object. - * We'll turn those off before putting it into tiered dhandle. - */ - WT_ERR(__wt_metadata_search(session, file_dhandle->name, &metaconf)); - __wt_verbose(session, WT_VERB_TIERED, "TIERED_OPEN: after switch meta conf %s %s", - dhandle->name, metaconf); - new_tiered_cfg[0] = metaconf; - new_tiered_cfg[1] = "tiered_object=false,readonly=false"; - WT_ERR(__wt_config_merge(session, new_tiered_cfg, NULL, &newconfig)); - __wt_free(session, dhandle->cfg[1]); - dhandle->cfg[1] = newconfig; - WT_ERR(__wt_config_merge(session, dhandle->cfg, NULL, &newconfig)); - WT_ERR(__wt_metadata_update(session, dhandle->name, newconfig)); } WT_ERR(__wt_btree_open(session, tiered_cfg)); WT_ERR(__wt_btree_switch_object(session, tiered->current_id, 0)); @@ -584,15 +602,21 @@ __wt_tiered_open(WT_SESSION_IMPL *session, const char *cfg[]) /* * __wt_tiered_close -- - * Close a tiered data handle. TODO: When this returns an actual meaningful return value, remove - * its entry from s_void. + * Close a tiered data handle. */ int -__wt_tiered_close(WT_SESSION_IMPL *session, WT_TIERED *tiered) +__wt_tiered_close(WT_SESSION_IMPL *session) +{ + return (__wt_btree_close(session)); +} + +/* + * __wt_tiered_discard -- + * Discard a tiered data handle. + */ +int +__wt_tiered_discard(WT_SESSION_IMPL *session, WT_TIERED *tiered) { -#if 0 - WT_DATA_HANDLE *dhandle; -#endif uint32_t i; __wt_free(session, tiered->key_format); @@ -605,19 +629,11 @@ __wt_tiered_close(WT_SESSION_IMPL *session, WT_TIERED *tiered) * close the other dhandles may be closed and freed before this dhandle. So just free the names. */ for (i = 0; i < WT_TIERED_MAX_TIERS; i++) { -#if 0 - dhandle = tiered->tiers[i].tier; - /* - * XXX We cannot decrement on connection close but we need to decrement on sweep close or - * other individual close. - */ - (void)__wt_atomic_subi32(&dhandle->session_inuse, 1); -#endif if (tiered->tiers[i].name != NULL) __wt_free(session, tiered->tiers[i].name); } - return (0); + return (__wt_btree_discard(session)); } /* diff --git a/src/third_party/wiredtiger/src/tiered/tiered_work.c b/src/third_party/wiredtiger/src/tiered/tiered_work.c index 8fa2634fcd2..b3fd1b7f9b5 100644 --- a/src/third_party/wiredtiger/src/tiered/tiered_work.c +++ b/src/third_party/wiredtiger/src/tiered/tiered_work.c @@ -28,7 +28,7 @@ __wt_tiered_work_free(WT_SESSION_IMPL *session, WT_TIERED_WORK_UNIT *entry) WT_STAT_CONN_INCR(session, flush_state_races); __wt_yield(); } - /* If all work is done signal any waiting thread waiting for sync. */ + /* If all work is done signal any waiting thread waiting for sync. */ if (WT_FLUSH_STATE_DONE(conn->flush_state)) __wt_cond_signal(session, conn->flush_cond); __wt_free(session, entry); diff --git a/src/third_party/wiredtiger/src/txn/txn.c b/src/third_party/wiredtiger/src/txn/txn.c index c73e8bcdb4c..9876159c4be 100644 --- a/src/third_party/wiredtiger/src/txn/txn.c +++ b/src/third_party/wiredtiger/src/txn/txn.c @@ -890,6 +890,15 @@ __txn_commit_timestamps_usage_check(WT_SESSION_IMPL *session, WT_TXN_OP *op, WT_ __wt_timestamp_to_string(op_ts, ts_string[0]), __wt_timestamp_to_string(prev_op_durable_ts, ts_string[1]))); + if (FLD_ISSET(ts_flags, WT_DHANDLE_TS_ORDERED) && prev_op_durable_ts != WT_TS_NONE && + !txn_has_ts) + WT_RET( + __wt_msg(session, + WT_COMMIT_TS_VERB_PREFIX "committing a transaction that updates a value without " + "a timestamp while the previous update (%s) is timestamped " + "on a table configured for strict ordering", + __wt_timestamp_to_string(prev_op_durable_ts, ts_string[1]))); + if (FLD_ISSET(ts_flags, WT_DHANDLE_TS_MIXED_MODE) && F_ISSET(txn, WT_TXN_HAS_TS_COMMIT) && op_ts != WT_TS_NONE && prev_op_durable_ts > op_ts) WT_RET(__wt_msg(session, @@ -965,9 +974,8 @@ __txn_fixup_prepared_update( hs_cursor->set_value(hs_cursor, &tw, tw.durable_stop_ts, tw.durable_start_ts, (uint64_t)WT_UPDATE_STANDARD, &hs_value); WT_ERR(hs_cursor->update(hs_cursor)); - } else { + } else WT_ERR(hs_cursor->remove(hs_cursor)); - } err: F_SET(txn, txn_flags); @@ -1129,9 +1137,15 @@ __txn_resolve_prepared_op(WT_SESSION_IMPL *session, WT_TXN_OP *op, bool commit, * and instead write nothing. */ WT_ERR(__wt_upd_alloc_tombstone(session, &tombstone, ¬_used)); +#ifdef HAVE_DIAGNOSTIC + WT_WITH_BTREE(session, op->btree, + ret = __wt_row_modify( + cbt, &cbt->iface.key, NULL, tombstone, WT_UPDATE_INVALID, false, false)); +#else WT_WITH_BTREE(session, op->btree, ret = __wt_row_modify(cbt, &cbt->iface.key, NULL, tombstone, WT_UPDATE_INVALID, false)); +#endif WT_ERR(ret); tombstone = NULL; } else if (ret == 0) @@ -1263,9 +1277,12 @@ __txn_commit_timestamps_assert(WT_SESSION_IMPL *session) used_ts = F_ISSET(txn, WT_TXN_HAS_TS_COMMIT) || F_ISSET(txn, WT_TXN_HAS_TS_DURABLE); /* - * Debugging checks on timestamps, if user requested them. + * Debugging checks on timestamps, if user requested them. We additionally don't expect recovery + * to be using timestamps when applying commits. If recovery is running, skip this assert to + * avoid failing the recovery process. */ - if (F_ISSET(txn, WT_TXN_TS_WRITE_ALWAYS) && !used_ts && txn->mod_count != 0) + if (F_ISSET(txn, WT_TXN_TS_WRITE_ALWAYS) && !used_ts && txn->mod_count != 0 && + !F_ISSET(S2C(session), WT_CONN_RECOVERING)) WT_RET_MSG(session, EINVAL, "commit_timestamp required and none set on this transaction"); if (F_ISSET(txn, WT_TXN_TS_WRITE_NEVER) && used_ts && txn->mod_count != 0) WT_RET_MSG( @@ -1431,15 +1448,16 @@ __wt_txn_commit(WT_SESSION_IMPL *session, const char *cfg[]) WT_UPDATE *upd; wt_timestamp_t candidate_durable_timestamp, prev_durable_timestamp; uint32_t fileid; - u_int i; + uint8_t previous_state; + u_int i, ft_resolution; #ifdef HAVE_DIAGNOSTIC u_int prepare_count; #endif bool locked, prepare, readonly, update_durable_ts; - txn = session->txn; conn = S2C(session); cursor = NULL; + txn = session->txn; txn_global = &conn->txn_global; #ifdef HAVE_DIAGNOSTIC prepare_count = 0; @@ -1564,6 +1582,7 @@ __wt_txn_commit(WT_SESSION_IMPL *session, const char *cfg[]) /* Note: we're going to commit: nothing can fail after this point. */ /* Process and free updates. */ + ft_resolution = 0; for (i = 0, op = txn->mod; i < txn->mod_count; i++, op++) { fileid = op->btree->id; switch (op->type) { @@ -1607,8 +1626,12 @@ __wt_txn_commit(WT_SESSION_IMPL *session, const char *cfg[]) } break; case WT_TXN_OP_REF_DELETE: - __wt_txn_op_set_timestamp(session, op); - break; + /* + * Fast-truncate operations are resolved in a second pass after failure is no longer + * possible. + */ + ++ft_resolution; + continue; case WT_TXN_OP_TRUNCATE_COL: case WT_TXN_OP_TRUNCATE_ROW: /* Other operations don't need timestamps. */ @@ -1620,11 +1643,6 @@ __wt_txn_commit(WT_SESSION_IMPL *session, const char *cfg[]) if (cursor != NULL) WT_CLEAR(cursor->key); } - txn->mod_count = 0; -#ifdef HAVE_DIAGNOSTIC - WT_ASSERT(session, txn->prepare_count == prepare_count); - txn->prepare_count = 0; -#endif if (cursor != NULL) { WT_ERR(cursor->close(cursor)); @@ -1632,6 +1650,36 @@ __wt_txn_commit(WT_SESSION_IMPL *session, const char *cfg[]) } /* + * Resolve any fast-truncate transactions and allow eviction to proceed on instantiated pages. + * This isn't done as part of the initial processing because until now the commit could still + * switch to an abort. The action allowing eviction to proceed is clearing the WT_UPDATE list, + * (if any), associated with the commit. We're the only consumer of that list and we no longer + * need it, and eviction knows it means abort or commit has completed on instantiated pages. + */ + for (i = 0, op = txn->mod; ft_resolution > 0 && i < txn->mod_count; i++, op++) + if (op->type == WT_TXN_OP_REF_DELETE) { + __wt_txn_op_set_timestamp(session, op); + + WT_REF_LOCK(session, op->u.ref, &previous_state); + if (previous_state == WT_REF_DELETED) + op->u.ref->ft_info.del->committed = 1; + else + __wt_free(session, op->u.ref->ft_info.update); + WT_REF_UNLOCK(op->u.ref, previous_state); + + __wt_txn_op_free(session, op); + + --ft_resolution; + } + WT_ASSERT(session, ft_resolution == 0); + + txn->mod_count = 0; +#ifdef HAVE_DIAGNOSTIC + WT_ASSERT(session, txn->prepare_count == prepare_count); + txn->prepare_count = 0; +#endif + + /* * If durable is set, we'll try to update the global durable timestamp with that value. If * durable isn't set, durable is implied to be the same as commit so we'll use that instead. */ @@ -1680,6 +1728,20 @@ __wt_txn_commit(WT_SESSION_IMPL *session, const char *cfg[]) */ if (!readonly) WT_IGNORE_RET(__wt_cache_eviction_check(session, false, false, NULL)); + + /* + * Stable timestamp cannot be concurrently increased greater than or equal to the prepared + * transaction's durable timestamp. Otherwise, checkpoint may only write partial updates of the + * transaction. + */ + if (prepare && txn->durable_timestamp <= txn_global->stable_timestamp) { + WT_ERR(__wt_verbose_dump_sessions(session, true)); + WT_ERR_PANIC(session, WT_PANIC, + "Stable timestamp is increased greater than or equal to the committing prepared " + "transaction's " + "durable timestamp"); + } + return (0); err: @@ -1733,7 +1795,7 @@ __wt_txn_prepare(WT_SESSION_IMPL *session, const char *cfg[]) "A transaction should not have been assigned a log record if WT_CONN_LOG_DEBUG mode is " "not enabled"); - /* Set the prepare timestamp. */ + /* Set the prepare timestamp. */ WT_RET(__wt_txn_set_timestamp(session, cfg)); if (!F_ISSET(txn, WT_TXN_HAS_TS_PREPARE)) diff --git a/src/third_party/wiredtiger/src/txn/txn_ckpt.c b/src/third_party/wiredtiger/src/txn/txn_ckpt.c index a8c5c8664a4..d6b683dd4f2 100644 --- a/src/third_party/wiredtiger/src/txn/txn_ckpt.c +++ b/src/third_party/wiredtiger/src/txn/txn_ckpt.c @@ -757,6 +757,7 @@ __txn_checkpoint(WT_SESSION_IMPL *session, const char *cfg[]) WT_TXN *txn; WT_TXN_GLOBAL *txn_global; WT_TXN_ISOLATION saved_isolation; + wt_off_t hs_size; wt_timestamp_t ckpt_tmp_ts; uint64_t fsync_duration_usecs, generation, hs_ckpt_duration_usecs; uint64_t time_start_fsync, time_stop_fsync, time_start_hs, time_stop_hs; @@ -766,6 +767,7 @@ __txn_checkpoint(WT_SESSION_IMPL *session, const char *cfg[]) conn = S2C(session); cache = conn->cache; + hs_size = 0; hs_dhandle = NULL; txn = session->txn; txn_global = &conn->txn_global; @@ -956,6 +958,12 @@ __txn_checkpoint(WT_SESSION_IMPL *session, const char *cfg[]) __checkpoint_verbose_track(session, "sync completed"); + /* If the history store file exists on disk, update its statistic. */ + if (F_ISSET(hs_dhandle, WT_DHANDLE_OPEN)) { + WT_ERR(__wt_block_manager_named_size(session, WT_HS_FILE, &hs_size)); + WT_STAT_CONN_SET(session, cache_hs_ondisk, hs_size); + } + /* * Commit the transaction now that we are sure that all files in the checkpoint have been * flushed to disk. It's OK to commit before checkpointing the metadata since we know that all @@ -1310,8 +1318,18 @@ __checkpoint_lock_dirty_tree_int(WT_SESSION_IMPL *session, bool is_checkpoint, b * checkpoint. */ WT_RET(__checkpoint_mark_skip(session, ckptbase, force)); - if (F_ISSET(btree, WT_BTREE_SKIP_CKPT)) + if (F_ISSET(btree, WT_BTREE_SKIP_CKPT)) { + /* + * If we decide to skip checkpointing, clear the delete flag on the checkpoints. The list of + * checkpoints will be cached for a future access. Which checkpoints need to be deleted can + * change in the meanwhile. + */ + WT_CKPT_FOREACH (ckptbase, ckpt) + if (F_ISSET(ckpt, WT_CKPT_DELETE)) + F_CLR(ckpt, WT_CKPT_DELETE); return (0); + } + /* * Lock the checkpoints that will be deleted. * @@ -1374,9 +1392,10 @@ __checkpoint_lock_dirty_tree( btree = S2BT(session); ckpt = ckptbase = NULL; - dhandle = session->dhandle; ckpt_bytes_allocated = 0; + dhandle = session->dhandle; name_alloc = NULL; + seen_ckpt_add = false; /* * Only referenced in diagnostic builds and gcc 5.1 isn't satisfied with wrapping the entire @@ -1453,17 +1472,7 @@ __checkpoint_lock_dirty_tree( WT_BTREE_CLEAN_CKPT(session, btree, 0); F_CLR(btree, WT_BTREE_OBSOLETE_PAGES); - /* - * Get the list of checkpoints for this file: We try to cache the ckptlist between the - * checkpoints. But there might not be one, as there are operations that can invalidate a - * ckptlist. So, use a cached ckptlist if there is one. Otherwise go through slow path of - * re-generating the ckptlist by reading the metadata. Also, we avoid using a cached checkpoint - * list for metadata. - */ - if (WT_IS_METADATA(dhandle) || - __wt_meta_saved_ckptlist_get(session, dhandle->name, &ckptbase) != 0) - WT_ERR( - __wt_meta_ckptlist_get(session, dhandle->name, true, &ckptbase, &ckpt_bytes_allocated)); + WT_ERR(__wt_meta_ckptlist_get(session, dhandle->name, true, &ckptbase, &ckpt_bytes_allocated)); /* We may be dropping specific checkpoints, check the configuration. */ if (cfg != NULL) { @@ -1512,7 +1521,6 @@ __checkpoint_lock_dirty_tree( * If we decided to skip checkpointing, we need to remove the new checkpoint entry we might have * appended to the list. */ - seen_ckpt_add = false; if (F_ISSET(btree, WT_BTREE_SKIP_CKPT)) { WT_CKPT_FOREACH_NAME_OR_ORDER (ckptbase, ckpt) { /* Checkpoint(s) to be added are always at the end of the list. */ @@ -1531,7 +1539,8 @@ __checkpoint_lock_dirty_tree( /* It is possible that we do not have any checkpoint in the list. */ err: __wt_meta_ckptlist_free(session, &ckptbase); - __wt_meta_saved_ckptlist_free(session); + btree->ckpt = NULL; + btree->ckpt_bytes_allocated = 0; } skip: __wt_free(session, name_alloc); @@ -1846,7 +1855,7 @@ fake: if (WT_IS_METADATA(dhandle) || !F_ISSET(session->txn, WT_TXN_RUNNING)) WT_ERR(__wt_checkpoint_sync(session, NULL)); - WT_ERR(__wt_meta_ckptlist_set(session, dhandle->name, btree->ckpt, &ckptlsn)); + WT_ERR(__wt_meta_ckptlist_set(session, dhandle, btree->ckpt, &ckptlsn)); /* * If we wrote a checkpoint (rather than faking one), we have to resolve it. Normally, tracking diff --git a/src/third_party/wiredtiger/src/txn/txn_recover.c b/src/third_party/wiredtiger/src/txn/txn_recover.c index debee377638..9f32e9346f0 100644 --- a/src/third_party/wiredtiger/src/txn/txn_recover.c +++ b/src/third_party/wiredtiger/src/txn/txn_recover.c @@ -563,7 +563,7 @@ __recovery_correct_write_gen(WT_SESSION_IMPL *session) while ((ret = cursor->next(cursor)) == 0) { WT_ERR(cursor->get_key(cursor, &uri)); - if (!WT_PREFIX_MATCH(uri, "file:")) + if (!WT_PREFIX_MATCH(uri, "file:") && !WT_PREFIX_MATCH(uri, "tiered:")) continue; WT_ERR(cursor->get_value(cursor, &config)); @@ -640,11 +640,11 @@ __recovery_setup_file(WT_RECOVERY *r, const char *uri, const char *config) } /* - * __recovery_free -- - * Free the recovery state. + * __recovery_close_cursors -- + * Close the logging recovery cursors. */ static int -__recovery_free(WT_RECOVERY *r) +__recovery_close_cursors(WT_RECOVERY *r) { WT_CURSOR *c; WT_DECL_RET; @@ -658,40 +658,60 @@ __recovery_free(WT_RECOVERY *r) WT_TRET(c->close(c)); } + r->nfiles = 0; __wt_free(session, r->files); return (ret); } /* - * __recovery_file_scan -- - * Scan the files referenced from the metadata and gather information about them for recovery. + * __recovery_file_scan_prefix -- + * Scan the files matching the prefix referenced from the metadata and gather information about + * them for recovery. */ static int -__recovery_file_scan(WT_RECOVERY *r) +__recovery_file_scan_prefix(WT_RECOVERY *r, const char *prefix, const char *ignore_suffix) { WT_CURSOR *c; WT_DECL_RET; int cmp; const char *uri, *config; - /* Scan through all files in the metadata. */ + /* Scan through all entries in the metadata matching the prefix. */ c = r->files[0].c; - c->set_key(c, "file:"); + c->set_key(c, prefix); if ((ret = c->search_near(c, &cmp)) != 0) { /* Is the metadata empty? */ WT_RET_NOTFOUND_OK(ret); return (0); } - if (cmp < 0) - WT_RET_NOTFOUND_OK(c->next(c)); + if (cmp < 0 && (ret = c->next(c)) != 0) { + /* No matching entries? */ + WT_RET_NOTFOUND_OK(ret); + return (0); + } for (; ret == 0; ret = c->next(c)) { WT_RET(c->get_key(c, &uri)); - if (!WT_PREFIX_MATCH(uri, "file:")) + if (!WT_PREFIX_MATCH(uri, prefix)) break; + if (ignore_suffix != NULL && WT_SUFFIX_MATCH(uri, ignore_suffix)) + continue; WT_RET(c->get_value(c, &config)); WT_RET(__recovery_setup_file(r, uri, config)); } WT_RET_NOTFOUND_OK(ret); + return (0); +} + +/* + * __recovery_file_scan -- + * Scan the files referenced from the metadata and gather information about them for recovery. + */ +static int +__recovery_file_scan(WT_RECOVERY *r) +{ + /* Scan through all files and tiered entries in the metadata. */ + WT_RET(__recovery_file_scan_prefix(r, "file:", ".wtobj")); + WT_RET(__recovery_file_scan_prefix(r, "tiered:", NULL)); /* * Set the connection level file id tracker, as such upon creation of a new file we'll begin @@ -967,6 +987,9 @@ __wt_txn_recover(WT_SESSION_IMPL *session, const char *cfg[]) WT_ERR(ret); done: + /* Close cached cursors, rollback-to-stable asserts exclusive access. */ + WT_ERR(__recovery_close_cursors(&r)); + WT_ERR(__recovery_set_checkpoint_timestamp(&r)); WT_ERR(__recovery_set_oldest_timestamp(&r)); WT_ERR(__recovery_set_checkpoint_snapshot(session)); @@ -1054,7 +1077,7 @@ done: FLD_SET(conn->log_flags, WT_CONN_LOG_RECOVER_DONE); err: - WT_TRET(__recovery_free(&r)); + WT_TRET(__recovery_close_cursors(&r)); __wt_free(session, config); FLD_CLR(conn->log_flags, WT_CONN_LOG_RECOVER_DIRTY); diff --git a/src/third_party/wiredtiger/src/txn/txn_rollback_to_stable.c b/src/third_party/wiredtiger/src/txn/txn_rollback_to_stable.c index 500d8cb6058..0a6b7e52cf9 100644 --- a/src/third_party/wiredtiger/src/txn/txn_rollback_to_stable.c +++ b/src/third_party/wiredtiger/src/txn/txn_rollback_to_stable.c @@ -8,45 +8,87 @@ #include "wt_internal.h" -#define WT_CHECK_RECOVERY_FLAG_TS_TXNID(session, txnid, durablets) \ - (durablets == WT_TS_NONE && F_ISSET(S2C(session), WT_CONN_RECOVERING) && \ - (txnid) >= S2C(session)->recovery_ckpt_snap_min) +#define WT_CHECK_RECOVERY_FLAG_TXNID(session, txnid) \ + (F_ISSET(S2C(session), WT_CONN_RECOVERING) && (txnid) >= S2C(session)->recovery_ckpt_snap_min) /* Enable rollback to stable verbose messaging during recovery. */ #define WT_VERB_RECOVERY_RTS(session) \ (F_ISSET(S2C(session), WT_CONN_RECOVERING) ? WT_VERB_RECOVERY | WT_VERB_RTS : WT_VERB_RTS) /* + * __rollback_delete_hs -- + * Delete the updates for a key in the history store until the first update (including) that is + * larger than or equal to the specified timestamp. + */ +static int +__rollback_delete_hs(WT_SESSION_IMPL *session, WT_ITEM *key, wt_timestamp_t ts) +{ + WT_CURSOR *hs_cursor; + WT_DECL_ITEM(hs_key); + WT_DECL_RET; + wt_timestamp_t hs_start_ts; + uint64_t hs_counter; + uint32_t hs_btree_id; + + /* Open a history store table cursor. */ + WT_RET(__wt_curhs_open(session, NULL, &hs_cursor)); + /* + * Rollback-to-stable operates exclusively (i.e., it is the only active operation in the system) + * outside the constraints of transactions. Therefore, there is no need for snapshot based + * visibility checks. + */ + F_SET(hs_cursor, WT_CURSTD_HS_READ_COMMITTED); + + WT_ERR(__wt_scr_alloc(session, 0, &hs_key)); + + /* + * Scan the history store for the given btree and key with maximum start timestamp to let the + * search point to the last version of the key and start traversing backwards to delete all the + * records until the first update with the start timestamp larger than or equal to the specified + * timestamp. + */ + hs_cursor->set_key(hs_cursor, 4, S2BT(session)->id, key, WT_TS_MAX, UINT64_MAX); + ret = __wt_curhs_search_near_before(session, hs_cursor); + for (; ret == 0; ret = hs_cursor->prev(hs_cursor)) { + WT_ERR(hs_cursor->get_key(hs_cursor, &hs_btree_id, hs_key, &hs_start_ts, &hs_counter)); + if (hs_start_ts < ts) + break; + WT_ERR(hs_cursor->remove(hs_cursor)); + WT_STAT_CONN_DATA_INCR(session, txn_rts_hs_removed); + if (hs_start_ts == ts) + WT_STAT_CONN_DATA_INCR(session, cache_hs_key_truncate_rts); + else + WT_STAT_CONN_DATA_INCR(session, cache_hs_key_truncate_rts_unstable); + } + WT_ERR_NOTFOUND_OK(ret, false); + +err: + __wt_scr_free(session, &hs_key); + WT_TRET(hs_cursor->close(hs_cursor)); + return (ret); +} + +/* * __rollback_abort_update -- * Abort updates in an update change with timestamps newer than the rollback timestamp. Also, * clear the history store flag for the first stable update in the update. */ -static void -__rollback_abort_update(WT_SESSION_IMPL *session, WT_UPDATE *first_upd, +static int +__rollback_abort_update(WT_SESSION_IMPL *session, WT_ITEM *key, WT_UPDATE *first_upd, wt_timestamp_t rollback_timestamp, bool *stable_update_found) { - WT_UPDATE *upd; + WT_UPDATE *stable_upd, *tombstone, *upd; char ts_string[2][WT_TS_INT_STRING_SIZE]; - *stable_update_found = false; + stable_upd = tombstone = NULL; + if (stable_update_found != NULL) + *stable_update_found = false; for (upd = first_upd; upd != NULL; upd = upd->next) { /* Skip the updates that are aborted. */ - if (upd->txnid == WT_TXN_ABORTED) { - if (upd == first_upd) - first_upd = upd->next; - } else if (rollback_timestamp < upd->durable_ts || - upd->prepare_state == WT_PREPARE_INPROGRESS) { - /* - * If any updates are aborted, all newer updates better be aborted as well. - * - * Timestamp ordering relies on the validations at the time of commit. Thus if the table - * is not configured for key consistency check, the timestamps could be out of order - * here. - */ - WT_ASSERT(session, - !F_ISSET(session->dhandle, WT_DHANDLE_TS_KEY_CONSISTENT) || upd == first_upd); - first_upd = upd->next; + if (upd->txnid == WT_TXN_ABORTED) + continue; + if (rollback_timestamp < upd->durable_ts || upd->prepare_state == WT_PREPARE_INPROGRESS) { __wt_verbose(session, WT_VERB_RECOVERY_RTS(session), "rollback to stable update aborted with txnid: %" PRIu64 " durable timestamp: %s and stable timestamp: %s, prepared: %s", @@ -59,7 +101,7 @@ __rollback_abort_update(WT_SESSION_IMPL *session, WT_UPDATE *first_upd, upd->durable_ts = upd->start_ts = WT_TS_NONE; } else { /* Valid update is found. */ - WT_ASSERT(session, first_upd == upd); + stable_upd = upd; break; } } @@ -70,40 +112,82 @@ __rollback_abort_update(WT_SESSION_IMPL *session, WT_UPDATE *first_upd, * history store. The next time when this update is moved into the history store, it will have a * different stop time point. */ - if (first_upd != NULL) { - F_CLR(first_upd, WT_UPDATE_HS); - *stable_update_found = true; + if (stable_upd != NULL) { + if (F_ISSET(stable_upd, WT_UPDATE_HS)) { + /* Find the update following a stable tombstone. */ + if (stable_upd->type == WT_UPDATE_TOMBSTONE) { + tombstone = stable_upd; + for (stable_upd = stable_upd->next; stable_upd != NULL; + stable_upd = stable_upd->next) { + if (stable_upd->txnid != WT_TXN_ABORTED) { + WT_ASSERT(session, + stable_upd->type != WT_UPDATE_TOMBSTONE && + F_ISSET(stable_upd, WT_UPDATE_HS)); + break; + } + } + } + + /* + * Delete the first stable update and any newer update from the history store. If the + * update following the stable tombstone is removed by obsolete check, no need to remove + * that update from the history store as it has a globally visible tombstone. In that + * case, it is enough to delete everything up until to the tombstone timestamp. + */ + WT_RET(__rollback_delete_hs( + session, key, stable_upd == NULL ? tombstone->start_ts : stable_upd->start_ts)); + + /* + * Clear the history store flag for the first stable update. Otherwise, it will not be + * moved to history store again. + */ + if (stable_upd != NULL) + F_CLR(stable_upd, WT_UPDATE_HS); + if (tombstone != NULL) + F_CLR(tombstone, WT_UPDATE_HS); + } + if (stable_update_found != NULL) + *stable_update_found = true; } -} -/* - * __rollback_abort_col_insert_list -- - * Apply the update abort check to each entry in an insert skip list. - */ -static void -__rollback_abort_col_insert_list(WT_SESSION_IMPL *session, WT_INSERT_HEAD *head, - wt_timestamp_t rollback_timestamp, bool *stable_update_found) -{ - WT_INSERT *ins; - WT_SKIP_FOREACH (ins, head) - if (ins->upd != NULL) - __rollback_abort_update(session, ins->upd, rollback_timestamp, stable_update_found); + return (0); } /* * __rollback_abort_insert_list -- * Apply the update abort check to each entry in an insert skip list. */ -static void -__rollback_abort_insert_list( - WT_SESSION_IMPL *session, WT_INSERT_HEAD *head, wt_timestamp_t rollback_timestamp) +static int +__rollback_abort_insert_list(WT_SESSION_IMPL *session, WT_PAGE *page, WT_INSERT_HEAD *head, + wt_timestamp_t rollback_timestamp, bool *stable_update_found) { + WT_DECL_ITEM(key); + WT_DECL_RET; WT_INSERT *ins; - bool stable_update_found; + uint64_t recno; + uint8_t *memp; + + WT_ERR( + __wt_scr_alloc(session, page->type == WT_PAGE_ROW_LEAF ? 0 : WT_INTPACK64_MAXSIZE, &key)); WT_SKIP_FOREACH (ins, head) - if (ins->upd != NULL) - __rollback_abort_update(session, ins->upd, rollback_timestamp, &stable_update_found); + if (ins->upd != NULL) { + if (page->type == WT_PAGE_ROW_LEAF) { + key->data = WT_INSERT_KEY(ins); + key->size = WT_INSERT_KEY_SIZE(ins); + } else { + recno = WT_INSERT_RECNO(ins); + memp = key->mem; + WT_ERR(__wt_vpack_uint(&memp, 0, recno)); + key->size = WT_PTRDIFF(memp, key->data); + } + WT_ERR(__rollback_abort_update( + session, key, ins->upd, rollback_timestamp, stable_update_found)); + } + +err: + __wt_scr_free(session, &key); + return (ret); } /* @@ -123,7 +207,11 @@ __rollback_col_modify(WT_SESSION_IMPL *session, WT_REF *ref, WT_UPDATE *upd, uin WT_ERR(__wt_col_search(&cbt, recno, ref, true, NULL)); /* Apply the modification. */ +#ifdef HAVE_DIAGNOSTIC + WT_ERR(__wt_col_modify(&cbt, recno, NULL, upd, WT_UPDATE_INVALID, true, false)); +#else WT_ERR(__wt_col_modify(&cbt, recno, NULL, upd, WT_UPDATE_INVALID, true)); +#endif err: /* Free any resources that may have been cached in the cursor. */ @@ -246,7 +334,7 @@ __rollback_check_if_txnid_non_committed(WT_SESSION_IMPL *session, uint64_t txnid */ static int __rollback_ondisk_fixup_key(WT_SESSION_IMPL *session, WT_REF *ref, WT_PAGE *page, WT_COL *cip, - WT_ROW *rip, wt_timestamp_t rollback_timestamp, bool replace, uint64_t recno) + WT_ROW *rip, wt_timestamp_t rollback_timestamp, uint64_t recno) { WT_CELL *kcell; WT_CELL_UNPACK_KV *unpack, _unpack; @@ -389,33 +477,15 @@ __rollback_ondisk_fixup_key(WT_SESSION_IMPL *session, WT_REF *ref, WT_PAGE *page if (hs_stop_durable_ts < newer_hs_durable_ts) WT_STAT_CONN_DATA_INCR(session, txn_rts_hs_stop_older_than_newer_start); - /* - * Stop processing when we find the newer version value of this key is stable according to - * the current version stop timestamp and transaction id when it is not appending the - * selected update to the update chain. Also it confirms that history store doesn't contains - * any newer version than the current version for the key. - */ /* Retrieve the time window from the history cursor. */ __wt_hs_upd_time_window(hs_cursor, &hs_tw); - if (!replace && - (hs_stop_durable_ts != WT_TS_NONE || - !__rollback_check_if_txnid_non_committed(session, hs_tw->stop_txn)) && - (hs_stop_durable_ts <= rollback_timestamp)) { - __wt_verbose(session, WT_VERB_RECOVERY_RTS(session), - "history store update valid with stop timestamp: %s, stable timestamp: %s, txnid: " - "%" PRIu64 " and type: %" PRIu8, - __wt_timestamp_to_string(hs_stop_durable_ts, ts_string[0]), - __wt_timestamp_to_string(rollback_timestamp, ts_string[1]), hs_tw->stop_txn, type); - break; - } /* * Stop processing when we find a stable update according to the given timestamp and * transaction id. */ - if ((hs_durable_ts != WT_TS_NONE || - !__rollback_check_if_txnid_non_committed(session, hs_tw->start_txn)) && - (hs_durable_ts <= rollback_timestamp)) { + if (!__rollback_check_if_txnid_non_committed(session, hs_tw->start_txn) && + hs_durable_ts <= rollback_timestamp) { __wt_verbose(session, WT_VERB_RECOVERY_RTS(session), "history store update valid with start timestamp: %s, durable timestamp: %s, stop " "timestamp: %s, stable timestamp: %s, txnid: %" PRIu64 " and type: %" PRIu8, @@ -453,18 +523,56 @@ __rollback_ondisk_fixup_key(WT_SESSION_IMPL *session, WT_REF *ref, WT_PAGE *page WT_STAT_CONN_DATA_INCR(session, cache_hs_key_truncate_rts_unstable); } - if (replace) { + /* + * If we found a history value that satisfied the given timestamp, add it to the update list. + * Otherwise remove the key by adding a tombstone. + */ + if (valid_update_found) { + /* Retrieve the time window from the history cursor. */ + __wt_hs_upd_time_window(hs_cursor, &hs_tw); + WT_ASSERT(session, + hs_tw->start_ts < unpack->tw.start_ts || hs_tw->start_txn < unpack->tw.start_txn); + WT_ERR(__wt_upd_alloc(session, &full_value, WT_UPDATE_STANDARD, &upd, NULL)); + + /* + * Set the transaction id of updates to WT_TXN_NONE when called from recovery, because the + * connections write generation will be initialized after rollback to stable and the updates + * in the cache will be problematic. The transaction id of pages which are in disk will be + * automatically reset as part of unpacking cell when loaded to cache. + */ + if (F_ISSET(S2C(session), WT_CONN_RECOVERING)) + upd->txnid = WT_TXN_NONE; + else + upd->txnid = hs_tw->start_txn; + upd->durable_ts = hs_tw->durable_start_ts; + upd->start_ts = hs_tw->start_ts; + __wt_verbose(session, WT_VERB_RECOVERY_RTS(session), + "update restored from history store txnid: %" PRIu64 ", start_ts: %s and durable_ts: %s", + upd->txnid, __wt_timestamp_to_string(upd->start_ts, ts_string[0]), + __wt_timestamp_to_string(upd->durable_ts, ts_string[1])); + + /* + * Set the flag to indicate that this update has been restored from history store for the + * rollback to stable operation. + */ + F_SET(upd, WT_UPDATE_RESTORED_FROM_HS); + WT_STAT_CONN_DATA_INCR(session, txn_rts_hs_restore_updates); + /* - * If we found a history value that satisfied the given timestamp, add it to the update - * list. Otherwise remove the key by adding a tombstone. + * We have a tombstone on the original update chain and it is stable according to the + * timestamp and txnid, we need to restore that as well. */ - if (valid_update_found) { - /* Retrieve the time window from the history cursor. */ - __wt_hs_upd_time_window(hs_cursor, &hs_tw); + if (!__rollback_check_if_txnid_non_committed(session, hs_tw->stop_txn) && + hs_stop_durable_ts <= rollback_timestamp) { + /* + * The restoring tombstone timestamp must be zero or less than previous update start + * timestamp or the on-disk update is an out of order prepared. + */ WT_ASSERT(session, - hs_tw->start_ts < unpack->tw.start_ts || hs_tw->start_txn < unpack->tw.start_txn); - WT_ERR(__wt_upd_alloc(session, &full_value, WT_UPDATE_STANDARD, &upd, NULL)); + hs_stop_durable_ts == WT_TS_NONE || hs_stop_durable_ts < newer_hs_durable_ts || + unpack->tw.prepare); + WT_ERR(__wt_upd_alloc_tombstone(session, &tombstone, NULL)); /* * Set the transaction id of updates to WT_TXN_NONE when called from recovery, because * the connections write generation will be initialized after rollback to stable and the @@ -472,80 +580,38 @@ __rollback_ondisk_fixup_key(WT_SESSION_IMPL *session, WT_REF *ref, WT_PAGE *page * disk will be automatically reset as part of unpacking cell when loaded to cache. */ if (F_ISSET(S2C(session), WT_CONN_RECOVERING)) - upd->txnid = WT_TXN_NONE; + tombstone->txnid = WT_TXN_NONE; else - upd->txnid = hs_tw->start_txn; - upd->durable_ts = hs_tw->durable_start_ts; - upd->start_ts = hs_tw->start_ts; + tombstone->txnid = hs_tw->stop_txn; + tombstone->durable_ts = hs_tw->durable_stop_ts; + tombstone->start_ts = hs_tw->stop_ts; __wt_verbose(session, WT_VERB_RECOVERY_RTS(session), - "update restored from history store txnid: %" PRIu64 - ", start_ts: %s and durable_ts: %s", - upd->txnid, __wt_timestamp_to_string(upd->start_ts, ts_string[0]), - __wt_timestamp_to_string(upd->durable_ts, ts_string[1])); + "tombstone restored from history store txnid: %" PRIu64 + ", start_ts: %s, durable_ts: %s", + tombstone->txnid, __wt_timestamp_to_string(tombstone->start_ts, ts_string[0]), + __wt_timestamp_to_string(tombstone->durable_ts, ts_string[1])); /* * Set the flag to indicate that this update has been restored from history store for * the rollback to stable operation. */ - F_SET(upd, WT_UPDATE_RESTORED_FROM_HS); - WT_STAT_CONN_DATA_INCR(session, txn_rts_hs_restore_updates); - - /* - * We have a tombstone on the original update chain and it is stable according to the - * timestamp and txnid, we need to restore that as well. - */ - if (!__rollback_check_if_txnid_non_committed(session, hs_tw->stop_txn) && - hs_stop_durable_ts <= rollback_timestamp) { - /* - * The restoring tombstone timestamp must be zero or less than previous update start - * timestamp or the on-disk update is an out of order prepared. - */ - WT_ASSERT(session, - hs_stop_durable_ts == WT_TS_NONE || hs_stop_durable_ts < newer_hs_durable_ts || - unpack->tw.prepare); - - WT_ERR(__wt_upd_alloc_tombstone(session, &tombstone, NULL)); - /* - * Set the transaction id of updates to WT_TXN_NONE when called from recovery, - * because the connections write generation will be initialized after rollback to - * stable and the updates in the cache will be problematic. The transaction id of - * pages which are in disk will be automatically reset as part of unpacking cell - * when loaded to cache. - */ - if (F_ISSET(S2C(session), WT_CONN_RECOVERING)) - tombstone->txnid = WT_TXN_NONE; - else - tombstone->txnid = hs_tw->stop_txn; - tombstone->durable_ts = hs_tw->durable_stop_ts; - tombstone->start_ts = hs_tw->stop_ts; - __wt_verbose(session, WT_VERB_RECOVERY_RTS(session), - "tombstone restored from history store txnid: %" PRIu64 - ", start_ts: %s, durable_ts: %s", - tombstone->txnid, __wt_timestamp_to_string(tombstone->start_ts, ts_string[0]), - __wt_timestamp_to_string(tombstone->durable_ts, ts_string[1])); - - /* - * Set the flag to indicate that this update has been restored from history store - * for the rollback to stable operation. - */ - F_SET(tombstone, WT_UPDATE_RESTORED_FROM_HS); + F_SET(tombstone, WT_UPDATE_RESTORED_FROM_HS); - tombstone->next = upd; - upd = tombstone; - WT_STAT_CONN_DATA_INCR(session, txn_rts_hs_restore_tombstones); - } - } else { - WT_ERR(__wt_upd_alloc_tombstone(session, &upd, NULL)); - WT_STAT_CONN_DATA_INCR(session, txn_rts_keys_removed); - __wt_verbose(session, WT_VERB_RECOVERY_RTS(session), "%p: key removed", (void *)key); + tombstone->next = upd; + upd = tombstone; + WT_STAT_CONN_DATA_INCR(session, txn_rts_hs_restore_tombstones); } - - if (rip != NULL) - WT_ERR(__rollback_row_modify(session, page, rip, upd)); - else - WT_ERR(__rollback_col_modify(session, ref, upd, recno)); + } else { + WT_ERR(__wt_upd_alloc_tombstone(session, &upd, NULL)); + WT_STAT_CONN_DATA_INCR(session, txn_rts_keys_removed); + __wt_verbose(session, WT_VERB_RECOVERY_RTS(session), "%p: key removed", (void *)key); } + if (rip != NULL) + WT_ERR(__rollback_row_modify(session, page, rip, upd)); + else + WT_ERR(__rollback_col_modify(session, ref, upd, recno)); + /* Finally remove that update from history store. */ if (valid_update_found) { WT_ERR(hs_cursor->remove(hs_cursor)); @@ -622,9 +688,8 @@ __rollback_abort_ondisk_kv(WT_SESSION_IMPL *session, WT_REF *ref, WT_COL *cip, W WT_STAT_CONN_DATA_INCR(session, txn_rts_sweep_hs_keys); } else return (0); - } else if (((vpack->tw.durable_start_ts > rollback_timestamp) || - (vpack->tw.durable_start_ts == WT_TS_NONE && - __rollback_check_if_txnid_non_committed(session, vpack->tw.start_txn))) || + } else if (vpack->tw.durable_start_ts > rollback_timestamp || + __rollback_check_if_txnid_non_committed(session, vpack->tw.start_txn) || (!WT_TIME_WINDOW_HAS_STOP(&vpack->tw) && prepared)) { __wt_verbose(session, WT_VERB_RECOVERY_RTS(session), "on-disk update aborted with start durable timestamp: %s, commit timestamp: %s, " @@ -633,8 +698,8 @@ __rollback_abort_ondisk_kv(WT_SESSION_IMPL *session, WT_REF *ref, WT_COL *cip, W __wt_timestamp_to_string(vpack->tw.start_ts, ts_string[1]), prepared ? "true" : "false", __wt_timestamp_to_string(rollback_timestamp, ts_string[2]), vpack->tw.start_txn); if (!F_ISSET(S2C(session), WT_CONN_IN_MEMORY)) - return (__rollback_ondisk_fixup_key( - session, ref, NULL, cip, rip, rollback_timestamp, true, recno)); + return ( + __rollback_ondisk_fixup_key(session, ref, NULL, cip, rip, rollback_timestamp, recno)); else { /* * In-memory database don't have a history store to provide a stable update, so remove @@ -644,10 +709,8 @@ __rollback_abort_ondisk_kv(WT_SESSION_IMPL *session, WT_REF *ref, WT_COL *cip, W WT_STAT_CONN_DATA_INCR(session, txn_rts_keys_removed); } } else if (WT_TIME_WINDOW_HAS_STOP(&vpack->tw) && - (((vpack->tw.durable_stop_ts > rollback_timestamp) || - (vpack->tw.durable_stop_ts == WT_TS_NONE && - __rollback_check_if_txnid_non_committed(session, vpack->tw.stop_txn))) || - prepared)) { + (vpack->tw.durable_stop_ts > rollback_timestamp || + __rollback_check_if_txnid_non_committed(session, vpack->tw.stop_txn) || prepared)) { /* * Clear the remove operation from the key by inserting the original on-disk value as a * standard update. @@ -665,7 +728,7 @@ __rollback_abort_ondisk_kv(WT_SESSION_IMPL *session, WT_REF *ref, WT_COL *cip, W WT_ASSERT(session, prepared == true); if (!F_ISSET(S2C(session), WT_CONN_IN_MEMORY)) return (__rollback_ondisk_fixup_key( - session, ref, NULL, cip, rip, rollback_timestamp, true, recno)); + session, ref, NULL, cip, rip, rollback_timestamp, recno)); else { /* * In-memory database don't have a history store to provide a stable update, so @@ -750,8 +813,8 @@ __rollback_abort_col_var(WT_SESSION_IMPL *session, WT_REF *ref, wt_timestamp_t r stable_update_found = false; if ((ins = WT_COL_UPDATE(page, cip)) != NULL) - __rollback_abort_col_insert_list( - session, ins, rollback_timestamp, &stable_update_found); + WT_RET(__rollback_abort_insert_list( + session, page, ins, rollback_timestamp, &stable_update_found)); if (!stable_update_found && page->dsk != NULL) { kcell = WT_COL_PTR(page, cip); @@ -768,7 +831,7 @@ __rollback_abort_col_var(WT_SESSION_IMPL *session, WT_REF *ref, wt_timestamp_t r /* Review the append list */ if ((ins = WT_COL_APPEND(page)) != NULL) - __rollback_abort_insert_list(session, ins, rollback_timestamp); + WT_RET(__rollback_abort_insert_list(session, page, ins, rollback_timestamp, NULL)); /* Mark the page as dirty to reconcile the page. */ if (page->modify) @@ -781,135 +844,22 @@ __rollback_abort_col_var(WT_SESSION_IMPL *session, WT_REF *ref, wt_timestamp_t r * Abort updates on a fixed length col leaf page with timestamps newer than the rollback * timestamp. */ -static void +static int __rollback_abort_col_fix(WT_SESSION_IMPL *session, WT_PAGE *page, wt_timestamp_t rollback_timestamp) { WT_INSERT_HEAD *ins; /* Review the changes to the original on-page data items */ if ((ins = WT_COL_UPDATE_SINGLE(page)) != NULL) - __rollback_abort_insert_list(session, ins, rollback_timestamp); + WT_RET(__rollback_abort_insert_list(session, page, ins, rollback_timestamp, NULL)); /* Review the append list */ if ((ins = WT_COL_APPEND(page)) != NULL) - __rollback_abort_insert_list(session, ins, rollback_timestamp); + WT_RET(__rollback_abort_insert_list(session, page, ins, rollback_timestamp, NULL)); /* Mark the page as dirty to reconcile the page. */ if (page->modify) __wt_page_modify_set(session, page); -} - -/* - * __rollback_abort_row_reconciled_page_internal -- - * Abort updates on a history store using the in-memory build reconciled page of data store. - */ -static int -__rollback_abort_row_reconciled_page_internal(WT_SESSION_IMPL *session, const void *image, - const uint8_t *addr, size_t addr_size, wt_timestamp_t rollback_timestamp) -{ - WT_DECL_RET; - WT_ITEM tmp; - WT_PAGE *mod_page; - WT_ROW *rip; - uint32_t i, page_flags; - const void *image_local; - - /* - * Don't pass an allocated buffer to the underlying block read function, force allocation of new - * memory of the appropriate size. - */ - WT_CLEAR(tmp); - - mod_page = NULL; - image_local = image; - - if (image_local == NULL) { - WT_RET(__wt_bt_read(session, &tmp, addr, addr_size)); - image_local = tmp.data; - } - - /* Don't free the passed image later. */ - page_flags = image != NULL ? 0 : WT_PAGE_DISK_ALLOC; - WT_ERR(__wt_page_inmem(session, NULL, image_local, page_flags, &mod_page)); - tmp.mem = NULL; - WT_ROW_FOREACH (mod_page, rip, i) - WT_ERR_NOTFOUND_OK(__rollback_ondisk_fixup_key( - session, NULL, mod_page, NULL, rip, rollback_timestamp, false, 0), - false); - -err: - if (mod_page != NULL) - __wt_page_out(session, &mod_page); - __wt_buf_free(session, &tmp); - - return (ret); -} - -/* - * __rollback_abort_row_reconciled_page -- - * Abort updates on a history store using the reconciled pages of data store. - */ -static int -__rollback_abort_row_reconciled_page( - WT_SESSION_IMPL *session, WT_PAGE *page, wt_timestamp_t rollback_timestamp) -{ - WT_MULTI *multi; - WT_PAGE_MODIFY *mod; - uint32_t multi_entry; - char ts_string[3][WT_TS_INT_STRING_SIZE]; - - if ((mod = page->modify) == NULL) - return (0); - - if (mod->rec_result == WT_PM_REC_REPLACE && - (mod->mod_replace.ta.newest_start_durable_ts > rollback_timestamp || - mod->mod_replace.ta.newest_stop_durable_ts > rollback_timestamp || - mod->mod_replace.ta.prepare)) { - __wt_verbose(session, WT_VERB_RECOVERY_RTS(session), - "reconciled replace block page history store update removal on-disk with start durable " - "timestamp: %s, stop durable timestamp: %s and stable timestamp: %s", - __wt_timestamp_to_string(mod->mod_replace.ta.newest_start_durable_ts, ts_string[0]), - __wt_timestamp_to_string(mod->mod_replace.ta.newest_stop_durable_ts, ts_string[1]), - __wt_timestamp_to_string(rollback_timestamp, ts_string[2])); - - /* Remove the history store newer updates. */ - if (!WT_IS_HS(session->dhandle)) - WT_RET(__rollback_abort_row_reconciled_page_internal(session, mod->u1.r.disk_image, - mod->u1.r.replace.addr, mod->u1.r.replace.size, rollback_timestamp)); - - /* - * As this page has newer aborts that are aborted, make sure to mark the page as dirty to - * let the reconciliation happens again on the page. Otherwise, the eviction may pick the - * already reconciled page to write to disk with newer updates. - */ - __wt_page_modify_set(session, page); - } else if (mod->rec_result == WT_PM_REC_MULTIBLOCK) { - for (multi = mod->mod_multi, multi_entry = 0; multi_entry < mod->mod_multi_entries; - ++multi, ++multi_entry) - if (multi->addr.ta.newest_start_durable_ts > rollback_timestamp || - multi->addr.ta.newest_stop_durable_ts > rollback_timestamp || - multi->addr.ta.prepare) { - __wt_verbose(session, WT_VERB_RECOVERY_RTS(session), - "reconciled multi block page history store update removal on-disk with start " - "durable timestamp: %s, stop durable timestamp: %s and stable timestamp: %s", - __wt_timestamp_to_string(multi->addr.ta.newest_start_durable_ts, ts_string[0]), - __wt_timestamp_to_string(multi->addr.ta.newest_stop_durable_ts, ts_string[1]), - __wt_timestamp_to_string(rollback_timestamp, ts_string[2])); - - /* Remove the history store newer updates. */ - if (!WT_IS_HS(session->dhandle)) - WT_RET(__rollback_abort_row_reconciled_page_internal(session, multi->disk_image, - multi->addr.addr, multi->addr.size, rollback_timestamp)); - - /* - * As this page has newer aborts that are aborted, make sure to mark the page as - * dirty to let the reconciliation happens again on the page. Otherwise, the - * eviction may pick the already reconciled page to write to disk with newer - * updates. - */ - __wt_page_modify_set(session, page); - } - } return (0); } @@ -921,6 +871,8 @@ __rollback_abort_row_reconciled_page( static int __rollback_abort_row_leaf(WT_SESSION_IMPL *session, WT_REF *ref, wt_timestamp_t rollback_timestamp) { + WT_DECL_ITEM(key); + WT_DECL_RET; WT_INSERT_HEAD *insert; WT_PAGE *page; WT_ROW *rip; @@ -930,11 +882,13 @@ __rollback_abort_row_leaf(WT_SESSION_IMPL *session, WT_REF *ref, wt_timestamp_t page = ref->page; + WT_RET(__wt_scr_alloc(session, 0, &key)); + /* * Review the insert list for keys before the first entry on the disk page. */ if ((insert = WT_ROW_INSERT_SMALLEST(page)) != NULL) - __rollback_abort_insert_list(session, insert, rollback_timestamp); + WT_ERR(__rollback_abort_insert_list(session, page, insert, rollback_timestamp, NULL)); /* * Review updates that belong to keys that are on the disk image, as well as for keys inserted @@ -942,30 +896,29 @@ __rollback_abort_row_leaf(WT_SESSION_IMPL *session, WT_REF *ref, wt_timestamp_t */ WT_ROW_FOREACH (page, rip, i) { stable_update_found = false; - if ((upd = WT_ROW_UPDATE(page, rip)) != NULL) - __rollback_abort_update(session, upd, rollback_timestamp, &stable_update_found); + if ((upd = WT_ROW_UPDATE(page, rip)) != NULL) { + WT_ERR(__wt_row_leaf_key(session, page, rip, key, false)); + WT_ERR( + __rollback_abort_update(session, key, upd, rollback_timestamp, &stable_update_found)); + } if ((insert = WT_ROW_INSERT(page, rip)) != NULL) - __rollback_abort_insert_list(session, insert, rollback_timestamp); + WT_ERR(__rollback_abort_insert_list(session, page, insert, rollback_timestamp, NULL)); /* * If there is no stable update found in the update list, abort any on-disk value. */ if (!stable_update_found) - WT_RET(__rollback_abort_ondisk_kv(session, ref, NULL, rip, rollback_timestamp, 0)); + WT_ERR(__rollback_abort_ondisk_kv(session, ref, NULL, rip, rollback_timestamp, 0)); } - /* - * If the configuration is not in-memory, abort history store updates from the reconciled pages - * of data store. - */ - if (!F_ISSET(S2C(session), WT_CONN_IN_MEMORY)) - WT_RET(__rollback_abort_row_reconciled_page(session, page, rollback_timestamp)); - /* Mark the page as dirty to reconcile the page. */ if (page->modify) __wt_page_modify_set(session, page); - return (0); + +err: + __wt_scr_free(session, &key); + return (ret); } /* @@ -1045,14 +998,14 @@ __rollback_page_needs_abort( prepared = vpack.ta.prepare; newest_txn = vpack.ta.newest_txn; result = (durable_ts > rollback_timestamp) || prepared || - WT_CHECK_RECOVERY_FLAG_TS_TXNID(session, newest_txn, durable_ts); + WT_CHECK_RECOVERY_FLAG_TXNID(session, newest_txn); } else if (addr != NULL) { tag = "address"; durable_ts = __rollback_get_ref_max_durable_timestamp(session, &addr->ta); prepared = addr->ta.prepare; newest_txn = addr->ta.newest_txn; result = (durable_ts > rollback_timestamp) || prepared || - WT_CHECK_RECOVERY_FLAG_TS_TXNID(session, newest_txn, durable_ts); + WT_CHECK_RECOVERY_FLAG_TXNID(session, newest_txn); } __wt_verbose(session, WT_VERB_RECOVERY_RTS(session), @@ -1072,19 +1025,11 @@ __rollback_abort_updates(WT_SESSION_IMPL *session, WT_REF *ref, wt_timestamp_t r { WT_PAGE *page; - /* Review deleted page saved to the ref. */ - if (ref->page_del != NULL && rollback_timestamp < ref->page_del->durable_timestamp) { - __wt_verbose( - session, WT_VERB_RECOVERY_RTS(session), "%p: deleted page rolled back", (void *)ref); - WT_RET(__wt_delete_page_rollback(session, ref)); - } - /* * If we have a ref with clean page, find out whether the page has any modifications that are * newer than the given timestamp. As eviction writes the newest version to page, even a clean * page may also contain modifications that need rollback. */ - WT_ASSERT(session, ref->page != NULL); page = ref->page; if (!__wt_page_is_modified(page) && !__rollback_page_needs_abort(session, ref, rollback_timestamp)) { @@ -1099,7 +1044,7 @@ __rollback_abort_updates(WT_SESSION_IMPL *session, WT_REF *ref, wt_timestamp_t r switch (page->type) { case WT_PAGE_COL_FIX: - __rollback_abort_col_fix(session, page, rollback_timestamp); + WT_RET(__rollback_abort_col_fix(session, page, rollback_timestamp)); break; case WT_PAGE_COL_VAR: WT_RET(__rollback_abort_col_var(session, ref, rollback_timestamp)); @@ -1124,19 +1069,30 @@ __rollback_abort_updates(WT_SESSION_IMPL *session, WT_REF *ref, wt_timestamp_t r /* * __rollback_abort_fast_truncate -- - * Abort fast truncate on this page newer than the timestamp. + * Abort fast truncate for an internal page of leaf pages. */ static int __rollback_abort_fast_truncate( WT_SESSION_IMPL *session, WT_REF *ref, wt_timestamp_t rollback_timestamp) { - /* Review deleted page saved to the ref. */ - if (ref->page_del != NULL && rollback_timestamp < ref->page_del->durable_timestamp) { - __wt_verbose( - session, WT_VERB_RECOVERY_RTS(session), "%p: deleted page rolled back", (void *)ref); - WT_RET(__wt_delete_page_rollback(session, ref)); - } + WT_REF *child_ref; + WT_INTL_FOREACH_BEGIN (session, ref->page, child_ref) { + /* + * A fast-truncate page is either in the WT_REF_DELETED state (where the WT_PAGE_DELETED + * structure has the timestamp information), or in an in-memory state where it started as a + * fast-truncate page which was then instantiated and the timestamp information moved to the + * individual WT_UPDATE structures. When reviewing internal pages, ignore the second case, + * an instantiated page is handled when the leaf page is visited. + */ + if (child_ref->state == WT_REF_DELETED && child_ref->ft_info.del != NULL && + rollback_timestamp < child_ref->ft_info.del->durable_timestamp) { + __wt_verbose(session, WT_VERB_RECOVERY_RTS(session), "%p: deleted page rolled back", + (void *)child_ref); + WT_RET(__wt_delete_page_rollback(session, child_ref)); + } + } + WT_INTL_FOREACH_END; return (0); } @@ -1174,25 +1130,19 @@ static int __rollback_to_stable_btree_walk(WT_SESSION_IMPL *session, wt_timestamp_t rollback_timestamp) { WT_DECL_RET; - WT_REF *child_ref, *ref; - - /* Set this flag to return error instead of panic if file is corrupted. */ - F_SET(session, WT_SESSION_QUIET_CORRUPT_FILE); + WT_REF *ref; /* Walk the tree, marking commits aborted where appropriate. */ ref = NULL; while ((ret = __wt_tree_walk_custom_skip(session, &ref, __wt_rts_page_skip, &rollback_timestamp, WT_READ_NO_EVICT | WT_READ_WONT_NEED)) == 0 && ref != NULL) - if (F_ISSET(ref, WT_REF_FLAG_INTERNAL)) { - WT_INTL_FOREACH_BEGIN (session, ref->page, child_ref) { - WT_RET(__rollback_abort_fast_truncate(session, child_ref, rollback_timestamp)); - } - WT_INTL_FOREACH_END; - } else + if (F_ISSET(ref, WT_REF_FLAG_INTERNAL)) + WT_WITH_PAGE_INDEX( + session, ret = __rollback_abort_fast_truncate(session, ref, rollback_timestamp)); + else WT_RET(__rollback_abort_updates(session, ref, rollback_timestamp)); - F_CLR(session, WT_SESSION_QUIET_CORRUPT_FILE); return (ret); } @@ -1205,7 +1155,6 @@ __rollback_to_stable_btree(WT_SESSION_IMPL *session, wt_timestamp_t rollback_tim { WT_BTREE *btree; WT_CONNECTION_IMPL *conn; - WT_DECL_RET; btree = S2BT(session); conn = S2C(session); @@ -1219,17 +1168,12 @@ __rollback_to_stable_btree(WT_SESSION_IMPL *session, wt_timestamp_t rollback_tim * Immediately durable files don't get their commits wiped. This case mostly exists to support * the semantic required for the oplog in MongoDB - updates that have been made to the oplog * should not be aborted. It also wouldn't be safe to roll back updates for any table that had - * it's records logged, since those updates would be recovered after a crash making them - * inconsistent. + * its records logged: those updates would be recovered after a crash, making them inconsistent. */ - if (__wt_btree_immediately_durable(session)) { - if (btree->id >= conn->stable_rollback_maxfile) - WT_RET_PANIC(session, EINVAL, "btree file ID %" PRIu32 " larger than max %" PRIu32, - btree->id, conn->stable_rollback_maxfile); + if (__wt_btree_immediately_durable(session)) return (0); - } - /* There is never anything to do for checkpoint handles */ + /* There is never anything to do for checkpoint handles. */ if (session->dhandle->checkpoint != NULL) return (0); @@ -1237,8 +1181,7 @@ __rollback_to_stable_btree(WT_SESSION_IMPL *session, wt_timestamp_t rollback_tim if (btree->root.page == NULL) return (0); - WT_WITH_PAGE_INDEX(session, ret = __rollback_to_stable_btree_walk(session, rollback_timestamp)); - return (ret); + return (__rollback_to_stable_btree_walk(session, rollback_timestamp)); } /* @@ -1409,211 +1352,238 @@ __rollback_progress_msg(WT_SESSION_IMPL *session, struct timespec rollback_start /* * __rollback_to_stable_btree_apply -- - * Perform rollback to stable to all files listed in the metadata, apart from the metadata and - * history store files. + * Perform rollback to stable on a single file. */ static int -__rollback_to_stable_btree_apply(WT_SESSION_IMPL *session) +__rollback_to_stable_btree_apply( + WT_SESSION_IMPL *session, const char *uri, const char *config, wt_timestamp_t rollback_timestamp) { WT_CONFIG ckptconf; WT_CONFIG_ITEM cval, value, key; - WT_CURSOR *cursor; WT_DECL_RET; WT_TXN_GLOBAL *txn_global; - wt_timestamp_t max_durable_ts, newest_start_durable_ts, newest_stop_durable_ts, - rollback_timestamp; - struct timespec rollback_timer; - uint64_t rollback_count, rollback_msg_count, rollback_txnid; - uint32_t btree_id; + wt_timestamp_t max_durable_ts, newest_start_durable_ts, newest_stop_durable_ts; size_t addr_size; + uint64_t rollback_txnid; + uint32_t btree_id, handle_open_flags; char ts_string[2][WT_TS_INT_STRING_SIZE]; - const char *config, *uri; - bool durable_ts_found, prepared_updates, has_txn_updates_gt_than_ckpt_snap; - bool dhandle_allocated, perform_rts; + bool dhandle_allocated, durable_ts_found, has_txn_updates_gt_than_ckpt_snap, perform_rts; + bool prepared_updates; + + /* Ignore non-file objects as well as the metadata and history store files. */ + if (!WT_PREFIX_MATCH(uri, "file:") || strcmp(uri, WT_HS_URI) == 0 || + strcmp(uri, WT_METAFILE_URI) == 0) + return (0); txn_global = &S2C(session)->txn_global; - rollback_count = rollback_msg_count = rollback_txnid = 0; + rollback_txnid = 0; addr_size = 0; + dhandle_allocated = false; - /* Initialize the verbose tracking timer. */ - __wt_epoch(session, &rollback_timer); + /* Find out the max durable timestamp of the object from checkpoint. */ + newest_start_durable_ts = newest_stop_durable_ts = WT_TS_NONE; + durable_ts_found = prepared_updates = has_txn_updates_gt_than_ckpt_snap = false; - /* - * Copy the stable timestamp, otherwise we'd need to lock it each time it's accessed. Even - * though the stable timestamp isn't supposed to be updated while rolling back, accessing it - * without a lock would violate protocol. - */ - WT_ORDERED_READ(rollback_timestamp, txn_global->stable_timestamp); - __wt_verbose(session, WT_VERB_RECOVERY_RTS(session), - "performing rollback to stable with stable timestamp: %s and oldest timestamp: %s", - __wt_timestamp_to_string(rollback_timestamp, ts_string[0]), - __wt_timestamp_to_string(txn_global->oldest_timestamp, ts_string[1])); + WT_RET(__wt_config_getones(session, config, "checkpoint", &cval)); + __wt_config_subinit(session, &ckptconf, &cval); + for (; __wt_config_next(&ckptconf, &key, &cval) == 0;) { + ret = __wt_config_subgets(session, &cval, "newest_start_durable_ts", &value); + if (ret == 0) { + newest_start_durable_ts = WT_MAX(newest_start_durable_ts, (wt_timestamp_t)value.val); + durable_ts_found = true; + } + WT_RET_NOTFOUND_OK(ret); + ret = __wt_config_subgets(session, &cval, "newest_stop_durable_ts", &value); + if (ret == 0) { + newest_stop_durable_ts = WT_MAX(newest_stop_durable_ts, (wt_timestamp_t)value.val); + durable_ts_found = true; + } + WT_RET_NOTFOUND_OK(ret); + ret = __wt_config_subgets(session, &cval, "prepare", &value); + if (ret == 0) { + if (value.val) + prepared_updates = true; + } + WT_RET_NOTFOUND_OK(ret); + ret = __wt_config_subgets(session, &cval, "newest_txn", &value); + if (value.len != 0) + rollback_txnid = (uint64_t)value.val; + WT_RET_NOTFOUND_OK(ret); + ret = __wt_config_subgets(session, &cval, "addr", &value); + if (ret == 0) + addr_size = value.len; + WT_RET_NOTFOUND_OK(ret); + } + max_durable_ts = WT_MAX(newest_start_durable_ts, newest_stop_durable_ts); + has_txn_updates_gt_than_ckpt_snap = WT_CHECK_RECOVERY_FLAG_TXNID(session, rollback_txnid); - WT_ASSERT(session, FLD_ISSET(session->lock_flags, WT_SESSION_LOCKED_SCHEMA)); - WT_RET(__wt_metadata_cursor(session, &cursor)); + /* Increment the inconsistent checkpoint stats counter. */ + if (has_txn_updates_gt_than_ckpt_snap) + WT_STAT_CONN_DATA_INCR(session, txn_rts_inconsistent_ckpt); - if (F_ISSET(S2C(session), WT_CONN_RECOVERING)) + /* + * The rollback to stable will skip the tables during recovery and shutdown in the following + * conditions. + * 1. Empty table. + * 2. Table has timestamped updates without a stable timestamp. + */ + if ((F_ISSET(S2C(session), WT_CONN_RECOVERING) || + F_ISSET(S2C(session), WT_CONN_CLOSING_TIMESTAMP)) && + (addr_size == 0 || + (txn_global->stable_timestamp == WT_TS_NONE && max_durable_ts != WT_TS_NONE))) { __wt_verbose(session, WT_VERB_RECOVERY_RTS(session), - "recovered checkpoint snapshot min: %" PRIu64 ", snapshot max: %" PRIu64 - ", snapshot count: %" PRIu32, - S2C(session)->recovery_ckpt_snap_min, S2C(session)->recovery_ckpt_snap_max, - S2C(session)->recovery_ckpt_snapshot_count); - - while ((ret = cursor->next(cursor)) == 0) { - /* Log a progress message. */ - __rollback_progress_msg(session, rollback_timer, rollback_count, &rollback_msg_count); - ++rollback_count; + "skip rollback to stable on file %s because %s", uri, + addr_size == 0 ? "its checkpoint address length is 0" : + "it has timestamped updates and the stable timestamp is 0"); + return (0); + } - WT_ERR(cursor->get_key(cursor, &uri)); - dhandle_allocated = perform_rts = false; + /* + * The rollback operation should be performed on this file based on the following: + * 1. The dhandle is present in the cache and tree is modified. + * 2. The checkpoint durable start/stop timestamp is greater than the rollback timestamp. + * 3. The checkpoint has prepared updates written to disk. + * 4. There is no durable timestamp in any checkpoint. + * 5. The checkpoint newest txn is greater than snapshot min txn id. + */ + WT_WITH_HANDLE_LIST_READ_LOCK(session, (ret = __wt_conn_dhandle_find(session, uri, NULL))); - /* Ignore metadata and history store files. */ - if (strcmp(uri, WT_METAFILE_URI) == 0 || strcmp(uri, WT_HS_URI) == 0) - continue; + perform_rts = ret == 0 && S2BT(session)->modified; - if (!WT_PREFIX_MATCH(uri, "file:")) - continue; + WT_ERR_NOTFOUND_OK(ret, false); - WT_ERR(cursor->get_value(cursor, &config)); + if (perform_rts || max_durable_ts > rollback_timestamp || prepared_updates || + !durable_ts_found || has_txn_updates_gt_than_ckpt_snap) { + /* + * MongoDB does not close all open handles before calling rollback-to-stable; otherwise, + * don't permit that behavior, the application is likely making a mistake. + */ +#ifdef WT_STANDALONE_BUILD + handle_open_flags = WT_DHANDLE_DISCARD | WT_DHANDLE_EXCLUSIVE; +#else + handle_open_flags = 0; +#endif + ret = __wt_session_get_dhandle(session, uri, NULL, NULL, handle_open_flags); + if (ret != 0) + WT_ERR_MSG(session, ret, "%s: unable to open handle%s", uri, + ret == EBUSY ? ", error indicates handle is unavailable due to concurrent use" : ""); + dhandle_allocated = true; - /* Find out the max durable timestamp of the object from checkpoint. */ - newest_start_durable_ts = newest_stop_durable_ts = WT_TS_NONE; - durable_ts_found = prepared_updates = has_txn_updates_gt_than_ckpt_snap = false; + __wt_verbose(session, WT_VERB_RECOVERY_RTS(session), + "tree rolled back with durable timestamp: %s, or when tree is modified: %s or " + "prepared updates: %s or when durable time is not found: %s or txnid: %" PRIu64 + " is greater than recovery checkpoint snap min: %s", + __wt_timestamp_to_string(max_durable_ts, ts_string[0]), + S2BT(session)->modified ? "true" : "false", prepared_updates ? "true" : "false", + !durable_ts_found ? "true" : "false", rollback_txnid, + has_txn_updates_gt_than_ckpt_snap ? "true" : "false"); + WT_ERR(__rollback_to_stable_btree(session, rollback_timestamp)); + } else + __wt_verbose(session, WT_VERB_RECOVERY_RTS(session), + "%s: tree skipped with durable timestamp: %s and stable timestamp: %s or txnid: %" PRIu64, + uri, __wt_timestamp_to_string(max_durable_ts, ts_string[0]), + __wt_timestamp_to_string(rollback_timestamp, ts_string[1]), rollback_txnid); - /* Get the btree ID. */ + /* + * Truncate history store entries for the non-timestamped table. + * Exceptions: + * 1. Modified tree - Scenarios where the tree is never checkpointed lead to zero + * durable timestamp even they are timestamped tables. Until we have a special + * indication of letting to know the table type other than checking checkpointed durable + * timestamp to WT_TS_NONE, we need this exception. + * 2. In-memory database - In this scenario, there is no history store to truncate. + */ + if ((!dhandle_allocated || !S2BT(session)->modified) && max_durable_ts == WT_TS_NONE && + !F_ISSET(S2C(session), WT_CONN_IN_MEMORY)) { WT_ERR(__wt_config_getones(session, config, "id", &cval)); btree_id = (uint32_t)cval.val; + WT_ERR(__rollback_to_stable_btree_hs_truncate(session, btree_id)); + } - WT_ERR(__wt_config_getones(session, config, "checkpoint", &cval)); - __wt_config_subinit(session, &ckptconf, &cval); - for (; __wt_config_next(&ckptconf, &key, &cval) == 0;) { - ret = __wt_config_subgets(session, &cval, "newest_start_durable_ts", &value); - if (ret == 0) { - newest_start_durable_ts = - WT_MAX(newest_start_durable_ts, (wt_timestamp_t)value.val); - durable_ts_found = true; - } - WT_ERR_NOTFOUND_OK(ret, false); - ret = __wt_config_subgets(session, &cval, "newest_stop_durable_ts", &value); - if (ret == 0) { - newest_stop_durable_ts = WT_MAX(newest_stop_durable_ts, (wt_timestamp_t)value.val); - durable_ts_found = true; - } - WT_ERR_NOTFOUND_OK(ret, false); - ret = __wt_config_subgets(session, &cval, "prepare", &value); - if (ret == 0) { - if (value.val) - prepared_updates = true; - } - WT_ERR_NOTFOUND_OK(ret, false); - ret = __wt_config_subgets(session, &cval, "newest_txn", &value); - if (value.len != 0) - rollback_txnid = (uint64_t)value.val; - WT_ERR_NOTFOUND_OK(ret, false); - ret = __wt_config_subgets(session, &cval, "addr", &value); - if (ret == 0) - addr_size = value.len; - WT_ERR_NOTFOUND_OK(ret, false); - } - max_durable_ts = WT_MAX(newest_start_durable_ts, newest_stop_durable_ts); - has_txn_updates_gt_than_ckpt_snap = - WT_CHECK_RECOVERY_FLAG_TS_TXNID(session, rollback_txnid, max_durable_ts); +err: + if (dhandle_allocated) + WT_TRET(__wt_session_release_dhandle(session)); + return (ret); +} - /* Increment the inconsistent checkpoint stats counter. */ - if (has_txn_updates_gt_than_ckpt_snap) - WT_STAT_CONN_DATA_INCR(session, txn_rts_inconsistent_ckpt); +/* + * __wt_rollback_to_stable_one -- + * Perform rollback to stable on a single object. + */ +int +__wt_rollback_to_stable_one(WT_SESSION_IMPL *session, const char *uri, bool *skipp) +{ + WT_DECL_RET; + wt_timestamp_t rollback_timestamp; + char *config; - /* - * The rollback to stable will skip the tables during recovery and shutdown in the following - * conditions. - * 1. Empty table. - * 2. Table has timestamped updates without a stable timestamp. - */ - if ((F_ISSET(S2C(session), WT_CONN_RECOVERING) || - F_ISSET(S2C(session), WT_CONN_CLOSING_TIMESTAMP)) && - (addr_size == 0 || - (txn_global->stable_timestamp == WT_TS_NONE && max_durable_ts != WT_TS_NONE))) { - __wt_verbose(session, WT_VERB_RECOVERY_RTS(session), - "skip rollback to stable on file %s because %s", uri, - addr_size == 0 ? "its checkpoint address length is 0" : - "it has timestamped updates and the stable timestamp is 0"); - continue; - } + /* + * This is confusing: the caller's boolean argument "skip" stops the schema-worker loop from + * processing this object and any underlying objects it may have (for example, a table with + * multiple underlying file objects). We rollback-to-stable all of the file objects an object + * may contain, so set the caller's skip argument to true on all file objects, else set the + * caller's skip argument to false so our caller continues down the tree of objects. + */ + *skipp = WT_PREFIX_MATCH(uri, "file:"); + if (!*skipp) + return (0); - /* - * The rollback operation should be performed on this file based on the following: - * 1. The dhandle is present in the cache and tree is modified. - * 2. The checkpoint durable start/stop timestamp is greater than the rollback timestamp. - * 3. There is no durable timestamp in any checkpoint. - * 4. The checkpoint newest txn is greater than snapshot min txn id - */ - WT_WITH_HANDLE_LIST_READ_LOCK(session, (ret = __wt_conn_dhandle_find(session, uri, NULL))); + WT_RET(__wt_metadata_search(session, uri, &config)); - if (ret == 0 && S2BT(session)->modified) - perform_rts = true; + /* Read the stable timestamp once, when we first start up. */ + WT_ORDERED_READ(rollback_timestamp, S2C(session)->txn_global.stable_timestamp); - WT_ERR_NOTFOUND_OK(ret, false); + F_SET(session, WT_SESSION_QUIET_CORRUPT_FILE); + ret = __rollback_to_stable_btree_apply(session, uri, config, rollback_timestamp); + F_CLR(session, WT_SESSION_QUIET_CORRUPT_FILE); - if (perform_rts || max_durable_ts > rollback_timestamp || prepared_updates || - !durable_ts_found || has_txn_updates_gt_than_ckpt_snap) { - /* Set this flag to return error instead of panic if file is corrupted. */ - F_SET(session, WT_SESSION_QUIET_CORRUPT_FILE); - ret = __wt_session_get_dhandle(session, uri, NULL, NULL, 0); - F_CLR(session, WT_SESSION_QUIET_CORRUPT_FILE); + __wt_free(session, config); - /* - * Ignore performing rollback to stable on files that does not exist or the files where - * corruption is detected. - */ - if ((ret == ENOENT) || - (ret == WT_ERROR && F_ISSET(S2C(session), WT_CONN_DATA_CORRUPTION))) { - __wt_verbose(session, WT_VERB_RECOVERY_RTS(session), - "ignore performing rollback to stable on %s because the file %s", uri, - ret == ENOENT ? "does not exist" : "is corrupted."); - continue; - } - WT_ERR(ret); + return (ret); +} - dhandle_allocated = true; - __wt_verbose(session, WT_VERB_RECOVERY_RTS(session), - "tree rolled back with durable timestamp: %s, or when tree is modified: %s or " - "prepared updates: %s or when durable time is not found: %s or txnid: %" PRIu64 - " is greater than recovery checkpoint snap min: %s", - __wt_timestamp_to_string(max_durable_ts, ts_string[0]), - S2BT(session)->modified ? "true" : "false", prepared_updates ? "true" : "false", - !durable_ts_found ? "true" : "false", rollback_txnid, - has_txn_updates_gt_than_ckpt_snap ? "true" : "false"); - WT_TRET(__rollback_to_stable_btree(session, rollback_timestamp)); - } else - __wt_verbose(session, WT_VERB_RECOVERY_RTS(session), - "tree skipped with durable timestamp: %s and stable timestamp: %s or txnid: %" PRIu64, - __wt_timestamp_to_string(max_durable_ts, ts_string[0]), - __wt_timestamp_to_string(rollback_timestamp, ts_string[1]), rollback_txnid); +/* + * __rollback_to_stable_btree_apply_all -- + * Perform rollback to stable to all files listed in the metadata, apart from the metadata and + * history store files. + */ +static int +__rollback_to_stable_btree_apply_all(WT_SESSION_IMPL *session, wt_timestamp_t rollback_timestamp) +{ + struct timespec rollback_timer; + WT_CURSOR *cursor; + WT_DECL_RET; + uint64_t rollback_count, rollback_msg_count; + const char *config, *uri; - /* - * Truncate history store entries for the non-timestamped table. - * Exceptions: - * 1. Modified tree - Scenarios where the tree is never checkpointed lead to zero - * durable timestamp even they are timestamped tables. Until we have a special - * indication of letting to know the table type other than checking checkpointed durable - * timestamp to WT_TS_NONE, We need this exception. - * 2. In-memory database - In this scenario, there is no history store to truncate. - */ + /* Initialize the verbose tracking timer. */ + __wt_epoch(session, &rollback_timer); + rollback_count = 0; + rollback_msg_count = 0; - if ((!dhandle_allocated || !S2BT(session)->modified) && max_durable_ts == WT_TS_NONE && - !F_ISSET(S2C(session), WT_CONN_IN_MEMORY)) - WT_TRET(__rollback_to_stable_btree_hs_truncate(session, btree_id)); + WT_RET(__wt_metadata_cursor(session, &cursor)); + while ((ret = cursor->next(cursor)) == 0) { + /* Log a progress message. */ + __rollback_progress_msg(session, rollback_timer, rollback_count, &rollback_msg_count); + ++rollback_count; - if (dhandle_allocated) - WT_TRET(__wt_session_release_dhandle(session)); + WT_ERR(cursor->get_key(cursor, &uri)); + WT_ERR(cursor->get_value(cursor, &config)); + + F_SET(session, WT_SESSION_QUIET_CORRUPT_FILE); + ret = __rollback_to_stable_btree_apply(session, uri, config, rollback_timestamp); + F_CLR(session, WT_SESSION_QUIET_CORRUPT_FILE); /* - * Continue when the table is corrupted and proceed to perform rollback to stable on other - * tables. + * Ignore rollback to stable failures on files that don't exist or files where corruption is + * detected. */ - if (ret == WT_ERROR && F_ISSET(S2C(session), WT_CONN_DATA_CORRUPTION)) + if (ret == ENOENT || (ret == WT_ERROR && F_ISSET(S2C(session), WT_CONN_DATA_CORRUPTION))) { + __wt_verbose(session, WT_VERB_RECOVERY_RTS(session), + "%s: skipped performing rollback to stable because the file %s", uri, + ret == ENOENT ? "does not exist" : "is corrupted."); continue; - + } WT_ERR(ret); } WT_ERR_NOTFOUND_OK(ret, false); @@ -1633,14 +1603,79 @@ err: static int __rollback_to_stable(WT_SESSION_IMPL *session, bool no_ckpt) { + WT_CACHE *cache; WT_CONNECTION_IMPL *conn; WT_DECL_RET; WT_TXN_GLOBAL *txn_global; + wt_timestamp_t rollback_timestamp; + size_t retries; + uint32_t cache_flags; + char ts_string[2][WT_TS_INT_STRING_SIZE]; conn = S2C(session); + cache = conn->cache; txn_global = &conn->txn_global; /* + * We're about to run a check for active transactions in the system to stop users from shooting + * themselves in the foot. Eviction threads may interfere with this check if they involve writes + * to the history store so we need to wait until the system is no longer evicting content. + * + * If we detect active evictions, we should wait a millisecond and check again. If we're waiting + * for evictions to quiesce for more than 2 minutes, we should give up on waiting and proceed + * with the transaction check anyway. + */ +#define WT_RTS_EVICT_MAX_RETRIES (2 * WT_MINUTE * WT_THOUSAND) + /* + * These are the types of evictions that can result in a history store operation. Since we want + * to avoid these happening concurrently with our check, we need to look for these flags. + */ +#define WT_CACHE_EVICT_HS_FLAGS \ + (WT_CACHE_EVICT_DIRTY | WT_CACHE_EVICT_UPDATES | WT_CACHE_EVICT_URGENT) + for (retries = 0; retries < WT_RTS_EVICT_MAX_RETRIES; ++retries) { + /* + * If we're shutting down or running with an in-memory configuration, we aren't at risk of + * racing with history store transactions. + */ + if (F_ISSET(conn, WT_CONN_CLOSING_TIMESTAMP | WT_CONN_IN_MEMORY)) + break; + + /* Check whether eviction has quiesced. */ + WT_ORDERED_READ(cache_flags, cache->flags); + if (!FLD_ISSET(cache_flags, WT_CACHE_EVICT_HS_FLAGS)) { + /* + * If we we find that the eviction flags are unset, interrupt the eviction server and + * acquire the pass lock to stop the server from setting the eviction flags AFTER this + * point and racing with our check. + */ + (void)__wt_atomic_addv32(&cache->pass_intr, 1); + __wt_spin_lock(session, &cache->evict_pass_lock); + (void)__wt_atomic_subv32(&cache->pass_intr, 1); + FLD_SET(session->lock_flags, WT_SESSION_LOCKED_PASS); + + /* + * Check that the flags didn't get set in between when we checked and when we acquired + * the server lock. If it did get set, release the locks and keep trying. If they're + * still unset, break out of this loop and commence our check. + */ + WT_ORDERED_READ(cache_flags, cache->flags); + if (!FLD_ISSET(cache_flags, WT_CACHE_EVICT_HS_FLAGS)) + break; + else { + __wt_spin_unlock(session, &cache->evict_pass_lock); + FLD_CLR(session->lock_flags, WT_SESSION_LOCKED_PASS); + } + } + /* If we're retrying, pause for a millisecond and let eviction make some progress. */ + __wt_sleep(0, WT_THOUSAND); + } + if (retries == WT_RTS_EVICT_MAX_RETRIES) { + WT_ERR(__wt_msg( + session, "timed out waiting for eviction to quiesce, running rollback to stable")); + WT_ASSERT(session, false && "Timed out waiting for eviction to quiesce prior to rts"); + } + + /* * Rollback to stable should ignore tombstones in the history store since it needs to scan the * entire table sequentially. */ @@ -1648,12 +1683,30 @@ __rollback_to_stable(WT_SESSION_IMPL *session, bool no_ckpt) WT_ERR(__rollback_to_stable_check(session)); + if (FLD_ISSET(session->lock_flags, WT_SESSION_LOCKED_PASS)) { + __wt_spin_unlock(session, &cache->evict_pass_lock); + FLD_CLR(session->lock_flags, WT_SESSION_LOCKED_PASS); + } + /* - * Allocate a non-durable btree bitstring. We increment the global value before using it, so the - * current value is already in use, and hence we need to add one here. + * Copy the stable timestamp, otherwise we'd need to lock it each time it's accessed. Even + * though the stable timestamp isn't supposed to be updated while rolling back, accessing it + * without a lock would violate protocol. */ - conn->stable_rollback_maxfile = conn->next_file_id + 1; - WT_ERR(__rollback_to_stable_btree_apply(session)); + WT_ORDERED_READ(rollback_timestamp, txn_global->stable_timestamp); + __wt_verbose(session, WT_VERB_RECOVERY_RTS(session), + "performing rollback to stable with stable timestamp: %s and oldest timestamp: %s", + __wt_timestamp_to_string(rollback_timestamp, ts_string[0]), + __wt_timestamp_to_string(txn_global->oldest_timestamp, ts_string[1])); + + if (F_ISSET(conn, WT_CONN_RECOVERING)) + __wt_verbose(session, WT_VERB_RECOVERY_RTS(session), + "recovered checkpoint snapshot min: %" PRIu64 ", snapshot max: %" PRIu64 + ", snapshot count: %" PRIu32, + conn->recovery_ckpt_snap_min, conn->recovery_ckpt_snap_max, + conn->recovery_ckpt_snapshot_count); + + WT_ERR(__rollback_to_stable_btree_apply_all(session, rollback_timestamp)); /* Rollback the global durable timestamp to the stable timestamp. */ txn_global->has_durable_timestamp = txn_global->has_stable_timestamp; @@ -1668,6 +1721,10 @@ __rollback_to_stable(WT_SESSION_IMPL *session, bool no_ckpt) WT_ERR(session->iface.checkpoint(&session->iface, "force=1")); err: + if (FLD_ISSET(session->lock_flags, WT_SESSION_LOCKED_PASS)) { + __wt_spin_unlock(session, &cache->evict_pass_lock); + FLD_CLR(session->lock_flags, WT_SESSION_LOCKED_PASS); + } F_CLR(session, WT_SESSION_ROLLBACK_TO_STABLE); return (ret); } diff --git a/src/third_party/wiredtiger/src/txn/txn_timestamp.c b/src/third_party/wiredtiger/src/txn/txn_timestamp.c index 6acd265fd2d..9fc79a78d72 100644 --- a/src/third_party/wiredtiger/src/txn/txn_timestamp.c +++ b/src/third_party/wiredtiger/src/txn/txn_timestamp.c @@ -187,10 +187,11 @@ __txn_global_query_timestamp(WT_SESSION_IMPL *session, wt_timestamp_t *tsp, cons */ if (ts == WT_TS_NONE) return (WT_NOTFOUND); - } else if (WT_STRING_MATCH("last_checkpoint", cval.str, cval.len)) - /* Read-only value forever. No lock needed. */ + } else if (WT_STRING_MATCH("last_checkpoint", cval.str, cval.len)) { + /* Read-only value forever. Make sure we don't used a cached version. */ + WT_BARRIER(); ts = txn_global->last_ckpt_timestamp; - else if (WT_STRING_MATCH("oldest", cval.str, cval.len)) { + } else if (WT_STRING_MATCH("oldest", cval.str, cval.len)) { if (!txn_global->has_oldest_timestamp) return (WT_NOTFOUND); ts = txn_global->oldest_timestamp; diff --git a/src/third_party/wiredtiger/src/utilities/util_load.h b/src/third_party/wiredtiger/src/utilities/util_load.h index 3f52ed439eb..5da81d5bfe3 100644 --- a/src/third_party/wiredtiger/src/utilities/util_load.h +++ b/src/third_party/wiredtiger/src/utilities/util_load.h @@ -22,9 +22,9 @@ int config_reorder(WT_SESSION *, char **); int config_update(WT_SESSION *, char **); /* Flags for util_load_json */ -/* AUTOMATIC FLAG VALUE GENERATION START */ +/* AUTOMATIC FLAG VALUE GENERATION START 0 */ #define LOAD_JSON_APPEND 0x1u /* append (ignore record number keys) */ #define LOAD_JSON_NO_OVERWRITE 0x2u /* don't overwrite existing data */ -/* AUTOMATIC FLAG VALUE GENERATION STOP */ +/* AUTOMATIC FLAG VALUE GENERATION STOP 32 */ int util_load_json(WT_SESSION *, const char *, uint32_t); diff --git a/src/third_party/wiredtiger/src/utilities/util_main.c b/src/third_party/wiredtiger/src/utilities/util_main.c index ee81b5f5d52..1d2d86e94ec 100644 --- a/src/third_party/wiredtiger/src/utilities/util_main.c +++ b/src/third_party/wiredtiger/src/utilities/util_main.c @@ -25,6 +25,17 @@ static const char *mongodb_config = "log=(enabled=true,path=journal,compressor=s #define REC_RECOVER "log=(recover=on)" #define SALVAGE "salvage=true" +/* + * wt_explicit_zero: clear a buffer, with precautions against being optimized away. + */ +static void +wt_explicit_zero(void *ptr, size_t len) +{ + /* Call through a volatile pointer to avoid removal even when it's a dead store. */ + static void *(*volatile memsetptr)(void *ptr, int ch, size_t len) = memset; + (void)memsetptr(ptr, '\0', len); +} + static void usage(void) { @@ -100,13 +111,15 @@ main(int argc, char *argv[]) case 'C': /* wiredtiger_open config */ cmd_config = __wt_optarg; break; - case 'E': /* secret key */ + case 'E': /* secret key */ + if (secretkey != NULL) + wt_explicit_zero(secretkey, strlen(secretkey)); free(secretkey); /* lint: set more than once */ if ((secretkey = strdup(__wt_optarg)) == NULL) { (void)util_err(NULL, errno, NULL); goto err; } - memset(__wt_optarg, 0, strlen(__wt_optarg)); + wt_explicit_zero(__wt_optarg, strlen(__wt_optarg)); break; case 'h': /* home directory */ home = __wt_optarg; @@ -286,6 +299,15 @@ open: goto err; } + if (secretkey != NULL) { + /* p contains a copy of secretkey, so zero both before freeing */ + wt_explicit_zero(p, strlen(p)); + wt_explicit_zero(secretkey, strlen(secretkey)); + } + free(p); + free(secretkey); + config = secretkey = p = NULL; + /* If we only want to verify the metadata, that is done in wiredtiger_open. We're done. */ if (func == NULL && meta_verify) goto done; @@ -304,6 +326,17 @@ err: } done: + /* may get here via either err or done before the free above happens */ + if (p != NULL) { + /* p may contain a copy of secretkey, so zero before freeing */ + wt_explicit_zero(p, strlen(p)); + free(p); + } + if (secretkey != NULL) { + wt_explicit_zero(secretkey, strlen(secretkey)); + free(secretkey); + } + if (conn != NULL) { /* Maintain backward compatibility. */ if (backward_compatible && @@ -315,9 +348,6 @@ done: ret = tret; } - free(p); - free(secretkey); - return (ret == 0 ? EXIT_SUCCESS : EXIT_FAILURE); } diff --git a/src/third_party/wiredtiger/test/cppsuite/Makefile.am b/src/third_party/wiredtiger/test/cppsuite/Makefile.am index 424e91e3276..82c30442482 100644 --- a/src/third_party/wiredtiger/test/cppsuite/Makefile.am +++ b/src/third_party/wiredtiger/test/cppsuite/Makefile.am @@ -11,7 +11,26 @@ all: all_TESTS= noinst_PROGRAMS= -run_SOURCES = tests/run.cxx +run_SOURCES = test_harness/core/component.cxx \ + test_harness/core/configuration.cxx \ + test_harness/core/throttle.cxx \ + test_harness/util/logger.cxx \ + test_harness/util/scoped_types.cxx \ + test_harness/workload/database_model.cxx \ + test_harness/workload/database_operation.cxx \ + test_harness/workload/random_generator.cxx \ + test_harness/workload/thread_context.cxx \ + test_harness/workload/workload_tracking.cxx \ + test_harness/workload/workload_validation.cxx \ + test_harness/checkpoint_manager.cxx \ + test_harness/connection_manager.cxx \ + test_harness/runtime_monitor.cxx \ + test_harness/test.cxx \ + test_harness/thread_manager.cxx \ + test_harness/timestamp_manager.cxx \ + test_harness/workload_generator.cxx \ + tests/run.cxx + noinst_PROGRAMS += run all_TESTS += run diff --git a/src/third_party/wiredtiger/test/cppsuite/configs/base_test_default.txt b/src/third_party/wiredtiger/test/cppsuite/configs/base_test_default.txt new file mode 100644 index 00000000000..986dd19373b --- /dev/null +++ b/src/third_party/wiredtiger/test/cppsuite/configs/base_test_default.txt @@ -0,0 +1,67 @@ +# Used as a base test for the framework. +duration_seconds=60, +cache_size_mb=200, +enable_logging=true, +checkpoint_manager= +( + enabled=true, + op_rate=5s +), +runtime_monitor= +( + stat_cache_size= + ( + enabled=true, + limit=1000 + ), + stat_db_size= + ( + enabled=true, + #10GB + limit=10000000000 + ), + postrun_statistics=[cache_hs_insert:0:100000000, cc_pages_removed:0:10000000], +), +timestamp_manager= +( + enabled=true, + oldest_lag=2, + op_rate=500ms, + stable_lag=2 +), +workload_generator= +( + populate_config= + ( + collection_count=100, + key_count_per_collection=50, + key_size=10, + thread_count=20, + value_size=20 + ), + insert_config= + ( + key_size=10, + op_rate=10ms, + ops_per_transaction=(max=100,min=0), + thread_count=5, + value_size=20 + ), + read_config= + ( + op_rate=3ms, + ops_per_transaction=(max=100,min=50), + thread_count=10 + ), + update_config= + ( + op_rate=15ms, + ops_per_transaction=(max=100,min=20), + thread_count=10, + value_size=20 + ) +), +workload_tracking= +( + op_rate=5s +) diff --git a/src/third_party/wiredtiger/test/cppsuite/configs/base_test_insert_heavy.txt b/src/third_party/wiredtiger/test/cppsuite/configs/base_test_insert_heavy.txt new file mode 100644 index 00000000000..154b5439653 --- /dev/null +++ b/src/third_party/wiredtiger/test/cppsuite/configs/base_test_insert_heavy.txt @@ -0,0 +1,54 @@ +# Used as a stress test for the framework. +duration_seconds=300, +cache_size_mb=2000, +checkpoint_manager= +( + enabled=true, + op_rate=20s +), +runtime_monitor= +( + stat_cache_size= + ( + enabled=false, + limit=100 + ) +), +timestamp_manager= +( + enabled=true, + oldest_lag=30, + op_rate=1s, + stable_lag=10 +), +workload_generator= +( + populate_config= + ( + collection_count=100, + key_count_per_collection=500, + key_size=100, + thread_count=20, + value_size=2000 + ), + insert_config= + ( + key_size=100, + op_rate=10ms, + ops_per_transaction=(max=100,min=0), + thread_count=50, + value_size=2000 + ), + read_config= + ( + op_rate=3ms, + ops_per_transaction=(max=100,min=50), + thread_count=10 + ), +), +# Tracking is currently disabled as verification can't handle rollbacks. +workload_tracking= +( + enabled=false, + op_rate=30s +) diff --git a/src/third_party/wiredtiger/test/cppsuite/configs/base_test_stress.txt b/src/third_party/wiredtiger/test/cppsuite/configs/base_test_stress.txt new file mode 100644 index 00000000000..1fed0b7300c --- /dev/null +++ b/src/third_party/wiredtiger/test/cppsuite/configs/base_test_stress.txt @@ -0,0 +1,70 @@ +# Used as a stress test for the framework. +duration_seconds=1500, +cache_size_mb=1000, +compression_enabled=true, +checkpoint_manager= +( + enabled=true, + op_rate=30s +), +runtime_monitor= +( + stat_cache_size= + ( + enabled=false, + limit=100 + ), + stat_db_size= + ( + enabled=true, + #At the end of the run the data files are approximately 2.3MB each. Which is a total of: + #1.15GB, the history store isn't significant. Give the workload an extra 200MB of margin. + limit=1350000000, + ) +), +timestamp_manager= +( + enabled=true, + oldest_lag=5, + op_rate=1s, + stable_lag=10 +), +workload_generator= +( + populate_config= + ( + collection_count=500, + #5GB of data + key_count_per_collection=1000, + key_size=100, + thread_count=20, + value_size=10000 + ), + insert_config= + ( + key_size=100, + op_rate=10ms, + ops_per_transaction=(max=20,min=10), + thread_count=25, + value_size=10000 + ), + read_config= + ( + op_rate=3ms, + ops_per_transaction=(max=100,min=50), + thread_count=20 + ), + update_config= + ( + op_rate=10s, + ops_per_transaction=(max=10,min=5), + thread_count=25, + value_size=10000 + ) +), +workload_tracking= +( + enabled=true, + # Run after checkpoint most of the time. + op_rate=40s +) diff --git a/src/third_party/wiredtiger/test/cppsuite/configs/config_poc_test_default.txt b/src/third_party/wiredtiger/test/cppsuite/configs/config_poc_test_default.txt deleted file mode 100644 index 74d35e1cbcd..00000000000 --- a/src/third_party/wiredtiger/test/cppsuite/configs/config_poc_test_default.txt +++ /dev/null @@ -1,31 +0,0 @@ -# Sets up a basic database with 2 collections and 5 keys and run thread for 10 seconds. -# All components are enabled. -# Used as a basic test for the framework. -duration_seconds=10, -cache_size_mb=1000, -checkpoint_manager= -( - enabled=true, - op_rate=5s -), -runtime_monitor= -( - stat_cache_size= - ( - enabled=true, - limit=100 - ) -), -workload_generator= -( - collection_count=2, - key_count=5, - key_size=1, - ops_per_transaction= - ( - min=5, - max=50 - ), - read_threads=1, - update_threads=1 -), diff --git a/src/third_party/wiredtiger/test/cppsuite/configs/config_poc_test_stress.txt b/src/third_party/wiredtiger/test/cppsuite/configs/config_poc_test_stress.txt deleted file mode 100644 index 34b6f9b89fe..00000000000 --- a/src/third_party/wiredtiger/test/cppsuite/configs/config_poc_test_stress.txt +++ /dev/null @@ -1,33 +0,0 @@ -# Sets up a basic database with 2 collections and 50000 keys and run thread for 10 seconds. -# All components are enabled. -# Used as a stress test for the framework. -duration_seconds=10, -cache_size_mb=5000, -enable_logging=true, -checkpoint_manager= -( - enabled=true, - op_rate=5s -), -runtime_monitor= -( - stat_cache_size= - ( - enabled=true, - limit=100 - ) -), -workload_generator= -( - collection_count=2, - key_count=50000, - key_size=10, - ops_per_transaction= - ( - min=5, - max=50 - ), - read_threads=1, - update_threads=1, - value_size=2000 -), diff --git a/src/third_party/wiredtiger/test/cppsuite/configs/config_example_test_default.txt b/src/third_party/wiredtiger/test/cppsuite/configs/example_test_default.txt index f5d5c916bdc..f5d5c916bdc 100644 --- a/src/third_party/wiredtiger/test/cppsuite/configs/config_example_test_default.txt +++ b/src/third_party/wiredtiger/test/cppsuite/configs/example_test_default.txt diff --git a/src/third_party/wiredtiger/test/cppsuite/configs/hs_cleanup_default.txt b/src/third_party/wiredtiger/test/cppsuite/configs/hs_cleanup_default.txt new file mode 100644 index 00000000000..5dc77f97645 --- /dev/null +++ b/src/third_party/wiredtiger/test/cppsuite/configs/hs_cleanup_default.txt @@ -0,0 +1,59 @@ +# Configuration for hs_cleanup. +# need to be defined. +duration_seconds=60, +cache_size_mb=200, +statistics_config= +( + type=all, + enable_logging=true +), +checkpoint_manager= +( + enabled=true, + op_rate=20s +), +runtime_monitor= +( + stat_cache_size= + ( + enabled=true, + limit=1000 + ), + stat_db_size= + ( + enabled=true, + #10GB + limit=10000000000 + ), + postrun_statistics=[cache_hs_insert:0:100000000, cc_pages_removed:0:10000000], +), +timestamp_manager= +( + enabled=true, + oldest_lag=5, + op_rate=1s, + stable_lag=5 +), +workload_generator= +( + populate_config= + ( + collection_count=10, + key_count_per_collection=500, + key_size=100, + thread_count=10, + value_size=10000 + ), + update_config= + ( + op_rate=10ms, + ops_per_transaction=(max=100,min=20), + thread_count=10, + value_size=10000 + ) +), +workload_tracking= +( + enabled=true, + op_rate=30s +) diff --git a/src/third_party/wiredtiger/test/cppsuite/configs/hs_cleanup_stress.txt b/src/third_party/wiredtiger/test/cppsuite/configs/hs_cleanup_stress.txt new file mode 100644 index 00000000000..f5b5793491d --- /dev/null +++ b/src/third_party/wiredtiger/test/cppsuite/configs/hs_cleanup_stress.txt @@ -0,0 +1,69 @@ +# Run for half an hour. +duration_seconds=1800, +cache_size_mb=200, +compression_enabled=true, +statistics_config= +( + type=all, + enable_logging=true +), +checkpoint_manager= +( + enabled=true, + op_rate=60s +), +runtime_monitor= +( + stat_cache_size= + ( + enabled=true, + limit=110 + ), + # The data files compress to around 25MB per table at the end of a run so 250MB total. + # +1.4GB for the history store. With an additional 150MB margin. + stat_db_size= + ( + enabled=true, + limit=1900000000, + ), + # Seems to insert around 490K records. Give it +-25K margin. + # Seems to remove 180K records. Give it a similar margin. + postrun_statistics=[cache_hs_insert:470000:515000, cc_pages_removed:170000:200000] +), +timestamp_manager= +( + enabled=true, + oldest_lag=2, + stable_lag=5 +), +workload_generator= +( + populate_config= + ( + collection_count=10, + key_count_per_collection=1000, + key_size=5, + thread_count=10, + value_size=100000 + ), + read_config= + ( + op_rate=5ms, + ops_per_transaction=(max=100,min=1), + thread_count=8 + ), + update_config= + ( + op_rate=10ms, + # Be careful to not aim too high with this config, if we fill the dirty cache and + # all threads are trying to update, they'll get pulled into eviction and will get stuck. + ops_per_transaction=(max=20,min=0), + thread_count=10, + value_size=100000 + ) +), +workload_tracking= +( + enabled=true, + op_rate=20s +) diff --git a/src/third_party/wiredtiger/test/cppsuite/create_test.sh b/src/third_party/wiredtiger/test/cppsuite/create_test.sh index 91f506e39e9..5076b43c7c3 100755 --- a/src/third_party/wiredtiger/test/cppsuite/create_test.sh +++ b/src/third_party/wiredtiger/test/cppsuite/create_test.sh @@ -23,7 +23,7 @@ if test -f "$FILE"; then fi # Check if default configuration associated to the test already exists. -CONFIG=configs/config_$1_default.txt +CONFIG=configs/$1_default.txt if test -f "$CONFIG"; then echo "$CONFIG cannot be created as it already exists." exit 1 @@ -32,7 +32,7 @@ fi # Copy the default template. cp tests/example_test.cxx $FILE echo "Created $FILE." -cp configs/config_example_test_default.txt $CONFIG +cp configs/example_test_default.txt $CONFIG echo "Created $CONFIG." # Replace example_test with the new test name. diff --git a/src/third_party/wiredtiger/test/cppsuite/test_harness/util/debug_utils.h b/src/third_party/wiredtiger/test/cppsuite/test_harness/checkpoint_manager.cxx index a2694f6987c..66836325acf 100644 --- a/src/third_party/wiredtiger/test/cppsuite/test_harness/util/debug_utils.h +++ b/src/third_party/wiredtiger/test/cppsuite/test_harness/checkpoint_manager.cxx @@ -26,32 +26,32 @@ * OTHER DEALINGS IN THE SOFTWARE. */ -#ifndef DEBUG_UTILS_H -#define DEBUG_UTILS_H +#include "checkpoint_manager.h" +#include "connection_manager.h" +#include "util/api_const.h" +#include "util/logger.h" -#include <iostream> - -/* Define helpful functions related to debugging. */ namespace test_harness { +checkpoint_manager::checkpoint_manager(configuration *configuration) + : component(CHECKPOINT_MANAGER, configuration) +{ +} -#define DEBUG_ERROR 0 -#define DEBUG_INFO 1 -#define DEBUG_TRACE 2 +void +checkpoint_manager::load() +{ + /* Load the general component things. */ + component::load(); -static int64_t _trace_level = 0; + /* Create session that we'll use for checkpointing. */ + if (_enabled) + _session = connection_manager::instance().create_session(); +} -/* Used to print out traces for debugging purpose. */ -static void -debug_print(const std::string &str, int64_t trace_type) +void +checkpoint_manager::do_work() { - if (_trace_level >= trace_type) { - if (trace_type >= DEBUG_ERROR) - std::cerr << str << std::endl; - else - std::cout << str << std::endl; - } + logger::log_msg(LOG_INFO, "Running checkpoint"); + testutil_check(_session->checkpoint(_session.get(), nullptr)); } - } // namespace test_harness - -#endif diff --git a/src/third_party/wiredtiger/test/cppsuite/test_harness/checkpoint_manager.h b/src/third_party/wiredtiger/test/cppsuite/test_harness/checkpoint_manager.h index d5396737c43..a127ee63657 100644 --- a/src/third_party/wiredtiger/test/cppsuite/test_harness/checkpoint_manager.h +++ b/src/third_party/wiredtiger/test/cppsuite/test_harness/checkpoint_manager.h @@ -29,45 +29,25 @@ #ifndef CHECKPOINT_MANAGER_H #define CHECKPOINT_MANAGER_H -#include "util/api_const.h" -#include "connection_manager.h" +#include "core/component.h" +#include "util/scoped_types.h" namespace test_harness { - class checkpoint_manager : public component { public: - explicit checkpoint_manager(configuration *configuration) - : component(CHECKPOINT_MANAGER, configuration) - { - } + explicit checkpoint_manager(configuration *configuration); virtual ~checkpoint_manager() = default; /* Delete the copy constructor and the assignment operator. */ checkpoint_manager(const checkpoint_manager &) = delete; checkpoint_manager &operator=(const checkpoint_manager &) = delete; - void - load() override final - { - /* Load the general component things. */ - component::load(); - - /* Create session that we'll use for checkpointing. */ - if (_enabled) - _session = connection_manager::instance().create_session(); - } - - void - do_work() override final - { - debug_print("Running checkpoint", DEBUG_INFO); - testutil_check(_session->checkpoint(_session, nullptr)); - } + void load() override final; + void do_work() override final; private: - WT_SESSION *_session = nullptr; + scoped_session _session; }; - } // namespace test_harness #endif diff --git a/src/third_party/wiredtiger/test/cppsuite/test_harness/connection_manager.cxx b/src/third_party/wiredtiger/test/cppsuite/test_harness/connection_manager.cxx new file mode 100644 index 00000000000..6ff134c7c96 --- /dev/null +++ b/src/third_party/wiredtiger/test/cppsuite/test_harness/connection_manager.cxx @@ -0,0 +1,95 @@ +/*- + * Public Domain 2014-present MongoDB, Inc. + * Public Domain 2008-2014 WiredTiger, Inc. + * + * This is free and unencumbered software released into the public domain. + * + * Anyone is free to copy, modify, publish, use, compile, sell, or + * distribute this software, either in source code form or as a compiled + * binary, for any purpose, commercial or non-commercial, and by any + * means. + * + * In jurisdictions that recognize copyright laws, the author or authors + * of this software dedicate any and all copyright interest in the + * software to the public domain. We make this dedication for the benefit + * of the public at large and to the detriment of our heirs and + * successors. We intend this dedication to be an overt act of + * relinquishment in perpetuity of all present and future rights to this + * software under copyright law. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. + * IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + */ + +#include "connection_manager.h" +#include "util/api_const.h" +#include "util/logger.h" + +namespace test_harness { +connection_manager & +connection_manager::instance() +{ + static connection_manager _instance; + return (_instance); +} + +void +connection_manager::close() +{ + if (_conn != nullptr) { + testutil_check(_conn->close(_conn, nullptr)); + _conn = nullptr; + } +} + +void +connection_manager::create(const std::string &config, const std::string &home) +{ + if (_conn != nullptr) { + logger::log_msg(LOG_ERROR, "Connection is not NULL, cannot be re-opened."); + testutil_die(EINVAL, "Connection is not NULL"); + } + logger::log_msg(LOG_INFO, "wiredtiger_open config: " + config); + + /* Create the working dir. */ + testutil_make_work_dir(home.c_str()); + + /* Open conn. */ + testutil_check(wiredtiger_open(home.c_str(), nullptr, config.c_str(), &_conn)); +} + +scoped_session +connection_manager::create_session() +{ + if (_conn == nullptr) { + logger::log_msg(LOG_ERROR, + "Connection is NULL, did you forget to call " + "connection_manager::create ?"); + testutil_die(EINVAL, "Connection is NULL"); + } + + _conn_mutex.lock(); + scoped_session session(_conn); + _conn_mutex.unlock(); + + return (session); +} + +/* + * set_timestamp calls into the connection API in a thread safe manner to set global timestamps. + */ +void +connection_manager::set_timestamp(const std::string &config) +{ + _conn_mutex.lock(); + testutil_check(_conn->set_timestamp(_conn, config.c_str())); + _conn_mutex.unlock(); +} + +connection_manager::connection_manager() {} +} // namespace test_harness diff --git a/src/third_party/wiredtiger/test/cppsuite/test_harness/connection_manager.h b/src/third_party/wiredtiger/test/cppsuite/test_harness/connection_manager.h index 29c76b59a2b..c6245160df1 100644 --- a/src/third_party/wiredtiger/test/cppsuite/test_harness/connection_manager.h +++ b/src/third_party/wiredtiger/test/cppsuite/test_harness/connection_manager.h @@ -29,6 +29,10 @@ #ifndef CONN_API_H #define CONN_API_H +/* Following definitions are required in order to use printing format specifiers in C++. */ +#define __STDC_LIMIT_MACROS +#define __STDC_FORMAT_MACROS + #include <mutex> extern "C" { @@ -36,8 +40,7 @@ extern "C" { #include "wiredtiger.h" } -#include "util/api_const.h" -#include "util/debug_utils.h" +#include "util/scoped_types.h" namespace test_harness { /* @@ -46,72 +49,26 @@ namespace test_harness { */ class connection_manager { public: + static connection_manager &instance(); + + public: /* No copies of the singleton allowed. */ connection_manager(connection_manager const &) = delete; connection_manager &operator=(connection_manager const &) = delete; - static connection_manager & - instance() - { - static connection_manager _instance; - return (_instance); - } - - void - close() - { - if (_conn != nullptr) { - testutil_check(_conn->close(_conn, NULL)); - _conn = nullptr; - } - } - - void - create(const std::string &config, const std::string &home = DEFAULT_DIR) - { - if (_conn != nullptr) { - debug_print("Connection is not NULL, cannot be re-opened.", DEBUG_ERROR); - testutil_die(EINVAL, "Connection is not NULL"); - } - - /* Create the working dir. */ - testutil_make_work_dir(home.c_str()); - - /* Open conn. */ - testutil_check(wiredtiger_open(home.c_str(), NULL, config.c_str(), &_conn)); - } - - WT_SESSION * - create_session() - { - WT_SESSION *session; - - if (_conn == nullptr) { - debug_print("Connection is NULL, did you forget to call connection_manager::create ?", - DEBUG_ERROR); - testutil_die(EINVAL, "Connection is NULL"); - } - - _conn_mutex.lock(); - testutil_check(_conn->open_session(_conn, NULL, NULL, &session)); - _conn_mutex.unlock(); - - return (session); - } + void close(); + void create(const std::string &config, const std::string &home = DEFAULT_DIR); + scoped_session create_session(); /* * set_timestamp calls into the connection API in a thread safe manner to set global timestamps. */ - void - set_timestamp(const std::string &config) - { - _conn_mutex.lock(); - testutil_check(_conn->set_timestamp(_conn, config.c_str())); - _conn_mutex.unlock(); - } + void set_timestamp(const std::string &config); + + private: + connection_manager(); private: - connection_manager() {} WT_CONNECTION *_conn = nullptr; std::mutex _conn_mutex; }; diff --git a/src/third_party/wiredtiger/test/cppsuite/test_harness/core/component.cxx b/src/third_party/wiredtiger/test/cppsuite/test_harness/core/component.cxx new file mode 100644 index 00000000000..b607f8a821d --- /dev/null +++ b/src/third_party/wiredtiger/test/cppsuite/test_harness/core/component.cxx @@ -0,0 +1,80 @@ +/*- + * Public Domain 2014-present MongoDB, Inc. + * Public Domain 2008-2014 WiredTiger, Inc. + * + * This is free and unencumbered software released into the public domain. + * + * Anyone is free to copy, modify, publish, use, compile, sell, or + * distribute this software, either in source code form or as a compiled + * binary, for any purpose, commercial or non-commercial, and by any + * means. + * + * In jurisdictions that recognize copyright laws, the author or authors + * of this software dedicate any and all copyright interest in the + * software to the public domain. We make this dedication for the benefit + * of the public at large and to the detriment of our heirs and + * successors. We intend this dedication to be an overt act of + * relinquishment in perpetuity of all present and future rights to this + * software under copyright law. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. + * IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + */ + +#include "component.h" +#include "test_harness/util/api_const.h" + +namespace test_harness { +component::component(const std::string &name, configuration *config) : _name(name), _config(config) +{ +} + +component::~component() +{ + delete _config; +} + +void +component::load() +{ + logger::log_msg(LOG_INFO, "Loading component: " + _name); + _enabled = _config->get_optional_bool(ENABLED, true); + _throttle = throttle(_config); + /* If we're not enabled we shouldn't be running. */ + _running = _enabled; +} + +void +component::run() +{ + logger::log_msg(LOG_INFO, "Running component: " + _name); + while (_enabled && _running) { + do_work(); + _throttle.sleep(); + } +} + +void +component::do_work() +{ + /* Not implemented. */ +} + +bool +component::enabled() const +{ + return (_enabled); +} + +void +component::finish() +{ + logger::log_msg(LOG_INFO, "Finishing component: " + _name); + _running = false; +} +} // namespace test_harness diff --git a/src/third_party/wiredtiger/test/cppsuite/test_harness/core/component.h b/src/third_party/wiredtiger/test/cppsuite/test_harness/core/component.h index 341932a0236..398ca11e442 100644 --- a/src/third_party/wiredtiger/test/cppsuite/test_harness/core/component.h +++ b/src/third_party/wiredtiger/test/cppsuite/test_harness/core/component.h @@ -39,12 +39,8 @@ namespace test_harness { */ class component { public: - component(const std::string &name, configuration *config) : _name(name), _config(config) {} - - ~component() - { - delete _config; - } + component(const std::string &name, configuration *config); + virtual ~component(); /* Delete the copy constructor and the assignment operator. */ component(const component &) = delete; @@ -54,15 +50,7 @@ class component { * The load function should perform all tasks required to setup the component for the main phase * of the test. An example operation performed in the load phase would be populating a database. */ - virtual void - load() - { - debug_print("Loading component: " + _name, DEBUG_INFO); - _enabled = _config->get_optional_bool(ENABLED, true); - _throttle = throttle(_config); - /* If we're not enabled we shouldn't be running. */ - _running = _enabled; - } + virtual void load(); /* * The run function provides a top level loop that calls the do_work function every X seconds as @@ -71,39 +59,18 @@ class component { * * If a component does not wish to use the standard run function, it can be overloaded. */ - virtual void - run() - { - debug_print("Running component: " + _name, DEBUG_INFO); - while (_enabled && _running) { - do_work(); - _throttle.sleep(); - } - } + virtual void run(); - virtual void - do_work() - { - /* Not implemented. */ - } + virtual void do_work(); - bool - enabled() const - { - return _enabled; - } + bool enabled() const; /* * The finish phase is a cleanup phase. Created objects are destroyed here and any final testing * requirements can be performed in this phase. An example could be the verification of the * database, or checking some relevant statistics. */ - virtual void - finish() - { - debug_print("Finishing component: " + _name, DEBUG_INFO); - _running = false; - } + virtual void finish(); protected: bool _enabled = false; diff --git a/src/third_party/wiredtiger/test/cppsuite/test_harness/core/configuration.cxx b/src/third_party/wiredtiger/test/cppsuite/test_harness/core/configuration.cxx new file mode 100644 index 00000000000..b6c53397955 --- /dev/null +++ b/src/third_party/wiredtiger/test/cppsuite/test_harness/core/configuration.cxx @@ -0,0 +1,295 @@ +/*- + * Public Domain 2014-present MongoDB, Inc. + * Public Domain 2008-2014 WiredTiger, Inc. + * + * This is free and unencumbered software released into the public domain. + * + * Anyone is free to copy, modify, publish, use, compile, sell, or + * distribute this software, either in source code form or as a compiled + * binary, for any purpose, commercial or non-commercial, and by any + * means. + * + * In jurisdictions that recognize copyright laws, the author or authors + * of this software dedicate any and all copyright interest in the + * software to the public domain. We make this dedication for the benefit + * of the public at large and to the detriment of our heirs and + * successors. We intend this dedication to be an overt act of + * relinquishment in perpetuity of all present and future rights to this + * software under copyright law. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. + * IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + */ + +#include <algorithm> +#include <stack> + +#include "configuration.h" + +namespace test_harness { +/* Static methods implementation. */ +static bool +config_item_to_bool(const WT_CONFIG_ITEM item) +{ + return (item.val != 0); +} + +static int64_t +config_item_to_int(const WT_CONFIG_ITEM item) +{ + return (item.val); +} + +static std::string +config_item_to_string(const WT_CONFIG_ITEM item) +{ + return std::string(item.str, item.len); +} + +static std::vector<std::string> +config_item_to_list(const WT_CONFIG_ITEM item) +{ + auto str = config_item_to_string(item); + + /* Get rid of the brackets. */ + testutil_assert(!str.empty() && str.front() == '[' && str.back() == ']'); + str.pop_back(); + str.erase(0, 1); + + return (split_string(str, ',')); +} + +/* configuration class implementation. */ +configuration::configuration(const std::string &test_config_name, const std::string &config) +{ + const auto *config_entry = __wt_test_config_match(test_config_name.c_str()); + if (config_entry == nullptr) + testutil_die(EINVAL, "failed to match test config name"); + std::string default_config = std::string(config_entry->base); + /* Merge in the default configuration. */ + _config = merge_default_config(default_config, config); + logger::log_msg(LOG_INFO, "Full config: " + _config); + + int ret = + wiredtiger_test_config_validate(nullptr, nullptr, test_config_name.c_str(), _config.c_str()); + if (ret != 0) + testutil_die(EINVAL, "failed to validate given config, ensure test config exists"); + ret = wiredtiger_config_parser_open(nullptr, _config.c_str(), _config.size(), &_config_parser); + if (ret != 0) + testutil_die(EINVAL, "failed to create configuration parser for provided config"); +} + +configuration::configuration(const WT_CONFIG_ITEM &nested) +{ + if (nested.type != WT_CONFIG_ITEM::WT_CONFIG_ITEM_STRUCT) + testutil_die(EINVAL, "provided config item isn't a structure"); + int ret = wiredtiger_config_parser_open(nullptr, nested.str, nested.len, &_config_parser); + if (ret != 0) + testutil_die(EINVAL, "failed to create configuration parser for provided sub config"); +} + +configuration::~configuration() +{ + if (_config_parser != nullptr) { + _config_parser->close(_config_parser); + _config_parser = nullptr; + } +} + +const std::string & +configuration::get_config() const +{ + return (_config); +} + +std::string +configuration::get_string(const std::string &key) +{ + return get<std::string>(key, false, types::STRING, "", config_item_to_string); +} + +std::string +configuration::get_optional_string(const std::string &key, const std::string &def) +{ + return get<std::string>(key, true, types::STRING, def, config_item_to_string); +} + +bool +configuration::get_bool(const std::string &key) +{ + return get<bool>(key, false, types::BOOL, false, config_item_to_bool); +} + +bool +configuration::get_optional_bool(const std::string &key, const bool def) +{ + return get<bool>(key, true, types::BOOL, def, config_item_to_bool); +} + +int64_t +configuration::get_int(const std::string &key) +{ + return get<int64_t>(key, false, types::INT, 0, config_item_to_int); +} + +int64_t +configuration::get_optional_int(const std::string &key, const int64_t def) +{ + return get<int64_t>(key, true, types::INT, def, config_item_to_int); +} + +configuration * +configuration::get_subconfig(const std::string &key) +{ + return get<configuration *>(key, false, types::STRUCT, nullptr, + [](WT_CONFIG_ITEM item) { return new configuration(item); }); +} + +configuration * +configuration::get_optional_subconfig(const std::string &key) +{ + return get<configuration *>(key, true, types::STRUCT, nullptr, + [](WT_CONFIG_ITEM item) { return new configuration(item); }); +} + +std::vector<std::string> +configuration::get_list(const std::string &key) +{ + return get<std::vector<std::string>>(key, false, types::LIST, {}, config_item_to_list); +} + +std::vector<std::string> +configuration::get_optional_list(const std::string &key) +{ + return get<std::vector<std::string>>(key, true, types::LIST, {}, config_item_to_list); +} + +template <typename T> +T +configuration::get( + const std::string &key, bool optional, types type, T def, T (*func)(WT_CONFIG_ITEM item)) +{ + WT_DECL_RET; + WT_CONFIG_ITEM value = {"", 0, 1, WT_CONFIG_ITEM::WT_CONFIG_ITEM_BOOL}; + const char *error_msg = "Configuration value doesn't match requested type"; + + ret = _config_parser->get(_config_parser, key.c_str(), &value); + if (ret == WT_NOTFOUND && optional) + return (def); + else if (ret != 0) + testutil_die(ret, "Error while finding config"); + + if (type == types::STRING && + (value.type != WT_CONFIG_ITEM::WT_CONFIG_ITEM_STRING && + value.type != WT_CONFIG_ITEM::WT_CONFIG_ITEM_ID)) + testutil_die(-1, error_msg); + else if (type == types::BOOL && value.type != WT_CONFIG_ITEM::WT_CONFIG_ITEM_BOOL) + testutil_die(-1, error_msg); + else if (type == types::INT && value.type != WT_CONFIG_ITEM::WT_CONFIG_ITEM_NUM) + testutil_die(-1, error_msg); + else if (type == types::STRUCT && value.type != WT_CONFIG_ITEM::WT_CONFIG_ITEM_STRUCT) + testutil_die(-1, error_msg); + else if (type == types::LIST && value.type != WT_CONFIG_ITEM::WT_CONFIG_ITEM_STRUCT) + testutil_die(-1, error_msg); + + return func(value); +} + +std::string +configuration::merge_default_config( + const std::string &default_config, const std::string &user_config) +{ + std::string merged_config; + auto split_default_config = split_config(default_config); + auto split_user_config = split_config(user_config); + auto user_it = split_user_config.begin(); + for (auto default_it = split_default_config.begin(); default_it != split_default_config.end(); + ++default_it) { + if (user_it->first != default_it->first) + /* The default does not exist in the user configuration, add it. */ + merged_config += default_it->first + "=" + default_it->second; + else { + /* If we have a sub config merge it in. */ + if (user_it->second[0] == '(') + merged_config += default_it->first + "=(" + + merge_default_config(default_it->second, user_it->second) + ')'; + else + /* Add the user configuration as it exists. */ + merged_config += user_it->first + "=" + user_it->second; + ++user_it; + } + /* Add a comma after every item we add except the last one. */ + if (split_default_config.end() - default_it != 1) + merged_config += ","; + } + return (merged_config); +} + +std::vector<std::pair<std::string, std::string>> +configuration::split_config(const std::string &config) +{ + std::string cut_config = config; + std::vector<std::pair<std::string, std::string>> split_config; + std::string key = "", value = ""; + bool in_subconfig = false; + bool expect_value = false; + std::stack<char> parens; + + /* All configuration strings must be at least 2 characters. */ + testutil_assert(config.size() > 1); + + /* Remove prefix and trailing "()". */ + if (config[0] == '(') + cut_config = config.substr(1, config.size() - 2); + + size_t start = 0, len = 0; + for (size_t i = 0; i < cut_config.size(); ++i) { + if (cut_config[i] == '(' || cut_config[i] == '[') { + parens.push(cut_config[i]); + in_subconfig = true; + } + if (cut_config[i] == ')' || cut_config[i] == ']') { + parens.pop(); + in_subconfig = !parens.empty(); + } + if (cut_config[i] == '=' && !in_subconfig) { + expect_value = true; + key = cut_config.substr(start, len); + start += len + 1; + len = 0; + continue; + } + if (cut_config[i] == ',' && !in_subconfig) { + expect_value = false; + if (start + len >= cut_config.size()) + break; + value = cut_config.substr(start, len); + start += len + 1; + len = 0; + split_config.push_back(std::make_pair(key, value)); + continue; + } + ++len; + } + if (expect_value) { + value = cut_config.substr(start, len); + split_config.push_back(std::make_pair(key, value)); + } + + /* We have to sort the config here otherwise we will match incorrectly while merging. */ + std::sort(split_config.begin(), split_config.end(), comparator); + return (split_config); +} + +bool +configuration::comparator( + std::pair<std::string, std::string> a, std::pair<std::string, std::string> b) +{ + return (a.first < b.first); +} +} // namespace test_harness diff --git a/src/third_party/wiredtiger/test/cppsuite/test_harness/core/configuration.h b/src/third_party/wiredtiger/test/cppsuite/test_harness/core/configuration.h index 7eaa96214cb..f34465904ad 100644 --- a/src/third_party/wiredtiger/test/cppsuite/test_harness/core/configuration.h +++ b/src/third_party/wiredtiger/test/cppsuite/test_harness/core/configuration.h @@ -29,61 +29,43 @@ #ifndef CONFIGURATION_H #define CONFIGURATION_H -#include <algorithm> #include <string> -#include <stack> +#include <vector> + +#include "test_harness/util/logger.h" extern "C" { #include "test_util.h" } -enum class types { BOOL, INT, STRING, STRUCT }; - namespace test_harness { -class configuration { - public: - configuration(const std::string &test_config_name, const std::string &config) - { - const auto *config_entry = __wt_test_config_match(test_config_name.c_str()); - if (config_entry == nullptr) - testutil_die(EINVAL, "failed to match test config name"); - std::string default_config = std::string(config_entry->base); - /* Merge in the default configuration. */ - _config = merge_default_config(default_config, config); - debug_print("Running with enriched config: " + _config, DEBUG_INFO); - - int ret = wiredtiger_test_config_validate( - nullptr, nullptr, test_config_name.c_str(), _config.c_str()); - if (ret != 0) - testutil_die(EINVAL, "failed to validate given config, ensure test config exists"); - ret = - wiredtiger_config_parser_open(nullptr, _config.c_str(), _config.size(), &_config_parser); - if (ret != 0) - testutil_die(EINVAL, "failed to create configuration parser for provided config"); +inline std::vector<std::string> +split_string(const std::string &str, const char delim) +{ + std::vector<std::string> splits; + std::string current_str; + for (const auto c : str) { + if (c == delim) { + if (!current_str.empty()) { + splits.push_back(current_str); + current_str.clear(); + } + } else + current_str.push_back(c); } + if (!current_str.empty()) + splits.push_back(std::move(current_str)); + return (splits); +} - configuration(const WT_CONFIG_ITEM &nested) - { - if (nested.type != WT_CONFIG_ITEM::WT_CONFIG_ITEM_STRUCT) - testutil_die(EINVAL, "provided config item isn't a structure"); - int ret = wiredtiger_config_parser_open(nullptr, nested.str, nested.len, &_config_parser); - if (ret != 0) - testutil_die(EINVAL, "failed to create configuration parser for provided sub config"); - } +class configuration { + public: + configuration(const std::string &test_config_name, const std::string &config); + configuration(const WT_CONFIG_ITEM &nested); - ~configuration() - { - if (_config_parser != nullptr) { - _config_parser->close(_config_parser); - _config_parser = nullptr; - } - } + ~configuration(); - const std::string & - get_config() const - { - return (_config); - } + const std::string &get_config() const; /* * Wrapper functions for retrieving basic configuration values. Ideally tests can avoid using @@ -93,194 +75,39 @@ class configuration { * component, the optional forms of the functions can be used. In this case a default value must * be passed and it will be set to that value. */ - std::string - get_string(const std::string &key) - { - return get<std::string>(key, false, types::STRING, "", config_item_to_string); - } - - std::string - get_optional_string(const std::string &key, const std::string &def) - { - return get<std::string>(key, true, types::STRING, def, config_item_to_string); - } - - bool - get_bool(const std::string &key) - { - return get<bool>(key, false, types::BOOL, false, config_item_to_bool); - } - - bool - get_optional_bool(const std::string &key, const bool def) - { - return get<bool>(key, true, types::BOOL, def, config_item_to_bool); - } - - int64_t - get_int(const std::string &key) - { - return get<int64_t>(key, false, types::INT, 0, config_item_to_int); - } - - int64_t - get_optional_int(const std::string &key, const int64_t def) - { - return get<int64_t>(key, true, types::INT, def, config_item_to_int); - } - - configuration * - get_subconfig(const std::string &key) - { - return get<configuration *>(key, false, types::STRUCT, nullptr, - [](WT_CONFIG_ITEM item) { return new configuration(item); }); - } + std::string get_string(const std::string &key); + std::string get_optional_string(const std::string &key, const std::string &def); + bool get_bool(const std::string &key); + bool get_optional_bool(const std::string &key, const bool def); + int64_t get_int(const std::string &key); + int64_t get_optional_int(const std::string &key, const int64_t def); + configuration *get_subconfig(const std::string &key); + configuration *get_optional_subconfig(const std::string &key); + std::vector<std::string> get_list(const std::string &key); + std::vector<std::string> get_optional_list(const std::string &key); private: - static bool - config_item_to_bool(const WT_CONFIG_ITEM item) - { - return (item.val != 0); - } - - static int64_t - config_item_to_int(const WT_CONFIG_ITEM item) - { - return (item.val); - } - - static std::string - config_item_to_string(const WT_CONFIG_ITEM item) - { - return std::string(item.str, item.len); - } + enum class types { BOOL, INT, LIST, STRING, STRUCT }; template <typename T> - T - get(const std::string &key, bool optional, types type, T def, T (*func)(WT_CONFIG_ITEM item)) - { - WT_DECL_RET; - WT_CONFIG_ITEM value = {"", 0, 1, WT_CONFIG_ITEM::WT_CONFIG_ITEM_BOOL}; - const char *error_msg = "Configuration value doesn't match requested type"; - - ret = _config_parser->get(_config_parser, key.c_str(), &value); - if (ret == WT_NOTFOUND && optional) - return (def); - else if (ret != 0) - testutil_die(ret, "Error while finding config"); - - if (type == types::STRING && - (value.type != WT_CONFIG_ITEM::WT_CONFIG_ITEM_STRING && - value.type != WT_CONFIG_ITEM::WT_CONFIG_ITEM_ID)) - testutil_die(-1, error_msg); - else if (type == types::BOOL && value.type != WT_CONFIG_ITEM::WT_CONFIG_ITEM_BOOL) - testutil_die(-1, error_msg); - else if (type == types::INT && value.type != WT_CONFIG_ITEM::WT_CONFIG_ITEM_NUM) - testutil_die(-1, error_msg); - else if (type == types::STRUCT && value.type != WT_CONFIG_ITEM::WT_CONFIG_ITEM_STRUCT) - testutil_die(-1, error_msg); - - return func(value); - } + T get(const std::string &key, bool optional, types type, T def, T (*func)(WT_CONFIG_ITEM item)); /* * Merge together two configuration strings, the user one and the default one. */ - static std::string - merge_default_config(const std::string &default_config, const std::string &user_config) - { - std::string merged_config; - auto split_default_config = split_config(default_config); - auto split_user_config = split_config(user_config); - auto user_it = split_user_config.begin(); - for (auto default_it = split_default_config.begin(); - default_it != split_default_config.end(); ++default_it) { - if (user_it->first != default_it->first) - /* The default does not exist in the user configuration, add it. */ - merged_config += default_it->first + "=" + default_it->second; - else { - /* If we have a sub config merge it in. */ - if (user_it->second[0] == '(') - merged_config += default_it->first + "=(" + - merge_default_config(default_it->second, user_it->second) + ')'; - else - /* Add the user configuration as it exists. */ - merged_config += user_it->first + "=" + user_it->second; - ++user_it; - } - /* Add a comma after every item we add except the last one. */ - if (split_default_config.end() - default_it != 1) - merged_config += ","; - } - return (merged_config); - } + static std::string merge_default_config( + const std::string &default_config, const std::string &user_config); /* * Split a config string into keys and values, taking care to not split incorrectly when we have - * a sub config. + * a sub config or array. */ - static std::vector<std::pair<std::string, std::string>> - split_config(const std::string &config) - { - std::string cut_config = config; - std::vector<std::pair<std::string, std::string>> split_config; - std::string key = "", value = ""; - bool in_subconfig = false; - bool expect_value = false; - std::stack<char> subconfig_parens; + static std::vector<std::pair<std::string, std::string>> split_config(const std::string &config); - /* All configuration strings must be at least 2 characters. */ - testutil_assert(config.size() > 1); - - /* Remove prefix and trailing "()". */ - if (config[0] == '(') - cut_config = config.substr(1, config.size() - 2); - - size_t start = 0, len = 0; - for (size_t i = 0; i < cut_config.size(); ++i) { - if (cut_config[i] == '(') { - subconfig_parens.push(cut_config[i]); - in_subconfig = true; - } - if (cut_config[i] == ')') { - subconfig_parens.pop(); - in_subconfig = !subconfig_parens.empty(); - } - if (cut_config[i] == '=' && !in_subconfig) { - expect_value = true; - key = cut_config.substr(start, len); - start += len + 1; - len = 0; - continue; - } - if (cut_config[i] == ',' && !in_subconfig) { - expect_value = false; - if (start + len >= cut_config.size()) - break; - value = cut_config.substr(start, len); - start += len + 1; - len = 0; - split_config.push_back(std::make_pair(key, value)); - continue; - } - ++len; - } - if (expect_value) { - value = cut_config.substr(start, len); - split_config.push_back(std::make_pair(key, value)); - } - - /* We have to sort the config here otherwise we will match incorrectly while merging. */ - std::sort(split_config.begin(), split_config.end(), comparator); - return (split_config); - } - - static bool - comparator(std::pair<std::string, std::string> a, std::pair<std::string, std::string> b) - { - return (a.first < b.first); - } + static bool comparator( + std::pair<std::string, std::string> a, std::pair<std::string, std::string> b); + private: std::string _config; WT_CONFIG_PARSER *_config_parser = nullptr; }; diff --git a/src/third_party/wiredtiger/test/cppsuite/test_harness/core/throttle.cxx b/src/third_party/wiredtiger/test/cppsuite/test_harness/core/throttle.cxx new file mode 100644 index 00000000000..af7b94e8f98 --- /dev/null +++ b/src/third_party/wiredtiger/test/cppsuite/test_harness/core/throttle.cxx @@ -0,0 +1,75 @@ +/*- + * Public Domain 2014-present MongoDB, Inc. + * Public Domain 2008-2014 WiredTiger, Inc. + * + * This is free and unencumbered software released into the public domain. + * + * Anyone is free to copy, modify, publish, use, compile, sell, or + * distribute this software, either in source code form or as a compiled + * binary, for any purpose, commercial or non-commercial, and by any + * means. + * + * In jurisdictions that recognize copyright laws, the author or authors + * of this software dedicate any and all copyright interest in the + * software to the public domain. We make this dedication for the benefit + * of the public at large and to the detriment of our heirs and + * successors. We intend this dedication to be an overt act of + * relinquishment in perpetuity of all present and future rights to this + * software under copyright law. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. + * IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + */ + +#include <thread> + +#include "configuration.h" +#include "test_harness/util/api_const.h" +#include "throttle.h" + +namespace test_harness { +throttle::throttle(const std::string &throttle_rate) +{ + std::string magnitude; + uint64_t multiplier = 0; + /* + * Find the ms, s, or m in the string. Searching for "ms" first as the following two searches + * would match as well. + */ + size_t pos = throttle_rate.find("ms"); + if (pos != std::string::npos) + multiplier = 1; + else { + pos = throttle_rate.find("s"); + if (pos != std::string::npos) + multiplier = 1000; + else { + pos = throttle_rate.find("m"); + if (pos != std::string::npos) + multiplier = 60 * 1000; + else + testutil_die(-1, "no rate specifier given"); + } + } + magnitude = throttle_rate.substr(0, pos); + /* This will throw if it can't cast, which is fine. */ + _ms = std::stoi(magnitude) * multiplier; +} + +/* Use optional and default to 1s per op in case something doesn't define this. */ +throttle::throttle(configuration *config) : throttle(config->get_optional_string(OP_RATE, "1s")) {} + +/* Default to a second per operation. */ +throttle::throttle() : throttle("1s") {} + +void +throttle::sleep() +{ + std::this_thread::sleep_for(std::chrono::milliseconds(_ms)); +} +} // namespace test_harness diff --git a/src/third_party/wiredtiger/test/cppsuite/test_harness/core/throttle.h b/src/third_party/wiredtiger/test/cppsuite/test_harness/core/throttle.h index da5c54fb2d0..24161ea956e 100644 --- a/src/third_party/wiredtiger/test/cppsuite/test_harness/core/throttle.h +++ b/src/third_party/wiredtiger/test/cppsuite/test_harness/core/throttle.h @@ -29,52 +29,23 @@ #ifndef THROTTLE_H #define THROTTLE_H -#include <thread> #include <string> -#include "configuration.h" +/* Forward declarations for classes to reduce compilation time and modules coupling. */ +class configuration; namespace test_harness { class throttle { public: - explicit throttle(const std::string &throttle_rate) - { - std::string magnitude; - uint64_t multiplier = 0; - /* - * Find the ms, s, or m in the string. Searching for "ms" first as the following two - * searches would match as well. - */ - size_t pos = throttle_rate.find("ms"); - if (pos != std::string::npos) - multiplier = 1; - else { - pos = throttle_rate.find("s"); - if (pos != std::string::npos) - multiplier = 1000; - else { - pos = throttle_rate.find("m"); - if (pos != std::string::npos) - multiplier = 60 * 1000; - else - testutil_die(-1, "no rate specifier given"); - } - } - magnitude = throttle_rate.substr(0, pos); - /* This will throw if it can't cast, which is fine. */ - _ms = std::stoi(magnitude) * multiplier; - } + explicit throttle(const std::string &throttle_rate); - explicit throttle(configuration *config) : throttle(config->get_string(OP_RATE)) {} + /* Use optional and default to 1s per op in case something doesn't define this. */ + explicit throttle(configuration *config); /* Default to a second per operation. */ - throttle() : throttle("1s") {} + throttle(); - void - sleep() - { - std::this_thread::sleep_for(std::chrono::milliseconds(_ms)); - } + void sleep(); private: uint64_t _ms = 1000; diff --git a/src/third_party/wiredtiger/test/cppsuite/test_harness/runtime_monitor.cxx b/src/third_party/wiredtiger/test/cppsuite/test_harness/runtime_monitor.cxx new file mode 100644 index 00000000000..ebb6520b465 --- /dev/null +++ b/src/third_party/wiredtiger/test/cppsuite/test_harness/runtime_monitor.cxx @@ -0,0 +1,292 @@ +/*- + * Public Domain 2014-present MongoDB, Inc. + * Public Domain 2008-2014 WiredTiger, Inc. + * + * This is free and unencumbered software released into the public domain. + * + * Anyone is free to copy, modify, publish, use, compile, sell, or + * distribute this software, either in source code form or as a compiled + * binary, for any purpose, commercial or non-commercial, and by any + * means. + * + * In jurisdictions that recognize copyright laws, the author or authors + * of this software dedicate any and all copyright interest in the + * software to the public domain. We make this dedication for the benefit + * of the public at large and to the detriment of our heirs and + * successors. We intend this dedication to be an overt act of + * relinquishment in perpetuity of all present and future rights to this + * software under copyright law. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. + * IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + */ + +#include "connection_manager.h" +#include "core/component.h" +#include "core/configuration.h" +#include "core/throttle.h" +#include "runtime_monitor.h" +#include "util/api_const.h" +#include "util/logger.h" + +namespace test_harness { +/* Static methods implementation. */ +static void +get_stat(scoped_cursor &cursor, int stat_field, int64_t *valuep) +{ + const char *desc, *pvalue; + cursor->set_key(cursor.get(), stat_field); + testutil_check(cursor->search(cursor.get())); + testutil_check(cursor->get_value(cursor.get(), &desc, &pvalue, valuep)); + testutil_check(cursor->reset(cursor.get())); +} + +static std::string +collection_name_to_file_name(const std::string &collection_name) +{ + /* Strip out the URI prefix. */ + const size_t colon_pos = collection_name.find(':'); + testutil_assert(colon_pos != std::string::npos); + const auto stripped_name = collection_name.substr(colon_pos + 1); + + /* Now add the directory and file extension. */ + return (std::string(DEFAULT_DIR) + "/" + stripped_name + ".wt"); +} + +/* Inline methods implementation. */ +/* + * The WiredTiger configuration API doesn't accept string statistic names when retrieving statistic + * values. This function provides the required mapping to statistic id. We should consider + * generating it programmatically in `stat.py` to avoid having to manually add a condition every + * time we want to observe a new postrun statistic. + */ +inline int +get_stat_field(const std::string &name) +{ + if (name == "cache_hs_insert") + return (WT_STAT_CONN_CACHE_HS_INSERT); + else if (name == "cc_pages_removed") + return (WT_STAT_CONN_CC_PAGES_REMOVED); + testutil_die(EINVAL, "get_stat_field: Stat \"%s\" is unrecognized", name.c_str()); +} + +/* runtime_statistic class implementation */ +runtime_statistic::runtime_statistic(configuration *config) +{ + _enabled = config->get_bool(ENABLED); +} + +bool +runtime_statistic::enabled() const +{ + return (_enabled); +} + +/* cache_limit_statistic class implementation */ +cache_limit_statistic::cache_limit_statistic(configuration *config) : runtime_statistic(config) +{ + limit = config->get_int(LIMIT); +} + +void +cache_limit_statistic::check(scoped_cursor &cursor) +{ + testutil_assert(cursor.get() != nullptr); + int64_t cache_bytes_image, cache_bytes_other, cache_bytes_max; + double use_percent; + /* Three statistics are required to compute cache use percentage. */ + get_stat(cursor, WT_STAT_CONN_CACHE_BYTES_IMAGE, &cache_bytes_image); + get_stat(cursor, WT_STAT_CONN_CACHE_BYTES_OTHER, &cache_bytes_other); + get_stat(cursor, WT_STAT_CONN_CACHE_BYTES_MAX, &cache_bytes_max); + /* + * Assert that we never exceed our configured limit for cache usage. Add 0.0 to avoid floating + * point conversion errors. + */ + use_percent = ((cache_bytes_image + cache_bytes_other + 0.0) / cache_bytes_max) * 100; + if (use_percent > limit) { + const std::string error_string = + "runtime_monitor: Cache usage exceeded during test! Limit: " + std::to_string(limit) + + " usage: " + std::to_string(use_percent); + testutil_die(-1, error_string.c_str()); + } else + logger::log_msg(LOG_TRACE, "Cache usage: " + std::to_string(use_percent)); +} + +/* db_size_statistic class implementation */ +db_size_statistic::db_size_statistic(configuration *config, database &database) + : runtime_statistic(config), _database(database) +{ + _limit = config->get_int(LIMIT); +#ifdef _WIN32 + Logger::log_msg("Database size checking is not implemented on Windows", LOG_ERROR); +#endif +} + +void +db_size_statistic::check(scoped_cursor &) +{ + const auto file_names = get_file_names(); +#ifndef _WIN32 + size_t db_size = 0; + for (const auto &name : file_names) { + struct stat sb; + if (stat(name.c_str(), &sb) == 0) { + db_size += sb.st_size; + logger::log_msg(LOG_TRACE, name + " was " + std::to_string(sb.st_size) + " bytes"); + } else + /* The only good reason for this to fail is if the file hasn't been created yet. */ + testutil_assert(errno == ENOENT); + } + logger::log_msg(LOG_TRACE, "Current database size is " + std::to_string(db_size) + " bytes"); + if (db_size > _limit) { + const std::string error_string = + "runtime_monitor: Database size limit exceeded during test! Limit: " + + std::to_string(_limit) + " db size: " + std::to_string(db_size); + testutil_die(-1, error_string.c_str()); + } +#else + static_cast<void>(file_names); + static_cast<void>(_database); + static_cast<void>(_limit); +#endif +} + +std::vector<std::string> +db_size_statistic::get_file_names() +{ + std::vector<std::string> file_names; + for (const auto &name : _database.get_collection_names()) + file_names.push_back(collection_name_to_file_name(name)); + + /* Add WiredTiger internal tables. */ + file_names.push_back(std::string(DEFAULT_DIR) + "/" + WT_HS_FILE); + file_names.push_back(std::string(DEFAULT_DIR) + "/" + WT_METAFILE); + + return (file_names); +} + +/* postrun_statistic_check class implementation */ +postrun_statistic_check::postrun_statistic::postrun_statistic( + std::string &&name, const int64_t min_limit, const int64_t max_limit) + : name(std::move(name)), field(get_stat_field(this->name)), min_limit(min_limit), + max_limit(max_limit) +{ +} + +postrun_statistic_check::postrun_statistic_check(configuration *config) +{ + const auto config_stats = config->get_list(POSTRUN_STATISTICS); + /* + * Each stat in the configuration is a colon separated list in the following format: + * <stat_name>:<min_limit>:<max_limit> + */ + for (const auto &c : config_stats) { + auto stat = split_string(c, ':'); + if (stat.size() != 3) + testutil_die(EINVAL, + "runtime_monitor: Each postrun statistic must follow the format of " + "\"stat_name:min_limit:max_limit\". Invalid format \"%s\" provided.", + c.c_str()); + const int min_limit = std::stoi(stat.at(1)), max_limit = std::stoi(stat.at(2)); + if (min_limit > max_limit) + testutil_die(EINVAL, + "runtime_monitor: The min limit of each postrun statistic must be less than or " + "equal to its max limit. Config=\"%s\" Min=%ld Max=%ld", + c.c_str(), min_limit, max_limit); + _stats.emplace_back(std::move(stat.at(0)), min_limit, max_limit); + } +} + +void +postrun_statistic_check::check(scoped_cursor &cursor) const +{ + bool success = true; + for (const auto &stat : _stats) { + if (!check_stat(cursor, stat)) + success = false; + } + if (!success) + testutil_die(-1, + "runtime_monitor: One or more postrun statistics were outside of their specified " + "limits."); +} + +bool +postrun_statistic_check::check_stat(scoped_cursor &cursor, const postrun_statistic &stat) const +{ + int64_t stat_value; + + testutil_assert(cursor.get() != nullptr); + get_stat(cursor, stat.field, &stat_value); + if (stat_value < stat.min_limit || stat_value > stat.max_limit) { + const std::string error_string = "runtime_monitor: Postrun stat \"" + stat.name + + "\" was outside of the specified limits. Min=" + std::to_string(stat.min_limit) + + " Max=" + std::to_string(stat.max_limit) + " Actual=" + std::to_string(stat_value); + logger::log_msg(LOG_ERROR, error_string); + return (false); + } + logger::log_msg(LOG_INFO, + "runtime_monitor: Final value of stat " + stat.name + " is: " + std::to_string(stat_value)); + return (true); +} + +/* runtime_monitor class implementation */ +runtime_monitor::runtime_monitor(configuration *config, database &database) + : component("runtime_monitor", config), _postrun_stats(config), _database(database) +{ +} + +runtime_monitor::~runtime_monitor() +{ + for (auto &it : _stats) + delete it; + _stats.clear(); +} + +void +runtime_monitor::load() +{ + configuration *sub_config; + std::string statistic_list; + + /* Load the general component things. */ + component::load(); + + if (_enabled) { + _session = connection_manager::instance().create_session(); + + /* Open our statistic cursor. */ + _cursor = _session.open_scoped_cursor(STATISTICS_URI); + + /* Load known runtime statistics. */ + sub_config = _config->get_subconfig(STAT_CACHE_SIZE); + _stats.push_back(new cache_limit_statistic(sub_config)); + delete sub_config; + + sub_config = _config->get_subconfig(STAT_DB_SIZE); + _stats.push_back(new db_size_statistic(sub_config, _database)); + delete sub_config; + } +} + +void +runtime_monitor::do_work() +{ + for (const auto &it : _stats) { + if (it->enabled()) + it->check(_cursor); + } +} + +void +runtime_monitor::finish() +{ + _postrun_stats.check(_cursor); + component::finish(); +} +} // namespace test_harness diff --git a/src/third_party/wiredtiger/test/cppsuite/test_harness/runtime_monitor.h b/src/third_party/wiredtiger/test/cppsuite/test_harness/runtime_monitor.h index 90aecc9aa59..e7e69302d1d 100644 --- a/src/third_party/wiredtiger/test/cppsuite/test_harness/runtime_monitor.h +++ b/src/third_party/wiredtiger/test/cppsuite/test_harness/runtime_monitor.h @@ -29,220 +29,109 @@ #ifndef RUNTIME_MONITOR_H #define RUNTIME_MONITOR_H +#include <string> +#include <vector> + extern "C" { +#include "test_util.h" #include "wiredtiger.h" } -#include "util/debug_utils.h" -#include "util/api_const.h" -#include "core/component.h" -#include "core/throttle.h" -#include "connection_manager.h" -#include "workload/database_operation.h" +#include "util/scoped_types.h" +#include "workload/database_model.h" + +/* Forward declarations for classes to reduce compilation time and modules coupling. */ +class configuration; namespace test_harness { -/* Static statistic get function. */ -static void -get_stat(WT_CURSOR *cursor, int stat_field, int64_t *valuep) -{ - const char *desc, *pvalue; - cursor->set_key(cursor, stat_field); - testutil_check(cursor->search(cursor)); - testutil_check(cursor->get_value(cursor, &desc, &pvalue, valuep)); -} -class statistic { +class runtime_statistic { public: - explicit statistic(configuration *config) - { - _enabled = config->get_bool(ENABLED); - } + explicit runtime_statistic(configuration *config); /* Check that the given statistic is within bounds. */ - virtual void check(WT_CURSOR *cursor) = 0; + virtual void check(scoped_cursor &cursor) = 0; /* Suppress warning about destructor being non-virtual. */ - virtual ~statistic() {} + virtual ~runtime_statistic() {} - bool - enabled() const - { - return _enabled; - } + bool enabled() const; protected: bool _enabled = false; }; -class cache_limit_statistic : public statistic { +class cache_limit_statistic : public runtime_statistic { public: - explicit cache_limit_statistic(configuration *config) : statistic(config) - { - limit = config->get_int(LIMIT); - } - - void - check(WT_CURSOR *cursor) override final - { - testutil_assert(cursor != nullptr); - int64_t cache_bytes_image, cache_bytes_other, cache_bytes_max; - double use_percent; - /* Three statistics are required to compute cache use percentage. */ - get_stat(cursor, WT_STAT_CONN_CACHE_BYTES_IMAGE, &cache_bytes_image); - get_stat(cursor, WT_STAT_CONN_CACHE_BYTES_OTHER, &cache_bytes_other); - get_stat(cursor, WT_STAT_CONN_CACHE_BYTES_MAX, &cache_bytes_max); - /* - * Assert that we never exceed our configured limit for cache usage. Add 0.0 to avoid - * floating point conversion errors. - */ - use_percent = ((cache_bytes_image + cache_bytes_other + 0.0) / cache_bytes_max) * 100; - if (use_percent > limit) { - const std::string error_string = - "runtime_monitor: Cache usage exceeded during test! Limit: " + std::to_string(limit) + - " usage: " + std::to_string(use_percent); - testutil_die(-1, error_string.c_str()); - } else - debug_print("Cache usage: " + std::to_string(use_percent), DEBUG_TRACE); - } + explicit cache_limit_statistic(configuration *config); + + void check(scoped_cursor &cursor) override final; private: int64_t limit; }; -static std::string -collection_name_to_file_name(const std::string &collection_name) -{ - /* Strip out the URI prefix. */ - const size_t colon_pos = collection_name.find(':'); - testutil_assert(colon_pos != std::string::npos); - const auto stripped_name = collection_name.substr(colon_pos + 1); - - /* Now add the directory and file extension. */ - return std::string(DEFAULT_DIR) + "/" + stripped_name + ".wt"; -} - -class db_size_statistic : public statistic { +class db_size_statistic : public runtime_statistic { public: - db_size_statistic(configuration *config, database &database) - : statistic(config), _database(database) - { - _limit = config->get_int(LIMIT); -#ifdef _WIN32 - debug_print("Database size checking is not implemented on Windows", DEBUG_ERROR); -#endif - } + db_size_statistic(configuration *config, database &database); virtual ~db_size_statistic() = default; /* Don't need the stat cursor for this. */ - void - check(WT_CURSOR *) override final - { - const auto file_names = get_file_names(); -#ifndef _WIN32 - size_t db_size = 0; - for (const auto &name : file_names) { - struct stat sb; - if (stat(name.c_str(), &sb) == 0) { - db_size += sb.st_size; - debug_print(name + " was " + std::to_string(sb.st_size) + " bytes", DEBUG_TRACE); - } else - /* The only good reason for this to fail is if the file hasn't been created yet. */ - testutil_assert(errno == ENOENT); - } - debug_print("Current database size is " + std::to_string(db_size) + " bytes", DEBUG_TRACE); - if (db_size > _limit) { - const std::string error_string = - "runtime_monitor: Database size limit exceeded during test! Limit: " + - std::to_string(_limit) + " db size: " + std::to_string(db_size); - testutil_die(-1, error_string.c_str()); - } -#else - static_cast<void>(file_names); - static_cast<void>(_database); - static_cast<void>(_limit); -#endif - } + void check(scoped_cursor &) override final; private: - std::vector<std::string> - get_file_names() - { - std::vector<std::string> file_names; - for (const auto &name : _database.get_collection_names()) - file_names.push_back(collection_name_to_file_name(name)); - - /* Add WiredTiger internal tables. */ - file_names.push_back(std::string(DEFAULT_DIR) + "/" + WT_HS_FILE); - file_names.push_back(std::string(DEFAULT_DIR) + "/" + WT_METAFILE); - - return file_names; - } + std::vector<std::string> get_file_names(); + private: database &_database; int64_t _limit; }; +class postrun_statistic_check { + public: + explicit postrun_statistic_check(configuration *config); + virtual ~postrun_statistic_check() = default; + + void check(scoped_cursor &cursor) const; + + private: + struct postrun_statistic { + postrun_statistic(std::string &&name, const int64_t min_limit, const int64_t max_limit); + + const std::string name; + const int field; + const int64_t min_limit, max_limit; + }; + + private: + bool check_stat(scoped_cursor &cursor, const postrun_statistic &stat) const; + + private: + std::vector<postrun_statistic> _stats; +}; + /* * The runtime monitor class is designed to track various statistics or other runtime signals * relevant to the given workload. */ class runtime_monitor : public component { public: - runtime_monitor(configuration *config, database &database) - : component("runtime_monitor", config), _database(database) - { - } - - ~runtime_monitor() - { - for (auto &it : _stats) - delete it; - _stats.clear(); - } + runtime_monitor(configuration *config, database &database); + ~runtime_monitor(); /* Delete the copy constructor and the assignment operator. */ runtime_monitor(const runtime_monitor &) = delete; runtime_monitor &operator=(const runtime_monitor &) = delete; - void - load() override final - { - configuration *sub_config; - std::string statistic_list; - - /* Load the general component things. */ - component::load(); - - if (_enabled) { - _session = connection_manager::instance().create_session(); - - /* Open our statistic cursor. */ - _session->open_cursor(_session, STATISTICS_URI, nullptr, nullptr, &_cursor); - - /* Load known statistics. */ - sub_config = _config->get_subconfig(STAT_CACHE_SIZE); - _stats.push_back(new cache_limit_statistic(sub_config)); - delete sub_config; - - sub_config = _config->get_subconfig(STAT_DB_SIZE); - _stats.push_back(new db_size_statistic(sub_config, _database)); - delete sub_config; - } - } - - void - do_work() override final - { - for (const auto &it : _stats) { - if (it->enabled()) - it->check(_cursor); - } - } + void load() override final; + void do_work() override final; + void finish() override final; private: - WT_CURSOR *_cursor = nullptr; - WT_SESSION *_session = nullptr; - std::vector<statistic *> _stats; + scoped_session _session; + scoped_cursor _cursor; + std::vector<runtime_statistic *> _stats; + postrun_statistic_check _postrun_stats; database &_database; }; } // namespace test_harness diff --git a/src/third_party/wiredtiger/test/cppsuite/test_harness/test.cxx b/src/third_party/wiredtiger/test/cppsuite/test_harness/test.cxx new file mode 100644 index 00000000000..e49590da7d7 --- /dev/null +++ b/src/third_party/wiredtiger/test/cppsuite/test_harness/test.cxx @@ -0,0 +1,187 @@ +/*- + * Public Domain 2014-present MongoDB, Inc. + * Public Domain 2008-2014 WiredTiger, Inc. + * + * This is free and unencumbered software released into the public domain. + * + * Anyone is free to copy, modify, publish, use, compile, sell, or + * distribute this software, either in source code form or as a compiled + * binary, for any purpose, commercial or non-commercial, and by any + * means. + * + * In jurisdictions that recognize copyright laws, the author or authors + * of this software dedicate any and all copyright interest in the + * software to the public domain. We make this dedication for the benefit + * of the public at large and to the detriment of our heirs and + * successors. We intend this dedication to be an overt act of + * relinquishment in perpetuity of all present and future rights to this + * software under copyright law. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. + * IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + */ + +/* Required to build using older versions of g++. */ +#include <cinttypes> +#include <mutex> + +#include "connection_manager.h" +#include "core/component.h" +#include "core/configuration.h" +#include "test.h" +#include "thread_manager.h" +#include "timestamp_manager.h" +#include "workload/workload_validation.h" +#include "util/api_const.h" + +namespace test_harness { +test::test(const test_args &args) : _args(args) +{ + _config = new configuration(args.test_name, args.test_config); + _checkpoint_manager = new checkpoint_manager(_config->get_subconfig(CHECKPOINT_MANAGER)); + _runtime_monitor = new runtime_monitor(_config->get_subconfig(RUNTIME_MONITOR), _database); + _timestamp_manager = new timestamp_manager(_config->get_subconfig(TIMESTAMP_MANAGER)); + _workload_tracking = new workload_tracking(_config->get_subconfig(WORKLOAD_TRACKING), + OPERATION_TRACKING_TABLE_CONFIG, TABLE_OPERATION_TRACKING, SCHEMA_TRACKING_TABLE_CONFIG, + TABLE_SCHEMA_TRACKING, _config->get_bool(COMPRESSION_ENABLED), *_timestamp_manager); + _workload_generator = new workload_generator(_config->get_subconfig(WORKLOAD_GENERATOR), this, + _timestamp_manager, _workload_tracking, _database); + _thread_manager = new thread_manager(); + + _database.set_timestamp_manager(_timestamp_manager); + _database.set_workload_tracking(_workload_tracking); + _database.set_create_config(_config->get_bool(COMPRESSION_ENABLED)); + + /* + * Ordering is not important here, any dependencies between components should be resolved + * internally by the components. + */ + _components = {_workload_tracking, _workload_generator, _timestamp_manager, _runtime_monitor, + _checkpoint_manager}; +} + +test::~test() +{ + delete _config; + delete _checkpoint_manager; + delete _runtime_monitor; + delete _timestamp_manager; + delete _thread_manager; + delete _workload_generator; + delete _workload_tracking; + _config = nullptr; + _checkpoint_manager = nullptr; + _runtime_monitor = nullptr; + _timestamp_manager = nullptr; + _thread_manager = nullptr; + _workload_generator = nullptr; + _workload_tracking = nullptr; + + _components.clear(); +} + +void +test::run() +{ + int64_t cache_size_mb, duration_seconds; + bool enable_logging, statistics_logging; + configuration *statistics_config; + std::string statistics_type; + /* Build the database creation config string. */ + std::string db_create_config = CONNECTION_CREATE; + + /* Enable snappy compression if required. */ + if (_config->get_bool(COMPRESSION_ENABLED)) + db_create_config += SNAPPY_EXT; + + /* Get the cache size. */ + cache_size_mb = _config->get_int(CACHE_SIZE_MB); + db_create_config += ",cache_size=" + std::to_string(cache_size_mb) + "MB"; + + /* Get the statistics configuration for this run. */ + statistics_config = _config->get_subconfig(STATISTICS_CONFIG); + statistics_type = statistics_config->get_string(TYPE); + statistics_logging = statistics_config->get_bool(ENABLE_LOGGING); + db_create_config += statistics_logging ? "," + std::string(STATISTICS_LOG) : ""; + db_create_config += ",statistics=(" + statistics_type + ")"; + /* Don't forget to delete. */ + delete statistics_config; + + /* Enable or disable write ahead logging. */ + enable_logging = _config->get_bool(ENABLE_LOGGING); + db_create_config += ",log=(enabled=" + std::string(enable_logging ? "true" : "false") + ")"; + + /* Add the user supplied wiredtiger open config. */ + db_create_config += _args.wt_open_config; + + /* Set up the test environment. */ + connection_manager::instance().create(db_create_config); + + /* Initiate the load stage of each component. */ + for (const auto &it : _components) + it->load(); + + /* Spawn threads for all component::run() functions. */ + for (const auto &it : _components) + _thread_manager->add_thread(&component::run, it); + + /* The initial population phase needs to be finished before starting the actual test. */ + while (_workload_generator->enabled() && !_workload_generator->db_populated()) + std::this_thread::sleep_for(std::chrono::milliseconds(10)); + + /* The test will run for the duration as defined in the config. */ + duration_seconds = _config->get_int(DURATION_SECONDS); + testutil_assert(duration_seconds >= 0); + logger::log_msg( + LOG_INFO, "Waiting {" + std::to_string(duration_seconds) + "} for testing to complete."); + std::this_thread::sleep_for(std::chrono::seconds(duration_seconds)); + + /* End the test by calling finish on all known components. */ + for (const auto &it : _components) + it->finish(); + + logger::log_msg(LOG_INFO, + "Joining all component threads.\n This could take a while as we need to wait" + " for all components to finish their current loop."); + _thread_manager->join(); + + /* Validation stage. */ + if (_workload_tracking->enabled()) { + workload_validation wv; + wv.validate(_workload_tracking->get_operation_table_name(), + _workload_tracking->get_schema_table_name(), + _workload_generator->get_database().get_collection_ids()); + } + + logger::log_msg(LOG_INFO, "SUCCESS"); +} + +workload_generator * +test::get_workload_generator() +{ + return (_workload_generator); +} + +runtime_monitor * +test::get_runtime_monitor() +{ + return (_runtime_monitor); +} + +timestamp_manager * +test::get_timestamp_manager() +{ + return (_timestamp_manager); +} + +thread_manager * +test::get_thread_manager() +{ + return (_thread_manager); +} +} // namespace test_harness diff --git a/src/third_party/wiredtiger/test/cppsuite/test_harness/test.h b/src/third_party/wiredtiger/test/cppsuite/test_harness/test.h index 2434704f6f9..875255e5969 100644 --- a/src/third_party/wiredtiger/test/cppsuite/test_harness/test.h +++ b/src/third_party/wiredtiger/test/cppsuite/test_harness/test.h @@ -29,71 +29,38 @@ #ifndef TEST_H #define TEST_H -/* Required to build using older versions of g++. */ -#include <cinttypes> #include <vector> -#include <mutex> +#include <string> extern "C" { #include "wiredtiger.h" } -#include "util/api_const.h" -#include "core/component.h" -#include "core/configuration.h" #include "checkpoint_manager.h" #include "connection_manager.h" #include "runtime_monitor.h" -#include "timestamp_manager.h" -#include "thread_manager.h" +#include "workload/database_operation.h" #include "workload_generator.h" -#include "workload/workload_validation.h" namespace test_harness { +class test_args { + public: + test_args(const std::string &config, const std::string &name, const std::string &wt_open_config) + : test_config(config), test_name(name), wt_open_config(wt_open_config) + { + } + const std::string test_config; + const std::string test_name; + const std::string wt_open_config; +}; + /* * The base class for a test, the standard usage pattern is to just call run(). */ class test : public database_operation { public: - test(const std::string &config, const std::string &name) - { - _config = new configuration(name, config); - _checkpoint_manager = new checkpoint_manager(_config->get_subconfig(CHECKPOINT_MANAGER)); - _runtime_monitor = new runtime_monitor(_config->get_subconfig(RUNTIME_MONITOR), _database); - _timestamp_manager = new timestamp_manager(_config->get_subconfig(TIMESTAMP_MANAGER)); - _workload_tracking = new workload_tracking(_config->get_subconfig(WORKLOAD_TRACKING), - OPERATION_TRACKING_TABLE_CONFIG, TABLE_OPERATION_TRACKING, SCHEMA_TRACKING_TABLE_CONFIG, - TABLE_SCHEMA_TRACKING); - _workload_generator = new workload_generator(_config->get_subconfig(WORKLOAD_GENERATOR), - this, _timestamp_manager, _workload_tracking, _database); - _thread_manager = new thread_manager(); - /* - * Ordering is not important here, any dependencies between components should be resolved - * internally by the components. - */ - _components = {_workload_tracking, _workload_generator, _timestamp_manager, - _runtime_monitor, _checkpoint_manager}; - } - - ~test() - { - delete _config; - delete _checkpoint_manager; - delete _runtime_monitor; - delete _timestamp_manager; - delete _thread_manager; - delete _workload_generator; - delete _workload_tracking; - _config = nullptr; - _checkpoint_manager = nullptr; - _runtime_monitor = nullptr; - _timestamp_manager = nullptr; - _thread_manager = nullptr; - _workload_generator = nullptr; - _workload_tracking = nullptr; - - _components.clear(); - } + test(const test_args &args); + ~test(); /* Delete the copy constructor and the assignment operator. */ test(const test &) = delete; @@ -102,87 +69,19 @@ class test : public database_operation { /* * The primary run function that most tests will be able to utilize without much other code. */ - virtual void - run() - { - int64_t cache_size_mb, duration_seconds; - bool enable_logging; - - /* Build the database creation config string. */ - std::string db_create_config = CONNECTION_CREATE; - - /* Get the cache size, and turn logging on or off. */ - cache_size_mb = _config->get_int(CACHE_SIZE_MB); - db_create_config += ",statistics=(fast),cache_size=" + std::to_string(cache_size_mb) + "MB"; - enable_logging = _config->get_bool(ENABLE_LOGGING); - db_create_config += ",log=(enabled=" + std::string(enable_logging ? "true" : "false") + ")"; - - /* Set up the test environment. */ - connection_manager::instance().create(db_create_config); - - /* Initiate the load stage of each component. */ - for (const auto &it : _components) - it->load(); - - /* Spawn threads for all component::run() functions. */ - for (const auto &it : _components) - _thread_manager->add_thread(&component::run, it); - - /* The initial population phase needs to be finished before starting the actual test. */ - while (_workload_generator->enabled() && !_workload_generator->db_populated()) - std::this_thread::sleep_for(std::chrono::milliseconds(10)); - - /* The test will run for the duration as defined in the config. */ - duration_seconds = _config->get_int(DURATION_SECONDS); - testutil_assert(duration_seconds >= 0); - std::this_thread::sleep_for(std::chrono::seconds(duration_seconds)); - - /* End the test by calling finish on all known components. */ - for (const auto &it : _components) - it->finish(); - _thread_manager->join(); - - /* Validation stage. */ - if (_workload_tracking->enabled()) { - workload_validation wv; - wv.validate(_workload_tracking->get_operation_table_name(), - _workload_tracking->get_schema_table_name(), _workload_generator->get_database()); - } - - debug_print("SUCCESS", DEBUG_INFO); - connection_manager::instance().close(); - } + virtual void run(); /* * Getters for all the major components, used if a test wants more control over the test * program. */ - workload_generator * - get_workload_generator() - { - return _workload_generator; - } - - runtime_monitor * - get_runtime_monitor() - { - return _runtime_monitor; - } - - timestamp_manager * - get_timestamp_manager() - { - return _timestamp_manager; - } - - thread_manager * - get_thread_manager() - { - return _thread_manager; - } + workload_generator *get_workload_generator(); + runtime_monitor *get_runtime_monitor(); + timestamp_manager *get_timestamp_manager(); + thread_manager *get_thread_manager(); private: - std::string _name; + const test_args &_args; std::vector<component *> _components; configuration *_config; checkpoint_manager *_checkpoint_manager = nullptr; @@ -191,6 +90,18 @@ class test : public database_operation { timestamp_manager *_timestamp_manager = nullptr; workload_generator *_workload_generator = nullptr; workload_tracking *_workload_tracking = nullptr; + /* + * FIX-ME-Test-Framework: We can't put this code in the destructor of `test` since it will run + * before the destructors of each of our members (meaning that sessions will get closed after + * the connection gets closed). To work around this, we've added a member with a destructor that + * closes the connection. + */ + struct connection_closer { + ~connection_closer() + { + connection_manager::instance().close(); + } + } _connection_closer; database _database; }; } // namespace test_harness diff --git a/src/third_party/wiredtiger/test/cppsuite/test_harness/thread_manager.cxx b/src/third_party/wiredtiger/test/cppsuite/test_harness/thread_manager.cxx new file mode 100644 index 00000000000..29d1e547151 --- /dev/null +++ b/src/third_party/wiredtiger/test/cppsuite/test_harness/thread_manager.cxx @@ -0,0 +1,59 @@ +/*- + * Public Domain 2014-present MongoDB, Inc. + * Public Domain 2008-2014 WiredTiger, Inc. + * + * This is free and unencumbered software released into the public domain. + * + * Anyone is free to copy, modify, publish, use, compile, sell, or + * distribute this software, either in source code form or as a compiled + * binary, for any purpose, commercial or non-commercial, and by any + * means. + * + * In jurisdictions that recognize copyright laws, the author or authors + * of this software dedicate any and all copyright interest in the + * software to the public domain. We make this dedication for the benefit + * of the public at large and to the detriment of our heirs and + * successors. We intend this dedication to be an overt act of + * relinquishment in perpetuity of all present and future rights to this + * software under copyright law. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. + * IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + */ + +#include "test_harness/util/logger.h" +#include "thread_manager.h" + +namespace test_harness { +thread_manager::~thread_manager() +{ + for (auto &it : _workers) { + if (it != nullptr && it->joinable()) { + logger::log_msg(LOG_ERROR, "You should've called join on the thread manager"); + it->join(); + } + delete it; + it = nullptr; + } + _workers.clear(); +} + +void +thread_manager::join() +{ + for (const auto &it : _workers) { + while (!it->joinable()) { + /* Helpful for diagnosing hangs. */ + logger::log_msg(LOG_TRACE, "Thread manager: Waiting to join."); + /* Check every so often to avoid spamming the log. */ + std::this_thread::sleep_for(std::chrono::milliseconds(100)); + } + it->join(); + } +} +} // namespace test_harness diff --git a/src/third_party/wiredtiger/test/cppsuite/test_harness/thread_manager.h b/src/third_party/wiredtiger/test/cppsuite/test_harness/thread_manager.h index d15d935584a..0fce6bfeb45 100644 --- a/src/third_party/wiredtiger/test/cppsuite/test_harness/thread_manager.h +++ b/src/third_party/wiredtiger/test/cppsuite/test_harness/thread_manager.h @@ -30,23 +30,13 @@ #define THREAD_MANAGER_H #include <thread> +#include <vector> namespace test_harness { /* Class that handles threads, from their initialization to their deletion. */ class thread_manager { public: - ~thread_manager() - { - for (auto &it : _workers) { - if (it != nullptr && it->joinable()) { - debug_print("You should've called join on the thread manager", DEBUG_ERROR); - it->join(); - } - delete it; - it = nullptr; - } - _workers.clear(); - } + ~thread_manager(); /* * Generic function to create threads that call member function of classes. @@ -55,22 +45,14 @@ class thread_manager { void add_thread(Callable &&fct, Args &&... args) { - std::thread *t = new std::thread(fct, args...); + std::thread *t = new std::thread(fct, std::forward<Args>(args)...); _workers.push_back(t); } /* * Complete the operations for all threads. */ - void - join() - { - for (const auto &it : _workers) { - while (!it->joinable()) { - } - it->join(); - } - } + void join(); private: std::vector<std::thread *> _workers; diff --git a/src/third_party/wiredtiger/test/cppsuite/test_harness/timestamp_manager.cxx b/src/third_party/wiredtiger/test/cppsuite/test_harness/timestamp_manager.cxx new file mode 100644 index 00000000000..7cc113f8ad8 --- /dev/null +++ b/src/third_party/wiredtiger/test/cppsuite/test_harness/timestamp_manager.cxx @@ -0,0 +1,138 @@ +/*- + * Public Domain 2014-present MongoDB, Inc. + * Public Domain 2008-2014 WiredTiger, Inc. + * + * This is free and unencumbered software released into the public domain. + * + * Anyone is free to copy, modify, publish, use, compile, sell, or + * distribute this software, either in source code form or as a compiled + * binary, for any purpose, commercial or non-commercial, and by any + * means. + * + * In jurisdictions that recognize copyright laws, the author or authors + * of this software dedicate any and all copyright interest in the + * software to the public domain. We make this dedication for the benefit + * of the public at large and to the detriment of our heirs and + * successors. We intend this dedication to be an overt act of + * relinquishment in perpetuity of all present and future rights to this + * software under copyright law. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. + * IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + */ + +#include <chrono> +#include <sstream> +#include <thread> + +#include "connection_manager.h" +#include "core/configuration.h" +#include "timestamp_manager.h" +#include "util/api_const.h" + +namespace test_harness { +const std::string +timestamp_manager::decimal_to_hex(uint64_t value) +{ + std::stringstream ss; + ss << std::hex << value; + std::string res(ss.str()); + return (res); +} + +timestamp_manager::timestamp_manager(configuration *config) : component("timestamp_manager", config) +{ +} + +void +timestamp_manager::load() +{ + component::load(); + int64_t oldest_lag = _config->get_int(OLDEST_LAG); + testutil_assert(oldest_lag >= 0); + /* Cast and then shift left to match the seconds position. */ + _oldest_lag = oldest_lag; + _oldest_lag <<= 32; + + int64_t stable_lag = _config->get_int(STABLE_LAG); + testutil_assert(stable_lag >= 0); + /* Cast and then shift left to match the seconds position. */ + _stable_lag = stable_lag; + _stable_lag <<= 32; +} + +void +timestamp_manager::do_work() +{ + std::string config; + /* latest_ts_s represents the time component of the latest timestamp provided. */ + wt_timestamp_t latest_ts_s; + + /* Timestamps are checked periodically. */ + latest_ts_s = get_time_now_s(); + + /* + * Keep a time window between the latest and stable ts less than the max defined in the + * configuration. + */ + testutil_assert(latest_ts_s >= _stable_ts); + if ((latest_ts_s - _stable_ts) > _stable_lag) { + logger::log_msg(LOG_INFO, "Timestamp_manager: Stable timestamp expired."); + _stable_ts = latest_ts_s; + config += std::string(STABLE_TS) + "=" + decimal_to_hex(_stable_ts); + } + + /* + * Keep a time window between the stable and oldest ts less than the max defined in the + * configuration. + */ + wt_timestamp_t new_oldest_ts = _oldest_ts; + testutil_assert(_stable_ts >= _oldest_ts); + if ((_stable_ts - _oldest_ts) > _oldest_lag) { + logger::log_msg(LOG_INFO, "Timestamp_manager: Oldest timestamp expired."); + new_oldest_ts = _stable_ts - _oldest_lag; + if (!config.empty()) + config += ","; + config += std::string(OLDEST_TS) + "=" + decimal_to_hex(new_oldest_ts); + } + + /* + * Save the new timestamps. Any timestamps that we're viewing from another thread should be set + * AFTER we've saved the new timestamps to avoid races where we sweep data that is not yet + * obsolete. + */ + if (!config.empty()) { + connection_manager::instance().set_timestamp(config); + _oldest_ts = new_oldest_ts; + } +} + +wt_timestamp_t +timestamp_manager::get_next_ts() +{ + uint64_t current_time = get_time_now_s(); + uint64_t increment = _increment_ts.fetch_add(1); + current_time |= (increment & 0x00000000FFFFFFFF); + return (current_time); +} + +wt_timestamp_t +timestamp_manager::get_oldest_ts() const +{ + return (_oldest_ts); +} + +uint64_t +timestamp_manager::get_time_now_s() const +{ + auto now = std::chrono::system_clock::now().time_since_epoch(); + uint64_t current_time_s = + static_cast<uint64_t>(std::chrono::duration_cast<std::chrono::seconds>(now).count()) << 32; + return (current_time_s); +} +} // namespace test_harness diff --git a/src/third_party/wiredtiger/test/cppsuite/test_harness/timestamp_manager.h b/src/third_party/wiredtiger/test/cppsuite/test_harness/timestamp_manager.h index adc5b734c05..e510614d077 100644 --- a/src/third_party/wiredtiger/test/cppsuite/test_harness/timestamp_manager.h +++ b/src/third_party/wiredtiger/test/cppsuite/test_harness/timestamp_manager.h @@ -30,114 +30,52 @@ #define TIMESTAMP_MANAGER_H #include <atomic> -#include <chrono> -#include <sstream> -#include <thread> +#include <string> #include "core/component.h" +/* Forward declarations for classes to reduce compilation time and modules coupling. */ +class configuration; + namespace test_harness { /* * The timestamp monitor class manages global timestamp state for all components in the test - * harness. It also manages the global timestamps within WiredTiger. All timestamps are in seconds - * unless specified otherwise. + * harness. It also manages the global timestamps within WiredTiger. + * + * The format of a timestamp is as follows, the first 32 bits represent the epoch time in seconds. + * The last 32 bits represent an increment for uniqueness. */ class timestamp_manager : public component { public: - timestamp_manager(configuration *config) : component("timestamp_manager", config) {} - - void - load() override final - { - component::load(); - - _oldest_lag = _config->get_int(OLDEST_LAG); - testutil_assert(_oldest_lag >= 0); - _stable_lag = _config->get_int(STABLE_LAG); - testutil_assert(_stable_lag >= 0); - } - - void - do_work() override final - { - std::string config; - /* latest_ts_s represents the time component of the latest timestamp provided. */ - wt_timestamp_t latest_ts_s; + static const std::string decimal_to_hex(uint64_t value); - /* Timestamps are checked periodically. */ - latest_ts_s = (_latest_ts >> 32); - /* - * Keep a time window between the latest and stable ts less than the max defined in the - * configuration. - */ - testutil_assert(latest_ts_s >= _stable_ts); - if ((latest_ts_s - _stable_ts) > _stable_lag) { - _stable_ts = latest_ts_s - _stable_lag; - config += std::string(STABLE_TS) + "=" + decimal_to_hex(_stable_ts); - } - - /* - * Keep a time window between the stable and oldest ts less than the max defined in the - * configuration. - */ - testutil_assert(_stable_ts >= _oldest_ts); - if ((_stable_ts - _oldest_ts) > _oldest_lag) { - _oldest_ts = _stable_ts - _oldest_lag; - if (!config.empty()) - config += ","; - config += std::string(OLDEST_TS) + "=" + decimal_to_hex(_oldest_ts); - } + public: + timestamp_manager(configuration *config); - /* Save the new timestamps. */ - if (!config.empty()) { - connection_manager::instance().set_timestamp(config); - config = ""; - } - } + void load() override final; + void do_work() override final; /* - * Get a unique commit timestamp. The first 32 bits represent the epoch time in seconds. The - * last 32 bits represent an increment for uniqueness. + * Get a unique timestamp. */ - wt_timestamp_t - get_next_ts() - { - uint64_t current_time = get_time_now_s(); - _increment_ts.fetch_add(1); - - current_time = (current_time << 32) | (_increment_ts & 0x00000000FFFFFFFF); - _latest_ts = current_time; - - return (_latest_ts); - } + wt_timestamp_t get_next_ts(); - static const std::string - decimal_to_hex(int64_t value) - { - std::stringstream ss; - ss << std::hex << value; - std::string res(ss.str()); - return (res); - } + wt_timestamp_t get_oldest_ts() const; private: - uint64_t - get_time_now_s() const - { - auto now = std::chrono::system_clock::now().time_since_epoch(); - uint64_t current_time_s = - static_cast<uint64_t>(std::chrono::duration_cast<std::chrono::seconds>(now).count()); - return (current_time_s); - } + /* Get the current time in seconds, bit shifted to the expected location. */ + uint64_t get_time_now_s() const; private: - std::atomic<wt_timestamp_t> _increment_ts; - wt_timestamp_t _latest_ts = 0U, _oldest_ts = 0U, _stable_ts = 0U; + std::atomic<wt_timestamp_t> _increment_ts{0}; + /* The tracking table sweep needs to read the oldest timestamp. */ + std::atomic<wt_timestamp_t> _oldest_ts{0U}; + wt_timestamp_t _stable_ts = 0U; /* * _oldest_lag is the time window between the stable and oldest timestamps. * _stable_lag is the time window between the latest and stable timestamps. */ - int64_t _oldest_lag = 0, _stable_lag = 0; + uint64_t _oldest_lag = 0U, _stable_lag = 0U; }; } // namespace test_harness diff --git a/src/third_party/wiredtiger/test/cppsuite/test_harness/util/api_const.h b/src/third_party/wiredtiger/test/cppsuite/test_harness/util/api_const.h index 146e98e38d6..1eef096aa29 100644 --- a/src/third_party/wiredtiger/test/cppsuite/test_harness/util/api_const.h +++ b/src/third_party/wiredtiger/test/cppsuite/test_harness/util/api_const.h @@ -42,12 +42,12 @@ static const char *WORKLOAD_TRACKING = "workload_tracking"; /* Configuration API consts. */ static const char *CACHE_SIZE_MB = "cache_size_mb"; static const char *COLLECTION_COUNT = "collection_count"; +static const char *COMPRESSION_ENABLED = "compression_enabled"; static const char *DURATION_SECONDS = "duration_seconds"; static const char *ENABLED = "enabled"; static const char *ENABLE_LOGGING = "enable_logging"; -static const char *INTERVAL = "interval"; static const char *INSERT_CONFIG = "insert_config"; -static const char *KEY_COUNT = "key_count"; +static const char *KEY_COUNT_PER_COLLECTION = "key_count_per_collection"; static const char *KEY_SIZE = "key_size"; static const char *LIMIT = "limit"; static const char *MAX = "max"; @@ -55,11 +55,15 @@ static const char *MIN = "min"; static const char *OLDEST_LAG = "oldest_lag"; static const char *OP_RATE = "op_rate"; static const char *OPS_PER_TRANSACTION = "ops_per_transaction"; +static const char *POPULATE_CONFIG = "populate_config"; +static const char *POSTRUN_STATISTICS = "postrun_statistics"; static const char *READ_CONFIG = "read_config"; static const char *STABLE_LAG = "stable_lag"; static const char *STAT_CACHE_SIZE = "stat_cache_size"; static const char *STAT_DB_SIZE = "stat_db_size"; +static const char *STATISTICS_CONFIG = "statistics_config"; static const char *THREAD_COUNT = "thread_count"; +static const char *TYPE = "type"; static const char *UPDATE_CONFIG = "update_config"; static const char *VALUE_SIZE = "value_size"; @@ -68,9 +72,23 @@ static const char *COMMIT_TS = "commit_timestamp"; static const char *CONNECTION_CREATE = "create"; static const char *OLDEST_TS = "oldest_timestamp"; static const char *STABLE_TS = "stable_timestamp"; +static const char *STATISTICS_LOG = "statistics_log=(json,wait=1)"; + +/* + * Use the Snappy compressor for stress testing to avoid excessive disk space usage. Our CMake + * builds load the `SNAPPY_PATH` automatically if it's enabled so we only need to infer the path to + * the library if it's not already set. + */ +#define BLKCMP_PFX "block_compressor=" +#define SNAPPY_BLK BLKCMP_PFX "snappy" +#define EXTPATH "../../ext/" +#ifndef SNAPPY_PATH +#define SNAPPY_PATH EXTPATH "compressors/snappy/.libs/libwiredtiger_snappy.so" +#endif +#define SNAPPY_EXT ",extensions=(" SNAPPY_PATH ")" /* Test harness consts. */ -static const char *DEFAULT_FRAMEWORK_SCHEMA = "key_format=S,value_format=S"; +static const char *DEFAULT_FRAMEWORK_SCHEMA = "key_format=S,value_format=S,"; static const char *TABLE_OPERATION_TRACKING = "table:operation_tracking"; static const char *TABLE_SCHEMA_TRACKING = "table:schema_tracking"; static const char *STATISTICS_URI = "statistics:"; diff --git a/src/third_party/wiredtiger/test/cppsuite/test_harness/util/logger.cxx b/src/third_party/wiredtiger/test/cppsuite/test_harness/util/logger.cxx new file mode 100644 index 00000000000..67b72eac172 --- /dev/null +++ b/src/third_party/wiredtiger/test/cppsuite/test_harness/util/logger.cxx @@ -0,0 +1,96 @@ +/*- + * Public Domain 2014-present MongoDB, Inc. + * Public Domain 2008-2014 WiredTiger, Inc. + * + * This is free and unencumbered software released into the public domain. + * + * Anyone is free to copy, modify, publish, use, compile, sell, or + * distribute this software, either in source code form or as a compiled + * binary, for any purpose, commercial or non-commercial, and by any + * means. + * + * In jurisdictions that recognize copyright laws, the author or authors + * of this software dedicate any and all copyright interest in the + * software to the public domain. We make this dedication for the benefit + * of the public at large and to the detriment of our heirs and + * successors. We intend this dedication to be an overt act of + * relinquishment in perpetuity of all present and future rights to this + * software under copyright law. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. + * IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + */ + +#include <chrono> +#include <iostream> +#include <mutex> +#include <sstream> +#include <thread> + +#include "logger.h" + +/* Define helpful functions related to debugging. */ +namespace test_harness { +/* Order of elements in this array corresponds to the definitions above. */ +const char *const LOG_LEVELS[] = {"ERROR", "WARN", "INFO", "TRACE"}; + +/* Mutex used by logger to synchronize printing. */ +static std::mutex _logger_mtx; + +/* Set default log level. */ +int64_t logger::trace_level = LOG_WARN; + +/* Include date in the logs by default. */ +bool logger::include_date = true; + +void +get_time(char *time_buf, size_t buf_size) +{ + size_t alloc_size; + struct tm *tm, _tm; + + /* Get time since epoch in nanoseconds. */ + uint64_t epoch_nanosec = std::chrono::high_resolution_clock::now().time_since_epoch().count(); + + /* Calculate time since epoch in seconds. */ + time_t time_epoch_sec = epoch_nanosec / WT_BILLION; + + tm = localtime_r(&time_epoch_sec, &_tm); + testutil_assert(tm != nullptr); + + alloc_size = + strftime(time_buf, buf_size, logger::include_date ? "[%Y-%m-%dT%H:%M:%S" : "[%H:%M:%S", tm); + + testutil_assert(alloc_size <= buf_size); + WT_IGNORE_RET(__wt_snprintf(&time_buf[alloc_size], buf_size - alloc_size, ".%" PRIu64 "Z]", + (uint64_t)epoch_nanosec % WT_BILLION)); +} + +/* Used to print out traces for debugging purpose. */ +void +logger::log_msg(int64_t trace_type, const std::string &str) +{ + if (logger::trace_level >= trace_type) { + testutil_assert( + trace_type >= LOG_ERROR && trace_type < sizeof(LOG_LEVELS) / sizeof(LOG_LEVELS[0])); + + char time_buf[64]; + get_time(time_buf, sizeof(time_buf)); + + std::ostringstream ss; + ss << time_buf << "[TID:" << std::this_thread::get_id() << "][" << LOG_LEVELS[trace_type] + << "]: " << str << std::endl; + + std::lock_guard<std::mutex> lg(_logger_mtx); + if (trace_type == LOG_ERROR) + std::cerr << ss.str(); + else + std::cout << ss.str(); + } +} +} // namespace test_harness diff --git a/src/third_party/wiredtiger/test/cppsuite/test_harness/util/logger.h b/src/third_party/wiredtiger/test/cppsuite/test_harness/util/logger.h new file mode 100644 index 00000000000..8d67bfd7e27 --- /dev/null +++ b/src/third_party/wiredtiger/test/cppsuite/test_harness/util/logger.h @@ -0,0 +1,76 @@ +/*- + * Public Domain 2014-present MongoDB, Inc. + * Public Domain 2008-2014 WiredTiger, Inc. + * + * This is free and unencumbered software released into the public domain. + * + * Anyone is free to copy, modify, publish, use, compile, sell, or + * distribute this software, either in source code form or as a compiled + * binary, for any purpose, commercial or non-commercial, and by any + * means. + * + * In jurisdictions that recognize copyright laws, the author or authors + * of this software dedicate any and all copyright interest in the + * software to the public domain. We make this dedication for the benefit + * of the public at large and to the detriment of our heirs and + * successors. We intend this dedication to be an overt act of + * relinquishment in perpetuity of all present and future rights to this + * software under copyright law. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. + * IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + */ + +#ifndef DEBUG_UTILS_H +#define DEBUG_UTILS_H + +/* Following definitions are required in order to use printing format specifiers in C++. */ +#define __STDC_LIMIT_MACROS +#define __STDC_FORMAT_MACROS + +#include <chrono> +#include <iostream> +#include <mutex> +#include <sstream> +#include <thread> + +extern "C" { +#include "test_util.h" +} + +/* Define helpful functions related to debugging. */ +namespace test_harness { + +/* Possible log levels. If you change anything here make sure you update LOG_LEVELS accordingly. */ +#define LOG_ERROR 0 +#define LOG_WARN 1 +#define LOG_INFO 2 +/* + * The trace log level can incur a performance overhead since the current logging implementation + * requires per-line locking. + */ +#define LOG_TRACE 3 + +class logger { + public: + /* Current log level. Default is LOG_WARN. */ + static int64_t trace_level; + + /* Include date in the logs if enabled. Default is true. */ + static bool include_date; + + public: + /* Used to print out traces for debugging purpose. */ + static void log_msg(int64_t trace_type, const std::string &str); + + /* Make sure the class will not be instantiated. */ + logger() = delete; +}; +} // namespace test_harness + +#endif diff --git a/src/third_party/wiredtiger/test/cppsuite/test_harness/util/scoped_types.cxx b/src/third_party/wiredtiger/test/cppsuite/test_harness/util/scoped_types.cxx new file mode 100644 index 00000000000..599a8214be2 --- /dev/null +++ b/src/third_party/wiredtiger/test/cppsuite/test_harness/util/scoped_types.cxx @@ -0,0 +1,162 @@ +/*- + * Public Domain 2014-present MongoDB, Inc. + * Public Domain 2008-2014 WiredTiger, Inc. + * + * This is free and unencumbered software released into the public domain. + * + * Anyone is free to copy, modify, publish, use, compile, sell, or + * distribute this software, either in source code form or as a compiled + * binary, for any purpose, commercial or non-commercial, and by any + * means. + * + * In jurisdictions that recognize copyright laws, the author or authors + * of this software dedicate any and all copyright interest in the + * software to the public domain. We make this dedication for the benefit + * of the public at large and to the detriment of our heirs and + * successors. We intend this dedication to be an overt act of + * relinquishment in perpetuity of all present and future rights to this + * software under copyright law. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. + * IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + */ + +#include <utility> + +#include "scoped_types.h" + +namespace test_harness { + +/* scoped_cursor implementation */ +scoped_cursor::scoped_cursor(WT_SESSION *session, const char *uri, const char *cfg) +{ + reinit(session, uri, cfg); +} + +scoped_cursor::scoped_cursor(scoped_cursor &&other) +{ + std::swap(_cursor, other._cursor); +} + +scoped_cursor::~scoped_cursor() +{ + if (_cursor != nullptr) + testutil_check(_cursor->close(_cursor)); +} + +/* + * Implement move assignment by move constructing a temporary and swapping its internals with the + * current cursor. This means that the currently held WT_CURSOR will get destroyed as the temporary + * falls out of the scope and we will steal the one that we're move assigning from. + */ +scoped_cursor & +scoped_cursor::operator=(scoped_cursor &&other) +{ + scoped_cursor tmp(std::move(other)); + std::swap(_cursor, tmp._cursor); + return (*this); +} + +void +scoped_cursor::reinit(WT_SESSION *session, const char *uri, const char *cfg) +{ + if (_cursor != nullptr) { + testutil_check(_cursor->close(_cursor)); + _cursor = nullptr; + } + if (session != nullptr) + testutil_check(session->open_cursor(session, uri, nullptr, cfg, &_cursor)); +} + +/* + * Override the dereference operators. The idea is that we should able to use this class as if it is + * a pointer to a WT_CURSOR. + */ +WT_CURSOR & +scoped_cursor::operator*() +{ + return (*_cursor); +} + +WT_CURSOR * +scoped_cursor::operator->() +{ + return (_cursor); +} + +WT_CURSOR * +scoped_cursor::get() +{ + return (_cursor); +} + +/* scoped_session implementation */ +scoped_session::scoped_session(WT_CONNECTION *conn) +{ + reinit(conn); +} + +scoped_session::~scoped_session() +{ + if (_session != nullptr) + testutil_check(_session->close(_session, nullptr)); +} + +scoped_session::scoped_session(scoped_session &&other) +{ + std::swap(_session, other._session); +} + +/* + * Implement move assignment by move constructing a temporary and swapping its internals with the + * current session. This means that the currently held WT_SESSION will get destroyed as the + * temporary falls out of the scope and we will steal the one that we're move assigning from. + */ +scoped_session & +scoped_session::operator=(scoped_session &&other) +{ + scoped_session tmp(std::move(other)); + std::swap(_session, tmp._session); + return (*this); +} + +void +scoped_session::reinit(WT_CONNECTION *conn) +{ + if (_session != nullptr) { + testutil_check(_session->close(_session, nullptr)); + _session = nullptr; + } + if (conn != nullptr) + testutil_check(conn->open_session(conn, nullptr, nullptr, &_session)); +} + +WT_SESSION & +scoped_session::operator*() +{ + return (*_session); +} + +WT_SESSION * +scoped_session::operator->() +{ + return (_session); +} + +WT_SESSION * +scoped_session::get() +{ + return (_session); +} + +scoped_cursor +scoped_session::open_scoped_cursor(const char *uri, const char *cfg) +{ + return (scoped_cursor(_session, uri, cfg)); +} +} // namespace test_harness diff --git a/src/third_party/wiredtiger/test/cppsuite/test_harness/util/scoped_types.h b/src/third_party/wiredtiger/test/cppsuite/test_harness/util/scoped_types.h new file mode 100644 index 00000000000..47b8592ede0 --- /dev/null +++ b/src/third_party/wiredtiger/test/cppsuite/test_harness/util/scoped_types.h @@ -0,0 +1,99 @@ +/*- + * Public Domain 2014-present MongoDB, Inc. + * Public Domain 2008-2014 WiredTiger, Inc. + * + * This is free and unencumbered software released into the public domain. + * + * Anyone is free to copy, modify, publish, use, compile, sell, or + * distribute this software, either in source code form or as a compiled + * binary, for any purpose, commercial or non-commercial, and by any + * means. + * + * In jurisdictions that recognize copyright laws, the author or authors + * of this software dedicate any and all copyright interest in the + * software to the public domain. We make this dedication for the benefit + * of the public at large and to the detriment of our heirs and + * successors. We intend this dedication to be an overt act of + * relinquishment in perpetuity of all present and future rights to this + * software under copyright law. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. + * IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + */ + +#ifndef SCOPED_TYPES_H +#define SCOPED_TYPES_H + +/* Following definitions are required in order to use printing format specifiers in C++. */ +#define __STDC_LIMIT_MACROS +#define __STDC_FORMAT_MACROS + +extern "C" { +#include "test_util.h" +} + +namespace test_harness { +class scoped_cursor { + public: + scoped_cursor() = default; + scoped_cursor(WT_SESSION *session, const char *uri, const char *cfg); + + /* Moving is ok but copying is not. */ + scoped_cursor(scoped_cursor &&other); + + virtual ~scoped_cursor(); + + scoped_cursor &operator=(scoped_cursor &&other); + scoped_cursor(const scoped_cursor &) = delete; + scoped_cursor &operator=(const scoped_cursor &) = delete; + + void reinit(WT_SESSION *session, const char *uri, const char *cfg); + + WT_CURSOR &operator*(); + WT_CURSOR *operator->(); + + WT_CURSOR *get(); + + private: + WT_CURSOR *_cursor = nullptr; +}; + +class scoped_session { + public: + scoped_session() = default; + explicit scoped_session(WT_CONNECTION *conn); + + virtual ~scoped_session(); + + /* Moving is ok but copying is not. */ + scoped_session(scoped_session &&other); + + scoped_session &operator=(scoped_session &&other); + + scoped_session(const scoped_session &) = delete; + scoped_session &operator=(const scoped_session &) = delete; + + void reinit(WT_CONNECTION *conn); + + /* + * Override the dereference operators. The idea is that we should able to use this class as if + * it is a pointer to a WT_SESSION. + */ + WT_SESSION &operator*(); + WT_SESSION *operator->(); + + WT_SESSION *get(); + + scoped_cursor open_scoped_cursor(const char *uri, const char *cfg = nullptr); + + private: + WT_SESSION *_session = nullptr; +}; + +} // namespace test_harness +#endif diff --git a/src/third_party/wiredtiger/test/cppsuite/test_harness/workload/database_model.cxx b/src/third_party/wiredtiger/test/cppsuite/test_harness/workload/database_model.cxx new file mode 100644 index 00000000000..9692ff9183a --- /dev/null +++ b/src/third_party/wiredtiger/test/cppsuite/test_harness/workload/database_model.cxx @@ -0,0 +1,156 @@ +/*- + * Public Domain 2014-present MongoDB, Inc. + * Public Domain 2008-2014 WiredTiger, Inc. + * + * This is free and unencumbered software released into the public domain. + * + * Anyone is free to copy, modify, publish, use, compile, sell, or + * distribute this software, either in source code form or as a compiled + * binary, for any purpose, commercial or non-commercial, and by any + * means. + * + * In jurisdictions that recognize copyright laws, the author or authors + * of this software dedicate any and all copyright interest in the + * software to the public domain. We make this dedication for the benefit + * of the public at large and to the detriment of our heirs and + * successors. We intend this dedication to be an overt act of + * relinquishment in perpetuity of all present and future rights to this + * software under copyright law. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. + * IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + */ + +#include <chrono> +#include <tuple> + +#include "../connection_manager.h" +#include "../timestamp_manager.h" +#include "../util/api_const.h" +#include "database_model.h" +#include "random_generator.h" +#include "workload_tracking.h" + +namespace test_harness { +/* collection class implementation */ +collection::collection(const uint64_t id, const uint64_t key_count, const std::string &name) + : id(id), _key_count(key_count), name(name) +{ +} + +uint64_t +collection::get_key_count() const +{ + return (_key_count); +} + +void +collection::increase_key_count(uint64_t increment) +{ + _key_count += increment; +} + +/* database class implementation */ +std::string +database::build_collection_name(const uint64_t id) +{ + return (std::string("table:collection_" + std::to_string(id))); +} + +void +database::add_collection(uint64_t key_count) +{ + std::lock_guard<std::mutex> lg(_mtx); + if (_session.get() == nullptr) + _session = connection_manager::instance().create_session(); + if (_collection_create_config.empty()) + testutil_die(EINVAL, "database_model: no collection create config specified!"); + uint64_t next_id = _next_collection_id++; + std::string collection_name = build_collection_name(next_id); + /* FIX-ME-Test-Framework: This will get removed when we split the model up. */ + _collections.emplace(std::piecewise_construct, std::forward_as_tuple(next_id), + std::forward_as_tuple(next_id, key_count, collection_name)); + testutil_check( + _session->create(_session.get(), collection_name.c_str(), _collection_create_config.c_str())); + _tracking->save_schema_operation( + tracking_operation::CREATE_COLLECTION, next_id, _tsm->get_next_ts()); +} + +collection & +database::get_collection(uint64_t id) +{ + std::lock_guard<std::mutex> lg(_mtx); + const auto it = _collections.find(id); + if (it == _collections.end()) + testutil_die(EINVAL, "tried to get collection that doesn't exist."); + return (it->second); +} + +collection & +database::get_random_collection() +{ + size_t collection_count = get_collection_count(); + /* Any caller should expect at least one collection to exist. */ + testutil_assert(collection_count != 0); + return (get_collection( + random_generator::instance().generate_integer<uint64_t>(0, collection_count - 1))); +} + +uint64_t +database::get_collection_count() +{ + std::lock_guard<std::mutex> lg(_mtx); + return (_collections.size()); +} + +std::vector<std::string> +database::get_collection_names() +{ + std::lock_guard<std::mutex> lg(_mtx); + std::vector<std::string> collection_names; + + for (auto const &it : _collections) + collection_names.push_back(it.second.name); + + return (collection_names); +} + +std::vector<uint64_t> +database::get_collection_ids() +{ + std::lock_guard<std::mutex> lg(_mtx); + std::vector<uint64_t> collection_ids; + + for (auto const &it : _collections) + collection_ids.push_back(it.first); + + return (collection_ids); +} + +void +database::set_timestamp_manager(timestamp_manager *tsm) +{ + testutil_assert(_tsm == nullptr); + _tsm = tsm; +} + +void +database::set_workload_tracking(workload_tracking *tracking) +{ + testutil_assert(_tracking == nullptr); + _tracking = tracking; +} + +void +database::set_create_config(bool use_compression) +{ + _collection_create_config = use_compression ? + std::string(DEFAULT_FRAMEWORK_SCHEMA) + std::string(SNAPPY_BLK) : + DEFAULT_FRAMEWORK_SCHEMA; +} +} // namespace test_harness diff --git a/src/third_party/wiredtiger/test/cppsuite/test_harness/workload/database_model.h b/src/third_party/wiredtiger/test/cppsuite/test_harness/workload/database_model.h index c9562954bfd..edbb5bd2675 100644 --- a/src/third_party/wiredtiger/test/cppsuite/test_harness/workload/database_model.h +++ b/src/third_party/wiredtiger/test/cppsuite/test_harness/workload/database_model.h @@ -29,97 +29,91 @@ #ifndef DATABASE_MODEL_H #define DATABASE_MODEL_H -#include <map> +#include <atomic> #include <string> +#include <map> -namespace test_harness { +#include "workload_tracking.h" +/* Forward declarations for classes to reduce compilation time and modules coupling. */ +class timestamp_manager; + +namespace test_harness { /* Key/Value type. */ typedef std::string key_value_t; -/* Representation of key states. */ -struct key_t { - bool exists; -}; +/* A collection is made of mapped key value objects. */ +class collection { + public: + collection(const uint64_t id, const uint64_t key_count, const std::string &name); + + /* Copies aren't allowed. */ + collection(const collection &) = delete; + collection &operator=(const collection &) = delete; + + uint64_t get_key_count() const; + + /* + * Adding new keys should generally be singly threaded per collection. If two threads both + * attempt to add keys using the incrementing id pattern they'd frequently conflict. + * + * The usage pattern is: + * 1. Call get_key_count to get the number of keys already existing. Add keys with id's equal + * to and greater than this value. + * 2. Once the transaction has successfully committed then call increase_key_count() with the + * number of added keys. + * + * The set of keys should always be contiguous such that other threads calling get_key_count + * will always know that the keys in existence are 0 -> _key_count - 1. + */ + void increase_key_count(uint64_t increment); -/* Representation of a value. */ -struct value_t { - key_value_t value; -}; + public: + const std::string name; + const uint64_t id; -/* A collection is made of mapped Key objects. */ -struct collection_t { - std::map<key_value_t, key_t> keys; - std::map<key_value_t, value_t> values; + private: + std::atomic<uint64_t> _key_count{0}; }; /* Representation of the collections in memory. */ class database { public: - std::vector<std::string> - get_collection_names() - { - std::lock_guard<std::mutex> lg(_mtx); - std::vector<std::string> collection_names; - - for (auto const &it : _collections) - collection_names.push_back(it.first); - - return (collection_names); - } - - std::map<key_value_t, key_t> - get_keys(const std::string &collection_name) - { - std::lock_guard<std::mutex> lg(_mtx); - return (_collections.at(collection_name).keys); - } - - void - add_collection(const std::string &collection_name) - { - std::lock_guard<std::mutex> lg(_mtx); - testutil_assert(_collections.find(collection_name) == _collections.end()); - _collections[collection_name] = {}; - } - - value_t - get_record(const std::string &collection_name, const char *key) - { - std::lock_guard<std::mutex> lg(_mtx); - return (_collections.at(collection_name).values.at(key)); - } - - void - insert_record(const std::string &collection_name, const char *key, const char *value) - { - std::lock_guard<std::mutex> lg(_mtx); - auto &c = _collections.at(collection_name); - c.keys[key].exists = true; - value_t v; - v.value = key_value_t(value); - c.values.emplace(key_value_t(key), v); - } - - void - update_record(const std::string &collection_name, const char *key, const char *value) - { - std::lock_guard<std::mutex> lg(_mtx); - auto &c = _collections.at(collection_name); - c.values.at(key).value = key_value_t(value); - } - - void - delete_record(const std::string &collection_name, const char *key) - { - std::lock_guard<std::mutex> lg(_mtx); - auto &c = _collections.at(collection_name); - c.keys.at(key).exists = false; - c.values.erase(key); - } + static std::string build_collection_name(const uint64_t id); + + public: + /* + * Add a new collection, this will create the underlying collection in the database. + */ + void add_collection(uint64_t key_count = 0); + + /* Get a collection using the id of the collection. */ + collection &get_collection(uint64_t id); + + /* Get a random collection. */ + collection &get_random_collection(); + + /* + * Retrieve the current collection count, collection names are indexed from 0 so when using this + * take care to avoid an off by one error. + */ + uint64_t get_collection_count(); + + /* FIX-ME-Test-Framework: Replace usages of this with get_collection_ids. */ + std::vector<std::string> get_collection_names(); + + std::vector<uint64_t> get_collection_ids(); + void set_timestamp_manager(timestamp_manager *tsm); + void set_workload_tracking(workload_tracking *tracking); + void set_create_config(bool use_compression); private: - std::map<std::string, collection_t> _collections; + std::string _collection_create_config = ""; + scoped_session _session; + timestamp_manager *_tsm = nullptr; + workload_tracking *_tracking = nullptr; + uint64_t _next_collection_id = 0; + std::map<uint64_t, collection> _collections; std::mutex _mtx; }; } // namespace test_harness diff --git a/src/third_party/wiredtiger/test/cppsuite/test_harness/workload/database_operation.cxx b/src/third_party/wiredtiger/test/cppsuite/test_harness/workload/database_operation.cxx new file mode 100644 index 00000000000..e18125446fe --- /dev/null +++ b/src/third_party/wiredtiger/test/cppsuite/test_harness/workload/database_operation.cxx @@ -0,0 +1,283 @@ +/*- + * Public Domain 2014-present MongoDB, Inc. + * Public Domain 2008-2014 WiredTiger, Inc. + * + * This is free and unencumbered software released into the public domain. + * + * Anyone is free to copy, modify, publish, use, compile, sell, or + * distribute this software, either in source code form or as a compiled + * binary, for any purpose, commercial or non-commercial, and by any + * means. + * + * In jurisdictions that recognize copyright laws, the author or authors + * of this software dedicate any and all copyright interest in the + * software to the public domain. We make this dedication for the benefit + * of the public at large and to the detriment of our heirs and + * successors. We intend this dedication to be an overt act of + * relinquishment in perpetuity of all present and future rights to this + * software under copyright law. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. + * IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + */ + +#include <cmath> +#include <map> + +#include "../connection_manager.h" +#include "../thread_manager.h" +#include "../util/api_const.h" +#include "database_operation.h" +#include "random_generator.h" +#include "workload_tracking.h" + +namespace test_harness { +/* Static methods. */ +static void +populate_worker(thread_context *tc) +{ + uint64_t collections_per_thread = tc->collection_count / tc->thread_count; + for (int64_t i = 0; i < collections_per_thread; ++i) { + collection &coll = tc->db.get_collection((tc->id * collections_per_thread) + i); + /* + * WiredTiger lets you open a cursor on a collection using the same pointer. When a session + * is closed, WiredTiger APIs close the cursors too. + */ + scoped_cursor cursor = tc->session.open_scoped_cursor(coll.name.c_str()); + for (uint64_t i = 0; i < tc->key_count; ++i) { + /* Start a txn. */ + tc->transaction.begin(); + if (!tc->insert(cursor, coll.id, i)) { + /* We failed to insert, and our transaction was rolled back retry. */ + --i; + continue; + } + tc->transaction.commit(); + } + } + logger::log_msg(LOG_TRACE, "Populate: thread {" + std::to_string(tc->id) + "} finished"); +} + +/* database_operation class implementation. */ +void +database_operation::populate( + database &database, timestamp_manager *tsm, configuration *config, workload_tracking *tracking) +{ + int64_t collection_count, key_count, key_size, thread_count, value_size; + std::vector<thread_context *> workers; + std::string collection_name; + thread_manager tm; + + /* Validate our config. */ + collection_count = config->get_int(COLLECTION_COUNT); + key_count = config->get_int(KEY_COUNT_PER_COLLECTION); + value_size = config->get_int(VALUE_SIZE); + thread_count = config->get_int(THREAD_COUNT); + testutil_assert(collection_count % thread_count == 0); + testutil_assert(value_size > 0); + key_size = config->get_int(KEY_SIZE); + testutil_assert(key_size > 0); + /* Keys must be unique. */ + testutil_assert(key_count <= pow(10, key_size)); + + logger::log_msg( + LOG_INFO, "Populate: " + std::to_string(collection_count) + " creating collections."); + + /* Create n collections as per the configuration. */ + for (int64_t i = 0; i < collection_count; ++i) + /* + * The database model will call into the API and create the collection, with its own + * session. + */ + database.add_collection(key_count); + + logger::log_msg( + LOG_INFO, "Populate: " + std::to_string(collection_count) + " collections created."); + + /* + * Spawn thread_count threads to populate the database, theoretically we should be IO bound + * here. + */ + for (int64_t i = 0; i < thread_count; ++i) { + thread_context *tc = new thread_context(i, thread_type::INSERT, config, + connection_manager::instance().create_session(), tsm, tracking, database); + workers.push_back(tc); + tm.add_thread(populate_worker, tc); + } + + /* Wait for our populate threads to finish and then join them. */ + logger::log_msg(LOG_INFO, "Populate: waiting for threads to complete."); + tm.join(); + + /* Cleanup our workers. */ + for (auto &it : workers) { + delete it; + it = nullptr; + } + logger::log_msg(LOG_INFO, "Populate: finished."); +} + +void +database_operation::insert_operation(thread_context *tc) +{ + logger::log_msg( + LOG_INFO, type_string(tc->type) + " thread {" + std::to_string(tc->id) + "} commencing."); + + /* Helper struct which stores a pointer to a collection and a cursor associated with it. */ + struct collection_cursor { + collection_cursor(collection &coll, scoped_cursor &&cursor) + : coll(coll), cursor(std::move(cursor)) + { + } + collection &coll; + scoped_cursor cursor; + }; + + /* Collection cursor vector. */ + std::vector<collection_cursor> ccv; + uint64_t collection_count = tc->db.get_collection_count(); + uint64_t collections_per_thread = collection_count / tc->thread_count; + /* Must have unique collections for each thread. */ + testutil_assert(collection_count % tc->thread_count == 0); + for (int i = tc->id * collections_per_thread; + i < (tc->id * collections_per_thread) + collections_per_thread && tc->running(); ++i) { + collection &coll = tc->db.get_collection(i); + scoped_cursor cursor = tc->session.open_scoped_cursor(coll.name.c_str()); + ccv.push_back({coll, std::move(cursor)}); + } + + uint64_t counter = 0; + while (tc->running()) { + uint64_t start_key = ccv[counter].coll.get_key_count(); + uint64_t added_count = 0; + bool committed = true; + tc->transaction.begin(); + + /* Collection cursor. */ + auto &cc = ccv[counter]; + while (tc->transaction.active() && tc->running()) { + /* Insert a key value pair. */ + if (!tc->insert(cc.cursor, cc.coll.id, start_key + added_count)) { + committed = false; + break; + } + added_count++; + tc->transaction.try_commit(); + /* Sleep the duration defined by the op_rate. */ + tc->sleep(); + } + /* Reset our cursor to avoid pinning content. */ + testutil_check(cc.cursor->reset(cc.cursor.get())); + + /* + * We need to inform the database model that we've added these keys as some other thread may + * rely on the key_count data. Only do so if we successfully committed. + */ + if (committed) + cc.coll.increase_key_count(added_count); + counter++; + if (counter == collections_per_thread) + counter = 0; + } + /* Make sure the last transaction is rolled back now the work is finished. */ + if (tc->transaction.active()) + tc->transaction.rollback(); +} + +void +database_operation::read_operation(thread_context *tc) +{ + logger::log_msg( + LOG_INFO, type_string(tc->type) + " thread {" + std::to_string(tc->id) + "} commencing."); + + std::map<uint64_t, scoped_cursor> cursors; + while (tc->running()) { + /* Get a collection and find a cached cursor. */ + collection &coll = tc->db.get_random_collection(); + + if (cursors.find(coll.id) == cursors.end()) + cursors.emplace(coll.id, std::move(tc->session.open_scoped_cursor(coll.name.c_str()))); + + /* Do a second lookup now that we know it exists. */ + auto &cursor = cursors[coll.id]; + + tc->transaction.begin(); + while (tc->transaction.active() && tc->running()) { + if (tc->next(cursor) == WT_ROLLBACK) + /* We got an error, our transaction has been rolled back. */ + break; + tc->transaction.add_op(); + tc->transaction.try_rollback(); + tc->sleep(); + } + /* Reset our cursor to avoid pinning content. */ + testutil_check(cursor->reset(cursor.get())); + } + /* Make sure the last transaction is rolled back now the work is finished. */ + if (tc->transaction.active()) + tc->transaction.rollback(); +} + +void +database_operation::update_operation(thread_context *tc) +{ + logger::log_msg( + LOG_INFO, type_string(tc->type) + " thread {" + std::to_string(tc->id) + "} commencing."); + /* Cursor map. */ + std::map<uint64_t, scoped_cursor> cursors; + + /* + * Loop while the test is running. + */ + while (tc->running()) { + /* + * Sleep the period defined by the op_rate in the configuration. Do this at the start of the + * loop as it could be skipped by a subsequent continue call. + */ + tc->sleep(); + + /* Choose a random collection to update. */ + collection &coll = tc->db.get_random_collection(); + + /* Look for existing cursors in our cursor cache. */ + if (cursors.find(coll.id) == cursors.end()) { + logger::log_msg(LOG_TRACE, + "Thread {" + std::to_string(tc->id) + + "} Creating cursor for collection: " + coll.name); + /* Open a cursor for the chosen collection. */ + scoped_cursor cursor = tc->session.open_scoped_cursor(coll.name.c_str()); + cursors.emplace(coll.id, std::move(cursor)); + } + + /* Start a transaction if possible. */ + tc->transaction.try_begin(); + + /* Get the cursor associated with the collection. */ + scoped_cursor &cursor = cursors[coll.id]; + + /* Choose a random key to update. */ + uint64_t key_id = + random_generator::instance().generate_integer<uint64_t>(0, coll.get_key_count() - 1); + bool successful_update = tc->update(cursor, coll.id, tc->key_to_string(key_id)); + + /* Reset our cursor to avoid pinning content. */ + testutil_check(cursor->reset(cursor.get())); + + /* We received a rollback in update. */ + if (!successful_update) + continue; + + /* Commit the current transaction if we're able to. */ + tc->transaction.try_commit(); + } + + /* Make sure the last operation is committed now the work is finished. */ + if (tc->transaction.active()) + tc->transaction.commit(); +} +} // namespace test_harness diff --git a/src/third_party/wiredtiger/test/cppsuite/test_harness/workload/database_operation.h b/src/third_party/wiredtiger/test/cppsuite/test_harness/workload/database_operation.h index 007d39da345..c8dd7370b0f 100644 --- a/src/third_party/wiredtiger/test/cppsuite/test_harness/workload/database_operation.h +++ b/src/third_party/wiredtiger/test/cppsuite/test_harness/workload/database_operation.h @@ -30,7 +30,6 @@ #define DATABASE_OPERATION_H #include "database_model.h" -#include "workload_tracking.h" #include "thread_context.h" namespace test_harness { @@ -39,226 +38,25 @@ class database_operation { /* * Function that performs the following steps using the configuration that is defined by the * test: - * - Create the working dir. - * - Open a connection. - * - Open a session. - * - Create n collections as per the configuration. + * - Creates N collections as per the configuration. + * - Creates M threads as per the configuration, each thread will: * - Open a cursor on each collection. - * - Insert m key/value pairs in each collection. Values are random strings which size is + * - Insert K key/value pairs in each collection. Values are random strings which size is * defined by the configuration. - * - Store in memory the created collections. */ - virtual void - populate(database &database, timestamp_manager *timestamp_manager, configuration *config, - workload_tracking *tracking) - { - WT_CURSOR *cursor; - WT_SESSION *session; - wt_timestamp_t ts; - int64_t collection_count, key_count, key_cpt, key_size, value_size; - std::string collection_name, cfg, home; - key_value_t generated_key, generated_value; - bool ts_enabled = timestamp_manager->enabled(); + virtual void populate(database &database, timestamp_manager *tsm, configuration *config, + workload_tracking *tracking); - cursor = nullptr; - collection_count = key_count = key_size = value_size = 0; + /* Basic insert operation that adds a new key every rate tick. */ + virtual void insert_operation(thread_context *tc); - /* Get a session. */ - session = connection_manager::instance().create_session(); - /* Create n collections as per the configuration and store each collection name. */ - collection_count = config->get_int(COLLECTION_COUNT); - for (size_t i = 0; i < collection_count; ++i) { - collection_name = "table:collection" + std::to_string(i); - database.add_collection(collection_name); - testutil_check( - session->create(session, collection_name.c_str(), DEFAULT_FRAMEWORK_SCHEMA)); - ts = timestamp_manager->get_next_ts(); - tracking->save_schema_operation( - tracking_operation::CREATE_COLLECTION, collection_name, ts); - } - debug_print(std::to_string(collection_count) + " collections created", DEBUG_TRACE); - - /* Open a cursor on each collection and use the configuration to insert key/value pairs. */ - key_count = config->get_int(KEY_COUNT); - value_size = config->get_int(VALUE_SIZE); - testutil_assert(value_size > 0); - key_size = config->get_int(KEY_SIZE); - testutil_assert(key_size > 0); - /* Keys must be unique. */ - testutil_assert(key_count <= pow(10, key_size)); - - for (const auto &collection_name : database.get_collection_names()) { - key_cpt = 0; - /* - * WiredTiger lets you open a cursor on a collection using the same pointer. When a - * session is closed, WiredTiger APIs close the cursors too. - */ - testutil_check( - session->open_cursor(session, collection_name.c_str(), NULL, NULL, &cursor)); - for (size_t i = 0; i < key_count; ++i) { - /* Generation of a unique key. */ - generated_key = number_to_string(key_size, key_cpt); - ++key_cpt; - /* - * Generation of a random string value using the size defined in the test - * configuration. - */ - generated_value = - random_generator::random_generator::instance().generate_string(value_size); - ts = timestamp_manager->get_next_ts(); - if (ts_enabled) - testutil_check(session->begin_transaction(session, "")); - insert(cursor, tracking, collection_name, generated_key.c_str(), - generated_value.c_str(), ts); - if (ts_enabled) { - cfg = std::string(COMMIT_TS) + "=" + timestamp_manager->decimal_to_hex(ts); - testutil_check(session->commit_transaction(session, cfg.c_str())); - } - } - } - debug_print("Populate stage done", DEBUG_TRACE); - } - - /* Basic read operation that walks a cursors across all collections. */ - virtual void - read_operation(thread_context *tc) - { - WT_CURSOR *cursor; - std::vector<WT_CURSOR *> cursors; - - /* Get a cursor for each collection in collection_names. */ - for (const auto &it : tc->database.get_collection_names()) { - testutil_check(tc->session->open_cursor(tc->session, it.c_str(), NULL, NULL, &cursor)); - cursors.push_back(cursor); - } - - while (!cursors.empty() && tc->running()) { - /* Walk each cursor. */ - for (const auto &it : cursors) { - if (it->next(it) != 0) - it->reset(it); - } - } - } - - /* - * Basic update operation that updates all the keys to a random value in each collection. - */ - virtual void - update_operation(thread_context *tc) - { - WT_CURSOR *cursor; - WT_DECL_RET; - wt_timestamp_t ts; - std::vector<WT_CURSOR *> cursors; - std::vector<std::string> collection_names = tc->database.get_collection_names(); - key_value_t key, generated_value; - const char *key_tmp; - uint64_t i = 0; - bool using_timestamps = tc->timestamp_manager->enabled(); - - /* Get a cursor for each collection in collection_names. */ - for (const auto &it : collection_names) { - testutil_check(tc->session->open_cursor(tc->session, it.c_str(), NULL, NULL, &cursor)); - cursors.push_back(cursor); - } - - /* - * Update each collection while the test is running. - */ - while (tc->running() && !collection_names.empty()) { - if (i >= collection_names.size()) - i = 0; - ret = cursors[i]->next(cursors[i]); - /* If we have reached the end of the collection, reset. */ - if (ret == WT_NOTFOUND) { - testutil_check(cursors[i]->reset(cursors[i])); - ++i; - } else if (ret != 0) - /* Stop updating in case of an error. */ - testutil_die(DEBUG_ERROR, "update_operation: cursor->next() failed: %d", ret); - else { - testutil_check(cursors[i]->get_key(cursors[i], &key_tmp)); - /* - * The retrieved key needs to be passed inside the update function. However, the - * update API doesn't guarantee our buffer will still be valid once it is called, as - * such we copy the buffer and then pass it into the API. - */ - key = key_value_t(key_tmp); - generated_value = - random_generator::random_generator::instance().generate_string(tc->value_size); - - /* Start a transaction. */ - if (!tc->transaction.active()) - tc->transaction.begin(tc->session, ""); - - ts = tc->timestamp_manager->get_next_ts(); - if (using_timestamps) - tc->transaction.set_commit_timestamp( - tc->session, timestamp_manager::decimal_to_hex(ts)); - - update(tc->tracking, cursors[i], collection_names[i], key.c_str(), - generated_value.c_str(), ts); - - /* Commit the current transaction. */ - tc->transaction.op_count++; - if (tc->transaction.can_commit()) - tc->transaction.commit(tc->session, ""); - } - tc->sleep(); - } - /* Make sure the last operation is committed now the work is finished. */ - if (tc->transaction.active()) - tc->transaction.commit(tc->session, ""); - } - - private: - /* WiredTiger APIs wrappers for single operations. */ - template <typename K, typename V> - void - insert(WT_CURSOR *cursor, workload_tracking *tracking, const std::string &collection_name, - const K &key, const V &value, wt_timestamp_t ts) - { - testutil_assert(cursor != nullptr); - - cursor->set_key(cursor, key); - cursor->set_value(cursor, value); - testutil_check(cursor->insert(cursor)); - debug_print("key/value inserted", DEBUG_TRACE); - - tracking->save_operation(tracking_operation::INSERT, collection_name, key, value, ts); - } - - template <typename K, typename V> - static void - update(workload_tracking *tracking, WT_CURSOR *cursor, const std::string &collection_name, - K key, V value, wt_timestamp_t ts) - { - testutil_assert(tracking != nullptr); - testutil_assert(cursor != nullptr); - - cursor->set_key(cursor, key); - cursor->set_value(cursor, value); - testutil_check(cursor->update(cursor)); - debug_print("key/value updated", DEBUG_TRACE); - - tracking->save_operation(tracking_operation::UPDATE, collection_name, key, value, ts); - } + /* Basic read operation that chooses a random collection and walks a cursor. */ + virtual void read_operation(thread_context *tc); /* - * Convert a number to a string. If the resulting string is less than the given length, padding - * of '0' is added. + * Basic update operation that chooses a random key and updates it. */ - static std::string - number_to_string(uint64_t size, uint64_t value) - { - std::string str, value_str = std::to_string(value); - testutil_assert(size >= value_str.size()); - uint64_t diff = size - value_str.size(); - std::string s(diff, '0'); - str = s.append(value_str); - return (str); - } + virtual void update_operation(thread_context *tc); }; } // namespace test_harness #endif diff --git a/src/third_party/wiredtiger/test/cppsuite/test_harness/workload/random_generator.cxx b/src/third_party/wiredtiger/test/cppsuite/test_harness/workload/random_generator.cxx new file mode 100644 index 00000000000..8a551b80a75 --- /dev/null +++ b/src/third_party/wiredtiger/test/cppsuite/test_harness/workload/random_generator.cxx @@ -0,0 +1,71 @@ +/*- + * Public Domain 2014-present MongoDB, Inc. + * Public Domain 2008-2014 WiredTiger, Inc. + * + * This is free and unencumbered software released into the public domain. + * + * Anyone is free to copy, modify, publish, use, compile, sell, or + * distribute this software, either in source code form or as a compiled + * binary, for any purpose, commercial or non-commercial, and by any + * means. + * + * In jurisdictions that recognize copyright laws, the author or authors + * of this software dedicate any and all copyright interest in the + * software to the public domain. We make this dedication for the benefit + * of the public at large and to the detriment of our heirs and + * successors. We intend this dedication to be an overt act of + * relinquishment in perpetuity of all present and future rights to this + * software under copyright law. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. + * IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + */ + +#include "random_generator.h" + +namespace test_harness { +random_generator & +random_generator::instance() +{ + thread_local random_generator _instance; + return (_instance); +} + +std::string +random_generator::generate_string(std::size_t length) +{ + std::string random_string; + + for (std::size_t i = 0; i < length; ++i) + random_string += _characters[_distribution(_generator)]; + + return (random_string); +} + +std::string +random_generator::generate_pseudo_random_string(std::size_t length) +{ + std::string random_string; + std::size_t start_location = _distribution(_generator); + + for (std::size_t i = 0; i < length; ++i) { + random_string += _characters[start_location]; + if (start_location == _characters.size() - 1) + start_location = 0; + else + start_location++; + } + return (random_string); +} + +random_generator::random_generator() +{ + _generator = std::mt19937(std::random_device{}()); + _distribution = std::uniform_int_distribution<>(0, _characters.size() - 1); +} +} // namespace test_harness diff --git a/src/third_party/wiredtiger/test/cppsuite/test_harness/workload/random_generator.h b/src/third_party/wiredtiger/test/cppsuite/test_harness/workload/random_generator.h index 7df4d7da3fb..cd6d7d4d7ec 100644 --- a/src/third_party/wiredtiger/test/cppsuite/test_harness/workload/random_generator.h +++ b/src/third_party/wiredtiger/test/cppsuite/test_harness/workload/random_generator.h @@ -30,49 +30,36 @@ #define RANDOM_GENERATOR_H #include <random> +#include <string> namespace test_harness { - /* Helper class to generate random values using uniform distributions. */ class random_generator { public: + static random_generator &instance(); + + public: /* No copies of the singleton allowed. */ random_generator(random_generator const &) = delete; random_generator &operator=(random_generator const &) = delete; - static random_generator & - instance() - { - static random_generator _instance; - return _instance; - } - /* Generate a random string of a given length. */ - std::string - generate_string(std::size_t length) - { - std::string random_string; - - for (std::size_t i = 0; i < length; ++i) - random_string += _characters[_distribution(_generator)]; + std::string generate_string(std::size_t length); - return (random_string); - } + /* Generate a pseudo random string which compresses better. */ + std::string generate_pseudo_random_string(std::size_t length); /* Generate a random integer between min and max. */ - int64_t - generate_integer(int64_t min, int64_t max) + template <typename T> + T + generate_integer(T min, T max) { - std::uniform_int_distribution<> dis(min, max); + std::uniform_int_distribution<T> dis(min, max); return dis(_generator); } private: - random_generator() - { - _generator = std::mt19937(std::random_device{}()); - _distribution = std::uniform_int_distribution<>(0, _characters.size() - 1); - } + random_generator(); std::mt19937 _generator; std::uniform_int_distribution<> _distribution; diff --git a/src/third_party/wiredtiger/test/cppsuite/test_harness/workload/thread_context.cxx b/src/third_party/wiredtiger/test/cppsuite/test_harness/workload/thread_context.cxx new file mode 100644 index 00000000000..321cc45b61b --- /dev/null +++ b/src/third_party/wiredtiger/test/cppsuite/test_harness/workload/thread_context.cxx @@ -0,0 +1,283 @@ +/*- + * Public Domain 2014-present MongoDB, Inc. + * Public Domain 2008-2014 WiredTiger, Inc. + * + * This is free and unencumbered software released into the public domain. + * + * Anyone is free to copy, modify, publish, use, compile, sell, or + * distribute this software, either in source code form or as a compiled + * binary, for any purpose, commercial or non-commercial, and by any + * means. + * + * In jurisdictions that recognize copyright laws, the author or authors + * of this software dedicate any and all copyright interest in the + * software to the public domain. We make this dedication for the benefit + * of the public at large and to the detriment of our heirs and + * successors. We intend this dedication to be an overt act of + * relinquishment in perpetuity of all present and future rights to this + * software under copyright law. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. + * IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + */ + +#include "../core/configuration.h" +#include "../timestamp_manager.h" +#include "../util/api_const.h" +#include "workload_tracking.h" +#include "random_generator.h" +#include "thread_context.h" + +namespace test_harness { +/* transaction_context class implementation */ +transaction_context::transaction_context( + configuration *config, timestamp_manager *timestamp_manager, WT_SESSION *session) + : _timestamp_manager(timestamp_manager), _session(session) +{ + /* Use optional here as our populate threads don't define this configuration. */ + configuration *transaction_config = config->get_optional_subconfig(OPS_PER_TRANSACTION); + if (transaction_config != nullptr) { + _min_op_count = transaction_config->get_optional_int(MIN, 1); + _max_op_count = transaction_config->get_optional_int(MAX, 1); + delete transaction_config; + } +} + +bool +transaction_context::active() const +{ + return (_in_txn); +} + +void +transaction_context::add_op() +{ + _op_count++; +} + +void +transaction_context::begin(const std::string &config) +{ + testutil_assert(!_in_txn); + testutil_check( + _session->begin_transaction(_session, config.empty() ? nullptr : config.c_str())); + /* This randomizes the number of operations to be executed in one transaction. */ + _target_op_count = + random_generator::instance().generate_integer<int64_t>(_min_op_count, _max_op_count); + _op_count = 0; + _in_txn = true; +} + +void +transaction_context::try_begin(const std::string &config) +{ + if (!_in_txn) + begin(config); +} + +void +transaction_context::commit(const std::string &config) +{ + testutil_assert(_in_txn); + testutil_check( + _session->commit_transaction(_session, config.empty() ? nullptr : config.c_str())); + _op_count = 0; + _in_txn = false; +} + +void +transaction_context::try_commit(const std::string &config) +{ + if (can_commit_rollback()) + commit(config); +} + +void +transaction_context::rollback(const std::string &config) +{ + testutil_assert(_in_txn); + testutil_check( + _session->rollback_transaction(_session, config.empty() ? nullptr : config.c_str())); + _op_count = 0; + _in_txn = false; +} + +void +transaction_context::try_rollback(const std::string &config) +{ + if (can_commit_rollback()) + rollback(config); +} + +void +transaction_context::set_commit_timestamp(wt_timestamp_t ts) +{ + /* We don't want to set zero timestamps on transactions if we're not using timestamps. */ + if (!_timestamp_manager->enabled()) + return; + std::string config = std::string(COMMIT_TS) + "=" + timestamp_manager::decimal_to_hex(ts); + testutil_check(_session->timestamp_transaction(_session, config.c_str())); +} + +bool +transaction_context::can_commit_rollback() +{ + return (_in_txn && _op_count >= _target_op_count); +} + +/* thread_context class implementation */ +thread_context::thread_context(uint64_t id, thread_type type, configuration *config, + scoped_session &&created_session, timestamp_manager *timestamp_manager, + workload_tracking *tracking, database &dbase) + : id(id), type(type), db(dbase), tsm(timestamp_manager), tracking(tracking), + session(std::move(created_session)), + transaction(transaction_context(config, timestamp_manager, session.get())), + /* These won't exist for certain threads which is why we use optional here. */ + collection_count(config->get_optional_int(COLLECTION_COUNT, 1)), + key_count(config->get_optional_int(KEY_COUNT_PER_COLLECTION, 1)), + key_size(config->get_optional_int(KEY_SIZE, 1)), + value_size(config->get_optional_int(VALUE_SIZE, 1)), + thread_count(config->get_int(THREAD_COUNT)) +{ + _throttle = throttle(config); + if (tracking->enabled()) + op_track_cursor = session.open_scoped_cursor(tracking->get_operation_table_name().c_str()); + + testutil_assert(key_size > 0 && value_size > 0); +} + +void +thread_context::finish() +{ + _running = false; +} + +std::string +thread_context::key_to_string(uint64_t key_id) +{ + std::string str, value_str = std::to_string(key_id); + testutil_assert(key_size >= value_str.size()); + uint64_t diff = key_size - value_str.size(); + std::string s(diff, '0'); + str = s.append(value_str); + return (str); +} + +bool +thread_context::update(scoped_cursor &cursor, uint64_t collection_id, const std::string &key) +{ + WT_DECL_RET; + std::string value; + wt_timestamp_t ts = tsm->get_next_ts(); + testutil_assert(tracking != nullptr); + testutil_assert(cursor.get() != nullptr); + + transaction.set_commit_timestamp(ts); + value = random_generator::instance().generate_pseudo_random_string(value_size); + cursor->set_key(cursor.get(), key.c_str()); + cursor->set_value(cursor.get(), value.c_str()); + ret = cursor->update(cursor.get()); + if (ret != 0) { + if (ret == WT_ROLLBACK) { + transaction.rollback(); + return (false); + } else + testutil_die(ret, "unhandled error while trying to update a key"); + } + ret = tracking->save_operation( + tracking_operation::INSERT, collection_id, key.c_str(), value.c_str(), ts, op_track_cursor); + if (ret != 0) { + if (ret == WT_ROLLBACK) { + transaction.rollback(); + return (false); + } else + testutil_die( + ret, "unhandled error while trying to save an update to the tracking table"); + } + transaction.add_op(); + return (true); +} + +bool +thread_context::insert(scoped_cursor &cursor, uint64_t collection_id, uint64_t key_id) +{ + WT_DECL_RET; + std::string key, value; + testutil_assert(tracking != nullptr); + testutil_assert(cursor.get() != nullptr); + + /* + * Get a timestamp to apply to the update. We still do this even if timestamps aren't enabled as + * it will return a value for the tracking table. + */ + wt_timestamp_t ts = tsm->get_next_ts(); + transaction.set_commit_timestamp(ts); + + key = key_to_string(key_id); + value = random_generator::instance().generate_pseudo_random_string(value_size); + + cursor->set_key(cursor.get(), key.c_str()); + cursor->set_value(cursor.get(), value.c_str()); + ret = cursor->insert(cursor.get()); + if (ret != 0) { + if (ret == WT_ROLLBACK) { + transaction.rollback(); + return (false); + } else + testutil_die(ret, "unhandled error while trying to insert a key"); + } + ret = tracking->save_operation( + tracking_operation::INSERT, collection_id, key.c_str(), value.c_str(), ts, op_track_cursor); + if (ret != 0) { + if (ret == WT_ROLLBACK) { + transaction.rollback(); + return (false); + } else + testutil_die( + ret, "unhandled error while trying to save an insert to the tracking table"); + } + transaction.add_op(); + return (true); +} + +int +thread_context::next(scoped_cursor &cursor) +{ + WT_DECL_RET; + + ret = cursor->next(cursor.get()); + + if (ret == WT_NOTFOUND) { + testutil_check(cursor->reset(cursor.get())); + return (ret); + } + + if (ret == WT_ROLLBACK) { + transaction.rollback(); + testutil_check(cursor->reset(cursor.get())); + return (ret); + } + + if (ret != 0) + testutil_die(ret, "cursor->next() failed with an unexpected error."); + + return (0); +} + +void +thread_context::sleep() +{ + _throttle.sleep(); +} + +bool +thread_context::running() const +{ + return (_running); +} +} // namespace test_harness diff --git a/src/third_party/wiredtiger/test/cppsuite/test_harness/workload/thread_context.h b/src/third_party/wiredtiger/test/cppsuite/test_harness/workload/thread_context.h index 63291ac9756..5255740f59c 100644 --- a/src/third_party/wiredtiger/test/cppsuite/test_harness/workload/thread_context.h +++ b/src/third_party/wiredtiger/test/cppsuite/test_harness/workload/thread_context.h @@ -29,80 +29,67 @@ #ifndef THREAD_CONTEXT_H #define THREAD_CONTEXT_H -#include "../core/throttle.h" -#include "../timestamp_manager.h" -#include "database_model.h" -#include "random_generator.h" -#include "workload_tracking.h" +#include <string> -namespace test_harness { -class transaction_context { - public: - explicit transaction_context(configuration *config) - { - configuration *transaction_config = config->get_subconfig(OPS_PER_TRANSACTION); - _min_op_count = transaction_config->get_int(MIN); - _max_op_count = transaction_config->get_int(MAX); - delete transaction_config; - } +extern "C" { +#include "test_util.h" +} - bool - active() const - { - return (_in_txn); - } +#include "../core/throttle.h" +#include "../workload/database_model.h" - /* - * The current transaction can be committed if: A transaction has started and the number of - * operations executed in the current transaction has exceeded the threshold. - */ - bool - can_commit() const - { - return (_in_txn && op_count >= _target_op_count); - } +/* Forward declarations for classes to reduce compilation time and modules coupling. */ +class configuration; +class timestamp_manager; +class workload_tracking; - void - commit(WT_SESSION *session, const std::string &config) - { - /* A transaction cannot be committed if not started. */ - testutil_assert(_in_txn); - testutil_check( - session->commit_transaction(session, config.empty() ? nullptr : config.c_str())); - _in_txn = false; +namespace test_harness { +enum thread_type { READ, INSERT, UPDATE }; + +static std::string +type_string(thread_type type) +{ + switch (type) { + case thread_type::INSERT: + return ("insert"); + case thread_type::READ: + return ("read"); + case thread_type::UPDATE: + return ("update"); + default: + testutil_die(EINVAL, "unexpected thread_type: %d", static_cast<int>(type)); } +} - void - begin(WT_SESSION *session, const std::string &config) - { - if (!_in_txn) { - testutil_check( - session->begin_transaction(session, config.empty() ? nullptr : config.c_str())); - /* This randomizes the number of operations to be executed in one transaction. */ - _target_op_count = - random_generator::instance().generate_integer(_min_op_count, _max_op_count); - op_count = 0; - _in_txn = true; - } - } +class transaction_context { + public: + explicit transaction_context( + configuration *config, timestamp_manager *timestamp_manager, WT_SESSION *session); + + bool active() const; + void add_op(); + void begin(const std::string &config = ""); + /* Begin a transaction if we are not currently in one. */ + void try_begin(const std::string &config = ""); + void commit(const std::string &config = ""); + /* Attempt to commit the transaction given the requirements are met. */ + void try_commit(const std::string &config = ""); + void rollback(const std::string &config = ""); + /* Attempt to rollback the transaction given the requirements are met. */ + void try_rollback(const std::string &config = ""); + /* Set a commit timestamp. */ + void set_commit_timestamp(wt_timestamp_t ts); - /* - * Set a commit timestamp. - */ - void - set_commit_timestamp(WT_SESSION *session, const std::string &ts) - { - std::string config = std::string(COMMIT_TS) + "=" + ts; - testutil_check(session->timestamp_transaction(session, config.c_str())); - } + private: + bool can_commit_rollback(); + private: /* * op_count is the current number of operations that have been executed in the current * transaction. */ - int64_t op_count = 0; + int64_t _op_count = 0; - private: /* * _min_op_count and _max_op_count are the minimum and maximum number of operations within one * transaction. is the current maximum number of operations that can be executed in the current @@ -112,51 +99,67 @@ class transaction_context { int64_t _max_op_count = INT64_MAX; int64_t _target_op_count = 0; bool _in_txn = false; + + WT_SESSION *_session = nullptr; + timestamp_manager *_timestamp_manager = nullptr; }; /* Container class for a thread and any data types it may need to interact with the database. */ class thread_context { public: - thread_context(configuration *config, timestamp_manager *timestamp_manager, - workload_tracking *tracking, database &db) - : database(db), timestamp_manager(timestamp_manager), tracking(tracking), - transaction(transaction_context(config)) - { - session = connection_manager::instance().create_session(); - _throttle = throttle(config); - - /* These won't exist for read threads which is why we use optional here. */ - key_size = config->get_optional_int(KEY_SIZE, 1); - value_size = config->get_optional_int(VALUE_SIZE, 1); - - testutil_assert(key_size > 0 && value_size > 0); - } + thread_context(uint64_t id, thread_type type, configuration *config, + scoped_session &&created_session, timestamp_manager *timestamp_manager, + workload_tracking *tracking, database &dbase); - void - finish() - { - _running = false; - } + virtual ~thread_context() = default; - void - sleep() - { - _throttle.sleep(); - } + void finish(); - bool - running() const - { - return (_running); - } + /* + * Convert a key_id to a string. If the resulting string is less than the given length, padding + * of '0' is added. + */ + std::string key_to_string(uint64_t key_id); - WT_SESSION *session; + /* + * Generic update function, takes a collection_id and key, will generate the value. + * + * Returns true if it successfully updates the key, false if it receives rollback from the API. + */ + bool update(scoped_cursor &cursor, uint64_t collection_id, const std::string &key); + + /* + * Generic insert function, takes a collection_id and key_id, will generate the value. + * + * Returns true if it successfully inserts the key, false if it receives rollback from the API. + */ + bool insert(scoped_cursor &cursor, uint64_t collection_id, uint64_t key_id); + + /* + * Generic next function. + * + * Handles rollback and not found internally, but will return the error code to the caller so + * the caller can distinguish between them. + */ + int next(scoped_cursor &cursor); + + void sleep(); + bool running() const; + + public: + scoped_session session; + scoped_cursor op_track_cursor; transaction_context transaction; - test_harness::timestamp_manager *timestamp_manager; - test_harness::workload_tracking *tracking; - test_harness::database &database; - int64_t key_size = 0; - int64_t value_size = 0; + timestamp_manager *tsm; + workload_tracking *tracking; + database &db; + const int64_t collection_count; + const int64_t key_count; + const int64_t key_size; + const int64_t value_size; + const int64_t thread_count; + const uint64_t id; + const thread_type type; private: throttle _throttle; diff --git a/src/third_party/wiredtiger/test/cppsuite/test_harness/workload/workload_tracking.cxx b/src/third_party/wiredtiger/test/cppsuite/test_harness/workload/workload_tracking.cxx new file mode 100644 index 00000000000..1211749ec28 --- /dev/null +++ b/src/third_party/wiredtiger/test/cppsuite/test_harness/workload/workload_tracking.cxx @@ -0,0 +1,168 @@ +/*- + * Public Domain 2014-present MongoDB, Inc. + * Public Domain 2008-2014 WiredTiger, Inc. + * + * This is free and unencumbered software released into the public domain. + * + * Anyone is free to copy, modify, publish, use, compile, sell, or + * distribute this software, either in source code form or as a compiled + * binary, for any purpose, commercial or non-commercial, and by any + * means. + * + * In jurisdictions that recognize copyright laws, the author or authors + * of this software dedicate any and all copyright interest in the + * software to the public domain. We make this dedication for the benefit + * of the public at large and to the detriment of our heirs and + * successors. We intend this dedication to be an overt act of + * relinquishment in perpetuity of all present and future rights to this + * software under copyright law. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. + * IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + */ + +#include "../connection_manager.h" +#include "../core/configuration.h" +#include "../util/scoped_types.h" +#include "workload_tracking.h" + +namespace test_harness { +workload_tracking::workload_tracking(configuration *_config, + const std::string &operation_table_config, const std::string &operation_table_name, + const std::string &schema_table_config, const std::string &schema_table_name, + const bool use_compression, timestamp_manager &tsm) + : component("workload_tracking", _config), _operation_table_config(operation_table_config), + _operation_table_name(operation_table_name), _schema_table_config(schema_table_config), + _schema_table_name(schema_table_name), _use_compression(use_compression), _tsm(tsm) +{ +} + +const std::string & +workload_tracking::get_schema_table_name() const +{ + return (_schema_table_name); +} + +const std::string & +workload_tracking::get_operation_table_name() const +{ + return (_operation_table_name); +} + +void +workload_tracking::load() +{ + component::load(); + + if (!_enabled) + return; + + /* Initiate schema tracking. */ + _session = connection_manager::instance().create_session(); + testutil_check( + _session->create(_session.get(), _schema_table_name.c_str(), _schema_table_config.c_str())); + _schema_track_cursor = _session.open_scoped_cursor(_schema_table_name.c_str()); + logger::log_msg(LOG_TRACE, "Schema tracking initiated"); + + /* Initiate operations tracking. */ + testutil_check(_session->create( + _session.get(), _operation_table_name.c_str(), _operation_table_config.c_str())); + logger::log_msg(LOG_TRACE, "Operations tracking created"); + + /* + * Open sweep cursor. This cursor will be used to clear out obsolete data from the tracking + * table. + */ + _sweep_cursor = _session.open_scoped_cursor(_operation_table_name.c_str()); + logger::log_msg(LOG_TRACE, "Tracking table sweep initialized"); +} + +void +workload_tracking::do_work() +{ + WT_DECL_RET; + wt_timestamp_t ts, oldest_ts; + uint64_t collection_id, sweep_collection_id; + int op_type; + const char *key, *value; + char *sweep_key; + bool globally_visible_update_found; + + key = sweep_key = nullptr; + globally_visible_update_found = false; + + /* Take a copy of the oldest so that we sweep with a consistent timestamp. */ + oldest_ts = _tsm.get_oldest_ts(); + + while ((ret = _sweep_cursor->prev(_sweep_cursor.get())) == 0) { + testutil_check(_sweep_cursor->get_key(_sweep_cursor.get(), &collection_id, &key, &ts)); + testutil_check(_sweep_cursor->get_value(_sweep_cursor.get(), &op_type, &value)); + /* + * If we're on a new key, reset the check. We want to track whether we have a globally + * visible update for the current key. + */ + if (sweep_key == nullptr || sweep_collection_id != collection_id || + strcmp(sweep_key, key) != 0) { + globally_visible_update_found = false; + if (sweep_key != nullptr) + free(sweep_key); + sweep_key = static_cast<char *>(dstrdup(key)); + sweep_collection_id = collection_id; + } + if (ts <= oldest_ts) { + if (globally_visible_update_found) { + if (logger::trace_level == LOG_TRACE) + logger::log_msg(LOG_TRACE, + std::string("workload tracking: Obsoleted update, key=") + sweep_key + + ", collection_id=" + std::to_string(collection_id) + + ", timestamp=" + std::to_string(ts) + + ", oldest_timestamp=" + std::to_string(oldest_ts) + ", value=" + value); + testutil_check(_sweep_cursor->remove(_sweep_cursor.get())); + } else if (static_cast<tracking_operation>(op_type) == tracking_operation::INSERT) { + if (logger::trace_level == LOG_TRACE) + logger::log_msg(LOG_TRACE, + std::string("workload tracking: Found globally visible update, key=") + + sweep_key + ", collection_id=" + std::to_string(collection_id) + + ", timestamp=" + std::to_string(ts) + + ", oldest_timestamp=" + std::to_string(oldest_ts) + ", value=" + value); + globally_visible_update_found = true; + } + } + } + + free(sweep_key); + + if (ret != WT_NOTFOUND) + testutil_die(LOG_ERROR, + "Tracking table sweep failed: cursor->next() returned an unexpected error %d.", ret); + + /* If we have a position, give it up. */ + testutil_check(_sweep_cursor->reset(_sweep_cursor.get())); +} + +void +workload_tracking::save_schema_operation( + const tracking_operation &operation, const uint64_t &collection_id, wt_timestamp_t ts) +{ + std::string error_message; + + if (!_enabled) + return; + + if (operation == tracking_operation::CREATE_COLLECTION || + operation == tracking_operation::DELETE_COLLECTION) { + _schema_track_cursor->set_key(_schema_track_cursor.get(), collection_id, ts); + _schema_track_cursor->set_value(_schema_track_cursor.get(), static_cast<int>(operation)); + testutil_check(_schema_track_cursor->insert(_schema_track_cursor.get())); + } else { + error_message = + "save_schema_operation: invalid operation " + std::to_string(static_cast<int>(operation)); + testutil_die(EINVAL, error_message.c_str()); + } +} +} // namespace test_harness diff --git a/src/third_party/wiredtiger/test/cppsuite/test_harness/workload/workload_tracking.h b/src/third_party/wiredtiger/test/cppsuite/test_harness/workload/workload_tracking.h index d44ee79e563..704f26ac84b 100644 --- a/src/third_party/wiredtiger/test/cppsuite/test_harness/workload/workload_tracking.h +++ b/src/third_party/wiredtiger/test/cppsuite/test_harness/workload/workload_tracking.h @@ -29,115 +29,65 @@ #ifndef WORKLOAD_TRACKING_H #define WORKLOAD_TRACKING_H +#include "../core/component.h" +#include "../timestamp_manager.h" +#include "../util/scoped_types.h" + /* - * Default schema for tracking operations on collections (key_format: Collection name / Key / + * Default schema for tracking operations on collections (key_format: Collection id / Key / * Timestamp, value_format: Operation type / Value) */ -#define OPERATION_TRACKING_KEY_FORMAT WT_UNCHECKED_STRING(SSQ) +#define OPERATION_TRACKING_KEY_FORMAT WT_UNCHECKED_STRING(QSQ) #define OPERATION_TRACKING_VALUE_FORMAT WT_UNCHECKED_STRING(iS) #define OPERATION_TRACKING_TABLE_CONFIG \ "key_format=" OPERATION_TRACKING_KEY_FORMAT ",value_format=" OPERATION_TRACKING_VALUE_FORMAT /* - * Default schema for tracking schema operations on collections (key_format: Collection name / + * Default schema for tracking schema operations on collections (key_format: Collection id / * Timestamp, value_format: Operation type) */ -#define SCHEMA_TRACKING_KEY_FORMAT WT_UNCHECKED_STRING(SQ) +#define SCHEMA_TRACKING_KEY_FORMAT WT_UNCHECKED_STRING(QQ) #define SCHEMA_TRACKING_VALUE_FORMAT WT_UNCHECKED_STRING(i) #define SCHEMA_TRACKING_TABLE_CONFIG \ "key_format=" SCHEMA_TRACKING_KEY_FORMAT ",value_format=" SCHEMA_TRACKING_VALUE_FORMAT namespace test_harness { /* Tracking operations. */ -enum class tracking_operation { CREATE_COLLECTION, DELETE_COLLECTION, DELETE_KEY, INSERT, UPDATE }; +enum class tracking_operation { CREATE_COLLECTION, DELETE_COLLECTION, DELETE_KEY, INSERT }; + /* Class used to track operations performed on collections */ class workload_tracking : public component { - public: workload_tracking(configuration *_config, const std::string &operation_table_config, const std::string &operation_table_name, const std::string &schema_table_config, - const std::string &schema_table_name) - : component("workload_tracking", _config), _operation_table_config(operation_table_config), - _operation_table_name(operation_table_name), _schema_table_config(schema_table_config), - _schema_table_name(schema_table_name) - { - } - - const std::string & - get_schema_table_name() const - { - return _schema_table_name; - } - - const std::string & - get_operation_table_name() const - { - return _operation_table_name; - } - - void - load() override final - { - WT_SESSION *session; - - component::load(); - - if (!_enabled) - return; + const std::string &schema_table_name, const bool use_compression, timestamp_manager &tsm); - /* Initiate schema tracking. */ - session = connection_manager::instance().create_session(); - testutil_check( - session->create(session, _schema_table_name.c_str(), _schema_table_config.c_str())); - testutil_check( - session->open_cursor(session, _schema_table_name.c_str(), NULL, NULL, &_cursor_schema)); - debug_print("Schema tracking initiated", DEBUG_TRACE); + const std::string &get_schema_table_name() const; + const std::string &get_operation_table_name() const; + void load() override final; - /* Initiate operations tracking. */ - testutil_check( - session->create(session, _operation_table_name.c_str(), _operation_table_config.c_str())); - testutil_check(session->open_cursor( - session, _operation_table_name.c_str(), NULL, NULL, &_cursor_operations)); - debug_print("Operations tracking created", DEBUG_TRACE); - } - - void - run() override final - { - /* Does not do anything. */ - } - - void - save_schema_operation( - const tracking_operation &operation, const std::string &collection_name, wt_timestamp_t ts) - { - std::string error_message; - - if (!_enabled) - return; + /* + * As every operation is tracked in the tracking table we need to clear out obsolete operations + * otherwise the file size grow continuously, as such we cleanup operations that are no longer + * relevant, i.e. older than the oldest timestamp. + */ + void do_work() override final; - if (operation == tracking_operation::CREATE_COLLECTION || - operation == tracking_operation::DELETE_COLLECTION) { - _cursor_schema->set_key(_cursor_schema, collection_name.c_str(), ts); - _cursor_schema->set_value(_cursor_schema, static_cast<int>(operation)); - testutil_check(_cursor_schema->insert(_cursor_schema)); - } else { - error_message = "save_schema_operation: invalid operation " + - std::to_string(static_cast<int>(operation)); - testutil_die(EINVAL, error_message.c_str()); - } - debug_print("save_schema_operation: workload tracking saved operation.", DEBUG_TRACE); - } + void save_schema_operation( + const tracking_operation &operation, const uint64_t &collection_id, wt_timestamp_t ts); template <typename K, typename V> - void - save_operation(const tracking_operation &operation, const std::string &collection_name, - const K &key, const V &value, wt_timestamp_t ts) + int + save_operation(const tracking_operation &operation, const uint64_t &collection_id, const K &key, + const V &value, wt_timestamp_t ts, scoped_cursor &op_track_cursor) { + WT_DECL_RET; std::string error_message; if (!_enabled) - return; + return (0); + + testutil_assert(op_track_cursor.get() != nullptr); if (operation == tracking_operation::CREATE_COLLECTION || operation == tracking_operation::DELETE_COLLECTION) { @@ -145,20 +95,23 @@ class workload_tracking : public component { "save_operation: invalid operation " + std::to_string(static_cast<int>(operation)); testutil_die(EINVAL, error_message.c_str()); } else { - _cursor_operations->set_key(_cursor_operations, collection_name.c_str(), key, ts); - _cursor_operations->set_value(_cursor_operations, static_cast<int>(operation), value); - testutil_check(_cursor_operations->insert(_cursor_operations)); + op_track_cursor->set_key(op_track_cursor.get(), collection_id, key, ts); + op_track_cursor->set_value(op_track_cursor.get(), static_cast<int>(operation), value); + ret = op_track_cursor->insert(op_track_cursor.get()); } - debug_print("save_operation: workload tracking saved operation.", DEBUG_TRACE); + return (ret); } private: - WT_CURSOR *_cursor_operations = nullptr; - WT_CURSOR *_cursor_schema = nullptr; + scoped_session _session; + scoped_cursor _schema_track_cursor; + scoped_cursor _sweep_cursor; const std::string _operation_table_config; const std::string _operation_table_name; const std::string _schema_table_config; const std::string _schema_table_name; + const bool _use_compression; + timestamp_manager &_tsm; }; } // namespace test_harness diff --git a/src/third_party/wiredtiger/test/cppsuite/test_harness/workload/workload_validation.cxx b/src/third_party/wiredtiger/test/cppsuite/test_harness/workload/workload_validation.cxx new file mode 100644 index 00000000000..833ec91d9e8 --- /dev/null +++ b/src/third_party/wiredtiger/test/cppsuite/test_harness/workload/workload_validation.cxx @@ -0,0 +1,279 @@ +/*- + * Public Domain 2014-present MongoDB, Inc. + * Public Domain 2008-2014 WiredTiger, Inc. + * + * This is free and unencumbered software released into the public domain. + * + * Anyone is free to copy, modify, publish, use, compile, sell, or + * distribute this software, either in source code form or as a compiled + * binary, for any purpose, commercial or non-commercial, and by any + * means. + * + * In jurisdictions that recognize copyright laws, the author or authors + * of this software dedicate any and all copyright interest in the + * software to the public domain. We make this dedication for the benefit + * of the public at large and to the detriment of our heirs and + * successors. We intend this dedication to be an overt act of + * relinquishment in perpetuity of all present and future rights to this + * software under copyright law. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. + * IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + */ + +#include <algorithm> + +#include "../connection_manager.h" +#include "database_model.h" +#include "workload_validation.h" + +namespace test_harness { +void +workload_validation::validate(const std::string &operation_table_name, + const std::string &schema_table_name, const std::vector<uint64_t> &known_collection_ids) +{ + WT_DECL_RET; + wt_timestamp_t tracked_timestamp; + std::vector<uint64_t> created_collections, deleted_collections; + uint64_t tracked_collection_id; + const char *tracked_key, *tracked_value; + int tracked_op_type; + uint64_t current_collection_id = 0; + + logger::log_msg(LOG_INFO, "Beginning validation."); + + scoped_session session = connection_manager::instance().create_session(); + + /* Retrieve the collections that were created and deleted during the test. */ + parse_schema_tracking_table( + session, schema_table_name, created_collections, deleted_collections); + + /* + * Make sure the deleted collections do not exist on disk. The created collections are checked + * in check_reference. + */ + for (auto const &it : deleted_collections) { + if (!verify_collection_file_state(session, it, false)) + testutil_die(LOG_ERROR, + "Validation failed: collection %s present on disk while it has been tracked as " + "deleted.", + database::build_collection_name(it).c_str()); + } + + /* + * All collections in memory should match those created in the schema tracking table. Dropping + * is currently not supported. + */ + std::sort(created_collections.begin(), created_collections.end()); + auto on_disk_collection_id = created_collections.begin(); + if (created_collections.size() != known_collection_ids.size()) + testutil_die(LOG_ERROR, + "Validation failed: collection state mismatch, expected %lu" + " collections to exist but have %lu on disk", + created_collections.size(), known_collection_ids.size()); + for (const auto id : known_collection_ids) { + if (id != *on_disk_collection_id) + testutil_die(LOG_ERROR, + "Validation failed: collection state mismatch expected " + "collection id %lu but got %lu.", + id, *on_disk_collection_id); + on_disk_collection_id++; + } + + /* Parse the tracking table. */ + validation_collection current_collection_records; + scoped_cursor cursor = session.open_scoped_cursor(operation_table_name.c_str()); + while ((ret = cursor->next(cursor.get())) == 0) { + testutil_check( + cursor->get_key(cursor.get(), &tracked_collection_id, &tracked_key, &tracked_timestamp)); + testutil_check(cursor->get_value(cursor.get(), &tracked_op_type, &tracked_value)); + + logger::log_msg(LOG_TRACE, + "Retrieved tracked values. \n Collection id: " + std::to_string(tracked_collection_id) + + "\n Key: " + std::string(tracked_key) + + "\n Timestamp: " + std::to_string(tracked_timestamp) + "\n Operation type: " + + std::to_string(tracked_op_type) + "\n Value: " + std::string(tracked_value)); + + /* + * Check if we've stepped over to the next collection. The tracking table is sorted by + * collection_id so this is correct. + */ + if (tracked_collection_id != current_collection_id) { + if (std::find(known_collection_ids.begin(), known_collection_ids.end(), + tracked_collection_id) == known_collection_ids.end()) + testutil_die(LOG_ERROR, + "Validation failed: The collection id %lu is not part of the known " + "collection set.", + tracked_collection_id); + if (tracked_collection_id < current_collection_id) + testutil_die(LOG_ERROR, "Validation failed: The collection id %lu is out of order.", + tracked_collection_id); + + /* + * Given that we've stepped over to the next collection we've built a full picture of + * the current collection and can now validate it. + */ + verify_collection(session, current_collection_id, current_collection_records); + + /* Begin processing the next collection. */ + current_collection_id = tracked_collection_id; + current_collection_records.clear(); + } + + /* + * Add the values from the tracking table to the current collection model. + */ + update_data_model(static_cast<tracking_operation>(tracked_op_type), + current_collection_records, current_collection_id, tracked_key, tracked_value); + }; + + /* The value of ret should be WT_NOTFOUND once the cursor has read all rows. */ + if (ret != WT_NOTFOUND) + testutil_die( + LOG_ERROR, "Validation failed: cursor->next() return an unexpected error %d.", ret); + + /* + * We still need to validate the last collection. But we can also end up here if there aren't + * any collections, check for that. + */ + if (known_collection_ids.size() != 0) + verify_collection(session, current_collection_id, current_collection_records); +} + +void +workload_validation::parse_schema_tracking_table(scoped_session &session, + const std::string &tracking_table_name, std::vector<uint64_t> &created_collections, + std::vector<uint64_t> &deleted_collections) +{ + wt_timestamp_t key_timestamp; + uint64_t key_collection_id; + int value_operation_type; + + scoped_cursor cursor = session.open_scoped_cursor(tracking_table_name.c_str()); + + while (cursor->next(cursor.get()) == 0) { + testutil_check(cursor->get_key(cursor.get(), &key_collection_id, &key_timestamp)); + testutil_check(cursor->get_value(cursor.get(), &value_operation_type)); + + logger::log_msg(LOG_TRACE, "Collection id is " + std::to_string(key_collection_id)); + logger::log_msg(LOG_TRACE, "Timestamp is " + std::to_string(key_timestamp)); + logger::log_msg(LOG_TRACE, "Operation type is " + std::to_string(value_operation_type)); + + if (static_cast<tracking_operation>(value_operation_type) == + tracking_operation::CREATE_COLLECTION) { + deleted_collections.erase(std::remove(deleted_collections.begin(), + deleted_collections.end(), key_collection_id), + deleted_collections.end()); + created_collections.push_back(key_collection_id); + } else if (static_cast<tracking_operation>(value_operation_type) == + tracking_operation::DELETE_COLLECTION) { + created_collections.erase(std::remove(created_collections.begin(), + created_collections.end(), key_collection_id), + created_collections.end()); + deleted_collections.push_back(key_collection_id); + } + } +} + +void +workload_validation::update_data_model(const tracking_operation &operation, + validation_collection &collection, const uint64_t collection_id, const char *key, + const char *value) +{ + if (operation == tracking_operation::DELETE_KEY) { + /* Search for the key validating that it exists. */ + const auto it = collection.find(key); + if (it == collection.end()) + testutil_die(LOG_ERROR, + "Validation failed: key deleted that doesn't exist. Collection id: %lu Key: %s", + collection_id, key); + else if (it->second.exists == false) + /* The key has been deleted twice. */ + testutil_die(LOG_ERROR, + "Validation failed: deleted key deleted again. Collection id: %lu Key: %s", + collection_id, it->first.c_str()); + + /* Update the key_state to deleted. */ + it->second.exists = false; + } else if (operation == tracking_operation::INSERT) + collection[key_value_t(key)] = key_state{true, key_value_t(value)}; + else + testutil_die(LOG_ERROR, "Validation failed: unexpected operation in the tracking table: %d", + static_cast<tracking_operation>(operation)); +} + +void +workload_validation::verify_collection( + scoped_session &session, const uint64_t collection_id, validation_collection &collection) +{ + /* Check the collection exists on disk. */ + if (!verify_collection_file_state(session, collection_id, true)) + testutil_die(LOG_ERROR, + "Validation failed: collection %lu not present on disk while it has been tracked as " + "created.", + collection_id); + + /* Walk through each key/value pair of the current collection. */ + for (const auto &record : collection) + verify_key_value(session, collection_id, record.first, record.second); +} + +bool +workload_validation::verify_collection_file_state( + scoped_session &session, const uint64_t collection_id, bool exists) const +{ + /* + * We don't necessarily expect to successfully open the cursor so don't create a scoped cursor. + */ + WT_CURSOR *cursor; + int ret = session->open_cursor(session.get(), + database::build_collection_name(collection_id).c_str(), nullptr, nullptr, &cursor); + if (ret == 0) + testutil_check(cursor->close(cursor)); + return (exists ? (ret == 0) : (ret != 0)); +} + +void +workload_validation::verify_key_value(scoped_session &session, const uint64_t collection_id, + const std::string &key, const key_state &key_state) +{ + WT_DECL_RET; + const char *retrieved_value; + + scoped_cursor cursor = + session.open_scoped_cursor(database::build_collection_name(collection_id).c_str()); + cursor->set_key(cursor.get(), key.c_str()); + ret = cursor->search(cursor.get()); + if (ret != 0) { + if (ret == WT_NOTFOUND && key_state.exists) + testutil_die(LOG_ERROR, + "Validation failed: Search failed to find key that should exist. Key: %s, " + "Collection_id: %lu", + key.c_str(), collection_id); + else + testutil_die(LOG_ERROR, + "Validation failed: Unexpected error while searching for a key. Key: %s, " + "Collection_id: %lu", + key.c_str(), collection_id); + } + if (key_state.exists == false) { + if (ret == 0) + testutil_die(LOG_ERROR, + "Validation failed: Key exists when it is expected to be deleted. Key: %s, " + "Collection_id: %lu", + key.c_str(), collection_id); + return; + } + testutil_check(cursor->get_value(cursor.get(), &retrieved_value)); + if (key_state.value != key_value_t(retrieved_value)) + testutil_die(LOG_ERROR, + "Validation failed: Value mismatch for key. Key: %s, Collection_id: %lu, Expected " + "value: %s, Found value: %s", + key.c_str(), collection_id, key_state.value.c_str(), retrieved_value); +} +} // namespace test_harness diff --git a/src/third_party/wiredtiger/test/cppsuite/test_harness/workload/workload_validation.h b/src/third_party/wiredtiger/test/cppsuite/test_harness/workload/workload_validation.h index 7f583a66888..ba745a225df 100644 --- a/src/third_party/wiredtiger/test/cppsuite/test_harness/workload/workload_validation.h +++ b/src/third_party/wiredtiger/test/cppsuite/test_harness/workload/workload_validation.h @@ -29,16 +29,24 @@ #ifndef WORKLOAD_VALIDATION_H #define WORKLOAD_VALIDATION_H +#include <map> #include <string> +#include <vector> extern "C" { #include "wiredtiger.h" } -#include "database_model.h" +#include "../util/scoped_types.h" namespace test_harness { - +struct key_state { + key_state() = default; + key_state(bool exists, const key_value_t &value) : exists(exists), value(value) {} + bool exists = false; + key_value_t value; +}; +typedef std::map<key_value_t, key_state> validation_collection; /* * Class that can validate database state and collection data. */ @@ -47,97 +55,13 @@ class workload_validation { /* * Validate the on disk data against what has been tracked during the test. This is done by * replaying the tracked operations so a representation in memory of the collections is created. - * This representation is then compared to what is on disk. operation_table_name: collection - * that contains all the operations about the key/value pairs in the different collections used - * during the test. schema_table_name: collection that contains all the operations about the - * creation or deletion of collections during the test. + * This representation is then compared to what is on disk. + * + * - operation_table_name: Table that contains all the operations performed on keys. + * - schema_table_name: Table that contains all the schema operations performed. */ - void - validate(const std::string &operation_table_name, const std::string &schema_table_name, - database &database) - { - WT_DECL_RET; - WT_CURSOR *cursor; - WT_SESSION *session; - wt_timestamp_t key_timestamp; - std::vector<std::string> created_collections, deleted_collections; - const char *key, *key_collection_name, *value; - int value_operation_type; - std::string collection_name; - - session = connection_manager::instance().create_session(); - - /* Retrieve the collections that were created and deleted during the test. */ - parse_schema_tracking_table( - session, schema_table_name, created_collections, deleted_collections); - - /* - * Make sure the deleted collections do not exist on disk. The created collections are - * checked in check_reference. - */ - for (auto const &it : deleted_collections) { - if (!verify_collection_state(session, it, false)) - testutil_die(DEBUG_ERROR, - "validate: collection %s present on disk while it has been tracked as deleted.", - it.c_str()); - } - - /* Parse the tracking table. */ - testutil_check( - session->open_cursor(session, operation_table_name.c_str(), NULL, NULL, &cursor)); - while ((ret = cursor->next(cursor)) == 0) { - testutil_check(cursor->get_key(cursor, &key_collection_name, &key, &key_timestamp)); - testutil_check(cursor->get_value(cursor, &value_operation_type, &value)); - - debug_print("Collection name is " + std::string(key_collection_name), DEBUG_TRACE); - debug_print("Key is " + std::string(key), DEBUG_TRACE); - debug_print("Timestamp is " + std::to_string(key_timestamp), DEBUG_TRACE); - debug_print("Operation type is " + std::to_string(value_operation_type), DEBUG_TRACE); - debug_print("Value is " + std::string(value), DEBUG_TRACE); - - /* - * If the cursor points to values from a collection that has been created during the - * test, update the data model. - */ - if (std::find(created_collections.begin(), created_collections.end(), - key_collection_name) != created_collections.end()) - update_data_model(static_cast<tracking_operation>(value_operation_type), - key_collection_name, key, value, database); - /* - * The collection should be part of the deleted collections if it has not be found in - * the created ones. - */ - else if (std::find(deleted_collections.begin(), deleted_collections.end(), - key_collection_name) == deleted_collections.end()) - testutil_die(DEBUG_ERROR, - "validate: The collection %s is not part of the created or deleted collections.", - key_collection_name); - - if (collection_name.empty()) - collection_name = key_collection_name; - else if (collection_name != key_collection_name) { - /* - * The data model is now fully updated for the last read collection. It can be - * checked. - */ - check_reference(session, collection_name, database); - collection_name = key_collection_name; - } - }; - - /* The value of ret should be WT_NOTFOUND once the cursor has read all rows. */ - if (ret != WT_NOTFOUND) - testutil_die(DEBUG_ERROR, "validate: cursor->next() %d.", ret); - - /* - * Once the cursor has read the entire table, the last parsed collection has not been - * checked yet. We still have to make sure collection_name has been updated. It will remain - * empty if there is no collections to check after the end of the test (no collections - * created or all deleted). - */ - if (!collection_name.empty()) - check_reference(session, collection_name, database); - } + void validate(const std::string &operation_table_name, const std::string &schema_table_name, + const std::vector<uint64_t> &known_collection_ids); private: /* @@ -145,158 +69,30 @@ class workload_validation { * collection_name: collection that contains the operations on the different collections during * the test. */ - void - parse_schema_tracking_table(WT_SESSION *session, const std::string &collection_name, - std::vector<std::string> &created_collections, std::vector<std::string> &deleted_collections) - { - WT_CURSOR *cursor; - wt_timestamp_t key_timestamp; - const char *key_collection_name; - int value_operation_type; - - testutil_check(session->open_cursor(session, collection_name.c_str(), NULL, NULL, &cursor)); - - while (cursor->next(cursor) == 0) { - testutil_check(cursor->get_key(cursor, &key_collection_name, &key_timestamp)); - testutil_check(cursor->get_value(cursor, &value_operation_type)); - - debug_print("Collection name is " + std::string(key_collection_name), DEBUG_TRACE); - debug_print("Timestamp is " + std::to_string(key_timestamp), DEBUG_TRACE); - debug_print("Operation type is " + std::to_string(value_operation_type), DEBUG_TRACE); - - if (static_cast<tracking_operation>(value_operation_type) == - tracking_operation::CREATE_COLLECTION) { - deleted_collections.erase(std::remove(deleted_collections.begin(), - deleted_collections.end(), key_collection_name), - deleted_collections.end()); - created_collections.push_back(key_collection_name); - } else if (static_cast<tracking_operation>(value_operation_type) == - tracking_operation::DELETE_COLLECTION) { - created_collections.erase(std::remove(created_collections.begin(), - created_collections.end(), key_collection_name), - created_collections.end()); - deleted_collections.push_back(key_collection_name); - } - } - } + void parse_schema_tracking_table(scoped_session &session, + const std::string &tracking_table_name, std::vector<uint64_t> &created_collections, + std::vector<uint64_t> &deleted_collections); /* Update the data model. */ - void - update_data_model(const tracking_operation &operation, const std::string &collection_name, - const char *key, const char *value, database &database) - { - switch (operation) { - case tracking_operation::DELETE_KEY: - /* - * Operations are parsed from the oldest to the most recent one. It is safe to assume - * the key has been inserted previously in an existing collection and can be safely - * deleted. - */ - database.delete_record(collection_name, key); - break; - case tracking_operation::INSERT: { - /* - * Keys are unique, it is safe to assume the key has not been encountered before. - */ - database.insert_record(collection_name, key, value); - break; - } - case tracking_operation::UPDATE: - database.update_record(collection_name, key, value); - break; - default: - testutil_die(DEBUG_ERROR, "Unexpected operation in the tracking table: %d", - static_cast<tracking_operation>(operation)); - break; - } - } + void update_data_model(const tracking_operation &operation, validation_collection &collection, + const uint64_t collection_id, const char *key, const char *value); /* - * Compare the tracked operations against what has been saved on disk. collection: - * representation in memory of the collection values and keys according to the tracking table. + * Compare the tracked operations against what has been saved on disk. */ - void - check_reference(WT_SESSION *session, const std::string &collection_name, database &database) - { - bool is_valid; - key_t key; - key_value_t key_str; - - /* Check the collection exists on disk. */ - if (!verify_collection_state(session, collection_name, true)) - testutil_die(DEBUG_ERROR, - "check_reference: collection %s not present on disk while it has been tracked as " - "created.", - collection_name.c_str()); - - /* Walk through each key/value pair of the current collection. */ - for (const auto &keys : database.get_keys(collection_name)) { - key_str = keys.first; - key = keys.second; - /* The key/value pair exists. */ - if (key.exists) - is_valid = (is_key_present(session, collection_name, key_str.c_str()) == true); - /* The key has been deleted. */ - else - is_valid = (is_key_present(session, collection_name, key_str.c_str()) == false); - - if (!is_valid) - testutil_die(DEBUG_ERROR, "check_reference: failed for key %s in collection %s.", - key_str.c_str(), collection_name.c_str()); - - /* Check the associated value is valid. */ - if (key.exists) { - if (!verify_value(session, collection_name, key_str.c_str(), - database.get_record(collection_name, key_str.c_str()).value)) - testutil_die(DEBUG_ERROR, - "check_reference: failed for key %s / value %s in collection %s.", - key_str.c_str(), - database.get_record(collection_name, key_str.c_str()).value.c_str(), - collection_name.c_str()); - } - } - } + void verify_collection( + scoped_session &session, const uint64_t collection_id, validation_collection &collection); /* * Check whether a collection exists on disk. exists: needs to be set to true if the collection * is expected to be existing, false otherwise. */ - bool - verify_collection_state( - WT_SESSION *session, const std::string &collection_name, bool exists) const - { - WT_CURSOR *cursor; - int ret = session->open_cursor(session, collection_name.c_str(), NULL, NULL, &cursor); - return (exists ? (ret == 0) : (ret != 0)); - } - - /* Check whether a keys exists in a collection on disk. */ - template <typename K> - bool - is_key_present(WT_SESSION *session, const std::string &collection_name, const K &key) - { - WT_CURSOR *cursor; - testutil_check(session->open_cursor(session, collection_name.c_str(), NULL, NULL, &cursor)); - cursor->set_key(cursor, key); - return (cursor->search(cursor) == 0); - } + bool verify_collection_file_state( + scoped_session &session, const uint64_t collection_id, bool exists) const; /* Verify the given expected value is the same on disk. */ - template <typename K, typename V> - bool - verify_value(WT_SESSION *session, const std::string &collection_name, const K &key, - const V &expected_value) - { - WT_CURSOR *cursor; - const char *value; - - testutil_check(session->open_cursor(session, collection_name.c_str(), NULL, NULL, &cursor)); - cursor->set_key(cursor, key); - testutil_check(cursor->search(cursor)); - testutil_check(cursor->get_value(cursor, &value)); - - return (key_value_t(value) == expected_value); - } + void verify_key_value(scoped_session &session, const uint64_t collection_id, + const std::string &key, const key_state &key_state); }; } // namespace test_harness diff --git a/src/third_party/wiredtiger/test/cppsuite/test_harness/workload_generator.cxx b/src/third_party/wiredtiger/test/cppsuite/test_harness/workload_generator.cxx new file mode 100644 index 00000000000..32676a59a6f --- /dev/null +++ b/src/third_party/wiredtiger/test/cppsuite/test_harness/workload_generator.cxx @@ -0,0 +1,149 @@ +/*- + * Public Domain 2014-present MongoDB, Inc. + * Public Domain 2008-2014 WiredTiger, Inc. + * + * This is free and unencumbered software released into the public domain. + * + * Anyone is free to copy, modify, publish, use, compile, sell, or + * distribute this software, either in source code form or as a compiled + * binary, for any purpose, commercial or non-commercial, and by any + * means. + * + * In jurisdictions that recognize copyright laws, the author or authors + * of this software dedicate any and all copyright interest in the + * software to the public domain. We make this dedication for the benefit + * of the public at large and to the detriment of our heirs and + * successors. We intend this dedication to be an overt act of + * relinquishment in perpetuity of all present and future rights to this + * software under copyright law. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. + * IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + */ + +#include <atomic> +#include <map> + +#include "connection_manager.h" +#include "core/configuration.h" +#include "core/throttle.h" +#include "util/api_const.h" +#include "workload/database_model.h" +#include "workload/database_operation.h" +#include "workload/random_generator.h" +#include "workload/workload_tracking.h" +#include "workload_generator.h" + +namespace test_harness { +/* operation_config class implementation */ +operation_config::operation_config(configuration *config, thread_type type) + : config(config), type(type), thread_count(config->get_int(THREAD_COUNT)) +{ +} + +std::function<void(test_harness::thread_context *)> +operation_config::get_func(database_operation *dbo) +{ + switch (type) { + case thread_type::INSERT: + return (std::bind(&database_operation::insert_operation, dbo, std::placeholders::_1)); + case thread_type::READ: + return (std::bind(&database_operation::read_operation, dbo, std::placeholders::_1)); + case thread_type::UPDATE: + return (std::bind(&database_operation::update_operation, dbo, std::placeholders::_1)); + default: + /* This may cause a separate testutil_die in type_string but that should be okay. */ + testutil_die(EINVAL, "unexpected thread_type: %s", type_string(type).c_str()); + } +} + +/* workload_generator class implementation */ +workload_generator::workload_generator(configuration *configuration, + database_operation *db_operation, timestamp_manager *timestamp_manager, + workload_tracking *tracking, database &database) + : component("workload_generator", configuration), _database(database), + _database_operation(db_operation), _timestamp_manager(timestamp_manager), _tracking(tracking) +{ +} + +workload_generator::~workload_generator() +{ + for (auto &it : _workers) + delete it; +} + +void +workload_generator::run() +{ + configuration *populate_config; + std::vector<operation_config> operation_configs; + uint64_t thread_id = 0; + + /* Retrieve useful parameters from the test configuration. */ + operation_configs.push_back( + operation_config(_config->get_subconfig(INSERT_CONFIG), thread_type::INSERT)); + operation_configs.push_back( + operation_config(_config->get_subconfig(READ_CONFIG), thread_type::READ)); + operation_configs.push_back( + operation_config(_config->get_subconfig(UPDATE_CONFIG), thread_type::UPDATE)); + populate_config = _config->get_subconfig(POPULATE_CONFIG); + + /* Populate the database. */ + _database_operation->populate(_database, _timestamp_manager, populate_config, _tracking); + _db_populated = true; + delete populate_config; + + /* Generate threads to execute read operations on the collections. */ + for (auto &it : operation_configs) { + if (it.thread_count != 0) + logger::log_msg(LOG_INFO, + "Workload_generator: Creating " + std::to_string(it.thread_count) + " " + + type_string(it.type) + " threads."); + for (size_t i = 0; i < it.thread_count && _running; ++i) { + thread_context *tc = new thread_context(thread_id++, it.type, it.config, + connection_manager::instance().create_session(), _timestamp_manager, _tracking, + _database); + _workers.push_back(tc); + _thread_manager.add_thread(it.get_func(_database_operation), tc); + } + /* + * Don't forget to delete the config we created earlier. While we do pass the config into + * the thread context it is not saved, so we are safe to do this. + */ + delete it.config; + + /* + * Reset the thread_id counter to 0 as we're only interested in knowing per operation type + * which thread we are. + */ + thread_id = 0; + } +} + +void +workload_generator::finish() +{ + component::finish(); + for (const auto &it : _workers) + it->finish(); + _thread_manager.join(); + logger::log_msg(LOG_TRACE, "Workload generator: run stage done"); +} + +database & +workload_generator::get_database() +{ + return (_database); +} + +bool +workload_generator::db_populated() const +{ + return (_db_populated); +} +} // namespace test_harness diff --git a/src/third_party/wiredtiger/test/cppsuite/test_harness/workload_generator.h b/src/third_party/wiredtiger/test/cppsuite/test_harness/workload_generator.h index 1980cf6ac6c..e29410f6116 100644 --- a/src/third_party/wiredtiger/test/cppsuite/test_harness/workload_generator.h +++ b/src/third_party/wiredtiger/test/cppsuite/test_harness/workload_generator.h @@ -30,98 +30,55 @@ #define WORKLOAD_GENERATOR_H #include <algorithm> -#include <atomic> -#include <map> +#include <functional> -#include "core/throttle.h" -#include "workload/database_model.h" +#include "core/component.h" +#include "workload/thread_context.h" +#include "thread_manager.h" #include "workload/database_operation.h" -#include "workload/random_generator.h" -#include "workload/workload_tracking.h" + +/* Forward declarations for classes to reduce compilation time and modules coupling. */ +class configuration; +class database; +class workload_tracking; namespace test_harness { /* + * Helper class to enable scalable operation types in the database_operation. + */ +class operation_config { + public: + operation_config(configuration *config, thread_type type); + + /* Returns a function pointer to the member function of the supplied database operation. */ + std::function<void(test_harness::thread_context *)> get_func(database_operation *dbo); + + public: + configuration *config; + const thread_type type; + const int64_t thread_count; +}; + +/* * Class that can execute operations based on a given configuration. */ class workload_generator : public component { public: workload_generator(configuration *configuration, database_operation *db_operation, - timestamp_manager *timestamp_manager, workload_tracking *tracking, database &database) - : component("workload_generator", configuration), _database(database), - _database_operation(db_operation), _timestamp_manager(timestamp_manager), - _tracking(tracking) - { - } + timestamp_manager *timestamp_manager, workload_tracking *tracking, database &database); - ~workload_generator() - { - for (auto &it : _workers) - delete it; - } + ~workload_generator(); /* Delete the copy constructor and the assignment operator. */ workload_generator(const workload_generator &) = delete; workload_generator &operator=(const workload_generator &) = delete; /* Do the work of the main part of the workload. */ - void - run() override final - { - configuration *read_config, *update_config, *insert_config; - - /* Populate the database. */ - _database_operation->populate(_database, _timestamp_manager, _config, _tracking); - _db_populated = true; - - /* Retrieve useful parameters from the test configuration. */ - update_config = _config->get_subconfig(UPDATE_CONFIG); - insert_config = _config->get_subconfig(INSERT_CONFIG); - read_config = _config->get_subconfig(READ_CONFIG); - - /* Generate threads to execute read operations on the collections. */ - for (size_t i = 0; i < read_config->get_int(THREAD_COUNT) && _running; ++i) { - thread_context *tc = - new thread_context(read_config, _timestamp_manager, _tracking, _database); - _workers.push_back(tc); - _thread_manager.add_thread( - &database_operation::read_operation, _database_operation, tc); - } - - /* Generate threads to execute update operations on the collections. */ - for (size_t i = 0; i < update_config->get_int(THREAD_COUNT) && _running; ++i) { - thread_context *tc = - new thread_context(update_config, _timestamp_manager, _tracking, _database); - _workers.push_back(tc); - _thread_manager.add_thread( - &database_operation::update_operation, _database_operation, tc); - } - - delete read_config; - delete update_config; - delete insert_config; - } - - void - finish() override final - { - component::finish(); - for (const auto &it : _workers) - it->finish(); - _thread_manager.join(); - debug_print("Workload generator: run stage done", DEBUG_TRACE); - } - - database & - get_database() - { - return (_database); - } + void run() override final; + void finish() override final; - bool - db_populated() const - { - return (_db_populated); - } + database &get_database(); + bool db_populated() const; private: database &_database; diff --git a/src/third_party/wiredtiger/test/cppsuite/tests/poc_test.cxx b/src/third_party/wiredtiger/test/cppsuite/tests/base_test.cxx index e1797747e40..7bb6e23205e 100755..100644 --- a/src/third_party/wiredtiger/test/cppsuite/tests/poc_test.cxx +++ b/src/third_party/wiredtiger/test/cppsuite/tests/base_test.cxx @@ -28,7 +28,13 @@ #include "test_harness/test.h" -class poc_test : public test_harness::test { +/* + * The "base test" that the framework uses, because its not overloading any of the database + * operation methods it will perform as they are defined and is therefore the "base". + * + * Can be used to create stress tests in various ways. + */ +class base_test : public test_harness::test { public: - poc_test(const std::string &config, const std::string &name) : test(config, name) {} + base_test(const test_harness::test_args &args) : test(args) {} }; diff --git a/src/third_party/wiredtiger/test/cppsuite/tests/example_test.cxx b/src/third_party/wiredtiger/test/cppsuite/tests/example_test.cxx index ab8c38dec16..a226fa33ff3 100644 --- a/src/third_party/wiredtiger/test/cppsuite/tests/example_test.cxx +++ b/src/third_party/wiredtiger/test/cppsuite/tests/example_test.cxx @@ -29,17 +29,17 @@ #include "test_harness/test.h" /* - * Class that defines operations that do nothing as an example. - * This shows how database operations can be overriden and customized. + * Class that defines operations that do nothing as an example. This shows how database operations + * can be overriden and customized. */ class example_test : public test_harness::test { public: - example_test(const std::string &config, const std::string &name) : test(config, name) {} + example_test(const test_harness::test_args &args) : test(args) {} void populate(test_harness::database &database, test_harness::timestamp_manager *_timestamp_manager, - test_harness::configuration *_config, test_harness::workload_tracking *tracking) - override final + test_harness::configuration *_config, + test_harness::workload_tracking *tracking) override final { std::cout << "populate: nothing done." << std::endl; } diff --git a/src/third_party/wiredtiger/test/cppsuite/tests/hs_cleanup.cxx b/src/third_party/wiredtiger/test/cppsuite/tests/hs_cleanup.cxx new file mode 100644 index 00000000000..4dd073cc115 --- /dev/null +++ b/src/third_party/wiredtiger/test/cppsuite/tests/hs_cleanup.cxx @@ -0,0 +1,88 @@ +/*- + * Public Domain 2014-present MongoDB, Inc. + * Public Domain 2008-2014 WiredTiger, Inc. + * + * This is free and unencumbered software released into the public domain. + * + * Anyone is free to copy, modify, publish, use, compile, sell, or + * distribute this software, either in source code form or as a compiled + * binary, for any purpose, commercial or non-commercial, and by any + * means. + * + * In jurisdictions that recognize copyright laws, the author or authors + * of this software dedicate any and all copyright interest in the + * software to the public domain. We make this dedication for the benefit + * of the public at large and to the detriment of our heirs and + * successors. We intend this dedication to be an overt act of + * relinquishment in perpetuity of all present and future rights to this + * software under copyright law. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. + * IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + */ + +#include "test_harness/test.h" + +#include "test_harness/connection_manager.h" + +using namespace test_harness; + +/* + * Here we want to age out entire pages, i.e. the stop time pair on a page should be globally + * visible. To do so we'll update ranges of keys with increasing timestamps which will age out the + * pre-existing data. It may not trigger a cleanup on the data file but should result in data + * getting cleaned up from the history store. + * + * This is then tracked using the associated statistic which can be found in the runtime_monitor. + */ +class hs_cleanup : public test { + public: + hs_cleanup(const test_args &args) : test(args) {} + + void + update_operation(thread_context *tc) override final + { + logger::log_msg( + LOG_INFO, type_string(tc->type) + " thread {" + std::to_string(tc->id) + "} commencing."); + + const char *key_tmp; + + collection &coll = tc->db.get_collection(tc->id); + + /* In this test each thread gets a single collection. */ + testutil_assert(tc->db.get_collection_count() == tc->thread_count); + scoped_cursor cursor = tc->session.open_scoped_cursor(coll.name.c_str()); + + /* We don't know the keyrange we're operating over here so we can't be much smarter here. */ + while (tc->running()) { + tc->sleep(); + + if (tc->next(cursor) != 0) + continue; + + testutil_check(cursor->get_key(cursor.get(), &key_tmp)); + + /* Start a transaction if possible. */ + tc->transaction.try_begin(); + + /* + * The retrieved key needs to be passed inside the update function. However, the update + * API doesn't guarantee our buffer will still be valid once it is called, as such we + * copy the buffer and then pass it into the API. + */ + if (!tc->update(cursor, coll.id, key_value_t(key_tmp))) + continue; + + /* Commit our transaction. */ + tc->transaction.try_commit(); + } + /* Ensure our last transaction is resolved. */ + if (tc->transaction.active()) + tc->transaction.commit(); + } +}; diff --git a/src/third_party/wiredtiger/test/cppsuite/tests/run.cxx b/src/third_party/wiredtiger/test/cppsuite/tests/run.cxx index 67d77116cf1..4e436ff7af3 100755 --- a/src/third_party/wiredtiger/test/cppsuite/tests/run.cxx +++ b/src/third_party/wiredtiger/test/cppsuite/tests/run.cxx @@ -30,11 +30,12 @@ #include <iostream> #include <string> -#include "test_harness/util/debug_utils.h" +#include "test_harness/util/logger.h" #include "test_harness/test.h" +#include "base_test.cxx" #include "example_test.cxx" -#include "poc_test.cxx" +#include "hs_cleanup.cxx" std::string parse_configuration_from_file(const std::string &filename) @@ -67,7 +68,8 @@ print_help() std::cout << std::endl; std::cout << "SYNOPSIS" << std::endl; std::cout << "\trun [OPTIONS]" << std::endl; - std::cout << "\trun -C [CONFIGURATION]" << std::endl; + std::cout << "\trun -C [WIREDTIGER_OPEN_CONFIGURATION]" << std::endl; + std::cout << "\trun -c [TEST_FRAMEWORK_CONFIGURATION]" << std::endl; std::cout << "\trun -f [FILE]" << std::endl; std::cout << "\trun -l [TRACEL_LEVEL]" << std::endl; std::cout << "\trun -t [TEST_NAME]" << std::endl; @@ -85,58 +87,66 @@ print_help() std::cout << std::endl; std::cout << "OPTIONS" << std::endl; std::cout << "\t-h Output a usage message and exit." << std::endl; - std::cout << "\t-C Configuration. Cannot be used with -f." << std::endl; + std::cout << "\t-C Additional wiredtiger open configuration." << std::endl; + std::cout << "\t-c Test framework configuration. Cannot be used with -f." << std::endl; std::cout << "\t-f File that contains the configuration. Cannot be used with -C." << std::endl; - std::cout << "\t-l Trace level from 0 (default) to 2." << std::endl; + std::cout << "\t-l Trace level from 0 to 3. " + "1 is the default level, all warnings and errors are logged." + << std::endl; std::cout << "\t-t Test name to be executed." << std::endl; } -void -value_missing_error(const std::string &str) -{ - test_harness::debug_print( - "Value missing for option " + str + ".\nTry './run -h' for more information.", DEBUG_ERROR); -} - /* * Run a specific test. - * test_name: specifies which test to run. - * config: defines the configuration used for the test. + * - test_name: specifies which test to run. + * - config: defines the configuration used for the test. */ int64_t -run_test(const std::string &test_name, const std::string &config) +run_test(const std::string &test_name, const std::string &config, const std::string &wt_open_config) { int error_code = 0; - test_harness::debug_print("Configuration\t:" + config, DEBUG_INFO); + test_harness::logger::log_msg(LOG_TRACE, "Configuration\t:" + config); - if (test_name == "poc_test") - poc_test(config, test_name).run(); + if (test_name == "base_test") + base_test(test_harness::test_args{config, test_name, wt_open_config}).run(); else if (test_name == "example_test") - example_test(config, test_name).run(); + example_test(test_harness::test_args{config, test_name, wt_open_config}).run(); + else if (test_name == "hs_cleanup") + hs_cleanup(test_harness::test_args{config, test_name, wt_open_config}).run(); else { - test_harness::debug_print("Test not found: " + test_name, DEBUG_ERROR); + test_harness::logger::log_msg(LOG_ERROR, "Test not found: " + test_name); error_code = -1; } if (error_code == 0) - test_harness::debug_print("Test " + test_name + " done.", DEBUG_INFO); + test_harness::logger::log_msg(LOG_INFO, "Test " + test_name + " done."); return (error_code); } +static std::string +get_default_config_path(const std::string &test_name) +{ + return ("configs/" + test_name + "_default.txt"); +} + int main(int argc, char *argv[]) { - std::string cfg, config_filename, test_name, current_test_name; + std::string cfg, config_filename, current_cfg, current_test_name, test_name, wt_open_config; int64_t error_code = 0; - const std::vector<std::string> all_tests = {"example_test", "poc_test"}; + const std::vector<std::string> all_tests = {"example_test", "hs_cleanup", "base_test"}; + + /* Set the program name for error messages. */ + (void)testutil_set_progname(argv); /* Parse args - * -C : Configuration. Cannot be used with -f. If no specific test is specified to be run, the - * same coniguration will be used for all existing tests. + * -C : Additional wiredtiger_open configuration. + * -c : Test framework configuration. Cannot be used with -f. If no specific test is specified + * to be run, the same configuration will be used for all existing tests. * -f : Filename that contains the configuration. Cannot be used with -C. If no specific test - * is specified to be run, the same coniguration will be used for all existing tests. + * is specified to be run, the same configuration will be used for all existing tests. * -l : Trace level. * -t : Test to run. All tests are run if not specified. */ @@ -145,59 +155,61 @@ main(int argc, char *argv[]) print_help(); return 0; } else if (std::string(argv[i]) == "-C") { + if ((i + 1) < argc) { + wt_open_config = argv[++i]; + /* Add a comma to the front if the user didn't supply one. */ + if (wt_open_config[0] != ',') + wt_open_config.insert(0, 1, ','); + } else + error_code = -1; + } else if (std::string(argv[i]) == "-c") { if (!config_filename.empty()) { - test_harness::debug_print("Option -C cannot be used with -f", DEBUG_ERROR); + test_harness::logger::log_msg(LOG_ERROR, "Option -C cannot be used with -f"); error_code = -1; } else if ((i + 1) < argc) cfg = argv[++i]; - else { - value_missing_error(argv[i]); + else error_code = -1; - } } else if (std::string(argv[i]) == "-f") { if (!cfg.empty()) { - test_harness::debug_print("Option -f cannot be used with -C", DEBUG_ERROR); + test_harness::logger::log_msg(LOG_ERROR, "Option -f cannot be used with -C"); error_code = -1; } else if ((i + 1) < argc) config_filename = argv[++i]; - else { - value_missing_error(argv[i]); + else error_code = -1; - } } else if (std::string(argv[i]) == "-t") { if ((i + 1) < argc) test_name = argv[++i]; - else { - value_missing_error(argv[i]); + else error_code = -1; - } } else if (std::string(argv[i]) == "-l") { if ((i + 1) < argc) - test_harness::_trace_level = std::stoi(argv[++i]); - else { - value_missing_error(argv[i]); + test_harness::logger::trace_level = std::stoi(argv[++i]); + else error_code = -1; - } - } + } else + error_code = -1; } if (error_code == 0) { - test_harness::debug_print( - "Trace level\t:" + std::to_string(test_harness::_trace_level), DEBUG_INFO); + test_harness::logger::log_msg( + LOG_INFO, "Trace level: " + std::to_string(test_harness::logger::trace_level)); if (test_name.empty()) { /* Run all tests. */ - test_harness::debug_print("Running all tests.", DEBUG_INFO); + test_harness::logger::log_msg(LOG_INFO, "Running all tests."); for (auto const &it : all_tests) { current_test_name = it; /* Configuration parsing. */ if (!config_filename.empty()) - cfg = parse_configuration_from_file(config_filename); - else if (cfg.empty()) { - config_filename = "configs/config_" + current_test_name + "_default.txt"; - cfg = parse_configuration_from_file(config_filename); - } - - error_code = run_test(current_test_name, cfg); + current_cfg = parse_configuration_from_file(config_filename); + else if (cfg.empty()) + current_cfg = + parse_configuration_from_file(get_default_config_path(current_test_name)); + else + current_cfg = cfg; + + error_code = run_test(current_test_name, current_cfg, wt_open_config); if (error_code != 0) break; } @@ -206,16 +218,17 @@ main(int argc, char *argv[]) /* Configuration parsing. */ if (!config_filename.empty()) cfg = parse_configuration_from_file(config_filename); - else if (cfg.empty()) { - config_filename = "configs/config_" + test_name + "_default.txt"; - cfg = parse_configuration_from_file(config_filename); - } - error_code = run_test(current_test_name, cfg); + else if (cfg.empty()) + cfg = parse_configuration_from_file(get_default_config_path(current_test_name)); + error_code = run_test(current_test_name, cfg, wt_open_config); } if (error_code != 0) - test_harness::debug_print("Test " + current_test_name + " failed.", DEBUG_ERROR); - } + test_harness::logger::log_msg(LOG_ERROR, "Test " + current_test_name + " failed."); + } else + test_harness::logger::log_msg(LOG_ERROR, + "Invalid command line arguments supplied. Try " + "'./run -h' for help."); return (error_code); } diff --git a/src/third_party/wiredtiger/test/csuite/incr_backup/main.c b/src/third_party/wiredtiger/test/csuite/incr_backup/main.c index b09e1b44da4..f8de2578e3d 100644 --- a/src/third_party/wiredtiger/test/csuite/incr_backup/main.c +++ b/src/third_party/wiredtiger/test/csuite/incr_backup/main.c @@ -110,7 +110,9 @@ extern int __wt_optind; extern char *__wt_optarg; /* - * The choices of operations we do to each table. + * The choices of operations we do to each table. Please do not initialize enum elements with custom + * values as there's an assumption that the first element has the default value of 0 and the last + * element is always reserved to check count on elements. */ typedef enum { INSERT, MODIFY, REMOVE, UPDATE, _OPERATION_TYPE_COUNT } OPERATION_TYPE; @@ -154,6 +156,20 @@ die(void) } /* + * Get operation type based on the number of changes + */ +static OPERATION_TYPE +get_operation_type(uint64_t change_count) +{ + int32_t op_type; + + op_type = ((change_count % CHANGES_PER_CYCLE) / KEYS_PER_TABLE); + testutil_assert(op_type <= _OPERATION_TYPE_COUNT); + + return (OPERATION_TYPE)op_type; +} + +/* * key_value -- * Return the key, value and operation type for a given change to a table. See "Cycle of changes * to a table" above. @@ -173,7 +189,7 @@ key_value(uint64_t change_count, char *key, size_t key_size, WT_ITEM *item, OPER char ch; key_num = change_count % KEYS_PER_TABLE; - *typep = op_type = (OPERATION_TYPE)((change_count % CHANGES_PER_CYCLE) / KEYS_PER_TABLE); + *typep = op_type = get_operation_type(change_count); testutil_check( __wt_snprintf(key, key_size, KEY_FORMAT, (int)(key_num % 100), (int)(key_num / 100))); @@ -370,7 +386,11 @@ table_changes(WT_SESSION *session, TABLE *table) item.size = table->max_value_size; key_value(change_count, key, sizeof(key), &item, &op_type); cur->set_key(cur, key); - testutil_assert(op_type < _OPERATION_TYPE_COUNT); + + /* + * To satisfy code analysis checks, we must handle all elements of the enum in the + * switch statement. + */ switch (op_type) { case INSERT: cur->set_value(cur, &item); @@ -393,7 +413,7 @@ table_changes(WT_SESSION *session, TABLE *table) testutil_check(cur->update(cur)); break; case _OPERATION_TYPE_COUNT: - break; + testutil_die(0, "Unexpected OPERATION_TYPE: _OPERATION_TYPE_COUNT"); } } free(value); @@ -682,11 +702,15 @@ check_table(WT_SESSION *session, TABLE *table) expect_records = 0; total_changes = table->change_count; boundary = total_changes % KEYS_PER_TABLE; - op_type = (OPERATION_TYPE)((total_changes % CHANGES_PER_CYCLE) / KEYS_PER_TABLE); + op_type = get_operation_type(total_changes); value = dcalloc(1, table->max_value_size); VERBOSE(3, "Checking: %s\n", table->name); - testutil_assert(op_type < _OPERATION_TYPE_COUNT); + + /* + * To satisfy code analysis checks, we must handle all elements of the enum in the switch + * statement. + */ switch (op_type) { case INSERT: expect_records = total_changes % KEYS_PER_TABLE; @@ -699,7 +723,7 @@ check_table(WT_SESSION *session, TABLE *table) expect_records = KEYS_PER_TABLE - (total_changes % KEYS_PER_TABLE); break; case _OPERATION_TYPE_COUNT: - break; + testutil_die(0, "Unexpected OPERATION_TYPE: _OPERATION_TYPE_COUNT"); } testutil_check(session->open_cursor(session, table->name, NULL, NULL, &cursor)); diff --git a/src/third_party/wiredtiger/test/csuite/random_directio/main.c b/src/third_party/wiredtiger/test/csuite/random_directio/main.c index eb84a9129b5..5f60b18bab4 100644 --- a/src/third_party/wiredtiger/test/csuite/random_directio/main.c +++ b/src/third_party/wiredtiger/test/csuite/random_directio/main.c @@ -883,8 +883,8 @@ check_db(uint32_t nth, uint32_t datasize, pid_t pid, bool directio, uint32_t fla gotid = (uint64_t)strtol(gotkey, &p, 10); testutil_assert(*p == KEY_SEP[0]); p++; - testutil_assert(isxdigit(*p)); - if (isdigit(*p)) + testutil_assert(isxdigit((unsigned char)*p)); + if (isdigit((unsigned char)*p)) gotth = (uint32_t)(*p - '0'); else if (*p >= 'a' && *p <= 'f') gotth = (uint32_t)((*p - 'a') + 10); diff --git a/src/third_party/wiredtiger/test/csuite/random_directio/util.c b/src/third_party/wiredtiger/test/csuite/random_directio/util.c index 43f0f0ad821..4b91501a2d8 100644 --- a/src/third_party/wiredtiger/test/csuite/random_directio/util.c +++ b/src/third_party/wiredtiger/test/csuite/random_directio/util.c @@ -37,7 +37,7 @@ * util.c * Utility functions for test that simulates system crashes. */ -#define COPY_BUF_SIZE ((size_t)(20 * 1024)) +#define COPY_BUF_SIZE ((size_t)(64 * 1024)) /* * copy_directory -- @@ -115,7 +115,7 @@ copy_directory(const char *fromdir, const char *todir, bool directio) */ if (buf == NULL) { if (directio) { - blksize = (size_t)sb.st_blksize; + blksize = (size_t)WT_MAX(sb.st_blksize, WT_BUFFER_ALIGNMENT_DEFAULT); testutil_assert(blksize < COPY_BUF_SIZE); /* * Make sure we have plenty of room for adjusting the pointer. diff --git a/src/third_party/wiredtiger/test/csuite/wt2999_join_extractor/main.c b/src/third_party/wiredtiger/test/csuite/wt2999_join_extractor/main.c index c0aa6a717c1..22b05deca31 100644 --- a/src/third_party/wiredtiger/test/csuite/wt2999_join_extractor/main.c +++ b/src/third_party/wiredtiger/test/csuite/wt2999_join_extractor/main.c @@ -42,13 +42,13 @@ custom_extract1(WT_EXTRACTOR *extractor, WT_SESSION *session, const WT_ITEM *key const WT_ITEM *value, WT_CURSOR *result_cursor) { WT_ITEM item; - int32_t v1; + int64_t v1; (void)extractor; (void)key; testutil_check(wiredtiger_struct_unpack(session, value->data, value->size, "u", &item)); - v1 = ((int *)item.data)[0]; + v1 = ((int64_t *)item.data)[0]; item.data = &v1; item.size = sizeof(v1); @@ -61,13 +61,13 @@ custom_extract2(WT_EXTRACTOR *extractor, WT_SESSION *session, const WT_ITEM *key const WT_ITEM *value, WT_CURSOR *result_cursor) { WT_ITEM item; - int32_t v2; + int64_t v2; (void)extractor; (void)key; testutil_check(wiredtiger_struct_unpack(session, value->data, value->size, "u", &item)); - v2 = ((int *)item.data)[1]; + v2 = ((int64_t *)item.data)[1]; item.data = &v2; item.size = sizeof(v2); @@ -86,7 +86,7 @@ main(int argc, char *argv[]) WT_CURSOR *cursor1, *cursor2, *jcursor; WT_ITEM k, v; WT_SESSION *session; - int32_t key, val[2]; + int64_t key, val[2]; int i, ret; opts = &_opts; diff --git a/src/third_party/wiredtiger/test/csuite/wt3184_dup_index_collator/main.c b/src/third_party/wiredtiger/test/csuite/wt3184_dup_index_collator/main.c index cf4ccb6f05f..ac05151f257 100644 --- a/src/third_party/wiredtiger/test/csuite/wt3184_dup_index_collator/main.c +++ b/src/third_party/wiredtiger/test/csuite/wt3184_dup_index_collator/main.c @@ -44,10 +44,18 @@ compare_int(int32_t a, int32_t b) } static int32_t -item_to_int(WT_ITEM *item) +item_to_int(const WT_ITEM *item) { + int32_t ret; + testutil_assert(item->size == sizeof(int32_t)); - return (*(int32_t *)item->data); + + /* + * Using memcpy instead of direct type cast to avoid undefined behavior sanitizer complaining + * about misaligned address. + */ + memcpy(&ret, item->data, sizeof(int32_t)); + return ret; } static int @@ -61,10 +69,9 @@ compare_int_items(WT_ITEM *itema, WT_ITEM *itemb) static void print_int_item(const char *str, const WT_ITEM *item) { - if (item->size > 0) { - testutil_assert(item->size == sizeof(int32_t)); - printf("%s%" PRId32, str, *(int32_t *)item->data); - } else + if (item->size > 0) + printf("%s%" PRId32, str, item_to_int(item)); + else printf("%s<empty>", str); } diff --git a/src/third_party/wiredtiger/test/csuite/wt3338_partial_update/main.c b/src/third_party/wiredtiger/test/csuite/wt3338_partial_update/main.c index a516cc23703..8b6b9c5f841 100644 --- a/src/third_party/wiredtiger/test/csuite/wt3338_partial_update/main.c +++ b/src/third_party/wiredtiger/test/csuite/wt3338_partial_update/main.c @@ -126,8 +126,11 @@ slow_apply_api(WT_ITEM *orig) tb = &_tb; /* Mess up anything not initialized in the buffers. */ - memset((uint8_t *)ta->mem + ta->size, 0xff, ta->memsize - ta->size); - memset((uint8_t *)tb->mem, 0xff, tb->memsize); + if ((ta->memsize - ta->size) > 0) + memset((uint8_t *)ta->mem + ta->size, 0xff, ta->memsize - ta->size); + + if (tb->memsize > 0) + memset((uint8_t *)tb->mem, 0xff, tb->memsize); /* * Process the entries to figure out how large a buffer we need. This is a bit pessimistic diff --git a/src/third_party/wiredtiger/test/csuite/wt3363_checkpoint_op_races/main.c b/src/third_party/wiredtiger/test/csuite/wt3363_checkpoint_op_races/main.c index c25b36351b0..069f2c5a5fe 100644 --- a/src/third_party/wiredtiger/test/csuite/wt3363_checkpoint_op_races/main.c +++ b/src/third_party/wiredtiger/test/csuite/wt3363_checkpoint_op_races/main.c @@ -28,7 +28,7 @@ /* * [TEST_TAGS] - * checkpoints:liveness:liveness + * checkpoint * [END_TAGS] */ diff --git a/src/third_party/wiredtiger/test/ctest_dir_sync.cmake b/src/third_party/wiredtiger/test/ctest_dir_sync.cmake new file mode 100644 index 00000000000..c15f6e336ec --- /dev/null +++ b/src/third_party/wiredtiger/test/ctest_dir_sync.cmake @@ -0,0 +1,48 @@ +# +# Public Domain 2014-present MongoDB, Inc. +# Public Domain 2008-2014 WiredTiger, Inc. +# +# This is free and unencumbered software released into the public domain. +# +# Anyone is free to copy, modify, publish, use, compile, sell, or +# distribute this software, either in source code form or as a compiled +# binary, for any purpose, commercial or non-commercial, and by any +# means. +# +# In jurisdictions that recognize copyright laws, the author or authors +# of this software dedicate any and all copyright interest in the +# software to the public domain. We make this dedication for the benefit +# of the public at large and to the detriment of our heirs and +# successors. We intend this dedication to be an overt act of +# relinquishment in perpetuity of all present and future rights to this +# software under copyright law. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +# IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. +# + +cmake_minimum_required(VERSION 3.10.0) + +if(NOT SYNC_DIR_SRC) + message(FATAL_ERROR "Missing a source directory to sync") +endif() + +if(NOT SYNC_DIR_DST) + message(FATAL_ERROR "Missing a destination directory to sync") +endif() + +# Get the list of files in the sync directory +file(GLOB files ${SYNC_DIR_SRC}/*) +# Check each file and copy over if it has changed +foreach (sync_file IN LISTS files) + get_filename_component(sync_file_basename ${sync_file} NAME) + execute_process(COMMAND ${CMAKE_COMMAND} -E copy_if_different + ${sync_file} + ${SYNC_DIR_DST}/${sync_file_basename} + ) +endforeach() diff --git a/src/third_party/wiredtiger/test/ctest_helpers.cmake b/src/third_party/wiredtiger/test/ctest_helpers.cmake index d11f494c8ce..622aea84160 100644 --- a/src/third_party/wiredtiger/test/ctest_helpers.cmake +++ b/src/third_party/wiredtiger/test/ctest_helpers.cmake @@ -104,6 +104,10 @@ function(create_test_executable target) target_link_libraries(${target} ${CREATE_TEST_LIBS}) endif() + if(ENABLE_TCMALLOC AND HAVE_LIBTCMALLOC) + target_link_libraries(${target} ${HAVE_LIBTCMALLOC}) + endif() + # If compiling for windows, additionally link in the shim library. if(WT_WIN) target_include_directories( @@ -132,14 +136,14 @@ function(create_test_executable target) # Useful if we need to setup an additional configs and environments needed to run the test executable. foreach(dir IN LISTS CREATE_TEST_ADDITIONAL_DIRECTORIES) get_filename_component(dir_basename ${dir} NAME) - # Copy the file to the given test/targets build directory. - add_custom_command(OUTPUT ${test_binary_dir}/${dir_basename} - COMMAND ${CMAKE_COMMAND} -E copy_directory - ${dir} - ${test_binary_dir}/${dir_basename} + # Copy the directory to the given test/targets build directory. + add_custom_target(sync_dir_${target}_${dir_basename} ALL + COMMAND ${CMAKE_COMMAND} + -DSYNC_DIR_SRC=${dir} + -DSYNC_DIR_DST=${test_binary_dir}/${dir_basename} + -P ${CMAKE_SOURCE_DIR}/test/ctest_dir_sync.cmake ) - add_custom_target(copy_dir_${target}_${dir_basename} DEPENDS ${test_binary_dir}/${dir_basename}) - add_dependencies(${target} copy_dir_${target}_${dir_basename}) + add_dependencies(${target} sync_dir_${target}_${dir_basename}) endforeach() endfunction() @@ -177,16 +181,18 @@ function(define_test_variants target) separate_arguments(variant_args UNIX_COMMAND ${curr_variant_args}) endif() # Create a variant directory to run the test in. - add_custom_target(${curr_variant_name}_test_dir - COMMAND ${CMAKE_COMMAND} -E make_directory ${CMAKE_CURRENT_BINARY_DIR}/${curr_variant_name}) + add_custom_command(OUTPUT ${curr_variant_name}_test_dir + COMMAND ${CMAKE_COMMAND} -E make_directory ${CMAKE_CURRENT_BINARY_DIR}/${curr_variant_name}_test_dir + ) + add_custom_target(create_dir_${curr_variant_name} DEPENDS ${curr_variant_name}_test_dir) # Ensure the variant target is created prior to building the test. - add_dependencies(${target} ${curr_variant_name}_test_dir) + add_dependencies(${target} create_dir_${curr_variant_name}) add_test( NAME ${curr_variant_name} COMMAND $<TARGET_FILE:${target}> ${variant_args} # Run each variant in its own subdirectory, allowing us to execute variants in # parallel. - WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}/${curr_variant_name} + WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}/${curr_variant_name}_test_dir ) list(APPEND defined_tests ${curr_variant_name}) endforeach() diff --git a/src/third_party/wiredtiger/test/evergreen.yml b/src/third_party/wiredtiger/test/evergreen.yml index 441b0dd3cc1..c1eefaa2349 100755 --- a/src/third_party/wiredtiger/test/evergreen.yml +++ b/src/third_party/wiredtiger/test/evergreen.yml @@ -44,16 +44,64 @@ functions: command: shell.exec params: script: | + set -o errexit + set -o verbose git clone https://github.com/wiredtiger/mongo-tests + "fetch mongo repo" : + command: shell.exec + params: + script: | + set -o errexit + set -o verbose + git clone https://github.com/mongodb/mongo + "import wiredtiger into mongo" : + command: shell.exec + params: + script: | + set -o errexit + set -o verbose + cp -a wiredtiger mongo/src/third_party/ + "compile mongodb" : + command: shell.exec + params: + shell: bash + working_dir: "mongo" + script: | + set -o errexit + set -o verbose + virtualenv -p python3 venv + source venv/bin/activate + pip3 install requirements_parser + pip3 install -r etc/pip/compile-requirements.txt + ./buildscripts/scons.py --variables-files=etc/scons/mongodbtoolchain_stable_gcc.vars --link-model=dynamic --ninja generate-ninja ICECC=icecc CCACHE=ccache + ninja -j$(nproc --all) install-mongod "configure wiredtiger": &configure_wiredtiger command: shell.exec params: - working_dir: "wiredtiger/build_posix" + working_dir: "wiredtiger" shell: bash script: | set -o errexit set -o verbose - if [ "$OS" != "Windows_NT" ]; then + # Check if the build variant has specified a build type, always default to + # Autoconf/Libtool if $is_cmake_build is not declared. + if [ ${is_cmake_build|false} = true ]; then + if [ "$OS" = "Windows_NT" ]; then + # Use the Windows powershell script to configure the CMake build. + # We execute it in a powershell environment as its easier to detect and source the Visual Studio + # toolchain in a native Windows environment. We can't easily execute the build in a cygwin environment. + powershell.exe -NonInteractive '.\test\evergreen\build_windows.ps1' -configure 1 ${windows_configure_flags|} + else + # Compiling with CMake. + . test/evergreen/find_cmake.sh + mkdir -p cmake_build + cd cmake_build + $CMAKE \ + ${posix_configure_flags|-DCMAKE_TOOLCHAIN_FILE=../build_cmake/toolchains/mongodbtoolchain_v3_gcc.cmake -DCMAKE_C_FLAGS="-ggdb" -DHAVE_DIAGNOSTIC=1 -DENABLE_PYTHON=1 -DENABLE_ZLIB=1 -DENABLE_STATIC=1 -DENABLE_STRICT=1 -DCMAKE_INSTALL_PREFIX=$(pwd)/LOCAL_INSTALL} -G "${cmake_generator|Ninja}" ./.. + fi + elif [ "$OS" != "Windows_NT" ]; then + # Compiling with Autoconf/Libtool. + cd build_posix sh reconf ${configure_env_vars|CC=/opt/mongodbtoolchain/v3/bin/gcc CXX=/opt/mongodbtoolchain/v3/bin/g++ PATH=/opt/mongodbtoolchain/v3/bin:$PATH ADD_CFLAGS="-ggdb -fPIC"} PYTHON="python3" \ ../configure ${configure_python_setting|} \ @@ -67,13 +115,23 @@ functions: script: | set -o errexit set -o verbose - if [ "Windows_NT" == "$OS" ]; then + if [ ${is_cmake_build|false} = true ]; then + if [ "$OS" = "Windows_NT" ]; then + # Use the Windows powershell script to execute Ninja build (can't execute directly in a cygwin environment). + powershell.exe '.\test\evergreen\build_windows.ps1 -build 1' + else + # Compiling with CMake generated Ninja file. + cd cmake_build + ${make_command|ninja} ${smp_command|} 2>&1 + fi + elif [ "Windows_NT" == "$OS" ]; then export "PATH=/cygdrive/c/Python39:/cygdrive/c/Python39/Scripts:$PATH" python --version python -m pip install scons==3.1.1 scons-3.1.1.bat "LIBPATH=c:\\python\\Python39\\libs" --enable-python=c:\\swigwin-3.0.2\\swig.exe --enable-diagnostic ${scons_smp_command|} else + # Compiling with Autoconf/Libtool Makefiles. cd build_posix ${make_command|make} ${smp_command|} 2>&1 @@ -226,16 +284,22 @@ functions: "make check all": command: shell.exec params: - working_dir: "wiredtiger/build_posix" + working_dir: "wiredtiger" script: | set -o errexit set -o verbose - - ${test_env_vars|} ${make_command|make} VERBOSE=1 check ${smp_command|} 2>&1 + if [ ${is_cmake_build|false} = true ]; then + . test/evergreen/find_cmake.sh + cd cmake_build + ${test_env_vars|} $CTEST -L check ${smp_command|} -VV 2>&1 + else + cd build_posix + ${test_env_vars|} ${make_command|make} VERBOSE=1 check ${smp_command|} 2>&1 + fi "unit test": command: shell.exec params: - working_dir: "wiredtiger/build_posix" + working_dir: "wiredtiger" script: | set -o errexit set -o verbose @@ -244,6 +308,11 @@ functions: export "PATH=/cygdrive/c/Python39:/cygdrive/c/Python39/Scripts:$PATH" export "PYTHONPATH=$(pwd)/../lang/python/wiredtiger):$(cygpath -w $(pwd)/../lang/python)" fi + if [ ${is_cmake_build|false} = true ]; then + cd cmake_build + else + cd build_posix + fi ${test_env_vars|} ${python_binary|python3} ../test/suite/run.py ${unit_test_args|-v 2} ${smp_command|} 2>&1 "format test": command: shell.exec @@ -555,27 +624,6 @@ tasks: - func: "upload artifact" - func: "cleanup" - - name: compile-msan - commands: - - func: "get project" - - func: "compile wiredtiger" - vars: - # We don't compile C++ for memory sanitized testing as it creates false positives - configure_env_vars: CC=/opt/mongodbtoolchain/v3/bin/clang CXX=skip PATH=/opt/mongodbtoolchain/v3/bin:$PATH CFLAGS="-fsanitize=memory -ggdb" - posix_configure_flags: --enable-silent-rules --enable-strict --enable-diagnostic --disable-static - - func: "upload artifact" - - func: "cleanup" - - - name: compile-ubsan - commands: - - func: "get project" - - func: "compile wiredtiger" - vars: - configure_env_vars: CC=/opt/mongodbtoolchain/v3/bin/gcc CXX=/opt/mongodbtoolchain/v3/bin/g++ PATH=/opt/mongodbtoolchain/v3/bin:$PATH CFLAGS="-fsanitize=undefined -ggdb" CXXFLAGS="-fsanitize=undefined -ggdb" - posix_configure_flags: --enable-silent-rules --enable-strict --enable-diagnostic - - func: "upload artifact" - - func: "cleanup" - # production build with --disable-shared - name: compile-production-disable-shared tags: ["pull_request"] @@ -669,27 +717,10 @@ tasks: depends_on: - name: compile commands: - - func: "fetch artifacts" + - func: "get project" - func: "compile wiredtiger" - func: "make check all" - - name: make-check-msan-test - depends_on: - - name: compile-msan - commands: - - func: "fetch artifacts" - vars: - dependent_task: compile-msan - - func: "compile wiredtiger" - vars: - # We don't compile C++ for memory sanitized testing as it creates false positives - configure_env_vars: CC=/opt/mongodbtoolchain/v3/bin/clang CXX=skip PATH=/opt/mongodbtoolchain/v3/bin:$PATH CFLAGS="-fsanitize=memory -ggdb" - posix_configure_flags: --enable-silent-rules --enable-strict --enable-diagnostic --disable-static - - func: "make check all" - vars: - smp_command: -j 1 - test_env_vars: MSAN_OPTIONS=abort_on_error=1:disable_coredump=0 MSAN_SYMBOLIZER_PATH=/opt/mongodbtoolchain/v3/bin/llvm-symbolizer TESTUTIL_SLOW_MACHINE=1 - - name: make-check-linux-no-ftruncate-test depends_on: - name: compile-linux-no-ftruncate @@ -890,28 +921,26 @@ tasks: # End of normal make check test tasks - - name: ubsan-test + # Start of cppsuite test tasks. + # All cppsuite pull request tasks must supply the relative path to the config file as we are in + # the build_posix working directory and the LD_LIBRARY_PATH is .libs. + + - name: cppsuite-base-test-default + tags: ["pull_request"] depends_on: - - name: compile-ubsan + - name: compile commands: - func: "fetch artifacts" - vars: - dependent_task: compile-ubsan - - func: "compile wiredtiger" - vars: - configure_env_vars: CC=/opt/mongodbtoolchain/v3/bin/gcc CXX=/opt/mongodbtoolchain/v3/bin/g++ PATH=/opt/mongodbtoolchain/v3/bin:$PATH CFLAGS="-fsanitize=undefined -ggdb" CXXFLAGS="-fsanitize=undefined -ggdb" - posix_configure_flags: --enable-silent-rules --enable-strict --enable-diagnostic - command: shell.exec params: - working_dir: "wiredtiger/build_posix/examples/c" + working_dir: "wiredtiger/build_posix/" script: | set -o errexit set -o verbose - UBSAN_OPTIONS=print_stacktrace=1:halt_on_error=1:abort_on_error=1:disable_coredump=0 ./ex_access - # Start of cppsuite test tasks. + ${test_env_vars|} $(pwd)/test/cppsuite/run -t base_test -C 'debug_mode=(cursor_copy=true)' -f test/cppsuite/configs/base_test_default.txt -l 2 - - name: poc-test-cpp + - name: cppsuite-hs-cleanup-default tags: ["pull_request"] depends_on: - name: compile @@ -924,7 +953,35 @@ tasks: set -o errexit set -o verbose - ${test_env_vars|} $(pwd)/test/cppsuite/run -t poc_test -f test/cppsuite/configs/config_poc_test_default.txt -l 1 + ${test_env_vars|} $(pwd)/test/cppsuite/run -t hs_cleanup -C 'debug_mode=(cursor_copy=true)' -f test/cppsuite/configs/hs_cleanup_default.txt -l 2 + + - name: cppsuite-base-test-stress + depends_on: + - name: compile + commands: + - func: "fetch artifacts" + - command: shell.exec + params: + working_dir: "wiredtiger/build_posix/test/cppsuite" + script: | + set -o errexit + set -o verbose + + ${test_env_vars|} $(pwd)/run -t base_test -f configs/base_test_stress.txt -l 2 + + - name: cppsuite-hs-cleanup-stress + depends_on: + - name: compile + commands: + - func: "fetch artifacts" + - command: shell.exec + params: + working_dir: "wiredtiger/build_posix/test/cppsuite" + script: | + set -o errexit + set -o verbose + + ${test_env_vars|} $(pwd)/run -t hs_cleanup -f configs/hs_cleanup_stress.txt -l 2 # End of cppsuite test tasks. # Start of csuite test tasks @@ -1736,21 +1793,6 @@ tasks: ${test_env_vars|} test/fops/t fi - - name: million-collection-test - commands: - - func: "get project" - - func: "fetch mongo-tests repo" - - command: shell.exec - params: - working_dir: mongo-tests - script: | - sudo su - set -o errexit - set -o verbose - ulimit -n 1000000 - ulimit -c unlimited - largescale/run-million-collection-test.sh . - - name: compatibility-test-for-newer-releases commands: - func: "get project" @@ -2506,6 +2548,49 @@ tasks: name: recovery-stress-test-3 tags: ["stress-test-3", "stress-test-zseries-3"] + - name: many-dhandle-stress-test + commands: + - func: "get project" + - func: "compile wiredtiger" + vars: + configure_env_vars: + CXX=/opt/mongodbtoolchain/v3/bin/g++ + PATH=/opt/mongodbtoolchain/v3/bin:$PATH ADD_CFLAGS="-ggdb -fPIC" + - command: shell.exec + params: + working_dir: "wiredtiger/bench/workgen/runner" + script: | + set -o errexit + set -o verbose + export "PATH=/opt/mongodbtoolchain/v3/bin:$PATH" + ./workgen_perf_check.sh many-dhandle-stress.py output.log 50 500 500 + + - name: many-collection-test + commands: + - command: timeout.update + params: + exec_timeout_secs: 86400 + timeout_secs: 86400 + - func: "fetch mongo repo" + - func: "get project" + - func: "import wiredtiger into mongo" + - func: "compile mongodb" + - func: "fetch mongo-tests repo" + - command: shell.exec + params: + working_dir: mongo-tests/largescale + shell: bash + script: | + set -o errexit + set -o verbose + sudo su + ulimit -n 1000000 + export "PATH=/opt/mongodbtoolchain/v3/bin:$PATH" + virtualenv -p python3 venv + source venv/bin/activate + pip3 install lorem pymongo + ./run_many_coll.sh ../../mongo/build/install/bin/mongod mongodb.log config/many-collection-testing many-collection clean-and-populate + - name: cyclomatic-complexity commands: - func: "get project" @@ -2541,10 +2626,6 @@ buildvariants: make_command: PATH=/opt/mongodbtoolchain/v3/bin:$PATH make tasks: - name: ".pull_request !.windows_only !.pull_request_compilers" - - name: compile-msan - - name: make-check-msan-test - - name: compile-ubsan - - name: ubsan-test - name: linux-directio distros: ubuntu1804-build - name: syscall-linux @@ -2563,6 +2644,23 @@ buildvariants: - name: static-wt-build-test - name: format-failure-configs-test +- name: ubuntu1804-cmake + display_name: "* Ubuntu 18.04 CMake" + run_on: + - ubuntu1804-test + expansions: + test_env_vars: LD_LIBRARY_PATH=$(pwd) WT_BUILDDIR=$(pwd) + posix_configure_flags: -DCMAKE_TOOLCHAIN_FILE=../build_cmake/toolchains/mongodbtoolchain_v3_gcc.cmake -DCMAKE_C_FLAGS="-ggdb" -DHAVE_DIAGNOSTIC=1 -DENABLE_PYTHON=1 -DENABLE_ZLIB=1 -DENABLE_SNAPPY=1 -DENABLE_STRICT=1 -DCMAKE_INSTALL_PREFIX=$(pwd)/LOCAL_INSTALL + python_binary: '/opt/mongodbtoolchain/v3/bin/python3' + smp_command: -j $(echo "`grep -c ^processor /proc/cpuinfo` * 2" | bc) + cmake_generator: Ninja + make_command: ninja + is_cmake_build: true + tasks: + - name: compile + - name: make-check-test + - name: unit-test + - name: ubuntu1804-asan display_name: "! Ubuntu 18.04 ASAN" run_on: @@ -2584,7 +2682,70 @@ buildvariants: LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libeatmydata.so PATH=/opt/mongodbtoolchain/v3/bin:$PATH LD_LIBRARY_PATH=$(pwd)/.libs top_srcdir=$(pwd)/.. top_builddir=$(pwd) tasks: - name: ".pull_request !.windows_only !.pull_request_compilers !.python" - - examples-c-test + - name: examples-c-test + +- name: ubuntu1804-msan + display_name: "! Ubuntu 18.04 MSAN" + run_on: + - ubuntu1804-test + expansions: + configure_env_vars: + # We don't compile C++ for memory sanitized testing as it creates false positives. + CC=/opt/mongodbtoolchain/v3/bin/clang + PATH=/opt/mongodbtoolchain/v3/bin:$PATH + CFLAGS="-fsanitize=memory -fno-omit-frame-pointer -fno-optimize-sibling-calls -O1 -ggdb" + posix_configure_flags: + --enable-silent-rules --enable-strict --enable-diagnostic + --disable-static --prefix=$(pwd)/LOCAL_INSTALL + smp_command: -j $(grep -c ^processor /proc/cpuinfo) + make_command: PATH=/opt/mongodbtoolchain/v3/bin:$PATH make + test_env_vars: + MSAN_OPTIONS="abort_on_error=1:disable_coredump=0:print_stacktrace=1" + MSAN_SYMBOLIZER_PATH=/opt/mongodbtoolchain/v3/bin/llvm-symbolizer + LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libeatmydata.so LD_LIBRARY_PATH=$(pwd)/.libs + PATH=/opt/mongodbtoolchain/v3/bin:$PATH top_srcdir=$(pwd)/.. top_builddir=$(pwd) + TESTUTIL_SLOW_MACHINE=1 + tasks: + - name: clang-analyzer + - name: compile + - name: compile-production-disable-shared + - name: compile-production-disable-static + - name: examples-c-production-disable-shared-test + - name: examples-c-production-disable-static-test + - name: format-stress-pull-request-test + - name: make-check-test + +- name: ubuntu1804-ubsan + display_name: "! Ubuntu 18.04 UBSAN" + run_on: + - ubuntu1804-test + expansions: + configure_env_vars: + CC=/opt/mongodbtoolchain/v3/bin/clang + CXX=/opt/mongodbtoolchain/v3/bin/clang++ + PATH=/opt/mongodbtoolchain/v3/bin:$PATH + CFLAGS="-fsanitize=undefined -fno-omit-frame-pointer -fno-optimize-sibling-calls -O1 -ggdb" + CXXFLAGS="-fsanitize=undefined -fno-omit-frame-pointer -fno-optimize-sibling-calls -O1 -ggdb" + posix_configure_flags: + --enable-silent-rules --enable-strict --enable-diagnostic + --disable-static --prefix=$(pwd)/LOCAL_INSTALL + smp_command: -j $(grep -c ^processor /proc/cpuinfo) + make_command: PATH=/opt/mongodbtoolchain/v3/bin:$PATH make + test_env_vars: + UBSAN_OPTIONS="detect_leaks=1:disable_coredump=0:external_symbolizer_path=/opt/mongodbtoolchain/v3/bin/llvm-symbolizer:halt_on_error=1:print_stacktrace=1" + LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libeatmydata.so LD_LIBRARY_PATH=$(pwd)/.libs + PATH=/opt/mongodbtoolchain/v3/bin:$PATH top_srcdir=$(pwd)/.. top_builddir=$(pwd) + tasks: + - name: clang-analyzer + - name: compile + - name: compile-production-disable-shared + - name: compile-production-disable-static + - name: examples-c-production-disable-shared-test + - name: examples-c-production-disable-static-test + - name: format-stress-pull-request-test + - name: make-check-test + - name: cppsuite-base-test-default + - name: cppsuite-hs-cleanup-default - name: ubuntu1804-compilers display_name: "! Ubuntu 18.04 Compilers" @@ -2618,6 +2779,41 @@ buildvariants: - name: ".stress-test-3" - name: ".stress-test-4" +- name: large-scale-tests + display_name: "Large scale tests" + batchtime: 480 # 3 times a day + run_on: + - ubuntu1804-test + expansions: + make_command: PATH=/opt/mongodbtoolchain/v3/bin:$PATH make + posix_configure_flags: + --enable-silent-rules --enable-python --enable-zlib --enable-snappy + --enable-strict --enable-static --prefix=$(pwd)/LOCAL_INSTALL + test_env_vars: + PATH=/opt/mongodbtoolchain/v3/bin:$PATH + tasks: + - name: many-dhandle-stress-test + - name: many-collection-test + distros: ubuntu1804-wt-large + +- name: cppsuite-stress-tests + display_name: "Cppsuite Stress Tests" + batchtime: 480 # 3 times a day + run_on: + - ubuntu1804-test + expansions: + test_env_vars: LD_LIBRARY_PATH=$(pwd)/../../.libs + make_command: PATH=/opt/mongodbtoolchain/v3/bin:$PATH make + posix_configure_flags: + --enable-silent-rules --enable-python --enable-zlib --enable-snappy + --enable-strict --enable-static + test_env_vars: + PATH=/opt/mongodbtoolchain/v3/bin:$PATH + tasks: + - name: compile + - name: cppsuite-hs-cleanup-stress + - name: cppsuite-base-test-stress + - name: package display_name: "~ Package" batchtime: 1440 # 1 day @@ -2667,10 +2863,6 @@ buildvariants: - name: unit-test - name: fops - name: time-shift-sensitivity-test - - name: compile-msan - - name: make-check-msan-test - - name: compile-ubsan - - name: ubsan-test - name: linux-directio distros: rhel80-build - name: syscall-linux @@ -2686,14 +2878,6 @@ buildvariants: - name: long-test - name: configure-combinations -- name: large-scale-tests - display_name: "~ Large scale tests" - batchtime: 1440 # 1 day - run_on: - - rhel80-build - tasks: - - name: million-collection-test - - name: code-statistics display_name: "Code statistics" batchtime: 1440 # 1 day @@ -2735,6 +2919,19 @@ buildvariants: - name: ".unit_test" - name: fops +- name: windows-64-cmake + display_name: "* Windows 64-bit CMake" + run_on: + - windows-64-vs2017-test + expansions: + python_binary: 'python' + is_cmake_build: true + test_env_vars: WT_BUILDDIR=$(pwd) + windows_configure_flags: -vcvars_bat "'C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\VC\Auxiliary\Build\vcvars64.bat'" + tasks: + - name: compile + - name: unit-test + - name: macos-1014 display_name: OS X 10.14 run_on: @@ -2748,12 +2945,27 @@ buildvariants: test_env_vars: PATH=/opt/mongodbtoolchain/v3/bin:$PATH DYLD_LIBRARY_PATH=$(pwd)/.libs top_srcdir=$(pwd)/.. top_builddir=$(pwd) tasks: - name: compile - # macos-1012 is a more stable distro that we want to use for our PR testing task - distros: macos-1012 - name: make-check-test - name: unit-test-with-compile - name: fops +- name: macos-1014-cmake + display_name: "* OS X 10.14 CMake" + run_on: + - macos-1014 + expansions: + posix_configure_flags: -DCMAKE_C_FLAGS="-ggdb" -DHAVE_DIAGNOSTIC=1 -DENABLE_PYTHON=1 -DENABLE_ZLIB=1 -DENABLE_STRICT=1 -DCMAKE_INSTALL_PREFIX=$(pwd)/LOCAL_INSTALL + python_binary: 'python3' + smp_command: -j $(sysctl -n hw.logicalcpu) + cmake_generator: "Unix Makefiles" + make_command: make + test_env_vars: DYLD_LIBRARY_PATH=$(pwd) WT_BUILDDIR=$(pwd) + is_cmake_build: true + tasks: + - name: compile + - name: make-check-test + - name: unit-test + - name: little-endian display_name: "~ Little-endian (x86)" run_on: @@ -2808,6 +3020,24 @@ buildvariants: - name: ".stress-test-ppc-1" - name: ".stress-test-ppc-2" +- name: ubuntu1804-ppc-cmake + display_name: "* Ubuntu 18.04 PPC CMake" + run_on: + - ubuntu1804-power8-test + batchtime: 10080 # 7 days + expansions: + test_env_vars: LD_LIBRARY_PATH=$(pwd) WT_BUILDDIR=$(pwd) + posix_configure_flags: -DCMAKE_TOOLCHAIN_FILE=../build_cmake/toolchains/mongodbtoolchain_v3_gcc.cmake -DCMAKE_C_FLAGS="-ggdb" -DHAVE_DIAGNOSTIC=1 -DENABLE_PYTHON=1 -DENABLE_ZLIB=1 -DENABLE_SNAPPY=1 -DENABLE_STRICT=1 -DCMAKE_INSTALL_PREFIX=$(pwd)/LOCAL_INSTALL + python_binary: '/opt/mongodbtoolchain/v3/bin/python3' + smp_command: -j $(grep -c ^processor /proc/cpuinfo) + cmake_generator: Ninja + make_command: ninja + is_cmake_build: true + tasks: + - name: compile + - name: make-check-test + - name: unit-test + - name: ubuntu1804-zseries display_name: "~ Ubuntu 18.04 zSeries" run_on: @@ -2828,3 +3058,21 @@ buildvariants: - name: ".stress-test-zseries-1" - name: ".stress-test-zseries-2" - name: ".stress-test-zseries-3" + +- name: ubuntu1804-zseries-cmake + display_name: "* Ubuntu 18.04 zSeries CMake" + run_on: + - ubuntu1804-zseries-test + batchtime: 10080 # 7 days + expansions: + test_env_vars: LD_LIBRARY_PATH=$(pwd) WT_BUILDDIR=$(pwd) + posix_configure_flags: -DCMAKE_TOOLCHAIN_FILE=../build_cmake/toolchains/mongodbtoolchain_v3_gcc.cmake -DCMAKE_C_FLAGS="-ggdb" -DHAVE_DIAGNOSTIC=1 -DENABLE_PYTHON=1 -DENABLE_ZLIB=1 -DENABLE_SNAPPY=1 -DENABLE_STRICT=1 -DCMAKE_INSTALL_PREFIX=$(pwd)/LOCAL_INSTALL + python_binary: '/opt/mongodbtoolchain/v3/bin/python3' + smp_command: -j $(grep -c ^processor /proc/cpuinfo) + cmake_generator: Ninja + make_command: ninja + is_cmake_build: true + tasks: + - name: compile + - name: make-check-test + - name: unit-test diff --git a/src/third_party/wiredtiger/test/evergreen/build_windows.ps1 b/src/third_party/wiredtiger/test/evergreen/build_windows.ps1 new file mode 100644 index 00000000000..57639d8b776 --- /dev/null +++ b/src/third_party/wiredtiger/test/evergreen/build_windows.ps1 @@ -0,0 +1,36 @@ +param ( + [bool]$configure = $false, + [bool]$build = $false, + [string]$vcvars_bat = "C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\VC\Auxiliary\Build\vcvars64.bat" +) + +# Source the vcvars to ensure we have access to the Visual Studio toolchain +cmd /c "`"$vcvars_bat`"&set" | +foreach { + if ($_ -match "=") { + $v = $_.split("="); set-item -force -path "ENV:\$($v[0])" -value "$($v[1])" + } +} + +# Ensure the PROCESSOR_ARCHITECTURE environment variable is set. This is sometimes not set +# when entering from a cygwin environment. +$env:PROCESSOR_ARCHITECTURE = "AMD64" + +# Ensure the swig binary location is in our PATH. +$env:Path += ";C:\swigwin-3.0.2" + +if (-not (Test-Path cmake_build)) { + mkdir cmake_build +} + +cd cmake_build + +# Configure build with CMake. +if( $configure -eq $true) { + C:\cmake\bin\cmake --no-warn-unused-cli -DSWIG_DIR="C:\swigwin-3.0.2" -DSWIG_EXECUTABLE="C:\swigwin-3.0.2\swig.exe" -DCMAKE_BUILD_TYPE='None' -DENABLE_PYTHON=1 -DENABLE_STRICT=1 -DCMAKE_TOOLCHAIN_FILE='..\build_cmake\toolchains\cl.cmake' -G "Ninja" ..\. +} + +# Execute Ninja build. +if( $build -eq $true) { + ninja +} diff --git a/src/third_party/wiredtiger/test/evergreen/compatibility_test_for_releases.sh b/src/third_party/wiredtiger/test/evergreen/compatibility_test_for_releases.sh index 39c6c4c2564..521a9787145 100755 --- a/src/third_party/wiredtiger/test/evergreen/compatibility_test_for_releases.sh +++ b/src/third_party/wiredtiger/test/evergreen/compatibility_test_for_releases.sh @@ -68,10 +68,10 @@ run_format() flags="-1q $(bflag $1)" args="" - args+="runs.type=row " # Temporarily disable column store tests args+="btree.prefix=0 " # Prefix testing isn't portable between releases args+="cache=80 " # Medium cache so there's eviction args+="checkpoints=1 " # Force periodic writes + args+="checksum=on " # Force checksums. args+="compression=snappy " # We only built with snappy, force the choice args+="data_source=table " args+="huffman_key=0 " # Not supoprted by newer releases @@ -80,6 +80,7 @@ run_format() args+="logging=1 " # Test log compatibility args+="logging_compression=snappy " # We only built with snappy, force the choice args+="rows=1000000 " + args+="runs.type=row " # Temporarily disable column store tests args+="salvage=0 " # Faster runs args+="timer=4 " args+="transaction.isolation=snapshot " # Older releases can't do lower isolation levels diff --git a/src/third_party/wiredtiger/test/evergreen/evg_cfg.py b/src/third_party/wiredtiger/test/evergreen/evg_cfg.py index 2caeda39101..e966ce0d69e 100755 --- a/src/third_party/wiredtiger/test/evergreen/evg_cfg.py +++ b/src/third_party/wiredtiger/test/evergreen/evg_cfg.py @@ -267,8 +267,9 @@ def evg_cfg(action, test_type): # We could get different string outputs when running 'git config remote.origin.url': # - 'git@github.com:wiredtiger/wiredtiger.git' (if run locally) # - 'ssh://git@github.com/wiredtiger/wiredtiger.git' (if run through SSH) + # - 'git://github.com/wiredtiger/wiredtiger' (if cloned anonymously) output = run('git config remote.origin.url') - if not 'github.com' in output or not 'wiredtiger.git' in output: + if not 'github.com' in output or (not 'wiredtiger.git' in output and output != 'git://github.com/wiredtiger/wiredtiger'): sys.exit("ERROR [%s]: need to run this script inside a wiredtiger repo" % prog) # Change directory to repo top level diff --git a/src/third_party/wiredtiger/test/evergreen/find_cmake.sh b/src/third_party/wiredtiger/test/evergreen/find_cmake.sh new file mode 100755 index 00000000000..72866bd4b53 --- /dev/null +++ b/src/third_party/wiredtiger/test/evergreen/find_cmake.sh @@ -0,0 +1,66 @@ +#!/bin/sh +set -o errexit # Exit the script with error if any of the commands fail + +# Adapted 'find_cmake' from mongo-c-driver evergreen infrastructure: +# https://github.com/mongodb/mongo-c-driver/blob/master/.evergreen/find-cmake.sh +find_cmake () +{ + if [ ! -z "$CMAKE" ]; then + return 0 + elif [ -f "/Applications/CMake.app/Contents/bin/cmake" ]; then + CMAKE="/Applications/CMake.app/Contents/bin/cmake" + CTEST="/Applications/CMake.app/Contents/bin/ctest" + elif [ -f "/opt/cmake/bin/cmake" ]; then + CMAKE="/opt/cmake/bin/cmake" + CTEST="/opt/cmake/bin/ctest" + elif command -v cmake 2>/dev/null; then + CMAKE=cmake + CTEST=ctest + elif uname -a | grep -iq 'x86_64 GNU/Linux'; then + if [ -f "$(pwd)/cmake-3.11.0/bin/cmake" ]; then + CMAKE="$(pwd)/cmake-3.11.0/bin/cmake" + CTEST="$(pwd)/cmake-3.11.0/bin/ctest" + return 0 + fi + curl --retry 5 https://cmake.org/files/v3.11/cmake-3.11.0-Linux-x86_64.tar.gz -sS --max-time 120 --fail --output cmake.tar.gz + mkdir cmake-3.11.0 + tar xzf cmake.tar.gz -C cmake-3.11.0 --strip-components=1 + CMAKE=$(pwd)/cmake-3.11.0/bin/cmake + CTEST=$(pwd)/cmake-3.11.0/bin/ctest + elif [ -f "/cygdrive/c/cmake/bin/cmake" ]; then + CMAKE="/cygdrive/c/cmake/bin/cmake" + CTEST="/cygdrive/c/cmake/bin/ctest" + elif [ -f "$(readlink -f cmake-install)"/bin/cmake ]; then + # If we have a custom cmake install from an earlier build step. + CMAKE="$(readlink -f cmake-install)/bin/cmake" + CTEST="$(readlink -f cmake-install)/bin/ctest" + fi + + if [ -z "$CMAKE" -o -z "$( $CMAKE --version 2>/dev/null )" ]; then + # Some images have no cmake yet, or a broken cmake (see: BUILD-8570) + echo "-- MAKE CMAKE --" + CMAKE_INSTALL_DIR=$(readlink -f cmake-install) + curl --retry 5 https://cmake.org/files/v3.11/cmake-3.11.0.tar.gz -sS --max-time 120 --fail --output cmake.tar.gz + tar xzf cmake.tar.gz + cd cmake-3.11.0 + ./bootstrap --prefix="${CMAKE_INSTALL_DIR}" + make -j8 + make install + cd .. + CMAKE="${CMAKE_INSTALL_DIR}/bin/cmake" + CTEST="${CMAKE_INSTALL_DIR}/bin/ctest" + echo "-- DONE MAKING CMAKE --" + fi + + echo "==========================================================" + echo "CMake and CTest environment variables, paths and versions:" + echo "CMAKE: ${CMAKE}" + echo "CTEST: ${CTEST}" + command -v ${CMAKE} + command -v ${CTEST} + ${CMAKE} --version + ${CTEST} --version + echo "==========================================================" +} + +find_cmake diff --git a/src/third_party/wiredtiger/test/format/bulk.c b/src/third_party/wiredtiger/test/format/bulk.c index c1decf24b6f..d1864741a77 100644 --- a/src/third_party/wiredtiger/test/format/bulk.c +++ b/src/third_party/wiredtiger/test/format/bulk.c @@ -60,7 +60,7 @@ bulk_commit_transaction(WT_SESSION *session) testutil_check(session->commit_transaction(session, buf)); /* Update the oldest timestamp, otherwise updates are pinned in memory. */ - timestamp_once(false, false); + timestamp_once(session, false, false); } /* diff --git a/src/third_party/wiredtiger/test/format/config.c b/src/third_party/wiredtiger/test/format/config.c index e6d9b6a9ae0..88964902d98 100644 --- a/src/third_party/wiredtiger/test/format/config.c +++ b/src/third_party/wiredtiger/test/format/config.c @@ -499,15 +499,21 @@ config_checksum(void) /* Choose a checksum mode if nothing was specified. */ if (!config_is_perm("disk.checksum")) switch (mmrand(NULL, 1, 10)) { - case 1: /* 10% */ + case 1: + case 2: + case 3: + case 4: /* 40% */ config_single("disk.checksum=on", false); break; - case 2: /* 10% */ + case 5: /* 10% */ config_single("disk.checksum=off", false); break; - default: /* 80% */ + case 6: /* 10% */ config_single("disk.checksum=uncompressed", false); break; + default: /* 40% */ + config_single("disk.checksum=unencrypted", false); + break; } } @@ -1082,7 +1088,7 @@ config_file(const char *name) break; } if (t == buf && *p == ']') { /* Closing brace, configuration starts after it. */ - while (isblank(*++p)) + while (isblank((unsigned char)*++p)) ; t = p--; } @@ -1368,9 +1374,11 @@ config_map_checksum(const char *s, u_int *vp) if (strcmp(s, "on") == 0) *vp = CHECKSUM_ON; else if (strcmp(s, "off") == 0) - *vp = CHECKSUM_ON; + *vp = CHECKSUM_OFF; else if (strcmp(s, "uncompressed") == 0) *vp = CHECKSUM_UNCOMPRESSED; + else if (strcmp(s, "unencrypted") == 0) + *vp = CHECKSUM_UNENCRYPTED; else testutil_die(EINVAL, "illegal checksum configuration: %s", s); } @@ -1411,6 +1419,8 @@ config_map_encryption(const char *s, u_int *vp) *vp = ENCRYPT_NONE; else if (strcmp(s, "rotn-7") == 0) *vp = ENCRYPT_ROTN_7; + else if (strcmp(s, "sodium") == 0) + *vp = ENCRYPT_SODIUM; else testutil_die(EINVAL, "illegal encryption configuration: %s", s); } diff --git a/src/third_party/wiredtiger/test/format/config.h b/src/third_party/wiredtiger/test/format/config.h index 0feb22f202c..07a6e2603ff 100644 --- a/src/third_party/wiredtiger/test/format/config.h +++ b/src/third_party/wiredtiger/test/format/config.h @@ -149,8 +149,8 @@ static CONFIG c[] = { {"checkpoint.wait", "seconds to wait if wiredtiger checkpoints configured", 0x0, 5, 100, 3600, &g.c_checkpoint_wait, NULL}, - {"disk.checksum", "checksum type (on | off | uncompressed)", C_IGNORE | C_STRING, 0, 0, 0, NULL, - &g.c_checksum}, + {"disk.checksum", "checksum type (on | off | uncompressed | unencrypted)", C_IGNORE | C_STRING, 0, + 0, 0, NULL, &g.c_checksum}, /* 5% */ {"disk.data_extend", "configure data file extension", C_BOOL, 5, 0, 0, &g.c_data_extend, NULL}, diff --git a/src/third_party/wiredtiger/test/format/format.h b/src/third_party/wiredtiger/test/format/format.h index cd46f43a781..619060e1881 100644 --- a/src/third_party/wiredtiger/test/format/format.h +++ b/src/third_party/wiredtiger/test/format/format.h @@ -35,14 +35,43 @@ #define EXTPATH "../../ext/" /* Extensions path */ +#ifndef LZ4_PATH #define LZ4_PATH EXTPATH "compressors/lz4/.libs/libwiredtiger_lz4.so" +#endif + +#ifndef SNAPPY_PATH #define SNAPPY_PATH EXTPATH "compressors/snappy/.libs/libwiredtiger_snappy.so" +#endif + +#ifndef ZLIB_PATH #define ZLIB_PATH EXTPATH "compressors/zlib/.libs/libwiredtiger_zlib.so" +#endif + +#ifndef ZSTD_PATH #define ZSTD_PATH EXTPATH "compressors/zstd/.libs/libwiredtiger_zstd.so" +#endif +#ifndef REVERSE_PATH #define REVERSE_PATH EXTPATH "collators/reverse/.libs/libwiredtiger_reverse_collator.so" +#endif +#ifndef ROTN_PATH #define ROTN_PATH EXTPATH "encryptors/rotn/.libs/libwiredtiger_rotn.so" +#endif + +#ifndef SODIUM_PATH +#define SODIUM_PATH EXTPATH "encryptors/sodium/.libs/libwiredtiger_sodium.so" +#endif + +/* + * To test the sodium encryptor, we use secretkey= rather than setting a keyid, because for a "real" + * (vs. test-only) encryptor, keyids require some kind of key server, and (a) setting one up for + * testing would be a nuisance and (b) currently the sodium encryptor doesn't support any anyway. + * + * It expects secretkey= to provide a hex-encoded 256-bit chacha20 key. This key will serve for + * testing purposes. + */ +#define SODIUM_TESTKEY "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef" #undef M #define M(v) ((v)*WT_MILLION) /* Million */ @@ -124,6 +153,12 @@ typedef struct { * that requires locking out transactional ops that set a timestamp. */ RWLOCK ts_lock; + /* + * Lock to prevent the stable timestamp from moving during the commit of prepared transactions. + * Otherwise, it may panic if the stable timestamp is moved to greater than or equal to the + * prepared transaction's durable timestamp when it is committing. + */ + RWLOCK prepare_commit_lock; uint64_t timestamp; /* Counter for timestamps */ uint64_t oldest_timestamp; /* Last timestamp used for oldest */ @@ -254,6 +289,7 @@ typedef struct { #define CHECKSUM_OFF 1 #define CHECKSUM_ON 2 #define CHECKSUM_UNCOMPRESSED 3 +#define CHECKSUM_UNENCRYPTED 4 u_int c_checksum_flag; /* Checksum flag value */ #define COMPRESS_NONE 1 @@ -266,6 +302,7 @@ typedef struct { #define ENCRYPT_NONE 1 #define ENCRYPT_ROTN_7 2 +#define ENCRYPT_SODIUM 3 u_int c_encryption_flag; /* Encryption flag value */ /* The page must be a multiple of the allocation size, and 512 always works. */ @@ -412,8 +449,8 @@ int snap_repeat_txn(WT_CURSOR *, TINFO *); void snap_repeat_update(TINFO *, bool); void snap_track(TINFO *, thread_op); void timestamp_init(void); -void timestamp_once(bool, bool); -void timestamp_teardown(void); +void timestamp_once(WT_SESSION *, bool, bool); +void timestamp_teardown(WT_SESSION *); int trace_config(const char *); void trace_init(void); void trace_ops_init(TINFO *); diff --git a/src/third_party/wiredtiger/test/format/format.i b/src/third_party/wiredtiger/test/format/format.i index b1530a6e18f..f0645ffb956 100644 --- a/src/third_party/wiredtiger/test/format/format.i +++ b/src/third_party/wiredtiger/test/format/format.i @@ -232,6 +232,38 @@ lock_writeunlock(WT_SESSION *session, RWLOCK *lock) } } +/* + * lock_readlock -- + * Wait to get read lock. + */ +static inline void +lock_readlock(WT_SESSION *session, RWLOCK *lock) +{ + testutil_assert(LOCK_INITIALIZED(lock)); + + if (lock->lock_type == LOCK_WT) { + __wt_readlock((WT_SESSION_IMPL *)session, &lock->l.wt); + } else { + testutil_check(pthread_rwlock_rdlock(&lock->l.pthread)); + } +} + +/* + * lock_writeunlock -- + * Release an exclusive lock. + */ +static inline void +lock_readunlock(WT_SESSION *session, RWLOCK *lock) +{ + testutil_assert(LOCK_INITIALIZED(lock)); + + if (lock->lock_type == LOCK_WT) { + __wt_readunlock((WT_SESSION_IMPL *)session, &lock->l.wt); + } else { + testutil_check(pthread_rwlock_unlock(&lock->l.pthread)); + } +} + #define trace_msg(fmt, ...) \ do { \ if (g.trace) { \ diff --git a/src/third_party/wiredtiger/test/format/ops.c b/src/third_party/wiredtiger/test/format/ops.c index 3fd5706efad..9ea615cb519 100644 --- a/src/third_party/wiredtiger/test/format/ops.c +++ b/src/third_party/wiredtiger/test/format/ops.c @@ -270,10 +270,6 @@ operations(u_int ops_seconds, bool lastrun) tinfo_init(); trace_msg("%s", "=============== thread ops start"); - /* Initialize locks to single-thread backups, failures, and timestamp updates. */ - lock_init(session, &g.backup_lock); - lock_init(session, &g.ts_lock); - for (i = 0; i < g.c_threads; ++i) { tinfo = tinfo_list[i]; testutil_check(__wt_thread_create(NULL, &tinfo->tid, ops, tinfo)); @@ -382,22 +378,20 @@ operations(u_int ops_seconds, bool lastrun) testutil_check(__wt_thread_join(NULL, ×tamp_tid)); g.workers_finished = false; - lock_destroy(session, &g.backup_lock); - lock_destroy(session, &g.ts_lock); - trace_msg("%s", "=============== thread ops stop"); /* * The system should be quiescent at this point, call rollback to stable. Generally, we expect * applications to do rollback-to-stable as part of the database open, but calling it outside of * the open path is expected in the case of applications that are "restarting" but skipping the - * close/re-open pair. + * close/re-open pair. Note we are not advancing the oldest timestamp, otherwise we wouldn't be + * able to replay operations from after rollback-to-stable completes. */ tinfo_rollback_to_stable(session); if (lastrun) { tinfo_teardown(); - timestamp_teardown(); + timestamp_teardown(session); } testutil_check(session->close(session, NULL)); @@ -505,6 +499,9 @@ commit_transaction(TINFO *tinfo, bool prepared) ts = 0; /* -Wconditional-uninitialized */ if (g.c_txn_timestamps) { + if (prepared) + lock_readlock(session, &g.prepare_commit_lock); + /* Lock out the oldest timestamp update. */ lock_writelock(session, &g.ts_lock); @@ -518,8 +515,11 @@ commit_transaction(TINFO *tinfo, bool prepared) } lock_writeunlock(session, &g.ts_lock); - } - testutil_check(session->commit_transaction(session, NULL)); + testutil_check(session->commit_transaction(session, NULL)); + if (prepared) + lock_readunlock(session, &g.prepare_commit_lock); + } else + testutil_check(session->commit_transaction(session, NULL)); /* Remember our oldest commit timestamp. */ tinfo->commit_ts = ts; diff --git a/src/third_party/wiredtiger/test/format/snap.c b/src/third_party/wiredtiger/test/format/snap.c index 1c934a5d187..e7709754523 100644 --- a/src/third_party/wiredtiger/test/format/snap.c +++ b/src/third_party/wiredtiger/test/format/snap.c @@ -162,7 +162,9 @@ snap_track(TINFO *tinfo, thread_op op) snap->kdata = drealloc(snap->kdata, ip->size); snap->kmemsize = ip->size; } - memcpy(snap->kdata, ip->data, snap->ksize = ip->size); + snap->ksize = ip->size; + if (ip->size > 0) + memcpy(snap->kdata, ip->data, ip->size); } if (op != REMOVE && op != TRUNCATE) { @@ -171,7 +173,9 @@ snap_track(TINFO *tinfo, thread_op op) snap->vdata = drealloc(snap->vdata, ip->size); snap->vmemsize = ip->size; } - memcpy(snap->vdata, ip->data, snap->vsize = ip->size); + snap->vsize = ip->size; + if (ip->size > 0) + memcpy(snap->vdata, ip->data, ip->size); } /* Move to the next slot, wrap at the end of the circular buffer. */ @@ -262,9 +266,12 @@ snap_verify(WT_CURSOR *cursor, TINFO *tinfo, SNAP_OPS *snap) } /* Check for simple matches. */ - if (ret == 0 && snap->op != REMOVE && value->size == snap->vsize && - memcmp(value->data, snap->vdata, value->size) == 0) - return (0); + if (ret == 0 && snap->op != REMOVE && value->size == snap->vsize) { + if (value->size == 0) + return (0); + if ((value->size > 0) && (memcmp(value->data, snap->vdata, value->size) == 0)) + return (0); + } if (ret == WT_NOTFOUND && snap->op == REMOVE) return (0); diff --git a/src/third_party/wiredtiger/test/format/t.c b/src/third_party/wiredtiger/test/format/t.c index 10f9763fa53..3656635e873 100644 --- a/src/third_party/wiredtiger/test/format/t.c +++ b/src/third_party/wiredtiger/test/format/t.c @@ -275,9 +275,15 @@ main(int argc, char *argv[]) timestamp_init(); trace_init(); + } + + /* Initialize locks to single-thread backups, failures, and timestamp updates. */ + lock_init(g.wts_session, &g.backup_lock); + lock_init(g.wts_session, &g.ts_lock); + lock_init(g.wts_session, &g.prepare_commit_lock); + if (!g.reopen) TIMED_MAJOR_OP(wts_load()); /* Load and verify initial records */ - } TIMED_MAJOR_OP(wts_verify(g.wts_conn, "verify")); TIMED_MAJOR_OP(wts_read_scan()); @@ -297,6 +303,10 @@ main(int argc, char *argv[]) */ TIMED_MAJOR_OP(wts_verify(g.wts_conn, "post-ops verify")); + lock_destroy(g.wts_session, &g.backup_lock); + lock_destroy(g.wts_session, &g.ts_lock); + lock_destroy(g.wts_session, &g.prepare_commit_lock); + track("shutting down", 0ULL, NULL); wts_close(&g.wts_conn, &g.wts_session); diff --git a/src/third_party/wiredtiger/test/format/util.c b/src/third_party/wiredtiger/test/format/util.c index 8c5efd007ee..c961d086516 100644 --- a/src/third_party/wiredtiger/test/format/util.c +++ b/src/third_party/wiredtiger/test/format/util.c @@ -259,7 +259,7 @@ timestamp_init(void) * Update the timestamp once. */ void -timestamp_once(bool allow_lag, bool final) +timestamp_once(WT_SESSION *session, bool allow_lag, bool final) { static const char *oldest_timestamp_str = "oldest_timestamp="; static const char *stable_timestamp_str = "stable_timestamp="; @@ -270,6 +270,9 @@ timestamp_once(bool allow_lag, bool final) conn = g.wts_conn; + /* Lock out transaction timestamp operations. */ + lock_writelock(session, &g.ts_lock); + if (final) g.oldest_timestamp = g.stable_timestamp = ++g.timestamp; else { @@ -291,10 +294,14 @@ timestamp_once(bool allow_lag, bool final) g.stable_timestamp = all_durable; } + lock_writeunlock(session, &g.ts_lock); + testutil_check(__wt_snprintf(buf, sizeof(buf), "%s%" PRIx64 ",%s%" PRIx64, oldest_timestamp_str, g.oldest_timestamp, stable_timestamp_str, g.stable_timestamp)); + lock_writelock(session, &g.prepare_commit_lock); testutil_check(conn->set_timestamp(conn, buf)); + lock_writeunlock(session, &g.prepare_commit_lock); trace_msg( "%-10s oldest=%" PRIu64 ", stable=%" PRIu64, "setts", g.oldest_timestamp, g.stable_timestamp); } @@ -319,9 +326,7 @@ timestamp(void *arg) while (!g.workers_finished) { random_sleep(&g.rnd, 15); - lock_writelock(session, &g.ts_lock); /* Lock out transaction timestamp operations. */ - timestamp_once(true, false); - lock_writeunlock(session, &g.ts_lock); + timestamp_once(session, true, false); } testutil_check(session->close(session, NULL)); @@ -333,13 +338,13 @@ timestamp(void *arg) * Wrap up timestamp operations. */ void -timestamp_teardown(void) +timestamp_teardown(WT_SESSION *session) { /* * Do a final bump of the oldest and stable timestamps, otherwise recent operations can prevent * verify from running. */ - timestamp_once(false, true); + timestamp_once(session, false, true); } /* diff --git a/src/third_party/wiredtiger/test/format/wts.c b/src/third_party/wiredtiger/test/format/wts.c index 48ea45b1022..e27f54a42a9 100644 --- a/src/third_party/wiredtiger/test/format/wts.c +++ b/src/third_party/wiredtiger/test/format/wts.c @@ -89,6 +89,36 @@ encryptor(uint32_t encrypt_flag) case ENCRYPT_ROTN_7: p = "rotn,keyid=7"; break; + case ENCRYPT_SODIUM: + p = "sodium,secretkey=" SODIUM_TESTKEY; + break; + default: + testutil_die(EINVAL, "illegal encryption flag: %#" PRIx32, encrypt_flag); + /* NOTREACHED */ + } + return (p); +} + +/* + * encryptor_at_open -- + * Configure encryption for wts_open(). + * + * This must set any secretkey. When keyids are in use it can return NULL. + */ +static const char * +encryptor_at_open(uint32_t encrypt_flag) +{ + const char *p; + + p = NULL; + switch (encrypt_flag) { + case ENCRYPT_NONE: + break; + case ENCRYPT_ROTN_7: + break; + case ENCRYPT_SODIUM: + p = "sodium,secretkey=" SODIUM_TESTKEY; + break; default: testutil_die(EINVAL, "illegal encryption flag: %#" PRIx32, encrypt_flag); /* NOTREACHED */ @@ -154,6 +184,7 @@ create_database(const char *home, WT_CONNECTION **connp) WT_CONNECTION *conn; size_t max; char config[8 * 1024], *p; + const char *enc; p = config; max = sizeof(config); @@ -190,8 +221,11 @@ create_database(const char *home, WT_CONNECTION **connp) compressor(g.c_logging_compression_flag)); /* Encryption. */ - if (g.c_encryption) - CONFIG_APPEND(p, ",encryption=(name=%s)", encryptor(g.c_encryption_flag)); + if (g.c_encryption) { + enc = encryptor(g.c_encryption_flag); + if (enc != NULL) + CONFIG_APPEND(p, ",encryption=(name=%s)", enc); + } /* Miscellaneous. */ #ifdef HAVE_POSIX_MEMALIGN @@ -255,11 +289,12 @@ create_database(const char *home, WT_CONNECTION **connp) CONFIG_APPEND(p, "]"); /* Extensions. */ - CONFIG_APPEND(p, ",extensions=[\"%s\", \"%s\", \"%s\", \"%s\", \"%s\", \"%s\"],", + CONFIG_APPEND(p, ",extensions=[\"%s\", \"%s\", \"%s\", \"%s\", \"%s\", \"%s\", \"%s\"],", g.c_reverse ? REVERSE_PATH : "", access(LZ4_PATH, R_OK) == 0 ? LZ4_PATH : "", access(ROTN_PATH, R_OK) == 0 ? ROTN_PATH : "", access(SNAPPY_PATH, R_OK) == 0 ? SNAPPY_PATH : "", - access(ZLIB_PATH, R_OK) == 0 ? ZLIB_PATH : "", access(ZSTD_PATH, R_OK) == 0 ? ZSTD_PATH : ""); + access(ZLIB_PATH, R_OK) == 0 ? ZLIB_PATH : "", access(ZSTD_PATH, R_OK) == 0 ? ZSTD_PATH : "", + access(SODIUM_PATH, R_OK) == 0 ? SODIUM_PATH : ""); /* * Put configuration file configuration options second to last. Put command line configuration @@ -343,6 +378,9 @@ create_object(WT_CONNECTION *conn) case CHECKSUM_UNCOMPRESSED: CONFIG_APPEND(p, ",checksum=\"uncompressed\""); break; + case CHECKSUM_UNENCRYPTED: + CONFIG_APPEND(p, ",checksum=\"unencrypted\""); + break; } /* Configure compression. */ @@ -425,19 +463,28 @@ void wts_open(const char *home, WT_CONNECTION **connp, WT_SESSION **sessionp, bool allow_verify) { WT_CONNECTION *conn; - const char *config; + size_t max; + char config[1024], *p; + const char *enc; *connp = NULL; *sessionp = NULL; + p = config; + max = sizeof(config); + config[0] = '\0'; + + enc = encryptor_at_open(g.c_encryption_flag); + if (enc != NULL) + CONFIG_APPEND(p, ",encryption=(name=%s)", enc); + /* If in-memory, there's only a single, shared WT_CONNECTION handle. */ if (g.c_in_memory != 0) conn = g.wts_conn_inmemory; else { - config = ""; #if WIREDTIGER_VERSION_MAJOR >= 10 if (g.c_verify && allow_verify) - config = ",verify_metadata=true"; + CONFIG_APPEND(p, ",verify_metadata=true"); #else WT_UNUSED(allow_verify); #endif diff --git a/src/third_party/wiredtiger/test/packing/intpack-test3.c b/src/third_party/wiredtiger/test/packing/intpack-test3.c index 2f8b00aacce..3207ac4ff19 100644 --- a/src/third_party/wiredtiger/test/packing/intpack-test3.c +++ b/src/third_party/wiredtiger/test/packing/intpack-test3.c @@ -110,19 +110,22 @@ test_spread(int64_t start, int64_t before, int64_t after) int main(void) { - int64_t i; + int64_t i, range, start_int64; + + range = 1025; + start_int64 = INT64_MAX - range; /* reduce start point by range to avoid integer overflow */ /* * Test all values in a range, to ensure pack/unpack of small numbers (which most actively use * different numbers of bits) works. */ test_spread(0, 100000, 100000); - test_spread(INT16_MAX, 1025, 1025); - test_spread(INT32_MAX, 1025, 1025); - test_spread(INT64_MAX, 1025, 1025); + test_spread(INT16_MAX, range, range); + test_spread(INT32_MAX, range, range); + test_spread(start_int64, range, range); /* Test bigger numbers */ - for (i = INT64_MAX; i > 0; i = i / 2) - test_spread(i, 1025, 1025); + for (i = start_int64; i > 0; i = i / 2) + test_spread(i, range, range); printf("\n"); return (0); diff --git a/src/third_party/wiredtiger/test/salvage/salvage.c b/src/third_party/wiredtiger/test/salvage/salvage.c index 7f3e1abe084..ba2929ac638 100644 --- a/src/third_party/wiredtiger/test/salvage/salvage.c +++ b/src/third_party/wiredtiger/test/salvage/salvage.c @@ -699,7 +699,7 @@ empty(int cnt) if (page_type == WT_PAGE_COL_FIX) for (i = 0; i < cnt; ++i) - testutil_assert(fputs("\\00\n", res_fp)); + testutil_assert(fputs("\\00\n", res_fp) != EOF); } /* diff --git a/src/third_party/wiredtiger/test/suite/hook_tiered.py b/src/third_party/wiredtiger/test/suite/hook_tiered.py index 5bb97ea399b..58ccc505416 100755 --- a/src/third_party/wiredtiger/test/suite/hook_tiered.py +++ b/src/third_party/wiredtiger/test/suite/hook_tiered.py @@ -34,16 +34,25 @@ # # Substitute tiered tables for regular (row-store) tables in Python tests. # -# These hooks can be used to run the existing cursor tests on tiered tables. -# They identify tests that create row-store tables and create tiered tables -# instead. The hook takes an optional argument to specify how many tiers -# to create. The default is 2. +# These hooks are intended for basic testing of tiered tables. There are several +# pieces of functionality here. # -# To run with 3 tiers per table: -# ../test/suite/run.py --hooks tiered=3 cursor +# 1. We add tiered storage parameters to the config when calling wiredtiger_open() +# +# 2. If we create an object that is *not* a table:, we add options to its config +# so that it will be stored local-only. Tiered storage isn't intended (yet?) for +# use with lsm or column store. +# +# 3. We add calls to flush_tier(). Currently we only flush after a checkpoint() call, +# but we should add others. +# +# 4. We stub out some functions that aren't supported by tiered tables. This will +# break tests of those functions. But often when they are used in other tests, we +# can get away with returning success without performing the operation. +# +# To run, for example, the cursor tests with these hooks enabled: +# ../test/suite/run.py --hooks tiered cursor # -# The hooks work with may other tests in the python suite but also encounter -# a variety of failures that I haven't tried to sort out. from __future__ import print_function import os, sys, wthooks @@ -52,66 +61,149 @@ from wttest import WiredTigerTestCase # These are the hook functions that are run when particular APIs are called. +# Add the local storage extension whenever we call wiredtiger_open +def wiredtiger_open_tiered(ignored_self, args): + auth_token = "test_token" + bucket = "mybucket" + extension_name = "local_store" + prefix = "pfx-" + extension_libs = WiredTigerTestCase.findExtension('storage_sources', extension_name) + if len(extension_libs) == 0: + raise Exception(extension_name + ' storage source extension not found') + + if not os.path.exists(bucket): + os.mkdir(bucket) + tier_string = ',tiered_storage=(auth_token=%s,' % auth_token + \ + 'bucket=%s,' % bucket + \ + 'bucket_prefix=%s,' % prefix + \ + 'name=%s),tiered_manager=(wait=0),' % extension_name + \ + 'extensions=[\"%s\"],' % extension_libs[0] + + args = list(args) # convert from a readonly tuple to a writeable list + args[-1] += tier_string # Modify the list + + WiredTigerTestCase.verbose(None, 3, + ' Calling wiredtiger_open with config = \'{}\''.format(args)) + + return args + +# Called to replace Connection.close +# Insert a call to flush_tier before closing connection. +def connection_close_replace(orig_connection_close, connection_self, config): + s = connection_self.open_session(None) + s.flush_tier(None) + s.close() + ret = orig_connection_close(connection_self, config) + return ret + +# Called to replace Session.alter +def session_alter_replace(orig_session_alter, session_self, uri, config): + # Alter isn't implemented for tiered tables. Only call it if this can't be the uri + # of a tiered table. Note this isn't a precise match for when we did/didn't create + # a tiered table, but we don't have the create config around to check. + ret = 0 + if not uri.startswith("table:"): + ret = orig_session_alter(session_self, uri, config) + return ret + +# Called to replace Session.checkpoint. +# We add a call to flush_tier after the checkpoint to make sure we are exercising tiered +# functionality. +def session_checkpoint_replace(orig_session_checkpoint, session_self, config): + ret = orig_session_checkpoint(session_self, config) + if ret != 0: + return ret + WiredTigerTestCase.verbose(None, 3, + ' Calling flush_tier() after checkpoint') + return session_self.flush_tier(None) + +# Called to replace Session.compact +def session_compact_replace(orig_session_compact, session_self, uri, config): + # Compact isn't implemented for tiered tables. Only call it if this can't be the uri + # of a tiered table. Note this isn't a precise match for when we did/didn't create + # a tiered table, but we don't have the create config around to check. + ret = 0 + if not uri.startswith("table:"): + ret = orig_session_compact(session_self, uri, config) + return ret + # Called to replace Session.create -def session_create_replace(ntiers, orig_session_create, session_self, uri, config): +def session_create_replace(orig_session_create, session_self, uri, config): if config == None: - base_config = "" - else: - base_config = config - - # If the test is creating a table (not colstore or lsm), create a tiered table instead, - # using arg to determine number of tiers. Otherwise just do the create as normal. - # - # NOTE: The following code uses the old API for creating tiered tables. As of WT-7173 - # this no longer works. It will be updated and fixed in WT-7440. - if (uri.startswith("table:") and "key_format=r" not in base_config and - "type=lsm" not in base_config): - tier_string = "" - for i in range(ntiers): - new_uri = uri.replace('table:', 'file:tier' + str(i) + '_') - orig_session_create(session_self, new_uri, config) - tier_string = tier_string + '"' + new_uri + '", ' - tier_config = 'type=tiered,tiered=(tiers=(' + tier_string[0:-2] + ')),' + base_config - WiredTigerTestCase.verbose(None, 3, - 'Creating tiered table {} with config = \'{}\''.format(uri, tier_config)) - ret = orig_session_create(session_self, uri, tier_config) + new_config = "" else: - ret = orig_session_create(session_self, uri, config) + new_config = config + + # If the test isn't creating a table (i.e., it's a column store or lsm) create it as a + # "local only" object. Otherwise we get tiered storage from the connection defaults. + if not uri.startswith("table:") or "key_format=r" in new_config or "type=lsm" in new_config: + new_config = new_config + ',tiered_storage=(name=none)' + + WiredTigerTestCase.verbose(None, 3, + ' Creating \'{}\' with config = \'{}\''.format(uri, new_config)) + ret = orig_session_create(session_self, uri, new_config) return ret # Called to replace Session.drop -def session_drop_replace(ntiers, orig_session_drop, session_self, uri, config): - # Drop isn't implemented for tiered tables. Only do the delete if this could be a - # uri we created a tiered table for. Note this isn't a precise match for when we - # did/didn't create a tiered table, but we don't have the create config around to check. +def session_drop_replace(orig_session_drop, session_self, uri, config): + # Drop isn't implemented for tiered tables. Only call it if this can't be the uri + # of a tiered table. Note this isn't a precise match for when we did/didn't create + # a tiered table, but we don't have the create config around to check. ret = 0 if not uri.startswith("table:"): ret = orig_session_drop(session_self, uri, config) return ret +# Called to replace Session.rename +def session_rename_replace(orig_session_rename, session_self, uri, newuri, config): + # Rename isn't implemented for tiered tables. Only call it if this can't be the uri + # of a tiered table. Note this isn't a precise match for when we did/didn't create + # a tiered table, but we don't have the create config around to check. + ret = 0 + if not uri.startswith("table:"): + ret = orig_session_rename(session_self, uri, newuri, config) + return ret + +# Called to replace Session.salvage +def session_salvage_replace(orig_session_salvage, session_self, uri, config): + # Salvage isn't implemented for tiered tables. Only call it if this can't be the uri + # of a tiered table. Note this isn't a precise match for when we did/didn't create + # a tiered table, but we don't have the create config around to check. + ret = 0 + if not uri.startswith("table:"): + ret = orig_session_salvage(session_self, uri, config) + return ret + # Called to replace Session.verify -def session_verify_replace(ntiers, orig_session_verify, session_self, uri): - return 0 +def session_verify_replace(orig_session_verify, session_self, uri, config): + # Verify isn't implemented for tiered tables. Only call it if this can't be the uri + # of a tiered table. Note this isn't a precise match for when we did/didn't create + # a tiered table, but we don't have the create config around to check. + ret = 0 + if not uri.startswith("table:"): + ret = orig_session_verify(session_self, uri, config) + return ret # Every hook file must have one or more classes descended from WiredTigerHook # This is where the hook functions are 'hooked' to API methods. class TieredHookCreator(wthooks.WiredTigerHookCreator): - def __init__(self, ntiers=0): - # Argument specifies the number of tiers to test. The default is 2. - if ntiers == None: - self.ntiers = 2 - else: - self.ntiers = int(ntiers) - - # Is this test one we should skip? We skip tests of features supported on standard - # tables but not tiered tables, specififically cursor caching and checkpoint cursors. + def __init__(self, arg=0): + # Caller can specify an optional command-line argument. We're not using it + # now, but this is where it would show up. + return + + # Is this test one we should skip? def skip_test(self, test): - skip = ["bulk_backup", - "checkpoint", - "test_cursor13_big", - "test_cursor13_drops", - "test_cursor13_dup", - "test_cursor13_reopens"] + # Skip any test that contains one of these strings as a substring + skip = ["backup", # Can't backup a tiered table + "cursor13_ckpt", # Checkpoint tests with cached cursors + "cursor13_drops", # Tests that require working drop implementation + "cursor13_dup", # More cursor cache tests + "cursor13_reopens", # More cursor cache tests + "lsm", # If the test name tells us it uses lsm ignore it + "test_config_json", # create replacement can't handle a json config string + "test_cursor_big", # Cursor caching verified with stats + "tiered"] for item in skip: if item in str(test): return True @@ -124,17 +216,39 @@ class TieredHookCreator(wthooks.WiredTigerHookCreator): return new_tests def setup_hooks(self): + orig_connection_close = self.Connection['close'] + self.Connection['close'] = (wthooks.HOOK_REPLACE, lambda s, config: + connection_close_replace(orig_connection_close, s, config)) + + orig_session_alter = self.Session['alter'] + self.Session['alter'] = (wthooks.HOOK_REPLACE, lambda s, uri, config: + session_alter_replace(orig_session_alter, s, uri, config)) + + orig_session_compact = self.Session['compact'] + self.Session['compact'] = (wthooks.HOOK_REPLACE, lambda s, uri, config: + session_compact_replace(orig_session_compact, s, uri, config)) + orig_session_create = self.Session['create'] self.Session['create'] = (wthooks.HOOK_REPLACE, lambda s, uri, config: - session_create_replace(self.ntiers, orig_session_create, s, uri, config)) + session_create_replace(orig_session_create, s, uri, config)) orig_session_drop = self.Session['drop'] - self.Session['drop'] = (wthooks.HOOK_REPLACE, lambda s, uri, config: - session_drop_replace(self.ntiers, orig_session_drop, s, uri, config)) + self.Session['drop'] = (wthooks.HOOK_REPLACE, lambda s, uri, config=None: + session_drop_replace(orig_session_drop, s, uri, config)) + + orig_session_rename = self.Session['rename'] + self.Session['rename'] = (wthooks.HOOK_REPLACE, lambda s, uri, newuri, config=None: + session_rename_replace(orig_session_rename, s, uri, newuri, config)) + + orig_session_salvage = self.Session['salvage'] + self.Session['salvage'] = (wthooks.HOOK_REPLACE, lambda s, uri, config: + session_salvage_replace(orig_session_salvage, s, uri, config)) orig_session_verify = self.Session['verify'] - self.Session['verify'] = (wthooks.HOOK_REPLACE, lambda s, uri: - session_verify_replace(self.ntiers, orig_session_verify, s, uri)) + self.Session['verify'] = (wthooks.HOOK_REPLACE, lambda s, uri, config: + session_verify_replace(orig_session_verify, s, uri, config)) + + self.wiredtiger['wiredtiger_open'] = (wthooks.HOOK_ARGS, wiredtiger_open_tiered) # Every hook file must have a top level initialize function, # returning a list of WiredTigerHook objects. diff --git a/src/third_party/wiredtiger/test/suite/test_backup01.py b/src/third_party/wiredtiger/test/suite/test_backup01.py index 9a7141d29ff..aa30471706e 100644 --- a/src/third_party/wiredtiger/test/suite/test_backup01.py +++ b/src/third_party/wiredtiger/test/suite/test_backup01.py @@ -27,7 +27,8 @@ # OTHER DEALINGS IN THE SOFTWARE. # # [TEST_TAGS] -# backup:correctness:full_backup +# wt_util +# backup:cursors # [END_TAGS] # diff --git a/src/third_party/wiredtiger/test/suite/test_backup11.py b/src/third_party/wiredtiger/test/suite/test_backup11.py index e15f1b01dd7..14023cb6ee3 100644 --- a/src/third_party/wiredtiger/test/suite/test_backup11.py +++ b/src/third_party/wiredtiger/test/suite/test_backup11.py @@ -25,6 +25,10 @@ # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. +# +# [TEST_TAGS] +# backup:cursors +# [END_TAGS] import wiredtiger, wttest import os, shutil diff --git a/src/third_party/wiredtiger/test/suite/test_base02.py b/src/third_party/wiredtiger/test/suite/test_base02.py index 7d272bb2aaf..9fb3fa0d82c 100644 --- a/src/third_party/wiredtiger/test/suite/test_base02.py +++ b/src/third_party/wiredtiger/test/suite/test_base02.py @@ -26,6 +26,10 @@ # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. # +# [TEST_TAGS] +# config_api +# [END_TAGS] +# # test_base02.py # Configuration # diff --git a/src/third_party/wiredtiger/test/suite/test_bug004.py b/src/third_party/wiredtiger/test/suite/test_bug004.py index cce2968e826..8af470d5161 100644 --- a/src/third_party/wiredtiger/test/suite/test_bug004.py +++ b/src/third_party/wiredtiger/test/suite/test_bug004.py @@ -26,6 +26,10 @@ # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. # +# [TEST_TAGS] +# reconciliation:overflow_keys +# [END_TAGS] +# # test_bug004.py # Regression tests. diff --git a/src/third_party/wiredtiger/test/suite/test_bug005.py b/src/third_party/wiredtiger/test/suite/test_bug005.py index 3e4a2d9bb3f..a74ea0a51ed 100644 --- a/src/third_party/wiredtiger/test/suite/test_bug005.py +++ b/src/third_party/wiredtiger/test/suite/test_bug005.py @@ -26,6 +26,10 @@ # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. # +# [TEST_TAGS] +# session_api:verify +# [END_TAGS] +# # test_bug005.py # Regression tests. diff --git a/src/third_party/wiredtiger/test/suite/test_bug008.py b/src/third_party/wiredtiger/test/suite/test_bug008.py index aada1b6c195..5b04cbffb8a 100644 --- a/src/third_party/wiredtiger/test/suite/test_bug008.py +++ b/src/third_party/wiredtiger/test/suite/test_bug008.py @@ -26,8 +26,13 @@ # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. # -# test_bug008.py -# Regression tests. +# [TEST_TAGS] +# cursors:search_near +# cursors:search +# [END_TAGS] +# +# test_bug008.py +# Regression tests for cursor search and cursor search near. import wiredtiger, wttest from wtdataset import SimpleDataSet diff --git a/src/third_party/wiredtiger/test/suite/test_bug024.py b/src/third_party/wiredtiger/test/suite/test_bug024.py index 661a35617c4..53e09543435 100644 --- a/src/third_party/wiredtiger/test/suite/test_bug024.py +++ b/src/third_party/wiredtiger/test/suite/test_bug024.py @@ -25,6 +25,10 @@ # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. +# +# [TEST_TAGS] +# connection_api:turtle_file +# [END_TAGS] from helper import copy_wiredtiger_home import wiredtiger, wttest diff --git a/src/third_party/wiredtiger/test/suite/test_checkpoint02.py b/src/third_party/wiredtiger/test/suite/test_checkpoint02.py index 66196ebe5f5..cf02983200e 100755 --- a/src/third_party/wiredtiger/test/suite/test_checkpoint02.py +++ b/src/third_party/wiredtiger/test/suite/test_checkpoint02.py @@ -27,7 +27,7 @@ # OTHER DEALINGS IN THE SOFTWARE. # # [TEST_TAGS] -# checkpoints:correctness:checkpoint_data +# checkpoint # [END_TAGS] # diff --git a/src/third_party/wiredtiger/test/suite/test_checkpoint03.py b/src/third_party/wiredtiger/test/suite/test_checkpoint03.py index 158a768f44d..0f4fba34186 100644 --- a/src/third_party/wiredtiger/test/suite/test_checkpoint03.py +++ b/src/third_party/wiredtiger/test/suite/test_checkpoint03.py @@ -27,7 +27,7 @@ # OTHER DEALINGS IN THE SOFTWARE. # # [TEST_TAGS] -# checkpoints:correctness:checkpoint_data +# checkpoint:history_store # [END_TAGS] # # test_checkpoint03.py diff --git a/src/third_party/wiredtiger/test/suite/test_checkpoint08.py b/src/third_party/wiredtiger/test/suite/test_checkpoint08.py index f6ea1a1c888..5a8e8fe647e 100755 --- a/src/third_party/wiredtiger/test/suite/test_checkpoint08.py +++ b/src/third_party/wiredtiger/test/suite/test_checkpoint08.py @@ -26,6 +26,10 @@ # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. # +# [TEST_TAGS] +# checkpoint:obsolete_data +# [END_TAGS] +# # test_checkpoint08.py # Test that the btree checkpoint is not skipped if there are obsolete pages. diff --git a/src/third_party/wiredtiger/test/suite/test_checkpoint_snapshot01.py b/src/third_party/wiredtiger/test/suite/test_checkpoint_snapshot01.py index 1f549bc7138..ce839469ed1 100644 --- a/src/third_party/wiredtiger/test/suite/test_checkpoint_snapshot01.py +++ b/src/third_party/wiredtiger/test/suite/test_checkpoint_snapshot01.py @@ -25,6 +25,10 @@ # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. +# +# [TEST_TAGS] +# checkpoint:metadata +# [END_TAGS] from helper import copy_wiredtiger_home import wiredtiger, wttest diff --git a/src/third_party/wiredtiger/test/suite/test_checkpoint_snapshot02.py b/src/third_party/wiredtiger/test/suite/test_checkpoint_snapshot02.py index 6e6edadd18e..2f93fef82b2 100644 --- a/src/third_party/wiredtiger/test/suite/test_checkpoint_snapshot02.py +++ b/src/third_party/wiredtiger/test/suite/test_checkpoint_snapshot02.py @@ -28,8 +28,8 @@ import fnmatch, os, shutil, threading, time from wtthread import checkpoint_thread, op_thread -from helper import copy_wiredtiger_home -import wiredtiger, wttest +from helper import simulate_crash_restart +import wttest from wtdataset import SimpleDataSet from wtscenario import make_scenarios from wiredtiger import stat @@ -47,23 +47,36 @@ class test_checkpoint_snapshot02(wttest.WiredTigerTestCase): uri = "table:test_checkpoint_snapshot02" nrows = 1000 + key_format_values = [ + ('column', dict(key_format='r')), + ('integer_row', dict(key_format='i')), + ] + + scenarios = make_scenarios(key_format_values) + def conn_config(self): config = 'cache_size=10MB,statistics=(all),statistics_log=(json,on_close,wait=1),log=(enabled=true),timing_stress_for_test=[checkpoint_slow]' return config - def large_updates(self, uri, value, ds, nrows): + def large_updates(self, uri, value, ds, nrows, commit_ts): # Update a large number of records. session = self.session cursor = session.open_cursor(uri) - for i in range(0, nrows): + for i in range(1, nrows+1): session.begin_transaction() cursor[ds.key(i)] = value - session.commit_transaction() + if commit_ts == 0: + session.commit_transaction() + else: + session.commit_transaction('commit_timestamp=' + timestamp_str(commit_ts)) cursor.close() - def check(self, check_value, uri, nrows): + def check(self, check_value, uri, nrows, read_ts): session = self.session - session.begin_transaction() + if read_ts == 0: + session.begin_transaction() + else: + session.begin_transaction('read_timestamp=' + timestamp_str(read_ts)) cursor = session.open_cursor(uri) count = 0 for k, v in cursor: @@ -74,26 +87,74 @@ class test_checkpoint_snapshot02(wttest.WiredTigerTestCase): def test_checkpoint_snapshot(self): + ds = SimpleDataSet(self, self.uri, 0, key_format=self.key_format, value_format="S",config='log=(enabled=false)') + ds.populate() + valuea = "aaaaa" * 100 + + self.large_updates(self.uri, valuea, ds, self.nrows, 0) + self.check(valuea, self.uri, self.nrows, 0) + + session1 = self.conn.open_session() + session1.begin_transaction() + cursor1 = session1.open_cursor(self.uri) + + for i in range(self.nrows+1, (self.nrows*2)+1): + cursor1.set_key(ds.key(i)) + cursor1.set_value(valuea) + self.assertEqual(cursor1.insert(), 0) + + # Create a checkpoint thread + done = threading.Event() + ckpt = checkpoint_thread(self.conn, done) + try: + ckpt.start() + # Sleep for sometime so that checkpoint starts before committing last transaction. + time.sleep(2) + session1.commit_transaction() + + finally: + done.set() + ckpt.join() + + #Simulate a crash by copying to a new directory(RESTART). + simulate_crash_restart(self, ".", "RESTART") + + # Check the table contains the last checkpointed value. + self.check(valuea, self.uri, self.nrows, 0) + + stat_cursor = self.session.open_cursor('statistics:', None, None) + inconsistent_ckpt = stat_cursor[stat.conn.txn_rts_inconsistent_ckpt][2] + keys_removed = stat_cursor[stat.conn.txn_rts_keys_removed][2] + stat_cursor.close() + + self.assertGreater(inconsistent_ckpt, 0) + self.assertGreaterEqual(keys_removed, 0) + + def test_checkpoint_snapshot_with_timestamp(self): + ds = SimpleDataSet(self, self.uri, 0, key_format="S", value_format="S",config='log=(enabled=false)') ds.populate() valuea = "aaaaa" * 100 - valueb = "bbbbb" * 100 - valuec = "ccccc" * 100 - valued = "ddddd" * 100 - cursor = self.session.open_cursor(self.uri) - self.large_updates(self.uri, valuea, ds, self.nrows) + # Pin oldest and stable timestamps to 10. + self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(10) + + ',stable_timestamp=' + timestamp_str(10)) - self.check(valuea, self.uri, self.nrows) + self.large_updates(self.uri, valuea, ds, self.nrows, 20) + self.check(valuea, self.uri, self.nrows, 20) session1 = self.conn.open_session() session1.begin_transaction() cursor1 = session1.open_cursor(self.uri) - for i in range(self.nrows, self.nrows*2): + for i in range(self.nrows+1, (self.nrows*2)+1): cursor1.set_key(ds.key(i)) cursor1.set_value(valuea) self.assertEqual(cursor1.insert(), 0) + session1.timestamp_transaction('commit_timestamp=' + timestamp_str(30)) + + # Set stable timestamp to 40 + self.conn.set_timestamp('stable_timestamp=' + timestamp_str(40)) # Create a checkpoint thread done = threading.Event() @@ -109,28 +170,88 @@ class test_checkpoint_snapshot02(wttest.WiredTigerTestCase): ckpt.join() #Simulate a crash by copying to a new directory(RESTART). - copy_wiredtiger_home(self, ".", "RESTART") + simulate_crash_restart(self, ".", "RESTART") + + # Check the table contains the last checkpointed value. + self.check(valuea, self.uri, self.nrows, 30) + + stat_cursor = self.session.open_cursor('statistics:', None, None) + inconsistent_ckpt = stat_cursor[stat.conn.txn_rts_inconsistent_ckpt][2] + keys_removed = stat_cursor[stat.conn.txn_rts_keys_removed][2] + stat_cursor.close() + + self.assertGreater(inconsistent_ckpt, 0) + self.assertGreaterEqual(keys_removed, 0) + + def test_checkpoint_snapshot_with_txnid_and_timestamp(self): + + ds = SimpleDataSet(self, self.uri, 0, key_format="S", value_format="S",config='log=(enabled=false)') + ds.populate() + valuea = "aaaaa" * 100 + valueb = "bbbbb" * 100 + + # Pin oldest and stable timestamps to 10. + self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(10) + + ',stable_timestamp=' + timestamp_str(10)) + + session1 = self.conn.open_session() + session1.begin_transaction() + + self.large_updates(self.uri, valuea, ds, self.nrows, 20) + self.check(valuea, self.uri, self.nrows, 20) + + session2 = self.conn.open_session() + session2.begin_transaction() + cursor2 = session2.open_cursor(self.uri) + + for i in range((self.nrows+1), (self.nrows*2)+1): + cursor2.set_key(ds.key(i)) + cursor2.set_value(valuea) + self.assertEqual(cursor2.insert(), 0) + session1.timestamp_transaction('commit_timestamp=' + timestamp_str(30)) - # Open the new directory. - self.conn = self.setUpConnectionOpen("RESTART") - self.session = self.setUpSessionOpen(self.conn) + # Set stable timestamp to 40 + self.conn.set_timestamp('stable_timestamp=' + timestamp_str(40)) + + # Create a checkpoint thread + done = threading.Event() + ckpt = checkpoint_thread(self.conn, done) + try: + ckpt.start() + # Sleep for sometime so that checkpoint starts before committing last transaction. + time.sleep(2) + session2.commit_transaction() + + finally: + done.set() + ckpt.join() + + session1.rollback_transaction() + #Simulate a crash by copying to a new directory(RESTART). + simulate_crash_restart(self, ".", "RESTART") # Check the table contains the last checkpointed value. - self.check(valuea, self.uri, self.nrows) + self.check(valuea, self.uri, self.nrows, 30) stat_cursor = self.session.open_cursor('statistics:', None, None) inconsistent_ckpt = stat_cursor[stat.conn.txn_rts_inconsistent_ckpt][2] keys_removed = stat_cursor[stat.conn.txn_rts_keys_removed][2] - keys_restored = stat_cursor[stat.conn.txn_rts_keys_restored][2] - pages_visited = stat_cursor[stat.conn.txn_rts_pages_visited][2] - upd_aborted = stat_cursor[stat.conn.txn_rts_upd_aborted][2] stat_cursor.close() self.assertGreater(inconsistent_ckpt, 0) - self.assertEqual(upd_aborted, 0) self.assertGreaterEqual(keys_removed, 0) - self.assertEqual(keys_restored, 0) - self.assertGreaterEqual(pages_visited, 0) + + simulate_crash_restart(self, "RESTART", "RESTART2") + # Check the table contains the last checkpointed value. + self.check(valuea, self.uri, self.nrows, 30) + + stat_cursor = self.session.open_cursor('statistics:', None, None) + inconsistent_ckpt = stat_cursor[stat.conn.txn_rts_inconsistent_ckpt][2] + keys_removed = stat_cursor[stat.conn.txn_rts_keys_removed][2] + stat_cursor.close() + + self.assertGreaterEqual(inconsistent_ckpt, 0) + self.assertEqual(keys_removed, 0) if __name__ == '__main__': wttest.run() diff --git a/src/third_party/wiredtiger/test/suite/test_checkpoint_snapshot03.py b/src/third_party/wiredtiger/test/suite/test_checkpoint_snapshot03.py index 09357a95332..293b16efaf5 100644 --- a/src/third_party/wiredtiger/test/suite/test_checkpoint_snapshot03.py +++ b/src/third_party/wiredtiger/test/suite/test_checkpoint_snapshot03.py @@ -25,6 +25,10 @@ # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. +# +# [TEST_TAGS] +# rollback_to_stable +# [END_TAGS] import fnmatch, os, shutil, threading, time from wtthread import checkpoint_thread, op_thread diff --git a/src/third_party/wiredtiger/test/suite/test_config02.py b/src/third_party/wiredtiger/test/suite/test_config02.py index d4a9ac1fb84..f731c9a6315 100755 --- a/src/third_party/wiredtiger/test/suite/test_config02.py +++ b/src/third_party/wiredtiger/test/suite/test_config02.py @@ -25,6 +25,11 @@ # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. +# +# [TEST_TAGS] +# connection_api:wiredtiger_open +# config_api +# [END_TAGS] import os import wiredtiger, wttest diff --git a/src/third_party/wiredtiger/test/suite/test_cursor06.py b/src/third_party/wiredtiger/test/suite/test_cursor06.py index 015cbed959d..54ec1519a86 100644 --- a/src/third_party/wiredtiger/test/suite/test_cursor06.py +++ b/src/third_party/wiredtiger/test/suite/test_cursor06.py @@ -25,6 +25,10 @@ # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. +# +# [TEST_TAGS] +# cursors:reconfigure +# [END_TAGS] import wiredtiger, wttest from wtdataset import SimpleDataSet, ComplexDataSet, ComplexLSMDataSet diff --git a/src/third_party/wiredtiger/test/suite/test_cursor17.py b/src/third_party/wiredtiger/test/suite/test_cursor17.py new file mode 100644 index 00000000000..f068279f8f1 --- /dev/null +++ b/src/third_party/wiredtiger/test/suite/test_cursor17.py @@ -0,0 +1,120 @@ +#!/usr/bin/env python +# +# Public Domain 2014-2020 MongoDB, Inc. +# Public Domain 2008-2014 WiredTiger, Inc. +# +# This is free and unencumbered software released into the public domain. +# +# Anyone is free to copy, modify, publish, use, compile, sell, or +# distribute this software, either in source code form or as a compiled +# binary, for any purpose, commercial or non-commercial, and by any +# means. +# +# In jurisdictions that recognize copyright laws, the author or authors +# of this software dedicate any and all copyright interest in the +# software to the public domain. We make this dedication for the benefit +# of the public at large and to the detriment of our heirs and +# successors. We intend this dedication to be an overt act of +# relinquishment in perpetuity of all present and future rights to this +# software under copyright law. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +# IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. + +import wiredtiger, wttest +from wtscenario import make_scenarios +from wiredtiger import stat, WT_NOTFOUND + +def timestamp_str(t): + return '%x' % t + +# test_cursor17.py +# Test the cursor traversal optimization for delete heavy workloads. This optimization enables +# cursor traversal mechanism to skip pages where all records on the page are deleted with a +# tombstone visible to the current transaction. +class test_cursor17(wttest.WiredTigerTestCase): + conn_config = 'cache_size=50MB,statistics=(all)' + session_config = 'isolation=snapshot' + + def get_stat(self, stat, uri): + stat_string = 'statistics:' + if (uri): + stat_string += uri + stat_cursor = self.session.open_cursor(stat_string) + val = stat_cursor[stat][2] + stat_cursor.close() + return val + + def test_cursor_skip_pages(self): + uri = 'table:test_cursor17' + create_params = 'key_format=i,value_format=S' + self.session.create(uri, create_params) + + value1 = 'a' * 500 + value2 = 'b' * 500 + total_keys = 40000 + + # Keep the oldest and the stable timestamp pinned. + self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(1)) + self.conn.set_timestamp('stable_timestamp=' + timestamp_str(2)) + cursor = self.session.open_cursor(uri) + + commit_timestamp = 3 + + # Insert predefined number of key-value pairs. + for key in range(total_keys): + self.session.begin_transaction() + cursor[key] = value1 + self.session.commit_transaction('commit_timestamp=' + timestamp_str(commit_timestamp)) + commit_timestamp += 1 + + # Delete everything on the table except for the first and the last KV pair. + for key in range(1, total_keys - 1): + self.session.begin_transaction() + cursor.set_key(key) + self.assertEqual(cursor.remove(),0) + self.session.commit_transaction('commit_timestamp=' + timestamp_str(commit_timestamp)) + commit_timestamp += 1 + + # Take a checkpoint to reconcile the pages. + self.session.checkpoint() + + self.session.begin_transaction('read_timestamp=' + timestamp_str(commit_timestamp)) + # Position the cursor on the first record. + cursor.set_key(0) + self.assertEqual(cursor.search(), 0) + # This should move the cursor to the last record. + self.assertEqual(cursor.next(), 0) + self.assertEqual(cursor.get_key(), total_keys - 1) + + # Check if we skipped any pages while moving the cursor. + # + # We calculate the number of pages we expect to skip based on the total number of leaf pages + # reported in the WT stats. We subtract 2 from the count of leaf pages in the table and test + # that we atleast skipped 80% of the expected number of pages. + + leaf_pages_in_table = self.get_stat(stat.dsrc.btree_row_leaf, uri) + expected_pages_skipped = ((leaf_pages_in_table - 2) * 8) // 10 + skipped_pages = self.get_stat(stat.conn.cursor_next_skip_page_count, None) + self.assertGreater(skipped_pages, expected_pages_skipped) + self.session.rollback_transaction() + + # Update a key in the middle of the table. + self.session.begin_transaction() + cursor[total_keys // 2] = value2 + self.session.commit_transaction('commit_timestamp=' + timestamp_str(commit_timestamp)) + commit_timestamp += 1 + + # Make sure we can reach a the record we updated in the middle of the table. + self.session.begin_transaction('read_timestamp=' + timestamp_str(commit_timestamp)) + # Position the cursor on the first record. + cursor.set_key(0) + self.assertEqual(cursor.search(), 0) + # This should move the cursor to the record we updated in the middle. + self.assertEqual(cursor.next(), 0) + self.assertEqual(cursor.get_key(), total_keys // 2) diff --git a/src/third_party/wiredtiger/test/suite/test_debug_mode05.py b/src/third_party/wiredtiger/test/suite/test_debug_mode05.py index 4fd2c27dd5d..f322dd98f7a 100644 --- a/src/third_party/wiredtiger/test/suite/test_debug_mode05.py +++ b/src/third_party/wiredtiger/test/suite/test_debug_mode05.py @@ -44,12 +44,11 @@ class test_debug_mode05(wttest.WiredTigerTestCase): def test_table_logging_rollback_to_stable(self): self.session.create(self.uri, 'key_format=i,value_format=u,log=(enabled=false)') - cursor = self.session.open_cursor(self.uri, None) - self.conn.set_timestamp('stable_timestamp=' + timestamp_str(100)) self.session.checkpoint() # Try doing a normal prepared txn and then rollback to stable. + cursor = self.session.open_cursor(self.uri, None) self.session.begin_transaction() for i in range(1, 50): cursor[i] = b'a' * 100 @@ -60,6 +59,7 @@ class test_debug_mode05(wttest.WiredTigerTestCase): self.session.timestamp_transaction( 'durable_timestamp=' + timestamp_str(250)) self.session.commit_transaction() + cursor.close() self.conn.rollback_to_stable() @@ -84,10 +84,12 @@ class test_debug_mode05(wttest.WiredTigerTestCase): self.conn.rollback_to_stable() self.session.begin_transaction() + cursor = self.session.open_cursor(self.uri, None) for i in range(1, 50): cursor[i] = b'b' * 100 self.session.commit_transaction( 'commit_timestamp=' + timestamp_str(450)) + cursor.close() self.conn.rollback_to_stable() diff --git a/src/third_party/wiredtiger/test/suite/test_debug_mode09.py b/src/third_party/wiredtiger/test/suite/test_debug_mode09.py index a44e25ca7ce..9c86d876276 100755 --- a/src/third_party/wiredtiger/test/suite/test_debug_mode09.py +++ b/src/third_party/wiredtiger/test/suite/test_debug_mode09.py @@ -36,7 +36,7 @@ import wttest # to do an update restore evict on a page, when the cache pressure requirements are not met. # This means setting eviction target low and cache size high. class test_debug_mode09(wttest.WiredTigerTestCase): - conn_config = 'cache_size=10MB,statistics=(all),eviction_target=10' + conn_config = 'cache_size=10MB,statistics=(all),eviction_target=10,debug_mode=(update_restore_evict=true)' uri = "table:test_debug_mode09" # Insert a bunch of data to trigger eviction @@ -57,17 +57,3 @@ class test_debug_mode09(wttest.WiredTigerTestCase): pages_update_restored = stat_cursor[stat.conn.cache_write_restore][2] stat_cursor.close() self.assertGreater(pages_update_restored, 0) - - # Restart the connection with update restore evict config and a clean table - self.close_conn() - self.conn_config += ",debug_mode=(update_restore_evict=true)" - self.reopen_conn(".", self.conn_config) - self.session.drop(self.uri) - self.session.create(self.uri, 'key_format=i,value_format=S') - - self.trigger_eviction(self.uri) - - stat_cursor = self.session.open_cursor('statistics:') - forced_pages_update_restore = stat_cursor[stat.conn.cache_write_restore][2] - stat_cursor.close() - self.assertGreater(forced_pages_update_restore, pages_update_restored) diff --git a/src/third_party/wiredtiger/test/suite/test_dictionary.py b/src/third_party/wiredtiger/test/suite/test_dictionary.py index b5baf170278..9465827741c 100644 --- a/src/third_party/wiredtiger/test/suite/test_dictionary.py +++ b/src/third_party/wiredtiger/test/suite/test_dictionary.py @@ -26,6 +26,10 @@ # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. # +# [TEST_TAGS] +# compression +# [END_TAGS] +# # test_dictionary.py # Smoke test dictionary compression. diff --git a/src/third_party/wiredtiger/test/suite/test_dump.py b/src/third_party/wiredtiger/test/suite/test_dump.py index 20f863dba8e..bc94d668f72 100644 --- a/src/third_party/wiredtiger/test/suite/test_dump.py +++ b/src/third_party/wiredtiger/test/suite/test_dump.py @@ -25,6 +25,10 @@ # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. +# +# [TEST_TAGS] +# wt_util +# [END_TAGS] import os, shutil import wiredtiger, wttest diff --git a/src/third_party/wiredtiger/test/suite/test_encrypt01.py b/src/third_party/wiredtiger/test/suite/test_encrypt01.py index 3aee3ecd90d..94acd8aa433 100644 --- a/src/third_party/wiredtiger/test/suite/test_encrypt01.py +++ b/src/third_party/wiredtiger/test/suite/test_encrypt01.py @@ -37,6 +37,16 @@ from wtscenario import make_scenarios # Test basic encryption class test_encrypt01(wttest.WiredTigerTestCase): + # To test the sodium encryptor, we use secretkey= rather than + # setting a keyid, because for a "real" (vs. test-only) encryptor, + # keyids require some kind of key server, and (a) setting one up + # for testing would be a nuisance and (b) currently the sodium + # encryptor doesn't support any anyway. + # + # It expects secretkey= to provide a hex-encoded 256-bit chacha20 key. + # This key will serve for testing purposes. + sodium_testkey = '0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef' + types = [ ('file', dict(uri='file:test_encrypt01')), ('table', dict(uri='table:test_encrypt01')), @@ -49,7 +59,9 @@ class test_encrypt01(wttest.WiredTigerTestCase): ('rotn', dict( sys_encrypt='rotn', sys_encrypt_args=',keyid=11', file_encrypt='rotn', file_encrypt_args=',keyid=13')), ('rotn-none', dict( sys_encrypt='rotn', sys_encrypt_args=',keyid=9', - file_encrypt='none', file_encrypt_args='')) + file_encrypt='none', file_encrypt_args='')), + ('sodium', dict( sys_encrypt='sodium', sys_encrypt_args=',secretkey=' + sodium_testkey, + file_encrypt='sodium', file_encrypt_args='')) ] compress = [ ('none', dict(log_compress=None, block_compress=None)), diff --git a/src/third_party/wiredtiger/test/suite/test_encrypt02.py b/src/third_party/wiredtiger/test/suite/test_encrypt02.py index ada16e2f172..c678a5a21e6 100644 --- a/src/third_party/wiredtiger/test/suite/test_encrypt02.py +++ b/src/third_party/wiredtiger/test/suite/test_encrypt02.py @@ -26,6 +26,10 @@ # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. # +# [TEST_TAGS] +# encryption +# [END_TAGS] +# # test_encrypt02.py # Encryption using passwords # @@ -37,6 +41,16 @@ from wtscenario import make_scenarios # Test basic encryption class test_encrypt02(wttest.WiredTigerTestCase, suite_subprocess): + # To test the sodium encryptor, we use secretkey= rather than + # setting a keyid, because for a "real" (vs. test-only) encryptor, + # keyids require some kind of key server, and (a) setting one up + # for testing would be a nuisance and (b) currently the sodium + # encryptor doesn't support any anyway. + # + # It expects secretkey= to provide a hex-encoded 256-bit chacha20 key. + # This key will serve for testing purposes. + sodium_testkey = '0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef' + uri = 'file:test_encrypt02' encrypt_type = [ ('noarg', dict( encrypt_args='name=rotn', secret_arg=None)), @@ -44,6 +58,8 @@ class test_encrypt02(wttest.WiredTigerTestCase, suite_subprocess): ('pass', dict( encrypt_args='name=rotn', secret_arg='ABC')), ('keyid-pass', dict( encrypt_args='name=rotn,keyid=11', secret_arg='ABC')), + ('sodium-pass', dict( encrypt_args='name=sodium', secret_arg=sodium_testkey)), + # The other combinations for sodium, which are rejected, are checked in encrypt08. ] scenarios = make_scenarios(encrypt_type) @@ -51,6 +67,7 @@ class test_encrypt02(wttest.WiredTigerTestCase, suite_subprocess): # Load the compression extension, skip the test if missing extlist.skip_if_missing = True extlist.extension('encryptors', 'rotn') + extlist.extension('encryptors', 'sodium') nrecords = 5000 bigvalue = "abcdefghij" * 1001 # len(bigvalue) = 10010 diff --git a/src/third_party/wiredtiger/test/suite/test_encrypt06.py b/src/third_party/wiredtiger/test/suite/test_encrypt06.py index f3b83cb23eb..be428f5a3cf 100755 --- a/src/third_party/wiredtiger/test/suite/test_encrypt06.py +++ b/src/third_party/wiredtiger/test/suite/test_encrypt06.py @@ -36,9 +36,22 @@ from wtscenario import make_scenarios # Test encryption, when on, does not leak any information class test_encrypt06(wttest.WiredTigerTestCase): + # To test the sodium encryptor, we use secretkey= rather than + # setting a keyid, because for a "real" (vs. test-only) encryptor, + # keyids require some kind of key server, and (a) setting one up + # for testing would be a nuisance and (b) currently the sodium + # encryptor doesn't support any anyway. + # + # Note that secretkey= is apparently not allowed with per-table + # encryption, so we don't test that. + # + # It expects secretkey= to provide a hex-encoded 256-bit chacha20 key. + # This key will serve for testing purposes. + sodium_testkey = '0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef' key11 = ',keyid=11,secretkey=XYZ' key13 = ',keyid=13' + sodiumkey = ',secretkey=' + sodium_testkey # Test with various combinations of tables with or without indices # and column groups, also with LSM. When 'match' is False, we @@ -85,6 +98,10 @@ class test_encrypt06(wttest.WiredTigerTestCase): sys_encrypt='rotn', sys_encrypt_args=key11, table0_encrypt='rotn', table0_encrypt_args=key13, table1_encrypt='none', table1_encrypt_args='')), + ('sodium-implied', dict( + sys_encrypt='sodium', sys_encrypt_args=sodiumkey, + table0_encrypt=None, table0_encrypt_args='', + table1_encrypt=None, table1_encrypt_args='')), ] scenarios = make_scenarios(encrypt, storagetype) nrecords = 1000 diff --git a/src/third_party/wiredtiger/test/suite/test_encrypt08.py b/src/third_party/wiredtiger/test/suite/test_encrypt08.py new file mode 100644 index 00000000000..e15dd851908 --- /dev/null +++ b/src/third_party/wiredtiger/test/suite/test_encrypt08.py @@ -0,0 +1,94 @@ +#!/usr/bin/env python +# +# Public Domain 2014-present MongoDB, Inc. +# Public Domain 2008-2014 WiredTiger, Inc. +# +# This is free and unencumbered software released into the public domain. +# +# Anyone is free to copy, modify, publish, use, compile, sell, or +# distribute this software, either in source code form or as a compiled +# binary, for any purpose, commercial or non-commercial, and by any +# means. +# +# In jurisdictions that recognize copyright laws, the author or authors +# of this software dedicate any and all copyright interest in the +# software to the public domain. We make this dedication for the benefit +# of the public at large and to the detriment of our heirs and +# successors. We intend this dedication to be an overt act of +# relinquishment in perpetuity of all present and future rights to this +# software under copyright law. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +# IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. +# +# test_encrypt08.py +# Test some error conditions with the libsodium encryption extension. +# + +import os, run, random +import wiredtiger, wttest +from wtscenario import make_scenarios + +# +# Test sodium encryption configuration. +# This exercises the error paths in the encryptor's customize method when +# used for system (not per-table) encryption. +# +class test_encrypt08(wttest.WiredTigerTestCase): + uri = 'file:test_encrypt08' + + # To test the sodium encryptor, we use secretkey= rather than + # setting a keyid, because for a "real" (vs. test-only) encryptor, + # keyids require some kind of key server, and (a) setting one up + # for testing would be a nuisance and (b) currently the sodium + # encryptor doesn't support any anyway. + # + # It expects secretkey= to provide a hex-encoded 256-bit chacha20 key. + # This key will serve for testing purposes. + sodium_testkey = '0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef' + + encrypt_type = [ + ('nokey', dict( sys_encrypt='', + msg='/no key given/')), + ('keyid', dict( sys_encrypt='keyid=123', + msg='/keyids not supported/')), + ('twokeys', dict( sys_encrypt='keyid=123,secretkey=' + sodium_testkey, + msg='/keys specified with both/')), + ('nothex', dict( sys_encrypt='secretkey=plop', + msg='/secret key not hex/')), + ('badsize', dict( sys_encrypt='secretkey=0123456789abcdef', + msg='/wrong secret key length/')), + ] + scenarios = make_scenarios(encrypt_type) + + def conn_extensions(self, extlist): + extlist.skip_if_missing = True + extlist.extension('encryptors', 'sodium') + + # Do not use conn_config to set the encryption, because that sets + # the encryption during open when we don't have control and can't + # catch exceptions. Instead we'll let the frameork open without + # encryption and then reopen ourselves. This seems to behave as + # desired (we get the intended errors from inside the encryptor) + # even though one might expect it to fail because it's reopening + # the database with different encryption. (If in the future it starts + # doing that, the workaround is to override setUpConnectionOpen. + # I'm not doing that now because it's quite a bit messier.) + + # (Re)open the database with bad encryption config. + def test_encrypt(self): + sysconfig = 'encryption=(name=sodium,{0}),'.format(self.sys_encrypt) + + with self.expectedStdoutPattern('Failed wiredtiger_open'): + self.assertRaisesWithMessage(wiredtiger.WiredTigerError, + lambda: + self.reopen_conn(config = sysconfig), + self.msg) + +if __name__ == '__main__': + wttest.run() diff --git a/src/third_party/wiredtiger/test/suite/test_encrypt09.py b/src/third_party/wiredtiger/test/suite/test_encrypt09.py new file mode 100644 index 00000000000..4232ab96a5f --- /dev/null +++ b/src/third_party/wiredtiger/test/suite/test_encrypt09.py @@ -0,0 +1,94 @@ +#!/usr/bin/env python +# +# Public Domain 2014-present MongoDB, Inc. +# Public Domain 2008-2014 WiredTiger, Inc. +# +# This is free and unencumbered software released into the public domain. +# +# Anyone is free to copy, modify, publish, use, compile, sell, or +# distribute this software, either in source code form or as a compiled +# binary, for any purpose, commercial or non-commercial, and by any +# means. +# +# In jurisdictions that recognize copyright laws, the author or authors +# of this software dedicate any and all copyright interest in the +# software to the public domain. We make this dedication for the benefit +# of the public at large and to the detriment of our heirs and +# successors. We intend this dedication to be an overt act of +# relinquishment in perpetuity of all present and future rights to this +# software under copyright law. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +# IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. +# +# test_encrypt09.py +# Test some error conditions with the libsodium encryption extension. +# + +import os, run, random +import wiredtiger, wttest +from wtscenario import make_scenarios + +# +# Test sodium encryption configuration. +# This exercises the error paths in the encryptor's customize method when +# used for per-table encryption. +# +class test_encrypt09(wttest.WiredTigerTestCase): + uri = 'file:test_encrypt09' + + # To test the sodium encryptor, we use secretkey= rather than + # setting a keyid, because for a "real" (vs. test-only) encryptor, + # keyids require some kind of key server, and (a) setting one up + # for testing would be a nuisance and (b) currently the sodium + # encryptor doesn't support any anyway. + # + # It expects secretkey= to provide a hex-encoded 256-bit chacha20 key. + # This key will serve for testing purposes. + sodium_testkey = '0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef' + + # The nokey case is not an error; if no key is given no separate encryptor is + # generated and no error occurs. + + # Note that the twokeys, nothex, and badsize cases do not (currently) get to + # the extension at all because (apparently) secretkey= is not allowed for + # per-table encryption. + + encrypt_type = [ + ('nokey', dict( file_encrypt='', + msg=None)), + ('keyid', dict( file_encrypt='keyid=123', + msg='/keyids not supported/')), + ('twokeys', dict( file_encrypt='keyid=123,secretkey=' + sodium_testkey, + msg='/unknown configuration key: .secretkey.:/')), + ('nothex', dict( file_encrypt='secretkey=plop', + msg='/unknown configuration key: .secretkey.:/')), + ('badsize', dict( file_encrypt='secretkey=0123456789abcdef', + msg='/unknown configuration key: .secretkey.:/')), + ] + scenarios = make_scenarios(encrypt_type) + + def conn_extensions(self, extlist): + extlist.skip_if_missing = True + extlist.extension('encryptors', 'sodium') + + def conn_config(self): + return 'encryption=(name=sodium,secretkey={0}),'.format(self.sodium_testkey) + + # Create a table with encryption values that are in error. + def test_encrypt(self): + params = 'key_format=S,value_format=S,encryption=(name=sodium,' + self.file_encrypt + ')' + + if self.msg is None: + self.session.create(self.uri, params) + else: + self.assertRaisesWithMessage(wiredtiger.WiredTigerError, lambda: + self.session.create(self.uri, params), self.msg) + +if __name__ == '__main__': + wttest.run() diff --git a/src/third_party/wiredtiger/test/suite/test_gc01.py b/src/third_party/wiredtiger/test/suite/test_gc01.py index 9ca16a9c961..950c1e4ce54 100755 --- a/src/third_party/wiredtiger/test/suite/test_gc01.py +++ b/src/third_party/wiredtiger/test/suite/test_gc01.py @@ -25,6 +25,10 @@ # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. +# +# [TEST_TAGS] +# checkpoint:garbage_collection +# [END_TAGS] import time from helper import copy_wiredtiger_home diff --git a/src/third_party/wiredtiger/test/suite/test_hs09.py b/src/third_party/wiredtiger/test/suite/test_hs09.py index a1808067a92..4217da2b02d 100644 --- a/src/third_party/wiredtiger/test/suite/test_hs09.py +++ b/src/third_party/wiredtiger/test/suite/test_hs09.py @@ -25,6 +25,10 @@ # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. +# +# [TEST_TAGS] +# history_store +# [END_TAGS] import wiredtiger, wttest from wiredtiger import stat diff --git a/src/third_party/wiredtiger/test/suite/test_hs15.py b/src/third_party/wiredtiger/test/suite/test_hs15.py index 5c14f064a4e..7d27d166a70 100644 --- a/src/third_party/wiredtiger/test/suite/test_hs15.py +++ b/src/third_party/wiredtiger/test/suite/test_hs15.py @@ -27,7 +27,7 @@ # OTHER DEALINGS IN THE SOFTWARE. # # [TEST_TAGS] -# caching_eviction:correctness:written_data +# history_store:eviction_checkpoint_interaction # [END_TAGS] # diff --git a/src/third_party/wiredtiger/test/suite/test_hs24.py b/src/third_party/wiredtiger/test/suite/test_hs24.py new file mode 100644 index 00000000000..4c4ee4a4b94 --- /dev/null +++ b/src/third_party/wiredtiger/test/suite/test_hs24.py @@ -0,0 +1,188 @@ +#!/usr/bin/env python +# +# Public Domain 2014-present MongoDB, Inc. +# Public Domain 2008-2014 WiredTiger, Inc. +# +# This is free and unencumbered software released into the public domain. +# +# Anyone is free to copy, modify, publish, use, compile, sell, or +# distribute this software, either in source code form or as a compiled +# binary, for any purpose, commercial or non-commercial, and by any +# means. +# +# In jurisdictions that recognize copyright laws, the author or authors +# of this software dedicate any and all copyright interest in the +# software to the public domain. We make this dedication for the benefit +# of the public at large and to the detriment of our heirs and +# successors. We intend this dedication to be an overt act of +# relinquishment in perpetuity of all present and future rights to this +# software under copyright law. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +# IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. + +import wttest, threading, wiredtiger +from helper import simulate_crash_restart + +def timestamp_str(t): + return '%x' % t + +# test_hs24.py +# Test that out of order timestamp fix racing with checkpointing the history store doesn't create inconsistent checkpoint. +class test_hs24(wttest.WiredTigerTestCase): + conn_config = 'cache_size=50MB,timing_stress_for_test=(history_store_checkpoint_delay)' + session_config = 'isolation=snapshot' + uri = 'table:test_hs24' + + value1 = 'a' * 500 + value2 = 'b' * 500 + value3 = 'c' * 500 + value4 = 'd' * 500 + def test_zero_ts(self): + self.session.create(self.uri, 'key_format=i,value_format=S') + self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(1)) + cursor = self.session.open_cursor(self.uri) + for i in range(0, 2000): + self.session.begin_transaction() + cursor[i] = self.value1 + self.session.commit_transaction('commit_timestamp=' + timestamp_str(4)) + self.session.begin_transaction() + cursor[i] = self.value2 + self.session.commit_transaction('commit_timestamp=' + timestamp_str(5)) + cursor.close() + self.conn.set_timestamp('stable_timestamp=' + timestamp_str(5)) + thread = threading.Thread(target=self.zero_ts_deletes) + thread.start() + self.session.checkpoint() + thread.join() + simulate_crash_restart(self, '.', "RESTART") + cursor = self.session.open_cursor(self.uri) + session2 = self.conn.open_session(None) + cursor2 = session2.open_cursor(self.uri) + self.session.begin_transaction('read_timestamp=' + timestamp_str(5)) + session2.begin_transaction('read_timestamp=' + timestamp_str(4)) + # Check the data store and the history store content is consistent. + # If we have a value in the data store, we should see the older + # version in the history store as well. + newer_data_visible = False + for i in range(0, 2000): + cursor.set_key(i) + cursor2.set_key(i) + ret = cursor.search() + ret2 = cursor2.search() + if not newer_data_visible: + newer_data_visible = ret != wiredtiger.WT_NOTFOUND + if newer_data_visible: + self.assertEquals(cursor.get_value(), self.value2) + self.assertEquals(cursor2.get_value(), self.value1) + else: + self.assertEquals(ret2, wiredtiger.WT_NOTFOUND) + session2.rollback_transaction() + self.session.rollback_transaction() + + def zero_ts_deletes(self): + session = self.setUpSessionOpen(self.conn) + cursor = session.open_cursor(self.uri) + for i in range(0, 2000): + session.begin_transaction() + cursor.set_key(i) + cursor.remove() + session.commit_transaction() + cursor.close() + session.close() + + def test_zero_commit(self): + self.session.create(self.uri, 'key_format=i,value_format=S') + self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(1)) + cursor = self.session.open_cursor(self.uri) + for i in range(0, 2000): + self.session.begin_transaction() + cursor[i] = self.value1 + self.session.commit_transaction('commit_timestamp=' + timestamp_str(4)) + self.session.begin_transaction() + cursor[i] = self.value2 + self.session.commit_transaction('commit_timestamp=' + timestamp_str(5)) + cursor.close() + self.conn.set_timestamp('stable_timestamp=' + timestamp_str(4)) + thread = threading.Thread(target=self.zero_ts_commits) + thread.start() + self.session.checkpoint() + thread.join() + simulate_crash_restart(self, '.', "RESTART") + cursor = self.session.open_cursor(self.uri) + self.session.begin_transaction('read_timestamp=' + timestamp_str(4)) + # Check we can only see the version committed by the zero timestamp + # commit thread before the checkpoint starts or value1. + newer_data_visible = False + for i in range(0, 2000): + value = cursor[i] + if not newer_data_visible: + newer_data_visible = value != self.value3 + if newer_data_visible: + self.assertEquals(value, self.value1) + else: + self.assertEquals(value, self.value3) + self.session.rollback_transaction() + + def zero_ts_commits(self): + session = self.setUpSessionOpen(self.conn) + cursor = session.open_cursor(self.uri) + for i in range(0, 2000): + session.begin_transaction() + cursor[i] = self.value3 + session.commit_transaction() + cursor.close() + session.close() + + def test_out_of_order_ts(self): + self.session.create(self.uri, 'key_format=i,value_format=S') + self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(1)) + cursor = self.session.open_cursor(self.uri) + for i in range(0, 2000): + self.session.begin_transaction() + cursor[i] = self.value1 + self.session.commit_transaction('commit_timestamp=' + timestamp_str(4)) + self.session.begin_transaction() + cursor[i] = self.value2 + self.session.commit_transaction('commit_timestamp=' + timestamp_str(5)) + self.conn.set_timestamp('stable_timestamp=' + timestamp_str(4)) + for i in range(0, 2000): + self.session.begin_transaction() + cursor[i] = self.value3 + self.session.commit_transaction('commit_timestamp=' + timestamp_str(6)) + cursor.close() + thread = threading.Thread(target=self.out_of_order_ts_commits) + thread.start() + self.session.checkpoint() + thread.join() + simulate_crash_restart(self, '.', "RESTART") + cursor = self.session.open_cursor(self.uri) + self.session.begin_transaction('read_timestamp=' + timestamp_str(4)) + # Check we can only see the version at timestamp 4, it's either + # committed by the out of order timestamp commit thread before the + # checkpoint starts or value1. + newer_data_visible = False + for i in range(0, 2000): + value = cursor[i] + if not newer_data_visible: + newer_data_visible = value != self.value4 + if newer_data_visible: + self.assertEquals(value, self.value1) + else: + self.assertEquals(value, self.value4) + self.session.rollback_transaction() + + def out_of_order_ts_commits(self): + session = self.setUpSessionOpen(self.conn) + cursor = session.open_cursor(self.uri) + for i in range(0, 2000): + session.begin_transaction() + cursor[i] = self.value4 + session.commit_transaction('commit_timestamp=' + timestamp_str(4)) + cursor.close() + session.close() diff --git a/src/third_party/wiredtiger/test/suite/test_hs25.py b/src/third_party/wiredtiger/test/suite/test_hs25.py new file mode 100644 index 00000000000..e353ce076fe --- /dev/null +++ b/src/third_party/wiredtiger/test/suite/test_hs25.py @@ -0,0 +1,73 @@ +#!/usr/bin/env python +# +# Public Domain 2014-present MongoDB, Inc. +# Public Domain 2008-2014 WiredTiger, Inc. +# +# This is free and unencumbered software released into the public domain. +# +# Anyone is free to copy, modify, publish, use, compile, sell, or +# distribute this software, either in source code form or as a compiled +# binary, for any purpose, commercial or non-commercial, and by any +# means. +# +# In jurisdictions that recognize copyright laws, the author or authors +# of this software dedicate any and all copyright interest in the +# software to the public domain. We make this dedication for the benefit +# of the public at large and to the detriment of our heirs and +# successors. We intend this dedication to be an overt act of +# relinquishment in perpetuity of all present and future rights to this +# software under copyright law. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +# IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. + +import wttest + +def timestamp_str(t): + return '%x' % t + +# test_hs25.py +# Ensure updates structure is correct when processing each key. +class test_hs25(wttest.WiredTigerTestCase): + conn_config = 'cache_size=50MB' + session_config = 'isolation=snapshot' + uri = 'table:test_hs25' + + def test_insert_updates_hs(self): + self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(1)) + self.conn.set_timestamp('stable_timestamp=' + timestamp_str(1)) + self.session.create(self.uri, 'key_format=i,value_format=S') + s = self.conn.open_session() + + # Update the first key. + cursor1 = self.session.open_cursor(self.uri) + self.session.begin_transaction() + cursor1[1] = 'a' + self.session.commit_transaction('commit_timestamp=' + timestamp_str(2)) + + # Update the second key. + self.session.begin_transaction() + cursor1[2] = 'a' + self.session.commit_transaction('commit_timestamp=' + timestamp_str(2)) + self.session.begin_transaction() + cursor1[2] = 'b' + self.session.commit_transaction('commit_timestamp=' + timestamp_str(3)) + + # Prepared update on the first key. + self.session.begin_transaction() + cursor1[1] = 'b' + cursor1[1] = 'c' + self.session.prepare_transaction('prepare_timestamp=' + timestamp_str(4)) + + # Run eviction cursor. + s.begin_transaction('ignore_prepare=true') + evict_cursor = s.open_cursor(self.uri, None, 'debug=(release_evict)') + self.assertEqual(evict_cursor[1], 'a') + self.assertEqual(evict_cursor[2], 'b') + s.rollback_transaction() + self.session.rollback_transaction() diff --git a/src/third_party/wiredtiger/test/suite/test_huffman02.py b/src/third_party/wiredtiger/test/suite/test_huffman02.py index 30da6804778..91bbbbc37bd 100644 --- a/src/third_party/wiredtiger/test/suite/test_huffman02.py +++ b/src/third_party/wiredtiger/test/suite/test_huffman02.py @@ -25,6 +25,10 @@ # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. +# +# [TEST_TAGS] +# huffman_encoding +# [END_TAGS] import os from suite_subprocess import suite_subprocess diff --git a/src/third_party/wiredtiger/test/suite/test_import06.py b/src/third_party/wiredtiger/test/suite/test_import06.py index fae5b3cfd35..083fc860a67 100644 --- a/src/third_party/wiredtiger/test/suite/test_import06.py +++ b/src/third_party/wiredtiger/test/suite/test_import06.py @@ -48,6 +48,16 @@ class test_import06(test_import_base): create_config = 'allocation_size={},key_format=u,log=(enabled=true),value_format=u,' \ 'block_compressor={},encryption=(name={})' + # To test the sodium encryptor, we use secretkey= rather than + # setting a keyid, because for a "real" (vs. test-only) encryptor, + # keyids require some kind of key server, and (a) setting one up + # for testing would be a nuisance and (b) currently the sodium + # encryptor doesn't support any anyway. + # + # It expects secretkey= to provide a hex-encoded 256-bit chacha20 key. + # This key will serve for testing purposes. + sodium_testkey = '0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef' + allocsizes = [ ('512', dict(allocsize='512')), ('1024', dict(allocsize='1024')), @@ -63,9 +73,10 @@ class test_import06(test_import_base): ('zstd', dict(compressor='zstd')), ] encryptors = [ - ('none', dict(encryptor='none')), - ('nop', dict(encryptor='none')), - ('rotn', dict(encryptor='rotn')), + ('none', dict(encryptor='none', encryptor_args='')), + ('nop', dict(encryptor='none', encryptor_args='')), + ('rotn', dict(encryptor='rotn', encryptor_args='')), + ('sodium', dict(encryptor='sodium', encryptor_args=',secretkey=' + sodium_testkey)), ] scenarios = make_scenarios(allocsizes, compressors, encryptors) @@ -77,7 +88,7 @@ class test_import06(test_import_base): def conn_config(self): return 'cache_size=50MB,log=(enabled),statistics=(all),encryption=(name={})'.format( - self.encryptor) + self.encryptor + self.encryptor_args) def test_import_repair(self): self.session.create(self.uri, diff --git a/src/third_party/wiredtiger/test/suite/test_import09.py b/src/third_party/wiredtiger/test/suite/test_import09.py index 7caa89055c0..a41aff32571 100644 --- a/src/third_party/wiredtiger/test/suite/test_import09.py +++ b/src/third_party/wiredtiger/test/suite/test_import09.py @@ -38,6 +38,16 @@ class test_import09(test_import_base): ntables = 1 session_config = 'isolation=snapshot' + # To test the sodium encryptor, we use secretkey= rather than + # setting a keyid, because for a "real" (vs. test-only) encryptor, + # keyids require some kind of key server, and (a) setting one up + # for testing would be a nuisance and (b) currently the sodium + # encryptor doesn't support any anyway. + # + # It expects secretkey= to provide a hex-encoded 256-bit chacha20 key. + # This key will serve for testing purposes. + sodium_testkey = '0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef' + allocsizes = [ ('512', dict(allocsize='512')), ('1024', dict(allocsize='1024')), @@ -53,9 +63,10 @@ class test_import09(test_import_base): ('zstd', dict(compressor='zstd')), ] encryptors = [ - ('none', dict(encryptor='none')), - ('nop', dict(encryptor='nop')), - ('rotn', dict(encryptor='rotn')), + ('none', dict(encryptor='none', encryptor_args='')), + ('nop', dict(encryptor='nop', encryptor_args='')), + ('rotn', dict(encryptor='rotn', encryptor_args='')), + ('sodium', dict(encryptor='sodium', encryptor_args=',secretkey=' + sodium_testkey)), ] tables = [ ('simple_table', dict( @@ -90,7 +101,7 @@ class test_import09(test_import_base): def conn_config(self): return 'cache_size=50MB,log=(enabled),statistics=(all),encryption=(name={})'.format( - self.encryptor) + self.encryptor + self.encryptor_args) def test_import_table_repair(self): # Add some tables & data and checkpoint. diff --git a/src/third_party/wiredtiger/test/suite/test_index02.py b/src/third_party/wiredtiger/test/suite/test_index02.py index de9fd2998c5..17e9c2b0f83 100755 --- a/src/third_party/wiredtiger/test/suite/test_index02.py +++ b/src/third_party/wiredtiger/test/suite/test_index02.py @@ -25,6 +25,10 @@ # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. +# +# [TEST_TAGS] +# indexes:search_near +# [END_TAGS] import wiredtiger, wttest from wtscenario import make_scenarios diff --git a/src/third_party/wiredtiger/test/suite/test_prepare08.py b/src/third_party/wiredtiger/test/suite/test_prepare08.py index c93a8984ab6..76a831b0a05 100644 --- a/src/third_party/wiredtiger/test/suite/test_prepare08.py +++ b/src/third_party/wiredtiger/test/suite/test_prepare08.py @@ -60,8 +60,11 @@ class test_prepare08(wttest.WiredTigerTestCase): self.session.commit_transaction('commit_timestamp=' + timestamp_str(ts)) cursor.close() - def check(self, ds, uri, nrows, value, ts): - cursor = self.session.open_cursor(uri) + def check(self, ds, uri, nrows, value, ts, release_evict): + if release_evict: + cursor = self.session.open_cursor(uri, None, "debug=(release_evict)") + else: + cursor = self.session.open_cursor(uri) self.session.begin_transaction('ignore_prepare=true,read_timestamp=' + timestamp_str(ts)) for i in range(1, nrows): cursor.set_key(ds.key(i)) @@ -70,6 +73,7 @@ class test_prepare08(wttest.WiredTigerTestCase): else: self.assertEquals(cursor.search(), 0) self.assertEquals(cursor.get_value(),value) + cursor.reset() self.session.commit_transaction() cursor.close() @@ -104,11 +108,11 @@ class test_prepare08(wttest.WiredTigerTestCase): self.updates(ds_2, uri_2, nrows, value_b, 30) # Verify the updates - self.check(ds_1, uri_1, nrows, value_a, 20) - self.check(ds_1, uri_1, nrows, value_b, 30) + self.check(ds_1, uri_1, nrows, value_a, 20, False) + self.check(ds_1, uri_1, nrows, value_b, 30, False) - self.check(ds_2, uri_2, nrows, value_a, 20) - self.check(ds_2, uri_2, nrows, value_b, 30) + self.check(ds_2, uri_2, nrows, value_a, 20, False) + self.check(ds_2, uri_2, nrows, value_b, 30, False) # Checkpoint self.session.checkpoint() @@ -127,14 +131,14 @@ class test_prepare08(wttest.WiredTigerTestCase): self.updates(ds_2, uri_2, nrows, value_d, 50) self.updates(ds_2, uri_2, nrows, value_e, 60) - self.check(ds_1, uri_1, nrows, value_a, 20) - self.check(ds_1, uri_1, nrows, value_b, 50) + self.check(ds_1, uri_1, nrows, value_a, 20, True) + self.check(ds_1, uri_1, nrows, value_b, 50, True) #rollback the prepared session session_p.rollback_transaction() - self.check(ds_1, uri_1, nrows, value_a, 20) - self.check(ds_1, uri_1, nrows, value_b, 50) + self.check(ds_1, uri_1, nrows, value_a, 20, False) + self.check(ds_1, uri_1, nrows, value_b, 50, False) # close sessions. cursor_p.close() @@ -173,11 +177,11 @@ class test_prepare08(wttest.WiredTigerTestCase): self.updates(ds_2, uri_2, nrows, value_b, 30) # Verify the updates - self.check(ds_1, uri_1, nrows, value_a, 20) - self.check(ds_1, uri_1, nrows, value_b, 30) + self.check(ds_1, uri_1, nrows, value_a, 20, False) + self.check(ds_1, uri_1, nrows, value_b, 30, False) - self.check(ds_2, uri_2, nrows, value_a, 20) - self.check(ds_2, uri_2, nrows, value_b, 30) + self.check(ds_2, uri_2, nrows, value_a, 20, False) + self.check(ds_2, uri_2, nrows, value_b, 30, False) # Checkpoint self.session.checkpoint() @@ -199,16 +203,16 @@ class test_prepare08(wttest.WiredTigerTestCase): self.updates(ds_2, uri_2, nrows, value_d, 50) self.updates(ds_2, uri_2, nrows, value_e, 60) - self.check(ds_1, uri_1, nrows, value_a, 20) - self.check(ds_1, uri_1, nrows, value_b, 30) - self.check(ds_1, uri_1, nrows, value_b, 50) + self.check(ds_1, uri_1, nrows, value_a, 20, True) + self.check(ds_1, uri_1, nrows, value_b, 30, True) + self.check(ds_1, uri_1, nrows, value_b, 50, True) # Commit the prepared session session_p.commit_transaction('commit_timestamp=' + timestamp_str(50) + ',durable_timestamp=' + timestamp_str(60)) - self.check(ds_1, uri_1, nrows, value_a, 20) - self.check(ds_1, uri_1, nrows, value_b, 30) - self.check(ds_1, uri_1, 0, None, 50) + self.check(ds_1, uri_1, nrows, value_a, 20, False) + self.check(ds_1, uri_1, nrows, value_b, 30, False) + self.check(ds_1, uri_1, 0, None, 50, False) # close sessions. cursor_p.close() @@ -269,16 +273,16 @@ class test_prepare08(wttest.WiredTigerTestCase): self.updates(ds_2, uri_2, nrows, value_d, 50) self.updates(ds_2, uri_2, nrows, value_e, 60) - self.check(ds_1, uri_1, nrows, value_a, 20) - self.check(ds_1, uri_1, 0, None, 30) - self.check(ds_1, uri_1, 0, None, 50) + self.check(ds_1, uri_1, nrows, value_a, 20, True) + self.check(ds_1, uri_1, 0, None, 30, True) + self.check(ds_1, uri_1, 0, None, 50, True) # Commit the prepared session session_p.commit_transaction('commit_timestamp=' + timestamp_str(50) + ',durable_timestamp=' + timestamp_str(60)) - self.check(ds_1, uri_1, nrows, value_a, 20) - self.check(ds_1, uri_1, 0, None, 30) - self.check(ds_1, uri_1, 0, None, 50) + self.check(ds_1, uri_1, nrows, value_a, 20, False) + self.check(ds_1, uri_1, 0, None, 30, False) + self.check(ds_1, uri_1, 0, None, 50, False) # close sessions. cursor_p.close() diff --git a/src/third_party/wiredtiger/test/suite/test_prepare09.py b/src/third_party/wiredtiger/test/suite/test_prepare09.py index f8b3c0ebd02..3c0aa546d08 100644 --- a/src/third_party/wiredtiger/test/suite/test_prepare09.py +++ b/src/third_party/wiredtiger/test/suite/test_prepare09.py @@ -25,6 +25,10 @@ # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. +# +# [TEST_TAGS] +# prepare +# [END_TAGS] import wiredtiger, wttest def timestamp_str(t): diff --git a/src/third_party/wiredtiger/test/suite/test_prepare12.py b/src/third_party/wiredtiger/test/suite/test_prepare12.py index 2663d12ad70..55d6c69c2d7 100644 --- a/src/third_party/wiredtiger/test/suite/test_prepare12.py +++ b/src/third_party/wiredtiger/test/suite/test_prepare12.py @@ -25,6 +25,10 @@ # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. +# +# [TEST_TAGS] +# eviction:prepare +# [END_TAGS] import wiredtiger, wttest def timestamp_str(t): diff --git a/src/third_party/wiredtiger/test/suite/test_prepare13.py b/src/third_party/wiredtiger/test/suite/test_prepare13.py index 152cc9219f3..8542a182328 100644 --- a/src/third_party/wiredtiger/test/suite/test_prepare13.py +++ b/src/third_party/wiredtiger/test/suite/test_prepare13.py @@ -26,6 +26,10 @@ # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. # +# [TEST_TAGS] +# truncate:prepare +# [END_TAGS] +# # test_prepare13.py # Fast-truncate fails when a page contains prepared updates. import wiredtiger, wttest diff --git a/src/third_party/wiredtiger/test/suite/test_prepare_cursor01.py b/src/third_party/wiredtiger/test/suite/test_prepare_cursor01.py index 6c29e00e2da..aa2e45bec32 100644 --- a/src/third_party/wiredtiger/test/suite/test_prepare_cursor01.py +++ b/src/third_party/wiredtiger/test/suite/test_prepare_cursor01.py @@ -25,6 +25,10 @@ # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. +# +# [TEST_TAGS] +# cursors:prepare +# [END_TAGS] import wiredtiger, wttest from wtdataset import SimpleDataSet, SimpleIndexDataSet diff --git a/src/third_party/wiredtiger/test/suite/test_prepare_hs02.py b/src/third_party/wiredtiger/test/suite/test_prepare_hs02.py index fd00db26341..5ba68e6a614 100644 --- a/src/third_party/wiredtiger/test/suite/test_prepare_hs02.py +++ b/src/third_party/wiredtiger/test/suite/test_prepare_hs02.py @@ -133,7 +133,7 @@ class test_prepare_hs02(wttest.WiredTigerTestCase, suite_subprocess): c[1] = 1 c[2] = 1 c[3] = 1 - self.session.commit_transaction('commit_timestamp=' + timestamp_str(301)) + self.session.commit_transaction('commit_timestamp=' + timestamp_str(302)) # Trigger a checkpoint, which could trigger reconciliation self.conn.set_timestamp('stable_timestamp=' + timestamp_str(350)) diff --git a/src/third_party/wiredtiger/test/suite/test_prepare_hs03.py b/src/third_party/wiredtiger/test/suite/test_prepare_hs03.py index 356ef7554f9..502a7ba9c8e 100644 --- a/src/third_party/wiredtiger/test/suite/test_prepare_hs03.py +++ b/src/third_party/wiredtiger/test/suite/test_prepare_hs03.py @@ -25,6 +25,11 @@ # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. +# +# [TEST_TAGS] +# salvage:prepare +# verify:prepare +# [END_TAGS] from helper import copy_wiredtiger_home import wiredtiger, wttest @@ -60,10 +65,21 @@ class test_prepare_hs03(wttest.WiredTigerTestCase): tablepointer.write('Bad!' * 1024) def corrupt_salvage_verify(self): + # An exclusive handle operation can fail if there is dirty data in the cache, closing the + # open handles before acquiring an exclusive handle will return EBUSY. A checkpoint should + # clear the dirty data, but eviction can re-dirty the cache between the checkpoint and the + # open attempt, we have to loop. + self.session.checkpoint() if self.corrupt == True: self.corrupt_table() - self.session.salvage(self.uri, "force") - self.session.verify(self.uri, None) + while True: + if not self.raisesBusy(lambda: self.session.salvage(self.uri, "force")): + break + self.session.checkpoint() + while True: + if not self.raisesBusy(lambda: self.session.verify(self.uri, None)): + break + self.session.checkpoint() def get_stat(self, stat): stat_cursor = self.session.open_cursor('statistics:') @@ -72,8 +88,6 @@ class test_prepare_hs03(wttest.WiredTigerTestCase): return val def prepare_updates(self, ds, nrows, nsessions, nkeys): - # Insert some records with commit timestamp, corrupt file and call salvage, verify before checkpoint. - # Commit some updates to get eviction and history store fired up commit_value = b"bbbbb" * 100 cursor = self.session.open_cursor(self.uri) @@ -85,11 +99,12 @@ class test_prepare_hs03(wttest.WiredTigerTestCase): self.session.commit_transaction('commit_timestamp=' + timestamp_str(1)) cursor.close() - # Corrupt the table, Call salvage to recover data from the corrupted table and call verify - self.corrupt_salvage_verify() + # Set the stable/oldest timstamps. + self.conn.set_timestamp('stable_timestamp=' + timestamp_str(1)) + self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(1)) - # Call checkpoint - self.session.checkpoint() + # Corrupt the table, call salvage to recover data from the corrupted table and call verify + self.corrupt_salvage_verify() hs_writes_start = self.get_stat(stat.conn.cache_write_hs) @@ -129,19 +144,13 @@ class test_prepare_hs03(wttest.WiredTigerTestCase): self.assertNotEquals(cursor.get_value(), prepare_value) cursor.close() - # Close all cursors and sessions, this will cause prepared updates to be - # rollback-ed + # Close all sessions (and cursors), this will cause prepared updates to be rolled back. for j in range (0, nsessions): - cursors[j].close() sessions[j].close() - # Corrupt the table, Call salvage to recover data from the corrupted table and call verify - self.corrupt_salvage_verify() - self.session.commit_transaction() - self.session.checkpoint() - # Corrupt the table, Call salvage to recover data from the corrupted table and call verify + # Corrupt the table, call salvage to recover data from the corrupted table and call verify self.corrupt_salvage_verify() # Finally, search for the keys inserted with commit timestamp @@ -180,8 +189,8 @@ class test_prepare_hs03(wttest.WiredTigerTestCase): cursor.close() self.session.commit_transaction() - # After simulating a crash, corrupt the table, call salvage to recover data from the corrupted table - # and call verify + # After simulating a crash, corrupt the table, call salvage to recover data from the + # corrupted table and call verify self.corrupt_salvage_verify() def test_prepare_hs(self): diff --git a/src/third_party/wiredtiger/test/suite/test_reconfig01.py b/src/third_party/wiredtiger/test/suite/test_reconfig01.py index 9a683252b7a..75a1c0bda9b 100644 --- a/src/third_party/wiredtiger/test/suite/test_reconfig01.py +++ b/src/third_party/wiredtiger/test/suite/test_reconfig01.py @@ -25,6 +25,10 @@ # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. +# +# [TEST_TAGS] +# connection_api:reconfigure +# [END_TAGS] import time import wiredtiger, wttest diff --git a/src/third_party/wiredtiger/test/suite/test_reconfig02.py b/src/third_party/wiredtiger/test/suite/test_reconfig02.py index bae9303e5b2..6eaeab95060 100644 --- a/src/third_party/wiredtiger/test/suite/test_reconfig02.py +++ b/src/third_party/wiredtiger/test/suite/test_reconfig02.py @@ -25,6 +25,10 @@ # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. +# +# [TEST_TAGS] +# connection_api:reconfigure +# [END_TAGS] import fnmatch, os, time import wiredtiger, wttest diff --git a/src/third_party/wiredtiger/test/suite/test_reconfig04.py b/src/third_party/wiredtiger/test/suite/test_reconfig04.py index 8119de46265..82dca85725c 100644 --- a/src/third_party/wiredtiger/test/suite/test_reconfig04.py +++ b/src/third_party/wiredtiger/test/suite/test_reconfig04.py @@ -25,6 +25,10 @@ # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. +# +# [TEST_TAGS] +# session_api:reconfigure +# [END_TAGS] import wiredtiger, wttest diff --git a/src/third_party/wiredtiger/test/suite/test_rollback_to_stable01.py b/src/third_party/wiredtiger/test/suite/test_rollback_to_stable01.py index da5b6d1ca91..7613e169fb1 100755 --- a/src/third_party/wiredtiger/test/suite/test_rollback_to_stable01.py +++ b/src/third_party/wiredtiger/test/suite/test_rollback_to_stable01.py @@ -30,8 +30,9 @@ import time from helper import copy_wiredtiger_home import wiredtiger, wttest from wtdataset import SimpleDataSet -from wiredtiger import stat +from wiredtiger import stat, wiredtiger_strerror, WiredTigerError, WT_ROLLBACK from wtscenario import make_scenarios +from time import sleep def timestamp_str(t): return '%x' % t @@ -39,53 +40,67 @@ def timestamp_str(t): # test_rollback_to_stable01.py # Shared base class used by rollback to stable tests. class test_rollback_to_stable_base(wttest.WiredTigerTestCase): + def retry_rollback(self, name, txn_session, code): + retry_limit = 100 + retries = 0 + completed = False + saved_exception = None + while not completed and retries < retry_limit: + if retries != 0: + self.pr("Retrying operation for " + name) + if txn_session: + txn_session.rollback_transaction() + sleep(0.1) + if txn_session: + txn_session.begin_transaction('isolation=snapshot') + self.pr("Began new transaction for " + name) + try: + code() + completed = True + except WiredTigerError as e: + rollback_str = wiredtiger_strerror(WT_ROLLBACK) + if rollback_str not in str(e): + raise(e) + retries += 1 + saved_exception = e + if not completed and saved_exception: + raise(saved_exception) + def large_updates(self, uri, value, ds, nrows, prepare, commit_ts): # Update a large number of records. session = self.session - cursor = session.open_cursor(uri) - for i in range(1, nrows + 1): - session.begin_transaction() - cursor[ds.key(i)] = value - if commit_ts == 0: - session.commit_transaction() - elif prepare: - session.prepare_transaction('prepare_timestamp=' + timestamp_str(commit_ts-1)) - session.timestamp_transaction('commit_timestamp=' + timestamp_str(commit_ts)) - session.timestamp_transaction('durable_timestamp=' + timestamp_str(commit_ts+1)) - session.commit_transaction() - else: - session.commit_transaction('commit_timestamp=' + timestamp_str(commit_ts)) - cursor.close() + try: + cursor = session.open_cursor(uri) + for i in range(1, nrows + 1): + session.begin_transaction() + cursor[ds.key(i)] = value + if commit_ts == 0: + session.commit_transaction() + elif prepare: + session.prepare_transaction('prepare_timestamp=' + timestamp_str(commit_ts-1)) + session.timestamp_transaction('commit_timestamp=' + timestamp_str(commit_ts)) + session.timestamp_transaction('durable_timestamp=' + timestamp_str(commit_ts+1)) + session.commit_transaction() + else: + session.commit_transaction('commit_timestamp=' + timestamp_str(commit_ts)) + cursor.close() + except WiredTigerError as e: + rollback_str = wiredtiger_strerror(WT_ROLLBACK) + if rollback_str in str(e): + session.rollback_transaction() + raise(e) def large_modifies(self, uri, value, ds, location, nbytes, nrows, prepare, commit_ts): # Load a slight modification. session = self.session - cursor = session.open_cursor(uri) - session.begin_transaction() - for i in range(1, nrows + 1): - cursor.set_key(i) - mods = [wiredtiger.Modify(value, location, nbytes)] - self.assertEqual(cursor.modify(mods), 0) - - if commit_ts == 0: - session.commit_transaction() - elif prepare: - session.prepare_transaction('prepare_timestamp=' + timestamp_str(commit_ts-1)) - session.timestamp_transaction('commit_timestamp=' + timestamp_str(commit_ts)) - session.timestamp_transaction('durable_timestamp=' + timestamp_str(commit_ts+1)) - session.commit_transaction() - else: - session.commit_transaction('commit_timestamp=' + timestamp_str(commit_ts)) - cursor.close() - - def large_removes(self, uri, ds, nrows, prepare, commit_ts): - # Remove a large number of records. - session = self.session - cursor = session.open_cursor(uri) - for i in range(1, nrows + 1): + try: + cursor = session.open_cursor(uri) session.begin_transaction() - cursor.set_key(i) - cursor.remove() + for i in range(1, nrows + 1): + cursor.set_key(i) + mods = [wiredtiger.Modify(value, location, nbytes)] + self.assertEqual(cursor.modify(mods), 0) + if commit_ts == 0: session.commit_transaction() elif prepare: @@ -95,7 +110,37 @@ class test_rollback_to_stable_base(wttest.WiredTigerTestCase): session.commit_transaction() else: session.commit_transaction('commit_timestamp=' + timestamp_str(commit_ts)) - cursor.close() + cursor.close() + except WiredTigerError as e: + rollback_str = wiredtiger_strerror(WT_ROLLBACK) + if rollback_str in str(e): + session.rollback_transaction() + raise(e) + + def large_removes(self, uri, ds, nrows, prepare, commit_ts): + # Remove a large number of records. + session = self.session + try: + cursor = session.open_cursor(uri) + for i in range(1, nrows + 1): + session.begin_transaction() + cursor.set_key(i) + cursor.remove() + if commit_ts == 0: + session.commit_transaction() + elif prepare: + session.prepare_transaction('prepare_timestamp=' + timestamp_str(commit_ts-1)) + session.timestamp_transaction('commit_timestamp=' + timestamp_str(commit_ts)) + session.timestamp_transaction('durable_timestamp=' + timestamp_str(commit_ts+1)) + session.commit_transaction() + else: + session.commit_transaction('commit_timestamp=' + timestamp_str(commit_ts)) + cursor.close() + except WiredTigerError as e: + rollback_str = wiredtiger_strerror(WT_ROLLBACK) + if rollback_str in str(e): + session.rollback_transaction() + raise(e) def check(self, check_value, uri, nrows, read_ts): session = self.session diff --git a/src/third_party/wiredtiger/test/suite/test_rollback_to_stable10.py b/src/third_party/wiredtiger/test/suite/test_rollback_to_stable10.py index 8aefee48b16..cfd0c2f3c46 100755 --- a/src/third_party/wiredtiger/test/suite/test_rollback_to_stable10.py +++ b/src/third_party/wiredtiger/test/suite/test_rollback_to_stable10.py @@ -29,41 +29,14 @@ import fnmatch, os, shutil, threading, time from helper import copy_wiredtiger_home, simulate_crash_restart from test_rollback_to_stable01 import test_rollback_to_stable_base -from wiredtiger import stat, wiredtiger_strerror, WiredTigerError, WT_ROLLBACK +from wiredtiger import stat from wtdataset import SimpleDataSet from wtscenario import make_scenarios from wtthread import checkpoint_thread, op_thread -from time import sleep def timestamp_str(t): return '%x' % t -def retry_rollback(self, name, txn_session, code): - retry_limit = 100 - retries = 0 - completed = False - saved_exception = None - while not completed and retries < retry_limit: - if retries != 0: - self.pr("Retrying operation for " + name) - if txn_session: - txn_session.rollback_transaction() - sleep(0.1) - if txn_session: - txn_session.begin_transaction('isolation=snapshot') - self.pr("Began new transaction for " + name) - try: - code() - completed = True - except WiredTigerError as e: - rollback_str = wiredtiger_strerror(WT_ROLLBACK) - if rollback_str not in str(e): - raise(e) - retries += 1 - saved_exception = e - if not completed and saved_exception: - raise(saved_exception) - # test_rollback_to_stable10.py # Test the rollback to stable operation performs sweeping history store. class test_rollback_to_stable10(test_rollback_to_stable_base): @@ -155,13 +128,13 @@ class test_rollback_to_stable10(test_rollback_to_stable_base): # Perform several updates in parallel with checkpoint. # Rollbacks may occur when checkpoint is running, so retry as needed. self.pr("updates") - retry_rollback(self, 'update ds1, e', None, + self.retry_rollback('update ds1, e', None, lambda: self.large_updates(uri_1, value_e, ds_1, nrows, self.prepare, 70)) - retry_rollback(self, 'update ds2, e', None, + self.retry_rollback('update ds2, e', None, lambda: self.large_updates(uri_2, value_e, ds_2, nrows, self.prepare, 70)) - retry_rollback(self, 'update ds1, f', None, + self.retry_rollback('update ds1, f', None, lambda: self.large_updates(uri_1, value_f, ds_1, nrows, self.prepare, 80)) - retry_rollback(self, 'update ds2, f', None, + self.retry_rollback('update ds2, f', None, lambda: self.large_updates(uri_2, value_f, ds_2, nrows, self.prepare, 80)) finally: done.set() @@ -289,7 +262,7 @@ class test_rollback_to_stable10(test_rollback_to_stable_base): session_p1 = self.conn.open_session() cursor_p1 = session_p1.open_cursor(uri_1) session_p1.begin_transaction('isolation=snapshot') - retry_rollback(self, 'update ds1', session_p1, + self.retry_rollback('update ds1', session_p1, lambda: prepare_range_updates( session_p1, cursor_p1, ds_1, value_e, nrows, 'prepare_timestamp=' + timestamp_str(69))) @@ -298,7 +271,7 @@ class test_rollback_to_stable10(test_rollback_to_stable_base): session_p2 = self.conn.open_session() cursor_p2 = session_p2.open_cursor(uri_2) session_p2.begin_transaction('isolation=snapshot') - retry_rollback(self, 'update ds2', session_p2, + self.retry_rollback('update ds2', session_p2, lambda: prepare_range_updates( session_p2, cursor_p2, ds_2, value_e, nrows, 'prepare_timestamp=' + timestamp_str(69))) diff --git a/src/third_party/wiredtiger/test/suite/test_rollback_to_stable13.py b/src/third_party/wiredtiger/test/suite/test_rollback_to_stable13.py index 66f66d62874..73f23b1f615 100644 --- a/src/third_party/wiredtiger/test/suite/test_rollback_to_stable13.py +++ b/src/third_party/wiredtiger/test/suite/test_rollback_to_stable13.py @@ -25,8 +25,6 @@ # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. - -import fnmatch, os, shutil, time from helper import simulate_crash_restart from test_rollback_to_stable01 import test_rollback_to_stable_base from wiredtiger import stat @@ -37,7 +35,8 @@ def timestamp_str(t): return '%x' % t # test_rollback_to_stable13.py -# Test the rollback to stable should roll back the tombstone in the history store. +# Test the rollback to stable should retain/restore the tombstone from +# the update list or from the history store for on-disk database. class test_rollback_to_stable13(test_rollback_to_stable_base): session_config = 'isolation=snapshot' @@ -54,7 +53,7 @@ class test_rollback_to_stable13(test_rollback_to_stable_base): scenarios = make_scenarios(key_format_values, prepare_values) def conn_config(self): - config = 'cache_size=500MB,statistics=(all),log=(enabled=true)' + config = 'cache_size=50MB,statistics=(all),log=(enabled=true)' return config def test_rollback_to_stable(self): @@ -98,6 +97,144 @@ class test_rollback_to_stable13(test_rollback_to_stable_base): self.conn.set_timestamp('stable_timestamp=' + timestamp_str(40)) self.session.checkpoint() + # Simulate a server crash and restart. + simulate_crash_restart(self, ".", "RESTART") + + # Check that the correct data is seen at and after the stable timestamp. + self.check(None, uri, 0, 50) + + # Check that we restore the correct value from the history store. + self.check(value_a, uri, nrows, 20) + + stat_cursor = self.session.open_cursor('statistics:', None, None) + restored_tombstones = stat_cursor[stat.conn.txn_rts_hs_restore_tombstones][2] + self.assertEqual(restored_tombstones, nrows) + + def test_rollback_to_stable_with_aborted_updates(self): + nrows = 1000 + + # Prepare transactions for column store table is not yet supported. + if self.prepare and self.key_format == 'r': + self.skipTest('Prepare transactions for column store table is not yet supported') + + # Create a table without logging. + uri = "table:rollback_to_stable13" + ds = SimpleDataSet( + self, uri, 0, key_format=self.key_format, value_format="S", config='split_pct=50,log=(enabled=false)') + ds.populate() + + # Pin oldest and stable to timestamp 10. + self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(10) + + ',stable_timestamp=' + timestamp_str(10)) + + value_a = "aaaaa" * 100 + value_b = "bbbbb" * 100 + value_c = "ccccc" * 100 + value_d = "ddddd" * 100 + + # Perform several updates. + self.large_updates(uri, value_a, ds, nrows, self.prepare, 20) + + # Update a large number of records and rollback. + cursor = self.session.open_cursor(uri) + for i in range(1, nrows + 1): + self.session.begin_transaction() + cursor[ds.key(i)] = value_b + self.session.rollback_transaction() + cursor.close() + + # Update a large number of records and rollback. + cursor = self.session.open_cursor(uri) + for i in range(1, nrows + 1): + self.session.begin_transaction() + cursor[ds.key(i)] = value_c + self.session.rollback_transaction() + cursor.close() + + # Perform several removes. + self.large_removes(uri, ds, nrows, self.prepare, 30) + + # Perform several updates. + self.large_updates(uri, value_d, ds, nrows, self.prepare, 60) + + # Verify data is visible and correct. + self.check(value_a, uri, nrows, 20) + self.check(None, uri, 0, 30) + self.check(value_d, uri, nrows, 60) + + # Pin stable to timestamp 50 if prepare otherwise 40. + if self.prepare: + self.conn.set_timestamp('stable_timestamp=' + timestamp_str(50)) + else: + self.conn.set_timestamp('stable_timestamp=' + timestamp_str(40)) + + self.session.checkpoint() + # Simulate a server crash and restart. + simulate_crash_restart(self, ".", "RESTART") + + # Check that the correct data is seen at and after the stable timestamp. + self.check(None, uri, 0, 50) + + # Check that we restore the correct value from the history store. + self.check(value_a, uri, nrows, 20) + + stat_cursor = self.session.open_cursor('statistics:', None, None) + restored_tombstones = stat_cursor[stat.conn.txn_rts_hs_restore_tombstones][2] + self.assertEqual(restored_tombstones, nrows) + + def test_rollback_to_stable_with_history_tombstone(self): + nrows = 1000 + + # Prepare transactions for column store table is not yet supported. + if self.prepare and self.key_format == 'r': + self.skipTest('Prepare transactions for column store table is not yet supported') + + # Create a table without logging. + uri = "table:rollback_to_stable13" + ds = SimpleDataSet( + self, uri, 0, key_format=self.key_format, value_format="S", config='split_pct=50,log=(enabled=false)') + ds.populate() + + # Pin oldest and stable to timestamp 10. + self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(10) + + ',stable_timestamp=' + timestamp_str(10)) + + value_a = "aaaaa" * 100 + value_b = "bbbbb" * 100 + value_c = "ccccc" * 100 + + # Perform several updates. + self.large_updates(uri, value_a, ds, nrows, self.prepare, 20) + + # Perform several removes. + self.large_removes(uri, ds, nrows, self.prepare, 30) + + # Perform several updates and removes in a single transaction. + cursor = self.session.open_cursor(uri) + for i in range(1, nrows + 1): + self.session.begin_transaction() + cursor[ds.key(i)] = value_b + cursor.set_key(ds.key(i)) + cursor.remove() + self.session.commit_transaction('commit_timestamp=' + timestamp_str(40)) + cursor.close() + + # Pin stable to timestamp 50 if prepare otherwise 40. + if self.prepare: + self.conn.set_timestamp('stable_timestamp=' + timestamp_str(50)) + else: + self.conn.set_timestamp('stable_timestamp=' + timestamp_str(40)) + + self.session.checkpoint() + + # Perform several updates and checkpoint. + self.large_updates(uri, value_c, ds, nrows, self.prepare, 60) + self.session.checkpoint() + + # Verify data is visible and correct. + self.check(value_a, uri, nrows, 20) + self.check(None, uri, 0, 40) + self.check(value_c, uri, nrows, 60) # Simulate a server crash and restart. simulate_crash_restart(self, ".", "RESTART") @@ -111,3 +248,51 @@ class test_rollback_to_stable13(test_rollback_to_stable_base): stat_cursor = self.session.open_cursor('statistics:', None, None) restored_tombstones = stat_cursor[stat.conn.txn_rts_hs_restore_tombstones][2] self.assertEqual(restored_tombstones, nrows) + + def test_rollback_to_stable_with_stable_remove(self): + nrows = 1000 + # Prepare transactions for column store table is not yet supported. + if self.prepare and self.key_format == 'r': + self.skipTest('Prepare transactions for column store table is not yet supported') + # Create a table without logging. + uri = "table:rollback_to_stable13" + ds = SimpleDataSet( + self, uri, 0, key_format=self.key_format, value_format="S", config='split_pct=50,log=(enabled=false)') + ds.populate() + # Pin oldest and stable to timestamp 10. + self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(10) + + ',stable_timestamp=' + timestamp_str(10)) + value_a = "aaaaa" * 100 + value_b = "bbbbb" * 100 + value_c = "ccccc" * 100 + # Perform several updates. + self.large_updates(uri, value_a, ds, nrows, self.prepare, 20) + # Perform several updates. + self.large_updates(uri, value_b, ds, nrows, self.prepare, 30) + # Perform several removes. + self.large_removes(uri, ds, nrows, self.prepare, 40) + # Pin stable to timestamp 50 if prepare otherwise 40. + if self.prepare: + self.conn.set_timestamp('stable_timestamp=' + timestamp_str(50)) + else: + self.conn.set_timestamp('stable_timestamp=' + timestamp_str(40)) + # Perform several updates and checkpoint. + self.large_updates(uri, value_c, ds, nrows, self.prepare, 60) + self.session.checkpoint() + # Verify data is visible and correct. + self.check(value_a, uri, nrows, 20) + self.check(None, uri, 0, 40) + self.check(value_c, uri, nrows, 60) + self.conn.rollback_to_stable() + # Perform several updates and checkpoint. + self.large_updates(uri, value_c, ds, nrows, self.prepare, 60) + self.session.checkpoint() + # Simulate a server crash and restart. + simulate_crash_restart(self, ".", "RESTART") + # Check that the correct data is seen at and after the stable timestamp. + self.check(None, uri, 0, 50) + # Check that we restore the correct value from the history store. + self.check(value_a, uri, nrows, 20) + stat_cursor = self.session.open_cursor('statistics:', None, None) + restored_tombstones = stat_cursor[stat.conn.txn_rts_hs_restore_tombstones][2] + self.assertEqual(restored_tombstones, nrows) diff --git a/src/third_party/wiredtiger/test/suite/test_rollback_to_stable14.py b/src/third_party/wiredtiger/test/suite/test_rollback_to_stable14.py index 66614938d3c..e7a5229555d 100755 --- a/src/third_party/wiredtiger/test/suite/test_rollback_to_stable14.py +++ b/src/third_party/wiredtiger/test/suite/test_rollback_to_stable14.py @@ -29,11 +29,10 @@ import fnmatch, os, shutil, threading, time from helper import simulate_crash_restart from test_rollback_to_stable01 import test_rollback_to_stable_base -from wiredtiger import stat, wiredtiger_strerror, WiredTigerError, WT_ROLLBACK +from wiredtiger import stat from wtdataset import SimpleDataSet from wtscenario import make_scenarios from wtthread import checkpoint_thread, op_thread -from time import sleep def timestamp_str(t): return '%x' % t @@ -44,32 +43,6 @@ def mod_val(value, char, location, nbytes=1): def append_val(value, char): return value + char -def retry_rollback(self, name, txn_session, code): - retry_limit = 100 - retries = 0 - completed = False - saved_exception = None - while not completed and retries < retry_limit: - if retries != 0: - self.pr("Retrying operation for " + name) - if txn_session: - txn_session.rollback_transaction() - sleep(0.1) - if txn_session: - txn_session.begin_transaction('isolation=snapshot') - self.pr("Began new transaction for " + name) - try: - code() - completed = True - except WiredTigerError as e: - rollback_str = wiredtiger_strerror(WT_ROLLBACK) - if rollback_str not in str(e): - raise(e) - retries += 1 - saved_exception = e - if not completed and saved_exception: - raise(saved_exception) - # test_rollback_to_stable14.py # Test the rollback to stable operation uses proper base update while restoring modifies from history store. class test_rollback_to_stable14(test_rollback_to_stable_base): @@ -147,13 +120,13 @@ class test_rollback_to_stable14(test_rollback_to_stable_base): # Perform several modifies in parallel with checkpoint. # Rollbacks may occur when checkpoint is running, so retry as needed. self.pr("modifies") - retry_rollback(self, 'modify ds1, W', None, + self.retry_rollback('modify ds1, W', None, lambda: self.large_modifies(uri, 'W', ds, 4, 1, nrows, self.prepare, 70)) - retry_rollback(self, 'modify ds1, X', None, + self.retry_rollback('modify ds1, X', None, lambda: self.large_modifies(uri, 'X', ds, 5, 1, nrows, self.prepare, 80)) - retry_rollback(self, 'modify ds1, Y', None, + self.retry_rollback('modify ds1, Y', None, lambda: self.large_modifies(uri, 'Y', ds, 6, 1, nrows, self.prepare, 90)) - retry_rollback(self, 'modify ds1, Z', None, + self.retry_rollback('modify ds1, Z', None, lambda: self.large_modifies(uri, 'Z', ds, 7, 1, nrows, self.prepare, 100)) finally: done.set() @@ -252,13 +225,13 @@ class test_rollback_to_stable14(test_rollback_to_stable_base): # Perform several modifies in parallel with checkpoint. # Rollbacks may occur when checkpoint is running, so retry as needed. self.pr("modifies") - retry_rollback(self, 'modify ds1, W', None, + self.retry_rollback('modify ds1, W', None, lambda: self.large_modifies(uri, 'W', ds, 4, 1, nrows, self.prepare, 70)) - retry_rollback(self, 'modify ds1, X', None, + self.retry_rollback('modify ds1, X', None, lambda: self.large_modifies(uri, 'X', ds, 5, 1, nrows, self.prepare, 80)) - retry_rollback(self, 'modify ds1, Y', None, + self.retry_rollback('modify ds1, Y', None, lambda: self.large_modifies(uri, 'Y', ds, 6, 1, nrows, self.prepare, 90)) - retry_rollback(self, 'modify ds1, Z', None, + self.retry_rollback('modify ds1, Z', None, lambda: self.large_modifies(uri, 'Z', ds, 7, 1, nrows, self.prepare, 100)) finally: done.set() @@ -355,13 +328,13 @@ class test_rollback_to_stable14(test_rollback_to_stable_base): # Perform several modifies in parallel with checkpoint. # Rollbacks may occur when checkpoint is running, so retry as needed. self.pr("modifies") - retry_rollback(self, 'modify ds1, W', None, + self.retry_rollback('modify ds1, W', None, lambda: self.large_modifies(uri, 'W', ds, len(value_modT), 1, nrows, self.prepare, 70)) - retry_rollback(self, 'modify ds1, X', None, + self.retry_rollback('modify ds1, X', None, lambda: self.large_modifies(uri, 'X', ds, len(value_modT) + 1, 1, nrows, self.prepare, 80)) - retry_rollback(self, 'modify ds1, Y', None, + self.retry_rollback('modify ds1, Y', None, lambda: self.large_modifies(uri, 'Y', ds, len(value_modT) + 2, 1, nrows, self.prepare, 90)) - retry_rollback(self, 'modify ds1, Z', None, + self.retry_rollback('modify ds1, Z', None, lambda: self.large_modifies(uri, 'Z', ds, len(value_modT) + 3, 1, nrows, self.prepare, 100)) finally: done.set() diff --git a/src/third_party/wiredtiger/test/suite/test_rollback_to_stable15.py b/src/third_party/wiredtiger/test/suite/test_rollback_to_stable15.py index 03a6aa396dd..b91691b99cd 100644 --- a/src/third_party/wiredtiger/test/suite/test_rollback_to_stable15.py +++ b/src/third_party/wiredtiger/test/suite/test_rollback_to_stable15.py @@ -77,6 +77,7 @@ class test_rollback_to_stable15(wttest.WiredTigerTestCase): self.assertEqual(v, check_value) count += 1 session.commit_transaction() + cursor.close() self.assertEqual(count, nrows) def test_rollback_to_stable(self): @@ -107,6 +108,7 @@ class test_rollback_to_stable15(wttest.WiredTigerTestCase): self.session.begin_transaction() cursor[i] = value30 self.session.commit_transaction('commit_timestamp=' + timestamp_str(5)) + cursor.close() #Set stable timestamp to 2 self.conn.set_timestamp('stable_timestamp=' + timestamp_str(2)) @@ -115,6 +117,7 @@ class test_rollback_to_stable15(wttest.WiredTigerTestCase): self.check(value20, uri, nrows - 1, 2) #Second Update to value30 at timestamp 7 + cursor = self.session.open_cursor(uri) for i in range(1, nrows): self.session.begin_transaction() cursor[i] = value30 @@ -125,6 +128,7 @@ class test_rollback_to_stable15(wttest.WiredTigerTestCase): self.session.begin_transaction() cursor[i] = value40 self.session.commit_transaction('commit_timestamp=' + timestamp_str(9)) + cursor.close() #Set stable timestamp to 7 self.conn.set_timestamp('stable_timestamp=' + timestamp_str(7)) diff --git a/src/third_party/wiredtiger/test/suite/test_rollback_to_stable16.py b/src/third_party/wiredtiger/test/suite/test_rollback_to_stable16.py index 0c0a3235e94..5ca08c18baf 100644 --- a/src/third_party/wiredtiger/test/suite/test_rollback_to_stable16.py +++ b/src/third_party/wiredtiger/test/suite/test_rollback_to_stable16.py @@ -25,6 +25,10 @@ # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. +# +# [TEST_TAGS] +# rollback_to_stable +# [END_TAGS] import os, shutil from helper import simulate_crash_restart diff --git a/src/third_party/wiredtiger/test/suite/test_rollback_to_stable18.py b/src/third_party/wiredtiger/test/suite/test_rollback_to_stable18.py index 68c2e8d0205..a9725841b8b 100644 --- a/src/third_party/wiredtiger/test/suite/test_rollback_to_stable18.py +++ b/src/third_party/wiredtiger/test/suite/test_rollback_to_stable18.py @@ -25,6 +25,11 @@ # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. +# +# [TEST_TAGS] +# rollback_to_stable +# aggregated_time_windows +# [END_TAGS] import fnmatch, os, shutil, time from helper import simulate_crash_restart diff --git a/src/third_party/wiredtiger/test/suite/test_rollback_to_stable19.py b/src/third_party/wiredtiger/test/suite/test_rollback_to_stable19.py index 284499dae64..dd98263ae41 100644 --- a/src/third_party/wiredtiger/test/suite/test_rollback_to_stable19.py +++ b/src/third_party/wiredtiger/test/suite/test_rollback_to_stable19.py @@ -113,6 +113,7 @@ class test_rollback_to_stable19(test_rollback_to_stable_base): cursor2.set_key(1) self.assertEquals(cursor2.search(), WT_NOTFOUND) self.session.commit_transaction() + cursor2.close() # Pin stable timestamp to 20. self.conn.set_timestamp('stable_timestamp=' + timestamp_str(20)) @@ -138,6 +139,7 @@ class test_rollback_to_stable19(test_rollback_to_stable_base): keys_removed = stat_cursor[stat.conn.txn_rts_keys_removed][2] self.assertGreater(upd_aborted, 0) self.assertGreater(keys_removed, 0) + stat_cursor.close() def test_rollback_to_stable_with_history(self): nrows = 1000 @@ -193,6 +195,7 @@ class test_rollback_to_stable19(test_rollback_to_stable_base): cursor2.set_key(1) self.assertEquals(cursor2.search(), WT_NOTFOUND) self.session.commit_transaction() + cursor2.close() # Pin stable timestamp to 40. self.conn.set_timestamp('stable_timestamp=' + timestamp_str(40)) diff --git a/src/third_party/wiredtiger/test/suite/test_rollback_to_stable21.py b/src/third_party/wiredtiger/test/suite/test_rollback_to_stable21.py index 8404af43df9..4e44b9c2d97 100644 --- a/src/third_party/wiredtiger/test/suite/test_rollback_to_stable21.py +++ b/src/third_party/wiredtiger/test/suite/test_rollback_to_stable21.py @@ -25,6 +25,11 @@ # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. +# +# [TEST_TAGS] +# rollback_to_stable:prepare +# rollback_to_stable:out_of_order_timestamps +# [END_TAGS] from wiredtiger import stat, WT_NOTFOUND from wtscenario import make_scenarios diff --git a/src/third_party/wiredtiger/test/suite/test_rollback_to_stable22.py b/src/third_party/wiredtiger/test/suite/test_rollback_to_stable22.py new file mode 100644 index 00000000000..ac56c6c6b5f --- /dev/null +++ b/src/third_party/wiredtiger/test/suite/test_rollback_to_stable22.py @@ -0,0 +1,84 @@ +#!/usr/bin/env python +# +# Public Domain 2014-present MongoDB, Inc. +# Public Domain 2008-2014 WiredTiger, Inc. +# +# This is free and unencumbered software released into the public domain. +# +# Anyone is free to copy, modify, publish, use, compile, sell, or +# distribute this software, either in source code form or as a compiled +# binary, for any purpose, commercial or non-commercial, and by any +# means. +# +# In jurisdictions that recognize copyright laws, the author or authors +# of this software dedicate any and all copyright interest in the +# software to the public domain. We make this dedication for the benefit +# of the public at large and to the detriment of our heirs and +# successors. We intend this dedication to be an overt act of +# relinquishment in perpetuity of all present and future rights to this +# software under copyright law. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +# IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. + +from test_rollback_to_stable01 import test_rollback_to_stable_base +from wtdataset import SimpleDataSet + +def timestamp_str(t): + return '%x' % t + +# test_rollback_to_stable22 +# Test history store operations conflicting with rollback to stable. We're trying to trigger a +# history store eviction concurrently to a rollback to stable call. We'll do this by limiting +# the cache size to 100MB and performing 100MB worth of inserts while periodically calling rollback +# to stable. +class test_rollback_to_stable22(test_rollback_to_stable_base): + conn_config = 'cache_size=100MB,statistics=(fast),statistics_log=(wait=1,json)' + session_config = 'isolation=snapshot' + prepare = False + + def test_rollback_to_stable(self): + nrows = 1000 + nds = 10 + + # Create a few tables and populate them with some initial data. + # + # Our way of preventing history store operations from interfering with rollback to stable's + # transaction check is by draining active evictions from each open dhandle. + # + # It's important that we have a few different tables to work with so that it's + # representative of a real situation. But also don't make the number too high relative to + # the number of updates or we may not have history for any of the tables. + ds_list = list() + for i in range(0, nds): + uri = 'table:rollback_to_stable22_{}'.format(i) + ds = SimpleDataSet( + self, uri, 0, key_format='i', value_format='S', config='log=(enabled=false)') + ds.populate() + ds_list.append(ds) + self.assertEqual(len(ds_list), nds) + + # 100 bytes of data are being inserted into 1000 rows. + # This happens 1000 iterations. + # Overall, that's 100MB of data which is guaranteed to kick start eviction. + for i in range(1, 1000): + # Generate a value, timestamp and table based off the index. + value = str(i)[0] * 100 + ts = i * 10 + ds = ds_list[i % nds] + + # Perform updates. + self.large_updates(ds.uri, value, ds, nrows, False, ts) + + # Every hundred updates, let's run rollback to stable. This is likely to happen during + # a history store eviction at least once. + if i % 100 == 0: + # Put the timestamp backwards so we can rollback the updates we just did. + stable_ts = (i - 1) * 10 + self.conn.set_timestamp('stable_timestamp=' + timestamp_str(stable_ts)) + self.conn.rollback_to_stable() diff --git a/src/third_party/wiredtiger/test/suite/test_schema03.py b/src/third_party/wiredtiger/test/suite/test_schema03.py index 3a1e5b50a52..1ab11a64f82 100755 --- a/src/third_party/wiredtiger/test/suite/test_schema03.py +++ b/src/third_party/wiredtiger/test/suite/test_schema03.py @@ -25,6 +25,11 @@ # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. +# +# [TEST_TAGS] +# schema_api +# indexes +# [END_TAGS] import os import suite_random diff --git a/src/third_party/wiredtiger/test/suite/test_search_near01.py b/src/third_party/wiredtiger/test/suite/test_search_near01.py index 2e54671c06c..80f50d2f97a 100644..100755 --- a/src/third_party/wiredtiger/test/suite/test_search_near01.py +++ b/src/third_party/wiredtiger/test/suite/test_search_near01.py @@ -215,6 +215,7 @@ class test_search_near01(wttest.WiredTigerTestCase): uri = 'table:test_row_search' self.session.create(uri, 'key_format=u,value_format=u') cursor = self.session.open_cursor(uri) + expect_count = self.get_stat(stat.conn.cursor_next_skip_lt_100) session2 = self.conn.open_session() l = "abcdefghijklmnopqrstuvwxyz" # Insert keys a -> z, except c @@ -238,8 +239,9 @@ class test_search_near01(wttest.WiredTigerTestCase): cursor2.set_key('c') cursor2.search_near() + expect_count += 1 skip_count = self.get_stat(stat.conn.cursor_next_skip_lt_100) - self.assertEqual(skip_count, 3) + self.assertEqual(skip_count, expect_count) session2.commit_transaction() # Perform an insertion and removal of a key next to another key, then search for the @@ -256,8 +258,9 @@ class test_search_near01(wttest.WiredTigerTestCase): cursor.set_key('dd') cursor.search_near() self.session.commit_transaction() + expect_count += 1 skip_count = self.get_stat(stat.conn.cursor_next_skip_lt_100) - self.assertEqual(skip_count, 4) + self.assertEqual(skip_count, expect_count) # Test a basic prepared scenario. def test_prepared(self): diff --git a/src/third_party/wiredtiger/test/suite/test_stat03.py b/src/third_party/wiredtiger/test/suite/test_stat03.py index 1f3434fece3..ca1eaa5c83f 100644 --- a/src/third_party/wiredtiger/test/suite/test_stat03.py +++ b/src/third_party/wiredtiger/test/suite/test_stat03.py @@ -25,7 +25,10 @@ # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. - +# +# [TEST_TAGS] +# cursors:statistics +# [END_TAGS] import itertools, wiredtiger, wttest from suite_subprocess import suite_subprocess from wiredtiger import stat diff --git a/src/third_party/wiredtiger/test/suite/test_stat_log02.py b/src/third_party/wiredtiger/test/suite/test_stat_log02.py index f08bfe0281b..cc808c388c9 100644 --- a/src/third_party/wiredtiger/test/suite/test_stat_log02.py +++ b/src/third_party/wiredtiger/test/suite/test_stat_log02.py @@ -25,6 +25,10 @@ # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. +# +# [TEST_TAGS] +# statistics +# [END_TAGS] import glob, json import helper, wiredtiger, wttest diff --git a/src/third_party/wiredtiger/test/suite/test_tiered02.py b/src/third_party/wiredtiger/test/suite/test_tiered02.py index 3416e581acb..6e50ec595d6 100755 --- a/src/third_party/wiredtiger/test/suite/test_tiered02.py +++ b/src/third_party/wiredtiger/test/suite/test_tiered02.py @@ -32,25 +32,20 @@ from wtdataset import SimpleDataSet # test_tiered02.py # Test tiered tree class test_tiered02(wttest.WiredTigerTestCase): - K = 1024 - M = 1024 * K - G = 1024 * M uri = "table:test_tiered02" auth_token = "test_token" bucket = "mybucket" bucket_prefix = "pfx_" extension_name = "local_store" - prefix = "pfx-" def conn_config(self): if not os.path.exists(self.bucket): os.mkdir(self.bucket) return \ - 'statistics=(all),' + \ 'tiered_storage=(auth_token=%s,' % self.auth_token + \ 'bucket=%s,' % self.bucket + \ - 'bucket_prefix=%s,' % self.prefix + \ + 'bucket_prefix=%s,' % self.bucket_prefix + \ 'name=%s),tiered_manager=(wait=0)' % self.extension_name # Load the local store extension, but skip the test if it is missing. @@ -66,6 +61,21 @@ class test_tiered02(wttest.WiredTigerTestCase): got = sorted(list(os.listdir(self.bucket))) self.pr('Flushed objects: ' + str(got)) if increase: + # WT-7639: we know that this assertion sometimes fails, + # we are collecting more data - we still want it to fail + # so it is noticed. + if len(got) <= self.flushed_objects: + from time import sleep + self.prout('directory items: {} is not greater than {}!'. + format(got, self.flushed_objects)) + self.prout('waiting to see if it resolves') + for i in range(0, 10): + self.prout('checking again') + newgot = sorted(list(os.listdir(self.bucket))) + if len(newgot) > self.flushed_objects: + self.prout('resolved, now see: {}'.format(newgot)) + break + sleep(i) self.assertGreater(len(got), self.flushed_objects) else: self.assertEqual(len(got), self.flushed_objects) @@ -93,31 +103,30 @@ class test_tiered02(wttest.WiredTigerTestCase): self.progress('flush_tier') self.session.flush_tier(None) self.confirm_flush() + ds.check() - # FIXME-WT-7589 reopening a connection does not yet work. - if False: - self.close_conn() - self.progress('reopen_conn') - self.reopen_conn() - # Check what was there before - ds = SimpleDataSet(self, self.uri, 10, config=args) - ds.check() + self.close_conn() + self.progress('reopen_conn') + self.reopen_conn() + # Check what was there before + ds = SimpleDataSet(self, self.uri, 10, config=args) + ds.check() self.progress('Create simple data set (50)') ds = SimpleDataSet(self, self.uri, 50, config=args) self.progress('populate') ds.populate() ds.check() + self.progress('open extra cursor on ' + self.uri) + cursor = self.session.open_cursor(self.uri, None, None) self.progress('checkpoint') self.session.checkpoint() + self.progress('flush_tier') self.session.flush_tier(None) + self.progress('flush_tier complete') self.confirm_flush() - # FIXME-WT-7589 This test works up to this point, then runs into trouble. - if True: - return - self.progress('Create simple data set (100)') ds = SimpleDataSet(self, self.uri, 100, config=args) self.progress('populate') @@ -134,12 +143,13 @@ class test_tiered02(wttest.WiredTigerTestCase): self.progress('populate') ds.populate() ds.check() + cursor.close() self.progress('close_conn') self.close_conn() - self.confirm_flush() # closing the connection does a checkpoint self.progress('reopen_conn') self.reopen_conn() + # Check what was there before ds = SimpleDataSet(self, self.uri, 200, config=args) ds.check() diff --git a/src/third_party/wiredtiger/test/suite/test_tiered03.py b/src/third_party/wiredtiger/test/suite/test_tiered03.py index 4c870e366c2..536bc00dd70 100755 --- a/src/third_party/wiredtiger/test/suite/test_tiered03.py +++ b/src/third_party/wiredtiger/test/suite/test_tiered03.py @@ -51,7 +51,6 @@ class test_tiered03(wttest.WiredTigerTestCase): bucket = "mybucket" bucket_prefix = "pfx_" extension_name = "local_store" - prefix = "pfx-" def conn_config(self): if not os.path.exists(self.bucket): @@ -60,7 +59,7 @@ class test_tiered03(wttest.WiredTigerTestCase): 'statistics=(all),' + \ 'tiered_storage=(auth_token=%s,' % self.auth_token + \ 'bucket=%s,' % self.bucket + \ - 'bucket_prefix=%s,' % self.prefix + \ + 'bucket_prefix=%s,' % self.bucket_prefix + \ 'name=%s)' % self.extension_name # Load the local store extension, but skip the test if it is missing. diff --git a/src/third_party/wiredtiger/test/suite/test_tiered04.py b/src/third_party/wiredtiger/test/suite/test_tiered04.py index 78a7e274e53..bca7f43fc4d 100755 --- a/src/third_party/wiredtiger/test/suite/test_tiered04.py +++ b/src/third_party/wiredtiger/test/suite/test_tiered04.py @@ -133,12 +133,7 @@ class test_tiered04(wttest.WiredTigerTestCase): self.check(c, 3) self.pr("flush tier again, holding open cursor") - # FIXME-WT-7591 Remove the extra cursor close and open surrounding the flush_tier call. - # Having a cursor open during a flush_tier does not yet work, so the test closes it, - # and reopens after the flush_tier. - c.close() self.session.flush_tier(None) - c = self.session.open_cursor(self.uri) c["3"] = "3" self.check(c, 4) diff --git a/src/third_party/wiredtiger/test/suite/test_tiered06.py b/src/third_party/wiredtiger/test/suite/test_tiered06.py index c797936a82b..d1eb9feae6f 100755 --- a/src/third_party/wiredtiger/test/suite/test_tiered06.py +++ b/src/third_party/wiredtiger/test/suite/test_tiered06.py @@ -47,14 +47,7 @@ class test_tiered06(wttest.WiredTigerTestCase): pdb.set_trace() def get_local_storage_source(self): - local = self.conn.get_storage_source('local_store') - - # Note: do not call local.terminate() . - # Since the local_storage extension has been loaded as a consequence of the - # wiredtiger_open call, WiredTiger already knows to call terminate when the connection - # closes. Calling it twice would attempt to free the same memory twice. - local.terminate = None - return local + return self.conn.get_storage_source('local_store') def test_local_basic(self): # Test some basic functionality of the storage source API, calling @@ -133,6 +126,7 @@ class test_tiered06(wttest.WiredTigerTestCase): self.assertEquals(fs.fs_directory_list(session, '', ''), ['foobar']) fs.terminate(session) + local.terminate(session) def test_local_write_read(self): # Write and read to a file non-sequentially. @@ -191,6 +185,7 @@ class test_tiered06(wttest.WiredTigerTestCase): else: self.assertEquals(in_block, a_block) fh.close(session) + local.terminate(session) def create_with_fs(self, fs, fname): session = self.session @@ -347,5 +342,12 @@ class test_tiered06(wttest.WiredTigerTestCase): self.check_dirlist(fs1, 'be', ['beagle']) self.check_dirlist(fs1, 'x', []) + # Terminate just one of the custom file systems. + # We should be able to terminate file systems, but we should + # also be able to terminate the storage source without terminating + # all the file systems we created. + fs1.terminate(session) + local.terminate(session) + if __name__ == '__main__': wttest.run() diff --git a/src/third_party/wiredtiger/test/suite/test_tiered07.py b/src/third_party/wiredtiger/test/suite/test_tiered07.py new file mode 100755 index 00000000000..61293325e56 --- /dev/null +++ b/src/third_party/wiredtiger/test/suite/test_tiered07.py @@ -0,0 +1,71 @@ +#!/usr/bin/env python +# +# Public Domain 2014-present MongoDB, Inc. +# Public Domain 2008-2014 WiredTiger, Inc. +# +# This is free and unencumbered software released into the public domain. +# +# Anyone is free to copy, modify, publish, use, compile, sell, or +# distribute this software, either in source code form or as a compiled +# binary, for any purpose, commercial or non-commercial, and by any +# means. +# +# In jurisdictions that recognize copyright laws, the author or authors +# of this software dedicate any and all copyright interest in the +# software to the public domain. We make this dedication for the benefit +# of the public at large and to the detriment of our heirs and +# successors. We intend this dedication to be an overt act of +# relinquishment in perpetuity of all present and future rights to this +# software under copyright law. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +# IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. + +import os, wiredtiger, wttest +StorageSource = wiredtiger.StorageSource # easy access to constants + +# test_tiered07.py +# Basic tiered storage API for schema operations. +class test_tiered07(wttest.WiredTigerTestCase): + uri = "table:test_tiered07" + newuri = "table:tier_rename" + + auth_token = "test_token" + bucket = "my_bucket" + bucket_prefix = "my_prefix" + extension_name = "local_store" + + def conn_extensions(self, extlist): + extlist.skip_if_missing = True + extlist.extension('storage_sources', self.extension_name) + + def conn_config(self): + os.mkdir(self.bucket) + return \ + 'tiered_storage=(auth_token=%s,' % self.auth_token + \ + 'bucket=%s,' % self.bucket + \ + 'bucket_prefix=%s,' % self.bucket_prefix + \ + 'name=%s,' % self.extension_name + \ + 'object_target_size=20M)' + + # Test calling schema APIs with a tiered table. + def test_tiered(self): + # Create a new tiered table. + self.session.create(self.uri, 'key_format=S') + + # Rename is not supported for tiered tables. + msg = "/is not supported/" + self.assertRaisesWithMessage(wiredtiger.WiredTigerError, + lambda:self.assertEquals(self.session.rename(self.uri, self.newuri, None), 0), msg) + + # Add a test for drop when implemented: + # Add data and flush tier. + # Drop table. Create new table with same name. + +if __name__ == '__main__': + wttest.run() diff --git a/src/third_party/wiredtiger/test/suite/test_tiered08.py b/src/third_party/wiredtiger/test/suite/test_tiered08.py new file mode 100755 index 00000000000..af5e7af32bc --- /dev/null +++ b/src/third_party/wiredtiger/test/suite/test_tiered08.py @@ -0,0 +1,145 @@ +#!/usr/bin/env python +# +# Public Domain 2014-present MongoDB, Inc. +# Public Domain 2008-2014 WiredTiger, Inc. +# +# This is free and unencumbered software released into the public domain. +# +# Anyone is free to copy, modify, publish, use, compile, sell, or +# distribute this software, either in source code form or as a compiled +# binary, for any purpose, commercial or non-commercial, and by any +# means. +# +# In jurisdictions that recognize copyright laws, the author or authors +# of this software dedicate any and all copyright interest in the +# software to the public domain. We make this dedication for the benefit +# of the public at large and to the detriment of our heirs and +# successors. We intend this dedication to be an overt act of +# relinquishment in perpetuity of all present and future rights to this +# software under copyright law. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +# IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. +# +# [TEST_TAGS] +# tiered_storage:checkpoint +# tiered_storage:flush_tier +# [END_TAGS] +# + +import os, threading, time, wiredtiger, wttest +from wiredtiger import stat +from wtthread import checkpoint_thread, flush_tier_thread + +# test_tiered08.py +# Run background checkpoints and flush_tier operations while inserting +# data into a table from another thread. +class test_tiered08(wttest.WiredTigerTestCase): + + batch_size = 100000 + + # Keep inserting keys until we've done this many flush and checkpoint ops. + ckpt_flush_target = 10 + + uri = "table:test_tiered08" + + auth_token = "test_token" + bucket = "mybucket" + bucket_prefix = "pfx_" + extension_name = "local_store" + + def conn_config(self): + if not os.path.exists(self.bucket): + os.mkdir(self.bucket) + return \ + 'statistics=(fast),' + \ + 'tiered_storage=(auth_token=%s,' % self.auth_token + \ + 'bucket=%s,' % self.bucket + \ + 'bucket_prefix=%s,' % self.bucket_prefix + \ + 'name=%s),tiered_manager=(wait=0)' % self.extension_name + + # Load the local store extension, but skip the test if it is missing. + def conn_extensions(self, extlist): + extlist.skip_if_missing = True + extlist.extension('storage_sources', self.extension_name) + + def get_stat(self, stat): + stat_cursor = self.session.open_cursor('statistics:') + val = stat_cursor[stat][2] + stat_cursor.close() + return val + + def key_gen(self, i): + return 'KEY' + str(i) + + def value_gen(self, i): + return 'VALUE_' + 'filler' * (i % 12) + str(i) + + # Populate the test table. Keep adding keys until the desired number of flush and + # checkpoint operations have happened. + def populate(self): + ckpt_count = 0 + flush_count = 0 + nkeys = 0 + + self.pr('Populating tiered table') + c = self.session.open_cursor(self.uri, None, None) + while ckpt_count < self.ckpt_flush_target or flush_count < self.ckpt_flush_target: + for i in range(nkeys, nkeys + self.batch_size): + c[self.key_gen(i)] = self.value_gen(i) + nkeys += self.batch_size + ckpt_count = self.get_stat(stat.conn.txn_checkpoint) + flush_count = self.get_stat(stat.conn.flush_tier) + c.close() + return nkeys + + def verify(self, key_count): + self.pr('Verifying tiered table') + c = self.session.open_cursor(self.uri, None, None) + for i in range(key_count): + self.assertEqual(c[self.key_gen(i)], self.value_gen(i)) + c.close() + + def test_tiered08(self): + + # FIXME-WT-7833 + # This test can trigger races in file handle access during flush_tier. + # We will re-enable it when that is fixed. + return + + cfg = self.conn_config() + self.pr('Config is: ' + cfg) + intl_page = 'internal_page_max=16K' + base_create = 'key_format=S,value_format=S,' + intl_page + self.session.create(self.uri, base_create) + + done = threading.Event() + ckpt = checkpoint_thread(self.conn, done) + flush = flush_tier_thread(self.conn, done) + + # Start background threads and give them a chance to start. + ckpt.start() + flush.start() + time.sleep(0.5) + + key_count = self.populate() + + done.set() + flush.join() + ckpt.join() + + self.verify(key_count) + + self.close_conn() + self.pr('Reopening tiered table') + self.reopen_conn() + + self.verify(key_count) + +if __name__ == '__main__': + wttest.run() diff --git a/src/third_party/wiredtiger/test/suite/test_timestamp04.py b/src/third_party/wiredtiger/test/suite/test_timestamp04.py index bc2f8669bcd..9aff0351e2d 100644 --- a/src/third_party/wiredtiger/test/suite/test_timestamp04.py +++ b/src/third_party/wiredtiger/test/suite/test_timestamp04.py @@ -150,6 +150,10 @@ class test_timestamp04(wttest.WiredTigerTestCase, suite_subprocess): # Setup an oldest timestamp to ensure state remains in cache. if k == 1: self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(1)) + cur_ts_log.close() + cur_ts_nolog.close() + cur_nots_log.close() + cur_nots_nolog.close() # Scenario: 1 # Check that we see all the inserted values(i.e 1) in all tables @@ -217,6 +221,10 @@ class test_timestamp04(wttest.WiredTigerTestCase, suite_subprocess): self.conn.set_timestamp('oldest_timestamp=' + stable_ts) # Update the values again in preparation for rolling back more. + cur_ts_log = self.session.open_cursor(self.table_ts_log) + cur_ts_nolog = self.session.open_cursor(self.table_ts_nolog) + cur_nots_log = self.session.open_cursor(self.table_nots_log) + cur_nots_nolog = self.session.open_cursor(self.table_nots_nolog) for k in keys: cur_nots_log[k] = 2 cur_nots_nolog[k] = 2 @@ -224,6 +232,10 @@ class test_timestamp04(wttest.WiredTigerTestCase, suite_subprocess): cur_ts_log[k] = 2 cur_ts_nolog[k] = 2 self.session.commit_transaction('commit_timestamp=' + timestamp_str(k + key_range)) + cur_ts_log.close() + cur_ts_nolog.close() + cur_nots_log.close() + cur_nots_nolog.close() # Scenario: 3 # Check that we see all values updated (i.e 2) in all tables. diff --git a/src/third_party/wiredtiger/test/suite/test_timestamp06.py b/src/third_party/wiredtiger/test/suite/test_timestamp06.py index e94d4657df7..d958a3ba3ed 100644 --- a/src/third_party/wiredtiger/test/suite/test_timestamp06.py +++ b/src/third_party/wiredtiger/test/suite/test_timestamp06.py @@ -154,6 +154,8 @@ class test_timestamp06(wttest.WiredTigerTestCase, suite_subprocess): cur_ts_nolog[k] = 3 self.session.commit_transaction('commit_timestamp=' + timestamp_str(301)) + cur_ts_log.close() + cur_ts_nolog.close() # Scenario: 1 # Check that we see all the latest values (i.e. 3) as per transaction diff --git a/src/third_party/wiredtiger/test/suite/test_timestamp18.py b/src/third_party/wiredtiger/test/suite/test_timestamp18.py index e6d0ac0ab0b..746551b1dec 100644 --- a/src/third_party/wiredtiger/test/suite/test_timestamp18.py +++ b/src/third_party/wiredtiger/test/suite/test_timestamp18.py @@ -26,6 +26,11 @@ # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. # +# [TEST_TAGS] +# transactions:mixed_mode_timestamps +# verify:prepare +# [END_TAGS] +# # test_timestamp18.py # Mixing timestamped and non-timestamped writes. # diff --git a/src/third_party/wiredtiger/test/suite/test_timestamp22.py b/src/third_party/wiredtiger/test/suite/test_timestamp22.py index 8dcbfbc3cd1..d03e18d8d72 100755 --- a/src/third_party/wiredtiger/test/suite/test_timestamp22.py +++ b/src/third_party/wiredtiger/test/suite/test_timestamp22.py @@ -417,6 +417,8 @@ class test_timestamp22(wttest.WiredTigerTestCase): commit_ts = self.oldest_ts if durable_ts < commit_ts: durable_ts = commit_ts + if durable_ts <= self.stable_ts: + durable_ts = self.stable_ts + 1 value = self.gen_value(iternum, commit_ts) self.updates(value, ds, do_prepare, commit_ts, durable_ts, read_ts) diff --git a/src/third_party/wiredtiger/test/suite/test_truncate01.py b/src/third_party/wiredtiger/test/suite/test_truncate01.py index c15c7ea2f92..39bffc5f163 100644 --- a/src/third_party/wiredtiger/test/suite/test_truncate01.py +++ b/src/third_party/wiredtiger/test/suite/test_truncate01.py @@ -26,6 +26,10 @@ # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. # +# [TEST_TAGS] +# truncate +# [END_TAGS] +# # test_truncate01.py # session level operations on tables # diff --git a/src/third_party/wiredtiger/test/suite/test_txn01.py b/src/third_party/wiredtiger/test/suite/test_txn01.py index 17677b911fa..9f77ed3e1eb 100644 --- a/src/third_party/wiredtiger/test/suite/test_txn01.py +++ b/src/third_party/wiredtiger/test/suite/test_txn01.py @@ -25,6 +25,10 @@ # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. +# +# [TEST_TAGS] +# transactions +# [END_TAGS] import wiredtiger, wttest from wtscenario import make_scenarios diff --git a/src/third_party/wiredtiger/test/suite/test_txn04.py b/src/third_party/wiredtiger/test/suite/test_txn04.py index f09e82bd0bd..ca41c90cf36 100644 --- a/src/third_party/wiredtiger/test/suite/test_txn04.py +++ b/src/third_party/wiredtiger/test/suite/test_txn04.py @@ -26,6 +26,11 @@ # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. # +# [TEST_TAGS] +# backup +# recovery +# [END_TAGS] +# # test_txn04.py # Transactions: hot backup and recovery # diff --git a/src/third_party/wiredtiger/test/suite/test_txn19.py b/src/third_party/wiredtiger/test/suite/test_txn19.py index 128375e577d..18c0636d603 100755 --- a/src/third_party/wiredtiger/test/suite/test_txn19.py +++ b/src/third_party/wiredtiger/test/suite/test_txn19.py @@ -26,6 +26,10 @@ # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. # +# [TEST_TAGS] +# recovery:log_files +# [END_TAGS] +# # test_txn19.py # Transactions: test recovery with corrupted log files # @@ -290,14 +294,13 @@ class test_txn19(wttest.WiredTigerTestCase, suite_subprocess): expect_fail = self.expect_recovery_failure() if expect_fail: - with self.expectedStdoutPattern('Failed wiredtiger_open'): - errmsg = '/WT_TRY_SALVAGE: database corruption detected/' - if self.kind == 'removal': - errmsg = '/No such file or directory/' - if self.kind == 'truncate': - errmsg = '/failed to read 128 bytes at offset 0/' - self.assertRaisesWithMessage(wiredtiger.WiredTigerError, - lambda: self.reopen_conn(newdir, self.base_config), errmsg) + errmsg = '/WT_TRY_SALVAGE: database corruption detected/' + if self.kind == 'removal': + errmsg = '/No such file or directory/' + if self.kind == 'truncate': + errmsg = '/failed to read 128 bytes at offset 0/' + self.assertRaisesWithMessage(wiredtiger.WiredTigerError, + lambda: self.reopen_conn(newdir, self.base_config), errmsg) else: if self.expect_warning_corruption(): with self.expectedStdoutPattern('log file .* corrupted'): @@ -488,9 +491,8 @@ class test_txn19_meta(wttest.WiredTigerTestCase, suite_subprocess): errmsg = '/is smaller than allocation size; file size=0, alloc size=4096/' if self.kind == 'removal': errmsg = '/No such file or directory/' - with self.expectedStdoutPattern('Failed wiredtiger_open'): - self.assertRaisesWithMessage(wiredtiger.WiredTigerError, - lambda: self.reopen_conn(dir, self.conn_config), errmsg) + self.assertRaisesWithMessage(wiredtiger.WiredTigerError, + lambda: self.reopen_conn(dir, self.conn_config), errmsg) else: # On non-windows platforms, we capture the renaming of WiredTiger.wt file. if os.name != 'nt' and self.filename == 'WiredTiger.turtle' and self.kind == 'removal': @@ -548,10 +550,9 @@ class test_txn19_meta(wttest.WiredTigerTestCase, suite_subprocess): # an error during the wiredtiger_open. But the nature of the # messages produced during the error is variable by which case # it is, and even variable from system to system. - with self.expectedStdoutPattern('.'): - self.assertRaisesWithMessage(wiredtiger.WiredTigerError, - lambda: self.reopen_conn(salvagedir, salvage_config), - '/.*/') + self.assertRaisesWithMessage(wiredtiger.WiredTigerError, + lambda: self.reopen_conn(salvagedir, salvage_config), + '/.*/') if __name__ == '__main__': wttest.run() diff --git a/src/third_party/wiredtiger/test/suite/test_txn22.py b/src/third_party/wiredtiger/test/suite/test_txn22.py index b96881c3ace..310c6f8a829 100755 --- a/src/third_party/wiredtiger/test/suite/test_txn22.py +++ b/src/third_party/wiredtiger/test/suite/test_txn22.py @@ -152,10 +152,9 @@ class test_txn22(wttest.WiredTigerTestCase, suite_subprocess): # Without salvage, they result in an error during the wiredtiger_open. # But the nature of the messages produced during the error is variable # by which case it is, and even variable from system to system. - with self.expectedStdoutPattern('.'): - self.assertRaisesWithMessage(wiredtiger.WiredTigerError, - lambda: self.reopen_conn(salvagedir, self.base_config), - '/.*/') + self.assertRaisesWithMessage(wiredtiger.WiredTigerError, + lambda: self.reopen_conn(salvagedir, self.base_config), + '/.*/') self.reopen_conn(salvagedir, salvage_config) if self.filename == 'test_txn22': @@ -165,10 +164,9 @@ class test_txn22(wttest.WiredTigerTestCase, suite_subprocess): # an error during the wiredtiger_open. But the nature of the # messages produced during the error is variable by which case # it is, and even variable from system to system. - with self.expectedStdoutPattern('.'): - self.assertRaisesWithMessage(wiredtiger.WiredTigerError, - lambda: self.reopen_conn(salvagedir, salvage_config), - '/.*/') + self.assertRaisesWithMessage(wiredtiger.WiredTigerError, + lambda: self.reopen_conn(salvagedir, salvage_config), + '/.*/') # The test may output the following error message while opening a file that # does not exist. Ignore that. diff --git a/src/third_party/wiredtiger/test/suite/test_txn26.py b/src/third_party/wiredtiger/test/suite/test_txn26.py index 75633b275e3..4618585b7f1 100644 --- a/src/third_party/wiredtiger/test/suite/test_txn26.py +++ b/src/third_party/wiredtiger/test/suite/test_txn26.py @@ -25,6 +25,10 @@ # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. +# +# [TEST_TAGS] +# transactions:timestamps +# [END_TAGS] import wiredtiger, wttest diff --git a/src/third_party/wiredtiger/test/suite/test_util11.py b/src/third_party/wiredtiger/test/suite/test_util11.py index 1e6046da291..583ad5b2b8d 100644 --- a/src/third_party/wiredtiger/test/suite/test_util11.py +++ b/src/third_party/wiredtiger/test/suite/test_util11.py @@ -25,6 +25,10 @@ # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. +# +# [TEST_TAGS] +# wt_util +# [END_TAGS] import os, struct from suite_subprocess import suite_subprocess diff --git a/src/third_party/wiredtiger/test/suite/test_version.py b/src/third_party/wiredtiger/test/suite/test_version.py index 977f0897c11..a73bbff9181 100644 --- a/src/third_party/wiredtiger/test/suite/test_version.py +++ b/src/third_party/wiredtiger/test/suite/test_version.py @@ -25,6 +25,10 @@ # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. +# +# [TEST_TAGS] +# connection_api +# [END_TAGS] import wiredtiger, wttest diff --git a/src/third_party/wiredtiger/test/suite/wttest.py b/src/third_party/wiredtiger/test/suite/wttest.py index ab4d7689d3f..1d22162f535 100755 --- a/src/third_party/wiredtiger/test/suite/wttest.py +++ b/src/third_party/wiredtiger/test/suite/wttest.py @@ -284,6 +284,27 @@ class WiredTigerTestCase(unittest.TestCase): self.skipped = True super(WiredTigerTestCase, self).skipTest(reason) + # Construct the expected filename for an extension library and return + # the name if the file exists. + @staticmethod + def findExtension(dirname, libname): + pat = '' + # When scanning for the extension library, we need to account for + # the binary paths produced by libtool and CMake. Libtool will export + # the library under '.libs'. + if os.path.exists(os.path.join(WiredTigerTestCase._builddir, 'ext', + dirname, libname, '.libs')): + pat = os.path.join(WiredTigerTestCase._builddir, 'ext', + dirname, libname, '.libs', 'libwiredtiger_*.so') + else: + pat = os.path.join(WiredTigerTestCase._builddir, 'ext', + dirname, libname, 'libwiredtiger_*.so') + filenames = glob.glob(pat) + if len(filenames) > 1: + raise Exception(self.shortid() + ": " + ext + + ": multiple extensions libraries found matching: " + pat) + return filenames + # Return the wiredtiger_open extension argument for # any needed shared library. def extensionsConfig(self): @@ -312,9 +333,7 @@ class WiredTigerTestCase(unittest.TestCase): ": extension is not named <dir>/<name>") libname = splits[1] dirname = splits[0] - pat = os.path.join(WiredTigerTestCase._builddir, 'ext', - dirname, libname, '.libs', 'libwiredtiger_*.so') - filenames = glob.glob(pat) + filenames = self.findExtension(dirname, libname) if len(filenames) == 0: if skipIfMissing: self.skipTest('extension "' + ext + '" not built') @@ -323,10 +342,6 @@ class WiredTigerTestCase(unittest.TestCase): raise Exception(self.shortid() + ": " + ext + ": no extensions library found matching: " + pat) - elif len(filenames) > 1: - raise Exception(self.shortid() + - ": " + ext + - ": multiple extensions libraries found matching: " + pat) complete = '"' + filenames[0] + '"' + extconf if ext in extfiles: if extfiles[ext] != complete: @@ -351,13 +366,7 @@ class WiredTigerTestCase(unittest.TestCase): # avoid confusion. sys.stdout.flush() conn_param = 'create,error_prefix="%s",%s' % (self.shortid(), config) - try: - conn = self.wiredtiger_open(home, conn_param) - except wiredtiger.WiredTigerError as e: - print("Failed wiredtiger_open: dir '%s', config '%s'" % \ - (home, conn_param)) - raise e - return conn + return self.wiredtiger_open(home, conn_param) # Replacement for wiredtiger.wiredtiger_open that returns # a proxied connection that knows to close it itself at the diff --git a/src/third_party/wiredtiger/test/suite/wtthread.py b/src/third_party/wiredtiger/test/suite/wtthread.py index eb8dd49f7f0..6e15f85dfb6 100755 --- a/src/third_party/wiredtiger/test/suite/wtthread.py +++ b/src/third_party/wiredtiger/test/suite/wtthread.py @@ -43,6 +43,20 @@ class checkpoint_thread(threading.Thread): sess.checkpoint() sess.close() +class flush_tier_thread(threading.Thread): + def __init__(self, conn, done): + self.conn = conn + self.done = done + threading.Thread.__init__(self) + + def run(self): + sess = self.conn.open_session() + while not self.done.isSet(): + # Sleep for 25 milliseconds. + time.sleep(0.0025) + sess.flush_tier() + sess.close() + class backup_thread(threading.Thread): def __init__(self, conn, backup_dir, done): self.backup_dir = backup_dir @@ -81,20 +95,16 @@ class backup_thread(threading.Thread): uri = "file:" + next_file uris.append(uri) - # TODO: We want a self.assertTrue here - be need to be a - # wttest to do that.. - if not compare_tables( - self, sess, uris, "checkpoint=WiredTigerCheckpoint"): - print("Error: checkpoint tables differ.") - else: - wttest.WiredTigerTestCase.printVerbose( - 3, "Checkpoint tables match") + # Add an assert to stop running the test if any difference in table contents + # is found. We would have liked to use self.assertTrue instead, but are unable + # to because backup_thread does not support this method unless it is a wttest. + wttest.WiredTigerTestCase.printVerbose(3, "Testing if checkpoint tables match:") + assert compare_tables(self, sess, uris) == True + wttest.WiredTigerTestCase.printVerbose(3, "Checkpoint tables match") - if not compare_tables(self, bkp_session, uris): - print("Error: backup tables differ.") - else: - wttest.WiredTigerTestCase.printVerbose( - 3, "Backup tables match") + wttest.WiredTigerTestCase.printVerbose(3, "Testing if backup tables match:") + assert compare_tables(self, bkp_session, uris) == True + wttest.WiredTigerTestCase.printVerbose(3, "Backup tables match") finally: if bkp_conn != None: bkp_conn.close() diff --git a/src/third_party/wiredtiger/test/test_coverage.md b/src/third_party/wiredtiger/test/test_coverage.md index 1b29241ff32..bc71f581c09 100644 --- a/src/third_party/wiredtiger/test/test_coverage.md +++ b/src/third_party/wiredtiger/test/test_coverage.md @@ -1,6 +1,56 @@ -|Component|Test Type|Testing Area|Description|Existing tests| -|---|---|---|---|---| -|Backup|Correctness|Full Backup|Full backup contains correct data|[../test/suite/test_backup01.py](../test/suite/test_backup01.py) -|Caching Eviction|Correctness|Written Data|Ensure that the data written out by eviction is correct after reading|[../test/suite/test_hs15.py](../test/suite/test_hs15.py) -|Checkpoints|Correctness|Checkpoint Data|On system with a complex, concurrent workload the correct versions of data appear in checkpoints|[../test/suite/test_checkpoint02.py](../test/suite/test_checkpoint02.py), [../test/suite/test_checkpoint03.py](../test/suite/test_checkpoint03.py) -|Checkpoints|Liveness|Liveness|Identify bugs and race conditions related to checkpoints that can cause deadlocks or livelocks|[../test/csuite/wt3363_checkpoint_op_races/main.c](../test/csuite/wt3363_checkpoint_op_races/main.c) +## Data Correctness tests: + +|Component|Sub-component|Existing tests| +|---|---|---| +|Checkpoint||[main.c](../test/csuite/wt3363_checkpoint_op_races/main.c) +## Functional Correctness tests: + +|Component|Sub-component|Existing tests| +|---|---|---| +|Aggregated Time Windows||[test_rollback_to_stable18.py](../test/suite/test_rollback_to_stable18.py) +|Backup||[test_txn04.py](../test/suite/test_txn04.py) +|Backup|Cursors|[test_backup01.py](../test/suite/test_backup01.py), [test_backup11.py](../test/suite/test_backup11.py) +|Checkpoint||[test_checkpoint02.py](../test/suite/test_checkpoint02.py) +|Checkpoint|Garbage Collection|[test_gc01.py](../test/suite/test_gc01.py) +|Checkpoint|History Store|[test_checkpoint03.py](../test/suite/test_checkpoint03.py) +|Checkpoint|Metadata|[test_checkpoint_snapshot01.py](../test/suite/test_checkpoint_snapshot01.py) +|Checkpoint|Obsolete Data|[test_checkpoint08.py](../test/suite/test_checkpoint08.py) +|Compression||[test_dictionary.py](../test/suite/test_dictionary.py) +|Config Api||[test_base02.py](../test/suite/test_base02.py), [test_config02.py](../test/suite/test_config02.py) +|Connection Api||[test_version.py](../test/suite/test_version.py) +|Connection Api|Reconfigure|[test_reconfig01.py](../test/suite/test_reconfig01.py), [test_reconfig02.py](../test/suite/test_reconfig02.py) +|Connection Api|Turtle File|[test_bug024.py](../test/suite/test_bug024.py) +|Connection Api|Wiredtiger Open|[test_config02.py](../test/suite/test_config02.py) +|Cursors|Prepare|[test_prepare_cursor01.py](../test/suite/test_prepare_cursor01.py) +|Cursors|Reconfigure|[test_cursor06.py](../test/suite/test_cursor06.py) +|Cursors|Search|[test_bug008.py](../test/suite/test_bug008.py) +|Cursors|Search Near|[test_bug008.py](../test/suite/test_bug008.py) +|Cursors|Statistics|[test_stat03.py](../test/suite/test_stat03.py) +|Encryption||[test_encrypt02.py](../test/suite/test_encrypt02.py) +|Eviction|Prepare|[test_prepare12.py](../test/suite/test_prepare12.py) +|History Store||[test_hs09.py](../test/suite/test_hs09.py) +|History Store|Eviction Checkpoint Interaction|[test_hs15.py](../test/suite/test_hs15.py) +|Huffman Encoding||[test_huffman02.py](../test/suite/test_huffman02.py) +|Indexes||[test_schema03.py](../test/suite/test_schema03.py) +|Indexes|Search Near|[test_index02.py](../test/suite/test_index02.py) +|Prepare||[test_prepare09.py](../test/suite/test_prepare09.py) +|Reconciliation|Overflow Keys|[test_bug004.py](../test/suite/test_bug004.py) +|Recovery||[test_txn04.py](../test/suite/test_txn04.py) +|Recovery|Log Files|[test_txn19.py](../test/suite/test_txn19.py) +|Rollback To Stable||[test_checkpoint_snapshot03.py](../test/suite/test_checkpoint_snapshot03.py), [test_rollback_to_stable16.py](../test/suite/test_rollback_to_stable16.py), [test_rollback_to_stable18.py](../test/suite/test_rollback_to_stable18.py) +|Rollback To Stable|Out Of Order Timestamps|[test_rollback_to_stable21.py](../test/suite/test_rollback_to_stable21.py) +|Rollback To Stable|Prepare|[test_rollback_to_stable21.py](../test/suite/test_rollback_to_stable21.py) +|Salvage|Prepare|[test_prepare_hs03.py](../test/suite/test_prepare_hs03.py) +|Schema Api||[test_schema03.py](../test/suite/test_schema03.py) +|Session Api|Reconfigure|[test_reconfig04.py](../test/suite/test_reconfig04.py) +|Session Api|Verify|[test_bug005.py](../test/suite/test_bug005.py) +|Statistics||[test_stat_log02.py](../test/suite/test_stat_log02.py) +|Tiered Storage|Checkpoint|[test_tiered08.py](../test/suite/test_tiered08.py) +|Tiered Storage|Flush Tier|[test_tiered08.py](../test/suite/test_tiered08.py) +|Transactions||[test_txn01.py](../test/suite/test_txn01.py) +|Transactions|Mixed Mode Timestamps|[test_timestamp18.py](../test/suite/test_timestamp18.py) +|Transactions|Timestamps|[test_txn26.py](../test/suite/test_txn26.py) +|Truncate||[test_truncate01.py](../test/suite/test_truncate01.py) +|Truncate|Prepare|[test_prepare13.py](../test/suite/test_prepare13.py) +|Verify|Prepare|[test_prepare_hs03.py](../test/suite/test_prepare_hs03.py), [test_timestamp18.py](../test/suite/test_timestamp18.py) +|Wt Util||[test_backup01.py](../test/suite/test_backup01.py), [test_dump.py](../test/suite/test_dump.py), [test_util11.py](../test/suite/test_util11.py) diff --git a/src/third_party/wiredtiger/test/utility/test_util.h b/src/third_party/wiredtiger/test/utility/test_util.h index 6689227d359..24e37e5acb9 100644 --- a/src/third_party/wiredtiger/test/utility/test_util.h +++ b/src/third_party/wiredtiger/test/utility/test_util.h @@ -102,31 +102,31 @@ typedef struct { * testutil_assert -- * Complain and quit if something isn't true. */ -#define testutil_assert(a) \ - do { \ - if (!(a)) \ - testutil_die(0, "%s/%d: %s", __func__, __LINE__, #a); \ +#define testutil_assert(a) \ + do { \ + if (!(a)) \ + testutil_die(0, "%s/%d: %s", __PRETTY_FUNCTION__, __LINE__, #a); \ } while (0) /* * testutil_assertfmt -- * Complain and quit if something isn't true. */ -#define testutil_assertfmt(a, fmt, ...) \ - do { \ - if (!(a)) \ - testutil_die(0, "%s/%d: %s: " fmt, __func__, __LINE__, #a, __VA_ARGS__); \ +#define testutil_assertfmt(a, fmt, ...) \ + do { \ + if (!(a)) \ + testutil_die(0, "%s/%d: %s: " fmt, __PRETTY_FUNCTION__, __LINE__, #a, __VA_ARGS__); \ } while (0) /* * testutil_check -- * Complain and quit if a function call fails. */ -#define testutil_check(call) \ - do { \ - int __r; \ - if ((__r = (call)) != 0) \ - testutil_die(__r, "%s/%d: %s", __func__, __LINE__, #call); \ +#define testutil_check(call) \ + do { \ + int __r; \ + if ((__r = (call)) != 0) \ + testutil_die(__r, "%s/%d: %s", __PRETTY_FUNCTION__, __LINE__, #call); \ } while (0) /* @@ -134,10 +134,10 @@ typedef struct { * Complain and quit if a function call fails, returning errno. The error test must be * specified, not just the call, because system calls fail in a variety of ways. */ -#define testutil_checksys(call) \ - do { \ - if (call) \ - testutil_die(errno, "%s/%d: %s", __func__, __LINE__, #call); \ +#define testutil_checksys(call) \ + do { \ + if (call) \ + testutil_die(errno, "%s/%d: %s", __PRETTY_FUNCTION__, __LINE__, #call); \ } while (0) /* @@ -148,7 +148,8 @@ typedef struct { do { \ int __r; \ if ((__r = (call)) != 0) \ - testutil_die(__r, "%s/%d: %s: " fmt, __func__, __LINE__, #call, __VA_ARGS__); \ + testutil_die( \ + __r, "%s/%d: %s: " fmt, __PRETTY_FUNCTION__, __LINE__, #call, __VA_ARGS__); \ } while (0) /* @@ -156,11 +157,11 @@ typedef struct { * Complain and quit if a function call fails. A special name because it appears in the * documentation. Allow any non-negative values. */ -#define error_sys_check(call) \ - do { \ - int __r; \ - if ((__r = (int)(call)) < 0 && __r != ENOTSUP) \ - testutil_die(__r, "%s/%d: %s", __func__, __LINE__, #call); \ +#define error_sys_check(call) \ + do { \ + int __r; \ + if ((__r = (int)(call)) < 0 && __r != ENOTSUP) \ + testutil_die(__r, "%s/%d: %s", __PRETTY_FUNCTION__, __LINE__, #call); \ } while (0) /* @@ -169,11 +170,11 @@ typedef struct { * documentation. Ignore ENOTSUP to allow library calls which might not be included in any * particular build. */ -#define error_check(call) \ - do { \ - int __r; \ - if ((__r = (call)) != 0 && __r != ENOTSUP) \ - testutil_die(__r, "%s/%d: %s", __func__, __LINE__, #call); \ +#define error_check(call) \ + do { \ + int __r; \ + if ((__r = (call)) != 0 && __r != ENOTSUP) \ + testutil_die(__r, "%s/%d: %s", __PRETTY_FUNCTION__, __LINE__, #call); \ } while (0) /* |