diff options
| author | Luke Chen <luke.chen@mongodb.com> | 2021-07-20 15:49:04 +1000 |
|---|---|---|
| committer | Evergreen Agent <no-reply@evergreen.mongodb.com> | 2021-07-20 07:16:16 +0000 |
| commit | 913cb830f1384da97f7e5c59723780849cdf0f71 (patch) | |
| tree | 09341fedfe75c50abf3f2bdfa8485f7b2433e0ba /src/third_party/wiredtiger/test/suite | |
| parent | 9f88f92a2d8aa1a6bff8180e9ae735d437ecd00b (diff) | |
| download | mongo-913cb830f1384da97f7e5c59723780849cdf0f71.tar.gz | |
Import wiredtiger: 87de78247074b291bbcf79c15d2a215af7cc2f7e from branch mongodb-5.0
ref: cca84e5abd..87de782470
for: 5.0.2
WT-6280 Fail eviction if out of order handling races with checkpoint
WT-6729 Quiesce eviction prior running rollback to stable's active transaction check
WT-6782 test_prepare_hs02 WT_ROLLBACK failure: conflict between concurrent operations
WT-7279 Allow multiple terminate calls for storage source extension
WT-7317 Improve the runtime_monitor component to track history store cleanup statistics
WT-7338 Copy the configuration directory when building the test framework
WT-7343 Write a script that executes many-collection-test.py
WT-7447 Fix the assert fire because onpage out of order update is not popped from the stack
WT-7473 Resolve "TODO: tiered" comments in code
WT-7507 Update salvage for a history store and timestamp world
WT-7520 Add start and stop values to automatic flag generation code
WT-7524 Refactor functions to obtain checkpoint list; Clear delete on skipping checkpoints
WT-7539 Add a configuration option that allows the user to specify debug modes.
WT-7543 Pass in the correct test name when constructing the default config path
WT-7552 Add UBSAN to automated WT testing
WT-7553 Loosen the restrictions around evicting fast-truncate pages to avoid cache-stuck failures
WT-7556 Fix test_rollback_to_stable10 failure cache_hs_ondisk is 0
WT-7591 Fixes to allow cursors to be open during flush_tier
WT-7603 Sort statistics to fix JSON output
WT-7605 Drop support for million-collection-test
WT-7609 Report on time taken to start and shutdown the database in many-coll-test
WT-7616 Create a pass or fail test based on many collection workgen workload
WT-7619 Add a new optimization to skip pages in cursor traversal when all entries on the page are deleted
WT-7626 We only ensure update restore eviction happened in test debug mode09
WT-7628 Return an error message when supplied invalid command line args in the test framework
WT-7629 Run clang format on .cxx files in the codebase.
WT-7632 Fix invalid argument in test_rollback_to_stable14
WT-7636 Fix leaked cursors by implementing scoped cursor and session types
WT-7640 Fix test_backup02 failure where checkpoint tables differ due to checkpoint cursor not supported
WT-7660 Rename poc_test to base_test in the cpp test framework and add insert_operation logic
WT-7665 Apply op tracking inserts in the workload transaction
WT-7667 Fix workgen JSON output
WT-7668 Overload the update method for the hs_cleanup test
WT-7670 Modify test tag format and tag additional python tests
WT-7675 Query last ckpt timestamp changes without taking checkpoint
WT-7676 Reformat wtperf backup to only read in files instead of wt_copy_and_sync
WT-7679 Create an evergreen test for many-dhandle-stress
WT-7683 Add python test hook to call flush_tier() during connection.close()
WT-7687 Stop tiered manager thread before internal thread
WT-7689 Fix double free in `__curhs_insert`
WT-7690 Fix coverity error when enum is compared against 0 (incr_backup:table_changes)
WT-7692 fix make check test failure on osx10 14 cmake
WT-7696 Fix coverity error - Unused variable in _rollback_to_stable_btree_apply_all
WT-7698 Decrease max_latency value in many dhandles scenario for workgen
WT-7699 Fix RTS handling to abort an out of order prepared transaction
WT-7705 Add an assert to ensure that there are no updates on the new disk image in update restore
WT-7706 Use same transaction update when on-disk value is an aborted prepared update
WT-7707 Simplify insert splits to use the splitting WT_REFs key
WT-7708 Add an assert to ensure the durable timestamp is larger than stable timestamp at the end of prepared commit
WT-7710 Fix to use history store btree to initialise history store cursor
WT-7715 Fix uninitialized bool in txn_ckpt.c
WT-7717 Change macros in local_store.c to use WT namespace
WT-7719 Change default value of ENABLE_STRICT to "OFF" (CMake Usability Improvements)
WT-7720 Update POSIX CMAKE doxygen documentation (CMake Usability Improvements)
WT-7721 Update test-format to reopen an existing database with different config
WT-7723 Delete the updates in the history store if they are rolled back or is the first stable update on the update chain
WT-7724 Fix race when running concurrent checkpoint and flush_tier
WT-7725 Add missing brackets around parameter in macro definition
WT-7726 Separating out the validation portion from the database model
WT-7727 Fix null pointer passed to memcpy() during 'format' test.
WT-7729 Fix to write out correct tiered information on checkpoint
WT-7730 Shifting the oldest and stable timestamps to match the commit timestamp format
WT-7739 Switch back to using MacOS 10.14 for Evergreen compile task
WT-7741 Fix misaligned address in crc32-x86.c
WT-7742 Fix misaligned address in wt3184_dup_index_collator/main.c
WT-7743 Fix integer overflow within wt2999_join_extractor csuite test
WT-7744 Fix null pointer within wt3338_partial_update csuite
WT-7746 Improve directory syncing with CMake helper 'create_test_executable'
WT-7748 Fix CMake library probes for libraries not on the default linker path
WT-7749 Assorted fixes for (and from) building and testing on NetBSD
WT-7751 Add an assert to ensure we never select an update that has been written to the history store for data store
WT-7752 Update packing code according to documentation
WT-7754 Fix race when updating block manager file handle
WT-7755 YSCB: Add a native implementation of YCSB to WTPERF.
WT-7756 RTS to clear the HS flag of an update following tombstone
WT-7760 Support array parsing in CppSuite config handling
WT-7761 Improve debug_print to include timestamp, thread_id and reorder args in cppsuite.
WT-7762 Create stressful configs for the two tests add them to evergreen.
WT-7763 Coverity issues found in the stress testing framework
WT-7765 Fix signed integer overflow in intpack-test3
WT-7766 Fix null pointer passed to memset in test_wt3338_partial_update
WT-7767 Code cleanup for curhs_remove and __txn_fixup_prepared_update
WT-7770 Fix issue linking TCMalloc in CMake
WT-7776 Add a hard limit on the number of modify updates before we instantiate a complete update
WT-7778 Fix null dereferencing, and return of incorrect allocation size
WT-7780 Guarantee log message sequencing in the test framework.
WT-7781 Avoid skipping python tests for 3rd party ext libraries in CMake builds
WT-7782 Separate implementation from headers in cppsuite test framework
WT-7783 Fix RTS to restore tombstone when an on-disk update is out of order prepare update
WT-7784 Enable RTS to use checkpoint snapshot on timestamp tables
WT-7795 Fix CppSuite failure "expected ) before PRIxMAX"
WT-7796 Scan the tracking table and delete parts of it that are obsolete.
WT-7797 Disable postrun stats in CppSuite testing
WT-7799 Do not report failure of wiredtiger_open in python tests to output
WT-7802 Remove data store same transaction update squash logic
WT-7804 Fix test_hs24 committing mixed mode update from the wrong value
WT-7807 Remove unused arg in backup_config
WT-7811 Fix test_hs24 not commiting from 0 timestamp
WT-7813 Stop inserting to history store if we see a prepared update
WT-7815 Properly initialize prev_upd_ts for ordered timestamp assertion
WT-7825 Fix test_hs24 key order
WT-7828 Move many-coll-test to ubuntu1804-wt-large and update thresholds accordingly
WT-7831 Clear transaction ids from previous run when repack the cell
WT-7832 Add an encryptor extension that uses the libsodium cryptography library.
WT-7836 Fixing a number of small issues in the cppsuite test framework
WT-7837 Clear updates structure in wt_hs_insert_updates to avoid firing assert
WT-7841 add "only unencrypted" checksum configuration, switch checksum default to "on"
WT-7843 Add missing macros to define PRIxMAX
WT-7846 Disable test_tiered08
WT-7851 Fix illegal checksum configuration in compatibility-test-for-newer-releases
WT-7852 Don't release eviction server lock when evicting pages
WT-7856 Enable flush test with open cursor in test_tiered04
Reverted ticket(s):
WT-7443 Add error message when bulk cursor can't get exclusive access to dhandle
Diffstat (limited to 'src/third_party/wiredtiger/test/suite')
77 files changed, 1649 insertions, 296 deletions
diff --git a/src/third_party/wiredtiger/test/suite/hook_tiered.py b/src/third_party/wiredtiger/test/suite/hook_tiered.py index cb7a44608ab..58ccc505416 100755 --- a/src/third_party/wiredtiger/test/suite/hook_tiered.py +++ b/src/third_party/wiredtiger/test/suite/hook_tiered.py @@ -87,6 +87,15 @@ def wiredtiger_open_tiered(ignored_self, args): return args +# Called to replace Connection.close +# Insert a call to flush_tier before closing connection. +def connection_close_replace(orig_connection_close, connection_self, config): + s = connection_self.open_session(None) + s.flush_tier(None) + s.close() + ret = orig_connection_close(connection_self, config) + return ret + # Called to replace Session.alter def session_alter_replace(orig_session_alter, session_self, uri, config): # Alter isn't implemented for tiered tables. Only call it if this can't be the uri @@ -207,14 +216,14 @@ class TieredHookCreator(wthooks.WiredTigerHookCreator): return new_tests def setup_hooks(self): + orig_connection_close = self.Connection['close'] + self.Connection['close'] = (wthooks.HOOK_REPLACE, lambda s, config: + connection_close_replace(orig_connection_close, s, config)) + orig_session_alter = self.Session['alter'] self.Session['alter'] = (wthooks.HOOK_REPLACE, lambda s, uri, config: session_alter_replace(orig_session_alter, s, uri, config)) - orig_session_close = self.Session['close'] - self.Session['close'] = (wthooks.HOOK_REPLACE, lambda s, config=None: - session_close_replace(orig_session_close, s, config)) - orig_session_compact = self.Session['compact'] self.Session['compact'] = (wthooks.HOOK_REPLACE, lambda s, uri, config: session_compact_replace(orig_session_compact, s, uri, config)) diff --git a/src/third_party/wiredtiger/test/suite/test_backup01.py b/src/third_party/wiredtiger/test/suite/test_backup01.py index 9a7141d29ff..aa30471706e 100644 --- a/src/third_party/wiredtiger/test/suite/test_backup01.py +++ b/src/third_party/wiredtiger/test/suite/test_backup01.py @@ -27,7 +27,8 @@ # OTHER DEALINGS IN THE SOFTWARE. # # [TEST_TAGS] -# backup:correctness:full_backup +# wt_util +# backup:cursors # [END_TAGS] # diff --git a/src/third_party/wiredtiger/test/suite/test_backup11.py b/src/third_party/wiredtiger/test/suite/test_backup11.py index e15f1b01dd7..14023cb6ee3 100644 --- a/src/third_party/wiredtiger/test/suite/test_backup11.py +++ b/src/third_party/wiredtiger/test/suite/test_backup11.py @@ -25,6 +25,10 @@ # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. +# +# [TEST_TAGS] +# backup:cursors +# [END_TAGS] import wiredtiger, wttest import os, shutil diff --git a/src/third_party/wiredtiger/test/suite/test_base02.py b/src/third_party/wiredtiger/test/suite/test_base02.py index 7d272bb2aaf..9fb3fa0d82c 100644 --- a/src/third_party/wiredtiger/test/suite/test_base02.py +++ b/src/third_party/wiredtiger/test/suite/test_base02.py @@ -26,6 +26,10 @@ # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. # +# [TEST_TAGS] +# config_api +# [END_TAGS] +# # test_base02.py # Configuration # diff --git a/src/third_party/wiredtiger/test/suite/test_bug004.py b/src/third_party/wiredtiger/test/suite/test_bug004.py index cce2968e826..8af470d5161 100644 --- a/src/third_party/wiredtiger/test/suite/test_bug004.py +++ b/src/third_party/wiredtiger/test/suite/test_bug004.py @@ -26,6 +26,10 @@ # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. # +# [TEST_TAGS] +# reconciliation:overflow_keys +# [END_TAGS] +# # test_bug004.py # Regression tests. diff --git a/src/third_party/wiredtiger/test/suite/test_bug005.py b/src/third_party/wiredtiger/test/suite/test_bug005.py index 3e4a2d9bb3f..a74ea0a51ed 100644 --- a/src/third_party/wiredtiger/test/suite/test_bug005.py +++ b/src/third_party/wiredtiger/test/suite/test_bug005.py @@ -26,6 +26,10 @@ # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. # +# [TEST_TAGS] +# session_api:verify +# [END_TAGS] +# # test_bug005.py # Regression tests. diff --git a/src/third_party/wiredtiger/test/suite/test_bug008.py b/src/third_party/wiredtiger/test/suite/test_bug008.py index aada1b6c195..5b04cbffb8a 100644 --- a/src/third_party/wiredtiger/test/suite/test_bug008.py +++ b/src/third_party/wiredtiger/test/suite/test_bug008.py @@ -26,8 +26,13 @@ # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. # -# test_bug008.py -# Regression tests. +# [TEST_TAGS] +# cursors:search_near +# cursors:search +# [END_TAGS] +# +# test_bug008.py +# Regression tests for cursor search and cursor search near. import wiredtiger, wttest from wtdataset import SimpleDataSet diff --git a/src/third_party/wiredtiger/test/suite/test_bug024.py b/src/third_party/wiredtiger/test/suite/test_bug024.py index 661a35617c4..53e09543435 100644 --- a/src/third_party/wiredtiger/test/suite/test_bug024.py +++ b/src/third_party/wiredtiger/test/suite/test_bug024.py @@ -25,6 +25,10 @@ # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. +# +# [TEST_TAGS] +# connection_api:turtle_file +# [END_TAGS] from helper import copy_wiredtiger_home import wiredtiger, wttest diff --git a/src/third_party/wiredtiger/test/suite/test_checkpoint02.py b/src/third_party/wiredtiger/test/suite/test_checkpoint02.py index 66196ebe5f5..cf02983200e 100755 --- a/src/third_party/wiredtiger/test/suite/test_checkpoint02.py +++ b/src/third_party/wiredtiger/test/suite/test_checkpoint02.py @@ -27,7 +27,7 @@ # OTHER DEALINGS IN THE SOFTWARE. # # [TEST_TAGS] -# checkpoints:correctness:checkpoint_data +# checkpoint # [END_TAGS] # diff --git a/src/third_party/wiredtiger/test/suite/test_checkpoint03.py b/src/third_party/wiredtiger/test/suite/test_checkpoint03.py index 158a768f44d..0f4fba34186 100644 --- a/src/third_party/wiredtiger/test/suite/test_checkpoint03.py +++ b/src/third_party/wiredtiger/test/suite/test_checkpoint03.py @@ -27,7 +27,7 @@ # OTHER DEALINGS IN THE SOFTWARE. # # [TEST_TAGS] -# checkpoints:correctness:checkpoint_data +# checkpoint:history_store # [END_TAGS] # # test_checkpoint03.py diff --git a/src/third_party/wiredtiger/test/suite/test_checkpoint08.py b/src/third_party/wiredtiger/test/suite/test_checkpoint08.py index f6ea1a1c888..5a8e8fe647e 100755 --- a/src/third_party/wiredtiger/test/suite/test_checkpoint08.py +++ b/src/third_party/wiredtiger/test/suite/test_checkpoint08.py @@ -26,6 +26,10 @@ # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. # +# [TEST_TAGS] +# checkpoint:obsolete_data +# [END_TAGS] +# # test_checkpoint08.py # Test that the btree checkpoint is not skipped if there are obsolete pages. diff --git a/src/third_party/wiredtiger/test/suite/test_checkpoint_snapshot01.py b/src/third_party/wiredtiger/test/suite/test_checkpoint_snapshot01.py index 1f549bc7138..ce839469ed1 100644 --- a/src/third_party/wiredtiger/test/suite/test_checkpoint_snapshot01.py +++ b/src/third_party/wiredtiger/test/suite/test_checkpoint_snapshot01.py @@ -25,6 +25,10 @@ # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. +# +# [TEST_TAGS] +# checkpoint:metadata +# [END_TAGS] from helper import copy_wiredtiger_home import wiredtiger, wttest diff --git a/src/third_party/wiredtiger/test/suite/test_checkpoint_snapshot02.py b/src/third_party/wiredtiger/test/suite/test_checkpoint_snapshot02.py index 6e6edadd18e..2f93fef82b2 100644 --- a/src/third_party/wiredtiger/test/suite/test_checkpoint_snapshot02.py +++ b/src/third_party/wiredtiger/test/suite/test_checkpoint_snapshot02.py @@ -28,8 +28,8 @@ import fnmatch, os, shutil, threading, time from wtthread import checkpoint_thread, op_thread -from helper import copy_wiredtiger_home -import wiredtiger, wttest +from helper import simulate_crash_restart +import wttest from wtdataset import SimpleDataSet from wtscenario import make_scenarios from wiredtiger import stat @@ -47,23 +47,36 @@ class test_checkpoint_snapshot02(wttest.WiredTigerTestCase): uri = "table:test_checkpoint_snapshot02" nrows = 1000 + key_format_values = [ + ('column', dict(key_format='r')), + ('integer_row', dict(key_format='i')), + ] + + scenarios = make_scenarios(key_format_values) + def conn_config(self): config = 'cache_size=10MB,statistics=(all),statistics_log=(json,on_close,wait=1),log=(enabled=true),timing_stress_for_test=[checkpoint_slow]' return config - def large_updates(self, uri, value, ds, nrows): + def large_updates(self, uri, value, ds, nrows, commit_ts): # Update a large number of records. session = self.session cursor = session.open_cursor(uri) - for i in range(0, nrows): + for i in range(1, nrows+1): session.begin_transaction() cursor[ds.key(i)] = value - session.commit_transaction() + if commit_ts == 0: + session.commit_transaction() + else: + session.commit_transaction('commit_timestamp=' + timestamp_str(commit_ts)) cursor.close() - def check(self, check_value, uri, nrows): + def check(self, check_value, uri, nrows, read_ts): session = self.session - session.begin_transaction() + if read_ts == 0: + session.begin_transaction() + else: + session.begin_transaction('read_timestamp=' + timestamp_str(read_ts)) cursor = session.open_cursor(uri) count = 0 for k, v in cursor: @@ -74,26 +87,74 @@ class test_checkpoint_snapshot02(wttest.WiredTigerTestCase): def test_checkpoint_snapshot(self): + ds = SimpleDataSet(self, self.uri, 0, key_format=self.key_format, value_format="S",config='log=(enabled=false)') + ds.populate() + valuea = "aaaaa" * 100 + + self.large_updates(self.uri, valuea, ds, self.nrows, 0) + self.check(valuea, self.uri, self.nrows, 0) + + session1 = self.conn.open_session() + session1.begin_transaction() + cursor1 = session1.open_cursor(self.uri) + + for i in range(self.nrows+1, (self.nrows*2)+1): + cursor1.set_key(ds.key(i)) + cursor1.set_value(valuea) + self.assertEqual(cursor1.insert(), 0) + + # Create a checkpoint thread + done = threading.Event() + ckpt = checkpoint_thread(self.conn, done) + try: + ckpt.start() + # Sleep for sometime so that checkpoint starts before committing last transaction. + time.sleep(2) + session1.commit_transaction() + + finally: + done.set() + ckpt.join() + + #Simulate a crash by copying to a new directory(RESTART). + simulate_crash_restart(self, ".", "RESTART") + + # Check the table contains the last checkpointed value. + self.check(valuea, self.uri, self.nrows, 0) + + stat_cursor = self.session.open_cursor('statistics:', None, None) + inconsistent_ckpt = stat_cursor[stat.conn.txn_rts_inconsistent_ckpt][2] + keys_removed = stat_cursor[stat.conn.txn_rts_keys_removed][2] + stat_cursor.close() + + self.assertGreater(inconsistent_ckpt, 0) + self.assertGreaterEqual(keys_removed, 0) + + def test_checkpoint_snapshot_with_timestamp(self): + ds = SimpleDataSet(self, self.uri, 0, key_format="S", value_format="S",config='log=(enabled=false)') ds.populate() valuea = "aaaaa" * 100 - valueb = "bbbbb" * 100 - valuec = "ccccc" * 100 - valued = "ddddd" * 100 - cursor = self.session.open_cursor(self.uri) - self.large_updates(self.uri, valuea, ds, self.nrows) + # Pin oldest and stable timestamps to 10. + self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(10) + + ',stable_timestamp=' + timestamp_str(10)) - self.check(valuea, self.uri, self.nrows) + self.large_updates(self.uri, valuea, ds, self.nrows, 20) + self.check(valuea, self.uri, self.nrows, 20) session1 = self.conn.open_session() session1.begin_transaction() cursor1 = session1.open_cursor(self.uri) - for i in range(self.nrows, self.nrows*2): + for i in range(self.nrows+1, (self.nrows*2)+1): cursor1.set_key(ds.key(i)) cursor1.set_value(valuea) self.assertEqual(cursor1.insert(), 0) + session1.timestamp_transaction('commit_timestamp=' + timestamp_str(30)) + + # Set stable timestamp to 40 + self.conn.set_timestamp('stable_timestamp=' + timestamp_str(40)) # Create a checkpoint thread done = threading.Event() @@ -109,28 +170,88 @@ class test_checkpoint_snapshot02(wttest.WiredTigerTestCase): ckpt.join() #Simulate a crash by copying to a new directory(RESTART). - copy_wiredtiger_home(self, ".", "RESTART") + simulate_crash_restart(self, ".", "RESTART") + + # Check the table contains the last checkpointed value. + self.check(valuea, self.uri, self.nrows, 30) + + stat_cursor = self.session.open_cursor('statistics:', None, None) + inconsistent_ckpt = stat_cursor[stat.conn.txn_rts_inconsistent_ckpt][2] + keys_removed = stat_cursor[stat.conn.txn_rts_keys_removed][2] + stat_cursor.close() + + self.assertGreater(inconsistent_ckpt, 0) + self.assertGreaterEqual(keys_removed, 0) + + def test_checkpoint_snapshot_with_txnid_and_timestamp(self): + + ds = SimpleDataSet(self, self.uri, 0, key_format="S", value_format="S",config='log=(enabled=false)') + ds.populate() + valuea = "aaaaa" * 100 + valueb = "bbbbb" * 100 + + # Pin oldest and stable timestamps to 10. + self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(10) + + ',stable_timestamp=' + timestamp_str(10)) + + session1 = self.conn.open_session() + session1.begin_transaction() + + self.large_updates(self.uri, valuea, ds, self.nrows, 20) + self.check(valuea, self.uri, self.nrows, 20) + + session2 = self.conn.open_session() + session2.begin_transaction() + cursor2 = session2.open_cursor(self.uri) + + for i in range((self.nrows+1), (self.nrows*2)+1): + cursor2.set_key(ds.key(i)) + cursor2.set_value(valuea) + self.assertEqual(cursor2.insert(), 0) + session1.timestamp_transaction('commit_timestamp=' + timestamp_str(30)) - # Open the new directory. - self.conn = self.setUpConnectionOpen("RESTART") - self.session = self.setUpSessionOpen(self.conn) + # Set stable timestamp to 40 + self.conn.set_timestamp('stable_timestamp=' + timestamp_str(40)) + + # Create a checkpoint thread + done = threading.Event() + ckpt = checkpoint_thread(self.conn, done) + try: + ckpt.start() + # Sleep for sometime so that checkpoint starts before committing last transaction. + time.sleep(2) + session2.commit_transaction() + + finally: + done.set() + ckpt.join() + + session1.rollback_transaction() + #Simulate a crash by copying to a new directory(RESTART). + simulate_crash_restart(self, ".", "RESTART") # Check the table contains the last checkpointed value. - self.check(valuea, self.uri, self.nrows) + self.check(valuea, self.uri, self.nrows, 30) stat_cursor = self.session.open_cursor('statistics:', None, None) inconsistent_ckpt = stat_cursor[stat.conn.txn_rts_inconsistent_ckpt][2] keys_removed = stat_cursor[stat.conn.txn_rts_keys_removed][2] - keys_restored = stat_cursor[stat.conn.txn_rts_keys_restored][2] - pages_visited = stat_cursor[stat.conn.txn_rts_pages_visited][2] - upd_aborted = stat_cursor[stat.conn.txn_rts_upd_aborted][2] stat_cursor.close() self.assertGreater(inconsistent_ckpt, 0) - self.assertEqual(upd_aborted, 0) self.assertGreaterEqual(keys_removed, 0) - self.assertEqual(keys_restored, 0) - self.assertGreaterEqual(pages_visited, 0) + + simulate_crash_restart(self, "RESTART", "RESTART2") + # Check the table contains the last checkpointed value. + self.check(valuea, self.uri, self.nrows, 30) + + stat_cursor = self.session.open_cursor('statistics:', None, None) + inconsistent_ckpt = stat_cursor[stat.conn.txn_rts_inconsistent_ckpt][2] + keys_removed = stat_cursor[stat.conn.txn_rts_keys_removed][2] + stat_cursor.close() + + self.assertGreaterEqual(inconsistent_ckpt, 0) + self.assertEqual(keys_removed, 0) if __name__ == '__main__': wttest.run() diff --git a/src/third_party/wiredtiger/test/suite/test_checkpoint_snapshot03.py b/src/third_party/wiredtiger/test/suite/test_checkpoint_snapshot03.py index 09357a95332..293b16efaf5 100644 --- a/src/third_party/wiredtiger/test/suite/test_checkpoint_snapshot03.py +++ b/src/third_party/wiredtiger/test/suite/test_checkpoint_snapshot03.py @@ -25,6 +25,10 @@ # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. +# +# [TEST_TAGS] +# rollback_to_stable +# [END_TAGS] import fnmatch, os, shutil, threading, time from wtthread import checkpoint_thread, op_thread diff --git a/src/third_party/wiredtiger/test/suite/test_config02.py b/src/third_party/wiredtiger/test/suite/test_config02.py index d4a9ac1fb84..f731c9a6315 100755 --- a/src/third_party/wiredtiger/test/suite/test_config02.py +++ b/src/third_party/wiredtiger/test/suite/test_config02.py @@ -25,6 +25,11 @@ # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. +# +# [TEST_TAGS] +# connection_api:wiredtiger_open +# config_api +# [END_TAGS] import os import wiredtiger, wttest diff --git a/src/third_party/wiredtiger/test/suite/test_cursor06.py b/src/third_party/wiredtiger/test/suite/test_cursor06.py index 015cbed959d..54ec1519a86 100644 --- a/src/third_party/wiredtiger/test/suite/test_cursor06.py +++ b/src/third_party/wiredtiger/test/suite/test_cursor06.py @@ -25,6 +25,10 @@ # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. +# +# [TEST_TAGS] +# cursors:reconfigure +# [END_TAGS] import wiredtiger, wttest from wtdataset import SimpleDataSet, ComplexDataSet, ComplexLSMDataSet diff --git a/src/third_party/wiredtiger/test/suite/test_cursor17.py b/src/third_party/wiredtiger/test/suite/test_cursor17.py new file mode 100644 index 00000000000..f068279f8f1 --- /dev/null +++ b/src/third_party/wiredtiger/test/suite/test_cursor17.py @@ -0,0 +1,120 @@ +#!/usr/bin/env python +# +# Public Domain 2014-2020 MongoDB, Inc. +# Public Domain 2008-2014 WiredTiger, Inc. +# +# This is free and unencumbered software released into the public domain. +# +# Anyone is free to copy, modify, publish, use, compile, sell, or +# distribute this software, either in source code form or as a compiled +# binary, for any purpose, commercial or non-commercial, and by any +# means. +# +# In jurisdictions that recognize copyright laws, the author or authors +# of this software dedicate any and all copyright interest in the +# software to the public domain. We make this dedication for the benefit +# of the public at large and to the detriment of our heirs and +# successors. We intend this dedication to be an overt act of +# relinquishment in perpetuity of all present and future rights to this +# software under copyright law. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +# IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. + +import wiredtiger, wttest +from wtscenario import make_scenarios +from wiredtiger import stat, WT_NOTFOUND + +def timestamp_str(t): + return '%x' % t + +# test_cursor17.py +# Test the cursor traversal optimization for delete heavy workloads. This optimization enables +# cursor traversal mechanism to skip pages where all records on the page are deleted with a +# tombstone visible to the current transaction. +class test_cursor17(wttest.WiredTigerTestCase): + conn_config = 'cache_size=50MB,statistics=(all)' + session_config = 'isolation=snapshot' + + def get_stat(self, stat, uri): + stat_string = 'statistics:' + if (uri): + stat_string += uri + stat_cursor = self.session.open_cursor(stat_string) + val = stat_cursor[stat][2] + stat_cursor.close() + return val + + def test_cursor_skip_pages(self): + uri = 'table:test_cursor17' + create_params = 'key_format=i,value_format=S' + self.session.create(uri, create_params) + + value1 = 'a' * 500 + value2 = 'b' * 500 + total_keys = 40000 + + # Keep the oldest and the stable timestamp pinned. + self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(1)) + self.conn.set_timestamp('stable_timestamp=' + timestamp_str(2)) + cursor = self.session.open_cursor(uri) + + commit_timestamp = 3 + + # Insert predefined number of key-value pairs. + for key in range(total_keys): + self.session.begin_transaction() + cursor[key] = value1 + self.session.commit_transaction('commit_timestamp=' + timestamp_str(commit_timestamp)) + commit_timestamp += 1 + + # Delete everything on the table except for the first and the last KV pair. + for key in range(1, total_keys - 1): + self.session.begin_transaction() + cursor.set_key(key) + self.assertEqual(cursor.remove(),0) + self.session.commit_transaction('commit_timestamp=' + timestamp_str(commit_timestamp)) + commit_timestamp += 1 + + # Take a checkpoint to reconcile the pages. + self.session.checkpoint() + + self.session.begin_transaction('read_timestamp=' + timestamp_str(commit_timestamp)) + # Position the cursor on the first record. + cursor.set_key(0) + self.assertEqual(cursor.search(), 0) + # This should move the cursor to the last record. + self.assertEqual(cursor.next(), 0) + self.assertEqual(cursor.get_key(), total_keys - 1) + + # Check if we skipped any pages while moving the cursor. + # + # We calculate the number of pages we expect to skip based on the total number of leaf pages + # reported in the WT stats. We subtract 2 from the count of leaf pages in the table and test + # that we atleast skipped 80% of the expected number of pages. + + leaf_pages_in_table = self.get_stat(stat.dsrc.btree_row_leaf, uri) + expected_pages_skipped = ((leaf_pages_in_table - 2) * 8) // 10 + skipped_pages = self.get_stat(stat.conn.cursor_next_skip_page_count, None) + self.assertGreater(skipped_pages, expected_pages_skipped) + self.session.rollback_transaction() + + # Update a key in the middle of the table. + self.session.begin_transaction() + cursor[total_keys // 2] = value2 + self.session.commit_transaction('commit_timestamp=' + timestamp_str(commit_timestamp)) + commit_timestamp += 1 + + # Make sure we can reach a the record we updated in the middle of the table. + self.session.begin_transaction('read_timestamp=' + timestamp_str(commit_timestamp)) + # Position the cursor on the first record. + cursor.set_key(0) + self.assertEqual(cursor.search(), 0) + # This should move the cursor to the record we updated in the middle. + self.assertEqual(cursor.next(), 0) + self.assertEqual(cursor.get_key(), total_keys // 2) diff --git a/src/third_party/wiredtiger/test/suite/test_debug_mode05.py b/src/third_party/wiredtiger/test/suite/test_debug_mode05.py index 4fd2c27dd5d..f322dd98f7a 100644 --- a/src/third_party/wiredtiger/test/suite/test_debug_mode05.py +++ b/src/third_party/wiredtiger/test/suite/test_debug_mode05.py @@ -44,12 +44,11 @@ class test_debug_mode05(wttest.WiredTigerTestCase): def test_table_logging_rollback_to_stable(self): self.session.create(self.uri, 'key_format=i,value_format=u,log=(enabled=false)') - cursor = self.session.open_cursor(self.uri, None) - self.conn.set_timestamp('stable_timestamp=' + timestamp_str(100)) self.session.checkpoint() # Try doing a normal prepared txn and then rollback to stable. + cursor = self.session.open_cursor(self.uri, None) self.session.begin_transaction() for i in range(1, 50): cursor[i] = b'a' * 100 @@ -60,6 +59,7 @@ class test_debug_mode05(wttest.WiredTigerTestCase): self.session.timestamp_transaction( 'durable_timestamp=' + timestamp_str(250)) self.session.commit_transaction() + cursor.close() self.conn.rollback_to_stable() @@ -84,10 +84,12 @@ class test_debug_mode05(wttest.WiredTigerTestCase): self.conn.rollback_to_stable() self.session.begin_transaction() + cursor = self.session.open_cursor(self.uri, None) for i in range(1, 50): cursor[i] = b'b' * 100 self.session.commit_transaction( 'commit_timestamp=' + timestamp_str(450)) + cursor.close() self.conn.rollback_to_stable() diff --git a/src/third_party/wiredtiger/test/suite/test_debug_mode09.py b/src/third_party/wiredtiger/test/suite/test_debug_mode09.py index a44e25ca7ce..9c86d876276 100755 --- a/src/third_party/wiredtiger/test/suite/test_debug_mode09.py +++ b/src/third_party/wiredtiger/test/suite/test_debug_mode09.py @@ -36,7 +36,7 @@ import wttest # to do an update restore evict on a page, when the cache pressure requirements are not met. # This means setting eviction target low and cache size high. class test_debug_mode09(wttest.WiredTigerTestCase): - conn_config = 'cache_size=10MB,statistics=(all),eviction_target=10' + conn_config = 'cache_size=10MB,statistics=(all),eviction_target=10,debug_mode=(update_restore_evict=true)' uri = "table:test_debug_mode09" # Insert a bunch of data to trigger eviction @@ -57,17 +57,3 @@ class test_debug_mode09(wttest.WiredTigerTestCase): pages_update_restored = stat_cursor[stat.conn.cache_write_restore][2] stat_cursor.close() self.assertGreater(pages_update_restored, 0) - - # Restart the connection with update restore evict config and a clean table - self.close_conn() - self.conn_config += ",debug_mode=(update_restore_evict=true)" - self.reopen_conn(".", self.conn_config) - self.session.drop(self.uri) - self.session.create(self.uri, 'key_format=i,value_format=S') - - self.trigger_eviction(self.uri) - - stat_cursor = self.session.open_cursor('statistics:') - forced_pages_update_restore = stat_cursor[stat.conn.cache_write_restore][2] - stat_cursor.close() - self.assertGreater(forced_pages_update_restore, pages_update_restored) diff --git a/src/third_party/wiredtiger/test/suite/test_dictionary.py b/src/third_party/wiredtiger/test/suite/test_dictionary.py index b5baf170278..9465827741c 100644 --- a/src/third_party/wiredtiger/test/suite/test_dictionary.py +++ b/src/third_party/wiredtiger/test/suite/test_dictionary.py @@ -26,6 +26,10 @@ # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. # +# [TEST_TAGS] +# compression +# [END_TAGS] +# # test_dictionary.py # Smoke test dictionary compression. diff --git a/src/third_party/wiredtiger/test/suite/test_dump.py b/src/third_party/wiredtiger/test/suite/test_dump.py index 20f863dba8e..bc94d668f72 100644 --- a/src/third_party/wiredtiger/test/suite/test_dump.py +++ b/src/third_party/wiredtiger/test/suite/test_dump.py @@ -25,6 +25,10 @@ # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. +# +# [TEST_TAGS] +# wt_util +# [END_TAGS] import os, shutil import wiredtiger, wttest diff --git a/src/third_party/wiredtiger/test/suite/test_encrypt01.py b/src/third_party/wiredtiger/test/suite/test_encrypt01.py index 3aee3ecd90d..94acd8aa433 100644 --- a/src/third_party/wiredtiger/test/suite/test_encrypt01.py +++ b/src/third_party/wiredtiger/test/suite/test_encrypt01.py @@ -37,6 +37,16 @@ from wtscenario import make_scenarios # Test basic encryption class test_encrypt01(wttest.WiredTigerTestCase): + # To test the sodium encryptor, we use secretkey= rather than + # setting a keyid, because for a "real" (vs. test-only) encryptor, + # keyids require some kind of key server, and (a) setting one up + # for testing would be a nuisance and (b) currently the sodium + # encryptor doesn't support any anyway. + # + # It expects secretkey= to provide a hex-encoded 256-bit chacha20 key. + # This key will serve for testing purposes. + sodium_testkey = '0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef' + types = [ ('file', dict(uri='file:test_encrypt01')), ('table', dict(uri='table:test_encrypt01')), @@ -49,7 +59,9 @@ class test_encrypt01(wttest.WiredTigerTestCase): ('rotn', dict( sys_encrypt='rotn', sys_encrypt_args=',keyid=11', file_encrypt='rotn', file_encrypt_args=',keyid=13')), ('rotn-none', dict( sys_encrypt='rotn', sys_encrypt_args=',keyid=9', - file_encrypt='none', file_encrypt_args='')) + file_encrypt='none', file_encrypt_args='')), + ('sodium', dict( sys_encrypt='sodium', sys_encrypt_args=',secretkey=' + sodium_testkey, + file_encrypt='sodium', file_encrypt_args='')) ] compress = [ ('none', dict(log_compress=None, block_compress=None)), diff --git a/src/third_party/wiredtiger/test/suite/test_encrypt02.py b/src/third_party/wiredtiger/test/suite/test_encrypt02.py index ada16e2f172..c678a5a21e6 100644 --- a/src/third_party/wiredtiger/test/suite/test_encrypt02.py +++ b/src/third_party/wiredtiger/test/suite/test_encrypt02.py @@ -26,6 +26,10 @@ # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. # +# [TEST_TAGS] +# encryption +# [END_TAGS] +# # test_encrypt02.py # Encryption using passwords # @@ -37,6 +41,16 @@ from wtscenario import make_scenarios # Test basic encryption class test_encrypt02(wttest.WiredTigerTestCase, suite_subprocess): + # To test the sodium encryptor, we use secretkey= rather than + # setting a keyid, because for a "real" (vs. test-only) encryptor, + # keyids require some kind of key server, and (a) setting one up + # for testing would be a nuisance and (b) currently the sodium + # encryptor doesn't support any anyway. + # + # It expects secretkey= to provide a hex-encoded 256-bit chacha20 key. + # This key will serve for testing purposes. + sodium_testkey = '0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef' + uri = 'file:test_encrypt02' encrypt_type = [ ('noarg', dict( encrypt_args='name=rotn', secret_arg=None)), @@ -44,6 +58,8 @@ class test_encrypt02(wttest.WiredTigerTestCase, suite_subprocess): ('pass', dict( encrypt_args='name=rotn', secret_arg='ABC')), ('keyid-pass', dict( encrypt_args='name=rotn,keyid=11', secret_arg='ABC')), + ('sodium-pass', dict( encrypt_args='name=sodium', secret_arg=sodium_testkey)), + # The other combinations for sodium, which are rejected, are checked in encrypt08. ] scenarios = make_scenarios(encrypt_type) @@ -51,6 +67,7 @@ class test_encrypt02(wttest.WiredTigerTestCase, suite_subprocess): # Load the compression extension, skip the test if missing extlist.skip_if_missing = True extlist.extension('encryptors', 'rotn') + extlist.extension('encryptors', 'sodium') nrecords = 5000 bigvalue = "abcdefghij" * 1001 # len(bigvalue) = 10010 diff --git a/src/third_party/wiredtiger/test/suite/test_encrypt06.py b/src/third_party/wiredtiger/test/suite/test_encrypt06.py index f3b83cb23eb..be428f5a3cf 100755 --- a/src/third_party/wiredtiger/test/suite/test_encrypt06.py +++ b/src/third_party/wiredtiger/test/suite/test_encrypt06.py @@ -36,9 +36,22 @@ from wtscenario import make_scenarios # Test encryption, when on, does not leak any information class test_encrypt06(wttest.WiredTigerTestCase): + # To test the sodium encryptor, we use secretkey= rather than + # setting a keyid, because for a "real" (vs. test-only) encryptor, + # keyids require some kind of key server, and (a) setting one up + # for testing would be a nuisance and (b) currently the sodium + # encryptor doesn't support any anyway. + # + # Note that secretkey= is apparently not allowed with per-table + # encryption, so we don't test that. + # + # It expects secretkey= to provide a hex-encoded 256-bit chacha20 key. + # This key will serve for testing purposes. + sodium_testkey = '0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef' key11 = ',keyid=11,secretkey=XYZ' key13 = ',keyid=13' + sodiumkey = ',secretkey=' + sodium_testkey # Test with various combinations of tables with or without indices # and column groups, also with LSM. When 'match' is False, we @@ -85,6 +98,10 @@ class test_encrypt06(wttest.WiredTigerTestCase): sys_encrypt='rotn', sys_encrypt_args=key11, table0_encrypt='rotn', table0_encrypt_args=key13, table1_encrypt='none', table1_encrypt_args='')), + ('sodium-implied', dict( + sys_encrypt='sodium', sys_encrypt_args=sodiumkey, + table0_encrypt=None, table0_encrypt_args='', + table1_encrypt=None, table1_encrypt_args='')), ] scenarios = make_scenarios(encrypt, storagetype) nrecords = 1000 diff --git a/src/third_party/wiredtiger/test/suite/test_encrypt08.py b/src/third_party/wiredtiger/test/suite/test_encrypt08.py new file mode 100644 index 00000000000..e15dd851908 --- /dev/null +++ b/src/third_party/wiredtiger/test/suite/test_encrypt08.py @@ -0,0 +1,94 @@ +#!/usr/bin/env python +# +# Public Domain 2014-present MongoDB, Inc. +# Public Domain 2008-2014 WiredTiger, Inc. +# +# This is free and unencumbered software released into the public domain. +# +# Anyone is free to copy, modify, publish, use, compile, sell, or +# distribute this software, either in source code form or as a compiled +# binary, for any purpose, commercial or non-commercial, and by any +# means. +# +# In jurisdictions that recognize copyright laws, the author or authors +# of this software dedicate any and all copyright interest in the +# software to the public domain. We make this dedication for the benefit +# of the public at large and to the detriment of our heirs and +# successors. We intend this dedication to be an overt act of +# relinquishment in perpetuity of all present and future rights to this +# software under copyright law. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +# IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. +# +# test_encrypt08.py +# Test some error conditions with the libsodium encryption extension. +# + +import os, run, random +import wiredtiger, wttest +from wtscenario import make_scenarios + +# +# Test sodium encryption configuration. +# This exercises the error paths in the encryptor's customize method when +# used for system (not per-table) encryption. +# +class test_encrypt08(wttest.WiredTigerTestCase): + uri = 'file:test_encrypt08' + + # To test the sodium encryptor, we use secretkey= rather than + # setting a keyid, because for a "real" (vs. test-only) encryptor, + # keyids require some kind of key server, and (a) setting one up + # for testing would be a nuisance and (b) currently the sodium + # encryptor doesn't support any anyway. + # + # It expects secretkey= to provide a hex-encoded 256-bit chacha20 key. + # This key will serve for testing purposes. + sodium_testkey = '0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef' + + encrypt_type = [ + ('nokey', dict( sys_encrypt='', + msg='/no key given/')), + ('keyid', dict( sys_encrypt='keyid=123', + msg='/keyids not supported/')), + ('twokeys', dict( sys_encrypt='keyid=123,secretkey=' + sodium_testkey, + msg='/keys specified with both/')), + ('nothex', dict( sys_encrypt='secretkey=plop', + msg='/secret key not hex/')), + ('badsize', dict( sys_encrypt='secretkey=0123456789abcdef', + msg='/wrong secret key length/')), + ] + scenarios = make_scenarios(encrypt_type) + + def conn_extensions(self, extlist): + extlist.skip_if_missing = True + extlist.extension('encryptors', 'sodium') + + # Do not use conn_config to set the encryption, because that sets + # the encryption during open when we don't have control and can't + # catch exceptions. Instead we'll let the frameork open without + # encryption and then reopen ourselves. This seems to behave as + # desired (we get the intended errors from inside the encryptor) + # even though one might expect it to fail because it's reopening + # the database with different encryption. (If in the future it starts + # doing that, the workaround is to override setUpConnectionOpen. + # I'm not doing that now because it's quite a bit messier.) + + # (Re)open the database with bad encryption config. + def test_encrypt(self): + sysconfig = 'encryption=(name=sodium,{0}),'.format(self.sys_encrypt) + + with self.expectedStdoutPattern('Failed wiredtiger_open'): + self.assertRaisesWithMessage(wiredtiger.WiredTigerError, + lambda: + self.reopen_conn(config = sysconfig), + self.msg) + +if __name__ == '__main__': + wttest.run() diff --git a/src/third_party/wiredtiger/test/suite/test_encrypt09.py b/src/third_party/wiredtiger/test/suite/test_encrypt09.py new file mode 100644 index 00000000000..4232ab96a5f --- /dev/null +++ b/src/third_party/wiredtiger/test/suite/test_encrypt09.py @@ -0,0 +1,94 @@ +#!/usr/bin/env python +# +# Public Domain 2014-present MongoDB, Inc. +# Public Domain 2008-2014 WiredTiger, Inc. +# +# This is free and unencumbered software released into the public domain. +# +# Anyone is free to copy, modify, publish, use, compile, sell, or +# distribute this software, either in source code form or as a compiled +# binary, for any purpose, commercial or non-commercial, and by any +# means. +# +# In jurisdictions that recognize copyright laws, the author or authors +# of this software dedicate any and all copyright interest in the +# software to the public domain. We make this dedication for the benefit +# of the public at large and to the detriment of our heirs and +# successors. We intend this dedication to be an overt act of +# relinquishment in perpetuity of all present and future rights to this +# software under copyright law. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +# IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. +# +# test_encrypt09.py +# Test some error conditions with the libsodium encryption extension. +# + +import os, run, random +import wiredtiger, wttest +from wtscenario import make_scenarios + +# +# Test sodium encryption configuration. +# This exercises the error paths in the encryptor's customize method when +# used for per-table encryption. +# +class test_encrypt09(wttest.WiredTigerTestCase): + uri = 'file:test_encrypt09' + + # To test the sodium encryptor, we use secretkey= rather than + # setting a keyid, because for a "real" (vs. test-only) encryptor, + # keyids require some kind of key server, and (a) setting one up + # for testing would be a nuisance and (b) currently the sodium + # encryptor doesn't support any anyway. + # + # It expects secretkey= to provide a hex-encoded 256-bit chacha20 key. + # This key will serve for testing purposes. + sodium_testkey = '0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef' + + # The nokey case is not an error; if no key is given no separate encryptor is + # generated and no error occurs. + + # Note that the twokeys, nothex, and badsize cases do not (currently) get to + # the extension at all because (apparently) secretkey= is not allowed for + # per-table encryption. + + encrypt_type = [ + ('nokey', dict( file_encrypt='', + msg=None)), + ('keyid', dict( file_encrypt='keyid=123', + msg='/keyids not supported/')), + ('twokeys', dict( file_encrypt='keyid=123,secretkey=' + sodium_testkey, + msg='/unknown configuration key: .secretkey.:/')), + ('nothex', dict( file_encrypt='secretkey=plop', + msg='/unknown configuration key: .secretkey.:/')), + ('badsize', dict( file_encrypt='secretkey=0123456789abcdef', + msg='/unknown configuration key: .secretkey.:/')), + ] + scenarios = make_scenarios(encrypt_type) + + def conn_extensions(self, extlist): + extlist.skip_if_missing = True + extlist.extension('encryptors', 'sodium') + + def conn_config(self): + return 'encryption=(name=sodium,secretkey={0}),'.format(self.sodium_testkey) + + # Create a table with encryption values that are in error. + def test_encrypt(self): + params = 'key_format=S,value_format=S,encryption=(name=sodium,' + self.file_encrypt + ')' + + if self.msg is None: + self.session.create(self.uri, params) + else: + self.assertRaisesWithMessage(wiredtiger.WiredTigerError, lambda: + self.session.create(self.uri, params), self.msg) + +if __name__ == '__main__': + wttest.run() diff --git a/src/third_party/wiredtiger/test/suite/test_gc01.py b/src/third_party/wiredtiger/test/suite/test_gc01.py index 9ca16a9c961..950c1e4ce54 100755 --- a/src/third_party/wiredtiger/test/suite/test_gc01.py +++ b/src/third_party/wiredtiger/test/suite/test_gc01.py @@ -25,6 +25,10 @@ # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. +# +# [TEST_TAGS] +# checkpoint:garbage_collection +# [END_TAGS] import time from helper import copy_wiredtiger_home diff --git a/src/third_party/wiredtiger/test/suite/test_hs09.py b/src/third_party/wiredtiger/test/suite/test_hs09.py index a1808067a92..4217da2b02d 100644 --- a/src/third_party/wiredtiger/test/suite/test_hs09.py +++ b/src/third_party/wiredtiger/test/suite/test_hs09.py @@ -25,6 +25,10 @@ # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. +# +# [TEST_TAGS] +# history_store +# [END_TAGS] import wiredtiger, wttest from wiredtiger import stat diff --git a/src/third_party/wiredtiger/test/suite/test_hs15.py b/src/third_party/wiredtiger/test/suite/test_hs15.py index 5c14f064a4e..7d27d166a70 100644 --- a/src/third_party/wiredtiger/test/suite/test_hs15.py +++ b/src/third_party/wiredtiger/test/suite/test_hs15.py @@ -27,7 +27,7 @@ # OTHER DEALINGS IN THE SOFTWARE. # # [TEST_TAGS] -# caching_eviction:correctness:written_data +# history_store:eviction_checkpoint_interaction # [END_TAGS] # diff --git a/src/third_party/wiredtiger/test/suite/test_hs24.py b/src/third_party/wiredtiger/test/suite/test_hs24.py new file mode 100644 index 00000000000..4c4ee4a4b94 --- /dev/null +++ b/src/third_party/wiredtiger/test/suite/test_hs24.py @@ -0,0 +1,188 @@ +#!/usr/bin/env python +# +# Public Domain 2014-present MongoDB, Inc. +# Public Domain 2008-2014 WiredTiger, Inc. +# +# This is free and unencumbered software released into the public domain. +# +# Anyone is free to copy, modify, publish, use, compile, sell, or +# distribute this software, either in source code form or as a compiled +# binary, for any purpose, commercial or non-commercial, and by any +# means. +# +# In jurisdictions that recognize copyright laws, the author or authors +# of this software dedicate any and all copyright interest in the +# software to the public domain. We make this dedication for the benefit +# of the public at large and to the detriment of our heirs and +# successors. We intend this dedication to be an overt act of +# relinquishment in perpetuity of all present and future rights to this +# software under copyright law. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +# IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. + +import wttest, threading, wiredtiger +from helper import simulate_crash_restart + +def timestamp_str(t): + return '%x' % t + +# test_hs24.py +# Test that out of order timestamp fix racing with checkpointing the history store doesn't create inconsistent checkpoint. +class test_hs24(wttest.WiredTigerTestCase): + conn_config = 'cache_size=50MB,timing_stress_for_test=(history_store_checkpoint_delay)' + session_config = 'isolation=snapshot' + uri = 'table:test_hs24' + + value1 = 'a' * 500 + value2 = 'b' * 500 + value3 = 'c' * 500 + value4 = 'd' * 500 + def test_zero_ts(self): + self.session.create(self.uri, 'key_format=i,value_format=S') + self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(1)) + cursor = self.session.open_cursor(self.uri) + for i in range(0, 2000): + self.session.begin_transaction() + cursor[i] = self.value1 + self.session.commit_transaction('commit_timestamp=' + timestamp_str(4)) + self.session.begin_transaction() + cursor[i] = self.value2 + self.session.commit_transaction('commit_timestamp=' + timestamp_str(5)) + cursor.close() + self.conn.set_timestamp('stable_timestamp=' + timestamp_str(5)) + thread = threading.Thread(target=self.zero_ts_deletes) + thread.start() + self.session.checkpoint() + thread.join() + simulate_crash_restart(self, '.', "RESTART") + cursor = self.session.open_cursor(self.uri) + session2 = self.conn.open_session(None) + cursor2 = session2.open_cursor(self.uri) + self.session.begin_transaction('read_timestamp=' + timestamp_str(5)) + session2.begin_transaction('read_timestamp=' + timestamp_str(4)) + # Check the data store and the history store content is consistent. + # If we have a value in the data store, we should see the older + # version in the history store as well. + newer_data_visible = False + for i in range(0, 2000): + cursor.set_key(i) + cursor2.set_key(i) + ret = cursor.search() + ret2 = cursor2.search() + if not newer_data_visible: + newer_data_visible = ret != wiredtiger.WT_NOTFOUND + if newer_data_visible: + self.assertEquals(cursor.get_value(), self.value2) + self.assertEquals(cursor2.get_value(), self.value1) + else: + self.assertEquals(ret2, wiredtiger.WT_NOTFOUND) + session2.rollback_transaction() + self.session.rollback_transaction() + + def zero_ts_deletes(self): + session = self.setUpSessionOpen(self.conn) + cursor = session.open_cursor(self.uri) + for i in range(0, 2000): + session.begin_transaction() + cursor.set_key(i) + cursor.remove() + session.commit_transaction() + cursor.close() + session.close() + + def test_zero_commit(self): + self.session.create(self.uri, 'key_format=i,value_format=S') + self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(1)) + cursor = self.session.open_cursor(self.uri) + for i in range(0, 2000): + self.session.begin_transaction() + cursor[i] = self.value1 + self.session.commit_transaction('commit_timestamp=' + timestamp_str(4)) + self.session.begin_transaction() + cursor[i] = self.value2 + self.session.commit_transaction('commit_timestamp=' + timestamp_str(5)) + cursor.close() + self.conn.set_timestamp('stable_timestamp=' + timestamp_str(4)) + thread = threading.Thread(target=self.zero_ts_commits) + thread.start() + self.session.checkpoint() + thread.join() + simulate_crash_restart(self, '.', "RESTART") + cursor = self.session.open_cursor(self.uri) + self.session.begin_transaction('read_timestamp=' + timestamp_str(4)) + # Check we can only see the version committed by the zero timestamp + # commit thread before the checkpoint starts or value1. + newer_data_visible = False + for i in range(0, 2000): + value = cursor[i] + if not newer_data_visible: + newer_data_visible = value != self.value3 + if newer_data_visible: + self.assertEquals(value, self.value1) + else: + self.assertEquals(value, self.value3) + self.session.rollback_transaction() + + def zero_ts_commits(self): + session = self.setUpSessionOpen(self.conn) + cursor = session.open_cursor(self.uri) + for i in range(0, 2000): + session.begin_transaction() + cursor[i] = self.value3 + session.commit_transaction() + cursor.close() + session.close() + + def test_out_of_order_ts(self): + self.session.create(self.uri, 'key_format=i,value_format=S') + self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(1)) + cursor = self.session.open_cursor(self.uri) + for i in range(0, 2000): + self.session.begin_transaction() + cursor[i] = self.value1 + self.session.commit_transaction('commit_timestamp=' + timestamp_str(4)) + self.session.begin_transaction() + cursor[i] = self.value2 + self.session.commit_transaction('commit_timestamp=' + timestamp_str(5)) + self.conn.set_timestamp('stable_timestamp=' + timestamp_str(4)) + for i in range(0, 2000): + self.session.begin_transaction() + cursor[i] = self.value3 + self.session.commit_transaction('commit_timestamp=' + timestamp_str(6)) + cursor.close() + thread = threading.Thread(target=self.out_of_order_ts_commits) + thread.start() + self.session.checkpoint() + thread.join() + simulate_crash_restart(self, '.', "RESTART") + cursor = self.session.open_cursor(self.uri) + self.session.begin_transaction('read_timestamp=' + timestamp_str(4)) + # Check we can only see the version at timestamp 4, it's either + # committed by the out of order timestamp commit thread before the + # checkpoint starts or value1. + newer_data_visible = False + for i in range(0, 2000): + value = cursor[i] + if not newer_data_visible: + newer_data_visible = value != self.value4 + if newer_data_visible: + self.assertEquals(value, self.value1) + else: + self.assertEquals(value, self.value4) + self.session.rollback_transaction() + + def out_of_order_ts_commits(self): + session = self.setUpSessionOpen(self.conn) + cursor = session.open_cursor(self.uri) + for i in range(0, 2000): + session.begin_transaction() + cursor[i] = self.value4 + session.commit_transaction('commit_timestamp=' + timestamp_str(4)) + cursor.close() + session.close() diff --git a/src/third_party/wiredtiger/test/suite/test_hs25.py b/src/third_party/wiredtiger/test/suite/test_hs25.py new file mode 100644 index 00000000000..e353ce076fe --- /dev/null +++ b/src/third_party/wiredtiger/test/suite/test_hs25.py @@ -0,0 +1,73 @@ +#!/usr/bin/env python +# +# Public Domain 2014-present MongoDB, Inc. +# Public Domain 2008-2014 WiredTiger, Inc. +# +# This is free and unencumbered software released into the public domain. +# +# Anyone is free to copy, modify, publish, use, compile, sell, or +# distribute this software, either in source code form or as a compiled +# binary, for any purpose, commercial or non-commercial, and by any +# means. +# +# In jurisdictions that recognize copyright laws, the author or authors +# of this software dedicate any and all copyright interest in the +# software to the public domain. We make this dedication for the benefit +# of the public at large and to the detriment of our heirs and +# successors. We intend this dedication to be an overt act of +# relinquishment in perpetuity of all present and future rights to this +# software under copyright law. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +# IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. + +import wttest + +def timestamp_str(t): + return '%x' % t + +# test_hs25.py +# Ensure updates structure is correct when processing each key. +class test_hs25(wttest.WiredTigerTestCase): + conn_config = 'cache_size=50MB' + session_config = 'isolation=snapshot' + uri = 'table:test_hs25' + + def test_insert_updates_hs(self): + self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(1)) + self.conn.set_timestamp('stable_timestamp=' + timestamp_str(1)) + self.session.create(self.uri, 'key_format=i,value_format=S') + s = self.conn.open_session() + + # Update the first key. + cursor1 = self.session.open_cursor(self.uri) + self.session.begin_transaction() + cursor1[1] = 'a' + self.session.commit_transaction('commit_timestamp=' + timestamp_str(2)) + + # Update the second key. + self.session.begin_transaction() + cursor1[2] = 'a' + self.session.commit_transaction('commit_timestamp=' + timestamp_str(2)) + self.session.begin_transaction() + cursor1[2] = 'b' + self.session.commit_transaction('commit_timestamp=' + timestamp_str(3)) + + # Prepared update on the first key. + self.session.begin_transaction() + cursor1[1] = 'b' + cursor1[1] = 'c' + self.session.prepare_transaction('prepare_timestamp=' + timestamp_str(4)) + + # Run eviction cursor. + s.begin_transaction('ignore_prepare=true') + evict_cursor = s.open_cursor(self.uri, None, 'debug=(release_evict)') + self.assertEqual(evict_cursor[1], 'a') + self.assertEqual(evict_cursor[2], 'b') + s.rollback_transaction() + self.session.rollback_transaction() diff --git a/src/third_party/wiredtiger/test/suite/test_huffman02.py b/src/third_party/wiredtiger/test/suite/test_huffman02.py index 30da6804778..91bbbbc37bd 100644 --- a/src/third_party/wiredtiger/test/suite/test_huffman02.py +++ b/src/third_party/wiredtiger/test/suite/test_huffman02.py @@ -25,6 +25,10 @@ # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. +# +# [TEST_TAGS] +# huffman_encoding +# [END_TAGS] import os from suite_subprocess import suite_subprocess diff --git a/src/third_party/wiredtiger/test/suite/test_import06.py b/src/third_party/wiredtiger/test/suite/test_import06.py index fae5b3cfd35..083fc860a67 100644 --- a/src/third_party/wiredtiger/test/suite/test_import06.py +++ b/src/third_party/wiredtiger/test/suite/test_import06.py @@ -48,6 +48,16 @@ class test_import06(test_import_base): create_config = 'allocation_size={},key_format=u,log=(enabled=true),value_format=u,' \ 'block_compressor={},encryption=(name={})' + # To test the sodium encryptor, we use secretkey= rather than + # setting a keyid, because for a "real" (vs. test-only) encryptor, + # keyids require some kind of key server, and (a) setting one up + # for testing would be a nuisance and (b) currently the sodium + # encryptor doesn't support any anyway. + # + # It expects secretkey= to provide a hex-encoded 256-bit chacha20 key. + # This key will serve for testing purposes. + sodium_testkey = '0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef' + allocsizes = [ ('512', dict(allocsize='512')), ('1024', dict(allocsize='1024')), @@ -63,9 +73,10 @@ class test_import06(test_import_base): ('zstd', dict(compressor='zstd')), ] encryptors = [ - ('none', dict(encryptor='none')), - ('nop', dict(encryptor='none')), - ('rotn', dict(encryptor='rotn')), + ('none', dict(encryptor='none', encryptor_args='')), + ('nop', dict(encryptor='none', encryptor_args='')), + ('rotn', dict(encryptor='rotn', encryptor_args='')), + ('sodium', dict(encryptor='sodium', encryptor_args=',secretkey=' + sodium_testkey)), ] scenarios = make_scenarios(allocsizes, compressors, encryptors) @@ -77,7 +88,7 @@ class test_import06(test_import_base): def conn_config(self): return 'cache_size=50MB,log=(enabled),statistics=(all),encryption=(name={})'.format( - self.encryptor) + self.encryptor + self.encryptor_args) def test_import_repair(self): self.session.create(self.uri, diff --git a/src/third_party/wiredtiger/test/suite/test_import09.py b/src/third_party/wiredtiger/test/suite/test_import09.py index 7caa89055c0..a41aff32571 100644 --- a/src/third_party/wiredtiger/test/suite/test_import09.py +++ b/src/third_party/wiredtiger/test/suite/test_import09.py @@ -38,6 +38,16 @@ class test_import09(test_import_base): ntables = 1 session_config = 'isolation=snapshot' + # To test the sodium encryptor, we use secretkey= rather than + # setting a keyid, because for a "real" (vs. test-only) encryptor, + # keyids require some kind of key server, and (a) setting one up + # for testing would be a nuisance and (b) currently the sodium + # encryptor doesn't support any anyway. + # + # It expects secretkey= to provide a hex-encoded 256-bit chacha20 key. + # This key will serve for testing purposes. + sodium_testkey = '0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef' + allocsizes = [ ('512', dict(allocsize='512')), ('1024', dict(allocsize='1024')), @@ -53,9 +63,10 @@ class test_import09(test_import_base): ('zstd', dict(compressor='zstd')), ] encryptors = [ - ('none', dict(encryptor='none')), - ('nop', dict(encryptor='nop')), - ('rotn', dict(encryptor='rotn')), + ('none', dict(encryptor='none', encryptor_args='')), + ('nop', dict(encryptor='nop', encryptor_args='')), + ('rotn', dict(encryptor='rotn', encryptor_args='')), + ('sodium', dict(encryptor='sodium', encryptor_args=',secretkey=' + sodium_testkey)), ] tables = [ ('simple_table', dict( @@ -90,7 +101,7 @@ class test_import09(test_import_base): def conn_config(self): return 'cache_size=50MB,log=(enabled),statistics=(all),encryption=(name={})'.format( - self.encryptor) + self.encryptor + self.encryptor_args) def test_import_table_repair(self): # Add some tables & data and checkpoint. diff --git a/src/third_party/wiredtiger/test/suite/test_index02.py b/src/third_party/wiredtiger/test/suite/test_index02.py index de9fd2998c5..17e9c2b0f83 100755 --- a/src/third_party/wiredtiger/test/suite/test_index02.py +++ b/src/third_party/wiredtiger/test/suite/test_index02.py @@ -25,6 +25,10 @@ # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. +# +# [TEST_TAGS] +# indexes:search_near +# [END_TAGS] import wiredtiger, wttest from wtscenario import make_scenarios diff --git a/src/third_party/wiredtiger/test/suite/test_prepare08.py b/src/third_party/wiredtiger/test/suite/test_prepare08.py index c93a8984ab6..76a831b0a05 100644 --- a/src/third_party/wiredtiger/test/suite/test_prepare08.py +++ b/src/third_party/wiredtiger/test/suite/test_prepare08.py @@ -60,8 +60,11 @@ class test_prepare08(wttest.WiredTigerTestCase): self.session.commit_transaction('commit_timestamp=' + timestamp_str(ts)) cursor.close() - def check(self, ds, uri, nrows, value, ts): - cursor = self.session.open_cursor(uri) + def check(self, ds, uri, nrows, value, ts, release_evict): + if release_evict: + cursor = self.session.open_cursor(uri, None, "debug=(release_evict)") + else: + cursor = self.session.open_cursor(uri) self.session.begin_transaction('ignore_prepare=true,read_timestamp=' + timestamp_str(ts)) for i in range(1, nrows): cursor.set_key(ds.key(i)) @@ -70,6 +73,7 @@ class test_prepare08(wttest.WiredTigerTestCase): else: self.assertEquals(cursor.search(), 0) self.assertEquals(cursor.get_value(),value) + cursor.reset() self.session.commit_transaction() cursor.close() @@ -104,11 +108,11 @@ class test_prepare08(wttest.WiredTigerTestCase): self.updates(ds_2, uri_2, nrows, value_b, 30) # Verify the updates - self.check(ds_1, uri_1, nrows, value_a, 20) - self.check(ds_1, uri_1, nrows, value_b, 30) + self.check(ds_1, uri_1, nrows, value_a, 20, False) + self.check(ds_1, uri_1, nrows, value_b, 30, False) - self.check(ds_2, uri_2, nrows, value_a, 20) - self.check(ds_2, uri_2, nrows, value_b, 30) + self.check(ds_2, uri_2, nrows, value_a, 20, False) + self.check(ds_2, uri_2, nrows, value_b, 30, False) # Checkpoint self.session.checkpoint() @@ -127,14 +131,14 @@ class test_prepare08(wttest.WiredTigerTestCase): self.updates(ds_2, uri_2, nrows, value_d, 50) self.updates(ds_2, uri_2, nrows, value_e, 60) - self.check(ds_1, uri_1, nrows, value_a, 20) - self.check(ds_1, uri_1, nrows, value_b, 50) + self.check(ds_1, uri_1, nrows, value_a, 20, True) + self.check(ds_1, uri_1, nrows, value_b, 50, True) #rollback the prepared session session_p.rollback_transaction() - self.check(ds_1, uri_1, nrows, value_a, 20) - self.check(ds_1, uri_1, nrows, value_b, 50) + self.check(ds_1, uri_1, nrows, value_a, 20, False) + self.check(ds_1, uri_1, nrows, value_b, 50, False) # close sessions. cursor_p.close() @@ -173,11 +177,11 @@ class test_prepare08(wttest.WiredTigerTestCase): self.updates(ds_2, uri_2, nrows, value_b, 30) # Verify the updates - self.check(ds_1, uri_1, nrows, value_a, 20) - self.check(ds_1, uri_1, nrows, value_b, 30) + self.check(ds_1, uri_1, nrows, value_a, 20, False) + self.check(ds_1, uri_1, nrows, value_b, 30, False) - self.check(ds_2, uri_2, nrows, value_a, 20) - self.check(ds_2, uri_2, nrows, value_b, 30) + self.check(ds_2, uri_2, nrows, value_a, 20, False) + self.check(ds_2, uri_2, nrows, value_b, 30, False) # Checkpoint self.session.checkpoint() @@ -199,16 +203,16 @@ class test_prepare08(wttest.WiredTigerTestCase): self.updates(ds_2, uri_2, nrows, value_d, 50) self.updates(ds_2, uri_2, nrows, value_e, 60) - self.check(ds_1, uri_1, nrows, value_a, 20) - self.check(ds_1, uri_1, nrows, value_b, 30) - self.check(ds_1, uri_1, nrows, value_b, 50) + self.check(ds_1, uri_1, nrows, value_a, 20, True) + self.check(ds_1, uri_1, nrows, value_b, 30, True) + self.check(ds_1, uri_1, nrows, value_b, 50, True) # Commit the prepared session session_p.commit_transaction('commit_timestamp=' + timestamp_str(50) + ',durable_timestamp=' + timestamp_str(60)) - self.check(ds_1, uri_1, nrows, value_a, 20) - self.check(ds_1, uri_1, nrows, value_b, 30) - self.check(ds_1, uri_1, 0, None, 50) + self.check(ds_1, uri_1, nrows, value_a, 20, False) + self.check(ds_1, uri_1, nrows, value_b, 30, False) + self.check(ds_1, uri_1, 0, None, 50, False) # close sessions. cursor_p.close() @@ -269,16 +273,16 @@ class test_prepare08(wttest.WiredTigerTestCase): self.updates(ds_2, uri_2, nrows, value_d, 50) self.updates(ds_2, uri_2, nrows, value_e, 60) - self.check(ds_1, uri_1, nrows, value_a, 20) - self.check(ds_1, uri_1, 0, None, 30) - self.check(ds_1, uri_1, 0, None, 50) + self.check(ds_1, uri_1, nrows, value_a, 20, True) + self.check(ds_1, uri_1, 0, None, 30, True) + self.check(ds_1, uri_1, 0, None, 50, True) # Commit the prepared session session_p.commit_transaction('commit_timestamp=' + timestamp_str(50) + ',durable_timestamp=' + timestamp_str(60)) - self.check(ds_1, uri_1, nrows, value_a, 20) - self.check(ds_1, uri_1, 0, None, 30) - self.check(ds_1, uri_1, 0, None, 50) + self.check(ds_1, uri_1, nrows, value_a, 20, False) + self.check(ds_1, uri_1, 0, None, 30, False) + self.check(ds_1, uri_1, 0, None, 50, False) # close sessions. cursor_p.close() diff --git a/src/third_party/wiredtiger/test/suite/test_prepare09.py b/src/third_party/wiredtiger/test/suite/test_prepare09.py index f8b3c0ebd02..3c0aa546d08 100644 --- a/src/third_party/wiredtiger/test/suite/test_prepare09.py +++ b/src/third_party/wiredtiger/test/suite/test_prepare09.py @@ -25,6 +25,10 @@ # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. +# +# [TEST_TAGS] +# prepare +# [END_TAGS] import wiredtiger, wttest def timestamp_str(t): diff --git a/src/third_party/wiredtiger/test/suite/test_prepare12.py b/src/third_party/wiredtiger/test/suite/test_prepare12.py index 2663d12ad70..55d6c69c2d7 100644 --- a/src/third_party/wiredtiger/test/suite/test_prepare12.py +++ b/src/third_party/wiredtiger/test/suite/test_prepare12.py @@ -25,6 +25,10 @@ # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. +# +# [TEST_TAGS] +# eviction:prepare +# [END_TAGS] import wiredtiger, wttest def timestamp_str(t): diff --git a/src/third_party/wiredtiger/test/suite/test_prepare13.py b/src/third_party/wiredtiger/test/suite/test_prepare13.py index 152cc9219f3..8542a182328 100644 --- a/src/third_party/wiredtiger/test/suite/test_prepare13.py +++ b/src/third_party/wiredtiger/test/suite/test_prepare13.py @@ -26,6 +26,10 @@ # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. # +# [TEST_TAGS] +# truncate:prepare +# [END_TAGS] +# # test_prepare13.py # Fast-truncate fails when a page contains prepared updates. import wiredtiger, wttest diff --git a/src/third_party/wiredtiger/test/suite/test_prepare_cursor01.py b/src/third_party/wiredtiger/test/suite/test_prepare_cursor01.py index 6c29e00e2da..aa2e45bec32 100644 --- a/src/third_party/wiredtiger/test/suite/test_prepare_cursor01.py +++ b/src/third_party/wiredtiger/test/suite/test_prepare_cursor01.py @@ -25,6 +25,10 @@ # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. +# +# [TEST_TAGS] +# cursors:prepare +# [END_TAGS] import wiredtiger, wttest from wtdataset import SimpleDataSet, SimpleIndexDataSet diff --git a/src/third_party/wiredtiger/test/suite/test_prepare_hs02.py b/src/third_party/wiredtiger/test/suite/test_prepare_hs02.py index fd00db26341..5ba68e6a614 100644 --- a/src/third_party/wiredtiger/test/suite/test_prepare_hs02.py +++ b/src/third_party/wiredtiger/test/suite/test_prepare_hs02.py @@ -133,7 +133,7 @@ class test_prepare_hs02(wttest.WiredTigerTestCase, suite_subprocess): c[1] = 1 c[2] = 1 c[3] = 1 - self.session.commit_transaction('commit_timestamp=' + timestamp_str(301)) + self.session.commit_transaction('commit_timestamp=' + timestamp_str(302)) # Trigger a checkpoint, which could trigger reconciliation self.conn.set_timestamp('stable_timestamp=' + timestamp_str(350)) diff --git a/src/third_party/wiredtiger/test/suite/test_prepare_hs03.py b/src/third_party/wiredtiger/test/suite/test_prepare_hs03.py index 356ef7554f9..502a7ba9c8e 100644 --- a/src/third_party/wiredtiger/test/suite/test_prepare_hs03.py +++ b/src/third_party/wiredtiger/test/suite/test_prepare_hs03.py @@ -25,6 +25,11 @@ # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. +# +# [TEST_TAGS] +# salvage:prepare +# verify:prepare +# [END_TAGS] from helper import copy_wiredtiger_home import wiredtiger, wttest @@ -60,10 +65,21 @@ class test_prepare_hs03(wttest.WiredTigerTestCase): tablepointer.write('Bad!' * 1024) def corrupt_salvage_verify(self): + # An exclusive handle operation can fail if there is dirty data in the cache, closing the + # open handles before acquiring an exclusive handle will return EBUSY. A checkpoint should + # clear the dirty data, but eviction can re-dirty the cache between the checkpoint and the + # open attempt, we have to loop. + self.session.checkpoint() if self.corrupt == True: self.corrupt_table() - self.session.salvage(self.uri, "force") - self.session.verify(self.uri, None) + while True: + if not self.raisesBusy(lambda: self.session.salvage(self.uri, "force")): + break + self.session.checkpoint() + while True: + if not self.raisesBusy(lambda: self.session.verify(self.uri, None)): + break + self.session.checkpoint() def get_stat(self, stat): stat_cursor = self.session.open_cursor('statistics:') @@ -72,8 +88,6 @@ class test_prepare_hs03(wttest.WiredTigerTestCase): return val def prepare_updates(self, ds, nrows, nsessions, nkeys): - # Insert some records with commit timestamp, corrupt file and call salvage, verify before checkpoint. - # Commit some updates to get eviction and history store fired up commit_value = b"bbbbb" * 100 cursor = self.session.open_cursor(self.uri) @@ -85,11 +99,12 @@ class test_prepare_hs03(wttest.WiredTigerTestCase): self.session.commit_transaction('commit_timestamp=' + timestamp_str(1)) cursor.close() - # Corrupt the table, Call salvage to recover data from the corrupted table and call verify - self.corrupt_salvage_verify() + # Set the stable/oldest timstamps. + self.conn.set_timestamp('stable_timestamp=' + timestamp_str(1)) + self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(1)) - # Call checkpoint - self.session.checkpoint() + # Corrupt the table, call salvage to recover data from the corrupted table and call verify + self.corrupt_salvage_verify() hs_writes_start = self.get_stat(stat.conn.cache_write_hs) @@ -129,19 +144,13 @@ class test_prepare_hs03(wttest.WiredTigerTestCase): self.assertNotEquals(cursor.get_value(), prepare_value) cursor.close() - # Close all cursors and sessions, this will cause prepared updates to be - # rollback-ed + # Close all sessions (and cursors), this will cause prepared updates to be rolled back. for j in range (0, nsessions): - cursors[j].close() sessions[j].close() - # Corrupt the table, Call salvage to recover data from the corrupted table and call verify - self.corrupt_salvage_verify() - self.session.commit_transaction() - self.session.checkpoint() - # Corrupt the table, Call salvage to recover data from the corrupted table and call verify + # Corrupt the table, call salvage to recover data from the corrupted table and call verify self.corrupt_salvage_verify() # Finally, search for the keys inserted with commit timestamp @@ -180,8 +189,8 @@ class test_prepare_hs03(wttest.WiredTigerTestCase): cursor.close() self.session.commit_transaction() - # After simulating a crash, corrupt the table, call salvage to recover data from the corrupted table - # and call verify + # After simulating a crash, corrupt the table, call salvage to recover data from the + # corrupted table and call verify self.corrupt_salvage_verify() def test_prepare_hs(self): diff --git a/src/third_party/wiredtiger/test/suite/test_reconfig01.py b/src/third_party/wiredtiger/test/suite/test_reconfig01.py index 9a683252b7a..75a1c0bda9b 100644 --- a/src/third_party/wiredtiger/test/suite/test_reconfig01.py +++ b/src/third_party/wiredtiger/test/suite/test_reconfig01.py @@ -25,6 +25,10 @@ # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. +# +# [TEST_TAGS] +# connection_api:reconfigure +# [END_TAGS] import time import wiredtiger, wttest diff --git a/src/third_party/wiredtiger/test/suite/test_reconfig02.py b/src/third_party/wiredtiger/test/suite/test_reconfig02.py index bae9303e5b2..6eaeab95060 100644 --- a/src/third_party/wiredtiger/test/suite/test_reconfig02.py +++ b/src/third_party/wiredtiger/test/suite/test_reconfig02.py @@ -25,6 +25,10 @@ # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. +# +# [TEST_TAGS] +# connection_api:reconfigure +# [END_TAGS] import fnmatch, os, time import wiredtiger, wttest diff --git a/src/third_party/wiredtiger/test/suite/test_reconfig04.py b/src/third_party/wiredtiger/test/suite/test_reconfig04.py index 8119de46265..82dca85725c 100644 --- a/src/third_party/wiredtiger/test/suite/test_reconfig04.py +++ b/src/third_party/wiredtiger/test/suite/test_reconfig04.py @@ -25,6 +25,10 @@ # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. +# +# [TEST_TAGS] +# session_api:reconfigure +# [END_TAGS] import wiredtiger, wttest diff --git a/src/third_party/wiredtiger/test/suite/test_rollback_to_stable01.py b/src/third_party/wiredtiger/test/suite/test_rollback_to_stable01.py index da5b6d1ca91..7613e169fb1 100755 --- a/src/third_party/wiredtiger/test/suite/test_rollback_to_stable01.py +++ b/src/third_party/wiredtiger/test/suite/test_rollback_to_stable01.py @@ -30,8 +30,9 @@ import time from helper import copy_wiredtiger_home import wiredtiger, wttest from wtdataset import SimpleDataSet -from wiredtiger import stat +from wiredtiger import stat, wiredtiger_strerror, WiredTigerError, WT_ROLLBACK from wtscenario import make_scenarios +from time import sleep def timestamp_str(t): return '%x' % t @@ -39,53 +40,67 @@ def timestamp_str(t): # test_rollback_to_stable01.py # Shared base class used by rollback to stable tests. class test_rollback_to_stable_base(wttest.WiredTigerTestCase): + def retry_rollback(self, name, txn_session, code): + retry_limit = 100 + retries = 0 + completed = False + saved_exception = None + while not completed and retries < retry_limit: + if retries != 0: + self.pr("Retrying operation for " + name) + if txn_session: + txn_session.rollback_transaction() + sleep(0.1) + if txn_session: + txn_session.begin_transaction('isolation=snapshot') + self.pr("Began new transaction for " + name) + try: + code() + completed = True + except WiredTigerError as e: + rollback_str = wiredtiger_strerror(WT_ROLLBACK) + if rollback_str not in str(e): + raise(e) + retries += 1 + saved_exception = e + if not completed and saved_exception: + raise(saved_exception) + def large_updates(self, uri, value, ds, nrows, prepare, commit_ts): # Update a large number of records. session = self.session - cursor = session.open_cursor(uri) - for i in range(1, nrows + 1): - session.begin_transaction() - cursor[ds.key(i)] = value - if commit_ts == 0: - session.commit_transaction() - elif prepare: - session.prepare_transaction('prepare_timestamp=' + timestamp_str(commit_ts-1)) - session.timestamp_transaction('commit_timestamp=' + timestamp_str(commit_ts)) - session.timestamp_transaction('durable_timestamp=' + timestamp_str(commit_ts+1)) - session.commit_transaction() - else: - session.commit_transaction('commit_timestamp=' + timestamp_str(commit_ts)) - cursor.close() + try: + cursor = session.open_cursor(uri) + for i in range(1, nrows + 1): + session.begin_transaction() + cursor[ds.key(i)] = value + if commit_ts == 0: + session.commit_transaction() + elif prepare: + session.prepare_transaction('prepare_timestamp=' + timestamp_str(commit_ts-1)) + session.timestamp_transaction('commit_timestamp=' + timestamp_str(commit_ts)) + session.timestamp_transaction('durable_timestamp=' + timestamp_str(commit_ts+1)) + session.commit_transaction() + else: + session.commit_transaction('commit_timestamp=' + timestamp_str(commit_ts)) + cursor.close() + except WiredTigerError as e: + rollback_str = wiredtiger_strerror(WT_ROLLBACK) + if rollback_str in str(e): + session.rollback_transaction() + raise(e) def large_modifies(self, uri, value, ds, location, nbytes, nrows, prepare, commit_ts): # Load a slight modification. session = self.session - cursor = session.open_cursor(uri) - session.begin_transaction() - for i in range(1, nrows + 1): - cursor.set_key(i) - mods = [wiredtiger.Modify(value, location, nbytes)] - self.assertEqual(cursor.modify(mods), 0) - - if commit_ts == 0: - session.commit_transaction() - elif prepare: - session.prepare_transaction('prepare_timestamp=' + timestamp_str(commit_ts-1)) - session.timestamp_transaction('commit_timestamp=' + timestamp_str(commit_ts)) - session.timestamp_transaction('durable_timestamp=' + timestamp_str(commit_ts+1)) - session.commit_transaction() - else: - session.commit_transaction('commit_timestamp=' + timestamp_str(commit_ts)) - cursor.close() - - def large_removes(self, uri, ds, nrows, prepare, commit_ts): - # Remove a large number of records. - session = self.session - cursor = session.open_cursor(uri) - for i in range(1, nrows + 1): + try: + cursor = session.open_cursor(uri) session.begin_transaction() - cursor.set_key(i) - cursor.remove() + for i in range(1, nrows + 1): + cursor.set_key(i) + mods = [wiredtiger.Modify(value, location, nbytes)] + self.assertEqual(cursor.modify(mods), 0) + if commit_ts == 0: session.commit_transaction() elif prepare: @@ -95,7 +110,37 @@ class test_rollback_to_stable_base(wttest.WiredTigerTestCase): session.commit_transaction() else: session.commit_transaction('commit_timestamp=' + timestamp_str(commit_ts)) - cursor.close() + cursor.close() + except WiredTigerError as e: + rollback_str = wiredtiger_strerror(WT_ROLLBACK) + if rollback_str in str(e): + session.rollback_transaction() + raise(e) + + def large_removes(self, uri, ds, nrows, prepare, commit_ts): + # Remove a large number of records. + session = self.session + try: + cursor = session.open_cursor(uri) + for i in range(1, nrows + 1): + session.begin_transaction() + cursor.set_key(i) + cursor.remove() + if commit_ts == 0: + session.commit_transaction() + elif prepare: + session.prepare_transaction('prepare_timestamp=' + timestamp_str(commit_ts-1)) + session.timestamp_transaction('commit_timestamp=' + timestamp_str(commit_ts)) + session.timestamp_transaction('durable_timestamp=' + timestamp_str(commit_ts+1)) + session.commit_transaction() + else: + session.commit_transaction('commit_timestamp=' + timestamp_str(commit_ts)) + cursor.close() + except WiredTigerError as e: + rollback_str = wiredtiger_strerror(WT_ROLLBACK) + if rollback_str in str(e): + session.rollback_transaction() + raise(e) def check(self, check_value, uri, nrows, read_ts): session = self.session diff --git a/src/third_party/wiredtiger/test/suite/test_rollback_to_stable10.py b/src/third_party/wiredtiger/test/suite/test_rollback_to_stable10.py index 8aefee48b16..cfd0c2f3c46 100755 --- a/src/third_party/wiredtiger/test/suite/test_rollback_to_stable10.py +++ b/src/third_party/wiredtiger/test/suite/test_rollback_to_stable10.py @@ -29,41 +29,14 @@ import fnmatch, os, shutil, threading, time from helper import copy_wiredtiger_home, simulate_crash_restart from test_rollback_to_stable01 import test_rollback_to_stable_base -from wiredtiger import stat, wiredtiger_strerror, WiredTigerError, WT_ROLLBACK +from wiredtiger import stat from wtdataset import SimpleDataSet from wtscenario import make_scenarios from wtthread import checkpoint_thread, op_thread -from time import sleep def timestamp_str(t): return '%x' % t -def retry_rollback(self, name, txn_session, code): - retry_limit = 100 - retries = 0 - completed = False - saved_exception = None - while not completed and retries < retry_limit: - if retries != 0: - self.pr("Retrying operation for " + name) - if txn_session: - txn_session.rollback_transaction() - sleep(0.1) - if txn_session: - txn_session.begin_transaction('isolation=snapshot') - self.pr("Began new transaction for " + name) - try: - code() - completed = True - except WiredTigerError as e: - rollback_str = wiredtiger_strerror(WT_ROLLBACK) - if rollback_str not in str(e): - raise(e) - retries += 1 - saved_exception = e - if not completed and saved_exception: - raise(saved_exception) - # test_rollback_to_stable10.py # Test the rollback to stable operation performs sweeping history store. class test_rollback_to_stable10(test_rollback_to_stable_base): @@ -155,13 +128,13 @@ class test_rollback_to_stable10(test_rollback_to_stable_base): # Perform several updates in parallel with checkpoint. # Rollbacks may occur when checkpoint is running, so retry as needed. self.pr("updates") - retry_rollback(self, 'update ds1, e', None, + self.retry_rollback('update ds1, e', None, lambda: self.large_updates(uri_1, value_e, ds_1, nrows, self.prepare, 70)) - retry_rollback(self, 'update ds2, e', None, + self.retry_rollback('update ds2, e', None, lambda: self.large_updates(uri_2, value_e, ds_2, nrows, self.prepare, 70)) - retry_rollback(self, 'update ds1, f', None, + self.retry_rollback('update ds1, f', None, lambda: self.large_updates(uri_1, value_f, ds_1, nrows, self.prepare, 80)) - retry_rollback(self, 'update ds2, f', None, + self.retry_rollback('update ds2, f', None, lambda: self.large_updates(uri_2, value_f, ds_2, nrows, self.prepare, 80)) finally: done.set() @@ -289,7 +262,7 @@ class test_rollback_to_stable10(test_rollback_to_stable_base): session_p1 = self.conn.open_session() cursor_p1 = session_p1.open_cursor(uri_1) session_p1.begin_transaction('isolation=snapshot') - retry_rollback(self, 'update ds1', session_p1, + self.retry_rollback('update ds1', session_p1, lambda: prepare_range_updates( session_p1, cursor_p1, ds_1, value_e, nrows, 'prepare_timestamp=' + timestamp_str(69))) @@ -298,7 +271,7 @@ class test_rollback_to_stable10(test_rollback_to_stable_base): session_p2 = self.conn.open_session() cursor_p2 = session_p2.open_cursor(uri_2) session_p2.begin_transaction('isolation=snapshot') - retry_rollback(self, 'update ds2', session_p2, + self.retry_rollback('update ds2', session_p2, lambda: prepare_range_updates( session_p2, cursor_p2, ds_2, value_e, nrows, 'prepare_timestamp=' + timestamp_str(69))) diff --git a/src/third_party/wiredtiger/test/suite/test_rollback_to_stable13.py b/src/third_party/wiredtiger/test/suite/test_rollback_to_stable13.py index 66f66d62874..73f23b1f615 100644 --- a/src/third_party/wiredtiger/test/suite/test_rollback_to_stable13.py +++ b/src/third_party/wiredtiger/test/suite/test_rollback_to_stable13.py @@ -25,8 +25,6 @@ # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. - -import fnmatch, os, shutil, time from helper import simulate_crash_restart from test_rollback_to_stable01 import test_rollback_to_stable_base from wiredtiger import stat @@ -37,7 +35,8 @@ def timestamp_str(t): return '%x' % t # test_rollback_to_stable13.py -# Test the rollback to stable should roll back the tombstone in the history store. +# Test the rollback to stable should retain/restore the tombstone from +# the update list or from the history store for on-disk database. class test_rollback_to_stable13(test_rollback_to_stable_base): session_config = 'isolation=snapshot' @@ -54,7 +53,7 @@ class test_rollback_to_stable13(test_rollback_to_stable_base): scenarios = make_scenarios(key_format_values, prepare_values) def conn_config(self): - config = 'cache_size=500MB,statistics=(all),log=(enabled=true)' + config = 'cache_size=50MB,statistics=(all),log=(enabled=true)' return config def test_rollback_to_stable(self): @@ -98,6 +97,144 @@ class test_rollback_to_stable13(test_rollback_to_stable_base): self.conn.set_timestamp('stable_timestamp=' + timestamp_str(40)) self.session.checkpoint() + # Simulate a server crash and restart. + simulate_crash_restart(self, ".", "RESTART") + + # Check that the correct data is seen at and after the stable timestamp. + self.check(None, uri, 0, 50) + + # Check that we restore the correct value from the history store. + self.check(value_a, uri, nrows, 20) + + stat_cursor = self.session.open_cursor('statistics:', None, None) + restored_tombstones = stat_cursor[stat.conn.txn_rts_hs_restore_tombstones][2] + self.assertEqual(restored_tombstones, nrows) + + def test_rollback_to_stable_with_aborted_updates(self): + nrows = 1000 + + # Prepare transactions for column store table is not yet supported. + if self.prepare and self.key_format == 'r': + self.skipTest('Prepare transactions for column store table is not yet supported') + + # Create a table without logging. + uri = "table:rollback_to_stable13" + ds = SimpleDataSet( + self, uri, 0, key_format=self.key_format, value_format="S", config='split_pct=50,log=(enabled=false)') + ds.populate() + + # Pin oldest and stable to timestamp 10. + self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(10) + + ',stable_timestamp=' + timestamp_str(10)) + + value_a = "aaaaa" * 100 + value_b = "bbbbb" * 100 + value_c = "ccccc" * 100 + value_d = "ddddd" * 100 + + # Perform several updates. + self.large_updates(uri, value_a, ds, nrows, self.prepare, 20) + + # Update a large number of records and rollback. + cursor = self.session.open_cursor(uri) + for i in range(1, nrows + 1): + self.session.begin_transaction() + cursor[ds.key(i)] = value_b + self.session.rollback_transaction() + cursor.close() + + # Update a large number of records and rollback. + cursor = self.session.open_cursor(uri) + for i in range(1, nrows + 1): + self.session.begin_transaction() + cursor[ds.key(i)] = value_c + self.session.rollback_transaction() + cursor.close() + + # Perform several removes. + self.large_removes(uri, ds, nrows, self.prepare, 30) + + # Perform several updates. + self.large_updates(uri, value_d, ds, nrows, self.prepare, 60) + + # Verify data is visible and correct. + self.check(value_a, uri, nrows, 20) + self.check(None, uri, 0, 30) + self.check(value_d, uri, nrows, 60) + + # Pin stable to timestamp 50 if prepare otherwise 40. + if self.prepare: + self.conn.set_timestamp('stable_timestamp=' + timestamp_str(50)) + else: + self.conn.set_timestamp('stable_timestamp=' + timestamp_str(40)) + + self.session.checkpoint() + # Simulate a server crash and restart. + simulate_crash_restart(self, ".", "RESTART") + + # Check that the correct data is seen at and after the stable timestamp. + self.check(None, uri, 0, 50) + + # Check that we restore the correct value from the history store. + self.check(value_a, uri, nrows, 20) + + stat_cursor = self.session.open_cursor('statistics:', None, None) + restored_tombstones = stat_cursor[stat.conn.txn_rts_hs_restore_tombstones][2] + self.assertEqual(restored_tombstones, nrows) + + def test_rollback_to_stable_with_history_tombstone(self): + nrows = 1000 + + # Prepare transactions for column store table is not yet supported. + if self.prepare and self.key_format == 'r': + self.skipTest('Prepare transactions for column store table is not yet supported') + + # Create a table without logging. + uri = "table:rollback_to_stable13" + ds = SimpleDataSet( + self, uri, 0, key_format=self.key_format, value_format="S", config='split_pct=50,log=(enabled=false)') + ds.populate() + + # Pin oldest and stable to timestamp 10. + self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(10) + + ',stable_timestamp=' + timestamp_str(10)) + + value_a = "aaaaa" * 100 + value_b = "bbbbb" * 100 + value_c = "ccccc" * 100 + + # Perform several updates. + self.large_updates(uri, value_a, ds, nrows, self.prepare, 20) + + # Perform several removes. + self.large_removes(uri, ds, nrows, self.prepare, 30) + + # Perform several updates and removes in a single transaction. + cursor = self.session.open_cursor(uri) + for i in range(1, nrows + 1): + self.session.begin_transaction() + cursor[ds.key(i)] = value_b + cursor.set_key(ds.key(i)) + cursor.remove() + self.session.commit_transaction('commit_timestamp=' + timestamp_str(40)) + cursor.close() + + # Pin stable to timestamp 50 if prepare otherwise 40. + if self.prepare: + self.conn.set_timestamp('stable_timestamp=' + timestamp_str(50)) + else: + self.conn.set_timestamp('stable_timestamp=' + timestamp_str(40)) + + self.session.checkpoint() + + # Perform several updates and checkpoint. + self.large_updates(uri, value_c, ds, nrows, self.prepare, 60) + self.session.checkpoint() + + # Verify data is visible and correct. + self.check(value_a, uri, nrows, 20) + self.check(None, uri, 0, 40) + self.check(value_c, uri, nrows, 60) # Simulate a server crash and restart. simulate_crash_restart(self, ".", "RESTART") @@ -111,3 +248,51 @@ class test_rollback_to_stable13(test_rollback_to_stable_base): stat_cursor = self.session.open_cursor('statistics:', None, None) restored_tombstones = stat_cursor[stat.conn.txn_rts_hs_restore_tombstones][2] self.assertEqual(restored_tombstones, nrows) + + def test_rollback_to_stable_with_stable_remove(self): + nrows = 1000 + # Prepare transactions for column store table is not yet supported. + if self.prepare and self.key_format == 'r': + self.skipTest('Prepare transactions for column store table is not yet supported') + # Create a table without logging. + uri = "table:rollback_to_stable13" + ds = SimpleDataSet( + self, uri, 0, key_format=self.key_format, value_format="S", config='split_pct=50,log=(enabled=false)') + ds.populate() + # Pin oldest and stable to timestamp 10. + self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(10) + + ',stable_timestamp=' + timestamp_str(10)) + value_a = "aaaaa" * 100 + value_b = "bbbbb" * 100 + value_c = "ccccc" * 100 + # Perform several updates. + self.large_updates(uri, value_a, ds, nrows, self.prepare, 20) + # Perform several updates. + self.large_updates(uri, value_b, ds, nrows, self.prepare, 30) + # Perform several removes. + self.large_removes(uri, ds, nrows, self.prepare, 40) + # Pin stable to timestamp 50 if prepare otherwise 40. + if self.prepare: + self.conn.set_timestamp('stable_timestamp=' + timestamp_str(50)) + else: + self.conn.set_timestamp('stable_timestamp=' + timestamp_str(40)) + # Perform several updates and checkpoint. + self.large_updates(uri, value_c, ds, nrows, self.prepare, 60) + self.session.checkpoint() + # Verify data is visible and correct. + self.check(value_a, uri, nrows, 20) + self.check(None, uri, 0, 40) + self.check(value_c, uri, nrows, 60) + self.conn.rollback_to_stable() + # Perform several updates and checkpoint. + self.large_updates(uri, value_c, ds, nrows, self.prepare, 60) + self.session.checkpoint() + # Simulate a server crash and restart. + simulate_crash_restart(self, ".", "RESTART") + # Check that the correct data is seen at and after the stable timestamp. + self.check(None, uri, 0, 50) + # Check that we restore the correct value from the history store. + self.check(value_a, uri, nrows, 20) + stat_cursor = self.session.open_cursor('statistics:', None, None) + restored_tombstones = stat_cursor[stat.conn.txn_rts_hs_restore_tombstones][2] + self.assertEqual(restored_tombstones, nrows) diff --git a/src/third_party/wiredtiger/test/suite/test_rollback_to_stable14.py b/src/third_party/wiredtiger/test/suite/test_rollback_to_stable14.py index 66614938d3c..e7a5229555d 100755 --- a/src/third_party/wiredtiger/test/suite/test_rollback_to_stable14.py +++ b/src/third_party/wiredtiger/test/suite/test_rollback_to_stable14.py @@ -29,11 +29,10 @@ import fnmatch, os, shutil, threading, time from helper import simulate_crash_restart from test_rollback_to_stable01 import test_rollback_to_stable_base -from wiredtiger import stat, wiredtiger_strerror, WiredTigerError, WT_ROLLBACK +from wiredtiger import stat from wtdataset import SimpleDataSet from wtscenario import make_scenarios from wtthread import checkpoint_thread, op_thread -from time import sleep def timestamp_str(t): return '%x' % t @@ -44,32 +43,6 @@ def mod_val(value, char, location, nbytes=1): def append_val(value, char): return value + char -def retry_rollback(self, name, txn_session, code): - retry_limit = 100 - retries = 0 - completed = False - saved_exception = None - while not completed and retries < retry_limit: - if retries != 0: - self.pr("Retrying operation for " + name) - if txn_session: - txn_session.rollback_transaction() - sleep(0.1) - if txn_session: - txn_session.begin_transaction('isolation=snapshot') - self.pr("Began new transaction for " + name) - try: - code() - completed = True - except WiredTigerError as e: - rollback_str = wiredtiger_strerror(WT_ROLLBACK) - if rollback_str not in str(e): - raise(e) - retries += 1 - saved_exception = e - if not completed and saved_exception: - raise(saved_exception) - # test_rollback_to_stable14.py # Test the rollback to stable operation uses proper base update while restoring modifies from history store. class test_rollback_to_stable14(test_rollback_to_stable_base): @@ -147,13 +120,13 @@ class test_rollback_to_stable14(test_rollback_to_stable_base): # Perform several modifies in parallel with checkpoint. # Rollbacks may occur when checkpoint is running, so retry as needed. self.pr("modifies") - retry_rollback(self, 'modify ds1, W', None, + self.retry_rollback('modify ds1, W', None, lambda: self.large_modifies(uri, 'W', ds, 4, 1, nrows, self.prepare, 70)) - retry_rollback(self, 'modify ds1, X', None, + self.retry_rollback('modify ds1, X', None, lambda: self.large_modifies(uri, 'X', ds, 5, 1, nrows, self.prepare, 80)) - retry_rollback(self, 'modify ds1, Y', None, + self.retry_rollback('modify ds1, Y', None, lambda: self.large_modifies(uri, 'Y', ds, 6, 1, nrows, self.prepare, 90)) - retry_rollback(self, 'modify ds1, Z', None, + self.retry_rollback('modify ds1, Z', None, lambda: self.large_modifies(uri, 'Z', ds, 7, 1, nrows, self.prepare, 100)) finally: done.set() @@ -252,13 +225,13 @@ class test_rollback_to_stable14(test_rollback_to_stable_base): # Perform several modifies in parallel with checkpoint. # Rollbacks may occur when checkpoint is running, so retry as needed. self.pr("modifies") - retry_rollback(self, 'modify ds1, W', None, + self.retry_rollback('modify ds1, W', None, lambda: self.large_modifies(uri, 'W', ds, 4, 1, nrows, self.prepare, 70)) - retry_rollback(self, 'modify ds1, X', None, + self.retry_rollback('modify ds1, X', None, lambda: self.large_modifies(uri, 'X', ds, 5, 1, nrows, self.prepare, 80)) - retry_rollback(self, 'modify ds1, Y', None, + self.retry_rollback('modify ds1, Y', None, lambda: self.large_modifies(uri, 'Y', ds, 6, 1, nrows, self.prepare, 90)) - retry_rollback(self, 'modify ds1, Z', None, + self.retry_rollback('modify ds1, Z', None, lambda: self.large_modifies(uri, 'Z', ds, 7, 1, nrows, self.prepare, 100)) finally: done.set() @@ -355,13 +328,13 @@ class test_rollback_to_stable14(test_rollback_to_stable_base): # Perform several modifies in parallel with checkpoint. # Rollbacks may occur when checkpoint is running, so retry as needed. self.pr("modifies") - retry_rollback(self, 'modify ds1, W', None, + self.retry_rollback('modify ds1, W', None, lambda: self.large_modifies(uri, 'W', ds, len(value_modT), 1, nrows, self.prepare, 70)) - retry_rollback(self, 'modify ds1, X', None, + self.retry_rollback('modify ds1, X', None, lambda: self.large_modifies(uri, 'X', ds, len(value_modT) + 1, 1, nrows, self.prepare, 80)) - retry_rollback(self, 'modify ds1, Y', None, + self.retry_rollback('modify ds1, Y', None, lambda: self.large_modifies(uri, 'Y', ds, len(value_modT) + 2, 1, nrows, self.prepare, 90)) - retry_rollback(self, 'modify ds1, Z', None, + self.retry_rollback('modify ds1, Z', None, lambda: self.large_modifies(uri, 'Z', ds, len(value_modT) + 3, 1, nrows, self.prepare, 100)) finally: done.set() diff --git a/src/third_party/wiredtiger/test/suite/test_rollback_to_stable15.py b/src/third_party/wiredtiger/test/suite/test_rollback_to_stable15.py index 03a6aa396dd..b91691b99cd 100644 --- a/src/third_party/wiredtiger/test/suite/test_rollback_to_stable15.py +++ b/src/third_party/wiredtiger/test/suite/test_rollback_to_stable15.py @@ -77,6 +77,7 @@ class test_rollback_to_stable15(wttest.WiredTigerTestCase): self.assertEqual(v, check_value) count += 1 session.commit_transaction() + cursor.close() self.assertEqual(count, nrows) def test_rollback_to_stable(self): @@ -107,6 +108,7 @@ class test_rollback_to_stable15(wttest.WiredTigerTestCase): self.session.begin_transaction() cursor[i] = value30 self.session.commit_transaction('commit_timestamp=' + timestamp_str(5)) + cursor.close() #Set stable timestamp to 2 self.conn.set_timestamp('stable_timestamp=' + timestamp_str(2)) @@ -115,6 +117,7 @@ class test_rollback_to_stable15(wttest.WiredTigerTestCase): self.check(value20, uri, nrows - 1, 2) #Second Update to value30 at timestamp 7 + cursor = self.session.open_cursor(uri) for i in range(1, nrows): self.session.begin_transaction() cursor[i] = value30 @@ -125,6 +128,7 @@ class test_rollback_to_stable15(wttest.WiredTigerTestCase): self.session.begin_transaction() cursor[i] = value40 self.session.commit_transaction('commit_timestamp=' + timestamp_str(9)) + cursor.close() #Set stable timestamp to 7 self.conn.set_timestamp('stable_timestamp=' + timestamp_str(7)) diff --git a/src/third_party/wiredtiger/test/suite/test_rollback_to_stable16.py b/src/third_party/wiredtiger/test/suite/test_rollback_to_stable16.py index 0c0a3235e94..5ca08c18baf 100644 --- a/src/third_party/wiredtiger/test/suite/test_rollback_to_stable16.py +++ b/src/third_party/wiredtiger/test/suite/test_rollback_to_stable16.py @@ -25,6 +25,10 @@ # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. +# +# [TEST_TAGS] +# rollback_to_stable +# [END_TAGS] import os, shutil from helper import simulate_crash_restart diff --git a/src/third_party/wiredtiger/test/suite/test_rollback_to_stable18.py b/src/third_party/wiredtiger/test/suite/test_rollback_to_stable18.py index 68c2e8d0205..a9725841b8b 100644 --- a/src/third_party/wiredtiger/test/suite/test_rollback_to_stable18.py +++ b/src/third_party/wiredtiger/test/suite/test_rollback_to_stable18.py @@ -25,6 +25,11 @@ # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. +# +# [TEST_TAGS] +# rollback_to_stable +# aggregated_time_windows +# [END_TAGS] import fnmatch, os, shutil, time from helper import simulate_crash_restart diff --git a/src/third_party/wiredtiger/test/suite/test_rollback_to_stable19.py b/src/third_party/wiredtiger/test/suite/test_rollback_to_stable19.py index 284499dae64..dd98263ae41 100644 --- a/src/third_party/wiredtiger/test/suite/test_rollback_to_stable19.py +++ b/src/third_party/wiredtiger/test/suite/test_rollback_to_stable19.py @@ -113,6 +113,7 @@ class test_rollback_to_stable19(test_rollback_to_stable_base): cursor2.set_key(1) self.assertEquals(cursor2.search(), WT_NOTFOUND) self.session.commit_transaction() + cursor2.close() # Pin stable timestamp to 20. self.conn.set_timestamp('stable_timestamp=' + timestamp_str(20)) @@ -138,6 +139,7 @@ class test_rollback_to_stable19(test_rollback_to_stable_base): keys_removed = stat_cursor[stat.conn.txn_rts_keys_removed][2] self.assertGreater(upd_aborted, 0) self.assertGreater(keys_removed, 0) + stat_cursor.close() def test_rollback_to_stable_with_history(self): nrows = 1000 @@ -193,6 +195,7 @@ class test_rollback_to_stable19(test_rollback_to_stable_base): cursor2.set_key(1) self.assertEquals(cursor2.search(), WT_NOTFOUND) self.session.commit_transaction() + cursor2.close() # Pin stable timestamp to 40. self.conn.set_timestamp('stable_timestamp=' + timestamp_str(40)) diff --git a/src/third_party/wiredtiger/test/suite/test_rollback_to_stable21.py b/src/third_party/wiredtiger/test/suite/test_rollback_to_stable21.py index 8404af43df9..4e44b9c2d97 100644 --- a/src/third_party/wiredtiger/test/suite/test_rollback_to_stable21.py +++ b/src/third_party/wiredtiger/test/suite/test_rollback_to_stable21.py @@ -25,6 +25,11 @@ # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. +# +# [TEST_TAGS] +# rollback_to_stable:prepare +# rollback_to_stable:out_of_order_timestamps +# [END_TAGS] from wiredtiger import stat, WT_NOTFOUND from wtscenario import make_scenarios diff --git a/src/third_party/wiredtiger/test/suite/test_rollback_to_stable22.py b/src/third_party/wiredtiger/test/suite/test_rollback_to_stable22.py new file mode 100644 index 00000000000..ac56c6c6b5f --- /dev/null +++ b/src/third_party/wiredtiger/test/suite/test_rollback_to_stable22.py @@ -0,0 +1,84 @@ +#!/usr/bin/env python +# +# Public Domain 2014-present MongoDB, Inc. +# Public Domain 2008-2014 WiredTiger, Inc. +# +# This is free and unencumbered software released into the public domain. +# +# Anyone is free to copy, modify, publish, use, compile, sell, or +# distribute this software, either in source code form or as a compiled +# binary, for any purpose, commercial or non-commercial, and by any +# means. +# +# In jurisdictions that recognize copyright laws, the author or authors +# of this software dedicate any and all copyright interest in the +# software to the public domain. We make this dedication for the benefit +# of the public at large and to the detriment of our heirs and +# successors. We intend this dedication to be an overt act of +# relinquishment in perpetuity of all present and future rights to this +# software under copyright law. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +# IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. + +from test_rollback_to_stable01 import test_rollback_to_stable_base +from wtdataset import SimpleDataSet + +def timestamp_str(t): + return '%x' % t + +# test_rollback_to_stable22 +# Test history store operations conflicting with rollback to stable. We're trying to trigger a +# history store eviction concurrently to a rollback to stable call. We'll do this by limiting +# the cache size to 100MB and performing 100MB worth of inserts while periodically calling rollback +# to stable. +class test_rollback_to_stable22(test_rollback_to_stable_base): + conn_config = 'cache_size=100MB,statistics=(fast),statistics_log=(wait=1,json)' + session_config = 'isolation=snapshot' + prepare = False + + def test_rollback_to_stable(self): + nrows = 1000 + nds = 10 + + # Create a few tables and populate them with some initial data. + # + # Our way of preventing history store operations from interfering with rollback to stable's + # transaction check is by draining active evictions from each open dhandle. + # + # It's important that we have a few different tables to work with so that it's + # representative of a real situation. But also don't make the number too high relative to + # the number of updates or we may not have history for any of the tables. + ds_list = list() + for i in range(0, nds): + uri = 'table:rollback_to_stable22_{}'.format(i) + ds = SimpleDataSet( + self, uri, 0, key_format='i', value_format='S', config='log=(enabled=false)') + ds.populate() + ds_list.append(ds) + self.assertEqual(len(ds_list), nds) + + # 100 bytes of data are being inserted into 1000 rows. + # This happens 1000 iterations. + # Overall, that's 100MB of data which is guaranteed to kick start eviction. + for i in range(1, 1000): + # Generate a value, timestamp and table based off the index. + value = str(i)[0] * 100 + ts = i * 10 + ds = ds_list[i % nds] + + # Perform updates. + self.large_updates(ds.uri, value, ds, nrows, False, ts) + + # Every hundred updates, let's run rollback to stable. This is likely to happen during + # a history store eviction at least once. + if i % 100 == 0: + # Put the timestamp backwards so we can rollback the updates we just did. + stable_ts = (i - 1) * 10 + self.conn.set_timestamp('stable_timestamp=' + timestamp_str(stable_ts)) + self.conn.rollback_to_stable() diff --git a/src/third_party/wiredtiger/test/suite/test_schema03.py b/src/third_party/wiredtiger/test/suite/test_schema03.py index 3a1e5b50a52..1ab11a64f82 100755 --- a/src/third_party/wiredtiger/test/suite/test_schema03.py +++ b/src/third_party/wiredtiger/test/suite/test_schema03.py @@ -25,6 +25,11 @@ # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. +# +# [TEST_TAGS] +# schema_api +# indexes +# [END_TAGS] import os import suite_random diff --git a/src/third_party/wiredtiger/test/suite/test_stat03.py b/src/third_party/wiredtiger/test/suite/test_stat03.py index 1f3434fece3..ca1eaa5c83f 100644 --- a/src/third_party/wiredtiger/test/suite/test_stat03.py +++ b/src/third_party/wiredtiger/test/suite/test_stat03.py @@ -25,7 +25,10 @@ # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. - +# +# [TEST_TAGS] +# cursors:statistics +# [END_TAGS] import itertools, wiredtiger, wttest from suite_subprocess import suite_subprocess from wiredtiger import stat diff --git a/src/third_party/wiredtiger/test/suite/test_stat_log02.py b/src/third_party/wiredtiger/test/suite/test_stat_log02.py index f08bfe0281b..cc808c388c9 100644 --- a/src/third_party/wiredtiger/test/suite/test_stat_log02.py +++ b/src/third_party/wiredtiger/test/suite/test_stat_log02.py @@ -25,6 +25,10 @@ # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. +# +# [TEST_TAGS] +# statistics +# [END_TAGS] import glob, json import helper, wiredtiger, wttest diff --git a/src/third_party/wiredtiger/test/suite/test_tiered02.py b/src/third_party/wiredtiger/test/suite/test_tiered02.py index 61f0ede77b5..6e50ec595d6 100755 --- a/src/third_party/wiredtiger/test/suite/test_tiered02.py +++ b/src/third_party/wiredtiger/test/suite/test_tiered02.py @@ -32,25 +32,20 @@ from wtdataset import SimpleDataSet # test_tiered02.py # Test tiered tree class test_tiered02(wttest.WiredTigerTestCase): - K = 1024 - M = 1024 * K - G = 1024 * M uri = "table:test_tiered02" auth_token = "test_token" bucket = "mybucket" bucket_prefix = "pfx_" extension_name = "local_store" - prefix = "pfx-" def conn_config(self): if not os.path.exists(self.bucket): os.mkdir(self.bucket) return \ - 'statistics=(all),' + \ 'tiered_storage=(auth_token=%s,' % self.auth_token + \ 'bucket=%s,' % self.bucket + \ - 'bucket_prefix=%s,' % self.prefix + \ + 'bucket_prefix=%s,' % self.bucket_prefix + \ 'name=%s),tiered_manager=(wait=0)' % self.extension_name # Load the local store extension, but skip the test if it is missing. @@ -122,6 +117,8 @@ class test_tiered02(wttest.WiredTigerTestCase): self.progress('populate') ds.populate() ds.check() + self.progress('open extra cursor on ' + self.uri) + cursor = self.session.open_cursor(self.uri, None, None) self.progress('checkpoint') self.session.checkpoint() @@ -146,16 +143,13 @@ class test_tiered02(wttest.WiredTigerTestCase): self.progress('populate') ds.populate() ds.check() + cursor.close() self.progress('close_conn') self.close_conn() self.progress('reopen_conn') self.reopen_conn() - # FIXME-WT-7589 This test works up to this point, then runs into trouble. - if True: - return - # Check what was there before ds = SimpleDataSet(self, self.uri, 200, config=args) ds.check() diff --git a/src/third_party/wiredtiger/test/suite/test_tiered03.py b/src/third_party/wiredtiger/test/suite/test_tiered03.py index 4c870e366c2..536bc00dd70 100755 --- a/src/third_party/wiredtiger/test/suite/test_tiered03.py +++ b/src/third_party/wiredtiger/test/suite/test_tiered03.py @@ -51,7 +51,6 @@ class test_tiered03(wttest.WiredTigerTestCase): bucket = "mybucket" bucket_prefix = "pfx_" extension_name = "local_store" - prefix = "pfx-" def conn_config(self): if not os.path.exists(self.bucket): @@ -60,7 +59,7 @@ class test_tiered03(wttest.WiredTigerTestCase): 'statistics=(all),' + \ 'tiered_storage=(auth_token=%s,' % self.auth_token + \ 'bucket=%s,' % self.bucket + \ - 'bucket_prefix=%s,' % self.prefix + \ + 'bucket_prefix=%s,' % self.bucket_prefix + \ 'name=%s)' % self.extension_name # Load the local store extension, but skip the test if it is missing. diff --git a/src/third_party/wiredtiger/test/suite/test_tiered04.py b/src/third_party/wiredtiger/test/suite/test_tiered04.py index 78a7e274e53..bca7f43fc4d 100755 --- a/src/third_party/wiredtiger/test/suite/test_tiered04.py +++ b/src/third_party/wiredtiger/test/suite/test_tiered04.py @@ -133,12 +133,7 @@ class test_tiered04(wttest.WiredTigerTestCase): self.check(c, 3) self.pr("flush tier again, holding open cursor") - # FIXME-WT-7591 Remove the extra cursor close and open surrounding the flush_tier call. - # Having a cursor open during a flush_tier does not yet work, so the test closes it, - # and reopens after the flush_tier. - c.close() self.session.flush_tier(None) - c = self.session.open_cursor(self.uri) c["3"] = "3" self.check(c, 4) diff --git a/src/third_party/wiredtiger/test/suite/test_tiered06.py b/src/third_party/wiredtiger/test/suite/test_tiered06.py index c797936a82b..d1eb9feae6f 100755 --- a/src/third_party/wiredtiger/test/suite/test_tiered06.py +++ b/src/third_party/wiredtiger/test/suite/test_tiered06.py @@ -47,14 +47,7 @@ class test_tiered06(wttest.WiredTigerTestCase): pdb.set_trace() def get_local_storage_source(self): - local = self.conn.get_storage_source('local_store') - - # Note: do not call local.terminate() . - # Since the local_storage extension has been loaded as a consequence of the - # wiredtiger_open call, WiredTiger already knows to call terminate when the connection - # closes. Calling it twice would attempt to free the same memory twice. - local.terminate = None - return local + return self.conn.get_storage_source('local_store') def test_local_basic(self): # Test some basic functionality of the storage source API, calling @@ -133,6 +126,7 @@ class test_tiered06(wttest.WiredTigerTestCase): self.assertEquals(fs.fs_directory_list(session, '', ''), ['foobar']) fs.terminate(session) + local.terminate(session) def test_local_write_read(self): # Write and read to a file non-sequentially. @@ -191,6 +185,7 @@ class test_tiered06(wttest.WiredTigerTestCase): else: self.assertEquals(in_block, a_block) fh.close(session) + local.terminate(session) def create_with_fs(self, fs, fname): session = self.session @@ -347,5 +342,12 @@ class test_tiered06(wttest.WiredTigerTestCase): self.check_dirlist(fs1, 'be', ['beagle']) self.check_dirlist(fs1, 'x', []) + # Terminate just one of the custom file systems. + # We should be able to terminate file systems, but we should + # also be able to terminate the storage source without terminating + # all the file systems we created. + fs1.terminate(session) + local.terminate(session) + if __name__ == '__main__': wttest.run() diff --git a/src/third_party/wiredtiger/test/suite/test_tiered08.py b/src/third_party/wiredtiger/test/suite/test_tiered08.py new file mode 100755 index 00000000000..af5e7af32bc --- /dev/null +++ b/src/third_party/wiredtiger/test/suite/test_tiered08.py @@ -0,0 +1,145 @@ +#!/usr/bin/env python +# +# Public Domain 2014-present MongoDB, Inc. +# Public Domain 2008-2014 WiredTiger, Inc. +# +# This is free and unencumbered software released into the public domain. +# +# Anyone is free to copy, modify, publish, use, compile, sell, or +# distribute this software, either in source code form or as a compiled +# binary, for any purpose, commercial or non-commercial, and by any +# means. +# +# In jurisdictions that recognize copyright laws, the author or authors +# of this software dedicate any and all copyright interest in the +# software to the public domain. We make this dedication for the benefit +# of the public at large and to the detriment of our heirs and +# successors. We intend this dedication to be an overt act of +# relinquishment in perpetuity of all present and future rights to this +# software under copyright law. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +# IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. +# +# [TEST_TAGS] +# tiered_storage:checkpoint +# tiered_storage:flush_tier +# [END_TAGS] +# + +import os, threading, time, wiredtiger, wttest +from wiredtiger import stat +from wtthread import checkpoint_thread, flush_tier_thread + +# test_tiered08.py +# Run background checkpoints and flush_tier operations while inserting +# data into a table from another thread. +class test_tiered08(wttest.WiredTigerTestCase): + + batch_size = 100000 + + # Keep inserting keys until we've done this many flush and checkpoint ops. + ckpt_flush_target = 10 + + uri = "table:test_tiered08" + + auth_token = "test_token" + bucket = "mybucket" + bucket_prefix = "pfx_" + extension_name = "local_store" + + def conn_config(self): + if not os.path.exists(self.bucket): + os.mkdir(self.bucket) + return \ + 'statistics=(fast),' + \ + 'tiered_storage=(auth_token=%s,' % self.auth_token + \ + 'bucket=%s,' % self.bucket + \ + 'bucket_prefix=%s,' % self.bucket_prefix + \ + 'name=%s),tiered_manager=(wait=0)' % self.extension_name + + # Load the local store extension, but skip the test if it is missing. + def conn_extensions(self, extlist): + extlist.skip_if_missing = True + extlist.extension('storage_sources', self.extension_name) + + def get_stat(self, stat): + stat_cursor = self.session.open_cursor('statistics:') + val = stat_cursor[stat][2] + stat_cursor.close() + return val + + def key_gen(self, i): + return 'KEY' + str(i) + + def value_gen(self, i): + return 'VALUE_' + 'filler' * (i % 12) + str(i) + + # Populate the test table. Keep adding keys until the desired number of flush and + # checkpoint operations have happened. + def populate(self): + ckpt_count = 0 + flush_count = 0 + nkeys = 0 + + self.pr('Populating tiered table') + c = self.session.open_cursor(self.uri, None, None) + while ckpt_count < self.ckpt_flush_target or flush_count < self.ckpt_flush_target: + for i in range(nkeys, nkeys + self.batch_size): + c[self.key_gen(i)] = self.value_gen(i) + nkeys += self.batch_size + ckpt_count = self.get_stat(stat.conn.txn_checkpoint) + flush_count = self.get_stat(stat.conn.flush_tier) + c.close() + return nkeys + + def verify(self, key_count): + self.pr('Verifying tiered table') + c = self.session.open_cursor(self.uri, None, None) + for i in range(key_count): + self.assertEqual(c[self.key_gen(i)], self.value_gen(i)) + c.close() + + def test_tiered08(self): + + # FIXME-WT-7833 + # This test can trigger races in file handle access during flush_tier. + # We will re-enable it when that is fixed. + return + + cfg = self.conn_config() + self.pr('Config is: ' + cfg) + intl_page = 'internal_page_max=16K' + base_create = 'key_format=S,value_format=S,' + intl_page + self.session.create(self.uri, base_create) + + done = threading.Event() + ckpt = checkpoint_thread(self.conn, done) + flush = flush_tier_thread(self.conn, done) + + # Start background threads and give them a chance to start. + ckpt.start() + flush.start() + time.sleep(0.5) + + key_count = self.populate() + + done.set() + flush.join() + ckpt.join() + + self.verify(key_count) + + self.close_conn() + self.pr('Reopening tiered table') + self.reopen_conn() + + self.verify(key_count) + +if __name__ == '__main__': + wttest.run() diff --git a/src/third_party/wiredtiger/test/suite/test_timestamp04.py b/src/third_party/wiredtiger/test/suite/test_timestamp04.py index bc2f8669bcd..9aff0351e2d 100644 --- a/src/third_party/wiredtiger/test/suite/test_timestamp04.py +++ b/src/third_party/wiredtiger/test/suite/test_timestamp04.py @@ -150,6 +150,10 @@ class test_timestamp04(wttest.WiredTigerTestCase, suite_subprocess): # Setup an oldest timestamp to ensure state remains in cache. if k == 1: self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(1)) + cur_ts_log.close() + cur_ts_nolog.close() + cur_nots_log.close() + cur_nots_nolog.close() # Scenario: 1 # Check that we see all the inserted values(i.e 1) in all tables @@ -217,6 +221,10 @@ class test_timestamp04(wttest.WiredTigerTestCase, suite_subprocess): self.conn.set_timestamp('oldest_timestamp=' + stable_ts) # Update the values again in preparation for rolling back more. + cur_ts_log = self.session.open_cursor(self.table_ts_log) + cur_ts_nolog = self.session.open_cursor(self.table_ts_nolog) + cur_nots_log = self.session.open_cursor(self.table_nots_log) + cur_nots_nolog = self.session.open_cursor(self.table_nots_nolog) for k in keys: cur_nots_log[k] = 2 cur_nots_nolog[k] = 2 @@ -224,6 +232,10 @@ class test_timestamp04(wttest.WiredTigerTestCase, suite_subprocess): cur_ts_log[k] = 2 cur_ts_nolog[k] = 2 self.session.commit_transaction('commit_timestamp=' + timestamp_str(k + key_range)) + cur_ts_log.close() + cur_ts_nolog.close() + cur_nots_log.close() + cur_nots_nolog.close() # Scenario: 3 # Check that we see all values updated (i.e 2) in all tables. diff --git a/src/third_party/wiredtiger/test/suite/test_timestamp06.py b/src/third_party/wiredtiger/test/suite/test_timestamp06.py index e94d4657df7..d958a3ba3ed 100644 --- a/src/third_party/wiredtiger/test/suite/test_timestamp06.py +++ b/src/third_party/wiredtiger/test/suite/test_timestamp06.py @@ -154,6 +154,8 @@ class test_timestamp06(wttest.WiredTigerTestCase, suite_subprocess): cur_ts_nolog[k] = 3 self.session.commit_transaction('commit_timestamp=' + timestamp_str(301)) + cur_ts_log.close() + cur_ts_nolog.close() # Scenario: 1 # Check that we see all the latest values (i.e. 3) as per transaction diff --git a/src/third_party/wiredtiger/test/suite/test_timestamp18.py b/src/third_party/wiredtiger/test/suite/test_timestamp18.py index e6d0ac0ab0b..746551b1dec 100644 --- a/src/third_party/wiredtiger/test/suite/test_timestamp18.py +++ b/src/third_party/wiredtiger/test/suite/test_timestamp18.py @@ -26,6 +26,11 @@ # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. # +# [TEST_TAGS] +# transactions:mixed_mode_timestamps +# verify:prepare +# [END_TAGS] +# # test_timestamp18.py # Mixing timestamped and non-timestamped writes. # diff --git a/src/third_party/wiredtiger/test/suite/test_timestamp22.py b/src/third_party/wiredtiger/test/suite/test_timestamp22.py index 8dcbfbc3cd1..d03e18d8d72 100755 --- a/src/third_party/wiredtiger/test/suite/test_timestamp22.py +++ b/src/third_party/wiredtiger/test/suite/test_timestamp22.py @@ -417,6 +417,8 @@ class test_timestamp22(wttest.WiredTigerTestCase): commit_ts = self.oldest_ts if durable_ts < commit_ts: durable_ts = commit_ts + if durable_ts <= self.stable_ts: + durable_ts = self.stable_ts + 1 value = self.gen_value(iternum, commit_ts) self.updates(value, ds, do_prepare, commit_ts, durable_ts, read_ts) diff --git a/src/third_party/wiredtiger/test/suite/test_truncate01.py b/src/third_party/wiredtiger/test/suite/test_truncate01.py index c15c7ea2f92..39bffc5f163 100644 --- a/src/third_party/wiredtiger/test/suite/test_truncate01.py +++ b/src/third_party/wiredtiger/test/suite/test_truncate01.py @@ -26,6 +26,10 @@ # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. # +# [TEST_TAGS] +# truncate +# [END_TAGS] +# # test_truncate01.py # session level operations on tables # diff --git a/src/third_party/wiredtiger/test/suite/test_txn01.py b/src/third_party/wiredtiger/test/suite/test_txn01.py index 17677b911fa..9f77ed3e1eb 100644 --- a/src/third_party/wiredtiger/test/suite/test_txn01.py +++ b/src/third_party/wiredtiger/test/suite/test_txn01.py @@ -25,6 +25,10 @@ # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. +# +# [TEST_TAGS] +# transactions +# [END_TAGS] import wiredtiger, wttest from wtscenario import make_scenarios diff --git a/src/third_party/wiredtiger/test/suite/test_txn04.py b/src/third_party/wiredtiger/test/suite/test_txn04.py index f09e82bd0bd..ca41c90cf36 100644 --- a/src/third_party/wiredtiger/test/suite/test_txn04.py +++ b/src/third_party/wiredtiger/test/suite/test_txn04.py @@ -26,6 +26,11 @@ # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. # +# [TEST_TAGS] +# backup +# recovery +# [END_TAGS] +# # test_txn04.py # Transactions: hot backup and recovery # diff --git a/src/third_party/wiredtiger/test/suite/test_txn19.py b/src/third_party/wiredtiger/test/suite/test_txn19.py index 128375e577d..18c0636d603 100755 --- a/src/third_party/wiredtiger/test/suite/test_txn19.py +++ b/src/third_party/wiredtiger/test/suite/test_txn19.py @@ -26,6 +26,10 @@ # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. # +# [TEST_TAGS] +# recovery:log_files +# [END_TAGS] +# # test_txn19.py # Transactions: test recovery with corrupted log files # @@ -290,14 +294,13 @@ class test_txn19(wttest.WiredTigerTestCase, suite_subprocess): expect_fail = self.expect_recovery_failure() if expect_fail: - with self.expectedStdoutPattern('Failed wiredtiger_open'): - errmsg = '/WT_TRY_SALVAGE: database corruption detected/' - if self.kind == 'removal': - errmsg = '/No such file or directory/' - if self.kind == 'truncate': - errmsg = '/failed to read 128 bytes at offset 0/' - self.assertRaisesWithMessage(wiredtiger.WiredTigerError, - lambda: self.reopen_conn(newdir, self.base_config), errmsg) + errmsg = '/WT_TRY_SALVAGE: database corruption detected/' + if self.kind == 'removal': + errmsg = '/No such file or directory/' + if self.kind == 'truncate': + errmsg = '/failed to read 128 bytes at offset 0/' + self.assertRaisesWithMessage(wiredtiger.WiredTigerError, + lambda: self.reopen_conn(newdir, self.base_config), errmsg) else: if self.expect_warning_corruption(): with self.expectedStdoutPattern('log file .* corrupted'): @@ -488,9 +491,8 @@ class test_txn19_meta(wttest.WiredTigerTestCase, suite_subprocess): errmsg = '/is smaller than allocation size; file size=0, alloc size=4096/' if self.kind == 'removal': errmsg = '/No such file or directory/' - with self.expectedStdoutPattern('Failed wiredtiger_open'): - self.assertRaisesWithMessage(wiredtiger.WiredTigerError, - lambda: self.reopen_conn(dir, self.conn_config), errmsg) + self.assertRaisesWithMessage(wiredtiger.WiredTigerError, + lambda: self.reopen_conn(dir, self.conn_config), errmsg) else: # On non-windows platforms, we capture the renaming of WiredTiger.wt file. if os.name != 'nt' and self.filename == 'WiredTiger.turtle' and self.kind == 'removal': @@ -548,10 +550,9 @@ class test_txn19_meta(wttest.WiredTigerTestCase, suite_subprocess): # an error during the wiredtiger_open. But the nature of the # messages produced during the error is variable by which case # it is, and even variable from system to system. - with self.expectedStdoutPattern('.'): - self.assertRaisesWithMessage(wiredtiger.WiredTigerError, - lambda: self.reopen_conn(salvagedir, salvage_config), - '/.*/') + self.assertRaisesWithMessage(wiredtiger.WiredTigerError, + lambda: self.reopen_conn(salvagedir, salvage_config), + '/.*/') if __name__ == '__main__': wttest.run() diff --git a/src/third_party/wiredtiger/test/suite/test_txn22.py b/src/third_party/wiredtiger/test/suite/test_txn22.py index b96881c3ace..310c6f8a829 100755 --- a/src/third_party/wiredtiger/test/suite/test_txn22.py +++ b/src/third_party/wiredtiger/test/suite/test_txn22.py @@ -152,10 +152,9 @@ class test_txn22(wttest.WiredTigerTestCase, suite_subprocess): # Without salvage, they result in an error during the wiredtiger_open. # But the nature of the messages produced during the error is variable # by which case it is, and even variable from system to system. - with self.expectedStdoutPattern('.'): - self.assertRaisesWithMessage(wiredtiger.WiredTigerError, - lambda: self.reopen_conn(salvagedir, self.base_config), - '/.*/') + self.assertRaisesWithMessage(wiredtiger.WiredTigerError, + lambda: self.reopen_conn(salvagedir, self.base_config), + '/.*/') self.reopen_conn(salvagedir, salvage_config) if self.filename == 'test_txn22': @@ -165,10 +164,9 @@ class test_txn22(wttest.WiredTigerTestCase, suite_subprocess): # an error during the wiredtiger_open. But the nature of the # messages produced during the error is variable by which case # it is, and even variable from system to system. - with self.expectedStdoutPattern('.'): - self.assertRaisesWithMessage(wiredtiger.WiredTigerError, - lambda: self.reopen_conn(salvagedir, salvage_config), - '/.*/') + self.assertRaisesWithMessage(wiredtiger.WiredTigerError, + lambda: self.reopen_conn(salvagedir, salvage_config), + '/.*/') # The test may output the following error message while opening a file that # does not exist. Ignore that. diff --git a/src/third_party/wiredtiger/test/suite/test_txn26.py b/src/third_party/wiredtiger/test/suite/test_txn26.py index 75633b275e3..4618585b7f1 100644 --- a/src/third_party/wiredtiger/test/suite/test_txn26.py +++ b/src/third_party/wiredtiger/test/suite/test_txn26.py @@ -25,6 +25,10 @@ # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. +# +# [TEST_TAGS] +# transactions:timestamps +# [END_TAGS] import wiredtiger, wttest diff --git a/src/third_party/wiredtiger/test/suite/test_util11.py b/src/third_party/wiredtiger/test/suite/test_util11.py index 1e6046da291..583ad5b2b8d 100644 --- a/src/third_party/wiredtiger/test/suite/test_util11.py +++ b/src/third_party/wiredtiger/test/suite/test_util11.py @@ -25,6 +25,10 @@ # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. +# +# [TEST_TAGS] +# wt_util +# [END_TAGS] import os, struct from suite_subprocess import suite_subprocess diff --git a/src/third_party/wiredtiger/test/suite/test_version.py b/src/third_party/wiredtiger/test/suite/test_version.py index 977f0897c11..a73bbff9181 100644 --- a/src/third_party/wiredtiger/test/suite/test_version.py +++ b/src/third_party/wiredtiger/test/suite/test_version.py @@ -25,6 +25,10 @@ # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. +# +# [TEST_TAGS] +# connection_api +# [END_TAGS] import wiredtiger, wttest diff --git a/src/third_party/wiredtiger/test/suite/wttest.py b/src/third_party/wiredtiger/test/suite/wttest.py index 92daaa62e37..1d22162f535 100755 --- a/src/third_party/wiredtiger/test/suite/wttest.py +++ b/src/third_party/wiredtiger/test/suite/wttest.py @@ -288,8 +288,17 @@ class WiredTigerTestCase(unittest.TestCase): # the name if the file exists. @staticmethod def findExtension(dirname, libname): - pat = os.path.join(WiredTigerTestCase._builddir, 'ext', - dirname, libname, '.libs', 'libwiredtiger_*.so') + pat = '' + # When scanning for the extension library, we need to account for + # the binary paths produced by libtool and CMake. Libtool will export + # the library under '.libs'. + if os.path.exists(os.path.join(WiredTigerTestCase._builddir, 'ext', + dirname, libname, '.libs')): + pat = os.path.join(WiredTigerTestCase._builddir, 'ext', + dirname, libname, '.libs', 'libwiredtiger_*.so') + else: + pat = os.path.join(WiredTigerTestCase._builddir, 'ext', + dirname, libname, 'libwiredtiger_*.so') filenames = glob.glob(pat) if len(filenames) > 1: raise Exception(self.shortid() + ": " + ext + @@ -357,13 +366,7 @@ class WiredTigerTestCase(unittest.TestCase): # avoid confusion. sys.stdout.flush() conn_param = 'create,error_prefix="%s",%s' % (self.shortid(), config) - try: - conn = self.wiredtiger_open(home, conn_param) - except wiredtiger.WiredTigerError as e: - print("Failed wiredtiger_open: dir '%s', config '%s'" % \ - (home, conn_param)) - raise e - return conn + return self.wiredtiger_open(home, conn_param) # Replacement for wiredtiger.wiredtiger_open that returns # a proxied connection that knows to close it itself at the diff --git a/src/third_party/wiredtiger/test/suite/wtthread.py b/src/third_party/wiredtiger/test/suite/wtthread.py index eb8dd49f7f0..6e15f85dfb6 100755 --- a/src/third_party/wiredtiger/test/suite/wtthread.py +++ b/src/third_party/wiredtiger/test/suite/wtthread.py @@ -43,6 +43,20 @@ class checkpoint_thread(threading.Thread): sess.checkpoint() sess.close() +class flush_tier_thread(threading.Thread): + def __init__(self, conn, done): + self.conn = conn + self.done = done + threading.Thread.__init__(self) + + def run(self): + sess = self.conn.open_session() + while not self.done.isSet(): + # Sleep for 25 milliseconds. + time.sleep(0.0025) + sess.flush_tier() + sess.close() + class backup_thread(threading.Thread): def __init__(self, conn, backup_dir, done): self.backup_dir = backup_dir @@ -81,20 +95,16 @@ class backup_thread(threading.Thread): uri = "file:" + next_file uris.append(uri) - # TODO: We want a self.assertTrue here - be need to be a - # wttest to do that.. - if not compare_tables( - self, sess, uris, "checkpoint=WiredTigerCheckpoint"): - print("Error: checkpoint tables differ.") - else: - wttest.WiredTigerTestCase.printVerbose( - 3, "Checkpoint tables match") + # Add an assert to stop running the test if any difference in table contents + # is found. We would have liked to use self.assertTrue instead, but are unable + # to because backup_thread does not support this method unless it is a wttest. + wttest.WiredTigerTestCase.printVerbose(3, "Testing if checkpoint tables match:") + assert compare_tables(self, sess, uris) == True + wttest.WiredTigerTestCase.printVerbose(3, "Checkpoint tables match") - if not compare_tables(self, bkp_session, uris): - print("Error: backup tables differ.") - else: - wttest.WiredTigerTestCase.printVerbose( - 3, "Backup tables match") + wttest.WiredTigerTestCase.printVerbose(3, "Testing if backup tables match:") + assert compare_tables(self, bkp_session, uris) == True + wttest.WiredTigerTestCase.printVerbose(3, "Backup tables match") finally: if bkp_conn != None: bkp_conn.close() |