diff options
| author | Luke Chen <luke.chen@mongodb.com> | 2018-01-29 13:44:37 +1100 |
|---|---|---|
| committer | Luke Chen <luke.chen@mongodb.com> | 2018-01-29 13:51:20 +1100 |
| commit | 1b4ae523b52673cbad589870ebc79cd55a4fe6a4 (patch) | |
| tree | 5469f11ec6d2b43012ffa9ef9c012e7f38408f32 | |
| parent | 08fe3aba7e9a5c8fa54426b68a503a852312d004 (diff) | |
| download | mongo-1b4ae523b52673cbad589870ebc79cd55a4fe6a4.tar.gz | |
Import wiredtiger: b85bcfde3b7e41a35017385fd219553e7028c427 from branch mongodb-3.6
ref: d8f244717b..b85bcfde3b
for: 3.6.3
WT-2705 High throughput cache bound many threaded workloads can experience long latencies
WT-3074 Automate a test to stress eviction walk with many active trees
WT-3079 Make sure eviction visits all trees
WT-3133 Detect or track long latency operations
WT-3295 Allow LSM to merge into custom data sources
WT-3565 Test and understand mixed timestamp/no-timestamp usage to same data
WT-3587 Remove HAVE_VERBOSE conditional compilation
WT-3597 Add a diagnostic check for updates to the same key out of timestamp order
WT-3632 Increase how granularly cache usage settings can be configured
WT-3654 Fix warning in Windows build on evergreen
WT-3695 format failed to report a stuck cache
WT-3716 Restore the WT_VERB_TEMPORARY verbose flag.
WT-3720 flags macros cast flags to unsigned values, hiding warnings.
WT-3725 Add statistics to locks around timestamp structures
WT-3732 Handle adding WT indices while cursors on the table are open
WT-3734 Fix undefined behavior in verbose output
WT-3738 Review internal session allocation accounting
WT-3740 race in page dirty-byte decrement.
WT-3750 Fast-path fs_directory_list of a single file.
WT-3753 Building on Windows --enable-java
WT-3767 Avoid lookaside instantiation for faster reads
WT-3768 Lookaside optimization: birthmarks
WT-3769 Fix a bug in reverse cursor walks with particular delete patterns and prefix compression enabled
WT-3772 Hot backup causes uncontrolled growth of WiredTigerPreplog files
WT-3774 Enhance Python lookaside testing to cover cursor modify
WT-3775 Improve commit timestamp is older than oldest timestamp error message
WT-3776 Cursor remove operation unpins page too early
WT-3779 Add support for string formats with WT_CURSOR::modify
WT-3780 Improve error messages on invalid WT_CURSOR::modify usage
WT-3783 Fix transaction isolation to use the correct enum
WT-3786 Transactions with timestamps should read their writes
WT-3787 test_compact02 failed as compaction halted due to eviction pressure
WT-3790 Switch statistics to rdtsc from epoch calls
WT-3792 LSM version 1 metadata incompatibility
WT-3793 WiredTiger page debug dump functions should unpack integer keys
WT-3794 Coverity 1383547 and lint
WT-3795 lint cleanups for the op-tracking software, reduce record write size.
WT-3796 Report a better error message if transaction commit fails
WT-3799 Test/format with timestamps enabled pin cache full
WT-3800 test_wt2834_join_bloom_fix hang
WT-3806 Make sure rdtsc values move forward in time
WT-3807 clang static analysis updates
WT-3809 Fix a bug in lookaside related to birthmarks
WT-3810 wt_rdtsc calibration needs to be longer and verify validity
WT-3811 Add basic ability to visualise operation tracking
WT-3812 debugging page output should handle complex key/value items.
WT-3816 Enable prefix compression on the lookaside table
WT-3818 __rec_txn_read() code order cleanup
WT-3819 clang static analysis improvements
WT-3820 Add a WT_SESSION.breakpoint method for Python debugging.
WT-3822 Update WiredTiger copyrights to 2018
WT-3824 tsc_nsec_ratio can compute to zero and cause divide by zero bug
WT-3825 Fix calculation of CPU ticks per unit time
WT-3826 random-abort test failure
WT-3827 test_compact02 failure
WT-3828 Link error on OS/X for __wt_process data reference
WT-3831 uninitialized buffer value in statlog server path comparison
WT-3832 Fixup shell script warning messages
WT-3833 test/format cache_minimum value error
WT-3835 cursor remove tries to return a key that doesn't exist
WT-3841 Fix error message pattern in timestamp09
WT-3842 full-build Friday & lint
WT-3844 Checkpoints can hang on limbo pages
WT-3845 Compiler warning in examples using GCC 5.4.0
WT-3846 Refine operation tracking visualization tool
WT-3852 Update debugging when committing at an earlier timestamp
WT-3853 LSM version 1 metadata incompatibility
WT-3860 Fix a list of lint issues
WT-3862 Remove invalid diagnostic assertion during reconciliation
WT-3863 Record the clock frequency in the optrack log file, so we can convert CPU cycles to nanoseconds when decoding the files.
WT-3865 avoid cache misses in WiredTiger tree-walk and hazard functions
WT-3871 format should read from a timestamp
WT-3874 Use custom collator for diagnostic verification of keys
725 files changed, 8142 insertions, 3255 deletions
diff --git a/src/third_party/wiredtiger/LICENSE b/src/third_party/wiredtiger/LICENSE index 8abd469a7b1..50389d6f4df 100644 --- a/src/third_party/wiredtiger/LICENSE +++ b/src/third_party/wiredtiger/LICENSE @@ -1,4 +1,4 @@ -Copyright (c) 2014-2017 MongoDB, Inc. +Copyright (c) 2014-2018 MongoDB, Inc. Copyright (c) 2008-2014 WiredTiger, Inc. All rights reserved. diff --git a/src/third_party/wiredtiger/NEWS b/src/third_party/wiredtiger/NEWS index ffcefd5f8c1..7bf3b0e7edb 100644 --- a/src/third_party/wiredtiger/NEWS +++ b/src/third_party/wiredtiger/NEWS @@ -1,6 +1,35 @@ Ticket reference tags refer to tickets in the MongoDB JIRA tracking system: https://jira.mongodb.org +WiredTiger release 3.0.0, 2018-01-08 +------------------------------------ + +See the upgrading documentation for details of API and behavior changes. + +Significant changes: +* WT-3039 Change the log file format to record a previous LSN record +* WT-3181 Add support for application defined transaction IDs via a mechanism called timestamps. +* WT-3310 Add support to WT_SESSION::alter to change table log setting +* WT-3389 Restructure page split code to hold a split generation for the entire operation. +* WT-3406 Fix a bug in reconciliation so that it ignores concurrent updates. +* WT-3418 Fix a block manager race in tree close/open +* WT-3435 Improvements to the cache overflow mechanism aka lookaside +* WT-3437 Improvements to auto tuning of number of eviction workers +* WT-3440 Add a log record when starting a checkpoint. +* WT-3461 Avoid hangs when system clocks move backwards by using CLOCK_MONOTONIC for pthread_cond_timedwait if possible. +* WT-3490 Fix a bug in WT_CURSOR.modify unaligned size_t access. +* WT-3495 Fix a bug so we don't ftruncate if log cursors are open +* WT-3497 Improve logging message when hitting the configured session limits +* WT-3537 Split pages in memory when nothing can be written +* WT-3556 Remove wtstats support +* WT-3681 Change recovery so that it doesn't truncate the last log file +* WT-3683 Allow eviction of clean pages with history when cache is stuck +* WT-3710 Get a page-level lock to ensure page splits are single threaded +* WT-3752 Allow trimming of obsolete modify updates. + +See JIRA changelog for a full listing: +https://jira.mongodb.org/projects/WT/versions/18401 + WiredTiger release 2.9.3, 2017-06-27 ------------------------------------ diff --git a/src/third_party/wiredtiger/README b/src/third_party/wiredtiger/README index 4def09abba6..234038d21d0 100644 --- a/src/third_party/wiredtiger/README +++ b/src/third_party/wiredtiger/README @@ -1,6 +1,6 @@ -WiredTiger 3.0.0: (June 27, 2017) +WiredTiger 3.0.1: (January 8, 2018) -This is version 3.0.0 of WiredTiger. +This is version 3.0.1 of WiredTiger. WiredTiger release packages and documentation can be found at: @@ -8,7 +8,7 @@ WiredTiger release packages and documentation can be found at: The documentation for this specific release can be found at: - http://source.wiredtiger.com/3.0.0/index.html + http://source.wiredtiger.com/3.0.1/index.html The WiredTiger source code can be found at: diff --git a/src/third_party/wiredtiger/RELEASE_INFO b/src/third_party/wiredtiger/RELEASE_INFO index ccdff34f2d5..38189c7be01 100644 --- a/src/third_party/wiredtiger/RELEASE_INFO +++ b/src/third_party/wiredtiger/RELEASE_INFO @@ -1,6 +1,6 @@ WIREDTIGER_VERSION_MAJOR=3 WIREDTIGER_VERSION_MINOR=0 -WIREDTIGER_VERSION_PATCH=0 +WIREDTIGER_VERSION_PATCH=1 WIREDTIGER_VERSION="$WIREDTIGER_VERSION_MAJOR.$WIREDTIGER_VERSION_MINOR.$WIREDTIGER_VERSION_PATCH" WIREDTIGER_RELEASE_DATE=`date "+%B %e, %Y"` diff --git a/src/third_party/wiredtiger/SConstruct b/src/third_party/wiredtiger/SConstruct index 96012456b34..2646d51378e 100644 --- a/src/third_party/wiredtiger/SConstruct +++ b/src/third_party/wiredtiger/SConstruct @@ -28,6 +28,9 @@ AddOption("--enable-diagnostic", dest="diagnostic", action="store_true", default AddOption("--enable-lz4", dest="lz4", type="string", nargs=1, action="store", help="Use LZ4 compression") +AddOption("--enable-java", dest="lang-java", type="string", nargs=1, action="store", + help="Build java extension, specify location of swig.exe binary and Java JDK dir separated by comma") + AddOption("--enable-python", dest="lang-python", type="string", nargs=1, action="store", help="Build Python extension, specify location of swig.exe binary") @@ -37,9 +40,6 @@ AddOption("--enable-snappy", dest="snappy", type="string", nargs=1, action="stor AddOption("--enable-tcmalloc", dest="tcmalloc", type="string", nargs=1, action="store", help="Use TCMalloc for memory allocation") -AddOption("--enable-verbose", dest="verbose", action="store_true", default=False, - help="Configure WiredTiger to support the verbose configuration string to wiredtiger_open") - AddOption("--enable-zlib", dest="zlib", type="string", nargs=1, action="store", help="Use zlib compression") @@ -183,9 +183,6 @@ if GetOption("lang-python"): env.Append(LIBPATH=[distutils.sysconfig.PREFIX + r"\libs"]) env.Append(CPPPATH=[distutils.sysconfig.get_python_inc()]) -if GetOption("verbose"): - env.Append(CPPDEFINES = ["HAVE_VERBOSE"]) - # Build WiredTiger.h file # @@ -355,6 +352,61 @@ if GetOption("lang-python"): Default(swiginstall, copySwig) +# Javap SWIG wrapper for WiredTiger +enableJava = GetOption("lang-java") +if enableJava and enableJava.count(",") == 1: + enableJavaPaths = enableJava.split(',') + + swigExe = enableJavaPaths[0] + javaPath = enableJavaPaths[1] + conf.env.Append(CPPPATH=[ javaPath + '/include']) + conf.env.Append(CPPPATH=[ javaPath + '/include/win32']) + + swigJavaFiles = ["lang/java/src/com/wiredtiger/db/AsyncOp.java", + "lang/java/src/com/wiredtiger/db/AsyncOpType.java", + "lang/java/src/com/wiredtiger/db/Connection.java", + "lang/java/src/com/wiredtiger/db/Cursor.java", + "lang/java/src/com/wiredtiger/db/Modify.java", + "lang/java/src/com/wiredtiger/db/SearchStatus.java", + "lang/java/src/com/wiredtiger/db/Session.java", + "lang/java/src/com/wiredtiger/db/WT_ITEM_HOLD.java", + "lang/java/src/com/wiredtiger/db/WT_MODIFY_LIST.java", + "lang/java/src/com/wiredtiger/db/wiredtiger.java", + "lang/java/src/com/wiredtiger/db/wiredtigerConstants.java", + "lang/java/src/com/wiredtiger/db/wiredtigerJNI.java"] + + swigCFile = "wiredtiger_wrap.c" + + swigFiles = env.Command( + swigJavaFiles + [swigCFile], '', + '"' + swigExe + '" -Wall -v -java -nodefaultctor -nodefaultdtor -package com.wiredtiger.db -outdir lang/java/src/com/wiredtiger/db -o wiredtiger_wrap.c lang/java/wiredtiger.i') + env.Depends(swigFiles, wtheader) + objectJavaWrap = env.Object(swigCFile) + env.Depends(objectJavaWrap, swigCFile) + + # + # Dynamically Loaded Library - wiredtiger_java.dll + wtjavadll = env.SharedLibrary( + target="wiredtiger_java", + source=wt_objs + [objectJavaWrap] + ['build_win/wiredtiger.def'], LIBS=wtlibs) + + env.Depends(wtjavadll, [filelistfile, version_file]) + Default(wtjavadll) + + # + # wiredtiger.jar + env['JAVAC'] = '"' + javaPath + '/bin/javac.exe"' + env['JAR'] = '"' + javaPath + '/bin/jar.exe"' + # Build classes + wtClasses = env.Java('lang/java/build', 'lang/java/src/') + env.Depends(wtClasses, swigJavaFiles) + # Pack classes in jar + wtJar = env.Command( 'lang/java/wiredtiger.jar', 'lang/java/build', env['JAR'] + " -cf $TARGET -C $SOURCE .") + env.Depends(wtJar, wtClasses) + Default(wtJar) +else: + print "Error using --enable-java, this option may contain two paths separated by comma, the first is the swig.exe binary and the second is the Java JDK directory. e.g. C:\Python27\python.exe C:\Python27\Scripts\scons.py --enable-java=\"C:\Program Files\swigwin-3.0.12\swig.exe\",\"C:\Program Files\Java\jdk1.8.0_151\"" + # Shim library of functions to emulate POSIX on Windows shim = env.Library("window_shim", ["test/windows/windows_shim.c"]) diff --git a/src/third_party/wiredtiger/api/leveldb/leveldb_wt.h b/src/third_party/wiredtiger/api/leveldb/leveldb_wt.h index b167e03192e..529e85acdca 100644 --- a/src/third_party/wiredtiger/api/leveldb/leveldb_wt.h +++ b/src/third_party/wiredtiger/api/leveldb/leveldb_wt.h @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/bench/workgen/runner/example_simple.py b/src/third_party/wiredtiger/bench/workgen/runner/example_simple.py index 626f7ca64a5..d33029cea1b 100755 --- a/src/third_party/wiredtiger/bench/workgen/runner/example_simple.py +++ b/src/third_party/wiredtiger/bench/workgen/runner/example_simple.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/bench/workgen/runner/example_txn.py b/src/third_party/wiredtiger/bench/workgen/runner/example_txn.py index 1b22dc10aba..43c3caadf8c 100644 --- a/src/third_party/wiredtiger/bench/workgen/runner/example_txn.py +++ b/src/third_party/wiredtiger/bench/workgen/runner/example_txn.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/bench/workgen/runner/insert_test.py b/src/third_party/wiredtiger/bench/workgen/runner/insert_test.py index 8380c6ba3eb..220292db3ed 100644 --- a/src/third_party/wiredtiger/bench/workgen/runner/insert_test.py +++ b/src/third_party/wiredtiger/bench/workgen/runner/insert_test.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/bench/workgen/runner/maintain_low_dirty_cache.py b/src/third_party/wiredtiger/bench/workgen/runner/maintain_low_dirty_cache.py index 695dcc86087..15ed46ec3c9 100644 --- a/src/third_party/wiredtiger/bench/workgen/runner/maintain_low_dirty_cache.py +++ b/src/third_party/wiredtiger/bench/workgen/runner/maintain_low_dirty_cache.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/bench/workgen/runner/multi_btree_heavy_stress.py b/src/third_party/wiredtiger/bench/workgen/runner/multi_btree_heavy_stress.py index 94dacfc4311..4d4f6ab818e 100644 --- a/src/third_party/wiredtiger/bench/workgen/runner/multi_btree_heavy_stress.py +++ b/src/third_party/wiredtiger/bench/workgen/runner/multi_btree_heavy_stress.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/bench/workgen/runner/runner/__init__.py b/src/third_party/wiredtiger/bench/workgen/runner/runner/__init__.py index 2d60e1522f5..7e73117af42 100644 --- a/src/third_party/wiredtiger/bench/workgen/runner/runner/__init__.py +++ b/src/third_party/wiredtiger/bench/workgen/runner/runner/__init__.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/bench/workgen/runner/runner/core.py b/src/third_party/wiredtiger/bench/workgen/runner/runner/core.py index a8977d9593e..bd4bcd6dbe8 100644 --- a/src/third_party/wiredtiger/bench/workgen/runner/runner/core.py +++ b/src/third_party/wiredtiger/bench/workgen/runner/runner/core.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/bench/workgen/runner/runner/latency.py b/src/third_party/wiredtiger/bench/workgen/runner/runner/latency.py index 8eaa10693a9..0274ce9b917 100644 --- a/src/third_party/wiredtiger/bench/workgen/runner/runner/latency.py +++ b/src/third_party/wiredtiger/bench/workgen/runner/runner/latency.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/bench/workgen/runner/small_btree.py b/src/third_party/wiredtiger/bench/workgen/runner/small_btree.py index 2bcc0188f30..2cc3065cb2b 100644 --- a/src/third_party/wiredtiger/bench/workgen/runner/small_btree.py +++ b/src/third_party/wiredtiger/bench/workgen/runner/small_btree.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/bench/workgen/runner/small_btree_reopen.py b/src/third_party/wiredtiger/bench/workgen/runner/small_btree_reopen.py index 252dd02aae9..62e6a861200 100644 --- a/src/third_party/wiredtiger/bench/workgen/runner/small_btree_reopen.py +++ b/src/third_party/wiredtiger/bench/workgen/runner/small_btree_reopen.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/bench/workgen/setup.py b/src/third_party/wiredtiger/bench/workgen/setup.py index 9fb5fa7b73a..439b9c474fe 100644 --- a/src/third_party/wiredtiger/bench/workgen/setup.py +++ b/src/third_party/wiredtiger/bench/workgen/setup.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/bench/workgen/workgen.cxx b/src/third_party/wiredtiger/bench/workgen/workgen.cxx index 31e21e6f6c9..f89356b836a 100644 --- a/src/third_party/wiredtiger/bench/workgen/workgen.cxx +++ b/src/third_party/wiredtiger/bench/workgen/workgen.cxx @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/bench/workgen/workgen.h b/src/third_party/wiredtiger/bench/workgen/workgen.h index 2a116e1c89e..dc15ab06bf8 100644 --- a/src/third_party/wiredtiger/bench/workgen/workgen.h +++ b/src/third_party/wiredtiger/bench/workgen/workgen.h @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/bench/workgen/workgen.swig b/src/third_party/wiredtiger/bench/workgen/workgen.swig index a33705f4350..61baae44493 100644 --- a/src/third_party/wiredtiger/bench/workgen/workgen.swig +++ b/src/third_party/wiredtiger/bench/workgen/workgen.swig @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/bench/workgen/workgen/__init__.py b/src/third_party/wiredtiger/bench/workgen/workgen/__init__.py index ff665bf9398..516485e400c 100644 --- a/src/third_party/wiredtiger/bench/workgen/workgen/__init__.py +++ b/src/third_party/wiredtiger/bench/workgen/workgen/__init__.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/bench/workgen/workgen_func.c b/src/third_party/wiredtiger/bench/workgen/workgen_func.c index 5ce2146a8e4..b23e9e99a3b 100644 --- a/src/third_party/wiredtiger/bench/workgen/workgen_func.c +++ b/src/third_party/wiredtiger/bench/workgen/workgen_func.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/bench/workgen/workgen_func.h b/src/third_party/wiredtiger/bench/workgen/workgen_func.h index ec7ecf0a504..949c8474a2a 100644 --- a/src/third_party/wiredtiger/bench/workgen/workgen_func.h +++ b/src/third_party/wiredtiger/bench/workgen/workgen_func.h @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/bench/workgen/workgen_int.h b/src/third_party/wiredtiger/bench/workgen/workgen_int.h index c7a5a7121e9..33a8a8f492f 100644 --- a/src/third_party/wiredtiger/bench/workgen/workgen_int.h +++ b/src/third_party/wiredtiger/bench/workgen/workgen_int.h @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/bench/workgen/workgen_time.h b/src/third_party/wiredtiger/bench/workgen/workgen_time.h index f33eb64d9c9..c1a0a2906dd 100644 --- a/src/third_party/wiredtiger/bench/workgen/workgen_time.h +++ b/src/third_party/wiredtiger/bench/workgen/workgen_time.h @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/bench/workgen/wtperf.py b/src/third_party/wiredtiger/bench/workgen/wtperf.py index 2837be6d064..9da8c37fd3a 100644 --- a/src/third_party/wiredtiger/bench/workgen/wtperf.py +++ b/src/third_party/wiredtiger/bench/workgen/wtperf.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/bench/wtperf/config.c b/src/third_party/wiredtiger/bench/wtperf/config.c index 1a4f46f5c37..d93baeed2f6 100644 --- a/src/third_party/wiredtiger/bench/wtperf/config.c +++ b/src/third_party/wiredtiger/bench/wtperf/config.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/bench/wtperf/config_opt.h b/src/third_party/wiredtiger/bench/wtperf/config_opt.h index 68bcd3e45f1..d61046fb828 100644 --- a/src/third_party/wiredtiger/bench/wtperf/config_opt.h +++ b/src/third_party/wiredtiger/bench/wtperf/config_opt.h @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/bench/wtperf/idle_table_cycle.c b/src/third_party/wiredtiger/bench/wtperf/idle_table_cycle.c index d0baa786ba9..f84e9ddaed5 100644 --- a/src/third_party/wiredtiger/bench/wtperf/idle_table_cycle.c +++ b/src/third_party/wiredtiger/bench/wtperf/idle_table_cycle.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/bench/wtperf/misc.c b/src/third_party/wiredtiger/bench/wtperf/misc.c index da48c600589..295948fd6b7 100644 --- a/src/third_party/wiredtiger/bench/wtperf/misc.c +++ b/src/third_party/wiredtiger/bench/wtperf/misc.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/bench/wtperf/runners/evict-fairness.wtperf b/src/third_party/wiredtiger/bench/wtperf/runners/evict-fairness.wtperf new file mode 100644 index 00000000000..94f55f6bea9 --- /dev/null +++ b/src/third_party/wiredtiger/bench/wtperf/runners/evict-fairness.wtperf @@ -0,0 +1,20 @@ +# This configuration is used to test the eviction fairness of visiting all tables. +# Create a set of tables with uneven distribution of data +conn_config="cache_size=1G,eviction=(threads_max=8),file_manager=(close_idle_time=100000),checkpoint=(wait=20,log_size=2GB),session_max=1000" +table_config="type=file" +table_count=200 +icount=0 +random_range=1000000000 +pareto=0 +range_partition=true +report_interval=5 + +run_ops=1000000 +populate_threads=0 +icount=0 +threads=((count=60,inserts=1)) + +# Warn if a latency over 1 second is seen +max_latency=1000 +sample_interval=5 +sample_rate=1 diff --git a/src/third_party/wiredtiger/bench/wtperf/runners/get_ckpt.py b/src/third_party/wiredtiger/bench/wtperf/runners/get_ckpt.py index da188ad47d4..2b52e066954 100755 --- a/src/third_party/wiredtiger/bench/wtperf/runners/get_ckpt.py +++ b/src/third_party/wiredtiger/bench/wtperf/runners/get_ckpt.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/bench/wtperf/track.c b/src/third_party/wiredtiger/bench/wtperf/track.c index 13ca85aabfd..798ee5d2144 100644 --- a/src/third_party/wiredtiger/bench/wtperf/track.c +++ b/src/third_party/wiredtiger/bench/wtperf/track.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/bench/wtperf/wtperf.c b/src/third_party/wiredtiger/bench/wtperf/wtperf.c index 9e18df9a0e3..4adb3db3c6c 100644 --- a/src/third_party/wiredtiger/bench/wtperf/wtperf.c +++ b/src/third_party/wiredtiger/bench/wtperf/wtperf.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/bench/wtperf/wtperf.h b/src/third_party/wiredtiger/bench/wtperf/wtperf.h index b17d082ddcf..7fb370e0b5c 100644 --- a/src/third_party/wiredtiger/bench/wtperf/wtperf.h +++ b/src/third_party/wiredtiger/bench/wtperf/wtperf.h @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/bench/wtperf/wtperf_opt.i b/src/third_party/wiredtiger/bench/wtperf/wtperf_opt.i index 10aebf36b01..24f114abcb6 100644 --- a/src/third_party/wiredtiger/bench/wtperf/wtperf_opt.i +++ b/src/third_party/wiredtiger/bench/wtperf/wtperf_opt.i @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/bench/wtperf/wtperf_throttle.c b/src/third_party/wiredtiger/bench/wtperf/wtperf_throttle.c index 75dad09ed50..008516168ad 100644 --- a/src/third_party/wiredtiger/bench/wtperf/wtperf_throttle.c +++ b/src/third_party/wiredtiger/bench/wtperf/wtperf_throttle.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/bench/wtperf/wtperf_truncate.c b/src/third_party/wiredtiger/bench/wtperf/wtperf_truncate.c index 5b794009afb..1f910b9a3a4 100644 --- a/src/third_party/wiredtiger/bench/wtperf/wtperf_truncate.c +++ b/src/third_party/wiredtiger/bench/wtperf/wtperf_truncate.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/build_posix/aclocal/options.m4 b/src/third_party/wiredtiger/build_posix/aclocal/options.m4 index 20d8ed08db5..7d0df5d65ac 100644 --- a/src/third_party/wiredtiger/build_posix/aclocal/options.m4 +++ b/src/third_party/wiredtiger/build_posix/aclocal/options.m4 @@ -253,18 +253,6 @@ AC_ARG_WITH(timestamp-size, AC_MSG_RESULT($with_timestamp_size) AC_DEFINE_UNQUOTED(WT_TIMESTAMP_SIZE, [$with_timestamp_size], [Size of a transaction timestamp in bytes]) -AH_TEMPLATE(HAVE_VERBOSE, [Enable verbose message configuration.]) -AC_MSG_CHECKING(if --enable-verbose option specified) -AC_ARG_ENABLE(verbose, - [AS_HELP_STRING([--enable-verbose], - [Enable verbose message configuration.])], r=$enableval, r=no) -case "$r" in -no) wt_cv_enable_verbose=no;; -*) AC_DEFINE(HAVE_VERBOSE) - wt_cv_enable_verbose=yes;; -esac -AC_MSG_RESULT($wt_cv_enable_verbose) - AC_MSG_CHECKING(if --enable-zlib option specified) AC_ARG_ENABLE(zlib, [AS_HELP_STRING([--enable-zlib], diff --git a/src/third_party/wiredtiger/build_posix/aclocal/version-set.m4 b/src/third_party/wiredtiger/build_posix/aclocal/version-set.m4 index 5e54ad1cb69..fa3fed8638b 100644 --- a/src/third_party/wiredtiger/build_posix/aclocal/version-set.m4 +++ b/src/third_party/wiredtiger/build_posix/aclocal/version-set.m4 @@ -2,8 +2,8 @@ dnl build by dist/s_version VERSION_MAJOR=3 VERSION_MINOR=0 -VERSION_PATCH=0 -VERSION_STRING='"WiredTiger 3.0.0: (June 27, 2017)"' +VERSION_PATCH=1 +VERSION_STRING='"WiredTiger 3.0.1: (January 8, 2018)"' AC_SUBST(VERSION_MAJOR) AC_SUBST(VERSION_MINOR) diff --git a/src/third_party/wiredtiger/build_posix/aclocal/version.m4 b/src/third_party/wiredtiger/build_posix/aclocal/version.m4 index 3c5980dbaad..9a6918366e9 100644 --- a/src/third_party/wiredtiger/build_posix/aclocal/version.m4 +++ b/src/third_party/wiredtiger/build_posix/aclocal/version.m4 @@ -1,2 +1,2 @@ dnl WiredTiger product version for AC_INIT. Maintained by dist/s_version -3.0.0 +3.0.1 diff --git a/src/third_party/wiredtiger/build_win/wiredtiger_config.h b/src/third_party/wiredtiger/build_win/wiredtiger_config.h index fa8425c4936..55431f59fae 100644 --- a/src/third_party/wiredtiger/build_win/wiredtiger_config.h +++ b/src/third_party/wiredtiger/build_win/wiredtiger_config.h @@ -124,9 +124,6 @@ /* Define to 1 if you have the <unistd.h> header file. */ /* #undef HAVE_UNISTD_H */ -/* Enable verbose message configuration. */ -/* #undef HAVE_VERBOSE */ - /* Define to 1 if you have the <x86intrin.h> header file. */ #define HAVE_X86INTRIN_H 1 diff --git a/src/third_party/wiredtiger/dist/api_data.py b/src/third_party/wiredtiger/dist/api_data.py index 68a8f4908d8..78dc25850cf 100644 --- a/src/third_party/wiredtiger/dist/api_data.py +++ b/src/third_party/wiredtiger/dist/api_data.py @@ -108,6 +108,18 @@ lsm_config = [ larger than this value. This overrides the \c memory_page_max setting''', min='512K', max='500MB'), + Config('merge_custom', '', r''' + configure the tree to merge into a custom data source''', + type='category', subconfig=[ + Config('prefix', '', r''' + custom data source prefix instead of \c "file"'''), + Config('start_generation', '0', r''' + merge generation at which the custom data source is used + (zero indicates no custom data source)''', + min='0', max='10'), + Config('suffix', '', r''' + custom data source suffix instead of \c ".lsm"'''), + ]), Config('merge_max', '15', r''' the maximum number of chunks to include in a merge operation''', min='2', max='100'), @@ -137,8 +149,10 @@ file_runtime_config = [ Config('commit_timestamp', 'none', r''' verify that timestamps should 'always' or 'never' be used on modifications with this table. Verification is 'none' - if mixed update use is allowed.''', - choices=['always','never','none']), + if mixed update use is allowed. If 'key_consistent' is + set then all updates to a specific key must be the same + with respect to timestamp usage or not.''', + choices=['always','key_consistent', 'never','none']), Config('read_timestamp', 'none', r''' verify that timestamps should 'always' or 'never' be used on reads with this table. Verification is 'none' @@ -442,29 +456,36 @@ connection_runtime_config = [ ]), Config('eviction_checkpoint_target', '5', r''' perform eviction at the beginning of checkpoints to bring the dirty - content in cache to this level, expressed as a percentage of the total - cache size. Ignored if set to zero or \c in_memory is \c true''', - min=0, max=99), + content in cache to this level. It is a percentage of the cache size if + the value is within the range of 0 to 100 or an absolute size when + greater than 100. The value is not allowed to exceed the \c cache_size. + Ignored if set to zero or \c in_memory is \c true''', + min=0, max='10TB'), Config('eviction_dirty_target', '5', r''' perform eviction in worker threads when the cache contains at least - this much dirty content, expressed as a percentage of the total cache - size.''', - min=1, max=99), + this much dirty content. It is a percentage of the cache size if the + value is within the range of 1 to 100 or an absolute size when greater + than 100. The value is not allowed to exceed the \c cache_size.''', + min=1, max='10TB'), Config('eviction_dirty_trigger', '20', r''' trigger application threads to perform eviction when the cache contains - at least this much dirty content, expressed as a percentage of the - total cache size. This setting only alters behavior if it is lower than - eviction_trigger''', - min=1, max=99), + at least this much dirty content. It is a percentage of the cache size + if the value is within the range of 1 to 100 or an absolute size when + greater than 100. The value is not allowed to exceed the \c cache_size. + This setting only alters behavior if it is lower than eviction_trigger + ''', min=1, max='10TB'), Config('eviction_target', '80', r''' perform eviction in worker threads when the cache contains at least - this much content, expressed as a percentage of the total cache size. - Must be less than \c eviction_trigger''', - min=10, max=99), + this much content. It is a percentage of the cache size if the value is + within the range of 10 to 100 or an absolute size when greater than 100. + The value is not allowed to exceed the \c cache_size.''', + min=10, max='10TB'), Config('eviction_trigger', '95', r''' trigger application threads to perform eviction when the cache contains - at least this much content, expressed as a percentage of the - total cache size''', min=10, max=99), + at least this much content. It is a percentage of the cache size if the + value is within the range of 10 to 100 or an absolute size when greater + than 100. The value is not allowed to exceed the \c cache_size.''', + min=10, max='10TB'), Config('file_manager', '', r''' control how file handles are managed''', type='category', subconfig=[ @@ -497,10 +518,25 @@ connection_runtime_config = [ Config('lsm_merge', 'true', r''' merge LSM chunks where possible (deprecated)''', type='boolean', undoc=True), + Config('operation_tracking', '', r''' + enable tracking of performance-critical functions. See + @ref operation_tracking for more information''', + type='category', subconfig=[ + Config('enabled', 'false', r''' + enable operation tracking subsystem''', + type='boolean'), + Config('path', '"."', r''' + the name of a directory into which operation tracking files are + written. The directory must already exist. If the value is not + an absolute path, the path is relative to the database home + (see @ref absolute_path for more information)'''), + ]), Config('shared_cache', '', r''' shared cache configuration options. A database should configure either a cache_size or a shared_cache not both. Enabling a - shared cache uses a session from the configured session_max''', + shared cache uses a session from the configured session_max. A + shared cache can not have absolute values configured for cache + eviction settings''', type='category', subconfig=[ Config('chunk', '10MB', r''' the granularity that a shared cache is redistributed''', @@ -525,7 +561,7 @@ connection_runtime_config = [ Maintain database statistics, which may impact performance. Choosing "all" maintains all statistics regardless of cost, "fast" maintains a subset of statistics that are relatively - inexpensive, "none" turns off all statistics. The "clear" + inexpensive, "none" turns off all statistics. The "clear" configuration resets statistics after they are gathered, where appropriate (for example, a cache size statistic is not cleared, while the count of cursor insert operations will @@ -546,8 +582,7 @@ connection_runtime_config = [ type='list', undoc=True, choices=[ 'checkpoint_slow', 'internal_page_split_race', 'page_split_race']), Config('verbose', '', r''' - enable messages for various events. Only available if WiredTiger - is configured with --enable-verbose. Options are given as a + enable messages for various events. Options are given as a list, such as <code>"verbose=[evictserver,read]"</code>''', type='list', choices=[ 'api', @@ -576,6 +611,7 @@ connection_runtime_config = [ 'salvage', 'shared_cache', 'split', + 'temporary', 'thread_group', 'timestamp', 'transaction', diff --git a/src/third_party/wiredtiger/dist/dist.py b/src/third_party/wiredtiger/dist/dist.py index ec47f509541..e4b76bdbab4 100644 --- a/src/third_party/wiredtiger/dist/dist.py +++ b/src/third_party/wiredtiger/dist/dist.py @@ -22,6 +22,14 @@ def all_c_files(): for line in glob.iglob('../test/*/*.[ci]'): yield line +# all_h_files -- +# Return list of all WiredTiger C include file names. +def all_h_files(): + file_re = re.compile(r'^\w') + for line in glob.iglob('../src/*/*.h'): + yield line + yield "../src/include/wiredtiger.in" + # source_dirs -- # Return a list of the WiredTiger source directory names. def source_dirs(): diff --git a/src/third_party/wiredtiger/dist/filelist b/src/third_party/wiredtiger/dist/filelist index 9755e24f3c7..79590313b89 100644 --- a/src/third_party/wiredtiger/dist/filelist +++ b/src/third_party/wiredtiger/dist/filelist @@ -112,6 +112,7 @@ src/meta/meta_ext.c src/meta/meta_table.c src/meta/meta_track.c src/meta/meta_turtle.c +src/optrack/optrack.c src/os_common/filename.c src/os_common/os_abort.c src/os_common/os_alloc.c diff --git a/src/third_party/wiredtiger/dist/flags.py b/src/third_party/wiredtiger/dist/flags.py index 28c91486e1a..0380cde1781 100644 --- a/src/third_party/wiredtiger/dist/flags.py +++ b/src/third_party/wiredtiger/dist/flags.py @@ -1,226 +1,60 @@ -# Output a C header file using the minimum number of distinct bits to ensure -# flags don't collide. - -import os, re, sys -from dist import compare_srcfile - -flags = { -################################################### -# Internal routine flag declarations -################################################### - 'log_scan' : [ - 'LOGSCAN_FIRST', - 'LOGSCAN_FROM_CKP', - 'LOGSCAN_ONE', - 'LOGSCAN_RECOVER', - ], - 'log_write' : [ - 'LOG_BACKGROUND', - 'LOG_DSYNC', - 'LOG_FLUSH', - 'LOG_FSYNC', - 'LOG_SYNC_ENABLED', - ], - 'page_read' : [ - 'READ_CACHE', - 'READ_IGNORE_CACHE_SIZE', - 'READ_LOOKASIDE', - 'READ_NOTFOUND_OK', - 'READ_NO_EMPTY', - 'READ_NO_GEN', - 'READ_NO_SPLIT', - 'READ_NO_WAIT', - 'READ_PREV', - 'READ_RESTART_OK', - 'READ_SKIP_INTL', - 'READ_TRUNCATE', - 'READ_WONT_NEED', - ], - 'rec_write' : [ - 'REC_CHECKPOINT', - 'REC_EVICT', - 'REC_IN_MEMORY', - 'REC_LOOKASIDE', - 'REC_SCRUB', - 'REC_UPDATE_RESTORE', - 'REC_VISIBILITY_ERR', - 'REC_VISIBLE_ALL', - ], - 'timing_stress_for_test' : [ - 'TIMING_STRESS_CHECKPOINT_SLOW', - 'TIMING_STRESS_INTERNAL_PAGE_SPLIT_RACE', - 'TIMING_STRESS_PAGE_SPLIT_RACE', - ], - 'txn_log_checkpoint' : [ - 'TXN_LOG_CKPT_CLEANUP', - 'TXN_LOG_CKPT_PREPARE', - 'TXN_LOG_CKPT_START', - 'TXN_LOG_CKPT_STOP', - 'TXN_LOG_CKPT_SYNC', - ], - 'txn_update_oldest' : [ - 'TXN_OLDEST_STRICT', - 'TXN_OLDEST_WAIT', - ], - 'verbose' : [ - 'VERB_API', - 'VERB_BLOCK', - 'VERB_CHECKPOINT', - 'VERB_CHECKPOINT_PROGRESS', - 'VERB_COMPACT', - 'VERB_EVICT', - 'VERB_EVICTSERVER', - 'VERB_EVICT_STUCK', - 'VERB_FILEOPS', - 'VERB_HANDLEOPS', - 'VERB_LOG', - 'VERB_LOOKASIDE', - 'VERB_LOOKASIDE_ACTIVITY', - 'VERB_LSM', - 'VERB_LSM_MANAGER', - 'VERB_METADATA', - 'VERB_MUTEX', - 'VERB_OVERFLOW', - 'VERB_READ', - 'VERB_REBALANCE', - 'VERB_RECONCILE', - 'VERB_RECOVERY', - 'VERB_RECOVERY_PROGRESS', - 'VERB_SALVAGE', - 'VERB_SHARED_CACHE', - 'VERB_SPLIT', - 'VERB_THREAD_GROUP', - 'VERB_TIMESTAMP', - 'VERB_TRANSACTION', - 'VERB_VERIFY', - 'VERB_VERSION', - 'VERB_WRITE', - ], - -################################################### -# Structure flag declarations -################################################### - 'conn' : [ - 'CONN_CACHE_POOL', - 'CONN_CKPT_SYNC', - 'CONN_CLOSING', - 'CONN_CLOSING_NO_MORE_OPENS', - 'CONN_EVICTION_NO_LOOKASIDE', - 'CONN_EVICTION_RUN', - 'CONN_IN_MEMORY', - 'CONN_LOOKASIDE_OPEN', - 'CONN_LEAK_MEMORY', - 'CONN_LSM_MERGE', - 'CONN_PANIC', - 'CONN_READONLY', - 'CONN_RECOVERING', - 'CONN_SERVER_ASYNC', - 'CONN_SERVER_CHECKPOINT', - 'CONN_SERVER_LOG', - 'CONN_SERVER_LSM', - 'CONN_SERVER_STATISTICS', - 'CONN_SERVER_SWEEP', - 'CONN_WAS_BACKUP', - ], - 'session' : [ - 'SESSION_CAN_WAIT', - 'SESSION_INTERNAL', - 'SESSION_IGNORE_CACHE_SIZE', - 'SESSION_LOCKED_CHECKPOINT', - 'SESSION_LOCKED_HANDLE_LIST_READ', - 'SESSION_LOCKED_HANDLE_LIST_WRITE', - 'SESSION_LOCKED_METADATA', - 'SESSION_LOCKED_PASS', - 'SESSION_LOCKED_SCHEMA', - 'SESSION_LOCKED_SLOT', - 'SESSION_LOCKED_TABLE_READ', - 'SESSION_LOCKED_TABLE_WRITE', - 'SESSION_LOCKED_TURTLE', - 'SESSION_LOGGING_INMEM', - 'SESSION_LOOKASIDE_CURSOR', - 'SESSION_NO_DATA_HANDLES', - 'SESSION_NO_RECONCILE', - 'SESSION_NO_LOGGING', - 'SESSION_NO_SCHEMA_LOCK', - 'SESSION_QUIET_CORRUPT_FILE', - 'SESSION_READ_WONT_NEED', - 'SESSION_SERVER_ASYNC', - ], - 'stat' : [ - 'STAT_CLEAR', - 'STAT_JSON', - 'STAT_ON_CLOSE', - 'STAT_TYPE_ALL', - 'STAT_TYPE_CACHE_WALK', - 'STAT_TYPE_FAST', - 'STAT_TYPE_SIZE', - 'STAT_TYPE_TREE_WALK', - ], -} - -flag_cnt = {} # Dictionary [flag] : [reference count] -flag_name = {} # Dictionary [flag] : [name ...] -name_mask = {} # Dictionary [name] : [used flag mask] - -# Step through the flags dictionary and build our local dictionaries. -for method in flags.items(): - name_mask[method[0]] = 0x0 - for flag in method[1]: - if flag == '__NONE__': - continue - if flag not in flag_cnt: - flag_cnt[flag] = 1 - flag_name[flag] = [] - else: - flag_cnt[flag] += 1 - flag_name[flag].append(method[0]) - -# Create list of possible bit masks. -bits = [2 ** i for i in range(0, 32)] - -# Walk the list of flags in reverse, sorted-by-reference count order. For -# each flag, find a bit that's not currently in use by any method using the -# flag. -flag_bit = {} # Dictionary [flag] : [bit value] -for f in sorted(flag_cnt.items(), key = lambda k_v : (-k_v[1], k_v[0])): - mask = 0xffffffff - for m in flag_name[f[0]]: - mask &= ~name_mask[m] - if mask == 0: - print >>sys.stderr,\ - "flags.py: ran out of flags at " + m + " method", - sys.exit(1) - for b in bits: - if mask & b: - mask = b - break - flag_bit[f[0]] = mask - for m in flag_name[f[0]]: - name_mask[m] |= mask - -# Print out the flag masks in hex. -# Assumes tab stops set to 8 characters. -flag_info = '' -for f in sorted(flag_cnt.items()): - flag_info += "#define\tWT_%s%s%#010x\n" %\ - (f[0],\ - "\t" * max(1, 6 - int((len('WT_') + len(f[0])) / 8)),\ - flag_bit[f[0]]) - -# Update the wiredtiger.in file with the flags information. -tmp_file = '__tmp' -tfile = open(tmp_file, 'w') -skip = 0 -for line in open('../src/include/flags.h', 'r'): - if skip: - if line.count('flags section: END'): - tfile.write('/*\n' + line) - skip = 0 - else: - tfile.write(line) - if line.count('flags section: BEGIN'): - skip = 1 - tfile.write(' */\n') - tfile.write(flag_info) -tfile.close() -compare_srcfile(tmp_file, '../src/include/flags.h') +#!/usr/bin/env python + +import re, sys +from dist import all_c_files, all_h_files, compare_srcfile + +# Automatically build flags values: read through all of the header files, and +# for each group of flags, sort them and give them a unique value. +# +# To add a new flag declare it at the top of the flags list as: +# #define WT_NEW_FLAG_NAME 0x0u +# +# and it will be automatically alphabetized and assigned the proper value. +def flag_declare(name): + tmp_file = '__tmp' + with open(name, 'r') as f: + tfile = open(tmp_file, 'w') + + lcnt = 0 + parsing = False + for line in f: + lcnt = lcnt + 1 + if line.find('AUTOMATIC FLAG VALUE GENERATION START') != -1: + header = line + defines = [] + parsing = True + elif line.find('AUTOMATIC FLAG VALUE GENERATION STOP') != -1: + # We only support 64 bits. + if len(defines) > 64: + print >>sys.stderr, name + ": line " +\ + str(lcnt) + ": exceeds maximum 64 bit flags" + sys.exit(1) + + # Calculate number of hex bytes, create format string + fmt = "0x%%0%dxu" % ((len(defines) + 3) / 4) + + tfile.write(header) + v = 1 + for d in sorted(defines): + tfile.write(re.sub("0x[01248u]*", fmt % v, d)) + v = v * 2 + tfile.write(line) + + parsing = False + elif parsing and line.find('#define') == -1: + print >>sys.stderr, name + ": line " +\ + str(lcnt) + ": unexpected flag line, no #define" + sys.exit(1) + elif parsing: + defines.append(line) + else: + tfile.write(line) + + tfile.close() + compare_srcfile(tmp_file, name) + +# Update function argument declarations. +for name in all_h_files(): + flag_declare(name) +for name in all_c_files(): + flag_declare(name) diff --git a/src/third_party/wiredtiger/dist/s_c_test_create b/src/third_party/wiredtiger/dist/s_c_test_create index 1d379664e75..66d1e24ec87 100755 --- a/src/third_party/wiredtiger/dist/s_c_test_create +++ b/src/third_party/wiredtiger/dist/s_c_test_create @@ -35,7 +35,7 @@ mkdir $CSUITE_DIRECTORY/$TEST_NAME (cat <<EOF /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/dist/s_clang-scan b/src/third_party/wiredtiger/dist/s_clang-scan new file mode 100644 index 00000000000..33d6f3e7d9e --- /dev/null +++ b/src/third_party/wiredtiger/dist/s_clang-scan @@ -0,0 +1,50 @@ +#! /bin/sh + +t=__wt.$$ +trap 'rm -rf $t /tmp/scan-build-*' 0 1 2 3 13 15 + +# Installation of the clang development package isn't standard, list a +# couple of the places we're using. +export PATH=$PATH:/usr/local/clang50/bin:/usr/local/llvm-devel/bin + +# Remove old reports. +rm -rf /tmp/scan-build-* + +# Find the top-level WiredTiger directory and move to there. +p="$PWD" +while test "$p" != "/" ; do + if test -d "$p/build_posix"; then + break; + fi + p=`dirname $p` +done +test "$p" != "/" || { + echo "$0: cannot find the WiredTiger top-level directory" + exit 1 +} + +cd $p || exit 1 +echo "$0: running scan-build in $p..." + +sh autogen.sh > /dev/null || exit 1 + +args="--use-cc=clang40 -no-failure-reports" +args="$args -disable-checker core.NullDereference" +scan-build $args ./configure 'CFLAGS=-g -Wall' \ + --disable-shared --enable-diagnostic --enable-strict > /dev/null +scan-build $args make -j 8 wt 2>&1 > $t + +make -j 8 distclean > /dev/null + +grep 'No bugs found' $t > /dev/null || { + echo "=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=" + echo 'unexpected scan-build output' + echo "=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=" + sed -e '/^--- /d' \ + -e '/^\/bin\/sh /d' \ + -e '/^depbase=/d' \ + -e '/^libtool: /d' < $t + exit 1 +} + +exit 0 diff --git a/src/third_party/wiredtiger/dist/s_clang-tidy b/src/third_party/wiredtiger/dist/s_clang-tidy new file mode 100644 index 00000000000..89d3fa4112a --- /dev/null +++ b/src/third_party/wiredtiger/dist/s_clang-tidy @@ -0,0 +1,85 @@ +#! /bin/sh + +t=__wt.$$ +trap 'rm -rf $t' 0 1 2 3 13 15 + +# Installation of the clang development package isn't standard, list a +# couple of the places we're using. +export PATH=$PATH:/usr/local/clang50/bin:/usr/local/llvm-devel/bin + +# Remove old reports. +rm -rf /tmp/scan-build-* + +# Find the top-level WiredTiger directory and move to there. +p="$PWD" +while test "$p" != "/" ; do + if test -d "$p/build_posix"; then + break; + fi + p=`dirname $p` +done +test "$p" != "/" || { + echo "$0: cannot find the WiredTiger top-level directory" + exit 1 +} + +cd $p || exit 1 +echo "$0: running clang-tidy in $p..." + +sh autogen.sh > /dev/null || exit 1 + +./configure --enable-diagnostic --enable-strict > /dev/null + +# We need a custom wiredtiger_config.h, clang-tidy doesn't know where to +# find the x86intrin.h include file. +cp wiredtiger_config.h $t +sed '/HAVE_X86INTRIN_H/d' < $t > wiredtiger_config.h + +# We need a custom verify_build.h, clang-tidy doesn't like the magic. +f=src/include/verify_build.h +rm -f $f.save +cp $f $f.save +echo '#define WT_STATIC_ASSERT(a)' > $f + +def="-D_GNU_SOURCE" +inc="-I. -Isrc/include" + +args="-checks=*" +args="$args,-android-cloexec-fopen" +args="$args,-clang-analyzer-core.NullDereference" +args="$args,-clang-analyzer-optin.performance.Padding" +args="$args,-google-readability-braces-around-statements" +args="$args,-hicpp-braces-around-statements" +args="$args,-hicpp-no-assembler" +args="$args,-hicpp-signed-bitwise" +args="$args,-llvm-header-guard" +args="$args,-llvm-include-order" +args="$args,-readability-braces-around-statements" +args="$args,-readability-inconsistent-declaration-parameter-name" +args="$args,-readability-named-parameter" +args="$args,-readability-non-const-parameter" + +# clang-tidy gets unhappy if we toss the whole tree at it, so run +# through a file at a time. +# Only specify -header once. +( +clang-tidy src/btree/bt_compact.c \ + "-header-filter=.*" "$args" -- $def $inc 2>&1 +for i in src/[a-np-z]*/*.c src/os_posix/*.c; do + clang-tidy $i "$args" -- $def $inc 2>&1 +done +) > $t + +echo "=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=" +echo 'clang-tidy output' +echo "=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=" +sed -e '/ warnings generated/d' \ + -e '/Suppressed.*warnings/d' \ + -e '/Use -header-filter/d' < $t + +# Restore verify_build.h. +f=src/include/verify_build.h +rm -f $f +mv $f.save $f + +exit 0 diff --git a/src/third_party/wiredtiger/dist/s_copyright b/src/third_party/wiredtiger/dist/s_copyright index 4a93be73fb6..170dba3efff 100755 --- a/src/third_party/wiredtiger/dist/s_copyright +++ b/src/third_party/wiredtiger/dist/s_copyright @@ -110,7 +110,8 @@ fi -o -name '*.cxx' \ -o -name '*.in' \ -o -name '*.java' \ - -o -name '*.py' | + -o -name '*.py' \ + -o -name '*.swig' | sed -e '/Makefile.in/d' \ -e '/^build_posix\//d' \ -e '/api\/leveldb\/basho\//d' \ diff --git a/src/third_party/wiredtiger/dist/s_define.list b/src/third_party/wiredtiger/dist/s_define.list index 4be8ceee0e3..cfae3106fcf 100644 --- a/src/third_party/wiredtiger/dist/s_define.list +++ b/src/third_party/wiredtiger/dist/s_define.list @@ -13,17 +13,17 @@ TXN_API_END WIN32_LEAN_AND_MEAN WT_ALIGN_CHECK WT_ATOMIC_CAS +WT_ATOMIC_CAS_FUNC WT_ATOMIC_FUNC WT_BLOCK_DESC_SIZE WT_BLOCK_HEADER_SIZE WT_CACHE_LINE_ALIGNMENT WT_CACHE_LINE_PAD_BEGIN WT_CACHE_LINE_PAD_END +WT_CLOCKDIFF_NS WT_CONN_CHECK_PANIC WT_DEADLOCK WT_DEBUG_BYTE -WT_SINGLE_THREAD_CHECK_START -WT_SINGLE_THREAD_CHECK_STOP WT_ERR_ERROR_OK WT_EXT_FOREACH_OFF WT_HANDLE_CLOSED @@ -36,6 +36,8 @@ WT_LOG_SLOT_MASK_ON WT_LOG_SLOT_MAXBITS WT_LOG_SLOT_UNBUFFERED_ISSET WT_LOOKASIDE_COMPRESSOR +WT_OPTRACK_BUFSIZE +WT_OPTRACK_MAXRECS WT_PACKED_STRUCT_BEGIN WT_PACKED_STRUCT_END WT_PADDING_CHECK @@ -45,6 +47,8 @@ WT_SESSION_LOCKED_CHECKPOINT WT_SESSION_LOCKED_TABLE_READ WT_SESSION_LOCKED_TABLE_WRITE WT_SESSION_LOCKED_TURTLE +WT_SINGLE_THREAD_CHECK_START +WT_SINGLE_THREAD_CHECK_STOP WT_SIZE_CHECK WT_STATS_FIELD_TO_OFFSET WT_STATS_SLOT_ID @@ -60,6 +64,9 @@ WT_STAT_INCRV_ATOMIC_BASE WT_STAT_INCRV_BASE WT_STAT_WRITE WT_TIMEDIFF_US +WT_TRACK_OP +WT_TRACK_OP_END +WT_TRACK_OP_INIT WT_TRET_ERROR_OK WT_UPDATE_SIZE WT_WITH_LOCK_NOWAIT diff --git a/src/third_party/wiredtiger/dist/s_funcs.list b/src/third_party/wiredtiger/dist/s_funcs.list index 42eacf9c068..a7653e5b497 100644 --- a/src/third_party/wiredtiger/dist/s_funcs.list +++ b/src/third_party/wiredtiger/dist/s_funcs.list @@ -27,6 +27,7 @@ __wt_log_scan __wt_nlpo2 __wt_nlpo2_round __wt_print_huffman_code +__wt_session_breakpoint __wt_stat_join_aggregate __wt_stat_join_clear_all __wt_stream_set_no_buffer diff --git a/src/third_party/wiredtiger/dist/s_stat b/src/third_party/wiredtiger/dist/s_stat index cf9303e5f95..32cf16fd3d4 100755 --- a/src/third_party/wiredtiger/dist/s_stat +++ b/src/third_party/wiredtiger/dist/s_stat @@ -25,6 +25,10 @@ cat << UNUSED_STAT_FIELDS lock_checkpoint_count lock_checkpoint_wait_application lock_checkpoint_wait_internal +lock_commit_timestamp_read_count +lock_commit_timestamp_wait_application +lock_commit_timestamp_wait_internal +lock_commit_timestamp_write_count lock_dhandle_read_count lock_dhandle_wait_application lock_dhandle_wait_internal @@ -32,6 +36,10 @@ lock_dhandle_write_count lock_metadata_count lock_metadata_wait_application lock_metadata_wait_internal +lock_read_timestamp_read_count +lock_read_timestamp_wait_application +lock_read_timestamp_wait_internal +lock_read_timestamp_write_count lock_schema_count lock_schema_wait_application lock_schema_wait_internal @@ -39,6 +47,10 @@ lock_table_read_count lock_table_wait_application lock_table_wait_internal lock_table_write_count +lock_txn_global_read_count +lock_txn_global_wait_application +lock_txn_global_wait_internal +lock_txn_global_write_count UNUSED_STAT_FIELDS echo "$search" diff --git a/src/third_party/wiredtiger/dist/s_string.ok b/src/third_party/wiredtiger/dist/s_string.ok index 1a252fdd615..5d2eb7427b6 100644 --- a/src/third_party/wiredtiger/dist/s_string.ok +++ b/src/third_party/wiredtiger/dist/s_string.ok @@ -162,6 +162,7 @@ HFS HHHH HHHHLL HHHLL +HILQr Hendrik HyperLevelDB ID's @@ -268,6 +269,7 @@ Noll Nul OOB OPLOG +OPTRACK OPTYPE OUTBUFF OVFL @@ -746,6 +748,7 @@ fsyncLock fsyncs ftruncate func +funcid fvisibility gcc gdb @@ -775,6 +778,7 @@ hashval havesize hdr highjack +hilq hotbackup hselasky html @@ -986,11 +990,13 @@ notsup notused novalue nowait +nsec nset nsnap nul nullptr nuls +num numSymbols numbare offpage @@ -1006,6 +1012,7 @@ openfile oplist oplog optimizations +optrack optype ori os @@ -1078,6 +1085,7 @@ rbrace rbracket rcursor rdonly +rdtsc rduppo readlock readonly @@ -1193,10 +1201,12 @@ tV tablename tcbench td +teardown tempdir testutil th tid +timedwait timestamp timestamps tmp @@ -1216,6 +1226,7 @@ trk trk's trunc trylock +tsc tupdate tvalue txn diff --git a/src/third_party/wiredtiger/dist/s_style b/src/third_party/wiredtiger/dist/s_style index 3ae14949fbf..3f4346173e6 100755 --- a/src/third_party/wiredtiger/dist/s_style +++ b/src/third_party/wiredtiger/dist/s_style @@ -61,6 +61,15 @@ else cat $t fi + if ! expr "$f" : 'examples/c/.*' > /dev/null && + ! expr "$f" : 'ext/.*' > /dev/null && + ! expr "$f" : 'src/include/wiredtiger_ext\.h' > /dev/null && + ! expr "$f" : 'src/txn/txn_ext\.c' > /dev/null && + grep WT_TXN_ISO_ $f; then + echo "$f: WT_TXN_ISO_XXX constants only for the extension API" + cat $t + fi + if ! expr "$f" : 'src/include/queue\.h' > /dev/null && egrep 'STAILQ_|SLIST_|\bLIST_' $f ; then echo "$f: use TAILQ for all lists" diff --git a/src/third_party/wiredtiger/dist/s_void b/src/third_party/wiredtiger/dist/s_void index 78a3340e3bf..9c5f6711da0 100755 --- a/src/third_party/wiredtiger/dist/s_void +++ b/src/third_party/wiredtiger/dist/s_void @@ -64,11 +64,13 @@ func_ok() -e '/int __wt_lsm_manager_pop_entry$/d' \ -e '/int __wt_once$/d' \ -e '/int __wt_posix_directory_list_free$/d' \ + -e '/int __wt_session_breakpoint$/d' \ -e '/int __wt_spin_init$/d' \ -e '/int __wt_spin_trylock$/d' \ -e '/int __wt_stat_connection_desc$/d' \ -e '/int __wt_stat_dsrc_desc$/d' \ -e '/int __wt_stat_join_desc$/d' \ + -e '/int __wt_txn_rollback_required$/d' \ -e '/int __wt_win_directory_list_free$/d' \ -e '/int bdb_compare_reverse$/d' \ -e '/int copyout_val$/d' \ diff --git a/src/third_party/wiredtiger/dist/stat_data.py b/src/third_party/wiredtiger/dist/stat_data.py index bbb581e904e..6cd3f219b4a 100644 --- a/src/third_party/wiredtiger/dist/stat_data.py +++ b/src/third_party/wiredtiger/dist/stat_data.py @@ -84,6 +84,10 @@ class LSMStat(Stat): prefix = 'LSM' def __init__(self, name, desc, flags=''): Stat.__init__(self, name, LSMStat.prefix, desc, flags) +class PerfHistStat(Stat): + prefix = 'perf' + def __init__(self, name, desc, flags=''): + Stat.__init__(self, name, PerfHistStat.prefix, desc, flags) class RecStat(Stat): prefix = 'reconciliation' def __init__(self, name, desc, flags=''): @@ -128,6 +132,7 @@ groups['memory'] = [ groups['system'] = [ ConnStat.prefix, DhandleStat.prefix, + PerfHistStat.prefix, SessionStat.prefix, ThreadStat.prefix ] @@ -267,6 +272,8 @@ connection_stats = [ CacheStat('cache_read_app_count', 'application threads page read from disk to cache count'), CacheStat('cache_read_app_time', 'application threads page read from disk to cache time (usecs)'), CacheStat('cache_read_lookaside', 'pages read into cache requiring lookaside entries'), + CacheStat('cache_read_lookaside_delay', 'pages read into cache with skipped lookaside entries needed later'), + CacheStat('cache_read_lookaside_skipped', 'pages read into cache skipping older lookaside entries'), CacheStat('cache_read_overflow', 'overflow pages read into cache'), CacheStat('cache_write', 'pages written from cache'), CacheStat('cache_write_app_count', 'application threads page write from cache to disk count'), @@ -309,6 +316,10 @@ connection_stats = [ LockStat('lock_checkpoint_count', 'checkpoint lock acquisitions'), LockStat('lock_checkpoint_wait_application', 'checkpoint lock application thread wait time (usecs)'), LockStat('lock_checkpoint_wait_internal', 'checkpoint lock internal thread wait time (usecs)'), + LockStat('lock_commit_timestamp_read_count', 'commit timestamp queue read lock acquisitions'), + LockStat('lock_commit_timestamp_wait_application', 'commit timestamp queue lock application thread time waiting for the dhandle lock (usecs)'), + LockStat('lock_commit_timestamp_wait_internal', 'commit timestamp queue lock internal thread time waiting for the dhandle lock (usecs)'), + LockStat('lock_commit_timestamp_write_count', 'commit timestamp queue write lock acquisitions'), LockStat('lock_dhandle_read_count', 'dhandle read lock acquisitions'), LockStat('lock_dhandle_wait_application', 'dhandle lock application thread time waiting for the dhandle lock (usecs)'), LockStat('lock_dhandle_wait_internal', 'dhandle lock internal thread time waiting for the dhandle lock (usecs)'), @@ -316,6 +327,10 @@ connection_stats = [ LockStat('lock_metadata_count', 'metadata lock acquisitions'), LockStat('lock_metadata_wait_application', 'metadata lock application thread wait time (usecs)'), LockStat('lock_metadata_wait_internal', 'metadata lock internal thread wait time (usecs)'), + LockStat('lock_read_timestamp_read_count', 'read timestamp queue read lock acquisitions'), + LockStat('lock_read_timestamp_wait_application', 'read timestamp queue lock application thread time waiting for the dhandle lock (usecs)'), + LockStat('lock_read_timestamp_wait_internal', 'read timestamp queue lock internal thread time waiting for the dhandle lock (usecs)'), + LockStat('lock_read_timestamp_write_count', 'read timestamp queue write lock acquisitions'), LockStat('lock_schema_count', 'schema lock acquisitions'), LockStat('lock_schema_wait_application', 'schema lock application thread wait time (usecs)'), LockStat('lock_schema_wait_internal', 'schema lock internal thread wait time (usecs)'), @@ -323,6 +338,10 @@ connection_stats = [ LockStat('lock_table_wait_application', 'table lock application thread time waiting for the table lock (usecs)'), LockStat('lock_table_wait_internal', 'table lock internal thread time waiting for the table lock (usecs)'), LockStat('lock_table_write_count', 'table write lock acquisitions'), + LockStat('lock_txn_global_read_count', 'txn global read lock acquisitions'), + LockStat('lock_txn_global_wait_application', 'txn global lock application thread time waiting for the dhandle lock (usecs)'), + LockStat('lock_txn_global_wait_internal', 'txn global lock internal thread time waiting for the dhandle lock (usecs)'), + LockStat('lock_txn_global_write_count', 'txn global write lock acquisitions'), ########################################## # Logging statistics @@ -389,6 +408,32 @@ connection_stats = [ LSMStat('lsm_work_units_done', 'tree maintenance operations executed'), ########################################## + # Performance Histogram Stats + ########################################## + PerfHistStat('perf_hist_fsread_latency_gt1000', 'file system read latency histogram (bucket 6) - 1000ms+'), + PerfHistStat('perf_hist_fsread_latency_lt50', 'file system read latency histogram (bucket 1) - 10-49ms'), + PerfHistStat('perf_hist_fsread_latency_lt100', 'file system read latency histogram (bucket 2) - 50-99ms'), + PerfHistStat('perf_hist_fsread_latency_lt250', 'file system read latency histogram (bucket 3) - 100-249ms'), + PerfHistStat('perf_hist_fsread_latency_lt500', 'file system read latency histogram (bucket 4) - 250-499ms'), + PerfHistStat('perf_hist_fsread_latency_lt1000', 'file system read latency histogram (bucket 5) - 500-999ms'), + PerfHistStat('perf_hist_fswrite_latency_gt1000', 'file system write latency histogram (bucket 6) - 1000ms+'), + PerfHistStat('perf_hist_fswrite_latency_lt50', 'file system write latency histogram (bucket 1) - 10-49ms'), + PerfHistStat('perf_hist_fswrite_latency_lt100', 'file system write latency histogram (bucket 2) - 50-99ms'), + PerfHistStat('perf_hist_fswrite_latency_lt250', 'file system write latency histogram (bucket 3) - 100-249ms'), + PerfHistStat('perf_hist_fswrite_latency_lt500', 'file system write latency histogram (bucket 4) - 250-499ms'), + PerfHistStat('perf_hist_fswrite_latency_lt1000', 'file system write latency histogram (bucket 5) - 500-999ms'), + PerfHistStat('perf_hist_opread_latency_gt10000', 'operation read latency histogram (bucket 5) - 10000us+'), + PerfHistStat('perf_hist_opread_latency_lt250', 'operation read latency histogram (bucket 1) - 100-249us'), + PerfHistStat('perf_hist_opread_latency_lt500', 'operation read latency histogram (bucket 2) - 250-499us'), + PerfHistStat('perf_hist_opread_latency_lt1000', 'operation read latency histogram (bucket 3) - 500-999us'), + PerfHistStat('perf_hist_opread_latency_lt10000', 'operation read latency histogram (bucket 4) - 1000-9999us'), + PerfHistStat('perf_hist_opwrite_latency_gt10000', 'operation write latency histogram (bucket 5) - 10000us+'), + PerfHistStat('perf_hist_opwrite_latency_lt250', 'operation write latency histogram (bucket 1) - 100-249us'), + PerfHistStat('perf_hist_opwrite_latency_lt500', 'operation write latency histogram (bucket 2) - 250-499us'), + PerfHistStat('perf_hist_opwrite_latency_lt1000', 'operation write latency histogram (bucket 3) - 500-999us'), + PerfHistStat('perf_hist_opwrite_latency_lt10000', 'operation write latency histogram (bucket 4) - 1000-9999us'), + +########################################## # Reconciliation statistics ########################################## RecStat('rec_page_delete', 'pages deleted'), @@ -447,19 +492,29 @@ connection_stats = [ TxnStat('txn_checkpoint_time_recent', 'transaction checkpoint most recent time (msecs)', 'no_clear,no_scale'), TxnStat('txn_checkpoint_time_total', 'transaction checkpoint total time (msecs)', 'no_clear,no_scale'), TxnStat('txn_commit', 'transactions committed'), - TxnStat('txn_commit_queue_head', 'transactions commit timestamp queue inserts to head'), - TxnStat('txn_commit_queue_inserts', 'transactions commit timestamp queue inserts total'), - TxnStat('txn_commit_queue_len', 'transactions commit timestamp queue length'), + TxnStat('txn_commit_queue_empty', 'commit timestamp queue insert to empty'), + TxnStat('txn_commit_queue_head', 'commit timestamp queue inserts to head'), + TxnStat('txn_commit_queue_inserts', 'commit timestamp queue inserts total'), + TxnStat('txn_commit_queue_len', 'commit timestamp queue length'), TxnStat('txn_fail_cache', 'transaction failures due to cache overflow'), TxnStat('txn_pinned_checkpoint_range', 'transaction range of IDs currently pinned by a checkpoint', 'no_clear,no_scale'), TxnStat('txn_pinned_range', 'transaction range of IDs currently pinned', 'no_clear,no_scale'), TxnStat('txn_pinned_snapshot_range', 'transaction range of IDs currently pinned by named snapshots', 'no_clear,no_scale'), TxnStat('txn_pinned_timestamp', 'transaction range of timestamps currently pinned', 'no_clear,no_scale'), TxnStat('txn_pinned_timestamp_oldest', 'transaction range of timestamps pinned by the oldest timestamp', 'no_clear,no_scale'), - TxnStat('txn_read_queue_head', 'transactions read timestamp queue inserts to head'), - TxnStat('txn_read_queue_inserts', 'transactions read timestamp queue inserts total'), - TxnStat('txn_read_queue_len', 'transactions read timestamp queue length'), + TxnStat('txn_query_ts', 'query timestamp calls'), + TxnStat('txn_read_queue_empty', 'read timestamp queue insert to empty'), + TxnStat('txn_read_queue_head', 'read timestamp queue inserts to head'), + TxnStat('txn_read_queue_inserts', 'read timestamp queue inserts total'), + TxnStat('txn_read_queue_len', 'read timestamp queue length'), TxnStat('txn_rollback', 'transactions rolled back'), + TxnStat('txn_set_ts', 'set timestamp calls'), + TxnStat('txn_set_ts_commit', 'set timestamp commit calls'), + TxnStat('txn_set_ts_commit_upd', 'set timestamp commit updates'), + TxnStat('txn_set_ts_oldest', 'set timestamp oldest calls'), + TxnStat('txn_set_ts_oldest_upd', 'set timestamp oldest updates'), + TxnStat('txn_set_ts_stable', 'set timestamp stable calls'), + TxnStat('txn_set_ts_stable_upd', 'set timestamp stable updates'), TxnStat('txn_snapshots_created', 'number of named snapshots created'), TxnStat('txn_snapshots_dropped', 'number of named snapshots dropped'), TxnStat('txn_sync', 'transaction sync calls'), diff --git a/src/third_party/wiredtiger/examples/c/ex_access.c b/src/third_party/wiredtiger/examples/c/ex_access.c index df8b0b499df..4ea06e1fad8 100644 --- a/src/third_party/wiredtiger/examples/c/ex_access.c +++ b/src/third_party/wiredtiger/examples/c/ex_access.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/examples/c/ex_all.c b/src/third_party/wiredtiger/examples/c/ex_all.c index dcd9dd406df..7091a0a76d6 100644 --- a/src/third_party/wiredtiger/examples/c/ex_all.c +++ b/src/third_party/wiredtiger/examples/c/ex_all.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/examples/c/ex_async.c b/src/third_party/wiredtiger/examples/c/ex_async.c index 3c7f5f55cd8..4367a757d25 100644 --- a/src/third_party/wiredtiger/examples/c/ex_async.c +++ b/src/third_party/wiredtiger/examples/c/ex_async.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/examples/c/ex_backup.c b/src/third_party/wiredtiger/examples/c/ex_backup.c index bf40d7573d8..18c34dbe4cc 100644 --- a/src/third_party/wiredtiger/examples/c/ex_backup.c +++ b/src/third_party/wiredtiger/examples/c/ex_backup.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/examples/c/ex_call_center.c b/src/third_party/wiredtiger/examples/c/ex_call_center.c index e4e4ab0b9cf..79a0239850f 100644 --- a/src/third_party/wiredtiger/examples/c/ex_call_center.c +++ b/src/third_party/wiredtiger/examples/c/ex_call_center.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/examples/c/ex_config_parse.c b/src/third_party/wiredtiger/examples/c/ex_config_parse.c index 8d418ffa186..a46740fd170 100644 --- a/src/third_party/wiredtiger/examples/c/ex_config_parse.c +++ b/src/third_party/wiredtiger/examples/c/ex_config_parse.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/examples/c/ex_cursor.c b/src/third_party/wiredtiger/examples/c/ex_cursor.c index 3a8700fb83c..54cee6be750 100644 --- a/src/third_party/wiredtiger/examples/c/ex_cursor.c +++ b/src/third_party/wiredtiger/examples/c/ex_cursor.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/examples/c/ex_data_source.c b/src/third_party/wiredtiger/examples/c/ex_data_source.c index 59cfde7313a..1a0b9f7f39a 100644 --- a/src/third_party/wiredtiger/examples/c/ex_data_source.c +++ b/src/third_party/wiredtiger/examples/c/ex_data_source.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. @@ -489,6 +489,21 @@ my_salvage(WT_DATA_SOURCE *dsrc, WT_SESSION *session, return (0); } +/*! [WT_DATA_SOURCE size] */ +static int +my_size(WT_DATA_SOURCE *dsrc, WT_SESSION *session, + const char *uri, wt_off_t *size) +/*! [WT_DATA_SOURCE size] */ +{ + /* Unused parameters */ + (void)dsrc; + (void)session; + (void)uri; + (void)size; + + return (0); +} + /*! [WT_DATA_SOURCE truncate] */ static int my_truncate(WT_DATA_SOURCE *dsrc, WT_SESSION *session, @@ -559,6 +574,19 @@ my_terminate(WT_DATA_SOURCE *dsrc, WT_SESSION *session) return (0); } +/*! [WT_DATA_SOURCE lsm_pre_merge] */ +static int +my_lsm_pre_merge(WT_DATA_SOURCE *dsrc, WT_CURSOR *source, WT_CURSOR *dest) +/*! [WT_DATA_SOURCE lsm_pre_merge] */ +{ + /* Unused parameters */ + (void)dsrc; + (void)source; + (void)dest; + + return (0); +} + static const char *home; int @@ -581,11 +609,13 @@ main(int argc, char *argv[]) my_open_cursor, my_rename, my_salvage, + my_size, my_truncate, my_range_truncate, my_verify, my_checkpoint, - my_terminate + my_terminate, + my_lsm_pre_merge }; error_check(conn->add_data_source(conn, "dsrc:", &my_dsrc, NULL)); /*! [WT_DATA_SOURCE register] */ diff --git a/src/third_party/wiredtiger/examples/c/ex_encrypt.c b/src/third_party/wiredtiger/examples/c/ex_encrypt.c index 3fdfa9a6d33..ed5112409f4 100644 --- a/src/third_party/wiredtiger/examples/c/ex_encrypt.c +++ b/src/third_party/wiredtiger/examples/c/ex_encrypt.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/examples/c/ex_event_handler.c b/src/third_party/wiredtiger/examples/c/ex_event_handler.c index d9ac4851bf7..1354a764927 100644 --- a/src/third_party/wiredtiger/examples/c/ex_event_handler.c +++ b/src/third_party/wiredtiger/examples/c/ex_event_handler.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/examples/c/ex_extending.c b/src/third_party/wiredtiger/examples/c/ex_extending.c index 7a845c1f6ad..9271f49a469 100644 --- a/src/third_party/wiredtiger/examples/c/ex_extending.c +++ b/src/third_party/wiredtiger/examples/c/ex_extending.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/examples/c/ex_extractor.c b/src/third_party/wiredtiger/examples/c/ex_extractor.c index 245d279ca4b..66e6066ad68 100644 --- a/src/third_party/wiredtiger/examples/c/ex_extractor.c +++ b/src/third_party/wiredtiger/examples/c/ex_extractor.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/examples/c/ex_file_system.c b/src/third_party/wiredtiger/examples/c/ex_file_system.c index 27807f452e6..f8c4e29007c 100644 --- a/src/third_party/wiredtiger/examples/c/ex_file_system.c +++ b/src/third_party/wiredtiger/examples/c/ex_file_system.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/examples/c/ex_hello.c b/src/third_party/wiredtiger/examples/c/ex_hello.c index fe21ff8057f..b5a88648b03 100644 --- a/src/third_party/wiredtiger/examples/c/ex_hello.c +++ b/src/third_party/wiredtiger/examples/c/ex_hello.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/examples/c/ex_log.c b/src/third_party/wiredtiger/examples/c/ex_log.c index 1912ffca8a6..77d6709f0db 100644 --- a/src/third_party/wiredtiger/examples/c/ex_log.c +++ b/src/third_party/wiredtiger/examples/c/ex_log.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/examples/c/ex_pack.c b/src/third_party/wiredtiger/examples/c/ex_pack.c index 98cd1510b92..c2bf4bc9a8b 100644 --- a/src/third_party/wiredtiger/examples/c/ex_pack.c +++ b/src/third_party/wiredtiger/examples/c/ex_pack.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/examples/c/ex_process.c b/src/third_party/wiredtiger/examples/c/ex_process.c index 06a62d10d39..70c5dd2713a 100644 --- a/src/third_party/wiredtiger/examples/c/ex_process.c +++ b/src/third_party/wiredtiger/examples/c/ex_process.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/examples/c/ex_schema.c b/src/third_party/wiredtiger/examples/c/ex_schema.c index f2ebd118104..28776aca668 100644 --- a/src/third_party/wiredtiger/examples/c/ex_schema.c +++ b/src/third_party/wiredtiger/examples/c/ex_schema.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/examples/c/ex_smoke.c b/src/third_party/wiredtiger/examples/c/ex_smoke.c index ab4e085d2f0..2647a706a54 100644 --- a/src/third_party/wiredtiger/examples/c/ex_smoke.c +++ b/src/third_party/wiredtiger/examples/c/ex_smoke.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. @@ -45,7 +45,11 @@ main(int argc, char *argv[]) * This code deliberately doesn't use the standard test_util macros, * we don't want to link against that code to smoke-test a build. */ - (void)system("rm -rf WT_HOME && mkdir WT_HOME"); + if ((ret = system("rm -rf WT_HOME && mkdir WT_HOME")) != 0) { + fprintf(stderr, + "Failed to clean up prior to running example.\n"); + return (EXIT_FAILURE); + } /* Open a connection to the database, creating it if necessary. */ if ((ret = wiredtiger_open("WT_HOME", NULL, "create", &conn)) != 0) { diff --git a/src/third_party/wiredtiger/examples/c/ex_stat.c b/src/third_party/wiredtiger/examples/c/ex_stat.c index 992f335c25e..bc13197038a 100644 --- a/src/third_party/wiredtiger/examples/c/ex_stat.c +++ b/src/third_party/wiredtiger/examples/c/ex_stat.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/examples/c/ex_sync.c b/src/third_party/wiredtiger/examples/c/ex_sync.c index 975ba3b928c..bc897126852 100644 --- a/src/third_party/wiredtiger/examples/c/ex_sync.c +++ b/src/third_party/wiredtiger/examples/c/ex_sync.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/examples/c/ex_thread.c b/src/third_party/wiredtiger/examples/c/ex_thread.c index fe14d67e44b..f709707bffc 100644 --- a/src/third_party/wiredtiger/examples/c/ex_thread.c +++ b/src/third_party/wiredtiger/examples/c/ex_thread.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/examples/java/com/wiredtiger/examples/ex_access.java b/src/third_party/wiredtiger/examples/java/com/wiredtiger/examples/ex_access.java index ed96ebce7d7..b9de70705da 100644 --- a/src/third_party/wiredtiger/examples/java/com/wiredtiger/examples/ex_access.java +++ b/src/third_party/wiredtiger/examples/java/com/wiredtiger/examples/ex_access.java @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/examples/java/com/wiredtiger/examples/ex_all.java b/src/third_party/wiredtiger/examples/java/com/wiredtiger/examples/ex_all.java index ff7d371fabd..6649e14bd05 100644 --- a/src/third_party/wiredtiger/examples/java/com/wiredtiger/examples/ex_all.java +++ b/src/third_party/wiredtiger/examples/java/com/wiredtiger/examples/ex_all.java @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/examples/java/com/wiredtiger/examples/ex_async.java b/src/third_party/wiredtiger/examples/java/com/wiredtiger/examples/ex_async.java index 92054464747..e9c1fb25271 100644 --- a/src/third_party/wiredtiger/examples/java/com/wiredtiger/examples/ex_async.java +++ b/src/third_party/wiredtiger/examples/java/com/wiredtiger/examples/ex_async.java @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/examples/java/com/wiredtiger/examples/ex_call_center.java b/src/third_party/wiredtiger/examples/java/com/wiredtiger/examples/ex_call_center.java index 921c7f9f57c..25efce8a319 100644 --- a/src/third_party/wiredtiger/examples/java/com/wiredtiger/examples/ex_call_center.java +++ b/src/third_party/wiredtiger/examples/java/com/wiredtiger/examples/ex_call_center.java @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/examples/java/com/wiredtiger/examples/ex_cursor.java b/src/third_party/wiredtiger/examples/java/com/wiredtiger/examples/ex_cursor.java index 4a57f3c35da..04e15b76f3e 100644 --- a/src/third_party/wiredtiger/examples/java/com/wiredtiger/examples/ex_cursor.java +++ b/src/third_party/wiredtiger/examples/java/com/wiredtiger/examples/ex_cursor.java @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/examples/java/com/wiredtiger/examples/ex_log.java b/src/third_party/wiredtiger/examples/java/com/wiredtiger/examples/ex_log.java index 5a76c43b13c..2d414c56776 100644 --- a/src/third_party/wiredtiger/examples/java/com/wiredtiger/examples/ex_log.java +++ b/src/third_party/wiredtiger/examples/java/com/wiredtiger/examples/ex_log.java @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/examples/java/com/wiredtiger/examples/ex_schema.java b/src/third_party/wiredtiger/examples/java/com/wiredtiger/examples/ex_schema.java index b7aa64f0c68..9f6ec90ec7a 100644 --- a/src/third_party/wiredtiger/examples/java/com/wiredtiger/examples/ex_schema.java +++ b/src/third_party/wiredtiger/examples/java/com/wiredtiger/examples/ex_schema.java @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/examples/java/com/wiredtiger/examples/ex_stat.java b/src/third_party/wiredtiger/examples/java/com/wiredtiger/examples/ex_stat.java index 799f0396756..9e7cc723bab 100644 --- a/src/third_party/wiredtiger/examples/java/com/wiredtiger/examples/ex_stat.java +++ b/src/third_party/wiredtiger/examples/java/com/wiredtiger/examples/ex_stat.java @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/examples/java/com/wiredtiger/examples/ex_thread.java b/src/third_party/wiredtiger/examples/java/com/wiredtiger/examples/ex_thread.java index 2476b3a4d41..49de7193f84 100644 --- a/src/third_party/wiredtiger/examples/java/com/wiredtiger/examples/ex_thread.java +++ b/src/third_party/wiredtiger/examples/java/com/wiredtiger/examples/ex_thread.java @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/examples/python/ex_access.py b/src/third_party/wiredtiger/examples/python/ex_access.py index 58ba64607e2..06140b9e0a5 100755 --- a/src/third_party/wiredtiger/examples/python/ex_access.py +++ b/src/third_party/wiredtiger/examples/python/ex_access.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/examples/python/ex_stat.py b/src/third_party/wiredtiger/examples/python/ex_stat.py index cd99c4f388b..6b009a727ca 100755 --- a/src/third_party/wiredtiger/examples/python/ex_stat.py +++ b/src/third_party/wiredtiger/examples/python/ex_stat.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/ext/collators/reverse/reverse_collator.c b/src/third_party/wiredtiger/ext/collators/reverse/reverse_collator.c index 3a589613427..2ad4e0fd27f 100644 --- a/src/third_party/wiredtiger/ext/collators/reverse/reverse_collator.c +++ b/src/third_party/wiredtiger/ext/collators/reverse/reverse_collator.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/ext/collators/revint/revint_collator.c b/src/third_party/wiredtiger/ext/collators/revint/revint_collator.c index 9952e922077..00c8bf93acb 100644 --- a/src/third_party/wiredtiger/ext/collators/revint/revint_collator.c +++ b/src/third_party/wiredtiger/ext/collators/revint/revint_collator.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/ext/compressors/lz4/lz4_compress.c b/src/third_party/wiredtiger/ext/compressors/lz4/lz4_compress.c index 279f0be6c36..1f32ff910d6 100644 --- a/src/third_party/wiredtiger/ext/compressors/lz4/lz4_compress.c +++ b/src/third_party/wiredtiger/ext/compressors/lz4/lz4_compress.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/ext/compressors/nop/nop_compress.c b/src/third_party/wiredtiger/ext/compressors/nop/nop_compress.c index 7cdb67c6bf2..f739ffa4777 100644 --- a/src/third_party/wiredtiger/ext/compressors/nop/nop_compress.c +++ b/src/third_party/wiredtiger/ext/compressors/nop/nop_compress.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/ext/compressors/snappy/snappy_compress.c b/src/third_party/wiredtiger/ext/compressors/snappy/snappy_compress.c index a86de5c3803..26aa3082bc0 100644 --- a/src/third_party/wiredtiger/ext/compressors/snappy/snappy_compress.c +++ b/src/third_party/wiredtiger/ext/compressors/snappy/snappy_compress.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/ext/compressors/zlib/zlib_compress.c b/src/third_party/wiredtiger/ext/compressors/zlib/zlib_compress.c index 3263b84bfaa..5ae54a25163 100644 --- a/src/third_party/wiredtiger/ext/compressors/zlib/zlib_compress.c +++ b/src/third_party/wiredtiger/ext/compressors/zlib/zlib_compress.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/ext/compressors/zstd/zstd_compress.c b/src/third_party/wiredtiger/ext/compressors/zstd/zstd_compress.c index d2ebaf20c4e..4f80694b0d1 100644 --- a/src/third_party/wiredtiger/ext/compressors/zstd/zstd_compress.c +++ b/src/third_party/wiredtiger/ext/compressors/zstd/zstd_compress.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/ext/datasources/helium/helium.c b/src/third_party/wiredtiger/ext/datasources/helium/helium.c index 4197c85bdd9..0cf9e979e9b 100644 --- a/src/third_party/wiredtiger/ext/datasources/helium/helium.c +++ b/src/third_party/wiredtiger/ext/datasources/helium/helium.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/ext/encryptors/nop/nop_encrypt.c b/src/third_party/wiredtiger/ext/encryptors/nop/nop_encrypt.c index 3bc0f0f1c71..0d04e51e395 100644 --- a/src/third_party/wiredtiger/ext/encryptors/nop/nop_encrypt.c +++ b/src/third_party/wiredtiger/ext/encryptors/nop/nop_encrypt.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/ext/encryptors/rotn/rotn_encrypt.c b/src/third_party/wiredtiger/ext/encryptors/rotn/rotn_encrypt.c index 5ffc8fcc1a3..f1e26eeaa23 100644 --- a/src/third_party/wiredtiger/ext/encryptors/rotn/rotn_encrypt.c +++ b/src/third_party/wiredtiger/ext/encryptors/rotn/rotn_encrypt.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/ext/extractors/csv/csv_extractor.c b/src/third_party/wiredtiger/ext/extractors/csv/csv_extractor.c index 9866e1d5b34..6ce64e240fe 100644 --- a/src/third_party/wiredtiger/ext/extractors/csv/csv_extractor.c +++ b/src/third_party/wiredtiger/ext/extractors/csv/csv_extractor.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/ext/test/fail_fs/fail_fs.c b/src/third_party/wiredtiger/ext/test/fail_fs/fail_fs.c index fd01ec66c68..bde1bfc48bf 100644 --- a/src/third_party/wiredtiger/ext/test/fail_fs/fail_fs.c +++ b/src/third_party/wiredtiger/ext/test/fail_fs/fail_fs.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/ext/test/kvs_bdb/kvs_bdb.c b/src/third_party/wiredtiger/ext/test/kvs_bdb/kvs_bdb.c index 8f857285b2b..0a897c3fdeb 100644 --- a/src/third_party/wiredtiger/ext/test/kvs_bdb/kvs_bdb.c +++ b/src/third_party/wiredtiger/ext/test/kvs_bdb/kvs_bdb.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. @@ -1026,11 +1026,13 @@ wiredtiger_extension_init(WT_CONNECTION *connection, WT_CONFIG_ARG *config) kvs_session_open_cursor, /* session.open_cursor */ kvs_session_rename, /* session.rename */ NULL, /* No session.salvage */ + NULL, /* No session.size */ kvs_session_truncate, /* session.truncate */ NULL, /* No range_truncate */ kvs_session_verify, /* session.verify */ NULL, /* session.checkpoint */ - kvs_terminate /* termination */ + kvs_terminate, /* termination */ + NULL /* lsm_pre_merge */ }; DATA_SOURCE *ds; DB_ENV *dbenv; diff --git a/src/third_party/wiredtiger/import.data b/src/third_party/wiredtiger/import.data index 56ca8b956ad..2113eef94fe 100644 --- a/src/third_party/wiredtiger/import.data +++ b/src/third_party/wiredtiger/import.data @@ -1,5 +1,5 @@ { - "commit": "d8f244717b6338063e0c20628bfa4bb65a821e0c", + "commit": "b85bcfde3b7e41a35017385fd219553e7028c427", "github": "wiredtiger/wiredtiger.git", "vendor": "wiredtiger", "branch": "mongodb-3.6" diff --git a/src/third_party/wiredtiger/lang/java/java_doc.i b/src/third_party/wiredtiger/lang/java/java_doc.i index 28990234e59..768f06e4ec1 100644 --- a/src/third_party/wiredtiger/lang/java/java_doc.i +++ b/src/third_party/wiredtiger/lang/java/java_doc.i @@ -54,6 +54,7 @@ COPYDOC(__wt_session, WT_SESSION, checkpoint) COPYDOC(__wt_session, WT_SESSION, snapshot) COPYDOC(__wt_session, WT_SESSION, transaction_pinned_range) COPYDOC(__wt_session, WT_SESSION, transaction_sync) +COPYDOC(__wt_session, WT_SESSION, breakpoint) COPYDOC(__wt_connection, WT_CONNECTION, async_flush) COPYDOC(__wt_connection, WT_CONNECTION, async_new_op) COPYDOC(__wt_connection, WT_CONNECTION, close) diff --git a/src/third_party/wiredtiger/lang/java/src/com/wiredtiger/db/AsyncCallback.java b/src/third_party/wiredtiger/lang/java/src/com/wiredtiger/db/AsyncCallback.java index b272d611255..52fd4dc5f06 100644 --- a/src/third_party/wiredtiger/lang/java/src/com/wiredtiger/db/AsyncCallback.java +++ b/src/third_party/wiredtiger/lang/java/src/com/wiredtiger/db/AsyncCallback.java @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/lang/java/src/com/wiredtiger/db/PackFormatInputStream.java b/src/third_party/wiredtiger/lang/java/src/com/wiredtiger/db/PackFormatInputStream.java index 5cf52a067b8..0579782d012 100644 --- a/src/third_party/wiredtiger/lang/java/src/com/wiredtiger/db/PackFormatInputStream.java +++ b/src/third_party/wiredtiger/lang/java/src/com/wiredtiger/db/PackFormatInputStream.java @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/lang/java/src/com/wiredtiger/db/PackInputStream.java b/src/third_party/wiredtiger/lang/java/src/com/wiredtiger/db/PackInputStream.java index 013f9601edb..7ada7ce6843 100644 --- a/src/third_party/wiredtiger/lang/java/src/com/wiredtiger/db/PackInputStream.java +++ b/src/third_party/wiredtiger/lang/java/src/com/wiredtiger/db/PackInputStream.java @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/lang/java/src/com/wiredtiger/db/PackOutputStream.java b/src/third_party/wiredtiger/lang/java/src/com/wiredtiger/db/PackOutputStream.java index 9af63db83c9..e5ab774878a 100644 --- a/src/third_party/wiredtiger/lang/java/src/com/wiredtiger/db/PackOutputStream.java +++ b/src/third_party/wiredtiger/lang/java/src/com/wiredtiger/db/PackOutputStream.java @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/lang/java/src/com/wiredtiger/db/PackUtil.java b/src/third_party/wiredtiger/lang/java/src/com/wiredtiger/db/PackUtil.java index 43b627cbf15..b5daaf7cbde 100644 --- a/src/third_party/wiredtiger/lang/java/src/com/wiredtiger/db/PackUtil.java +++ b/src/third_party/wiredtiger/lang/java/src/com/wiredtiger/db/PackUtil.java @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/lang/java/src/com/wiredtiger/db/WiredTigerException.java b/src/third_party/wiredtiger/lang/java/src/com/wiredtiger/db/WiredTigerException.java index 233e1598c5d..a93a829c62e 100644 --- a/src/third_party/wiredtiger/lang/java/src/com/wiredtiger/db/WiredTigerException.java +++ b/src/third_party/wiredtiger/lang/java/src/com/wiredtiger/db/WiredTigerException.java @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/lang/java/src/com/wiredtiger/db/WiredTigerPackingException.java b/src/third_party/wiredtiger/lang/java/src/com/wiredtiger/db/WiredTigerPackingException.java index 73d279f9e85..245acc86240 100644 --- a/src/third_party/wiredtiger/lang/java/src/com/wiredtiger/db/WiredTigerPackingException.java +++ b/src/third_party/wiredtiger/lang/java/src/com/wiredtiger/db/WiredTigerPackingException.java @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/lang/java/src/com/wiredtiger/db/WiredTigerPanicException.java b/src/third_party/wiredtiger/lang/java/src/com/wiredtiger/db/WiredTigerPanicException.java index 4d82ad3d5df..8baf6eea283 100644 --- a/src/third_party/wiredtiger/lang/java/src/com/wiredtiger/db/WiredTigerPanicException.java +++ b/src/third_party/wiredtiger/lang/java/src/com/wiredtiger/db/WiredTigerPanicException.java @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/lang/java/src/com/wiredtiger/db/WiredTigerRollbackException.java b/src/third_party/wiredtiger/lang/java/src/com/wiredtiger/db/WiredTigerRollbackException.java index 84c7e0803a3..b322c2a2608 100644 --- a/src/third_party/wiredtiger/lang/java/src/com/wiredtiger/db/WiredTigerRollbackException.java +++ b/src/third_party/wiredtiger/lang/java/src/com/wiredtiger/db/WiredTigerRollbackException.java @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/lang/java/wiredtiger.i b/src/third_party/wiredtiger/lang/java/wiredtiger.i index 4c22a0af43b..dfac0e9ec8d 100644 --- a/src/third_party/wiredtiger/lang/java/wiredtiger.i +++ b/src/third_party/wiredtiger/lang/java/wiredtiger.i @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/lang/python/setup.py b/src/third_party/wiredtiger/lang/python/setup.py index c88b268fcff..2f86785d511 100644 --- a/src/third_party/wiredtiger/lang/python/setup.py +++ b/src/third_party/wiredtiger/lang/python/setup.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/lang/python/setup_pip.py b/src/third_party/wiredtiger/lang/python/setup_pip.py index 2ddca407e6b..9aa263c8ac9 100644 --- a/src/third_party/wiredtiger/lang/python/setup_pip.py +++ b/src/third_party/wiredtiger/lang/python/setup_pip.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/lang/python/wiredtiger.i b/src/third_party/wiredtiger/lang/python/wiredtiger.i index e976af6a284..0c95be87c1b 100644 --- a/src/third_party/wiredtiger/lang/python/wiredtiger.i +++ b/src/third_party/wiredtiger/lang/python/wiredtiger.i @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. @@ -511,7 +511,6 @@ COMPARE_NOTFOUND_OK(__wt_cursor::_search_near) %exception wiredtiger_version; %exception diagnostic_build; %exception timestamp_build; -%exception verbose_build; /* WT_ASYNC_OP customization. */ /* First, replace the varargs get / set methods with Python equivalents. */ @@ -1013,18 +1012,9 @@ int diagnostic_build() { int timestamp_build() { return WT_TIMESTAMP_SIZE > 0; } - -int verbose_build() { -#ifdef HAVE_VERBOSE - return 1; -#else - return 0; -#endif -} %} int diagnostic_build(); int timestamp_build(); -int verbose_build(); /* Remove / rename parts of the C API that we don't want in Python. */ %immutable __wt_cursor::session; diff --git a/src/third_party/wiredtiger/lang/python/wiredtiger/fpacking.py b/src/third_party/wiredtiger/lang/python/wiredtiger/fpacking.py index 8ae4c1cf99f..a8be044e748 100644 --- a/src/third_party/wiredtiger/lang/python/wiredtiger/fpacking.py +++ b/src/third_party/wiredtiger/lang/python/wiredtiger/fpacking.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/lang/python/wiredtiger/intpacking.py b/src/third_party/wiredtiger/lang/python/wiredtiger/intpacking.py index ed1f00ceb37..c8ee61a37a8 100644 --- a/src/third_party/wiredtiger/lang/python/wiredtiger/intpacking.py +++ b/src/third_party/wiredtiger/lang/python/wiredtiger/intpacking.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/lang/python/wiredtiger/packing.py b/src/third_party/wiredtiger/lang/python/wiredtiger/packing.py index fb674538b76..604b437995b 100644 --- a/src/third_party/wiredtiger/lang/python/wiredtiger/packing.py +++ b/src/third_party/wiredtiger/lang/python/wiredtiger/packing.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/lang/python/wiredtiger/pip_init.py b/src/third_party/wiredtiger/lang/python/wiredtiger/pip_init.py index 71c35fabd57..593c41da357 100644 --- a/src/third_party/wiredtiger/lang/python/wiredtiger/pip_init.py +++ b/src/third_party/wiredtiger/lang/python/wiredtiger/pip_init.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/src/async/async_api.c b/src/third_party/wiredtiger/src/async/async_api.c index 0152238456d..db755db198a 100644 --- a/src/third_party/wiredtiger/src/async/async_api.c +++ b/src/third_party/wiredtiger/src/async/async_api.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/async/async_op.c b/src/third_party/wiredtiger/src/async/async_op.c index dc8a3cdf348..13b2f9ba64f 100644 --- a/src/third_party/wiredtiger/src/async/async_op.c +++ b/src/third_party/wiredtiger/src/async/async_op.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/async/async_worker.c b/src/third_party/wiredtiger/src/async/async_worker.c index 48961da9e42..6dfddced8e7 100644 --- a/src/third_party/wiredtiger/src/async/async_worker.c +++ b/src/third_party/wiredtiger/src/async/async_worker.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/block/block_addr.c b/src/third_party/wiredtiger/src/block/block_addr.c index 6a016776175..2ead0e229bc 100644 --- a/src/third_party/wiredtiger/src/block/block_addr.c +++ b/src/third_party/wiredtiger/src/block/block_addr.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/block/block_ckpt.c b/src/third_party/wiredtiger/src/block/block_ckpt.c index 64d0283a8a3..55df1527e98 100644 --- a/src/third_party/wiredtiger/src/block/block_ckpt.c +++ b/src/third_party/wiredtiger/src/block/block_ckpt.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -62,7 +62,6 @@ __wt_block_checkpoint_load(WT_SESSION_IMPL *session, WT_BLOCK *block, ci = NULL; -#ifdef HAVE_VERBOSE if (WT_VERBOSE_ISSET(session, WT_VERB_CHECKPOINT)) { if (addr != NULL) { WT_ERR(__wt_scr_alloc(session, 0, &tmp)); @@ -72,7 +71,6 @@ __wt_block_checkpoint_load(WT_SESSION_IMPL *session, WT_BLOCK *block, "%s: load-checkpoint: %s", block->name, addr == NULL ? "[Empty]" : (const char *)tmp->data); } -#endif /* * There's a single checkpoint in the file that can be written, all of @@ -545,7 +543,6 @@ __ckpt_process(WT_SESSION_IMPL *session, WT_BLOCK *block, WT_CKPT *ckptbase) !F_ISSET(ckpt, WT_CKPT_DELETE)) continue; -#ifdef HAVE_VERBOSE if (WT_VERBOSE_ISSET(session, WT_VERB_CHECKPOINT)) { if (tmp == NULL) WT_ERR(__wt_scr_alloc(session, 0, &tmp)); @@ -555,7 +552,6 @@ __ckpt_process(WT_SESSION_IMPL *session, WT_BLOCK *block, WT_CKPT *ckptbase) "%s: delete-checkpoint: %s: %s", block->name, ckpt->name, (const char *)tmp->data); } -#endif /* * Find the checkpoint into which we'll roll this checkpoint's * blocks: it's the next real checkpoint in the list, and it diff --git a/src/third_party/wiredtiger/src/block/block_compact.c b/src/third_party/wiredtiger/src/block/block_compact.c index 1e510e08596..9c85250d8bd 100644 --- a/src/third_party/wiredtiger/src/block/block_compact.c +++ b/src/third_party/wiredtiger/src/block/block_compact.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -8,9 +8,7 @@ #include "wt_internal.h" -#ifdef HAVE_VERBOSE static void __block_dump_avail(WT_SESSION_IMPL *, WT_BLOCK *, bool); -#endif /* * __wt_block_compact_start -- @@ -40,19 +38,15 @@ __wt_block_compact_start(WT_SESSION_IMPL *session, WT_BLOCK *block) int __wt_block_compact_end(WT_SESSION_IMPL *session, WT_BLOCK *block) { - WT_UNUSED(session); - /* Restore the original allocation plan. */ __wt_block_configure_first_fit(block, false); -#ifdef HAVE_VERBOSE /* Dump the results of the compaction pass. */ if (WT_VERBOSE_ISSET(session, WT_VERB_COMPACT)) { __wt_spin_lock(session, &block->live_lock); __block_dump_avail(session, block, false); __wt_spin_unlock(session, &block->live_lock); } -#endif return (0); } @@ -80,11 +74,9 @@ __wt_block_compact_skip(WT_SESSION_IMPL *session, WT_BLOCK *block, bool *skipp) __wt_spin_lock(session, &block->live_lock); -#ifdef HAVE_VERBOSE /* Dump the current state of the file. */ if (WT_VERBOSE_ISSET(session, WT_VERB_COMPACT)) __block_dump_avail(session, block, true); -#endif /* Sum the available bytes in the initial 80% and 90% of the file. */ avail_eighty = avail_ninety = 0; @@ -186,7 +178,6 @@ __wt_block_compact_page_skip(WT_SESSION_IMPL *session, } __wt_spin_unlock(session, &block->live_lock); -#ifdef HAVE_VERBOSE if (WT_VERBOSE_ISSET(session, WT_VERB_COMPACT)) { ++block->compact_pages_reviewed; if (*skipp) @@ -194,12 +185,10 @@ __wt_block_compact_page_skip(WT_SESSION_IMPL *session, else ++block->compact_pages_written; } -#endif return (0); } -#ifdef HAVE_VERBOSE /* * __block_dump_avail -- * Dump out the avail list so we can see what compaction will look like. @@ -276,4 +265,3 @@ __block_dump_avail(WT_SESSION_IMPL *session, WT_BLOCK *block, bool start) (uintmax_t)((v * 100) / (wt_off_t)el->bytes)); } } -#endif diff --git a/src/third_party/wiredtiger/src/block/block_ext.c b/src/third_party/wiredtiger/src/block/block_ext.c index c8d8bc3032f..3995fdb4c94 100644 --- a/src/third_party/wiredtiger/src/block/block_ext.c +++ b/src/third_party/wiredtiger/src/block/block_ext.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/block/block_map.c b/src/third_party/wiredtiger/src/block/block_map.c index 847f2393043..5973693fc2d 100644 --- a/src/third_party/wiredtiger/src/block/block_map.c +++ b/src/third_party/wiredtiger/src/block/block_map.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/block/block_mgr.c b/src/third_party/wiredtiger/src/block/block_mgr.c index a2a3fd922a8..a2790863961 100644 --- a/src/third_party/wiredtiger/src/block/block_mgr.c +++ b/src/third_party/wiredtiger/src/block/block_mgr.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/block/block_open.c b/src/third_party/wiredtiger/src/block/block_open.c index 5f2ab947acb..7c7e0e5c525 100644 --- a/src/third_party/wiredtiger/src/block/block_open.c +++ b/src/third_party/wiredtiger/src/block/block_open.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/block/block_read.c b/src/third_party/wiredtiger/src/block/block_read.c index 86b0cad13db..cd419566e40 100644 --- a/src/third_party/wiredtiger/src/block/block_read.c +++ b/src/third_party/wiredtiger/src/block/block_read.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/block/block_session.c b/src/third_party/wiredtiger/src/block/block_session.c index 7f757e16e17..a439e1724b1 100644 --- a/src/third_party/wiredtiger/src/block/block_session.c +++ b/src/third_party/wiredtiger/src/block/block_session.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/block/block_slvg.c b/src/third_party/wiredtiger/src/block/block_slvg.c index ece8eccf8e1..82a5d6f414d 100644 --- a/src/third_party/wiredtiger/src/block/block_slvg.c +++ b/src/third_party/wiredtiger/src/block/block_slvg.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/block/block_vrfy.c b/src/third_party/wiredtiger/src/block/block_vrfy.c index c7439104f83..98791131a9c 100644 --- a/src/third_party/wiredtiger/src/block/block_vrfy.c +++ b/src/third_party/wiredtiger/src/block/block_vrfy.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -20,10 +20,8 @@ static int __verify_set_file_size(WT_SESSION_IMPL *, WT_BLOCK *, WT_CKPT *); /* The bit list ignores the first block: convert to/from a frag/offset. */ #define WT_wt_off_TO_FRAG(block, off) \ ((off) / (block)->allocsize - 1) -#ifdef HAVE_VERBOSE #define WT_FRAG_TO_OFF(block, frag) \ (((wt_off_t)((frag) + 1)) * (block)->allocsize) -#endif /* * __wt_block_verify_start -- @@ -385,8 +383,6 @@ __verify_filefrag_add(WT_SESSION_IMPL *session, WT_BLOCK *block, { uint64_t f, frag, frags, i; - WT_UNUSED(type); /* !HAVE_VERBOSE */ - __wt_verbose(session, WT_VERB_VERIFY, "add file block%s%s%s at %" PRIuMAX "-%" PRIuMAX " (%" PRIuMAX ")", type == NULL ? "" : " (", @@ -464,7 +460,6 @@ __verify_filefrag_chk(WT_SESSION_IMPL *session, WT_BLOCK *block) __bit_set(block->fragfile, last); } -#ifdef HAVE_VERBOSE if (!WT_VERBOSE_ISSET(session, WT_VERB_VERIFY)) continue; @@ -472,7 +467,6 @@ __verify_filefrag_chk(WT_SESSION_IMPL *session, WT_BLOCK *block) "file range %" PRIuMAX "-%" PRIuMAX " never verified", (uintmax_t)WT_FRAG_TO_OFF(block, first), (uintmax_t)WT_FRAG_TO_OFF(block, last)); -#endif } if (count == 0) return (0); @@ -557,7 +551,6 @@ __verify_ckptfrag_chk(WT_SESSION_IMPL *session, WT_BLOCK *block) __bit_clear(block->fragckpt, last); } -#ifdef HAVE_VERBOSE if (!WT_VERBOSE_ISSET(session, WT_VERB_VERIFY)) continue; @@ -565,7 +558,6 @@ __verify_ckptfrag_chk(WT_SESSION_IMPL *session, WT_BLOCK *block) "checkpoint range %" PRIuMAX "-%" PRIuMAX " never verified", (uintmax_t)WT_FRAG_TO_OFF(block, first), (uintmax_t)WT_FRAG_TO_OFF(block, last)); -#endif } if (count == 0) diff --git a/src/third_party/wiredtiger/src/block/block_write.c b/src/third_party/wiredtiger/src/block/block_write.c index 4344cb8497d..db9141a654f 100644 --- a/src/third_party/wiredtiger/src/block/block_write.c +++ b/src/third_party/wiredtiger/src/block/block_write.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/bloom/bloom.c b/src/third_party/wiredtiger/src/bloom/bloom.c index 6f4050b3eb6..edc68169365 100644 --- a/src/third_party/wiredtiger/src/bloom/bloom.c +++ b/src/third_party/wiredtiger/src/bloom/bloom.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/btree/bt_compact.c b/src/third_party/wiredtiger/src/btree/bt_compact.c index 63015312232..0a11f40d6d4 100644 --- a/src/third_party/wiredtiger/src/btree/bt_compact.c +++ b/src/third_party/wiredtiger/src/btree/bt_compact.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/btree/bt_curnext.c b/src/third_party/wiredtiger/src/btree/bt_curnext.c index a5fb19a20e7..6737af9996b 100644 --- a/src/third_party/wiredtiger/src/btree/bt_curnext.c +++ b/src/third_party/wiredtiger/src/btree/bt_curnext.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -136,8 +136,9 @@ new_page: if (cbt->ins == NULL) __cursor_set_recno(cbt, WT_INSERT_RECNO(cbt->ins)); if ((upd = __wt_txn_read(session, cbt->ins->upd)) == NULL) continue; - if (upd->type == WT_UPDATE_DELETED) { - if (__wt_txn_upd_visible_all(session, upd)) + if (upd->type == WT_UPDATE_TOMBSTONE) { + if (upd->txnid != WT_TXN_NONE && + __wt_txn_upd_visible_all(session, upd)) ++cbt->page_deleted_count; continue; } @@ -195,8 +196,9 @@ new_page: /* Find the matching WT_COL slot. */ upd = cbt->ins == NULL ? NULL : __wt_txn_read(session, cbt->ins->upd); if (upd != NULL) { - if (upd->type == WT_UPDATE_DELETED) { - if (__wt_txn_upd_visible_all(session, upd)) + if (upd->type == WT_UPDATE_TOMBSTONE) { + if (upd->txnid != WT_TXN_NONE && + __wt_txn_upd_visible_all(session, upd)) ++cbt->page_deleted_count; continue; } @@ -311,8 +313,9 @@ __cursor_row_next(WT_CURSOR_BTREE *cbt, bool newpage) new_insert: if ((ins = cbt->ins) != NULL) { if ((upd = __wt_txn_read(session, ins->upd)) == NULL) continue; - if (upd->type == WT_UPDATE_DELETED) { - if (__wt_txn_upd_visible_all(session, upd)) + if (upd->type == WT_UPDATE_TOMBSTONE) { + if (upd->txnid != WT_TXN_NONE && + __wt_txn_upd_visible_all(session, upd)) ++cbt->page_deleted_count; continue; } @@ -342,8 +345,9 @@ new_insert: if ((ins = cbt->ins) != NULL) { cbt->slot = cbt->row_iteration_slot / 2 - 1; rip = &page->pg_row[cbt->slot]; upd = __wt_txn_read(session, WT_ROW_UPDATE(page, rip)); - if (upd != NULL && upd->type == WT_UPDATE_DELETED) { - if (__wt_txn_upd_visible_all(session, upd)) + if (upd != NULL && upd->type == WT_UPDATE_TOMBSTONE) { + if (upd->txnid != WT_TXN_NONE && + __wt_txn_upd_visible_all(session, upd)) ++cbt->page_deleted_count; continue; } @@ -417,9 +421,10 @@ __cursor_key_order_check_row( WT_PANIC_ERR(session, EINVAL, "WT_CURSOR.%s out-of-order returns: returned key %s then key %s", next ? "next" : "prev", - __wt_buf_set_printable( - session, cbt->lastkey->data, cbt->lastkey->size, a), - __wt_buf_set_printable(session, key->data, key->size, b)); + __wt_buf_set_printable_format(session, + cbt->lastkey->data, cbt->lastkey->size, btree->key_format, a), + __wt_buf_set_printable_format(session, + key->data, key->size, btree->key_format, b)); err: __wt_scr_free(session, &a); __wt_scr_free(session, &b); diff --git a/src/third_party/wiredtiger/src/btree/bt_curprev.c b/src/third_party/wiredtiger/src/btree/bt_curprev.c index 50a755f1431..068a9915ab9 100644 --- a/src/third_party/wiredtiger/src/btree/bt_curprev.c +++ b/src/third_party/wiredtiger/src/btree/bt_curprev.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -282,8 +282,9 @@ new_page: if (cbt->ins == NULL) __cursor_set_recno(cbt, WT_INSERT_RECNO(cbt->ins)); if ((upd = __wt_txn_read(session, cbt->ins->upd)) == NULL) continue; - if (upd->type == WT_UPDATE_DELETED) { - if (__wt_txn_upd_visible_all(session, upd)) + if (upd->type == WT_UPDATE_TOMBSTONE) { + if (upd->txnid != WT_TXN_NONE && + __wt_txn_upd_visible_all(session, upd)) ++cbt->page_deleted_count; continue; } @@ -342,8 +343,9 @@ new_page: if (cbt->recno < cbt->ref->ref_recno) upd = cbt->ins == NULL ? NULL : __wt_txn_read(session, cbt->ins->upd); if (upd != NULL) { - if (upd->type == WT_UPDATE_DELETED) { - if (__wt_txn_upd_visible_all(session, upd)) + if (upd->type == WT_UPDATE_TOMBSTONE) { + if (upd->txnid != WT_TXN_NONE && + __wt_txn_upd_visible_all(session, upd)) ++cbt->page_deleted_count; continue; } @@ -468,8 +470,9 @@ __cursor_row_prev(WT_CURSOR_BTREE *cbt, bool newpage) new_insert: if ((ins = cbt->ins) != NULL) { if ((upd = __wt_txn_read(session, ins->upd)) == NULL) continue; - if (upd->type == WT_UPDATE_DELETED) { - if (__wt_txn_upd_visible_all(session, upd)) + if (upd->type == WT_UPDATE_TOMBSTONE) { + if (upd->txnid != WT_TXN_NONE && + __wt_txn_upd_visible_all(session, upd)) ++cbt->page_deleted_count; continue; } @@ -501,8 +504,9 @@ new_insert: if ((ins = cbt->ins) != NULL) { cbt->slot = cbt->row_iteration_slot / 2 - 1; rip = &page->pg_row[cbt->slot]; upd = __wt_txn_read(session, WT_ROW_UPDATE(page, rip)); - if (upd != NULL && upd->type == WT_UPDATE_DELETED) { - if (__wt_txn_upd_visible_all(session, upd)) + if (upd != NULL && upd->type == WT_UPDATE_TOMBSTONE) { + if (upd->txnid != WT_TXN_NONE && + __wt_txn_upd_visible_all(session, upd)) ++cbt->page_deleted_count; continue; } diff --git a/src/third_party/wiredtiger/src/btree/bt_cursor.c b/src/third_party/wiredtiger/src/btree/bt_cursor.c index e5d5e64194f..8c7170e6180 100644 --- a/src/third_party/wiredtiger/src/btree/bt_cursor.c +++ b/src/third_party/wiredtiger/src/btree/bt_cursor.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -55,15 +55,48 @@ __cursor_state_restore(WT_CURSOR *cursor, WT_CURFILE_STATE *state) * Return if we have a page pinned. */ static inline bool -__cursor_page_pinned(WT_CURSOR_BTREE *cbt, bool eviction_ok) +__cursor_page_pinned(WT_CURSOR_BTREE *cbt) { + WT_CURSOR *cursor; + WT_SESSION_IMPL *session; + + cursor = &cbt->iface; + session = (WT_SESSION_IMPL *)cursor->session; + + /* + * Check the page active flag, asserting the page reference with any + * external key. + */ + if (!F_ISSET(cbt, WT_CBT_ACTIVE)) { + WT_ASSERT((WT_SESSION_IMPL *)cursor->session, + cbt->ref == NULL && !F_ISSET(cursor, WT_CURSTD_KEY_INT)); + return (false); + } + + /* + * Check if the key references the page. When returning from search, + * the page is active and the key is internal. After the application + * sets a key, the key is external, and the page is useless. + */ + if (!F_ISSET(cursor, WT_CURSTD_KEY_INT)) + return (false); + + /* + * Fail if the page is flagged for forced eviction (so we periodically + * release pages grown too large). + */ + if (cbt->ref->page->read_gen == WT_READGEN_OLDEST) + return (false); + /* - * Optionally fail the page-pinned test when the page is flagged for - * forced eviction (so we periodically release pages grown too large). - * The test is optional as not all callers can release pinned pages. + * If we are doing an update, we need a page with history. Release the + * page so we get it again with history if required. */ - return (F_ISSET(cbt, WT_CBT_ACTIVE) && - (!eviction_ok || cbt->ref->page->read_gen != WT_READGEN_OLDEST)); + if (F_ISSET(&session->txn, WT_TXN_UPDATE) && + cbt->ref->state != WT_REF_MEM) + return (false); + + return (true); } /* @@ -146,7 +179,7 @@ __cursor_disable_bulk(WT_SESSION_IMPL *session, WT_BTREE *btree) * __cursor_fix_implicit -- * Return if search went past the end of the tree. */ -static inline int +static inline bool __cursor_fix_implicit(WT_BTREE *btree, WT_CURSOR_BTREE *cbt) { /* @@ -229,7 +262,7 @@ __wt_cursor_valid(WT_CURSOR_BTREE *cbt, WT_UPDATE **updp) */ if (cbt->ins != NULL && (upd = __wt_txn_read(session, cbt->ins->upd)) != NULL) { - if (upd->type == WT_UPDATE_DELETED) + if (upd->type == WT_UPDATE_TOMBSTONE) return (false); if (updp != NULL) *updp = upd; @@ -302,7 +335,7 @@ __wt_cursor_valid(WT_CURSOR_BTREE *cbt, WT_UPDATE **updp) page->modify->mod_row_update != NULL && (upd = __wt_txn_read(session, page->modify->mod_row_update[cbt->slot])) != NULL) { - if (upd->type == WT_UPDATE_DELETED) + if (upd->type == WT_UPDATE_TOMBSTONE) return (false); if (updp != NULL) *updp = upd; @@ -468,7 +501,7 @@ __wt_btcur_search(WT_CURSOR_BTREE *cbt) * from the root. */ valid = false; - if (__cursor_page_pinned(cbt, true)) { + if (__cursor_page_pinned(cbt)) { __wt_txn_cursor_op(session); WT_ERR(btree->type == BTREE_ROW ? @@ -565,7 +598,7 @@ __wt_btcur_search_near(WT_CURSOR_BTREE *cbt, int *exactp) * existing record. */ valid = false; - if (btree->type == BTREE_ROW && __cursor_page_pinned(cbt, true)) { + if (btree->type == BTREE_ROW && __cursor_page_pinned(cbt)) { __wt_txn_cursor_op(session); WT_ERR(__cursor_row_search(session, cbt, cbt->ref, true)); @@ -691,14 +724,12 @@ __wt_btcur_insert(WT_CURSOR_BTREE *cbt) /* * If inserting with overwrite configured, and positioned to an on-page - * key, the update doesn't require another search. The cursor won't be - * positioned on a page with an external key set, but be sure. Cursors - * configured for append aren't included, regardless of whether or not - * they meet all other criteria. + * key, the update doesn't require another search. Cursors configured + * for append aren't included, regardless of whether or not they meet + * all other criteria. */ - if (__cursor_page_pinned(cbt, true) && - F_ISSET_ALL(cursor, WT_CURSTD_KEY_INT | WT_CURSTD_OVERWRITE) && - !append_key) { + if (__cursor_page_pinned(cbt) && + F_ISSET(cursor, WT_CURSTD_OVERWRITE) && !append_key) { WT_ERR(__wt_txn_autocommit_check(session)); /* * The cursor position may not be exact (the cursor's comparison @@ -771,7 +802,7 @@ retry: WT_ERR(__cursor_func_init(cbt, true)); WT_ERR(__cursor_col_modify(session, cbt, WT_UPDATE_STANDARD)); if (append_key) - cbt->iface.recno = cbt->recno; + cursor->recno = cbt->recno; } err: if (ret == WT_RESTART) { @@ -784,7 +815,7 @@ done: /* Insert doesn't maintain a position across calls, clear resources. */ if (ret == 0) { F_CLR(cursor, WT_CURSTD_KEY_SET | WT_CURSTD_VALUE_SET); if (append_key) - F_SET(cursor, WT_CURSTD_KEY_INT); + F_SET(cursor, WT_CURSTD_KEY_EXT); } WT_TRET(__cursor_reset(cbt)); if (ret != 0) @@ -884,12 +915,12 @@ err: if (ret == WT_RESTART) { int __wt_btcur_remove(WT_CURSOR_BTREE *cbt) { + enum { NO_POSITION, POSITIONED, SEARCH_POSITION } positioned; WT_BTREE *btree; WT_CURFILE_STATE state; WT_CURSOR *cursor; WT_DECL_RET; WT_SESSION_IMPL *session; - bool positioned; btree = cbt->btree; cursor = &cbt->iface; @@ -902,8 +933,27 @@ __wt_btcur_remove(WT_CURSOR_BTREE *cbt) /* * WT_CURSOR.remove has a unique semantic, the cursor stays positioned * if it starts positioned, otherwise clear the cursor on completion. + * + * However, if we unpin the page (because the page is in WT_REF_LIMBO or + * it was selected for forcible eviction), and every item on the page is + * deleted, eviction can delete the page and our subsequent search will + * re-instantiate an empty page for us, with no key/value pairs. Cursor + * remove will search that page and return not-found, which is OK unless + * cursor-overwrite is configured (which causes cursor remove to return + * success even if there's no item to delete). In that case, we're + * supposed to return a positioned cursor, but there's nothing to which + * we can position, and we'll fail attempting to point the cursor at the + * key on the page to satisfy the positioned requirement. + * + * Do the best we can: If we start with a positioned cursor, and we let + * go of our pinned page, reset our state to use the search position, + * that is, use a successful search to return to a "positioned" state. + * If we start with a positioned cursor, let go of our pinned page, and + * the search fails, leave the cursor's key set so the cursor appears + * positioned to the application. */ - positioned = F_ISSET(cursor, WT_CURSTD_KEY_INT); + positioned = + F_ISSET(cursor, WT_CURSTD_KEY_INT) ? POSITIONED : NO_POSITION; /* Save the cursor state. */ __cursor_state_save(cursor, &state); @@ -912,9 +962,7 @@ __wt_btcur_remove(WT_CURSOR_BTREE *cbt) * If remove positioned to an on-page key, the remove doesn't require * another search. We don't care about the "overwrite" configuration * because regardless of the overwrite setting, any existing record is - * removed, and the record must exist with a positioned cursor. The - * cursor won't be positioned on a page with an external key set, but - * be sure. + * removed, and the record must exist with a positioned cursor. * * There's trickiness in the page-pinned check. By definition a remove * operation leaves a cursor positioned if it's initially positioned. @@ -929,8 +977,7 @@ __wt_btcur_remove(WT_CURSOR_BTREE *cbt) * that's all a positioned cursor implies), but it's probably safer to * avoid page eviction entirely in the positioned case. */ - if (__cursor_page_pinned(cbt, !positioned) && - F_ISSET(cursor, WT_CURSTD_KEY_INT)) { + if (__cursor_page_pinned(cbt)) { WT_ERR(__wt_txn_autocommit_check(session)); /* @@ -940,8 +987,8 @@ __wt_btcur_remove(WT_CURSOR_BTREE *cbt) */ cbt->compare = 0; ret = btree->type == BTREE_ROW ? - __cursor_row_modify(session, cbt, WT_UPDATE_DELETED) : - __cursor_col_modify(session, cbt, WT_UPDATE_DELETED); + __cursor_row_modify(session, cbt, WT_UPDATE_TOMBSTONE) : + __cursor_col_modify(session, cbt, WT_UPDATE_TOMBSTONE); if (ret == 0) goto done; @@ -959,6 +1006,9 @@ __wt_btcur_remove(WT_CURSOR_BTREE *cbt) goto err; } + if (positioned == POSITIONED) + positioned = SEARCH_POSITION; + /* * The pinned page goes away if we do a search, get a local copy of any * pinned key and discard any value (remove discards any previous @@ -980,7 +1030,7 @@ retry: WT_ERR(__cursor_func_init(cbt, true)); if (cbt->compare != 0 || !__wt_cursor_valid(cbt, NULL)) WT_ERR(WT_NOTFOUND); - ret = __cursor_row_modify(session, cbt, WT_UPDATE_DELETED); + ret = __cursor_row_modify(session, cbt, WT_UPDATE_TOMBSTONE); } else { WT_ERR(__cursor_col_search(session, cbt, NULL)); @@ -1008,7 +1058,7 @@ retry: WT_ERR(__cursor_func_init(cbt, true)); cbt->recno = cursor->recno; } else ret = __cursor_col_modify( - session, cbt, WT_UPDATE_DELETED); + session, cbt, WT_UPDATE_TOMBSTONE); } err: if (ret == WT_RESTART) { @@ -1017,27 +1067,51 @@ err: if (ret == WT_RESTART) { goto retry; } - /* - * If the cursor is configured to overwrite and the record is not found, - * that is exactly what we want, return success. - */ - if (F_ISSET(cursor, WT_CURSTD_OVERWRITE) && ret == WT_NOTFOUND) - ret = 0; +done: if (ret == 0) { + F_CLR(cursor, WT_CURSTD_VALUE_SET); + switch (positioned) { + case NO_POSITION: + /* + * Never positioned and we leave it that way, clear any + * key and reset the cursor. + */ + F_CLR(cursor, WT_CURSTD_KEY_SET); + WT_TRET(__cursor_reset(cbt)); + break; + case POSITIONED: + /* + * Positioned and we used the pinned page, leave the key + * alone, whatever it is. + */ + break; + case SEARCH_POSITION: + /* + * Positioned and we did a search anyway, get a key to + * return. + */ + WT_TRET(__wt_key_return(session, cbt)); + break; + } + } -done: /* - * If the cursor was positioned, it stays positioned, point the cursor - * at an internal copy of the key. Otherwise, there's no position or - * key/value. - */ - if (ret == 0) - F_CLR(cursor, WT_CURSTD_KEY_SET | WT_CURSTD_VALUE_SET); - if (ret == 0 && positioned) - WT_TRET(__wt_key_return(session, cbt)); - else + if (ret != 0) { WT_TRET(__cursor_reset(cbt)); - if (ret != 0) __cursor_state_restore(cursor, &state); + /* + * If the cursor is configured to overwrite and the record isn't + * found, that is exactly what we want, return success. Note we + * set clear the return value after everything else, the clause + * above dealing with the cursor position is only correct if we + * were successful. If search failed after positioned is set to + * SEARCH_POSITION, we cannot return a key. The only action to + * take is to set the cursor to its original key, which we just + * did. + */ + if (F_ISSET(cursor, WT_CURSTD_OVERWRITE) && ret == WT_NOTFOUND) + ret = 0; + } + return (ret); } @@ -1068,12 +1142,9 @@ __btcur_update(WT_CURSOR_BTREE *cbt, WT_ITEM *value, u_int modify_type) * If update positioned to an on-page key, the update doesn't require * another search. We don't care about the "overwrite" configuration * because regardless of the overwrite setting, any existing record is - * updated, and the record must exist with a positioned cursor. The - * cursor won't be positioned on a page with an external key set, but - * be sure. + * updated, and the record must exist with a positioned cursor. */ - if (__cursor_page_pinned(cbt, true) && - F_ISSET(cursor, WT_CURSTD_KEY_INT)) { + if (__cursor_page_pinned(cbt)) { WT_ERR(__wt_txn_autocommit_check(session)); /* @@ -1168,20 +1239,21 @@ done: if (ret == 0) WT_TRET(__cursor_kv_return( session, cbt, cbt->modify_update)); break; - case WT_UPDATE_RESERVED: + case WT_UPDATE_RESERVE: /* * WT_CURSOR.reserve doesn't return any value. */ F_CLR(cursor, WT_CURSTD_VALUE_SET); /* FALLTHROUGH */ - case WT_UPDATE_MODIFIED: + case WT_UPDATE_MODIFY: /* * WT_CURSOR.modify has already created the return value * and our job is to leave it untouched. */ WT_TRET(__wt_key_return(session, cbt)); break; - case WT_UPDATE_DELETED: + case WT_UPDATE_BIRTHMARK: + case WT_UPDATE_TOMBSTONE: default: WT_TRET(__wt_illegal_value(session, NULL)); break; @@ -1268,8 +1340,7 @@ __wt_btcur_modify(WT_CURSOR_BTREE *cbt, WT_MODIFY *entries, int nentries) WT_ERR(__wt_btcur_search(cbt)); orig = cursor->value.size; - WT_ERR(__wt_modify_apply_api( - session, &cursor->value, entries, nentries)); + WT_ERR(__wt_modify_apply_api(session, cursor, entries, nentries)); new = cursor->value.size; WT_ERR(__cursor_size_chk(session, &cursor->value)); if (new > orig) @@ -1290,7 +1361,7 @@ __wt_btcur_modify(WT_CURSOR_BTREE *cbt, WT_MODIFY *entries, int nentries) ret = __btcur_update(cbt, &cursor->value, WT_UPDATE_STANDARD); else if ((ret = __wt_modify_pack(session, &modify, entries, nentries)) == 0) - ret = __btcur_update(cbt, modify, WT_UPDATE_MODIFIED); + ret = __btcur_update(cbt, modify, WT_UPDATE_MODIFY); if (overwrite) F_SET(cursor, WT_CURSTD_OVERWRITE); @@ -1330,7 +1401,7 @@ __wt_btcur_reserve(WT_CURSOR_BTREE *cbt) /* WT_CURSOR.reserve is update-without-overwrite and a special value. */ overwrite = F_ISSET(cursor, WT_CURSTD_OVERWRITE); F_CLR(cursor, WT_CURSTD_OVERWRITE); - ret = __btcur_update(cbt, &cursor->value, WT_UPDATE_RESERVED); + ret = __btcur_update(cbt, &cursor->value, WT_UPDATE_RESERVE); if (overwrite) F_SET(cursor, WT_CURSTD_OVERWRITE); return (ret); @@ -1510,7 +1581,7 @@ retry: WT_RET(__wt_btcur_search(start)); F_MASK((WT_CURSOR *)start, WT_CURSTD_KEY_SET) == WT_CURSTD_KEY_INT); for (;;) { - if ((ret = rmfunc(session, start, WT_UPDATE_DELETED)) != 0) + if ((ret = rmfunc(session, start, WT_UPDATE_TOMBSTONE)) != 0) break; if (stop != NULL && __cursor_equals(start, stop)) @@ -1568,7 +1639,7 @@ retry: WT_RET(__wt_btcur_search(start)); for (;;) { value = (const uint8_t *)start->iface.value.data; if (*value != 0 && - (ret = rmfunc(session, start, WT_UPDATE_DELETED)) != 0) + (ret = rmfunc(session, start, WT_UPDATE_TOMBSTONE)) != 0) break; if (stop != NULL && __cursor_equals(start, stop)) diff --git a/src/third_party/wiredtiger/src/btree/bt_debug.c b/src/third_party/wiredtiger/src/btree/bt_debug.c index 3df5920830c..325aec853e5 100644 --- a/src/third_party/wiredtiger/src/btree/bt_debug.c +++ b/src/third_party/wiredtiger/src/btree/bt_debug.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -20,13 +20,16 @@ struct __wt_dbg { * When using the standard event handlers, the debugging output has to * do its own message handling because its output isn't line-oriented. */ - FILE *fp; /* Optional file handle */ - WT_ITEM *msg; /* Buffered message */ + FILE *fp; /* Optional file handle */ + WT_ITEM *msg; /* Buffered message */ int (*f)(WT_DBG *, const char *, ...) /* Function to write */ WT_GCC_FUNC_DECL_ATTRIBUTE((format (printf, 2, 3))); - WT_ITEM *tmp; /* Temporary space */ + const char *key_format; + const char *value_format; + + WT_ITEM *tmp; /* Temporary space */ }; static const /* Output separator */ @@ -102,7 +105,7 @@ __debug_bytes(WT_DBG *ds, const void *data_arg, size_t size) /* * __debug_item -- - * Dump a single data/size pair, with an optional tag. + * Dump a single data/size item, with an optional tag. */ static int __debug_item(WT_DBG *ds, const char *tag, const void *data_arg, size_t size) @@ -115,6 +118,33 @@ __debug_item(WT_DBG *ds, const char *tag, const void *data_arg, size_t size) } /* + * __debug_item_key -- + * Dump a single data/size key item, with an optional tag. + */ +static int +__debug_item_key(WT_DBG *ds, const char *tag, const void *data_arg, size_t size) +{ + return (ds->f(ds, "\t%s%s{%s}\n", + tag == NULL ? "" : tag, tag == NULL ? "" : " ", + __wt_buf_set_printable_format( + ds->session, data_arg, size, ds->key_format, ds->tmp))); +} + +/* + * __debug_item_value -- + * Dump a single data/size value item, with an optional tag. + */ +static int +__debug_item_value( + WT_DBG *ds, const char *tag, const void *data_arg, size_t size) +{ + return (ds->f(ds, "\t%s%s{%s}\n", + tag == NULL ? "" : tag, tag == NULL ? "" : " ", + __wt_buf_set_printable_format( + ds->session, data_arg, size, ds->value_format, ds->tmp))); +} + +/* * __dmsg_event -- * Send a debug message to the event handler. */ @@ -193,6 +223,8 @@ __dmsg_file(WT_DBG *ds, const char *fmt, ...) static int __debug_config(WT_SESSION_IMPL *session, WT_DBG *ds, const char *ofile) { + WT_BTREE *btree; + memset(ds, 0, sizeof(WT_DBG)); ds->session = session; @@ -213,6 +245,9 @@ __debug_config(WT_SESSION_IMPL *session, WT_DBG *ds, const char *ofile) ds->f = __dmsg_file; } + btree = S2BT_SAFE(session); + ds->key_format = btree->key_format; + ds->value_format = btree->value_format; return (0); } @@ -557,8 +592,10 @@ __wt_debug_tree_shape( return (__dmsg_wrapup(ds)); } -#define WT_DEBUG_TREE_LEAF 0x01 /* Debug leaf pages */ -#define WT_DEBUG_TREE_WALK 0x02 /* Descend the tree */ +/* AUTOMATIC FLAG VALUE GENERATION START */ +#define WT_DEBUG_TREE_LEAF 0x1u /* Debug leaf pages */ +#define WT_DEBUG_TREE_WALK 0x2u /* Descend the tree */ +/* AUTOMATIC FLAG VALUE GENERATION STOP */ /* * __wt_debug_tree_all -- @@ -920,7 +957,7 @@ __debug_page_row_int(WT_DBG *ds, WT_PAGE *page, uint32_t flags) WT_INTL_FOREACH_BEGIN(session, page, ref) { __wt_ref_key(page, ref, &p, &len); - WT_RET(__debug_item(ds, "K", p, len)); + WT_RET(__debug_item_key(ds, "K", p, len)); WT_RET(__debug_ref(ds, ref)); } WT_INTL_FOREACH_END; @@ -965,7 +1002,7 @@ __debug_page_row_leaf(WT_DBG *ds, WT_PAGE *page) /* Dump the page's K/V pairs. */ WT_ROW_FOREACH(page, rip, i) { WT_ERR(__wt_row_leaf_key(session, page, rip, key, false)); - WT_ERR(__debug_item(ds, "K", key->data, key->size)); + WT_ERR(__debug_item_key(ds, "K", key->data, key->size)); if ((cell = __wt_row_leaf_value_cell(page, rip, NULL)) == NULL) WT_ERR(ds->f(ds, "\tV {}\n")); @@ -1014,7 +1051,7 @@ __debug_row_skip(WT_DBG *ds, WT_INSERT_HEAD *head) WT_INSERT *ins; WT_SKIP_FOREACH(ins, head) { - WT_RET(__debug_item(ds, + WT_RET(__debug_item_key(ds, "insert", WT_INSERT_KEY(ins), WT_INSERT_KEY_SIZE(ins))); WT_RET(__debug_update(ds, ins->upd, false)); } @@ -1022,11 +1059,11 @@ __debug_row_skip(WT_DBG *ds, WT_INSERT_HEAD *head) } /* - * __debug_modified -- - * Dump a modified update. + * __debug_modify -- + * Dump a modify update. */ static int -__debug_modified(WT_DBG *ds, WT_UPDATE *upd) +__debug_modify(WT_DBG *ds, WT_UPDATE *upd) { size_t nentries, data_size, offset, size; const size_t *p; @@ -1061,16 +1098,19 @@ __debug_update(WT_DBG *ds, WT_UPDATE *upd, bool hexbyte) { for (; upd != NULL; upd = upd->next) { switch (upd->type) { - case WT_UPDATE_DELETED: - WT_RET(ds->f(ds, "\tvalue {deleted}\n")); + case WT_UPDATE_INVALID: + WT_RET(ds->f(ds, "\tvalue {invalid}\n")); + break; + case WT_UPDATE_BIRTHMARK: + WT_RET(ds->f(ds, "\tvalue {birthmark}\n")); break; - case WT_UPDATE_MODIFIED: - WT_RET(ds->f(ds, "\tvalue {modified: ")); - WT_RET(__debug_modified(ds, upd)); + case WT_UPDATE_MODIFY: + WT_RET(ds->f(ds, "\tvalue {modify: ")); + WT_RET(__debug_modify(ds, upd)); WT_RET(ds->f(ds, "}\n")); break; - case WT_UPDATE_RESERVED: - WT_RET(ds->f(ds, "\tvalue {reserved}\n")); + case WT_UPDATE_RESERVE: + WT_RET(ds->f(ds, "\tvalue {reserve}\n")); break; case WT_UPDATE_STANDARD: if (hexbyte) { @@ -1078,9 +1118,12 @@ __debug_update(WT_DBG *ds, WT_UPDATE *upd, bool hexbyte) WT_RET(__debug_hex_byte(ds, *upd->data)); WT_RET(ds->f(ds, "}\n")); } else - WT_RET(__debug_item(ds, + WT_RET(__debug_item_value(ds, "value", upd->data, upd->size)); break; + case WT_UPDATE_TOMBSTONE: + WT_RET(ds->f(ds, "\tvalue {tombstone}\n")); + break; } if (upd->txnid == WT_TXN_ABORTED) WT_RET(ds->f(ds, "\t" "txn aborted")); @@ -1246,10 +1289,8 @@ __debug_cell_data(WT_DBG *ds, * Column-store references to deleted cells return a NULL cell * reference. */ - if (unpack == NULL) { - WT_RET(__debug_item(ds, tag, "deleted", strlen("deleted"))); - return (0); - } + if (unpack == NULL) + return (__debug_item(ds, tag, "deleted", strlen("deleted"))); switch (unpack->raw) { case WT_CELL_ADDR_DEL: @@ -1260,28 +1301,32 @@ __debug_cell_data(WT_DBG *ds, case WT_CELL_KEY_OVFL_RM: case WT_CELL_VALUE_OVFL_RM: p = __wt_cell_type_string(unpack->raw); - WT_RET(__debug_item(ds, tag, p, strlen(p))); - break; + return (__debug_item(ds, tag, p, strlen(p))); + } + + WT_RET(__wt_scr_alloc(session, 256, &buf)); + WT_ERR(page == NULL ? + __wt_dsk_cell_data_ref(session, page_type, unpack, buf) : + __wt_page_cell_data_ref(session, page, unpack, buf)); + + switch (unpack->raw) { case WT_CELL_KEY: case WT_CELL_KEY_OVFL: case WT_CELL_KEY_PFX: case WT_CELL_KEY_SHORT: case WT_CELL_KEY_SHORT_PFX: + WT_ERR(__debug_item_key(ds, tag, buf->data, buf->size)); + break; case WT_CELL_VALUE: case WT_CELL_VALUE_COPY: case WT_CELL_VALUE_OVFL: case WT_CELL_VALUE_SHORT: - WT_RET(__wt_scr_alloc(session, 256, &buf)); - ret = page == NULL ? - __wt_dsk_cell_data_ref(session, page_type, unpack, buf) : - __wt_page_cell_data_ref(session, page, unpack, buf); - if (ret == 0) - WT_RET(__debug_item(ds, tag, buf->data, buf->size)); - __wt_scr_free(session, &buf); + WT_ERR(__debug_item_value(ds, tag, buf->data, buf->size)); break; - WT_ILLEGAL_VALUE(session); + WT_ILLEGAL_VALUE_ERR(session); } +err: __wt_scr_free(session, &buf); return (ret); } #endif diff --git a/src/third_party/wiredtiger/src/btree/bt_delete.c b/src/third_party/wiredtiger/src/btree/bt_delete.c index 20e592d12bc..a728341e033 100644 --- a/src/third_party/wiredtiger/src/btree/bt_delete.c +++ b/src/third_party/wiredtiger/src/btree/bt_delete.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -158,6 +158,7 @@ __wt_delete_page_rollback(WT_SESSION_IMPL *session, WT_REF *ref) for (sleep_count = yield_count = 0;;) { switch (ref->state) { case WT_REF_DISK: + case WT_REF_LIMBO: case WT_REF_LOOKASIDE: case WT_REF_READING: WT_ASSERT(session, 0); /* Impossible, assert */ @@ -341,7 +342,7 @@ __wt_delete_page_instantiate(WT_SESSION_IMPL *session, WT_REF *ref) */ for (i = 0, size = 0; i < page->entries; ++i) { WT_ERR(__wt_calloc_one(session, &upd)); - upd->type = WT_UPDATE_DELETED; + upd->type = WT_UPDATE_TOMBSTONE; if (page_del == NULL) upd->txnid = WT_TXN_NONE; /* Globally visible */ diff --git a/src/third_party/wiredtiger/src/btree/bt_discard.c b/src/third_party/wiredtiger/src/btree/bt_discard.c index 1aae991a407..66974c70e04 100644 --- a/src/third_party/wiredtiger/src/btree/bt_discard.c +++ b/src/third_party/wiredtiger/src/btree/bt_discard.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -16,14 +16,13 @@ static void __free_skip_array( WT_SESSION_IMPL *, WT_INSERT_HEAD **, uint32_t, bool); static void __free_skip_list(WT_SESSION_IMPL *, WT_INSERT *, bool); static void __free_update(WT_SESSION_IMPL *, WT_UPDATE **, uint32_t, bool); -static void __page_out_int(WT_SESSION_IMPL *, WT_PAGE **, bool); /* - * __wt_ref_out_int -- + * __wt_ref_out -- * Discard an in-memory page, freeing all memory associated with it. */ void -__wt_ref_out_int(WT_SESSION_IMPL *session, WT_REF *ref, bool rewrite) +__wt_ref_out(WT_SESSION_IMPL *session, WT_REF *ref) { /* * A version of the page-out function that allows us to make additional @@ -57,25 +56,15 @@ __wt_ref_out_int(WT_SESSION_IMPL *session, WT_REF *ref, bool rewrite) } #endif - __page_out_int(session, &ref->page, rewrite); + __wt_page_out(session, &ref->page); } /* - * __wt_ref_out -- + * __wt_page_out -- * Discard an in-memory page, freeing all memory associated with it. */ void -__wt_ref_out(WT_SESSION_IMPL *session, WT_REF *ref) -{ - __wt_ref_out_int(session, ref, false); -} - -/* - * __page_out_int -- - * Discard an in-memory page, freeing all memory associated with it. - */ -static void -__page_out_int(WT_SESSION_IMPL *session, WT_PAGE **pagep, bool rewrite) +__wt_page_out(WT_SESSION_IMPL *session, WT_PAGE **pagep) { WT_PAGE *page; WT_PAGE_HEADER *dsk; @@ -113,7 +102,7 @@ __page_out_int(WT_SESSION_IMPL *session, WT_PAGE **pagep, bool rewrite) } /* Update the cache's information. */ - __wt_cache_page_evict(session, page, rewrite); + __wt_cache_page_evict(session, page); dsk = (WT_PAGE_HEADER *)page->dsk; if (F_ISSET_ATOMIC(page, WT_PAGE_DISK_ALLOC)) @@ -158,16 +147,6 @@ __page_out_int(WT_SESSION_IMPL *session, WT_PAGE **pagep, bool rewrite) } /* - * __wt_page_out -- - * Discard an in-memory page, freeing all memory associated with it. - */ -void -__wt_page_out(WT_SESSION_IMPL *session, WT_PAGE **pagep) -{ - __page_out_int(session, pagep, false); -} - -/* * __free_page_modify -- * Discard the page's associated modification structures. */ @@ -310,20 +289,12 @@ __wt_free_ref( break; } - /* - * Free any address allocation; if there's no linked WT_REF page, it - * must be allocated. - */ + /* Free any address allocation. */ __wt_ref_addr_free(session, ref); - /* - * Free any lookaside or page-deleted information. We only expect a - * lookaside structure for lookaside references, but can see - * page-deleted information in other cases (such as WT_REF_MEM). - */ - if (ref->state == WT_REF_LOOKASIDE) - __wt_free(session, ref->page_las); - else if (ref->page_del != NULL) { + /* Free any lookaside or page-deleted information. */ + __wt_free(session, ref->page_las); + if (ref->page_del != NULL) { __wt_free(session, ref->page_del->update_list); __wt_free(session, ref->page_del); } diff --git a/src/third_party/wiredtiger/src/btree/bt_handle.c b/src/third_party/wiredtiger/src/btree/bt_handle.c index 4ab88cea01e..370e81673d8 100644 --- a/src/third_party/wiredtiger/src/btree/bt_handle.c +++ b/src/third_party/wiredtiger/src/btree/bt_handle.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -292,9 +292,6 @@ __btree_conf(WT_SESSION_IMPL *session, WT_CKPT *ckpt) const char **cfg, *enc_cfg[] = { NULL, NULL }; bool fixed; - WT_UNUSED(maj_version); /* !HAVE_VERBOSE */ - WT_UNUSED(min_version); /* !HAVE_VERBOSE */ - btree = S2BT(session); cfg = btree->dhandle->cfg; conn = S2C(session); @@ -403,13 +400,20 @@ __btree_conf(WT_SESSION_IMPL *session, WT_CKPT *ckpt) cfg, "assert.commit_timestamp", &cval)); if (WT_STRING_MATCH("always", cval.str, cval.len)) { FLD_SET(btree->assert_flags, WT_ASSERT_COMMIT_TS_ALWAYS); - FLD_CLR(btree->assert_flags, WT_ASSERT_COMMIT_TS_NEVER); + FLD_CLR(btree->assert_flags, + WT_ASSERT_COMMIT_TS_KEYS | WT_ASSERT_COMMIT_TS_NEVER); + } else if (WT_STRING_MATCH("key_consistent", cval.str, cval.len)) { + FLD_SET(btree->assert_flags, WT_ASSERT_COMMIT_TS_KEYS); + FLD_CLR(btree->assert_flags, + WT_ASSERT_COMMIT_TS_ALWAYS | WT_ASSERT_COMMIT_TS_NEVER); } else if (WT_STRING_MATCH("never", cval.str, cval.len)) { FLD_SET(btree->assert_flags, WT_ASSERT_COMMIT_TS_NEVER); - FLD_CLR(btree->assert_flags, WT_ASSERT_COMMIT_TS_ALWAYS); + FLD_CLR(btree->assert_flags, + WT_ASSERT_COMMIT_TS_ALWAYS | WT_ASSERT_COMMIT_TS_KEYS); } else FLD_CLR(btree->assert_flags, - WT_ASSERT_COMMIT_TS_ALWAYS | WT_ASSERT_COMMIT_TS_NEVER); + WT_ASSERT_COMMIT_TS_ALWAYS | + WT_ASSERT_COMMIT_TS_KEYS | WT_ASSERT_COMMIT_TS_NEVER); WT_RET(__wt_config_gets(session, cfg, "assert.read_timestamp", &cval)); if (WT_STRING_MATCH("always", cval.str, cval.len)) { FLD_SET(btree->assert_flags, WT_ASSERT_READ_TS_ALWAYS); @@ -810,7 +814,7 @@ __btree_page_sizes(WT_SESSION_IMPL *session) btree->maxmempage = (uint64_t)cval.val; if (!F_ISSET(conn, WT_CONN_CACHE_POOL) && (cache_size = conn->cache_size) > 0) - btree->maxmempage = WT_MIN(btree->maxmempage, + btree->maxmempage = (uint64_t)WT_MIN(btree->maxmempage, (conn->cache->eviction_dirty_trigger * cache_size) / 1000); /* Enforce a lower bound of a single disk leaf page */ diff --git a/src/third_party/wiredtiger/src/btree/bt_huffman.c b/src/third_party/wiredtiger/src/btree/bt_huffman.c index 7e3b2582097..6d8ea284ab5 100644 --- a/src/third_party/wiredtiger/src/btree/bt_huffman.c +++ b/src/third_party/wiredtiger/src/btree/bt_huffman.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/btree/bt_io.c b/src/third_party/wiredtiger/src/btree/bt_io.c index e443fbdc94b..007513fd581 100644 --- a/src/third_party/wiredtiger/src/btree/bt_io.c +++ b/src/third_party/wiredtiger/src/btree/bt_io.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -173,7 +173,6 @@ __wt_bt_write(WT_SESSION_IMPL *session, WT_ITEM *buf, uint8_t *addr, size_t *addr_sizep, bool checkpoint, bool checkpoint_io, bool compressed) { - struct timespec start, stop; WT_BM *bm; WT_BTREE *btree; WT_DECL_ITEM(ctmp); @@ -183,6 +182,7 @@ __wt_bt_write(WT_SESSION_IMPL *session, WT_ITEM *buf, WT_KEYED_ENCRYPTOR *kencryptor; WT_PAGE_HEADER *dsk; size_t dst_len, len, result_len, size, src_len; + uint64_t time_start, time_stop; uint8_t *dst, *src; int compression_failed; /* Extension API, so not a bool. */ bool data_checksum, encrypted, timer; @@ -190,6 +190,7 @@ __wt_bt_write(WT_SESSION_IMPL *session, WT_ITEM *buf, btree = S2BT(session); bm = btree->bm; encrypted = false; + time_start = time_stop = 0; /* Checkpoint calls are different than standard calls. */ WT_ASSERT(session, @@ -353,6 +354,7 @@ __wt_bt_write(WT_SESSION_IMPL *session, WT_ITEM *buf, * configured. */ data_checksum = true; /* -Werror=maybe-uninitialized */ + WT_NOT_READ(data_checksum); switch (btree->checksum) { case CKSUM_ON: data_checksum = true; @@ -366,7 +368,7 @@ __wt_bt_write(WT_SESSION_IMPL *session, WT_ITEM *buf, } timer = !F_ISSET(session, WT_SESSION_INTERNAL); if (timer) - __wt_epoch(session, &start); + time_start = __wt_clock(session); /* Call the block manager to write the block. */ WT_ERR(checkpoint ? @@ -376,10 +378,10 @@ __wt_bt_write(WT_SESSION_IMPL *session, WT_ITEM *buf, /* Update some statistics now that the write is done */ if (timer) { - __wt_epoch(session, &stop); + time_stop = __wt_clock(session); WT_STAT_CONN_INCR(session, cache_write_app_count); WT_STAT_CONN_INCRV(session, cache_write_app_time, - WT_TIMEDIFF_US(stop, start)); + WT_CLOCKDIFF_US(time_stop, time_start)); } WT_STAT_CONN_INCR(session, cache_write); diff --git a/src/third_party/wiredtiger/src/btree/bt_misc.c b/src/third_party/wiredtiger/src/btree/bt_misc.c index 04b607082d1..b1d5a8b0782 100644 --- a/src/third_party/wiredtiger/src/btree/bt_misc.c +++ b/src/third_party/wiredtiger/src/btree/bt_misc.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/btree/bt_ovfl.c b/src/third_party/wiredtiger/src/btree/bt_ovfl.c index d65073a398f..ce0ee706923 100644 --- a/src/third_party/wiredtiger/src/btree/bt_ovfl.c +++ b/src/third_party/wiredtiger/src/btree/bt_ovfl.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/btree/bt_page.c b/src/third_party/wiredtiger/src/btree/bt_page.c index e2d4fa01fa7..8219ee0d7ed 100644 --- a/src/third_party/wiredtiger/src/btree/bt_page.c +++ b/src/third_party/wiredtiger/src/btree/bt_page.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/btree/bt_random.c b/src/third_party/wiredtiger/src/btree/bt_random.c index 268b040bd6e..03b5039b00b 100644 --- a/src/third_party/wiredtiger/src/btree/bt_random.c +++ b/src/third_party/wiredtiger/src/btree/bt_random.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -232,6 +232,7 @@ restart: /* descent = pindex->index[__wt_random(&session->rnd) % entries]; if (descent->state == WT_REF_DISK || + descent->state == WT_REF_LIMBO || descent->state == WT_REF_LOOKASIDE || descent->state == WT_REF_MEM) break; @@ -240,6 +241,7 @@ restart: /* for (i = 0; i < entries; ++i) { descent = pindex->index[i]; if (descent->state == WT_REF_DISK || + descent->state == WT_REF_LIMBO || descent->state == WT_REF_LOOKASIDE || descent->state == WT_REF_MEM) break; diff --git a/src/third_party/wiredtiger/src/btree/bt_read.c b/src/third_party/wiredtiger/src/btree/bt_read.c index 19ff15fb21e..afaf6c82aa5 100644 --- a/src/third_party/wiredtiger/src/btree/bt_read.c +++ b/src/third_party/wiredtiger/src/btree/bt_read.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -8,9 +8,6 @@ #include "wt_internal.h" -static void __btree_verbose_lookaside_read( - WT_SESSION_IMPL *, uint32_t, uint64_t); - /* * __col_instantiate -- * Update a column-store page entry based on a lookaside table update list. @@ -70,6 +67,142 @@ __row_instantiate(WT_SESSION_IMPL *session, } /* + * __las_page_skip_locked -- + * Check if we can skip reading a locked page with lookaside entries. + */ +static inline bool +__las_page_skip_locked(WT_SESSION_IMPL *session, WT_REF *ref) +{ + WT_TXN *txn; + + txn = &session->txn; + + /* + * Skip lookaside pages if reading without a timestamp and all the + * updates in lookaside are in the past. + * + * Lookaside eviction preferentially chooses the newest updates when + * creating page images with no stable timestamp. If a stable timestamp + * has been set, we have to visit the page because eviction chooses old + * version of records in that case. + * + * One case where we may need to visit the page is if lookaside eviction + * is active in tree 2 when a checkpoint has started and is working its + * way through tree 1. In that case, lookaside may have created a page + * image with updates in the future of the checkpoint. + * + * We also need to instantiate a lookaside page if this is an update + * operation in progress. + */ + if (ref->page_las->invalid) + return (false); + + if (F_ISSET(txn, WT_TXN_UPDATE)) + return (false); + + if (!F_ISSET(txn, WT_TXN_HAS_SNAPSHOT)) + return (false); + + if (WT_TXNID_LE(txn->snap_min, ref->page_las->las_max_txn)) + return (false); + + if (!F_ISSET(txn, WT_TXN_HAS_TS_READ) && ref->page_las->las_skew_newest) + return (true); + +#ifdef HAVE_TIMESTAMPS + /* + * Skip lookaside pages if reading as of a timestamp, we evicted new + * versions of data and all the updates are in the past. + */ + if (F_ISSET(&session->txn, WT_TXN_HAS_TS_READ) && + ref->page_las->las_skew_newest && + __wt_timestamp_cmp( + &ref->page_las->onpage_timestamp, &session->txn.read_timestamp) < 0) + return (true); + + /* + * Skip lookaside pages if reading as of a timestamp, we evicted old + * versions of data and all the updates are in the future. + */ + if (F_ISSET(&session->txn, WT_TXN_HAS_TS_READ) && + !ref->page_las->las_skew_newest && + __wt_timestamp_cmp( + &ref->page_las->min_timestamp, &session->txn.read_timestamp) > 0) + return (true); +#endif + + return (false); +} + +/* + * __las_page_skip -- + * Check if we can skip reading a page with lookaside entries. + */ +static inline bool +__las_page_skip(WT_SESSION_IMPL *session, WT_REF *ref) +{ + uint32_t previous_state; + bool skip; + + if ((previous_state = ref->state) != WT_REF_LIMBO && + previous_state != WT_REF_LOOKASIDE) + return (false); + + if (!__wt_atomic_casv32(&ref->state, previous_state, WT_REF_LOCKED)) + return (false); + + skip = __las_page_skip_locked(session, ref); + + /* Restore the state and push the change. */ + ref->state = previous_state; + WT_FULL_BARRIER(); + + return (skip); +} + +/* + * __las_page_instantiate_verbose -- + * Create a verbose message to display at most once per checkpoint when + * performing a lookaside table read. + */ +static void +__las_page_instantiate_verbose(WT_SESSION_IMPL *session, uint64_t las_pageid) +{ + WT_CACHE *cache; + uint64_t ckpt_gen_current, ckpt_gen_last; + + if (!WT_VERBOSE_ISSET(session, + WT_VERB_LOOKASIDE | WT_VERB_LOOKASIDE_ACTIVITY)) + return; + + cache = S2C(session)->cache; + ckpt_gen_current = __wt_gen(session, WT_GEN_CHECKPOINT); + ckpt_gen_last = cache->las_verb_gen_read; + + /* + * This message is throttled to one per checkpoint. To do this we + * track the generation of the last checkpoint for which the message + * was printed and check against the current checkpoint generation. + */ + if (WT_VERBOSE_ISSET(session, WT_VERB_LOOKASIDE) || + ckpt_gen_current > ckpt_gen_last) { + /* + * Attempt to atomically replace the last checkpoint generation + * for which this message was printed. If the atomic swap fails + * we have raced and the winning thread will print the message. + */ + if (__wt_atomic_casv64(&cache->las_verb_gen_read, + ckpt_gen_last, ckpt_gen_current)) { + __wt_verbose(session, + WT_VERB_LOOKASIDE | WT_VERB_LOOKASIDE_ACTIVITY, + "Read from lookaside file triggered for " + "file ID %" PRIu32 ", page ID %" PRIu64, + S2BT(session)->id, las_pageid); + } + } +} + +/* * __las_page_instantiate -- * Instantiate lookaside update records in a recently read page. */ @@ -97,6 +230,10 @@ __las_page_instantiate(WT_SESSION_IMPL *session, WT_REF *ref, uint32_t btree_id) session_flags = 0; /* [-Werror=maybe-uninitialized] */ WT_CLEAR(las_key); + __las_page_instantiate_verbose(session, ref->page_las->las_pageid); + WT_STAT_CONN_INCR(session, cache_read_lookaside); + WT_STAT_DATA_INCR(session, cache_read_lookaside); + __wt_btcur_init(session, &cbt); __wt_btcur_open(&cbt); @@ -314,18 +451,19 @@ __evict_force_check(WT_SESSION_IMPL *session, WT_REF *ref) static int __page_read(WT_SESSION_IMPL *session, WT_REF *ref, uint32_t flags) { - struct timespec start, stop; WT_BTREE *btree; WT_DECL_RET; WT_ITEM tmp; WT_PAGE *page; size_t addr_size; - uint32_t page_flags, new_state, previous_state; + uint64_t time_start, time_stop; + uint32_t page_flags, final_state, new_state, previous_state; const uint8_t *addr; bool timer; btree = S2BT(session); page = NULL; + time_start = time_stop = 0; /* * Don't pass an allocated buffer to the underlying block read function, @@ -348,6 +486,7 @@ __page_read(WT_SESSION_IMPL *session, WT_REF *ref, uint32_t flags) new_state = WT_REF_READING; break; case WT_REF_DELETED: + case WT_REF_LIMBO: case WT_REF_LOOKASIDE: new_state = WT_REF_LOCKED; break; @@ -357,6 +496,20 @@ __page_read(WT_SESSION_IMPL *session, WT_REF *ref, uint32_t flags) if (!__wt_atomic_casv32(&ref->state, previous_state, new_state)) return (0); + final_state = WT_REF_MEM; + + /* + * If we already have the page image, just instantiate the history. + * + * We need exclusive access because other threads could be reading the + * page without history and we can't change the state underneath them. + */ + if (previous_state == WT_REF_LIMBO) { + if (__wt_hazard_check(session, ref) != NULL) + goto err; + goto skip_read; + } + /* * Get the address: if there is no address, the page was deleted or had * only lookaside entries, and a subsequent search or insert is forcing @@ -379,66 +532,89 @@ __page_read(WT_SESSION_IMPL *session, WT_REF *ref, uint32_t flags) */ timer = !F_ISSET(session, WT_SESSION_INTERNAL); if (timer) - __wt_epoch(session, &start); + time_start = __wt_clock(session); WT_ERR(__wt_bt_read(session, &tmp, addr, addr_size)); if (timer) { - __wt_epoch(session, &stop); + time_stop = __wt_clock(session); WT_STAT_CONN_INCR(session, cache_read_app_count); WT_STAT_CONN_INCRV(session, cache_read_app_time, - WT_TIMEDIFF_US(stop, start)); + WT_CLOCKDIFF_US(time_stop, time_start)); } /* * Build the in-memory version of the page. Clear our local reference to * the allocated copy of the disk image on return, the in-memory object * steals it. + * + * If a page is read with eviction disabled, we don't count evicting it + * as progress. Since disabling eviction allows pages to be read even + * when the cache is full, we want to avoid workloads repeatedly reading + * a page with eviction disabled (e.g., a metadata page), then evicting + * that page and deciding that is a sign that eviction is unstuck. */ page_flags = WT_DATA_IN_ITEM(&tmp) ? WT_PAGE_DISK_ALLOC : WT_PAGE_DISK_MAPPED; if (LF_ISSET(WT_READ_IGNORE_CACHE_SIZE)) - FLD_SET(page_flags, WT_PAGE_READ_NO_EVICT); + FLD_SET(page_flags, WT_PAGE_EVICT_NO_PROGRESS); WT_ERR(__wt_page_inmem(session, ref, tmp.data, page_flags, &page)); tmp.mem = NULL; -skip_read: + /* + * The WT_REF lookaside state should match the page-header state of + * any page we read. + */ + WT_ASSERT(session, + (previous_state != WT_REF_LIMBO && + previous_state != WT_REF_LOOKASIDE) || + ref->page->dsk == NULL || + F_ISSET(ref->page->dsk, WT_PAGE_LAS_UPDATE)); + /* * If reading for a checkpoint, there's no additional work to do, the * page on disk is correct as written. */ - if (session->dhandle->checkpoint != NULL) + if (session->dhandle->checkpoint != NULL) { + WT_ASSERT(session, previous_state == WT_REF_DISK); goto done; + } - /* If the page was deleted, instantiate that information. */ - if (previous_state == WT_REF_DELETED) +skip_read: + switch (previous_state) { + case WT_REF_DELETED: + /* If the page was deleted, instantiate that information. */ WT_ERR(__wt_delete_page_instantiate(session, ref)); + break; + case WT_REF_LOOKASIDE: + if (__las_page_skip_locked(session, ref)) { + WT_STAT_CONN_INCR( + session, cache_read_lookaside_skipped); + ref->page_las->eviction_to_lookaside = true; + final_state = WT_REF_LIMBO; + break; + } + /* FALLTHROUGH */ + case WT_REF_LIMBO: + /* Instantiate updates from the database's lookaside table. */ + if (previous_state == WT_REF_LIMBO) + WT_STAT_CONN_INCR(session, cache_read_lookaside_delay); - /* - * Instantiate updates from the database's lookaside table. The page - * flag was set when the page was written, potentially a long time ago. - * We only care if the lookaside table is currently active, check that - * before doing any work. - */ - if (previous_state == WT_REF_LOOKASIDE) { - WT_ASSERT(session, (ref->page->dsk == NULL || - F_ISSET(ref->page->dsk, WT_PAGE_LAS_UPDATE))); - - __btree_verbose_lookaside_read( - session, btree->id, ref->page_las->las_pageid); - WT_STAT_CONN_INCR(session, cache_read_lookaside); - WT_STAT_DATA_INCR(session, cache_read_lookaside); WT_ERR(__las_page_instantiate(session, ref, btree->id)); /* * The page is instantiated so we no longer need the lookaside - * entries. Note that we are discarding updates so the page - * must be marked available even if these operations fail. + * entries. Note we are discarding updates so the page must be + * marked available even if these operations fail. + * + * Don't free WT_REF.page_las, there may be concurrent readers. */ WT_TRET(__wt_las_remove_block( session, NULL, btree->id, ref->page_las->las_pageid)); - __wt_free(session, ref->page_las); + + ref->page_las->eviction_to_lookaside = false; + break; } -done: WT_PUBLISH(ref->state, WT_REF_MEM); +done: WT_PUBLISH(ref->state, final_state); return (ret); err: /* @@ -446,7 +622,7 @@ err: /* * it discarded the page, but not the disk image. Discard the page * and separately discard the disk image in all cases. */ - if (ref->page != NULL) + if (ref->page != NULL && previous_state != WT_REF_LIMBO) __wt_ref_out(session, ref); WT_PUBLISH(ref->state, previous_state); @@ -456,74 +632,6 @@ err: /* } /* - * __las_page_skip -- - * Check if we can skip reading a page with lookaside entries. - */ -static inline bool -__las_page_skip(WT_SESSION_IMPL *session, WT_REF *ref) -{ - WT_TXN *txn; - bool skip; - - txn = &session->txn; - skip = false; - - if (!__wt_atomic_casv32(&ref->state, WT_REF_LOOKASIDE, WT_REF_LOCKED)) - return (false); - - /* - * Skip lookaside pages if reading without a timestamp and all the - * updates in lookaside are in the past. - * - * If we skip a lookaside page, the tree cannot be left clean: - * lookaside entries must be resolved before the tree can be discarded. - * - * Lookaside eviction preferentially chooses the newest updates when - * creating page image with no stable timestamp. If a stable timestamp - * has been set, we have to visit the page because eviction chooses old - * version of records in that case. - * - * One case where we may need to visit the page is if lookaside - * eviction is active in tree 2 when a checkpoint has started and is - * working its way through tree 1. In that case, lookaside may have - * created a page image with updates in the future of the checkpoint. - */ - if (!F_ISSET(txn, WT_TXN_HAS_SNAPSHOT)) - goto done; - - if (WT_TXNID_LE(txn->snap_min, ref->page_las->las_max_txn)) - goto done; - - if (!F_ISSET(txn, WT_TXN_HAS_TS_READ) && - ref->page_las->las_skew_newest) { - skip = true; - goto done; - } - -#ifdef HAVE_TIMESTAMPS - /* - * Skip lookaside pages if reading as of a timestamp and all the - * updates are in the future. - */ - WT_ASSERT(session, - !F_ISSET(&session->txn, WT_TXN_HAS_TS_READ) || - __wt_timestamp_cmp(&ref->page_las->onpage_timestamp, - &session->txn.read_timestamp) <= 0); - - if (F_ISSET(&session->txn, WT_TXN_HAS_TS_READ) && - !ref->page_las->las_skew_newest && - __wt_timestamp_cmp( - &ref->page_las->min_timestamp, &session->txn.read_timestamp) > 0) { - skip = true; - goto done; - } -#endif - -done: WT_PUBLISH(ref->state, WT_REF_LOOKASIDE); - return (skip); -} - -/* * __wt_page_in_func -- * Acquire a hazard pointer to a page; if the page is not in-memory, * read it from the disk and build an in-memory version. @@ -539,6 +647,7 @@ __wt_page_in_func(WT_SESSION_IMPL *session, WT_REF *ref, uint32_t flags WT_DECL_RET; WT_PAGE *page; uint64_t sleep_cnt, wait_cnt; + uint32_t current_state; int force_attempts; bool busy, cache_work, did_read, stalled, wont_need; @@ -558,7 +667,7 @@ __wt_page_in_func(WT_SESSION_IMPL *session, WT_REF *ref, uint32_t flags for (did_read = wont_need = stalled = false, force_attempts = 0, sleep_cnt = wait_cnt = 0;;) { - switch (ref->state) { + switch (current_state = ref->state) { case WT_REF_DELETED: if (LF_ISSET(WT_READ_NO_EMPTY) && __wt_delete_page_skip(session, ref, false)) @@ -568,6 +677,12 @@ __wt_page_in_func(WT_SESSION_IMPL *session, WT_REF *ref, uint32_t flags if (LF_ISSET(WT_READ_CACHE)) { if (!LF_ISSET(WT_READ_LOOKASIDE)) return (WT_NOTFOUND); + /* + * If we skip a lookaside page, the tree + * cannot be left clean: lookaside entries + * must be resolved before the tree can be + * discarded. + */ if (__las_page_skip(session, ref)) { __wt_tree_modify_set(session); return (WT_NOTFOUND); @@ -627,6 +742,7 @@ read: /* break; case WT_REF_SPLIT: return (WT_RESTART); + case WT_REF_LIMBO: case WT_REF_MEM: /* * The page is in memory. @@ -652,6 +768,22 @@ read: /* WT_STAT_CONN_INCR(session, page_busy_blocked); break; } + /* + * If we are a limbo page check whether we need to + * instantiate the history. By having a hazard pointer + * we can use the locked version. + */ + if (current_state == WT_REF_LIMBO && + ((!LF_ISSET(WT_READ_CACHE) || + LF_ISSET(WT_READ_LOOKASIDE)) && + !__las_page_skip_locked(session, ref))) { + WT_RET(__wt_hazard_clear(session, ref)); + goto read; + } + if (current_state == WT_REF_LIMBO && + LF_ISSET(WT_READ_CACHE) && + LF_ISSET(WT_READ_LOOKASIDE)) + __wt_tree_modify_set(session); /* * Check if the page requires forced eviction. @@ -679,6 +811,7 @@ read: /* /* If forced eviction fails, stall. */ if (ret == EBUSY) { ret = 0; + WT_NOT_READ(ret); WT_STAT_CONN_INCR(session, page_forcible_evict_blocked); stalled = true; @@ -765,52 +898,3 @@ skip_evict: /* WT_STAT_CONN_INCRV(session, page_sleep, sleep_cnt); } } - -/* - * __btree_verbose_lookaside_read -- - * Create a verbose message to display at most once per checkpoint when - * performing a lookaside table read. - */ -static void -__btree_verbose_lookaside_read( - WT_SESSION_IMPL *session, uint32_t las_id, uint64_t las_pageid) -{ -#ifdef HAVE_VERBOSE - WT_CACHE *cache; - uint64_t ckpt_gen_current, ckpt_gen_last; - - if (!WT_VERBOSE_ISSET(session, - WT_VERB_LOOKASIDE | WT_VERB_LOOKASIDE_ACTIVITY)) - return; - - cache = S2C(session)->cache; - ckpt_gen_current = __wt_gen(session, WT_GEN_CHECKPOINT); - ckpt_gen_last = cache->las_verb_gen_read; - - /* - * This message is throttled to one per checkpoint. To do this we - * track the generation of the last checkpoint for which the message - * was printed and check against the current checkpoint generation. - */ - if (WT_VERBOSE_ISSET(session, WT_VERB_LOOKASIDE) || - ckpt_gen_current > ckpt_gen_last) { - /* - * Attempt to atomically replace the last checkpoint generation - * for which this message was printed. If the atomic swap fails - * we have raced and the winning thread will print the message. - */ - if (__wt_atomic_casv64(&cache->las_verb_gen_read, - ckpt_gen_last, ckpt_gen_current)) { - __wt_verbose(session, - WT_VERB_LOOKASIDE | WT_VERB_LOOKASIDE_ACTIVITY, - "Read from lookaside file triggered for " - "file ID %" PRIu32 ", page ID %" PRIu64, - las_id, las_pageid); - } - } -#else - WT_UNUSED(session); - WT_UNUSED(las_id); - WT_UNUSED(las_pageid); -#endif -} diff --git a/src/third_party/wiredtiger/src/btree/bt_rebalance.c b/src/third_party/wiredtiger/src/btree/bt_rebalance.c index c6d9253b2d3..ed23d8de215 100644 --- a/src/third_party/wiredtiger/src/btree/bt_rebalance.c +++ b/src/third_party/wiredtiger/src/btree/bt_rebalance.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/btree/bt_ret.c b/src/third_party/wiredtiger/src/btree/bt_ret.c index b24a4e1db45..7d0da631e2b 100644 --- a/src/third_party/wiredtiger/src/btree/bt_ret.c +++ b/src/third_party/wiredtiger/src/btree/bt_ret.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -49,7 +49,7 @@ __key_return(WT_SESSION_IMPL *session, WT_CURSOR_BTREE *cbt) * itself because our caller might do another search in * this table using the key we return, and we'd corrupt * the search key during any subsequent search that used - * the temporary buffer. + * the temporary buffer). */ tmp = cbt->row_key; cbt->row_key = cbt->tmp; @@ -150,6 +150,7 @@ __wt_value_return_upd(WT_SESSION_IMPL *session, WT_UPDATE **listp, *list[WT_MODIFY_ARRAY_SIZE]; size_t allocated_bytes; u_int i; + bool skipped_birthmark; cursor = &cbt->iface; allocated_bytes = 0; @@ -166,21 +167,33 @@ __wt_value_return_upd(WT_SESSION_IMPL *session, cursor->value.size = upd->size; return (0); } - WT_ASSERT(session, upd->type == WT_UPDATE_MODIFIED); + WT_ASSERT(session, upd->type == WT_UPDATE_MODIFY); /* * Find a complete update that's visible to us, tracking modifications * that are visible to us. */ - for (i = 0, listp = list; upd != NULL; upd = upd->next) { - if (upd->txnid == WT_TXN_ABORTED || - (!ignore_visibility && !__wt_txn_upd_visible(session, upd))) + for (i = 0, listp = list, skipped_birthmark = false; + upd != NULL; + upd = upd->next) { + if (upd->txnid == WT_TXN_ABORTED) continue; + if (!ignore_visibility && !__wt_txn_upd_visible(session, upd)) { + if (upd->type == WT_UPDATE_BIRTHMARK) + skipped_birthmark = true; + continue; + } + + if (upd->type == WT_UPDATE_BIRTHMARK) { + upd = NULL; + break; + } + if (WT_UPDATE_DATA_VALUE(upd)) break; - if (upd->type == WT_UPDATE_MODIFIED) { + if (upd->type == WT_UPDATE_MODIFY) { /* * Update lists are expected to be short, but it's not * guaranteed. There's sufficient room on the stack to @@ -200,30 +213,45 @@ __wt_value_return_upd(WT_SESSION_IMPL *session, } /* - * If we hit the end of the chain, roll forward from the update item we - * found, otherwise, from the original page's value. + * If there's no visible update and we skipped a birthmark, the base + * item is an empty item (in other words, birthmarks we can't read act + * as tombstones). + * If there's no visible update and we didn't skip a birthmark, the base + * item is the on-page item, which must be globally visible. + * If there's a visible update and it's a tombstone, the base item is an + * empty item. + * If there's a visible update and it's not a tombstone, the base item + * is the on-page item. */ if (upd == NULL) { - /* - * Callers of this function set the cursor slot to an impossible - * value to check we're not trying to return on-page values when - * the update list should have been sufficient (which happens, - * for example, if an update list was truncated, deleting some - * standard update required by a previous modify update). Assert - * the case. - */ - WT_ASSERT(session, cbt->slot != UINT32_MAX); + if (skipped_birthmark) + WT_ERR(__wt_buf_set(session, &cursor->value, "", 0)); + else { + /* + * Callers of this function set the cursor slot to an + * impossible value to check we don't try and return + * on-page values when the update list should have been + * sufficient (which happens, for example, if an update + * list was truncated, deleting some standard update + * required by a previous modify update). Assert the + * case. + */ + WT_ASSERT(session, cbt->slot != UINT32_MAX); - WT_ERR(__value_return(session, cbt)); - } else if (upd->type == WT_UPDATE_DELETED) + WT_ERR(__value_return(session, cbt)); + } + } else if (upd->type == WT_UPDATE_TOMBSTONE) WT_ERR(__wt_buf_set(session, &cursor->value, "", 0)); else WT_ERR(__wt_buf_set(session, &cursor->value, upd->data, upd->size)); + /* + * Once we have a base item, roll forward through any visible modify + * updates. + */ while (i > 0) - WT_ERR(__wt_modify_apply( - session, &cursor->value, listp[--i]->data)); + WT_ERR(__wt_modify_apply(session, cursor, listp[--i]->data)); err: if (allocated_bytes != 0) __wt_free(session, listp); diff --git a/src/third_party/wiredtiger/src/btree/bt_slvg.c b/src/third_party/wiredtiger/src/btree/bt_slvg.c index e2da77348f0..ebb01bf818a 100644 --- a/src/third_party/wiredtiger/src/btree/bt_slvg.c +++ b/src/third_party/wiredtiger/src/btree/bt_slvg.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -106,10 +106,12 @@ struct __wt_track { } col; } u; -#define WT_TRACK_CHECK_START 0x01 /* Row: initial key updated */ -#define WT_TRACK_CHECK_STOP 0x02 /* Row: last key updated */ -#define WT_TRACK_MERGE 0x04 /* Page requires merging */ -#define WT_TRACK_OVFL_REFD 0x08 /* Overflow page referenced */ +/* AUTOMATIC FLAG VALUE GENERATION START */ +#define WT_TRACK_CHECK_START 0x1u /* Row: initial key updated */ +#define WT_TRACK_CHECK_STOP 0x2u /* Row: last key updated */ +#define WT_TRACK_MERGE 0x4u /* Page requires merging */ +#define WT_TRACK_OVFL_REFD 0x8u /* Overflow page referenced */ +/* AUTOMATIC FLAG VALUE GENERATION STOP */ u_int flags; }; @@ -1893,8 +1895,6 @@ __slvg_row_build_leaf( uint32_t i, skip_start, skip_stop; int cmp; - WT_UNUSED(ss); /* !HAVE_VERBOSE */ - btree = S2BT(session); page = NULL; diff --git a/src/third_party/wiredtiger/src/btree/bt_split.c b/src/third_party/wiredtiger/src/btree/bt_split.c index bf7ea54adb0..36bbe48b407 100644 --- a/src/third_party/wiredtiger/src/btree/bt_split.c +++ b/src/third_party/wiredtiger/src/btree/bt_split.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -720,6 +720,7 @@ __split_parent(WT_SESSION_IMPL *session, WT_REF *ref, WT_REF **ref_new, /* Start making real changes to the tree, errors are fatal. */ complete = WT_ERR_PANIC; + WT_NOT_READ(complete); /* Encourage a race */ __page_split_timing_stress(session, @@ -757,16 +758,6 @@ __split_parent(WT_SESSION_IMPL *session, WT_REF *ref, WT_REF **ref_new, */ if (discard) { /* - * Page-delete information is only read when the WT_REF state is - * WT_REF_DELETED. The page-delete memory wasn't added to the - * parent's footprint, ignore it here. - */ - if (ref->page_del != NULL) { - __wt_free(session, ref->page_del->update_list); - __wt_free(session, ref->page_del); - } - - /* * Set the discarded WT_REF state to split, ensuring we don't * race with any discard of the WT_REF deleted fields. */ @@ -842,12 +833,18 @@ __split_parent(WT_SESSION_IMPL *session, WT_REF *ref, WT_REF **ref_new, } /* - * If this page was fast-truncated, any attached structure - * should have been freed before now. + * The page-delete and lookaside memory weren't added to the + * parent's footprint, ignore it here. */ - WT_ASSERT(session, next_ref->page_del == NULL); + if (next_ref->page_del != NULL) { + __wt_free(session, next_ref->page_del->update_list); + __wt_free(session, next_ref->page_del); + } + __wt_free(session, next_ref->page_las); + /* Free the backing block and address. */ WT_TRET(__wt_ref_block_free(session, next_ref)); + WT_TRET(__split_safe_free( session, split_gen, exclusive, next_ref, sizeof(WT_REF))); parent_decr += sizeof(WT_REF); @@ -1476,6 +1473,12 @@ __split_multi_inmem( WT_ERR(__wt_row_search( session, key, ref, &cbt, true, true)); + /* + * Birthmarks should only be applied to on-page values. + */ + WT_ASSERT(session, cbt.compare == 0 || + upd->type != WT_UPDATE_BIRTHMARK); + /* Apply the modification. */ WT_ERR(__wt_row_modify(session, &cbt, key, NULL, upd, WT_UPDATE_INVALID, true)); @@ -1567,7 +1570,7 @@ __split_multi_inmem_fail(WT_SESSION_IMPL *session, WT_PAGE *orig, WT_REF *ref) /* * __wt_multi_to_ref -- - * Move a multi-block list into an array of WT_REF structures. + * Move a multi-block entry into a WT_REF structure. */ int __wt_multi_to_ref(WT_SESSION_IMPL *session, @@ -2254,12 +2257,17 @@ __wt_split_rewrite(WT_SESSION_IMPL *session, WT_REF *ref, WT_MULTI *multi) * * Pages with unresolved changes are not marked clean during * reconciliation, do it now. + * + * Don't count this as eviction making progress, we did a one-for-one + * rewrite of a page in memory, typical in the case of cache pressure. */ __wt_page_modify_clear(session, page); - __wt_ref_out_int(session, ref, true); + F_SET_ATOMIC(page, WT_PAGE_EVICT_NO_PROGRESS); + __wt_ref_out(session, ref); /* Swap the new page into place. */ ref->page = new->page; + WT_PUBLISH(ref->state, WT_REF_MEM); __wt_free(session, new); diff --git a/src/third_party/wiredtiger/src/btree/bt_stat.c b/src/third_party/wiredtiger/src/btree/bt_stat.c index 9be476b87c6..c00b2bfda39 100644 --- a/src/third_party/wiredtiger/src/btree/bt_stat.c +++ b/src/third_party/wiredtiger/src/btree/bt_stat.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -179,20 +179,20 @@ __stat_page_col_var( */ WT_SKIP_FOREACH(ins, WT_COL_UPDATE(page, cip)) { switch (ins->upd->type) { - case WT_UPDATE_DELETED: - if (!orig_deleted) { - ++deleted_cnt; - --entry_cnt; - } - break; - case WT_UPDATE_MODIFIED: + case WT_UPDATE_MODIFY: case WT_UPDATE_STANDARD: if (orig_deleted) { --deleted_cnt; ++entry_cnt; } break; - case WT_UPDATE_RESERVED: + case WT_UPDATE_RESERVE: + break; + case WT_UPDATE_TOMBSTONE: + if (!orig_deleted) { + ++deleted_cnt; + --entry_cnt; + } break; } } @@ -201,14 +201,14 @@ __stat_page_col_var( /* Walk any append list. */ WT_SKIP_FOREACH(ins, WT_COL_APPEND(page)) switch (ins->upd->type) { - case WT_UPDATE_DELETED: - ++deleted_cnt; - break; - case WT_UPDATE_MODIFIED: + case WT_UPDATE_MODIFY: case WT_UPDATE_STANDARD: ++entry_cnt; break; - case WT_UPDATE_RESERVED: + case WT_UPDATE_RESERVE: + break; + case WT_UPDATE_TOMBSTONE: + ++deleted_cnt; break; } @@ -277,8 +277,8 @@ __stat_page_row_leaf( * key on the page. */ WT_SKIP_FOREACH(ins, WT_ROW_INSERT_SMALLEST(page)) - if (ins->upd->type != WT_UPDATE_DELETED && - ins->upd->type != WT_UPDATE_RESERVED) + if (ins->upd->type != WT_UPDATE_RESERVE && + ins->upd->type != WT_UPDATE_TOMBSTONE) ++entry_cnt; /* @@ -288,8 +288,8 @@ __stat_page_row_leaf( WT_ROW_FOREACH(page, rip, i) { upd = WT_ROW_UPDATE(page, rip); if (upd == NULL || - (upd->type != WT_UPDATE_DELETED && - upd->type != WT_UPDATE_RESERVED)) + (upd->type != WT_UPDATE_RESERVE && + upd->type != WT_UPDATE_TOMBSTONE)) ++entry_cnt; if (upd == NULL && (cell = __wt_row_leaf_value_cell(page, rip, NULL)) != NULL && @@ -298,8 +298,8 @@ __stat_page_row_leaf( /* Walk K/V pairs inserted after the on-page K/V pair. */ WT_SKIP_FOREACH(ins, WT_ROW_INSERT(page, rip)) - if (ins->upd->type != WT_UPDATE_DELETED && - ins->upd->type != WT_UPDATE_RESERVED) + if (ins->upd->type != WT_UPDATE_RESERVE && + ins->upd->type != WT_UPDATE_TOMBSTONE) ++entry_cnt; } diff --git a/src/third_party/wiredtiger/src/btree/bt_sync.c b/src/third_party/wiredtiger/src/btree/bt_sync.c index 2338d5be8ed..8600c7d6555 100644 --- a/src/third_party/wiredtiger/src/btree/bt_sync.c +++ b/src/third_party/wiredtiger/src/btree/bt_sync.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -114,7 +114,6 @@ __sync_dup_walk( static int __sync_file(WT_SESSION_IMPL *session, WT_CACHE_OP syncop) { - struct timespec end, start; WT_BTREE *btree; WT_CONNECTION_IMPL *conn; WT_DECL_RET; @@ -122,7 +121,7 @@ __sync_file(WT_SESSION_IMPL *session, WT_CACHE_OP syncop) WT_REF *prev, *walk; WT_TXN *txn; uint64_t internal_bytes, internal_pages, leaf_bytes, leaf_pages; - uint64_t oldest_id, saved_pinned_id; + uint64_t oldest_id, saved_pinned_id, time_start, time_stop; uint32_t flags; bool timer, tried_eviction; @@ -131,6 +130,7 @@ __sync_file(WT_SESSION_IMPL *session, WT_CACHE_OP syncop) prev = walk = NULL; txn = &session->txn; tried_eviction = false; + time_start = time_stop = 0; flags = WT_READ_CACHE | WT_READ_NO_GEN; internal_bytes = leaf_bytes = 0; @@ -138,7 +138,7 @@ __sync_file(WT_SESSION_IMPL *session, WT_CACHE_OP syncop) saved_pinned_id = WT_SESSION_TXN_STATE(session)->pinned_id; timer = WT_VERBOSE_ISSET(session, WT_VERB_CHECKPOINT); if (timer) - __wt_epoch(session, &start); + time_start = __wt_clock(session); switch (syncop) { case WT_SYNC_WRITE_LEAVES: @@ -330,7 +330,7 @@ __sync_file(WT_SESSION_IMPL *session, WT_CACHE_OP syncop) } if (timer) { - __wt_epoch(session, &end); + time_stop = __wt_clock(session); __wt_verbose(session, WT_VERB_CHECKPOINT, "__sync_file WT_SYNC_%s wrote: %" PRIu64 " leaf pages (%" PRIu64 "B), %" PRIu64 @@ -338,7 +338,7 @@ __sync_file(WT_SESSION_IMPL *session, WT_CACHE_OP syncop) syncop == WT_SYNC_WRITE_LEAVES ? "WRITE_LEAVES" : "CHECKPOINT", leaf_pages, leaf_bytes, internal_pages, internal_bytes, - WT_TIMEDIFF_MS(end, start)); + WT_CLOCKDIFF_MS(time_stop, time_start)); } err: /* On error, clear any left-over tree walk. */ diff --git a/src/third_party/wiredtiger/src/btree/bt_upgrade.c b/src/third_party/wiredtiger/src/btree/bt_upgrade.c index a7fe3283218..6c50a294893 100644 --- a/src/third_party/wiredtiger/src/btree/bt_upgrade.c +++ b/src/third_party/wiredtiger/src/btree/bt_upgrade.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/btree/bt_vrfy.c b/src/third_party/wiredtiger/src/btree/bt_vrfy.c index c3b5d926a8f..c10a9256769 100644 --- a/src/third_party/wiredtiger/src/btree/bt_vrfy.c +++ b/src/third_party/wiredtiger/src/btree/bt_vrfy.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/btree/bt_vrfy_dsk.c b/src/third_party/wiredtiger/src/btree/bt_vrfy_dsk.c index de80f207621..aae50ed636c 100644 --- a/src/third_party/wiredtiger/src/btree/bt_vrfy_dsk.c +++ b/src/third_party/wiredtiger/src/btree/bt_vrfy_dsk.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/btree/bt_walk.c b/src/third_party/wiredtiger/src/btree/bt_walk.c index b68c6b9c5c6..22921d7d378 100644 --- a/src/third_party/wiredtiger/src/btree/bt_walk.c +++ b/src/third_party/wiredtiger/src/btree/bt_walk.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -293,7 +293,7 @@ __tree_walk_internal(WT_SESSION_IMPL *session, WT_DECL_RET; WT_PAGE_INDEX *pindex; WT_REF *couple, *couple_orig, *ref; - uint32_t slot; + uint32_t current_state, slot; bool empty_internal, initial_descent, prev, skip; btree = S2BT(session); @@ -460,7 +460,8 @@ restart: /* * If we see any child states other than deleted, the * page isn't empty. */ - if (ref->state != WT_REF_DELETED && + current_state = ref->state; + if (current_state != WT_REF_DELETED && !LF_ISSET(WT_READ_TRUNCATE)) empty_internal = false; @@ -470,11 +471,12 @@ restart: /* * fast-path some common cases. */ if (LF_ISSET(WT_READ_NO_WAIT) && - ref->state != WT_REF_MEM) + current_state != WT_REF_MEM && + current_state != WT_REF_LIMBO) break; /* Skip lookaside pages if not requested. */ - if (ref->state == WT_REF_LOOKASIDE && + if (current_state == WT_REF_LOOKASIDE && !LF_ISSET(WT_READ_LOOKASIDE)) break; } else if (LF_ISSET(WT_READ_TRUNCATE)) { @@ -482,7 +484,7 @@ restart: /* * Avoid pulling a deleted page back in to try * to delete it again. */ - if (ref->state == WT_REF_DELETED && + if (current_state == WT_REF_DELETED && __wt_delete_page_skip(session, ref, false)) break; /* @@ -502,7 +504,7 @@ restart: /* /* * Try to skip deleted pages visible to us. */ - if (ref->state == WT_REF_DELETED && + if (current_state == WT_REF_DELETED && __wt_delete_page_skip(session, ref, false)) break; } @@ -516,6 +518,7 @@ restart: /* */ if (ret == WT_NOTFOUND) { ret = 0; + WT_NOT_READ(ret); break; } @@ -662,8 +665,8 @@ __wt_tree_walk_count(WT_SESSION_IMPL *session, int __wt_tree_walk_custom_skip( WT_SESSION_IMPL *session, WT_REF **refp, - int (*skip_func)(WT_SESSION_IMPL *, WT_REF *, void *, bool *), - void *func_cookie, uint32_t flags) + int (*skip_func)(WT_SESSION_IMPL *, WT_REF *, void *, bool *), + void *func_cookie, uint32_t flags) { return (__tree_walk_internal( session, refp, NULL, skip_func, func_cookie, flags)); diff --git a/src/third_party/wiredtiger/src/btree/col_modify.c b/src/third_party/wiredtiger/src/btree/col_modify.c index a51428dc62b..7270c49a9f5 100644 --- a/src/third_party/wiredtiger/src/btree/col_modify.c +++ b/src/third_party/wiredtiger/src/btree/col_modify.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -39,13 +39,13 @@ __wt_col_modify(WT_SESSION_IMPL *session, WT_CURSOR_BTREE *cbt, append = logged = false; if (upd_arg == NULL) { - if (modify_type == WT_UPDATE_DELETED || - modify_type == WT_UPDATE_RESERVED) { + if (modify_type == WT_UPDATE_RESERVE || + modify_type == WT_UPDATE_TOMBSTONE) { /* * Fixed-size column-store doesn't have on-page deleted * values, it's a nul byte. */ - if (modify_type == WT_UPDATE_DELETED && + if (modify_type == WT_UPDATE_TOMBSTONE && btree->type == BTREE_COL_FIX) { modify_type = WT_UPDATE_STANDARD; value = &col_fix_remove; @@ -67,6 +67,9 @@ __wt_col_modify(WT_SESSION_IMPL *session, WT_CURSOR_BTREE *cbt, } } + /* We're going to modify the page, we should have loaded history. */ + WT_ASSERT(session, cbt->ref->state != WT_REF_LIMBO); + /* If we don't yet have a modify structure, we'll need one. */ WT_RET(__wt_page_modify_init(session, page)); mod = page->modify; @@ -200,7 +203,7 @@ __wt_col_modify(WT_SESSION_IMPL *session, WT_CURSOR_BTREE *cbt, } /* If the update was successful, add it to the in-memory log. */ - if (logged && modify_type != WT_UPDATE_RESERVED) + if (logged && modify_type != WT_UPDATE_RESERVE) WT_ERR(__wt_txn_log_op(session, cbt)); if (0) { diff --git a/src/third_party/wiredtiger/src/btree/col_srch.c b/src/third_party/wiredtiger/src/btree/col_srch.c index 10bc3894a0d..5c0e066647a 100644 --- a/src/third_party/wiredtiger/src/btree/col_srch.c +++ b/src/third_party/wiredtiger/src/btree/col_srch.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/btree/row_key.c b/src/third_party/wiredtiger/src/btree/row_key.c index a016568898f..ca8e2418857 100644 --- a/src/third_party/wiredtiger/src/btree/row_key.c +++ b/src/third_party/wiredtiger/src/btree/row_key.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/btree/row_modify.c b/src/third_party/wiredtiger/src/btree/row_modify.c index a2aaeb7673f..8b1e4d78f54 100644 --- a/src/third_party/wiredtiger/src/btree/row_modify.c +++ b/src/third_party/wiredtiger/src/btree/row_modify.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -61,6 +61,9 @@ __wt_row_modify(WT_SESSION_IMPL *session, WT_CURSOR_BTREE *cbt, upd = upd_arg; logged = false; + /* We're going to modify the page, we should have loaded history. */ + WT_ASSERT(session, cbt->ref->state != WT_REF_LIMBO); + /* If we don't yet have a modify structure, we'll need one. */ WT_RET(__wt_page_modify_init(session, page)); mod = page->modify; @@ -202,7 +205,7 @@ __wt_row_modify(WT_SESSION_IMPL *session, WT_CURSOR_BTREE *cbt, &ins, ins_size, skipdepth, exclusive)); } - if (logged && modify_type != WT_UPDATE_RESERVED) + if (logged && modify_type != WT_UPDATE_RESERVE) WT_ERR(__wt_txn_log_op(session, cbt)); if (0) { @@ -273,8 +276,9 @@ __wt_update_alloc(WT_SESSION_IMPL *session, const WT_ITEM *value, * Allocate the WT_UPDATE structure and room for the value, then copy * the value into place. */ - if (modify_type == WT_UPDATE_DELETED || - modify_type == WT_UPDATE_RESERVED) + if (modify_type == WT_UPDATE_BIRTHMARK || + modify_type == WT_UPDATE_RESERVE || + modify_type == WT_UPDATE_TOMBSTONE) WT_RET(__wt_calloc(session, 1, WT_UPDATE_SIZE, &upd)); else { WT_RET(__wt_calloc( @@ -321,7 +325,8 @@ __wt_update_obsolete_check( continue; if (!__wt_txn_upd_visible_all(session, upd)) first = NULL; - else if (first == NULL && WT_UPDATE_DATA_VALUE(upd)) + else if (first == NULL && (WT_UPDATE_DATA_VALUE(upd) || + upd->type == WT_UPDATE_BIRTHMARK)) first = upd; } @@ -355,24 +360,3 @@ __wt_update_obsolete_check( return (NULL); } - -/* - * __wt_update_obsolete_free -- - * Free an obsolete update list. - */ -void -__wt_update_obsolete_free( - WT_SESSION_IMPL *session, WT_PAGE *page, WT_UPDATE *upd) -{ - WT_UPDATE *next; - size_t size; - - /* Free a WT_UPDATE list. */ - for (size = 0; upd != NULL; upd = next) { - next = upd->next; - size += WT_UPDATE_MEMSIZE(upd); - __wt_free(session, upd); - } - if (size != 0) - __wt_cache_page_inmem_decr(session, page, size); -} diff --git a/src/third_party/wiredtiger/src/btree/row_srch.c b/src/third_party/wiredtiger/src/btree/row_srch.c index 68299a396ba..e75b307812c 100644 --- a/src/third_party/wiredtiger/src/btree/row_srch.c +++ b/src/third_party/wiredtiger/src/btree/row_srch.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/cache/cache_las.c b/src/third_party/wiredtiger/src/cache/cache_las.c index e0d90ad836d..5fa46cb7fb2 100644 --- a/src/third_party/wiredtiger/src/cache/cache_las.c +++ b/src/third_party/wiredtiger/src/cache/cache_las.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -333,7 +333,6 @@ __wt_las_cursor_close( static int __las_insert_block_verbose(WT_SESSION_IMPL *session, WT_MULTI *multi) { -#ifdef HAVE_VERBOSE WT_CACHE *cache; WT_CONNECTION_IMPL *conn; #ifdef HAVE_TIMESTAMPS @@ -342,7 +341,8 @@ __las_insert_block_verbose(WT_SESSION_IMPL *session, WT_MULTI *multi) char hex_timestamp[9]; /* Enough for disabled string */ #endif uint64_t ckpt_gen_current, ckpt_gen_last; - uint32_t btree_id, pct_dirty, pct_full; + uint32_t btree_id; + double pct_dirty, pct_full; btree_id = S2BT(session)->id; @@ -380,12 +380,12 @@ __las_insert_block_verbose(WT_SESSION_IMPL *session, WT_MULTI *multi) "file ID %" PRIu32 ", page ID %" PRIu64 ". " "Max txn ID %" PRIu64 ", min timestamp %s, skewed %s. " "Entries now in lookaside file: %" PRId64 ", " - "cache dirty: %" PRIu32 "%% , " - "cache use: %" PRIu32 "%%", + "cache dirty: %2.3f%% , " + "cache use: %2.3f%%", btree_id, multi->page_las.las_pageid, multi->page_las.las_max_txn, hex_timestamp, - multi->page_las.las_skew_newest? "newest" : "oldest", + multi->page_las.las_skew_newest ? "newest" : "oldest", WT_STAT_READ(conn->stats, cache_lookaside_entries), pct_dirty, pct_full); } @@ -393,10 +393,6 @@ __las_insert_block_verbose(WT_SESSION_IMPL *session, WT_MULTI *multi) /* Never skip updating the tracked generation */ if (WT_VERBOSE_ISSET(session, WT_VERB_LOOKASIDE)) cache->las_verb_gen_write = ckpt_gen_current; -#else - WT_UNUSED(session); - WT_UNUSED(multi); -#endif return (0); } @@ -433,7 +429,7 @@ __wt_las_insert_block(WT_SESSION_IMPL *session, WT_CURSOR *cursor, /* Wrap all the updates in a transaction. */ las_session = (WT_SESSION_IMPL *)cursor->session; WT_RET(__wt_txn_begin(las_session, NULL)); - las_session->txn.isolation = WT_TXN_ISO_READ_UNCOMMITTED; + las_session->txn.isolation = WT_ISO_READ_UNCOMMITTED; /* * Make sure there are no leftover entries (e.g., from a handle @@ -495,18 +491,16 @@ __wt_las_insert_block(WT_SESSION_IMPL *session, WT_CURSOR *cursor, continue; switch (upd->type) { - case WT_UPDATE_DELETED: - las_value.size = 0; - break; - case WT_UPDATE_MODIFIED: + case WT_UPDATE_MODIFY: case WT_UPDATE_STANDARD: las_value.data = upd->data; las_value.size = upd->size; break; - case WT_UPDATE_RESERVED: - WT_ASSERT(session, - upd->type != WT_UPDATE_RESERVED); - continue; + case WT_UPDATE_BIRTHMARK: + case WT_UPDATE_TOMBSTONE: + las_value.size = 0; + break; + WT_ILLEGAL_VALUE_ERR(session); } cursor->set_key(cursor, @@ -516,8 +510,25 @@ __wt_las_insert_block(WT_SESSION_IMPL *session, WT_CURSOR *cursor, las_timestamp.data = &upd->timestamp; las_timestamp.size = WT_TIMESTAMP_SIZE; #endif - cursor->set_value(cursor, - upd->txnid, &las_timestamp, upd->type, &las_value); + /* + * If saving a non-zero length value on the page, save a + * birthmark instead of duplicating it in the lookaside + * table. (We check the length because row-store doesn't + * write zero-length data items.) + */ + if (multi->page_las.las_skew_newest && + upd == list->onpage_upd && + upd->size > 0 && + (upd->type == WT_UPDATE_STANDARD || + upd->type == WT_UPDATE_MODIFY)) { + las_value.size = 0; + cursor->set_value(cursor, + upd->txnid, &las_timestamp, + WT_UPDATE_BIRTHMARK, &las_value); + } else + cursor->set_value(cursor, + upd->txnid, &las_timestamp, + upd->type, &las_value); /* * Using update looks a little strange because the keys @@ -533,7 +544,7 @@ __wt_las_insert_block(WT_SESSION_IMPL *session, WT_CURSOR *cursor, if (insert_cnt > 0) { WT_STAT_CONN_INCRV( session, cache_lookaside_entries, insert_cnt); - __wt_atomic_add64( + (void)__wt_atomic_add64( &S2C(session)->cache->las_entry_count, insert_cnt); WT_ERR(__las_insert_block_verbose(session, multi)); } @@ -639,7 +650,7 @@ __wt_las_remove_block(WT_SESSION_IMPL *session, */ if (local_cursor) { WT_ERR(__wt_txn_begin(las_session, NULL)); - las_session->txn.isolation = WT_TXN_ISO_READ_UNCOMMITTED; + las_session->txn.isolation = WT_ISO_READ_UNCOMMITTED; local_txn = true; } @@ -725,12 +736,10 @@ __las_sweep_init(WT_SESSION_IMPL *session) cache->las_sweep_dropmin = UINT32_MAX; cache->las_sweep_dropmax = 0; for (i = 0; i < cache->las_dropped_next; i++) { - cache->las_sweep_dropmin = WT_MIN( - cache->las_sweep_dropmin, - cache->las_dropped[i]); - cache->las_sweep_dropmax = WT_MAX( - cache->las_sweep_dropmax, - cache->las_dropped[i]); + cache->las_sweep_dropmin = + WT_MIN(cache->las_sweep_dropmin, cache->las_dropped[i]); + cache->las_sweep_dropmax = + WT_MAX(cache->las_sweep_dropmax, cache->las_dropped[i]); } /* Initialize the bitmap. */ diff --git a/src/third_party/wiredtiger/src/checksum/arm64/crc32-arm64.c b/src/third_party/wiredtiger/src/checksum/arm64/crc32-arm64.c index 4316ee3d14e..d0c989fd757 100644 --- a/src/third_party/wiredtiger/src/checksum/arm64/crc32-arm64.c +++ b/src/third_party/wiredtiger/src/checksum/arm64/crc32-arm64.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/src/checksum/software/checksum.c b/src/third_party/wiredtiger/src/checksum/software/checksum.c index a880d38894c..1228c9a0ce1 100644 --- a/src/third_party/wiredtiger/src/checksum/software/checksum.c +++ b/src/third_party/wiredtiger/src/checksum/software/checksum.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/src/checksum/x86/crc32-x86.c b/src/third_party/wiredtiger/src/checksum/x86/crc32-x86.c index 1c2c08fa1c1..8de987d4fc9 100644 --- a/src/third_party/wiredtiger/src/checksum/x86/crc32-x86.c +++ b/src/third_party/wiredtiger/src/checksum/x86/crc32-x86.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/src/config/config.c b/src/third_party/wiredtiger/src/config/config.c index d48c39de6b5..53f62daa5fb 100644 --- a/src/third_party/wiredtiger/src/config/config.c +++ b/src/third_party/wiredtiger/src/config/config.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/config/config_api.c b/src/third_party/wiredtiger/src/config/config_api.c index 3b37732f49b..a9993cb1e11 100644 --- a/src/third_party/wiredtiger/src/config/config_api.c +++ b/src/third_party/wiredtiger/src/config/config_api.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -97,7 +97,7 @@ wiredtiger_config_parser_open(WT_SESSION *wt_session, */ int wiredtiger_config_validate(WT_SESSION *wt_session, - WT_EVENT_HANDLER *handler, const char *name, const char *config) + WT_EVENT_HANDLER *event_handler, const char *name, const char *config) { const WT_CONFIG_ENTRY *ep, **epp; WT_CONNECTION_IMPL *conn, dummy_conn; @@ -108,9 +108,9 @@ wiredtiger_config_validate(WT_SESSION *wt_session, /* * It's a logic error to specify both a session and an event handler. */ - if (session != NULL && handler != NULL) + if (session != NULL && event_handler != NULL) WT_RET_MSG(session, EINVAL, - "wiredtiger_config_validate error handler ignored when " + "wiredtiger_config_validate event handler ignored when " "a session also specified"); /* @@ -118,13 +118,13 @@ wiredtiger_config_validate(WT_SESSION *wt_session, * a fake session/connection pair and configure the event handler. */ conn = NULL; - if (session == NULL && handler != NULL) { + if (session == NULL && event_handler != NULL) { WT_CLEAR(dummy_conn); conn = &dummy_conn; session = conn->default_session = &conn->dummy_session; session->iface.connection = &conn->iface; session->name = "wiredtiger_config_validate"; - __wt_event_handler_set(session, handler); + __wt_event_handler_set(session, event_handler); } if (session != NULL) conn = S2C(session); diff --git a/src/third_party/wiredtiger/src/config/config_check.c b/src/third_party/wiredtiger/src/config/config_check.c index 8038ae89413..d4d11c5a02e 100644 --- a/src/third_party/wiredtiger/src/config/config_check.c +++ b/src/third_party/wiredtiger/src/config/config_check.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/config/config_collapse.c b/src/third_party/wiredtiger/src/config/config_collapse.c index 155b700f2dd..26636873902 100644 --- a/src/third_party/wiredtiger/src/config/config_collapse.c +++ b/src/third_party/wiredtiger/src/config/config_collapse.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/config/config_def.c b/src/third_party/wiredtiger/src/config/config_def.c index 14234badcfa..cc3ea7c9d52 100644 --- a/src/third_party/wiredtiger/src/config/config_def.c +++ b/src/third_party/wiredtiger/src/config/config_def.c @@ -106,6 +106,13 @@ static const WT_CONFIG_CHECK }; static const WT_CONFIG_CHECK + confchk_wiredtiger_open_operation_tracking_subconfigs[] = { + { "enabled", "boolean", NULL, NULL, NULL, 0 }, + { "path", "string", NULL, NULL, NULL, 0 }, + { NULL, NULL, NULL, NULL, NULL, 0 } +}; + +static const WT_CONFIG_CHECK confchk_wiredtiger_open_shared_cache_subconfigs[] = { { "chunk", "int", NULL, "min=1MB,max=10TB", NULL, 0 }, { "name", "string", NULL, NULL, NULL, 0 }, @@ -142,16 +149,18 @@ static const WT_CONFIG_CHECK confchk_WT_CONNECTION_reconfigure[] = { NULL, NULL, confchk_wiredtiger_open_eviction_subconfigs, 2 }, { "eviction_checkpoint_target", "int", - NULL, "min=0,max=99", + NULL, "min=0,max=10TB", NULL, 0 }, { "eviction_dirty_target", "int", - NULL, "min=1,max=99", + NULL, "min=1,max=10TB", NULL, 0 }, { "eviction_dirty_trigger", "int", - NULL, "min=1,max=99", + NULL, "min=1,max=10TB", + NULL, 0 }, + { "eviction_target", "int", NULL, "min=10,max=10TB", NULL, 0 }, + { "eviction_trigger", "int", + NULL, "min=10,max=10TB", NULL, 0 }, - { "eviction_target", "int", NULL, "min=10,max=99", NULL, 0 }, - { "eviction_trigger", "int", NULL, "min=10,max=99", NULL, 0 }, { "file_manager", "category", NULL, NULL, confchk_wiredtiger_open_file_manager_subconfigs, 3 }, @@ -162,6 +171,9 @@ static const WT_CONFIG_CHECK confchk_WT_CONNECTION_reconfigure[] = { NULL, NULL, confchk_wiredtiger_open_lsm_manager_subconfigs, 2 }, { "lsm_merge", "boolean", NULL, NULL, NULL, 0 }, + { "operation_tracking", "category", + NULL, NULL, + confchk_wiredtiger_open_operation_tracking_subconfigs, 2 }, { "shared_cache", "category", NULL, NULL, confchk_wiredtiger_open_shared_cache_subconfigs, 5 }, @@ -183,8 +195,8 @@ static const WT_CONFIG_CHECK confchk_WT_CONNECTION_reconfigure[] = { "\"lookaside_activity\",\"lsm\",\"lsm_manager\",\"metadata\"," "\"mutex\",\"overflow\",\"read\",\"rebalance\",\"reconcile\"," "\"recovery\",\"recovery_progress\",\"salvage\",\"shared_cache\"," - "\"split\",\"thread_group\",\"timestamp\",\"transaction\"," - "\"verify\",\"version\",\"write\"]", + "\"split\",\"temporary\",\"thread_group\",\"timestamp\"," + "\"transaction\",\"verify\",\"version\",\"write\"]", NULL, 0 }, { NULL, NULL, NULL, NULL, NULL, 0 } }; @@ -205,7 +217,8 @@ static const WT_CONFIG_CHECK confchk_WT_CURSOR_reconfigure[] = { static const WT_CONFIG_CHECK confchk_assert_subconfigs[] = { { "commit_timestamp", "string", - NULL, "choices=[\"always\",\"never\",\"none\"]", + NULL, "choices=[\"always\",\"key_consistent\",\"never\"," + "\"none\"]", NULL, 0 }, { "read_timestamp", "string", NULL, "choices=[\"always\",\"never\",\"none\"]", @@ -277,6 +290,14 @@ static const WT_CONFIG_CHECK }; static const WT_CONFIG_CHECK + confchk_WT_SESSION_create_merge_custom_subconfigs[] = { + { "prefix", "string", NULL, NULL, NULL, 0 }, + { "start_generation", "int", NULL, "min=0,max=10", NULL, 0 }, + { "suffix", "string", NULL, NULL, NULL, 0 }, + { NULL, NULL, NULL, NULL, NULL, 0 } +}; + +static const WT_CONFIG_CHECK confchk_WT_SESSION_create_lsm_subconfigs[] = { { "auto_throttle", "boolean", NULL, NULL, NULL, 0 }, { "bloom", "boolean", NULL, NULL, NULL, 0 }, @@ -287,6 +308,9 @@ static const WT_CONFIG_CHECK { "chunk_count_limit", "int", NULL, NULL, NULL, 0 }, { "chunk_max", "int", NULL, "min=100MB,max=10TB", NULL, 0 }, { "chunk_size", "int", NULL, "min=512K,max=500MB", NULL, 0 }, + { "merge_custom", "category", + NULL, NULL, + confchk_WT_SESSION_create_merge_custom_subconfigs, 3 }, { "merge_max", "int", NULL, "min=2,max=100", NULL, 0 }, { "merge_min", "int", NULL, "max=100", NULL, 0 }, { NULL, NULL, NULL, NULL, NULL, 0 } @@ -346,7 +370,7 @@ static const WT_CONFIG_CHECK confchk_WT_SESSION_create[] = { confchk_WT_SESSION_create_log_subconfigs, 1 }, { "lsm", "category", NULL, NULL, - confchk_WT_SESSION_create_lsm_subconfigs, 11 }, + confchk_WT_SESSION_create_lsm_subconfigs, 12 }, { "memory_page_max", "int", NULL, "min=512B,max=10TB", NULL, 0 }, @@ -680,7 +704,7 @@ static const WT_CONFIG_CHECK confchk_lsm_meta[] = { confchk_WT_SESSION_create_log_subconfigs, 1 }, { "lsm", "category", NULL, NULL, - confchk_WT_SESSION_create_lsm_subconfigs, 11 }, + confchk_WT_SESSION_create_lsm_subconfigs, 12 }, { "memory_page_max", "int", NULL, "min=512B,max=10TB", NULL, 0 }, @@ -781,16 +805,18 @@ static const WT_CONFIG_CHECK confchk_wiredtiger_open[] = { NULL, NULL, confchk_wiredtiger_open_eviction_subconfigs, 2 }, { "eviction_checkpoint_target", "int", - NULL, "min=0,max=99", + NULL, "min=0,max=10TB", NULL, 0 }, { "eviction_dirty_target", "int", - NULL, "min=1,max=99", + NULL, "min=1,max=10TB", NULL, 0 }, { "eviction_dirty_trigger", "int", - NULL, "min=1,max=99", + NULL, "min=1,max=10TB", + NULL, 0 }, + { "eviction_target", "int", NULL, "min=10,max=10TB", NULL, 0 }, + { "eviction_trigger", "int", + NULL, "min=10,max=10TB", NULL, 0 }, - { "eviction_target", "int", NULL, "min=10,max=99", NULL, 0 }, - { "eviction_trigger", "int", NULL, "min=10,max=99", NULL, 0 }, { "exclusive", "boolean", NULL, NULL, NULL, 0 }, { "extensions", "list", NULL, NULL, NULL, 0 }, { "file_extend", "list", @@ -810,6 +836,9 @@ static const WT_CONFIG_CHECK confchk_wiredtiger_open[] = { { "lsm_merge", "boolean", NULL, NULL, NULL, 0 }, { "mmap", "boolean", NULL, NULL, NULL, 0 }, { "multiprocess", "boolean", NULL, NULL, NULL, 0 }, + { "operation_tracking", "category", + NULL, NULL, + confchk_wiredtiger_open_operation_tracking_subconfigs, 2 }, { "readonly", "boolean", NULL, NULL, NULL, 0 }, { "session_max", "int", NULL, "min=1", NULL, 0 }, { "session_scratch_max", "int", NULL, NULL, NULL, 0 }, @@ -839,8 +868,8 @@ static const WT_CONFIG_CHECK confchk_wiredtiger_open[] = { "\"lookaside_activity\",\"lsm\",\"lsm_manager\",\"metadata\"," "\"mutex\",\"overflow\",\"read\",\"rebalance\",\"reconcile\"," "\"recovery\",\"recovery_progress\",\"salvage\",\"shared_cache\"," - "\"split\",\"thread_group\",\"timestamp\",\"transaction\"," - "\"verify\",\"version\",\"write\"]", + "\"split\",\"temporary\",\"thread_group\",\"timestamp\"," + "\"transaction\",\"verify\",\"version\",\"write\"]", NULL, 0 }, { "write_through", "list", NULL, "choices=[\"data\",\"log\"]", @@ -876,16 +905,18 @@ static const WT_CONFIG_CHECK confchk_wiredtiger_open_all[] = { NULL, NULL, confchk_wiredtiger_open_eviction_subconfigs, 2 }, { "eviction_checkpoint_target", "int", - NULL, "min=0,max=99", + NULL, "min=0,max=10TB", NULL, 0 }, { "eviction_dirty_target", "int", - NULL, "min=1,max=99", + NULL, "min=1,max=10TB", NULL, 0 }, { "eviction_dirty_trigger", "int", - NULL, "min=1,max=99", + NULL, "min=1,max=10TB", + NULL, 0 }, + { "eviction_target", "int", NULL, "min=10,max=10TB", NULL, 0 }, + { "eviction_trigger", "int", + NULL, "min=10,max=10TB", NULL, 0 }, - { "eviction_target", "int", NULL, "min=10,max=99", NULL, 0 }, - { "eviction_trigger", "int", NULL, "min=10,max=99", NULL, 0 }, { "exclusive", "boolean", NULL, NULL, NULL, 0 }, { "extensions", "list", NULL, NULL, NULL, 0 }, { "file_extend", "list", @@ -905,6 +936,9 @@ static const WT_CONFIG_CHECK confchk_wiredtiger_open_all[] = { { "lsm_merge", "boolean", NULL, NULL, NULL, 0 }, { "mmap", "boolean", NULL, NULL, NULL, 0 }, { "multiprocess", "boolean", NULL, NULL, NULL, 0 }, + { "operation_tracking", "category", + NULL, NULL, + confchk_wiredtiger_open_operation_tracking_subconfigs, 2 }, { "readonly", "boolean", NULL, NULL, NULL, 0 }, { "session_max", "int", NULL, "min=1", NULL, 0 }, { "session_scratch_max", "int", NULL, NULL, NULL, 0 }, @@ -934,8 +968,8 @@ static const WT_CONFIG_CHECK confchk_wiredtiger_open_all[] = { "\"lookaside_activity\",\"lsm\",\"lsm_manager\",\"metadata\"," "\"mutex\",\"overflow\",\"read\",\"rebalance\",\"reconcile\"," "\"recovery\",\"recovery_progress\",\"salvage\",\"shared_cache\"," - "\"split\",\"thread_group\",\"timestamp\",\"transaction\"," - "\"verify\",\"version\",\"write\"]", + "\"split\",\"temporary\",\"thread_group\",\"timestamp\"," + "\"transaction\",\"verify\",\"version\",\"write\"]", NULL, 0 }, { "version", "string", NULL, NULL, NULL, 0 }, { "write_through", "list", @@ -970,16 +1004,18 @@ static const WT_CONFIG_CHECK confchk_wiredtiger_open_basecfg[] = { NULL, NULL, confchk_wiredtiger_open_eviction_subconfigs, 2 }, { "eviction_checkpoint_target", "int", - NULL, "min=0,max=99", + NULL, "min=0,max=10TB", NULL, 0 }, { "eviction_dirty_target", "int", - NULL, "min=1,max=99", + NULL, "min=1,max=10TB", NULL, 0 }, { "eviction_dirty_trigger", "int", - NULL, "min=1,max=99", + NULL, "min=1,max=10TB", + NULL, 0 }, + { "eviction_target", "int", NULL, "min=10,max=10TB", NULL, 0 }, + { "eviction_trigger", "int", + NULL, "min=10,max=10TB", NULL, 0 }, - { "eviction_target", "int", NULL, "min=10,max=99", NULL, 0 }, - { "eviction_trigger", "int", NULL, "min=10,max=99", NULL, 0 }, { "extensions", "list", NULL, NULL, NULL, 0 }, { "file_extend", "list", NULL, "choices=[\"data\",\"log\"]", @@ -997,6 +1033,9 @@ static const WT_CONFIG_CHECK confchk_wiredtiger_open_basecfg[] = { { "lsm_merge", "boolean", NULL, NULL, NULL, 0 }, { "mmap", "boolean", NULL, NULL, NULL, 0 }, { "multiprocess", "boolean", NULL, NULL, NULL, 0 }, + { "operation_tracking", "category", + NULL, NULL, + confchk_wiredtiger_open_operation_tracking_subconfigs, 2 }, { "readonly", "boolean", NULL, NULL, NULL, 0 }, { "session_max", "int", NULL, "min=1", NULL, 0 }, { "session_scratch_max", "int", NULL, NULL, NULL, 0 }, @@ -1024,8 +1063,8 @@ static const WT_CONFIG_CHECK confchk_wiredtiger_open_basecfg[] = { "\"lookaside_activity\",\"lsm\",\"lsm_manager\",\"metadata\"," "\"mutex\",\"overflow\",\"read\",\"rebalance\",\"reconcile\"," "\"recovery\",\"recovery_progress\",\"salvage\",\"shared_cache\"," - "\"split\",\"thread_group\",\"timestamp\",\"transaction\"," - "\"verify\",\"version\",\"write\"]", + "\"split\",\"temporary\",\"thread_group\",\"timestamp\"," + "\"transaction\",\"verify\",\"version\",\"write\"]", NULL, 0 }, { "version", "string", NULL, NULL, NULL, 0 }, { "write_through", "list", @@ -1060,16 +1099,18 @@ static const WT_CONFIG_CHECK confchk_wiredtiger_open_usercfg[] = { NULL, NULL, confchk_wiredtiger_open_eviction_subconfigs, 2 }, { "eviction_checkpoint_target", "int", - NULL, "min=0,max=99", + NULL, "min=0,max=10TB", NULL, 0 }, { "eviction_dirty_target", "int", - NULL, "min=1,max=99", + NULL, "min=1,max=10TB", NULL, 0 }, { "eviction_dirty_trigger", "int", - NULL, "min=1,max=99", + NULL, "min=1,max=10TB", + NULL, 0 }, + { "eviction_target", "int", NULL, "min=10,max=10TB", NULL, 0 }, + { "eviction_trigger", "int", + NULL, "min=10,max=10TB", NULL, 0 }, - { "eviction_target", "int", NULL, "min=10,max=99", NULL, 0 }, - { "eviction_trigger", "int", NULL, "min=10,max=99", NULL, 0 }, { "extensions", "list", NULL, NULL, NULL, 0 }, { "file_extend", "list", NULL, "choices=[\"data\",\"log\"]", @@ -1087,6 +1128,9 @@ static const WT_CONFIG_CHECK confchk_wiredtiger_open_usercfg[] = { { "lsm_merge", "boolean", NULL, NULL, NULL, 0 }, { "mmap", "boolean", NULL, NULL, NULL, 0 }, { "multiprocess", "boolean", NULL, NULL, NULL, 0 }, + { "operation_tracking", "category", + NULL, NULL, + confchk_wiredtiger_open_operation_tracking_subconfigs, 2 }, { "readonly", "boolean", NULL, NULL, NULL, 0 }, { "session_max", "int", NULL, "min=1", NULL, 0 }, { "session_scratch_max", "int", NULL, NULL, NULL, 0 }, @@ -1114,8 +1158,8 @@ static const WT_CONFIG_CHECK confchk_wiredtiger_open_usercfg[] = { "\"lookaside_activity\",\"lsm\",\"lsm_manager\",\"metadata\"," "\"mutex\",\"overflow\",\"read\",\"rebalance\",\"reconcile\"," "\"recovery\",\"recovery_progress\",\"salvage\",\"shared_cache\"," - "\"split\",\"thread_group\",\"timestamp\",\"transaction\"," - "\"verify\",\"version\",\"write\"]", + "\"split\",\"temporary\",\"thread_group\",\"timestamp\"," + "\"transaction\",\"verify\",\"version\",\"write\"]", NULL, 0 }, { "write_through", "list", NULL, "choices=[\"data\",\"log\"]", @@ -1180,11 +1224,12 @@ static const WT_CONFIG_ENTRY config_entries[] = { "file_manager=(close_handle_minimum=250,close_idle_time=30," "close_scan_interval=10),log=(archive=true,prealloc=true," "zero_fill=false),lsm_manager=(merge=true,worker_thread_max=4)," - "lsm_merge=true,shared_cache=(chunk=10MB,name=,quota=0,reserve=0," - "size=500MB),statistics=none,statistics_log=(json=false," - "on_close=false,sources=,timestamp=\"%b %d %H:%M:%S\",wait=0)," + "lsm_merge=true,operation_tracking=(enabled=false,path=\".\")," + "shared_cache=(chunk=10MB,name=,quota=0,reserve=0,size=500MB)," + "statistics=none,statistics_log=(json=false,on_close=false," + "sources=,timestamp=\"%b %d %H:%M:%S\",wait=0)," "timing_stress_for_test=,verbose=", - confchk_WT_CONNECTION_reconfigure, 21 + confchk_WT_CONNECTION_reconfigure, 22 }, { "WT_CONNECTION.rollback_to_stable", "", @@ -1247,7 +1292,8 @@ static const WT_CONFIG_ENTRY config_entries[] = { "leaf_value_max=0,log=(enabled=true),lsm=(auto_throttle=true," "bloom=true,bloom_bit_count=16,bloom_config=,bloom_hash_count=8," "bloom_oldest=false,chunk_count_limit=0,chunk_max=5GB," - "chunk_size=10MB,merge_max=15,merge_min=0),memory_page_max=5MB," + "chunk_size=10MB,merge_custom=(prefix=,start_generation=0," + "suffix=),merge_max=15,merge_min=0),memory_page_max=5MB," "os_cache_dirty_max=0,os_cache_max=0,prefix_compression=false," "prefix_compression_min=4,source=,split_deepen_min_child=0," "split_deepen_per_child=0,split_pct=90,type=file,value_format=u", @@ -1387,7 +1433,8 @@ static const WT_CONFIG_ENTRY config_entries[] = { "leaf_page_max=32KB,leaf_value_max=0,log=(enabled=true)," "lsm=(auto_throttle=true,bloom=true,bloom_bit_count=16," "bloom_config=,bloom_hash_count=8,bloom_oldest=false," - "chunk_count_limit=0,chunk_max=5GB,chunk_size=10MB,merge_max=15," + "chunk_count_limit=0,chunk_max=5GB,chunk_size=10MB," + "merge_custom=(prefix=,start_generation=0,suffix=),merge_max=15," "merge_min=0),memory_page_max=5MB,old_chunks=," "os_cache_dirty_max=0,os_cache_max=0,prefix_compression=false," "prefix_compression_min=4,split_deepen_min_child=0," @@ -1414,14 +1461,15 @@ static const WT_CONFIG_ENTRY config_entries[] = { "log=(archive=true,compressor=,enabled=false,file_max=100MB," "path=\".\",prealloc=true,recover=on,zero_fill=false)," "lsm_manager=(merge=true,worker_thread_max=4),lsm_merge=true," - "mmap=true,multiprocess=false,readonly=false,session_max=100," + "mmap=true,multiprocess=false,operation_tracking=(enabled=false," + "path=\".\"),readonly=false,session_max=100," "session_scratch_max=2MB,shared_cache=(chunk=10MB,name=,quota=0," "reserve=0,size=500MB),statistics=none,statistics_log=(json=false" ",on_close=false,path=\".\",sources=,timestamp=\"%b %d %H:%M:%S\"" ",wait=0),timing_stress_for_test=,transaction_sync=(enabled=false" ",method=fsync),use_environment=true,use_environment_priv=false," "verbose=,write_through=", - confchk_wiredtiger_open, 42 + confchk_wiredtiger_open, 43 }, { "wiredtiger_open_all", "async=(enabled=false,ops_max=1024,threads=2),buffer_alignment=-1" @@ -1438,14 +1486,15 @@ static const WT_CONFIG_ENTRY config_entries[] = { "log=(archive=true,compressor=,enabled=false,file_max=100MB," "path=\".\",prealloc=true,recover=on,zero_fill=false)," "lsm_manager=(merge=true,worker_thread_max=4),lsm_merge=true," - "mmap=true,multiprocess=false,readonly=false,session_max=100," + "mmap=true,multiprocess=false,operation_tracking=(enabled=false," + "path=\".\"),readonly=false,session_max=100," "session_scratch_max=2MB,shared_cache=(chunk=10MB,name=,quota=0," "reserve=0,size=500MB),statistics=none,statistics_log=(json=false" ",on_close=false,path=\".\",sources=,timestamp=\"%b %d %H:%M:%S\"" ",wait=0),timing_stress_for_test=,transaction_sync=(enabled=false" ",method=fsync),use_environment=true,use_environment_priv=false," "verbose=,version=(major=0,minor=0),write_through=", - confchk_wiredtiger_open_all, 43 + confchk_wiredtiger_open_all, 44 }, { "wiredtiger_open_basecfg", "async=(enabled=false,ops_max=1024,threads=2),buffer_alignment=-1" @@ -1460,13 +1509,14 @@ static const WT_CONFIG_ENTRY config_entries[] = { "log=(archive=true,compressor=,enabled=false,file_max=100MB," "path=\".\",prealloc=true,recover=on,zero_fill=false)," "lsm_manager=(merge=true,worker_thread_max=4),lsm_merge=true," - "mmap=true,multiprocess=false,readonly=false,session_max=100," + "mmap=true,multiprocess=false,operation_tracking=(enabled=false," + "path=\".\"),readonly=false,session_max=100," "session_scratch_max=2MB,shared_cache=(chunk=10MB,name=,quota=0," "reserve=0,size=500MB),statistics=none,statistics_log=(json=false" ",on_close=false,path=\".\",sources=,timestamp=\"%b %d %H:%M:%S\"" ",wait=0),timing_stress_for_test=,transaction_sync=(enabled=false" ",method=fsync),verbose=,version=(major=0,minor=0),write_through=", - confchk_wiredtiger_open_basecfg, 37 + confchk_wiredtiger_open_basecfg, 38 }, { "wiredtiger_open_usercfg", "async=(enabled=false,ops_max=1024,threads=2),buffer_alignment=-1" @@ -1481,13 +1531,14 @@ static const WT_CONFIG_ENTRY config_entries[] = { "log=(archive=true,compressor=,enabled=false,file_max=100MB," "path=\".\",prealloc=true,recover=on,zero_fill=false)," "lsm_manager=(merge=true,worker_thread_max=4),lsm_merge=true," - "mmap=true,multiprocess=false,readonly=false,session_max=100," + "mmap=true,multiprocess=false,operation_tracking=(enabled=false," + "path=\".\"),readonly=false,session_max=100," "session_scratch_max=2MB,shared_cache=(chunk=10MB,name=,quota=0," "reserve=0,size=500MB),statistics=none,statistics_log=(json=false" ",on_close=false,path=\".\",sources=,timestamp=\"%b %d %H:%M:%S\"" ",wait=0),timing_stress_for_test=,transaction_sync=(enabled=false" ",method=fsync),verbose=,write_through=", - confchk_wiredtiger_open_usercfg, 36 + confchk_wiredtiger_open_usercfg, 37 }, { NULL, NULL, NULL, 0 } }; diff --git a/src/third_party/wiredtiger/src/config/config_ext.c b/src/third_party/wiredtiger/src/config/config_ext.c index 82e5a3f1788..0992e305e9c 100644 --- a/src/third_party/wiredtiger/src/config/config_ext.c +++ b/src/third_party/wiredtiger/src/config/config_ext.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/config/config_upgrade.c b/src/third_party/wiredtiger/src/config/config_upgrade.c index 5f2770b1691..8b8c2919d24 100644 --- a/src/third_party/wiredtiger/src/config/config_upgrade.c +++ b/src/third_party/wiredtiger/src/config/config_upgrade.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/conn/api_version.c b/src/third_party/wiredtiger/src/conn/api_version.c index c4f3d978c1e..45f67dffc98 100644 --- a/src/third_party/wiredtiger/src/conn/api_version.c +++ b/src/third_party/wiredtiger/src/conn/api_version.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/conn/conn_api.c b/src/third_party/wiredtiger/src/conn/conn_api.c index fd8fd6763db..d9f93e8bdce 100644 --- a/src/third_party/wiredtiger/src/conn/conn_api.c +++ b/src/third_party/wiredtiger/src/conn/conn_api.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -1084,6 +1084,8 @@ err: /* WT_TRET(wt_session->close(wt_session, config)); } + WT_TRET(__wt_async_flush(session)); + /* * Disable lookaside eviction: it doesn't help us shut down and can * lead to pages being marked dirty, causing spurious assertions to @@ -1794,7 +1796,7 @@ err: /* /* Simple structure for name and flag configuration searches. */ typedef struct { const char *name; - uint32_t flag; + uint64_t flag; } WT_NAME_FLAG; /* @@ -1831,6 +1833,7 @@ __wt_verbose_config(WT_SESSION_IMPL *session, const char *cfg[]) { "salvage", WT_VERB_SALVAGE }, { "shared_cache", WT_VERB_SHARED_CACHE }, { "split", WT_VERB_SPLIT }, + { "temporary", WT_VERB_TEMPORARY }, { "thread_group", WT_VERB_THREAD_GROUP }, { "timestamp", WT_VERB_TIMESTAMP }, { "transaction", WT_VERB_TRANSACTION }, @@ -1843,7 +1846,7 @@ __wt_verbose_config(WT_SESSION_IMPL *session, const char *cfg[]) WT_CONNECTION_IMPL *conn; WT_DECL_RET; const WT_NAME_FLAG *ft; - uint32_t flags; + uint64_t flags; conn = S2C(session); @@ -1852,17 +1855,8 @@ __wt_verbose_config(WT_SESSION_IMPL *session, const char *cfg[]) flags = 0; for (ft = verbtypes; ft->name != NULL; ft++) { if ((ret = __wt_config_subgets( - session, &cval, ft->name, &sval)) == 0 && sval.val != 0) { -#ifdef HAVE_VERBOSE + session, &cval, ft->name, &sval)) == 0 && sval.val != 0) LF_SET(ft->flag); -#else - WT_RET_MSG(session, EINVAL, - "Verbose option specified when WiredTiger built " - "without verbose support. Add --enable-verbose to " - "configure command and rebuild to include support " - "for verbose messages"); -#endif - } WT_RET_NOTFOUND_OK(ret); } @@ -2002,7 +1996,7 @@ __wt_timing_stress_config(WT_SESSION_IMPL *session, const char *cfg[]) WT_CONNECTION_IMPL *conn; WT_DECL_RET; const WT_NAME_FLAG *ft; - uint32_t flags; + uint64_t flags; conn = S2C(session); @@ -2172,6 +2166,44 @@ err: API_END_RET(session, ret); } /* + * __conn_session_size -- + * Return the session count for this run. + */ +static int +__conn_session_size( + WT_SESSION_IMPL *session, const char *cfg[], uint32_t *vp) +{ + WT_CONFIG_ITEM cval; + int64_t v; + + /* + * Start with 20 internal sessions to cover threads the application + * can't configure (for example, checkpoint or statistics log server + * threads). + */ +#define WT_EXTRA_INTERNAL_SESSIONS 20 + v = WT_EXTRA_INTERNAL_SESSIONS; + + /* Then, add in the thread counts applications can configure. */ + WT_RET(__wt_config_gets(session, cfg, "async.threads", &cval)); + v += cval.val; + + WT_RET(__wt_config_gets(session, cfg, "eviction.threads_max", &cval)); + v += cval.val; + + WT_RET(__wt_config_gets( + session, cfg, "lsm_manager.worker_thread_max", &cval)); + v += cval.val; + + WT_RET(__wt_config_gets(session, cfg, "session_max", &cval)); + v += cval.val; + + *vp = (uint32_t)v; + + return (0); +} + +/* * __conn_chk_file_system -- * Check the configured file system. */ @@ -2198,6 +2230,16 @@ __conn_chk_file_system(WT_SESSION_IMPL *session, bool readonly) } WT_CONN_SET_FILE_SYSTEM_REQ(fs_size); + /* + * The lower-level API for returning the first matching entry was added + * later and not documented because it's an optimization for high-end + * filesystems doing logging, specifically pre-allocating log files. + * Check for the API and fall back to the standard API if not available. + */ + if (conn->file_system->fs_directory_list_single == NULL) + conn->file_system->fs_directory_list_single = + conn->file_system->fs_directory_list; + return (0); } @@ -2244,7 +2286,7 @@ wiredtiger_dummy_session_init( */ int wiredtiger_open(const char *home, WT_EVENT_HANDLER *event_handler, - const char *config, WT_CONNECTION **wt_connp) + const char *config, WT_CONNECTION **connectionp) { static const WT_CONNECTION stdc = { __conn_async_flush, @@ -2292,7 +2334,7 @@ wiredtiger_open(const char *home, WT_EVENT_HANDLER *event_handler, const char *cfg[] = { NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL }; - *wt_connp = NULL; + *connectionp = NULL; conn = NULL; session = NULL; @@ -2498,8 +2540,10 @@ wiredtiger_open(const char *home, WT_EVENT_HANDLER *event_handler, WT_ERR(__wt_verbose_config(session, cfg)); WT_ERR(__wt_timing_stress_config(session, cfg)); - WT_ERR(__wt_config_gets(session, cfg, "session_max", &cval)); - conn->session_size = (uint32_t)cval.val + WT_EXTRA_INTERNAL_SESSIONS; + /* Set up operation tracking if configured. */ + WT_ERR(__wt_conn_optrack_setup(session, cfg, false)); + + WT_ERR(__conn_session_size(session, cfg, &conn->session_size)); WT_ERR(__wt_config_gets(session, cfg, "session_scratch_max", &cval)); conn->session_scratch_max = (size_t)cval.val; @@ -2635,7 +2679,7 @@ wiredtiger_open(const char *home, WT_EVENT_HANDLER *event_handler, WT_ERR(__wt_connection_workers(session, cfg)); WT_STATIC_ASSERT(offsetof(WT_CONNECTION_IMPL, iface) == 0); - *wt_connp = &conn->iface; + *connectionp = &conn->iface; err: /* Discard the scratch buffers. */ __wt_scr_free(session, &encbuf); diff --git a/src/third_party/wiredtiger/src/conn/conn_cache.c b/src/third_party/wiredtiger/src/conn/conn_cache.c index a7a9b8e2729..871190380f7 100644 --- a/src/third_party/wiredtiger/src/conn/conn_cache.c +++ b/src/third_party/wiredtiger/src/conn/conn_cache.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -9,6 +9,47 @@ #include "wt_internal.h" /* + * __cache_config_abs_to_pct -- + * Cache configuration values can be either a percentage or an absolute + * size, this function converts an absolute size to a percentage. + */ +static inline int +__cache_config_abs_to_pct(WT_SESSION_IMPL *session, + double *param, const char *param_name, bool shared) +{ + WT_CONNECTION_IMPL *conn; + double input; + + conn = S2C(session); + + WT_ASSERT(session, param != NULL); + input = *param; + + /* + * Anything above 100 is an absolute value; convert it to percentage. + */ + if (input > 100.0) { + /* + * In a shared cache configuration the cache size changes + * regularly. Therefore, we require a percentage setting and do + * not allow an absolute size setting. + */ + if (shared) + WT_RET_MSG(session, EINVAL, + "Shared cache configuration requires a percentage " + "value for %s", param_name); + /* An absolute value can't exceed the cache size. */ + if (input > conn->cache_size) + WT_RET_MSG(session, EINVAL, + "%s should not exceed cache size", param_name); + + *param = (input * 100.0) / (conn->cache_size); + } + + return (0); +} + +/* * __cache_config_local -- * Configure the underlying cache. */ @@ -37,17 +78,26 @@ __cache_config_local(WT_SESSION_IMPL *session, bool shared, const char *cfg[]) cache->overhead_pct = (u_int)cval.val; WT_RET(__wt_config_gets(session, cfg, "eviction_target", &cval)); - cache->eviction_target = (u_int)cval.val; + cache->eviction_target = (double)cval.val; + WT_RET(__cache_config_abs_to_pct( + session, &(cache->eviction_target), "eviction target", shared)); WT_RET(__wt_config_gets(session, cfg, "eviction_trigger", &cval)); - cache->eviction_trigger = (u_int)cval.val; + cache->eviction_trigger = (double)cval.val; + WT_RET(__cache_config_abs_to_pct( + session, &(cache->eviction_trigger), "eviction trigger", shared)); WT_RET(__wt_config_gets( session, cfg, "eviction_checkpoint_target", &cval)); - cache->eviction_checkpoint_target = (u_int)cval.val; + cache->eviction_checkpoint_target = (double)cval.val; + WT_RET(__cache_config_abs_to_pct(session, + &(cache->eviction_checkpoint_target), + "eviction checkpoint target", shared)); WT_RET(__wt_config_gets(session, cfg, "eviction_dirty_target", &cval)); - cache->eviction_dirty_target = (u_int)cval.val; + cache->eviction_dirty_target = (double)cval.val; + WT_RET(__cache_config_abs_to_pct(session, + &(cache->eviction_dirty_target), "eviction dirty target", shared)); /* * Don't allow the dirty target to be larger than the overall @@ -66,7 +116,10 @@ __cache_config_local(WT_SESSION_IMPL *session, bool shared, const char *cfg[]) cache->eviction_dirty_target; WT_RET(__wt_config_gets(session, cfg, "eviction_dirty_trigger", &cval)); - cache->eviction_dirty_trigger = (u_int)cval.val; + cache->eviction_dirty_trigger = (double)cval.val; + WT_RET(__cache_config_abs_to_pct(session, + &(cache->eviction_dirty_trigger), "eviction dirty trigger", + shared)); /* * Don't allow the dirty trigger to be larger than the overall diff --git a/src/third_party/wiredtiger/src/conn/conn_cache_pool.c b/src/third_party/wiredtiger/src/conn/conn_cache_pool.c index ade63b00af6..720df3c465d 100644 --- a/src/third_party/wiredtiger/src/conn/conn_cache_pool.c +++ b/src/third_party/wiredtiger/src/conn/conn_cache_pool.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -304,6 +304,7 @@ __wt_conn_cache_pool_destroy(WT_SESSION_IMPL *session) conn = S2C(session); cache = conn->cache; cp_locked = found = false; + WT_NOT_READ(cp_locked); cp = __wt_process.cache_pool; if (!F_ISSET(conn, WT_CONN_CACHE_POOL)) @@ -338,6 +339,7 @@ __wt_conn_cache_pool_destroy(WT_SESSION_IMPL *session) */ __wt_spin_unlock(session, &cp->cache_pool_lock); cp_locked = false; + WT_NOT_READ(cp_locked); FLD_CLR(cache->pool_flags, WT_CACHE_POOL_RUN); __wt_cond_signal(session, cp->cache_pool_cond); @@ -570,8 +572,8 @@ __cache_pool_adjust(WT_SESSION_IMPL *session, WT_CACHE *cache; WT_CACHE_POOL *cp; WT_CONNECTION_IMPL *entry; + double pct_full; uint64_t adjustment, highest_percentile, pressure, reserved, smallest; - u_int pct_full; bool busy, decrease_ok, grow, pool_full; *adjustedp = false; @@ -579,7 +581,7 @@ __cache_pool_adjust(WT_SESSION_IMPL *session, cp = __wt_process.cache_pool; grow = false; pool_full = cp->currently_used >= cp->size; - pct_full = 0; + pct_full = 0.0; /* Highest as a percentage, avoid 0 */ highest_percentile = (highest / 100) + 1; @@ -611,7 +613,7 @@ __cache_pool_adjust(WT_SESSION_IMPL *session, entry->default_session, false, true, &pct_full); __wt_verbose(session, WT_VERB_SHARED_CACHE, - "\t%5" PRIu64 ", %3" PRIu64 ", %2" PRIu32 ", %d, %2u", + "\t%5" PRIu64 ", %3" PRIu64 ", %2" PRIu32 ", %d, %2.3f", entry->cache_size >> 20, pressure, cache->cp_skip_count, busy, pct_full); @@ -674,8 +676,9 @@ __cache_pool_adjust(WT_SESSION_IMPL *session, * potentially a negative feedback loop in the * balance algorithm. */ - smallest = (100 * __wt_cache_bytes_inuse(cache)) / - cache->eviction_trigger; + smallest = + (uint64_t)((100 * __wt_cache_bytes_inuse(cache)) / + cache->eviction_trigger); if (entry->cache_size > smallest) adjustment = WT_MIN(cp->chunk, (entry->cache_size - smallest) / 2); diff --git a/src/third_party/wiredtiger/src/conn/conn_ckpt.c b/src/third_party/wiredtiger/src/conn/conn_ckpt.c index a47524af2d7..d302a59c928 100644 --- a/src/third_party/wiredtiger/src/conn/conn_ckpt.c +++ b/src/third_party/wiredtiger/src/conn/conn_ckpt.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/conn/conn_dhandle.c b/src/third_party/wiredtiger/src/conn/conn_dhandle.c index 3cf075e52f8..d7355bffa59 100644 --- a/src/third_party/wiredtiger/src/conn/conn_dhandle.c +++ b/src/third_party/wiredtiger/src/conn/conn_dhandle.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/conn/conn_handle.c b/src/third_party/wiredtiger/src/conn/conn_handle.c index 42ae866b329..42b8510cbd3 100644 --- a/src/third_party/wiredtiger/src/conn/conn_handle.c +++ b/src/third_party/wiredtiger/src/conn/conn_handle.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/conn/conn_log.c b/src/third_party/wiredtiger/src/conn/conn_log.c index 8819bf8b970..9097e10ef5a 100644 --- a/src/third_party/wiredtiger/src/conn/conn_log.c +++ b/src/third_party/wiredtiger/src/conn/conn_log.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -599,6 +599,7 @@ __log_file_server(void *arg) WT_ERR(__wt_fsync(session, log->log_fh, true)); __wt_spin_lock(session, &log->log_sync_lock); locked = true; + WT_NOT_READ(locked); /* * The sync LSN could have advanced while we * were writing to disk. @@ -865,12 +866,11 @@ err: WT_PANIC_MSG(session, ret, "log wrlsn server error"); static WT_THREAD_RET __log_server(void *arg) { - struct timespec start, now; WT_CONNECTION_IMPL *conn; WT_DECL_RET; WT_LOG *log; WT_SESSION_IMPL *session; - uint64_t timediff; + uint64_t time_start, time_stop, timediff; bool did_work, signalled; session = arg; @@ -949,11 +949,11 @@ __log_server(void *arg) } /* Wait until the next event. */ - __wt_epoch(session, &start); + time_start = __wt_clock(session); __wt_cond_auto_wait_signal( session, conn->log_cond, did_work, NULL, &signalled); - __wt_epoch(session, &now); - timediff = WT_TIMEDIFF_MS(now, start); + time_stop = __wt_clock(session); + timediff = WT_CLOCKDIFF_MS(time_stop, time_start); } if (0) { diff --git a/src/third_party/wiredtiger/src/conn/conn_open.c b/src/third_party/wiredtiger/src/conn/conn_open.c index e72fa5c00a4..a2856843354 100644 --- a/src/third_party/wiredtiger/src/conn/conn_open.c +++ b/src/third_party/wiredtiger/src/conn/conn_open.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -147,6 +147,13 @@ __wt_connection_close(WT_CONNECTION_IMPL *conn) if (conn->lock_fh != NULL) WT_TRET(__wt_close(session, &conn->lock_fh)); + /* Close any optrack files */ + if (session->optrack_fh != NULL) + WT_TRET(__wt_close(session, &session->optrack_fh)); + + /* Close operation tracking */ + WT_TRET(__wt_conn_optrack_teardown(session, false)); + /* Close any file handles left open. */ WT_TRET(__wt_close_connection_close(session)); diff --git a/src/third_party/wiredtiger/src/conn/conn_reconfig.c b/src/third_party/wiredtiger/src/conn/conn_reconfig.c index 1bde68e1762..294812ff4d0 100644 --- a/src/third_party/wiredtiger/src/conn/conn_reconfig.c +++ b/src/third_party/wiredtiger/src/conn/conn_reconfig.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -57,6 +57,107 @@ __wt_conn_compat_config(WT_SESSION_IMPL *session, const char **cfg) } /* + * __wt_conn_optrack_setup -- + * Set up operation logging. + */ +int +__wt_conn_optrack_setup(WT_SESSION_IMPL *session, + const char *cfg[], bool reconfig) +{ + WT_CONFIG_ITEM cval; + WT_CONNECTION_IMPL *conn; + WT_DECL_ITEM(buf); + WT_DECL_RET; + + conn = S2C(session); + + /* Once an operation tracking path has been set it can't be changed. */ + if (!reconfig) { + WT_RET(__wt_config_gets(session, + cfg, "operation_tracking.path", &cval)); + WT_RET(__wt_strndup(session, + cval.str, cval.len, &conn->optrack_path)); + } + + WT_RET(__wt_config_gets(session, + cfg, "operation_tracking.enabled", &cval)); + if (cval.val == 0) { + if (F_ISSET(conn, WT_CONN_OPTRACK)) { + WT_RET(__wt_conn_optrack_teardown(session, reconfig)); + F_CLR(conn, WT_CONN_OPTRACK); + } + return (0); + } + if (F_ISSET(conn, WT_CONN_READONLY)) + /* Operation tracking isn't supported in read-only mode */ + WT_RET_MSG(session, EINVAL, + "Operation tracking is incompatible with read only " + "configuration."); + if (F_ISSET(conn, WT_CONN_OPTRACK)) + /* Already enabled, nothing else to do */ + return (0); + + /* + * Operation tracking files will include the ID of the creating process + * in their name, so we can distinguish between log files created by + * different WiredTiger processes in the same directory. We cache the + * process id for future use. + */ + conn->optrack_pid = __wt_process_id(); + + /* + * Open the file in the same directory that will hold a map of + * translations between function names and function IDs. If the file + * exists, remove it. + */ + WT_RET(__wt_scr_alloc(session, 0, &buf)); + WT_ERR(__wt_filename_construct(session, conn->optrack_path, + "optrack-map", conn->optrack_pid, UINT32_MAX, buf)); + WT_ERR(__wt_open(session, + (const char *)buf->data, WT_FS_OPEN_FILE_TYPE_REGULAR, + WT_FS_OPEN_CREATE, &conn->optrack_map_fh)); + + WT_ERR(__wt_spin_init(session, + &conn->optrack_map_spinlock, "optrack map spinlock")); + + WT_ERR(__wt_malloc(session, WT_OPTRACK_BUFSIZE, + &conn->dummy_session.optrack_buf)); + + /* Set operation tracking on */ + F_SET(conn, WT_CONN_OPTRACK); + +err: __wt_scr_free(session, &buf); + return (ret); +} + +/* + * __wt_conn_optrack_teardown -- + * Clean up connection-wide resources used for operation logging. + */ +int +__wt_conn_optrack_teardown(WT_SESSION_IMPL *session, bool reconfig) +{ + WT_CONNECTION_IMPL *conn; + WT_DECL_RET; + + conn = S2C(session); + + if (!reconfig) + /* Looks like we are shutting down */ + __wt_free(session, conn->optrack_path); + + if (!F_ISSET(conn, WT_CONN_OPTRACK)) + return (0); + + __wt_spin_destroy(session, &conn->optrack_map_spinlock); + + WT_TRET(__wt_close(session, &conn->optrack_map_fh)); + __wt_free(session, conn->dummy_session.optrack_buf); + + return (ret); +} + +/* * __wt_conn_statistics_config -- * Set statistics configuration. */ @@ -189,6 +290,7 @@ __wt_conn_reconfig(WT_SESSION_IMPL *session, const char **cfg) WT_WITH_CHECKPOINT_LOCK(session, ret = __wt_conn_compat_config(session, cfg)); WT_ERR(ret); + WT_ERR(__wt_conn_optrack_setup(session, cfg, true)); WT_ERR(__wt_conn_statistics_config(session, cfg)); WT_ERR(__wt_async_reconfig(session, cfg)); WT_ERR(__wt_cache_config(session, true, cfg)); diff --git a/src/third_party/wiredtiger/src/conn/conn_stat.c b/src/third_party/wiredtiger/src/conn/conn_stat.c index 815390acfac..14a1570c138 100644 --- a/src/third_party/wiredtiger/src/conn/conn_stat.c +++ b/src/third_party/wiredtiger/src/conn/conn_stat.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -503,7 +503,6 @@ __statlog_log_one(WT_SESSION_IMPL *session, WT_ITEM *path, WT_ITEM *tmp) struct timespec ts; struct tm *tm, _tm; WT_CONNECTION_IMPL *conn; - WT_FSTREAM *log_stream; conn = S2C(session); @@ -516,16 +515,16 @@ __statlog_log_one(WT_SESSION_IMPL *session, WT_ITEM *path, WT_ITEM *tmp) WT_RET_MSG(session, ENOMEM, "strftime path conversion"); /* If the path has changed, cycle the log file. */ - if ((log_stream = conn->stat_fs) == NULL || + if (conn->stat_fs == NULL || path == NULL || strcmp(tmp->mem, path->mem) != 0) { WT_RET(__wt_fclose(session, &conn->stat_fs)); - if (path != NULL) - (void)strcpy(path->mem, tmp->mem); WT_RET(__wt_fopen(session, tmp->mem, WT_FS_OPEN_CREATE | WT_FS_OPEN_FIXED, WT_STREAM_APPEND, - &log_stream)); + &conn->stat_fs)); + + if (path != NULL) + WT_RET(__wt_buf_setstr(session, path, tmp->mem)); } - conn->stat_fs = log_stream; /* Create the entry prefix for this time of day. */ if (strftime(tmp->mem, tmp->memsize, conn->stat_format, tm) == 0) @@ -582,6 +581,7 @@ __statlog_on_close(WT_SESSION_IMPL *session) "Attempt to log statistics while a server is running"); WT_RET(__wt_scr_alloc(session, strlen(conn->stat_path) + 128, &tmp)); + WT_ERR(__wt_buf_setstr(session, tmp, "")); WT_ERR(__statlog_log_one(session, NULL, tmp)); err: __wt_scr_free(session, &tmp); @@ -613,9 +613,6 @@ __statlog_server(void *arg) session = arg; conn = S2C(session); - WT_CLEAR(path); - WT_CLEAR(tmp); - /* * We need a temporary place to build a path and an entry prefix. * The length of the path plus 128 should be more than enough. @@ -623,8 +620,12 @@ __statlog_server(void *arg) * We also need a place to store the current path, because that's * how we know when to close/re-open the file. */ + WT_CLEAR(path); WT_ERR(__wt_buf_init(session, &path, strlen(conn->stat_path) + 128)); + WT_ERR(__wt_buf_setstr(session, &path, "")); + WT_CLEAR(tmp); WT_ERR(__wt_buf_init(session, &tmp, strlen(conn->stat_path) + 128)); + WT_ERR(__wt_buf_setstr(session, &tmp, "")); for (;;) { /* Wait until the next event. */ diff --git a/src/third_party/wiredtiger/src/conn/conn_sweep.c b/src/third_party/wiredtiger/src/conn/conn_sweep.c index 06e441a3037..4d2267d176b 100644 --- a/src/third_party/wiredtiger/src/conn/conn_sweep.c +++ b/src/third_party/wiredtiger/src/conn/conn_sweep.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/cursor/cur_backup.c b/src/third_party/wiredtiger/src/cursor/cur_backup.c index ecb0e02929f..679d1474f8a 100644 --- a/src/third_party/wiredtiger/src/cursor/cur_backup.c +++ b/src/third_party/wiredtiger/src/cursor/cur_backup.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/cursor/cur_bulk.c b/src/third_party/wiredtiger/src/cursor/cur_bulk.c index 56bcbb741f7..4c181bd9458 100644 --- a/src/third_party/wiredtiger/src/cursor/cur_bulk.c +++ b/src/third_party/wiredtiger/src/cursor/cur_bulk.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/cursor/cur_config.c b/src/third_party/wiredtiger/src/cursor/cur_config.c index 6c198315e33..a7b7b2bf979 100644 --- a/src/third_party/wiredtiger/src/cursor/cur_config.c +++ b/src/third_party/wiredtiger/src/cursor/cur_config.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/cursor/cur_ds.c b/src/third_party/wiredtiger/src/cursor/cur_ds.c index 9bd90fad5f5..bd0dcd02eef 100644 --- a/src/third_party/wiredtiger/src/cursor/cur_ds.c +++ b/src/third_party/wiredtiger/src/cursor/cur_ds.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/cursor/cur_dump.c b/src/third_party/wiredtiger/src/cursor/cur_dump.c index 740d7ecc819..a15bf2ae642 100644 --- a/src/third_party/wiredtiger/src/cursor/cur_dump.c +++ b/src/third_party/wiredtiger/src/cursor/cur_dump.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/cursor/cur_file.c b/src/third_party/wiredtiger/src/cursor/cur_file.c index 22ba6d1dee1..9d6f031807f 100644 --- a/src/third_party/wiredtiger/src/cursor/cur_file.c +++ b/src/third_party/wiredtiger/src/cursor/cur_file.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -9,6 +9,13 @@ #include "wt_internal.h" /* + * Define functions that increment histogram statistics for cursor read and + * write operations latency. + */ +WT_STAT_USECS_HIST_INCR_FUNC(opread, perf_hist_opread_latency, 100) +WT_STAT_USECS_HIST_INCR_FUNC(opwrite, perf_hist_opwrite_latency, 100) + +/* * __curfile_compare -- * WT_CURSOR->compare method for the btree cursor type. */ @@ -90,6 +97,7 @@ __curfile_next(WT_CURSOR *cursor) /* Next maintains a position, key and value. */ WT_ASSERT(session, + F_ISSET(cbt, WT_CBT_ACTIVE) && F_MASK(cursor, WT_CURSTD_KEY_SET) == WT_CURSTD_KEY_INT && F_MASK(cursor, WT_CURSTD_VALUE_SET) == WT_CURSTD_VALUE_INT); @@ -115,6 +123,7 @@ __wt_curfile_next_random(WT_CURSOR *cursor) /* Next-random maintains a position, key and value. */ WT_ASSERT(session, + F_ISSET(cbt, WT_CBT_ACTIVE) && F_MASK(cursor, WT_CURSTD_KEY_SET) == WT_CURSTD_KEY_INT && F_MASK(cursor, WT_CURSTD_VALUE_SET) == WT_CURSTD_VALUE_INT); @@ -139,6 +148,7 @@ __curfile_prev(WT_CURSOR *cursor) /* Prev maintains a position, key and value. */ WT_ASSERT(session, + F_ISSET(cbt, WT_CBT_ACTIVE) && F_MASK(cursor, WT_CURSTD_KEY_SET) == WT_CURSTD_KEY_INT && F_MASK(cursor, WT_CURSTD_VALUE_SET) == WT_CURSTD_VALUE_INT); @@ -163,6 +173,7 @@ __curfile_reset(WT_CURSOR *cursor) /* Reset maintains no position, key or value. */ WT_ASSERT(session, + !F_ISSET(cbt, WT_CBT_ACTIVE) && F_MASK(cursor, WT_CURSTD_KEY_SET) == 0 && F_MASK(cursor, WT_CURSTD_VALUE_SET) == 0); @@ -179,15 +190,21 @@ __curfile_search(WT_CURSOR *cursor) WT_CURSOR_BTREE *cbt; WT_DECL_RET; WT_SESSION_IMPL *session; + uint64_t time_start, time_stop; cbt = (WT_CURSOR_BTREE *)cursor; CURSOR_API_CALL(cursor, session, search, cbt->btree); WT_ERR(__cursor_checkkey(cursor)); + time_start = __wt_clock(session); WT_ERR(__wt_btcur_search(cbt)); + time_stop = __wt_clock(session); + __wt_stat_usecs_hist_incr_opread(session, + WT_CLOCKDIFF_US(time_stop, time_start)); /* Search maintains a position, key and value. */ WT_ASSERT(session, + F_ISSET(cbt, WT_CBT_ACTIVE) && F_MASK(cursor, WT_CURSTD_KEY_SET) == WT_CURSTD_KEY_INT && F_MASK(cursor, WT_CURSTD_VALUE_SET) == WT_CURSTD_VALUE_INT); @@ -204,15 +221,21 @@ __curfile_search_near(WT_CURSOR *cursor, int *exact) WT_CURSOR_BTREE *cbt; WT_DECL_RET; WT_SESSION_IMPL *session; + uint64_t time_start, time_stop; cbt = (WT_CURSOR_BTREE *)cursor; CURSOR_API_CALL(cursor, session, search_near, cbt->btree); WT_ERR(__cursor_checkkey(cursor)); + time_start = __wt_clock(session); WT_ERR(__wt_btcur_search_near(cbt, exact)); + time_stop = __wt_clock(session); + __wt_stat_usecs_hist_incr_opread(session, + WT_CLOCKDIFF_US(time_stop, time_start)); /* Search-near maintains a position, key and value. */ WT_ASSERT(session, + F_ISSET(cbt, WT_CBT_ACTIVE) && F_MASK(cursor, WT_CURSTD_KEY_SET) == WT_CURSTD_KEY_INT && F_MASK(cursor, WT_CURSTD_VALUE_SET) == WT_CURSTD_VALUE_INT); @@ -229,6 +252,7 @@ __curfile_insert(WT_CURSOR *cursor) WT_CURSOR_BTREE *cbt; WT_DECL_RET; WT_SESSION_IMPL *session; + uint64_t time_start, time_stop; cbt = (WT_CURSOR_BTREE *)cursor; CURSOR_UPDATE_API_CALL_BTREE(cursor, session, insert, cbt->btree); @@ -237,17 +261,23 @@ __curfile_insert(WT_CURSOR *cursor) WT_ERR(__cursor_checkkey(cursor)); WT_ERR(__cursor_checkvalue(cursor)); + time_start = __wt_clock(session); WT_ERR(__wt_btcur_insert(cbt)); + time_stop = __wt_clock(session); + __wt_stat_usecs_hist_incr_opwrite(session, + WT_CLOCKDIFF_US(time_stop, time_start)); /* * Insert maintains no position, key or value (except for column-store * appends, where we are returning a key). */ WT_ASSERT(session, - (F_ISSET(cursor, WT_CURSTD_APPEND) && - F_MASK(cursor, WT_CURSTD_KEY_SET) == WT_CURSTD_KEY_INT) || + !F_ISSET(cbt, WT_CBT_ACTIVE) && + ((F_ISSET(cursor, WT_CURSTD_APPEND) && + F_MASK(cursor, WT_CURSTD_KEY_SET) == WT_CURSTD_KEY_EXT) || (!F_ISSET(cursor, WT_CURSTD_APPEND) && - F_MASK(cursor, WT_CURSTD_KEY_SET) == 0)); + F_MASK(cursor, WT_CURSTD_KEY_SET) == 0))); + WT_ASSERT(session, F_MASK(cursor, WT_CURSTD_VALUE_SET) == 0); err: CURSOR_UPDATE_API_END(session, ret); return (ret); @@ -307,6 +337,7 @@ __curfile_modify(WT_CURSOR *cursor, WT_MODIFY *entries, int nentries) * always an internal value. */ WT_ASSERT(session, + F_ISSET(cbt, WT_CBT_ACTIVE) && F_MASK(cursor, WT_CURSTD_KEY_SET) == WT_CURSTD_KEY_INT); WT_ASSERT(session, F_MASK(cursor, WT_CURSTD_VALUE_SET) != 0); @@ -324,16 +355,22 @@ __curfile_update(WT_CURSOR *cursor) WT_CURSOR_BTREE *cbt; WT_DECL_RET; WT_SESSION_IMPL *session; + uint64_t time_start, time_stop; cbt = (WT_CURSOR_BTREE *)cursor; CURSOR_UPDATE_API_CALL_BTREE(cursor, session, update, cbt->btree); WT_ERR(__cursor_checkkey(cursor)); WT_ERR(__cursor_checkvalue(cursor)); + time_start = __wt_clock(session); WT_ERR(__wt_btcur_update(cbt)); + time_stop = __wt_clock(session); + __wt_stat_usecs_hist_incr_opwrite(session, + WT_CLOCKDIFF_US(time_stop, time_start)); /* Update maintains a position, key and value. */ WT_ASSERT(session, + F_ISSET(cbt, WT_CBT_ACTIVE) && F_MASK(cursor, WT_CURSTD_KEY_SET) == WT_CURSTD_KEY_INT && F_MASK(cursor, WT_CURSTD_VALUE_SET) == WT_CURSTD_VALUE_INT); @@ -351,22 +388,24 @@ __curfile_remove(WT_CURSOR *cursor) WT_CURSOR_BTREE *cbt; WT_DECL_RET; WT_SESSION_IMPL *session; + uint64_t time_start, time_stop; cbt = (WT_CURSOR_BTREE *)cursor; CURSOR_REMOVE_API_CALL(cursor, session, cbt->btree); WT_ERR(__cursor_checkkey(cursor)); + time_start = __wt_clock(session); WT_ERR(__wt_btcur_remove(cbt)); + time_stop = __wt_clock(session); + __wt_stat_usecs_hist_incr_opwrite(session, + WT_CLOCKDIFF_US(time_stop, time_start)); /* * Remove with a search-key is fire-and-forget, no position and no key. - * Remove starting from a position maintains the position and a key. - * We don't know which it was at this layer, so can only assert the key - * is not set at all, or internal. There's never a value. + * Remove starting from a position maintains the position and a key, + * but the key can end up being internal, external, or not set, there's + * nothing to assert. There's never a value. */ - WT_ASSERT(session, - F_MASK(cursor, WT_CURSTD_KEY_SET) == 0 || - F_MASK(cursor, WT_CURSTD_KEY_SET) == WT_CURSTD_KEY_INT); WT_ASSERT(session, F_MASK(cursor, WT_CURSTD_VALUE_SET) == 0); err: CURSOR_UPDATE_API_END(session, ret); @@ -398,6 +437,7 @@ __curfile_reserve(WT_CURSOR *cursor) * each successful reserve operation. */ WT_ASSERT(session, + F_ISSET(cbt, WT_CBT_ACTIVE) && F_MASK(cursor, WT_CURSTD_KEY_SET) == WT_CURSTD_KEY_INT); WT_ASSERT(session, F_MASK(cursor, WT_CURSTD_VALUE_SET) == 0); @@ -554,11 +594,12 @@ __curfile_create(WT_SESSION_IMPL *session, __wt_btcur_open(cbt); /* - * WT_CURSOR.modify supported on 'u' value formats, but the fast-path - * through the btree code requires log file format changes, it's not - * available in all versions. + * WT_CURSOR.modify supported on 'S' and 'u' value formats, but the + * fast-path through the btree code requires log file format changes, + * it's not available in all versions. */ - if (WT_STREQ(cursor->value_format, "u") && + if ((WT_STREQ(cursor->value_format, "S") || + WT_STREQ(cursor->value_format, "u")) && S2C(session)->compat_major >= WT_LOG_V2) cursor->modify = __curfile_modify; diff --git a/src/third_party/wiredtiger/src/cursor/cur_index.c b/src/third_party/wiredtiger/src/cursor/cur_index.c index 6b1ef6312e8..eb82effe702 100644 --- a/src/third_party/wiredtiger/src/cursor/cur_index.c +++ b/src/third_party/wiredtiger/src/cursor/cur_index.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/cursor/cur_join.c b/src/third_party/wiredtiger/src/cursor/cur_join.c index bcd3943122d..699774809f5 100644 --- a/src/third_party/wiredtiger/src/cursor/cur_join.c +++ b/src/third_party/wiredtiger/src/cursor/cur_join.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/cursor/cur_json.c b/src/third_party/wiredtiger/src/cursor/cur_json.c index 2b46dc39622..87f8899d9c8 100644 --- a/src/third_party/wiredtiger/src/cursor/cur_json.c +++ b/src/third_party/wiredtiger/src/cursor/cur_json.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -204,14 +204,13 @@ __json_struct_size(WT_SESSION_IMPL *session, const void *buffer, WT_RET( __json_unpack_put(session, &pv, NULL, 0, &name, &result)); } - if (ret == WT_NOTFOUND) - ret = 0; + WT_RET_NOTFOUND_OK(ret); /* Be paranoid - __pack_write should never overflow. */ WT_ASSERT(session, p <= end); *presult = result; - return (ret); + return (0); } /* @@ -258,15 +257,14 @@ __json_struct_unpackv(WT_SESSION_IMPL *session, jbuf += jsize; jbufsize -= jsize; } - if (ret == WT_NOTFOUND) - ret = 0; + WT_RET_NOTFOUND_OK(ret); /* Be paranoid - __unpack_read should never overflow. */ WT_ASSERT(session, p <= end); WT_ASSERT(session, jbufsize == 1); - return (ret); + return (0); } /* @@ -754,13 +752,11 @@ __json_pack_struct(WT_SESSION_IMPL *session, void *buffer, size_t size, WT_RET(__pack_write(session, &pv, &p, (size_t)(end - p))); multi = true; } + WT_RET_NOTFOUND_OK(ret); /* Be paranoid - __pack_write should never overflow. */ WT_ASSERT(session, p <= end); - if (ret != WT_NOTFOUND) - return (ret); - return (0); } @@ -778,16 +774,17 @@ __json_pack_size( { WT_CONFIG_ITEM name; WT_DECL_PACK_VALUE(pv); + WT_DECL_RET; WT_PACK pack; WT_PACK_NAME packname; - size_t toksize, total; + size_t toksize, v; const char *tokstart; bool multi; __pack_name_init(session, names, iskey, &packname); multi = false; WT_RET(__pack_init(session, &pack, fmt)); - for (total = 0; __pack_next(&pack, &pv) == 0;) { + for (*sizep = 0; (ret = __pack_next(&pack, &pv)) == 0;) { if (multi) JSON_EXPECT_TOKEN(session, jstr, ','); JSON_EXPECT_TOKEN_GET(session, jstr, 's', tokstart, toksize); @@ -799,13 +796,15 @@ __json_pack_size( iskey ? "key" : "value", (int)name.len, name.str); JSON_EXPECT_TOKEN(session, jstr, ':'); WT_PACK_JSON_GET(session, pv, jstr); - total += __pack_size(session, &pv); + WT_RET(__pack_size(session, &pv, &v)); + *sizep += v; multi = true; } + WT_RET_NOTFOUND_OK(ret); + /* check end of string */ JSON_EXPECT_TOKEN(session, jstr, 0); - *sizep = total; return (0); } diff --git a/src/third_party/wiredtiger/src/cursor/cur_log.c b/src/third_party/wiredtiger/src/cursor/cur_log.c index bcfac1bfe3e..8f664f4c2cd 100644 --- a/src/third_party/wiredtiger/src/cursor/cur_log.c +++ b/src/third_party/wiredtiger/src/cursor/cur_log.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/cursor/cur_metadata.c b/src/third_party/wiredtiger/src/cursor/cur_metadata.c index d9aeed1fccd..2536de8c455 100644 --- a/src/third_party/wiredtiger/src/cursor/cur_metadata.c +++ b/src/third_party/wiredtiger/src/cursor/cur_metadata.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/cursor/cur_stat.c b/src/third_party/wiredtiger/src/cursor/cur_stat.c index f6228b9c853..39921e11edc 100644 --- a/src/third_party/wiredtiger/src/cursor/cur_stat.c +++ b/src/third_party/wiredtiger/src/cursor/cur_stat.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/cursor/cur_std.c b/src/third_party/wiredtiger/src/cursor/cur_std.c index 9cfa3203aec..d7b23be75d6 100644 --- a/src/third_party/wiredtiger/src/cursor/cur_std.c +++ b/src/third_party/wiredtiger/src/cursor/cur_std.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -96,9 +96,17 @@ __wt_cursor_equals_notsup(WT_CURSOR *cursor, WT_CURSOR *other, int *equalp) int __wt_cursor_modify_notsup(WT_CURSOR *cursor, WT_MODIFY *entries, int nentries) { + WT_SESSION_IMPL *session; + WT_UNUSED(entries); WT_UNUSED(nentries); + if (cursor->value_format != NULL && strlen(cursor->value_format) != 0) { + session = (WT_SESSION_IMPL *)cursor->session; + WT_RET_MSG(session, ENOTSUP, + "WT_CURSOR.modify only supported for 'S' and 'u' value " + "formats"); + } return (__wt_cursor_notsup(cursor)); } @@ -626,8 +634,7 @@ __cursor_modify(WT_CURSOR *cursor, WT_MODIFY *entries, int nentries) /* Get the current value, apply the modifications. */ WT_ERR(cursor->search(cursor)); - WT_ERR(__wt_modify_apply_api( - session, &cursor->value, entries, nentries)); + WT_ERR(__wt_modify_apply_api(session, cursor, entries, nentries)); /* We know both key and value are set, "overwrite" doesn't matter. */ ret = cursor->update(cursor); @@ -814,10 +821,11 @@ __wt_cursor_init(WT_CURSOR *cursor, F_SET(cursor, WT_CURSTD_RAW); /* - * WT_CURSOR.modify supported on 'u' value formats, but may have been - * already initialized (file cursors have a faster implementation). + * WT_CURSOR.modify supported on 'S' and 'u' value formats, but may have + * been already initialized (file cursors have a faster implementation). */ - if (WT_STREQ(cursor->value_format, "u") && + if ((WT_STREQ(cursor->value_format, "S") || + WT_STREQ(cursor->value_format, "u")) && cursor->modify == __wt_cursor_modify_notsup) cursor->modify = __cursor_modify; diff --git a/src/third_party/wiredtiger/src/cursor/cur_table.c b/src/third_party/wiredtiger/src/cursor/cur_table.c index 429f75208f2..ab2382946bd 100644 --- a/src/third_party/wiredtiger/src/cursor/cur_table.c +++ b/src/third_party/wiredtiger/src/cursor/cur_table.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -547,7 +547,7 @@ __curtable_insert(WT_CURSOR *cursor) */ F_CLR(primary, WT_CURSTD_KEY_SET | WT_CURSTD_VALUE_SET); if (F_ISSET(primary, WT_CURSTD_APPEND)) - F_SET(primary, WT_CURSTD_KEY_INT); + F_SET(primary, WT_CURSTD_KEY_EXT); err: CURSOR_UPDATE_API_END(session, ret); return (ret); diff --git a/src/third_party/wiredtiger/src/docs/build-javadoc.sh b/src/third_party/wiredtiger/src/docs/build-javadoc.sh index 69cb1186467..29dce173261 100755 --- a/src/third_party/wiredtiger/src/docs/build-javadoc.sh +++ b/src/third_party/wiredtiger/src/docs/build-javadoc.sh @@ -8,5 +8,5 @@ CLASSPATH=$THRIFT_HOME/libthrift.jar:$SLF4J_JAR javadoc -public -d $DOCS/java \ -stylesheetfile $DOCS/style/javadoc.css \ -use -link http://java.sun.com/j2se/1.5.0/docs/api/ \ -header '<b>WiredTiger API</b><br><font size="-1"> version '$WT_VERSION'</font>' \ - -windowtitle 'WiredTiger Java API' -bottom '<font size=1>Copyright (c) 2008-2017 MongoDB, Inc. All rights reserved.</font>' \ + -windowtitle 'WiredTiger Java API' -bottom '<font size=1>Copyright (c) 2008-2018 MongoDB, Inc. All rights reserved.</font>' \ com.wiredtiger com.wiredtiger.util diff --git a/src/third_party/wiredtiger/src/docs/build-posix.dox b/src/third_party/wiredtiger/src/docs/build-posix.dox index 3e7f8f37acd..7192a9c2322 100644 --- a/src/third_party/wiredtiger/src/docs/build-posix.dox +++ b/src/third_party/wiredtiger/src/docs/build-posix.dox @@ -142,10 +142,6 @@ Build the WiredTiger <a href="http://www.python.org">Python</a> API. Configure WiredTiger for <a href="http://code.google.com/p/snappy/">snappy</a> compression; see @ref compression for more information. -@par \c --enable-verbose -Configure WiredTiger to support the \c verbose configuration string to -::wiredtiger_open. - @par \c --enable-zlib Configure WiredTiger for <a href="http://www.zlib.net/">zlib</a> compression; see @ref compression for more information. diff --git a/src/third_party/wiredtiger/src/docs/build-windows.dox b/src/third_party/wiredtiger/src/docs/build-windows.dox index 79df6b9712d..dcd16e30ad1 100644 --- a/src/third_party/wiredtiger/src/docs/build-windows.dox +++ b/src/third_party/wiredtiger/src/docs/build-windows.dox @@ -89,6 +89,10 @@ Configure WiredTiger to sleep and wait for a debugger to attach on failure. Configure WiredTiger to perform various run-time diagnostic tests. <b>DO NOT</b> configure this option in production environments. +@par \c --enable-java +Build the WiredTiger <a href="http://www.java.com">Java</a> API, +requires path to swig.exe and path to Java JDK directory, comma separated. + @par \c --enable-python Build the WiredTiger <a href="http://www.python.org">Python</a> API, requires path to swig.exe on Windows. @@ -97,10 +101,6 @@ requires path to swig.exe on Windows. Configure WiredTiger for <a href="http://code.google.com/p/snappy/">snappy</a> compression; see @ref compression for more information. -@par \c --enable-verbose -Configure WiredTiger to support the \c verbose configuration string to -::wiredtiger_open. - @par \c --enable-zlib Configure WiredTiger for <a href="http://www.zlib.net/">zlib</a> compression; see @ref compression for more information. diff --git a/src/third_party/wiredtiger/src/docs/error-handling.dox b/src/third_party/wiredtiger/src/docs/error-handling.dox index 7b7c0cd03d5..8e866b8c123 100644 --- a/src/third_party/wiredtiger/src/docs/error-handling.dox +++ b/src/third_party/wiredtiger/src/docs/error-handling.dox @@ -87,10 +87,11 @@ standard API. Note that ::wiredtiger_strerror is not thread-safe. @m_if{c} -@section error_handling_event Error handling using the WT_EVENT_HANDLER +@section event_message_handling Message handling using the WT_EVENT_HANDLER -Specific error handling can be configured by passing an implementation -of WT_EVENT_HANDLER to ::wiredtiger_open or WT_CONNECTION::open_session. +Specific error and other message handling can be configured by passing an +implementation of WT_EVENT_HANDLER to ::wiredtiger_open or +WT_CONNECTION::open_session. For example, both informational and error messages might be passed to an application-specific logging function that added a timestamp and logged diff --git a/src/third_party/wiredtiger/src/docs/operation_tracking.dox b/src/third_party/wiredtiger/src/docs/operation_tracking.dox new file mode 100644 index 00000000000..b49ec1aa91a --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/operation_tracking.dox @@ -0,0 +1,39 @@ +/*! @m_page{{c,java},operation_tracking,Track function calls} + +When this option is enabled, WiredTiger tracks all API calls as well as certain +functions that are deemed important for performance, such as eviction-related +functions. Tracking is performed by generating a log record when the WiredTiger +library enters and exits a tracked function. A log record contains a function +name and its timestamp. For the purposes of efficiency, the timestamp on most +platforms is obtained directly from the CPU register, so the units of time +correspond to processor clock ticks. + +By default, the log files are produced in the sub-directory +`operation_tracking` of the database home directory. To change the +destination directory, use the `path` sub-option of the +`operation_tracking` option. + +Operation tracking produces two kinds of files: the binary log files +and the map file. The map file is needed to convert the binary logs +into text. The map file contains the mapping between numeric function +ids used in the log files and the human-readable function name. The +map file is called `optrack-map.<pid>`, where `pid` is the id of +the process that generated the map file. Log files have names that +look like `optrack.<pid>.<tid>`, where `tid` is the numeric id of the +WiredTiger session that generated the log file. + +To convert binary log files to text, run the wt_optrack_decode.py script in the +tools directory of the WiredTiger distribution. The script takes as arguments +the list of log files and the name of the map file with the `-m` option. Suppose +you would like to convert the log files generated by a process with `pid +1234`. Then you would run the script as follows: + +``` +python wt_optrack_decode.py -m optrack-map.1234 optrack.1234.* +``` + +Enabling operation tracking may generate performance overhead if the tracked +functions are executed frequently. Please be aware of this consequence and +measure your performance before deciding whether to enable operation tracking. + +*/ diff --git a/src/third_party/wiredtiger/src/docs/programming.dox b/src/third_party/wiredtiger/src/docs/programming.dox index 7274fb8078a..3ddb0c376c5 100644 --- a/src/third_party/wiredtiger/src/docs/programming.dox +++ b/src/third_party/wiredtiger/src/docs/programming.dox @@ -44,6 +44,7 @@ each of which is ordered by one or more columns. - @subpage in_memory - @subpage cursor_join - @subpage cursor_log +- @subpage operation_tracking - @ref transaction_named_snapshots - @subpage rebalance - @subpage shared_cache diff --git a/src/third_party/wiredtiger/src/docs/spell.ok b/src/third_party/wiredtiger/src/docs/spell.ok index 112e5e815fe..d5347709248 100644 --- a/src/third_party/wiredtiger/src/docs/spell.ok +++ b/src/third_party/wiredtiger/src/docs/spell.ok @@ -44,6 +44,7 @@ Google's HyperDex HyperLevelDB IEC +JDK JavaScript KMS LD @@ -368,6 +369,7 @@ objectsin ol oltp oob +optrack optype os outlier @@ -382,6 +384,7 @@ pget php plantuml png +pid posix pre prepends @@ -462,6 +465,7 @@ substring superset sys sz +tid tRuE tablename tcl diff --git a/src/third_party/wiredtiger/src/docs/style/footer.html b/src/third_party/wiredtiger/src/docs/style/footer.html index 12d25422f89..755affdbb7d 100644 --- a/src/third_party/wiredtiger/src/docs/style/footer.html +++ b/src/third_party/wiredtiger/src/docs/style/footer.html @@ -3,13 +3,13 @@ <div id="nav-path" class="navpath"><!-- id is needed for treeview function! --> <ul> $navpath - <li class="footer">Copyright (c) 2008-2017 MongoDB, Inc. All rights reserved. Contact <a href="mailto:info@wiredtiger.com">info@wiredtiger.com</a> for more information.</li> + <li class="footer">Copyright (c) 2008-2018 MongoDB, Inc. All rights reserved. Contact <a href="mailto:info@wiredtiger.com">info@wiredtiger.com</a> for more information.</li> </ul> </div> <!--END GENERATE_TREEVIEW--> <!--BEGIN !GENERATE_TREEVIEW--> <hr class="footer"/><address class="footer"><small> -Copyright (c) 2008-2017 MongoDB, Inc. All rights reserved. Contact <a href="mailto:info@wiredtiger.com">info@wiredtiger.com</a> for more information. +Copyright (c) 2008-2018 MongoDB, Inc. All rights reserved. Contact <a href="mailto:info@wiredtiger.com">info@wiredtiger.com</a> for more information. </small></address> <!--END !GENERATE_TREEVIEW--> </body> diff --git a/src/third_party/wiredtiger/src/docs/tools/doxfilter.py b/src/third_party/wiredtiger/src/docs/tools/doxfilter.py index 301142269c3..3a9f0f6836a 100755 --- a/src/third_party/wiredtiger/src/docs/tools/doxfilter.py +++ b/src/third_party/wiredtiger/src/docs/tools/doxfilter.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/src/docs/tools/fixlinks.py b/src/third_party/wiredtiger/src/docs/tools/fixlinks.py index 1887665d5be..009c1a5aa30 100755 --- a/src/third_party/wiredtiger/src/docs/tools/fixlinks.py +++ b/src/third_party/wiredtiger/src/docs/tools/fixlinks.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/src/docs/top/main.dox b/src/third_party/wiredtiger/src/docs/top/main.dox index 1bfb623c0a0..e4de22ff042 100644 --- a/src/third_party/wiredtiger/src/docs/top/main.dox +++ b/src/third_party/wiredtiger/src/docs/top/main.dox @@ -6,12 +6,12 @@ WiredTiger is an high performance, scalable, production quality, NoSQL, @section releases Releases <table> -@row{<b>WiredTiger 2.9.3</b> (current), +@row{<b>WiredTiger 3.0.0</b> (current), + <a href="releases/wiredtiger-3.0.0.tar.bz2"><b>[Release package]</b></a>, + <a href="3.0.0/index.html"><b>[Documentation]</b></a>} +@row{<b>WiredTiger 2.9.3</b> (previous), <a href="releases/wiredtiger-2.9.3.tar.bz2"><b>[Release package]</b></a>, <a href="2.9.3/index.html"><b>[Documentation]</b></a>} -@row{<b>WiredTiger 2.9.2</b> (previous), - <a href="releases/wiredtiger-2.9.2.tar.bz2"><b>[Release package]</b></a>, - <a href="2.9.2/index.html"><b>[Documentation]</b></a>} @row{<b>Development branch</b>, <a href="https://github.com/wiredtiger/wiredtiger"><b>[Source code]</b></a>, <a href="develop/index.html"><b>[Documentation]</b></a>} diff --git a/src/third_party/wiredtiger/src/docs/transactions.dox b/src/third_party/wiredtiger/src/docs/transactions.dox index f66923aac3f..d27e796b8e9 100644 --- a/src/third_party/wiredtiger/src/docs/transactions.dox +++ b/src/third_party/wiredtiger/src/docs/transactions.dox @@ -201,8 +201,7 @@ been used. This is enforced by assertions in diagnostic builds, if applications violate this rule, data consistency can be violated. The commits to a particular data item must be performed in timestamp order. -Again, this is only checked in diagnostic builds and if applications violate -this rule, data consistency can be violated. +If applications violate this rule, data consistency can be violated. @subsection Timestamp support in the extension API diff --git a/src/third_party/wiredtiger/src/docs/upgrading.dox b/src/third_party/wiredtiger/src/docs/upgrading.dox index 09cafbf480d..2e4990e8a33 100644 --- a/src/third_party/wiredtiger/src/docs/upgrading.dox +++ b/src/third_party/wiredtiger/src/docs/upgrading.dox @@ -28,6 +28,16 @@ The performance visualization tool \c wtstats has been removed and is no longer supported. </dd> +<dt>::wiredtiger_open cache configuration changes</dt> +<dd> +The cache configuration options \c eviction_checkpoint_target, \c +eviction_dirty_target, \c eviction_dirty_trigger, \c eviction_target and \c +eviction_trigger have changed. The options can now take absolute size. It would +be a percentage of the cache size if the value is within the range of 0 to 100 +or an absolute size when greater than 100. This API change is compatible with +existing usage. +</dd> + </dl><hr> @section version_292 Upgrading to Version 2.9.2 <dl> diff --git a/src/third_party/wiredtiger/src/evict/evict_file.c b/src/third_party/wiredtiger/src/evict/evict_file.c index 13e2823d234..f2b632ca06d 100644 --- a/src/third_party/wiredtiger/src/evict/evict_file.c +++ b/src/third_party/wiredtiger/src/evict/evict_file.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/evict/evict_lru.c b/src/third_party/wiredtiger/src/evict/evict_lru.c index 343d29d47cf..39c84764070 100644 --- a/src/third_party/wiredtiger/src/evict/evict_lru.c +++ b/src/third_party/wiredtiger/src/evict/evict_lru.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -230,7 +230,6 @@ __wt_evict_server_wake(WT_SESSION_IMPL *session) conn = S2C(session); cache = conn->cache; -#ifdef HAVE_VERBOSE if (WT_VERBOSE_ISSET(session, WT_VERB_EVICTSERVER)) { uint64_t bytes_inuse, bytes_max; @@ -244,7 +243,6 @@ __wt_evict_server_wake(WT_SESSION_IMPL *session) bytes_inuse <= bytes_max ? "<=" : ">", bytes_max / WT_MEGABYTE); } -#endif __wt_cond_signal(session, cache->evict_cond); } @@ -371,9 +369,7 @@ err: WT_PANIC_MSG(session, ret, "cache eviction thread error"); static int __evict_server(WT_SESSION_IMPL *session, bool *did_work) { -#if defined(HAVE_DIAGNOSTIC) || defined(HAVE_VERBOSE) struct timespec now; -#endif WT_CACHE *cache; WT_CONNECTION_IMPL *conn; WT_DECL_RET; @@ -421,13 +417,19 @@ __evict_server(WT_SESSION_IMPL *session, bool *did_work) /* Eviction is stuck, check if we have made progress. */ if (*did_work) { -#if defined(HAVE_DIAGNOSTIC) || defined(HAVE_VERBOSE) - __wt_epoch(session, &cache->stuck_time); +#if !defined(HAVE_DIAGNOSTIC) + /* Need verbose check only if not in diagnostic build */ + if (WT_VERBOSE_ISSET(session, WT_VERB_EVICT_STUCK)) #endif + __wt_epoch(session, &cache->stuck_time); return (0); } -#if defined(HAVE_DIAGNOSTIC) || defined(HAVE_VERBOSE) +#if !defined(HAVE_DIAGNOSTIC) + /* Need verbose check only if not in diagnostic build */ + if (!WT_VERBOSE_ISSET(session, WT_VERB_EVICT_STUCK)) + return (0); +#endif /* * If we're stuck for 5 minutes in diagnostic mode, or the verbose * evict_stuck flag is configured, log the cache and transaction state. @@ -447,11 +449,10 @@ __evict_server(WT_SESSION_IMPL *session, bool *did_work) #if defined(HAVE_DIAGNOSTIC) __wt_err(session, ETIMEDOUT, "Cache stuck for too long, giving up"); - ret = ETIMEDOUT; - WT_TRET(__wt_verbose_dump_txn(session)); - WT_TRET(__wt_verbose_dump_cache(session)); - return (ret); -#elif defined(HAVE_VERBOSE) + WT_RET(__wt_verbose_dump_txn(session)); + WT_RET(__wt_verbose_dump_cache(session)); + return (ETIMEDOUT); +#else if (WT_VERBOSE_ISSET(session, WT_VERB_EVICT_STUCK)) { WT_RET(__wt_verbose_dump_txn(session)); WT_RET(__wt_verbose_dump_cache(session)); @@ -461,7 +462,6 @@ __evict_server(WT_SESSION_IMPL *session, bool *did_work) } #endif } -#endif return (0); } @@ -492,12 +492,14 @@ __wt_evict_create(WT_SESSION_IMPL *session) session_flags, __wt_evict_thread_chk, __wt_evict_thread_run, __wt_evict_thread_stop)); -#if defined(HAVE_DIAGNOSTIC) || defined(HAVE_VERBOSE) /* * Ensure the cache stuck timer is initialized when starting eviction. */ - __wt_epoch(session, &conn->cache->stuck_time); +#if !defined(HAVE_DIAGNOSTIC) + /* Need verbose check only if not in diagnostic build */ + if (WT_VERBOSE_ISSET(session, WT_VERB_EVICTSERVER)) #endif + __wt_epoch(session, &conn->cache->stuck_time); /* * Allow queues to be populated now that the eviction threads @@ -555,11 +557,17 @@ __evict_update_work(WT_SESSION_IMPL *session) WT_BTREE *las_tree; WT_CACHE *cache; WT_CONNECTION_IMPL *conn; + double dirty_target, dirty_trigger, target, trigger; uint64_t bytes_inuse, bytes_max, dirty_inuse; conn = S2C(session); cache = conn->cache; + dirty_target = cache->eviction_dirty_target; + dirty_trigger = cache->eviction_dirty_trigger; + target = cache->eviction_target; + trigger = cache->eviction_trigger; + /* Clear previous state. */ cache->flags = 0; @@ -587,13 +595,13 @@ __evict_update_work(WT_SESSION_IMPL *session) bytes_inuse = __wt_cache_bytes_inuse(cache); if (__wt_eviction_clean_needed(session, NULL)) F_SET(cache, WT_CACHE_EVICT_CLEAN | WT_CACHE_EVICT_CLEAN_HARD); - else if (bytes_inuse > (cache->eviction_target * bytes_max) / 100) + else if (bytes_inuse > (target * bytes_max) / 100) F_SET(cache, WT_CACHE_EVICT_CLEAN); dirty_inuse = __wt_cache_dirty_leaf_inuse(cache); if (__wt_eviction_dirty_needed(session, NULL)) F_SET(cache, WT_CACHE_EVICT_DIRTY | WT_CACHE_EVICT_DIRTY_HARD); - else if (dirty_inuse > (cache->eviction_dirty_target * bytes_max) / 100) + else if (dirty_inuse > (uint64_t)(dirty_target * bytes_max) / 100) F_SET(cache, WT_CACHE_EVICT_DIRTY); /* @@ -608,10 +616,9 @@ __evict_update_work(WT_SESSION_IMPL *session) * Scrub dirty pages and keep them in cache if we are less than half * way to the clean or dirty trigger. */ - if (bytes_inuse < ((cache->eviction_target + cache->eviction_trigger) * - bytes_max) / 200 && dirty_inuse < (uint64_t) - ((cache->eviction_dirty_target + cache->eviction_dirty_trigger) * - bytes_max) / 200) + if (bytes_inuse < (uint64_t)((target + trigger) * bytes_max) / 200 && + dirty_inuse < + (uint64_t)((dirty_target + dirty_trigger) * bytes_max) / 200) F_SET(cache, WT_CACHE_EVICT_SCRUB); /* @@ -624,9 +631,8 @@ __evict_update_work(WT_SESSION_IMPL *session) if (!F_ISSET(conn, WT_CONN_EVICTION_NO_LOOKASIDE) && (__wt_cache_stuck(session) || (__wt_cache_lookaside_score(cache) > 80 && - dirty_inuse > (uint64_t) - ((cache->eviction_dirty_target + cache->eviction_dirty_trigger) * - bytes_max) / 200))) + dirty_inuse > + (uint64_t)((dirty_target + dirty_trigger) * bytes_max) / 200))) F_SET(cache, WT_CACHE_EVICT_LOOKASIDE); /* @@ -651,27 +657,27 @@ __evict_update_work(WT_SESSION_IMPL *session) static int __evict_pass(WT_SESSION_IMPL *session) { - struct timespec now, prev; WT_CACHE *cache; WT_CONNECTION_IMPL *conn; WT_TXN_GLOBAL *txn_global; uint64_t eviction_progress, oldest_id, prev_oldest_id; + uint64_t time_now, time_prev; u_int loop; conn = S2C(session); cache = conn->cache; txn_global = &conn->txn_global; + time_prev = 0; /* [-Wconditional-uninitialized] */ /* Track whether pages are being evicted and progress is made. */ eviction_progress = cache->eviction_progress; prev_oldest_id = txn_global->oldest_id; - WT_CLEAR(prev); /* Evict pages from the cache. */ for (loop = 0; cache->pass_intr == 0; loop++) { - __wt_epoch(session, &now); + time_now = __wt_clock(session); if (loop == 0) - prev = now; + time_prev = time_now; __evict_tune_workers(session); /* @@ -739,7 +745,7 @@ __evict_pass(WT_SESSION_IMPL *session) * transactions and writing updates to the lookaside table. */ if (eviction_progress == cache->eviction_progress) { - if (WT_TIMEDIFF_MS(now, prev) >= 20 && + if (WT_CLOCKDIFF_MS(time_now, time_prev) >= 20 && F_ISSET(cache, WT_CACHE_EVICT_CLEAN_HARD | WT_CACHE_EVICT_DIRTY_HARD)) { if (cache->evict_aggressive_score < 100) @@ -749,7 +755,7 @@ __evict_pass(WT_SESSION_IMPL *session) txn_global->current != oldest_id && cache->evict_aggressive_score < 100) ++cache->evict_aggressive_score; - prev = now; + time_prev = time_now; prev_oldest_id = oldest_id; } @@ -779,12 +785,11 @@ __evict_pass(WT_SESSION_IMPL *session) __wt_verbose(session, WT_VERB_EVICTSERVER, "%s", "unable to reach eviction goal"); break; - } else { - if (cache->evict_aggressive_score > 0) - --cache->evict_aggressive_score; - loop = 0; - eviction_progress = cache->eviction_progress; } + if (cache->evict_aggressive_score > 0) + --cache->evict_aggressive_score; + loop = 0; + eviction_progress = cache->eviction_progress; } return (0); } @@ -1158,7 +1163,9 @@ __evict_lru_pages(WT_SESSION_IMPL *session, bool is_server) { WT_CONNECTION_IMPL *conn; WT_DECL_RET; + WT_TRACK_OP_DECL; + WT_TRACK_OP_INIT(session); conn = S2C(session); /* @@ -1175,6 +1182,7 @@ __evict_lru_pages(WT_SESSION_IMPL *session, bool is_server) __wt_cond_wait( session, conn->evict_threads.wait_cond, 10000, NULL); + WT_TRACK_OP_END(session); return (ret == WT_NOTFOUND ? 0 : ret); } @@ -1188,9 +1196,11 @@ __evict_lru_walk(WT_SESSION_IMPL *session) WT_CACHE *cache; WT_DECL_RET; WT_EVICT_QUEUE *queue, *other_queue; + WT_TRACK_OP_DECL; uint64_t read_gen_oldest; uint32_t candidates, entries; + WT_TRACK_OP_INIT(session); cache = S2C(session)->cache; /* Age out the score of how much the queue has been empty recently. */ @@ -1212,12 +1222,14 @@ __evict_lru_walk(WT_SESSION_IMPL *session) */ if (__evict_queue_full(queue) && cache->evict_empty_score < WT_EVICT_SCORE_CUTOFF) - return (0); + goto err; /* Get some more pages to consider for eviction. */ - if ((ret = __evict_walk(cache->walk_session, queue)) == EBUSY) - return (0); /* An interrupt was requested, give up. */ - WT_RET_NOTFOUND_OK(ret); + if ((ret = __evict_walk(cache->walk_session, queue)) == EBUSY) { + ret = 0; + goto err; /* An interrupt was requested, give up. */ + } + WT_ERR_NOTFOUND_OK(ret); /* * If the queue we are filling is empty, pages are being requested @@ -1271,7 +1283,7 @@ __evict_lru_walk(WT_SESSION_IMPL *session) queue->evict_candidates = 0; queue->evict_current = NULL; __wt_spin_unlock(session, &queue->evict_lock); - return (0); + goto err; } /* Decide how many of the candidates we're going to try and evict. */ @@ -1330,7 +1342,8 @@ __evict_lru_walk(WT_SESSION_IMPL *session) */ __wt_cond_signal(session, S2C(session)->evict_threads.wait_cond); - return (0); +err: WT_TRACK_OP_END(session); + return (ret); } /* @@ -1345,9 +1358,12 @@ __evict_walk(WT_SESSION_IMPL *session, WT_EVICT_QUEUE *queue) WT_CONNECTION_IMPL *conn; WT_DATA_HANDLE *dhandle; WT_DECL_RET; + WT_TRACK_OP_DECL; u_int max_entries, retries, slot, start_slot, total_candidates; bool dhandle_locked, incr; + WT_TRACK_OP_INIT(session); + conn = S2C(session); cache = conn->cache; btree = NULL; @@ -1527,9 +1543,10 @@ err: if (dhandle_locked) * let our caller know. */ if (queue->evict_entries == slot && cache->pass_intr == 0) - return (WT_NOTFOUND); + ret = WT_NOTFOUND; queue->evict_entries = slot; + WT_TRACK_OP_END(session); return (ret); } @@ -2036,12 +2053,13 @@ fast: /* If the page can't be evicted, give up. */ * point keeping a page pinned, since it may be the only candidate in * an idle tree. * - * If we land on a page requiring forced eviction, move on to the next - * page: we want this page evicted as quickly as possible. + * If we land on a page requiring forced eviction, or that isn't an + * ordinary in-memory page (e.g., WT_REF_LIMBO), move until we find an + * ordinary page: we should not prevent exclusive access to the page + * until the next walk. */ if (ref != NULL) { if (__wt_ref_is_root(ref) || evict == start || give_up || - WT_READGEN_EVICT_SOON(ref->page->read_gen) || ref->page->memory_footprint >= btree->splitmempage) { if (restarts == 0) WT_STAT_CONN_INCR( @@ -2049,9 +2067,11 @@ fast: /* If the page can't be evicted, give up. */ WT_RET(__wt_page_release( cache->walk_session, ref, walk_flags)); ref = NULL; - } else if (WT_READGEN_EVICT_SOON(ref->page->read_gen)) - WT_RET_NOTFOUND_OK(__wt_tree_walk_count( - session, &ref, &refs_walked, walk_flags)); + } else + while (ref != NULL && (ref->state != WT_REF_MEM || + WT_READGEN_EVICT_SOON(ref->page->read_gen))) + WT_RET_NOTFOUND_OK(__wt_tree_walk_count( + session, &ref, &refs_walked, walk_flags)); btree->evict_ref = ref; } @@ -2075,7 +2095,7 @@ __evict_get_ref( WT_CACHE *cache; WT_EVICT_ENTRY *evict; WT_EVICT_QUEUE *queue, *other_queue, *urgent_queue; - uint32_t candidates; + uint32_t candidates, previous_state; bool is_app, server_only, urgent_ok; *btreep = NULL; @@ -2200,8 +2220,10 @@ __evict_get_ref( * multiple attempts to evict it. For pages that are already * being evicted, this operation will fail and we will move on. */ - if (!__wt_atomic_casv32( - &evict->ref->state, WT_REF_MEM, WT_REF_LOCKED)) { + if (((previous_state = evict->ref->state) != WT_REF_MEM && + previous_state != WT_REF_LIMBO) || + !__wt_atomic_casv32( + &evict->ref->state, previous_state, WT_REF_LOCKED)) { __evict_list_clear(session, evict); continue; } @@ -2242,18 +2264,22 @@ __evict_get_ref( static int __evict_page(WT_SESSION_IMPL *session, bool is_server) { - struct timespec enter, leave; WT_BTREE *btree; WT_CACHE *cache; WT_DECL_RET; WT_REF *ref; + WT_TRACK_OP_DECL; + uint64_t time_start, time_stop; bool app_timer; + WT_TRACK_OP_INIT(session); + WT_RET(__evict_get_ref(session, is_server, &btree, &ref)); WT_ASSERT(session, ref->state == WT_REF_LOCKED); app_timer = false; cache = S2C(session)->cache; + time_start = time_stop = 0; /* * An internal session flags either the server itself or an eviction @@ -2272,7 +2298,7 @@ __evict_page(WT_SESSION_IMPL *session, bool is_server) cache->app_evicts++; if (WT_STAT_ENABLED(session)) { app_timer = true; - __wt_epoch(session, &enter); + time_start = __wt_clock(session); } } @@ -2292,10 +2318,12 @@ __evict_page(WT_SESSION_IMPL *session, bool is_server) (void)__wt_atomic_subv32(&btree->evict_busy, 1); if (app_timer) { - __wt_epoch(session, &leave); + time_stop = __wt_clock(session); WT_STAT_CONN_INCRV(session, - application_evict_time, WT_TIMEDIFF_US(leave, enter)); + application_evict_time, + WT_CLOCKDIFF_US(time_stop, time_start)); } + WT_TRACK_OP_END(session); return (ret); } @@ -2306,19 +2334,22 @@ __evict_page(WT_SESSION_IMPL *session, bool is_server) */ int __wt_cache_eviction_worker( - WT_SESSION_IMPL *session, bool busy, bool readonly, u_int pct_full) + WT_SESSION_IMPL *session, bool busy, bool readonly, double pct_full) { - struct timespec enter, leave; WT_CACHE *cache; WT_CONNECTION_IMPL *conn; WT_DECL_RET; + WT_TRACK_OP_DECL; WT_TXN_GLOBAL *txn_global; WT_TXN_STATE *txn_state; - uint64_t initial_progress, max_progress; + uint64_t initial_progress, max_progress, time_start, time_stop; bool timer; + WT_TRACK_OP_INIT(session); + conn = S2C(session); cache = conn->cache; + time_start = time_stop = 0; txn_global = &conn->txn_global; txn_state = WT_SESSION_TXN_STATE(session); @@ -2326,11 +2357,8 @@ __wt_cache_eviction_worker( * It is not safe to proceed if the eviction server threads aren't * setup yet. */ - if (!conn->evict_server_running) - return (0); - - if (busy && pct_full < 100) - return (0); + if (!conn->evict_server_running || (busy && pct_full < 100.0)) + goto done; /* Wake the eviction server if we need to do work. */ __wt_evict_server_wake(session); @@ -2339,7 +2367,7 @@ __wt_cache_eviction_worker( timer = WT_STAT_ENABLED(session) && !F_ISSET(session, WT_SESSION_INTERNAL); if (timer) - __wt_epoch(session, &enter); + time_start = __wt_clock(session); for (initial_progress = cache->eviction_progress;; ret = 0) { /* @@ -2351,7 +2379,8 @@ __wt_cache_eviction_worker( if (__wt_cache_stuck(session) && __wt_txn_am_oldest(session)) { --cache->evict_aggressive_score; WT_STAT_CONN_INCR(session, txn_fail_cache); - WT_ERR(WT_ROLLBACK); + WT_ERR(__wt_txn_rollback_required(session, + "oldest transaction rolled back for eviction")); } /* @@ -2370,7 +2399,7 @@ __wt_cache_eviction_worker( /* See if eviction is still needed. */ if (!__wt_eviction_needed(session, busy, readonly, &pct_full) || - ((pct_full < 100 || cache->eviction_scrub_limit > 0.0) && + ((pct_full < 100.0 || cache->eviction_scrub_limit > 0.0) && (cache->eviction_progress > initial_progress + max_progress))) break; @@ -2406,11 +2435,13 @@ __wt_cache_eviction_worker( } err: if (timer) { - __wt_epoch(session, &leave); + time_stop = __wt_clock(session); WT_STAT_CONN_INCRV(session, - application_cache_time, WT_TIMEDIFF_US(leave, enter)); + application_cache_time, + WT_CLOCKDIFF_US(time_stop, time_start)); } +done: WT_TRACK_OP_END(session); return (ret); /* NOTREACHED */ } @@ -2552,7 +2583,9 @@ __verbose_dump_cache_single(WT_SESSION_IMPL *session, dhandle->checkpoint != NULL ? dhandle->checkpoint : "<live>", btree->evict_disabled != 0 ? "eviction disabled" : "", btree->evict_disabled_open ? " at open" : "")); - if (intl_pages != 0) + if (intl_pages == 0) + WT_RET(__wt_msg(session, "internal: 0 pages")); + else WT_RET(__wt_msg(session, "internal: " "%" PRIu64 " pages, " @@ -2569,7 +2602,9 @@ __verbose_dump_cache_single(WT_SESSION_IMPL *session, intl_dirty_bytes / WT_MEGABYTE, intl_bytes_max / WT_MEGABYTE, intl_dirty_bytes_max / WT_MEGABYTE)); - if (leaf_pages != 0) + if (leaf_pages == 0) + WT_RET(__wt_msg(session, "leaf: 0 pages")); + else WT_RET(__wt_msg(session, "leaf: " "%" PRIu64 " pages, " @@ -2603,22 +2638,25 @@ __wt_verbose_dump_cache(WT_SESSION_IMPL *session) WT_CONNECTION_IMPL *conn; WT_DATA_HANDLE *dhandle; WT_DECL_RET; + double pct; uint64_t total_bytes, total_dirty_bytes; - u_int pct; + bool needed; conn = S2C(session); total_bytes = total_dirty_bytes = 0; - pct = 0; /* [-Werror=uninitialized] */ + pct = 0.0; /* [-Werror=uninitialized] */ WT_RET(__wt_msg(session, "%s", WT_DIVIDER)); WT_RET(__wt_msg(session, "cache dump")); WT_RET(__wt_msg(session, "cache full: %s", __wt_cache_full(session) ? "yes" : "no")); - WT_RET(__wt_msg(session, "cache clean check: %s (%u%%)", - __wt_eviction_clean_needed(session, &pct) ? "yes" : "no", pct)); - WT_RET(__wt_msg(session, "cache dirty check: %s (%u%%)", - __wt_eviction_dirty_needed(session, &pct) ? "yes" : "no", pct)); + needed = __wt_eviction_clean_needed(session, &pct); + WT_RET(__wt_msg(session, + "cache clean check: %s (%2.3f%%)", needed ? "yes" : "no", pct)); + needed = __wt_eviction_dirty_needed(session, &pct); + WT_RET(__wt_msg(session, + "cache dirty check: %s (%2.3f%%)", needed ? "yes" : "no", pct)); for (dhandle = NULL;;) { WT_WITH_HANDLE_LIST_READ_LOCK(session, diff --git a/src/third_party/wiredtiger/src/evict/evict_page.c b/src/third_party/wiredtiger/src/evict/evict_page.c index 7a84f90eb81..6d904a2004b 100644 --- a/src/third_party/wiredtiger/src/evict/evict_page.c +++ b/src/third_party/wiredtiger/src/evict/evict_page.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -52,15 +52,15 @@ __evict_exclusive(WT_SESSION_IMPL *session, WT_REF *ref) int __wt_page_release_evict(WT_SESSION_IMPL *session, WT_REF *ref) { - struct timespec start, stop; WT_BTREE *btree; WT_DECL_RET; WT_PAGE *page; + uint64_t time_start, time_stop; bool locked, too_big; btree = S2BT(session); page = ref->page; - __wt_epoch(session, &start); + time_start = __wt_clock(session); /* * Take some care with order of operations: if we release the hazard @@ -83,12 +83,12 @@ __wt_page_release_evict(WT_SESSION_IMPL *session, WT_REF *ref) * we have one of two pairs of stats to increment. */ ret = __wt_evict(session, ref, false); - __wt_epoch(session, &stop); + time_stop = __wt_clock(session); if (ret == 0) { if (too_big) { WT_STAT_CONN_INCR(session, cache_eviction_force); WT_STAT_CONN_INCRV(session, cache_eviction_force_time, - WT_TIMEDIFF_US(stop, start)); + WT_CLOCKDIFF_US(time_stop, time_start)); } else { /* * If the page isn't too big, we are evicting it because @@ -98,12 +98,12 @@ __wt_page_release_evict(WT_SESSION_IMPL *session, WT_REF *ref) WT_STAT_CONN_INCR(session, cache_eviction_force_delete); WT_STAT_CONN_INCRV(session, cache_eviction_force_delete_time, - WT_TIMEDIFF_US(stop, start)); + WT_CLOCKDIFF_US(time_stop, time_start)); } } else { WT_STAT_CONN_INCR(session, cache_eviction_force_fail); WT_STAT_CONN_INCRV(session, cache_eviction_force_fail_time, - WT_TIMEDIFF_US(stop, start)); + WT_CLOCKDIFF_US(time_stop, time_start)); } (void)__wt_atomic_subv32(&btree->evict_busy, 1); @@ -268,9 +268,16 @@ __evict_page_clean_update(WT_SESSION_IMPL *session, WT_REF *ref, bool closing) * Discard the page and update the reference structure; if the page has * an address, it's a disk page; if it has no address, it's a deleted * page re-instantiated (for example, by searching) and never written. + * + * If evicting a WT_REF_LIMBO reference, we get to here and transition + * back to WT_REF_LOOKASIDE. */ __wt_ref_out(session, ref); - if (ref->addr == NULL) { + if (!closing && ref->page_las != NULL && + ref->page_las->eviction_to_lookaside) { + ref->page_las->eviction_to_lookaside = false; + WT_PUBLISH(ref->state, WT_REF_LOOKASIDE); + } else if (ref->addr == NULL) { WT_WITH_PAGE_INDEX(session, ret = __evict_delete_ref(session, ref, closing)); WT_RET_BUSY_OK(ret); @@ -299,16 +306,14 @@ __evict_page_dirty_update(WT_SESSION_IMPL *session, WT_REF *ref, bool closing) switch (mod->rec_result) { case WT_PM_REC_EMPTY: /* Page is empty */ /* - * Update the parent to reference a deleted page. The fact that - * reconciliation left the page "empty" means there's no older - * transaction in the system that might need to see an earlier - * version of the page. For that reason, we clear the address - * of the page, if we're forced to "read" into that namespace, - * we'll instantiate a new page instead of trying to read from - * the backing store. + * Update the parent to reference a deleted page. Reconciliation + * left the page "empty", so there's no older transaction in the + * system that might need to see an earlier version of the page. + * There's no backing address, if we're forced to "read" into + * that namespace, we instantiate a new page instead of trying + * to read from the backing store. */ __wt_ref_out(session, ref); - ref->addr = NULL; WT_WITH_PAGE_INDEX(session, ret = __evict_delete_ref(session, ref, closing)); WT_RET_BUSY_OK(ret); @@ -347,9 +352,7 @@ __evict_page_dirty_update(WT_SESSION_IMPL *session, WT_REF *ref, bool closing) * Publish: a barrier to ensure the structure fields are set * before the state change makes the page available to readers. */ - if (mod->mod_replace.addr == NULL) - ref->addr = NULL; - else { + if (mod->mod_replace.addr != NULL) { WT_RET(__wt_calloc_one(session, &addr)); *addr = mod->mod_replace; mod->mod_replace.addr = NULL; @@ -361,6 +364,7 @@ __evict_page_dirty_update(WT_SESSION_IMPL *session, WT_REF *ref, bool closing) * Eviction wants to keep this page if we have a disk image, * re-instantiate the page in memory, else discard the page. */ + __wt_free(session, ref->page_las); if (mod->mod_disk_image == NULL) { if (mod->mod_page_las.las_pageid != 0) { WT_RET( diff --git a/src/third_party/wiredtiger/src/evict/evict_stat.c b/src/third_party/wiredtiger/src/evict/evict_stat.c index b8878f1ce2f..4fcef60a04d 100644 --- a/src/third_party/wiredtiger/src/evict/evict_stat.c +++ b/src/third_party/wiredtiger/src/evict/evict_stat.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/include/api.h b/src/third_party/wiredtiger/src/include/api.h index 2e3a2fe220f..847d6c5ee01 100644 --- a/src/third_party/wiredtiger/src/include/api.h +++ b/src/third_party/wiredtiger/src/include/api.h @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -36,10 +36,16 @@ /* Standard entry points to the API: declares/initializes local variables. */ #define API_SESSION_INIT(s, h, n, dh) \ + WT_TRACK_OP_DECL; \ WT_DATA_HANDLE *__olddh = (s)->dhandle; \ const char *__oldname = (s)->name; \ (s)->dhandle = (dh); \ (s)->name = (s)->lastop = #h "." #n; \ + /* \ + * No code before this line, otherwise error handling won't be \ + * correct. \ + */ \ + WT_TRACK_OP_INIT(s); \ WT_SINGLE_THREAD_CHECK_START(s); \ WT_ERR(WT_SESSION_CHECK_PANIC(s)); \ __wt_verbose((s), WT_VERB_API, "%s", "CALL: " #h ":" #n) @@ -57,38 +63,51 @@ #define API_END(s, ret) \ if ((s) != NULL) { \ - WT_SINGLE_THREAD_CHECK_STOP(s); \ - (s)->dhandle = __olddh; \ - (s)->name = __oldname; \ + WT_TRACK_OP_END(s); \ + WT_SINGLE_THREAD_CHECK_STOP(s); \ if (F_ISSET(&(s)->txn, WT_TXN_RUNNING) && \ (ret) != 0 && \ (ret) != WT_NOTFOUND && \ (ret) != WT_DUPLICATE_KEY) \ F_SET(&(s)->txn, WT_TXN_ERROR); \ + /* \ + * No code after this line, otherwise error handling \ + * won't be correct. \ + */ \ + (s)->dhandle = __olddh; \ + (s)->name = __oldname; \ } \ } while (0) /* An API call wrapped in a transaction if necessary. */ #define TXN_API_CALL(s, h, n, bt, config, cfg) do { \ - bool __autotxn = false; \ + bool __autotxn = false, __update = false; \ API_CALL(s, h, n, bt, config, cfg); \ __wt_txn_timestamp_flags(s); \ __autotxn = !F_ISSET(&(s)->txn, WT_TXN_AUTOCOMMIT | WT_TXN_RUNNING);\ if (__autotxn) \ - F_SET(&(s)->txn, WT_TXN_AUTOCOMMIT) + F_SET(&(s)->txn, WT_TXN_AUTOCOMMIT); \ + __update = !F_ISSET(&(s)->txn, WT_TXN_UPDATE); \ + if (__update) \ + F_SET(&(s)->txn, WT_TXN_UPDATE); \ /* An API call wrapped in a transaction if necessary. */ #define TXN_API_CALL_NOCONF(s, h, n, dh) do { \ - bool __autotxn = false; \ + bool __autotxn = false, __update = false; \ API_CALL_NOCONF(s, h, n, dh); \ __wt_txn_timestamp_flags(s); \ __autotxn = !F_ISSET(&(s)->txn, WT_TXN_AUTOCOMMIT | WT_TXN_RUNNING);\ if (__autotxn) \ - F_SET(&(s)->txn, WT_TXN_AUTOCOMMIT) + F_SET(&(s)->txn, WT_TXN_AUTOCOMMIT); \ + __update = !F_ISSET(&(s)->txn, WT_TXN_UPDATE); \ + if (__update) \ + F_SET(&(s)->txn, WT_TXN_UPDATE); \ /* End a transactional API call, optional retry on deadlock. */ #define TXN_API_END_RETRY(s, ret, retry) \ API_END(s, ret); \ + if (__update) \ + F_CLR(&(s)->txn, WT_TXN_UPDATE); \ if (__autotxn) { \ if (F_ISSET(&(s)->txn, WT_TXN_AUTOCOMMIT)) \ F_CLR(&(s)->txn, WT_TXN_AUTOCOMMIT); \ diff --git a/src/third_party/wiredtiger/src/include/async.h b/src/third_party/wiredtiger/src/include/async.h index 53a7d982ba5..769aa4cfeec 100644 --- a/src/third_party/wiredtiger/src/include/async.h +++ b/src/third_party/wiredtiger/src/include/async.h @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/include/bitstring.i b/src/third_party/wiredtiger/src/include/bitstring.i index 237b4b98afe..d45ef25c86b 100644 --- a/src/third_party/wiredtiger/src/include/bitstring.i +++ b/src/third_party/wiredtiger/src/include/bitstring.i @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/src/include/block.h b/src/third_party/wiredtiger/src/include/block.h index 1a445bfac0b..01a7617789a 100644 --- a/src/third_party/wiredtiger/src/include/block.h +++ b/src/third_party/wiredtiger/src/include/block.h @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -341,7 +341,11 @@ struct __wt_block_header { */ uint32_t checksum; /* 04-07: checksum */ -#define WT_BLOCK_DATA_CKSUM 0x01 /* Block data is part of the checksum */ + /* + * No automatic generation: flag values cannot change, they're written + * to disk. + */ +#define WT_BLOCK_DATA_CKSUM 0x1u /* Block data is part of the checksum */ uint8_t flags; /* 08: flags */ /* diff --git a/src/third_party/wiredtiger/src/include/bloom.h b/src/third_party/wiredtiger/src/include/bloom.h index a0efc0bf1fa..6fe8cb2c1d0 100644 --- a/src/third_party/wiredtiger/src/include/bloom.h +++ b/src/third_party/wiredtiger/src/include/bloom.h @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/include/btmem.h b/src/third_party/wiredtiger/src/include/btmem.h index c5cdfe5850a..7fbf27a1fff 100644 --- a/src/third_party/wiredtiger/src/include/btmem.h +++ b/src/third_party/wiredtiger/src/include/btmem.h @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -8,6 +8,33 @@ #define WT_RECNO_OOB 0 /* Illegal record number */ +/* AUTOMATIC FLAG VALUE GENERATION START */ +#define WT_READ_CACHE 0x0001u +#define WT_READ_IGNORE_CACHE_SIZE 0x0002u +#define WT_READ_LOOKASIDE 0x0004u +#define WT_READ_NOTFOUND_OK 0x0008u +#define WT_READ_NO_EMPTY 0x0010u +#define WT_READ_NO_GEN 0x0020u +#define WT_READ_NO_SPLIT 0x0040u +#define WT_READ_NO_WAIT 0x0080u +#define WT_READ_PREV 0x0100u +#define WT_READ_RESTART_OK 0x0200u +#define WT_READ_SKIP_INTL 0x0400u +#define WT_READ_TRUNCATE 0x0800u +#define WT_READ_WONT_NEED 0x1000u +/* AUTOMATIC FLAG VALUE GENERATION STOP */ + +/* AUTOMATIC FLAG VALUE GENERATION START */ +#define WT_REC_CHECKPOINT 0x01u +#define WT_REC_EVICT 0x02u +#define WT_REC_IN_MEMORY 0x04u +#define WT_REC_LOOKASIDE 0x08u +#define WT_REC_SCRUB 0x10u +#define WT_REC_UPDATE_RESTORE 0x20u +#define WT_REC_VISIBILITY_ERR 0x40u +#define WT_REC_VISIBLE_ALL 0x80u +/* AUTOMATIC FLAG VALUE GENERATION STOP */ + /* * WT_PAGE_HEADER -- * Blocks have a common header, a WT_PAGE_HEADER structure followed by a @@ -41,11 +68,15 @@ struct __wt_page_header { uint8_t type; /* 24: page type */ -#define WT_PAGE_COMPRESSED 0x01 /* Page is compressed on disk */ -#define WT_PAGE_EMPTY_V_ALL 0x02 /* Page has all zero-length values */ -#define WT_PAGE_EMPTY_V_NONE 0x04 /* Page has no zero-length values */ -#define WT_PAGE_ENCRYPTED 0x08 /* Page is encrypted on disk */ -#define WT_PAGE_LAS_UPDATE 0x10 /* Page updates in lookaside store */ + /* + * No automatic generation: flag values cannot change, they're written + * to disk. + */ +#define WT_PAGE_COMPRESSED 0x01u /* Page is compressed on disk */ +#define WT_PAGE_EMPTY_V_ALL 0x02u /* Page has all zero-length values */ +#define WT_PAGE_EMPTY_V_NONE 0x04u /* Page has no zero-length values */ +#define WT_PAGE_ENCRYPTED 0x08u /* Page is encrypted on disk */ +#define WT_PAGE_LAS_UPDATE 0x10u /* Page updates in lookaside store */ uint8_t flags; /* 25: flags */ /* @@ -145,8 +176,10 @@ struct __wt_ovfl_reuse { * skiplist entry; if reconciliation fails for any reason, discard the * newly added skiplist entries, along with their underlying blocks. */ -#define WT_OVFL_REUSE_INUSE 0x01 -#define WT_OVFL_REUSE_JUST_ADDED 0x02 +/* AUTOMATIC FLAG VALUE GENERATION START */ +#define WT_OVFL_REUSE_INUSE 0x1u +#define WT_OVFL_REUSE_JUST_ADDED 0x2u +/* AUTOMATIC FLAG VALUE GENERATION STOP */ uint8_t flags; /* @@ -191,19 +224,22 @@ struct __wt_ovfl_reuse { "key_format=" WT_UNCHECKED_STRING(QIQu) \ ",value_format=" WT_UNCHECKED_STRING(QuBu) \ ",block_compressor=" WT_LOOKASIDE_COMPRESSOR \ - ",leaf_value_max=64MB" + ",leaf_value_max=64MB" \ + ",prefix_compression=true" /* * WT_PAGE_LOOKASIDE -- * Related information for on-disk pages with lookaside entries. */ struct __wt_page_lookaside { - uint64_t las_pageid; /* Page ID in lookaside */ - uint64_t las_max_txn; /* Maximum transaction ID in - lookaside */ - WT_DECL_TIMESTAMP(min_timestamp) /* Min timestamp in lookaside */ - WT_DECL_TIMESTAMP(onpage_timestamp) /* Max timestamp on page */ - bool las_skew_newest; /* On-page skewed to newest */ + uint64_t las_pageid; /* Page ID in lookaside */ + uint64_t las_max_txn; /* Max transaction ID in lookaside */ + WT_DECL_TIMESTAMP(min_timestamp)/* Min timestamp in lookaside */ + /* Max timestamp on page */ + WT_DECL_TIMESTAMP(onpage_timestamp) + bool eviction_to_lookaside; /* Revert to lookaside on eviction */ + bool las_skew_newest; /* On-page skewed to newest */ + bool invalid; /* History is required correct reads */ }; /* @@ -604,14 +640,16 @@ struct __wt_page { #define WT_PAGE_ROW_LEAF 7 /* Row-store leaf page */ uint8_t type; /* Page type */ -#define WT_PAGE_BUILD_KEYS 0x01 /* Keys have been built in memory */ -#define WT_PAGE_DISK_ALLOC 0x02 /* Disk image in allocated memory */ -#define WT_PAGE_DISK_MAPPED 0x04 /* Disk image in mapped memory */ -#define WT_PAGE_EVICT_LRU 0x08 /* Page is on the LRU queue */ -#define WT_PAGE_OVERFLOW_KEYS 0x10 /* Page has overflow keys */ -#define WT_PAGE_READ_NO_EVICT 0x20 /* Page read with eviction disabled */ -#define WT_PAGE_SPLIT_INSERT 0x40 /* A leaf page was split for append */ -#define WT_PAGE_UPDATE_IGNORE 0x80 /* Ignore updates on page discard */ +/* AUTOMATIC FLAG VALUE GENERATION START */ +#define WT_PAGE_BUILD_KEYS 0x01u /* Keys have been built in memory */ +#define WT_PAGE_DISK_ALLOC 0x02u /* Disk image in allocated memory */ +#define WT_PAGE_DISK_MAPPED 0x04u /* Disk image in mapped memory */ +#define WT_PAGE_EVICT_LRU 0x08u /* Page is on the LRU queue */ +#define WT_PAGE_EVICT_NO_PROGRESS 0x10u /* Eviction doesn't count as progress */ +#define WT_PAGE_OVERFLOW_KEYS 0x20u /* Page has overflow keys */ +#define WT_PAGE_SPLIT_INSERT 0x40u /* A leaf page was split for append */ +#define WT_PAGE_UPDATE_IGNORE 0x80u /* Ignore updates on page discard */ +/* AUTOMATIC FLAG VALUE GENERATION STOP */ uint8_t flags_atomic; /* Atomic flags, use F_*_ATOMIC */ uint8_t unused[2]; /* Unused padding */ @@ -685,6 +723,10 @@ struct __wt_page { * row-store leaf pages without reading them if they don't reference * overflow items. * + * WT_REF_LIMBO: + * The page image has been loaded into memory but there is additional + * history in the lookaside table that has not been applied. + * * WT_REF_LOCKED: * Locked for exclusive access. In eviction, this page or a parent has * been selected for eviction; once hazard pointers are checked, the page @@ -758,11 +800,12 @@ struct __wt_ref { #define WT_REF_DISK 0 /* Page is on disk */ #define WT_REF_DELETED 1 /* Page is on disk, but deleted */ -#define WT_REF_LOCKED 2 /* Page locked for exclusive access */ -#define WT_REF_LOOKASIDE 3 /* Page is on disk with lookaside */ -#define WT_REF_MEM 4 /* Page is in cache and valid */ -#define WT_REF_READING 5 /* Page being read */ -#define WT_REF_SPLIT 6 /* Parent page split (WT_REF dead) */ +#define WT_REF_LIMBO 2 /* Page is in cache without history */ +#define WT_REF_LOCKED 3 /* Page locked for exclusive access */ +#define WT_REF_LOOKASIDE 4 /* Page is on disk with lookaside */ +#define WT_REF_MEM 5 /* Page is in cache and valid */ +#define WT_REF_READING 6 /* Page being read */ +#define WT_REF_SPLIT 7 /* Parent page split (WT_REF dead) */ volatile uint32_t state; /* Page state */ /* @@ -784,16 +827,14 @@ struct __wt_ref { #undef ref_ikey #define ref_ikey key.ikey - union { - WT_PAGE_DELETED *page_del; /* Deleted page information */ - WT_PAGE_LOOKASIDE *page_las; /* Lookaside information */ - }; + WT_PAGE_DELETED *page_del; /* Deleted page information */ + WT_PAGE_LOOKASIDE *page_las; /* Lookaside information */ }; /* * WT_REF_SIZE is the expected structure size -- we verify the build to ensure * the compiler hasn't inserted padding which would break the world. */ -#define WT_REF_SIZE 48 +#define WT_REF_SIZE 56 /* * WT_ROW -- @@ -939,15 +980,17 @@ struct __wt_update { uint32_t size; /* data length */ #define WT_UPDATE_INVALID 0 /* diagnostic check */ -#define WT_UPDATE_DELETED 1 /* deleted */ -#define WT_UPDATE_MODIFIED 2 /* partial-update modify value */ -#define WT_UPDATE_RESERVED 3 /* reserved */ +#define WT_UPDATE_BIRTHMARK 1 /* transaction for on-page value */ +#define WT_UPDATE_MODIFY 2 /* partial-update modify value */ +#define WT_UPDATE_RESERVE 3 /* reserved */ #define WT_UPDATE_STANDARD 4 /* complete value */ +#define WT_UPDATE_TOMBSTONE 5 /* deleted */ uint8_t type; /* type (one byte to conserve memory) */ /* If the update includes a complete value. */ #define WT_UPDATE_DATA_VALUE(upd) \ - ((upd)->type == WT_UPDATE_STANDARD || (upd)->type == WT_UPDATE_DELETED) + ((upd)->type == WT_UPDATE_STANDARD || \ + (upd)->type == WT_UPDATE_TOMBSTONE) #if WT_TIMESTAMP_SIZE != 8 WT_DECL_TIMESTAMP(timestamp) /* unaligned uint8_t array timestamp */ diff --git a/src/third_party/wiredtiger/src/include/btree.h b/src/third_party/wiredtiger/src/include/btree.h index 8a3273d1b6b..96f6309aba4 100644 --- a/src/third_party/wiredtiger/src/include/btree.h +++ b/src/third_party/wiredtiger/src/include/btree.h @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -97,10 +97,13 @@ struct __wt_btree { uint64_t maxmempage; /* In-memory page max size */ uint64_t splitmempage; /* In-memory split trigger size */ -#define WT_ASSERT_COMMIT_TS_ALWAYS 0x0001 -#define WT_ASSERT_COMMIT_TS_NEVER 0x0002 -#define WT_ASSERT_READ_TS_ALWAYS 0x0004 -#define WT_ASSERT_READ_TS_NEVER 0x0008 +/* AUTOMATIC FLAG VALUE GENERATION START */ +#define WT_ASSERT_COMMIT_TS_ALWAYS 0x01u +#define WT_ASSERT_COMMIT_TS_KEYS 0x02u +#define WT_ASSERT_COMMIT_TS_NEVER 0x04u +#define WT_ASSERT_READ_TS_ALWAYS 0x08u +#define WT_ASSERT_READ_TS_NEVER 0x10u +/* AUTOMATIC FLAG VALUE GENERATION STOP */ uint32_t assert_flags; /* Debugging assertion information */ void *huffman_key; /* Key huffman encoding */ @@ -192,21 +195,23 @@ struct __wt_btree { } evict_start_type; /* - * Flag values up to 0xff are reserved for WT_DHANDLE_XXX. + * Flag values up to 0xff are reserved for WT_DHANDLE_XXX. We don't + * automatically generate these flag values for that reason, there's + * no way to start at an offset. */ -#define WT_BTREE_ALTER 0x000100 /* Handle is for alter */ -#define WT_BTREE_BULK 0x000200 /* Bulk-load handle */ -#define WT_BTREE_CLOSED 0x000400 /* Handle closed */ -#define WT_BTREE_IGNORE_CACHE 0x000800 /* Cache-resident object */ -#define WT_BTREE_IN_MEMORY 0x001000 /* Cache-resident object */ -#define WT_BTREE_LOOKASIDE 0x002000 /* Look-aside table */ -#define WT_BTREE_NO_CHECKPOINT 0x004000 /* Disable checkpoints */ -#define WT_BTREE_NO_LOGGING 0x008000 /* Disable logging */ -#define WT_BTREE_REBALANCE 0x010000 /* Handle is for rebalance */ -#define WT_BTREE_SALVAGE 0x020000 /* Handle is for salvage */ -#define WT_BTREE_SKIP_CKPT 0x040000 /* Handle skipped checkpoint */ -#define WT_BTREE_UPGRADE 0x080000 /* Handle is for upgrade */ -#define WT_BTREE_VERIFY 0x100000 /* Handle is for verify */ +#define WT_BTREE_ALTER 0x000100u /* Handle is for alter */ +#define WT_BTREE_BULK 0x000200u /* Bulk-load handle */ +#define WT_BTREE_CLOSED 0x000400u /* Handle closed */ +#define WT_BTREE_IGNORE_CACHE 0x000800u /* Cache-resident object */ +#define WT_BTREE_IN_MEMORY 0x001000u /* Cache-resident object */ +#define WT_BTREE_LOOKASIDE 0x002000u /* Look-aside table */ +#define WT_BTREE_NO_CHECKPOINT 0x004000u /* Disable checkpoints */ +#define WT_BTREE_NO_LOGGING 0x008000u /* Disable logging */ +#define WT_BTREE_REBALANCE 0x010000u /* Handle is for rebalance */ +#define WT_BTREE_SALVAGE 0x020000u /* Handle is for salvage */ +#define WT_BTREE_SKIP_CKPT 0x040000u /* Handle skipped checkpoint */ +#define WT_BTREE_UPGRADE 0x080000u /* Handle is for upgrade */ +#define WT_BTREE_VERIFY 0x100000u /* Handle is for verify */ uint32_t flags; }; diff --git a/src/third_party/wiredtiger/src/include/btree.i b/src/third_party/wiredtiger/src/include/btree.i index 560cc8eb212..3a6413162f3 100644 --- a/src/third_party/wiredtiger/src/include/btree.i +++ b/src/third_party/wiredtiger/src/include/btree.i @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -257,7 +257,7 @@ __wt_cache_page_byte_dirty_decr( * Take care to read the dirty-byte count only once in case * we're racing with updates. */ - orig = page->modify->bytes_dirty; + WT_ORDERED_READ(orig, page->modify->bytes_dirty); decr = WT_MIN(size, orig); if (__wt_atomic_cassize( &page->modify->bytes_dirty, orig, orig - decr)) @@ -400,7 +400,7 @@ __wt_cache_page_image_incr(WT_SESSION_IMPL *session, uint32_t size) * Evict pages from the cache. */ static inline void -__wt_cache_page_evict(WT_SESSION_IMPL *session, WT_PAGE *page, bool rewrite) +__wt_cache_page_evict(WT_SESSION_IMPL *session, WT_PAGE *page) { WT_BTREE *btree; WT_CACHE *cache; @@ -448,17 +448,8 @@ __wt_cache_page_evict(WT_SESSION_IMPL *session, WT_PAGE *page, bool rewrite) /* * Track if eviction makes progress. This is used in various places to * determine whether eviction is stuck. - * - * We don't count rewrites as progress. - * - * Further, if a page was read with eviction disabled, we don't count - * evicting a it as progress. Since disabling eviction allows pages to - * be read even when the cache is full, we want to avoid workloads - * repeatedly reading a page with eviction disabled (e.g., from the - * metadata), then evicting that page and deciding that is a sign that - * eviction is unstuck. */ - if (!rewrite && !F_ISSET_ATOMIC(page, WT_PAGE_READ_NO_EVICT)) + if (!F_ISSET_ATOMIC(page, WT_PAGE_EVICT_NO_PROGRESS)) (void)__wt_atomic_addv64(&cache->eviction_progress, 1); } @@ -1236,7 +1227,7 @@ __wt_leaf_page_can_split(WT_SESSION_IMPL *session, WT_PAGE *page) * size, split as soon as there are 5 items on the page. */ #define WT_MAX_SPLIT_COUNT 5 - if (page->memory_footprint > btree->maxleafpage * 2) { + if (page->memory_footprint > (size_t)btree->maxleafpage * 2) { for (count = 0, ins = ins_head->head[0]; ins != NULL; ins = ins->next[0]) { @@ -1469,7 +1460,7 @@ __wt_page_release(WT_SESSION_IMPL *session, WT_REF *ref, uint32_t flags) (LF_ISSET(WT_READ_NO_SPLIT) || (!inmem_split && F_ISSET(session, WT_SESSION_NO_RECONCILE)))) { if (!WT_SESSION_IS_CHECKPOINT(session)) - __wt_page_evict_urgent(session, ref); + (void)__wt_page_evict_urgent(session, ref); } else { WT_RET_BUSY_OK(__wt_page_release_evict(session, ref)); return (0); diff --git a/src/third_party/wiredtiger/src/include/btree_cmp.i b/src/third_party/wiredtiger/src/include/btree_cmp.i index 9efbf8f618f..8f8e0e83717 100644 --- a/src/third_party/wiredtiger/src/include/btree_cmp.i +++ b/src/third_party/wiredtiger/src/include/btree_cmp.i @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/include/buf.i b/src/third_party/wiredtiger/src/include/buf.i index 8ff52f86ced..e4b9dcb2c79 100644 --- a/src/third_party/wiredtiger/src/include/buf.i +++ b/src/third_party/wiredtiger/src/include/buf.i @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/include/cache.h b/src/third_party/wiredtiger/src/include/cache.h index a2c0e95293c..7a49f388826 100644 --- a/src/third_party/wiredtiger/src/include/cache.h +++ b/src/third_party/wiredtiger/src/include/cache.h @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -89,9 +89,7 @@ struct __wt_cache { uint64_t worker_evicts; /* Pages evicted by worker threads */ uint64_t evict_max_page_size; /* Largest page seen at eviction */ -#if defined(HAVE_DIAGNOSTIC) || defined(HAVE_VERBOSE) struct timespec stuck_time; /* Stuck time */ -#endif /* * Read information. @@ -107,12 +105,16 @@ struct __wt_cache { WT_CONDVAR *evict_cond; /* Eviction server condition */ WT_SPINLOCK evict_walk_lock; /* Eviction walk location */ - u_int eviction_dirty_target; /* Percent to allow dirty */ - u_int eviction_dirty_trigger; /* Percent to trigger dirty eviction */ - u_int eviction_trigger; /* Percent to trigger eviction */ - u_int eviction_target; /* Percent to end eviction */ + /* + * Eviction threshold percentages use double type to allow for + * specifying percentages less than one. + */ + double eviction_dirty_target; /* Percent to allow dirty */ + double eviction_dirty_trigger; /* Percent to trigger dirty eviction */ + double eviction_trigger; /* Percent to trigger eviction */ + double eviction_target; /* Percent to end eviction */ - u_int eviction_checkpoint_target;/* Percent to reduce dirty + double eviction_checkpoint_target;/* Percent to reduce dirty to during checkpoint scrubs */ double eviction_scrub_limit; /* Percent of cache to trigger dirty eviction during checkpoint @@ -233,17 +235,21 @@ struct __wt_cache { /* * Flags. */ -#define WT_CACHE_POOL_MANAGER 0x001 /* The active cache pool manager */ -#define WT_CACHE_POOL_RUN 0x002 /* Cache pool thread running */ +/* AUTOMATIC FLAG VALUE GENERATION START */ +#define WT_CACHE_POOL_MANAGER 0x1u /* The active cache pool manager */ +#define WT_CACHE_POOL_RUN 0x2u /* Cache pool thread running */ +/* AUTOMATIC FLAG VALUE GENERATION STOP */ uint32_t pool_flags; /* Cache pool flags */ -#define WT_CACHE_EVICT_CLEAN 0x001 /* Evict clean pages */ -#define WT_CACHE_EVICT_CLEAN_HARD 0x002 /* Clean % blocking app threads */ -#define WT_CACHE_EVICT_DIRTY 0x004 /* Evict dirty pages */ -#define WT_CACHE_EVICT_DIRTY_HARD 0x008 /* Dirty % blocking app threads */ -#define WT_CACHE_EVICT_LOOKASIDE 0x010 /* Try lookaside eviction */ -#define WT_CACHE_EVICT_SCRUB 0x020 /* Scrub dirty pages */ -#define WT_CACHE_EVICT_URGENT 0x040 /* Pages are in the urgent queue */ +/* AUTOMATIC FLAG VALUE GENERATION START */ +#define WT_CACHE_EVICT_CLEAN 0x01u /* Evict clean pages */ +#define WT_CACHE_EVICT_CLEAN_HARD 0x02u /* Clean % blocking app threads */ +#define WT_CACHE_EVICT_DIRTY 0x04u /* Evict dirty pages */ +#define WT_CACHE_EVICT_DIRTY_HARD 0x08u /* Dirty % blocking app threads */ +#define WT_CACHE_EVICT_LOOKASIDE 0x10u /* Try lookaside eviction */ +#define WT_CACHE_EVICT_SCRUB 0x20u /* Scrub dirty pages */ +#define WT_CACHE_EVICT_URGENT 0x40u /* Pages are in the urgent queue */ +/* AUTOMATIC FLAG VALUE GENERATION STOP */ #define WT_CACHE_EVICT_ALL (WT_CACHE_EVICT_CLEAN | WT_CACHE_EVICT_DIRTY) uint32_t flags; }; @@ -272,6 +278,8 @@ struct __wt_cache_pool { uint8_t pool_managed; /* Cache pool has a manager thread */ -#define WT_CACHE_POOL_ACTIVE 0x01 /* Cache pool is active */ +/* AUTOMATIC FLAG VALUE GENERATION START */ +#define WT_CACHE_POOL_ACTIVE 0x1u /* Cache pool is active */ +/* AUTOMATIC FLAG VALUE GENERATION STOP */ uint8_t flags; }; diff --git a/src/third_party/wiredtiger/src/include/cache.i b/src/third_party/wiredtiger/src/include/cache.i index 00f45dcd44f..fc127942d02 100644 --- a/src/third_party/wiredtiger/src/include/cache.i +++ b/src/third_party/wiredtiger/src/include/cache.i @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -255,7 +255,7 @@ __wt_session_can_wait(WT_SESSION_IMPL *session) * volume of data in cache. */ static inline bool -__wt_eviction_clean_needed(WT_SESSION_IMPL *session, u_int *pct_fullp) +__wt_eviction_clean_needed(WT_SESSION_IMPL *session, double *pct_fullp) { WT_CACHE *cache; uint64_t bytes_inuse, bytes_max; @@ -270,7 +270,7 @@ __wt_eviction_clean_needed(WT_SESSION_IMPL *session, u_int *pct_fullp) bytes_inuse = __wt_cache_bytes_inuse(cache); if (pct_fullp != NULL) - *pct_fullp = (u_int)((100 * bytes_inuse) / bytes_max); + *pct_fullp = ((100.0 * bytes_inuse) / bytes_max); return (bytes_inuse > (cache->eviction_trigger * bytes_max) / 100); } @@ -281,7 +281,7 @@ __wt_eviction_clean_needed(WT_SESSION_IMPL *session, u_int *pct_fullp) * volume of dirty data in cache. */ static inline bool -__wt_eviction_dirty_needed(WT_SESSION_IMPL *session, u_int *pct_fullp) +__wt_eviction_dirty_needed(WT_SESSION_IMPL *session, double *pct_fullp) { WT_CACHE *cache; double dirty_trigger; @@ -297,10 +297,10 @@ __wt_eviction_dirty_needed(WT_SESSION_IMPL *session, u_int *pct_fullp) dirty_inuse = __wt_cache_dirty_leaf_inuse(cache); if (pct_fullp != NULL) - *pct_fullp = (u_int)((100 * dirty_inuse) / bytes_max); + *pct_fullp = ((100.0 * dirty_inuse) / bytes_max); if ((dirty_trigger = cache->eviction_scrub_limit) < 1.0) - dirty_trigger = (double)cache->eviction_dirty_trigger; + dirty_trigger = cache->eviction_dirty_trigger; return (dirty_inuse > (uint64_t)(dirty_trigger * bytes_max) / 100); } @@ -312,10 +312,10 @@ __wt_eviction_dirty_needed(WT_SESSION_IMPL *session, u_int *pct_fullp) */ static inline bool __wt_eviction_needed( - WT_SESSION_IMPL *session, bool busy, bool readonly, u_int *pct_fullp) + WT_SESSION_IMPL *session, bool busy, bool readonly, double *pct_fullp) { WT_CACHE *cache; - u_int pct_dirty, pct_full; + double pct_dirty, pct_full; bool clean_needed, dirty_needed; cache = S2C(session)->cache; @@ -330,7 +330,7 @@ __wt_eviction_needed( clean_needed = __wt_eviction_clean_needed(session, &pct_full); if (readonly) { dirty_needed = false; - pct_dirty = 0; + pct_dirty = 0.0; } else dirty_needed = __wt_eviction_dirty_needed(session, &pct_dirty); @@ -339,9 +339,9 @@ __wt_eviction_needed( * we involve the application thread. */ if (pct_fullp != NULL) - *pct_fullp = (u_int)WT_MAX(0, 100 - WT_MIN( - (int)cache->eviction_trigger - (int)pct_full, - (int)cache->eviction_dirty_trigger - (int)pct_dirty)); + *pct_fullp = WT_MAX(0.0, 100.0 - WT_MIN( + cache->eviction_trigger - pct_full, + cache->eviction_dirty_trigger - pct_dirty)); /* * Only check the dirty trigger when the session is not busy. @@ -381,7 +381,7 @@ __wt_cache_eviction_check( WT_BTREE *btree; WT_TXN_GLOBAL *txn_global; WT_TXN_STATE *txn_state; - u_int pct_full; + double pct_full; if (didworkp != NULL) *didworkp = false; diff --git a/src/third_party/wiredtiger/src/include/cell.i b/src/third_party/wiredtiger/src/include/cell.i index 6ea7d5d68d2..644222ad569 100644 --- a/src/third_party/wiredtiger/src/include/cell.i +++ b/src/third_party/wiredtiger/src/include/cell.i @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/include/column.i b/src/third_party/wiredtiger/src/include/column.i index c95d338f980..6212cda19f3 100644 --- a/src/third_party/wiredtiger/src/include/column.i +++ b/src/third_party/wiredtiger/src/include/column.i @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/include/compact.h b/src/third_party/wiredtiger/src/include/compact.h index d74090c286c..67386a25852 100644 --- a/src/third_party/wiredtiger/src/include/compact.h +++ b/src/third_party/wiredtiger/src/include/compact.h @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/include/config.h b/src/third_party/wiredtiger/src/include/config.h index 3698290f21b..4764ce0fd9f 100644 --- a/src/third_party/wiredtiger/src/include/config.h +++ b/src/third_party/wiredtiger/src/include/config.h @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/include/connection.h b/src/third_party/wiredtiger/src/include/connection.h index 9288618c87e..aef6b2d6777 100644 --- a/src/third_party/wiredtiger/src/include/connection.h +++ b/src/third_party/wiredtiger/src/include/connection.h @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -25,6 +25,9 @@ struct __wt_process { /* Locked: connection queue */ TAILQ_HEAD(__wt_connection_impl_qh, __wt_connection_impl) connqh; WT_CACHE_POOL *cache_pool; +#define WT_TSC_DEFAULT_RATIO 1.0 + double tsc_nsec_ratio; /* rdtsc ticks to nanoseconds */ + bool use_epochtime; /* use expensive time */ /* Checksum function */ #define __wt_checksum(chunk, len) __wt_process.checksum(chunk, len) @@ -104,12 +107,6 @@ struct __wt_named_extractor { }; /* - * Allocate some additional slots for internal sessions so the user cannot - * configure too few sessions for us to run. - */ -#define WT_EXTRA_INTERNAL_SESSIONS 20 - -/* * WT_CONN_CHECK_PANIC -- * Check if we've panicked and return the appropriate error. */ @@ -192,6 +189,12 @@ struct __wt_connection_impl { /* Configuration */ const WT_CONFIG_ENTRY **config_entries; + const char *optrack_path; /* Directory for operation logs */ + WT_FH *optrack_map_fh; /* Name to id translation file. */ + WT_SPINLOCK optrack_map_spinlock; /* Translation file spinlock. */ + uintmax_t optrack_pid; /* Cache the process ID. */ + uint16_t optrack_uid; /* Unique function ID */ + void **foc; /* Free-on-close array */ size_t foc_cnt; /* Array entries */ size_t foc_size; /* Array size */ @@ -312,15 +315,17 @@ struct __wt_connection_impl { const char *stat_stamp; /* Statistics log entry timestamp */ uint64_t stat_usecs; /* Statistics log period */ -#define WT_CONN_LOG_ARCHIVE 0x001 /* Archive is enabled */ -#define WT_CONN_LOG_DOWNGRADED 0x002 /* Running older version */ -#define WT_CONN_LOG_ENABLED 0x004 /* Logging is enabled */ -#define WT_CONN_LOG_EXISTED 0x008 /* Log files found */ -#define WT_CONN_LOG_FORCE_DOWNGRADE 0x010 /* Force downgrade */ -#define WT_CONN_LOG_RECOVER_DIRTY 0x020 /* Recovering unclean */ -#define WT_CONN_LOG_RECOVER_DONE 0x040 /* Recovery completed */ -#define WT_CONN_LOG_RECOVER_ERR 0x080 /* Error if recovery required */ -#define WT_CONN_LOG_ZERO_FILL 0x100 /* Manually zero files */ +/* AUTOMATIC FLAG VALUE GENERATION START */ +#define WT_CONN_LOG_ARCHIVE 0x001u /* Archive is enabled */ +#define WT_CONN_LOG_DOWNGRADED 0x002u /* Running older version */ +#define WT_CONN_LOG_ENABLED 0x004u /* Logging is enabled */ +#define WT_CONN_LOG_EXISTED 0x008u /* Log files found */ +#define WT_CONN_LOG_FORCE_DOWNGRADE 0x010u /* Force downgrade */ +#define WT_CONN_LOG_RECOVER_DIRTY 0x020u /* Recovering unclean */ +#define WT_CONN_LOG_RECOVER_DONE 0x040u /* Recovery completed */ +#define WT_CONN_LOG_RECOVER_ERR 0x080u /* Error if recovery required */ +#define WT_CONN_LOG_ZERO_FILL 0x100u /* Manually zero files */ +/* AUTOMATIC FLAG VALUE GENERATION STOP */ uint32_t log_flags; /* Global logging configuration */ WT_CONDVAR *log_cond; /* Log server wait mutex */ WT_SESSION_IMPL *log_session; /* Log server session */ @@ -391,22 +396,64 @@ struct __wt_connection_impl { wt_off_t data_extend_len; /* file_extend data length */ wt_off_t log_extend_len; /* file_extend log length */ -#define WT_DIRECT_IO_CHECKPOINT 0x01 /* Checkpoints */ -#define WT_DIRECT_IO_DATA 0x02 /* Data files */ -#define WT_DIRECT_IO_LOG 0x04 /* Log files */ - uint32_t direct_io; /* O_DIRECT, FILE_FLAG_NO_BUFFERING */ - - uint32_t write_through; /* FILE_FLAG_WRITE_THROUGH */ +/* AUTOMATIC FLAG VALUE GENERATION START */ +#define WT_DIRECT_IO_CHECKPOINT 0x1u /* Checkpoints */ +#define WT_DIRECT_IO_DATA 0x2u /* Data files */ +#define WT_DIRECT_IO_LOG 0x4u /* Log files */ +/* AUTOMATIC FLAG VALUE GENERATION STOP */ + uint64_t direct_io; /* O_DIRECT, FILE_FLAG_NO_BUFFERING */ + uint64_t write_through; /* FILE_FLAG_WRITE_THROUGH */ bool mmap; /* mmap configuration */ int page_size; /* OS page size for mmap alignment */ - uint32_t verbose; + +/* AUTOMATIC FLAG VALUE GENERATION START */ +#define WT_VERB_API 0x000000001u +#define WT_VERB_BLOCK 0x000000002u +#define WT_VERB_CHECKPOINT 0x000000004u +#define WT_VERB_CHECKPOINT_PROGRESS 0x000000008u +#define WT_VERB_COMPACT 0x000000010u +#define WT_VERB_EVICT 0x000000020u +#define WT_VERB_EVICTSERVER 0x000000040u +#define WT_VERB_EVICT_STUCK 0x000000080u +#define WT_VERB_FILEOPS 0x000000100u +#define WT_VERB_HANDLEOPS 0x000000200u +#define WT_VERB_LOG 0x000000400u +#define WT_VERB_LOOKASIDE 0x000000800u +#define WT_VERB_LOOKASIDE_ACTIVITY 0x000001000u +#define WT_VERB_LSM 0x000002000u +#define WT_VERB_LSM_MANAGER 0x000004000u +#define WT_VERB_METADATA 0x000008000u +#define WT_VERB_MUTEX 0x000010000u +#define WT_VERB_OVERFLOW 0x000020000u +#define WT_VERB_READ 0x000040000u +#define WT_VERB_REBALANCE 0x000080000u +#define WT_VERB_RECONCILE 0x000100000u +#define WT_VERB_RECOVERY 0x000200000u +#define WT_VERB_RECOVERY_PROGRESS 0x000400000u +#define WT_VERB_SALVAGE 0x000800000u +#define WT_VERB_SHARED_CACHE 0x001000000u +#define WT_VERB_SPLIT 0x002000000u +#define WT_VERB_TEMPORARY 0x004000000u +#define WT_VERB_THREAD_GROUP 0x008000000u +#define WT_VERB_TIMESTAMP 0x010000000u +#define WT_VERB_TRANSACTION 0x020000000u +#define WT_VERB_VERIFY 0x040000000u +#define WT_VERB_VERSION 0x080000000u +#define WT_VERB_WRITE 0x100000000u +/* AUTOMATIC FLAG VALUE GENERATION STOP */ + uint64_t verbose; /* * Variable with flags for which subsystems the diagnostic stress timing * delays have been requested. */ - uint32_t timing_stress_flags; +/* AUTOMATIC FLAG VALUE GENERATION START */ +#define WT_TIMING_STRESS_CHECKPOINT_SLOW 0x1u +#define WT_TIMING_STRESS_INTERNAL_PAGE_SPLIT_RACE 0x2u +#define WT_TIMING_STRESS_PAGE_SPLIT_RACE 0x4u +/* AUTOMATIC FLAG VALUE GENERATION STOP */ + uint64_t timing_stress_flags; #define WT_STDERR(s) (&S2C(s)->wt_stderr) #define WT_STDOUT(s) (&S2C(s)->wt_stdout) @@ -418,5 +465,28 @@ struct __wt_connection_impl { */ WT_FILE_SYSTEM *file_system; +/* AUTOMATIC FLAG VALUE GENERATION START */ +#define WT_CONN_CACHE_POOL 0x000001u +#define WT_CONN_CKPT_SYNC 0x000002u +#define WT_CONN_CLOSING 0x000004u +#define WT_CONN_CLOSING_NO_MORE_OPENS 0x000008u +#define WT_CONN_EVICTION_NO_LOOKASIDE 0x000010u +#define WT_CONN_EVICTION_RUN 0x000020u +#define WT_CONN_IN_MEMORY 0x000040u +#define WT_CONN_LEAK_MEMORY 0x000080u +#define WT_CONN_LOOKASIDE_OPEN 0x000100u +#define WT_CONN_LSM_MERGE 0x000200u +#define WT_CONN_OPTRACK 0x000400u +#define WT_CONN_PANIC 0x000800u +#define WT_CONN_READONLY 0x001000u +#define WT_CONN_RECOVERING 0x002000u +#define WT_CONN_SERVER_ASYNC 0x004000u +#define WT_CONN_SERVER_CHECKPOINT 0x008000u +#define WT_CONN_SERVER_LOG 0x010000u +#define WT_CONN_SERVER_LSM 0x020000u +#define WT_CONN_SERVER_STATISTICS 0x040000u +#define WT_CONN_SERVER_SWEEP 0x080000u +#define WT_CONN_WAS_BACKUP 0x100000u +/* AUTOMATIC FLAG VALUE GENERATION STOP */ uint32_t flags; }; diff --git a/src/third_party/wiredtiger/src/include/ctype.i b/src/third_party/wiredtiger/src/include/ctype.i index 3855ae653a5..e69facdb251 100644 --- a/src/third_party/wiredtiger/src/include/ctype.i +++ b/src/third_party/wiredtiger/src/include/ctype.i @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/include/cursor.h b/src/third_party/wiredtiger/src/include/cursor.h index 8d2f2c80c2a..31dec1d24f6 100644 --- a/src/third_party/wiredtiger/src/include/cursor.h +++ b/src/third_party/wiredtiger/src/include/cursor.h @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -74,7 +74,9 @@ struct __wt_cursor_backup { size_t list_allocated; size_t list_next; -#define WT_CURBACKUP_LOCKER 0x01 /* Hot-backup started */ +/* AUTOMATIC FLAG VALUE GENERATION START */ +#define WT_CURBACKUP_LOCKER 0x1u /* Hot-backup started */ +/* AUTOMATIC FLAG VALUE GENERATION STOP */ uint8_t flags; }; #define WT_CURSOR_BACKUP_ID(cursor) (((WT_CURSOR_BACKUP *)(cursor))->maxid) @@ -209,14 +211,15 @@ struct __wt_cursor_btree { uint64_t lastrecno; #endif -#define WT_CBT_ACTIVE 0x01 /* Active in the tree */ -#define WT_CBT_ITERATE_APPEND 0x02 /* Col-store: iterating append list */ -#define WT_CBT_ITERATE_NEXT 0x04 /* Next iteration configuration */ -#define WT_CBT_ITERATE_PREV 0x08 /* Prev iteration configuration */ -#define WT_CBT_NO_TXN 0x10 /* Non-transactional cursor - (e.g. on a checkpoint) */ -#define WT_CBT_SEARCH_SMALLEST 0x20 /* Row-store: small-key insert list */ -#define WT_CBT_VAR_ONPAGE_MATCH 0x40 /* Var-store: on-page recno match */ +/* AUTOMATIC FLAG VALUE GENERATION START */ +#define WT_CBT_ACTIVE 0x01u /* Active in the tree */ +#define WT_CBT_ITERATE_APPEND 0x02u /* Col-store: iterating append list */ +#define WT_CBT_ITERATE_NEXT 0x04u /* Next iteration configuration */ +#define WT_CBT_ITERATE_PREV 0x08u /* Prev iteration configuration */ +#define WT_CBT_NO_TXN 0x10u /* Non-txn cursor (e.g. a checkpoint) */ +#define WT_CBT_SEARCH_SMALLEST 0x20u /* Row-store: small-key insert list */ +#define WT_CBT_VAR_ONPAGE_MATCH 0x40u /* Var-store: on-page recno match */ +/* AUTOMATIC FLAG VALUE GENERATION STOP */ #define WT_CBT_POSITION_MASK /* Flags associated with position */ \ (WT_CBT_ITERATE_APPEND | WT_CBT_ITERATE_NEXT | WT_CBT_ITERATE_PREV | \ @@ -336,12 +339,14 @@ struct __wt_cursor_join_endpoint { uint8_t recno_buf[10]; /* holds packed recno */ WT_CURSOR *cursor; -#define WT_CURJOIN_END_LT 0x01 /* include values < cursor */ -#define WT_CURJOIN_END_EQ 0x02 /* include values == cursor */ -#define WT_CURJOIN_END_GT 0x04 /* include values > cursor */ +/* AUTOMATIC FLAG VALUE GENERATION START */ +#define WT_CURJOIN_END_EQ 0x1u /* include values == cursor */ +#define WT_CURJOIN_END_GT 0x2u /* include values > cursor */ +#define WT_CURJOIN_END_LT 0x4u /* include values < cursor */ +#define WT_CURJOIN_END_OWN_CURSOR 0x8u /* must close cursor */ +/* AUTOMATIC FLAG VALUE GENERATION STOP */ #define WT_CURJOIN_END_GE (WT_CURJOIN_END_GT | WT_CURJOIN_END_EQ) #define WT_CURJOIN_END_LE (WT_CURJOIN_END_LT | WT_CURJOIN_END_EQ) -#define WT_CURJOIN_END_OWN_CURSOR 0x08 /* must close cursor */ uint8_t flags; /* range for this endpoint */ }; #define WT_CURJOIN_END_RANGE(endp) \ @@ -365,11 +370,12 @@ struct __wt_cursor_join_entry { uint32_t bloom_hash_count; /* hash functions in bloom */ uint64_t count; /* approx number of matches */ -#define WT_CURJOIN_ENTRY_BLOOM 0x01 /* use a bloom filter */ -#define WT_CURJOIN_ENTRY_DISJUNCTION 0x02 /* endpoints are or-ed */ -#define WT_CURJOIN_ENTRY_FALSE_POSITIVES 0x04 /* after bloom filter do not - * filter false positives */ -#define WT_CURJOIN_ENTRY_OWN_BLOOM 0x08 /* this entry owns the bloom */ +/* AUTOMATIC FLAG VALUE GENERATION START */ +#define WT_CURJOIN_ENTRY_BLOOM 0x1u /* use a bloom filter */ +#define WT_CURJOIN_ENTRY_DISJUNCTION 0x2u /* endpoints are or-ed */ +#define WT_CURJOIN_ENTRY_FALSE_POSITIVES 0x4u /* don't filter false pos */ +#define WT_CURJOIN_ENTRY_OWN_BLOOM 0x8u /* this entry owns the bloom */ +/* AUTOMATIC FLAG VALUE GENERATION STOP */ uint8_t flags; WT_CURSOR_JOIN_ENDPOINT *ends; /* reference endpoints */ @@ -392,9 +398,11 @@ struct __wt_cursor_join { u_int entries_next; uint8_t recno_buf[10]; /* holds packed recno */ -#define WT_CURJOIN_DISJUNCTION 0x01 /* Entries are or-ed */ -#define WT_CURJOIN_ERROR 0x02 /* Error in initialization */ -#define WT_CURJOIN_INITIALIZED 0x04 /* Successful initialization */ +/* AUTOMATIC FLAG VALUE GENERATION START */ +#define WT_CURJOIN_DISJUNCTION 0x1u /* Entries are or-ed */ +#define WT_CURJOIN_ERROR 0x2u /* Error in initialization */ +#define WT_CURJOIN_INITIALIZED 0x4u /* Successful initialization */ +/* AUTOMATIC FLAG VALUE GENERATION STOP */ uint8_t flags; }; @@ -419,7 +427,9 @@ struct __wt_cursor_log { uint32_t rectype; /* Record type */ uint64_t txnid; /* Record txnid */ -#define WT_CURLOG_ARCHIVE_LOCK 0x01 /* Archive lock held */ +/* AUTOMATIC FLAG VALUE GENERATION START */ +#define WT_CURLOG_ARCHIVE_LOCK 0x1u /* Archive lock held */ +/* AUTOMATIC FLAG VALUE GENERATION STOP */ uint8_t flags; }; @@ -429,9 +439,11 @@ struct __wt_cursor_metadata { WT_CURSOR *file_cursor; /* Queries of regular metadata */ WT_CURSOR *create_cursor; /* Extra cursor for create option */ -#define WT_MDC_CREATEONLY 0x01 -#define WT_MDC_ONMETADATA 0x02 -#define WT_MDC_POSITIONED 0x04 +/* AUTOMATIC FLAG VALUE GENERATION START */ +#define WT_MDC_CREATEONLY 0x1u +#define WT_MDC_ONMETADATA 0x2u +#define WT_MDC_POSITIONED 0x4u +/* AUTOMATIC FLAG VALUE GENERATION STOP */ uint8_t flags; }; diff --git a/src/third_party/wiredtiger/src/include/cursor.i b/src/third_party/wiredtiger/src/include/cursor.i index ceef5061075..a4e986c4325 100644 --- a/src/third_party/wiredtiger/src/include/cursor.i +++ b/src/third_party/wiredtiger/src/include/cursor.i @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/include/dhandle.h b/src/third_party/wiredtiger/src/include/dhandle.h index 2844c6e3e73..a18881d8ea9 100644 --- a/src/third_party/wiredtiger/src/include/dhandle.h +++ b/src/third_party/wiredtiger/src/include/dhandle.h @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -48,11 +48,11 @@ if ((dhandle) == NULL) \ (dhandle) = TAILQ_FIRST(head); \ else { \ - WT_DHANDLE_RELEASE(dhandle); \ - (dhandle) = TAILQ_NEXT(dhandle, field); \ + WT_DHANDLE_RELEASE(dhandle); \ + (dhandle) = TAILQ_NEXT(dhandle, field); \ } \ if ((dhandle) != NULL) \ - WT_DHANDLE_ACQUIRE(dhandle); \ + WT_DHANDLE_ACQUIRE(dhandle); \ } while (0) /* @@ -103,12 +103,14 @@ struct __wt_data_handle { WT_DSRC_STATS *stat_array; /* Flags values over 0xff are reserved for WT_BTREE_* */ -#define WT_DHANDLE_DEAD 0x01 /* Dead, awaiting discard */ -#define WT_DHANDLE_DISCARD 0x02 /* Close on release */ -#define WT_DHANDLE_DISCARD_KILL 0x04 /* Mark dead on release */ -#define WT_DHANDLE_EXCLUSIVE 0x08 /* Exclusive access */ -#define WT_DHANDLE_IS_METADATA 0x10 /* Metadata handle */ -#define WT_DHANDLE_LOCK_ONLY 0x20 /* Handle only used as a lock */ -#define WT_DHANDLE_OPEN 0x40 /* Handle is open */ +/* AUTOMATIC FLAG VALUE GENERATION START */ +#define WT_DHANDLE_DEAD 0x01u /* Dead, awaiting discard */ +#define WT_DHANDLE_DISCARD 0x02u /* Close on release */ +#define WT_DHANDLE_DISCARD_KILL 0x04u /* Mark dead on release */ +#define WT_DHANDLE_EXCLUSIVE 0x08u /* Exclusive access */ +#define WT_DHANDLE_IS_METADATA 0x10u /* Metadata handle */ +#define WT_DHANDLE_LOCK_ONLY 0x20u /* Handle only used as a lock */ +#define WT_DHANDLE_OPEN 0x40u /* Handle is open */ +/* AUTOMATIC FLAG VALUE GENERATION STOP */ uint32_t flags; }; diff --git a/src/third_party/wiredtiger/src/include/dlh.h b/src/third_party/wiredtiger/src/include/dlh.h index d02523b03d1..a5fcaf17355 100644 --- a/src/third_party/wiredtiger/src/include/dlh.h +++ b/src/third_party/wiredtiger/src/include/dlh.h @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/include/error.h b/src/third_party/wiredtiger/src/include/error.h index 3b3c0769c90..a4ef4757ec9 100644 --- a/src/third_party/wiredtiger/src/include/error.h +++ b/src/third_party/wiredtiger/src/include/error.h @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -61,7 +61,6 @@ } while (0) #define WT_RET_BUSY_OK(a) WT_RET_ERROR_OK(a, EBUSY) #define WT_RET_NOTFOUND_OK(a) WT_RET_ERROR_OK(a, WT_NOTFOUND) - /* Set "ret" if not already set. */ #define WT_TRET(a) do { \ int __ret; \ @@ -137,15 +136,7 @@ * there's no portable way to remove the comma before an empty __VA_ARGS__ * value. */ -#ifdef HAVE_VERBOSE #define __wt_verbose(session, flag, fmt, ...) do { \ if (WT_VERBOSE_ISSET(session, flag)) \ __wt_verbose_worker(session, fmt, __VA_ARGS__); \ } while (0) -#else -#define __wt_verbose(session, flag, fmt, ...) do { \ - WT_UNUSED(session); \ - WT_UNUSED(flag); \ - WT_UNUSED(fmt); \ -} while (0) -#endif diff --git a/src/third_party/wiredtiger/src/include/extern.h b/src/third_party/wiredtiger/src/include/extern.h index 272544b08f7..3674d9218da 100644 --- a/src/third_party/wiredtiger/src/include/extern.h +++ b/src/third_party/wiredtiger/src/include/extern.h @@ -129,7 +129,6 @@ extern int __wt_delete_page(WT_SESSION_IMPL *session, WT_REF *ref, bool *skipp) extern void __wt_delete_page_rollback(WT_SESSION_IMPL *session, WT_REF *ref); extern bool __wt_delete_page_skip(WT_SESSION_IMPL *session, WT_REF *ref, bool visible_all); extern int __wt_delete_page_instantiate(WT_SESSION_IMPL *session, WT_REF *ref) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); -extern void __wt_ref_out_int(WT_SESSION_IMPL *session, WT_REF *ref, bool rewrite); extern void __wt_ref_out(WT_SESSION_IMPL *session, WT_REF *ref); extern void __wt_page_out(WT_SESSION_IMPL *session, WT_PAGE **pagep); extern void __wt_free_ref( WT_SESSION_IMPL *session, WT_REF *ref, int page_type, bool free_pages); @@ -198,7 +197,6 @@ extern int __wt_row_modify(WT_SESSION_IMPL *session, WT_CURSOR_BTREE *cbt, const extern int __wt_row_insert_alloc(WT_SESSION_IMPL *session, const WT_ITEM *key, u_int skipdepth, WT_INSERT **insp, size_t *ins_sizep) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_update_alloc(WT_SESSION_IMPL *session, const WT_ITEM *value, WT_UPDATE **updp, size_t *sizep, u_int modify_type) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern WT_UPDATE *__wt_update_obsolete_check( WT_SESSION_IMPL *session, WT_PAGE *page, WT_UPDATE *upd); -extern void __wt_update_obsolete_free( WT_SESSION_IMPL *session, WT_PAGE *page, WT_UPDATE *upd); extern int __wt_search_insert(WT_SESSION_IMPL *session, WT_CURSOR_BTREE *cbt, WT_INSERT_HEAD *ins_head, WT_ITEM *srch_key) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_row_search(WT_SESSION_IMPL *session, WT_ITEM *srch_key, WT_REF *leaf, WT_CURSOR_BTREE *cbt, bool insert, bool restore) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern bool __wt_las_nonempty(WT_SESSION_IMPL *session); @@ -286,6 +284,8 @@ extern int __wt_connection_open(WT_CONNECTION_IMPL *conn, const char *cfg[]) WT_ extern int __wt_connection_close(WT_CONNECTION_IMPL *conn) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_connection_workers(WT_SESSION_IMPL *session, const char *cfg[]) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_conn_compat_config(WT_SESSION_IMPL *session, const char **cfg) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); +extern int __wt_conn_optrack_setup(WT_SESSION_IMPL *session, const char *cfg[], bool reconfig) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); +extern int __wt_conn_optrack_teardown(WT_SESSION_IMPL *session, bool reconfig) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_conn_statistics_config(WT_SESSION_IMPL *session, const char *cfg[]) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_conn_reconfig(WT_SESSION_IMPL *session, const char **cfg) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern void __wt_conn_stat_init(WT_SESSION_IMPL *session); @@ -367,7 +367,7 @@ extern int __wt_evict_create(WT_SESSION_IMPL *session) WT_GCC_FUNC_DECL_ATTRIBUT extern int __wt_evict_destroy(WT_SESSION_IMPL *session) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_evict_file_exclusive_on(WT_SESSION_IMPL *session) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern void __wt_evict_file_exclusive_off(WT_SESSION_IMPL *session); -extern int __wt_cache_eviction_worker( WT_SESSION_IMPL *session, bool busy, bool readonly, u_int pct_full) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); +extern int __wt_cache_eviction_worker( WT_SESSION_IMPL *session, bool busy, bool readonly, double pct_full) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern bool __wt_page_evict_urgent(WT_SESSION_IMPL *session, WT_REF *ref); extern void __wt_evict_priority_set(WT_SESSION_IMPL *session, uint64_t v); extern void __wt_evict_priority_clear(WT_SESSION_IMPL *session); @@ -375,7 +375,7 @@ extern int __wt_verbose_dump_cache(WT_SESSION_IMPL *session) WT_GCC_FUNC_DECL_AT extern int __wt_page_release_evict(WT_SESSION_IMPL *session, WT_REF *ref) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_evict(WT_SESSION_IMPL *session, WT_REF *ref, bool closing) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern void __wt_curstat_cache_walk(WT_SESSION_IMPL *session); -extern int __wt_log_printf(WT_SESSION_IMPL *session, const char *fmt, ...) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); +extern int __wt_log_printf(WT_SESSION_IMPL *session, const char *format, ...) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern void __wt_log_ckpt(WT_SESSION_IMPL *session, WT_LSN *ckp_lsn); extern int __wt_log_flush_lsn(WT_SESSION_IMPL *session, WT_LSN *lsn, bool start) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern void __wt_log_background(WT_SESSION_IMPL *session, WT_LSN *lsn); @@ -464,8 +464,8 @@ extern int __wt_lsm_meta_write(WT_SESSION_IMPL *session, WT_LSM_TREE *lsm_tree, extern int __wt_curstat_lsm_init( WT_SESSION_IMPL *session, const char *uri, WT_CURSOR_STAT *cst) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_lsm_tree_close_all(WT_SESSION_IMPL *session) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_lsm_tree_bloom_name(WT_SESSION_IMPL *session, WT_LSM_TREE *lsm_tree, uint32_t id, const char **retp) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); -extern int __wt_lsm_tree_chunk_name(WT_SESSION_IMPL *session, WT_LSM_TREE *lsm_tree, uint32_t id, const char **retp) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); -extern int __wt_lsm_tree_set_chunk_size( WT_SESSION_IMPL *session, WT_LSM_CHUNK *chunk) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); +extern int __wt_lsm_tree_chunk_name(WT_SESSION_IMPL *session, WT_LSM_TREE *lsm_tree, uint32_t id, uint32_t generation, const char **retp) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); +extern int __wt_lsm_tree_set_chunk_size( WT_SESSION_IMPL *session, WT_LSM_TREE *lsm_tree, WT_LSM_CHUNK *chunk) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_lsm_tree_setup_chunk( WT_SESSION_IMPL *session, WT_LSM_TREE *lsm_tree, WT_LSM_CHUNK *chunk) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_lsm_tree_setup_bloom( WT_SESSION_IMPL *session, WT_LSM_TREE *lsm_tree, WT_LSM_CHUNK *chunk) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_lsm_tree_create(WT_SESSION_IMPL *session, const char *uri, bool exclusive, const char *config) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); @@ -528,8 +528,12 @@ extern int __wt_meta_track_destroy(WT_SESSION_IMPL *session) WT_GCC_FUNC_DECL_AT extern int __wt_turtle_init(WT_SESSION_IMPL *session) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_turtle_read(WT_SESSION_IMPL *session, const char *key, char **valuep) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_turtle_update(WT_SESSION_IMPL *session, const char *key, const char *value) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); +extern void __wt_optrack_record_funcid( WT_SESSION_IMPL *session, const char *func, uint16_t *func_idp); +extern int __wt_optrack_open_file(WT_SESSION_IMPL *session) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); +extern size_t __wt_optrack_flush_buffer(WT_SESSION_IMPL *s); extern int __wt_filename(WT_SESSION_IMPL *session, const char *name, char **path) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_nfilename( WT_SESSION_IMPL *session, const char *name, size_t namelen, char **path) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); +extern int __wt_filename_construct(WT_SESSION_IMPL *session, const char *path, const char *file_prefix, uintmax_t id_1, uint32_t id_2, WT_ITEM *buf) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_remove_if_exists(WT_SESSION_IMPL *session, const char *name, bool durable) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_copy_and_sync(WT_SESSION *wt_session, const char *from, const char *to) WT_GCC_FUNC_DECL_ATTRIBUTE((visibility("default"))) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern void __wt_abort(WT_SESSION_IMPL *session) WT_GCC_FUNC_DECL_ATTRIBUTE((noreturn)); @@ -553,14 +557,14 @@ extern int __wt_fopen(WT_SESSION_IMPL *session, const char *name, uint32_t open_ extern void __wt_os_stdio(WT_SESSION_IMPL *session); extern int __wt_getopt( const char *progname, int nargc, char *const *nargv, const char *ostr) WT_GCC_FUNC_DECL_ATTRIBUTE((visibility("default"))) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern uint64_t __wt_strtouq(const char *nptr, char **endptr, int base) WT_GCC_FUNC_DECL_ATTRIBUTE((visibility("default"))); -extern int __wt_ext_struct_pack(WT_EXTENSION_API *wt_api, WT_SESSION *wt_session, void *buffer, size_t size, const char *fmt, ...) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); -extern int __wt_ext_struct_size(WT_EXTENSION_API *wt_api, WT_SESSION *wt_session, size_t *sizep, const char *fmt, ...) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); -extern int __wt_ext_struct_unpack(WT_EXTENSION_API *wt_api, WT_SESSION *wt_session, const void *buffer, size_t size, const char *fmt, ...) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); +extern int __wt_ext_struct_pack(WT_EXTENSION_API *wt_api, WT_SESSION *wt_session, void *buffer, size_t len, const char *fmt, ...) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); +extern int __wt_ext_struct_size(WT_EXTENSION_API *wt_api, WT_SESSION *wt_session, size_t *lenp, const char *fmt, ...) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); +extern int __wt_ext_struct_unpack(WT_EXTENSION_API *wt_api, WT_SESSION *wt_session, const void *buffer, size_t len, const char *fmt, ...) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_struct_check(WT_SESSION_IMPL *session, const char *fmt, size_t len, bool *fixedp, uint32_t *fixed_lenp) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_struct_confchk(WT_SESSION_IMPL *session, WT_CONFIG_ITEM *v) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); -extern int __wt_struct_size(WT_SESSION_IMPL *session, size_t *sizep, const char *fmt, ...) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); -extern int __wt_struct_pack(WT_SESSION_IMPL *session, void *buffer, size_t size, const char *fmt, ...) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); -extern int __wt_struct_unpack(WT_SESSION_IMPL *session, const void *buffer, size_t size, const char *fmt, ...) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); +extern int __wt_struct_size(WT_SESSION_IMPL *session, size_t *lenp, const char *fmt, ...) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); +extern int __wt_struct_pack(WT_SESSION_IMPL *session, void *buffer, size_t len, const char *fmt, ...) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); +extern int __wt_struct_unpack(WT_SESSION_IMPL *session, const void *buffer, size_t len, const char *fmt, ...) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_struct_repack(WT_SESSION_IMPL *session, const char *infmt, const char *outfmt, const WT_ITEM *inbuf, WT_ITEM *outbuf) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_ext_pack_start(WT_EXTENSION_API *wt_api, WT_SESSION *wt_session, const char *format, void *buffer, size_t size, WT_PACK_STREAM **psp) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_ext_unpack_start(WT_EXTENSION_API *wt_api, WT_SESSION *wt_session, const char *format, const void *buffer, size_t size, WT_PACK_STREAM **psp) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); @@ -637,6 +641,7 @@ extern int __wt_open_cursor(WT_SESSION_IMPL *session, const char *uri, WT_CURSOR extern int __wt_session_create( WT_SESSION_IMPL *session, const char *uri, const char *config) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_session_range_truncate(WT_SESSION_IMPL *session, const char *uri, WT_CURSOR *start, WT_CURSOR *stop) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern const char *__wt_session_strerror(WT_SESSION *wt_session, int error); +extern int __wt_session_breakpoint(WT_SESSION *wt_session) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_open_session(WT_CONNECTION_IMPL *conn, WT_EVENT_HANDLER *event_handler, const char *config, bool open_metadata, WT_SESSION_IMPL **sessionp) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_open_internal_session(WT_CONNECTION_IMPL *conn, const char *name, bool open_metadata, uint32_t session_flags, WT_SESSION_IMPL **sessionp) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_session_compact_check_timeout(WT_SESSION_IMPL *session) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); @@ -693,7 +698,6 @@ extern void __wt_stash_discard(WT_SESSION_IMPL *session); extern int __wt_stash_add(WT_SESSION_IMPL *session, int which, uint64_t generation, void *p, size_t len) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern void __wt_stash_discard_all(WT_SESSION_IMPL *session_safe, WT_SESSION_IMPL *session); extern int __wt_library_init(void) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); -extern void __wt_breakpoint(void); extern uint64_t __wt_hash_city64(const void *s, size_t len); extern uint64_t __wt_hash_fnv64(const void *string, size_t len); extern int @@ -719,8 +723,8 @@ extern void __wt_print_huffman_code(void *huffman_arg, uint16_t symbol); extern int __wt_huffman_encode(WT_SESSION_IMPL *session, void *huffman_arg, const uint8_t *from_arg, size_t from_len, WT_ITEM *to_buf) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_huffman_decode(WT_SESSION_IMPL *session, void *huffman_arg, const uint8_t *from_arg, size_t from_len, WT_ITEM *to_buf) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_modify_pack(WT_SESSION_IMPL *session, WT_ITEM **modifyp, WT_MODIFY *entries, int nentries) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); -extern int __wt_modify_apply_api( WT_SESSION_IMPL *session, WT_ITEM *value, WT_MODIFY *entries, int nentries) WT_GCC_FUNC_DECL_ATTRIBUTE((visibility("default"))) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); -extern int __wt_modify_apply(WT_SESSION_IMPL *session, WT_ITEM *value, const void *modify) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); +extern int __wt_modify_apply_api(WT_SESSION_IMPL *session, WT_CURSOR *cursor, WT_MODIFY *entries, int nentries) WT_GCC_FUNC_DECL_ATTRIBUTE((visibility("default"))) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); +extern int __wt_modify_apply( WT_SESSION_IMPL *session, WT_CURSOR *cursor, const void *modify) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_rwlock_init(WT_SESSION_IMPL *session, WT_RWLOCK *l) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern void __wt_rwlock_destroy(WT_SESSION_IMPL *session, WT_RWLOCK *l); extern int __wt_try_readlock(WT_SESSION_IMPL *session, WT_RWLOCK *l) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); @@ -738,17 +742,19 @@ extern uint32_t __wt_rduppo2(uint32_t n, uint32_t po2); extern void __wt_random_init(WT_RAND_STATE volatile *rnd_state) WT_GCC_FUNC_DECL_ATTRIBUTE((visibility("default"))); extern void __wt_random_init_seed( WT_SESSION_IMPL *session, WT_RAND_STATE volatile *rnd_state) WT_GCC_FUNC_DECL_ATTRIBUTE((visibility("default"))); extern uint32_t __wt_random(WT_RAND_STATE volatile *rnd_state) WT_GCC_FUNC_DECL_ATTRIBUTE((visibility("default"))); -extern int __wt_buf_grow_worker(WT_SESSION_IMPL *session, WT_ITEM *buf, size_t size) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); -extern int __wt_buf_fmt(WT_SESSION_IMPL *session, WT_ITEM *buf, const char *fmt, ...) WT_GCC_FUNC_DECL_ATTRIBUTE((format (printf, 3, 4))) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); -extern int __wt_buf_catfmt(WT_SESSION_IMPL *session, WT_ITEM *buf, const char *fmt, ...) WT_GCC_FUNC_DECL_ATTRIBUTE((format (printf, 3, 4))) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); +extern int __wt_buf_grow_worker(WT_SESSION_IMPL *session, WT_ITEM *buf, size_t size) WT_GCC_FUNC_DECL_ATTRIBUTE((visibility("default"))) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); +extern int __wt_buf_fmt(WT_SESSION_IMPL *session, WT_ITEM *buf, const char *fmt, ...) WT_GCC_FUNC_DECL_ATTRIBUTE((format (printf, 3, 4))) WT_GCC_FUNC_DECL_ATTRIBUTE((visibility("default"))) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); +extern int __wt_buf_catfmt(WT_SESSION_IMPL *session, WT_ITEM *buf, const char *fmt, ...) WT_GCC_FUNC_DECL_ATTRIBUTE((format (printf, 3, 4))) WT_GCC_FUNC_DECL_ATTRIBUTE((visibility("default"))) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern const char *__wt_buf_set_printable( WT_SESSION_IMPL *session, const void *p, size_t size, WT_ITEM *buf); +extern const char *__wt_buf_set_printable_format(WT_SESSION_IMPL *session, const void *buffer, size_t size, const char *format, WT_ITEM *buf); extern const char *__wt_buf_set_size( WT_SESSION_IMPL *session, uint64_t size, bool exact, WT_ITEM *buf); extern int __wt_scr_alloc_func(WT_SESSION_IMPL *session, size_t size, WT_ITEM **scratchp #ifdef HAVE_DIAGNOSTIC , const char *file, int line #endif - ); + ) + WT_GCC_FUNC_DECL_ATTRIBUTE((visibility("default"))); extern void __wt_scr_discard(WT_SESSION_IMPL *session); extern void *__wt_ext_scr_alloc( WT_EXTENSION_API *wt_api, WT_SESSION *wt_session, size_t size); extern void __wt_ext_scr_free(WT_EXTENSION_API *wt_api, WT_SESSION *wt_session, void *p); @@ -779,6 +785,7 @@ extern void __wt_thread_group_start_one( WT_SESSION_IMPL *session, WT_THREAD_GRO extern void __wt_thread_group_stop_one(WT_SESSION_IMPL *session, WT_THREAD_GROUP *group); extern void __wt_epoch(WT_SESSION_IMPL *session, struct timespec *tsp) WT_GCC_FUNC_DECL_ATTRIBUTE((visibility("default"))); extern void __wt_seconds(WT_SESSION_IMPL *session, time_t *timep); +extern uint64_t __wt_clock_to_nsec(uint64_t end, uint64_t begin); extern void __wt_txn_release_snapshot(WT_SESSION_IMPL *session); extern void __wt_txn_get_snapshot(WT_SESSION_IMPL *session); extern int __wt_txn_update_oldest(WT_SESSION_IMPL *session, uint32_t flags) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); @@ -787,6 +794,7 @@ extern int __wt_txn_reconfigure(WT_SESSION_IMPL *session, const char *config) WT extern void __wt_txn_release(WT_SESSION_IMPL *session); extern int __wt_txn_commit(WT_SESSION_IMPL *session, const char *cfg[]) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_txn_rollback(WT_SESSION_IMPL *session, const char *cfg[]) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); +extern int __wt_txn_rollback_required(WT_SESSION_IMPL *session, const char *reason) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_txn_init(WT_SESSION_IMPL *session, WT_SESSION_IMPL *session_ret) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern void __wt_txn_stats_update(WT_SESSION_IMPL *session); extern void __wt_txn_destroy(WT_SESSION_IMPL *session); diff --git a/src/third_party/wiredtiger/src/include/extern_posix.h b/src/third_party/wiredtiger/src/include/extern_posix.h index 624cd815dad..d4119fdb07a 100644 --- a/src/third_party/wiredtiger/src/include/extern_posix.h +++ b/src/third_party/wiredtiger/src/include/extern_posix.h @@ -1,6 +1,7 @@ /* DO NOT EDIT: automatically built by dist/s_prototypes. */ extern int __wt_posix_directory_list(WT_FILE_SYSTEM *file_system, WT_SESSION *wt_session, const char *directory, const char *prefix, char ***dirlistp, uint32_t *countp) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); +extern int __wt_posix_directory_list_single(WT_FILE_SYSTEM *file_system, WT_SESSION *wt_session, const char *directory, const char *prefix, char ***dirlistp, uint32_t *countp) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_posix_directory_list_free(WT_FILE_SYSTEM *file_system, WT_SESSION *wt_session, char **dirlist, uint32_t count) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_dlopen(WT_SESSION_IMPL *session, const char *path, WT_DLH **dlhp) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_dlsym(WT_SESSION_IMPL *session, WT_DLH *dlh, const char *name, bool fail, void *sym_ret) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); @@ -29,5 +30,6 @@ extern int __wt_thread_create(WT_SESSION_IMPL *session, wt_thread_t *tidret, WT_ extern int __wt_thread_join(WT_SESSION_IMPL *session, wt_thread_t tid) WT_GCC_FUNC_DECL_ATTRIBUTE((visibility("default"))) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern void __wt_thread_id(uintmax_t *id) WT_GCC_FUNC_DECL_ATTRIBUTE((visibility("default"))); extern int __wt_thread_str(char *buf, size_t buflen) WT_GCC_FUNC_DECL_ATTRIBUTE((visibility("default"))) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); +extern uintmax_t __wt_process_id(void); extern void __wt_epoch_raw(WT_SESSION_IMPL *session, struct timespec *tsp); extern void __wt_yield(void) WT_GCC_FUNC_DECL_ATTRIBUTE((visibility("default"))); diff --git a/src/third_party/wiredtiger/src/include/extern_win.h b/src/third_party/wiredtiger/src/include/extern_win.h index ae4195b20a1..bb5329127f5 100644 --- a/src/third_party/wiredtiger/src/include/extern_win.h +++ b/src/third_party/wiredtiger/src/include/extern_win.h @@ -1,6 +1,7 @@ /* DO NOT EDIT: automatically built by dist/s_prototypes. */ extern int __wt_win_directory_list(WT_FILE_SYSTEM *file_system, WT_SESSION *wt_session, const char *directory, const char *prefix, char ***dirlistp, uint32_t *countp) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); +extern int __wt_win_directory_list_single(WT_FILE_SYSTEM *file_system, WT_SESSION *wt_session, const char *directory, const char *prefix, char ***dirlistp, uint32_t *countp) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_win_directory_list_free(WT_FILE_SYSTEM *file_system, WT_SESSION *wt_session, char **dirlist, uint32_t count) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_dlopen(WT_SESSION_IMPL *session, const char *path, WT_DLH **dlhp) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_dlsym(WT_SESSION_IMPL *session, WT_DLH *dlh, const char *name, bool fail, void *sym_ret) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); @@ -27,6 +28,7 @@ extern int __wt_thread_create(WT_SESSION_IMPL *session, wt_thread_t *tidret, WT_ extern int __wt_thread_join(WT_SESSION_IMPL *session, wt_thread_t tid) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern void __wt_thread_id(uintmax_t *id); extern int __wt_thread_str(char *buf, size_t buflen) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); +extern uintmax_t __wt_process_id(void); extern void __wt_epoch_raw(WT_SESSION_IMPL *session, struct timespec *tsp); extern int __wt_to_utf16_string( WT_SESSION_IMPL *session, const char*utf8, WT_ITEM **outbuf) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); extern int __wt_to_utf8_string( WT_SESSION_IMPL *session, const wchar_t*wide, WT_ITEM **outbuf) WT_GCC_FUNC_DECL_ATTRIBUTE((warn_unused_result)); diff --git a/src/third_party/wiredtiger/src/include/flags.h b/src/third_party/wiredtiger/src/include/flags.h deleted file mode 100644 index b191e8fe01d..00000000000 --- a/src/third_party/wiredtiger/src/include/flags.h +++ /dev/null @@ -1,130 +0,0 @@ -/* - * DO NOT EDIT: automatically built by dist/flags.py. - * flags section: BEGIN - */ -#define WT_CONN_CACHE_POOL 0x00000001 -#define WT_CONN_CKPT_SYNC 0x00000002 -#define WT_CONN_CLOSING 0x00000004 -#define WT_CONN_CLOSING_NO_MORE_OPENS 0x00000008 -#define WT_CONN_EVICTION_NO_LOOKASIDE 0x00000010 -#define WT_CONN_EVICTION_RUN 0x00000020 -#define WT_CONN_IN_MEMORY 0x00000040 -#define WT_CONN_LEAK_MEMORY 0x00000080 -#define WT_CONN_LOOKASIDE_OPEN 0x00000100 -#define WT_CONN_LSM_MERGE 0x00000200 -#define WT_CONN_PANIC 0x00000400 -#define WT_CONN_READONLY 0x00000800 -#define WT_CONN_RECOVERING 0x00001000 -#define WT_CONN_SERVER_ASYNC 0x00002000 -#define WT_CONN_SERVER_CHECKPOINT 0x00004000 -#define WT_CONN_SERVER_LOG 0x00008000 -#define WT_CONN_SERVER_LSM 0x00010000 -#define WT_CONN_SERVER_STATISTICS 0x00020000 -#define WT_CONN_SERVER_SWEEP 0x00040000 -#define WT_CONN_WAS_BACKUP 0x00080000 -#define WT_LOGSCAN_FIRST 0x00000001 -#define WT_LOGSCAN_FROM_CKP 0x00000002 -#define WT_LOGSCAN_ONE 0x00000004 -#define WT_LOGSCAN_RECOVER 0x00000008 -#define WT_LOG_BACKGROUND 0x00000001 -#define WT_LOG_DSYNC 0x00000002 -#define WT_LOG_FLUSH 0x00000004 -#define WT_LOG_FSYNC 0x00000008 -#define WT_LOG_SYNC_ENABLED 0x00000010 -#define WT_READ_CACHE 0x00000001 -#define WT_READ_IGNORE_CACHE_SIZE 0x00000002 -#define WT_READ_LOOKASIDE 0x00000004 -#define WT_READ_NOTFOUND_OK 0x00000008 -#define WT_READ_NO_EMPTY 0x00000010 -#define WT_READ_NO_GEN 0x00000020 -#define WT_READ_NO_SPLIT 0x00000040 -#define WT_READ_NO_WAIT 0x00000080 -#define WT_READ_PREV 0x00000100 -#define WT_READ_RESTART_OK 0x00000200 -#define WT_READ_SKIP_INTL 0x00000400 -#define WT_READ_TRUNCATE 0x00000800 -#define WT_READ_WONT_NEED 0x00001000 -#define WT_REC_CHECKPOINT 0x00000001 -#define WT_REC_EVICT 0x00000002 -#define WT_REC_IN_MEMORY 0x00000004 -#define WT_REC_LOOKASIDE 0x00000008 -#define WT_REC_SCRUB 0x00000010 -#define WT_REC_UPDATE_RESTORE 0x00000020 -#define WT_REC_VISIBILITY_ERR 0x00000040 -#define WT_REC_VISIBLE_ALL 0x00000080 -#define WT_SESSION_CAN_WAIT 0x00000001 -#define WT_SESSION_IGNORE_CACHE_SIZE 0x00000002 -#define WT_SESSION_INTERNAL 0x00000004 -#define WT_SESSION_LOCKED_CHECKPOINT 0x00000008 -#define WT_SESSION_LOCKED_HANDLE_LIST_READ 0x00000010 -#define WT_SESSION_LOCKED_HANDLE_LIST_WRITE 0x00000020 -#define WT_SESSION_LOCKED_METADATA 0x00000040 -#define WT_SESSION_LOCKED_PASS 0x00000080 -#define WT_SESSION_LOCKED_SCHEMA 0x00000100 -#define WT_SESSION_LOCKED_SLOT 0x00000200 -#define WT_SESSION_LOCKED_TABLE_READ 0x00000400 -#define WT_SESSION_LOCKED_TABLE_WRITE 0x00000800 -#define WT_SESSION_LOCKED_TURTLE 0x00001000 -#define WT_SESSION_LOGGING_INMEM 0x00002000 -#define WT_SESSION_LOOKASIDE_CURSOR 0x00004000 -#define WT_SESSION_NO_DATA_HANDLES 0x00008000 -#define WT_SESSION_NO_LOGGING 0x00010000 -#define WT_SESSION_NO_RECONCILE 0x00020000 -#define WT_SESSION_NO_SCHEMA_LOCK 0x00040000 -#define WT_SESSION_QUIET_CORRUPT_FILE 0x00080000 -#define WT_SESSION_READ_WONT_NEED 0x00100000 -#define WT_SESSION_SERVER_ASYNC 0x00200000 -#define WT_STAT_CLEAR 0x00000001 -#define WT_STAT_JSON 0x00000002 -#define WT_STAT_ON_CLOSE 0x00000004 -#define WT_STAT_TYPE_ALL 0x00000008 -#define WT_STAT_TYPE_CACHE_WALK 0x00000010 -#define WT_STAT_TYPE_FAST 0x00000020 -#define WT_STAT_TYPE_SIZE 0x00000040 -#define WT_STAT_TYPE_TREE_WALK 0x00000080 -#define WT_TIMING_STRESS_CHECKPOINT_SLOW 0x00000001 -#define WT_TIMING_STRESS_INTERNAL_PAGE_SPLIT_RACE 0x00000002 -#define WT_TIMING_STRESS_PAGE_SPLIT_RACE 0x00000004 -#define WT_TXN_LOG_CKPT_CLEANUP 0x00000001 -#define WT_TXN_LOG_CKPT_PREPARE 0x00000002 -#define WT_TXN_LOG_CKPT_START 0x00000004 -#define WT_TXN_LOG_CKPT_STOP 0x00000008 -#define WT_TXN_LOG_CKPT_SYNC 0x00000010 -#define WT_TXN_OLDEST_STRICT 0x00000001 -#define WT_TXN_OLDEST_WAIT 0x00000002 -#define WT_VERB_API 0x00000001 -#define WT_VERB_BLOCK 0x00000002 -#define WT_VERB_CHECKPOINT 0x00000004 -#define WT_VERB_CHECKPOINT_PROGRESS 0x00000008 -#define WT_VERB_COMPACT 0x00000010 -#define WT_VERB_EVICT 0x00000020 -#define WT_VERB_EVICTSERVER 0x00000040 -#define WT_VERB_EVICT_STUCK 0x00000080 -#define WT_VERB_FILEOPS 0x00000100 -#define WT_VERB_HANDLEOPS 0x00000200 -#define WT_VERB_LOG 0x00000400 -#define WT_VERB_LOOKASIDE 0x00000800 -#define WT_VERB_LOOKASIDE_ACTIVITY 0x00001000 -#define WT_VERB_LSM 0x00002000 -#define WT_VERB_LSM_MANAGER 0x00004000 -#define WT_VERB_METADATA 0x00008000 -#define WT_VERB_MUTEX 0x00010000 -#define WT_VERB_OVERFLOW 0x00020000 -#define WT_VERB_READ 0x00040000 -#define WT_VERB_REBALANCE 0x00080000 -#define WT_VERB_RECONCILE 0x00100000 -#define WT_VERB_RECOVERY 0x00200000 -#define WT_VERB_RECOVERY_PROGRESS 0x00400000 -#define WT_VERB_SALVAGE 0x00800000 -#define WT_VERB_SHARED_CACHE 0x01000000 -#define WT_VERB_SPLIT 0x02000000 -#define WT_VERB_THREAD_GROUP 0x04000000 -#define WT_VERB_TIMESTAMP 0x08000000 -#define WT_VERB_TRANSACTION 0x10000000 -#define WT_VERB_VERIFY 0x20000000 -#define WT_VERB_VERSION 0x40000000 -#define WT_VERB_WRITE 0x80000000 -/* - * flags section: END - * DO NOT EDIT: automatically built by dist/flags.py. - */ diff --git a/src/third_party/wiredtiger/src/include/gcc.h b/src/third_party/wiredtiger/src/include/gcc.h index 043d9b44356..deae5bf06a9 100644 --- a/src/third_party/wiredtiger/src/include/gcc.h +++ b/src/third_party/wiredtiger/src/include/gcc.h @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -94,61 +94,76 @@ * with some versions of clang. See http://llvm.org/bugs/show_bug.cgi?id=21499 * for details. */ -#define WT_ATOMIC_CAS(ptr, oldval, newval) \ - (__sync_val_compare_and_swap(ptr, oldval, newval) == oldval) +#define WT_ATOMIC_CAS(ptr, old, new) \ + (__sync_val_compare_and_swap(ptr, old, new) == (old)) #else -#define WT_ATOMIC_CAS(ptr, oldval, newval) \ - __sync_bool_compare_and_swap(ptr, oldval, newval) +#define WT_ATOMIC_CAS(ptr, old, new) \ + __sync_bool_compare_and_swap(ptr, old, new) #endif +#define WT_ATOMIC_CAS_FUNC(name, vp_arg, old_arg, new_arg) \ +static inline bool \ +__wt_atomic_cas##name(vp_arg, old_arg, new_arg) \ +{ \ + return (WT_ATOMIC_CAS(vp, old, new)); \ +} +WT_ATOMIC_CAS_FUNC(8, uint8_t *vp, uint8_t old, uint8_t new) +WT_ATOMIC_CAS_FUNC(16, uint16_t *vp, uint16_t old, uint16_t new) +WT_ATOMIC_CAS_FUNC(32, uint32_t *vp, uint32_t old, uint32_t new) +WT_ATOMIC_CAS_FUNC(v32, \ + volatile uint32_t *vp, volatile uint32_t old, volatile uint32_t new) +WT_ATOMIC_CAS_FUNC(i32, int32_t *vp, int32_t old, int32_t new) +WT_ATOMIC_CAS_FUNC(iv32, \ + volatile int32_t *vp, volatile int32_t old, volatile int32_t new) +WT_ATOMIC_CAS_FUNC(64, uint64_t *vp, uint64_t old, uint64_t new) +WT_ATOMIC_CAS_FUNC(v64, \ + volatile uint64_t *vp, volatile uint64_t old, volatile uint64_t new) +WT_ATOMIC_CAS_FUNC(i64, int64_t *vp, int64_t old, int64_t new) +WT_ATOMIC_CAS_FUNC(iv64, \ + volatile int64_t *vp, volatile int64_t old, volatile int64_t new) +WT_ATOMIC_CAS_FUNC(size, size_t *vp, size_t old, size_t new) -#define WT_ATOMIC_FUNC(name, ret, type) \ +/* + * __wt_atomic_cas_ptr -- + * Pointer compare and swap. + */ +static inline bool +__wt_atomic_cas_ptr(void *vp, void *old, void *new) +{ + return (WT_ATOMIC_CAS((void **)vp, old, new)); +} + +#define WT_ATOMIC_FUNC(name, ret, vp_arg, v_arg) \ static inline ret \ -__wt_atomic_add##name(type *vp, type v) \ +__wt_atomic_add##name(vp_arg, v_arg) \ { \ return (__sync_add_and_fetch(vp, v)); \ } \ static inline ret \ -__wt_atomic_fetch_add##name(type *vp, type v) \ +__wt_atomic_fetch_add##name(vp_arg, v_arg) \ { \ return (__sync_fetch_and_add(vp, v)); \ } \ static inline ret \ -__wt_atomic_store##name(type *vp, type v) \ +__wt_atomic_store##name(vp_arg, v_arg) \ { \ return (__sync_lock_test_and_set(vp, v)); \ } \ static inline ret \ -__wt_atomic_sub##name(type *vp, type v) \ +__wt_atomic_sub##name(vp_arg, v_arg) \ { \ return (__sync_sub_and_fetch(vp, v)); \ -} \ -static inline bool \ -__wt_atomic_cas##name(type *vp, type old, type new) \ -{ \ - return (WT_ATOMIC_CAS(vp, old, new)); \ -} - -WT_ATOMIC_FUNC(8, uint8_t, uint8_t) -WT_ATOMIC_FUNC(16, uint16_t, uint16_t) -WT_ATOMIC_FUNC(32, uint32_t, uint32_t) -WT_ATOMIC_FUNC(v32, uint32_t, volatile uint32_t) -WT_ATOMIC_FUNC(i32, int32_t, int32_t) -WT_ATOMIC_FUNC(iv32, int32_t, volatile int32_t) -WT_ATOMIC_FUNC(64, uint64_t, uint64_t) -WT_ATOMIC_FUNC(v64, uint64_t, volatile uint64_t) -WT_ATOMIC_FUNC(i64, int64_t, int64_t) -WT_ATOMIC_FUNC(iv64, int64_t, volatile int64_t) -WT_ATOMIC_FUNC(size, size_t, size_t) - -/* - * __wt_atomic_cas_ptr -- - * Pointer compare and swap. - */ -static inline bool -__wt_atomic_cas_ptr(void *vp, void *old, void *new) -{ - return (WT_ATOMIC_CAS((void **)vp, old, new)); } +WT_ATOMIC_FUNC(8, uint8_t, uint8_t *vp, uint8_t v) +WT_ATOMIC_FUNC(16, uint16_t, uint16_t *vp, uint16_t v) +WT_ATOMIC_FUNC(32, uint32_t, uint32_t *vp, uint32_t v) +WT_ATOMIC_FUNC(v32, uint32_t, volatile uint32_t *vp, volatile uint32_t v) +WT_ATOMIC_FUNC(i32, int32_t, int32_t *vp, int32_t v) +WT_ATOMIC_FUNC(iv32, int32_t, volatile int32_t *vp, volatile int32_t v) +WT_ATOMIC_FUNC(64, uint64_t, uint64_t *vp, uint64_t v) +WT_ATOMIC_FUNC(v64, uint64_t, volatile uint64_t *vp, volatile uint64_t v) +WT_ATOMIC_FUNC(i64, int64_t, int64_t *vp, int64_t v) +WT_ATOMIC_FUNC(iv64, int64_t, volatile int64_t *vp, volatile int64_t v) +WT_ATOMIC_FUNC(size, size_t, size_t *vp, size_t v) /* Compile read-write barrier */ #define WT_BARRIER() __asm__ volatile("" ::: "memory") diff --git a/src/third_party/wiredtiger/src/include/hardware.h b/src/third_party/wiredtiger/src/include/hardware.h index 3ff198be3c7..073348cf69c 100644 --- a/src/third_party/wiredtiger/src/include/hardware.h +++ b/src/third_party/wiredtiger/src/include/hardware.h @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/include/intpack.i b/src/third_party/wiredtiger/src/include/intpack.i index 9cc6acc2d7e..aa50baa2f5f 100644 --- a/src/third_party/wiredtiger/src/include/intpack.i +++ b/src/third_party/wiredtiger/src/include/intpack.i @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/include/lint.h b/src/third_party/wiredtiger/src/include/lint.h index c02e25592c0..c2884c1b3e0 100644 --- a/src/third_party/wiredtiger/src/include/lint.h +++ b/src/third_party/wiredtiger/src/include/lint.h @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -84,8 +84,8 @@ __wt_atomic_cas_ptr(void *vp, void *orig, void *new) { return (false); } -static inline void WT_BARRIER(void) { } -static inline void WT_FULL_BARRIER(void) { } -static inline void WT_PAUSE(void) { } -static inline void WT_READ_BARRIER(void) { } -static inline void WT_WRITE_BARRIER(void) { } +static inline void WT_BARRIER(void) {} +static inline void WT_FULL_BARRIER(void) {} +static inline void WT_PAUSE(void) {} +static inline void WT_READ_BARRIER(void) {} +static inline void WT_WRITE_BARRIER(void) {} diff --git a/src/third_party/wiredtiger/src/include/log.h b/src/third_party/wiredtiger/src/include/log.h index 3ec874ad5b6..aa3f0164454 100644 --- a/src/third_party/wiredtiger/src/include/log.h +++ b/src/third_party/wiredtiger/src/include/log.h @@ -1,11 +1,26 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * * See the file LICENSE for redistribution information. */ +/* AUTOMATIC FLAG VALUE GENERATION START */ +#define WT_LOGSCAN_FIRST 0x1u +#define WT_LOGSCAN_FROM_CKP 0x2u +#define WT_LOGSCAN_ONE 0x4u +#define WT_LOGSCAN_RECOVER 0x8u +/* AUTOMATIC FLAG VALUE GENERATION STOP */ + +/* AUTOMATIC FLAG VALUE GENERATION START */ +#define WT_LOG_BACKGROUND 0x01u +#define WT_LOG_DSYNC 0x02u +#define WT_LOG_FLUSH 0x04u +#define WT_LOG_FSYNC 0x08u +#define WT_LOG_SYNC_ENABLED 0x10u +/* AUTOMATIC FLAG VALUE GENERATION STOP */ + /* * WT_LSN -- * A log sequence number, representing a position in the transaction log. @@ -78,6 +93,17 @@ union __wt_lsn { ((size) - offsetof(WT_LOG_RECORD, record)) /* + * We allocate the buffer size, but trigger a slot switch when we cross + * the maximum size of half the buffer. If a record is more than the buffer + * maximum then we trigger a slot switch and write that record unbuffered. + * We use a larger buffer to provide overflow space so that we can switch + * once we cross the threshold. + */ +#define WT_LOG_SLOT_BUF_SIZE (256 * 1024) /* Must be power of 2 */ +#define WT_LOG_SLOT_BUF_MAX ((uint32_t)log->slot_buf_size / 2) +#define WT_LOG_SLOT_UNBUFFERED (WT_LOG_SLOT_BUF_SIZE << 1) + +/* * Possible values for the consolidation array slot states: * * WT_LOG_SLOT_CLOSE - slot is in use but closed to new joins. @@ -92,6 +118,12 @@ union __wt_lsn { * a few special states, reserve the top few bits for state. That makes * the maximum size less than 32 bits for both joined and released. */ +/* + * XXX + * The log slot bits are signed and should be rewritten as unsigned. For now, + * give the logging subsystem its own flags macro. + */ +#define FLD_LOG_SLOT_ISSET(field, mask) (((field) & (uint64_t)(mask)) != 0) /* * The high bit is reserved for the special states. If the high bit is @@ -101,17 +133,6 @@ union __wt_lsn { #define WT_LOG_SLOT_WRITTEN (-2) /* Slot data written, not processed */ /* - * We allocate the buffer size, but trigger a slot switch when we cross - * the maximum size of half the buffer. If a record is more than the buffer - * maximum then we trigger a slot switch and write that record unbuffered. - * We use a larger buffer to provide overflow space so that we can switch - * once we cross the threshold. - */ -#define WT_LOG_SLOT_BUF_SIZE (256 * 1024) /* Must be power of 2 */ -#define WT_LOG_SLOT_BUF_MAX ((uint32_t)log->slot_buf_size / 2) -#define WT_LOG_SLOT_UNBUFFERED (WT_LOG_SLOT_BUF_SIZE << 1) - -/* * If new slot states are added, adjust WT_LOG_SLOT_BITS and * WT_LOG_SLOT_MASK_OFF accordingly for how much of the top 32 * bits we are using. More slot states here will reduce the maximum @@ -155,8 +176,8 @@ union __wt_lsn { /* Slot is in use, but closed to new joins */ #define WT_LOG_SLOT_CLOSED(state) \ (WT_LOG_SLOT_ACTIVE(state) && \ - (FLD64_ISSET((uint64_t)(state), WT_LOG_SLOT_CLOSE) && \ - !FLD64_ISSET((uint64_t)(state), WT_LOG_SLOT_RESERVED))) + (FLD_LOG_SLOT_ISSET((uint64_t)(state), WT_LOG_SLOT_CLOSE) && \ + !FLD_LOG_SLOT_ISSET((uint64_t)(state), WT_LOG_SLOT_RESERVED))) /* Slot is in use, all data copied into buffer */ #define WT_LOG_SLOT_INPROGRESS(state) \ (WT_LOG_SLOT_RELEASED(state) != WT_LOG_SLOT_JOINED(state)) @@ -167,7 +188,7 @@ union __wt_lsn { #define WT_LOG_SLOT_OPEN(state) \ (WT_LOG_SLOT_ACTIVE(state) && \ !WT_LOG_SLOT_UNBUFFERED_ISSET(state) && \ - !FLD64_ISSET((uint64_t)(state), WT_LOG_SLOT_CLOSE) && \ + !FLD_LOG_SLOT_ISSET((uint64_t)(state), WT_LOG_SLOT_CLOSE) && \ WT_LOG_SLOT_JOINED(state) < WT_LOG_SLOT_BUF_MAX) struct __wt_logslot { @@ -183,11 +204,13 @@ struct __wt_logslot { WT_FH *slot_fh; /* File handle for this group */ WT_ITEM slot_buf; /* Buffer for grouped writes */ -#define WT_SLOT_CLOSEFH 0x01 /* Close old fh on release */ -#define WT_SLOT_FLUSH 0x02 /* Wait for write */ -#define WT_SLOT_SYNC 0x04 /* Needs sync on release */ -#define WT_SLOT_SYNC_DIR 0x08 /* Directory sync on release */ - uint32_t flags; /* Flags */ +/* AUTOMATIC FLAG VALUE GENERATION START */ +#define WT_SLOT_CLOSEFH 0x1u /* Close old fh on release */ +#define WT_SLOT_FLUSH 0x2u /* Wait for write */ +#define WT_SLOT_SYNC 0x4u /* Needs sync on release */ +#define WT_SLOT_SYNC_DIR 0x8u /* Directory sync on release */ +/* AUTOMATIC FLAG VALUE GENERATION STOP */ + uint32_t flags; WT_CACHE_LINE_PAD_END }; @@ -203,10 +226,13 @@ struct __wt_myslot { WT_LOGSLOT *slot; /* Slot I'm using */ wt_off_t end_offset; /* My end offset in buffer */ wt_off_t offset; /* Slot buffer offset */ -#define WT_MYSLOT_CLOSE 0x01 /* This thread is closing the slot */ -#define WT_MYSLOT_NEEDS_RELEASE 0x02 /* This thread is releasing the slot */ -#define WT_MYSLOT_UNBUFFERED 0x04 /* Write directly */ - uint32_t flags; /* Flags */ + +/* AUTOMATIC FLAG VALUE GENERATION START */ +#define WT_MYSLOT_CLOSE 0x1u /* This thread is closing the slot */ +#define WT_MYSLOT_NEEDS_RELEASE 0x2u /* This thread is releasing the slot */ +#define WT_MYSLOT_UNBUFFERED 0x4u /* Write directly */ +/* AUTOMATIC FLAG VALUE GENERATION STOP */ + uint32_t flags; }; #define WT_LOG_END_HEADER log->allocsize @@ -276,9 +302,11 @@ struct __wt_log { uint64_t write_calls; /* Calls to log_write */ #endif -#define WT_LOG_FORCE_NEWFILE 0x01 /* Force switch to new log file */ -#define WT_LOG_OPENED 0x02 /* Log subsystem successfully open */ -#define WT_LOG_TRUNCATE_NOTSUP 0x04 /* File system truncate not supported */ +/* AUTOMATIC FLAG VALUE GENERATION START */ +#define WT_LOG_FORCE_NEWFILE 0x1u /* Force switch to new log file */ +#define WT_LOG_OPENED 0x2u /* Log subsystem successfully open */ +#define WT_LOG_TRUNCATE_NOTSUP 0x4u /* File system truncate not supported */ +/* AUTOMATIC FLAG VALUE GENERATION STOP */ uint32_t flags; }; @@ -286,8 +314,12 @@ struct __wt_log_record { uint32_t len; /* 00-03: Record length including hdr */ uint32_t checksum; /* 04-07: Checksum of the record */ -#define WT_LOG_RECORD_COMPRESSED 0x01 /* Compressed except hdr */ -#define WT_LOG_RECORD_ENCRYPTED 0x02 /* Encrypted except hdr */ + /* + * No automatic generation: flag values cannot change, they're written + * to disk. + */ +#define WT_LOG_RECORD_COMPRESSED 0x01u /* Compressed except hdr */ +#define WT_LOG_RECORD_ENCRYPTED 0x02u /* Encrypted except hdr */ uint16_t flags; /* 08-09: Flags */ uint8_t unused[2]; /* 10-11: Padding */ uint32_t mem_len; /* 12-15: Uncompressed len if needed */ @@ -317,7 +349,7 @@ __wt_log_record_byteswap(WT_LOG_RECORD *record) * The log file's description. */ struct __wt_log_desc { -#define WT_LOG_MAGIC 0x101064 +#define WT_LOG_MAGIC 0x101064u uint32_t log_magic; /* 00-03: Magic number */ #define WT_LOG_VERSION 2 uint16_t version; /* 04-05: Log version */ @@ -357,7 +389,9 @@ __wt_log_desc_byteswap(WT_LOG_DESC *desc) /* * Flags for __wt_txn_op_printlog. */ -#define WT_TXN_PRINTLOG_HEX 0x0001 /* Add hex output */ +/* AUTOMATIC FLAG VALUE GENERATION START */ +#define WT_TXN_PRINTLOG_HEX 0x1u /* Add hex output */ +/* AUTOMATIC FLAG VALUE GENERATION STOP */ /* * WT_LOG_REC_DESC -- diff --git a/src/third_party/wiredtiger/src/include/log.i b/src/third_party/wiredtiger/src/include/log.i index 8c7e5dc65e8..3151ca4ff86 100644 --- a/src/third_party/wiredtiger/src/include/log.i +++ b/src/third_party/wiredtiger/src/include/log.i @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/include/lsm.h b/src/third_party/wiredtiger/src/include/lsm.h index 0e026d90da5..3102f1754cf 100644 --- a/src/third_party/wiredtiger/src/include/lsm.h +++ b/src/third_party/wiredtiger/src/include/lsm.h @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -66,16 +66,17 @@ struct __wt_cursor_lsm { u_int update_count; /* Updates performed. */ -#define WT_CLSM_ACTIVE 0x001 /* Incremented the session count */ -#define WT_CLSM_BULK 0x002 /* Open for snapshot isolation */ -#define WT_CLSM_ITERATE_NEXT 0x004 /* Forward iteration */ -#define WT_CLSM_ITERATE_PREV 0x008 /* Backward iteration */ -#define WT_CLSM_MERGE 0x010 /* Merge cursor, don't update */ -#define WT_CLSM_MINOR_MERGE 0x020 /* Minor merge, include tombstones */ -#define WT_CLSM_MULTIPLE 0x040 /* Multiple cursors have values for the - current key */ -#define WT_CLSM_OPEN_READ 0x080 /* Open for reads */ -#define WT_CLSM_OPEN_SNAPSHOT 0x100 /* Open for snapshot isolation */ +/* AUTOMATIC FLAG VALUE GENERATION START */ +#define WT_CLSM_ACTIVE 0x001u /* Incremented the session count */ +#define WT_CLSM_BULK 0x002u /* Open for snapshot isolation */ +#define WT_CLSM_ITERATE_NEXT 0x004u /* Forward iteration */ +#define WT_CLSM_ITERATE_PREV 0x008u /* Backward iteration */ +#define WT_CLSM_MERGE 0x010u /* Merge cursor, don't update */ +#define WT_CLSM_MINOR_MERGE 0x020u /* Minor merge, include tombstones */ +#define WT_CLSM_MULTIPLE 0x040u /* Multiple cursors have values */ +#define WT_CLSM_OPEN_READ 0x080u /* Open for reads */ +#define WT_CLSM_OPEN_SNAPSHOT 0x100u /* Open for snapshot isolation */ +/* AUTOMATIC FLAG VALUE GENERATION STOP */ uint32_t flags; }; @@ -112,11 +113,13 @@ struct __wt_lsm_chunk { int8_t evicted; /* 1/0: in-memory chunk was evicted */ uint8_t flushing; /* 1/0: chunk flush in progress */ -#define WT_LSM_CHUNK_BLOOM 0x01 -#define WT_LSM_CHUNK_HAS_TIMESTAMP 0x02 -#define WT_LSM_CHUNK_MERGING 0x04 -#define WT_LSM_CHUNK_ONDISK 0x08 -#define WT_LSM_CHUNK_STABLE 0x10 +/* AUTOMATIC FLAG VALUE GENERATION START */ +#define WT_LSM_CHUNK_BLOOM 0x01u +#define WT_LSM_CHUNK_HAS_TIMESTAMP 0x02u +#define WT_LSM_CHUNK_MERGING 0x04u +#define WT_LSM_CHUNK_ONDISK 0x08u +#define WT_LSM_CHUNK_STABLE 0x10u +/* AUTOMATIC FLAG VALUE GENERATION STOP */ uint32_t flags; }; @@ -125,11 +128,13 @@ struct __wt_lsm_chunk { * type of work they will execute, and by work units to define which action * is required. */ -#define WT_LSM_WORK_BLOOM 0x01 /* Create a bloom filter */ -#define WT_LSM_WORK_DROP 0x02 /* Drop unused chunks */ -#define WT_LSM_WORK_FLUSH 0x04 /* Flush a chunk to disk */ -#define WT_LSM_WORK_MERGE 0x08 /* Look for a tree merge */ -#define WT_LSM_WORK_SWITCH 0x10 /* Switch to new in-memory chunk */ +/* AUTOMATIC FLAG VALUE GENERATION START */ +#define WT_LSM_WORK_BLOOM 0x01u /* Create a bloom filter */ +#define WT_LSM_WORK_DROP 0x02u /* Drop unused chunks */ +#define WT_LSM_WORK_FLUSH 0x04u /* Flush a chunk to disk */ +#define WT_LSM_WORK_MERGE 0x08u /* Look for a tree merge */ +#define WT_LSM_WORK_SWITCH 0x10u /* Switch to new in-memory chunk */ +/* AUTOMATIC FLAG VALUE GENERATION STOP */ /* * WT_LSM_WORK_UNIT -- @@ -138,7 +143,9 @@ struct __wt_lsm_chunk { struct __wt_lsm_work_unit { TAILQ_ENTRY(__wt_lsm_work_unit) q; /* Worker unit queue */ uint32_t type; /* Type of operation */ -#define WT_LSM_WORK_FORCE 0x0001 /* Force operation */ +/* AUTOMATIC FLAG VALUE GENERATION START */ +#define WT_LSM_WORK_FORCE 0x1u /* Force operation */ +/* AUTOMATIC FLAG VALUE GENERATION STOP */ uint32_t flags; /* Flags for operation */ WT_LSM_TREE *lsm_tree; }; @@ -172,7 +179,9 @@ struct __wt_lsm_manager { #define WT_LSM_MIN_WORKERS 3 WT_LSM_WORKER_ARGS lsm_worker_cookies[WT_LSM_MAX_WORKERS]; -#define WT_LSM_MANAGER_SHUTDOWN 0x01 /* Manager has shut down */ +/* AUTOMATIC FLAG VALUE GENERATION START */ +#define WT_LSM_MANAGER_SHUTDOWN 0x1u /* Manager has shut down */ +/* AUTOMATIC FLAG VALUE GENERATION STOP */ uint32_t flags; }; @@ -192,6 +201,11 @@ struct __wt_lsm_tree { const char *key_format, *value_format; const char *bloom_config, *file_config; + uint32_t custom_generation; /* Level at which a custom data source + should be used for merges. */ + const char *custom_prefix; /* Prefix for custom data source */ + const char *custom_suffix; /* Suffix for custom data source */ + WT_COLLATOR *collator; const char *collator_name; int collator_owned; @@ -226,9 +240,11 @@ struct __wt_lsm_tree { uint64_t chunk_max; /* Maximum chunk a merge creates */ u_int merge_min, merge_max; -#define WT_LSM_BLOOM_MERGED 0x00000001 -#define WT_LSM_BLOOM_OFF 0x00000002 -#define WT_LSM_BLOOM_OLDEST 0x00000004 +/* AUTOMATIC FLAG VALUE GENERATION START */ +#define WT_LSM_BLOOM_MERGED 0x1u +#define WT_LSM_BLOOM_OFF 0x2u +#define WT_LSM_BLOOM_OLDEST 0x4u +/* AUTOMATIC FLAG VALUE GENERATION STOP */ uint32_t bloom; /* Bloom creation policy */ WT_LSM_CHUNK **chunk; /* Array of active LSM chunks */ @@ -274,10 +290,12 @@ struct __wt_lsm_tree { * flags here are not protected for concurrent access, don't put * anything here that is susceptible to races. */ -#define WT_LSM_TREE_COMPACTING 0x01 /* Tree being compacted */ -#define WT_LSM_TREE_MERGES 0x02 /* Tree should run merges */ -#define WT_LSM_TREE_OPEN 0x04 /* The tree is open */ -#define WT_LSM_TREE_THROTTLE 0x08 /* Throttle updates */ +/* AUTOMATIC FLAG VALUE GENERATION START */ +#define WT_LSM_TREE_COMPACTING 0x1u /* Tree being compacted */ +#define WT_LSM_TREE_MERGES 0x2u /* Tree should run merges */ +#define WT_LSM_TREE_OPEN 0x4u /* The tree is open */ +#define WT_LSM_TREE_THROTTLE 0x8u /* Throttle updates */ +/* AUTOMATIC FLAG VALUE GENERATION STOP */ uint32_t flags; }; diff --git a/src/third_party/wiredtiger/src/include/meta.h b/src/third_party/wiredtiger/src/include/meta.h index 2dd77157caa..e64e06d08f8 100644 --- a/src/third_party/wiredtiger/src/include/meta.h +++ b/src/third_party/wiredtiger/src/include/meta.h @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -73,9 +73,11 @@ struct __wt_ckpt { void *bpriv; /* Block manager private */ -#define WT_CKPT_ADD 0x01 /* Checkpoint to be added */ -#define WT_CKPT_DELETE 0x02 /* Checkpoint to be deleted */ -#define WT_CKPT_FAKE 0x04 /* Checkpoint is a fake */ -#define WT_CKPT_UPDATE 0x08 /* Checkpoint requires update */ +/* AUTOMATIC FLAG VALUE GENERATION START */ +#define WT_CKPT_ADD 0x1u /* Checkpoint to be added */ +#define WT_CKPT_DELETE 0x2u /* Checkpoint to be deleted */ +#define WT_CKPT_FAKE 0x4u /* Checkpoint is a fake */ +#define WT_CKPT_UPDATE 0x8u /* Checkpoint requires update */ +/* AUTOMATIC FLAG VALUE GENERATION STOP */ uint32_t flags; }; diff --git a/src/third_party/wiredtiger/src/include/misc.h b/src/third_party/wiredtiger/src/include/misc.h index 2435d37ee20..0ddeb046353 100644 --- a/src/third_party/wiredtiger/src/include/misc.h +++ b/src/third_party/wiredtiger/src/include/misc.h @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -11,6 +11,7 @@ * and unused function return values. */ #define WT_UNUSED(var) (void)(var) +#define WT_NOT_READ(var) (void)(var) #define WT_IGNORE_RET(call) do { \ int __ignored_ret; \ __ignored_ret = (call); \ @@ -132,15 +133,13 @@ * hex constant might be a negative integer), and to ensure the hex constant is * the correct size before applying the bitwise not operator. */ -#define FLD_CLR(field, mask) ((void)((field) &= ~(uint32_t)(mask))) -#define FLD_MASK(field, mask) ((field) & (uint32_t)(mask)) +#define FLD_CLR(field, mask) ((void)((field) &= ~(mask))) +#define FLD_MASK(field, mask) ((field) & (mask)) #define FLD_ISSET(field, mask) (FLD_MASK(field, mask) != 0) -#define FLD64_ISSET(field, mask) (((field) & (uint64_t)(mask)) != 0) -#define FLD_SET(field, mask) ((void)((field) |= (uint32_t)(mask))) +#define FLD_SET(field, mask) ((void)((field) |= (mask))) #define F_CLR(p, mask) FLD_CLR((p)->flags, mask) #define F_ISSET(p, mask) FLD_ISSET((p)->flags, mask) -#define F_ISSET_ALL(p, mask) (FLD_MASK((p)->flags, mask) == (mask)) #define F_MASK(p, mask) FLD_MASK((p)->flags, mask) #define F_SET(p, mask) FLD_SET((p)->flags, mask) @@ -195,12 +194,8 @@ } while (0) /* Verbose messages. */ -#ifdef HAVE_VERBOSE #define WT_VERBOSE_ISSET(session, f) \ (FLD_ISSET(S2C(session)->verbose, f)) -#else -#define WT_VERBOSE_ISSET(session, f) 0 -#endif #define WT_CLEAR(s) \ memset(&(s), 0, sizeof(s)) diff --git a/src/third_party/wiredtiger/src/include/misc.i b/src/third_party/wiredtiger/src/include/misc.i index bedd3121037..05c0733d4ce 100644 --- a/src/third_party/wiredtiger/src/include/misc.i +++ b/src/third_party/wiredtiger/src/include/misc.i @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -30,6 +30,49 @@ __wt_hex(int c) } /* + * __wt_rdtsc -- + * Get a timestamp from CPU registers. + */ +static inline uint64_t +__wt_rdtsc(void) { +#if defined (__i386) + { + uint64_t x; + + __asm__ volatile ("rdtsc" : "=A" (x)); + return (x); + } +#elif defined (__amd64) + { + uint64_t a, d; + + __asm__ volatile ("rdtsc" : "=a" (a), "=d" (d)); + return ((d << 32) | a); + } +#else + return (0); +#endif +} + +/* + * __wt_clock -- + * Obtain a timestamp via either a CPU register or via a system call on + * platforms where obtaining it directly from the hardware register is + * not supported. + */ +static inline uint64_t +__wt_clock(WT_SESSION_IMPL *session) +{ + struct timespec tsp; + + if (__wt_process.use_epochtime) { + __wt_epoch(session, &tsp); + return ((uint64_t)(tsp.tv_sec * WT_BILLION + tsp.tv_nsec)); + } + return (__wt_rdtsc()); +} + +/* * __wt_strdup -- * ANSI strdup function. */ diff --git a/src/third_party/wiredtiger/src/include/msvc.h b/src/third_party/wiredtiger/src/include/msvc.h index d9bbd2d505b..3194e536b9b 100644 --- a/src/third_party/wiredtiger/src/include/msvc.h +++ b/src/third_party/wiredtiger/src/include/msvc.h @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/include/mutex.h b/src/third_party/wiredtiger/src/include/mutex.h index 7aeb6160f43..2a3fc7448f8 100644 --- a/src/third_party/wiredtiger/src/include/mutex.h +++ b/src/third_party/wiredtiger/src/include/mutex.h @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/include/mutex.i b/src/third_party/wiredtiger/src/include/mutex.i index 871ccf63be8..8a2699f031d 100644 --- a/src/third_party/wiredtiger/src/include/mutex.i +++ b/src/third_party/wiredtiger/src/include/mutex.i @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -293,21 +293,21 @@ __wt_spin_unlock(WT_SESSION_IMPL *session, WT_SPINLOCK *t) static inline void __wt_spin_lock_track(WT_SESSION_IMPL *session, WT_SPINLOCK *t) { - struct timespec enter, leave; + uint64_t time_start, time_stop; int64_t **stats; if (t->stat_count_off != -1 && WT_STAT_ENABLED(session)) { - __wt_epoch(session, &enter); + time_start = __wt_clock(session); __wt_spin_lock(session, t); - __wt_epoch(session, &leave); + time_stop = __wt_clock(session); stats = (int64_t **)S2C(session)->stats; stats[session->stat_bucket][t->stat_count_off]++; if (F_ISSET(session, WT_SESSION_INTERNAL)) stats[session->stat_bucket][t->stat_int_usecs_off] += - (int64_t)WT_TIMEDIFF_US(leave, enter); + (int64_t)WT_CLOCKDIFF_US(time_stop, time_start); else stats[session->stat_bucket][t->stat_app_usecs_off] += - (int64_t)WT_TIMEDIFF_US(leave, enter); + (int64_t)WT_CLOCKDIFF_US(time_stop, time_start); } else __wt_spin_lock(session, t); } diff --git a/src/third_party/wiredtiger/src/include/optrack.h b/src/third_party/wiredtiger/src/include/optrack.h new file mode 100644 index 00000000000..8593a20d2c4 --- /dev/null +++ b/src/third_party/wiredtiger/src/include/optrack.h @@ -0,0 +1,87 @@ +/*- + * Copyright (c) 2014-2018 MongoDB, Inc. + * Copyright (c) 2008-2014 WiredTiger, Inc. + * All rights reserved. + * + * See the file LICENSE for redistribution information. + */ + +#define WT_OPTRACK_MAXRECS (16384) +#define WT_OPTRACK_BUFSIZE (WT_OPTRACK_MAXRECS * sizeof(WT_OPTRACK_RECORD)) +#define WT_OPTRACK_VERSION 2 + +/* + * WT_OPTRACK_HEADER -- + * A header in the operation tracking log file. The internal session + * identifier is a boolean: 1 if the session is internal, 0 otherwise. + */ +struct __wt_optrack_header { + uint32_t optrack_version; + uint32_t optrack_session_internal; + uint32_t optrack_tsc_nsec_ratio; +}; + +/* + * WT_OPTRACK_RECORD -- + * A structure for logging function entry and exit events. + * + * We pad the record so that the size of the entire record is 16 bytes. If we + * don't do this, the compiler will pad it for us, because we keep records in + * the record buffer array and each new record must be aligned on the 8-byte + * boundary, since its first element is an 8-byte timestamp. Instead of letting + * the compiler insert the padding silently, we pad explicitly, so that whoever + * writes the binary decoder can refer to this struct to find out the record + * size. + * + * The operation id included in this structure is a unique address of a function + * in the binary. As we log operations, we keep track of the correspondence + * between function addresses and their names. When the log file is decoded, + * operations identifiers are replaced with function names. Therefore, the + * present design assumes that the user will be inserting the tracking macros + * on function boundaries: when we enter into the function and when we exit + * from it. + */ +struct __wt_optrack_record { + uint64_t op_timestamp; /* timestamp */ + uint16_t op_id; /* function ID */ + uint16_t op_type; /* start/stop */ + uint8_t padding[4]; +}; + +#define WT_TRACK_OP(s, optype) do { \ + WT_OPTRACK_RECORD *__tr; \ + __tr = &((s)->optrack_buf[ \ + (s)->optrackbuf_ptr % WT_OPTRACK_MAXRECS]); \ + __tr->op_timestamp = __wt_clock(s); \ + __tr->op_id = __func_id; \ + __tr->op_type = optype; \ + \ + if (++(s)->optrackbuf_ptr == WT_OPTRACK_MAXRECS) { \ + (s)->optrack_offset += __wt_optrack_flush_buffer(s); \ + (s)->optrackbuf_ptr = 0; \ + } \ +} while (0) + +/* + * We do not synchronize access to optrack buffer pointer under the assumption + * that there is no more than one thread using a given session. This assumption + * does not always hold. When it does not, we might have a race. In this case, + * we may lose a few log records. We prefer to risk losing a few log records + * occasionally in order not to synchronize this code, which is intended to be + * very lightweight. + * Exclude the default session (ID 0) because it can be used by multiple + * threads and it is also used in error paths during failed open calls. + */ +#define WT_TRACK_OP_DECL \ + static uint16_t __func_id = 0 +#define WT_TRACK_OP_INIT(s) \ + if (F_ISSET(S2C(s), WT_CONN_OPTRACK) && (s)->id != 0) { \ + if (__func_id == 0) \ + __wt_optrack_record_funcid( \ + s, __func__, &__func_id); \ + WT_TRACK_OP(s, 0); \ + } + +#define WT_TRACK_OP_END(s) \ + if (F_ISSET(S2C(s), WT_CONN_OPTRACK) && (s)->id != 0) \ + WT_TRACK_OP(s, 1); diff --git a/src/third_party/wiredtiger/src/include/os.h b/src/third_party/wiredtiger/src/include/os.h index ec1860d19a6..c31619f2f96 100644 --- a/src/third_party/wiredtiger/src/include/os.h +++ b/src/third_party/wiredtiger/src/include/os.h @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -65,6 +65,15 @@ #define WT_TIMEDIFF_SEC(end, begin) \ (WT_TIMEDIFF_NS((end), (begin)) / WT_BILLION) +#define WT_CLOCKDIFF_NS(end, begin) \ + (__wt_clock_to_nsec(end, begin)) +#define WT_CLOCKDIFF_US(end, begin) \ + (WT_CLOCKDIFF_NS(end, begin) / WT_THOUSAND) +#define WT_CLOCKDIFF_MS(end, begin) \ + (WT_CLOCKDIFF_NS(end, begin) / WT_MILLION) +#define WT_CLOCKDIFF_SEC(end, begin) \ + (WT_CLOCKDIFF_NS(end, begin) / WT_BILLION) + #define WT_TIMECMP(t1, t2) \ ((t1).tv_sec < (t2).tv_sec ? -1 : \ (t1).tv_sec == (t2).tv_sec ? \ @@ -155,9 +164,11 @@ struct __wt_fstream { wt_off_t size; /* File size */ WT_ITEM buf; /* Data */ -#define WT_STREAM_APPEND 0x01 /* Open a stream for append */ -#define WT_STREAM_READ 0x02 /* Open a stream for read */ -#define WT_STREAM_WRITE 0x04 /* Open a stream for write */ +/* AUTOMATIC FLAG VALUE GENERATION START */ +#define WT_STREAM_APPEND 0x1u /* Open a stream for append */ +#define WT_STREAM_READ 0x2u /* Open a stream for read */ +#define WT_STREAM_WRITE 0x4u /* Open a stream for write */ +/* AUTOMATIC FLAG VALUE GENERATION STOP */ uint32_t flags; int (*close)(WT_SESSION_IMPL *, WT_FSTREAM *); diff --git a/src/third_party/wiredtiger/src/include/os_fhandle.i b/src/third_party/wiredtiger/src/include/os_fhandle.i index e5177e64b57..7c09a83132c 100644 --- a/src/third_party/wiredtiger/src/include/os_fhandle.i +++ b/src/third_party/wiredtiger/src/include/os_fhandle.i @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -7,6 +7,13 @@ */ /* + * Define functions that increment histogram statistics for filesystem + * operations latency. + */ +WT_STAT_MSECS_HIST_INCR_FUNC(fsread, perf_hist_fsread_latency, 10) +WT_STAT_MSECS_HIST_INCR_FUNC(fswrite, perf_hist_fswrite_latency, 10) + +/* * __wt_fsync -- * POSIX fsync. */ @@ -94,6 +101,7 @@ __wt_read( WT_SESSION_IMPL *session, WT_FH *fh, wt_off_t offset, size_t len, void *buf) { WT_DECL_RET; + uint64_t time_start, time_stop; __wt_verbose(session, WT_VERB_HANDLEOPS, "%s: handle-read: %" WT_SIZET_FMT " at %" PRIuMAX, @@ -101,10 +109,14 @@ __wt_read( WT_STAT_CONN_INCR_ATOMIC(session, thread_read_active); WT_STAT_CONN_INCR(session, read_io); + time_start = __wt_clock(session); ret = fh->handle->fh_read( fh->handle, (WT_SESSION *)session, offset, len, buf); + time_stop = __wt_clock(session); + __wt_stat_msecs_hist_incr_fsread(session, + WT_CLOCKDIFF_MS(time_stop, time_start)); WT_STAT_CONN_DECR_ATOMIC(session, thread_read_active); return (ret); } @@ -157,6 +169,7 @@ __wt_write(WT_SESSION_IMPL *session, WT_FH *fh, wt_off_t offset, size_t len, const void *buf) { WT_DECL_RET; + uint64_t time_start, time_stop; WT_ASSERT(session, !F_ISSET(S2C(session), WT_CONN_READONLY) || WT_STRING_MATCH(fh->name, @@ -175,10 +188,14 @@ __wt_write(WT_SESSION_IMPL *session, WT_STAT_CONN_INCR(session, write_io); WT_STAT_CONN_INCR_ATOMIC(session, thread_write_active); + time_start = __wt_clock(session); ret = fh->handle->fh_write( fh->handle, (WT_SESSION *)session, offset, len, buf); + time_stop = __wt_clock(session); + __wt_stat_msecs_hist_incr_fswrite(session, + WT_CLOCKDIFF_MS(time_stop, time_start)); WT_STAT_CONN_DECR_ATOMIC(session, thread_write_active); return (ret); } diff --git a/src/third_party/wiredtiger/src/include/os_fs.i b/src/third_party/wiredtiger/src/include/os_fs.i index c81d3f5dec6..ecd7ca802a9 100644 --- a/src/third_party/wiredtiger/src/include/os_fs.i +++ b/src/third_party/wiredtiger/src/include/os_fs.i @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -38,6 +38,37 @@ __wt_fs_directory_list(WT_SESSION_IMPL *session, } /* + * __wt_fs_directory_list_single -- + * Return a single matching file from a directory. + */ +static inline int +__wt_fs_directory_list_single(WT_SESSION_IMPL *session, + const char *dir, const char *prefix, char ***dirlistp, u_int *countp) +{ + WT_DECL_RET; + WT_FILE_SYSTEM *file_system; + WT_SESSION *wt_session; + char *path; + + *dirlistp = NULL; + *countp = 0; + + __wt_verbose(session, WT_VERB_FILEOPS, + "%s: directory-list-single: prefix %s", + dir, prefix == NULL ? "all" : prefix); + + WT_RET(__wt_filename(session, dir, &path)); + + file_system = S2C(session)->file_system; + wt_session = (WT_SESSION *)session; + ret = file_system->fs_directory_list_single( + file_system, wt_session, path, prefix, dirlistp, countp); + + __wt_free(session, path); + return (ret); +} + +/* * __wt_fs_directory_list_free -- * Free memory allocated by __wt_fs_directory_list. */ diff --git a/src/third_party/wiredtiger/src/include/os_fstream.i b/src/third_party/wiredtiger/src/include/os_fstream.i index 1561274b388..09ce37b7058 100644 --- a/src/third_party/wiredtiger/src/include/os_fstream.i +++ b/src/third_party/wiredtiger/src/include/os_fstream.i @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/include/os_windows.h b/src/third_party/wiredtiger/src/include/os_windows.h index ea54d00af1f..f5e270a4954 100644 --- a/src/third_party/wiredtiger/src/include/os_windows.h +++ b/src/third_party/wiredtiger/src/include/os_windows.h @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/include/packing.i b/src/third_party/wiredtiger/src/include/packing.i index d5181738fbd..34a1bb62edb 100644 --- a/src/third_party/wiredtiger/src/include/packing.i +++ b/src/third_party/wiredtiger/src/include/packing.i @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -253,14 +253,15 @@ next: if (pack->cur == pack->end) * __pack_size -- * Get the size of a packed value. */ -static inline size_t -__pack_size(WT_SESSION_IMPL *session, WT_PACK_VALUE *pv) +static inline int +__pack_size(WT_SESSION_IMPL *session, WT_PACK_VALUE *pv, size_t *vp) { size_t s, pad; switch (pv->type) { case 'x': - return (pv->size); + *vp = pv->size; + return (0); case 'j': case 'J': case 'K': @@ -276,7 +277,8 @@ __pack_size(WT_SESSION_IMPL *session, WT_PACK_VALUE *pv) WT_ASSERT(session, len >= 0); s = (size_t)len + (pv->type == 'K' ? 0 : 1); } - return (s); + *vp = s; + return (0); case 's': case 'S': if (pv->type == 's' || pv->havesize) { @@ -284,7 +286,8 @@ __pack_size(WT_SESSION_IMPL *session, WT_PACK_VALUE *pv) WT_ASSERT(session, s != 0); } else s = strlen(pv->u.s) + 1; - return (s); + *vp = s; + return (0); case 'U': case 'u': s = pv->u.item.size; @@ -295,28 +298,33 @@ __pack_size(WT_SESSION_IMPL *session, WT_PACK_VALUE *pv) pad = pv->size - s; if (pv->type == 'U') s += __wt_vsize_uint(s + pad); - return (s + pad); + *vp = s + pad; + return (0); case 'b': case 'B': case 't': - return (1); + *vp = 1; + return (0); case 'h': case 'i': case 'l': case 'q': - return (__wt_vsize_int(pv->u.i)); + *vp = __wt_vsize_int(pv->u.i); + return (0); case 'H': case 'I': case 'L': case 'Q': case 'r': - return (__wt_vsize_uint(pv->u.u)); + *vp = __wt_vsize_uint(pv->u.u); + return (0); case 'R': - return (sizeof(uint64_t)); + *vp = sizeof(uint64_t); + return (0); } - __wt_err(session, EINVAL, "unknown pack-value type: %c", (int)pv->type); - return ((size_t)-1); + WT_RET_MSG( + session, EINVAL, "unknown pack-value type: %c", (int)pv->type); } /* @@ -635,13 +643,11 @@ __wt_struct_packv(WT_SESSION_IMPL *session, WT_PACK_GET(session, pv, ap); WT_RET(__pack_write(session, &pv, &p, (size_t)(end - p))); } + WT_RET_NOTFOUND_OK(ret); /* Be paranoid - __pack_write should never overflow. */ WT_ASSERT(session, p <= end); - if (ret != WT_NOTFOUND) - return (ret); - return (0); } @@ -654,22 +660,26 @@ __wt_struct_sizev( WT_SESSION_IMPL *session, size_t *sizep, const char *fmt, va_list ap) { WT_DECL_PACK_VALUE(pv); + WT_DECL_RET; WT_PACK pack; - size_t total; + size_t v; + + *sizep = 0; if (fmt[0] != '\0' && fmt[1] == '\0') { pv.type = fmt[0]; WT_PACK_GET(session, pv, ap); - *sizep = __pack_size(session, &pv); - return (0); + return (__pack_size(session, &pv, sizep)); } WT_RET(__pack_init(session, &pack, fmt)); - for (total = 0; __pack_next(&pack, &pv) == 0;) { + while ((ret = __pack_next(&pack, &pv)) == 0) { WT_PACK_GET(session, pv, ap); - total += __pack_size(session, &pv); + WT_RET(__pack_size(session, &pv, &v)); + *sizep += v; } - *sizep = total; + WT_RET_NOTFOUND_OK(ret); + return (0); } @@ -701,13 +711,11 @@ __wt_struct_unpackv(WT_SESSION_IMPL *session, WT_RET(__unpack_read(session, &pv, &p, (size_t)(end - p))); WT_UNPACK_PUT(session, pv, ap); } + WT_RET_NOTFOUND_OK(ret); /* Be paranoid - __pack_write should never overflow. */ WT_ASSERT(session, p <= end); - if (ret != WT_NOTFOUND) - return (ret); - return (0); } diff --git a/src/third_party/wiredtiger/src/include/posix.h b/src/third_party/wiredtiger/src/include/posix.h index 23a4d178e98..0b4bf919327 100644 --- a/src/third_party/wiredtiger/src/include/posix.h +++ b/src/third_party/wiredtiger/src/include/posix.h @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/include/schema.h b/src/third_party/wiredtiger/src/include/schema.h index 80513f1174b..3a60b0f9c67 100644 --- a/src/third_party/wiredtiger/src/include/schema.h +++ b/src/third_party/wiredtiger/src/include/schema.h @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -40,7 +40,10 @@ struct __wt_index { const char *idxkey_format; /* Index key format (hides primary) */ const char *exkey_format; /* Key format for custom extractors */ -#define WT_INDEX_IMMUTABLE 0x01 + +/* AUTOMATIC FLAG VALUE GENERATION START */ +#define WT_INDEX_IMMUTABLE 0x1u +/* AUTOMATIC FLAG VALUE GENERATION STOP */ uint32_t flags; /* Index configuration flags */ }; diff --git a/src/third_party/wiredtiger/src/include/serial.i b/src/third_party/wiredtiger/src/include/serial.i index d471ebb399c..02f15cdb8af 100644 --- a/src/third_party/wiredtiger/src/include/serial.i +++ b/src/third_party/wiredtiger/src/include/serial.i @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -157,9 +157,10 @@ __wt_col_append_serial(WT_SESSION_IMPL *session, WT_PAGE *page, size_t new_ins_size, uint64_t *recnop, u_int skipdepth, bool exclusive) { WT_DECL_RET; - WT_INSERT *new_ins = *new_insp; + WT_INSERT *new_ins; /* Clear references to memory we now own and must free on error. */ + new_ins = *new_insp; *new_insp = NULL; /* Check for page write generation wrap. */ @@ -206,11 +207,12 @@ __wt_insert_serial(WT_SESSION_IMPL *session, WT_PAGE *page, size_t new_ins_size, u_int skipdepth, bool exclusive) { WT_DECL_RET; - WT_INSERT *new_ins = *new_insp; + WT_INSERT *new_ins; u_int i; bool simple; /* Clear references to memory we now own and must free on error. */ + new_ins = *new_insp; *new_insp = NULL; /* Check for page write generation wrap. */ @@ -262,11 +264,13 @@ __wt_update_serial(WT_SESSION_IMPL *session, WT_PAGE *page, WT_UPDATE **srch_upd, WT_UPDATE **updp, size_t upd_size, bool exclusive) { WT_DECL_RET; - WT_UPDATE *obsolete, *upd = *updp; + WT_UPDATE *obsolete, *upd; wt_timestamp_t *obsolete_timestamp; + size_t size; uint64_t txn; /* Clear references to memory we now own and must free on error. */ + upd = *updp; *updp = NULL; /* Check for page write generation wrap. */ @@ -329,9 +333,20 @@ __wt_update_serial(WT_SESSION_IMPL *session, WT_PAGE *page, return (0); obsolete = __wt_update_obsolete_check(session, page, upd->next); + + /* + * Decrement the dirty byte count while holding the page lock, else we + * can race with checkpoints cleaning a page. + */ + for (size = 0, upd = obsolete; upd != NULL; upd = upd->next) + size += WT_UPDATE_MEMSIZE(upd); + if (size != 0) + __wt_cache_page_inmem_decr(session, page, size); + WT_PAGE_UNLOCK(session, page); + if (obsolete != NULL) - __wt_update_obsolete_free(session, page, obsolete); + __wt_free_update_list(session, obsolete); return (0); } diff --git a/src/third_party/wiredtiger/src/include/session.h b/src/third_party/wiredtiger/src/include/session.h index 23cf136d0aa..0f3b2488e9f 100644 --- a/src/third_party/wiredtiger/src/include/session.h +++ b/src/third_party/wiredtiger/src/include/session.h @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -148,6 +148,30 @@ struct __wt_session_impl { /* Sessions have an associated statistics bucket based on its ID. */ u_int stat_bucket; /* Statistics bucket offset */ +/* AUTOMATIC FLAG VALUE GENERATION START */ +#define WT_SESSION_CAN_WAIT 0x000001u +#define WT_SESSION_IGNORE_CACHE_SIZE 0x000002u +#define WT_SESSION_INTERNAL 0x000004u +#define WT_SESSION_LOCKED_CHECKPOINT 0x000008u +#define WT_SESSION_LOCKED_HANDLE_LIST_READ 0x000010u +#define WT_SESSION_LOCKED_HANDLE_LIST_WRITE 0x000020u +#define WT_SESSION_LOCKED_METADATA 0x000040u +#define WT_SESSION_LOCKED_PASS 0x000080u +#define WT_SESSION_LOCKED_SCHEMA 0x000100u +#define WT_SESSION_LOCKED_SLOT 0x000200u +#define WT_SESSION_LOCKED_TABLE_READ 0x000400u +#define WT_SESSION_LOCKED_TABLE_WRITE 0x000800u +#define WT_SESSION_LOCKED_TURTLE 0x001000u +#define WT_SESSION_LOGGING_INMEM 0x002000u +#define WT_SESSION_LOOKASIDE_CURSOR 0x004000u +#define WT_SESSION_NO_DATA_HANDLES 0x008000u +#define WT_SESSION_NO_LOGGING 0x010000u +#define WT_SESSION_NO_RECONCILE 0x020000u +#define WT_SESSION_NO_SCHEMA_LOCK 0x040000u +#define WT_SESSION_QUIET_CORRUPT_FILE 0x080000u +#define WT_SESSION_READ_WONT_NEED 0x100000u +#define WT_SESSION_SERVER_ASYNC 0x200000u +/* AUTOMATIC FLAG VALUE GENERATION STOP */ uint32_t flags; /* @@ -213,4 +237,12 @@ struct __wt_session_impl { uint32_t hazard_inuse; /* Hazard pointer array slots in-use */ uint32_t nhazard; /* Count of active hazard pointers */ WT_HAZARD *hazard; /* Hazard pointer array */ + + /* + * Operation tracking. + */ + WT_OPTRACK_RECORD *optrack_buf; + u_int optrackbuf_ptr; + uint64_t optrack_offset; + WT_FH *optrack_fh; }; diff --git a/src/third_party/wiredtiger/src/include/stat.h b/src/third_party/wiredtiger/src/include/stat.h index 438333858b9..8b8c3a55a6c 100644 --- a/src/third_party/wiredtiger/src/include/stat.h +++ b/src/third_party/wiredtiger/src/include/stat.h @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -84,6 +84,17 @@ #define WT_STATS_FIELD_TO_OFFSET(stats, fld) \ (int)(&(stats)[0]->fld - (int64_t *)(stats)[0]) +/* AUTOMATIC FLAG VALUE GENERATION START */ +#define WT_STAT_CLEAR 0x01u +#define WT_STAT_JSON 0x02u +#define WT_STAT_ON_CLOSE 0x04u +#define WT_STAT_TYPE_ALL 0x08u +#define WT_STAT_TYPE_CACHE_WALK 0x10u +#define WT_STAT_TYPE_FAST 0x20u +#define WT_STAT_TYPE_SIZE 0x40u +#define WT_STAT_TYPE_TREE_WALK 0x80u +/* AUTOMATIC FLAG VALUE GENERATION STOP */ + /* * Sum the values from all structures in the array. */ @@ -248,6 +259,51 @@ __wt_stats_clear(void *stats_arg, int slot) } while (0) /* + * Construct histogram increment functions to put the passed value into the + * right bucket. Bucket ranges, represented by various statistics, depend upon + * whether the passed value is in milliseconds or microseconds. Also values + * less than a given minimum are ignored and not put in any bucket. This floor + * value keeps us from having an excessively large smallest values. + */ +#define WT_STAT_MSECS_HIST_INCR_FUNC(name, stat, min_val) \ +static inline void \ +__wt_stat_msecs_hist_incr_##name(WT_SESSION_IMPL *session, uint64_t msecs) \ +{ \ + if (msecs < (min_val)) \ + return; \ + if (msecs < 50) \ + WT_STAT_CONN_INCR(session, stat##_lt50); \ + else if (msecs < 100) \ + WT_STAT_CONN_INCR(session, stat##_lt100); \ + else if (msecs < 250) \ + WT_STAT_CONN_INCR(session, stat##_lt250); \ + else if (msecs < 500) \ + WT_STAT_CONN_INCR(session, stat##_lt500); \ + else if (msecs < 1000) \ + WT_STAT_CONN_INCR(session, stat##_lt1000); \ + else \ + WT_STAT_CONN_INCR(session, stat##_gt1000); \ +} + +#define WT_STAT_USECS_HIST_INCR_FUNC(name, stat, min_val) \ +static inline void \ +__wt_stat_usecs_hist_incr_##name(WT_SESSION_IMPL *session, uint64_t usecs) \ +{ \ + if (usecs < (min_val)) \ + return; \ + if (usecs < 250) \ + WT_STAT_CONN_INCR(session, stat##_lt250); \ + else if (usecs < 500) \ + WT_STAT_CONN_INCR(session, stat##_lt500); \ + else if (usecs < 1000) \ + WT_STAT_CONN_INCR(session, stat##_lt1000); \ + else if (usecs < 10000) \ + WT_STAT_CONN_INCR(session, stat##_lt10000); \ + else \ + WT_STAT_CONN_INCR(session, stat##_gt10000); \ +} + +/* * DO NOT EDIT: automatically built by dist/stat.py. */ /* Statistics section: BEGIN */ @@ -364,6 +420,8 @@ struct __wt_connection_stats { int64_t cache_eviction_pages_queued_oldest; int64_t cache_read; int64_t cache_read_lookaside; + int64_t cache_read_lookaside_skipped; + int64_t cache_read_lookaside_delay; int64_t cache_pages_requested; int64_t cache_eviction_pages_seen; int64_t cache_eviction_fail; @@ -413,6 +471,10 @@ struct __wt_connection_stats { int64_t lock_checkpoint_count; int64_t lock_checkpoint_wait_application; int64_t lock_checkpoint_wait_internal; + int64_t lock_commit_timestamp_wait_application; + int64_t lock_commit_timestamp_wait_internal; + int64_t lock_commit_timestamp_read_count; + int64_t lock_commit_timestamp_write_count; int64_t lock_dhandle_wait_application; int64_t lock_dhandle_wait_internal; int64_t lock_dhandle_read_count; @@ -420,6 +482,10 @@ struct __wt_connection_stats { int64_t lock_metadata_count; int64_t lock_metadata_wait_application; int64_t lock_metadata_wait_internal; + int64_t lock_read_timestamp_wait_application; + int64_t lock_read_timestamp_wait_internal; + int64_t lock_read_timestamp_read_count; + int64_t lock_read_timestamp_write_count; int64_t lock_schema_count; int64_t lock_schema_wait_application; int64_t lock_schema_wait_internal; @@ -427,6 +493,10 @@ struct __wt_connection_stats { int64_t lock_table_wait_internal; int64_t lock_table_read_count; int64_t lock_table_write_count; + int64_t lock_txn_global_wait_application; + int64_t lock_txn_global_wait_internal; + int64_t lock_txn_global_read_count; + int64_t lock_txn_global_write_count; int64_t log_slot_switch_busy; int64_t log_force_ckpt_sleep; int64_t log_bytes_payload; @@ -473,6 +543,28 @@ struct __wt_connection_stats { int64_t log_compress_len; int64_t log_slot_coalesced; int64_t log_close_yields; + int64_t perf_hist_fsread_latency_lt50; + int64_t perf_hist_fsread_latency_lt100; + int64_t perf_hist_fsread_latency_lt250; + int64_t perf_hist_fsread_latency_lt500; + int64_t perf_hist_fsread_latency_lt1000; + int64_t perf_hist_fsread_latency_gt1000; + int64_t perf_hist_fswrite_latency_lt50; + int64_t perf_hist_fswrite_latency_lt100; + int64_t perf_hist_fswrite_latency_lt250; + int64_t perf_hist_fswrite_latency_lt500; + int64_t perf_hist_fswrite_latency_lt1000; + int64_t perf_hist_fswrite_latency_gt1000; + int64_t perf_hist_opread_latency_lt250; + int64_t perf_hist_opread_latency_lt500; + int64_t perf_hist_opread_latency_lt1000; + int64_t perf_hist_opread_latency_lt10000; + int64_t perf_hist_opread_latency_gt10000; + int64_t perf_hist_opwrite_latency_lt250; + int64_t perf_hist_opwrite_latency_lt500; + int64_t perf_hist_opwrite_latency_lt1000; + int64_t perf_hist_opwrite_latency_lt10000; + int64_t perf_hist_opwrite_latency_gt10000; int64_t rec_page_delete_fast; int64_t rec_pages; int64_t rec_pages_eviction; @@ -518,8 +610,24 @@ struct __wt_connection_stats { int64_t page_del_rollback_blocked; int64_t child_modify_blocked_page; int64_t tree_descend_blocked; + int64_t txn_commit_queue_empty; + int64_t txn_commit_queue_head; + int64_t txn_commit_queue_inserts; + int64_t txn_commit_queue_len; int64_t txn_snapshots_created; int64_t txn_snapshots_dropped; + int64_t txn_query_ts; + int64_t txn_read_queue_empty; + int64_t txn_read_queue_head; + int64_t txn_read_queue_inserts; + int64_t txn_read_queue_len; + int64_t txn_set_ts; + int64_t txn_set_ts_commit; + int64_t txn_set_ts_commit_upd; + int64_t txn_set_ts_oldest; + int64_t txn_set_ts_oldest_upd; + int64_t txn_set_ts_stable; + int64_t txn_set_ts_stable_upd; int64_t txn_begin; int64_t txn_checkpoint_running; int64_t txn_checkpoint_generation; @@ -540,13 +648,7 @@ struct __wt_connection_stats { int64_t txn_pinned_timestamp; int64_t txn_pinned_timestamp_oldest; int64_t txn_sync; - int64_t txn_commit_queue_head; - int64_t txn_commit_queue_inserts; - int64_t txn_commit_queue_len; int64_t txn_commit; - int64_t txn_read_queue_head; - int64_t txn_read_queue_inserts; - int64_t txn_read_queue_len; int64_t txn_rollback; int64_t txn_update_conflict; }; diff --git a/src/third_party/wiredtiger/src/include/swap.h b/src/third_party/wiredtiger/src/include/swap.h index bd28296e668..ab690430550 100644 --- a/src/third_party/wiredtiger/src/include/swap.h +++ b/src/third_party/wiredtiger/src/include/swap.h @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/include/thread_group.h b/src/third_party/wiredtiger/src/include/thread_group.h index 97eda6ab674..5b856e3a59c 100644 --- a/src/third_party/wiredtiger/src/include/thread_group.h +++ b/src/third_party/wiredtiger/src/include/thread_group.h @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -21,11 +21,13 @@ struct __wt_thread { * WT_THREAD and thread-group function flags, merged because * WT_THREAD_PANIC_FAIL appears in both groups. */ -#define WT_THREAD_ACTIVE 0x01 /* thread is active or paused */ -#define WT_THREAD_CAN_WAIT 0x02 /* WT_SESSION_CAN_WAIT */ -#define WT_THREAD_LOOKASIDE 0x04 /* open lookaside cursor */ -#define WT_THREAD_PANIC_FAIL 0x08 /* panic if the thread fails */ -#define WT_THREAD_RUN 0x10 /* thread is running */ +/* AUTOMATIC FLAG VALUE GENERATION START */ +#define WT_THREAD_ACTIVE 0x01u /* thread is active or paused */ +#define WT_THREAD_CAN_WAIT 0x02u /* WT_SESSION_CAN_WAIT */ +#define WT_THREAD_LOOKASIDE 0x04u /* open lookaside cursor */ +#define WT_THREAD_PANIC_FAIL 0x08u /* panic if the thread fails */ +#define WT_THREAD_RUN 0x10u /* thread is running */ +/* AUTOMATIC FLAG VALUE GENERATION STOP */ uint32_t flags; /* diff --git a/src/third_party/wiredtiger/src/include/txn.h b/src/third_party/wiredtiger/src/include/txn.h index 6b78c78a5cd..3f48368b303 100644 --- a/src/third_party/wiredtiger/src/include/txn.h +++ b/src/third_party/wiredtiger/src/include/txn.h @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -10,6 +10,19 @@ #define WT_TXN_FIRST 1 /* First transaction to run. */ #define WT_TXN_ABORTED UINT64_MAX /* Update rolled back, ignore. */ +/* AUTOMATIC FLAG VALUE GENERATION START */ +#define WT_TXN_LOG_CKPT_CLEANUP 0x01u +#define WT_TXN_LOG_CKPT_PREPARE 0x02u +#define WT_TXN_LOG_CKPT_START 0x04u +#define WT_TXN_LOG_CKPT_STOP 0x08u +#define WT_TXN_LOG_CKPT_SYNC 0x10u +/* AUTOMATIC FLAG VALUE GENERATION STOP */ + +/* AUTOMATIC FLAG VALUE GENERATION START */ +#define WT_TXN_OLDEST_STRICT 0x1u +#define WT_TXN_OLDEST_WAIT 0x2u +/* AUTOMATIC FLAG VALUE GENERATION STOP */ + /* * Transaction ID comparison dealing with edge cases. * @@ -246,19 +259,25 @@ struct __wt_txn { WT_ITEM *ckpt_snapshot; bool full_ckpt; -#define WT_TXN_AUTOCOMMIT 0x00001 -#define WT_TXN_ERROR 0x00002 -#define WT_TXN_HAS_ID 0x00004 -#define WT_TXN_HAS_SNAPSHOT 0x00008 -#define WT_TXN_HAS_TS_COMMIT 0x00010 -#define WT_TXN_HAS_TS_READ 0x00020 -#define WT_TXN_NAMED_SNAPSHOT 0x00040 -#define WT_TXN_PUBLIC_TS_COMMIT 0x00080 -#define WT_TXN_PUBLIC_TS_READ 0x00100 -#define WT_TXN_READONLY 0x00200 -#define WT_TXN_RUNNING 0x00400 -#define WT_TXN_SYNC_SET 0x00800 -#define WT_TXN_TS_COMMIT_ALWAYS 0x01000 -#define WT_TXN_TS_COMMIT_NEVER 0x02000 + const char *rollback_reason; /* If rollback, the reason */ + +/* AUTOMATIC FLAG VALUE GENERATION START */ +#define WT_TXN_AUTOCOMMIT 0x0001u +#define WT_TXN_ERROR 0x0002u +#define WT_TXN_HAS_ID 0x0004u +#define WT_TXN_HAS_SNAPSHOT 0x0008u +#define WT_TXN_HAS_TS_COMMIT 0x0010u +#define WT_TXN_HAS_TS_READ 0x0020u +#define WT_TXN_NAMED_SNAPSHOT 0x0040u +#define WT_TXN_PUBLIC_TS_COMMIT 0x0080u +#define WT_TXN_PUBLIC_TS_READ 0x0100u +#define WT_TXN_READONLY 0x0200u +#define WT_TXN_RUNNING 0x0400u +#define WT_TXN_SYNC_SET 0x0800u +#define WT_TXN_TS_COMMIT_ALWAYS 0x1000u +#define WT_TXN_TS_COMMIT_KEYS 0x2000u +#define WT_TXN_TS_COMMIT_NEVER 0x4000u +#define WT_TXN_UPDATE 0x8000u +/* AUTOMATIC FLAG VALUE GENERATION STOP */ uint32_t flags; }; diff --git a/src/third_party/wiredtiger/src/include/txn.i b/src/third_party/wiredtiger/src/include/txn.i index 1683ce8fbe2..fbd89195746 100644 --- a/src/third_party/wiredtiger/src/include/txn.i +++ b/src/third_party/wiredtiger/src/include/txn.i @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -26,6 +26,8 @@ __wt_txn_timestamp_flags(WT_SESSION_IMPL *session) return; if (FLD_ISSET(btree->assert_flags, WT_ASSERT_COMMIT_TS_ALWAYS)) F_SET(&session->txn, WT_TXN_TS_COMMIT_ALWAYS); + if (FLD_ISSET(btree->assert_flags, WT_ASSERT_COMMIT_TS_KEYS)) + F_SET(&session->txn, WT_TXN_TS_COMMIT_KEYS); if (FLD_ISSET(btree->assert_flags, WT_ASSERT_COMMIT_TS_NEVER)) F_SET(&session->txn, WT_TXN_TS_COMMIT_NEVER); } @@ -494,13 +496,25 @@ __wt_txn_upd_visible(WT_SESSION_IMPL *session, WT_UPDATE *upd) static inline WT_UPDATE * __wt_txn_read(WT_SESSION_IMPL *session, WT_UPDATE *upd) { - /* Skip reserved place-holders, they're never visible. */ - for (; upd != NULL; upd = upd->next) - if (upd->type != WT_UPDATE_RESERVED && + static WT_UPDATE tombstone = { + .txnid = WT_TXN_NONE, .type = WT_UPDATE_TOMBSTONE + }; + bool skipped_birthmark; + + for (skipped_birthmark = false; upd != NULL; upd = upd->next) { + /* Skip reserved place-holders, they're never visible. */ + if (upd->type != WT_UPDATE_RESERVE && __wt_txn_upd_visible(session, upd)) break; + /* An invisible birthmark is equivalent to a tombstone. */ + if (upd->type == WT_UPDATE_BIRTHMARK) + skipped_birthmark = true; + } + + if (upd == NULL && skipped_birthmark) + upd = &tombstone; - return (upd); + return (upd == NULL || upd->type == WT_UPDATE_BIRTHMARK ? NULL : upd); } /* @@ -721,7 +735,8 @@ __wt_txn_update_check(WT_SESSION_IMPL *session, WT_UPDATE *upd) session, txn_update_conflict); WT_STAT_DATA_INCR( session, txn_update_conflict); - return (WT_ROLLBACK); + return (__wt_txn_rollback_required(session, + "conflict between concurrent operations")); } upd = upd->next; } diff --git a/src/third_party/wiredtiger/src/include/verify_build.h b/src/third_party/wiredtiger/src/include/verify_build.h index 3973f786a90..8071efe4432 100644 --- a/src/third_party/wiredtiger/src/include/verify_build.h +++ b/src/third_party/wiredtiger/src/include/verify_build.h @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/include/wiredtiger.in b/src/third_party/wiredtiger/src/include/wiredtiger.in index d4c1fa956c3..53067bf44ab 100644 --- a/src/third_party/wiredtiger/src/include/wiredtiger.in +++ b/src/third_party/wiredtiger/src/include/wiredtiger.in @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -125,9 +125,11 @@ struct __wt_item { /*! Managed memory size (internal use). */ size_t memsize; -#define WT_ITEM_ALIGNED 0x00000001 -#define WT_ITEM_INUSE 0x00000002 /*! Object flags (internal use). */ +/* AUTOMATIC FLAG VALUE GENERATION START */ +#define WT_ITEM_ALIGNED 0x1u +#define WT_ITEM_INUSE 0x2u +/* AUTOMATIC FLAG VALUE GENERATION STOP */ uint32_t flags; #endif }; @@ -149,8 +151,11 @@ struct __wt_modify { /*! * The zero-based byte offset in the value where the new data is placed. * - * If the offset is past the end of the value, nul bytes are appended to - * the value up to the specified offset. + * If the offset is past the end of the value, padding bytes are + * appended to the value up to the specified offset. If the value is a + * string (value format \c S), the padding byte is a space. If the value + * is a raw byte array accessed using a WT_ITEM structure (value format + * \c u), the padding byte is a nul. */ size_t offset; @@ -483,11 +488,12 @@ struct __wt_cursor { * Both the key and value must be set and the record must already exist; * the record will be updated. * - * Modification structures are applied in order, and later modifications - * can update earlier modifications. + * Modifications are specified in WT_MODIFY structures. Modifications + * are applied in order and later modifications can update earlier ones. * - * The modify method is only supported on raw byte arrays accessed using - * a WT_ITEM structure, that is, a format type of \c u. + * The modify method is only supported on strings (value format type + * \c S), or raw byte arrays accessed using a WT_ITEM structure (value + * format type \c u). * * Calling the WT_CURSOR::modify method outside of snapshot isolation * can lead to unexpected results. While \c read-committed isolation @@ -673,22 +679,24 @@ struct __wt_cursor { */ const char *internal_uri; -#define WT_CURSTD_APPEND 0x00001 -#define WT_CURSTD_BULK 0x00002 -#define WT_CURSTD_DUMP_HEX 0x00004 -#define WT_CURSTD_DUMP_JSON 0x00008 -#define WT_CURSTD_DUMP_PRINT 0x00010 -#define WT_CURSTD_JOINED 0x00020 -#define WT_CURSTD_KEY_EXT 0x00040 /* Key points out of the tree. */ -#define WT_CURSTD_KEY_INT 0x00080 /* Key points into the tree. */ +/* AUTOMATIC FLAG VALUE GENERATION START */ +#define WT_CURSTD_APPEND 0x0001u +#define WT_CURSTD_BULK 0x0002u +#define WT_CURSTD_DUMP_HEX 0x0004u +#define WT_CURSTD_DUMP_JSON 0x0008u +#define WT_CURSTD_DUMP_PRINT 0x0010u +#define WT_CURSTD_JOINED 0x0020u +#define WT_CURSTD_KEY_EXT 0x0040u /* Key points out of the tree. */ +#define WT_CURSTD_KEY_INT 0x0080u /* Key points into the tree. */ +#define WT_CURSTD_META_INUSE 0x0100u +#define WT_CURSTD_OPEN 0x0200u +#define WT_CURSTD_OVERWRITE 0x0400u +#define WT_CURSTD_RAW 0x0800u +#define WT_CURSTD_RAW_SEARCH 0x1000u +#define WT_CURSTD_VALUE_EXT 0x2000u /* Value points out of the tree. */ +#define WT_CURSTD_VALUE_INT 0x4000u /* Value points into the tree. */ +/* AUTOMATIC FLAG VALUE GENERATION STOP */ #define WT_CURSTD_KEY_SET (WT_CURSTD_KEY_EXT | WT_CURSTD_KEY_INT) -#define WT_CURSTD_META_INUSE 0x00100 -#define WT_CURSTD_OPEN 0x00200 -#define WT_CURSTD_OVERWRITE 0x00400 -#define WT_CURSTD_RAW 0x00800 -#define WT_CURSTD_RAW_SEARCH 0x01000 -#define WT_CURSTD_VALUE_EXT 0x02000 /* Value points out of the tree. */ -#define WT_CURSTD_VALUE_INT 0x04000 /* Value points into the tree. */ #define WT_CURSTD_VALUE_SET (WT_CURSTD_VALUE_EXT | WT_CURSTD_VALUE_INT) uint32_t flags; #endif @@ -1320,6 +1328,20 @@ struct __wt_session { * for chunks to be temporarily larger than this value. This overrides * the \c memory_page_max setting., an integer between 512K and 500MB; * default \c 10MB.} + * @config{ merge_custom = (, + * configure the tree to merge into a custom data source., a set of + * related configuration options defined below.} + * @config{ prefix, + * custom data source prefix instead of \c "file"., a string; default + * empty.} + * @config{ start + * _generation, merge generation at which the custom data source is used + * (zero indicates no custom data source)., an integer between 0 and 10; + * default \c 0.} + * @config{ suffix, + * custom data source suffix instead of \c ".lsm"., a string; default + * empty.} + * @config{ ),,} * @config{ merge_max, the * maximum number of chunks to include in a merge operation., an integer * between 2 and 100; default \c 15.} @@ -1502,10 +1524,10 @@ struct __wt_session { * called). * * @param session the session handle - * @param fmt a printf format specifier + * @param format a printf format specifier * @errors */ - int __F(log_printf)(WT_SESSION *session, const char *fmt, ...); + int __F(log_printf)(WT_SESSION *session, const char *format, ...); /*! * Rebalance a table or file, see @ref rebalance. @@ -1906,6 +1928,19 @@ struct __wt_session { */ int __F(transaction_sync)(WT_SESSION *session, const char *config); /*! @} */ + +#ifndef DOXYGEN + /*! + * Call into the library. + * + * This method is used for breakpoints and to set other configuration + * when debugging layers not directly supporting those features. + * + * @param session the session handle + * @errors + */ + int __F(breakpoint)(WT_SESSION *session); +#endif }; /*! @@ -2081,27 +2116,36 @@ struct __wt_connection { * current eviction load., an integer between 1 and 20; default \c 1.} * @config{ ),,} * @config{eviction_checkpoint_target, perform eviction at the beginning - * of checkpoints to bring the dirty content in cache to this level\, - * expressed as a percentage of the total cache size. Ignored if set to - * zero or \c in_memory is \c true., an integer between 0 and 99; - * default \c 5.} + * of checkpoints to bring the dirty content in cache to this level. It + * is a percentage of the cache size if the value is within the range of + * 0 to 100 or an absolute size when greater than 100. The value is not + * allowed to exceed the \c cache_size. Ignored if set to zero or \c + * in_memory is \c true., an integer between 0 and 10TB; default \c 5.} * @config{eviction_dirty_target, perform eviction in worker threads - * when the cache contains at least this much dirty content\, expressed - * as a percentage of the total cache size., an integer between 1 and - * 99; default \c 5.} + * when the cache contains at least this much dirty content. It is a + * percentage of the cache size if the value is within the range of 1 to + * 100 or an absolute size when greater than 100. The value is not + * allowed to exceed the \c cache_size., an integer between 1 and 10TB; + * default \c 5.} * @config{eviction_dirty_trigger, trigger application threads to * perform eviction when the cache contains at least this much dirty - * content\, expressed as a percentage of the total cache size. This - * setting only alters behavior if it is lower than eviction_trigger., - * an integer between 1 and 99; default \c 20.} + * content. It is a percentage of the cache size if the value is within + * the range of 1 to 100 or an absolute size when greater than 100. The + * value is not allowed to exceed the \c cache_size. This setting only + * alters behavior if it is lower than eviction_trigger., an integer + * between 1 and 10TB; default \c 20.} * @config{eviction_target, perform eviction in worker threads when the - * cache contains at least this much content\, expressed as a percentage - * of the total cache size. Must be less than \c eviction_trigger., an - * integer between 10 and 99; default \c 80.} + * cache contains at least this much content. It is a percentage of the + * cache size if the value is within the range of 10 to 100 or an + * absolute size when greater than 100. The value is not allowed to + * exceed the \c cache_size., an integer between 10 and 10TB; default \c + * 80.} * @config{eviction_trigger, trigger application threads to perform - * eviction when the cache contains at least this much content\, - * expressed as a percentage of the total cache size., an integer - * between 10 and 99; default \c 95.} + * eviction when the cache contains at least this much content. It is a + * percentage of the cache size if the value is within the range of 10 + * to 100 or an absolute size when greater than 100. The value is not + * allowed to exceed the \c cache_size., an integer between 10 and 10TB; + * default \c 95.} * @config{file_manager = (, control how file handles are managed., a * set of related configuration options defined below.} * @config{ close_handle_minimum, number of @@ -2137,16 +2181,29 @@ struct __wt_connection { * thread uses a session handle from the configured session_max., an * integer between 3 and 20; default \c 4.} * @config{ ),,} + * @config{operation_tracking = (, enable tracking of + * performance-critical functions. See @ref operation_tracking for more + * information., a set of related configuration options defined below.} + * @config{ enabled, enable operation tracking + * subsystem., a boolean flag; default \c false.} + * @config{ path, the name of a directory into + * which operation tracking files are written. The directory must + * already exist. If the value is not an absolute path\, the path is + * relative to the database home (see @ref absolute_path for more + * information)., a string; default \c ".".} + * @config{ ),,} * @config{shared_cache = (, shared cache configuration options. A * database should configure either a cache_size or a shared_cache not * both. Enabling a shared cache uses a session from the configured - * session_max., a set of related configuration options defined below.} - * @config{ chunk, the granularity that a shared - * cache is redistributed., an integer between 1MB and 10TB; default \c - * 10MB.} - * @config{ name, the name of a cache that - * is shared between databases or \c "none" when no shared cache is - * configured., a string; default \c none.} + * session_max. A shared cache can not have absolute values configured + * for cache eviction settings., a set of related configuration options + * defined below.} + * @config{ chunk, the + * granularity that a shared cache is redistributed., an integer between + * 1MB and 10TB; default \c 10MB.} + * @config{ name, + * the name of a cache that is shared between databases or \c "none" + * when no shared cache is configured., a string; default \c none.} * @config{ quota, maximum size of cache this * database can be allocated from the shared cache. Defaults to the * entire shared cache size., an integer; default \c 0.} @@ -2196,17 +2253,17 @@ struct __wt_connection { * write of the log records; setting this value above 0 configures * statistics logging., an integer between 0 and 100000; default \c 0.} * @config{ ),,} - * @config{verbose, enable messages for various events. Only available - * if WiredTiger is configured with --enable-verbose. Options are given - * as a list\, such as <code>"verbose=[evictserver\,read]"</code>., a - * list\, with values chosen from the following options: \c "api"\, \c - * "block"\, \c "checkpoint"\, \c "checkpoint_progress"\, \c "compact"\, - * \c "evict"\, \c "evict_stuck"\, \c "evictserver"\, \c "fileops"\, \c - * "handleops"\, \c "log"\, \c "lookaside"\, \c "lookaside_activity"\, - * \c "lsm"\, \c "lsm_manager"\, \c "metadata"\, \c "mutex"\, \c - * "overflow"\, \c "read"\, \c "rebalance"\, \c "reconcile"\, \c - * "recovery"\, \c "recovery_progress"\, \c "salvage"\, \c - * "shared_cache"\, \c "split"\, \c "thread_group"\, \c "timestamp"\, \c + * @config{verbose, enable messages for various events. Options are + * given as a list\, such as + * <code>"verbose=[evictserver\,read]"</code>., a list\, with values + * chosen from the following options: \c "api"\, \c "block"\, \c + * "checkpoint"\, \c "checkpoint_progress"\, \c "compact"\, \c "evict"\, + * \c "evict_stuck"\, \c "evictserver"\, \c "fileops"\, \c "handleops"\, + * \c "log"\, \c "lookaside"\, \c "lookaside_activity"\, \c "lsm"\, \c + * "lsm_manager"\, \c "metadata"\, \c "mutex"\, \c "overflow"\, \c + * "read"\, \c "rebalance"\, \c "reconcile"\, \c "recovery"\, \c + * "recovery_progress"\, \c "salvage"\, \c "shared_cache"\, \c "split"\, + * \c "temporary"\, \c "thread_group"\, \c "timestamp"\, \c * "transaction"\, \c "verify"\, \c "version"\, \c "write"; default * empty.} * @configend @@ -2266,8 +2323,8 @@ struct __wt_connection { * @snippet ex_all.c Open a session * * @param connection the connection handle - * @param errhandler An error handler. If <code>NULL</code>, the - * connection's error handler is used. See @ref error_handling_event + * @param event_handler An event handler. If <code>NULL</code>, the + * connection's event handler is used. See @ref event_message_handling * for more information. * @configstart{WT_CONNECTION.open_session, see dist/api_data.py} * @config{ignore_cache_size, when set\, operations performed by this @@ -2284,7 +2341,7 @@ struct __wt_connection { * @errors */ int __F(open_session)(WT_CONNECTION *connection, - WT_EVENT_HANDLER *errhandler, const char *config, + WT_EVENT_HANDLER *event_handler, const char *config, WT_SESSION **sessionp); /*! @} */ @@ -2556,9 +2613,9 @@ struct __wt_connection { * * @param home The path to the database home directory. See @ref home * for more information. - * @param errhandler An error handler. If <code>NULL</code>, a builtin error - * handler is installed that writes error messages to stderr. See - * @ref error_handling_event for more information. + * @param event_handler An event handler. If <code>NULL</code>, a default + * event handler is installed that writes error messages to stderr. See + * @ref event_message_handling for more information. * @configstart{wiredtiger_open, see dist/api_data.py} * @config{async = (, asynchronous operations configuration options., a set of * related configuration options defined below.} @@ -2669,25 +2726,32 @@ struct __wt_connection { * @config{ * ),,} * @config{eviction_checkpoint_target, perform eviction at the beginning of - * checkpoints to bring the dirty content in cache to this level\, expressed as - * a percentage of the total cache size. Ignored if set to zero or \c in_memory - * is \c true., an integer between 0 and 99; default \c 5.} + * checkpoints to bring the dirty content in cache to this level. It is a + * percentage of the cache size if the value is within the range of 0 to 100 or + * an absolute size when greater than 100. The value is not allowed to exceed + * the \c cache_size. Ignored if set to zero or \c in_memory is \c true., an + * integer between 0 and 10TB; default \c 5.} * @config{eviction_dirty_target, perform eviction in worker threads when the - * cache contains at least this much dirty content\, expressed as a percentage - * of the total cache size., an integer between 1 and 99; default \c 5.} + * cache contains at least this much dirty content. It is a percentage of the + * cache size if the value is within the range of 1 to 100 or an absolute size + * when greater than 100. The value is not allowed to exceed the \c cache_size., + * an integer between 1 and 10TB; default \c 5.} * @config{eviction_dirty_trigger, trigger application threads to perform - * eviction when the cache contains at least this much dirty content\, expressed - * as a percentage of the total cache size. This setting only alters behavior - * if it is lower than eviction_trigger., an integer between 1 and 99; default - * \c 20.} + * eviction when the cache contains at least this much dirty content. It is a + * percentage of the cache size if the value is within the range of 1 to 100 or + * an absolute size when greater than 100. The value is not allowed to exceed + * the \c cache_size. This setting only alters behavior if it is lower than + * eviction_trigger., an integer between 1 and 10TB; default \c 20.} * @config{eviction_target, perform eviction in worker threads when the cache - * contains at least this much content\, expressed as a percentage of the total - * cache size. Must be less than \c eviction_trigger., an integer between 10 - * and 99; default \c 80.} + * contains at least this much content. It is a percentage of the cache size if + * the value is within the range of 10 to 100 or an absolute size when greater + * than 100. The value is not allowed to exceed the \c cache_size., an integer + * between 10 and 10TB; default \c 80.} * @config{eviction_trigger, trigger application threads to perform eviction - * when the cache contains at least this much content\, expressed as a - * percentage of the total cache size., an integer between 10 and 99; default \c - * 95.} + * when the cache contains at least this much content. It is a percentage of + * the cache size if the value is within the range of 10 to 100 or an absolute + * size when greater than 100. The value is not allowed to exceed the \c + * cache_size., an integer between 10 and 10TB; default \c 95.} * @config{exclusive, fail if the database already exists\, generally used with * the \c create option., a boolean flag; default \c false.} * @config{extensions, list of shared library extensions to load (using dlopen). @@ -2759,6 +2823,16 @@ struct __wt_connection { * start an RPC server for primary processes and use RPC for secondary * processes). <b>Not yet supported in WiredTiger</b>., a boolean flag; default * \c false.} + * @config{operation_tracking = (, enable tracking of performance-critical + * functions. See @ref operation_tracking for more information., a set of + * related configuration options defined below.} + * @config{ enabled, enable operation tracking + * subsystem., a boolean flag; default \c false.} + * @config{ path, the name of a directory into which + * operation tracking files are written. The directory must already exist. If + * the value is not an absolute path\, the path is relative to the database home + * (see @ref absolute_path for more information)., a string; default \c ".".} + * @config{ ),,} * @config{readonly, open connection in read-only mode. The database must * exist. All methods that may modify a database are disabled. See @ref * readonly for more information., a boolean flag; default \c false.} @@ -2766,8 +2840,9 @@ struct __wt_connection { * threads)., an integer greater than or equal to 1; default \c 100.} * @config{shared_cache = (, shared cache configuration options. A database * should configure either a cache_size or a shared_cache not both. Enabling a - * shared cache uses a session from the configured session_max., a set of - * related configuration options defined below.} + * shared cache uses a session from the configured session_max. A shared cache + * can not have absolute values configured for cache eviction settings., a set + * of related configuration options defined below.} * @config{ chunk, the granularity that a shared cache is * redistributed., an integer between 1MB and 10TB; default \c 10MB.} * @config{ name, the name of a cache that is shared @@ -2839,8 +2914,7 @@ struct __wt_connection { * WIREDTIGER_HOME environment variables even if the process is running with * special privileges. See @ref home for more information., a boolean flag; * default \c false.} - * @config{verbose, enable messages for various events. Only available if - * WiredTiger is configured with --enable-verbose. Options are given as a + * @config{verbose, enable messages for various events. Options are given as a * list\, such as <code>"verbose=[evictserver\,read]"</code>., a list\, with * values chosen from the following options: \c "api"\, \c "block"\, \c * "checkpoint"\, \c "checkpoint_progress"\, \c "compact"\, \c "evict"\, \c @@ -2848,8 +2922,9 @@ struct __wt_connection { * \c "lookaside"\, \c "lookaside_activity"\, \c "lsm"\, \c "lsm_manager"\, \c * "metadata"\, \c "mutex"\, \c "overflow"\, \c "read"\, \c "rebalance"\, \c * "reconcile"\, \c "recovery"\, \c "recovery_progress"\, \c "salvage"\, \c - * "shared_cache"\, \c "split"\, \c "thread_group"\, \c "timestamp"\, \c - * "transaction"\, \c "verify"\, \c "version"\, \c "write"; default empty.} + * "shared_cache"\, \c "split"\, \c "temporary"\, \c "thread_group"\, \c + * "timestamp"\, \c "transaction"\, \c "verify"\, \c "version"\, \c "write"; + * default empty.} * @config{write_through, Use \c FILE_FLAG_WRITE_THROUGH on Windows to write to * files. Ignored on non-Windows systems. Options are given as a list\, such * as <code>"write_through=[data]"</code>. Configuring \c write_through requires @@ -2868,7 +2943,7 @@ struct __wt_connection { * @errors */ int wiredtiger_open(const char *home, - WT_EVENT_HANDLER *errhandler, const char *config, + WT_EVENT_HANDLER *event_handler, const char *config, WT_CONNECTION **connectionp) WT_ATTRIBUTE_LIBRARY_VISIBLE; /*! @@ -2916,14 +2991,15 @@ struct __wt_async_callback { struct __wt_event_handler { /*! * Callback to handle error messages; by default, error messages are - * written to the stderr stream. See @ref error_handling. + * written to the stderr stream. See @ref event_message_handling for + * more information. * * Errors that require the application to exit and restart will have * their \c error value set to \c WT_PANIC. The application can exit - * immediately when \c WT_PANIC is passed to an error handler, there + * immediately when \c WT_PANIC is passed to an event handler, there * is no reason to return into WiredTiger. * - * Error handler returns are not ignored: if the handler returns + * Event handler returns are not ignored: if the handler returns * non-zero, the error may cause the WiredTiger function posting the * event to fail, and may even cause operation or library failure. * @@ -2940,7 +3016,8 @@ struct __wt_event_handler { /*! * Callback to handle informational messages; by default, informational - * messages are written to the stdout stream. See @ref error_handling. + * messages are written to the stdout stream. See + * @ref event_message_handling for more information. * * Message handler returns are not ignored: if the handler returns * non-zero, the error may cause the WiredTiger function posting the @@ -2955,8 +3032,9 @@ struct __wt_event_handler { WT_SESSION *session, const char *message); /*! - * Callback to handle progress messages; by default, no progress - * messages are written. See @ref error_handling. + * Callback to handle progress messages; by default, progress messages + * are not written. See @ref event_message_handling for more + * information. * * Progress handler returns are not ignored: if the handler returns * non-zero, the error may cause the WiredTiger function posting the @@ -3012,12 +3090,12 @@ struct __wt_event_handler { * * @param session the session handle * @param buffer a pointer to a packed byte array - * @param size the number of valid bytes in the buffer + * @param len the number of valid bytes in the buffer * @param format the data format, see @ref packing * @errors */ int wiredtiger_struct_pack(WT_SESSION *session, - void *buffer, size_t size, const char *format, ...) + void *buffer, size_t len, const char *format, ...) WT_ATTRIBUTE_LIBRARY_VISIBLE; /*! @@ -3030,13 +3108,13 @@ int wiredtiger_struct_pack(WT_SESSION *session, * @snippet ex_all.c Get the packed size * * @param session the session handle - * @param sizep a location where the number of bytes needed for the + * @param lenp a location where the number of bytes needed for the * matching call to ::wiredtiger_struct_pack is returned * @param format the data format, see @ref packing * @errors */ int wiredtiger_struct_size(WT_SESSION *session, - size_t *sizep, const char *format, ...) WT_ATTRIBUTE_LIBRARY_VISIBLE; + size_t *lenp, const char *format, ...) WT_ATTRIBUTE_LIBRARY_VISIBLE; /*! * Unpack a structure from a buffer. @@ -3048,12 +3126,12 @@ int wiredtiger_struct_size(WT_SESSION *session, * * @param session the session handle * @param buffer a pointer to a packed byte array - * @param size the number of valid bytes in the buffer + * @param len the number of valid bytes in the buffer * @param format the data format, see @ref packing * @errors */ int wiredtiger_struct_unpack(WT_SESSION *session, - const void *buffer, size_t size, const char *format, ...) + const void *buffer, size_t len, const char *format, ...) WT_ATTRIBUTE_LIBRARY_VISIBLE; #if !defined(SWIG) @@ -3257,9 +3335,9 @@ struct __wt_config_item { * ::wiredtiger_open. * @param session the session handle (may be \c NULL if the database not yet * opened). - * @param errhandler An error handler (used if \c session is \c NULL; if both - * \c session and \c errhandler are \c NULL, error messages will be written to - * stderr). + * @param event_handler An event handler (used if \c session is \c NULL; if both + * \c session and \c event_handler are \c NULL, error messages will be written + * to stderr). * @param name the WiredTiger function or method to validate. * @param config the configuration string being parsed. * @returns zero for success, non-zero to indicate an error. @@ -3267,7 +3345,7 @@ struct __wt_config_item { * @snippet ex_all.c Validate a configuration string */ int wiredtiger_config_validate(WT_SESSION *session, - WT_EVENT_HANDLER *errhandler, const char *name, const char *config) + WT_EVENT_HANDLER *event_handler, const char *name, const char *config) WT_ATTRIBUTE_LIBRARY_VISIBLE; #endif @@ -3864,6 +3942,14 @@ struct __wt_data_source { const char *uri, WT_CONFIG_ARG *config); /*! + * Callback to get the size of an object. + * + * @snippet ex_data_source.c WT_DATA_SOURCE size + */ + int (*size)(WT_DATA_SOURCE *dsrc, WT_SESSION *session, + const char *uri, wt_off_t *size); + + /*! * Callback to truncate an object. * * @snippet ex_data_source.c WT_DATA_SOURCE truncate @@ -3904,6 +3990,17 @@ struct __wt_data_source { * @snippet ex_data_source.c WT_DATA_SOURCE terminate */ int (*terminate)(WT_DATA_SOURCE *dsrc, WT_SESSION *session); + + /*! + * If non-NULL, a callback performed before an LSM merge. + * + * @param[in] source a cursor configured with the data being merged + * @param[in] dest a cursor on the new object being filled by the merge + * + * @snippet ex_data_source.c WT_DATA_SOURCE lsm_pre_merge + */ + int (*lsm_pre_merge)( + WT_DATA_SOURCE *dsrc, WT_CURSOR *source, WT_CURSOR *dest); }; /*! @@ -4118,31 +4215,45 @@ typedef enum { WT_FS_OPEN_FILE_TYPE_REGULAR /*!< open a regular file */ } WT_FS_OPEN_FILE_TYPE; +#ifdef DOXYGEN /*! WT_FILE_SYSTEM::open_file flags: random access pattern */ -#define WT_FS_OPEN_ACCESS_RAND 0x001 +#define WT_FS_OPEN_ACCESS_RAND 0x0 /*! WT_FILE_SYSTEM::open_file flags: sequential access pattern */ -#define WT_FS_OPEN_ACCESS_SEQ 0x002 +#define WT_FS_OPEN_ACCESS_SEQ 0x0 /*! WT_FILE_SYSTEM::open_file flags: create if does not exist */ -#define WT_FS_OPEN_CREATE 0x004 +#define WT_FS_OPEN_CREATE 0x0 /*! WT_FILE_SYSTEM::open_file flags: direct I/O requested */ -#define WT_FS_OPEN_DIRECTIO 0x008 +#define WT_FS_OPEN_DIRECTIO 0x0 /*! WT_FILE_SYSTEM::open_file flags: file creation must be durable */ -#define WT_FS_OPEN_DURABLE 0x010 +#define WT_FS_OPEN_DURABLE 0x0 /*! * WT_FILE_SYSTEM::open_file flags: return EBUSY if exclusive use not available */ -#define WT_FS_OPEN_EXCLUSIVE 0x020 -#ifndef DOXYGEN -#define WT_FS_OPEN_FIXED 0x040 /* Path not home relative (internal) */ -#endif +#define WT_FS_OPEN_EXCLUSIVE 0x0 /*! WT_FILE_SYSTEM::open_file flags: open is read-only */ -#define WT_FS_OPEN_READONLY 0x080 +#define WT_FS_OPEN_READONLY 0x0 /*! * WT_FILE_SYSTEM::remove or WT_FILE_SYSTEM::rename flags: the remove or rename * operation must be durable */ -#define WT_FS_DURABLE 0x001 +#define WT_FS_DURABLE 0x0 +#else +/* AUTOMATIC FLAG VALUE GENERATION START */ +#define WT_FS_OPEN_ACCESS_RAND 0x01u +#define WT_FS_OPEN_ACCESS_SEQ 0x02u +#define WT_FS_OPEN_CREATE 0x04u +#define WT_FS_OPEN_DIRECTIO 0x08u +#define WT_FS_OPEN_DURABLE 0x10u +#define WT_FS_OPEN_EXCLUSIVE 0x20u +#define WT_FS_OPEN_FIXED 0x40u /* Path not home relative (internal) */ +#define WT_FS_OPEN_READONLY 0x80u +/* AUTOMATIC FLAG VALUE GENERATION STOP */ + +/* AUTOMATIC FLAG VALUE GENERATION START */ +#define WT_FS_DURABLE 0x1u +/* AUTOMATIC FLAG VALUE GENERATION STOP */ +#endif /*! * The interface implemented by applications to provide a custom file system @@ -4178,6 +4289,15 @@ struct __wt_file_system { WT_SESSION *session, const char *directory, const char *prefix, char ***dirlist, uint32_t *countp); +#if !defined(DOXYGEN) + /* + * Return a single file name for the named directory. + */ + int (*fs_directory_list_single)(WT_FILE_SYSTEM *file_system, + WT_SESSION *session, const char *directory, const char *prefix, + char ***dirlist, uint32_t *countp); +#endif + /*! * Free memory allocated by WT_FILE_SYSTEM::directory_list. * @@ -4908,415 +5028,528 @@ extern int wiredtiger_extension_terminate(WT_CONNECTION *connection); #define WT_STAT_CONN_CACHE_READ 1105 /*! cache: pages read into cache requiring lookaside entries */ #define WT_STAT_CONN_CACHE_READ_LOOKASIDE 1106 +/*! cache: pages read into cache skipping older lookaside entries */ +#define WT_STAT_CONN_CACHE_READ_LOOKASIDE_SKIPPED 1107 +/*! + * cache: pages read into cache with skipped lookaside entries needed + * later + */ +#define WT_STAT_CONN_CACHE_READ_LOOKASIDE_DELAY 1108 /*! cache: pages requested from the cache */ -#define WT_STAT_CONN_CACHE_PAGES_REQUESTED 1107 +#define WT_STAT_CONN_CACHE_PAGES_REQUESTED 1109 /*! cache: pages seen by eviction walk */ -#define WT_STAT_CONN_CACHE_EVICTION_PAGES_SEEN 1108 +#define WT_STAT_CONN_CACHE_EVICTION_PAGES_SEEN 1110 /*! cache: pages selected for eviction unable to be evicted */ -#define WT_STAT_CONN_CACHE_EVICTION_FAIL 1109 +#define WT_STAT_CONN_CACHE_EVICTION_FAIL 1111 /*! cache: pages walked for eviction */ -#define WT_STAT_CONN_CACHE_EVICTION_WALK 1110 +#define WT_STAT_CONN_CACHE_EVICTION_WALK 1112 /*! cache: pages written from cache */ -#define WT_STAT_CONN_CACHE_WRITE 1111 +#define WT_STAT_CONN_CACHE_WRITE 1113 /*! cache: pages written requiring in-memory restoration */ -#define WT_STAT_CONN_CACHE_WRITE_RESTORE 1112 +#define WT_STAT_CONN_CACHE_WRITE_RESTORE 1114 /*! cache: percentage overhead */ -#define WT_STAT_CONN_CACHE_OVERHEAD 1113 +#define WT_STAT_CONN_CACHE_OVERHEAD 1115 /*! cache: tracked bytes belonging to internal pages in the cache */ -#define WT_STAT_CONN_CACHE_BYTES_INTERNAL 1114 +#define WT_STAT_CONN_CACHE_BYTES_INTERNAL 1116 /*! cache: tracked bytes belonging to leaf pages in the cache */ -#define WT_STAT_CONN_CACHE_BYTES_LEAF 1115 +#define WT_STAT_CONN_CACHE_BYTES_LEAF 1117 /*! cache: tracked dirty bytes in the cache */ -#define WT_STAT_CONN_CACHE_BYTES_DIRTY 1116 +#define WT_STAT_CONN_CACHE_BYTES_DIRTY 1118 /*! cache: tracked dirty pages in the cache */ -#define WT_STAT_CONN_CACHE_PAGES_DIRTY 1117 +#define WT_STAT_CONN_CACHE_PAGES_DIRTY 1119 /*! cache: unmodified pages evicted */ -#define WT_STAT_CONN_CACHE_EVICTION_CLEAN 1118 +#define WT_STAT_CONN_CACHE_EVICTION_CLEAN 1120 /*! connection: auto adjusting condition resets */ -#define WT_STAT_CONN_COND_AUTO_WAIT_RESET 1119 +#define WT_STAT_CONN_COND_AUTO_WAIT_RESET 1121 /*! connection: auto adjusting condition wait calls */ -#define WT_STAT_CONN_COND_AUTO_WAIT 1120 +#define WT_STAT_CONN_COND_AUTO_WAIT 1122 /*! connection: detected system time went backwards */ -#define WT_STAT_CONN_TIME_TRAVEL 1121 +#define WT_STAT_CONN_TIME_TRAVEL 1123 /*! connection: files currently open */ -#define WT_STAT_CONN_FILE_OPEN 1122 +#define WT_STAT_CONN_FILE_OPEN 1124 /*! connection: memory allocations */ -#define WT_STAT_CONN_MEMORY_ALLOCATION 1123 +#define WT_STAT_CONN_MEMORY_ALLOCATION 1125 /*! connection: memory frees */ -#define WT_STAT_CONN_MEMORY_FREE 1124 +#define WT_STAT_CONN_MEMORY_FREE 1126 /*! connection: memory re-allocations */ -#define WT_STAT_CONN_MEMORY_GROW 1125 +#define WT_STAT_CONN_MEMORY_GROW 1127 /*! connection: pthread mutex condition wait calls */ -#define WT_STAT_CONN_COND_WAIT 1126 +#define WT_STAT_CONN_COND_WAIT 1128 /*! connection: pthread mutex shared lock read-lock calls */ -#define WT_STAT_CONN_RWLOCK_READ 1127 +#define WT_STAT_CONN_RWLOCK_READ 1129 /*! connection: pthread mutex shared lock write-lock calls */ -#define WT_STAT_CONN_RWLOCK_WRITE 1128 +#define WT_STAT_CONN_RWLOCK_WRITE 1130 /*! connection: total fsync I/Os */ -#define WT_STAT_CONN_FSYNC_IO 1129 +#define WT_STAT_CONN_FSYNC_IO 1131 /*! connection: total read I/Os */ -#define WT_STAT_CONN_READ_IO 1130 +#define WT_STAT_CONN_READ_IO 1132 /*! connection: total write I/Os */ -#define WT_STAT_CONN_WRITE_IO 1131 +#define WT_STAT_CONN_WRITE_IO 1133 /*! cursor: cursor create calls */ -#define WT_STAT_CONN_CURSOR_CREATE 1132 +#define WT_STAT_CONN_CURSOR_CREATE 1134 /*! cursor: cursor insert calls */ -#define WT_STAT_CONN_CURSOR_INSERT 1133 +#define WT_STAT_CONN_CURSOR_INSERT 1135 /*! cursor: cursor modify calls */ -#define WT_STAT_CONN_CURSOR_MODIFY 1134 +#define WT_STAT_CONN_CURSOR_MODIFY 1136 /*! cursor: cursor next calls */ -#define WT_STAT_CONN_CURSOR_NEXT 1135 +#define WT_STAT_CONN_CURSOR_NEXT 1137 /*! cursor: cursor prev calls */ -#define WT_STAT_CONN_CURSOR_PREV 1136 +#define WT_STAT_CONN_CURSOR_PREV 1138 /*! cursor: cursor remove calls */ -#define WT_STAT_CONN_CURSOR_REMOVE 1137 +#define WT_STAT_CONN_CURSOR_REMOVE 1139 /*! cursor: cursor reserve calls */ -#define WT_STAT_CONN_CURSOR_RESERVE 1138 +#define WT_STAT_CONN_CURSOR_RESERVE 1140 /*! cursor: cursor reset calls */ -#define WT_STAT_CONN_CURSOR_RESET 1139 +#define WT_STAT_CONN_CURSOR_RESET 1141 /*! cursor: cursor restarted searches */ -#define WT_STAT_CONN_CURSOR_RESTART 1140 +#define WT_STAT_CONN_CURSOR_RESTART 1142 /*! cursor: cursor search calls */ -#define WT_STAT_CONN_CURSOR_SEARCH 1141 +#define WT_STAT_CONN_CURSOR_SEARCH 1143 /*! cursor: cursor search near calls */ -#define WT_STAT_CONN_CURSOR_SEARCH_NEAR 1142 +#define WT_STAT_CONN_CURSOR_SEARCH_NEAR 1144 /*! cursor: cursor update calls */ -#define WT_STAT_CONN_CURSOR_UPDATE 1143 +#define WT_STAT_CONN_CURSOR_UPDATE 1145 /*! cursor: truncate calls */ -#define WT_STAT_CONN_CURSOR_TRUNCATE 1144 +#define WT_STAT_CONN_CURSOR_TRUNCATE 1146 /*! data-handle: connection data handles currently active */ -#define WT_STAT_CONN_DH_CONN_HANDLE_COUNT 1145 +#define WT_STAT_CONN_DH_CONN_HANDLE_COUNT 1147 /*! data-handle: connection sweep candidate became referenced */ -#define WT_STAT_CONN_DH_SWEEP_REF 1146 +#define WT_STAT_CONN_DH_SWEEP_REF 1148 /*! data-handle: connection sweep dhandles closed */ -#define WT_STAT_CONN_DH_SWEEP_CLOSE 1147 +#define WT_STAT_CONN_DH_SWEEP_CLOSE 1149 /*! data-handle: connection sweep dhandles removed from hash list */ -#define WT_STAT_CONN_DH_SWEEP_REMOVE 1148 +#define WT_STAT_CONN_DH_SWEEP_REMOVE 1150 /*! data-handle: connection sweep time-of-death sets */ -#define WT_STAT_CONN_DH_SWEEP_TOD 1149 +#define WT_STAT_CONN_DH_SWEEP_TOD 1151 /*! data-handle: connection sweeps */ -#define WT_STAT_CONN_DH_SWEEPS 1150 +#define WT_STAT_CONN_DH_SWEEPS 1152 /*! data-handle: session dhandles swept */ -#define WT_STAT_CONN_DH_SESSION_HANDLES 1151 +#define WT_STAT_CONN_DH_SESSION_HANDLES 1153 /*! data-handle: session sweep attempts */ -#define WT_STAT_CONN_DH_SESSION_SWEEPS 1152 +#define WT_STAT_CONN_DH_SESSION_SWEEPS 1154 /*! lock: checkpoint lock acquisitions */ -#define WT_STAT_CONN_LOCK_CHECKPOINT_COUNT 1153 +#define WT_STAT_CONN_LOCK_CHECKPOINT_COUNT 1155 /*! lock: checkpoint lock application thread wait time (usecs) */ -#define WT_STAT_CONN_LOCK_CHECKPOINT_WAIT_APPLICATION 1154 +#define WT_STAT_CONN_LOCK_CHECKPOINT_WAIT_APPLICATION 1156 /*! lock: checkpoint lock internal thread wait time (usecs) */ -#define WT_STAT_CONN_LOCK_CHECKPOINT_WAIT_INTERNAL 1155 +#define WT_STAT_CONN_LOCK_CHECKPOINT_WAIT_INTERNAL 1157 +/*! + * lock: commit timestamp queue lock application thread time waiting for + * the dhandle lock (usecs) + */ +#define WT_STAT_CONN_LOCK_COMMIT_TIMESTAMP_WAIT_APPLICATION 1158 +/*! + * lock: commit timestamp queue lock internal thread time waiting for the + * dhandle lock (usecs) + */ +#define WT_STAT_CONN_LOCK_COMMIT_TIMESTAMP_WAIT_INTERNAL 1159 +/*! lock: commit timestamp queue read lock acquisitions */ +#define WT_STAT_CONN_LOCK_COMMIT_TIMESTAMP_READ_COUNT 1160 +/*! lock: commit timestamp queue write lock acquisitions */ +#define WT_STAT_CONN_LOCK_COMMIT_TIMESTAMP_WRITE_COUNT 1161 /*! * lock: dhandle lock application thread time waiting for the dhandle * lock (usecs) */ -#define WT_STAT_CONN_LOCK_DHANDLE_WAIT_APPLICATION 1156 +#define WT_STAT_CONN_LOCK_DHANDLE_WAIT_APPLICATION 1162 /*! * lock: dhandle lock internal thread time waiting for the dhandle lock * (usecs) */ -#define WT_STAT_CONN_LOCK_DHANDLE_WAIT_INTERNAL 1157 +#define WT_STAT_CONN_LOCK_DHANDLE_WAIT_INTERNAL 1163 /*! lock: dhandle read lock acquisitions */ -#define WT_STAT_CONN_LOCK_DHANDLE_READ_COUNT 1158 +#define WT_STAT_CONN_LOCK_DHANDLE_READ_COUNT 1164 /*! lock: dhandle write lock acquisitions */ -#define WT_STAT_CONN_LOCK_DHANDLE_WRITE_COUNT 1159 +#define WT_STAT_CONN_LOCK_DHANDLE_WRITE_COUNT 1165 /*! lock: metadata lock acquisitions */ -#define WT_STAT_CONN_LOCK_METADATA_COUNT 1160 +#define WT_STAT_CONN_LOCK_METADATA_COUNT 1166 /*! lock: metadata lock application thread wait time (usecs) */ -#define WT_STAT_CONN_LOCK_METADATA_WAIT_APPLICATION 1161 +#define WT_STAT_CONN_LOCK_METADATA_WAIT_APPLICATION 1167 /*! lock: metadata lock internal thread wait time (usecs) */ -#define WT_STAT_CONN_LOCK_METADATA_WAIT_INTERNAL 1162 +#define WT_STAT_CONN_LOCK_METADATA_WAIT_INTERNAL 1168 +/*! + * lock: read timestamp queue lock application thread time waiting for + * the dhandle lock (usecs) + */ +#define WT_STAT_CONN_LOCK_READ_TIMESTAMP_WAIT_APPLICATION 1169 +/*! + * lock: read timestamp queue lock internal thread time waiting for the + * dhandle lock (usecs) + */ +#define WT_STAT_CONN_LOCK_READ_TIMESTAMP_WAIT_INTERNAL 1170 +/*! lock: read timestamp queue read lock acquisitions */ +#define WT_STAT_CONN_LOCK_READ_TIMESTAMP_READ_COUNT 1171 +/*! lock: read timestamp queue write lock acquisitions */ +#define WT_STAT_CONN_LOCK_READ_TIMESTAMP_WRITE_COUNT 1172 /*! lock: schema lock acquisitions */ -#define WT_STAT_CONN_LOCK_SCHEMA_COUNT 1163 +#define WT_STAT_CONN_LOCK_SCHEMA_COUNT 1173 /*! lock: schema lock application thread wait time (usecs) */ -#define WT_STAT_CONN_LOCK_SCHEMA_WAIT_APPLICATION 1164 +#define WT_STAT_CONN_LOCK_SCHEMA_WAIT_APPLICATION 1174 /*! lock: schema lock internal thread wait time (usecs) */ -#define WT_STAT_CONN_LOCK_SCHEMA_WAIT_INTERNAL 1165 +#define WT_STAT_CONN_LOCK_SCHEMA_WAIT_INTERNAL 1175 /*! * lock: table lock application thread time waiting for the table lock * (usecs) */ -#define WT_STAT_CONN_LOCK_TABLE_WAIT_APPLICATION 1166 +#define WT_STAT_CONN_LOCK_TABLE_WAIT_APPLICATION 1176 /*! * lock: table lock internal thread time waiting for the table lock * (usecs) */ -#define WT_STAT_CONN_LOCK_TABLE_WAIT_INTERNAL 1167 +#define WT_STAT_CONN_LOCK_TABLE_WAIT_INTERNAL 1177 /*! lock: table read lock acquisitions */ -#define WT_STAT_CONN_LOCK_TABLE_READ_COUNT 1168 +#define WT_STAT_CONN_LOCK_TABLE_READ_COUNT 1178 /*! lock: table write lock acquisitions */ -#define WT_STAT_CONN_LOCK_TABLE_WRITE_COUNT 1169 +#define WT_STAT_CONN_LOCK_TABLE_WRITE_COUNT 1179 +/*! + * lock: txn global lock application thread time waiting for the dhandle + * lock (usecs) + */ +#define WT_STAT_CONN_LOCK_TXN_GLOBAL_WAIT_APPLICATION 1180 +/*! + * lock: txn global lock internal thread time waiting for the dhandle + * lock (usecs) + */ +#define WT_STAT_CONN_LOCK_TXN_GLOBAL_WAIT_INTERNAL 1181 +/*! lock: txn global read lock acquisitions */ +#define WT_STAT_CONN_LOCK_TXN_GLOBAL_READ_COUNT 1182 +/*! lock: txn global write lock acquisitions */ +#define WT_STAT_CONN_LOCK_TXN_GLOBAL_WRITE_COUNT 1183 /*! log: busy returns attempting to switch slots */ -#define WT_STAT_CONN_LOG_SLOT_SWITCH_BUSY 1170 +#define WT_STAT_CONN_LOG_SLOT_SWITCH_BUSY 1184 /*! log: force checkpoint calls slept */ -#define WT_STAT_CONN_LOG_FORCE_CKPT_SLEEP 1171 +#define WT_STAT_CONN_LOG_FORCE_CKPT_SLEEP 1185 /*! log: log bytes of payload data */ -#define WT_STAT_CONN_LOG_BYTES_PAYLOAD 1172 +#define WT_STAT_CONN_LOG_BYTES_PAYLOAD 1186 /*! log: log bytes written */ -#define WT_STAT_CONN_LOG_BYTES_WRITTEN 1173 +#define WT_STAT_CONN_LOG_BYTES_WRITTEN 1187 /*! log: log files manually zero-filled */ -#define WT_STAT_CONN_LOG_ZERO_FILLS 1174 +#define WT_STAT_CONN_LOG_ZERO_FILLS 1188 /*! log: log flush operations */ -#define WT_STAT_CONN_LOG_FLUSH 1175 +#define WT_STAT_CONN_LOG_FLUSH 1189 /*! log: log force write operations */ -#define WT_STAT_CONN_LOG_FORCE_WRITE 1176 +#define WT_STAT_CONN_LOG_FORCE_WRITE 1190 /*! log: log force write operations skipped */ -#define WT_STAT_CONN_LOG_FORCE_WRITE_SKIP 1177 +#define WT_STAT_CONN_LOG_FORCE_WRITE_SKIP 1191 /*! log: log records compressed */ -#define WT_STAT_CONN_LOG_COMPRESS_WRITES 1178 +#define WT_STAT_CONN_LOG_COMPRESS_WRITES 1192 /*! log: log records not compressed */ -#define WT_STAT_CONN_LOG_COMPRESS_WRITE_FAILS 1179 +#define WT_STAT_CONN_LOG_COMPRESS_WRITE_FAILS 1193 /*! log: log records too small to compress */ -#define WT_STAT_CONN_LOG_COMPRESS_SMALL 1180 +#define WT_STAT_CONN_LOG_COMPRESS_SMALL 1194 /*! log: log release advances write LSN */ -#define WT_STAT_CONN_LOG_RELEASE_WRITE_LSN 1181 +#define WT_STAT_CONN_LOG_RELEASE_WRITE_LSN 1195 /*! log: log scan operations */ -#define WT_STAT_CONN_LOG_SCANS 1182 +#define WT_STAT_CONN_LOG_SCANS 1196 /*! log: log scan records requiring two reads */ -#define WT_STAT_CONN_LOG_SCAN_REREADS 1183 +#define WT_STAT_CONN_LOG_SCAN_REREADS 1197 /*! log: log server thread advances write LSN */ -#define WT_STAT_CONN_LOG_WRITE_LSN 1184 +#define WT_STAT_CONN_LOG_WRITE_LSN 1198 /*! log: log server thread write LSN walk skipped */ -#define WT_STAT_CONN_LOG_WRITE_LSN_SKIP 1185 +#define WT_STAT_CONN_LOG_WRITE_LSN_SKIP 1199 /*! log: log sync operations */ -#define WT_STAT_CONN_LOG_SYNC 1186 +#define WT_STAT_CONN_LOG_SYNC 1200 /*! log: log sync time duration (usecs) */ -#define WT_STAT_CONN_LOG_SYNC_DURATION 1187 +#define WT_STAT_CONN_LOG_SYNC_DURATION 1201 /*! log: log sync_dir operations */ -#define WT_STAT_CONN_LOG_SYNC_DIR 1188 +#define WT_STAT_CONN_LOG_SYNC_DIR 1202 /*! log: log sync_dir time duration (usecs) */ -#define WT_STAT_CONN_LOG_SYNC_DIR_DURATION 1189 +#define WT_STAT_CONN_LOG_SYNC_DIR_DURATION 1203 /*! log: log write operations */ -#define WT_STAT_CONN_LOG_WRITES 1190 +#define WT_STAT_CONN_LOG_WRITES 1204 /*! log: logging bytes consolidated */ -#define WT_STAT_CONN_LOG_SLOT_CONSOLIDATED 1191 +#define WT_STAT_CONN_LOG_SLOT_CONSOLIDATED 1205 /*! log: maximum log file size */ -#define WT_STAT_CONN_LOG_MAX_FILESIZE 1192 +#define WT_STAT_CONN_LOG_MAX_FILESIZE 1206 /*! log: number of pre-allocated log files to create */ -#define WT_STAT_CONN_LOG_PREALLOC_MAX 1193 +#define WT_STAT_CONN_LOG_PREALLOC_MAX 1207 /*! log: pre-allocated log files not ready and missed */ -#define WT_STAT_CONN_LOG_PREALLOC_MISSED 1194 +#define WT_STAT_CONN_LOG_PREALLOC_MISSED 1208 /*! log: pre-allocated log files prepared */ -#define WT_STAT_CONN_LOG_PREALLOC_FILES 1195 +#define WT_STAT_CONN_LOG_PREALLOC_FILES 1209 /*! log: pre-allocated log files used */ -#define WT_STAT_CONN_LOG_PREALLOC_USED 1196 +#define WT_STAT_CONN_LOG_PREALLOC_USED 1210 /*! log: records processed by log scan */ -#define WT_STAT_CONN_LOG_SCAN_RECORDS 1197 +#define WT_STAT_CONN_LOG_SCAN_RECORDS 1211 /*! log: slot close lost race */ -#define WT_STAT_CONN_LOG_SLOT_CLOSE_RACE 1198 +#define WT_STAT_CONN_LOG_SLOT_CLOSE_RACE 1212 /*! log: slot close unbuffered waits */ -#define WT_STAT_CONN_LOG_SLOT_CLOSE_UNBUF 1199 +#define WT_STAT_CONN_LOG_SLOT_CLOSE_UNBUF 1213 /*! log: slot closures */ -#define WT_STAT_CONN_LOG_SLOT_CLOSES 1200 +#define WT_STAT_CONN_LOG_SLOT_CLOSES 1214 /*! log: slot join atomic update races */ -#define WT_STAT_CONN_LOG_SLOT_RACES 1201 +#define WT_STAT_CONN_LOG_SLOT_RACES 1215 /*! log: slot join calls atomic updates raced */ -#define WT_STAT_CONN_LOG_SLOT_YIELD_RACE 1202 +#define WT_STAT_CONN_LOG_SLOT_YIELD_RACE 1216 /*! log: slot join calls did not yield */ -#define WT_STAT_CONN_LOG_SLOT_IMMEDIATE 1203 +#define WT_STAT_CONN_LOG_SLOT_IMMEDIATE 1217 /*! log: slot join calls found active slot closed */ -#define WT_STAT_CONN_LOG_SLOT_YIELD_CLOSE 1204 +#define WT_STAT_CONN_LOG_SLOT_YIELD_CLOSE 1218 /*! log: slot join calls slept */ -#define WT_STAT_CONN_LOG_SLOT_YIELD_SLEEP 1205 +#define WT_STAT_CONN_LOG_SLOT_YIELD_SLEEP 1219 /*! log: slot join calls yielded */ -#define WT_STAT_CONN_LOG_SLOT_YIELD 1206 +#define WT_STAT_CONN_LOG_SLOT_YIELD 1220 /*! log: slot join found active slot closed */ -#define WT_STAT_CONN_LOG_SLOT_ACTIVE_CLOSED 1207 +#define WT_STAT_CONN_LOG_SLOT_ACTIVE_CLOSED 1221 /*! log: slot joins yield time (usecs) */ -#define WT_STAT_CONN_LOG_SLOT_YIELD_DURATION 1208 +#define WT_STAT_CONN_LOG_SLOT_YIELD_DURATION 1222 /*! log: slot transitions unable to find free slot */ -#define WT_STAT_CONN_LOG_SLOT_NO_FREE_SLOTS 1209 +#define WT_STAT_CONN_LOG_SLOT_NO_FREE_SLOTS 1223 /*! log: slot unbuffered writes */ -#define WT_STAT_CONN_LOG_SLOT_UNBUFFERED 1210 +#define WT_STAT_CONN_LOG_SLOT_UNBUFFERED 1224 /*! log: total in-memory size of compressed records */ -#define WT_STAT_CONN_LOG_COMPRESS_MEM 1211 +#define WT_STAT_CONN_LOG_COMPRESS_MEM 1225 /*! log: total log buffer size */ -#define WT_STAT_CONN_LOG_BUFFER_SIZE 1212 +#define WT_STAT_CONN_LOG_BUFFER_SIZE 1226 /*! log: total size of compressed records */ -#define WT_STAT_CONN_LOG_COMPRESS_LEN 1213 +#define WT_STAT_CONN_LOG_COMPRESS_LEN 1227 /*! log: written slots coalesced */ -#define WT_STAT_CONN_LOG_SLOT_COALESCED 1214 +#define WT_STAT_CONN_LOG_SLOT_COALESCED 1228 /*! log: yields waiting for previous log file close */ -#define WT_STAT_CONN_LOG_CLOSE_YIELDS 1215 +#define WT_STAT_CONN_LOG_CLOSE_YIELDS 1229 +/*! perf: file system read latency histogram (bucket 1) - 10-49ms */ +#define WT_STAT_CONN_PERF_HIST_FSREAD_LATENCY_LT50 1230 +/*! perf: file system read latency histogram (bucket 2) - 50-99ms */ +#define WT_STAT_CONN_PERF_HIST_FSREAD_LATENCY_LT100 1231 +/*! perf: file system read latency histogram (bucket 3) - 100-249ms */ +#define WT_STAT_CONN_PERF_HIST_FSREAD_LATENCY_LT250 1232 +/*! perf: file system read latency histogram (bucket 4) - 250-499ms */ +#define WT_STAT_CONN_PERF_HIST_FSREAD_LATENCY_LT500 1233 +/*! perf: file system read latency histogram (bucket 5) - 500-999ms */ +#define WT_STAT_CONN_PERF_HIST_FSREAD_LATENCY_LT1000 1234 +/*! perf: file system read latency histogram (bucket 6) - 1000ms+ */ +#define WT_STAT_CONN_PERF_HIST_FSREAD_LATENCY_GT1000 1235 +/*! perf: file system write latency histogram (bucket 1) - 10-49ms */ +#define WT_STAT_CONN_PERF_HIST_FSWRITE_LATENCY_LT50 1236 +/*! perf: file system write latency histogram (bucket 2) - 50-99ms */ +#define WT_STAT_CONN_PERF_HIST_FSWRITE_LATENCY_LT100 1237 +/*! perf: file system write latency histogram (bucket 3) - 100-249ms */ +#define WT_STAT_CONN_PERF_HIST_FSWRITE_LATENCY_LT250 1238 +/*! perf: file system write latency histogram (bucket 4) - 250-499ms */ +#define WT_STAT_CONN_PERF_HIST_FSWRITE_LATENCY_LT500 1239 +/*! perf: file system write latency histogram (bucket 5) - 500-999ms */ +#define WT_STAT_CONN_PERF_HIST_FSWRITE_LATENCY_LT1000 1240 +/*! perf: file system write latency histogram (bucket 6) - 1000ms+ */ +#define WT_STAT_CONN_PERF_HIST_FSWRITE_LATENCY_GT1000 1241 +/*! perf: operation read latency histogram (bucket 1) - 100-249us */ +#define WT_STAT_CONN_PERF_HIST_OPREAD_LATENCY_LT250 1242 +/*! perf: operation read latency histogram (bucket 2) - 250-499us */ +#define WT_STAT_CONN_PERF_HIST_OPREAD_LATENCY_LT500 1243 +/*! perf: operation read latency histogram (bucket 3) - 500-999us */ +#define WT_STAT_CONN_PERF_HIST_OPREAD_LATENCY_LT1000 1244 +/*! perf: operation read latency histogram (bucket 4) - 1000-9999us */ +#define WT_STAT_CONN_PERF_HIST_OPREAD_LATENCY_LT10000 1245 +/*! perf: operation read latency histogram (bucket 5) - 10000us+ */ +#define WT_STAT_CONN_PERF_HIST_OPREAD_LATENCY_GT10000 1246 +/*! perf: operation write latency histogram (bucket 1) - 100-249us */ +#define WT_STAT_CONN_PERF_HIST_OPWRITE_LATENCY_LT250 1247 +/*! perf: operation write latency histogram (bucket 2) - 250-499us */ +#define WT_STAT_CONN_PERF_HIST_OPWRITE_LATENCY_LT500 1248 +/*! perf: operation write latency histogram (bucket 3) - 500-999us */ +#define WT_STAT_CONN_PERF_HIST_OPWRITE_LATENCY_LT1000 1249 +/*! perf: operation write latency histogram (bucket 4) - 1000-9999us */ +#define WT_STAT_CONN_PERF_HIST_OPWRITE_LATENCY_LT10000 1250 +/*! perf: operation write latency histogram (bucket 5) - 10000us+ */ +#define WT_STAT_CONN_PERF_HIST_OPWRITE_LATENCY_GT10000 1251 /*! reconciliation: fast-path pages deleted */ -#define WT_STAT_CONN_REC_PAGE_DELETE_FAST 1216 +#define WT_STAT_CONN_REC_PAGE_DELETE_FAST 1252 /*! reconciliation: page reconciliation calls */ -#define WT_STAT_CONN_REC_PAGES 1217 +#define WT_STAT_CONN_REC_PAGES 1253 /*! reconciliation: page reconciliation calls for eviction */ -#define WT_STAT_CONN_REC_PAGES_EVICTION 1218 +#define WT_STAT_CONN_REC_PAGES_EVICTION 1254 /*! reconciliation: pages deleted */ -#define WT_STAT_CONN_REC_PAGE_DELETE 1219 +#define WT_STAT_CONN_REC_PAGE_DELETE 1255 /*! reconciliation: split bytes currently awaiting free */ -#define WT_STAT_CONN_REC_SPLIT_STASHED_BYTES 1220 +#define WT_STAT_CONN_REC_SPLIT_STASHED_BYTES 1256 /*! reconciliation: split objects currently awaiting free */ -#define WT_STAT_CONN_REC_SPLIT_STASHED_OBJECTS 1221 +#define WT_STAT_CONN_REC_SPLIT_STASHED_OBJECTS 1257 /*! session: open cursor count */ -#define WT_STAT_CONN_SESSION_CURSOR_OPEN 1222 +#define WT_STAT_CONN_SESSION_CURSOR_OPEN 1258 /*! session: open session count */ -#define WT_STAT_CONN_SESSION_OPEN 1223 +#define WT_STAT_CONN_SESSION_OPEN 1259 /*! session: table alter failed calls */ -#define WT_STAT_CONN_SESSION_TABLE_ALTER_FAIL 1224 +#define WT_STAT_CONN_SESSION_TABLE_ALTER_FAIL 1260 /*! session: table alter successful calls */ -#define WT_STAT_CONN_SESSION_TABLE_ALTER_SUCCESS 1225 +#define WT_STAT_CONN_SESSION_TABLE_ALTER_SUCCESS 1261 /*! session: table alter unchanged and skipped */ -#define WT_STAT_CONN_SESSION_TABLE_ALTER_SKIP 1226 +#define WT_STAT_CONN_SESSION_TABLE_ALTER_SKIP 1262 /*! session: table compact failed calls */ -#define WT_STAT_CONN_SESSION_TABLE_COMPACT_FAIL 1227 +#define WT_STAT_CONN_SESSION_TABLE_COMPACT_FAIL 1263 /*! session: table compact successful calls */ -#define WT_STAT_CONN_SESSION_TABLE_COMPACT_SUCCESS 1228 +#define WT_STAT_CONN_SESSION_TABLE_COMPACT_SUCCESS 1264 /*! session: table create failed calls */ -#define WT_STAT_CONN_SESSION_TABLE_CREATE_FAIL 1229 +#define WT_STAT_CONN_SESSION_TABLE_CREATE_FAIL 1265 /*! session: table create successful calls */ -#define WT_STAT_CONN_SESSION_TABLE_CREATE_SUCCESS 1230 +#define WT_STAT_CONN_SESSION_TABLE_CREATE_SUCCESS 1266 /*! session: table drop failed calls */ -#define WT_STAT_CONN_SESSION_TABLE_DROP_FAIL 1231 +#define WT_STAT_CONN_SESSION_TABLE_DROP_FAIL 1267 /*! session: table drop successful calls */ -#define WT_STAT_CONN_SESSION_TABLE_DROP_SUCCESS 1232 +#define WT_STAT_CONN_SESSION_TABLE_DROP_SUCCESS 1268 /*! session: table rebalance failed calls */ -#define WT_STAT_CONN_SESSION_TABLE_REBALANCE_FAIL 1233 +#define WT_STAT_CONN_SESSION_TABLE_REBALANCE_FAIL 1269 /*! session: table rebalance successful calls */ -#define WT_STAT_CONN_SESSION_TABLE_REBALANCE_SUCCESS 1234 +#define WT_STAT_CONN_SESSION_TABLE_REBALANCE_SUCCESS 1270 /*! session: table rename failed calls */ -#define WT_STAT_CONN_SESSION_TABLE_RENAME_FAIL 1235 +#define WT_STAT_CONN_SESSION_TABLE_RENAME_FAIL 1271 /*! session: table rename successful calls */ -#define WT_STAT_CONN_SESSION_TABLE_RENAME_SUCCESS 1236 +#define WT_STAT_CONN_SESSION_TABLE_RENAME_SUCCESS 1272 /*! session: table salvage failed calls */ -#define WT_STAT_CONN_SESSION_TABLE_SALVAGE_FAIL 1237 +#define WT_STAT_CONN_SESSION_TABLE_SALVAGE_FAIL 1273 /*! session: table salvage successful calls */ -#define WT_STAT_CONN_SESSION_TABLE_SALVAGE_SUCCESS 1238 +#define WT_STAT_CONN_SESSION_TABLE_SALVAGE_SUCCESS 1274 /*! session: table truncate failed calls */ -#define WT_STAT_CONN_SESSION_TABLE_TRUNCATE_FAIL 1239 +#define WT_STAT_CONN_SESSION_TABLE_TRUNCATE_FAIL 1275 /*! session: table truncate successful calls */ -#define WT_STAT_CONN_SESSION_TABLE_TRUNCATE_SUCCESS 1240 +#define WT_STAT_CONN_SESSION_TABLE_TRUNCATE_SUCCESS 1276 /*! session: table verify failed calls */ -#define WT_STAT_CONN_SESSION_TABLE_VERIFY_FAIL 1241 +#define WT_STAT_CONN_SESSION_TABLE_VERIFY_FAIL 1277 /*! session: table verify successful calls */ -#define WT_STAT_CONN_SESSION_TABLE_VERIFY_SUCCESS 1242 +#define WT_STAT_CONN_SESSION_TABLE_VERIFY_SUCCESS 1278 /*! thread-state: active filesystem fsync calls */ -#define WT_STAT_CONN_THREAD_FSYNC_ACTIVE 1243 +#define WT_STAT_CONN_THREAD_FSYNC_ACTIVE 1279 /*! thread-state: active filesystem read calls */ -#define WT_STAT_CONN_THREAD_READ_ACTIVE 1244 +#define WT_STAT_CONN_THREAD_READ_ACTIVE 1280 /*! thread-state: active filesystem write calls */ -#define WT_STAT_CONN_THREAD_WRITE_ACTIVE 1245 +#define WT_STAT_CONN_THREAD_WRITE_ACTIVE 1281 /*! thread-yield: application thread time evicting (usecs) */ -#define WT_STAT_CONN_APPLICATION_EVICT_TIME 1246 +#define WT_STAT_CONN_APPLICATION_EVICT_TIME 1282 /*! thread-yield: application thread time waiting for cache (usecs) */ -#define WT_STAT_CONN_APPLICATION_CACHE_TIME 1247 +#define WT_STAT_CONN_APPLICATION_CACHE_TIME 1283 /*! * thread-yield: connection close blocked waiting for transaction state * stabilization */ -#define WT_STAT_CONN_TXN_RELEASE_BLOCKED 1248 +#define WT_STAT_CONN_TXN_RELEASE_BLOCKED 1284 /*! thread-yield: connection close yielded for lsm manager shutdown */ -#define WT_STAT_CONN_CONN_CLOSE_BLOCKED_LSM 1249 +#define WT_STAT_CONN_CONN_CLOSE_BLOCKED_LSM 1285 /*! thread-yield: data handle lock yielded */ -#define WT_STAT_CONN_DHANDLE_LOCK_BLOCKED 1250 +#define WT_STAT_CONN_DHANDLE_LOCK_BLOCKED 1286 /*! * thread-yield: get reference for page index and slot time sleeping * (usecs) */ -#define WT_STAT_CONN_PAGE_INDEX_SLOT_REF_BLOCKED 1251 +#define WT_STAT_CONN_PAGE_INDEX_SLOT_REF_BLOCKED 1287 /*! thread-yield: log server sync yielded for log write */ -#define WT_STAT_CONN_LOG_SERVER_SYNC_BLOCKED 1252 +#define WT_STAT_CONN_LOG_SERVER_SYNC_BLOCKED 1288 /*! thread-yield: page acquire busy blocked */ -#define WT_STAT_CONN_PAGE_BUSY_BLOCKED 1253 +#define WT_STAT_CONN_PAGE_BUSY_BLOCKED 1289 /*! thread-yield: page acquire eviction blocked */ -#define WT_STAT_CONN_PAGE_FORCIBLE_EVICT_BLOCKED 1254 +#define WT_STAT_CONN_PAGE_FORCIBLE_EVICT_BLOCKED 1290 /*! thread-yield: page acquire locked blocked */ -#define WT_STAT_CONN_PAGE_LOCKED_BLOCKED 1255 +#define WT_STAT_CONN_PAGE_LOCKED_BLOCKED 1291 /*! thread-yield: page acquire read blocked */ -#define WT_STAT_CONN_PAGE_READ_BLOCKED 1256 +#define WT_STAT_CONN_PAGE_READ_BLOCKED 1292 /*! thread-yield: page acquire time sleeping (usecs) */ -#define WT_STAT_CONN_PAGE_SLEEP 1257 +#define WT_STAT_CONN_PAGE_SLEEP 1293 /*! * thread-yield: page delete rollback time sleeping for state change * (usecs) */ -#define WT_STAT_CONN_PAGE_DEL_ROLLBACK_BLOCKED 1258 +#define WT_STAT_CONN_PAGE_DEL_ROLLBACK_BLOCKED 1294 /*! thread-yield: page reconciliation yielded due to child modification */ -#define WT_STAT_CONN_CHILD_MODIFY_BLOCKED_PAGE 1259 +#define WT_STAT_CONN_CHILD_MODIFY_BLOCKED_PAGE 1295 /*! * thread-yield: tree descend one level yielded for split page index * update */ -#define WT_STAT_CONN_TREE_DESCEND_BLOCKED 1260 +#define WT_STAT_CONN_TREE_DESCEND_BLOCKED 1296 +/*! transaction: commit timestamp queue insert to empty */ +#define WT_STAT_CONN_TXN_COMMIT_QUEUE_EMPTY 1297 +/*! transaction: commit timestamp queue inserts to head */ +#define WT_STAT_CONN_TXN_COMMIT_QUEUE_HEAD 1298 +/*! transaction: commit timestamp queue inserts total */ +#define WT_STAT_CONN_TXN_COMMIT_QUEUE_INSERTS 1299 +/*! transaction: commit timestamp queue length */ +#define WT_STAT_CONN_TXN_COMMIT_QUEUE_LEN 1300 /*! transaction: number of named snapshots created */ -#define WT_STAT_CONN_TXN_SNAPSHOTS_CREATED 1261 +#define WT_STAT_CONN_TXN_SNAPSHOTS_CREATED 1301 /*! transaction: number of named snapshots dropped */ -#define WT_STAT_CONN_TXN_SNAPSHOTS_DROPPED 1262 +#define WT_STAT_CONN_TXN_SNAPSHOTS_DROPPED 1302 +/*! transaction: query timestamp calls */ +#define WT_STAT_CONN_TXN_QUERY_TS 1303 +/*! transaction: read timestamp queue insert to empty */ +#define WT_STAT_CONN_TXN_READ_QUEUE_EMPTY 1304 +/*! transaction: read timestamp queue inserts to head */ +#define WT_STAT_CONN_TXN_READ_QUEUE_HEAD 1305 +/*! transaction: read timestamp queue inserts total */ +#define WT_STAT_CONN_TXN_READ_QUEUE_INSERTS 1306 +/*! transaction: read timestamp queue length */ +#define WT_STAT_CONN_TXN_READ_QUEUE_LEN 1307 +/*! transaction: set timestamp calls */ +#define WT_STAT_CONN_TXN_SET_TS 1308 +/*! transaction: set timestamp commit calls */ +#define WT_STAT_CONN_TXN_SET_TS_COMMIT 1309 +/*! transaction: set timestamp commit updates */ +#define WT_STAT_CONN_TXN_SET_TS_COMMIT_UPD 1310 +/*! transaction: set timestamp oldest calls */ +#define WT_STAT_CONN_TXN_SET_TS_OLDEST 1311 +/*! transaction: set timestamp oldest updates */ +#define WT_STAT_CONN_TXN_SET_TS_OLDEST_UPD 1312 +/*! transaction: set timestamp stable calls */ +#define WT_STAT_CONN_TXN_SET_TS_STABLE 1313 +/*! transaction: set timestamp stable updates */ +#define WT_STAT_CONN_TXN_SET_TS_STABLE_UPD 1314 /*! transaction: transaction begins */ -#define WT_STAT_CONN_TXN_BEGIN 1263 +#define WT_STAT_CONN_TXN_BEGIN 1315 /*! transaction: transaction checkpoint currently running */ -#define WT_STAT_CONN_TXN_CHECKPOINT_RUNNING 1264 +#define WT_STAT_CONN_TXN_CHECKPOINT_RUNNING 1316 /*! transaction: transaction checkpoint generation */ -#define WT_STAT_CONN_TXN_CHECKPOINT_GENERATION 1265 +#define WT_STAT_CONN_TXN_CHECKPOINT_GENERATION 1317 /*! transaction: transaction checkpoint max time (msecs) */ -#define WT_STAT_CONN_TXN_CHECKPOINT_TIME_MAX 1266 +#define WT_STAT_CONN_TXN_CHECKPOINT_TIME_MAX 1318 /*! transaction: transaction checkpoint min time (msecs) */ -#define WT_STAT_CONN_TXN_CHECKPOINT_TIME_MIN 1267 +#define WT_STAT_CONN_TXN_CHECKPOINT_TIME_MIN 1319 /*! transaction: transaction checkpoint most recent time (msecs) */ -#define WT_STAT_CONN_TXN_CHECKPOINT_TIME_RECENT 1268 +#define WT_STAT_CONN_TXN_CHECKPOINT_TIME_RECENT 1320 /*! transaction: transaction checkpoint scrub dirty target */ -#define WT_STAT_CONN_TXN_CHECKPOINT_SCRUB_TARGET 1269 +#define WT_STAT_CONN_TXN_CHECKPOINT_SCRUB_TARGET 1321 /*! transaction: transaction checkpoint scrub time (msecs) */ -#define WT_STAT_CONN_TXN_CHECKPOINT_SCRUB_TIME 1270 +#define WT_STAT_CONN_TXN_CHECKPOINT_SCRUB_TIME 1322 /*! transaction: transaction checkpoint total time (msecs) */ -#define WT_STAT_CONN_TXN_CHECKPOINT_TIME_TOTAL 1271 +#define WT_STAT_CONN_TXN_CHECKPOINT_TIME_TOTAL 1323 /*! transaction: transaction checkpoints */ -#define WT_STAT_CONN_TXN_CHECKPOINT 1272 +#define WT_STAT_CONN_TXN_CHECKPOINT 1324 /*! * transaction: transaction checkpoints skipped because database was * clean */ -#define WT_STAT_CONN_TXN_CHECKPOINT_SKIPPED 1273 +#define WT_STAT_CONN_TXN_CHECKPOINT_SKIPPED 1325 /*! transaction: transaction failures due to cache overflow */ -#define WT_STAT_CONN_TXN_FAIL_CACHE 1274 +#define WT_STAT_CONN_TXN_FAIL_CACHE 1326 /*! * transaction: transaction fsync calls for checkpoint after allocating * the transaction ID */ -#define WT_STAT_CONN_TXN_CHECKPOINT_FSYNC_POST 1275 +#define WT_STAT_CONN_TXN_CHECKPOINT_FSYNC_POST 1327 /*! * transaction: transaction fsync duration for checkpoint after * allocating the transaction ID (usecs) */ -#define WT_STAT_CONN_TXN_CHECKPOINT_FSYNC_POST_DURATION 1276 +#define WT_STAT_CONN_TXN_CHECKPOINT_FSYNC_POST_DURATION 1328 /*! transaction: transaction range of IDs currently pinned */ -#define WT_STAT_CONN_TXN_PINNED_RANGE 1277 +#define WT_STAT_CONN_TXN_PINNED_RANGE 1329 /*! transaction: transaction range of IDs currently pinned by a checkpoint */ -#define WT_STAT_CONN_TXN_PINNED_CHECKPOINT_RANGE 1278 +#define WT_STAT_CONN_TXN_PINNED_CHECKPOINT_RANGE 1330 /*! * transaction: transaction range of IDs currently pinned by named * snapshots */ -#define WT_STAT_CONN_TXN_PINNED_SNAPSHOT_RANGE 1279 +#define WT_STAT_CONN_TXN_PINNED_SNAPSHOT_RANGE 1331 /*! transaction: transaction range of timestamps currently pinned */ -#define WT_STAT_CONN_TXN_PINNED_TIMESTAMP 1280 +#define WT_STAT_CONN_TXN_PINNED_TIMESTAMP 1332 /*! * transaction: transaction range of timestamps pinned by the oldest * timestamp */ -#define WT_STAT_CONN_TXN_PINNED_TIMESTAMP_OLDEST 1281 +#define WT_STAT_CONN_TXN_PINNED_TIMESTAMP_OLDEST 1333 /*! transaction: transaction sync calls */ -#define WT_STAT_CONN_TXN_SYNC 1282 -/*! transaction: transactions commit timestamp queue inserts to head */ -#define WT_STAT_CONN_TXN_COMMIT_QUEUE_HEAD 1283 -/*! transaction: transactions commit timestamp queue inserts total */ -#define WT_STAT_CONN_TXN_COMMIT_QUEUE_INSERTS 1284 -/*! transaction: transactions commit timestamp queue length */ -#define WT_STAT_CONN_TXN_COMMIT_QUEUE_LEN 1285 +#define WT_STAT_CONN_TXN_SYNC 1334 /*! transaction: transactions committed */ -#define WT_STAT_CONN_TXN_COMMIT 1286 -/*! transaction: transactions read timestamp queue inserts to head */ -#define WT_STAT_CONN_TXN_READ_QUEUE_HEAD 1287 -/*! transaction: transactions read timestamp queue inserts total */ -#define WT_STAT_CONN_TXN_READ_QUEUE_INSERTS 1288 -/*! transaction: transactions read timestamp queue length */ -#define WT_STAT_CONN_TXN_READ_QUEUE_LEN 1289 +#define WT_STAT_CONN_TXN_COMMIT 1335 /*! transaction: transactions rolled back */ -#define WT_STAT_CONN_TXN_ROLLBACK 1290 +#define WT_STAT_CONN_TXN_ROLLBACK 1336 /*! transaction: update conflicts */ -#define WT_STAT_CONN_TXN_UPDATE_CONFLICT 1291 +#define WT_STAT_CONN_TXN_UPDATE_CONFLICT 1337 /*! * @} diff --git a/src/third_party/wiredtiger/src/include/wiredtiger_ext.h b/src/third_party/wiredtiger/src/include/wiredtiger_ext.h index bc61c43e29d..f33402129ff 100644 --- a/src/third_party/wiredtiger/src/include/wiredtiger_ext.h +++ b/src/third_party/wiredtiger/src/include/wiredtiger_ext.h @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/include/wt_internal.h b/src/third_party/wiredtiger/src/include/wt_internal.h index b25ed08e30f..df3ee2da174 100644 --- a/src/third_party/wiredtiger/src/include/wt_internal.h +++ b/src/third_party/wiredtiger/src/include/wt_internal.h @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -35,6 +35,7 @@ extern "C" { #endif #include <errno.h> #include <fcntl.h> +#include <float.h> #include <inttypes.h> #ifdef _WIN32 #include <io.h> @@ -246,6 +247,10 @@ struct __wt_named_extractor; typedef struct __wt_named_extractor WT_NAMED_EXTRACTOR; struct __wt_named_snapshot; typedef struct __wt_named_snapshot WT_NAMED_SNAPSHOT; +struct __wt_optrack_header; + typedef struct __wt_optrack_header WT_OPTRACK_HEADER; +struct __wt_optrack_record; + typedef struct __wt_optrack_record WT_OPTRACK_RECORD; struct __wt_ovfl_reuse; typedef struct __wt_ovfl_reuse WT_OVFL_REUSE; struct __wt_ovfl_track; @@ -352,10 +357,10 @@ union __wt_rand_state; #include "cursor.h" #include "dlh.h" #include "error.h" -#include "flags.h" #include "log.h" #include "lsm.h" #include "meta.h" +#include "optrack.h" #include "os.h" #include "schema.h" #include "thread_group.h" @@ -374,6 +379,7 @@ union __wt_rand_state; #include "ctype.i" /* required by packing.i */ #include "intpack.i" /* required by cell.i, packing.i */ +#include "misc.i" /* required by mutex.i */ #include "buf.i" /* required by cell.i */ #include "cache.i" /* required by txn.i */ @@ -387,7 +393,6 @@ union __wt_rand_state; #include "column.i" #include "cursor.i" #include "log.i" -#include "misc.i" #include "os_fhandle.i" #include "os_fs.i" #include "os_fstream.i" diff --git a/src/third_party/wiredtiger/src/log/log.c b/src/third_party/wiredtiger/src/log/log.c index 17c77f532bb..167297c5c80 100644 --- a/src/third_party/wiredtiger/src/log/log.c +++ b/src/third_party/wiredtiger/src/log/log.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -16,21 +16,22 @@ static int __log_write_internal( #define WT_LOG_COMPRESS_SKIP (offsetof(WT_LOG_RECORD, record)) #define WT_LOG_ENCRYPT_SKIP (offsetof(WT_LOG_RECORD, record)) -/* Flags to __log_openfile */ -#define WT_LOG_OPEN_CREATE_OK 0x01 +/* AUTOMATIC FLAG VALUE GENERATION START */ +#define WT_LOG_OPEN_CREATE_OK 0x1u /* Flag to __log_openfile() */ +/* AUTOMATIC FLAG VALUE GENERATION STOP */ /* * __wt_log_printf -- * Write a text message to the log. */ int -__wt_log_printf(WT_SESSION_IMPL *session, const char *fmt, ...) +__wt_log_printf(WT_SESSION_IMPL *session, const char *format, ...) { WT_DECL_RET; va_list ap; - va_start(ap, fmt); - ret = __wt_log_vprintf(session, fmt, ap); + va_start(ap, format); + ret = __wt_log_vprintf(session, format, ap); va_end(ap); return (ret); } @@ -80,6 +81,28 @@ __log_get_files(WT_SESSION_IMPL *session, } /* + * __log_get_files_single -- + * Retrieve a single log-related file of the given prefix type. + */ +static int +__log_get_files_single(WT_SESSION_IMPL *session, + const char *file_prefix, char ***filesp, u_int *countp) +{ + WT_CONNECTION_IMPL *conn; + const char *log_path; + + *countp = 0; + *filesp = NULL; + + conn = S2C(session); + log_path = conn->log_path; + if (log_path == NULL) + log_path = ""; + return (__wt_fs_directory_list_single( + session, log_path, file_prefix, filesp, countp)); +} + +/* * __log_prealloc_remove -- * Remove all previously created pre-allocated files. */ @@ -258,11 +281,10 @@ __wt_log_background(WT_SESSION_IMPL *session, WT_LSN *lsn) int __wt_log_force_sync(WT_SESSION_IMPL *session, WT_LSN *min_lsn) { - struct timespec fsync_start, fsync_stop; WT_DECL_RET; WT_FH *log_fh; WT_LOG *log; - uint64_t fsync_duration_usecs; + uint64_t fsync_duration_usecs, time_start, time_stop; log = S2C(session)->log; log_fh = NULL; @@ -289,10 +311,10 @@ __wt_log_force_sync(WT_SESSION_IMPL *session, WT_LSN *min_lsn) "log_force_sync: sync directory %s to LSN %" PRIu32 "/%" PRIu32, log->log_dir_fh->name, min_lsn->l.file, min_lsn->l.offset); - __wt_epoch(session, &fsync_start); + time_start = __wt_clock(session); WT_ERR(__wt_fsync(session, log->log_dir_fh, true)); - __wt_epoch(session, &fsync_stop); - fsync_duration_usecs = WT_TIMEDIFF_US(fsync_stop, fsync_start); + time_stop = __wt_clock(session); + fsync_duration_usecs = WT_CLOCKDIFF_US(time_stop, time_start); log->sync_dir_lsn = *min_lsn; WT_STAT_CONN_INCR(session, log_sync_dir); WT_STAT_CONN_INCRV(session, @@ -312,10 +334,10 @@ __wt_log_force_sync(WT_SESSION_IMPL *session, WT_LSN *min_lsn) __wt_verbose(session, WT_VERB_LOG, "log_force_sync: sync %s to LSN %" PRIu32 "/%" PRIu32, log_fh->name, min_lsn->l.file, min_lsn->l.offset); - __wt_epoch(session, &fsync_start); + time_start = __wt_clock(session); WT_ERR(__wt_fsync(session, log_fh, true)); - __wt_epoch(session, &fsync_stop); - fsync_duration_usecs = WT_TIMEDIFF_US(fsync_stop, fsync_start); + time_stop = __wt_clock(session); + fsync_duration_usecs = WT_CLOCKDIFF_US(time_stop, time_start); log->sync_lsn = *min_lsn; WT_STAT_CONN_INCR(session, log_sync); WT_STAT_CONN_INCRV(session, @@ -484,18 +506,8 @@ static int __log_filename(WT_SESSION_IMPL *session, uint32_t id, const char *file_prefix, WT_ITEM *buf) { - const char *log_path; - - log_path = S2C(session)->log_path; - - if (log_path != NULL && log_path[0] != '\0') - WT_RET(__wt_buf_fmt(session, buf, "%s/%s.%010" PRIu32, - log_path, file_prefix, id)); - else - WT_RET(__wt_buf_fmt(session, buf, "%s.%010" PRIu32, - file_prefix, id)); - - return (0); + return (__wt_filename_construct(session, + S2C(session)->log_path, file_prefix, UINTMAX_MAX, id, buf)); } /* @@ -1012,6 +1024,7 @@ err: __wt_scr_free(session, &buf); static int __log_alloc_prealloc(WT_SESSION_IMPL *session, uint32_t to_num) { + WT_CONNECTION_IMPL *conn; WT_DECL_ITEM(from_path); WT_DECL_ITEM(to_path); WT_DECL_RET; @@ -1023,15 +1036,15 @@ __log_alloc_prealloc(WT_SESSION_IMPL *session, uint32_t to_num) /* * If there are no pre-allocated files, return WT_NOTFOUND. */ - log = S2C(session)->log; + conn = S2C(session); + log = conn->log; logfiles = NULL; - WT_ERR(__log_get_files(session, WT_LOG_PREPNAME, &logfiles, &logcount)); + WT_ERR(__log_get_files_single( + session, WT_LOG_PREPNAME, &logfiles, &logcount)); if (logcount == 0) return (WT_NOTFOUND); - /* - * We have a file to use. Just use the first one. - */ + /* We have a file to use. */ WT_ERR(__wt_log_extract_lognum(session, logfiles[0], &from_num)); WT_ERR(__wt_scr_alloc(session, 0, &from_path)); @@ -1137,7 +1150,14 @@ __log_newfile(WT_SESSION_IMPL *session, bool conn_open, bool *created) * If we need to create the log file, do so now. */ if (create_log) { - log->prep_missed++; + /* + * Increment the missed pre-allocated file counter only + * if a hot backup is not in progress. We are deliberately + * not using pre-allocated log files during backup + * (see comment above). + */ + if (!conn->hot_backup) + log->prep_missed++; WT_RET(__wt_log_allocfile( session, log->fileid, WT_LOG_FILENAME)); } @@ -1709,12 +1729,11 @@ err: __wt_free(session, buf); int __wt_log_release(WT_SESSION_IMPL *session, WT_LOGSLOT *slot, bool *freep) { - struct timespec fsync_start, fsync_stop; WT_CONNECTION_IMPL *conn; WT_DECL_RET; WT_LOG *log; WT_LSN sync_lsn; - uint64_t fsync_duration_usecs; + uint64_t fsync_duration_usecs, time_start, time_stop; int64_t release_buffered, release_bytes; bool locked; @@ -1825,11 +1844,11 @@ __wt_log_release(WT_SESSION_IMPL *session, WT_LOGSLOT *slot, bool *freep) "/%" PRIu32, log->log_dir_fh->name, sync_lsn.l.file, sync_lsn.l.offset); - __wt_epoch(session, &fsync_start); + time_start = __wt_clock(session); WT_ERR(__wt_fsync(session, log->log_dir_fh, true)); - __wt_epoch(session, &fsync_stop); + time_stop = __wt_clock(session); fsync_duration_usecs = - WT_TIMEDIFF_US(fsync_stop, fsync_start); + WT_CLOCKDIFF_US(time_stop, time_start); log->sync_dir_lsn = sync_lsn; WT_STAT_CONN_INCR(session, log_sync_dir); WT_STAT_CONN_INCRV(session, @@ -1847,11 +1866,11 @@ __wt_log_release(WT_SESSION_IMPL *session, WT_LOGSLOT *slot, bool *freep) log->log_fh->name, sync_lsn.l.file, sync_lsn.l.offset); WT_STAT_CONN_INCR(session, log_sync); - __wt_epoch(session, &fsync_start); + time_start = __wt_clock(session); WT_ERR(__wt_fsync(session, log->log_fh, true)); - __wt_epoch(session, &fsync_stop); + time_stop = __wt_clock(session); fsync_duration_usecs = - WT_TIMEDIFF_US(fsync_stop, fsync_start); + WT_CLOCKDIFF_US(time_stop, time_start); WT_STAT_CONN_INCRV(session, log_sync_duration, fsync_duration_usecs); log->sync_lsn = sync_lsn; diff --git a/src/third_party/wiredtiger/src/log/log_slot.c b/src/third_party/wiredtiger/src/log/log_slot.c index 61dfb82083d..fc8181e2460 100644 --- a/src/third_party/wiredtiger/src/log/log_slot.c +++ b/src/third_party/wiredtiger/src/log/log_slot.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -100,7 +100,7 @@ __log_slot_close( WT_LOG *log; int64_t end_offset, new_state, old_state; #ifdef HAVE_DIAGNOSTIC - struct timespec begin, now; + uint64_t time_start, time_stop; int count; #endif @@ -133,7 +133,8 @@ retry: /* * If someone completely processed this slot, we're done. */ - if (FLD64_ISSET((uint64_t)slot->slot_state, WT_LOG_SLOT_RESERVED)) { + if (FLD_LOG_SLOT_ISSET( + (uint64_t)slot->slot_state, WT_LOG_SLOT_RESERVED)) { WT_STAT_CONN_INCR(session, log_slot_close_race); return (WT_NOTFOUND); } @@ -160,7 +161,7 @@ retry: */ #ifdef HAVE_DIAGNOSTIC count = 0; - __wt_epoch(session, &begin); + time_start = __wt_clock(session); #endif if (WT_LOG_SLOT_UNBUFFERED_ISSET(old_state)) { while (slot->slot_unbuffered == 0) { @@ -169,8 +170,9 @@ retry: #ifdef HAVE_DIAGNOSTIC ++count; if (count > WT_MILLION) { - __wt_epoch(session, &now); - if (WT_TIMEDIFF_SEC(now, begin) > 10) { + time_stop = __wt_clock(session); + if (WT_CLOCKDIFF_SEC( + time_stop, time_start) > 10) { __wt_errx(session, "SLOT_CLOSE: Slot %" PRIu32 " Timeout unbuffered, state 0x%" PRIx64 " unbuffered %" PRId64, @@ -211,7 +213,7 @@ __log_slot_new(WT_SESSION_IMPL *session) WT_LOGSLOT *slot; int32_t i, pool_i; #ifdef HAVE_DIAGNOSTIC - struct timespec begin, now; + uint64_t time_start, time_stop; int count; #endif @@ -229,7 +231,7 @@ __log_slot_new(WT_SESSION_IMPL *session) #ifdef HAVE_DIAGNOSTIC count = 0; - __wt_epoch(session, &begin); + time_start = __wt_clock(session); #endif /* * Keep trying until we can find a free slot. @@ -269,8 +271,8 @@ __log_slot_new(WT_SESSION_IMPL *session) #ifdef HAVE_DIAGNOSTIC ++count; if (count > WT_MILLION) { - __wt_epoch(session, &now); - if (WT_TIMEDIFF_SEC(now, begin) > 10) { + time_stop = __wt_clock(session); + if (WT_CLOCKDIFF_SEC(time_stop, time_start) > 10) { __wt_errx(session, "SLOT_NEW: Timeout free slot"); __log_slot_dump(session); @@ -482,7 +484,7 @@ __wt_log_slot_destroy(WT_SESSION_IMPL *session) */ for (i = 0; i < WT_SLOT_POOL; i++) { slot = &log->slot_pool[i]; - if (!FLD64_ISSET( + if (!FLD_LOG_SLOT_ISSET( (uint64_t)slot->slot_state, WT_LOG_SLOT_RESERVED)) { rel = WT_LOG_SLOT_RELEASED_BUFFERED(slot->slot_state); if (rel != 0) @@ -503,17 +505,17 @@ void __wt_log_slot_join(WT_SESSION_IMPL *session, uint64_t mysize, uint32_t flags, WT_MYSLOT *myslot) { - struct timespec start, stop; WT_CONNECTION_IMPL *conn; WT_LOG *log; WT_LOGSLOT *slot; - uint64_t usecs; + uint64_t time_start, time_stop, usecs; int64_t flag_state, new_state, old_state, released; int32_t join_offset, new_join, wait_cnt; bool closed, diag_yield, raced, slept, unbuffered, yielded; conn = S2C(session); log = conn->log; + time_start = time_stop = 0; WT_ASSERT(session, !F_ISSET(session, WT_SESSION_LOCKED_SLOT)); WT_ASSERT(session, mysize != 0); @@ -575,7 +577,7 @@ __wt_log_slot_join(WT_SESSION_IMPL *session, uint64_t mysize, ++wait_cnt; } if (!yielded) - __wt_epoch(session, &start); + time_start = __wt_clock(session); yielded = true; /* * The slot is no longer open or we lost the race to @@ -596,8 +598,8 @@ __wt_log_slot_join(WT_SESSION_IMPL *session, uint64_t mysize, WT_STAT_CONN_INCR(session, log_slot_immediate); else { WT_STAT_CONN_INCR(session, log_slot_yield); - __wt_epoch(session, &stop); - usecs = WT_TIMEDIFF_US(stop, start); + time_stop = __wt_clock(session); + usecs = WT_CLOCKDIFF_US(time_stop, time_start); WT_STAT_CONN_INCRV(session, log_slot_yield_duration, usecs); if (closed) WT_STAT_CONN_INCR(session, log_slot_yield_close); diff --git a/src/third_party/wiredtiger/src/log/log_sys.c b/src/third_party/wiredtiger/src/log/log_sys.c index ad65aaf9a8a..3a019c68acc 100644 --- a/src/third_party/wiredtiger/src/log/log_sys.c +++ b/src/third_party/wiredtiger/src/log/log_sys.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/lsm/lsm_cursor.c b/src/third_party/wiredtiger/src/lsm/lsm_cursor.c index e3b2b50ad28..c5a3935813a 100644 --- a/src/third_party/wiredtiger/src/lsm/lsm_cursor.c +++ b/src/third_party/wiredtiger/src/lsm/lsm_cursor.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -557,13 +557,19 @@ retry: if (F_ISSET(clsm, WT_CLSM_MERGE)) { if (strcmp(cursor->uri, chunk->uri) != 0) break; - /* Make sure the checkpoint config matches. */ - checkpoint = ((WT_CURSOR_BTREE *)cursor)-> - btree->dhandle->checkpoint; - if (checkpoint == NULL && - F_ISSET(chunk, WT_LSM_CHUNK_ONDISK) && - !chunk->empty) - break; + /* + * Make sure the checkpoint config matches when not + * using a custom data source. + */ + if (lsm_tree->custom_generation == 0 || + chunk->generation < lsm_tree->custom_generation) { + checkpoint = ((WT_CURSOR_BTREE *)cursor)-> + btree->dhandle->checkpoint; + if (checkpoint == NULL && + F_ISSET(chunk, WT_LSM_CHUNK_ONDISK) && + !chunk->empty) + break; + } /* Make sure the Bloom config matches. */ if (clsm->chunks[ngood]->bloom == NULL && @@ -721,13 +727,19 @@ err: WT_ASSERT( session, strcmp(cursor->uri, chunk->uri) == 0); - /* Make sure the checkpoint config matches. */ - checkpoint = ((WT_CURSOR_BTREE *)cursor)-> - btree->dhandle->checkpoint; - WT_ASSERT(session, - (F_ISSET(chunk, WT_LSM_CHUNK_ONDISK) && - !chunk->empty) ? - checkpoint != NULL : checkpoint == NULL); + /* + * Make sure the checkpoint config matches when not + * using a custom data source. + */ + if (lsm_tree->custom_generation == 0 || + chunk->generation < lsm_tree->custom_generation) { + checkpoint = ((WT_CURSOR_BTREE *)cursor)-> + btree->dhandle->checkpoint; + WT_ASSERT(session, + (F_ISSET(chunk, WT_LSM_CHUNK_ONDISK) && + !chunk->empty) ? + checkpoint != NULL : checkpoint == NULL); + } /* Make sure the Bloom config matches. */ WT_ASSERT(session, @@ -1826,7 +1838,8 @@ __wt_clsm_open(WT_SESSION_IMPL *session, WT_ERR(__wt_clsm_open_bulk(clsm, cfg)); if (0) { -err: if (clsm != NULL) +err: + if (clsm != NULL) WT_TRET(__wt_clsm_close(cursor)); else if (lsm_tree != NULL) __wt_lsm_tree_release(session, lsm_tree); diff --git a/src/third_party/wiredtiger/src/lsm/lsm_cursor_bulk.c b/src/third_party/wiredtiger/src/lsm/lsm_cursor_bulk.c index 354e1626153..1dbf268577c 100644 --- a/src/third_party/wiredtiger/src/lsm/lsm_cursor_bulk.c +++ b/src/third_party/wiredtiger/src/lsm/lsm_cursor_bulk.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/lsm/lsm_manager.c b/src/third_party/wiredtiger/src/lsm/lsm_manager.c index 62f2c7795be..40ff5fc0b26 100644 --- a/src/third_party/wiredtiger/src/lsm/lsm_manager.c +++ b/src/third_party/wiredtiger/src/lsm/lsm_manager.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/lsm/lsm_merge.c b/src/third_party/wiredtiger/src/lsm/lsm_merge.c index 7a20686fb97..1a2568a0d64 100644 --- a/src/third_party/wiredtiger/src/lsm/lsm_merge.c +++ b/src/third_party/wiredtiger/src/lsm/lsm_merge.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -137,6 +137,25 @@ __lsm_merge_aggressive_update(WT_SESSION_IMPL *session, WT_LSM_TREE *lsm_tree) } /* + * __lsm_merge_clear -- + * Clear merge flag on chunks that was set during __lsm_merge_span. + */ +static void +__lsm_merge_clear(WT_SESSION_IMPL *session, WT_LSM_TREE *lsm_tree, + u_int start, u_int nchunks) +{ + WT_LSM_CHUNK *chunk; + u_int i; + + for (i = 0; i < nchunks; i++) { + chunk = lsm_tree->chunk[start + i]; + WT_ASSERT(session, + F_ISSET(chunk, WT_LSM_CHUNK_MERGING)); + F_CLR(chunk, WT_LSM_CHUNK_MERGING); + } +} + +/* * __lsm_merge_span -- * Figure out the best span of chunks to merge. Return an error if * there is no need to do any merges. Called with the LSM tree @@ -146,12 +165,14 @@ static int __lsm_merge_span(WT_SESSION_IMPL *session, WT_LSM_TREE *lsm_tree, u_int id, u_int *start, u_int *end, uint64_t *records) { - WT_LSM_CHUNK *chunk, *previous, *youngest; + WT_LSM_CHUNK *chunk, *youngest; uint64_t chunk_size, record_count; - uint32_t aggressive, max_gap, max_gen, max_level; - u_int end_chunk, i, merge_max, merge_min, nchunks, start_chunk; + uint32_t aggressive, max_gap, max_level; + u_int end_chunk, merge_max, merge_min, nchunks, start_chunk; u_int oldest_gen, youngest_gen; - +#ifdef HAVE_DIAGNOSTIC + u_int i; +#endif /* Clear the return parameters */ *start = *end = 0; *records = 0; @@ -235,6 +256,14 @@ retry_find: break; /* + * If we have enough chunks for a merge and the next chunk is + * in too high a generation, stop. + */ + if (nchunks >= merge_min && + chunk->generation > youngest_gen + max_gap) + break; + + /* * If the size of the chunks selected so far exceeds the * configured maximum chunk size, stop. Keep going if we can * slide the window further into the tree: we don't want to @@ -255,21 +284,10 @@ retry_find: if (oldest_gen - youngest_gen > max_gap) break; - /* - * If we have enough chunks for a merge and the next chunk is - * in too high a generation, stop. - */ - if (nchunks >= merge_min) { - previous = lsm_tree->chunk[start_chunk]; - max_gen = youngest->generation + max_gap; - if (previous->generation <= max_gen && - chunk->generation > max_gen) - break; - } - F_SET(chunk, WT_LSM_CHUNK_MERGING); record_count += chunk->count; --start_chunk; + ++nchunks; /* * If the merge would be too big, or we have a full window @@ -280,12 +298,15 @@ retry_find: (nchunks == merge_max && start_chunk > 0 && chunk->generation == lsm_tree->chunk[start_chunk - 1]->generation)) { - WT_ASSERT(session, - F_ISSET(youngest, WT_LSM_CHUNK_MERGING)); - F_CLR(youngest, WT_LSM_CHUNK_MERGING); - record_count -= youngest->count; - chunk_size -= youngest->size; + /* + * Try again with smaller range. Unfortunately all the + * intermediate state will be reset. Since there's no + * easy way to restore youngest_gen and oldest_gen. + */ + __lsm_merge_clear( + session, lsm_tree, start_chunk, nchunks); --end_chunk; + goto retry_find; } else if (nchunks == merge_max) /* We've found the best full merge we can */ break; @@ -308,12 +329,7 @@ retry_find: * Don't do merges that are too small or across too many generations. */ if (nchunks < merge_min || oldest_gen - youngest_gen > max_gap) { - for (i = 0; i < nchunks; i++) { - chunk = lsm_tree->chunk[start_chunk + i]; - WT_ASSERT(session, - F_ISSET(chunk, WT_LSM_CHUNK_MERGING)); - F_CLR(chunk, WT_LSM_CHUNK_MERGING); - } + __lsm_merge_clear(session, lsm_tree, start_chunk, nchunks); /* * If we didn't find a merge with appropriate gaps, try again * with a smaller range. @@ -417,6 +433,7 @@ __wt_lsm_merge(WT_SESSION_IMPL *session, WT_LSM_TREE *lsm_tree, u_int id) WT_ERR(__wt_calloc_one(session, &chunk)); created_chunk = true; chunk->id = dest_id; + chunk->generation = generation; if (FLD_ISSET(lsm_tree->bloom, WT_LSM_BLOOM_MERGED) && (FLD_ISSET(lsm_tree->bloom, WT_LSM_BLOOM_OLDEST) || @@ -453,6 +470,20 @@ __wt_lsm_merge(WT_SESSION_IMPL *session, WT_LSM_TREE *lsm_tree, u_int id) cfg[2] = NULL; WT_ERR(__wt_open_cursor(session, chunk->uri, NULL, cfg, &dest)); + if (lsm_tree->custom_generation != 0 && + chunk->generation >= lsm_tree->custom_generation) { + WT_DATA_SOURCE *dsrc = + __wt_schema_get_source(session, chunk->uri); + + if (dsrc != NULL && dsrc->lsm_pre_merge != NULL) { + /* Call the callback. */ + WT_ERR(dsrc->lsm_pre_merge(dsrc, src, dest)); + + /* Make sure the source is ready to start the scan. */ + WT_ERR(src->reset(src)); + } + } + #define LSM_MERGE_CHECK_INTERVAL WT_THOUSAND for (insert_count = 0; (ret = src->next(src)) == 0; insert_count++) { if (insert_count % LSM_MERGE_CHECK_INTERVAL == 0) { @@ -541,7 +572,7 @@ __wt_lsm_merge(WT_SESSION_IMPL *session, WT_LSM_TREE *lsm_tree, u_int id) in_sync = false; WT_ERR_NOTFOUND_OK(ret); - WT_ERR(__wt_lsm_tree_set_chunk_size(session, chunk)); + WT_ERR(__wt_lsm_tree_set_chunk_size(session, lsm_tree, chunk)); __wt_lsm_tree_writelock(session, lsm_tree); locked = true; @@ -567,7 +598,6 @@ __wt_lsm_merge(WT_SESSION_IMPL *session, WT_LSM_TREE *lsm_tree, u_int id) if (create_bloom) F_SET(chunk, WT_LSM_CHUNK_BLOOM); chunk->count = insert_count; - chunk->generation = generation; F_SET(chunk, WT_LSM_CHUNK_ONDISK); /* diff --git a/src/third_party/wiredtiger/src/lsm/lsm_meta.c b/src/third_party/wiredtiger/src/lsm/lsm_meta.c index 3a4960aaec1..de12f19380b 100644 --- a/src/third_party/wiredtiger/src/lsm/lsm_meta.c +++ b/src/third_party/wiredtiger/src/lsm/lsm_meta.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -104,7 +104,8 @@ __lsm_meta_read_v0( lsm_tree->chunk[nchunks++] = chunk; chunk->id = (uint32_t)lv.val; WT_RET(__wt_lsm_tree_chunk_name(session, - lsm_tree, chunk->id, &chunk->uri)); + lsm_tree, chunk->id, + chunk->generation, &chunk->uri)); F_SET(chunk, WT_LSM_CHUNK_ONDISK | WT_LSM_CHUNK_STABLE); @@ -200,6 +201,24 @@ __lsm_meta_read_v1( cv.str, cv.len, &lsm_tree->collator_name)); } + /* lsm.merge_custom does not appear in all V1 LSM metadata. */ + lsm_tree->custom_generation = 0; + if ((ret = __wt_config_getones( + session, lsmconf, "lsm.merge_custom.start_generation", &cv)) == 0) + lsm_tree->custom_generation = (uint32_t)cv.val; + WT_ERR_NOTFOUND_OK(ret); + if (lsm_tree->custom_generation != 0) { + WT_ERR(__wt_config_getones( + session, lsmconf, "lsm.merge_custom.prefix", &cv)); + WT_ERR(__wt_strndup(session, + cv.str, cv.len, &lsm_tree->custom_prefix)); + + WT_ERR(__wt_config_getones( + session, lsmconf, "lsm.merge_custom.suffix", &cv)); + WT_ERR(__wt_strndup(session, + cv.str, cv.len, &lsm_tree->custom_suffix)); + } + WT_ERR(__wt_config_getones(session, lsmconf, "lsm.auto_throttle", &cv)); if (cv.val) F_SET(lsm_tree, WT_LSM_TREE_THROTTLE); @@ -265,8 +284,7 @@ __lsm_meta_read_v1( lsm_tree->last = (u_int)cv.val; WT_ERR(__wt_config_getones(session, lsmconf, "chunks", &cv)); __wt_config_subinit(session, &lparser, &cv); - for (nchunks = 0; (ret = - __wt_config_next(&lparser, &lk, &lv)) == 0; ) { + for (nchunks = 0; (ret = __wt_config_next(&lparser, &lk, &lv)) == 0;) { if (WT_STRING_MATCH("id", lk.str, lk.len)) { WT_ERR(__wt_realloc_def(session, &lsm_tree->chunk_alloc, @@ -274,25 +292,23 @@ __lsm_meta_read_v1( WT_ERR(__wt_calloc_one(session, &chunk)); lsm_tree->chunk[nchunks++] = chunk; chunk->id = (uint32_t)lv.val; - WT_ERR(__wt_lsm_tree_chunk_name(session, - lsm_tree, chunk->id, &chunk->uri)); - F_SET(chunk, - WT_LSM_CHUNK_ONDISK | - WT_LSM_CHUNK_STABLE); + F_SET(chunk, WT_LSM_CHUNK_ONDISK | WT_LSM_CHUNK_STABLE); } else if (WT_STRING_MATCH("bloom", lk.str, lk.len)) { WT_ERR(__wt_lsm_tree_bloom_name( session, lsm_tree, chunk->id, &chunk->bloom_uri)); F_SET(chunk, WT_LSM_CHUNK_BLOOM); - continue; } else if (WT_STRING_MATCH("chunk_size", lk.str, lk.len)) { chunk->size = (uint64_t)lv.val; - continue; } else if (WT_STRING_MATCH("count", lk.str, lk.len)) { chunk->count = (uint64_t)lv.val; - continue; } else if (WT_STRING_MATCH("generation", lk.str, lk.len)) { chunk->generation = (uint32_t)lv.val; - continue; + /* + * Id appears first, but we need both id and generation + * to create the name. + */ + WT_ERR(__wt_lsm_tree_chunk_name(session, lsm_tree, + chunk->id, chunk->generation, &chunk->uri)); } } WT_ERR_NOTFOUND_OK(ret); @@ -300,8 +316,7 @@ __lsm_meta_read_v1( WT_ERR(__wt_config_getones(session, lsmconf, "old_chunks", &cv)); __wt_config_subinit(session, &lparser, &cv); - for (nchunks = 0; (ret = - __wt_config_next(&lparser, &lk, &lv)) == 0; ) { + for (nchunks = 0; (ret = __wt_config_next(&lparser, &lk, &lv)) == 0;) { if (WT_STRING_MATCH("bloom", lk.str, lk.len)) { WT_ERR(__wt_strndup(session, lv.str, lv.len, &chunk->bloom_uri)); @@ -473,7 +488,10 @@ __wt_lsm_meta_write(WT_SESSION_IMPL *session, WT_LSM_TREE *lsm_tree, chunk = lsm_tree->chunk[i]; if (i > 0) WT_ERR(__wt_buf_catfmt(session, buf, ",")); - WT_ERR(__wt_buf_catfmt(session, buf, "id=%" PRIu32, chunk->id)); + WT_ERR(__wt_buf_catfmt( + session, buf, "id=%" PRIu32, chunk->id)); + WT_ERR(__wt_buf_catfmt( + session, buf, ",generation=%" PRIu32, chunk->generation)); if (F_ISSET(chunk, WT_LSM_CHUNK_BLOOM)) WT_ERR(__wt_buf_catfmt(session, buf, ",bloom")); if (chunk->size != 0) @@ -482,8 +500,6 @@ __wt_lsm_meta_write(WT_SESSION_IMPL *session, WT_LSM_TREE *lsm_tree, if (chunk->count != 0) WT_ERR(__wt_buf_catfmt( session, buf, ",count=%" PRIu64, chunk->count)); - WT_ERR(__wt_buf_catfmt( - session, buf, ",generation=%" PRIu32, chunk->generation)); } WT_ERR(__wt_buf_catfmt(session, buf, "]")); WT_ERR(__wt_buf_catfmt(session, buf, ",old_chunks=[")); diff --git a/src/third_party/wiredtiger/src/lsm/lsm_stat.c b/src/third_party/wiredtiger/src/lsm/lsm_stat.c index 3688069d200..590aad926fc 100644 --- a/src/third_party/wiredtiger/src/lsm/lsm_stat.c +++ b/src/third_party/wiredtiger/src/lsm/lsm_stat.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/lsm/lsm_tree.c b/src/third_party/wiredtiger/src/lsm/lsm_tree.c index c5b63df099f..1b92028072d 100644 --- a/src/third_party/wiredtiger/src/lsm/lsm_tree.c +++ b/src/third_party/wiredtiger/src/lsm/lsm_tree.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -28,6 +28,8 @@ __lsm_tree_discard_state(WT_SESSION_IMPL *session, WT_LSM_TREE *lsm_tree) __wt_free(session, lsm_tree->key_format); __wt_free(session, lsm_tree->value_format); __wt_free(session, lsm_tree->collator_name); + __wt_free(session, lsm_tree->custom_prefix); + __wt_free(session, lsm_tree->custom_suffix); __wt_free(session, lsm_tree->bloom_config); __wt_free(session, lsm_tree->file_config); @@ -208,14 +210,22 @@ err: __wt_scr_free(session, &tmp); */ int __wt_lsm_tree_chunk_name(WT_SESSION_IMPL *session, - WT_LSM_TREE *lsm_tree, uint32_t id, const char **retp) + WT_LSM_TREE *lsm_tree, uint32_t id, uint32_t generation, const char **retp) { WT_DECL_ITEM(tmp); WT_DECL_RET; WT_RET(__wt_scr_alloc(session, 0, &tmp)); - WT_ERR(__wt_buf_fmt( - session, tmp, "file:%s-%06" PRIu32 ".lsm", lsm_tree->filename, id)); + + if (lsm_tree->custom_generation != 0 && + generation >= lsm_tree->custom_generation) + WT_ERR(__wt_buf_fmt(session, tmp, "%s:%s-%06" PRIu32 "%s", + lsm_tree->custom_prefix, lsm_tree->filename, id, + lsm_tree->custom_suffix)); + else + WT_ERR(__wt_buf_fmt(session, tmp, "file:%s-%06" PRIu32 ".lsm", + lsm_tree->filename, id)); + WT_ERR(__wt_strndup(session, tmp->data, tmp->size, retp)); err: __wt_scr_free(session, &tmp); @@ -229,16 +239,32 @@ err: __wt_scr_free(session, &tmp); */ int __wt_lsm_tree_set_chunk_size( - WT_SESSION_IMPL *session, WT_LSM_CHUNK *chunk) + WT_SESSION_IMPL *session, WT_LSM_TREE *lsm_tree, WT_LSM_CHUNK *chunk) { + WT_DATA_SOURCE *dsrc; wt_off_t size; const char *filename; - filename = chunk->uri; - if (!WT_PREFIX_SKIP(filename, "file:")) - WT_RET_MSG(session, EINVAL, - "Expected a 'file:' URI: %s", chunk->uri); - WT_RET(__wt_fs_size(session, filename, &size)); + size = 0; + if (lsm_tree->custom_generation != 0 && + chunk->generation >= lsm_tree->custom_generation) { + dsrc = __wt_schema_get_source(session, chunk->uri); + /* + * We can only retrieve a size if the data source exposes the + * information. + */ + if (dsrc != NULL && dsrc->size != NULL) { + /* Call the callback. */ + WT_RET(dsrc->size( + dsrc, (WT_SESSION*)session, chunk->uri, &size)); + } + } else { + filename = chunk->uri; + if (!WT_PREFIX_SKIP(filename, "file:")) + WT_RET_MSG(session, EINVAL, + "Expected a 'file:' URI: %s", chunk->uri); + WT_RET(__wt_fs_size(session, filename, &size)); + } chunk->size = (uint64_t)size; @@ -257,10 +283,13 @@ __lsm_tree_cleanup_old(WT_SESSION_IMPL *session, const char *uri) WT_DECL_RET; const char *cfg[] = { WT_CONFIG_BASE(session, WT_SESSION_drop), "force", NULL }; - bool exists; + bool exists, is_file; - WT_RET(__wt_fs_exist(session, uri + strlen("file:"), &exists)); - if (exists) + exists = false; + is_file = WT_PREFIX_MATCH(uri, "file:"); + if (is_file) + WT_RET(__wt_fs_exist(session, uri + strlen("file:"), &exists)); + if (!is_file || exists) WT_WITH_SCHEMA_LOCK(session, ret = __wt_schema_drop(session, uri, cfg)); return (ret); @@ -280,7 +309,7 @@ __wt_lsm_tree_setup_chunk( WT_RET(__wt_spin_init(session, &chunk->timestamp_spinlock, "LSM chunk timestamp")); WT_RET(__wt_lsm_tree_chunk_name( - session, lsm_tree, chunk->id, &chunk->uri)); + session, lsm_tree, chunk->id, chunk->generation, &chunk->uri)); /* * If the underlying file exists, drop the chunk first - there may be @@ -883,6 +912,7 @@ __wt_lsm_tree_drop( bool locked; locked = false; + WT_NOT_READ(locked); /* Get the LSM tree. */ WT_RET(__wt_lsm_tree_get(session, name, true, &lsm_tree)); @@ -942,6 +972,7 @@ __wt_lsm_tree_rename(WT_SESSION_IMPL *session, old = NULL; locked = false; + WT_NOT_READ(locked); /* Get the LSM tree. */ WT_RET(__wt_lsm_tree_get(session, olduri, true, &lsm_tree)); @@ -959,8 +990,8 @@ __wt_lsm_tree_rename(WT_SESSION_IMPL *session, old = chunk->uri; chunk->uri = NULL; - WT_ERR(__wt_lsm_tree_chunk_name( - session, lsm_tree, chunk->id, &chunk->uri)); + WT_ERR(__wt_lsm_tree_chunk_name(session, lsm_tree, + chunk->id, chunk->generation, &chunk->uri)); WT_ERR(__wt_schema_rename(session, old, chunk->uri, cfg)); __wt_free(session, old); @@ -1010,8 +1041,10 @@ __wt_lsm_tree_truncate( bool locked; WT_UNUSED(cfg); + chunk = NULL; locked = false; + WT_NOT_READ(locked); /* Get the LSM tree. */ WT_RET(__wt_lsm_tree_get(session, name, true, &lsm_tree)); @@ -1349,9 +1382,10 @@ __wt_lsm_tree_worker(WT_SESSION_IMPL *session, u_int i; bool exclusive, locked, need_release; - locked = false; - need_release = false; + locked = need_release = false; + WT_NOT_READ(locked); exclusive = FLD_ISSET(open_flags, WT_DHANDLE_EXCLUSIVE); + WT_RET(__wt_lsm_tree_get(session, uri, exclusive, &lsm_tree)); need_release = true; @@ -1390,20 +1424,18 @@ __wt_lsm_tree_worker(WT_SESSION_IMPL *session, */ if (FLD_ISSET(open_flags, WT_BTREE_ALTER)) { WT_ERR(__wt_lsm_meta_write(session, lsm_tree, cfg[0])); - /* - * We're about to discard the tree so we do not need to - * release it later. - */ - need_release = false; + + locked = false; if (exclusive) __wt_lsm_tree_writeunlock(session, lsm_tree); else __wt_lsm_tree_readunlock(session, lsm_tree); - locked = false; + /* * We rewrote the meta-data. Discard the tree and the next * access will reopen it. */ + need_release = false; WT_WITH_HANDLE_LIST_WRITE_LOCK(session, ret = __lsm_tree_discard(session, lsm_tree, false)); WT_ERR(ret); diff --git a/src/third_party/wiredtiger/src/lsm/lsm_work_unit.c b/src/third_party/wiredtiger/src/lsm/lsm_work_unit.c index 76827f7888c..29ece08004d 100644 --- a/src/third_party/wiredtiger/src/lsm/lsm_work_unit.c +++ b/src/third_party/wiredtiger/src/lsm/lsm_work_unit.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -326,6 +326,7 @@ __wt_lsm_checkpoint_chunk(WT_SESSION_IMPL *session, bool flush_set, release_dhandle; flush_set = release_dhandle = false; + WT_NOT_READ(flush_set); /* * If the chunk is already checkpointed, make sure it is also evicted. @@ -338,9 +339,10 @@ __wt_lsm_checkpoint_chunk(WT_SESSION_IMPL *session, ret = __lsm_discard_handle(session, chunk->uri, NULL)); if (ret == 0) chunk->evicted = 1; - else if (ret == EBUSY) + else if (ret == EBUSY) { ret = 0; - else + WT_NOT_READ(ret); + } else WT_RET_MSG(session, ret, "discard handle"); } if (F_ISSET(chunk, WT_LSM_CHUNK_ONDISK)) { @@ -409,7 +411,7 @@ __wt_lsm_checkpoint_chunk(WT_SESSION_IMPL *session, WT_ERR_MSG(session, ret, "LSM checkpoint"); /* Now the file is written, get the chunk size. */ - WT_ERR(__wt_lsm_tree_set_chunk_size(session, chunk)); + WT_ERR(__wt_lsm_tree_set_chunk_size(session, lsm_tree, chunk)); ++lsm_tree->chunks_flushed; diff --git a/src/third_party/wiredtiger/src/lsm/lsm_worker.c b/src/third_party/wiredtiger/src/lsm/lsm_worker.c index 8e89cf39099..82f72bdf355 100644 --- a/src/third_party/wiredtiger/src/lsm/lsm_worker.c +++ b/src/third_party/wiredtiger/src/lsm/lsm_worker.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/meta/meta_apply.c b/src/third_party/wiredtiger/src/meta/meta_apply.c index dbd3293dc2e..26d91c060ef 100644 --- a/src/third_party/wiredtiger/src/meta/meta_apply.c +++ b/src/third_party/wiredtiger/src/meta/meta_apply.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/meta/meta_ckpt.c b/src/third_party/wiredtiger/src/meta/meta_ckpt.c index 24f991cf549..a2edee8aac4 100644 --- a/src/third_party/wiredtiger/src/meta/meta_ckpt.c +++ b/src/third_party/wiredtiger/src/meta/meta_ckpt.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/meta/meta_ext.c b/src/third_party/wiredtiger/src/meta/meta_ext.c index b1d1d2be28f..7e28ba1d5cd 100644 --- a/src/third_party/wiredtiger/src/meta/meta_ext.c +++ b/src/third_party/wiredtiger/src/meta/meta_ext.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/meta/meta_table.c b/src/third_party/wiredtiger/src/meta/meta_table.c index 83688bbd591..d338c7769aa 100644 --- a/src/third_party/wiredtiger/src/meta/meta_table.c +++ b/src/third_party/wiredtiger/src/meta/meta_table.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/meta/meta_track.c b/src/third_party/wiredtiger/src/meta/meta_track.c index 0ad9a2aa429..6c92434aa2c 100644 --- a/src/third_party/wiredtiger/src/meta/meta_track.c +++ b/src/third_party/wiredtiger/src/meta/meta_track.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/meta/meta_turtle.c b/src/third_party/wiredtiger/src/meta/meta_turtle.c index f3e768c0948..8ce13384c0a 100644 --- a/src/third_party/wiredtiger/src/meta/meta_turtle.c +++ b/src/third_party/wiredtiger/src/meta/meta_turtle.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/optrack/optrack.c b/src/third_party/wiredtiger/src/optrack/optrack.c new file mode 100644 index 00000000000..8258a715927 --- /dev/null +++ b/src/third_party/wiredtiger/src/optrack/optrack.c @@ -0,0 +1,118 @@ +/*- + * Copyright (c) 2014-2018 MongoDB, Inc. + * Copyright (c) 2008-2014 WiredTiger, Inc. + * All rights reserved. + * + * See the file LICENSE for redistribution information. + */ + +#include "wt_internal.h" + +/* + * __wt_optrack_record_funcid -- + * Record optrack function id + */ +void +__wt_optrack_record_funcid( + WT_SESSION_IMPL *session, const char *func, uint16_t *func_idp) +{ + WT_CONNECTION_IMPL *conn; + WT_DECL_ITEM(tmp); + WT_DECL_RET; + wt_off_t fsize; + + conn = S2C(session); + + WT_ERR(__wt_scr_alloc(session, strlen(func) + 32, &tmp)); + + __wt_spin_lock(session, &conn->optrack_map_spinlock); + if (*func_idp == 0) { + *func_idp = ++conn->optrack_uid; + + WT_ERR(__wt_buf_fmt( + session, tmp, "%" PRIu16 " %s\n", *func_idp, func)); + WT_ERR(__wt_filesize(session, conn->optrack_map_fh, &fsize)); + WT_ERR(__wt_write(session, + conn->optrack_map_fh, fsize, tmp->size, tmp->data)); + } + + if (0) { +err: WT_PANIC_MSG(session, ret, "%s", __func__); + } + + __wt_spin_unlock(session, &conn->optrack_map_spinlock); + __wt_scr_free(session, &tmp); +} + +/* + * __wt_optrack_open_file -- + * Open the per-session operation-tracking file. + */ +int +__wt_optrack_open_file(WT_SESSION_IMPL *session) +{ + WT_CONNECTION_IMPL *conn; + WT_DECL_ITEM(buf); + WT_DECL_RET; + WT_OPTRACK_HEADER optrack_header = { WT_OPTRACK_VERSION, 0, + (uint32_t)WT_TSC_DEFAULT_RATIO * WT_THOUSAND }; + + conn = S2C(session); + + if (!F_ISSET(conn, WT_CONN_OPTRACK)) + return (WT_ERROR); + + WT_RET(__wt_scr_alloc(session, 0, &buf)); + WT_ERR(__wt_filename_construct(session, conn->optrack_path, + "optrack", conn->optrack_pid, session->id, buf)); + WT_ERR(__wt_open(session, + (const char *)buf->data, WT_FS_OPEN_FILE_TYPE_REGULAR, + WT_FS_OPEN_CREATE, &session->optrack_fh)); + + /* Indicate whether this is an internal session */ + if (F_ISSET(session, WT_SESSION_INTERNAL)) + optrack_header.optrack_session_internal = 1; + + /* + * Record the clock ticks to nanoseconds ratio. Multiply it by one + * thousand, so we can use a fixed width integer. + */ + optrack_header.optrack_tsc_nsec_ratio = + (uint32_t)(__wt_process.tsc_nsec_ratio * WT_THOUSAND); + + /* Write the header into the operation-tracking file. */ + WT_ERR(session->optrack_fh->handle->fh_write( + session->optrack_fh->handle, (WT_SESSION *)session, + 0, sizeof(WT_OPTRACK_HEADER), &optrack_header)); + + session->optrack_offset = sizeof(WT_OPTRACK_HEADER); + + if (0) { +err: WT_TRET(__wt_close(session, &session->optrack_fh)); + } + __wt_scr_free(session, &buf); + + return (ret); +} + +/* + * __wt_optrack_flush_buffer -- + * Flush optrack buffer. Returns the number of bytes flushed to the file. + */ +size_t +__wt_optrack_flush_buffer(WT_SESSION_IMPL *s) +{ + WT_DECL_RET; + + if (s->optrack_fh == NULL) + if (__wt_optrack_open_file(s)) + return (0); + + ret = s->optrack_fh->handle->fh_write(s->optrack_fh->handle, + (WT_SESSION *)s, (wt_off_t)s->optrack_offset, + s->optrackbuf_ptr * sizeof(WT_OPTRACK_RECORD), s->optrack_buf); + if (ret == 0) + return (s->optrackbuf_ptr * sizeof(WT_OPTRACK_RECORD)); + else + return (0); +} diff --git a/src/third_party/wiredtiger/src/os_common/filename.c b/src/third_party/wiredtiger/src/os_common/filename.c index 16825410dc3..6eebe545242 100644 --- a/src/third_party/wiredtiger/src/os_common/filename.c +++ b/src/third_party/wiredtiger/src/os_common/filename.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -54,6 +54,27 @@ err: __wt_free(session, buf); } /* + * __wt_filename_construct -- + * Given unique identifiers, return a WT_ITEM of a generated file name of + * the given prefix type. Any identifier that is 0 will be skipped. + */ +int +__wt_filename_construct(WT_SESSION_IMPL *session, const char *path, + const char *file_prefix, uintmax_t id_1, uint32_t id_2, WT_ITEM *buf) +{ + if (path != NULL && path[0] != '\0') + WT_RET(__wt_buf_catfmt( + session, buf, "%s%s", path, __wt_path_separator())); + WT_RET(__wt_buf_catfmt(session, buf, "%s", file_prefix)); + if (id_1 != UINTMAX_MAX) + WT_RET(__wt_buf_catfmt(session, buf, ".%010" PRIuMAX, id_1)); + if (id_2 != UINT32_MAX) + WT_RET(__wt_buf_catfmt(session, buf, ".%010" PRIu32, id_2)); + + return (0); +} + +/* * __wt_remove_if_exists -- * Remove a file if it exists. */ diff --git a/src/third_party/wiredtiger/src/os_common/os_abort.c b/src/third_party/wiredtiger/src/os_common/os_abort.c index ebef001ce67..a725ad9151d 100644 --- a/src/third_party/wiredtiger/src/os_common/os_abort.c +++ b/src/third_party/wiredtiger/src/os_common/os_abort.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/os_common/os_alloc.c b/src/third_party/wiredtiger/src/os_common/os_alloc.c index eae81586f65..bec28c67c9c 100644 --- a/src/third_party/wiredtiger/src/os_common/os_alloc.c +++ b/src/third_party/wiredtiger/src/os_common/os_alloc.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/os_common/os_errno.c b/src/third_party/wiredtiger/src/os_common/os_errno.c index d88d06d7610..0f57658ab23 100644 --- a/src/third_party/wiredtiger/src/os_common/os_errno.c +++ b/src/third_party/wiredtiger/src/os_common/os_errno.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/os_common/os_fhandle.c b/src/third_party/wiredtiger/src/os_common/os_fhandle.c index a353353216b..3e5ad67e031 100644 --- a/src/third_party/wiredtiger/src/os_common/os_fhandle.c +++ b/src/third_party/wiredtiger/src/os_common/os_fhandle.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -135,9 +135,8 @@ static inline int __open_verbose( WT_SESSION_IMPL *session, const char *name, int file_type, u_int flags) { -#ifdef HAVE_VERBOSE - WT_DECL_RET; WT_DECL_ITEM(tmp); + WT_DECL_RET; const char *file_type_tag, *sep; if (!WT_VERBOSE_ISSET(session, WT_VERB_FILEOPS)) @@ -193,13 +192,6 @@ __open_verbose( err: __wt_scr_free(session, &tmp); return (ret); -#else - WT_UNUSED(session); - WT_UNUSED(name); - WT_UNUSED(file_type); - WT_UNUSED(flags); - return (0); -#endif } /* @@ -219,6 +211,8 @@ __wt_open(WT_SESSION_IMPL *session, WT_ASSERT(session, file_type != 0); /* A file type is required. */ + *fhp = NULL; + conn = S2C(session); file_system = conn->file_system; fh = NULL; diff --git a/src/third_party/wiredtiger/src/os_common/os_fs_inmemory.c b/src/third_party/wiredtiger/src/os_common/os_fs_inmemory.c index e669ea2802d..cb7a05e05d9 100644 --- a/src/third_party/wiredtiger/src/os_common/os_fs_inmemory.c +++ b/src/third_party/wiredtiger/src/os_common/os_fs_inmemory.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/os_common/os_fstream.c b/src/third_party/wiredtiger/src/os_common/os_fstream.c index 7ceda73ce0a..184c4054e06 100644 --- a/src/third_party/wiredtiger/src/os_common/os_fstream.c +++ b/src/third_party/wiredtiger/src/os_common/os_fstream.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/os_common/os_fstream_stdio.c b/src/third_party/wiredtiger/src/os_common/os_fstream_stdio.c index 82e82b5f3e5..760abe84af5 100644 --- a/src/third_party/wiredtiger/src/os_common/os_fstream_stdio.c +++ b/src/third_party/wiredtiger/src/os_common/os_fstream_stdio.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/os_common/os_getopt.c b/src/third_party/wiredtiger/src/os_common/os_getopt.c index ca516ca62e5..caf1d7eba45 100644 --- a/src/third_party/wiredtiger/src/os_common/os_getopt.c +++ b/src/third_party/wiredtiger/src/os_common/os_getopt.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/src/os_common/os_strtouq.c b/src/third_party/wiredtiger/src/os_common/os_strtouq.c index 1cedfbdcb08..1ad2e05a22b 100644 --- a/src/third_party/wiredtiger/src/os_common/os_strtouq.c +++ b/src/third_party/wiredtiger/src/os_common/os_strtouq.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/os_posix/os_dir.c b/src/third_party/wiredtiger/src/os_posix/os_dir.c index 205ca389f43..2c2cb084a91 100644 --- a/src/third_party/wiredtiger/src/os_posix/os_dir.c +++ b/src/third_party/wiredtiger/src/os_posix/os_dir.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -11,13 +11,13 @@ #include <dirent.h> /* - * __wt_posix_directory_list -- + * __directory_list_worker -- * Get a list of files from a directory, POSIX version. */ -int -__wt_posix_directory_list(WT_FILE_SYSTEM *file_system, +static int +__directory_list_worker(WT_FILE_SYSTEM *file_system, WT_SESSION *wt_session, const char *directory, - const char *prefix, char ***dirlistp, uint32_t *countp) + const char *prefix, char ***dirlistp, uint32_t *countp, bool single) { struct dirent *dp; DIR *dirp; @@ -63,19 +63,20 @@ __wt_posix_directory_list(WT_FILE_SYSTEM *file_system, session, &dirallocsz, count + 1, &entries)); WT_ERR(__wt_strdup(session, dp->d_name, &entries[count])); ++count; + + if (single) + break; } *dirlistp = entries; *countp = count; -err: if (dirp != NULL) { - WT_SYSCALL(closedir(dirp), tret); - if (tret != 0) { - __wt_err(session, tret, - "%s: directory-list: closedir", directory); - if (ret == 0) - ret = tret; - } +err: WT_SYSCALL(closedir(dirp), tret); + if (tret != 0) { + __wt_err(session, tret, + "%s: directory-list: closedir", directory); + if (ret == 0) + ret = tret; } if (ret == 0) @@ -90,6 +91,32 @@ err: if (dirp != NULL) { } /* + * __wt_posix_directory_list -- + * Get a list of files from a directory, POSIX version. + */ +int +__wt_posix_directory_list(WT_FILE_SYSTEM *file_system, + WT_SESSION *wt_session, const char *directory, + const char *prefix, char ***dirlistp, uint32_t *countp) +{ + return (__directory_list_worker(file_system, + wt_session, directory, prefix, dirlistp, countp, false)); +} + +/* + * __wt_posix_directory_list_single -- + * Get one file from a directory, POSIX version. + */ +int +__wt_posix_directory_list_single(WT_FILE_SYSTEM *file_system, + WT_SESSION *wt_session, const char *directory, + const char *prefix, char ***dirlistp, uint32_t *countp) +{ + return (__directory_list_worker(file_system, + wt_session, directory, prefix, dirlistp, countp, true)); +} + +/* * __wt_posix_directory_list_free -- * Free memory returned by __wt_posix_directory_list. */ diff --git a/src/third_party/wiredtiger/src/os_posix/os_dlopen.c b/src/third_party/wiredtiger/src/os_posix/os_dlopen.c index 154b15a886c..b48e9d2f4db 100644 --- a/src/third_party/wiredtiger/src/os_posix/os_dlopen.c +++ b/src/third_party/wiredtiger/src/os_posix/os_dlopen.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/os_posix/os_fallocate.c b/src/third_party/wiredtiger/src/os_posix/os_fallocate.c index 5c57c5964b5..6f6e6a6bdc2 100644 --- a/src/third_party/wiredtiger/src/os_posix/os_fallocate.c +++ b/src/third_party/wiredtiger/src/os_posix/os_fallocate.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/os_posix/os_fs.c b/src/third_party/wiredtiger/src/os_posix/os_fs.c index c72c116015f..ca810fa8d88 100644 --- a/src/third_party/wiredtiger/src/os_posix/os_fs.c +++ b/src/third_party/wiredtiger/src/os_posix/os_fs.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. @@ -776,6 +776,8 @@ __wt_os_posix(WT_SESSION_IMPL *session) /* Initialize the POSIX jump table. */ file_system->fs_directory_list = __wt_posix_directory_list; + file_system->fs_directory_list_single = + __wt_posix_directory_list_single; file_system->fs_directory_list_free = __wt_posix_directory_list_free; file_system->fs_exist = __posix_fs_exist; file_system->fs_open_file = __posix_open_file; diff --git a/src/third_party/wiredtiger/src/os_posix/os_getenv.c b/src/third_party/wiredtiger/src/os_posix/os_getenv.c index 5b5a52cb273..0570f117595 100644 --- a/src/third_party/wiredtiger/src/os_posix/os_getenv.c +++ b/src/third_party/wiredtiger/src/os_posix/os_getenv.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/os_posix/os_map.c b/src/third_party/wiredtiger/src/os_posix/os_map.c index 5e625a49bac..b9ec284e124 100644 --- a/src/third_party/wiredtiger/src/os_posix/os_map.c +++ b/src/third_party/wiredtiger/src/os_posix/os_map.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/os_posix/os_mtx_cond.c b/src/third_party/wiredtiger/src/os_posix/os_mtx_cond.c index 533d2a0ab08..daaf55d65d2 100644 --- a/src/third_party/wiredtiger/src/os_posix/os_mtx_cond.c +++ b/src/third_party/wiredtiger/src/os_posix/os_mtx_cond.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -57,8 +57,11 @@ __wt_cond_wait_signal(WT_SESSION_IMPL *session, WT_CONDVAR *cond, { struct timespec ts; WT_DECL_RET; + WT_TRACK_OP_DECL; bool locked; + WT_TRACK_OP_INIT(session); + locked = false; /* Fast path if already signalled. */ @@ -135,8 +138,10 @@ err: (void)__wt_atomic_subi32(&cond->waiters, 1); if (locked) WT_TRET(pthread_mutex_unlock(&cond->mtx)); - if (ret == 0) + if (ret == 0) { + WT_TRACK_OP_END(session); return; + } WT_PANIC_MSG(session, ret, "pthread_cond_wait: %s", cond->name); } diff --git a/src/third_party/wiredtiger/src/os_posix/os_once.c b/src/third_party/wiredtiger/src/os_posix/os_once.c index d2913997711..1e123c84a53 100644 --- a/src/third_party/wiredtiger/src/os_posix/os_once.c +++ b/src/third_party/wiredtiger/src/os_posix/os_once.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/os_posix/os_pagesize.c b/src/third_party/wiredtiger/src/os_posix/os_pagesize.c index 09c52c41fe5..360f772d328 100644 --- a/src/third_party/wiredtiger/src/os_posix/os_pagesize.c +++ b/src/third_party/wiredtiger/src/os_posix/os_pagesize.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/os_posix/os_path.c b/src/third_party/wiredtiger/src/os_posix/os_path.c index fc1a0fd4910..1f0a5f0dee6 100644 --- a/src/third_party/wiredtiger/src/os_posix/os_path.c +++ b/src/third_party/wiredtiger/src/os_posix/os_path.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/os_posix/os_priv.c b/src/third_party/wiredtiger/src/os_posix/os_priv.c index 0e0f5dfb190..3c981ce51c2 100644 --- a/src/third_party/wiredtiger/src/os_posix/os_priv.c +++ b/src/third_party/wiredtiger/src/os_posix/os_priv.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/os_posix/os_setvbuf.c b/src/third_party/wiredtiger/src/os_posix/os_setvbuf.c index a916ef79311..44836fba0d0 100644 --- a/src/third_party/wiredtiger/src/os_posix/os_setvbuf.c +++ b/src/third_party/wiredtiger/src/os_posix/os_setvbuf.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/os_posix/os_sleep.c b/src/third_party/wiredtiger/src/os_posix/os_sleep.c index 67c0aaa375c..88209d7308e 100644 --- a/src/third_party/wiredtiger/src/os_posix/os_sleep.c +++ b/src/third_party/wiredtiger/src/os_posix/os_sleep.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/os_posix/os_snprintf.c b/src/third_party/wiredtiger/src/os_posix/os_snprintf.c index 3ac0183f5ec..ff0bdd46251 100644 --- a/src/third_party/wiredtiger/src/os_posix/os_snprintf.c +++ b/src/third_party/wiredtiger/src/os_posix/os_snprintf.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/os_posix/os_thread.c b/src/third_party/wiredtiger/src/os_posix/os_thread.c index dc4d49ad493..7dd803e1b00 100644 --- a/src/third_party/wiredtiger/src/os_posix/os_thread.c +++ b/src/third_party/wiredtiger/src/os_posix/os_thread.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -110,3 +110,13 @@ __wt_thread_str(char *buf, size_t buflen) "%" PRIuMAX ":%p", (uintmax_t)getpid(), (void *)self)); #endif } + +/* + * __wt_process_id -- + * Return the process ID assigned by the operating system. + */ +uintmax_t +__wt_process_id(void) +{ + return ((uintmax_t)getpid()); +} diff --git a/src/third_party/wiredtiger/src/os_posix/os_time.c b/src/third_party/wiredtiger/src/os_posix/os_time.c index 1b7a9359531..7db4522a3e6 100644 --- a/src/third_party/wiredtiger/src/os_posix/os_time.c +++ b/src/third_party/wiredtiger/src/os_posix/os_time.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/os_posix/os_yield.c b/src/third_party/wiredtiger/src/os_posix/os_yield.c index 3190e9e7062..2c0aeb4c94e 100644 --- a/src/third_party/wiredtiger/src/os_posix/os_yield.c +++ b/src/third_party/wiredtiger/src/os_posix/os_yield.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/os_win/os_dir.c b/src/third_party/wiredtiger/src/os_win/os_dir.c index 69235659f04..3b78106f3b4 100644 --- a/src/third_party/wiredtiger/src/os_win/os_dir.c +++ b/src/third_party/wiredtiger/src/os_win/os_dir.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -9,13 +9,13 @@ #include "wt_internal.h" /* - * __wt_win_directory_list -- + * __directory_list_worker -- * Get a list of files from a directory, MSVC version. */ -int -__wt_win_directory_list(WT_FILE_SYSTEM *file_system, +static int +__directory_list_worker(WT_FILE_SYSTEM *file_system, WT_SESSION *wt_session, const char *directory, - const char *prefix, char ***dirlistp, uint32_t *countp) + const char *prefix, char ***dirlistp, uint32_t *countp, bool single) { DWORD windows_error; HANDLE findhandle; @@ -82,7 +82,7 @@ __wt_win_directory_list(WT_FILE_SYSTEM *file_system, WT_ERR(__wt_strdup(session, file_utf8->data, &entries[count])); ++count; __wt_scr_free(session, &file_utf8); - } while (FindNextFileW(findhandle, &finddata) != 0); + } while (!single && FindNextFileW(findhandle, &finddata) != 0); *dirlistp = entries; *countp = count; @@ -116,6 +116,32 @@ err: if (findhandle != INVALID_HANDLE_VALUE) } /* + * __wt_win_directory_list -- + * Get a list of files from a directory, MSVC version. + */ +int +__wt_win_directory_list(WT_FILE_SYSTEM *file_system, + WT_SESSION *wt_session, const char *directory, + const char *prefix, char ***dirlistp, uint32_t *countp) +{ + return (__directory_list_worker(file_system, + wt_session, directory, prefix, dirlistp, countp, false)); +} + +/* + * __wt_win_directory_list_single -- + * Get a single file from a directory, MSVC version. + */ +int +__wt_win_directory_list_single(WT_FILE_SYSTEM *file_system, + WT_SESSION *wt_session, const char *directory, + const char *prefix, char ***dirlistp, uint32_t *countp) +{ + return (__directory_list_worker(file_system, + wt_session, directory, prefix, dirlistp, countp, true)); +} + +/* * __wt_win_directory_list_free -- * Free memory returned by __wt_win_directory_list, Windows version. */ diff --git a/src/third_party/wiredtiger/src/os_win/os_dlopen.c b/src/third_party/wiredtiger/src/os_win/os_dlopen.c index 9ee4d703c7a..a4e7ebe4fbb 100644 --- a/src/third_party/wiredtiger/src/os_win/os_dlopen.c +++ b/src/third_party/wiredtiger/src/os_win/os_dlopen.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/os_win/os_fs.c b/src/third_party/wiredtiger/src/os_win/os_fs.c index 1410a7bad03..66f4de87299 100644 --- a/src/third_party/wiredtiger/src/os_win/os_fs.c +++ b/src/third_party/wiredtiger/src/os_win/os_fs.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -651,6 +651,7 @@ __wt_os_win(WT_SESSION_IMPL *session) /* Initialize the Windows jump table. */ file_system->fs_directory_list = __wt_win_directory_list; + file_system->fs_directory_list_single = __wt_win_directory_list_single; file_system->fs_directory_list_free = __wt_win_directory_list_free; file_system->fs_exist = __win_fs_exist; file_system->fs_open_file = __win_open_file; diff --git a/src/third_party/wiredtiger/src/os_win/os_getenv.c b/src/third_party/wiredtiger/src/os_win/os_getenv.c index b7b7f765656..d3f779396f6 100644 --- a/src/third_party/wiredtiger/src/os_win/os_getenv.c +++ b/src/third_party/wiredtiger/src/os_win/os_getenv.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/os_win/os_map.c b/src/third_party/wiredtiger/src/os_win/os_map.c index 25835d2615d..2223a6037ef 100644 --- a/src/third_party/wiredtiger/src/os_win/os_map.c +++ b/src/third_party/wiredtiger/src/os_win/os_map.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -82,8 +82,6 @@ __wt_win_unmap(WT_FILE_HANDLE *file_handle, WT_SESSION *wt_session, WT_FILE_HANDLE_WIN *win_fh; WT_SESSION_IMPL *session; - WT_UNUSED(length); /* !HAVE_VERBOSE */ - win_fh = (WT_FILE_HANDLE_WIN *)file_handle; session = (WT_SESSION_IMPL *)wt_session; diff --git a/src/third_party/wiredtiger/src/os_win/os_mtx_cond.c b/src/third_party/wiredtiger/src/os_win/os_mtx_cond.c index 9d4339c8731..4394d83afef 100644 --- a/src/third_party/wiredtiger/src/os_win/os_mtx_cond.c +++ b/src/third_party/wiredtiger/src/os_win/os_mtx_cond.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/os_win/os_once.c b/src/third_party/wiredtiger/src/os_win/os_once.c index dd21c58b8af..2b607c04c62 100644 --- a/src/third_party/wiredtiger/src/os_win/os_once.c +++ b/src/third_party/wiredtiger/src/os_win/os_once.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/os_win/os_pagesize.c b/src/third_party/wiredtiger/src/os_win/os_pagesize.c index 07b1c3afc5c..45afa35ff25 100644 --- a/src/third_party/wiredtiger/src/os_win/os_pagesize.c +++ b/src/third_party/wiredtiger/src/os_win/os_pagesize.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/os_win/os_path.c b/src/third_party/wiredtiger/src/os_win/os_path.c index 78ad3bda509..1c6cdc76ceb 100644 --- a/src/third_party/wiredtiger/src/os_win/os_path.c +++ b/src/third_party/wiredtiger/src/os_win/os_path.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/os_win/os_priv.c b/src/third_party/wiredtiger/src/os_win/os_priv.c index acc3793255a..ce48aa5fa4e 100644 --- a/src/third_party/wiredtiger/src/os_win/os_priv.c +++ b/src/third_party/wiredtiger/src/os_win/os_priv.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/os_win/os_setvbuf.c b/src/third_party/wiredtiger/src/os_win/os_setvbuf.c index 78e42ecf4b5..6a7f02fe106 100644 --- a/src/third_party/wiredtiger/src/os_win/os_setvbuf.c +++ b/src/third_party/wiredtiger/src/os_win/os_setvbuf.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/os_win/os_sleep.c b/src/third_party/wiredtiger/src/os_win/os_sleep.c index 477474e0665..448447f35ef 100644 --- a/src/third_party/wiredtiger/src/os_win/os_sleep.c +++ b/src/third_party/wiredtiger/src/os_win/os_sleep.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/os_win/os_snprintf.c b/src/third_party/wiredtiger/src/os_win/os_snprintf.c index 20231b468c6..c55422981eb 100644 --- a/src/third_party/wiredtiger/src/os_win/os_snprintf.c +++ b/src/third_party/wiredtiger/src/os_win/os_snprintf.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/os_win/os_thread.c b/src/third_party/wiredtiger/src/os_win/os_thread.c index 1d549cf4712..f27ea84181e 100644 --- a/src/third_party/wiredtiger/src/os_win/os_thread.c +++ b/src/third_party/wiredtiger/src/os_win/os_thread.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -96,3 +96,13 @@ __wt_thread_str(char *buf, size_t buflen) "%" PRIu64 ":%" PRIu64, (uint64_t)GetCurrentProcessId(), (uint64_t)GetCurrentThreadId)); } + +/* + * __wt_process_id -- + * Return the process ID assigned by the operating system. + */ +uintmax_t +__wt_process_id(void) +{ + return (uintmax_t)GetCurrentProcessId(); +} diff --git a/src/third_party/wiredtiger/src/os_win/os_time.c b/src/third_party/wiredtiger/src/os_win/os_time.c index 4e9d4595fae..284d2a63931 100644 --- a/src/third_party/wiredtiger/src/os_win/os_time.c +++ b/src/third_party/wiredtiger/src/os_win/os_time.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/os_win/os_utf8.c b/src/third_party/wiredtiger/src/os_win/os_utf8.c index f7bab41c81f..077c39db3ef 100644 --- a/src/third_party/wiredtiger/src/os_win/os_utf8.c +++ b/src/third_party/wiredtiger/src/os_win/os_utf8.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/os_win/os_winerr.c b/src/third_party/wiredtiger/src/os_win/os_winerr.c index c7748d80fb2..8c4b9d8e098 100644 --- a/src/third_party/wiredtiger/src/os_win/os_winerr.c +++ b/src/third_party/wiredtiger/src/os_win/os_winerr.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/os_win/os_yield.c b/src/third_party/wiredtiger/src/os_win/os_yield.c index e38fc21e16b..b347d2d510b 100644 --- a/src/third_party/wiredtiger/src/os_win/os_yield.c +++ b/src/third_party/wiredtiger/src/os_win/os_yield.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/packing/pack_api.c b/src/third_party/wiredtiger/src/packing/pack_api.c index ee7ce6c4c0d..db0485c459a 100644 --- a/src/third_party/wiredtiger/src/packing/pack_api.c +++ b/src/third_party/wiredtiger/src/packing/pack_api.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -14,7 +14,7 @@ */ int wiredtiger_struct_pack(WT_SESSION *wt_session, - void *buffer, size_t size, const char *fmt, ...) + void *buffer, size_t len, const char *format, ...) { WT_DECL_RET; WT_SESSION_IMPL *session; @@ -22,8 +22,8 @@ wiredtiger_struct_pack(WT_SESSION *wt_session, session = (WT_SESSION_IMPL *)wt_session; - va_start(ap, fmt); - ret = __wt_struct_packv(session, buffer, size, fmt, ap); + va_start(ap, format); + ret = __wt_struct_packv(session, buffer, len, format, ap); va_end(ap); return (ret); @@ -35,7 +35,7 @@ wiredtiger_struct_pack(WT_SESSION *wt_session, */ int wiredtiger_struct_size(WT_SESSION *wt_session, - size_t *sizep, const char *fmt, ...) + size_t *lenp, const char *format, ...) { WT_DECL_RET; WT_SESSION_IMPL *session; @@ -43,8 +43,8 @@ wiredtiger_struct_size(WT_SESSION *wt_session, session = (WT_SESSION_IMPL *)wt_session; - va_start(ap, fmt); - ret = __wt_struct_sizev(session, sizep, fmt, ap); + va_start(ap, format); + ret = __wt_struct_sizev(session, lenp, format, ap); va_end(ap); return (ret); @@ -56,7 +56,7 @@ wiredtiger_struct_size(WT_SESSION *wt_session, */ int wiredtiger_struct_unpack(WT_SESSION *wt_session, - const void *buffer, size_t size, const char *fmt, ...) + const void *buffer, size_t len, const char *format, ...) { WT_DECL_RET; WT_SESSION_IMPL *session; @@ -64,8 +64,8 @@ wiredtiger_struct_unpack(WT_SESSION *wt_session, session = (WT_SESSION_IMPL *)wt_session; - va_start(ap, fmt); - ret = __wt_struct_unpackv(session, buffer, size, fmt, ap); + va_start(ap, format); + ret = __wt_struct_unpackv(session, buffer, len, format, ap); va_end(ap); return (ret); @@ -77,7 +77,7 @@ wiredtiger_struct_unpack(WT_SESSION *wt_session, */ int __wt_ext_struct_pack(WT_EXTENSION_API *wt_api, WT_SESSION *wt_session, - void *buffer, size_t size, const char *fmt, ...) + void *buffer, size_t len, const char *fmt, ...) { WT_DECL_RET; WT_SESSION_IMPL *session; @@ -87,7 +87,7 @@ __wt_ext_struct_pack(WT_EXTENSION_API *wt_api, WT_SESSION *wt_session, ((WT_CONNECTION_IMPL *)wt_api->conn)->default_session; va_start(ap, fmt); - ret = __wt_struct_packv(session, buffer, size, fmt, ap); + ret = __wt_struct_packv(session, buffer, len, fmt, ap); va_end(ap); return (ret); @@ -99,7 +99,7 @@ __wt_ext_struct_pack(WT_EXTENSION_API *wt_api, WT_SESSION *wt_session, */ int __wt_ext_struct_size(WT_EXTENSION_API *wt_api, WT_SESSION *wt_session, - size_t *sizep, const char *fmt, ...) + size_t *lenp, const char *fmt, ...) { WT_DECL_RET; WT_SESSION_IMPL *session; @@ -109,7 +109,7 @@ __wt_ext_struct_size(WT_EXTENSION_API *wt_api, WT_SESSION *wt_session, ((WT_CONNECTION_IMPL *)wt_api->conn)->default_session; va_start(ap, fmt); - ret = __wt_struct_sizev(session, sizep, fmt, ap); + ret = __wt_struct_sizev(session, lenp, fmt, ap); va_end(ap); return (ret); @@ -121,7 +121,7 @@ __wt_ext_struct_size(WT_EXTENSION_API *wt_api, WT_SESSION *wt_session, */ int __wt_ext_struct_unpack(WT_EXTENSION_API *wt_api, WT_SESSION *wt_session, - const void *buffer, size_t size, const char *fmt, ...) + const void *buffer, size_t len, const char *fmt, ...) { WT_DECL_RET; WT_SESSION_IMPL *session; @@ -131,7 +131,7 @@ __wt_ext_struct_unpack(WT_EXTENSION_API *wt_api, WT_SESSION *wt_session, ((WT_CONNECTION_IMPL *)wt_api->conn)->default_session; va_start(ap, fmt); - ret = __wt_struct_unpackv(session, buffer, size, fmt, ap); + ret = __wt_struct_unpackv(session, buffer, len, fmt, ap); va_end(ap); return (ret); diff --git a/src/third_party/wiredtiger/src/packing/pack_impl.c b/src/third_party/wiredtiger/src/packing/pack_impl.c index d40043fc13c..285f8ddfdaa 100644 --- a/src/third_party/wiredtiger/src/packing/pack_impl.c +++ b/src/third_party/wiredtiger/src/packing/pack_impl.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -25,9 +25,7 @@ __wt_struct_check(WT_SESSION_IMPL *session, WT_RET(__pack_initn(session, &pack, fmt, len)); for (fields = 0; (ret = __pack_next(&pack, &pv)) == 0; fields++) ; - - if (ret != WT_NOTFOUND) - return (ret); + WT_RET_NOTFOUND_OK(ret); if (fixedp != NULL && fixed_lenp != NULL) { if (fields == 0) { @@ -58,13 +56,13 @@ __wt_struct_confchk(WT_SESSION_IMPL *session, WT_CONFIG_ITEM *v) * Calculate the size of a packed byte string. */ int -__wt_struct_size(WT_SESSION_IMPL *session, size_t *sizep, const char *fmt, ...) +__wt_struct_size(WT_SESSION_IMPL *session, size_t *lenp, const char *fmt, ...) { WT_DECL_RET; va_list ap; va_start(ap, fmt); - ret = __wt_struct_sizev(session, sizep, fmt, ap); + ret = __wt_struct_sizev(session, lenp, fmt, ap); va_end(ap); return (ret); @@ -76,13 +74,13 @@ __wt_struct_size(WT_SESSION_IMPL *session, size_t *sizep, const char *fmt, ...) */ int __wt_struct_pack(WT_SESSION_IMPL *session, - void *buffer, size_t size, const char *fmt, ...) + void *buffer, size_t len, const char *fmt, ...) { WT_DECL_RET; va_list ap; va_start(ap, fmt); - ret = __wt_struct_packv(session, buffer, size, fmt, ap); + ret = __wt_struct_packv(session, buffer, len, fmt, ap); va_end(ap); return (ret); @@ -94,13 +92,13 @@ __wt_struct_pack(WT_SESSION_IMPL *session, */ int __wt_struct_unpack(WT_SESSION_IMPL *session, - const void *buffer, size_t size, const char *fmt, ...) + const void *buffer, size_t len, const char *fmt, ...) { WT_DECL_RET; va_list ap; va_start(ap, fmt); - ret = __wt_struct_unpackv(session, buffer, size, fmt, ap); + ret = __wt_struct_unpackv(session, buffer, len, fmt, ap); va_end(ap); return (ret); diff --git a/src/third_party/wiredtiger/src/packing/pack_stream.c b/src/third_party/wiredtiger/src/packing/pack_stream.c index dc2925acaf3..80dfe906bdf 100644 --- a/src/third_party/wiredtiger/src/packing/pack_stream.c +++ b/src/third_party/wiredtiger/src/packing/pack_stream.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -24,7 +24,7 @@ struct __wt_pack_stream { */ int wiredtiger_pack_start(WT_SESSION *wt_session, - const char *format, void *buffer, size_t len, WT_PACK_STREAM **psp) + const char *format, void *buffer, size_t size, WT_PACK_STREAM **psp) { WT_DECL_RET; WT_PACK_STREAM *ps; @@ -34,7 +34,7 @@ wiredtiger_pack_start(WT_SESSION *wt_session, WT_RET(__wt_calloc_one(session, &ps)); WT_ERR(__pack_init(session, &ps->pack, format)); ps->p = ps->start = buffer; - ps->end = ps->p + len; + ps->end = ps->p + size; *psp = ps; if (0) { @@ -48,8 +48,8 @@ err: (void)wiredtiger_pack_close(ps, NULL); * Open a stream for unpacking. */ int -wiredtiger_unpack_start(WT_SESSION *wt_session, const char *format, - const void *buffer, size_t size, WT_PACK_STREAM **psp) +wiredtiger_unpack_start(WT_SESSION *wt_session, + const char *format, const void *buffer, size_t size, WT_PACK_STREAM **psp) { return (wiredtiger_pack_start( wt_session, format, (void *)buffer, size, psp)); diff --git a/src/third_party/wiredtiger/src/reconcile/rec_track.c b/src/third_party/wiredtiger/src/reconcile/rec_track.c index d0a6df2a7ec..0a5eb7eacc2 100644 --- a/src/third_party/wiredtiger/src/reconcile/rec_track.c +++ b/src/third_party/wiredtiger/src/reconcile/rec_track.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -36,9 +36,6 @@ __ovfl_discard_verbose( WT_CELL_UNPACK *unpack, _unpack; WT_DECL_ITEM(tmp); - WT_UNUSED(page); /* !HAVE_VERBOSE */ - WT_UNUSED(tag); /* !HAVE_VERBOSE */ - WT_RET(__wt_scr_alloc(session, 512, &tmp)); unpack = &_unpack; @@ -172,10 +169,6 @@ __ovfl_reuse_verbose(WT_SESSION_IMPL *session, { WT_DECL_ITEM(tmp); - WT_UNUSED(page); /* !HAVE_VERBOSE */ - WT_UNUSED(reuse); /* !HAVE_VERBOSE */ - WT_UNUSED(tag); /* !HAVE_VERBOSE */ - WT_RET(__wt_scr_alloc(session, 64, &tmp)); __wt_verbose(session, WT_VERB_OVERFLOW, diff --git a/src/third_party/wiredtiger/src/reconcile/rec_write.c b/src/third_party/wiredtiger/src/reconcile/rec_write.c index 233e0ec61f6..d9b415a76cd 100644 --- a/src/third_party/wiredtiger/src/reconcile/rec_write.c +++ b/src/third_party/wiredtiger/src/reconcile/rec_write.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -387,8 +387,10 @@ __wt_reconcile(WT_SESSION_IMPL *session, WT_REF *ref, * have a snapshot. This doesn't apply to checkpoints: there are * (rare) cases where we write data at read-uncommitted isolation. */ - WT_ASSERT(session, !LF_ISSET(WT_REC_UPDATE_RESTORE) || - LF_ISSET(WT_REC_VISIBLE_ALL)); + WT_ASSERT(session, + !LF_ISSET(WT_REC_LOOKASIDE) || !LF_ISSET(WT_REC_UPDATE_RESTORE)); + WT_ASSERT(session, + !LF_ISSET(WT_REC_UPDATE_RESTORE) || LF_ISSET(WT_REC_VISIBLE_ALL)); WT_ASSERT(session, !LF_ISSET(WT_REC_EVICT) || LF_ISSET(WT_REC_VISIBLE_ALL) || F_ISSET(&session->txn, WT_TXN_HAS_SNAPSHOT)); @@ -797,9 +799,13 @@ __rec_root_write(WT_SESSION_IMPL *session, WT_PAGE *page, uint32_t flags) /* * Create a new root page, initialize the array of child references, * mark it dirty, then write it. + * + * Don't count the eviction of this page as progress, checkpoint can + * repeatedly create and discard these pages. */ WT_RET(__wt_page_alloc(session, page->type, mod->mod_multi_entries, false, &next)); + F_SET_ATOMIC(next, WT_PAGE_EVICT_NO_PROGRESS); WT_INTL_INDEX_GET(session, next, pindex); for (i = 0; i < mod->mod_multi_entries; ++i) { @@ -1086,9 +1092,11 @@ __rec_init(WT_SESSION_IMPL *session, /* * The fake cursor used to figure out modified update values points to - * the enclosing WT_REF as a way to access the page. + * the enclosing WT_REF as a way to access the page, and also needs to + * set the format. */ r->update_modify_cbt.ref = ref; + r->update_modify_cbt.iface.value_format = btree->value_format; return (0); } @@ -1208,6 +1216,13 @@ __rec_append_orig_value(WT_SESSION_IMPL *session, __wt_txn_upd_visible_all(session, upd)) return (0); + /* Add the original value after birthmarks. */ + if (upd->type == WT_UPDATE_BIRTHMARK) { + WT_ASSERT(session, unpack != NULL && + unpack->type != WT_CELL_DEL); + break; + } + /* Leave reference at the last item in the chain. */ if (upd->next == NULL) break; @@ -1226,7 +1241,7 @@ __rec_append_orig_value(WT_SESSION_IMPL *session, size = 0; /* -Wconditional-uninitialized */ if (unpack == NULL || unpack->type == WT_CELL_DEL) WT_RET(__wt_update_alloc(session, - NULL, &append, &size, WT_UPDATE_DELETED)); + NULL, &append, &size, WT_UPDATE_TOMBSTONE)); else { WT_RET(__wt_scr_alloc(session, 0, &tmp)); WT_ERR(__wt_page_cell_data_ref(session, page, unpack, tmp)); @@ -1235,15 +1250,28 @@ __rec_append_orig_value(WT_SESSION_IMPL *session, } /* - * Set the entry's transaction information to the lowest possible value. - * Since cleared memory matches the lowest possible transaction ID and - * timestamp, do nothing. + * If we're saving the original value for a birthmark, transfer over + * the transaction ID and clear out the birthmark update. * - * Append the new entry to the update list. + * Else, set the entry's transaction information to the lowest possible + * value. Cleared memory matches the lowest possible transaction ID and + * timestamp, do nothing. */ + if (upd->type == WT_UPDATE_BIRTHMARK) { + append->txnid = upd->txnid; + __wt_timestamp_set(&append->timestamp, &upd->timestamp); + append->next = upd->next; + } + + /* Append the new entry into the update list. */ WT_PUBLISH(upd->next, append); __wt_cache_page_inmem_incr(session, page, size); + if (upd->type == WT_UPDATE_BIRTHMARK) { + upd->type = WT_UPDATE_STANDARD; + upd->txnid = WT_TXN_ABORTED; + } + err: __wt_scr_free(session, &tmp); return (ret); } @@ -1263,7 +1291,7 @@ __rec_txn_read(WT_SESSION_IMPL *session, WT_RECONCILE *r, wt_timestamp_t *timestampp; size_t upd_memsize; uint64_t max_txn, txnid; - bool all_visible, uncommitted; + bool all_visible, skipped_birthmark, uncommitted; #ifdef HAVE_TIMESTAMPS WT_UPDATE *first_ts_upd; @@ -1278,7 +1306,7 @@ __rec_txn_read(WT_SESSION_IMPL *session, WT_RECONCILE *r, first_txn_upd = NULL; upd_memsize = 0; max_txn = WT_TXN_NONE; - uncommitted = false; + skipped_birthmark = uncommitted = false; /* * If called with a WT_INSERT item, use its WT_UPDATE list (which must @@ -1338,15 +1366,15 @@ __rec_txn_read(WT_SESSION_IMPL *session, WT_RECONCILE *r, * committed update. Regular eviction checks that the maximum * transaction ID and timestamp seen are stable. * - * Lookaside eviction tries to choose the same version as a - * subsequent checkpoint, so that checkpoint can skip over - * pages with lookaside entries. If the application has - * supplied a stable timestamp, we assume (a) that it is old, - * and (b) that the next checkpoint will use it, so we wait to - * see a stable update. If there is no stable timestamp, we - * assume the next checkpoint will write the most recent - * version (but we save enough information that checkpoint can - * fix things up if we choose an update that is too new). + * Lookaside and update/restore eviction try to choose the same + * version as a subsequent checkpoint, so that checkpoint can + * skip over pages with lookaside entries. If the application + * has supplied a stable timestamp, we assume (a) that it is + * old, and (b) that the next checkpoint will use it, so we wait + * to see a stable update. If there is no stable timestamp, we + * assume the next checkpoint will write the most recent version + * (but we save enough information that checkpoint can fix + * things up if we choose an update that is too new). */ if (*updp == NULL && r->las_skew_newest) *updp = upd; @@ -1371,6 +1399,9 @@ __rec_txn_read(WT_SESSION_IMPL *session, WT_RECONCILE *r, return (EBUSY); } + if (upd->type == WT_UPDATE_BIRTHMARK) + skipped_birthmark = true; + continue; } @@ -1383,14 +1414,31 @@ __rec_txn_read(WT_SESSION_IMPL *session, WT_RECONCILE *r, *updp = upd; } + /* Keep track of the selected update. */ + upd = *updp; + /* Reconciliation should never see an aborted or reserved update. */ - WT_ASSERT(session, *updp == NULL || - ((*updp)->txnid != WT_TXN_ABORTED && - (*updp)->type != WT_UPDATE_RESERVED)); + WT_ASSERT(session, upd == NULL || + (upd->txnid != WT_TXN_ABORTED && upd->type != WT_UPDATE_RESERVE)); /* If all of the updates were aborted, quit. */ - if (first_txn_upd == NULL) + if (first_txn_upd == NULL) { + WT_ASSERT(session, upd == NULL); return (0); + } + + /* If no updates were skipped, record that we're making progress. */ + if (upd == first_txn_upd) + r->update_used = true; + + /* + * The checkpoint transaction is special. Make sure we never write + * metadata updates from a checkpoint in a concurrent session. + */ + WT_ASSERT(session, !WT_IS_METADATA(session->dhandle) || + upd == NULL || upd->txnid == WT_TXN_NONE || + upd->txnid != S2C(session)->txn_global.checkpoint_state.id || + WT_SESSION_IS_CHECKPOINT(session)); /* * Track the most recent transaction in the page. We store this in the @@ -1402,25 +1450,26 @@ __rec_txn_read(WT_SESSION_IMPL *session, WT_RECONCILE *r, r->max_txn = max_txn; #ifdef HAVE_TIMESTAMPS + /* Update the maximum timestamp. */ if (first_ts_upd != NULL && __wt_timestamp_cmp(&r->max_timestamp, &first_ts_upd->timestamp) < 0) __wt_timestamp_set(&r->max_timestamp, &first_ts_upd->timestamp); -#endif - /* - * The checkpoint transaction is special. Make sure we never write - * metadata updates from a checkpoint in a concurrent session. - */ - WT_ASSERT(session, !WT_IS_METADATA(session->dhandle) || - *updp == NULL || (*updp)->txnid == WT_TXN_NONE || - (*updp)->txnid != S2C(session)->txn_global.checkpoint_state.id || - WT_SESSION_IS_CHECKPOINT(session)); + /* Update the maximum on-page timestamp. */ + if (upd != NULL && + __wt_timestamp_cmp(&upd->timestamp, &r->max_onpage_timestamp) > 0) + __wt_timestamp_set(&r->max_onpage_timestamp, &upd->timestamp); +#endif /* - * If there are no skipped updates, record that we're making progress. + * If the update we chose was a birthmark, or we are doing + * update-restore and we skipped a birthmark, the original on-page + * value must be retained. */ - if (*updp == first_txn_upd) - r->update_used = true; + if (upd != NULL && + (upd->type == WT_UPDATE_BIRTHMARK || + (F_ISSET(r, WT_REC_UPDATE_RESTORE) && skipped_birthmark))) + *updp = NULL; /* * Check if all updates on the page are visible. If not, it must stay @@ -1435,7 +1484,7 @@ __rec_txn_read(WT_SESSION_IMPL *session, WT_RECONCILE *r, #else timestampp = NULL; #endif - all_visible = *updp == first_txn_upd && !uncommitted && + all_visible = upd == first_txn_upd && !uncommitted && (F_ISSET(r, WT_REC_VISIBLE_ALL) ? __wt_txn_visible_all(session, max_txn, timestampp) : __wt_txn_visible(session, max_txn, timestampp)); @@ -1443,12 +1492,12 @@ __rec_txn_read(WT_SESSION_IMPL *session, WT_RECONCILE *r, if (all_visible) goto check_original_value; + r->leave_dirty = true; + if (F_ISSET(r, WT_REC_VISIBILITY_ERR)) WT_PANIC_RET(session, EINVAL, "reconciliation error, update not visible"); - r->leave_dirty = true; - /* * If not trying to evict the page, we know what we'll write and we're * done. @@ -1516,8 +1565,6 @@ check_original_value: * Returning an update means the original on-page value might be lost, * and that's a problem if there's a reader that needs it. There are * several cases: - * - any update with no backing record (because we will store an empty - * value on page and returning that is wrong). * - any update from a modify operation (because the modify has to be * applied to a stable update, not the new on-page update), * - any lookaside table eviction (because the backing disk image is @@ -1531,12 +1578,6 @@ check_original_value: WT_RET( __rec_append_orig_value(session, page, first_upd, vpack)); -#ifdef HAVE_TIMESTAMPS - if ((upd = *updp) != NULL && - __wt_timestamp_cmp(&upd->timestamp, &r->max_onpage_timestamp) > 0) - __wt_timestamp_set(&r->max_onpage_timestamp, &upd->timestamp); -#endif - return (0); } @@ -1754,6 +1795,7 @@ __rec_child_modify(WT_SESSION_IMPL *session, */ break; + case WT_REF_LIMBO: case WT_REF_LOOKASIDE: /* * On disk, with lookaside updates. @@ -3387,16 +3429,18 @@ __rec_split_write_supd(WT_SESSION_IMPL *session, r->supd_next = j; } -done: /* Track the oldest timestamp seen so far. */ - multi->page_las.las_skew_newest = r->las_skew_newest; - multi->page_las.las_max_txn = r->max_txn; - WT_ASSERT(session, r->max_txn != WT_TXN_NONE); +done: if (F_ISSET(r, WT_REC_LOOKASIDE)) { + /* Track the oldest lookaside timestamp seen so far. */ + multi->page_las.las_skew_newest = r->las_skew_newest; + multi->page_las.las_max_txn = r->max_txn; + WT_ASSERT(session, r->max_txn != WT_TXN_NONE); #ifdef HAVE_TIMESTAMPS - __wt_timestamp_set( - &multi->page_las.min_timestamp, &r->min_saved_timestamp); - __wt_timestamp_set( - &multi->page_las.onpage_timestamp, &r->max_onpage_timestamp); + __wt_timestamp_set(&multi->page_las.min_timestamp, + &r->min_saved_timestamp); + __wt_timestamp_set(&multi->page_las.onpage_timestamp, + &r->max_onpage_timestamp); #endif + } err: __wt_scr_free(session, &key); return (ret); @@ -4626,10 +4670,7 @@ record_loop: /* if (upd != NULL) { switch (upd->type) { - case WT_UPDATE_DELETED: - deleted = true; - break; - case WT_UPDATE_MODIFIED: + case WT_UPDATE_MODIFY: cbt->slot = WT_COL_SLOT(page, cip); WT_ERR(__wt_value_return_upd( session, cbt, upd, @@ -4642,6 +4683,9 @@ record_loop: /* data = upd->data; size = upd->size; break; + case WT_UPDATE_TOMBSTONE: + deleted = true; + break; WT_ILLEGAL_VALUE_ERR(session); } } else if (vpack->raw == WT_CELL_VALUE_OVFL_RM) { @@ -4867,10 +4911,7 @@ compare: /* deleted = true; else switch (upd->type) { - case WT_UPDATE_DELETED: - deleted = true; - break; - case WT_UPDATE_MODIFIED: + case WT_UPDATE_MODIFY: /* * Impossible slot, there's no backing * on-page item. @@ -4887,6 +4928,9 @@ compare: /* data = upd->data; size = upd->size; break; + case WT_UPDATE_TOMBSTONE: + deleted = true; + break; WT_ILLEGAL_VALUE_ERR(session); } @@ -5434,7 +5478,33 @@ __rec_row_leaf(WT_SESSION_IMPL *session, page, vpack, F_ISSET(r, WT_REC_EVICT))); switch (upd->type) { - case WT_UPDATE_DELETED: + case WT_UPDATE_MODIFY: + cbt->slot = WT_ROW_SLOT(page, rip); + WT_ERR(__wt_value_return_upd(session, cbt, upd, + F_ISSET(r, WT_REC_VISIBLE_ALL))); + WT_ERR(__rec_cell_build_val(session, r, + cbt->iface.value.data, + cbt->iface.value.size, (uint64_t)0)); + dictionary = true; + break; + case WT_UPDATE_STANDARD: + /* + * If no value, nothing needs to be copied. + * Otherwise, build the value's chunk from the + * update value. + */ + if (upd->size == 0) { + val->buf.data = NULL; + val->cell_len = + val->len = val->buf.size = 0; + } else { + WT_ERR(__rec_cell_build_val(session, r, + upd->data, upd->size, + (uint64_t)0)); + dictionary = true; + } + break; + case WT_UPDATE_TOMBSTONE: /* * If this key/value pair was deleted, we're * done. @@ -5473,33 +5543,6 @@ __rec_row_leaf(WT_SESSION_IMPL *session, /* Proceed with appended key/value pairs. */ goto leaf_insert; - case WT_UPDATE_MODIFIED: - cbt->slot = WT_ROW_SLOT(page, rip); - WT_ERR(__wt_value_return_upd( - session, cbt, upd, - F_ISSET(r, WT_REC_VISIBLE_ALL))); - WT_ERR(__rec_cell_build_val(session, r, - cbt->iface.value.data, - cbt->iface.value.size, (uint64_t)0)); - dictionary = true; - break; - case WT_UPDATE_STANDARD: - /* - * If no value, nothing needs to be copied. - * Otherwise, build the value's chunk from the - * update value. - */ - if (upd->size == 0) { - val->buf.data = NULL; - val->cell_len = - val->len = val->buf.size = 0; - } else { - WT_ERR(__rec_cell_build_val(session, r, - upd->data, upd->size, - (uint64_t)0)); - dictionary = true; - } - break; WT_ILLEGAL_VALUE_ERR(session); } } @@ -5591,6 +5634,7 @@ build: WT_ERR(__wt_dsk_cell_data_ref(session, WT_PAGE_ROW_LEAF, kpack, r->cur)); key_onpage_ovfl = false; + WT_NOT_READ(key_onpage_ovfl); } /* @@ -5688,17 +5732,14 @@ __rec_row_leaf_insert(WT_SESSION_IMPL *session, WT_RECONCILE *r, WT_INSERT *ins) } switch (upd->type) { - case WT_UPDATE_DELETED: - continue; - case WT_UPDATE_MODIFIED: + case WT_UPDATE_MODIFY: /* * Impossible slot, there's no backing on-page * item. */ cbt->slot = UINT32_MAX; WT_RET(__wt_value_return_upd( - session, cbt, upd, - F_ISSET(r, WT_REC_VISIBLE_ALL))); + session, cbt, upd, F_ISSET(r, WT_REC_VISIBLE_ALL))); WT_RET(__rec_cell_build_val(session, r, cbt->iface.value.data, cbt->iface.value.size, (uint64_t)0)); @@ -5711,6 +5752,8 @@ __rec_row_leaf_insert(WT_SESSION_IMPL *session, WT_RECONCILE *r, WT_INSERT *ins) r, upd->data, upd->size, (uint64_t)0)); break; + case WT_UPDATE_TOMBSTONE: + continue; WT_ILLEGAL_VALUE(session); } @@ -6088,7 +6131,7 @@ __rec_write_wrapup_err(WT_SESSION_IMPL *session, WT_RECONCILE *r, WT_PAGE *page) /* * __rec_las_wrapup -- - * Copy all of the saved updates into the database's lookaside buffer. + * Copy all of the saved updates into the database's lookaside table. */ static int __rec_las_wrapup(WT_SESSION_IMPL *session, WT_RECONCILE *r) diff --git a/src/third_party/wiredtiger/src/schema/schema_alter.c b/src/third_party/wiredtiger/src/schema/schema_alter.c index 3b52b4bd222..a957969e6cc 100644 --- a/src/third_party/wiredtiger/src/schema/schema_alter.c +++ b/src/third_party/wiredtiger/src/schema/schema_alter.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/schema/schema_create.c b/src/third_party/wiredtiger/src/schema/schema_create.c index 291778068f4..13298f56e9e 100644 --- a/src/third_party/wiredtiger/src/schema/schema_create.c +++ b/src/third_party/wiredtiger/src/schema/schema_create.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -396,18 +396,28 @@ __create_index(WT_SESSION_IMPL *session, "should be <table name>:<index name>: %s", name); /* - * Note: it would be better to get the table exclusive here, while + * Note: it would be better to keep the table exclusive here, while * changing its indexes. We don't because some operation we perform * below reacquire the table handle (such as opening a cursor on the - * table in order to fill the index). If we get the handle exclusive + * table in order to fill the index). If we keep the handle exclusive * here, those operations wanting ordinary access will conflict, - * leading to errors. + * leading to errors. At the same time, we don't want to allow + * table cursors that have already been fully opened to remain open + * across this call. * - * Instead, we rely on the global table lock to protect the set of - * available indexes. + * Temporarily getting the table exclusively serves the purpose + * of ensuring that cursors on the table that are already open + * must at least be closed before this call proceeds. */ tlen = (size_t)(idxname++ - tablename); if ((ret = __wt_schema_get_table( + session, tablename, tlen, true, WT_DHANDLE_EXCLUSIVE, &table)) != 0) + WT_RET_MSG(session, ret, + "Can't create an index for table: %.*s", + (int)tlen, tablename); + WT_RET(__wt_schema_release_table(session, table)); + + if ((ret = __wt_schema_get_table( session, tablename, tlen, true, 0, &table)) != 0) WT_RET_MSG(session, ret, "Can't create an index for a non-existent table: %.*s", diff --git a/src/third_party/wiredtiger/src/schema/schema_drop.c b/src/third_party/wiredtiger/src/schema/schema_drop.c index 4ee34f68a8c..12b577d0ea6 100644 --- a/src/third_party/wiredtiger/src/schema/schema_drop.c +++ b/src/third_party/wiredtiger/src/schema/schema_drop.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -124,7 +124,14 @@ __drop_table( * being reopened while it is being dropped. One issue is that the * WT_WITHOUT_LOCKS macro can drop and reacquire the global table lock, * avoiding deadlocks while waiting for LSM operation to quiesce. + * + * Temporarily getting the table exclusively serves the purpose + * of ensuring that cursors on the table that are already open + * must at least be closed before this call proceeds. */ + WT_ERR(__wt_schema_get_table_uri(session, uri, true, + WT_DHANDLE_EXCLUSIVE, &table)); + WT_ERR(__wt_schema_release_table(session, table)); WT_ERR(__wt_schema_get_table_uri(session, uri, true, 0, &table)); /* Drop the column groups. */ diff --git a/src/third_party/wiredtiger/src/schema/schema_list.c b/src/third_party/wiredtiger/src/schema/schema_list.c index 11257968b75..0f8806dd462 100644 --- a/src/third_party/wiredtiger/src/schema/schema_list.c +++ b/src/third_party/wiredtiger/src/schema/schema_list.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/schema/schema_open.c b/src/third_party/wiredtiger/src/schema/schema_open.c index 3b27c8300e2..0964483d57c 100644 --- a/src/third_party/wiredtiger/src/schema/schema_open.c +++ b/src/third_party/wiredtiger/src/schema/schema_open.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/schema/schema_plan.c b/src/third_party/wiredtiger/src/schema/schema_plan.c index 69ae3f4ce4b..abe76013e12 100644 --- a/src/third_party/wiredtiger/src/schema/schema_plan.c +++ b/src/third_party/wiredtiger/src/schema/schema_plan.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -97,7 +97,7 @@ __wt_schema_colcheck(WT_SESSION_IMPL *session, WT_RET(__pack_init(session, &pack, key_format)); for (kcols = 0; (ret = __pack_next(&pack, &pv)) == 0; kcols++) ; - WT_RET_TEST(ret != WT_NOTFOUND, ret); + WT_RET_NOTFOUND_OK(ret); WT_RET(__pack_init(session, &pack, value_format)); for (vcols = 0; (ret = __pack_next(&pack, &pv)) == 0; vcols++) diff --git a/src/third_party/wiredtiger/src/schema/schema_project.c b/src/third_party/wiredtiger/src/schema/schema_project.c index a682d63d002..9bc776bf8f2 100644 --- a/src/third_party/wiredtiger/src/schema/schema_project.c +++ b/src/third_party/wiredtiger/src/schema/schema_project.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -92,7 +92,7 @@ __wt_schema_project_in(WT_SESSION_IMPL *session, if (pv.type == 'S' || pv.type == 's') pv.u.s = ""; - len = __pack_size(session, &pv); + WT_RET(__pack_size(session, &pv, &len)); WT_RET(__wt_buf_grow(session, buf, buf->size + len)); p = (uint8_t *)buf->mem + buf->size; @@ -121,7 +121,7 @@ __wt_schema_project_in(WT_SESSION_IMPL *session, } old_len = (size_t)(next - p); - len = __pack_size(session, &pv); + WT_RET(__pack_size(session, &pv, &len)); offset = WT_PTRDIFF(p, buf->mem); WT_RET(__wt_buf_grow(session, buf, buf->size + len)); @@ -310,7 +310,7 @@ __wt_schema_project_slice(WT_SESSION_IMPL *session, WT_CURSOR **cp, if (pv.type == 'S' || pv.type == 's') pv.u.s = ""; - len = __pack_size(session, &pv); + WT_RET(__pack_size(session, &pv, &len)); WT_RET(__wt_buf_grow(session, buf, buf->size + len)); p = (uint8_t *)buf->data + buf->size; @@ -357,7 +357,7 @@ __wt_schema_project_slice(WT_SESSION_IMPL *session, WT_CURSOR **cp, __wt_tolower((u_char)vpv.type)); pv.u = vpv.u; - len = __pack_size(session, &pv); + WT_RET(__pack_size(session, &pv, &len)); offset = WT_PTRDIFF(p, buf->data); /* * Avoid growing the buffer if the value fits. @@ -463,7 +463,7 @@ __wt_schema_project_merge(WT_SESSION_IMPL *session, __wt_tolower((u_char)pv.type) == __wt_tolower((u_char)vpv.type)); vpv.u = pv.u; - len = __pack_size(session, &vpv); + WT_RET(__pack_size(session, &vpv, &len)); WT_RET(__wt_buf_grow(session, value, value->size + len)); vp = (uint8_t *)value->mem + value->size; diff --git a/src/third_party/wiredtiger/src/schema/schema_rename.c b/src/third_party/wiredtiger/src/schema/schema_rename.c index 9a61e39fe2e..0a8a80a214e 100644 --- a/src/third_party/wiredtiger/src/schema/schema_rename.c +++ b/src/third_party/wiredtiger/src/schema/schema_rename.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/schema/schema_stat.c b/src/third_party/wiredtiger/src/schema/schema_stat.c index 06518ffd50a..b86149bdb3a 100644 --- a/src/third_party/wiredtiger/src/schema/schema_stat.c +++ b/src/third_party/wiredtiger/src/schema/schema_stat.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/schema/schema_truncate.c b/src/third_party/wiredtiger/src/schema/schema_truncate.c index 7d5168fe72f..8eefdc1da5d 100644 --- a/src/third_party/wiredtiger/src/schema/schema_truncate.c +++ b/src/third_party/wiredtiger/src/schema/schema_truncate.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/schema/schema_util.c b/src/third_party/wiredtiger/src/schema/schema_util.c index f4625cd8ec4..ceec6db6cb5 100644 --- a/src/third_party/wiredtiger/src/schema/schema_util.c +++ b/src/third_party/wiredtiger/src/schema/schema_util.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/schema/schema_worker.c b/src/third_party/wiredtiger/src/schema/schema_worker.c index ce930c20818..1012749efc3 100644 --- a/src/third_party/wiredtiger/src/schema/schema_worker.c +++ b/src/third_party/wiredtiger/src/schema/schema_worker.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/session/session_api.c b/src/third_party/wiredtiger/src/session/session_api.c index d81735234a0..fd091cb5b13 100644 --- a/src/third_party/wiredtiger/src/session/session_api.c +++ b/src/third_party/wiredtiger/src/session/session_api.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -191,6 +191,24 @@ __session_close(WT_SESSION *wt_session, const char *config) /* Free transaction information. */ __wt_txn_destroy(session); + /* + * Close the file where we tracked long operations. Do this before + * releasing resources, as we do scratch buffer management when we flush + * optrack buffers to disk + */ + if (F_ISSET(conn, WT_CONN_OPTRACK)) { + if (session->optrackbuf_ptr > 0) { + WT_IGNORE_RET((int)__wt_optrack_flush_buffer(session)); + WT_IGNORE_RET(__wt_close(session, + &session->optrack_fh)); + /* Indicate that the file is closed */ + session->optrack_fh = NULL; + } + + /* Free the operation tracking buffer */ + __wt_free(session, session->optrack_buf); + } + /* Release common session resources. */ WT_TRET(__wt_session_release_resources(session)); @@ -489,7 +507,8 @@ __session_alter(WT_SESSION *wt_session, const char *uri, const char *config) ret = __wt_schema_worker(session, uri, __wt_alter, NULL, cfg, WT_BTREE_ALTER | WT_DHANDLE_EXCLUSIVE))); -err: if (ret != 0) +err: + if (ret != 0) WT_STAT_CONN_INCR(session, session_table_alter_fail); else WT_STAT_CONN_INCR(session, session_table_alter_success); @@ -580,7 +599,8 @@ __session_create(WT_SESSION *wt_session, const char *uri, const char *config) ret = __wt_session_create(session, uri, config); -err: if (ret != 0) +err: + if (ret != 0) WT_STAT_CONN_INCR(session, session_table_create_fail); else WT_STAT_CONN_INCR(session, session_table_create_success); @@ -731,7 +751,8 @@ __session_rebalance(WT_SESSION *wt_session, const char *uri, const char *config) ret = __wt_schema_worker(session, uri, __wt_bt_rebalance, NULL, cfg, WT_DHANDLE_EXCLUSIVE | WT_BTREE_REBALANCE))); -err: if (ret != 0) +err: + if (ret != 0) WT_STAT_CONN_INCR(session, session_table_rebalance_fail); else WT_STAT_CONN_INCR(session, session_table_rebalance_success); @@ -782,8 +803,8 @@ __session_rename(WT_SESSION *wt_session, WT_WITH_SCHEMA_LOCK(session, WT_WITH_TABLE_WRITE_LOCK(session, ret = __wt_schema_rename(session, uri, newuri, cfg)))); - -err: if (ret != 0) +err: + if (ret != 0) WT_STAT_CONN_INCR(session, session_table_rename_fail); else WT_STAT_CONN_INCR(session, session_table_rename_success); @@ -887,7 +908,8 @@ __session_drop(WT_SESSION *wt_session, const char *uri, const char *config) ret = __wt_schema_drop(session, uri, cfg))); } -err: if (ret != 0) +err: + if (ret != 0) WT_STAT_CONN_INCR(session, session_table_drop_fail); else WT_STAT_CONN_INCR(session, session_table_drop_success); @@ -1078,7 +1100,8 @@ __session_salvage(WT_SESSION *wt_session, const char *uri, const char *config) ret = __wt_schema_worker(session, uri, __wt_salvage, NULL, cfg, WT_DHANDLE_EXCLUSIVE | WT_BTREE_SALVAGE))); -err: if (ret != 0) +err: + if (ret != 0) WT_STAT_CONN_INCR(session, session_table_salvage_fail); else WT_STAT_CONN_INCR(session, session_table_salvage_success); @@ -1395,7 +1418,8 @@ __session_verify(WT_SESSION *wt_session, const char *uri, const char *config) ret = __wt_schema_worker(session, uri, __wt_verify, NULL, cfg, WT_DHANDLE_EXCLUSIVE | WT_BTREE_VERIFY))); -err: if (ret != 0) +err: + if (ret != 0) WT_STAT_CONN_INCR(session, session_table_verify_fail); else WT_STAT_CONN_INCR(session, session_table_verify_success); @@ -1443,7 +1467,9 @@ __session_commit_transaction(WT_SESSION *wt_session, const char *config) txn = &session->txn; if (F_ISSET(txn, WT_TXN_ERROR) && txn->mod_count != 0) WT_ERR_MSG(session, EINVAL, - "failed transaction requires rollback"); + "failed transaction requires rollback%s%s", + txn->rollback_reason == NULL ? "" : ": ", + txn->rollback_reason == NULL ? "" : txn->rollback_reason); if (ret == 0) ret = __wt_txn_commit(session, cfg); @@ -1552,13 +1578,13 @@ __transaction_sync_run_chk(WT_SESSION_IMPL *session) static int __session_transaction_sync(WT_SESSION *wt_session, const char *config) { - struct timespec now, start; WT_CONFIG_ITEM cval; WT_CONNECTION_IMPL *conn; WT_DECL_RET; WT_LOG *log; WT_SESSION_IMPL *session; uint64_t remaining_usec, timeout_ms, waited_ms; + uint64_t time_start, time_stop; session = (WT_SESSION_IMPL *)wt_session; SESSION_API_CALL(session, transaction_sync, config, cfg); @@ -1600,18 +1626,18 @@ __session_transaction_sync(WT_SESSION *wt_session, const char *config) if (timeout_ms == 0) WT_ERR(ETIMEDOUT); - __wt_epoch(session, &start); /* * Keep checking the LSNs until we find it is stable or we reach * our timeout, or there's some other reason to quit. */ + time_start = __wt_clock(session); while (__wt_log_cmp(&session->bg_sync_lsn, &log->sync_lsn) > 0) { if (!__transaction_sync_run_chk(session)) WT_ERR(ETIMEDOUT); __wt_cond_signal(session, conn->log_file_cond); - __wt_epoch(session, &now); - waited_ms = WT_TIMEDIFF_MS(now, start); + time_stop = __wt_clock(session); + waited_ms = WT_CLOCKDIFF_MS(time_stop, time_start); if (waited_ms < timeout_ms) { remaining_usec = (timeout_ms - waited_ms) * WT_THOUSAND; __wt_cond_wait(session, log->log_sync_cond, @@ -1755,6 +1781,19 @@ __wt_session_strerror(WT_SESSION *wt_session, int error) } /* + * __wt_session_breakpoint -- + * A place to put a breakpoint, if you need one, or call some check + * code. + */ +int +__wt_session_breakpoint(WT_SESSION *wt_session) +{ + WT_UNUSED(wt_session); + + return (0); +} + +/* * __open_session -- * Allocate a session handle. */ @@ -1791,7 +1830,8 @@ __open_session(WT_CONNECTION_IMPL *conn, __session_checkpoint, __session_snapshot, __session_transaction_pinned_range, - __session_transaction_sync + __session_transaction_sync, + __wt_session_breakpoint }, stds_readonly = { NULL, NULL, @@ -1820,7 +1860,8 @@ __open_session(WT_CONNECTION_IMPL *conn, __session_checkpoint_readonly, __session_snapshot, __session_transaction_pinned_range, - __session_transaction_sync_readonly + __session_transaction_sync_readonly, + __wt_session_breakpoint }; WT_DECL_RET; WT_SESSION_IMPL *session, *session_ret; @@ -1847,9 +1888,9 @@ __open_session(WT_CONNECTION_IMPL *conn, break; if (i == conn->session_size) WT_ERR_MSG(session, WT_ERROR, - "out of sessions, only configured to support %" PRIu32 - " sessions (including %d additional internal sessions)", - conn->session_size, WT_EXTRA_INTERNAL_SESSIONS); + "out of sessions, configured for %" PRIu32 " (including " + "internal sessions)", + conn->session_size); /* * If the active session count is increasing, update it. We don't worry @@ -1904,6 +1945,12 @@ __open_session(WT_CONNECTION_IMPL *conn, /* Cache the offset of this session's statistics bucket. */ session_ret->stat_bucket = WT_STATS_SLOT_ID(session); + /* Allocate the buffer for operation tracking */ + if (F_ISSET(conn, WT_CONN_OPTRACK)) { + WT_ERR(__wt_malloc( + session, WT_OPTRACK_BUFSIZE, &session_ret->optrack_buf)); + session_ret->optrackbuf_ptr = 0; + } /* * Configuration: currently, the configuration for open_session is the * same as session.reconfigure, so use that function. diff --git a/src/third_party/wiredtiger/src/session/session_compact.c b/src/third_party/wiredtiger/src/session/session_compact.c index aa2f1bc3bd8..96761ff8745 100644 --- a/src/third_party/wiredtiger/src/session/session_compact.c +++ b/src/third_party/wiredtiger/src/session/session_compact.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/session/session_dhandle.c b/src/third_party/wiredtiger/src/session/session_dhandle.c index accdf615e37..caa775686cf 100644 --- a/src/third_party/wiredtiger/src/session/session_dhandle.c +++ b/src/third_party/wiredtiger/src/session/session_dhandle.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/session/session_salvage.c b/src/third_party/wiredtiger/src/session/session_salvage.c index 5a67bd1f7ac..0a5181dc218 100644 --- a/src/third_party/wiredtiger/src/session/session_salvage.c +++ b/src/third_party/wiredtiger/src/session/session_salvage.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/support/cond_auto.c b/src/third_party/wiredtiger/src/support/cond_auto.c index 2d43eb3bf79..61db334de24 100644 --- a/src/third_party/wiredtiger/src/support/cond_auto.c +++ b/src/third_party/wiredtiger/src/support/cond_auto.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/support/crypto.c b/src/third_party/wiredtiger/src/support/crypto.c index 6208d83b0f2..db59d3c6ccb 100644 --- a/src/third_party/wiredtiger/src/support/crypto.c +++ b/src/third_party/wiredtiger/src/support/crypto.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/support/err.c b/src/third_party/wiredtiger/src/support/err.c index a3b87b6edde..c7cfa0c3932 100644 --- a/src/third_party/wiredtiger/src/support/err.c +++ b/src/third_party/wiredtiger/src/support/err.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/support/generation.c b/src/third_party/wiredtiger/src/support/generation.c index 8bdbcc078fb..34f3ab6afb9 100644 --- a/src/third_party/wiredtiger/src/support/generation.c +++ b/src/third_party/wiredtiger/src/support/generation.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/support/global.c b/src/third_party/wiredtiger/src/support/global.c index fd0ecbe1b65..d1271e0d427 100644 --- a/src/third_party/wiredtiger/src/support/global.c +++ b/src/third_party/wiredtiger/src/support/global.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -41,6 +41,68 @@ __wt_endian_check(void) } /* + * __global_calibrate_ticks -- + * Calibrate a ratio from rdtsc ticks to nanoseconds. + */ +static void +__global_calibrate_ticks(void) +{ + /* + * Default to using __wt_epoch until we have a good value for the ratio. + */ + __wt_process.tsc_nsec_ratio = WT_TSC_DEFAULT_RATIO; + __wt_process.use_epochtime = true; + +#if defined (__i386) || defined (__amd64) + { + struct timespec start, stop; + double ratio; + uint64_t diff_nsec, diff_tsc, min_nsec, min_tsc; + uint64_t tries, tsc_start, tsc_stop; + volatile uint64_t i; + + /* + * Run this calibration loop a few times to make sure we get a + * reading that does not have a potential scheduling shift in it. + * The inner loop is CPU intensive but a scheduling change in the + * middle could throw off calculations. Take the minimum amount + * of time and compute the ratio. + */ + min_nsec = min_tsc = UINT64_MAX; + for (tries = 0; tries < 3; ++tries) { + /* This needs to be CPU intensive and large enough. */ + __wt_epoch(NULL, &start); + tsc_start = __wt_rdtsc(); + for (i = 0; i < 100 * WT_MILLION; i++) + ; + tsc_stop = __wt_rdtsc(); + __wt_epoch(NULL, &stop); + diff_nsec = WT_TIMEDIFF_NS(stop, start); + diff_tsc = tsc_stop - tsc_start; + + /* If the clock didn't tick over, we don't have a sample. */ + if (diff_nsec == 0 || diff_tsc == 0) + continue; + min_nsec = WT_MIN(min_nsec, diff_nsec); + min_tsc = WT_MIN(min_tsc, diff_tsc); + } + + /* + * Only use rdtsc if we got a good reading. One reason this might fail + * is that the system's clock granularity is not fine-grained enough. + */ + if (min_nsec != UINT64_MAX) { + ratio = (double)min_tsc / (double)min_nsec; + if (ratio > DBL_EPSILON) { + __wt_process.tsc_nsec_ratio = ratio; + __wt_process.use_epochtime = false; + } + } + } +#endif +} + +/* * __wt_global_once -- * Global initialization, run once. */ @@ -56,13 +118,9 @@ __wt_global_once(void) } __wt_checksum_init(); + __global_calibrate_ticks(); TAILQ_INIT(&__wt_process.connqh); - -#ifdef HAVE_DIAGNOSTIC - /* Load debugging code the compiler might optimize out. */ - __wt_breakpoint(); -#endif } /* @@ -91,19 +149,3 @@ __wt_library_init(void) } return (__wt_pthread_once_failed); } - -#ifdef HAVE_DIAGNOSTIC -/* - * __wt_breakpoint -- - * A simple place to put a breakpoint, if you need one. - */ -void -__wt_breakpoint(void) -{ - /* - * Yield the processor (just to keep the compiler from optimizing the - * function out). - */ - __wt_yield(); -} -#endif diff --git a/src/third_party/wiredtiger/src/support/hash_city.c b/src/third_party/wiredtiger/src/support/hash_city.c index e14368d3529..402e4f08394 100644 --- a/src/third_party/wiredtiger/src/support/hash_city.c +++ b/src/third_party/wiredtiger/src/support/hash_city.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/src/support/hash_fnv.c b/src/third_party/wiredtiger/src/support/hash_fnv.c index aad698229fd..c544e2d6096 100644 --- a/src/third_party/wiredtiger/src/support/hash_fnv.c +++ b/src/third_party/wiredtiger/src/support/hash_fnv.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/src/support/hazard.c b/src/third_party/wiredtiger/src/support/hazard.c index d2d2e855088..815c876f444 100644 --- a/src/third_party/wiredtiger/src/support/hazard.c +++ b/src/third_party/wiredtiger/src/support/hazard.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -72,6 +72,7 @@ __wt_hazard_set(WT_SESSION_IMPL *session, WT_REF *ref, bool *busyp ) { WT_HAZARD *hp; + uint32_t current_state; *busyp = false; @@ -84,7 +85,8 @@ __wt_hazard_set(WT_SESSION_IMPL *session, WT_REF *ref, bool *busyp * eviction and splits, we re-check it after a barrier to make sure * we have a valid reference. */ - if (ref->state != WT_REF_MEM) { + current_state = ref->state; + if (current_state != WT_REF_LIMBO && current_state != WT_REF_MEM) { *busyp = true; return (0); } @@ -132,8 +134,8 @@ __wt_hazard_set(WT_SESSION_IMPL *session, WT_REF *ref, bool *busyp * Do the dance: * * The memory location which makes a page "real" is the WT_REF's state - * of WT_REF_MEM, which can be set to WT_REF_LOCKED at any time by the - * page eviction server. + * of WT_REF_LIMBO or WT_REF_MEM, which can be set to WT_REF_LOCKED + * at any time by the page eviction server. * * Add the WT_REF reference to the session's hazard list and flush the * write, then see if the page's state is still valid. If so, we can @@ -152,9 +154,10 @@ __wt_hazard_set(WT_SESSION_IMPL *session, WT_REF *ref, bool *busyp /* * Check if the page state is still valid, where valid means a - * state of WT_REF_MEM. + * state of WT_REF_LIMBO or WT_REF_MEM. */ - if (ref->state == WT_REF_MEM) { + current_state = ref->state; + if (current_state == WT_REF_LIMBO || current_state == WT_REF_MEM) { ++session->nhazard; /* diff --git a/src/third_party/wiredtiger/src/support/hex.c b/src/third_party/wiredtiger/src/support/hex.c index 58730b1505b..69b33bad368 100644 --- a/src/third_party/wiredtiger/src/support/hex.c +++ b/src/third_party/wiredtiger/src/support/hex.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/support/huffman.c b/src/third_party/wiredtiger/src/support/huffman.c index e0e8f47e45c..f784352c981 100644 --- a/src/third_party/wiredtiger/src/support/huffman.c +++ b/src/third_party/wiredtiger/src/support/huffman.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/support/modify.c b/src/third_party/wiredtiger/src/support/modify.c index 42541ffc3ef..0d75ffa5679 100644 --- a/src/third_party/wiredtiger/src/support/modify.c +++ b/src/third_party/wiredtiger/src/support/modify.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -56,11 +56,16 @@ __wt_modify_pack(WT_SESSION_IMPL *session, * Apply a single modify structure change to the buffer. */ static int -__modify_apply_one(WT_SESSION_IMPL *session, WT_ITEM *value, +__modify_apply_one(WT_SESSION_IMPL *session, WT_CURSOR *cursor, size_t data_size, size_t offset, size_t size, const uint8_t *data) { + WT_ITEM *value; size_t len; uint8_t *from, *to; + bool sformat; + + value = &cursor->value; + sformat = cursor->value_format[0] == 'S'; /* * Grow the buffer to the maximum size we'll need. This is pessimistic @@ -89,15 +94,26 @@ __modify_apply_one(WT_SESSION_IMPL *session, WT_ITEM *value, } /* + * Decrement the size to discard the trailing nul (done after growing + * the buffer to ensure it can be restored without further checking). + */ + if (sformat) + --value->size; + + /* * If appending bytes past the end of the value, initialize gap bytes * and copy the new bytes into place. */ if (value->size <= offset) { if (value->size < offset) - memset((uint8_t *)value->data + - value->size, 0, offset - value->size); + memset((uint8_t *)value->data + value->size, + sformat ? ' ' : 0, offset - value->size); memmove((uint8_t *)value->data + offset, data, data_size); value->size = offset + data_size; + + /* Restore the trailing nul. */ + if (sformat) + ((char *)value->data)[value->size++] = '\0'; return (0); } @@ -149,6 +165,10 @@ __modify_apply_one(WT_SESSION_IMPL *session, WT_ITEM *value, value->size += (data_size - size); } + /* Restore the trailing nul. */ + if (sformat) + ((char *)value->data)[value->size++] = '\0'; + return (0); } @@ -158,14 +178,14 @@ __modify_apply_one(WT_SESSION_IMPL *session, WT_ITEM *value, * interface. */ int -__wt_modify_apply_api( - WT_SESSION_IMPL *session, WT_ITEM *value, WT_MODIFY *entries, int nentries) +__wt_modify_apply_api(WT_SESSION_IMPL *session, + WT_CURSOR *cursor, WT_MODIFY *entries, int nentries) WT_GCC_FUNC_ATTRIBUTE((visibility("default"))) { int i; for (i = 0; i < nentries; ++i) - WT_RET(__modify_apply_one(session, value, entries[i].data.size, + WT_RET(__modify_apply_one(session, cursor, entries[i].data.size, entries[i].offset, entries[i].size, entries[i].data.data)); return (0); @@ -176,7 +196,8 @@ __wt_modify_apply_api( * Apply a single set of WT_MODIFY changes to a buffer. */ int -__wt_modify_apply(WT_SESSION_IMPL *session, WT_ITEM *value, const void *modify) +__wt_modify_apply( + WT_SESSION_IMPL *session, WT_CURSOR *cursor, const void *modify) { size_t data_size, nentries, offset, size; const size_t *p; @@ -198,7 +219,7 @@ __wt_modify_apply(WT_SESSION_IMPL *session, WT_ITEM *value, const void *modify) memcpy(&offset, p++, sizeof(size_t)); memcpy(&size, p++, sizeof(size_t)); WT_RET(__modify_apply_one( - session, value, data_size, offset, size, data)); + session, cursor, data_size, offset, size, data)); } return (0); diff --git a/src/third_party/wiredtiger/src/support/mtx_rw.c b/src/third_party/wiredtiger/src/support/mtx_rw.c index 1e7a8a1b4df..572592b9fbc 100644 --- a/src/third_party/wiredtiger/src/support/mtx_rw.c +++ b/src/third_party/wiredtiger/src/support/mtx_rw.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. @@ -165,8 +165,8 @@ __read_blocked(WT_SESSION_IMPL *session) void __wt_readlock(WT_SESSION_IMPL *session, WT_RWLOCK *l) { - struct timespec enter, leave; WT_RWLOCK new, old; + uint64_t time_start, time_stop; int64_t **stats; int16_t writers_active; uint8_t ticket; @@ -176,6 +176,7 @@ __wt_readlock(WT_SESSION_IMPL *session, WT_RWLOCK *l) WT_STAT_CONN_INCR(session, rwlock_read); stats = (int64_t **)S2C(session)->stats; set_stats = (l->stat_read_count_off != -1 && WT_STAT_ENABLED(session)); + time_start = time_stop = 0; if (set_stats) stats[session->stat_bucket][l->stat_read_count_off]++; @@ -236,7 +237,7 @@ stall: __wt_cond_wait(session, } if (set_stats) - __wt_epoch(session, &enter); + time_start = __wt_clock(session); /* Wait for our group to start. */ for (pause_cnt = 0; ticket != l->u.s.current; pause_cnt++) { if (pause_cnt < 1000) @@ -251,13 +252,13 @@ stall: __wt_cond_wait(session, } } if (set_stats) { - __wt_epoch(session, &leave); + time_stop = __wt_clock(session); if (F_ISSET(session, WT_SESSION_INTERNAL)) stats[session->stat_bucket][l->stat_int_usecs_off] += - (int64_t)WT_TIMEDIFF_US(leave, enter); + (int64_t)WT_CLOCKDIFF_US(time_stop, time_start); else stats[session->stat_bucket][l->stat_app_usecs_off] += - (int64_t)WT_TIMEDIFF_US(leave, enter); + (int64_t)WT_CLOCKDIFF_US(time_stop, time_start); } /* @@ -362,8 +363,8 @@ __write_blocked(WT_SESSION_IMPL *session) void __wt_writelock(WT_SESSION_IMPL *session, WT_RWLOCK *l) { - struct timespec enter, leave; WT_RWLOCK new, old; + uint64_t time_start, time_stop; int64_t **stats; uint8_t ticket; int pause_cnt; @@ -372,6 +373,7 @@ __wt_writelock(WT_SESSION_IMPL *session, WT_RWLOCK *l) WT_STAT_CONN_INCR(session, rwlock_write); stats = (int64_t **)S2C(session)->stats; set_stats = (l->stat_write_count_off != -1 && WT_STAT_ENABLED(session)); + time_start = time_stop = 0; if (set_stats) stats[session->stat_bucket][l->stat_write_count_off]++; @@ -405,7 +407,7 @@ __wt_writelock(WT_SESSION_IMPL *session, WT_RWLOCK *l) * we have the lock. */ if (set_stats) - __wt_epoch(session, &enter); + time_start = __wt_clock(session); for (pause_cnt = 0, old.u.v = l->u.v; ticket != old.u.s.current || old.u.s.readers_active != 0; pause_cnt++, old.u.v = l->u.v) { @@ -421,13 +423,13 @@ __wt_writelock(WT_SESSION_IMPL *session, WT_RWLOCK *l) } } if (set_stats) { - __wt_epoch(session, &leave); + time_stop = __wt_clock(session); if (F_ISSET(session, WT_SESSION_INTERNAL)) stats[session->stat_bucket][l->stat_int_usecs_off] += - (int64_t)WT_TIMEDIFF_US(leave, enter); + (int64_t)WT_CLOCKDIFF_US(time_stop, time_start); else stats[session->stat_bucket][l->stat_app_usecs_off] += - (int64_t)WT_TIMEDIFF_US(leave, enter); + (int64_t)WT_CLOCKDIFF_US(time_stop, time_start); } /* diff --git a/src/third_party/wiredtiger/src/support/pow.c b/src/third_party/wiredtiger/src/support/pow.c index 2fb193afca8..60012feb88c 100644 --- a/src/third_party/wiredtiger/src/support/pow.c +++ b/src/third_party/wiredtiger/src/support/pow.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/src/support/rand.c b/src/third_party/wiredtiger/src/support/rand.c index 64ed2341eeb..9f76efb28e2 100644 --- a/src/third_party/wiredtiger/src/support/rand.c +++ b/src/third_party/wiredtiger/src/support/rand.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/src/support/scratch.c b/src/third_party/wiredtiger/src/support/scratch.c index 8596f4e1fe6..2ead79a1c1c 100644 --- a/src/third_party/wiredtiger/src/support/scratch.c +++ b/src/third_party/wiredtiger/src/support/scratch.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -15,6 +15,7 @@ */ int __wt_buf_grow_worker(WT_SESSION_IMPL *session, WT_ITEM *buf, size_t size) + WT_GCC_FUNC_ATTRIBUTE((visibility("default"))) { size_t offset; bool copy_data; @@ -68,6 +69,7 @@ __wt_buf_grow_worker(WT_SESSION_IMPL *session, WT_ITEM *buf, size_t size) int __wt_buf_fmt(WT_SESSION_IMPL *session, WT_ITEM *buf, const char *fmt, ...) WT_GCC_FUNC_ATTRIBUTE((format (printf, 3, 4))) + WT_GCC_FUNC_ATTRIBUTE((visibility("default"))) { WT_DECL_RET; size_t len; @@ -102,6 +104,7 @@ __wt_buf_fmt(WT_SESSION_IMPL *session, WT_ITEM *buf, const char *fmt, ...) int __wt_buf_catfmt(WT_SESSION_IMPL *session, WT_ITEM *buf, const char *fmt, ...) WT_GCC_FUNC_ATTRIBUTE((format (printf, 3, 4))) + WT_GCC_FUNC_ATTRIBUTE((visibility("default"))) { WT_DECL_RET; size_t len, space; @@ -141,8 +144,8 @@ __wt_buf_catfmt(WT_SESSION_IMPL *session, WT_ITEM *buf, const char *fmt, ...) /* * __wt_buf_set_printable -- - * Set the contents of the buffer to a printable representation of a - * byte string. + * Set the contents of the buffer to a printable representation of a byte + * string. */ const char * __wt_buf_set_printable( @@ -156,6 +159,85 @@ __wt_buf_set_printable( } /* + * __wt_buf_set_printable_format -- + * Set the contents of the buffer to a printable representation of a byte + * string, based on a format. + */ +const char * +__wt_buf_set_printable_format(WT_SESSION_IMPL *session, + const void *buffer, size_t size, const char *format, WT_ITEM *buf) +{ + WT_DECL_ITEM(tmp); + WT_DECL_PACK_VALUE(pv); + WT_DECL_RET; + WT_PACK pack; + const uint8_t *p, *end; + const char *retp, *sep; + + p = (const uint8_t *)buffer; + end = p + size; + + WT_ERR(__wt_buf_init(session, buf, 0)); + + WT_ERR(__pack_init(session, &pack, format)); + for (sep = ""; (ret = __pack_next(&pack, &pv)) == 0;) { + WT_ERR(__unpack_read(session, &pv, &p, (size_t)(end - p))); + switch (pv.type) { + case 'x': + break; + case 's': + case 'S': + WT_ERR(__wt_buf_catfmt( + session, buf, "%s%s", sep, pv.u.s)); + sep = ","; + break; + case 'U': + case 'u': + if (pv.u.item.size == 0) + break; + + if (tmp == NULL) + WT_ERR(__wt_scr_alloc(session, 0, &tmp)); + WT_ERR(__wt_buf_catfmt(session, buf, "%s%s", + sep, __wt_buf_set_printable( + session, pv.u.item.data, pv.u.item.size, tmp))); + break; + case 'b': + case 'h': + case 'i': + case 'l': + case 'q': + WT_ERR(__wt_buf_catfmt( + session, buf, "%s%" PRId64, sep, pv.u.i)); + sep = ","; + break; + case 'B': + case 't': + case 'H': + case 'I': + case 'L': + case 'Q': + case 'r': + case 'R': + WT_ERR(__wt_buf_catfmt( + session, buf, "%s%" PRIu64, sep, pv.u.u)); + sep = ","; + break; + WT_ILLEGAL_VALUE_ERR(session); + } + } + WT_ERR_NOTFOUND_OK(ret); + +err: __wt_scr_free(session, &tmp); + if (ret == 0) + return ((const char *)buf->data); + + retp = "failed to create printable output"; + __wt_err(session, ret, "%s: %s", __func__, retp); + return (retp); +} + +/* * __wt_buf_set_size -- * Set the contents of the buffer to a printable representation of a * byte size. @@ -207,6 +289,7 @@ __wt_scr_alloc_func(WT_SESSION_IMPL *session, size_t size, WT_ITEM **scratchp , const char *file, int line #endif ) + WT_GCC_FUNC_ATTRIBUTE((visibility("default"))) { WT_DECL_RET; WT_ITEM *buf, **p, **best, **slot; diff --git a/src/third_party/wiredtiger/src/support/stat.c b/src/third_party/wiredtiger/src/support/stat.c index eeb1f2d26e9..926176d6024 100644 --- a/src/third_party/wiredtiger/src/support/stat.c +++ b/src/third_party/wiredtiger/src/support/stat.c @@ -832,6 +832,8 @@ static const char * const __stats_connection_desc[] = { "cache: pages queued for urgent eviction during walk", "cache: pages read into cache", "cache: pages read into cache requiring lookaside entries", + "cache: pages read into cache skipping older lookaside entries", + "cache: pages read into cache with skipped lookaside entries needed later", "cache: pages requested from the cache", "cache: pages seen by eviction walk", "cache: pages selected for eviction unable to be evicted", @@ -881,6 +883,10 @@ static const char * const __stats_connection_desc[] = { "lock: checkpoint lock acquisitions", "lock: checkpoint lock application thread wait time (usecs)", "lock: checkpoint lock internal thread wait time (usecs)", + "lock: commit timestamp queue lock application thread time waiting for the dhandle lock (usecs)", + "lock: commit timestamp queue lock internal thread time waiting for the dhandle lock (usecs)", + "lock: commit timestamp queue read lock acquisitions", + "lock: commit timestamp queue write lock acquisitions", "lock: dhandle lock application thread time waiting for the dhandle lock (usecs)", "lock: dhandle lock internal thread time waiting for the dhandle lock (usecs)", "lock: dhandle read lock acquisitions", @@ -888,6 +894,10 @@ static const char * const __stats_connection_desc[] = { "lock: metadata lock acquisitions", "lock: metadata lock application thread wait time (usecs)", "lock: metadata lock internal thread wait time (usecs)", + "lock: read timestamp queue lock application thread time waiting for the dhandle lock (usecs)", + "lock: read timestamp queue lock internal thread time waiting for the dhandle lock (usecs)", + "lock: read timestamp queue read lock acquisitions", + "lock: read timestamp queue write lock acquisitions", "lock: schema lock acquisitions", "lock: schema lock application thread wait time (usecs)", "lock: schema lock internal thread wait time (usecs)", @@ -895,6 +905,10 @@ static const char * const __stats_connection_desc[] = { "lock: table lock internal thread time waiting for the table lock (usecs)", "lock: table read lock acquisitions", "lock: table write lock acquisitions", + "lock: txn global lock application thread time waiting for the dhandle lock (usecs)", + "lock: txn global lock internal thread time waiting for the dhandle lock (usecs)", + "lock: txn global read lock acquisitions", + "lock: txn global write lock acquisitions", "log: busy returns attempting to switch slots", "log: force checkpoint calls slept", "log: log bytes of payload data", @@ -941,6 +955,28 @@ static const char * const __stats_connection_desc[] = { "log: total size of compressed records", "log: written slots coalesced", "log: yields waiting for previous log file close", + "perf: file system read latency histogram (bucket 1) - 10-49ms", + "perf: file system read latency histogram (bucket 2) - 50-99ms", + "perf: file system read latency histogram (bucket 3) - 100-249ms", + "perf: file system read latency histogram (bucket 4) - 250-499ms", + "perf: file system read latency histogram (bucket 5) - 500-999ms", + "perf: file system read latency histogram (bucket 6) - 1000ms+", + "perf: file system write latency histogram (bucket 1) - 10-49ms", + "perf: file system write latency histogram (bucket 2) - 50-99ms", + "perf: file system write latency histogram (bucket 3) - 100-249ms", + "perf: file system write latency histogram (bucket 4) - 250-499ms", + "perf: file system write latency histogram (bucket 5) - 500-999ms", + "perf: file system write latency histogram (bucket 6) - 1000ms+", + "perf: operation read latency histogram (bucket 1) - 100-249us", + "perf: operation read latency histogram (bucket 2) - 250-499us", + "perf: operation read latency histogram (bucket 3) - 500-999us", + "perf: operation read latency histogram (bucket 4) - 1000-9999us", + "perf: operation read latency histogram (bucket 5) - 10000us+", + "perf: operation write latency histogram (bucket 1) - 100-249us", + "perf: operation write latency histogram (bucket 2) - 250-499us", + "perf: operation write latency histogram (bucket 3) - 500-999us", + "perf: operation write latency histogram (bucket 4) - 1000-9999us", + "perf: operation write latency histogram (bucket 5) - 10000us+", "reconciliation: fast-path pages deleted", "reconciliation: page reconciliation calls", "reconciliation: page reconciliation calls for eviction", @@ -986,8 +1022,24 @@ static const char * const __stats_connection_desc[] = { "thread-yield: page delete rollback time sleeping for state change (usecs)", "thread-yield: page reconciliation yielded due to child modification", "thread-yield: tree descend one level yielded for split page index update", + "transaction: commit timestamp queue insert to empty", + "transaction: commit timestamp queue inserts to head", + "transaction: commit timestamp queue inserts total", + "transaction: commit timestamp queue length", "transaction: number of named snapshots created", "transaction: number of named snapshots dropped", + "transaction: query timestamp calls", + "transaction: read timestamp queue insert to empty", + "transaction: read timestamp queue inserts to head", + "transaction: read timestamp queue inserts total", + "transaction: read timestamp queue length", + "transaction: set timestamp calls", + "transaction: set timestamp commit calls", + "transaction: set timestamp commit updates", + "transaction: set timestamp oldest calls", + "transaction: set timestamp oldest updates", + "transaction: set timestamp stable calls", + "transaction: set timestamp stable updates", "transaction: transaction begins", "transaction: transaction checkpoint currently running", "transaction: transaction checkpoint generation", @@ -1008,13 +1060,7 @@ static const char * const __stats_connection_desc[] = { "transaction: transaction range of timestamps currently pinned", "transaction: transaction range of timestamps pinned by the oldest timestamp", "transaction: transaction sync calls", - "transaction: transactions commit timestamp queue inserts to head", - "transaction: transactions commit timestamp queue inserts total", - "transaction: transactions commit timestamp queue length", "transaction: transactions committed", - "transaction: transactions read timestamp queue inserts to head", - "transaction: transactions read timestamp queue inserts total", - "transaction: transactions read timestamp queue length", "transaction: transactions rolled back", "transaction: update conflicts", }; @@ -1166,6 +1212,8 @@ __wt_stat_connection_clear_single(WT_CONNECTION_STATS *stats) stats->cache_eviction_pages_queued_oldest = 0; stats->cache_read = 0; stats->cache_read_lookaside = 0; + stats->cache_read_lookaside_skipped = 0; + stats->cache_read_lookaside_delay = 0; stats->cache_pages_requested = 0; stats->cache_eviction_pages_seen = 0; stats->cache_eviction_fail = 0; @@ -1215,6 +1263,10 @@ __wt_stat_connection_clear_single(WT_CONNECTION_STATS *stats) stats->lock_checkpoint_count = 0; stats->lock_checkpoint_wait_application = 0; stats->lock_checkpoint_wait_internal = 0; + stats->lock_commit_timestamp_wait_application = 0; + stats->lock_commit_timestamp_wait_internal = 0; + stats->lock_commit_timestamp_read_count = 0; + stats->lock_commit_timestamp_write_count = 0; stats->lock_dhandle_wait_application = 0; stats->lock_dhandle_wait_internal = 0; stats->lock_dhandle_read_count = 0; @@ -1222,6 +1274,10 @@ __wt_stat_connection_clear_single(WT_CONNECTION_STATS *stats) stats->lock_metadata_count = 0; stats->lock_metadata_wait_application = 0; stats->lock_metadata_wait_internal = 0; + stats->lock_read_timestamp_wait_application = 0; + stats->lock_read_timestamp_wait_internal = 0; + stats->lock_read_timestamp_read_count = 0; + stats->lock_read_timestamp_write_count = 0; stats->lock_schema_count = 0; stats->lock_schema_wait_application = 0; stats->lock_schema_wait_internal = 0; @@ -1229,6 +1285,10 @@ __wt_stat_connection_clear_single(WT_CONNECTION_STATS *stats) stats->lock_table_wait_internal = 0; stats->lock_table_read_count = 0; stats->lock_table_write_count = 0; + stats->lock_txn_global_wait_application = 0; + stats->lock_txn_global_wait_internal = 0; + stats->lock_txn_global_read_count = 0; + stats->lock_txn_global_write_count = 0; stats->log_slot_switch_busy = 0; stats->log_force_ckpt_sleep = 0; stats->log_bytes_payload = 0; @@ -1275,6 +1335,28 @@ __wt_stat_connection_clear_single(WT_CONNECTION_STATS *stats) stats->log_compress_len = 0; stats->log_slot_coalesced = 0; stats->log_close_yields = 0; + stats->perf_hist_fsread_latency_lt50 = 0; + stats->perf_hist_fsread_latency_lt100 = 0; + stats->perf_hist_fsread_latency_lt250 = 0; + stats->perf_hist_fsread_latency_lt500 = 0; + stats->perf_hist_fsread_latency_lt1000 = 0; + stats->perf_hist_fsread_latency_gt1000 = 0; + stats->perf_hist_fswrite_latency_lt50 = 0; + stats->perf_hist_fswrite_latency_lt100 = 0; + stats->perf_hist_fswrite_latency_lt250 = 0; + stats->perf_hist_fswrite_latency_lt500 = 0; + stats->perf_hist_fswrite_latency_lt1000 = 0; + stats->perf_hist_fswrite_latency_gt1000 = 0; + stats->perf_hist_opread_latency_lt250 = 0; + stats->perf_hist_opread_latency_lt500 = 0; + stats->perf_hist_opread_latency_lt1000 = 0; + stats->perf_hist_opread_latency_lt10000 = 0; + stats->perf_hist_opread_latency_gt10000 = 0; + stats->perf_hist_opwrite_latency_lt250 = 0; + stats->perf_hist_opwrite_latency_lt500 = 0; + stats->perf_hist_opwrite_latency_lt1000 = 0; + stats->perf_hist_opwrite_latency_lt10000 = 0; + stats->perf_hist_opwrite_latency_gt10000 = 0; stats->rec_page_delete_fast = 0; stats->rec_pages = 0; stats->rec_pages_eviction = 0; @@ -1320,8 +1402,24 @@ __wt_stat_connection_clear_single(WT_CONNECTION_STATS *stats) stats->page_del_rollback_blocked = 0; stats->child_modify_blocked_page = 0; stats->tree_descend_blocked = 0; + stats->txn_commit_queue_empty = 0; + stats->txn_commit_queue_head = 0; + stats->txn_commit_queue_inserts = 0; + stats->txn_commit_queue_len = 0; stats->txn_snapshots_created = 0; stats->txn_snapshots_dropped = 0; + stats->txn_query_ts = 0; + stats->txn_read_queue_empty = 0; + stats->txn_read_queue_head = 0; + stats->txn_read_queue_inserts = 0; + stats->txn_read_queue_len = 0; + stats->txn_set_ts = 0; + stats->txn_set_ts_commit = 0; + stats->txn_set_ts_commit_upd = 0; + stats->txn_set_ts_oldest = 0; + stats->txn_set_ts_oldest_upd = 0; + stats->txn_set_ts_stable = 0; + stats->txn_set_ts_stable_upd = 0; stats->txn_begin = 0; /* not clearing txn_checkpoint_running */ /* not clearing txn_checkpoint_generation */ @@ -1342,13 +1440,7 @@ __wt_stat_connection_clear_single(WT_CONNECTION_STATS *stats) /* not clearing txn_pinned_timestamp */ /* not clearing txn_pinned_timestamp_oldest */ stats->txn_sync = 0; - stats->txn_commit_queue_head = 0; - stats->txn_commit_queue_inserts = 0; - stats->txn_commit_queue_len = 0; stats->txn_commit = 0; - stats->txn_read_queue_head = 0; - stats->txn_read_queue_inserts = 0; - stats->txn_read_queue_len = 0; stats->txn_rollback = 0; stats->txn_update_conflict = 0; } @@ -1536,6 +1628,10 @@ __wt_stat_connection_aggregate( WT_STAT_READ(from, cache_eviction_pages_queued_oldest); to->cache_read += WT_STAT_READ(from, cache_read); to->cache_read_lookaside += WT_STAT_READ(from, cache_read_lookaside); + to->cache_read_lookaside_skipped += + WT_STAT_READ(from, cache_read_lookaside_skipped); + to->cache_read_lookaside_delay += + WT_STAT_READ(from, cache_read_lookaside_delay); to->cache_pages_requested += WT_STAT_READ(from, cache_pages_requested); to->cache_eviction_pages_seen += @@ -1590,6 +1686,14 @@ __wt_stat_connection_aggregate( WT_STAT_READ(from, lock_checkpoint_wait_application); to->lock_checkpoint_wait_internal += WT_STAT_READ(from, lock_checkpoint_wait_internal); + to->lock_commit_timestamp_wait_application += + WT_STAT_READ(from, lock_commit_timestamp_wait_application); + to->lock_commit_timestamp_wait_internal += + WT_STAT_READ(from, lock_commit_timestamp_wait_internal); + to->lock_commit_timestamp_read_count += + WT_STAT_READ(from, lock_commit_timestamp_read_count); + to->lock_commit_timestamp_write_count += + WT_STAT_READ(from, lock_commit_timestamp_write_count); to->lock_dhandle_wait_application += WT_STAT_READ(from, lock_dhandle_wait_application); to->lock_dhandle_wait_internal += @@ -1603,6 +1707,14 @@ __wt_stat_connection_aggregate( WT_STAT_READ(from, lock_metadata_wait_application); to->lock_metadata_wait_internal += WT_STAT_READ(from, lock_metadata_wait_internal); + to->lock_read_timestamp_wait_application += + WT_STAT_READ(from, lock_read_timestamp_wait_application); + to->lock_read_timestamp_wait_internal += + WT_STAT_READ(from, lock_read_timestamp_wait_internal); + to->lock_read_timestamp_read_count += + WT_STAT_READ(from, lock_read_timestamp_read_count); + to->lock_read_timestamp_write_count += + WT_STAT_READ(from, lock_read_timestamp_write_count); to->lock_schema_count += WT_STAT_READ(from, lock_schema_count); to->lock_schema_wait_application += WT_STAT_READ(from, lock_schema_wait_application); @@ -1616,6 +1728,14 @@ __wt_stat_connection_aggregate( WT_STAT_READ(from, lock_table_read_count); to->lock_table_write_count += WT_STAT_READ(from, lock_table_write_count); + to->lock_txn_global_wait_application += + WT_STAT_READ(from, lock_txn_global_wait_application); + to->lock_txn_global_wait_internal += + WT_STAT_READ(from, lock_txn_global_wait_internal); + to->lock_txn_global_read_count += + WT_STAT_READ(from, lock_txn_global_read_count); + to->lock_txn_global_write_count += + WT_STAT_READ(from, lock_txn_global_write_count); to->log_slot_switch_busy += WT_STAT_READ(from, log_slot_switch_busy); to->log_force_ckpt_sleep += WT_STAT_READ(from, log_force_ckpt_sleep); to->log_bytes_payload += WT_STAT_READ(from, log_bytes_payload); @@ -1669,6 +1789,50 @@ __wt_stat_connection_aggregate( to->log_compress_len += WT_STAT_READ(from, log_compress_len); to->log_slot_coalesced += WT_STAT_READ(from, log_slot_coalesced); to->log_close_yields += WT_STAT_READ(from, log_close_yields); + to->perf_hist_fsread_latency_lt50 += + WT_STAT_READ(from, perf_hist_fsread_latency_lt50); + to->perf_hist_fsread_latency_lt100 += + WT_STAT_READ(from, perf_hist_fsread_latency_lt100); + to->perf_hist_fsread_latency_lt250 += + WT_STAT_READ(from, perf_hist_fsread_latency_lt250); + to->perf_hist_fsread_latency_lt500 += + WT_STAT_READ(from, perf_hist_fsread_latency_lt500); + to->perf_hist_fsread_latency_lt1000 += + WT_STAT_READ(from, perf_hist_fsread_latency_lt1000); + to->perf_hist_fsread_latency_gt1000 += + WT_STAT_READ(from, perf_hist_fsread_latency_gt1000); + to->perf_hist_fswrite_latency_lt50 += + WT_STAT_READ(from, perf_hist_fswrite_latency_lt50); + to->perf_hist_fswrite_latency_lt100 += + WT_STAT_READ(from, perf_hist_fswrite_latency_lt100); + to->perf_hist_fswrite_latency_lt250 += + WT_STAT_READ(from, perf_hist_fswrite_latency_lt250); + to->perf_hist_fswrite_latency_lt500 += + WT_STAT_READ(from, perf_hist_fswrite_latency_lt500); + to->perf_hist_fswrite_latency_lt1000 += + WT_STAT_READ(from, perf_hist_fswrite_latency_lt1000); + to->perf_hist_fswrite_latency_gt1000 += + WT_STAT_READ(from, perf_hist_fswrite_latency_gt1000); + to->perf_hist_opread_latency_lt250 += + WT_STAT_READ(from, perf_hist_opread_latency_lt250); + to->perf_hist_opread_latency_lt500 += + WT_STAT_READ(from, perf_hist_opread_latency_lt500); + to->perf_hist_opread_latency_lt1000 += + WT_STAT_READ(from, perf_hist_opread_latency_lt1000); + to->perf_hist_opread_latency_lt10000 += + WT_STAT_READ(from, perf_hist_opread_latency_lt10000); + to->perf_hist_opread_latency_gt10000 += + WT_STAT_READ(from, perf_hist_opread_latency_gt10000); + to->perf_hist_opwrite_latency_lt250 += + WT_STAT_READ(from, perf_hist_opwrite_latency_lt250); + to->perf_hist_opwrite_latency_lt500 += + WT_STAT_READ(from, perf_hist_opwrite_latency_lt500); + to->perf_hist_opwrite_latency_lt1000 += + WT_STAT_READ(from, perf_hist_opwrite_latency_lt1000); + to->perf_hist_opwrite_latency_lt10000 += + WT_STAT_READ(from, perf_hist_opwrite_latency_lt10000); + to->perf_hist_opwrite_latency_gt10000 += + WT_STAT_READ(from, perf_hist_opwrite_latency_gt10000); to->rec_page_delete_fast += WT_STAT_READ(from, rec_page_delete_fast); to->rec_pages += WT_STAT_READ(from, rec_pages); to->rec_pages_eviction += WT_STAT_READ(from, rec_pages_eviction); @@ -1743,10 +1907,33 @@ __wt_stat_connection_aggregate( to->child_modify_blocked_page += WT_STAT_READ(from, child_modify_blocked_page); to->tree_descend_blocked += WT_STAT_READ(from, tree_descend_blocked); + to->txn_commit_queue_empty += + WT_STAT_READ(from, txn_commit_queue_empty); + to->txn_commit_queue_head += + WT_STAT_READ(from, txn_commit_queue_head); + to->txn_commit_queue_inserts += + WT_STAT_READ(from, txn_commit_queue_inserts); + to->txn_commit_queue_len += WT_STAT_READ(from, txn_commit_queue_len); to->txn_snapshots_created += WT_STAT_READ(from, txn_snapshots_created); to->txn_snapshots_dropped += WT_STAT_READ(from, txn_snapshots_dropped); + to->txn_query_ts += WT_STAT_READ(from, txn_query_ts); + to->txn_read_queue_empty += WT_STAT_READ(from, txn_read_queue_empty); + to->txn_read_queue_head += WT_STAT_READ(from, txn_read_queue_head); + to->txn_read_queue_inserts += + WT_STAT_READ(from, txn_read_queue_inserts); + to->txn_read_queue_len += WT_STAT_READ(from, txn_read_queue_len); + to->txn_set_ts += WT_STAT_READ(from, txn_set_ts); + to->txn_set_ts_commit += WT_STAT_READ(from, txn_set_ts_commit); + to->txn_set_ts_commit_upd += + WT_STAT_READ(from, txn_set_ts_commit_upd); + to->txn_set_ts_oldest += WT_STAT_READ(from, txn_set_ts_oldest); + to->txn_set_ts_oldest_upd += + WT_STAT_READ(from, txn_set_ts_oldest_upd); + to->txn_set_ts_stable += WT_STAT_READ(from, txn_set_ts_stable); + to->txn_set_ts_stable_upd += + WT_STAT_READ(from, txn_set_ts_stable_upd); to->txn_begin += WT_STAT_READ(from, txn_begin); to->txn_checkpoint_running += WT_STAT_READ(from, txn_checkpoint_running); @@ -1781,16 +1968,7 @@ __wt_stat_connection_aggregate( to->txn_pinned_timestamp_oldest += WT_STAT_READ(from, txn_pinned_timestamp_oldest); to->txn_sync += WT_STAT_READ(from, txn_sync); - to->txn_commit_queue_head += - WT_STAT_READ(from, txn_commit_queue_head); - to->txn_commit_queue_inserts += - WT_STAT_READ(from, txn_commit_queue_inserts); - to->txn_commit_queue_len += WT_STAT_READ(from, txn_commit_queue_len); to->txn_commit += WT_STAT_READ(from, txn_commit); - to->txn_read_queue_head += WT_STAT_READ(from, txn_read_queue_head); - to->txn_read_queue_inserts += - WT_STAT_READ(from, txn_read_queue_inserts); - to->txn_read_queue_len += WT_STAT_READ(from, txn_read_queue_len); to->txn_rollback += WT_STAT_READ(from, txn_rollback); to->txn_update_conflict += WT_STAT_READ(from, txn_update_conflict); } diff --git a/src/third_party/wiredtiger/src/support/thread_group.c b/src/third_party/wiredtiger/src/support/thread_group.c index ba0c57e9468..50abe64bbe6 100644 --- a/src/third_party/wiredtiger/src/support/thread_group.c +++ b/src/third_party/wiredtiger/src/support/thread_group.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/support/time.c b/src/third_party/wiredtiger/src/support/time.c index 240a77591a3..842b50fad09 100644 --- a/src/third_party/wiredtiger/src/support/time.c +++ b/src/third_party/wiredtiger/src/support/time.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -71,3 +71,23 @@ __wt_seconds(WT_SESSION_IMPL *session, time_t *timep) *timep = t.tv_sec; } + +/* + * __wt_clock_to_nsec -- + * Convert from clock ticks to nanoseconds. + */ +uint64_t +__wt_clock_to_nsec(uint64_t end, uint64_t begin) +{ + double clock_diff; + + /* + * If the ticks were reset, consider it an invalid check and just + * return zero as the time difference because we cannot compute + * anything meaningful. + */ + if (end < begin) + return (0); + clock_diff = (double)(end - begin); + return ((uint64_t)(clock_diff / __wt_process.tsc_nsec_ratio)); +} diff --git a/src/third_party/wiredtiger/src/txn/txn.c b/src/third_party/wiredtiger/src/txn/txn.c index fb61b037723..5d70acb90f0 100644 --- a/src/third_party/wiredtiger/src/txn/txn.c +++ b/src/third_party/wiredtiger/src/txn/txn.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -367,7 +367,6 @@ __wt_txn_update_oldest(WT_SESSION_IMPL *session, uint32_t flags) if (WT_TXNID_LT(txn_global->last_running, last_running)) { txn_global->last_running = last_running; -#ifdef HAVE_VERBOSE /* Output a verbose message about long-running transactions, * but only when some progress is being made. */ if (WT_VERBOSE_ISSET(session, WT_VERB_TRANSACTION) && @@ -380,7 +379,6 @@ __wt_txn_update_oldest(WT_SESSION_IMPL *session, uint32_t flags) oldest_session->lastop, oldest_session->txn.snap_min); } -#endif } done: __wt_writeunlock(session, &txn_global->rwlock); @@ -444,7 +442,7 @@ __wt_txn_config(WT_SESSION_IMPL *session, const char *cfg[]) #ifdef HAVE_TIMESTAMPS wt_timestamp_t ts; WT_TXN_GLOBAL *txn_global; - char timestamp_buf[2 * WT_TIMESTAMP_SIZE + 1]; + char hex_timestamp[2][2 * WT_TIMESTAMP_SIZE + 1]; bool round_to_oldest; txn_global = &S2C(session)->txn_global; @@ -462,11 +460,13 @@ __wt_txn_config(WT_SESSION_IMPL *session, const char *cfg[]) * avoid a race between checking and setting transaction * timestamp. */ + WT_RET(__wt_timestamp_to_hex_string(session, + hex_timestamp[0], &ts)); __wt_readlock(session, &txn_global->rwlock); if (__wt_timestamp_cmp(&ts, &txn_global->oldest_timestamp) < 0) { WT_RET(__wt_timestamp_to_hex_string(session, - timestamp_buf, &ts)); + hex_timestamp[1], &txn_global->oldest_timestamp)); /* * If given read timestamp is earlier than oldest * timestamp then round the read timestamp to @@ -478,8 +478,8 @@ __wt_txn_config(WT_SESSION_IMPL *session, const char *cfg[]) else { __wt_readunlock(session, &txn_global->rwlock); WT_RET_MSG(session, EINVAL, "read timestamp " - "%s older than oldest timestamp", - timestamp_buf); + "%s older than oldest timestamp %s", + hex_timestamp[0], hex_timestamp[1]); } } else { __wt_timestamp_set(&txn->read_timestamp, &ts); @@ -499,8 +499,8 @@ __wt_txn_config(WT_SESSION_IMPL *session, const char *cfg[]) * critical section. */ __wt_verbose(session, WT_VERB_TIMESTAMP, "Read " - "timestamp %s : Rounded to oldest timestamp", - timestamp_buf); + "timestamp %s : Rounded to oldest timestamp %s", + hex_timestamp[0], hex_timestamp[1]); } #else WT_RET_MSG(session, EINVAL, "read_timestamp requires a " @@ -594,10 +594,90 @@ __wt_txn_release(WT_SESSION_IMPL *session) __wt_txn_release_snapshot(session); txn->isolation = session->isolation; + txn->rollback_reason = NULL; + /* Ensure the transaction flags are cleared on exit */ txn->flags = 0; } +#ifdef HAVE_TIMESTAMPS +/* + * __txn_commit_timestamp_validate -- + * Validate that timestamp provided to commit is legal. + */ +static inline int +__txn_commit_timestamp_validate(WT_SESSION_IMPL *session) +{ + WT_TXN *txn; + WT_TXN_OP *op; + WT_UPDATE *upd; + u_int i; + bool op_used_ts, upd_used_ts; + + txn = &session->txn; + + /* + * Debugging checks on timestamps, if user requested them. + */ + if (F_ISSET(txn, WT_TXN_TS_COMMIT_ALWAYS) && + !F_ISSET(txn, WT_TXN_HAS_TS_COMMIT) && + txn->mod_count != 0) + WT_RET_MSG(session, EINVAL, "commit_timestamp required and " + "none set on this transaction"); + if (F_ISSET(txn, WT_TXN_TS_COMMIT_NEVER) && + F_ISSET(txn, WT_TXN_HAS_TS_COMMIT) && + txn->mod_count != 0) + WT_RET_MSG(session, EINVAL, "no commit_timestamp required and " + "timestamp set on this transaction"); + + /* + * If we're not doing any key consistency checking, we're done. + */ + if (!F_ISSET(txn, WT_TXN_TS_COMMIT_KEYS)) + return (0); + + /* + * Error on any valid update structures for the same key that + * are at a later timestamp or use timestamps inconsistently. + */ + for (i = 0, op = txn->mod; i < txn->mod_count; i++, op++) + if (op->type == WT_TXN_OP_BASIC_TS || + op->type == WT_TXN_OP_BASIC) { + /* + * Skip over any aborted update structures. + */ + upd = op->u.upd->next; + while (upd != NULL && upd->txnid == WT_TXN_ABORTED) + upd = upd->next; + + /* + * Check the timestamp on this update with the + * first valid update in the chain. They're in + * most recent order. + */ + if (upd == NULL) + continue; + /* + * Check for consistent per-key timestamp usage. + * If timestamps are or are not used originally then + * they should be used the same way always. Check + * timestamps are used in order. + */ + op_used_ts = + __wt_timestamp_iszero(&op->u.upd->timestamp); + upd_used_ts = __wt_timestamp_iszero(&upd->timestamp); + if (op_used_ts != upd_used_ts) + WT_RET_MSG(session, EINVAL, + "per-key timestamps used inconsistently"); + if (__wt_timestamp_cmp(&op->u.upd->timestamp, + &upd->timestamp) < 0) + WT_RET_MSG(session, EINVAL, + "out of order timestamps"); + } + return (0); +} +#endif + /* * __wt_txn_commit -- * Commit the current transaction. @@ -647,20 +727,9 @@ __wt_txn_commit(WT_SESSION_IMPL *session, const char *cfg[]) } #ifdef HAVE_TIMESTAMPS - /* - * Debugging checks on timestamps, if user requested them. - */ - if (F_ISSET(txn, WT_TXN_TS_COMMIT_ALWAYS) && - !F_ISSET(txn, WT_TXN_HAS_TS_COMMIT) && - txn->mod_count != 0) - WT_ERR_MSG(session, EINVAL, "commit_timestamp required and " - "none set on this transaction"); - if (F_ISSET(txn, WT_TXN_TS_COMMIT_NEVER) && - F_ISSET(txn, WT_TXN_HAS_TS_COMMIT) && - txn->mod_count != 0) - WT_ERR_MSG(session, EINVAL, "no commit_timestamp required and " - "timestamp set on this transaction"); + WT_ERR(__txn_commit_timestamp_validate(session)); #endif + /* * The default sync setting is inherited from the connection, but can * be overridden by an explicit "sync" setting for this transaction. @@ -749,7 +818,7 @@ __wt_txn_commit(WT_SESSION_IMPL *session, const char *cfg[]) * Switch reserved operations to abort to * simplify obsolete update list truncation. */ - if (op->u.upd->type == WT_UPDATE_RESERVED) { + if (op->u.upd->type == WT_UPDATE_RESERVE) { op->u.upd->txnid = WT_TXN_ABORTED; break; } @@ -942,6 +1011,18 @@ __wt_txn_rollback(WT_SESSION_IMPL *session, const char *cfg[]) } /* + * __wt_txn_rollback_required -- + * Prepare to log a reason if the user attempts to use the transaction to + * do anything other than rollback. + */ +int +__wt_txn_rollback_required(WT_SESSION_IMPL *session, const char *reason) +{ + session->txn.rollback_reason = reason; + return (WT_ROLLBACK); +} + +/* * __wt_txn_init -- * Initialize a session's transaction data. */ @@ -1061,13 +1142,15 @@ __wt_txn_global_init(WT_SESSION_IMPL *session, const char *cfg[]) WT_RET(__wt_spin_init( session, &txn_global->id_lock, "transaction id lock")); - WT_RET(__wt_rwlock_init(session, &txn_global->rwlock)); + WT_RWLOCK_INIT_TRACKED(session, &txn_global->rwlock, txn_global); WT_RET(__wt_rwlock_init(session, &txn_global->visibility_rwlock)); - WT_RET(__wt_rwlock_init(session, &txn_global->commit_timestamp_rwlock)); + WT_RWLOCK_INIT_TRACKED(session, + &txn_global->commit_timestamp_rwlock, commit_timestamp); TAILQ_INIT(&txn_global->commit_timestamph); - WT_RET(__wt_rwlock_init(session, &txn_global->read_timestamp_rwlock)); + WT_RWLOCK_INIT_TRACKED(session, + &txn_global->read_timestamp_rwlock, read_timestamp); TAILQ_INIT(&txn_global->read_timestamph); WT_RET(__wt_rwlock_init(session, &txn_global->nsnap_rwlock)); @@ -1158,6 +1241,7 @@ __wt_verbose_dump_txn_one(WT_SESSION_IMPL *session, WT_TXN *txn) const char *iso_tag; iso_tag = "INVALID"; + WT_NOT_READ(iso_tag); switch (txn->isolation) { case WT_ISO_READ_COMMITTED: iso_tag = "WT_ISO_READ_COMMITTED"; diff --git a/src/third_party/wiredtiger/src/txn/txn_ckpt.c b/src/third_party/wiredtiger/src/txn/txn_ckpt.c index c82187daf85..616816f0e8d 100644 --- a/src/third_party/wiredtiger/src/txn/txn_ckpt.c +++ b/src/third_party/wiredtiger/src/txn/txn_ckpt.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -369,25 +369,24 @@ __wt_checkpoint_get_handles(WT_SESSION_IMPL *session, const char *cfg[]) static void __checkpoint_reduce_dirty_cache(WT_SESSION_IMPL *session) { - struct timespec last, start, stop; WT_CACHE *cache; WT_CONNECTION_IMPL *conn; double current_dirty, delta, scrub_min; uint64_t bytes_written_last, bytes_written_start, bytes_written_total; uint64_t cache_size, max_write; uint64_t current_us, stepdown_us, total_ms, work_us; + uint64_t time_last, time_start, time_stop; bool progress; conn = S2C(session); cache = conn->cache; /* Give up if scrubbing is disabled. */ - if (cache->eviction_checkpoint_target == 0 || + if (cache->eviction_checkpoint_target < DBL_EPSILON || cache->eviction_checkpoint_target >= cache->eviction_dirty_trigger) return; - __wt_epoch(session, &start); - last = start; + time_last = time_start = __wt_clock(session); bytes_written_last = 0; bytes_written_start = cache->bytes_written; cache_size = conn->cache_size; @@ -437,7 +436,7 @@ __checkpoint_reduce_dirty_cache(WT_SESSION_IMPL *session) for (;;) { current_dirty = (100.0 * __wt_cache_dirty_leaf_inuse(cache)) / cache_size; - if (current_dirty <= (double)cache->eviction_checkpoint_target) + if (current_dirty <= cache->eviction_checkpoint_target) break; /* @@ -448,8 +447,8 @@ __checkpoint_reduce_dirty_cache(WT_SESSION_IMPL *session) break; __wt_sleep(0, stepdown_us / 10); - __wt_epoch(session, &stop); - current_us = WT_TIMEDIFF_US(stop, last); + time_stop = __wt_clock(session); + current_us = WT_CLOCKDIFF_US(time_stop, time_last); bytes_written_total = cache->bytes_written - bytes_written_start; @@ -503,11 +502,11 @@ __checkpoint_reduce_dirty_cache(WT_SESSION_IMPL *session) WT_MAX(cache->eviction_dirty_target, current_dirty - delta); WT_STAT_CONN_SET(session, txn_checkpoint_scrub_target, cache->eviction_scrub_limit); - __wt_epoch(session, &last); + time_last = __wt_clock(session); } - __wt_epoch(session, &stop); - total_ms = WT_TIMEDIFF_MS(stop, start); + time_stop = __wt_clock(session); + total_ms = WT_CLOCKDIFF_MS(time_stop, time_start); WT_STAT_CONN_SET(session, txn_checkpoint_scrub_time, total_ms); } @@ -575,7 +574,6 @@ __checkpoint_stats(WT_SESSION_IMPL *session) static void __checkpoint_verbose_track(WT_SESSION_IMPL *session, const char *msg) { -#ifdef HAVE_VERBOSE struct timespec stop; WT_CONNECTION_IMPL *conn; uint64_t msec; @@ -593,10 +591,6 @@ __checkpoint_verbose_track(WT_SESSION_IMPL *session, const char *msg) ": Full database checkpoint %s", msec, __wt_gen(session, WT_GEN_CHECKPOINT), msg); -#else - WT_UNUSED(session); - WT_UNUSED(msg); -#endif } /* @@ -745,14 +739,13 @@ __checkpoint_prepare(WT_SESSION_IMPL *session, const char *cfg[]) static int __txn_checkpoint(WT_SESSION_IMPL *session, const char *cfg[]) { - struct timespec fsync_start, fsync_stop; WT_CACHE *cache; WT_CONNECTION_IMPL *conn; WT_DECL_RET; WT_TXN *txn; WT_TXN_GLOBAL *txn_global; WT_TXN_ISOLATION saved_isolation; - uint64_t fsync_duration_usecs, generation; + uint64_t fsync_duration_usecs, generation, time_start, time_stop; u_int i; bool failed, full, idle, logging, tracking; void *saved_meta_next; @@ -887,10 +880,10 @@ __txn_checkpoint(WT_SESSION_IMPL *session, const char *cfg[]) * Checkpoints have to hit disk (it would be reasonable to configure for * lazy checkpoints, but we don't support them yet). */ - __wt_epoch(session, &fsync_start); + time_start = __wt_clock(session); WT_ERR(__checkpoint_apply(session, cfg, __wt_checkpoint_sync)); - __wt_epoch(session, &fsync_stop); - fsync_duration_usecs = WT_TIMEDIFF_US(fsync_stop, fsync_start); + time_stop = __wt_clock(session); + fsync_duration_usecs = WT_CLOCKDIFF_US(time_stop, time_start); WT_STAT_CONN_INCR(session, txn_checkpoint_fsync_post); WT_STAT_CONN_SET(session, txn_checkpoint_fsync_post_duration, fsync_duration_usecs); diff --git a/src/third_party/wiredtiger/src/txn/txn_ext.c b/src/third_party/wiredtiger/src/txn/txn_ext.c index 38d2a08a13c..e0401c83be5 100644 --- a/src/third_party/wiredtiger/src/txn/txn_ext.c +++ b/src/third_party/wiredtiger/src/txn/txn_ext.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/txn/txn_log.c b/src/third_party/wiredtiger/src/txn/txn_log.c index bd1b2239e2d..d31a9e27583 100644 --- a/src/third_party/wiredtiger/src/txn/txn_log.c +++ b/src/third_party/wiredtiger/src/txn/txn_log.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -25,6 +25,7 @@ __txn_op_log_row_key_check(WT_SESSION_IMPL *session, WT_CURSOR_BTREE *cbt) WT_ITEM key; WT_PAGE *page; WT_ROW *rip; + int cmp; cursor = &cbt->iface; WT_ASSERT(session, F_ISSET(cursor, WT_CURSTD_KEY_SET)); @@ -50,8 +51,9 @@ __txn_op_log_row_key_check(WT_SESSION_IMPL *session, WT_CURSOR_BTREE *cbt) key.size = WT_INSERT_KEY_SIZE(cbt->ins); } - WT_ASSERT(session, key.size == cursor->key.size && - memcmp(key.data, cursor->key.data, key.size) == 0); + WT_ASSERT(session, __wt_compare( + session, cbt->btree->collator, &key, &cursor->key, &cmp) == 0); + WT_ASSERT(session, cmp == 0); __wt_buf_free(session, &key); } @@ -85,11 +87,7 @@ __txn_op_log(WT_SESSION_IMPL *session, __txn_op_log_row_key_check(session, cbt); #endif switch (upd->type) { - case WT_UPDATE_DELETED: - WT_RET(__wt_logop_row_remove_pack( - session, logrec, op->fileid, &cursor->key)); - break; - case WT_UPDATE_MODIFIED: + case WT_UPDATE_MODIFY: WT_RET(__wt_logop_row_modify_pack( session, logrec, op->fileid, &cursor->key, &value)); break; @@ -97,6 +95,10 @@ __txn_op_log(WT_SESSION_IMPL *session, WT_RET(__wt_logop_row_put_pack( session, logrec, op->fileid, &cursor->key, &value)); break; + case WT_UPDATE_TOMBSTONE: + WT_RET(__wt_logop_row_remove_pack( + session, logrec, op->fileid, &cursor->key)); + break; WT_ILLEGAL_VALUE(session); } } else { @@ -104,11 +106,7 @@ __txn_op_log(WT_SESSION_IMPL *session, WT_ASSERT(session, recno != WT_RECNO_OOB); switch (upd->type) { - case WT_UPDATE_DELETED: - WT_RET(__wt_logop_col_remove_pack( - session, logrec, op->fileid, recno)); - break; - case WT_UPDATE_MODIFIED: + case WT_UPDATE_MODIFY: WT_RET(__wt_logop_col_modify_pack( session, logrec, op->fileid, recno, &value)); break; @@ -116,6 +114,10 @@ __txn_op_log(WT_SESSION_IMPL *session, WT_RET(__wt_logop_col_put_pack( session, logrec, op->fileid, recno, &value)); break; + case WT_UPDATE_TOMBSTONE: + WT_RET(__wt_logop_col_remove_pack( + session, logrec, op->fileid, recno)); + break; WT_ILLEGAL_VALUE(session); } } diff --git a/src/third_party/wiredtiger/src/txn/txn_nsnap.c b/src/third_party/wiredtiger/src/txn/txn_nsnap.c index 487631aa687..ed1d5407de4 100644 --- a/src/third_party/wiredtiger/src/txn/txn_nsnap.c +++ b/src/third_party/wiredtiger/src/txn/txn_nsnap.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/txn/txn_recover.c b/src/third_party/wiredtiger/src/txn/txn_recover.c index 97d45eb9d98..e058d08cc17 100644 --- a/src/third_party/wiredtiger/src/txn/txn_recover.c +++ b/src/third_party/wiredtiger/src/txn/txn_recover.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -136,8 +136,7 @@ __txn_op_apply( * than using cursor modify to create a partial update * (for no particular reason than simplicity). */ - WT_ERR(__wt_modify_apply( - session, &cursor->value, value.data)); + WT_ERR(__wt_modify_apply(session, cursor, value.data)); WT_ERR(cursor->insert(cursor)); } break; @@ -204,8 +203,7 @@ __txn_op_apply( * than using cursor modify to create a partial update * (for no particular reason than simplicity). */ - WT_ERR(__wt_modify_apply( - session, &cursor->value, value.data)); + WT_ERR(__wt_modify_apply(session, cursor, value.data)); WT_ERR(cursor->insert(cursor)); } break; diff --git a/src/third_party/wiredtiger/src/txn/txn_rollback_to_stable.c b/src/third_party/wiredtiger/src/txn/txn_rollback_to_stable.c index c68d00d7503..0af70c4090d 100644 --- a/src/third_party/wiredtiger/src/txn/txn_rollback_to_stable.c +++ b/src/third_party/wiredtiger/src/txn/txn_rollback_to_stable.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -49,7 +49,7 @@ __txn_rollback_to_stable_lookaside_fixup(WT_SESSION_IMPL *session) F_SET(session, WT_SESSION_READ_WONT_NEED); /* Walk the file. */ - for (; (ret = cursor->next(cursor)) == 0; ) { + while ((ret = cursor->next(cursor)) == 0) { WT_ERR(cursor->get_key(cursor, &las_pageid, &las_id, &las_counter, &las_key)); @@ -251,22 +251,6 @@ __txn_abort_newer_updates( } /* - * __txn_rollback_to_stable_custom_skip -- - * Return if custom rollback requires we read this page. - */ -static int -__txn_rollback_to_stable_custom_skip( - WT_SESSION_IMPL *session, WT_REF *ref, void *context, bool *skipp) -{ - WT_UNUSED(context); - WT_UNUSED(session); - - /* Review all pages that are in memory. */ - *skipp = !(ref->state == WT_REF_MEM || ref->state == WT_REF_DELETED); - return (0); -} - -/* * __txn_rollback_to_stable_btree_walk -- * Called for each open handle - choose to either skip or wipe the commits */ @@ -275,22 +259,24 @@ __txn_rollback_to_stable_btree_walk( WT_SESSION_IMPL *session, wt_timestamp_t *rollback_timestamp) { WT_DECL_RET; - WT_PAGE *page; WT_REF *ref; /* Walk the tree, marking commits aborted where appropriate. */ ref = NULL; - while ((ret = __wt_tree_walk_custom_skip(session, &ref, - __txn_rollback_to_stable_custom_skip, - NULL, WT_READ_NO_EVICT)) == 0 && ref != NULL) { - page = ref->page; + while ((ret = __wt_tree_walk(session, &ref, + WT_READ_CACHE | WT_READ_LOOKASIDE | WT_READ_NO_EVICT)) == 0 && + ref != NULL) { + if (ref->page_las != NULL && + __wt_timestamp_cmp(rollback_timestamp, + &ref->page_las->onpage_timestamp) < 0) + ref->page_las->invalid = true; /* Review deleted page saved to the ref */ if (ref->page_del != NULL && __wt_timestamp_cmp( rollback_timestamp, &ref->page_del->timestamp) < 0) __wt_delete_page_rollback(session, ref); - if (!__wt_page_is_modified(page)) + if (!__wt_page_is_modified(ref->page)) continue; WT_RET(__txn_abort_newer_updates( @@ -369,7 +355,7 @@ __txn_rollback_to_stable_btree(WT_SESSION_IMPL *session, const char *cfg[]) */ WT_WITH_TIMESTAMP_READLOCK(session, &txn_global->rwlock, __wt_timestamp_set( - &rollback_timestamp, &txn_global->stable_timestamp)); + &rollback_timestamp, &txn_global->stable_timestamp)); /* * Ensure the eviction server is out of the file - we don't diff --git a/src/third_party/wiredtiger/src/txn/txn_timestamp.c b/src/third_party/wiredtiger/src/txn/txn_timestamp.c index 6735946fbb3..41ac970f14e 100644 --- a/src/third_party/wiredtiger/src/txn/txn_timestamp.c +++ b/src/third_party/wiredtiger/src/txn/txn_timestamp.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -65,25 +65,20 @@ __wt_timestamp_to_hex_string( /* * __wt_verbose_timestamp -- - * Output a verbose message along with the specified timestamp + * Output a verbose message along with the specified timestamp. */ void __wt_verbose_timestamp(WT_SESSION_IMPL *session, const wt_timestamp_t *ts, const char *msg) { -#ifdef HAVE_VERBOSE char timestamp_buf[2 * WT_TIMESTAMP_SIZE + 1]; - if (__wt_timestamp_to_hex_string(session, timestamp_buf, ts) != 0) + if (!WT_VERBOSE_ISSET(session, WT_VERB_TIMESTAMP) || + (__wt_timestamp_to_hex_string(session, timestamp_buf, ts) != 0)) return; __wt_verbose(session, WT_VERB_TIMESTAMP, "Timestamp %s : %s", timestamp_buf, msg); -#else - WT_UNUSED(session); - WT_UNUSED(ts); - WT_UNUSED(msg); -#endif } /* @@ -202,6 +197,7 @@ __txn_global_query_timestamp( conn = S2C(session); txn_global = &conn->txn_global; + WT_STAT_CONN_INCR(session, txn_query_ts); WT_RET(__wt_config_gets(session, cfg, "get", &cval)); if (WT_STRING_MATCH("all_committed", cval.str, cval.len)) { if (!txn_global->has_commit_timestamp) @@ -309,7 +305,7 @@ __wt_txn_update_pinned_timestamp(WT_SESSION_IMPL *session, bool force) WT_WITH_TIMESTAMP_READLOCK(session, &txn_global->rwlock, __wt_timestamp_set( - &oldest_timestamp, &txn_global->oldest_timestamp)); + &oldest_timestamp, &txn_global->oldest_timestamp)); /* Scan to find the global pinned timestamp. */ if ((ret = __txn_global_query_timestamp( @@ -324,7 +320,7 @@ __wt_txn_update_pinned_timestamp(WT_SESSION_IMPL *session, bool force) if (txn_global->has_pinned_timestamp && !force) { WT_WITH_TIMESTAMP_READLOCK(session, &txn_global->rwlock, __wt_timestamp_set( - &last_pinned_timestamp, &txn_global->pinned_timestamp)); + &last_pinned_timestamp, &txn_global->pinned_timestamp)); if (__wt_timestamp_cmp( &pinned_timestamp, &last_pinned_timestamp) <= 0) @@ -359,17 +355,24 @@ __wt_txn_global_set_timestamp(WT_SESSION_IMPL *session, const char *cfg[]) WT_CONFIG_ITEM commit_cval, oldest_cval, stable_cval; bool has_commit, has_oldest, has_stable; + WT_STAT_CONN_INCR(session, txn_set_ts); WT_RET(__wt_config_gets_def(session, cfg, "commit_timestamp", 0, &commit_cval)); has_commit = commit_cval.len != 0; + if (has_commit) + WT_STAT_CONN_INCR(session, txn_set_ts_commit); WT_RET(__wt_config_gets_def(session, cfg, "oldest_timestamp", 0, &oldest_cval)); has_oldest = oldest_cval.len != 0; + if (has_oldest) + WT_STAT_CONN_INCR(session, txn_set_ts_oldest); WT_RET(__wt_config_gets_def(session, cfg, "stable_timestamp", 0, &stable_cval)); has_stable = stable_cval.len != 0; + if (has_stable) + WT_STAT_CONN_INCR(session, txn_set_ts_stable); /* If no timestamp was supplied, there's nothing to do. */ if (!has_commit && !has_oldest && !has_stable) @@ -381,6 +384,7 @@ __wt_txn_global_set_timestamp(WT_SESSION_IMPL *session, const char *cfg[]) WT_TXN_GLOBAL *txn_global; wt_timestamp_t commit_ts, oldest_ts, stable_ts; wt_timestamp_t last_oldest_ts, last_stable_ts; + char hex_timestamp[2][2 * WT_TIMESTAMP_SIZE + 1]; bool force; txn_global = &S2C(session)->txn_global; @@ -429,17 +433,25 @@ __wt_txn_global_set_timestamp(WT_SESSION_IMPL *session, const char *cfg[]) if (has_commit && (has_oldest || txn_global->has_oldest_timestamp) && __wt_timestamp_cmp(&oldest_ts, &commit_ts) > 0) { __wt_readunlock(session, &txn_global->rwlock); + WT_RET(__wt_timestamp_to_hex_string( + session, hex_timestamp[0], &oldest_ts)); + WT_RET(__wt_timestamp_to_hex_string( + session, hex_timestamp[1], &commit_ts)); WT_RET_MSG(session, EINVAL, - "set_timestamp: oldest timestamp must not be later than " - "commit timestamp"); + "set_timestamp: oldest timestamp %s must not be later than " + "commit timestamp %s", hex_timestamp[0], hex_timestamp[1]); } if (has_commit && (has_stable || txn_global->has_stable_timestamp) && __wt_timestamp_cmp(&stable_ts, &commit_ts) > 0) { __wt_readunlock(session, &txn_global->rwlock); + WT_RET(__wt_timestamp_to_hex_string( + session, hex_timestamp[0], &stable_ts)); + WT_RET(__wt_timestamp_to_hex_string( + session, hex_timestamp[1], &commit_ts)); WT_RET_MSG(session, EINVAL, - "set_timestamp: stable timestamp must not be later than " - "commit timestamp"); + "set_timestamp: stable timestamp %s must not be later than " + "commit timestamp %s", hex_timestamp[0], hex_timestamp[1]); } /* @@ -451,9 +463,13 @@ __wt_txn_global_set_timestamp(WT_SESSION_IMPL *session, const char *cfg[]) (has_stable || txn_global->has_stable_timestamp) && __wt_timestamp_cmp(&oldest_ts, &stable_ts) > 0) { __wt_readunlock(session, &txn_global->rwlock); + WT_RET(__wt_timestamp_to_hex_string( + session, hex_timestamp[0], &oldest_ts)); + WT_RET(__wt_timestamp_to_hex_string( + session, hex_timestamp[1], &stable_ts)); WT_RET_MSG(session, EINVAL, - "set_timestamp: oldest timestamp must not be later than " - "stable timestamp"); + "set_timestamp: oldest timestamp %s must not be later than " + "stable timestamp %s", hex_timestamp[0], hex_timestamp[1]); } __wt_readunlock(session, &txn_global->rwlock); @@ -484,6 +500,7 @@ set: __wt_writelock(session, &txn_global->rwlock); if (has_commit) { __wt_timestamp_set(&txn_global->commit_timestamp, &commit_ts); txn_global->has_commit_timestamp = true; + WT_STAT_CONN_INCR(session, txn_set_ts_commit_upd); __wt_verbose_timestamp(session, &commit_ts, "Updated global commit timestamp"); } @@ -492,6 +509,7 @@ set: __wt_writelock(session, &txn_global->rwlock); force || __wt_timestamp_cmp( &oldest_ts, &txn_global->oldest_timestamp) > 0)) { __wt_timestamp_set(&txn_global->oldest_timestamp, &oldest_ts); + WT_STAT_CONN_INCR(session, txn_set_ts_oldest_upd); txn_global->has_oldest_timestamp = true; txn_global->oldest_is_pinned = false; __wt_verbose_timestamp(session, &oldest_ts, @@ -502,6 +520,7 @@ set: __wt_writelock(session, &txn_global->rwlock); force || __wt_timestamp_cmp( &stable_ts, &txn_global->stable_timestamp) > 0)) { __wt_timestamp_set(&txn_global->stable_timestamp, &stable_ts); + WT_STAT_CONN_INCR(session, txn_set_ts_stable_upd); txn_global->has_stable_timestamp = true; txn_global->stable_is_pinned = false; __wt_verbose_timestamp(session, &stable_ts, @@ -532,29 +551,41 @@ __wt_timestamp_validate(WT_SESSION_IMPL *session, const char *name, { WT_TXN *txn = &session->txn; WT_TXN_GLOBAL *txn_global = &S2C(session)->txn_global; + wt_timestamp_t oldest_ts, stable_ts; char hex_timestamp[2 * WT_TIMESTAMP_SIZE + 1]; - bool older_than_oldest_ts, older_than_stable_ts; + bool has_oldest_ts, has_stable_ts; /* + * Added this redundant initialization to circumvent build failure. + */ + __wt_timestamp_set_zero(&oldest_ts); + __wt_timestamp_set_zero(&stable_ts); + /* * Compare against the oldest and the stable timestamp. Return an error * if the given timestamp is older than oldest and/or stable timestamp. */ WT_WITH_TIMESTAMP_READLOCK(session, &txn_global->rwlock, - older_than_oldest_ts = (cmp_oldest && - txn_global->has_oldest_timestamp && - __wt_timestamp_cmp(ts, &txn_global->oldest_timestamp) < 0); - older_than_stable_ts = (cmp_stable && - txn_global->has_stable_timestamp && - __wt_timestamp_cmp(ts, &txn_global->stable_timestamp) < 0)); - - if (older_than_oldest_ts) + if ((has_oldest_ts = txn_global->has_oldest_timestamp)) + __wt_timestamp_set(&oldest_ts, &txn_global->oldest_timestamp); + if ((has_stable_ts = txn_global->has_stable_timestamp)) + __wt_timestamp_set(&stable_ts, &txn_global->stable_timestamp)); + + if (cmp_oldest && has_oldest_ts && + __wt_timestamp_cmp(ts, &oldest_ts) < 0) { + WT_RET(__wt_timestamp_to_hex_string(session, hex_timestamp, + &oldest_ts)); WT_RET_MSG(session, EINVAL, - "%s timestamp %.*s older than oldest timestamp", - name, (int)cval->len, cval->str); - if (older_than_stable_ts) + "%s timestamp %.*s older than oldest timestamp %s", + name, (int)cval->len, cval->str, hex_timestamp); + } + if (cmp_stable && has_stable_ts && + __wt_timestamp_cmp(ts, &stable_ts) < 0) { + WT_RET(__wt_timestamp_to_hex_string(session, hex_timestamp, + &stable_ts)); WT_RET_MSG(session, EINVAL, - "%s timestamp %.*s older than stable timestamp", - name, (int)cval->len, cval->str); + "%s timestamp %.*s older than stable timestamp %s", + name, (int)cval->len, cval->str, hex_timestamp); + } /* * Compare against the commit timestamp of the current transaction. @@ -640,8 +671,10 @@ __wt_txn_set_commit_timestamp(WT_SESSION_IMPL *session) __wt_timestamp_set(&txn->first_commit_timestamp, &ts); __wt_writelock(session, &txn_global->commit_timestamp_rwlock); - for (prev = TAILQ_LAST(&txn_global->commit_timestamph, __wt_txn_cts_qh); - prev != NULL && + prev = TAILQ_LAST(&txn_global->commit_timestamph, __wt_txn_cts_qh); + if (prev == NULL) + WT_STAT_CONN_INCR(session, txn_commit_queue_empty); + for (; prev != NULL && __wt_timestamp_cmp(&prev->first_commit_timestamp, &ts) > 0; prev = TAILQ_PREV(prev, __wt_txn_cts_qh, commit_timestampq)) ; @@ -698,8 +731,10 @@ __wt_txn_set_read_timestamp(WT_SESSION_IMPL *session) return; __wt_writelock(session, &txn_global->read_timestamp_rwlock); - for (prev = TAILQ_LAST(&txn_global->read_timestamph, __wt_txn_rts_qh); - prev != NULL && __wt_timestamp_cmp( + prev = TAILQ_LAST(&txn_global->read_timestamph, __wt_txn_rts_qh); + if (prev == NULL) + WT_STAT_CONN_INCR(session, txn_read_queue_empty); + for (; prev != NULL && __wt_timestamp_cmp( &prev->read_timestamp, &txn->read_timestamp) > 0; prev = TAILQ_PREV(prev, __wt_txn_rts_qh, read_timestampq)) ; diff --git a/src/third_party/wiredtiger/src/utilities/util.h b/src/third_party/wiredtiger/src/utilities/util.h index adf3f844295..1dda8c2fb61 100644 --- a/src/third_party/wiredtiger/src/utilities/util.h +++ b/src/third_party/wiredtiger/src/utilities/util.h @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -48,7 +48,7 @@ int util_rebalance(WT_SESSION *, int, char *[]); int util_rename(WT_SESSION *, int, char *[]); int util_salvage(WT_SESSION *, int, char *[]); int util_stat(WT_SESSION *, int, char *[]); -int util_str2recno(WT_SESSION *, const char *p, uint64_t *recnop); +int util_str2num(WT_SESSION *, const char *, bool, uint64_t *); int util_truncate(WT_SESSION *, int, char *[]); int util_upgrade(WT_SESSION *, int, char *[]); char *util_uri(WT_SESSION *, const char *, const char *); diff --git a/src/third_party/wiredtiger/src/utilities/util_alter.c b/src/third_party/wiredtiger/src/utilities/util_alter.c index da6316b2364..b57c2d7b090 100644 --- a/src/third_party/wiredtiger/src/utilities/util_alter.c +++ b/src/third_party/wiredtiger/src/utilities/util_alter.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/utilities/util_backup.c b/src/third_party/wiredtiger/src/utilities/util_backup.c index bf4d7b67671..f0296de7ba8 100644 --- a/src/third_party/wiredtiger/src/utilities/util_backup.c +++ b/src/third_party/wiredtiger/src/utilities/util_backup.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -11,53 +11,31 @@ static int copy(WT_SESSION *, const char *, const char *); static int usage(void); -/* - * append_target -- - * Build a list of comma-separated targets. - */ -static int -append_target(WT_SESSION *session, const char *target, char **bufp) -{ - static size_t len = 0, remain = 0; - static char *buf = NULL; - static bool first = true; - - /* 20 bytes of slop */ - if (buf == NULL || remain < strlen(target) + 20) { - len += strlen(target) + 512; - remain += strlen(target) + 512; - if ((buf = realloc(buf, len)) == NULL) - return (util_err(session, errno, NULL)); - *bufp = buf; - } - if (first) { - first = false; - strcpy(buf, "target=("); - } else - buf[strlen(buf) - 1] = ','; /* overwrite previous ")" */ - strcat(buf, "\""); - strcat(buf, target); - strcat(buf, "\")"); - remain -= strlen(target) + 1; - - return (0); -} - int util_backup(WT_SESSION *session, int argc, char *argv[]) { WT_CURSOR *cursor; + WT_DECL_ITEM(tmp); WT_DECL_RET; + WT_SESSION_IMPL *session_impl; int ch; - char *config; const char *directory, *name; + bool target; - config = NULL; + session_impl = (WT_SESSION_IMPL *)session; + + target = false; while ((ch = __wt_getopt(progname, argc, argv, "t:")) != EOF) switch (ch) { case 't': - if (append_target(session, __wt_optarg, &config)) - return (1); + if (!target) { + WT_ERR(__wt_scr_alloc(session_impl, 0, &tmp)); + WT_ERR(__wt_buf_fmt( + session_impl, tmp, "%s", "target=(")); + } + WT_ERR(__wt_buf_catfmt(session_impl, tmp, + "%s\"%s\"", target ? "," : "", __wt_optarg)); + target = true; break; case '?': default: @@ -72,8 +50,12 @@ util_backup(WT_SESSION *session, int argc, char *argv[]) } directory = *argv; - if ((ret = session->open_cursor( - session, "backup:", NULL, config, &cursor)) != 0) { + /* Terminate any target. */ + if (target) + WT_ERR(__wt_buf_catfmt(session_impl, tmp, "%s", ")")); + + if ((ret = session->open_cursor(session, "backup:", + NULL, target ? (char *)tmp->data : NULL, &cursor)) != 0) { fprintf(stderr, "%s: cursor open(backup:) failed: %s\n", progname, session->strerror(session, ret)); goto err; @@ -94,7 +76,7 @@ util_backup(WT_SESSION *session, int argc, char *argv[]) goto err; } -err: free(config); +err: __wt_scr_free(session_impl, &tmp); return (ret); } diff --git a/src/third_party/wiredtiger/src/utilities/util_compact.c b/src/third_party/wiredtiger/src/utilities/util_compact.c index c8963a8fda6..3236100c030 100644 --- a/src/third_party/wiredtiger/src/utilities/util_compact.c +++ b/src/third_party/wiredtiger/src/utilities/util_compact.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/utilities/util_cpyright.c b/src/third_party/wiredtiger/src/utilities/util_cpyright.c index 0cfba056387..98e5c919a48 100644 --- a/src/third_party/wiredtiger/src/utilities/util_cpyright.c +++ b/src/third_party/wiredtiger/src/utilities/util_cpyright.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -11,7 +11,7 @@ void util_copyright(void) { - printf("%s\n", "Copyright (c) 2008-2017 MongoDB, Inc."); + printf("%s\n", "Copyright (c) 2008-2018 MongoDB, Inc."); printf("%s\n\n", "All rights reserved."); printf("%s\n\n", diff --git a/src/third_party/wiredtiger/src/utilities/util_create.c b/src/third_party/wiredtiger/src/utilities/util_create.c index 2c7a87fd406..a8ada11ccb1 100644 --- a/src/third_party/wiredtiger/src/utilities/util_create.c +++ b/src/third_party/wiredtiger/src/utilities/util_create.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/utilities/util_downgrade.c b/src/third_party/wiredtiger/src/utilities/util_downgrade.c index 4263727242c..c999fb068da 100644 --- a/src/third_party/wiredtiger/src/utilities/util_downgrade.c +++ b/src/third_party/wiredtiger/src/utilities/util_downgrade.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -28,7 +28,6 @@ util_downgrade(WT_SESSION *session, WT_CONNECTION *conn, int argc, char *argv[]) return (usage()); } argc -= __wt_optind; - argv += __wt_optind; /* * The release argument is required. diff --git a/src/third_party/wiredtiger/src/utilities/util_drop.c b/src/third_party/wiredtiger/src/utilities/util_drop.c index 460c9a6de57..7c4e06da341 100644 --- a/src/third_party/wiredtiger/src/utilities/util_drop.c +++ b/src/third_party/wiredtiger/src/utilities/util_drop.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/utilities/util_dump.c b/src/third_party/wiredtiger/src/utilities/util_dump.c index f768d323afb..73e0e097c31 100644 --- a/src/third_party/wiredtiger/src/utilities/util_dump.c +++ b/src/third_party/wiredtiger/src/utilities/util_dump.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -33,14 +33,17 @@ int util_dump(WT_SESSION *session, int argc, char *argv[]) { WT_CURSOR *cursor; + WT_DECL_ITEM(tmp); WT_DECL_RET; - size_t len; + WT_SESSION_IMPL *session_impl; int ch, i; - char *checkpoint, *config, *p, *simpleuri, *uri; + char *checkpoint, *p, *simpleuri, *uri; bool hex, json, reverse; + session_impl = (WT_SESSION_IMPL *)session; + hex = json = reverse = false; - checkpoint = config = simpleuri = uri = NULL; + checkpoint = simpleuri = uri = NULL; cursor = NULL; while ((ch = __wt_getopt(progname, argc, argv, "c:f:jrx")) != EOF) switch (ch) { @@ -85,6 +88,7 @@ util_dump(WT_SESSION *session, int argc, char *argv[]) dump_prefix(session, hex, json) != 0)) goto err; + WT_RET(__wt_scr_alloc(session_impl, 0, &tmp)); for (i = 0; i < argc; i++) { if (json && i > 0) if (dump_json_separator(session) != 0) @@ -96,24 +100,14 @@ util_dump(WT_SESSION *session, int argc, char *argv[]) if ((uri = util_uri(session, argv[i], "table")) == NULL) goto err; - len = - checkpoint == NULL ? 0 : strlen("checkpoint=") + - strlen(checkpoint) + 1; - len += strlen(json ? "dump=json" : - (hex ? "dump=hex" : "dump=print")); - if ((config = malloc(len + 10)) == NULL) - goto err; - if (checkpoint == NULL) - config[0] = '\0'; - else { - (void)strcpy(config, "checkpoint="); - (void)strcat(config, checkpoint); - (void)strcat(config, ","); - } - (void)strcat(config, json ? "dump=json" : - (hex ? "dump=hex" : "dump=print")); + WT_ERR(__wt_buf_set(session_impl, tmp, "", 0)); + if (checkpoint != NULL) + WT_ERR(__wt_buf_catfmt( + session_impl, tmp, "checkpoint=%s,", checkpoint)); + WT_ERR(__wt_buf_catfmt(session_impl, tmp, + "dump=%s", json ? "json" : (hex ? "hex" : "print"))); if ((ret = session->open_cursor( - session, uri, NULL, config, &cursor)) != 0) { + session, uri, NULL, (char *)tmp->data, &cursor)) != 0) { fprintf(stderr, "%s: cursor open(%s) failed: %s\n", progname, uri, session->strerror(session, ret)); goto err; @@ -147,7 +141,7 @@ util_dump(WT_SESSION *session, int argc, char *argv[]) err: ret = 1; } - free(config); + __wt_scr_free(session_impl, &tmp); free(uri); free(simpleuri); if (cursor != NULL && (ret = cursor->close(cursor)) != 0) { diff --git a/src/third_party/wiredtiger/src/utilities/util_dump.h b/src/third_party/wiredtiger/src/utilities/util_dump.h index 7f037cc659a..fa66c5e1789 100644 --- a/src/third_party/wiredtiger/src/utilities/util_dump.h +++ b/src/third_party/wiredtiger/src/utilities/util_dump.h @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/utilities/util_list.c b/src/third_party/wiredtiger/src/utilities/util_list.c index e64768bf1e5..7ff17394f79 100644 --- a/src/third_party/wiredtiger/src/utilities/util_list.c +++ b/src/third_party/wiredtiger/src/utilities/util_list.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/utilities/util_load.c b/src/third_party/wiredtiger/src/utilities/util_load.c index 813ed7481e3..3ed21fc591c 100644 --- a/src/third_party/wiredtiger/src/utilities/util_load.c +++ b/src/third_party/wiredtiger/src/utilities/util_load.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/utilities/util_load.h b/src/third_party/wiredtiger/src/utilities/util_load.h index 53fce665ddc..c7ba98591e9 100644 --- a/src/third_party/wiredtiger/src/utilities/util_load.h +++ b/src/third_party/wiredtiger/src/utilities/util_load.h @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -22,7 +22,9 @@ int config_reorder(WT_SESSION *, char **); int config_update(WT_SESSION *, char **); /* Flags for util_load_json */ -#define LOAD_JSON_APPEND 0x0001 /* append (ignore record number keys) */ -#define LOAD_JSON_NO_OVERWRITE 0x0002 /* don't overwrite existing data */ +/* AUTOMATIC FLAG VALUE GENERATION START */ +#define LOAD_JSON_APPEND 0x1u /* append (ignore record number keys) */ +#define LOAD_JSON_NO_OVERWRITE 0x2u /* don't overwrite existing data */ +/* AUTOMATIC FLAG VALUE GENERATION STOP */ int util_load_json(WT_SESSION *, const char *, uint32_t); diff --git a/src/third_party/wiredtiger/src/utilities/util_load_json.c b/src/third_party/wiredtiger/src/utilities/util_load_json.c index 92c30bdd78b..91358fd29f0 100644 --- a/src/third_party/wiredtiger/src/utilities/util_load_json.c +++ b/src/third_party/wiredtiger/src/utilities/util_load_json.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -354,8 +354,9 @@ json_top_level(WT_SESSION *session, JSON_INPUT_STATE *ins, uint32_t flags) WT_DECL_RET; static const char *json_markers[] = { "\"config\"", "\"colgroups\"", "\"indices\"", "\"data\"", NULL }; + uint64_t curversion; + int toktype; char *config, *tableuri; - int curversion, toktype; bool hasversion; memset(&cl, 0, sizeof(cl)); @@ -380,8 +381,10 @@ json_top_level(WT_SESSION *session, JSON_INPUT_STATE *ins, uint32_t flags) } hasversion = true; JSON_EXPECT(session, ins, 's'); - if ((curversion = atoi(ins->tokstart + 1)) <= 0 || - curversion > DUMP_JSON_SUPPORTED_VERSION) { + if ((ret = util_str2num(session, + ins->tokstart + 1, false, &curversion)) != 0) + goto err; + if (curversion > DUMP_JSON_SUPPORTED_VERSION) { ret = util_err(session, ENOTSUP, "unsupported JSON dump version \"%.*s\"", (int)(ins->toklen - 1), ins->tokstart + 1); diff --git a/src/third_party/wiredtiger/src/utilities/util_loadtext.c b/src/third_party/wiredtiger/src/utilities/util_loadtext.c index 1053ab89694..1519a0e4fa5 100644 --- a/src/third_party/wiredtiger/src/utilities/util_loadtext.c +++ b/src/third_party/wiredtiger/src/utilities/util_loadtext.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/utilities/util_main.c b/src/third_party/wiredtiger/src/utilities/util_main.c index d712a2bec05..2d08c4c5274 100644 --- a/src/third_party/wiredtiger/src/utilities/util_main.c +++ b/src/third_party/wiredtiger/src/utilities/util_main.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -41,7 +41,7 @@ usage(void) "\t" "compact\t compact an object\n" "\t" "copyright copyright information\n" "\t" "create\t create an object\n" - "\t" "downgrade\t downgrade an database\n" + "\t" "downgrade downgrade a database\n" "\t" "drop\t drop an object\n" "\t" "dump\t dump an object\n" "\t" "list\t list database objects\n" diff --git a/src/third_party/wiredtiger/src/utilities/util_misc.c b/src/third_party/wiredtiger/src/utilities/util_misc.c index d0fe35ff370..63d02a32472 100644 --- a/src/third_party/wiredtiger/src/utilities/util_misc.c +++ b/src/third_party/wiredtiger/src/utilities/util_misc.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -94,13 +94,13 @@ util_read_line(WT_SESSION *session, ULINE *l, bool eof_expected, bool *eofp) } /* - * util_str2recno -- - * Convert a string to a record number. + * util_str2num -- + * Convert a string to a number. */ int -util_str2recno(WT_SESSION *session, const char *p, uint64_t *recnop) +util_str2num(WT_SESSION *session, const char *p, bool endnul, uint64_t *vp) { - uint64_t recno; + uint64_t v; char *endptr; /* @@ -112,16 +112,20 @@ util_str2recno(WT_SESSION *session, const char *p, uint64_t *recnop) goto format; errno = 0; - recno = __wt_strtouq(p, &endptr, 0); - if (recno == ULLONG_MAX && errno == ERANGE) - return ( - util_err(session, ERANGE, "%s: invalid record number", p)); + v = __wt_strtouq(p, &endptr, 0); + if (v == ULLONG_MAX && errno == ERANGE) + return (util_err(session, ERANGE, "%s: invalid number", p)); - if (endptr[0] != '\0') -format: return ( - util_err(session, EINVAL, "%s: invalid record number", p)); + /* + * In most cases we expect the number to be a string and end with a + * nul byte (and we want to confirm that because it's a user-entered + * command-line argument), but we allow the caller to configure that + * test off. + */ + if (endnul && endptr[0] != '\0') +format: return (util_err(session, EINVAL, "%s: invalid number", p)); - *recnop = recno; + *vp = v; return (0); } diff --git a/src/third_party/wiredtiger/src/utilities/util_printlog.c b/src/third_party/wiredtiger/src/utilities/util_printlog.c index 2e5ae3aa926..a0181ff97f2 100644 --- a/src/third_party/wiredtiger/src/utilities/util_printlog.c +++ b/src/third_party/wiredtiger/src/utilities/util_printlog.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -35,7 +35,6 @@ util_printlog(WT_SESSION *session, int argc, char *argv[]) return (usage()); } argc -= __wt_optind; - argv += __wt_optind; /* There should not be any more arguments. */ if (argc != 0) diff --git a/src/third_party/wiredtiger/src/utilities/util_read.c b/src/third_party/wiredtiger/src/utilities/util_read.c index 9a5954f95e4..ab2b2a79968 100644 --- a/src/third_party/wiredtiger/src/utilities/util_read.c +++ b/src/third_party/wiredtiger/src/utilities/util_read.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -74,7 +74,7 @@ util_read(WT_SESSION *session, int argc, char *argv[]) */ for (rval = false; *++argv != NULL;) { if (rkey) { - if (util_str2recno(session, *argv, &recno)) + if (util_str2num(session, *argv, true, &recno)) return (1); cursor->set_key(cursor, recno); } else diff --git a/src/third_party/wiredtiger/src/utilities/util_rebalance.c b/src/third_party/wiredtiger/src/utilities/util_rebalance.c index f58f086e777..151474923c7 100644 --- a/src/third_party/wiredtiger/src/utilities/util_rebalance.c +++ b/src/third_party/wiredtiger/src/utilities/util_rebalance.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/utilities/util_rename.c b/src/third_party/wiredtiger/src/utilities/util_rename.c index 51e936100ff..2321fbb32b5 100644 --- a/src/third_party/wiredtiger/src/utilities/util_rename.c +++ b/src/third_party/wiredtiger/src/utilities/util_rename.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/utilities/util_salvage.c b/src/third_party/wiredtiger/src/utilities/util_salvage.c index dc311b5ee9a..ee7b64d8f98 100644 --- a/src/third_party/wiredtiger/src/utilities/util_salvage.c +++ b/src/third_party/wiredtiger/src/utilities/util_salvage.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/utilities/util_stat.c b/src/third_party/wiredtiger/src/utilities/util_stat.c index f9c66a15885..eb3f64c6240 100644 --- a/src/third_party/wiredtiger/src/utilities/util_stat.c +++ b/src/third_party/wiredtiger/src/utilities/util_stat.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/utilities/util_truncate.c b/src/third_party/wiredtiger/src/utilities/util_truncate.c index 101fa23c0cb..02b25086657 100644 --- a/src/third_party/wiredtiger/src/utilities/util_truncate.c +++ b/src/third_party/wiredtiger/src/utilities/util_truncate.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/utilities/util_upgrade.c b/src/third_party/wiredtiger/src/utilities/util_upgrade.c index 1ad95bf3cf2..6672af8fd39 100644 --- a/src/third_party/wiredtiger/src/utilities/util_upgrade.c +++ b/src/third_party/wiredtiger/src/utilities/util_upgrade.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/utilities/util_verbose.c b/src/third_party/wiredtiger/src/utilities/util_verbose.c index 979cd7451c1..2f675a66bc2 100644 --- a/src/third_party/wiredtiger/src/utilities/util_verbose.c +++ b/src/third_party/wiredtiger/src/utilities/util_verbose.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/utilities/util_verify.c b/src/third_party/wiredtiger/src/utilities/util_verify.c index ac8bfe8a5bc..0c28120995c 100644 --- a/src/third_party/wiredtiger/src/utilities/util_verify.c +++ b/src/third_party/wiredtiger/src/utilities/util_verify.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * diff --git a/src/third_party/wiredtiger/src/utilities/util_write.c b/src/third_party/wiredtiger/src/utilities/util_write.c index fa03201cd1f..da958f86c2d 100644 --- a/src/third_party/wiredtiger/src/utilities/util_write.c +++ b/src/third_party/wiredtiger/src/utilities/util_write.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2014-2017 MongoDB, Inc. + * Copyright (c) 2014-2018 MongoDB, Inc. * Copyright (c) 2008-2014 WiredTiger, Inc. * All rights reserved. * @@ -92,7 +92,7 @@ util_write(WT_SESSION *session, int argc, char *argv[]) while (*++argv != NULL) { if (!append) { if (rkey) { - if (util_str2recno(session, *argv, &recno)) + if (util_str2num(session, *argv, true, &recno)) return (1); cursor->set_key(cursor, recno); } else diff --git a/src/third_party/wiredtiger/test/bloom/test_bloom.c b/src/third_party/wiredtiger/test/bloom/test_bloom.c index e545565057b..10607a3719c 100644 --- a/src/third_party/wiredtiger/test/bloom/test_bloom.c +++ b/src/third_party/wiredtiger/test/bloom/test_bloom.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/checkpoint/checkpointer.c b/src/third_party/wiredtiger/test/checkpoint/checkpointer.c index ae499c2e96b..170cde22fa1 100644 --- a/src/third_party/wiredtiger/test/checkpoint/checkpointer.c +++ b/src/third_party/wiredtiger/test/checkpoint/checkpointer.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/checkpoint/test_checkpoint.c b/src/third_party/wiredtiger/test/checkpoint/test_checkpoint.c index a646482b392..d5673f51cb2 100644 --- a/src/third_party/wiredtiger/test/checkpoint/test_checkpoint.c +++ b/src/third_party/wiredtiger/test/checkpoint/test_checkpoint.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/checkpoint/test_checkpoint.h b/src/third_party/wiredtiger/test/checkpoint/test_checkpoint.h index 36551211b7e..93cebf762fd 100644 --- a/src/third_party/wiredtiger/test/checkpoint/test_checkpoint.h +++ b/src/third_party/wiredtiger/test/checkpoint/test_checkpoint.h @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/checkpoint/workers.c b/src/third_party/wiredtiger/test/checkpoint/workers.c index cd32db6746f..c72b4b897b1 100644 --- a/src/third_party/wiredtiger/test/checkpoint/workers.c +++ b/src/third_party/wiredtiger/test/checkpoint/workers.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/csuite/Makefile.am b/src/third_party/wiredtiger/test/csuite/Makefile.am index f564d85c6d8..5a16a419aba 100644 --- a/src/third_party/wiredtiger/test/csuite/Makefile.am +++ b/src/third_party/wiredtiger/test/csuite/Makefile.am @@ -95,6 +95,10 @@ test_wt3363_checkpoint_op_races_SOURCES = wt3363_checkpoint_op_races/main.c noinst_PROGRAMS += test_wt3363_checkpoint_op_races all_TESTS += test_wt3363_checkpoint_op_races +test_wt3874_pad_byte_collator_SOURCES = wt3874_pad_byte_collator/main.c +noinst_PROGRAMS += test_wt3874_pad_byte_collator +all_TESTS += test_wt3874_pad_byte_collator + test_rwlock_SOURCES = rwlock/main.c noinst_PROGRAMS += test_rwlock all_TESTS += test_rwlock diff --git a/src/third_party/wiredtiger/test/csuite/random_abort/main.c b/src/third_party/wiredtiger/test/csuite/random_abort/main.c index ad49f01dde5..e99ed5ecd4d 100644 --- a/src/third_party/wiredtiger/test/csuite/random_abort/main.c +++ b/src/third_party/wiredtiger/test/csuite/random_abort/main.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. @@ -391,9 +391,13 @@ main(int argc, char *argv[]) */ for (last_key = UINT64_MAX;; ++count, last_key = key) { ret = fscanf(fp, "%" SCNu64 "\n", &key); - if (ret != EOF && ret != 1) - testutil_die(errno, "fscanf"); - if (ret == EOF) + /* + * Consider anything other than clear success in + * getting the key to be EOF. We've seen file system + * issues where the file ends with zeroes on a 4K + * boundary and does not return EOF but a ret of zero. + */ + if (ret != 1) break; /* * If we're unlucky, the last line may be a partially diff --git a/src/third_party/wiredtiger/test/csuite/rwlock/main.c b/src/third_party/wiredtiger/test/csuite/rwlock/main.c index f1bad3f538f..2b4e9144fe4 100644 --- a/src/third_party/wiredtiger/test/csuite/rwlock/main.c +++ b/src/third_party/wiredtiger/test/csuite/rwlock/main.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/csuite/scope/main.c b/src/third_party/wiredtiger/test/csuite/scope/main.c index e9035775ba5..eed9207951a 100644 --- a/src/third_party/wiredtiger/test/csuite/scope/main.c +++ b/src/third_party/wiredtiger/test/csuite/scope/main.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/csuite/timestamp_abort/main.c b/src/third_party/wiredtiger/test/csuite/timestamp_abort/main.c index 79b232b532a..2cf9a69110c 100644 --- a/src/third_party/wiredtiger/test/csuite/timestamp_abort/main.c +++ b/src/third_party/wiredtiger/test/csuite/timestamp_abort/main.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. @@ -476,7 +476,7 @@ print_missing(REPORT *r, const char *fname, const char *msg) ". Then keys %" PRIu64 "-%" PRIu64 " exist." " Key range %" PRIu64 "-%" PRIu64 "\n", fname, msg, - r->exist_key - r->first_miss - 1, + (r->exist_key - r->first_miss) - 1, r->first_miss, r->exist_key - 1, r->exist_key, r->last_key, r->first_key, r->last_key); @@ -655,7 +655,8 @@ main(int argc, char *argv[]) "rm -rf ../%s.SAVE && mkdir ../%s.SAVE && " "cp -p WiredTigerLog.* ../%s.SAVE", home, home, home)); - (void)system(buf); + if ((status = system(buf)) < 0) + testutil_die(status, "system: %s", buf); printf("Open database, run recovery and verify content\n"); /* diff --git a/src/third_party/wiredtiger/test/csuite/truncated_log/main.c b/src/third_party/wiredtiger/test/csuite/truncated_log/main.c index b5da289efb3..be28a8d9500 100644 --- a/src/third_party/wiredtiger/test/csuite/truncated_log/main.c +++ b/src/third_party/wiredtiger/test/csuite/truncated_log/main.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/csuite/wt1965_col_efficiency/main.c b/src/third_party/wiredtiger/test/csuite/wt1965_col_efficiency/main.c index 265fef8e76f..39fc9dc03a2 100644 --- a/src/third_party/wiredtiger/test/csuite/wt1965_col_efficiency/main.c +++ b/src/third_party/wiredtiger/test/csuite/wt1965_col_efficiency/main.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/csuite/wt2246_col_append/main.c b/src/third_party/wiredtiger/test/csuite/wt2246_col_append/main.c index bf2d4b8ffc4..fca3d8fcd00 100644 --- a/src/third_party/wiredtiger/test/csuite/wt2246_col_append/main.c +++ b/src/third_party/wiredtiger/test/csuite/wt2246_col_append/main.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/csuite/wt2323_join_visibility/main.c b/src/third_party/wiredtiger/test/csuite/wt2323_join_visibility/main.c index b779281a0c0..0de304e61bd 100644 --- a/src/third_party/wiredtiger/test/csuite/wt2323_join_visibility/main.c +++ b/src/third_party/wiredtiger/test/csuite/wt2323_join_visibility/main.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/csuite/wt2403_lsm_workload/main.c b/src/third_party/wiredtiger/test/csuite/wt2403_lsm_workload/main.c index 214276bda21..c039cd80d31 100644 --- a/src/third_party/wiredtiger/test/csuite/wt2403_lsm_workload/main.c +++ b/src/third_party/wiredtiger/test/csuite/wt2403_lsm_workload/main.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/csuite/wt2447_join_main_table/main.c b/src/third_party/wiredtiger/test/csuite/wt2447_join_main_table/main.c index 8ad721c8d51..85a581d73c9 100644 --- a/src/third_party/wiredtiger/test/csuite/wt2447_join_main_table/main.c +++ b/src/third_party/wiredtiger/test/csuite/wt2447_join_main_table/main.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/csuite/wt2535_insert_race/main.c b/src/third_party/wiredtiger/test/csuite/wt2535_insert_race/main.c index f5848203f37..a0239e40a8a 100644 --- a/src/third_party/wiredtiger/test/csuite/wt2535_insert_race/main.c +++ b/src/third_party/wiredtiger/test/csuite/wt2535_insert_race/main.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/csuite/wt2592_join_schema/main.c b/src/third_party/wiredtiger/test/csuite/wt2592_join_schema/main.c index acdb6d80b27..1bc202954ab 100644 --- a/src/third_party/wiredtiger/test/csuite/wt2592_join_schema/main.c +++ b/src/third_party/wiredtiger/test/csuite/wt2592_join_schema/main.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/csuite/wt2695_checksum/main.c b/src/third_party/wiredtiger/test/csuite/wt2695_checksum/main.c index 3bd9bfca3c0..8a130f8d958 100644 --- a/src/third_party/wiredtiger/test/csuite/wt2695_checksum/main.c +++ b/src/third_party/wiredtiger/test/csuite/wt2695_checksum/main.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/csuite/wt2719_reconfig/main.c b/src/third_party/wiredtiger/test/csuite/wt2719_reconfig/main.c index 58e2a0bc113..739f8c77fc4 100644 --- a/src/third_party/wiredtiger/test/csuite/wt2719_reconfig/main.c +++ b/src/third_party/wiredtiger/test/csuite/wt2719_reconfig/main.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. @@ -149,7 +149,6 @@ static const char * const list[] = { ",statistics_log=(wait=37)", ",statistics_log=(wait=0)", -#ifdef HAVE_VERBOSE ",verbose=(\"api\")", ",verbose=(\"block\")", ",verbose=(\"checkpoint\")", @@ -176,7 +175,6 @@ static const char * const list[] = { ",verbose=(\"version\")", ",verbose=(\"write\")", ",verbose=()" -#endif }; static int diff --git a/src/third_party/wiredtiger/test/csuite/wt2834_join_bloom_fix/main.c b/src/third_party/wiredtiger/test/csuite/wt2834_join_bloom_fix/main.c index acce1007f79..de05c459e10 100644 --- a/src/third_party/wiredtiger/test/csuite/wt2834_join_bloom_fix/main.c +++ b/src/third_party/wiredtiger/test/csuite/wt2834_join_bloom_fix/main.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. @@ -67,11 +67,15 @@ main(int argc, char *argv[]) memset(opts, 0, sizeof(*opts)); testutil_check(testutil_parse_opts(argc, argv, opts)); testutil_make_work_dir(opts->home); + testutil_progress(opts, "start"); testutil_check(wiredtiger_open(opts->home, NULL, "create,cache_size=250M", &opts->conn)); + testutil_progress(opts, "wiredtiger_open"); testutil_check( opts->conn->open_session(opts->conn, NULL, NULL, &session)); + testutil_progress(opts, "sessions opened"); + /* * Note: repeated primary key 'id' as 'id2'. This makes * it easier to dump an index and know which record we're @@ -97,6 +101,7 @@ main(int argc, char *argv[]) testutil_check(session->create(session, balanceuri, "columns=(balance)")); testutil_check(session->create(session, flaguri, "columns=(flag)")); + testutil_progress(opts, "setup complete"); /* * Insert a single record with all items we are search for, @@ -110,7 +115,9 @@ main(int argc, char *argv[]) testutil_check(maincur->close(maincur)); testutil_check(session->close(session, NULL)); + testutil_progress(opts, "populate start"); populate(opts); + testutil_progress(opts, "populate end"); testutil_check(opts->conn->open_session( opts->conn, NULL, NULL, &session)); @@ -159,10 +166,11 @@ main(int argc, char *argv[]) testutil_assert(ret == WT_NOTFOUND); testutil_assert(count == 0); + testutil_progress(opts, "cleanup starting"); +#if 0 testutil_cleanup(opts); - /* NOTREACHED */ - - return (0); +#endif + return (EXIT_SUCCESS); } void diff --git a/src/third_party/wiredtiger/test/csuite/wt2853_perf/main.c b/src/third_party/wiredtiger/test/csuite/wt2853_perf/main.c index 3bee610bfa6..93fb94836c0 100644 --- a/src/third_party/wiredtiger/test/csuite/wt2853_perf/main.c +++ b/src/third_party/wiredtiger/test/csuite/wt2853_perf/main.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. @@ -97,12 +97,15 @@ main(int argc, char *argv[]) sharedopts->bloom = BLOOM; testutil_check(testutil_parse_opts(argc, argv, opts)); testutil_make_work_dir(opts->home); + testutil_progress(opts, "start"); testutil_check(wiredtiger_open(opts->home, NULL, "create,cache_size=1G", &opts->conn)); + testutil_progress(opts, "wiredtiger_open"); testutil_check( opts->conn->open_session(opts->conn, NULL, NULL, &session)); + testutil_progress(opts, "sessions opened"); /* * Note: id is repeated as id2. This makes it easier to @@ -143,6 +146,7 @@ main(int argc, char *argv[]) testutil_check(maincur->insert(maincur)); testutil_check(maincur->close(maincur)); testutil_check(session->close(session, NULL)); + testutil_progress(opts, "setup complete"); for (i = 0; i < N_INSERT_THREAD; ++i) { insert_args[i].threadnum = i; @@ -161,6 +165,7 @@ main(int argc, char *argv[]) testutil_check(pthread_create( &get_tid[i], NULL, thread_get, &get_args[i])); } + testutil_progress(opts, "threads started"); /* * Wait for insert threads to finish. When they @@ -175,6 +180,7 @@ main(int argc, char *argv[]) for (i = 0; i < N_GET_THREAD; ++i) testutil_check(pthread_join(get_tid[i], NULL)); + testutil_progress(opts, "threads joined"); fprintf(stderr, "\n"); for (i = 0; i < N_GET_THREAD; ++i) { fprintf(stderr, " thread %d did %d joins (%d fails)\n", i, @@ -183,7 +189,10 @@ main(int argc, char *argv[]) } testutil_assert(nfail == 0); + testutil_progress(opts, "cleanup starting"); +#if 0 testutil_cleanup(opts); +#endif return (0); } @@ -212,6 +221,7 @@ thread_insert(void *arg) testutil_check(session->open_cursor(session, opts->uri, NULL, NULL, &maincur)); + testutil_progress(opts, "insert start"); for (i = 0; i < N_INSERT; i++) { /* * Insert threads may stomp on each other's records; @@ -242,6 +252,7 @@ thread_insert(void *arg) fprintf(stderr, "."); (void)time(&curtime); if ((elapsed = difftime(curtime, prevtime)) > 5.0) { + testutil_progress(opts, "insert time gap"); fprintf(stderr, "\n" "GAP: %.0f secs after %d inserts\n", elapsed, i); @@ -250,6 +261,7 @@ thread_insert(void *arg) prevtime = curtime; } } + testutil_progress(opts, "insert end"); testutil_check(maincur->close(maincur)); testutil_check(session->close(session, NULL)); return (NULL); @@ -281,6 +293,7 @@ thread_get(void *arg) testutil_check(session->open_cursor( session, sharedopts->posturi, NULL, NULL, &postcur)); + testutil_progress(opts, "get start"); for (threadargs->njoins = 0; threadargs->done == 0; threadargs->njoins++) { testutil_check(session->begin_transaction(session, NULL)); @@ -290,20 +303,14 @@ thread_get(void *arg) testutil_check(postcur->get_key(postcur, &post)); testutil_check(postcur->get_value(postcur, &post2, &bal, &extra, &flag, &key)); - testutil_assert(post == post2); - if (post != 54321) - break; + testutil_assert((flag > 0 && bal < 0) || + (flag == 0 && bal >= 0)); maincur->set_key(maincur, key); testutil_check(maincur->search(maincur)); testutil_check(maincur->get_value(maincur, &post2, &bal2, &extra, &flag2, &key2)); testutil_check(maincur->reset(maincur)); - testutil_assert(key == key2); - testutil_assert(post == post2); - testutil_assert(bal == bal2); - testutil_assert(flag == flag2); - testutil_assert((flag2 > 0 && bal2 < 0) || (flag2 == 0 && bal2 >= 0)); } @@ -318,6 +325,7 @@ thread_get(void *arg) (void)time(&curtime); if ((elapsed = difftime(curtime, prevtime)) > 5.0) { + testutil_progress(opts, "get time gap"); fprintf(stderr, "\n" "GAP: %.0f secs after %d gets\n", elapsed, threadargs->njoins); @@ -325,6 +333,7 @@ thread_get(void *arg) } prevtime = curtime; } + testutil_progress(opts, "get end"); testutil_check(postcur->close(postcur)); testutil_check(maincur->close(maincur)); testutil_check(session->close(session, NULL)); diff --git a/src/third_party/wiredtiger/test/csuite/wt2909_checkpoint_integrity/main.c b/src/third_party/wiredtiger/test/csuite/wt2909_checkpoint_integrity/main.c index 22b77d8f4f4..fd713e50ba0 100644 --- a/src/third_party/wiredtiger/test/csuite/wt2909_checkpoint_integrity/main.c +++ b/src/third_party/wiredtiger/test/csuite/wt2909_checkpoint_integrity/main.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/csuite/wt2999_join_extractor/main.c b/src/third_party/wiredtiger/test/csuite/wt2999_join_extractor/main.c index 194ff143610..502c0d05a31 100644 --- a/src/third_party/wiredtiger/test/csuite/wt2999_join_extractor/main.c +++ b/src/third_party/wiredtiger/test/csuite/wt2999_join_extractor/main.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/csuite/wt3120_filesys/main.c b/src/third_party/wiredtiger/test/csuite/wt3120_filesys/main.c index 2fae85017d4..82d8cae5d04 100644 --- a/src/third_party/wiredtiger/test/csuite/wt3120_filesys/main.c +++ b/src/third_party/wiredtiger/test/csuite/wt3120_filesys/main.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/csuite/wt3135_search_near_collator/main.c b/src/third_party/wiredtiger/test/csuite/wt3135_search_near_collator/main.c index 103a502f808..d880093a074 100644 --- a/src/third_party/wiredtiger/test/csuite/wt3135_search_near_collator/main.c +++ b/src/third_party/wiredtiger/test/csuite/wt3135_search_near_collator/main.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/csuite/wt3184_dup_index_collator/main.c b/src/third_party/wiredtiger/test/csuite/wt3184_dup_index_collator/main.c index c6dd726e58a..94592bc8e59 100644 --- a/src/third_party/wiredtiger/test/csuite/wt3184_dup_index_collator/main.c +++ b/src/third_party/wiredtiger/test/csuite/wt3184_dup_index_collator/main.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/csuite/wt3338_partial_update/main.c b/src/third_party/wiredtiger/test/csuite/wt3338_partial_update/main.c index e4b265985f7..fb246f87da1 100644 --- a/src/third_party/wiredtiger/test/csuite/wt3338_partial_update/main.c +++ b/src/third_party/wiredtiger/test/csuite/wt3338_partial_update/main.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. @@ -253,21 +253,21 @@ static int nruns = 1000; * Run some tests. */ static void -modify_run(WT_ITEM *local, WT_ITEM *library, bool verbose) +modify_run(WT_CURSOR *cursor, WT_ITEM *local, bool verbose) { int i, j; for (i = 0; i < nruns; ++i) { - modify_init(local, library); + modify_init(local, &cursor->value); for (j = 0; j < 1000; ++j) { modify_build(); slow_apply_api(local); testutil_check(__wt_modify_apply_api( - NULL, library, entries, nentries)); + NULL, cursor, entries, nentries)); - diff(local, library); + diff(local, &cursor->value); } if (verbose) { printf("%d (%d%%)\r", i, (i * 100) / nruns); @@ -282,7 +282,8 @@ int main(int argc, char *argv[]) { TEST_OPTS *opts, _opts; - WT_ITEM *library, _library, *local, _local; + WT_CURSOR *cursor, _cursor; + WT_ITEM *local, _local; if (testutil_is_flag_set("TESTUTIL_ENABLE_LONG_TESTS")) nruns = 10000; @@ -300,17 +301,18 @@ main(int argc, char *argv[]) /* Set up replacement information. */ modify_repl_init(); - /* Allocate a pair of buffers. */ + /* We need two items, one of them hooked into fake cursor. */ local = &_local; memset(&_local, 0, sizeof(_local)); - library = &_library; - memset(&_library, 0, sizeof(_library)); + cursor = &_cursor; + memset(&_cursor, 0, sizeof(_cursor)); + cursor->value_format = "u"; /* Run the test. */ - modify_run(local, library, opts->verbose); + modify_run(cursor, local, opts->verbose); __wt_buf_free(NULL, local); - __wt_buf_free(NULL, library); + __wt_buf_free(NULL, &cursor->value); testutil_cleanup(opts); return (EXIT_SUCCESS); diff --git a/src/third_party/wiredtiger/test/csuite/wt3363_checkpoint_op_races/main.c b/src/third_party/wiredtiger/test/csuite/wt3363_checkpoint_op_races/main.c index 9cb1ab0f4c6..47a0f9df760 100644 --- a/src/third_party/wiredtiger/test/csuite/wt3363_checkpoint_op_races/main.c +++ b/src/third_party/wiredtiger/test/csuite/wt3363_checkpoint_op_races/main.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/csuite/wt3874_pad_byte_collator/main.c b/src/third_party/wiredtiger/test/csuite/wt3874_pad_byte_collator/main.c new file mode 100644 index 00000000000..c985b1f8f32 --- /dev/null +++ b/src/third_party/wiredtiger/test/csuite/wt3874_pad_byte_collator/main.c @@ -0,0 +1,115 @@ +/*- + * Public Domain 2014-2018 MongoDB, Inc. + * Public Domain 2008-2014 WiredTiger, Inc. + * + * This is free and unencumbered software released into the public domain. + * + * Anyone is free to copy, modify, publish, use, compile, sell, or + * distribute this software, either in source code form or as a compiled + * binary, for any purpose, commercial or non-commercial, and by any + * means. + * + * In jurisdictions that recognize copyright laws, the author or authors + * of this software dedicate any and all copyright interest in the + * software to the public domain. We make this dedication for the benefit + * of the public at large and to the detriment of our heirs and + * successors. We intend this dedication to be an overt act of + * relinquishment in perpetuity of all present and future rights to this + * software under copyright law. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. + * IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + */ +#include "test_util.h" + +/* + * JIRA ticket reference: WT-3874 + * Test case description: Set up a collator that only uses the first + * byte of a record for comparison; all other bytes are considered padding. + * With that collator for a table, insert an item, then remove that + * item (with different padding). + * Failure mode: An assertion is fired when we get back the key as stored + * in the record, if we compare it to the given key without taking into + * account the collator. + */ + +#include <assert.h> +#include <stdio.h> +#include <stdlib.h> +#include <string.h> + +#include <wiredtiger.h> + +#define KEY_SIZE 20 + +static int +my_compare(WT_COLLATOR *collator, WT_SESSION *session, + const WT_ITEM *v1, const WT_ITEM *v2, int *cmp) +{ + (void)collator; + (void)session; + + if (v1->size < 1 || v2->size < 1) + return (EINVAL); + *cmp = strncmp((const char *)v1->data, (const char *)v2->data, 1); + return (0); +} + +static WT_COLLATOR my_coll = { my_compare, NULL, NULL }; + +int +main(int argc, char *argv[]) +{ + TEST_OPTS *opts, _opts; + WT_CONNECTION *conn; + WT_SESSION *session; + WT_CURSOR *cursor; + char buf[KEY_SIZE]; + WT_ITEM key; + + opts = &_opts; + memset(opts, 0, sizeof(*opts)); + srand(123); + + testutil_check(testutil_parse_opts(argc, argv, opts)); + testutil_make_work_dir(opts->home); + + testutil_check(wiredtiger_open(opts->home, NULL, "create,log=(enabled)", + &opts->conn)); + conn = opts->conn; + testutil_check(conn->add_collator(conn, "my_coll", &my_coll, NULL)); + testutil_check(conn->open_session(opts->conn, NULL, NULL, &session)); + + testutil_check(session->create(session, "table:main", + "key_format=u,value_format=u,collator=my_coll")); + + testutil_check(session->open_cursor( + session, "table:main", NULL, NULL, &cursor)); + + memset(buf, 'X', sizeof(buf)); + buf[0] = 'a'; + + key.data = buf; + key.size = sizeof(buf); + cursor->set_key(cursor, &key); + cursor->set_value(cursor, &key); + testutil_check(cursor->insert(cursor)); + + testutil_check(session->checkpoint(session, NULL)); + + /* Use a different padding. */ + memset(buf, 'Y', sizeof(buf)); + buf[0] = 'a'; + + cursor->set_key(cursor, &key); + testutil_check(cursor->remove(cursor)); + + testutil_check(session->close(session, NULL)); + testutil_cleanup(opts); + return (EXIT_SUCCESS); +} diff --git a/src/third_party/wiredtiger/test/cursor_order/cursor_order.c b/src/third_party/wiredtiger/test/cursor_order/cursor_order.c index 2c9cc1410de..b2e847f880c 100644 --- a/src/third_party/wiredtiger/test/cursor_order/cursor_order.c +++ b/src/third_party/wiredtiger/test/cursor_order/cursor_order.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/cursor_order/cursor_order.h b/src/third_party/wiredtiger/test/cursor_order/cursor_order.h index ab9f94850df..12a5732bd0f 100644 --- a/src/third_party/wiredtiger/test/cursor_order/cursor_order.h +++ b/src/third_party/wiredtiger/test/cursor_order/cursor_order.h @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/cursor_order/cursor_order_file.c b/src/third_party/wiredtiger/test/cursor_order/cursor_order_file.c index 3473cb16607..4706703b054 100644 --- a/src/third_party/wiredtiger/test/cursor_order/cursor_order_file.c +++ b/src/third_party/wiredtiger/test/cursor_order/cursor_order_file.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/cursor_order/cursor_order_ops.c b/src/third_party/wiredtiger/test/cursor_order/cursor_order_ops.c index 576860483ba..2ff4b5ed5fb 100644 --- a/src/third_party/wiredtiger/test/cursor_order/cursor_order_ops.c +++ b/src/third_party/wiredtiger/test/cursor_order/cursor_order_ops.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/fops/file.c b/src/third_party/wiredtiger/test/fops/file.c index 62f1cc8b5bf..9437230afbd 100644 --- a/src/third_party/wiredtiger/test/fops/file.c +++ b/src/third_party/wiredtiger/test/fops/file.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/fops/fops.c b/src/third_party/wiredtiger/test/fops/fops.c index 9f2257d8e96..99c333be4a0 100644 --- a/src/third_party/wiredtiger/test/fops/fops.c +++ b/src/third_party/wiredtiger/test/fops/fops.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/fops/t.c b/src/third_party/wiredtiger/test/fops/t.c index fcbbdcabd73..e748918a08a 100644 --- a/src/third_party/wiredtiger/test/fops/t.c +++ b/src/third_party/wiredtiger/test/fops/t.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. @@ -160,7 +160,8 @@ wt_startup(char *config_open) testutil_make_work_dir(home); testutil_check(__wt_snprintf(config_buf, sizeof(config_buf), - "create,error_prefix=\"%s\",cache_size=5MB%s%s", + "create,error_prefix=\"%s\",cache_size=5MB%s%s," + "operation_tracking=(enabled=false)", progname, config_open == NULL ? "" : ",", config_open == NULL ? "" : config_open)); diff --git a/src/third_party/wiredtiger/test/fops/thread.h b/src/third_party/wiredtiger/test/fops/thread.h index 0df36025be0..a936d62f521 100644 --- a/src/third_party/wiredtiger/test/fops/thread.h +++ b/src/third_party/wiredtiger/test/fops/thread.h @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/format/backup.c b/src/third_party/wiredtiger/test/format/backup.c index df4fcd323dc..5ad37905cd0 100644 --- a/src/third_party/wiredtiger/test/format/backup.c +++ b/src/third_party/wiredtiger/test/format/backup.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/format/bdb.c b/src/third_party/wiredtiger/test/format/bdb.c index f3dd9e44f17..a3a9ad3a62f 100644 --- a/src/third_party/wiredtiger/test/format/bdb.c +++ b/src/third_party/wiredtiger/test/format/bdb.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/format/bulk.c b/src/third_party/wiredtiger/test/format/bulk.c index 398471786e6..4f207037419 100644 --- a/src/third_party/wiredtiger/test/format/bulk.c +++ b/src/third_party/wiredtiger/test/format/bulk.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/format/compact.c b/src/third_party/wiredtiger/test/format/compact.c index 2df3839f67b..7331112da9b 100644 --- a/src/third_party/wiredtiger/test/format/compact.c +++ b/src/third_party/wiredtiger/test/format/compact.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/format/config.c b/src/third_party/wiredtiger/test/format/config.c index 769ed608e64..709eb9a4a26 100644 --- a/src/third_party/wiredtiger/test/format/config.c +++ b/src/third_party/wiredtiger/test/format/config.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/format/config.h b/src/third_party/wiredtiger/test/format/config.h index 7ac65147462..565df91d46b 100644 --- a/src/third_party/wiredtiger/test/format/config.h +++ b/src/third_party/wiredtiger/test/format/config.h @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. @@ -35,19 +35,19 @@ typedef struct { const char *desc; /* Configuration description */ /* Value is a boolean, yes if roll of 1-to-100 is <= CONFIG->min. */ -#define C_BOOL 0x001 +#define C_BOOL 0x01u /* Not a simple randomization, handle outside the main loop. */ -#define C_IGNORE 0x002 +#define C_IGNORE 0x02u /* Value was set from command-line or file, ignore for all runs. */ -#define C_PERM 0x004 +#define C_PERM 0x04u /* Value isn't random for this run, ignore just for this run. */ -#define C_TEMP 0x008 +#define C_TEMP 0x08u /* Value is a string. */ -#define C_STRING 0x020 +#define C_STRING 0x20u u_int flags; uint32_t min; /* Minimum value */ @@ -103,7 +103,7 @@ static CONFIG c[] = { { "cache_minimum", "minimum size of the cache in MB", - C_IGNORE, 1, 0, 100 * 1024, &g.c_cache_minimum, NULL }, + C_IGNORE, 0, 0, 100 * 1024, &g.c_cache_minimum, NULL }, { "checkpoints", "type of checkpoints (on | off | wiredtiger)", diff --git a/src/third_party/wiredtiger/test/format/format.h b/src/third_party/wiredtiger/test/format/format.h index af66e166f47..9191a73a134 100644 --- a/src/third_party/wiredtiger/test/format/format.h +++ b/src/third_party/wiredtiger/test/format/format.h @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. @@ -267,7 +267,8 @@ typedef struct { uint64_t rollback; uint64_t deadlock; - uint64_t timestamp; /* last committed timestamp */ + uint64_t commit_timestamp; /* last committed timestamp */ + uint64_t read_timestamp; /* read timestamp */ bool quit; /* thread should quit */ diff --git a/src/third_party/wiredtiger/test/format/lrt.c b/src/third_party/wiredtiger/test/format/lrt.c index fab3b54126e..fdf91508dd6 100644 --- a/src/third_party/wiredtiger/test/format/lrt.c +++ b/src/third_party/wiredtiger/test/format/lrt.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/format/ops.c b/src/third_party/wiredtiger/test/format/ops.c index 4c5576654d2..6ac2f10af95 100644 --- a/src/third_party/wiredtiger/test/format/ops.c +++ b/src/third_party/wiredtiger/test/format/ops.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. @@ -273,35 +273,6 @@ wts_ops(int lastrun) free(tinfo_list); } -/* - * isolation_config -- - * Return an isolation configuration. - */ -static inline u_int -isolation_config(WT_RAND_STATE *rnd, WT_SESSION *session) -{ - u_int v; - const char *config; - - if ((v = g.c_isolation_flag) == ISOLATION_RANDOM) - v = mmrand(rnd, 2, 4); - switch (v) { - case ISOLATION_READ_UNCOMMITTED: - config = "isolation=read-uncommitted"; - break; - case ISOLATION_READ_COMMITTED: - config = "isolation=read-committed"; - break; - case ISOLATION_SNAPSHOT: - default: - v = ISOLATION_SNAPSHOT; - config = "isolation=snapshot"; - break; - } - testutil_check(session->reconfigure(session, config)); - return (v); -} - typedef struct { uint64_t keyno; /* Row number */ @@ -482,6 +453,58 @@ snap_check(WT_CURSOR *cursor, } /* + * begin_transaction -- + * Choose an isolation configuration and begin a transaction. + */ +static void +begin_transaction(TINFO *tinfo, WT_SESSION *session, u_int *iso_configp) +{ + u_int v; + const char *config; + char config_buf[64]; + + if ((v = g.c_isolation_flag) == ISOLATION_RANDOM) + v = mmrand(&tinfo->rnd, 2, 4); + switch (v) { + case ISOLATION_READ_UNCOMMITTED: + config = "isolation=read-uncommitted"; + break; + case ISOLATION_READ_COMMITTED: + config = "isolation=read-committed"; + break; + case ISOLATION_SNAPSHOT: + default: + v = ISOLATION_SNAPSHOT; + config = "isolation=snapshot"; + if (g.c_txn_timestamps) { + /* + * Set the thread's read timestamp to the current value + * before allocating a new read timestamp. This + * guarantees the oldest timestamp won't move past the + * allocated timestamp before the transaction begins. + */ + tinfo->read_timestamp = g.timestamp; + tinfo->read_timestamp = + __wt_atomic_addv64(&g.timestamp, 1); + testutil_check(__wt_snprintf( + config_buf, sizeof(config_buf), + "read_timestamp=%" PRIx64, tinfo->read_timestamp)); + config = config_buf; + } + break; + } + *iso_configp = v; + + testutil_check(session->begin_transaction(session, config)); + + /* + * It's OK for the oldest timestamp to move past a running query, clear + * the thread's read timestamp, it no longer needs to be pinned. + */ + tinfo->read_timestamp = 0; +} + +/* * commit_transaction -- * Commit a transaction */ @@ -492,6 +515,12 @@ commit_transaction(TINFO *tinfo, WT_SESSION *session) char config_buf[64]; if (g.c_txn_timestamps) { + /* + * Update the thread's update timestamp with the current value + * to prevent the oldest timestamp moving past our allocated + * timestamp before the commit completes. + */ + tinfo->commit_timestamp = g.timestamp; ts = __wt_atomic_addv64(&g.timestamp, 1); testutil_check(__wt_snprintf( config_buf, sizeof(config_buf), @@ -500,12 +529,12 @@ commit_transaction(TINFO *tinfo, WT_SESSION *session) session->commit_transaction(session, config_buf)); /* - * Update the thread's last-committed timestamp. Don't let the - * compiler re-order this statement, if we were to race with - * the timestamp thread, it might see our thread update before - * the transaction commit. + * Clear the thread's active timestamp: it no longer needs to + * be pinned. Don't let the compiler re-order this statement, + * if we were to race with the timestamp thread, it might see + * our thread update before the transaction commit. */ - WT_PUBLISH(tinfo->timestamp, ts); + WT_PUBLISH(tinfo->commit_timestamp, 0); } else testutil_check(session->commit_transaction(session, NULL)); ++tinfo->commit; @@ -646,10 +675,7 @@ ops(void *arg) */ if (!SINGLETHREADED && !intxn && mmrand(&tinfo->rnd, 1, 100) >= g.c_txn_freq) { - iso_config = isolation_config(&tinfo->rnd, session); - testutil_check( - session->begin_transaction(session, NULL)); - + begin_transaction(tinfo, session, &iso_config); snap = iso_config == ISOLATION_SNAPSHOT ? snap_list : NULL; intxn = true; diff --git a/src/third_party/wiredtiger/test/format/rebalance.c b/src/third_party/wiredtiger/test/format/rebalance.c index 195130cfa68..c1b20b92857 100644 --- a/src/third_party/wiredtiger/test/format/rebalance.c +++ b/src/third_party/wiredtiger/test/format/rebalance.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/format/salvage.c b/src/third_party/wiredtiger/test/format/salvage.c index e18137b4f01..c631ea599e3 100644 --- a/src/third_party/wiredtiger/test/format/salvage.c +++ b/src/third_party/wiredtiger/test/format/salvage.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/format/t.c b/src/third_party/wiredtiger/test/format/t.c index 02ed0a2da60..6077a67a541 100644 --- a/src/third_party/wiredtiger/test/format/t.c +++ b/src/third_party/wiredtiger/test/format/t.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/format/util.c b/src/third_party/wiredtiger/test/format/util.c index 3c61ab5a66b..cf75c98129a 100644 --- a/src/third_party/wiredtiger/test/format/util.c +++ b/src/third_party/wiredtiger/test/format/util.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. @@ -608,25 +608,32 @@ timestamp(void *arg) */ while (!g.workers_finished) { /* - * Find the lowest committed timestamp. The timestamp thread - * starts before the operational threads, wait for them. + * Find the lowest in-use timestamp. The timestamp thread starts + * before the operational threads, wait for them. */ - oldest_timestamp = UINT64_MAX; + oldest_timestamp = g.timestamp; for (i = 0; i < g.c_threads; ++i) { tinfo = tinfo_list[i]; - this_ts = tinfo->timestamp; - if (this_ts != 0 && - this_ts < oldest_timestamp) + this_ts = tinfo->commit_timestamp; + if (this_ts != 0 && this_ts < oldest_timestamp) + oldest_timestamp = this_ts; + this_ts = tinfo->read_timestamp; + if (this_ts != 0 && this_ts < oldest_timestamp) oldest_timestamp = this_ts; } - if (oldest_timestamp == UINT64_MAX) { + + /* + * Don't try to update until we've committed some transactions + * with timestamps. + */ + if (oldest_timestamp == 0) { __wt_sleep(1, 0); continue; } /* - * Don't get more than 100 transactions or more than 15 seconds - * out of date. + * If less than 100 transactions out of date, wait up to 15 + * seconds before updating. */ WT_READ_BARRIER(); testutil_assert(oldest_timestamp <= g.timestamp); @@ -642,6 +649,7 @@ timestamp(void *arg) config_buf, sizeof(config_buf), "oldest_timestamp=%" PRIx64, oldest_timestamp)); testutil_check(conn->set_timestamp(conn, config_buf)); + __wt_seconds((WT_SESSION_IMPL *)session, &last); usecs = mmrand(NULL, 5, 40); __wt_sleep(0, usecs); diff --git a/src/third_party/wiredtiger/test/format/wts.c b/src/third_party/wiredtiger/test/format/wts.c index ddcd14cfd55..0b234cc772b 100644 --- a/src/third_party/wiredtiger/test/format/wts.c +++ b/src/third_party/wiredtiger/test/format/wts.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/huge/huge.c b/src/third_party/wiredtiger/test/huge/huge.c index 145f5720cb6..11d6bbdc312 100644 --- a/src/third_party/wiredtiger/test/huge/huge.c +++ b/src/third_party/wiredtiger/test/huge/huge.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/java/com/wiredtiger/test/AsyncTest.java b/src/third_party/wiredtiger/test/java/com/wiredtiger/test/AsyncTest.java index 11d98fb3b4e..e2db98e3d70 100644 --- a/src/third_party/wiredtiger/test/java/com/wiredtiger/test/AsyncTest.java +++ b/src/third_party/wiredtiger/test/java/com/wiredtiger/test/AsyncTest.java @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/java/com/wiredtiger/test/AutoCloseTest.java b/src/third_party/wiredtiger/test/java/com/wiredtiger/test/AutoCloseTest.java index e4f720ede8b..a4a5416b398 100644 --- a/src/third_party/wiredtiger/test/java/com/wiredtiger/test/AutoCloseTest.java +++ b/src/third_party/wiredtiger/test/java/com/wiredtiger/test/AutoCloseTest.java @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/java/com/wiredtiger/test/BackupCursorTest.java b/src/third_party/wiredtiger/test/java/com/wiredtiger/test/BackupCursorTest.java index af0a2784589..69572ef9a19 100644 --- a/src/third_party/wiredtiger/test/java/com/wiredtiger/test/BackupCursorTest.java +++ b/src/third_party/wiredtiger/test/java/com/wiredtiger/test/BackupCursorTest.java @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/java/com/wiredtiger/test/ConcurrentCloseTest.java b/src/third_party/wiredtiger/test/java/com/wiredtiger/test/ConcurrentCloseTest.java index 3759057ef1f..8a939729c73 100644 --- a/src/third_party/wiredtiger/test/java/com/wiredtiger/test/ConcurrentCloseTest.java +++ b/src/third_party/wiredtiger/test/java/com/wiredtiger/test/ConcurrentCloseTest.java @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/java/com/wiredtiger/test/ConfigTest.java b/src/third_party/wiredtiger/test/java/com/wiredtiger/test/ConfigTest.java index 432aa245afa..2e39263ceb6 100644 --- a/src/third_party/wiredtiger/test/java/com/wiredtiger/test/ConfigTest.java +++ b/src/third_party/wiredtiger/test/java/com/wiredtiger/test/ConfigTest.java @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/java/com/wiredtiger/test/CursorTest.java b/src/third_party/wiredtiger/test/java/com/wiredtiger/test/CursorTest.java index 28c92dd8a8d..02896f834cf 100644 --- a/src/third_party/wiredtiger/test/java/com/wiredtiger/test/CursorTest.java +++ b/src/third_party/wiredtiger/test/java/com/wiredtiger/test/CursorTest.java @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/java/com/wiredtiger/test/CursorTest02.java b/src/third_party/wiredtiger/test/java/com/wiredtiger/test/CursorTest02.java index 10705997352..09229fb5c53 100644 --- a/src/third_party/wiredtiger/test/java/com/wiredtiger/test/CursorTest02.java +++ b/src/third_party/wiredtiger/test/java/com/wiredtiger/test/CursorTest02.java @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/java/com/wiredtiger/test/CursorTest03.java b/src/third_party/wiredtiger/test/java/com/wiredtiger/test/CursorTest03.java index 73c7a22f69d..d2df05dbbb0 100644 --- a/src/third_party/wiredtiger/test/java/com/wiredtiger/test/CursorTest03.java +++ b/src/third_party/wiredtiger/test/java/com/wiredtiger/test/CursorTest03.java @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/java/com/wiredtiger/test/ExceptionTest.java b/src/third_party/wiredtiger/test/java/com/wiredtiger/test/ExceptionTest.java index 99719225b05..d165270b3f7 100644 --- a/src/third_party/wiredtiger/test/java/com/wiredtiger/test/ExceptionTest.java +++ b/src/third_party/wiredtiger/test/java/com/wiredtiger/test/ExceptionTest.java @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/java/com/wiredtiger/test/PackTest.java b/src/third_party/wiredtiger/test/java/com/wiredtiger/test/PackTest.java index 302313169cd..6623a6face3 100644 --- a/src/third_party/wiredtiger/test/java/com/wiredtiger/test/PackTest.java +++ b/src/third_party/wiredtiger/test/java/com/wiredtiger/test/PackTest.java @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/java/com/wiredtiger/test/PackTest02.java b/src/third_party/wiredtiger/test/java/com/wiredtiger/test/PackTest02.java index 517afd1ec03..edef545619b 100644 --- a/src/third_party/wiredtiger/test/java/com/wiredtiger/test/PackTest02.java +++ b/src/third_party/wiredtiger/test/java/com/wiredtiger/test/PackTest02.java @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/java/com/wiredtiger/test/PackTest03.java b/src/third_party/wiredtiger/test/java/com/wiredtiger/test/PackTest03.java index 81e7987f987..f24648ff0aa 100644 --- a/src/third_party/wiredtiger/test/java/com/wiredtiger/test/PackTest03.java +++ b/src/third_party/wiredtiger/test/java/com/wiredtiger/test/PackTest03.java @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/java/com/wiredtiger/test/WiredTigerSuite.java b/src/third_party/wiredtiger/test/java/com/wiredtiger/test/WiredTigerSuite.java index c77ff5b3507..78687c71041 100644 --- a/src/third_party/wiredtiger/test/java/com/wiredtiger/test/WiredTigerSuite.java +++ b/src/third_party/wiredtiger/test/java/com/wiredtiger/test/WiredTigerSuite.java @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/manydbs/manydbs.c b/src/third_party/wiredtiger/test/manydbs/manydbs.c index 61fd3d01776..a6574d21d72 100644 --- a/src/third_party/wiredtiger/test/manydbs/manydbs.c +++ b/src/third_party/wiredtiger/test/manydbs/manydbs.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/mciproject.yml b/src/third_party/wiredtiger/test/mciproject.yml index 16e103e5366..271f392ae7c 100644 --- a/src/third_party/wiredtiger/test/mciproject.yml +++ b/src/third_party/wiredtiger/test/mciproject.yml @@ -60,10 +60,10 @@ tasks: set -o errexit set -o verbose if [ "Windows_NT" = "$OS" ]; then - scons.bat --enable-python=c:\\swigwin-3.0.2\\swig.exe --enable-diagnostic --enable-verbose ${smp_command|} + scons.bat --enable-python=c:\\swigwin-3.0.2\\swig.exe --enable-diagnostic ${smp_command|} else ./build_posix/reconf - ${configure_env_vars|} ./configure --enable-diagnostic --enable-python --enable-zlib --enable-strict --enable-verbose + ${configure_env_vars|} ./configure --enable-diagnostic --enable-python --enable-zlib --enable-strict ${make_command|make} ${smp_command|} 2>&1 ${test_env_vars|} TESTUTIL_ENABLE_LONG_TESTS=1 ${make_command|make} VERBOSE=1 check 2>&1 fi @@ -113,7 +113,7 @@ tasks: set -o errexit set -o verbose - scons.bat ${smp_command|} "CFLAGS=/Gv /wd4090 /wd4996 /we4047 /we4024 /TC /we4100 /w4133" wiredtiger.dll libwiredtiger.lib + scons.bat ${smp_command|} "CFLAGS=/Gv /wd4090 /wd4996 /we4047 /we4024 /TC /we4100 /we4133" wiredtiger.dll libwiredtiger.lib - name: fops depends_on: diff --git a/src/third_party/wiredtiger/test/packing/intpack-test.c b/src/third_party/wiredtiger/test/packing/intpack-test.c index 3d2797a3553..7bc3f1f519b 100644 --- a/src/third_party/wiredtiger/test/packing/intpack-test.c +++ b/src/third_party/wiredtiger/test/packing/intpack-test.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. @@ -40,6 +40,12 @@ main(void) memset(buf, 0xff, sizeof(buf)); /* -Werror=maybe-uninitialized */ + /* + * Required on some systems to pull in parts of the library + * for which we have data references. + */ + testutil_check(__wt_library_init()); + for (ncalls = 0, i = 0; i < 10000000; i++) { for (s = 0; s < 50; s += 5) { ++ncalls; diff --git a/src/third_party/wiredtiger/test/packing/intpack-test2.c b/src/third_party/wiredtiger/test/packing/intpack-test2.c index 7fa455842b5..b1f4b8756e7 100644 --- a/src/third_party/wiredtiger/test/packing/intpack-test2.c +++ b/src/third_party/wiredtiger/test/packing/intpack-test2.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. @@ -38,6 +38,12 @@ main(void) memset(buf, 0xff, sizeof(buf)); /* -Werror=maybe-uninitialized */ + /* + * Required on some systems to pull in parts of the library + * for which we have data references. + */ + testutil_check(__wt_library_init()); + for (i = 1; i < 1LL << 60; i <<= 1) { end = buf; testutil_check( diff --git a/src/third_party/wiredtiger/test/packing/intpack-test3.c b/src/third_party/wiredtiger/test/packing/intpack-test3.c index 2f74c84c6d8..8076ca5cd52 100644 --- a/src/third_party/wiredtiger/test/packing/intpack-test3.c +++ b/src/third_party/wiredtiger/test/packing/intpack-test3.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. @@ -45,6 +45,12 @@ test_value(int64_t val) sinput = val; soutput = 0; /* -Werror=maybe-uninitialized */ + /* + * Required on some systems to pull in parts of the library + * for which we have data references. + */ + testutil_check(__wt_library_init()); + p = buf; testutil_check(__wt_vpack_int(&p, sizeof(buf), sinput)); used_len = (size_t)(p - buf); diff --git a/src/third_party/wiredtiger/test/packing/packing-test.c b/src/third_party/wiredtiger/test/packing/packing-test.c index e53664ee9c5..89946c4a64d 100644 --- a/src/third_party/wiredtiger/test/packing/packing-test.c +++ b/src/third_party/wiredtiger/test/packing/packing-test.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. @@ -58,6 +58,12 @@ check(const char *fmt, ...) int main(void) { + /* + * Required on some systems to pull in parts of the library + * for which we have data references. + */ + testutil_check(__wt_library_init()); + check("iii", 0, 101, -99); check("3i", 0, 101, -99); check("iS", 42, "forty two"); diff --git a/src/third_party/wiredtiger/test/readonly/readonly.c b/src/third_party/wiredtiger/test/readonly/readonly.c index e01b5b85fbd..b5342831320 100644 --- a/src/third_party/wiredtiger/test/readonly/readonly.c +++ b/src/third_party/wiredtiger/test/readonly/readonly.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. @@ -44,9 +44,9 @@ static const char * const uri = "table:main"; #define ENV_CONFIG \ "create,log=(file_max=10M,archive=false,enabled)," \ - "transaction_sync=(enabled,method=none)" -#define ENV_CONFIG_RD "readonly=true" -#define ENV_CONFIG_WR "readonly=false" + "operation_tracking=(enabled=false),transaction_sync=(enabled,method=none)" +#define ENV_CONFIG_RD "operation_tracking=(enabled=false),readonly=true" +#define ENV_CONFIG_WR "operation_tracking=(enabled=false),readonly=false" #define MAX_VAL 4096 #define MAX_KV 10000 diff --git a/src/third_party/wiredtiger/test/salvage/salvage.c b/src/third_party/wiredtiger/test/salvage/salvage.c index 9eb2a9de383..9c8a90d37b9 100644 --- a/src/third_party/wiredtiger/test/salvage/salvage.c +++ b/src/third_party/wiredtiger/test/salvage/salvage.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/helper.py b/src/third_party/wiredtiger/test/suite/helper.py index 2f9bbf8aa68..7aec06f5260 100644 --- a/src/third_party/wiredtiger/test/suite/helper.py +++ b/src/third_party/wiredtiger/test/suite/helper.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/run.py b/src/third_party/wiredtiger/test/suite/run.py index 8a936de584b..6b668ad3e07 100644 --- a/src/third_party/wiredtiger/test/suite/run.py +++ b/src/third_party/wiredtiger/test/suite/run.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/suite_random.py b/src/third_party/wiredtiger/test/suite/suite_random.py index 16a8b89113c..f31f92410aa 100644 --- a/src/third_party/wiredtiger/test/suite/suite_random.py +++ b/src/third_party/wiredtiger/test/suite/suite_random.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/suite_subprocess.py b/src/third_party/wiredtiger/test/suite/suite_subprocess.py index 71aab9c5422..c2e9d99f691 100644 --- a/src/third_party/wiredtiger/test/suite/suite_subprocess.py +++ b/src/third_party/wiredtiger/test/suite/suite_subprocess.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_alter01.py b/src/third_party/wiredtiger/test/suite/test_alter01.py index 7a143afb32c..aabe071ef03 100644 --- a/src/third_party/wiredtiger/test/suite/test_alter01.py +++ b/src/third_party/wiredtiger/test/suite/test_alter01.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_alter02.py b/src/third_party/wiredtiger/test/suite/test_alter02.py index 3a288d7edf1..d32538c80f1 100644 --- a/src/third_party/wiredtiger/test/suite/test_alter02.py +++ b/src/third_party/wiredtiger/test/suite/test_alter02.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_assert01.py b/src/third_party/wiredtiger/test/suite/test_assert01.py index 3a4f8e4127a..acdb3d9ee69 100644 --- a/src/third_party/wiredtiger/test/suite/test_assert01.py +++ b/src/third_party/wiredtiger/test/suite/test_assert01.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_assert02.py b/src/third_party/wiredtiger/test/suite/test_assert02.py index d264273c3a0..ff1a4bd05e5 100644 --- a/src/third_party/wiredtiger/test/suite/test_assert02.py +++ b/src/third_party/wiredtiger/test/suite/test_assert02.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_assert03.py b/src/third_party/wiredtiger/test/suite/test_assert03.py index 36d4936a82e..c78991aba33 100644 --- a/src/third_party/wiredtiger/test/suite/test_assert03.py +++ b/src/third_party/wiredtiger/test/suite/test_assert03.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_assert04.py b/src/third_party/wiredtiger/test/suite/test_assert04.py new file mode 100644 index 00000000000..d582535fd0f --- /dev/null +++ b/src/third_party/wiredtiger/test/suite/test_assert04.py @@ -0,0 +1,349 @@ +#!/usr/bin/env python +# +# Public Domain 2014-2018 MongoDB, Inc. +# Public Domain 2008-2014 WiredTiger, Inc. +# +# This is free and unencumbered software released into the public domain. +# +# Anyone is free to copy, modify, publish, use, compile, sell, or +# distribute this software, either in source code form or as a compiled +# binary, for any purpose, commercial or non-commercial, and by any +# means. +# +# In jurisdictions that recognize copyright laws, the author or authors +# of this software dedicate any and all copyright interest in the +# software to the public domain. We make this dedication for the benefit +# of the public at large and to the detriment of our heirs and +# successors. We intend this dedication to be an overt act of +# relinquishment in perpetuity of all present and future rights to this +# software under copyright law. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +# IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. +# +# test_assert04.py +# Timestamps: verify consistency usage on keys +# + +from suite_subprocess import suite_subprocess +import wiredtiger, wttest + +def timestamp_str(t): + return '%x' % t + +class test_assert04(wttest.WiredTigerTestCase, suite_subprocess): + def test_timestamp_alter(self): + if not wiredtiger.timestamp_build(): + self.skipTest('requires a timestamp build') + + base = 'assert04' + uri = 'file:' + base + cfg_on = 'assert=(commit_timestamp=key_consistent)' + cfg_off = 'assert=(commit_timestamp=none)' + msg_ooo='/out of order/' + msg_usage='/used inconsistently/' + + # Create the table without the key consistency checking turned on. + # Create a few items breaking the rules. Then alter the setting and + # verify the inconsistent usage is detected. + self.session.create(uri, 'key_format=S,value_format=S') + + # Insert a data item at timestamp 2. + c = self.session.open_cursor(uri) + self.session.begin_transaction() + self.session.timestamp_transaction( + 'commit_timestamp=' + timestamp_str(2)) + c['key_ts1'] = 'value2' + self.session.commit_transaction() + c.close() + + # Modify the data item at timestamp 1. + c = self.session.open_cursor(uri) + self.session.begin_transaction() + self.session.timestamp_transaction( + 'commit_timestamp=' + timestamp_str(1)) + c['key_ts1'] = 'value1' + self.session.commit_transaction() + c.close() + + # Insert a non-timestamped item. Then modify with a timestamp. And + # again modify without a timestamp. + c = self.session.open_cursor(uri) + self.session.begin_transaction() + c['key_nots'] = 'value_nots' + self.session.commit_transaction() + c.close() + + c = self.session.open_cursor(uri) + self.session.begin_transaction() + self.session.timestamp_transaction( + 'commit_timestamp=' + timestamp_str(2)) + c['key_nots'] = 'value2' + self.session.commit_transaction() + c.close() + + c = self.session.open_cursor(uri) + self.session.begin_transaction() + c['key_nots'] = 'value_nots2' + self.session.commit_transaction() + c.close() + + # We must move the oldest timestamp forward in order to alter. + # Otherwise alter's closing of the file will fail with EBUSY. + self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(2)) + + # Now alter the setting and make sure we detect incorrect usage. + self.session.alter(uri, cfg_on) + + # Detect decreasing timestamp. + c = self.session.open_cursor(uri) + self.session.begin_transaction() + self.session.timestamp_transaction( + 'commit_timestamp=' + timestamp_str(5)) + c['key_ts1'] = 'value5' + self.session.commit_transaction() + c.close() + + c = self.session.open_cursor(uri) + self.session.begin_transaction() + self.session.timestamp_transaction( + 'commit_timestamp=' + timestamp_str(4)) + c['key_ts1'] = 'value4' + self.assertRaisesWithMessage(wiredtiger.WiredTigerError, + lambda: self.session.commit_transaction(), msg_ooo) + c.close() + + # Detect not using a timestamp. + c = self.session.open_cursor(uri) + self.session.begin_transaction() + c['key_ts1'] = 'value_nots3' + self.assertRaisesWithMessage(wiredtiger.WiredTigerError, + lambda: self.session.commit_transaction(), msg_usage) + c.close() + + # Detect using a timestamp on the non-timestamp key. + c = self.session.open_cursor(uri) + self.session.begin_transaction() + c['key_nots'] = 'value_nots3' + self.session.commit_transaction() + c.close() + + c = self.session.open_cursor(uri) + self.session.begin_transaction() + self.session.timestamp_transaction( + 'commit_timestamp=' + timestamp_str(3)) + c['key_nots'] = 'value3' + self.assertRaisesWithMessage(wiredtiger.WiredTigerError, + lambda: self.session.commit_transaction(), msg_usage) + c.close() + + c = self.session.open_cursor(uri) + self.assertEquals(c['key_ts1'], 'value5') + self.assertEquals(c['key_nots'], 'value_nots3') + c.close() + + # Now alter the setting again and detection is off. + self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(5)) + self.session.alter(uri, cfg_off) + + # Detection is off we can successfully change the same key with and + # without a timestamp. + c = self.session.open_cursor(uri) + self.session.begin_transaction() + c['key_nots'] = 'value_nots4' + self.session.commit_transaction() + c.close() + + c = self.session.open_cursor(uri) + self.session.begin_transaction() + self.session.timestamp_transaction( + 'commit_timestamp=' + timestamp_str(6)) + c['key_nots'] = 'value6' + self.session.commit_transaction() + c.close() + + def test_timestamp_usage(self): + if not wiredtiger.timestamp_build(): + self.skipTest('requires a timestamp build') + + base = 'assert04' + uri = 'file:' + base + msg_ooo='/out of order/' + msg_usage='/used inconsistently/' + + # Create the table with the key consistency checking turned on. + # That checking will verify any individual key is always or never + # used with a timestamp. And if it is used with a timestamp that + # the timestamps are in increasing order for that key. + self.session.create(uri, 'key_format=S,value_format=S,assert=(commit_timestamp=key_consistent)') + + # Insert a data item at timestamp 2. + c = self.session.open_cursor(uri) + self.session.begin_transaction() + self.session.timestamp_transaction( + 'commit_timestamp=' + timestamp_str(2)) + c['key_ts1'] = 'value2' + self.session.commit_transaction() + c.close() + + # Modify the data item at timestamp 1. We should detect it is wrong. + c = self.session.open_cursor(uri) + self.session.begin_transaction() + self.session.timestamp_transaction( + 'commit_timestamp=' + timestamp_str(1)) + c['key_ts1'] = 'value1' + self.assertRaisesWithMessage(wiredtiger.WiredTigerError, + lambda: self.session.commit_transaction(), msg_ooo) + c.close() + + # Make sure we can successfully add a different key at timestamp 1. + c = self.session.open_cursor(uri) + self.session.begin_transaction() + self.session.timestamp_transaction( + 'commit_timestamp=' + timestamp_str(1)) + c['key_ts2'] = 'value1' + self.session.commit_transaction() + c.close() + + # + # Insert key_ts3 at timestamp 10 and key_ts4 at 15. + # Then modify both keys in one transaction at timestamp 13. + # We should not be allowed to modify the one from 15. + # So the whole transaction should fail. + # + c = self.session.open_cursor(uri) + self.session.begin_transaction() + self.session.timestamp_transaction( + 'commit_timestamp=' + timestamp_str(10)) + c['key_ts3'] = 'value10' + self.session.commit_transaction() + self.session.begin_transaction() + self.session.timestamp_transaction( + 'commit_timestamp=' + timestamp_str(15)) + c['key_ts4'] = 'value15' + self.session.commit_transaction() + + c = self.session.open_cursor(uri) + self.session.begin_transaction() + self.session.timestamp_transaction( + 'commit_timestamp=' + timestamp_str(13)) + c['key_ts3'] = 'value13' + c['key_ts4'] = 'value13' + self.assertRaisesWithMessage(wiredtiger.WiredTigerError, + lambda: self.session.commit_transaction(), msg_ooo) + c.close() + + c = self.session.open_cursor(uri) + self.assertEquals(c['key_ts3'], 'value10') + self.assertEquals(c['key_ts4'], 'value15') + c.close() + + # + # Separately, we should be able to update key_ts3 at timestamp 10 + # but not update key_ts4 inserted at timestamp 15. + # + c = self.session.open_cursor(uri) + self.session.begin_transaction() + self.session.timestamp_transaction( + 'commit_timestamp=' + timestamp_str(13)) + c['key_ts3'] = 'value13' + self.session.commit_transaction() + + c = self.session.open_cursor(uri) + self.session.begin_transaction() + self.session.timestamp_transaction( + 'commit_timestamp=' + timestamp_str(13)) + c['key_ts4'] = 'value13' + self.assertRaisesWithMessage(wiredtiger.WiredTigerError, + lambda: self.session.commit_transaction(), msg_ooo) + c.close() + + # Make sure multiple update attempts still fail and eventually + # succeed with a later timestamp. This tests that aborted entries + # in the update chain are not considered for the timestamp check. + c = self.session.open_cursor(uri) + self.session.begin_transaction() + self.session.timestamp_transaction( + 'commit_timestamp=' + timestamp_str(14)) + c['key_ts4'] = 'value14' + self.assertRaisesWithMessage(wiredtiger.WiredTigerError, + lambda: self.session.commit_transaction(), msg_ooo) + c.close() + c = self.session.open_cursor(uri) + self.assertEquals(c['key_ts4'], 'value15') + c.close() + + c = self.session.open_cursor(uri) + self.session.begin_transaction() + self.session.timestamp_transaction( + 'commit_timestamp=' + timestamp_str(16)) + c['key_ts4'] = 'value16' + self.session.commit_transaction() + c.close() + c = self.session.open_cursor(uri) + self.assertEquals(c['key_ts4'], 'value16') + c.close() + + # Now try to modify a key previously used with timestamps without + # one. We should get the inconsistent usage message. + c = self.session.open_cursor(uri) + self.session.begin_transaction() + c['key_ts4'] = 'value_nots' + self.assertRaisesWithMessage(wiredtiger.WiredTigerError, + lambda: self.session.commit_transaction(), msg_usage) + c.close() + + c = self.session.open_cursor(uri) + self.session.begin_transaction() + c['key_ts4'] = 'value_nots' + self.assertRaisesWithMessage(wiredtiger.WiredTigerError, + lambda: self.session.commit_transaction(), msg_usage) + c.close() + c = self.session.open_cursor(uri) + self.assertEquals(c['key_ts4'], 'value16') + c.close() + + # Now confirm the other way. Create a key without a timestamp and then + # attempt to modify it with a timestamp. The only error checking that + # makes sense here is the inconsistent usage. + c = self.session.open_cursor(uri) + self.session.begin_transaction() + c['key_nots'] = 'value_nots' + self.session.commit_transaction() + c.close() + + c = self.session.open_cursor(uri) + self.session.begin_transaction() + self.session.timestamp_transaction( + 'commit_timestamp=' + timestamp_str(16)) + c['key_nots'] = 'value16' + self.assertRaisesWithMessage(wiredtiger.WiredTigerError, + lambda: self.session.commit_transaction(), msg_usage) + c.close() + + c = self.session.open_cursor(uri) + self.session.begin_transaction() + c['key_nots'] = 'value_nots1' + self.session.commit_transaction() + c.close() + + c = self.session.open_cursor(uri) + self.session.begin_transaction() + self.session.timestamp_transaction( + 'commit_timestamp=' + timestamp_str(17)) + c['key_nots'] = 'value17' + self.assertRaisesWithMessage(wiredtiger.WiredTigerError, + lambda: self.session.commit_transaction(), msg_usage) + c.close() + + c = self.session.open_cursor(uri) + self.assertEquals(c['key_nots'], 'value_nots1') + c.close() + +if __name__ == '__main__': + wttest.run() diff --git a/src/third_party/wiredtiger/test/suite/test_async01.py b/src/third_party/wiredtiger/test/suite/test_async01.py index 4faaad6b8f4..432325db9be 100644 --- a/src/third_party/wiredtiger/test/suite/test_async01.py +++ b/src/third_party/wiredtiger/test/suite/test_async01.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_async02.py b/src/third_party/wiredtiger/test/suite/test_async02.py index fbd743fec29..1d3095551ab 100644 --- a/src/third_party/wiredtiger/test/suite/test_async02.py +++ b/src/third_party/wiredtiger/test/suite/test_async02.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_async03.py b/src/third_party/wiredtiger/test/suite/test_async03.py index cf993071d73..4af5a6b01bd 100644 --- a/src/third_party/wiredtiger/test/suite/test_async03.py +++ b/src/third_party/wiredtiger/test/suite/test_async03.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_autoclose.py b/src/third_party/wiredtiger/test/suite/test_autoclose.py index ce152b24fe3..cfe016d66fe 100644 --- a/src/third_party/wiredtiger/test/suite/test_autoclose.py +++ b/src/third_party/wiredtiger/test/suite/test_autoclose.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_backup01.py b/src/third_party/wiredtiger/test/suite/test_backup01.py index 52d71ab53bb..2eb6efc4eed 100644 --- a/src/third_party/wiredtiger/test/suite/test_backup01.py +++ b/src/third_party/wiredtiger/test/suite/test_backup01.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_backup02.py b/src/third_party/wiredtiger/test/suite/test_backup02.py index 7d8f653feae..f4f5219440b 100644 --- a/src/third_party/wiredtiger/test/suite/test_backup02.py +++ b/src/third_party/wiredtiger/test/suite/test_backup02.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. @@ -32,7 +32,7 @@ from wtthread import backup_thread, checkpoint_thread, op_thread from wtscenario import make_scenarios # test_backup02.py -# Run background checkpoints and backsups repeatedly while doing inserts +# Run background checkpoints and backups repeatedly while doing inserts # in another thread class test_backup02(wttest.WiredTigerTestCase): scenarios = make_scenarios([ diff --git a/src/third_party/wiredtiger/test/suite/test_backup03.py b/src/third_party/wiredtiger/test/suite/test_backup03.py index 7d0bfd5eaaf..da94dfe5f17 100644 --- a/src/third_party/wiredtiger/test/suite/test_backup03.py +++ b/src/third_party/wiredtiger/test/suite/test_backup03.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_backup04.py b/src/third_party/wiredtiger/test/suite/test_backup04.py index 9f40ae2427b..13b2436d7ad 100644 --- a/src/third_party/wiredtiger/test/suite/test_backup04.py +++ b/src/third_party/wiredtiger/test/suite/test_backup04.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_backup05.py b/src/third_party/wiredtiger/test/suite/test_backup05.py index fb44de04694..96b7cbea58e 100644 --- a/src/third_party/wiredtiger/test/suite/test_backup05.py +++ b/src/third_party/wiredtiger/test/suite/test_backup05.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_backup06.py b/src/third_party/wiredtiger/test/suite/test_backup06.py index d416ba035b5..1540e114bb8 100644 --- a/src/third_party/wiredtiger/test/suite/test_backup06.py +++ b/src/third_party/wiredtiger/test/suite/test_backup06.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_backup07.py b/src/third_party/wiredtiger/test/suite/test_backup07.py index 8332815b0ca..b15ab274a8b 100644 --- a/src/third_party/wiredtiger/test/suite/test_backup07.py +++ b/src/third_party/wiredtiger/test/suite/test_backup07.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_base01.py b/src/third_party/wiredtiger/test/suite/test_base01.py index f39ec3eb739..1a2133b17bb 100644 --- a/src/third_party/wiredtiger/test/suite/test_base01.py +++ b/src/third_party/wiredtiger/test/suite/test_base01.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_base02.py b/src/third_party/wiredtiger/test/suite/test_base02.py index 5e1140a5700..6af674c6402 100644 --- a/src/third_party/wiredtiger/test/suite/test_base02.py +++ b/src/third_party/wiredtiger/test/suite/test_base02.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_base03.py b/src/third_party/wiredtiger/test/suite/test_base03.py index ad1629db77e..8e7b44ed092 100644 --- a/src/third_party/wiredtiger/test/suite/test_base03.py +++ b/src/third_party/wiredtiger/test/suite/test_base03.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_base04.py b/src/third_party/wiredtiger/test/suite/test_base04.py index f9fdddce157..e78af565679 100644 --- a/src/third_party/wiredtiger/test/suite/test_base04.py +++ b/src/third_party/wiredtiger/test/suite/test_base04.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_base05.py b/src/third_party/wiredtiger/test/suite/test_base05.py index 5ba6d5eda4b..847cbd0a738 100644 --- a/src/third_party/wiredtiger/test/suite/test_base05.py +++ b/src/third_party/wiredtiger/test/suite/test_base05.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_baseconfig.py b/src/third_party/wiredtiger/test/suite/test_baseconfig.py index 3a5778b3bb5..57c2c7a203c 100644 --- a/src/third_party/wiredtiger/test/suite/test_baseconfig.py +++ b/src/third_party/wiredtiger/test/suite/test_baseconfig.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_bug001.py b/src/third_party/wiredtiger/test/suite/test_bug001.py index 4353dad5e68..9882cb2c700 100644 --- a/src/third_party/wiredtiger/test/suite/test_bug001.py +++ b/src/third_party/wiredtiger/test/suite/test_bug001.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_bug003.py b/src/third_party/wiredtiger/test/suite/test_bug003.py index 799c004e17d..b44bb33c69b 100644 --- a/src/third_party/wiredtiger/test/suite/test_bug003.py +++ b/src/third_party/wiredtiger/test/suite/test_bug003.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_bug004.py b/src/third_party/wiredtiger/test/suite/test_bug004.py index a47bdc6dd1e..c94ea70225c 100644 --- a/src/third_party/wiredtiger/test/suite/test_bug004.py +++ b/src/third_party/wiredtiger/test/suite/test_bug004.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_bug005.py b/src/third_party/wiredtiger/test/suite/test_bug005.py index 6d099bf2708..c6c4f6d18a6 100644 --- a/src/third_party/wiredtiger/test/suite/test_bug005.py +++ b/src/third_party/wiredtiger/test/suite/test_bug005.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_bug006.py b/src/third_party/wiredtiger/test/suite/test_bug006.py index 505325de200..c65780a176c 100644 --- a/src/third_party/wiredtiger/test/suite/test_bug006.py +++ b/src/third_party/wiredtiger/test/suite/test_bug006.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_bug007.py b/src/third_party/wiredtiger/test/suite/test_bug007.py index 806d75d8394..b654fbd260c 100644 --- a/src/third_party/wiredtiger/test/suite/test_bug007.py +++ b/src/third_party/wiredtiger/test/suite/test_bug007.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_bug008.py b/src/third_party/wiredtiger/test/suite/test_bug008.py index cb0bb390ad4..d0377f318ee 100644 --- a/src/third_party/wiredtiger/test/suite/test_bug008.py +++ b/src/third_party/wiredtiger/test/suite/test_bug008.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_bug009.py b/src/third_party/wiredtiger/test/suite/test_bug009.py index 7f2af55f2d0..25041921055 100644 --- a/src/third_party/wiredtiger/test/suite/test_bug009.py +++ b/src/third_party/wiredtiger/test/suite/test_bug009.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_bug010.py b/src/third_party/wiredtiger/test/suite/test_bug010.py index 1584c173053..01d604e5a07 100644 --- a/src/third_party/wiredtiger/test/suite/test_bug010.py +++ b/src/third_party/wiredtiger/test/suite/test_bug010.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_bug011.py b/src/third_party/wiredtiger/test/suite/test_bug011.py index 2c3fd831f93..02c43231499 100644 --- a/src/third_party/wiredtiger/test/suite/test_bug011.py +++ b/src/third_party/wiredtiger/test/suite/test_bug011.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_bug012.py b/src/third_party/wiredtiger/test/suite/test_bug012.py index ae80a9c7179..fd38f97bc46 100644 --- a/src/third_party/wiredtiger/test/suite/test_bug012.py +++ b/src/third_party/wiredtiger/test/suite/test_bug012.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_bug013.py b/src/third_party/wiredtiger/test/suite/test_bug013.py index a15bd42c9da..d8db304543e 100644 --- a/src/third_party/wiredtiger/test/suite/test_bug013.py +++ b/src/third_party/wiredtiger/test/suite/test_bug013.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_bug014.py b/src/third_party/wiredtiger/test/suite/test_bug014.py index 81e47bc331b..dc96e4c7f82 100644 --- a/src/third_party/wiredtiger/test/suite/test_bug014.py +++ b/src/third_party/wiredtiger/test/suite/test_bug014.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_bug015.py b/src/third_party/wiredtiger/test/suite/test_bug015.py index 5b2a64dc76e..e93b62bcd50 100644 --- a/src/third_party/wiredtiger/test/suite/test_bug015.py +++ b/src/third_party/wiredtiger/test/suite/test_bug015.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_bug016.py b/src/third_party/wiredtiger/test/suite/test_bug016.py index a2a40118008..5c14f59d8cd 100644 --- a/src/third_party/wiredtiger/test/suite/test_bug016.py +++ b/src/third_party/wiredtiger/test/suite/test_bug016.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_bug017.py b/src/third_party/wiredtiger/test/suite/test_bug017.py index 43aeee07bb6..e85be77fffc 100644 --- a/src/third_party/wiredtiger/test/suite/test_bug017.py +++ b/src/third_party/wiredtiger/test/suite/test_bug017.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_bug018.py b/src/third_party/wiredtiger/test/suite/test_bug018.py index 7d20ebcaacb..f32140d1987 100644 --- a/src/third_party/wiredtiger/test/suite/test_bug018.py +++ b/src/third_party/wiredtiger/test/suite/test_bug018.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. @@ -65,7 +65,7 @@ class test_bug018(wttest.WiredTigerTestCase): # table out from underneath WiredTiger. We do this right before # closing the connection so that the write error happens during close # when writing out the final data. Allow table 1 to succeed and force - # an erorr writing out table 2. + # an error writing out table 2. # # This is Linux-specific code to figure out the file descriptor. for f in os.listdir('/proc/self/fd'): diff --git a/src/third_party/wiredtiger/test/suite/test_bulk01.py b/src/third_party/wiredtiger/test/suite/test_bulk01.py index 5cbb722d7a5..f718e377e05 100644 --- a/src/third_party/wiredtiger/test/suite/test_bulk01.py +++ b/src/third_party/wiredtiger/test/suite/test_bulk01.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_bulk02.py b/src/third_party/wiredtiger/test/suite/test_bulk02.py index de9ebec5204..ec20c3242c3 100644 --- a/src/third_party/wiredtiger/test/suite/test_bulk02.py +++ b/src/third_party/wiredtiger/test/suite/test_bulk02.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_checkpoint01.py b/src/third_party/wiredtiger/test/suite/test_checkpoint01.py index 1964a94b31c..38569ad090e 100644 --- a/src/third_party/wiredtiger/test/suite/test_checkpoint01.py +++ b/src/third_party/wiredtiger/test/suite/test_checkpoint01.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_checkpoint02.py b/src/third_party/wiredtiger/test/suite/test_checkpoint02.py index 3a0a47d8163..e88c3bfd146 100644 --- a/src/third_party/wiredtiger/test/suite/test_checkpoint02.py +++ b/src/third_party/wiredtiger/test/suite/test_checkpoint02.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_colgap.py b/src/third_party/wiredtiger/test/suite/test_colgap.py index 1ea55e11d78..af2022a74ee 100644 --- a/src/third_party/wiredtiger/test/suite/test_colgap.py +++ b/src/third_party/wiredtiger/test/suite/test_colgap.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_collator.py b/src/third_party/wiredtiger/test/suite/test_collator.py index 320c4e7d7b4..e1dd2bb39a2 100644 --- a/src/third_party/wiredtiger/test/suite/test_collator.py +++ b/src/third_party/wiredtiger/test/suite/test_collator.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. @@ -108,6 +108,7 @@ class test_collator(wttest.WiredTigerTestCase): key = c.get_key() self.assertEqual(value, expect) i += 1 + cursor.close() self.assertEqual(self.nentries, i) for i in range(0, self.nindices): c = icursor[i] diff --git a/src/third_party/wiredtiger/test/suite/test_compact01.py b/src/third_party/wiredtiger/test/suite/test_compact01.py index cfe5c909b1f..4a0926e8faa 100644 --- a/src/third_party/wiredtiger/test/suite/test_compact01.py +++ b/src/third_party/wiredtiger/test/suite/test_compact01.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_compact02.py b/src/third_party/wiredtiger/test/suite/test_compact02.py index eb1eb641191..ffa05fb92db 100644 --- a/src/third_party/wiredtiger/test/suite/test_compact02.py +++ b/src/third_party/wiredtiger/test/suite/test_compact02.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. @@ -30,7 +30,7 @@ # Test that compact reduces the file size. # -import wiredtiger, wttest +import time, wiredtiger, wttest from wiredtiger import stat from wtscenario import make_scenarios @@ -146,7 +146,14 @@ class test_compact02(wttest.WiredTigerTestCase): self.session.checkpoint() # 5. Call compact. - self.session.compact(self.uri, None) + # Compact can collide with eviction, if that happens we retry. Wait for + # up to a minute, the check for EBUSY should mean we're not retrying on + # real errors. + for i in range(1, 15): + if not self.raisesBusy( + lambda: self.session.compact(self.uri, None)): + break + time.sleep(4) # 6. Get stats on compacted table. sz = self.getSize() diff --git a/src/third_party/wiredtiger/test/suite/test_compat01.py b/src/third_party/wiredtiger/test/suite/test_compat01.py index ce87e072fc9..f4d37a18971 100644 --- a/src/third_party/wiredtiger/test/suite/test_compat01.py +++ b/src/third_party/wiredtiger/test/suite/test_compat01.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_compress01.py b/src/third_party/wiredtiger/test/suite/test_compress01.py index 1190a9dbe00..aa6f5104216 100644 --- a/src/third_party/wiredtiger/test/suite/test_compress01.py +++ b/src/third_party/wiredtiger/test/suite/test_compress01.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_config01.py b/src/third_party/wiredtiger/test/suite/test_config01.py index 5252d805e07..595f2ff9496 100644 --- a/src/third_party/wiredtiger/test/suite/test_config01.py +++ b/src/third_party/wiredtiger/test/suite/test_config01.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_config02.py b/src/third_party/wiredtiger/test/suite/test_config02.py index 441aa41d218..eaf0c867893 100644 --- a/src/third_party/wiredtiger/test/suite/test_config02.py +++ b/src/third_party/wiredtiger/test/suite/test_config02.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_config03.py b/src/third_party/wiredtiger/test/suite/test_config03.py index 810d399613a..49ea1d70b30 100644 --- a/src/third_party/wiredtiger/test/suite/test_config03.py +++ b/src/third_party/wiredtiger/test/suite/test_config03.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_config04.py b/src/third_party/wiredtiger/test/suite/test_config04.py index b09189be8ea..c3d7e3b8f49 100644 --- a/src/third_party/wiredtiger/test/suite/test_config04.py +++ b/src/third_party/wiredtiger/test/suite/test_config04.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. @@ -140,7 +140,6 @@ class test_config04(wttest.WiredTigerTestCase): def test_eviction(self): self.common_test('eviction_target=84,eviction_trigger=94') - # Note def test_eviction_bad(self): self.assertRaisesWithMessage(wiredtiger.WiredTigerError, lambda: @@ -154,6 +153,68 @@ class test_config04(wttest.WiredTigerTestCase): 'eviction_trigger=86'), "/eviction target must be lower than the eviction trigger/") + def test_eviction_absolute(self): + self.common_test('eviction_target=50MB,eviction_trigger=60MB,' + 'eviction_dirty_target=20MB,eviction_dirty_trigger=15MB,' + 'eviction_checkpoint_target=13MB') + + def test_eviction_abs_and_pct(self): + self.common_test('eviction_target=50,eviction_trigger=60MB,' + 'eviction_dirty_target=20,eviction_dirty_trigger=15MB') + + def test_eviction_abs_less_than_one_pct(self): + self.wiredtiger_open('.','create,cache_size=8GB,eviction_target=70MB,' + 'eviction_trigger=75MB') + + # Test that eviction_target must be lower than eviction_trigger + def test_eviction_absolute_bad(self): + self.assertRaisesWithMessage(wiredtiger.WiredTigerError, lambda: + self.wiredtiger_open('.','create,eviction_target=70MB,' + 'eviction_trigger=60MB'), + '/eviction target must be lower than the eviction trigger/') + + def test_eviction_abs_and_pct_bad(self): + self.assertRaisesWithMessage(wiredtiger.WiredTigerError, lambda: + self.wiredtiger_open('.','create,eviction_target=50,' + 'eviction_trigger=40MB'), + '/eviction target must be lower than the eviction trigger/') + + def test_eviction_abs_and_pct_bad2(self): + self.assertRaisesWithMessage(wiredtiger.WiredTigerError, lambda: + self.wiredtiger_open('.','create,eviction_target=50MB,' + 'eviction_trigger=40'), + '/eviction target must be lower than the eviction trigger/') + + def test_eviction_tgt_abs_too_large(self): + self.assertRaisesWithMessage(wiredtiger.WiredTigerError, lambda: + self.wiredtiger_open('.','create,cache_size=500MB,' + 'eviction_target=1G'), + '/eviction target should not exceed cache size/') + + def test_eviction_trigger_abs_too_large(self): + self.assertRaisesWithMessage(wiredtiger.WiredTigerError, lambda: + self.wiredtiger_open('.','create,cache_size=500MB,' + 'eviction_trigger=1G'), + '/eviction trigger should not exceed cache size/') + + def test_eviction_dirty_tgt_abs_too_large(self): + self.assertRaisesWithMessage(wiredtiger.WiredTigerError, lambda: + self.wiredtiger_open('.','create,cache_size=500MB,' + 'eviction_dirty_target=1G'), + '/eviction dirty target should not exceed cache size/') + + def test_eviction_dirty_trigggr_abs_too_large(self): + self.assertRaisesWithMessage(wiredtiger.WiredTigerError, lambda: + self.wiredtiger_open('.','create,cache_size=500MB,' + 'eviction_dirty_trigger=1G'), + '/eviction dirty trigger should not exceed cache size/') + + def test_eviction_checkpoint_tgt_abs_too_large(self): + self.assertRaisesWithMessage(wiredtiger.WiredTigerError, lambda: + self.wiredtiger_open('.','create,cache_size=500MB,' + 'eviction_checkpoint_target=1G'), + '/eviction checkpoint target should not exceed cache size/') + def test_invalid_config(self): msg = '/Unbalanced brackets/' self.assertRaisesWithMessage(wiredtiger.WiredTigerError, diff --git a/src/third_party/wiredtiger/test/suite/test_config05.py b/src/third_party/wiredtiger/test/suite/test_config05.py index 5960f01dc8e..21ddd14662d 100644 --- a/src/third_party/wiredtiger/test/suite/test_config05.py +++ b/src/third_party/wiredtiger/test/suite/test_config05.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. @@ -90,7 +90,7 @@ class test_config05(wttest.WiredTigerTestCase): self.conn = self.wiredtiger_open('.', 'create,session_max=1') self.assertRaisesWithMessage(wiredtiger.WiredTigerError, lambda: [self.conn.open_session(None) for i in range(100)], - '/configured to support/') + '/out of sessions/') def test_exclusive_create(self): self.conn = self.wiredtiger_open('.', 'create,exclusive') diff --git a/src/third_party/wiredtiger/test/suite/test_config06.py b/src/third_party/wiredtiger/test/suite/test_config06.py index f39fe2d3a4f..ef041a78449 100644 --- a/src/third_party/wiredtiger/test/suite/test_config06.py +++ b/src/third_party/wiredtiger/test/suite/test_config06.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_cursor01.py b/src/third_party/wiredtiger/test/suite/test_cursor01.py index 99bdb6182c7..ae98d1688cd 100644 --- a/src/third_party/wiredtiger/test/suite/test_cursor01.py +++ b/src/third_party/wiredtiger/test/suite/test_cursor01.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_cursor02.py b/src/third_party/wiredtiger/test/suite/test_cursor02.py index 35dc2587b1f..ae146ae6e36 100644 --- a/src/third_party/wiredtiger/test/suite/test_cursor02.py +++ b/src/third_party/wiredtiger/test/suite/test_cursor02.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_cursor03.py b/src/third_party/wiredtiger/test/suite/test_cursor03.py index 8910dc741a4..f44c4ed0b36 100644 --- a/src/third_party/wiredtiger/test/suite/test_cursor03.py +++ b/src/third_party/wiredtiger/test/suite/test_cursor03.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_cursor04.py b/src/third_party/wiredtiger/test/suite/test_cursor04.py index b7457ec623d..cfc6953788c 100644 --- a/src/third_party/wiredtiger/test/suite/test_cursor04.py +++ b/src/third_party/wiredtiger/test/suite/test_cursor04.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_cursor05.py b/src/third_party/wiredtiger/test/suite/test_cursor05.py index e0cce3dcb5e..67b69560a8a 100644 --- a/src/third_party/wiredtiger/test/suite/test_cursor05.py +++ b/src/third_party/wiredtiger/test/suite/test_cursor05.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_cursor06.py b/src/third_party/wiredtiger/test/suite/test_cursor06.py index 280d6f09171..c812ff33a9b 100644 --- a/src/third_party/wiredtiger/test/suite/test_cursor06.py +++ b/src/third_party/wiredtiger/test/suite/test_cursor06.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_cursor07.py b/src/third_party/wiredtiger/test/suite/test_cursor07.py index 94989dd68a2..c9967825867 100644 --- a/src/third_party/wiredtiger/test/suite/test_cursor07.py +++ b/src/third_party/wiredtiger/test/suite/test_cursor07.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_cursor08.py b/src/third_party/wiredtiger/test/suite/test_cursor08.py index 82b4a3b7c7c..cef4d5d6166 100644 --- a/src/third_party/wiredtiger/test/suite/test_cursor08.py +++ b/src/third_party/wiredtiger/test/suite/test_cursor08.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_cursor09.py b/src/third_party/wiredtiger/test/suite/test_cursor09.py index de9ae5163b6..feb918553d9 100644 --- a/src/third_party/wiredtiger/test/suite/test_cursor09.py +++ b/src/third_party/wiredtiger/test/suite/test_cursor09.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_cursor10.py b/src/third_party/wiredtiger/test/suite/test_cursor10.py index 11fb43825ad..91e6997ff06 100644 --- a/src/third_party/wiredtiger/test/suite/test_cursor10.py +++ b/src/third_party/wiredtiger/test/suite/test_cursor10.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_cursor11.py b/src/third_party/wiredtiger/test/suite/test_cursor11.py index 1f3ea1555f2..5afbe626e42 100644 --- a/src/third_party/wiredtiger/test/suite/test_cursor11.py +++ b/src/third_party/wiredtiger/test/suite/test_cursor11.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_cursor12.py b/src/third_party/wiredtiger/test/suite/test_cursor12.py index b2c1065b316..50204274b94 100644 --- a/src/third_party/wiredtiger/test/suite/test_cursor12.py +++ b/src/third_party/wiredtiger/test/suite/test_cursor12.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. @@ -39,13 +39,17 @@ class test_cursor12(wttest.WiredTigerTestCase): ('recno', dict(keyfmt='r')), ('string', dict(keyfmt='S')), ] + valuefmt = [ + ('item', dict(valuefmt='u')), + ('string', dict(valuefmt='S')), + ] types = [ ('file', dict(uri='file:modify')), ('lsm', dict(uri='lsm:modify')), ('table', dict(uri='table:modify')), ] # Skip record number keys with LSM. - scenarios = filter_scenarios(make_scenarios(types, keyfmt), + scenarios = filter_scenarios(make_scenarios(types, keyfmt, valuefmt), lambda name, d: not ('lsm' in d['uri'] and d['keyfmt'] == 'r')) # List with original value, final value, and modifications to get @@ -73,7 +77,7 @@ class test_cursor12(wttest.WiredTigerTestCase): 'mods' : [['--', 8, 2]] },{ 'o' : 'ABCDEFGH', # append with gap - 'f' : 'ABCDEFGH\00\00--', + 'f' : 'ABCDEFGH --', 'mods' : [['--', 10, 2]] },{ 'o' : 'ABCDEFGH', # multiple replacements @@ -85,7 +89,7 @@ class test_cursor12(wttest.WiredTigerTestCase): 'mods' : [['+', 1, 1], ['+', 1, 1], ['+', 1, 1], ['-', 1, 1]] },{ 'o' : 'ABCDEFGH', # multiple overlapping gap replacements - 'f' : 'ABCDEFGH\00\00--', + 'f' : 'ABCDEFGH --', 'mods' : [['+', 10, 1], ['+', 10, 1], ['+', 10, 1], ['--', 10, 2]] },{ 'o' : 'ABCDEFGH', # shrink beginning @@ -200,7 +204,8 @@ class test_cursor12(wttest.WiredTigerTestCase): c.set_key(ds.key(row)) self.assertEquals(c.search(), 0) - self.assertEquals(c.get_value(), i['f']) + v = c.get_value() + self.assertEquals(v.replace("\x00", " "), i['f']) if not single: row = row + 1 @@ -215,7 +220,8 @@ class test_cursor12(wttest.WiredTigerTestCase): for i in self.list: c.set_key(ds.key(row)) self.assertEquals(c.search(), 0) - self.assertEquals(c.get_value(), i['f']) + v = c.get_value() + self.assertEquals(v.replace("\x00", " "), i['f']) if not single: row = row + 1 @@ -224,21 +230,21 @@ class test_cursor12(wttest.WiredTigerTestCase): # Smoke-test the modify API, operating on a group of records. def test_modify_smoke(self): ds = SimpleDataSet(self, - self.uri, 100, key_format=self.keyfmt, value_format='u') + self.uri, 100, key_format=self.keyfmt, value_format=self.valuefmt) ds.populate() self.modify_load(ds, False) # Smoke-test the modify API, operating on a single record def test_modify_smoke_single(self): ds = SimpleDataSet(self, - self.uri, 100, key_format=self.keyfmt, value_format='u') + self.uri, 100, key_format=self.keyfmt, value_format=self.valuefmt) ds.populate() self.modify_load(ds, True) # Smoke-test the modify API, closing and re-opening the database. def test_modify_smoke_reopen(self): ds = SimpleDataSet(self, - self.uri, 100, key_format=self.keyfmt, value_format='u') + self.uri, 100, key_format=self.keyfmt, value_format=self.valuefmt) ds.populate() self.modify_load(ds, False) @@ -260,7 +266,7 @@ class test_cursor12(wttest.WiredTigerTestCase): # Populate a database, and checkpoint it so it exists after recovery. ds = SimpleDataSet(self, - self.uri, 100, key_format=self.keyfmt, value_format='u') + self.uri, 100, key_format=self.keyfmt, value_format=self.valuefmt) ds.populate() self.session.checkpoint() self.modify_load(ds, False) @@ -278,7 +284,7 @@ class test_cursor12(wttest.WiredTigerTestCase): # Check that we can perform a large number of modifications to a record. def test_modify_many(self): ds = SimpleDataSet(self, - self.uri, 20, key_format=self.keyfmt, value_format='u') + self.uri, 20, key_format=self.keyfmt, value_format=self.valuefmt) ds.populate() c = self.session.open_cursor(self.uri, None) @@ -301,7 +307,7 @@ class test_cursor12(wttest.WiredTigerTestCase): # Check that modify returns not-found after a delete. def test_modify_delete(self): ds = SimpleDataSet(self, - self.uri, 20, key_format=self.keyfmt, value_format='u') + self.uri, 20, key_format=self.keyfmt, value_format=self.valuefmt) ds.populate() c = self.session.open_cursor(self.uri, None) @@ -319,7 +325,7 @@ class test_cursor12(wttest.WiredTigerTestCase): # and after it's aborted. def test_modify_abort(self): ds = SimpleDataSet(self, - self.uri, 20, key_format=self.keyfmt, value_format='u') + self.uri, 20, key_format=self.keyfmt, value_format=self.valuefmt) ds.populate() # Start a transaction. diff --git a/src/third_party/wiredtiger/test/suite/test_cursor_compare.py b/src/third_party/wiredtiger/test/suite/test_cursor_compare.py index 7dceb9a86cb..0d69a7a985f 100644 --- a/src/third_party/wiredtiger/test/suite/test_cursor_compare.py +++ b/src/third_party/wiredtiger/test/suite/test_cursor_compare.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_cursor_pin.py b/src/third_party/wiredtiger/test/suite/test_cursor_pin.py index 91690ef6ed2..34641585552 100644 --- a/src/third_party/wiredtiger/test/suite/test_cursor_pin.py +++ b/src/third_party/wiredtiger/test/suite/test_cursor_pin.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_cursor_random.py b/src/third_party/wiredtiger/test/suite/test_cursor_random.py index c7736e322e1..27445fe3cc8 100644 --- a/src/third_party/wiredtiger/test/suite/test_cursor_random.py +++ b/src/third_party/wiredtiger/test/suite/test_cursor_random.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_cursor_random02.py b/src/third_party/wiredtiger/test/suite/test_cursor_random02.py index 11ea8e1f489..5303da6488e 100644 --- a/src/third_party/wiredtiger/test/suite/test_cursor_random02.py +++ b/src/third_party/wiredtiger/test/suite/test_cursor_random02.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_cursor_tracker.py b/src/third_party/wiredtiger/test/suite/test_cursor_tracker.py index dee3c6d1b45..02f73cc0978 100644 --- a/src/third_party/wiredtiger/test/suite/test_cursor_tracker.py +++ b/src/third_party/wiredtiger/test/suite/test_cursor_tracker.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_debug_info.py b/src/third_party/wiredtiger/test/suite/test_debug_info.py index 97fa8efbc86..70ab6e058fa 100644 --- a/src/third_party/wiredtiger/test/suite/test_debug_info.py +++ b/src/third_party/wiredtiger/test/suite/test_debug_info.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_drop.py b/src/third_party/wiredtiger/test/suite/test_drop.py index 4be311b8bb2..deea7d6f240 100644 --- a/src/third_party/wiredtiger/test/suite/test_drop.py +++ b/src/third_party/wiredtiger/test/suite/test_drop.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_drop02.py b/src/third_party/wiredtiger/test/suite/test_drop02.py index 7ab891daf15..c2fd5b346ff 100644 --- a/src/third_party/wiredtiger/test/suite/test_drop02.py +++ b/src/third_party/wiredtiger/test/suite/test_drop02.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_drop_create.py b/src/third_party/wiredtiger/test/suite/test_drop_create.py index eb851c3212f..63c833a2e41 100644 --- a/src/third_party/wiredtiger/test/suite/test_drop_create.py +++ b/src/third_party/wiredtiger/test/suite/test_drop_create.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_dump.py b/src/third_party/wiredtiger/test/suite/test_dump.py index 37f4572b5c9..89ae5a81c97 100644 --- a/src/third_party/wiredtiger/test/suite/test_dump.py +++ b/src/third_party/wiredtiger/test/suite/test_dump.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_dupc.py b/src/third_party/wiredtiger/test/suite/test_dupc.py index 6e35eb361a0..ba1726b0bf0 100644 --- a/src/third_party/wiredtiger/test/suite/test_dupc.py +++ b/src/third_party/wiredtiger/test/suite/test_dupc.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_durability01.py b/src/third_party/wiredtiger/test/suite/test_durability01.py index 97c89aabc4c..b4344677696 100644 --- a/src/third_party/wiredtiger/test/suite/test_durability01.py +++ b/src/third_party/wiredtiger/test/suite/test_durability01.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_empty.py b/src/third_party/wiredtiger/test/suite/test_empty.py index ad57bc78036..32726e26c25 100644 --- a/src/third_party/wiredtiger/test/suite/test_empty.py +++ b/src/third_party/wiredtiger/test/suite/test_empty.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_encrypt01.py b/src/third_party/wiredtiger/test/suite/test_encrypt01.py index 5b4be01c861..8d305ee9283 100644 --- a/src/third_party/wiredtiger/test/suite/test_encrypt01.py +++ b/src/third_party/wiredtiger/test/suite/test_encrypt01.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_encrypt02.py b/src/third_party/wiredtiger/test/suite/test_encrypt02.py index c62828cf607..6df8e212057 100644 --- a/src/third_party/wiredtiger/test/suite/test_encrypt02.py +++ b/src/third_party/wiredtiger/test/suite/test_encrypt02.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_encrypt03.py b/src/third_party/wiredtiger/test/suite/test_encrypt03.py index 85be38a27ae..ce6cfbc22e6 100644 --- a/src/third_party/wiredtiger/test/suite/test_encrypt03.py +++ b/src/third_party/wiredtiger/test/suite/test_encrypt03.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_encrypt04.py b/src/third_party/wiredtiger/test/suite/test_encrypt04.py index 3f9058404a8..fae455084a4 100644 --- a/src/third_party/wiredtiger/test/suite/test_encrypt04.py +++ b/src/third_party/wiredtiger/test/suite/test_encrypt04.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_encrypt05.py b/src/third_party/wiredtiger/test/suite/test_encrypt05.py index d4653b2e9b6..fa5ed483462 100644 --- a/src/third_party/wiredtiger/test/suite/test_encrypt05.py +++ b/src/third_party/wiredtiger/test/suite/test_encrypt05.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_encrypt06.py b/src/third_party/wiredtiger/test/suite/test_encrypt06.py index 62e32597f3d..2187182b193 100644 --- a/src/third_party/wiredtiger/test/suite/test_encrypt06.py +++ b/src/third_party/wiredtiger/test/suite/test_encrypt06.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_encrypt07.py b/src/third_party/wiredtiger/test/suite/test_encrypt07.py index 4846a520b00..d4cb7461e1b 100644 --- a/src/third_party/wiredtiger/test/suite/test_encrypt07.py +++ b/src/third_party/wiredtiger/test/suite/test_encrypt07.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_env01.py b/src/third_party/wiredtiger/test/suite/test_env01.py index c4ce7f69dd2..5d83f800a8b 100644 --- a/src/third_party/wiredtiger/test/suite/test_env01.py +++ b/src/third_party/wiredtiger/test/suite/test_env01.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_excl.py b/src/third_party/wiredtiger/test/suite/test_excl.py index 539d599fe32..d12264d72d2 100644 --- a/src/third_party/wiredtiger/test/suite/test_excl.py +++ b/src/third_party/wiredtiger/test/suite/test_excl.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_hazard.py b/src/third_party/wiredtiger/test/suite/test_hazard.py index 73c63099c85..36a9d040fb5 100644 --- a/src/third_party/wiredtiger/test/suite/test_hazard.py +++ b/src/third_party/wiredtiger/test/suite/test_hazard.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_home.py b/src/third_party/wiredtiger/test/suite/test_home.py index 667d466266b..32a2984dd28 100644 --- a/src/third_party/wiredtiger/test/suite/test_home.py +++ b/src/third_party/wiredtiger/test/suite/test_home.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_huffman01.py b/src/third_party/wiredtiger/test/suite/test_huffman01.py index 04a13210e40..83560f4e4b0 100644 --- a/src/third_party/wiredtiger/test/suite/test_huffman01.py +++ b/src/third_party/wiredtiger/test/suite/test_huffman01.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_huffman02.py b/src/third_party/wiredtiger/test/suite/test_huffman02.py index e009734ffb6..924394566c5 100644 --- a/src/third_party/wiredtiger/test/suite/test_huffman02.py +++ b/src/third_party/wiredtiger/test/suite/test_huffman02.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_index01.py b/src/third_party/wiredtiger/test/suite/test_index01.py index bd3794bf730..a802c66a716 100644 --- a/src/third_party/wiredtiger/test/suite/test_index01.py +++ b/src/third_party/wiredtiger/test/suite/test_index01.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_index02.py b/src/third_party/wiredtiger/test/suite/test_index02.py index d2b7b66dfe3..7ded6757c38 100644 --- a/src/third_party/wiredtiger/test/suite/test_index02.py +++ b/src/third_party/wiredtiger/test/suite/test_index02.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_index03.py b/src/third_party/wiredtiger/test/suite/test_index03.py new file mode 100644 index 00000000000..d4670a71937 --- /dev/null +++ b/src/third_party/wiredtiger/test/suite/test_index03.py @@ -0,0 +1,71 @@ +#!/usr/bin/env python +# +# Public Domain 2014-2018 MongoDB, Inc. +# Public Domain 2008-2014 WiredTiger, Inc. +# +# This is free and unencumbered software released into the public domain. +# +# Anyone is free to copy, modify, publish, use, compile, sell, or +# distribute this software, either in source code form or as a compiled +# binary, for any purpose, commercial or non-commercial, and by any +# means. +# +# In jurisdictions that recognize copyright laws, the author or authors +# of this software dedicate any and all copyright interest in the +# software to the public domain. We make this dedication for the benefit +# of the public at large and to the detriment of our heirs and +# successors. We intend this dedication to be an overt act of +# relinquishment in perpetuity of all present and future rights to this +# software under copyright law. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +# IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. + +import wiredtiger, wttest + +# test_index03.py +# Make sure cursors cannot stay open while a new index is created. +class test_index03(wttest.WiredTigerTestCase): + + def key(self, i): + return str('%015d' % i) + + def value(self, i): + return tuple([str(i+1), str(i+2), str(i+3), str(i+4)]) + + def test_index_create(self): + uri = 'table:test_index03' + index1_uri = 'index:test_index03:indx1' + index2_uri = 'index:test_index03:indx2' + config = ',key_format=S,value_format=SSSS' + + session = self.session + session.create(uri, 'columns=(key,col1,col2,col3,col4)' + config) + session.create(index1_uri, 'columns=(col1)' + config) + c1 = session.open_cursor(uri, None) + + # Having cursors open across index creates is not currently allowed. + with self.expectedStderrPattern("Can't create an index for table"): + self.assertRaises(wiredtiger.WiredTigerError, + lambda: session.create(index2_uri, 'columns=(col2)' + config)) + c1.close() + + session.create(index2_uri, 'columns=(col2)' + config) + c1 = session.open_cursor(uri, None) + # Having cursors open across drops is not currently allowed. + # On the drop side, we need to begin using the cursor + for i in xrange(100, 200): + c1[self.key(i)] = self.value(i) + + self.assertRaises(wiredtiger.WiredTigerError, + lambda: session.drop(index2_uri)) + c1.close() + session.drop(index2_uri) + +if __name__ == '__main__': + wttest.run() diff --git a/src/third_party/wiredtiger/test/suite/test_inmem01.py b/src/third_party/wiredtiger/test/suite/test_inmem01.py index 694bcabbe77..e100643af1a 100644 --- a/src/third_party/wiredtiger/test/suite/test_inmem01.py +++ b/src/third_party/wiredtiger/test/suite/test_inmem01.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_inmem02.py b/src/third_party/wiredtiger/test/suite/test_inmem02.py index f2340f6af69..b691f3f9cc8 100644 --- a/src/third_party/wiredtiger/test/suite/test_inmem02.py +++ b/src/third_party/wiredtiger/test/suite/test_inmem02.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_intpack.py b/src/third_party/wiredtiger/test/suite/test_intpack.py index 215ebc8856a..68c9f7b5566 100644 --- a/src/third_party/wiredtiger/test/suite/test_intpack.py +++ b/src/third_party/wiredtiger/test/suite/test_intpack.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_join01.py b/src/third_party/wiredtiger/test/suite/test_join01.py index 167f4793ce4..6204c4deed2 100644 --- a/src/third_party/wiredtiger/test/suite/test_join01.py +++ b/src/third_party/wiredtiger/test/suite/test_join01.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_join02.py b/src/third_party/wiredtiger/test/suite/test_join02.py index 7b85791f17a..f97e496f3c6 100644 --- a/src/third_party/wiredtiger/test/suite/test_join02.py +++ b/src/third_party/wiredtiger/test/suite/test_join02.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_join03.py b/src/third_party/wiredtiger/test/suite/test_join03.py index 552e27632d2..efa445cb619 100644 --- a/src/third_party/wiredtiger/test/suite/test_join03.py +++ b/src/third_party/wiredtiger/test/suite/test_join03.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_join04.py b/src/third_party/wiredtiger/test/suite/test_join04.py index c5ba1ad8c79..cd72e4599f7 100644 --- a/src/third_party/wiredtiger/test/suite/test_join04.py +++ b/src/third_party/wiredtiger/test/suite/test_join04.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_join05.py b/src/third_party/wiredtiger/test/suite/test_join05.py index aedf7a04c24..16a8a2bd1eb 100644 --- a/src/third_party/wiredtiger/test/suite/test_join05.py +++ b/src/third_party/wiredtiger/test/suite/test_join05.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_join06.py b/src/third_party/wiredtiger/test/suite/test_join06.py index c3d2aa2b9ca..52d93f55ea0 100644 --- a/src/third_party/wiredtiger/test/suite/test_join06.py +++ b/src/third_party/wiredtiger/test/suite/test_join06.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_join07.py b/src/third_party/wiredtiger/test/suite/test_join07.py index 87bcc8040d3..7f7c791b236 100644 --- a/src/third_party/wiredtiger/test/suite/test_join07.py +++ b/src/third_party/wiredtiger/test/suite/test_join07.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_join08.py b/src/third_party/wiredtiger/test/suite/test_join08.py index cdcd89a207a..d098dbdc0d3 100644 --- a/src/third_party/wiredtiger/test/suite/test_join08.py +++ b/src/third_party/wiredtiger/test/suite/test_join08.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_join09.py b/src/third_party/wiredtiger/test/suite/test_join09.py index 0441349803e..36d0c5dd18d 100644 --- a/src/third_party/wiredtiger/test/suite/test_join09.py +++ b/src/third_party/wiredtiger/test/suite/test_join09.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_jsondump01.py b/src/third_party/wiredtiger/test/suite/test_jsondump01.py index 13eb7e7be26..4f0c3f71576 100644 --- a/src/third_party/wiredtiger/test/suite/test_jsondump01.py +++ b/src/third_party/wiredtiger/test/suite/test_jsondump01.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_jsondump02.py b/src/third_party/wiredtiger/test/suite/test_jsondump02.py index 5c6bf810e08..f0aed597d4e 100644 --- a/src/third_party/wiredtiger/test/suite/test_jsondump02.py +++ b/src/third_party/wiredtiger/test/suite/test_jsondump02.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_las.py b/src/third_party/wiredtiger/test/suite/test_las.py index 07938c6d80b..f38b11138d2 100644 --- a/src/third_party/wiredtiger/test/suite/test_las.py +++ b/src/third_party/wiredtiger/test/suite/test_las.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. @@ -41,7 +41,7 @@ class test_las(wttest.WiredTigerTestCase): return 'cache_size=50MB' def large_updates(self, session, uri, value, ds, nrows, timestamp=False): - # Insert a large number of records, we'll hang if the lookaside table + # Update a large number of records, we'll hang if the lookaside table # isn't doing its thing. cursor = session.open_cursor(uri) for i in range(1, 10000): @@ -54,6 +54,23 @@ class test_las(wttest.WiredTigerTestCase): session.commit_transaction('commit_timestamp=' + timestamp_str(i + 1)) cursor.close() + def large_modifies(self, session, uri, offset, ds, nrows, timestamp=False): + # Modify a large number of records, we'll hang if the lookaside table + # isn't doing its thing. + cursor = session.open_cursor(uri) + for i in range(1, 10000): + if timestamp == True: + session.begin_transaction() + cursor.set_key(ds.key(nrows + i)) + mods = [] + mod = wiredtiger.Modify('A', offset, 1) + mods.append(mod) + + self.assertEqual(cursor.modify(mods), 0) + if timestamp == True: + session.commit_transaction('commit_timestamp=' + timestamp_str(i + 1)) + cursor.close() + def durable_check(self, check_value, uri, ds, nrows): # Checkpoint and backup so as to simulate recovery self.session.checkpoint() @@ -66,8 +83,9 @@ class test_las(wttest.WiredTigerTestCase): # Skip the initial rows, which were not updated for i in range(0, nrows+1): self.assertEquals(cursor.next(), 0) - #print "Check value : " + str(check_value) - #print "value : " + str(cursor.get_value()) + if (check_value != cursor.get_value()): + print "Check value : " + str(check_value) + print "value : " + str(cursor.get_value()) self.assertTrue(check_value == cursor.get_value()) cursor.close() session.close() @@ -77,7 +95,7 @@ class test_las(wttest.WiredTigerTestCase): # Create a small table. uri = "table:test_las" nrows = 100 - ds = SimpleDataSet(self, uri, nrows, key_format="S") + ds = SimpleDataSet(self, uri, nrows, key_format="S", value_format='u') ds.populate() bigvalue = "aaaaa" * 100 @@ -112,16 +130,30 @@ class test_las(wttest.WiredTigerTestCase): session2.close() # Scenario: 3 + # Check to see LAS working with modify operations + bigvalue3 = "ccccc" * 100 + bigvalue3 = 'AA' + bigvalue3[2:] + session2 = self.conn.open_session() + session2.begin_transaction('isolation=snapshot') + # Apply two modify operations - replacing the first two items with 'A' + self.large_modifies(self.session, uri, 0, ds, nrows) + self.large_modifies(self.session, uri, 1, ds, nrows) + # Check to see the value after recovery + self.durable_check(bigvalue3, uri, ds, nrows) + session2.rollback_transaction() + session2.close() + + # Scenario: 4 # Check to see LAS working with old timestamp - bigvalue3 = "ddddd" * 100 + bigvalue4 = "ddddd" * 100 self.conn.set_timestamp('stable_timestamp=' + timestamp_str(1)) - self.large_updates(self.session, uri, bigvalue3, ds, nrows, timestamp=True) + self.large_updates(self.session, uri, bigvalue4, ds, nrows, timestamp=True) # Check to see data can be see only till the stable_timestamp - self.durable_check(bigvalue2, uri, ds, nrows) + self.durable_check(bigvalue3, uri, ds, nrows) self.conn.set_timestamp('stable_timestamp=' + timestamp_str(i + 1)) # Check to see latest data can be seen - self.durable_check(bigvalue3, uri, ds, nrows) + self.durable_check(bigvalue4, uri, ds, nrows) if __name__ == '__main__': wttest.run() diff --git a/src/third_party/wiredtiger/test/suite/test_lsm01.py b/src/third_party/wiredtiger/test/suite/test_lsm01.py index 8a9972261fd..aaee405d40e 100644 --- a/src/third_party/wiredtiger/test/suite/test_lsm01.py +++ b/src/third_party/wiredtiger/test/suite/test_lsm01.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_lsm02.py b/src/third_party/wiredtiger/test/suite/test_lsm02.py index c35dfa43646..94b8fe5d00c 100644 --- a/src/third_party/wiredtiger/test/suite/test_lsm02.py +++ b/src/third_party/wiredtiger/test/suite/test_lsm02.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_lsm03.py b/src/third_party/wiredtiger/test/suite/test_lsm03.py index 0eb02d546f0..daf9f3d4bc4 100644 --- a/src/third_party/wiredtiger/test/suite/test_lsm03.py +++ b/src/third_party/wiredtiger/test/suite/test_lsm03.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_lsm04.py b/src/third_party/wiredtiger/test/suite/test_lsm04.py index 83e3bb552cf..82b937a713b 100644 --- a/src/third_party/wiredtiger/test/suite/test_lsm04.py +++ b/src/third_party/wiredtiger/test/suite/test_lsm04.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_metadata_cursor01.py b/src/third_party/wiredtiger/test/suite/test_metadata_cursor01.py index f9476a06642..9fc2130fc9a 100644 --- a/src/third_party/wiredtiger/test/suite/test_metadata_cursor01.py +++ b/src/third_party/wiredtiger/test/suite/test_metadata_cursor01.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_nsnap01.py b/src/third_party/wiredtiger/test/suite/test_nsnap01.py index ee97e4f9985..10e9885325a 100644 --- a/src/third_party/wiredtiger/test/suite/test_nsnap01.py +++ b/src/third_party/wiredtiger/test/suite/test_nsnap01.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_nsnap02.py b/src/third_party/wiredtiger/test/suite/test_nsnap02.py index 689c704c97e..0d3266c4476 100644 --- a/src/third_party/wiredtiger/test/suite/test_nsnap02.py +++ b/src/third_party/wiredtiger/test/suite/test_nsnap02.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_nsnap03.py b/src/third_party/wiredtiger/test/suite/test_nsnap03.py index 7be6557d458..4f8663d0112 100644 --- a/src/third_party/wiredtiger/test/suite/test_nsnap03.py +++ b/src/third_party/wiredtiger/test/suite/test_nsnap03.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_nsnap04.py b/src/third_party/wiredtiger/test/suite/test_nsnap04.py index f53c9b5b3cd..ed373fc033b 100644 --- a/src/third_party/wiredtiger/test/suite/test_nsnap04.py +++ b/src/third_party/wiredtiger/test/suite/test_nsnap04.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_overwrite.py b/src/third_party/wiredtiger/test/suite/test_overwrite.py index 0929db549b2..b875ec7d89a 100644 --- a/src/third_party/wiredtiger/test/suite/test_overwrite.py +++ b/src/third_party/wiredtiger/test/suite/test_overwrite.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_pack.py b/src/third_party/wiredtiger/test/suite/test_pack.py index a24ef4fdfe1..99b1480069f 100644 --- a/src/third_party/wiredtiger/test/suite/test_pack.py +++ b/src/third_party/wiredtiger/test/suite/test_pack.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. @@ -74,6 +74,8 @@ class test_pack(wttest.WiredTigerTestCase): forw_idx.set_key(*v) self.assertEquals(forw_idx.search(), 0) self.assertEquals(forw_idx.get_value(), 1234) + forw.close() + forw_idx.close() def test_packing(self): self.check('iii', 0, 101, -99) diff --git a/src/third_party/wiredtiger/test/suite/test_readonly01.py b/src/third_party/wiredtiger/test/suite/test_readonly01.py index ee5f78294f4..b05a0d5bd4a 100644 --- a/src/third_party/wiredtiger/test/suite/test_readonly01.py +++ b/src/third_party/wiredtiger/test/suite/test_readonly01.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. @@ -44,8 +44,8 @@ class test_readonly01(wttest.WiredTigerTestCase, suite_subprocess): # We want a list of directory writable or readonly. # basecfg_list = [ - ('basecfg', dict(basecfg='config_base=true,')), - ('no_basecfg', dict(basecfg='config_base=false,')), + ('basecfg', dict(basecfg='config_base=true,operation_tracking=(enabled=false),')), + ('no_basecfg', dict(basecfg='config_base=false,operation_tracking=(enabled=false),')), ] dir_list = [ ('write', dict(dirchmod=False)), diff --git a/src/third_party/wiredtiger/test/suite/test_readonly02.py b/src/third_party/wiredtiger/test/suite/test_readonly02.py index 3d3de8186d9..a8868d25163 100644 --- a/src/third_party/wiredtiger/test/suite/test_readonly02.py +++ b/src/third_party/wiredtiger/test/suite/test_readonly02.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. @@ -42,11 +42,16 @@ class test_readonly02(wttest.WiredTigerTestCase, suite_subprocess): entries = 10 conn_params = \ - 'create,statistics=(fast),log=(enabled,file_max=100K,zero_fill=true),' + 'create,statistics=(fast),' + \ + 'log=(enabled,file_max=100K,zero_fill=true),' + \ + 'operation_tracking=(enabled=false),' conn_params_rd = \ - 'create,readonly=true,statistics=(fast),log=(enabled,zero_fill=false),' + 'create,readonly=true,statistics=(fast),' + \ + 'log=(enabled,zero_fill=false),' + \ + 'operation_tracking=(enabled=false),' conn_params_rdcfg = \ - 'create,readonly=true,statistics=(fast),log=(enabled),' + 'create,readonly=true,statistics=(fast),log=(enabled),' + \ + 'operation_tracking=(enabled=false),' # # Run to make sure incompatible configuration options return an error. diff --git a/src/third_party/wiredtiger/test/suite/test_readonly03.py b/src/third_party/wiredtiger/test/suite/test_readonly03.py index 474e23981a2..d20a8907119 100644 --- a/src/third_party/wiredtiger/test/suite/test_readonly03.py +++ b/src/third_party/wiredtiger/test/suite/test_readonly03.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. @@ -40,8 +40,8 @@ class test_readonly03(wttest.WiredTigerTestCase, suite_subprocess): uri2 = 'table:test_readonly03_2' create = True - conn_params = 'create,log=(enabled),' - conn_params_rd = 'readonly=true' + conn_params = 'create,log=(enabled),operation_tracking=(enabled=false),' + conn_params_rd = 'readonly=true,operation_tracking=(enabled=false),' session_ops = [ 'alter', 'create', 'compact', 'drop', 'log_flush', 'log_printf', 'rebalance', 'rename', 'salvage', 'truncate', 'upgrade', ] diff --git a/src/third_party/wiredtiger/test/suite/test_rebalance.py b/src/third_party/wiredtiger/test/suite/test_rebalance.py index 867d71b6d35..4d55225e17a 100644 --- a/src/third_party/wiredtiger/test/suite/test_rebalance.py +++ b/src/third_party/wiredtiger/test/suite/test_rebalance.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_reconfig01.py b/src/third_party/wiredtiger/test/suite/test_reconfig01.py index 646b8622a72..fc78ea709a4 100644 --- a/src/third_party/wiredtiger/test/suite/test_reconfig01.py +++ b/src/third_party/wiredtiger/test/suite/test_reconfig01.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. @@ -75,6 +75,13 @@ class test_reconfig01(wttest.WiredTigerTestCase): self.conn.reconfigure("eviction=(threads_min=2)") # Set min and max the same. self.conn.reconfigure("eviction=(threads_min=6,threads_max=6)") + # Set target and trigger with an absolute value. + self.conn.reconfigure("eviction_target=50M,eviction_trigger=100M") + # Set dirty target and trigger with an absolute value + self.conn.reconfigure("eviction_dirty_target=20M," + "eviction_dirty_trigger=40M") + # Set eviction checkpoint target with an absolute value + self.conn.reconfigure("eviction_checkpoint_target=50M") def test_reconfig_lsm_manager(self): # We create and populate a tiny LSM so that we can start off with diff --git a/src/third_party/wiredtiger/test/suite/test_reconfig02.py b/src/third_party/wiredtiger/test/suite/test_reconfig02.py index 0694f3621d8..2508017fd9b 100644 --- a/src/third_party/wiredtiger/test/suite/test_reconfig02.py +++ b/src/third_party/wiredtiger/test/suite/test_reconfig02.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_reconfig03.py b/src/third_party/wiredtiger/test/suite/test_reconfig03.py index 3ab21735bf0..6ebaf8cd662 100644 --- a/src/third_party/wiredtiger/test/suite/test_reconfig03.py +++ b/src/third_party/wiredtiger/test/suite/test_reconfig03.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_reconfig04.py b/src/third_party/wiredtiger/test/suite/test_reconfig04.py index 37288150d35..d5fa606c2e4 100644 --- a/src/third_party/wiredtiger/test/suite/test_reconfig04.py +++ b/src/third_party/wiredtiger/test/suite/test_reconfig04.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_rename.py b/src/third_party/wiredtiger/test/suite/test_rename.py index 0e55a445cd4..4977cc3e07b 100644 --- a/src/third_party/wiredtiger/test/suite/test_rename.py +++ b/src/third_party/wiredtiger/test/suite/test_rename.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_reserve.py b/src/third_party/wiredtiger/test/suite/test_reserve.py index 23159ed7f8a..4b81a47c1c9 100644 --- a/src/third_party/wiredtiger/test/suite/test_reserve.py +++ b/src/third_party/wiredtiger/test/suite/test_reserve.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_salvage.py b/src/third_party/wiredtiger/test/suite/test_salvage.py index 14045afa21e..e1a2349b6b4 100644 --- a/src/third_party/wiredtiger/test/suite/test_salvage.py +++ b/src/third_party/wiredtiger/test/suite/test_salvage.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_schema01.py b/src/third_party/wiredtiger/test/suite/test_schema01.py index 983593dad83..066a708faf5 100644 --- a/src/third_party/wiredtiger/test/suite/test_schema01.py +++ b/src/third_party/wiredtiger/test/suite/test_schema01.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_schema02.py b/src/third_party/wiredtiger/test/suite/test_schema02.py index e34063aa66b..dc50d0e8737 100644 --- a/src/third_party/wiredtiger/test/suite/test_schema02.py +++ b/src/third_party/wiredtiger/test/suite/test_schema02.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_schema03.py b/src/third_party/wiredtiger/test/suite/test_schema03.py index ce6e3684980..12d35d3b949 100644 --- a/src/third_party/wiredtiger/test/suite/test_schema03.py +++ b/src/third_party/wiredtiger/test/suite/test_schema03.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_schema04.py b/src/third_party/wiredtiger/test/suite/test_schema04.py index 765040ae73f..5987176e7c6 100644 --- a/src/third_party/wiredtiger/test/suite/test_schema04.py +++ b/src/third_party/wiredtiger/test/suite/test_schema04.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_schema05.py b/src/third_party/wiredtiger/test/suite/test_schema05.py index f3a75447ee4..0837851e123 100644 --- a/src/third_party/wiredtiger/test/suite/test_schema05.py +++ b/src/third_party/wiredtiger/test/suite/test_schema05.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. @@ -122,6 +122,7 @@ class test_schema05(wttest.WiredTigerTestCase): value = c.get_value() self.assertEqual(value, expect) i += 1 + cursor.close() self.assertEqual(self.nentries, i) for i in range(0, self.nindices): icursor[i].close() diff --git a/src/third_party/wiredtiger/test/suite/test_schema06.py b/src/third_party/wiredtiger/test/suite/test_schema06.py index ef8434a1eaa..f1352a8b87c 100644 --- a/src/third_party/wiredtiger/test/suite/test_schema06.py +++ b/src/third_party/wiredtiger/test/suite/test_schema06.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_schema07.py b/src/third_party/wiredtiger/test/suite/test_schema07.py index 8de0c477157..1fa92dbbc36 100644 --- a/src/third_party/wiredtiger/test/suite/test_schema07.py +++ b/src/third_party/wiredtiger/test/suite/test_schema07.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_shared_cache01.py b/src/third_party/wiredtiger/test/suite/test_shared_cache01.py index 5b348a0ca87..9ebdd5093fc 100644 --- a/src/third_party/wiredtiger/test/suite/test_shared_cache01.py +++ b/src/third_party/wiredtiger/test/suite/test_shared_cache01.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. @@ -158,6 +158,39 @@ class test_shared_cache01(wttest.WiredTigerTestCase): self.add_records(sess, 0, nops) self.closeConnections() + # Opening a connection with absolute values for eviction config should fail + def test_shared_cache_absolute_evict_config(self): + nops = 1000 + self.assertRaisesWithMessage(wiredtiger.WiredTigerError, + lambda: self.openConnections(['WT_TEST1', 'WT_TEST2'], + pool_opts = ',shared_cache=(name=pool,size=50M,reserve=20M),' + 'eviction_target=10M,'), '/Shared cache configuration requires a ' + 'percentage value for eviction target/') + + self.assertRaisesWithMessage(wiredtiger.WiredTigerError, + lambda: self.openConnections(['WT_TEST1', 'WT_TEST2'], + pool_opts = ',shared_cache=(name=pool,size=50M,reserve=20M),' + 'eviction_trigger=10M,'), '/Shared cache configuration requires a ' + 'percentage value for eviction trigger/') + + self.assertRaisesWithMessage(wiredtiger.WiredTigerError, + lambda: self.openConnections(['WT_TEST1', 'WT_TEST2'], + pool_opts = ',shared_cache=(name=pool,size=50M,reserve=20M),' + 'eviction_dirty_target=10M,'), '/Shared cache configuration ' + 'requires a percentage value for eviction dirty target/') + + self.assertRaisesWithMessage(wiredtiger.WiredTigerError, + lambda: self.openConnections(['WT_TEST1', 'WT_TEST2'], + pool_opts = ',shared_cache=(name=pool,size=50M,reserve=20M),' + 'eviction_dirty_trigger=10M,'), '/Shared cache configuration ' + 'requires a percentage value for eviction dirty trigger/') + + self.assertRaisesWithMessage(wiredtiger.WiredTigerError, + lambda: self.openConnections(['WT_TEST1', 'WT_TEST2'], + pool_opts = ',shared_cache=(name=pool,size=50M,reserve=20M),' + 'eviction_checkpoint_target=10M,'), '/Shared cache configuration ' + 'requires a percentage value for eviction checkpoint target/') + # Test verbose output @unittest.skip("Verbose output handling") def test_shared_cache_verbose(self): diff --git a/src/third_party/wiredtiger/test/suite/test_shared_cache02.py b/src/third_party/wiredtiger/test/suite/test_shared_cache02.py index c6e5209ff8a..3d5b29f1969 100644 --- a/src/third_party/wiredtiger/test/suite/test_shared_cache02.py +++ b/src/third_party/wiredtiger/test/suite/test_shared_cache02.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. @@ -162,5 +162,35 @@ class test_shared_cache02(wttest.WiredTigerTestCase): self.closeConnections() + # Test reconfigure with absolute value for eviction config fails + def test_shared_cache_reconfig04(self): + nops = 1000 + self.openConnections(['WT_TEST1', 'WT_TEST2'], + pool_opts = ',shared_cache=(name=pool,size=50M,reserve=20M),') + + for sess in self.sessions: + sess.create(self.uri, "key_format=S,value_format=S") + self.add_records(sess, 0, nops) + + connection = self.conns[0] + # Reconfiguring with absolute value of eviction trigger should fail. + self.assertRaisesWithMessage(wiredtiger.WiredTigerError, + lambda: connection.reconfigure("shared_cache=(name=pool," + "size=20M,reserve=10M),eviction_trigger=10M"),'/Shared cache ' + 'configuration requires a percentage value for eviction trigger/') + + connection = self.conns[1] + # Reconfiguring with absolute value for eviction target should fail. + self.assertRaisesWithMessage(wiredtiger.WiredTigerError, + lambda: connection.reconfigure("shared_cache=(name=pool," + "size=20M,reserve=10M),eviction_target=10M"),'/Shared cache ' + 'configuration requires a percentage value for eviction target/') + + # Reconfigure with percentage value for eviction target passes + self.conns[0].reconfigure("shared_cache=(name=pool,reserve=20M)," + "eviction_target=50") + + self.closeConnections() + if __name__ == '__main__': wttest.run() diff --git a/src/third_party/wiredtiger/test/suite/test_split.py b/src/third_party/wiredtiger/test/suite/test_split.py index b3de91d3cdb..7dc7edb7ed8 100644 --- a/src/third_party/wiredtiger/test/suite/test_split.py +++ b/src/third_party/wiredtiger/test/suite/test_split.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_stat01.py b/src/third_party/wiredtiger/test/suite/test_stat01.py index 03f0507ced4..e106bf63c51 100644 --- a/src/third_party/wiredtiger/test/suite/test_stat01.py +++ b/src/third_party/wiredtiger/test/suite/test_stat01.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_stat02.py b/src/third_party/wiredtiger/test/suite/test_stat02.py index d7f66424c5d..5c093fcc697 100644 --- a/src/third_party/wiredtiger/test/suite/test_stat02.py +++ b/src/third_party/wiredtiger/test/suite/test_stat02.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_stat03.py b/src/third_party/wiredtiger/test/suite/test_stat03.py index d486cbda0b6..ffe5dc1a110 100644 --- a/src/third_party/wiredtiger/test/suite/test_stat03.py +++ b/src/third_party/wiredtiger/test/suite/test_stat03.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_stat04.py b/src/third_party/wiredtiger/test/suite/test_stat04.py index af5f0e282bf..6b98b9d0bb8 100644 --- a/src/third_party/wiredtiger/test/suite/test_stat04.py +++ b/src/third_party/wiredtiger/test/suite/test_stat04.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_stat05.py b/src/third_party/wiredtiger/test/suite/test_stat05.py index 6478bb5e58c..f78004ae55e 100644 --- a/src/third_party/wiredtiger/test/suite/test_stat05.py +++ b/src/third_party/wiredtiger/test/suite/test_stat05.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_stat06.py b/src/third_party/wiredtiger/test/suite/test_stat06.py index fb8110624c4..75ab41594f1 100644 --- a/src/third_party/wiredtiger/test/suite/test_stat06.py +++ b/src/third_party/wiredtiger/test/suite/test_stat06.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_stat_log01.py b/src/third_party/wiredtiger/test/suite/test_stat_log01.py index 8f17be042d6..a82d8338ee4 100644 --- a/src/third_party/wiredtiger/test/suite/test_stat_log01.py +++ b/src/third_party/wiredtiger/test/suite/test_stat_log01.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_stat_log02.py b/src/third_party/wiredtiger/test/suite/test_stat_log02.py index 96f4ccf3da8..c0395baff11 100644 --- a/src/third_party/wiredtiger/test/suite/test_stat_log02.py +++ b/src/third_party/wiredtiger/test/suite/test_stat_log02.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_sweep01.py b/src/third_party/wiredtiger/test/suite/test_sweep01.py index 4d11942dc54..4939c270635 100644 --- a/src/third_party/wiredtiger/test/suite/test_sweep01.py +++ b/src/third_party/wiredtiger/test/suite/test_sweep01.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. @@ -44,7 +44,7 @@ class test_sweep01(wttest.WiredTigerTestCase, suite_subprocess): numkv = 1000 conn_config = 'file_manager=(close_handle_minimum=0,' + \ 'close_idle_time=6,close_scan_interval=2),' + \ - 'statistics=(fast),' + 'statistics=(fast),operation_tracking=(enabled=false),' types = [ ('row', dict(tabletype='row', diff --git a/src/third_party/wiredtiger/test/suite/test_sweep02.py b/src/third_party/wiredtiger/test/suite/test_sweep02.py index 76931ecbfbd..e50cfb949bb 100644 --- a/src/third_party/wiredtiger/test/suite/test_sweep02.py +++ b/src/third_party/wiredtiger/test/suite/test_sweep02.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_sweep03.py b/src/third_party/wiredtiger/test/suite/test_sweep03.py index 5ff747b1056..356d270ddea 100644 --- a/src/third_party/wiredtiger/test/suite/test_sweep03.py +++ b/src/third_party/wiredtiger/test/suite/test_sweep03.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_timestamp01.py b/src/third_party/wiredtiger/test/suite/test_timestamp01.py index 09a264e2afd..0610b8aea7b 100644 --- a/src/third_party/wiredtiger/test/suite/test_timestamp01.py +++ b/src/third_party/wiredtiger/test/suite/test_timestamp01.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_timestamp02.py b/src/third_party/wiredtiger/test/suite/test_timestamp02.py index 60a6eef3a55..ab1572a7c06 100644 --- a/src/third_party/wiredtiger/test/suite/test_timestamp02.py +++ b/src/third_party/wiredtiger/test/suite/test_timestamp02.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_timestamp03.py b/src/third_party/wiredtiger/test/suite/test_timestamp03.py index c340d258d62..db1d97365d7 100644 --- a/src/third_party/wiredtiger/test/suite/test_timestamp03.py +++ b/src/third_party/wiredtiger/test/suite/test_timestamp03.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_timestamp04.py b/src/third_party/wiredtiger/test/suite/test_timestamp04.py index f7052448208..48ec7fac9a6 100644 --- a/src/third_party/wiredtiger/test/suite/test_timestamp04.py +++ b/src/third_party/wiredtiger/test/suite/test_timestamp04.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. @@ -27,7 +27,7 @@ # OTHER DEALINGS IN THE SOFTWARE. # # test_timestamp04.py -# Timestamps: Test that rollback_to_stable obeys expected visibility rules +# Timestamps: Test that rollback_to_stable obeys expected visibility rules. # from suite_subprocess import suite_subprocess @@ -49,7 +49,7 @@ class test_timestamp04(wttest.WiredTigerTestCase, suite_subprocess): ('V2', dict(conn_config=',log=(enabled)', using_log=True)), ] - # Minimum cache_size requirement of lsm is 31MB + # Minimum cache_size requirement of lsm is 31MB. types = [ ('col_fix', dict(empty=1, cacheSize='cache_size=20MB', extra_config=',key_format=r,value_format=8t')), ('col_var', dict(empty=0, cacheSize='cache_size=20MB', extra_config=',key_format=r')), @@ -68,14 +68,12 @@ class test_timestamp04(wttest.WiredTigerTestCase, suite_subprocess): cur = session.open_cursor(tablename, None) if missing == False: actual = dict((k, v) for k, v in cur if v != 0) - if prn == True: - print "CHECK : Expected" - print expected - print "CHECK : Actual" - print actual + if actual != expected: + print "missing: ", sorted(set(expected) - set(actual)) + print "extras: ", sorted(set(actual) - set(expected)) self.assertTrue(actual == expected) - # Search for the expected items as well as iterating + # Search for the expected items as well as iterating. for k, v in expected.iteritems(): if missing == False: self.assertEqual(cur[k], v, "for key " + str(k)) @@ -114,7 +112,7 @@ class test_timestamp04(wttest.WiredTigerTestCase, suite_subprocess): self.ConnectionOpen(self.cacheSize) # Configure small page sizes to ensure eviction comes through and we - # have a somewhat complex tree + # have a somewhat complex tree. config_default = 'key_format=i,value_format=i,memory_page_max=32k,leaf_page_max=8k,internal_page_max=8k' config_nolog = ',log=(enabled=false)' # @@ -133,7 +131,7 @@ class test_timestamp04(wttest.WiredTigerTestCase, suite_subprocess): self.session.create(self.table_nots_nolog, config_default + config_nolog + self.extra_config) cur_nots_nolog = self.session.open_cursor(self.table_nots_nolog) - # Insert keys each with timestamp=key, in some order + # Insert keys each with timestamp=key, in some order. key_range = 10000 keys = range(1, key_range + 1) @@ -168,41 +166,41 @@ class test_timestamp04(wttest.WiredTigerTestCase, suite_subprocess): self.conn.rollback_to_stable() # Check that we see the inserted value (i.e. 1) for all the keys in - # non-timestamp tables + # non-timestamp tables. self.check(self.session, 'read_timestamp=' + latest_ts, self.table_nots_log, dict((k, 1) for k in keys[:])) self.check(self.session, 'read_timestamp=' + latest_ts, self.table_nots_nolog, dict((k, 1) for k in keys[:])) # For non-logged tables the behavior is consistent across connections - # with or without log enabled + # with or without log enabled. # Check that we see the inserted value (i.e. 1) for the keys in a - # timestamp table till the stable_timestamp only. + # timestamped table until the stable_timestamp only. self.check(self.session, 'read_timestamp=' + latest_ts, self.table_ts_nolog, dict((k, 1) for k in keys[:(key_range / 2)])) self.check(self.session, 'read_timestamp=' + latest_ts, self.table_ts_nolog, dict((k, 1) for k in keys[(key_range / 2 + 1):]), missing=True) - # For logged tables behavior changes for rollback_to_stable based on + # For logged tables, the behavior of rollback_to_stable changes based on # whether connection level logging is enabled or not. if self.using_log == True: - # When log is enabled, none of the keys will be rolled back. - # Check that we see all the keys + # When the log is enabled, none of the keys will be rolled back. + # Check that we see all the keys. self.check(self.session, 'read_timestamp=' + latest_ts, self.table_ts_log, dict((k, 1) for k in keys[:])) else: - # When log is disabled, keys will be rolled back till stable_timestamp - # Check that we see the insertions are rolled back in timestamp tables - # till the stable_timestamp + # When the log is disabled, the keys will be rolled back until stable_timestamp. + # Check that we see the insertions are rolled back in timestamped tables + # until the stable_timestamp. self.check(self.session, 'read_timestamp=' + latest_ts, self.table_ts_log, dict((k, 1) for k in keys[:(key_range / 2)])) self.check(self.session, 'read_timestamp=' + latest_ts, self.table_ts_log, dict((k, 1) for k in keys[(key_range / 2 + 1):]), missing=True) - # Bump the oldest timestamp, we're not going back... + # Bump the oldest timestamp, we're not going back. self.conn.set_timestamp('oldest_timestamp=' + stable_ts) - # Update the values again in preparation for rolling back more + # Update the values again in preparation for rolling back more. for k in keys: cur_nots_log[k] = 2 cur_nots_nolog[k] = 2 @@ -212,7 +210,7 @@ class test_timestamp04(wttest.WiredTigerTestCase, suite_subprocess): self.session.commit_transaction('commit_timestamp=' + timestamp_str(k + key_range)) # Scenario: 3 - # Check that we see all values updated (i.e 2) in all tables + # Check that we see all values updated (i.e 2) in all tables. latest_ts = timestamp_str(2 * key_range) self.check(self.session, 'read_timestamp=' + latest_ts, self.table_nots_log, dict((k, 2) for k in keys[:])) @@ -225,20 +223,20 @@ class test_timestamp04(wttest.WiredTigerTestCase, suite_subprocess): # Scenario: 4 # Advance the stable_timestamp by a quarter range and rollback. - # three-quarter timestamps will be rolled back. + # Three-fourths of the later timestamps will be rolled back. stable_ts = timestamp_str(key_range + key_range / 4) self.conn.set_timestamp('stable_timestamp=' + stable_ts) self.conn.rollback_to_stable() # Check that we see the updated value (i.e. 2) for all the keys in - # non-timestamp tables + # non-timestamped tables. self.check(self.session, 'read_timestamp=' + latest_ts, self.table_nots_log, dict((k, 2) for k in keys[:])) self.check(self.session, 'read_timestamp=' + latest_ts, self.table_nots_nolog, dict((k, 2) for k in keys[:])) # For non-logged tables the behavior is consistent across connections - # with or without log enabled - # Check that we see only half key ranges in timestamp tables. we see + # with or without log enabled. + # Check that we see only half key ranges in timestamp tables. We see # the updated value (i.e. 2) for the first quarter keys and old values # (i.e. 1) for the second quarter keys. self.check(self.session, 'read_timestamp=' + latest_ts, @@ -251,12 +249,12 @@ class test_timestamp04(wttest.WiredTigerTestCase, suite_subprocess): # whether connection level logging is enabled or not. if self.using_log == True: # When log is enabled, none of the keys will be rolled back. - # Check that we see all the keys + # Check that we see all the keys. self.check(self.session, 'read_timestamp=' + latest_ts, self.table_ts_log, dict((k, 2) for k in keys[:])) else: - # When log is disabled, keys will be rolled back till stable_timestamp - # Check that we see only half key ranges in timestamp tables. we see + # When log is disabled, keys will be rolled back until the stable_timestamp. + # Check that we see only half the key ranges in timestamped tables. We see # the updated value (i.e. 2) for the first quarter keys and old values # (i.e. 1) for the second quarter keys. self.check(self.session, 'read_timestamp=' + latest_ts, diff --git a/src/third_party/wiredtiger/test/suite/test_timestamp05.py b/src/third_party/wiredtiger/test/suite/test_timestamp05.py index e8433c27a1e..be369c3fa69 100644 --- a/src/third_party/wiredtiger/test/suite/test_timestamp05.py +++ b/src/third_party/wiredtiger/test/suite/test_timestamp05.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_timestamp06.py b/src/third_party/wiredtiger/test/suite/test_timestamp06.py index ab4440656df..37e3d0da2d2 100644 --- a/src/third_party/wiredtiger/test/suite/test_timestamp06.py +++ b/src/third_party/wiredtiger/test/suite/test_timestamp06.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_timestamp07.py b/src/third_party/wiredtiger/test/suite/test_timestamp07.py index 214cc2ee164..e4d281f2038 100644 --- a/src/third_party/wiredtiger/test/suite/test_timestamp07.py +++ b/src/third_party/wiredtiger/test/suite/test_timestamp07.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. @@ -191,6 +191,7 @@ class test_timestamp07(wttest.WiredTigerTestCase, suite_subprocess): c2 = self.session.open_cursor(uri2) self.session.create(uri3, 'key_format=i,value_format=S') c3 = self.session.open_cursor(uri3) + # print "tables created" # Insert keys 1..nkeys each with timestamp=key, in some order. orig_keys = range(1, self.nkeys+1) @@ -204,6 +205,8 @@ class test_timestamp07(wttest.WiredTigerTestCase, suite_subprocess): c3[k] = self.value self.session.commit_transaction('commit_timestamp=' + timestamp_str(k)) + # print "value inserted in all tables, reading..." + # Now check that we see the expected state when reading at each # timestamp. for k in orig_keys: @@ -212,6 +215,8 @@ class test_timestamp07(wttest.WiredTigerTestCase, suite_subprocess): self.check(self.session, 'read_timestamp=' + timestamp_str(k), k + 1, None) + # print "all values read, updating timestamps" + # Bump the oldest timestamp, we're not going back... self.assertTimestampsEqual(self.conn.query_timestamp(), timestamp_str(self.nkeys)) self.oldts = self.stablets = timestamp_str(self.nkeys) @@ -219,6 +224,8 @@ class test_timestamp07(wttest.WiredTigerTestCase, suite_subprocess): self.conn.set_timestamp('stable_timestamp=' + self.stablets) # print "Oldest " + self.oldts + # print "inserting value2 in all tables" + # Update them and retry. random.shuffle(keys) count = 0 @@ -241,6 +248,7 @@ class test_timestamp07(wttest.WiredTigerTestCase, suite_subprocess): # Take a checkpoint using the given configuration. Then verify # whether value2 appears in a copy of that data or not. + # print "check_stable 1" self.check_stable(self.value2, 0, self.nkeys, self.nkeys if self.using_log else 0) # Update the stable timestamp to the latest, but not the oldest @@ -248,6 +256,7 @@ class test_timestamp07(wttest.WiredTigerTestCase, suite_subprocess): # timestamp is moved we should see all keys with value2. self.stablets = timestamp_str(self.nkeys*2) self.conn.set_timestamp('stable_timestamp=' + self.stablets) + # print "check_stable 2" self.check_stable(self.value2, self.nkeys, self.nkeys, self.nkeys) # If we're not using the log we're done. @@ -281,6 +290,7 @@ class test_timestamp07(wttest.WiredTigerTestCase, suite_subprocess): # of that data or not. Both tables that are logged should see # all the data regardless of timestamps. The table that is not # logged should not see any of it. + # print "check_stable 3" self.check_stable(self.value3, 0, self.nkeys, self.nkeys) if __name__ == '__main__': diff --git a/src/third_party/wiredtiger/test/suite/test_timestamp08.py b/src/third_party/wiredtiger/test/suite/test_timestamp08.py index 3dfc3b65bc8..e72e3c62b58 100644 --- a/src/third_party/wiredtiger/test/suite/test_timestamp08.py +++ b/src/third_party/wiredtiger/test/suite/test_timestamp08.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_timestamp09.py b/src/third_party/wiredtiger/test/suite/test_timestamp09.py index 0e3a6b8dab3..9b7d88bf64e 100644 --- a/src/third_party/wiredtiger/test/suite/test_timestamp09.py +++ b/src/third_party/wiredtiger/test/suite/test_timestamp09.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. @@ -109,7 +109,7 @@ class test_timestamp09(wttest.WiredTigerTestCase, suite_subprocess): self.assertRaisesWithMessage(wiredtiger.WiredTigerError, lambda: self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(3) + ',stable_timestamp=' + timestamp_str(1)), - '/oldest timestamp must not be later than stable timestamp/') + '/oldest timestamp 0*3 must not be later than stable timestamp 0*1/') # Oldest timestamp is 3 at the moment, trying to set it to an earlier # timestamp is a no-op. @@ -128,7 +128,7 @@ class test_timestamp09(wttest.WiredTigerTestCase, suite_subprocess): self.assertRaisesWithMessage(wiredtiger.WiredTigerError, lambda: self.conn.set_timestamp('oldest_timestamp=' + timestamp_str(6)), - '/oldest timestamp must not be later than stable timestamp/') + '/oldest timestamp 0*6 must not be later than stable timestamp 0*5/') # Commit timestamp >= Stable timestamp. # Check both timestamp_transaction and commit_transaction API. diff --git a/src/third_party/wiredtiger/test/suite/test_timestamp11.py b/src/third_party/wiredtiger/test/suite/test_timestamp11.py new file mode 100644 index 00000000000..f98b7c47b2b --- /dev/null +++ b/src/third_party/wiredtiger/test/suite/test_timestamp11.py @@ -0,0 +1,150 @@ +#!/usr/bin/env python +# +# Public Domain 2014-2018 MongoDB, Inc. +# Public Domain 2008-2014 WiredTiger, Inc. +# +# This is free and unencumbered software released into the public domain. +# +# Anyone is free to copy, modify, publish, use, compile, sell, or +# distribute this software, either in source code form or as a compiled +# binary, for any purpose, commercial or non-commercial, and by any +# means. +# +# In jurisdictions that recognize copyright laws, the author or authors +# of this software dedicate any and all copyright interest in the +# software to the public domain. We make this dedication for the benefit +# of the public at large and to the detriment of our heirs and +# successors. We intend this dedication to be an overt act of +# relinquishment in perpetuity of all present and future rights to this +# software under copyright law. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +# IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. +# +# test_timestamp11.py +# Timestamps: mixed timestamp usage +# + +from suite_subprocess import suite_subprocess +import wiredtiger, wttest + +def timestamp_str(t): + return '%x' % t + +class test_timestamp11(wttest.WiredTigerTestCase, suite_subprocess): + def test_timestamp_range(self): + if not wiredtiger.timestamp_build(): + self.skipTest('requires a timestamp build') + + base = 'timestamp11' + uri = 'file:' + base + self.session.create(uri, 'key_format=S,value_format=S') + + # Test that mixed timestamp usage where some transactions use timestamps + # and others don't behave in the expected way. + + # Insert two data items at timestamp 2 + c = self.session.open_cursor(uri) + self.session.begin_transaction() + self.session.timestamp_transaction( + 'commit_timestamp=' + timestamp_str(2)) + c['key'] = 'value2' + c['key2'] = 'value2' + self.session.commit_transaction() + c.close() + + # + # Modify one key without a timestamp and modify the other with a + # later timestamp. + # + c = self.session.open_cursor(uri) + self.session.begin_transaction() + self.session.timestamp_transaction( + 'commit_timestamp=' + timestamp_str(5)) + c['key'] = 'value5' + self.session.commit_transaction() + c.close() + + c = self.session.open_cursor(uri) + self.session.begin_transaction() + c['key2'] = 'valueNOTS' + self.session.commit_transaction() + c.close() + + # + # Set the stable timestamp and then roll back to it. The first key + # should roll back to the original value and the second key should + # remain at the non-timestamped value. Also the non-timestamped value + # stays regardless of rollbacks or reading at a timestamp. + # + stable_ts = timestamp_str(2) + self.conn.set_timestamp('stable_timestamp=' + stable_ts) + self.conn.rollback_to_stable() + + c = self.session.open_cursor(uri) + self.session.begin_transaction() + self.assertEquals(c['key'], 'value2') + self.assertEquals(c['key2'], 'valueNOTS') + self.session.commit_transaction() + c.close() + + c = self.session.open_cursor(uri) + self.session.begin_transaction('read_timestamp=' + stable_ts) + self.assertEquals(c['key'], 'value2') + self.assertEquals(c['key2'], 'valueNOTS') + self.session.commit_transaction() + c.close() + + # + # Repeat but swapping the keys using or not using timestamps. + # + c = self.session.open_cursor(uri) + self.session.begin_transaction() + self.session.timestamp_transaction( + 'commit_timestamp=' + timestamp_str(5)) + c['key2'] = 'value5' + self.session.commit_transaction() + c.close() + + c = self.session.open_cursor(uri) + self.session.begin_transaction() + c['key'] = 'valueNOTS' + self.session.commit_transaction() + c.close() + + # Read with each timestamp and without any timestamp. + # + # Without a timestamp. We should see the latest value for each. + c = self.session.open_cursor(uri) + self.session.begin_transaction() + self.assertEquals(c['key'], 'valueNOTS') + self.assertEquals(c['key2'], 'value5') + self.session.commit_transaction() + c.close() + + # With timestamp 2. Both non-timestamped values override the original + # value at timestamp 2. + c = self.session.open_cursor(uri) + self.session.begin_transaction('read_timestamp=' + stable_ts) + self.assertEquals(c['key'], 'valueNOTS') + self.assertEquals(c['key2'], 'valueNOTS') + self.session.commit_transaction() + c.close() + + # With timestamp 5. We rolled back the first one and never re-inserted + # one at that timestamp and inserted without a timestamp. For the second + # we inserted at timestamp 5 after the non-timestamped insert. + c = self.session.open_cursor(uri) + self.session.begin_transaction('read_timestamp=' + timestamp_str(5)) + self.assertEquals(c['key'], 'valueNOTS') + self.assertEquals(c['key2'], 'value5') + self.session.commit_transaction() + c.close() + +if __name__ == '__main__': + wttest.run() diff --git a/src/third_party/wiredtiger/test/suite/test_truncate01.py b/src/third_party/wiredtiger/test/suite/test_truncate01.py index 88d29d8443a..b26c9cf8fe1 100644 --- a/src/third_party/wiredtiger/test/suite/test_truncate01.py +++ b/src/third_party/wiredtiger/test/suite/test_truncate01.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_truncate02.py b/src/third_party/wiredtiger/test/suite/test_truncate02.py index 06fa6bfc94f..cb68d90d9ff 100644 --- a/src/third_party/wiredtiger/test/suite/test_truncate02.py +++ b/src/third_party/wiredtiger/test/suite/test_truncate02.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_truncate03.py b/src/third_party/wiredtiger/test/suite/test_truncate03.py index 613ab772571..2344234b38a 100644 --- a/src/third_party/wiredtiger/test/suite/test_truncate03.py +++ b/src/third_party/wiredtiger/test/suite/test_truncate03.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_txn01.py b/src/third_party/wiredtiger/test/suite/test_txn01.py index e0030909331..78c334db62f 100644 --- a/src/third_party/wiredtiger/test/suite/test_txn01.py +++ b/src/third_party/wiredtiger/test/suite/test_txn01.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_txn02.py b/src/third_party/wiredtiger/test/suite/test_txn02.py index 6e54270d3ad..94b939596d1 100644 --- a/src/third_party/wiredtiger/test/suite/test_txn02.py +++ b/src/third_party/wiredtiger/test/suite/test_txn02.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_txn03.py b/src/third_party/wiredtiger/test/suite/test_txn03.py index 53e9b8e6206..c5f9a38f904 100644 --- a/src/third_party/wiredtiger/test/suite/test_txn03.py +++ b/src/third_party/wiredtiger/test/suite/test_txn03.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_txn04.py b/src/third_party/wiredtiger/test/suite/test_txn04.py index 470e37d6a9c..26b71c1af74 100644 --- a/src/third_party/wiredtiger/test/suite/test_txn04.py +++ b/src/third_party/wiredtiger/test/suite/test_txn04.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_txn05.py b/src/third_party/wiredtiger/test/suite/test_txn05.py index 045cd570d8a..52e27f14612 100644 --- a/src/third_party/wiredtiger/test/suite/test_txn05.py +++ b/src/third_party/wiredtiger/test/suite/test_txn05.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_txn06.py b/src/third_party/wiredtiger/test/suite/test_txn06.py index 520c25f9b86..b9e88088707 100644 --- a/src/third_party/wiredtiger/test/suite/test_txn06.py +++ b/src/third_party/wiredtiger/test/suite/test_txn06.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. @@ -40,11 +40,6 @@ class test_txn06(wttest.WiredTigerTestCase, suite_subprocess): source_uri = 'table:' + tablename + "_src" nrows = 100000 - def conn_config(self): - if not wiredtiger.verbose_build(): - self.skipTest('requires a verbose build') - return '' - def test_long_running(self): # Populate a table SimpleDataSet(self, self.source_uri, self.nrows).populate() diff --git a/src/third_party/wiredtiger/test/suite/test_txn07.py b/src/third_party/wiredtiger/test/suite/test_txn07.py index fe1bdd346a0..58a3e2093ce 100644 --- a/src/third_party/wiredtiger/test/suite/test_txn07.py +++ b/src/third_party/wiredtiger/test/suite/test_txn07.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_txn08.py b/src/third_party/wiredtiger/test/suite/test_txn08.py index 5ba0a529a31..41122be58ef 100644 --- a/src/third_party/wiredtiger/test/suite/test_txn08.py +++ b/src/third_party/wiredtiger/test/suite/test_txn08.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_txn09.py b/src/third_party/wiredtiger/test/suite/test_txn09.py index cc5771ef681..421c4e69887 100644 --- a/src/third_party/wiredtiger/test/suite/test_txn09.py +++ b/src/third_party/wiredtiger/test/suite/test_txn09.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_txn10.py b/src/third_party/wiredtiger/test/suite/test_txn10.py index d27f83bf2e4..32cf13e0ea4 100644 --- a/src/third_party/wiredtiger/test/suite/test_txn10.py +++ b/src/third_party/wiredtiger/test/suite/test_txn10.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_txn11.py b/src/third_party/wiredtiger/test/suite/test_txn11.py index 0cb2dfd615a..4e689bac1b3 100644 --- a/src/third_party/wiredtiger/test/suite/test_txn11.py +++ b/src/third_party/wiredtiger/test/suite/test_txn11.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_txn12.py b/src/third_party/wiredtiger/test/suite/test_txn12.py index 21065083633..f94f116abe8 100644 --- a/src/third_party/wiredtiger/test/suite/test_txn12.py +++ b/src/third_party/wiredtiger/test/suite/test_txn12.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_txn13.py b/src/third_party/wiredtiger/test/suite/test_txn13.py index ec6f2813457..1c1224a545d 100644 --- a/src/third_party/wiredtiger/test/suite/test_txn13.py +++ b/src/third_party/wiredtiger/test/suite/test_txn13.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_txn14.py b/src/third_party/wiredtiger/test/suite/test_txn14.py index 2245f49ae85..104db8a9da2 100644 --- a/src/third_party/wiredtiger/test/suite/test_txn14.py +++ b/src/third_party/wiredtiger/test/suite/test_txn14.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_txn15.py b/src/third_party/wiredtiger/test/suite/test_txn15.py index 762c0613735..ddc78678c01 100644 --- a/src/third_party/wiredtiger/test/suite/test_txn15.py +++ b/src/third_party/wiredtiger/test/suite/test_txn15.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_txn16.py b/src/third_party/wiredtiger/test/suite/test_txn16.py index 921b67e3f6b..916b735a9d9 100644 --- a/src/third_party/wiredtiger/test/suite/test_txn16.py +++ b/src/third_party/wiredtiger/test/suite/test_txn16.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_txn17.py b/src/third_party/wiredtiger/test/suite/test_txn17.py index 213dd67ff53..a15acb4aed3 100644 --- a/src/third_party/wiredtiger/test/suite/test_txn17.py +++ b/src/third_party/wiredtiger/test/suite/test_txn17.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_txn18.py b/src/third_party/wiredtiger/test/suite/test_txn18.py index ec3cc7bae00..30bf0070325 100644 --- a/src/third_party/wiredtiger/test/suite/test_txn18.py +++ b/src/third_party/wiredtiger/test/suite/test_txn18.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. @@ -93,7 +93,7 @@ class test_txn18(wttest.WiredTigerTestCase, suite_subprocess): lambda:self.wiredtiger_open(errdir, self.conn_recerror), msg) # If recover=error is run on the directory and returns an error, - # make sure when we subsequenty open with recover=on it properly + # make sure when we subsequently open with recover=on it properly # recovers all the data. self.assertRaisesWithMessage(wiredtiger.WiredTigerError, lambda:self.wiredtiger_open(newdir, self.conn_recerror), msg) diff --git a/src/third_party/wiredtiger/test/suite/test_unicode01.py b/src/third_party/wiredtiger/test/suite/test_unicode01.py index 21d6a714268..06d7381d353 100644 --- a/src/third_party/wiredtiger/test/suite/test_unicode01.py +++ b/src/third_party/wiredtiger/test/suite/test_unicode01.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_upgrade.py b/src/third_party/wiredtiger/test/suite/test_upgrade.py index 6672daf11d6..40cc1c50b1e 100644 --- a/src/third_party/wiredtiger/test/suite/test_upgrade.py +++ b/src/third_party/wiredtiger/test/suite/test_upgrade.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_util01.py b/src/third_party/wiredtiger/test/suite/test_util01.py index a181acd5568..19ac54af821 100644 --- a/src/third_party/wiredtiger/test/suite/test_util01.py +++ b/src/third_party/wiredtiger/test/suite/test_util01.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_util02.py b/src/third_party/wiredtiger/test/suite/test_util02.py index b9bc3bd26c1..7d2ea5aca9d 100644 --- a/src/third_party/wiredtiger/test/suite/test_util02.py +++ b/src/third_party/wiredtiger/test/suite/test_util02.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_util03.py b/src/third_party/wiredtiger/test/suite/test_util03.py index e5e4f624991..908cd11bbf5 100644 --- a/src/third_party/wiredtiger/test/suite/test_util03.py +++ b/src/third_party/wiredtiger/test/suite/test_util03.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_util04.py b/src/third_party/wiredtiger/test/suite/test_util04.py index cbfd63b6b65..fde92a6a629 100644 --- a/src/third_party/wiredtiger/test/suite/test_util04.py +++ b/src/third_party/wiredtiger/test/suite/test_util04.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_util07.py b/src/third_party/wiredtiger/test/suite/test_util07.py index 7d3d6ec5f37..f6efdea8830 100644 --- a/src/third_party/wiredtiger/test/suite/test_util07.py +++ b/src/third_party/wiredtiger/test/suite/test_util07.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_util08.py b/src/third_party/wiredtiger/test/suite/test_util08.py index 3c4561da263..8f209e6c5af 100644 --- a/src/third_party/wiredtiger/test/suite/test_util08.py +++ b/src/third_party/wiredtiger/test/suite/test_util08.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_util09.py b/src/third_party/wiredtiger/test/suite/test_util09.py index 3138ea087a2..8cd283628ba 100644 --- a/src/third_party/wiredtiger/test/suite/test_util09.py +++ b/src/third_party/wiredtiger/test/suite/test_util09.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_util11.py b/src/third_party/wiredtiger/test/suite/test_util11.py index 68cb751d364..1a07535108d 100644 --- a/src/third_party/wiredtiger/test/suite/test_util11.py +++ b/src/third_party/wiredtiger/test/suite/test_util11.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_util12.py b/src/third_party/wiredtiger/test/suite/test_util12.py index 6f4638e93aa..ea66b7678a3 100644 --- a/src/third_party/wiredtiger/test/suite/test_util12.py +++ b/src/third_party/wiredtiger/test/suite/test_util12.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_util13.py b/src/third_party/wiredtiger/test/suite/test_util13.py index 79dc232f5ef..9d6391f029c 100644 --- a/src/third_party/wiredtiger/test/suite/test_util13.py +++ b/src/third_party/wiredtiger/test/suite/test_util13.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_util14.py b/src/third_party/wiredtiger/test/suite/test_util14.py index e2a9f41f0d4..194d64147a5 100644 --- a/src/third_party/wiredtiger/test/suite/test_util14.py +++ b/src/third_party/wiredtiger/test/suite/test_util14.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_util15.py b/src/third_party/wiredtiger/test/suite/test_util15.py index 33096e71bee..0c2bb4061fe 100644 --- a/src/third_party/wiredtiger/test/suite/test_util15.py +++ b/src/third_party/wiredtiger/test/suite/test_util15.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_util16.py b/src/third_party/wiredtiger/test/suite/test_util16.py index 00e68c1017a..1f32e4beb4a 100644 --- a/src/third_party/wiredtiger/test/suite/test_util16.py +++ b/src/third_party/wiredtiger/test/suite/test_util16.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_util17.py b/src/third_party/wiredtiger/test/suite/test_util17.py index decc1fabf1d..ea669e58983 100644 --- a/src/third_party/wiredtiger/test/suite/test_util17.py +++ b/src/third_party/wiredtiger/test/suite/test_util17.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_verify.py b/src/third_party/wiredtiger/test/suite/test_verify.py index 615b8e278ac..16ba49ef4a7 100644 --- a/src/third_party/wiredtiger/test/suite/test_verify.py +++ b/src/third_party/wiredtiger/test/suite/test_verify.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/test_version.py b/src/third_party/wiredtiger/test/suite/test_version.py index c854a393c2b..7738af985da 100644 --- a/src/third_party/wiredtiger/test/suite/test_version.py +++ b/src/third_party/wiredtiger/test/suite/test_version.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/wtdataset.py b/src/third_party/wiredtiger/test/suite/wtdataset.py index 3093f550e8b..04dd5315919 100644 --- a/src/third_party/wiredtiger/test/suite/wtdataset.py +++ b/src/third_party/wiredtiger/test/suite/wtdataset.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/wtscenario.py b/src/third_party/wiredtiger/test/suite/wtscenario.py index 7bdfc4f23bc..76824e428df 100644 --- a/src/third_party/wiredtiger/test/suite/wtscenario.py +++ b/src/third_party/wiredtiger/test/suite/wtscenario.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/suite/wttest.py b/src/third_party/wiredtiger/test/suite/wttest.py index c654370718c..40d8bd689cd 100644 --- a/src/third_party/wiredtiger/test/suite/wttest.py +++ b/src/third_party/wiredtiger/test/suite/wttest.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. @@ -490,6 +490,39 @@ class WiredTigerTestCase(unittest.TestCase): with self.expectedStderr(message): self.assertRaises(exceptionType, expr) + def assertRaisesException(self, exceptionType, expr, + exceptionString=None, optional=False): + """ + Like TestCase.assertRaises(), with some additional options. + If the exceptionString argument is used, the exception's string + must match it. If optional is set, then no assertion occurs + if the exception doesn't occur. + Returns true if the assertion is raised. + """ + raised = False + try: + expr() + except BaseException, err: + if not isinstance(err, exceptionType): + self.fail('Exception of incorrect type raised, got type: ' + \ + str(type(err))) + if exceptionString != None and exceptionString != str(err): + self.fail('Exception with incorrect string raised, got: "' + \ + str(err) + '"') + raised = True + if not raised and not optional: + self.fail('no assertion raised') + return raised + + def raisesBusy(self, expr): + """ + Execute the expression, returning true if a 'Resource busy' + exception is raised, returning false if no exception is raised. + Any other exception raises a test suite failure. + """ + return self.assertRaisesException(wiredtiger.WiredTigerError, \ + expr, exceptionString='Resource busy', optional=True) + def assertTimestampsEqual(self, ts1, ts2): """ TestCase.assertEqual() for timestamps diff --git a/src/third_party/wiredtiger/test/suite/wtthread.py b/src/third_party/wiredtiger/test/suite/wtthread.py index 54fc4a1961e..77408b57c5e 100644 --- a/src/third_party/wiredtiger/test/suite/wtthread.py +++ b/src/third_party/wiredtiger/test/suite/wtthread.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/syscall/syscall.py b/src/third_party/wiredtiger/test/syscall/syscall.py index 1caa718b4fc..5567db9cc6b 100644 --- a/src/third_party/wiredtiger/test/syscall/syscall.py +++ b/src/third_party/wiredtiger/test/syscall/syscall.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/syscall/wt2336_base/base.run b/src/third_party/wiredtiger/test/syscall/wt2336_base/base.run index db455c97474..328d5b8b7dd 100644 --- a/src/third_party/wiredtiger/test/syscall/wt2336_base/base.run +++ b/src/third_party/wiredtiger/test/syscall/wt2336_base/base.run @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/syscall/wt2336_base/main.c b/src/third_party/wiredtiger/test/syscall/wt2336_base/main.c index f22af235c19..54f9999a126 100644 --- a/src/third_party/wiredtiger/test/syscall/wt2336_base/main.c +++ b/src/third_party/wiredtiger/test/syscall/wt2336_base/main.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/thread/file.c b/src/third_party/wiredtiger/test/thread/file.c index 16488259d4d..716931c58df 100644 --- a/src/third_party/wiredtiger/test/thread/file.c +++ b/src/third_party/wiredtiger/test/thread/file.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/thread/rw.c b/src/third_party/wiredtiger/test/thread/rw.c index bda54d388b5..dc46e9b595d 100644 --- a/src/third_party/wiredtiger/test/thread/rw.c +++ b/src/third_party/wiredtiger/test/thread/rw.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/thread/stats.c b/src/third_party/wiredtiger/test/thread/stats.c index 9eadcce2214..675a1bcfd87 100644 --- a/src/third_party/wiredtiger/test/thread/stats.c +++ b/src/third_party/wiredtiger/test/thread/stats.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/thread/t.c b/src/third_party/wiredtiger/test/thread/t.c index 3dc196eef5d..f509b6d73e2 100644 --- a/src/third_party/wiredtiger/test/thread/t.c +++ b/src/third_party/wiredtiger/test/thread/t.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/thread/thread.h b/src/third_party/wiredtiger/test/thread/thread.h index bcba442b4c1..27d5287d30e 100644 --- a/src/third_party/wiredtiger/test/thread/thread.h +++ b/src/third_party/wiredtiger/test/thread/thread.h @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/utility/misc.c b/src/third_party/wiredtiger/test/utility/misc.c index 9d8fa28d3d7..a632e832f53 100644 --- a/src/third_party/wiredtiger/test/utility/misc.c +++ b/src/third_party/wiredtiger/test/utility/misc.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. @@ -148,6 +148,24 @@ testutil_make_work_dir(const char *dir) } /* + * testutil_progress -- + * Print a progress message to the progress file. + */ +void +testutil_progress(TEST_OPTS *opts, const char *message) +{ + FILE *fp; + time_t now; + + if ((fp = fopen(opts->progress_file_name, "a")) == NULL) + testutil_die(errno, "fopen"); + (void)time(&now); + fprintf(fp, "[%" PRIuMAX "] %s\n", (uintmax_t)now, message); + if (fclose(fp) != 0) + testutil_die(errno, "fclose"); +} + +/* * testutil_cleanup -- * Delete the existing work directory and free the options structure. */ @@ -161,6 +179,7 @@ testutil_cleanup(TEST_OPTS *opts) testutil_clean_work_dir(opts->home); free(opts->uri); + free(opts->progress_file_name); free(opts->home); } diff --git a/src/third_party/wiredtiger/test/utility/parse_opts.c b/src/third_party/wiredtiger/test/utility/parse_opts.c index bcdd7d3f7bd..9e4f309615e 100644 --- a/src/third_party/wiredtiger/test/utility/parse_opts.c +++ b/src/third_party/wiredtiger/test/utility/parse_opts.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. @@ -119,6 +119,14 @@ testutil_parse_opts(int argc, char * const *argv, TEST_OPTS *opts) opts->home, len, "WT_TEST.%s", opts->progname)); } + /* + * Setup the progress file name. + */ + len = strlen(opts->home) + 20; + opts->progress_file_name = dmalloc(len); + testutil_check(__wt_snprintf(opts->progress_file_name, len, + "%s/progress.txt", opts->home)); + /* Setup the default URI string */ len = strlen("table:") + strlen(opts->progname) + 10; opts->uri = dmalloc(len); diff --git a/src/third_party/wiredtiger/test/utility/test_util.h b/src/third_party/wiredtiger/test/utility/test_util.h index 7500df8d5e5..1054412adfe 100644 --- a/src/third_party/wiredtiger/test/utility/test_util.h +++ b/src/third_party/wiredtiger/test/utility/test_util.h @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. @@ -48,6 +48,7 @@ /* Generic option parsing structure shared by all test cases. */ typedef struct { char *home; + char *progress_file_name; const char *progname; enum { TABLE_COL=1, /* Fixed-length column store */ TABLE_FIX=2, /* Variable-length column store */ @@ -240,6 +241,7 @@ void testutil_cleanup(TEST_OPTS *); bool testutil_is_flag_set(const char *); void testutil_make_work_dir(const char *); int testutil_parse_opts(int, char * const *, TEST_OPTS *); +void testutil_progress(TEST_OPTS *, const char *); void testutil_work_dir_from_path(char *, size_t, const char *); WT_THREAD_RET thread_append(void *); diff --git a/src/third_party/wiredtiger/test/utility/thread.c b/src/third_party/wiredtiger/test/utility/thread.c index ed8c7e5ae73..1e1bd0bf575 100644 --- a/src/third_party/wiredtiger/test/utility/thread.c +++ b/src/third_party/wiredtiger/test/utility/thread.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/windows/windows_shim.c b/src/third_party/wiredtiger/test/windows/windows_shim.c index 8986c1a5ae1..b40a9e01a42 100644 --- a/src/third_party/wiredtiger/test/windows/windows_shim.c +++ b/src/third_party/wiredtiger/test/windows/windows_shim.c @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/windows/windows_shim.h b/src/third_party/wiredtiger/test/windows/windows_shim.h index 5a065ed8a79..2f46be9daee 100644 --- a/src/third_party/wiredtiger/test/windows/windows_shim.h +++ b/src/third_party/wiredtiger/test/windows/windows_shim.h @@ -1,5 +1,5 @@ /*- - * Public Domain 2014-2017 MongoDB, Inc. + * Public Domain 2014-2018 MongoDB, Inc. * Public Domain 2008-2014 WiredTiger, Inc. * * This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/test/wtperf/test_conf_dump.py b/src/third_party/wiredtiger/test/wtperf/test_conf_dump.py index bbfb8e819e9..e0063eb3a0a 100644 --- a/src/third_party/wiredtiger/test/wtperf/test_conf_dump.py +++ b/src/third_party/wiredtiger/test/wtperf/test_conf_dump.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. diff --git a/src/third_party/wiredtiger/tools/optrack/find-latency-spikes.py b/src/third_party/wiredtiger/tools/optrack/find-latency-spikes.py new file mode 100755 index 00000000000..0870ced29f0 --- /dev/null +++ b/src/third_party/wiredtiger/tools/optrack/find-latency-spikes.py @@ -0,0 +1,1140 @@ +#!/usr/bin/env python +# +# Public Domain 2014-2018 MongoDB, Inc. +# Public Domain 2008-2014 WiredTiger, Inc. +# +# This is free and unencumbered software released into the public domain. +# +# Anyone is free to copy, modify, publish, use, compile, sell, or +# distribute this software, either in source code form or as a compiled +# binary, for any purpose, commercial or non-commercial, and by any +# means. +# +# In jurisdictions that recognize copyright laws, the author or authors +# of this software dedicate any and all copyright interest in the +# software to the public domain. We make this dedication for the benefit +# of the public at large and to the detriment of our heirs and +# successors. We intend this dedication to be an overt act of +# relinquishment in perpetuity of all present and future rights to this +# software under copyright law. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +# IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. +#!/usr/bin/env python + +import argparse +from bokeh.layouts import column +from bokeh.models import ColumnDataSource, CustomJS, HoverTool, FixedTicker +from bokeh.models import LabelSet, Legend, LegendItem +from bokeh.models import NumeralTickFormatter, OpenURL, Range1d, TapTool +from bokeh.models.annotations import Label +from bokeh.plotting import figure, output_file, reset_output, save, show +from bokeh.resources import CDN +import matplotlib +import numpy as np +import os +import pandas as pd +import sys +import traceback + +# A directory where we store cross-file plots for each bucket of the outlier +# histogram. +# +bucketDir = "BUCKET-FILES"; + +# A static list of available CSS colors +colorList = []; + +# Codes for various colors for printing of informational and error messages. +# +class color: + PURPLE = '\033[95m' + CYAN = '\033[96m' + DARKCYAN = '\033[36m' + BLUE = '\033[94m' + GREEN = '\033[92m' + YELLOW = '\033[93m' + RED = '\033[91m' + BOLD = '\033[1m' + UNDERLINE = '\033[4m' + END = '\033[0m' + +# A function name mapped to its corresponding color. +# +funcToColor = {}; +lastColorUsed = 0; + +# The smallest and the largest timestamps seen across all files. +# +firstTimeStamp = sys.maxsize; +lastTimeStamp = 0; + +# A dictionary that holds function-specific threshold values telling +# us when the function is to be considered an outlier. These values +# would be read from a config file, if supplied by the user. +# +outlierThresholdDict = {}; +outlierPrettyNames = {}; + +# A dictionary that holds a reference to the raw dataframe for each file. +# +perFileDataFrame = {}; + +# A dictionary that holds the intervals data per function. +# +perFuncDF = {}; + +# Data frames and largest stack depth for each file. +perFileDataFrame = {}; +perFileLargestStackDepth = {}; + +plotWidth = 1200; +pixelsForTitle = 30; +pixelsPerHeightUnit = 30; +pixelsPerWidthUnit = 5; + +# The name of the time units that were used when recording timestamps. +# We assume that it's nanoseconds by default. Alternative units can be +# set in the configuration file. +# +timeUnitString = "nanoseconds"; + +# The coefficient by which we multiply the standard deviation when +# setting the outlier threshold, in case it is not specified by the user. +# +STDEV_MULT = 2; + +def initColorList(): + + global colorList; + + colorList = matplotlib.colors.cnames.keys(); + + for color in colorList: + # Some browsers break if you try to give them 'sage' + if (color == "sage"): + colorList.remove(color); + +# +# Each unique function name gets a unique color. +# If we run out of colors, we repeat them from the +# beginning of the list. +# +def getColorForFunction(function): + + global colorList; + global lastColorUsed; + global funcToColor; + + if not funcToColor.has_key(function): + funcToColor[function] = colorList[lastColorUsed % len(colorList)]; + lastColorUsed += 1; + + return funcToColor[function]; + +# +# An intervalEnd is a tuple of three items. +# item #0 is the timestamp, +# item #1 is the event type, +# item #2 is the function name. +# +def getIntervalData(intervalBeginningsStack, intervalEnd, logfile): + + errorOccurred = False; + matchFound = False; + + if (intervalEnd[1] != 1): + logfile.write( + "getIntervaldata: only rows with event type 1 can be used.\n"); + logfile.write(str(intervalEnd) + "\n"); + return None; + + if (len(intervalBeginningsStack) < 1): + logfile.write("Nothing on the intervalBeginningsStack. " + + "I cannot find the beginning for this interval.\n"); + logfile.write(str(intervalEnd) + "\n"); + return None; + + while (not matchFound): + intervalBegin = intervalBeginningsStack.pop(); + if (intervalBegin is None): + logfile.write("Could not find the matching operation begin record" + + " for the following operation end record: \n"); + logfile.write(str(intervalEnd) + "\n"); + return None; + if (intervalBegin[2] != intervalEnd[2]): + logfile.write("Operation end record does not match the available " + + "operation begin record. " + + "Your log file may be incomplete.\n" + + "Skipping the begin record.\n"); + logfile.write("Begin: " + str(intervalBegin) + "\n"); + logfile.write("End: " + str(intervalEnd) + "\n"); + errorOccurred = True; + else: + matchFound = True; + + return intervalBegin[0], intervalEnd[0], intervalEnd[2], errorOccurred; + +def plotOutlierHistogram(dataframe, maxOutliers, func, durationThreshold, + averageDuration, maxDuration): + + global pixelsForTitle; + global pixelsPerHeightUnit; + global plotWidth; + global timeUnitString; + + cds = ColumnDataSource(dataframe); + + figureTitle = "Occurrences of " + func + " that took longer than " \ + + durationThreshold + "."; + + hover = HoverTool(tooltips = [ + ("interval start", "@lowerbound{0,0}"), + ("interval end", "@upperbound{0,0}")]); + + TOOLS = [hover, "tap, reset"]; + + p = figure(title = figureTitle, plot_width = plotWidth, + plot_height = min(500, (max(5, (maxOutliers + 1)) \ + * pixelsPerHeightUnit + \ + pixelsForTitle)), + x_axis_label = "Execution timeline (" + timeUnitString + ")", + y_axis_label = "Number of outliers", + tools = TOOLS, toolbar_location="above"); + + y_ticker_max = p.plot_height / pixelsPerHeightUnit; + y_ticker_step = max(1, (maxOutliers + 1)/y_ticker_max); + y_upper_bound = (maxOutliers / y_ticker_step + 1) * y_ticker_step; + + p.yaxis.ticker = FixedTicker(ticks = + range(0, y_upper_bound, y_ticker_step)); + p.ygrid.ticker = FixedTicker(ticks = + range(0, y_upper_bound, y_ticker_step)); + p.xaxis.formatter = NumeralTickFormatter(format="0,"); + + p.y_range = Range1d(0, y_upper_bound); + + p.quad(left = 'lowerbound', right = 'upperbound', bottom = 'bottom', + top = 'height', color = funcToColor[func], source = cds, + nonselection_fill_color=funcToColor[func], + nonselection_fill_alpha = 1.0, + line_color = "lightgrey", + selection_fill_color = funcToColor[func], + selection_line_color="grey" + ); + + # Add an annotation to the chart + # + y_max = dataframe['height'].max(); + text = "Average duration: " + '{0:,.0f}'.format(averageDuration) + \ + ". Maximum duration: " + '{0:,.0f}'.format(maxDuration) + "."; + mytext = Label(x=0, y=y_upper_bound-y_ticker_step, text=text, + text_color = "grey", text_font = "helvetica", + text_font_size = "10pt", + text_font_style = "italic"); + p.add_layout(mytext); + + url = "@bucketfiles"; + taptool = p.select(type=TapTool); + taptool.callback = OpenURL(url=url); + + return p; + +# From all timestamps subtract the smallest observed timestamp, so that +# our execution timeline begins at zero. +# Cleanup the data to remove incomplete records and fix their effects. +# +def normalizeIntervalData(): + + global firstTimeStamp; + global perFileDataFrame; + + print(color.BLUE + color.BOLD + "Normalizing data..." + color.END); + + for file, df in perFileDataFrame.iteritems(): + df['origstart'] = df['start']; + df['start'] = df['start'] - firstTimeStamp; + df['end'] = df['end'] - firstTimeStamp; + +def reportDataError(logfile, logfilename): + + if (logfile is not sys.stdout): + print(color.BOLD + color.RED + "Your data may have errors. " + + "Check the file " + logfilename + " for details." + color.END); + return True; + +# +# Go over all operation records in the dataframe and assign stack depths. +# +def assignStackDepths(dataframe): + + stack = []; + + df = dataframe.sort_values(by=['start']); + df = df.reset_index(drop = True); + + for i in range(len(df.index)): + + myStartTime = df.at[i, 'start']; + + # Pop all items off stack whose end time is earlier than my + # start time. They are not part of my stack, so I don't want to + # count them. + # + while (len(stack) > 0 and stack[-1] < myStartTime): + stack.pop(); + + df.at[i, 'stackdepth'] = len(stack); + stack.append(df.at[i, 'end']); + + return df; + +def createCallstackSeries(data, logfilename): + + global firstTimeStamp; + global lastTimeStamp; + + colors = []; + beginIntervals = []; + dataFrame = None; + endIntervals = []; + errorReported = False; + functionNames = []; + intervalBeginningsStack = []; + largestStackDepth = 0; + logfile = None; + thisIsFirstRow = True; + + # Let's open the log file. + try: + logfile = open(logfilename, "w"); + except: + logfile = sys.stdout; + + for row in data.itertuples(): + # row[0] is the timestamp, row[1] is the event type, + # row[2] is the function name. + # + if (row[1] == 0): + intervalBeginningsStack.append(row); + elif (row[1] == 1): + try: + intervalBegin, intervalEnd, function, error\ + = getIntervalData(intervalBeginningsStack, row, logfile); + if (error and (not errorReported)): + errorReported = reportDataError(logfile, logfilename); + except: + if (not errorReported): + errorReported = reportDataError(logfile, logfilename); + continue; + + if (intervalBegin < firstTimeStamp): + firstTimeStamp = intervalBegin; + if (intervalEnd > lastTimeStamp): + lastTimeStamp = intervalEnd; + + colors.append(getColorForFunction(function)); + beginIntervals.append(intervalBegin); + endIntervals.append(intervalEnd); + functionNames.append(function); + + else: + print("Invalid event in this line:"); + print(str(row[0]) + " " + str(row[1]) + " " + str(row[2])); + continue; + + if (len(intervalBeginningsStack) > 0): + logfile.write(str(len(intervalBeginningsStack)) + " operations had a " + + "begin record, but no matching end records. " + + "Please check that your operation tracking macros " + + "are properly inserted.\n"); + if (not errorReported): + errorReported = reportDataError(logfile, logfilename); + intervalBeginningsStack = []; + + dict = {}; + dict['color'] = colors; + dict['start'] = beginIntervals; + dict['end'] = endIntervals; + dict['function'] = functionNames; + dict['stackdepth'] = [0] * len(beginIntervals); + + dataframe = pd.DataFrame(data=dict); + dataframe = assignStackDepths(dataframe); + + dataframe['durations'] = dataframe['end'] - dataframe['start']; + dataframe['stackdepthNext'] = dataframe['stackdepth'] + 1; + + return dataframe; + +# For each function we only show the legend once. In this dictionary we +# keep track of colors already used. +# +colorAlreadyUsedInLegend = {}; + +def createLegendFigure(legendDict): + + global pixelsForTitle; + global plotWidth; + + FUNCS_PER_ROW = 5; + HSPACE_BETWEEN_FUNCS = 10; + VSPACE_BETWEEN_FUNCS = 1; + + funcs = []; + colors = []; + x_coords = []; + y_coords = []; + pixelsForLegendItem = 20; + + # Get a sorted list of functions and their + # corresponding colors. + # + for func in sorted(legendDict.keys()): + funcs.append(func); + colors.append(legendDict[func]); + + # Figure out the coordinates of functions on the plot + # + for i in range(len(funcs)): + + x_coord = i % (FUNCS_PER_ROW) + 1; + x_coord += i % (FUNCS_PER_ROW) * HSPACE_BETWEEN_FUNCS; + x_coords.append(x_coord); + + y_coord = (i/FUNCS_PER_ROW) + 1; + y_coord += (i/FUNCS_PER_ROW) * VSPACE_BETWEEN_FUNCS; + y_coords.append(y_coord); + + data = {}; + data['func'] = funcs; + data['color'] = colors; + data['left'] = x_coords; + data['bottom'] = y_coords; + + df = pd.DataFrame(data=data); + + max_ycoord = df['bottom'].max(); + df['bottom'] = (max_ycoord + 1) - df['bottom']; + + df['right'] = df['left'] + 1; + df['top'] = df['bottom'] + 1; + + cds = ColumnDataSource(df); + + p = figure(title="TRACKED FUNCTIONS", + plot_width=plotWidth, + plot_height = max((max_ycoord + 2) * pixelsForLegendItem, 90), + tools = [], toolbar_location="above", + x_range = (0, (FUNCS_PER_ROW + 1)* HSPACE_BETWEEN_FUNCS), + y_range = (0, max_ycoord + 2), + x_axis_label = "", + y_axis_label = ""); + + p.title.align = "center"; + p.title.text_font_style = "normal"; + + p.xaxis.axis_line_color = "lightgrey"; + p.xaxis.major_tick_line_color = None; + p.xaxis.minor_tick_line_color = None; + p.xaxis.major_label_text_font_size = '0pt'; + + p.yaxis.axis_line_color = "lightgrey"; + p.yaxis.major_tick_line_color = None; + p.yaxis.minor_tick_line_color = None; + p.yaxis.major_label_text_font_size = '0pt'; + + p.xgrid.grid_line_color = None; + p.ygrid.grid_line_color = None; + + p.outline_line_width = 1 + p.outline_line_alpha = 1 + p.outline_line_color = "lightgrey" + + p.quad(left = 'left', right = 'right', bottom = 'bottom', + top = 'top', color = 'color', line_color = "lightgrey", + line_width = 0.5, source=cds); + + labels = LabelSet(x='right', y='bottom', text='func', level='glyph', + text_font_size = "10pt", + x_offset=3, y_offset=0, source=cds, render_mode='canvas'); + p.add_layout(labels); + + return p; + +def generateBucketChartForFile(figureName, dataframe, y_max, x_min, x_max): + + global colorAlreadyUsedInLegend; + global funcToColor; + global plotWidth; + global timeUnitString; + + MAX_ITEMS_PER_LEGEND = 10; + numLegends = 0; + legendItems = {}; + pixelsPerStackLevel = 30; + pixelsPerLegend = 60; + pixelsForTitle = 30; + + cds = ColumnDataSource(dataframe); + + hover = HoverTool(tooltips=[ + ("function", "@function"), + ("duration", "@durations{0,0}"), + ("log file begin timestamp", "@origstart{0,0}") + ]); + + TOOLS = [hover]; + + p = figure(title=figureName, plot_width=plotWidth, + x_range = (x_min, x_max), + y_range = (0, y_max+1), + x_axis_label = "Time (" + timeUnitString + ")", + y_axis_label = "Stack depth", + tools = TOOLS, toolbar_location="above"); + + # No minor ticks or labels on the y-axis + p.yaxis.major_tick_line_color = None; + p.yaxis.minor_tick_line_color = None; + p.yaxis.major_label_text_font_size = '0pt'; + p.yaxis.ticker = FixedTicker(ticks = range(0, y_max+1)); + p.ygrid.ticker = FixedTicker(ticks = range(0, y_max+1)); + + p.xaxis.formatter = NumeralTickFormatter(format="0,"); + p.title.text_font_style = "bold"; + + p.quad(left = 'start', right = 'end', bottom = 'stackdepth', + top = 'stackdepthNext', color = 'color', line_color = "lightgrey", + line_width = 0.5, source=cds); + + for func, fColor in funcToColor.iteritems(): + + # If this function is not present in this dataframe, + # we don't care about it. + # + boolVec = (dataframe['function'] == func); + fDF = dataframe[boolVec]; + if (fDF.size == 0): + continue; + + # If we already added a color to any legend, we don't + # add it again to avoid redundancy in the charts and + # in order not to waste space. + # + if (colorAlreadyUsedInLegend.has_key(fColor)): + continue; + else: + colorAlreadyUsedInLegend[fColor] = True; + + legendItems[func] = fColor; + + # Plot height is the function of the maximum call stack and the number of + # legends + p.plot_height = max((y_max+1) * pixelsPerStackLevel, 100) + pixelsForTitle; + + return p, legendItems; + +def generateEmptyDataset(): + + dict = {}; + dict['color'] = [0]; + dict['durations'] = [0]; + dict['start'] = [0]; + dict['end'] = [0]; + dict['function'] = [""]; + dict['stackdepth'] = [0]; + dict['stackdepthNext'] = [0]; + + return pd.DataFrame(data=dict); + +# +# Here we generate plots that span all the input files. Each plot shows +# the timelines for all files, stacked vertically. The timeline shows +# the function callstacks over time from this file. +# +# Since a single timeline is too large to fit on a single screen, we generate +# a separate HTML file with plots for bucket "i". A bucket is a vertical slice +# across the timelines for all files. We call it a bucket, because it +# corresponds to a bucket in the outlier histogram. +# +def generateCrossFilePlotsForBucket(i, lowerBound, upperBound, navigatorDF): + + global bucketDir; + global colorAlreadyUsedInLegend; + global timeUnitString; + + aggregateLegendDict = {}; + figuresForAllFiles = []; + fileName = bucketDir + "/bucket-" + str(i) + ".html"; + + reset_output(); + + intervalTitle = "Interval #" + str(i) + ". {:,}".format(lowerBound) + \ + " to " + "{:,}".format(upperBound) + \ + " " + timeUnitString + "."; + + # Generate a navigator chart, which shows where we are in the + # trace and allows moving around the trace. + # + navigatorFigure = generateNavigatorFigure(navigatorDF, i, intervalTitle); + figuresForAllFiles.append(navigatorFigure); + + # The following dictionary keeps track of legends. We need + # a legend for each new HTML file. So we reset the dictionary + # before generating a new file. + # + colorAlreadyUsedInLegend = {}; + + # Select from the dataframe for this file the records whose 'start' + # and 'end' timestamps fall within the lower and upper bound. + # + for fname in sorted(perFileDataFrame.keys()): + + fileDF = perFileDataFrame[fname]; + + # Select operations whose start timestamp falls within + # the current interval, delimited by lowerBound and upperBound. + # + startInBucket = fileDF.loc[(fileDF['start'] >= lowerBound) + & (fileDF['start'] < upperBound)]; + + # Select operations whose end timestamp falls within + # the current interval, delimited by lowerBound and upperBound. + # + endInBucket = fileDF.loc[(fileDF['end'] > lowerBound) + & (fileDF['end'] <= upperBound)]; + + # Select operations that begin before this interval and end after + # this interval, but continue throughout this interval. The interval + # is delimited by lowerBound and upperBound. + # + spanBucket = fileDF.loc[(fileDF['start'] < lowerBound) + & (fileDF['end'] > upperBound)]; + + frames = [startInBucket, endInBucket, spanBucket]; + bucketDF = pd.concat(frames).drop_duplicates().reset_index(drop=True); + + if (bucketDF.size == 0): + continue; + + # If the end of the function is outside the interval, let's pretend + # that it is within the interval, otherwise we won't see any data about + # it when we hover. This won't have the effect of showing wrong + # data to the user. + # + mask = bucketDF.end >= upperBound; + bucketDF.loc[mask, 'end'] = upperBound-1; + + # Same adjustment as above if the start of the operation falls outside + # the interval's lower bound. + # + mask = bucketDF.start < lowerBound; + bucketDF.loc[mask, 'start'] = lowerBound; + + largestStackDepth = bucketDF['stackdepthNext'].max(); + figureTitle = fname; + + figure, legendDict = generateBucketChartForFile(figureTitle, bucketDF, + largestStackDepth, + lowerBound, upperBound); + aggregateLegendDict.update(legendDict); + figuresForAllFiles.append(figure); + + # Create the legend for this file and insert it after the navigator figure + if (len(aggregateLegendDict) > 0): + legendFigure = createLegendFigure(aggregateLegendDict); + figuresForAllFiles.insert(1, legendFigure); + + save(column(figuresForAllFiles), filename = fileName, + title=intervalTitle, resources=CDN); + + return fileName; + +# Generate a plot that shows a view of the entire timeline in a form of +# intervals. By clicking on an interval we can navigate to that interval. +# +def generateNavigatorFigure(dataframe, i, title): + + global pixelsForTitle; + global pixelsPerHeightUnit; + global plotWidth; + + # Generate the colors, such that the current interval is shown in a + # different color than the rest. + # + numIntervals = dataframe['intervalnumber'].size; + color = ["white" for x in range(numIntervals)]; + color[i] = "salmon"; + dataframe['color'] = color; + + cds = ColumnDataSource(dataframe); + + title = title + " CLICK TO NAVIGATE"; + + hover = HoverTool(tooltips = [ + ("interval #", "@intervalnumber"), + ("interval start", "@intervalbegin{0,0}"), + ("interval end", "@intervalend{0,0}")]); + + TOOLS = [hover, "tap"]; + + p = figure(title = title, plot_width = plotWidth, + x_range = (0, numIntervals), + plot_height = 2 * pixelsPerHeightUnit + pixelsForTitle, + x_axis_label = "", + y_axis_label = "", tools = TOOLS, + toolbar_location="above"); + + # No minor ticks or labels on the y-axis + p.yaxis.major_tick_line_color = None; + p.yaxis.minor_tick_line_color = None; + p.yaxis.major_label_text_font_size = '0pt'; + p.yaxis.ticker = FixedTicker(ticks = range(0, 1)); + p.ygrid.ticker = FixedTicker(ticks = range(0, 1)); + + p.xaxis.formatter = NumeralTickFormatter(format="0,"); + + p.title.align = "center"; + p.title.text_font_style = "normal"; + + p.quad(left = 'intervalnumber', right = 'intervalnumbernext', + bottom = 0, top = 2, color = 'color', source = cds, + nonselection_fill_color='color', + nonselection_fill_alpha = 1.0, + line_color = "aliceblue", + selection_fill_color = "white", + selection_line_color="lightgrey" + ); + + url = "@bucketfiles"; + taptool = p.select(type=TapTool); + taptool.callback = OpenURL(url=url); + + return p; + + +# Create a dataframe describing all time intervals, which will later be used +# to generate a plot allowing us to navigate along the execution by clicking +# on different intervals. +# +def createIntervalNavigatorDF(numBuckets, timeUnitsPerBucket): + + global bucketDir; + + bucketFiles = []; + bucketID = []; + intervalBegin = []; + intervalEnd = []; + + for i in range(numBuckets): + + lBound = i * timeUnitsPerBucket; + uBound = (i+1) * timeUnitsPerBucket; + fileName = "bucket-" + str(i) + ".html"; + + bucketID.append(i); + intervalBegin.append(lBound); + intervalEnd.append(uBound); + bucketFiles.append(fileName); + + data = {}; + data['bucketfiles'] = bucketFiles; + data['intervalbegin'] = intervalBegin; + data['intervalend'] = intervalEnd; + data['intervalnumber'] = bucketID; + + dataframe = pd.DataFrame(data=data); + dataframe['intervalnumbernext'] = dataframe['intervalnumber'] + 1; + return dataframe; + +# Generate plots of time series slices across all files for each bucket +# in the outlier histogram. Save each cross-file slice to an HTML file. +# +def generateTSSlicesForBuckets(): + + global firstTimeStamp; + global lastTimeStamp; + global plotWidth; + global pixelsPerWidthUnit; + + bucketFilenames = []; + + numBuckets = plotWidth / pixelsPerWidthUnit; + timeUnitsPerBucket = (lastTimeStamp - firstTimeStamp) / numBuckets; + + navigatorDF = createIntervalNavigatorDF(numBuckets, timeUnitsPerBucket); + + for i in range(numBuckets): + lowerBound = i * timeUnitsPerBucket; + upperBound = (i+1) * timeUnitsPerBucket; + + fileName = generateCrossFilePlotsForBucket(i, lowerBound, upperBound, + navigatorDF); + + percentComplete = float(i) / float(numBuckets) * 100; + print(color.BLUE + color.BOLD + " Generating timeline charts... "), + sys.stdout.write("%d%% complete \r" % (percentComplete) ); + sys.stdout.flush(); + bucketFilenames.append(fileName); + + print(color.END); + + return bucketFilenames; + +def processFile(fname): + + global perFileDataFrame; + global perFuncDF; + + rawData = pd.read_csv(fname, + header=None, delimiter=" ", + index_col=2, + names=["Event", "Function", "Timestamp"], + dtype={"Event": np.int32, "Timestamp": np.int64}, + thousands=","); + + print(color.BOLD + color.BLUE + + "Processing file " + str(fname) + color.END); + iDF = createCallstackSeries(rawData, "." + fname + ".log"); + + perFileDataFrame[fname] = iDF; + + for func in funcToColor.keys(): + + funcDF = iDF.loc[lambda iDF: iDF.function == func, :]; + funcDF = funcDF.drop(columns = ['function']); + + if (not perFuncDF.has_key(func)): + perFuncDF[func] = funcDF; + else: + perFuncDF[func] = pd.concat([perFuncDF[func], funcDF]); + + +# +# For each function, split the timeline into buckets. In each bucket +# show how many times this function took an unusually long time to +# execute. +# +# The parameter durationThreshold tells us when a function should be +# considered as unusually long. If this parameter is "-1" we count +# all functions whose duration exceeded the average by more than +# two standard deviations. +# +def createOutlierHistogramForFunction(func, funcDF, bucketFilenames): + + global firstTimeStamp; + global lastTimeStamp; + global plotWidth; + global pixelsPerWidthUnit; + global timeUnitString; + global STDEV_MULT; + + durationThreshold = 0; + durationThresholdDescr = ""; + + # + # funcDF is a list of functions along with their start and end + # interval and durations. We need to create a new dataframe where + # we separate the entire timeline into a fixed number of periods + # and for each period compute how many outlier durations were + # observed. Then we create a histogram from this data. + + # Subtract the smallest timestamp from all the interval data. + funcDF['start'] = funcDF['start'] - firstTimeStamp; + funcDF['end'] = funcDF['end'] - firstTimeStamp; + + funcDF = funcDF.sort_values(by=['start']); + + averageDuration = funcDF['durations'].mean(); + maxDuration = funcDF['durations'].max(); + + if (outlierThresholdDict.has_key(func)): + durationThreshold = outlierThresholdDict[func]; + durationThresholdDescr = outlierPrettyNames[func]; + elif (outlierThresholdDict.has_key("*")): + durationThreshold = outlierThresholdDict["*"]; + durationThresholdDescr = outlierPrettyNames["*"]; + else: + # Signal that we will use standard deviation + durationThreshold = -STDEV_MULT; + + if (durationThreshold < 0): # this is a stdev multiplier + mult = -durationThreshold; + stdDev = funcDF['durations'].std(); + durationThreshold = averageDuration + mult * stdDev; + durationThresholdDescr = '{0:,.0f}'.format(durationThreshold) \ + + " " + timeUnitString + " (" + str(mult) + \ + " standard deviations)"; + + numBuckets = plotWidth / pixelsPerWidthUnit; + timeUnitsPerBucket = (lastTimeStamp - firstTimeStamp) / numBuckets; + lowerBounds = []; + upperBounds = []; + bucketHeights = []; + maxOutliers = 0; + + for i in range(numBuckets): + lowerBound = i * timeUnitsPerBucket; + upperBound = (i+1) * timeUnitsPerBucket; + + bucketDF = funcDF.loc[(funcDF['start'] >= lowerBound) + & (funcDF['start'] < upperBound) + & (funcDF['durations'] >= durationThreshold)]; + + numOutliers = bucketDF.size; + if (numOutliers > maxOutliers): + maxOutliers = numOutliers; + + lowerBounds.append(lowerBound); + upperBounds.append(upperBound); + bucketHeights.append(numOutliers); + + if (maxOutliers == 0): + return None; + + dict = {}; + dict['lowerbound'] = lowerBounds; + dict['upperbound'] = upperBounds; + dict['height'] = bucketHeights; + dict['bottom'] = [0] * len(lowerBounds); + dict['bucketfiles'] = bucketFilenames; + + dataframe = pd.DataFrame(data=dict); + + return plotOutlierHistogram(dataframe, maxOutliers, func, + durationThresholdDescr, averageDuration, + maxDuration); + +# +# Return the string naming the time units used to measure time stamps, +# depending on how many time units there are in a second. +# +def getTimeUnitString(unitsPerSecond): + + if unitsPerSecond == 1000: + return "milliseconds"; + elif unitsPerSecond == 1000000: + return "microseconds"; + elif unitsPerSecond == 1000000000: + return "nanoseconds"; + else: + return "CPU cycles"; + +# +# The configuration file tells us which functions should be considered +# outliers. All comment lines must begin with '#'. +# +# The first non-comment line of the file must tell us how to interpret +# the measurement units in the trace file. It must have a single number +# telling us how many time units are contained in a second. This should +# be the same time units used in the trace file. For example, if the trace +# file contains timestamps measured in milliseconds, the number would be 1000, +# it the timestamp is in nanoseconds, the number would be 1000000000. +# If timestamps were measured in clock cycles, the number +# must tell us how many times the CPU clock ticks per second on the processor +# where the trace was gathered. +# +# The remaining lines must have the format: +# <func_name> <outlier_threshold> [units] +# +# For example, if you would like to flag as outliers all instances of +# __cursor_row_search that took longer than 200ms, you would specify this as: +# +# __cursor_row_search 200 ms +# +# You can use * as the wildcard for all function. No other wildcard options are +# supported at the moment. +# +# Acceptable units are: +# +# s -- for seconds +# ms -- for milliseconds +# us -- for microseconds +# ns -- for nanoseconds +# stdev -- for standard deviations. +# +# If no units are supplied, the same unit as the one used for the timestamp +# in the trace files is assumed. +# +# If there is a valid configuration file, but the function does not appear in +# it, we will not generate an outlier histogram for this function. Use the +# wildcard symbol to include all functions. +# +def parseConfigFile(fname): + + global outlierThresholdDict; + global outlierPrettyNames; + global timeUnitString; + + configFile = None; + firstNonCommentLine = True; + unitsPerSecond = -1; + unitsPerMillisecond = 0.0; + unitsPerMicrosecond = 0.0; + unitsPerNanosecond = 0.0; + + try: + configFile = open(fname, "r"); + except: + print(color.BOLD + color.RED + + "Could not open " + fname + " for reading." + color.END); + return False; + + for line in configFile: + + if (line[0] == "#"): + continue; + elif (firstNonCommentLine): + try: + unitsPerSecond = int(line); + unitsPerMillisecond = unitsPerSecond / 1000; + unitsPerMicrosecond = unitsPerSecond / 1000000; + unitsPerNanosecond = unitsPerSecond / 1000000000; + + timeUnitString = getTimeUnitString(unitsPerSecond); + + firstNonCommentLine = False; + except ValueError: + print(color.BOLD + color.RED + + "Could not parse the number of measurement units " + + "per second. This must be the first value in the " + + "config file." + color.END); + return False; + else: + func = ""; + number = 0; + threshold = 0.0; + units = ""; + + words = line.split(); + try: + func = words[0]; + number = int(words[1]); + units = words[2]; + except ValueError: + print(color.BOLD + color.RED + + "While parsing the config file, could not understand " + + "the following line: " + color.END); + print(line); + continue; + + # Now convert the number to the baseline units and record in the + # dictionary. + # + if (units == "s"): + threshold = unitsPerSecond * number; + elif (units == "ms"): + threshold = unitsPerMillisecond * number; + elif (units == "us"): + threshold = unitsPerMicrosecond * number; + elif (units == "ns"): + threshold = unitsPerNanosecond * number; + elif (units == "stdev"): + threshold = -units; + # We record it as negative, so that we know + # this is a standard deviation. We will compute + # the actual value once we know the average. + else: + print(color.BOLD + color.RED + + "While parsing the config file, could not understand " + + "the following line: " + color.END); + print(line); + continue; + + outlierThresholdDict[func] = threshold; + outlierPrettyNames[func] = str(number) + " " + units; + + # We were given an empty config file + if (firstNonCommentLine): + return False; + + print outlierThresholdDict; + return True; + + +def main(): + + global arrowLeftImg; + global arrowRightImg; + global bucketDir; + global perFuncDF; + + configSupplied = False; + figuresForAllFunctions = []; + + # Set up the argument parser + # + parser = argparse.ArgumentParser(description= + 'Visualize operation log'); + parser.add_argument('files', type=str, nargs='*', + help='log files to process'); + parser.add_argument('-c', '--config', dest='configFile', default=''); + args = parser.parse_args(); + + if (len(args.files) == 0): + parser.print_help(); + sys.exit(1); + + # Get names of standard CSS colors that we will use for the legend + initColorList(); + + # Read the configuration file, if supplied. + if (args.configFile != ''): + configSupplied = parseConfigFile(args.configFile); + + if (not configSupplied): + pluralSuffix = ""; + if (STDEV_MULT > 1): + pluralSuffix = "s"; + print(color.BLUE + color.BOLD + + "Will deem as outliers all function instances whose runtime " + + "was " + str(STDEV_MULT) + " standard deviation" + pluralSuffix + + " greater than the average runtime for that function." + + color.END); + + + # Create a directory for the files that display the data summarized + # in each bucket of the outlier histogram. We call these "bucket files". + # + if not os.path.exists(bucketDir): + os.makedirs(bucketDir); + + # Parallelize this later, so we are working on files in parallel. + for fname in args.files: + processFile(fname); + + # Normalize all intervals by subtracting the first timestamp. + normalizeIntervalData(); + + # Generate plots of time series slices across all files for each bucket + # in the outlier histogram. Save each cross-file slice to an HTML file. + # + fileNameList = generateTSSlicesForBuckets(); + + totalFuncs = len(perFuncDF.keys()); + i = 0; + # Generate a histogram of outlier durations + for func in sorted(perFuncDF.keys()): + funcDF = perFuncDF[func]; + figure = createOutlierHistogramForFunction(func, funcDF, fileNameList); + if (figure is not None): + figuresForAllFunctions.append(figure); + + i += 1; + percentComplete = float(i) / float(totalFuncs) * 100; + print(color.BLUE + color.BOLD + " Generating outlier histograms... "), + sys.stdout.write("%d%% complete \r" % (percentComplete) ); + sys.stdout.flush(); + + print(color.END); + reset_output(); + output_file(filename = "WT-outliers.html", title="Outlier histograms"); + show(column(figuresForAllFunctions)); + +if __name__ == '__main__': + main() diff --git a/src/third_party/wiredtiger/tools/optrack/wt_optrack_decode.py b/src/third_party/wiredtiger/tools/optrack/wt_optrack_decode.py new file mode 100755 index 00000000000..380cab50eba --- /dev/null +++ b/src/third_party/wiredtiger/tools/optrack/wt_optrack_decode.py @@ -0,0 +1,332 @@ +#!/usr/bin/env python +# +# Public Domain 2014-2018 MongoDB, Inc. +# Public Domain 2008-2014 WiredTiger, Inc. +# +# This is free and unencumbered software released into the public domain. +# +# Anyone is free to copy, modify, publish, use, compile, sell, or +# distribute this software, either in source code form or as a compiled +# binary, for any purpose, commercial or non-commercial, and by any +# means. +# +# In jurisdictions that recognize copyright laws, the author or authors +# of this software dedicate any and all copyright interest in the +# software to the public domain. We make this dedication for the benefit +# of the public at large and to the detriment of our heirs and +# successors. We intend this dedication to be an overt act of +# relinquishment in perpetuity of all present and future rights to this +# software under copyright law. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +# IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. + +import argparse +import colorsys +from multiprocessing import Process +import multiprocessing +import os +import os.path +import struct +import sys +import subprocess +import time +import traceback + +# +# This log version must be the same as that defined in ../src/include/optrack.h +# +currentLogVersion = 2; + +class color: + PURPLE = '\033[95m' + CYAN = '\033[96m' + DARKCYAN = '\033[36m' + BLUE = '\033[94m' + GREEN = '\033[92m' + YELLOW = '\033[93m' + RED = '\033[91m' + BOLD = '\033[1m' + UNDERLINE = '\033[4m' + END = '\033[0m' + +functionMap = {}; + +def buildTranslationMap(mapFileName): + + mapFile = None; + + if not os.path.exists(mapFileName): + return False; + + try: + mapFile = open(mapFileName, "r"); + except: + print(color.BOLD + color.RED); + print("Could not open " + mapFileName + " for reading"); + print(color.END); + return; + + # Read lines from the map file and build an in-memory map + # of translations. Each line has a function ID followed by space and + # followed by the function name. + # + lines = mapFile.readlines(); # a map file is usually small + + for line in lines: + + words = line.split(" "); + if (len(words) < 2): + continue; + + try: + funcID = int(words[0]); + except: + continue; + + funcName = words[1].strip(); + + functionMap[funcID] = funcName; + + return True; + +def funcIDtoName(funcID): + + if (functionMap.has_key(funcID)): + return functionMap[funcID]; + else: + return "NULL"; + +# +# The format of the record is written down in src/include/optrack.h +# file in the WiredTiger source tree. The current implementation assumes +# a record of three fields. The first field is the 8-byte timestamp. +# The second field is the 2-byte function ID. The third field is the +# 2-byte operation type: '0' for function entry, '1' for function exit. +# The record size would be padded to 16 bytes in the C implementation by +# the compiler, because we keep an array of records, and each new record +# has to be 8-byte aligned, since the first field has the size 8 bytes. +# So we explicitly pad the track record structure in the implementation +# to make it clear what the record size is. +# +def parseOneRecord(file): + + bytesRead = ""; + record = (); + RECORD_SIZE = 16; + + try: + bytesRead = file.read(RECORD_SIZE); + except: + return None; + + if (len(bytesRead) < RECORD_SIZE): + return None; + + record = struct.unpack('Qhhxxxx', bytesRead); + + return record; + +# +# HEADER_SIZE must be the same as the size of WT_OPTRACK_HEADER +# structure defined in ../src/include/optrack.h +# +def validateHeader(file): + + global currentLogVersion; + + bytesRead = ""; + HEADER_SIZE = 12; + + try: + bytesRead = file.read(HEADER_SIZE); + except: + return False, -1; + + if (len(bytesRead) < HEADER_SIZE): + return False, -1; + + version, threadType, tsc_nsec = struct.unpack('III', bytesRead); + + if (version == currentLogVersion): + return True, threadType, tsc_nsec; + else: + return False, -1, 1; + +def getStringFromThreadType(threadType): + + if (threadType == 0): + return "external"; + elif (threadType == 1): + return "internal"; + else: + return unknown; + + +def parseFile(fileName): + + done = False; + file = None; + threadType = 0; + threadTypeString = None; + tsc_nsec_ratio = 1.0; + outputFile = None; + outputFileName = ""; + totalRecords = 0; + validVersion = False; + + print(color.BOLD + "Processing file " + fileName + color.END); + + # Open the log file for reading + try: + file = open(fileName, "r"); + except: + print(color.BOLD + color.RED + + "Could not open " + fileName + " for reading" + color.END); + return; + + # Read and validate log header + validVersion, threadType, tsc_nsec_ratio = validateHeader(file); + if (not validVersion): + return; + + # Find out if this log file was generated by an internal or an + # external thread. This will be reflected in the output file name. + # + threadTypeString = getStringFromThreadType(threadType); + + # This ratio tells us how many clock ticks there are in a nanosecond + # on the processor on which this trace file was generated. When the WT + # library logs this ratio, it multiplies it by 1000. So we have to divide + # it back to get an accurate ratio. + tsc_nsec_ratio = float(tsc_nsec_ratio) / 1000.0; + + print("TSC_NSEC ratio parsed: " + '{0:,.4f}'.format(tsc_nsec_ratio)); + + # Open the text file for writing + try: + outputFileName = fileName + "-" + threadTypeString + ".txt"; + outputFile = open(outputFileName, "w"); + except: + print(color.BOLD + color.RED + + "Could not open file " + outputfileName + ".txt for writing." + + color.END); + return; + + print(color.BOLD + color.PURPLE + + "Writing to output file " + outputFileName + "." + color.END); + + while (not done): + record = parseOneRecord(file); + + if ((record is None) or len(record) < 3): + done = True; + else: + try: + time = float(record[0]) / tsc_nsec_ratio; + funcName = funcIDtoName(record[1]); + opType = record[2]; + + outputFile.write(str(opType) + " " + funcName + " " + + str(int(time)) + + "\n"); + totalRecords += 1; + except: + exc_type, exc_value, exc_traceback = sys.exc_info() + traceback.print_exception(exc_type, exc_value, exc_traceback); + print(color.BOLD + color.RED); + print("Could not write record " + str(record) + + " to file " + fileName + ".txt."); + print(color.END); + done = True; + + print("Wrote " + str(totalRecords) + " records to " + outputFileName + "."); + file.close(); + outputFile.close(); + +def waitOnOneProcess(runningProcesses): + + success = False; + for fname, p in runningProcesses.items(): + if (not p.is_alive()): + del runningProcesses[fname]; + success = True; + + # If we have not found a terminated process, sleep for a while + if (not success): + time.sleep(5); + +def main(): + + runnableProcesses = {}; + returnValues = {}; + spawnedProcesses = {}; + successfullyProcessedFiles = []; + targetParallelism = multiprocessing.cpu_count(); + terminatedProcesses = {}; + + parser = argparse.ArgumentParser(description= + 'Convert WiredTiger operation \ + tracking logs from binary to \ + text format.'); + + parser.add_argument('files', type=str, nargs='*', + help='optrack log files to process'); + + parser.add_argument('-j', dest='jobParallelism', type=int, + default='0'); + + parser.add_argument('-m', '--mapfile', dest='mapFileName', type=str, + default='optrack-map'); + + args = parser.parse_args(); + + print("Running with the following parameters:"); + for key, value in vars(args).items(): + print ("\t" + key + ": " + str(value)); + + # Parse the map of function ID to name translations. + if (buildTranslationMap(args.mapFileName) is False): + print("Failed to locate or parse the map file " + + args.mapFileName); + print("Cannot proceed."); + return; + + # Determine the target job parallelism + if (args.jobParallelism > 0): + targetParallelism = args.jobParallelism; + if (targetParallelism == 0): + targetParallelism = len(args.files); + print(color.BLUE + color.BOLD + + "Will process " + str(targetParallelism) + " files in parallel." + + color.END); + + # Prepare the processes that will parse files, one per file + if (len(args.files) > 0): + for fname in args.files: + p = Process(target=parseFile, args=(fname,)); + runnableProcesses[fname] = p; + + # Spawn these processes, not exceeding the desired parallelism + while (len(runnableProcesses) > 0): + while (len(spawnedProcesses) < targetParallelism + and len(runnableProcesses) > 0): + + fname, p = runnableProcesses.popitem(); + p.start(); + spawnedProcesses[fname] = p; + + # Find at least one terminated process + waitOnOneProcess(spawnedProcesses); + + # Wait for all processes to terminate + while (len(spawnedProcesses) > 0): + waitOnOneProcess(spawnedProcesses); + +if __name__ == '__main__': + main() diff --git a/src/third_party/wiredtiger/tools/wt_ckpt_decode.py b/src/third_party/wiredtiger/tools/wt_ckpt_decode.py index 0d45a652063..082387f5622 100644 --- a/src/third_party/wiredtiger/tools/wt_ckpt_decode.py +++ b/src/third_party/wiredtiger/tools/wt_ckpt_decode.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Public Domain 2014-2017 MongoDB, Inc. +# Public Domain 2014-2018 MongoDB, Inc. # Public Domain 2008-2014 WiredTiger, Inc. # # This is free and unencumbered software released into the public domain. |