Import wiredtiger: 911c940adab547d36ac305fc627a79e637fa3c40 from branch mongodb-3.2r3.2.10-rc0

ref: dddca65..911c940ada
for: 3.2.10

SERVER-24971  Excessive memory held by sessions when application threads do evictions
SERVER-25843  Coverity analysis defect 99856: Redundant test
SERVER-25845  Coverity analysis defect 99859: Explicit null dereferenced
SERVER-25846  Coverity analysis defect 99861: Dereference after null check
WT-1162       Add latency to Jenkins wtperf tests and plots
WT-2026       Maximum pages size at eviction too large
WT-2221       Document which statistics are available via a "fast" configuration vs. an "all" configuration
WT-2233       Investigate changing when the eviction server switches to aggressive mode.
WT-2239       Make sure LSM cursors read up to date dsk_gen, it was racing with compact
WT-2323       Allocate a transaction id at the beginning of join cursor iteration
WT-2353       Failure to create async threads as part of a wiredtiger_open call will cause a hang
WT-2380       Make scripts fail if code doesn't match style
WT-2486       Update make check so that it runs faster
WT-2555       make format run on Windows
WT-2578       remove write barriers from the TAILQ_INSERT_XXX macros
WT-2631       nullptr is passed for parameters marked with attribute non-null
WT-2638       ftruncate may not be supported
WT-2645       wt dump: push the complexity of collecting metadata into a dump cursor
WT-2648       cache-line alignment for new ports
WT-2665       Limit allocator fragmentation in WiredTiger
WT-2678       The metadata should not imply that an empty value is true
WT-2688       configure --enable-python doesn't check for availability of swig
WT-2693       Check open_cursor error paths for consistent handling
WT-2695       Integrate s390x accelerated crc32c support
WT-2708       split child-update race with reconciliation/eviction
WT-2711       Change statistics log configuration options
WT-2719       add fuzz testing for WiredTiger options and reconfiguration.
WT-2728       Don't re-read log file headers during log_flush
WT-2729       Focus eviction walks in largest trees
WT-2730       cursor next/prev can return the wrong key/value pair when crossing a page boundary
WT-2731       Raw compression can create pages that are larger than expected
WT-2732       Coverity analysis defect 99665: Redundant test
WT-2734       Improve documentation of eviction behavior
WT-2737       Scrub dirty pages rather than evicting them
WT-2738       Remove the ability to change the default checkpoint name
WT-2739       pluggable file systems documentation cleanups
WT-2743       Thread count statistics always report 0
WT-2744       partial line even with line buffering set
WT-2746       track checkpoint I/O separately from eviction I/O
WT-2751       column-store statistics incorrectly calculates the number of entries
WT-2752       Fixes to zipfian wtperf workload config
WT-2755       flexelint configuration treats size_t as 4B type
WT-2756       Upgrade the autoconf archive package to check for swig 3.0
WT-2757       Column tables behave differently when column names are provided
WT-2759       Releasing the hot-backup lock doesn't require the schema lock.
WT-2760       Fix a bug in backup related to directory sync. Change the filesystem API to make durable the default
WT-2762       wtstats tool fails if checkpoint runs
WT-2763       Unit test test_intpack failing on OSX
WT-2764       Optimize checkpoints to reduce throughput disruption
WT-2765       wt dump: indices need to be shown in the dump output
WT-2766       Don't count eviction of lookaside file pages for the purpose of checking stuck cache
WT-2767       test suite needs way to run an individual scenario
WT-2769       Update documentation to reflect correct limits of memory_page_max
WT-2770       Add statistics tracking schema operations
WT-2772       Investigate log performance testing weirdness
WT-2773       search_near in indexes does not find exact matches
WT-2774       minor cleanups/improvements
WT-2778       Python test suite: make scenario initialization consistent
WT-2779       Raw compression created unexpectedly large pages on disk
WT-2781       Enhance bulk cursor option with an option to return immediately on contention
WT-2782       Missing a fs_directory_list_free in ex_file_system.c
WT-2783       wtperf multi-btree.wtperf dumps core on Mac
WT-2785       Scrub dirty pages rather than evicting them: single-page reconciliation
WT-2787       Include src/include/wiredtiger_ext.h is problematic
WT-2788       Java: freed memory overwrite during handle close can cause JNI crash
WT-2791       Enhance OS X Evergreen unit test
WT-2793       wtperf config improvements
WT-2795       Update documentation around read-only configuration
WT-2796       Memory leak in reconciliation uncovered by stress testing
WT-2798       Crash vulnerability with nojournal after create during checkpoint
WT-2800       Illegal file format in test/format on PPC
WT-2801       Crash vulnerability from eviction of metadata during checkpoint
WT-2802       Transaction commit causes heap-use-after free
WT-2803       Add verbose functionality to WT Evergreen tests
WT-2804       Don't read values in a tree without a snapshot
WT-2805       Infinite recursion if error streams fail
WT-2806       wtperf allocation size off-by-one
WT-2807       Switch Jenkins performance tests to tcmalloc
WT-2811       Reconciliation asserts that transaction time has gone backwards
WT-2812       Error when reconfiguring cache targets
WT-2813       small cache usage stuck even with large cache
WT-2814       Enhance wtperf to support single-op truncate mode
WT-2816       Improve WiredTiger eviction performance
WT-2817       Investigate performance regression in develop, add workload to wtperf/runners
WT-2818       The page visibility check when queuing pages for eviction is overly restrictive
WT-2820       add gcc warn_unused_result attribute
WT-2822       panic mutex and other functions that cannot fail
WT-2823       support file handles without a truncate method
WT-2824       wtperf displays connection and table create configurations twice
WT-2826       clang38 false positive on uninitialized variable.
WT-2827       checkpoint log_size configuration improvements
WT-2828       Make long wtperf tests reflect mongoDB usage
WT-2829       Switch automated testing to use enable-strict configure option
WT-2832       Python test uses hard-coded temporary directory
WT-2834       Join cursor: discrepancy with bloom filters
WT-2835       WT_CONNECTION.leak-memory can skip memory map and cache cleanup
WT-2838       Don't free session handles on close if leak memory is configured
WT-2839       lint: Ignoring return value of function
WT-2840       clang analysis: garbage values
WT-2841       Jenkins Valgrind runner is reporting errors in test wt2719_reconfig
WT-2842       split wtperf's configuration into per-database and per-run parts
WT-2843       Fix a bug in recovery if there is no filesystem truncate support
WT-2846       Several bugs related to reconfiguring eviction server at runtime
WT-2847       Merge fair locks into read/write locks.
WT-2850       clang 4.1 attribute warnings when building
WT-2853       Multi threaded reader writer example shows temporary slowdown or lockup
WT-2857       POSIX ftruncate calls should be #ifdef'd HAVE_FTRUNCATE
WT-2862       Fix lint error in test case for forced eviction with multiple cursors
WT-2863       Support UTF-8 paths on Windows
WT-2865       eviction thread error failure
WT-2866       Eviction server algorithm tuning
WT-2867       Review and fix barrier usage in __lsm_tree_close
WT-2868       Add sample_interval to checkpoint-stress wtperf config
WT-2869       Performance regression on secondaries
WT-2870       Rename wtperf checkpoint schema jobs
WT-2871       __wt_verbose has the wrong GCC format attributes
WT-2872       Recent stuck cache test/stress failures.
WT-2873       Refactor CRC32 code
WT-2875       Test test_wt2853_perf can run too long under valgrind
WT-2876       Extend wtperf to support a log like table
WT-2878       Verbose changes affected performance
WT-2881       Add -Wpedantic to clang compiler warning flags
WT-2883       wiredtiger_open with verbose=handleops recursive loop
WT-2885       __wt_checkpoint_signal lint
WT-2886       Decide how in-memory configuration and eviction_dirty_target interact
WT-2888       Switch functions to return void where possible
WT-2892       hot backup can race with block truncate
WT-2896       Coverity #1362535: resource leak
WT-2897       Checkpoints can become corrupted on failure
WT-2901       Add option to disable checkpoint dirty stepdown phase
WT-2903       Reduce the impact of checkpoint scrubbing on applications

1 files changed, 21 insertions, 41 deletions

diff --git a/src/third_party/wiredtiger/src/btree/bt_sync.c b/src/third_party/wiredtiger/src/btree/bt_sync.c
index da6c53aa316..b41179a565d 100644
--- a/src/third_party/wiredtiger/src/btree/bt_sync.c
+++ b/src/third_party/wiredtiger/src/btree/bt_sync.c
@@ -84,7 +84,8 @@ __sync_file(WT_SESSION_IMPL *session, WT_CACHE_OP syncop)
 					WT_ERR(__wt_txn_get_snapshot(session));
 				leaf_bytes += page->memory_footprint;
 				++leaf_pages;
-				WT_ERR(__wt_reconcile(session, walk, NULL, 0));
+				WT_ERR(__wt_reconcile(
+				    session, walk, NULL, WT_CHECKPOINTING));
 			}
 		}
 		break;
@@ -92,7 +93,7 @@ __sync_file(WT_SESSION_IMPL *session, WT_CACHE_OP syncop)
 		/*
 		 * If we are flushing a file at read-committed isolation, which
 		 * is of particular interest for flushing the metadata to make
-		 * schema-changing operation durable, get a transactional
+		 * a schema-changing operation durable, get a transactional
 		 * snapshot now.
 		 *
 		 * All changes committed up to this point should be included.
@@ -126,7 +127,17 @@ __sync_file(WT_SESSION_IMPL *session, WT_CACHE_OP syncop)
 		 */
 		WT_PUBLISH(btree->checkpointing, WT_CKPT_PREPARE);
 
-		WT_ERR(__wt_evict_file_exclusive_on(session));
+		/*
+		 * Sync for checkpoint allows splits to happen while the queue
+		 * is being drained, but not reconciliation. We need to do this,
+		 * since draining the queue can take long enough for hot pages
+		 * to grow significantly larger than the configured maximum
+		 * size.
+		 */
+		F_SET(btree, WT_BTREE_NO_RECONCILE);
+		ret = __wt_evict_file_exclusive_on(session);
+		F_CLR(btree, WT_BTREE_NO_RECONCILE);
+		WT_ERR(ret);
 		__wt_evict_file_exclusive_off(session);
 
 		WT_PUBLISH(btree->checkpointing, WT_CKPT_RUNNING);
@@ -183,7 +194,8 @@ __sync_file(WT_SESSION_IMPL *session, WT_CACHE_OP syncop)
 				leaf_bytes += page->memory_footprint;
 				++leaf_pages;
 			}
-			WT_ERR(__wt_reconcile(session, walk, NULL, 0));
+			WT_ERR(__wt_reconcile(
+			    session, walk, NULL, WT_CHECKPOINTING));
 		}
 		break;
 	case WT_SYNC_CLOSE:
@@ -194,7 +206,7 @@ __sync_file(WT_SESSION_IMPL *session, WT_CACHE_OP syncop)
 
 	if (WT_VERBOSE_ISSET(session, WT_VERB_CHECKPOINT)) {
 		WT_ERR(__wt_epoch(session, &end));
-		WT_ERR(__wt_verbose(session, WT_VERB_CHECKPOINT,
+		__wt_verbose(session, WT_VERB_CHECKPOINT,
 		    "__sync_file WT_SYNC_%s wrote:\n\t %" PRIu64
 		    " bytes, %" PRIu64 " pages of leaves\n\t %" PRIu64
 		    " bytes, %" PRIu64 " pages of internal\n\t"
@@ -202,7 +214,7 @@ __sync_file(WT_SESSION_IMPL *session, WT_CACHE_OP syncop)
 		    syncop == WT_SYNC_WRITE_LEAVES ?
 		    "WRITE_LEAVES" : "CHECKPOINT",
 		    leaf_bytes, leaf_pages, internal_bytes, internal_pages,
-		    WT_TIMEDIFF_MS(end, start)));
+		    WT_TIMEDIFF_MS(end, start));
 	}
 
 err:	/* On error, clear any left-over tree walk. */
@@ -217,41 +229,9 @@ err:	/* On error, clear any left-over tree walk. */
 	    saved_snap_min == WT_TXN_NONE)
 		__wt_txn_release_snapshot(session);
 
-	if (btree->checkpointing != WT_CKPT_OFF) {
-		/*
-		 * Update the checkpoint generation for this handle so visible
-		 * updates newer than the checkpoint can be evicted.
-		 *
-		 * This has to be published before eviction is enabled again,
-		 * so that eviction knows that the checkpoint has completed.
-		 */
-		WT_PUBLISH(btree->checkpoint_gen,
-		    conn->txn_global.checkpoint_gen);
-		WT_STAT_FAST_DATA_SET(session,
-		    btree_checkpoint_generation, btree->checkpoint_gen);
-
-		/*
-		 * Clear the checkpoint flag and push the change; not required,
-		 * but publishing the change means stalled eviction gets moving
-		 * as soon as possible.
-		 */
-		btree->checkpointing = WT_CKPT_OFF;
-		WT_FULL_BARRIER();
-
-		/*
-		 * If this tree was being skipped by the eviction server during
-		 * the checkpoint, clear the wait.
-		 */
-		btree->evict_walk_period = 0;
-
-		/*
-		 * Wake the eviction server, in case application threads have
-		 * stalled while the eviction server decided it couldn't make
-		 * progress.  Without this, application threads will be stalled
-		 * until the eviction server next wakes.
-		 */
-		WT_TRET(__wt_evict_server_wake(session));
-	}
+	/* Clear the checkpoint flag and push the change. */
+	if (btree->checkpointing != WT_CKPT_OFF)
+		WT_PUBLISH(btree->checkpointing, WT_CKPT_OFF);
 
 	__wt_spin_unlock(session, &btree->flush_lock);