Import wiredtiger: 2e9744d11a65c63ba7445060dc78371250f04051 from branch mongodb-3.6

ref: 6173a98979..2e9744d11a
for: 3.5.11

WT-2309       Add yields and/or sleeps in #DIAGNOSTIC mode
WT-3047       Add mode aimed at uncovering race conditions in split code
WT-3308       Add statistics tracking around yield loops
WT-3316       Add new engineering section to reference guide documentation
WT-3338       Optimize cursor modify
WT-3380       Special case 8-byte timestamps
WT-3387       Add support for a stable timestamp
WT-3389       Restructure split code to hold a split generation for the entire operation.
WT-3406       Reconciliation is choosing reserved records for writing.
WT-3410       Add developer documentation for table rename
WT-3412       Add backoff logic to the btree delete and walk yield loops
WT-3418       block manager object race
WT-3422       WiredTiger upgrading documents out of date
WT-3432       workgen needs braces around an "if" body
WT-3433       session->alter method should not be supported in read-only mode
WT-3439       lint/cleanup
WT-3440       Add a log record when starting a checkpoint
WT-3442       Coverity 1378213: false positive on diagnostic assignment.
WT-3446       Temporarily disable timestamp testing in test/checkpoint
WT-3447       test_stat_log02 can assert before table stats are printed
WT-3461       Avoid long sleeps when the system clock is adjusted
WT-3463       Add recovery of backup to test_timestamp03.py
WT-3466       Track the first commit timestamp for each transaction
WT-3467       Minor lint/cleanup

10 files changed, 251 insertions, 5 deletions

diff --git a/src/third_party/wiredtiger/src/docs/Doxyfile b/src/third_party/wiredtiger/src/docs/Doxyfile
index e7382e2bc5e..8292df18e47 100644
--- a/src/third_party/wiredtiger/src/docs/Doxyfile
+++ b/src/third_party/wiredtiger/src/docs/Doxyfile
@@ -232,7 +232,9 @@ ALIASES                = "notyet{1}=Note: <b>"\1"</b> not yet supported in Wired
                          "configstart{2}=@param config\n  Configuration string, see @ref config_strings. Permitted values:\n  <table>@hrow{Name,Effect,Values}" \
                          "config{3}=  @row{<tt>\1</tt>,\2,\3}" \
                          "configend=  </table>" \
-                         "configempty{2}=@param config\n  Configuration string, see @ref config_strings.  No values currently permitted."
+                         "configempty{2}=@param config\n  Configuration string, see @ref config_strings.  No values currently permitted." \
+                         "plantuml_start{1}=\image html \1\n\image latex \1\n<!-- PlantUML template begins" \
+                         "plantuml_end=PlantUML template end -->"
 
 # This tag can be used to specify a number of word-keyword mappings (TCL only). 
 # A mapping has the form "name=value". For example adding 
diff --git a/src/third_party/wiredtiger/src/docs/devdoc-index.dox b/src/third_party/wiredtiger/src/docs/devdoc-index.dox
new file mode 100644
index 00000000000..ba809d7af43
--- /dev/null
+++ b/src/third_party/wiredtiger/src/docs/devdoc-index.dox
@@ -0,0 +1,12 @@
+/*! @page devdoc-index Developer Documentation
+
+Most applications begin to make use of WiredTiger by creating a table (or other
+data object) to store their data in. Create is one of several schema operations
+available in WiredTiger.
+
+For more information on how schema operations are implemented in WiredTiger,
+see:
+
+- @subpage devdoc-schema
+
+*/
diff --git a/src/third_party/wiredtiger/src/docs/devdoc-schema.dox b/src/third_party/wiredtiger/src/docs/devdoc-schema.dox
new file mode 100644
index 00000000000..509fbf16d67
--- /dev/null
+++ b/src/third_party/wiredtiger/src/docs/devdoc-schema.dox
@@ -0,0 +1,208 @@
+/*! @page devdoc-schema Schema operations
+
+A schema defines the format of the application data in WiredTiger. WiredTiger
+supports various types of schemas (See @ref schema for more
+information), operated upon through a WT_SESSION reference. This section details
+the internals of these various schema operations.
+
+Schema operations cause an update to the metadata and are performed under a
+schema lock to avoid concurrent operations on the same object. The following
+sequence of steps define a generic schema operation:
+
+@plantuml_start{schema_generic.png}
+@startuml{schema_generic.png}
+:A schema operation;
+partition with-schema-lock {
+    :perform operation on underlying data-object;
+    :update metadata-file;
+    :checkpoint and sync metadata;
+}
+stop
+@enduml
+@plantuml_end
+
+@section schema_create Schema Create
+
+The create schema operation is responsible for creating the underlying data
+object on the filesystem with the right parameters and then creating an entry
+for this new object into the metadata. The sequence of operations involved in a
+create for various schema types are as follows:
+
+@plantuml_start{schema_create.png}
+@startuml{schema_create.png}
+:WT_SESSION->create(.,name,.)
+(__session_create());
+
+partition session-API-call {
+    :API session init with NULL dhandle;
+    :exit if PANIC flag set;
+    :exit if invalid configuration;
+}
+
+:validate "name" and if passed "type" config parameter;
+note right
+    "name" parameter is called as "uri" internally.
+    "type" is usually not passed and generally
+    implied from the uri.
+end note
+
+partition with-schema-lock {
+    partition with-table-lock {
+        :turn on meta tracking;
+        :check uri}
+
+        split
+            :uri matches "file:"
+            ("file" is the underlying
+             type for all the objects);
+        split again
+            :uri matches "colgroup:";
+            :__create_colgroup();
+        split again
+            :uri matches "table:";
+            :__create_table();
+        split again
+            :uri matches "lsm:";
+            :__wt_lsm_tree_create();
+        split again
+            :uri matches "index:";
+            :__create_index();
+        split again
+            :matches a named data-source;
+            :__create_data_source();
+        end split
+
+        partition __create_file() {
+            :exit if file exists;
+            :validate allocation size;
+            :block manager creates the file:
+             1.create file using __wt_open()
+             2.write an initial descriptor to file
+             3.fsync and close the file handle;
+            if (metadata-file?) then (yes)
+            else (no)
+                :update metadata with file
+                 configuration and version;
+            endif
+            :check if file setup correctly by
+             getting btree handle with
+             WT_DHANDLE_EXCLUSIVE set;
+            if (metatracking on?) then (yes)
+                :track locked handle*;
+            else (no)
+                :release btree -
+                 sync and close;
+            endif
+        }
+
+        partition turn-off-meta-tracking {
+            if (errors?) then (yes)
+                :unroll operations;
+            else (no)
+                if (logging?) then (yes)
+                    :sync log;
+                else (no)
+                endif
+                :checkpoint and sync;
+            endif
+            :apply post-commit ops:
+             release tracked (handle) btree* -
+             sync and close;
+            note right
+                if meta tracking is on, this btree
+                was being tracked as locked. As part
+                of tuning off meta tracking, we sync
+                and close this btree
+            end note
+        }
+    }
+}
+
+:API-end;
+
+stop
+@enduml
+@plantuml_end
+
+@section schema_rename Schema Rename
+
+The rename schema operation is responsible for renaming the underlying data
+object on the filesystem and updating the metadata accordingly. The sequence of
+operations involved in a rename for various schema types are as follows:
+
+@plantuml_start{schema_rename.png}
+@startuml{schema_rename.png}
+:WT_SESSION->rename(old-uri, new-uri, .)
+(__session_rename());
+:session-API-call;
+
+partition with-checkpoint-lock {
+    partition with-schema-lock {
+        partition with-table-write-lock {
+            :validate new uri-type to match the old type;
+            :turn on meta tracking;
+            :check uri}
+
+            split
+                :uri matches "file:"
+                ("file" is the underlying
+                 type for all the objects);
+            split again
+                :uri matches "lsm:";
+                :__wt_lsm_tree_rename();
+            split again
+                :matches a named data-source;
+                :WT_DATA_SOURCE::rename();
+            split again
+                :uri matches "table:";
+                partition __rename_table() {
+                    :rename colgroups and indices represented by the table:
+                     1. extract names from the uri
+                     2. create new uri with existing types and configuration
+                     3. recursive call the rename operation on individual
+                        colgroup and index with the old and the new uri
+                     4. remove old entry for colgroups and indices from
+                        the metadata table and add the new ones;
+                    :close and remove table handle from the session;
+                    :remove old table entry from the metadata table
+                     and add a new one;
+                }
+            end split
+
+            partition __rename_file() {
+                :fail if backup cursor open and schema operations will conflict;
+                :close btree handles in the file;
+                :fail if file with the old name doesn't exist or with the new
+                 name exists;
+                :remove old file entries and add new in the metadata;
+                :rename the underlying file;
+                if (meta-tracking?) then (yes)
+                    :track filesystem op;
+                else (no)
+                endif
+            }
+
+            :bump schema generation number to ignore stale data;
+
+            partition turn-off-meta-tracking {
+                if (errors?) then (yes)
+                    :unroll operations;
+                else (no)
+                    if (logging?) then (yes)
+                        :sync log;
+                    else (no)
+                    endif
+                    :checkpoint and sync;
+                endif
+            }
+        }
+    }
+}
+
+:API-end;
+
+stop
+@enduml
+@plantuml_end
+
+*/
diff --git a/src/third_party/wiredtiger/src/docs/images/plantuml_gen_img/schema_create.png b/src/third_party/wiredtiger/src/docs/images/plantuml_gen_img/schema_create.png
new file mode 100644
index 00000000000..d2699646e9c
--- /dev/null
+++ b/src/third_party/wiredtiger/src/docs/images/plantuml_gen_img/schema_create.png
diff --git a/src/third_party/wiredtiger/src/docs/images/plantuml_gen_img/schema_generic.png b/src/third_party/wiredtiger/src/docs/images/plantuml_gen_img/schema_generic.png
new file mode 100644
index 00000000000..790584b3f7b
--- /dev/null
+++ b/src/third_party/wiredtiger/src/docs/images/plantuml_gen_img/schema_generic.png
diff --git a/src/third_party/wiredtiger/src/docs/images/plantuml_gen_img/schema_rename.png b/src/third_party/wiredtiger/src/docs/images/plantuml_gen_img/schema_rename.png
new file mode 100644
index 00000000000..b0aa560946e
--- /dev/null
+++ b/src/third_party/wiredtiger/src/docs/images/plantuml_gen_img/schema_rename.png
diff --git a/src/third_party/wiredtiger/src/docs/introduction.dox b/src/third_party/wiredtiger/src/docs/introduction.dox
index 52936a04d3c..3a6886869b0 100644
--- a/src/third_party/wiredtiger/src/docs/introduction.dox
+++ b/src/third_party/wiredtiger/src/docs/introduction.dox
@@ -59,4 +59,8 @@ To browse the WiredTiger source code repository or contact us, see:
 
 - @subpage community
 
+For more information on WiredTiger internals, see:
+
+- @subpage devdoc-index
+
 */
diff --git a/src/third_party/wiredtiger/src/docs/spell.ok b/src/third_party/wiredtiger/src/docs/spell.ok
index 5d629f4c49f..112e5e815fe 100644
--- a/src/third_party/wiredtiger/src/docs/spell.ok
+++ b/src/third_party/wiredtiger/src/docs/spell.ok
@@ -15,6 +15,7 @@ Coverity
 Coverity's
 DB's
 DBTs
+DHANDLE
 DONTNEED
 Datastore
 DbCursor
@@ -185,6 +186,8 @@ desc
 destructor
 destructors
 dev
+devdoc
+dhandle
 disjunction
 disjunctions
 distclean
@@ -208,6 +211,7 @@ endcond
 endian
 endif
 endinternal
+enduml
 english
 env
 eof
@@ -317,6 +321,7 @@ memalloc
 memfree
 memp
 metadata
+metatracking
 minkey
 mixin
 mixins
@@ -375,6 +380,7 @@ perf
 petabyte
 pget
 php
+plantuml
 png
 posix
 pre
@@ -440,6 +446,7 @@ sql
 src
 ssd
 startsync
+startuml
 statlog
 stderr
 str
diff --git a/src/third_party/wiredtiger/src/docs/transactions.dox b/src/third_party/wiredtiger/src/docs/transactions.dox
index 8a05de9b5f5..d9cc72dcf24 100644
--- a/src/third_party/wiredtiger/src/docs/transactions.dox
+++ b/src/third_party/wiredtiger/src/docs/transactions.dox
@@ -164,7 +164,7 @@ timestamp size is 8 bytes (i.e., 64 bits).  Setting a size of zero disables
 transaction timestamp functionality.
 
 Applications can assign explicit commit timestamps to transactions, then read
-"as of" a timestamp.  Timestamps and are communicated to WiredTiger using a
+"as of" a timestamp.  Timestamps are communicated to WiredTiger using a
 lower case hexadecimal encoding, so the encoded value can be twice as long as
 the raw timestamp value.
 
@@ -177,6 +177,13 @@ WiredTiger can discard history before the specified point.  It is critical
 that the oldest timestamp update frequently or the cache can become full of
 updates, reducing performance.
 
+Setting a stable timestamp in WT_CONNECTION::set_timestamp indicates a
+known stable location that is sufficient for durability.  During a checkpoint
+the state of a table will be saved only as of the stable timestamp.  Newer
+updates after that stable timestamp will not be included in the checkpoint.
+That can be overridden in the call to WT_SESSION::checkpoint.  It is expected
+that the stable timestamp is updated frequently.
+
 Commit timestamps cannot be set in the past of any read timestamp that has
 been used.  This is enforced by assertions in diagnostic builds, if
 applications violate this rule, data consistency can be violated.
diff --git a/src/third_party/wiredtiger/src/docs/upgrade.dox b/src/third_party/wiredtiger/src/docs/upgrade.dox
index 7952efa0343..9f8331c9b1f 100644
--- a/src/third_party/wiredtiger/src/docs/upgrade.dox
+++ b/src/third_party/wiredtiger/src/docs/upgrade.dox
@@ -60,11 +60,17 @@ the WT_SESSION::upgrade method.
 Applications wanting the ability to downgrade to previous releases of
 WiredTiger are constrained in some important ways:
 
-- The underlying data file formats must not have changed between the
+- The underlying table file formats must not have changed between the
 release currently being run and the release to which you are
-downgrading. In other words, data file format changes are not backward
-compatible, and an upgraded data file cannot be downgraded without being
+downgrading. In other words, table file format changes are not backward
+compatible, and an upgraded table file cannot be downgraded without being
 dumped and re-loaded.
+- If the underlying log file format changed and you have logging enabled
+in the database, in order to retain the ability to downgrade to earlier
+releases you must use \c compatibility option in the call to
+::wiredtiger_open or WT_CONNECTION::reconfigure.  When running at an
+earlier compatibility release setting, the log files generated will be
+compatible with earlier binaries.
 <br><br>
 - Applications concerned with downgrading should configure the
 ::wiredtiger_open \c config_base value to \c false, so WiredTiger does