diff options
Diffstat (limited to 'src/third_party/wiredtiger/src/docs')
99 files changed, 10921 insertions, 0 deletions
diff --git a/src/third_party/wiredtiger/src/docs/Doxyfile b/src/third_party/wiredtiger/src/docs/Doxyfile new file mode 100644 index 00000000000..4c1682de6eb --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/Doxyfile @@ -0,0 +1,1856 @@ +# Doxyfile 1.8.2 + +# This file describes the settings to be used by the documentation system +# doxygen (www.doxygen.org) for a project +# +# All text after a hash (#) is considered a comment and will be ignored +# The format is: +# TAG = value [value, ...] +# For lists items can also be appended using: +# TAG += value [value, ...] +# Values that contain spaces should be placed between quotes (" ") + +#--------------------------------------------------------------------------- +# Project related configuration options +#--------------------------------------------------------------------------- + +# This tag specifies the encoding used for all characters in the config file +# that follow. The default is UTF-8 which is also the encoding used for all +# text before the first occurrence of this tag. Doxygen uses libiconv (or the +# iconv built into libc) for the transcoding. See +# http://www.gnu.org/software/libiconv for the list of possible encodings. + +DOXYFILE_ENCODING = UTF-8 + +# The PROJECT_NAME tag is a single word (or sequence of words) that should +# identify the project. Note that if you do not use Doxywizard you need +# to put quotes around the project name if it contains spaces. + +PROJECT_NAME = WiredTiger + +# The PROJECT_NUMBER tag can be used to enter a project or revision number. +# This could be handy for archiving the generated documentation or +# if some version control system is used. + +PROJECT_NUMBER = "Version 1.0" + +# Using the PROJECT_BRIEF tag one can provide an optional one line description +# for a project that appears at the top of each page and should give viewer +# a quick idea about the purpose of the project. Keep the description short. + +PROJECT_BRIEF = + +# With the PROJECT_LOGO tag one can specify an logo or icon that is +# included in the documentation. The maximum height of the logo should not +# exceed 55 pixels and the maximum width should not exceed 200 pixels. +# Doxygen will copy the logo to the output directory. + +PROJECT_LOGO = images/LogoFinal-header.png + +# The OUTPUT_DIRECTORY tag is used to specify the (relative or absolute) +# base path where the generated documentation will be put. +# If a relative path is entered, it will be relative to the location +# where doxygen was started. If left blank the current directory will be used. + +OUTPUT_DIRECTORY = ../../docs + +# If the CREATE_SUBDIRS tag is set to YES, then doxygen will create +# 4096 sub-directories (in 2 levels) under the output directory of each output +# format and will distribute the generated files over these directories. +# Enabling this option can be useful when feeding doxygen a huge amount of +# source files, where putting all generated files in the same directory would +# otherwise cause performance problems for the file system. + +CREATE_SUBDIRS = NO + +# The OUTPUT_LANGUAGE tag is used to specify the language in which all +# documentation generated by doxygen is written. Doxygen will use this +# information to generate all constant output in the proper language. +# The default language is English, other supported languages are: +# Afrikaans, Arabic, Brazilian, Catalan, Chinese, Chinese-Traditional, +# Croatian, Czech, Danish, Dutch, Esperanto, Farsi, Finnish, French, German, +# Greek, Hungarian, Italian, Japanese, Japanese-en (Japanese with English +# messages), Korean, Korean-en, Lithuanian, Norwegian, Macedonian, Persian, +# Polish, Portuguese, Romanian, Russian, Serbian, Serbian-Cyrillic, Slovak, +# Slovene, Spanish, Swedish, Ukrainian, and Vietnamese. + +OUTPUT_LANGUAGE = English + +# If the BRIEF_MEMBER_DESC tag is set to YES (the default) Doxygen will +# include brief member descriptions after the members that are listed in +# the file and class documentation (similar to JavaDoc). +# Set to NO to disable this. + +BRIEF_MEMBER_DESC = YES + +# If the REPEAT_BRIEF tag is set to YES (the default) Doxygen will prepend +# the brief description of a member or function before the detailed description. +# Note: if both HIDE_UNDOC_MEMBERS and BRIEF_MEMBER_DESC are set to NO, the +# brief descriptions will be completely suppressed. + +REPEAT_BRIEF = YES + +# This tag implements a quasi-intelligent brief description abbreviator +# that is used to form the text in various listings. Each string +# in this list, if found as the leading text of the brief description, will be +# stripped from the text and the result after processing the whole list, is +# used as the annotated text. Otherwise, the brief description is used as-is. +# If left blank, the following values are used ("$name" is automatically +# replaced with the name of the entity): "The $name class" "The $name widget" +# "The $name file" "is" "provides" "specifies" "contains" +# "represents" "a" "an" "the" + +ABBREVIATE_BRIEF = "The $name class" \ + "The $name widget" \ + "The $name file" \ + is \ + provides \ + specifies \ + contains \ + represents \ + a \ + an \ + the + +# If the ALWAYS_DETAILED_SEC and REPEAT_BRIEF tags are both set to YES then +# Doxygen will generate a detailed section even if there is only a brief +# description. + +ALWAYS_DETAILED_SEC = NO + +# If the INLINE_INHERITED_MEMB tag is set to YES, doxygen will show all +# inherited members of a class in the documentation of that class as if those +# members were ordinary class members. Constructors, destructors and assignment +# operators of the base classes will not be shown. + +INLINE_INHERITED_MEMB = NO + +# If the FULL_PATH_NAMES tag is set to YES then Doxygen will prepend the full +# path before files name in the file list and in the header files. If set +# to NO the shortest path that makes the file name unique will be used. + +FULL_PATH_NAMES = NO + +# If the FULL_PATH_NAMES tag is set to YES then the STRIP_FROM_PATH tag +# can be used to strip a user-defined part of the path. Stripping is +# only done if one of the specified strings matches the left-hand part of +# the path. The tag can be used to show relative paths in the file list. +# If left blank the directory from which doxygen is run is used as the +# path to strip. Note that you specify absolute paths here, but also +# relative paths, which will be relative from the directory where doxygen is +# started. + +STRIP_FROM_PATH = + +# The STRIP_FROM_INC_PATH tag can be used to strip a user-defined part of +# the path mentioned in the documentation of a class, which tells +# the reader which header file to include in order to use a class. +# If left blank only the name of the header file containing the class +# definition is used. Otherwise one should specify the include paths that +# are normally passed to the compiler using the -I flag. + +STRIP_FROM_INC_PATH = ../include/ + +# If the SHORT_NAMES tag is set to YES, doxygen will generate much shorter +# (but less readable) file names. This can be useful if your file system +# doesn't support long names like on DOS, Mac, or CD-ROM. + +SHORT_NAMES = NO + +# If the JAVADOC_AUTOBRIEF tag is set to YES then Doxygen +# will interpret the first line (until the first dot) of a JavaDoc-style +# comment as the brief description. If set to NO, the JavaDoc +# comments will behave just like regular Qt-style comments +# (thus requiring an explicit @brief command for a brief description.) + +JAVADOC_AUTOBRIEF = NO + +# If the QT_AUTOBRIEF tag is set to YES then Doxygen will +# interpret the first line (until the first dot) of a Qt-style +# comment as the brief description. If set to NO, the comments +# will behave just like regular Qt-style comments (thus requiring +# an explicit \brief command for a brief description.) + +QT_AUTOBRIEF = YES + +# The MULTILINE_CPP_IS_BRIEF tag can be set to YES to make Doxygen +# treat a multi-line C++ special comment block (i.e. a block of //! or /// +# comments) as a brief description. This used to be the default behaviour. +# The new default is to treat a multi-line C++ comment block as a detailed +# description. Set this tag to YES if you prefer the old behaviour instead. + +MULTILINE_CPP_IS_BRIEF = NO + +# If the INHERIT_DOCS tag is set to YES (the default) then an undocumented +# member inherits the documentation from any documented member that it +# re-implements. + +INHERIT_DOCS = YES + +# If the SEPARATE_MEMBER_PAGES tag is set to YES, then doxygen will produce +# a new page for each member. If set to NO, the documentation of a member will +# be part of the file/class/namespace that contains it. + +SEPARATE_MEMBER_PAGES = NO + +# The TAB_SIZE tag can be used to set the number of spaces in a tab. +# Doxygen uses this value to replace tabs by spaces in code fragments. + +TAB_SIZE = 8 + +# This tag can be used to specify a number of aliases that acts +# as commands in the documentation. An alias has the form "name=value". +# For example adding "sideeffect=\par Side Effects:\n" will allow you to +# put the command \sideeffect (or @sideeffect) in the documentation, which +# will result in a user-defined paragraph with heading "Side Effects:". +# You can put \n's in the value part of an alias to insert newlines. + +ALIASES = "notyet{1}=Note: <b>"\1"</b> not yet supported in WiredTiger.\n@todo fix when \1 supported\n\n" \ + "errors=@returns zero on success and a non-zero error code on failure. See @ref error_returns \"Error Returns\" for details." \ + "ebusy_errors=@returns zero on success, EBUSY if there are open cursors on the object and a non-zero error code on failure. See @ref error_returns \"Error Returns\" for details." \ + "ex_ref{1}=@ref \1 \"\1\"" \ + "ref_single=@ref" \ + "subpage_single=@subpage" \ + "hrow{1}=<tr><th>\1</th></tr>" \ + "hrow{2}=<tr><th>\1</th><th>\2</th></tr>" \ + "hrow{3}=<tr><th>\1</th><th>\2</th><th>\3</th></tr>" \ + "hrow{4}=<tr><th>\1</th><th>\2</th><th>\3</th><th>\4</th></tr>" \ + "hrow{5}=<tr><th>\1</th><th>\2</th><th>\3</th><th>\4</th><th>\5</th></tr>" \ + "row{1}=<tr><td>\1</td></tr>" \ + "row{2}=<tr><td>\1</td><td>\2</td></tr>" \ + "row{3}=<tr><td>\1</td><td>\2</td><td>\3</td></tr>" \ + "row{4}=<tr><td>\1</td><td>\2</td><td>\3</td><td>\4</td></tr>" \ + "row{5}=<tr><td>\1</td><td>\2</td><td>\3</td><td>\4</td><td>\5</td></tr>" \ + "configstart{2}=@param config\n Configuration string, see @ref config_strings. Permitted values:\n <table>@hrow{Name,Effect,Values}" \ + "config{3}= @row{<tt>\1</tt>,\2,\3}" \ + "configend= </table>" \ + "configempty{2}=@param config\n Configuration string, see @ref config_strings. No values currently permitted." + +# This tag can be used to specify a number of word-keyword mappings (TCL only). +# A mapping has the form "name=value". For example adding +# "class=itcl::class" will allow you to use the command class in the +# itcl::class meaning. + +TCL_SUBST = + +# Set the OPTIMIZE_OUTPUT_FOR_C tag to YES if your project consists of C +# sources only. Doxygen will then generate output that is more tailored for C. +# For instance, some of the names that are used will be different. The list +# of all members will be omitted, etc. + +OPTIMIZE_OUTPUT_FOR_C = NO + +# Set the OPTIMIZE_OUTPUT_JAVA tag to YES if your project consists of Java +# sources only. Doxygen will then generate output that is more tailored for +# Java. For instance, namespaces will be presented as packages, qualified +# scopes will look different, etc. + +OPTIMIZE_OUTPUT_JAVA = YES + +# Set the OPTIMIZE_FOR_FORTRAN tag to YES if your project consists of Fortran +# sources only. Doxygen will then generate output that is more tailored for +# Fortran. + +OPTIMIZE_FOR_FORTRAN = NO + +# Set the OPTIMIZE_OUTPUT_VHDL tag to YES if your project consists of VHDL +# sources. Doxygen will then generate output that is tailored for +# VHDL. + +OPTIMIZE_OUTPUT_VHDL = NO + +# Doxygen selects the parser to use depending on the extension of the files it +# parses. With this tag you can assign which parser to use for a given +# extension. Doxygen has a built-in mapping, but you can override or extend it +# using this tag. The format is ext=language, where ext is a file extension, +# and language is one of the parsers supported by doxygen: IDL, Java, +# Javascript, CSharp, C, C++, D, PHP, Objective-C, Python, Fortran, VHDL, C, +# C++. For instance to make doxygen treat .inc files as Fortran files (default +# is PHP), and .f files as C (default is Fortran), use: inc=Fortran f=C. Note +# that for custom extensions you also need to set FILE_PATTERNS otherwise the +# files are not read by doxygen. + +EXTENSION_MAPPING = in=C + +# If MARKDOWN_SUPPORT is enabled (the default) then doxygen pre-processes all +# comments according to the Markdown format, which allows for more readable +# documentation. See http://daringfireball.net/projects/markdown/ for details. +# The output of markdown processing is further processed by doxygen, so you +# can mix doxygen, HTML, and XML commands with Markdown formatting. +# Disable only in case of backward compatibilities issues. + +MARKDOWN_SUPPORT = YES + +# When enabled doxygen tries to link words that correspond to documented classes, +# or namespaces to their corresponding documentation. Such a link can be +# prevented in individual cases by by putting a % sign in front of the word or +# globally by setting AUTOLINK_SUPPORT to NO. + +AUTOLINK_SUPPORT = YES + +# If you use STL classes (i.e. std::string, std::vector, etc.) but do not want +# to include (a tag file for) the STL sources as input, then you should +# set this tag to YES in order to let doxygen match functions declarations and +# definitions whose arguments contain STL classes (e.g. func(std::string); v.s. +# func(std::string) {}). This also makes the inheritance and collaboration +# diagrams that involve STL classes more complete and accurate. + +BUILTIN_STL_SUPPORT = NO + +# If you use Microsoft's C++/CLI language, you should set this option to YES to +# enable parsing support. + +CPP_CLI_SUPPORT = NO + +# Set the SIP_SUPPORT tag to YES if your project consists of sip sources only. +# Doxygen will parse them like normal C++ but will assume all classes use public +# instead of private inheritance when no explicit protection keyword is present. + +SIP_SUPPORT = NO + +# For Microsoft's IDL there are propget and propput attributes to indicate +# getter and setter methods for a property. Setting this option to YES (the +# default) will make doxygen replace the get and set methods by a property in +# the documentation. This will only work if the methods are indeed getting or +# setting a simple type. If this is not the case, or you want to show the +# methods anyway, you should set this option to NO. + +IDL_PROPERTY_SUPPORT = YES + +# If member grouping is used in the documentation and the DISTRIBUTE_GROUP_DOC +# tag is set to YES, then doxygen will reuse the documentation of the first +# member in the group (if any) for the other members of the group. By default +# all members of a group must be documented explicitly. + +DISTRIBUTE_GROUP_DOC = NO + +# Set the SUBGROUPING tag to YES (the default) to allow class member groups of +# the same type (for instance a group of public functions) to be put as a +# subgroup of that type (e.g. under the Public Functions section). Set it to +# NO to prevent subgrouping. Alternatively, this can be done per class using +# the \nosubgrouping command. + +SUBGROUPING = YES + +# When the INLINE_GROUPED_CLASSES tag is set to YES, classes, structs and +# unions are shown inside the group in which they are included (e.g. using +# @ingroup) instead of on a separate page (for HTML and Man pages) or +# section (for LaTeX and RTF). + +INLINE_GROUPED_CLASSES = NO + +# When the INLINE_SIMPLE_STRUCTS tag is set to YES, structs, classes, and +# unions with only public data fields will be shown inline in the documentation +# of the scope in which they are defined (i.e. file, namespace, or group +# documentation), provided this scope is documented. If set to NO (the default), +# structs, classes, and unions are shown on a separate page (for HTML and Man +# pages) or section (for LaTeX and RTF). + +INLINE_SIMPLE_STRUCTS = YES + +# When TYPEDEF_HIDES_STRUCT is enabled, a typedef of a struct, union, or enum +# is documented as struct, union, or enum with the name of the typedef. So +# typedef struct TypeS {} TypeT, will appear in the documentation as a struct +# with name TypeT. When disabled the typedef will appear as a member of a file, +# namespace, or class. And the struct will be named TypeS. This can typically +# be useful for C code in case the coding convention dictates that all compound +# types are typedef'ed and only the typedef is referenced, never the tag name. + +TYPEDEF_HIDES_STRUCT = YES + +# Similar to the SYMBOL_CACHE_SIZE the size of the symbol lookup cache can be +# set using LOOKUP_CACHE_SIZE. This cache is used to resolve symbols given +# their name and scope. Since this can be an expensive process and often the +# same symbol appear multiple times in the code, doxygen keeps a cache of +# pre-resolved symbols. If the cache is too small doxygen will become slower. +# If the cache is too large, memory is wasted. The cache size is given by this +# formula: 2^(16+LOOKUP_CACHE_SIZE). The valid range is 0..9, the default is 0, +# corresponding to a cache size of 2^16 = 65536 symbols. + +LOOKUP_CACHE_SIZE = 0 + +#--------------------------------------------------------------------------- +# Build related configuration options +#--------------------------------------------------------------------------- + +# If the EXTRACT_ALL tag is set to YES doxygen will assume all entities in +# documentation are documented, even if no documentation was available. +# Private class members and static file members will be hidden unless +# the EXTRACT_PRIVATE and EXTRACT_STATIC tags are set to YES + +EXTRACT_ALL = NO + +# If the EXTRACT_PRIVATE tag is set to YES all private members of a class +# will be included in the documentation. + +EXTRACT_PRIVATE = NO + +# If the EXTRACT_PACKAGE tag is set to YES all members with package or internal +# scope will be included in the documentation. + +EXTRACT_PACKAGE = NO + +# If the EXTRACT_STATIC tag is set to YES all static members of a file +# will be included in the documentation. + +EXTRACT_STATIC = NO + +# If the EXTRACT_LOCAL_CLASSES tag is set to YES classes (and structs) +# defined locally in source files will be included in the documentation. +# If set to NO only classes defined in header files are included. + +EXTRACT_LOCAL_CLASSES = YES + +# This flag is only useful for Objective-C code. When set to YES local +# methods, which are defined in the implementation section but not in +# the interface are included in the documentation. +# If set to NO (the default) only methods in the interface are included. + +EXTRACT_LOCAL_METHODS = NO + +# If this flag is set to YES, the members of anonymous namespaces will be +# extracted and appear in the documentation as a namespace called +# 'anonymous_namespace{file}', where file will be replaced with the base +# name of the file that contains the anonymous namespace. By default +# anonymous namespaces are hidden. + +EXTRACT_ANON_NSPACES = NO + +# If the HIDE_UNDOC_MEMBERS tag is set to YES, Doxygen will hide all +# undocumented members of documented classes, files or namespaces. +# If set to NO (the default) these members will be included in the +# various overviews, but no documentation section is generated. +# This option has no effect if EXTRACT_ALL is enabled. + +HIDE_UNDOC_MEMBERS = NO + +# If the HIDE_UNDOC_CLASSES tag is set to YES, Doxygen will hide all +# undocumented classes that are normally visible in the class hierarchy. +# If set to NO (the default) these classes will be included in the various +# overviews. This option has no effect if EXTRACT_ALL is enabled. + +HIDE_UNDOC_CLASSES = NO + +# If the HIDE_FRIEND_COMPOUNDS tag is set to YES, Doxygen will hide all +# friend (class|struct|union) declarations. +# If set to NO (the default) these declarations will be included in the +# documentation. + +HIDE_FRIEND_COMPOUNDS = NO + +# If the HIDE_IN_BODY_DOCS tag is set to YES, Doxygen will hide any +# documentation blocks found inside the body of a function. +# If set to NO (the default) these blocks will be appended to the +# function's detailed documentation block. + +HIDE_IN_BODY_DOCS = NO + +# The INTERNAL_DOCS tag determines if documentation +# that is typed after a \internal command is included. If the tag is set +# to NO (the default) then the documentation will be excluded. +# Set it to YES to include the internal documentation. + +INTERNAL_DOCS = NO + +# If the CASE_SENSE_NAMES tag is set to NO then Doxygen will only generate +# file names in lower-case letters. If set to YES upper-case letters are also +# allowed. This is useful if you have classes or files whose names only differ +# in case and if your file system supports case sensitive file names. Windows +# and Mac users are advised to set this option to NO. + +CASE_SENSE_NAMES = NO + +# If the HIDE_SCOPE_NAMES tag is set to NO (the default) then Doxygen +# will show members with their full class and namespace scopes in the +# documentation. If set to YES the scope will be hidden. + +HIDE_SCOPE_NAMES = NO + +# If the SHOW_INCLUDE_FILES tag is set to YES (the default) then Doxygen +# will put a list of the files that are included by a file in the documentation +# of that file. + +SHOW_INCLUDE_FILES = NO + +# If the FORCE_LOCAL_INCLUDES tag is set to YES then Doxygen +# will list include files with double quotes in the documentation +# rather than with sharp brackets. + +FORCE_LOCAL_INCLUDES = NO + +# If the INLINE_INFO tag is set to YES (the default) then a tag [inline] +# is inserted in the documentation for inline members. + +INLINE_INFO = YES + +# If the SORT_MEMBER_DOCS tag is set to YES (the default) then doxygen +# will sort the (detailed) documentation of file and class members +# alphabetically by member name. If set to NO the members will appear in +# declaration order. + +SORT_MEMBER_DOCS = YES + +# If the SORT_BRIEF_DOCS tag is set to YES then doxygen will sort the +# brief documentation of file, namespace and class members alphabetically +# by member name. If set to NO (the default) the members will appear in +# declaration order. + +SORT_BRIEF_DOCS = NO + +# If the SORT_MEMBERS_CTORS_1ST tag is set to YES then doxygen +# will sort the (brief and detailed) documentation of class members so that +# constructors and destructors are listed first. If set to NO (the default) +# the constructors will appear in the respective orders defined by +# SORT_MEMBER_DOCS and SORT_BRIEF_DOCS. +# This tag will be ignored for brief docs if SORT_BRIEF_DOCS is set to NO +# and ignored for detailed docs if SORT_MEMBER_DOCS is set to NO. + +SORT_MEMBERS_CTORS_1ST = NO + +# If the SORT_GROUP_NAMES tag is set to YES then doxygen will sort the +# hierarchy of group names into alphabetical order. If set to NO (the default) +# the group names will appear in their defined order. + +SORT_GROUP_NAMES = YES + +# If the SORT_BY_SCOPE_NAME tag is set to YES, the class list will be +# sorted by fully-qualified names, including namespaces. If set to +# NO (the default), the class list will be sorted only by class name, +# not including the namespace part. +# Note: This option is not very useful if HIDE_SCOPE_NAMES is set to YES. +# Note: This option applies only to the class list, not to the +# alphabetical list. + +SORT_BY_SCOPE_NAME = YES + +# If the STRICT_PROTO_MATCHING option is enabled and doxygen fails to +# do proper type resolution of all parameters of a function it will reject a +# match between the prototype and the implementation of a member function even +# if there is only one candidate or it is obvious which candidate to choose +# by doing a simple string match. By disabling STRICT_PROTO_MATCHING doxygen +# will still accept a match between prototype and implementation in such cases. + +STRICT_PROTO_MATCHING = NO + +# The GENERATE_TODOLIST tag can be used to enable (YES) or +# disable (NO) the todo list. This list is created by putting \todo +# commands in the documentation. + +GENERATE_TODOLIST = NO + +# The GENERATE_TESTLIST tag can be used to enable (YES) or +# disable (NO) the test list. This list is created by putting \test +# commands in the documentation. + +GENERATE_TESTLIST = YES + +# The GENERATE_BUGLIST tag can be used to enable (YES) or +# disable (NO) the bug list. This list is created by putting \bug +# commands in the documentation. + +GENERATE_BUGLIST = YES + +# The GENERATE_DEPRECATEDLIST tag can be used to enable (YES) or +# disable (NO) the deprecated list. This list is created by putting +# \deprecated commands in the documentation. + +GENERATE_DEPRECATEDLIST= YES + +# The ENABLED_SECTIONS tag can be used to enable conditional +# documentation sections, marked by \if sectionname ... \endif. + +ENABLED_SECTIONS = + +# The MAX_INITIALIZER_LINES tag determines the maximum number of lines +# the initial value of a variable or macro consists of for it to appear in +# the documentation. If the initializer consists of more lines than specified +# here it will be hidden. Use a value of 0 to hide initializers completely. +# The appearance of the initializer of individual variables and macros in the +# documentation can be controlled using \showinitializer or \hideinitializer +# command in the documentation regardless of this setting. + +MAX_INITIALIZER_LINES = 0 + +# Set the SHOW_USED_FILES tag to NO to disable the list of files generated +# at the bottom of the documentation of classes and structs. If set to YES the +# list will mention the files that were used to generate the documentation. + +SHOW_USED_FILES = NO + +# Set the SHOW_FILES tag to NO to disable the generation of the Files page. +# This will remove the Files entry from the Quick Index and from the +# Folder Tree View (if specified). The default is YES. + +SHOW_FILES = NO + +# Set the SHOW_NAMESPACES tag to NO to disable the generation of the +# Namespaces page. This will remove the Namespaces entry from the Quick Index +# and from the Folder Tree View (if specified). The default is YES. + +SHOW_NAMESPACES = NO + +# The FILE_VERSION_FILTER tag can be used to specify a program or script that +# doxygen should invoke to get the current version for each file (typically from +# the version control system). Doxygen will invoke the program by executing (via +# popen()) the command <command> <input-file>, where <command> is the value of +# the FILE_VERSION_FILTER tag, and <input-file> is the name of an input file +# provided by doxygen. Whatever the program writes to standard output +# is used as the file version. See the manual for examples. + +FILE_VERSION_FILTER = + +# The LAYOUT_FILE tag can be used to specify a layout file which will be parsed +# by doxygen. The layout file controls the global structure of the generated +# output files in an output format independent way. To create the layout file +# that represents doxygen's defaults, run doxygen with the -l option. +# You can optionally specify a file name after the option, if omitted +# DoxygenLayout.xml will be used as the name of the layout file. + +LAYOUT_FILE = style/DoxygenLayout.xml + +# The CITE_BIB_FILES tag can be used to specify one or more bib files +# containing the references data. This must be a list of .bib files. The +# .bib extension is automatically appended if omitted. Using this command +# requires the bibtex tool to be installed. See also +# http://en.wikipedia.org/wiki/BibTeX for more info. For LaTeX the style +# of the bibliography can be controlled using LATEX_BIB_STYLE. To use this +# feature you need bibtex and perl available in the search path. + +CITE_BIB_FILES = + +#--------------------------------------------------------------------------- +# configuration options related to warning and progress messages +#--------------------------------------------------------------------------- + +# The QUIET tag can be used to turn on/off the messages that are generated +# by doxygen. Possible values are YES and NO. If left blank NO is used. + +QUIET = NO + +# The WARNINGS tag can be used to turn on/off the warning messages that are +# generated by doxygen. Possible values are YES and NO. If left blank +# NO is used. + +WARNINGS = YES + +# If WARN_IF_UNDOCUMENTED is set to YES, then doxygen will generate warnings +# for undocumented members. If EXTRACT_ALL is set to YES then this flag will +# automatically be disabled. + +WARN_IF_UNDOCUMENTED = YES + +# If WARN_IF_DOC_ERROR is set to YES, doxygen will generate warnings for +# potential errors in the documentation, such as not documenting some +# parameters in a documented function, or documenting parameters that +# don't exist or using markup commands wrongly. + +WARN_IF_DOC_ERROR = YES + +# The WARN_NO_PARAMDOC option can be enabled to get warnings for +# functions that are documented, but have no documentation for their parameters +# or return value. If set to NO (the default) doxygen will only warn about +# wrong or incomplete parameter documentation, but not about the absence of +# documentation. + +WARN_NO_PARAMDOC = YES + +# The WARN_FORMAT tag determines the format of the warning messages that +# doxygen can produce. The string should contain the $file, $line, and $text +# tags, which will be replaced by the file and line number from which the +# warning originated and the warning text. Optionally the format may contain +# $version, which will be replaced by the version of the file (if it could +# be obtained via FILE_VERSION_FILTER) + +WARN_FORMAT = "$file:$line: $text" + +# The WARN_LOGFILE tag can be used to specify a file to which warning +# and error messages should be written. If left blank the output is written +# to stderr. + +WARN_LOGFILE = doxygen.log + +#--------------------------------------------------------------------------- +# configuration options related to the input files +#--------------------------------------------------------------------------- + +# The INPUT tag can be used to specify the files and/or directories that contain +# documented source files. You may enter file names like "myfile.cpp" or +# directories like "/usr/src/myproject". Separate the files or directories +# with spaces. + +INPUT = ../include/wiredtiger.in \ + ../include/wiredtiger_ext.h \ + . + +# This tag can be used to specify the character encoding of the source files +# that doxygen parses. Internally doxygen uses the UTF-8 encoding, which is +# also the default input encoding. Doxygen uses libiconv (or the iconv built +# into libc) for the transcoding. See http://www.gnu.org/software/libiconv for +# the list of possible encodings. + +INPUT_ENCODING = UTF-8 + +# If the value of the INPUT tag contains directories, you can use the +# FILE_PATTERNS tag to specify one or more wildcard pattern (like *.cpp +# and *.h) to filter out the source-files in the directories. If left +# blank the following patterns are tested: +# *.c *.cc *.cxx *.cpp *.c++ *.d *.java *.ii *.ixx *.ipp *.i++ *.inl *.h *.hh +# *.hxx *.hpp *.h++ *.idl *.odl *.cs *.php *.php3 *.inc *.m *.mm *.dox *.py +# *.f90 *.f *.for *.vhd *.vhdl + +FILE_PATTERNS = *.c \ + *.cc \ + *.cxx \ + *.cpp \ + *.c++ \ + *.d \ + *.java \ + *.ii \ + *.ixx \ + *.ipp \ + *.i++ \ + *.inl \ + *.h \ + *.hh \ + *.hxx \ + *.hpp \ + *.h++ \ + *.idl \ + *.odl \ + *.cs \ + *.php \ + *.php3 \ + *.inc \ + *.m \ + *.md \ + *.mm \ + *.dox \ + *.py \ + *.f90 \ + *.f \ + *.vhd \ + *.vhdl + +# The RECURSIVE tag can be used to turn specify whether or not subdirectories +# should be searched for input files as well. Possible values are YES and NO. +# If left blank NO is used. + +RECURSIVE = YES + +# The EXCLUDE tag can be used to specify files and/or directories that should be +# excluded from the INPUT source files. This way you can easily exclude a +# subdirectory from a directory tree whose root is specified with the INPUT tag. +# Note that relative paths are relative to the directory from which doxygen is +# run. + +EXCLUDE = bdb-map.dox \ + design.dox \ + processes.dox \ + sql-map.dox \ + tools \ + top + +# The EXCLUDE_SYMLINKS tag can be used to select whether or not files or +# directories that are symbolic links (a Unix file system feature) are excluded +# from the input. + +EXCLUDE_SYMLINKS = NO + +# If the value of the INPUT tag contains directories, you can use the +# EXCLUDE_PATTERNS tag to specify one or more wildcard patterns to exclude +# certain files from those directories. Note that the wildcards are matched +# against the file with absolute path, so to exclude all test directories +# for example use the pattern */test/* + +EXCLUDE_PATTERNS = + +# The EXCLUDE_SYMBOLS tag can be used to specify one or more symbol names +# (namespaces, classes, functions, etc.) that should be excluded from the +# output. The symbol name can be a fully qualified name, a word, or if the +# wildcard * is used, a substring. Examples: ANamespace, AClass, +# AClass::ANamespace, ANamespace::*Test + +EXCLUDE_SYMBOLS = __F \ + doc_* + +# The EXAMPLE_PATH tag can be used to specify one or more files or +# directories that contain example code fragments that are included (see +# the \include command). + +EXAMPLE_PATH = ../../examples/c \ + ../../examples/java/com/wiredtiger/examples/ \ + ../../ext/compressors/nop \ + ../../ext/encryptors/nop \ + ../../ext/encryptors/rotn + +# If the value of the EXAMPLE_PATH tag contains directories, you can use the +# EXAMPLE_PATTERNS tag to specify one or more wildcard pattern (like *.cpp +# and *.h) to filter out the source-files in the directories. If left +# blank all files are included. + +EXAMPLE_PATTERNS = + +# If the EXAMPLE_RECURSIVE tag is set to YES then subdirectories will be +# searched for input files to be used with the \include or \dontinclude +# commands irrespective of the value of the RECURSIVE tag. +# Possible values are YES and NO. If left blank NO is used. + +EXAMPLE_RECURSIVE = NO + +# The IMAGE_PATH tag can be used to specify one or more files or +# directories that contain image that are included in the documentation (see +# the \image command). + +IMAGE_PATH = images + +# The INPUT_FILTER tag can be used to specify a program that doxygen should +# invoke to filter for each input file. Doxygen will invoke the filter program +# by executing (via popen()) the command <filter> <input-file>, where <filter> +# is the value of the INPUT_FILTER tag, and <input-file> is the name of an +# input file. Doxygen will then use the output that the filter program writes +# to standard output. If FILTER_PATTERNS is specified, this tag will be +# ignored. + +INPUT_FILTER = + +# The FILTER_PATTERNS tag can be used to specify filters on a per file pattern +# basis. Doxygen will compare the file name with each pattern and apply the +# filter if there is a match. The filters are a list of the form: +# pattern=filter (like *.cpp=my_cpp_filter). See INPUT_FILTER for further +# info on how filters are used. If FILTER_PATTERNS is empty or if +# non of the patterns match the file name, INPUT_FILTER is applied. + +FILTER_PATTERNS = *.py=tools/pyfilter \ + *.dox=tools/doxfilter + +# If the FILTER_SOURCE_FILES tag is set to YES, the input filter (if set using +# INPUT_FILTER) will be used to filter the input files when producing source +# files to browse (i.e. when SOURCE_BROWSER is set to YES). + +FILTER_SOURCE_FILES = NO + +# The FILTER_SOURCE_PATTERNS tag can be used to specify source filters per file +# pattern. A pattern will override the setting for FILTER_PATTERN (if any) +# and it is also possible to disable source filtering for a specific pattern +# using *.ext= (so without naming a filter). This option only has effect when +# FILTER_SOURCE_FILES is enabled. + +FILTER_SOURCE_PATTERNS = + +#--------------------------------------------------------------------------- +# configuration options related to source browsing +#--------------------------------------------------------------------------- + +# If the SOURCE_BROWSER tag is set to YES then a list of source files will +# be generated. Documented entities will be cross-referenced with these sources. +# Note: To get rid of all source code in the generated output, make sure also +# VERBATIM_HEADERS is set to NO. + +SOURCE_BROWSER = NO + +# Setting the INLINE_SOURCES tag to YES will include the body +# of functions and classes directly in the documentation. + +INLINE_SOURCES = NO + +# Setting the STRIP_CODE_COMMENTS tag to YES (the default) will instruct +# doxygen to hide any special comment blocks from generated source code +# fragments. Normal C, C++ and Fortran comments will always remain visible. + +STRIP_CODE_COMMENTS = YES + +# If the REFERENCED_BY_RELATION tag is set to YES +# then for each documented function all documented +# functions referencing it will be listed. + +REFERENCED_BY_RELATION = NO + +# If the REFERENCES_RELATION tag is set to YES +# then for each documented function all documented entities +# called/used by that function will be listed. + +REFERENCES_RELATION = NO + +# If the REFERENCES_LINK_SOURCE tag is set to YES (the default) +# and SOURCE_BROWSER tag is set to YES, then the hyperlinks from +# functions in REFERENCES_RELATION and REFERENCED_BY_RELATION lists will +# link to the source code. Otherwise they will link to the documentation. + +REFERENCES_LINK_SOURCE = NO + +# If the USE_HTAGS tag is set to YES then the references to source code +# will point to the HTML generated by the htags(1) tool instead of doxygen +# built-in source browser. The htags tool is part of GNU's global source +# tagging system (see http://www.gnu.org/software/global/global.html). You +# will need version 4.8.6 or higher. + +USE_HTAGS = NO + +# If the VERBATIM_HEADERS tag is set to YES (the default) then Doxygen +# will generate a verbatim copy of the header file for each class for +# which an include is specified. Set to NO to disable this. + +VERBATIM_HEADERS = NO + +#--------------------------------------------------------------------------- +# configuration options related to the alphabetical class index +#--------------------------------------------------------------------------- + +# If the ALPHABETICAL_INDEX tag is set to YES, an alphabetical index +# of all compounds will be generated. Enable this if the project +# contains a lot of classes, structs, unions or interfaces. + +ALPHABETICAL_INDEX = NO + +# If the alphabetical index is enabled (see ALPHABETICAL_INDEX) then +# the COLS_IN_ALPHA_INDEX tag can be used to specify the number of columns +# in which this list will be split (can be a number in the range [1..20]) + +COLS_IN_ALPHA_INDEX = 5 + +# In case all classes in a project start with a common prefix, all +# classes will be put under the same header in the alphabetical index. +# The IGNORE_PREFIX tag can be used to specify one or more prefixes that +# should be ignored while generating the index headers. + +IGNORE_PREFIX = WT_ + +#--------------------------------------------------------------------------- +# configuration options related to the HTML output +#--------------------------------------------------------------------------- + +# If the GENERATE_HTML tag is set to YES (the default) Doxygen will +# generate HTML output. + +GENERATE_HTML = YES + +# The HTML_OUTPUT tag is used to specify where the HTML docs will be put. +# If a relative path is entered the value of OUTPUT_DIRECTORY will be +# put in front of it. If left blank `html' will be used as the default path. + +HTML_OUTPUT = . + +# The HTML_FILE_EXTENSION tag can be used to specify the file extension for +# each generated HTML page (for example: .htm,.php,.asp). If it is left blank +# doxygen will generate files with .html extension. + +HTML_FILE_EXTENSION = .html + +# The HTML_HEADER tag can be used to specify a personal HTML header for +# each generated HTML page. If it is left blank doxygen will generate a +# standard header. Note that when using a custom header you are responsible +# for the proper inclusion of any scripts and style sheets that doxygen +# needs, which is dependent on the configuration options used. +# It is advised to generate a default header using "doxygen -w html +# header.html footer.html stylesheet.css YourConfigFile" and then modify +# that header. Note that the header is subject to change so you typically +# have to redo this when upgrading to a newer version of doxygen or when +# changing the value of configuration settings such as GENERATE_TREEVIEW! + +HTML_HEADER = style/header.html + +# The HTML_FOOTER tag can be used to specify a personal HTML footer for +# each generated HTML page. If it is left blank doxygen will generate a +# standard footer. + +HTML_FOOTER = style/footer.html + +# The HTML_STYLESHEET tag can be used to specify a user-defined cascading +# style sheet that is used by each HTML page. It can be used to +# fine-tune the look of the HTML output. If left blank doxygen will +# generate a default style sheet. Note that it is recommended to use +# HTML_EXTRA_STYLESHEET instead of this one, as it is more robust and this +# tag will in the future become obsolete. + +HTML_STYLESHEET = + +# The HTML_EXTRA_STYLESHEET tag can be used to specify an additional +# user-defined cascading style sheet that is included after the standard +# style sheets created by doxygen. Using this option one can overrule +# certain style aspects. This is preferred over using HTML_STYLESHEET +# since it does not replace the standard style sheet and is therefor more +# robust against future updates. Doxygen will copy the style sheet file to +# the output directory. + +HTML_EXTRA_STYLESHEET = style/wiredtiger.css + +# The HTML_EXTRA_FILES tag can be used to specify one or more extra images or +# other source files which should be copied to the HTML output directory. Note +# that these files will be copied to the base HTML output directory. Use the +# $relpath$ marker in the HTML_HEADER and/or HTML_FOOTER files to load these +# files. In the HTML_STYLESHEET file, use the file name only. Also note that +# the files will be copied as-is; there are no commands or markers available. + +HTML_EXTRA_FILES = + +# The HTML_COLORSTYLE_HUE tag controls the color of the HTML output. +# Doxygen will adjust the colors in the style sheet and background images +# according to this color. Hue is specified as an angle on a colorwheel, +# see http://en.wikipedia.org/wiki/Hue for more information. +# For instance the value 0 represents red, 60 is yellow, 120 is green, +# 180 is cyan, 240 is blue, 300 purple, and 360 is red again. +# The allowed range is 0 to 359. + +HTML_COLORSTYLE_HUE = 34 + +# The HTML_COLORSTYLE_SAT tag controls the purity (or saturation) of +# the colors in the HTML output. For a value of 0 the output will use +# grayscales only. A value of 255 will produce the most vivid colors. + +HTML_COLORSTYLE_SAT = 81 + +# The HTML_COLORSTYLE_GAMMA tag controls the gamma correction applied to +# the luminance component of the colors in the HTML output. Values below +# 100 gradually make the output lighter, whereas values above 100 make +# the output darker. The value divided by 100 is the actual gamma applied, +# so 80 represents a gamma of 0.8, The value 220 represents a gamma of 2.2, +# and 100 does not change the gamma. + +HTML_COLORSTYLE_GAMMA = 96 + +# If the HTML_TIMESTAMP tag is set to YES then the footer of each generated HTML +# page will contain the date and time when the page was generated. Setting +# this to NO can help when comparing the output of multiple runs. + +HTML_TIMESTAMP = YES + +# If the HTML_DYNAMIC_SECTIONS tag is set to YES then the generated HTML +# documentation will contain sections that can be hidden and shown after the +# page has loaded. + +HTML_DYNAMIC_SECTIONS = YES + +# With HTML_INDEX_NUM_ENTRIES one can control the preferred number of +# entries shown in the various tree structured indices initially; the user +# can expand and collapse entries dynamically later on. Doxygen will expand +# the tree to such a level that at most the specified number of entries are +# visible (unless a fully collapsed tree already exceeds this amount). +# So setting the number of entries 1 will produce a full collapsed tree by +# default. 0 is a special value representing an infinite number of entries +# and will result in a full expanded tree by default. + +HTML_INDEX_NUM_ENTRIES = 100 + +# If the GENERATE_DOCSET tag is set to YES, additional index files +# will be generated that can be used as input for Apple's Xcode 3 +# integrated development environment, introduced with OSX 10.5 (Leopard). +# To create a documentation set, doxygen will generate a Makefile in the +# HTML output directory. Running make will produce the docset in that +# directory and running "make install" will install the docset in +# ~/Library/Developer/Shared/Documentation/DocSets so that Xcode will find +# it at startup. +# See http://developer.apple.com/tools/creatingdocsetswithdoxygen.html +# for more information. + +GENERATE_DOCSET = NO + +# When GENERATE_DOCSET tag is set to YES, this tag determines the name of the +# feed. A documentation feed provides an umbrella under which multiple +# documentation sets from a single provider (such as a company or product suite) +# can be grouped. + +DOCSET_FEEDNAME = "Doxygen generated docs" + +# When GENERATE_DOCSET tag is set to YES, this tag specifies a string that +# should uniquely identify the documentation set bundle. This should be a +# reverse domain-name style string, e.g. com.mycompany.MyDocSet. Doxygen +# will append .docset to the name. + +DOCSET_BUNDLE_ID = org.doxygen.Project + +# When GENERATE_PUBLISHER_ID tag specifies a string that should uniquely +# identify the documentation publisher. This should be a reverse domain-name +# style string, e.g. com.mycompany.MyDocSet.documentation. + +DOCSET_PUBLISHER_ID = org.doxygen.Publisher + +# The GENERATE_PUBLISHER_NAME tag identifies the documentation publisher. + +DOCSET_PUBLISHER_NAME = Publisher + +# If the GENERATE_HTMLHELP tag is set to YES, additional index files +# will be generated that can be used as input for tools like the +# Microsoft HTML help workshop to generate a compiled HTML help file (.chm) +# of the generated HTML documentation. + +GENERATE_HTMLHELP = NO + +# If the GENERATE_HTMLHELP tag is set to YES, the CHM_FILE tag can +# be used to specify the file name of the resulting .chm file. You +# can add a path in front of the file if the result should not be +# written to the html output directory. + +CHM_FILE = + +# If the GENERATE_HTMLHELP tag is set to YES, the HHC_LOCATION tag can +# be used to specify the location (absolute path including file name) of +# the HTML help compiler (hhc.exe). If non-empty doxygen will try to run +# the HTML help compiler on the generated index.hhp. + +HHC_LOCATION = + +# If the GENERATE_HTMLHELP tag is set to YES, the GENERATE_CHI flag +# controls if a separate .chi index file is generated (YES) or that +# it should be included in the master .chm file (NO). + +GENERATE_CHI = NO + +# If the GENERATE_HTMLHELP tag is set to YES, the CHM_INDEX_ENCODING +# is used to encode HtmlHelp index (hhk), content (hhc) and project file +# content. + +CHM_INDEX_ENCODING = + +# If the GENERATE_HTMLHELP tag is set to YES, the BINARY_TOC flag +# controls whether a binary table of contents is generated (YES) or a +# normal table of contents (NO) in the .chm file. + +BINARY_TOC = NO + +# The TOC_EXPAND flag can be set to YES to add extra items for group members +# to the contents of the HTML help documentation and to the tree view. + +TOC_EXPAND = NO + +# If the GENERATE_QHP tag is set to YES and both QHP_NAMESPACE and +# QHP_VIRTUAL_FOLDER are set, an additional index file will be generated +# that can be used as input for Qt's qhelpgenerator to generate a +# Qt Compressed Help (.qch) of the generated HTML documentation. + +GENERATE_QHP = NO + +# If the QHG_LOCATION tag is specified, the QCH_FILE tag can +# be used to specify the file name of the resulting .qch file. +# The path specified is relative to the HTML output folder. + +QCH_FILE = + +# The QHP_NAMESPACE tag specifies the namespace to use when generating +# Qt Help Project output. For more information please see +# http://doc.trolltech.com/qthelpproject.html#namespace + +QHP_NAMESPACE = org.doxygen.Project + +# The QHP_VIRTUAL_FOLDER tag specifies the namespace to use when generating +# Qt Help Project output. For more information please see +# http://doc.trolltech.com/qthelpproject.html#virtual-folders + +QHP_VIRTUAL_FOLDER = doc + +# If QHP_CUST_FILTER_NAME is set, it specifies the name of a custom filter to +# add. For more information please see +# http://doc.trolltech.com/qthelpproject.html#custom-filters + +QHP_CUST_FILTER_NAME = + +# The QHP_CUST_FILT_ATTRS tag specifies the list of the attributes of the +# custom filter to add. For more information please see +# <a href="http://doc.trolltech.com/qthelpproject.html#custom-filters"> +# Qt Help Project / Custom Filters</a>. + +QHP_CUST_FILTER_ATTRS = + +# The QHP_SECT_FILTER_ATTRS tag specifies the list of the attributes this +# project's +# filter section matches. +# <a href="http://doc.trolltech.com/qthelpproject.html#filter-attributes"> +# Qt Help Project / Filter Attributes</a>. + +QHP_SECT_FILTER_ATTRS = + +# If the GENERATE_QHP tag is set to YES, the QHG_LOCATION tag can +# be used to specify the location of Qt's qhelpgenerator. +# If non-empty doxygen will try to run qhelpgenerator on the generated +# .qhp file. + +QHG_LOCATION = + +# If the GENERATE_ECLIPSEHELP tag is set to YES, additional index files +# will be generated, which together with the HTML files, form an Eclipse help +# plugin. To install this plugin and make it available under the help contents +# menu in Eclipse, the contents of the directory containing the HTML and XML +# files needs to be copied into the plugins directory of eclipse. The name of +# the directory within the plugins directory should be the same as +# the ECLIPSE_DOC_ID value. After copying Eclipse needs to be restarted before +# the help appears. + +GENERATE_ECLIPSEHELP = NO + +# A unique identifier for the eclipse help plugin. When installing the plugin +# the directory name containing the HTML and XML files should also have +# this name. + +ECLIPSE_DOC_ID = org.doxygen.Project + +# The DISABLE_INDEX tag can be used to turn on/off the condensed index (tabs) +# at top of each HTML page. The value NO (the default) enables the index and +# the value YES disables it. Since the tabs have the same information as the +# navigation tree you can set this option to NO if you already set +# GENERATE_TREEVIEW to YES. + +DISABLE_INDEX = NO + +# The GENERATE_TREEVIEW tag is used to specify whether a tree-like index +# structure should be generated to display hierarchical information. +# If the tag value is set to YES, a side panel will be generated +# containing a tree-like index structure (just like the one that +# is generated for HTML Help). For this to work a browser that supports +# JavaScript, DHTML, CSS and frames is required (i.e. any modern browser). +# Windows users are probably better off using the HTML help feature. +# Since the tree basically has the same information as the tab index you +# could consider to set DISABLE_INDEX to NO when enabling this option. + +GENERATE_TREEVIEW = YES + +# The ENUM_VALUES_PER_LINE tag can be used to set the number of enum values +# (range [0,1..20]) that doxygen will group on one line in the generated HTML +# documentation. Note that a value of 0 will completely suppress the enum +# values from appearing in the overview section. + +ENUM_VALUES_PER_LINE = 4 + +# If the treeview is enabled (see GENERATE_TREEVIEW) then this tag can be +# used to set the initial width (in pixels) of the frame in which the tree +# is shown. + +TREEVIEW_WIDTH = 200 + +# When the EXT_LINKS_IN_WINDOW option is set to YES doxygen will open +# links to external symbols imported via tag files in a separate window. + +EXT_LINKS_IN_WINDOW = NO + +# Use this tag to change the font size of Latex formulas included +# as images in the HTML documentation. The default is 10. Note that +# when you change the font size after a successful doxygen run you need +# to manually remove any form_*.png images from the HTML output directory +# to force them to be regenerated. + +FORMULA_FONTSIZE = 10 + +# Use the FORMULA_TRANPARENT tag to determine whether or not the images +# generated for formulas are transparent PNGs. Transparent PNGs are +# not supported properly for IE 6.0, but are supported on all modern browsers. +# Note that when changing this option you need to delete any form_*.png files +# in the HTML output before the changes have effect. + +FORMULA_TRANSPARENT = YES + +# Enable the USE_MATHJAX option to render LaTeX formulas using MathJax +# (see http://www.mathjax.org) which uses client side Javascript for the +# rendering instead of using prerendered bitmaps. Use this if you do not +# have LaTeX installed or if you want to formulas look prettier in the HTML +# output. When enabled you may also need to install MathJax separately and +# configure the path to it using the MATHJAX_RELPATH option. + +USE_MATHJAX = NO + +# When MathJax is enabled you need to specify the location relative to the +# HTML output directory using the MATHJAX_RELPATH option. The destination +# directory should contain the MathJax.js script. For instance, if the mathjax +# directory is located at the same level as the HTML output directory, then +# MATHJAX_RELPATH should be ../mathjax. The default value points to +# the MathJax Content Delivery Network so you can quickly see the result without +# installing MathJax. However, it is strongly recommended to install a local +# copy of MathJax from http://www.mathjax.org before deployment. + +MATHJAX_RELPATH = http://www.mathjax.org/mathjax + +# The MATHJAX_EXTENSIONS tag can be used to specify one or MathJax extension +# names that should be enabled during MathJax rendering. + +MATHJAX_EXTENSIONS = + +# When the SEARCHENGINE tag is enabled doxygen will generate a search box +# for the HTML output. The underlying search engine uses javascript +# and DHTML and should work on any modern browser. Note that when using +# HTML help (GENERATE_HTMLHELP), Qt help (GENERATE_QHP), or docsets +# (GENERATE_DOCSET) there is already a search function so this one should +# typically be disabled. For large projects the javascript based search engine +# can be slow, then enabling SERVER_BASED_SEARCH may provide a better solution. + +SEARCHENGINE = NO + +# When the SERVER_BASED_SEARCH tag is enabled the search engine will be +# implemented using a PHP enabled web server instead of at the web client +# using Javascript. Doxygen will generate the search PHP script and index +# file to put on the web server. The advantage of the server +# based approach is that it scales better to large projects and allows +# full text search. The disadvantages are that it is more difficult to setup +# and does not have live searching capabilities. + +SERVER_BASED_SEARCH = NO + +#--------------------------------------------------------------------------- +# configuration options related to the LaTeX output +#--------------------------------------------------------------------------- + +# If the GENERATE_LATEX tag is set to YES (the default) Doxygen will +# generate Latex output. + +GENERATE_LATEX = NO + +# The LATEX_OUTPUT tag is used to specify where the LaTeX docs will be put. +# If a relative path is entered the value of OUTPUT_DIRECTORY will be +# put in front of it. If left blank `latex' will be used as the default path. + +LATEX_OUTPUT = latex + +# The LATEX_CMD_NAME tag can be used to specify the LaTeX command name to be +# invoked. If left blank `latex' will be used as the default command name. +# Note that when enabling USE_PDFLATEX this option is only used for +# generating bitmaps for formulas in the HTML output, but not in the +# Makefile that is written to the output directory. + +LATEX_CMD_NAME = latex + +# The MAKEINDEX_CMD_NAME tag can be used to specify the command name to +# generate index for LaTeX. If left blank `makeindex' will be used as the +# default command name. + +MAKEINDEX_CMD_NAME = makeindex + +# If the COMPACT_LATEX tag is set to YES Doxygen generates more compact +# LaTeX documents. This may be useful for small projects and may help to +# save some trees in general. + +COMPACT_LATEX = NO + +# The PAPER_TYPE tag can be used to set the paper type that is used +# by the printer. Possible values are: a4, letter, legal and +# executive. If left blank a4wide will be used. + +PAPER_TYPE = a4wide + +# The EXTRA_PACKAGES tag can be to specify one or more names of LaTeX +# packages that should be included in the LaTeX output. + +EXTRA_PACKAGES = + +# The LATEX_HEADER tag can be used to specify a personal LaTeX header for +# the generated latex document. The header should contain everything until +# the first chapter. If it is left blank doxygen will generate a +# standard header. Notice: only use this tag if you know what you are doing! + +LATEX_HEADER = + +# The LATEX_FOOTER tag can be used to specify a personal LaTeX footer for +# the generated latex document. The footer should contain everything after +# the last chapter. If it is left blank doxygen will generate a +# standard footer. Notice: only use this tag if you know what you are doing! + +LATEX_FOOTER = + +# If the PDF_HYPERLINKS tag is set to YES, the LaTeX that is generated +# is prepared for conversion to pdf (using ps2pdf). The pdf file will +# contain links (just like the HTML output) instead of page references +# This makes the output suitable for online browsing using a pdf viewer. + +PDF_HYPERLINKS = YES + +# If the USE_PDFLATEX tag is set to YES, pdflatex will be used instead of +# plain latex in the generated Makefile. Set this option to YES to get a +# higher quality PDF documentation. + +USE_PDFLATEX = YES + +# If the LATEX_BATCHMODE tag is set to YES, doxygen will add the \\batchmode. +# command to the generated LaTeX files. This will instruct LaTeX to keep +# running if errors occur, instead of asking the user for help. +# This option is also used when generating formulas in HTML. + +LATEX_BATCHMODE = NO + +# If LATEX_HIDE_INDICES is set to YES then doxygen will not +# include the index chapters (such as File Index, Compound Index, etc.) +# in the output. + +LATEX_HIDE_INDICES = YES + +# If LATEX_SOURCE_CODE is set to YES then doxygen will include +# source code with syntax highlighting in the LaTeX output. +# Note that which sources are shown also depends on other settings +# such as SOURCE_BROWSER. + +LATEX_SOURCE_CODE = YES + +# The LATEX_BIB_STYLE tag can be used to specify the style to use for the +# bibliography, e.g. plainnat, or ieeetr. The default style is "plain". See +# http://en.wikipedia.org/wiki/BibTeX for more info. + +LATEX_BIB_STYLE = plain + +#--------------------------------------------------------------------------- +# configuration options related to the RTF output +#--------------------------------------------------------------------------- + +# If the GENERATE_RTF tag is set to YES Doxygen will generate RTF output +# The RTF output is optimized for Word 97 and may not look very pretty with +# other RTF readers or editors. + +GENERATE_RTF = NO + +# The RTF_OUTPUT tag is used to specify where the RTF docs will be put. +# If a relative path is entered the value of OUTPUT_DIRECTORY will be +# put in front of it. If left blank `rtf' will be used as the default path. + +RTF_OUTPUT = rtf + +# If the COMPACT_RTF tag is set to YES Doxygen generates more compact +# RTF documents. This may be useful for small projects and may help to +# save some trees in general. + +COMPACT_RTF = NO + +# If the RTF_HYPERLINKS tag is set to YES, the RTF that is generated +# will contain hyperlink fields. The RTF file will +# contain links (just like the HTML output) instead of page references. +# This makes the output suitable for online browsing using WORD or other +# programs which support those fields. +# Note: wordpad (write) and others do not support links. + +RTF_HYPERLINKS = NO + +# Load style sheet definitions from file. Syntax is similar to doxygen's +# config file, i.e. a series of assignments. You only have to provide +# replacements, missing definitions are set to their default value. + +RTF_STYLESHEET_FILE = + +# Set optional variables used in the generation of an rtf document. +# Syntax is similar to doxygen's config file. + +RTF_EXTENSIONS_FILE = + +#--------------------------------------------------------------------------- +# configuration options related to the man page output +#--------------------------------------------------------------------------- + +# If the GENERATE_MAN tag is set to YES (the default) Doxygen will +# generate man pages + +GENERATE_MAN = YES + +# The MAN_OUTPUT tag is used to specify where the man pages will be put. +# If a relative path is entered the value of OUTPUT_DIRECTORY will be +# put in front of it. If left blank `man' will be used as the default path. + +MAN_OUTPUT = man + +# The MAN_EXTENSION tag determines the extension that is added to +# the generated man pages (default is the subroutine's section .3) + +MAN_EXTENSION = .3 + +# If the MAN_LINKS tag is set to YES and Doxygen generates man output, +# then it will generate one additional man file for each entity +# documented in the real man page(s). These additional files +# only source the real man page, but without them the man command +# would be unable to find the correct page. The default is NO. + +MAN_LINKS = NO + +#--------------------------------------------------------------------------- +# configuration options related to the XML output +#--------------------------------------------------------------------------- + +# If the GENERATE_XML tag is set to YES Doxygen will +# generate an XML file that captures the structure of +# the code including all documentation. + +GENERATE_XML = NO + +#--------------------------------------------------------------------------- +# configuration options for the AutoGen Definitions output +#--------------------------------------------------------------------------- + +# If the GENERATE_AUTOGEN_DEF tag is set to YES Doxygen will +# generate an AutoGen Definitions (see autogen.sf.net) file +# that captures the structure of the code including all +# documentation. Note that this feature is still experimental +# and incomplete at the moment. + +GENERATE_AUTOGEN_DEF = NO + +#--------------------------------------------------------------------------- +# configuration options related to the Perl module output +#--------------------------------------------------------------------------- + +# If the GENERATE_PERLMOD tag is set to YES Doxygen will +# generate a Perl module file that captures the structure of +# the code including all documentation. Note that this +# feature is still experimental and incomplete at the +# moment. + +GENERATE_PERLMOD = NO + +# If the PERLMOD_LATEX tag is set to YES Doxygen will generate +# the necessary Makefile rules, Perl scripts and LaTeX code to be able +# to generate PDF and DVI output from the Perl module output. + +PERLMOD_LATEX = NO + +# If the PERLMOD_PRETTY tag is set to YES the Perl module output will be +# nicely formatted so it can be parsed by a human reader. This is useful +# if you want to understand what is going on. On the other hand, if this +# tag is set to NO the size of the Perl module output will be much smaller +# and Perl will parse it just the same. + +PERLMOD_PRETTY = YES + +# The names of the make variables in the generated doxyrules.make file +# are prefixed with the string contained in PERLMOD_MAKEVAR_PREFIX. +# This is useful so different doxyrules.make files included by the same +# Makefile don't overwrite each other's variables. + +PERLMOD_MAKEVAR_PREFIX = + +#--------------------------------------------------------------------------- +# Configuration options related to the preprocessor +#--------------------------------------------------------------------------- + +# If the ENABLE_PREPROCESSING tag is set to YES (the default) Doxygen will +# evaluate all C-preprocessor directives found in the sources and include +# files. + +ENABLE_PREPROCESSING = YES + +# If the MACRO_EXPANSION tag is set to YES Doxygen will expand all macro +# names in the source code. If set to NO (the default) only conditional +# compilation will be performed. Macro expansion can be done in a controlled +# way by setting EXPAND_ONLY_PREDEF to YES. + +MACRO_EXPANSION = YES + +# If the EXPAND_ONLY_PREDEF and MACRO_EXPANSION tags are both set to YES +# then the macro expansion is limited to the macros specified with the +# PREDEFINED and EXPAND_AS_DEFINED tags. + +EXPAND_ONLY_PREDEF = NO + +# If the SEARCH_INCLUDES tag is set to YES (the default) the includes files +# pointed to by INCLUDE_PATH will be searched when a #include is found. + +SEARCH_INCLUDES = YES + +# The INCLUDE_PATH tag can be used to specify one or more directories that +# contain include files that are not input files but should be processed by +# the preprocessor. + +INCLUDE_PATH = + +# You can use the INCLUDE_FILE_PATTERNS tag to specify one or more wildcard +# patterns (like *.h and *.hpp) to filter out the header-files in the +# directories. If left blank, the patterns specified with FILE_PATTERNS will +# be used. + +INCLUDE_FILE_PATTERNS = + +# The PREDEFINED tag can be used to specify one or more macro names that +# are defined before the preprocessor is started (similar to the -D option of +# gcc). The argument of the tag is a list of macros of the form: name +# or name=definition (no spaces). If the definition and the = are +# omitted =1 is assumed. To prevent a macro definition from being +# undefined via #undef or recursively expanded use the := operator +# instead of the = operator. + +PREDEFINED = DOXYGEN \ + __wt_async_callback:=WT_ASYNC_CALLBACK \ + __wt_async_op:=WT_ASYNC_OP \ + __wt_collator:=WT_COLLATOR \ + __wt_compressor:=WT_COMPRESSOR \ + __wt_config_arg:=WT_CONFIG_ARG \ + __wt_config_item:=WT_CONFIG_ITEM \ + __wt_config_parser:=WT_CONFIG_PARSER \ + __wt_config_scan:=WT_CONFIG_SCAN \ + __wt_connection:=WT_CONNECTION \ + __wt_cursor:=WT_CURSOR \ + __wt_data_source:=WT_DATA_SOURCE \ + __wt_encryptor:=WT_ENCRYPTOR \ + __wt_event_handler:=WT_EVENT_HANDLER \ + __wt_extension_api:=WT_EXTENSION_API \ + __wt_extractor:=WT_EXTRACTOR \ + __wt_item:=WT_ITEM \ + __wt_lsn:=WT_LSN \ + __wt_session:=WT_SESSION \ + __wt_txn_notify:=WT_TXN_NOTIFY \ + WT_HANDLE_CLOSED(x):=x \ + WT_HANDLE_NULLABLE(x):=x + +# If the MACRO_EXPANSION and EXPAND_ONLY_PREDEF tags are set to YES then +# this tag can be used to specify a list of macro names that should be expanded. +# The macro definition that is found in the sources will be used. +# Use the PREDEFINED tag if you want to use a different macro definition that +# overrules the definition found in the source code. + +EXPAND_AS_DEFINED = + +# If the SKIP_FUNCTION_MACROS tag is set to YES (the default) then +# doxygen's preprocessor will remove all references to function-like macros +# that are alone on a line, have an all uppercase name, and do not end with a +# semicolon, because these will confuse the parser if not removed. + +SKIP_FUNCTION_MACROS = YES + +#--------------------------------------------------------------------------- +# Configuration::additions related to external references +#--------------------------------------------------------------------------- + +# The TAGFILES option can be used to specify one or more tagfiles. For each +# tag file the location of the external documentation should be added. The +# format of a tag file without this location is as follows: +# TAGFILES = file1 file2 ... +# Adding location for the tag files is done as follows: +# TAGFILES = file1=loc1 "file2 = loc2" ... +# where "loc1" and "loc2" can be relative or absolute paths +# or URLs. Note that each tag file must have a unique name (where the name does +# NOT include the path). If a tag file is not located in the directory in which +# doxygen is run, you must also specify the path to the tagfile here. + +TAGFILES = + +# When a file name is specified after GENERATE_TAGFILE, doxygen will create +# a tag file that is based on the input files it reads. + +GENERATE_TAGFILE = + +# If the ALLEXTERNALS tag is set to YES all external classes will be listed +# in the class index. If set to NO only the inherited external classes +# will be listed. + +ALLEXTERNALS = NO + +# If the EXTERNAL_GROUPS tag is set to YES all external groups will be listed +# in the modules index. If set to NO, only the current project's groups will +# be listed. + +EXTERNAL_GROUPS = YES + +# The PERL_PATH should be the absolute path and name of the perl script +# interpreter (i.e. the result of `which perl'). + +PERL_PATH = /usr/bin/perl + +#--------------------------------------------------------------------------- +# Configuration options related to the dot tool +#--------------------------------------------------------------------------- + +# If the CLASS_DIAGRAMS tag is set to YES (the default) Doxygen will +# generate a inheritance diagram (in HTML, RTF and LaTeX) for classes with base +# or super classes. Setting the tag to NO turns the diagrams off. Note that +# this option also works with HAVE_DOT disabled, but it is recommended to +# install and use dot, since it yields more powerful graphs. + +CLASS_DIAGRAMS = NO + +# You can define message sequence charts within doxygen comments using the \msc +# command. Doxygen will then run the mscgen tool (see +# http://www.mcternan.me.uk/mscgen/) to produce the chart and insert it in the +# documentation. The MSCGEN_PATH tag allows you to specify the directory where +# the mscgen tool resides. If left empty the tool is assumed to be found in the +# default search path. + +MSCGEN_PATH = + +# If set to YES, the inheritance and collaboration graphs will hide +# inheritance and usage relations if the target is undocumented +# or is not a class. + +HIDE_UNDOC_RELATIONS = YES + +# If you set the HAVE_DOT tag to YES then doxygen will assume the dot tool is +# available from the path. This tool is part of Graphviz, a graph visualization +# toolkit from AT&T and Lucent Bell Labs. The other options in this section +# have no effect if this option is set to NO (the default) + +HAVE_DOT = NO + +# The DOT_NUM_THREADS specifies the number of dot invocations doxygen is +# allowed to run in parallel. When set to 0 (the default) doxygen will +# base this on the number of processors available in the system. You can set it +# explicitly to a value larger than 0 to get control over the balance +# between CPU load and processing speed. + +DOT_NUM_THREADS = 0 + +# By default doxygen will use the Helvetica font for all dot files that +# doxygen generates. When you want a differently looking font you can specify +# the font name using DOT_FONTNAME. You need to make sure dot is able to find +# the font, which can be done by putting it in a standard location or by setting +# the DOTFONTPATH environment variable or by setting DOT_FONTPATH to the +# directory containing the font. + +DOT_FONTNAME = + +# The DOT_FONTSIZE tag can be used to set the size of the font of dot graphs. +# The default size is 10pt. + +DOT_FONTSIZE = 10 + +# By default doxygen will tell dot to use the Helvetica font. +# If you specify a different font using DOT_FONTNAME you can use DOT_FONTPATH to +# set the path where dot can find it. + +DOT_FONTPATH = + +# If the CLASS_GRAPH and HAVE_DOT tags are set to YES then doxygen +# will generate a graph for each documented class showing the direct and +# indirect inheritance relations. Setting this tag to YES will force the +# CLASS_DIAGRAMS tag to NO. + +CLASS_GRAPH = YES + +# If the COLLABORATION_GRAPH and HAVE_DOT tags are set to YES then doxygen +# will generate a graph for each documented class showing the direct and +# indirect implementation dependencies (inheritance, containment, and +# class references variables) of the class with other documented classes. + +COLLABORATION_GRAPH = YES + +# If the GROUP_GRAPHS and HAVE_DOT tags are set to YES then doxygen +# will generate a graph for groups, showing the direct groups dependencies + +GROUP_GRAPHS = YES + +# If the UML_LOOK tag is set to YES doxygen will generate inheritance and +# collaboration diagrams in a style similar to the OMG's Unified Modeling +# Language. + +UML_LOOK = NO + +# If the UML_LOOK tag is enabled, the fields and methods are shown inside +# the class node. If there are many fields or methods and many nodes the +# graph may become too big to be useful. The UML_LIMIT_NUM_FIELDS +# threshold limits the number of items for each type to make the size more +# managable. Set this to 0 for no limit. Note that the threshold may be +# exceeded by 50% before the limit is enforced. + +UML_LIMIT_NUM_FIELDS = 10 + +# If set to YES, the inheritance and collaboration graphs will show the +# relations between templates and their instances. + +TEMPLATE_RELATIONS = NO + +# If the ENABLE_PREPROCESSING, SEARCH_INCLUDES, INCLUDE_GRAPH, and HAVE_DOT +# tags are set to YES then doxygen will generate a graph for each documented +# file showing the direct and indirect include dependencies of the file with +# other documented files. + +INCLUDE_GRAPH = YES + +# If the ENABLE_PREPROCESSING, SEARCH_INCLUDES, INCLUDED_BY_GRAPH, and +# HAVE_DOT tags are set to YES then doxygen will generate a graph for each +# documented header file showing the documented files that directly or +# indirectly include this file. + +INCLUDED_BY_GRAPH = YES + +# If the CALL_GRAPH and HAVE_DOT options are set to YES then +# doxygen will generate a call dependency graph for every global function +# or class method. Note that enabling this option will significantly increase +# the time of a run. So in most cases it will be better to enable call graphs +# for selected functions only using the \callgraph command. + +CALL_GRAPH = NO + +# If the CALLER_GRAPH and HAVE_DOT tags are set to YES then +# doxygen will generate a caller dependency graph for every global function +# or class method. Note that enabling this option will significantly increase +# the time of a run. So in most cases it will be better to enable caller +# graphs for selected functions only using the \callergraph command. + +CALLER_GRAPH = NO + +# If the GRAPHICAL_HIERARCHY and HAVE_DOT tags are set to YES then doxygen +# will generate a graphical hierarchy of all classes instead of a textual one. + +GRAPHICAL_HIERARCHY = YES + +# If the DIRECTORY_GRAPH and HAVE_DOT tags are set to YES +# then doxygen will show the dependencies a directory has on other directories +# in a graphical way. The dependency relations are determined by the #include +# relations between the files in the directories. + +DIRECTORY_GRAPH = YES + +# The DOT_IMAGE_FORMAT tag can be used to set the image format of the images +# generated by dot. Possible values are svg, png, jpg, or gif. +# If left blank png will be used. If you choose svg you need to set +# HTML_FILE_EXTENSION to xhtml in order to make the SVG files +# visible in IE 9+ (other browsers do not have this requirement). + +DOT_IMAGE_FORMAT = png + +# If DOT_IMAGE_FORMAT is set to svg, then this option can be set to YES to +# enable generation of interactive SVG images that allow zooming and panning. +# Note that this requires a modern browser other than Internet Explorer. +# Tested and working are Firefox, Chrome, Safari, and Opera. For IE 9+ you +# need to set HTML_FILE_EXTENSION to xhtml in order to make the SVG files +# visible. Older versions of IE do not have SVG support. + +INTERACTIVE_SVG = NO + +# The tag DOT_PATH can be used to specify the path where the dot tool can be +# found. If left blank, it is assumed the dot tool can be found in the path. + +DOT_PATH = + +# The DOTFILE_DIRS tag can be used to specify one or more directories that +# contain dot files that are included in the documentation (see the +# \dotfile command). + +DOTFILE_DIRS = + +# The MSCFILE_DIRS tag can be used to specify one or more directories that +# contain msc files that are included in the documentation (see the +# \mscfile command). + +MSCFILE_DIRS = + +# The DOT_GRAPH_MAX_NODES tag can be used to set the maximum number of +# nodes that will be shown in the graph. If the number of nodes in a graph +# becomes larger than this value, doxygen will truncate the graph, which is +# visualized by representing a node as a red box. Note that doxygen if the +# number of direct children of the root node in a graph is already larger than +# DOT_GRAPH_MAX_NODES then the graph will not be shown at all. Also note +# that the size of a graph can be further restricted by MAX_DOT_GRAPH_DEPTH. + +DOT_GRAPH_MAX_NODES = 50 + +# The MAX_DOT_GRAPH_DEPTH tag can be used to set the maximum depth of the +# graphs generated by dot. A depth value of 3 means that only nodes reachable +# from the root by following a path via at most 3 edges will be shown. Nodes +# that lay further from the root node will be omitted. Note that setting this +# option to 1 or 2 may greatly reduce the computation time needed for large +# code bases. Also note that the size of a graph can be further restricted by +# DOT_GRAPH_MAX_NODES. Using a depth of 0 means no depth restriction. + +MAX_DOT_GRAPH_DEPTH = 0 + +# Set the DOT_TRANSPARENT tag to YES to generate images with a transparent +# background. This is disabled by default, because dot on Windows does not +# seem to support this out of the box. Warning: Depending on the platform used, +# enabling this option may lead to badly anti-aliased labels on the edges of +# a graph (i.e. they become hard to read). + +DOT_TRANSPARENT = NO + +# Set the DOT_MULTI_TARGETS tag to YES allow dot to generate multiple output +# files in one run (i.e. multiple -o and -T options on the command line). This +# makes dot run faster, but since only newer versions of dot (>1.8.10) +# support this, this feature is disabled by default. + +DOT_MULTI_TARGETS = NO + +# If the GENERATE_LEGEND tag is set to YES (the default) Doxygen will +# generate a legend page explaining the meaning of the various boxes and +# arrows in the dot generated graphs. + +GENERATE_LEGEND = YES + +# If the DOT_CLEANUP tag is set to YES (the default) Doxygen will +# remove the intermediate dot files that are used to generate +# the various graphs. + +DOT_CLEANUP = YES diff --git a/src/third_party/wiredtiger/src/docs/Makefile b/src/third_party/wiredtiger/src/docs/Makefile new file mode 100644 index 00000000000..71173523a0a --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/Makefile @@ -0,0 +1,7 @@ +all: + @(cd ../../dist && sh s_docs -t) + +clean: + @(cd ../../dist && sh s_docs -a) + +.PHONY: all clean diff --git a/src/third_party/wiredtiger/src/docs/admin.dox b/src/third_party/wiredtiger/src/docs/admin.dox new file mode 100644 index 00000000000..d31911366ec --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/admin.dox @@ -0,0 +1,12 @@ +/*! @page admin Managing WiredTiger databases + +This section covers topics of interest for system administrators +managing WiredTiger applications: + +- @subpage home +- @subpage database_config +- @ref backup_util +- @subpage filesystems +- @subpage security + + */ diff --git a/src/third_party/wiredtiger/src/docs/architecture.dox b/src/third_party/wiredtiger/src/docs/architecture.dox new file mode 100644 index 00000000000..840910139c9 --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/architecture.dox @@ -0,0 +1,114 @@ +/*! @page architecture WiredTiger Architecture + +The WiredTiger data engine is a high performance, scalable, transactional, +production quality, open source, NoSQL data engine, created to maximize the +value of each computer you buy: + +- WiredTiger offers both low latency and high throughput (in-cache reads +require no latching, writes typically require a single latch), + +- WiredTiger handles data sets much larger than RAM without performance +or resource degradation, + +- WiredTiger has predictable behavior under heavy access and large +volumes of data, + +- WiredTiger offers transactional semantics without blocking, + +- WiredTiger stores are not corrupted by torn writes, reverting to the +last checkpoint after system failure, + +- WiredTiger supports petabyte tables, records up to 4GB, and record +numbers up to 64-bits. + +WiredTiger's design is focused on a few core principles: + +@section multi_core Multi-core scaling + +WiredTiger scales on modern, multi-CPU architectures. Using a variety of +programming techniques such as hazard pointers, lock-free algorithms, fast +latching and message passing, WiredTiger performs more work per CPU core than +alternative engines. + +WiredTiger's transactions use optimistic concurrency control algorithms that +avoid the bottleneck of a centralized lock manager. Transactional operations +in one thread do not block operations in other threads, but strong isolation is +provided and update conflicts are detected to preserve data consistency. + +@section cache Hot caches + +WiredTiger supports both row-oriented storage (where all columns of a +row are stored together), and column-oriented storage (where groups of +columns are stored in separate files), resulting in more efficient +memory use. When reading and writing column-stores, only the columns +required for any particular query are maintained in memory. +Column-store keys are derived from the value's location in the table +rather than being physically stored in the table, further minimizing +memory requirements. Finally, row-and column-stores can be +mixed-and-matched at the table level: for example, a row-store index can +be created on a column-store table. + +WiredTiger supports @ref lsm, where updates are buffered in small files +that fit in cache for fast random updates, then automatically merged into +larger files in the background so that read latency approaches that of +traditional Btree files. LSM trees automatically create Bloom filters to +avoid unnecessary reads from files that cannot containing matching keys. + +WiredTiger supports different-sized Btree internal and leaf pages in the +same file. Applications can maximize the amount of data transferred in +each I/O by configuring large leaf pages, and still minimize CPU cache +misses when searching the tree. + +WiredTiger supports key prefix compression and value dictionaries, +reducing the amount of memory keys and values require. + +WiredTiger supports static encoding with a configurable Huffman engine, +which typically reduces the amount of information maintained in memory +by 20-50%. + +@section io Making I/O more valuable + +WiredTiger uses compact file formats to minimize on-disk overhead. +WiredTiger does not store page content indexing information on disk, +instead, WiredTiger instantiates content indexing information either +when pages are read from disk or on demand. This simplifies the on-disk +file format and in the case of small key/value pairs, typically reduces +the amount of information written to disk by 20-50%. + +WiredTiger supports variable-length pages, meaning there is less wasted +space for large objects, and no need for compaction as pages grow and +shrink naturally when key/value pairs are inserted or deleted. + +WiredTiger supports block compression on table pages. Because +WiredTiger supports variable-length pages, pages do not have to shrink +by a fixed amount in order to benefit from block compression. Block +compression is selectable on a per-table basis, allowing applications +to choose the compression algorithm most appropriate for their data. +Block compression typically reduces the amount of information written +to disk by 30-80%. + +WiredTiger supports leaf pages of up to 512MB in size. Disk seeks are +less likely when reading large amounts of data from disk, significantly +improving table scan performance. + +Also, as noted in the @ref cache section, WiredTiger supports +column-store formats, prefix compression and static encoding. While +each of these features makes WiredTiger's use of memory more efficient, +they also maximize the amount of useful data transferred per disk I/O. + +@section quality Production quality + +WiredTiger is production quality, supported software, engineered for the +most demanding application environments. For example, as a no-overwrite +data engine, torn writes can never corrupt a WiredTiger data store. + +WiredTiger includes verification support so you can verify data sets, +and salvage support as a last-ditch protection: data can be retrieved +even if it somehow becomes corrupted. + +@section nosql NoSQL and Open Source + +WiredTiger is an Open Source, NoSQL data engine. See the @ref license +for details. + +*/ diff --git a/src/third_party/wiredtiger/src/docs/async.dox b/src/third_party/wiredtiger/src/docs/async.dox new file mode 100644 index 00000000000..fcc0e590369 --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/async.dox @@ -0,0 +1,135 @@ +/*! @m_page{{c,java},async,Asynchronous operations} + +WiredTiger supports asynchronous operations; as an example of where this +can be useful, a server application handling requests from a network as +fast as possible may want its worker threads to initiate a unit of work +and then immediately respond to the next request, rather than waiting +for the results of the first request. + +WiredTiger supports asynchronous operations through the WT_ASYNC_OP +handle. The work unit represented by the WT_ASYNC_OP handle is queued +by the application and performed by an internal WiredTiger worker +thread. When the work unit completes, the WiredTiger thread makes a +callback to notify the application the operation is finished, along with +providing any results and error values. + +The asynchronous operation handle operates in a manner similar to a +WT_CURSOR handle. An asynchronous operation includes: + +- getter/setters for key and value fields +- encoding of fields to store in the data source +- methods to modify and retrieve specific data (for example, insert and update) +- a method to compact a table + +The WT_ASYNC_OP handle does not survive after the callback function +returns into the WiredTiger library. When the application callback +returns the handle is returned to the system pool. The application +callback must copy out any key, value or other information that it needs +before the callback function returns. + +@section async_config Configuring asynchronous operations + +To perform asynchronous operations, the application must first include +the \c async configuration option when ::wiredtiger_open is called. +Additional configuration parameters include the number of WiredTiger +worker threads created to handle the incoming queue of operations and +the maximum number of simultaneous asynchronous operations that are +expected. + +For example, the following configures an application for asynchronous +operations, with a maximum of 10 asynchronous operations and 2 supporting +threads: + +@snippet ex_async.c async example connection + +If the number of requests exceeds the configured maximum number, a +WT_ASYNC_OP handle won't immediately be available and an error will be +returned to the application when it attempts to allocate a handle. If +the number of configured worker threads are unable to keep up with the +requests, requests will be forced to wait for worker threads to become +available. + +@section async_alloc Allocating an asynchronous operations handle + +A WT_ASYNC_OP handle is allocated using the WT_CONNECTION::async_new_op +method. This method takes an existing object URI and a callback. For +example: + +@snippet ex_async.c async handle allocation + +To aid the application in matching up an asynchronous operation with a +subsequent call to the callback function, every handle contains a unique +\c uint64_t identifier and ::WT_ASYNC_OPTYPE type. The \c identifier is +assigned when the handle is allocated and the \c type is assigned when +the asynchronous operation is queued. + +To retrieve the id, use the WT_ASYNC_OP::get_id method: + +@snippet ex_async.c async get identifier + +To retrieve the ::WT_ASYNC_OPTYPE type, use the WT_ASYNC_OP::get_type method: + +@snippet ex_async.c async get type + +WiredTiger only allows a limited number of method calls back into the +library using the WT_ASYNC_OP handle, while in the callback function. +The application is allowed to retrieve than handle's key, value, +identifier and the operation type from the WT_ASYNC_OP handle. + +Here is a complete example callback function implementation, from +the example program @ex_ref{ex_async.c}: + +@snippet ex_async.c async example callback implementation + +@m_if{java} +@snippet ex_async.c async example callback implementation part 2 +@m_endif + +@section async_operations Executing asynchronous operations + +The WT_ASYNC_OP handle behaves similarly to the WT_CURSOR handle, that +is, the key and value are initialized and then an operation is +performed. + +For example, the following code does an asynchronous insert into the +table: + +@snippet ex_async.c async insert + +For example, the following code does an asynchronous search of the +table: + +@snippet ex_async.c async search + +When a database contains multiple tables, it may be desired to compact +several tables in parallel without having to manage separate threads +to each call WT_SESSION::compact. Alternatively, compacting several +tables serially may take much longer. The WT_ASYNC_OP::compact method +allows the application to compact multiple objects asynchronously. + +@snippet ex_async.c async compaction + +@section async_flush Waiting for outstanding operations to complete + +The WT_CONNECTION::async_flush method can be used to wait for all +previous operations to complete. When that call returns, all previously +queued operations are guaranteed to have been completed and their +callback functions have returned. + +@snippet ex_async.c async flush + +Because WT_CONNECTION::close implicitly does a WT_CONNECTION::async_flush, +the call is not required in all applications. + +@section async_transactions Asynchronous operations and transactions + +Each asynchronous worker thread operates in its own session, executing +a single asynchronous operation with the context of the session's +transaction. Therefore, there is no way to combine multiple, related +updates into a single transaction when using asynchronous operations. + +The transaction is committed if the operation was successful and the +application callback returns success, otherwise the transaction is +rolled back. + +*/ diff --git a/src/third_party/wiredtiger/src/docs/backup.dox b/src/third_party/wiredtiger/src/docs/backup.dox new file mode 100644 index 00000000000..7742e698ac8 --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/backup.dox @@ -0,0 +1,135 @@ +/*! @class doc_bulk_durability + +Bulk-loads are not commit-level durable, that is, the creation and +bulk-load of an object will not appear in the database log files.\ For +this reason, applications doing incremental backups after a full backup +should repeat the full backup step after doing a bulk-load to make the +bulk-load durable.\ In addition, incremental backups after a bulk-load +can cause recovery to report errors because there are log records that +apply to data files which don't appear in the backup. + +*/ + +/*! @m_page{{c,java},backup,Backups} + +WiredTiger cursors provide access to data from a variety of sources. +One of these sources is the list of files required to perform a backup +of the database. The list may be the files required by all of the +objects in the database, or a subset of the objects in the database. + +WiredTiger backups are "on-line" or "hot" backups, and applications may +continue to read and write the databases while a snapshot is taken. + +@section backup_process Backup from an application + +1. Open a cursor on the \c "backup:" data source, which begins the + process of a backup. + +2. Copy each file returned by the WT_CURSOR::next method to the backup + location, for example, a different directory. Do not reuse backup + locations unless all files have first been removed from them, in + other words, remove any previous backup information before using a + backup location. + +3. Close the cursor; the cursor must not be closed until all of the + files have been copied. + +The directory into which the files are copied may subsequently be +specified as an directory to the ::wiredtiger_open function and +accessed as a WiredTiger database home. + +Copying the database files for a backup does not require any special +alignment or block size (specifically, Linux or Windows filesystems that +do not support read/write isolation can be safely read for backups). + +The cursor must not be closed until all of the files have been copied, +however, there is no requirement the files be copied in any order or in +any relationship to the WT_CURSOR::next calls, only that all files have +been copied before the cursor is closed. For example, applications might +aggregate the file names from the cursor and then list the file names as +arguments to a file archiver such as the system tar utility. + +During the period the backup cursor is open, database checkpoints can +be created, but no checkpoints can be deleted. This may result in +significant file growth. + +The following is a programmatic example of creating a backup: + +@snippet ex_all.c backup + +In cases where the backup is desired for a checkpoint other than the +most recent, applications can discard all checkpoints subsequent to the +checkpoint they want using the WT_SESSION::checkpoint method. For +example: + +@snippet ex_all.c backup of a checkpoint + +@section backup_util Backup from the command line + +The @ref_single util_backup command may also be used to create backups: + +@code +rm -rf /path/database.backup && + mkdir /path/database.backup && + wt -h /path/database.source backup /path/database.backup +@endcode + +@section backup_incremental Incremental backup + +Once a backup has been done, it can be rolled forward incrementally by +adding log files to the backup copy. Adding log files to the copy +decreases potential data loss from switching to the copy, but increases +the recovery time necessary to switch to the copy. To reset the +recovery time necessary to switch to the copy, perform a full backup of +the database. For example, an application might do a full backup of the +database once a week during a quiet period, and then incrementally copy +new log files into the backup directory for the rest of the week. +Incremental backups may also save time when the tables are very large. + +@copydoc doc_bulk_durability + +By default, WiredTiger automatically removes log files no longer +required for recovery. Applications wanting to use log files for +incremental backup must first disable automatic log file removal using +the \c log=(archive=false) configuration to ::wiredtiger_open. + +The following is the procedure for incrementally backing up a database +and removing log files from the original database home: + +1. Perform a full backup of the database (as described above). + +2. Perform a full database checkpoint. + +3. Open a cursor on the \c "backup:" data source, with the + \c "target=(\"log:\\")" target specified, which begins the + process of an incremental backup. + +4. Copy each log file returned by the WT_CURSOR::next method to the backup + directory. It is not an error to copy a log file which has been copied + before, but care should be taken to ensure each log file is completely copied + as the most recent log file may change in size while being copied. + +5. If all log files have been successfully copied, archive the log + files by calling the WT_SESSION::truncate method with the URI + <code>log:</code> and specifying the backup cursor as the + start cursor to that method. + +6. Close the backup cursor. + +Steps 2-6 can be repeated any number of times before step 1 is +repeated. These steps can be repeated as long as the backup database +directory has not been opened, recovery run and become live. Once +the database becomes live, you must repeat all steps 1-6 to another, +different backup database directory. + +@section backup_o_direct Backup and O_DIRECT + +Many Linux systems do not support mixing \c O_DIRECT and memory mapping +or normal I/O to the same file. If \c O_DIRECT is configured for data +or log files on Linux systems (using the wiredtiger_open \c direct_io +configuration), any program used to copy files during backup should also +specify \c O_DIRECT when configuring its file access. Likewise, when +\c O_DIRECT is not configured by the database application, programs +copying files should not configure \c O_DIRECT. + +*/ diff --git a/src/third_party/wiredtiger/src/docs/basic-api.dox b/src/third_party/wiredtiger/src/docs/basic-api.dox new file mode 100644 index 00000000000..2b810e6676b --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/basic-api.dox @@ -0,0 +1,126 @@ +/*! @m_page{{c,java},basic_api,Getting Started with the API} + +WiredTiger applications will generally use the following classes to access +and manage data: + + - a WT_CONNECTION represents a connection to a database. Most + applications will only open one connection to a database for each process. + All methods in WT_CONNECTION are thread safe. + + - a WT_SESSION represents a context in which database operations are + performed. Sessions are opened on a specified connection, and + applications must open a single session for each thread accessing the + database. + + - a WT_CURSOR represents a cursor over a collection of data. Cursors are + opened in the context of a session (which may have an associated + transaction), and can query and update records. In the common case, a + cursor is used to access records in a table. However, cursors can be used + on subsets of tables (such as a single column or a projection of multiple + columns), as an interface to statistics, configuration data or + application-specific data sources. + +Handles and operations are @ref config_strings "configured using strings", +which keeps the set of methods in the API relatively small and makes the +interface very similar regardless of the programming language used in the +application. WiredTiger supports the C, C++, Java and Python programming +languages (among others). + +By default, WiredTiger works as a traditional key/value store, where the +keys and values are +@m_if{c} +raw byte arrays accessed using a WT_ITEM structure. +@m_else +raw byte arrays. +@m_endif +Keys and values may be up to (4GB - 512B) bytes in size, but depending +on how WT_SESSION::create "maximum item sizes" are configured, +large key and value items will be stored on overflow pages. + +WiredTiger also supports a @ref schema "schema layer" so that keys and +values types can be chosen from a list, or composite keys or values made +up of columns with any combination of types. The size (4GB - 512B) byte +limit on keys and values still applies. + +All applications that use WiredTiger will be structured roughly as follows. +The code below is taken from the complete example program +@ex_ref{ex_access.c}. + +@section basic_connection Connecting to a database + +To access a database, first open a connection and create a session handle +for the single thread accessing the database: + +@snippet ex_access.c access example connection + +The configuration string @c "create" is passed to ::wiredtiger_open to +indicate the database should be created if it does not already exist. + +@m_if{c} +The code block above also shows simple error handling with +::wiredtiger_strerror (a function that returns a string describing an +error code passed as its argument). More complex error handling can be +configured by passing an implementation of WT_EVENT_HANDLER to +::wiredtiger_open or WT_CONNECTION::open_session. +@m_endif +@m_if{java} +The code block above also shows simple error handling by catching +WiredTigerException. +@m_endif + +@section basic_create_table Creating a table + +Create a table we can use to store data: + +@snippet ex_access.c access example table create + +This call creates a table called @c "access", configured to use strings +for its key and value columns. (See @ref schema for more information +on tables with other types of key and value columns.) + +@section basic_cursors Accessing data with cursors + +Now that we have a table, we open a cursor to perform some operations +on it: + +@snippet ex_access.c access example cursor open + +Here, the string @c "table:access" specifies that we are opening the +cursor on the table named @c "access". + +Then we insert a new row into the table. The WT_CURSOR::set_key and +WT_CURSOR::set_value calls put the application's key and value into +the cursor, respectively. The WT_CURSOR::insert call creates a +record containing that value and inserts it into the table. + +@snippet ex_access.c access example cursor insert + +Now we iterate through all of the records in the table, printing them out +as we go: + +@snippet ex_access.c access example cursor list + +Note that the key and value parts of the records are returned as C +strings because the table was created that way (even if it was created +by a previous run of the example). No data extraction or conversion is +required in the application. + +Because the cursor was positioned in the table after the WT_CURSOR::insert +call, we had to re-position it using the WT_CURSOR::first call; if we +weren't using the cursor for the call to WT_CURSOR::insert above, this loop +would simplify to: + +@code + while ((ret = cursor->next(cursor)) == 0) { + ... + } +@endcode + +@section basic_close Closing handles + +Lastly, we close the connection, which implicitly closes the cursor and +session handles: + +@snippet ex_access.c access example close + + */ diff --git a/src/third_party/wiredtiger/src/docs/build-javadoc.sh b/src/third_party/wiredtiger/src/docs/build-javadoc.sh new file mode 100755 index 00000000000..be886937070 --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/build-javadoc.sh @@ -0,0 +1,12 @@ +DOCS=`dirname $0` +TOP=$DOCS/.. +. $TOP/config.sh + +CLASSPATH=$THRIFT_HOME/libthrift.jar:$SLF4J_JAR javadoc -public -d $DOCS/java \ + -source 1.5 \ + -sourcepath $TOP/lang/java/src:$TOP/src/server/gen-java \ + -stylesheetfile $DOCS/style/javadoc.css \ + -use -link http://java.sun.com/j2se/1.5.0/docs/api/ \ + -header '<b>WiredTiger API</b><br><font size="-1"> version '$WT_VERSION'</font>' \ + -windowtitle 'WiredTiger Java API' -bottom '<font size=1>Copyright (c) 2008-2016 MongoDB, Inc. All rights reserved.</font>' \ + com.wiredtiger com.wiredtiger.util diff --git a/src/third_party/wiredtiger/src/docs/build-posix.dox b/src/third_party/wiredtiger/src/docs/build-posix.dox new file mode 100644 index 00000000000..f61b199bff2 --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/build-posix.dox @@ -0,0 +1,215 @@ +/*! @page build-posix Building and installing WiredTiger on POSIX + +@section posix_github Building using Git and GitHub + +Skip this step if you are building from a WiredTiger release package, +and proceed with @ref posix_building. + +To build from the WiredTiger GitHub repository requires +<a href="http://git-scm.com/">git</a>, +<a href="http://www.gnu.org/software/autoconf/autoconf.html">autoconf</a>, +<a href="http://www.gnu.org/software/autoconf/automake.html">automake</a>, +<a href="http://www.gnu.org/software/libtool/libtool.html">libtool</a> and +related tools. The standard options for those tools can be specified +when configuring and building WiredTiger. + +First, clone the repository: + +@code +git clone git://github.com/wiredtiger/wiredtiger.git +@endcode + +Second, run \c autogen.sh to create the \c configure script: + +@code +cd wiredtiger +sh autogen.sh +@endcode + +Now proceed with @ref posix_building. + +@section posix_building Building WiredTiger + +To build the WiredTiger software on a POSIX system, change directory to +the top-level directory, then configure and build the software: + +@code +cd wiredtiger +./configure && make +@endcode + +To rebuild from scratch, discard any previous configuration by cleaning +out the build area: + +@code +make distclean +@endcode + +To see additional configuration options, run: + +@code +./configure --help +@endcode + +WiredTiger uses +<a href="http://www.gnu.org/software/autoconf/autoconf.html">autoconf</a> +<a href="http://www.gnu.org/software/autoconf/automake.html">automake</a>, +and <a href="http://www.gnu.org/software/libtool/libtool.html">libtool</a> +to create the configure script and Makefiles. The standard options for those +tools can be specified when configuring and building WiredTiger. + +@section posix_installing Installing WiredTiger + +The WiredTiger software consists of a library and a single standalone +utility. + +WiredTiger's distribution follows the GNU Coding Standards installation +guidelines, and by default WiredTiger builds and installs four versions +of the library in <code>/usr/local/lib</code>. For example: + +@code +file /usr/local/lib/libwiredtiger* +/usr/local/lib/libwiredtiger-1.0.0.so: ELF 64-bit LSB shared object, x86-64, version 1 (FreeBSD), dynamically linked, not stripped +/usr/local/lib/libwiredtiger.a: current ar archive +/usr/local/lib/libwiredtiger.la: libtool library file +/usr/local/lib/libwiredtiger.so: symbolic link to `libwiredtiger-1.0.0.so' +@endcode + +WiredTiger uses +<a href="http://www.gnu.org/software/libtool/libtool.html">libtool</a> to +build the libraries. By default, both shared and static libraries are built. +To build only static libraries, configure WiredTiger using the +\c --disable-shared argument. To build only shared libraries, configure using +WiredTiger using the \c --disable-static argument. + +In addition, WiredTiger installs a standalone utility program named +<code>wt</code>. By default, this utility is installed in +<code>/usr/local/bin/wt</code>. + +To install WiredTiger: + +@code +make install +@endcode + +To uninstall WiredTiger: + +@code +make uninstall +@endcode + +To install WiredTiger's libraries or binaries into alternate locations, +use the configuration or installation options described in the +<a href="http://www.gnu.org/prep/standards/">GNU coding standards</a> +documentation. For example, to install the libraries and binaries into +a different location: + +@code +./configure --prefix=/c/wiredtiger +@endcode + +@section posix_configure Configuring WiredTiger + +The WiredTiger software supports some additional configuration options: + +@par \c --enable-attach +Configure WiredTiger to sleep and wait for a debugger to attach on failure. +<b>DO NOT</b> configure this option in production environments. + +@par \c --enable-bzip2 +Configure WiredTiger for <a href="http://www.bzip.org/">bzip2</a> +compression; see @ref compression for more information. + +@par \c --enable-diagnostic +Configure WiredTiger to perform various run-time diagnostic tests. +<b>DO NOT</b> configure this option in production environments. + +@par \c --enable-java +Build the WiredTiger Java API. + +@par \c --enable-leveldb[=basho, hyper, rocksdb] +Include a Google, Inc. compatible LevelDB API. Optionally, specify \c basho +to build a version of the LevelDB API compatible with Basho's Riak +product, specify \c hyper to build a version of the LevelDB API compatible +with the HyperDex product, and specify \c rocksdb to build a version of the +LevelDB API compatible with applications using Facebook's RocksDB +engine. See @ref leveldb for more information. + +@par \c --enable-lz4 +Configure WiredTiger for <a href="https://github.com/Cyan4973/lz4">LZ4</a> +compression; see @ref compression for more information. + +@par \c --enable-python +Build the WiredTiger <a href="http://www.python.org">Python</a> API. + +@par \c --enable-snappy +Configure WiredTiger for <a href="http://code.google.com/p/snappy/">snappy</a> +compression; see @ref compression for more information. + +@par \c --enable-verbose +Configure WiredTiger to support the \c verbose configuration string to +::wiredtiger_open. + +@par \c --enable-zlib +Configure WiredTiger for <a href="http://www.zlib.net/">zlib</a> +compression; see @ref compression for more information. + +@par <code>--with-builtins</code> +Configure WiredTiger to include support for extensions in the main library. +This avoids requiring additional libraries for supported extensions. Currently +supported options are \c lz4, \c snappy and \c zlib. + +@par <code>--with-python-prefix</code> +Configure WiredTiger to install Python libraries to a non-standard Python +install location. + +@par <code>--with-spinlock[=pthread, pthread_adaptive, gcc]</code> +Configure WiredTiger to use a specific mutex type for serialization; +options are \c pthread (the default, which configures WiredTiger to use +POSIX 1003.1c pthread mutexes), \c pthread_adaptive (which configures +WiredTiger to use POSIX 1003.1c pthread mutexes configured to be +adaptive (where that functionality is available), or \c gcc (which +configures WiredTiger to use gcc-based spinlocks). + +@section posix_compiler Changing compiler or loader options + +To change the compiler or loader behavior during the build, use the +<code>CC</code>, <code>CFLAGS</code>, <code>LDFLAGS</code>, or +<code>LIBS</code> environment variables: + +@par \c CC +The compiler. +@par \c CFLAGS +Compiler flags. +@par \c LDFLAGS +Loader flags. +@par \c LIBS +Additional libraries. + +For example, to specify a different compiler: + +@code +env CC=mygcc ./configure +@endcode + +By default, WiredTiger builds with the \c -O3 compiler optimization flag +unless the \c --enable-debug configuration option is specified, in which +case the \c -g compiler flag is used instead. For example, to specify +a different level of optimization: + +@code +env CFLAGS=-Os ./configure +@endcode + +To specify a different set of include files: + +@code +env CFLAGS=-I/usr/local/include ./configure +@endcode + +To specify an additional library: + +@code +env LIBS="-lrtf -lmin" LDFLAGS=-L/usr/local/lib ./configure +@endcode + */ diff --git a/src/third_party/wiredtiger/src/docs/build-pydoc.sh b/src/third_party/wiredtiger/src/docs/build-pydoc.sh new file mode 100755 index 00000000000..5e6e3635be5 --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/build-pydoc.sh @@ -0,0 +1,6 @@ +DOCS=`dirname $0` +TOP=$DOCS/.. +. $TOP/config.sh + +cd python +PYTHONPATH=../../lang/python/src:$THRIFT_HOME/lib/python2.6/site-packages pydoc -w wiredtiger diff --git a/src/third_party/wiredtiger/src/docs/build-windows.dox b/src/third_party/wiredtiger/src/docs/build-windows.dox new file mode 100644 index 00000000000..79df6b9712d --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/build-windows.dox @@ -0,0 +1,108 @@ +/*! @page build-windows Building and installing WiredTiger on Windows + +@section windows_github Building using Git and GitHub + +Skip this step if you are building from a WiredTiger release package, +and proceed with @ref windows_building + +First, clone the repository: + +@code +git clone git://github.com/wiredtiger/wiredtiger.git +@endcode + +Now proceed with @ref windows_building + +@section windows_building Building on Windows + +Building WiredTiger on Windows requires +<a href="http://www.scons.org">SCons</a> as well as the Microsoft Visual +C++ compiler in Microsoft Visual Studio. + +Change directory to the top-level directory, then build the software: + +@code +cd wiredtiger +scons +@endcode + +To rebuild from scratch, discard any previous configuration by cleaning +out the build area: + +@code +scons -c +@endcode + +To see additional configuration options, run: + +@code +scons --help +@endcode + +To build the python language support, a 64-bit version of Python is +required. The required version is referred to as x86-64 on +<a href="www.python.org">Python.org</a> or x64 on +<a href="http://www.activestate.com/activepython/downloads"> +ActiveState Python</a>. + +@section windows_installing Installing WiredTiger + +The WiredTiger software consists of a library and a single standalone +utility. + +WiredTiger's distribution follows the GNU Coding Standards installation +guidelines, and by default WiredTiger builds and installs a static library +and dll version of the library. + +@code +file <root directory>/package/bin/ +wt.exe: x64 standalone executable +wiredtiger.dll: x64 dynamically linked library + +file <root directory>/package/lib/ +libwiredtiger.lib: x64 static library +wiredtiger.lib: x64 import library for dll +@endcode + +To install WiredTiger: + +@code +scons install +@endcode + +To install WiredTiger's libraries or binaries into alternate locations, +you can use the --prefix configuration option. + +@code +scons --prefix=c:\wiredtiger install +@endcode + +@section windows_configure Configuring WiredTiger + +The WiredTiger software supports some additional configuration options: + +@par \c --enable-attach +Configure WiredTiger to sleep and wait for a debugger to attach on failure. +<b>DO NOT</b> configure this option in production environments. + +@par \c --enable-diagnostic +Configure WiredTiger to perform various run-time diagnostic tests. +<b>DO NOT</b> configure this option in production environments. + +@par \c --enable-python +Build the WiredTiger <a href="http://www.python.org">Python</a> API, +requires path to swig.exe on Windows. + +@par \c --enable-snappy +Configure WiredTiger for <a href="http://code.google.com/p/snappy/">snappy</a> +compression; see @ref compression for more information. + +@par \c --enable-verbose +Configure WiredTiger to support the \c verbose configuration string to +::wiredtiger_open. + +@par \c --enable-zlib +Configure WiredTiger for <a href="http://www.zlib.net/">zlib</a> +compression; see @ref compression for more information. + + */ diff --git a/src/third_party/wiredtiger/src/docs/checkpoint.dox b/src/third_party/wiredtiger/src/docs/checkpoint.dox new file mode 100644 index 00000000000..523c0887859 --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/checkpoint.dox @@ -0,0 +1,90 @@ +/*! @m_page{{c,java},checkpoint,Checkpoint durability} + +WiredTiger supports checkpoint durability by default, and optionally +commit-level durability when logging is enabled. In most applications, +commit-level durability impacts performance more than checkpoint +durability; checkpoints offer basic operation durability across +application or system failure without impacting performance (although +the creation of each checkpoint is a relatively heavy-weight operation). +See @ref durability for information on commit-level durability. + +Checkpoints with transactional logging enabled allow logging to archive +older log files (if configured to do so) while still retaining +commit-level durability and the ability to run recovery. + +A checkpoint is automatically created whenever a modified data source +is closed. Additionally, checkpoints of one or more data sources, or +the entire database, can be explicitly created using the +WT_SESSION::checkpoint method. Checkpoints can also be scheduled +periodically based on elapsed time or data size with the \c checkpoint +configuration to ::wiredtiger_open. + +All transactional updates committed before a checkpoint are made durable +by the checkpoint, therefore the frequency of checkpoints limits the +volume of data that may be lost due to application or system failure. + +When WiredTiger data sources are first opened, they are opened in the +state of the most recent checkpoint taken on the file, in other words, +updates after the most recent checkpoint will not appear in the data +source. If no checkpoint is found when the data source is opened, the +data source will appear empty. + +@section checkpoint_server Automatic checkpoints + +WiredTiger will automatically checkpoint the entire database when the +\c checkpoint configuration parameter to ::wiredtiger_open is set. When +this configuration is used, an internal server thread is created. + +The period between checkpoints can be defined either in seconds via \c +wait, or, if logging is enabled, as the number of bytes written to the log +since the last checkpoint via \c log_size, or both. If both periods are +defined then the checkpoint occurs as soon as either threshold has occurred +and both are reset once the checkpoint is complete. If using \c log_size, +we recommend that the size selected be a multiple of the log file size for +archiving purposes. + +@section checkpoint_cursors Checkpoint cursors + +Cursors are normally opened in the most recent version of a data source. +However, a checkpoint configuration string may be provided to +WT_SESSION::open_cursor, opening a read-only, static view of the data +source. This provides a limited form of time-travel, as the static view +is not changed by subsequent checkpoints, and will persist until the +checkpoint cursor is closed. + +@section checkpoint_naming Checkpoint naming + +Additionally, checkpoints that do not include LSM trees may optionally +be given names by the application. Checkpoints named by the application +persist until explicitly discarded or the application creates a new +checkpoint with the same name (which replaces the previous checkpoint +of that name). If the previous checkpoint cannot be replaced, either +because a cursor is reading from the previous checkpoint, or backups are +in progress, the checkpoint will fail. Because named checkpoints +persist until discarded or replaced, they can be used to periodically +snapshot data for later use. + +Internal checkpoints (that is, checkpoints not named by the application) +use the reserved name "WiredTigerCheckpoint". Applications can open the +most recent of these checkpoints by specifying "WiredTigerCheckpoint" +as the checkpoint name to WT_SESSION::open_cursor. Creating a new +internal checkpoint drops all previous internal checkpoints, if +possible; if a previous internal checkpoint cannot be dropped for any +reason, the checkpoint will ignore the previous checkpoint and continue. +Subsequent checkpoints will drop those ignored checkpoints when it +becomes possible. + +The \c -c option to the \c wt command line utility \c list command will +list a data source's checkpoints, with time stamp, in a human-readable +format. + +@section checkpoint_compaction Checkpoints and file compaction + +Checkpoints share file blocks, and dropping a checkpoint may or may not +make file blocks available for re-use, depending on whether the dropped +checkpoint contained the last reference to a file block. +Because checkpoints named by the application are not discarded until +explicitly discarded or replaced, they may prevent WT_SESSION::compact +from accomplishing anything due to shared file blocks. + + */ diff --git a/src/third_party/wiredtiger/src/docs/command-line.dox b/src/third_party/wiredtiger/src/docs/command-line.dox new file mode 100644 index 00000000000..e2b376d5e3f --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/command-line.dox @@ -0,0 +1,401 @@ +/*! @page command_line WiredTiger command line utility + +WiredTiger includes a command line utility, \c wt. + +@section util_global_synopsis SYNOPSIS +<code>wt [-LRVv] [-C config] [-E secretkey ] [-h directory] command [command-specific arguments]</code> + +@section util_global_description DESCRIPTION +The \c wt tool is a command-line utility that provides access to +various pieces of the WiredTiger functionality. + +@section util_global_options OPTIONS +There are several global options: + +@par <code>-C config</code> +Specify configuration strings for the ::wiredtiger_open function. +@par <code>-E secretkey</code> +Specify an encryption secret key for the ::wiredtiger_open function. +@par <code>-h directory</code> +Specify a database home directory. +@par <code>-L</code> +Forcibly turn off logging subsystem for debugging purposes. +@par <code>-R</code> +Run recovery if the underlying database is configured to do so. +@par <code>-V</code> +Display WiredTiger version and exit. +@par <code>-v</code> +Set verbose output. + +Unless otherwise described by a \c wt command, the \c wt tool exits zero +on success and non-zero on error. + +The \c wt tool supports several commands. If configured in the underlying +database, some commands will run recovery when opening the database. If +the user wants to force recovery on any command, use the \c -R option. +In general, commands that modify the database or tables will run recovery +by default and commands that only read data will not run recovery. + +<hr> +@section util_backup wt backup +Perform a backup of a database or set of data sources. + +The \c backup command performs a backup of the database, copying the +database files to a \c specified directory, which can be subsequently +opened as a WiredTiger database. See @ref backup for more information, +and @ref file_permissions for specifics on the copied file permissions. + +@subsection util_backup_synopsis Synopsis +<code>wt [-RVv] [-C config] [-E secretkey ] [-h directory] backup [-t uri] directory</code> + +@subsection util_backup_options Options +The following are command-specific options for the \c backup command: + +@par <code>-t uri</code> +By default, the \c backup command does a backup of the entire database; +the \c -t option changes the \c backup command to do a backup of only +the named data sources. + +<hr> +@section util_compact wt compact +Compact a table or file. + +The \c compact command attempts to rewrite the specified table or file +to consume less disk space. + +@subsection util_compact_synopsis Synopsis +<code>wt [-RVv] [-C config] [-E secretkey ] [-h directory] compact uri</code> + +@subsection util_compact_options Options +The \c compact command has no command-specific options. + +<hr> +@section util_create wt create +Create a table or file. + +The \c create command creates the specified \c uri with the specified +configuration. It is equivalent to a call to WT_SESSION::create with +the specified string arguments. + +@subsection util_create_synopsis Synopsis +<code>wt [-RVv] [-C config] [-E secretkey ] [-h directory] create [-c config] uri</code> + +@subsection util_create_options Options +The following are command-specific options for the \c create command: + +@par <code>-c</code> +Include a configuration string to be passed to WT_SESSION::create. + +<hr> +@section util_drop wt drop +Drop a table or file. + +The \c drop command drops the specified \c uri. It is equivalent to a +call to WT_SESSION::drop with the "force" configuration argument. + +@subsection util_drop_synopsis Synopsis +<code>wt [-RVv] [-C config] [-E secretkey ] [-h directory] drop uri</code> + +@subsection util_drop_options Options +The \c drop command has no command-specific options. + +<hr> +@section util_dump wt dump +Export data in a text format. + +The \c dump command outputs the specified table in a portable format +which can be re-loaded into a new table using the \c load command. + +See @subpage dump_formats for details of the dump file formats. + +@subsection util_dump_synopsis Synopsis +<code>wt [-RVv] [-C config] [-E secretkey ] [-h directory] dump [-jrx] [-c checkpoint] [-f output] uri</code> + +@subsection util_dump_options Options +The following are command-specific options for the \c dump command: + +@par <code>-c</code> +By default, the \c dump command opens the most recent version of the data +source; the \c -c option changes the \c dump command to open the named +checkpoint. + +@par <code>-f</code> +By default, the \c dump command output is written to the standard output; +the \c -f option re-directs the output to the specified file. + +@par <code>-j</code> +Dump in JSON (<a href="http://www.json.org">JavaScript Object Notation</a>) +format. + +@par <code>-r</code> +Dump in reverse order, from largest key to smallest. + +@par <code>-x</code> +Dump all characters in a hexadecimal encoding (the default is to leave +printable characters unencoded). + +<hr> +@section util_list wt list +List the tables and files in the database. + +By default, the \c list command prints out the tables and files stored in +the database. If a URI is specified as an argument, only information about +that data source is printed. + +@subsection util_list_synopsis Synopsis +<code>wt [-RVv] [-C config] [-E secretkey ] [-h directory] list [-cv] [uri]</code> + +@subsection util_list_options Options +The following are command-specific options for the \c list command: + +@par <code>-c</code> +If the \c -c option is specified, the data source's checkpoints are printed +in a human-readable format. + +@par <code>-v</code> +If the \c -v option is specified, the data source's complete schema table +value is printed. + +<hr> +@section util_load wt load +Load a table or file from dump output. + +The \c load command reads the standard input for data and loads it into +a table or file, creating the table or file if it does not yet exist. +The data should be the format produced by the \c dump command; see +@ref dump_formats for details. + +By default, if the table or file already exists, data in the file or +table will be overwritten by the new data (use the \c -n option to +make an attempt to overwrite existing data return an error). + +@subsection util_load_synopsis Synopsis +<code>wt [-RVv] [-C config] [-E secretkey ] [-h directory] load [-ajn] [-f input] [-r name] [uri configuration ...]</code> + +@subsection util_load_options Options +The following are command-specific options for the \c load command: + +@par <code>-a</code> +If the \c -a option is specified, record number keys in the input are +ignored and the data is appended to the data source and assigned new record +number keys. The \c -a option is only applicable when loading into a +column store. + +@par <code>-f</code> +By default, the \c load command reads from the standard input; the \c +-f option reads the input from the specified file. + +@par <code>-j</code> +Load input in the JSON (<a href="http://www.json.org">JavaScript Object +Notation</a>) format that was created by the <code>dump -j</code> command. + +@par <code>-n</code> +By default, input data will overwrite existing data where the key/value +pair already exists in the data source; the \c -n option causes the \c +load command to fail if there's an attempt to overwrite already existing +data. + +@par <code>-r</code> +By default, the \c load command uses the table or file name taken from the +input; the \c -r option renames the data source. + +Additionally, \c uri and \c configuration pairs may be specified to the +\c load command. These configuration pairs can be used to modify the +configuration values from the dump header passed to the WT_SESSION::create +call. + +The \c uri part of the configuration pair should match only one of the +objects being loaded, but may be a prefix of the object being matched. +For example, the following two sets of configuration pairs are +equivalent in the case of loading a single table named \c xxx. + +@code +table block_allocation=first +table:xxx block_allocation=first +@endcode + +It's an error, however, to specify a matching prefix that matches more +than a single object being loaded. + +Multiple \c configuration arguments may be specified. For example, the +following two sets of configuration pairs are equivalent: + +@code +table:xxx block_allocation=first,prefix_compress=false +table:xxx block_allocation=first table:xxx prefix_compress=false +@endcode + +<hr> +@section util_loadtext wt loadtext +Load text into a table or file. + +The \c loadtext command reads the standard input for text and loads it +into a table or file. The input data should be printable characters, +with newline delimiters for each key or value. + +The \c loadtext command does not create the file if it does not yet +exist. + +In the case of inserting values into a column-store table or file, each +value is appended to the table or file; in the case of inserting values +into a row-store table or file, lines are handled in pairs, where the +first line is the key and the second line is the value. If the +row-store table or file already exists, data in the table or file will +be overwritten by the new data. + +@subsection util_loadtext_synopsis Synopsis +<code>wt [-RVv] [-C config] [-E secretkey ] [-h directory] loadtext [-f input]</code> + +@subsection util_loadtext_options Options +The following are command-specific options for the \c loadtext command: + +@par <code>-f</code> +By default, the \c loadtext command reads from the standard input; the +\c -f option reads the input from the specified file. + +<hr> +@section util_printlog wt printlog +Display the database log. + +The \c printlog command outputs the database log. + +@subsection util_printlog_synopsis Synopsis +<code>wt [-RVv] [-C config] [-E secretkey ] [-h directory] printlog [-x] [-f output]</code> + +@subsection util_printlog_options Options +The following are command-specific options for the \c printlog command: + +@par <code>-f</code> +By default, the \c printlog command output is written to the standard +output; the \c -f option re-directs the output to the specified file. + +@par <code>-x</code> +Keys and value items in the log are printed in hex format in addition +to the default string format. + +<hr> +@section util_read wt read +Read records from a table or file. + +The \c read command prints out the records associated with the specified +keys from the specified data source. The data source must be configured +with string or record number keys and string values. + +The \c read command exits non-zero if a specified record is not found. + +@subsection util_read_synopsis Synopsis +<code>wt [-RVv] [-C config] [-E secretkey ] [-h directory] read uri key ...</code> + +@subsection util_read_options Options +The \c read command has no command-specific options. + +<hr> +@section util_rename wt rename +Rename a table or file. + +The \c rename command renames the specified table or file. + +@subsection util_rename_synopsis Synopsis +<code>wt [-RVv] [-C config] [-E secretkey ] [-h directory] rename uri name</code> + +@subsection util_rename_options Options +The \c rename command has no command-specific options. + +<hr> +@section util_salvage wt salvage +Recover data from a corrupted file. + +The \c salvage command salvages the specified data source, discarding any +data that cannot be recovered. Underlying files are re-written in +place, overwriting the original file contents. + +@subsection util_salvage_synopsis Synopsis +<code>wt [-RVv] [-C config] [-E secretkey ] [-h directory] salvage [-F force] uri</code> + +@subsection util_salvage_options Options +The following are command-specific options for the \c salvage command: + +@par <code>-F</code> +By default, salvage will refuse to salvage files that fail basic tests +(for example, files that don't appear to be in a WiredTiger format). +The \c -F option forces the salvage of the file, regardless. + +<hr> +@section util_stat wt stat +Display database or data source statistics. + +The \c stat command outputs run-time statistics for the WiredTiger +engine, or, if specified, for the URI on the command-line. + +@subsection util_stat_synopsis Synopsis +<code>wt [-RVv] [-C config] [-E secretkey ] [-h directory] stat [-f] [uri]</code> + +@subsection util_stat_options Options +The following are command-specific options for the \c stat command: + +@par <code>-f</code> +Include only "fast" statistics in the output (equivalent to passing +<code>statistics=(fast)</code>) to WT_SESSION::open_cursor. + +<hr> +@section util_upgrade wt upgrade +Upgrade a table or file. + +The \c upgrade command upgrades the specified table or file, exiting +success if the data source is up-to-date, and failure if the data source +cannot be upgraded. + +@subsection util_upgrade_synopsis Synopsis +<code>wt [-RVv] [-C config] [-E secretkey ] [-h directory] upgrade uri</code> + +@subsection util_upgrade_options Options +The \c upgrade command has no command-specific options. + +<hr> +@section util_verify wt verify +Check the structural integrity of a table or file. + +The \c verify command verifies the specified table or file, exiting +success if the data source is correct, and failure if the data source is +corrupted. + +@subsection util_verify_synopsis Synopsis +<code>wt [-RVv] [-C config] [-E secretkey ] [-h directory] verify uri</code> + +@subsection util_verify_options Options +The \c verify command has no command-specific options. + +<hr> +@section util_write wt write +Write records to a table or file. + +The \c write command stores records into the specified data source. +The data source must be configured with string or record number keys and +string values. + +If the \c write command is called with the \c -a option, each +command-line argument is a single value to be appended to the specified +column-store data source. If the \c write command is not called with +the \c -a option, the command-line arguments are key/value pairs. + +Attempting to overwrite an already existing record will fail. + +@subsection util_write_synopsis Synopsis +<code> +wt [-RVv] [-C config] [-E secretkey ] [-h directory] write -a uri value ... +<br> +wt [-RVv] [-C config] [-E secretkey ] [-h directory] write [-o] uri key value ... +</code> + +@subsection util_write_options Options +The following are command-specific options for the \c write command: + +@par <code>-a</code> +Append each value as a new record in the data source. + +@par <code>-o</code> +By default, attempting to overwrite an already existing record will +fail. The \c -o option changes \c write to overwrite previously +existing records. + +*/ diff --git a/src/third_party/wiredtiger/src/docs/community.dox b/src/third_party/wiredtiger/src/docs/community.dox new file mode 100644 index 00000000000..de51cbca535 --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/community.dox @@ -0,0 +1,23 @@ +/*! @page community WiredTiger community and contact information + +MongoDB Inc. and the WiredTiger community maintain and develop +<a href="https://github.com/wiredtiger"><b>WiredTiger on GitHub</b></a>, +and all contributors are welcome! + +All source code and revision histories for this software are available +in the <a href="https://github.com/wiredtiger/wiredtiger"><b>WiredTiger +source tree repository</b></a>. + +Please submit any feature suggestions and bug reports at +<a href="http://jira.mongodb.org/browse/WT"> +<b>WiredTiger's Issues page</b></a>. + +Discussion of WiredTiger issues and development can also be found in +the +<a href="http://groups.google.com/group/wiredtiger-users"><b>WiredTiger +Google Group</b></a>. + +To contact MongoDB, Inc. about WiredTiger, please send email to +<a mailto="info@wiredtiger.com"><b>info@wiredtiger.com</b></a>. + +*/ diff --git a/src/third_party/wiredtiger/src/docs/compact.dox b/src/third_party/wiredtiger/src/docs/compact.dox new file mode 100644 index 00000000000..9d37bf77214 --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/compact.dox @@ -0,0 +1,12 @@ +/*! @m_page{{c,java},compact,Compaction} + +The WT_SESSION::compact method can be used to compact data sources. + +The data source does not have to be quiescent, compaction may be +performed on an active data source. + +Because checkpoints named by the application are not discarded until +explicitly removed or replaced, they may prevent WT_SESSION::compact +from accomplishing anything. + + */ diff --git a/src/third_party/wiredtiger/src/docs/compression.dox b/src/third_party/wiredtiger/src/docs/compression.dox new file mode 100644 index 00000000000..56715e20752 --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/compression.dox @@ -0,0 +1,136 @@ +/*! @m_page{{c,java},compression,Compressors} + +This section explains how to configure WiredTiger's builtin support for +the bzip2, lz4, snappy and zlib compression engines. + +@section compression_bzip2 Using bzip2 compression + +To use the builtin support for +<a href="http://www.bzip.org/">Julian Seward's bzip2</a> +compression, first check that bzip2 is installed in include and library +directories searched by the compiler. Once bzip2 is installed, you can +enable bzip2 using the \c --enable-bzip2 option to configure. + +If bzip2 is installed in a location not normally searched by the +compiler toolchain, you'll need to modify the \c CPPFLAGS and \c LDFLAGS +to indicate these locations. For example, with the bzip2 includes and +libraries installed in \c /usr/local/include and \c /usr/local/lib, you +would run configure with the following additional arguments: + +@code +--enable-bzip2 CPPFLAGS="-I/usr/local/include" LDFLAGS="-L/usr/local/include" +@endcode + +When opening the WiredTiger database, load the bzip2 shared library as +an extension. For example, with the WiredTiger library installed in +\c /usr/local/lib, you would use the following extension: + +@snippet ex_all.c Configure bzip2 extension + +Finally, when creating the WiredTiger object, set \c block_compressor +to \c bzip2: + +@snippet ex_all.c Create a bzip2 compressed table + +@section compression_lz4 Using LZ4 compression + +To use the builtin support for Yann Collet's +<a href="https://github.com/Cyan4973/lz4">LZ4</a> +compression, first check that LZ4 is installed in include and library +directories searched by the compiler. Once LZ4 is installed, you can +enable LZ4 using the \c --enable-lz4 option to configure. + +If LZ4 is installed in a location not normally searched by the +compiler toolchain, you'll need to modify the \c CPPFLAGS and \c LDFLAGS +to indicate these locations. For example, with the LZ4 includes and +libraries installed in \c /usr/local/include and \c /usr/local/lib, you +would run configure with the following additional arguments: + +@code +--enable-lz4 CPPFLAGS="-I/usr/local/include" LDFLAGS="-L/usr/local/include" +@endcode + +When opening the WiredTiger database, load the LZ4 shared library as +an extension. For example, with the WiredTiger library installed in +\c /usr/local/lib, you would use the following extension: + +@snippet ex_all.c Configure lz4 extension + +Finally, when creating the WiredTiger object, set \c block_compressor +to \c lz4: + +@snippet ex_all.c Create a lz4 compressed table + +@section compression_snappy Using snappy compression + +To use the builtin support for +<a href="http://code.google.com/p/snappy/">Google's snappy</a> +compression, first check that snappy is installed in include and library +directories searched by the compiler. Once snappy is installed, you can +enable snappy using the \c --enable-snappy option to configure. + +If snappy is installed in a location not normally searched by the +compiler toolchain, you'll need to modify the \c CPPFLAGS and \c LDFLAGS +to indicate these locations. For example, with the snappy includes and +libraries installed in \c /usr/local/include and \c /usr/local/lib, you +would run configure with the following additional arguments: + +@code +--enable-snappy CPPFLAGS="-I/usr/local/include" LDFLAGS="-L/usr/local/include" +@endcode + +When opening the WiredTiger database, load the snappy shared library as +an extension. For example, with the WiredTiger library installed in +\c /usr/local/lib, you would use the following extension: + +@snippet ex_all.c Configure snappy extension + +Finally, when creating the WiredTiger object, set \c block_compressor +to \c snappy: + +@snippet ex_all.c Create a snappy compressed table + +@section compression_zlib Using zlib compression + +To use the builtin support for Greg Roelofs' and Mark Adler's +<a href="http://www.zlib.net/">zlib</a> +compression, first check that zlib is installed in include and library +directories searched by the compiler. Once zlib is installed, you can +enable zlib using the \c --enable-zlib option to configure. + +If zlib is installed in a location not normally searched by the compiler +toolchain, you'll need to modify the \c CPPFLAGS and \c LDFLAGS to +indicate these locations. For example, with the zlib includes and +libraries installed in \c /usr/local/include and \c /usr/local/lib, you +would run configure with the following additional arguments: + +@code +--enable-zlib CPPFLAGS="-I/usr/local/include" LDFLAGS="-L/usr/local/include" +@endcode + +When opening the WiredTiger database, load the zlib shared library as +an extension. For example, with the WiredTiger library installed in +\c /usr/local/lib, you would use the following extension: + +@snippet ex_all.c Configure zlib extension + +Finally, when creating the WiredTiger object, set \c block_compressor +to \c zlib: + +@snippet ex_all.c Create a zlib compressed table + +@section compression_upgrading Upgrading compression engines + +WiredTiger does not store information with file blocks to identify the +compression engine used to compressed the block. Applications wanting +to upgrade to some future compression engine (without requiring a file +dump and re-load), should ensure each compressed block includes enough +information to identify the compression engine used, so its compression +code can correctly decompress old and new blocks. + +@section compression_custom Custom compression engines + +WiredTiger may be extended by adding custom compression engines; see +WT_COMPRESSOR for more information. + + */ diff --git a/src/third_party/wiredtiger/src/docs/config-strings.dox b/src/third_party/wiredtiger/src/docs/config-strings.dox new file mode 100644 index 00000000000..a583573214f --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/config-strings.dox @@ -0,0 +1,127 @@ +/*! @m_page{{c,java},config_strings,Configuration Strings} + +@section config_intro Introduction + +Methods in WiredTiger take a configuration string to provide optional +arguments and configure non-standard behaviors. These strings are +simple comma-separated lists of <code>"<key>=<value>"</code> pairs, and +all have the same format: + +<pre> + [key['='value]][','[key['='value]]]* +</pre> + +Keys and values and values that consist of alphanumeric characters can be +specified directly. More precisely, keys or values that match this regular +expression do not require quotes: + +<pre> + [-_0-9A-Za-z./][^\\t\\r\\n :=,\\])}]* +</pre> + +Configuration strings are case-sensitive (for example, "true" is a +valid boolean value, but "True" is not). + +More complex keys and values can be specified by quoting them with double +quotes. + +For example, a configuration string is used when opening a connection +to a database to specify if the database should be created and to set +the cache size: + +@snippet ex_all.c Open a connection + +Configuration strings are also used to configure the table schema. For +example, to create a table that uses C language strings for keys and +values: + +@snippet ex_all.c Create a table + +To handle more complex schema configuration, such as specifying multiple +columns in a table, values are nested using parentheses. For example: + +@snippet ex_all.c Create a table with columns + +All types of parentheses are treated equivalently by the parser. + +When an integer values is expected, the value may have multiplier characters +appended, as follows: + +<table> +@hrow{Character, Meaning, Change to value} +@row{\c B or \c b, byte, no change} +@row{\c K or \c k, kilobyte, multiply by 2^10} +@row{\c M or \c m, megabyte, multiply by 2^20} +@row{\c G or \c g, gigabyte, multiply by 2^30} +@row{\c T or \c t, terabyte, multiply by 2^40} +@row{\c P or \c p, petabyte, multiply by 2^50} +</table> + +For example, the value \c 500B is the same as entering the number \c 500, the +value \c 500K is the same as \c 512000 and the string \c 500GB is the same as +\c 536870912000. + +Values of type of "boolean" can be set to any of \c false, \c true, \c 0 or +\c 1. If no value is specified for a key, the value \c 1 is implied. For +example, all of the following forms of the \c overwrite configuration string +are identical: + +@snippet ex_all.c boolean configuration string example + +Configuration strings are processed in order from left to right, with +later settings overriding earlier ones (unless multiple settings for a +key are supported by the method). + +Superfluous commas and whitespace in the configuration string are ignored +(including at the beginning and end of the string), so it is always safe to +combine two configuration strings by concatenating them with a comma in +between. + +@m_if{c} +Empty configuration strings may be represented in C or C++ by passing +<code>NULL</code>. +@m_endif +@m_if{java} +Empty configuration strings may be represented in Java by passing +<code>null</code>. +@m_endif + +@section config_json JavaScript Object Notation (JSON) compatibility + +WiredTiger configuration strings are compatible with +<a href="http://www.json.org/">JavaScript Object Notation (JSON)</a>, +and will accept additional formatting as follows: + +- parentheses may be round or square brackets or curly braces: + <code>'()'</code>, <code>'[]'</code> or <code>'{}'</code> +- the whole configuration string may optionally be wrapped in parentheses +- the key/value separator can be a colon: <code>':'</code> +- keys and values may be in double quotes: <code>"key" = "value"</code> +- quoted strings are interpreted as UTF-8 values + +The result of this relaxed parsing is that applications may pass strings +representing valid <a href="http://json.org/">JSON objects</a> wherever +configuration strings are required. + +For example, in Python, code might look as follows: + +\code + import json + config = json.dumps({ + "key_format" : "r", + "value_format" : "5sHQ", + "columns" : ("id", "country", "year", "population"), + "colgroup.population" : ["population"], + "index.country_year" : ["country", "year"] + }) +\endcode + +Because JSON compatibility allows colons to be used as key/value separators, +WiredTiger URIs may require quoting. For example, the following +WT_SESSION::checkpoint call specifies a set of URIs as checkpoint targets, +using double-quote characters to keep the parser from treating the colon +characters as JSON name/value separators: + +@snippet ex_all.c JSON quoting example + +*/ diff --git a/src/third_party/wiredtiger/src/docs/cursor-join.dox b/src/third_party/wiredtiger/src/docs/cursor-join.dox new file mode 100644 index 00000000000..51da6b174bf --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/cursor-join.dox @@ -0,0 +1,19 @@ +/*! @m_page{{c,java},cursor_join,Join cursors} + +Join cursors provide a way to iterate over a subset of a table, where the subset is specified by relationships with reference cursors. + +A join cursor is created with WT_SESSION::open_cursor using a \c +"join:table:<name>" URI prefix. Then reference cursors are positioned to +keys on indices and joined to the join cursor using WT_SESSION::join calls. +The result is a join cursor that can be iterated to satisfy the join +equation. + +Here is an example using join cursors: + +@snippet ex_schema.c Join cursors + +Joins support various comparison operators: \c "eq", \c "gt", \c "ge", \c "lt", \c "le". Ranges with lower and upper bounds can also be specified, by joining two cursors on the same index, for example, one with \c "compare=ge" and another \c "compare=lt". In addition to joining indices, the main table can be joined so that a range of primary keys can be specified. + +All the joins should be done on the join cursor before WT_CURSOR::next is called. Calling WT_CURSOR::next on a join cursor for the first time populates any bloom filters and performs other initialization. The join cursor's key is the primary key (the key for the main table), and its value is the entire set of values of the main table. A join cursor can be created with a projection by appending \c "(col1,col2,...)" to the URI if a different set of values is needed. + +*/ diff --git a/src/third_party/wiredtiger/src/docs/cursor-log.dox b/src/third_party/wiredtiger/src/docs/cursor-log.dox new file mode 100644 index 00000000000..fd247dd7105 --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/cursor-log.dox @@ -0,0 +1,114 @@ +/*! @m_page{{c,java},cursor_log,Log cursors} + +WiredTiger cursors provide access to data from a variety of sources, and +one of these sources is the records in the transaction log files. Log +files may not be present in every WiredTiger database, only databases +that have been configured for logging using the \c log configuration for +::wiredtiger_open. In databases with log files, a log cursor provides +access to the log records. Although log cursors are read-only, +applications may store records in the log using WT_SESSION::log_printf. + +Each physical WiredTiger log record represents one or more operations +in the database. When a log record represents more than a single +operation in the database, all of the operations in a log record will +be part of the same transaction, however, there is no corresponding +guarantee that all operations in a transaction must appear in the same +log record. + +The following examples are taken from the complete example program +@ex_ref{ex_log.c}. + +To open a log cursor on the database: + +@snippet ex_log.c log cursor open + +A log cursor's key is a unique log record identifier, plus a +@m_if{c} +uint32_t +@m_else +int +@m_endif +operation counter within that log record. When a log record maps +one-to-one to a transaction (in other words, the returned log record has +the only database operation the transaction made), the operation counter +returned for the key will be zero. + +The unique log record identifier maps to a WT_LSN data structure, which +has two fields: WT_LSN::id, the log file identifier, and WT_LSN::offset, +the offset of the log record in the log file. + +Here is an example of getting the log cursor's key: + +@snippet ex_log.c log cursor get_key + +The log cursor's value is comprised of six fields: + +@m_if{c} +- a \c uint64_t transaction ID (set for commit records only, otherwise 0), +- a \c uint32_t record type +- a \c uint32_t operation type (set for commit records only, otherwise 0) +- a \c uint32_t file id (if applicable, otherwise 0) +@m_else +- a \c long transaction ID (set for commit records only, otherwise 0), +- a \c int record type +- a \c int operation type (set for commit records only, otherwise 0) +- a \c int file id (if applicable, otherwise 0) +@m_endif +- the operation key (commit records only, otherwise empty) +- the operation value + +The transaction ID may not be unique across recovery, that is, closing +and reopening the database may result in transaction IDs smaller than +previously seen transaction IDs. + +The record and operation types are taken from @ref_single log_types; +typically, the only record or operation type applications are concerned with +is ::WT_LOGREC_MESSAGE, which is a log record generated by the application. + +The file ID may not be unique across recovery, that is, closing and +reopening the database may result in file IDs changing. Additionally, +there is currently no way to map file IDs to file names or higher-level +objects. + +Here is an example of getting the log cursor's value: + +@snippet ex_log.c log cursor get_value + +For clarity, imagine a set of three log records: + +- the first with a single operation, +- the second with five operations, +- the third with a single operation. + +The log cursor's WT_CURSOR::next call will return a total of seven +records. The first time the log cursor will return a key with a unique +log ID, a unique transaction ID, and an operation counter of 0. The +next five returns from the log cursor will have a common log ID, a +common transaction ID, and operation counters starting at 1 and ending +at 5. The final return from the log cursor will again have a unique log +ID, a unique transaction ID, and an operation counter of 0. + +Here's a more complete example of walking the file file and displaying +the results: + +@snippet ex_log.c log cursor walk + +The log cursor's key can be used to search for specific records in the +log (assuming the record still exists and has not been archived), by +setting the key and calling WT_CURSOR::search. However, it is not +possible to search for a specific operation within a log record, and the +key's operation counter is ignored when the key is set. The result of +a search for a log record with more than one operation is always the +first operation in the log record. + +Here is an example of setting the log cursor's key: + +@snippet ex_log.c log cursor set_key + +Log cursors are read-only, however applications can insert their own log +records using WT_SESSION::log_printf. Here is an example of adding an +application record into the database log: + +@snippet ex_log.c log cursor printf + +*/ diff --git a/src/third_party/wiredtiger/src/docs/cursor-ops.dox b/src/third_party/wiredtiger/src/docs/cursor-ops.dox new file mode 100644 index 00000000000..b743d81db57 --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/cursor-ops.dox @@ -0,0 +1,153 @@ +/*! @m_page{{c,java},cursor_ops,Cursor operations} + +Common operations in WiredTiger are performed using WT_CURSOR handles. +A cursor includes: + +- a position within a data source +- getter/setters for key and value fields +- encoding of fields to store in the data source +- methods to navigate within and iterate through the data + +@section cursor_opening Opening a cursor + +Cursors are created using the WT_SESSION::open_cursor method. For +example, from the program @ex_ref{ex_cursor.c}: + +@snippet ex_cursor.c open cursor #1 + +Another example from the same program: + +@snippet ex_cursor.c open cursor #2 + +In addition to traditional data sources, cursors in WiredTiger are used +to access projections and even created data sources such as the run-time +statistics: + +@snippet ex_cursor.c open cursor #3 + +See @ref cursors for more information on available cursor types. + +@section cursor_closing Closing a cursor + +Cursors remain open until either WT_CURSOR::close is called or the cursor's +session is closed, which may either be in WT_SESSION::close or +WT_CONNECTION::close. + +@section cursor_position Positioning a cursor + +Cursors may be positioned at the beginning of the data source, the end of +the data source, at an exact key within the data source, and near a key +within the data source. + +To invalidate the position of a cursor so that subsequent iterations start +from the beginning or end of the data source, use the WT_CURSOR::reset method: + +@snippet ex_cursor.c cursor reset + +To move a cursor forward in the data source, use the cursor WT_CURSOR::next +method: + +@snippet ex_cursor.c cursor next + +If the WT_CURSOR::next method is called on a cursor without a position +in the data source, it is positioned at the beginning of the data source. + +To move a cursor backward in the data source, use the cursor WT_CURSOR::prev +method: + +@snippet ex_cursor.c cursor prev + +If the WT_CURSOR::prev method is called on a cursor without a position +in the data source, it is positioned at the end of the data source. + +To position a cursor at a specific location in the data source, use the +WT_CURSOR::search method: + +@snippet ex_cursor.c cursor search + +To position a cursor at or near a location in the data source, use the +WT_CURSOR::search_near method: + +@snippet ex_cursor.c cursor search near + +Cursor positions do not survive transactions: cursors that are open during +WT_SESSION::begin_transaction, WT_SESSION::commit_transaction or +WT_SESSION::rollback_transaction will lose their position as if +WT_CURSOR::reset was called. + +Cursors can be configured to move to a random position with WT_CURSOR::next +is called, see @subpage cursor_random for details. + +@section cursor_writes Inserting and updating + +To insert new data, and optionally update existing data, using a cursor, +use the WT_CURSOR::insert method: + +@snippet ex_cursor.c cursor insert + +To update existing data using a cursor, use the WT_CURSOR::update method: + +@snippet ex_cursor.c cursor update + +To remove existing data using a cursor, use the WT_CURSOR::remove method: + +@snippet ex_cursor.c cursor remove + +The WT_SESSION::open_cursor \c overwrite configuration is \c true by default, +causing WT_CURSOR::insert, WT_CURSOR::remove and WT_CURSOR::update to ignore +the current state of the record, and these methods will succeed regardless of +whether or not the record previously exists. + +When an application configures \c overwrite to \c false, WT_CURSOR::insert will +fail with ::WT_DUPLICATE_KEY if the record previously exists, and +WT_CURSOR::update and WT_CURSOR::remove will fail with ::WT_NOTFOUND if the +record does not previously exist. + +@section cursor_error Cursor position after error + +After any cursor handle method failure, the cursor's position is +undetermined. For cursor operations that expect a key to be set before the +operation begins (including WT_CURSOR::search, WT_CURSOR::insert, +WT_CURSOR::update and WT_CURSOR::remove), the application's key and value +will not be cleared by an error. + +Applications that cannot re-position the cursor after failure must +duplicate the cursor by calling WT_SESSION::open_cursor and passing the +cursor as the \c to_dup parameter before calling a cursor method that will +attempt to re-position the cursor. Cursor duplication is not supported +for the backup, config and statistics cursor types. + +@section cursor_memory_scoping Cursor key/value memory scoping + +When applications pass a pointer (either to a WT_ITEM or a string), to +WT_CURSOR::set_key or WT_CURSOR::set_value, WiredTiger does not copy the +memory referenced by the pointer. For this reason, the application must +keep the referenced memory unchanged and valid until the next operation +that successfully positions the cursor, modifies the underlying data, +or the cursor is reset or closed (discarding its resources). The +operations that position the cursor are WT_CURSOR::next, WT_CURSOR::prev, +WT_CURSOR::search and WT_CURSOR::search_near; the operations that modify +the underlying data are WT_CURSOR::insert, WT_CURSOR::update and +WT_CURSOR::remove. + +If a cursor operation fails (for example, due to a ::WT_ROLLBACK error), +it may be retried without calling WT_CURSOR::set_key or +WT_CURSOR::set_value again. That is, the cursor may still reference the +application-supplied memory until the cursor is successfully positioned, +underlying data is modified, or the cursor is closed or reset. + +@m_if{c} +Any pointers returned by WT_CURSOR::get_key or WT_CURSOR::get_value are +only valid until a subsequent cursor call that successfully positions +the cursor, modifies the underlying data, or the cursor is reset or +closed. These pointers may reference private WiredTiger data structures +that may not be modified or freed by the application. If a longer scope +is required, the application must make a copy of the memory before the +cursor is re-used, closed or reset. + +The comments in this example code explain when the application can safely +modify memory passed to WT_CURSOR::set_key or WT_CURSOR::set_value: + +@snippet ex_scope.c cursor scope operation +@m_endif + */ diff --git a/src/third_party/wiredtiger/src/docs/cursor-random.dox b/src/third_party/wiredtiger/src/docs/cursor-random.dox new file mode 100644 index 00000000000..a0a3212be6d --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/cursor-random.dox @@ -0,0 +1,28 @@ +/*! @m_page{{c,java},cursor_random,Cursor random} + +The \c next_random configuration to the WT_SESSION::open_cursor method +configures the cursor to return a pseudo-random record from a row-store +object (the \c next_random configuration is not supported on other types +of objects). + +Applications should use the WT_CURSOR::next method to retrieve records +from the object, most other cursor methods are not supported. For +example, it's not possible to update using a cursor configured for +random retrieval. + +By default, each returned record is pseudo-randomly selected from the +underlying object as a whole. That can lead to skewed results when the +underlying tree structure is unbalanced or records are not uniformly +distributed. In such cases, the \c next_random_sample_size configuration +can also be specified. Setting \c next_random_sample_size configures the +number of samples the application expects to take using the cursor. A +cursor configured using \c next_random_sample_size divides the object +into \c next_random_sample_size pieces, and each subsequent retrieval +returns a record from the next one of those pieces. + +For example, setting \c next_random_sample_percent to \c 10 would cause +the cursor to sequentially return records from each tenth part of the +object. Setting \c next_random_sample_percent to \c 1000 would cause the +cursor to sequentially return records from each .1% of the object. + + */ diff --git a/src/third_party/wiredtiger/src/docs/cursors.dox b/src/third_party/wiredtiger/src/docs/cursors.dox new file mode 100644 index 00000000000..b6271951f91 --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/cursors.dox @@ -0,0 +1,144 @@ +/*! @m_page{{c,java},cursors,Cursors} + +Common operations in WiredTiger are performed using WT_CURSOR handles. +A cursor includes: + +- a position within a data source +- getter/setters for key and value fields +- encoding of fields to store in the data source +- methods to navigate within and iterate through the data + +See @subpage cursor_ops for a description of how to use cursors. + +@section cursor_types Cursor types + +@copydoc doc_cursor_types + +See the following for more details: + +- @subpage data_sources +- @ref metadata +- @ref cursor_log +- @ref cursor_join + +@section cursor_projections Projections + +Cursors on tables and indices can return a subset of columns. This is done by +listing the column names in parenthesis in the <code>uri</code> parameter to +WT_SESSION::open_cursor. Only the fields from the listed columns are returned +by WT_CURSOR::get_value. + +The @ex_ref{ex_schema.c} example creates a table where the value format is +\c "5sHq", where the initial string is the country, the short is a year, +and the long is a population. The following example lists just the country +and year columns from the table record values: + +@snippet ex_schema.c Return a subset of values from the table + +This is particularly useful with index cursors, because if all columns in +the projection are available in the index (including primary key columns, +which are the values of the index), the data can be read from the index +without accessing any column groups. See @ref schema_index_projections for +more information. + +@section cursors_transactions Cursors and Transactions + +If there is a transaction active in a session, cursors operate in the +context of that transaction. Reads performed while a transaction is +active inherit the isolation level of the transaction, and updates +performed within a transaction are made durable by calling +WT_SESSION::commit_transaction, or discarded by calling +WT_SESSION::rollback_transaction. + +If no transaction is active, cursor reads are performed at the isolation +level of the session, set with the \c isolation configuration key to +WT_CONNECTION::open_session and successful updates are automatically +committed before the update operation completes. + +Any operation that consists of multiple related updates should be +enclosed in an explicit transaction to ensure that the updates are +applied atomically. + +At \c read-committed (the default) or \c snapshot isolation levels, +committed changes from concurrent transactions become visible when no +cursor is positioned. In other words, at these isolation levels, all +cursors in a session read from a stable snapshot while any cursor in the +session remains positioned. + +Cursor positions survive transaction boundaries, unless a transaction +is rolled back. When a transaction is rolled-back either implicitly +or explicitly, all cursors in the session are reset as if the +WT_CURSOR::reset method was called, discarding any cursor position as +well as any key and value. + +See @ref transactions for more information. + +@section cursors_eviction Cursors and Eviction + +Cursor positions hold resources that can inhibit the eviction of memory +pages. If a cursor is active on a page being considered for eviction, +the eviction will defer until the cursor is moved or reset. To avoid +this and to keep resources freed in general, an application should call +WT_CURSOR::reset during times it does not need to keep the cursor +positioned. A cursor that has been reset is not active and will not +inhibit eviction. + +@section cursor_raw Raw mode + +Cursors can be configured for raw mode by specifying the \c "raw" config +keyword to WT_SESSION::open_cursor. In this mode, the methods +WT_CURSOR::get_key, WT_CURSOR::get_value, WT_CURSOR::set_key and +WT_CURSOR::set_value all take a single +@m_if{c} +WT_ITEM +@m_else +byte array +@m_endif +in the variable-length +argument list instead of a separate argument for each column. + +@m_if{c} +WT_ITEM structures do not need to be cleared before use. + +For WT_CURSOR::get_key and WT_CURSOR::get_value in raw mode, the WT_ITEM +can be split into columns by calling WT_EXTENSION_API::struct_unpack +with the cursor's \c key_format or \c value_format, respectively. For +WT_CURSOR::set_key and WT_CURSOR::set_value in raw mode, the WT_ITEM +should be equivalent to calling WT_EXTENSION_API::struct_pack for the +cursor's \c key_format or \c value_format, respectively. + +The @ex_ref{ex_schema.c} example creates a table where the value format is +\c "5sHq", where the initial string is the country, the short is a year, +and the long is a population. +@m_endif +The following example lists the table record +values, using raw mode: + +@snippet ex_schema.c List the records in the table using raw mode. + +Raw mode can be used in combination with projections. The following +example lists just the country and year columns from the table record +values, using raw mode: + +@snippet ex_schema.c Return a subset of values from the table using raw mode + +@section metadata Reading WiredTiger Metadata + +WiredTiger cursors provide access to data from a variety of sources. +One of these sources is the list of objects in the database. + +To retrieve the list of database objects, open a cursor on the uri +<code>metadata:</code>. Each returned key will be a database object and +each returned value will be the information stored in the metadata for +object named by the key. + +For example: + +@snippet ex_all.c Open a cursor on the metadata + +To retrieve value strings that are valid arguments for calls to +WT_SESSION::create, open a cursor on <code>metadata:create</code>. + +The metadata cursor is read-only, and the metadata cannot be modified. + +*/ diff --git a/src/third_party/wiredtiger/src/docs/custom-collators.dox b/src/third_party/wiredtiger/src/docs/custom-collators.dox new file mode 100644 index 00000000000..03dd0cf0ff0 --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/custom-collators.dox @@ -0,0 +1,55 @@ +/*! @page custom_collators Custom Collators + +@section custom_collators_intro Introduction to Custom Collators + +WiredTiger tables order records based on their keys. Cursors traverse +records in key order using WT_CURSOR::next, or in reverse order using +WT_CURSOR::prev. By default, WiredTiger uses lexicographic ordering, by +comparing the raw bytes of each key to determine their order. + +The built-in encoding of types (including integers and strings) is designed +to make lexicographic ordering match the natural ordering, including when +the key consists of multiple columns, each of which can be a different +type. + +Applications that need custom ordering of keys can either change the +serialized representation so that the lexicographic order matches the +required order, or implement the WT_COLLATOR interface to change the +comparison routine that WiredTiger uses. + +Applications must register their WT_COLLATOR implementations using +WT_CONNECTION::add_collator. They are then configured by passing +\c "collator=..." to WT_SESSION::create when creating a table or index. + +See @ex_ref{ex_extending.c} for more details about how to implement custom +collators. + +@section custom_collators_recovery Custom Collators and Recovery + +If logging is enabled, WiredTiger will run recovery as required in +::wiredtiger_open. Any custom collators in use must be registered before +recovery runs. This is described in more detail in +@ref extensions_recovery. + +@section custom_collators_indices Custom Collators for Indices + +Custom collators can be used with indices, but they must take into account +how WiredTiger indices are implemented. The primary key columns are +implicitly appended to the logical index key columns in order to create the +key that is stored in the index. This is done so that the key stored in +the index is unique even when multiple records have the same values for the +index key columns. + +A collator must give an unambiguous ordering to records in the index, so it +must use the primary key columns as well as the index columns when +comparing two index records. + +What this means in practice is that if the table has \c key_format=r and +the index is on a string column, then the index cursor will have +\c key_format=S, but the actual keys stored in the index will have +\c key_format=Sr (with the primary key appended). + +Collators usually call ::wiredtiger_struct_unpack with the appropriate +format to split the index key into fields that are used for the comparison. + +*/ diff --git a/src/third_party/wiredtiger/src/docs/custom-data-sources.dox b/src/third_party/wiredtiger/src/docs/custom-data-sources.dox new file mode 100644 index 00000000000..9082ee95dea --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/custom-data-sources.dox @@ -0,0 +1,242 @@ +/*! @page custom_data_sources Custom Data Sources + +Applications can implement their own custom data sources underneath +WiredTiger using the WT_DATA_SOURCE interface. Each data source should +support a set of methods for a different URI type (for example, in the +same way WiredTiger supports the built-in type "file:", an application +data source might support the type "dsrc:"). + +The WiredTiger distribution includes an example of a complete custom +data source implementation (based on Oracle's Berkeley DB database +engine), in the file test/format/kvs_bdb.c. This example implementation is +public domain software, please feel free to use this code as a prototype +implementation of other custom data sources. + +Applications register their WT_DATA_SOURCE interface implementations +with WiredTiger using the WT_CONNECTION::add_data_source method: + +@snippet ex_data_source.c WT_DATA_SOURCE register + +@section custom_ds_methods WT_DATA_SOURCE methods + +Any of the WT_DATA_SOURCE methods may be initialized to NULL. Calling +uninitialized WT_DATA_SOURCE methods through a WiredTiger API will +result in an "operation not supported" error. For example, an +underlying data source that does not support a compaction operation +should set the \c compact field of their WT_DATA_SOURCE structure to +NULL. + +@subsection custom_ds_create WT_DATA_SOURCE::create method + +When implementing a custom data source, you may want to consider the 'r' +key and 't' value formats, and whether or not they should be implemented. + +The 'r' key format indicates keys are \c uint64_t typed record numbers. +In this case, cursor methods will be passed and/or return a record +number in the \c recno field of the WT_CURSOR structure. + +Cursor methods for data sources supporting fixed-length bit field +column-stores (that is, a store with an 'r' type key and 't' type value) +should accept and/or return a single byte in the value field of the +WT_CURSOR structure, where between 1 and 8 of the low-order bits of +that byte contain the bit-field's value. For example, if the value +format is "3t", the key's value is the bottom 3 bits. + +@section custom_ds_cursor_methods WT_CURSOR methods + +The WT_DATA_SOURCE::open_cursor method is responsible for allocating and +returning a WT_CURSOR structure. The only fields of the WT_CURSOR +structure that should be initialized by WT_DATA_SOURCE::open_cursor are +a subset of the underlying methods it supports. + +The following methods of the WT_CURSOR structure must be initialized: +<ul> +<li>WT_CURSOR::close +<li>WT_CURSOR::next +<li>WT_CURSOR::prev +<li>WT_CURSOR::reset +<li>WT_CURSOR::search +<li>WT_CURSOR::search_near +<li>WT_CURSOR::insert +<li>WT_CURSOR::update +<li>WT_CURSOR::remove +</ul> + +No other methods of the WT_CURSOR structure should be initialized, for +example initializing WT_CURSOR::get_key or WT_CURSOR::set_key will have +no effect. + +Incorrectly configuring the WT_CURSOR methods will likely result in a core +dump. + +The data source's WT_CURSOR::close method is responsible for discarding +the allocated WT_CURSOR structure returned by WT_DATA_SOURCE::open_cursor. + +@subsection custom_ds_cursor_insert WT_CURSOR::insert method + +Custom data sources supporting column-stores (that is, a store with an +'r' type key) should consider the \c append configuration string +optionally specified for the WT_DATA_SOURCE::open_cursor method. The +\c append string configures WT_CURSOR::insert to allocate and return an +new record number key. + +Custom data sources should consider the \c overwrite configuration string +optionally specified for the WT_DATA_SOURCE::open_cursor method. If the +\c overwrite configuration is \c true, WT_CURSOR::insert, WT_CURSOR::remove +and WT_CURSOR::update should ignore the current state of the record, and +these methods will succeed regardless of whether or not the record previously +exists. + +When an application configures \c overwrite to \c false, WT_CURSOR::insert +should fail with ::WT_DUPLICATE_KEY if the record previously exists, and +WT_CURSOR::update and WT_CURSOR::remove will fail with ::WT_NOTFOUND if the +record does not previously exist. + +Custom data sources supporting fixed-length bit field column-stores (that +is, a store with an 'r' type key and 't' type value) may, but are not +required to, support the semantic that inserting a new record after the +current maximum record in a store implicitly creates the missing records +as records with a value of 0. + +@section custom_ds_cursor_fields WT_CURSOR key/value fields + +Custom data source cursor methods are expected to use the \c recno, +\c key and \c value fields of the WT_CURSOR handle. + +The \c recno field is a uint64_t type and is the record number set or +retrieved using the cursor when the data source was configured for +record number keys. + +The \c key and \c value fields are WT_ITEM structures. The \c key.data, +\c key.size, \c value.data and \c value.size fields are read by the +cursor methods storing items in the underlying data source and set by +the cursor methods retrieving items from the underlying data source. + +@section custom_ds_error_handling Error handling + +Some specific errors should be mapped to WiredTiger errors: +<ul> +<li> +attempts to insert an existing key should return WT_DUPLICATE_KEY +<li> +failure to find a key/value pair should return WT_NOTFOUND +<li> +fatal errors requiring the database restart should return WT_PANIC +</ul> + +Otherwise, successful return from custom data source methods should be +indicated by a return value of zero; error returns may be any value +other than zero or an error in WiredTiger's @ref error_handling. A simple +approach is to always return either a system error value (that is, \c +errno), or \c WT_ERROR. Error messages can be displayed using the +WT_SESSION::msg_printf method. For example: + +@snippet ex_data_source.c WT_DATA_SOURCE error message + +@section custom_ds_config Configuration strings + +@subsection custom_ds_config_parsing Parsing configuration strings + +Configuration information is provided to each WT_DATA_SOURCE method as an +argument. This configuration information can be parsed using the +WT_EXTENSION_API::config method, and is returned in a WT_CONFIG_ITEM +structure. + +For example, the \c append, \c overwrite \c key_format and \c value_format +configuration strings may be required for the WT_DATA_SOURCE::open_cursor +method to correctly configure itself. + +A \c key_format value of "r" indicates the data source is being configured +to use record numbers as keys. In this case initialized WT_CURSOR methods +must take their key value from the cursor's \c recno field, and not the +cursor's \c key field. (It is not required that record numbers be supported +by a custom data source, the WT_DATA_SOURCE::open_cursor method can return +an error if an application attempts to configure a data source with a +\c key_format of "r".) + +The WT_DATA_SOURCE::open_cursor method might retrieve the boolean or +integer value of a configuration string as follows: + +@snippet ex_data_source.c WT_EXTENSION_CONFIG boolean +@snippet ex_data_source.c WT_EXTENSION_CONFIG integer + +The WT_DATA_SOURCE::open_cursor method might retrieve the string value +of a configuration string as follows: + +@snippet ex_data_source.c WT_EXTENSION config_get + +@subsection custom_ds_config_add Creating data-specific configuration strings + +Applications can add their own configuration strings to WiredTiger +methods using WT_CONNECTION::configure_method. + +WT_CONNECTION::configure_method takes the following arguments: + +- the method being extended, where the name is the concatenation of the handle +name, a period and the method name. For example, \c "WT_SESSION.create" +would add new configuration strings to the WT_SESSION::create method, +and \c "WT_SESSION.open_cursor" would add the configuration string to the +WT_SESSION::open_cursor method. + +- the object type of the data source being extended. For example, \c "table:" +would extend the configuration arguments for table objects, and \c "my_data:" +could be used to extend the configuration arguments for a data source with +URIs beginning with the "my_data" prefix. A NULL value for the object type +implies all object types. + +- the additional configuration string, which consists of the name of the +configuration string and an optional default value. The default value +is specified by appending an equals sign and a value. For example, for +a new configuration string with the name \c "device", specifying either +\c "device=/path" or \c "device=35" would configure the default values. + +- the type of the additional configuration, which must one of \c "boolean", +\c "int", \c "list" or \c "string". + +- value checking information for the additional configuration, or NULL if +no checking information is provided. + +For example, an application might add new boolean, integer, list or +string type configuration strings as follows: + +@snippet ex_data_source.c WT_DATA_SOURCE configure boolean +@snippet ex_data_source.c WT_DATA_SOURCE configure integer +@snippet ex_data_source.c WT_DATA_SOURCE configure list +@snippet ex_data_source.c WT_DATA_SOURCE configure string + +Once these additional configuration calls have returned, application calls to +the WT_SESSION::open_cursor method could then include configuration strings +such as \c my_boolean=false, or \c my_integer=37, or \c my_source=/home. + +Additional checking information can be provided for \c int, \c list or +\c string type configuration strings. + +For integers, either or both of a maximum or minimum value can be +provided, so an error will result if the application attempts to set the +value outside of the acceptable range: + +@snippet ex_data_source.c WT_DATA_SOURCE configure integer with checking + +For lists and strings, a set of valid choices can also be provided, so +an error will result if the application attempts to set the value to a +string not listed as a valid choice: + +@snippet ex_data_source.c WT_DATA_SOURCE configure string with checking +@snippet ex_data_source.c WT_DATA_SOURCE configure list with checking + +@section custom_ds_cursor_collator WT_COLLATOR + +Custom data sources do not support custom ordering of records, and +attempting to create a custom data source with a collator configured +will fail. + +@section custom_data_source_cursor_serialize Serialization + +WiredTiger does not serialize calls to the WT_DATA_SOURCE methods or to +cursor methods configured by WT_DATA_SOURCE::open_cursor, and the +methods may be called from multiple threads concurrently. It is the +responsibility of the implementation to protect any shared data. For +example, object operations such as WT_DATA_SOURCE::drop might not be +permitted while there are open cursors for the WT_DATA_SOURCE object. + +*/ diff --git a/src/third_party/wiredtiger/src/docs/custom-extractors.dox b/src/third_party/wiredtiger/src/docs/custom-extractors.dox new file mode 100644 index 00000000000..31dfed94b75 --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/custom-extractors.dox @@ -0,0 +1,73 @@ +/*! @page custom_extractors Custom Extractors + +@section custom_extractors_intro Introduction to Custom Extractors + +A WiredTiger table can have zero or more associated indices. An index +uses a different key to locate records than the table, and usually only +stores a short key for each record, with the (larger) value in the +table. + +WiredTiger tables must be created with column names in order to create +an index. This is required so that index cursors can support +projections, and because WiredTiger optimizes some cases of "simple" +tables without column names. + +When the full schema of your records can be described in WiredTiger's +packing format, you can create an index by specifying which columns from +the record should appear in the index. However, for more complex +records, or to associate multiple index keys to each record, +applications can instead specify a custom extractor by implementing the +WT_EXTRACTOR interface. + +The main method in the interface is WT_EXTRACTOR::extract. This is +called by WiredTiger each time a record is updated in a table. The +\c extract method should determine the index key(s) and call +WT_CURSOR::set_key followed by WT_CURSOR::insert on the supplied +\c result_cursor for each index key. + +If any operation fails, WT_EXTRACTOR::extract must return the failure +to WiredTiger, or the index could become out of sync with the table. + +Note that the extract callback is called for all operations that update +the table, not just inserts. The callback sets the key and uses the +WT_CURSOR::insert method to return the index key(s). WiredTiger will +perform the required operation to keep the index in sync with the table. + +Applications must register their WT_EXTRACTOR implementations using +WT_CONNECTION::add_extractor. This is often done by creating a +@ref extensions "WiredTiger extension". They are then configured by +passing \c "extractor=..." to WT_SESSION::create when creating an index. + +See @ex_ref{ex_extractor.c} for an example of how to implement custom +extractors. + +@section custom_extractors_notes Implementation notes + +A WiredTiger index is a row store where the key columns contain all of +the secondary and primary key columns, but only the secondary key +columns are visible to applications. The value is empty, and +WiredTiger's on-disk format optimizes for this case (empty values take +up no space on disk). + +Custom extractors only need to calculate the public index key columns. +The \c result_cursor will be configured with a \c key_format +corresponding to what was supplied to WT_SESSION::create when the index +was created. WiredTiger will append the (hidden) primary key when +populating the index. + +If column names are specified for an index with a custom extractor, it +is not permitted to use any column names from the table key. Custom +index keys can include columns from the table value, but the extracted +value must be equal to the value from that column of the record or the +results of using a projection cursor on the index will be undefined. + +@section custom_extractors_raw Custom Collators in raw mode + +If a custom extractor needs to operate in raw mode on the +\c result_cursor, it must take into account an implementation detail. +To avoid rewriting the extracted key, WiredTiger appends a padding byte +to the raw key using a \c 'x' format. See @ref schema_format_types for +more information. If the callback operates in raw mode, it must also +append this padding byte. + +*/ diff --git a/src/third_party/wiredtiger/src/docs/data-sources.dox b/src/third_party/wiredtiger/src/docs/data-sources.dox new file mode 100644 index 00000000000..d09d1cbc1b8 --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/data-sources.dox @@ -0,0 +1,142 @@ +/*! @class doc_cursor_types + +The following are the builtin basic cursor types: +<table> +@hrow{URI, Type, Notes} +@row{<tt>table:\<table name\>[\<projection\>]</tt>, + table cursor, + table key\, table value(s) with optional projection of columns} +@row{<tt>colgroup:\<table name\>:\<column group name\></tt>, + column group cursor, + table key\, column group value(s)} +@row{<tt>index:\<table name\>:\<index name\>[\<projection\>]</tt>, + index cursor, + key=index key\, value=table value(s) with optional projection + of columns} +@row{<tt>join:table:\<table name\>[\<projection\>]</tt>, + join cursor, + key=table key\, value=table value(s) with optional projection + of columns} +</table> +Some administrative tasks can be accomplished using the following special +cursor types that give access to data managed by WiredTiger: +<table> +@hrow{URI, Type, Notes} +@row{<tt>backup:</tt>, + backup cursor, + key=<code>string</code>\, see @ref backup for details} +@row{<code>log:</code>, + log cursor, + key=<code>(long fileID\, long offset\, int seqno)</code>\,<br> + value=<code>(uint64_t txnid\, uint32_t rectype\,<br> + uint32_t optype\, uint32_t fileid\,<br> + WT_ITEM key\, WT_ITEM value)</code>\,<br> + see @ref cursor_log for details} +@row{<tt>metadata:[create]</tt>, + metadata cursor (optionally only returning configuration strings for + WT_SESSION::create if <code>create</code> is appended, + key=<code>string</code>\, value=<code>string</code>\,<br> + see @ref metadata for details} +@row{<tt>statistics:[\<data source URI\>]</tt>, + database or data source statistics cursor, + key=<code>int id</code>\,<br> + value=<code>(string description\, + string value\, uint64_t value)</code>\,<br> + see @ref data_statistics for details} +</table> +Advanced applications may also open the following low-level cursor types: +<table> +@hrow{URI, Type, Notes} +@row{<tt>file:\<file name\></tt>, + file cursor, + file key\, file value(s)} +@row{<tt>lsm:\<name\></tt>, + LSM cursor (key=LSM key\, value=LSM value), LSM key\, LSM value\,<br> + see @ref lsm} +</table> +*/ + +/* ----------------- */ + +/*! @m_page{{c,java},data_sources,Data Sources} + +WiredTiger provides access to data from a variety of sources. At the +lowest level, data may be stored in a file using a tree structure. A +relational schema supporting tables, indices and column groups is +layered on top of file. Additional sources include LSM trees and +statistics, and applications can further extend the supported types by +implementing the ::WT_DATA_SOURCE interface. + +Common operations on all data sources are performed using WT_CURSOR +handles. See @ref cursor_ops for a description of how to use +cursors. + +@section data_builtin Builtin data sources + +@copydoc doc_cursor_types + +@subsection data_files Raw Files + +WiredTiger's schema layer can be bypassed by opening cursors with a \c +"file:" URI, using the name of the underlying file. This can be useful for +seeing the contents of a column group or index without reading all of the +columns from the table. + +For example, if an index becomes inconsistent with its primary, a file +cursor can read from the index without errors (even though some of the keys +that are returned may not exist in the primary). + +@subsection data_indices Table Index data + +When an index is created for a table, records are inserted into the index +whenever the table is updated. These records use a different key to the +primary table, as specified when the index is created with the +WT_SESSION::create method. + +A cursor opened on an index has the specified index columns as its key, +accessed by WT_CURSOR::set_key and WT_CURSOR::get_key. The value columns +default to returning the value columns from the table, but this can be +overridden by configuring a projection cursor (see @ref cursor_projections), +which can access the table key columns or a subset of the value columns. + +@subsection data_statistics Statistics Data + +Statistics cursors can be used to retrieve run-time statistics about a +WiredTiger database as well as statistics for individual data sources. +The statistics are at two levels: per-database and per-individual data +source. Database-wide statistics are retrieved with the \c "statistics:" +URI; individual data source statistics are available by specifying +\c "statistics:<data source URI>". + +The statistic key is an integer from the list of keys in +@ref_single statistics_keys "Statistics Keys". Statistics cursors return +three values from the WT_CURSOR::get_value call: a printable description of +the statistic, a printable version of the entry's value, and the entry's +unsigned 64-bit integral value, respectively. + +The cursor's statistics values are loaded when the cursor is opened and +remain unchanged for the life of the cursor, unless the cursor is reset +by calling the WT_CURSOR::reset method, which reloads the values. + +The following is an example of printing run-time statistics about the +WiredTiger engine: + +@snippet ex_stat.c statistics database function + +The following is an example of printing statistics about a table: + +@snippet ex_stat.c statistics table function + +Both examples can use a common display routine that iterates through the +statistics until the cursor returns the end of the list. + +@snippet ex_stat.c statistics display function + +Individual statistics values can be retrieved by searching for the +corresponding key, as shown in the following example: + +@snippet ex_stat.c statistics retrieve by key + +See @ref_single tune_statistics for more examples of how statistics can be +used. +*/ diff --git a/src/third_party/wiredtiger/src/docs/database-config.dox b/src/third_party/wiredtiger/src/docs/database-config.dox new file mode 100644 index 00000000000..b8dee0be980 --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/database-config.dox @@ -0,0 +1,93 @@ +/*! @page database_config Database Configuration + +WiredTiger databases are configured using the ::wiredtiger_open call, +which takes a configuration string argument. In addition to this, the +database configuration can be specified using the \c WiredTiger.config +file or the WIREDTIGER_CONFIG environment variable. + +When a WiredTiger database is created, the configuration string passed +to ::wiredtiger_open is saved to a WiredTiger home directory file named +\c WiredTiger.basecfg, and that configuration file is read whenever the +database is subsequently opened. + +The configuration string to the ::wiredtiger_open call allows applications +to do per run settings (such as with or without statistics) or override +original creation settings. The user config file and the environment +variable allow system administrators to override application settings +without recompilation. + +@section config_order Configuration ordering + +When a database is created or opened, the order of configuration is: + +- Any \c WiredTiger.basecfg file, +- which is overridden by the ::wiredtiger_open configuration string argument. +- which is overridden by the \c WiredTiger.config file, +- which is overridden by the \c WIREDTIGER_CONFIG environment variable, + +@section config_env WIREDTIGER_CONFIG environment variable + +If the \c WIREDTIGER_CONFIG environment variable is set, it is read as +a configuration string. + +If the process is running with special privileges and \c WIREDTIGER_CONFIG +is set, the ::wiredtiger_open function must be configured with the +\c use_environment_priv flag, or the call to ::wiredtiger_open will fail. +See @ref home for more details. + +@section config_file WiredTiger.config file + +If a file named \c WiredTiger.config appears in the WiredTiger home +directory, it is read as a configuration string. + +The file is minimally parsed in order to build configuration strings for +the WiredTiger configuration parser: + +<ul> +<li>A backslash (<b><tt>\\</tt></b>) followed by any character other +than a newline character leaves both characters untouched; otherwise, +if a backslash is followed by a newline character, both the backslash +and the newline character are discarded. + +<li>Any text between double-quote pairs (<b><tt>"</tt></b>) is left +untouched, including newline and white-space characters. Backslash +characters escape double-quote characters: a backslash escaped +double-quote character can neither start nor end a quoted string. + +<li>Comments are discarded. If the first non-white-space character +following an unquoted and unescaped newline character is a hash mark +(<b><tt>#</tt></b>), all characters up to the next newline character are +discarded. The ending newline character cannot be escaped or quoted, +once a comment line is identified, all characters to the next newline +are simply discarded. + +<li>Otherwise, all lines are concatenated and the newline characters +replaced with commas. +</ul> + +@section config_base WiredTiger.basecfg file + +When a WiredTiger database is created, non-default configuration +information specified to ::wiredtiger_open is saved to a WiredTiger home +directory file named \c WiredTiger.basecfg, and that configuration file +is read whenever the database is subsequently opened. + +For example, if a shared library extensions are required, those settings +will be saved to the base configuration file so it's easier to write +general-purpose applications that might not be aware of how a particular +database was created and configured. + +Users should not not edit this file, set the \c WIREDTIGER_CONFIG +environment variable or create a \c WiredTiger.config file instead, as +those settings override the settings in the base configuration file. + +This file is a potential problem for applications that want to retain +the ability to downgrade to previous releases. If a new database is +created after an upgrade, the base configuration file might have +configuration strings that would cause a previous version of the +application to fail. For this reason, applications that may want to +downgrade should configure the ::wiredtiger_open value \c config_base +to \c false so the base configuration file is not written as part of +database create. + +*/ diff --git a/src/third_party/wiredtiger/src/docs/dump-formats.dox b/src/third_party/wiredtiger/src/docs/dump-formats.dox new file mode 100644 index 00000000000..2a903f9c790 --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/dump-formats.dox @@ -0,0 +1,104 @@ +/*! @page dump_formats Dump Formats + +The @ref util_dump command produces a flat-text representation of a +table that can be loaded by @ref util_load. This page describes the +output formats of the @ref util_dump command. + +@section dump_formats_json JSON dump format + +JSON (<a href="http://www.json.org">JavaScript Object Notation</a>) +dump files use the standard JSON data-interchange format to specify +the objects, and may be interpreted by any JSON reader. + +The format is a JSON object where each key is the URI passed to +WT_SESSION::create and the corresponding value is a JSON array of two +entries. The first entry in this array is a JSON object composed of +configuration information: the "config" key has the configuration +string used with WT_SESSION::create, the "colgroups" and "indices" +keys have values that are arrays of objects that are in turn composed +of configuration information. The second entry is a JSON array, with +each entry an object representing a row of data. If the columns were +named in the configuration string used with WT_SESSION::create, those +names are used for keys, otherwise predictable names (for example, +"key0", "value0", "value1") are generated. The values in this object +are the values for each column in the record. + +Here is some sample output: + +@code +{ + "table:planets" : [ + { + "config" : "columns=(id,name,distance),key_format=i,value_format=Si", + "colgroups" : [], + "indices" : [ + { + "uri" : "index:planets:names", + "config" : "columns=(name),key_format=Si,source=\"file:astronomy.wt\",type=file" + } + ] + }, + [ + { +"id" : 1, +"name" : "Mercury", +"distance" : 57910000 + }, + { +"id" : 2, +"name" : "Venus", +"distance" : 108200000 + }, + ... + ] + ] +} +@endcode + +@section dump_formats_text Text dump format + +Text dump files have three parts, a prefix, a header and a body. + +The dump prefix includes basic information about the dump including the +WiredTiger version that created the dump and the dump format. The dump +format consists of a line beginning with \c "Format=", and contains the +following information: + +<table> +@hrow{String, Meaning} +@row{hex, the dumped data is in a hexadecimal dump format} +@row{print, the dumped data is in a printable format} +</table> + +The dump header follows a single \c "Header" line in the file and +consists of paired key and value lines, where the key is the URI passed +to WT_SESSION::create and the value is corresponding configuration +string. The table or file can be recreated by calling +WT_SESSION::create for each pair of lines in the header. + +The dump body follows a single \c "Data" line in the file and consists +of a text representation of the records in the table. Each record is a +represented by a pair of lines: the first line is the key and the second +line is the value. These lines are encoded in one of two formats: a +printable format and a hexadecimal format. + +The printable format consists of literal printable characters, and +hexadecimal encoded non-printable characters. Encoded characters are +written as three separate characters: a backslash character followed by +two hexadecimal characters (first the high nibble and then the low +nibble). For example, a newline character in the ASCII character set +would be encoded as \c "\0a" and an escape character would be encoded +as \c "\1b". Backslash characters which do not precede a hexadecimal +encoding are paired, that is, the characters \c "\\" should be +interpreted as a single backslash character. + +The hexadecimal format consists of encoded characters, where each +literal character is written as a pair of characters (first the +high-nibble and then the low-nibble). For example, "0a" would be an +ASCII newline character and "1b" would be an ASCII escape character. + +Because the definition of "printable" may depend on the application's +locale, dump files in the printable output format may be less portable +than dump files in the hexadecimal output format. + + */ diff --git a/src/third_party/wiredtiger/src/docs/durability.dox b/src/third_party/wiredtiger/src/docs/durability.dox new file mode 100644 index 00000000000..30683335978 --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/durability.dox @@ -0,0 +1,98 @@ +/*! @m_page{{c,java},durability,Commit-level durability} + +WiredTiger supports checkpoint durability by default, and optionally +commit-level durability when logging is enabled. In most applications, +commit-level durability impacts performance more than checkpoint +durability; checkpoints offer basic operation durability across +application or system failure without impacting performance (although +the creation of each checkpoint is a relatively heavy-weight operation). +See @ref checkpoint for information on checkpoint durability. + +Commit-level durability is implemented using a write-ahead log and is +enabled using the \c log=(enabled) configuration to ::wiredtiger_open. +When logging is enabled, WiredTiger writes records to the log for each +transaction. + +Transactions define which updates are made durable together; see +@ref transactions for details. By default, log records are buffered in memory +when written by WT_SESSION::commit_transaction. See +@ref durability_group_commit for information about durability options. + +When the transactional log is enabled, calling ::wiredtiger_open +automatically performs a recovery step when opening the database that +applies whatever changes from the log are required to bring the database +up to date with the most recent transactional state. This recovery step +may require extensions be available when it runs (for example, collators +and compression). Therefore, applications doing recovery must configure +extensions with the \c extensions keyword to ::wiredtiger_open consistently +whenever re-opening the database. + +Recovery is required after the failure of any thread of control in the +application, where the failed thread might have been executing inside +of the WiredTiger library or open WiredTiger handles have been lost. In +most applications, if any thread of control exits unexpectedly, the +application will close and re-open the database. + +@section durability_checkpoint Checkpoints + +Checkpoints of the database should still be performed periodically when +commit-level durability is configured, either explicitly from the +application or periodically based on elapsed time or data size with the +\c checkpoint configuration to ::wiredtiger_open. + +Database checkpoints are necessary for two reasons: First, log files can +only be archived after a checkpoint completes, and so the frequency of +checkpoints determines the disk space required by log files. Second, +checkpoints bound the time required for recovery to complete after +application or system failure by limiting the log records that need to +be processed. + +@section durability_backup Backups + +With logging enabled, partial backups (backups where not all of the +database objects are copied), may result in error messages during +recovery, because data files referenced in the logs might not be found. +Applications should either copy all objects and log files if +commit-level durability of the copied database is required, or +alternatively, copy only selected objects when backing up and not copy +log files at all, then fall back to checkpoint durability when switching +to the backup. + +@section durability_bulk Bulk loads + +@copydoc doc_bulk_durability + +@section durability_archiving Log file archival + +WiredTiger log files are named "WiredTigerLog.[number]" where "[number]" +is a 10-digit value, for example, WiredTigerLog.0000000001". The log +file with the largest number in its name is the most recent log file +written. The log file size can be set using the \c log configuration +to ::wiredtiger_open. + +By default, WiredTiger automatically removes log files no longer +required for recovery. Applications wanting to archive log files +instead must disable log file removal using the \c log=(archive=false) +configuration to ::wiredtiger_open. + +Log files may be removed or archived after a checkpoint has completed, +as long as there's not a backup in progress. Immediately after the checkpoint +has completed, only the most recent log file is needed for recovery, and all +other log files can be removed or archived. Note that there must always +be at least one log file for the database. + +Open log cursors prevents WiredTiger from automatically removing log files. +Therefore, we recommend proactively closing log cursors when done with them. +Applications manually removing log files should take care that no log cursors +are opened in the log when removing files or errors may occur when trying +to read a log record in a file that was removed. + +@section durability_tuning Tuning commit-level durability + +@subsection durability_group_commit Group commit +@copydoc doc_tune_durability_group_commit + +@subsection durability_flush_config Flush call configuration +@copydoc doc_tune_durability_flush_config + + */ diff --git a/src/third_party/wiredtiger/src/docs/encryption.dox b/src/third_party/wiredtiger/src/docs/encryption.dox new file mode 100644 index 00000000000..e3051cf25e9 --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/encryption.dox @@ -0,0 +1,124 @@ +/*! @m_page{{c,java},encryption,Encryptors} + +@section encryption_overview Overview of Encryption in WiredTiger +@ref encryption_custom "Custom encryption engines" may be used to +extend WiredTiger. WiredTiger does not currently offer builtin +support for any particular encryption algorithm. +@ref encryption_examples "Example encryption code" is provided +to demonstrate how encryption extensions are created. + +\warning The encryption infrastructure included in WiredTiger, when used with a +strong encryption algorithm, is intended to protect data stored in files +(that is, <em>encryption at rest</em>). The table content (keys, values), +the metadata pertaining to data (table, index, column names, and other +configuration information) as well as the database log files are encrypted +on disk. Decryption occurs when the data is read into memory; thus an +attacker having the ability to directly read system memory will have access +to unencrypted data. Many systems may also page memory to a backing disk +under load. Access to any such \em paging or \em swap devices must be +considered when planning the security of a system. + +The encryption extension must be loaded in the ::wiredtiger_open call. +See @subpage_single extensions for details on how extensions are loaded. +Also, encryption is specified using \c encryption= in the configuration +for the ::wiredtiger_open call. This configuration establishes +the encryption algorithm and keys to be used for database log files and a subset +of the WiredTiger metadata files. By default, this same encryption +is also used for all data files. We call this the <em>system</em> encryption. + +It is also possible to use different encryption options when individual +data files are first created, using the \c encryption= configuration in +the WT_SESSION::create call. Such options override the default +(<em>system</em>) encryption that was indicated in the ::wiredtiger_open +call for the individual data file. It is possible to turn encryption off +for individual files, to use a different encryptor, or to specify a +different \c keyid. + +Overriding the system encryption for a table does not override +the system encryption for indices on that table, nor does it override +the system encryption for column groups specified on that table. +Encryption for column groups and indices must specified when they +are created, if they are to be different than the system encryption. + +It is an error to specify encryption in a WT_SESSION::create call when it +was not specified in the ::wiredtiger_open call. This prevents accidental +exposure of the file's data in log files, which would be written in the +clear in such a scenario. + +\warning When using separate keys for individual data files or tables, the +key used for the \em system encryption continues to have fundamental +importance. The database log, protected by the \em system encryption, +contains a shared stream of changes to all data files. Thus, if the \em +system key is exposed, even when per-file keys are not exposed, an attacker +can read database log files, and hence has access to data in individual +files. + +@section encryption_parameters Encryption keyid and secretkey + +Two parameters, \c keyid and \c secretkey, may be specified +when configuring encryption for ::wiredtiger_open to allow the possibility +of varying the algorithm according to different keys. + +The configuration parameter <code>encryption=(keyid=<em>identifier</em>)</code> +may be used in ::wiredtiger_open or WT_SESSION::create calls. This is intended +to reference a key stored using a Key Management Solution (KMS). +The \c keyid given to ::wiredtiger_open is stored in the clear in +WiredTiger configuration files; it should never contain sensitive +information. As an example, with a \c keyid of \c "customerABC", +the encryptor would consult the KMS to return a key previously stored +for \c "customerABC". The encryptor will use the returned key when +applying the encryption. To effectively use the \c keyid, a custom +encryptor must implement the WT_ENCRYPTOR::customize callback. +It is during \c customize that the encryptor has an opportunity to use +the \c keyid to fetch the actual key. The \c customize function is +called on the first use of a \c keyid, and the same \em customized encryptor +will be used with each use of the same \c keyid. + +The configuration parameter <code>encryption=(secretkey)</code> is used +only in the ::wiredtiger_open call. The value of the secretkey +is never stored on disk in any form, so it must always be provided +when WiredTiger is reopened (again, with the ::wiredtiger_open call). +The secretkey is available to the encryptor during the +WT_ENCRYPTOR::customize callback, during which the encryptor may +be \em customized to keep the secretkey or a transformation of it +for use during the WT_ENCRYPTOR::encrypt and WT_ENCRYPTOR::decrypt callbacks. + +If a \c secretkey is used, it must be provided using the \c -E option +when using the \c wt utility. Specifying \c keyid is not needed with the \c wt +utility, as the \c keyid is stored in the clear on disk by WiredTiger. +Any additional \c keyid values needed to decrypt data files +are stored in WiredTiger metadata using the system encryptor. + +@section encryption_custom Custom encryption engines + +WiredTiger may be extended by adding custom encryption engines +that we call \em encryptors. Custom encryptors must be coded in the C language. +Once packaged, they can be used in any language. + +See @subpage_single extensions for general details on extending WiredTiger, +and see WT_ENCRYPTOR for the encryptor interface. + +Custom encryptors are registered by calling WT_CONNECTION::add_encryptor, +this creates an encryptor name that may be referenced using the +<code>encryption=(name=...</code> configuration string in the ::wiredtiger_open +or WT_SESSION::create call. + +@section encryption_examples Encryption examples + +There are two kinds of example code with overlapping functionality. +A simple, self contained encryption example is in @ex_ref{ex_encrypt.c}. +This example includes a small encryptor that rotates letters in the +alphabet by a fixed amount, based on the value of \c keyid. This example +also shows how encryption is configured within an application. +The second set of examples are in \c ext/encryptors. These are +encryptors only (no application level code), showing how encryptors +might be packaged in a loadable shared library. +@ex_ref{nop_encrypt.c} merely copies its input to its output. +@ex_ref{rotn_encrypt.c} is an extended version of the example that +rotates the alphabet. It adds a twist in that a \c secretkey can +be specified, changing the rotation per letter. The Python test suite +uses the rotn encryptor to help test the encryption framework. + +Please note that these samples are for demonstration use only. +They do not provide any security. + */ diff --git a/src/third_party/wiredtiger/src/docs/error-handling.dox b/src/third_party/wiredtiger/src/docs/error-handling.dox new file mode 100644 index 00000000000..d1291e38ff0 --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/error-handling.dox @@ -0,0 +1,68 @@ +/*! @m_page{{c,java},error_handling,Error handling} + +WiredTiger operations return a value of 0 on success and a non-zero +value on error. Error codes may be either positive or negative: +positive error codes are standard error codes as described for +POSIX-like systems (for example, EINVAL or EBUSY), negative error codes +are WiredTiger-specific (for example, WT_ROLLBACK). + +WiredTiger-specific error codes always appear in the -31,800 to -31,999 +range. + +@m_if{java} +Informational return values, like <code>wiredtiger.WT_NOTFOUND</code> +or <code>wiredtiger.WT_DUPLICATE_KEY</code> or 0 (success), +are directly returned by APIs. More severe errors +are thrown as \c WiredTigerException, which may be caught by the +application. + +The \c WiredTigerRollbackException is a specific type of \c WiredTigerException, +it is thrown when there is a conflict between concurrent operations. +An application that catches this exception should call rollback() on +the relevant transaction, and retry as necessary. + +The \c WiredTigerPanicException is a specific type of \c WiredTigerException, +it is thrown when there is an underlying problem that requires the +application to exit and restart. + +The following is a complete list of possible WiredTiger-specific +return values, all constants defined in the com.wiredtiger.db.wiredtiger class: +@m_else +The following is a list of possible WiredTiger-specific errors: +@m_endif + +@if IGNORE_BUILT_BY_API_ERR_BEGIN +@endif + +@par <code>WT_ROLLBACK</code> +This error is generated when an operation cannot be completed due to a conflict with concurrent operations. The operation may be retried; if a transaction is in progress, it should be rolled back and the operation retried in a new transaction. + +@par <code>WT_DUPLICATE_KEY</code> +This error is generated when the application attempts to insert a record with the same key as an existing record without the 'overwrite' configuration to WT_SESSION::open_cursor. + +@par <code>WT_ERROR</code> +This error is returned when an error is not covered by a specific error return. + +@par <code>WT_NOTFOUND</code> +This error indicates an operation did not find a value to return. This includes cursor search and other operations where no record matched the cursor's search key such as WT_CURSOR::update or WT_CURSOR::remove. + +@par <code>WT_PANIC</code> +This error indicates an underlying problem that requires the application exit and restart. The application can exit immediately when \c WT_PANIC is returned from a WiredTiger interface, no further WiredTiger calls are required. + +@par <code>WT_RUN_RECOVERY</code> +This error is generated when wiredtiger_open is configured to return an error if recovery is required to use the database. + +@if IGNORE_BUILT_BY_API_ERR_END +@endif + +The ::wiredtiger_strerror function returns the standard message +associated with any WiredTiger, ISO C99, or POSIX 1003.1-2001 function: + +@snippet ex_all.c Display an error + +@m_if{c} +More complex error handling can be configured by passing an implementation +of WT_EVENT_HANDLER to ::wiredtiger_open or WT_CONNECTION::open_session. +@m_endif + +*/ diff --git a/src/third_party/wiredtiger/src/docs/examples.dox b/src/third_party/wiredtiger/src/docs/examples.dox new file mode 100644 index 00000000000..3ed7357b52c --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/examples.dox @@ -0,0 +1,58 @@ +/*! +@example ex_access.c +Create, insert and access a simple table. + +@example ex_async.c +Show how to configure and use asynchronous operations. + +@example ex_call_center.c +A more complex schema based on a call center example, showing how to map +some SQL constructs onto the WiredTiger API. + +@example ex_config.c +Shows how to configure some properties of the database and tables. + +@example ex_cursor.c +Shows some common cursor types and operations. + +@example ex_encrypt.c +Shows how to extend WiredTiger with a simple encryption algorithm. + +@example nop_encrypt.c +Shows the basic framework for building an encryptor as a plug in library. + +@example rotn_encrypt.c +Shows a simple encryptor as a plug in library. + +@example ex_extending.c +Shows how to extend WiredTiger with application-specific collations, +extractors and cursor types. + +@example ex_extractor.c +Shows how to extend WiredTiger with a more complex custom extractor. + +@example ex_hello.c +This is an example of how to create and open a database. + +@example ex_pack.c +Shows basic packing and unpacking of fields. + +@cond sharing +@notyet{inter-process sharing} +@example ex_process.c +Shows how to connect to a database from multiple processes. +@endcond + +@example ex_schema.c +Shows how to create column-oriented data and access individual columns. + +@example ex_stat.c +Shows how to access database and table statistics. + +@example ex_log.c +Shows how to access the database log files. + +@example ex_thread.c +Shows how to access a database with multiple threads. + + */ diff --git a/src/third_party/wiredtiger/src/docs/extensions.dox b/src/third_party/wiredtiger/src/docs/extensions.dox new file mode 100644 index 00000000000..e6754eb93e8 --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/extensions.dox @@ -0,0 +1,60 @@ +/*! @page extensions Extending WiredTiger + +WiredTiger can be extended in various ways, including: + +- adding @ref compression_custom, @ref encryption_custom, +- @ref custom_data_sources, and +- @ref WT_COLLATOR "by implementing the WT_COLLATOR interface". +- @ref WT_EXTRACTOR "by implementing the WT_EXTRACTOR interface". + +Code that implements these interfaces can use the @ref wt_ext. + +@section extensions_loadable Loadable extensions + +Loadable extensions, also known as modules, are libraries of code that can +be loaded at runtime to add functionality to WiredTiger. To build a +loadable extension, use <code>gcc -shared ...</code> or +<code>libtool -module ...</code>, or the equivalent for your system. + +Extensions can be loaded on an open connection by calling +WT_CONNECTION::load_extension. + +Extensions can be loaded during ::wiredtiger_open by passing the +\c extensions configuration when the database is created. When used this +way, the extensions will be loaded each time the database is opened, +whether by your application or the @ref command_line. + +The extension entry point, which defaults to ::wiredtiger_extension_init, +is called for each loadable module. Applications must supply this entry +point, which in turn usually calls WT_CONNECTION::add_data_source, +WT_CONNECTION::add_encryptor, WT_CONNECTION::add_collator and/or +WT_CONNECTION::add_compressor to add functionality to WiredTiger. + +@section extensions_recovery Extensions and recovery + +Some extensions, in particular WT_COLLATOR and WT_COMPRESSOR, are required +in order to run recovery if \c logging is enabled. This means that they +must be loaded using the \c extensions keyword to ::wiredtiger_open, +because recovery runs before ::wiredtiger_open returns the WT_CONNECTION +handle to the application. + +@section extensions_local Local extensions + +If it is not feasible to separate application logic into a loadable +extension separate from the executable, applications can use the reserved +name \c local as a path in the \c extensions list. This will search for +the entry point in the running application, and should usually override the +\c entry symbol. + +Here is an example of a local entry point: + +<code> +wiredtiger_open(path, NULL, + "create,extensions=[local=(entry=my_extension_init)]") +</code> + +Note that databases created in this way can only be opened by applications +that include the specified entry point. In particular, <b>they cannot be +accessed using the @ref command_line</b>. + + */ diff --git a/src/third_party/wiredtiger/src/docs/file-formats.dox b/src/third_party/wiredtiger/src/docs/file-formats.dox new file mode 100644 index 00000000000..8346024953a --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/file-formats.dox @@ -0,0 +1,164 @@ +/*! @m_page{{c,java},file_formats,File formats and compression} + +@section file_formats_formats File formats + +WiredTiger supports two underlying file formats: row-store and +column-store, where both are B+tree implementations of key/value stores. +WiredTiger also supports @ref lsm, implemented as a tree of B+trees. + +In a row-store, both keys and data are variable-length byte strings. In +a column-store, keys are 64-bit record numbers (key_format type 'r'), +and values are either variable- or fixed-length byte strings. + +Generally, row-stores are faster for queries where all of the columns +are required by every lookup (because there's only a single set of +meta-data pages to read into the cache and search). Column-stores are +faster when most queries require only a subset of the columns (because +columns can be separated into multiple files and only the columns being +returned need be present in the cache). + +Row-store keys and values, and variable-length column-store values, can +be up to (4GB - 512B) in length. Keys and values too large to fit on a +normal page are stored as overflow items in the file, and are likely to +require additional file I/O to access. + +Fixed-length column-store values (value_format type 't'), are limited +to 8-bits, and only values between 0 and 255 may be stored. +Additionally, there is no out-of-band fixed-length "deleted" value, and +deleting a value is the same as storing a value of 0. For the same +reason, storing a value of 0 will cause cursor scans to skip the record. + +WiredTiger does not support duplicate data items: there can be only a +single value associated with any given key, and applications are +responsible for creating unique key/value pairs. + +WiredTiger allocates space from the underlying files in block units. +The minimum file allocation unit WiredTiger supports is 512B and the +maximum is 512MB. File offsets are signed 8B values, making the maximum +file size very, very large. + +@section file_formats_choice Choosing a file format + +The row-store format is the default choice for most applications. When +the primary key is a record number, there are advantages to storing +columns in separate files, or the underlying data is a set of bits, +column-store format may be a better choice. + +Both row- and column-store formats can maintain high volumes of writes, +but for data sets requiring sustained, extreme write throughput, +@ref lsm are usually a better choice. For applications that do not +require extreme write throughput, row- or column-store is likely to be +a better choice because the read throughput is better than with LSM trees +(an effect that becomes more pronounced as additional read threads are +added). + +Applications with complex schemas may also benefit from using multiple +storage formats, that is, using a combination of different formats in +the database, and even in individual tables (for example, a sparse, wide +table configured with a column-store primary, where indexes are stored +in an LSM tree). + +Finally, as WiredTiger makes it easy to switch back-and-forth between +storage configurations, it's usually worthwhile benchmarking possible +configurations when there is any question. + +@section file_formats_compression File formats and compression + +Row-stores support four types of compression: key prefix compression, +dictionary compression, Huffman encoding and block compression. + +- Key prefix compression reduces the size requirement of both in-memory +and on-disk objects by storing any identical key prefix only once per +page. + + The cost is additional CPU and memory when operating on the in-memory tree. +Specifically, sequential cursor movement through prefix-compressed page in +reverse (but not forward) order, or the random lookup of a key/value pair will +allocate sufficient memory to hold some number of uncompressed keys. So, for +example, if key prefix compression only saves a small number of bytes per key, +the additional memory cost of instantiating the uncompressed key may mean +prefix compression is not worthwhile. Further, in cases where the +on-disk cost is the primary concern, block compression may mean prefix +compression is less useful. + + Applications may limit the use of prefix compression by configuring the +minimum number of bytes that must be gained before prefix compression is +used with the WT_SESSION::create method's \c prefix_compression_min +configuration string. + + Key prefix compression is disabled by default. + +- Dictionary compression reduces the size requirement of both the +in-memory and on-disk objects by storing any identical value only once +per page. The cost is minor additional CPU and memory use when writing +pages to disk. + + Dictionary compression is disabled by default. + +- Huffman encoding reduces the size requirement of both the in-memory +and on-disk objects by compressing individual key/value items, and can +be separately configured either or both keys and values. The cost is +additional CPU and memory use when searching the in-memory tree (if keys +are encoded), and additional CPU and memory use when returning values +from the in-memory tree and when writing pages to disk. Note the +additional CPU cost of Huffman encoding can be high, and should be +considered. (See @subpage_single huffman for details.) + + Huffman encoding is disabled by default. + +- Block compression reduces the size requirement of on-disk objects by +compressing blocks of the backing object's file. The cost is additional +CPU and memory use when reading and writing pages to disk. Note the +additional CPU cost of block compression can be high, and should be +considered. (See @ref compression for details.) + + Block compression is disabled by default. + +Column-stores with variable-length byte string values support four +types of compression: run-length encoding, dictionary compression, +Huffman encoding and block compression. + +- Run-length encoding reduces the size requirement of both the in-memory +and on-disk objects by storing sequential, duplicate values in the store +only a single time (with an associated count). The cost is minor +additional CPU and memory use when returning values from the in-memory +tree and when writing pages to disk. + + Run-length encoding is always enabled and cannot be turned off. + +- Dictionary compression reduces the size requirement of both the +in-memory and on-disk objects by storing any identical value only once +per page. The cost is minor additional CPU and memory use when +returning values from the in-memory tree and when writing pages to disk. + + Dictionary compression is disabled by default. + +- Huffman encoding reduces the size requirement of both the in-memory +and on-disk objects by compressing individual value items. The cost is +additional CPU and memory use when returning values from the in-memory +tree and when writing pages to disk. Note the additional CPU cost of +Huffman encoding can be high, and should be considered. +(See @ref_single huffman for details.) + + Huffman encoding is disabled by default. + +- Block compression reduces the size requirement of on-disk objects by +compressing blocks of the backing object's file. The cost is additional +CPU and memory use when reading and writing pages to disk. Note the +additional CPU cost of block compression can be high, and should be +considered. (See @ref compression for details.) + + Block compression is disabled by default. + +Column-stores with fixed-length byte values support a single type of +compression: block compression. + +- Block compression reduces the size requirement of on-disk objects by +compressing blocks of the backing object's file. The cost is additional +CPU and memory use when reading and writing pages to disk. Note the +additional CPU cost of block compression can be high, and should be +considered. (See @ref compression for details.) + + Block compression is disabled by default. + +*/ diff --git a/src/third_party/wiredtiger/src/docs/filesystems.dox b/src/third_party/wiredtiger/src/docs/filesystems.dox new file mode 100644 index 00000000000..1d54a233eea --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/filesystems.dox @@ -0,0 +1,13 @@ +/*! @page filesystems Filesystems + +@section remote Remote file systems + +WiredTiger objects may be stored on remote file systems if the remote +file system conforms to ISO/IEC 9945-1:1996 (POSIX.1). In the case of +read-only objects, multiple systems may access the objects +simultaneously; objects which are written cannot be accessed by more +than a single system. Because remote file systems are often slower than +local file systems, using a remote file system for storage may degrade +performance. + +*/ diff --git a/src/third_party/wiredtiger/src/docs/helium.dox b/src/third_party/wiredtiger/src/docs/helium.dox new file mode 100644 index 00000000000..5e09cb6801b --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/helium.dox @@ -0,0 +1,125 @@ +/*! @page helium WiredTiger Helium support + +WiredTiger supports Levyx Inc., Helium Data Store volumes as a data-source. + +To configure one or more Helium volumes as WiredTiger data sources, take +the following steps. + +@section helium_build Building the WiredTiger Helium Support + +To build the Helium support, use the configuration option \c --with-helium=DIR. +For example: + +@code +% cd wiredtiger +% ls /usr/local/lib/Helium +Helium Programmer's Reference.pdf libhe.a +README.TXT libhe.so +he.h +% ./configure --with-helium=/usr/local/lib/Helium && make +@endcode + +@section helium_load Loading the WiredTiger Helium Support + +Next, add code to your application to load the Helium shared library. + +The following example loads the Helium shared library, configuring and +naming two separate Helium volumes. The first volume is named \c dev1, +the second volume is named \c dev2. Volume \c dev1 has two underlying +physical Helium devices, \c /dev/disk3s1 and \c /dev/disk4s1. Volume +\c dev2 has a single underlying physical Helium device, \c /dev/disk5s1. + +@code +#define HELIUM_LIBRARY_PATH "test/helium/.libs/libwiredtiger_helium.so"" +ret = connection->load_extension(connection, HELIUM_LIBRARY_PATH, + "config=[" + "dev1=[helium_devices=[\"he://.//dev/disk3s1,/dev/disk4s1\"]," + "helium_o_volume_truncate=1]," + "dev2=[helium_devices=[\"he://.//dev/disk5s1\"]," + "helium_o_volume_truncate=1]]"); +@endcode + +The \c helium_devices configuration string takes a WiredTiger string +which is a comma-separated list of Helium devices. (Note the quoting +required for that to be possible.) + +In this example, both Helium volumes are configured to be truncated when +first opened, and all previously existing contents discarded. + +When configuring a Helium volume, the following non-standard configuration +strings are supported: + +<table> +@hrow{String, Type, Meaning} +@row{helium_devices, list, WiredTiger URI to Helium volume mapping} +@row{helium_env_read_cache_size, int, struct he_env read_cache_size value} +@row{helium_env_write_cache_size, int, struct he_env write_cache_size value} +@row{helium_o_volume_truncate, boolean, HE_O_VOLUME_TRUNCATE flag} +</table> + +With the exception of the configuration string \c helium_devices (which +is WiredTiger specific), see the Helium documentation for details on +their use. + +@section helium_objects Creating WiredTiger objects on Helium volumes + +When creating WiredTiger objects on Helium volumes, the volume names are +used as part of the URI specified to WiredTiger methods such as +WT_SESSION::create or WT_SESSION::rename, separated from the object name +by a single slash character. + +Additionally, the \c helium \c type configuration string must be included. + +The following example creates a table named \c access on the Helium +volume \c dev1, and then opens a cursor on the table: + +@code +WT_CURSOR *cursor; +WT_SESSION *session; + +/* Create the access table. */ +ret = session->create( + session, "table:dev1/access", "key_format=S,value_format=S,type=helium"); + +/* Open a cursor on the access table. */ +ret = session->open_cursor(session, "table:dev1/access", NULL, NULL, &cursor); +@endcode + +When calling WT_SESSION::create to create an object on a Helium volume, +the following additional configuration strings are supported: + +<table> +@hrow{String, Type, Meaning} +@row{helium_o_compress, boolean, HE_I_COMPRESS flag} +@row{helium_o_truncate, boolean, HE_O_TRUNCATE flag} +</table> + +See the Helium device documentation for details on their use. + +For example, creating and truncating a table could be done as follows: + +@code +WT_SESSION *session; + +/* Create and truncate the access table. */ +ret = session->create(session, "table:dev1/access", + "key_format=S,value_format=S,type=helium,helium_o_truncate=1"); +@endcode + +@section helium_notes Helium notes + +- Helium volumes do not support database backups. +- Helium volumes do not support named checkpoints. +- Helium volumes do not support compression of any kind. +- Helium volumes do not support bulk load as a special case, and configuring +cursors for bulk load has no effect. +- Inserting a new record after the current maximum record in a fixed-length +bit field column-store (that is, a store with an 'r' type key and 't' type +value) does not implicitly create the missing records. + +@section helium_limitations Helium limitations + +- WiredTiger transactions cannot include operations on both Helium volumes +and other stores; this will be corrected in a future release. + +*/ diff --git a/src/third_party/wiredtiger/src/docs/home.dox b/src/third_party/wiredtiger/src/docs/home.dox new file mode 100644 index 00000000000..3cde64877b2 --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/home.dox @@ -0,0 +1,31 @@ +/*! @page home Database Home Directory + +WiredTiger file naming is based on the \c home argument specified to the +::wiredtiger_open function and the \c WIREDTIGER_HOME environment +variable, determined by the following steps: + +-# If a \c home argument is specified to the ::wiredtiger_open function, +its value is used as the database directory. +-# If the \c WIREDTIGER_HOME environment variable is not set, the database +directory is the process' current working directory. No current working +directory path is maintained by the WiredTiger software and changing the +working directory after opening the WiredTiger database may cause failure. +-# If the process is running with special privileges, the ::wiredtiger_open +function must be configured with the \c use_environment_priv flag. The +\c use_environment_priv flag is intended for applications that have or +acquire special privileges and wish to use an environment-specified home +directory: if such an application does not configure the ::wiredtiger_open +function with the \c use_environment_priv flag, setting an environment +value will cause the open to fail. On ISO/IEC 9945-1:1990 (POSIX.1) +systems, "special privileges" is defined as a real user ID not equal to the +effective user ID, or the real group ID not equal to the effective group +ID. +-# Otherwise, the value of the \c WIREDTIGER_HOME environment variable is +used as the database directory. + +Finally, consider security when configuring WiredTiger to use the +\c WIREDTIGER_HOME environment variable, especially in applications which +run with permissions other than the user's. Such applications are +potentially vulnerable to allowing users access to databases they could not +otherwise access. +*/ diff --git a/src/third_party/wiredtiger/src/docs/huffman.dox b/src/third_party/wiredtiger/src/docs/huffman.dox new file mode 100644 index 00000000000..41e4be5ced3 --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/huffman.dox @@ -0,0 +1,66 @@ +/*! @page huffman Huffman Encoding + +Keys in row-stores and variable-length values in either row- or +column-stores can be compressed with Huffman encoding. + +Huffman compression is maintained in memory as well as on disk, and can +increase the amount of usable data the cache can hold as well as +decrease the size of the data on disk. The additional CPU cost of +Huffman coding can be high, and should be considered. + +To configure Huffman encoding for the key in a row-store, specify \c +huffman_key=english, \c huffman_key=utf8<file> or \c +huffman_key=utf16<file> in the configuration passed to \c +WT_SESSION::create. + +To configure Huffman encoding for a variable-length value in either a +row-store or a column-store, specify \c huffman_value=english, \c +huffman_value=utf8<file> or \c huffman_value=utf16<file> in the +configuration passed to \c WT_SESSION::create. + +Setting Huffman encoding to \c english configures WiredTiger to use a +built-in English language frequency table. The English language +frequency table is based on \c "Case-sensitive letter and bigram +frequency counts from large-scale English corpora", by Michael N. Jones +and D.J.K. Mewhort, modified to support space and tab characters. + +Setting Huffman encoding to \c utf8<file> or \c utf16<file> configures +WiredTiger to use a frequency table read from a file. (Note: the \c < +and \c > characters are not literal, and should not appear in the +string.) + +The frequency table file format is lines containing pairs of unsigned +integers separated by whitespace. The first integer is the symbol +value, the second integer is the frequency value. Symbol values may be +specified as hexadecimal numbers (with a leading \c 0x prefix), or as +integers. For example, an English-language frequency table for the +characters \c 0 through \c 9 might look like this: + +@code +0x30 546233 +0x31 460946 +0x32 333499 +0x33 187606 +0x34 192528 +0x35 374413 +0x36 153865 +0x37 120094 +0x38 182627 +0x39 282364 +@endcode + +Frequency table symbol values must be unique. In the case of \c utf8 +files, symbol values must be in the range of 0 to 255. In the case of +\c utf16 files, symbol values must be in the range of 0 to 65,535. +Frequency values do not need to be unique, but must be in the range of +0 to the maximum 32-bit unsigned integer value (4,294,967,295), where +the lower a frequency value, the less likely the byte value is to occur. + +Any symbol values not listed in the frequency table are assumed to have +frequencies of 0. Input containing symbol values that did not appear +in the frequency table (or appeared in the frequency table, but with +frequency values of 0), are accepted, but will not compress as well as +if they are listed in the frequency table, with frequency values other +than 0. + +*/ diff --git a/src/third_party/wiredtiger/src/docs/images/LogoFace-watermark.png b/src/third_party/wiredtiger/src/docs/images/LogoFace-watermark.png Binary files differnew file mode 100644 index 00000000000..7af37f88a7c --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/images/LogoFace-watermark.png diff --git a/src/third_party/wiredtiger/src/docs/images/LogoFinal-header.png b/src/third_party/wiredtiger/src/docs/images/LogoFinal-header.png Binary files differnew file mode 100644 index 00000000000..fcbdb27ae81 --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/images/LogoFinal-header.png diff --git a/src/third_party/wiredtiger/src/docs/images/architecture.pdf b/src/third_party/wiredtiger/src/docs/images/architecture.pdf Binary files differnew file mode 100644 index 00000000000..ff97ddff2b4 --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/images/architecture.pdf diff --git a/src/third_party/wiredtiger/src/docs/images/architecture.png b/src/third_party/wiredtiger/src/docs/images/architecture.png Binary files differnew file mode 100644 index 00000000000..c5b72bc05e0 --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/images/architecture.png diff --git a/src/third_party/wiredtiger/src/docs/images/wtstats.png b/src/third_party/wiredtiger/src/docs/images/wtstats.png Binary files differnew file mode 100644 index 00000000000..f65a2871b6f --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/images/wtstats.png diff --git a/src/third_party/wiredtiger/src/docs/introduction.dox b/src/third_party/wiredtiger/src/docs/introduction.dox new file mode 100644 index 00000000000..52936a04d3c --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/introduction.dox @@ -0,0 +1,62 @@ +/*! @mainpage Reference Guide + +WiredTiger is an high performance, scalable, production quality, NoSQL, +Open Source extensible platform for data management. + +WiredTiger supports row-oriented storage (where all columns of a row are +stored together), column-oriented storage (where columns are stored in +groups, allowing for more efficient access and storage of column +subsets) and log-structured merge trees (LSM), for sustained throughput +under random insert workloads. + +WiredTiger includes ACID transactions with standard isolation levels and +durability at both checkpoint and fine-grained granularity. + +WiredTiger can be used as a simple key/value store, but also has a +complete schema layer, including indices and projections. + +WiredTiger should be generally portable to any 64-bit system supporting +the ANSI C99, POSIX 1003.1 and POSIX 1003.1c (threads extension) standards. + +For more information on the WiredTiger architecture and why it might be +right for your project, see: + +- @subpage architecture + +For more information about building and installing WiredTiger, see: + +- @subpage build-posix +- @subpage build-windows +- @subpage leveldb + +For more information about writing WiredTiger applications, see: + +- @subpage programming +- @subpage programming_lang_java +(The Java API is not available on Windows.) +- @ref wt "WiredTiger API reference manual" + +For more information about administrating WiredTiger databases, see: + +- @subpage command_line +- @subpage admin + +For release change logs and upgrading information, see: + +- @subpage md_changelog +- @subpage upgrading + +WiredTiger is Open Source; for more information, see: + +- @subpage license + +WiredTiger releases are tested on Linux, FreeBSD and OS X; for more +information, see: + +- @subpage testing + +To browse the WiredTiger source code repository or contact us, see: + +- @subpage community + +*/ diff --git a/src/third_party/wiredtiger/src/docs/keyvalue.dox b/src/third_party/wiredtiger/src/docs/keyvalue.dox new file mode 100644 index 00000000000..f375886b030 --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/keyvalue.dox @@ -0,0 +1,19 @@ +/*! @page keyvalue Key/Value pairs + +Underlying WiredTiger files are key/value stores, that is, entries in +the file are paired keys and values. These keys and values can be +accessed directly, and a file cursor will only return the data from the +single file, without reference to any other related files or data. For +example, if an index were lost or to become inconsistent with its +primary, a file cursor could read from the index without problems (even +though some of the returned keys might not exist in the primary). + +In the case of a row-store file, keys and values are both +variable-length byte strings, up to (4GB - 512B) in length. + +In the case of a column-store file, keys are 64-bit record numbers, +where the first record in the file is record number 1. Values are +either variable-length byte strings, up to (4GB - 512B) in length, or +bit fields, where a bit field is 1 to 8 bits. + + */ diff --git a/src/third_party/wiredtiger/src/docs/leveldb.dox b/src/third_party/wiredtiger/src/docs/leveldb.dox new file mode 100644 index 00000000000..9233adf5971 --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/leveldb.dox @@ -0,0 +1,54 @@ +/*! +@page leveldb Building and installing the LevelDB compatibility API + +WiredTiger includes an implementation of the LevelDB API to ease integration and performance testing with existing applications. + +@section leveldb_build Building and using WiredTiger's LevelDB API + +Here are the steps to build and run an existing application against WiredTiger's LevelDB API: + +<ol> +<li> choose an installation directory for WiredTiger. The default is: + + <code> + WT_INSTALL=/usr/local + </code> + + but anywhere is fine. + +<li> build WiredTiger with LevelDB support: + + <code> + ./configure --enable-leveldb --with-builtins=snappy --prefix=$WT_INSTALL + make install + </code> + + Change the above to \c --enable-leveldb=basho to build a version of + the LevelDB API compatible with Basho's Riak product, or + \c --enable-leveldb=hyper to build a version of the LevelDB API + compatible with the HyperDex product, or + \c --enable-leveldb=rocksdb to build a version of the LevelDB API + compatible with applications using Facebook's RocksDB engine. + +<li> compile and link the application against WiredTiger's LevelDB API: + + <code> + ./configure CPPFLAGS="-I$WT_INSTALL/include/wiredtiger" LDFLAGS="-L$WT_INSTALL/lib/wiredtiger" ... && make + </code> + +<li> start the application: + + <code> + LD_LIBRARY_PATH=$WT_INSTALL/lib/wiredtiger \<normal start command\> + </code> +</ol> + +@section leveldb_caveats Caveats + +The core LevelDB features are well supported, though some configuration options may be ignored. In the extended versions of the API, some rarely used features have not been implemented, and will cause either build or runtime errors. + +The WiredTiger on-disk format is not compatible with LevelDB. Data will need to be dumped and reloaded by the application. + +The WiredTiger LevelDB API is not guaranteed to be binary compatible with LevelDB. We do not support using \c LD_PRELOAD to switch an existing binary built with stock LevelDB to WiredTiger without compiling against the installed include files. + + */ diff --git a/src/third_party/wiredtiger/src/docs/license.dox b/src/third_party/wiredtiger/src/docs/license.dox new file mode 100644 index 00000000000..febced2c6af --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/license.dox @@ -0,0 +1,79 @@ +/*! @page license WiredTiger licensing + +The complete WiredTiger software package is Open Source software: you +are welcome to modify and redistribute it under the terms of +<a href="http://www.gnu.org/licenses/gpl-2.0-standalone.html"> +<b>version 2</b></a> or +<a href="http://www.gnu.org/licenses/gpl-3.0-standalone.html"> +<b>version 3</b></a> of the +<b>GNU General Public License</b></a> +as published by the Free Software Foundation. This program is +distributed in the hope that it will be useful, but WITHOUT ANY +WARRANTY; without even the implied warranty of MERCHANTABILITY or +FITNESS FOR A PARTICULAR PURPOSE. See the +<b>GNU General Public License</b></a> for details. + +Additionally, portions of the WiredTiger distribution are distributed +under the terms of the +<a href="http://www.opensource.org/licenses/BSD-3-Clause"> +BSD-3-Clause License</a>. These files have +<a href="http://www.opensource.org/licenses/BSD-3-Clause"> +BSD-3-Clause License</a> +copyright notices, and may be freely used and redistributed under the +terms of that notice. + +Additionally, portions of the WiredTiger distribution are public domain +software. Public domain files have notices releasing the software into +the public domain and may be freely used and redistributed. + +For a license to use the WiredTiger software under conditions other than +those described above, or for technical support for this software, please +contact MongoDB, Inc. at +<a mailto="info@wiredtiger.com">info@wiredtiger.com</a>. + +@section license_library 3rd party software included in the WiredTiger library + +Every build of the WiredTiger library binary includes the following 3rd +party software, distributed under their license terms. Redistribution +of the WiredTiger library should comply with these copyrights. + +<table> +@hrow{Distribution Files, Copyright Holder, License} +@row{\c src/include/bitstring.i, University of California\, Berkeley, <a href="http://www.opensource.org/licenses/BSD-3-Clause">BSD-3-Clause License</a>} +@row{\c src/include/queue.h, University of California\, Berkeley, <a href="http://www.opensource.org/licenses/BSD-3-Clause">BSD-3-Clause License</a>} +@row{\c src/os_posix/os_getopt.c, University of California\, Berkeley, <a href="http://www.opensource.org/licenses/BSD-3-Clause">BSD-3-Clause License</a>} +@row{\c src/support/hash_city.c, Google\, Inc., <a href="http://www.opensource.org/licenses/MIT">The MIT License</a>} +@row{\c src/support/hash_fnv.c, Authors, Public Domain} +</table> + +@section license_leveldb 3rd party software optionally included in the WiredTiger library: LevelDB + +If the \c --enable-leveldb configuration option is specified when +configuring the WiredTiger build, additional 3rd party software is +included in the WiredTiger LevelDB library binary, distributed under +their license terms. Redistribution of the WiredTiger library built +with the \c --enable-leveldb configuration option should comply with +these copyrights. + +<table> +@hrow{Distribution Files, Copyright Holder, License} +@row{\c api/leveldb/basho/*, Basho Technologies\, Inc., <a href="http://opensource.org/licenses/Apache-2.0">Apache License\, Version 2.0</a>} +@row{\c api/leveldb/hyperleveldb/*, The HyperLevelDB Authors, <a href="http://www.opensource.org/licenses/BSD-3-Clause">BSD-3-Clause License</a>} +@row{\c api/leveldb/leveldb/*, Google\, Inc., <a href="http://www.opensource.org/licenses/BSD-3-Clause">BSD-3-Clause License</a>} +@row{\c api/leveldb/rocksdb/*, Facebook\, Inc., <a href="http://www.opensource.org/licenses/BSD-3-Clause">BSD-3-Clause License</a>} +</table> + +@section license_distribution 3rd party software included in the WiredTiger distribution + +The WiredTiger software distribution includes additional 3rd party software +licensed under a variety of licenses. <b>This software is NOT included +in the WiredTiger library binary as built by the distribution.</b> This +software is used for peripheral tasks, such as building the WiredTiger +documentation, testing the WiredTiger software or graphically displaying +WiredTiger run-time output. If you extract portions of the WiredTiger +software distribution for inclusion in other packages, or redistribute +selected portions of the WiredTiger sources, please review the copyright +notices and LICENSE files included in the WiredTiger distribution for +the terms and conditions of such redistribution. + +*/ diff --git a/src/third_party/wiredtiger/src/docs/lsm.dox b/src/third_party/wiredtiger/src/docs/lsm.dox new file mode 100644 index 00000000000..08512f27c92 --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/lsm.dox @@ -0,0 +1,133 @@ +/*! @m_page{{c,java},lsm,Log-Structured Merge Trees} + +@section lsm_background Background + +A common requirement is sustained throughput under a workload that consists +of random inserts, where either the key range is chosen so that inserts are +very unlikely to conflict (e.g., 128-bit hashes), or where inserts are +expected to overwrite existing values. + +With traditional btree variants, inserts are very fast while the data set +remains in cache, but once the tree overflows the cache, performance drops +significantly. There are two factors involved: + + -# once the data fills the cache, new inserts have some probability of going +to a page that is not in cache, requiring a read; and + -# the cache is full of dirty pages, so pages have to be written to free +space in the cache before the read can be satisfied. + +@section lsm_description Description of LSM trees + +Log-Structured Merge Trees are described in this paper by Patrick O'Neil, +Edward Cheng, Dieter Gawlick and Elizabeth O'Neil: +http://www.cs.umb.edu/~poneil/lsmtree.pdf + +A logical tree is split into several physical pieces so that the +most-recently-updated portion of data is in a tree that fits entirely in +memory. The size of the in-memory chunk can be configured with the +\c "lsm=(chunk_size)" configuration key to WT_SESSION::create. + +Once the in-memory tree reaches a threshold size, a new in-memory tree is +created and the old tree synced to disk. Once written to disk, trees are +read-only, though they are merged in the background with other on-disk +trees to reduce the cost of reads. + +With this structure, "blind writes" can be performed entirely on the +in-memory tree. Deletes are implemented by inserting a special "tombstone" +record into the in-memory tree. + +@section lsm_api Interface to LSM trees + +An LSM tree can be created as follows, in much the same way as a +WiredTiger btree file: + +@code +@m_if{c} +session->create(session, "table:bucket", "type=lsm,key_format=S,value_format=S"); +@m_else +session.create("table:bucket", "type=lsm,key_format=S,value_format=S"); +@m_endif +@endcode + +Once created, the LSM tree can be accessed using the same cursor interface +as other data sources in WiredTiger: + +@code +@m_if{c} +WT_CURSOR *c; + +session->open_cursor(session, "table:bucket", NULL, NULL, &c); +for(;;) { + c->set_key(c, "key"); + c->set_value(c, "value"); + c->insert(c); +} +@m_else +Cursor c = session.open_cursor("table:bucket", null, null); +for(;;) { + c.putKeyString("key"); + c.putValueString("value"); + c.insert(); +} +@m_endif +@endcode + +If an LSM cursor is configured with \c "overwrite=false" passed to +WT_SESSION::open_cursor, the result will be a search through the levels of the +LSM tree before every modification. + +@section lsm_merge Merging + +A background thread is opened for each active LSM tree. This thread is +responsible for both writing old chunks to stable storage, and for merging +multiple chunks together so that reads can be satisfied from a small number +of files. There is currently no way to configure merges: they are performed +automatically by the background thread. + +@section lsm_bloom Bloom filters + +WiredTiger creates a Bloom filter when merging. This is an additional file +that contains a configurable number of bits per key (default 8). Keys are +hashed a configurable number of times (default 4), and the corresponding +bits set. The Bloom filter is used to avoid reading from a chunk if the key +cannot be present. + +With the defaults, the Bloom filter only requires one byte per key, so +it usually fits in cache. The Bloom parameters can be configured with +\c "lsm=(bloom_bit_count)" and \c "lsm=(bloom_hash_count)" configuration +keys to WT_SESSION::create. The Bloom file can be configured with the +\c "lsm=(bloom_config)" key. + +@section lsm_schema Creating tables using LSM trees + +Tables or indices can be stored using LSM trees. Schema support is provided +for LSM as an extension to the WT_SESSION::create method: + +@code +@m_if{c} +session->create(session, "table:T", "type=lsm"); +@m_else +session.create("table:T", "type=lsm"); +@m_endif +@endcode + +The default type for all schema objects will continue to be btree. + +@section lsm_caveats Caveats + +@subsection lsm_hazard Hazard configuration + +Reads from an LSM cursor may need to position a cursor in each active chunk. +The number of chunks depends on the chunk size, and how many chunks have +been merged. There must be at least as many hazard pointers available as +there are chunks in the tree for each cursor that is open on the LSM tree. +The number of hazard pointers is configured with the \c "hazard_max" +configuration key to ::wiredtiger_open. + +@subsection lsm_checkpoints Named checkpoints + +Named checkpoints are not supported on LSM trees, and cursors cannot be +opened with a non-empty \c "checkpoint" configuration (that is, only the +most recent standard checkpoint can be read). + + */ diff --git a/src/third_party/wiredtiger/src/docs/namespace.dox b/src/third_party/wiredtiger/src/docs/namespace.dox new file mode 100644 index 00000000000..8657338e369 --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/namespace.dox @@ -0,0 +1,39 @@ +/*! @m_page{{c,java},namespace,Name spaces} + +@section namespace_env Process' environment name space + +WiredTiger's environment variables begin with the string "WIREDTIGER". + +@m_if{c} +@section namespace_c C language name space + +WiredTiger's public function names begin with the string "wiredtiger". + +WiredTiger's public \c \#define and structure \c typedef declarations +begin with the string "WT_". + +WiredTiger's private function names begin with the string "__wt_". +@m_endif + +@m_if{java} +@section namespace_c Java language name space + +WiredTiger's public classes are in the \c com.wiredtiger.db package. + +The \c com.wiredtiger.db.wiredtiger class contains static methods +and inherited constants. +@m_endif + +@section namespace_filesystem File system name space + +WiredTiger's files begin with the string "WiredTiger"; applications +should not create files in the WiredTiger file system name space. + +@m_if{c} +@section namespace_error Error return name space + +WiredTiger reserves all values from -31,800 to -31,999 as possible error +return values. +@m_endif + +*/ diff --git a/src/third_party/wiredtiger/src/docs/packing.dox b/src/third_party/wiredtiger/src/docs/packing.dox new file mode 100644 index 00000000000..aca9cd72b46 --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/packing.dox @@ -0,0 +1,44 @@ +/*! @m_page{{c,java},packing,Packing and Unpacking Data} + +WiredTiger's data packing uses format strings similar to those specified in the +Python struct module: + http://docs.python.org/library/struct + +The first character of the format string can be used to indicate the byte +order, size and alignment of the packed data, according to the following +table: + +<table> +@hrow{Character, Byte order, Size, Alignment} +@row{<tt>.</tt>, big-endian, packed, none} +@row{<tt>></tt>, big-endian, standard, none} +@row{<tt><</tt>, little-endian, standard, none} +@row{<tt>\@</tt>, native, native, native} +</table> + +If the first character is not one of these, '.' (big-endian, packed) is +assumed: it naturally sorts in lexicographic order, and the packed format +uses variable-sized encoding of values to reduce the data size. + +@notyet{little-endian format} +Only the default big-endian, packed format is currently supported. + +The remaining characters in the format string specify the type of each field +to be packed into or unpacked from a byte array. See @ref schema_column_types +for the list of supported types. + +@m_if{c} +@todo Describe the variable-length integer packing in sufficient detail that it can be re-implemented in other programming languages or in network clients. + +@section config_examples Code samples + +The code below is taken from the complete example program @ex_ref{ex_pack.c}. It demonstrates how to pack three integer values into a buffer and then unpack them again. + +@snippet ex_pack.c packing +@m_else +In Java, data is packed and unpacked using cursor put* and get* operations, +for example: +@snippet ex_schema.c Insert and list records +@m_endif + + */ diff --git a/src/third_party/wiredtiger/src/docs/processes.dox b/src/third_party/wiredtiger/src/docs/processes.dox new file mode 100644 index 00000000000..87f715792f9 --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/processes.dox @@ -0,0 +1,28 @@ +/*! @page processes Sharing Between Processes + +@notyet{inter-process sharing} + +WiredTiger includes a server that provides remote access to databases. The primary process to open a database can optionally start a server that handles requests from other processes. + +The remote interface is the way languages other than C/C++ are supported, and the interface for client processes in multiprocess C/C++ applications. + +The server can be embedded in an application or run from the command line. To embed the RPC server in your application, pass <code>"multiprocess"</code> in the configuration string to ::wiredtiger_open. Note that in this case, when your application exits, all client connections are forcibly closed. + +For details on running a standalone RPC server, see @ref utility. + +@section processes_sharing Opening connections from multiple processes + +When ::wiredtiger_open is called for a database, one of the following occurs: + +# no process has the database open, it was closed cleanly. In this case, the opening process becomes the primary process and the database is opened without recovery. +# no process has the database open, but it was not closed cleanly. In this case, the process becomes the primary and recovery is run before the database is opened. See @ref transaction_recovery for details. +# another process has the database open and is running the RPC server, in which case the opening process becomes a client. +# another process has the database open but is not running the RPC server, in which case the open fails. + +@section processes_example Code samples + +The code below is taken from the complete example program @ex_ref{ex_process.c}. + +@snippet ex_process.c processes + + */ diff --git a/src/third_party/wiredtiger/src/docs/programming.dox b/src/third_party/wiredtiger/src/docs/programming.dox new file mode 100644 index 00000000000..5d79edd660b --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/programming.dox @@ -0,0 +1,81 @@ +/*! @m_page{{c,java},programming,Writing WiredTiger applications} + +This section covers topics of interest for programmers writing +@m_if{c} +WiredTiger applications. +@m_else +WiredTiger applications in Java. +@m_endif + +We follow SQL terminology: a database is set of tables managed together. +Tables consist of rows, where each row is a key and its associated +value. Tables may optionally have an associated schema, splitting the +value into a set of columns. Tables may also have associated indices, +each of which is ordered by one or more columns. + +<h2>Using the API</h2> +- @subpage basic_api +- @subpage config_strings +- @subpage cursors +- @subpage transactions +- @subpage error_handling + +<h2>Storage options</h2> +- @subpage schema +- @subpage lsm +- @subpage file_formats +- @subpage compression +- @subpage encryption + +<h2>Programming notes</h2> +- @subpage threads +- @subpage namespace +@m_if{c} +- @subpage signals +@m_endif + +<h2>Advanced topics</h2> +- @subpage async +- @subpage backup +- @subpage compact +- @subpage checkpoint +- @subpage durability +- @subpage cursor_join +- @subpage cursor_log +- @ref transaction_named_snapshots +- @subpage rebalance +- @subpage shared_cache +- @subpage statistics +- @subpage_single upgrade + +@m_if{c} +<h2>Extending WiredTiger</h2> +- @subpage extensions +- @subpage custom_collators +- @subpage custom_extractors +- @subpage custom_data_sources +- @subpage helium +@m_endif + +<h2>Performance monitoring and tuning</h2> +- @subpage_single tune_statistics +- @subpage_single wtperf +- @subpage_single wtstats +<p> +- @subpage_single tune_memory_allocator +- @subpage_single tune_page_sizes +- @subpage_single tune_cache +- @subpage_single tune_bulk_load +- @subpage_single tune_cursor_persist +- @subpage_single tune_read_only +- @subpage_single tune_durability +- @subpage_single tune_checksum +- @subpage_single tune_compression +- @subpage_single tune_file_alloc +- @subpage_single tune_system_buffer_cache +- @subpage_single tune_transparent_huge_pages +- @subpage_single tune_close +- @subpage_single tune_mutex +- @subpage_single tune_zone_reclaim + + */ diff --git a/src/third_party/wiredtiger/src/docs/rebalance.dox b/src/third_party/wiredtiger/src/docs/rebalance.dox new file mode 100644 index 00000000000..a6acfe07ef5 --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/rebalance.dox @@ -0,0 +1,14 @@ +/*! @m_page{{c,java},rebalance,Rebalance} + +The WT_SESSION::rebalance method can be used to rebalance data sources' +underlying btrees. If a tree has become unbalanced (that is, one part of +the tree is excessively deep), WT_SESSION::rebalance rewrites the tree +as a balanced tree. + +The data source must be quiescent. + +The WT_SESSION::rebalance method should never be needed, as WiredTiger +btrees are maintained as balanced trees. It is only provided as a tool +to handle the unexpected. + + */ diff --git a/src/third_party/wiredtiger/src/docs/schema.dox b/src/third_party/wiredtiger/src/docs/schema.dox new file mode 100644 index 00000000000..65ad7f6919c --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/schema.dox @@ -0,0 +1,403 @@ +/*! @m_page{{c,java},schema,Schema, Columns, Column Groups, Indices and Projections} + +While many tables have simple key/value pairs for records, WiredTiger +also supports more complex data patterns. + +@section schema_intro Tables, rows and columns + +A table is a logical representation of data consisting of cells in rows +and columns. For example, a database might have a simple table +including an employee identifier, last name and first name, and a +salary: + +<table> +@hrow{Employee ID, Last Name, First Name, Salary} +@row{1, Smith, Joe, 40000} +@row{2, Jones, Mary, 50000} +@row{3, Johnson, Cathy, 44000} +</table> + +A row-oriented database would store all of the values for the first +employee in the first row, then the next employee's values in the next +row, and so on: + +<pre> + 1,Smith,Joe,40000 + 2,Jones,Mary,50000 + 3,Johnson,Cathy,44000 +</pre> + +A column-oriented database would store all of the values of a column +together, then the values of the next column, and so on: + +<pre> + 1,2,3 + Smith,Jones,Johnson + Joe,Mary,Cathy + 40000,50000,44000 +</pre> + +WiredTiger supports both storage formats, and can mix and match the storage +of columns within a logical table. + +A \em table in WiredTiger consist of one or more \em "column groups" that +together hold all of the columns in primary key order; and zero or more \em +indices that enable fast lookup of records by columns in orders other than +the primary key. + +Applications describe the format of their data by supplying a schema to +WT_SESSION::create. This specifies how the application's data can be split +into fields and mapped onto rows and columns. + +@section schema_column_types Column types + +By default, WiredTiger works as a traditional key/value store, where the +keys and values are raw byte arrays accessed using a WT_ITEM structure. +Key and value types may also be chosen from a list, or composed of multiple +columns with any combination of types. Keys and values may be up to +(<code>4GB - 512B</code>) bytes in size. + +See @subpage_single keyvalue for more details on raw key / value items. + +@section schema_format_types Format types + +WiredTiger's uses format strings similar to those specified in the Python +struct module to describe the types of columns in a table: + http://docs.python.org/library/struct + +<table> +@hrow{Format, C Type, Java type, Python type, Notes} +@row{\c x, N/A, N/A, N/A, pad byte\, no associated value} +@row{\c b, \c int8_t, \c byte, \c int, signed byte} +@row{\c B, \c uint8_t, \c byte, \c int, unsigned byte} +@row{\c h, \c int16_t, \c short, \c int, signed 16-bit} +@row{\c H, \c uint16_t, \c short, \c int, unsigned 16-bit} +@row{\c i, \c int32_t, \c int, \c int, signed 32-bit} +@row{\c I, \c uint32_t, \c int, \c int, unsigned 32-bit} +@row{\c l, \c int32_t, \c int, \c int, signed 32-bit} +@row{\c L, \c uint32_t, \c int, \c int, unsigned 32-bit} +@row{\c q, \c int64_t, \c long, \c int, signed 64-bit} +@row{\c Q, \c uint64_t, \c long, \c int, unsigned 64-bit} +@row{\c r, \c uint64_t, \c long, \c int, record number} +@row{\c s, \c char[], \c String, \c str, fixed-length string} +@row{\c S, \c char[], \c String, \c str, NUL-terminated string} +@row{\c t, \c uint8_t, \c byte, \c int, fixed-length bit field} +@row{\c u, <code>WT_ITEM *</code>, <code>byte[]</code>, \c str, + raw byte array} +</table> + +The \c 'r' type is used for record number keys in column stores. It is +otherwise identical to the \c 'Q' type. + +The \c 's' type is used for fixed-length strings. If it is preceded by +a size, that indicates the number of bytes to store; the default is a +length of 1 byte. + +The \c 'S' type is encoded as a C language string terminated by a +NUL character. +@m_if{java} +Because of this, the associated Java String may not contain the NUL character. +@m_endif + +The \c 't' type is used for fixed-length bit field values. If +it is preceded by a size, that indicates the number of bits to store, +between 1 and 8. That number of low-order bits will be stored in the +table. The default is a size of 1 bit: that is, a boolean. +@m_if{c} +C applications must always use a \c uint8_t type (or equivalently, +<code>unsigned char</code>) for calls to WT_CURSOR::set_value, and a +pointer to the same for calls to WT_CURSOR::get_value. +@m_endif +If a bit field +value is combined with other types in a packing format, it is equivalent to +\c 'B', and a full byte is used to store it. + +When referenced by a record number (that is, a key format of +<code>'r'</code>), the \c 't' type will be stored in a fixed-length +column-store, and will not have an out-of-band value to indicate the record +does not exist. In this case, a 0 byte value is used to indicate the +record does not exist. This means removing a record with WT_CURSOR::remove +is equivalent to storing a value of 0 in the record with WT_CURSOR::update +(and storing a value of 0 in the record will cause cursor scans to skip the +record). Additionally, creating a record past the end of an object implies +the creation of any missing intermediate records, with byte values of 0. + +The \c 'u' type is for raw byte arrays: if it appears at the end of a +format string (including in the default \c "u" format for untyped tables), +the size is not stored explicitly. When \c 'u' appears within a format +string, the size is stored as a 32-bit integer in the same byte order as +the rest of the format string, followed by the data. + +There is a default collator that gives lexicographic (byte-wise) +comparisons, and the default encoding is designed so that lexicographic +ordering of encoded keys is usually the expected ordering. For example, +the variable-length encoding of integers is designed so that they have the +natural integer ordering under the default collator. + +See @subpage packing for details of WiredTiger's packing format. + +WiredTiger can also be extended with custom collators by implementing the +@m_if{c} +WT_COLLATOR interface. +@m_else +WT_COLLATOR interface (C only). +@m_endif + +@section schema_key_and_value_formats Key and value formats + +Every table has a key format and a value format as describe in +@ref schema_column_types. These types are configured when the table is +created by passing \c key_format and \c value_format keys to +WT_SESSION::create. + +For example, a simple row-store table with strings as both keys and +values would be created as follows: + +@snippet ex_all.c Create a table + +A simple column-store table with strings for values would be created as +follows: + +@snippet ex_all.c Create a column-store table + +@section schema_cursor_formats Cursor formats + +Cursors for a table have the same key format as the table itself. +@m_if{c} +The key columns of a cursor are set with WT_CURSOR::set_key and accessed with +WT_CURSOR::get_key. WT_CURSOR::set_key is analogous to \c printf, +and takes a list of value +@m_else +The key columns of a cursor are set with the \c Cursor.putKey* methods +and accessed with the \c Cursor.getKey* methods. \c Cursor.putKey* methods +must be called +@m_endif +in the order the key columns are configured in \c +key_format. + +For example, setting the key for a row-store table with strings as keys +would be done as follows: + +@snippet ex_all.c Set the cursor's string key + +For example, setting the key for a column-store table would be done as +follows: + +@snippet ex_all.c Set the cursor's record number key + +A more complex example, setting a composite key for a row-store table where +the key_format was \c "SiH", would be done as follows: + +@snippet ex_all.c Set the cursor's composite key + +The key's values are accessed with +@m_if{c} +WT_CURSOR::get_key, which is analogous +to \c scanf, and takes a list of pointers to values in the same order: +@m_else +successive calls to \c Cursor.getKey* methods: +@m_endif + +@snippet ex_all.c Get the cursor's string key +@snippet ex_all.c Get the cursor's record number key +@snippet ex_all.c Get the cursor's composite key + +Cursors for a table have the same value format as the table, unless a +projection is configured with WT_SESSION::open_cursor. See +@ref cursor_projections for more information. + +@m_if{c} +WT_CURSOR::set_value is used to set value columns, and +WT_CURSOR::get_value is used to get value columns, in the same way as +described for WT_CURSOR::set_key and WT_CURSOR::get_key. +@m_else +\c The Cursor.putValue* methods are used to set value columns, and +\c Cursor.getValue* are used to get value columns, in the same way as +described for \c Cursor.putKey* and \c Cursor.getKey*. +@m_endif + +@section schema_columns Columns + +The columns in a table can be assigned names by passing a \c columns key to +WT_SESSION::create. The column names are assigned first to the columns in +the \c key_format, and then to the columns in \c value_format. There must be +a name for every column, and no column names may be repeated. + +For example, a column-store table with an employee ID as the key and +three columns (department, salary and first year of employment), might +be created as follows: + +@snippet ex_all.c Create a table with columns + +In this example, the key's column name is \c id, and the value's column +names are \c department, \c salary, and \c year-started (where \c id maps to +the column format \c r, \c department maps to the column value format \c +S, \c salary maps to the value format \c i and \c year-started maps to the +value format \c H). + +Once the table is created, there is no need to call WT_SESSION::create +during subsequent runs of the application. However, it's worthwhile +making the call anyway as it both verifies the table exists and the +table schema matches the schema expected by the application. + +@section schema_column_groups Column groups + +Once column names are assigned, they can be used to configure column +groups. Column groups are primarily used to define storage in order to +tune cache behavior, as each column group is stored in a separate file. + +There are two steps involved in setting up column groups: first, pass a +list of names for the column groups in the \c colgroups configuration +key to WT_SESSION::create. Then make a call to WT_SESSION::create for +each column group, using the URI <code>colgroup:\<table\>:\<colgroup +name\></code> and a \c columns key in the configuration. Every column +must appear in at least one column group; columns can be listed in +multiple column groups, causing the column to be stored in multiple +files. + +For example, consider the following data being stored in a WiredTiger +table: + +@snippet ex_schema.c schema declaration + +If we primarily wanted to access the population information by itself, +but still wanted population information included when accessing other +information, we might store all of the columns in one file, and store +an additional copy of the population column in another file: + +@snippet ex_schema.c Create a table with column groups + +Column groups always have the same key as the table. This is particularly +useful for column stores, because record numbers are not stored explicitly +on disk, so there is no repetition of keys across multiple files. Keys +will be replicated in multiple files in the case of row-store column groups. + +A cursor can be opened on a column group by passing the column group's +URI to the WT_SESSION::open_cursor method. For example, the population +can be retrieved from both of the column groups we created: + +@snippet ex_schema.c Read population from the primary column group +@snippet ex_schema.c Read population from the standalone column group + +Key columns may not be included in the list of columns for a column +group. Because column groups always have the same key as the table, key +columns for column groups are retrieved using WT_CURSOR::get_key, not +WT_CURSOR::get_value. + +@section schema_indices Indices + +Columns are also used to create and configure indices on tables. + +Table indices are automatically updated whenever the table is modified. + +Table index cursors are read-only and cannot be used for update operations. + +To create a table index, call WT_SESSION::create using the URI +<code>index:\<table\>:\<index name\></code>, listing a column in the +configuration. + +Continuing the example, we might open an index on the \c country column: + +@snippet ex_schema.c Create an index + +Cursors are opened on indices by passing the index's URI to the +WT_SESSION::open_cursor method. + +Index cursors use the specified index key columns for +@m_if{c} +WT_CURSOR::get_key and WT_CURSOR::set_key. +@m_else +\c Cursor.getKey* and \c Cursor.putKey* calls. +@m_endif +For example, we can retrieve information from +the \c country index as follows: + +@snippet ex_schema.c Search in a simple index + +To create an index with a composite key, specify more than one column +to the WT_SESSION::create call: + +@snippet ex_schema.c Create an index with a composite key + +To retrieve information from a composite index requires a more complicated +@m_if{c} +WT_CURSOR::set_key call, +@m_else +set of \c Cursor.putKey* calls, +@m_endif +but is otherwise the same: + +@snippet ex_schema.c Search in a composite index + +@section schema_index_immutable Immutable indices + +It is possible to create an index with the \c immutable configuration +setting enabled. This setting tells WiredTiger that the index keys for a +record do not change when records are updated. This is an optimization +that it saves a remove and insert into the index whenever a value in the +primary table is updated. + +If immutable is configured when updates should alter the content of +the index it is possible to corrupt data. + +An example of using an immutable index is: + +@snippet ex_schema.c Create an immutable index + +@section schema_index_projections Index cursor projections + +By default, index cursors return all of the table's value columns from +@m_if{c} +WT_CURSOR::get_value. +@m_else +\c Cursor.getValue* calls. +@m_endif +The application can specify that a subset of the +usual columns should be returned in calls to +@m_if{c} +WT_CURSOR::get_value +@m_else +\c Cursor.getValue +@m_endif +by appending a list of columns to the \c uri parameter of the +WT_SESSION::open_cursor call. This is called a \em projection, see +@ref cursor_projections for more details. + +In the case of index cursors, a projection can be used to avoid lookups +in column groups that do not hold columns relevant to the operation. + +The following example will return just the table's primary key (a record +number, in this case) from the index: + +@snippet ex_schema.c Return the table's record number key using an index + +Here is an example of a projection that returns a subset of columns from +the index: + +@snippet ex_schema.c Return a subset of the value columns from an index + +For performance reasons, it may be desirable to include all columns for +a performance-critical operation in an index, so that it is possible to +perform index-only lookups where no column group from the table is +accessed. In this case, all of the "hot" columns should be included in the +index (always list the "real" index key columns first, so they will +determine the sort order). Then, open a cursor on the index that doesn't +return any value columns, and no column group will be accessed. + +@snippet ex_schema.c Access only the index + +Index cursors for column-store objects may not be created using the +record number as the index key (there is no use for a secondary index +on a column-store where the index key is the record number). + +@section schema_examples Code samples + +The code included above was taken from the complete example program +@ex_ref{ex_schema.c}. + +Here is another example program, @ex_ref{ex_call_center.c}. + +@snippet ex_call_center.c call-center decl +@snippet ex_call_center.c call-center work + + */ diff --git a/src/third_party/wiredtiger/src/docs/security.dox b/src/third_party/wiredtiger/src/docs/security.dox new file mode 100644 index 00000000000..331f74d969b --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/security.dox @@ -0,0 +1,25 @@ +/*! @page security Security + +@section directory_permissions Database directory permissions + +All WiredTiger files are stored in the database home directory, and the +WiredTiger database directory should have its permissions set to ensure +database objects are not accessible to users without appropriate +permissions. See @ref home for more information. + +@section file_permissions File permissions + +WiredTiger creates file system objects readable and writable by the +process owner, group and user, as modified by the process' umask value. +The group ownership of created file system objects may vary depending +on the system, and is not controlled by WiredTiger. + +@section environment_variables Environment variables + +Consider security when configuring WiredTiger to use the WIREDTIGER_HOME +environment variable, especially in applications which run with +permissions other than the user's. Such applications are potentially +vulnerable to allowing users access to databases they could not +otherwise access. See @ref home for more information. + +*/ diff --git a/src/third_party/wiredtiger/src/docs/shared-cache.dox b/src/third_party/wiredtiger/src/docs/shared-cache.dox new file mode 100644 index 00000000000..95422caabda --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/shared-cache.dox @@ -0,0 +1,41 @@ +/*! @m_page{{c,java},shared_cache,Per-process shared caches} + +WiredTiger supports sharing a single cache among multiple databases within +a process. + +An application configures a shared cache by specifying a shared_cache name +to the ::wiredtiger_open function. Applications can optionally +set a minimum amount of cache any connection in the pool will be assigned +and the granularity at which the cache pool is redistributed among +connections - called the chunk size. + +The shared cache implementation assigns a certain amount of cache to each +participating database. Each database manages its allocated cache +as it would when not using a shared cache - thus databases using a shared +cache can have different eviction policies. There is a thread that monitors +the cache usage of each database and redistributes the cache among +participants according to where it is most likely to improve performance. +The cache is redistributed in chunks which are of a configurable size. Once +a database has had a chunk of cache added or removed it will be given +time to start effectively using that cache before it is considered for further +adjustment. If a small chunk size is configured it will take longer for the +shared cache to adjust to changes in participants. Reallocation of resources +happens periodically and whenever a database joins the shared cache. + +The reallocation of resources is determined by comparing the amount of eviction +activity in a particular database to that of the other participating databases. + +When a database is opened it will be allocated the amount of cache configured +as the shared cache minimum, regardless of whether the cache pool is currently +fully utilized. Other databases will have their assigned cache size +reduced so the total cache size used will return within the bounds - there +may be a period when the actual usage exceeds the configured maximum. +This is especially likely if many databases join the shared cache in a short +period. When a database is closed any resources it is using are distributed +among the other databases. + +WiredTiger shared cache tuning options can be configured when first opening a +database via ::wiredtiger_open or changed after open using the +WT_CONNECTION::reconfigure method. + + */ diff --git a/src/third_party/wiredtiger/src/docs/signals.dox b/src/third_party/wiredtiger/src/docs/signals.dox new file mode 100644 index 00000000000..a081370159d --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/signals.dox @@ -0,0 +1,11 @@ +/*! @page signals Signal handling + +WiredTiger has no signal handlers and does not modify process signal handler behavior in any way. + +WiredTiger restarts any system calls interrupted by a signal. + +WiredTiger applications exiting as a result of a signal should exit WiredTiger gracefully, closing any open WiredTiger handles, before terminating the process. + +WiredTiger is not re-entrant, and applications using WiredTiger should not make WiredTiger API calls from signal handlers. + +*/ diff --git a/src/third_party/wiredtiger/src/docs/spell.ok b/src/third_party/wiredtiger/src/docs/spell.ok new file mode 100644 index 00000000000..80597302cbb --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/spell.ok @@ -0,0 +1,489 @@ +personal_ws-1.1 en 200 +ACM +APIs +ActiveState +Adler's +Atomicity +BLOBs +CFLAGS +CPPFLAGS +Cheng +Christoph +Collet's +Coverity +Coverity's +DB's +DBTs +Datastore +DbCursor +DbEnv +DbMultiple +EAGAIN +EB +EBUSY +ECMA +EINVAL +ENCRYPTOR +EmpId +Encryptors +Facebook +Facebook's +FlexeLint +FreeBSD +FreeBSD's +GCC +Gawlick +Gimpel +GitHub +Google +Google's +HyperDex +HyperLevelDB +IEC +JavaScript +KMS +LD +LDFLAGS +LIBS +LLVM +LOGREC +LRVv +LSB +LSM +LZ +Lameter +LevelDB +Levyx +MERCHANTABILITY +MVCC's +Makefiles +Mewhort +MongoDB +Multithreaded +NOTFOUND +NUMA +NoSQL +OPTYPE +PRELOAD +README +Rebalance +RedHat +RepMgr +Riak +RocksDB +Roelofs +Rrx +SCons +Seward's +SiH +TXT +URIs +WiredTiger +WiredTiger's +WiredTigerCheckpoint +WiredTigerException +WiredTigerLog +WiredTigerPanicException +WiredTigerRollbackException +WiredTigerStat +WiredTigerTestCase +Yann +Za +aR +abstime +ack'ed +ajn +alloc +allocator +allocators +allocsize +ao +api +apiflags +ar +archiver +arg +async +atomicity +autoconf +autogen +automake +basecfg +basho +bdb +bdbmap +benchmarking +bigram +bindir +bitstring +bool +boolean +booleans +br +btree +btrees +bufs +builtin +builtins +bzip +cachesize +calc +callbk +cd +cdb +cds +changelog +checksum +checksums +ckp +clickable +colgroup +colgroups +combinatorial +command's +comparator +cond +config +configurign +conn +const +control's +copydoc +cpp +crashless +cursortype +customerABC +cv +dN +dNLen +dNOff +dT +dataN +dataitem +dataset +datasets +datastore +dbc +dbformat +dbm +dbt +decl +decrypt +decrypted +del +desc +destructor +destructors +dev +distclean +dl +dll +dlp +dontlock +dp +ds +dsrc +dsync +dt +dumpfile +dup +dups +encrypt +encryptor +encryptors +endcode +endcond +endian +endif +endinternal +english +env +eof +eq +erlang +errno +exe +fadvise +failchk +fd's +fdatasync +fieldname +fileID +fileformats +fileid +filename +filesystem +filesystems +fillfactor +firstname +fnv +forw +fput +freelist +fsync +gcc +gdbm +ge +getKey +getValue +getopt +getter +gid +github +gnuplot +hb +hotbackup +href +hrow +hsearch +html +htmlinclude +huffman +hugepage +hyperleveldb +icount +iflag +indices +init +intl +inuse +io +ip +je +jemalloc +jitter +jni +jrx +json +kb +keyexist +keyfmt +keyid +keyname +keyvalue +kvs +lang +lastname +le +len +leveldb +li +libdir +libhe +libkvs +libtool +libwiredtiger +lmin +ln +loadtext +logc +lookup +lookups +lrtf +lsm +lsn +lt +lz +mailto +mainpage +malloc +malloc'd +marshalled +marshalling +maxintlitem +maxintlpage +maxleafitem +maxleafpage +md +mem +memalloc +memfree +memp +metadata +minkey +mixin +mixins +mkdir +mmap +mpool +mpoolfile +msg +msgs +multi +multiprocess +multithreaded +multithreading +multiversion +mutex +mutexes +mutexing +mvcc +mygcc +mytable +namespace +ndary +ndbm +newsite +nocase +nocasecoll +nodup +noflush +nolocking +nommap +nop +nosql +nosync +notgranted +notyet +nowait +nul +num +numa +nvd +objectsin +ol +oltp +oob +optype +os +outlier +outliers +ovfl +pareto +pcoll +pdf +perf +petabyte +pget +php +png +posix +pre +primary's +printf +printlog +printvalue +priv +proc +pthread +pthreads +putKey +putKeyString +putValue +putValueString +py +qnx +rVv +rdbms +rdlock +readlock +realclean +realloc +realloc'd +rebalance +recno +recnoN +recnum +recoverability +recs +rectype +relinking +rerequests +ret +rf +rmw +ro +rocksdb +rotn +rpc +runnable +runtime +rwlock +sHQ +sHq +scalable +scanf +schemas +scons +secretkey +selectable +seqname +seqno +serializable +sess +skiplist +spinlock +spinlocks +sql +src +ssd +startsync +statlog +str +strerror +strftime +strget +struct +structs +subdatabases +subdirectory +subpage +substring +superset +sys +sz +tRuE +tablename +tcl +tcmalloc +td +th +thang +timestamp +timestamps +todo +toolchain +touchpad +tpc +tpcb +trackpad +transactional +transactionally +tt +txn +txnid +txns +typedef +uint +ul +umask +unallocated +unencoded +unencrypted +unescaped +unicode +uninstall +unittest +untyped +uri +useconds +usr +utf +util +valign +valuefmt +vec +versa +vm +warmup +whitespace +wiredtiger +workQ +writelock +writelocks +wrlock +wtperf +wtstats +xa +yieldcpu +zlib diff --git a/src/third_party/wiredtiger/src/docs/sql-map.dox b/src/third_party/wiredtiger/src/docs/sql-map.dox new file mode 100644 index 00000000000..8f288565436 --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/sql-map.dox @@ -0,0 +1,32 @@ +/*! \page sql_mapping Mapping SQL onto the WiredTiger API + +@todo decide whether to keep this page, if so, fill it out + +An RDBMS has the SQL language, a parser and a query planner. All of them +sit on top of WiredTiger, as they would on top of any key-value store. + +The difference with WiredTiger's API is that the query plan is going to be +based on fundamental operations including: + +# sequential scan through a range; +# join (various flavors); +# sort (group by); +# projection, and so on. + +Those operations would be implemented as different WiredTiger cursor types, +using WT_CONNECTION::add_cursor_type. + +The cursor types could be implemented as one or more extensions, loaded +automatically whenever the database is opened for the first time. Then the +cursor types would always be there, and the query planner just asks for a +cursor that combines together data from various other cursors and comes out as +a set of key/data pairs it can deal with. + +With the right set of special cursor types are implemented, one cursor could +return the SQL result set. That is, combining cursors would implement a +query plan. And that could happen over a socket, mixing local processing +with server-side processing. + +This same interface could deal with many types of compression, transforming +data on the fly in various ways (like views), and so on. + */ diff --git a/src/third_party/wiredtiger/src/docs/statistics.dox b/src/third_party/wiredtiger/src/docs/statistics.dox new file mode 100644 index 00000000000..453da34c51a --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/statistics.dox @@ -0,0 +1,164 @@ +/*! @m_page{{c,java},statistics,Statistics} + +WiredTiger can be configured to maintain a variety of run-time and +data-source statistics. As maintaining statistics may involve updating +shared-memory data structures and traversing disk-based data structures, +configuring statistics may decrease application performance. + +The ::wiredtiger_open \c statistics configuration must be set in order +for statistics to be maintained. When configured to \c none (the +default), no statistics are maintained and attempting to read the +statistics will result in an error. Alternatively, the \c fast +configuration maintains a subset of the statistics which are relatively +inexpensive, and the \c all configuration maintains all statistics +regardless of cost. + +The following example configures WiredTiger to maintain all statistics, +regardless of cost: + +@snippet ex_all.c Statistics configuration + +Statistics are gathered and returned to the application using a statistics +cursor, which returns key/value pairs to the application. See +@ref data_statistics for information about accessing the gathered statistics. + +When the WT_SESSION::open_cursor method is called to open a statistics +cursor, the statistics configuration specifies the statistics to be +gathered, similarly to the ::wiredtiger_open function. The \c fast +configuration gathers the subset of the statistics that are relatively +inexpensive, and the \c all configuration gathers all statistics, +regardless of cost. If no configuration is specified, the current +database statistics configuration is assumed. + +The configuration of the WT_SESSION::open_cursor method must agree with the +database configuration: if the database is configured to maintain \c fast +statistics, attempts to open a statistics cursor in \c all mode will fail. +If the database has been configured to maintain \c all statistics, the +statistics cursor can be configured in either \c fast or \c all modes. +For example, an application might configure the database to maintain all +available statistics, but the application might gather expensive statistics +less frequently than inexpensive ones. + +The following example opens a statistics cursor on the database: + +@snippet ex_all.c Statistics cursor database + +The following example opens a statistics cursor on a table: + +@snippet ex_all.c Statistics cursor table + +The following example opens a statistics cursor on a table, but gathering +only the relatively inexpensive statistics: + +@snippet ex_all.c Statistics cursor table fast + +The WT_SESSION::open_cursor method and the ::wiredtiger_open function +also support the statistics configuration value \c clear. + +When \c clear is specified to the WT_SESSION::open_cursor method, gathered +statistics will be reset, where appropriate, after they are gathered. For +example, a cache size statistic is not cleared because it's not expected to +change rapidly over time, while the count of cursor insert operations will +be cleared. This allows applications to easily monitor changes in the +system over time. + +The following example gathers all statistics for a table, regardless of +cost, and then clears them: + +@snippet ex_all.c Statistics cursor clear configuration + +When \c clear is specified to the ::wiredtiger_open function, gathered +statistics will be reset, where appropriate, after they are gathered. +This applies to all statistics cursors (as if \c clear was configured +when the cursor was opened), as well as statistics logging, when it is +configured. + +The following example configures WiredTiger to maintain only relatively +inexpensive statistics, and to clears statistics after they are gathered +or logged: + +@snippet ex_all.c Statistics clear configuration + +@section statistics_log Statistics logging + +WiredTiger will optionally log database statistics into a file when the +the ::wiredtiger_open \c statistics_log configuration is set. + +The resulting statistics can be displayed using the \c wtstats visualization +tool. For more information, see @ref_single wtstats. + +The following example logs statistics every 30 seconds: + +@snippet ex_all.c Statistics logging + +Each record is formatted as a space-separated timestamp, unsigned 64-bit +value and a variable length string which describes the statistic. + +The timestamp format may be changed with the \c statistics_log.timestamp +configuration string. The \c timestamp value may contain ISO C90 standard +strftime conversion specifications. + +The statistics the database is configured to maintain are logged, that +is, configuring the database to maintain \c fast or \c all statistics +will modify the statistics that will be logged. + +Statistics for specific underlying data sources may be included by adding +a list of data source URIs to the \c statistics_log configuration string: + +@snippet ex_all.c Statistics logging with a table + +Statistics for all underlying data sources of a particular type may be +included by adding a partial data source URI to the \c statistics_log +configuration string: + +@snippet ex_all.c Statistics logging with all tables + +When database statistics are logged, the database home will be the first +space-separated entry for each record in the log file. For example: + +@code +Mar 08 11:38:23 463 /database/home pthread mutex condition wait calls +Mar 08 11:38:23 0 /database/home files currently open +Mar 08 11:38:23 1855437 /database/home total heap memory allocations +Mar 08 11:38:23 1856622 /database/home total heap memory frees +Mar 08 11:38:23 1 /database/home total heap memory re-allocations +Mar 08 11:38:23 472 /database/home total read I/Os +@endcode + +When data source statistics are logged, the data source's URI will be +the first space-separated entry for each record in the log file. For +example: + +@code +Mar 20 10:42:36 21 table:mytable compressed pages written +Mar 20 10:42:36 0 table:mytable page written failed to compress +Mar 20 10:42:36 5 table:mytable page written was too small to compress +Mar 20 10:42:36 586 table:mytable cursor insert calls +Mar 20 10:42:36 0 table:mytable bulk-loaded cursor-insert calls +@endcode + +No statistics are logged for any data source for which a handle is not +currently open in the database, nor will any statistics requiring the +traversal of a tree (as if the \c statistics_fast configuration string +were set). + +The location of the log files may be changed with the \c statistics_log.path +configuration string. The \c path value value may contain ISO C90 standard +strftime conversion specifications. WiredTiger will not create non-existent +directories in the path, they must exist before ::wiredtiger_open is called. + +The following example logs statistics into files named with the month, +day and year: + +@snippet ex_all.c Statistics logging with path + +A Python script that parses the default logging output and uses the +<a href="http://www.gnuplot.info/">gnuplot</a>, utility to generate +Portable Network Graphics (PNG) format graphs is included in the +WiredTiger distribution in the file \c tools/statlog.py. + +@m_if{c} +To interactively examine statistics results, see @ref wtstats. +@m_endif + +*/ diff --git a/src/third_party/wiredtiger/src/docs/style/DoxygenLayout.xml b/src/third_party/wiredtiger/src/docs/style/DoxygenLayout.xml new file mode 100644 index 00000000000..fba7c189509 --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/style/DoxygenLayout.xml @@ -0,0 +1,187 @@ +<doxygenlayout version="1.0"> + <!-- Navigation index tabs for HTML output --> + <navindex> + <tab type="mainpage" visible="yes" title=""/> + <tab type="pages" visible="yes" title="" intro=""/> + <tab type="modules" visible="yes" title="" intro=""/> + <tab type="namespaces" visible="yes" title=""> + <tab type="namespacelist" visible="yes" title="" intro=""/> + <tab type="namespacemembers" visible="yes" title="" intro=""/> + </tab> + <tab type="classes" visible="no" title=""> + <tab type="classlist" visible="yes" title="" intro=""/> + <tab type="classindex" visible="$ALPHABETICAL_INDEX" title=""/> + <tab type="hierarchy" visible="yes" title="" intro=""/> + <tab type="classmembers" visible="yes" title="" intro=""/> + </tab> + <tab type="files" visible="yes" title=""> + <tab type="filelist" visible="yes" title="" intro=""/> + <tab type="globals" visible="yes" title="" intro=""/> + </tab> + <tab type="examples" visible="yes" title="" intro=""/> + <tab type="user" url="community.html" visible="yes" title="Community"/> + <tab type="user" url="license.html" visible="yes" title="License"/> + </navindex> + + <!-- Layout definition for a class page --> + <class> + <briefdescription visible="yes"/> + <includes visible="$SHOW_INCLUDE_FILES"/> + <inheritancegraph visible="$CLASS_GRAPH"/> + <collaborationgraph visible="$COLLABORATION_GRAPH"/> + <allmemberslink visible="yes"/> + <memberdecl> + <nestedclasses visible="yes" title=""/> + <publictypes title=""/> + <publicslots title=""/> + <signals title=""/> + <publicmethods title=""/> + <publicstaticmethods title=""/> + <publicattributes title=""/> + <publicstaticattributes title=""/> + <protectedtypes title=""/> + <protectedslots title=""/> + <protectedmethods title=""/> + <protectedstaticmethods title=""/> + <protectedattributes title=""/> + <protectedstaticattributes title=""/> + <packagetypes title=""/> + <packagemethods title=""/> + <packagestaticmethods title=""/> + <packageattributes title=""/> + <packagestaticattributes title=""/> + <properties title=""/> + <events title=""/> + <privatetypes title=""/> + <privateslots title=""/> + <privatemethods title=""/> + <privatestaticmethods title=""/> + <privateattributes title=""/> + <privatestaticattributes title=""/> + <friends title=""/> + <related title="" subtitle=""/> + <membergroups visible="yes"/> + </memberdecl> + <detaileddescription title=""/> + <memberdef> + <inlineclasses title=""/> + <typedefs title=""/> + <enums title=""/> + <constructors title=""/> + <functions title=""/> + <related title=""/> + <variables title=""/> + <properties title=""/> + <events title=""/> + </memberdef> + <usedfiles visible="$SHOW_USED_FILES"/> + <authorsection visible="yes"/> + </class> + + <!-- Layout definition for a namespace page --> + <namespace> + <briefdescription visible="yes"/> + <memberdecl> + <nestednamespaces visible="yes" title=""/> + <classes visible="yes" title=""/> + <typedefs title=""/> + <enums title=""/> + <functions title=""/> + <variables title=""/> + <membergroups visible="yes"/> + </memberdecl> + <detaileddescription title=""/> + <memberdef> + <inlineclasses title=""/> + <typedefs title=""/> + <enums title=""/> + <functions title=""/> + <variables title=""/> + </memberdef> + <authorsection visible="yes"/> + </namespace> + + <!-- Layout definition for a file page --> + <file> + <briefdescription visible="yes"/> + <includes visible="$SHOW_INCLUDE_FILES"/> + <includegraph visible="$INCLUDE_GRAPH"/> + <includedbygraph visible="$INCLUDED_BY_GRAPH"/> + <sourcelink visible="yes"/> + <memberdecl> + <classes visible="yes" title=""/> + <namespaces visible="yes" title=""/> + <defines title=""/> + <typedefs title=""/> + <enums title=""/> + <functions title=""/> + <variables title=""/> + <membergroups visible="yes"/> + </memberdecl> + <detaileddescription title=""/> + <memberdef> + <inlineclasses title=""/> + <defines title=""/> + <typedefs title=""/> + <enums title=""/> + <functions title=""/> + <variables title=""/> + </memberdef> + <authorsection/> + </file> + + <!-- Layout definition for a group page --> + <group> + <briefdescription visible="yes"/> + <groupgraph visible="$GROUP_GRAPHS"/> + <memberdecl> + <classes visible="yes" title=""/> + <namespaces visible="yes" title=""/> + <nestedgroups visible="yes" title=""/> + <files visible="yes" title=""/> + <defines title=""/> + <typedefs title=""/> + <enums title=""/> + <enumvalues title=""/> + <functions title=""/> + <variables title=""/> + <signals title=""/> + <publicslots title=""/> + <protectedslots title=""/> + <privateslots title=""/> + <events title=""/> + <properties title=""/> + <friends title=""/> + <membergroups visible="yes"/> + </memberdecl> + <detaileddescription title=""/> + <memberdef> + <pagedocs/> + <inlineclasses title=""/> + <defines title=""/> + <typedefs title=""/> + <enums title=""/> + <enumvalues title=""/> + <functions title=""/> + <variables title=""/> + <signals title=""/> + <publicslots title=""/> + <protectedslots title=""/> + <privateslots title=""/> + <events title=""/> + <properties title=""/> + <friends title=""/> + </memberdef> + <authorsection visible="yes"/> + </group> + + <!-- Layout definition for a directory page --> + <directory> + <briefdescription visible="yes"/> + <directorygraph visible="yes"/> + <memberdecl> + <files visible="yes"/> + </memberdecl> + <detaileddescription title=""/> + </directory> +</doxygenlayout> diff --git a/src/third_party/wiredtiger/src/docs/style/background_navigation.png b/src/third_party/wiredtiger/src/docs/style/background_navigation.png Binary files differnew file mode 100644 index 00000000000..d59e96ddef6 --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/style/background_navigation.png diff --git a/src/third_party/wiredtiger/src/docs/style/doxygen.png b/src/third_party/wiredtiger/src/docs/style/doxygen.png Binary files differnew file mode 100644 index 00000000000..f0a274bbaff --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/style/doxygen.png diff --git a/src/third_party/wiredtiger/src/docs/style/footer.html b/src/third_party/wiredtiger/src/docs/style/footer.html new file mode 100644 index 00000000000..e5a7b30eef5 --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/style/footer.html @@ -0,0 +1,16 @@ +<!-- start footer part --> +<!--BEGIN GENERATE_TREEVIEW--> +<div id="nav-path" class="navpath"><!-- id is needed for treeview function! --> + <ul> + $navpath + <li class="footer">Copyright (c) 2008-2016 MongoDB, Inc. All rights reserved. Contact <a href="mailto:info@wiredtiger.com">info@wiredtiger.com</a> for more information.</li> + </ul> +</div> +<!--END GENERATE_TREEVIEW--> +<!--BEGIN !GENERATE_TREEVIEW--> +<hr class="footer"/><address class="footer"><small> +Copyright (c) 2008-2016 MongoDB, Inc. All rights reserved. Contact <a href="mailto:info@wiredtiger.com">info@wiredtiger.com</a> for more information. +</small></address> +<!--END !GENERATE_TREEVIEW--> +</body> +</html> diff --git a/src/third_party/wiredtiger/src/docs/style/header.html b/src/third_party/wiredtiger/src/docs/style/header.html new file mode 100644 index 00000000000..0c045a750d1 --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/style/header.html @@ -0,0 +1,57 @@ +<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> +<html xmlns="http://www.w3.org/1999/xhtml"> +<head> +<meta http-equiv="Content-Type" content="text/xhtml;charset=UTF-8"/> +<meta http-equiv="X-UA-Compatible" content="IE=9"/> +<!--BEGIN PROJECT_NAME--><title>$projectname: $title</title><!--END PROJECT_NAME--> +<!--BEGIN !PROJECT_NAME--><title>$title</title><!--END !PROJECT_NAME--> +<link href="$relpath$tabs.css" rel="stylesheet" type="text/css"/> +<script type="text/javascript" src="$relpath$jquery.js"></script> +<script type="text/javascript" src="$relpath$dynsections.js"></script> +$treeview +$search +$mathjax +<link href="$relpath$$stylesheet" rel="stylesheet" type="text/css" /> +$extrastylesheet +</head> +<body> +<div id="top"><!-- do not remove this div, it is closed by doxygen! --> + +<!--BEGIN TITLEAREA--> +<div id="titlearea"> +<table cellspacing="0" cellpadding="0"> + <tbody> + <tr style="height: 56px;"> + <!--BEGIN PROJECT_LOGO--> + <td id="projectlogo"><a href="http://wiredtiger.com/"><img alt="Logo" src="$relpath$$projectlogo" alt="WiredTiger" /></a></td> + <!--END PROJECT_LOGO--> + <!--BEGIN PROJECT_NAME--> + <td style="padding-left: 0.5em;"> + <div id="projectname"> + <!--BEGIN PROJECT_NUMBER--> <span id="projectnumber">$projectnumber</span><!--END PROJECT_NUMBER--> + </div> + <!--BEGIN PROJECT_BRIEF--><div id="projectbrief">$projectbrief</div><!--END PROJECT_BRIEF--> + </td> + <!--END PROJECT_NAME--> + <!--BEGIN !PROJECT_NAME--> + <!--BEGIN PROJECT_BRIEF--> + <td style="padding-left: 0.5em;"> + <div id="projectbrief">$projectbrief</div> + </td> + <!--END PROJECT_BRIEF--> + <!--END !PROJECT_NAME--> + <!--BEGIN DISABLE_INDEX--> + <!--BEGIN SEARCHENGINE--> + <td>$searchbox</td> + <!--END SEARCHENGINE--> + <!--END DISABLE_INDEX--> + </tr> + </tbody> +</table> +</div> +<div class="banner"> + <a href="https://github.com/wiredtiger/wiredtiger">Fork me on GitHub</a> + <a class="last" href="http://groups.google.com/group/wiredtiger-users">Join my user group</a> +</div> +<!--END TITLEAREA--> +<!-- end header part --> diff --git a/src/third_party/wiredtiger/src/docs/style/img_downArrow.png b/src/third_party/wiredtiger/src/docs/style/img_downArrow.png Binary files differnew file mode 100644 index 00000000000..024a03c2654 --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/style/img_downArrow.png diff --git a/src/third_party/wiredtiger/src/docs/style/javadoc.css b/src/third_party/wiredtiger/src/docs/style/javadoc.css new file mode 100644 index 00000000000..f345b05b158 --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/style/javadoc.css @@ -0,0 +1,35 @@ +/* Javadoc style sheet */ + +/* Define colors, fonts and other style attributes here to override the defaults */ + +/* Page background color */ +body { background-color: #FFFFFF } + +/* Table colors */ +.TableHeadingColor { background: #CCCCFF } /* Dark mauve */ +.TableSubHeadingColor { background: #EEEEFF } /* Light mauve */ +.TableRowColor { background: #FFFFFF } /* White */ + +/* Font used in left-hand frame lists */ +.FrameTitleFont { font-size: 10pts; font-family: Helvetica, Arial, san-serif } +.FrameHeadingFont { font-size: 10pts; font-family: Helvetica, Arial, san-serif } +.FrameItemFont { font-size: 10pts; font-family: Helvetica, Arial, san-serif } + +/* Example of smaller, sans-serif font in frames */ +/* .FrameItemFont { font-size: 10pt; font-family: Helvetica, Arial, sans-serif } */ + +/* Navigation bar fonts and colors */ +.NavBarCell1 { background-color:#EEEEFF;} /* Red */ +.NavBarCell1Rev { background-color:#00008B;} /* Light red */ +.NavBarFont1 { font-family: Arial, Helvetica, sans-serif; color:#000000;} +.NavBarFont1Rev { font-family: Arial, Helvetica, sans-serif; color:#FFFFFF;} + +.NavBarCell2 { font-family: Arial, Helvetica, sans-serif; background-color:#FFFFFF;} +.NavBarCell3 { font-family: Arial, Helvetica, sans-serif; background-color:#FFFFFF;} + +pre.code +{ + border: solid thin; + padding: 2px; + background-color: #ffffcc; +} diff --git a/src/third_party/wiredtiger/src/docs/style/tabs.css b/src/third_party/wiredtiger/src/docs/style/tabs.css new file mode 100644 index 00000000000..c543eb00429 --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/style/tabs.css @@ -0,0 +1,128 @@ +/* tabs styles, based on http://www.alistapart.com/articles/slidingdoors */ + +div.navigation { + display:block; + float:left; + clear:both; + width:100%; + padding-top:10px; + padding-bottom:30px; + border-bottom:1px dotted #E0E0E0; + height: 60px; + background:url(background_navigation.png); +} + +div.contents { + float:left; + clear:both; + +} + +address { + display:none; +} + +DIV.tabs +{ + float : left; + width : 100%; + margin-bottom : 4px; +} + +DIV.tabs UL +{ + margin : 0px; + padding-left : 0px; + list-style : none; +} + +DIV.tabs LI, DIV.tabs FORM +{ + display : inline; + margin : 0px; + padding : 0px; + background-color:#FFFFFF; +} + +DIV.tabs FORM +{ + float : right; +} + +DIV.tabs A +{ + float : left; + font-size : 80%; + font-weight : bold; + text-decoration : none; +} + +DIV.tabs A:hover +{ + background-position: 100% -150px; +} + +DIV.tabs A:link, DIV.tabs A:visited, +DIV.tabs A:active, DIV.tabs A:hover +{ + color: #1A419D; +} + +DIV.tabs SPAN +{ + float : left; + display : block; + padding : 5px 9px; + white-space : nowrap; + border-top:1px dotted #E0E0E0; + border-bottom:1px dotted #E0E0E0; +} + +DIV.tabs #MSearchBox +{ + float : right; + display : inline; + font-size : 1em; +} + +DIV.tabs TD +{ + font-size : 80%; + font-weight : bold; + text-decoration : none; +} + + + +/* Commented Backslash Hack hides rule from IE5-Mac \*/ +DIV.tabs SPAN {float : none;} +/* End IE5-Mac hack */ + +DIV.tabs A:hover SPAN +{ + background-position: 0% -150px; +} + +DIV.tabs LI.current A +{ + background-position: 100% -150px; + border-width : 0px; + background-color: #F0F0F0; + background: #F0F0F0 url(img_downArrow.png) center bottom no-repeat; +} + +DIV.tabs LI.current SPAN +{ + background-position: 0% -150px; + padding-bottom : 6px; +} + +DIV.navpath +{ + background : none; + border : none; + border-bottom : 1px solid #84B0C7; + text-align : center; + margin : 2px; + padding : 2px; +} diff --git a/src/third_party/wiredtiger/src/docs/style/wiredtiger.css b/src/third_party/wiredtiger/src/docs/style/wiredtiger.css new file mode 100644 index 00000000000..b16b21f2029 --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/style/wiredtiger.css @@ -0,0 +1,71 @@ +/* WiredTiger-specific styling. */ +body, table, div, p, dl { + font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; + color: #373737; + font-size: 13px; + line-height: 1.3; +} + +h1 { + font-size: 120%; +} + +h2 { + font-size: 110%; +} + +/* @group Link Styling */ + +a { + color: #3D578C; +} + +.tablist { + width: 800px; +} + +.tablist a[href="pages.html"] { + display: none; +} + +.contents a:visited { + color: #4665A2; +} + +#projectlogo { + vertical-align: middle; +} + +.banner { + position: absolute; top: -2px; right: 1em; + overflow: hidden; + background-color: #e57504; + background-image: -webkit-linear-gradient(top, #dc7202, #ee7906); + background-image: -moz-linear-gradient(top, #dc7202, #ee7906); + background-image: -ms-linear-gradient(top, #dc7202, #ee7906); + border-radius: 0 0 8px 8px; + -moz-border-radius: 0 0 8px 8px; + box-shadow: 0 0 1em #888; +} + +.banner a { + border: 1px solid #ebaa65; + border-top-style: none; + color: #fff; + display: block; + font: bold 9pt sans-serif; + margin: 0.05em 0 0.05em 0; + padding: 0.5em 2em; + text-align: center; + text-decoration: none; + text-shadow: 0 0 0.5em #444; +} + +.banner a.last { + border: none; +} + +@media print +{ + .banner { display: none; } +} diff --git a/src/third_party/wiredtiger/src/docs/testing.dox b/src/third_party/wiredtiger/src/docs/testing.dox new file mode 100644 index 00000000000..7d454d54212 --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/testing.dox @@ -0,0 +1,77 @@ +/*! +@page testing WiredTiger testing + +WiredTiger uses a combination of several different tools and test +programs for testing WiredTiger. + +WiredTiger testing is done automatically using the <a +href="http://jenkins-ci.org/">Jenkins</a> continuous integration testing +framework. This allows us to be confident we don't introduce problems +in a non-standard configuration, or performance regressions. + +@section test_suite Unit Test Suite + +WiredTiger's primary functionality and regression testing is done in a +Python unit test suite (found in the source tree under \c test/suite). + +The WiredTiger Python test suite includes approximately 10,000 separate +tests which are run on all platforms supported by WiredTiger. Each test +is intended to test a single operation in a reproducible way, making it +easy to diagnose errors. The test suite runs multiple test cases in +parallel, allowing it to be run in a relatively short period of time. + +The WiredTiger unit test suite includes tests that cover: +- WiredTiger functionality (for example, cursors, transactions and recovery), +- Combinations of WiredTiger's configuration settings and APIs, +- Bug regression tests. + +The WiredTiger Python test suite is built using the WiredTiger Python +API and the Python unittest functionality (the test suite requires at +least Python version 2.6). + +The WiredTiger test suite automatically runs as part of every commit +into the WiredTiger GitHub source tree. + +@section testing_performance Performance Testing + +Performance testing is primarily done using the \c bench/wtperf utility. +A variety of database configurations are run based on the scripts in \c +bench/wtperf/runners. + +WiredTiger performance tests are automatically run as part of each +commit into the \c develop branch of the WiredTiger GitHub source tree +and compared against previous runs to detect performance regressions. + +@section testing_stress Stress Testing + +Stress testing is primarily done using the \c test/format utility. This +test program randomly configures a database and then runs some number +of randomly selected operations, using some number of randomly selected +threads, on that database. WiredTiger stress testing is run continuously +on the WiredTiger GitHub \c develop branch. + +@section testing_threads Concurrency testing + +Concurrency testing is primarily done using the \c test/format utility. +Additionally, the \c test/thread and \c test/fops test utilities test +specific heavily threaded operations. WiredTiger concurrency testing +is run continuously on the WiredTiger GitHub \c develop branch. + +@section testing_static Static analysis + +WiredTiger static analysis is done using three tools: + +- The <a href="https://www.coverity.com/">Coverity, +Inc.</a>software analysis tool; current results and historic defect +reports are available at <a +href="https://scan.coverity.com/projects/1018">Coverity's WiredTiger +page</a>. + +- The Gimpel Software <a +href="http://www.gimpel.com/html/flex.htm">FlexeLint</a> implementation +of the UNIX lint tool. + +- The University of Illinois LLVM project's <a +href="http://clang-analyzer.llvm.org/">Clang Static Analyzer</a>. + + */ diff --git a/src/third_party/wiredtiger/src/docs/threads.dox b/src/third_party/wiredtiger/src/docs/threads.dox new file mode 100644 index 00000000000..16b0d7940da --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/threads.dox @@ -0,0 +1,46 @@ +/*! @m_page{{c,java},threads,Multithreading} + +All WT_CONNECTION methods are thread safe, and WT_CONNECTION handles can +be shared between threads. Applications typically open a single +connection to each database, per process. Multi-threaded programs +must wait for all other threads to exit before closing the WT_CONNECTION +handle because that will implicitly close all other handles. +Alternatively, a thread can set an event handler using +WT_CONNECTION::open_session to be notified when a WT_CONNECTION close is +in progress. + +WT_SESSION and WT_CURSOR methods are not thread safe and WT_SESSION and +WT_CURSOR handles cannot be accessed concurrently by multiple threads. +Applications typically open one WT_SESSION handle for each thread +accessing a database, and then one or more WT_CURSOR handles within the +session. + +WT_SESSION and WT_CURSOR methods may be accessed by different threads +serially (for example, a pool of threads managed by the application with +a set of shared session or cursor handles). There is no thread-local +state in WiredTiger, and no built-in synchronization of session or +cursor handles, either, so if multiple threads access a session or +cursor handle, access must be serialized by the application. + +WT_ASYNC_OP methods are not thread-safe, and must be accessed by only a +single thread at a time. WT_ASYNC_OP methods may be accessed by +different threads serially (and that is expected to happen when the +asynchronous callback function runs). + +@section threads_example Code samples + +The code below is taken from the complete example program +@ex_ref{ex_thread.c}. + +This is an example of a thread entry point. A new session is opened for +the thread and used for all operations within that thread. + +@snippet ex_thread.c thread scan + +Here is the main function that starts the threads. It opens a single +connection, shared between the threads, and closes the connection after +waiting for all of the threads to exit. + +@snippet ex_thread.c thread main + + */ diff --git a/src/third_party/wiredtiger/src/docs/tools/doxfilter b/src/third_party/wiredtiger/src/docs/tools/doxfilter new file mode 100755 index 00000000000..298e23f6cbb --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/tools/doxfilter @@ -0,0 +1,30 @@ +#!/bin/sh +# +# Public Domain 2008-2012 WiredTiger, Inc. +# +# This is free and unencumbered software released into the public domain. +# +# Anyone is free to copy, modify, publish, use, compile, sell, or +# distribute this software, either in source code form or as a compiled +# binary, for any purpose, commercial or non-commercial, and by any +# means. +# +# In jurisdictions that recognize copyright laws, the author or authors +# of this software dedicate any and all copyright interest in the +# software to the public domain. We make this dedication for the benefit +# of the public at large and to the detriment of our heirs and +# successors. We intend this dedication to be an overt act of +# relinquishment in perpetuity of all present and future rights to this +# software under copyright law. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +# IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. + +tooldir=`dirname $0` + +python $tooldir/doxfilter.py "$@" diff --git a/src/third_party/wiredtiger/src/docs/tools/doxfilter.py b/src/third_party/wiredtiger/src/docs/tools/doxfilter.py new file mode 100755 index 00000000000..b2d5f857df1 --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/tools/doxfilter.py @@ -0,0 +1,194 @@ +#!/usr/bin/env python +# +# Public Domain 2014-2016 MongoDB, Inc. +# Public Domain 2008-2014 WiredTiger, Inc. +# +# This is free and unencumbered software released into the public domain. +# +# Anyone is free to copy, modify, publish, use, compile, sell, or +# distribute this software, either in source code form or as a compiled +# binary, for any purpose, commercial or non-commercial, and by any +# means. +# +# In jurisdictions that recognize copyright laws, the author or authors +# of this software dedicate any and all copyright interest in the +# software to the public domain. We make this dedication for the benefit +# of the public at large and to the detriment of our heirs and +# successors. We intend this dedication to be an overt act of +# relinquishment in perpetuity of all present and future rights to this +# software under copyright law. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +# IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. + + +# This Python script is run as part of generating the documentation for the +# WiredTiger reference manual. It changes comments to Javadoc style +# (i.e., from "/*!" to "/**"), because the latter are configured to not +# search for brief descriptions at the beginning of pages. +# It also processes any page marked with @m_page specially to create +# multiple per-language versions of the page. + +import re, sys + +progname = 'doxfilter.py' +linenum = 0 +filename = '<unknown>' + +def err(arg): + sys.stderr.write(filename + ':' + str(linenum) + ': ERROR: ' + arg + '\n') + sys.exit(1) + +def java_post_substitutions(source): + result = source + for datatype in [['WT_CONNECTION', 'Connection'], + ['WT_CURSOR', 'Cursor'], + ['WT_SESSION', 'Session'], + ['WT_ASYNC_OPTYPE', 'AsyncOpType'], + ['WT_ASYNC_OP', 'AsyncOp']]: + fromdt = datatype[0] + todt = datatype[1] + + # e.g. replace("WT_CONNECTION::", "Connection."). + # replace("WT_CONNECTION", "Connection") + # replace("::WT_CONNECTION", "Connection") + # etc. + result = result.replace(fromdt + '::', todt + '.') + result = re.sub(r':*' + fromdt, todt, result) + + # We fix back any 'ref' entries, since we don't have + # many java versions of these refered-to pages. + # + # Ideally, we'd have a way ('@m_ref'?) to indicate which + # ones could refer to a Java version. + result = result.replace('ref ' + todt + '.', 'ref ' + fromdt + '::') + result = result.replace('::wiredtiger_open', '\c wiredtiger.open') + return result + +def process_lang(lang, lines): + result = '' + lang_ext = '.' + lang + class_suffix = None + if lang == 'c': + lang_suffix = "" + lang_desc="" + elif lang == 'java': + lang_suffix = "_lang_java" + lang_desc=" in Java" + else: + err('@m_page contains illegal lang: ' + lang) + condstack = [True] + linenum = 0 + mif_pat = re.compile(r'^\s*@m_if{([^,}]*)}') + melse_pat = re.compile(r'^\s*@m_else\s*$') + mendif_pat = re.compile(r'^\s*@m_endif\s*$') + mpage_pat = re.compile(r'@m_page{{([^}]*)},([^,}]*),([^}]*)}') + mpage_rep = r'@page \2' + lang_suffix + r' \3 ' + lang_desc + ref_pat = re.compile(r'@ref\s+(\w*)') + ref_rep = r'@ref \1' + lang_suffix + snip_pat = re.compile(r'@snippet ex_([^.]*)[.]c\s+(.*)') + snip_rep = r'@snippet ex_\1' + lang_ext + r' \2' + section_pat = re.compile(r'(^@\w*section)\s+(\w*)') + section_rep = r'\1 \2' + lang_suffix + subpage_pat = re.compile(r'@subpage\s+(\w*)') + subpage_rep = r'@subpage \1' + lang_suffix + exref_pat = re.compile(r'@ex_ref{ex_([^.]*)[.]c}') + if lang == 'c': + exref_rep = r'@ex_ref{ex_\1' + lang_ext + '}' + else: + # Though we have java examples, we don't have references + # to them working yet, so strip the @ex_ref. + exref_rep = r'ex_\1' + lang_ext + + # Any remaining @m_foo{...} aliases are + # diverted to @c_foo{...} or @java_foo{...} + mgeneric_pat = re.compile(r'@m_([^ }]*)') + mgeneric_rep = r'@' + lang + r'_\1' + for line in lines: + linenum += 1 + if lang != 'c': + line = re.sub(exref_pat, exref_rep, line) + line = re.sub(ref_pat, ref_rep, line) + line = re.sub(section_pat, section_rep, line) + line = re.sub(snip_pat, snip_rep, line) + line = re.sub(mpage_pat, mpage_rep, line) + line = re.sub(subpage_pat, subpage_rep, line) + if '@m_if' in line: + m = re.search(mif_pat, line) + if not m: + err('@m_if incorrect syntax') + iflang = m.groups()[0] + if iflang != 'java' and iflang != 'c': + err('@m_if unknown language') + condstack.append(iflang == lang) + elif '@m_else' in line: + if not re.search(melse_pat, line): + err('@m_else has extraneous stuff') + if len(condstack) <= 1: + err('@m_else missing if') + condstack[-1] = not condstack[-1] + elif '@m_endif' in line: + if not re.search(mendif_pat, line): + err('@m_endif has extraneous stuff') + if len(condstack) <= 1: + err('@m_endif missing if') + condstack.pop() + else: + if condstack[-1]: + # Do generic @m_... macros last + line = re.sub(mgeneric_pat, + mgeneric_rep, line) + result += line + '\n' + if lang == 'java': + result = java_post_substitutions(result) + if len(condstack) != 1: + err('non matching @m_if/@m_endif') + return result + +# Collect all lines between @m_page and the comment end to +# be processed multiple times, once for each language. +def process_multilang(source): + result = '' + in_mpage = False + mpage_content = [] + mpage_pat = re.compile(r'@m_page{{([^}]*)},([^,}]*),([^}]*)}') + mpage_end_pat = re.compile(r'\*/') + for line in source.split('\n'): + m = re.search(mpage_pat, line) + if line.count('@m_page') > 0 and not m: + err('@m_page incorrect syntax') + if m: + if in_mpage: + err('multiple @m_page without end of comment') + else: + in_mpage = True + langs = m.groups()[0].split(',') + if in_mpage: + mpage_content.append(line) + else: + result += line + '\n' + if re.search(mpage_end_pat, line): + if in_mpage: + in_mpage = False + for lang in langs: + result += process_lang(lang, mpage_content) + mpage_content = [] + return result + +def process(source): + source = source.replace(r'/*!', r'/**') + if '@m_' in source: + source = process_multilang(source) + return source + +if __name__ == '__main__': + for f in sys.argv[1:]: + filename = f + with open(f, 'r') as infile: + sys.stdout.write(process(infile.read())) + sys.exit(0) diff --git a/src/third_party/wiredtiger/src/docs/tools/doxypy.py b/src/third_party/wiredtiger/src/docs/tools/doxypy.py new file mode 100755 index 00000000000..54fef5f03a5 --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/tools/doxypy.py @@ -0,0 +1,414 @@ +#!/usr/bin/env python + +__applicationName__ = "doxypy" +__blurb__ = """ +doxypy is an input filter for Doxygen. It preprocesses python +files so that docstrings of classes and functions are reformatted +into Doxygen-conform documentation blocks. +""" + +__doc__ = __blurb__ + \ +""" +In order to make Doxygen preprocess files through doxypy, simply +add the following lines to your Doxyfile: + FILTER_SOURCE_FILES = YES + INPUT_FILTER = "python /path/to/doxypy.py" +""" + +__version__ = "0.4.2" +__date__ = "14th October 2009" +__website__ = "http://code.foosel.org/doxypy" + +__author__ = ( + "Philippe 'demod' Neumann (doxypy at demod dot org)", + "Gina 'foosel' Haeussge (gina at foosel dot net)" +) + +__licenseName__ = "GPL v2" +__license__ = """This program is free software: you can redistribute it and/or modify +it under the terms of the GNU General Public License as published by +the Free Software Foundation, either version 2 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details. + +You should have received a copy of the GNU General Public License +along with this program. If not, see <http://www.gnu.org/licenses/>. +""" + +import sys +import re + +from optparse import OptionParser, OptionGroup + +class FSM(object): + """Implements a finite state machine. + + Transitions are given as 4-tuples, consisting of an origin state, a target + state, a condition for the transition (given as a reference to a function + which gets called with a given piece of input) and a pointer to a function + to be called upon the execution of the given transition. + """ + + """ + @var transitions holds the transitions + @var current_state holds the current state + @var current_input holds the current input + @var current_transition hold the currently active transition + """ + + def __init__(self, start_state=None, transitions=[]): + self.transitions = transitions + self.current_state = start_state + self.current_input = None + self.current_transition = None + + def setStartState(self, state): + self.current_state = state + + def addTransition(self, from_state, to_state, condition, callback): + self.transitions.append([from_state, to_state, condition, callback]) + + def makeTransition(self, input): + """Makes a transition based on the given input. + + @param input input to parse by the FSM + """ + for transition in self.transitions: + [from_state, to_state, condition, callback] = transition + if from_state == self.current_state: + match = condition(input) + if match: + self.current_state = to_state + self.current_input = input + self.current_transition = transition + if options.debug: + print >>sys.stderr, "# FSM: executing (%s -> %s) for line '%s'" % (from_state, to_state, input) + callback(match) + return + +class Doxypy(object): + def __init__(self): + string_prefixes = "[uU]?[rR]?" + + self.start_single_comment_re = re.compile("^\s*%s(''')" % string_prefixes) + self.end_single_comment_re = re.compile("(''')\s*$") + + self.start_double_comment_re = re.compile("^\s*%s(\"\"\")" % string_prefixes) + self.end_double_comment_re = re.compile("(\"\"\")\s*$") + + self.single_comment_re = re.compile("^\s*%s(''').*(''')\s*$" % string_prefixes) + self.double_comment_re = re.compile("^\s*%s(\"\"\").*(\"\"\")\s*$" % string_prefixes) + + self.defclass_re = re.compile("^(\s*)(def .+:|class .+:)") + self.empty_re = re.compile("^\s*$") + self.hashline_re = re.compile("^\s*#.*$") + self.importline_re = re.compile("^\s*(import |from .+ import)") + + self.multiline_defclass_start_re = re.compile("^(\s*)(def|class)(\s.*)?$") + self.multiline_defclass_end_re = re.compile(":\s*$") + + ## Transition list format + # ["FROM", "TO", condition, action] + transitions = [ + ### FILEHEAD + + # single line comments + ["FILEHEAD", "FILEHEAD", self.single_comment_re.search, self.appendCommentLine], + ["FILEHEAD", "FILEHEAD", self.double_comment_re.search, self.appendCommentLine], + + # multiline comments + ["FILEHEAD", "FILEHEAD_COMMENT_SINGLE", self.start_single_comment_re.search, self.appendCommentLine], + ["FILEHEAD_COMMENT_SINGLE", "FILEHEAD", self.end_single_comment_re.search, self.appendCommentLine], + ["FILEHEAD_COMMENT_SINGLE", "FILEHEAD_COMMENT_SINGLE", self.catchall, self.appendCommentLine], + ["FILEHEAD", "FILEHEAD_COMMENT_DOUBLE", self.start_double_comment_re.search, self.appendCommentLine], + ["FILEHEAD_COMMENT_DOUBLE", "FILEHEAD", self.end_double_comment_re.search, self.appendCommentLine], + ["FILEHEAD_COMMENT_DOUBLE", "FILEHEAD_COMMENT_DOUBLE", self.catchall, self.appendCommentLine], + + # other lines + ["FILEHEAD", "FILEHEAD", self.empty_re.search, self.appendFileheadLine], + ["FILEHEAD", "FILEHEAD", self.hashline_re.search, self.appendFileheadLine], + ["FILEHEAD", "FILEHEAD", self.importline_re.search, self.appendFileheadLine], + ["FILEHEAD", "DEFCLASS", self.defclass_re.search, self.resetCommentSearch], + ["FILEHEAD", "DEFCLASS_MULTI", self.multiline_defclass_start_re.search, self.resetCommentSearch], + ["FILEHEAD", "DEFCLASS_BODY", self.catchall, self.appendFileheadLine], + + ### DEFCLASS + + # single line comments + ["DEFCLASS", "DEFCLASS_BODY", self.single_comment_re.search, self.appendCommentLine], + ["DEFCLASS", "DEFCLASS_BODY", self.double_comment_re.search, self.appendCommentLine], + + # multiline comments + ["DEFCLASS", "COMMENT_SINGLE", self.start_single_comment_re.search, self.appendCommentLine], + ["COMMENT_SINGLE", "DEFCLASS_BODY", self.end_single_comment_re.search, self.appendCommentLine], + ["COMMENT_SINGLE", "COMMENT_SINGLE", self.catchall, self.appendCommentLine], + ["DEFCLASS", "COMMENT_DOUBLE", self.start_double_comment_re.search, self.appendCommentLine], + ["COMMENT_DOUBLE", "DEFCLASS_BODY", self.end_double_comment_re.search, self.appendCommentLine], + ["COMMENT_DOUBLE", "COMMENT_DOUBLE", self.catchall, self.appendCommentLine], + + # other lines + ["DEFCLASS", "DEFCLASS", self.empty_re.search, self.appendDefclassLine], + ["DEFCLASS", "DEFCLASS", self.defclass_re.search, self.resetCommentSearch], + ["DEFCLASS", "DEFCLASS_MULTI", self.multiline_defclass_start_re.search, self.resetCommentSearch], + ["DEFCLASS", "DEFCLASS_BODY", self.catchall, self.stopCommentSearch], + + ### DEFCLASS_BODY + + ["DEFCLASS_BODY", "DEFCLASS", self.defclass_re.search, self.startCommentSearch], + ["DEFCLASS_BODY", "DEFCLASS_MULTI", self.multiline_defclass_start_re.search, self.startCommentSearch], + ["DEFCLASS_BODY", "DEFCLASS_BODY", self.catchall, self.appendNormalLine], + + ### DEFCLASS_MULTI + ["DEFCLASS_MULTI", "DEFCLASS", self.multiline_defclass_end_re.search, self.appendDefclassLine], + ["DEFCLASS_MULTI", "DEFCLASS_MULTI", self.catchall, self.appendDefclassLine], + ] + + self.fsm = FSM("FILEHEAD", transitions) + self.outstream = sys.stdout + + self.output = [] + self.comment = [] + self.filehead = [] + self.defclass = [] + self.indent = "" + + def __closeComment(self): + """Appends any open comment block and triggering block to the output.""" + + if options.autobrief: + if len(self.comment) == 1 \ + or (len(self.comment) > 2 and self.comment[1].strip() == ''): + self.comment[0] = self.__docstringSummaryToBrief(self.comment[0]) + + if self.comment: + block = self.makeCommentBlock() + self.output.extend(block) + + if self.defclass: + self.output.extend(self.defclass) + + def __docstringSummaryToBrief(self, line): + """Adds \\brief to the docstrings summary line. + + A \\brief is prepended, provided no other doxygen command is at the + start of the line. + """ + stripped = line.strip() + if stripped and not stripped[0] in ('@', '\\'): + return "\\brief " + line + else: + return line + + def __flushBuffer(self): + """Flushes the current outputbuffer to the outstream.""" + if self.output: + try: + if options.debug: + print >>sys.stderr, "# OUTPUT: ", self.output + print >>self.outstream, "\n".join(self.output) + self.outstream.flush() + except IOError: + # Fix for FS#33. Catches "broken pipe" when doxygen closes + # stdout prematurely upon usage of INPUT_FILTER, INLINE_SOURCES + # and FILTER_SOURCE_FILES. + pass + self.output = [] + + def catchall(self, input): + """The catchall-condition, always returns true.""" + return True + + def resetCommentSearch(self, match): + """Restarts a new comment search for a different triggering line. + + Closes the current commentblock and starts a new comment search. + """ + if options.debug: + print >>sys.stderr, "# CALLBACK: resetCommentSearch" + self.__closeComment() + self.startCommentSearch(match) + + def startCommentSearch(self, match): + """Starts a new comment search. + + Saves the triggering line, resets the current comment and saves + the current indentation. + """ + if options.debug: + print >>sys.stderr, "# CALLBACK: startCommentSearch" + self.defclass = [self.fsm.current_input] + self.comment = [] + self.indent = match.group(1) + + def stopCommentSearch(self, match): + """Stops a comment search. + + Closes the current commentblock, resets the triggering line and + appends the current line to the output. + """ + if options.debug: + print >>sys.stderr, "# CALLBACK: stopCommentSearch" + self.__closeComment() + + self.defclass = [] + self.output.append(self.fsm.current_input) + + def appendFileheadLine(self, match): + """Appends a line in the FILEHEAD state. + + Closes the open comment block, resets it and appends the current line. + """ + if options.debug: + print >>sys.stderr, "# CALLBACK: appendFileheadLine" + self.__closeComment() + self.comment = [] + self.output.append(self.fsm.current_input) + + def appendCommentLine(self, match): + """Appends a comment line. + + The comment delimiter is removed from multiline start and ends as + well as singleline comments. + """ + if options.debug: + print >>sys.stderr, "# CALLBACK: appendCommentLine" + (from_state, to_state, condition, callback) = self.fsm.current_transition + + # single line comment + if (from_state == "DEFCLASS" and to_state == "DEFCLASS_BODY") \ + or (from_state == "FILEHEAD" and to_state == "FILEHEAD"): + # remove comment delimiter from begin and end of the line + activeCommentDelim = match.group(1) + line = self.fsm.current_input + self.comment.append(line[line.find(activeCommentDelim)+len(activeCommentDelim):line.rfind(activeCommentDelim)]) + + if (to_state == "DEFCLASS_BODY"): + self.__closeComment() + self.defclass = [] + # multiline start + elif from_state == "DEFCLASS" or from_state == "FILEHEAD": + # remove comment delimiter from begin of the line + activeCommentDelim = match.group(1) + line = self.fsm.current_input + self.comment.append(line[line.find(activeCommentDelim)+len(activeCommentDelim):]) + # multiline end + elif to_state == "DEFCLASS_BODY" or to_state == "FILEHEAD": + # remove comment delimiter from end of the line + activeCommentDelim = match.group(1) + line = self.fsm.current_input + self.comment.append(line[0:line.rfind(activeCommentDelim)]) + if (to_state == "DEFCLASS_BODY"): + self.__closeComment() + self.defclass = [] + # in multiline comment + else: + # just append the comment line + self.comment.append(self.fsm.current_input) + + def appendNormalLine(self, match): + """Appends a line to the output.""" + if options.debug: + print >>sys.stderr, "# CALLBACK: appendNormalLine" + self.output.append(self.fsm.current_input) + + def appendDefclassLine(self, match): + """Appends a line to the triggering block.""" + if options.debug: + print >>sys.stderr, "# CALLBACK: appendDefclassLine" + self.defclass.append(self.fsm.current_input) + + def makeCommentBlock(self): + """Indents the current comment block with respect to the current + indentation level. + + @returns a list of indented comment lines + """ + doxyStart = "##" + commentLines = self.comment + + commentLines = map(lambda x: "%s# %s" % (self.indent, x), commentLines) + l = [self.indent + doxyStart] + l.extend(commentLines) + + return l + + def parse(self, input): + """Parses a python file given as input string and returns the doxygen- + compatible representation. + + @param input the python code to parse + @returns the modified python code + """ + lines = input.split("\n") + + for line in lines: + self.fsm.makeTransition(line) + + if self.fsm.current_state == "DEFCLASS": + self.__closeComment() + + return "\n".join(self.output) + + def parseFile(self, filename): + """Parses a python file given as input string and returns the doxygen- + compatible representation. + + @param input the python code to parse + @returns the modified python code + """ + f = open(filename, 'r') + + for line in f: + self.parseLine(line.rstrip('\r\n')) + if self.fsm.current_state == "DEFCLASS": + self.__closeComment() + self.__flushBuffer() + f.close() + + def parseLine(self, line): + """Parse one line of python and flush the resulting output to the + outstream. + + @param line the python code line to parse + """ + self.fsm.makeTransition(line) + self.__flushBuffer() + +def optParse(): + """Parses commandline options.""" + parser = OptionParser(prog=__applicationName__, version="%prog " + __version__) + + parser.set_usage("%prog [options] filename") + parser.add_option("--autobrief", + action="store_true", dest="autobrief", + help="use the docstring summary line as \\brief description" + ) + parser.add_option("--debug", + action="store_true", dest="debug", + help="enable debug output on stderr" + ) + + ## parse options + global options + (options, filename) = parser.parse_args() + + if not filename: + print >>sys.stderr, "No filename given." + sys.exit(-1) + + return filename[0] + +def main(): + """Starts the parser on the file given by the filename as the first + argument on the commandline. + """ + filename = optParse() + fsm = Doxypy() + fsm.parseFile(filename) + +if __name__ == "__main__": + main() diff --git a/src/third_party/wiredtiger/src/docs/tools/fixlinks.py b/src/third_party/wiredtiger/src/docs/tools/fixlinks.py new file mode 100755 index 00000000000..7163246e3bd --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/tools/fixlinks.py @@ -0,0 +1,67 @@ +#!/usr/bin/env python +# +# Public Domain 2014-2016 MongoDB, Inc. +# Public Domain 2008-2014 WiredTiger, Inc. +# +# This is free and unencumbered software released into the public domain. +# +# Anyone is free to copy, modify, publish, use, compile, sell, or +# distribute this software, either in source code form or as a compiled +# binary, for any purpose, commercial or non-commercial, and by any +# means. +# +# In jurisdictions that recognize copyright laws, the author or authors +# of this software dedicate any and all copyright interest in the +# software to the public domain. We make this dedication for the benefit +# of the public at large and to the detriment of our heirs and +# successors. We intend this dedication to be an overt act of +# relinquishment in perpetuity of all present and future rights to this +# software under copyright law. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +# IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. +# +# This Python script is run as part of generating the documentation for the +# WiredTiger Python API. It runs after doxypy, and turns method names in +# comments into references to the corresponding function in the C API. + +import re, sys + +def process(source): + # Replace standard struct wrapper wording + source = re.sub(r'(\s+#.*)Proxy of C (\w+) struct', + lambda m: ('%sPython wrapper around C ::%s%s@copydoc ::%s' % + (m.group(1), m.group(2).strip('_').upper(), m.group(1), + m.group(2).strip('_').upper())), source) + + # Replace lowercase class names with upper case + source = re.sub(r'(\s+#.*)__(wt_\w+)::', + lambda m: ('%s%s::' % (m.group(1), m.group(2).upper())), source) + + # Replace "char" with "string" in comments + source = re.sub(r'(\s+#.*)\bconst char \*', r'\1string', source) + source = re.sub(r'(\s+#.*)\bchar const \*', r'\1string', source) + source = re.sub(r'(\s+#.*)\bchar\b', r'\1string', source) + + # Copy documentation -- methods, then global functions + source = re.sub(r'(\s+# )(\w+)\(self, (connection|cursor|session).*', + lambda m: ('%s%s%s@copydoc WT_%s::%s' % + (m.group(0), m.group(1), m.group(1), m.group(3).upper(), m.group(2))), + source) + + source = re.sub(r'(\s+# )(wiredtiger_\w+)\(.*', + lambda m: ('%s%s%s@copydoc ::%s' % + (m.group(0), m.group(1), m.group(1), m.group(2))), source) + + # Replace "self, handle" with "self" -- these are typedef'ed away + source = re.sub(r'(\s+#.*self), + (?:connection|cursor|session)', r'\1', source) + return source + +if __name__ == '__main__': + sys.stdout.write(process(sys.stdin.read())) diff --git a/src/third_party/wiredtiger/src/docs/tools/pyfilter b/src/third_party/wiredtiger/src/docs/tools/pyfilter new file mode 100755 index 00000000000..7ef80325bd1 --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/tools/pyfilter @@ -0,0 +1,30 @@ +#!/bin/sh +# +# Public Domain 2008-2012 WiredTiger, Inc. +# +# This is free and unencumbered software released into the public domain. +# +# Anyone is free to copy, modify, publish, use, compile, sell, or +# distribute this software, either in source code form or as a compiled +# binary, for any purpose, commercial or non-commercial, and by any +# means. +# +# In jurisdictions that recognize copyright laws, the author or authors +# of this software dedicate any and all copyright interest in the +# software to the public domain. We make this dedication for the benefit +# of the public at large and to the detriment of our heirs and +# successors. We intend this dedication to be an overt act of +# relinquishment in perpetuity of all present and future rights to this +# software under copyright law. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +# IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. + +tooldir=`dirname $0` + +python $tooldir/doxypy.py "$@" | python $tooldir/fixlinks.py diff --git a/src/third_party/wiredtiger/src/docs/top/Doxyfile b/src/third_party/wiredtiger/src/docs/top/Doxyfile new file mode 100644 index 00000000000..ed4f2eb8c3b --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/top/Doxyfile @@ -0,0 +1,8 @@ +# Override settings in the normal documentation build for the landing page. + +PROJECT_NUMBER = "Developer Site" +OUTPUT_DIRECTORY = ../../docs/top +INPUT = top community.dox license.dox +EXCLUDE = + +GENERATE_TREEVIEW = NO diff --git a/src/third_party/wiredtiger/src/docs/top/main.dox b/src/third_party/wiredtiger/src/docs/top/main.dox new file mode 100644 index 00000000000..7e670541e7d --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/top/main.dox @@ -0,0 +1,26 @@ +/*! @mainpage WiredTiger Developer Site + +WiredTiger is an high performance, scalable, production quality, NoSQL, +@subpage license "Open Source" extensible platform for data management. + +@section releases Releases + +<table> +@row{<b>WiredTiger 2.7.0</b> (current), + <a href="releases/wiredtiger-2.7.0.tar.bz2"><b>[Release package]</b></a>, + <a href="2.7.0/index.html"><b>[Documentation]</b></a>} +@row{<b>WiredTiger 2.6.1</b> (previous), + <a href="releases/wiredtiger-2.6.1.tar.bz2"><b>[Release package]</b></a>, + <a href="2.6.1/index.html"><b>[Documentation]</b></a>} +@row{<b>Development branch</b>, + <a href="https://github.com/wiredtiger/wiredtiger"><b>[Source code]</b></a>, + <a href="develop/index.html"><b>[Documentation]</b></a>} +</table> + +@section contact Contact Us + +To ask questions or discuss issues related to using WiredTiger, + +- Visit our <a href="http://groups.google.com/group/wiredtiger-users"><b>User Group</b></a> +- Send email to <a href="mailto:info@wiredtiger.com"><b>info@wiredtiger.com</b></a> +*/ diff --git a/src/third_party/wiredtiger/src/docs/transactions.dox b/src/third_party/wiredtiger/src/docs/transactions.dox new file mode 100644 index 00000000000..bbbd2d52296 --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/transactions.dox @@ -0,0 +1,151 @@ +/*! @m_page{{c,java},transactions,Transactions} + +@section transactions_acid ACID properties + +Transactions provide a powerful abstraction for multiple threads to +operate on data concurrently because they have the following properties: + +- Atomicity: all or none of a transaction is completed. +- Consistency: if each transaction maintains some property when considered + separately, then the combined effect of executing the transactions + concurrently will maintain the same property. +- Isolation: developers can reason about transactions as if they run + single-threaded. +- Durability: once a transaction commits, its updates cannot be lost. + +WiredTiger supports transactions with the following caveats to the ACID +properties: + +- the maximum level of isolation supported is snapshot isolation. + See @ref transaction_isolation for more details. +- transactional updates are made durable by a combination of checkpoints + and logging. See @ref checkpoint for information on checkpoint durability + and @ref durability for information on commit-level durability. +- each transaction's uncommitted changes must fit in memory: for + efficiency, WiredTiger does not write to the log until a transaction + commits. + +@section transactions_api Transactional API + +In WiredTiger, transaction operations are methods off the WT_SESSION +class. + +Applications call WT_SESSION::begin_transaction to start a new transaction. +Operations subsequently performed using that WT_SESSION handle, including +operations on any cursors open in that WT_SESSION handle (whether opened +before or after the WT_SESSION::begin_transaction call), are part of the +transaction and their effects committed by calling +WT_SESSION::commit_transaction, or discarded by calling +WT_SESSION::rollback_transaction. + +If WT_SESSION::commit_transaction returns an error for any reason, the +transaction was rolled back, not committed. + +When transactions are used, data operations can encounter a conflict and +fail with the ::WT_ROLLBACK error. If this error occurs, transactions +should be rolled back with WT_SESSION::rollback_transaction and the +operation retried. + +The WT_SESSION::rollback_transaction method implicitly resets all +cursors in the session as if the WT_CURSOR::reset method was called, +discarding any cursor position as well as any key and value. + +@snippet ex_all.c transaction commit/rollback + +@section transactions_implicit Implicit transactions + +If a cursor is used when no explicit transaction is active in a session, +reads are performed at the isolation level of the session, set with the +\c isolation key to WT_CONNECTION::open_session, and successful updates +are automatically committed before the update operation returns. + +Any operation consisting of multiple related updates should be enclosed +in an explicit transaction to ensure the updates are applied atomically. + +If an implicit transaction successfully commits, the cursors in the +WT_SESSION remain positioned. If an implicit transaction fails, all +cursors in the WT_SESSION are reset, as if WT_CURSOR::reset were called, +discarding any position or key/value information they may have. + +See @ref cursors_transactions for more information. + +@section transactions_concurrency Concurrency control + +WiredTiger uses optimistic concurrency control algorithms. This avoids +the bottleneck of a centralized lock manager and ensures transactional +operations do not block: reads do not block writes, and vice versa. + +Further, writes do not block writes, although concurrent transactions +updating the same value will fail with ::WT_ROLLBACK. Some applications +may benefit from application-level synchronization to avoid repeated +attempts to rollback and update the same value. + +Operations in transactions may also fail with the ::WT_ROLLBACK error if +some resource cannot be allocated after repeated attempts. For example, if +the cache is not large enough to hold the updates required to satisfy +transactional readers, an operation may fail and return ::WT_ROLLBACK. + +@section transaction_isolation Isolation levels + +WiredTiger supports <code>read-uncommitted</code>, +<code>read-committed</code> and <code>snapshot</code> isolation levels; +the default isolation level is <code>read-committed</code>. + +- <code>read-uncommitted</code>: +Transactions can see changes made by other transactions before those +transactions are committed. Dirty reads, non-repeatable reads and +phantoms are possible. + +- <code>read-committed</code>: +Transactions cannot see changes made by other transactions before those +transactions are committed. Dirty reads are not possible; +non-repeatable reads and phantoms are possible. Committed changes from +concurrent transactions become visible when no cursor is positioned in +the read-committed transaction. + +- <code>snapshot</code>: +Transactions read the versions of records committed before the transaction +started. Dirty reads and non-repeatable reads are not possible; phantoms +are possible.<br><br> +Snapshot isolation is a strong guarantee, but not equivalent to a +single-threaded execution of the transactions, known as serializable +isolation. Concurrent transactions T1 and T2 running under snapshot +isolation may both commit and produce a state that neither (T1 followed +by T2) nor (T2 followed by T1) could have produced, if there is overlap +between T1's reads and T2's writes, and between T1's writes and T2's +reads. + +The transaction isolation level can be configured on a per-transaction +basis: + +@snippet ex_all.c transaction isolation + +Additionally, the default transaction isolation can be configured and +re-configured on a per-session basis: + +@snippet ex_all.c session isolation configuration + +@snippet ex_all.c session isolation re-configuration + +@section transaction_named_snapshots Named Snapshots + +Applications can create named snapshots by calling WT_SESSION::snapshot +with a configuration that includes <code>"name=foo"</code>. +This configuration creates a new named snapshot, as if a snapshot isolation +transaction were started at the time of the WT_SESSION::snapshot call. + +Subsequent transactions can be started "as of" that snapshot by calling +WT_SESSION::begin_transaction with a configuration that includes +<code>snapshot=foo</code>. That transaction will run at snapshot isolation +as if the transaction started at the time of the WT_SESSION::snapshot +call that created the snapshot. + +Named snapshots keep data pinned in cache as if a real transaction were +running for the time that the named transaction is active. The resources +associated with named snapshots should be released by calling +WT_SESSION::snapshot with a configuration that includes +<code>"drop="</code>. See WT_SESSION::snapshot documentation for details of +the semantics supported by the drop configuration. + +Named snapshots are not durable: they do not survive WT_CONNECTION::close. + */ diff --git a/src/third_party/wiredtiger/src/docs/tune-bulk-load.dox b/src/third_party/wiredtiger/src/docs/tune-bulk-load.dox new file mode 100644 index 00000000000..f5d28436dca --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/tune-bulk-load.dox @@ -0,0 +1,39 @@ +/*! @page tune_bulk_load Bulk-load + +When loading a large amount of data into a new object, using a cursor +with the \c bulk configuration string enabled and loading the data in +sorted order will be much faster than doing out-of-order inserts. + +WiredTiger cursors can be configured for bulk-load using the \c bulk +configuration keyword to WT_SESSION::open_cursor. Bulk-load is a "fast +path" for quickly loading a large number of rows. Bulk-load may only +be used on newly created objects, and an object being bulk-loaded is not +accessible from other cursors. + +Cursors configured for bulk-load only support the WT_CURSOR::insert and +WT_CURSOR::close methods. Bulk load inserts are non-transactional: they +cannot be rolled back and ignore the transactional state of the WT_SESSION +in which they are opened. + +When doing a bulk-load insert, keys must be inserted in sorted order. +When doing a bulk-load insert into a column-store object, any skipped +records will be created as already-deleted rows. If a column-store +bulk-load cursor is configured with \c append, the cursor key will be +ignored and each inserted row will be assigned the next sequential +record number. + +When using the \c sort utility on a Linux or other POSIX-like system to +pre-sort keys, the locale specified by the environment affects the sort +order and may not match the default sort order used by WiredTiger. Set +\c LC_ALL=C in the process' environment to configure the traditional sort +order that uses native byte values. + +When bulk-loading fixed-length column store objects, the \c bulk +configuration string value \c bitmap allows chunks of a memory resident +bitmap to be loaded directly into an object by passing a WT_ITEM to +WT_CURSOR::set_value, where the size field indicates the number of +records in the bitmap (as specified by the object's \c value_format +configuration). Bulk-loaded bitmap values must end on a byte boundary +relative to the bit count (except for the last set of values loaded). + + */ diff --git a/src/third_party/wiredtiger/src/docs/tune-cache.dox b/src/third_party/wiredtiger/src/docs/tune-cache.dox new file mode 100644 index 00000000000..c9603085905 --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/tune-cache.dox @@ -0,0 +1,97 @@ +/*! @page tune_cache Cache and eviction tuning + +@section tuning_cache_size Cache size + +The size of the cache is the single most important tuning knob for a +WiredTiger application. Ideally the cache should be configured to be +large enough to hold an application's working set. + +The cache size for the database is normally configured by setting the +\c cache_size configuration string when calling the ::wiredtiger_open +function. The cache size can be adjusted after the open call with +WT_CONNECTION::reconfigure. + +An example of setting a cache size to 500MB: + +@snippet ex_config.c configure cache size + +The effectiveness of the chosen cache size can be measured by reviewing +the page eviction statistics for the database. + +@section tuning_cache_resident Cache resident objects + +Objects can be created as cache resident - that is their contents will +remain in cache and never be considered for the purposes of cache +eviction. Cache residence can be configured with the WT_SESSION::create +"cache_resident" configuration string. LSM tree objects do not support +the "cache_resident" setting. + +Configuring a cache resident object has several effects: + +- Once the object's pages have been created or instantiated in memory + no further I/O cost is ever paid for object access, minimizing + potential latency. +- Cache resident objects can be accessed faster than objects tracked for + potential eviction. +- If cache resident objects require a significant proportion of the + configured cache size then non cache-resident objects can incur + significantly higher I/O churn. +- If cache resident objects require more space than the configured cache + size, then further operations will either return error or stall until + space is made available by closing objects. + +An example of configuring a cache-resident object: + +@snippet ex_all.c Create a cache-resident object + +@section cache_eviction Eviction tuning + +When an application approaches the maximum cache size, WiredTiger begins +eviction to stop memory use from growing too large, approximating a +least-recently-used algorithm. + +WiredTiger provides several configuration options for tuning how pages +are evicted from the cache. Different settings will improve performance +depending on an application's particular workload. Customizing the +eviction configuration settings can reduce latency spikes in +application threads and can improve throughput in some applications. + +WiredTiger eviction tuning options can be configured when first opening +a database via ::wiredtiger_open, or changed after open with +WT_CONNECTION::reconfigure. + +The \c eviction_trigger configuration value is the occupied percentage +of the total cache size that causes eviction to start. By default, +WiredTiger begins evicting pages when the cache is 95% full. An +application concerned about a latency spike as the cache becomes full +might want to begin eviction earlier. + +The \c eviction_target configuration value is the overall target for +eviction, expressed as a percentage of total cache size; that is, once +eviction begins, it will proceed until the target percentage of bytes +in the cache is reached. Note the \c eviction_target configuration +value is ignored until eviction is triggered. + +The \c eviction_dirty_target configuration value is the overall dirty +byte target for eviction, expressed as a percentage of total cache size; +that is, once eviction begins, it will proceed until the target +percentage of dirty bytes in the cache is reached. Note the +\c eviction_dirty_target configuration value is ignored until eviction +is triggered. + +@snippet ex_all.c Eviction configuration + +By default, WiredTiger cache eviction is handled by a single, separate +thread. In a large, busy cache, a single thread will be insufficient +(especially when the eviction thread must wait for I/O). The +\c eviction=(threads_min) and \c eviction=(threads_max) configuration +values can be used to configure the minimum and maximum number of +additional threads WiredTiger will create to keep up with the +application eviction load. Finally, if the Wiredtiger eviction threads +are unable to keep up with application demand for cache space, +application threads will be tasked with eviction as well, potentially +resulting in latency spikes. + +@snippet ex_all.c Eviction worker configuration + + */ diff --git a/src/third_party/wiredtiger/src/docs/tune-checksum.dox b/src/third_party/wiredtiger/src/docs/tune-checksum.dox new file mode 100644 index 00000000000..bd7b2fb6ecc --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/tune-checksum.dox @@ -0,0 +1,22 @@ +/*! @page tune_checksum Checksums + +WiredTiger optionally checksums file reads and writes to detect storage +failures. In read-only applications, or when file compression provides +any necessary checksum functionality, or when using backing storage +systems where blocks require no validation, performance can be increased +by turning off checksum support when calling the WT_SESSION::create +method. + +Checksums can be configured to be "off" or "on", as well as "uncompressed". +The "uncompressed" configuration checksums blocks not otherwise protected +by compression. For example, in a system where the compression algorithms +provide sufficient protection against corruption, and when writing a block +which is too small to be usefully compressed, setting the checksum +configuration value to "uncompressed" causes WiredTiger to checksum the +blocks which are not compressed: + +@snippet ex_all.c Configure checksums to uncompressed + +The default WiredTiger configuration is \c uncompressed. + + */ diff --git a/src/third_party/wiredtiger/src/docs/tune-close.dox b/src/third_party/wiredtiger/src/docs/tune-close.dox new file mode 100644 index 00000000000..dd615e3e460 --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/tune-close.dox @@ -0,0 +1,13 @@ +/*! @page tune_close Connection close + +Closing a WT_CONNECTION:: handle can be expensive, in part because of +the cost of freeing memory (for example, the memory allocated for a +large cache). If the process is exiting regardless, configuring +WT_CONNECTION::close to leak memory on close can significantly speed up +the close. + +An example of configuring WT_CONNECTION::close to leak memory: + +@snippet ex_all.c Connection close leaking memory + + */ diff --git a/src/third_party/wiredtiger/src/docs/tune-compression.dox b/src/third_party/wiredtiger/src/docs/tune-compression.dox new file mode 100644 index 00000000000..bb675337a0d --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/tune-compression.dox @@ -0,0 +1,25 @@ +/*! @page tune_compression Compression + +WiredTiger includes a number of optional compression techniques. Configuring +compression generally decreases on-disk and in-memory resource requirements +and the amount of I/O, and increases CPU cost when rows are read and written. + +Configuring compression may change application throughput. For example, +in applications using solid-state drives (where I/O is less expensive), +turning off compression may increase application performance by reducing +CPU costs; in applications where I/O costs are more expensive, turning on +compression may increase application performance by reducing the overall +number of I/O operations. + +An example of turning on row-store key prefix compression: + +@snippet ex_all.c Configure key prefix compression on + +An example of turning on row-store or column-store dictionary compression: + +@snippet ex_all.c Configure dictionary compression on + +See @ref file_formats_compression for more information on available +compression techniques. + + */ diff --git a/src/third_party/wiredtiger/src/docs/tune-cursor-persist.dox b/src/third_party/wiredtiger/src/docs/tune-cursor-persist.dox new file mode 100644 index 00000000000..183a877434e --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/tune-cursor-persist.dox @@ -0,0 +1,18 @@ +/*! @page tune_cursor_persist Cursor persistence + +Opening a new cursor is a relatively expensive operation in WiredTiger +(especially in table objects and Log-Structured Merge Trees (LSM) trees, +where a logical cursor may require multiple, underlying object cursors), +and caching cursors can improve performance. On the other hand, cursors +hold positions in objects, and therefore long-lived cursor positions can +decrease performance by blocking page eviction or looking like a +long-lived transaction. + +Best practices are to cache cursors, but use the WT_CURSOR::reset method +to discard the cursor's position in the object when the position is no +longer needed. Additionally, use the WT_CURSOR::insert method instead +of the WT_CURSOR::update method when there's no need to hold a position +in the object, because the WT_CURSOR::insert method never holds a cursor +position and there's no need to call WT_CURSOR::reset. + + */ diff --git a/src/third_party/wiredtiger/src/docs/tune-durability.dox b/src/third_party/wiredtiger/src/docs/tune-durability.dox new file mode 100644 index 00000000000..0b4644f3ec5 --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/tune-durability.dox @@ -0,0 +1,119 @@ +/*! @class doc_tune_durability_group_commit + +WiredTiger automatically groups the flush operations for threads that +commit concurrently into single calls. This usually means +multi-threaded workloads will achieve higher throughput than +single-threaded workloads because the operating system can flush data +more efficiently to the disk. No application-level configuration is +required for this feature. + + */ + +/*! @class doc_tune_durability_flush_config + +By default, log records are written to an in-memory buffer before +WT_SESSION::commit_transaction returns, giving highest performance but +not ensuring durability. The durability guarantees can be stricter but +will impact performance. + +If \c transaction_sync=(enabled=false) is configured to ::wiredtiger_open, +log records may be buffered in memory, and only flushed to disk by +checkpoints, when log files switch or calls to WT_SESSION::commit_transaction +with \c sync=on. (Note that any call to WT_SESSION::commit_transaction +with \c sync=on will flush the log records for all committed transactions, +not just the transaction where the configuration is set.) This provides the +minimal guarantees, but will be significantly faster than other configurations. + +If \c transaction_sync=(enabled=true), \c transaction_sync=(method) +further configures the method used to flush log records to disk. By +default, the configured value is \c fsync, which calls the operating +system's \c fsync call (of \c fdatasync if available) as each commit completes. + +If the value is set to \c dsync, the \c O_DSYNC or \c O_SYNC +flag to the operating system's \c open call will be specified when the +file is opened. (The durability guarantee of the \c fsync and \c dsync +configurations are the same, and in our experience the \c open flags are +slower, this configuration is only included for systems where that may +not be the case.) + +If the value is set to \c none, the operating system's \c write call +will be called as each commit completes but does not flush to disk. +This setting gives durability at the application level but not at +the system level. + +When a log file fills and the system moves to the next log file, the +previous log file will always be flushed to disk prior to close. So +when running in a durability mode that does not flush to disk, the risk +is bounded by the most recent log file change. + +Here is the expected performance of durability modes, in order from the +fastest to the slowest (and from the fewest durability guarantees to the +most durability guarantees). + +<table> +@hrow{Durability Mode, Notes} +@row{<code>log=(enabled=false)</code>, checkpoint-level durability} +@row{<code>log=(enabled)\,transaction_sync=(enabled=false)</code>, + in-memory buffered logging configured; updates durable after + checkpoint or after \c sync is set in WT_SESSION::begin_transaction} +@row{<code>log=(enabled)\,transaction_sync=(enabled=true\,method=none)</code>, + logging configured; updates durable after application failure\, + but not after system failure} +@row{<code>log=(enabled)\,transaction_sync=(enabled=true\,method=fsync)</code>, + logging configured; updates durable on application or system + failure} +@row{<code>log=(enabled)\,transaction_sync=(enabled=true\,method=dsync)</code>, + logging configured; updates durable on application or system + failure} +</table> + +The durability setting can also be controlled directly on a per-transaction +basis via the WT_SESSION::commit_transaction method. +The WT_SESSION::commit_transaction supports several durability modes with +the \c sync configuration that override the connection level settings. + +If \c sync=on is configured then this commit operation will wait for its +log records, and all earlier ones, to be durable to the extent specified +by the \c transaction_sync=(method) setting before returning. + +If \c sync=off is configured then this commit operation will write its +records into the in-memory buffer and return immediately. + +If \c sync=background is configured then this commit operation will write +its record to an in-memory buffer, and will return. Prior to returning it +will signal an internal WiredTiger worker thread to synchronize this log +record. The caller may then check the status of that background +synchronization with the WT_SESSION::transaction_sync method. + +The durability of the write-ahead log can be controlled independently +as well via the WT_SESSION::log_flush method. +The WT_SESSION::log_flush supports several durability modes with +the \c sync configuration that immediately act upon the log. + +If \c sync=on is configured then this flush will force the current +log and all earlier records to be durable on disk before returning. +This method call overrides the \c transaction_sync setting and +forces the data out via \c fsync. + +If \c sync=off is configured then this flush operation will force the +logging subsystem to write any outstanding in-memory buffers to the +file system before returning. + +If \c sync=background is configured then this flush operation will force +the signalling of a background synchronization operation. +The caller may then check the status of that background +synchronization with the WT_SESSION::transaction_sync method. + */ + +/*! @page tune_durability Commit-level durability + +There are some considerations when configuring commit-level durability +that can affect performance. + +@section tune_durability_group_commit Group commit +@copydoc doc_tune_durability_group_commit + +@section tune_durability_flush_config Flush call configuration +@copydoc doc_tune_durability_flush_config + + */ diff --git a/src/third_party/wiredtiger/src/docs/tune-file-alloc.dox b/src/third_party/wiredtiger/src/docs/tune-file-alloc.dox new file mode 100644 index 00000000000..062a0170ac8 --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/tune-file-alloc.dox @@ -0,0 +1,30 @@ +/*! @page tune_file_alloc File allocation + +@section tuning_system_file_block_grow File growth + +It's faster on some filesystems to grow a file in chunks rather than to +extend it a block at a time as new blocks are written. By configuring +the wiredtiger_open functions \c file_extend value, applications can +grow files ahead of the blocks being written. + +@snippet ex_all.c Configure file_extend + +The system calls used to extend files in chunks on some systems require +locking across the system call, which may decrease performance. Before +configuring a \c file_extend value, before and after throughput should +be carefully evaluated. + +@section tuning_system_file_block_allocation File block allocation + +By default, when file blocks are being reused, WiredTiger attempts to +avoid file fragmentation by selecting the smallest available block +rather than splitting a larger available block into two. The \c +block_allocation configuration string to WT_SESSION::create can be set +to \c first to change the algorithm to first-fit, that is, take the +first available block in the file. Applications where file size is more +of an issue than file fragmentation (for example, applications with +fixed-size blocks) might want to configure this way. + +@snippet ex_all.c Configure block_allocation + + */ diff --git a/src/third_party/wiredtiger/src/docs/tune-memory-allocator.dox b/src/third_party/wiredtiger/src/docs/tune-memory-allocator.dox new file mode 100644 index 00000000000..a619708f816 --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/tune-memory-allocator.dox @@ -0,0 +1,18 @@ +/*! @page tune_memory_allocator Memory allocator + +The performance of WiredTiger applications, especially heavily-threaded +applications can be dominated by memory allocation because the WiredTiger +engine frees and re-allocates memory as part of many queries. Replacing +the system's malloc implementation with one that has better threaded +performance (for example, +<a href="http://goog-perftools.sourceforge.net/doc/tcmalloc.html"> +Google's tcmalloc</a>, or +<a href="http://www.canonware.com/jemalloc">FreeBSD's jemalloc</a>), +can dramatically improve throughput. + +As different memory allocators have different overhead and different +workloads will have different heap allocation sizes and patterns, +applications may need to set their allocator overhead using the +\c cache_overhead configuration to the wiredtiger_open:: call. + + */ diff --git a/src/third_party/wiredtiger/src/docs/tune-mutex.dox b/src/third_party/wiredtiger/src/docs/tune-mutex.dox new file mode 100644 index 00000000000..713d9f0d8d0 --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/tune-mutex.dox @@ -0,0 +1,17 @@ +/*! @page tune_mutex Mutexes + +The performance of WiredTiger applications can be sensitive to type of +mutex used for serialization. WiredTiger can be configured, using the +\c --with-spinlock argument during configuration to use one of three +different mutex types: \c gcc, \c pthread and \c pthread_adaptive mutexes. + +The \c gcc option configures GCC compiler (or compatible compilers such +as clang), mutexes; the \c pthread option configures POSIX 1003.1c +pthread mutexes; the \c pthread_adaptive option configures POSIX 1003.1c +pthread mutexes with adaptive behavior (where that behavior is +supported, for example, most Linux platforms). + +By default, WiredTiger is configured to use \c pthread, that is, +non-adaptive POSIX 1003.1c pthread mutexes. + + */ diff --git a/src/third_party/wiredtiger/src/docs/tune-page-sizes.dox b/src/third_party/wiredtiger/src/docs/tune-page-sizes.dox new file mode 100644 index 00000000000..130e047a02d --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/tune-page-sizes.dox @@ -0,0 +1,142 @@ +/*! @page tune_page_sizes Page and overflow key/value sizes + +There are seven page and key/value size configuration strings: + +- allocation size (\c allocation_size), +- page sizes (\c internal_page_max and \c leaf_page_max), +- key and value sizes (\c internal_key_max, \c leaf_key_max and \c leaf_value_max), and the +- page-split percentage (\c split_pct). + +All seven are specified to the WT_SESSION::create method, in other +words, they are configurable on a per-file basis. + +Applications commonly configure page sizes, based on their workload's +typical key and value size. Once the correct page size has been chosen, +appropriate defaults for the other configuration values are derived from +the page sizes, and relatively few applications will need to modify the +other page and key/value size configuration options. + +An example of configuring page and key/value sizes: + +@snippet ex_all.c Create a table and configure the page size + +@section tune_page_sizes_sizes Page, key and value sizes + +The \c internal_page_max and \c leaf_page_max configuration values +specify a maximum size for Btree internal and leaf pages. That is, when +an internal or leaf page grows past that size, it splits into multiple +pages. Generally, internal pages should be sized to fit into on-chip +caches in order to minimize cache misses when searching the tree, while +leaf pages should be sized to maximize I/O performance (if reading from +disk is necessary, it is usually desirable to read a large amount of +data, assuming some locality of reference in the application's access +pattern). + +The default page size configurations (2KB for \c internal_page_max, 32KB +for \c leaf_page_max), are appropriate for applications with relatively +small keys and values. + +- Applications doing full-table scans through out-of-memory workloads +might increase both internal and leaf page sizes to transfer more data +per I/O. +- Applications focused on read/write amplification might decrease the page +size to better match the underlying storage block size. + +When block compression has been configured, configured page sizes will +not match the actual size of the page on disk. Block compression in +WiredTiger happens within the I/O subsystem, and so a page might split +even if subsequent compression would result in a resulting page size +small enough to leave as a single page. In other words, page sizes are +based on in-memory sizes, not on-disk sizes. Applications needing to +write specific sized blocks may want to consider implementing a +WT_COMPRESSOR::compress_raw function. + +The page sizes also determine the default size of overflow items, that +is, keys and values too large to easily store on a page. Overflow items +are stored separately in the file from the page where the item logically +appears, and so reading or writing an overflow item is more expensive +than an on-page item, normally requiring additional I/O. Additionally, +overflow values are not cached in memory. This means overflow items +won't affect the caching behavior of the application, but it also means +that each time an overflow value is read, it is re-read from disk. + +For both of these reasons, applications should avoid creating large +numbers of commonly referenced overflow items. This is especially +important for keys, as keys on internal pages are referenced during +random searches, not just during data retrieval. Generally, +applications should make every attempt to avoid creating overflow keys. + +- Applications with large keys and values, and concerned with latency, +might increase the page size to avoid creating overflow items, in order +to avoid the additional cost of retrieving them. + +- Applications with large keys and values, doing random searches, might +decrease the page size to avoid wasting cache space on overflow items +that aren't likely to be needed. + +- Applications with large keys and values, doing table scans, might +increase the page size to avoid creating overflow items, as the overflow +items must be read into memory in all cases, anyway. + +The \c internal_key_max, \c leaf_key_max and \c leaf_value_max +configuration values allow applications to change the size at which a +key or value will be treated as an overflow item. + +The value of \c internal_key_max is relative to the maximum internal +page size. Because the number of keys on an internal page determines +the depth of the tree, the \c internal_key_max value can only be +adjusted within a certain range, and the configured value will be +automatically adjusted by WiredTiger, if necessary to ensure a +reasonable number of keys fit on an internal page. + +The values of \c leaf_key_max and \c leaf_value_max are not relative to +the maximum leaf page size. If either is larger than the maximum page +size, the page size will be ignored when the larger keys and values are +being written, and a larger page will be created as necessary. + +Most applications should not need to tune the maximum key and value +sizes. Applications requiring a small page size, but also having +latency concerns such that the additional work to retrieve an overflow +item is an issue, may find them useful. + +An example of configuring a large leaf overflow value: + +@snippet ex_all.c Create a table and configure a large leaf value max + +@section tune_page_sizes_split_percentage Split percentage + +The \c split_pct configuration string configures the size of a split +page. When a page grows sufficiently large that it must be written as +multiple disk blocks, the newly written block size is \c split_pct +percent of the maximum page size. This value should be selected to +avoid creating a large number of tiny pages or repeatedly splitting +whenever new entries are inserted. For example, if the maximum page +size is 1MB, a \c split_pct value of 10% would potentially result in +creating a large number of 100KB pages, which may not be optimal for +future I/O. Or, if the maximum page size is 1MB, a \c split_pct value +of 90% would potentially result in repeatedly splitting pages as the +split pages grow to 1MB over and over. The default value for \c +split_pct is 75%, intended to keep large pages relatively large, while +still giving split pages room to grow. + +Most applications should not need to tune the split percentage size. + +@section tune_page_sizes_allocation_size Allocation size + +The \c allocation_size configuration value is the underlying unit of +allocation for the file. As the unit of file allocation, it sets the +minimum page size and how much space is wasted when storing small +amounts of data and overflow items. For example, if the allocation size +is set to 4KB, an overflow item of 18,000 bytes requires 5 allocation +units and wastes about 2KB of space. If the allocation size is 16KB, +the same overflow item would waste more than 10KB. + +The default allocation size is 4KB, chosen for compatibility with +virtual memory page sizes and direct I/O requirements on common server +platforms. + +Most applications should not need to tune the allocation size; it is +primarily intended for applications coping with the specific +requirements some file systems make to support features like direct I/O. + +*/ diff --git a/src/third_party/wiredtiger/src/docs/tune-read-only.dox b/src/third_party/wiredtiger/src/docs/tune-read-only.dox new file mode 100644 index 00000000000..98db7e0782e --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/tune-read-only.dox @@ -0,0 +1,26 @@ +/*! @page tune_read_only Read-only objects + +Cursors opened on checkpoints (either named, or using the special "last +checkpoint" name "WiredTigerCheckpoint") are read-only objects. Unless +memory mapping is configured off (using the "mmap" configuration string +to ::wiredtiger_open), read-only objects are mapped into process memory +instead of being read through the WiredTiger cache. Using read-only +objects where possible minimizes the amount of buffer cache memory +required by WiredTiger applications and the work required for buffer +cache management, as well as reducing the number of memory copies from +the operating system buffer cache into application memory. + +To open a named checkpoint, use the configuration string "checkpoint" +to the WT_SESSION::open_cursor method: +@snippet ex_all.c open a named checkpoint + +To open the last checkpoint taken in the object, use the configuration +string "checkpoint" with the name "WiredTigerCheckpoint" to the +WT_SESSION::open_cursor method: +@snippet ex_all.c open the default checkpoint + +Reading from checkpoints is only supported on row- and column-store +objects, LSM cursors cannot be opened with a non-empty \c "checkpoint" +configuration. + + */ diff --git a/src/third_party/wiredtiger/src/docs/tune-statistics.dox b/src/third_party/wiredtiger/src/docs/tune-statistics.dox new file mode 100644 index 00000000000..1fa67d8b7cd --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/tune-statistics.dox @@ -0,0 +1,39 @@ +/*! @page tune_statistics Performance monitoring with statistics + +WiredTiger optionally maintains a variety of statistics, when the +\c statistics configuration string is specified to ::wiredtiger_open; +see @ref statistics for general information about statistics, and +@ref data_statistics for information about accessing the statistics. + +The resulting statistics can be displayed using the \c wtstats visualization +tool. For more information, see @ref wtstats. + +<b> +Note that maintaining run-time statistics involves updating +shared-memory data structures and may decrease application performance. +</b> + +The statistics gathered by WiredTiger can be combined to derive information +about the system's behavior. For example, a cursor can be opened on the +statistics for a table: + +@snippet ex_stat.c statistics calculate open table stats + +Then this code calculates the "fragmentation" of a table, defined +here as the percentage of the table that is not part of the current +checkpoint: + +@snippet ex_stat.c statistics calculate table fragmentation + +The following example calculates the "write amplification", defined here as +the ratio of bytes written to the filesystem versus the total bytes +inserted, updated and removed by the application. + +@snippet ex_stat.c statistics calculate write amplification + +Both examples use this helper function to retrieve statistics values from a +cursor: + +@snippet ex_stat.c statistics calculation helper function + + */ diff --git a/src/third_party/wiredtiger/src/docs/tune-system-buffer-cache.dox b/src/third_party/wiredtiger/src/docs/tune-system-buffer-cache.dox new file mode 100644 index 00000000000..1919c66997a --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/tune-system-buffer-cache.dox @@ -0,0 +1,115 @@ +/*! @page tune_system_buffer_cache System buffer cache + +@section tuning_system_buffer_cache_direct_io Direct I/O + +WiredTiger optionally supports direct I/O. Configuring direct I/O may +be useful for applications wanting to: +- minimize the operating system cache effects of I/O to and from +WiredTiger's buffer cache, +- avoid double-buffering of blocks in WiredTiger's cache and the +operating system buffer cache, and +- avoid stalling underlying solid-state drives by writing a large number +of dirty blocks. + +Direct I/O is configured using the "direct_io" configuration string to +the ::wiredtiger_open function. An example of configuring direct I/O +for WiredTiger's data files: + +@snippet ex_all.c Configure direct_io for data files + +On Windows, the "direct_io" configuration string controls whether the operating +system cache is used to buffer reads, and writes to disk, i.e., +FILE_FLAG_NO_BUFFERING. When "direct_io" is off, Windows will use free RAM to +cache access to files. This may had adverse effects because Windows may page out +the WiredTiger buffer cache instead of its file cache. An additional +configuration string "write_through" controls whether the disk is allowed to +cache the writes. Enabling this flag increases write latency as the drive must +ensure all writes are persisted to disk, but it ensures write durability. To get +the equivalent of \c O_DIRECT on Windows, "direct_io", and "write_through" must +be both set. + +Direct I/O implies a writing thread waits for the write to complete +(which is a slower operation than writing into the system buffer cache), +and configuring direct I/O is likely to decrease overall application +performance. + +Many Linux systems do not support mixing \c O_DIRECT and memory +mapping or normal I/O to the same file, and attempting to do so can +result in data loss or corruption. For this reason: + +- WiredTiger silently ignores the setting of the \c mmap configuration +to the wiredtiger_open function in those cases, and will never memory +map a file which is configured for direct I/O. + +- If \c O_DIRECT is configured for data files on Linux systems, any +system utilities used to copy data files for the purposes of backup +should also specify \c O_DIRECT when configuring their file access. A +standard Linux system utility that supports \c O_DIRECT is the \c dd +utility, when using the \c iflag=direct command-line option. + +Additionally, Windows, and many Linux systems require specific alignment for +buffers used for I/O when direct I/O is configured, and using the wrong +alignment can cause data loss or corruption. When direct I/O is +configured on Windows, and Linux systems, WiredTiger aligns I/O buffers to 4KB; +if different alignment is required by your system, the \c buffer_alignment +configuration to the wiredtiger_open call should be configured to the +correct value. + +Finally, if direct I/O is configured on any system, WiredTiger will +silently change the file unit allocation size and the maximum leaf and +internal page sizes to be at least as large as the \c buffer_alignment +value as well as a multiple of that value. + +Direct I/O is based on the non-standard \c O_DIRECT flag to the POSIX +1003.1 open system call and may not be available on all platforms. + +@section tuning_system_buffer_cache_os_cache_dirty_max os_cache_dirty_max + +As well as direct I/O, WiredTiger supports two additional configuration +options related to the system buffer cache: + +The first is \c os_cache_dirty_max, the maximum dirty bytes an object +is allowed to have in the system buffer cache. Once this many bytes +from an object are written into the system buffer cache, WiredTiger will +attempt to schedule writes for all of the dirty blocks the object has +in the system buffer cache. This configuration option allows +applications to flush dirty blocks from the object, avoiding stalling +any underlying drives when the object is subsequently flushed to disk +as part of a durability operation. + +An example of configuring \c os_cache_dirty_max: + +@snippet ex_all.c os_cache_dirty_max configuration + +The \c os_cache_dirty_max configuration may not be used in combination +with direct I/O. + +The \c os_cache_dirty_max configuration is based on the non-standard +Linux \c sync_file_range system call and will be ignored if set +and that call is not available. + +@section tuning_system_buffer_cache_os_cache_max os_cache_max + +The second configuration option related to the system buffer cache is +\c os_cache_max, the maximum bytes an object is allowed to have in the +system buffer cache. Once this many bytes from an object are either +read into or written from the system buffer cache, WiredTiger will +attempt to evict all of the object's blocks from the buffer cache. This +configuration option allows applications to evict blocks from the system +buffer cache to limit double-buffering and system buffer cache overhead. + +An example of configuring \c os_cache_max: + +@snippet ex_all.c os_cache_max configuration + +The \c os_cache_max configuration may not be used in combination with +direct I/O. + +The \c os_cache_max configuration is based on the POSIX 1003.1 standard +\c posix_fadvise system call and may not available on all platforms. + +Configuring direct I/O, \c os_cache_dirty_max or \c os_cache_max all +have the side effect of turning off memory-mapping of objects in +WiredTiger. + + */ diff --git a/src/third_party/wiredtiger/src/docs/tune-transparent-huge-pages.dox b/src/third_party/wiredtiger/src/docs/tune-transparent-huge-pages.dox new file mode 100644 index 00000000000..1484bd84c18 --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/tune-transparent-huge-pages.dox @@ -0,0 +1,16 @@ +/*! @page tune_transparent_huge_pages Linux transparent huge pages + +The performance of WiredTiger applications can be adversely affected +when transparent huge pages are enabled on Linux systems. Some Linux +systems enable transparent huge pages by default. See the RedHat +Enterprise Linux system documentation's +<a href="https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Performance_Tuning_Guide/s-memory-transhuge.html"> +Huge Pages and Transparent Huge Pages</a> for more information. + +To disable transparent huge pages run the following command as the root user: + +@code +echo never > /sys/kernel/mm/transparent_hugepage/enabled +@endcode + + */ diff --git a/src/third_party/wiredtiger/src/docs/tune-zone-reclaim.dox b/src/third_party/wiredtiger/src/docs/tune-zone-reclaim.dox new file mode 100644 index 00000000000..d6b8d635f47 --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/tune-zone-reclaim.dox @@ -0,0 +1,18 @@ +/*! @page tune_zone_reclaim Linux zone reclamation memory management + +The performance of WiredTiger applications can be sensitive to the zone +reclamation algorithms used in Linux Non Uniform Memory Management (NUMA). +Details of NUMA with specific reference to Linux can be found in the +ACM Queue article +<a href="http://queue.acm.org/detail.cfm?id=2513149"> +NUMA (Non-Uniform Memory Access): An Overview</a>, by Christoph Lameter. +If a WiredTiger database cache is assigned a significant proportion of +system memory, it will likely be beneficial to disable zone reclamation. +Zone reclamation can be disabled by running the following command as the +root user: + +@code +echo 0 > /proc/sys/vm/zone_reclaim_mode +@endcode + + */ diff --git a/src/third_party/wiredtiger/src/docs/upgrade.dox b/src/third_party/wiredtiger/src/docs/upgrade.dox new file mode 100644 index 00000000000..7952efa0343 --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/upgrade.dox @@ -0,0 +1,83 @@ +/*! @page upgrade Upgrading and downgrading databases + +The upgrade/downgrade process is only necessary for major or minor +WiredTiger releases (releases where the major or minor version number +changed). Patch releases (where only the patch number changed), do not +contain any API or file format changes, and do not require anything +other than a link step or shared library installation for upgrading or +downgrading. However, the complete WiredTiger library must be re-built +from scratch when changing versions, as there is no guarantee modules +from one version of the library will work with modules from any other +version. + +The rest of this page discusses upgrading and downgrading WiredTiger +major and minor releases. Specifically, by upgrading, we mean migrating +an application and its databases to a newer WiredTiger release, and by +downgrading, we mean reverting to an older version. + +When upgrading or downgrading between WiredTiger releases, there are two +components to consider: the WiredTiger library API and the underlying +data files used for the database (for example, log files or B+tree +files). + +As part of each WiredTiger release, a new @ref upgrading documentation +section is created, describing changes in the WiredTiger API and in the +format of underlying files, if any. Changes to the format of the +underlying data files are extremely rare, while changes to WiredTiger +API are more common. Generally, when the WiredTiger API changes, the +changes are backward compatible and applications do not require +modifications in order to upgrade to the new release. + +Note it is not necessary to upgrade (downgrade) to intermediate +releases, you can ignore intermediate releases and upgrade (downgrade) +directly to the release you plan to use. + +To upgrade to a new release of WiredTiger: + +-# Review the information for each release after the WiredTiger release +you're currently using, up to and including the release you plan to use. +Changes to the WiredTiger API are normally backward compatible, that is, +you can continue to use the same WiredTiger configuration values used +previously. However, it will likely enhance program maintainability to +switch to newer versions of the configuration strings, as documentation +for the deprecated values will no longer be available. +<br><br> +-# Once you have reviewed the relevant release notes and modified your +application as necessary, install the new version of the application by +relinking or installing a new version of the WiredTiger shared library. +<br><br> +-# Shut down the old version of the application. +<br><br> +-# Start the new version of the application. +<br><br> +If the upgrading documentation for any release after the previous +release and up to and including the release you are using, specifies +underlying data file formats are no longer backward compatible and will +not be upgraded at run-time, a data upgrade step is required before the +data can be accessed. In this case, data files must be upgraded using +the WT_SESSION::upgrade method. + +Applications wanting the ability to downgrade to previous releases of +WiredTiger are constrained in some important ways: + +- The underlying data file formats must not have changed between the +release currently being run and the release to which you are +downgrading. In other words, data file format changes are not backward +compatible, and an upgraded data file cannot be downgraded without being +dumped and re-loaded. +<br><br> +- Applications concerned with downgrading should configure the +::wiredtiger_open \c config_base value to \c false, so WiredTiger does +not write the base configuration file file when creating databases. +Because the base configuration file is written using a current version +of the WiredTiger API, writing this file could cause applications using +previous versions of the WiredTiger API to fail. See @ref config_base +for more information. + +To downgrade to a previous release of WiredTiger: + +-# Shut down the new version of the application. +<br><br> +-# Start the old version of the application. + +*/ diff --git a/src/third_party/wiredtiger/src/docs/upgrading.dox b/src/third_party/wiredtiger/src/docs/upgrading.dox new file mode 100644 index 00000000000..e0239919f0b --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/upgrading.dox @@ -0,0 +1,952 @@ +/*! @page upgrading Upgrading WiredTiger applications + +@section version_271 Upgrading to Version 2.7.1 +<dl> +<dt>Column-store bulk-load cursors</dt> +<dd> +Historically, bulk-load of a column-store object ignored any key set in +the cursor and automatically assigned each inserted row the next +sequential record number for its key. In the 2.7.1 release, column-store +objects match row-store behavior and require the cursor key be set +before an insert. (This also allows allows sparse tables to be created +in column-store objects, any skipped records are created as +already-deleted rows.) To match the previous behavior, specify the +\c append configuration string when opening the column-store bulk-load +cursor; this causes the cursor's key to be ignored and each inserted row +will be assigned the next record number. +</dd> + +<dt>Change to WT_SESSION::truncate with URI</dt> +<dd> +If using the WT_SESSION::truncate API with a file: URI for a full table +truncate, underlying algorithmic changes result in some visible differences. +This call can now return WT_ROLLBACK. Applications should be prepared to +handle this error. This method no longer requires exclusive access to the +table. Also the underlying disk space may not be immediately +reclaimed when the call returns. The performance of this API may differ +from earlier releases. +</dd> + +</dl><hr> +@section version_270 Upgrading to Version 2.7.0 + +<dl> +<dt>Change to config_base=false</dt> +<dd> +If \c config_base=false in the config passed directly to ::wiredtiger_open, +any existing base configuration file will now be ignored. If an application +was relying on this behavior, a connection will be opened with different +settings after upgrading, which could lead to errors or unexpected behavior. +</dd> + +<dt>Statistic change</dt> +<dd> +The statistic "pages split during eviction" was replaced. It has been +replaced by a pair of statistics "internal pages split during eviction" and +"leaf pages split during eviction". +</dd> + +<dt>Change to WT_CURSOR::insert</dt> +<dd> +The WT_CURSOR::insert method in this release has slightly different semantics +with respect to referencing application memory. In previous releases, +WT_CURSOR::insert continued to reference application-memory specified to +either WT_CURSOR::set_key or WT_CURSOR::set_value after a successful return, +which could potentially lead to a core dump if the application freed that +memory before a subsequent call to a WT_CURSOR:: method without an intermediate +WT_CURSOR::set_key or WT_CURSOR::set_value call. In the 2.7.0 release, +WT_CURSOR::insert behaves like the other cursor methods and does not reference +application memory after a successful return. Applications depending on the +previous semantic will require modifications to set the cursor's key and/or +value after a successful WT_CURSOR::insert call. +</dd> + +<dt>WT_SESSION.verify</dt> +<dd> +The WT_SESSION.verify method in this release has a new configuration +option, \c strict. By default, with \c strict set to \c false, +WT_SESSION.verify will no longer return an error for problems that do +not impact the future use of the object (for example, if a leaked block +were detected, the application can continue to run). WT_SESSION.verify +will continue to output an error message whenever an error is detected, +only the final return value is affected. This change allows +applications to verify objects and continue if at all possible. +Applications in development should configure \c strict to \c true in +order to terminate the application whenever an error is detected. +</dd> + +</dl><hr> +@section version_261 Upgrading to Version 2.6.1 + +<dl> +<dt>Move the per transaction sync configuration</dt> +<dd> +Move the sync configuration setting from WT_SESSION::begin_transaction to +WT_SESSION::commit_transaction. Change the configuration from a boolean to a +string with options of on, off, background. Deprecated support for specifying +the configuration to WT_SESSION::begin_transaction is maintained for now. +</dd> + +</dl><hr> +@section version_260 Upgrading to Version 2.6.0 + +<dl> +<dt>LZ4 compression</dt> +<dd> +The LZ4 compression support has been updated in this release in non-backward +compatible ways; <b>tables and files compressed using LZ4 compression with +WiredTiger 2.5.3 must be dumped then re-loaded using WiredTiger 2.6.0.</b> +</dd> + +<dt>wiredtiger_open file manager configuration changes</dt> +<dd> +The configuration settings related to file handle management have changed +significantly in this release. If your application uses the file_manager +configuration setting, please refer to the API documentation for the latest +settings. +</dd> + +<dt>Statistic change</dt> +<dd> +The statistic "log read operations" was removed. The function that +used that statistic was never called inside WiredTiger and was also +removed. +</dd> + +<dt>File handle closing</dt> +<dd> +In WiredTiger, a separate thread of control periodically reviews open +objects, closing ones which have been idle for some period of time and +discarding them from the cache. In previous WiredTiger releases, this +review was independent of the number of open file handles. In this +release, WiredTiger only reviews the file handles if at least 250 file +handles are open. How often file handles are reviewed (the default is +30 seconds), and how many file handles must be opened before review is +triggered, can be configured using the \c file_manager configuration +values to the ::wiredtiger_open call. +</dd> + +<dt>Table drop changes</dt> +<dd> +There has been a functionality change to the WT_SESSION::drop API when the +\c force configuration option is included. The table is now dropped in the +background and does not flush data from cache. +</dd> + +</dl><hr> +@section version_253 Upgrading to Version 2.5.3 +<dl> + +<dt>Configuration string case-sensitivity</dt> +<dd> +In previous WiredTiger releases, there were several cases where configuration +strings were treated in a case-sensitive manner (for example, it was possible +to specify "True", "true" or even "tRuE" as a boolean value). For consistency, +in this release all WiredTiger configuration strings are case-sensitive, and +only "true" will be accepted. +</dd> + +<dt>Statistics cursor refresh</dt> +<dd> +In previous releases of WiredTiger, a statistics cursor made a snapshot +of the relevant statistics before the first statistics value was read, +and those values remained unchanged from that point on, no matter how +the cursor was used. In the WiredTiger 2.5.2 release, calling the +WT_CURSOR::reset method to reset the cursor refreshes the statistics +returned by the cursor. +</dd> + +<dt>Buffer alignment on Linux</dt> +<dd> +In previous releases of WiredTiger, all buffers used for I/O were aligned +to 4KB boundaries by default. In the WiredTiger 2.5.2 release, alignment +is only enforced when direct I/O is configured. +</dd> + +</dl><hr> +@section version_252 Upgrading to Version 2.5.2 + +There are no special upgrade steps required. + +@section version_251 Upgrading to Version 2.5.1 +<dl> + +<dt>Statistics cursors on data sources</dt> +<dd> +Opening a statistics cursor on a data source without a \c checkpoint name +no longer aggregates statistics for all open checkpoints on that data +source. The returned statistics will only be for the in-memory version of +the data source. + +For applications that read and write from ordinary tables (without +specifying a \c checkpoint), there will be no change. Applications that +opened cursors on checkpoints and relied on their statistics being +aggregated into a single statistics cursor will need to open statistics +cursors on each checkpoint. +</dd> + +</dl><hr> +@section version_250 Upgrading to Version 2.5.0 +<dl> + +<dt>WT_STAT_CONN_LOG_BYTES_USER renamed WT_STAT_CONN_LOG_BYTES_PAYLOAD</dt> +<dd> +The statistic for the number of bytes written to the log minus the overhead of +log record headers and padding was renamed to make the intent clearer. Any +applications using the old name will need to be updated. +</dd> + +<dt>"none" configuration for collators, compressors and extractors</dt> +<dd> +Collators, compressors and extractors can now be disabled with an explicit +\c "none" value as an alternative to using an empty string. Any applications +using the name \c "none" for a collator, compressor or extractor will need to +be updated. +</dd> + +<dt>maximum keys and value sizes +<dd> +The WT_SESSION::create \c internal_item_max and \c leaf_item_max +configuration strings are now deprecated in favor of the +\c internal_key_max, \c leaf_key_max, and \c leaf_value_max +configuration strings. See @ref tune_page_sizes for more information. +</dd> + +</dl><hr> +@section version_241 Upgrading to Version 2.4.1 +<dl> + +<dt>WT_DEADLOCK renamed</dt> +<dd> +The \c WT_DEADLOCK error return has been deprecated in favor of ::WT_ROLLBACK +to clarify that WT_SESSION::rollback_transaction should be called; no program +changes are required. +</dd> +<dt>Statistics keys changed</dt> +<dd> +The names of WiredTiger statistics have been updated to be more consistently +named, and simpler to categorize. Any application that was parsing the +strings output by statistics will need to be updated. +</dd> + +</dl><hr> +@section version_240 Upgrading to Version 2.4.0 +<dl> + +<dt>Default configuration file changes</dt> +<dd> +WiredTiger creates a configuration file when a database is first created. This +release adds a version number to that configuration file, and functionality +to automatically translate configuration settings between versions moving +forward. +</dd> +<dt>Cursors no longer reset on transaction begin or commit</dt> +<dd> +In previous versions of WiredTiger, all cursors in a session were reset at +transaction boundaries (WT_SESSION::begin_transaction, +WT_SESSION::commit_transaction and WT_SESSION::rollback_transaction). +Now they are only reset by WT_SESSION::rollback_transaction. + +This change means that a cursor can be used to iterate through a table and +perform transactional updates based on the visited records without the cursor +losing its position. + +Applications relying on this behavior (for example, to avoid pinning the +underlying cursor resources), should reset cursors explicitly when the +position is no longer required. +</dd> + +<dt>WT_COLLATOR interface changes</dt> +<dd> +Add a new WT_COLLATOR::customize callback that WiredTiger will call (if set) +for each data source configured to use the collator. Applications using the +existing WT_COLLATOR interface that do not require the new functionality should +set this callback to \c NULL. +</dd> +<dt>::wiredtiger_open lsm_merge option changed</dt> +<dd> +The global lsm_merge configuration setting has been moved into +the \c lsm_manager option group. To disable merges in all LSM +trees pass \c lsm_manger=(merge=false) to ::wiredtiger_open. +</dd> +<dt>Java include path search changes</dt> +<dd> +We have updated the configure logic used to search for Java Native Interface +files. This fixes problems building WiredTiger's Java API, particularly on +OS X, but may introduce problems if your build procedure relied on the old +search behavior. +</dd> +<dt>Default mutex implementation changed</dt> +<dd> +The default mutex implementation has been changed from adaptive pthread +mutexes to non-adaptive pthread mutexes. Installations can explicitly +select adaptive pthread mutexes by specifying +\c --with-spinlock=pthread_adaptive at configuration time. +</dd> +<dt>LSM merge threads option change</dt> +<dd> +The WT_SESSION::create \c lsm=(merge_threads) configuration option has been +replaced by the W::wiredtiger_open \c lsm_manager=(worker_thread_max) option. +The new version specifies a set of LSM threads that are shared across all +LSM trees in a database, the older configuration was per LSM table. +</dd> + +</dl><hr> +@section version_231 Upgrading to Version 2.3.1 +<dl> + +<dt>::wiredtiger_open eviction_workers configuration changed</dt> +<dd> +The \c eviction_workers configuration setting has been replaced by +\c eviction=(threads_min) and \c eviction=(threads_max) settings. + +There is also a semantic change because \c eviction_workers used to +configure additional threads whereas the new settings configure the total +number of threads involved with eviction. +</dd> + +</dl><hr> +@section version_230 Upgrading to Version 2.3.0 + +There are no special upgrade steps required. + +@section version_221 Upgrading to Version 2.2.1 +<dl> + +<dt>::wiredtiger_open configuration parsing order changed</dt> +<dd> +In the 2.2.1 release, the order that configuration strings are +parsed and override earlier values changed. Applications using +the \c Wiredtiger.config file or \c WIREDTIGER_CONFIG environment +variable may need to change. The old order: +<ol> +<li> default ::wiredtiger_open configuration</li> +<li> base configuration file, created with the database</li> +<li> user configuration file \c Wiredtiger.config</li> +<li> user environment variable \c WIREDTIGER_CONFIG</li> +<li> configuration string passed in to ::wiredtiger_open</li> +</ol> +<p>In the new order the user's configuration settings override:</p> +<ol> +<li> default ::wiredtiger_open configuration</li> +<li> base configuration file, created with the database</li> +<li> configuration string passed in to ::wiredtiger_open</li> +<li> user configuration file \c Wiredtiger.config</li> +<li> user environment variable \c WIREDTIGER_CONFIG</li> +</ol> + +</dd> +<dt>\c os_cache_dirty_max off for LSM</dt> +<dd> +In some earlier versions of WiredTiger, creating an LSM table automatically +configured \c os_cache_dirty_max, causing additional system calls that slowed +some workloads. Applications that benefit from this setting should set it +explicitly in WT_SESSION::create. +</dd> + +<dt>\c transaction_sync setting change</dt> +<dd> +In the 2.2.1 release of WiredTiger the ::wiredtiger_open \c transaction_sync +configuration setting has changed from a string value to a structure with two +fields: \c method, which corresponds to the previous value (\c dsync, \c fsync +or \c none), and \c enabled, which determines whether there is a sync on +every transaction commit by default. Applications can enable or disable sync +for a specific transaction with the \c sync setting to +WT_SESSION::begin_transaction. +</dd> + +</dl><hr> +@section version_220 Upgrading to Version 2.2.0 +<dl> + +<dt>WT_SESSION::create prefix_compression disabled by default</dt> +<dd> +In the 2.2.0 release, prefix compression default to \c false. Applications +that benefit from prefix compression will need to explicitly set +\c prefix_compression=true when creating tables. +</dd> + +<dt>wiredtiger_open verbose message changes</dt> +<dd> +In the 2.2.0 release it is now necessary to include \c --enable-verbose +in the configure command to be able to use verbose messages. +</dd> + +</dl><hr> +@section version_212 Upgrading to Version 2.1.2 +<dl> + +<dt>::wiredtiger_open shared_cache configuration changes</dt> +<dd> +In the 2.1.2 release of WiredTiger the ::wiredtiger_open \c shared_cache +configuration option group have changed. The option that was named \c enable is +no longer available. To enable a \c shared_cache it is compulsory to name +the pool being shared. +We are now also enforcing that only one of \c cache_size and \c shared_cache +are specified in the ::wiredtiger_open configuration string. +</dd> + +</dl><hr> +@section version_211 Upgrading to Version 2.1.1 +<dl> + +<dt>WT_EXTENSION_API::config methods</dt> +<dd> +In the 2.1.1 release of WiredTiger the configuration string parsing API +has been changed and added to a new public handle. The +WT_EXTENSION_API::config_strget, WT_EXTENSION_API::config_scan_begin, +WT_EXTENSION_API::config_scan_next and WT_EXTENSION_API::config_scan_end +have been removed. They have been replaced by a +WT_EXTENSION_API::config_parser_open method, which can be used to parse +configuration strings. See the WT_CONFIG_PARSER documentation for +examples on how to use the updated API. +</dd> + +</dl><hr> +@section version_21 Upgrading to Version 2.1 +<dl> + +<dt>WT_ITEM::size type</dt> +<dd> +In the 2.1 release of WiredTiger WT_ITEM::size type has changed from +\c uint32_t to \c size_t. Applications may require modifications to +resolve compile-time errors. +</dd> + +<dt>WT_COMPRESSOR::compress_raw signature</dt> +<dd> +In the 2.1 release of WiredTiger, the behavior of the compress_raw +callback has changed so that it will only be retried if it returns +\c EAGAIN. If it returns zero and sets \c result_slots to zero, +WiredTiger will assume that raw compression has failed and will fall +back to calling WT_COMPRESSOR::compress. +</dd> + +<dt>Transaction sync default setting</dt> +<dd> +In the 2.1 release of WiredTiger the ::wiredtiger_open \c transaction_sync +configuration setting default value has changed from "dsync" to "fsync". +This is due to enhancements to the group commit implementation in +WiredTiger - which mean that greater throughput can be achieved with +explicit "fsync" calls than by enabling "dsync" on a file handle. +Applications that don't execute concurrent transactions may see better +throughput with transaction_sync set to "dsync". +</dd> + +</dl><hr> +@section version_20 Upgrading to Version 2.0 +<dl> + +<dt>File format changes</dt> +<dd> +The underlying file format is unchanged in 2.0 +</dd> + +<dt>WT_SESSION::create LSM configuration options</dt> +<dd> +In the 2.0 release of WiredTiger the LSM configuration options have been +collected into a configuration option subgroup. All configuration options +to WT_SESSION::create that previously had a prefix of \c lsm_ now belong to +the \c lsm configuration group. If you are explicitly configuring any of the +following options, you should review the WT_SESSION::create documentation for +details of the updated syntax: lsm_auto_throttle, lsm_bloom, lsm_bloom_config, +lsm_bloom_bit_count, lsm_bloom_hash_count, lsm_bloom_oldest, lsm_chunk_max, +lsm_chunk_size, lsm_merge_max and lsm_merge_threads. +</dd> + +</dl><hr> +@section version_166 Upgrading to Version 1.6.6 +<dl> + +<dt>File format changes</dt> +<dd> +The underlying file formats changed in the 1.6.6 release; tables and files +should be dumped and re-loaded into a new database. +</dd> + +<dt>WT_SESSION::compact trigger configuration</dt> +<dd> +In previous releases, the \c trigger configuration string to the +WT_SESSION::compact method specified a requirement to initiate compaction; in +the 1.6.6 release, this configuration string has been removed, and compaction +will be attempted if it seems likely at least 10% of the file can be recovered. +Applications may require modifications to resolve run-time errors. +</dd> + +<dt>Statistics configuration</dt> +<dd> +In previous releases, the ::wiredtiger_open function took a \c statistics +configuration, which defaulted to false; when set to true, additional, +generally performance-expensive statistics were maintained by the +database, above and beyond a default set of statistics. In version +1.6.6, the \c statistics configuration is a list which may be set to +"all", "fast" or "none". When set to "none", no statistics are +maintained by the database; when set to "fast", only relatively +performance-inexpensive statistics are maintained, and when set to +"all", all statistics are maintained, regardless of cost. + +In previous releases, the ::wiredtiger_open function took a \c statistics_log +configuration which logged the performance-inexpensive database +statistics to a file. In version 1.6.6, the \c statistics_log +configuration logs whatever statistics are configured for the database. +If the database is configured with \c statistics to "none", no +statistics will be logged to the file; if the database is configured +with "all" or "fast", the corresponding statistics will be logged to the +file. + +In previous releases, the WT_SESSION::cursor method took +\c statistics_clear and a \c statistics_fast configurations. The +\c statistics_clear configuration defaulted to false; when set to true, +statistics counters were reset after they were gathered by the cursor. +The \c statistics_fast configuration defaulted to true; when set to +true, the cursor only gathered performance-inexpensive statistics for +the cursor, and when set to false, the cursor gathered all available +statistics, regardless of cost. + +In version 1.6.6, these two configuration booleans have been replaced +with a new configuration list \c statistics, which may be set from the +values "clear", "fast" and "all". When "fast" is configured, only +relatively performance-inexpensive statistics are gathered, and when +"all" is configured, all statistics are gathered, regardless of cost. +When "clear" is configured, statistics counters are reset after they are +gathered. + +Additionally, in version 1.6.6, statistics cursors must be configured +to agree with the database statistics configuration; when the database +statistics are configured to "none", attempts to open a statistics +cursor will fail; when the database statistics are configured to "fast", +a statistics cursor must also be configured to "fast"; when the database +statistics are configured to "all", a statistics cursor may be +configured to either "fast" or "all". Opening a statistics cursor +without configuring either "fast" or "all" will configure the cursor to +be the same as the current database configuration. + +Applications may require modifications to resolve run-time errors; +application statistics configuration and cursors should be reviewed to +confirm they are configured for the desired behavior; +</dd> + +<dt>WT_EVENT_HANDLER interface changes</dt> +<dd> +Add a new WT_EVENT_HANDLER::handle_close callback that WiredTiger will call +any time it automatically closes an application session or cursor handle. + +Additionally add a WT_SESSION parameter into the existing +WT_EVENT_HANDLER::handle_error, WT_EVENT_HANDLER::handle_message and +WT_EVENT_HANDLER::handle_progress callback functions. +</dd> + +</dl><hr> +@section version_165 Upgrading to Version 1.6.5 +<dl> + +<dt>WT_CURSOR::insert behavior</dt> +<dd> +In previous releases, the WT_CURSOR::insert ended positioned at the inserted +record. To minimize the cursor resources held by applications inserting many +records, the WT_CURSOR::insert method has been changed to end without any +position. Application insert cursors should be reviewed to confirm they do +not attempt to iterate after an insert. +</dd> + +<dt>WT_SESSION::open_cursor statistics_fast configuration</dt> +<dd> +In previous releases, the default \c statistics_fast configuration to +the WT_SESSION::open_cursor method was \c false; in the 1.6.5 release, +the default statistics_fast configuration is \c true. Applications +opening statistics cursors should be reviewed to confirm they have the +correct behavior. +</dd> + +<dt>Synchronous checkpoint configuration</dt> +<dd> +The \c sync configuration key to ::wiredtiger_open has been renamed \c checkpoint_sync. +</dd> + +</dl><hr> +@section version_164 Upgrading to Version 1.6.4 +<dl> + +<dt>File format changes</dt> +<dd> +The underlying file formats changed in the 1.6.4 release; tables and files +should be dumped and re-loaded into a new database. +</dd> + +<dt>wt utility load command</dt> +<dd> +The default behavior of the \c wt utility's \c load command has been +changed to overwrite existing data, by default, and the \c -o flag to +the \c load command (overwrite existing data) has been replaced with the +\c -n flag (do not overwrite existing data). Applications requiring +the previous default behavior of not overwriting existing data should +add the \c -n option to their command line configuration; applications +previously using the \c -o option on their command line configurations +should remove it. +</dd> + +</dl><hr> +@section version_163 Upgrading to Version 1.6.3 +<dl> + +<dt>Cursor overwrite configuration</dt> +<dd> +In previous releases, the WT_SESSION::open_cursor \c overwrite configuration +string behaved inconsistently across Btree and LSM data sources. In Btree, +\c overwrite was \c false by default and was limited to the WT_CURSOR::insert +method, changing an insert to succeed regardless of whether or not the record +previously existed. In LSM trees, \c overwrite was \c true by default, and +applied to the WT_CURSOR::insert, WT_CURSOR::remove and WT_CURSOR::update +methods, configuring all three methods to ignore the existing state of the +record. + +In the 1.6.3 release, the \c overwrite configuration is consistent across both +Btree and LSM tree data sources. For performance reasons, the default is the +behavior previously described for LSM trees: in other words, \c overwrite is +\c true by default, causing WT_CURSOR::insert, WT_CURSOR::remove and +WT_CURSOR::update to ignore the current state of the record, and these methods +will succeed regardless of whether or not the record previously exists. When +an application configures \c overwrite to \c false, WT_CURSOR::insert will fail +with ::WT_DUPLICATE_KEY if the record previously exists, and WT_CURSOR::update +and WT_CURSOR::remove will fail with ::WT_NOTFOUND if the record does not +previously exist. + +<b>This is a potentially serious API change that will not be detected by +compilation.</b> Application cursors should be reviewed to confirm they are +configured for the desired behavior. +</dd> + +<dt>::wiredtiger_open no longer accepts a \c transactional configuration</dt> +<dd> +The \c transactional configuration key has been removed from +::wiredtiger_open. Any application setting it should simply remove it, +no change in application behavior is needed. +</dd> + +</dl><hr> +@section version_162 Upgrading to Version 1.6.2 +<dl> + +<dt>Table of WiredTiger extension methods</dt> +<dd> +New functionality was added to the list of WiredTiger extension methods; +applications using the extension methods will require recompilation. +</dd> + +<dt>WT_SESSION::create no longer accepts a "source" configuration</dt> +<dd> +The \c "source" configuration key has been removed from WT_SESSION::create. +Normal applications should not have been using it, and there were a number of +bugs associated with it. +</dd> + +<dt>Default checksum configuration</dt> +<dd> +The default file checksum configuration was changed to \c uncompressed, +which means blocks that are compressed will no longer also include a +checksum, by default. Applications using compression insufficient for +the purposes of corrupted block identification should change their file +checksum configuration to \c on. +</dd> + +</dl><hr> +@section version_161 Upgrading to Version 1.6.1 +<dl> + +<dt>Default page sizes</dt> +<dd> +In the 1.6.1 release, the default for the WT_SESSION::create configuration +string \c allocation_size changed from 512B to 4KB, and the default for the +configuration string \c internal_page_max changed from 2KB to 4KB. +Applications wanting to create files with smaller allocation or internal +page sizes will need to set those configuration values explicitly. +</dd> + +<dt>Shared cache configuration</dt> +<dd> +In the 1.6.1 release, an explicit shared_cache=(enable=boolean) option was +added to the wiredtiger_open configuration options. Existing applications that +use shared cache functionality will need to add the enable option to the +configuration string. The default value for the option is false. +</dd> + +<dt>WT_COMPRESSOR::compress_raw signature</dt> +<dd> +In the 1.6.1 release, the \c split_pct argument to the +WT_COMPRESSOR::compress_raw function changed type from \c u_int to \c int, +applications may require modification to avoid compiler warnings. +</dd> + +</dl><hr> +@section version_160 Upgrading to Version 1.6.0 +<dl> + +<dt>File format changes</dt> +<dd> +The underlying file formats changed in the 1.6.0 release; tables and files +should be dumped and re-loaded into a new database. +</dd> + +</dl><hr> +@section version_153 Upgrading to Version 1.5.3 +<dl> + +<dt>Configuration strings</dt> +<dd> +An undocumented feature where configuration string case was ignored has +been removed, and all configuration strings are now case-dependent. +Applications may require modifications to resolve run-time errors. +</dd> + +<dt>Loading extensions and WT_EXTENSION_API</dt> +<dd> +The following changes are only applicable to applications loading +extensions and/or using the WiredTiger extension functions described in +WT_EXTENSION_API. + +<ul> +<li> +The signature of ::wiredtiger_extension_init has changed +from <code>(WT_SESSION *session, WT_EXTENSION_API *api)</code> to +<code>(WT_CONNECTION *connection)</code>. As no WT_EXTENSION_API +handle reference is passed to the function, the +WT_CONNECTION::get_extension_api has been added to support retrieval of +the extension API. Applications may require modifications. + +<li> +The type of all configuration arguments to extension methods has changed +from \c "const char *" to \c "WT_CONFIG_ARG *", and the WT_EXTENSION::config +method added to support configuration parsing; applications may require +modifications. + +<li> +The undocumented \c wiredtiger_XXX defines for WT_EXTENSION_API extension +methods have been removed from the \c wiredtiger_ext.h include file; +applications should instead use the method handles referenced by the +WT_EXTENSION_API handle to call extension functions. + +<li> +The extension API methods have all changed to require an additional +parameter, the WT_EXTENSION_API method handle; applications may require +modifications. + +<li> +The WT_SESSION::msg_printf method was replaced by WT_EXTENSION_API::msg_printf; +applications may require modifications. +</ul> +</dd> + +<dt>WT_DATA_SOURCE</dt> +<dd> +The following changes are only applicable to applications providing new +implementations of the WiredTiger WT_DATA_SOURCE class. + +<ul> +<li> +The WT_DATA_SOURCE class has three new methods: WT_DATA_SOURCE::compact, +WT_DATA_SOURCE::salvage, and WT_DATA_SOURCE::verify; applications may +require modifications to resolve compile errors. + +<li> +The \c owner argument to the WT_DATA_SOURCE::open_cursor method has been +removed; applications may require modifications to resolve compile errors. + +<li> +The \c exclusive argument to the WT_DATA_SOURCE::create method has been +removed; applications may require modifications to resolve compile errors. +</ul> +</dd> + +</dl><hr> +@section version_143 Upgrading to Version 1.4.3 +<dl> + +<dt>Statistics</dt> +<dd> +WiredTiger statistics are no longer maintained by default; to configure +statistics, use the \c statistics configuration string to the +::wiredtiger_open function. +</dd> + +</dl><hr> +@section version_139 Upgrading to Version 1.3.9 +<dl> + +<dt>Compression</dt> +<dd> +A new member, WT_COMPRESSOR::compress_raw, was added to the WT_COMPRESSOR +extension API. Applications using the WT_COMPRESSOR extension API should +add a NULL as the second field of that structure. +</dd> + +<dt>Checksums</dt> +<dd> +The WT_SESSION::create method's \c checksum configuration string has +been changed from a boolean type to a string type. Applications using +the checksum configuration string should change a value of \c true to +the string \c on, and a value of \c false to the string \c off or the +string \c uncompressed. +</dd> + +<dt>File format changes</dt> +<dd> +The underlying file formats changed in the 1.3.9 release; tables and files +should be dumped and re-loaded into a new database. +</dd> + +</dl><hr> +@section version_138 Upgrading to Version 1.3.8 +<dl> + +<dt>Statistics keys</dt> +<dd> +The @ref statistics_keys "statistics key constants" have been renamed to use +all capitals, and use consistent prefixes to distinguish between connection +statistics and statistics for data sources. +</dd> + +</dl><hr> +@section version_136 Upgrading to Version 1.3.6 +<dl> + +<dt>Installed library names</dt> +<dd> +The installed WiredTiger extension library names changed to limit +namespace pollution: + +<table> +@hrow{Library, Previous Name, New Name} +@row{Bzip2 compression, bzip2_compress.a, libwiredtiger_bzip2.a} +@row{, bzip2_compress.la, libwiredtiger_bzip2.la} +@row{, bzip2_compress.so, libwiredtiger_bzip2.so} +@row{Snappy compression, snappy_compress.a, libwiredtiger_snappy.a} +@row{, snappy_compress.la, libwiredtiger_snappy.la} +@row{, snappy_compress.so, libwiredtiger_snappy.so} +@row{No-op compression, nop_compress.a, No longer installed} +@row{, nop_compress.la, No longer installed} +@row{, nop_compress.so, No longer installed} +@row{Reverse order collator, reverse_collator.a, No longer installed} +@row{, reverse_collator.la, No longer installed} +@row{, reverse_collator.so, No longer installed} +</table> +</dd> + +<dt>Built-in compression names</dt> +<dd> +The built-in compression name arguments to the WT_SESSION:create +\c block_compressor configuration string changed for consistency: + +<table> +@hrow{Extension, Previous Name, New Name} +@row{Bzip2 compression, "bzip2_compress", "bzip2"} +@row{Snappy compression, "snappy_compress", "snappy"} +</table> +</dd> + +</dl><hr> +@section version_135 Upgrading to Version 1.3.5 +<dl> + +<dt>File format changes</dt> +<dd> +The underlying file formats changed in the 1.3.5 release; tables and files +should be dumped and re-loaded into a new database. +</dd> + +</dl><hr> +@section version_13 Upgrading to Version 1.3 +<dl> + +<dt>Checkpoint and Snapshot</dt> +<dd> +The checkpoint functionality supported by WT_SESSION::checkpoint and the +snapshot functionality supported by WT_SESSION::sync have been merged +into a single piece of functionality. + +<ul> +<li>WT_SESSION.checkpoint<br> +The WT_SESSION::checkpoint method's \c snapshot configuration string has +been renamed to \c name. The name assigned to checkpoints without a +specified \c name configuration is now \c "WiredTigerCheckpoint". + +<li>WT_SESSION.drop<br> +In releases before 1.3, the WT_SESSION::drop method was used to delete +snapshots. In 1.3, the functionality of deleting snapshots has been +moved to the WT_SESSION::checkpoint method, specifically, snapshots are +discarded using the WT_SESSION::checkpoint method's \c drop configuration +string. + +<li>WT_SESSION.sync<br> +The WT_SESSION::sync method has been removed from the 1.3 release; the +functionality of creating an object snapshot has moved to the +WT_SESSION::checkpoint method, specifically, creating a snapshot of a +one or more objects is done using the WT_SESSION::checkpoint method's +\c target configuration string. + +<li>wt drop -s<br> +The \c -s option to the \c drop command for the \c wt command line +utility has been removed, and object snapshots may no longer be removed +from the command line. + +<li>wt dump, list -s<br> +The \c -s options to the \c dump and \c list commands for the \c wt +command line utility have been renamed to be \c -c. +</ul> +</dd> + +<dt>WT_SESSION.open_cursor</dt> +<dd> +In releases before 1.3, the WT_SESSION::open_cursor method could duplicate +cursors that were not positioned in an object; in 1.3, a cursor must be +positioned in order to be duplicated. +</dd> + +<dt>Transactional cursors</dt> +<dd> +In releases before 1.3, ending a transaction by calling the +WT_SESSION::commit_transaction or WT_SESSION::rollback_transaction methods +implicitly closed all open cursors; in 1.3, the cursors remain open, but are +reset (discarding their positions and cursor values). This means applications +must change to either close cursors explicitly, or rely on an eventual +WT_SESSION::close or WT_CONNECTION::close methods to implicitly close open +cursors. +</dd> + +<dt>Default transactional isolation level</dt> +<dd> +In releases before 1.3, the default isolation level for transaction was +\c snapshot, and the default isolation level for non-transaction operations +was \c read-uncommitted; in 1.3, the default isolation level for all +operations is \c read-committed. + +The default can be overridden for a session using the \c isolation setting +in WT_CONNECTION::open_cursor. +</dd> + +<dt>WT_SESSION.truncate</dt> +<dd> +In releases before 1.3, the WT_SESSION::truncate method required cursors used +for truncation of a cursor range to reference existing keys in the object; +in 1.3, the WT_SESSION::truncate method has been changed to allow cursors to +reference any valid key in the object's name space so applications may discard +portions of the object name space without knowing exactly what records the +object contains. +</dd> + +<dt>WT_CURSOR.equals</dt> +<dd> +In releases before 1.3, the WT_CURSOR::equals method returned zero/non-zero +to indicate cursor equality; in 1.3, the WT_CURSOR::equals method has +been replaced with WT_CURSOR::compare, which compares two cursors and +returns a cursor comparison status (less than 0, equal to 0, or greater than +0) depending on the cursors' key order. +</dd> + +<dt>File format changes</dt> +<dd> +The underlying file formats changed in the 1.3 release; tables and files +should be dumped and re-loaded into a new database. +</dd> + +</dl> +*/ diff --git a/src/third_party/wiredtiger/src/docs/wtperf.dox b/src/third_party/wiredtiger/src/docs/wtperf.dox new file mode 100644 index 00000000000..64e25978dd8 --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/wtperf.dox @@ -0,0 +1,269 @@ +/*! @page wtperf Simulating workloads with wtperf + +The WiredTiger distribution includes a tool that can be used to simulate +workloads in WiredTiger, in the directory \c bench/wtperf. + +The \c wtperf utility generally has two phases, the populate phase which +creates a database and then populates an object in that database, and a +workload phase, that does some set of operations on the object. + +For example, the following configuration uses a single thread to +populate a file object with 500,000 records in a 500MB cache. The +workload phase consists of 8 threads running for two minutes, all +reading from the file. + +@code +conn_config="cache_size=500MB" +table_config="type=file" +icount=500000 +run_time=120 +populate_threads=1 +threads=((count=8,reads=1)) +@endcode + +In most cases, where the workload is the only interesting phase, the +populate phase can be performed once and the workload phase run +repeatedly (for more information, see the wtperf \c create configuration +variable). + +The \c conn_config configuration supports setting any WiredTiger +connection configuration value. This is commonly used to configure +statistics with regular reports, to obtain more information from the +run: + +@code +conn_config="cache_size=20G,statistics=(fast,clear),statistics_log=(wait=600)" +report_interval=5 +@endcode + +Note quoting must be used when passing values to Wiredtiger +configuration, as opposed to configuring the \c wtperf utility itself. + +The \c table_config configuration supports setting any WiredTiger object +creation configuration value, for example, the above test can be +converted to using an LSM store instead of a B+tree store, with +additional LSM configuration, by changing \c conn_config to: + +@code +table_config="lsm=(chunk_size=5MB),type=lsm,os_cache_dirty_max=16MB" +@endcode + +More complex workloads can be configured by creating more threads doing +inserts and updates as well as reads. For example, to configure two +inserting threads two threads doing a mixture of inserts, reads and +updates: + +@code +threads=((count=2,inserts=1),(count=2,inserts=1,reads=1,updates=1)) +@endcode + +Example \c wtperf configuration files can be found in the +\c bench/wtperf/runners/ directory. + +There are also a number of command line arguments that can be passed +to \c wtperf: +@par <code>-C config</code> +Specify configuration strings for the ::wiredtiger_open function. +This argument is additive to the \c conn_config parameter in the +configuration file. +@par <code>-h directory</code> +Specify a database home directory. The default is \c ./WT_TEST. +@par <code>-m monitor_directory</code> +Specify a directory for all monitoring related files. +The default is the database home directory. +@par <code>-O config_file</code> +Specify the configuration file to run. +@par <code>-o config</code> +Specify configuration strings for the \c wtperf program. +This argument will override settings in the configuration file. +@par <code>-T config</code> +Specify configuration strings for the WT_SESSION::create function. +This argument is additive to the \c table_config parameter in the +configuration file. + +@section monitor Monitoring wtperf + +Like all WiredTiger applications, the \c wtperf command can be configured +with statistics logging, and the resulting output displayed using the +\c wtstats visualization tool. For more information, see @ref wtstats. + +In addition to statistics logging, \c wtperf can monitor performance and +operation latency times. Monitoring is enabled using the \c sample_interval +configuration. For example to record information every 10 seconds, set the +following on the command line or add it to the \c wtperf configuration file: + +@code +sample_interval=10 +@endcode + +Enabling monitoring causes \c wtperf to create a file \c monitor in the +database home directory (or another directory as specified using the +\c -m option to \c wtperf). + +The same visualization tool, \c wtstats, can be used to view a combined +chart with both the \c monitor output and the statistics logging output +at the same time. + +The following example shows how to run the \c medium-btree.wtperf configuration +with monitoring enabled, and then generate a graph. + +@code +# Change into the WiredTiger directory. +cd wiredtiger + +# Configure and build WiredTiger if not already built. +./configure && make + +# Remove and re-create the run directory. +rm -rf WTPERF_RUN && mkdir WTPERF_RUN + +# Run the medium-btree.wtperf workload, sampling performance every 5 seconds. +bench/wtperf/wtperf \ + -h WTPERF_RUN \ + -o sample_interval=5 \ + -O bench/wtperf/runners/medium-btree.wtperf + +# Use the visualization tool to create HTML graph output; the output file is +# named wtstats.html. +python tools/wtstats/wtstats.py WTPERF_RUN/monitor + +# Possible alternatives if statistics logging also enabled: +# python tools/wtstats/wtstats.py WTPERF_RUN/monitor WTPERF_RUN/WiredTigerStat* +# python tools/wtstats/wtstats.py WTPERF_RUN +@endcode + +The python command creates a file named \c wtstats.html in the current +working directory. You can open the generated HTML document in your browser +and see the generated statistics. + +@section config Wtperf configuration options + +The following is a list of the currently available \c wtperf +configuration options: + +\if START_AUTO_GENERATED_WTPERF_CONFIGURATION +DO NOT EDIT: THIS PART OF THE FILE IS GENERATED BY dist/s_docs. +\endif + +@par async_threads (unsigned int, default=0) +number of async worker threads +@par checkpoint_interval (unsigned int, default=120) +checkpoint every interval seconds during the workload phase. +@par checkpoint_stress_rate (unsigned int, default=0) +checkpoint every rate operations during the populate phase in the +populate thread(s), 0 to disable +@par checkpoint_threads (unsigned int, default=0) +number of checkpoint threads +@par conn_config (string, default=create) +connection configuration string +@par compact (boolean, default=false) +post-populate compact for LSM merging activity +@par compression (string, default=none) +compression extension. Allowed configuration values are: 'none', +'bzip', 'lz4', 'snappy', 'zlib' +@par create (boolean, default=true) +do population phase; false to use existing database +@par database_count (unsigned int, default=1) +number of WiredTiger databases to use. Each database will execute the +workload using a separate home directory and complete set of worker +threads +@par drop_tables (unsigned int, default=0) +Whether to drop all tables at the end of the run, and report time +taken to do the drop. +@par icount (unsigned int, default=5000) +number of records to initially populate. If multiple tables are +configured the count is spread evenly across all tables. +@par idle_table_cycle (unsigned int, default=0) +Enable regular create and drop of idle tables, value is the maximum +number of seconds a create or drop is allowed before flagging an +error. Default 0 which means disabled. +@par index (boolean, default=false) +Whether to create an index on the value field. +@par insert_rmw (boolean, default=false) +execute a read prior to each insert in workload phase +@par key_sz (unsigned int, default=20) +key size +@par log_partial (boolean, default=false) +perform partial logging on first table only. +@par min_throughput (unsigned int, default=0) +notify if any throughput measured is less than this amount. Aborts or +prints warning based on min_throughput_fatal setting. Requires +sample_interval to be configured +@par min_throughput_fatal (boolean, default=false) +print warning (false) or abort (true) of min_throughput failure. +@par max_latency (unsigned int, default=0) +notify if any latency measured exceeds this number of +milliseconds.Aborts or prints warning based on min_throughput_fatal +setting. Requires sample_interval to be configured +@par max_latency_fatal (boolean, default=false) +print warning (false) or abort (true) of max_latency failure. +@par pareto (unsigned int, default=0) +use pareto distribution for random numbers. Zero to disable, otherwise +a percentage indicating how aggressive the distribution should be. +@par populate_ops_per_txn (unsigned int, default=0) +number of operations to group into each transaction in the populate +phase, zero for auto-commit +@par populate_threads (unsigned int, default=1) +number of populate threads, 1 for bulk load +@par random_range (unsigned int, default=0) +if non zero choose a value from within this range as the key for +insert operations +@par random_value (boolean, default=false) +generate random content for the value +@par read_range (unsigned int, default=0) +scan a range of keys after each search +@par reopen_connection (boolean, default=true) +close and reopen the connection between populate and workload phases +@par report_interval (unsigned int, default=2) +output throughput information every interval seconds, 0 to disable +@par run_ops (unsigned int, default=0) +total read, insert and update workload operations +@par run_time (unsigned int, default=0) +total workload seconds +@par sample_interval (unsigned int, default=0) +performance logging every interval seconds, 0 to disable +@par sample_rate (unsigned int, default=50) +how often the latency of operations is measured. One for every +operation,two for every second operation, three for every third +operation etc. +@par sess_config (string, default=) +session configuration string +@par table_config (string, default=key_format=S,value_format=S,type=lsm,exclusive=true,allocation_size=4kb,internal_page_max=64kb,leaf_page_max=4kb,split_pct=100) +table configuration string +@par table_count (unsigned int, default=1) +number of tables to run operations over. Keys are divided evenly over +the tables. Cursors are held open on all tables. Default 1, maximum +99999. +@par table_count_idle (unsigned int, default=0) +number of tables to create, that won't be populated. Default 0. +@par threads (string, default=) +workload configuration: each 'count' entry is the total number of +threads, and the 'insert', 'read' and 'update' entries are the ratios +of insert, read and update operations done by each worker thread; If a +throttle value is provided each thread will do a maximum of that +number of operations per second; multiple workload configurations may +be specified per threads configuration; for example, a more complex +threads configuration might be +'threads=((count=2,reads=1)(count=8,reads=1,inserts=2,updates=1))' +which would create 2 threads doing nothing but reads and 8 threads +each doing 50% inserts and 25% reads and updates. Allowed +configuration values are 'count', 'throttle', 'reads', 'inserts', +'updates', 'truncate', 'truncate_pct' and 'truncate_count'. There are +also behavior modifiers, supported modifiers are 'ops_per_txn' +@par transaction_config (string, default=) +transaction configuration string, relevant when populate_opts_per_txn +is nonzero +@par table_name (string, default=test) +table name +@par value_sz (unsigned int, default=100) +value size +@par verbose (unsigned int, default=1) +verbosity +@par warmup (unsigned int, default=0) +How long to run the workload phase before starting measurements + +\if STOP_AUTO_GENERATED_WTPERF_CONFIGURATION +DO NOT EDIT: THIS PART OF THE FILE IS GENERATED BY dist/s_docs. +\endif + +*/ diff --git a/src/third_party/wiredtiger/src/docs/wtstats.dox b/src/third_party/wiredtiger/src/docs/wtstats.dox new file mode 100644 index 00000000000..c4218049a4a --- /dev/null +++ b/src/third_party/wiredtiger/src/docs/wtstats.dox @@ -0,0 +1,119 @@ +/*! @page wtstats Visualizing performance with wtstats + +The WiredTiger distribution includes the \b wtstats tool that can be +used to examine information generated using statistics logging (see +@ref statistics_log) and wtperf monitoring output (see @ref wtperf). + +After running an application with statistics logging configured, the +statistics output files will be in the database home directory. By +default they are the files named with a \c WiredTigerStat prefix and a +day and hour timestamp. + +To create an HTML file with a graphical representation of the statistics +run the following command, replacing \c wiredtiger with the path to the +WiredTiger build directory, and replacing \c database-directory with the +application's database home directory: + +\code{.sh} +python wiredtiger/tools/wtstats/wtstats.py database-directory +\endcode + +If not all of the statistics files are being processed, a subset can be +specified to the tool. For example: + +\code{.sh} +python wiredtiger/tools/wtstats.py database-directory/WiredTigerStat.06.* +\endcode + +In either case, a \c wtstats.html file will be generated in the current +directory that you can open in your browser. + +Additional options are available, use <tt>wtstats.py --help</tt> to +display them. + +The following is an example \c wtstats.html display: + +\image html wtstats.png "wtstats.html" + +@section wtstats_display Manipulating the wtstats display + +When you open the generated HTML file, you will see an empty page with +some basic instructions in the middle, a sidebar containing grouped +sections on the left, and buttons to toggle axis settings at the top +right. + +To look at specific statistics, open the respective section by clicking +on the section heading or the little arrow on the right. For example, +click on the word "connection" in the sidebar to reveal the _connection_ +stats. Inside a group, click on any of the contained stats to toggle +them on or off. You should see the stats graph appear in the main +window. To toggle the entire group on or off, you can click on the +circle next to the group heading. To look at one specific isolated +statistic, you can shift-click on the stat. This will disable all but +the chosen stat. Shift-clicking it again will return to the previous +selection. + +Depending on your WiredTiger settings, you may have gathered a large +number of stats. Use the search bar above the stats groups to filter +stats on keywords. For example, typing `mem` into the search bar will +show all stats that have the substring "mem" in them. Press the "Clear" +button to clear the filter again. + +Once some stats are showing in the main view panel, use the mouse to +hover over the graphs. A cross-hair will appear highlighting the data +point closest to the mouse cursor and showing the stat name and value +(y-axis). A small gray label below the x-axis will additionally show the +exact x-axis value for the current data point. + +You can zoom into the graph via various mouse/trackpad gestures (some +of which may not be supported by your input device): + +- use the scroll wheel (mouse) +- use two-finger vertical scroll (touchpad) +- use pinch-to-zoom gesture (touchpad) +- use double-click to zoom in, shift+double-click to zoom out (mouse, touchpad) + +If you zoom into a dataset far enough to see individual data points, +they will be displayed as little circles along the graph line. + +You can pan the graphs left/right by click-dragging the area +horizontally. + +If your data contains a lot of sample points (more than pixels available +in the current window), \c wtstats will sub-sample your data for +efficiency reasons. When this happens, a yellow warning box will appear +at the top next to the buttons. Due to sub-sampling, narrow spikes in a +graph may not always be visible and you may experience some jitter when +panning the data left/right. To ensure that all data points are +rendered, widen your browser window or zoom into the data until the +sub-sampling warning disappears. + +The \c wtstats display lets you switch modes for both the x- and y-axis +with the buttons at the top right. + +The x-axis button switches between relative and absolute time, with the +default being relative time. + +In "relative time" mode, every individual stat is assumed to start at +time 0, and the x-axis displays the duration in seconds that have passed +since the start. A useful use case for relative mode is when you record +multiple stat files of the same data workload in succession, and want +to compare them against each other. Relative mode will show the stat +files overlapping, despite them being recorded at different times. + +Absolute time plots the stats at their exact recording times. The x-axis +shows date and time of the events. If stat files are recorded at +different times, they appear in the graph at different intervals on the +x-axis. This mode is most useful to correlate events in \c wtstats with +other data sources, for example system profiling or monitoring data, +based on their timestamp. + +The y-axis button toggles between linear and logarithmic scaling mode, +the default is linear. Linear mode is most useful if the visible stats +have similar value ranges. However, if you compare stats of different +magnitudes, for example "bytes in cache" vs. "files open", you can +switch to "log-scale" mode to avoid having one graph dominate the entire +range. The same is true for outliers in a single graph: if the outlier +dominates the range, switch to log mode for better scaling. + +*/ |
