Convert the testing wiki page into a page in the documentation.

3 files changed, 220 insertions, 63 deletions

diff --git a/src/docs/introduction.dox b/src/docs/introduction.dox
index 80a7a4c2cd6..4ecd9aca744 100644
--- a/src/docs/introduction.dox
+++ b/src/docs/introduction.dox
@@ -48,6 +48,8 @@ For more information about using WiredTiger, see:
 
 - @subpage admin
 
+- @subpage testing
+
 - @subpage license
 
 To browse the WiredTiger source code or contact us, see:
diff --git a/src/docs/spell.ok b/src/docs/spell.ok
index 9b3dea67212..0fbe636fb84 100644
--- a/src/docs/spell.ok
+++ b/src/docs/spell.ok
@@ -1,71 +1,19 @@
 personal_ws-1.1 en 200
+ack'ed
 ACM
-APIs
 Adler's
-Atomicity
-BLOBs
-CFLAGS
-CPPFLAGS
-Cheng
-Christoph
-DB's
-DBTs
-DbCursor
-DbEnv
-DbMultiple
-EAGAIN
-EB
-EBUSY
-ECMA
-EINVAL
-EmpId
-FreeBSD
-FreeBSD's
-GCC
-Gawlick
-GitHub
-Google
-Google's
-IEC
-JavaScript
-LDFLAGS
-LIBS
-LSB
-LSM
-Lameter
-Levyx
-MERCHANTABILITY
-MVCC's
-Makefiles
-Mewhort
-NOTFOUND
-NUMA
-NoSQL
-README
-RedHat
-RepMgr
-Roelofs
-Rrx
-Seward's
-SiS
-TXT
-URIs
-Vv
-WiredTiger
-WiredTiger's
-WiredTigerCheckpoint
-Za
-aR
-ack'ed
 alloc
 allocator
 allocsize
 ao
 api
 apiflags
+APIs
 ar
+aR
 archiver
 arg
+Atomicity
 autoconf
 automake
 basecfg
@@ -74,6 +22,7 @@ bdbmap
 bigram
 bindir
 bitstring
+BLOBs
 bool
 boolean
 booleans
@@ -88,12 +37,16 @@ callbk
 cd
 cdb
 cds
+CFLAGS
 changelog
 checksum
 checksums
+Cheng
+Christoph
 ckp
 colgroup
 colgroups
+combinatorial
 command's
 comparator
 cond
@@ -103,19 +56,23 @@ conn
 const
 control's
 cpp
+CPPFLAGS
 crashless
 cursortype
 cv
-dN
-dNLen
-dNOff
-dT
-dataN
 dataitem
+dataN
 datasets
+datastore
+Datastore
 dbc
+DbCursor
+DbEnv
 dbm
+DbMultiple
+DB's
 dbt
+DBTs
 decl
 del
 desc
@@ -125,15 +82,25 @@ dev
 distclean
 dl
 dlp
+dN
+dNLen
+dNOff
 dontlock
 dp
 ds
 dsrc
 dsync
 dt
+dT
 dumpfile
 dup
 dups
+EAGAIN
+EB
+EBUSY
+ECMA
+EINVAL
+EmpId
 endcode
 endcond
 endian
@@ -156,16 +123,24 @@ filesystems
 fillfactor
 firstname
 fnv
+forw
 fput
+FreeBSD
+FreeBSD's
 freelist
 fsync
+Gawlick
 gcc
+GCC
 gdbm
 getopt
 getter
 gid
 github
+GitHub
 gnuplot
+Google
+Google's
 hb
 hotbackup
 href
@@ -174,6 +149,7 @@ hsearch
 html
 huffman
 hugepage
+IEC
 iflag
 indices
 init
@@ -181,6 +157,7 @@ intl
 inuse
 io
 ip
+JavaScript
 je
 jemalloc
 jni
@@ -191,12 +168,16 @@ keyfmt
 keyname
 keyvalue
 kvs
+Lameter
 lastname
+LDFLAGS
 len
+Levyx
 li
 libdir
 libhe
 libkvs
+LIBS
 libtool
 libwiredtiger
 lmin
@@ -206,11 +187,14 @@ logc
 lookup
 lookups
 lrtf
+LSB
 lsm
+LSM
 lsn
 lt
 mailto
 mainpage
+Makefiles
 malloc
 malloc'd
 marshalled
@@ -223,8 +207,12 @@ md
 memalloc
 memfree
 memp
+MERCHANTABILITY
 metadata
+Mewhort
 minkey
+mixin
+mixins
 mkdir
 mmap
 mpool
@@ -233,12 +221,15 @@ msg
 msgs
 multi
 multiprocess
+multithreaded
+Multithreaded
 multithreading
 multiversion
 mutex
 mutexes
 mutexing
 mvcc
+MVCC's
 mygcc
 mytable
 namespace
@@ -253,13 +244,16 @@ nolocking
 nommap
 nop
 nosql
+NoSQL
 nosync
+NOTFOUND
 notgranted
 notyet
 nowait
 nul
 num
 numa
+NUMA
 objectsin
 ol
 oltp
@@ -287,6 +281,7 @@ qnx
 rdbms
 rdlock
 readlock
+README
 readonly
 realclean
 realloc
@@ -296,20 +291,27 @@ recnoN
 recnum
 reconf
 recs
+RedHat
+RepMgr
 rerequests
 ret
 rf
 ro
+Roelofs
 rpc
+Rrx
+runnable
 rwlock
-sHQ
-sHq
 scalable
 scanf
 schemas
 selectable
 seqname
 serializable
+Seward's
+sHq
+sHQ
+SiS
 spinlock
 spinlocks
 sql
@@ -344,6 +346,7 @@ transactionally
 tt
 txn
 txns
+TXT
 typedef
 uint
 ul
@@ -351,9 +354,12 @@ umask
 unallocated
 unencoded
 unescaped
+unicode
 uninstall
+unittest
 untyped
 uri
+URIs
 useconds
 usr
 utf
@@ -363,12 +369,19 @@ valuefmt
 vec
 versa
 vm
+Vv
 whitespace
 wiredtiger
+WiredTiger
+WiredTigerCheckpoint
+WiredTiger's
+WiredTigerTestCase
 workQ
 writelock
 writelocks
 wrlock
+wtperf
 xa
 yieldcpu
+Za
 zlib
diff --git a/src/docs/testing.dox b/src/docs/testing.dox
new file mode 100644
index 00000000000..271623fd7de
--- /dev/null
+++ b/src/docs/testing.dox
@@ -0,0 +1,142 @@
+/*!
+@page testing How WiredTiger is tested
+
+There are several parts to the testing story for Wired Tiger:
+
+* functionality coverage
+* capacity/limits testing
+* bug regression testing
+* multiple platform testing
+* multithreading testing
+* stress testing
+* performance testing
+
+This part of the test plan is for the first four items on the list above. That is, testing each Wired Tiger feature or configuration, and that those features and configurations also work in every combination. In addition some tests of capacity or limits, such that can be accomplished within our test suite. And finally, the ability to add test cases that trigger particular past bugs. All this will be tried on all platforms we are interested in. This testing will use the Python API along with a (currently thin) framework being built on top of Python's unittest facility. We call this our 'test suite'.
+
+Our goal for the test suite is to have an easy to run package of tests that gives good feature and code coverage, and has repeatable errors. It should be runnable (at least in some form) in a short amount of time -- we'll probably eventually have subsets that can be run in various lengths of time, for example, a set of tests under an hour to accommodate a 'smoke test'.
+
+To quickly discuss the other parts of testing mentioned above:
+
+@section testing_threads Multithreading
+
+Multithreaded testing is difficult. For the moment, there is a multithreading test (in test/thread) written in C that is the primary engine for testing this. There will be some simple Python tests for multithreading, though this will not be a primary emphasis. One important issue is that we expect the failures resulting from the python test base to be mostly repeatable. Adding a large multithreading component with uncontrolled interactions is in opposition to this goal. We will be periodically evaluating other ways to repeatably test multithreaded behavior.
+
+@section testing_stress Stress testing
+
+Similarly, stress testing is mostly handled by the test/format tool. Stress tests can take significant time to run, which goes against a goal of having a reasonable amount of time to run the entire suite (that is, less than several days!)
+
+@section testing_performance Performance testing
+
+We test performance using the wtperf tool, running a variety of configurations on every push, and tracking any performance regressions.
+
+@section testing_unit Basic functionality coverage
+
+There are several dimensions to the basic testing.
+
+@subsection testing_unit_access Access methods
+
+* row: (K/V pair)
+* var: (like 64 bit recno) with arbitrary length value
+* fix: (like 64 bit recno) but fixed (0-8 bit) value
+
+@subsection testing_unit_datastore Datastore operations
+
+* key based operations - create, update, delete
+* cursor based operations - forw/back iteration, search, search near
+
+@subsection testing_unit_indices Secondary indices
+create, update, delete from both primary and or secondary. multiple secondaries, cursors, etc.
+
+@subsection testing_unit_types Data variations
+
+* key/value length
+* key/value 'values': e.g. english, binary, sparse, unicode, etc.
+* data types for 'typed' access (key_format, value_format)
+
+The above areas are the 'basics' of the test suite.  We also 'mix in' other dimensions:
+
+@subsection testing_unit_config Disk storage variations
+
+* block_compressor off vs. bzip2
+* {internal,leaf}_node_{min,max}
+* column_{internal,leaf}_extend
+* huffman encoding for key,value,both,none
+* prefix_compression
+* cache_size
+* allocation_size variations
+* btree (there are no other options currently)
+* split_min, split_pct
+* internal_key_truncate
+* key_gap
+
+Other than changing configuration options, the tests will be the same, and we expect the results to be the same. These tests must be done completely - the full combinatorial subset of these options against our basic tests.
+
+Some notes:
+
+* via dump or other stats, we may be able to detect that the desired configuration options were indeed used.
+* Some combinations may in fact be illegal, some options may only apply to a particular access method, etc.
+
+@subsection testing_unit_more More functionality tests
+
+Many of these may rightly belong up in the basic set of tests which get mixed in with the invisible tests. Some may only need to be performed once (do we really need to test variations in file names, or statistics cursors against every block size?) Some qualify as dimensions that we combine with all the other tests.
+
+@subsection testing_unit_columns Columns
+
+* column lists
+* column groups (colgroups)
+
+@subsection testing_unit_misc Miscellany
+
+* opening with exclusive=true
+* variations in file names (current, relative, abs directory)
+* cursor types: bulk, dump, statistics, printable, raw
+
+@subsection testing_unit_callbacks Callbacks
+
+collator, compressor, ..., events
+
+@subsection testing_unit_transactions Transaction and isolation
+
+Cursor isolation levels: snapshot, read-committed, read-uncommitted.
+
+@subsection testing_unit_util Command-line utility
+
+* wt dump
+* wt load
+* wt salvage
+* wt verify
+
+@subsection testing_unit_limits Capacity / Limits testing.
+
+In any of the above functional tests, where appropriate, we should test:
+
+* multiple (connections, sessions, cursors, etc.)
+* capacity testing, e.g. testing 10000 simultaneously opened cursors, etc.
+* wherever there are limits, test them.
+* wherever there are no limits, run to some reasonable capacity.
+* capacity and multiples in combinations
+
+@subsection testing_unit_platforms Testing on multiple platforms.
+
+Once we have a test suite, it should be straightforward to try it on whatever platform we can get WT running on.
+
+@subsection testing_unit_regression Regression tests.
+
+Individual tests will be made that correspond to bug conditions we've encountered. Whether they belong in the basic category or one of the mixins will be determined on a case by case basis.
+
+@subsection testing_unit_strategies Testing strategies
+
+Currently, we have a test suite built with Python's unittest. It has allowed us to write a small number of individual tests, that we will expand to fill out many of the dimensions above.
+
+To test the various configuration options in Python, we'll be able to leverage:
+
+* inheritance
+* setting up suites in unittest
+
+To mixin configurations we modified WiredTigerTestCase to accept configuration strings, using this technique: [[http://eli.thegreenplace.net/2011/08/02/python-unit-testing-parametrized-test-cases/]]
+
+One challenge is how to test a big combinatorial problem. python unittest scenarios let us create a simple test of scenarios to test. We now have a 'multiply_scenarios' function that takes several scenario tests (e.g. 10 different page sizes vs. 10 different cache sizes vs. 15 key combinations vs. 2 (huffman on/off), etc.) and multiplies them out.
+
+The problem with this approach is that we end up with too many tests. So we've enhanced this to allow each entry in to have a 'P' variable. This is treated as a probability (0.0 <= P <= 1.0), and when we multiply scenarios, we produce a resultant P variable that is the product of each multiplicand. Each final scenario (one of the zillions) now has a final probability and based on a random number generator, we decide whether to prune it from the final list. For now the random generator is 'predictable', so we generate the same list on each run. Perhaps in the future we can run many more of the combinations on some test server. For now, it satisfies my desire for trying out combinations in a practical way, as well as codifying the parameters that we'd like tested.
+
+ */