1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
|
/*!
@page testing WiredTiger testing
WiredTiger uses a combination of several different tools and test
programs for testing WiredTiger.
WiredTiger testing is done automatically using the <a
href="http://jenkins-ci.org/">Jenkins</a> and <a
href="https://github.com/evergreen-ci/evergreen/wiki">Evergreen</a> continuous integration testing
frameworks. This allows us to be confident we don't introduce problems in a non-standard
configuration, or performance regressions.
@section test_suite Unit Test Suite
@subsection test_python Python
WiredTiger's primary functionality and regression testing is done in a
Python unit test suite (found in the source tree under \c test/suite).
The WiredTiger Python test suite includes approximately 10,000 separate
tests which are run on all platforms supported by WiredTiger. Each test
is intended to test a single operation in a reproducible way, making it
easy to diagnose errors. The test suite runs multiple test cases in
parallel, allowing it to be run in a relatively short period of time.
The WiredTiger unit test suite includes tests that cover:
- WiredTiger functionality (for example, cursors, transactions and recovery),
- Combinations of WiredTiger's configuration settings and APIs,
- Bug regression tests.
The WiredTiger Python test suite is built using the WiredTiger Python
API and the Python unittest functionality (the test suite requires at
least Python version 2.6).
The WiredTiger test suite automatically runs as part of every commit
into the WiredTiger GitHub source tree.
@subsection test_csuite CSuite
@subsection test_cppsuite CPPSuite
@subsection test_format Format
@subsection test_perf WT Perf
@subsection test_workgen Workgen
@section testing_compatibility Compatibility Testing
@section testing_correctness Correctness Testing
@section testing_performance Performance Testing
Performance testing is primarily done using the \c bench/wtperf utility.
A variety of database configurations are run based on the scripts in \c
bench/wtperf/runners.
WiredTiger performance tests are automatically run as part of each
commit into the \c develop branch of the WiredTiger GitHub source tree
and compared against previous runs to detect performance regressions.
Performance testing is also done using the \c bench/workgen utility.
@section testing_stress Stress Testing
Stress testing is primarily done using the \c test/format utility. This
test program randomly configures a database and then runs some number
of randomly selected operations, using some number of randomly selected
threads, on that database. WiredTiger stress testing is run continuously
on the WiredTiger GitHub \c develop branch.
@section testing_threads Concurrency testing
Concurrency testing is primarily done using the \c test/format utility.
Additionally, the \c test/thread and \c test/fops test utilities test
specific heavily threaded operations. WiredTiger concurrency testing
is run continuously on the WiredTiger GitHub \c develop branch.
@section testing_runtime Runtime correctness checking
ASAN, MSAN, UBSAN, Valgrind
@section testing_static Static analysis
WiredTiger static analysis is done using three tools:
- The <a href="https://www.coverity.com/">Coverity,
Inc.</a>software analysis tool; current results and historic defect
reports are available at <a
href="https://scan.coverity.com/projects/1018">Coverity's WiredTiger
page</a>.
- The Gimpel Software <a
href="http://www.gimpel.com/html/flex.htm">FlexeLint</a> implementation
of the UNIX lint tool.
- The University of Illinois LLVM project's <a
href="http://clang-analyzer.llvm.org/">Clang Static Analyzer</a>.
@section testing_coverage Coverage
@subsection coverage_tests Tests
The <a href="https://github.com/wiredtiger/wiredtiger/blob/develop/test/test_coverage.md"> test
coverage documentation</a> is generated in each branch, the link provided is to the current
\c develop version of the document.
@subsection coverage_tags Tagging scheme
Each test file shall contain a tag using the following scheme:
@code{.console}
[TEST_TAGS]
<COMPONENT>:<TESTING_TYPE>:<TESTING_AREA>
[END_TAGS]
@endcode
One test file can have multiple tags:
@code{.console}
[TEST_TAGS]
backup:correctness:full_backup
checkpoints:correctness:checkpoint_data
caching_eviction:correctness:written_data
[END_TAGS]
`@endcode
If a test file shall be ignored, the following tag can be used:
@code{.console}
[TEST_TAGS]
ignored_file
[END_TAGS]
@endcode
@subsection coverage_code Code
WiredTiger code coverage is measured by gcov.
@subsection coverage_cyclomatic Cyclomatic Complexity
*/
|
