1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
|
/*! @page wtperf Simulating workloads with wtperf
The WiredTiger distribution includes a tool that can be used to simulate
workloads in WiredTiger, in the directory \c bench/wtperf.
The \c wtperf utility generally has two phases, the populate phase which
creates a database and then populates an object in that database, and a
workload phase, that does some set of operations on the object.
For example, the following configuration uses a single thread to
populate a file object with 500,000 records in a 500MB cache. The
workload phase consists of 8 threads running for two minutes, all
reading from the file.
@code
conn_config="cache_size=500MB"
table_config="type=file"
icount=500000
run_time=120
populate_threads=1
threads=((count=8,reads=1))
@endcode
In most cases, where the workload is the only interesting phase, the
populate phase can be performed once and the workload phase run
repeatedly (for more information, see the wtperf \c create configuration
variable).
The \c conn_config configuration supports setting any WiredTiger
connection configuration value. This is commonly used to configure
statistics with regular reports, to obtain more information from the
run:
@code
conn_config="cache_size=20G,statistics=(fast,clear),statistics_log=(wait=600)"
report_interval=5
@endcode
Note quoting must be used when passing values to Wiredtiger
configuration, as opposed to configuring the \c wtperf utility itself.
The \c table_config configuration supports setting any WiredTiger object
creation configuration value, for example, the above test can be
converted to using an LSM store instead of a B+tree store, with
additional LSM configuration, by changing \c conn_config to:
@code
table_config="lsm=(chunk_size=5MB),type=lsm,os_cache_dirty_max=16MB"
@endcode
More complex workloads can be configured by creating more threads doing
inserts and updates as well as reads. For example, to configure two
inserting threads two threads doing a mixture of inserts, reads and
updates:
@code
threads=((count=2,inserts=1),(count=2,inserts=1,reads=1,updates=1))
@endcode
Example \c wtperf configuration files can be found in the
\c bench/wtperf/runners/ directory.
There are also a number of command line arguments that can be passed
to \c wtperf:
@par <code>-C config</code>
Specify configuration strings for the ::wiredtiger_open function.
This argument is additive to the \c conn_config parameter in the
configuration file.
@par <code>-h directory</code>
Specify a database home directory. The default is \c ./WT_TEST.
@par <code>-m monitor_directory</code>
Specify a directory for all monitoring related files.
The default is the database home directory.
@par <code>-O config_file</code>
Specify the configuration file to run.
@par <code>-o config</code>
Specify configuration strings for the \c wtperf program.
This argument will override settings in the configuration file.
@par <code>-T config</code>
Specify configuration strings for the WT_SESSION::create function.
This argument is additive to the \c table_config parameter in the
configuration file.
@section monitor Monitoring wtperf performance
In addition to testing a particular configuration, \c wtperf can monitor
performance and operation latency times. Monitoring can be enabled
using the \c sample_interval configuration setting. For example to
record information every 10 seconds, add the following to the \c wtperf
configuration file:
@code
sample_interval=10
@endcode
Enabling monitoring causes \c wtperf to create a file \c monitor in the
database home directory, or another directory as specified in the \c -m
option to \c wtperf.
A visualization tool to see the performance over the course of the
\c wtperf run can be found in \c tools/wtperf_stats.py.
The following example shows how to run the \c large-lsm.wtperf
configuration to a subdirectory of \c build_posix called \c WTPERF_RUN with
monitoring enabled, and then generate a graph. This example assumes you
have all the necessary tools already installed to build WiredTiger,
Python and python-nvd3.
@code
cd $WIREDTIGER_SOURCE
./configure && make
mkdir WTPERF_RUN && ./bench/wtperf/wtperf -h WTPERF_RUN -O ./bench/wtperf/runners/large-lsm.wtperf
cd WTPERF_RUN; python ../tools/wtperf_stats.py monitor
@endcode
The python tool creates a file named \c wtperf_stats.html in the current
working directory. You can open the generated HTML document in your browser
and see the generated statistics. The keys in the graph are clickable,
including the graph included on this page. Double clicking on one
of the keys will show only that item. Single clicking will
enable or disable a particular item.
The script as a few optional arguments. For example, <code> -o file </code>
or <code> --output file </code> to redirect the HTML output to another filename,
and \c --abstime to display time on the X-axis in absolute time.
For example, the file will have content like this:
\htmlinclude wtperf-sample.html
@section config Wtperf configuration options
The following is a list of the currently available \c wtperf
configuration options:
\if START_AUTO_GENERATED_WTPERF_CONFIGURATION
DO NOT EDIT: THIS PART OF THE FILE IS GENERATED BY dist/s_docs.
\endif
@par async_threads (unsigned int, default=0)
number of async worker threads
@par checkpoint_interval (unsigned int, default=120)
checkpoint every interval seconds during the workload phase.
@par checkpoint_stress_rate (unsigned int, default=0)
checkpoint every rate operations during the populate phase in the
populate thread(s), 0 to disable
@par checkpoint_threads (unsigned int, default=0)
number of checkpoint threads
@par conn_config (string, default=create)
connection configuration string
@par compact (boolean, default=false)
post-populate compact for LSM merging activity
@par compression (string, default=none)
compression extension. Allowed configuration values are: 'none',
'bzip', 'snappy', 'zlib'
@par create (boolean, default=true)
do population phase; false to use existing database
@par database_count (unsigned int, default=1)
number of WiredTiger databases to use. Each database will execute the
workload using a separate home directory and complete set of worker
threads
@par icount (unsigned int, default=5000)
number of records to initially populate. If multiple tables are
configured, each table has this many items inserted.
@par insert_rmw (boolean, default=false)
execute a read prior to each insert in workload phase
@par key_sz (unsigned int, default=20)
key size
@par min_throughput (unsigned int, default=0)
abort if any throughput measured is less than this amount. Requires
sample_interval to be configured
@par max_latency (unsigned int, default=0)
abort if any latency measured exceeds this number of
milliseconds.Requires sample_interval to be configured
@par pareto (boolean, default=false)
use pareto 80/20 distribution for random numbers
@par populate_ops_per_txn (unsigned int, default=0)
number of operations to group into each transaction in the populate
phase, zero for auto-commit
@par populate_threads (unsigned int, default=1)
number of populate threads, 1 for bulk load
@par random_range (unsigned int, default=0)
if non zero choose a value from within this range as the key for
insert operations
@par random_value (boolean, default=false)
generate random content for the value
@par report_interval (unsigned int, default=2)
output throughput information every interval seconds, 0 to disable
@par run_ops (unsigned int, default=0)
total read, insert and update workload operations
@par run_time (unsigned int, default=0)
total workload seconds
@par sample_interval (unsigned int, default=0)
performance logging every interval seconds, 0 to disable
@par sample_rate (unsigned int, default=50)
how often the latency of operations is measured. One for every
operation,two for every second operation, three for every third
operation etc.
@par sess_config (string, default=)
session configuration string
@par table_config (string, default=key_format=S,value_format=S,type=lsm,exclusive=true,allocation_size=4kb,internal_page_max=64kb,leaf_page_max=4kb,split_pct=100)
table configuration string
@par table_count (unsigned int, default=1)
number of tables to run operations over. Keys are divided evenly over
the tables. Default 1, maximum 99.
@par threads (string, default=)
workload configuration: each 'count' entry is the total number of
threads, and the 'insert', 'read' and 'update' entries are the ratios
of insert, read and update operations done by each worker thread; If a
throttle value is provided each thread will do a maximum of that
number of operations per second; multiple workload configurations may
be specified; for example, a more complex threads configuration might
be 'threads=((count=2,reads=1)(count=8,reads=1,inserts=2,updates=1))'
which would create 2 threads doing nothing but reads and 8 threads
each doing 50% inserts and 25% reads and updates. Allowed
configuration values are 'count', 'throttle', 'reads', 'inserts',
'updates'
@par transaction_config (string, default=)
transaction configuration string, relevant when populate_opts_per_txn
is nonzero
@par table_name (string, default=test)
table name
@par value_sz (unsigned int, default=100)
value size
@par verbose (unsigned int, default=1)
verbosity
@par warmup (unsigned int, default=0)
How long to run the workload phase before starting measurements
\if STOP_AUTO_GENERATED_WTPERF_CONFIGURATION
DO NOT EDIT: THIS PART OF THE FILE IS GENERATED BY dist/s_docs.
\endif
*/
|