1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
|
ctest_test
----------
Perform the :ref:`CTest Test Step` as a :ref:`Dashboard Client`.
::
ctest_test([BUILD <build-dir>] [APPEND]
[START <start-number>]
[END <end-number>]
[STRIDE <stride-number>]
[EXCLUDE <exclude-regex>]
[INCLUDE <include-regex>]
[EXCLUDE_LABEL <label-exclude-regex>]
[INCLUDE_LABEL <label-include-regex>]
[EXCLUDE_FIXTURE <regex>]
[EXCLUDE_FIXTURE_SETUP <regex>]
[EXCLUDE_FIXTURE_CLEANUP <regex>]
[PARALLEL_LEVEL <level>]
[RESOURCE_SPEC_FILE <file>]
[TEST_LOAD <threshold>]
[SCHEDULE_RANDOM <ON|OFF>]
[STOP_ON_FAILURE]
[STOP_TIME <time-of-day>]
[RETURN_VALUE <result-var>]
[CAPTURE_CMAKE_ERROR <result-var>]
[REPEAT <mode>:<n>]
[OUTPUT_JUNIT <file>]
[QUIET]
)
..
_note: If updating the argument list here, please also update the argument
list documentation for :command:`ctest_memcheck` as well.
Run tests in the project build tree and store results in
``Test.xml`` for submission with the :command:`ctest_submit` command.
The options are:
``BUILD <build-dir>``
Specify the top-level build directory. If not given, the
:variable:`CTEST_BINARY_DIRECTORY` variable is used.
``APPEND``
Mark ``Test.xml`` for append to results previously submitted to a
dashboard server since the last :command:`ctest_start` call.
Append semantics are defined by the dashboard server in use.
This does *not* cause results to be appended to a ``.xml`` file
produced by a previous call to this command.
``START <start-number>``
Specify the beginning of a range of test numbers.
``END <end-number>``
Specify the end of a range of test numbers.
``STRIDE <stride-number>``
Specify the stride by which to step across a range of test numbers.
``EXCLUDE <exclude-regex>``
Specify a regular expression matching test names to exclude.
``INCLUDE <include-regex>``
Specify a regular expression matching test names to include.
Tests not matching this expression are excluded.
``EXCLUDE_LABEL <label-exclude-regex>``
Specify a regular expression matching test labels to exclude.
``INCLUDE_LABEL <label-include-regex>``
Specify a regular expression matching test labels to include.
Tests not matching this expression are excluded.
``EXCLUDE_FIXTURE <regex>``
.. versionadded:: 3.7
If a test in the set of tests to be executed requires a particular fixture,
that fixture's setup and cleanup tests would normally be added to the test
set automatically. This option prevents adding setup or cleanup tests for
fixtures matching the ``<regex>``. Note that all other fixture behavior is
retained, including test dependencies and skipping tests that have fixture
setup tests that fail.
``EXCLUDE_FIXTURE_SETUP <regex>``
.. versionadded:: 3.7
Same as ``EXCLUDE_FIXTURE`` except only matching setup tests are excluded.
``EXCLUDE_FIXTURE_CLEANUP <regex>``
.. versionadded:: 3.7
Same as ``EXCLUDE_FIXTURE`` except only matching cleanup tests are excluded.
``PARALLEL_LEVEL <level>``
Specify a positive number representing the number of tests to
be run in parallel.
``RESOURCE_SPEC_FILE <file>``
.. versionadded:: 3.16
Specify a
:ref:`resource specification file <ctest-resource-specification-file>`. See
:ref:`ctest-resource-allocation` for more information.
``TEST_LOAD <threshold>``
.. versionadded:: 3.4
While running tests in parallel, try not to start tests when they
may cause the CPU load to pass above a given threshold. If not
specified the :variable:`CTEST_TEST_LOAD` variable will be checked,
and then the :option:`--test-load <ctest --test-load>` command-line
argument to :manual:`ctest(1)`. See also the ``TestLoad`` setting
in the :ref:`CTest Test Step`.
``REPEAT <mode>:<n>``
.. versionadded:: 3.17
Run tests repeatedly based on the given ``<mode>`` up to ``<n>`` times.
The modes are:
``UNTIL_FAIL``
Require each test to run ``<n>`` times without failing in order to pass.
This is useful in finding sporadic failures in test cases.
``UNTIL_PASS``
Allow each test to run up to ``<n>`` times in order to pass.
Repeats tests if they fail for any reason.
This is useful in tolerating sporadic failures in test cases.
``AFTER_TIMEOUT``
Allow each test to run up to ``<n>`` times in order to pass.
Repeats tests only if they timeout.
This is useful in tolerating sporadic timeouts in test cases
on busy machines.
``SCHEDULE_RANDOM <ON|OFF>``
Launch tests in a random order. This may be useful for detecting
implicit test dependencies.
``STOP_ON_FAILURE``
.. versionadded:: 3.18
Stop the execution of the tests once one has failed.
``STOP_TIME <time-of-day>``
Specify a time of day at which the tests should all stop running.
``RETURN_VALUE <result-var>``
Store in the ``<result-var>`` variable ``0`` if all tests passed.
Store non-zero if anything went wrong.
``CAPTURE_CMAKE_ERROR <result-var>``
.. versionadded:: 3.7
Store in the ``<result-var>`` variable -1 if there are any errors running
the command and prevent ctest from returning non-zero if an error occurs.
``OUTPUT_JUNIT <file>``
.. versionadded:: 3.21
Write test results to ``<file>`` in JUnit XML format. If ``<file>`` is a
relative path, it will be placed in the build directory. If ``<file>``
already exists, it will be overwritten. Note that the resulting JUnit XML
file is **not** uploaded to CDash because it would be redundant with
CTest's ``Test.xml`` file.
``QUIET``
.. versionadded:: 3.3
Suppress any CTest-specific non-error messages that would have otherwise
been printed to the console. Output from the underlying test command is not
affected. Summary info detailing the percentage of passing tests is also
unaffected by the ``QUIET`` option.
See also the :variable:`CTEST_CUSTOM_MAXIMUM_PASSED_TEST_OUTPUT_SIZE`,
:variable:`CTEST_CUSTOM_MAXIMUM_FAILED_TEST_OUTPUT_SIZE` and
:variable:`CTEST_CUSTOM_TEST_OUTPUT_TRUNCATION` variables.
.. _`Additional Test Measurements`:
Additional Test Measurements
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
CTest can parse the output of your tests for extra measurements to report
to CDash.
When run as a :ref:`Dashboard Client`, CTest will include these custom
measurements in the ``Test.xml`` file that gets uploaded to CDash.
Check the `CDash test measurement documentation
<https://github.com/Kitware/CDash/blob/master/docs/test_measurements.md>`_
for more information on the types of test measurements that CDash recognizes.
.. versionadded: 3.22
CTest can parse custom measurements from tags named
``<CTestMeasurement>`` or ``<CTestMeasurementFile>``. The older names
``<DartMeasurement>`` and ``<DartMeasurementFile>`` are still supported.
The following example demonstrates how to output a variety of custom test
measurements.
.. code-block:: c++
std::cout <<
"<CTestMeasurement type=\"numeric/double\" name=\"score\">28.3</CTestMeasurement>"
<< std::endl;
std::cout <<
"<CTestMeasurement type=\"text/string\" name=\"color\">red</CTestMeasurement>"
<< std::endl;
std::cout <<
"<CTestMeasurement type=\"text/link\" name=\"CMake URL\">https://cmake.org</CTestMeasurement>"
<< std::endl;
std::cout <<
"<CTestMeasurement type=\"text/preformatted\" name=\"Console Output\">" <<
"line 1.\n" <<
" \033[31;1m line 2. Bold red, and indented!\033[0;0ml\n" <<
"line 3. Not bold or indented...\n" <<
"</CTestMeasurement>" << std::endl;
Image Measurements
""""""""""""""""""
The following example demonstrates how to upload test images to CDash.
.. code-block:: c++
std::cout <<
"<CTestMeasurementFile type=\"image/jpg\" name=\"TestImage\">" <<
"/dir/to/test_img.jpg</CTestMeasurementFile>" << std::endl;
std::cout <<
"<CTestMeasurementFile type=\"image/gif\" name=\"ValidImage\">" <<
"/dir/to/valid_img.gif</CTestMeasurementFile>" << std::endl;
std::cout <<
"<CTestMeasurementFile type=\"image/png\" name=\"AlgoResult\"> <<
"/dir/to/img.png</CTestMeasurementFile>"
<< std::endl;
Images will be displayed together in an interactive comparison mode on CDash
if they are provided with two or more of the following names.
* ``TestImage``
* ``ValidImage``
* ``BaselineImage``
* ``DifferenceImage2``
By convention, ``TestImage`` is the image generated by your test, and
``ValidImage`` (or ``BaselineImage``) is basis of comparison used to determine
if the test passed or failed.
If another image name is used it will be displayed by CDash as a static image
separate from the interactive comparison UI.
Attached Files
""""""""""""""
.. versionadded:: 3.21
The following example demonstrates how to upload non-image files to CDash.
.. code-block:: c++
std::cout <<
"<CTestMeasurementFile type=\"file\" name=\"TestInputData1\">" <<
"/dir/to/data1.csv</CTestMeasurementFile>\n" <<
"<CTestMeasurementFile type=\"file\" name=\"TestInputData2\">" <<
"/dir/to/data2.csv</CTestMeasurementFile>" << std::endl;
If the name of the file to upload is known at configure time, you can use the
:prop_test:`ATTACHED_FILES` or :prop_test:`ATTACHED_FILES_ON_FAIL` test
properties instead.
Custom Details
""""""""""""""
.. versionadded:: 3.21
The following example demonstrates how to specify a custom value for the
``Test Details`` field displayed on CDash.
.. code-block:: c++
std::cout <<
"<CTestDetails>My Custom Details Value</CTestDetails>" << std::endl;
.. _`Additional Labels`:
Additional Labels
"""""""""""""""""
.. versionadded:: 3.22
The following example demonstrates how to add additional labels to a test
at runtime.
.. code-block:: c++
std::cout <<
"<CTestLabel>Custom Label 1</CTestLabel>\n" <<
"<CTestLabel>Custom Label 2</CTestLabel>" << std::endl;
Use the :prop_test:`LABELS` test property instead for labels that can be
determined at configure time.
|
