1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
|
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<title>Boost Internal Regression Test Suite</title>
</head>
<body bgcolor="#FFFFFF" text="#000000">
<table border="1" bgcolor="#007F7F" cellpadding="2">
<tr>
<td bgcolor="#FFFFFF"><img src="../c++boost.gif" alt="c++boost.gif (8819 bytes)" width="277" height="86"></td>
<td><a href="../index.htm"><font face="Arial,Helvetica" color="#FFFFFF"><big>Home</big></font></a></td>
<td><a href="../libs/libraries.htm"><font face="Arial,Helvetica" color="#FFFFFF"><big>Libraries</big></font></a></td>
<td><a href="../people/people.htm"><font face="Arial,Helvetica" color="#FFFFFF"><big>People</big></font></a></td>
<td><a href="../more/faq.htm"><font face="Arial,Helvetica" color="#FFFFFF"><big>FAQ</big></font></a></td>
<td><a href="../more/index.htm"><font face="Arial,Helvetica" color="#FFFFFF"><big>More</big></font></a></td>
</tr>
</table>
<h1>Boost Internal Regression Test Suite</h1>
Boost's internal regression test suite produces the
<a href="../status/compiler_status.html">compiler status tables</a>.
<p>Although not ordinarily run by Boost library users, it is documented here for
the benefit of Boost developers, and for Boost users porting to a new platform.</p>
<h2>Requirements</h2>
The test suite has been designed to meet to the following requirements.
<ul>
<li>Pure ISO C++, no tools required except a C++ compiler.
<li>Support for tests which expect an error
<li>Flexible configuration
<li>HTML output
</ul>
These requirements rule out any script-based approach such as dejagnu
(requires Tcl and expect) or even shell scripts.
<h2>Running Regression Tests</h2>
The implementation is provided in a single source file named
<a href="../status/regression.cpp">regression.cpp</a>.
<p>
You should be able to compile and link this file using whatever C++
compiler is at your disposition. However, you may need to configure
both the compiler and the standard library to use "strict" ISO
compliance mode. Also, you need to extend the search path for include
files with the main boost directory so that the header file
<a href="../boost/config.hpp">boost/config.hpp</a>
can be found. This header file is required to work around compiler
deficiencies.
<p>
You can then start the resulting executable to run regression tests.
By default, the regression test program reads the file "compiler.cfg"
in the current directory to determine the list of compilers and their
invocation syntax. After that, it reads the file "regression.cfg" in
the current directory to determine the regression tests to run. The
results of the regression tests are written in an HTML formatted text
file. This file is by default named "cs-OS.html" in the current
directory, where "OS" is a placeholder for the name of the operating
system (e.g., "linux" or "win32").
<p>
To generate the
<a href="../status/compiler_status.html">compiler status tables</a>,
boost uses the files
<a href="../status/compiler.cfg">status/compiler.cfg</a>
and
<a href="../status/regression.cfg">status/regression.cfg</a>.
<p>
The regression test program accepts some command-line options to alter
its behavior.
<p>
<table border="1">
<tr>
<td>-h <em>or</em> --help</td>
<td>prints a help message
</tr>
<tr>
<td>--config <em>file</em></td>
<td>Use <em>file</em> instead of "compiler.cfg" as the compiler
configuration file. This allows for private compiler setups.</td>
</tr>
<tr>
<td>--tests <em>file</em></td>
<td>Use <em>file</em> instead of "regression.cfg" as the tests
configuration file. This allows individual libraries to specify
additional tests not to be published in the main
<a href="../status/compiler_status.html">compiler status tables</a>.
</td>
</tr>
<tr>
<td>--boost <em>path</em></td>
<td>Use <em>path</em> as the filesystem path to the main boost
directory. The default is "..", i.e. the parent directory.</td>
</tr>
<tr>
<td>--output <em>file</em><br>-o <em>file</em></td>
<td>Write the HTML output to <em>file</em> instead of the default
"cs-OS.html".</td>
</tr>
<tr>
<td>--compiler <em>name</em></td>
<td>Run the tests only with compiler <em>name</em>. The <em>name</em>
must be defined in the second line of an applicable compiler
configuration (see below). The default is to run the tests with all
compilers suitable for the platform.</td>
</tr>
<tr>
<td>--diff</td>
<td>Read the HTML output file before writing it. In the HTML output,
highlight differences in test outcomes compared to the previous
run.</td>
</tr>
<tr>
<td><em>test</em></td>
<td>Run only the named test. The syntax is the same as in the
configuration file (see below).</td>
</tr>
</table>
<p>
When running only a selected test, you must also provide an alternate
HTML output filename with "--output" so that the full test output is
not accidentally overwritten.
<p>
You should redirect the output (<code>std::cout</code>) and error
(<code>std::cerr</code>) channels to a suitable log file for later
inspection.
<h2>Configuration Files</h2>
In both configuration files, single-line comments starting with "//"
at the leftmost column are ignored.
<h3>Compiler Configuration</h3>
The compiler configuration file can contain descriptions for an
arbitrary number of compilers. Each compiler is configured by a block
of six consecutive text lines.
<ol>
<li>Name of the operating system for which the entry is applicable
(e.g., "linux" or "win32").
<li>Name of the compiler; should be unique within one operating
system. The name of the compiler should not contain the version
number, because it is expected that regression tests are always run
with the most recent compiler version available.
<li>Name and version number of the compiler. This is printed on
<code>std::cout</code> prior to running a test with that compiler.
<li>Command-line invocation of the compiler to compile a single source
file to an object file.
<li>Command-line invocation of the compiler to compile a single source
file to an executable.
<li>Identification of the compiler for inclusion in the HTML output;
may contain HTML tags such as <br>.
</ol>
The two command-lines are subject to the following substitutions:
<ul>
<li>Any string starting with a dollar symbol "$", then containing an
arbitrary number of underscores, digits, or uppercase letters, is
replaced by the value of the associated environment variable.
<li>%include is replaced by the path given by the "--boost" command
line option (".." by default). This should be used to extend the
search path of your compiler so that the boost header files are found.
<li>%source is the name of the source file to be compiled, qualified
with the path given by the "--boost" command-line option (".." by
default).
</ul>
<h3>Test Configuration</h3>
The test configuration file can contain descriptions for an arbitrary
number of tests. Each test is described by a single line. The first
word (up to the first space) is the type of the test, the next word
gives the filename of the test relative the directory to be given by the
"--boost" command-line option. Optionally, additional words are
passed on as command-line arguments when the test is executed (only
for types "run" and "run-fail"). In these arguments, "%boost" is
replaced by the path given by the "--boost" command-line option (".."
by default).
<p>
The following test types are available:
<ul>
<li>compile: The given test file should compile successfully.
Otherwise, the test fails.
<li>compile-fail: The given test file should not compile successfully,
but instead the compiler should give an error message. If the test
does compile successfully, the test fails.
<li>link: The given test file should compile and link successfully.
Otherwise, the test fails. In particular, a <code>main</code>
function must be present in the test file.
<li>link-fail: The given test file should not compile and link
successfully. It is not specified whether the compile or the link
should not succeed. If the test does compile and link successfully,
the test fails.
<li>run: The given test file should compile and link successfully.
After that, the resulting executable is invoked and should return a
zero exit code. If any of these steps fail, the test fails.
<li>run-fail: The given test file is compiled, linked, and, if an
executable was successfully generated, it is invoked. Either
compiling or linking should fail or the resulting executable should
return a non-zero exit code. If compiling and linking succeeds and
the resulting executable returns a zero exit code, the test fails.
</ul>
<h2>Adapting for a New Platform</h2>
In order to adapt the regression test suite for a new platform, a few
changes to the <a href="../status/regression.cpp">regression.cpp</a>
are required.
<ul>
<li>Add an appropriate check for your platform in
<code>get_host()</code> and return a unique string identifying the
platform (this string is used to filter the compile configuration
file).
<li>Verify that <code>get_system_configuration()</code> is
appropriately defined for your platform. For a Unix platform, it most
likely is. The function may return a verbose HTML string describing
the platform.
</ul>
You also need to configure the compilers available on your platform in
"compiler.cfg" as described above.
<p>You may need to add an entry for the compiler to <a href="../boost/config.hpp">boost/config.hpp</a>,
but only if entries for the compiler is not already present, and the compiler
doesn't fully conform to the ISO C++ Standard.</p>
<h2>History</h2>
<p>The regression.cpp test program was contributed
by Jens Maurer, generalizing and improving an earlier program by Beman Dawes.</p>
<hr>
2001-01-30<br>
<a href="../people/jens_maurer.htm">Jens Maurer</a>
</body>
</html>
|
