1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
|
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=windows-1252">
<meta name="GENERATOR" content="Microsoft FrontPage 5.0">
<meta name="ProgId" content="FrontPage.Editor.Document">
<title>Boost Test Policies and Protocols</title>
</head>
<body bgcolor="#FFFFFF" text="#000000">
<table border="1" bgcolor="#007F7F" cellpadding="2">
<tr>
<td bgcolor="#FFFFFF"><img src="../boost.png" alt="boost.png (6897 bytes)" width="277" height="86"></td>
<td><a href="../index.htm"><font face="Arial" color="#FFFFFF"><big>Home</big></font></a></td>
<td><a href="../libs/libraries.htm"><font face="Arial" color="#FFFFFF"><big>Libraries</big></font></a></td>
<td><a href="../people/people.htm"><font face="Arial" color="#FFFFFF"><big>People</big></font></a></td>
<td><a href="faq.htm"><font face="Arial" color="#FFFFFF"><big>FAQ</big></font></a></td>
<td><a href="index.htm"><font face="Arial" color="#FFFFFF"><big>More</big></font></a></td>
</tr>
</table>
<h1>Boost Test Policies and Protocols</h1>
<p>The Boost libraries are intended to be both reliable and portable.
Every experienced programmer knows that means each library must be tested against a suitable number of test cases, on a wide range of platforms,
and then tested again (regression tested) every time a change is made and before
every release. </p>
<p>"Quality assurance based on a wide range of targeted tests" as one
of the key answers to C.A.R
Hoare's question
"How did software get so reliable without proof."</p>
<h2>Regression test</h2>
<p>Boost uses an automatic <a href="regression.html"> regression test suite</a> which generates HTML
<a href="../status/compiler_status.html">compiler
status tables</a>.</p>
<h2>Test Policy</h2>
<h3>Required</h3>
<ul>
<li>Every Boost library should supply one or more suitable test programs to be
exercised by the Boost <a href="regression.html"> regression test suite</a>. In addition to
the usual compile-link-run tests expecting successful completion,
compile-only or compile-and-link-only tests may be performed, and success
for the test may be defined as failure of the steps.</li>
<li>Test program execution must report errors by returning a non-zero value. They
may also write to stdout or stderr, but that output should be relatively
brief. Regardless of other output, a non-zero return value is the only
way the regression test framework will recognize an error has
occurred. Note that test programs to be included in the status tables must
compile, link, and run quickly since the tests are executed many, many,
times.</li>
<li>Libraries with time consuming tests should be divided into a
fast-execution basic test program for the status tables, and a separate
full-coverage test program for exhaustive test cases. The basic test
should concentrate on compilation issues so that the status tables
accurately reflect the library's likelihood of correct compilation on a
platform.</li>
<li>If for any reason the usual test policies do not apply to a particular
library, an alternate test strategy must be implemented.</li>
<li>A <a href="regression.html#Adding_new_test">Jamfile</a> to drive the
regression tests for the library. </li>
</ul>
<h3>Optional (but highly recommended)</h3>
<p>The <a href="../libs/test/index.html">Boost Test Library</a> provides many
useful components which ease the construction of test programs.</p>
<ul>
<li>Use the library's
<a href="../libs/test/doc/components/test_tools/index.html">Test Tools</a> for the construction of simple test
programs that do not need much structure.</li>
<li>Use the library's
<a href="../libs/test/doc/components/utf/index.html">Unit Test
Framework</a> for the construction of more complex test programs that need to
be structured into individual tests
and test suites.</li>
</ul>
<h2>Suggested Protocol for Fixing Bugs or Adding Features.</h2>
<ul>
<li>First, add regression test cases that detects the bug or tests the
feature. Sometimes adding one case suggests similar untested cases, and they
are added too.</li>
<li>Second, for bugs, run the regression test and verify that the bug is now
detected.</li>
<li>Third, then, and only then, fix the bug or add the feature.</li>
<li>Finally, rerun the full regression tests - sometimes the change breaks
something else.</li>
</ul>
<h2>History</h2>
<p><a href="regression.html#History">See Regression Test History</a>.</p>
<h2>Acknowledgements</h2>
<p>Written by Beman Dawes. Jens Maurer, Paul Moore, Gary Powell and Jeremy Siek contributed helpful suggestions.</p>
<hr>
<p>Revised <!--webbot bot="Timestamp" S-Type="EDITED" S-Format="%d %B, %Y" startspan -->08 January, 2004<!--webbot bot="Timestamp" endspan i-checksum="38708" -->
</p>
<p>© Copyright Beman Dawes 2001</p>
<p> Use, modification, and distribution are subject to the Boost Software
License, Version 1.0. (See accompanying file <a href="../LICENSE_1_0.txt">
LICENSE_1_0.txt</a> or copy at <a href="http://www.boost.org/LICENSE_1_0.txt">
www.boost.org/LICENSE_1_0.txt</a>)</p>
</body>
</html>
|
