1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
|
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
"http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>
<meta name="generator" content=
"HTML Tidy for Linux/x86 (vers 1st March 2002), see www.w3.org">
<!--tidy options: -i -wrap 78 -->
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<title>A testing system for Boost.Build</title>
<style type="text/css">
hr { color: black }
p.revision { text-align: right; font-style: italic }
pre.code { margin-left: 2em }
pre.example { margin-left: 2em; border: solid black thin }
pre.output { margin-left: 2em }
img.banner { border: 0; float: left }
h1 { text-align: right }
br.clear { clear: left }
div.attention { color: red }
</style>
</head>
<body>
<p><a href="../../../../index.htm"><img class="banner" height="86" width=
"277" alt="C++ Boost" src="../../../../boost.png"></a></p>
<h1>A testing system for Boost.Build<br class="clear">
</h1>
<hr>
<dl class="page-index">
<dt><a href="#sec-intro">Introduction for users</a></dt>
<dd>
<dl class="page-index">
<dt><a href="#sec-command-line-options">Command line options</a></dt>
</dl>
</dd>
<dt><a href="#sec-developers">Introduction for developers</a></dt>
<dd>
<dl class="page-index">
<dt><a href="#sec-intro-changing">Changing the working
directory</a></dt>
<dt><a href="#sec-intro-examining">Examining the working directory and
changing it</a></dt>
<dt><a href="#sec-intro-results">Test result</a></dt>
</dl>
</dd>
<dt><a href="#sec-reference">Reference documentation</a></dt>
<dd>
<dl class="page-index">
<dt><a href="#method-__init__">Method __init__</a></dt>
<dt><a href="#method-set_tree">Method <tt>set_tree</tt></a></dt>
<dt><a href="#method-write">Method <tt>write</tt></a></dt>
<dt><a href="#method-copy">Method <tt>copy</tt></a></dt>
<dt><a href="#method-touch">Method <tt>touch</tt></a></dt>
<dt><a href="#method-run_build_system">Method
<tt>run_build_system</tt></a></dt>
<dt><a href="#method-read">Method <tt>read</tt></a></dt>
<dt><a href="#method-read_and_strip">Method
<tt>read_and_strip</tt></a></dt>
<dt><a href="#methods-expectations">Methods for declaring
expectations</a></dt>
<dt><a href="#methods-ignoring">Methods for ignoring
changes</a></dt>
<dt><a href="#methods-result">Methods for explicitly specifying
results</a></dt>
<dt><a href="#class-list">Helper class <tt>List</tt></a></dt>
</dl>
</dd>
</dl>
<hr>
<h2><a name="sec-intro">Introduction for users</a></h2>
<p>The testing system for Boost.Build is a small set of Python modules and
scripts for automatically testing user-obversable behaviour. It uses
components from testing systems of <a href="http://www.scons.org">Scons</a>
and <a href="http://subversion.tigris.org">Subversion</a>, together with
some additional functionality.</p>
<p>To run the tests you need to:</p>
<ol>
<li>Get the source tree of Boost.Build (located at <tt>tools/build</tt>
in Boost)</li>
<li>Have <a href="http://www.python.org">Python</a> installed. Version
2.1 is known to work.</li>
<li>Build Boost.Jam. See <a href=
"../engine/index.html">$boost_build_root/engine/index.html</a> for
instructions.</li>
<li>Configure at least one toolset. You can edit <tt>site-config.jam</tt>
or <tt>user-config.jam</tt> to add new toolsets. Or you can create file
<tt>test-config.jam</tt> in <tt>$boost_build_root/test</tt> directory. In
this case, <tt>site-config.jam</tt> and <tt>user-config.jam</tt> will be
ignored for testing.</li>
</ol>
<p>When all is set, you can run all the tests using the <tt>test_all.py</tt>
script or you can run a specific test by starting its Python script
directly.</p>
<p>Examples:</p>
<pre class="code">
python test_all.py
python generators_test.py
</pre>
<p>If everything is OK, you will see a list of passed tests. Otherwise, a
failure will be reported.</p>
<h3><a name="sec-command-line-options">Command line options</a></h3>
<p>Test scripts will use the toolset you configured to be the default or
you can specify a specific one on the command line:</p>
<pre class="code">
python test_all.py borland
python generators_test.py msvc-7.1
</pre>
<p>Other test script flags you can specify on the command line are:</p>
<ul>
<li><tt>--default-bjam</tt> -- By default the test system will use the
Boost Jam executable found built in its default development build
location. This option makes it use the default one available on your
system, i.e. the one found in the system path.</li>
<li><tt>--preserve</tt> -- In case of a failed test its working
directory will be copied to the "failed_test" directory under the
current directory.</li>
<li><tt>--verbose</tt> -- Makes the test system and the run build system
display additional output. Note though that this may cause tests that
check the build system output to fail.</li>
</ul>
<h2><a name="sec-developers">Introduction for developers</a></h2>
<p>It is suggested that every new functionality come together with tests,
and that bugfixes are accompanied by tests. There's no need to say that
tests are good, but two points are extremely important:</p>
<ul>
<li>For an interpreted language like Jam, without any static checks,
testing is simply the only sefeguard we can have.</li>
<li>Good tests allow us to change internal design much more safely, and we
have not gotten everything nailed down yet.</li>
</ul>
<p>Adding a new test is simple:</p>
<ol>
<li>Go to <tt>$boost_build_root/test/test_all.py</tt> and add new test
name to the list at the end of the file. Suppose the test name is "hello".
</li>
<li>Add a new python module, in this example "hello.py", to do the actual
testing.</li>
</ol>
<p>The module, in general will perform these basic actions:</p>
<ol>
<li>Set up the initial working directory state</li>
<li>
Run the build system and check the results:
<ol>
<li>generated output,</li>
<li>changes made to the working directory,</li>
<li>new content of the working directory.</li>
</ol>
</li>
<li>Add, remove or touch files or change their content and then repeat
the previous step until satisfied.</li>
<li>Clean up</li>
</ol>
<p>The "hello.py" module might contain:</p>
<pre class="example">
from BoostBuild import List
# Create a temporary working directory
t = BoostBuild.Tester()
# Create the needed files
t.write("jamroot.jam", "")
t.write("jamfile.jam", """
exe hello : hello.cpp ;
""")
t.write("hello.cpp", """
int main()
{
return 0;
}
""")
t.run_build_system()
# First, create a list of three pathnames.
file_list = List("bin/$toolset/debug/") * List("hello.exe hello.obj")
# Second, assert that those files were added as result of the last build system invocation.
t.expect_addition(file_list)
# Invoke the build system once again.
t.run_build_system("clean")
# Check if the files added previously were removed.
t.expect_removal(file_list)
# Remove temporary directories
t.cleanup()
</pre>
<p>The <tt>test</tt> directory contains a file "template.py" which can be
used as a start for your own tests.</p>
<p>Overview of the most important methods of class <tt>Tester</tt> follows.
</p>
<h3><a name="sec-intro-changing">Changing the working directory</a></h3>
<p>The class <tt>Tester</tt> creates a temporary directory in its
constructor and changes to that directory. It can be modified by calling
these methods:</p>
<ul>
<li><tt>set_tree</tt> -- sets the content of the working directory to be
equal to the content of the specified directory. This method is
preferrable when directory tree for testing is large.</li>
<li><tt>write</tt> -- sets the content of file in a working directory.
This is optimal if you want to create a directory tree with 3-4 small
files.</li>
<li><tt>touch</tt> -- changes the modification times of a file</li>
</ul>
<h3><a name="sec-intro-examining">Examining the working directory and
changing it</a></h3>
<p>The method <tt>read</tt>, inherited from the <tt>TestCmd</tt> class, can
be used to read any file in the working directory and check its content.
<tt>Tester</tt> adds another method for tracking changes. Whenever the build
system is run (using <a href="#method-run_build_system"><tt>run_build_system
</tt></a>), the working dir state before and after running is recorded. In
addition, difference between the two states -- i.e. lists of files that were
added, removed, modified or touched -- are stored in two member variables -
<tt>tree_difference</tt> and <tt>unexpected_difference</tt>.</p>
<p>After than, the test author may specify that some change is expected, for
example, by calling <tt>expect_addition("foo")</tt>. This call will check if
the file was indeed added, and if so, will remove its name from the list of
added files in <tt>unexpected_difference</tt>. Likewise, it is possible to
specify that some changes are not interesting, for example a call to
<tt>ignore("*.obj")</tt> will just remove every file with the ".obj"
extension from <tt>unexpected_difference</tt>.</p>
<p>When test has finished with expectations and ignoring, the member
<tt>unexpected_difference</tt> will contain the list of all changes not yet
accounted for. It is possible to assure that this list is empty by calling
the <tt>expect_nothing_more</tt> member function.</p>
<h3><a name="sec-intro-results">Test result</a></h3>
<p>Any of the <tt>expect*</tt> methods below will fail the test if the
expectation is not met. It is also possible to perform manually arbitrary
test and explicitly cause the test to either pass or fail. Ordinary
filesystem functions can be used to work with the directory tree. Methods
<tt>pass_test</tt> and <tt>fail_test</tt> are used to explicitly give the
test outcome.</p>
<p>Typically, after test termination, the working directory is erased. See
the <a href="#sec-command-line-options">"--preserve" command line option</a>
for information on how to preserve the working directory content for failed
tests for debugging purposes.</p>
<h2 id="sec-reference">Reference documentation</h2>
<p>The test system is composed of class <tt>Tester</tt>, derived form
<tt>TestCmd.TestCmd</tt>, and helper class <tt>List</tt>. <tt>Tester</tt>
and <tt>List</tt> methods are described below.</p>
<p>The documentation frequently refers to <tt>filename</tt>. In all cases,
files are specified in unix style: a sequence of components, separated by
"/". This is true on all platforms. In some contexts a list of files is
allowed. In those cases any object with a sequence interface is allowed.</p>
<h3><a name="method-__init__">Method <tt>__init__(self, arguments="",
executable="bjam", match=TestCmd.match_exact, boost_build_path=None,
translate_suffixes=True, pass_toolset=True, use_test_config=True,
ignore_toolset_requirements=True, workdir="", **keywords)</tt></a></h3>
<p><b>Optional arguments:</b></p>
<ul>
<li><tt>arguments</tt>
-- Arguments passed to the run executable.</li>
<li><tt>executable</tt>
-- Name of the executable to invoke.</li>
<li><tt>match</tt>
-- Function to use for compating actual and expected file contents.
</li>
<li><tt>boost_build_path</tt>
-- Boost build path to be passed to the run executable.</li>
<li><tt>translate_suffixes</tt>
-- Whether to update suffixes on the the file names passed from the
test script so they match those actually created by the current
toolset. For example, static library files are specified by using
the .lib suffix but when the 'gcc' toolset is used it actually
creates them using the .a suffix.</li>
<li><tt>pass_toolset</tt>
-- Whether the test system should pass the specified toolset to the
run executable.</li>
<li><tt>use_test_config</tt>
-- Whether the test system should tell the run executable to read in
the test_config.jam configuration file.</li>
<li><tt>ignore_toolset_requirements</tt>
-- Whether the test system should tell the run executable to ignore
toolset requirements.</li>
<li><tt>workdir</tt>
-- Indicates an absolute directory where the test will be run from.
</li>
</ul>
<p><b>Optional arguments inherited from the base class:</b></p>
<ul>
<li><tt>description</tt>
-- Test description string displayed in case of a failed test.</li>
<li><tt>subdir</tt>
-- List of subdirectories to automatically create under the working
directory. Each subdirectory needs to be specified separately
parent coming before its child.</li>
<li><tt>verbose</tt>
-- Flag that may be used to enable more verbose test system output.
Note that it does not also enable more verbose build system output
like the <a href="#sec-command-line-options">"--verbose" command
line option</a> does.</li>
</ul>
<p><b>Effects:</b></p>
<ol>
<li>Remembers the current working directory in member
<tt>original_workdir</tt>.</li>
<li>Determines the location of the executable (<code>bjam</code> by
default) and build system files, assuming that the current directory is
<tt>tools/build/test</tt>. Formulates jam invocation command, which
will include explicit setting for the <tt>BOOST_BUILD_PATH</tt> variable
and arguments passed to this methods, if any. This command will be used
by subsequent invocation of <a href="#method-run_build_system"><tt>
run_build_system</tt></a>. Finally, initializes the base class.</li>
<li>Changes the current working directory to the temporary working
directory created by the base constructor.</li>
<li>If you want to run a test in an existing directory, pass it as
<tt>workdir</tt>.</li>
<li> Most parameters passed to this constructor function may be overruled
for each specific test system run using <a href=
"#method-run_build_system"><tt>run_build_system</tt></a> parameters.
</ol>
<h3><a name="method-set_tree">Method <tt>set_tree(self,
tree_location)</tt></a></h3>
<p><b>Effects:</b></p>
<p>Replaces the content of the current working directory with the content
of directory at <tt>tree_location</tt>. If <tt>tree_location</tt> is not
absolute pathname, it will be treated as relative to
<tt>self.original_workdir</tt>. This methods also explicitly makes the
copied files writeable.</p>
<h3><a name="method-write">Method <tt>write(self, name,
content)</tt></a></h3>
<p><b>Effects:</b></p>
<p>Writes the specified content to the file given by <tt>name</tt> under
the temporary working directory. If the file already exists, it is
overwritten. Any required directories are automatically created.</p>
<h3><a name="method-copy">Method <tt>copy(self, src, dst)</tt></a></h3>
<p><b>Effects:</b></p>
<p>Equvivalent to <tt>self.write(self.read(src), dst)</tt>.</p>
<h3><a name="method-touch">Method <tt>touch(self, names)</tt></a></h3>
<p><b>Effects:</b></p>
<p>Sets the access and modification times for all files in <tt>names</tt> to
the current time. All the elements in <tt>names</tt> should be relative
paths.</p>
<h3><a name="method-run_build_system">Method <tt>run_build_system(self,
extra_args="", subdir="", stdout=None, stderr="", status=0, match=None,
pass_toolset=None, use_test_config=None, ignore_toolset_requirements=None,
expected_duration=None, **kw)</tt></a></h3>
<p><b>Effects:</b></p>
<ol>
<li>Stores the state of the working directory in
<tt>self.previous_tree</tt>.</li>
<li>Changes to <tt>subdir</tt>, if it is specified. It is relative to
the <tt>original_workdir</tt> or the workdir specified in
<tt>__init</tt>.</li>
<li>Invokes the <tt>bjam</tt> executable, passing <tt>extra_args</tt>
to it. The binary should be located under
<tt><test_invocation_dir>/../jam/src/bin.<platform></tt>.
This is to make sure tests use the version of jam build from CVS.</li>
<li>Compares the stdout, stderr and exit status of build system
invocation with values to appropriate parameters, if they are not
<tt>None</tt>. If any difference is found, the test fails.</li>
<li>If the <tt>expected_duration</tt> parameter is specified then it
represents the maximal allowed time in seconds for the test to run. The
test will be marked as failed if its duration is greater than the given
<tt>expected_duration</tt> parameter value.</li>
<li>Stores the new state of the working directory in <tt>self.tree</tt>.
Computes the difference between previous and current trees and stores them
in variables <tt>self.tree_difference</tt> and
<tt>self.unexpected_difference</tt>. Both variables are instances of class
<tt>tree.Trees_different</tt>, which have four attributes:
<tt>added_files</tt>, <tt>removed_files</tt>, <tt>modified_files</tt> and
<tt>touched_files</tt>. Each is a list of strings.</p></li>
</ol>
<h3><a name="method-read">Method <tt>read(self, name)</tt></a></h3>
<p><b>Effects:</b></p>
<p>Read the specified file and returns it content. Raises an exception is
the file is absent.</p>
<h3><a name="method-read_and_strip">Method <tt>read_and_strip(self, name)
</tt></a></h3>
<p><b>Effects:</b></p>
<p>Read the specified file and returns it content, after removing trailing
whitespace from every line. Raises an exception is the file is absent.</p>
<p><b>Rationale:</b></p>
<p>Althought this method is questionable, there are a lot of cases when jam
or shells it uses insert spaces. It seems that introducing this method is
much simpler than dealing with all those cases.</p>
<h3><a name="methods-expectations">Methods for declaring expectations</a>
</h3>
<p>Accordingly to the number of changes kinds that are detected, there are
four methods that specify that test author expects a specific change to
occur. They check <tt>self.unexpected_difference</tt>, and if the change is
present there, it is removed. Otherwise, test fails.</p>
<p>Each method accepts a list of names. Those names use <tt>/</tt> path
separator on all systems. Additionaly, the test system translates suffixes
appropriately. For the test to be portable, suffixes should use Windows
convention: <tt>exe</tt> for executables, <tt>dll</tt> for dynamic libraries
and <tt>lib</tt> for static libraries. Lastly, the string "$toolset" in file
names is replaced by the name of tested toolset.</p>
<p><b>Note:</b> The <tt>List</tt> helper class might be useful to create
lists of names.</p>
<p><b>Note:</b> The file content can be examined using the
<tt>TestCmd.read</tt> function.</p>
<p>The members are:</p>
<ul>
<li>expect_addition</li>
<li>expect_removal</li>
<li>expect_modification</li>
<li>expect_nothing</li>
</ul>
<p>Note that <tt>expect_modification</tt> is used to check that a either
file content or timestamp has changed. The rationale is that some compilers
change content even if sources does not change, and it's easier to have a
method which checks for both content and time changes.</p>
<p>There's also a member <tt>expect_nothing_more</tt>, which checks that all
the changes are either expected or ignored, in other words that
<tt>unexpected_difference</tt> is empty by now.</p>
<p>Lastly, there's a method to compare file content with expected content:
</p>
<tt>expect_content(self, name, content, exact=0)</tt>
<p>The method fails the test if the content of file identified by 'name' is
different from 'content'. If 'exact' is true, the file content is used
as-is, otherwise, two transformations are applied:</p>
<ul>
<li>The <tt>read_and_strip</tt> method is used to read the file, which
removes trailing whitespace</li>
<li>Each backslash in the file content is converted to forward slash.</li>
</ul>
<h3><a name="methods-ignoring">Methods for ignoring changes</a></h3>
<p>There are five methods which ignore changes made to the working tree.
They silently remove elements from <tt>self.unexpected_difference</tt>, and
don't generate error if element is not found. They accept shell style
wildcard.</p>
<p>The following methods correspond to four kinds of changes:</p>
<ul>
<li>ignore_addition(self, wildcard)</li>
<li>ignore_removal(self, wildcard)</li>
<li>ignore_modification(self, wildcard)</li>
<li>ignore_touch(self, wilcard)</li>
</ul>
<p>The method <tt>ignore(self, wildcard)</tt> ignores all the changes made
to files that match a wildcard.</p>
<h3><a name="methods-result">Methods for explicitly specifying results</a>
</h3>
<h4>Method <tt>pass_test(self, condition=1)</tt></h4>
<div class="attention">
At this moment, the method should not be used.
</div>
<h4>Method <tt>fail_test(self, condition=1)</tt></h4>
<p><b>Effects:</b> Cause the test to fail if <tt>condition</tt> is true.</p>
<h3><a name="class-list">Helper class <tt>List</tt></a></h3>
The class has sequence interface and two additional methods.
<h4>Method <tt>__init__(self, string)</tt></h4>
<p><b>Effects:</b> Splits the string on unescaped spaces and tabs. The split
components can further be retrieved using standard sequence access.</p>
<h4>Method <tt>__mul__(self, other)</tt></h4>
<p><b>Effects:</b> Returns an <tt>List</tt> instance, which elements are all
possible concatenations of two string, first of which is from <tt>self</tt>,
and second of which is from <tt>other</tt>.</p>
<p>The class also defines <tt>__str__</tt> and <tt>__repr__</tt> methods.
Finally, there's <tt>__coerce__</tt> method which allows to convert strings
to instances of <tt>List</tt>.</p>
<p><b>Example:</b></p>
<pre>
l = "a b" * List("c d")
for e in l:
print e
</pre>
<p>will output:</p>
<pre>
ac
ad
bc
bd
</pre>
<hr>
<p class="revision">Last modified: May 02, 2008</p>
<p>© Copyright Vladimir Prus 2002, 2003, 2004, 2005.<br>
© Copyright Jurko Gospodnetic 2008.<br>
Distributed under the Boost Software License, Version 1.0.
(See accompanying file LICENSE_1_0.txt or http://www.boost.org/LICENSE_1_0.txt)</p>
</body>
</html>
|
