1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
|
<?xml version="1.0" encoding="latin1" ?>
<!DOCTYPE chapter SYSTEM "chapter.dtd">
<chapter>
<header>
<copyright>
<year>2002</year><year>2013</year>
<holder>Ericsson AB. All Rights Reserved.</holder>
</copyright>
<legalnotice>
The contents of this file are subject to the Erlang Public License,
Version 1.1, (the "License"); you may not use this file except in
compliance with the License. You should have received a copy of the
Erlang Public License along with this software. If not, it can be
retrieved online at http://www.erlang.org/.
Software distributed under the License is distributed on an "AS IS"
basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See
the License for the specific language governing rights and limitations
under the License.
</legalnotice>
<title>Test Server Basics</title>
<prepared>Siri Hansen</prepared>
<docno></docno>
<date></date>
<rev></rev>
<file>basics_chapter.xml</file>
</header>
<section>
<title>Introduction</title>
<p><em>Test Server</em> is a portable test tool for automated
testing of Erlang programs and OTP applications. It provides an
interface for running test programs directly with Test Server
as well as an interface for integrating Test Server
with a framework application. The latter makes it possible to use
Test Server as the engine of a higher level test tool
application.</p>
<p>It is strongly recommended that Test Server be used from inside
a framework application, rather than interfaced directly for
running test programs. Test Server can be pretty difficult to use
since it's a very general and quite extensive and complex
application. Furthermore, the <c>test_server_ctrl</c> functions
are not meant to be used from within the actual test programs. The
framework should handle communication with Test Server and deal
with the more complex aspects of this interaction automatically so
that a higher level interface may be provided for the tester. For
test tool usage to be productive, a simpler, more intuitive and
(if required) more specific interface is required than what Test Server
can provide.</p>
<p>OTP delivers a general purpose framework for Test Server, called
<em>Common Test</em>. This application is a tool well suited for
automated black box testing of target systems of <em>any kind</em>
(not necessarily implemented in Erlang). Common Test is also a very
useful tool for white box testing of Erlang programs and OTP
applications. Unless a more specific functionality and/or user
interface is required (in which case you might need to implement
your own framework), Common Test should do the job for
you. Please read the Common Test User's Guide and reference manual
for more information.</p>
<p>Under normal circumstances, knowledge about the Test Server
application is not required for using the Common Test framework.
However, if you want to use Test Server without a framework,
or learn how to integrate it with your own framework, please read on...
</p>
</section>
<section>
<title>Getting started</title>
<p>Testing when using Test Server is done by running test
suites. A test suite is a number of test cases, where each test
case tests one or more things. The test case is the smallest unit
that the test server deals with. One or more test cases are
grouped together into one ordinary Erlang module, which is called
a test suite. Several test suite modules can be grouped together
in special test specification files representing whole application
and/or system test "jobs".
</p>
<p>The test suite Erlang module must follow a certain interface,
which is specified by Test Server. See the section on writing
test suites for details about this.
</p>
<p>Each test case is considered a success if it returns to the
caller, no matter what the returned value is. An exception to this
is the return value <c>{skip, Reason}</c> which indicates that the
test case is skipped. A failure is specified as a crash, no matter
what the crash reason is.
</p>
<p>As a test suite runs, all information (including output to
stdout) is recorded in several different log files. A minimum of
information is displayed to the user console. This only include
start and stop information, plus a note for each failed test case.
</p>
<p>The result from each test case is recorded in an HTML log file
which is created for each test run. Every test case gets one row
in a table presenting total time, whether the case was successful
or not, if it was skipped, and possibly also a comment. The HTML
file has links to each test case's logfile, which may be viewed
from e.g. Netscape or any other HTML capable browser.
</p>
<p>The Test Server consists of three parts:
</p>
<list type="bulleted">
<item>The part that executes the test suites and
provides support for the test suite author is called
<c>test_server</c>. This is described in the chapter about
writing test cases in this user's guide, and in the reference
manual for the <c>test_server</c> module.</item>
<item>The controlling part, which provides the low level
operator interface, starts and stops slave nodes and writes
log files, is called
<c>test_server_ctrl</c>. The Test Server Controller should not
be used directly when running tests. Instead a framework built
on top of it should be used. More information
about how to write your own framework can be found
in this user's guide and in the reference manual for the
<c>test_server_ctrl</c> module.</item>
</list>
</section>
<section>
<title>Definition of terms</title>
<taglist>
<tag><em>conf(iguration) case</em></tag>
<item>This is a group of test cases which need some specific
configuration. A conf case contains an initiation function which
sets up a specific configuration, one or more test cases using
this configuration, and a cleanup function which restores the
configuration. A conf case is specified in a test specification
either like this:<c>{conf,InitFunc,ListOfCases,CleanupFunc}</c>,
or this: <c>{conf,Properties,InitFunc,ListOfCases,CleanupFunc}</c>
</item>
<tag><em>datadir</em></tag>
<item>Data directory for a test suite. This directory contains
any files used by the test suite, e.g. additional erlang
modules, c code or data files. If the data directory contains
code which must be compiled before the test suite is run, it
should also contain a makefile source called Makefile.src
defining how to compile.
</item>
<tag><em>documentation clause</em></tag>
<item>One of the function clauses in a test case. This clause
shall return a list of strings describing what the test case
tests.
</item>
<tag><em>execution clause</em></tag>
<item>One of the function clauses in a test case. This clause
implements the actual test case, i.e. calls the functions that
shall be tested and checks results. The clause shall crash if it
fails.
</item>
<tag><em>major log file</em></tag>
<item>This is the test suites log file.
</item>
<tag><em>Makefile.src</em></tag>
<item>This file is used by the test server framework to generate
a makefile for a datadir. It contains some special characters
which are replaced according to the platform currently tested.
</item>
<tag><em>minor log file</em></tag>
<item>This is a separate log file for each test case.
</item>
<tag><em>privdir</em></tag>
<item>Private directory for a test suite. This directory should
be used when the test suite needs to write to files.
</item>
<tag><em>skip case</em></tag>
<item>A test case which shall be skipped.
</item>
<tag><em>specification clause</em></tag>
<item>One of the function clauses in a test case. This clause
shall return an empty list, a test specification or
<c>{skip,Reason}</c>. If an empty list is returned, it means
that the test case shall be executed, and so it must also have
an execution clause.
</item>
<tag><em>test case</em></tag>
<item>A single test included in a test suite. Typically it tests
one function in a module or application. A test case is
implemented as a function in a test suite module. The function
can have three clauses, the documentation-, specification- and
execution clause.
</item>
<tag><em>test specification</em></tag>
<item>A specification of which test suites and test cases to
run. There can be test specifications on three different levels
in a test. The top level is a test specification file which
roughly specifies what to test for a whole application. Then
there is a test specification for each test suite returned from
the <c>all(suite)</c> function in the suite. And there can also
be a test specification returned from the specification clause
of a test case.
</item>
<tag><em>test specification file</em></tag>
<item>This is a text file containing the test specification for
an application. The file has the extension ".spec" or
".spec.Platform", where Platform is e.g. "vxworks".
</item>
<tag><em>test suite</em></tag>
<item>An erlang module containing a collection of test cases for
a specific application or module.
</item>
<tag><em>topcase</em></tag>
<item>The first "command" in a test specification file. This
command contains the test specification, like this:
<c>{topcase,TestSpecification}</c></item>
</taglist>
</section>
</chapter>
|
