From 2ef840647acadb489d54332f6a218dcf2e629ff9 Mon Sep 17 00:00:00 2001
From: tmanevik The Common Test Framework provides a high level
- operator interface for testing. It adds the following features to
- the Erlang/OTP Test Server: The When Common Test starts, it will automatically attempt to compile any
+ When
-
+
If compilation should fail for one or more suites, the compilation errors - are printed to tty and the operator is asked if the test run should proceed +
If compilation fails for one or more suites, the compilation errors
+ are printed to tty and the operator is asked if the test run is to proceed
without the missing suites, or be aborted. If the operator chooses to proceed,
- it is noted in the HTML log which tests have missing suites. Also, for each failed
- compilation, the failed tests counter in the return value of
-
Any help module (i.e. regular Erlang module with name not ending with
- "_SUITE") that resides in the same test object directory as a suite
- which is part of the test, will also be automatically compiled. A help
- module will not be mistaken for a test suite (unless it has a "_SUITE"
- name of course). All help modules in a particular test object directory
- are compiled no matter if all or only particular suites in the directory
+ the tests having missing suites are noted in the HTML log. If
Any help module (that is, regular Erlang module with name not ending with + "_SUITE") that resides in the same test object directory as a suite, + which is part of the test, is also automatically compiled. A help + module is not mistaken for a test suite (unless it has a "_SUITE" name). + All help modules in a particular test object directory + are compiled, no matter if all or only particular suites in the directory are part of the test.
If test suites or help modules include header files stored in other
- locations than the test directory, you may specify these include directories
- by means of the
Example (bash):
Common Test will pass all include directories (specified either with the
-
It is also possible to specify include directories in test specifications - (see below).
+Include directories can also be specified in test specifications,
+ see
If the user wants to run all test suites for a test object (or OTP application)
- by specifying only the top directory (e.g. with the
If the user wants to run all test suites for a test object (or an OTP application)
+ by specifying only the top directory (for example, with start flag/option
It is possible to disable the automatic compilation feature by using the
-
To disable the automatic compilation feature, use flag
+
The
The
Examples:
-Examples:
++ $ ct_run -config $CFGS/sys1.cfg $CFGS/sys2.cfg -dir $SYS1_TEST $SYS2_TEST + $ ct_run -userconfig ct_config_xml $CFGS/sys1.xml $CFGS/sys2.xml -dir $SYS1_TEST $SYS2_TEST + $ ct_run -suite $SYS1_TEST/setup_SUITE $SYS2_TEST/config_SUITE + $ ct_run -suite $SYS1_TEST/setup_SUITE -case start stop + $ ct_run -suite $SYS1_TEST/setup_SUITE -group installation -case start stop-
It is also possible to combine the
This has the same effect as calling:
- -For more details on
Other flags that may be used with
The flags
+ $ ct_run -dir ./testdir -suite x_SUITE y_SUITE+ +
This has the same effect as the following:
++ $ ct_run -suite ./testdir/x_SUITE ./testdir/y_SUITE+ +
For details, see
+
The following flags can also be used with
+
Lists all available start flags.
Specifies where the HTML log files are to be written.
Associates the test run with a name that gets printed + in the overview HTML log files.
Refreshes the top-level HTML index files.
Starts web-based GUI (described later).
Starts interactive shell mode (described later).
Steps through test cases using the Erlang Debugger (described later).
Uses test specification as input (described later).
Allows user-specific terms in a test specification (described later).
, tells
Points out a user HTML style sheet (described later).
To perform code coverage test (see
+
To specify if the
To install
+
To install
+
To install
+
Directories passed to Common Test may have either relative or absolute paths.
To enable or disable
+
Arbitrary start flags to the Erlang Runtime System may also be passed as
+ Specifies include directories (described earlier). Disables the automatic test suite compilation feature (described earlier). Aborts the test run if one or more suites fail to compile (described earlier). Extends Enables automatic Tells Tells Tells On time-out, the test run is aborted when the current test job is finished. If Provides a decryption key for
+ Points out a file containing a decryption key for
+ Switches off HTML enhancements that can be incompatible with older browsers. Enables modification of the logging behavior, see
+ Sets
Directories passed to
Any start flags to the Erlang runtime system (application
Example:
Note how in this example, the absolute path of the
The absolute path of directory
The
If auto skipped test cases should not affect the exit status, you may change the default - behaviour using start flag:
--exit_status ignore_config+
If auto-skipped test cases do not affect the exit status. The default + behavior can be changed using start flag:
++ -exit_status ignore_config-
Executing Executing
For more information about the
For more information about the
Common Test provides an Erlang API for running tests. The main (and most - flexible) function for specifying and executing tests is called +
is with
The function returns the test result, represented by the tuple: +
The function returns the test result, represented by the tuple
The default start option
The default start option
During execution of tests, started with +
During execution of tests started with
Note that in order to use the
-
To use the functions
+
For detailed documentation about
-
For details, see
+
With the
- [-case-]]]>
or (in the Erlang shell):
+ [-caseThe syntax in the Erlang shell is as follows:
- ct:run_test([{group,GroupsNamesOrPaths}, {case,Cases}]).]]>+ ct:run_test([{group,GroupsNamesOrPaths}, {case,Cases}]).]]> -
The
Parameter
Given a group name, say
As you may have already realized, this means that if there is more than - one way to locate a group (and its test cases) in a path, the result of the - group search operation is a number of tests, all of which will be performed. - Common Test actually interprets a group specification that consists of a - single name this way:
+This means that if there is more than one way to locate a group
+ (and its test cases) in a path, the result of the group search operation
+ is a number of tests, all of which are to be performed.
+
"Search and find all paths in the group definitions tree that lead - to the specified group and, for each path, create a test which (1) executes - all configuration functions in the path to the specified group, then (2) - executes all - or all matching - test cases in this group, as well as (3) - all - or all matching - test cases in all sub groups of the group". -
+ to the specified group and, for each path, create a test that does the following, + in order: +It is also possible for the user to specify a specific group path with
- the
The user can specify a specific group path with
+ parameter
or similarly in the Erlang shell (requires a list within the groups list):
+The syntax in the Erlang shell is as follows (requires a list within the groups list):
The last group in the specified path will be the terminating group in
- the test, i.e. no sub groups following this group will be executed. In the
- example above,
The last group in the specified path is the terminating group in
+ the test, that is, no subgroups following this group are executed. In the
+ previous example,
Note that the group path specification doesn't necessarily
+ The group path specification does not necessarily
have to include all groups in the path to the terminating group.
- Common Test will search for all matching paths if given an incomplete group
- path.
Note also that it's possible to combine group names and group paths with the
-
Group names and group paths can be combined with parameter
+
Examples:
++ -module(x_SUITE). + ... + %% The group definitions: + groups() -> + [{top1,[],[tc11,tc12, + {sub11,[],[tc12,tc13]}, + {sub12,[],[tc14,tc15, + {sub121,[],[tc12,tc16]}]}]}, + + {top2,[],[{group,sub21},{group,sub22}]}, + {sub21,[],[tc21,{group,sub2X2}]}, + {sub22,[],[{group,sub221},tc21,tc22,{group,sub2X2}]}, + {sub221,[],[tc21,tc23]}, + {sub2X2,[],[tc21,tc24]}].+ +
The following executes two tests, one for all cases and all subgroups
+ under
+ $ ct_run -suite "x_SUITE" -group all + 1> ct:run_test([{suite,"x_SUITE"}, {group,all}]).+
Using
Examples:
+The following executes one test for all cases and subgroups under
- -module(x_SUITE). - ... - %% The group definitions: - groups() -> - [{top1,[],[tc11,tc12, - {sub11,[],[tc12,tc13]}, - {sub12,[],[tc14,tc15, - {sub121,[],[tc12,tc16]}]}]}, - - {top2,[],[{group,sub21},{group,sub22}]}, - {sub21,[],[tc21,{group,sub2X2}]}, - {sub22,[],[{group,sub221},tc21,tc22,{group,sub2X2}]}, - {sub221,[],[tc21,tc23]}, - {sub2X2,[],[tc21,tc24]}]. --
Two tests will be executed, one for all cases and all sub groups under
This will execute one test for all cases and sub groups under
This will run a test that executes
This will execute
This will search
This will execute two tests, one that includes all cases and sub groups under
-
In this example, Common Test will find and execute two tests, one for the path from
-
Here, by specifying the unique path:
In this example only the test cases for
If a test case that belongs to a group (according to the group definition), is executed - without a group specification, i.e. simply by means of (command line):
+ $ ct_run -suite "x_SUITE" -group top1 + 1> ct:run_test([{suite,"x_SUITE"}, {group,[top1]}]). + +The following runs a test executing
+ $ ct_run -suite "x_SUITE" -group top1 -case tc12 + 1> ct:run_test([{suite,"x_SUITE"}, {group,[top1]}, {testcase,[tc12]}]).+ +
The following executes
+ $ ct_run -suite "x_SUITE" -group [top1] -case tc12 + 1> ct:run_test([{suite,"x_SUITE"}, {group,[[top1]]}, {testcase,[tc12]}]).+ +
The following searches
+ $ ct_run -suite "x_SUITE" -group top1 -case tc16 + 1> ct:run_test([{suite,"x_SUITE"}, {group,[top1]}, {testcase,[tc16]}]).+
Using the specific path
The following executes two tests, one including all cases and subgroups under
+
+ $ ct_run -suite "x_SUITE" -group sub12 [sub12] + 1> ct:run_test([{suite,"x_SUITE"}, {group,[sub12,[sub12]]}]).+ +
In the following example,
+ $ ct_run -suite "x_SUITE" -group sub2X2 + 1> ct:run_test([{suite,"x_SUITE"}, {group,[sub2X2]}]).+ +
In the following example, by specifying the unique path
+ $ ct_run -suite "x_SUITE" -group [sub21,sub2X2] + 1> ct:run_test([{suite,"x_SUITE"}, {group,[[sub21,sub2X2]]}]).+ +
The following executes only the test cases for
+ $ ct_run -suite "x_SUITE" -group [sub22] -case tc22 tc21 + 1> ct:run_test([{suite,"x_SUITE"}, {group,[[sub22]]}, {testcase,[tc22,tc21]}]).+ +
If a test case belonging to a group (according to the group definition) is executed + without a group specification, that is, simply by + (using the command line):
or (Erlang shell):
+or (using the Erlang shell):
then Common Test ignores the group definition and executes the test case in the scope of the - test suite only (no group configuration functions are called).
+then
The group specification feature, exactly as it has been presented in this section, can also +
The group specification feature, as presented in this section, can also
be used in
You can start Common Test in an interactive shell mode where no - automatic testing is performed. Instead, in this mode, Common Test +
You can start
The shell mode is useful e.g. for debugging test suites, for analysing +
The shell mode is useful, for example, for debugging test suites, analyzing and debugging the SUT during "simulated" test case execution, and - for trying out various operations during test suite development.
- -To invoke the interactive shell mode, you can start an Erlang shell
- manually and call
To start the interactive shell mode, start an Erlang shell
+ manually and call
If you use the
Examples:
+If no config file is given with the
If no configuration file is specified with command
If any functions using "required config data" (e.g. ct_telnet or
- ct_ftp functions) are to be called from the erlang shell, config
- data must first be required with
Example:
+If any functions using "required configuration data" (for example, functions
+
Example:
- 1> ct:require(unix_telnet, unix). - ok - 2> ct_telnet:open(unix_telnet). - {ok,<0.105.0>} - 4> ct_telnet:cmd(unix_telnet, "ls ."). - {ok,["ls .","file1 ...",...]} -+ 1> ct:require(unix_telnet, unix). + ok + 2> ct_telnet:open(unix_telnet). + {ok,<0.105.0>} + 4> ct_telnet:cmd(unix_telnet, "ls ."). + {ok,["ls .","file1 ...",...]} -
Everything that Common Test normally prints in the test case logs,
- will in the interactive mode be written to a log named
-
Everything that
If you wish to exit the interactive mode (e.g. to start an
- automated test run with
If you wish to exit the interactive mode (for example, to start an automated
+ test run with
By means of
If no extra options are given with the
Using
If no extra options are specified with flag/option
Common Test enables the Debugger auto attach feature, which means +
The step functionality can be used together with the
The step functionality can be used together with flag/option
The most flexible way to specify what to test, is to use a so - called test specification. A test specification is a sequence of +
The most flexible way to specify what to test, is to use a
+ test specification, which is a sequence of
Erlang terms. The terms are normally declared in one or more text files
(see
With configuration terms it is possible to e.g. label the test
- run (similar to
Configuration terms can be combined with
With configuration terms it is, for example, possible to do the following:
+Configuration terms can be combined with
With test specification terms it is possible to state exactly - which tests should run and in which order. A test term specifies + terms (for example, log directory, label, style sheet, and auto-compilation).
+ +With test specification terms, it is possible to state exactly + which tests to run and in which order. A test term specifies either one or more suites, one or more test case groups (possibly nested), or one or more test cases in a group (or in multiple groups) or in a suite.
-An arbitrary number of test terms may be declared in sequence. - Common Test will by default compile the terms into one or more tests - to be performed in one resulting test run. Note that a term that - specifies a set of test cases will "swallow" one that only - specifies a subset of these cases. E.g. the result of merging - one term that specifies that all cases in suite S should be + +
Any number of test terms can be declared in sequence.
+
A test term can also specify one or more test suites, groups, - or test cases to be skipped. Skipped suites, groups and cases - are not executed and show up in the HTML log files as - SKIPPED.
+ or test cases to be skipped. Skipped suites, groups, and cases + are not executed and show up in the HTML log files asWhen multiple test specification files are given at startup (either +
When multiple test specification files are specified at startup (either
with
Joining a number of specifications, or running them separately, can - also be accomplished with (and may be combined with) test specification - file inclusion, described next.
+ also be accomplished with (and can be combined with) test specification + file inclusion.With the
With the term
Example:
+- %% In specification file "a.spec" - {specs, join, ["b.spec", "c.spec"]}. - {specs, separate, ["d.spec", "e.spec"]}. - %% Config and test terms follow - ...+ %% In specification file "a.spec" + {specs, join, ["b.spec", "c.spec"]}. + {specs, separate, ["d.spec", "e.spec"]}. + %% Config and test terms follow + ... +
In this example, the test terms defined in files "b.spec" and "c.spec" - will be joined with the terms in the source specification "a.spec" + are joined with the terms in source specification "a.spec" (if any). The inclusion of specifications "d.spec" and - "e.spec" will result in two separate, and independent, test runs (i.e. - one for each included specification).
-Note that the
Option
Joined specifications share common configuration settings, such as
the list of
If
Note that it is always the
Notice that it is always the
The
The term
When a test case group is specified, the resulting test
- executes the
The test specification utilizes the same mechanism for specifying
- test case groups by means of names and paths, as explained in the
-
The
The test specification uses the same mechanism for specifying
+ test case groups through names and paths, as explained in section
+
Element
Below is the test specification syntax. Test specifications can
- be used to run tests both in a single test host environment and
- in a distributed Common Test environment (Large Scale
- Testing). The node parameters in the Test specifications can be used to run tests both in a single
+ test host environment and in a distributed
+
Config terms:
+ (as most flags have a corresponding configuration term) +Configuration terms:
- {merge_tests, Bool}. - - {define, Constant, Value}. - - {specs, InclSpecsOption, TestSpecs}. - - {node, NodeAlias, Node}. - - {init, InitOptions}. - {init, [NodeAlias], InitOptions}. - - {label, Label}. - {label, NodeRefs, Label}. - - {verbosity, VerbosityLevels}. - {verbosity, NodeRefs, VerbosityLevels}. - - {stylesheet, CSSFile}. - {stylesheet, NodeRefs, CSSFile}. - - {silent_connections, ConnTypes}. - {silent_connections, NodeRefs, ConnTypes}. - - {multiply_timetraps, N}. - {multiply_timetraps, NodeRefs, N}. - - {scale_timetraps, Bool}. - {scale_timetraps, NodeRefs, Bool}. - - {cover, CoverSpecFile}. - {cover, NodeRefs, CoverSpecFile}. - - {cover_stop, Bool}. - {cover_stop, NodeRefs, Bool}. - - {include, IncludeDirs}. - {include, NodeRefs, IncludeDirs}. - - {auto_compile, Bool}, - {auto_compile, NodeRefs, Bool}, - - {abort_if_missing_suites, Bool}, - {abort_if_missing_suites, NodeRefs, Bool}, + {merge_tests, Bool}. - {config, ConfigFiles}. - {config, ConfigDir, ConfigBaseNames}. - {config, NodeRefs, ConfigFiles}. - {config, NodeRefs, ConfigDir, ConfigBaseNames}. - - {userconfig, {CallbackModule, ConfigStrings}}. - {userconfig, NodeRefs, {CallbackModule, ConfigStrings}}. - - {logdir, LogDir}. - {logdir, NodeRefs, LogDir}. - - {logopts, LogOpts}. - {logopts, NodeRefs, LogOpts}. - - {create_priv_dir, PrivDirOption}. - {create_priv_dir, NodeRefs, PrivDirOption}. - - {event_handler, EventHandlers}. - {event_handler, NodeRefs, EventHandlers}. - {event_handler, EventHandlers, InitArgs}. - {event_handler, NodeRefs, EventHandlers, InitArgs}. - - {ct_hooks, CTHModules}. - {ct_hooks, NodeRefs, CTHModules}. - - {enable_builtin_hooks, Bool}. - - {basic_html, Bool}. - {basic_html, NodeRefs, Bool}. - - {release_shell, Bool}.+ {define, Constant, Value}. + + {specs, InclSpecsOption, TestSpecs}. + + {node, NodeAlias, Node}. + + {init, InitOptions}. + {init, [NodeAlias], InitOptions}. + + {label, Label}. + {label, NodeRefs, Label}. + + {verbosity, VerbosityLevels}. + {verbosity, NodeRefs, VerbosityLevels}. + + {stylesheet, CSSFile}. + {stylesheet, NodeRefs, CSSFile}. + + {silent_connections, ConnTypes}. + {silent_connections, NodeRefs, ConnTypes}. + + {multiply_timetraps, N}. + {multiply_timetraps, NodeRefs, N}. + + {scale_timetraps, Bool}. + {scale_timetraps, NodeRefs, Bool}. + + {cover, CoverSpecFile}. + {cover, NodeRefs, CoverSpecFile}. + + {cover_stop, Bool}. + {cover_stop, NodeRefs, Bool}. + + {include, IncludeDirs}. + {include, NodeRefs, IncludeDirs}. + + {auto_compile, Bool}, + {auto_compile, NodeRefs, Bool}, + + {abort_if_missing_suites, Bool}, + {abort_if_missing_suites, NodeRefs, Bool}, + + {config, ConfigFiles}. + {config, ConfigDir, ConfigBaseNames}. + {config, NodeRefs, ConfigFiles}. + {config, NodeRefs, ConfigDir, ConfigBaseNames}. + + {userconfig, {CallbackModule, ConfigStrings}}. + {userconfig, NodeRefs, {CallbackModule, ConfigStrings}}. + + {logdir, LogDir}. + {logdir, NodeRefs, LogDir}. + + {logopts, LogOpts}. + {logopts, NodeRefs, LogOpts}. + + {create_priv_dir, PrivDirOption}. + {create_priv_dir, NodeRefs, PrivDirOption}. + + {event_handler, EventHandlers}. + {event_handler, NodeRefs, EventHandlers}. + {event_handler, EventHandlers, InitArgs}. + {event_handler, NodeRefs, EventHandlers, InitArgs}. + + {ct_hooks, CTHModules}. + {ct_hooks, NodeRefs, CTHModules}. + + {enable_builtin_hooks, Bool}. + + {basic_html, Bool}. + {basic_html, NodeRefs, Bool}. + + {release_shell, Bool}. -
Test terms:
+Test terms:
- {suites, Dir, Suites}. - {suites, NodeRefs, Dir, Suites}. - - {groups, Dir, Suite, Groups}. - {groups, NodeRefs, Dir, Suite, Groups}. - - {groups, Dir, Suite, Groups, {cases,Cases}}. - {groups, NodeRefs, Dir, Suite, Groups, {cases,Cases}}. - - {cases, Dir, Suite, Cases}. - {cases, NodeRefs, Dir, Suite, Cases}. - - {skip_suites, Dir, Suites, Comment}. - {skip_suites, NodeRefs, Dir, Suites, Comment}. - - {skip_groups, Dir, Suite, GroupNames, Comment}. - {skip_groups, NodeRefs, Dir, Suite, GroupNames, Comment}. - - {skip_cases, Dir, Suite, Cases, Comment}. - {skip_cases, NodeRefs, Dir, Suite, Cases, Comment}.- -
Types:
+ {suites, Dir, Suites}. + {suites, NodeRefs, Dir, Suites}. + + {groups, Dir, Suite, Groups}. + {groups, NodeRefs, Dir, Suite, Groups}. + + {groups, Dir, Suite, Groups, {cases,Cases}}. + {groups, NodeRefs, Dir, Suite, Groups, {cases,Cases}}. + + {cases, Dir, Suite, Cases}. + {cases, NodeRefs, Dir, Suite, Cases}. + + {skip_suites, Dir, Suites, Comment}. + {skip_suites, NodeRefs, Dir, Suites, Comment}. + + {skip_groups, Dir, Suite, GroupNames, Comment}. + {skip_groups, NodeRefs, Dir, Suite, GroupNames, Comment}. + + {skip_cases, Dir, Suite, Cases, Comment}. + {skip_cases, NodeRefs, Dir, Suite, Cases, Comment}. + +Types:
- Bool = true | false - Constant = atom() - Value = term() - InclSpecsOption = join | separate - TestSpecs = string() | [string()] - NodeAlias = atom() - Node = node() - NodeRef = NodeAlias | Node | master - NodeRefs = all_nodes | [NodeRef] | NodeRef - InitOptions = term() - Label = atom() | string() - VerbosityLevels = integer() | [{Category,integer()}] - Category = atom() - CSSFile = string() - ConnTypes = all | [atom()] - N = integer() - CoverSpecFile = string() - IncludeDirs = string() | [string()] - ConfigFiles = string() | [string()] - ConfigDir = string() - ConfigBaseNames = string() | [string()] - CallbackModule = atom() - ConfigStrings = string() | [string()] - LogDir = string() - LogOpts = [term()] - PrivDirOption = auto_per_run | auto_per_tc | manual_per_tc - EventHandlers = atom() | [atom()] - InitArgs = [term()] - CTHModules = [CTHModule | - {CTHModule, CTHInitArgs} | - {CTHModule, CTHInitArgs, CTHPriority}] - CTHModule = atom() - CTHInitArgs = term() - Dir = string() - Suites = atom() | [atom()] | all - Suite = atom() - Groups = GroupPath | [GroupPath] | GroupSpec | [GroupSpec] | all - GroupPath = [GroupName] - GroupSpec = GroupName | {GroupName,Properties} | {GroupName,Properties,GroupSpec} - GroupName = atom() - GroupNames = GroupName | [GroupName] - Cases = atom() | [atom()] | all - Comment = string() | ""- -
The difference between the
The difference between the
- {config, ["/home/testuser/tests/config/nodeA.cfg", - "/home/testuser/tests/config/nodeB.cfg"]}. - - {config, "/home/testuser/tests/config", ["nodeA.cfg","nodeB.cfg"]}.+ {config, ["/home/testuser/tests/config/nodeA.cfg", + "/home/testuser/tests/config/nodeB.cfg"]}. + + {config, "/home/testuser/tests/config", ["nodeA.cfg","nodeB.cfg"]}. -
Any relative paths specified in the test specification, will be
- relative to the directory which contains the test specification file, if
+ Any relative paths, specified in the test specification, are
+ relative to the directory containing the test specification file if
The path is relative to the top-level log directory if
The
The term
For the sake of readability, the name of the constant must always
- begin with an upper case letter, or a
The main benefit of constants is that they can be used to reduce the size - (and avoid repetition) of long strings, such as file paths. Compare these - terms:
+ (and avoid repetition) of long strings, such as file paths. +Examples:
- %% 1a. no constant - {config, "/home/testuser/tests/config", ["nodeA.cfg","nodeB.cfg"]}. - {suites, "/home/testuser/tests/suites", all}. - - %% 1b. with constant - {define, 'TESTDIR', "/home/testuser/tests"}. - {config, "'TESTDIR'/config", ["nodeA.cfg","nodeB.cfg"]}. - {suites, "'TESTDIR'/suites", all}. - - %% 2a. no constants - {config, [testnode@host1, testnode@host2], "../config", ["nodeA.cfg","nodeB.cfg"]}. - {suites, [testnode@host1, testnode@host2], "../suites", [x_SUITE, y_SUITE]}. - - %% 2b. with constants - {define, 'NODE', testnode}. - {define, 'NODES', ['NODE'@host1, 'NODE'@host2]}. - {config, 'NODES', "../config", ["nodeA.cfg","nodeB.cfg"]}. - {suites, 'NODES', "../suites", [x_SUITE, y_SUITE]}.+ %% 1a. no constant + {config, "/home/testuser/tests/config", ["nodeA.cfg","nodeB.cfg"]}. + {suites, "/home/testuser/tests/suites", all}. + + %% 1b. with constant + {define, 'TESTDIR', "/home/testuser/tests"}. + {config, "'TESTDIR'/config", ["nodeA.cfg","nodeB.cfg"]}. + {suites, "'TESTDIR'/suites", all}. + + %% 2a. no constants + {config, [testnode@host1, testnode@host2], "../config", ["nodeA.cfg","nodeB.cfg"]}. + {suites, [testnode@host1, testnode@host2], "../suites", [x_SUITE, y_SUITE]}. + + %% 2b. with constants + {define, 'NODE', testnode}. + {define, 'NODES', ['NODE'@host1, 'NODE'@host2]}. + {config, 'NODES', "../config", ["nodeA.cfg","nodeB.cfg"]}. + {suites, 'NODES', "../suites", [x_SUITE, y_SUITE]}.
Constants make the test specification term
- %% using the old alias term - {config, "/home/testuser/tests/config/nodeA.cfg"}. - {alias, suite_dir, "/home/testuser/tests/suites"}. - {groups, suite_dir, x_SUITE, group1}. - - %% replacing with constants - {define, 'TestDir', "/home/testuser/tests"}. - {define, 'CfgDir', "'TestDir'/config"}. - {define, 'SuiteDir', "'TestDir'/suites"}. - {config, 'CfgDir', "nodeA.cfg"}. - {groups, 'SuiteDir', x_SUITE, group1}.+ %% using the old alias term + {config, "/home/testuser/tests/config/nodeA.cfg"}. + {alias, suite_dir, "/home/testuser/tests/suites"}. + {groups, suite_dir, x_SUITE, group1}. + + %% replacing with constants + {define, 'TestDir', "/home/testuser/tests"}. + {define, 'CfgDir', "'TestDir'/config"}. + {define, 'SuiteDir', "'TestDir'/suites"}. + {config, 'CfgDir', "nodeA.cfg"}. + {groups, 'SuiteDir', x_SUITE, group1}. -
Actually, constants could well replace the
Constants can well replace term
Here follows a simple test specification example:
- {define, 'Top', "/home/test"}. - {define, 'T1', "'Top'/t1"}. - {define, 'T2', "'Top'/t2"}. - {define, 'T3', "'Top'/t3"}. - {define, 'CfgFile', "config.cfg"}. - - {logdir, "'Top'/logs"}. - - {config, ["'T1'/'CfgFile'", "'T2'/'CfgFile'", "'T3'/'CfgFile'"]}. - - {suites, 'T1', all}. - {skip_suites, 'T1', [t1B_SUITE,t1D_SUITE], "Not implemented"}. - {skip_cases, 'T1', t1A_SUITE, [test3,test4], "Irrelevant"}. - {skip_cases, 'T1', t1C_SUITE, [test1], "Ignore"}. - - {suites, 'T2', [t2B_SUITE,t2C_SUITE]}. - {cases, 'T2', t2A_SUITE, [test4,test1,test7]}. - - {skip_suites, 'T3', all, "Not implemented"}.+ {define, 'Top', "/home/test"}. + {define, 'T1', "'Top'/t1"}. + {define, 'T2', "'Top'/t2"}. + {define, 'T3', "'Top'/t3"}. + {define, 'CfgFile', "config.cfg"}. + + {logdir, "'Top'/logs"}. + + {config, ["'T1'/'CfgFile'", "'T2'/'CfgFile'", "'T3'/'CfgFile'"]}. + + {suites, 'T1', all}. + {skip_suites, 'T1', [t1B_SUITE,t1D_SUITE], "Not implemented"}. + {skip_cases, 'T1', t1A_SUITE, [test3,test4], "Irrelevant"}. + {skip_cases, 'T1', t1C_SUITE, [test1], "Ignore"}. + + {suites, 'T2', [t2B_SUITE,t2C_SUITE]}. + {cases, 'T2', t2A_SUITE, [test4,test1,test7]}. + + {skip_suites, 'T3', all, "Not implemented"}.
The example specifies the following:
-With the
With term
It is possible for the user to provide a test specification that
- includes (for Common Test) unrecognizable terms. If this is desired,
- the
The user can provide a test specification including (for
It's possible to look up terms in the current test specification
- (i.e. the spec that's been used to configure and run the current test).
- The function
Terms in the current test specification
+ (that is, the specification that has been used to configure and run the current test)
+ can be looked up.
+ The function
For example, in the test specification:
- ... - {label, my_server_smoke_test}. - {config, "../../my_server_setup.cfg"}. - {config, "../../my_server_interface.cfg"}. - ...-
And in e.g. a test suite or a CT hook function:
+ ... + {label, my_server_smoke_test}. + {config, "../../my_server_setup.cfg"}. + {config, "../../my_server_interface.cfg"}. + ... +And in, for example, a test suite or a
- ... - [{label,[{_Node,TestType}]}, {config,CfgFiles}] = - ct:get_testspec_terms([label,config]), + ... + [{label,[{_Node,TestType}]}, {config,CfgFiles}] = + ct:get_testspec_terms([label,config]), - [verify_my_server_cfg(TestType, CfgFile) || {Node,CfgFile} <- CfgFiles, - Node == node()]; - ...+ [verify_my_server_cfg(TestType, CfgFile) || {Node,CfgFile} <- CfgFiles, + Node == node()]; + ...
The web based GUI, VTS, is started with the +
The web-based GUI, Virtual Test Server (VTS), is started with the
Examples:
+From the GUI you can run tests and view the result and the logs.
-Note that
Example:
+Example:
Note that the browser must run as a separate OS process or VTS will hang!
-If no specific browser start command is specified, Firefox will
- be the default browser on Unix platforms and Internet Explorer on Windows.
- If Common Test fails to start a browser automatically, or The browser must run as a separate OS process, otherwise VTS hangs.
If no specific browser start command is specified, Firefox is
+ the default browser on Unix platforms, and Internet Explorer on Windows.
+ If
As the execution of the test suites proceed, events are logged in - four different ways:
+ the following four different ways: -Typically the operator, who may run hundreds or thousands of - test cases, doesn't want to fill the console with details - about, or printouts from, the specific test cases. By default, the - operator will only see:
+Typically the operator, possibly running hundreds or thousands of + test cases, does not want to fill the console with details + about, or printouts from, specific test cases. By default, the + operator only sees the following:
-If/when the operator wants to dig deeper into the general results, or
- the result of a specific test case, he should do so by
- following the links in the HTML presentation and take a look in the
- major or minor log files. The "all_runs.html" page is a practical
- starting point usually. It's located in
To dig deeper into the general results, or
+ the result of a specific test case, the operator can do so by
+ following the links in the HTML presentation and read the
+ major or minor log files. The "all_runs.html" page is a good
+ starting point. It is located in
An "index.html" page is written for each test run (i.e. stored in - the "ct_run" directory tagged with node name, date and time). This - file gives a short overview of all individual tests performed in the - same test run. The test names follow this convention:
-An "index.html" page is written for each test run (that is, stored in
+ the
On the test run index page there is a link to the Common Test +
The "test run index" page includes a link to the
On the test run index page it is noted if a test has missing - suites (i.e. suites that Common Test has failed to +
The "test run index" page indicates if a test has missing
+ suites (that is, suites that
The major log file shows a detailed report of the test run. It includes test suite and test case names, execution time, the - exact reason for failures etc. The information is available in both + exact reason for failures, and so on. The information is available in both a file with textual and with HTML representation. The HTML file shows a - summary which gives a good overview of the test run. It also has links + summary that gives a good overview of the test run. It also has links to each individual test case log file for quick viewing with an HTML browser.
The minor log files contain full details of every single test
- case, each one in a separate file. This way, it should be
+ case, each in a separate file. This way, it is
straightforward to compare the latest results to that of previous
- test runs, even if the set of test cases changes. If SASL is running,
- its logs will also be printed to the current minor log file by the
+ test runs, even if the set of test cases changes. If application
The full name of the minor log file (i.e. the name of the file +
The full name of the minor log file (that is, the name of the file
including the absolute directory path) can be read during execution
- of the test case. It comes as value in the tuple
+ of the test case. It comes as value in tuple
Which information goes where is user configurable via the - test server controller. Three threshold values determine what - comes out on screen, and in the major or minor log files. See - the OTP Test Server manual for information. The contents that - goes to the HTML log file is fixed however and cannot be altered.
- -The log files are written continously during a test run and links are - always created initially when a test starts. This makes it possible - to follow test progress simply by refreshing pages in the HTML browser. + +
The log files are written continuously during a test run and links are + always created initially when a test starts. Thevtest progress can therefore + be followed simply by refreshing pages in the HTML browser. Statistics totals are not presented until a test is complete however.
With the
With
With
With start flag
The HTML version of the test suite source code is not + generated during the test run (and is consequently not available + in the log file system).
For example, if a test is started with:
then printouts during the test made by successive calls to
instead of each
instead of each
By clicking the name in the column header of any table (e.g. "Ok", "Case", "Time", etc),
- the table rows are sorted in whatever order makes sense for the type of value (e.g.
- numerical for "Ok" or "Time", and alphabetical for "Case"). The sorting is performed
- by means of JavaScript code, automatically inserted into the HTML log files. Common Test
- uses the
By clicking the name in the column header of any table
+ (for example, "Ok", "Case", "Time", and so on), the table rows are sorted
+ in whatever order makes sense for the type of value (for example,
+ numerical for "Ok" or "Time", and alphabetical for "Case"). The sorting is
+ performed through JavaScript code, automatically inserted into the HTML
+ log files.
On the test suites overview page you find a link to the Unexpected I/O Log.
- In this log, Common Test saves printouts made with
-
The test suites overview page includes a link to the Unexpected I/O Log.
+ In this log,
On the Common Test Framework Log page you find links to the so called
- Pre- and Post Test I/O Log. In this log, Common Test saves printouts made with
-
The
Note that logging to file with
Logging to file with
Common Test uses an HTML Style Sheet (CSS file) to control the look of - the HTML log files generated during test runs. If, for some reason, the - log files are not displayed correctly in the browser of your - choice, or you prefer a more primitive ("pre Common Test v1.6") look - of the logs, use the start flag/option:
-basic_html-
This disables the use of Style Sheets, as well as JavaScripts (see - table sorting above).
++ basic_html+
This disables the use of style sheets and JavaScripts (see
+
Common Test includes an optional feature to allow +
- div.sys_config { background:blue; color:white } - div.sys_state { background:yellow; color:black } - div.error { background:red; color:white }+ div.sys_config { background:blue; color:white } + div.sys_state { background:yellow; color:black } + div.error { background:red; color:white } -
To install the CSS file (Common Test inlines the definition in the
- HTML code), the name may be provided when executing
To install the CSS file (
Example:
- $ ct_run -dir $TEST/prog -stylesheet $TEST/styles/test_categories.css+ $ ct_run -dir $TEST/prog -stylesheet $TEST/styles/test_categories.css -
Categories in a CSS file installed with the
Categories in a CSS file installed with flag
It is also possible to install style sheets on a per suite and - per test case basis. Example:
+Style sheets can also be installed on a per suite and + per test case basis.
+Example:
- -module(my_SUITE). - ... - suite() -> [..., {stylesheet,"suite_categories.css"}, ...]. - ... - my_testcase(_) -> - ... - ct:log(sys_config, "Test node version: ~p", [VersionInfo]), - ... - ct:log(sys_state, "Connections: ~p", [ConnectionInfo]), - ... - ct:pal(error, "Error ~p detected! Info: ~p", [SomeFault,ErrorInfo]), - ct:fail(SomeFault).+ -module(my_SUITE). + ... + suite() -> [..., {stylesheet,"suite_categories.css"}, ...]. + ... + my_testcase(_) -> + ... + ct:log(sys_config, "Test node version: ~p", [VersionInfo]), + ... + ct:log(sys_state, "Connections: ~p", [ConnectionInfo]), + ... + ct:pal(error, "Error ~p detected! Info: ~p", [SomeFault,ErrorInfo]), + ct:fail(SomeFault).
If the style sheet is installed as in this example, the categories are
private to the suite in question. They can be used by all test cases in the
- suite, but can not be used by other suites. A suite private style sheet,
- if specified, will be used in favour of a global style sheet (one specified
- with the
In a tuple
The
Argument
You can order Common Test to repeat the tests you specify. You can choose - to repeat tests a certain number of times, repeat tests for a specific period of time, +
You can order
The duration time,
When timeout occurs, Common Test will never abort the ongoing test case, since
- this might leave the system under test in an undefined, and possibly bad, state.
- Instead Common Test will by default finish the current test
- run before stopping. If the
Log files from every single repeated test run is saved in normal Common Test fashion (see above). - Common Test may later support an optional feature to only store the last (and possibly - the first) set of logs of repeated test runs, but for now the user must be careful not - to run out of disk space if tests are repeated during long periods of time.
- -Note that for each test run that is part of a repeated session, information about the - particular test run is printed in the Common Test Framework Log. There you can read - the repetition number, remaining time, etc.
- -Example 1:
+The duration time is specified as
The finish time can be specified as
When time-out occurs,
As
Log files from every repeated test run is saved in normal
For each test run that is part of a repeated session, information about the
+ particular test run is printed in the
Example 1:
- $ ct_run -dir $TEST_ROOT/to1 $TEST_ROOT/to2 -duration 001000 -force_stop-
Here the suites in test directory to1, followed by the suites in to2, will be executed
- in one test run. A timeout event will occur after 10 minutes. As long as there is time
- left, Common Test will repeat the test run (i.e. starting over with the to1 test).
- When the timeout occurs, Common Test will stop as soon as the current job is finished
- (because of the
Example 2:
+ $ ct_run -dir $TEST_ROOT/to1 $TEST_ROOT/to2 -duration 001000 -force_stop + +Here, the suites in test directory
Example 2:
- $ ct_run -dir $TEST_ROOT/to1 $TEST_ROOT/to2 -duration 001000 -forces_stop skip_rest-
Here the same test run as in Example 1, but with the
-
Example 3:
+ $ ct_run -dir $TEST_ROOT/to1 $TEST_ROOT/to2 -duration 001000 -forces_stop skip_rest + +Here, the same tests as in Example 1 are run, but with flag
Example 3:
- $ date - Fri Sep 28 15:00:00 MEST 2007 - - $ ct_run -dir $TEST_ROOT/to1 $TEST_ROOT/to2 -until 160000-
Here the same test run as in the example above will be executed (and possibly repeated). - In this example, however, the timeout will occur after 1 hour and when that happens, - Common Test will finish the entire test run before stopping (i.e. the to1 and to2 test - will always both be executed in the same test run).
+ $ date + Fri Sep 28 15:00:00 MEST 2007 + + $ ct_run -dir $TEST_ROOT/to1 $TEST_ROOT/to2 -until 160000 + +Here, the same test run as in the previous examples are executed (and possibly repeated).
+ However, when the time-out occurs, after 1 hour,
Example 4:
+Example 4:
- $ ct_run -dir $TEST_ROOT/to1 $TEST_ROOT/to2 -repeat 5-
Here the test run, including both the to1 and the to2 test, will be repeated 5 times.
+ $ ct_run -dir $TEST_ROOT/to1 $TEST_ROOT/to2 -repeat 5 + +Here, the test run, including both the
This feature should not be confused with the Do not confuse this feature with the
The protocol handling processes in Common Test, implemented by ct_telnet,
- ct_ssh, ct_ftp etc, do verbose printing to the test case logs. This can be switched off
- by means of the
The protocol handling processes in
- ct_run -silent_connections [conn_types] -+ ct_run -silent_connections [conn_types] -
where
Here,
Example:
+Example 1:
- ct_run ... -silent_connections ssh telnet-
switches off logging for ssh and telnet connections.
+ ct_run ... -silent_connections ssh telnet +This switches off logging for SSH and Telnet connections.
+ +Example 2:
- ct_run ... -silent_connections-
switches off logging for all connection types.
+ ct_run ... -silent_connections +This switches off logging for all connection types.
-Fatal communication error and reconnection attempts will always be printed even - if logging has been suppressed for the connection type in question. However, operations - such as sending and receiving data will be performed silently.
+Fatal communication error and reconnection attempts are always printed, even if + logging has been suppressed for the connection type in question. However, operations + such as sending and receiving data are performed silently.
-It is possible to also specify
Example:
+Example 3:
- - -module(my_SUITE). + -module(my_SUITE). - suite() -> [..., {silent_connections,[telnet,ssh]}, ...]. + suite() -> [..., {silent_connections,[telnet,ssh]}, ...]. - ... + ... - my_testcase1() -> - [{silent_connections,[ssh]}]. + my_testcase1() -> + [{silent_connections,[ssh]}]. - my_testcase1(_) -> - ... + my_testcase1(_) -> + ... - my_testcase2(_) -> - ... -+ my_testcase2(_) -> + ... -
In this example,
In this example,
The
Start flag/option
Note that in the current Common Test version, the
-
In the current