1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
|
How the test suites work in Gitano
==================================
Gitano's test suite is a functional black-box suite built in the `yarn` testing
language. The testing stuff all lives in the testing directory at the top
level of the Gitano git repository.
The test suite can run in both an SSH mode and an HTTP mode. In the former the
accesses to the Gitano instances created during the tests are made by simulated
SSH access. In the latter they are made by simulating HTTP access to the CGIs
which form the Gitano interface.
To run the test suite, simply:
make test
If you want to run a specific scenario from the test suite:
make test SCENARIO="whatever scenario name"
If you are making changes and just want to 'smoke test' Gitano to ensure that
you have not made any fundamental compiler errors etc, then run:
make basictest
The basictest run will only run in SSH mode by default, you can switch that
to HTTP mode if you are changing the CGI code by running:
make basictest TEST_PROTO=http
The protocol being used during a test run is exposed in the GTT_PROTO
environment variable passed to implementations.
Yarn implementations
--------------------
The original (and primary) Yarn is part of the `cmdtest` suite by Lars
Wirzenius, is written in Python 2, and can be found in the `cmdtest` Debian
package or at <http://git.liw.fi/cgi-bin/cgit/cgit.cgi/cmdtest/> at the time of
writing this document.
The Gitano test suite is also compatible with `rsyarn` by Daniel Silverstone,
is written in Rust (needs 1.13 or better), and can be found at
<https://git.gitano.org.uk/rsyarn.git> at the time of writing this document.
Other Yarn implementations may be compatible, but the Makefile may need
tweaking if they do not expose the same cmdline arguments. Patches are welcome
to support other Yarn implementations' cmdline syntax.
You may select a different Yarn tool by setting the 'YARN' make variable on
the command line, and you may pass additional arguments as may be necessary by
setting the 'YARN_ARGS' variable on the command line, for example if you wish
to use `rsyarn` and have it run five tests in parallel, you might run:
make test YARN=rsyarn YARN_ARGS=-j5
This can be a significant speed boost when your computer is capable of running
multiple of the scenarios in parallel, for example on the author's slow old
laptop, the full yarn suite takes over two minutes to run using a single
threaded approach, or only 40 seconds if five scenarios are run in parallel.
Note: The test suite was originally written with single-threaded execution in
mind, and though the Yarns allows for arbitrary numbers of scenarios to run in
parallel, you should be careful not to overload your system when running the
test suite.
HTTP mode test dependencies
---------------------------
HTTP tests work via password access rather than key access, so the `htpasswd`
binary must be in your `PATH`. In Debian (and derived distributions) this can
be accomplised by installing the `apache2-utils` package.
HTTP tests require starting an HTTP server on an unused port. The test suite
implements this using `lighttpd`, so that binary must be in your `PATH`. In
Debian (and derived distributions) this can be accomplished by installing the
`lighttpd` package. This package may start the server by default, so you are
likely to want to run `systemctl disable --now lighttpd.service` to stop it.
The tests default to running servers on ports starting at 8080 and walking up
to find a free one (up to a maximum of ten times). If there is a better port
to use set `HTTP_FIRST_TEST_PORT`. This limitation means that if your `yarn`
implementation parallelises scenarios, you must limit the parallelism to ten or
else the HTTP tests will not run successfully.
The fundamental parts of the `testing/` directory are
=====================================================
* `gitano-test-tool.in`: This is a tool which helps the test suite to do
complex operations including simulating different UNIX users, setting up a
Gitano instance, etc.
* `http-unwrap`: This tool helps gitano-test-tool to unwrap HTTP responses
generated by the test suite when running in HTTP mode.
* `keys/`: This directory contains a set of keys which are used in various
parts of the test suite
* `admin-patches/`: This directory contains patches to the gitano-admin.git
which can be applied during tests. If you change the skeleton ruleset (or
otherwise) for Gitano, you may need to refresh these patches.
* `library.yarn`: This contains the implementations for the test suite yarn
statements.
* `01_*.yarn`: These are considered the basic tests
* `02_*.yarn`: These tests are focussed around validating the commands
specifically. The test might not use only the command in question, but that
is their goal.
* `03_*.yarn` These tests are more integration tests which look to simulate
more complex user use-cases.
Tip and tricks when writing tests for Gitano
============================================
* Until this point is taken out of this document, testing explicitly against
stderr is only guaranteed to work in ssh-only tests. Guard those scenarios
with an 'ASSUMING' which checks. For example, 'ASSUMING rsync is possible'
checks this.
* Where there are checks for 'stdout' or 'stderr' you can also write the same
kinds of statements for 'the output' which will consider a concatenation of
the two.
* Remember that the stdout/stderr of a step is not retained by default by all
yarn implementations, even in --snapshot mode. If you need the output of
a shell command for debugging purposes, be sure to write it to a file.
* There is a utility "IMPLEMENTS THEN failure ensues" which always fails but
also dumps a whole bunch of information to its output which can be a quick
way to debug a scenario without needing to --snapshot which slows things
down.
* Remember that access to Gitano behaves differently over SSH and over HTTP.
If your test assumes the behaviour of one or the other you must either guard
as in the first point or write your IMPLEMENTS to cope.
* As of the time of writing, Gitano's Yarns are written in shell. You should
only assume POSIX shell functionality. We default to `/bin/sh` which on all
sensible systems is a simple shell such as `dash`. If your `/bin/sh` is,
however, `bash` then you should take extra care not to write bashisms into
your test implementations.
|
