1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
|
Tempest Test Removal Procedure
==============================
Historically tempest was the only way of doing functional testing and
integration testing in OpenStack. This was mostly only an artifact of tempest
being the only proven pattern for doing this, not an artifact of a design
decision. However, moving forward as functional testing is being spun up in
each individual project we really only want tempest to be the integration test
suite it was intended to be; testing the high level interactions between
projects through REST API requests. In this model there are probably existing
tests that aren't the best fit living in tempest. However, since tempest is
largely still the only gating test suite in this space we can't carelessly rip
out everything from the tree. This document outlines the procedure which was
developed to ensure we minimize the risk for removing something of value from
the tempest tree.
This procedure might seem overly conservative and slow paced, but this is by
design to try and ensure we don't remove something that is actually providing
value. Having potential duplication between testing is not a big deal
especially compared to the alternative of removing something which is actually
providing value and is actively catching bugs, or blocking incorrect patches
from landing.
Proposing a test removal
------------------------
3 prong rule for removal
^^^^^^^^^^^^^^^^^^^^^^^^
In the proposal etherpad we'll be looking for answers to 3 questions
#. The tests proposed for removal must have equiv. coverage in a different
project's test suite (whether this is another gating test project, or an in
tree functional test suite). For API tests preferably the other project will
have a similar source of friction in place to prevent breaking api changes
so that we don't regress and let breaking api changes slip through the
gate.
#. The test proposed for removal has a failure rate < 0.50% in the gate over
the past release (the value and interval will likely be adjusted in the
future)
.. _`prong #3`:
#. There must not be an external user/consumer of tempest
that depends on the test proposed for removal
The answers to 1 and 2 are easy to verify. For 1 just provide a link to the new
test location. If you are linking to the tempest removal patch please also put
a Depends-On in the commit message for the commit which moved the test into
another repo.
For prong 2 you can use OpenStack-Health:
Using OpenStack-Health
""""""""""""""""""""""
Go to: http://status.openstack.org/openstack-health and then navigate to a per
test page for six months. You'll end up with a page that will graph the success
and failure rates on the bottom graph. For example, something like `this URL`_.
.. _this URL: http://status.openstack.org/openstack-health/#/test/tempest.scenario.test_volume_boot_pattern.TestVolumeBootPatternV2.test_volume_boot_pattern?groupKey=project&resolutionKey=day&duration=P6M
The Old Way using subunit2sql directly
""""""""""""""""""""""""""""""""""""""
SELECT * from tests where test_id like "%test_id%";
(where $test_id is the full test_id, but truncated to the class because of
setUpClass or tearDownClass failures)
You can access the infra mysql subunit2sql db w/ read-only permissions with:
* hostname: logstash.openstack.org
* username: query
* password: query
* db_name: subunit2sql
For example if you were trying to remove the test with the id:
tempest.api.compute.admin.test_flavors_negative.FlavorsAdminNegativeTestJSON.test_get_flavor_details_for_deleted_flavor
you would run the following:
#. run: "mysql -u query -p -h logstash.openstack.org subunit2sql" to connect
to the subunit2sql db
#. run the query: MySQL [subunit2sql]> select * from tests where test_id like
"tempest.api.compute.admin.test_flavors_negative.FlavorsAdminNegativeTestJSON%";
which will return a table of all the tests in the class (but it will also
catch failures in setUpClass and tearDownClass)
#. paste the output table with numbers and the mysql command you ran to
generate it into the etherpad.
Eventually a cli interface will be created to make that a bit more friendly.
Also a dashboard is in the works so we don't need to manually run the command.
The intent of the 2nd prong is to verify that moving the test into a project
specific testing is preventing bugs (assuming the tempest tests were catching
issues) from bubbling up a layer into tempest jobs. If we're seeing failure
rates above a certain threshold in the gate checks that means the functional
testing isn't really being effective in catching that bug (and therefore
blocking it from landing) and having the testing run in tempest still has
value.
However for the 3rd prong verification is a bit more subjective. The original
intent of this prong was mostly for refstack/defcore and also for things that
running on the stable branches. We don't want to remove any tests if that
would break our api consistency checking between releases, or something that
defcore/refstack is depending on being in tempest. It's worth pointing out
that if a test is used in defcore as part of interop testing then it will
probably have continuing value being in tempest as part of the
integration/integrated tests in general. This is one area where some overlap
is expected between testing in projects and tempest, which is not a bad thing.
Discussing the 3rd prong
""""""""""""""""""""""""
There are 2 approaches to addressing the 3rd prong. Either it can be raised
during a qa meeting during the tempest discussion. Please put it on the agenda
well ahead of the scheduled meeting. Since the meeting time will be well known
ahead of time anyone who depends on the tests will have ample time beforehand
to outline any concerns on the before the meeting. To give ample time for
people to respond to removal proposals please add things to the agenda by the
Monday before the meeting.
The other option is to raise the removal on the openstack-dev mailing list.
(for example see: http://lists.openstack.org/pipermail/openstack-dev/2016-February/086218.html )
This will raise the issue to the wider community and attract at least the same
(most likely more) attention than discussing it during the irc meeting. The
only downside is that it might take more time to get a response, given the
nature of ML.
Exceptions to this procedure
----------------------------
For the most part all tempest test removals have to go through this procedure
there are a couple of exceptions though:
#. The class of testing has been decided to be outside the scope of tempest.
#. A revert for a patch which added a broken test, or testing which didn't
actually run in the gate (basically any revert for something which
shouldn't have been added)
#. Tests that would become out of scope as a consequence of an API change,
as described in `API Compatibility`_.
Such tests cannot live in Tempest because of the branchless nature of
Tempest. Such test must still honor `prong #3`_.
For the first exception type the only types of testing in tree which have been
declared out of scope at this point are:
* The CLI tests (which should be completely removed at this point)
* Neutron Adv. Services testing (which should be completely removed at this
point)
* XML API Tests (which should be completely removed at this point)
* EC2 API/boto tests (which should be completely removed at this point)
For tests that fit into this category the only criteria for removal is that
there is equivalent testing elsewhere.
Tempest Scope
^^^^^^^^^^^^^
Starting in the liberty cycle tempest has defined a set of projects which
are defined as in scope for direct testing in tempest. As of today that list
is:
* Keystone
* Nova
* Glance
* Cinder
* Neutron
* Swift
anything that lives in tempest which doesn't test one of these projects can be
removed assuming there is equivalent testing elsewhere. Preferably using the
`tempest plugin mechanism`_
to maintain continuity after migrating the tests out of tempest.
.. _tempest plugin mechanism: http://docs.openstack.org/developer/tempest/plugin.html
API Compatibility
"""""""""""""""""
If an API introduces a non-discoverable, backward incompatible change, and
such change is not backported to all versions supported by Tempest, tests for
that API cannot live in Tempest anymore.
This is because tests would not be able to know or control which API response
to expect, and thus would not be able to enforce a specific behavior.
If a test exists in Tempest that would meet this criteria as consequence of a
change, the test must be removed according to the procedure discussed into
this document. The API change should not be merged until all conditions
required for test removal can be met.
|
