1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
|
# End-to-end Testing
## What is end-to-end testing?
End-to-end testing is a strategy used to check whether your application works
as expected across the entire software stack and architecture, including
integration of all micro-services and components that are supposed to work
together.
## Branch naming
If your contribution contains **only** changes under the
[`qa/` folder](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/qa), you can
speed up the CI process by following some branch naming conventions. You have
three choices:
| Branch name | Valid example |
|:----------------------|:-----------------------------|
| Starting with `qa/` | `qa/new-oauth-login-test` |
| Starting with `qa-` | `qa-new-oauth-login-test` |
| Ending in `-qa` | `123-new-oauth-login-test-qa` |
If your branch name matches any of the above, it will run only the QA-related
jobs.
If it does not, the whole application test suite will run (including QA-related
jobs).
## How do we test GitLab?
We use [Omnibus GitLab][omnibus-gitlab] to build GitLab packages and then we
test these packages using the [GitLab QA orchestrator][gitlab-qa] tool, which is
a black-box testing framework for the API and the UI.
### Testing nightly builds
We run scheduled pipeline each night to test nightly builds created by Omnibus.
You can find these nightly pipelines at [gitlab-org/quality/nightly/pipelines][quality-nightly-pipelines].
Results are reported in the `#qa-nightly` Slack channel.
### Testing staging
We run scheduled pipeline each night to test staging.
You can find these nightly pipelines at [gitlab-org/quality/staging/pipelines][quality-staging-pipelines].
Results are reported in the `#qa-staging` Slack channel.
### Testing code in merge requests
#### Using the `package-and-qa-manual` job
It is possible to run end-to-end tests for a merge request, eventually being run in
a pipeline in the [`gitlab-qa`](https://gitlab.com/gitlab-org/gitlab-qa/) project,
by triggering the `package-and-qa-manual` manual action in the `test` stage (not
available for forks).
**This runs end-to-end tests against a custom Omnibus package built from your
merge request's changes.**
Manual action that starts end-to-end tests is also available in merge requests
in [Omnibus GitLab][omnibus-gitlab].
Below you can read more about how to use it and how does it work.
#### How does it work?
Currently, we are using _multi-project pipeline_-like approach to run QA
pipelines.
```mermaid
graph LR
A1 -.->|1. Triggers an omnibus-gitlab pipeline and wait for it to be done| A2
B2[`Trigger-qa` stage<br>`Trigger:qa-test` job] -.->|2. Triggers a gitlab-qa pipeline and wait for it to be done| A3
subgraph "gitlab-ce/ee pipeline"
A1[`test` stage<br>`package-and-qa-manual` job]
end
subgraph "omnibus-gitlab pipeline"
A2[`Trigger-docker` stage<br>`Trigger:gitlab-docker` job] -->|once done| B2
end
subgraph "gitlab-qa pipeline"
A3>QA jobs run] -.->|3. Reports back the pipeline result to the `package-and-qa-manual` job<br>and post the result on the original commit tested| A1
end
```
1. Developer triggers a manual action, that can be found in CE / EE merge
requests. This starts a chain of pipelines in multiple projects.
1. The script being executed triggers a pipeline in [Omnibus GitLab][omnibus-gitlab]
and waits for the resulting status. We call this a _status attribution_.
1. GitLab packages are being built in the [Omnibus GitLab][omnibus-gitlab]
pipeline. Packages are then pushed to its Container Registry.
1. When packages are ready, and available in the registry, a final step in the
[Omnibus GitLab][omnibus-gitlab] pipeline, triggers a new
GitLab QA pipeline (those with access can view them at `https://gitlab.com/gitlab-org/gitlab-qa/pipelines`). It also waits for a resulting status.
1. GitLab QA pulls images from the registry, spins-up containers and runs tests
against a test environment that has been just orchestrated by the `gitlab-qa`
tool.
1. The result of the GitLab QA pipeline is being
propagated upstream, through Omnibus, back to the CE / EE merge request.
#### Using the `review-qa-all` jobs
On every pipeline during the `test` stage, the `review-qa-smoke` job is
automatically started: it runs the QA smoke suite against the
[Review App][review-apps].
You can also manually start the `review-qa-all`: it runs the full QA suite
against the [Review App][review-apps].
**This runs end-to-end tests against a Review App based on [the official GitLab
Helm chart][helm-chart], itself deployed with custom
[Cloud Native components][cng] built from your merge request's changes.**
See [Review Apps][review-apps] for more details about Review Apps.
[helm-chart]: https://gitlab.com/gitlab-org/charts/gitlab/
[cng]: https://gitlab.com/gitlab-org/build/CNG
## How do I run the tests?
There are two main options for running the tests. If you simply want to run the
existing tests against a live GitLab instance or against a pre-built docker image
you can use the [GitLab QA orchestrator][gitlab-qa-readme]. See also [examples
of the test scenarios you can run via the orchestrator](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/what_tests_can_be_run.md#examples).
On the other hand, if you would like to run against a local development GitLab
environment, you can use the [GitLab Development Kit (GDK)](https://gitlab.com/gitlab-org/gitlab-development-kit/).
Please refer to the instructions in the [QA README](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/qa/README.md#how-can-i-use-it)
and the section below.
## How do I write tests?
In order to write new tests, you first need to learn more about GitLab QA
architecture. See the [documentation about it][gitlab-qa-architecture].
Once you decided where to put [test environment orchestration scenarios] and
[instance-level scenarios], take a look at the [GitLab QA README][instance-qa-readme],
the [GitLab QA orchestrator README][gitlab-qa-readme], and [the already existing
instance-level scenarios][instance-level scenarios].
Continued reading:
- [Quick Start Guide](quick_start_guide.md)
- [Style Guide](style_guide.md)
- [Best Practices](best_practices.md)
## Where can I ask for help?
You can ask question in the `#quality` channel on Slack (GitLab internal) or
you can find an issue you would like to work on in
[the `gitlab-ce` issue tracker][gitlab-ce-issues],
[the `gitlab-ee` issue tracker][gitlab-ce-issues], or
[the `gitlab-qa` issue tracker][gitlab-qa-issues].
[omnibus-gitlab]: https://gitlab.com/gitlab-org/omnibus-gitlab
[gitlab-qa]: https://gitlab.com/gitlab-org/gitlab-qa
[gitlab-qa-readme]: https://gitlab.com/gitlab-org/gitlab-qa/tree/master/README.md
[quality-nightly-pipelines]: https://gitlab.com/gitlab-org/quality/nightly/pipelines
[quality-staging-pipelines]: https://gitlab.com/gitlab-org/quality/staging/pipelines
[review-apps]: ../review_apps.md
[gitlab-qa-architecture]: https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/architecture.md
[gitlab-qa-issues]: https://gitlab.com/gitlab-org/gitlab-qa/issues?label_name%5B%5D=new+scenario
[gitlab-ce-issues]: https://gitlab.com/gitlab-org/gitlab-ce/issues?label_name[]=QA&label_name[]=test
[gitlab-ee-issues]: https://gitlab.com/gitlab-org/gitlab-ee/issues?label_name[]=QA&label_name[]=test
[test environment orchestration scenarios]: https://gitlab.com/gitlab-org/gitlab-qa/tree/master/lib/gitlab/qa/scenario
[instance-level scenarios]: https://gitlab.com/gitlab-org/gitlab-ce/tree/master/qa/qa/specs/features
[Page objects documentation]: https://gitlab.com/gitlab-org/gitlab-ce/tree/master/qa/qa/page/README.md
[instance-qa-readme]: https://gitlab.com/gitlab-org/gitlab-ce/tree/master/qa/README.md
[instance-qa-examples]: https://gitlab.com/gitlab-org/gitlab-ce/tree/master/qa/qa
|
