1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
|
# Pipelines for the GitLab project
Pipelines for `gitlab-org/gitlab` and `gitlab-org/gitlab-foss` (as well as the
`dev` instance's mirrors) are configured in the usual
[`.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab-ci.yml)
which itself includes files under
[`.gitlab/ci/`](https://gitlab.com/gitlab-org/gitlab/tree/master/.gitlab/ci)
for easier maintenance.
We're striving to [dogfood](https://about.gitlab.com/handbook/engineering/#dogfooding)
GitLab [CI/CD features and best-practices](../ci/yaml/README.md)
as much as possible.
## Stages
The current stages are:
- `prepare`: This stage includes jobs that prepare artifacts that are needed by
jobs in subsequent stages.
- `quick-test`: This stage includes test jobs that should run first and fail the
pipeline early (currently used to run Geo tests when the branch name starts
with `geo-`, `geo/`, or ends with `-geo`).
- `test`: This stage includes most of the tests, DB/migration jobs, and static analysis jobs.
- `review-prepare`: This stage includes a job that build the CNG images that are
later used by the (Helm) Review App deployment (see
[Review Apps](testing_guide/review_apps.md) for details).
- `review`: This stage includes jobs that deploy the GitLab and Docs Review Apps.
- `qa`: This stage includes jobs that perform QA tasks against the Review App
that is deployed in the previous stage.
- `notification`: This stage includes jobs that sends notifications about pipeline status.
- `post-test`: This stage includes jobs that build reports or gather data from
the previous stages' jobs (e.g. coverage, Knapsack metadata etc.).
- `pages`: This stage includes a job that deploys the various reports as
GitLab pages (e.g. <https://gitlab-org.gitlab.io/gitlab/coverage-ruby/>,
<https://gitlab-org.gitlab.io/gitlab/coverage-javascript/>,
<https://gitlab-org.gitlab.io/gitlab/webpack-report/>).
## Default image
The default image is currently
`gitlab.com/gitlab-org/gitlab-build-images:ruby-2.6.3-golang-1.11-git-2.22-chrome-73.0-node-12.x-yarn-1.16-postgresql-9.6-graphicsmagick-1.3.33`.
It includes Ruby 2.6.3, Go 1.11, Git 2.22, Chrome 73, Node 12, Yarn 1.16,
PostgreSQL 9.6, and Graphics Magick 1.3.33.
The images used in our pipelines are configured in the
[`gitlab-org/gitlab-build-images`](https://gitlab.com/gitlab-org/gitlab-build-images)
project, which is push-mirrored to <https://dev.gitlab.org/gitlab/gitlab-build-images>
for redundancy.
The current version of the build images can be found in the
["Used by GitLab CE/EE section"](https://gitlab.com/gitlab-org/gitlab-build-images/blob/master/.gitlab-ci.yml).
## Default variables
In addition to the [predefined variables](../ci/variables/predefined_variables.md),
each pipeline includes the following [variables](../ci/variables/README.md):
- `RAILS_ENV: "test"`
- `NODE_ENV: "test"`
- `SIMPLECOV: "true"`
- `GIT_DEPTH: "50"`
- `GIT_SUBMODULE_STRATEGY: "none"`
- `GET_SOURCES_ATTEMPTS: "3"`
- `KNAPSACK_RSPEC_SUITE_REPORT_PATH: knapsack/${CI_PROJECT_NAME}/rspec_report-master.json`
- `FLAKY_RSPEC_SUITE_REPORT_PATH: rspec_flaky/report-suite.json`
- `BUILD_ASSETS_IMAGE: "false"`
- `ES_JAVA_OPTS: "-Xms256m -Xmx256m"`
- `ELASTIC_URL: "http://elastic:changeme@docker.elastic.co-elasticsearch-elasticsearch:9200"`
## Common job definitions
Most of the jobs [extend from a few CI definitions](../ci/yaml/README.md#extends)
that are scoped to a single
[configuration parameter](../ci/yaml/README.md#configuration-parameters).
These common definitions are:
- `.default-tags`: Ensures a job has the `gitlab-org` tag to ensure it's using
our dedicated runners.
- `.default-retry`: Allows a job to retry upon `unknown_failure`, `api_failure`,
`runner_system_failure`.
- `.default-before_script`: Allows a job to use a default `before_script` definition
suitable for Ruby/Rails tasks that may need a database running (e.g. tests).
- `.default-cache`: Allows a job to use a default `cache` definition suitable for
Ruby/Rails and frontend tasks.
- `.default-only`: Restricts the cases where a job is created. This currently
includes `master`, `/^[\d-]+-stable(-ee)?$/` (stable branches),
`/^\d+-\d+-auto-deploy-\d+$/` (security branches), `merge_requests`, `tags`.
Note that jobs won't be created for branches with this default configuration.
- `.only-review`: Only creates a job for the `gitlab-org` namespace and if
Kubernetes integration is available. Also, prevents a job from being created
for `master` and auto-deploy branches.
- `.only-review-schedules`: Same as `.only-review` but also restrict a job to
only run for [schedules](../user/project/pipelines/schedules.md).
- `.only-canonical-schedules`: Only creates a job for scheduled pipelines in
the `gitlab-org/gitlab` and `gitlab-org/gitlab-foss` projects
- `.use-pg9`: Allows a job to use the `postgres:9.6` and `redis:alpine` services.
- `.use-pg10`: Allows a job to use the `postgres:10.9` and `redis:alpine` services.
- `.use-pg9-ee`: Same as `.use-pg9` but also use the
`docker.elastic.co/elasticsearch/elasticsearch:5.6.12` services.
- `.use-pg10-ee`: Same as `.use-pg10` but also use the
`docker.elastic.co/elasticsearch/elasticsearch:5.6.12` services.
- `.only-ee`: Only creates a job for the `gitlab` project.
- `.only-ee-as-if-foss`: Same as `.only-ee` but simulate the FOSS project by
setting the `FOSS_ONLY='1'` environment variable.
## Changes detection
If a job extends from `.default-only` (and most of the jobs should), it can restrict
the cases where it should be created
[based on the changes](../ci/yaml/README.md#onlychangesexceptchanges)
from a commit or MR by extending from the following CI definitions:
- `.only-code-changes`: Allows a job to only be created upon code-related changes.
- `.only-qa-changes`: Allows a job to only be created upon QA-related changes.
- `.only-docs-changes`: Allows a job to only be created upon docs-related changes.
- `.only-code-qa-changes`: Allows a job to only be created upon code-related or QA-related changes.
- `.only-graphql-changes`: Allows a job to only be created upon graphql-related changes.
**See <https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab/ci/global.gitlab-ci.yml>
for the list of exact patterns.**
## Directed acyclic graph
We're using the [`needs:`](../ci/yaml/README.md#needs) keyword to
execute jobs out of order for the following jobs:
```mermaid
graph RL;
A[setup-test-env];
B["gitlab:assets:compile pull-push-cache<br/>(master only)"];
C[gitlab:assets:compile pull-cache];
D["cache gems<br/>(master and tags only)"];
E[review-build-cng];
F[build-qa-image];
G[review-deploy];
G2["schedule:review-deploy<br/>(master only)"];
H[karma];
I[jest];
J["compile-assets pull-push-cache<br/>(master only)"];
K[compile-assets pull-cache];
L[webpack-dev-server];
M[coverage];
N[pages];
O[static-analysis];
P["schedule:package-and-qa<br/>(master schedule only)"];
Q[package-and-qa];
R[package-and-qa-manual];
S["RSpec<br/>(e.g. rspec unit pg9)"]
T[retrieve-tests-metadata];
subgraph "`prepare` stage"
A
F
K
J
T
end
subgraph "`test` stage"
B --> |needs| A;
C --> |needs| A;
D --> |needs| A;
H -.-> |needs and depends on| A;
H -.-> |needs and depends on| K;
I -.-> |needs and depends on| A;
I -.-> |needs and depends on| K;
L -.-> |needs and depends on| A;
L -.-> |needs and depends on| K;
O -.-> |needs and depends on| A;
O -.-> |needs and depends on| K;
S -.-> |needs and depends on| A;
S -.-> |needs and depends on| K;
S -.-> |needs and depends on| T;
downtime_check --> |needs and depends on| A;
db:* --> |needs| A;
gitlab:setup --> |needs| A;
downtime_check --> |needs and depends on| A;
graphql-docs-verify --> |needs| A;
end
subgraph "`review-prepare` stage"
E --> |needs| C;
X["schedule:review-build-cng<br/>(master schedule only)"] --> |needs| C;
end
subgraph "`review` stage"
G --> |needs| E;
G2 --> |needs| E;
end
subgraph "`qa` stage"
Q --> |needs| C;
Q --> |needs| F;
R --> |needs| C;
R --> |needs| F;
P --> |needs| C;
P --> |needs| F;
review-qa-smoke -.-> |needs and depends on| G;
review-qa-all -.-> |needs and depends on| G;
review-performance -.-> |needs and depends on| G;
X2["schedule:review-performance<br/>(master only)"] -.-> |needs and depends on| G2;
dast -.-> |needs and depends on| G;
end
subgraph "`notification` stage"
NOTIFICATION1["schedule:package-and-qa:notify-success<br>(on_success)"] -.-> |needs| P;
NOTIFICATION2["schedule:package-and-qa:notify-failure<br>(on_failure)"] -.-> |needs| P;
end
subgraph "`post-test` stage"
M
end
subgraph "`pages` stage"
N -.-> |depends on| C;
N -.-> |depends on| H;
N -.-> |depends on| M;
end
```
## Test jobs
Consult [GitLab tests in the Continuous Integration (CI) context](testing_guide/ci.md)
for more information.
## Review app jobs
Consult the [Review Apps](testing_guide/review_apps.md) dedicated page for more information.
---
[Return to Development documentation](README.md)
|
