1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
|
# Review Apps
Review Apps are automatically deployed by each pipeline, both in
[CE](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22010) and
[EE](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/6665).
## How does it work?
### CD/CD architecture diagram
<details>
<summary>Show mermaid source</summary>
<pre>
graph TD
B1 -.->|2. once gitlab:assets:compile is done,<br />triggers a CNG-mirror pipeline and wait for it to be done| A2
C1 -.->|2. once review-build-cng is done,<br />Helm deploys the Review App using the Cloud<br/>Native images built by the CNG-mirror pipeline| A3
subgraph gitlab-ce/ee `test` stage
A1[gitlab:assets:compile]
B1[review-build-cng] -->|1. wait for| A1
C1[review-deploy] -->|1. wait for| B1
D1[review-qa-smoke] -->|1. wait for| C1
D1[review-qa-smoke] -.->|2. once review-deploy is done| E1>gitlab-qa runs the smoke<br/>suite against the Review App]
end
subgraph CNG-mirror pipeline
A2>Cloud Native images are built];
end
subgraph GCP `gitlab-review-apps` project
A3>"Cloud Native images are deployed to the<br />`review-apps-ce` or `review-apps-ee` Kubernetes (GKE) cluster"];
end
</pre>
</details>
### Detailed explanation
1. On every [pipeline][gitlab-pipeline] during the `test` stage, the
[`review-build-cng`][review-build-cng] and
[`review-deploy`][review-deploy] jobs are automatically started.
- The [`review-deploy`][review-deploy] job waits for the
[`review-build-cng`][review-build-cng] job to finish.
- The [`review-build-cng`][review-build-cng] job waits for the
[`gitlab:assets:compile`][gitlab:assets:compile] job to finish since the
[`CNG-mirror`][cng-mirror] pipeline triggered in the following step depends on it.
1. Once the [`gitlab:assets:compile`][gitlab:assets:compile] job is done,
[`review-build-cng`][review-build-cng] [triggers a pipeline][cng-pipeline]
in the [`CNG-mirror`][cng-mirror] project.
- The [`CNG-mirror`][cng-pipeline] pipeline creates the Docker images of
each component (e.g. `gitlab-rails-ee`, `gitlab-shell`, `gitaly` etc.)
based on the commit from the [GitLab pipeline][gitlab-pipeline] and store
them in its [registry][cng-mirror-registry].
- We use the [`CNG-mirror`][cng-mirror] project so that the `CNG`, (**C**loud
**N**ative **G**itLab), project's registry is not overloaded with a
lot of transient Docker images.
1. Once the [`review-build-cng`][review-build-cng] job is done, the
[`review-deploy`][review-deploy] job deploys the Review App using
[the official GitLab Helm chart][helm-chart] to the
[`review-apps-ce`][review-apps-ce] / [`review-apps-ee`][review-apps-ee]
Kubernetes cluster on GCP.
- The actual scripts used to deploy the Review App can be found at
[`scripts/review_apps/review-apps.sh`][review-apps.sh].
- These scripts are basically
[our official Auto DevOps scripts][Auto-DevOps.gitlab-ci.yml] where the
default CNG images are overridden with the images built and stored in the
[`CNG-mirror` project's registry][cng-mirror-registry].
- Since we're using [the official GitLab Helm chart][helm-chart], this means
you get a dedicated environment for your branch that's very close to what
it would look in production.
1. Once the [`review-deploy`][review-deploy] job succeeds, you should be able to
use your Review App thanks to the direct link to it from the MR widget. The
default username is `root` and its password can be found in the 1Password
secure note named **gitlab-{ce,ee} Review App's root password**.
**Additional notes:**
- The Kubernetes cluster is connected to the `gitlab-{ce,ee}` projects using
[GitLab's Kubernetes integration][gitlab-k8s-integration]. This basically
allows to have a link to the Review App directly from the merge request
widget.
- If the Review App deployment fails, you can simply retry it (there's no need
to run the [`review-stop`][gitlab-ci-yml] job first).
- The manual [`review-stop`][gitlab-ci-yml] in the `test` stage can be used to
stop a Review App manually, and is also started by GitLab once a branch is
deleted.
- Review Apps are cleaned up regularly using a pipeline schedule that runs
the [`schedule:review-cleanup`][gitlab-ci-yml] job.
## QA runs
On every [pipeline][gitlab-pipeline] during the `test` stage, the
`review-qa-smoke` job is automatically started: it runs the smoke QA suite.
You can also manually start the `review-qa-all`: it runs the full QA suite.
Note that both jobs first wait for the `review-deploy` job to be finished.
## How to?
### Find my Review App slug?
1. Open the `review-deploy` job.
1. Look for `Checking for previous deployment of review-*`.
1. For instance for `Checking for previous deployment of review-qa-raise-e-12chm0`,
your Review App slug would be `review-qa-raise-e-12chm0` in this case.
### Run a Rails console?
1. [Filter Workloads by your Review App slug](https://console.cloud.google.com/kubernetes/workload?project=gitlab-review-apps)
, e.g. `review-29951-issu-id2qax`.
1. Find and open the `task-runner` Deployment, e.g. `review-29951-issu-id2qax-task-runner`.
1. Click on the Pod in the "Managed pods" section, e.g. `review-29951-issu-id2qax-task-runner-d5455cc8-2lsvz`.
1. Click on the `KUBECTL` dropdown, then `Exec` -> `task-runner`.
1. Replace `-c task-runner -- ls` with `-- /srv/gitlab/bin/rails c` from the
default command or
- Run `kubectl exec --namespace review-apps-ce -it review-29951-issu-id2qax-task-runner-d5455cc8-2lsvz -- /srv/gitlab/bin/rails c`
and
- Replace `review-apps-ce` with `review-apps-ee` if the Review App
is running EE, and
- Replace `review-29951-issu-id2qax-task-runner-d5455cc8-2lsvz`
with your Pod's name.
### Dig into a Pod's logs?
1. [Filter Workloads by your Review App slug](https://console.cloud.google.com/kubernetes/workload?project=gitlab-review-apps)
, e.g. `review-1979-1-mul-dnvlhv`.
1. Find and open the `migrations` Deployment, e.g.
`review-1979-1-mul-dnvlhv-migrations.1`.
1. Click on the Pod in the "Managed pods" section, e.g.
`review-1979-1-mul-dnvlhv-migrations.1-nqwtx`.
1. Click on the `Container logs` link.
## Frequently Asked Questions
**Isn't it too much to trigger CNG image builds on every test run? This creates
thousands of unused Docker images.**
> We have to start somewhere and improve later. Also, we're using the
CNG-mirror project to store these Docker images so that we can just wipe out
the registry at some point, and use a new fresh, empty one.
**How big are the Kubernetes clusters (`review-apps-ce` and `review-apps-ee`)?**
> The clusters are currently set up with a single pool of preemptible nodes,
with a minimum of 1 node and a maximum of 100 nodes.
**What are the machine running on the cluster?**
> We're currently using `n1-standard-4` (4 vCPUs, 15 GB memory) machines.
**How do we secure this from abuse? Apps are open to the world so we need to
find a way to limit it to only us.**
> This isn't enabled for forks.
[charts-1068]: https://gitlab.com/charts/gitlab/issues/1068
[gitlab-pipeline]: https://gitlab.com/gitlab-org/gitlab-ce/pipelines/44362587
[gitlab:assets:compile]: https://gitlab.com/gitlab-org/gitlab-ce/-/jobs/149511610
[review-build-cng]: https://gitlab.com/gitlab-org/gitlab-ce/-/jobs/149511623
[review-deploy]: https://gitlab.com/gitlab-org/gitlab-ce/-/jobs/149511624
[cng-mirror]: https://gitlab.com/gitlab-org/build/CNG-mirror
[cng-pipeline]: https://gitlab.com/gitlab-org/build/CNG-mirror/pipelines/44364657
[cng-mirror-registry]: https://gitlab.com/gitlab-org/build/CNG-mirror/container_registry
[helm-chart]: https://gitlab.com/charts/gitlab/
[review-apps-ce]: https://console.cloud.google.com/kubernetes/clusters/details/us-central1-a/review-apps-ce?project=gitlab-review-apps
[review-apps-ee]: https://console.cloud.google.com/kubernetes/clusters/details/us-central1-b/review-apps-ee?project=gitlab-review-apps
[review-apps.sh]: https://gitlab.com/gitlab-org/gitlab-ee/blob/master/scripts/review_apps/review-apps.sh
[automated_cleanup.rb]: https://gitlab.com/gitlab-org/gitlab-ee/blob/master/scripts/review_apps/automated_cleanup.rb
[Auto-DevOps.gitlab-ci.yml]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/lib/gitlab/ci/templates/Auto-DevOps.gitlab-ci.yml
[gitlab-ci-yml]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/.gitlab-ci.yml
[gitlab-k8s-integration]: https://docs.gitlab.com/ee/user/project/clusters/index.html
[password-bug]: https://gitlab.com/gitlab-org/gitlab-ce/issues/53621
---
[Return to Testing documentation](index.md)
|
