summaryrefslogtreecommitdiff
path: root/doc/topics/autodevops/upgrading_auto_deploy_dependencies.md
blob: c45390e935d3c91eb2508ba0b63aa3dc0c6f9d71 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
---
stage: Release
group: Release
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
type: reference
---

# Upgrading deployments for newer Auto Deploy dependencies (Auto Deploy template, auto-deploy-image and auto-deploy-app chart)

[Auto Deploy](stages.md#auto-deploy) is a feature that deploys your application to a Kubernetes cluster.
It consists of several dependencies:

- [Auto Deploy template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Deploy.gitlab-ci.yml) is a set of pipeline jobs and scripts that makes use of `auto-deploy-image`.
- [`auto-deploy-image`](https://gitlab.com/gitlab-org/cluster-integration/auto-deploy-image) is the executable image that communicates with the Kubernetes cluster.
- [`auto-deploy-app chart`](https://gitlab.com/gitlab-org/cluster-integration/auto-deploy-image/-/tree/master/assets/auto-deploy-app) is the Helm chart for deploying your application.

The `auto-deploy-image` and `auto-deploy-app` charts use [Semantic Versioning](https://semver.org/).
By default, your Auto DevOps project keeps using the stable and non-breaking version.
However, these dependencies could be upgraded in a major version release of GitLab
with breaking changes requiring you to upgrade your deployments.

This guide explains how to upgrade your deployments with newer or different major versions of Auto Deploy dependencies.

## Verify dependency versions

The process to check the current versions differs depending on which template you
are using. First verify which template is in use:

- For self-managed instances, the [stable Auto Deploy template bundled with the GitLab package](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Deploy.gitlab-ci.yml)
  is being used.
- [The GitLab.com stable Auto Deploy template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Deploy.gitlab-ci.yml)
  is being used if **one** of the following is true:
  - Your Auto DevOps project doesn't have a `.gitlab-ci.yml` file.
  - Your Auto DevOps project has a `.gitlab-ci.yml` and [includes](../../ci/yaml/README.md#includetemplate)
    the `Auto-DevOps.gitlab-ci.yml` template.
- [The latest Auto Deploy template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Deploy.latest.gitlab-ci.yml)
  is being used if **both** of the following is true:
  - Your Auto DevOps project has a `.gitlab-ci.yml` file and [includes](../../ci/yaml/README.md#includetemplate)
    the `Auto-DevOps.gitlab-ci.yml` template.
  - It also includes [the latest Auto Deploy template](#early-adopters)

If you know what template is being used:

- The `auto-deploy-image` version is in the template (for example `auto-deploy-image:v1.0.3`).
- The `auto-deploy-app` chart version is [in the auto-deploy-image repository](https://gitlab.com/gitlab-org/cluster-integration/auto-deploy-image/-/blob/v1.0.3/assets/auto-deploy-app/Chart.yaml)
  (for example `version: 1.0.3`).

## Compatibility

The following table explains the version compatibility between GitLab and Auto Deploy dependencies:

| GitLab version   | `auto-deploy-image` version | Notes |
|------------------|-----------------------------|-------|
| v10.0 to v14.0   | v0.1.0 to v2.0.0            | v0 and v1 auto-deploy-image are backwards compatible. |
| v13.4 and higher | v2.0.0 and higher           | v2 auto-deploy-image contains breaking changes, as explained in the [upgrade guide](#upgrade-deployments-to-the-v2-auto-deploy-image). |

You can find the current stable version of auto-deploy-image in the [Auto Deploy stable template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Deploy.gitlab-ci.yml).

## Upgrade Guide

Projects using Auto DevOps must use the unmodified chart managed by GitLab.
[Customized charts](customize.md#custom-helm-chart) are unsupported.

### Upgrade deployments to the v1 `auto-deploy-image`

The v1 chart is backward compatible with the v0 chart, so no configuration changes are needed.

### Upgrade deployments to the v2 `auto-deploy-image`

The v2 auto-deploy-image contains multiple dependency and architectural changes.
If your Auto DevOps project has an active environment deployed with the v1 `auto-deploy-image`,
please proceed with the following upgrade guide. Otherwise, you can skip this process.

#### Kubernetes 1.16+

The v2 auto-deploy-image drops support for Kubernetes 1.15 and lower. If you need to upgrade your
Kubernetes cluster, follow your cloud provider's instructions. Here's
[an example on GKE](https://cloud.google.com/kubernetes-engine/docs/how-to/upgrading-a-cluster).

#### Helm 3

> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/228609) in GitLab 13.4.

The `auto-deploy-image` uses the Helm binary to manipulate the releases.
Previously, `auto-deploy-image` used Helm v2, which used Tiller in a cluster.
In the v2 `auto-deploy-image`, it uses Helm v3 that doesn't require Tiller anymore.

If your Auto DevOps project has an active environment that was deployed with the v1
`auto-deploy-image`, use the following steps to upgrade to v2, which uses Helm 3:

1. Modify your `.gitlab-ci.yml` with:

   ```yaml
   include:
     - template: Auto-DevOps.gitlab-ci.yml
     - remote: https://gitlab.com/hfyngvason/ci-templates/-/raw/master/Helm-2to3.gitlab-ci.yml

   variables:
     # If this variable is not present, the migration jobs will not show up
     MIGRATE_HELM_2TO3: "true"

   .auto-deploy:
     image: registry.gitlab.com/gitlab-org/cluster-integration/auto-deploy-image:v2.0.0-beta.1
     variables:
       AUTO_DEVOPS_FORCE_DEPLOY_V2: 1
   ```

1. Run the `<environment-name>:helm-2to3:migrate` job.
1. Deploy your environment as usual. This deployment uses Helm 3.
1. If the deployment succeeds, you can safely run `environment:helm-2to3:cleanup`.
   This deletes all Helm 2 release data from the namespace.

   If you accidentally delete the Helm 2 releases before you are ready, the `<environment-name>:helm2to3:migrate`
   job saves a backup for 1 week in a job artifact called `helm-2-release-backups`.
   The backup is in a Kubernetes manifest file that can be restored using
   `kubectl apply -f $backup`.
1. Remove the `MIGRATE_HELM_2TO3` variable.

#### In-Cluster PostgreSQL Channel 2

The v2 auto-deploy-image drops support for [legacy in-cluster PostgreSQL](upgrading_postgresql.md).
If your Kubernetes cluster still depends on it, [upgrade and migrate your data](upgrading_postgresql.md)
with the [v1 auto-deploy-image](#use-a-specific-version-of-auto-deploy-dependencies).

#### Traffic routing change for canary deployments and incremental rollouts

> [Introduced](https://gitlab.com/gitlab-org/cluster-integration/auto-deploy-image/-/merge_requests/109) in GitLab 13.4.

Auto Deploy supports advanced deployment strategies such as [canary deployments](customize.md#deploy-policy-for-canary-environments)
and [incremental rollouts](../../ci/environments/incremental_rollouts.md).

Previously, `auto-deploy-image` created one service to balance the traffic between
unstable and stable tracks by changing the replica ratio. In the v2 `auto-deploy-image`,
it controls the traffic with [Canary Ingress](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#canary).

For more details, see the [v2 `auto-deploy-app` chart resource architecture](#v2-chart-resource-architecture).

If your Auto DevOps project has active `canary` or `rollout` track releases in the
`production` environment deployed with the v1 `auto-deploy-image`, use the following
steps to upgrade to v2:

1. Verify your project is [using the v1 `auto-deploy-image`](#verify-dependency-versions).
   If not, [specify the version](#use-a-specific-version-of-auto-deploy-dependencies).
1. If you're in the process of deploying `canary` or `rollout` deployments, promote
   them to `production` first to delete the unstable tracks.
1. Verify your project is [using the v2 `auto-deploy-image`](#verify-dependency-versions).
   If not, [specify the version](#use-a-specific-version-of-auto-deploy-dependencies).
1. Add an `AUTO_DEVOPS_FORCE_DEPLOY_V2` environment variable with a value of `true`
   in the GitLab CI/CD settings.
1. Create a new pipeline and run the `production` job to renew the resource architecture
   with the v2 `auto-deploy-app chart`.
1. Remove the `AUTO_DEVOPS_FORCE_DEPLOY_V2` environment variable.

### Use a specific version of Auto Deploy dependencies

To use a specifc version of Auto Deploy dependencies, specify the previous Auto Deploy
stable template that contains the [desired version of `auto-deploy-image` and `auto-deploy-app`](#verify-dependency-versions).

For example, if the template is bundled in GitLab v13.3, change your `.gitlab-ci.yml` to:

```yaml
include:
  - template: Auto-DevOps.gitlab-ci.yml
  - remote: https://gitlab.com/gitlab-org/gitlab/-/blob/v13.3.0-ee/lib/gitlab/ci/templates/Jobs/Deploy.gitlab-ci.yml
```

### Ignore warnings and continue deploying

If you are certain that the new chart version is safe to be deployed, you can add
the `AUTO_DEVOPS_FORCE_DEPLOY_V<major-version-number>` [environment variable](customize.md#build-and-deployment)
to force the deployment to continue.

For example, if you want to deploy the `v2.0.0` chart on a deployment that previously
used the `v0.17.0` chart, add `AUTO_DEVOPS_FORCE_DEPLOY_V2`.

## Early adopters

If you want to use the latest beta or unstable version of `auto-deploy-image`, include
the latest Auto Deploy template into your `.gitlab-ci.yml`:

```yaml
include:
  - template: Auto-DevOps.gitlab-ci.yml
  - remote: https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Deploy.latest.gitlab-ci.yml
```

WARNING:
Using a beta or unstable `auto-deploy-image` could cause unrecoverable damage to
your environments. Do not test it with important projects or environments.

The next stable template update is [planned for GitLab v14.0](https://gitlab.com/gitlab-org/gitlab/-/issues/232788).

## Resource Architectures of the `auto-deploy-app` chart

### v0 and v1 chart resource architecture

```mermaid
graph TD;
subgraph gl-managed-app
Z[Nginx Ingress]
end
Z[Nginx Ingress] --> A(Ingress);
Z[Nginx Ingress] --> B(Ingress);
subgraph stg namespace
B[Ingress] --> H(...);
end
subgraph prd namespace
A[Ingress] --> D(Service);
D[Service] --> E(Deployment:Pods:app:stable);
D[Service] --> F(Deployment:Pods:app:canary);
D[Service] --> I(Deployment:Pods:app:rollout);
E(Deployment:Pods:app:stable)---id1[(Pods:Postgres)]
F(Deployment:Pods:app:canary)---id1[(Pods:Postgres)]
I(Deployment:Pods:app:rollout)---id1[(Pods:Postgres)]
end
```

### v2 chart resource architecture

```mermaid
graph TD;
subgraph gl-managed-app
Z[Nginx Ingress]
end
Z[Nginx Ingress] --> A(Ingress);
Z[Nginx Ingress] --> B(Ingress);
Z[Nginx Ingress] --> |If canary is present or incremental rollout/|J(Canary Ingress);
subgraph stg namespace
B[Ingress] --> H(...);
end
subgraph prd namespace
subgraph stable track
A[Ingress] --> D[Service];
D[Service] --> E(Deployment:Pods:app:stable);
end
subgraph canary track
J(Canary Ingress) --> K[Service]
K[Service] --> F(Deployment:Pods:app:canary);
end
E(Deployment:Pods:app:stable)---id1[(Pods:Postgres)]
F(Deployment:Pods:app:canary)---id1[(Pods:Postgres)]
end
```

## Troubleshooting

### Major version mismatch warning

If deploying a chart that has a major version that is different from the previous one,
the new chart might not be correctly deployed. This could be due to an architectural
change. If that happens, the deployment job fails with a message similar to:

```plaintext
*************************************************************************************
                                   [WARNING]
Detected a major version difference between the the chart that is currently deploying (auto-deploy-app-v0.7.0), and the previously deployed chart (auto-deploy-app-v1.0.0).
A new major version might not be backward compatible with the current release (production). The deployment could fail or be stuck in an unrecoverable status.
...
```

To clear this error message and resume deployments, you must do one of the following:

- Manually [upgrade the chart version](#upgrade-guide).
- [Use a specific chart version](#use-a-specific-version-of-auto-deploy-dependencies).

### Error: `missing key "app.kubernetes.io/managed-by": must be set to "Helm"`

If your cluster has a deployment that was deployed with the v1 `auto-deploy-image`,
you might encounter the following error:

- `Error: rendered manifests contain a resource that already exists. Unable to continue with install: Secret "production-postgresql" in namespace "<project-name>-production" exists and cannot be imported into the current release: invalid ownership metadata; label validation error: missing key "app.kubernetes.io/managed-by": must be set to "Helm"; annotation validation error: missing key "meta.helm.sh/release-name": must be set to "production-postgresql"; annotation validation error: missing key "meta.helm.sh/release-namespace": must be set to "<project-name>-production"`

This is because the previous deployment was deployed with Helm2, which is not compatible with Helm3.
To resolve the problem, please follow the [upgrade guide](#upgrade-deployments-to-the-v2-auto-deploy-image).