summaryrefslogtreecommitdiff
path: root/doc/migrate_ci_to_ce/README.md
blob: e12ea9a9ad75d4ebf4a9e0dc1bbdd7fb3dd0712d (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
## Migrate GitLab CI to GitLab CE/EE

## Notice

**You need to have working GitLab CI 7.14 to perform migration.
The older versions are not supported and will most likely break migration procedure.**

This migration can't be done online and takes significant amount of time.
Make sure to plan it ahead.

If you are running older version please follow the upgrade guide first:
https://gitlab.com/gitlab-org/gitlab-ci/blob/master/doc/update/7.13-to-7.14.md

The migration is divided into a two parts:
1. **[CI]** You will be making a changes to GitLab CI instance.
1. **[CE]** You will be making a changes to GitLab CE/EE instance.

### 1. Stop CI server [CI]

    sudo service gitlab_ci stop
    
### 2. Backup [CI]

**The migration procedure is database breaking.
You need to create backup if you still want to access GitLab CI in case of failure.**

```bash
cd /home/gitlab_ci/gitlab-ci
sudo -u gitlab_ci -H bundle exec backup:create RAILS_ENV=production
```

### 3. Prepare GitLab CI database to migration [CI]
    
Copy and paste the command in terminal to rename all tables.
This also breaks your database structure disallowing you to use it anymore.

    cat <<EOF | bundle exec rails dbconsole production
    ALTER TABLE application_settings RENAME TO ci_application_settings;
    ALTER TABLE builds RENAME TO ci_builds;
    ALTER TABLE commits RENAME TO ci_commits;
    ALTER TABLE events RENAME TO ci_events;
    ALTER TABLE jobs RENAME TO ci_jobs;
    ALTER TABLE projects RENAME TO ci_projects;
    ALTER TABLE runner_projects RENAME TO ci_runner_projects;
    ALTER TABLE runners RENAME TO ci_runners;
    ALTER TABLE services RENAME TO ci_services;
    ALTER TABLE tags RENAME TO ci_tags;
    ALTER TABLE taggings RENAME TO ci_taggings;
    ALTER TABLE trigger_requests RENAME TO ci_trigger_requests;
    ALTER TABLE triggers RENAME TO ci_triggers;
    ALTER TABLE variables RENAME TO ci_variables;
    ALTER TABLE web_hooks RENAME TO ci_web_hooks;
    EOF

### 4. Dump GitLab CI database [CI]

First check used database and credentials on GitLab CI and GitLab CE/EE:

1. To check it on GitLab CI:

    cat /home/gitlab_ci/gitlab-ci/config/database.yml
    
1. To check it on GitLab CE/EE:

    cat /home/git/gitlab/config/database.yml

Please first check the database engine used for GitLab CI and GitLab CE/EE.

1. If your GitLab CI uses **mysql2** and GitLab CE/EE uses it too.
Please follow **Dump MySQL** guide.

1. If your GitLab CI uses **postgres** and GitLab CE/EE uses **postgres**.
Please follow **Dump PostgreSQL** guide.

1. If your GitLab CI uses **mysql2** and GitLab CE/EE uses **postgres**.
Please follow **Dump MySQL and migrate to PostgreSQL** guide.

**Remember credentials stored for accessing GitLab CI.
You will need to put these credentials into commands executed below.**

    $ cat config/database.yml                                                                                                                                                                                                                        [10:06:55]
    #
    # PRODUCTION
    #
    production:
      adapter: postgresql or mysql2
      encoding: utf8
      reconnect: false
      database: GITLAB_CI_DATABASE
      pool: 5
      username: DB_USERNAME
      password: DB_PASSWORD
      host: DB_HOSTNAME
      port: DB_PORT
      # socket: /tmp/mysql.sock

#### a. Dump MySQL
 
    mysqldump --default-character-set=utf8 --complete-insert --no-create-info \
      --host=DB_USERNAME --port=DB_PORT --user=DB_HOSTNAME -p 
      GITLAB_CI_DATABASE \
      ci_application_settings ci_builds ci_commits ci_events ci_jobs ci_projects \
      ci_runner_projects ci_runners ci_services ci_tags ci_taggings ci_trigger_requests \
      ci_triggers ci_variables ci_web_hooks > gitlab_ci.sql
      
#### b. Dump PostgreSQL
  
    pg_dump -h DB_HOSTNAME -U DB_USERNAME -p DB_PORT --data-only GITLAB_CI_DATABASE -t "ci_*" > gitlab_ci.sql

#### c. Dump MySQL and migrate to PostgreSQL

    # Dump existing MySQL database first
    mysqldump --default-character-set=utf8 --compatible=postgresql --complete-insert \
      --host=DB_USERNAME --port=DB_PORT --user=DB_HOSTNAME -p 
      GITLAB_CI_DATABASE \
      ci_application_settings ci_builds ci_commits ci_events ci_jobs ci_projects \
      ci_runner_projects ci_runners ci_services ci_tags ci_taggings ci_trigger_requests \
      ci_triggers ci_variables ci_web_hooks > gitlab_ci.sql.tmp
      
    # Convert database to be compatible with PostgreSQL
    git clone https://github.com/gitlabhq/mysql-postgresql-converter.git -b gitlab
    python mysql-postgresql-converter/db_converter.py gitlab_ci.sql.tmp gitlab_ci.sql.tmp2
    ed -s gitlab_ci.sql.tmp2 < mysql-postgresql-converter/move_drop_indexes.ed
    
    # Filter to only include INSERT statements
    grep "^\(START\|SET\|INSERT\|COMMIT\)" gitlab_ci.sql.tmp2 > gitlab_ci.sql
    
### 5. Make sure that your GitLab CE/EE is 8.0 [CE]

Please verify that you use GitLab CE/EE 8.0.
If not, please follow the update guide: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/update/7.14-to-8.0.md

### 6. Stop GitLab CE/EE [CE]

Before you can migrate data you need to stop GitLab CE/EE first.

    sudo service gitlab stop
    
### 7. Backup GitLab CE/EE [CE]

This migration poses a **significant risk** of breaking your GitLab CE/EE. 
**You should create the GitLab CI/EE backup before doing it.**

    cd /home/git/gitlab
    sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production

### 8. Copy secret tokens [CE]

The `secrets.yml` file stores encryption keys for secure variables.

You need to copy the content of `config/secrets.yml` to the same file in GitLab CE.

    sudo cp /home/gitlab_ci/gitlab-ci/config/secrets.yml /home/git/gitlab/config/secrets.yml
    sudo chown git:git /home/git/gitlab/config/secrets.yml
    sudo chown 0600 /home/git/gitlab/config/secrets.yml
    
### 9. New configuration options for `gitlab.yml` [CE]

There are new configuration options available for [`gitlab.yml`](config/gitlab.yml.example).
View them with the command below and apply them manually to your current `gitlab.yml`:

```sh
git diff origin/7-14-stable:config/gitlab.yml.example origin/8-0-stable:config/gitlab.yml.example
```

The new options include configuration of GitLab CI that are now being part of GitLab CE and EE.

### 10. Copy build logs [CE]

You need to copy the contents of `builds/` to the same directory in GitLab CE/EE.

    sudo rsync -av /home/gitlab_ci/gitlab-ci/builds /home/git/gitlab/builds
    sudo chown -R git:git /home/git/gitlab/builds

The build traces are usually quite big so it will take a significant amount of time.

### 11. Import GitLab CI database [CE]

The one of the last steps is to import existing GitLab CI database.

    sudo mv /home/gitlab_ci/gitlab-ci/gitlab_ci.sql /home/git/gitlab/gitlab_ci.sql
    sudo chown git:git /home/git/gitlab/gitlab_ci.sql
    sudo -u git -H bundle exec rake ci:migrate CI_DUMP=/home/git/gitlab/gitlab_ci.sql RAILS_ENV=production

The task does:
1. Delete data from all existing CI tables
1. Import database data
1. Fix database auto increments
1. Fix tags assigned to Builds and Runners
1. Fix services used by CI

### 12. Start GitLab [CE]

You can start GitLab CI/EE now and see if everything is working.

    sudo service gitlab start

### 13. Update nginx [CI]
    
Now get back to GitLab CI and update **Nginx** configuration in order to:
1. Have all existing runners able to communicate with a migrated GitLab CI.
1. Have GitLab able send build triggers to CI address specified in Project's settings -> Services -> GitLab CI.

You need to edit `/etc/nginx/sites-available/gitlab_ci` and paste:
    
    # GITLAB CI
    server {
      listen 80 default_server;         # e.g., listen 192.168.1.1:80;
      server_name YOUR_CI_SERVER_FQDN;  # e.g., server_name source.example.com;
    
      access_log  /var/log/nginx/gitlab_ci_access.log;
      error_log   /var/log/nginx/gitlab_ci_error.log;
    
      # expose API to fix runners
      location /api {
        proxy_read_timeout    300;
        proxy_connect_timeout 300;
        proxy_redirect        off;
        proxy_set_header      X-Real-IP $remote_addr;
    
        # You need to specify your DNS servers that are able to resolve YOUR_GITLAB_SERVER_FQDN
        resolver 8.8.8.8 8.8.4.4;
        proxy_pass $scheme://YOUR_GITLAB_SERVER_FQDN/ci$request_uri;
      }
    
      # redirect all other CI requests
      location / {
        return 301 $scheme://YOUR_GITLAB_SERVER_FQDN/ci$request_uri;
      }
    
      # adjust this to match the largest build log your runners might submit,
      # set to 0 to disable limit
      client_max_body_size 10m;
    }

Make sure to fill the blanks to match your setup:
1. **YOUR_CI_SERVER_FQDN**: The existing public facing address of GitLab CI, eg. ci.gitlab.com.
1. **YOUR_GITLAB_SERVER_FQDN**: The public facing address of GitLab CE/EE, eg. gitlab.com.

**Make sure to not remove the `/ci$request_uri`. This is required to properly forward the requests.**

You should also make sure that you can do:
1. `curl https://YOUR_GITLAB_SERVER_FQDN/` from your previous GitLab CI server.
1. `curl https://YOUR_CI_SERVER_FQDN/` from your GitLab CE/EE server.

## Check your configuration

    sudo nginx -t

## Restart nginx

    sudo /etc/init.d/nginx restart

### 14. Done!

If everything went OK you should be able to access all your GitLab CI data by pointing your browser to:
https://gitlab.example.com/ci/.

The GitLab CI should also work when using the previous address, redirecting you to the GitLab CE/EE.

**Enjoy!**