Merge branch 'rs-ci-migration-guide' into 'master'

Reformat and copy edit the CI-to-CE migration guide

[ci skip]

See merge request !1335

1 files changed, 264 insertions, 220 deletions

diff --git a/doc/migrate_ci_to_ce/README.md b/doc/migrate_ci_to_ce/README.md
index 1e45f29dbb2..c5eb4534c70 100644
--- a/doc/migrate_ci_to_ce/README.md
+++ b/doc/migrate_ci_to_ce/README.md
@@ -1,280 +1,324 @@
-## Migrate GitLab CI to GitLab CE/EE
+## Migrate GitLab CI to GitLab CE or EE
 
-## Notice
+Beginning with version 8.0 of GitLab Community Edition (CE) and Enterprise
+Edition (EE), GitLab CI is no longer its own application, but is instead built
+into the CE and EE applications.
 
-**You need to have working GitLab CI 7.14 to perform migration.
-The older versions are not supported and will most likely break migration procedure.**
+This guide will detail the process of migrating your CI installation and data
+into your GitLab CE or EE installation.
 
-This migration can't be done online and takes significant amount of time.
-Make sure to plan it ahead.
+### Before we begin
 
-If you are running older version please follow the upgrade guide first:
-https://gitlab.com/gitlab-org/gitlab-ci/blob/master/doc/update/7.13-to-7.14.md
+**You need to have a working installation of GitLab CI version 8.0 to perform
+this migration. The older versions are not supported and will most likely break
+this migration procedure.**
 
-The migration is divided into a two parts:
-1. **[CI]** You will be making a changes to GitLab CI instance.
-1. **[CE]** You will be making a changes to GitLab CE/EE instance.
+This migration cannot be performed online and takes a significant amount of
+time. Make sure to plan ahead.
 
-### 1. Stop CI server [CI]
+If you are running a version of GitLab CI prior to 8.0 please follow the
+appropriate [update guide](https://gitlab.com/gitlab-org/gitlab-ci/tree/master/doc/update/)
+before proceeding.
 
-    sudo service gitlab_ci stop
-    
-### 2. Backup [CI]
+The migration is divided into four parts and covers both manual and Omnibus
+installations:
 
-**The migration procedure is database breaking.
-You need to create backup if you still want to access GitLab CI in case of failure.**
+1. [GitLab CI](#part-i-gitlab-ci)
+1. [Gitlab CE (or EE)](#part-ii-gitlab-ce-or-ee)
+1. [Nginx configuration](#part-iii-nginx-configuration)
+1. [Finishing Up](#part-iv-finishing-up)
 
-```bash
-cd /home/gitlab_ci/gitlab-ci
-sudo -u gitlab_ci -H bundle exec backup:create RAILS_ENV=production
-```
+### Part I: GitLab CI
 
-### 3. Prepare GitLab CI database to migration [CI]
-    
-Copy and paste the command in terminal to rename all tables.
-This also breaks your database structure disallowing you to use it anymore.
-
-    cat <<EOF | bundle exec rails dbconsole production
-    ALTER TABLE application_settings RENAME TO ci_application_settings;
-    ALTER TABLE builds RENAME TO ci_builds;
-    ALTER TABLE commits RENAME TO ci_commits;
-    ALTER TABLE events RENAME TO ci_events;
-    ALTER TABLE jobs RENAME TO ci_jobs;
-    ALTER TABLE projects RENAME TO ci_projects;
-    ALTER TABLE runner_projects RENAME TO ci_runner_projects;
-    ALTER TABLE runners RENAME TO ci_runners;
-    ALTER TABLE services RENAME TO ci_services;
-    ALTER TABLE tags RENAME TO ci_tags;
-    ALTER TABLE taggings RENAME TO ci_taggings;
-    ALTER TABLE trigger_requests RENAME TO ci_trigger_requests;
-    ALTER TABLE triggers RENAME TO ci_triggers;
-    ALTER TABLE variables RENAME TO ci_variables;
-    ALTER TABLE web_hooks RENAME TO ci_web_hooks;
-    EOF
-
-### 4. Remove CI cronjob
+#### 1. Stop GitLab CI
 
-```
-cd /home/gitlab_ci/gitlab-ci
-sudo -u gitlab_ci -H bundle exec whenever --clear-crontab
-```
+    # Manual installation
+    sudo service gitlab_ci stop
 
-### 5. Dump GitLab CI database [CI]
+    # Omnibus installation
+    sudo gitlab-ctl stop ci-unicorn ci-sidekiq
 
-First check used database and credentials on GitLab CI and GitLab CE/EE:
+#### 2. Create a backup
 
-1. To check it on GitLab CI:
+The migration procedure modifies the structure of the CI database. If something
+goes wrong, you will not be able to revert to a previous version without a
+backup.
 
-    cat /home/gitlab_ci/gitlab-ci/config/database.yml
-    
-1. To check it on GitLab CE/EE:
+If your GitLab CI installation uses **MySQL** and your GitLab CE (or EE)
+installation uses **PostgreSQL** you'll need to convert the CI database by
+setting a `MYSQL_TO_POSTGRESQL` flag.
 
+If you use the Omnibus package you most likely use **PostgreSQL** on both GitLab
+CE (or EE) and CI.
+
+You can check which database each install is using by viewing their
+database configuration files:
+
+    cat /home/gitlab_ci/gitlab-ci/config/database.yml
     cat /home/git/gitlab/config/database.yml
 
-Please first check the database engine used for GitLab CI and GitLab CE/EE.
-
-1. If your GitLab CI uses **mysql2** and GitLab CE/EE uses it too.
-Please follow **Dump MySQL** guide.
-
-1. If your GitLab CI uses **postgres** and GitLab CE/EE uses **postgres**.
-Please follow **Dump PostgreSQL** guide.
-
-1. If your GitLab CI uses **mysql2** and GitLab CE/EE uses **postgres**.
-Please follow **Dump MySQL and migrate to PostgreSQL** guide.
-
-**Remember credentials stored for accessing GitLab CI.
-You will need to put these credentials into commands executed below.**
-
-    $ cat config/database.yml                                                                                                                                                                                                                        [10:06:55]
-    #
-    # PRODUCTION
-    #
-    production:
-      adapter: postgresql or mysql2
-      encoding: utf8
-      reconnect: false
-      database: GITLAB_CI_DATABASE
-      pool: 5
-      username: DB_USERNAME
-      password: DB_PASSWORD
-      host: DB_HOSTNAME
-      port: DB_PORT
-      # socket: /tmp/mysql.sock
-
-#### a. Dump MySQL
- 
-    mysqldump --default-character-set=utf8 --complete-insert --no-create-info \
-      --host=DB_USERNAME --port=DB_PORT --user=DB_HOSTNAME -p 
-      GITLAB_CI_DATABASE \
-      ci_application_settings ci_builds ci_commits ci_events ci_jobs ci_projects \
-      ci_runner_projects ci_runners ci_services ci_tags ci_taggings ci_trigger_requests \
-      ci_triggers ci_variables ci_web_hooks > gitlab_ci.sql
-      
-#### b. Dump PostgreSQL
-  
-    pg_dump -h DB_HOSTNAME -U DB_USERNAME -p DB_PORT --data-only GITLAB_CI_DATABASE -t "ci_*" > gitlab_ci.sql
-
-#### c. Dump MySQL and migrate to PostgreSQL
-
-    # Dump existing MySQL database first
-    mysqldump --default-character-set=utf8 --compatible=postgresql --complete-insert \
-      --host=DB_USERNAME --port=DB_PORT --user=DB_HOSTNAME -p 
-      GITLAB_CI_DATABASE \
-      ci_application_settings ci_builds ci_commits ci_events ci_jobs ci_projects \
-      ci_runner_projects ci_runners ci_services ci_tags ci_taggings ci_trigger_requests \
-      ci_triggers ci_variables ci_web_hooks > gitlab_ci.sql.tmp
-      
-    # Convert database to be compatible with PostgreSQL
-    git clone https://github.com/gitlabhq/mysql-postgresql-converter.git -b gitlab
-    python mysql-postgresql-converter/db_converter.py gitlab_ci.sql.tmp gitlab_ci.sql.tmp2
-    ed -s gitlab_ci.sql.tmp2 < mysql-postgresql-converter/move_drop_indexes.ed
-    
-    # Filter to only include INSERT statements
-    grep "^\(START\|SET\|INSERT\|COMMIT\)" gitlab_ci.sql.tmp2 > gitlab_ci.sql
-    
-### 6. Make sure that your GitLab CE/EE is 8.0 [CE]
-
-Please verify that you use GitLab CE/EE 8.0.
-If not, please follow the update guide: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/update/7.14-to-8.0.md
-
-### 7. Stop GitLab CE/EE [CE]
-
-Before you can migrate data you need to stop GitLab CE/EE first.
+- If both applications use the same database `adapter`, create the backup with
+  this command:
+
+        # Manual installation
+        cd /home/gitlab_ci/gitlab-ci
+        sudo -u gitlab_ci -H bundle exec backup:create RAILS_ENV=production
+
+        # Omnibus installation
+        sudo gitlab-ci-rake backup:create
+
+- If CI uses MySQL, and CE (or EE) uses PostgreSQL, create the backup with this
+  command (note the `MYSQL_TO_POSTGRESQL` flag):
+
+        # Manual installation
+        cd /home/gitlab_ci/gitlab-ci
+        sudo -u gitlab_ci -H bundle exec backup:create RAILS_ENV=production MYSQL_TO_POSTGRESQL=1
 
+        # Omnibus installation
+        sudo gitlab-ci-rake backup:create MYSQL_TO_POSTGRESQL=1
+
+#### 3. Remove cronjob
+
+**Note:** This step is only required for **manual installations**. Omnibus users
+can [skip to the next step](#part-ii-gitlab-ce-or-ee).
+
+    # Manual installation
+    cd /home/gitlab_ci/gitlab-ci
+    sudo -u gitlab_ci -H bundle exec whenever --clear-crontab
+
+### Part II: GitLab CE (or EE)
+
+#### 1. Ensure GitLab is updated
+
+Your GitLab CE or EE installation **must be version 8.0**. If it's not, follow
+the [update guide](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/update/7.14-to-8.0.md)
+before proceeding.
+
+If you use the Omnibus packages simply run `apt-get upgrade` to install the
+latest version.
+
+#### 2. Prevent CI usage during the migration process
+
+As an administrator, go to **Admin Area** -> **Settings**, and under **Continuous
+Integration** uncheck **Disable to prevent CI usage until rake ci:migrate is run
+(8.0 only)**.
+
+This will disable the CI integration and prevent users from creating CI projects
+until the migration process is completed.
+
+#### 3. Stop GitLab
+
+Before you can migrate data you need to stop the GitLab service first:
+
+    # Manual installation
     sudo service gitlab stop
-    
-### 8. Backup GitLab CE/EE [CE]
 
-This migration poses a **significant risk** of breaking your GitLab CE/EE. 
-**You should create the GitLab CI/EE backup before doing it.**
+    # Omnibus installation
+    sudo gitlab-ctl stop unicorn sidekiq
+
+#### 4. Create a backup
 
+This migration poses a **significant risk** of breaking your GitLab
+installation. Create a backup before proceeding:
+
+    # Manual installation
     cd /home/git/gitlab
     sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production
 
-### 9. Copy secret tokens [CE]
+    # Omnibus installation
+    sudo gitlab-rake gitlab:backup:create
+
+It's possible to speed up backup creation by skipping repositories and uploads:
+
+    # Manual installation
+    cd /home/git/gitlab
+    sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production SKIP=repositories,uploads
+
+    # Omnibus installation
+    sudo gitlab-rake gitlab:backup:create SKIP=repositories,uploads
+
+#### 5. Copy secret tokens from CI
 
 The `secrets.yml` file stores encryption keys for secure variables.
 
-You need to copy the content of `config/secrets.yml` to the same file in GitLab CE.
+- **Manual installations** need to copy the contents of GitLab CI's
+  `config/secrets.yml` file to the same file in GitLab CE:
 
+    ```bash
+    # Manual installation
     sudo cp /home/gitlab_ci/gitlab-ci/config/secrets.yml /home/git/gitlab/config/secrets.yml
     sudo chown git:git /home/git/gitlab/config/secrets.yml
     sudo chown 0600 /home/git/gitlab/config/secrets.yml
-    
-### 10. New configuration options for `gitlab.yml` [CE]
+    ```
+
+- **Omnibus installations** where GitLab CI and CE (or EE) are on the same
+  server don't need to do anything further, because the secrets are stored in
+  `/etc/gitlab/gitlab-secrets.json`.
 
-There are new configuration options available for [`gitlab.yml`](config/gitlab.yml.example).
-View them with the command below and apply them manually to your current `gitlab.yml`:
+- **Omnibus installations** where GitLab CI is on a different server than CE (or
+  EE) will need to:
+    1. On the CI server, copy the `db_key_base` value from
+       `/etc/gitlab/gitlab-secrets.json`
+    1. On the CE (or EE) server, add `gitlab_ci['db_key_base'] =
+       "VALUE_FROM_ABOVE"` to the `/etc/gitlab/gitlab.rb` file and run `sudo
+       gitlab-ctl reconfigure`
 
-```sh
-git diff origin/7-14-stable:config/gitlab.yml.example origin/8-0-stable:config/gitlab.yml.example
+#### 6. New configuration options for `gitlab.yml`
+
+**Note:** This step is only required for **manual installations**. Omnibus users
+can [skip to the next step](#7-copy-backup-from-gitlab-ci).
+
+There are new configuration options available for `gitlab.yml`. View them with
+the command below and apply them manually to your current `gitlab.yml`:
+
+    git diff origin/7-14-stable:config/gitlab.yml.example origin/8-0-stable:config/gitlab.yml.example
+
+The new options include configuration settings for GitLab CI.
+
+#### 7. Copy backup from GitLab CI
+
+```bash
+# Manual installation
+sudo cp -v /home/gitlab_ci/gitlab-ci/tmp/backups/*_gitlab_ci_backup.tar /home/git/gitlab/tmp/backups
+sudo chown git:git /home/git/gitlab/tmp/backups/*_gitlab_ci_backup.tar
+
+# Omnibus installation
+sudo cp -v /var/opt/gitlab/ci-backups/*_gitlab_ci_backup.tar /var/opt/gitlab/backups/
+sudo chown git:git /var/opt/gitlab/backups/*_gitlab_ci_backup.tar
 ```
 
-The new options include configuration of GitLab CI that are now being part of GitLab CE and EE.
+If moving across the servers you can use `scp`.
+However, this requires you to provide an authorized key or password to login to
+the GitLab CE (or EE) server from the CI server. You can try to use ssh-agent
+from your local machine to have that: login to your GitLab CI server using
+`ssh -A`.
 
-### 11. Copy build logs [CE]
+```bash
+# Manual installation
+scp /home/gitlab_ci/gitlab-ci/tmp/backups/*_gitlab_ci_backup.tar root@gitlab.example.com:/home/git/gitlab/tmp/backup
 
-You need to copy the contents of `builds/` to the same directory in GitLab CE/EE.
+# Omnibus installation
+scp /var/opt/gitlab/ci-backups/*_gitlab_ci_backup.tar root@gitlab.example.com:/var/opt/gitlab/backups/
+```
 
-    sudo rsync -av /home/gitlab_ci/gitlab-ci/builds /home/git/gitlab/builds
-    sudo chown -R git:git /home/git/gitlab/builds
+#### 8. Import GitLab CI backup
 
-The build traces are usually quite big so it will take a significant amount of time.
+Now you'll import the GitLab CI database dump that you created earlier into the
+GitLab CE or EE database:
 
-### 12. Import GitLab CI database [CE]
+    # Manual installation
+    sudo -u git -H bundle exec rake ci:migrate RAILS_ENV=production
 
-The one of the last steps is to import existing GitLab CI database.
+    # Omnibus installation
+    sudo gitlab-rake ci:migrate
 
-    sudo mv /home/gitlab_ci/gitlab-ci/gitlab_ci.sql /home/git/gitlab/gitlab_ci.sql
-    sudo chown git:git /home/git/gitlab/gitlab_ci.sql
-    sudo -u git -H bundle exec rake ci:migrate CI_DUMP=/home/git/gitlab/gitlab_ci.sql RAILS_ENV=production
+This task will take some time.
 
-The task does:
-1. Delete data from all existing CI tables
-1. Import database data
-1. Fix database auto increments
-1. Fix tags assigned to Builds and Runners
-1. Fix services used by CI
+This migration task automatically re-enables the CI setting that you
+[disabled earlier](#2-prevent-ci-usage-during-the-migration-process).
 
-### 13. Start GitLab [CE]
+#### 9. Start GitLab
 
-You can start GitLab CI/EE now and see if everything is working.
+You can start GitLab CE (or EE) now and see if everything is working:
 
+    # Manual installation
     sudo service gitlab start
 
-### 14. Update nginx [CI]
-    
-Now get back to GitLab CI and update **Nginx** configuration in order to:
-1. Have all existing runners able to communicate with a migrated GitLab CI.
-1. Have GitLab able send build triggers to CI address specified in Project's settings -> Services -> GitLab CI.
-
-You need to edit `/etc/nginx/sites-available/gitlab_ci` and paste:
-    
-    # GITLAB CI
-    server {
-      listen 80 default_server;         # e.g., listen 192.168.1.1:80;
-      server_name YOUR_CI_SERVER_FQDN;  # e.g., server_name source.example.com;
-    
-      access_log  /var/log/nginx/gitlab_ci_access.log;
-      error_log   /var/log/nginx/gitlab_ci_error.log;
-    
-      # expose API to fix runners
-      location /api {
-        proxy_read_timeout    300;
-        proxy_connect_timeout 300;
-        proxy_redirect        off;
-        proxy_set_header      X-Real-IP $remote_addr;
-    
-        # You need to specify your DNS servers that are able to resolve YOUR_GITLAB_SERVER_FQDN
-        resolver 8.8.8.8 8.8.4.4;
-        proxy_pass $scheme://YOUR_GITLAB_SERVER_FQDN/ci$request_uri;
-      }
-      
-      # expose build endpoint to allow trigger builds
-      location ~ ^/projects/\d+/build$ {
-        proxy_read_timeout    300;
-        proxy_connect_timeout 300;
-        proxy_redirect        off;
-        proxy_set_header      X-Real-IP $remote_addr;
-    
-        # You need to specify your DNS servers that are able to resolve YOUR_GITLAB_SERVER_FQDN
-        resolver 8.8.8.8 8.8.4.4;
-        proxy_pass $scheme://YOUR_GITLAB_SERVER_FQDN/ci$request_uri;
-      }
-    
-      # redirect all other CI requests
-      location / {
-        return 301 $scheme://YOUR_GITLAB_SERVER_FQDN/ci$request_uri;
-      }
-    
-      # adjust this to match the largest build log your runners might submit,
-      # set to 0 to disable limit
-      client_max_body_size 10m;
-    }
-
-Make sure to fill the blanks to match your setup:
-1. **YOUR_CI_SERVER_FQDN**: The existing public facing address of GitLab CI, eg. ci.gitlab.com.
-1. **YOUR_GITLAB_SERVER_FQDN**: The public facing address of GitLab CE/EE, eg. gitlab.com.
-
-**Make sure to not remove the `/ci$request_uri`. This is required to properly forward the requests.**
-
-You should also make sure that you can do:
+    # Omnibus installation
+    sudo gitlab-ctl restart unicorn sidekiq
+
+### Part III: Nginx configuration
+
+This section is only required for **manual installations**. Omnibus users can
+[skip to the final step](#part-iv-finishing-up).
+
+#### 1. Update Nginx configuration
+
+To ensure that your existing CI runners are able to communicate with the
+migrated installation, and that existing build triggers still work, you'll need
+to update your Nginx configuration to redirect requests for the old locations to
+the new ones.
+
+Edit `/etc/nginx/sites-available/gitlab_ci` and paste:
+
+```nginx
+# GITLAB CI
+server {
+  listen 80 default_server;         # e.g., listen 192.168.1.1:80;
+  server_name YOUR_CI_SERVER_FQDN;  # e.g., server_name source.example.com;
+
+  access_log  /var/log/nginx/gitlab_ci_access.log;
+  error_log   /var/log/nginx/gitlab_ci_error.log;
+
+  # expose API to fix runners
+  location /api {
+    proxy_read_timeout    300;
+    proxy_connect_timeout 300;
+    proxy_redirect        off;
+    proxy_set_header      X-Real-IP $remote_addr;
+
+    # You need to specify your DNS servers that are able to resolve YOUR_GITLAB_SERVER_FQDN
+    resolver 8.8.8.8 8.8.4.4;
+    proxy_pass $scheme://YOUR_GITLAB_SERVER_FQDN/ci$request_uri;
+  }
+
+  # expose build endpoint to allow trigger builds
+  location ~ ^/projects/\d+/build$ {
+    proxy_read_timeout    300;
+    proxy_connect_timeout 300;
+    proxy_redirect        off;
+    proxy_set_header      X-Real-IP $remote_addr;
+
+    # You need to specify your DNS servers that are able to resolve YOUR_GITLAB_SERVER_FQDN
+    resolver 8.8.8.8 8.8.4.4;
+    proxy_pass $scheme://YOUR_GITLAB_SERVER_FQDN/ci$request_uri;
+  }
+
+  # redirect all other CI requests
+  location / {
+    return 301 $scheme://YOUR_GITLAB_SERVER_FQDN/ci$request_uri;
+  }
+
+  # adjust this to match the largest build log your runners might submit,
+  # set to 0 to disable limit
+  client_max_body_size 10m;
+}
+```
+
+Make sure you substitute these placeholder values with your real ones:
+
+1. `YOUR_CI_SERVER_FQDN`: The existing public-facing address of your GitLab CI
+   install (e.g., `ci.gitlab.com`).
+1. `YOUR_GITLAB_SERVER_FQDN`: The current public-facing address of your GitLab
+   CE (or EE) install (e.g., `gitlab.com`).
+
+**Make sure not to remove the `/ci$request_uri` part. This is required to
+properly forward the requests.**
+
+You should also make sure that you can:
+
 1. `curl https://YOUR_GITLAB_SERVER_FQDN/` from your previous GitLab CI server.
-1. `curl https://YOUR_CI_SERVER_FQDN/` from your GitLab CE/EE server.
+1. `curl https://YOUR_CI_SERVER_FQDN/` from your GitLab CE (or EE) server.
 
-## Check your configuration
+#### 2. Check Nginx configuration
 
     sudo nginx -t
 
-## Restart nginx
+#### 3. Restart Nginx
 
     sudo /etc/init.d/nginx restart
 
-### 15. Done!
-
-If everything went OK you should be able to access all your GitLab CI data by pointing your browser to:
-https://gitlab.example.com/ci/.
+### Part IV: Finishing Up
 
-The GitLab CI should also work when using the previous address, redirecting you to the GitLab CE/EE.
+If everything went well you should be able to access your migrated CI install by
+visiting `https://gitlab.example.com/ci/`. If you visit the old GitLab CI
+address, you should be redirected to the new one.
 
 **Enjoy!**
+
+### Troubleshooting
+
+#### Restore from backup
+
+If something went wrong and you need to restore a backup, consult the [Backup
+restoration](../raketasks/backup_restore.md) guide.