Update/improve CI migration docs for 8.0.1

1 files changed, 147 insertions, 189 deletions

diff --git a/doc/migrate_ci_to_ce/README.md b/doc/migrate_ci_to_ce/README.md
index 1f050e2def2..c2df33a22e0 100644
--- a/doc/migrate_ci_to_ce/README.md
+++ b/doc/migrate_ci_to_ce/README.md
@@ -7,230 +7,198 @@ into the CE and EE applications.
 This guide will detail the process of migrating your CI installation and data
 into your GitLab CE or EE installation.
 
-### Before we begin
+We recommend that you read through the entire migration process in this
+document before beginning.
 
-**You need to have a working installation of GitLab CI version 8.0 to perform
-this migration. The older versions are not supported and will most likely break
-this migration procedure.**
+### Overview
 
-This migration cannot be performed online and takes a significant amount of
-time. Make sure to plan ahead.
+In this document we assume you have a GitLab server and a GitLab CI server. It
+does not matter if these are the same machine.
 
-If you are running a version of GitLab CI prior to 8.0 please follow the
-appropriate [update guide](https://gitlab.com/gitlab-org/gitlab-ci/tree/master/doc/update/)
-before proceeding.
+The migration consists of three parts: updating GitLab and GitLab CI, moving
+data, and redirecting traffic.
 
-The migration is divided into four parts and covers both manual and Omnibus
-installations:
+Please note that CI builds triggered on your GitLab server in the time between
+updating to 8.0 and finishing the migration will be lost.
 
-1. [GitLab CI](#part-i-gitlab-ci)
-1. [Gitlab CE (or EE)](#part-ii-gitlab-ce-or-ee)
-1. [Nginx configuration](#part-iii-nginx-configuration)
-1. [Finishing Up](#part-iv-finishing-up)
+### Before upgrading
 
-### Part I: GitLab CI
+- (1) Make sure that the backup script on both servers can connect to the database.
 
-#### 1. Stop GitLab CI
-
-    # Manual installation
-    sudo service gitlab_ci stop
-
-    # Omnibus installation
-    sudo gitlab-ctl stop ci-unicorn ci-sidekiq
-
-#### 2. Create a backup
-
-The migration procedure modifies the structure of the CI database. If something
-goes wrong, you will not be able to revert to a previous version without a
-backup.
-
-If your GitLab CI installation uses **MySQL** and your GitLab CE (or EE)
-installation uses **PostgreSQL** you'll need to convert the CI database by
-setting a `MYSQL_TO_POSTGRESQL` flag.
-
-If you use the Omnibus package you most likely use **PostgreSQL** on both GitLab
-CE (or EE) and CI.
-
-You can check which database each install is using by viewing their
-database configuration files:
-
-    cat /home/gitlab_ci/gitlab-ci/config/database.yml
-    cat /home/git/gitlab/config/database.yml
-
-- If both applications use the same database `adapter`, create the backup with
-  this command:
-
-        # Manual installation
-        cd /home/gitlab_ci/gitlab-ci
-        sudo -u gitlab_ci -H bundle exec rake backup:create RAILS_ENV=production
-
-        # Omnibus installation
-        sudo gitlab-ci-rake backup:create
-
-- If CI uses MySQL, and CE (or EE) uses PostgreSQL, create the backup with this
-  command (note the `MYSQL_TO_POSTGRESQL` flag):
-
-        # Manual installation
-        cd /home/gitlab_ci/gitlab-ci
-        sudo -u gitlab_ci -H bundle exec rake backup:create RAILS_ENV=production MYSQL_TO_POSTGRESQL=1
-
-        # Omnibus installation
-        sudo gitlab-ci-rake backup:create MYSQL_TO_POSTGRESQL=1
-
-#### 3. Remove cronjob
-
-**Note:** This step is only required for **manual installations**. Omnibus users
-can [skip to the next step](#part-ii-gitlab-ce-or-ee).
-
-    # Manual installation
-    cd /home/gitlab_ci/gitlab-ci
-    sudo -u gitlab_ci -H bundle exec whenever --clear-crontab
-
-### Part II: GitLab CE (or EE)
-
-#### 1. Ensure GitLab is updated
-
-Your GitLab CE or EE installation **must be version 8.0**. If it's not, follow
-the [update guide](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/update/7.14-to-8.0.md)
-before proceeding.
-
-If you use the Omnibus packages simply run `apt-get upgrade` to install the
-latest version.
-
-#### 2. Prevent CI usage during the migration process
-
-As an administrator, go to **Admin Area** -> **Settings**, and under **Continuous
-Integration** uncheck **Disable to prevent CI usage until rake ci:migrate is run
-(8.0 only)**.
-
-This will disable the CI integration and prevent users from creating CI projects
-until the migration process is completed.
-
-#### 3. Stop GitLab
-
-Before you can migrate data you need to stop the GitLab service first:
-
-    # Manual installation
-    sudo service gitlab stop
-
-    # Omnibus installation
-    sudo gitlab-ctl stop unicorn sidekiq
+```
+# CI server
+# Omnibus
+sudo gitlab-ci-rake backup:create
 
-#### 4. Create a backup
+# Source
+cd /home/gitlab_ci/gitlab-ci
+sudo -u gitlab_ci -H bundle exec rake backup:create RAILS_ENV=production
+```
 
-This migration poses a **significant risk** of breaking your GitLab
-installation. Create a backup before proceeding:
+```
+# GitLab server
+# Omnibus
+sudo gitlab-rake gitlab:backup:create SKIP=repositories,uploads
 
-    # Manual installation
-    cd /home/git/gitlab
-    sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production
+# Source
+cd /home/git/gitlab
+sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production SKIP=repositories,uploads
+```
 
-    # Omnibus installation
-    sudo gitlab-rake gitlab:backup:create
+If this fails you need to fix it before upgrading to 8.0. Also see
+https://about.gitlab.com/getting-help/
 
-It's possible to speed up backup creation by skipping repositories and uploads:
+- (2) Check what databases you use on your GitLab server and your CI server.
+  Look for the 'adapter:' line. If your CI server and your GitLab server use
+the same database adapter no special care is needed. If your CI server uses
+MySQL and your GitLab server uses PostgreSQL you need to pass a special option
+during the 'Moving data' part. **If your CI server uses PostgreSQL and your
+GitLab server uses MySQL you cannot migrate your CI data to GitLab 8.0.***
 
-    # Manual installation
-    cd /home/git/gitlab
-    sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production SKIP=repositories,uploads
+- (3) Decide where to store CI build traces on GitLab server. GitLab CI uses
+  files on disk to store CI build traces. The default path for these build
+traces is `/var/opt/gitlab/gitlab-ci/build` (Omnibus) or
+`/home/git/gitlab/builds` (Source). If you are storing your repository data in
+a special location, or if you are using NFS, you should make sure that you
+store build traces on the same storage as your Git repositories.
 
-    # Omnibus installation
-    sudo gitlab-rake gitlab:backup:create SKIP=repositories,uploads
+```
+# CI server
+# Omnibus
+sudo gitlab-ci-rake env:info
 
-#### 5. Copy secret tokens from CI
+# Source
+cd /home/gitlab_ci/gitlab-ci
+sudo -u gitlab_ci -H bundle exec rake env:info RAILS_ENV=production
+```
 
-The `secrets.yml` file stores encryption keys for secure variables.
+```
+# GitLab server
+# Omnibus
+sudo gitlab-rake gitlab:env:info
 
-- **Manual installations** need to copy the contents of GitLab CI's
-  `config/secrets.yml` file to the same file in GitLab CE:
+# Source
+cd /home/git/gitlab
+sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production
+```
 
-    ```bash
-    # Manual installation
-    sudo cp /home/gitlab_ci/gitlab-ci/config/secrets.yml /home/git/gitlab/config/secrets.yml
-    sudo chown git:git /home/git/gitlab/config/secrets.yml
-    sudo chown 0600 /home/git/gitlab/config/secrets.yml
-    ```
+### Upgrading
 
-- **Omnibus installations** where GitLab CI and CE (or EE) are on the same
-  server don't need to do anything further, because the secrets are stored in
-  `/etc/gitlab/gitlab-secrets.json`.
+From this point on, GitLab CI will be unavailable for your end users.
 
-- **Omnibus installations** where GitLab CI is on a different server than CE (or
-  EE) will need to:
-    1. On the CI server, copy the `db_key_base` value from
-       `/etc/gitlab/gitlab-secrets.json`
-    1. On the CE (or EE) server, add `gitlab_ci['db_key_base'] =
-       "VALUE_FROM_ABOVE"` to the `/etc/gitlab/gitlab.rb` file and run `sudo
-       gitlab-ctl reconfigure`
+- (1) First upgrade your GitLab server to version 8.0:
+https://about.gitlab.com/update/
+- (2) After you update, go to the admin panel and temporarily disable CI.  As
+  an administrator, go to **Admin Area** -> **Settings**, and under
+**Continuous Integration** uncheck **Disable to prevent CI usage until rake
+ci:migrate is run (8.0 only)**.
+- (3) If you want to use custom CI settings (e.g. change where builds are
+  stored), please update `/etc/gitlab/gitlab.rb` (Omnibus) or
+`/home/git/gitlab/config/gitlab.yml` (Source).
+- (4) Now upgrade GitLab CI to version 8.0. If you are using Omnibus packages,
+  this may have already happened when you upgraded GitLab to 8.0.
 
-#### 6. New configuration options for `gitlab.yml`
+- (5) Disable GitLab CI after upgrading to 8.0.
 
-**Note:** This step is only required for **manual installations**. Omnibus users
-can [skip to the next step](#7-copy-backup-from-gitlab-ci).
+```
+# CI server
+# Omnibus
+sudo gitlab-ctl stop ci-unicorn
+sudo gitlab-ctl stop ci-sidekiq
+
+# Source
+sudo service gitlab_ci stop
+cd /home/gitlab_ci/gitlab-ci
+sudo -u gitlab_ci -H bundle exec whenever --clear-crontab
+```
 
-There are new configuration options available for `gitlab.yml`. View them with
-the command below and apply them manually to your current `gitlab.yml`:
+### Moving data
 
-    git diff origin/7-14-stable:config/gitlab.yml.example origin/8-0-stable:config/gitlab.yml.example
+- (1) Move the database encryption key from your CI server to your GitLab server.
 
-The new options include configuration settings for GitLab CI.
+```
+# CI server
+# Omnibus
+sudo gitlab-ci-rake backup:show_secrets
 
-#### 7. Copy backup from GitLab CI
+# Source
+cd /home/gitlab_ci/gitlab-ci
+sudo -u gitlab_ci -H bundle exec rake backup:show_secrets RAILS_ENV=production
+```
 
-```bash
-# Manual installation
-sudo cp -v /home/gitlab_ci/gitlab-ci/tmp/backups/*_gitlab_ci_backup.tar /home/git/gitlab/tmp/backups
-sudo chown git:git /home/git/gitlab/tmp/backups/*_gitlab_ci_backup.tar
+- (2) Create your final CI data export. If you are converting from MySQL to
+  PostgreSQL, add ` MYSQL_TO_POSTGRESQL=1` to the end of the rake command. When
+the command finishes it will print the path to your data export archive; you
+will need this file later.
 
-# Omnibus installation
-sudo cp -v /var/opt/gitlab/ci-backups/*_gitlab_ci_backup.tar /var/opt/gitlab/backups/
-sudo chown git:git /var/opt/gitlab/backups/*_gitlab_ci_backup.tar
 ```
+# CI server
+# Omnibus
+sudo gitlab-ci-rake backup:create
 
-If moving across the servers you can use `scp`.
-However, this requires you to provide an authorized key or password to login to
-the GitLab CE (or EE) server from the CI server. You can try to use ssh-agent
-from your local machine to have that: login to your GitLab CI server using
-`ssh -A`.
+# Source
+cd /home/git/gitlab
+sudo -u git -H bundle exec rake backup:create RAILS_ENV=production
+```
 
-```bash
-# Manual installation
-scp /home/gitlab_ci/gitlab-ci/tmp/backups/*_gitlab_ci_backup.tar root@gitlab.example.com:/home/git/gitlab/tmp/backup
+- (3) Copy your CI data archive to your GitLab server. If you were running
+  GitLab and GitLab CI on the same server you can skip this step. There are
+many ways to do this, below we use SSH agent forwarding and 'scp', which will
+be easy and fast for most setups. You can also copy the data archive first from
+the CI server to your laptop and then from your laptop to the GitLab server.
 
-# Omnibus installation
-scp /var/opt/gitlab/ci-backups/*_gitlab_ci_backup.tar root@gitlab.example.com:/var/opt/gitlab/backups/
+```
+# Start from your laptop
+ssh -A ci_admin@ci_server.example
+# Now on the CI server
+scp /path/to/12345_gitlab_ci_backup.tar gitlab_admin@gitlab_server.example:~
 ```
 
-#### 8. Import GitLab CI backup
+- (4) Make the CI data archive discoverable for GitLab. We assume below that
+  you store backups in the default path, adjust the command if necessary.
 
-Now you'll import the GitLab CI database dump that you created earlier into the
-GitLab CE or EE database:
+```
+# GitLab server
+# Omnibus
+sudo mv /path/to/12345_gitlab_ci_backup.tar /var/opt/gitlab/backups/
 
-    # Manual installation
-    sudo -u git -H bundle exec rake ci:migrate RAILS_ENV=production
+# Source
+sudo mv /path/to/12345_gitlab_ci_backup.tar /home/git/gitlab/tmp/backups/
+```
 
-    # Omnibus installation
-    sudo gitlab-rake ci:migrate
+- (5) Import the CI data into GitLab.
 
-This task will take some time.
+```
+# GitLab server
+# Omnibus
+sudo gitlab-rake ci:migrate
 
-This migration task automatically re-enables the CI setting that you
-[disabled earlier](#2-prevent-ci-usage-during-the-migration-process).
+# Source
+cd /home/git/gitlab
+sudo -u git -H bundle exec rake ci:migrate RAILS_ENV=production
+```
 
-#### 9. Start GitLab
+- (6) Restart GitLab
 
-You can start GitLab CE (or EE) now and see if everything is working:
+```
+# GitLab server
+# Omnibus
+sudo gitlab-ctl hup unicorn
+sudo gitlab-ctl restart sidekiq
 
-    # Manual installation
-    sudo service gitlab start
+# Source
+sudo service gitlab reload
+```
 
-    # Omnibus installation
-    sudo gitlab-ctl restart unicorn sidekiq
+### Redirecting traffic
 
-### Part III: Nginx configuration
+If you were running GitLab CI with Omnibus packages and you were using the
+internal NGINX configuration your CI service should now be available both at
+`ci.example.com` (the old address) and `gitlab.example.com/ci`. You are done!
 
-This section is only required for **manual installations**. Omnibus users can
-[skip to the final step](#part-iv-finishing-up).
+If you installed GitLab CI from source we now need to configure a redirect in
+NGINX so that existing CI runners can keep using the old CI server address, and
+so that existing links to your CI server keep working.
 
 #### 1. Update Nginx configuration
 
@@ -296,16 +264,6 @@ You should also make sure that you can:
 
     sudo /etc/init.d/nginx restart
 
-### Part IV: Finishing Up
-
-If everything went well you should be able to access your migrated CI install by
-visiting `https://gitlab.example.com/ci/`. If you visit the old GitLab CI
-address, you should be redirected to the new one.
-
-**Enjoy!**
-
-### Troubleshooting
-
 #### Restore from backup
 
 If something went wrong and you need to restore a backup, consult the [Backup