Merge branch 'drop_db_before_restore' into 'master'

Reload the schema before restoring a database backup

If a user tries to downgrade and restore after a failed upgrade,
the database may still contain newer tables. Reload the older
schema before restoring the database to avoid future upgrade
problems. Also, add a rake task to help users add migration
versions to the database so it's easier to recover from these
errors if they do occur. Fixes #13419

See merge request !2807

3 files changed, 87 insertions, 0 deletions

diff --git a/doc/raketasks/backup_restore.md b/doc/raketasks/backup_restore.md
index f6d1234ac4a..4329ac30a1c 100644
--- a/doc/raketasks/backup_restore.md
+++ b/doc/raketasks/backup_restore.md
@@ -249,6 +249,9 @@ reconfigure` after changing `gitlab-secrets.json`.
 ### Installation from source
 
 ```
+# Stop processes that are connected to the database
+sudo service gitlab stop
+
 bundle exec rake gitlab:backup:restore RAILS_ENV=production
 ```
 
diff --git a/doc/update/README.md b/doc/update/README.md
index 109d5de3fa2..0241f036830 100644
--- a/doc/update/README.md
+++ b/doc/update/README.md
@@ -15,3 +15,4 @@ Depending on the installation method and your GitLab version, there are multiple
 
 - [MySQL to PostgreSQL](mysql_to_postgresql.md) guides you through migrating your database from MySQL to PostgreSQL.
 - [MySQL installation guide](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/install/database_mysql.md) contains additional information about configuring GitLab to work with a MySQL database.
+- [Restoring from backup after a failed upgrade](restore_after_failure.md)
diff --git a/doc/update/restore_after_failure.md b/doc/update/restore_after_failure.md
new file mode 100644
index 00000000000..01c52aae7f5
--- /dev/null
+++ b/doc/update/restore_after_failure.md
@@ -0,0 +1,83 @@
+# Restoring from backup after a failed upgrade
+
+Upgrades are usually smooth and restoring from backup is a rare occurrence.
+However, it's important to know how to recover when problems do arise.
+
+## Roll back to an earlier version and restore a backup
+
+In some cases after a failed upgrade, the fastest solution is to roll back to
+the previous version you were using.
+
+First, roll back the code or package. For source installations this involves
+checking out the older version (branch or tag). For Omnibus installations this
+means installing the older .deb or .rpm package. Then, restore from a backup.
+Follow the instructions in the
+[Backup and Restore](../raketasks/backup_restore.md#restore-a-previously-created-backup)
+documentation.
+
+## Potential problems on the next upgrade
+
+When a rollback is necessary it can produce problems on subsequent upgrade
+attempts. This is because some tables may have been added during the failed
+upgrade. If these tables are still present after you restore from the
+older backup it can lead to migration failures on future upgrades.
+
+Starting in GitLab 8.6 we drop all tables prior to importing the backup to
+prevent this problem. If you've restored a backup to a version prior to 8.6 you
+may need to manually correct the problem next time you upgrade.
+
+Example error:
+
+```
+== 20151103134857 CreateLfsObjects: migrating =================================
+-- create_table(:lfs_objects)
+rake aborted!
+StandardError: An error has occurred, this and all later migrations canceled:
+
+PG::DuplicateTable: ERROR:  relation "lfs_objects" already exists
+```
+
+Copy the version from the error. In this case the version number is
+`20151103134857`.
+
+>**WARNING:** Use the following steps only if you are certain this is what you
+need to do.
+
+### GitLab 8.6+
+
+Pass the version to a database rake task to manually mark the migration as
+complete.
+
+```
+# Source install
+sudo -u git -H bundle exec rake gitlab:db:mark_migration_complete[20151103134857] RAILS_ENV=production
+
+# Omnibus install
+sudo gitlab-rake gitlab:db:mark_migration_complete[20151103134857]
+```
+
+Once the migration is successfully marked, run the rake `db:migrate` task again.
+You will likely have to repeat this process several times until all failed
+migrations are marked complete.
+
+### GitLab < 8.6
+
+```
+# Source install
+sudo -u git -H bundle exec rails console production
+
+# Omnibus install
+sudo gitlab-rails console
+```
+
+At the Rails console, type the following commands:
+
+```
+ActiveRecord::Base.connection.execute("INSERT INTO schema_migrations (version) VALUES('20151103134857')")
+exit
+```
+
+Once the migration is successfully marked, run the rake `db:migrate` task again.
+You will likely have to repeat this process several times until all failed
+migrations are marked complete.
+