Merge branch 'background-migrations' into 'master'

Support for post deployment migrations

Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/22133

See merge request !6572

7 files changed, 168 insertions, 0 deletions

diff --git a/CHANGELOG.md b/CHANGELOG.md
index 24b9219d9cf..3423e810c2b 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -72,6 +72,51 @@ Please view this file on the master branch, on stable branches it's out of date.
 - Fix and improve `Sortable.highest_label_priority`. !7165
 - Fixed sticky merge request tabs when sidebar is pinned. !7167
 - Only remove right connector of first build of last stage. !7179
+  - Backups do not fail anymore when using tar on annex and custom_hooks only. !5814
+  - Adds user project membership expired event to clarify why user was removed (Callum Dryden)
+  - Trim leading and trailing whitespace on project_path (Linus Thiel)
+  - Prevent award emoji via notes for issues/MRs authored by user (barthc)
+  - Adds an optional path parameter to the Commits API to filter commits by path (Luis HGO)
+  - Fix extra space on Build sidebar on Firefox !7060
+  - Fix mobile layout issues in admin user overview page !7087
+  - Fix HipChat notifications rendering (airatshigapov, eisnerd)
+  - Refactor Jira service to use jira-ruby gem
+  - Add hover to trash icon in notes !7008 (blackst0ne)
+  - Only show one error message for an invalid email !5905 (lycoperdon)
+  - Fix sidekiq stats in admin area (blackst0ne)
+  - Created cycle analytics bundle JavaScript file
+  - API: Fix booleans not recognized as such when using the `to_boolean` helper
+  - Removed delete branch tooltip !6954
+  - Stop unauthorized users dragging on milestone page (blackst0ne)
+  - Restore issue boards welcome message when a project is created !6899
+  - Escape ref and path for relative links !6050 (winniehell)
+  - Fixed link typo on /help/ui to Alerts section. !6915 (Sam Rose)
+  - Fix filtering of milestones with quotes in title (airatshigapov)
+  - Refactor less readable existance checking code from CoffeeScript !6289 (jlogandavison)
+  - Update mail_room and enable sentinel support to Reply By Email (!7101)
+  - Add task completion status in Issues and Merge Requests tabs: "X of Y tasks completed" (!6527, @gmesalazar)
+  - Added support for executing post deployment migrations
+  - Simpler arguments passed to named_route on toggle_award_url helper method
+  - Fix typo in framework css class. !7086 (Daniel Voogsgerd)
+  - New issue board list dropdown stays open after adding a new list
+  - Fix: Backup restore doesn't clear cache
+  - API: Fix project deploy keys 400 and 500 errors when adding an existing key. !6784 (Joshua Welsh)
+  - Replace jquery.cookie plugin with js.cookie !7085
+  - Use MergeRequestsClosingIssues cache data on Issue#closed_by_merge_requests method
+  - Fix Sign in page 'Forgot your password?' link overlaps on medium-large screens
+  - Show full status link on MR & commit pipelines
+  - Fix documents and comments on Build API `scope`
+  - Refactor email, use setter method instead AR callbacks for email attribute (Semyon Pupkov)
+  - Shortened merge request modal to let clipboard button not overlap
+  - In all filterable drop downs, put input field in focus only after load is complete (Ido @leibo)
+
+## 8.13.2
+  - Fix builds dropdown overlapping bug !7124
+  - Fix applying labels for GitHub-imported MRs !7139
+  - Fix importing MR comments from GitHub !7139
+  - Modify GitHub importer to be retryable !7003
+  - Fix and improve `Sortable.highest_label_priority`
+  - Fixed sticky merge request tabs when sidebar is pinned
 
 ## 8.13.1 (2016-10-25)
 
diff --git a/config/initializers/0_post_deployment_migrations.rb b/config/initializers/0_post_deployment_migrations.rb
new file mode 100644
index 00000000000..0068a03d214
--- /dev/null
+++ b/config/initializers/0_post_deployment_migrations.rb
@@ -0,0 +1,12 @@
+# Post deployment migrations are included by default. This file must be loaded
+# before other initializers as Rails may otherwise memoize a list of migrations
+# excluding the post deployment migrations.
+unless ENV['SKIP_POST_DEPLOYMENT_MIGRATIONS']
+  path = Rails.root.join('db', 'post_migrate').to_s
+
+  Rails.application.config.paths['db/migrate'] << path
+
+  # Rails memoizes migrations at certain points where it won't read the above
+  # path just yet. As such we must also update the following list of paths.
+  ActiveRecord::Migrator.migrations_paths << path
+end
diff --git a/db/post_migrate/.gitkeep b/db/post_migrate/.gitkeep
new file mode 100644
index 00000000000..e69de29bb2d
--- /dev/null
+++ b/db/post_migrate/.gitkeep
diff --git a/doc/development/README.md b/doc/development/README.md
index 14d6f08e43a..3f2151bbe8e 100644
--- a/doc/development/README.md
+++ b/doc/development/README.md
@@ -41,6 +41,7 @@
 
 - [What requires downtime?](what_requires_downtime.md)
 - [Adding database indexes](adding_database_indexes.md)
+- [Post Deployment Migrations](post_deployment_migrations.md)
 
 ## Compliance
 
diff --git a/doc/development/post_deployment_migrations.md b/doc/development/post_deployment_migrations.md
new file mode 100644
index 00000000000..cfc91539bee
--- /dev/null
+++ b/doc/development/post_deployment_migrations.md
@@ -0,0 +1,75 @@
+# Post Deployment Migrations
+
+Post deployment migrations are regular Rails migrations that can optionally be
+executed after a deployment. By default these migrations are executed alongside
+the other migrations. To skip these migrations you will have to set the
+environment variable `SKIP_POST_DEPLOYMENT_MIGRATIONS` to a non-empty value
+when running `rake db:migrate`.
+
+For example, this would run all migrations including any post deployment
+migrations:
+
+```bash
+bundle exec rake db:migrate
+```
+
+This however will skip post deployment migrations:
+
+```bash
+SKIP_POST_DEPLOYMENT_MIGRATIONS=true bundle exec rake db:migrate
+```
+
+## Deployment Integration
+
+Say you're using Chef for deploying new versions of GitLab and you'd like to run
+post deployment migrations after deploying a new version. Let's assume you
+normally use the command `chef-client` to do so. To make use of this feature
+you'd have to run this command as follows:
+
+```bash
+SKIP_POST_DEPLOYMENT_MIGRATIONS=true sudo chef-client
+```
+
+Once all servers have been updated you can run `chef-client` again on a single
+server _without_ the environment variable.
+
+The process is similar for other deployment techniques: first you would deploy
+with the environment variable set, then you'll essentially re-deploy a single
+server but with the variable _unset_.
+
+## Creating Migrations
+
+To create a post deployment migration you can use the following Rails generator:
+
+```bash
+bundle exec rails g post_deployment_migration migration_name_here
+```
+
+This will generate the migration file in `db/post_migrate`. These migrations
+behave exactly like regular Rails migrations.
+
+## Use Cases
+
+Post deployment migrations can be used to perform migrations that mutate state
+that an existing version of GitLab depends on. For example, say you want to
+remove a column from a table. This requires downtime as a GitLab instance
+depends on this column being present while it's running. Normally you'd follow
+these steps in such a case:
+
+1. Stop the GitLab instance
+2. Run the migration removing the column
+3. Start the GitLab instance again
+
+Using post deployment migrations we can instead follow these steps:
+
+1. Deploy a new version of GitLab while ignoring post deployment migrations
+2. Re-run `rake db:migrate` but without the environment variable set
+
+Here we don't need any downtime as the migration takes place _after_ a new
+version (which doesn't depend on the column anymore) has been deployed.
+
+Some other examples where these migrations are useful:
+
+* Cleaning up data generated due to a bug in GitLab
+* Removing tables
+* Migrating jobs from one Sidekiq queue to another
diff --git a/generator_templates/rails/post_deployment_migration/migration.rb b/generator_templates/rails/post_deployment_migration/migration.rb
new file mode 100644
index 00000000000..1a7b8d5bf35
--- /dev/null
+++ b/generator_templates/rails/post_deployment_migration/migration.rb
@@ -0,0 +1,22 @@
+# See http://doc.gitlab.com/ce/development/migration_style_guide.html
+# for more information on how to write migrations for GitLab.
+
+class <%= migration_class_name %> < ActiveRecord::Migration
+  include Gitlab::Database::MigrationHelpers
+
+  DOWNTIME = false
+
+  # When using the methods "add_concurrent_index" or "add_column_with_default"
+  # you must disable the use of transactions as these methods can not run in an
+  # existing transaction. When using "add_concurrent_index" make sure that this
+  # method is the _only_ method called in the migration, any other changes
+  # should go in a separate migration. This ensures that upon failure _only_ the
+  # index creation fails and can be retried or reverted easily.
+  #
+  # To disable transactions uncomment the following line and remove these
+  # comments:
+  # disable_ddl_transaction!
+
+  def change
+  end
+end
diff --git a/lib/generators/rails/post_deployment_migration/post_deployment_migration_generator.rb b/lib/generators/rails/post_deployment_migration/post_deployment_migration_generator.rb
new file mode 100644
index 00000000000..392bd640192
--- /dev/null
+++ b/lib/generators/rails/post_deployment_migration/post_deployment_migration_generator.rb
@@ -0,0 +1,13 @@
+module Rails
+  class PostDeploymentMigrationGenerator < Rails::Generators::NamedBase
+    def create_migration_file
+      timestamp = Time.now.strftime('%Y%m%d%H%I%S')
+
+      template "migration.rb", "db/post_migrate/#{timestamp}_#{file_name}.rb"
+    end
+
+    def migration_class_name
+      file_name.camelize
+    end
+  end
+end