diff options
Diffstat (limited to 'doc/development/migration_style_guide.md')
-rw-r--r-- | doc/development/migration_style_guide.md | 56 |
1 files changed, 44 insertions, 12 deletions
diff --git a/doc/development/migration_style_guide.md b/doc/development/migration_style_guide.md index e0e21319f47..64d8b22f1b8 100644 --- a/doc/development/migration_style_guide.md +++ b/doc/development/migration_style_guide.md @@ -45,9 +45,12 @@ work it needs to perform and how long it takes to complete: One exception is a migration that takes longer but is absolutely critical for the application to operate correctly. For example, you might have indices that enforce unique tuples, or that are needed for query performance in critical parts of the application. In cases where the migration would be unacceptably slow, however, a better option might be to guard the feature with a [feature flag](feature_flags/index.md) and perform a post-deployment migration instead. The feature can then be turned on after the migration finishes. + + Migrations used to add new models are also part of these regular schema migrations. The only differences are the Rails command used to generate the migrations and the additional generated files, one for the model and one for the model's spec. 1. [**Post-deployment migrations.**](database/post_deployment_migrations.md) These are Rails migrations in `db/post_migrate` and - run _after_ new application code has been deployed (for GitLab.com after the production deployment has finished). - They can be used for schema changes that aren't critical for the application to operate, or data migrations that take at most a few minutes. + are run independently from the GitLab.com deployments. Pending post migrations are executed on a daily basis at the discretion + of release manager through the [post-deploy migration pipeline](https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/post_deploy_migration/readme.md#how-to-determine-if-a-post-deploy-migration-has-been-executed-on-gitlabcom). + These migrations can be used for schema changes that aren't critical for the application to operate, or data migrations that take at most a few minutes. Common examples for schema changes that should run post-deploy include: - Clean-ups, like removing unused columns. - Adding non-critical indices on high-traffic tables. @@ -88,7 +91,7 @@ Keep in mind that all durations should be measured against GitLab.com. |----|----|---| | Regular migrations | `<= 3 minutes` | A valid exception are changes without which application functionality or performance would be severely degraded and which cannot be delayed. | | Post-deployment migrations | `<= 10 minutes` | A valid exception are schema changes, since they must not happen in background migrations. | -| Background migrations | `> 10 minutes` | Since these are suitable for larger tables, it's not possible to set a precise timing guideline, however, any single query must stay below [`1 second` execution time](query_performance.md#timing-guidelines-for-queries) with cold caches. | +| Background migrations | `> 10 minutes` | Since these are suitable for larger tables, it's not possible to set a precise timing guideline, however, any single query must stay below [`1 second` execution time](database/query_performance.md#timing-guidelines-for-queries) with cold caches. | ## Decide which database to target @@ -108,6 +111,20 @@ bundle exec rails g migration migration_name_here This generates the migration file in `db/migrate`. +### Regular schema migrations to add new models + +To create a new model you can use the following Rails generator: + +```shell +bundle exec rails g model model_name_here +``` + +This will generate: + +- the migration file in `db/migrate` +- the model file in `app/models` +- the spec file in `spec/models` + ## Schema Changes Changes to the schema should be committed to `db/structure.sql`. This @@ -119,7 +136,7 @@ columns manually for existing tables as this causes confusion to other people using `db/structure.sql` generated by Rails. NOTE: -[Creating an index asynchronously requires two merge requests.](adding_database_indexes.md#add-a-migration-to-create-the-index-synchronously) +[Creating an index asynchronously requires two merge requests.](database/adding_database_indexes.md#add-a-migration-to-create-the-index-synchronously) When done, commit the schema change in the merge request that adds the index with `add_concurrent_index`. @@ -245,7 +262,7 @@ When using a single-transaction migration, a transaction holds a database connec for the duration of the migration, so you must make sure the actions in the migration do not take too much time: GitLab.com's production database has a `15s` timeout, so in general, the cumulative execution time in a migration should aim to fit comfortably -in that limit. Singular query timings should fit within the [standard limit](query_performance.md#timing-guidelines-for-queries) +in that limit. Singular query timings should fit within the [standard limit](database/query_performance.md#timing-guidelines-for-queries) In case you need to insert, update, or delete a significant amount of data, you: @@ -268,7 +285,7 @@ which is a "versioned" class. For new migrations, the latest version should be u can be looked up in `Gitlab::Database::Migration::MIGRATION_CLASSES`) to use the latest version of migration helpers. -In this example, we use version 1.0 of the migration class: +In this example, we use version 2.0 of the migration class: ```ruby class TestMigration < Gitlab::Database::Migration[2.0] @@ -580,7 +597,7 @@ end Verify the index is not being used anymore with this Thanos query: ```sql -sum(rate(pg_stat_user_indexes_idx_tup_read{env="gprd", indexrelname="index_ci_name", type="patroni-ci"}[5m])) +sum by (type)(rate(pg_stat_user_indexes_idx_scan{env="gprd", indexrelname="index_groups_on_parent_id_id"}[5m])) ``` Note that it is not necessary to check if the index exists prior to @@ -611,7 +628,7 @@ might not be required, like: Additionally, wide indexes are not required to match all filter criteria of queries, we just need to cover enough columns so that the index lookup has a small enough selectivity. Please review our -[Adding Database indexes](adding_database_indexes.md) guide for more details. +[Adding Database indexes](database/adding_database_indexes.md) guide for more details. When adding an index to a non-empty table make sure to use the method `add_concurrent_index` instead of the regular `add_index` method. @@ -640,7 +657,7 @@ end You must explicitly name indexes that are created with more complex definitions beyond table name, column names, and uniqueness constraint. -Consult the [Adding Database Indexes](adding_database_indexes.md#requirements-for-naming-indexes) +Consult the [Adding Database Indexes](database/adding_database_indexes.md#requirements-for-naming-indexes) guide for more details. If you need to add a unique index, please keep in mind there is the possibility @@ -658,7 +675,7 @@ If a migration requires conditional logic based on the absence or presence of an index, you must test for existence of that index using its name. This helps avoids problems with how Rails compares index definitions, which can lead to unexpected results. For more details, review the -[Adding Database Indexes](adding_database_indexes.md#why-explicit-names-are-required) +[Adding Database Indexes](database/adding_database_indexes.md#why-explicit-names-are-required) guide. The easiest way to test for existence of an index by name is to use the @@ -739,6 +756,21 @@ If a backport adding a column with a default value is needed for %12.9 or earlie it should use `add_column_with_default` helper. If a [large table](https://gitlab.com/gitlab-org/gitlab/-/blob/master/rubocop/rubocop-migrations.yml#L3) is involved, backporting to %12.9 is contraindicated. +## Removing the column default for non-nullable columns + +If you have added a non-nullable column, and used the default value to populate +existing data, you need to keep that default value around until at least after +the application code is updated. You cannot remove the default value in the +same migration, as the migrations run before the model code is updated and +models will have an old schema cache, meaning they won't know about this column +and won't be able to set it. In this case it's recommended to: + +1. Add the column with default value in a normal migration. +1. Remove the default in a post-deployment migration. + +The post-deployment migration happens after the application restarts, +ensuring the new column has been discovered. + ## Changing the column default One might think that changing a default column with `change_column_default` is an @@ -1196,8 +1228,8 @@ If using a model in the migrations, you should first [clear the column cache](https://api.rubyonrails.org/classes/ActiveRecord/ModelSchema/ClassMethods.html#method-i-reset_column_information) using `reset_column_information`. -If using a model that leverages single table inheritance (STI), there are [special -considerations](single_table_inheritance.md#in-migrations). +If using a model that leverages single table inheritance (STI), there are +[special considerations](database/single_table_inheritance.md#in-migrations). This avoids problems where a column that you are using was altered and cached in a previous migration. |