diff options
Diffstat (limited to 'doc/administration/raketasks/storage.md')
| -rw-r--r-- | doc/administration/raketasks/storage.md | 278 |
1 files changed, 113 insertions, 165 deletions
diff --git a/doc/administration/raketasks/storage.md b/doc/administration/raketasks/storage.md index 1e487a317c9..30f50c24138 100644 --- a/doc/administration/raketasks/storage.md +++ b/doc/administration/raketasks/storage.md @@ -1,214 +1,162 @@ -# Repository Storage Rake Tasks +# Repository storage Rake tasks **(CORE ONLY)** -This is a collection of Rake tasks you can use to help you list and migrate -existing projects and attachments associated with it from Legacy storage to -the new Hashed storage type. +This is a collection of Rake tasks to help you list and migrate +existing projects and their attachments to the new +[hashed storage](../repository_storage_types.md) that GitLab +uses to organize the Git data. -You can read more about the storage types [here][storage-types]. +## List projects and attachments -## Migrate existing projects to Hashed storage +The following Rake tasks will list the projects and attachments that are +available on legacy and hashed storage. -Before migrating your existing projects, you should -[enable hashed storage][storage-migration] for the new projects as well. +### On legacy storage -This task will schedule all your existing projects and attachments associated with it to be migrated to the -**Hashed** storage type: +To have a summary and then a list of projects and their attachments using legacy storage: -**Omnibus Installation** +- **Omnibus installation** -```shell -sudo gitlab-rake gitlab:storage:migrate_to_hashed -``` - -**Source Installation** - -```shell -sudo -u git -H bundle exec rake gitlab:storage:migrate_to_hashed RAILS_ENV=production -``` - -They both also accept a range as environment variable: - -```shell -# to migrate any non migrated project from ID 20 to 50. -export ID_FROM=20 -export ID_TO=50 -``` - -You can monitor the progress in the **Admin Area > Monitoring > Background Jobs** page. -There is a specific Queue you can watch to see how long it will take to finish: -`hashed_storage:hashed_storage_project_migrate` - -After it reaches zero, you can confirm every project has been migrated by running the commands bellow. -If you find it necessary, you can run this migration script again to schedule missing projects. - -Any error or warning will be logged in Sidekiq's log file. - -NOTE: **Note:** -If Geo is enabled, each project that is successfully migrated generates an event to replicate the changes on any **secondary** nodes. - -You only need the `gitlab:storage:migrate_to_hashed` Rake task to migrate your repositories, but we have additional -commands below that helps you inspect projects and attachments in both legacy and hashed storage. - -## Rollback from Hashed storage to Legacy storage - -If you need to rollback the storage migration for any reason, you can follow the steps described here. - -NOTE: **Note:** Hashed Storage will be required in future version of GitLab. - -To prevent new projects from being created in the Hashed storage, -you need to undo the [enable hashed storage][storage-migration] changes. - -This task will schedule all your existing projects and associated attachments to be rolled back to the -Legacy storage type. - -For Omnibus installations, run the following: - -```shell -sudo gitlab-rake gitlab:storage:rollback_to_legacy -``` - -For source installations, run the following: - -```shell -sudo -u git -H bundle exec rake gitlab:storage:rollback_to_legacy RAILS_ENV=production -``` - -Both commands accept a range as environment variable: - -```shell -# to rollback any migrated project from ID 20 to 50. -export ID_FROM=20 -export ID_TO=50 -``` - -You can monitor the progress in the **Admin Area > Monitoring > Background Jobs** page. -On the **Queues** tab, you can watch the `hashed_storage:hashed_storage_project_rollback` queue to see how long the process will take to finish. - -After it reaches zero, you can confirm every project has been rolled back by running the commands bellow. -If some projects weren't rolled back, you can run this rollback script again to schedule further rollbacks. - -Any error or warning will be logged in Sidekiq's log file. - -## List projects on Legacy storage + ```shell + # Projects + sudo gitlab-rake gitlab:storage:legacy_projects + sudo gitlab-rake gitlab:storage:list_legacy_projects -To have a simple summary of projects using **Legacy** storage: + # Attachments + sudo gitlab-rake gitlab:storage:legacy_attachments + sudo gitlab-rake gitlab:storage:list_legacy_attachments + ``` -**Omnibus Installation** +- **Source installation** -```shell -sudo gitlab-rake gitlab:storage:legacy_projects -``` - -**Source Installation** - -```shell -sudo -u git -H bundle exec rake gitlab:storage:legacy_projects RAILS_ENV=production -``` + ```shell + # Projects + sudo -u git -H bundle exec rake gitlab:storage:legacy_projects RAILS_ENV=production + sudo -u git -H bundle exec rake gitlab:storage:list_legacy_projects RAILS_ENV=production -To list projects using **Legacy** storage: + # Attachments + sudo -u git -H bundle exec rake gitlab:storage:legacy_attachments RAILS_ENV=production + sudo -u git -H bundle exec rake gitlab:storage:list_legacy_attachments RAILS_ENV=production + ``` -**Omnibus Installation** - -```shell -sudo gitlab-rake gitlab:storage:list_legacy_projects -``` - -**Source Installation** - -```shell -sudo -u git -H bundle exec rake gitlab:storage:list_legacy_projects RAILS_ENV=production +### On hashed storage -``` +To have a summary and then a list of projects and their attachments using hashed storage: -## List projects on Hashed storage +- **Omnibus installation** -To have a simple summary of projects using **Hashed** storage: + ```shell + # Projects + sudo gitlab-rake gitlab:storage:hashed_projects + sudo gitlab-rake gitlab:storage:list_hashed_projects -**Omnibus Installation** + # Attachments + sudo gitlab-rake gitlab:storage:hashed_attachments + sudo gitlab-rake gitlab:storage:list_hashed_attachments + ``` -```shell -sudo gitlab-rake gitlab:storage:hashed_projects -``` +- **Source installation** -**Source Installation** + ```shell + # Projects + sudo -u git -H bundle exec rake gitlab:storage:hashed_projects RAILS_ENV=production + sudo -u git -H bundle exec rake gitlab:storage:list_hashed_projects RAILS_ENV=production -```shell -sudo -u git -H bundle exec rake gitlab:storage:hashed_projects RAILS_ENV=production -``` + # Attachments + sudo -u git -H bundle exec rake gitlab:storage:hashed_attachments RAILS_ENV=production + sudo -u git -H bundle exec rake gitlab:storage:list_hashed_attachments RAILS_ENV=production + ``` -To list projects using **Hashed** storage: +## Migrate to hashed storage -**Omnibus Installation** +NOTE: **Note:** +In GitLab 13.0, [hashed storage](../repository_storage_types.md#hashed-storage) +is enabled by default and the legacy storage is deprecated. +Support for legacy storage will be removed in GitLab 14.0. If you're on GitLab +13.0 and later, switching new projects to legacy storage is not possible. +The option to choose between hashed and legacy storage in the admin area has +been disabled. -```shell -sudo gitlab-rake gitlab:storage:list_hashed_projects -``` +This task will schedule all your existing projects and attachments associated +with it to be migrated to the **Hashed** storage type: -**Source Installation** +- **Omnibus installation** -```shell -sudo -u git -H bundle exec rake gitlab:storage:list_hashed_projects RAILS_ENV=production -``` + ```shell + sudo gitlab-rake gitlab:storage:migrate_to_hashed + ``` -## List attachments on Legacy storage +- **Source installation** -To have a simple summary of project attachments using **Legacy** storage: + ```shell + sudo -u git -H bundle exec rake gitlab:storage:migrate_to_hashed RAILS_ENV=production + ``` -**Omnibus Installation** +If you have any existing integration, you may want to do a small rollout first, +to validate. You can do so by specifying an ID range with the operation by using +the environment variables `ID_FROM` and `ID_TO`. For example, to limit the rollout +to project IDs 50 to 100 in an Omnibus GitLab installation: ```shell -sudo gitlab-rake gitlab:storage:legacy_attachments +sudo gitlab-rake gitlab:storage:migrate_to_hashed ID_FROM=50 ID_TO=100 ``` -**Source Installation** +You can monitor the progress in the **{admin}** **Admin Area > Monitoring > Background Jobs** page. +There is a specific queue you can watch to see how long it will take to finish: +`hashed_storage:hashed_storage_project_migrate`. -```shell -sudo -u git -H bundle exec rake gitlab:storage:legacy_attachments RAILS_ENV=production -``` - -To list project attachments using **Legacy** storage: - -**Omnibus Installation** +After it reaches zero, you can confirm every project has been migrated by running the commands bellow. +If you find it necessary, you can run this migration script again to schedule missing projects. -```shell -sudo gitlab-rake gitlab:storage:list_legacy_attachments -``` +Any error or warning will be logged in Sidekiq's log file. -**Source Installation** +NOTE: **Note:** +If [Geo](../geo/replication/index.md) is enabled, each project that is successfully migrated +generates an event to replicate the changes on any **secondary** nodes. -```shell -sudo -u git -H bundle exec rake gitlab:storage:list_legacy_attachments RAILS_ENV=production -``` +You only need the `gitlab:storage:migrate_to_hashed` Rake task to migrate your repositories, but we have additional +commands below that helps you inspect projects and attachments in both legacy and hashed storage. -## List attachments on Hashed storage +## Rollback from hashed storage to legacy storage -To have a simple summary of project attachments using **Hashed** storage: +NOTE: **Deprecated:** +In GitLab 13.0, [hashed storage](../repository_storage_types.md#hashed-storage) +is enabled by default and the legacy storage is deprecated. +Support for legacy storage will be removed in GitLab 14.0. If you're on GitLab +13.0 and later, switching new projects to legacy storage is not possible. +The option to choose between hashed and legacy storage in the admin area has +been disabled. -**Omnibus Installation** +This task will schedule all your existing projects and associated attachments to be rolled back to the +legacy storage type. -```shell -sudo gitlab-rake gitlab:storage:hashed_attachments -``` +- **Omnibus installation** -**Source Installation** + ```shell + sudo gitlab-rake gitlab:storage:rollback_to_legacy + ``` -```shell -sudo -u git -H bundle exec rake gitlab:storage:hashed_attachments RAILS_ENV=production -``` +- **Source installation** -To list project attachments using **Hashed** storage: + ```shell + sudo -u git -H bundle exec rake gitlab:storage:rollback_to_legacy RAILS_ENV=production + ``` -**Omnibus Installation** +If you have any existing integration, you may want to do a small rollback first, +to validate. You can do so by specifying an ID range with the operation by using +the environment variables `ID_FROM` and `ID_TO`. For example, to limit the rollout +to project IDs 50 to 100 in an Omnibus GitLab installation: ```shell -sudo gitlab-rake gitlab:storage:list_hashed_attachments +sudo gitlab-rake gitlab:storage:rollback_to_legacy ID_FROM=50 ID_TO=100 ``` -**Source Installation** +You can monitor the progress in the **{admin}** **Admin Area > Monitoring > Background Jobs** page. +On the **Queues** tab, you can watch the `hashed_storage:hashed_storage_project_rollback` queue to see how long the process will take to finish. -```shell -sudo -u git -H bundle exec rake gitlab:storage:list_hashed_attachments RAILS_ENV=production -``` +After it reaches zero, you can confirm every project has been rolled back by running the commands bellow. +If some projects weren't rolled back, you can run this rollback script again to schedule further rollbacks. +Any error or warning will be logged in Sidekiq's log file. -[storage-types]: ../repository_storage_types.md -[storage-migration]: ../repository_storage_types.md#how-to-migrate-to-hashed-storage +If you have a Geo setup, the rollback will not be reflected automatically +on the **secondary** node. You may need to wait for a backfill operation to kick-in and remove +the remaining repositories from the special `@hashed/` folder manually. |