1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
|
---
stage: Enablement
group: Distribution
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
---
# Repository storage Rake tasks **(FREE SELF)**
This is a collection of Rake tasks to help you list and migrate
existing projects and their attachments to the new
[hashed storage](../repository_storage_types.md) that GitLab
uses to organize the Git data.
## List projects and attachments
The following Rake tasks will list the projects and attachments that are
available on legacy and hashed storage.
### On legacy storage
To have a summary and then a list of projects and their attachments using legacy storage:
- **Omnibus installation**
```shell
# Projects
sudo gitlab-rake gitlab:storage:legacy_projects
sudo gitlab-rake gitlab:storage:list_legacy_projects
# Attachments
sudo gitlab-rake gitlab:storage:legacy_attachments
sudo gitlab-rake gitlab:storage:list_legacy_attachments
```
- **Source installation**
```shell
# Projects
sudo -u git -H bundle exec rake gitlab:storage:legacy_projects RAILS_ENV=production
sudo -u git -H bundle exec rake gitlab:storage:list_legacy_projects RAILS_ENV=production
# Attachments
sudo -u git -H bundle exec rake gitlab:storage:legacy_attachments RAILS_ENV=production
sudo -u git -H bundle exec rake gitlab:storage:list_legacy_attachments RAILS_ENV=production
```
### On hashed storage
To have a summary and then a list of projects and their attachments using hashed storage:
- **Omnibus installation**
```shell
# Projects
sudo gitlab-rake gitlab:storage:hashed_projects
sudo gitlab-rake gitlab:storage:list_hashed_projects
# Attachments
sudo gitlab-rake gitlab:storage:hashed_attachments
sudo gitlab-rake gitlab:storage:list_hashed_attachments
```
- **Source installation**
```shell
# Projects
sudo -u git -H bundle exec rake gitlab:storage:hashed_projects RAILS_ENV=production
sudo -u git -H bundle exec rake gitlab:storage:list_hashed_projects RAILS_ENV=production
# Attachments
sudo -u git -H bundle exec rake gitlab:storage:hashed_attachments RAILS_ENV=production
sudo -u git -H bundle exec rake gitlab:storage:list_hashed_attachments RAILS_ENV=production
```
## Migrate to hashed storage
WARNING:
In GitLab 13.0, [hashed storage](../repository_storage_types.md#hashed-storage)
is enabled by default and the legacy storage is deprecated.
Support for legacy storage will be removed in GitLab 14.0. If you're on GitLab
13.0 and later, switching new projects to legacy storage is not possible.
The option to choose between hashed and legacy storage in the admin area has
been disabled.
This task must be run on any machine that has Rails/Sidekiq configured and will
schedule all your existing projects and attachments associated with it to be
migrated to the **Hashed** storage type:
- **Omnibus installation**
```shell
sudo gitlab-rake gitlab:storage:migrate_to_hashed
```
- **Source installation**
```shell
sudo -u git -H bundle exec rake gitlab:storage:migrate_to_hashed RAILS_ENV=production
```
If you have any existing integration, you may want to do a small rollout first,
to validate. You can do so by specifying an ID range with the operation by using
the environment variables `ID_FROM` and `ID_TO`. For example, to limit the rollout
to project IDs 50 to 100 in an Omnibus GitLab installation:
```shell
sudo gitlab-rake gitlab:storage:migrate_to_hashed ID_FROM=50 ID_TO=100
```
You can monitor the progress in the **Admin Area > Monitoring > Background Jobs** page.
There is a specific queue you can watch to see how long it will take to finish:
`hashed_storage:hashed_storage_project_migrate`.
After it reaches zero, you can confirm every project has been migrated by running the commands above.
If you find it necessary, you can run this migration script again to schedule missing projects.
Any error or warning will be logged in Sidekiq's log file.
If [Geo](../geo/index.md) is enabled, each project that is successfully migrated
generates an event to replicate the changes on any **secondary** nodes.
You only need the `gitlab:storage:migrate_to_hashed` Rake task to migrate your repositories, but we have additional
commands below that helps you inspect projects and attachments in both legacy and hashed storage.
## Rollback from hashed storage to legacy storage
WARNING:
In GitLab 13.0, [hashed storage](../repository_storage_types.md#hashed-storage)
is enabled by default and the legacy storage is deprecated.
Support for legacy storage will be removed in GitLab 14.0. If you're on GitLab
13.0 and later, switching new projects to legacy storage is not possible.
The option to choose between hashed and legacy storage in the admin area has
been disabled.
This task will schedule all your existing projects and associated attachments to be rolled back to the
legacy storage type.
- **Omnibus installation**
```shell
sudo gitlab-rake gitlab:storage:rollback_to_legacy
```
- **Source installation**
```shell
sudo -u git -H bundle exec rake gitlab:storage:rollback_to_legacy RAILS_ENV=production
```
If you have any existing integration, you may want to do a small rollback first,
to validate. You can do so by specifying an ID range with the operation by using
the environment variables `ID_FROM` and `ID_TO`. For example, to limit the rollout
to project IDs 50 to 100 in an Omnibus GitLab installation:
```shell
sudo gitlab-rake gitlab:storage:rollback_to_legacy ID_FROM=50 ID_TO=100
```
You can monitor the progress in the **Admin Area > Monitoring > Background Jobs** page.
On the **Queues** tab, you can watch the `hashed_storage:hashed_storage_project_rollback` queue to see how long the process will take to finish.
After it reaches zero, you can confirm every project has been rolled back by running the commands above.
If some projects weren't rolled back, you can run this rollback script again to schedule further rollbacks.
Any error or warning will be logged in Sidekiq's log file.
If you have a Geo setup, the rollback will not be reflected automatically
on the **secondary** node. You may need to wait for a backfill operation to kick-in and remove
the remaining repositories from the special `@hashed/` folder manually.
|
