diff options
Diffstat (limited to 'doc/raketasks/backup_restore.md')
| -rw-r--r-- | doc/raketasks/backup_restore.md | 211 |
1 files changed, 206 insertions, 5 deletions
diff --git a/doc/raketasks/backup_restore.md b/doc/raketasks/backup_restore.md index 9de063eec6f..9db1f09cf61 100644 --- a/doc/raketasks/backup_restore.md +++ b/doc/raketasks/backup_restore.md @@ -12,11 +12,11 @@ An application data backup creates an archive file that contains the database, all repositories and all attachments. You can only restore a backup to **exactly the same version and type (CE/EE)** -of GitLab on which it was created. The best way to migrate your repositories -from one server to another is through a backup and restore. +of GitLab on which it was created. The best way to [migrate your projects +from one server to another](#migrate-to-a-new-server) is through a backup and restore. WARNING: -GitLab doesn't back up items that aren't stored in the file system. If you're +GitLab doesn't back up items that aren't stored on the file system. If you're using [object storage](../administration/object_storage.md), be sure to enable backups with your object storage provider, if desired. @@ -68,6 +68,7 @@ including: Backups do not include: - [Mattermost data](https://docs.mattermost.com/administration/config-settings.html#file-storage) +- Redis (and thus Sidekiq jobs) WARNING: GitLab does not back up any configuration files (`/etc/gitlab`), TLS keys and certificates, or system @@ -983,7 +984,7 @@ your installation is using PgBouncer, for either performance reasons or when usi Next, restore `/etc/gitlab/gitlab-secrets.json` if necessary, [as previously mentioned](#restore-prerequisites). -Reconfigure, restart and check GitLab: +Reconfigure, restart and [check](../administration/raketasks/maintenance.md#check-gitlab-configuration) GitLab: ```shell sudo gitlab-ctl reconfigure @@ -999,6 +1000,14 @@ the target for the restore. sudo gitlab-rake gitlab:doctor:secrets ``` +For added assurance, you can perform [an integrity check on the uploaded files](../administration/raketasks/check.md#uploaded-files-integrity): + +```shell +sudo gitlab-rake gitlab:artifacts:check +sudo gitlab-rake gitlab:lfs:check +sudo gitlab-rake gitlab:uploads:check +``` + ### Restore for Docker image and GitLab Helm chart installations For GitLab installations using the Docker image or the GitLab Helm chart on a @@ -1180,7 +1189,7 @@ has a longer discussion explaining the potential problems. To prevent writes to the Git repository data, there are two possible approaches: -- Use [maintenance mode](../administration/maintenance_mode/index.md) to place GitLab in a read-only state. +- Use [maintenance mode](../administration/maintenance_mode/index.md) **(PREMIUM SELF)** to place GitLab in a read-only state. - Create explicit downtime by stopping all Gitaly services before backing up the repositories: ```shell @@ -1282,6 +1291,198 @@ sudo GITLAB_BACKUP_PGHOST=192.168.1.10 GITLAB_BACKUP_PGPORT=5432 /opt/gitlab/bin See the [PostgreSQL documentation](https://www.postgresql.org/docs/12/libpq-envars.html) for more details on what these parameters do. +## Migrate to a new server + +<!-- some details borrowed from GitLab.com move from Azure to GCP detailed at https://gitlab.com/gitlab-com/migration/-/blob/master/.gitlab/issue_templates/failover.md --> + +You can use GitLab backup and restore to migrate your instance to a new server. This section outlines a typical procedure for a GitLab deployment running on a single server. +If you're running GitLab Geo, an alternative option is [Geo disaster recovery for planned failover](../administration/geo/disaster_recovery/planned_failover.md). + +WARNING: +Avoid uncoordinated data processing by both the new and old servers, where multiple +servers could connect concurrently and process the same data. For example, when using +[incoming email](../administration/incoming_email.md), if both GitLab instances are +processing email at the same time, then both instances will end up missing some data. +This type of problem can occur with other services as well, such as a +[non-packaged database](https://docs.gitlab.com/omnibus/settings/database.html#using-a-non-packaged-postgresql-database-management-server), +a non-packaged Redis instance, or non-packaged Sidekiq. + +Prerequisites: + +- Some time before your migration, consider notifying your users of upcoming + scheduled maintenance with a [broadcast message banner](../user/admin_area/broadcast_messages.md). +- Ensure your backups are complete and current. Create a complete system-level backup, or + take a snapshot of all servers involved in the migration, in case destructive commands + (like `rm`) are run incorrectly. + +### Prepare the new server + +To prepare the new server: + +1. Copy the + [SSH host keys](https://superuser.com/questions/532040/copy-ssh-keys-from-one-server-to-another-server/532079#532079) + from the old server to avoid man-in-the-middle attack warnings. +1. [Install and configure GitLab](https://about.gitlab.com/install) except + [incoming email](../administration/incoming_email.md): + 1. Install GitLab. + 1. Configure by copying `/etc/gitlab` files from the old server to the new server, and update as necessary. + Read the + [Omnibus configuration backup and restore instructions](https://docs.gitlab.com/omnibus/settings/backups.html) for more detail. + 1. If applicable, disable [incoming email](../administration/incoming_email.md). + 1. Block new CI/CD jobs from starting upon initial startup after the backup and restore. + Edit `/etc/gitlab/gitlab.rb` and set the following: + + ```ruby + nginx['custom_gitlab_server_config'] = "location /api/v4/jobs/request {\n deny all;\n return 503;\n}\n" + ``` + + 1. Reconfigure GitLab: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +1. Stop GitLab to avoid any potential unnecessary and unintentional data processing: + + ```shell + sudo gitlab-ctl stop + ``` + +1. Configure the new server to allow receiving the Redis database and GitLab backup files: + + ```shell + sudo rm -f /var/opt/gitlab/redis/dump.rdb + sudo chown <your-linux-username> /var/opt/gitlab/redis + sudo mkdir /var/opt/gitlab/backups + sudo chown <your-linux-username> /var/opt/gitlab/backups + ``` + +### Prepare and transfer content from the old server + +1. Ensure you have an up-to-date system-level backup or snapshot of the old server. +1. Enable [maintenance mode](../administration/maintenance_mode/index.md) **(PREMIUM SELF)**, + if supported by your GitLab edition. +1. Block new CI/CD jobs from starting: + 1. Edit `/etc/gitlab/gitlab.rb`, and set the following: + + ```ruby + nginx['custom_gitlab_server_config'] = "location /api/v4/jobs/request {\n deny all;\n return 503;\n}\n" + ``` + + 1. Reconfigure GitLab: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +1. Disable periodic background jobs: + 1. On the top bar, select **Menu > Admin**. + 1. On the left sidebar, select **Monitoring > Background Jobs**. + 1. Under the Sidekiq dashboard, select **Cron** tab and then + **Disable All**. +1. Wait for the currently running CI/CD jobs to finish, or accept that jobs that have not completed may be lost. + To view jobs currently running, on the left sidebar, select **Overviews > Jobs**, + and then select **Running**. +1. Wait for Sidekiq jobs to finish: + 1. On the left sidebar, select **Monitoring > Background Jobs**. + 1. Under the Sidekiq dashboard, select **Queues** and then **Live Poll**. + Wait for **Busy** and **Enqueued** to drop to 0. + These queues contain work that has been submitted by your users; + shutting down before these jobs complete may cause the work to be lost. + Make note of the numbers shown in the Sidekiq dashboard for post-migration verification. +1. Flush the Redis database to disk, and stop GitLab other than the services needed for migration: + + ```shell + sudo /opt/gitlab/embedded/bin/redis-cli -s /var/opt/gitlab/redis/redis.socket save && sudo gitlab-ctl stop && sudo gitlab-ctl start postgresql + ``` + +1. Create a GitLab backup: + + ```shell + sudo gitlab-backup create + ``` + +1. Disable the following GitLab services and prevent unintentional restarts by adding the following to the bottom of `/etc/gitlab/gitlab.rb`: + + ```ruby + alertmanager['enable'] = false + gitlab_exporter['enable'] = false + gitlab_pages['enable'] = false + gitlab_workhorse['enable'] = false + grafana['enable'] = false + logrotate['enable'] = false + gitlab_rails['incoming_email_enabled'] = false + nginx['enable'] = false + node_exporter['enable'] = false + postgres_exporter['enable'] = false + postgresql['enable'] = false + prometheus['enable'] = false + puma['enable'] = false + redis['enable'] = false + redis_exporter['enable'] = false + registry['enable'] = false + sidekiq['enable'] = false + ``` + +1. Reconfigure GitLab: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +1. Verify everything is stopped, and confirm no services are running: + + ```shell + sudo gitlab-ctl status + ``` + +1. Transfer the Redis database and GitLab backups to the new server: + + ```shell + sudo scp /var/opt/gitlab/redis/dump.rdb <your-linux-username>@new-server:/var/opt/gitlab/redis + sudo scp /var/opt/gitlab/backups/your-backup.tar <your-linux-username>@new-server:/var/opt/gitlab/backups + ``` + +### Restore data on the new server + +1. Restore appropriate file system permissions: + + ```shell + sudo chown gitlab-redis /var/opt/gitlab/redis + sudo chown gitlab-redis:gitlab-redis /var/opt/gitlab/redis/dump.rdb + sudo chown git:root /var/opt/gitlab/backups + sudo chown git:git /var/opt/gitlab/backups/your-backup.tar + ``` + +1. [Restore the GitLab backup](#restore-gitlab). +1. Verify that the Redis database restored correctly: + 1. On the top bar, select **Menu > Admin**. + 1. On the left sidebar, select **Monitoring > Background Jobs**. + 1. Under the Sidekiq dashboard, verify that the numbers + match with what was shown on the old server. + 1. While still under the Sidekiq dashboard, select **Cron** and then **Enable All** + to re-enable periodic background jobs. +1. Test that read-only operations on the GitLab instance work as expected. For example, browse through project repository files, merge requests, and issues. +1. Disable [Maintenance Mode](../administration/maintenance_mode/index.md) **(PREMIUM SELF)**, if previously enabled. +1. Test that the GitLab instance is working as expected. +1. If applicable, re-enable [incoming email](../administration/incoming_email.md) and test it is working as expected. +1. Update your DNS or load balancer to point at the new server. +1. Unblock new CI/CD jobs from starting by removing the custom NGINX config + you added previously: + + ```ruby + # The following line must be removed + nginx['custom_gitlab_server_config'] = "location /api/v4/jobs/request {\n deny all;\n return 503;\n}\n" + ``` + +1. Reconfigure GitLab: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +1. Remove the scheduled maintenance [broadcast message banner](../user/admin_area/broadcast_messages.md). + ## Additional notes This documentation is for GitLab Community and Enterprise Edition. We back up |