diff options
Diffstat (limited to 'doc/administration/geo/disaster_recovery/index.md')
| -rw-r--r-- | doc/administration/geo/disaster_recovery/index.md | 351 |
1 files changed, 212 insertions, 139 deletions
diff --git a/doc/administration/geo/disaster_recovery/index.md b/doc/administration/geo/disaster_recovery/index.md index 43e5dc1d224..d1ea2978202 100644 --- a/doc/administration/geo/disaster_recovery/index.md +++ b/doc/administration/geo/disaster_recovery/index.md @@ -38,7 +38,7 @@ order to avoid unnecessary data loss. WARNING: If the **primary** node goes offline, there may be data saved on the **primary** node -that has not been replicated to the **secondary** node. This data should be treated +that have not been replicated to the **secondary** node. This data should be treated as lost if you proceed. If an outage on the **primary** node happens, you should do everything possible to @@ -46,62 +46,53 @@ avoid a split-brain situation where writes can occur in two different GitLab instances, complicating recovery efforts. So to prepare for the failover, we must disable the **primary** node. -1. SSH into the **primary** node to stop and disable GitLab, if possible: +- If you have SSH access: - ```shell - sudo gitlab-ctl stop - ``` - - Prevent GitLab from starting up again if the server unexpectedly reboots: + 1. SSH into the **primary** node to stop and disable GitLab: - ```shell - sudo systemctl disable gitlab-runsvdir - ``` - - NOTE: - (**CentOS only**) In CentOS 6 or older, there is no easy way to prevent GitLab from being - started if the machine reboots isn't available (see [Omnibus GitLab issue #3058](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/3058)). - It may be safest to uninstall the GitLab package completely: - - ```shell - yum remove gitlab-ee - ``` + ```shell + sudo gitlab-ctl stop + ``` - NOTE: - (**Ubuntu 14.04 LTS**) If you are using an older version of Ubuntu - or any other distribution based on the Upstart init system, you can prevent GitLab - from starting if the machine reboots by doing the following: + 1. Prevent GitLab from starting up again if the server unexpectedly reboots: - ```shell - initctl stop gitlab-runsvvdir - echo 'manual' > /etc/init/gitlab-runsvdir.override - initctl reload-configuration - ``` + ```shell + sudo systemctl disable gitlab-runsvdir + ``` -1. If you do not have SSH access to the **primary** node, take the machine offline and - prevent it from rebooting by any means at your disposal. - Since there are many ways you may prefer to accomplish this, we will avoid a - single recommendation. You may need to: +- If you do not have SSH access to the **primary** node, take the machine offline and + prevent it from rebooting by any means at your disposal. + Since there are many ways you may prefer to accomplish this, we will avoid a + single recommendation. You may need to: - - Reconfigure the load balancers. - - Change DNS records (for example, point the primary DNS record to the - **secondary** node to stop usage of the **primary** node). - - Stop the virtual servers. - - Block traffic through a firewall. - - Revoke object storage permissions from the **primary** node. - - Physically disconnect a machine. + - Reconfigure the load balancers. + - Change DNS records (for example, point the primary DNS record to the + **secondary** node to stop usage of the **primary** node). + - Stop the virtual servers. + - Block traffic through a firewall. + - Revoke object storage permissions from the **primary** node. + - Physically disconnect a machine. -1. If you plan to [update the primary domain DNS record](#step-4-optional-updating-the-primary-domain-dns-record), - you may wish to lower the TTL now to speed up propagation. + If you plan to [update the primary domain DNS record](#step-4-optional-updating-the-primary-domain-dns-record), + you may wish to lower the TTL now to speed up propagation. ### Step 3. Promoting a **secondary** node +WARNING: +In GitLab 13.2 and 13.3, promoting a secondary node to a primary while the +secondary is paused fails. Do not pause replication before promoting a +secondary. If the node is paused, be sure to resume before promoting. +This issue has been fixed in GitLab 13.4 and later. + Note the following when promoting a secondary: - If replication was paused on the secondary node (for example as a part of upgrading, while you were running a version of GitLab earlier than 13.4), you _must_ [enable the node by using the database](../replication/troubleshooting.md#message-activerecordrecordinvalid-validation-failed-enabled-geo-primary-node-cannot-be-disabled) - before proceeding. + before proceeding. If the secondary node + [has been paused](../../geo/index.md#pausing-and-resuming-replication), the promotion + performs a point-in-time recovery to the last known state. + Data that was created on the primary while the secondary was paused will be lost. - A new **secondary** should not be added at this time. If you want to add a new **secondary**, do this after you have completed the entire process of promoting the **secondary** to the **primary**. @@ -117,62 +108,48 @@ Note the following when promoting a secondary: sudo -i ``` -1. Edit `/etc/gitlab/gitlab.rb` to reflect its new status as **primary** by - removing any lines that enabled the `geo_secondary_role`: - - Users of GitLab 13.5 or later can skip this step, due to the appropriate - roles being enabled or disabled during the promotion in the following - step. +1. If you're using GitLab 13.5 and later, skip this step. If not, edit + `/etc/gitlab/gitlab.rb` and remove any of the following lines that + might be present: ```ruby - ## In pre-11.5 documentation, the role was enabled as follows. Remove this line. geo_secondary_role['enable'] = true - - ## In 11.5+ documentation, the role was enabled as follows. Remove this line. roles ['geo_secondary_role'] ``` -1. Promote the **secondary** node to the **primary** node. - - WARNING: - In GitLab 13.2 and 13.3, promoting a secondary node to a primary while the - secondary is paused fails. Do not pause replication before promoting a - secondary. If the node is paused, be sure to resume before promoting. This - issue has been fixed in GitLab 13.4 and later. - - WARNING: - If the secondary node [has been paused](../../geo/index.md#pausing-and-resuming-replication), this performs - a point-in-time recovery to the last known state. - Data that was created on the primary while the secondary was paused will be lost. +1. Promote the **secondary** node to the **primary** node: - NOTE: - In GitLab 13.7 and earlier, if you have a data type with zero items to sync, - this command reports `ERROR - Replication is not up-to-date` even if - replication is actually up-to-date. If replication and verification output - shows that it is complete, you can add `--skip-preflight-checks` to make the - command complete promotion. This bug was fixed in GitLab 13.8 and later. + - To promote the secondary node to primary along with [preflight checks](planned_failover.md#preflight-checks): - To promote the secondary node to primary along with preflight checks: + ```shell + gitlab-ctl promote-to-primary-node + ``` - ```shell - gitlab-ctl promote-to-primary-node - ``` + - If you have already run the preflight checks separately or don't want to run them, + you can skip them with: - If you have already run the [preflight checks](planned_failover.md#preflight-checks) separately or don't want to run them, you can skip preflight checks with: + ```shell + gitlab-ctl promote-to-primary-node --skip-preflight-checks + ``` - ```shell - gitlab-ctl promote-to-primary-node --skip-preflight-checks - ``` + NOTE: + In GitLab 13.7 and earlier, if you have a data type with zero items to sync + and don't skip the preflight checks, promoting the secondary reports + `ERROR - Replication is not up-to-date` even if replication is actually + up-to-date. If replication and verification output + shows that it is complete, you can skip the preflight checks to make the + command complete promotion. This bug was fixed in GitLab 13.8 and later. - You can also promote the secondary node to primary **without any further confirmation**, even when preflight checks fail: + - To promote the secondary node to primary **without any further confirmation**, + even when preflight checks fail: - ```shell - gitlab-ctl promote-to-primary-node --force - ``` + ```shell + gitlab-ctl promote-to-primary-node --force + ``` -1. Verify you can connect to the newly promoted **primary** node using the URL used +1. Verify you can connect to the newly-promoted **primary** node using the URL used previously for the **secondary** node. -1. If successful, the **secondary** node has now been promoted to the **primary** node. +1. If successful, the **secondary** node is now promoted to the **primary** node. #### Promoting a **secondary** node with multiple servers @@ -181,17 +158,6 @@ conjunction with multiple servers, as it can only perform changes on a **secondary** with only a single machine. Instead, you must do this manually. -WARNING: -In GitLab 13.2 and 13.3, promoting a secondary node to a primary while the -secondary is paused fails. Do not pause replication before promoting a -secondary. If the node is paused, be sure to resume before promoting. This -issue has been fixed in GitLab 13.4 and later. - -WARNING: -If the secondary node [has been paused](../../geo/index.md#pausing-and-resuming-replication), this performs -a point-in-time recovery to the last known state. -Data that was created on the primary while the secondary was paused will be lost. - 1. SSH in to the database node in the **secondary** and trigger PostgreSQL to promote to read-write: @@ -201,20 +167,17 @@ Data that was created on the primary while the secondary was paused will be lost In GitLab 12.8 and earlier, see [Message: `sudo: gitlab-pg-ctl: command not found`](../replication/troubleshooting.md#message-sudo-gitlab-pg-ctl-command-not-found). -1. Edit `/etc/gitlab/gitlab.rb` on every machine in the **secondary** to - reflect its new status as **primary** by removing any lines that enabled the - `geo_secondary_role`: +1. Edit `/etc/gitlab/gitlab.rb` on every node in the **secondary** site to + reflect its new status as **primary** by removing any of the following + lines that might be present: ```ruby - ## In pre-11.5 documentation, the role was enabled as follows. Remove this line. geo_secondary_role['enable'] = true - - ## In 11.5+ documentation, the role was enabled as follows. Remove this line. roles ['geo_secondary_role'] ``` - After making these changes [Reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) each - machine so the changes take effect. + After making these changes, [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) + on each machine so the changes take effect. 1. Promote the **secondary** to **primary**. SSH into a single application server and execute: @@ -223,9 +186,9 @@ Data that was created on the primary while the secondary was paused will be lost sudo gitlab-rake geo:set_secondary_as_primary ``` -1. Verify you can connect to the newly promoted **primary** using the URL used +1. Verify you can connect to the newly-promoted **primary** using the URL used previously for the **secondary**. -1. Success! The **secondary** has now been promoted to **primary**. +1. If successful, the **secondary** node is now promoted to the **primary** node. #### Promoting a **secondary** node with a Patroni standby cluster @@ -234,17 +197,6 @@ conjunction with a Patroni standby cluster, as it can only perform changes on a **secondary** with only a single machine. Instead, you must do this manually. -WARNING: -In GitLab 13.2 and 13.3, promoting a secondary node to a primary while the -secondary is paused fails. Do not pause replication before promoting a -secondary. If the node is paused, be sure to resume before promoting. This -issue has been fixed in GitLab 13.4 and later. - -WARNING: -If the secondary node [has been paused](../../geo/index.md#pausing-and-resuming-replication), this performs -a point-in-time recovery to the last known state. -Data that was created on the primary while the secondary was paused will be lost. - 1. SSH in to the Standby Leader database node in the **secondary** and trigger PostgreSQL to promote to read-write: @@ -252,13 +204,10 @@ Data that was created on the primary while the secondary was paused will be lost sudo gitlab-ctl promote-db ``` -1. Edit `/etc/gitlab/gitlab.rb` on every application and Sidekiq nodes in the secondary to reflect its new status as primary by removing any lines that enabled the `geo_secondary_role`: +1. Edit `/etc/gitlab/gitlab.rb` on every application and Sidekiq nodes in the secondary to reflect its new status as primary by removing any of the following lines that might be present: ```ruby - ## In pre-11.5 documentation, the role was enabled as follows. Remove this line. geo_secondary_role['enable'] = true - - ## In 11.5+ documentation, the role was enabled as follows. Remove this line. roles ['geo_secondary_role'] ``` @@ -280,9 +229,9 @@ Data that was created on the primary while the secondary was paused will be lost sudo gitlab-rake geo:set_secondary_as_primary ``` -1. Verify you can connect to the newly promoted **primary** using the URL used previously for the **secondary**. - -1. Success! The **secondary** has now been promoted to **primary**. +1. Verify you can connect to the newly-promoted **primary** using the URL used + previously for the **secondary**. +1. If successful, the **secondary** node is now promoted to the **primary** node. #### Promoting a **secondary** node with an external PostgreSQL database @@ -292,12 +241,12 @@ node with GitLab and the database on the same machine. As a result, a manual pro required: 1. Promote the replica database associated with the **secondary** site. This will - set the database to read-write: - - Amazon RDS - [Promoting a Read Replica to Be a Standalone DB Instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ReadRepl.html#USER_ReadRepl.Promote) - - Azure Database for PostgreSQL - [Stop replication](https://docs.microsoft.com/en-us/azure/postgresql/howto-read-replicas-portal#stop-replication) - - Other external PostgreSQL databases - save the script below in you secondary node, for example - `/tmp/geo_promote.sh`, and modify the connection parameters to match your - environment. Then, execute it to promote the replica: + set the database to read-write. The instructions vary depending on where your database is hosted: + - [Amazon RDS](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ReadRepl.html#USER_ReadRepl.Promote) + - [Azure PostgreSQL](https://docs.microsoft.com/en-us/azure/postgresql/howto-read-replicas-portal#stop-replication) + - For other external PostgreSQL databases, save the following script in you + secondary node, for example `/tmp/geo_promote.sh`, and modify the connection + parameters to match your environment. Then, execute it to promote the replica: ```shell #!/bin/bash @@ -318,14 +267,11 @@ required: ``` 1. Edit `/etc/gitlab/gitlab.rb` on every node in the **secondary** site to - reflect its new status as **primary** by removing any lines that enabled the - `geo_secondary_role`: + reflect its new status as **primary** by removing any of the following + lines that might be present: ```ruby - ## In GitLab 11.4 and earlier, remove this line. geo_secondary_role['enable'] = true - - ## In GitLab 11.5 and later, remove this line. roles ['geo_secondary_role'] ``` @@ -339,10 +285,9 @@ required: sudo gitlab-rake geo:set_secondary_as_primary ``` -1. Verify you can connect to the newly promoted **primary** site using the URL used - previously for the **secondary** site. - -1. Success! The **secondary** site has now been promoted to **primary**. +1. Verify you can connect to the newly-promoted **primary** using the URL used + previously for the **secondary**. +1. If successful, the **secondary** node is now promoted to the **primary** node. ### Step 4. (Optional) Updating the primary domain DNS record @@ -443,7 +388,7 @@ and after that you also need two extra steps. sudo -i ``` -1. Edit `/etc/gitlab/gitlab.rb` +1. Edit `/etc/gitlab/gitlab.rb`: ```ruby ## Enable a Geo Primary role (if you haven't yet) @@ -468,7 +413,7 @@ and after that you also need two extra steps. (For more details about these settings you can read [Configure the primary server](../setup/database.md#step-1-configure-the-primary-server)) 1. Save the file and reconfigure GitLab for the database listen changes and - the replication slot changes to be applied. + the replication slot changes to be applied: ```shell gitlab-ctl reconfigure @@ -501,6 +446,134 @@ Now we need to make each **secondary** node listen to changes on the new **prima to [initiate the replication process](../setup/database.md#step-3-initiate-the-replication-process) again but this time for another **primary** node. All the old replication settings will be overwritten. +## Promoting a secondary Geo cluster in GitLab Cloud Native Helm Charts + +When updating a Cloud Native Geo deployment, the process for updating any node that is external to the secondary Kubernetes cluster does not differ from the non Cloud Native approach. As such, you can always defer to [Promoting a secondary Geo node in single-secondary configurations](#promoting-a-secondary-geo-node-in-single-secondary-configurations) for more information. + +The following sections assume you are using the `gitlab` namespace. If you used a different namespace when setting up your cluster, you should also replace `--namespace gitlab` with your namespace. + +WARNING: +In GitLab 13.2 and 13.3, promoting a secondary site to a primary while the +secondary is paused fails. Do not pause replication before promoting a +secondary. If the site is paused, be sure to resume before promoting. This +issue has been fixed in GitLab 13.4 and later. + +### Step 1. Permanently disable the **primary** cluster + +WARNING: +If the **primary** site goes offline, there may be data saved on the **primary** site +that has not been replicated to the **secondary** site. This data should be treated +as lost if you proceed. + +If an outage on the **primary** site happens, you should do everything possible to +avoid a split-brain situation where writes can occur in two different GitLab +instances, complicating recovery efforts. So to prepare for the failover, you +must disable the **primary** site: + +- If you have access to the **primary** Kubernetes cluster, connect to it and disable the GitLab webservice and Sidekiq pods: + + ```shell + kubectl --namespace gitlab scale deploy gitlab-geo-webservice-default --replicas=0 + kubectl --namespace gitlab scale deploy gitlab-geo-sidekiq-all-in-1-v1 --replicas=0 + ``` + +- If you do not have access to the **primary** Kubernetes cluster, take the cluster offline and + prevent it from coming back online by any means at your disposal. + Since there are many ways you may prefer to accomplish this, we will avoid a + single recommendation. You may need to: + + - Reconfigure the load balancers. + - Change DNS records (for example, point the primary DNS record to the + **secondary** site to stop usage of the **primary** site). + - Stop the virtual servers. + - Block traffic through a firewall. + - Revoke object storage permissions from the **primary** site. + - Physically disconnect a machine. + +### Step 2. Promote all **secondary** nodes external to the cluster + +WARNING: +If the secondary site [has been paused](../../geo/index.md#pausing-and-resuming-replication), this performs +a point-in-time recovery to the last known state. +Data that was created on the primary while the secondary was paused will be lost. + +1. SSH in to the database node in the **secondary** and trigger PostgreSQL to + promote to read-write: + + ```shell + sudo gitlab-ctl promote-db + ``` + + In GitLab 12.8 and earlier, see [Message: `sudo: gitlab-pg-ctl: command not found`](../replication/troubleshooting.md#message-sudo-gitlab-pg-ctl-command-not-found). + +1. Edit `/etc/gitlab/gitlab.rb` on the database node in the **secondary** site to + reflect its new status as **primary** by removing any lines that enabled the + `geo_secondary_role`: + + NOTE: + Depending on your architecture these steps will need to be run on any GitLab node that is external to the **secondary** Kubernetes cluster. + + ```ruby + ## In pre-11.5 documentation, the role was enabled as follows. Remove this line. + geo_secondary_role['enable'] = true + + ## In 11.5+ documentation, the role was enabled as follows. Remove this line. + roles ['geo_secondary_role'] + ``` + + After making these changes, [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) on the database node. + +### Step 3. Promote the **secondary** cluster + +1. Find the task runner pod: + + ```shell + kubectl --namespace gitlab get pods -lapp=task-runner + ``` + +1. Promote the secondary: + + ```shell + kubectl --namespace gitlab exec -ti gitlab-geo-task-runner-XXX -- gitlab-rake geo:set_secondary_as_primary + ``` + +1. Update the existing cluster configuration. + + You can retrieve the existing config with Helm: + + ```shell + helm --namespace gitlab get values gitlab-geo > gitlab.yaml + ``` + + The existing config will contain a section for Geo that should resemble: + + ```yaml + geo: + enabled: true + role: secondary + nodeName: secondary.example.com + psql: + host: geo-2.db.example.com + port: 5431 + password: + secret: geo + key: geo-postgresql-password + ``` + + To promote the **secondary** cluster to a **primary** cluster, update `role: secondary` to `role: primary`. + + You can remove the entire `psql` section if the cluster will remain as a primary site, this refers to the tracking database and will be ignored whilst the cluster is acting as a primary site. + + Update the cluster with the new config: + + ```shell + helm upgrade --install --version <current Chart version> gitlab-geo gitlab/gitlab --namespace gitlab -f gitlab.yaml + ``` + +1. Verify you can connect to the newly promoted primary using the URL used previously for the secondary. + +1. Success! The secondary has now been promoted to primary. + ## Troubleshooting This section was moved to [another location](../replication/troubleshooting.md#fixing-errors-during-a-failover-or-when-promoting-a-secondary-to-a-primary-node). |