diff options
Diffstat (limited to 'doc/administration/geo')
| -rw-r--r-- | doc/administration/geo/disaster_recovery/index.md | 351 | ||||
| -rw-r--r-- | doc/administration/geo/glossary.md | 2 | ||||
| -rw-r--r-- | doc/administration/geo/index.md | 4 | ||||
| -rw-r--r-- | doc/administration/geo/replication/datatypes.md | 89 | ||||
| -rw-r--r-- | doc/administration/geo/replication/disable_geo.md | 20 | ||||
| -rw-r--r-- | doc/administration/geo/replication/docker_registry.md | 44 | ||||
| -rw-r--r-- | doc/administration/geo/replication/geo_validation_tests.md | 4 | ||||
| -rw-r--r-- | doc/administration/geo/replication/remove_geo_node.md | 58 | ||||
| -rw-r--r-- | doc/administration/geo/replication/remove_geo_site.md | 58 | ||||
| -rw-r--r-- | doc/administration/geo/replication/security_review.md | 10 | ||||
| -rw-r--r-- | doc/administration/geo/replication/usage.md | 27 | ||||
| -rw-r--r-- | doc/administration/geo/replication/using_a_geo_server.md | 27 | ||||
| -rw-r--r-- | doc/administration/geo/replication/version_specific_updates.md | 2 | ||||
| -rw-r--r-- | doc/administration/geo/setup/database.md | 2 | ||||
| -rw-r--r-- | doc/administration/geo/setup/index.md | 2 |
15 files changed, 397 insertions, 303 deletions
diff --git a/doc/administration/geo/disaster_recovery/index.md b/doc/administration/geo/disaster_recovery/index.md index 43e5dc1d224..d1ea2978202 100644 --- a/doc/administration/geo/disaster_recovery/index.md +++ b/doc/administration/geo/disaster_recovery/index.md @@ -38,7 +38,7 @@ order to avoid unnecessary data loss. WARNING: If the **primary** node goes offline, there may be data saved on the **primary** node -that has not been replicated to the **secondary** node. This data should be treated +that have not been replicated to the **secondary** node. This data should be treated as lost if you proceed. If an outage on the **primary** node happens, you should do everything possible to @@ -46,62 +46,53 @@ avoid a split-brain situation where writes can occur in two different GitLab instances, complicating recovery efforts. So to prepare for the failover, we must disable the **primary** node. -1. SSH into the **primary** node to stop and disable GitLab, if possible: +- If you have SSH access: - ```shell - sudo gitlab-ctl stop - ``` - - Prevent GitLab from starting up again if the server unexpectedly reboots: + 1. SSH into the **primary** node to stop and disable GitLab: - ```shell - sudo systemctl disable gitlab-runsvdir - ``` - - NOTE: - (**CentOS only**) In CentOS 6 or older, there is no easy way to prevent GitLab from being - started if the machine reboots isn't available (see [Omnibus GitLab issue #3058](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/3058)). - It may be safest to uninstall the GitLab package completely: - - ```shell - yum remove gitlab-ee - ``` + ```shell + sudo gitlab-ctl stop + ``` - NOTE: - (**Ubuntu 14.04 LTS**) If you are using an older version of Ubuntu - or any other distribution based on the Upstart init system, you can prevent GitLab - from starting if the machine reboots by doing the following: + 1. Prevent GitLab from starting up again if the server unexpectedly reboots: - ```shell - initctl stop gitlab-runsvvdir - echo 'manual' > /etc/init/gitlab-runsvdir.override - initctl reload-configuration - ``` + ```shell + sudo systemctl disable gitlab-runsvdir + ``` -1. If you do not have SSH access to the **primary** node, take the machine offline and - prevent it from rebooting by any means at your disposal. - Since there are many ways you may prefer to accomplish this, we will avoid a - single recommendation. You may need to: +- If you do not have SSH access to the **primary** node, take the machine offline and + prevent it from rebooting by any means at your disposal. + Since there are many ways you may prefer to accomplish this, we will avoid a + single recommendation. You may need to: - - Reconfigure the load balancers. - - Change DNS records (for example, point the primary DNS record to the - **secondary** node to stop usage of the **primary** node). - - Stop the virtual servers. - - Block traffic through a firewall. - - Revoke object storage permissions from the **primary** node. - - Physically disconnect a machine. + - Reconfigure the load balancers. + - Change DNS records (for example, point the primary DNS record to the + **secondary** node to stop usage of the **primary** node). + - Stop the virtual servers. + - Block traffic through a firewall. + - Revoke object storage permissions from the **primary** node. + - Physically disconnect a machine. -1. If you plan to [update the primary domain DNS record](#step-4-optional-updating-the-primary-domain-dns-record), - you may wish to lower the TTL now to speed up propagation. + If you plan to [update the primary domain DNS record](#step-4-optional-updating-the-primary-domain-dns-record), + you may wish to lower the TTL now to speed up propagation. ### Step 3. Promoting a **secondary** node +WARNING: +In GitLab 13.2 and 13.3, promoting a secondary node to a primary while the +secondary is paused fails. Do not pause replication before promoting a +secondary. If the node is paused, be sure to resume before promoting. +This issue has been fixed in GitLab 13.4 and later. + Note the following when promoting a secondary: - If replication was paused on the secondary node (for example as a part of upgrading, while you were running a version of GitLab earlier than 13.4), you _must_ [enable the node by using the database](../replication/troubleshooting.md#message-activerecordrecordinvalid-validation-failed-enabled-geo-primary-node-cannot-be-disabled) - before proceeding. + before proceeding. If the secondary node + [has been paused](../../geo/index.md#pausing-and-resuming-replication), the promotion + performs a point-in-time recovery to the last known state. + Data that was created on the primary while the secondary was paused will be lost. - A new **secondary** should not be added at this time. If you want to add a new **secondary**, do this after you have completed the entire process of promoting the **secondary** to the **primary**. @@ -117,62 +108,48 @@ Note the following when promoting a secondary: sudo -i ``` -1. Edit `/etc/gitlab/gitlab.rb` to reflect its new status as **primary** by - removing any lines that enabled the `geo_secondary_role`: - - Users of GitLab 13.5 or later can skip this step, due to the appropriate - roles being enabled or disabled during the promotion in the following - step. +1. If you're using GitLab 13.5 and later, skip this step. If not, edit + `/etc/gitlab/gitlab.rb` and remove any of the following lines that + might be present: ```ruby - ## In pre-11.5 documentation, the role was enabled as follows. Remove this line. geo_secondary_role['enable'] = true - - ## In 11.5+ documentation, the role was enabled as follows. Remove this line. roles ['geo_secondary_role'] ``` -1. Promote the **secondary** node to the **primary** node. - - WARNING: - In GitLab 13.2 and 13.3, promoting a secondary node to a primary while the - secondary is paused fails. Do not pause replication before promoting a - secondary. If the node is paused, be sure to resume before promoting. This - issue has been fixed in GitLab 13.4 and later. - - WARNING: - If the secondary node [has been paused](../../geo/index.md#pausing-and-resuming-replication), this performs - a point-in-time recovery to the last known state. - Data that was created on the primary while the secondary was paused will be lost. +1. Promote the **secondary** node to the **primary** node: - NOTE: - In GitLab 13.7 and earlier, if you have a data type with zero items to sync, - this command reports `ERROR - Replication is not up-to-date` even if - replication is actually up-to-date. If replication and verification output - shows that it is complete, you can add `--skip-preflight-checks` to make the - command complete promotion. This bug was fixed in GitLab 13.8 and later. + - To promote the secondary node to primary along with [preflight checks](planned_failover.md#preflight-checks): - To promote the secondary node to primary along with preflight checks: + ```shell + gitlab-ctl promote-to-primary-node + ``` - ```shell - gitlab-ctl promote-to-primary-node - ``` + - If you have already run the preflight checks separately or don't want to run them, + you can skip them with: - If you have already run the [preflight checks](planned_failover.md#preflight-checks) separately or don't want to run them, you can skip preflight checks with: + ```shell + gitlab-ctl promote-to-primary-node --skip-preflight-checks + ``` - ```shell - gitlab-ctl promote-to-primary-node --skip-preflight-checks - ``` + NOTE: + In GitLab 13.7 and earlier, if you have a data type with zero items to sync + and don't skip the preflight checks, promoting the secondary reports + `ERROR - Replication is not up-to-date` even if replication is actually + up-to-date. If replication and verification output + shows that it is complete, you can skip the preflight checks to make the + command complete promotion. This bug was fixed in GitLab 13.8 and later. - You can also promote the secondary node to primary **without any further confirmation**, even when preflight checks fail: + - To promote the secondary node to primary **without any further confirmation**, + even when preflight checks fail: - ```shell - gitlab-ctl promote-to-primary-node --force - ``` + ```shell + gitlab-ctl promote-to-primary-node --force + ``` -1. Verify you can connect to the newly promoted **primary** node using the URL used +1. Verify you can connect to the newly-promoted **primary** node using the URL used previously for the **secondary** node. -1. If successful, the **secondary** node has now been promoted to the **primary** node. +1. If successful, the **secondary** node is now promoted to the **primary** node. #### Promoting a **secondary** node with multiple servers @@ -181,17 +158,6 @@ conjunction with multiple servers, as it can only perform changes on a **secondary** with only a single machine. Instead, you must do this manually. -WARNING: -In GitLab 13.2 and 13.3, promoting a secondary node to a primary while the -secondary is paused fails. Do not pause replication before promoting a -secondary. If the node is paused, be sure to resume before promoting. This -issue has been fixed in GitLab 13.4 and later. - -WARNING: -If the secondary node [has been paused](../../geo/index.md#pausing-and-resuming-replication), this performs -a point-in-time recovery to the last known state. -Data that was created on the primary while the secondary was paused will be lost. - 1. SSH in to the database node in the **secondary** and trigger PostgreSQL to promote to read-write: @@ -201,20 +167,17 @@ Data that was created on the primary while the secondary was paused will be lost In GitLab 12.8 and earlier, see [Message: `sudo: gitlab-pg-ctl: command not found`](../replication/troubleshooting.md#message-sudo-gitlab-pg-ctl-command-not-found). -1. Edit `/etc/gitlab/gitlab.rb` on every machine in the **secondary** to - reflect its new status as **primary** by removing any lines that enabled the - `geo_secondary_role`: +1. Edit `/etc/gitlab/gitlab.rb` on every node in the **secondary** site to + reflect its new status as **primary** by removing any of the following + lines that might be present: ```ruby - ## In pre-11.5 documentation, the role was enabled as follows. Remove this line. geo_secondary_role['enable'] = true - - ## In 11.5+ documentation, the role was enabled as follows. Remove this line. roles ['geo_secondary_role'] ``` - After making these changes [Reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) each - machine so the changes take effect. + After making these changes, [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) + on each machine so the changes take effect. 1. Promote the **secondary** to **primary**. SSH into a single application server and execute: @@ -223,9 +186,9 @@ Data that was created on the primary while the secondary was paused will be lost sudo gitlab-rake geo:set_secondary_as_primary ``` -1. Verify you can connect to the newly promoted **primary** using the URL used +1. Verify you can connect to the newly-promoted **primary** using the URL used previously for the **secondary**. -1. Success! The **secondary** has now been promoted to **primary**. +1. If successful, the **secondary** node is now promoted to the **primary** node. #### Promoting a **secondary** node with a Patroni standby cluster @@ -234,17 +197,6 @@ conjunction with a Patroni standby cluster, as it can only perform changes on a **secondary** with only a single machine. Instead, you must do this manually. -WARNING: -In GitLab 13.2 and 13.3, promoting a secondary node to a primary while the -secondary is paused fails. Do not pause replication before promoting a -secondary. If the node is paused, be sure to resume before promoting. This -issue has been fixed in GitLab 13.4 and later. - -WARNING: -If the secondary node [has been paused](../../geo/index.md#pausing-and-resuming-replication), this performs -a point-in-time recovery to the last known state. -Data that was created on the primary while the secondary was paused will be lost. - 1. SSH in to the Standby Leader database node in the **secondary** and trigger PostgreSQL to promote to read-write: @@ -252,13 +204,10 @@ Data that was created on the primary while the secondary was paused will be lost sudo gitlab-ctl promote-db ``` -1. Edit `/etc/gitlab/gitlab.rb` on every application and Sidekiq nodes in the secondary to reflect its new status as primary by removing any lines that enabled the `geo_secondary_role`: +1. Edit `/etc/gitlab/gitlab.rb` on every application and Sidekiq nodes in the secondary to reflect its new status as primary by removing any of the following lines that might be present: ```ruby - ## In pre-11.5 documentation, the role was enabled as follows. Remove this line. geo_secondary_role['enable'] = true - - ## In 11.5+ documentation, the role was enabled as follows. Remove this line. roles ['geo_secondary_role'] ``` @@ -280,9 +229,9 @@ Data that was created on the primary while the secondary was paused will be lost sudo gitlab-rake geo:set_secondary_as_primary ``` -1. Verify you can connect to the newly promoted **primary** using the URL used previously for the **secondary**. - -1. Success! The **secondary** has now been promoted to **primary**. +1. Verify you can connect to the newly-promoted **primary** using the URL used + previously for the **secondary**. +1. If successful, the **secondary** node is now promoted to the **primary** node. #### Promoting a **secondary** node with an external PostgreSQL database @@ -292,12 +241,12 @@ node with GitLab and the database on the same machine. As a result, a manual pro required: 1. Promote the replica database associated with the **secondary** site. This will - set the database to read-write: - - Amazon RDS - [Promoting a Read Replica to Be a Standalone DB Instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ReadRepl.html#USER_ReadRepl.Promote) - - Azure Database for PostgreSQL - [Stop replication](https://docs.microsoft.com/en-us/azure/postgresql/howto-read-replicas-portal#stop-replication) - - Other external PostgreSQL databases - save the script below in you secondary node, for example - `/tmp/geo_promote.sh`, and modify the connection parameters to match your - environment. Then, execute it to promote the replica: + set the database to read-write. The instructions vary depending on where your database is hosted: + - [Amazon RDS](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ReadRepl.html#USER_ReadRepl.Promote) + - [Azure PostgreSQL](https://docs.microsoft.com/en-us/azure/postgresql/howto-read-replicas-portal#stop-replication) + - For other external PostgreSQL databases, save the following script in you + secondary node, for example `/tmp/geo_promote.sh`, and modify the connection + parameters to match your environment. Then, execute it to promote the replica: ```shell #!/bin/bash @@ -318,14 +267,11 @@ required: ``` 1. Edit `/etc/gitlab/gitlab.rb` on every node in the **secondary** site to - reflect its new status as **primary** by removing any lines that enabled the - `geo_secondary_role`: + reflect its new status as **primary** by removing any of the following + lines that might be present: ```ruby - ## In GitLab 11.4 and earlier, remove this line. geo_secondary_role['enable'] = true - - ## In GitLab 11.5 and later, remove this line. roles ['geo_secondary_role'] ``` @@ -339,10 +285,9 @@ required: sudo gitlab-rake geo:set_secondary_as_primary ``` -1. Verify you can connect to the newly promoted **primary** site using the URL used - previously for the **secondary** site. - -1. Success! The **secondary** site has now been promoted to **primary**. +1. Verify you can connect to the newly-promoted **primary** using the URL used + previously for the **secondary**. +1. If successful, the **secondary** node is now promoted to the **primary** node. ### Step 4. (Optional) Updating the primary domain DNS record @@ -443,7 +388,7 @@ and after that you also need two extra steps. sudo -i ``` -1. Edit `/etc/gitlab/gitlab.rb` +1. Edit `/etc/gitlab/gitlab.rb`: ```ruby ## Enable a Geo Primary role (if you haven't yet) @@ -468,7 +413,7 @@ and after that you also need two extra steps. (For more details about these settings you can read [Configure the primary server](../setup/database.md#step-1-configure-the-primary-server)) 1. Save the file and reconfigure GitLab for the database listen changes and - the replication slot changes to be applied. + the replication slot changes to be applied: ```shell gitlab-ctl reconfigure @@ -501,6 +446,134 @@ Now we need to make each **secondary** node listen to changes on the new **prima to [initiate the replication process](../setup/database.md#step-3-initiate-the-replication-process) again but this time for another **primary** node. All the old replication settings will be overwritten. +## Promoting a secondary Geo cluster in GitLab Cloud Native Helm Charts + +When updating a Cloud Native Geo deployment, the process for updating any node that is external to the secondary Kubernetes cluster does not differ from the non Cloud Native approach. As such, you can always defer to [Promoting a secondary Geo node in single-secondary configurations](#promoting-a-secondary-geo-node-in-single-secondary-configurations) for more information. + +The following sections assume you are using the `gitlab` namespace. If you used a different namespace when setting up your cluster, you should also replace `--namespace gitlab` with your namespace. + +WARNING: +In GitLab 13.2 and 13.3, promoting a secondary site to a primary while the +secondary is paused fails. Do not pause replication before promoting a +secondary. If the site is paused, be sure to resume before promoting. This +issue has been fixed in GitLab 13.4 and later. + +### Step 1. Permanently disable the **primary** cluster + +WARNING: +If the **primary** site goes offline, there may be data saved on the **primary** site +that has not been replicated to the **secondary** site. This data should be treated +as lost if you proceed. + +If an outage on the **primary** site happens, you should do everything possible to +avoid a split-brain situation where writes can occur in two different GitLab +instances, complicating recovery efforts. So to prepare for the failover, you +must disable the **primary** site: + +- If you have access to the **primary** Kubernetes cluster, connect to it and disable the GitLab webservice and Sidekiq pods: + + ```shell + kubectl --namespace gitlab scale deploy gitlab-geo-webservice-default --replicas=0 + kubectl --namespace gitlab scale deploy gitlab-geo-sidekiq-all-in-1-v1 --replicas=0 + ``` + +- If you do not have access to the **primary** Kubernetes cluster, take the cluster offline and + prevent it from coming back online by any means at your disposal. + Since there are many ways you may prefer to accomplish this, we will avoid a + single recommendation. You may need to: + + - Reconfigure the load balancers. + - Change DNS records (for example, point the primary DNS record to the + **secondary** site to stop usage of the **primary** site). + - Stop the virtual servers. + - Block traffic through a firewall. + - Revoke object storage permissions from the **primary** site. + - Physically disconnect a machine. + +### Step 2. Promote all **secondary** nodes external to the cluster + +WARNING: +If the secondary site [has been paused](../../geo/index.md#pausing-and-resuming-replication), this performs +a point-in-time recovery to the last known state. +Data that was created on the primary while the secondary was paused will be lost. + +1. SSH in to the database node in the **secondary** and trigger PostgreSQL to + promote to read-write: + + ```shell + sudo gitlab-ctl promote-db + ``` + + In GitLab 12.8 and earlier, see [Message: `sudo: gitlab-pg-ctl: command not found`](../replication/troubleshooting.md#message-sudo-gitlab-pg-ctl-command-not-found). + +1. Edit `/etc/gitlab/gitlab.rb` on the database node in the **secondary** site to + reflect its new status as **primary** by removing any lines that enabled the + `geo_secondary_role`: + + NOTE: + Depending on your architecture these steps will need to be run on any GitLab node that is external to the **secondary** Kubernetes cluster. + + ```ruby + ## In pre-11.5 documentation, the role was enabled as follows. Remove this line. + geo_secondary_role['enable'] = true + + ## In 11.5+ documentation, the role was enabled as follows. Remove this line. + roles ['geo_secondary_role'] + ``` + + After making these changes, [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) on the database node. + +### Step 3. Promote the **secondary** cluster + +1. Find the task runner pod: + + ```shell + kubectl --namespace gitlab get pods -lapp=task-runner + ``` + +1. Promote the secondary: + + ```shell + kubectl --namespace gitlab exec -ti gitlab-geo-task-runner-XXX -- gitlab-rake geo:set_secondary_as_primary + ``` + +1. Update the existing cluster configuration. + + You can retrieve the existing config with Helm: + + ```shell + helm --namespace gitlab get values gitlab-geo > gitlab.yaml + ``` + + The existing config will contain a section for Geo that should resemble: + + ```yaml + geo: + enabled: true + role: secondary + nodeName: secondary.example.com + psql: + host: geo-2.db.example.com + port: 5431 + password: + secret: geo + key: geo-postgresql-password + ``` + + To promote the **secondary** cluster to a **primary** cluster, update `role: secondary` to `role: primary`. + + You can remove the entire `psql` section if the cluster will remain as a primary site, this refers to the tracking database and will be ignored whilst the cluster is acting as a primary site. + + Update the cluster with the new config: + + ```shell + helm upgrade --install --version <current Chart version> gitlab-geo gitlab/gitlab --namespace gitlab -f gitlab.yaml + ``` + +1. Verify you can connect to the newly promoted primary using the URL used previously for the secondary. + +1. Success! The secondary has now been promoted to primary. + ## Troubleshooting This section was moved to [another location](../replication/troubleshooting.md#fixing-errors-during-a-failover-or-when-promoting-a-secondary-to-a-primary-node). diff --git a/doc/administration/geo/glossary.md b/doc/administration/geo/glossary.md index 98a29c27a89..1ec552326aa 100644 --- a/doc/administration/geo/glossary.md +++ b/doc/administration/geo/glossary.md @@ -6,7 +6,7 @@ type: howto --- -# Geo Glossary +# Geo Glossary **(PREMIUM SELF)** NOTE: We are updating the Geo documentation, user interface and commands to reflect these changes. Not all pages comply with diff --git a/doc/administration/geo/index.md b/doc/administration/geo/index.md index 296500e35e2..f1884f297e8 100644 --- a/doc/administration/geo/index.md +++ b/doc/administration/geo/index.md @@ -54,7 +54,7 @@ Geo provides: ### Gitaly Cluster Geo should not be confused with [Gitaly Cluster](../gitaly/praefect.md). For more information about -the difference between Geo and Gitaly Cluster, see [Gitaly Cluster compared to Geo](../gitaly/praefect.md#gitaly-cluster-compared-to-geo). +the difference between Geo and Gitaly Cluster, see [Gitaly Cluster compared to Geo](../gitaly/index.md#gitaly-cluster-compared-to-geo). ## How it works @@ -281,7 +281,6 @@ WARNING: This list of limitations only reflects the latest version of GitLab. If you are using an older version, extra limitations may be in place. - Pushing directly to a **secondary** node redirects (for HTTP) or proxies (for SSH) the request to the **primary** node instead of [handling it directly](https://gitlab.com/gitlab-org/gitlab/-/issues/1381), except when using Git over HTTP with credentials embedded within the URI. For example, `https://user:password@secondary.tld`. -- Cloning, pulling, or pushing repositories that exist on the **primary** node but not on the **secondary** nodes where [selective synchronization](replication/configuration.md#selective-synchronization) does not include the project is not supported over SSH [but support is planned](https://gitlab.com/groups/gitlab-org/-/epics/2562). HTTP(S) is supported. - The **primary** node has to be online for OAuth login to happen. Existing sessions and Git are not affected. Support for the **secondary** node to use an OAuth provider independent from the primary is [being planned](https://gitlab.com/gitlab-org/gitlab/-/issues/208465). - The installation takes multiple manual steps that together can take about an hour depending on circumstances. We are working on improving this experience. See [Omnibus GitLab issue #2978](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/2978) for details. - Real-time updates of issues/merge requests (for example, via long polling) doesn't work on the **secondary** node. @@ -289,6 +288,7 @@ This list of limitations only reflects the latest version of GitLab. If you are - Object pools for forked project deduplication work only on the **primary** node, and are duplicated on the **secondary** node. - GitLab Runners cannot register with a **secondary** node. Support for this is [planned for the future](https://gitlab.com/gitlab-org/gitlab/-/issues/3294). - Geo **secondary** nodes can not be configured to [use high-availability configurations of PostgreSQL](https://gitlab.com/groups/gitlab-org/-/epics/2536). +- [Selective synchronization](replication/configuration.md#selective-synchronization) only limits what repositories are replicated. The entire PostgreSQL data is still replicated. Selective synchronization is not built to accomodate compliance / export control use cases. ### Limitations on replication/verification diff --git a/doc/administration/geo/replication/datatypes.md b/doc/administration/geo/replication/datatypes.md index dfb460c9f46..61f99257844 100644 --- a/doc/administration/geo/replication/datatypes.md +++ b/doc/administration/geo/replication/datatypes.md @@ -33,8 +33,9 @@ verification methods: | Git | Project wiki repository | Geo with Gitaly | Gitaly Checksum | | Git | Project designs repository | Geo with Gitaly | Gitaly Checksum | | Git | Object pools for forked project deduplication | Geo with Gitaly | _Not implemented_ | -| Git | Project Snippets | Geo with Gitaly | _Not implemented_ | -| Git | Personal Snippets | Geo with Gitaly | _Not implemented_ | +| Git | Project Snippets | Geo with Gitaly | Gitaly Checksum | +| Git | Personal Snippets | Geo with Gitaly | Gitaly Checksum | +| Git | Group wiki repository | Geo with Gitaly | _Not implemented_ | | Blobs | User uploads _(file system)_ | Geo with API | _Not implemented_ | | Blobs | User uploads _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | | Blobs | LFS objects _(file system)_ | Geo with API | _Not implemented_ | @@ -51,8 +52,10 @@ verification methods: | Blobs | Versioned Terraform State _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | | Blobs | External Merge Request Diffs _(file system)_ | Geo with API | _Not implemented_ | | Blobs | External Merge Request Diffs _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | +| Blobs | Pipeline artifacts _(file system)_ | Geo with API | SHA256 checksum | +| Blobs | Pipeline artifacts _(object storage)_ | Geo with API/Managed (*2*) | SHA256 checksum | -- (*1*): Redis replication can be used as part of HA with Redis sentinel. It's not used between Geo nodes. +- (*1*): Redis replication can be used as part of HA with Redis sentinel. It's not used between Geo sites. - (*2*): Object storage replication can be performed by Geo or by your object storage provider/appliance native replication feature. @@ -68,7 +71,7 @@ performance limitations when using a remote file system). Communication is done via Gitaly's own gRPC API. There are three possible ways of synchronization: -- Using regular Git clone/fetch from one Geo node to another (with special authentication). +- Using regular Git clone/fetch from one Geo site to another (with special authentication). - Using repository snapshots (for when the first method fails or repository is corrupt). - Manual trigger from the Admin Area (a combination of both of the above). @@ -82,7 +85,7 @@ They all live in the same shard and share the same base name with a `-wiki` and for Wiki and Design Repository cases. Besides that, there are snippet repositories. They can be connected to a project or to some specific user. -Both types will be synced to a secondary node. +Both types will be synced to a secondary site. ### Blobs @@ -95,7 +98,7 @@ GitLab stores files and blobs such as Issue attachments or LFS objects into eith - A Storage Appliance that exposes an Object Storage-compatible API. When using the file system store instead of Object Storage, you need to use network mounted file systems -to run GitLab when using more than one server. +to run GitLab when using more than one node. With respect to replication and verification: @@ -113,10 +116,10 @@ as well as permissions and credentials. PostgreSQL can also hold some level of cached data like HTML rendered Markdown, cached merge-requests diff (this can also be configured to be offloaded to object storage). -We use PostgreSQL's own replication functionality to replicate data from the **primary** to **secondary** nodes. +We use PostgreSQL's own replication functionality to replicate data from the **primary** to **secondary** sites. We use Redis both as a cache store and to hold persistent data for our background jobs system. Because both -use-cases has data that are exclusive to the same Geo node, we don't replicate it between nodes. +use-cases have data that are exclusive to the same Geo site, we don't replicate it between sites. Elasticsearch is an optional database, that can enable advanced searching capabilities, like improved Advanced Search in both source-code level and user generated content in Issues / Merge-Requests and discussions. Currently it's not @@ -125,7 +128,7 @@ supported in Geo. ## Limitations on replication/verification The following table lists the GitLab features along with their replication -and verification status on a **secondary** node. +and verification status on a **secondary** site. You can keep track of the progress to implement the missing items in these epics/issues: @@ -165,40 +168,40 @@ Feature.enable(:geo_package_file_replication) WARNING: Features not on this list, or with **No** in the **Replicated** column, -are not replicated on the **secondary** node. Failing over without manually +are not replicated to a **secondary** site. Failing over without manually replicating data from those features will cause the data to be **lost**. -If you wish to use those features on a **secondary** node, or to execute a failover +If you wish to use those features on a **secondary** site, or to execute a failover successfully, you must replicate their data using some other means. -| Feature | Replicated (added in GitLab version) | Verified (added in GitLab version) | Object Storage replication (see [Geo with Object Storage](object_storage.md)) | Notes | -|:---------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------|:----------------------------------------------------------|:-------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| [Application data in PostgreSQL](../../postgresql/index.md) | **Yes** (10.2) | **Yes** (10.2) | No | | -| [Project repository](../../../user/project/repository/) | **Yes** (10.2) | **Yes** (10.7) | No | | -| [Project wiki repository](../../../user/project/wiki/) | **Yes** (10.2) | **Yes** (10.7) | No | -| [Group wiki repository](../../../user/group/index.md#group-wikis) | [**Yes** (13.10)](https://gitlab.com/gitlab-org/gitlab/-/issues/208147) | No | No | Behind feature flag `geo_group_wiki_repository_replication`, enabled by default | -| [Uploads](../../uploads.md) | **Yes** (10.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | No | Verified only on transfer or manually using [Integrity Check Rake Task](../../raketasks/check.md) on both nodes and comparing the output between them. | -| [LFS objects](../../lfs/index.md) | **Yes** (10.2) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8922) | Via Object Storage provider if supported. Native Geo support (Beta). | Verified only on transfer or manually using [Integrity Check Rake Task](../../raketasks/check.md) on both nodes and comparing the output between them. GitLab versions 11.11.x and 12.0.x are affected by [a bug that prevents any new LFS objects from replicating](https://gitlab.com/gitlab-org/gitlab/-/issues/32696). | -| [Personal snippets](../../../user/snippets.md) | **Yes** (10.2) | **Yes** (10.2) | No | | -| [Project snippets](../../../user/snippets.md) | **Yes** (10.2) | **Yes** (10.2) | No | | -| [CI job artifacts (other than Job Logs)](../../../ci/pipelines/job_artifacts.md) | **Yes** (10.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8923) | Via Object Storage provider if supported. Native Geo support (Beta) . | Verified only manually using [Integrity Check Rake Task](../../raketasks/check.md) on both nodes and comparing the output between them | -| [Job logs](../../job_logs.md) | **Yes** (10.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8923) | Via Object Storage provider if supported. Native Geo support (Beta). | Verified only on transfer or manually using [Integrity Check Rake Task](../../raketasks/check.md) on both nodes and comparing the output between them | -| [Object pools for forked project deduplication](../../../development/git_object_deduplication.md) | **Yes** | No | No | | -| [Container Registry](../../packages/container_registry.md) | **Yes** (12.3) | No | No | Disabled by default. See [instructions](docker_registry.md) to enable. | -| [Content in object storage (beta)](object_storage.md) | **Yes** (12.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/13845) | No | | -| [Project designs repository](../../../user/project/issues/design_management.md) | **Yes** (12.7) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/32467) | Via Object Storage provider if supported. Native Geo support (Beta). | | -| [Package Registry for npm](../../../user/packages/npm_registry/index.md) | **Yes** (13.2) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | -| [Package Registry for Maven](../../../user/packages/maven_repository/index.md) | **Yes** (13.2) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | -| [Package Registry for Conan](../../../user/packages/conan_repository/index.md) | **Yes** (13.2) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | -| [Package Registry for NuGet](../../../user/packages/nuget_repository/index.md) | **Yes** (13.2) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | -| [Package Registry for PyPI](../../../user/packages/pypi_repository/index.md) | **Yes** (13.2) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | -| [Package Registry for Composer](../../../user/packages/composer_repository/index.md) | **Yes** (13.2) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | -| [Package Registry for generic packages](../../../user/packages/generic_packages/index.md) | **Yes** (13.5) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | -| [Versioned Terraform State](../../terraform_state.md) | **Yes** (13.5) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_terraform_state_version_replication`, enabled by default | -| [External merge request diffs](../../merge_request_diffs.md) | **Yes** (13.5) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_merge_request_diff_replication`, enabled by default | -| [Versioned snippets](../../../user/snippets.md#versioned-snippets) | [**Yes** (13.7)](https://gitlab.com/groups/gitlab-org/-/epics/2809) | [No](https://gitlab.com/groups/gitlab-org/-/epics/2810) | No | | -| [Server-side Git hooks](../../server_hooks.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1867) | No | No | | -| [Elasticsearch integration](../../../integration/elasticsearch.md) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/1186) | No | No | | -| [GitLab Pages](../../pages/index.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/589) | No | No | | -| [CI Pipeline Artifacts](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/ci/pipeline_artifact.rb) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/238464) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Persists additional artifacts after a pipeline completes | -| [Dependency proxy images](../../../user/packages/dependency_proxy/index.md) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/259694) | No | No | Blocked on [Geo: Secondary Mimicry](https://gitlab.com/groups/gitlab-org/-/epics/1528). Note that replication of this cache is not needed for Disaster Recovery purposes because it can be recreated from external sources. | -| [Vulnerability Export](../../../user/application_security/vulnerability_report/#export-vulnerability-details) | [Not planned](https://gitlab.com/groups/gitlab-org/-/epics/3111) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Not planned because they are ephemeral and sensitive. They can be regenerated on demand. | +|Feature | Replicated (added in GitLab version) | Verified (added in GitLab version) | Object Storage replication (see [Geo with Object Storage](object_storage.md)) | Notes | +|:--------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------|:------------------------------------------------------------------------|:------------------------------------------------------------------------------|:------| +|[Application data in PostgreSQL](../../postgresql/index.md) | **Yes** (10.2) | **Yes** (10.2) | No | | +|[Project repository](../../../user/project/repository/) | **Yes** (10.2) | **Yes** (10.7) | No | | +|[Project wiki repository](../../../user/project/wiki/) | **Yes** (10.2) | **Yes** (10.7) | No | | +|[Group wiki repository](../../../user/project/wiki/index.md#group-wikis) | [**Yes** (13.10)](https://gitlab.com/gitlab-org/gitlab/-/issues/208147) | No | No | Behind feature flag `geo_group_wiki_repository_replication`, enabled by default. | +|[Uploads](../../uploads.md) | **Yes** (10.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | No | Verified only on transfer or manually using [Integrity Check Rake Task](../../raketasks/check.md) on both sites and comparing the output between them. | +|[LFS objects](../../lfs/index.md) | **Yes** (10.2) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8922) | Via Object Storage provider if supported. Native Geo support (Beta). | Verified only on transfer or manually using [Integrity Check Rake Task](../../raketasks/check.md) on both sites and comparing the output between them. GitLab versions 11.11.x and 12.0.x are affected by [a bug that prevents any new LFS objects from replicating](https://gitlab.com/gitlab-org/gitlab/-/issues/32696). | +|[Personal snippets](../../../user/snippets.md) | **Yes** (10.2) | **Yes** (10.2) | No | | +|[Project snippets](../../../user/snippets.md) | **Yes** (10.2) | **Yes** (10.2) | No | | +|[CI job artifacts (other than Job Logs)](../../../ci/pipelines/job_artifacts.md) | **Yes** (10.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8923) | Via Object Storage provider if supported. Native Geo support (Beta). | Verified only manually using [Integrity Check Rake Task](../../raketasks/check.md) on both sites and comparing the output between them. | +|[CI Pipeline Artifacts](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/ci/pipeline_artifact.rb) | [**Yes** (13.11)](https://gitlab.com/gitlab-org/gitlab/-/issues/238464) | [**Yes** (13.11)](https://gitlab.com/gitlab-org/gitlab/-/issues/238464) | Via Object Storage provider if supported. Native Geo support (Beta). | Persists additional artifacts after a pipeline completes | +|[Job logs](../../job_logs.md) | **Yes** (10.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8923) | Via Object Storage provider if supported. Native Geo support (Beta). | Verified only on transfer or manually using [Integrity Check Rake Task](../../raketasks/check.md) on both sites and comparing the output between them. | +|[Object pools for forked project deduplication](../../../development/git_object_deduplication.md) | **Yes** | No | No | | +|[Container Registry](../../packages/container_registry.md) | **Yes** (12.3) | No | No | Disabled by default. See [instructions](docker_registry.md) to enable. | +|[Content in object storage (beta)](object_storage.md) | **Yes** (12.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/13845) | No | | +|[Project designs repository](../../../user/project/issues/design_management.md) | **Yes** (12.7) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/32467) | Via Object Storage provider if supported. Native Geo support (Beta). | | +|[Package Registry for npm](../../../user/packages/npm_registry/index.md) | **Yes** (13.2) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | +|[Package Registry for Maven](../../../user/packages/maven_repository/index.md) | **Yes** (13.2) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | +|[Package Registry for Conan](../../../user/packages/conan_repository/index.md) | **Yes** (13.2) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | +|[Package Registry for NuGet](../../../user/packages/nuget_repository/index.md) | **Yes** (13.2) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | +|[Package Registry for PyPI](../../../user/packages/pypi_repository/index.md) | **Yes** (13.2) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | +|[Package Registry for Composer](../../../user/packages/composer_repository/index.md) | **Yes** (13.2) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | +|[Package Registry for generic packages](../../../user/packages/generic_packages/index.md) | **Yes** (13.5) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | +|[Versioned Terraform State](../../terraform_state.md) | **Yes** (13.5) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_terraform_state_version_replication`, enabled by default. | +|[External merge request diffs](../../merge_request_diffs.md) | **Yes** (13.5) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_merge_request_diff_replication`, enabled by default. | +|[Versioned snippets](../../../user/snippets.md#versioned-snippets) | [**Yes** (13.7)](https://gitlab.com/groups/gitlab-org/-/epics/2809) | [No](https://gitlab.com/groups/gitlab-org/-/epics/2810) | No | | +|[Server-side Git hooks](../../server_hooks.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1867) | No | No | | +|[Elasticsearch integration](../../../integration/elasticsearch.md) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/1186) | No | No | | +|[GitLab Pages](../../pages/index.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/589) | No | No | | +|[Dependency proxy images](../../../user/packages/dependency_proxy/index.md) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/259694) | No | No | Blocked on [Geo: Secondary Mimicry](https://gitlab.com/groups/gitlab-org/-/epics/1528). Note that replication of this cache is not needed for Disaster Recovery purposes because it can be recreated from external sources. | +|[Vulnerability Export](../../../user/application_security/vulnerability_report/#export-vulnerability-details) | [Not planned](https://gitlab.com/groups/gitlab-org/-/epics/3111) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Not planned because they are ephemeral and sensitive. They can be regenerated on demand. | diff --git a/doc/administration/geo/replication/disable_geo.md b/doc/administration/geo/replication/disable_geo.md index a26adcefef5..c71cf80d0c1 100644 --- a/doc/administration/geo/replication/disable_geo.md +++ b/doc/administration/geo/replication/disable_geo.md @@ -12,26 +12,26 @@ situation and you want to disable Geo momentarily, you can use these instruction Geo setup. There should be no functional difference between disabling Geo and having an active Geo setup with -no secondary Geo nodes if you remove them correctly. +no secondary Geo sites if you remove them correctly. To disable Geo, follow these steps: -1. [Remove all secondary Geo nodes](#remove-all-secondary-geo-nodes). -1. [Remove the primary node from the UI](#remove-the-primary-node-from-the-ui). +1. [Remove all secondary Geo sites](#remove-all-secondary-geo-sites). +1. [Remove the primary site from the UI](#remove-the-primary-site-from-the-ui). 1. [Remove secondary replication slots](#remove-secondary-replication-slots). 1. [Remove Geo-related configuration](#remove-geo-related-configuration). 1. [(Optional) Revert PostgreSQL settings to use a password and listen on an IP](#optional-revert-postgresql-settings-to-use-a-password-and-listen-on-an-ip). -## Remove all secondary Geo nodes +## Remove all secondary Geo sites -To disable Geo, you need to first remove all your secondary Geo nodes, which means replication will not happen -anymore on these nodes. You can follow our docs to [remove your secondary Geo nodes](remove_geo_node.md). +To disable Geo, you need to first remove all your secondary Geo sites, which means replication will not happen +anymore on these sites. You can follow our docs to [remove your secondary Geo sites](remove_geo_site.md). -If the current node that you want to keep using is a secondary node, you need to first promote it to primary. -You can use our steps on [how to promote a secondary node](../disaster_recovery/#step-3-promoting-a-secondary-node) +If the current site that you want to keep using is a secondary site, you need to first promote it to primary. +You can use our steps on [how to promote a secondary site](../disaster_recovery/#step-3-promoting-a-secondary-node) to do that. -## Remove the primary node from the UI +## Remove the primary site from the UI 1. Go to **Admin Area > Geo** (`/admin/geo/nodes`). 1. Click the **Remove** button for the **primary** node. @@ -59,7 +59,7 @@ Geo node in a PostgreSQL console (`sudo gitlab-psql`): ## Remove Geo-related configuration -1. SSH into your primary Geo node and log in as root: +1. For each node on your primary Geo site, SSH into the node and log in as root: ```shell sudo -i diff --git a/doc/administration/geo/replication/docker_registry.md b/doc/administration/geo/replication/docker_registry.md index 1669abbc52a..ea73614511f 100644 --- a/doc/administration/geo/replication/docker_registry.md +++ b/doc/administration/geo/replication/docker_registry.md @@ -5,16 +5,16 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Docker Registry for a secondary node **(PREMIUM SELF)** +# Docker Registry for a secondary site **(PREMIUM SELF)** You can set up a [Docker Registry](https://docs.docker.com/registry/) on your -**secondary** Geo node that mirrors the one on the **primary** Geo node. +**secondary** Geo site that mirrors the one on the **primary** Geo site. ## Storage support Docker Registry currently supports a few types of storage. If you choose a distributed storage (`azure`, `gcs`, `s3`, `swift`, or `oss`) for your Docker -Registry on the **primary** node, you can use the same storage for a **secondary** +Registry on the **primary** site, you can use the same storage for a **secondary** Docker Registry as well. For more information, read the [Load balancing considerations](https://docs.docker.com/registry/deploying/#load-balancing-considerations) when deploying the Registry, and how to set up the storage driver for the GitLab @@ -24,22 +24,22 @@ integrated [Container Registry](../../packages/container_registry.md#use-object- You can enable a storage-agnostic replication so it can be used for cloud or local storage. Whenever a new image is pushed to the -**primary** node, each **secondary** node will pull it to its own container +**primary** site, each **secondary** site will pull it to its own container repository. To configure Docker Registry replication: -1. Configure the [**primary** node](#configure-primary-node). -1. Configure the [**secondary** node](#configure-secondary-node). +1. Configure the [**primary** site](#configure-primary-site). +1. Configure the [**secondary** site](#configure-secondary-site). 1. Verify Docker Registry [replication](#verify-replication). -### Configure **primary** node +### Configure **primary** site Make sure that you have Container Registry set up and working on -the **primary** node before following the next steps. +the **primary** site before following the next steps. We need to make Docker Registry send notification events to the -**primary** node. +**primary** site. 1. SSH into your GitLab **primary** server and login as root: @@ -85,27 +85,29 @@ We need to make Docker Registry send notification events to the gitlab-ctl reconfigure ``` -### Configure **secondary** node +### Configure **secondary** site Make sure you have Container Registry set up and working on -the **secondary** node before following the next steps. +the **secondary** site before following the next steps. -The following steps should be done on each **secondary** node you're +The following steps should be done on each **secondary** site you're expecting to see the Docker images replicated. -Because we need to allow the **secondary** node to communicate securely with -the **primary** node Container Registry, we need to have a single key -pair for all the nodes. The **secondary** node will use this key to +Because we need to allow the **secondary** site to communicate securely with +the **primary** site Container Registry, we need to have a single key +pair for all the sites. The **secondary** site will use this key to generate a short-lived JWT that is pull-only-capable to access the -**primary** node Container Registry. +**primary** site Container Registry. -1. SSH into the **secondary** node and login as the `root` user: +For each application and Sidekiq node on the **secondary** site: + +1. SSH into the node and login as the `root` user: ```shell sudo -i ``` -1. Copy `/var/opt/gitlab/gitlab-rails/etc/gitlab-registry.key` from the **primary** to the **secondary** node. +1. Copy `/var/opt/gitlab/gitlab-rails/etc/gitlab-registry.key` from the **primary** to the node. 1. Edit `/etc/gitlab/gitlab.rb`: @@ -114,7 +116,7 @@ generate a short-lived JWT that is pull-only-capable to access the gitlab_rails['geo_registry_replication_primary_api_url'] = 'https://primary.example.com:5050/' # Primary registry address, it will be used by the secondary node to directly communicate to primary registry ``` -1. Reconfigure the **secondary** node for the change to take effect: +1. Reconfigure the node for the change to take effect: ```shell gitlab-ctl reconfigure @@ -123,6 +125,6 @@ generate a short-lived JWT that is pull-only-capable to access the ### Verify replication To verify Container Registry replication is working, go to **Admin Area > Geo** -(`/admin/geo/nodes`) on the **secondary** node. +(`/admin/geo/nodes`) on the **secondary** site. The initial replication, or "backfill", will probably still be in progress. -You can monitor the synchronization process on each Geo node from the **primary** node's **Geo Nodes** dashboard in your browser. +You can monitor the synchronization process on each Geo site from the **primary** site's **Geo Nodes** dashboard in your browser. diff --git a/doc/administration/geo/replication/geo_validation_tests.md b/doc/administration/geo/replication/geo_validation_tests.md index f050d3e708c..06fd3cd70be 100644 --- a/doc/administration/geo/replication/geo_validation_tests.md +++ b/doc/administration/geo/replication/geo_validation_tests.md @@ -148,13 +148,13 @@ The following are PostgreSQL upgrade validation tests we performed. [PostgreSQL 11 upgrade procedure for Geo installations](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4975): - Description: Prior to making PostgreSQL 11 the default version of PostgreSQL in GitLab 12.10, we - tested upgrading to PostgreSQL 11 in Geo deployments on GitLab 12.9. + tested upgrading to PostgreSQL 11 in Geo deployments in GitLab 12.9. - Outcome: Partially successful. Issues were discovered in multi-node configurations with a separate tracking database and concerns were raised about allowing automatic upgrades when Geo enabled. - Follow up issues: - [`replicate-geo-database` incorrectly tries to back up repositories](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5241). - [`pg-upgrade` fails to upgrade a standalone Geo tracking database](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5242). - - [`revert-pg-upgrade` fails to downgrade the PostgreSQL data of a Geo secondary’s standalone tracking database](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5243). + - [`revert-pg-upgrade` fails to downgrade the PostgreSQL data of a Geo secondary's standalone tracking database](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5243). - [Timeout error on Geo secondary read-replica near the end of `gitlab-ctl pg-upgrade`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5235). [Verify Geo installation with PostgreSQL 11](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4971): diff --git a/doc/administration/geo/replication/remove_geo_node.md b/doc/administration/geo/replication/remove_geo_node.md index 06cf5375f0d..09ea84b6c4b 100644 --- a/doc/administration/geo/replication/remove_geo_node.md +++ b/doc/administration/geo/replication/remove_geo_node.md @@ -1,58 +1,8 @@ --- -stage: Enablement -group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: howto +redirect_to: '../../geo/replication/remove_geo_site.md' --- -# Removing secondary Geo nodes **(PREMIUM SELF)** +This document was moved to [another location](../../geo/replication/remove_geo_site.md). -**Secondary** nodes can be removed from the Geo cluster using the Geo administration page of the **primary** node. To remove a **secondary** node: - -1. Go to **Admin Area > Geo** (`/admin/geo/nodes`). -1. Click the **Remove** button for the **secondary** node you want to remove. -1. Confirm by clicking **Remove** when the prompt appears. - -Once removed from the Geo administration page, you must stop and uninstall the **secondary** node: - -1. On the **secondary** node, stop GitLab: - - ```shell - sudo gitlab-ctl stop - ``` - -1. On the **secondary** node, uninstall GitLab: - - ```shell - # Stop gitlab and remove its supervision process - sudo gitlab-ctl uninstall - - # Debian/Ubuntu - sudo dpkg --remove gitlab-ee - - # Redhat/Centos - sudo rpm --erase gitlab-ee - ``` - -Once GitLab has been uninstalled from the **secondary** node, the replication slot must be dropped from the **primary** node's database as follows: - -1. On the **primary** node, start a PostgreSQL console session: - - ```shell - sudo gitlab-psql - ``` - - NOTE: - Using `gitlab-rails dbconsole` will not work, because managing replication slots requires superuser permissions. - -1. Find the name of the relevant replication slot. This is the slot that is specified with `--slot-name` when running the replicate command: `gitlab-ctl replicate-geo-database`. - - ```sql - SELECT * FROM pg_replication_slots; - ``` - -1. Remove the replication slot for the **secondary** node: - - ```sql - SELECT pg_drop_replication_slot('<name_of_slot>'); - ``` +<!-- This redirect file can be deleted after 2022-04-01 --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/geo/replication/remove_geo_site.md b/doc/administration/geo/replication/remove_geo_site.md new file mode 100644 index 00000000000..a42a4c4eb47 --- /dev/null +++ b/doc/administration/geo/replication/remove_geo_site.md @@ -0,0 +1,58 @@ +--- +stage: Enablement +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +type: howto +--- + +# Removing secondary Geo sites **(PREMIUM SELF)** + +**Secondary** sites can be removed from the Geo cluster using the Geo administration page of the **primary** site. To remove a **secondary** site: + +1. Go to **Admin Area > Geo** (`/admin/geo/nodes`). +1. Select the **Remove** button for the **secondary** site you want to remove. +1. Confirm by selecting **Remove** when the prompt appears. + +Once removed from the Geo administration page, you must stop and uninstall the **secondary** site. For each node on your secondary Geo site: + +1. Stop GitLab: + + ```shell + sudo gitlab-ctl stop + ``` + +1. Uninstall GitLab: + + ```shell + # Stop gitlab and remove its supervision process + sudo gitlab-ctl uninstall + + # Debian/Ubuntu + sudo dpkg --remove gitlab-ee + + # Redhat/Centos + sudo rpm --erase gitlab-ee + ``` + +Once GitLab has been uninstalled from each node on the **secondary** site, the replication slot must be dropped from the **primary** site's database as follows: + +1. On the **primary** site's database node, start a PostgreSQL console session: + + ```shell + sudo gitlab-psql + ``` + + NOTE: + Using `gitlab-rails dbconsole` will not work, because managing replication slots requires superuser permissions. + +1. Find the name of the relevant replication slot. This is the slot that is specified with `--slot-name` when running the replicate command: `gitlab-ctl replicate-geo-database`. + + ```sql + SELECT * FROM pg_replication_slots; + ``` + +1. Remove the replication slot for the **secondary** site: + + ```sql + SELECT pg_drop_replication_slot('<name_of_slot>'); + ``` diff --git a/doc/administration/geo/replication/security_review.md b/doc/administration/geo/replication/security_review.md index c31881910bc..abb84b95623 100644 --- a/doc/administration/geo/replication/security_review.md +++ b/doc/administration/geo/replication/security_review.md @@ -34,7 +34,7 @@ from [owasp.org](https://owasp.org/). ### How can the data be classified into categories according to its sensitivity? - The GitLab model of sensitivity is centered around public vs. internal vs. - private projects. Geo replicates them all indiscriminately. “Selective sync” + private projects. Geo replicates them all indiscriminately. "Selective sync" exists for files and repositories (but not database content), which would permit only less-sensitive projects to be replicated to a **secondary** node if desired. - See also: [GitLab data classification policy](https://about.gitlab.com/handbook/engineering/security/data-classification-standard.html). @@ -81,7 +81,7 @@ from [owasp.org](https://owasp.org/). considered an administrator with super-user privileges. - See also: [more granular access control](https://gitlab.com/gitlab-org/gitlab/-/issues/18242) (not Geo-specific). -- Much of Geo’s integration (database replication, for instance) must be +- Much of Geo's integration (database replication, for instance) must be configured with the application, typically by system administrators. ### What administrative capabilities does the application offer? @@ -165,7 +165,7 @@ from [owasp.org](https://owasp.org/). ### What aspects of the product may or may not be hosted via the cloud computing model? -- GitLab is “cloud native” and this applies to Geo as much as to the rest of the +- GitLab is "cloud native" and this applies to Geo as much as to the rest of the product. Deployment in clouds is a common and supported scenario. ## If applicable, what approach(es) to cloud computing will be taken (Managed Hosting versus "Pure" Cloud, a "full machine" approach such as AWS-EC2 versus a "hosted database" approach such as AWS-RDS and Azure, etc)? @@ -223,7 +223,7 @@ from [owasp.org](https://owasp.org/). ### What data input validation requirements have been defined? -- **Secondary** nodes must have a faithful replication of the **primary** node’s data. +- **Secondary** nodes must have a faithful replication of the **primary** node's data. ### What data does the application store and how? @@ -235,7 +235,7 @@ from [owasp.org](https://owasp.org/). rest. A subset of database columns are encrypted at rest using the `db_otp_key`. - A static secret shared across all hosts in a GitLab deployment. - In transit, data should be encrypted, although the application does permit - communication to proceed unencrypted. The two main transits are the **secondary** node’s + communication to proceed unencrypted. The two main transits are the **secondary** node's replication process for PostgreSQL, and for Git repositories/files. Both should be protected using TLS, with the keys for that managed via Omnibus per existing configuration for end-user access to GitLab. diff --git a/doc/administration/geo/replication/usage.md b/doc/administration/geo/replication/usage.md new file mode 100644 index 00000000000..2fcc0565567 --- /dev/null +++ b/doc/administration/geo/replication/usage.md @@ -0,0 +1,27 @@ +--- +stage: Enablement +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +type: howto +--- + +<!-- Please update EE::GitLab::GeoGitAccess::GEO_SERVER_DOCS_URL if this file is moved) --> + +# Using a Geo Site **(PREMIUM SELF)** + +After you set up the [database replication and configure the Geo nodes](../index.md#setup-instructions), use your closest GitLab site as you would do with the primary one. + +You can push directly to a **secondary** site (for both HTTP, SSH including Git LFS), and the request will be proxied to the primary site instead ([introduced](https://about.gitlab.com/releases/2018/09/22/gitlab-11-3-released/) in [GitLab Premium](https://about.gitlab.com/pricing/#self-managed) 11.3). + +Example of the output you will see when pushing to a **secondary** site: + +```shell +$ git push +remote: +remote: This request to a Geo secondary node will be forwarded to the +remote: Geo primary node: +remote: +remote: ssh://git@primary.geo/user/repo.git +remote: +Everything up-to-date +``` diff --git a/doc/administration/geo/replication/using_a_geo_server.md b/doc/administration/geo/replication/using_a_geo_server.md index 7578b6bf61a..e48e750f710 100644 --- a/doc/administration/geo/replication/using_a_geo_server.md +++ b/doc/administration/geo/replication/using_a_geo_server.md @@ -1,27 +1,8 @@ --- -stage: Enablement -group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: howto +redirect_to: '../../geo/replication/usage.md' --- -<!-- Please update EE::GitLab::GeoGitAccess::GEO_SERVER_DOCS_URL if this file is moved) --> +This document was moved to [another location](../../geo/replication/usage.md). -# Using a Geo Server **(PREMIUM SELF)** - -After you set up the [database replication and configure the Geo nodes](../index.md#setup-instructions), use your closest GitLab node as you would a normal standalone GitLab instance. - -Pushing directly to a **secondary** node (for both HTTP, SSH including Git LFS) was [introduced](https://about.gitlab.com/releases/2018/09/22/gitlab-11-3-released/) in [GitLab Premium](https://about.gitlab.com/pricing/#self-managed) 11.3. - -Example of the output you will see when pushing to a **secondary** node: - -```shell -$ git push -remote: -remote: You're pushing to a Geo secondary. We'll help you by proxying this -remote: request to the primary: -remote: -remote: ssh://git@primary.geo/user/repo.git -remote: -Everything up-to-date -``` +<!-- This redirect file can be deleted after 2022-04-01 --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/geo/replication/version_specific_updates.md b/doc/administration/geo/replication/version_specific_updates.md index 883e5d44b69..4f0a4dc638c 100644 --- a/doc/administration/geo/replication/version_specific_updates.md +++ b/doc/administration/geo/replication/version_specific_updates.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Version-specific update instructions +# Version-specific update instructions **(PREMIUM SELF)** Review this page for update instructions for your version. These steps accompany the [general steps](updating_the_geo_nodes.md#general-update-steps) diff --git a/doc/administration/geo/setup/database.md b/doc/administration/geo/setup/database.md index 7128d4283d1..9c2cc8fc62e 100644 --- a/doc/administration/geo/setup/database.md +++ b/doc/administration/geo/setup/database.md @@ -672,7 +672,7 @@ For each Patroni instance on the secondary site: ## Migrating from repmgr to Patroni 1. Before migrating, it is recommended that there is no replication lag between the primary and secondary sites and that replication is paused. In GitLab 13.2 and later, you can pause and resume replication with `gitlab-ctl geo-replication-pause` and `gitlab-ctl geo-replication-resume` on a Geo secondary database node. -1. Follow the [instructions to migrate repmgr to Patroni](../../postgresql/replication_and_failover.md#switching-from-repmgr-to-patroni). When configuring Patroni on each primary site database node, add `patroni['replicaton_slots'] = { '<slot_name>' => 'physical' }` +1. Follow the [instructions to migrate repmgr to Patroni](../../postgresql/replication_and_failover.md#switching-from-repmgr-to-patroni). When configuring Patroni on each primary site database node, add `patroni['replication_slots'] = { '<slot_name>' => 'physical' }` to `gitlab.rb` where `<slot_name>` is the name of the replication slot for your Geo secondary. This will ensure that Patroni recognizes the replication slot as permanent and will not drop it upon restarting. 1. If database replication to the secondary was paused before migration, resume replication once Patroni is confirmed working on the primary. diff --git a/doc/administration/geo/setup/index.md b/doc/administration/geo/setup/index.md index 8308cbfa82e..5ec18e29f21 100644 --- a/doc/administration/geo/setup/index.md +++ b/doc/administration/geo/setup/index.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Setting up Geo +# Setting up Geo **(PREMIUM SELF)** These instructions assume you have a working instance of GitLab. They guide you through: |