diff options
| author | GitLab Bot <gitlab-bot@gitlab.com> | 2021-08-19 09:08:42 +0000 |
|---|---|---|
| committer | GitLab Bot <gitlab-bot@gitlab.com> | 2021-08-19 09:08:42 +0000 |
| commit | b76ae638462ab0f673e5915986070518dd3f9ad3 (patch) | |
| tree | bdab0533383b52873be0ec0eb4d3c66598ff8b91 /doc/administration/geo | |
| parent | 434373eabe7b4be9593d18a585fb763f1e5f1a6f (diff) | |
| download | gitlab-ce-b76ae638462ab0f673e5915986070518dd3f9ad3.tar.gz | |
Add latest changes from gitlab-org/gitlab@14-2-stable-eev14.2.0-rc42
Diffstat (limited to 'doc/administration/geo')
16 files changed, 277 insertions, 355 deletions
diff --git a/doc/administration/geo/disaster_recovery/background_verification.md b/doc/administration/geo/disaster_recovery/background_verification.md index f03cd64c14e..65e0ffd4366 100644 --- a/doc/administration/geo/disaster_recovery/background_verification.md +++ b/doc/administration/geo/disaster_recovery/background_verification.md @@ -161,7 +161,7 @@ If the **primary** and **secondary** nodes have a checksum verification mismatch  1. Go to the project's repository directory on both **primary** and **secondary** nodes - (the path is usually `/var/opt/gitlab/git-data/repositories`). Note that if `git_data_dirs` + (the path is usually `/var/opt/gitlab/git-data/repositories`). If `git_data_dirs` is customized, check the directory layout on your server to be sure: ```shell @@ -198,7 +198,7 @@ nodes, and comparing the output between them. In GitLab EE 12.1, Geo calculates checksums for attachments, LFS objects, and archived traces on secondary nodes after the transfer, compares it with the -stored checksums, and rejects transfers if mismatched. Please note that Geo +stored checksums, and rejects transfers if mismatched. Geo currently does not support an automatic way to verify these data if they have been synced before GitLab EE 12.1. diff --git a/doc/administration/geo/disaster_recovery/planned_failover.md b/doc/administration/geo/disaster_recovery/planned_failover.md index 5c15523ac78..8b55bebb42b 100644 --- a/doc/administration/geo/disaster_recovery/planned_failover.md +++ b/doc/administration/geo/disaster_recovery/planned_failover.md @@ -118,7 +118,7 @@ On the **secondary** node: objects aren't yet replicated (shown in gray), consider giving the node more time to complete -  +  If any objects are failing to replicate, this should be investigated before scheduling the maintenance window. Following a planned failover, anything that diff --git a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md index 16ae5bde062..27990748071 100644 --- a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md +++ b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md @@ -72,7 +72,7 @@ On the **secondary** node: objects aren't yet replicated (shown in gray), consider giving the node more time to complete. -  +  If any objects are failing to replicate, this should be investigated before scheduling the maintenance window. After a planned failover, anything that @@ -94,7 +94,7 @@ follow these steps to avoid unnecessary data loss: 1. Until a [read-only mode](https://gitlab.com/gitlab-org/gitlab/-/issues/14609) is implemented, updates must be prevented from happening manually to the - **primary**. Note that your **secondary** node still needs read-only + **primary**. Your **secondary** node still needs read-only access to the **primary** node during the maintenance window: 1. At the scheduled time, using your cloud provider or your node's firewall, block diff --git a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md index 36c9d46d650..9d5e65cd194 100644 --- a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md +++ b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md @@ -57,7 +57,7 @@ and there should be no failures (shown in red). If a large proportion of objects aren't yet replicated (shown in gray), consider giving the node more time to complete. - + If any objects are failing to replicate, this should be investigated before scheduling the maintenance window. After a planned failover, anything that @@ -79,7 +79,7 @@ follow these steps to avoid unnecessary data loss: 1. Until a [read-only mode](https://gitlab.com/gitlab-org/gitlab/-/issues/14609) is implemented, updates must be prevented from happening manually to the - **primary**. Note that your **secondary** node still needs read-only + **primary**. Your **secondary** node still needs read-only access to the **primary** node during the maintenance window: 1. At the scheduled time, using your cloud provider or your node's firewall, block diff --git a/doc/administration/geo/index.md b/doc/administration/geo/index.md index 295a448c432..7175d41abd8 100644 --- a/doc/administration/geo/index.md +++ b/doc/administration/geo/index.md @@ -101,12 +101,12 @@ From the perspective of a user performing Git operations: - The **primary** site behaves as a full read-write GitLab instance. - **Secondary** sites are read-only but proxy Git push operations to the **primary** site. This makes **secondary** sites appear to support push operations themselves. -To simplify the diagram, some necessary components are omitted. Note that: +To simplify the diagram, some necessary components are omitted. - Git over SSH requires [`gitlab-shell`](https://gitlab.com/gitlab-org/gitlab-shell) and OpenSSH. - Git over HTTPS required [`gitlab-workhorse`](https://gitlab.com/gitlab-org/gitlab-workhorse). -Note that a **secondary** site needs two different PostgreSQL databases: +A **secondary** site needs two different PostgreSQL databases: - A read-only database instance that streams data from the main GitLab database. - [Another database instance](#geo-tracking-database) used internally by the **secondary** site to record what data has been replicated. @@ -193,11 +193,8 @@ This list of limitations only reflects the latest version of GitLab. If you are - The **primary** site has to be online for OAuth login to happen. Existing sessions and Git are not affected. Support for the **secondary** site to use an OAuth provider independent from the primary is [being planned](https://gitlab.com/gitlab-org/gitlab/-/issues/208465). - The installation takes multiple manual steps that together can take about an hour depending on circumstances. We are working on improving this experience. See [Omnibus GitLab issue #2978](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/2978) for details. - Real-time updates of issues/merge requests (for example, via long polling) doesn't work on the **secondary** site. -- [Selective synchronization](replication/configuration.md#selective-synchronization) applies only to files and repositories. Other datasets are replicated to the **secondary** site in full, making it inappropriate for use as an access control mechanism. -- Object pools for forked project deduplication work only on the **primary** site, and are duplicated on the **secondary** site. - GitLab Runners cannot register with a **secondary** site. Support for this is [planned for the future](https://gitlab.com/gitlab-org/gitlab/-/issues/3294). -- Configuring Geo **secondary** sites to [use high-availability configurations of PostgreSQL](https://gitlab.com/groups/gitlab-org/-/epics/2536) is currently in **alpha** support. -- [Selective synchronization](replication/configuration.md#selective-synchronization) only limits what repositories are replicated. The entire PostgreSQL data is still replicated. Selective synchronization is not built to accommodate compliance / export control use cases. +- [Selective synchronization](replication/configuration.md#selective-synchronization) only limits what repositories and files are replicated. The entire PostgreSQL data is still replicated. Selective synchronization is not built to accommodate compliance / export control use cases. ### Limitations on replication/verification diff --git a/doc/administration/geo/replication/configuration.md b/doc/administration/geo/replication/configuration.md index e8ffa1ae91a..5b22741f578 100644 --- a/doc/administration/geo/replication/configuration.md +++ b/doc/administration/geo/replication/configuration.md @@ -7,35 +7,35 @@ type: howto # Geo configuration **(PREMIUM SELF)** -## Configuring a new **secondary** node +## Configuring a new **secondary** site NOTE: -This is the final step in setting up a **secondary** Geo node. Stages of the +This is the final step in setting up a **secondary** Geo site. Stages of the setup process must be completed in the documented order. Before attempting the steps in this stage, [complete all prior stages](../setup/index.md#using-omnibus-gitlab). -The basic steps of configuring a **secondary** node are to: +The basic steps of configuring a **secondary** site are to: -- Replicate required configurations between the **primary** node and the **secondary** nodes. -- Configure a tracking database on each **secondary** node. -- Start GitLab on each **secondary** node. +- Replicate required configurations between the **primary** site and the **secondary** sites. +- Configure a tracking database on each **secondary** site. +- Start GitLab on each **secondary** site. You are encouraged to first read through all the steps before executing them in your testing/production environment. NOTE: -**Do not** set up any custom authentication for the **secondary** nodes. This is handled by the **primary** node. +**Do not** set up any custom authentication for the **secondary** sites. This is handled by the **primary** site. Any change that requires access to the **Admin Area** needs to be done in the -**primary** node because the **secondary** node is a read-only replica. +**primary** site because the **secondary** site is a read-only replica. ### Step 1. Manually replicate secret GitLab values GitLab stores a number of secret values in the `/etc/gitlab/gitlab-secrets.json` -file which *must* be the same on all nodes. Until there is -a means of automatically replicating these between nodes (see [issue #3789](https://gitlab.com/gitlab-org/gitlab/-/issues/3789)), -they must be manually replicated to the **secondary** node. +file which *must* be the same on all of a site's nodes. Until there is +a means of automatically replicating these between sites (see [issue #3789](https://gitlab.com/gitlab-org/gitlab/-/issues/3789)), +they must be manually replicated to **all nodes of the secondary site**. -1. SSH into the **primary** node, and execute the command below: +1. SSH into a **Rails node on your primary** site, and execute the command below: ```shell sudo cat /etc/gitlab/gitlab-secrets.json @@ -43,7 +43,7 @@ they must be manually replicated to the **secondary** node. This displays the secrets that need to be replicated, in JSON format. -1. SSH into the **secondary** node and login as the `root` user: +1. SSH **into each node on your secondary Geo site** and login as the `root` user: ```shell sudo -i @@ -55,7 +55,7 @@ they must be manually replicated to the **secondary** node. mv /etc/gitlab/gitlab-secrets.json /etc/gitlab/gitlab-secrets.json.`date +%F` ``` -1. Copy `/etc/gitlab/gitlab-secrets.json` from the **primary** node to the **secondary** node, or +1. Copy `/etc/gitlab/gitlab-secrets.json` from the **Rails node on your primary** site to **each node on your secondary** site, or copy-and-paste the file contents between nodes: ```shell @@ -72,28 +72,28 @@ they must be manually replicated to the **secondary** node. chmod 0600 /etc/gitlab/gitlab-secrets.json ``` -1. Reconfigure the **secondary** node for the change to take effect: +1. Reconfigure **each Rails, Sidekiq and Gitaly nodes on your secondary** site for the change to take effect: ```shell gitlab-ctl reconfigure gitlab-ctl restart ``` -### Step 2. Manually replicate the **primary** node's SSH host keys +### Step 2. Manually replicate the **primary** site's SSH host keys GitLab integrates with the system-installed SSH daemon, designating a user (typically named `git`) through which all access requests are handled. In a [Disaster Recovery](../disaster_recovery/index.md) situation, GitLab system -administrators promote a **secondary** node to the **primary** node. DNS records for the -**primary** domain should also be updated to point to the new **primary** node -(previously a **secondary** node). Doing so avoids the need to update Git remotes and API URLs. +administrators promote a **secondary** site to the **primary** site. DNS records for the +**primary** domain should also be updated to point to the new **primary** site +(previously a **secondary** site). Doing so avoids the need to update Git remotes and API URLs. -This causes all SSH requests to the newly promoted **primary** node to +This causes all SSH requests to the newly promoted **primary** site to fail due to SSH host key mismatch. To prevent this, the primary SSH host -keys must be manually replicated to the **secondary** node. +keys must be manually replicated to the **secondary** site. -1. SSH into the **secondary** node and login as the `root` user: +1. SSH into **each node on your secondary** site and login as the `root` user: ```shell sudo -i @@ -105,34 +105,34 @@ keys must be manually replicated to the **secondary** node. find /etc/ssh -iname ssh_host_* -exec cp {} {}.backup.`date +%F` \; ``` -1. Copy OpenSSH host keys from the **primary** node: +1. Copy OpenSSH host keys from the **primary** site: - If you can access your **primary** node using the **root** user: + If you can access one of the **nodes on your primary** site serving SSH traffic (usually, the main GitLab Rails application nodes) using the **root** user: ```shell - # Run this from the secondary node, change `<primary_node_fqdn>` for the IP or FQDN of the server + # Run this from the secondary site, change `<primary_site_fqdn>` for the IP or FQDN of the server scp root@<primary_node_fqdn>:/etc/ssh/ssh_host_*_key* /etc/ssh ``` If you only have access through a user with `sudo` privileges: ```shell - # Run this from your primary node: + # Run this from the node on your primary site: sudo tar --transform 's/.*\///g' -zcvf ~/geo-host-key.tar.gz /etc/ssh/ssh_host_*_key* - # Run this from your secondary node: - scp <user_with_sudo>@<primary_node_fqdn>:geo-host-key.tar.gz . + # Run this on each node on your secondary site: + scp <user_with_sudo>@<primary_site_fqdn>:geo-host-key.tar.gz . tar zxvf ~/geo-host-key.tar.gz -C /etc/ssh ``` -1. On your **secondary** node, ensure the file permissions are correct: +1. On **each node on your secondary** site, ensure the file permissions are correct: ```shell chown root:root /etc/ssh/ssh_host_*_key* chmod 0600 /etc/ssh/ssh_host_*_key* ``` -1. To verify key fingerprint matches, execute the following command on both nodes: +1. To verify key fingerprint matches, execute the following command on both primary and secondary nodes on each site: ```shell for file in /etc/ssh/ssh_host_*_key; do ssh-keygen -lf $file; done @@ -160,7 +160,7 @@ keys must be manually replicated to the **secondary** node. NOTE: The output for private keys and public keys command should generate the same fingerprint. -1. Restart `sshd` on your **secondary** node: +1. Restart `sshd` on **each node on your secondary** site: ```shell # Debian or Ubuntu installations @@ -175,31 +175,34 @@ keys must be manually replicated to the **secondary** node. SSH into your GitLab **secondary** server in a new terminal. If you are unable to connect, verify the permissions are correct according to the previous steps. -### Step 3. Add the **secondary** node +### Step 3. Add the **secondary** site -1. SSH into your GitLab **secondary** server and login as root: +1. SSH into **each Rails and Sidekiq node on your secondary** site and login as root: ```shell sudo -i ``` -1. Edit `/etc/gitlab/gitlab.rb` and add a **unique** name for your node. You need this in the next steps: +1. Edit `/etc/gitlab/gitlab.rb` and add a **unique** name for your site. You need this in the next steps: ```ruby - # The unique identifier for the Geo node. - gitlab_rails['geo_node_name'] = '<node_name_here>' + ## + ## The unique identifier for the Geo site. See + ## https://docs.gitlab.com/ee/user/admin_area/geo_nodes.html#common-settings + ## + gitlab_rails['geo_node_name'] = '<site_name_here>' ``` -1. Reconfigure the **secondary** node for the change to take effect: +1. Reconfigure **each Rails and Sidekiq node on your secondary** site for the change to take effect: ```shell gitlab-ctl reconfigure ``` -1. On the top bar of the primary node, select **Menu >** **{admin}** **Admin**. -1. On the left sidebar, select **Geo > Nodes**. -1. Select **Add site**. -  +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Geo > Sites**. +1. Select **New site**. +  1. Fill in **Name** with the `gitlab_rails['geo_node_name']` in `/etc/gitlab/gitlab.rb`. These values must always match *exactly*, character for character. @@ -207,10 +210,10 @@ keys must be manually replicated to the **secondary** node. values must always match, but it doesn't matter if one ends with a `/` and the other doesn't. 1. Optionally, choose which groups or storage shards should be replicated by the - **secondary** node. Leave blank to replicate all. Read more in + **secondary** site. Leave blank to replicate all. Read more in [selective synchronization](#selective-synchronization). -1. Select **Add node** to add the **secondary** node. -1. SSH into your GitLab **secondary** server and restart the services: +1. Select **Add site** to add the **secondary** site. +1. SSH into **each Rails, and Sidekiq node on your secondary** site and restart the services: ```shell gitlab-ctl restart @@ -222,58 +225,58 @@ keys must be manually replicated to the **secondary** node. gitlab-rake gitlab:geo:check ``` -1. SSH into your **primary** server and login as root to verify the - **secondary** node is reachable or there are any common issue with your Geo setup: +1. SSH into a **Rails or Sidekiq server on your primary** site and login as root to verify the + **secondary** site is reachable or there are any common issue with your Geo setup: ```shell gitlab-rake gitlab:geo:check ``` -Once added to the Geo administration page and restarted, the **secondary** node automatically starts -replicating missing data from the **primary** node in a process known as **backfill**. -Meanwhile, the **primary** node starts to notify each **secondary** node of any changes, so -that the **secondary** node can act on those notifications immediately. +Once added to the Geo administration page and restarted, the **secondary** site automatically starts +replicating missing data from the **primary** site in a process known as **backfill**. +Meanwhile, the **primary** site starts to notify each **secondary** site of any changes, so +that the **secondary** site can act on those notifications immediately. -Be sure the _secondary_ node is running and accessible. You can sign in to the -_secondary_ node with the same credentials as were used with the _primary_ node. +Be sure the _secondary_ site is running and accessible. You can sign in to the +_secondary_ site with the same credentials as were used with the _primary_ site. -### Step 4. (Optional) Configuring the **secondary** node to trust the **primary** node +### Step 4. (Optional) Configuring the **secondary** site to trust the **primary** site -You can safely skip this step if your **primary** node uses a CA-issued HTTPS certificate. +You can safely skip this step if your **primary** site uses a CA-issued HTTPS certificate. -If your **primary** node is using a self-signed certificate for *HTTPS* support, you -need to add that certificate to the **secondary** node's trust store. Retrieve the -certificate from the **primary** node and follow +If your **primary** site is using a self-signed certificate for *HTTPS* support, you +need to add that certificate to the **secondary** site's trust store. Retrieve the +certificate from the **primary** site and follow [these instructions](https://docs.gitlab.com/omnibus/settings/ssl.html) -on the **secondary** node. +on the **secondary** site. ### Step 5. Enable Git access over HTTP/HTTPS Geo synchronizes repositories over HTTP/HTTPS, and therefore requires this clone -method to be enabled. This is enabled by default, but if converting an existing node to Geo it should be checked: +method to be enabled. This is enabled by default, but if converting an existing site to Geo it should be checked: -On the **primary** node: +On the **primary** site: 1. On the top bar, select **Menu >** **{admin}** **Admin**. 1. On the left sidebar, select **Settings > General**. 1. Expand **Visibility and access controls**. 1. Ensure "Enabled Git access protocols" is set to either "Both SSH and HTTP(S)" or "Only HTTP(S)". -### Step 6. Verify proper functioning of the **secondary** node +### Step 6. Verify proper functioning of the **secondary** site -You can sign in to the **secondary** node with the same credentials you used with -the **primary** node. After you sign in: +You can sign in to the **secondary** site with the same credentials you used with +the **primary** site. After you sign in: 1. On the top bar, select **Menu >** **{admin}** **Admin**. -1. On the left sidebar, select **Geo > Nodes**. -1. Verify that it's correctly identified as a **secondary** Geo node, and that +1. On the left sidebar, select **Geo > Sites**. +1. Verify that it's correctly identified as a **secondary** Geo site, and that Geo is enabled. The initial replication, or 'backfill', is probably still in progress. You -can monitor the synchronization process on each Geo node from the **primary** -node's **Geo Nodes** dashboard in your browser. +can monitor the synchronization process on each Geo site from the **primary** +site's **Geo Sites** dashboard in your browser. - + If your installation isn't working properly, check the [troubleshooting document](troubleshooting.md). @@ -286,10 +289,10 @@ The two most obvious issues that can become apparent in the dashboard are: - You are using a custom certificate or custom CA (see the [troubleshooting document](troubleshooting.md)). - The instance is firewalled (check your firewall rules). -Please note that disabling a **secondary** node stops the synchronization process. +Disabling a **secondary** site stops the synchronization process. -Please note that if `git_data_dirs` is customized on the **primary** node for multiple -repository shards you must duplicate the same configuration on each **secondary** node. +If `git_data_dirs` is customized on the **primary** site for multiple +repository shards you must duplicate the same configuration on each **secondary** site. Point your users to the [Using a Geo Site guide](usage.md). @@ -304,7 +307,7 @@ Currently, this is what is synced: ## Selective synchronization Geo supports selective synchronization, which allows administrators to choose -which projects should be synchronized by **secondary** nodes. +which projects should be synchronized by **secondary** sites. A subset of projects can be chosen, either by group or by storage shard. The former is ideal for replicating data belonging to a subset of users, while the latter is more suited to progressively rolling out Geo to a large GitLab @@ -312,22 +315,22 @@ instance. It is important to note that selective synchronization: -1. Does not restrict permissions from **secondary** nodes. -1. Does not hide project metadata from **secondary** nodes. +1. Does not restrict permissions from **secondary** sites. +1. Does not hide project metadata from **secondary** sites. - Since Geo currently relies on PostgreSQL replication, all project metadata - gets replicated to **secondary** nodes, but repositories that have not been + gets replicated to **secondary** sites, but repositories that have not been selected are empty. 1. Does not reduce the number of events generated for the Geo event log. - - The **primary** node generates events as long as any **secondary** nodes are present. - Selective synchronization restrictions are implemented on the **secondary** nodes, - not the **primary** node. + - The **primary** site generates events as long as any **secondary** sites are present. + Selective synchronization restrictions are implemented on the **secondary** sites, + not the **primary** site. ### Git operations on unreplicated repositories > [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2562) in GitLab 12.10 for HTTP(S) and in GitLab 13.0 for SSH. Git clone, pull, and push operations over HTTP(S) and SSH are supported for repositories that -exist on the **primary** node but not on **secondary** nodes. This situation can occur +exist on the **primary** site but not on **secondary** sites. This situation can occur when: - Selective synchronization does not include the project attached to the repository. @@ -335,7 +338,7 @@ when: ## Upgrading Geo -See the [updating the Geo nodes document](updating_the_geo_nodes.md). +See the [updating the Geo sites document](updating_the_geo_nodes.md). ## Troubleshooting diff --git a/doc/administration/geo/replication/datatypes.md b/doc/administration/geo/replication/datatypes.md index a56d9dc813c..a696e5410e5 100644 --- a/doc/administration/geo/replication/datatypes.md +++ b/doc/administration/geo/replication/datatypes.md @@ -32,7 +32,6 @@ verification methods: | Git | Project repository | Geo with Gitaly | Gitaly Checksum | | Git | Project wiki repository | Geo with Gitaly | Gitaly Checksum | | Git | Project designs repository | Geo with Gitaly | Gitaly Checksum | -| Git | Object pools for forked project deduplication | Geo with Gitaly | _Not implemented_ | | Git | Project Snippets | Geo with Gitaly | Gitaly Checksum | | Git | Personal Snippets | Geo with Gitaly | Gitaly Checksum | | Git | Group wiki repository | Geo with Gitaly | _Not implemented_ | @@ -47,8 +46,8 @@ verification methods: | Blobs | Container registry _(file system)_ | Geo with API/Docker API | _Not implemented_ | | Blobs | Container registry _(object storage)_ | Geo with API/Managed/Docker API (*2*) | _Not implemented_ | | Blobs | Package registry _(file system)_ | Geo with API | SHA256 checksum | -| Blobs | Package registry _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | -| Blobs | Versioned Terraform State _(file system)_ | Geo with API | SHA256 checksum | +| Blobs | Package registry _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | +| Blobs | Versioned Terraform State _(file system)_ | Geo with API | SHA256 checksum | | Blobs | Versioned Terraform State _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | | Blobs | External Merge Request Diffs _(file system)_ | Geo with API | _Not implemented_ | | Blobs | External Merge Request Diffs _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | @@ -69,6 +68,8 @@ or using LVM. It requires no special file system and can work with NFS or a mounted Storage Appliance (there may be performance limitations when using a remote file system). +Geo will trigger garbage collection in Gitaly to [deduplicate forked repositories](../../../development/git_object_deduplication.md#git-object-deduplication-and-gitlab-geo) on Geo secondary sites. + Communication is done via Gitaly's own gRPC API. There are three possible ways of synchronization: - Using regular Git clone/fetch from one Geo site to another (with special authentication). @@ -186,7 +187,6 @@ successfully, you must replicate their data using some other means. |[CI job artifacts (other than Job Logs)](../../../ci/pipelines/job_artifacts.md) | **Yes** (10.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8923) | Via Object Storage provider if supported. Native Geo support (Beta). | Verified only manually using [Integrity Check Rake Task](../../raketasks/check.md) on both sites and comparing the output between them. | |[CI Pipeline Artifacts](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/ci/pipeline_artifact.rb) | [**Yes** (13.11)](https://gitlab.com/gitlab-org/gitlab/-/issues/238464) | [**Yes** (13.11)](https://gitlab.com/gitlab-org/gitlab/-/issues/238464) | Via Object Storage provider if supported. Native Geo support (Beta). | Persists additional artifacts after a pipeline completes | |[Job logs](../../job_logs.md) | **Yes** (10.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8923) | Via Object Storage provider if supported. Native Geo support (Beta). | Verified only on transfer or manually using [Integrity Check Rake Task](../../raketasks/check.md) on both sites and comparing the output between them. | -|[Object pools for forked project deduplication](../../../development/git_object_deduplication.md) | **Yes** | No | No | | |[Container Registry](../../packages/container_registry.md) | **Yes** (12.3) | No | No | Disabled by default. See [instructions](docker_registry.md) to enable. | |[Content in object storage (beta)](object_storage.md) | **Yes** (12.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/13845) | No | | |[Infrastructure Registry for Terraform Module](../../../user/packages/terraform_module_registry/index.md) | **Yes** (14.0) | [**Yes**](#limitation-of-verification-for-files-in-object-storage) (14.0) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | @@ -200,11 +200,11 @@ successfully, you must replicate their data using some other means. |[Package Registry for generic packages](../../../user/packages/generic_packages/index.md) | **Yes** (13.5) | [**Yes**](#limitation-of-verification-for-files-in-object-storage) (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | |[Versioned Terraform State](../../terraform_state.md) | **Yes** (13.5) | [**Yes**](#limitation-of-verification-for-files-in-object-storage) (13.12) | Via Object Storage provider if supported. Native Geo support (Beta). | Replication is behind the feature flag `geo_terraform_state_version_replication`, enabled by default. Verification was behind the feature flag `geo_terraform_state_version_verification`, which was removed in 14.0| |[External merge request diffs](../../merge_request_diffs.md) | **Yes** (13.5) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Replication is behind the feature flag `geo_merge_request_diff_replication`, enabled by default. Verification is under development, behind the feature flag `geo_merge_request_diff_verification`, introduced in 14.0.| -|[Versioned snippets](../../../user/snippets.md#versioned-snippets) | [**Yes** (13.7)](https://gitlab.com/groups/gitlab-org/-/epics/2809) | [No](https://gitlab.com/groups/gitlab-org/-/epics/2810) | No | | +|[Versioned snippets](../../../user/snippets.md#versioned-snippets) | [**Yes** (13.7)](https://gitlab.com/groups/gitlab-org/-/epics/2809) | [**Yes** (14.2)](https://gitlab.com/groups/gitlab-org/-/epics/2810) | No | Verification was implemented behind the feature flag `geo_snippet_repository_verification` in 13.11, and the feature flag was removed in 14.2. | |[Server-side Git hooks](../../server_hooks.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1867) | No | No | | |[Elasticsearch integration](../../../integration/elasticsearch.md) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/1186) | No | No | | |[GitLab Pages](../../pages/index.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/589) | No | Via Object Storage provider if supported. **No** native Geo support (Beta). | | -|[Dependency proxy images](../../../user/packages/dependency_proxy/index.md) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/259694) | No | No | Blocked on [Geo: Secondary Mimicry](https://gitlab.com/groups/gitlab-org/-/epics/1528). Note that replication of this cache is not needed for Disaster Recovery purposes because it can be recreated from external sources. | +|[Dependency proxy images](../../../user/packages/dependency_proxy/index.md) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/259694) | No | No | Blocked on [Geo: Secondary Mimicry](https://gitlab.com/groups/gitlab-org/-/epics/1528). Replication of this cache is not needed for Disaster Recovery purposes because it can be recreated from external sources. | |[Vulnerability Export](../../../user/application_security/vulnerability_report/#export-vulnerability-details) | [Not planned](https://gitlab.com/groups/gitlab-org/-/epics/3111) | No | | Not planned because they are ephemeral and sensitive. They can be regenerated on demand. | #### Limitation of verification for files in Object Storage diff --git a/doc/administration/geo/replication/img/adding_a_secondary_node_v13_3.png b/doc/administration/geo/replication/img/adding_a_secondary_v13_3.png Binary files differindex e43ace99666..e43ace99666 100644 --- a/doc/administration/geo/replication/img/adding_a_secondary_node_v13_3.png +++ b/doc/administration/geo/replication/img/adding_a_secondary_v13_3.png diff --git a/doc/administration/geo/replication/img/geo_node_dashboard_v14_0.png b/doc/administration/geo/replication/img/geo_dashboard_v14_0.png Binary files differindex 6d183fc6bd2..6d183fc6bd2 100644 --- a/doc/administration/geo/replication/img/geo_node_dashboard_v14_0.png +++ b/doc/administration/geo/replication/img/geo_dashboard_v14_0.png diff --git a/doc/administration/geo/replication/img/geo_node_health_v14_0.png b/doc/administration/geo/replication/img/geo_site_health_v14_0.png Binary files differindex 4c640522569..4c640522569 100644 --- a/doc/administration/geo/replication/img/geo_node_health_v14_0.png +++ b/doc/administration/geo/replication/img/geo_site_health_v14_0.png diff --git a/doc/administration/geo/replication/location_aware_git_url.md b/doc/administration/geo/replication/location_aware_git_url.md index a80c293149e..ce1af27611e 100644 --- a/doc/administration/geo/replication/location_aware_git_url.md +++ b/doc/administration/geo/replication/location_aware_git_url.md @@ -20,7 +20,7 @@ as well. NOTE: You can also use a load balancer to distribute web UI or API traffic to -[multiple Geo **secondary** sites](../../../user/admin_area/geo_nodes.md#multiple-secondary-nodes-behind-a-load-balancer). +[multiple Geo **secondary** sites](../../../user/admin_area/geo_nodes.md#multiple-secondary-sites-behind-a-load-balancer). Importantly, the **primary** site cannot yet be included. See the feature request [Support putting the **primary** behind a Geo node load balancer](https://gitlab.com/gitlab-org/gitlab/-/issues/10888). @@ -107,7 +107,7 @@ You can customize the: - SSH remote URL to use the location-aware `git.example.com`. To do so, change the SSH remote URL's host by setting `gitlab_rails['gitlab_ssh_host']` in `gitlab.rb` of web nodes. - HTTP remote URL as shown in - [Custom Git clone URL for HTTP(S)](../../../user/admin_area/settings/visibility_and_access_controls.md#custom-git-clone-url-for-https). + [Custom Git clone URL for HTTP(S)](../../../user/admin_area/settings/visibility_and_access_controls.md#customize-git-clone-url-for-https). ## Example Git request handling behavior diff --git a/doc/administration/geo/replication/multiple_servers.md b/doc/administration/geo/replication/multiple_servers.md index ea2488b65fb..7db210d31f4 100644 --- a/doc/administration/geo/replication/multiple_servers.md +++ b/doc/administration/geo/replication/multiple_servers.md @@ -17,71 +17,59 @@ described, it is possible to adapt these instructions to your needs. _[diagram source - GitLab employees only](https://docs.google.com/drawings/d/1z0VlizKiLNXVVVaERFwgsIOuEgjcUqDTWPdQYsE7Z4c/edit)_ -The topology above assumes the **primary** and **secondary** Geo clusters +The topology above assumes the **primary** and **secondary** Geo sites are located in two separate locations, on their own virtual network with private IP addresses. The network is configured such that all machines in one geographic location can communicate with each other using their private IP addresses. The IP addresses given are examples and may be different depending on the network topology of your deployment. -The only external way to access the two Geo deployments is by HTTPS at +The only external way to access the two Geo sites is by HTTPS at `gitlab.us.example.com` and `gitlab.eu.example.com` in the example above. NOTE: -The **primary** and **secondary** Geo deployments must be able to communicate to each other over HTTPS. +The **primary** and **secondary** Geo sites must be able to communicate to each other over HTTPS. ## Redis and PostgreSQL for multiple nodes -Geo supports: - -- Redis and PostgreSQL on the **primary** node configured for multiple nodes. -- Redis on **secondary** nodes configured for multiple nodes. - -NOTE: -Support for PostgreSQL on **secondary** nodes in multi-node configuration -[is planned](https://gitlab.com/groups/gitlab-org/-/epics/2536). - Because of the additional complexity involved in setting up this configuration for PostgreSQL and Redis, it is not covered by this Geo multi-node documentation. For more information on setting up a multi-node PostgreSQL cluster and Redis cluster using the Omnibus GitLab package, see: -- [PostgreSQL multi-node documentation](../../postgresql/replication_and_failover.md) +- [Geo multi-node database replication](../setup/database.md#multi-node-database-replication) - [Redis multi-node documentation](../../redis/replication_and_failover.md) NOTE: It is possible to use cloud hosted services for PostgreSQL and Redis, but this is beyond the scope of this document. -## Prerequisites: Two working GitLab multi-node clusters +## Prerequisites: Two independently working GitLab multi-node sites -One cluster serves as the **primary** node. Use the -[GitLab multi-node documentation](../../reference_architectures/index.md) to set this up. If +One GitLab site serves as the Geo **primary** site. Use the +[GitLab reference architectures documentation](../../reference_architectures/index.md) +to set this up. You can use different reference architecture sizes for each Geo site. If you already have a working GitLab instance that is in-use, it can be used as a -**primary**. +**primary** site. -The second cluster serves as the **secondary** node. Again, use the -[GitLab multi-node documentation](../../reference_architectures/index.md) to set this up. +The second GitLab site serves as the Geo **secondary** site. Again, use the +[GitLab reference architectures documentation](../../reference_architectures/index.md) to set this up. It's a good idea to log in and test it. However, be aware that its data is -wiped out as part of the process of replicating from the **primary** node. +wiped out as part of the process of replicating from the **primary** site. -## Configure the GitLab cluster to be the **primary** node +## Configure a GitLab site to be the Geo **primary** site -The following steps enable a GitLab cluster to serve as the **primary** node. +The following steps enable a GitLab site to serve as the Geo **primary** site. -### Step 1: Configure the **primary** frontend servers +### Step 1: Configure the **primary** frontend nodes 1. Edit `/etc/gitlab/gitlab.rb` and add the following: ```ruby ## - ## Enable the Geo primary role + ## The unique identifier for the Geo site. See + ## https://docs.gitlab.com/ee/user/admin_area/geo_nodes.html#common-settings ## - roles ['geo_primary_role'] - - ## - ## The unique identifier for the Geo node. - ## - gitlab_rails['geo_node_name'] = '<node_name_here>' + gitlab_rails['geo_node_name'] = '<site_name_here>' ## ## Disable automatic migrations @@ -93,155 +81,63 @@ After making these changes, [reconfigure GitLab](../../restart_gitlab.md#omnibus NOTE: PostgreSQL and Redis should have already been disabled on the -application servers during normal GitLab multi-node setup. Connections -from the application servers to services on the backend servers should +application nodes during normal GitLab multi-node setup. Connections +from the application nodes to services on the backend nodes should have also been configured. See multi-node configuration documentation for [PostgreSQL](../../postgresql/replication_and_failover.md#configuring-the-application-nodes) and [Redis](../../redis/replication_and_failover.md#example-configuration-for-the-gitlab-application). -### Step 2: Configure the **primary** database - -1. Edit `/etc/gitlab/gitlab.rb` and add the following: - - ```ruby - ## - ## Configure the Geo primary role and the PostgreSQL role - ## - roles ['geo_primary_role', 'postgres_role'] - ``` +## Configure the other GitLab site to be a Geo **secondary** site -## Configure a **secondary** node - -A **secondary** cluster is similar to any other GitLab multi-node cluster, with two +A **secondary** site is similar to any other GitLab multi-node site, with three major differences: -- The main PostgreSQL database is a read-only replica of the **primary** node's +- The main PostgreSQL database is a read-only replica of the Geo **primary** site's PostgreSQL database. -- There is also a single PostgreSQL database for the **secondary** cluster, - called the "tracking database", which tracks the synchronization state of - various resources. +- There is an additional PostgreSQL database for each Geo **secondary** site, + called the "Geo tracking database", which tracks the replication and verification + state of various resources. +- There is an additional GitLab service [`geo-logcursor`](../index.md#geo-log-cursor) Therefore, we set up the multi-node components one by one and include deviations from the normal multi-node setup. However, we highly recommend configuring a -brand-new cluster first, as if it were not part of a Geo setup. This allows -verifying that it is a working cluster. And only then should it be modified -for use as a Geo **secondary**. This helps to separate Geo setup problems from -unrelated problems. +brand-new GitLab site first, as if it were not part of a Geo setup. This allows +verifying that it is a working GitLab site. And only then should it be modified +for use as a Geo **secondary** site. This helps to separate Geo setup problems from +unrelated multi-node configuration problems. -### Step 1: Configure the Redis and Gitaly services on the **secondary** node +### Step 1: Configure the Redis and Gitaly services on the Geo **secondary** site Configure the following services, again using the non-Geo multi-node documentation: - [Configuring Redis for GitLab](../../redis/replication_and_failover.md#example-configuration-for-the-gitlab-application) for multiple nodes. - [Gitaly](../../gitaly/index.md), which stores data that is - synchronized from the **primary** node. + synchronized from the Geo **primary** site. NOTE: [NFS](../../nfs.md) can be used in place of Gitaly but is not recommended. -### Step 2: Configure the main read-only replica PostgreSQL database on the **secondary** node - -NOTE: -The following documentation assumes the database runs on -a single node only. Multi-node PostgreSQL on **secondary** nodes is -[not currently supported](https://gitlab.com/groups/gitlab-org/-/epics/2536). - -Configure the [**secondary** database](../setup/database.md) as a read-only replica of -the **primary** database. Use the following as a guide. - -1. Generate an MD5 hash of the desired password for the database user that the - GitLab application uses to access the read-replica database: - - Note that the username (`gitlab` by default) is incorporated into the hash. - - ```shell - gitlab-ctl pg-password-md5 gitlab - # Enter password: <your_password_here> - # Confirm password: <your_password_here> - # fca0b89a972d69f00eb3ec98a5838484 - ``` - - Use this hash to fill in `<md5_hash_of_your_password>` in the next step. +### Step 2: Configure Postgres streaming replication -1. Edit `/etc/gitlab/gitlab.rb` in the replica database machine, and add the - following: - - ```ruby - ## - ## Configure the Geo secondary role and the PostgreSQL role - ## - roles ['geo_secondary_role', 'postgres_role'] - - ## - ## The unique identifier for the Geo node. - ## This should match the secondary's application node. - ## - gitlab_rails['geo_node_name'] = '<node_name_here>' - - ## - ## Secondary address - ## - replace '<secondary_node_ip>' with the public or VPC address of your Geo secondary node - ## - replace '<tracking_database_ip>' with the public or VPC address of your Geo tracking database node - ## - postgresql['listen_address'] = '<secondary_node_ip>' - postgresql['md5_auth_cidr_addresses'] = ['<secondary_node_ip>/32', '<tracking_database_ip>/32'] - - ## - ## Database credentials password (defined previously in primary node) - ## - replicate same values here as defined in primary node - ## - postgresql['sql_user_password'] = '<md5_hash_of_your_password>' - gitlab_rails['db_password'] = '<your_password_here>' - - ## - ## When running the Geo tracking database on a separate machine, disable it - ## here and allow connections from the tracking database host. And ensure - ## the tracking database IP is in postgresql['md5_auth_cidr_addresses'] above. - ## - geo_postgresql['enable'] = false - - ## - ## Disable all other services that aren't needed. Note that we had to enable - ## geo_secondary_role to cause some configuration changes to postgresql, but - ## the role enables single-node services by default. - ## - alertmanager['enable'] = false - consul['enable'] = false - geo_logcursor['enable'] = false - gitaly['enable'] = false - gitlab_exporter['enable'] = false - gitlab_workhorse['enable'] = false - nginx['enable'] = false - node_exporter['enable'] = false - pgbouncer_exporter['enable'] = false - prometheus['enable'] = false - redis['enable'] = false - redis_exporter['enable'] = false - patroni['enable'] = false - sidekiq['enable'] = false - sidekiq_cluster['enable'] = false - puma['enable'] = false - ``` - -After making these changes, [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) so the changes take effect. +Follow the [Geo database replication instructions](../setup/database.md). If using an external PostgreSQL instance, refer also to [Geo with external PostgreSQL instances](../setup/external_database.md). -### Step 3: Configure the tracking database on the **secondary** node +### Step 3: Configure the Geo tracking database on the Geo **secondary** site -NOTE: -This documentation assumes the tracking database runs on -only a single machine, rather than as a PostgreSQL cluster. +If you want to run the Geo tracking database in a multi-node PostgreSQL cluster, +then follow [Configuring Patroni cluster for the tracking PostgreSQL database](../setup/database.md#configuring-patroni-cluster-for-the-tracking-postgresql-database). -Configure the tracking database. +If you want to run the Geo tracking database on a single node, then follow +the instructions below. 1. Generate an MD5 hash of the desired password for the database user that the GitLab application uses to access the tracking database: - Note that the username (`gitlab_geo` by default) is incorporated into the + The username (`gitlab_geo` by default) is incorporated into the hash. ```shell @@ -254,8 +150,8 @@ Configure the tracking database. Use this hash to fill in `<tracking_database_password_md5_hash>` in the next step. -1. Edit `/etc/gitlab/gitlab.rb` in the tracking database machine, and add the - following: +1. On the machine where the Geo tracking database is intended to run, add the + following to `/etc/gitlab/gitlab.rb`: ```ruby ## @@ -271,29 +167,8 @@ Configure the tracking database. geo_postgresql['md5_auth_cidr_addresses'] = ['<replica_database_ip>/32'] gitlab_rails['db_host'] = '<replica_database_ip>' - # Prevent reconfigure from attempting to run migrations on the replica DB + # Prevent reconfigure from attempting to run migrations on the replica database gitlab_rails['auto_migrate'] = false - - ## - ## Ensure unnecessary services are disabled - ## - alertmanager['enable'] = false - consul['enable'] = false - geo_logcursor['enable'] = false - gitaly['enable'] = false - gitlab_exporter['enable'] = false - gitlab_workhorse['enable'] = false - nginx['enable'] = false - node_exporter['enable'] = false - pgbouncer_exporter['enable'] = false - postgresql['enable'] = false - prometheus['enable'] = false - redis['enable'] = false - redis_exporter['enable'] = false - patroni['enable'] = false - sidekiq['enable'] = false - sidekiq_cluster['enable'] = false - puma['enable'] = false ``` After making these changes, [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) so the changes take effect. @@ -301,29 +176,42 @@ After making these changes, [reconfigure GitLab](../../restart_gitlab.md#omnibus If using an external PostgreSQL instance, refer also to [Geo with external PostgreSQL instances](../setup/external_database.md). -### Step 4: Configure the frontend application servers on the **secondary** node +### Step 4: Configure the frontend application nodes on the Geo **secondary** site -In the architecture overview, there are two machines running the GitLab -application services. These services are enabled selectively in the -configuration. +In the minimal [architecture diagram](#architecture-overview) above, there are two +machines running the GitLab application services. These services are enabled +selectively in the configuration. -Configure the GitLab Rails application servers following the relevant steps +Configure the GitLab Rails application nodes following the relevant steps outlined in the [reference architectures](../../reference_architectures/index.md), then make the following modifications: -1. Edit `/etc/gitlab/gitlab.rb` on each application server in the **secondary** - cluster, and add the following: +1. Edit `/etc/gitlab/gitlab.rb` on each application node in the Geo **secondary** + site, and add the following: ```ruby ## - ## Enable the Geo secondary role + ## Enable GitLab application services. The application_role enables many services. + ## Alternatively, you can choose to enable or disable specific services on + ## different nodes to aid in horizontal scaling and separation of concerns. ## - roles ['geo_secondary_role', 'application_role'] + roles ['application_role'] + + ## `application_role` already enables this. You only need this line if + ## you selectively enable individual services that depend on Rails, like + ## `puma`, `sidekiq`, `geo-logcursor`, etc. + gitlab_rails['enable'] = true ## - ## The unique identifier for the Geo node. + ## Enable Geo Log Cursor service ## - gitlab_rails['geo_node_name'] = '<node_name_here>' + geo_logcursor['enable'] = true + + ## + ## The unique identifier for the Geo site. See + ## https://docs.gitlab.com/ee/user/admin_area/geo_nodes.html#common-settings + ## + gitlab_rails['geo_node_name'] = '<site_name_here>' ## ## Disable automatic migrations @@ -331,12 +219,11 @@ then make the following modifications: gitlab_rails['auto_migrate'] = false ## - ## Configure the connection to the tracking DB. And disable application - ## servers from running tracking databases. + ## Configure the connection to the tracking database ## + geo_secondary['enable'] = true geo_secondary['db_host'] = '<geo_tracking_db_host>' geo_secondary['db_password'] = '<geo_tracking_db_password>' - geo_postgresql['enable'] = false ## ## Configure connection to the streaming replica database, if you haven't @@ -353,7 +240,7 @@ then make the following modifications: ## ## If you are using custom users not managed by Omnibus, you need to specify - ## UIDs and GIDs like below, and ensure they match between servers in a + ## UIDs and GIDs like below, and ensure they match between nodes in a ## cluster to avoid permissions issues ## user['uid'] = 9000 @@ -369,14 +256,17 @@ If you had set up PostgreSQL cluster using the omnibus package and had set `postgresql['sql_user_password'] = 'md5 digest of secret'`, keep in mind that `gitlab_rails['db_password']` and `geo_secondary['db_password']` contains the plaintext passwords. This is used to let the Rails -servers connect to the databases. +nodes connect to the databases. NOTE: -Make sure that current node IP is listed in `postgresql['md5_auth_cidr_addresses']` setting of your remote database. +Make sure that current node's IP is listed in +`postgresql['md5_auth_cidr_addresses']` setting of the read-replica database to +allow Rails on this node to connect to Postgres. After making these changes [Reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) so the changes take effect. -On the secondary the following GitLab frontend services are enabled: +In the [architecture overview](#architecture-overview) topology, the following GitLab +services are enabled on the "frontend" nodes: - `geo-logcursor` - `gitlab-pages` @@ -388,62 +278,41 @@ On the secondary the following GitLab frontend services are enabled: - `sidekiq` - `puma` -Verify these services by running `sudo gitlab-ctl status` on the frontend -application servers. +Verify these services exist by running `sudo gitlab-ctl status` on the frontend +application nodes. -### Step 5: Set up the LoadBalancer for the **secondary** node +### Step 5: Set up the LoadBalancer for the Geo **secondary** site -In this topology, a load balancer is required at each geographic location to -route traffic to the application servers. +The minimal [architecture diagram](#architecture-overview) above shows a load +balancer at each geographic location to route traffic to the application nodes. See [Load Balancer for GitLab with multiple nodes](../../load_balancer.md) for more information. -### Step 6: Configure the backend application servers on the **secondary** node +### Step 6: Configure the backend application nodes on the Geo **secondary** site -The minimal reference architecture diagram above shows all application services +The minimal [architecture diagram](#architecture-overview) above shows all application services running together on the same machines. However, for multiple nodes we [strongly recommend running all services separately](../../reference_architectures/index.md). -For example, a Sidekiq server could be configured similarly to the frontend -application servers above, with some changes to run only the `sidekiq` service: +For example, a Sidekiq node could be configured similarly to the frontend +application nodes above, with some changes to run only the `sidekiq` service: -1. Edit `/etc/gitlab/gitlab.rb` on each Sidekiq server in the **secondary** - cluster, and add the following: +1. Edit `/etc/gitlab/gitlab.rb` on each Sidekiq node in the Geo **secondary** + site, and add the following: ```ruby ## - ## Enable the Geo secondary role - ## - roles ['geo_secondary_role'] - - ## ## Enable the Sidekiq service ## sidekiq['enable'] = true + gitlab_rails['enable'] = true ## - ## Ensure unnecessary services are disabled - ## - alertmanager['enable'] = false - consul['enable'] = false - gitaly['enable'] = false - gitlab_exporter['enable'] = false - gitlab_workhorse['enable'] = false - nginx['enable'] = false - node_exporter['enable'] = false - pgbouncer_exporter['enable'] = false - postgresql['enable'] = false - prometheus['enable'] = false - redis['enable'] = false - redis_exporter['enable'] = false - patroni['enable'] = false - puma['enable'] = false - - ## - ## The unique identifier for the Geo node. + ## The unique identifier for the Geo site. See + ## https://docs.gitlab.com/ee/user/admin_area/geo_nodes.html#common-settings ## - gitlab_rails['geo_node_name'] = '<node_name_here>' + gitlab_rails['geo_node_name'] = '<site_name_here>' ## ## Disable automatic migrations @@ -451,12 +320,11 @@ application servers above, with some changes to run only the `sidekiq` service: gitlab_rails['auto_migrate'] = false ## - ## Configure the connection to the tracking DB. And disable application - ## servers from running tracking databases. + ## Configure the connection to the tracking database ## + geo_secondary['enable'] = true geo_secondary['db_host'] = '<geo_tracking_db_host>' geo_secondary['db_password'] = '<geo_tracking_db_password>' - geo_postgresql['enable'] = false ## ## Configure connection to the streaming replica database, if you haven't @@ -473,7 +341,7 @@ application servers above, with some changes to run only the `sidekiq` service: ## ## If you are using custom users not managed by Omnibus, you need to specify - ## UIDs and GIDs like below, and ensure they match between servers in a + ## UIDs and GIDs like below, and ensure they match between nodes in a ## cluster to avoid permissions issues ## user['uid'] = 9000 @@ -484,8 +352,8 @@ application servers above, with some changes to run only the `sidekiq` service: registry['gid'] = 9002 ``` - You can similarly configure a server to run only the `geo-logcursor` service + You can similarly configure a node to run only the `geo-logcursor` service with `geo_logcursor['enable'] = true` and disabling Sidekiq with `sidekiq['enable'] = false`. - These servers do not need to be attached to the load balancer. + These nodes do not need to be attached to the load balancer. diff --git a/doc/administration/geo/replication/object_storage.md b/doc/administration/geo/replication/object_storage.md index 90a41ed3e1c..d3931c0ba80 100644 --- a/doc/administration/geo/replication/object_storage.md +++ b/doc/administration/geo/replication/object_storage.md @@ -9,6 +9,11 @@ type: howto Geo can be used in combination with Object Storage (AWS S3, or other compatible object storage). +The storage method for files is recorded in the database, and the database is replicated +from the **primary** Geo site to the **secondary** Geo site, so the **secondary** Geo site +must match the storage method of the **primary** Geo site. +Therefore, if the **primary** Geo site uses object storage, the **secondary** Geo site must use it too. + Currently, **secondary** sites can use either: - The same storage bucket as the **primary** site. diff --git a/doc/administration/geo/replication/troubleshooting.md b/doc/administration/geo/replication/troubleshooting.md index d63e927627a..e072696b37c 100644 --- a/doc/administration/geo/replication/troubleshooting.md +++ b/doc/administration/geo/replication/troubleshooting.md @@ -39,7 +39,7 @@ to help identify if something is wrong: - Is the node's secondary tracking database connected? - Is the node's secondary tracking database up-to-date? - + For information on how to resolve common errors reported from the UI, see [Fixing Common Errors](#fixing-common-errors). @@ -159,6 +159,27 @@ This machine's Geo node name matches a database record ... no doc/administration/geo/replication/troubleshooting.md#can-geo-detect-the-current-node-correctly ``` +### Message: `WARNING: oldest xmin is far in the past` and `pg_wal` size growing + +If a replication slot is inactive, +the `pg_wal` logs corresponding to the slot are reserved forever +(or until the slot is active again). This causes continuous disk usage growth +and the following messages appear repeatedly in the +[PostgreSQL logs](../../logs.md#postgresql-logs): + +```plaintext +WARNING: oldest xmin is far in the past +HINT: Close open transactions soon to avoid wraparound problems. +You might also need to commit or roll back old prepared transactions, or drop stale replication slots. +``` + +To fix this, do the following: + +1. [Connect to the primary database](https://docs.gitlab.com/omnibus/settings/database.html#connecting-to-the-bundled-postgresql-database). +1. Run `SELECT * FROM pg_replication_slots;`. +1. Note the `slot_name` that reports `active` as `f` (false). +1. Follow [all these steps to remove that Geo site](remove_geo_site.md). + ## Fixing errors found when running the Geo check Rake task When running this Rake task, you may see errors if the nodes are not properly configured: @@ -299,7 +320,7 @@ sudo gitlab-ctl \ --backup-timeout=21600 ``` -This will give the initial replication up to six hours to complete, rather than +This gives the initial replication up to six hours to complete, rather than the default thirty minutes. Adjust as required for your installation. ### Message: "PANIC: could not write to file `pg_xlog/xlogtemp.123`: No space left on device" @@ -314,7 +335,7 @@ log data to build up in `pg_xlog`. Removing the unused slots can reduce the amou ``` NOTE: - Using `gitlab-rails dbconsole` will not work, because managing replication slots requires superuser permissions. + Using `gitlab-rails dbconsole` does not work, because managing replication slots requires superuser permissions. 1. View your replication slots with: @@ -325,7 +346,8 @@ log data to build up in `pg_xlog`. Removing the unused slots can reduce the amou Slots where `active` is `f` are not active. - When this slot should be active, because you have a **secondary** node configured using that slot, - log in to that **secondary** node and check the PostgreSQL logs why the replication is not running. + log in to that **secondary** node and check the [PostgreSQL logs](../../logs.md#postgresql-logs) + to view why the replication is not running. - If you are no longer using the slot (for example, you no longer have Geo enabled), you can remove it with in the PostgreSQL console session: @@ -336,13 +358,12 @@ Slots where `active` is `f` are not active. ### Message: "ERROR: canceling statement due to conflict with recovery" -This error may rarely occur under normal usage, and the system is resilient +This error occurs infrequently under normal usage, and the system is resilient enough to recover. However, under certain conditions, some database queries on secondaries may run -excessively long, which increases the frequency of this error. At some point, -some of these queries will never be able to complete due to being canceled -every time. +excessively long, which increases the frequency of this error. This can lead to a situation +where some queries never complete due to being canceled on every replication. These long-running queries are [planned to be removed in the future](https://gitlab.com/gitlab-org/gitlab/-/issues/34269), @@ -483,7 +504,7 @@ Then reconfigure GitLab: sudo gitlab-ctl reconfigure ``` -This will increase the timeout to four hours (14400 seconds). Choose a time +This increases the timeout to four hours (14400 seconds). Choose a time long enough to accommodate a full clone of your largest repositories. ### New LFS objects are never replicated @@ -521,7 +542,7 @@ to start again from scratch, there are a few steps that can help you: gitlab-ctl stop geo-logcursor ``` - You can watch Sidekiq logs to know when Sidekiq jobs processing have finished: + You can watch the [Sidekiq logs](../../logs.md#sidekiq-logs) to know when Sidekiq jobs processing has finished: ```shell gitlab-ctl tail sidekiq @@ -542,9 +563,8 @@ to start again from scratch, there are a few steps that can help you: 1. _(Optional)_ Rename other data folders and create new ones WARNING: - You may still have files on the **secondary** node that have been removed from **primary** node but - removal have not been reflected. If you skip this step, they will never be removed - from this Geo node. + You may still have files on the **secondary** node that have been removed from the **primary** node, but this + removal has not been reflected. If you skip this step, these files are not removed at all from the Geo node. Any uploaded content like file attachments, avatars or LFS objects are stored in a subfolder in one of the two paths below: @@ -639,12 +659,12 @@ Counts: #### If you are promoting a Geo secondary site running on a single server -`gitlab-ctl promotion-preflight-checks` will fail due to the existence of +`gitlab-ctl promotion-preflight-checks` fails due to the existence of `failed` rows in the `geo_design_registry` table. Use the [previous snippet](#design-repository-failures-on-mirrored-projects-and-project-imports) to determine the actual replication status of Design repositories. -`gitlab-ctl promote-to-primary-node` will fail since it runs preflight checks. +`gitlab-ctl promote-to-primary-node` fails since it runs preflight checks. If the [previous snippet](#design-repository-failures-on-mirrored-projects-and-project-imports) shows that all designs are synced, then you can use the `--skip-preflight-checks` option or the `--force` option to move forward with @@ -652,7 +672,7 @@ promotion. #### If you are promoting a Geo secondary site running on multiple servers -`gitlab-ctl promotion-preflight-checks` will fail due to the existence of +`gitlab-ctl promotion-preflight-checks` fails due to the existence of `failed` rows in the `geo_design_registry` table. Use the [previous snippet](#design-repository-failures-on-mirrored-projects-and-project-imports) to determine the actual replication status of Design repositories. @@ -744,7 +764,7 @@ When you need to run the `gitlab-pg-ctl` command to promote the PostgreSQL read-replica database. -In GitLab 12.8 and earlier, this command will fail with the message: +In GitLab 12.8 and earlier, this command fails with the message: ```plaintext sudo: gitlab-pg-ctl: command not found @@ -803,7 +823,7 @@ If you notice for some reason there are more artifacts on the Geo secondary node than on the Geo primary node, you can use the Rake task to [cleanup orphan artifact files](../../../raketasks/cleanup.md#remove-orphan-artifact-files). -On a Geo **secondary** node, this command will also clean up all Geo +On a Geo **secondary** node, this command also cleans up all Geo registry record related to the orphan files on disk. ## Fixing sign in errors diff --git a/doc/administration/geo/setup/database.md b/doc/administration/geo/setup/database.md index bc4128deb4a..fa343f7eb40 100644 --- a/doc/administration/geo/setup/database.md +++ b/doc/administration/geo/setup/database.md @@ -69,11 +69,14 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o sudo -i ``` -1. Edit `/etc/gitlab/gitlab.rb` and add a **unique** name for your node: +1. Edit `/etc/gitlab/gitlab.rb` and add a **unique** name for your site: ```ruby - # The unique identifier for the Geo node. - gitlab_rails['geo_node_name'] = '<node_name_here>' + ## + ## The unique identifier for the Geo site. See + ## https://docs.gitlab.com/ee/user/admin_area/geo_nodes.html#common-settings + ## + gitlab_rails['geo_node_name'] = '<site_name_here>' ``` 1. Reconfigure the **primary** node for the change to take effect: @@ -207,7 +210,12 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o ```ruby ## ## Geo Primary role - ## - configure dependent flags automatically to enable Geo + ## - Configures Postgres settings for replication + ## - Prevents automatic upgrade of Postgres since it requires downtime of + ## streaming replication to Geo secondary sites + ## - Enables standard single-node GitLab services like NGINX, Puma, Redis, + ## Sidekiq, etc. If you are segregating services, then you will need to + ## explicitly disable unwanted services. ## roles(['geo_primary_role']) @@ -571,6 +579,13 @@ Leader instance**: patroni['password'] = 'PATRONI_API_PASSWORD' patroni['replication_password'] = 'PLAIN_TEXT_POSTGRESQL_REPLICATION_PASSWORD' + # Add all patroni nodes to the allowlist + patroni['allowlist'] = %w[ + 127.0.0.1/32 + PATRONI_PRIMARY1_IP/32 PATRONI_PRIMARY2_IP/32 PATRONI_PRIMARY3_IP/32 + PATRONI_SECONDARY1_IP/32 PATRONI_SECONDARY2_IP/32 PATRONI_SECONDARY3_IP/32 + ] + # We list all secondary instances as they can all become a Standby Leader postgresql['md5_auth_cidr_addresses'] = %w[ PATRONI_PRIMARY1_IP/32 PATRONI_PRIMARY2_IP/32 PATRONI_PRIMARY3_IP/32 PATRONI_PRIMARY_PGBOUNCER/32 @@ -725,6 +740,13 @@ For each Patroni instance on the secondary site: # Any other instance that needs access to the database as per documentation ] + + # Add patroni nodes to the allowlist + patroni['allowlist'] = %w[ + 127.0.0.1/32 + PATRONI_SECONDARY1_IP/32 PATRONI_SECONDARY2_IP/32 PATRONI_SECONDARY3_IP/32 + ] + patroni['standby_cluster']['enable'] = true patroni['standby_cluster']['host'] = 'INTERNAL_LOAD_BALANCER_PRIMARY_IP' patroni['standby_cluster']['port'] = INTERNAL_LOAD_BALANCER_PRIMARY_PORT @@ -903,6 +925,12 @@ For each Patroni instance on the secondary site for the tracking database: # Any other instance that needs access to the database as per documentation ] + # Add patroni nodes to the allowlist + patroni['allowlist'] = %w[ + 127.0.0.1/32 + PATRONI_TRACKINGDB1_IP/32 PATRONI_TRACKINGDB2_IP/32 PATRONI_TRACKINGDB3_IP/32 + ] + # Patroni configuration patroni['username'] = 'PATRONI_API_USERNAME' patroni['password'] = 'PATRONI_API_PASSWORD' diff --git a/doc/administration/geo/setup/external_database.md b/doc/administration/geo/setup/external_database.md index 3ec84f1268b..56d196b54ec 100644 --- a/doc/administration/geo/setup/external_database.md +++ b/doc/administration/geo/setup/external_database.md @@ -34,9 +34,10 @@ developed and tested. We aim to be compatible with most external roles ['geo_primary_role'] ## - ## The unique identifier for the Geo site. + ## The unique identifier for the Geo site. See + ## https://docs.gitlab.com/ee/user/admin_area/geo_nodes.html#common-settings ## - gitlab_rails['geo_node_name'] = '<geo_site_name_here>' + gitlab_rails['geo_node_name'] = '<site_name_here>' ``` 1. Reconfigure the **primary** node for the change to take effect: @@ -242,7 +243,7 @@ the tracking database on port 5432. 1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) -1. The reconfigure should automatically create the database. If needed, you can perform this task manually. Note that this task (whether run by itself or during reconfigure) requires the database user to be a superuser. +1. The reconfigure should automatically create the database. If needed, you can perform this task manually. This task (whether run by itself or during reconfigure) requires the database user to be a superuser. ```shell gitlab-rake geo:db:create |
