diff options
Diffstat (limited to 'doc/administration/geo')
30 files changed, 309 insertions, 646 deletions
diff --git a/doc/administration/geo/disaster_recovery/background_verification.md b/doc/administration/geo/disaster_recovery/background_verification.md index f2854d0538b..f573b64b5f1 100644 --- a/doc/administration/geo/disaster_recovery/background_verification.md +++ b/doc/administration/geo/disaster_recovery/background_verification.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Automatic background verification **(PREMIUM ONLY)** +# Automatic background verification **(PREMIUM SELF)** NOTE: Automatic background verification of repositories and wikis was added in @@ -146,8 +146,8 @@ If the **primary** and **secondary** nodes have a checksum verification mismatch **Edit** button:  -1. On the project admin page get the **Gitaly storage name**, and **Gitaly relative path**: -  +1. On the project administration page get the **Gitaly storage name**, and **Gitaly relative path**: +  1. Navigate to the project's repository directory on both **primary** and **secondary** nodes (the path is usually `/var/opt/gitlab/git-data/repositories`). Note that if `git_data_dirs` diff --git a/doc/administration/geo/disaster_recovery/bring_primary_back.md b/doc/administration/geo/disaster_recovery/bring_primary_back.md index 3f86e03bd7f..d06f11bbe09 100644 --- a/doc/administration/geo/disaster_recovery/bring_primary_back.md +++ b/doc/administration/geo/disaster_recovery/bring_primary_back.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Bring a demoted primary node back online **(PREMIUM ONLY)** +# Bring a demoted primary node back online **(PREMIUM SELF)** After a failover, it is possible to fail back to the demoted **primary** node to restore your original configuration. This process consists of two steps: diff --git a/doc/administration/geo/disaster_recovery/index.md b/doc/administration/geo/disaster_recovery/index.md index 2b0a3e2f114..43e5dc1d224 100644 --- a/doc/administration/geo/disaster_recovery/index.md +++ b/doc/administration/geo/disaster_recovery/index.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Disaster Recovery (Geo) **(PREMIUM ONLY)** +# Disaster Recovery (Geo) **(PREMIUM SELF)** Geo replicates your database, your Git repositories, and few other assets. We will support and replicate more data in the future, that will enable you to @@ -145,6 +145,13 @@ Note the following when promoting a secondary: a point-in-time recovery to the last known state. Data that was created on the primary while the secondary was paused will be lost. + NOTE: + In GitLab 13.7 and earlier, if you have a data type with zero items to sync, + this command reports `ERROR - Replication is not up-to-date` even if + replication is actually up-to-date. If replication and verification output + shows that it is complete, you can add `--skip-preflight-checks` to make the + command complete promotion. This bug was fixed in GitLab 13.8 and later. + To promote the secondary node to primary along with preflight checks: ```shell @@ -245,12 +252,6 @@ Data that was created on the primary while the secondary was paused will be lost sudo gitlab-ctl promote-db ``` -1. Disable Patroni auto-failover: - - ```shell - sudo gitlab-ctl patroni pause - ``` - 1. Edit `/etc/gitlab/gitlab.rb` on every application and Sidekiq nodes in the secondary to reflect its new status as primary by removing any lines that enabled the `geo_secondary_role`: ```ruby @@ -273,12 +274,6 @@ Data that was created on the primary while the secondary was paused will be lost sudo gitlab-ctl reconfigure ``` -1. Resume Patroni auto-failover: - - ```shell - sudo gitlab-ctl patroni resume - ``` - 1. Promote the **secondary** to **primary**. SSH into a single application server and execute: ```shell diff --git a/doc/administration/geo/disaster_recovery/planned_failover.md b/doc/administration/geo/disaster_recovery/planned_failover.md index 1468c5cd39d..e64d0d4983e 100644 --- a/doc/administration/geo/disaster_recovery/planned_failover.md +++ b/doc/administration/geo/disaster_recovery/planned_failover.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Disaster recovery for planned failover **(PREMIUM ONLY)** +# Disaster recovery for planned failover **(PREMIUM SELF)** The primary use-case of Disaster Recovery is to ensure business continuity in the event of unplanned outage, but it can be used in conjunction with a planned @@ -45,6 +45,12 @@ be found in `/var/opt/gitlab/gitlab-rails/shared/pages` if using Omnibus). ## Preflight checks +NOTE: +In GitLab 13.7 and earlier, if you have a data type with zero items to sync, +this command reports `ERROR - Replication is not up-to-date` even if +replication is actually up-to-date. This bug was fixed in GitLab 13.8 and +later. + Run this command to list out all preflight checks and automatically check if replication and verification are complete before scheduling a planned failover to ensure the process will go smoothly: ```shell @@ -138,41 +144,10 @@ will take to finish syncing. An example message would be: ## Prevent updates to the **primary** node -Until a [read-only mode](https://gitlab.com/gitlab-org/gitlab/-/issues/14609) is implemented, updates must be prevented -from happening manually. Note that your **secondary** node still needs read-only -access to the **primary** node during the maintenance window. - -1. At the scheduled time, using your cloud provider or your node's firewall, block - all HTTP, HTTPS and SSH traffic to/from the **primary** node, **except** for your IP and - the **secondary** node's IP. - - For instance, you might run the following commands on the server(s) making up your **primary** node: - - ```shell - sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 22 -j ACCEPT - sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 22 -j ACCEPT - sudo iptables -A INPUT --destination-port 22 -j REJECT - - sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 80 -j ACCEPT - sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 80 -j ACCEPT - sudo iptables -A INPUT --tcp-dport 80 -j REJECT - - sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 443 -j ACCEPT - sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 443 -j ACCEPT - sudo iptables -A INPUT --tcp-dport 443 -j REJECT - ``` - - From this point, users will be unable to view their data or make changes on the - **primary** node. They will also be unable to log in to the **secondary** node. - However, existing sessions will work for the remainder of the maintenance period, and - public data will be accessible throughout. - -1. Verify the **primary** node is blocked to HTTP traffic by visiting it in browser via - another IP. The server should refuse connection. +To ensure that all data is replicated to a secondary site, updates (write requests) need to +be disabled on the primary site: -1. Verify the **primary** node is blocked to Git over SSH traffic by attempting to pull an - existing Git repository with an SSH remote URL. The server should refuse - connection. +1. Enable [maintenance mode](../../maintenance_mode/index.md). 1. Disable non-Geo periodic background jobs on the **primary** node by navigating to **Admin Area > Monitoring > Background Jobs > Cron**, pressing `Disable All`, diff --git a/doc/administration/geo/disaster_recovery/promotion_runbook.md b/doc/administration/geo/disaster_recovery/promotion_runbook.md deleted file mode 100644 index eef1e7ae9dd..00000000000 --- a/doc/administration/geo/disaster_recovery/promotion_runbook.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -redirect_to: runbooks/planned_failover_single_node.md ---- - -This document was moved to [another location](runbooks/planned_failover_single_node.md). - -<!-- This redirect file can be deleted after February 1, 2021. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md index f17990ce5f8..3227fafca0f 100644 --- a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md +++ b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md @@ -9,7 +9,7 @@ WARNING: This runbook is in **alpha**. For complete, production-ready documentation, see the [disaster recovery documentation](../index.md). -# Disaster Recovery (Geo) promotion runbooks **(PREMIUM ONLY)** +# Disaster Recovery (Geo) promotion runbooks **(PREMIUM SELF)** ## Geo planned failover for a multi-node configuration diff --git a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md index a2a9350e411..7f311d172ef 100644 --- a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md +++ b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md @@ -9,7 +9,7 @@ WARNING: This runbook is in **alpha**. For complete, production-ready documentation, see the [disaster recovery documentation](../index.md). -# Disaster Recovery (Geo) promotion runbooks **(PREMIUM ONLY)** +# Disaster Recovery (Geo) promotion runbooks **(PREMIUM SELF)** ## Geo planned failover for a single-node configuration @@ -233,12 +233,25 @@ To promote the secondary node: check if replication and verification are complete before scheduling a planned failover to ensure the process will go smoothly: + NOTE: + In GitLab 13.7 and earlier, if you have a data type with zero items to sync, + this command reports `ERROR - Replication is not up-to-date` even if + replication is actually up-to-date. This bug was fixed in GitLab 13.8 and + later. + ```shell gitlab-ctl promotion-preflight-checks ``` 1. Promote the **secondary**: + NOTE: + In GitLab 13.7 and earlier, if you have a data type with zero items to sync, + this command reports `ERROR - Replication is not up-to-date` even if + replication is actually up-to-date. If replication and verification output + shows that it is complete, you can add `--skip-preflight-checks` to make the + command complete promotion. This bug was fixed in GitLab 13.8 and later. + ```shell gitlab-ctl promote-to-primary-node ``` diff --git a/doc/administration/geo/index.md b/doc/administration/geo/index.md index 1985ea2e04c..296500e35e2 100644 --- a/doc/administration/geo/index.md +++ b/doc/administration/geo/index.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Geo **(PREMIUM ONLY)** +# Geo **(PREMIUM SELF)** > - Introduced in GitLab Enterprise Edition 8.9. > - Using Geo in combination with diff --git a/doc/administration/geo/replication/configuration.md b/doc/administration/geo/replication/configuration.md index 5adf256f6fa..154815efa51 100644 --- a/doc/administration/geo/replication/configuration.md +++ b/doc/administration/geo/replication/configuration.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Geo configuration **(PREMIUM ONLY)** +# Geo configuration **(PREMIUM SELF)** ## Configuring a new **secondary** node @@ -229,7 +229,7 @@ keys must be manually replicated to the **secondary** node. gitlab-rake gitlab:geo:check ``` -Once added to the admin panel and restarted, the **secondary** node will automatically start +Once added to the Geo administration page and restarted, the **secondary** node will automatically start replicating missing data from the **primary** node in a process known as **backfill**. Meanwhile, the **primary** node will start to notify each **secondary** node of any changes, so that the **secondary** node can act on those notifications immediately. @@ -299,7 +299,7 @@ Currently, this is what is synced: ## Selective synchronization -Geo supports selective synchronization, which allows admins to choose +Geo supports selective synchronization, which allows administrators to choose which projects should be synchronized by **secondary** nodes. A subset of projects can be chosen, either by group or by storage shard. The former is ideal for replicating data belonging to a subset of users, while the diff --git a/doc/administration/geo/replication/database.md b/doc/administration/geo/replication/database.md deleted file mode 100644 index 099a73e93d8..00000000000 --- a/doc/administration/geo/replication/database.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -redirect_to: '../setup/database.md' ---- - -This document was moved to [another location](../setup/index.md). - -<!-- This redirect file can be deleted after February 1, 2021. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/geo/replication/datatypes.md b/doc/administration/geo/replication/datatypes.md index 0d2922c3347..f2913dd55ce 100644 --- a/doc/administration/geo/replication/datatypes.md +++ b/doc/administration/geo/replication/datatypes.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Geo data types support **(PREMIUM ONLY)** +# Geo data types support **(PREMIUM SELF)** A Geo data type is a specific class of data that is required by one or more GitLab features to store relevant information. @@ -35,21 +35,21 @@ verification methods: | Git | Object pools for forked project deduplication | Geo with Gitaly | _Not implemented_ | | Git | Project Snippets | Geo with Gitaly | _Not implemented_ | | Git | Personal Snippets | Geo with Gitaly | _Not implemented_ | -| Blobs | User uploads _(filesystem)_ | Geo with API | _Not implemented_ | +| Blobs | User uploads _(file system)_ | Geo with API | _Not implemented_ | | Blobs | User uploads _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | -| Blobs | LFS objects _(filesystem)_ | Geo with API | _Not implemented_ | +| Blobs | LFS objects _(file system)_ | Geo with API | _Not implemented_ | | Blobs | LFS objects _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | -| Blobs | CI job artifacts _(filesystem)_ | Geo with API | _Not implemented_ | +| Blobs | CI job artifacts _(file system)_ | Geo with API | _Not implemented_ | | Blobs | CI job artifacts _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | -| Blobs | Archived CI build traces _(filesystem)_ | Geo with API | _Not implemented_ | +| Blobs | Archived CI build traces _(file system)_ | Geo with API | _Not implemented_ | | Blobs | Archived CI build traces _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | -| Blobs | Container registry _(filesystem)_ | Geo with API/Docker API | _Not implemented_ | +| Blobs | Container registry _(file system)_ | Geo with API/Docker API | _Not implemented_ | | Blobs | Container registry _(object storage)_ | Geo with API/Managed/Docker API (*2*) | _Not implemented_ | -| Blobs | Package registry _(filesystem)_ | Geo with API | _Not implemented_ | +| Blobs | Package registry _(file system)_ | Geo with API | _Not implemented_ | | Blobs | Package registry _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | -| Blobs | Versioned Terraform State _(filesystem)_ | Geo with API | _Not implemented_ | +| Blobs | Versioned Terraform State _(file system)_ | Geo with API | _Not implemented_ | | Blobs | Versioned Terraform State _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | -| Blobs | External Merge Request Diffs _(filesystem)_ | Geo with API | _Not implemented_ | +| Blobs | External Merge Request Diffs _(file system)_ | Geo with API | _Not implemented_ | | Blobs | External Merge Request Diffs _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | - (*1*): Redis replication can be used as part of HA with Redis sentinel. It's not used between Geo nodes. @@ -63,14 +63,14 @@ is responsible for allowing access and operations on the locally stored Git repo on a machine with a single disk, multiple disks mounted as a single mount-point (like with a RAID array), or using LVM. -It requires no special filesystem and can work with NFS or a mounted Storage Appliance (there may be -performance limitations when using a remote filesystem). +It requires no special file system and can work with NFS or a mounted Storage Appliance (there may be +performance limitations when using a remote file system). Communication is done via Gitaly's own gRPC API. There are three possible ways of synchronization: - Using regular Git clone/fetch from one Geo node to another (with special authentication). - Using repository snapshots (for when the first method fails or repository is corrupt). -- Manual trigger from the Admin UI (a combination of both of the above). +- Manual trigger from the Admin Area (a combination of both of the above). Each project can have at most 3 different repositories: @@ -88,13 +88,13 @@ Both types will be synced to a secondary node. GitLab stores files and blobs such as Issue attachments or LFS objects into either: -- The filesystem in a specific location. +- The file system in a specific location. - An [Object Storage](../../object_storage.md) solution. Object Storage solutions can be: - Cloud based like Amazon S3 Google Cloud Storage. - Hosted by you (like MinIO). - A Storage Appliance that exposes an Object Storage-compatible API. -When using the filesystem store instead of Object Storage, you need to use network mounted filesystems +When using the file system store instead of Object Storage, you need to use network mounted file systems to run GitLab when using more than one server. With respect to replication and verification: @@ -144,9 +144,9 @@ The replication for some data types is behind a corresponding feature flag: > - They're enabled on GitLab.com. > - They can't be enabled or disabled per-project. > - They are recommended for production use. -> - For GitLab self-managed instances, GitLab administrators can opt to [disable them](#enable-or-disable-replication-for-some-data-types). **(CORE ONLY)** +> - For GitLab self-managed instances, GitLab administrators can opt to [disable them](#enable-or-disable-replication-for-some-data-types). **(FREE SELF)** -#### Enable or disable replication (for some data types) **(CORE ONLY)** +#### Enable or disable replication (for some data types) Replication for some data types are released behind feature flags that are **enabled by default**. [GitLab administrators with access to the GitLab Rails console](../../feature_flags.md) can opt to disable it for your instance. You can find feature flag names of each of those data types in the notes column of the table below. @@ -173,7 +173,7 @@ successfully, you must replicate their data using some other means. | Feature | Replicated (added in GitLab version) | Verified (added in GitLab version) | Object Storage replication (see [Geo with Object Storage](object_storage.md)) | Notes | |:---------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------|:----------------------------------------------------------|:-------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | [Application data in PostgreSQL](../../postgresql/index.md) | **Yes** (10.2) | **Yes** (10.2) | No | | -| [Project repository](../../..//user/project/repository/) | **Yes** (10.2) | **Yes** (10.7) | No | | +| [Project repository](../../../user/project/repository/) | **Yes** (10.2) | **Yes** (10.7) | No | | | [Project wiki repository](../../../user/project/wiki/) | **Yes** (10.2) | **Yes** (10.7) | No | | [Group wiki repository](../../../user/group/index.md#group-wikis) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/208147) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/208147) | No | | | [Uploads](../../uploads.md) | **Yes** (10.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | No | Verified only on transfer or manually using [Integrity Check Rake Task](../../raketasks/check.md) on both nodes and comparing the output between them. | @@ -186,13 +186,13 @@ successfully, you must replicate their data using some other means. | [Container Registry](../../packages/container_registry.md) | **Yes** (12.3) | No | No | Disabled by default. See [instructions](docker_registry.md) to enable. | | [Content in object storage (beta)](object_storage.md) | **Yes** (12.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/13845) | No | | | [Project designs repository](../../../user/project/issues/design_management.md) | **Yes** (12.7) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/32467) | Via Object Storage provider if supported. Native Geo support (Beta). | | -| [NPM Registry](../../../user/packages/npm_registry/index.md) | **Yes** (13.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | -| [Maven Repository](../../../user/packages/maven_repository/index.md) | **Yes** (13.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | -| [Conan Repository](../../../user/packages/conan_repository/index.md) | **Yes** (13.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | -| [NuGet Repository](../../../user/packages/nuget_repository/index.md) | **Yes** (13.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | -| [PyPI Repository](../../../user/packages/pypi_repository/index.md) | **Yes** (13.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | -| [Composer Repository](../../../user/packages/composer_repository/index.md) | **Yes** (13.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | -| [Generic packages](../../../user/packages/generic_packages/index.md) | **Yes** (13.5) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | +| [Package Registry for npm](../../../user/packages/npm_registry/index.md) | **Yes** (13.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | +| [Package Registry for Maven](../../../user/packages/maven_repository/index.md) | **Yes** (13.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | +| [Package Registry for Conan](../../../user/packages/conan_repository/index.md) | **Yes** (13.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | +| [Package Registry for NuGet](../../../user/packages/nuget_repository/index.md) | **Yes** (13.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | +| [Package Registry for PyPI](../../../user/packages/pypi_repository/index.md) | **Yes** (13.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | +| [Package Registry for Composer](../../../user/packages/composer_repository/index.md) | **Yes** (13.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | +| [Package Registry for generic packages](../../../user/packages/generic_packages/index.md) | **Yes** (13.5) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | | [Versioned Terraform State](../../terraform_state.md) | **Yes** (13.5) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_terraform_state_version_replication`, enabled by default | | [External merge request diffs](../../merge_request_diffs.md) | **Yes** (13.5) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_merge_request_diff_replication`, enabled by default | | [Versioned snippets](../../../user/snippets.md#versioned-snippets) | [**Yes** (13.7)](https://gitlab.com/groups/gitlab-org/-/epics/2809) | [No](https://gitlab.com/groups/gitlab-org/-/epics/2810) | No | | @@ -201,4 +201,4 @@ successfully, you must replicate their data using some other means. | [GitLab Pages](../../pages/index.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/589) | No | No | | | [CI Pipeline Artifacts](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/ci/pipeline_artifact.rb) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/238464) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Persists additional artifacts after a pipeline completes | | [Dependency proxy images](../../../user/packages/dependency_proxy/index.md) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/259694) | No | No | Blocked on [Geo: Secondary Mimicry](https://gitlab.com/groups/gitlab-org/-/epics/1528). Note that replication of this cache is not needed for Disaster Recovery purposes because it can be recreated from external sources. | -| [Vulnerability Export](../../../user/application_security/security_dashboard/#export-vulnerabilities) | [Not planned](https://gitlab.com/groups/gitlab-org/-/epics/3111) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Not planned because they are ephemeral and sensitive. They can be regenerated on demand. | +| [Vulnerability Export](../../../user/application_security/vulnerability_report/#export-vulnerabilities) | [Not planned](https://gitlab.com/groups/gitlab-org/-/epics/3111) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Not planned because they are ephemeral and sensitive. They can be regenerated on demand. | diff --git a/doc/administration/geo/replication/disable_geo.md b/doc/administration/geo/replication/disable_geo.md index e8767a56266..a26adcefef5 100644 --- a/doc/administration/geo/replication/disable_geo.md +++ b/doc/administration/geo/replication/disable_geo.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Disabling Geo **(PREMIUM ONLY)** +# Disabling Geo **(PREMIUM SELF)** If you want to revert to a regular Omnibus setup after a test, or you have encountered a Disaster Recovery situation and you want to disable Geo momentarily, you can use these instructions to disable your diff --git a/doc/administration/geo/replication/docker_registry.md b/doc/administration/geo/replication/docker_registry.md index f92a878662e..1669abbc52a 100644 --- a/doc/administration/geo/replication/docker_registry.md +++ b/doc/administration/geo/replication/docker_registry.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Docker Registry for a secondary node **(PREMIUM ONLY)** +# Docker Registry for a secondary node **(PREMIUM SELF)** You can set up a [Docker Registry](https://docs.docker.com/registry/) on your **secondary** Geo node that mirrors the one on the **primary** Geo node. diff --git a/doc/administration/geo/replication/external_database.md b/doc/administration/geo/replication/external_database.md deleted file mode 100644 index dfaf6819fa0..00000000000 --- a/doc/administration/geo/replication/external_database.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -redirect_to: '../setup/external_database.md' ---- - -This document was moved to [another location](../setup/external_database.md). - -<!-- This redirect file can be deleted after February 1, 2021. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/geo/replication/faq.md b/doc/administration/geo/replication/faq.md index ab8199c3717..73a36f5e674 100644 --- a/doc/administration/geo/replication/faq.md +++ b/doc/administration/geo/replication/faq.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Geo Frequently Asked Questions **(PREMIUM ONLY)** +# Geo Frequently Asked Questions **(PREMIUM SELF)** ## What are the minimum requirements to run Geo? diff --git a/doc/administration/geo/replication/geo_validation_tests.md b/doc/administration/geo/replication/geo_validation_tests.md index 2d701458e66..f050d3e708c 100644 --- a/doc/administration/geo/replication/geo_validation_tests.md +++ b/doc/administration/geo/replication/geo_validation_tests.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Geo validation tests **(PREMIUM ONLY)** +# Geo validation tests **(PREMIUM SELF)** The Geo team performs manual testing and validation on common deployment configurations to ensure that Geo works when upgrading between minor GitLab versions and major PostgreSQL database versions. @@ -53,7 +53,7 @@ The following are GitLab upgrade validation tests we performed. - Outcome: Partial success because we did not run the looping pipeline during the demo to validate zero-downtime. - Follow up issues: - - [Clarify hup Puma/Unicorn should include deploy node](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5460) + - [Clarify how Puma/Unicorn should include deploy node](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5460) - [Investigate MR creation failure after upgrade to 12.9.10](https://gitlab.com/gitlab-org/gitlab/-/issues/223282) Closed as false positive. ### February 2020 @@ -128,7 +128,7 @@ The following are PostgreSQL upgrade validation tests we performed. PostgreSQL 12 with a database cluster on the primary is not recommended until the issues are resolved. - Known issues for PostgreSQL clusters: - [Ensure Patroni detects PostgreSQL update](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5423) - - [Allow configuring permanent replication slots in patroni](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5628) + - [Allow configuring permanent replication slots in Patroni](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5628) ### August 2020 diff --git a/doc/administration/geo/replication/high_availability.md b/doc/administration/geo/replication/high_availability.md deleted file mode 100644 index 1da4246db1f..00000000000 --- a/doc/administration/geo/replication/high_availability.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -redirect_to: 'multiple_servers.md' ---- - -This document was moved to [another location](multiple_servers.md). - -<!-- This redirect file can be deleted after February 1, 2021. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/geo/replication/index.md b/doc/administration/geo/replication/index.md deleted file mode 100644 index 955e05491d2..00000000000 --- a/doc/administration/geo/replication/index.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -redirect_to: '../index.md' ---- - -This document was moved to [another location](../index.md). - -<!-- This redirect file can be deleted after February 1, 2021. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/geo/replication/location_aware_git_url.md b/doc/administration/geo/replication/location_aware_git_url.md index f2cf8c9bda0..0eea792d374 100644 --- a/doc/administration/geo/replication/location_aware_git_url.md +++ b/doc/administration/geo/replication/location_aware_git_url.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Location-aware Git remote URL with AWS Route53 **(PREMIUM ONLY)** +# Location-aware Git remote URL with AWS Route53 **(PREMIUM SELF)** You can provide GitLab users with a single remote URL that automatically uses the Geo node closest to them. This means users don't need to update their Git diff --git a/doc/administration/geo/replication/multiple_servers.md b/doc/administration/geo/replication/multiple_servers.md index 0ab972c9077..f83e0e14e54 100644 --- a/doc/administration/geo/replication/multiple_servers.md +++ b/doc/administration/geo/replication/multiple_servers.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Geo for multiple nodes **(PREMIUM ONLY)** +# Geo for multiple nodes **(PREMIUM SELF)** This document describes a minimal reference architecture for running Geo in a multi-node configuration. If your multi-node setup differs from the one diff --git a/doc/administration/geo/replication/object_storage.md b/doc/administration/geo/replication/object_storage.md index ce791d66b34..ad419f999b3 100644 --- a/doc/administration/geo/replication/object_storage.md +++ b/doc/administration/geo/replication/object_storage.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Geo with Object storage **(PREMIUM ONLY)** +# Geo with Object storage **(PREMIUM SELF)** Geo can be used in combination with Object Storage (AWS S3, or other compatible object storage). @@ -29,7 +29,7 @@ WARNING: This is a [**beta** feature](https://about.gitlab.com/handbook/product/#beta) and is not ready yet for production use at any scale. The main limitations are a lack of testing at scale and no verification of any replicated data. **Secondary** nodes can replicate files stored on the **primary** node regardless of -whether they are stored on the local filesystem or in object storage. +whether they are stored on the local file system or in object storage. To enable GitLab replication, you must: diff --git a/doc/administration/geo/replication/remove_geo_node.md b/doc/administration/geo/replication/remove_geo_node.md index 519b93b447b..26150a6f536 100644 --- a/doc/administration/geo/replication/remove_geo_node.md +++ b/doc/administration/geo/replication/remove_geo_node.md @@ -5,15 +5,15 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Removing secondary Geo nodes **(PREMIUM ONLY)** +# Removing secondary Geo nodes **(PREMIUM SELF)** -**Secondary** nodes can be removed from the Geo cluster using the Geo admin page of the **primary** node. To remove a **secondary** node: +**Secondary** nodes can be removed from the Geo cluster using the Geo administration page of the **primary** node. To remove a **secondary** node: 1. Navigate to **Admin Area > Geo** (`/admin/geo/nodes`). 1. Click the **Remove** button for the **secondary** node you want to remove. 1. Confirm by clicking **Remove** when the prompt appears. -Once removed from the Geo admin page, you must stop and uninstall the **secondary** node: +Once removed from the Geo administration page, you must stop and uninstall the **secondary** node: 1. On the **secondary** node, stop GitLab: diff --git a/doc/administration/geo/replication/security_review.md b/doc/administration/geo/replication/security_review.md index 7c15662340c..c31881910bc 100644 --- a/doc/administration/geo/replication/security_review.md +++ b/doc/administration/geo/replication/security_review.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Geo security review (Q&A) **(PREMIUM ONLY)** +# Geo security review (Q&A) **(PREMIUM SELF)** The following security review of the Geo feature set focuses on security aspects of the feature as they apply to customers running their own GitLab instances. The review @@ -78,7 +78,7 @@ from [owasp.org](https://owasp.org/). ### Who has administrative capabilities in the application? - Nothing Geo-specific. Any user where `admin: true` is set in the database is - considered an admin with super-user privileges. + considered an administrator with super-user privileges. - See also: [more granular access control](https://gitlab.com/gitlab-org/gitlab/-/issues/18242) (not Geo-specific). - Much of Geo’s integration (database replication, for instance) must be @@ -194,7 +194,7 @@ from [owasp.org](https://owasp.org/). administrator via SSH, and then back out to the **secondary** node in the same manner. In particular, this includes the PostgreSQL replication credentials and a secret key (`db_key_base`) which is used to decrypt certain columns in the database. - The `db_key_base` secret is stored unencrypted on the filesystem, in + The `db_key_base` secret is stored unencrypted on the file system, in `/etc/gitlab/gitlab-secrets.json`, along with a number of other secrets. There is no at-rest protection for them. @@ -217,7 +217,7 @@ from [owasp.org](https://owasp.org/). - **Secondary** nodes and **primary** nodes interact via HTTP/HTTPS (secured with JSON web tokens) and via PostgreSQL streaming replication. -- Within a **primary** node or **secondary** node, the SSOT is the filesystem and the database +- Within a **primary** node or **secondary** node, the SSOT is the file system and the database (including Geo tracking database on **secondary** node). The various internal components are orchestrated to make alterations to these stores. @@ -231,7 +231,7 @@ from [owasp.org](https://owasp.org/). ### What data is or may need to be encrypted and what key management requirements have been defined? -- Neither **primary** nodes or **secondary** nodes encrypt Git repository or filesystem data at +- Neither **primary** nodes or **secondary** nodes encrypt Git repository or file system data at rest. A subset of database columns are encrypted at rest using the `db_otp_key`. - A static secret shared across all hosts in a GitLab deployment. - In transit, data should be encrypted, although the application does permit @@ -287,5 +287,5 @@ from [owasp.org](https://owasp.org/). ### What application auditing requirements have been defined? How are audit and debug logs accessed, stored, and secured? -- Structured JSON log is written to the filesystem, and can also be ingested +- Structured JSON log is written to the file system, and can also be ingested into a Kibana installation for further analysis. diff --git a/doc/administration/geo/replication/troubleshooting.md b/doc/administration/geo/replication/troubleshooting.md index 2c0160ed02a..3b1f60a0f3f 100644 --- a/doc/administration/geo/replication/troubleshooting.md +++ b/doc/administration/geo/replication/troubleshooting.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Troubleshooting Geo **(PREMIUM ONLY)** +# Troubleshooting Geo **(PREMIUM SELF)** Setting up Geo requires careful attention to details and sometimes it's easy to miss a step. @@ -219,7 +219,7 @@ sudo gitlab-rake gitlab:geo:check ``` - Ensure that you have added the secondary node in the Admin Area of the **primary** node. - - Ensure that you entered the `external_url` or `gitlab_rails['geo_node_name']` when adding the secondary node in the admin are of the **primary** node. + - Ensure that you entered the `external_url` or `gitlab_rails['geo_node_name']` when adding the secondary node in the Admin Area of the **primary** node. - Prior to GitLab 12.4, edit the secondary node in the Admin Area of the **primary** node and ensure that there is a trailing `/` in the `Name` field. 1. Check returns `Exception: PG::UndefinedTable: ERROR: relation "geo_nodes" does not exist` @@ -396,6 +396,69 @@ In GitLab 13.4, a seed project is added when GitLab is first installed. This mak on a new Geo secondary node. There is an [issue to account for seed projects](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5618) when checking the database. +### Message: `Synchronization failed - Error syncing repository` + +WARNING: +If large repositories are affected by this problem, +their resync may take a long time and cause significant load on your Geo nodes, +storage and network systems. + +If you get the error `Synchronization failed - Error syncing repository` along with the following log messages, this indicates that the expected `geo` remote is not present in the `.git/config` file +of a repository on the secondary Geo node's filesystem: + +```json +{ + "created": "@1603481145.084348757", + "description": "Error received from peer unix:/var/opt/gitlab/gitaly/gitaly.socket", + … + "grpc_message": "exit status 128", + "grpc_status": 13 +} +{ … + "grpc.request.fullMethod": "/gitaly.RemoteService/FindRemoteRootRef", + "grpc.request.glProjectPath": "<namespace>/<project>", + … + "level": "error", + "msg": "fatal: 'geo' does not appear to be a git repository + fatal: Could not read from remote repository. …", +} +``` + +To solve this: + +1. Log into the secondary Geo node. + +1. Back up [the `.git` folder](../../repository_storage_types.md#translating-hashed-storage-paths). + +1. Optional: [Spot-check](../../troubleshooting/log_parsing.md#find-all-projects-affected-by-a-fatal-git-problem)) + a few of those IDs whether they indeed correspond + to a project with known Geo replication failures. + Use `fatal: 'geo'` as the `grep` term and the following API call: + + ```shell + curl --request GET --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/<first_failed_geo_sync_ID>" + ``` + +1. Enter the [Rails console](../../troubleshooting/navigating_gitlab_via_rails_console.md) and run: + + ```ruby + failed_geo_syncs = Geo::ProjectRegistry.failed.pluck(:id) + failed_geo_syncs.each do |fgs| + puts Geo::ProjectRegistry.failed.find(fgs).project_id + end + ``` + +1. Run the following commands to reset each project's + Geo-related attributes and execute a new sync: + + ```ruby + failed_geo_syncs.each do |fgs| + registry = Geo::ProjectRegistry.failed.find(fgs) + registry.update(resync_repository: true, force_to_redownload_repository: false, repository_retry_count: 0) + Geo::RepositorySyncService.new(registry.project).execute + end + ``` + ### Very large repositories never successfully synchronize on the **secondary** node GitLab places a timeout on all repository clones, including project imports @@ -678,19 +741,19 @@ sudo /opt/gitlab/embedded/bin/gitlab-pg-ctl promote GitLab 12.9 and later are [unaffected by this error](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5147). -### Two-factor authentication is broken after a failover +### Message: `ERROR - Replication is not up-to-date` during `gitlab-ctl promotion-preflight-checks` + +In GitLab 13.7 and earlier, if you have a data type with zero items to sync, +this command reports `ERROR - Replication is not up-to-date` even if +replication is actually up-to-date. This bug was fixed in GitLab 13.8 and +later. -The setup instructions for Geo prior to 10.5 failed to replicate the -`otp_key_base` secret, which is used to encrypt the two-factor authentication -secrets stored in the database. If it differs between **primary** and **secondary** -nodes, users with two-factor authentication enabled won't be able to log in -after a failover. +### Message: `ERROR - Replication is not up-to-date` during `gitlab-ctl promote-to-primary-node` -If you still have access to the old **primary** node, you can follow the -instructions in the -[Upgrading to GitLab 10.5](../replication/version_specific_updates.md#updating-to-gitlab-105) -section to resolve the error. Otherwise, the secret is lost and you'll need to -[reset two-factor authentication for all users](../../../security/two_factor_authentication.md#disabling-2fa-for-everyone). +In GitLab 13.7 and earlier, if you have a data type with zero items to sync, +this command reports `ERROR - Replication is not up-to-date` even if +replication is actually up-to-date. If replication and verification output +shows that it is complete, you can add `--skip-preflight-checks` to make the command complete promotion. This bug was fixed in GitLab 13.8 and later. ## Expired artifacts @@ -717,7 +780,7 @@ node's URL matches its external URL. ## Fixing common errors -This section documents common errors reported in the Admin UI and how to fix them. +This section documents common errors reported in the Admin Area and how to fix them. ### Geo database configuration file is missing diff --git a/doc/administration/geo/replication/tuning.md b/doc/administration/geo/replication/tuning.md index 183180e5ade..a4aad3dec68 100644 --- a/doc/administration/geo/replication/tuning.md +++ b/doc/administration/geo/replication/tuning.md @@ -5,11 +5,11 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Tuning Geo **(PREMIUM ONLY)** +# Tuning Geo **(PREMIUM SELF)** ## Changing the sync/verification capacity values -In the Geo admin page at **Admin Area > Geo** (`/admin/geo/nodes`), +In **Admin Area > Geo** (`/admin/geo/nodes`), there are several variables that can be tuned to improve performance of Geo: - Repository sync capacity diff --git a/doc/administration/geo/replication/updating_the_geo_nodes.md b/doc/administration/geo/replication/updating_the_geo_nodes.md index ff3a1e8689b..0c68adf162d 100644 --- a/doc/administration/geo/replication/updating_the_geo_nodes.md +++ b/doc/administration/geo/replication/updating_the_geo_nodes.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Updating the Geo nodes **(PREMIUM ONLY)** +# Updating the Geo nodes **(PREMIUM SELF)** WARNING: Read these sections carefully before updating your Geo nodes. Not following diff --git a/doc/administration/geo/replication/using_a_geo_server.md b/doc/administration/geo/replication/using_a_geo_server.md index 743b1b384db..7578b6bf61a 100644 --- a/doc/administration/geo/replication/using_a_geo_server.md +++ b/doc/administration/geo/replication/using_a_geo_server.md @@ -7,7 +7,7 @@ type: howto <!-- Please update EE::GitLab::GeoGitAccess::GEO_SERVER_DOCS_URL if this file is moved) --> -# Using a Geo Server **(PREMIUM ONLY)** +# Using a Geo Server **(PREMIUM SELF)** After you set up the [database replication and configure the Geo nodes](../index.md#setup-instructions), use your closest GitLab node as you would a normal standalone GitLab instance. diff --git a/doc/administration/geo/replication/version_specific_updates.md b/doc/administration/geo/replication/version_specific_updates.md index 0d8eba555ab..be2ce0ac2c0 100644 --- a/doc/administration/geo/replication/version_specific_updates.md +++ b/doc/administration/geo/replication/version_specific_updates.md @@ -5,23 +5,37 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Version specific update instructions +# Version-specific update instructions -Check this document if it includes instructions for the version you are updating. -These steps go together with the [general steps](updating_the_geo_nodes.md#general-update-steps) +Review this page for update instructions for your version. These steps +accompany the [general steps](updating_the_geo_nodes.md#general-update-steps) for updating Geo nodes. +## Updating to GitLab 13.7 + +We've detected an issue with the `FetchRemove` call used by Geo secondaries. +This causes performance issues as we execute reference transaction hooks for +each updated reference. Delay any upgrade attempts until this is in the +[13.7.5 patch release.](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/3002). +More details are available [in this issue](https://gitlab.com/gitlab-org/git/-/issues/79). + ## Updating to GitLab 13.5 -In GitLab 13.5, there is a [regression that prevents viewing a list of container repositories and registries](https://gitlab.com/gitlab-org/gitlab/-/issues/285475) on Geo secondaries. This issue is fixed in GitLab 13.6.1 and later. +GitLab 13.5 has a [regression that prevents viewing a list of container repositories and registries](https://gitlab.com/gitlab-org/gitlab/-/issues/285475) +on Geo secondaries. This issue is fixed in GitLab 13.6.1 and later. ## Updating to GitLab 13.3 -In GitLab 13.3, Geo removed the PostgreSQL [Foreign Data Wrapper](https://www.postgresql.org/docs/11/postgres-fdw.html) dependency for the tracking database. +In GitLab 13.3, Geo removed the PostgreSQL [Foreign Data Wrapper](https://www.postgresql.org/docs/11/postgres-fdw.html) +dependency for the tracking database. -The FDW server, user, and the extension will be removed during the upgrade process on each **secondary** node. The GitLab settings related to the FDW in the `/etc/gitlab/gitlab.rb` have been deprecated and can be safely removed. +The FDW server, user, and the extension will be removed during the upgrade +process on each secondary node. The GitLab settings related to the FDW in the +`/etc/gitlab/gitlab.rb` have been deprecated and can be safely removed. -There are some scenarios like using an external PostgreSQL instance for the tracking database where the FDW settings must be removed manually. Enter the PostgreSQL console of that instance and remove them: +There are some scenarios like using an external PostgreSQL instance for the +tracking database where the FDW settings must be removed manually. Enter the +PostgreSQL console of that instance and remove them: ```shell DROP SERVER gitlab_secondary CASCADE; @@ -36,7 +50,10 @@ upgrade to GitLab 13.4 or later. ## Updating to GitLab 13.2 -In GitLab 13.2, promoting a secondary node to a primary while the secondary is paused fails. **Do not pause replication before promoting a secondary.** If the node is paused, please resume before promoting. To avoid this issue, upgrade to GitLab 13.4 or later. +In GitLab 13.2, promoting a secondary node to a primary while the secondary is +paused fails. Do not pause replication before promoting a secondary. If the +node is paused, be sure to resume before promoting. To avoid this issue, +upgrade to GitLab 13.4 or later. ## Updating to GitLab 13.0 @@ -59,12 +76,13 @@ GitLab 12.9.0 through GitLab 12.9.3 are affected by [a bug that stops repository verification](https://gitlab.com/gitlab-org/gitlab/-/issues/213523). The issue is fixed in GitLab 12.9.4. Upgrade to GitLab 12.9.4 or later. -By default, GitLab 12.9 will attempt to automatically update the embedded -PostgreSQL server to 10.12 from 9.6, which requires downtime on secondaries -while reinitializing streaming replication. For the recommended procedure, see the +By default, GitLab 12.9 attempts to update the embedded PostgreSQL server +version from 9.6 to 10.12, which requires downtime on secondaries while +reinitializing streaming replication. For the recommended procedure, see the [Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). -This can be temporarily disabled by running the following before updating: +You can temporarily disable this behavior by running the following before +updating: ```shell sudo touch /etc/gitlab/disable-postgresql-upgrade @@ -72,12 +90,13 @@ sudo touch /etc/gitlab/disable-postgresql-upgrade ## Updating to GitLab 12.8 -By default, GitLab 12.8 will attempt to automatically update the embedded -PostgreSQL server to 10.12 from 9.6, which requires downtime on secondaries -while reinitializing streaming replication. For the recommended procedure, see -the [Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). +By default, GitLab 12.8 attempts to update the embedded PostgreSQL server +version from 9.6 to 10.12, which requires downtime on secondaries while +reinitializing streaming replication. For the recommended procedure, see the +[Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). -This can be temporarily disabled by running the following before updating: +You can temporarily disable this behavior by running the following before +updating: ```shell sudo touch /etc/gitlab/disable-postgresql-upgrade @@ -87,18 +106,17 @@ sudo touch /etc/gitlab/disable-postgresql-upgrade WARNING: Only upgrade to GitLab 12.7.5 or later. Do not upgrade to versions 12.7.0 -through 12.7.4 because there is [an initialization order -bug](https://gitlab.com/gitlab-org/gitlab/-/issues/199672) that causes Geo -**secondaries** to set the incorrect database connection pool size. [The -fix](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/24021) was +through 12.7.4 because there is [an initialization order bug](https://gitlab.com/gitlab-org/gitlab/-/issues/199672) that causes Geo secondaries to set the incorrect database connection pool size. +[The fix](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/24021) was shipped in 12.7.5. -By default, GitLab 12.7 will attempt to automatically update the embedded -PostgreSQL server to 10.9 from 9.6, which requires downtime on secondaries -while reinitializing streaming replication. For the recommended procedure, see -the [Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). +By default, GitLab 12.7 attempts to update the embedded PostgreSQL server +version from 9.6 to 10.9, which requires downtime on secondaries while +reinitializing streaming replication. For the recommended procedure, see the +[Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). -This can be temporarily disabled by running the following before updating: +You can temporarily disable this behavior by running the following before +updating: ```shell sudo touch /etc/gitlab/disable-postgresql-upgrade @@ -106,12 +124,13 @@ sudo touch /etc/gitlab/disable-postgresql-upgrade ## Updating to GitLab 12.6 -By default, GitLab 12.6 will attempt to automatically update the embedded -PostgreSQL server to 10.9 from 9.6, which requires downtime on secondaries -while reinitializing streaming replication. For the recommended procedure, see -the [Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). +By default, GitLab 12.6 attempts to update the embedded PostgreSQL server +version from 9.6 to 10.9, which requires downtime on secondaries while +reinitializing streaming replication. For the recommended procedure, see the +[Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). -This can be temporarily disabled by running the following before updating: +You can temporarily disable this behavior by running the following before +updating: ```shell sudo touch /etc/gitlab/disable-postgresql-upgrade @@ -119,12 +138,13 @@ sudo touch /etc/gitlab/disable-postgresql-upgrade ## Updating to GitLab 12.5 -By default, GitLab 12.5 will attempt to automatically update the embedded -PostgreSQL server to 10.9 from 9.6, which requires downtime on secondaries -while reinitializing streaming replication. For the recommended procedure, see -the [Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). +By default, GitLab 12.5 attempts to update the embedded PostgreSQL server +version from 9.6 to 10.9, which requires downtime on secondaries while +reinitializing streaming replication. For the recommended procedure, see the +[Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). -This can be temporarily disabled by running the following before updating: +You can temporarily disable this behavior by running the following before +updating: ```shell sudo touch /etc/gitlab/disable-postgresql-upgrade @@ -132,12 +152,13 @@ sudo touch /etc/gitlab/disable-postgresql-upgrade ## Updating to GitLab 12.4 -By default, GitLab 12.4 will attempt to automatically update the embedded -PostgreSQL server to 10.9 from 9.6, which requires downtime on secondaries -while reinitializing streaming replication. For the recommended procedure, see -the [Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). +By default, GitLab 12.4 attempts to update the embedded PostgreSQL server +version from 9.6 to 10.9, which requires downtime on secondaries while +reinitializing streaming replication. For the recommended procedure, see the +[Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). -This can be temporarily disabled by running the following before updating: +You can temporarily disable this behavior by running the following before +updating: ```shell sudo touch /etc/gitlab/disable-postgresql-upgrade @@ -146,9 +167,9 @@ sudo touch /etc/gitlab/disable-postgresql-upgrade ## Updating to GitLab 12.3 WARNING: -If the existing PostgreSQL server version is 9.6.x, it is recommended to -upgrade to GitLab 12.4 or later. By default, GitLab 12.3 attempts to update the -embedded PostgreSQL server from 9.6 to 10.9. In certain circumstances, it will +If the existing PostgreSQL server version is 9.6.x, we recommend upgrading to +GitLab 12.4 or later. By default, GitLab 12.3 attempts to update the embedded +PostgreSQL server version from 9.6 to 10.9. In certain circumstances, it can fail. For more information, see the [Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). @@ -160,23 +181,23 @@ For the recommended procedure, see the ## Updating to GitLab 12.2 WARNING: -If the existing PostgreSQL server version is 9.6.x, it is recommended to -upgrade to GitLab 12.4 or later. By default, GitLab 12.2 attempts to update the -embedded PostgreSQL server from 9.6 to 10.9. In certain circumstances, it will +If the existing PostgreSQL server version is 9.6.x, we recommend upgrading to +GitLab 12.4 or later. By default, GitLab 12.2 attempts to update the embedded +PostgreSQL server version from 9.6 to 10.9. In certain circumstances, it can fail. For more information, see the [Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). -Additionally, if the PostgreSQL upgrade does not fail, a successful upgrade +Additionally, if the PostgreSQL upgrade doesn't fail, a successful upgrade requires downtime for secondaries while reinitializing streaming replication. For the recommended procedure, see the [Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). GitLab 12.2 includes the following minor PostgreSQL updates: -- To version `9.6.14` if you run PostgreSQL 9.6. -- To version `10.9` if you run PostgreSQL 10. +- To version `9.6.14`, if you run PostgreSQL 9.6. +- To version `10.9`, if you run PostgreSQL 10. -This update will occur even if major PostgreSQL updates are disabled. +This update occurs even if major PostgreSQL updates are disabled. Before [refreshing Foreign Data Wrapper during a Geo upgrade](https://docs.gitlab.com/omnibus/update/README.html#run-post-deployment-migrations-and-checks), restart the Geo tracking database: @@ -185,14 +206,15 @@ restart the Geo tracking database: sudo gitlab-ctl restart geo-postgresql ``` -The restart avoids a version mismatch when PostgreSQL tries to load the FDW extension. +The restart avoids a version mismatch when PostgreSQL tries to load the FDW +extension. ## Updating to GitLab 12.1 WARNING: -If the existing PostgreSQL server version is 9.6.x, it is recommended to -upgrade to GitLab 12.4 or later. By default, GitLab 12.1 attempts to update the -embedded PostgreSQL server from 9.6 to 10.9. In certain circumstances, it will +If the existing PostgreSQL server version is 9.6.x, we recommend upgrading to +GitLab 12.4 or later. By default, GitLab 12.1 attempts to update the embedded +PostgreSQL server version from 9.6 to 10.9. In certain circumstances, it can fail. For more information, see the [Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). @@ -206,419 +228,11 @@ For the recommended procedure, see the WARNING: This version is affected by a [bug that results in new LFS objects not being replicated to Geo secondary nodes](https://gitlab.com/gitlab-org/gitlab/-/issues/32696). -The issue is fixed in GitLab 12.1; be sure to upgrade to GitLab 12.1 or later. +The issue is fixed in GitLab 12.1. Be sure to upgrade to GitLab 12.1 or later. ## Updating to GitLab 11.11 WARNING: This version is affected by a [bug that results in new LFS objects not being replicated to Geo secondary nodes](https://gitlab.com/gitlab-org/gitlab/-/issues/32696). -The issue is fixed in GitLab 12.1; be sure to upgrade to GitLab 12.1 or later. - -## Updating to GitLab 10.8 - -Before 10.8, broadcast messages would not propagate without flushing -the cache on the **secondary** nodes. This has been fixed in 10.8, but -requires one last cache flush on each **secondary** node: - -```shell -sudo gitlab-rake cache:clear -``` - -## Updating to GitLab 10.6 - -In 10.4, we started to recommend that you define a password for database user (`gitlab`). - -We now require this change as we use this password to enable the Foreign Data Wrapper, as a way to optimize -the Geo Tracking Database. We are also improving security by disabling the use of **trust** -authentication method. - -1. **(primary)** Login to your **primary** node and run: - - ```shell - gitlab-ctl pg-password-md5 gitlab - # Enter password: <your_password_here> - # Confirm password: <your_password_here> - # fca0b89a972d69f00eb3ec98a5838484 - ``` - - Copy the generated hash and edit `/etc/gitlab/gitlab.rb`: - - ```ruby - # Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab` - postgresql['sql_user_password'] = '<md5_hash_of_your_password>' - - # Every node that runs Unicorn or Sidekiq needs to have the database - # password specified as below. - # This must be present in all application nodes. - gitlab_rails['db_password'] = '<your_password_here>' - ``` - - Still in the configuration file, locate and remove the `trust_auth_cidr_address`: - - ```ruby - postgresql['trust_auth_cidr_addresses'] = ['127.0.0.1/32','1.2.3.4/32'] # <- Remove this - ``` - -1. **(primary)** Reconfigure and restart: - - ```shell - sudo gitlab-ctl reconfigure - sudo gitlab-ctl restart - ``` - -1. **(secondary)** Login to all **secondary** nodes and edit `/etc/gitlab/gitlab.rb`: - - ```ruby - # Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab` - postgresql['sql_user_password'] = '<md5_hash_of_your_password>' - - # Every node that runs Unicorn or Sidekiq needs to have the database - # password specified as below. - # This must be present in all application nodes. - gitlab_rails['db_password'] = '<your_password_here>' - - # Enable Foreign Data Wrapper - geo_secondary['db_fdw'] = true - - # Secondary address in CIDR format, for example '5.6.7.8/32' - postgresql['md5_auth_cidr_addresses'] = ['<secondary_node_ip>/32'] - ``` - - Still in the configuration file, locate and remove the `trust_auth_cidr_address`: - - ```ruby - postgresql['trust_auth_cidr_addresses'] = ['127.0.0.1/32','5.6.7.8/32'] # <- Remove this - ``` - -1. **(secondary)** Reconfigure and restart: - - ```shell - sudo gitlab-ctl reconfigure - sudo gitlab-ctl restart - ``` - -## Updating to GitLab 10.5 - -For Geo Disaster Recovery to work with minimum downtime, your **secondary** node -should use the same set of secrets as the **primary** node. However, setup instructions -prior to the 10.5 release only synchronized the `db_key_base` secret. - -To rectify this error on existing installations, you should **overwrite** the -contents of `/etc/gitlab/gitlab-secrets.json` on each **secondary** node with the -contents of `/etc/gitlab/gitlab-secrets.json` on the **primary** node, then run the -following command on each **secondary** node: - -```shell -sudo gitlab-ctl reconfigure -``` - -If you do not perform this step, you may find that two-factor authentication -[is broken following DR](troubleshooting.md#two-factor-authentication-is-broken-after-a-failover). - -To prevent SSH requests to the newly promoted **primary** node from failing -due to SSH host key mismatch when updating the **primary** node domain's DNS record -you should perform the step to [Manually replicate **primary** SSH host keys](configuration.md#step-2-manually-replicate-the-primary-nodes-ssh-host-keys) in each -**secondary** node. - -## Updating to GitLab 10.3 - -### Support for SSH repository synchronization removed - -In GitLab 10.2, synchronizing secondaries over SSH was deprecated. In 10.3, -support is removed entirely. All installations will switch to the HTTP/HTTPS -cloning method instead. Before updating, ensure that all your Geo nodes are -configured to use this method and that it works for your installation. In -particular, ensure that [Git access over HTTP/HTTPS is enabled](configuration.md#step-5-enable-git-access-over-httphttps). - -Synchronizing repositories over the public Internet using HTTP is insecure, so -you should ensure that you have HTTPS configured before updating. Note that -file synchronization is **also** insecure in these cases! - -## Updating to GitLab 10.2 - -### Secure PostgreSQL replication - -Support for TLS-secured PostgreSQL replication has been added. If you are -currently using PostgreSQL replication across the open internet without an -external means of securing the connection (e.g., a site-to-site VPN), then you -should immediately reconfigure your **primary** and **secondary** PostgreSQL instances -according to the [updated instructions](../setup/database.md). - -If you *are* securing the connections externally and wish to continue doing so, -ensure you include the new option `--sslmode=prefer` in future invocations of -`gitlab-ctl replicate-geo-database`. - -### HTTPS repository sync - -Support for replicating repositories and wikis over HTTP/HTTPS has been added. -Replicating over SSH has been deprecated, and support for this option will be -removed in a future release. - -To switch to HTTP/HTTPS replication, log into the **primary** node as an admin and visit -**Admin Area > Geo** (`/admin/geo/nodes`). For each **secondary** node listed, -press the "Edit" button, change the "Repository cloning" setting from -"SSH (deprecated)" to "HTTP/HTTPS", and press "Save changes". This should take -effect immediately. - -Any new secondaries should be created using HTTP/HTTPS replication - this is the -default setting. - -After you've verified that HTTP/HTTPS replication is working, you should remove -the now-unused SSH keys from your secondaries, as they may cause problems if the -**secondary** node if ever promoted to a **primary** node: - -1. **(secondary)** Login to **all** your **secondary** nodes and run: - - ```ruby - sudo -u git -H rm ~git/.ssh/id_rsa ~git/.ssh/id_rsa.pub - ``` - -### Hashed Storage - -WARNING: -Hashed storage is in **Alpha**. It is considered experimental and not -production-ready. See [Hashed Storage](../../repository_storage_types.md) for more detail. - -If you previously enabled Hashed Storage and migrated all your existing -projects to Hashed Storage, disabling hashed storage will not migrate projects -to their previous project based storage path. As such, once enabled and -migrated we recommend leaving Hashed Storage enabled. - -## Updating to GitLab 10.1 - -WARNING: -Hashed storage is in **Alpha**. It is considered experimental and not -production-ready. See [Hashed Storage](../../repository_storage_types.md) for more detail. - -[Hashed storage](../../repository_storage_types.md) was introduced in -GitLab 10.0, and a [migration path](../../raketasks/storage.md) for -existing repositories was added in GitLab 10.1. - -## Updating to GitLab 10.0 - -In GitLab 10.0 and later, we require all **Geo** systems to [use SSH key lookups via -the database](../../operations/fast_ssh_key_lookup.md) to avoid having to maintain consistency of the -`authorized_keys` file for SSH access. Failing to do this will prevent users -from being able to clone via SSH. - -Note that in older versions of Geo, attachments downloaded on the **secondary** -nodes would be saved to the wrong directory. We recommend that you do the -following to clean this up. - -On the **secondary** Geo nodes, run as root: - -```shell -mv /var/opt/gitlab/gitlab-rails/working /var/opt/gitlab/gitlab-rails/working.old -mkdir /var/opt/gitlab/gitlab-rails/working -chmod 700 /var/opt/gitlab/gitlab-rails/working -chown git:git /var/opt/gitlab/gitlab-rails/working -``` - -You may delete `/var/opt/gitlab/gitlab-rails/working.old` any time. - -Once this is done, we advise restarting GitLab on the **secondary** nodes for the -new working directory to be used: - -```shell -sudo gitlab-ctl restart -``` - -## Updating from GitLab 9.3 or older - -If you started running Geo on GitLab 9.3 or older, we recommend that you -resync your **secondary** PostgreSQL databases to use replication slots. If you -started using Geo with GitLab 9.4 or 10.x, no further action should be -required because replication slots are used by default. However, if you -started with GitLab 9.3 and updated later, you should still follow the -instructions below. - -When in doubt, it doesn't hurt to do a resync. The easiest way to do this in -Omnibus is the following: - -1. Make sure you have Omnibus GitLab on the **primary** server. -1. Run `gitlab-ctl reconfigure` and `gitlab-ctl restart postgresql`. This will enable replication slots on the **primary** database. -1. Check the steps about defining `postgresql['sql_user_password']`, `gitlab_rails['db_password']`. -1. Make sure `postgresql['max_replication_slots']` matches the number of **secondary** Geo nodes locations. -1. Install GitLab on the **secondary** server. -1. Re-run the [database replication process](../setup/database.md#step-3-initiate-the-replication-process). - -## Updating to GitLab 9.0 - -> **IMPORTANT**: -With GitLab 9.0, the PostgreSQL version is updated to 9.6 and manual steps are -required to update the **secondary** nodes and keep the Streaming Replication -working. Downtime is required, so plan ahead. - -The following steps apply only if you update from a 8.17 GitLab version to -9.0+. For previous versions, update to GitLab 8.17 first before attempting to -update to 9.0+. - ---- - -Make sure to follow the steps in the exact order as they appear below and pay -extra attention in what node (either **primary** or **secondary**) you execute them! Each step -is prepended with the relevant node for better clarity: - -1. **(secondary)** Log in to **all** your **secondary** nodes and stop all services: - - ```ruby - sudo gitlab-ctl stop - ``` - -1. **(secondary)** Make a backup of the `recovery.conf` file on **all** - **secondary** nodes to preserve PostgreSQL's credentials: - - ```shell - sudo cp /var/opt/gitlab/postgresql/data/recovery.conf /var/opt/gitlab/ - ``` - -1. **(primary)** Update the **primary** node to GitLab 9.0 following the - [regular update docs](../../../update/README.md). At the end of the - update, the **primary** node will be running with PostgreSQL 9.6. - -1. **(primary)** To prevent a de-synchronization of the repository replication, - stop all services except `postgresql` as we will use it to re-initialize the - **secondary** node's database: - - ```shell - sudo gitlab-ctl stop - sudo gitlab-ctl start postgresql - ``` - -1. **(secondary)** Run the following steps on each of the **secondary** nodes: - - 1. **(secondary)** Stop all services: - - ```shell - sudo gitlab-ctl stop - ``` - - 1. **(secondary)** Prevent running database migrations: - - ```shell - sudo touch /etc/gitlab/skip-auto-migrations - ``` - - 1. **(secondary)** Move the old database to another directory: - - ```shell - sudo mv /var/opt/gitlab/postgresql{,.bak} - ``` - - 1. **(secondary)** Update to GitLab 9.0 following the [regular update docs](../../../update/README.md). - At the end of the update, the node will be running with PostgreSQL 9.6. - - 1. **(secondary)** Make sure all services are up: - - ```shell - sudo gitlab-ctl start - ``` - - 1. **(secondary)** Reconfigure GitLab: - - ```shell - sudo gitlab-ctl reconfigure - ``` - - 1. **(secondary)** Run the PostgreSQL upgrade command: - - ```shell - sudo gitlab-ctl pg-upgrade - ``` - - 1. **(secondary)** See the stored credentials for the database that you will - need to re-initialize the replication: - - ```shell - sudo grep -s primary_conninfo /var/opt/gitlab/recovery.conf - ``` - - 1. **(secondary)** Save the snippet below in a file, let's say `/tmp/replica.sh`. Modify the - embedded paths if necessary: - - ```shell - #!/bin/bash - - PORT="5432" - USER="gitlab_replicator" - echo --------------------------------------------------------------- - echo WARNING: Make sure this script is run from the secondary server - echo --------------------------------------------------------------- - echo - echo Enter the IP or FQDN of the primary PostgreSQL server - read HOST - echo Enter the password for $USER@$HOST - read -s PASSWORD - echo Enter the required sslmode - read SSLMODE - - echo Stopping PostgreSQL and all GitLab services - sudo service gitlab stop - sudo service postgresql stop - - echo Backing up postgresql.conf - sudo -u postgres mv /var/opt/gitlab/postgresql/data/postgresql.conf /var/opt/gitlab/postgresql/ - - echo Cleaning up old cluster directory - sudo -u postgres rm -rf /var/opt/gitlab/postgresql/data - - echo Starting base backup as the replicator user - echo Enter the password for $USER@$HOST - sudo -u postgres /opt/gitlab/embedded/bin/pg_basebackup -h $HOST -D /var/opt/gitlab/postgresql/data -U gitlab_replicator -v -x -P - - echo Writing recovery.conf file - sudo -u postgres bash -c "cat > /var/opt/gitlab/postgresql/data/recovery.conf <<- _EOF1_ - standby_mode = 'on' - primary_conninfo = 'host=$HOST port=$PORT user=$USER password=$PASSWORD sslmode=$SSLMODE' - _EOF1_ - " - - echo Restoring postgresql.conf - sudo -u postgres mv /var/opt/gitlab/postgresql/postgresql.conf /var/opt/gitlab/postgresql/data/ - - echo Starting PostgreSQL - sudo service postgresql start - ``` - - 1. **(secondary)** Run the recovery script using the credentials from the - previous step: - - ```shell - sudo bash /tmp/replica.sh - ``` - - 1. **(secondary)** Reconfigure GitLab: - - ```shell - sudo gitlab-ctl reconfigure - ``` - - 1. **(secondary)** Start all services: - - ```shell - sudo gitlab-ctl start - ``` - - 1. **(secondary)** Repeat the steps for the remaining **secondary** nodes. - -1. **(primary)** After all **secondary** nodes are updated, start all services in - **primary** node: - - ```shell - sudo gitlab-ctl start - ``` - -### Update tracking database on **secondary** node - -After updating a **secondary** node, you might need to run migrations on the -tracking database. The tracking database was added in GitLab 9.1, and is -required in GitLab 10.0 and later. - -1. Run database migrations on tracking database: - - ```shell - sudo gitlab-rake geo:db:migrate - ``` - -1. Repeat this step for each **secondary** node. +The issue is fixed in GitLab 12.1. Be sure to upgrade to GitLab 12.1 or later. diff --git a/doc/administration/geo/setup/database.md b/doc/administration/geo/setup/database.md index 95f67b20ab5..1272e7d1419 100644 --- a/doc/administration/geo/setup/database.md +++ b/doc/administration/geo/setup/database.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Geo database replication **(PREMIUM ONLY)** +# Geo database replication **(PREMIUM SELF)** NOTE: If your GitLab installation uses external (not managed by Omnibus) PostgreSQL @@ -472,33 +472,31 @@ information, see [High Availability with Omnibus GitLab](../../postgresql/replic ## Patroni support Support for Patroni is intended to replace `repmgr` as a -[highly availabile PostgreSQL solution](../../postgresql/replication_and_failover.md) +[highly available PostgreSQL solution](../../postgresql/replication_and_failover.md) on the primary node, but it can also be used for PostgreSQL HA on a secondary -node. +site. Starting with GitLab 13.5, Patroni is available for _experimental_ use with Geo -primary and secondary nodes. Due to its experimental nature, Patroni support is +primary and secondary sites. Due to its experimental nature, Patroni support is subject to change without notice. This experimental implementation has the following limitations: -- Whenever a new Leader is elected, the PgBouncer instance must be reconfigured - to point to the new Leader. -- Whenever a new Leader is elected on the primary node, the Standby Leader on - the secondary needs to be reconfigured to point to the new Leader. - Whenever `gitlab-ctl reconfigure` runs on a Patroni Leader instance, there's a chance the node will be demoted due to the required short-time restart. To avoid this, you can pause auto-failover by running `gitlab-ctl patroni pause`. - After a reconfigure, it unpauses on its own. + After a reconfigure, it resumes on its own. -For instructions about how to set up Patroni on the primary node, see the +For instructions about how to set up Patroni on the primary site, see the [PostgreSQL replication and failover with Omnibus GitLab](../../postgresql/replication_and_failover.md#patroni) page. -If you are currently using `repmgr` on your Geo primary, see [these instructions](#migrating-from-repmgr-to-patroni) for migrating from `repmgr` to Patroni. +If you are currently using `repmgr` on your Geo primary site, see [these instructions](#migrating-from-repmgr-to-patroni) for migrating from `repmgr` to Patroni. -A production-ready and secure setup requires at least three Patroni instances on -the primary site, and a similar configuration on the secondary sites. Be sure to -use password credentials and other database best practices. +A production-ready and secure setup requires at least three Consul nodes, three +Patroni nodes, one internal load-balancing node on the primary site, and a similar +configuration for the secondary site. The internal load balancer provides a single +endpoint for connecting to the Patroni cluster's leader whenever a new leader is +elected. Be sure to use [password credentials](../..//postgresql/replication_and_failover.md#database-authorization-for-patroni) and other database best practices. Similar to `repmgr`, using Patroni on a secondary node is optional. @@ -555,7 +553,51 @@ Leader instance**: gitlab-ctl reconfigure ``` -### Step 2. Configure a Standby cluster on the secondary site +### Step 2. Configure the internal load balancer on the primary site + +To avoid reconfiguring the Standby Leader on the secondary site whenever a new +Leader is elected on the primary site, we'll need to set up a TCP internal load +balancer which will give a single endpoint for connecting to the Patroni +cluster's Leader. + +The Omnibus GitLab packages do not include a Load Balancer. Here's how you +could do it with [HAProxy](https://www.haproxy.org/). + +The following IPs and names will be used as an example: + +- `10.6.0.21`: Patroni 1 (`patroni1.internal`) +- `10.6.0.21`: Patroni 2 (`patroni2.internal`) +- `10.6.0.22`: Patroni 3 (`patroni3.internal`) + +```plaintext +global + log /dev/log local0 + log localhost local1 notice + log stdout format raw local0 + +defaults + log global + default-server inter 3s fall 3 rise 2 on-marked-down shutdown-sessions + +frontend internal-postgresql-tcp-in + bind *:5000 + mode tcp + option tcplog + + default_backend postgresql + +backend postgresql + option httpchk + http-check expect status 200 + + server patroni1.internal 10.6.0.21:5432 maxconn 100 check port 8008 + server patroni2.internal 10.6.0.22:5432 maxconn 100 check port 8008 + server patroni3.internal 10.6.0.23.195:5432 maxconn 100 check port 8008 +``` + +Refer to your preferred Load Balancer's documentation for further guidance. + +### Step 3. Configure a Standby cluster on the secondary site NOTE: If you are converting a secondary site to a Patroni Cluster, you must start @@ -589,8 +631,8 @@ For each Patroni instance on the secondary site: patroni['enable'] = false patroni['standby_cluster']['enable'] = true - patroni['standby_cluster']['host'] = 'PATRONI_PRIMARY_LEADER_IP' # This needs to be changed anytime the primary Leader changes - patroni['standby_cluster']['port'] = 5432 + patroni['standby_cluster']['host'] = 'INTERNAL_LOAD_BALANCER_PRIMARY_IP' + patroni['standby_cluster']['port'] = INTERNAL_LOAD_BALANCER_PRIMARY_PORT patroni['standby_cluster']['primary_slot_name'] = 'geo_secondary' # Or the unique replication slot name you setup before patroni['replication_password'] = 'PLAIN_TEXT_POSTGRESQL_REPLICATION_PASSWORD' patroni['use_pg_rewind'] = true @@ -642,10 +684,11 @@ With Patroni it's now possible to support that. In order to migrate the existing 1. Make sure you have a Consul cluster setup on the secondary (similar to how you set it up on the primary). 1. [Configure a permanent replication slot](#step-1-configure-patroni-permanent-replication-slot-on-the-primary-site). -1. [Configure a Standby Cluster](#step-2-configure-a-standby-cluster-on-the-secondary-site) +1. [Configure the internal load balancer](#step-2-configure-the-internal-load-balancer-on-the-primary-site). +1. [Configure a Standby Cluster](#step-3-configure-a-standby-cluster-on-the-secondary-site) on that single node machine. - -You will end up with a "Standby Cluster" with a single node. That allows you to later on add additional patroni nodes + +You will end up with a "Standby Cluster" with a single node. That allows you to later on add additional Patroni nodes by following the same instructions above. ## Troubleshooting diff --git a/doc/administration/geo/setup/external_database.md b/doc/administration/geo/setup/external_database.md index 1fb41fbb53a..8e7d8049467 100644 --- a/doc/administration/geo/setup/external_database.md +++ b/doc/administration/geo/setup/external_database.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Geo with external PostgreSQL instances **(PREMIUM ONLY)** +# Geo with external PostgreSQL instances **(PREMIUM SELF)** This document is relevant if you are using a PostgreSQL instance that is *not managed by Omnibus*. This includes cloud-managed instances like AWS RDS, or |