diff options
Diffstat (limited to 'doc/administration/geo')
13 files changed, 222 insertions, 113 deletions
diff --git a/doc/administration/geo/disaster_recovery/bring_primary_back.md b/doc/administration/geo/disaster_recovery/bring_primary_back.md index 83081e2cef6..b73d3ba52a2 100644 --- a/doc/administration/geo/disaster_recovery/bring_primary_back.md +++ b/doc/administration/geo/disaster_recovery/bring_primary_back.md @@ -45,9 +45,12 @@ To bring the former **primary** node up to date: all the writes to this node](planned_failover.md#prevent-updates-to-the-primary-node) during this procedure. -1. [Setup database replication](../setup/database.md). Note that in this - case, **primary** node refers to the current **primary** node, and **secondary** node refers to the - former **primary** node. +1. [Set up database replication](../setup/database.md). In this case, the **secondary** node + refers to the former **primary** node. + 1. If [PgBouncer](../../postgresql/pgbouncer.md) was enabled on the **current secondary** node + (when it was a primary node) disable it by editing `/etc/gitlab/gitlab.rb` + and running `sudo gitlab-ctl reconfigure`. + 1. You can then set up database replication on the **secondary** node. If you have lost your original **primary** node, follow the [setup instructions](../setup/index.md) to set up a new **secondary** node. diff --git a/doc/administration/geo/disaster_recovery/index.md b/doc/administration/geo/disaster_recovery/index.md index 5876a8c36e6..1e9b430834c 100644 --- a/doc/administration/geo/disaster_recovery/index.md +++ b/doc/administration/geo/disaster_recovery/index.md @@ -133,6 +133,12 @@ Note the following when promoting a secondary: ``` 1. Promote the **secondary** node to the **primary** node. + DANGER: **Warning:** + In GitLab 13.2 and 13.3, promoting a secondary node to a primary while the + secondary is paused fails. Do not pause replication before promoting a + secondary. If the node is paused, be sure to resume before promoting. This + issue has been fixed in GitLab 13.4 and later. + CAUTION: **Caution:** If the secondary node [has been paused](../../geo/index.md#pausing-and-resuming-replication), this performs a point-in-time recovery to the last known state. @@ -167,8 +173,14 @@ conjunction with multiple servers, as it can only perform changes on a **secondary** with only a single machine. Instead, you must do this manually. +DANGER: **Warning:** +In GitLab 13.2 and 13.3, promoting a secondary node to a primary while the +secondary is paused fails. Do not pause replication before promoting a +secondary. If the node is paused, be sure to resume before promoting. This +issue has been fixed in GitLab 13.4 and later. + CAUTION: **Caution:** -If the secondary node [has been paused](../../geo/index.md#pausing-and-resuming-replication), this performs + If the secondary node [has been paused](../../geo/index.md#pausing-and-resuming-replication), this performs a point-in-time recovery to the last known state. Data that was created on the primary while the secondary was paused will be lost. diff --git a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md index c89b7929b13..3864445bbf4 100644 --- a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md +++ b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md @@ -227,6 +227,12 @@ conjunction with multiple servers, as it can only perform changes on a **secondary** with only a single machine. Instead, you must do this manually. +DANGER: **Warning:** +In GitLab 13.2 and 13.3, promoting a secondary node to a primary while the +secondary is paused fails. Do not pause replication before promoting a +secondary. If the node is paused, be sure to resume before promoting. This +issue has been fixed in GitLab 13.4 and later. + CAUTION: **Caution:** If the secondary node [has been paused](../../../geo/index.md#pausing-and-resuming-replication), this performs a point-in-time recovery to the last known state. diff --git a/doc/administration/geo/index.md b/doc/administration/geo/index.md index 0bfdf1d2d65..02b907ae237 100644 --- a/doc/administration/geo/index.md +++ b/doc/administration/geo/index.md @@ -49,7 +49,7 @@ Geo provides: - Read-only **secondary** nodes: Maintain one **primary** GitLab node while still enabling read-only **secondary** nodes for each of your distributed teams. - Authentication system hooks: **Secondary** nodes receives all authentication data (like user accounts and logins) from the **primary** instance. -- An intuitive UI: **Secondary** nodes utilize the same web interface your team has grown accustomed to. In addition, there are visual notifications that block write operations and make it clear that a user is on a **secondary** node. +- An intuitive UI: **Secondary** nodes use the same web interface your team has grown accustomed to. In addition, there are visual notifications that block write operations and make it clear that a user is on a **secondary** node. ## How it works @@ -116,6 +116,7 @@ The following are required to run Geo: - [Ubuntu](https://ubuntu.com) 16.04+ - PostgreSQL 11+ with [Streaming Replication](https://wiki.postgresql.org/wiki/Streaming_Replication) - Git 2.9+ +- Git-lfs 2.4.2+ on the user side when using LFS - All nodes must run the same GitLab version. Additionally, check GitLab's [minimum requirements](../../install/requirements.md), @@ -195,6 +196,12 @@ For information on how to update your Geo nodes to the latest GitLab version, se > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/35913) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.2. +DANGER: **Warning:** +In GitLab 13.2 and 13.3, promoting a secondary node to a primary while the +secondary is paused fails. Do not pause replication before promoting a +secondary. If the node is paused, be sure to resume before promoting. This +issue has been fixed in GitLab 13.4 and later. + CAUTION: **Caution:** Pausing and resuming of replication is currently only supported for Geo installations using an Omnibus GitLab-managed database. External databases are currently not supported. diff --git a/doc/administration/geo/replication/configuration.md b/doc/administration/geo/replication/configuration.md index 8c818bcfbe2..43a422c4bc3 100644 --- a/doc/administration/geo/replication/configuration.md +++ b/doc/administration/geo/replication/configuration.md @@ -229,19 +229,10 @@ replicating missing data from the **primary** node in a process known as **backf Meanwhile, the **primary** node will start to notify each **secondary** node of any changes, so that the **secondary** node can act on those notifications immediately. -Make sure the **secondary** node is running and accessible. -You can login to the **secondary** node with the same credentials as used for the **primary** node. +Be sure the _secondary_ node is running and accessible. You can sign in to the +_secondary_ node with the same credentials as were used with the _primary_ node. -### Step 4. Enabling Hashed Storage - -Using Hashed Storage significantly improves Geo replication. Project and group -renames no longer require synchronization between nodes. - -1. Visit the **primary** node's **Admin Area > Settings > Repository** - (`/admin/application_settings/repository`) in your browser. -1. In the **Repository storage** section, check **Use hashed storage paths for newly created and renamed projects**. - -### Step 5. (Optional) Configuring the **secondary** node to trust the **primary** node +### Step 4. (Optional) Configuring the **secondary** node to trust the **primary** node You can safely skip this step if your **primary** node uses a CA-issued HTTPS certificate. @@ -251,21 +242,23 @@ certificate from the **primary** node and follow [these instructions](https://docs.gitlab.com/omnibus/settings/ssl.html) on the **secondary** node. -### Step 6. Enable Git access over HTTP/HTTPS +### Step 5. Enable Git access over HTTP/HTTPS Geo synchronizes repositories over HTTP/HTTPS, and therefore requires this clone -method to be enabled. Navigate to **Admin Area > Settings** -(`/admin/application_settings/general`) on the **primary** node, and set -`Enabled Git access protocols` to `Both SSH and HTTP(S)` or `Only HTTP(S)`. +method to be enabled. This is enabled by default, but if converting an existing node to Geo it should be checked: + +1. Navigate to **Admin Area > Settings** (`/admin/application_settings/general`) on the **primary** node. +1. Expand "Visibility and access controls". +1. Ensure "Enabled Git access protocols" is set to either "Both SSH and HTTP(S)" or "Only HTTP(S)". -### Step 7. Verify proper functioning of the **secondary** node +### Step 6. Verify proper functioning of the **secondary** node Your **secondary** node is now configured! -You can login to the **secondary** node with the same credentials you used for the -**primary** node. Visit the **secondary** node's **Admin Area > Geo** -(`/admin/geo/nodes`) in your browser to check if it's correctly identified as a -**secondary** Geo node and if Geo is enabled. +You can sign in to the _secondary_ node with the same credentials you used with +the _primary_ node. Visit the _secondary_ node's **Admin Area > Geo** +(`/admin/geo/nodes`) in your browser to determine if it's correctly identified +as a _secondary_ Geo node, and if Geo is enabled. The initial replication, or 'backfill', will probably still be in progress. You can monitor the synchronization process on each Geo node from the **primary** diff --git a/doc/administration/geo/replication/datatypes.md b/doc/administration/geo/replication/datatypes.md index d1fa2d503be..9e896f2c07c 100644 --- a/doc/administration/geo/replication/datatypes.md +++ b/doc/administration/geo/replication/datatypes.md @@ -127,8 +127,11 @@ and verification status on a **secondary** node. You can keep track of the progress to implement the missing items in these epics/issues: -- [Unreplicated Data Types](https://gitlab.com/groups/gitlab-org/-/epics/893) -- [Verify all replicated data](https://gitlab.com/groups/gitlab-org/-/epics/1430) +- [Geo: Build a scalable, self-service Geo replication and verification framework](https://gitlab.com/groups/gitlab-org/-/epics/2161) +- [Geo: Improve the self-service Geo replication framework](https://gitlab.com/groups/gitlab-org/-/epics/3761) +- [Geo: Move existing blobs to framework](https://gitlab.com/groups/gitlab-org/-/epics/3588) +- [Geo: Add unreplicated data types](https://gitlab.com/groups/gitlab-org/-/epics/893) +- [Geo: Support GitLab Pages](https://gitlab.com/groups/gitlab-org/-/epics/589) ### Replicated data types behind a feature flag @@ -157,18 +160,19 @@ To enable, such as for package file replication: Feature.enable(:geo_package_file_replication) ``` -DANGER: **Danger:** +DANGER: **Warning:** Features not on this list, or with **No** in the **Replicated** column, are not replicated on the **secondary** node. Failing over without manually replicating data from those features will cause the data to be **lost**. If you wish to use those features on a **secondary** node, or to execute a failover successfully, you must replicate their data using some other means. -| Feature | Replicated (added in GitLab version) | Verified (added in GitLab version) | Object Storage replication (please see [Geo with Object Storage](object_storage.md)) | Notes | +| Feature | Replicated (added in GitLab version) | Verified (added in GitLab version) | Object Storage replication (see [Geo with Object Storage](object_storage.md)) | Notes | |:---------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------|:----------------------------------------------------------|:-------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | [Application data in PostgreSQL](../../postgresql/index.md) | **Yes** (10.2) | **Yes** (10.2) | No | | | [Project repository](../../..//user/project/repository/) | **Yes** (10.2) | **Yes** (10.7) | No | | -| [Project wiki repository](../../../user/project/wiki/) | **Yes** (10.2) | **Yes** (10.7) | No | | +| [Project wiki repository](../../../user/project/wiki/) | **Yes** (10.2) | **Yes** (10.7) | No | +| [Group wiki repository](../../../user/group/index.md#group-wikis) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/208147) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/208147) | No | | | [Uploads](../../uploads.md) | **Yes** (10.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | No | Verified only on transfer or manually using [Integrity Check Rake Task](../../raketasks/check.md) on both nodes and comparing the output between them. | | [LFS objects](../../lfs/index.md) | **Yes** (10.2) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8922) | Via Object Storage provider if supported. Native Geo support (Beta). | Verified only on transfer or manually using [Integrity Check Rake Task](../../raketasks/check.md) on both nodes and comparing the output between them. GitLab versions 11.11.x and 12.0.x are affected by [a bug that prevents any new LFS objects from replicating](https://gitlab.com/gitlab-org/gitlab/-/issues/32696). | | [Personal snippets](../../../user/snippets.md#personal-snippets) | **Yes** (10.2) | **Yes** (10.2) | No | | @@ -176,7 +180,7 @@ successfully, you must replicate their data using some other means. | [CI job artifacts (other than Job Logs)](../../../ci/pipelines/job_artifacts.md) | **Yes** (10.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8923) | Via Object Storage provider if supported. Native Geo support (Beta) . | Verified only manually using [Integrity Check Rake Task](../../raketasks/check.md) on both nodes and comparing the output between them | | [Job logs](../../job_logs.md) | **Yes** (10.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8923) | Via Object Storage provider if supported. Native Geo support (Beta). | Verified only on transfer or manually using [Integrity Check Rake Task](../../raketasks/check.md) on both nodes and comparing the output between them | | [Object pools for forked project deduplication](../../../development/git_object_deduplication.md) | **Yes** | No | No | | -| [Container Registry](../../packages/container_registry.md) | **Yes** (12.3) | No | No | Disabled by default. See [instructions](./docker_registry.md) to enable. | +| [Container Registry](../../packages/container_registry.md) | **Yes** (12.3) | No | No | Disabled by default. See [instructions](docker_registry.md) to enable. | | [Content in object storage (beta)](object_storage.md) | **Yes** (12.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/13845) | No | | | [Project designs repository](../../../user/project/issues/design_management.md) | **Yes** (12.7) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/32467) | Via Object Storage provider if supported. Native Geo support (Beta). | | | [NPM Registry](../../../user/packages/npm_registry/index.md) | **Yes** (13.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | @@ -187,7 +191,7 @@ successfully, you must replicate their data using some other means. | [Composer Repository](../../../user/packages/composer_repository/index.md) | **Yes** (13.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | | [Generic packages](../../../user/packages/generic_packages/index.md) | **Yes** (13.5) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | | [Versioned Terraform State](../../terraform_state.md) | **Yes** (13.5) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_terraform_state_version_replication`, enabled by default | -| [External merge request diffs](../../merge_request_diffs.md) | **Yes** (13.5) | No | Behind feature flag `geo_merge_request_diff_replication`, enabled by default | | +| [External merge request diffs](../../merge_request_diffs.md) | **Yes** (13.5) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_merge_request_diff_replication`, enabled by default | | [Versioned snippets](../../../user/snippets.md#versioned-snippets) | [No](https://gitlab.com/groups/gitlab-org/-/epics/2809) | [No](https://gitlab.com/groups/gitlab-org/-/epics/2810) | No | | | [Server-side Git hooks](../../server_hooks.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1867) | No | No | | | [Elasticsearch integration](../../../integration/elasticsearch.md) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/1186) | No | No | | diff --git a/doc/administration/geo/replication/disable_geo.md b/doc/administration/geo/replication/disable_geo.md index 14a11d9c1e3..dabf4499f9d 100644 --- a/doc/administration/geo/replication/disable_geo.md +++ b/doc/administration/geo/replication/disable_geo.md @@ -25,7 +25,7 @@ To disable Geo, follow these steps: ## Remove all secondary Geo nodes To disable Geo, you need to first remove all your secondary Geo nodes, which means replication will not happen -anymore on these nodes. You can follow our docs to [remove your secondary Geo nodes](./remove_geo_node.md). +anymore on these nodes. You can follow our docs to [remove your secondary Geo nodes](remove_geo_node.md). If the current node that you want to keep using is a secondary node, you need to first promote it to primary. You can use our steps on [how to promote a secondary node](../disaster_recovery/#step-3-promoting-a-secondary-node) diff --git a/doc/administration/geo/replication/location_aware_git_url.md b/doc/administration/geo/replication/location_aware_git_url.md index ddcdea736e7..82fda8b0fc2 100644 --- a/doc/administration/geo/replication/location_aware_git_url.md +++ b/doc/administration/geo/replication/location_aware_git_url.md @@ -43,8 +43,8 @@ In any case, you require: - A working GitLab **secondary** node. - A Route53 Hosted Zone managing your domain. -If you have not yet setup a Geo **primary** node and **secondary** node, please consult -[the Geo setup instructions](../index.md#setup-instructions). +If you haven't yet set up a Geo _primary_ node and _secondary_ node, see the +[Geo setup instructions](../index.md#setup-instructions). ## Create a traffic policy diff --git a/doc/administration/geo/replication/object_storage.md b/doc/administration/geo/replication/object_storage.md index 642d31f6298..3f5ba1939b8 100644 --- a/doc/administration/geo/replication/object_storage.md +++ b/doc/administration/geo/replication/object_storage.md @@ -72,7 +72,7 @@ If you are using Google Cloud Storage, consider using Or you can use the [Storage Transfer Service](https://cloud.google.com/storage-transfer/docs/), although this only supports daily synchronization. -For manual synchronization, or scheduled by `cron`, please have a look at: +For manual synchronization, or scheduled by `cron`, see: - [`s3cmd sync`](https://s3tools.org/s3cmd-sync) - [`gsutil rsync`](https://cloud.google.com/storage/docs/gsutil/commands/rsync) diff --git a/doc/administration/geo/replication/troubleshooting.md b/doc/administration/geo/replication/troubleshooting.md index b62c5c6f460..15e3d3ff0a8 100644 --- a/doc/administration/geo/replication/troubleshooting.md +++ b/doc/administration/geo/replication/troubleshooting.md @@ -720,7 +720,8 @@ GitLab cannot find or doesn't have permission to access the `database_geo.yml` c In an Omnibus GitLab installation, the file should be in `/var/opt/gitlab/gitlab-rails/etc`. If it doesn't exist or inadvertent changes have been made to it, run `sudo gitlab-ctl reconfigure` to restore it to its correct state. -If this path is mounted on a remote volume, please check your volume configuration and that it has correct permissions. +If this path is mounted on a remote volume, ensure your volume configuration +has the correct permissions. ### An existing tracking database cannot be reused @@ -782,3 +783,13 @@ To resolve this issue: using IPv6 to send its status to the **primary** node. If it is, add an entry to the **primary** node using IPv4 in the `/etc/hosts` file. Alternatively, you should [enable IPv6 on the **primary** node](https://docs.gitlab.com/omnibus/settings/nginx.html#setting-the-nginx-listen-address-or-addresses). + +## Fixing client errors + +### Authorization errors from LFS HTTP(s) client requests + +You may have problems if you're running a version of [Git LFS](https://git-lfs.github.com/) before 2.4.2. +As noted in [this authentication issue](https://github.com/git-lfs/git-lfs/issues/3025), +requests redirected from the secondary to the primary node do not properly send the +Authorization header. This may result in either an infinite `Authorization <-> Redirect` +loop, or Authorization errors. diff --git a/doc/administration/geo/replication/updating_the_geo_nodes.md b/doc/administration/geo/replication/updating_the_geo_nodes.md index 55ddccb1d98..af59e45250c 100644 --- a/doc/administration/geo/replication/updating_the_geo_nodes.md +++ b/doc/administration/geo/replication/updating_the_geo_nodes.md @@ -8,7 +8,9 @@ type: howto # Updating the Geo nodes **(PREMIUM ONLY)** CAUTION: **Warning:** -Please ensure you read these sections carefully before updating your Geo nodes! Not following version-specific update steps may result in unexpected downtime. Please [contact support](https://about.gitlab.com/support/#contact-support) if you have any specific questions. +Read these sections carefully before updating your Geo nodes. Not following +version-specific update steps may result in unexpected downtime. If you have +any specific questions, [contact Support](https://about.gitlab.com/support/#contact-support). Updating Geo nodes involves performing: @@ -47,4 +49,4 @@ everything is working correctly: 1. Test the data replication by pushing code to the **primary** node and see if it is received by **secondary** nodes. -If you encounter any issues, please consult the [Geo troubleshooting guide](troubleshooting.md). +If you encounter any issues, see the [Geo troubleshooting guide](troubleshooting.md). diff --git a/doc/administration/geo/replication/version_specific_updates.md b/doc/administration/geo/replication/version_specific_updates.md index 85c869eff92..9ea5283812e 100644 --- a/doc/administration/geo/replication/version_specific_updates.md +++ b/doc/administration/geo/replication/version_specific_updates.md @@ -24,34 +24,41 @@ DROP SERVER gitlab_secondary CASCADE; DROP EXTENSION IF EXISTS postgres_fdw; ``` +DANGER: **Warning:** +In GitLab 13.3, promoting a secondary node to a primary while the secondary is +paused fails. Do not pause replication before promoting a secondary. If the +node is paused, be sure to resume before promoting. To avoid this issue, +upgrade to GitLab 13.4 or later. + +## Updating to GitLab 13.2 + +In GitLab 13.2, promoting a secondary node to a primary while the secondary is paused fails. **Do not pause replication before promoting a secondary.** If the node is paused, please resume before promoting. To avoid this issue, upgrade to GitLab 13.4 or later. + ## Updating to GitLab 13.0 Upgrading to GitLab 13.0 requires GitLab 12.10 to already be using PostgreSQL -version 11. Please see -[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance) -for the recommended procedure. +version 11. For the recommended procedure, see the +[Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). ## Updating to GitLab 12.10 -GitLab 12.10 does not attempt to automatically update the embedded PostgreSQL -server when using Geo, because the PostgreSQL upgrade requires downtime on -secondaries while reinitializing streaming replication. It must be upgraded -manually. Please see -[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance) -for the recommended procedure. +GitLab 12.10 doesn't attempt to update the embedded PostgreSQL server when +using Geo, because the PostgreSQL upgrade requires downtime for secondaries +while reinitializing streaming replication. It must be upgraded manually. For +the recommended procedure, see the +[Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). ## Updating to GitLab 12.9 CAUTION: **Warning:** GitLab 12.9.0 through GitLab 12.9.3 are affected by [a bug that stops repository verification](https://gitlab.com/gitlab-org/gitlab/-/issues/213523). -The issue is fixed in GitLab 12.9.4. Please upgrade to GitLab 12.9.4 or later. +The issue is fixed in GitLab 12.9.4. Upgrade to GitLab 12.9.4 or later. By default, GitLab 12.9 will attempt to automatically update the embedded PostgreSQL server to 10.12 from 9.6, which requires downtime on secondaries -while reinitializing streaming replication. Please see -[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance) -for the recommended procedure. +while reinitializing streaming replication. For the recommended procedure, see the +[Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). This can be temporarily disabled by running the following before updating: @@ -63,9 +70,8 @@ sudo touch /etc/gitlab/disable-postgresql-upgrade By default, GitLab 12.8 will attempt to automatically update the embedded PostgreSQL server to 10.12 from 9.6, which requires downtime on secondaries -while reinitializing streaming replication. Please see -[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance) -for the recommended procedure. +while reinitializing streaming replication. For the recommended procedure, see +the [Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). This can be temporarily disabled by running the following before updating: @@ -75,7 +81,7 @@ sudo touch /etc/gitlab/disable-postgresql-upgrade ## Updating to GitLab 12.7 -DANGER: **Danger:** +DANGER: **Warning:** Only upgrade to GitLab 12.7.5 or later. Do not upgrade to versions 12.7.0 through 12.7.4 because there is [an initialization order bug](https://gitlab.com/gitlab-org/gitlab/-/issues/199672) that causes Geo @@ -84,10 +90,9 @@ fix](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/24021) was shipped in 12.7.5. By default, GitLab 12.7 will attempt to automatically update the embedded -PostgreSQL server to 10.9 from 9.6, which requires downtime on secondaries while -reinitializing streaming replication. Please see -[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance) -for the recommended procedure. +PostgreSQL server to 10.9 from 9.6, which requires downtime on secondaries +while reinitializing streaming replication. For the recommended procedure, see +the [Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). This can be temporarily disabled by running the following before updating: @@ -98,10 +103,9 @@ sudo touch /etc/gitlab/disable-postgresql-upgrade ## Updating to GitLab 12.6 By default, GitLab 12.6 will attempt to automatically update the embedded -PostgreSQL server to 10.9 from 9.6, which requires downtime on secondaries while -reinitializing streaming replication. Please see -[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance) -for the recommended procedure. +PostgreSQL server to 10.9 from 9.6, which requires downtime on secondaries +while reinitializing streaming replication. For the recommended procedure, see +the [Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). This can be temporarily disabled by running the following before updating: @@ -112,10 +116,9 @@ sudo touch /etc/gitlab/disable-postgresql-upgrade ## Updating to GitLab 12.5 By default, GitLab 12.5 will attempt to automatically update the embedded -PostgreSQL server to 10.9 from 9.6, which requires downtime on secondaries while -reinitializing streaming replication. Please see -[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance) -for the recommended procedure. +PostgreSQL server to 10.9 from 9.6, which requires downtime on secondaries +while reinitializing streaming replication. For the recommended procedure, see +the [Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). This can be temporarily disabled by running the following before updating: @@ -126,10 +129,9 @@ sudo touch /etc/gitlab/disable-postgresql-upgrade ## Updating to GitLab 12.4 By default, GitLab 12.4 will attempt to automatically update the embedded -PostgreSQL server to 10.9 from 9.6, which requires downtime on secondaries while -reinitializing streaming replication. Please see -[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance) -for the recommended procedure. +PostgreSQL server to 10.9 from 9.6, which requires downtime on secondaries +while reinitializing streaming replication. For the recommended procedure, see +the [Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). This can be temporarily disabled by running the following before updating: @@ -139,35 +141,31 @@ sudo touch /etc/gitlab/disable-postgresql-upgrade ## Updating to GitLab 12.3 -DANGER: **Danger:** +DANGER: **Warning:** If the existing PostgreSQL server version is 9.6.x, it is recommended to -upgrade to GitLab 12.4 or newer. By default, GitLab 12.3 will attempt to -automatically update the embedded PostgreSQL server to 10.9 from 9.6. In -certain circumstances, it will fail. Please see -[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance) -for more information. +upgrade to GitLab 12.4 or later. By default, GitLab 12.3 attempts to update the +embedded PostgreSQL server from 9.6 to 10.9. In certain circumstances, it will +fail. For more information, see the +[Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). -Additionally, if the PostgreSQL upgrade does not fail, a successful upgrade -requires downtime on secondaries while reinitializing streaming replication. -Please see -[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance) -for the recommended procedure. +Additionally, if the PostgreSQL upgrade doesn't fail, a successful upgrade +requires downtime for secondaries while reinitializing streaming replication. +For the recommended procedure, see the +[Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). ## Updating to GitLab 12.2 -DANGER: **Danger:** +DANGER: **Warning:** If the existing PostgreSQL server version is 9.6.x, it is recommended to -upgrade to GitLab 12.4 or newer. By default, GitLab 12.2 will attempt to -automatically update the embedded PostgreSQL server to 10.9 from 9.6. In -certain circumstances, it will fail. Please see -[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance) -for more information. +upgrade to GitLab 12.4 or later. By default, GitLab 12.2 attempts to update the +embedded PostgreSQL server from 9.6 to 10.9. In certain circumstances, it will +fail. For more information, see the +[Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). Additionally, if the PostgreSQL upgrade does not fail, a successful upgrade -requires downtime on secondaries while reinitializing streaming replication. -Please see -[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance) -for the recommended procedure. +requires downtime for secondaries while reinitializing streaming replication. +For the recommended procedure, see the +[Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). GitLab 12.2 includes the following minor PostgreSQL updates: @@ -187,33 +185,31 @@ The restart avoids a version mismatch when PostgreSQL tries to load the FDW exte ## Updating to GitLab 12.1 -DANGER: **Danger:** +DANGER: **Warning:** If the existing PostgreSQL server version is 9.6.x, it is recommended to -upgrade to GitLab 12.4 or newer. By default, GitLab 12.1 will attempt to -automatically update the embedded PostgreSQL server to 10.9 from 9.6. In -certain circumstances, it will fail. Please see -[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance) -for more information. +upgrade to GitLab 12.4 or later. By default, GitLab 12.1 attempts to update the +embedded PostgreSQL server from 9.6 to 10.9. In certain circumstances, it will +fail. For more information, see the +[Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). -Additionally, if the PostgreSQL upgrade does not fail, a successful upgrade -requires downtime on secondaries while reinitializing streaming replication. -Please see -[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance) -for the recommended procedure. +Additionally, if the PostgreSQL upgrade doesn't fail, a successful upgrade +requires downtime for secondaries while reinitializing streaming replication. +For the recommended procedure, see the +[Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). ## Updating to GitLab 12.0 CAUTION: **Warning:** -This version is affected by [a bug that results in new LFS objects not being replicated to -Geo secondary nodes](https://gitlab.com/gitlab-org/gitlab/-/issues/32696). The issue is fixed -in GitLab 12.1. Please upgrade to GitLab 12.1 or later. +This version is affected by a [bug that results in new LFS objects not being +replicated to Geo secondary nodes](https://gitlab.com/gitlab-org/gitlab/-/issues/32696). +The issue is fixed in GitLab 12.1; be sure to upgrade to GitLab 12.1 or later. ## Updating to GitLab 11.11 CAUTION: **Warning:** -This version is affected by [a bug that results in new LFS objects not being replicated to -Geo secondary nodes](https://gitlab.com/gitlab-org/gitlab/-/issues/32696). The issue is fixed -in GitLab 12.1. Please upgrade to GitLab 12.1 or later. +This version is affected by a [bug that results in new LFS objects not being +replicated to Geo secondary nodes](https://gitlab.com/gitlab-org/gitlab/-/issues/32696). +The issue is fixed in GitLab 12.1; be sure to upgrade to GitLab 12.1 or later. ## Updating to GitLab 10.8 @@ -329,7 +325,7 @@ In GitLab 10.2, synchronizing secondaries over SSH was deprecated. In 10.3, support is removed entirely. All installations will switch to the HTTP/HTTPS cloning method instead. Before updating, ensure that all your Geo nodes are configured to use this method and that it works for your installation. In -particular, ensure that [Git access over HTTP/HTTPS is enabled](configuration.md#step-6-enable-git-access-over-httphttps). +particular, ensure that [Git access over HTTP/HTTPS is enabled](configuration.md#step-5-enable-git-access-over-httphttps). Synchronizing repositories over the public Internet using HTTP is insecure, so you should ensure that you have HTTPS configured before updating. Note that @@ -433,7 +429,7 @@ required because replication slots are used by default. However, if you started with GitLab 9.3 and updated later, you should still follow the instructions below. -When in doubt, it does not hurt to do a resync. The easiest way to do this in +When in doubt, it doesn't hurt to do a resync. The easiest way to do this in Omnibus is the following: 1. Make sure you have Omnibus GitLab on the **primary** server. diff --git a/doc/administration/geo/setup/database.md b/doc/administration/geo/setup/database.md index 09b9c71aeb7..24e55d26997 100644 --- a/doc/administration/geo/setup/database.md +++ b/doc/administration/geo/setup/database.md @@ -474,6 +474,81 @@ high-availability configuration with a cluster of nodes supporting a Geo **primary** node and another cluster of nodes supporting a Geo **secondary** node. For more information, see [High Availability with Omnibus GitLab](../../postgresql/replication_and_failover.md). +## Patroni support + +Support for Patroni is intended to replace `repmgr` as a +[highly availabile PostgreSQL solution](../../postgresql/replication_and_failover.md) +on the primary node, but it can also be used for PostgreSQL HA on a secondary +node. + +Starting with GitLab 13.5, Patroni is available for _experimental_ use with Geo +primary and secondary nodes. Due to its experimental nature, Patroni support is +subject to change without notice. + +This experimental implementation has the following limitations: + +- Whenever a new Leader is elected, the PgBouncer instance must be reconfigured + to point to the new Leader. +- Whenever a new Leader is elected on the primary node, the Standby Leader on + the secondary needs to be reconfigured to point to the new Leader. +- Whenever `gitlab-ctl reconfigure` runs on a Patroni Leader instance, there's a + chance the node will be demoted due to the required short-time restart. To + avoid this, you can pause auto-failover by running `gitlab-ctl patroni pause`. + After a reconfigure, it unpauses on its own. + +For instructions about how to set up Patroni on the primary node, see the +[PostgreSQL replication and failover with Omnibus GitLab](../../postgresql/replication_and_failover.md#patroni) page. + +A production-ready and secure setup requires at least three Patroni instances on +the primary, and a similar configuration on the secondary nodes. Be sure to use +password credentials and other database best practices. + +Similar to `repmgr`, using Patroni on a secondary node is optional. + +To set up database replication with Patroni on a secondary node, configure a +_permanent replication slot_ on the primary node's Patroni cluster, and ensure +password authentication is used. + +On Patroni instances for the primary node, add the following to the +`/etc/gitlab/gitlab.rb` file: + +```ruby +# You need one entry for each secondary, with a unique name following PostgreSQL slot_name constraints: +# +# Configuration syntax will be: 'unique_slotname' => { 'type' => 'physical' }, +# We don't support setting a permanent replication slot for logical replication type +patroni['replication_slots'] = { + 'geo_secondary' => { 'type' => 'physical' } +} + +postgresql['md5_auth_cidr_addresses'] = [ + 'PATRONI_PRIMARY1_IP/32', 'PATRONI_PRIMARY2_IP/32', 'PATRONI_PRIMARY3_IP/32', 'PATRONI_PRIMARY_PGBOUNCER/32', + 'PATRONI_SECONDARY1_IP/32', 'PATRONI_SECONDARY2_IP/32', 'PATRONI_SECONDARY3_IP/32' # we list all secondary instances as they can all become a Standby Leader + # any other instance that needs access to the database as per documentation +] + +postgresql['pgbouncer_user_password'] = 'PGBOUNCER_PASSWORD_HASH' +postgresql['sql_replication_password'] = 'POSTGRESQL_REPLICATION_PASSWORD_HASH' +postgresql['sql_user_password'] = 'POSTGRESQL_PASSWORD_HASH' +``` + +On Patroni instances for the secondary node, add the following to the +`/etc/gitlab/gitlab.rb` file: + +```ruby +postgresql['md5_auth_cidr_addresses'] = [ + 'PATRONI_SECONDARY1_IP/32', 'PATRONI_SECONDARY2_IP/32', 'PATRONI_SECONDARY3_IP/32', 'PATRONI_SECONDARY_PGBOUNCER/32', + # any other instance that needs access to the database as per documentation +] + +patroni['enable'] = true +patroni['standby_cluster']['enable'] = true +patroni['standby_cluster']['host'] = 'PATRONI_PRIMARY_LEADER_IP' # this needs to be changed anytime the primary Leader changes +patroni['standby_cluster']['port'] = 5432 +patroni['standby_cluster']['primary_slot_name'] = 'geo_secondary' # or the unique replication slot name you setup before +patroni['replication_password'] = 'PLAIN_TEXT_POSTGRESQL_REPLICATION_PASSWORD' +``` + ## Troubleshooting Read the [troubleshooting document](../replication/troubleshooting.md). |