diff options
author | GitLab Bot <gitlab-bot@gitlab.com> | 2020-07-20 12:26:25 +0000 |
---|---|---|
committer | GitLab Bot <gitlab-bot@gitlab.com> | 2020-07-20 12:26:25 +0000 |
commit | a09983ae35713f5a2bbb100981116d31ce99826e (patch) | |
tree | 2ee2af7bd104d57086db360a7e6d8c9d5d43667a /doc/administration/postgresql | |
parent | 18c5ab32b738c0b6ecb4d0df3994000482f34bd8 (diff) | |
download | gitlab-ce-a09983ae35713f5a2bbb100981116d31ce99826e.tar.gz |
Add latest changes from gitlab-org/gitlab@13-2-stable-ee
Diffstat (limited to 'doc/administration/postgresql')
-rw-r--r-- | doc/administration/postgresql/index.md | 36 | ||||
-rw-r--r-- | doc/administration/postgresql/replication_and_failover.md | 234 | ||||
-rw-r--r-- | doc/administration/postgresql/standalone.md | 3 |
3 files changed, 260 insertions, 13 deletions
diff --git a/doc/administration/postgresql/index.md b/doc/administration/postgresql/index.md new file mode 100644 index 00000000000..7e0a2f3cae1 --- /dev/null +++ b/doc/administration/postgresql/index.md @@ -0,0 +1,36 @@ +--- +type: reference +--- + +# Configuring PostgreSQL for scaling + +In this section, you'll be guided through configuring a PostgreSQL database to +be used with GitLab in one of our [Scalable and Highly Available Setups](../reference_architectures/index.md). +There are essentially three setups to choose from. + +## PostgreSQL replication and failover with Omnibus GitLab **(PREMIUM ONLY)** + +This setup is for when you have installed GitLab using the +[Omnibus GitLab **Enterprise Edition** (EE) package](https://about.gitlab.com/install/?version=ee). + +All the tools that are needed like PostgreSQL, PgBouncer, Repmgr are bundled in +the package, so you can it to set up the whole PostgreSQL infrastructure (primary, replica). + +[> Read how to set up PostgreSQL replication and failover using Omnibus GitLab](replication_and_failover.md) + +## Standalone PostgreSQL using Omnibus GitLab **(CORE ONLY)** + +This setup is for when you have installed the +[Omnibus GitLab packages](https://about.gitlab.com/install/) (CE or EE), +to use the bundled PostgreSQL having only its service enabled. + +[> Read how to set up a standalone PostgreSQL instance using Omnibus GitLab](standalone.md) + +## Provide your own PostgreSQL instance **(CORE ONLY)** + +This setup is for when you have installed GitLab using the +[Omnibus GitLab packages](https://about.gitlab.com/install/) (CE or EE), +or installed it [from source](../../install/installation.md), but you want to use +your own external PostgreSQL server. + +[> Read how to set up an external PostgreSQL instance](external.md) diff --git a/doc/administration/postgresql/replication_and_failover.md b/doc/administration/postgresql/replication_and_failover.md index aa95b983d20..5f550f09e5b 100644 --- a/doc/administration/postgresql/replication_and_failover.md +++ b/doc/administration/postgresql/replication_and_failover.md @@ -1,16 +1,15 @@ # PostgreSQL replication and failover with Omnibus GitLab **(PREMIUM ONLY)** -> Important notes: -> -> - This document will focus only on configuration supported with [GitLab Premium](https://about.gitlab.com/pricing/), using the Omnibus GitLab package. -> - If you are a Community Edition or Starter user, consider using a cloud hosted solution. -> - This document will not cover installations from source. -> -> - If a setup with replication and failover is not what you were looking for, see the [database configuration document](https://docs.gitlab.com/omnibus/settings/database.html) -> for the Omnibus GitLab packages. -> -> Please read this document fully before attempting to configure PostgreSQL with -> replication and failover for GitLab. +This document will focus only on configuration supported with [GitLab Premium](https://about.gitlab.com/pricing/), using the Omnibus GitLab package. +If you are a Community Edition or Starter user, consider using a cloud hosted solution. +This document will not cover installations from source. + +If a setup with replication and failover is not what you were looking for, see +the [database configuration document](https://docs.gitlab.com/omnibus/settings/database.html) +for the Omnibus GitLab packages. + +It's recommended to read this document fully before attempting to configure PostgreSQL with +replication and failover for GitLab. ## Architecture @@ -967,7 +966,8 @@ after it has been restored to service. gitlab-ctl restart repmgrd ``` - CAUTION: **Warning:** When the server is brought back online, and before + CAUTION: **Warning:** + When the server is brought back online, and before you switch it to a standby node, repmgr will report that there are two masters. If there are any clients that are still attempting to write to the old master, this will cause a split, and the old master will need to be resynced from @@ -1127,3 +1127,213 @@ If you're running into an issue with a component not outlined here, be sure to c - [Consul](../high_availability/consul.md#troubleshooting) - [PostgreSQL](https://docs.gitlab.com/omnibus/settings/database.html#troubleshooting) - [GitLab application](../high_availability/gitlab.md#troubleshooting) + +## Patroni + +NOTE: **Note:** +Starting from GitLab 13.1, Patroni is available for **experimental** use to replace repmgr. Due to its +experimental nature, Patroni support is **subject to change without notice.** + +Patroni is an opinionated solution for PostgreSQL high-availability. It takes the control of PostgreSQL, overrides its +configuration and manages its lifecycle (start, stop, restart). This is a more active approach when compared to repmgr. +Both repmgr and Patroni are both supported and available. But Patroni will be the default (and perhaps the only) option +for PostgreSQL 12 clustering and cascading replication for Geo deployments. + +The [architecture](#example-recommended-setup-manual-steps) (that was mentioned above) does not change for Patroni. +You do not need any special consideration for Patroni while provisioning your database nodes. Patroni heavily relies on +Consul to store the state of the cluster and elect a leader. Any failure in Consul cluster and its leader election will +propagate to Patroni cluster as well. + +Similar to repmgr, Patroni monitors the cluster and handles failover. When the primary node fails it works with Consul +to notify PgBouncer. However, as opposed to repmgr, on failure, Patroni handles the transitioning of the old primary to +a replica and rejoins it to the cluster automatically. So you do not need any manual operation for recovering the +cluster as you do with repmgr. + +With Patroni the connection flow is slightly different. Patroni on each node connects to Consul agent to join the +cluster. Only after this point it decides if the node is the primary or a replica. Based on this decision, it configures +and starts PostgreSQL which it communicates with directly over a Unix socket. This implies that if Consul cluster is not +functional or does not have a leader, Patroni and by extension PostgreSQL will not start. Patroni also exposes a REST +API which can be accessed via its [default port](https://docs.gitlab.com/omnibus/package-information/defaults.html#patroni) +on each node. + +### Configuring Patroni cluster + +You must enable Patroni explicitly to be able to use it (with `patroni['enable'] = true`). When Patroni is enabled +repmgr will be disabled automatically. + +Any PostgreSQL configuration item that controls replication, for example `wal_level`, `max_wal_senders`, etc, are strictly +controlled by Patroni and will override the original settings that you make with the `postgresql[...]` configuration key. +Hence, they are all separated and placed under `patroni['postgresql'][...]`. This behavior is limited to replication. +Patroni honours any other PostgreSQL configuration that was made with the `postgresql[...]` configuration key. For example, +`max_wal_senders` by default is set to `5`. If you wish to change this you must set it with the `patroni['postgresql']['max_wal_senders']` +configuration key. + +The configuration of Patroni node is very similar to a repmgr but shorter. When Patroni is enabled, first you can ignore +any replication setting of PostgreSQL (it will be overwritten anyway). Then you can remove any `repmgr[...]` or +repmgr-specific configuration as well. Especially, make sure that you remove `postgresql['shared_preload_libraries'] = 'repmgr_funcs'`. + +Here is an example similar to [the one that was done with repmgr](#configuring-the-database-nodes): + +```ruby +# Disable all components except PostgreSQL and Repmgr and Consul +roles['postgres_role'] + +# Enable Patroni +patroni['enable'] = true + +# PostgreSQL configuration +postgresql['listen_address'] = '0.0.0.0' + +# Disable automatic database migrations +gitlab_rails['auto_migrate'] = false + +# Configure the Consul agent +consul['services'] = %w(postgresql) + +# START user configuration +# Please set the real values as explained in Required Information section +# +# Replace PGBOUNCER_PASSWORD_HASH with a generated md5 value +postgresql['pgbouncer_user_password'] = 'PGBOUNCER_PASSWORD_HASH' +# Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value +postgresql['sql_user_password'] = 'POSTGRESQL_PASSWORD_HASH' + +# Replace X with value of number of db nodes + 1 (OPTIONAL the default value is 5) +patroni['postgresql']['max_wal_senders'] = X +patroni['postgresql']['max_replication_slots'] = X + +# Replace XXX.XXX.XXX.XXX/YY with Network Address +postgresql['trust_auth_cidr_addresses'] = %w(XXX.XXX.XXX.XXX/YY) + +# Replace placeholders: +# +# Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z +# with the addresses gathered for CONSUL_SERVER_NODES +consul['configuration'] = { + retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z) +} +# +# END user configuration +``` + +You do not need an additional or different configuration for replica nodes. As a matter of fact, you don't have to have +a predetermined primary node. Therefore all database nodes use the same configuration. + +Once the configuration of a node is done, you must [reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) +on each node for the changes to take effect. + +Generally, when Consul cluster is ready, the first node that [reconfigures](../restart_gitlab.md#omnibus-gitlab-reconfigure) +becomes the leader. You do not need to sequence the nodes reconfiguration. You can run them in parallel or in any order. +If you choose an arbitrary order you do not have any predetermined master. + +As opposed to repmgr, once the nodes are reconfigured you do not need any further action or additional command to join +the replicas. + +#### Database authorization for Patroni + +Patroni uses Unix socket to manage PostgreSQL instance. Therefore, the connection from the `local` socket must be trusted. + +Also, replicas use the replication user (`gitlab_replicator` by default) to communicate with the leader. For this user, +you can choose between `trust` and `md5` authentication. If you set `postgresql['sql_replication_password']`, +Patroni will use `md5` authentication, otherwise it falls back to `trust`. You must to specify the cluster CIDR in +`postgresql['md5_auth_cidr_addresses']` or `postgresql['trust_auth_cidr_addresses']` respectively. + +### Interacting with Patroni cluster + +You can use `gitlab-ctl patroni members` to check the status of the cluster members. To check the status of each node +`gitlab-ctl patroni` provides two additional sub-commands, `check-leader` and `check-replica` which indicate if a node +is the primary or a replica. + +When Patroni is enabled, you don't have direct control over `postgresql` service. Patroni will signal PostgreSQL's startup, +shutdown, and restart. For example, for shutting down PostgreSQL on a node, you must shutdown Patroni on the same node +with: + +```shell +sudo gitlab-ctl stop patroni +``` + +### Manual failover procedure for Patroni + +While Patroni supports automatic failover, you also have the ability to perform +a manual one, where you have two slightly different options: + +- **Failover**: allows you to perform a manual failover when there are no healthy nodes. + You can perform this action in any PostgreSQL node: + + ```shell + sudo gitlab-ctl patroni failover + ``` + +- **Switchover**: only works when the cluster is healthy and allows you to schedule a switchover (it can happen immediately). + You can perform this action in any PostgreSQL node: + + ```shell + sudo gitlab-ctl patroni switchover + ``` + +For further details on this subject, see the +[Patroni documentation](https://patroni.readthedocs.io/en/latest/rest_api.html#switchover-and-failover-endpoints). + +### Recovering the Patroni cluster + +To recover the old primary and rejoin it to the cluster as a replica, you can simply start Patroni with: + +```shell +sudo gitlab-ctl start patroni +``` + +No further configuration or intervention is needed. + +### Maintenance procedure for Patroni + +With Patroni enabled, you can run a planned maintenance. If you want to do some maintenance work on one node and you +don't want Patroni to manage it, you can use put it into maintenance mode: + +```shell +sudo gitlab-ctl patroni pause +``` + +When Patroni runs in a paused mode, it does not change the state of PostgreSQL. Once you are done you can resume Patroni: + +```shell +sudo gitlab-ctl patroni resume +``` + +For further details, see [Patroni documentation on this subject](https://patroni.readthedocs.io/en/latest/pause.html). + +### Switching from repmgr to Patroni + +CAUTION: **Warning:** +Although switching from repmgr to Patroni is fairly straightforward the other way around is not. Rolling back from +Patroni to repmgr can be complicated and may involve deletion of data directory. If you need to do that, please contact +GitLab support. + +You can switch an exiting database cluster to use Patroni instead of repmgr with the following steps: + +1. Stop repmgr on all replica nodes and lastly with the primary node: + + ```shell + sudo gitlab-ctl stop repmgrd + ``` + +1. Stop PostgreSQL on all replica nodes: + + ```shell + sudo gitlab-ctl stop postgresql + ``` + + NOTE: **Note:** + Ensure that there is no `walsender` process running on the primary node. + `ps aux | grep walsender` must not show any running process. + +1. On the primary node, [configure Patroni](#configuring-patroni-cluster). Remove `repmgr` and any other + repmgr-specific configuration. Also remove any configuration that is related to PostgreSQL replication. +1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) on the primary node. It will become + the leader. You can check this with: + + ```shell + sudo gitlab-ctl tail patroni + ``` + +1. Repeat the last two steps for all replica nodes. `gitlab.rb` should look the same on all nodes. +1. Optional: You can remove `gitlab_repmgr` database and role on the primary. diff --git a/doc/administration/postgresql/standalone.md b/doc/administration/postgresql/standalone.md index 3e7826ce009..2747749066e 100644 --- a/doc/administration/postgresql/standalone.md +++ b/doc/administration/postgresql/standalone.md @@ -53,7 +53,8 @@ together with Omnibus GitLab. This is recommended as part of our gitlab_rails['auto_migrate'] = false ``` - NOTE: **Note:** The role `postgres_role` was introduced with GitLab 10.3 + NOTE: **Note:** + The role `postgres_role` was introduced with GitLab 10.3 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. Note the PostgreSQL node's IP address or hostname, port, and |