diff options
Diffstat (limited to 'doc/administration/postgresql')
| -rw-r--r-- | doc/administration/postgresql/external.md | 6 | ||||
| -rw-r--r-- | doc/administration/postgresql/index.md | 5 | ||||
| -rw-r--r-- | doc/administration/postgresql/pgbouncer.md | 32 | ||||
| -rw-r--r-- | doc/administration/postgresql/replication_and_failover.md | 117 | ||||
| -rw-r--r-- | doc/administration/postgresql/standalone.md | 14 |
5 files changed, 161 insertions, 13 deletions
diff --git a/doc/administration/postgresql/external.md b/doc/administration/postgresql/external.md index 632b68fb014..4a164c66578 100644 --- a/doc/administration/postgresql/external.md +++ b/doc/administration/postgresql/external.md @@ -1,3 +1,9 @@ +--- +stage: Enablement +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Configure GitLab using an external PostgreSQL service If you're hosting GitLab on a cloud provider, you can optionally use a diff --git a/doc/administration/postgresql/index.md b/doc/administration/postgresql/index.md index 2720d8e696b..c7ec46db654 100644 --- a/doc/administration/postgresql/index.md +++ b/doc/administration/postgresql/index.md @@ -1,11 +1,14 @@ --- +stage: Enablement +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: reference --- # Configuring PostgreSQL for scaling In this section, you'll be guided through configuring a PostgreSQL database to -be used with GitLab in one of our [Scalable and Highly Available Setups](../reference_architectures/index.md). +be used with GitLab in one of our [reference architectures](../reference_architectures/index.md). There are essentially three setups to choose from. ## PostgreSQL replication and failover with Omnibus GitLab **(PREMIUM ONLY)** diff --git a/doc/administration/postgresql/pgbouncer.md b/doc/administration/postgresql/pgbouncer.md index 9db3e017359..7760b197267 100644 --- a/doc/administration/postgresql/pgbouncer.md +++ b/doc/administration/postgresql/pgbouncer.md @@ -1,4 +1,7 @@ --- +stage: Enablement +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: reference --- @@ -145,6 +148,35 @@ ote_pid | tls (1 row) ``` +## Procedure for bypassing PgBouncer + +Some database changes have to be done directly, and not through PgBouncer. This includes database restores and GitLab upgrades (because of the database migrations). + +1. To find the primary node, run the following on a database node: + + ```shell + sudo gitlab-ctl repmgr cluster show + ``` + +1. Edit `/etc/gitlab/gitlab.rb` on the application node you're performing the task on, and update + `gitlab_rails['db_host']` and `gitlab_rails['db_port']` with the database + primary's host and port. + +1. Run reconfigure: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +Once you've performed the tasks or procedure, switch back to using PgBouncer: + +1. Change back `/etc/gitlab/gitlab.rb` to point to PgBouncer. +1. Run reconfigure: + + ```shell + sudo gitlab-ctl reconfigure + ``` + ## Troubleshooting In case you are experiencing any issues connecting through PgBouncer, the first diff --git a/doc/administration/postgresql/replication_and_failover.md b/doc/administration/postgresql/replication_and_failover.md index bc2af167e6c..0a40b61fd3c 100644 --- a/doc/administration/postgresql/replication_and_failover.md +++ b/doc/administration/postgresql/replication_and_failover.md @@ -1,10 +1,16 @@ +--- +stage: Enablement +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # PostgreSQL replication and failover with Omnibus GitLab **(PREMIUM ONLY)** -This document will focus only on configuration supported with [GitLab Premium](https://about.gitlab.com/pricing/), using the Omnibus GitLab package. -If you are a Community Edition or Starter user, consider using a cloud hosted solution. -This document will not cover installations from source. +This document focuses on configuration supported with [GitLab Premium](https://about.gitlab.com/pricing/), using the Omnibus GitLab package. +If you're a Community Edition or Starter user, consider using a cloud hosted solution. +This document doesn't cover installations from source. -If a setup with replication and failover is not what you were looking for, see +If a setup with replication and failover isn't what you were looking for, see the [database configuration document](https://docs.gitlab.com/omnibus/settings/database.html) for the Omnibus GitLab packages. @@ -87,9 +93,9 @@ information. #### Network information -PostgreSQL does not listen on any network interface by default. It needs to know -which IP address to listen on in order to be accessible to other services. -Similarly, PostgreSQL access is controlled based on the network source. +PostgreSQL doesn't listen on any network interface by default. It needs to know +which IP address to listen on to be accessible to other services. Similarly, +PostgreSQL access is controlled based on the network source. This is why you will need: @@ -419,7 +425,7 @@ Check the [Troubleshooting section](#troubleshooting) before proceeding. 1. Make sure you collect [`CONSUL_SERVER_NODES`](#consul-information), [`CONSUL_PASSWORD_HASH`](#consul-information), and [`PGBOUNCER_PASSWORD_HASH`](#pgbouncer-information) before executing the next step. -1. One each node, edit the `/etc/gitlab/gitlab.rb` config file and replace values noted in the `# START user configuration` section as below: +1. One each node, edit the `/etc/gitlab/gitlab.rb` configuration file and replace values noted in the `# START user configuration` section as below: ```ruby # Disable all components except PgBouncer and Consul agent @@ -1263,6 +1269,11 @@ with: sudo gitlab-ctl stop patroni ``` +Note that stopping or restarting Patroni service on the leader node will trigger the automatic failover. If you +want to signal Patroni to reload its configuration or restart PostgreSQL process without triggering the failover, you +must use the `reload` or `restart` sub-commands of `gitlab-ctl patroni` instead. These two sub-commands are wrappers of +the same `patronictl` commands. + ### Manual failover procedure for Patroni While Patroni supports automatic failover, you also have the ability to perform @@ -1348,3 +1359,93 @@ You can switch an exiting database cluster to use Patroni instead of repmgr with 1. Repeat the last two steps for all replica nodes. `gitlab.rb` should look the same on all nodes. 1. Optional: You can remove `gitlab_repmgr` database and role on the primary. + +### Upgrading PostgreSQL major version in a Patroni cluster + +As of GitLab 13.3, PostgreSQL 11.7 and 12.3 are both shipped with Omnibus GitLab. GitLab still +uses PostgreSQL 11 by default. Therefore `gitlab-ctl pg-upgrade` does not automatically upgrade +to PostgreSQL 12. If you want to upgrade to PostgreSQL 12, you must ask for it explicitly. + +CAUTION: **Warning:** +The procedure for upgrading PostgreSQL in a Patroni cluster is different than when upgrading using repmgr. +The following outlines the key differences and important considerations that need to be accounted for when +upgrading PostgreSQL. + +Here are a few key facts that you must consider before upgrading PostgreSQL: + +- The main point is that you will have to **shut down the Patroni cluster**. This means that your + GitLab deployment will be down for the duration of database upgrade or, at least, as long as your leader + node is upgraded. This can be **a significant downtime depending on the size of your database**. + +- Upgrading PostgreSQL creates a new data directory with a new control data. From Patroni's perspective + this is a new cluster that needs to be bootstrapped again. Therefore, as part of the upgrade procedure, + the cluster state, which is stored in Consul, will be wiped out. Once the upgrade is completed, Patroni + will be instructed to bootstrap a new cluster. **Note that this will change your _cluster ID_**. + +- The procedures for upgrading leader and replicas are not the same. That is why it is important to use the + right procedure on each node. + +- Upgrading a replica node **deletes the data directory and resynchronizes it** from the leader using the + configured replication method (currently `pg_basebackup` is the only available option). It might take some + time for replica to catch up with the leader, depending on the size of your database. + +- An overview of the upgrade procedure is outlined in [Patoni's documentation](https://patroni.readthedocs.io/en/latest/existing_data.html#major-upgrade-of-postgresql-version). + You can still use `gitlab-ctl pg-upgrade` which implements this procedure with a few adjustments. + +Considering these, you should carefully plan your PostgreSQL upgrade: + +1. Find out which node is the leader and which node is a replica: + + ```shell + gitlab-ctl patroni members + ``` + + NOTE: **Note:** + `gitlab-ctl pg-upgrade` tries to detect the role of the node. If for any reason the auto-detection + does not work or you believe it did not detect the role correctly, you can use the `--leader` or `--replica` + arguments to manually override it. + +1. Stop Patroni **only on replicas**. + + ```shell + sudo gitlab-ctl stop patroni + ``` + +1. Enable the maintenance mode on the **application node**: + + ```shell + sudo gitlab-ctl deploy-page up + ``` + +1. Upgrade PostgreSQL on **the leader node** and make sure that the upgrade is completed successfully: + + ```shell + sudo gitlab-ctl pg-upgrade -V 12 + ``` + +1. Check the status of the leader and cluster. You can only proceed if you have a healthy leader: + + ```shell + gitlab-ctl patroni check-leader + + # OR + + gitlab-ctl patroni members + ``` + +1. You can now disable the maintenance mode on the **application node**: + + ```shell + sudo gitlab-ctl deploy-page down + ``` + +1. Upgrade PostgreSQL **on replicas** (you can do this in parallel on all of them): + + ```shell + sudo gitlab-ctl pg-upgrade -V 12 + ``` + +NOTE: **Note:** +Reverting PostgreSQL upgrade with `gitlab-ctl revert-pg-upgrade` has the same considerations as +`gitlab-ctl pg-upgrade`. You should follow the same procedure by first stopping the replicas, +then reverting the leader, and finally reverting the replicas. diff --git a/doc/administration/postgresql/standalone.md b/doc/administration/postgresql/standalone.md index 2747749066e..2ac74e8a4a0 100644 --- a/doc/administration/postgresql/standalone.md +++ b/doc/administration/postgresql/standalone.md @@ -1,15 +1,21 @@ +--- +stage: Enablement +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Standalone PostgreSQL using Omnibus GitLab **(CORE ONLY)** If you wish to have your database service hosted separately from your GitLab -application server(s), you can do this using the PostgreSQL binaries packaged +application servers, you can do this using the PostgreSQL binaries packaged together with Omnibus GitLab. This is recommended as part of our [reference architecture for up to 2,000 users](../reference_architectures/2k_users.md). ## Setting it up -1. SSH into the PostgreSQL server. -1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab - package you want using **steps 1 and 2** from the GitLab downloads page. +1. SSH in to the PostgreSQL server. +1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab + package you want using *steps 1 and 2* from the GitLab downloads page. - Do not complete any other steps on the download page. 1. Generate a password hash for PostgreSQL. This assumes you will use the default username of `gitlab` (recommended). The command will request a password |