Add latest changes from gitlab-org/gitlab@13-2-stable-ee

3 files changed, 260 insertions, 13 deletions

diff --git a/doc/administration/postgresql/index.md b/doc/administration/postgresql/index.md
new file mode 100644
index 00000000000..7e0a2f3cae1
--- /dev/null
+++ b/doc/administration/postgresql/index.md
@@ -0,0 +1,36 @@
+---
+type: reference
+---
+
+# Configuring PostgreSQL for scaling
+
+In this section, you'll be guided through configuring a PostgreSQL database to
+be used with GitLab in one of our [Scalable and Highly Available Setups](../reference_architectures/index.md).
+There are essentially three setups to choose from.
+
+## PostgreSQL replication and failover with Omnibus GitLab **(PREMIUM ONLY)**
+
+This setup is for when you have installed GitLab using the
+[Omnibus GitLab **Enterprise Edition** (EE) package](https://about.gitlab.com/install/?version=ee).
+
+All the tools that are needed like PostgreSQL, PgBouncer, Repmgr are bundled in
+the package, so you can it to set up the whole PostgreSQL infrastructure (primary, replica).
+
+[> Read how to set up PostgreSQL replication and failover using Omnibus GitLab](replication_and_failover.md)
+
+## Standalone PostgreSQL using Omnibus GitLab **(CORE ONLY)**
+
+This setup is for when you have installed the
+[Omnibus GitLab packages](https://about.gitlab.com/install/) (CE or EE),
+to use the bundled PostgreSQL having only its service enabled.
+
+[> Read how to set up a standalone PostgreSQL instance using Omnibus GitLab](standalone.md)
+
+## Provide your own PostgreSQL instance **(CORE ONLY)**
+
+This setup is for when you have installed GitLab using the
+[Omnibus GitLab packages](https://about.gitlab.com/install/) (CE or EE),
+or installed it [from source](../../install/installation.md), but you want to use
+your own external PostgreSQL server.
+
+[> Read how to set up an external PostgreSQL instance](external.md)
diff --git a/doc/administration/postgresql/replication_and_failover.md b/doc/administration/postgresql/replication_and_failover.md
index aa95b983d20..5f550f09e5b 100644
--- a/doc/administration/postgresql/replication_and_failover.md
+++ b/doc/administration/postgresql/replication_and_failover.md
@@ -1,16 +1,15 @@
 # PostgreSQL replication and failover with Omnibus GitLab **(PREMIUM ONLY)**
 
-> Important notes:
->
-> - This document will focus only on configuration supported with [GitLab Premium](https://about.gitlab.com/pricing/), using the Omnibus GitLab package.
-> - If you are a Community Edition or Starter user, consider using a cloud hosted solution.
-> - This document will not cover installations from source.
->
-> - If a setup with replication and failover is not what you were looking for, see the [database configuration document](https://docs.gitlab.com/omnibus/settings/database.html)
->   for the Omnibus GitLab packages.
->
-> Please read this document fully before attempting to configure PostgreSQL with
-> replication and failover for GitLab.
+This document will focus only on configuration supported with [GitLab Premium](https://about.gitlab.com/pricing/), using the Omnibus GitLab package.
+If you are a Community Edition or Starter user, consider using a cloud hosted solution.
+This document will not cover installations from source.
+
+If a setup with replication and failover is not what you were looking for, see
+the [database configuration document](https://docs.gitlab.com/omnibus/settings/database.html)
+for the Omnibus GitLab packages.
+
+It's recommended to read this document fully before attempting to configure PostgreSQL with
+replication and failover for GitLab.
 
 ## Architecture
 
@@ -967,7 +966,8 @@ after it has been restored to service.
   gitlab-ctl restart repmgrd
   ```
 
-  CAUTION: **Warning:** When the server is brought back online, and before
+  CAUTION: **Warning:**
+  When the server is brought back online, and before
   you switch it to a standby node, repmgr will report that there are two masters.
   If there are any clients that are still attempting to write to the old master,
   this will cause a split, and the old master will need to be resynced from
@@ -1127,3 +1127,213 @@ If you're running into an issue with a component not outlined here, be sure to c
 - [Consul](../high_availability/consul.md#troubleshooting)
 - [PostgreSQL](https://docs.gitlab.com/omnibus/settings/database.html#troubleshooting)
 - [GitLab application](../high_availability/gitlab.md#troubleshooting)
+
+## Patroni
+
+NOTE: **Note:**
+Starting from GitLab 13.1, Patroni is available for **experimental** use to replace repmgr. Due to its
+experimental nature, Patroni support is **subject to change without notice.**
+
+Patroni is an opinionated solution for PostgreSQL high-availability. It takes the control of PostgreSQL, overrides its
+configuration and manages its lifecycle (start, stop, restart). This is a more active approach when compared to repmgr.
+Both repmgr and Patroni are both supported and available. But Patroni will be the default (and perhaps the only) option
+for PostgreSQL 12 clustering and cascading replication for Geo deployments.
+
+The [architecture](#example-recommended-setup-manual-steps) (that was mentioned above) does not change for Patroni.
+You do not need any special consideration for Patroni while provisioning your database nodes. Patroni heavily relies on
+Consul to store the state of the cluster and elect a leader. Any failure in Consul cluster and its leader election will
+propagate to Patroni cluster as well.
+
+Similar to repmgr, Patroni monitors the cluster and handles failover. When the primary node fails it works with Consul
+to notify PgBouncer. However, as opposed to repmgr, on failure, Patroni handles the transitioning of the old primary to
+a replica and rejoins it to the cluster automatically. So you do not need any manual operation for recovering the
+cluster as you do with repmgr.
+
+With Patroni the connection flow is slightly different. Patroni on each node connects to Consul agent to join the
+cluster. Only after this point it decides if the node is the primary or a replica. Based on this decision, it configures
+and starts PostgreSQL which it communicates with directly over a Unix socket. This implies that if Consul cluster is not
+functional or does not have a leader, Patroni and by extension PostgreSQL will not start. Patroni also exposes a REST
+API which can be accessed via its [default port](https://docs.gitlab.com/omnibus/package-information/defaults.html#patroni)
+on each node.
+
+### Configuring Patroni cluster
+
+You must enable Patroni explicitly to be able to use it (with `patroni['enable'] = true`). When Patroni is enabled
+repmgr will be disabled automatically.
+
+Any PostgreSQL configuration item that controls replication, for example `wal_level`, `max_wal_senders`, etc, are strictly
+controlled by Patroni and will override the original settings that you make with the `postgresql[...]` configuration key.
+Hence, they are all separated and placed under `patroni['postgresql'][...]`. This behavior is limited to replication.
+Patroni honours any other PostgreSQL configuration that was made with the `postgresql[...]` configuration key. For example,
+`max_wal_senders` by default is set to `5`. If you wish to change this you must set it with the `patroni['postgresql']['max_wal_senders']`
+configuration key.
+
+The configuration of Patroni node is very similar to a repmgr but shorter. When Patroni is enabled, first you can ignore
+any replication setting of PostgreSQL (it will be overwritten anyway). Then you can remove any `repmgr[...]` or
+repmgr-specific configuration as well. Especially, make sure that you remove `postgresql['shared_preload_libraries'] = 'repmgr_funcs'`.
+
+Here is an example similar to [the one that was done with repmgr](#configuring-the-database-nodes):
+
+```ruby
+# Disable all components except PostgreSQL and Repmgr and Consul
+roles['postgres_role']
+
+# Enable Patroni
+patroni['enable'] = true
+
+# PostgreSQL configuration
+postgresql['listen_address'] = '0.0.0.0'
+
+# Disable automatic database migrations
+gitlab_rails['auto_migrate'] = false
+
+# Configure the Consul agent
+consul['services'] = %w(postgresql)
+
+# START user configuration
+# Please set the real values as explained in Required Information section
+#
+# Replace PGBOUNCER_PASSWORD_HASH with a generated md5 value
+postgresql['pgbouncer_user_password'] = 'PGBOUNCER_PASSWORD_HASH'
+# Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value
+postgresql['sql_user_password'] = 'POSTGRESQL_PASSWORD_HASH'
+
+# Replace X with value of number of db nodes + 1 (OPTIONAL the default value is 5)
+patroni['postgresql']['max_wal_senders'] = X
+patroni['postgresql']['max_replication_slots'] = X
+
+# Replace XXX.XXX.XXX.XXX/YY with Network Address
+postgresql['trust_auth_cidr_addresses'] = %w(XXX.XXX.XXX.XXX/YY)
+
+# Replace placeholders:
+#
+# Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z
+# with the addresses gathered for CONSUL_SERVER_NODES
+consul['configuration'] = {
+  retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z)
+}
+#
+# END user configuration
+```
+
+You do not need an additional or different configuration for replica nodes. As a matter of fact, you don't have to have
+a predetermined primary node. Therefore all database nodes use the same configuration.
+
+Once the configuration of a node is done, you must [reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure)
+on each node for the changes to take effect.
+
+Generally, when Consul cluster is ready, the first node that [reconfigures](../restart_gitlab.md#omnibus-gitlab-reconfigure)
+becomes the leader. You do not need to sequence the nodes reconfiguration. You can run them in parallel or in any order.
+If you choose an arbitrary order you do not have any predetermined master.
+
+As opposed to repmgr, once the nodes are reconfigured you do not need any further action or additional command to join
+the replicas.
+
+#### Database authorization for Patroni
+
+Patroni uses Unix socket to manage PostgreSQL instance. Therefore, the connection from the `local` socket must be trusted.
+
+Also, replicas use the replication user (`gitlab_replicator` by default) to communicate with the leader. For this user,
+you can choose between `trust` and `md5` authentication. If you set `postgresql['sql_replication_password']`,
+Patroni will use `md5` authentication, otherwise it falls back to `trust`. You must to specify the cluster CIDR in
+`postgresql['md5_auth_cidr_addresses']` or `postgresql['trust_auth_cidr_addresses']` respectively.
+
+### Interacting with Patroni cluster
+
+You can use `gitlab-ctl patroni members` to check the status of the cluster members. To check the status of each node
+`gitlab-ctl patroni` provides two additional sub-commands, `check-leader` and `check-replica` which indicate if a node
+is the primary or a replica.
+
+When Patroni is enabled, you don't have direct control over `postgresql` service. Patroni will signal PostgreSQL's startup,
+shutdown, and restart. For example, for shutting down PostgreSQL on a node, you must shutdown Patroni on the same node
+with:
+
+```shell
+sudo gitlab-ctl stop patroni
+```
+
+### Manual failover procedure for Patroni
+
+While Patroni supports automatic failover, you also have the ability to perform
+a manual one, where you have two slightly different options:
+
+- **Failover**: allows you to perform a manual failover when there are no healthy nodes.
+  You can perform this action in any PostgreSQL node:
+
+  ```shell
+  sudo gitlab-ctl patroni failover
+  ```
+
+- **Switchover**: only works when the cluster is healthy and allows you to schedule a switchover (it can happen immediately).
+  You can perform this action in any PostgreSQL node:
+
+  ```shell
+  sudo gitlab-ctl patroni switchover
+  ```
+
+For further details on this subject, see the
+[Patroni documentation](https://patroni.readthedocs.io/en/latest/rest_api.html#switchover-and-failover-endpoints).
+
+### Recovering the Patroni cluster
+
+To recover the old primary and rejoin it to the cluster as a replica, you can simply start Patroni with:
+
+```shell
+sudo gitlab-ctl start patroni
+```
+
+No further configuration or intervention is needed.
+
+### Maintenance procedure for Patroni
+
+With Patroni enabled, you can run a planned maintenance. If you want to do some maintenance work on one node and you
+don't want Patroni to manage it, you can use put it into maintenance mode:
+
+```shell
+sudo gitlab-ctl patroni pause
+```
+
+When Patroni runs in a paused mode, it does not change the state of PostgreSQL. Once you are done you can resume Patroni:
+
+```shell
+sudo gitlab-ctl patroni resume
+```
+
+For further details, see [Patroni documentation on this subject](https://patroni.readthedocs.io/en/latest/pause.html).
+
+### Switching from repmgr to Patroni
+
+CAUTION: **Warning:**
+Although switching from repmgr to Patroni is fairly straightforward the other way around is not. Rolling back from
+Patroni to repmgr can be complicated and may involve deletion of data directory. If you need to do that, please contact
+GitLab support.
+
+You can switch an exiting database cluster to use Patroni instead of repmgr with the following steps:
+
+1. Stop repmgr on all replica nodes and lastly with the primary node:
+
+   ```shell
+   sudo gitlab-ctl stop repmgrd
+   ```
+
+1. Stop PostgreSQL on all replica nodes:
+
+   ```shell
+   sudo gitlab-ctl stop postgresql
+   ```
+
+   NOTE: **Note:**
+   Ensure that there is no `walsender` process running on the primary node.
+   `ps aux | grep walsender` must not show any running process.  
+
+1. On the primary node, [configure Patroni](#configuring-patroni-cluster). Remove `repmgr` and any other
+   repmgr-specific configuration. Also remove any configuration that is related to PostgreSQL replication.
+1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) on the primary node. It will become
+   the leader. You can check this with:
+
+   ```shell
+   sudo gitlab-ctl tail patroni
+   ```
+
+1. Repeat the last two steps for all replica nodes. `gitlab.rb` should look the same on all nodes.
+1. Optional: You can remove `gitlab_repmgr` database and role on the primary.
diff --git a/doc/administration/postgresql/standalone.md b/doc/administration/postgresql/standalone.md
index 3e7826ce009..2747749066e 100644
--- a/doc/administration/postgresql/standalone.md
+++ b/doc/administration/postgresql/standalone.md
@@ -53,7 +53,8 @@ together with Omnibus GitLab. This is recommended as part of our
    gitlab_rails['auto_migrate'] = false
    ```
 
-   NOTE: **Note:** The role `postgres_role` was introduced with GitLab 10.3
+   NOTE: **Note:**
+   The role `postgres_role` was introduced with GitLab 10.3
 
 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
 1. Note the PostgreSQL node's IP address or hostname, port, and