Add latest changes from gitlab-org/gitlab@14-0-stable-eev14.0.0-rc42

8 files changed, 1122 insertions, 1283 deletions

diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md
index e4a699b962c..4627b27a45e 100644
--- a/doc/administration/reference_architectures/10k_users.md
+++ b/doc/administration/reference_architectures/10k_users.md
@@ -17,29 +17,34 @@ full list of reference architectures, see
 
 | Service                                    | Nodes       | Configuration           | GCP              | AWS          | Azure     |
 |--------------------------------------------|-------------|-------------------------|------------------|--------------|-----------|
-| External load balancing node               | 1           | 2 vCPU, 1.8 GB memory   | `n1-highcpu-2`   | `c5.large`   | `F2s v2`  |
-| Consul*                                    | 3           | 2 vCPU, 1.8 GB memory   | `n1-highcpu-2`   | `c5.large`   | `F2s v2`  |
-| PostgreSQL*                                | 3           | 8 vCPU, 30 GB memory    | `n1-standard-8`  | `m5.2xlarge` | `D8s v3`  |
-| PgBouncer*                                 | 3           | 2 vCPU, 1.8 GB memory   | `n1-highcpu-2`   | `c5.large`   | `F2s v2`  |
-| Internal load balancing node               | 1           | 2 vCPU, 1.8 GB memory   | `n1-highcpu-2`   | `c5.large`   | `F2s v2`  |
-| Redis - Cache**                            | 3           | 4 vCPU, 15 GB memory    | `n1-standard-4`  | `m5.xlarge`  | `D4s v3`  |
-| Redis - Queues / Shared State**            | 3           | 4 vCPU, 15 GB memory    | `n1-standard-4`  | `m5.xlarge`  | `D4s v3`  |
-| Redis Sentinel - Cache**                   | 3           | 1 vCPU, 3.75 GB memory  | `n1-standard-1`  | `c5.large`   | `A1 v2`   |
-| Redis Sentinel - Queues / Shared State**   | 3           | 1 vCPU, 3.75 GB memory  | `n1-standard-1`  | `c5.large`   | `A1 v2`   |
+| External load balancing node(3)            | 1           | 2 vCPU, 1.8 GB memory   | `n1-highcpu-2`   | `c5.large`   | `F2s v2`  |
+| Consul(1)                                  | 3           | 2 vCPU, 1.8 GB memory   | `n1-highcpu-2`   | `c5.large`   | `F2s v2`  |
+| PostgreSQL(1)                              | 3           | 8 vCPU, 30 GB memory    | `n1-standard-8`  | `m5.2xlarge` | `D8s v3`  |
+| PgBouncer(1)                               | 3           | 2 vCPU, 1.8 GB memory   | `n1-highcpu-2`   | `c5.large`   | `F2s v2`  |
+| Internal load balancing node(3)            | 1           | 2 vCPU, 1.8 GB memory   | `n1-highcpu-2`   | `c5.large`   | `F2s v2`  |
+| Redis - Cache(2)                           | 3           | 4 vCPU, 15 GB memory    | `n1-standard-4`  | `m5.xlarge`  | `D4s v3`  |
+| Redis - Queues / Shared State(2)           | 3           | 4 vCPU, 15 GB memory    | `n1-standard-4`  | `m5.xlarge`  | `D4s v3`  |
+| Redis Sentinel - Cache(2)                  | 3           | 1 vCPU, 3.75 GB memory  | `n1-standard-1`  | `c5.large`   | `A1 v2`   |
+| Redis Sentinel - Queues / Shared State(2)  | 3           | 1 vCPU, 3.75 GB memory  | `n1-standard-1`  | `c5.large`   | `A1 v2`   |
 | Gitaly                                     | 3           | 16 vCPU, 60 GB memory   | `n1-standard-16` | `m5.4xlarge` | `D16s v3` |
 | Praefect                                   | 3           | 2 vCPU, 1.8 GB memory   | `n1-highcpu-2`   | `c5.large`   | `F2s v2`  |
-| Praefect PostgreSQL*                       | 1+          | 2 vCPU, 1.8 GB memory   | `n1-highcpu-2`   | `c5.large`   | `F2s v2`  |
+| Praefect PostgreSQL(1)                     | 1+          | 2 vCPU, 1.8 GB memory   | `n1-highcpu-2`   | `c5.large`   | `F2s v2`  |
 | Sidekiq                                    | 4           | 4 vCPU, 15 GB memory    | `n1-standard-4`  | `m5.xlarge`  | `D4s v3`  |
 | GitLab Rails                               | 3           | 32 vCPU, 28.8 GB memory | `n1-highcpu-32`  | `c5.9xlarge` | `F32s v2` |
 | Monitoring node                            | 1           | 4 vCPU, 3.6 GB memory   | `n1-highcpu-4`   | `c5.xlarge`  | `F4s v2`  |
-| Object storage                             | n/a         | n/a                     | n/a              | n/a          | n/a       |
-| NFS server                                 | 1           | 4 vCPU, 3.6 GB memory   | `n1-highcpu-4`   | `c5.xlarge`  | `F4s v2`  |
+| Object storage(4)                          | n/a         | n/a                     | n/a              | n/a          | n/a       |
+| NFS server (optional, not recommended)     | 1           | 4 vCPU, 3.6 GB memory   | `n1-highcpu-4`   | `c5.xlarge`  | `F4s v2`  |
+
+<!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix -->
+<!-- markdownlint-disable MD029 -->
+1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work, however Azure Database for PostgreSQL is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to performance issues. Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery.
+2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work.
+3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work.
+4. Should be run on reputable third party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work.
+<!-- markdownlint-enable MD029 -->
 
 NOTE:
-Components marked with * can be optionally run on reputable
-third party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work.
-Components marked with ** can be optionally run on reputable
-third party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work.
+For all PaaS solutions that involve configuring instances, it is strongly recommended to implement a minimum of three nodes in three different availability zones to align with resilient cloud architecture practices.
 
 ```plantuml
 @startuml 10k
@@ -157,7 +162,7 @@ To set up GitLab and its components to accommodate up to 10,000 users:
    provides access to the Git repositories.
 1. [Configure Sidekiq](#configure-sidekiq).
 1. [Configure the main GitLab Rails application](#configure-gitlab-rails)
-   to run Puma/Unicorn, Workhorse, GitLab Shell, and to serve all frontend
+   to run Puma, Workhorse, GitLab Shell, and to serve all frontend
    requests (which include UI, API, and Git over HTTP/SSH).
 1. [Configure Prometheus](#configure-prometheus) to monitor your GitLab
    environment.
@@ -411,11 +416,6 @@ The following IPs will be used as an example:
 - `10.6.0.12`: Consul 2
 - `10.6.0.13`: Consul 3
 
-NOTE:
-The configuration processes for the other servers in your reference architecture will
-use the `/etc/gitlab/gitlab-secrets.json` file from your Consul server to connect
-with the other servers.
-
 To configure Consul:
 
 1. SSH in to the server that will host Consul.
@@ -426,10 +426,9 @@ To configure Consul:
 1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
 
    ```ruby
-   roles ['consul_role']
+   roles(['consul_role'])
 
    ## Enable service discovery for Prometheus
-   consul['enable'] = true
    consul['monitoring_service_discovery'] =  true
 
    ## The IPs of the Consul server nodes
@@ -446,7 +445,11 @@ To configure Consul:
    gitlab_rails['auto_migrate'] = false
    ```
 
+1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
+   the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
+
 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
+
 1. Go through the steps again for all the other Consul nodes, and
    make sure you set up the correct IPs.
 
@@ -538,6 +541,15 @@ in the second step, do not supply the `EXTERNAL_URL` value.
    sudo gitlab-ctl pg-password-md5 pgbouncer
    ```
 
+1. Generate a password hash for the PostgreSQL replication username/password pair. This assumes you will use the default
+   username of `gitlab_replicator` (recommended). The command will request a password
+   and a confirmation. Use the value that is output by this command in the next step
+   as the value of `<postgresql_replication_password_hash>`:
+
+   ```shell
+   sudo gitlab-ctl pg-password-md5 gitlab_replicator
+   ```
+
 1. Generate a password hash for the Consul database username/password pair. This assumes you will use the default
    username of `gitlab-consul` (recommended). The command will request a password
    and confirmation. Use the value that is output by this command in the next
@@ -550,19 +562,21 @@ in the second step, do not supply the `EXTERNAL_URL` value.
 1. On every database node, edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section:
 
    ```ruby
-   # Disable all components except PostgreSQL, Patroni, and Consul
-   roles ['postgres_role']
+   # Disable all components except Patroni and Consul
+   roles(['patroni_role'])
 
    # PostgreSQL configuration
    postgresql['listen_address'] = '0.0.0.0'
 
-   # Enable Patroni
-   patroni['enable'] = true
-   # Set `max_wal_senders` to one more than the number of database nodes in the cluster.
+   # Sets `max_replication_slots` to double the number of database nodes.
+   # Patroni uses one extra slot per node when initiating the replication.
+   patroni['postgresql']['max_replication_slots'] = 8
+
+   # Set `max_wal_senders` to one more than the number of replication slots in the cluster.
    # This is used to prevent replication from using up all of the
    # available database connections.
-   patroni['postgresql']['max_wal_senders'] = 4
-   patroni['postgresql']['max_replication_slots'] = 4
+   patroni['postgresql']['max_wal_senders'] = 9
+
    # Incoming recommended value for max connections is 500. See https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5691.
    patroni['postgresql']['max_connections'] = 500
 
@@ -570,7 +584,6 @@ in the second step, do not supply the `EXTERNAL_URL` value.
    gitlab_rails['auto_migrate'] = false
 
    # Configure the Consul agent
-   consul['enable'] = true
    consul['services'] = %w(postgresql)
    ## Enable service discovery for Prometheus
    consul['monitoring_service_discovery'] =  true
@@ -580,6 +593,8 @@ in the second step, do not supply the `EXTERNAL_URL` value.
    #
    # Replace PGBOUNCER_PASSWORD_HASH with a generated md5 value
    postgresql['pgbouncer_user_password'] = '<pgbouncer_password_hash>'
+   # Replace POSTGRESQL_REPLICATION_PASSWORD_HASH with a generated md5 value
+   postgresql['sql_replication_password'] = '<postgresql_replication_password_hash>'
    # Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value
    postgresql['sql_user_password'] = '<postgresql_password_hash>'
 
@@ -603,9 +618,8 @@ PostgreSQL, with Patroni managing its failover, will default to use `pg_rewind`
 Like most failover handling methods, this has a small chance of leading to data loss.
 Learn more about the various [Patroni replication methods](../postgresql/replication_and_failover.md#selecting-the-appropriate-patroni-replication-method).
 
-1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace
-   the file of the same name on this server. If that file is not on this server,
-   add the file from your Consul server to this server.
+1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
+   the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
 
 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
 
@@ -620,21 +634,7 @@ are supported and can be added if needed.
 
 #### PostgreSQL post-configuration
 
-SSH in to the **primary node**:
-
-1. Open a database prompt:
-
-   ```shell
-   gitlab-psql -d gitlabhq_production
-   ```
-
-1. Make sure the `pg_trgm` extension is enabled (it might already be):
-
-   ```shell
-   CREATE EXTENSION pg_trgm;
-   ```
-
-1. Exit the database prompt by typing `\q` and Enter.
+SSH in to any of the Patroni nodes on the **primary site**:
 
 1. Check the status of the leader and cluster:
 
@@ -676,7 +676,7 @@ The following IPs will be used as an example:
 
    ```ruby
    # Disable all components except Pgbouncer and Consul agent
-   roles ['pgbouncer_role']
+   roles(['pgbouncer_role'])
 
    # Configure PgBouncer
    pgbouncer['admin_users'] = %w(pgbouncer gitlab-consul)
@@ -693,7 +693,6 @@ The following IPs will be used as an example:
 
    # Configure Consul agent
    consul['watchers'] = %w(postgresql)
-   consul['enable'] = true
    consul['configuration'] = {
    retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13)
    }
@@ -705,9 +704,8 @@ The following IPs will be used as an example:
    node_exporter['listen_address'] = '0.0.0.0:9100'
    ```
 
-1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace
-   the file of the same name on this server. If that file is not on this server,
-   add the file from your Consul server to this server.
+1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
+   the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
 
 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
 
@@ -826,8 +824,8 @@ a node and change its status from primary to replica (and vice versa).
 1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
 
    ```ruby
-   # Specify server role as 'redis_master_role'
-   roles ['redis_master_role']
+   # Specify server role as 'redis_master_role' and enable Consul agent
+   roles(['redis_master_role', 'consul_role'])
 
    # IP address pointing to a local IP that the other machines can reach to.
    # You can also set bind to '0.0.0.0' which listen in all interfaces.
@@ -849,7 +847,6 @@ a node and change its status from primary to replica (and vice versa).
    redis['maxmemory_samples'] = 5
 
    ## Enable service discovery for Prometheus
-   consul['enable'] = true
    consul['monitoring_service_discovery'] =  true
 
    ## The IPs of the Consul server nodes
@@ -861,19 +858,22 @@ a node and change its status from primary to replica (and vice versa).
    # Set the network addresses that the exporters will listen on
    node_exporter['listen_address'] = '0.0.0.0:9100'
    redis_exporter['listen_address'] = '0.0.0.0:9121'
+   redis_exporter['flags'] = {
+        'redis.addr' => 'redis://10.6.0.51:6379',
+        'redis.password' => 'redis-password-goes-here',
+   }
 
    # Prevent database migrations from running on upgrade automatically
    gitlab_rails['auto_migrate'] = false
    ```
 
-1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace
-   the file of the same name on this server. If that file is not on this server,
-   add the file from your Consul server to this server.
+1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
+   the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
 
 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
 
 You can specify multiple roles, like sentinel and Redis, as:
-`roles ['redis_sentinel_role', 'redis_master_role']`. Read more about
+`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about
 [roles](https://docs.gitlab.com/omnibus/roles/).
 
 #### Configure the replica Redis Cache nodes
@@ -886,8 +886,8 @@ You can specify multiple roles, like sentinel and Redis, as:
 1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
 
    ```ruby
-   # Specify server role as 'redis_replica_role'
-   roles ['redis_replica_role']
+   # Specify server role as 'redis_replica_role' and enable Consul agent
+   roles(['redis_replica_role', 'consul_role'])
 
    # IP address pointing to a local IP that the other machines can reach to.
    # You can also set bind to '0.0.0.0' which listen in all interfaces.
@@ -916,7 +916,6 @@ You can specify multiple roles, like sentinel and Redis, as:
    redis['maxmemory_samples'] = 5
 
    ## Enable service discovery for Prometheus
-   consul['enable'] = true
    consul['monitoring_service_discovery'] =  true
 
    ## The IPs of the Consul server nodes
@@ -928,21 +927,24 @@ You can specify multiple roles, like sentinel and Redis, as:
    # Set the network addresses that the exporters will listen on
    node_exporter['listen_address'] = '0.0.0.0:9100'
    redis_exporter['listen_address'] = '0.0.0.0:9121'
+   redis_exporter['flags'] = {
+        'redis.addr' => 'redis://10.6.0.52:6379',
+        'redis.password' => 'redis-password-goes-here',
+   }
 
    # Prevent database migrations from running on upgrade automatically
    gitlab_rails['auto_migrate'] = false
    ```
 
-1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace
-   the file of the same name on this server. If that file is not on this server,
-   add the file from your Consul server to this server.
+1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
+   the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
 
 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
 1. Go through the steps again for all the other replica nodes, and
    make sure to set up the IPs correctly.
 
 You can specify multiple roles, like sentinel and Redis, as:
-`roles ['redis_sentinel_role', 'redis_master_role']`. Read more about
+`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about
 [roles](https://docs.gitlab.com/omnibus/roles/).
 
 These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after
@@ -984,7 +986,7 @@ To configure the Sentinel Cache server:
 1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
 
    ```ruby
-   roles ['redis_sentinel_role']
+   roles(['redis_sentinel_role', 'consul_role'])
 
    ## Must be the same in every sentinel node
    redis['master_name'] = 'gitlab-redis-cache'
@@ -1048,7 +1050,6 @@ To configure the Sentinel Cache server:
    #sentinel['failover_timeout'] = 60000
 
    ## Enable service discovery for Prometheus
-   consul['enable'] = true
    consul['monitoring_service_discovery'] =  true
 
    ## The IPs of the Consul server nodes
@@ -1065,9 +1066,8 @@ To configure the Sentinel Cache server:
    gitlab_rails['auto_migrate'] = false
    ```
 
-1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace
-   the file of the same name on this server. If that file is not on this server,
-   add the file from your Consul server to this server.
+1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
+   the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
 
 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
 1. Go through the steps again for all the other Consul/Sentinel nodes, and
@@ -1097,8 +1097,8 @@ a node and change its status from primary to replica (and vice versa).
 1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
 
    ```ruby
-   # Specify server role as 'redis_master_role'
-   roles ['redis_master_role']
+   # Specify server role as 'redis_master_role' and enable Consul agent
+   roles(['redis_master_role', 'consul_role'])
 
    # IP address pointing to a local IP that the other machines can reach to.
    # You can also set bind to '0.0.0.0' which listen in all interfaces.
@@ -1114,7 +1114,6 @@ a node and change its status from primary to replica (and vice versa).
    redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER'
 
    ## Enable service discovery for Prometheus
-   consul['enable'] = true
    consul['monitoring_service_discovery'] =  true
 
    ## The IPs of the Consul server nodes
@@ -1131,14 +1130,13 @@ a node and change its status from primary to replica (and vice versa).
    gitlab_rails['auto_migrate'] = false
    ```
 
-1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace
-   the file of the same name on this server. If that file is not on this server,
-   add the file from your Consul server to this server.
+1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
+   the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
 
 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
 
 You can specify multiple roles, like sentinel and Redis, as:
-`roles ['redis_sentinel_role', 'redis_master_role']`. Read more about
+`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about
 [roles](https://docs.gitlab.com/omnibus/roles/).
 
 #### Configure the replica Redis Queues nodes
@@ -1151,8 +1149,8 @@ You can specify multiple roles, like sentinel and Redis, as:
 1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
 
    ```ruby
-   # Specify server role as 'redis_replica_role'
-   roles ['redis_replica_role']
+   # Specify server role as 'redis_replica_role' and enable Consul agent
+   roles(['redis_replica_role', 'consul_role'])
 
    # IP address pointing to a local IP that the other machines can reach to.
    # You can also set bind to '0.0.0.0' which listen in all interfaces.
@@ -1175,7 +1173,6 @@ You can specify multiple roles, like sentinel and Redis, as:
    #redis['master_port'] = 6379
 
    ## Enable service discovery for Prometheus
-   consul['enable'] = true
    consul['monitoring_service_discovery'] =  true
 
    ## The IPs of the Consul server nodes
@@ -1192,16 +1189,15 @@ You can specify multiple roles, like sentinel and Redis, as:
    gitlab_rails['auto_migrate'] = false
    ```
 
-1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace
-   the file of the same name on this server. If that file is not on this server,
-   add the file from your Consul server to this server.
+1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
+   the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
 
 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
 1. Go through the steps again for all the other replica nodes, and
    make sure to set up the IPs correctly.
 
 You can specify multiple roles, like sentinel and Redis, as:
-`roles ['redis_sentinel_role', 'redis_master_role']`. Read more about
+`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about
 [roles](https://docs.gitlab.com/omnibus/roles/).
 
 These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after
@@ -1243,7 +1239,7 @@ To configure the Sentinel Queues server:
 1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
 
    ```ruby
-   roles ['redis_sentinel_role']
+   roles(['redis_sentinel_role', 'consul_role'])
 
    ## Must be the same in every sentinel node
    redis['master_name'] = 'gitlab-redis-persistent'
@@ -1307,7 +1303,6 @@ To configure the Sentinel Queues server:
    #sentinel['failover_timeout'] = 60000
 
    ## Enable service discovery for Prometheus
-   consul['enable'] = true
    consul['monitoring_service_discovery'] =  true
 
    ## The IPs of the Consul server nodes
@@ -1324,17 +1319,8 @@ To configure the Sentinel Queues server:
    gitlab_rails['auto_migrate'] = false
    ```
 
-1. To prevent database migrations from running on upgrade, run:
-
-   ```shell
-   sudo touch /etc/gitlab/skip-auto-reconfigure
-   ```
-
-   Only the primary GitLab application server should handle migrations.
-
-1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace
-   the file of the same name on this server. If that file is not on this server,
-   add the file from your Consul server to this server.
+1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
+   the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
 
 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
 1. Go through the steps again for all the other Sentinel nodes, and
@@ -1397,9 +1383,7 @@ in the second step, do not supply the `EXTERNAL_URL` value.
 
    ```ruby
    # Disable all components except PostgreSQL and Consul
-   roles ['postgres_role']
-   repmgr['enable'] = false
-   patroni['enable'] = false
+   roles(['postgres_role', 'consul_role'])
 
    # PostgreSQL configuration
    postgresql['listen_address'] = '0.0.0.0'
@@ -1409,7 +1393,6 @@ in the second step, do not supply the `EXTERNAL_URL` value.
    gitlab_rails['auto_migrate'] = false
 
    # Configure the Consul agent
-   consul['enable'] = true
    ## Enable service discovery for Prometheus
    consul['monitoring_service_discovery'] =  true
 
@@ -1435,7 +1418,11 @@ in the second step, do not supply the `EXTERNAL_URL` value.
    # END user configuration
    ```
 
+1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
+   the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
+
 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
+
 1. Follow the [post configuration](#praefect-postgresql-post-configuration).
 
 <div align="right">
@@ -1533,19 +1520,18 @@ To configure the Praefect nodes, on each one:
 1. Edit the `/etc/gitlab/gitlab.rb` file to configure Praefect:
 
    ```ruby
-   # Avoid running unnecessary services on the Gitaly server
+   # Avoid running unnecessary services on the Praefect server
+   gitaly['enable'] = false
    postgresql['enable'] = false
    redis['enable'] = false
-   nginx['enable'] = false
    puma['enable'] = false
-   unicorn['enable'] = false
    sidekiq['enable'] = false
    gitlab_workhorse['enable'] = false
-   grafana['enable'] = false
-
-   # If you run a separate monitoring node you can disable these services
-   alertmanager['enable'] = false
    prometheus['enable'] = false
+   alertmanager['enable'] = false
+   grafana['enable'] = false
+   gitlab_exporter['enable'] = false
+   nginx['enable'] = false
 
    # Praefect Configuration
    praefect['enable'] = true
@@ -1583,19 +1569,20 @@ To configure the Praefect nodes, on each one:
    # server ('praefect') and in git_data_dirs on Gitaly nodes ('gitaly-1')
    praefect['virtual_storages'] = {
      'default' => {
-       'gitaly-1' => {
-         'address' => 'tcp://10.6.0.91:8075',
-         'token'   => '<praefect_internal_token>',
-         'primary' => true
-       },
-       'gitaly-2' => {
-         'address' => 'tcp://10.6.0.92:8075',
-         'token'   => '<praefect_internal_token>'
-       },
-       'gitaly-3' => {
-         'address' => 'tcp://10.6.0.93:8075',
-         'token'   => '<praefect_internal_token>'
-       },
+       'nodes' => {
+         'gitaly-1' => {
+           'address' => 'tcp://10.6.0.91:8075',
+           'token'   => '<praefect_internal_token>'
+         },
+         'gitaly-2' => {
+           'address' => 'tcp://10.6.0.92:8075',
+           'token'   => '<praefect_internal_token>'
+         },
+         'gitaly-3' => {
+           'address' => 'tcp://10.6.0.93:8075',
+           'token'   => '<praefect_internal_token>'
+         },
+       }
      }
    }
 
@@ -1612,11 +1599,25 @@ To configure the Praefect nodes, on each one:
    # END user configuration
    ```
 
-   1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and
-      then replace the file of the same name on this server. If that file isn't on
-      this server, add the file from your Consul server to this server.
+1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
+   the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
+
+1. Praefect requires to run some database migrations, much like the main GitLab application. For this
+   you should select **one Praefect node only to run the migrations**, AKA the _Deploy Node_. This node
+   must be configured first before the others as follows:
+   
+   1. In the `/etc/gitlab/gitlab.rb` file, change the `praefect['auto_migrate']` setting value from `false` to `true`
+
+   1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run:
+
+   ```shell
+   sudo touch /etc/gitlab/skip-auto-reconfigure
+   ```
+   
+   1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect and
+      to run the Praefect database migrations.
 
-   1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
+1. On all other Praefect nodes, [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
 
 ### Configure Gitaly
 
@@ -1660,21 +1661,17 @@ On each node:
    storage paths, enable the network listener, and to configure the token:
 
    ```ruby
-   # /etc/gitlab/gitlab.rb
-
    # Avoid running unnecessary services on the Gitaly server
    postgresql['enable'] = false
    redis['enable'] = false
-   nginx['enable'] = false
    puma['enable'] = false
-   unicorn['enable'] = false
    sidekiq['enable'] = false
    gitlab_workhorse['enable'] = false
-   grafana['enable'] = false
-
-   # If you run a separate monitoring node you can disable these services
-   alertmanager['enable'] = false
    prometheus['enable'] = false
+   alertmanager['enable'] = false
+   grafana['enable'] = false
+   gitlab_exporter['enable'] = false
+   nginx['enable'] = false
 
    # Prevent database migrations from running on upgrade automatically
    gitlab_rails['auto_migrate'] = false
@@ -1682,9 +1679,11 @@ On each node:
    # Configure the gitlab-shell API callback URL. Without this, `git push` will
    # fail. This can be your 'front door' GitLab URL or an internal load
    # balancer.
-   # Don't forget to copy `/etc/gitlab/gitlab-secrets.json` from web server to Gitaly server.
    gitlab_rails['internal_api_url'] = 'https://gitlab.example.com'
 
+   # Gitaly 
+   gitaly['enable'] = true
+
    # Make Gitaly accept connections on all network interfaces. You must use
    # firewalls to restrict access to this address/port.
    # Comment out following line if you only want to support TLS connections
@@ -1726,9 +1725,8 @@ On each node:
      })
      ```
 
-1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and
-   then replace the file of the same name on this server. If that file isn't on
-   this server, add the file from your Consul server to this server.
+1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
+   the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
 
 1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
 
@@ -1835,28 +1833,19 @@ To configure the Sidekiq nodes, on each one:
 1. Open `/etc/gitlab/gitlab.rb` with your editor:
 
    ```ruby
-   ########################################
-   #####        Services Disabled       ###
-   ########################################
-
-   nginx['enable'] = false
-   grafana['enable'] = false
-   prometheus['enable'] = false
-   alertmanager['enable'] = false
+   # Avoid running unnecessary services on the Sidekiq server
    gitaly['enable'] = false
-   gitlab_workhorse['enable'] = false
-   nginx['enable'] = false
-   puma['enable'] = false
-   postgres_exporter['enable'] = false
    postgresql['enable'] = false
    redis['enable'] = false
-   redis_exporter['enable'] = false
+   puma['enable'] = false
+   gitlab_workhorse['enable'] = false
+   prometheus['enable'] = false
+   alertmanager['enable'] = false
+   grafana['enable'] = false
    gitlab_exporter['enable'] = false
+   nginx['enable'] = false
 
-   ########################################
-   ####              Redis              ###
-   ########################################
-
+   # Redis
    ## Redis connection details
    ## First cluster that will host the cache
    gitlab_rails['redis_cache_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER>@gitlab-redis-cache'
@@ -1888,13 +1877,10 @@ To configure the Sidekiq nodes, on each one:
      {host: '10.6.0.83', port: 26379},
    ]
 
-   #######################################
-   ###              Gitaly             ###
-   #######################################
-
-   # git_data_dirs get configured for the Praefect virtual storage
-   # Address is Internal Load Balancer for Praefect
-   # Token is praefect_external_token
+   # Gitaly Cluster
+   ## git_data_dirs get configured for the Praefect virtual storage
+   ## Address is Internal Load Balancer for Praefect
+   ## Token is praefect_external_token
    git_data_dirs({
      "default" => {
        "gitaly_address" => "tcp://10.6.0.40:2305", # internal load balancer IP
@@ -1902,20 +1888,17 @@ To configure the Sidekiq nodes, on each one:
      }
    })
 
-   #######################################
-   ###            Postgres             ###
-   #######################################
+   # PostgreSQL
    gitlab_rails['db_host'] = '10.6.0.40' # internal load balancer IP
    gitlab_rails['db_port'] = 6432
    gitlab_rails['db_password'] = '<postgresql_user_password>'
    gitlab_rails['db_adapter'] = 'postgresql'
    gitlab_rails['db_encoding'] = 'unicode'
-   # Prevent database migrations from running on upgrade automatically
+   ## Prevent database migrations from running on upgrade automatically
    gitlab_rails['auto_migrate'] = false
 
-   #######################################
-   ###      Sidekiq configuration      ###
-   #######################################
+   # Sidekiq
+   sidekiqp['enable'] = true
    sidekiq['listen_address'] = "0.0.0.0"
 
    # Set number of Sidekiq queue processes to the same number as available CPUs
@@ -1924,9 +1907,7 @@ To configure the Sidekiq nodes, on each one:
    # Set number of Sidekiq threads per queue process to the recommend number of 10
    sidekiq['max_concurrency'] = 10
 
-   #######################################
-   ###     Monitoring configuration    ###
-   #######################################
+   # Monitoring
    consul['enable'] = true
    consul['monitoring_service_discovery'] =  true
 
@@ -1934,18 +1915,15 @@ To configure the Sidekiq nodes, on each one:
       retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13)
    }
 
-   # Set the network addresses that the exporters will listen on
+   ## Set the network addresses that the exporters will listen on
    node_exporter['listen_address'] = '0.0.0.0:9100'
 
-   # Rails Status for prometheus
+   ## Add the monitoring node's IP address to the monitoring whitelist
    gitlab_rails['monitoring_whitelist'] = ['10.6.0.151/32', '127.0.0.0/8']
 
-   #############################
-   ###     Object storage    ###
-   #############################
-
-   # This is an example for configuring Object Storage on GCP
-   # Replace this config with your chosen Object Storage provider as desired
+   # Object Storage
+   ## This is an example for configuring Object Storage on GCP
+   ## Replace this config with your chosen Object Storage provider as desired
    gitlab_rails['object_store']['connection'] = {
      'provider' => 'Google',
      'google_project' => '<gcp-project-name>',
@@ -1958,11 +1936,26 @@ To configure the Sidekiq nodes, on each one:
    gitlab_rails['object_store']['objects']['packages']['bucket'] = "<gcp-packages-bucket-name>"
    gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = "<gcp-dependency-proxy-bucket-name>"
    gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = "<gcp-terraform-state-bucket-name>"
+
+   gitlab_rails['backup_upload_connection'] = {
+     'provider' => 'Google',
+     'google_project' => '<gcp-project-name>',
+     'google_json_key_location' => '<path-to-gcp-service-account-key>'
+   }
+   gitlab_rails['backup_upload_remote_directory'] = "<gcp-backups-state-bucket-name>"
+   ```
+
+1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
+   the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
+
+1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run:
+
+   ```shell
+   sudo touch /etc/gitlab/skip-auto-reconfigure
    ```
 
-1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace
-   the file of the same name on this server. If that file is not on this server,
-   add the file from your Consul server to this server.
+   Only a single designated node should handle migrations as detailed in the
+   [GitLab Rails post-configuration](#gitlab-rails-post-configuration) section.
 
 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
 
@@ -1993,9 +1986,6 @@ On each node perform the following:
 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab
    package of your choice. Be sure to follow _only_ installation steps 1 and 2
    on the page.
-1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace
-   the file of the same name on this server. If that file is not on this server,
-   add the file from your Consul server to this server.
 
 1. Edit `/etc/gitlab/gitlab.rb` and use the following configuration.
    To maintain uniformity of links across nodes, the `external_url`
@@ -2017,7 +2007,7 @@ On each node perform the following:
    })
 
    ## Disable components that will not be on the GitLab application server
-   roles ['application_role']
+   roles(['application_role'])
    gitaly['enable'] = false
    nginx['enable'] = true
    sidekiq['enable'] = false
@@ -2090,9 +2080,15 @@ On each node perform the following:
    gitlab_rails['object_store']['objects']['packages']['bucket'] = "<gcp-packages-bucket-name>"
    gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = "<gcp-dependency-proxy-bucket-name>"
    gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = "<gcp-terraform-state-bucket-name>"
+
+   gitlab_rails['backup_upload_connection'] = {
+     'provider' => 'Google',
+     'google_project' => '<gcp-project-name>',
+     'google_json_key_location' => '<path-to-gcp-service-account-key>'
+   }
+   gitlab_rails['backup_upload_remote_directory'] = "<gcp-backups-state-bucket-name>"
    ```
 
-1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
 1. If you're using [Gitaly with TLS support](#gitaly-cluster-tls-support), make sure the
    `git_data_dirs` entry is configured with `tls` instead of `tcp`:
 
@@ -2111,6 +2107,20 @@ On each node perform the following:
       sudo cp cert.pem /etc/gitlab/trusted-certs/
       ```
 
+1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
+   the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
+
+1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run:
+
+   ```shell
+   sudo touch /etc/gitlab/skip-auto-reconfigure
+   ```
+
+   Only a single designated node should handle migrations as detailed in the
+   [GitLab Rails post-configuration](#gitlab-rails-post-configuration) section.
+
+1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
+
 1. If you're [using NFS](#configure-nfs-optional):
    1. If necessary, install the NFS client utility packages using the following
       commands:
@@ -2150,7 +2160,8 @@ On each node perform the following:
       registry['gid'] = 9002
       ```
 
-1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
+   1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
+
 1. Confirm the node can connect to Gitaly:
 
    ```shell
@@ -2214,55 +2225,34 @@ To configure the Monitoring node:
 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab
    package of your choice. Be sure to follow _only_ installation steps 1 and 2
    on the page.
-1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace
-   the file of the same name on this server. If that file is not on this server,
-   add the file from your Consul server to this server.
 
 1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
 
    ```ruby
-   external_url 'http://gitlab.example.com'
+   roles(['monitoring_role', 'consul_role'])
 
-   # Disable all other services
-   alertmanager['enable'] = false
-   gitaly['enable'] = false
-   gitlab_exporter['enable'] = false
-   gitlab_workhorse['enable'] = false
-   nginx['enable'] = true
-   postgres_exporter['enable'] = false
-   postgresql['enable'] = false
-   redis['enable'] = false
-   redis_exporter['enable'] = false
-   sidekiq['enable'] = false
-   puma['enable'] = false
-   unicorn['enable'] = false
-   node_exporter['enable'] = false
-   gitlab_exporter['enable'] = false
+   external_url 'http://gitlab.example.com'
 
-   # Enable Prometheus
-   prometheus['enable'] = true
+   # Prometheus
    prometheus['listen_address'] = '0.0.0.0:9090'
    prometheus['monitor_kubernetes'] = false
 
-   # Enable Login form
-   grafana['disable_login_form'] = false
-
-   # Enable Grafana
-   grafana['enable'] = true
+   # Grafana
    grafana['admin_password'] = '<grafana_password>'
+   grafana['disable_login_form'] = false
 
    # Enable service discovery for Prometheus
-   consul['enable'] = true
    consul['monitoring_service_discovery'] =  true
    consul['configuration'] = {
       retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13)
    }
 
-   # Prevent database migrations from running on upgrade automatically
-   gitlab_rails['auto_migrate'] = false
+   # Nginx - For Grafana access
+   nginx['enable'] = true
    ```
 
-1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
+1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
+
 1. In the GitLab UI, set `admin/application_settings/metrics_and_profiling` > Metrics - Grafana to `/-/grafana` to
 `http[s]://<MONITOR NODE>/-/grafana`
 
@@ -2395,35 +2385,46 @@ time use Google Cloud’s Kubernetes Engine (GKE) and associated machine types,
 and CPU requirements should translate to most other providers. We hope to update this in the
 future with further specific cloud provider details.
 
-| Service                                               | Nodes | Configuration           | GCP              | Allocatable CPUs and Memory |
-|-------------------------------------------------------|-------|-------------------------|------------------|-----------------------------|
-| Webservice                                            | 4     | 32 vCPU, 28.8 GB memory | `n1-standard-32` | 127.5 vCPU, 118 GB memory   |
-| Sidekiq                                               | 4     | 4 vCPU, 15 GB memory    | `n1-standard-4`  | 15.5 vCPU, 50 GB memory     |
-| Supporting services such as NGINX, Prometheus, etc... | 2     | 4 vCPU, 15 GB memory    | `n1-standard-4`  | 7.75 vCPU, 25 GB memory     |
+| Service                                               | Nodes(1) | Configuration           | GCP              | Allocatable CPUs and Memory |
+|-------------------------------------------------------|----------|-------------------------|------------------|-----------------------------|
+| Webservice                                            | 4        | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | 127.5 vCPU, 118 GB memory   |
+| Sidekiq                                               | 4        | 4 vCPU, 15 GB memory    | `n1-standard-4`  | 15.5 vCPU, 50 GB memory     |
+| Supporting services such as NGINX, Prometheus, etc.   | 2        | 4 vCPU, 15 GB memory    | `n1-standard-4`  | 7.75 vCPU, 25 GB memory     |
+
+<!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix -->
+<!-- markdownlint-disable MD029 -->
+1. Nodes configuration is shown as it is forced to ensure pod vcpu / memory ratios and avoid scaling during **performance testing**.
+   In production deployments there is no need to assign pods to nodes. A minimum of three nodes in three different availability zones is strongly recommended to align with resilient cloud architecture practices.
+<!-- markdownlint-enable MD029 -->
 
 Next are the backend components that run on static compute VMs via Omnibus (or External PaaS
 services where applicable):
 
 | Service                                    | Nodes | Configuration           | GCP              |
 |--------------------------------------------|-------|-------------------------|------------------|
-| Consul*                                    | 3     | 2 vCPU, 1.8 GB memory   | `n1-highcpu-2`   |
-| PostgreSQL*                                | 3     | 8 vCPU, 30 GB memory    | `n1-standard-8`  |
-| PgBouncer*                                 | 3     | 2 vCPU, 1.8 GB memory   | `n1-highcpu-2`   |
-| Internal load balancing node               | 1     | 2 vCPU, 1.8 GB memory   | `n1-highcpu-2`   |
-| Redis - Cache**                            | 3     | 4 vCPU, 15 GB memory    | `n1-standard-4`  |
-| Redis - Queues / Shared State**            | 3     | 4 vCPU, 15 GB memory    | `n1-standard-4`  |
-| Redis Sentinel - Cache**                   | 3     | 1 vCPU, 3.75 GB memory   | `n1-standard-1`  |
-| Redis Sentinel - Queues / Shared State**   | 3     | 1 vCPU, 3.75 GB memory   | `n1-standard-1`  |
+| Consul(1)                                  | 3     | 2 vCPU, 1.8 GB memory   | `n1-highcpu-2`   |
+| PostgreSQL(1)                              | 3     | 8 vCPU, 30 GB memory    | `n1-standard-8`  |
+| PgBouncer(1)                               | 3     | 2 vCPU, 1.8 GB memory   | `n1-highcpu-2`   |
+| Internal load balancing node(3)            | 1     | 2 vCPU, 1.8 GB memory   | `n1-highcpu-2`   |
+| Redis - Cache(2)                           | 3     | 4 vCPU, 15 GB memory    | `n1-standard-4`  |
+| Redis - Queues / Shared State(2)           | 3     | 4 vCPU, 15 GB memory    | `n1-standard-4`  |
+| Redis Sentinel - Cache(2)                  | 3     | 1 vCPU, 3.75 GB memory   | `n1-standard-1`  |
+| Redis Sentinel - Queues / Shared State(2)  | 3     | 1 vCPU, 3.75 GB memory   | `n1-standard-1`  |
 | Gitaly                                     | 3     | 16 vCPU, 60 GB memory   | `n1-standard-16` |
 | Praefect                                   | 3     | 2 vCPU, 1.8 GB memory   | `n1-highcpu-2`   |
-| Praefect PostgreSQL*                       | 1+    | 2 vCPU, 1.8 GB memory   | `n1-highcpu-2`   |
-| Object storage                             | n/a   | n/a                     | n/a              |
+| Praefect PostgreSQL(1)                     | 1+    | 2 vCPU, 1.8 GB memory   | `n1-highcpu-2`   |
+| Object storage(4)                          | n/a   | n/a                     | n/a              |
+
+<!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix -->
+<!-- markdownlint-disable MD029 -->
+1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work, however Azure Database for PostgreSQL is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to performance issues. Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery.
+2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work.
+3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work.
+4. Should be run on reputable third party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work.
+<!-- markdownlint-enable MD029 -->
 
 NOTE:
-Components marked with * can be optionally run on reputable
-third party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work.
-Components marked with ** can be optionally run on reputable
-third party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work.
+For all PaaS solutions that involve configuring instances, it is strongly recommended to implement a minimum of three nodes in three different availability zones to align with resilient cloud architecture practices.
 
 ```plantuml
 @startuml 10k
diff --git a/doc/administration/reference_architectures/25k_users.md b/doc/administration/reference_architectures/25k_users.md
index 129386d6ce5..1f72c45c2b7 100644
--- a/doc/administration/reference_architectures/25k_users.md
+++ b/doc/administration/reference_architectures/25k_users.md
@@ -17,29 +17,34 @@ full list of reference architectures, see
 
 | Service                                  | Nodes       | Configuration           | GCP              | AWS          | Azure     |
 |------------------------------------------|-------------|-------------------------|------------------|--------------|-----------|
-| External load balancing node             | 1           | 4 vCPU, 3.6 GB memory   | `n1-highcpu-4`   | `c5.xlarge`  | `F4s v2`  |
-| Consul*                                  | 3           | 2 vCPU, 1.8 GB memory   | `n1-highcpu-2`   | `c5.large`   | `F2s v2`  |
-| PostgreSQL*                              | 3           | 16 vCPU, 60 GB memory   | `n1-standard-1`  | `m5.4xlarge` | `D16s v3` |
-| PgBouncer*                               | 3           | 2 vCPU, 1.8 GB memory   | `n1-highcpu-2`   | `c5.large`   | `F2s v2`  |
-| Internal load balancing node             | 1           | 4 vCPU, 3.6GB memory    | `n1-highcpu-4`   | `c5.large`   | `F2s v2`  |
-| Redis - Cache**                          | 3           | 4 vCPU, 15 GB memory    | `n1-standard-4`  | `m5.xlarge`  | `D4s v3`  |
-| Redis - Queues / Shared State**          | 3           | 4 vCPU, 15 GB memory    | `n1-standard-4`  | `m5.xlarge`  | `D4s v3`  |
-| Redis Sentinel - Cache**                 | 3           | 1 vCPU, 3.75 GB memory  | `n1-standard-1`  | `c5.large`   | `A1 v2`   |
-| Redis Sentinel - Queues / Shared State** | 3           | 1 vCPU, 3.75 GB memory  | `n1-standard-1`  | `c5.large`   | `A1 v2`   |
+| External load balancing node(3)          | 1           | 4 vCPU, 3.6 GB memory   | `n1-highcpu-4`   | `c5.xlarge`  | `F4s v2`  |
+| Consul(1)                                | 3           | 2 vCPU, 1.8 GB memory   | `n1-highcpu-2`   | `c5.large`   | `F2s v2`  |
+| PostgreSQL(1)                            | 3           | 16 vCPU, 60 GB memory   | `n1-standard-1`  | `m5.4xlarge` | `D16s v3` |
+| PgBouncer(1)                             | 3           | 2 vCPU, 1.8 GB memory   | `n1-highcpu-2`   | `c5.large`   | `F2s v2`  |
+| Internal load balancing node(3)          | 1           | 4 vCPU, 3.6GB memory    | `n1-highcpu-4`   | `c5.large`   | `F2s v2`  |
+| Redis - Cache(2)                         | 3           | 4 vCPU, 15 GB memory    | `n1-standard-4`  | `m5.xlarge`  | `D4s v3`  |
+| Redis - Queues / Shared State(2)         | 3           | 4 vCPU, 15 GB memory    | `n1-standard-4`  | `m5.xlarge`  | `D4s v3`  |
+| Redis Sentinel - Cache(2)                | 3           | 1 vCPU, 3.75 GB memory  | `n1-standard-1`  | `c5.large`   | `A1 v2`   |
+| Redis Sentinel - Queues / Shared State(2)| 3           | 1 vCPU, 3.75 GB memory  | `n1-standard-1`  | `c5.large`   | `A1 v2`   |
 | Gitaly                                   | 3           | 32 vCPU, 120 GB memory  | `n1-standard-32` | `m5.8xlarge` | `D32s v3` |
 | Praefect                                 | 3           | 4 vCPU, 3.6 GB memory   | `n1-highcpu-4`   | `c5.xlarge`  | `F4s v2`  |
-| Praefect PostgreSQL*                     | 1+          | 2 vCPU, 1.8 GB memory   | `n1-highcpu-2`   | `c5.large`   | `F2s v2`  |
+| Praefect PostgreSQL(1)                   | 1+          | 2 vCPU, 1.8 GB memory   | `n1-highcpu-2`   | `c5.large`   | `F2s v2`  |
 | Sidekiq                                  | 4           | 4 vCPU, 15 GB memory    | `n1-standard-4`  | `m5.xlarge`  | `D4s v3`  |
 | GitLab Rails                             | 5           | 32 vCPU, 28.8 GB memory | `n1-highcpu-32`  | `c5.9xlarge` | `F32s v2` |
 | Monitoring node                          | 1           | 4 vCPU, 3.6 GB memory   | `n1-highcpu-4`   | `c5.xlarge`  | `F4s v2`  |
-| Object storage                           | n/a         | n/a                     | n/a              | n/a          | n/a       |
-| NFS server                               | 1           | 4 vCPU, 3.6 GB memory   | `n1-highcpu-4`   | `c5.xlarge`  | `F4s v2`  |
+| Object storage(4)                        | n/a         | n/a                     | n/a              | n/a          | n/a       |
+| NFS server (optional, not recommended)   | 1           | 4 vCPU, 3.6 GB memory   | `n1-highcpu-4`   | `c5.xlarge`  | `F4s v2`  |
+
+<!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix -->
+<!-- markdownlint-disable MD029 -->
+1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work, however Azure Database for PostgreSQL is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to performance issues. Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery.
+2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work.
+3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work.
+4. Should be run on reputable third party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work.
+<!-- markdownlint-enable MD029 -->
 
 NOTE:
-Components marked with * can be optionally run on reputable
-third party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work.
-Components marked with ** can be optionally run on reputable
-third party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work.
+For all PaaS solutions that involve configuring instances, it is strongly recommended to implement a minimum of three nodes in three different availability zones to align with resilient cloud architecture practices.
 
 ```plantuml
 @startuml 25k
@@ -157,7 +162,7 @@ To set up GitLab and its components to accommodate up to 25,000 users:
    provides access to the Git repositories.
 1. [Configure Sidekiq](#configure-sidekiq).
 1. [Configure the main GitLab Rails application](#configure-gitlab-rails)
-   to run Puma/Unicorn, Workhorse, GitLab Shell, and to serve all frontend
+   to run Puma, Workhorse, GitLab Shell, and to serve all frontend
    requests (which include UI, API, and Git over HTTP/SSH).
 1. [Configure Prometheus](#configure-prometheus) to monitor your GitLab
    environment.
@@ -413,11 +418,6 @@ The following IPs will be used as an example:
 - `10.6.0.12`: Consul 2
 - `10.6.0.13`: Consul 3
 
-NOTE:
-The configuration processes for the other servers in your reference architecture will
-use the `/etc/gitlab/gitlab-secrets.json` file from your Consul server to connect
-with the other servers.
-
 To configure Consul:
 
 1. SSH in to the server that will host Consul.
@@ -428,10 +428,9 @@ To configure Consul:
 1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
 
    ```ruby
-   roles ['consul_role']
+   roles(['consul_role'])
 
    ## Enable service discovery for Prometheus
-   consul['enable'] = true
    consul['monitoring_service_discovery'] =  true
 
    ## The IPs of the Consul server nodes
@@ -448,7 +447,11 @@ To configure Consul:
    gitlab_rails['auto_migrate'] = false
    ```
 
+1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
+   the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
+
 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
+
 1. Go through the steps again for all the other Consul nodes, and
    make sure you set up the correct IPs.
 
@@ -540,6 +543,15 @@ in the second step, do not supply the `EXTERNAL_URL` value.
    sudo gitlab-ctl pg-password-md5 pgbouncer
    ```
 
+1. Generate a password hash for the PostgreSQL replication username/password pair. This assumes you will use the default
+   username of `gitlab_replicator` (recommended). The command will request a password
+   and a confirmation. Use the value that is output by this command in the next step
+   as the value of `<postgresql_replication_password_hash>`:
+
+   ```shell
+   sudo gitlab-ctl pg-password-md5 gitlab_replicator
+   ```
+
 1. Generate a password hash for the Consul database username/password pair. This assumes you will use the default
    username of `gitlab-consul` (recommended). The command will request a password
    and confirmation. Use the value that is output by this command in the next
@@ -552,19 +564,21 @@ in the second step, do not supply the `EXTERNAL_URL` value.
 1. On every database node, edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section:
 
    ```ruby
-   # Disable all components except PostgreSQL, Patroni, and Consul
-   roles ['postgres_role']
+   # Disable all components except Patroni and Consul
+   roles(['patroni_role'])
 
    # PostgreSQL configuration
    postgresql['listen_address'] = '0.0.0.0'
 
-   # Enable Patroni
-   patroni['enable'] = true
-   # Set `max_wal_senders` to one more than the number of database nodes in the cluster.
+   # Sets `max_replication_slots` to double the number of database nodes.
+   # Patroni uses one extra slot per node when initiating the replication.
+   patroni['postgresql']['max_replication_slots'] = 8
+
+   # Set `max_wal_senders` to one more than the number of replication slots in the cluster.
    # This is used to prevent replication from using up all of the
    # available database connections.
-   patroni['postgresql']['max_wal_senders'] = 4
-   patroni['postgresql']['max_replication_slots'] = 4
+   patroni['postgresql']['max_wal_senders'] = 9
+
    # Incoming recommended value for max connections is 500. See https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5691.
    patroni['postgresql']['max_connections'] = 500
 
@@ -572,7 +586,6 @@ in the second step, do not supply the `EXTERNAL_URL` value.
    gitlab_rails['auto_migrate'] = false
 
    # Configure the Consul agent
-   consul['enable'] = true
    consul['services'] = %w(postgresql)
    ## Enable service discovery for Prometheus
    consul['monitoring_service_discovery'] =  true
@@ -582,6 +595,8 @@ in the second step, do not supply the `EXTERNAL_URL` value.
    #
    # Replace PGBOUNCER_PASSWORD_HASH with a generated md5 value
    postgresql['pgbouncer_user_password'] = '<pgbouncer_password_hash>'
+   # Replace POSTGRESQL_REPLICATION_PASSWORD_HASH with a generated md5 value
+   postgresql['sql_replication_password'] = '<postgresql_replication_password_hash>'
    # Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value
    postgresql['sql_user_password'] = '<postgresql_password_hash>'
 
@@ -605,9 +620,8 @@ PostgreSQL, with Patroni managing its failover, will default to use `pg_rewind`
 Like most failover handling methods, this has a small chance of leading to data loss.
 Learn more about the various [Patroni replication methods](../postgresql/replication_and_failover.md#selecting-the-appropriate-patroni-replication-method).
 
-1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace
-   the file of the same name on this server. If that file is not on this server,
-   add the file from your Consul server to this server.
+1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
+   the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
 
 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
 
@@ -622,21 +636,7 @@ are supported and can be added if needed.
 
 #### PostgreSQL post-configuration
 
-SSH in to the **primary node**:
-
-1. Open a database prompt:
-
-   ```shell
-   gitlab-psql -d gitlabhq_production
-   ```
-
-1. Make sure the `pg_trgm` extension is enabled (it might already be):
-
-   ```shell
-   CREATE EXTENSION pg_trgm;
-   ```
-
-1. Exit the database prompt by typing `\q` and Enter.
+SSH in to any of the Patroni nodes on the **primary site**:
 
 1. Check the status of the leader and cluster:
 
@@ -678,7 +678,7 @@ The following IPs will be used as an example:
 
    ```ruby
    # Disable all components except Pgbouncer and Consul agent
-   roles ['pgbouncer_role']
+   roles(['pgbouncer_role'])
 
    # Configure PgBouncer
    pgbouncer['admin_users'] = %w(pgbouncer gitlab-consul)
@@ -695,7 +695,6 @@ The following IPs will be used as an example:
 
    # Configure Consul agent
    consul['watchers'] = %w(postgresql)
-   consul['enable'] = true
    consul['configuration'] = {
    retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13)
    }
@@ -707,9 +706,8 @@ The following IPs will be used as an example:
    node_exporter['listen_address'] = '0.0.0.0:9100'
    ```
 
-1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace
-   the file of the same name on this server. If that file is not on this server,
-   add the file from your Consul server to this server.
+1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
+   the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
 
 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
 
@@ -828,8 +826,8 @@ a node and change its status from primary to replica (and vice versa).
 1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
 
    ```ruby
-   # Specify server role as 'redis_master_role'
-   roles ['redis_master_role']
+   # Specify server role as 'redis_master_role' and enable Consul agent
+   roles(['redis_master_role', 'consul_role']
 
    # IP address pointing to a local IP that the other machines can reach to.
    # You can also set bind to '0.0.0.0' which listen in all interfaces.
@@ -851,7 +849,6 @@ a node and change its status from primary to replica (and vice versa).
    redis['maxmemory_samples'] = 5
 
    ## Enable service discovery for Prometheus
-   consul['enable'] = true
    consul['monitoring_service_discovery'] =  true
 
    ## The IPs of the Consul server nodes
@@ -863,19 +860,22 @@ a node and change its status from primary to replica (and vice versa).
    # Set the network addresses that the exporters will listen on
    node_exporter['listen_address'] = '0.0.0.0:9100'
    redis_exporter['listen_address'] = '0.0.0.0:9121'
+   redis_exporter['flags'] = {
+        'redis.addr' => 'redis://10.6.0.51:6379',
+        'redis.password' => 'redis-password-goes-here',
+   }   
 
    # Prevent database migrations from running on upgrade automatically
    gitlab_rails['auto_migrate'] = false
    ```
 
-1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace
-   the file of the same name on this server. If that file is not on this server,
-   add the file from your Consul server to this server.
+1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
+   the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
 
 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
 
 You can specify multiple roles, like sentinel and Redis, as:
-`roles ['redis_sentinel_role', 'redis_master_role']`. Read more about
+`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about
 [roles](https://docs.gitlab.com/omnibus/roles/).
 
 #### Configure the replica Redis Cache nodes
@@ -888,8 +888,8 @@ You can specify multiple roles, like sentinel and Redis, as:
 1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
 
    ```ruby
-   # Specify server role as 'redis_replica_role'
-   roles ['redis_replica_role']
+   # Specify server role as 'redis_replica_role' and enable Consul agent
+   roles(['redis_replica_role', 'consul_role']
 
    # IP address pointing to a local IP that the other machines can reach to.
    # You can also set bind to '0.0.0.0' which listen in all interfaces.
@@ -918,7 +918,6 @@ You can specify multiple roles, like sentinel and Redis, as:
    redis['maxmemory_samples'] = 5
 
    ## Enable service discovery for Prometheus
-   consul['enable'] = true
    consul['monitoring_service_discovery'] =  true
 
    ## The IPs of the Consul server nodes
@@ -930,21 +929,25 @@ You can specify multiple roles, like sentinel and Redis, as:
    # Set the network addresses that the exporters will listen on
    node_exporter['listen_address'] = '0.0.0.0:9100'
    redis_exporter['listen_address'] = '0.0.0.0:9121'
+   redis_exporter['flags'] = {
+        'redis.addr' => 'redis://10.6.0.52:6379',
+        'redis.password' => 'redis-password-goes-here',
+   }
 
    # Prevent database migrations from running on upgrade automatically
    gitlab_rails['auto_migrate'] = false
    ```
 
-1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace
-   the file of the same name on this server. If that file is not on this server,
-   add the file from your Consul server to this server.
+1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
+   the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
 
 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
+
 1. Go through the steps again for all the other replica nodes, and
    make sure to set up the IPs correctly.
 
 You can specify multiple roles, like sentinel and Redis, as:
-`roles ['redis_sentinel_role', 'redis_master_role']`. Read more about
+`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about
 [roles](https://docs.gitlab.com/omnibus/roles/).
 
 These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after
@@ -986,7 +989,7 @@ To configure the Sentinel Cache server:
 1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
 
    ```ruby
-   roles ['redis_sentinel_role']
+   roles(['redis_sentinel_role', 'consul_role'])
 
    ## Must be the same in every sentinel node
    redis['master_name'] = 'gitlab-redis-cache'
@@ -1050,7 +1053,6 @@ To configure the Sentinel Cache server:
    #sentinel['failover_timeout'] = 60000
 
    ## Enable service discovery for Prometheus
-   consul['enable'] = true
    consul['monitoring_service_discovery'] =  true
 
    ## The IPs of the Consul server nodes
@@ -1067,11 +1069,11 @@ To configure the Sentinel Cache server:
    gitlab_rails['auto_migrate'] = false
    ```
 
-1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace
-   the file of the same name on this server. If that file is not on this server,
-   add the file from your Consul server to this server.
+1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
+   the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
 
 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
+
 1. Go through the steps again for all the other Consul/Sentinel nodes, and
    make sure you set up the correct IPs.
 
@@ -1099,8 +1101,8 @@ a node and change its status from primary to replica (and vice versa).
 1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
 
    ```ruby
-   # Specify server role as 'redis_master_role'
-   roles ['redis_master_role']
+   # Specify server role as 'redis_master_role' and enable Consul agent
+   roles(['redis_master_role', 'consul_role'])
 
    # IP address pointing to a local IP that the other machines can reach to.
    # You can also set bind to '0.0.0.0' which listen in all interfaces.
@@ -1116,7 +1118,6 @@ a node and change its status from primary to replica (and vice versa).
    redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER'
 
    ## Enable service discovery for Prometheus
-   consul['enable'] = true
    consul['monitoring_service_discovery'] =  true
 
    ## The IPs of the Consul server nodes
@@ -1133,14 +1134,13 @@ a node and change its status from primary to replica (and vice versa).
    gitlab_rails['auto_migrate'] = false
    ```
 
-1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace
-   the file of the same name on this server. If that file is not on this server,
-   add the file from your Consul server to this server.
+1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
+   the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
 
 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
 
 You can specify multiple roles, like sentinel and Redis, as:
-`roles ['redis_sentinel_role', 'redis_master_role']`. Read more about
+`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about
 [roles](https://docs.gitlab.com/omnibus/roles/).
 
 #### Configure the replica Redis Queues nodes
@@ -1153,8 +1153,8 @@ You can specify multiple roles, like sentinel and Redis, as:
 1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
 
    ```ruby
-   # Specify server role as 'redis_replica_role'
-   roles ['redis_replica_role']
+   # Specify server role as 'redis_replica_role' and enable Consul agent
+   roles(['redis_replica_role', 'consul_role'])
 
    # IP address pointing to a local IP that the other machines can reach to.
    # You can also set bind to '0.0.0.0' which listen in all interfaces.
@@ -1177,7 +1177,6 @@ You can specify multiple roles, like sentinel and Redis, as:
    #redis['master_port'] = 6379
 
    ## Enable service discovery for Prometheus
-   consul['enable'] = true
    consul['monitoring_service_discovery'] =  true
 
    ## The IPs of the Consul server nodes
@@ -1189,21 +1188,25 @@ You can specify multiple roles, like sentinel and Redis, as:
    # Set the network addresses that the exporters will listen on
    node_exporter['listen_address'] = '0.0.0.0:9100'
    redis_exporter['listen_address'] = '0.0.0.0:9121'
+   redis_exporter['flags'] = {
+        'redis.addr' => 'redis://10.6.0.62:6379',
+        'redis.password' => 'redis-password-goes-here',
+   }
 
    # Prevent database migrations from running on upgrade automatically
    gitlab_rails['auto_migrate'] = false
    ```
 
-1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace
-   the file of the same name on this server. If that file is not on this server,
-   add the file from your Consul server to this server.
+1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
+   the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
 
 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
+
 1. Go through the steps again for all the other replica nodes, and
    make sure to set up the IPs correctly.
 
 You can specify multiple roles, like sentinel and Redis, as:
-`roles ['redis_sentinel_role', 'redis_master_role']`. Read more about
+`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about
 [roles](https://docs.gitlab.com/omnibus/roles/).
 
 These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after
@@ -1245,7 +1248,7 @@ To configure the Sentinel Queues server:
 1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
 
    ```ruby
-   roles ['redis_sentinel_role']
+   roles(['redis_sentinel_role', 'consul_role'])
 
    ## Must be the same in every sentinel node
    redis['master_name'] = 'gitlab-redis-persistent'
@@ -1309,7 +1312,6 @@ To configure the Sentinel Queues server:
    #sentinel['failover_timeout'] = 60000
 
    ## Enable service discovery for Prometheus
-   consul['enable'] = true
    consul['monitoring_service_discovery'] =  true
 
    ## The IPs of the Consul server nodes
@@ -1326,7 +1328,10 @@ To configure the Sentinel Queues server:
    gitlab_rails['auto_migrate'] = false
    ```
 
-1. To prevent database migrations from running on upgrade, run:
+1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
+   the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
+
+1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run:
 
    ```shell
    sudo touch /etc/gitlab/skip-auto-reconfigure
@@ -1334,11 +1339,8 @@ To configure the Sentinel Queues server:
 
    Only the primary GitLab application server should handle migrations.
 
-1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace
-   the file of the same name on this server. If that file is not on this server,
-   add the file from your Consul server to this server.
-
 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
+
 1. Go through the steps again for all the other Sentinel nodes, and
    make sure you set up the correct IPs.
 
@@ -1399,9 +1401,7 @@ in the second step, do not supply the `EXTERNAL_URL` value.
 
    ```ruby
    # Disable all components except PostgreSQL and Consul
-   roles ['postgres_role']
-   repmgr['enable'] = false
-   patroni['enable'] = false
+   roles(['postgres_role', 'consul_role'])
 
    # PostgreSQL configuration
    postgresql['listen_address'] = '0.0.0.0'
@@ -1411,7 +1411,6 @@ in the second step, do not supply the `EXTERNAL_URL` value.
    gitlab_rails['auto_migrate'] = false
 
    # Configure the Consul agent
-   consul['enable'] = true
    ## Enable service discovery for Prometheus
    consul['monitoring_service_discovery'] =  true
 
@@ -1437,7 +1436,11 @@ in the second step, do not supply the `EXTERNAL_URL` value.
    # END user configuration
    ```
 
+1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
+   the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
+
 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
+
 1. Follow the [post configuration](#praefect-postgresql-post-configuration).
 
 <div align="right">
@@ -1535,19 +1538,18 @@ To configure the Praefect nodes, on each one:
 1. Edit the `/etc/gitlab/gitlab.rb` file to configure Praefect:
 
    ```ruby
-   # Avoid running unnecessary services on the Gitaly server
+   # Avoid running unnecessary services on the Praefect server
+   gitaly['enable'] = false
    postgresql['enable'] = false
    redis['enable'] = false
-   nginx['enable'] = false
    puma['enable'] = false
-   unicorn['enable'] = false
    sidekiq['enable'] = false
    gitlab_workhorse['enable'] = false
-   grafana['enable'] = false
-
-   # If you run a separate monitoring node you can disable these services
-   alertmanager['enable'] = false
    prometheus['enable'] = false
+   alertmanager['enable'] = false
+   grafana['enable'] = false
+   gitlab_exporter['enable'] = false
+   nginx['enable'] = false
 
    # Praefect Configuration
    praefect['enable'] = true
@@ -1585,19 +1587,20 @@ To configure the Praefect nodes, on each one:
    # server ('praefect') and in git_data_dirs on Gitaly nodes ('gitaly-1')
    praefect['virtual_storages'] = {
      'default' => {
-       'gitaly-1' => {
-         'address' => 'tcp://10.6.0.91:8075',
-         'token'   => '<praefect_internal_token>',
-         'primary' => true
-       },
-       'gitaly-2' => {
-         'address' => 'tcp://10.6.0.92:8075',
-         'token'   => '<praefect_internal_token>'
-       },
-       'gitaly-3' => {
-         'address' => 'tcp://10.6.0.93:8075',
-         'token'   => '<praefect_internal_token>'
-       },
+       'nodes' => {
+         'gitaly-1' => {
+           'address' => 'tcp://10.6.0.91:8075',
+           'token'   => '<praefect_internal_token>'
+         },
+         'gitaly-2' => {
+           'address' => 'tcp://10.6.0.92:8075',
+           'token'   => '<praefect_internal_token>'
+         },
+         'gitaly-3' => {
+           'address' => 'tcp://10.6.0.93:8075',
+           'token'   => '<praefect_internal_token>'
+         },
+       }
      }
    }
 
@@ -1614,11 +1617,25 @@ To configure the Praefect nodes, on each one:
    # END user configuration
    ```
 
-   1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and
-      then replace the file of the same name on this server. If that file isn't on
-      this server, add the file from your Consul server to this server.
+1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
+the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
+
+1. Praefect requires to run some database migrations, much like the main GitLab application. For this
+   you should select **one Praefect node only to run the migrations**, AKA the _Deploy Node_. This node
+   must be configured first before the others as follows:
+   
+   1. In the `/etc/gitlab/gitlab.rb` file, change the `praefect['auto_migrate']` setting value from `false` to `true`
+
+   1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run:
 
-   1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
+   ```shell
+   sudo touch /etc/gitlab/skip-auto-reconfigure
+   ```
+   
+   1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect and
+      to run the Praefect database migrations.
+
+1. On all other Praefect nodes, [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
 
 ### Configure Gitaly
 
@@ -1662,21 +1679,17 @@ On each node:
    storage paths, enable the network listener, and to configure the token:
 
    ```ruby
-   # /etc/gitlab/gitlab.rb
-
    # Avoid running unnecessary services on the Gitaly server
    postgresql['enable'] = false
    redis['enable'] = false
-   nginx['enable'] = false
    puma['enable'] = false
-   unicorn['enable'] = false
    sidekiq['enable'] = false
    gitlab_workhorse['enable'] = false
-   grafana['enable'] = false
-
-   # If you run a separate monitoring node you can disable these services
-   alertmanager['enable'] = false
    prometheus['enable'] = false
+   alertmanager['enable'] = false
+   grafana['enable'] = false
+   gitlab_exporter['enable'] = false
+   nginx['enable'] = false
 
    # Prevent database migrations from running on upgrade automatically
    gitlab_rails['auto_migrate'] = false
@@ -1684,9 +1697,11 @@ On each node:
    # Configure the gitlab-shell API callback URL. Without this, `git push` will
    # fail. This can be your 'front door' GitLab URL or an internal load
    # balancer.
-   # Don't forget to copy `/etc/gitlab/gitlab-secrets.json` from web server to Gitaly server.
    gitlab_rails['internal_api_url'] = 'https://gitlab.example.com'
 
+   # Gitaly 
+   gitaly['enable'] = true
+
    # Make Gitaly accept connections on all network interfaces. You must use
    # firewalls to restrict access to this address/port.
    # Comment out following line if you only want to support TLS connections
@@ -1728,9 +1743,8 @@ On each node:
      })
      ```
 
-1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and
-   then replace the file of the same name on this server. If that file isn't on
-   this server, add the file from your Consul server to this server.
+1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
+   the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
 
 1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
 
@@ -1837,28 +1851,19 @@ To configure the Sidekiq nodes, on each one:
 1. Open `/etc/gitlab/gitlab.rb` with your editor:
 
    ```ruby
-   ########################################
-   #####        Services Disabled       ###
-   ########################################
-
-   nginx['enable'] = false
-   grafana['enable'] = false
-   prometheus['enable'] = false
-   alertmanager['enable'] = false
+   # Avoid running unnecessary services on the Sidekiq server
    gitaly['enable'] = false
-   gitlab_workhorse['enable'] = false
-   nginx['enable'] = false
-   puma['enable'] = false
-   postgres_exporter['enable'] = false
    postgresql['enable'] = false
    redis['enable'] = false
-   redis_exporter['enable'] = false
+   puma['enable'] = false
+   gitlab_workhorse['enable'] = false
+   prometheus['enable'] = false
+   alertmanager['enable'] = false
+   grafana['enable'] = false
    gitlab_exporter['enable'] = false
+   nginx['enable'] = false
 
-   ########################################
-   ####              Redis              ###
-   ########################################
-
+   # Redis
    ## Redis connection details
    ## First cluster that will host the cache
    gitlab_rails['redis_cache_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER>@gitlab-redis-cache'
@@ -1890,13 +1895,10 @@ To configure the Sidekiq nodes, on each one:
      {host: '10.6.0.83', port: 26379},
    ]
 
-   #######################################
-   ###              Gitaly             ###
-   #######################################
-
-   # git_data_dirs get configured for the Praefect virtual storage
-   # Address is Internal Load Balancer for Praefect
-   # Token is praefect_external_token
+   # Gitaly Cluster
+   ## git_data_dirs get configured for the Praefect virtual storage
+   ## Address is Internal Load Balancer for Praefect
+   ## Token is praefect_external_token
    git_data_dirs({
      "default" => {
        "gitaly_address" => "tcp://10.6.0.40:2305", # internal load balancer IP
@@ -1904,20 +1906,17 @@ To configure the Sidekiq nodes, on each one:
      }
    })
 
-   #######################################
-   ###            Postgres             ###
-   #######################################
+   # PostgreSQL
    gitlab_rails['db_host'] = '10.6.0.20' # internal load balancer IP
    gitlab_rails['db_port'] = 6432
    gitlab_rails['db_password'] = '<postgresql_user_password>'
    gitlab_rails['db_adapter'] = 'postgresql'
    gitlab_rails['db_encoding'] = 'unicode'
-   # Prevent database migrations from running on upgrade automatically
+   ## Prevent database migrations from running on upgrade automatically
    gitlab_rails['auto_migrate'] = false
 
-   #######################################
-   ###      Sidekiq configuration      ###
-   #######################################
+   # Sidekiq
+   sidekiq['enable'] = true
    sidekiq['listen_address'] = "0.0.0.0"
 
    # Set number of Sidekiq queue processes to the same number as available CPUs
@@ -1926,9 +1925,7 @@ To configure the Sidekiq nodes, on each one:
    # Set number of Sidekiq threads per queue process to the recommend number of 10
    sidekiq['max_concurrency'] = 10
 
-   #######################################
-   ###     Monitoring configuration    ###
-   #######################################
+   # Monitoring
    consul['enable'] = true
    consul['monitoring_service_discovery'] =  true
 
@@ -1936,16 +1933,13 @@ To configure the Sidekiq nodes, on each one:
       retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13)
    }
 
-   # Set the network addresses that the exporters will listen on
+   ## Set the network addresses that the exporters will listen on
    node_exporter['listen_address'] = '0.0.0.0:9100'
 
-   # Rails Status for prometheus
+   ## Add the monitoring node's IP address to the monitoring whitelist
    gitlab_rails['monitoring_whitelist'] = ['10.6.0.151/32', '127.0.0.0/8']
 
-   #############################
-   ###     Object storage    ###
-   #############################
-
+   # Object Storage
    # This is an example for configuring Object Storage on GCP
    # Replace this config with your chosen Object Storage provider as desired
    gitlab_rails['object_store']['connection'] = {
@@ -1960,11 +1954,26 @@ To configure the Sidekiq nodes, on each one:
    gitlab_rails['object_store']['objects']['packages']['bucket'] = "<gcp-packages-bucket-name>"
    gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = "<gcp-dependency-proxy-bucket-name>"
    gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = "<gcp-terraform-state-bucket-name>"
+
+   gitlab_rails['backup_upload_connection'] = {
+     'provider' => 'Google',
+     'google_project' => '<gcp-project-name>',
+     'google_json_key_location' => '<path-to-gcp-service-account-key>'
+   }
+   gitlab_rails['backup_upload_remote_directory'] = "<gcp-backups-state-bucket-name>"
+   ```
+
+1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
+   the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
+
+1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run:
+
+   ```shell
+   sudo touch /etc/gitlab/skip-auto-reconfigure
    ```
 
-1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace
-   the file of the same name on this server. If that file is not on this server,
-   add the file from your Consul server to this server.
+   Only a single designated node should handle migrations as detailed in the
+   [GitLab Rails post-configuration](#gitlab-rails-post-configuration) section.
 
 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
 
@@ -1997,9 +2006,6 @@ On each node perform the following:
 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab
    package of your choice. Be sure to follow _only_ installation steps 1 and 2
    on the page.
-1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace
-   the file of the same name on this server. If that file is not on this server,
-   add the file from your Consul server to this server.
 
 1. Edit `/etc/gitlab/gitlab.rb` and use the following configuration.
    To maintain uniformity of links across nodes, the `external_url`
@@ -2021,7 +2027,7 @@ On each node perform the following:
    })
 
    ## Disable components that will not be on the GitLab application server
-   roles ['application_role']
+   roles(['application_role'])
    gitaly['enable'] = false
    nginx['enable'] = true
    sidekiq['enable'] = false
@@ -2094,9 +2100,15 @@ On each node perform the following:
    gitlab_rails['object_store']['objects']['packages']['bucket'] = "<gcp-packages-bucket-name>"
    gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = "<gcp-dependency-proxy-bucket-name>"
    gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = "<gcp-terraform-state-bucket-name>"
+
+   gitlab_rails['backup_upload_connection'] = {
+     'provider' => 'Google',
+     'google_project' => '<gcp-project-name>',
+     'google_json_key_location' => '<path-to-gcp-service-account-key>'
+   }
+   gitlab_rails['backup_upload_remote_directory'] = "<gcp-backups-state-bucket-name>"
    ```
 
-1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
 1. If you're using [Gitaly with TLS support](#gitaly-cluster-tls-support), make sure the
    `git_data_dirs` entry is configured with `tls` instead of `tcp`:
 
@@ -2115,6 +2127,20 @@ On each node perform the following:
       sudo cp cert.pem /etc/gitlab/trusted-certs/
       ```
 
+1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
+   the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
+
+1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run:
+
+   ```shell
+   sudo touch /etc/gitlab/skip-auto-reconfigure
+   ```
+
+   Only a single designated node should handle migrations as detailed in the
+   [GitLab Rails post-configuration](#gitlab-rails-post-configuration) section.
+
+1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
+
 1. If you're [using NFS](#configure-nfs-optional):
    1. If necessary, install the NFS client utility packages using the following
       commands:
@@ -2154,7 +2180,7 @@ On each node perform the following:
       registry['gid'] = 9002
       ```
 
-1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
+   1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
 1. Confirm the node can connect to Gitaly:
 
    ```shell
@@ -2218,52 +2244,30 @@ To configure the Monitoring node:
 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab
    package of your choice. Be sure to follow _only_ installation steps 1 and 2
    on the page.
-1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace
-   the file of the same name on this server. If that file is not on this server,
-   add the file from your Consul server to this server.
 
 1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
 
    ```ruby
-   external_url 'http://gitlab.example.com'
+   roles(['monitoring_role', 'consul_role'])
 
-   # Disable all other services
-   alertmanager['enable'] = false
-   gitaly['enable'] = false
-   gitlab_exporter['enable'] = false
-   gitlab_workhorse['enable'] = false
-   nginx['enable'] = true
-   postgres_exporter['enable'] = false
-   postgresql['enable'] = false
-   redis['enable'] = false
-   redis_exporter['enable'] = false
-   sidekiq['enable'] = false
-   puma['enable'] = false
-   unicorn['enable'] = false
-   node_exporter['enable'] = false
-   gitlab_exporter['enable'] = false
+   external_url 'http://gitlab.example.com'
 
-   # Enable Prometheus
-   prometheus['enable'] = true
+   # Prometheus
    prometheus['listen_address'] = '0.0.0.0:9090'
    prometheus['monitor_kubernetes'] = false
 
-   # Enable Login form
-   grafana['disable_login_form'] = false
-
-   # Enable Grafana
-   grafana['enable'] = true
+   # Grafana
    grafana['admin_password'] = '<grafana_password>'
+   grafana['disable_login_form'] = false
 
    # Enable service discovery for Prometheus
-   consul['enable'] = true
    consul['monitoring_service_discovery'] =  true
    consul['configuration'] = {
       retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13)
    }
 
-   # Prevent database migrations from running on upgrade automatically
-   gitlab_rails['auto_migrate'] = false
+   # Nginx - For Grafana access
+   nginx['enable'] = true
    ```
 
 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
diff --git a/doc/administration/reference_architectures/2k_users.md b/doc/administration/reference_architectures/2k_users.md
index 69e261cfbe6..7db3a343e0b 100644
--- a/doc/administration/reference_architectures/2k_users.md
+++ b/doc/administration/reference_architectures/2k_users.md
@@ -18,20 +18,24 @@ For a full list of reference architectures, see
 
 | Service                                  | Nodes  | Configuration           | GCP             | AWS          | Azure    |
 |------------------------------------------|--------|-------------------------|-----------------|--------------|----------|
-| Load balancer                            | 1      | 2 vCPU, 1.8 GB memory   | `n1-highcpu-2`  | `c5.large`   | `F2s v2` |
-| PostgreSQL*                              | 1      | 2 vCPU, 7.5 GB memory   | `n1-standard-2` | `m5.large`   | `D2s v3` |
-| Redis**                                  | 1      | 1 vCPU, 3.75 GB memory  | `n1-standard-1` | `m5.large`   | `D2s v3` |
+| Load balancer(3)                         | 1      | 2 vCPU, 1.8 GB memory   | `n1-highcpu-2`  | `c5.large`   | `F2s v2` |
+| PostgreSQL(1)                            | 1      | 2 vCPU, 7.5 GB memory   | `n1-standard-2` | `m5.large`   | `D2s v3` |
+| Redis(2)                                 | 1      | 1 vCPU, 3.75 GB memory  | `n1-standard-1` | `m5.large`   | `D2s v3` |
 | Gitaly                                   | 1      | 4 vCPU, 15 GB memory    | `n1-standard-4` | `m5.xlarge`  | `D4s v3` |
 | GitLab Rails                             | 2      | 8 vCPU, 7.2 GB memory   | `n1-highcpu-8`  | `c5.2xlarge` | `F8s v2` |
 | Monitoring node                          | 1      | 2 vCPU, 1.8 GB memory   | `n1-highcpu-2`  | `c5.large`   | `F2s v2` |
-| Object storage                           | n/a    | n/a                     | n/a             | n/a          | n/a      |
+| Object storage(4)                        | n/a    | n/a                     | n/a             | n/a          | n/a      |
 | NFS server (optional, not recommended)   | 1      | 4 vCPU, 3.6 GB memory   | `n1-highcpu-4`  | `c5.xlarge`  | `F4s v2` |
 
+<!-- markdownlint-disable MD029 -->
+1. Can be optionally run on reputable third party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work, however Azure Database for PostgreSQL is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to performance issues. Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery.
+2. Can be optionally run as reputable third party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work.
+3. Can be optionally run as reputable third party load balancing services (LB PaaS). AWS ELB is known to work.
+4. Should be run on reputable third party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work.
+<!-- markdownlint-enable MD029 -->
+
 NOTE:
-Components marked with * can be optionally run on reputable
-third party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work.
-Components marked with ** can be optionally run on reputable
-third party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work.
+For all PaaS solutions that involve configuring instances, it is strongly recommended to implement a minimum of three nodes in three different availability zones to align with resilient cloud architecture practices.
 
 ```plantuml
 @startuml 2k
@@ -84,7 +88,7 @@ To set up GitLab and its components to accommodate up to 2,000 users:
 1. [Configure Gitaly](#configure-gitaly), which provides access to the Git
    repositories.
 1. [Configure the main GitLab Rails application](#configure-gitlab-rails)
-   to run Puma/Unicorn, Workhorse, GitLab Shell, and to serve all frontend
+   to run Puma, Workhorse, GitLab Shell, and to serve all frontend
    requests (which include UI, API, and Git over HTTP/SSH).
 1. [Configure Prometheus](#configure-prometheus) to monitor your GitLab
    environment.
@@ -265,10 +269,8 @@ further configuration steps.
      database. Example: `%w(123.123.123.123/32 123.123.123.234/32)`
 
    ```ruby
-   # Disable all components except PostgreSQL
-   roles ['postgres_role']
-   patroni['enable'] = false
-   consul['enable'] = false
+   # Disable all components except PostgreSQL related ones
+   roles(['postgres_role'])
    prometheus['enable'] = false
    alertmanager['enable'] = false
    pgbouncer_exporter['enable'] = false
@@ -295,6 +297,9 @@ further configuration steps.
    gitlab_rails['auto_migrate'] = false
    ```
 
+1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
+   the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
+
 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
 1. Note the PostgreSQL node's IP address or hostname, port, and
    plain text password. These will be necessary when configuring the [GitLab
@@ -346,20 +351,18 @@ Omnibus:
    ```ruby
    ## Enable Redis
    redis['enable'] = true
-
-   ## Disable all other services
+   
+   # Avoid running unnecessary services on the Redis server
+   gitaly['enable'] = false
+   postgresql['enable'] = false
+   puma['enable'] = false
    sidekiq['enable'] = false
    gitlab_workhorse['enable'] = false
-   puma['enable'] = false
-   unicorn['enable'] = false
-   postgresql['enable'] = false
-   nginx['enable'] = false
    prometheus['enable'] = false
    alertmanager['enable'] = false
-   pgbouncer_exporter['enable'] = false
-   gitlab_exporter['enable'] = false
-   gitaly['enable'] = false
    grafana['enable'] = false
+   gitlab_exporter['enable'] = false
+   nginx['enable'] = false
 
    redis['bind'] = '0.0.0.0'
    redis['port'] = 6379
@@ -376,7 +379,11 @@ Omnibus:
    }
    ```
 
+1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
+   the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
+
 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
+
 1. Note the Redis node's IP address or hostname, port, and
    Redis password. These will be necessary when [configuring the GitLab
    application servers](#configure-gitlab-rails) later.
@@ -455,16 +462,14 @@ To configure the Gitaly server, on the server node you want to use for Gitaly:
    # Avoid running unnecessary services on the Gitaly server
    postgresql['enable'] = false
    redis['enable'] = false
-   nginx['enable'] = false
    puma['enable'] = false
-   unicorn['enable'] = false
    sidekiq['enable'] = false
    gitlab_workhorse['enable'] = false
-   grafana['enable'] = false
-
-   # If you run a separate monitoring node you can disable these services
-   alertmanager['enable'] = false
    prometheus['enable'] = false
+   alertmanager['enable'] = false
+   grafana['enable'] = false
+   gitlab_exporter['enable'] = false
+   nginx['enable'] = false
 
    # Prevent database migrations from running on upgrade automatically
    gitlab_rails['auto_migrate'] = false
@@ -472,9 +477,11 @@ To configure the Gitaly server, on the server node you want to use for Gitaly:
    # Configure the gitlab-shell API callback URL. Without this, `git push` will
    # fail. This can be your 'front door' GitLab URL or an internal load
    # balancer.
-   # Don't forget to copy `/etc/gitlab/gitlab-secrets.json` from web server to Gitaly server.
    gitlab_rails['internal_api_url'] = 'https://gitlab.example.com'
 
+   # Gitaly
+   gitaly['enable'] = true
+
    # Make Gitaly accept connections on all network interfaces. You must use
    # firewalls to restrict access to this address/port.
    # Comment out following line if you only want to support TLS connections
@@ -494,7 +501,11 @@ To configure the Gitaly server, on the server node you want to use for Gitaly:
    })
    ```
 
-1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
+1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
+   the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
+
+1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
+
 1. Confirm that Gitaly can perform callbacks to the internal API:
 
    ```shell
@@ -629,7 +640,7 @@ On each node perform the following:
    })
 
    ## Disable components that will not be on the GitLab application server
-   roles ['application_role']
+   roles(['application_role'])
    gitaly['enable'] = false
    nginx['enable'] = true
 
@@ -658,10 +669,7 @@ On each node perform the following:
    gitlab_rails['monitoring_whitelist'] = ['<MONITOR NODE IP>/32', '127.0.0.0/8']
    nginx['status']['options']['allow'] = ['<MONITOR NODE IP>/32', '127.0.0.0/8']
 
-   #############################
-   ###     Object storage    ###
-   #############################
-
+   # Object Storage
    # This is an example for configuring Object Storage on GCP
    # Replace this config with your chosen Object Storage provider as desired
    gitlab_rails['object_store']['connection'] = {
@@ -677,6 +685,13 @@ On each node perform the following:
    gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = "<gcp-dependency-proxy-bucket-name>"
    gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = "<gcp-terraform-state-bucket-name>"
 
+   gitlab_rails['backup_upload_connection'] = {
+     'provider' => 'Google',
+     'google_project' => '<gcp-project-name>',
+     'google_json_key_location' => '<path-to-gcp-service-account-key>'
+   }
+   gitlab_rails['backup_upload_remote_directory'] = "<gcp-backups-state-bucket-name>"
+
    ## Uncomment and edit the following options if you have set up NFS
    ##
    ## Prevent GitLab from starting if NFS data mounts are not available
@@ -710,7 +725,20 @@ On each node perform the following:
       sudo cp cert.pem /etc/gitlab/trusted-certs/
       ```
 
-1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
+1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
+   the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
+
+1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run:
+
+   ```shell
+   sudo touch /etc/gitlab/skip-auto-reconfigure
+   ```
+
+   Only a single designated node should handle migrations as detailed in the
+   [GitLab Rails post-configuration](#gitlab-rails-post-configuration) section.
+
+1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
+
 1. Run `sudo gitlab-rake gitlab:gitaly:check` to confirm the node can connect to Gitaly.
 1. Tail the logs to see the requests:
 
@@ -718,11 +746,6 @@ On each node perform the following:
    sudo gitlab-ctl tail gitaly
    ```
 
-1. Save the `/etc/gitlab/gitlab-secrets.json` file from one of the two
-   application nodes and install it on the other application node and the
-   [Gitaly node](#configure-gitaly) and
-   [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
-
 When you specify `https` in the `external_url`, as in the previous example,
 GitLab expects that the SSL certificates are in `/etc/gitlab/ssl/`. If the
 certificates aren't present, NGINX will fail to start. For more information, see
@@ -765,38 +788,21 @@ running [Prometheus](../monitoring/prometheus/index.md) and
 1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
 
    ```ruby
+   roles(['monitoring_role'])
+
    external_url 'http://gitlab.example.com'
 
-   # Enable Prometheus
-   prometheus['enable'] = true
+   # Prometheus
    prometheus['listen_address'] = '0.0.0.0:9090'
    prometheus['monitor_kubernetes'] = false
 
-   # Enable Login form
-   grafana['disable_login_form'] = false
-
-   # Enable Grafana
+   # Grafana
    grafana['enable'] = true
-   grafana['admin_password'] = 'toomanysecrets'
+   grafana['admin_password'] = '<grafana_password>'
+   grafana['disable_login_form'] = false
 
-   # Disable all other services
-   alertmanager['enable'] = false
-   gitaly['enable'] = false
-   gitlab_exporter['enable'] = false
-   gitlab_workhorse['enable'] = false
+   # Nginx - For Grafana access
    nginx['enable'] = true
-   postgres_exporter['enable'] = false
-   postgresql['enable'] = false
-   redis['enable'] = false
-   redis_exporter['enable'] = false
-   sidekiq['enable'] = false
-   puma['enable'] = false
-   unicorn['enable'] = false
-   node_exporter['enable'] = false
-   gitlab_exporter['enable'] = false
-
-   # Prevent database migrations from running on upgrade automatically
-   gitlab_rails['auto_migrate'] = false
    ```
 
 1. Prometheus also needs some scrape configurations to pull all the data from the various
diff --git a/doc/administration/reference_architectures/3k_users.md b/doc/administration/reference_architectures/3k_users.md
index d81f98a35d4..bca5e4c3dab 100644
--- a/doc/administration/reference_architectures/3k_users.md
+++ b/doc/administration/reference_architectures/3k_users.md
@@ -27,26 +27,31 @@ For a full list of reference architectures, see
 
 | Service                                    | Nodes       | Configuration         | GCP             | AWS          | Azure    |
 |--------------------------------------------|-------------|-----------------------|-----------------|--------------|----------|
-| External load balancing node               | 1           | 2 vCPU, 1.8 GB memory | `n1-highcpu-2`  | `c5.large`   | `F2s v2` |
-| Redis**                                    | 3           | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large`   | `D2s v3` |
-| Consul* + Sentinel**                       | 3           | 2 vCPU, 1.8 GB memory | `n1-highcpu-2`  | `c5.large`   | `F2s v2` |
-| PostgreSQL*                                | 3           | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large`   | `D2s v3` |
-| PgBouncer*                                 | 3           | 2 vCPU, 1.8 GB memory | `n1-highcpu-2`  | `c5.large`   | `F2s v2` |
-| Internal load balancing node               | 1           | 2 vCPU, 1.8 GB memory | `n1-highcpu-2`  | `c5.large`   | `F2s v2` |
+| External load balancing node(3)            | 1           | 2 vCPU, 1.8 GB memory | `n1-highcpu-2`  | `c5.large`   | `F2s v2` |
+| Redis(2)                                   | 3           | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large`   | `D2s v3` |
+| Consul(1) + Sentinel(2)                    | 3           | 2 vCPU, 1.8 GB memory | `n1-highcpu-2`  | `c5.large`   | `F2s v2` |
+| PostgreSQL(1)                              | 3           | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large`   | `D2s v3` |
+| PgBouncer(1)                               | 3           | 2 vCPU, 1.8 GB memory | `n1-highcpu-2`  | `c5.large`   | `F2s v2` |
+| Internal load balancing node(3)            | 1           | 2 vCPU, 1.8 GB memory | `n1-highcpu-2`  | `c5.large`   | `F2s v2` |
 | Gitaly                                     | 3           | 4 vCPU, 15 GB memory  | `n1-standard-4` | `m5.xlarge`  | `D4s v3` |
 | Praefect                                   | 3           | 2 vCPU, 1.8 GB memory | `n1-highcpu-2`  | `c5.large`   | `F2s v2` |
-| Praefect PostgreSQL*                       | 1+          | 2 vCPU, 1.8 GB memory | `n1-highcpu-2`  | `c5.large`   | `F2s v2` |
+| Praefect PostgreSQL(1)                     | 1+          | 2 vCPU, 1.8 GB memory | `n1-highcpu-2`  | `c5.large`   | `F2s v2` |
 | Sidekiq                                    | 4           | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large`   | `D2s v3` |
 | GitLab Rails                               | 3           | 8 vCPU, 7.2 GB memory | `n1-highcpu-8`  | `c5.2xlarge` | `F8s v2` |
 | Monitoring node                            | 1           | 2 vCPU, 1.8 GB memory | `n1-highcpu-2`  | `c5.large`   | `F2s v2` |
-| Object storage                             | n/a         | n/a                   | n/a             | n/a          | n/a      |
+| Object storage(4)                          | n/a         | n/a                   | n/a             | n/a          | n/a      |
 | NFS server (optional, not recommended)     | 1           | 4 vCPU, 3.6 GB memory | `n1-highcpu-4`  | `c5.xlarge`  | `F4s v2` |
 
+<!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix -->
+<!-- markdownlint-disable MD029 -->
+1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work, however Azure Database for PostgreSQL is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to performance issues. Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery.
+2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work.
+3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work.
+4. Should be run on reputable third party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work.
+<!-- markdownlint-enable MD029 -->
+
 NOTE:
-Components marked with * can be optionally run on reputable
-third party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work.
-Components marked with ** can be optionally run on reputable
-third party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work.
+For all PaaS solutions that involve configuring instances, it is strongly recommended to implement a minimum of three nodes in three different availability zones to align with resilient cloud architecture practices.
 
 ```plantuml
 @startuml 3k
@@ -169,7 +174,7 @@ To set up GitLab and its components to accommodate up to 3,000 users:
    provides access to the Git repositories.
 1. [Configure Sidekiq](#configure-sidekiq).
 1. [Configure the main GitLab Rails application](#configure-gitlab-rails)
-   to run Puma/Unicorn, Workhorse, GitLab Shell, and to serve all frontend
+   to run Puma, Workhorse, GitLab Shell, and to serve all frontend
    requests (which include UI, API, and Git over HTTP/SSH).
 1. [Configure Prometheus](#configure-prometheus) to monitor your GitLab
    environment.
@@ -470,8 +475,8 @@ a node and change its status from primary to replica (and vice versa).
 1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
 
    ```ruby
-   # Specify server role as 'redis_master_role'
-   roles ['redis_master_role']
+   # Specify server role as 'redis_master_role' and enable Consul agent
+   roles(['redis_master_role', 'consul_role'])
 
    # IP address pointing to a local IP that the other machines can reach to.
    # You can also set bind to '0.0.0.0' which listen in all interfaces.
@@ -487,7 +492,6 @@ a node and change its status from primary to replica (and vice versa).
    redis['password'] = 'redis-password-goes-here'
 
    ## Enable service discovery for Prometheus
-   consul['enable'] = true
    consul['monitoring_service_discovery'] =  true
 
    ## The IPs of the Consul server nodes
@@ -508,6 +512,9 @@ a node and change its status from primary to replica (and vice versa).
    gitlab_rails['auto_migrate'] = false
    ```
 
+1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
+   the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
+
 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
 
 You can specify multiple roles, like sentinel and Redis, as:
@@ -546,8 +553,8 @@ run: redis-exporter: (pid 30075) 76861s; run: log: (pid 29674) 76896s
 1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
 
    ```ruby
-   # Specify server role as 'redis_replica_role'
-   roles ['redis_replica_role']
+   # Specify server role as 'redis_replica_role' and enable Consul agent
+   roles(['redis_replica_role', 'consul_role'])
 
    # IP address pointing to a local IP that the other machines can reach to.
    # You can also set bind to '0.0.0.0' which listen in all interfaces.
@@ -570,7 +577,6 @@ run: redis-exporter: (pid 30075) 76861s; run: log: (pid 29674) 76896s
    #redis['master_port'] = 6379
 
    ## Enable service discovery for Prometheus
-   consul['enable'] = true
    consul['monitoring_service_discovery'] =  true
 
    ## The IPs of the Consul server nodes
@@ -591,12 +597,15 @@ run: redis-exporter: (pid 30075) 76861s; run: log: (pid 29674) 76896s
    gitlab_rails['auto_migrate'] = false
    ```
 
+1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
+   the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
+
 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
 1. Go through the steps again for all the other replica nodes, and
    make sure to set up the IPs correctly.
 
 You can specify multiple roles, like sentinel and Redis, as:
-`roles ['redis_sentinel_role', 'redis_master_role']`. Read more about
+`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about
 [roles](https://docs.gitlab.com/omnibus/roles/).
 
 These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after
@@ -638,7 +647,7 @@ To configure the Sentinel:
 1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
 
    ```ruby
-   roles ['redis_sentinel_role', 'consul_role']
+   roles(['redis_sentinel_role', 'consul_role'])
 
    # Must be the same in every sentinel node
    redis['master_name'] = 'gitlab-redis'
@@ -702,7 +711,6 @@ To configure the Sentinel:
    # sentinel['failover_timeout'] = 60000
 
    ## Enable service discovery for Prometheus
-   consul['enable'] = true
    consul['monitoring_service_discovery'] =  true
 
    ## The IPs of the Consul server nodes
@@ -720,7 +728,11 @@ To configure the Sentinel:
    gitlab_rails['auto_migrate'] = false
    ```
 
+1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
+   the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
+
 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
+
 1. Go through the steps again for all the other Consul/Sentinel nodes, and
    make sure you set up the correct IPs.
 
@@ -813,6 +825,15 @@ in the second step, do not supply the `EXTERNAL_URL` value.
    sudo gitlab-ctl pg-password-md5 pgbouncer
    ```
 
+1. Generate a password hash for the PostgreSQL replication username/password pair. This assumes you will use the default
+   username of `gitlab_replicator` (recommended). The command will request a password
+   and a confirmation. Use the value that is output by this command in the next step
+   as the value of `<postgresql_replication_password_hash>`:
+   
+   ```shell
+   sudo gitlab-ctl pg-password-md5 gitlab_replicator
+   ```
+
 1. Generate a password hash for the Consul database username/password pair. This assumes you will use the default
    username of `gitlab-consul` (recommended). The command will request a password
    and confirmation. Use the value that is output by this command in the next
@@ -825,27 +846,28 @@ in the second step, do not supply the `EXTERNAL_URL` value.
 1. On every database node, edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section:
 
    ```ruby
-   # Disable all components except PostgreSQL, Patroni, and Consul
-   roles ['postgres_role']
-
+   # Disable all components except Patroni and Consul
+   roles(['patroni_role'])
+   
    # PostgreSQL configuration
    postgresql['listen_address'] = '0.0.0.0'
 
-   # Enable Patroni
-   patroni['enable'] = true
-   # Set `max_wal_senders` to one more than the number of database nodes in the cluster.
+   # Sets `max_replication_slots` to double the number of database nodes.
+   # Patroni uses one extra slot per node when initiating the replication.
+   patroni['postgresql']['max_replication_slots'] = 6
+
+   # Set `max_wal_senders` to one more than the number of replication slots in the cluster.
    # This is used to prevent replication from using up all of the
    # available database connections.
-   patroni['postgresql']['max_wal_senders'] = 4
-   patroni['postgresql']['max_replication_slots'] = 4
+   patroni['postgresql']['max_wal_senders'] = 7
+
    # Incoming recommended value for max connections is 500. See https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5691.
    patroni['postgresql']['max_connections'] = 500
 
    # Prevent database migrations from running on upgrade automatically
    gitlab_rails['auto_migrate'] = false
-
+   
    # Configure the Consul agent
-   consul['enable'] = true
    consul['services'] = %w(postgresql)
    ## Enable service discovery for Prometheus
    consul['monitoring_service_discovery'] =  true
@@ -855,6 +877,8 @@ in the second step, do not supply the `EXTERNAL_URL` value.
    #
    # Replace PGBOUNCER_PASSWORD_HASH with a generated md5 value
    postgresql['pgbouncer_user_password'] = '<pgbouncer_password_hash>'
+   # Replace POSTGRESQL_REPLICATION_PASSWORD_HASH with a generated md5 value
+   postgresql['sql_replication_password'] = '<postgresql_replication_password_hash>'
    # Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value
    postgresql['sql_user_password'] = '<postgresql_password_hash>'
 
@@ -878,9 +902,8 @@ PostgreSQL, with Patroni managing its failover, will default to use `pg_rewind`
 Like most failover handling methods, this has a small chance of leading to data loss.
 Learn more about the various [Patroni replication methods](../postgresql/replication_and_failover.md#selecting-the-appropriate-patroni-replication-method).
 
-1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace
-   the file of the same name on this server. If that file is not on this server,
-   add the file from your Consul server to this server.
+1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
+   the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
 
 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
 
@@ -895,22 +918,7 @@ are supported and can be added if needed.
 
 #### PostgreSQL post-configuration
 
-SSH in to the **primary node**:
-
-1. Open a database prompt:
-
-   ```shell
-   gitlab-psql -d gitlabhq_production
-   ```
-
-1. Enable the `pg_trgm` and `btree_gist` extensions:
-
-   ```shell
-   CREATE EXTENSION pg_trgm;
-   CREATE EXTENSION btree_gist;
-   ```
-
-1. Exit the database prompt by typing `\q` and Enter.
+SSH in to any of the Patroni nodes on the **primary site**:
 
 1. Check the status of the leader and cluster:
 
@@ -952,7 +960,7 @@ The following IPs will be used as an example:
 
    ```ruby
    # Disable all components except Pgbouncer and Consul agent
-   roles ['pgbouncer_role']
+   roles(['pgbouncer_role'])
 
    # Configure PgBouncer
    pgbouncer['admin_users'] = %w(pgbouncer gitlab-consul)
@@ -969,7 +977,6 @@ The following IPs will be used as an example:
 
    # Configure Consul agent
    consul['watchers'] = %w(postgresql)
-   consul['enable'] = true
    consul['configuration'] = {
    retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13)
    }
@@ -982,6 +989,9 @@ The following IPs will be used as an example:
    pgbouncer_exporter['listen_address'] = '0.0.0.0:9188'
    ```
 
+1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
+   the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
+
 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
 
 1. Create a `.pgpass` file so Consul is able to
@@ -1097,9 +1107,7 @@ in the second step, do not supply the `EXTERNAL_URL` value.
 
    ```ruby
    # Disable all components except PostgreSQL and Consul
-   roles ['postgres_role']
-   repmgr['enable'] = false
-   patroni['enable'] = false
+   roles(['postgres_role', 'consul_role'])
 
    # PostgreSQL configuration
    postgresql['listen_address'] = '0.0.0.0'
@@ -1109,7 +1117,6 @@ in the second step, do not supply the `EXTERNAL_URL` value.
    gitlab_rails['auto_migrate'] = false
 
    # Configure the Consul agent
-   consul['enable'] = true
    ## Enable service discovery for Prometheus
    consul['monitoring_service_discovery'] =  true
 
@@ -1135,6 +1142,9 @@ in the second step, do not supply the `EXTERNAL_URL` value.
    # END user configuration
    ```
 
+1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
+   the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
+
 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
 1. Follow the [post configuration](#praefect-postgresql-post-configuration).
 
@@ -1213,7 +1223,7 @@ Praefect requires several secret tokens to secure communications across the Clus
 
 Gitaly Cluster nodes are configured in Praefect via a `virtual storage`. Each storage contains
 the details of each Gitaly node that makes up the cluster. Each storage is also given a name
-and this name is used in several areas of the config. In this guide, the name of the storage will be
+and this name is used in several areas of the configuration. In this guide, the name of the storage will be
 `default`. Also, this guide is geared towards new installs, if upgrading an existing environment
 to use Gitaly Cluster, you may need to use a different name.
 Refer to the [Praefect documentation](../gitaly/praefect.md#praefect) for more info.
@@ -1233,19 +1243,18 @@ To configure the Praefect nodes, on each one:
 1. Edit the `/etc/gitlab/gitlab.rb` file to configure Praefect:
 
    ```ruby
-   # Avoid running unnecessary services on the Gitaly server
+   # Avoid running unnecessary services on the Praefect server
+   gitaly['enable'] = false
    postgresql['enable'] = false
    redis['enable'] = false
-   nginx['enable'] = false
    puma['enable'] = false
-   unicorn['enable'] = false
    sidekiq['enable'] = false
    gitlab_workhorse['enable'] = false
-   grafana['enable'] = false
-
-   # If you run a separate monitoring node you can disable these services
-   alertmanager['enable'] = false
    prometheus['enable'] = false
+   alertmanager['enable'] = false
+   grafana['enable'] = false
+   gitlab_exporter['enable'] = false
+   nginx['enable'] = false
 
    # Praefect Configuration
    praefect['enable'] = true
@@ -1283,19 +1292,20 @@ To configure the Praefect nodes, on each one:
    # server ('praefect') and in git_data_dirs on Gitaly nodes ('gitaly-1')
    praefect['virtual_storages'] = {
      'default' => {
-       'gitaly-1' => {
-         'address' => 'tcp://10.6.0.91:8075',
-         'token'   => '<praefect_internal_token>',
-         'primary' => true
-       },
-       'gitaly-2' => {
-         'address' => 'tcp://10.6.0.92:8075',
-         'token'   => '<praefect_internal_token>'
-       },
-       'gitaly-3' => {
-         'address' => 'tcp://10.6.0.93:8075',
-         'token'   => '<praefect_internal_token>'
-       },
+       'nodes' => {
+         'gitaly-1' => {
+           'address' => 'tcp://10.6.0.91:8075',
+           'token'   => '<praefect_internal_token>'
+         },
+         'gitaly-2' => {
+           'address' => 'tcp://10.6.0.92:8075',
+           'token'   => '<praefect_internal_token>'
+         },
+         'gitaly-3' => {
+           'address' => 'tcp://10.6.0.93:8075',
+           'token'   => '<praefect_internal_token>'
+         },
+       }
      }
    }
 
@@ -1312,11 +1322,25 @@ To configure the Praefect nodes, on each one:
    # END user configuration
    ```
 
-   1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and
-      then replace the file of the same name on this server. If that file isn't on
-      this server, add the file from your Consul server to this server.
+1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
+the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
 
-   1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
+1. Praefect requires to run some database migrations, much like the main GitLab application. For this
+   you should select **one Praefect node only to run the migrations**, AKA the _Deploy Node_. This node
+   must be configured first before the others as follows:
+   
+   1. In the `/etc/gitlab/gitlab.rb` file, change the `praefect['auto_migrate']` setting value from `false` to `true`
+
+   1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run:
+
+   ```shell
+   sudo touch /etc/gitlab/skip-auto-reconfigure
+   ```
+   
+   1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect and
+      to run the Praefect database migrations.
+
+1. On all other Praefect nodes, [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
 
 ### Configure Gitaly
 
@@ -1360,29 +1384,27 @@ On each node:
    storage paths, enable the network listener, and to configure the token:
 
    ```ruby
-   # /etc/gitlab/gitlab.rb
-
    # Avoid running unnecessary services on the Gitaly server
    postgresql['enable'] = false
    redis['enable'] = false
-   nginx['enable'] = false
    puma['enable'] = false
-   unicorn['enable'] = false
    sidekiq['enable'] = false
    gitlab_workhorse['enable'] = false
-   grafana['enable'] = false
-
-   # If you run a separate monitoring node you can disable these services
-   alertmanager['enable'] = false
    prometheus['enable'] = false
+   alertmanager['enable'] = false
+   grafana['enable'] = false
+   gitlab_exporter['enable'] = false
+   nginx['enable'] = false
 
    # Prevent database migrations from running on upgrade automatically
    gitlab_rails['auto_migrate'] = false
 
+   # Gitaly
+   gitaly['enable'] = true
+
    # Configure the gitlab-shell API callback URL. Without this, `git push` will
    # fail. This can be your 'front door' GitLab URL or an internal load
    # balancer.
-   # Don't forget to copy `/etc/gitlab/gitlab-secrets.json` from web server to Gitaly server.
    gitlab_rails['internal_api_url'] = 'https://gitlab.example.com'
 
    # Make Gitaly accept connections on all network interfaces. You must use
@@ -1426,9 +1448,8 @@ On each node:
      })
      ```
 
-1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and
-   then replace the file of the same name on this server. If that file isn't on
-   this server, add the file from your Consul server to this server.
+1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
+   the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
 
 1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
 
@@ -1537,29 +1558,19 @@ To configure the Sidekiq nodes, one each one:
 1. Open `/etc/gitlab/gitlab.rb` with your editor:
 
    ```ruby
-   ########################################
-   #####        Services Disabled       ###
-   ########################################
-
-   nginx['enable'] = false
-   grafana['enable'] = false
-   prometheus['enable'] = false
-   alertmanager['enable'] = false
+   # Avoid running unnecessary services on the Sidekiq server
    gitaly['enable'] = false
-   gitlab_workhorse['enable'] = false
-   nginx['enable'] = false
-   puma['enable'] = false
-   postgres_exporter['enable'] = false
    postgresql['enable'] = false
    redis['enable'] = false
-   redis_exporter['enable'] = false
+   puma['enable'] = false
+   gitlab_workhorse['enable'] = false
+   prometheus['enable'] = false
+   alertmanager['enable'] = false
+   grafana['enable'] = false
    gitlab_exporter['enable'] = false
+   nginx['enable'] = false
 
-   ########################################
-   ####              Redis              ###
-   ########################################
-
-   ## Must be the same in every sentinel node
+   # Redis
    redis['master_name'] = 'gitlab-redis'
 
    ## The same password for Redis authentication you set up for the master node.
@@ -1572,13 +1583,10 @@ To configure the Sidekiq nodes, one each one:
       {'host' => '10.6.0.13', 'port' => 26379},
    ]
 
-   #######################################
-   ###              Gitaly             ###
-   #######################################
-
-   # git_data_dirs get configured for the Praefect virtual storage
-   # Address is Internal Load Balancer for Praefect
-   # Token is praefect_external_token
+   # Gitaly Cluster
+   ## git_data_dirs get configured for the Praefect virtual storage
+   ## Address is Internal Load Balancer for Praefect
+   ## Token is praefect_external_token
    git_data_dirs({
      "default" => {
        "gitaly_address" => "tcp://10.6.0.40:2305", # internal load balancer IP
@@ -1586,31 +1594,26 @@ To configure the Sidekiq nodes, one each one:
      }
    })
 
-   #######################################
-   ###            Postgres             ###
-   #######################################
+   # PostgreSQL
    gitlab_rails['db_host'] = '10.6.0.40' # internal load balancer IP
    gitlab_rails['db_port'] = 6432
    gitlab_rails['db_password'] = '<postgresql_user_password>'
    gitlab_rails['db_adapter'] = 'postgresql'
    gitlab_rails['db_encoding'] = 'unicode'
-   # Prevent database migrations from running on upgrade automatically
+   ## Prevent database migrations from running on upgrade automatically
    gitlab_rails['auto_migrate'] = false
 
-   #######################################
-   ###      Sidekiq configuration      ###
-   #######################################
+   # Sidekiq
+   sidekiq['enable'] = true
    sidekiq['listen_address'] = "0.0.0.0"
 
-   # Set number of Sidekiq queue processes to the same number as available CPUs
+   ## Set number of Sidekiq queue processes to the same number as available CPUs
    sidekiq['queue_groups'] = ['*'] * 2
 
-   # Set number of Sidekiq threads per queue process to the recommend number of 10
+   ## Set number of Sidekiq threads per queue process to the recommend number of 10
    sidekiq['max_concurrency'] = 10
 
-   #######################################
-   ###     Monitoring configuration    ###
-   #######################################
+   # Monitoring
    consul['enable'] = true
    consul['monitoring_service_discovery'] =  true
 
@@ -1618,19 +1621,16 @@ To configure the Sidekiq nodes, one each one:
       retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13)
    }
 
-   # Set the network addresses that the exporters will listen on
+   ## Set the network addresses that the exporters will listen on
    node_exporter['listen_address'] = '0.0.0.0:9100'
 
-   # Rails Status for prometheus
+   ## Add the monitoring node's IP address to the monitoring whitelist
    gitlab_rails['monitoring_whitelist'] = ['10.6.0.81/32', '127.0.0.0/8']
    gitlab_rails['prometheus_address'] = '10.6.0.81:9090'
 
-   #############################
-   ###     Object storage    ###
-   #############################
-
-   # This is an example for configuring Object Storage on GCP
-   # Replace this config with your chosen Object Storage provider as desired
+   # Object Storage
+   ## This is an example for configuring Object Storage on GCP
+   ## Replace this config with your chosen Object Storage provider as desired
    gitlab_rails['object_store']['connection'] = {
      'provider' => 'Google',
      'google_project' => '<gcp-project-name>',
@@ -1644,9 +1644,28 @@ To configure the Sidekiq nodes, one each one:
    gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = "<gcp-dependency-proxy-bucket-name>"
    gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = "<gcp-terraform-state-bucket-name>"
 
+   gitlab_rails['backup_upload_connection'] = {
+     'provider' => 'Google',
+     'google_project' => '<gcp-project-name>',
+     'google_json_key_location' => '<path-to-gcp-service-account-key>'
+   }
+   gitlab_rails['backup_upload_remote_directory'] = "<gcp-backups-state-bucket-name>"
    ```
 
+1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
+   the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
+
+1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run:
+
+   ```shell
+   sudo touch /etc/gitlab/skip-auto-reconfigure
+   ```
+
+   Only a single designated node should handle migrations as detailed in the
+   [GitLab Rails post-configuration](#gitlab-rails-post-configuration) section.
+
 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
+
 1. Verify the GitLab services are running:
 
    ```shell
@@ -1728,7 +1747,7 @@ On each node perform the following:
    })
 
    ## Disable components that will not be on the GitLab application server
-   roles ['application_role']
+   roles(['application_role'])
    gitaly['enable'] = false
    nginx['enable'] = true
    sidekiq['enable'] = false
@@ -1793,10 +1812,7 @@ On each node perform the following:
    #registry['uid'] = 9002
    #registry['gid'] = 9002
 
-   #############################
-   ###     Object storage    ###
-   #############################
-
+   # Object storage
    # This is an example for configuring Object Storage on GCP
    # Replace this config with your chosen Object Storage provider as desired
    gitlab_rails['object_store']['connection'] = {
@@ -1811,6 +1827,13 @@ On each node perform the following:
    gitlab_rails['object_store']['objects']['packages']['bucket'] = "<gcp-packages-bucket-name>"
    gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = "<gcp-dependency-proxy-bucket-name>"
    gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = "<gcp-terraform-state-bucket-name>"
+
+   gitlab_rails['backup_upload_connection'] = {
+     'provider' => 'Google',
+     'google_project' => '<gcp-project-name>',
+     'google_json_key_location' => '<path-to-gcp-service-account-key>'
+   }
+   gitlab_rails['backup_upload_remote_directory'] = "<gcp-backups-state-bucket-name>"
    ```
 
 1. If you're using [Gitaly with TLS support](#gitaly-cluster-tls-support), make sure the
@@ -1831,7 +1854,20 @@ On each node perform the following:
       sudo cp cert.pem /etc/gitlab/trusted-certs/
       ```
 
-1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
+1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
+   the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
+
+1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run:
+
+   ```shell
+   sudo touch /etc/gitlab/skip-auto-reconfigure
+   ```
+
+   Only a single designated node should handle migrations as detailed in the
+   [GitLab Rails post-configuration](#gitlab-rails-post-configuration) section.
+
+1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
+
 1. Run `sudo gitlab-rake gitlab:gitaly:check` to confirm the node can connect to Gitaly.
 1. Tail the logs to see the requests:
 
@@ -1839,11 +1875,6 @@ On each node perform the following:
    sudo gitlab-ctl tail gitaly
    ```
 
-1. Save the `/etc/gitlab/gitlab-secrets.json` file from one of the two
-   application nodes and install it on the other application node, the
-   [Gitaly node](#configure-gitaly) and the [Sidekiq node](#configure-sidekiq) and
-   [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
-
 1. Verify the GitLab services are running:
 
    ```shell
@@ -1902,45 +1933,26 @@ running [Prometheus](../monitoring/prometheus/index.md) and
 1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
 
    ```ruby
-   external_url 'http://gitlab.example.com'
+   roles(['monitoring_role', 'consul_role'])
 
-   # Disable all other services
-   alertmanager['enable'] = false
-   gitaly['enable'] = false
-   gitlab_exporter['enable'] = false
-   gitlab_workhorse['enable'] = false
-   nginx['enable'] = true
-   postgres_exporter['enable'] = false
-   postgresql['enable'] = false
-   redis['enable'] = false
-   redis_exporter['enable'] = false
-   sidekiq['enable'] = false
-   puma['enable'] = false
-   unicorn['enable'] = false
-   node_exporter['enable'] = false
-   gitlab_exporter['enable'] = false
+   external_url 'http://gitlab.example.com'
 
-   # Enable Prometheus
-   prometheus['enable'] = true
+   # Prometheus
    prometheus['listen_address'] = '0.0.0.0:9090'
    prometheus['monitor_kubernetes'] = false
 
-   # Enable Login form
-   grafana['disable_login_form'] = false
-
-   # Enable Grafana
-   grafana['enable'] = true
+   # Grafana
    grafana['admin_password'] = '<grafana_password>'
+   grafana['disable_login_form'] = false
 
    # Enable service discovery for Prometheus
-   consul['enable'] = true
    consul['monitoring_service_discovery'] =  true
    consul['configuration'] = {
       retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13)
    }
 
-   # Prevent database migrations from running on upgrade automatically
-   gitlab_rails['auto_migrate'] = false
+   # Nginx - For Grafana access
+   nginx['enable'] = true
    ```
 
 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
@@ -2074,7 +2086,7 @@ but with smaller performance requirements, several modifications can be consider
   - PostgreSQL: Can be run on reputable Cloud PaaS solutions such as Google Cloud SQL or AWS RDS. In this setup, the PgBouncer and Consul nodes are no longer required:
     - Consul may still be desired if [Prometheus](../monitoring/prometheus/index.md) auto discovery is a requirement, otherwise you would need to [manually add scrape configurations](../monitoring/prometheus/index.md#adding-custom-scrape-configurations) for all nodes.
       - As Redis Sentinel runs on the same box as Consul in this architecture, it may need to be run on a separate box if Redis is still being run via Omnibus.
-  - Redis: Can be run on reputable Cloud PaaS solutions such as Google Memorystore and AWS Elasticache. In this setup, the Redis Sentinel is no longer required.
+  - Redis: Can be run on reputable Cloud PaaS solutions such as Google Memorystore and AWS ElastiCache. In this setup, the Redis Sentinel is no longer required.
 
 <div align="right">
   <a type="button" class="btn btn-default" href="#setup-components">
diff --git a/doc/administration/reference_architectures/50k_users.md b/doc/administration/reference_architectures/50k_users.md
index 5cc463c953d..b3324cb75fb 100644
--- a/doc/administration/reference_architectures/50k_users.md
+++ b/doc/administration/reference_architectures/50k_users.md
@@ -17,29 +17,34 @@ full list of reference architectures, see
 
 | Service                                  | Nodes       | Configuration           | GCP              | AWS           | Azure     |
 |------------------------------------------|-------------|-------------------------|------------------|---------------|-----------|
-| External load balancing node             | 1           | 8 vCPU, 7.2 GB memory   | `n1-highcpu-8`   | `c5.2xlarge`  | `F8s v2`  |
-| Consul*                                  | 3           | 2 vCPU, 1.8 GB memory   | `n1-highcpu-2`   | `c5.large`    | `F2s v2`  |
-| PostgreSQL*                              | 3           | 32 vCPU, 120 GB memory  | `n1-standard-32` | `m5.8xlarge`  | `D32s v3` |
-| PgBouncer*                               | 3           | 2 vCPU, 1.8 GB memory   | `n1-highcpu-2`   | `c5.large`    | `F2s v2`  |
-| Internal load balancing node             | 1           | 8 vCPU, 7.2 GB memory   | `n1-highcpu-8`   | `c5.2xlarge`  | `F8s v2`  |
-| Redis - Cache**                          | 3           | 4 vCPU, 15 GB memory    | `n1-standard-4`  | `m5.xlarge`   | `D4s v3`  |
-| Redis - Queues / Shared State**          | 3           | 4 vCPU, 15 GB memory    | `n1-standard-4`  | `m5.xlarge`   | `D4s v3`  |
-| Redis Sentinel - Cache**                 | 3           | 1 vCPU, 3.75 GB memory  | `n1-standard-1`  | `c5.large`    | `A1 v2`   |
-| Redis Sentinel - Queues / Shared State** | 3           | 1 vCPU, 3.75 GB memory  | `n1-standard-1`  | `c5.large`    | `A1 v2`   |
+| External load balancing node(3)          | 1           | 8 vCPU, 7.2 GB memory   | `n1-highcpu-8`   | `c5.2xlarge`  | `F8s v2`  |
+| Consul(1)                                | 3           | 2 vCPU, 1.8 GB memory   | `n1-highcpu-2`   | `c5.large`    | `F2s v2`  |
+| PostgreSQL(1)                            | 3           | 32 vCPU, 120 GB memory  | `n1-standard-32` | `m5.8xlarge`  | `D32s v3` |
+| PgBouncer(1)                             | 3           | 2 vCPU, 1.8 GB memory   | `n1-highcpu-2`   | `c5.large`    | `F2s v2`  |
+| Internal load balancing node(3)          | 1           | 8 vCPU, 7.2 GB memory   | `n1-highcpu-8`   | `c5.2xlarge`  | `F8s v2`  |
+| Redis - Cache(2)                         | 3           | 4 vCPU, 15 GB memory    | `n1-standard-4`  | `m5.xlarge`   | `D4s v3`  |
+| Redis - Queues / Shared State(2)         | 3           | 4 vCPU, 15 GB memory    | `n1-standard-4`  | `m5.xlarge`   | `D4s v3`  |
+| Redis Sentinel - Cache(2)                | 3           | 1 vCPU, 3.75 GB memory  | `n1-standard-1`  | `c5.large`    | `A1 v2`   |
+| Redis Sentinel - Queues / Shared State(2)| 3           | 1 vCPU, 3.75 GB memory  | `n1-standard-1`  | `c5.large`    | `A1 v2`   |
 | Gitaly                                   | 3           | 64 vCPU, 240 GB memory  | `n1-standard-64` | `m5.16xlarge` | `D64s v3` |
 | Praefect                                 | 3           | 4 vCPU, 3.6 GB memory   | `n1-highcpu-4`   | `c5.xlarge`   | `F4s v2`  |
-| Praefect PostgreSQL*                     | 1+          | 2 vCPU, 1.8 GB memory   | `n1-highcpu-2`   | `c5.large`    | `F2s v2`  |
+| Praefect PostgreSQL(1)                   | 1+          | 2 vCPU, 1.8 GB memory   | `n1-highcpu-2`   | `c5.large`    | `F2s v2`  |
 | Sidekiq                                  | 4           | 4 vCPU, 15 GB memory    | `n1-standard-4`  | `m5.xlarge`   | `D4s v3`  |
 | GitLab Rails                             | 12          | 32 vCPU, 28.8 GB memory | `n1-highcpu-32`  | `c5.9xlarge`  | `F32s v2` |
 | Monitoring node                          | 1           | 4 vCPU, 3.6 GB memory   | `n1-highcpu-4`   | `c5.xlarge`   | `F4s v2`  |
-| Object storage                           | n/a         | n/a                     | n/a              | n/a           | n/a       |
-| NFS server                               | 1           | 4 vCPU, 3.6 GB memory   | `n1-highcpu-4`   | `c5.xlarge`   | `F4s v2`  |
+| Object storage(4)                        | n/a         | n/a                     | n/a              | n/a           | n/a       |
+| NFS server (optional, not recommended)   | 1           | 4 vCPU, 3.6 GB memory   | `n1-highcpu-4`   | `c5.xlarge`  | `F4s v2`   |
+
+<!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix -->
+<!-- markdownlint-disable MD029 -->
+1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work, however Azure Database for PostgreSQL is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to performance issues. Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery.
+2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work.
+3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work.
+4. Should be run on reputable third party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work.
+<!-- markdownlint-enable MD029 -->
 
 NOTE:
-Components marked with * can be optionally run on reputable
-third party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work.
-Components marked with ** can be optionally run on reputable
-third party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work.
+For all PaaS solutions that involve configuring instances, it is strongly recommended to implement a minimum of three nodes in three different availability zones to align with resilient cloud architecture practices.
 
 ```plantuml
 @startuml 50k
@@ -157,7 +162,7 @@ To set up GitLab and its components to accommodate up to 50,000 users:
    provides access to the Git repositories.
 1. [Configure Sidekiq](#configure-sidekiq).
 1. [Configure the main GitLab Rails application](#configure-gitlab-rails)
-   to run Puma/Unicorn, Workhorse, GitLab Shell, and to serve all frontend
+   to run Puma, Workhorse, GitLab Shell, and to serve all frontend
    requests (which include UI, API, and Git over HTTP/SSH).
 1. [Configure Prometheus](#configure-prometheus) to monitor your GitLab
    environment.
@@ -420,11 +425,6 @@ The following IPs will be used as an example:
 - `10.6.0.12`: Consul 2
 - `10.6.0.13`: Consul 3
 
-NOTE:
-The configuration processes for the other servers in your reference architecture will
-use the `/etc/gitlab/gitlab-secrets.json` file from your Consul server to connect
-with the other servers.
-
 To configure Consul:
 
 1. SSH in to the server that will host Consul.
@@ -435,10 +435,9 @@ To configure Consul:
 1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
 
    ```ruby
-   roles ['consul_role']
+   roles(['consul_role'])
 
    ## Enable service discovery for Prometheus
-   consul['enable'] = true
    consul['monitoring_service_discovery'] =  true
 
    ## The IPs of the Consul server nodes
@@ -455,7 +454,11 @@ To configure Consul:
    gitlab_rails['auto_migrate'] = false
    ```
 
+1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
+   the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
+
 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
+
 1. Go through the steps again for all the other Consul nodes, and
    make sure you set up the correct IPs.
 
@@ -547,6 +550,15 @@ in the second step, do not supply the `EXTERNAL_URL` value.
    sudo gitlab-ctl pg-password-md5 pgbouncer
    ```
 
+1. Generate a password hash for the PostgreSQL replication username/password pair. This assumes you will use the default
+   username of `gitlab_replicator` (recommended). The command will request a password
+   and a confirmation. Use the value that is output by this command in the next step
+   as the value of `<postgresql_replication_password_hash>`:
+
+   ```shell
+   sudo gitlab-ctl pg-password-md5 gitlab_replicator
+   ```
+
 1. Generate a password hash for the Consul database username/password pair. This assumes you will use the default
    username of `gitlab-consul` (recommended). The command will request a password
    and confirmation. Use the value that is output by this command in the next
@@ -559,19 +571,21 @@ in the second step, do not supply the `EXTERNAL_URL` value.
 1. On every database node, edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section:
 
    ```ruby
-   # Disable all components except PostgreSQL, Patroni, and Consul
-   roles ['postgres_role']
+   # Disable all components except Patroni and Consul
+   roles(['patroni_role'])
 
    # PostgreSQL configuration
    postgresql['listen_address'] = '0.0.0.0'
 
-   # Enable Patroni
-   patroni['enable'] = true
-   # Set `max_wal_senders` to one more than the number of database nodes in the cluster.
+   # Sets `max_replication_slots` to double the number of database nodes.
+   # Patroni uses one extra slot per node when initiating the replication.
+   patroni['postgresql']['max_replication_slots'] = 8
+
+   # Set `max_wal_senders` to one more than the number of replication slots in the cluster.
    # This is used to prevent replication from using up all of the
    # available database connections.
-   patroni['postgresql']['max_wal_senders'] = 4
-   patroni['postgresql']['max_replication_slots'] = 4
+   patroni['postgresql']['max_wal_senders'] = 9
+
    # Incoming recommended value for max connections is 500. See https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5691.
    patroni['postgresql']['max_connections'] = 500
 
@@ -589,6 +603,8 @@ in the second step, do not supply the `EXTERNAL_URL` value.
    #
    # Replace PGBOUNCER_PASSWORD_HASH with a generated md5 value
    postgresql['pgbouncer_user_password'] = '<pgbouncer_password_hash>'
+   # Replace POSTGRESQL_REPLICATION_PASSWORD_HASH with a generated md5 value
+   postgresql['sql_replication_password'] = '<postgresql_replication_password_hash>'
    # Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value
    postgresql['sql_user_password'] = '<postgresql_password_hash>'
 
@@ -612,9 +628,8 @@ PostgreSQL, with Patroni managing its failover, will default to use `pg_rewind`
 Like most failover handling methods, this has a small chance of leading to data loss.
 Learn more about the various [Patroni replication methods](../postgresql/replication_and_failover.md#selecting-the-appropriate-patroni-replication-method).
 
-1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace
-   the file of the same name on this server. If that file is not on this server,
-   add the file from your Consul server to this server.
+1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
+   the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
 
 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
 
@@ -629,21 +644,7 @@ are supported and can be added if needed.
 
 #### PostgreSQL post-configuration
 
-SSH in to the **primary node**:
-
-1. Open a database prompt:
-
-   ```shell
-   gitlab-psql -d gitlabhq_production
-   ```
-
-1. Make sure the `pg_trgm` extension is enabled (it might already be):
-
-   ```shell
-   CREATE EXTENSION pg_trgm;
-   ```
-
-1. Exit the database prompt by typing `\q` and Enter.
+SSH in to any of the Patroni nodes on the **primary site**:
 
 1. Check the status of the leader and cluster:
 
@@ -685,7 +686,7 @@ The following IPs will be used as an example:
 
    ```ruby
    # Disable all components except Pgbouncer and Consul agent
-   roles ['pgbouncer_role']
+   roles(['pgbouncer_role'])
 
    # Configure PgBouncer
    pgbouncer['admin_users'] = %w(pgbouncer gitlab-consul)
@@ -702,7 +703,6 @@ The following IPs will be used as an example:
 
    # Configure Consul agent
    consul['watchers'] = %w(postgresql)
-   consul['enable'] = true
    consul['configuration'] = {
    retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13)
    }
@@ -714,9 +714,8 @@ The following IPs will be used as an example:
    node_exporter['listen_address'] = '0.0.0.0:9100'
    ```
 
-1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace
-   the file of the same name on this server. If that file is not on this server,
-   add the file from your Consul server to this server.
+1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
+   the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
 
 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
 
@@ -835,8 +834,8 @@ a node and change its status from primary to replica (and vice versa).
 1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
 
    ```ruby
-   # Specify server role as 'redis_master_role'
-   roles ['redis_master_role']
+   # Specify server role as 'redis_master_role' and enable Consul agent
+   roles(['redis_master_role', 'consul_role'])
 
    # IP address pointing to a local IP that the other machines can reach to.
    # You can also set bind to '0.0.0.0' which listen in all interfaces.
@@ -858,7 +857,6 @@ a node and change its status from primary to replica (and vice versa).
    redis['maxmemory_samples'] = 5
 
    ## Enable service discovery for Prometheus
-   consul['enable'] = true
    consul['monitoring_service_discovery'] =  true
 
    ## The IPs of the Consul server nodes
@@ -870,19 +868,22 @@ a node and change its status from primary to replica (and vice versa).
    # Set the network addresses that the exporters will listen on
    node_exporter['listen_address'] = '0.0.0.0:9100'
    redis_exporter['listen_address'] = '0.0.0.0:9121'
-
+   redis_exporter['flags'] = {
+        'redis.addr' => 'redis://10.6.0.51:6379',
+        'redis.password' => 'redis-password-goes-here',
+   }
+   
    # Prevent database migrations from running on upgrade automatically
    gitlab_rails['auto_migrate'] = false
    ```
 
-1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace
-   the file of the same name on this server. If that file is not on this server,
-   add the file from your Consul server to this server.
+1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
+   the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
 
 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
 
 You can specify multiple roles, like sentinel and Redis, as:
-`roles ['redis_sentinel_role', 'redis_master_role']`. Read more about
+`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about
 [roles](https://docs.gitlab.com/omnibus/roles/).
 
 #### Configure the replica Redis Cache nodes
@@ -895,8 +896,8 @@ You can specify multiple roles, like sentinel and Redis, as:
 1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
 
    ```ruby
-   # Specify server role as 'redis_replica_role'
-   roles ['redis_replica_role']
+   # Specify server role as 'redis_replica_role' and enable Consul agent
+   roles(['redis_replica_role', 'consul_role'])
 
    # IP address pointing to a local IP that the other machines can reach to.
    # You can also set bind to '0.0.0.0' which listen in all interfaces.
@@ -925,7 +926,6 @@ You can specify multiple roles, like sentinel and Redis, as:
    redis['maxmemory_samples'] = 5
 
    ## Enable service discovery for Prometheus
-   consul['enable'] = true
    consul['monitoring_service_discovery'] =  true
 
    ## The IPs of the Consul server nodes
@@ -937,21 +937,25 @@ You can specify multiple roles, like sentinel and Redis, as:
    # Set the network addresses that the exporters will listen on
    node_exporter['listen_address'] = '0.0.0.0:9100'
    redis_exporter['listen_address'] = '0.0.0.0:9121'
+   redis_exporter['flags'] = {
+        'redis.addr' => 'redis://10.6.0.52:6379',
+        'redis.password' => 'redis-password-goes-here',
+   }
 
    # Prevent database migrations from running on upgrade automatically
    gitlab_rails['auto_migrate'] = false
    ```
 
-1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace
-   the file of the same name on this server. If that file is not on this server,
-   add the file from your Consul server to this server.
+1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
+   the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
 
 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
+
 1. Go through the steps again for all the other replica nodes, and
    make sure to set up the IPs correctly.
 
 You can specify multiple roles, like sentinel and Redis, as:
-`roles ['redis_sentinel_role', 'redis_master_role']`. Read more about
+`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about
 [roles](https://docs.gitlab.com/omnibus/roles/).
 
 These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after
@@ -993,7 +997,7 @@ To configure the Sentinel Cache server:
 1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
 
    ```ruby
-   roles ['redis_sentinel_role']
+   roles(['redis_sentinel_role', 'consul_role'])
 
    ## Must be the same in every sentinel node
    redis['master_name'] = 'gitlab-redis-cache'
@@ -1057,7 +1061,6 @@ To configure the Sentinel Cache server:
    #sentinel['failover_timeout'] = 60000
 
    ## Enable service discovery for Prometheus
-   consul['enable'] = true
    consul['monitoring_service_discovery'] =  true
 
    ## The IPs of the Consul server nodes
@@ -1074,11 +1077,11 @@ To configure the Sentinel Cache server:
    gitlab_rails['auto_migrate'] = false
    ```
 
-1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace
-   the file of the same name on this server. If that file is not on this server,
-   add the file from your Consul server to this server.
+1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
+   the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
 
 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
+
 1. Go through the steps again for all the other Consul/Sentinel nodes, and
    make sure you set up the correct IPs.
 
@@ -1106,8 +1109,8 @@ a node and change its status from primary to replica (and vice versa).
 1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
 
    ```ruby
-   # Specify server role as 'redis_master_role'
-   roles ['redis_master_role']
+   # Specify server role as 'redis_master_role' and enable Consul agent
+   roles(['redis_master_role', 'consul_role'])
 
    # IP address pointing to a local IP that the other machines can reach to.
    # You can also set bind to '0.0.0.0' which listen in all interfaces.
@@ -1123,7 +1126,6 @@ a node and change its status from primary to replica (and vice versa).
    redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER'
 
    ## Enable service discovery for Prometheus
-   consul['enable'] = true
    consul['monitoring_service_discovery'] =  true
 
    ## The IPs of the Consul server nodes
@@ -1140,9 +1142,8 @@ a node and change its status from primary to replica (and vice versa).
    gitlab_rails['auto_migrate'] = false
    ```
 
-1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace
-   the file of the same name on this server. If that file is not on this server,
-   add the file from your Consul server to this server.
+1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
+   the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
 
 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
 
@@ -1160,8 +1161,8 @@ You can specify multiple roles, like sentinel and Redis, as:
 1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
 
    ```ruby
-   # Specify server role as 'redis_replica_role'
-   roles ['redis_replica_role']
+   # Specify server role as 'redis_replica_role' and enable Consul agent
+   roles(['redis_replica_role', 'consul_role'])
 
    # IP address pointing to a local IP that the other machines can reach to.
    # You can also set bind to '0.0.0.0' which listen in all interfaces.
@@ -1184,7 +1185,6 @@ You can specify multiple roles, like sentinel and Redis, as:
    #redis['master_port'] = 6379
 
    ## Enable service discovery for Prometheus
-   consul['enable'] = true
    consul['monitoring_service_discovery'] =  true
 
    ## The IPs of the Consul server nodes
@@ -1201,16 +1201,16 @@ You can specify multiple roles, like sentinel and Redis, as:
    gitlab_rails['auto_migrate'] = false
    ```
 
-1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace
-   the file of the same name on this server. If that file is not on this server,
-   add the file from your Consul server to this server.
+1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
+   the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
 
 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
+
 1. Go through the steps again for all the other replica nodes, and
    make sure to set up the IPs correctly.
 
 You can specify multiple roles, like sentinel and Redis, as:
-`roles ['redis_sentinel_role', 'redis_master_role']`. Read more about
+`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about
 [roles](https://docs.gitlab.com/omnibus/roles/).
 
 These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after
@@ -1252,7 +1252,7 @@ To configure the Sentinel Queues server:
 1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
 
    ```ruby
-   roles ['redis_sentinel_role']
+   roles(['redis_sentinel_role', 'consul_role'])
 
    ## Must be the same in every sentinel node
    redis['master_name'] = 'gitlab-redis-persistent'
@@ -1316,7 +1316,6 @@ To configure the Sentinel Queues server:
    #sentinel['failover_timeout'] = 60000
 
    ## Enable service discovery for Prometheus
-   consul['enable'] = true
    consul['monitoring_service_discovery'] =  true
 
    ## The IPs of the Consul server nodes
@@ -1333,7 +1332,7 @@ To configure the Sentinel Queues server:
    gitlab_rails['auto_migrate'] = false
    ```
 
-1. To prevent database migrations from running on upgrade, run:
+1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run:
 
    ```shell
    sudo touch /etc/gitlab/skip-auto-reconfigure
@@ -1341,11 +1340,11 @@ To configure the Sentinel Queues server:
 
    Only the primary GitLab application server should handle migrations.
 
-1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace
-   the file of the same name on this server. If that file is not on this server,
-   add the file from your Consul server to this server.
+1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
+   the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
 
 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
+
 1. Go through the steps again for all the other Sentinel nodes, and
    make sure you set up the correct IPs.
 
@@ -1406,9 +1405,7 @@ in the second step, do not supply the `EXTERNAL_URL` value.
 
    ```ruby
    # Disable all components except PostgreSQL and Consul
-   roles ['postgres_role']
-   repmgr['enable'] = false
-   patroni['enable'] = false
+   roles(['postgres_role', 'consul_role'])
 
    # PostgreSQL configuration
    postgresql['listen_address'] = '0.0.0.0'
@@ -1418,7 +1415,6 @@ in the second step, do not supply the `EXTERNAL_URL` value.
    gitlab_rails['auto_migrate'] = false
 
    # Configure the Consul agent
-   consul['enable'] = true
    ## Enable service discovery for Prometheus
    consul['monitoring_service_discovery'] =  true
 
@@ -1444,7 +1440,11 @@ in the second step, do not supply the `EXTERNAL_URL` value.
    # END user configuration
    ```
 
+1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
+   the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
+
 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
+
 1. Follow the [post configuration](#praefect-postgresql-post-configuration).
 
 <div align="right">
@@ -1542,19 +1542,18 @@ To configure the Praefect nodes, on each one:
 1. Edit the `/etc/gitlab/gitlab.rb` file to configure Praefect:
 
    ```ruby
-   # Avoid running unnecessary services on the Gitaly server
+   # Avoid running unnecessary services on the Praefect server
+   gitaly['enable'] = false
    postgresql['enable'] = false
    redis['enable'] = false
-   nginx['enable'] = false
    puma['enable'] = false
-   unicorn['enable'] = false
    sidekiq['enable'] = false
    gitlab_workhorse['enable'] = false
-   grafana['enable'] = false
-
-   # If you run a separate monitoring node you can disable these services
-   alertmanager['enable'] = false
    prometheus['enable'] = false
+   alertmanager['enable'] = false
+   grafana['enable'] = false
+   gitlab_exporter['enable'] = false
+   nginx['enable'] = false
 
    # Praefect Configuration
    praefect['enable'] = true
@@ -1592,19 +1591,20 @@ To configure the Praefect nodes, on each one:
    # server ('praefect') and in git_data_dirs on Gitaly nodes ('gitaly-1')
    praefect['virtual_storages'] = {
      'default' => {
-       'gitaly-1' => {
-         'address' => 'tcp://10.6.0.91:8075',
-         'token'   => '<praefect_internal_token>',
-         'primary' => true
-       },
-       'gitaly-2' => {
-         'address' => 'tcp://10.6.0.92:8075',
-         'token'   => '<praefect_internal_token>'
-       },
-       'gitaly-3' => {
-         'address' => 'tcp://10.6.0.93:8075',
-         'token'   => '<praefect_internal_token>'
-       },
+       'nodes' => {
+         'gitaly-1' => {
+           'address' => 'tcp://10.6.0.91:8075',
+           'token'   => '<praefect_internal_token>'
+         },
+         'gitaly-2' => {
+           'address' => 'tcp://10.6.0.92:8075',
+           'token'   => '<praefect_internal_token>'
+         },
+         'gitaly-3' => {
+           'address' => 'tcp://10.6.0.93:8075',
+           'token'   => '<praefect_internal_token>'
+         },
+       }
      }
    }
 
@@ -1621,11 +1621,25 @@ To configure the Praefect nodes, on each one:
    # END user configuration
    ```
 
-   1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and
-      then replace the file of the same name on this server. If that file isn't on
-      this server, add the file from your Consul server to this server.
+1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
+the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
 
-   1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
+1. Praefect requires to run some database migrations, much like the main GitLab application. For this
+   you should select **one Praefect node only to run the migrations**, AKA the _Deploy Node_. This node
+   must be configured first before the others as follows:
+   
+   1. In the `/etc/gitlab/gitlab.rb` file, change the `praefect['auto_migrate']` setting value from `false` to `true`
+
+   1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run:
+
+   ```shell
+   sudo touch /etc/gitlab/skip-auto-reconfigure
+   ```
+   
+   1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect and
+      to run the Praefect database migrations.
+
+1. On all other Praefect nodes, [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
 
 ### Configure Gitaly
 
@@ -1669,21 +1683,17 @@ On each node:
    storage paths, enable the network listener, and to configure the token:
 
    ```ruby
-   # /etc/gitlab/gitlab.rb
-
    # Avoid running unnecessary services on the Gitaly server
    postgresql['enable'] = false
    redis['enable'] = false
-   nginx['enable'] = false
    puma['enable'] = false
-   unicorn['enable'] = false
    sidekiq['enable'] = false
    gitlab_workhorse['enable'] = false
-   grafana['enable'] = false
-
-   # If you run a separate monitoring node you can disable these services
-   alertmanager['enable'] = false
    prometheus['enable'] = false
+   alertmanager['enable'] = false
+   grafana['enable'] = false
+   gitlab_exporter['enable'] = false
+   nginx['enable'] = false
 
    # Prevent database migrations from running on upgrade automatically
    gitlab_rails['auto_migrate'] = false
@@ -1691,9 +1701,11 @@ On each node:
    # Configure the gitlab-shell API callback URL. Without this, `git push` will
    # fail. This can be your 'front door' GitLab URL or an internal load
    # balancer.
-   # Don't forget to copy `/etc/gitlab/gitlab-secrets.json` from web server to Gitaly server.
    gitlab_rails['internal_api_url'] = 'https://gitlab.example.com'
 
+   # Gitaly 
+   gitaly['enable'] = true
+
    # Make Gitaly accept connections on all network interfaces. You must use
    # firewalls to restrict access to this address/port.
    # Comment out following line if you only want to support TLS connections
@@ -1735,9 +1747,8 @@ On each node:
      })
      ```
 
-1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and
-   then replace the file of the same name on this server. If that file isn't on
-   this server, add the file from your Consul server to this server.
+1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
+   the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
 
 1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
 
@@ -1844,28 +1855,19 @@ To configure the Sidekiq nodes, on each one:
 1. Open `/etc/gitlab/gitlab.rb` with your editor:
 
    ```ruby
-   ########################################
-   #####        Services Disabled       ###
-   ########################################
-
-   nginx['enable'] = false
-   grafana['enable'] = false
-   prometheus['enable'] = false
-   alertmanager['enable'] = false
+   # Avoid running unnecessary services on the Sidekiq server
    gitaly['enable'] = false
-   gitlab_workhorse['enable'] = false
-   nginx['enable'] = false
-   puma['enable'] = false
-   postgres_exporter['enable'] = false
    postgresql['enable'] = false
    redis['enable'] = false
-   redis_exporter['enable'] = false
+   puma['enable'] = false
+   gitlab_workhorse['enable'] = false
+   prometheus['enable'] = false
+   alertmanager['enable'] = false
+   grafana['enable'] = false
    gitlab_exporter['enable'] = false
+   nginx['enable'] = false
 
-   ########################################
-   ####              Redis              ###
-   ########################################
-
+   # Redis
    ## Redis connection details
    ## First cluster that will host the cache
    gitlab_rails['redis_cache_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER>@gitlab-redis-cache'
@@ -1897,10 +1899,7 @@ To configure the Sidekiq nodes, on each one:
      {host: '10.6.0.83', port: 26379},
    ]
 
-   #######################################
-   ###              Gitaly             ###
-   #######################################
-
+   # Gitaly
    # git_data_dirs get configured for the Praefect virtual storage
    # Address is Internal Load Balancer for Praefect
    # Token is praefect_external_token
@@ -1911,31 +1910,26 @@ To configure the Sidekiq nodes, on each one:
      }
    })
 
-   #######################################
-   ###            Postgres             ###
-   #######################################
+   # PostgreSQL
    gitlab_rails['db_host'] = '10.6.0.20' # internal load balancer IP
    gitlab_rails['db_port'] = 6432
    gitlab_rails['db_password'] = '<postgresql_user_password>'
    gitlab_rails['db_adapter'] = 'postgresql'
    gitlab_rails['db_encoding'] = 'unicode'
-   # Prevent database migrations from running on upgrade automatically
+   ## Prevent database migrations from running on upgrade automatically
    gitlab_rails['auto_migrate'] = false
 
-   #######################################
-   ###      Sidekiq configuration      ###
-   #######################################
+   # Sidekiq
+   sidekiq['enable'] = true
    sidekiq['listen_address'] = "0.0.0.0"
 
-   # Set number of Sidekiq queue processes to the same number as available CPUs
+   ## Set number of Sidekiq queue processes to the same number as available CPUs
    sidekiq['queue_groups'] = ['*'] * 4
 
-   # Set number of Sidekiq threads per queue process to the recommend number of 10
+   ## Set number of Sidekiq threads per queue process to the recommend number of 10
    sidekiq['max_concurrency'] = 10
 
-   #######################################
-   ###     Monitoring configuration    ###
-   #######################################
+   # Monitoring 
    consul['enable'] = true
    consul['monitoring_service_discovery'] =  true
 
@@ -1946,15 +1940,12 @@ To configure the Sidekiq nodes, on each one:
    # Set the network addresses that the exporters will listen on
    node_exporter['listen_address'] = '0.0.0.0:9100'
 
-   # Rails Status for prometheus
+   ## Add the monitoring node's IP address to the monitoring whitelist
    gitlab_rails['monitoring_whitelist'] = ['10.6.0.151/32', '127.0.0.0/8']
 
-   #############################
-   ###     Object storage    ###
-   #############################
-
-   # This is an example for configuring Object Storage on GCP
-   # Replace this config with your chosen Object Storage provider as desired
+   # Object storage
+   ## This is an example for configuring Object Storage on GCP
+   ## Replace this config with your chosen Object Storage provider as desired
    gitlab_rails['object_store']['connection'] = {
      'provider' => 'Google',
      'google_project' => '<gcp-project-name>',
@@ -1967,11 +1958,26 @@ To configure the Sidekiq nodes, on each one:
    gitlab_rails['object_store']['objects']['packages']['bucket'] = "<gcp-packages-bucket-name>"
    gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = "<gcp-dependency-proxy-bucket-name>"
    gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = "<gcp-terraform-state-bucket-name>"
+
+   gitlab_rails['backup_upload_connection'] = {
+     'provider' => 'Google',
+     'google_project' => '<gcp-project-name>',
+     'google_json_key_location' => '<path-to-gcp-service-account-key>'
+   }
+   gitlab_rails['backup_upload_remote_directory'] = "<gcp-backups-state-bucket-name>"
+   ```
+
+1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
+   the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
+
+1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run:
+
+   ```shell
+   sudo touch /etc/gitlab/skip-auto-reconfigure
    ```
 
-1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace
-   the file of the same name on this server. If that file is not on this server,
-   add the file from your Consul server to this server.
+   Only a single designated node should handle migrations as detailed in the
+   [GitLab Rails post-configuration](#gitlab-rails-post-configuration) section.
 
 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
 
@@ -2011,9 +2017,6 @@ On each node perform the following:
 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab
    package of your choice. Be sure to follow _only_ installation steps 1 and 2
    on the page.
-1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace
-   the file of the same name on this server. If that file is not on this server,
-   add the file from your Consul server to this server.
 
 1. Edit `/etc/gitlab/gitlab.rb` and use the following configuration.
    To maintain uniformity of links across nodes, the `external_url`
@@ -2035,7 +2038,7 @@ On each node perform the following:
    })
 
    ## Disable components that will not be on the GitLab application server
-   roles ['application_role']
+   roles(['application_role'])
    gitaly['enable'] = false
    nginx['enable'] = true
    sidekiq['enable'] = false
@@ -2108,9 +2111,15 @@ On each node perform the following:
    gitlab_rails['object_store']['objects']['packages']['bucket'] = "<gcp-packages-bucket-name>"
    gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = "<gcp-dependency-proxy-bucket-name>"
    gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = "<gcp-terraform-state-bucket-name>"
+
+   gitlab_rails['backup_upload_connection'] = {
+     'provider' => 'Google',
+     'google_project' => '<gcp-project-name>',
+     'google_json_key_location' => '<path-to-gcp-service-account-key>'
+   }
+   gitlab_rails['backup_upload_remote_directory'] = "<gcp-backups-state-bucket-name>"
    ```
 
-1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
 1. If you're using [Gitaly with TLS support](#gitaly-cluster-tls-support), make sure the
    `git_data_dirs` entry is configured with `tls` instead of `tcp`:
 
@@ -2129,6 +2138,20 @@ On each node perform the following:
       sudo cp cert.pem /etc/gitlab/trusted-certs/
       ```
 
+1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
+   the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
+
+1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run:
+
+   ```shell
+   sudo touch /etc/gitlab/skip-auto-reconfigure
+   ```
+
+   Only a single designated node should handle migrations as detailed in the
+   [GitLab Rails post-configuration](#gitlab-rails-post-configuration) section.
+
+1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
+
 1. If you're [using NFS](#configure-nfs-optional):
    1. If necessary, install the NFS client utility packages using the following
       commands:
@@ -2168,7 +2191,7 @@ On each node perform the following:
       registry['gid'] = 9002
       ```
 
-1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
+   1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
 1. Confirm the node can connect to Gitaly:
 
    ```shell
@@ -2232,52 +2255,30 @@ To configure the Monitoring node:
 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab
    package of your choice. Be sure to follow _only_ installation steps 1 and 2
    on the page.
-1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace
-   the file of the same name on this server. If that file is not on this server,
-   add the file from your Consul server to this server.
 
 1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
 
    ```ruby
-   external_url 'http://gitlab.example.com'
+   roles(['monitoring_role', 'consul_role'])
 
-   # Disable all other services
-   alertmanager['enable'] = false
-   gitaly['enable'] = false
-   gitlab_exporter['enable'] = false
-   gitlab_workhorse['enable'] = false
-   nginx['enable'] = true
-   postgres_exporter['enable'] = false
-   postgresql['enable'] = false
-   redis['enable'] = false
-   redis_exporter['enable'] = false
-   sidekiq['enable'] = false
-   puma['enable'] = false
-   unicorn['enable'] = false
-   node_exporter['enable'] = false
-   gitlab_exporter['enable'] = false
+   external_url 'http://gitlab.example.com'
 
-   # Enable Prometheus
-   prometheus['enable'] = true
+   # Prometheus
    prometheus['listen_address'] = '0.0.0.0:9090'
    prometheus['monitor_kubernetes'] = false
 
-   # Enable Login form
-   grafana['disable_login_form'] = false
-
-   # Enable Grafana
-   grafana['enable'] = true
+   # Grafana
    grafana['admin_password'] = '<grafana_password>'
+   grafana['disable_login_form'] = false
 
    # Enable service discovery for Prometheus
-   consul['enable'] = true
    consul['monitoring_service_discovery'] =  true
    consul['configuration'] = {
       retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13)
    }
 
-   # Prevent database migrations from running on upgrade automatically
-   gitlab_rails['auto_migrate'] = false
+   # Nginx - For Grafana access
+   nginx['enable'] = true
    ```
 
 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
diff --git a/doc/administration/reference_architectures/5k_users.md b/doc/administration/reference_architectures/5k_users.md
index 792dd7020c7..9952df196c9 100644
--- a/doc/administration/reference_architectures/5k_users.md
+++ b/doc/administration/reference_architectures/5k_users.md
@@ -24,26 +24,31 @@ costly-to-operate environment by using the
 
 | Service                                    | Nodes       | Configuration           | GCP             | AWS          | Azure    |
 |--------------------------------------------|-------------|-------------------------|-----------------|--------------|----------|
-| External load balancing node               | 1           | 2 vCPU, 1.8 GB memory   | `n1-highcpu-2`  | `c5.large`   | `F2s v2` |
-| Redis**                                    | 3           | 2 vCPU, 7.5 GB memory   | `n1-standard-2` | `m5.large`   | `D2s v3` |
-| Consul* + Sentinel**                       | 3           | 2 vCPU, 1.8 GB memory   | `n1-highcpu-2`  | `c5.large`   | `F2s v2` |
-| PostgreSQL*                                | 3           | 4 vCPU, 15 GB memory    | `n1-standard-4` | `m5.xlarge`  | `D4s v3` |
-| PgBouncer*                                 | 3           | 2 vCPU, 1.8 GB memory   | `n1-highcpu-2`  | `c5.large`   | `F2s v2` |
-| Internal load balancing node               | 1           | 2 vCPU, 1.8 GB memory   | `n1-highcpu-2`  | `c5.large`   | `F2s v2` |
+| External load balancing node(3)            | 1           | 2 vCPU, 1.8 GB memory   | `n1-highcpu-2`  | `c5.large`   | `F2s v2` |
+| Redis(2)                                   | 3           | 2 vCPU, 7.5 GB memory   | `n1-standard-2` | `m5.large`   | `D2s v3` |
+| Consul(1) + Sentinel(2)                    | 3           | 2 vCPU, 1.8 GB memory   | `n1-highcpu-2`  | `c5.large`   | `F2s v2` |
+| PostgreSQL(1)                              | 3           | 4 vCPU, 15 GB memory    | `n1-standard-4` | `m5.xlarge`  | `D4s v3` |
+| PgBouncer(1)                               | 3           | 2 vCPU, 1.8 GB memory   | `n1-highcpu-2`  | `c5.large`   | `F2s v2` |
+| Internal load balancing node(3)            | 1           | 2 vCPU, 1.8 GB memory   | `n1-highcpu-2`  | `c5.large`   | `F2s v2` |
 | Gitaly                                     | 3           | 8 vCPU, 30 GB memory    | `n1-standard-8` | `m5.2xlarge` | `D8s v3` |
 | Praefect                                   | 3           | 2 vCPU, 1.8 GB memory   | `n1-highcpu-2`  | `c5.large`   | `F2s v2` |
-| Praefect PostgreSQL*                       | 1+          | 2 vCPU, 1.8 GB memory   | `n1-highcpu-2`  | `c5.large`   | `F2s v2` |
+| Praefect PostgreSQL(1)                     | 1+          | 2 vCPU, 1.8 GB memory   | `n1-highcpu-2`  | `c5.large`   | `F2s v2` |
 | Sidekiq                                    | 4           | 2 vCPU, 7.5 GB memory   | `n1-standard-2` | `m5.large`   | `D2s v3` |
 | GitLab Rails                               | 3           | 16 vCPU, 14.4 GB memory | `n1-highcpu-16` | `c5.4xlarge` | `F16s v2`|
 | Monitoring node                            | 1           | 2 vCPU, 1.8 GB memory   | `n1-highcpu-2`  | `c5.large`   | `F2s v2` |
-| Object storage                             | n/a         | n/a                     | n/a             | n/a          | n/a      |
+| Object storage(4)                          | n/a         | n/a                     | n/a             | n/a          | n/a      |
 | NFS server (optional, not recommended)     | 1           | 4 vCPU, 3.6 GB memory   | `n1-highcpu-4`  | `c5.xlarge`  | `F4s v2` |
 
+<!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix -->
+<!-- markdownlint-disable MD029 -->
+1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work, however Azure Database for PostgreSQL is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to performance issues. Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery.
+2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work.
+3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work.
+4. Should be run on reputable third party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work.
+<!-- markdownlint-enable MD029 -->
+
 NOTE:
-Components marked with * can be optionally run on reputable
-third party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work.
-Components marked with ** can be optionally run on reputable
-third party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work.
+For all PaaS solutions that involve configuring instances, it is strongly recommended to implement a minimum of three nodes in three different availability zones to align with resilient cloud architecture practices.
 
 ```plantuml
 @startuml 5k
@@ -161,7 +166,7 @@ To set up GitLab and its components to accommodate up to 5,000 users:
    provides access to the Git repositories.
 1. [Configure Sidekiq](#configure-sidekiq).
 1. [Configure the main GitLab Rails application](#configure-gitlab-rails)
-   to run Puma/Unicorn, Workhorse, GitLab Shell, and to serve all frontend
+   to run Puma, Workhorse, GitLab Shell, and to serve all frontend
    requests (which include UI, API, and Git over HTTP/SSH).
 1. [Configure Prometheus](#configure-prometheus) to monitor your GitLab
    environment.
@@ -462,8 +467,8 @@ a node and change its status from primary to replica (and vice versa).
 1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
 
    ```ruby
-   # Specify server role as 'redis_master_role'
-   roles ['redis_master_role']
+   # Specify server role as 'redis_master_role' and enable Consul agent
+   roles(['redis_master_role', 'consul_role'])
 
    # IP address pointing to a local IP that the other machines can reach to.
    # You can also set bind to '0.0.0.0' which listen in all interfaces.
@@ -479,7 +484,6 @@ a node and change its status from primary to replica (and vice versa).
    redis['password'] = 'redis-password-goes-here'
 
    ## Enable service discovery for Prometheus
-   consul['enable'] = true
    consul['monitoring_service_discovery'] =  true
 
    ## The IPs of the Consul server nodes
@@ -500,10 +504,13 @@ a node and change its status from primary to replica (and vice versa).
    gitlab_rails['auto_migrate'] = false
    ```
 
+1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
+   the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
+
 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
 
 You can specify multiple roles, like sentinel and Redis, as:
-`roles ['redis_sentinel_role', 'redis_master_role']`. Read more about
+`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about
 [roles](https://docs.gitlab.com/omnibus/roles/).
 
 You can list the current Redis Primary, Replica status via:
@@ -538,8 +545,8 @@ run: redis-exporter: (pid 30075) 76861s; run: log: (pid 29674) 76896s
 1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
 
    ```ruby
-   # Specify server role as 'redis_replica_role'
-   roles ['redis_replica_role']
+   # Specify server role as 'redis_replica_role' and enable Consul agent
+   roles(['redis_replica_role', 'consul_role'])
 
    # IP address pointing to a local IP that the other machines can reach to.
    # You can also set bind to '0.0.0.0' which listen in all interfaces.
@@ -562,7 +569,6 @@ run: redis-exporter: (pid 30075) 76861s; run: log: (pid 29674) 76896s
    #redis['master_port'] = 6379
 
    ## Enable service discovery for Prometheus
-   consul['enable'] = true
    consul['monitoring_service_discovery'] =  true
 
    ## The IPs of the Consul server nodes
@@ -583,12 +589,15 @@ run: redis-exporter: (pid 30075) 76861s; run: log: (pid 29674) 76896s
    gitlab_rails['auto_migrate'] = false
    ```
 
+1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
+   the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
+
 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
 1. Go through the steps again for all the other replica nodes, and
    make sure to set up the IPs correctly.
 
 You can specify multiple roles, like sentinel and Redis, as:
-`roles ['redis_sentinel_role', 'redis_master_role']`. Read more about
+`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about
 [roles](https://docs.gitlab.com/omnibus/roles/).
 
 These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after
@@ -630,7 +639,7 @@ To configure the Sentinel:
 1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
 
    ```ruby
-   roles ['redis_sentinel_role', 'consul_role']
+   roles(['redis_sentinel_role', 'consul_role'])
 
    # Must be the same in every sentinel node
    redis['master_name'] = 'gitlab-redis'
@@ -694,7 +703,6 @@ To configure the Sentinel:
    # sentinel['failover_timeout'] = 60000
 
    ## Enable service discovery for Prometheus
-   consul['enable'] = true
    consul['monitoring_service_discovery'] =  true
 
    ## The IPs of the Consul server nodes
@@ -712,6 +720,9 @@ To configure the Sentinel:
    gitlab_rails['auto_migrate'] = false
    ```
 
+1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
+   the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
+
 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
 1. Go through the steps again for all the other Consul/Sentinel nodes, and
    make sure you set up the correct IPs.
@@ -805,6 +816,15 @@ in the second step, do not supply the `EXTERNAL_URL` value.
    sudo gitlab-ctl pg-password-md5 pgbouncer
    ```
 
+1. Generate a password hash for the PostgreSQL replication username/password pair. This assumes you will use the default
+   username of `gitlab_replicator` (recommended). The command will request a password
+   and a confirmation. Use the value that is output by this command in the next step
+   as the value of `<postgresql_replication_password_hash>`:
+
+   ```shell
+   sudo gitlab-ctl pg-password-md5 gitlab_replicator
+   ```
+
 1. Generate a password hash for the Consul database username/password pair. This assumes you will use the default
    username of `gitlab-consul` (recommended). The command will request a password
    and confirmation. Use the value that is output by this command in the next
@@ -817,19 +837,21 @@ in the second step, do not supply the `EXTERNAL_URL` value.
 1. On every database node, edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section:
 
    ```ruby
-   # Disable all components except PostgreSQL, Patroni, and Consul
-   roles ['postgres_role']
+   # Disable all components except Patroni and Consul
+   roles(['patroni_role'])
 
    # PostgreSQL configuration
    postgresql['listen_address'] = '0.0.0.0'
 
-   # Enable Patroni
-   patroni['enable'] = true
-   # Set `max_wal_senders` to one more than the number of database nodes in the cluster.
+   # Sets `max_replication_slots` to double the number of database nodes.
+   # Patroni uses one extra slot per node when initiating the replication.
+   patroni['postgresql']['max_replication_slots'] = 8
+   
+   # Set `max_wal_senders` to one more than the number of replication slots in the cluster.
    # This is used to prevent replication from using up all of the
    # available database connections.
-   patroni['postgresql']['max_wal_senders'] = 4
-   patroni['postgresql']['max_replication_slots'] = 4
+   patroni['postgresql']['max_wal_senders'] = 9
+
    # Incoming recommended value for max connections is 500. See https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5691.
    patroni['postgresql']['max_connections'] = 500
 
@@ -837,7 +859,6 @@ in the second step, do not supply the `EXTERNAL_URL` value.
    gitlab_rails['auto_migrate'] = false
 
    # Configure the Consul agent
-   consul['enable'] = true
    consul['services'] = %w(postgresql)
    ## Enable service discovery for Prometheus
    consul['monitoring_service_discovery'] =  true
@@ -847,6 +868,8 @@ in the second step, do not supply the `EXTERNAL_URL` value.
    #
    # Replace PGBOUNCER_PASSWORD_HASH with a generated md5 value
    postgresql['pgbouncer_user_password'] = '<pgbouncer_password_hash>'
+   # Replace POSTGRESQL_REPLICATION_PASSWORD_HASH with a generated md5 value
+   postgresql['sql_replication_password'] = '<postgresql_replication_password_hash>'
    # Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value
    postgresql['sql_user_password'] = '<postgresql_password_hash>'
 
@@ -870,9 +893,8 @@ PostgreSQL, with Patroni managing its failover, will default to use `pg_rewind`
 Like most failover handling methods, this has a small chance of leading to data loss.
 Learn more about the various [Patroni replication methods](../postgresql/replication_and_failover.md#selecting-the-appropriate-patroni-replication-method).
 
-1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace
-   the file of the same name on this server. If that file is not on this server,
-   add the file from your Consul server to this server.
+1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
+   the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
 
 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
 
@@ -887,21 +909,7 @@ are supported and can be added if needed.
 
 #### PostgreSQL post-configuration
 
-SSH in to the **primary node**:
-
-1. Open a database prompt:
-
-   ```shell
-   gitlab-psql -d gitlabhq_production
-   ```
-
-1. Enable the `pg_trgm` extension:
-
-   ```shell
-   CREATE EXTENSION pg_trgm;
-   ```
-
-1. Exit the database prompt by typing `\q` and Enter.
+SSH in to any of the Patroni nodes on the **primary site**:
 
 1. Check the status of the leader and cluster:
 
@@ -943,7 +951,7 @@ The following IPs will be used as an example:
 
    ```ruby
    # Disable all components except Pgbouncer and Consul agent
-   roles ['pgbouncer_role']
+   roles(['pgbouncer_role'])
 
    # Configure PgBouncer
    pgbouncer['admin_users'] = %w(pgbouncer gitlab-consul)
@@ -960,7 +968,6 @@ The following IPs will be used as an example:
 
    # Configure Consul agent
    consul['watchers'] = %w(postgresql)
-   consul['enable'] = true
    consul['configuration'] = {
    retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13)
    }
@@ -973,6 +980,9 @@ The following IPs will be used as an example:
    pgbouncer_exporter['listen_address'] = '0.0.0.0:9188'
    ```
 
+1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
+   the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
+
 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
 
 1. Create a `.pgpass` file so Consul is able to
@@ -1088,9 +1098,7 @@ in the second step, do not supply the `EXTERNAL_URL` value.
 
    ```ruby
    # Disable all components except PostgreSQL and Consul
-   roles ['postgres_role']
-   repmgr['enable'] = false
-   patroni['enable'] = false
+   roles(['postgres_role', 'consul_role'])
 
    # PostgreSQL configuration
    postgresql['listen_address'] = '0.0.0.0'
@@ -1100,7 +1108,6 @@ in the second step, do not supply the `EXTERNAL_URL` value.
    gitlab_rails['auto_migrate'] = false
 
    # Configure the Consul agent
-   consul['enable'] = true
    ## Enable service discovery for Prometheus
    consul['monitoring_service_discovery'] =  true
 
@@ -1126,7 +1133,11 @@ in the second step, do not supply the `EXTERNAL_URL` value.
    # END user configuration
    ```
 
+1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
+   the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
+
 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
+
 1. Follow the [post configuration](#praefect-postgresql-post-configuration).
 
 <div align="right">
@@ -1224,19 +1235,18 @@ To configure the Praefect nodes, on each one:
 1. Edit the `/etc/gitlab/gitlab.rb` file to configure Praefect:
 
    ```ruby
-   # Avoid running unnecessary services on the Gitaly server
+   # Avoid running unnecessary services on the Praefect server
+   gitaly['enable'] = false
    postgresql['enable'] = false
    redis['enable'] = false
-   nginx['enable'] = false
    puma['enable'] = false
-   unicorn['enable'] = false
    sidekiq['enable'] = false
    gitlab_workhorse['enable'] = false
-   grafana['enable'] = false
-
-   # If you run a separate monitoring node you can disable these services
-   alertmanager['enable'] = false
    prometheus['enable'] = false
+   alertmanager['enable'] = false
+   grafana['enable'] = false
+   gitlab_exporter['enable'] = false
+   nginx['enable'] = false
 
    # Praefect Configuration
    praefect['enable'] = true
@@ -1274,19 +1284,20 @@ To configure the Praefect nodes, on each one:
    # server ('praefect') and in git_data_dirs on Gitaly nodes ('gitaly-1')
    praefect['virtual_storages'] = {
      'default' => {
-       'gitaly-1' => {
-         'address' => 'tcp://10.6.0.91:8075',
-         'token'   => '<praefect_internal_token>',
-         'primary' => true
-       },
-       'gitaly-2' => {
-         'address' => 'tcp://10.6.0.92:8075',
-         'token'   => '<praefect_internal_token>'
-       },
-       'gitaly-3' => {
-         'address' => 'tcp://10.6.0.93:8075',
-         'token'   => '<praefect_internal_token>'
-       },
+       'nodes' => {
+         'gitaly-1' => {
+           'address' => 'tcp://10.6.0.91:8075',
+           'token'   => '<praefect_internal_token>'
+         },
+         'gitaly-2' => {
+           'address' => 'tcp://10.6.0.92:8075',
+           'token'   => '<praefect_internal_token>'
+         },
+         'gitaly-3' => {
+           'address' => 'tcp://10.6.0.93:8075',
+           'token'   => '<praefect_internal_token>'
+         },
+       }
      }
    }
 
@@ -1303,11 +1314,25 @@ To configure the Praefect nodes, on each one:
    # END user configuration
    ```
 
-   1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and
-      then replace the file of the same name on this server. If that file isn't on
-      this server, add the file from your Consul server to this server.
+1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
+the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
+
+1. Praefect requires to run some database migrations, much like the main GitLab application. For this
+   you should select **one Praefect node only to run the migrations**, AKA the _Deploy Node_. This node
+   must be configured first before the others as follows:
+   
+   1. In the `/etc/gitlab/gitlab.rb` file, change the `praefect['auto_migrate']` setting value from `false` to `true`
+
+   1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run:
+
+   ```shell
+   sudo touch /etc/gitlab/skip-auto-reconfigure
+   ```
+   
+   1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect and
+      to run the Praefect database migrations.
 
-   1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
+1. On all other Praefect nodes, [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
 
 ### Configure Gitaly
 
@@ -1351,21 +1376,17 @@ On each node:
    storage paths, enable the network listener, and to configure the token:
 
    ```ruby
-   # /etc/gitlab/gitlab.rb
-
    # Avoid running unnecessary services on the Gitaly server
    postgresql['enable'] = false
    redis['enable'] = false
-   nginx['enable'] = false
    puma['enable'] = false
-   unicorn['enable'] = false
    sidekiq['enable'] = false
    gitlab_workhorse['enable'] = false
-   grafana['enable'] = false
-
-   # If you run a separate monitoring node you can disable these services
-   alertmanager['enable'] = false
    prometheus['enable'] = false
+   alertmanager['enable'] = false
+   grafana['enable'] = false
+   gitlab_exporter['enable'] = false
+   nginx['enable'] = false
 
    # Prevent database migrations from running on upgrade automatically
    gitlab_rails['auto_migrate'] = false
@@ -1373,9 +1394,11 @@ On each node:
    # Configure the gitlab-shell API callback URL. Without this, `git push` will
    # fail. This can be your 'front door' GitLab URL or an internal load
    # balancer.
-   # Don't forget to copy `/etc/gitlab/gitlab-secrets.json` from web server to Gitaly server.
    gitlab_rails['internal_api_url'] = 'https://gitlab.example.com'
 
+   # Gitaly
+   gitaly['enable'] = true
+
    # Make Gitaly accept connections on all network interfaces. You must use
    # firewalls to restrict access to this address/port.
    # Comment out following line if you only want to support TLS connections
@@ -1417,9 +1440,8 @@ On each node:
      })
      ```
 
-1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and
-   then replace the file of the same name on this server. If that file isn't on
-   this server, add the file from your Consul server to this server.
+1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
+   the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
 
 1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
 
@@ -1526,28 +1548,19 @@ To configure the Sidekiq nodes, one each one:
 1. Open `/etc/gitlab/gitlab.rb` with your editor:
 
    ```ruby
-   ########################################
-   #####        Services Disabled       ###
-   ########################################
-
-   nginx['enable'] = false
-   grafana['enable'] = false
-   prometheus['enable'] = false
-   alertmanager['enable'] = false
+   # Avoid running unnecessary services on the Sidekiq server
    gitaly['enable'] = false
-   gitlab_workhorse['enable'] = false
-   nginx['enable'] = false
-   puma['enable'] = false
-   postgres_exporter['enable'] = false
    postgresql['enable'] = false
    redis['enable'] = false
-   redis_exporter['enable'] = false
+   puma['enable'] = false
+   gitlab_workhorse['enable'] = false
+   prometheus['enable'] = false
+   alertmanager['enable'] = false
+   grafana['enable'] = false
    gitlab_exporter['enable'] = false
+   nginx['enable'] = false
 
-   ########################################
-   ####              Redis              ###
-   ########################################
-
+   # Redis
    ## Must be the same in every sentinel node
    redis['master_name'] = 'gitlab-redis'
 
@@ -1561,13 +1574,10 @@ To configure the Sidekiq nodes, one each one:
       {'host' => '10.6.0.13', 'port' => 26379},
    ]
 
-   #######################################
-   ###              Gitaly             ###
-   #######################################
-
-   # git_data_dirs get configured for the Praefect virtual storage
-   # Address is Internal Load Balancer for Praefect
-   # Token is praefect_external_token
+   # Gitaly Cluster
+   ## git_data_dirs get configured for the Praefect virtual storage
+   ## Address is Internal Load Balancer for Praefect
+   ## Token is praefect_external_token
    git_data_dirs({
      "default" => {
        "gitaly_address" => "tcp://10.6.0.40:2305", # internal load balancer IP
@@ -1575,31 +1585,26 @@ To configure the Sidekiq nodes, one each one:
      }
    })
 
-   #######################################
-   ###            Postgres             ###
-   #######################################
+   # PostgreSQL
    gitlab_rails['db_host'] = '10.6.0.40' # internal load balancer IP
    gitlab_rails['db_port'] = 6432
    gitlab_rails['db_password'] = '<postgresql_user_password>'
    gitlab_rails['db_adapter'] = 'postgresql'
    gitlab_rails['db_encoding'] = 'unicode'
-   # Prevent database migrations from running on upgrade automatically
+   ## Prevent database migrations from running on upgrade automatically
    gitlab_rails['auto_migrate'] = false
 
-   #######################################
-   ###      Sidekiq configuration      ###
-   #######################################
+   # Sidekiq
+   sidekiq['enable'] = true
    sidekiq['listen_address'] = "0.0.0.0"
 
-   # Set number of Sidekiq queue processes to the same number as available CPUs
+   ## Set number of Sidekiq queue processes to the same number as available CPUs
    sidekiq['queue_groups'] = ['*'] * 4
 
-   # Set number of Sidekiq threads per queue process to the recommend number of 10
+   ## Set number of Sidekiq threads per queue process to the recommend number of 10
    sidekiq['max_concurrency'] = 10
 
-   #######################################
-   ###     Monitoring configuration    ###
-   #######################################
+   # Monitoring
    consul['enable'] = true
    consul['monitoring_service_discovery'] =  true
 
@@ -1607,19 +1612,16 @@ To configure the Sidekiq nodes, one each one:
       retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13)
    }
 
-   # Set the network addresses that the exporters will listen on
+   ## Set the network addresses that the exporters will listen on
    node_exporter['listen_address'] = '0.0.0.0:9100'
 
-   # Rails Status for prometheus
+   ## Add the monitoring node's IP address to the monitoring whitelist
    gitlab_rails['monitoring_whitelist'] = ['10.6.0.81/32', '127.0.0.0/8']
    gitlab_rails['prometheus_address'] = '10.6.0.81:9090'
 
-   #############################
-   ###     Object storage    ###
-   #############################
-
-   # This is an example for configuring Object Storage on GCP
-   # Replace this config with your chosen Object Storage provider as desired
+   # Object Storage
+   ## This is an example for configuring Object Storage on GCP
+   ## Replace this config with your chosen Object Storage provider as desired
    gitlab_rails['object_store']['connection'] = {
      'provider' => 'Google',
      'google_project' => '<gcp-project-name>',
@@ -1632,9 +1634,29 @@ To configure the Sidekiq nodes, one each one:
    gitlab_rails['object_store']['objects']['packages']['bucket'] = "<gcp-packages-bucket-name>"
    gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = "<gcp-dependency-proxy-bucket-name>"
    gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = "<gcp-terraform-state-bucket-name>"
+
+   gitlab_rails['backup_upload_connection'] = {
+     'provider' => 'Google',
+     'google_project' => '<gcp-project-name>',
+     'google_json_key_location' => '<path-to-gcp-service-account-key>'
+   }
+   gitlab_rails['backup_upload_remote_directory'] = "<gcp-backups-state-bucket-name>"
    ```
 
-1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
+1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
+   the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
+
+1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run:
+
+   ```shell
+   sudo touch /etc/gitlab/skip-auto-reconfigure
+   ```
+
+   Only a single designated node should handle migrations as detailed in the
+   [GitLab Rails post-configuration](#gitlab-rails-post-configuration) section.
+
+1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
+
 1. Verify the GitLab services are running:
 
    ```shell
@@ -1716,7 +1738,7 @@ On each node perform the following:
    })
 
    ## Disable components that will not be on the GitLab application server
-   roles ['application_role']
+   roles(['application_role'])
    gitaly['enable'] = false
    nginx['enable'] = true
    sidekiq['enable'] = false
@@ -1785,6 +1807,13 @@ On each node perform the following:
    gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = "<gcp-dependency-proxy-bucket-name>"
    gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = "<gcp-terraform-state-bucket-name>"
 
+   gitlab_rails['backup_upload_connection'] = {
+     'provider' => 'Google',
+     'google_project' => '<gcp-project-name>',
+     'google_json_key_location' => '<path-to-gcp-service-account-key>'
+   }
+   gitlab_rails['backup_upload_remote_directory'] = "<gcp-backups-state-bucket-name>"
+
    ## Uncomment and edit the following options if you have set up NFS
    ##
    ## Prevent GitLab from starting if NFS data mounts are not available
@@ -1819,7 +1848,20 @@ On each node perform the following:
       sudo cp cert.pem /etc/gitlab/trusted-certs/
       ```
 
-1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
+1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
+   the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
+
+1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run:
+
+   ```shell
+   sudo touch /etc/gitlab/skip-auto-reconfigure
+   ```
+
+   Only a single designated node should handle migrations as detailed in the
+   [GitLab Rails post-configuration](#gitlab-rails-post-configuration) section.
+
+1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
+
 1. Run `sudo gitlab-rake gitlab:gitaly:check` to confirm the node can connect to Gitaly.
 1. Tail the logs to see the requests:
 
@@ -1827,11 +1869,6 @@ On each node perform the following:
    sudo gitlab-ctl tail gitaly
    ```
 
-1. Save the `/etc/gitlab/gitlab-secrets.json` file from one of the two
-   application nodes and install it on the other application node, the
-   [Gitaly node](#configure-gitaly) and the [Sidekiq node](#configure-sidekiq) and
-   [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
-
 1. Verify the GitLab services are running:
 
    ```shell
@@ -1890,45 +1927,26 @@ running [Prometheus](../monitoring/prometheus/index.md) and
 1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
 
    ```ruby
-   external_url 'http://gitlab.example.com'
+   roles(['monitoring_role', 'consul_role'])
 
-   # Disable all other services
-   alertmanager['enable'] = false
-   gitaly['enable'] = false
-   gitlab_exporter['enable'] = false
-   gitlab_workhorse['enable'] = false
-   nginx['enable'] = true
-   postgres_exporter['enable'] = false
-   postgresql['enable'] = false
-   redis['enable'] = false
-   redis_exporter['enable'] = false
-   sidekiq['enable'] = false
-   puma['enable'] = false
-   unicorn['enable'] = false
-   node_exporter['enable'] = false
-   gitlab_exporter['enable'] = false
+   external_url 'http://gitlab.example.com'
 
-   # Enable Prometheus
-   prometheus['enable'] = true
+   # Prometheus
    prometheus['listen_address'] = '0.0.0.0:9090'
    prometheus['monitor_kubernetes'] = false
 
-   # Enable Login form
-   grafana['disable_login_form'] = false
-
-   # Enable Grafana
-   grafana['enable'] = true
+   # Grafana
    grafana['admin_password'] = '<grafana_password>'
+   grafana['disable_login_form'] = false
 
    # Enable service discovery for Prometheus
-   consul['enable'] = true
    consul['monitoring_service_discovery'] =  true
    consul['configuration'] = {
       retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13)
    }
 
-   # Prevent database migrations from running on upgrade automatically
-   gitlab_rails['auto_migrate'] = false
+   # Nginx - For Grafana access
+   nginx['enable'] = true
    ```
 
 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
diff --git a/doc/administration/reference_architectures/index.md b/doc/administration/reference_architectures/index.md
index 6698737af6a..49024365e30 100644
--- a/doc/administration/reference_architectures/index.md
+++ b/doc/administration/reference_architectures/index.md
@@ -138,12 +138,12 @@ As long as at least one of each component is online and capable of handling the
 ### Automated database failover **(PREMIUM SELF)**
 
 > - Level of complexity: **High**
-> - Required domain knowledge: PgBouncer, Repmgr or Patroni, shared storage, distributed systems
+> - Required domain knowledge: PgBouncer, Patroni, shared storage, distributed systems
 
 By adding automatic failover for database systems, you can enable higher uptime
 with additional database nodes. This extends the default database with
 cluster management and failover policies.
-[PgBouncer in conjunction with Repmgr or Patroni](../postgresql/replication_and_failover.md)
+[PgBouncer in conjunction with Patroni](../postgresql/replication_and_failover.md)
 is recommended.
 
 ### Instance level replication with GitLab Geo **(PREMIUM SELF)**
diff --git a/doc/administration/reference_architectures/troubleshooting.md b/doc/administration/reference_architectures/troubleshooting.md
index d5f57965a80..4b07cff7de2 100644
--- a/doc/administration/reference_architectures/troubleshooting.md
+++ b/doc/administration/reference_architectures/troubleshooting.md
@@ -206,211 +206,8 @@ To make sure your configuration is correct:
 
 ## Troubleshooting Gitaly
 
-If you have any problems when using standalone Gitaly nodes, first
-[check all the versions are up to date](../gitaly/index.md#check-versions-when-using-standalone-gitaly-servers).
-
-### `gitaly-debug`
-
-The `gitaly-debug` command provides "production debugging" tools for Gitaly and Git
-performance. It is intended to help production engineers and support
-engineers investigate Gitaly performance problems.
-
-If you're using GitLab 11.6 or newer, this tool should be installed on
-your GitLab / Gitaly server already at `/opt/gitlab/embedded/bin/gitaly-debug`.
-If you're investigating an older GitLab version you can compile this
-tool offline and copy the executable to your server:
-
-```shell
-git clone https://gitlab.com/gitlab-org/gitaly.git
-cd cmd/gitaly-debug
-GOOS=linux GOARCH=amd64 go build -o gitaly-debug
-```
-
-To see the help page of `gitaly-debug` for a list of supported sub-commands, run:
-
-```shell
-gitaly-debug -h
-```
-
-### Commits, pushes, and clones return a 401
-
-```plaintext
-remote: GitLab: 401 Unauthorized
-```
-
-You will need to sync your `gitlab-secrets.json` file with your GitLab
-app nodes.
-
-### Client side gRPC logs
-
-Gitaly uses the [gRPC](https://grpc.io/) RPC framework. The Ruby gRPC
-client has its own log file which may contain useful information when
-you are seeing Gitaly errors. You can control the log level of the
-gRPC client with the `GRPC_LOG_LEVEL` environment variable. The
-default level is `WARN`.
-
-You can run a gRPC trace with:
-
-```shell
-sudo GRPC_TRACE=all GRPC_VERBOSITY=DEBUG gitlab-rake gitlab:gitaly:check
-```
-
-### Observing `gitaly-ruby` traffic
-
-[`gitaly-ruby`](../gitaly/configure_gitaly.md#gitaly-ruby) is an internal implementation detail of Gitaly,
-so, there's not that much visibility into what goes on inside
-`gitaly-ruby` processes.
-
-If you have Prometheus set up to scrape your Gitaly process, you can see
-request rates and error codes for individual RPCs in `gitaly-ruby` by
-querying `grpc_client_handled_total`. Strictly speaking, this metric does
-not differentiate between `gitaly-ruby` and other RPCs, but in practice
-(as of GitLab 11.9), all gRPC calls made by Gitaly itself are internal
-calls from the main Gitaly process to one of its `gitaly-ruby` sidecars.
-
-Assuming your `grpc_client_handled_total` counter only observes Gitaly,
-the following query shows you RPCs are (most likely) internally
-implemented as calls to `gitaly-ruby`:
-
-```prometheus
-sum(rate(grpc_client_handled_total[5m])) by (grpc_method) > 0
-```
-
-### Repository changes fail with a `401 Unauthorized` error
-
-If you're running Gitaly on its own server and notice that users can
-successfully clone and fetch repositories (via both SSH and HTTPS), but can't
-push to them or make changes to the repository in the web UI without getting a
-`401 Unauthorized` message, then it's possible Gitaly is failing to authenticate
-with the other nodes due to having the wrong secrets file.
-
-Confirm the following are all true:
-
-- When any user performs a `git push` to any repository on this Gitaly node, it
-  fails with the following error (note the `401 Unauthorized`):
-
-  ```shell
-  remote: GitLab: 401 Unauthorized
-  To <REMOTE_URL>
-  ! [remote rejected] branch-name -> branch-name (pre-receive hook declined)
-  error: failed to push some refs to '<REMOTE_URL>'
-  ```
-
-- When any user adds or modifies a file from the repository using the GitLab
-  UI, it immediately fails with a red `401 Unauthorized` banner.
-- Creating a new project and [initializing it with a README](../../user/project/working_with_projects.md#blank-projects)
-  successfully creates the project but doesn't create the README.
-- When [tailing the logs](https://docs.gitlab.com/omnibus/settings/logs.html#tail-logs-in-a-console-on-the-server) on an app node and reproducing the error, you get `401` errors
-  when reaching the [`/api/v4/internal/allowed`](../../development/internal_api.md) endpoint:
-
-  ```shell
-  # api_json.log
-  {
-    "time": "2019-07-18T00:30:14.967Z",
-    "severity": "INFO",
-    "duration": 0.57,
-    "db": 0,
-    "view": 0.57,
-    "status": 401,
-    "method": "POST",
-    "path": "\/api\/v4\/internal\/allowed",
-    "params": [
-      {
-        "key": "action",
-        "value": "git-receive-pack"
-      },
-      {
-        "key": "changes",
-        "value": "REDACTED"
-      },
-      {
-        "key": "gl_repository",
-        "value": "REDACTED"
-      },
-      {
-        "key": "project",
-        "value": "\/path\/to\/project.git"
-      },
-      {
-        "key": "protocol",
-        "value": "web"
-      },
-      {
-        "key": "env",
-        "value": "{\"GIT_ALTERNATE_OBJECT_DIRECTORIES\":[],\"GIT_ALTERNATE_OBJECT_DIRECTORIES_RELATIVE\":[],\"GIT_OBJECT_DIRECTORY\":null,\"GIT_OBJECT_DIRECTORY_RELATIVE\":null}"
-      },
-      {
-        "key": "user_id",
-        "value": "2"
-      },
-      {
-        "key": "secret_token",
-        "value": "[FILTERED]"
-      }
-    ],
-    "host": "gitlab.example.com",
-    "ip": "REDACTED",
-    "ua": "Ruby",
-    "route": "\/api\/:version\/internal\/allowed",
-    "queue_duration": 4.24,
-    "gitaly_calls": 0,
-    "gitaly_duration": 0,
-    "correlation_id": "XPUZqTukaP3"
-  }
-
-  # nginx_access.log
-  [IP] - - [18/Jul/2019:00:30:14 +0000] "POST /api/v4/internal/allowed HTTP/1.1" 401 30 "" "Ruby"
-  ```
-
-To fix this problem, confirm that your `gitlab-secrets.json` file
-on the Gitaly node matches the one on all other nodes. If it doesn't match,
-update the secrets file on the Gitaly node to match the others, then
-[reconfigure the node](../restart_gitlab.md#omnibus-gitlab-reconfigure).
-
-### Command line tools cannot connect to Gitaly
-
-If you are having trouble connecting to a Gitaly node with command line (CLI) tools, and certain actions result in a `14: Connect Failed` error message, it means that gRPC cannot reach your Gitaly node.
-
-Verify that you can reach Gitaly via TCP:
-
-```shell
-sudo gitlab-rake gitlab:tcp_check[GITALY_SERVER_IP,GITALY_LISTEN_PORT]
-```
-
-If the TCP connection fails, check your network settings and your firewall rules. If the TCP connection succeeds, your networking and firewall rules are correct.
-
-If you use proxy servers in your command line environment, such as Bash, these can interfere with your gRPC traffic.
-
-If you use Bash or a compatible command line environment, run the following commands to determine whether you have proxy servers configured:
-
-```shell
-echo $http_proxy
-echo $https_proxy
-```
-
-If either of these variables have a value, your Gitaly CLI connections may be getting routed through a proxy which cannot connect to Gitaly.
-
-To remove the proxy setting, run the following commands (depending on which variables had values):
-
-```shell
-unset http_proxy
-unset https_proxy
-```
-
-### Gitaly not listening on new address after reconfiguring
-
-When updating the `gitaly['listen_addr']` or `gitaly['prometheus_listen_addr']` values, Gitaly may continue to listen on the old address after a `sudo gitlab-ctl reconfigure`.
-
-When this occurs, performing a `sudo gitlab-ctl restart` will resolve the issue. This will no longer be necessary after [this issue](https://gitlab.com/gitlab-org/gitaly/-/issues/2521) is resolved.
-
-### Permission denied errors appearing in Gitaly logs when accessing repositories from a standalone Gitaly node
-
-If this error occurs even though file permissions are correct, it's likely that
-the Gitaly node is experiencing
-[clock drift](https://en.wikipedia.org/wiki/Clock_drift).
-
-Please ensure that the GitLab and Gitaly nodes are synchronized and use an NTP time
-server to keep them synchronized if possible.
+For troubleshooting information, see Gitaly and Gitaly Cluster
+[troubleshooting information](../gitaly/index.md).
 
 ## Troubleshooting the GitLab Rails application