Add latest changes from gitlab-org/gitlab@13-0-stable-ee

4 files changed, 297 insertions, 190 deletions

diff --git a/doc/administration/gitaly/img/praefect_architecture_v12_10.png b/doc/administration/gitaly/img/praefect_architecture_v12_10.png
index 7b8f1138b23..024a12b0a5d 100644
--- a/doc/administration/gitaly/img/praefect_architecture_v12_10.png
+++ b/doc/administration/gitaly/img/praefect_architecture_v12_10.png
diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md
index 20b9f2104de..14b0a6bd450 100644
--- a/doc/administration/gitaly/index.md
+++ b/doc/administration/gitaly/index.md
@@ -10,6 +10,11 @@ On this page, *Gitaly server* refers to a standalone node that only runs Gitaly
 and *Gitaly client* is a GitLab Rails app node that runs all other processes
 except Gitaly.
 
+CAUTION: **Caution:**
+From GitLab 13.0, using NFS for Git repositories is deprecated. In GitLab 14.0,
+support for NFS for Git repositories is scheduled to be removed. Upgrade to
+[Gitaly Cluster](praefect.md) as soon as possible.
+
 ## Architecture
 
 Here's a high-level architecture overview of how Gitaly is used.
@@ -69,7 +74,7 @@ The following list depicts what the network architecture of Gitaly is:
 - A GitLab server can use one or more Gitaly servers.
 - Gitaly addresses must be specified in such a way that they resolve
   correctly for ALL Gitaly clients.
-- Gitaly clients are: Unicorn, Sidekiq, GitLab Workhorse,
+- Gitaly clients are: Puma/Unicorn, Sidekiq, GitLab Workhorse,
   GitLab Shell, Elasticsearch Indexer, and Gitaly itself.
 - A Gitaly server must be able to make RPC calls **to itself** via its own
   `(Gitaly address, Gitaly token)` pair as specified in `/config/gitlab.yml`.
@@ -101,13 +106,16 @@ Omnibus GitLab or install it from source:
   **_do not_** provide the `EXTERNAL_URL=` value.
 - From source: [Install Gitaly](../../install/installation.md#install-gitaly).
 
-### 2. Client side token configuration
+### 2. Authentication
 
-Configure a token on the instance that runs the GitLab Rails application.
+Gitaly and GitLab use two shared secrets for authentication, one to authenticate gRPC requests
+to Gitaly, and a second for authentication callbacks from GitLab-Shell to the GitLab internal API.
 
 **For Omnibus GitLab**
 
-1. On the client node(s), edit `/etc/gitlab/gitlab.rb`:
+To configure the Gitaly token:
+
+1. On the client server, edit `/etc/gitlab/gitlab.rb`:
 
    ```ruby
    gitlab_rails['gitaly_token'] = 'abc123secret'
@@ -115,9 +123,42 @@ Configure a token on the instance that runs the GitLab Rails application.
 
 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
 
+1. On the Gitaly server, edit `/etc/gitlab/gitlab.rb`:
+
+   ```ruby
+   gitaly['auth_token'] = 'abc123secret'
+   ```
+
+1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
+
+There are two ways to configure the GitLab-Shell token:
+
+1. Copy `/etc/gitlab/gitlab-secrets.json` from the client server to same path on the Gitaly server.
+1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
+
+**OR**
+
+1. On the client server, edit `/etc/gitlab/gitlab.rb`:
+
+   ```ruby
+   gitlab_shell['secret_token'] = 'shellsecret'
+   ```
+
+1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
+
+1. On the Gitaly server, edit `/etc/gitlab/gitlab.rb`:
+
+   ```ruby
+   gitlab_shell['secret_token'] = 'shellsecret'
+   ```
+
+1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
+
 **For installations from source**
 
-1. On the client node(s), edit `/home/git/gitlab/config/gitlab.yml`:
+1. Copy `/home/git/gitlab/.gitlab_shell_secret` from the client server to the same path on the Gitaly
+server.
+1. On the client server, edit `/home/git/gitlab/config/gitlab.yml`:
 
    ```yaml
    gitlab:
@@ -138,12 +179,6 @@ documentation on configuring Gitaly
 authentication](https://gitlab.com/gitlab-org/gitaly/blob/master/doc/configuration/README.md#authentication)
 .
 
-Gitaly must trigger some callbacks to GitLab via GitLab Shell. As a result,
-the GitLab Shell secret must be the same between the other GitLab servers and
-the Gitaly server. The easiest way to accomplish this is to copy `/etc/gitlab/gitlab-secrets.json`
-from an existing GitLab server to the Gitaly server. Without this shared secret,
-Git operations in GitLab will result in an API error.
-
 **For Omnibus GitLab**
 
 1. Edit `/etc/gitlab/gitlab.rb`:
@@ -160,17 +195,17 @@ Git operations in GitLab will result in an API error.
    postgresql['enable'] = false
    redis['enable'] = false
    nginx['enable'] = false
-   unicorn['enable'] = false
+   puma['enable'] = false
    sidekiq['enable'] = false
    gitlab_workhorse['enable'] = false
    grafana['enable'] = false
 
-   # If you run a seperate monitoring node you can disable these services
+   # If you run a separate monitoring node you can disable these services
    alertmanager['enable'] = false
    prometheus['enable'] = false
 
-   # If you don't run a seperate monitoring node you can
-   # Enable Prometheus access & disable these extra services
+   # If you don't run a separate monitoring node you can
+   # enable Prometheus access & disable these extra services.
    # This makes Prometheus listen on all interfaces. You must use firewalls to restrict access to this address/port.
    # prometheus['listen_address'] = '0.0.0.0:9090'
    # prometheus['monitor_kubernetes'] = false
@@ -189,10 +224,6 @@ Git operations in GitLab will result in an API error.
    # Don't forget to copy `/etc/gitlab/gitlab-secrets.json` from web server to Gitaly server.
    gitlab_rails['internal_api_url'] = 'https://gitlab.example.com'
 
-   # Authentication token to ensure only authorized servers can communicate with
-   # Gitaly server
-   gitaly['auth_token'] = 'abc123secret'
-
    # Make Gitaly accept connections on all network interfaces. You must use
    # firewalls to restrict access to this address/port.
    # Comment out following line if you only want to support TLS connections
@@ -230,6 +261,8 @@ Git operations in GitLab will result in an API error.
    ```
 
 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
+1. Run `sudo /opt/gitlab/embedded/service/gitlab-shell/bin/check -config /opt/gitlab/embedded/service/gitlab-shell/config.yml`
+to confirm that Gitaly can perform callbacks to the internal API.
 
 **For installations from source**
 
@@ -271,7 +304,15 @@ Git operations in GitLab will result in an API error.
    path = '/srv/gitlab/git-data/repositories'
    ```
 
+1. On each Gitaly server, edit `/home/git/gitlab-shell/config.yml`:
+
+   ```yaml
+   gitlab_url: https://gitlab.example.com
+   ```
+
 1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source).
+1. Run `sudo -u git /home/git/gitlab-shell/bin/check -config /home/git/gitlab-shell/config.yml`
+to confirm that Gitaly can perform callbacks to the internal API.
 
 ### 4. Converting clients to use the Gitaly server
 
@@ -302,11 +343,10 @@ can read and write to `/mnt/gitlab/storage2`.
      'storage1' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' },
      'storage2' => { 'gitaly_address' => 'tcp://gitaly2.internal:8075' },
    })
-
-   gitlab_rails['gitaly_token'] = 'abc123secret'
    ```
 
 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
+1. Run `sudo gitlab-rake gitlab:gitaly:check` to confirm the client can connect to Gitaly.
 1. Tail the logs to see the requests:
 
    ```shell
@@ -330,9 +370,6 @@ can read and write to `/mnt/gitlab/storage2`.
          storage2:
            gitaly_address: tcp://gitaly2.internal:8075
            path: /some/dummy/path
-
-     gitaly:
-       token: 'abc123secret'
    ```
 
    NOTE: **Note:**
@@ -341,6 +378,8 @@ can read and write to `/mnt/gitlab/storage2`.
    [this issue](https://gitlab.com/gitlab-org/gitaly/issues/1282) is resolved.
 
 1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source).
+1. Run `sudo -u git -H bundle exec rake gitlab:gitaly:check RAILS_ENV=production` to
+confirm the client can connect to Gitaly.
 1. Tail the logs to see the requests:
 
    ```shell
@@ -397,9 +436,9 @@ with a Gitaly instance that listens for secure connections you will need to use
 scheme in the `gitaly_address` of the corresponding storage entry in the GitLab configuration.
 
 You will need to bring your own certificates as this isn't provided automatically.
-The certificate to be used needs to be installed on all Gitaly nodes, and the
-certificate (or CA of certificate) on all
-client nodes that communicate with it following the procedure described in
+The certificate, or its certificate authority, must be installed on all Gitaly
+nodes (including the Gitaly node using the certificate) and on all client nodes
+that communicate with it following the procedure described in
 [GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates).
 
 NOTE: **Note**
@@ -430,17 +469,32 @@ To configure Gitaly with TLS:
    })
 
    gitlab_rails['gitaly_token'] = 'abc123secret'
+   gitlab_shell['secret_token'] = 'shellsecret'
    ```
 
 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) on client node(s).
+1. On the client node(s), copy the cert into the `/etc/gitlab/trusted-certs`:
+
+  ```shell
+  sudo cp cert.pem /etc/gitlab/trusted-certs/
+  ```
+
 1. On the Gitaly server, create the `/etc/gitlab/ssl` directory and copy your key and certificate there:
 
    ```shell
    sudo mkdir -p /etc/gitlab/ssl
    sudo chmod 755 /etc/gitlab/ssl
    sudo cp key.pem cert.pem /etc/gitlab/ssl/
+   sudo chmod 644 key.pem cert.pem
    ```
 
+1. Copy the cert to `/etc/gitlab/trusted-certs` so Gitaly will trust the cert when
+calling into itself:
+
+  ```shell
+  sudo cp /etc/gitlab/ssl/cert.pem /etc/gitlab/trusted-certs/
+  ```
+
 1. On the Gitaly server node(s), edit `/etc/gitlab/gitlab.rb` and add:
 
    <!--
@@ -463,6 +517,13 @@ To configure Gitaly with TLS:
 
 **For installations from source**
 
+1. On the client node(s), add the cert to the system trusted certs:
+
+   ```shell
+   sudo cp cert.pem /usr/local/share/ca-certificates/gitaly.crt
+   sudo update-ca-certificates
+   ```
+
 1. On the client node(s), edit `/home/git/gitlab/config/gitlab.yml` as follows:
 
    ```yaml
@@ -488,13 +549,32 @@ To configure Gitaly with TLS:
    data will be stored in this folder. This will no longer be necessary after
    [this issue](https://gitlab.com/gitlab-org/gitaly/issues/1282) is resolved.
 
-1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source) on client node(s).
+1. Save the file and[restart GitLab](../restart_gitlab.md#installations-from-source)
+on client node(s).
+1. Copy `/home/git/gitlab/.gitlab_shell_secret` from the client server to the same
+path on the Gitaly server.
+1. On the Gitaly server, create or edit `/etc/default/gitlab` and add:
+
+   ```shell
+   export SSL_CERT_DIR=/etc/gitlab/ssl
+   ```
+
+1. Save the file.
 1. Create the `/etc/gitlab/ssl` directory and copy your key and certificate there:
 
    ```shell
    sudo mkdir -p /etc/gitlab/ssl
-   sudo chmod 700 /etc/gitlab/ssl
+   sudo chmod 755 /etc/gitlab/ssl
    sudo cp key.pem cert.pem /etc/gitlab/ssl/
+   sudo chmod 644 key.pem cert.pem
+   ```
+
+1. On the Gitaly server, add the cert to the system trusted certs so Gitaly will trust it
+when calling into itself:
+
+   ```shell
+   sudo cp cert.pem /usr/local/share/ca-certificates/gitaly.crt
+   sudo update-ca-certificates
    ```
 
 1. On the Gitaly server node(s), edit `/home/git/gitaly/config.toml` and add:
@@ -781,7 +861,7 @@ two checks. The result of both of these checks is cached.
     see if we can access filesystem underneath the Gitaly server
     directly. If so, use the Rugged patch.
 
-To see if GitLab Rails can access the repo filesystem directly, we use
+To see if GitLab Rails can access the repository filesystem directly, we use
 the following heuristic:
 
 - Gitaly ensures that the filesystem has a metadata file in its root
diff --git a/doc/administration/gitaly/praefect.md b/doc/administration/gitaly/praefect.md
index 19a69dc348c..0ea83f0e090 100644
--- a/doc/administration/gitaly/praefect.md
+++ b/doc/administration/gitaly/praefect.md
@@ -1,35 +1,70 @@
-# Praefect: High Availability
+---
+type: reference
+---
 
-NOTE: **Note:** Praefect is an experimental service, and data loss is likely.
+# Gitaly Cluster
 
-Praefect is an optional reverse-proxy for [Gitaly](../index.md) to manage a
-cluster of Gitaly nodes for high availability. Initially, high availability
-be implemented through asynchronous replication. If a Gitaly node becomes
-unavailable, it will be possible to fail over to a warm Gitaly replica.
+[Gitaly](index.md), the service that provides storage for Git repositories, can
+be run in a clustered configuration to increase fault tolerance. In this
+configuration, every Git repository is stored on every Gitaly node in the
+cluster. Multiple clusters (or shards), can be configured.
 
-The first minimal version will support:
+NOTE: **Note:**
+Gitaly Clusters can be created using [GitLab Core](https://about.gitlab.com/pricing/#self-managed)
+and higher tiers. However, technical support is limited to GitLab Premium and Ultimate customers
+only. Not available in GitLab.com.
+
+Praefect is a router and transaction manager for Gitaly, and a required
+component for running a Gitaly Cluster.
+
+![Architecture diagram](img/praefect_architecture_v12_10.png)
+
+Using a Gitaly Cluster increase fault tolerance by:
+
+- Replicating write operations to warm standby Gitaly nodes.
+- Detecting Gitaly node failures.
+- Automatically routing Git requests to an available Gitaly node.
+
+The availability objectives for Gitaly clusters are:
+
+- **Recovery Point Objective (RPO):** Less than 1 minute.
+
+  Writes are replicated asynchronously. Any writes that have not been replicated
+  to the newly promoted primary are lost.
+
+  [Strong Consistency](https://gitlab.com/groups/gitlab-org/-/epics/1189) is
+  planned to improve this to "no loss".
+
+- **Recovery Time Objective (RTO):** Less than 10 seconds.
+
+  Outages are detected by a health checks run by each Praefect node every
+  second. Failover requires ten consecutive failed health checks on each
+  Praefect node.
+
+  [Faster outage detection](https://gitlab.com/gitlab-org/gitaly/-/issues/2608)
+  is planned to improve this to less than 1 second.
+
+The current version supports:
 
 - Eventual consistency of the secondary replicas.
-- Automatic fail over from the primary to the secondary.
+- Automatic failover from the primary to the secondary.
 - Reporting of possible data loss if replication queue is non empty.
+- Marking the newly promoted primary read only if possible data loss is
+  detected.
 
 Follow the [HA Gitaly epic](https://gitlab.com/groups/gitlab-org/-/epics/1489)
-for updates and roadmap.
-
-## Requirements for configuring Gitaly for High Availability
+for improvements including
+[horizontally distributing reads](https://gitlab.com/groups/gitlab-org/-/epics/2013).
 
-NOTE: **Note:** this reference architecture is not highly available because
-Praefect is a single point of failure.
+## Requirements for configuring a Gitaly Cluster
 
-The minimal [alpha](https://about.gitlab.com/handbook/product/#alpha-beta-ga)
-reference architecture additionally requires:
+The minimum recommended configuration for a Gitaly Cluster requires:
 
-- 1 Praefect node
-- 1 PostgreSQL server (PostgreSQL 9.6 or newer)
+- 1 load balancer
+- 1 PostgreSQL server (PostgreSQL 11 or newer)
+- 3 Praefect nodes
 - 3 Gitaly nodes (1 primary, 2 secondary)
 
-![Alpha architecture diagram](img/praefect_architecture_v12_10.png)
-
 See the [design
 document](https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/design_ha.md)
 for implementation details.
@@ -43,6 +78,7 @@ package (highly recommended), follow the steps below:
 1. [Configuring the Praefect database](#postgresql)
 1. [Configuring the Praefect proxy/router](#praefect)
 1. [Configuring each Gitaly node](#gitaly) (once for each Gitaly node)
+1. [Configure the load balancer](#load-balancer)
 1. [Updating the GitLab server configuration](#gitlab)
 1. [Configure Grafana](#grafana)
 
@@ -51,8 +87,8 @@ package (highly recommended), follow the steps below:
 Before beginning, you should already have a working GitLab instance. [Learn how
 to install GitLab](https://about.gitlab.com/install/).
 
-Provision a PostgreSQL server (PostgreSQL 9.6 or newer). Configuration through
-the GitLab Omnibus distribution is not yet supported. Follow this
+Provision a PostgreSQL server (PostgreSQL 11 or newer). Configuration through
+the Omnibus GitLab distribution is not yet supported. Follow this
 [issue](https://gitlab.com/gitlab-org/gitaly/issues/2476) for updates.
 
 Prepare all your new nodes by [installing
@@ -64,6 +100,7 @@ GitLab](https://about.gitlab.com/install/).
 
 You will need the IP/host address for each node.
 
+1. `LOAD_BALANCER_SERVER_ADDRESS`: the IP/hots address of the load balancer
 1. `POSTGRESQL_SERVER_ADDRESS`: the IP/host address of the PostgreSQL server
 1. `PRAEFECT_HOST`: the IP/host address of the Praefect server
 1. `GITALY_HOST`: the IP/host address of each Gitaly server
@@ -98,17 +135,19 @@ We will note in the instructions below where these secrets are required.
 
 ### PostgreSQL
 
-NOTE: **Note:** don't reuse the GitLab application database for the Praefect
-database.
+NOTE: **Note:** do not store the GitLab application database and the Praefect
+database on the same PostgreSQL server if using
+[Geo](../geo/replication/index.md). The replication state is internal to each instance
+of GitLab and should not be replicated.
 
 To complete this section you will need:
 
 - 1 Praefect node
-- 1 PostgreSQL server (PostgreSQL 9.6 or newer)
+- 1 PostgreSQL server (PostgreSQL 11 or newer)
   - An SQL user with permissions to create databases
 
 During this section, we will configure the PostgreSQL server, from the Praefect
-node, using `psql` which is installed by GitLab Omnibus.
+node, using `psql` which is installed by Omnibus GitLab.
 
 1. SSH into the **Praefect** node and login as root:
 
@@ -173,7 +212,7 @@ application server, or a Gitaly node.
    nginx['enable'] = false
    prometheus['enable'] = false
    grafana['enable'] = false
-   unicorn['enable'] = false
+   puma['enable'] = false
    sidekiq['enable'] = false
    gitlab_workhorse['enable'] = false
    gitaly['enable'] = false
@@ -189,16 +228,12 @@ application server, or a Gitaly node.
 1. Configure **Praefect** to listen on network interfaces by editing
    `/etc/gitlab/gitlab.rb`:
 
-   You will need to replace:
-
-   - `PRAEFECT_HOST` with the IP address or hostname of the Praefect node
-
    ```ruby
-   praefect['listen_addr'] = 'PRAEFECT_HOST:2305'
+   praefect['listen_addr'] = '0.0.0.0:2305'
 
    # Enable Prometheus metrics access to Praefect. You must use firewalls
    # to restrict access to this address/port.
-   praefect['prometheus_listen_addr'] = 'PRAEFECT_HOST:9652'
+   praefect['prometheus_listen_addr'] = '0.0.0.0:9652'
    ```
 
 1. Configure a strong `auth_token` for **Praefect** by editing
@@ -245,9 +280,9 @@ application server, or a Gitaly node.
 1. Configure the **Praefect** cluster to connect to each Gitaly node in the
    cluster by editing `/etc/gitlab/gitlab.rb`.
 
-   In the example below we have configured one cluster named `praefect`. This
-   cluster has three Gitaly nodes `gitaly-1`, `gitaly-2`, and `gitaly-3`, which
-   will be replicas of each other.
+   In the example below we have configured one virtual storage (or shard) named
+   `storage-1`. This cluster has three Gitaly nodes `gitaly-1`, `gitaly-2`, and
+   `gitaly-3`, which will be replicas of each other.
 
    Replace `PRAEFECT_INTERNAL_TOKEN` with a strong secret, which will be used by
    Praefect when communicating with Gitaly nodes in the cluster. This token is
@@ -260,13 +295,13 @@ application server, or a Gitaly node.
 
    NOTE: **Note:** The `gitaly-1` node is currently denoted the primary. This
    can be used to manually fail from one node to another. This will be removed
-   in the future to allow for automatic failover.
+   in the [future](https://gitlab.com/gitlab-org/gitaly/-/issues/2634).
 
    ```ruby
    # Name of storage hash must match storage name in git_data_dirs on GitLab
    # server ('praefect') and in git_data_dirs on Gitaly nodes ('gitaly-1')
    praefect['virtual_storages'] = {
-     'praefect' => {
+     'storage-1' => {
        'gitaly-1' => {
          'address' => 'tcp://GITALY_HOST:8075',
          'token'   => 'PRAEFECT_INTERNAL_TOKEN',
@@ -284,75 +319,63 @@ application server, or a Gitaly node.
    }
    ```
 
-1. Save the changes to `/etc/gitlab/gitlab.rb` and [reconfigure Praefect](../restart_gitlab.md#omnibus-gitlab-reconfigure):
+1. Enable the database replication queue:
 
-   ```shell
-   gitlab-ctl reconfigure
+   ```ruby
+   praefect['postgres_queue_enabled'] = true
    ```
 
-1. Verify that Praefect can reach PostgreSQL:
+   In the next release, database replication queue will be enabled by default.
+   See [issue #2615](https://gitlab.com/gitlab-org/gitaly/-/issues/2615).
 
-   ```shell
-   sudo -u git /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml sql-ping
+1. Enable automatic failover by editing `/etc/gitlab/gitlab.rb`:
+
+   ```ruby
+   praefect['failover_enabled'] = true
+   praefect['failover_election_strategy'] = 'sql'
    ```
 
-   If the check fails, make sure you have followed the steps correctly. If you
-   edit `/etc/gitlab/gitlab.rb`, remember to run `sudo gitlab-ctl reconfigure`
-   again before trying the `sql-ping` command.
+   When automatic failover is enabled, Praefect checks the health of internal
+   Gitaly nodes. If the primary has a certain amount of health checks fail, it
+   will promote one of the secondaries to be primary, and demote the primary to
+   be a secondary.
 
-#### Automatic failover
+   NOTE: **Note:** Database leader election will be [enabled by default in the
+   future](https://gitlab.com/gitlab-org/gitaly/-/issues/2682).
 
-When automatic failover is enabled, Praefect will do automatic detection of the health of internal Gitaly nodes. If the
-primary has a certain amount of health checks fail, it will decide to promote one of the secondaries to be primary, and
-demote the primary to be a secondary.
+   Caution, **automatic failover** favors availability over consistency and will
+   cause data loss if changes have not been replicated to the newly elected
+   primary. In the next release, leader election will [prefer to promote up to
+   date replicas](https://gitlab.com/gitlab-org/gitaly/-/issues/2642), and it
+   will be an option to favor consistency by marking [out-of-date repositories
+   read-only](https://gitlab.com/gitlab-org/gitaly/-/issues/2630).
 
-1. To enable automatic failover, edit `/etc/gitlab/gitlab.rb`:
+1. Save the changes to `/etc/gitlab/gitlab.rb` and [reconfigure
+   Praefect](../restart_gitlab.md#omnibus-gitlab-reconfigure):
 
-   ```ruby
-   # failover_enabled turns on automatic failover
-   praefect['failover_enabled'] = true
-   praefect['virtual_storages'] = {
-     'praefect' => {
-       'gitaly-1' => {
-         'address' => 'tcp://GITALY_HOST:8075',
-         'token'   => 'PRAEFECT_INTERNAL_TOKEN',
-         'primary' => true
-       },
-       'gitaly-2' => {
-         'address' => 'tcp://GITALY_HOST:8075',
-         'token'   => 'PRAEFECT_INTERNAL_TOKEN'
-       },
-       'gitaly-3' => {
-         'address' => 'tcp://GITALY_HOST:8075',
-         'token'   => 'PRAEFECT_INTERNAL_TOKEN'
-       }
-     }
-   }
+   ```shell
+   gitlab-ctl reconfigure
    ```
 
-1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
+1. To ensure that Praefect [has updated its Prometheus listen
+   address](https://gitlab.com/gitlab-org/gitaly/-/issues/2734), [restart
+   Gitaly](../restart_gitlab.md#omnibus-gitlab-restart):
 
-Below is the picture when Praefect starts up with the config.toml above:
+   ```shell
+   gitlab-ctl restart praefect
+   ```
 
-```mermaid
-graph TD
-  A[Praefect] -->|Mutator RPC| B(internal_storage_0)
-  B --> |Replication|C[internal_storage_1]
-```
+1. Verify that Praefect can reach PostgreSQL:
 
-Let's say suddenly `internal_storage_0` goes down. Praefect will detect this and
-automatically switch over to `internal_storage_1`, and `internal_storage_0` will serve as a secondary:
+   ```shell
+   sudo -u git /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml sql-ping
+   ```
 
-```mermaid
-graph TD
-  A[Praefect] -->|Mutator RPC| B(internal_storage_1)
-  B --> |Replication|C[internal_storage_0]
-```
+   If the check fails, make sure you have followed the steps correctly. If you
+   edit `/etc/gitlab/gitlab.rb`, remember to run `sudo gitlab-ctl reconfigure`
+   again before trying the `sql-ping` command.
 
-NOTE: **Note:**: Currently this feature is supported for setups that only have 1 Praefect instance. Praefect instances running,
-for example behind a load balancer, `failover_enabled` should be disabled. The reason is The reason is because there
-is no coordination that currently happens across different Praefect instances, so there could be a situation where
-two Praefect instances think two different Gitaly nodes are the primary.
+**The steps above must be completed for each Praefect node!**
 
 ### Gitaly
 
@@ -365,7 +388,7 @@ To complete this section you will need:
   These should be dedicated nodes, do not run other services on these nodes.
 
 Every Gitaly server assigned to the Praefect cluster needs to be configured. The
-configuration is the same as a normal [standalone Gitaly server](../index.md),
+configuration is the same as a normal [standalone Gitaly server](index.md),
 except:
 
 - the storage names are exposed to Praefect, not GitLab
@@ -403,7 +426,7 @@ documentation](index.md#3-gitaly-server-configuration).
    nginx['enable'] = false
    prometheus['enable'] = false
    grafana['enable'] = false
-   unicorn['enable'] = false
+   puma['enable'] = false
    sidekiq['enable'] = false
    gitlab_workhorse['enable'] = false
    prometheus_monitoring['enable'] = false
@@ -419,18 +442,14 @@ documentation](index.md#3-gitaly-server-configuration).
 1. Configure **Gitaly** to listen on network interfaces by editing
    `/etc/gitlab/gitlab.rb`:
 
-   You will need to replace:
-
-   - `GITALY_HOST` with the IP address or hostname of the Gitaly node
-
    ```ruby
    # Make Gitaly accept connections on all network interfaces.
    # Use firewalls to restrict access to this address/port.
-   gitaly['listen_addr'] = 'GITALY_HOST:8075'
+   gitaly['listen_addr'] = '0.0.0.0:8075'
 
    # Enable Prometheus metrics access to Gitaly. You must use firewalls
    # to restrict access to this address/port.
-   gitaly['prometheus_listen_addr'] = 'GITALY_HOST:9236'
+   gitaly['prometheus_listen_addr'] = '0.0.0.0:9236'
    ```
 
 1. Configure a strong `auth_token` for **Gitaly** by editing
@@ -445,7 +464,7 @@ documentation](index.md#3-gitaly-server-configuration).
 1. Configure the GitLab Shell `secret_token`, and `internal_api_url` which are
    needed for `git push` operations.
 
-   If you have already configured [Gitaly on its own server](../index.md)
+   If you have already configured [Gitaly on its own server](index.md)
 
    ```ruby
    gitlab_shell['secret_token'] = 'GITLAB_SHELL_SECRET_TOKEN'
@@ -458,12 +477,12 @@ documentation](index.md#3-gitaly-server-configuration).
 
 1. Configure the storage location for Git data by setting `git_data_dirs` in
    `/etc/gitlab/gitlab.rb`. Each Gitaly node should have a unique storage name
-   (eg `gitaly-1`).
+   (such as `gitaly-1`).
 
    Instead of configuring `git_data_dirs` uniquely for each Gitaly node, it is
    often easier to have include the configuration for all Gitaly nodes on every
    Gitaly node. This is supported because the Praefect `virtual_storages`
-   configuration maps each storage name (eg `gitaly-1`) to a specific node, and
+   configuration maps each storage name (such as `gitaly-1`) to a specific node, and
    requests are routed accordingly. This means every Gitaly node in your fleet
    can share the same configuration.
 
@@ -484,13 +503,16 @@ documentation](index.md#3-gitaly-server-configuration).
    })
    ```
 
-1. Save the changes to `/etc/gitlab/gitlab.rb` and [reconfigure Gitaly](../restart_gitlab.md#omnibus-gitlab-reconfigure):
+1. Save the changes to `/etc/gitlab/gitlab.rb` and [reconfigure
+   Gitaly](../restart_gitlab.md#omnibus-gitlab-reconfigure):
 
    ```shell
    gitlab-ctl reconfigure
    ```
 
-1. To ensure that Gitaly [has updated its Prometheus listen address](https://gitlab.com/gitlab-org/gitaly/-/issues/2521), [restart Gitaly](../restart_gitlab.md#omnibus-gitlab-restart):
+1. To ensure that Gitaly [has updated its Prometheus listen
+   address](https://gitlab.com/gitlab-org/gitaly/-/issues/2734), [restart
+   Gitaly](../restart_gitlab.md#omnibus-gitlab-restart):
 
    ```shell
    gitlab-ctl restart gitaly
@@ -508,37 +530,23 @@ config.
    sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml dial-nodes
    ```
 
-1. Enable automatic failover by editing `/etc/gitlab/gitlab.rb`:
+### Load Balancer
 
-   ```ruby
-   praefect['failover_enabled'] = true
-   ```
-
-   When automatic failover is enabled, Praefect checks the health of internal
-   Gitaly nodes. If the primary has a certain amount of health checks fail, it
-   will promote one of the secondaries to be primary, and demote the primary to
-   be a secondary.
+In a highly available Gitaly configuration, a load balancer is needed to route
+internal traffic from the GitLab application to the Praefect nodes. The
+specifics on which load balancer to use or the exact configuration is beyond the
+scope of the GitLab documentation.
 
-   Manual failover is possible by updating `praefect['virtual_storages']` and
-   nominating a new primary node.
+We hope that if you’re managing HA systems like GitLab, you have a load balancer
+of choice already. Some examples include [HAProxy](https://www.haproxy.org/)
+(open-source), [Google Internal Load Balancer](https://cloud.google.com/load-balancing/docs/internal/),
+[AWS Elastic Load Balancer](https://aws.amazon.com/elasticloadbalancing/), F5
+Big-IP LTM, and Citrix Net Scaler. This documentation will outline what ports
+and protocols you need configure.
 
-1. By default, Praefect will nominate a primary Gitaly node for each
-   shard and store the state of the primary in local memory. This state
-   does not persist across restarts and will cause a split brain
-   if multiple Praefect nodes are used for redundancy.
-
-   To avoid this limitation, enable the SQL election strategy:
-
-    ```ruby
-    praefect['failover_election_strategy'] = 'sql'
-    ```
-
-1. Save the changes to `/etc/gitlab/gitlab.rb` and [reconfigure
-   Praefect](../restart_gitlab.md#omnibus-gitlab-reconfigure):
-
-   ```shell
-   gitlab-ctl reconfigure
-   ```
+| LB Port | Backend Port | Protocol |
+|---------|--------------|----------|
+| 2305    | 2305         | TCP      |
 
 ### GitLab
 
@@ -555,7 +563,7 @@ Particular attention should be shown to:
 - the storage name added to `git_data_dirs` in this section must match the
   storage name under `praefect['virtual_storages']` on the Praefect node. This
   was set in the [Praefect](#praefect) section of this guide. This document uses
-  `praefect` as the Praefect storage name.
+  `storage-1` as the Praefect storage name.
 
 1. SSH into the **GitLab** node and login as root:
 
@@ -563,12 +571,23 @@ Particular attention should be shown to:
    sudo -i
    ```
 
+1. Configure the `external_url` so that files could be served by GitLab
+   by proper endpoint access by editing `/etc/gitlab/gitlab.rb`:
+
+   You will need to replace `GITLAB_SERVER_URL` with the real external facing
+   URL on which current GitLab instance is serving:
+
+   ```ruby
+   external_url 'GITLAB_SERVER_URL'
+   ```
+
 1. Add the Praefect cluster as a storage location by editing
    `/etc/gitlab/gitlab.rb`.
 
    You will need to replace:
 
-   - `PRAEFECT_HOST` with the IP address or hostname of the Praefect node
+   - `LOAD_BALANCER_SERVER_ADDRESS` with the IP address or hostname of the load
+     balancer.
    - `GITLAB_HOST` with the IP address or hostname of the GitLab server
    - `PRAEFECT_EXTERNAL_TOKEN` with the real secret
 
@@ -577,18 +596,18 @@ Particular attention should be shown to:
      "default" => {
        "gitaly_address" => "tcp://GITLAB_HOST:8075"
      },
-     "praefect" => {
-       "gitaly_address" => "tcp://PRAEFECT_HOST:2305",
+     "storage-1" => {
+       "gitaly_address" => "tcp://LOAD_BALANCER_SERVER_ADDRESS:2305",
        "gitaly_token" => 'PRAEFECT_EXTERNAL_TOKEN'
      }
    })
    ```
 
-1. Allow Gitaly to listen on a tcp port by editing
+1. Allow Gitaly to listen on a TCP port by editing
    `/etc/gitlab/gitlab.rb`
 
    ```ruby
-   gitaly['listen_addr'] = 'GITLAB_HOST:8075'
+   gitaly['listen_addr'] = '0.0.0.0:8075'
    ```
 
 1. Configure the `gitlab_shell['secret_token']` so that callbacks from Gitaly
@@ -601,16 +620,6 @@ Particular attention should be shown to:
    gitlab_shell['secret_token'] = 'GITLAB_SHELL_SECRET_TOKEN'
    ```
 
-1. Configure the `external_url` so that files could be served by GitLab
-   by proper endpoint access by editing `/etc/gitlab/gitlab.rb`:
-
-   You will need to replace `GITLAB_SERVER_URL` with the real external facing URL on which
-   current GitLab instance is serving:
-
-   ```ruby
-   external_url 'GITLAB_SERVER_URL'
-   ```
-
 1. Add Prometheus monitoring settings by editing `/etc/gitlab/gitlab.rb`.
 
    You will need to replace:
@@ -624,7 +633,9 @@ Particular attention should be shown to:
        'job_name' => 'praefect',
        'static_configs' => [
          'targets' => [
-           'PRAEFECT_HOST:9652' # praefect
+           'PRAEFECT_HOST:9652', # praefect-1
+           'PRAEFECT_HOST:9652', # praefect-2
+           'PRAEFECT_HOST:9652', # praefect-3
          ]
        ]
      },
@@ -647,6 +658,14 @@ Particular attention should be shown to:
    gitlab-ctl reconfigure
    ```
 
+1. To ensure that Gitaly [has updated its Prometheus listen
+   address](https://gitlab.com/gitlab-org/gitaly/-/issues/2734), [restart
+   Gitaly](../restart_gitlab.md#omnibus-gitlab-restart):
+
+   ```shell
+   gitlab-ctl restart gitaly
+   ```
+
 1. Verify each `gitlab-shell` on each Gitaly instance can reach GitLab. On each Gitaly instance run:
 
    ```shell
@@ -725,6 +744,13 @@ Praefect regularly checks the health of each backend Gitaly node. This
 information can be used to automatically failover to a new primary node if the
 current primary node is found to be unhealthy.
 
+- **PostgreSQL (recommended):** Enabled by setting
+  `praefect['failover_election_strategy'] = sql`. This configuration
+  option will allow multiple Praefect nodes to coordinate via the
+  PostgreSQL database to elect a primary Gitaly node. This configuration
+  will cause Praefect nodes to elect a new primary, monitor its health,
+  and elect a new primary if the current one has not been reachable in
+  10 seconds by a majority of the Praefect nodes.
 - **Manual:** Automatic failover is disabled. The primary node can be
   reconfigured in `/etc/gitlab/gitlab.rb` on the Praefect node. Modify the
   `praefect['virtual_storages']` field by moving the `primary = true` to promote
@@ -735,13 +761,6 @@ current primary node is found to be unhealthy.
   checks fail for the current primary backend Gitaly node, and new primary will
   be elected. **Do not use with multiple Praefect nodes!** Using with multiple
   Praefect nodes is likely to result in a split brain.
-- **PostgreSQL:** Enabled by setting
-  `praefect['failover_election_strategy'] = sql`. This configuration
-  option will allow multiple Praefect nodes to coordinate via the
-  PostgreSQL database to elect a primary Gitaly node. This configuration
-  will cause Praefect nodes to elect a new primary, monitor its health,
-  and elect a new primary if the current one has not been reachable in
-  10 seconds by a majority of the Praefect nodes.
 
 NOTE: **Note:**: Praefect does not yet account for replication lag on
 the secondaries during the election process, so data loss can occur
@@ -753,13 +772,13 @@ strategy in the future.
 
 ## Identifying Impact of a Primary Node Failure
 
-When a primary Gitaly node fails, there is a chance of dataloss. Dataloss can occur if there were outstanding replication jobs the secondaries did not manage to process before the failure. The Praefect `dataloss` subcommand helps identify these cases by counting the number of dead replication jobs for each repository within a given timeframe.
+When a primary Gitaly node fails, there is a chance of data loss. Data loss can occur if there were outstanding replication jobs the secondaries did not manage to process before the failure. The Praefect `dataloss` sub-command helps identify these cases by counting the number of dead replication jobs for each repository within a given time frame.
 
 ```shell
 sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml dataloss -from <rfc3339-time> -to <rfc3339-time>
 ```
 
-If the timeframe is not specified, dead replication jobs from the last six hours are counted:
+If the time frame is not specified, dead replication jobs from the last six hours are counted:
 
 ```shell
 sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml dataloss
@@ -770,19 +789,27 @@ example/repository-2: 4 jobs
 example/repository-3: 2 jobs
 ```
 
-To specify a timeframe in UTC, run:
+To specify a time frame in UTC, run:
 
 ```shell
 sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml dataloss -from 2020-01-02T00:00:00+00:00 -to 2020-01-02T00:02:00+00:00
 ```
 
+### Checking repository checksums
+
+To check a project's checksums across all nodes, the Praefect replicas Rake task can be used:
+
+```shell
+sudo gitlab-rake "gitlab:praefect:replicas[project_id]"
+```
+
 ## Backend Node Recovery
 
 When a Praefect backend node fails and is no longer able to
 replicate changes, the backend node will start to drift from the primary. If
 that node eventually recovers, it will need to be reconciled with the current
 primary. The primary node is considered the single source of truth for the
-state of a shard. The Praefect `reconcile` subcommand allows for the manual
+state of a shard. The Praefect `reconcile` sub-command allows for the manual
 reconciliation between a backend node and the current primary.
 
 Run the following command on the Praefect server after all placeholders
diff --git a/doc/administration/gitaly/reference.md b/doc/administration/gitaly/reference.md
index 6b6919247fe..e718d8953ca 100644
--- a/doc/administration/gitaly/reference.md
+++ b/doc/administration/gitaly/reference.md
@@ -149,7 +149,7 @@ before and after. If the hit ratio does not improve, the higher limit is
 probably not making a meaningful difference. Here is an example
 Prometheus query to see the hit rate:
 
-```text
+```plaintext
 sum(rate(gitaly_catfile_cache_total{type="hit"}[5m])) / sum(rate(gitaly_catfile_cache_total{type=~"(hit)|(miss)"}[5m]))
 ```
 
@@ -258,7 +258,7 @@ You can adjust the `concurrency` of each RPC endpoint.
 | ---- | ---- | -------- | ----------- |
 | `concurrency` | array | yes | An array of RPC endpoints. |
 | `rpc` | string | no | The name of the RPC endpoint (`/gitaly.RepositoryService/GarbageCollect`). |
-| `max_per_repo` | integer | no | Concurrency per RPC per repo. |
+| `max_per_repo` | integer | no | Concurrency per RPC per repository. |
 
 Example: