Add latest changes from gitlab-org/gitlab@14-6-stable-eev14.6.0-rc42

5 files changed, 700 insertions, 448 deletions

diff --git a/doc/administration/gitaly/configure_gitaly.md b/doc/administration/gitaly/configure_gitaly.md
index d0841f4e607..b31a02aae0a 100644
--- a/doc/administration/gitaly/configure_gitaly.md
+++ b/doc/administration/gitaly/configure_gitaly.md
@@ -350,6 +350,10 @@ leading to `Error creating pipeline` and `Commit not found` errors, or stale dat
 As the final step, you must update Gitaly clients to switch from using local Gitaly service to use
 the Gitaly servers you just configured.
 
+NOTE:
+GitLab requires a `default` repository storage to be configured.
+[Read more about this limitation](#gitlab-requires-a-default-repository-storage).
+
 This can be risky because anything that prevents your Gitaly clients from reaching the Gitaly
 servers causes all Gitaly requests to fail. For example, any sort of network, firewall, or name
 resolution problems.
@@ -489,6 +493,18 @@ gitaly['key_path'] = "/etc/gitlab/ssl/key.pem"
 `path` can be included only for storage shards on the local Gitaly server.
 If it's excluded, default Git storage directory is used for that storage shard.
 
+### GitLab requires a default repository storage
+
+When adding Gitaly servers to an environment, you might want to replace the original `default` Gitaly service. However, you can't
+reconfigure the GitLab application servers to remove the `default` entry from `git_data_dirs` because GitLab requires a
+`git_data_dirs` entry called `default`. [Read more](https://gitlab.com/gitlab-org/gitlab/-/issues/36175) about this limitation.
+
+To work around the limitation:
+
+1. Define an additional storage location on the new Gitaly service and configure the additional storage to be `default`.
+1. In the [Admin Area](../repository_storage_paths.md#configure-where-new-repositories-are-stored), set `default` to a weight of zero
+   to prevent repositories being stored there.
+
 ### Disable Gitaly where not required (optional)
 
 If you run Gitaly [as a remote service](#run-gitaly-on-its-own-server), consider
@@ -605,7 +621,7 @@ To configure Gitaly with TLS:
 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
 1. Verify Gitaly traffic is being served over TLS by
    [observing the types of Gitaly connections](#observe-type-of-gitaly-connections).
-1. (Optional) Improve security by:
+1. Optional. Improve security by:
    1. Disabling non-TLS connections by commenting out or deleting `gitaly['listen_addr']` in
       `/etc/gitlab/gitlab.rb`.
    1. Saving the file.
@@ -681,7 +697,7 @@ To configure Gitaly with TLS:
 1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source).
 1. Verify Gitaly traffic is being served over TLS by
    [observing the types of Gitaly connections](#observe-type-of-gitaly-connections).
-1. (Optional) Improve security by:
+1. Optional. Improve security by:
    1. Disabling non-TLS connections by commenting out or deleting `listen_addr` in
       `/home/git/gitaly/config.toml`.
    1. Saving the file.
diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md
index c689530e12c..f99bbf21840 100644
--- a/doc/administration/gitaly/index.md
+++ b/doc/administration/gitaly/index.md
@@ -189,8 +189,7 @@ The availability objectives for Gitaly clusters assuming a single node failure a
   Writes are replicated asynchronously. Any writes that have not been replicated
   to the newly promoted primary are lost.
 
-  [Strong consistency](#strong-consistency) can be used to avoid loss in some
-  circumstances.
+  [Strong consistency](#strong-consistency) prevents loss in some circumstances.
 
 - **Recovery Time Objective (RTO):** Less than 10 seconds.
   Outages are detected by a health check run by each Praefect node every
@@ -284,8 +283,7 @@ Gitaly Cluster provides the following features:
 - [Replication factor](#replication-factor) of repositories for increased redundancy.
 - [Automatic failover](praefect.md#automatic-failover-and-primary-election-strategies) from the
   primary Gitaly node to secondary Gitaly nodes.
-- Reporting of possible [data loss](praefect.md#check-for-data-loss) if replication queue is
-  non-empty.
+- Reporting of possible [data loss](recovery.md#check-for-data-loss) if replication queue isn't empty.
 
 Follow the [Gitaly Cluster epic](https://gitlab.com/groups/gitlab-org/-/epics/1489) for improvements
 including [horizontally distributing reads](https://gitlab.com/groups/gitlab-org/-/epics/2013).
@@ -323,18 +321,26 @@ You can [monitor distribution of reads](#monitor-gitaly-cluster) using Prometheu
 > - In GitLab 13.3, disabled unless primary-wins voting strategy is disabled.
 > - From GitLab 13.4, enabled by default.
 > - From GitLab 13.5, you must use Git v2.28.0 or higher on Gitaly nodes to enable strong consistency.
-> - From GitLab 13.6, primary-wins voting strategy and `gitaly_reference_transactions_primary_wins` feature flag were removed from the source code.
+> - From GitLab 13.6, primary-wins voting strategy and the `gitaly_reference_transactions_primary_wins` feature flag was removed.
+> - From GitLab 14.0, [Gitaly Cluster only supports strong consistency](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/3575), and the `gitaly_reference_transactions` feature flag was removed.
 
-By default, Gitaly Cluster guarantees eventual consistency by replicating all writes to secondary
-Gitaly nodes after the write to the primary Gitaly node has happened.
+Gitaly Cluster provides strong consistency by writing changes synchronously to all healthy, up-to-date replicas. If a
+replica is outdated or unhealthy at the time of the transaction, the write is asynchronously replicated to it.
 
-Praefect can instead provide strong consistency by creating a transaction and writing changes to all
-Gitaly nodes at once.
+If strong consistency is unavailable, Gitaly Cluster guarantees eventual consistency. In this case. Gitaly Cluster
+replicates all writes to secondary Gitaly nodes after the write to the primary Gitaly node has occurred.
 
-If enabled, transactions are only available for a subset of RPCs. For more information, see the
-[strong consistency epic](https://gitlab.com/groups/gitlab-org/-/epics/1189).
+Strong consistency:
 
-For configuration information, see [Configure strong consistency](praefect.md#configure-strong-consistency).
+- Is the primary replication method in GitLab 14.0 and later. A subset of operations still use replication jobs
+  (eventual consistency) instead of strong consistency. Refer to the
+  [strong consistency epic](https://gitlab.com/groups/gitlab-org/-/epics/1189) for more information.
+- Must be configured in GitLab versions 13.1 to 13.12. For configuration information, refer to either:
+  - Documentation on your GitLab instance at `/help`.
+  - The [13.12 documentation](https://docs.gitlab.com/13.12/ee/administration/gitaly/praefect.html#strong-consistency).
+- Is unavailable in GitLab 13.0 and earlier.
+
+For more information on monitoring strong consistency, see the Gitaly Cluster [Prometheus metrics documentation](#monitor-gitaly-cluster).
 
 #### Replication factor
 
@@ -368,6 +374,10 @@ WARNING:
 Some [known database inconsistency issues](#known-issues) exist in Gitaly Cluster. We recommend you
 remain on your current service for now.
 
+NOTE:
+GitLab requires a `default` repository storage to be configured.
+[Read more about this limitation](configure_gitaly.md#gitlab-requires-a-default-repository-storage).
+
 ### Migrate off Gitaly Cluster
 
 If you have repositories stored on a Gitaly Cluster, but you'd like to migrate
@@ -513,6 +523,10 @@ To monitor [strong consistency](#strong-consistency), you can use the following
 
 You can also monitor the [Praefect logs](../logs.md#praefect-logs).
 
+## Recover from failure
+
+Gitaly Cluster can [recover from certain types of failure](recovery.md).
+
 ## Do not bypass Gitaly
 
 GitLab doesn't advise directly accessing Gitaly repositories stored on disk with a Git client,
diff --git a/doc/administration/gitaly/praefect.md b/doc/administration/gitaly/praefect.md
index da456131a52..d3a8662080f 100644
--- a/doc/administration/gitaly/praefect.md
+++ b/doc/administration/gitaly/praefect.md
@@ -215,6 +215,38 @@ The database used by Praefect is now configured.
 If you see Praefect database errors after configuring PostgreSQL, see
 [troubleshooting steps](troubleshooting.md#relation-does-not-exist-errors).
 
+#### Reads distribution caching
+
+Praefect performance can be improved by additionally configuring the `database_direct`
+settings:
+
+```ruby
+praefect['database_direct_host'] = POSTGRESQL_HOST
+praefect['database_direct_port'] = 5432
+
+# Use the following to override parameters of direct database connection.
+# Comment out where the parameters are the same for both connections.
+
+praefect['database_direct_user'] = 'praefect'
+praefect['database_direct_password'] = PRAEFECT_SQL_PASSWORD
+praefect['database_direct_dbname'] = 'praefect_production'
+#praefect['database_direct_sslmode'] = '...'
+#praefect['database_direct_sslcert'] = '...'
+#praefect['database_direct_sslkey'] = '...'
+#praefect['database_direct_sslrootcert'] = '...'
+```
+
+Once configured, this connection is automatically used for the
+[SQL LISTEN](https://www.postgresql.org/docs/11/sql-listen.html) feature and
+allows Praefect to receive notifications from PostgreSQL for cache invalidation.
+
+Verify this feature is working by looking for the following log entry in the Praefect
+log:
+
+```plaintext
+reads distribution caching is enabled by configuration
+```
+
 #### Use PgBouncer
 
 To reduce PostgreSQL resource consumption, we recommend setting up and configuring
@@ -223,7 +255,7 @@ this, you must point Praefect to PgBouncer by setting Praefect database paramete
 
 ```ruby
 praefect['database_host'] = PGBOUNCER_HOST
-praefect['database_port'] = 6432
+praefect['database_port'] = 5432
 praefect['database_user'] = 'praefect'
 praefect['database_password'] = PRAEFECT_SQL_PASSWORD
 praefect['database_dbname'] = 'praefect_production'
@@ -1073,31 +1105,6 @@ To get started quickly:
 Congratulations! You've configured an observable fault-tolerant Praefect
 cluster.
 
-## Configure strong consistency
-
-To enable [strong consistency](index.md#strong-consistency):
-
-- In GitLab 13.5, you must use Git v2.28.0 or higher on Gitaly nodes to enable strong consistency.
-- In GitLab 13.4 and later, the strong consistency voting strategy has been improved and enabled by default.
-  Instead of requiring all nodes to agree, only the primary and half of the secondaries need to agree.
-- In GitLab 13.3, reference transactions are enabled by default with a primary-wins strategy.
-  This strategy causes all transactions to succeed for the primary and thus does not ensure strong consistency.
-  To enable strong consistency, disable the `:gitaly_reference_transactions_primary_wins` feature flag.
-- In GitLab 13.2, enable the `:gitaly_reference_transactions` feature flag.
-- In GitLab 13.1, enable the `:gitaly_reference_transactions` and `:gitaly_hooks_rpc`
-  feature flags.
-
-Changing feature flags requires [access to the Rails console](../feature_flags.md#start-the-gitlab-rails-console).
-In the Rails console, enable or disable the flags as required. For example:
-
-```ruby
-Feature.enable(:gitaly_reference_transactions)
-Feature.disable(:gitaly_reference_transactions_primary_wins)
-```
-
-For information on monitoring strong consistency, see the
-[relevant documentation](index.md#monitor-gitaly-cluster).
-
 ## Configure replication factor
 
 WARNING:
@@ -1153,8 +1160,7 @@ Praefect regularly checks the health of each Gitaly node. This is used to automa
 to a newly-elected primary Gitaly node if the current primary node is found to be unhealthy.
 
 We recommend using [repository-specific primary nodes](#repository-specific-primary-nodes). This is
-[planned to be the only available election strategy](https://gitlab.com/gitlab-org/gitaly/-/issues/3574)
-from GitLab 14.0.
+[the only available election strategy](https://gitlab.com/gitlab-org/gitaly/-/issues/3574) from GitLab 14.0.
 
 ### Repository-specific primary nodes
 
@@ -1268,7 +1274,7 @@ To migrate existing clusters:
 ### Deprecated election strategies
 
 WARNING:
-The below election strategies are deprecated and are scheduled for removal in GitLab 14.0.
+The below election strategies are deprecated and were removed in GitLab 14.0.
 Migrate to [repository-specific primary nodes](#repository-specific-primary-nodes).
 
 - **PostgreSQL:** Enabled by default until GitLab 14.0, and equivalent to:
@@ -1287,397 +1293,3 @@ Migrate to [repository-specific primary nodes](#repository-specific-primary-node
   If a sufficient number of health checks fail for the current primary Gitaly node, a new primary is
   elected. **Do not use with multiple Praefect nodes!** Using with multiple Praefect nodes is
   likely to result in a split brain.
-
-## Primary Node Failure
-
-Gitaly Cluster recovers from a failing primary Gitaly node by promoting a healthy secondary as the
-new primary.
-
-In GitLab 14.1 and later, Gitaly Cluster:
-
-- Elects a healthy secondary with a fully up to date copy of the repository as the new primary.
-- Repository becomes unavailable if there are no fully up to date copies of it on healthy secondaries.
-
-To minimize data loss in GitLab 13.0 to 14.0, Gitaly Cluster:
-
-- Switches repositories that are outdated on the new primary to [read-only mode](#read-only-mode).
-- Elects the secondary with the least unreplicated writes from the primary to be the new
-  primary. Because there can still be some unreplicated writes,
-  [data loss can occur](#check-for-data-loss).
-
-### Read-only mode
-
-> - Introduced in GitLab 13.0 as [generally available](https://about.gitlab.com/handbook/product/gitlab-the-product/#generally-available-ga).
-> - Between GitLab 13.0 and GitLab 13.2, read-only mode applied to the whole virtual storage and occurred whenever failover occurred.
-> - [In GitLab 13.3 and later](https://gitlab.com/gitlab-org/gitaly/-/issues/2862), read-only mode applies on a per-repository basis and only occurs if a new primary is out of date.
-new primary. If the failed primary contained unreplicated writes, [data loss can occur](#check-for-data-loss).
-> - Removed in GitLab 14.1. Instead, repositories [become unavailable](#unavailable-repositories).
-
-When Gitaly Cluster switches to a new primary in GitLab 13.0 to 14.0, repositories enter
-read-only mode if they are out of date. This can happen after failing over to an outdated
-secondary. Read-only mode eases data recovery efforts by preventing writes that may conflict
-with the unreplicated writes on other nodes.
-
-To enable writes again in GitLab 13.0 to 14.0, an administrator can:
-
-1. [Check](#check-for-data-loss) for data loss.
-1. Attempt to [recover](#data-recovery) missing data.
-1. Either [enable writes](#enable-writes-or-accept-data-loss) in the virtual storage or
-   [accept data loss](#enable-writes-or-accept-data-loss) if necessary, depending on the version of
-   GitLab.
-
-## Unavailable repositories
-
-> - From GitLab 13.0 through 14.0, repositories became read-only if they were outdated on the primary but fully up to date on a healthy secondary. `dataloss` sub-command displays read-only repositories by default through these versions.
-> - Since GitLab 14.1, Praefect contains more responsive failover logic which immediately fails over to one of the fully up to date secondaries rather than placing the repository in read-only mode. Since GitLab 14.1, the `dataloss` sub-command displays repositories which are unavailable due to having no fully up to date copies on healthy Gitaly nodes.
-
-A repository is unavailable if all of its up to date replicas are unavailable. Unavailable repositories are
-not accessible through Praefect to prevent serving stale data that may break automated tooling.
-
-### Check for data loss
-
-The Praefect `dataloss` subcommand identifies:
-
-- Copies of repositories in GitLab 13.0 to GitLab 14.0 that at are likely to be outdated.
-  This can help identify potential data loss after a failover.
-- Repositories in GitLab 14.1 and later that are unavailable. This helps identify potential
-  data loss and repositories which are no longer accessible because all of their up-to-date
-  replicas copies are unavailable.
-
-The following parameters are available:
-
-- `-virtual-storage` that specifies which virtual storage to check. Because they might require
-  an administrator to intervene, the default behavior is to display:
-  - In GitLab 13.0 to 14.0, copies of read-only repositories.
-  - In GitLab 14.1 and later, unavailable repositories.
-- In GitLab 14.1 and later, [`-partially-unavailable`](#unavailable-replicas-of-available-repositories)
-  that specifies whether to include in the output repositories that are available but have
-  some assigned copies that are not available.
-
-NOTE:
-`dataloss` is still in beta and the output format is subject to change.
-
-To check for repositories with outdated primaries or for unavailable repositories, run:
-
-```shell
-sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml dataloss [-virtual-storage <virtual-storage>]
-```
-
-Every configured virtual storage is checked if none is specified:
-
-```shell
-sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml dataloss
-```
-
-Repositories are listed in the output that have either:
-
-- An outdated copy of the repository on the primary, in GitLab 13.0 to GitLab 14.0.
-- No healthy and fully up-to-date copies available, in GitLab 14.1 and later.
-
-The following information is printed for each repository:
-
-- A repository's relative path to the storage directory identifies each repository and groups the related
-  information.
-- The repository's current status is printed in parentheses next to the disk path:
-  - In GitLab 13.0 to 14.0, either `(read-only)` if the repository's primary node is outdated
-    and can't accept writes. Otherwise, `(writable)`.
-  - In GitLab 14.1 and later, `(unavailable)` is printed next to the disk path if the
-    repository is unavailable.
-- The primary field lists the repository's current primary. If the repository has no primary, the field shows
-  `No Primary`.
-- The In-Sync Storages lists replicas which have replicated the latest successful write and all writes
-  preceding it.
-- The Outdated Storages lists replicas which contain an outdated copy of the repository. Replicas which have no copy
-  of the repository but should contain it are also listed here. The maximum number of changes the replica is missing
-  is listed next to replica. It's important to notice that the outdated replicas may be fully up to date or contain
-  later changes but Praefect can't guarantee it.
-
-Additional information includes:
-
-- Whether a node is assigned to host the repository is listed with each node's status.
-  `assigned host` is printed next to nodes that are assigned to store the repository. The
-  text is omitted if the node contains a copy of the repository but is not assigned to store
-  the repository. Such copies aren't kept in sync by Praefect, but may act as replication
-  sources to bring assigned copies up to date.
-- In GitLab 14.1 and later, `unhealthy` is printed next to the copies that are located
-  on unhealthy Gitaly nodes.
-
-Example output:
-
-```shell
-Virtual storage: default
-  Outdated repositories:
-    @hashed/3f/db/3fdba35f04dc8c462986c992bcf875546257113072a909c162f7e470e581e278.git (unavailable):
-      Primary: gitaly-1
-      In-Sync Storages:
-        gitaly-2, assigned host, unhealthy
-      Outdated Storages:
-        gitaly-1 is behind by 3 changes or less, assigned host
-        gitaly-3 is behind by 3 changes or less
-```
-
-A confirmation is printed out when every repository is available. For example:
-
-```shell
-Virtual storage: default
-  All repositories are available!
-```
-
-#### Unavailable replicas of available repositories
-
-NOTE:
-In GitLab 14.0 and earlier, the flag is `-partially-replicated` and the output shows any repositories with assigned nodes with outdated
-copies.
-
-To also list information of repositories which are available but are unavailable from some of the assigned nodes,
-use the `-partially-unavailable` flag.
-
-A repository is available if there is a healthy, up to date replica available. Some of the assigned secondary
-replicas may be temporarily unavailable for access while they are waiting to replicate the latest changes.
-
-```shell
-sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml dataloss [-virtual-storage <virtual-storage>] [-partially-unavailable]
-```
-
-Example output:
-
-```shell
-Virtual storage: default
-  Outdated repositories:
-    @hashed/3f/db/3fdba35f04dc8c462986c992bcf875546257113072a909c162f7e470e581e278.git:
-      Primary: gitaly-1
-      In-Sync Storages:
-        gitaly-1, assigned host
-      Outdated Storages:
-        gitaly-2 is behind by 3 changes or less, assigned host
-        gitaly-3 is behind by 3 changes or less
-```
-
-With the `-partially-unavailable` flag set, a confirmation is printed out if every assigned replica is fully up to
-date and healthy.
-
-For example:
-
-```shell
-Virtual storage: default
-  All repositories are fully available on all assigned storages!
-```
-
-### Check repository checksums
-
-To check a project's repository checksums across on all Gitaly nodes, run the
-[replicas Rake task](../raketasks/praefect.md#replica-checksums) on the main GitLab node.
-
-### Accept data loss
-
-WARNING:
-`accept-dataloss` causes permanent data loss by overwriting other versions of the repository. Data
-[recovery efforts](#data-recovery) must be performed before using it.
-
-If it is not possible to bring one of the up to date replicas back online, you may have to accept data
-loss. When accepting data loss, Praefect marks the chosen replica of the repository as the latest version
-and replicates it to the other assigned Gitaly nodes. This process overwrites any other version of the
-repository so care must be taken.
-
-```shell
-sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml accept-dataloss
--virtual-storage <virtual-storage> -repository <relative-path> -authoritative-storage <storage-name>
-```
-
-### Enable writes or accept data loss
-
-WARNING:
-`accept-dataloss` causes permanent data loss by overwriting other versions of the repository.
-Data [recovery efforts](#data-recovery) must be performed before using it.
-
-Praefect provides the following subcommands to re-enable writes or accept data loss:
-
-- In GitLab 13.2 and earlier, `enable-writes` to re-enable virtual storage for writes after
-  data recovery attempts:
-
-  ```shell
-  sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml enable-writes -virtual-storage <virtual-storage>
-  ```
-
-- In GitLab 13.3 and later, if it is not possible to bring one of the up to date nodes back
-  online, you may have to accept data loss:
-
-  ```shell
-  sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml accept-dataloss -virtual-storage <virtual-storage> -repository <relative-path> -authoritative-storage <storage-name>
-  ```
-
-  When accepting data loss, Praefect:
-
-  1. Marks the chosen copy of the repository as the latest version.
-  1. Replicates the copy to the other assigned Gitaly nodes.
-
-  This process overwrites any other copy of the repository so care must be taken.
-
-## Data recovery
-
-If a Gitaly node fails replication jobs for any reason, it ends up hosting outdated versions of the
-affected repositories. Praefect provides tools for:
-
-- [Automatic](#automatic-reconciliation) reconciliation, for GitLab 13.4 and later.
-- [Manual](#manual-reconciliation) reconciliation, for:
-  - GitLab 13.3 and earlier.
-  - Repositories upgraded to GitLab 13.4 and later without entries in the `repositories` table. In
-    GitLab 13.6 and later, [a migration is run](https://gitlab.com/gitlab-org/gitaly/-/issues/3033)
-    when Praefect starts for these repositories.
-
-These tools reconcile the outdated repositories to bring them fully up to date again.
-
-### Automatic reconciliation
-
-> [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/2717) in GitLab 13.4.
-
-Praefect automatically reconciles repositories that are not up to date. By default, this is done every
-five minutes. For each outdated repository on a healthy Gitaly node, the Praefect picks a
-random, fully up-to-date replica of the repository on another healthy Gitaly node to replicate from. A
-replication job is scheduled only if there are no other replication jobs pending for the target
-repository.
-
-The reconciliation frequency can be changed via the configuration. The value can be any valid
-[Go duration value](https://golang.org/pkg/time/#ParseDuration). Values below 0 disable the feature.
-
-Examples:
-
-```ruby
-praefect['reconciliation_scheduling_interval'] = '5m' # the default value
-```
-
-```ruby
-praefect['reconciliation_scheduling_interval'] = '30s' # reconcile every 30 seconds
-```
-
-```ruby
-praefect['reconciliation_scheduling_interval'] = '0' # disable the feature
-```
-
-### Manual reconciliation
-
-WARNING:
-The `reconcile` sub-command was removed in GitLab 14.1. Use [automatic reconciliation](#automatic-reconciliation) instead. Manual reconciliation may produce excess replication jobs and is limited in functionality. Manual reconciliation does not work when [repository-specific primary nodes](#repository-specific-primary-nodes) are
-enabled.
-
-The Praefect `reconcile` sub-command allows for the manual reconciliation between two Gitaly nodes. The
-command replicates every repository on a later version on the reference storage to the target storage.
-
-```shell
-sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml reconcile -virtual <virtual-storage> -reference <up-to-date-storage> -target <outdated-storage> -f
-```
-
-- Replace the placeholder `<virtual-storage>` with the virtual storage containing the Gitaly node storage to be checked.
-- Replace the placeholder `<up-to-date-storage>` with the Gitaly storage name containing up to date repositories.
-- Replace the placeholder `<outdated-storage>` with the Gitaly storage name containing outdated repositories.
-
-### Manually remove repositories
-
-> [Introduced](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/3767) in GitLab 14.3.
-
-The `remove-repository` Praefect sub-command removes repositories from a Gitaly Cluster. It removes
-all state associated with a given repository including:
-
-- On-disk repositories on all relevant Gitaly nodes.
-- Any database state tracked by Praefect.
-
-```shell
-sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml remove-repository -virtual-storage <virtual-storage> -repository <repository>
-```
-
-- `-virtual-storage` is the virtual storage the repository is located in. Virtual storages are configured in `/etc/gitlab/gitlab.rb` under `praefect['virtual_storages]` and looks like the following:
-
-  ```ruby
-  praefect['virtual_storages'] = {
-    'default' => {
-      ...
-    },
-    'storage-1' => {
-      ...
-    }
-  }
-  ```
-
-  In this example, the virtual storage to specify is `default` or `storage-1`.
-
-- `-repository` is the repository's relative path in the storage [beginning with `@hashed`](../repository_storage_types.md#hashed-storage).
-  For example:
-
-  ```plaintext
-  @hashed/f5/ca/f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b.git
-  ```
-
-Parts of the repository can continue to exist after running `remove-repository`. This can be because of:
-
-- A deletion error.
-- An in-flight RPC call targeting the repository.
-
-If this occurs, run `remove-repository` again.
-
-### Manually list untracked repositories
-
-> [Introduced](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/3926) in GitLab 14.4.
-
-The `list-untracked-repositories` Praefect sub-command lists repositories of the Gitaly Cluster that both:
-
-- Exist for at least one Gitaly storage.
-- Aren't tracked in the Praefect database.
-
-The command outputs:
-
-- Result to `STDOUT` and the command's logs.
-- Errors to `STDERR`.
-
-Each entry is a complete JSON string with a newline at the end (configurable using the
-`-delimiter` flag). For example:
-
-```plaintext
-sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml list-untracked-repositories
-{"virtual_storage":"default","storage":"gitaly-1","relative_path":"@hashed/ab/cd/abcd123456789012345678901234567890123456789012345678901234567890.git"}
-{"virtual_storage":"default","storage":"gitaly-1","relative_path":"@hashed/ab/cd/abcd123456789012345678901234567890123456789012345678901234567891.git"}
-```
-
-### Manually track repositories
-
-> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/5658) in GitLab 14.4.
-
-The `track-repository` Praefect sub-command adds repositories on disk to the Praefect database to be tracked.
-
-```shell
-sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml track-repository -virtual-storage <virtual-storage> -repository <repository>
-```
-
-- `-virtual-storage` is the virtual storage the repository is located in. Virtual storages are configured in `/etc/gitlab/gitlab.rb` under `praefect['virtual_storages]` and looks like the following:
-
-  ```ruby
-  praefect['virtual_storages'] = {
-    'default' => {
-      ...
-    },
-    'storage-1' => {
-      ...
-    }
-  }
-  ```
-
-  In this example, the virtual storage to specify is `default` or `storage-1`.
-
-- `-repository` is the repository's relative path in the storage [beginning with `@hashed`](../repository_storage_types.md#hashed-storage).
-  For example:
-
-  ```plaintext
-  @hashed/f5/ca/f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b.git
-  ```
-
-- `-authoritative-storage` is the storage we want Praefect to treat as the primary. Required if
-  [per-repository replication](#configure-replication-factor) is set as the replication strategy.
-
-The command outputs:
-
-- Results to `STDOUT` and the command's logs.
-- Errors to `STDERR`.
-
-This command fails if:
-
-- The repository is already being tracked by the Praefect database.
-- The repository does not exist on disk.
diff --git a/doc/administration/gitaly/recovery.md b/doc/administration/gitaly/recovery.md
new file mode 100644
index 00000000000..e1b9a73908d
--- /dev/null
+++ b/doc/administration/gitaly/recovery.md
@@ -0,0 +1,418 @@
+---
+stage: Create
+group: Gitaly
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
+type: reference
+---
+
+# Recovery options
+
+Gitaly Cluster can [recover from certain types of failure](recovery.md).
+
+## Primary Node Failure
+
+Gitaly Cluster recovers from a failing primary Gitaly node by promoting a healthy secondary as the
+new primary.
+
+In GitLab 14.1 and later, Gitaly Cluster:
+
+- Elects a healthy secondary with a fully up to date copy of the repository as the new primary.
+- Repository becomes unavailable if there are no fully up to date copies of it on healthy secondaries.
+
+To minimize data loss in GitLab 13.0 to 14.0, Gitaly Cluster:
+
+- Switches repositories that are outdated on the new primary to [read-only mode](#read-only-mode).
+- Elects the secondary with the least unreplicated writes from the primary to be the new
+  primary. Because there can still be some unreplicated writes,
+  [data loss can occur](#check-for-data-loss).
+
+### Read-only mode
+
+> - Introduced in GitLab 13.0 as [generally available](https://about.gitlab.com/handbook/product/gitlab-the-product/#generally-available-ga).
+> - Between GitLab 13.0 and GitLab 13.2, read-only mode applied to the whole virtual storage and occurred whenever failover occurred.
+> - [In GitLab 13.3 and later](https://gitlab.com/gitlab-org/gitaly/-/issues/2862), read-only mode applies on a per-repository basis and only occurs if a new primary is out of date.
+new primary. If the failed primary contained unreplicated writes, [data loss can occur](#check-for-data-loss).
+> - Removed in GitLab 14.1. Instead, repositories [become unavailable](#unavailable-repositories).
+
+When Gitaly Cluster switches to a new primary in GitLab 13.0 to 14.0, repositories enter
+read-only mode if they are out of date. This can happen after failing over to an outdated
+secondary. Read-only mode eases data recovery efforts by preventing writes that may conflict
+with the unreplicated writes on other nodes.
+
+To enable writes again in GitLab 13.0 to 14.0, an administrator can:
+
+1. [Check](#check-for-data-loss) for data loss.
+1. Attempt to [recover](#data-recovery) missing data.
+1. Either [enable writes](#enable-writes-or-accept-data-loss) in the virtual storage or
+   [accept data loss](#enable-writes-or-accept-data-loss) if necessary, depending on the version of
+   GitLab.
+
+## Unavailable repositories
+
+> - From GitLab 13.0 through 14.0, repositories became read-only if they were outdated on the primary but fully up to date on a healthy secondary. `dataloss` sub-command displays read-only repositories by default through these versions.
+> - Since GitLab 14.1, Praefect contains more responsive failover logic which immediately fails over to one of the fully up to date secondaries rather than placing the repository in read-only mode. Since GitLab 14.1, the `dataloss` sub-command displays repositories which are unavailable due to having no fully up to date copies on healthy Gitaly nodes.
+
+A repository is unavailable if all of its up to date replicas are unavailable. Unavailable repositories are
+not accessible through Praefect to prevent serving stale data that may break automated tooling.
+
+### Check for data loss
+
+The Praefect `dataloss` subcommand identifies:
+
+- Copies of repositories in GitLab 13.0 to GitLab 14.0 that at are likely to be outdated.
+  This can help identify potential data loss after a failover.
+- Repositories in GitLab 14.1 and later that are unavailable. This helps identify potential
+  data loss and repositories which are no longer accessible because all of their up-to-date
+  replicas copies are unavailable.
+
+The following parameters are available:
+
+- `-virtual-storage` that specifies which virtual storage to check. Because they might require
+  an administrator to intervene, the default behavior is to display:
+  - In GitLab 13.0 to 14.0, copies of read-only repositories.
+  - In GitLab 14.1 and later, unavailable repositories.
+- In GitLab 14.1 and later, [`-partially-unavailable`](#unavailable-replicas-of-available-repositories)
+  that specifies whether to include in the output repositories that are available but have
+  some assigned copies that are not available.
+
+NOTE:
+`dataloss` is still in beta and the output format is subject to change.
+
+To check for repositories with outdated primaries or for unavailable repositories, run:
+
+```shell
+sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml dataloss [-virtual-storage <virtual-storage>]
+```
+
+Every configured virtual storage is checked if none is specified:
+
+```shell
+sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml dataloss
+```
+
+Repositories are listed in the output that have either:
+
+- An outdated copy of the repository on the primary, in GitLab 13.0 to GitLab 14.0.
+- No healthy and fully up-to-date copies available, in GitLab 14.1 and later.
+
+The following information is printed for each repository:
+
+- A repository's relative path to the storage directory identifies each repository and groups the related
+  information.
+- The repository's current status is printed in parentheses next to the disk path:
+  - In GitLab 13.0 to 14.0, either `(read-only)` if the repository's primary node is outdated
+    and can't accept writes. Otherwise, `(writable)`.
+  - In GitLab 14.1 and later, `(unavailable)` is printed next to the disk path if the
+    repository is unavailable.
+- The primary field lists the repository's current primary. If the repository has no primary, the field shows
+  `No Primary`.
+- The In-Sync Storages lists replicas which have replicated the latest successful write and all writes
+  preceding it.
+- The Outdated Storages lists replicas which contain an outdated copy of the repository. Replicas which have no copy
+  of the repository but should contain it are also listed here. The maximum number of changes the replica is missing
+  is listed next to replica. It's important to notice that the outdated replicas may be fully up to date or contain
+  later changes but Praefect can't guarantee it.
+
+Additional information includes:
+
+- Whether a node is assigned to host the repository is listed with each node's status.
+  `assigned host` is printed next to nodes that are assigned to store the repository. The
+  text is omitted if the node contains a copy of the repository but is not assigned to store
+  the repository. Such copies aren't kept in sync by Praefect, but may act as replication
+  sources to bring assigned copies up to date.
+- In GitLab 14.1 and later, `unhealthy` is printed next to the copies that are located
+  on unhealthy Gitaly nodes.
+
+Example output:
+
+```shell
+Virtual storage: default
+  Outdated repositories:
+    @hashed/3f/db/3fdba35f04dc8c462986c992bcf875546257113072a909c162f7e470e581e278.git (unavailable):
+      Primary: gitaly-1
+      In-Sync Storages:
+        gitaly-2, assigned host, unhealthy
+      Outdated Storages:
+        gitaly-1 is behind by 3 changes or less, assigned host
+        gitaly-3 is behind by 3 changes or less
+```
+
+A confirmation is printed out when every repository is available. For example:
+
+```shell
+Virtual storage: default
+  All repositories are available!
+```
+
+#### Unavailable replicas of available repositories
+
+NOTE:
+In GitLab 14.0 and earlier, the flag is `-partially-replicated` and the output shows any repositories with assigned nodes with outdated
+copies.
+
+To also list information of repositories which are available but are unavailable from some of the assigned nodes,
+use the `-partially-unavailable` flag.
+
+A repository is available if there is a healthy, up to date replica available. Some of the assigned secondary
+replicas may be temporarily unavailable for access while they are waiting to replicate the latest changes.
+
+```shell
+sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml dataloss [-virtual-storage <virtual-storage>] [-partially-unavailable]
+```
+
+Example output:
+
+```shell
+Virtual storage: default
+  Outdated repositories:
+    @hashed/3f/db/3fdba35f04dc8c462986c992bcf875546257113072a909c162f7e470e581e278.git:
+      Primary: gitaly-1
+      In-Sync Storages:
+        gitaly-1, assigned host
+      Outdated Storages:
+        gitaly-2 is behind by 3 changes or less, assigned host
+        gitaly-3 is behind by 3 changes or less
+```
+
+With the `-partially-unavailable` flag set, a confirmation is printed out if every assigned replica is fully up to
+date and healthy.
+
+For example:
+
+```shell
+Virtual storage: default
+  All repositories are fully available on all assigned storages!
+```
+
+### Check repository checksums
+
+To check a project's repository checksums across on all Gitaly nodes, run the
+[replicas Rake task](../raketasks/praefect.md#replica-checksums) on the main GitLab node.
+
+### Accept data loss
+
+WARNING:
+`accept-dataloss` causes permanent data loss by overwriting other versions of the repository. Data
+[recovery efforts](#data-recovery) must be performed before using it.
+
+If it is not possible to bring one of the up to date replicas back online, you may have to accept data
+loss. When accepting data loss, Praefect marks the chosen replica of the repository as the latest version
+and replicates it to the other assigned Gitaly nodes. This process overwrites any other version of the
+repository so care must be taken.
+
+```shell
+sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml accept-dataloss
+-virtual-storage <virtual-storage> -repository <relative-path> -authoritative-storage <storage-name>
+```
+
+### Enable writes or accept data loss
+
+WARNING:
+`accept-dataloss` causes permanent data loss by overwriting other versions of the repository.
+Data [recovery efforts](#data-recovery) must be performed before using it.
+
+Praefect provides the following subcommands to re-enable writes or accept data loss:
+
+- In GitLab 13.2 and earlier, `enable-writes` to re-enable virtual storage for writes after
+  data recovery attempts:
+
+  ```shell
+  sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml enable-writes -virtual-storage <virtual-storage>
+  ```
+
+- In GitLab 13.3 and later, if it is not possible to bring one of the up to date nodes back
+  online, you may have to accept data loss:
+
+  ```shell
+  sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml accept-dataloss -virtual-storage <virtual-storage> -repository <relative-path> -authoritative-storage <storage-name>
+  ```
+
+  When accepting data loss, Praefect:
+
+  1. Marks the chosen copy of the repository as the latest version.
+  1. Replicates the copy to the other assigned Gitaly nodes.
+
+  This process overwrites any other copy of the repository so care must be taken.
+
+## Data recovery
+
+If a Gitaly node fails replication jobs for any reason, it ends up hosting outdated versions of the
+affected repositories. Praefect provides tools for:
+
+- [Automatic](#automatic-reconciliation) reconciliation, for GitLab 13.4 and later.
+- [Manual](#manual-reconciliation) reconciliation, for:
+  - GitLab 13.3 and earlier.
+  - Repositories upgraded to GitLab 13.4 and later without entries in the `repositories` table. In
+    GitLab 13.6 and later, [a migration is run](https://gitlab.com/gitlab-org/gitaly/-/issues/3033)
+    when Praefect starts for these repositories.
+
+These tools reconcile the outdated repositories to bring them fully up to date again.
+
+### Automatic reconciliation
+
+> [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/2717) in GitLab 13.4.
+
+Praefect automatically reconciles repositories that are not up to date. By default, this is done every
+five minutes. For each outdated repository on a healthy Gitaly node, the Praefect picks a
+random, fully up-to-date replica of the repository on another healthy Gitaly node to replicate from. A
+replication job is scheduled only if there are no other replication jobs pending for the target
+repository.
+
+The reconciliation frequency can be changed via the configuration. The value can be any valid
+[Go duration value](https://pkg.go.dev/time#ParseDuration). Values below 0 disable the feature.
+
+Examples:
+
+```ruby
+praefect['reconciliation_scheduling_interval'] = '5m' # the default value
+```
+
+```ruby
+praefect['reconciliation_scheduling_interval'] = '30s' # reconcile every 30 seconds
+```
+
+```ruby
+praefect['reconciliation_scheduling_interval'] = '0' # disable the feature
+```
+
+### Manual reconciliation
+
+WARNING:
+The `reconcile` sub-command was removed in GitLab 14.1. Use [automatic reconciliation](#automatic-reconciliation) instead.
+Manual reconciliation may produce excess replication jobs and is limited in functionality. Manual reconciliation does not
+work when [repository-specific primary nodes](praefect.md#repository-specific-primary-nodes) are enabled.
+
+The Praefect `reconcile` sub-command allows for the manual reconciliation between two Gitaly nodes. The
+command replicates every repository on a later version on the reference storage to the target storage.
+
+```shell
+sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml reconcile -virtual <virtual-storage> -reference <up-to-date-storage> -target <outdated-storage> -f
+```
+
+- Replace the placeholder `<virtual-storage>` with the virtual storage containing the Gitaly node storage to be checked.
+- Replace the placeholder `<up-to-date-storage>` with the Gitaly storage name containing up to date repositories.
+- Replace the placeholder `<outdated-storage>` with the Gitaly storage name containing outdated repositories.
+
+### Manually remove repositories
+
+> - [Introduced](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/3767) in GitLab 14.3.
+> - [Introduced](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/4054) in GitLab 14.6, support for dry-run mode.
+
+The `remove-repository` Praefect sub-command removes a repository from a Gitaly Cluster, and all state associated with a given repository including:
+
+- On-disk repositories on all relevant Gitaly nodes.
+- Any database state tracked by Praefect.
+
+In GitLab 14.6 and later, by default, the command operates in dry-run mode. In earlier versions, the command didn't support dry-run mode. For example:
+
+```shell
+sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml remove-repository -virtual-storage <virtual-storage> -repository <repository>
+```
+
+- Replace `<virtual-storage>` with the name of the virtual storage containing the repository.
+- Replace `<repository>` with the relative path of the repository to remove.
+- In GitLab 14.6 and later, add `-apply` to run the command outside of dry-run mode and remove the repository. For example:
+
+  ```shell
+  sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml remove-repository -virtual-storage <virtual-storage> -repository <repository> -apply
+  ```
+
+- `-virtual-storage` is the virtual storage the repository is located in. Virtual storages are configured in `/etc/gitlab/gitlab.rb` under `praefect['virtual_storages]` and looks like the following:
+
+  ```ruby
+  praefect['virtual_storages'] = {
+    'default' => {
+      ...
+    },
+    'storage-1' => {
+      ...
+    }
+  }
+  ```
+
+  In this example, the virtual storage to specify is `default` or `storage-1`.
+
+- `-repository` is the repository's relative path in the storage [beginning with `@hashed`](../repository_storage_types.md#hashed-storage).
+  For example:
+
+  ```plaintext
+  @hashed/f5/ca/f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b.git
+  ```
+
+Parts of the repository can continue to exist after running `remove-repository`. This can be because of:
+
+- A deletion error.
+- An in-flight RPC call targeting the repository.
+
+If this occurs, run `remove-repository` again.
+
+### Manually list untracked repositories
+
+> [Introduced](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/3926) in GitLab 14.4.
+
+The `list-untracked-repositories` Praefect sub-command lists repositories of the Gitaly Cluster that both:
+
+- Exist for at least one Gitaly storage.
+- Aren't tracked in the Praefect database.
+
+The command outputs:
+
+- Result to `STDOUT` and the command's logs.
+- Errors to `STDERR`.
+
+Each entry is a complete JSON string with a newline at the end (configurable using the
+`-delimiter` flag). For example:
+
+```plaintext
+sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml list-untracked-repositories
+{"virtual_storage":"default","storage":"gitaly-1","relative_path":"@hashed/ab/cd/abcd123456789012345678901234567890123456789012345678901234567890.git"}
+{"virtual_storage":"default","storage":"gitaly-1","relative_path":"@hashed/ab/cd/abcd123456789012345678901234567890123456789012345678901234567891.git"}
+```
+
+### Manually track repositories
+
+> - [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/5658) in GitLab 14.4.
+> - [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/5789) in GitLab 14.6, support for immediate replication.
+
+The `track-repository` Praefect sub-command adds repositories on disk to the Praefect database to be tracked.
+
+```shell
+sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml track-repository -virtual-storage <virtual-storage> -repository <repository> -replicate-immediately
+```
+
+- `-virtual-storage` is the virtual storage the repository is located in. Virtual storages are configured in `/etc/gitlab/gitlab.rb` under `praefect['virtual_storages]` and looks like the following:
+
+  ```ruby
+  praefect['virtual_storages'] = {
+    'default' => {
+      ...
+    },
+    'storage-1' => {
+      ...
+    }
+  }
+  ```
+
+  In this example, the virtual storage to specify is `default` or `storage-1`.
+
+- `-repository` is the repository's relative path in the storage [beginning with `@hashed`](../repository_storage_types.md#hashed-storage).
+  For example:
+
+  ```plaintext
+  @hashed/f5/ca/f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b.git
+  ```
+
+- `-authoritative-storage` is the storage we want Praefect to treat as the primary. Required if
+  [per-repository replication](praefect.md#configure-replication-factor) is set as the replication strategy.
+- `-replicate-immediately`, available in GitLab 14.6 and later, causes the command to replicate the repository to its secondaries immediately.
+   Otherwise, replication jobs are scheduled for execution in the database and are picked up by a Praefect background process.
+
+The command outputs:
+
+- Results to `STDOUT` and the command's logs.
+- Errors to `STDERR`.
+
+This command fails if:
+
+- The repository is already being tracked by the Praefect database.
+- The repository does not exist on disk.
diff --git a/doc/administration/gitaly/troubleshooting.md b/doc/administration/gitaly/troubleshooting.md
index d6d93b8af94..fdd281c1a90 100644
--- a/doc/administration/gitaly/troubleshooting.md
+++ b/doc/administration/gitaly/troubleshooting.md
@@ -153,7 +153,7 @@ Confirm the following are all true:
 
 - When any user adds or modifies a file from the repository using the GitLab
   UI, it immediately fails with a red `401 Unauthorized` banner.
-- Creating a new project and [initializing it with a README](../../user/project/working_with_projects.md#blank-projects)
+- Creating a new project and [initializing it with a README](../../user/project/working_with_projects.md#create-a-blank-project)
   successfully creates the project but doesn't create the README.
 - When [tailing the logs](https://docs.gitlab.com/omnibus/settings/logs.html#tail-logs-in-a-console-on-the-server)
   on a Gitaly client and reproducing the error, you get `401` errors
@@ -328,10 +328,94 @@ experiencing [clock drift](https://en.wikipedia.org/wiki/Clock_drift).
 Please ensure that the GitLab and Gitaly nodes are synchronized and use an NTP time
 server to keep them synchronized if possible.
 
+### Health check warnings
+
+The following warning in `/var/log/gitlab/praefect/current` can be ignored.
+
+```plaintext
+"error":"full method name not found: /grpc.health.v1.Health/Check",
+"msg":"error when looking up method info"
+```
+
+### File not found errors
+
+The following errors in `/var/log/gitlab/gitaly/current` can be ignored.
+They are caused by the GitLab Rails application checking for specific files
+that do not exist in a repository.
+
+```plaintext
+"error":"not found: .gitlab/route-map.yml"
+"error":"not found: Dockerfile"
+"error":"not found: .gitlab-ci.yml"
+```
+
 ## Troubleshoot Praefect (Gitaly Cluster)
 
 The following sections provide possible solutions to Gitaly Cluster errors.
 
+### Check cluster health
+
+> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/) in GitLab 14.6.
+
+The `check` Praefect sub-command runs a series of checks to determine the health of the Gitaly Cluster.
+
+```shell
+gitlab-ctl praefect check
+```
+
+The following sections describe the checks that are run.
+
+#### Praefect migrations
+
+Because Database migrations must be up to date for Praefect to work correctly, checks if Praefect migrations are up to date.
+
+If this check fails:
+
+1. See the `schema_migrations` table in the database to see which migrations have run.
+1. Run `praefect sql-migrate` to bring the migrations up to date.
+
+#### Node connectivity and disk access
+
+Checks if Praefect can reach all of its Gitaly nodes, and if each Gitaly node has read and write access to all of its storages.
+
+If this check fails:
+
+1. Confirm the network addresses and tokens are set up correctly:
+   - In the Praefect configuration.
+   - In each Gitaly node's configuration.
+1. On the Gitaly nodes, check that the `gitaly` process being run as `git`. There might be a permissions issue that is preventing Gitaly from
+   accessing its storage directories.
+1. Confirm that there are no issues with the network that connects Praefect to Gitaly nodes.
+
+#### Database read and write access
+
+Checks if Praefect can read from and write to the database.
+
+If this check fails:
+
+1. See if the Praefect database is in recovery mode. In recovery mode, tables may be read only. To check, run:
+
+   ```sql
+   select pg_is_in_recovery()
+   ```
+
+1. Confirm that the user that Praefect uses to connect to PostgreSQL has read and write access to the database.
+1. See if the database has been placed into read-only mode. To check, run:
+
+   ```sql
+   show default_transaction_read_only
+   ```
+
+#### Inaccessible repositories
+
+Checks how many repositories are inaccessible because they are missing a primary assignment, or their primary is unavailable.
+
+If this check fails:
+
+1. See if any Gitaly nodes are down. Run `praefect ping-nodes` to check.
+1. Check if there is a high load on the Praefect database. If the Praefect database is slow to respond, it can lead health checks failing to persist
+   to the database, leading Praefect to think nodes are unhealthy.
+
 ### Praefect errors in logs
 
 If you receive an error, check `/var/log/gitlab/gitlab-rails/production.log`.
@@ -353,17 +437,107 @@ Here are common errors and potential causes:
 
 ### Determine primary Gitaly node
 
-To determine the current primary Gitaly node for a specific Praefect node:
+To determine the primary node of a repository:
 
-- Use the `Shard Primary Election` [Grafana chart](praefect.md#grafana) on the
-  [`Gitlab Omnibus - Praefect` dashboard](https://gitlab.com/gitlab-org/grafana-dashboards/-/blob/master/omnibus/praefect.json).
-  This is recommended.
-- If you do not have Grafana set up, use the following command on each host of each
-  Praefect node:
+- In GitLab 14.6 and later, use the [`praefect metadata`](#view-repository-metadata) subcommand.
+- In GitLab 13.12 to GitLab 14.5 with [repository-specific primaries](praefect.md#repository-specific-primary-nodes),
+  use the [`gitlab:praefect:replicas` Rake task](../raketasks/praefect.md#replica-checksums).
+- With legacy election strategies in GitLab 13.12 and earlier, the primary was the same for all repositories in a virtual storage.
+  To determine the current primary Gitaly node for a specific virtual storage:
 
-  ```shell
-  curl localhost:9652/metrics | grep gitaly_praefect_primaries`
-  ```
+  - Use the `Shard Primary Election` [Grafana chart](praefect.md#grafana) on the
+    [`Gitlab Omnibus - Praefect` dashboard](https://gitlab.com/gitlab-org/grafana-dashboards/-/blob/master/omnibus/praefect.json).
+    This is recommended.
+  - If you do not have Grafana set up, use the following command on each host of each
+    Praefect node:
+
+    ```shell
+    curl localhost:9652/metrics | grep gitaly_praefect_primaries`
+    ```
+
+### View repository metadata
+
+> [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/3481) in GitLab 14.6.
+
+Gitaly Cluster maintains a [metadata database](index.md#components) about the repositories stored on the cluster. Use the `praefect metadata` subcommand
+to inspect the metadata for troubleshooting.
+
+You can retrieve a repository's metadata by its Praefect-assigned repository ID:
+
+```shell
+sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml metadata -repository-id <repository-id>
+```
+
+You can also retrieve a repository's metadata by its virtual storage and relative path:
+
+```shell
+sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml metadata -virtual-storage <virtual-storage> -relative-path <relative-path>
+```
+
+#### Examples
+
+To retrieve the metadata for a repository with a Praefect-assigned repository ID of 1:
+
+```shell
+sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml metadata -repository-id 1
+```
+
+To retrieve the metadata for a repository with virtual storage `default` and relative path `@hashed/b1/7e/b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9.git`:
+
+```shell
+sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml metadata -virtual-storage default -relative-path @hashed/b1/7e/b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9.git
+```
+
+Either of these examples retrieve the following metadata for an example repository:
+
+```plaintext
+Repository ID: 54771
+Virtual Storage: "default"
+Relative Path: "@hashed/b1/7e/b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9.git"
+Replica Path: "@hashed/b1/7e/b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9.git"
+Primary: "gitaly-1"
+Generation: 1
+Replicas:
+- Storage: "gitaly-1"
+  Assigned: true
+  Generation: 1, fully up to date
+  Healthy: true
+  Valid Primary: true
+- Storage: "gitaly-2"
+  Assigned: true
+  Generation: 0, behind by 1 changes
+  Healthy: true
+  Valid Primary: false
+- Storage: "gitaly-3"
+  Assigned: true
+  Generation: replica not yet created
+  Healthy: false
+  Valid Primary: false
+```
+
+#### Available metadata
+
+The metadata retrieved by `praefect metadata` includes the fields in the following tables.
+
+| Field             | Description                                                                                                        |
+|:------------------|:-------------------------------------------------------------------------------------------------------------------|
+| `Repository ID`   | Permanent unique ID assigned to the repository by Praefect. Different to the ID GitLab uses for repositories.      |
+| `Virtual Storage` | Name of the virtual storage the repository is stored in.                                                           |
+| `Relative Path`   | Repository's path in the virtual storage.                                                                          |
+| `Replica Path`    | Where on the Gitaly node's disk the repository's replicas are stored.                                                |
+| `Primary`         | Current primary of the repository.                                                                                 |
+| `Generation`      | Used by Praefect to track repository changes. Each write in the repository increments the repository's generation. |
+| `Replicas`        | A list of replicas that exist or are expected to exist.                                                            |
+
+For each replica, the following metadata is available:
+
+| `Replicas` Field | Description                                                                                                                                                                                                                                                                                                                                                                                                                                            |
+|:-----------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `Storage`        | Name of the Gitaly storage that contains the replica.                                                                                                                                                                                                                                                                                                                                                                                                  |
+| `Assigned`       | Indicates whether the replica is expected to exist in the storage. Can be `false` if a Gitaly node is removed from the cluster or if the storage contains an extra copy after the repository's replication factor was decreased.                                                                                                                                                                                                                       |
+| `Generation`     | Latest confirmed generation of the replica. It indicates:<br><br>- The replica is fully up to date if the generation matches the repository's generation.<br>- The replica is outdated if the replica's generation is less than the repository's generation.<br>- `replica not yet created` if the replica does not yet exist at all on the storage.                                                                                                          |
+| `Healthy`        | Indicates whether the Gitaly node that is hosting this replica is considered healthy by the consensus of Praefect nodes.                                                                                                                                                                                                                                                                                                                               |
+| `Valid Primary`  | Indicates whether the replica is fit to serve as the primary node. If the repository's primary is not a valid primary, a failover occurs on the next write to the repository if there is another replica that is a valid primary. A replica is a valid primary if:<br><br>- It is stored on a healthy Gitaly node.<br>- It is fully up to date.<br>- It is not targeted by a pending deletion job from decreasing replication factor.<br>- It is assigned. |
 
 ### Check that repositories are in sync
 
@@ -371,7 +545,7 @@ Is [some cases](index.md#known-issues) the Praefect database can get out of sync
 a given repository is fully synced on all nodes, run the [`gitlab:praefect:replicas` Rake task](../raketasks/praefect.md#replica-checksums)
 that checksums the repository on all Gitaly nodes.
 
-The [Praefect dataloss](praefect.md#check-for-data-loss) command only checks the state of the repo in the Praefect database, and cannot
+The [Praefect dataloss](recovery.md#check-for-data-loss) command only checks the state of the repo in the Praefect database, and cannot
 be relied to detect sync problems in this scenario.
 
 ### Relation does not exist errors
@@ -409,3 +583,21 @@ This indicates that the virtual storage name used in the
 [`git_data_dirs` setting](praefect.md#gitaly) for GitLab.
 
 Resolve this by matching the virtual storage names used in Praefect and GitLab configuration.
+
+### Gitaly Cluster performance issues on cloud platforms
+
+Praefect does not require a lot of CPU or memory, and can run on small virtual machines.
+Cloud services may place other limits on the resources that small VMs can use, such as
+disk IO and network traffic.
+
+Praefect nodes generate a lot of network traffic. The following symptoms can be observed if their network bandwidth has
+been throttled by the cloud service:
+
+- Poor performance of Git operations.
+- High network latency.
+- High memory use by Praefect.
+
+Possible solutions:
+
+- Provision larger VMs to gain access to larger network traffic allowances.
+- Use your cloud service's monitoring and logging to check that the Praefect nodes are not exhausting their traffic allowances.