Add latest changes from gitlab-org/gitlab@14-3-stable-eev14.3.0-rc42

4 files changed, 106 insertions, 26 deletions

diff --git a/doc/administration/gitaly/configure_gitaly.md b/doc/administration/gitaly/configure_gitaly.md
index 5e8cbac42c1..d0841f4e607 100644
--- a/doc/administration/gitaly/configure_gitaly.md
+++ b/doc/administration/gitaly/configure_gitaly.md
@@ -41,9 +41,8 @@ However, Gitaly can be deployed to its own server, which can benefit GitLab inst
 multiple machines.
 
 NOTE:
-When configured to run on their own servers, Gitaly servers
-[must be upgraded](https://docs.gitlab.com/omnibus/update/#upgrading-gitaly-servers) before Gitaly
-clients in your cluster.
+When configured to run on their own servers, Gitaly servers must be
+[upgraded](../../update/package/index.md) before Gitaly clients in your cluster.
 
 The process for setting up Gitaly on its own server is:
 
@@ -340,6 +339,12 @@ disable enforcement. For more information, see the documentation on configuring
 1. Run `sudo -u git /home/git/gitaly/gitaly-hooks check /home/git/gitaly/config.toml`
    to confirm that Gitaly can perform callbacks to the GitLab internal API.
 
+WARNING:
+If directly copying repository data from a GitLab server to Gitaly, ensure that the metadata file,
+default path `/var/opt/gitlab/git-data/repositories/.gitaly-metadata`, is not included in the transfer.
+Copying this file causes GitLab to use the [Rugged patches](index.md#direct-access-to-git-in-gitlab) for repositories hosted on the Gitaly server,
+leading to `Error creating pipeline` and `Commit not found` errors, or stale data.
+
 ### Configure Gitaly clients
 
 As the final step, you must update Gitaly clients to switch from using local Gitaly service to use
diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md
index bca83e903ac..7a7aac884ed 100644
--- a/doc/administration/gitaly/index.md
+++ b/doc/administration/gitaly/index.md
@@ -87,11 +87,8 @@ Gitaly comes pre-configured with Omnibus GitLab, which is a configuration
 - Omnibus GitLab installations for up to 2000 users, see [specific Gitaly configuration instructions](../reference_architectures/2k_users.md#configure-gitaly).
 - Source installations or custom Gitaly installations, see [Configure Gitaly](configure_gitaly.md).
 
-GitLab installations for more than 2000 users should use Gitaly Cluster.
-
-NOTE:
-If not set in GitLab, feature flags are read as false from the console and Gitaly uses their
-default value. The default value depends on the GitLab version.
+GitLab installations for more than 2000 active users performing daily Git write operation may be
+best suited by using Gitaly Cluster.
 
 ## Gitaly Cluster
 
@@ -137,7 +134,7 @@ In this example:
 - The [replication factor](#replication-factor) is `3`. There are three copies maintained
   of each repository.
 
-The availability objectives for Gitaly clusters are:
+The availability objectives for Gitaly clusters assuming a single node failure are:
 
 - **Recovery Point Objective (RPO):** Less than 1 minute.
 
@@ -155,6 +152,58 @@ The availability objectives for Gitaly clusters are:
   Faster outage detection, to improve this speed to less than 1 second,
   is tracked [in this issue](https://gitlab.com/gitlab-org/gitaly/-/issues/2608).
 
+WARNING:
+If complete cluster failure occurs, disaster recovery plans should be executed. These can affect the
+RPO and RTO discussed above.
+
+### Architecture and configuration recommendations
+
+The following table provides recommendations based on your
+requirements. Users means concurrent users actively performing
+simultaneous Git Operations.
+
+Gitaly services are present in every GitLab installation and always coordinates Git repository storage and
+retrieval. Gitaly can be as simple as a single background service when operating on a single instance Omnibus
+GitLab (All of GitLab on one machine). Gitaly can be separated into it's own instance and it can be configured in
+a full cluster configuration depending on scaling and availability requirements.
+
+The GitLab Reference Architectures provide guidance for what Gitaly configuration is advisable at each of the scales.
+The Gitaly configuration is noted by the architecture diagrams and the table of required resources.
+
+| User Scaling                                                 | Reference Architecture To Use                                | GitLab Instance Configuration                | Gitaly Configuration            | Git Repository Storage             | Instances Dedicated to Gitaly Services |
+| ------------------------------------------------------------ | ------------------------------------------------------------ | -------------------------------------------- | ------------------------------- | ---------------------------------- | -------------------------------------- |
+| Up to 1000 Users                                             | [1K](../reference_architectures/1k_users.md)                 | Single Instance for all of GitLab            | Already Integrated as a Service | Local Disk                         | 0                                      |
+| Up to 2999 Users                                             | [2K](../reference_architectures/2k_users.md)                 | Horizontally Scaled GitLab Instance (Non-HA) | Single Gitaly Server            | Local Disk of Gitaly Instance      | 1                                      |
+| 3000 Users and Over                                          | [3K](../reference_architectures/1k_users.md)                 | Horizontally Scaled GitLab Instance with HA  | Gitaly Cluster                  | Local Disk of Each Gitaly Instance | 8                                      |
+| RTO/RPO Requirements for AZ Failure Tolerance Regardless of User Scale | [3K (with downscaling)](../reference_architectures/3k_users.md) | Custom (1)                      | Gitaly Cluster                  | Local Disk of Each Gitaly Instance | 8                                      |
+
+1. If you feel that you need AZ Failure Tolerance for user scaling lower than 3K, please contact Customer Success
+   to discuss your RTO and RPO needs and what options exist to meet those objectives.
+
+WARNING:
+At present, some [known database inconsistency issues](#known-issues-impacting-gitaly-cluster)
+exist in Gitaly Cluster. It is our recommendation that for now, you remain on your current service.
+We will adjust the date for NFS support removal if this applies to you.
+
+### Known issues impacting Gitaly Cluster
+
+The following table outlines current known issues impacting the use of Gitaly Cluster. For
+the most up to date status of these issues, please refer to the referenced issues / epics.
+
+| Issue                                                   | Summary                          |
+| Gitaly Cluster + Geo can cause database inconsistencies | There are some conditions during Geo replication that can cause database inconsistencies with Gitaly Cluster. These have been identified and are being resolved by updating Gitaly Cluster to [identify repositories with a unique and persistent identifier](https://gitlab.com/gitlab-org/gitaly/-/issues/3485). |
+| Database inconsistencies due to repository access outside of Gitaly Cluster's control | Operations that write to the repository storage which do not go through normal Gitaly Cluster methods can cause database inconsistencies. These can include (but are not limited to) snapshot restoration for cluster node disks, node upgrades which modify files under Git control, or any other disk operation that may touch repository storage external to GitLab. The Gitaly team is actively working to provide manual commands to [reconcile the Praefect database with the repository storage](https://gitlab.com/groups/gitlab-org/-/epics/6723). |
+
+### Snapshot backup and recovery limitations
+
+Gitaly Cluster does not support snapshot backups because these can cause issues where the
+Praefect database becomes out of sync with the disk storage. Because of how Praefect rebuilds
+the replication metadata of Gitaly disk information during a restore, we recommend using the
+[official backup and restore Rake tasks](../../raketasks/backup_restore.md).
+
+To track progress on work on a solution for manually re-synchronizing the Praefect database
+with disk storage, see [this epic](https://gitlab.com/groups/gitlab-org/-/epics/6575).
+
 ### Virtual storage
 
 Virtual storage makes it viable to have a single repository storage in GitLab to simplify repository
@@ -306,7 +355,10 @@ For configuration information, see [Configure replication factor](praefect.md#co
 
 For more information on configuring Gitaly Cluster, see [Configure Gitaly Cluster](praefect.md).
 
-### Migrate to Gitaly Cluster
+## Migrate to Gitaly Cluster
+
+We recommend you migrate to Gitaly Cluster if your
+[requirements recommend](#architecture-and-configuration-recommendations) Gitaly Cluster.
 
 Whether migrating to Gitaly Cluster because of [NFS support deprecation](index.md#nfs-deprecation-notice)
 or to move from single Gitaly nodes, the basic process involves:
@@ -318,6 +370,11 @@ or to move from single Gitaly nodes, the basic process involves:
    Gitaly Cluster, existing repositories stored outside Gitaly Cluster must be moved. There is no
    automatic migration but the moves can be scheduled with the GitLab API.
 
+WARNING:
+At present, some [known database inconsistency issues](#known-issues-impacting-gitaly-cluster)
+exist in Gitaly Cluster. It is our recommendation that for now, you remain on your current service.
+We will adjust the date for NFS support removal if this applies to you.
+
 ## Monitor Gitaly and Gitaly Cluster
 
 You can use the available logs and [Prometheus metrics](../monitoring/prometheus/index.md) to
@@ -449,6 +506,8 @@ To monitor [strong consistency](#strong-consistency), you can use the following
 - `gitaly_hook_transaction_voting_delay_seconds`, the client-side delay introduced by waiting for
   the transaction to be committed.
 
+You can also monitor the [Praefect logs](../logs.md#praefect-logs).
+
 ## Do not bypass Gitaly
 
 GitLab doesn't advise directly accessing Gitaly repositories stored on disk with a Git client,
@@ -539,12 +598,6 @@ To see if GitLab can access the repository file system directly, we use the foll
 Direct Git access is enable by default in Omnibus GitLab because it fills in the correct repository
 paths in the GitLab configuration file `config/gitlab.yml`. This satisfies the UUID check.
 
-WARNING:
-If directly copying repository data from a GitLab server to Gitaly, ensure that the metadata file,
-default path `/var/opt/gitlab/git-data/repositories/.gitaly-metadata`, is not included in the transfer.
-Copying this file causes GitLab to use the Rugged patches for repositories hosted on the Gitaly server,
-leading to `Error creating pipeline` and `Commit not found` errors, or stale data.
-
 ### Transition to Gitaly Cluster
 
 For the sake of removing complexity, we must remove direct Git access in GitLab. However, we can't
diff --git a/doc/administration/gitaly/praefect.md b/doc/administration/gitaly/praefect.md
index 4af7f1a58a5..eb666f1caf4 100644
--- a/doc/administration/gitaly/praefect.md
+++ b/doc/administration/gitaly/praefect.md
@@ -20,10 +20,6 @@ Configure Gitaly Cluster using either:
 
 Smaller GitLab installations may need only [Gitaly itself](index.md).
 
-NOTE:
-Upgrade instructions for Omnibus GitLab installations
-[are available](https://docs.gitlab.com/omnibus/update/#gitaly-cluster).
-
 ## Requirements
 
 The minimum recommended configuration for a Gitaly Cluster requires:
@@ -279,7 +275,7 @@ you need to prepare PostgreSQL server with [setup instruction](#manual-database-
 
 ```ruby
 pgbouncer['databases'] = {
-  # Other database configuation including gitlabhq_production
+  # Other database configuration including gitlabhq_production
   ...
 
   praefect_production: {
@@ -353,6 +349,7 @@ If there are multiple Praefect nodes:
 
 To complete this section you need a [configured PostgreSQL server](#postgresql), including:
 
+WARNING:
 Praefect should be run on a dedicated node. Do not run Praefect on the
 application server, or a Gitaly node.
 
@@ -994,7 +991,7 @@ Particular attention should be shown to:
 
 1. Check that the Praefect storage is configured to store new repositories:
 
-   1. On the top bar, select **Menu >** **{admin}** **Admin**.
+   1. On the top bar, select **Menu > Admin**.
    1. On the left sidebar, select **Settings > Repository**.
    1. Expand the **Repository storage** section.
 
@@ -1574,3 +1571,28 @@ sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.t
 - Replace the placeholder `<virtual-storage>` with the virtual storage containing the Gitaly node storage to be checked.
 - Replace the placeholder `<up-to-date-storage>` with the Gitaly storage name containing up to date repositories.
 - Replace the placeholder `<outdated-storage>` with the Gitaly storage name containing outdated repositories.
+
+### Manually remove repositories
+
+> [Introduced](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/3767) in GitLab 14.3.
+
+The `remove-repository` Praefect sub-command removes repositories from a Gitaly Cluster. It removes
+all state associated with a given repository including:
+
+- On-disk repositories on all relevant Gitaly nodes.
+- Any database state tracked by Praefect.
+
+```shell
+sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml remove-repository -virtual-storage <virtual-storage> -repository <repository>
+```
+
+- `-virtual-storage` is the virtual storage the repository is located in.
+- `-repository` is the repository's relative path in the storage.
+
+Sometimes parts of the repository continue to exist after running `remove-repository`. This can be caused
+because of:
+
+- A deletion error.
+- An in-flight RPC call targeting the repository.
+
+If this occurs, run `remove-repository` again.
diff --git a/doc/administration/gitaly/troubleshooting.md b/doc/administration/gitaly/troubleshooting.md
index 3dd700968f9..1b53a0994f5 100644
--- a/doc/administration/gitaly/troubleshooting.md
+++ b/doc/administration/gitaly/troubleshooting.md
@@ -23,7 +23,7 @@ See also [Gitaly timeout](../../user/admin_area/settings/gitaly_timeouts.md) set
 When using standalone Gitaly servers, you must make sure they are the same version
 as GitLab to ensure full compatibility:
 
-1. On the top bar, select **Menu >** **{admin}** **Admin** on your GitLab instance.
+1. On the top bar, select **Menu > Admin** on your GitLab instance.
 1. On the left sidebar, select **Overview > Gitaly Servers**.
 1. Confirm all Gitaly servers indicate that they are up to date.
 
@@ -226,8 +226,8 @@ update the secrets file on the Gitaly server to match the Gitaly client, then
 ### Repository pushes fail with a `deny updating a hidden ref` error
 
 Due to [a change](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/3426)
-introduced in GitLab 13.12, Gitaly has read-only, internal GitLab references that users are not 
-permitted to update. If you attempt to update internal references with `git push --mirror`, Git 
+introduced in GitLab 13.12, Gitaly has read-only, internal GitLab references that users are not
+permitted to update. If you attempt to update internal references with `git push --mirror`, Git
 returns the rejection error, `deny updating a hidden ref`.
 
 The following references are read-only:
@@ -243,7 +243,7 @@ To mirror-push branches and tags only, and avoid attempting to mirror-push prote
 git push origin +refs/heads/*:refs/heads/* +refs/tags/*:refs/tags/*
 ```
 
-Any other namespaces that the admin wants to push can be included there as well via additional patterns. 
+Any other namespaces that the admin wants to push can be included there as well via additional patterns.
 
 ### Command line tools cannot connect to Gitaly