Add latest changes from gitlab-org/gitlab@master

12 files changed, 420 insertions, 7 deletions

diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md
index 4630b6d8259..24ad2b502c2 100644
--- a/doc/administration/gitaly/index.md
+++ b/doc/administration/gitaly/index.md
@@ -1016,6 +1016,15 @@ When updating the `gitaly['listen_addr']` or `gitaly['prometheus_listen_addr']`
 
 When this occurs, performing a `sudo gitlab-ctl restart` will resolve the issue. This will no longer be necessary after [this issue](https://gitlab.com/gitlab-org/gitaly/issues/2521) is resolved.
 
+### Permission denied errors appearing in Gitaly logs when accessing repositories from a standalone Gitaly node
+
+If this error occurs even though file permissions are correct, it's likely that
+the Gitaly node is experiencing
+[clock drift](https://en.wikipedia.org/wiki/Clock_drift).
+
+Please ensure that the GitLab and Gitaly nodes are synchronized and use an NTP time
+server to keep them synchronized if possible.
+
 ### Praefect
 
 Praefect is an experimental daemon that allows for replication of the Git data.
diff --git a/doc/administration/gitaly/praefect.md b/doc/administration/gitaly/praefect.md
index 9fb61d93f73..c35a3aea976 100644
--- a/doc/administration/gitaly/praefect.md
+++ b/doc/administration/gitaly/praefect.md
@@ -573,6 +573,127 @@ Particular attention should be shown to:
 
 Congratulations! You have configured a highly available Praefect cluster.
 
+### Failover
+
+There are two ways to do a failover from one internal Gitaly node to another as the primary. Manually, or automatically.
+
+As an example, in this `config.toml` we have one virtual storage named "default" with two internal Gitaly nodes behind it.
+One is deemed the "primary". This means that read and write traffic will go to `internal_storage_0`, and writes
+will get replicated to `internal_storage_1`:
+
+```toml
+socket_path = "/path/to/Praefect.socket"
+
+# failover_enabled will enable automatic failover
+failover_enabled = false
+
+[logging]
+format = "json"
+level = "info"
+
+[[virtual_storage]]
+name = "default"
+
+[[virtual_storage.node]]
+  name = "internal_storage_0"
+  address = "tcp://localhost:9999"
+  primary = true
+  token = "supersecret"
+
+[[virtual_storage.node]]
+  name = "internal_storage_1"
+  address = "tcp://localhost:9998"
+  token = "supersecret"
+```
+
+#### Manual failover
+
+In order to failover from using one internal Gitaly node to using another, a manual failover step can be used. Unless `failover_enabled` is set to `true`
+in the `config.toml`, the only way to fail over from one primary to using another node as the primary is to do a manual failover.
+
+1. Move `primary = true` from the current `[[virtual_storage.node]]` to another node in `/etc/gitlab/gitlab.rb`:
+
+   ```ruby
+   praefect['virtual_storages'] = {
+     'praefect' => {
+       'gitaly-1' => {
+         'address' => 'tcp://GITALY_HOST:8075',
+         'token'   => 'PRAEFECT_INTERNAL_TOKEN',
+         # no longer the primary
+       },
+       'gitaly-2' => {
+         'address' => 'tcp://GITALY_HOST:8075',
+         'token'   => 'PRAEFECT_INTERNAL_TOKEN',
+         # this is the new primary
+         'primary' => true
+       },
+       'gitaly-3' => {
+         'address' => 'tcp://GITALY_HOST:8075',
+         'token'   => 'PRAEFECT_INTERNAL_TOKEN',
+       }
+     }
+   }
+   ```
+
+1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
+
+On a restart, Praefect will send write traffic to `internal_storage_1`. `internal_storage_0` is the new secondary now,
+and replication jobs will be created to replicate repository data to `internal_storage_0` **from** `internal_storage_1`
+
+#### Automatic failover
+
+When automatic failover is enabled, Praefect will do automatic detection of the health of internal Gitaly nodes. If the
+primary has a certain amount of healthchecks fail, it will decide to promote one of the secondaries to be primary, and
+demote the primary to be a secondary.
+
+1. To enable automatic failover, edit `/etc/gitlab/gitlab.rb`:
+
+   ```ruby
+   # failover_enabled turns on automatic failover
+   praefect['failover_enabled'] = true
+   praefect['virtual_storages'] = {
+     'praefect' => {
+       'gitaly-1' => {
+         'address' => 'tcp://GITALY_HOST:8075',
+         'token'   => 'PRAEFECT_INTERNAL_TOKEN',
+         'primary' => true
+       },
+       'gitaly-2' => {
+         'address' => 'tcp://GITALY_HOST:8075',
+         'token'   => 'PRAEFECT_INTERNAL_TOKEN'
+       },
+       'gitaly-3' => {
+         'address' => 'tcp://GITALY_HOST:8075',
+         'token'   => 'PRAEFECT_INTERNAL_TOKEN'
+       }
+     }
+   }
+   ```
+
+1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
+
+Below is the picture when Praefect starts up with the config.toml above:
+
+```mermaid
+graph TD
+  A[Praefect] -->|Mutator RPC| B(internal_storage_0)
+  B --> |Replication|C[internal_storage_1]
+```
+
+Let's say suddenly `internal_storage_0` goes down. Praefect will detect this and
+automatically switch over to `internal_storage_1`, and `internal_storage_0` will serve as a secondary:
+
+```mermaid
+graph TD
+  A[Praefect] -->|Mutator RPC| B(internal_storage_1)
+  B --> |Replication|C[internal_storage_0]
+```
+
+NOTE: **Note:**: Currently this feature is supported for setups that only have 1 Praefect instance. Praefect instances running,
+for example behind a load balancer, `failover_enabled` should be disabled. The reason is The reason is because there
+is no coordination that currently happens across different Praefect instances, so there could be a situation where
+two Praefect instances think two different Gitaly nodes are the primary.
+
 ## Migrating existing repositories to Praefect
 
 If your GitLab instance already has repositories, these won't be migrated
diff --git a/doc/administration/high_availability/README.md b/doc/administration/high_availability/README.md
index c74beb11241..88f1079dec5 100644
--- a/doc/administration/high_availability/README.md
+++ b/doc/administration/high_availability/README.md
@@ -110,7 +110,7 @@ them.
 | [PgBouncer](../../development/architecture.md#pgbouncer)                                                            | Database Pool Manager                                               | [PgBouncer HA configuration](pgbouncer.md) **(PREMIUM ONLY)** |
 | [Redis](../../development/architecture.md#redis)[^3] with Redis Sentinel                                            | Key/Value store for shared data with HA watcher service             | [Redis HA configuration](redis.md)                            |
 | [Gitaly](../../development/architecture.md#gitaly)[^2] [^5] [^7]                                                    | Recommended high-level storage for Git repository data              | [Gitaly HA configuration](gitaly.md)                          |
-| [Sidekiq](../../development/architecture.md#sidekiq)                                                                | Asynchronous/Background jobs                                        |                                                               |
+| [Sidekiq](../../development/architecture.md#sidekiq)                                                                | Asynchronous/Background jobs                                        | [Sidekiq configuration](sidekiq.md)                           |
 | [GitLab application nodes](../../development/architecture.md#unicorn)[^1]                                           | (Unicorn / Puma, Workhorse) - Web-requests (UI, API, Git over HTTP) | [GitLab app HA/scaling configuration](gitlab.md)              |
 | [Prometheus](../../development/architecture.md#prometheus) and [Grafana](../../development/architecture.md#grafana) | GitLab environment monitoring                                       | [Monitoring node for scaling/HA](monitoring_node.md)          |
 
diff --git a/doc/administration/high_availability/sidekiq.md b/doc/administration/high_availability/sidekiq.md
new file mode 100644
index 00000000000..3d120ea8f09
--- /dev/null
+++ b/doc/administration/high_availability/sidekiq.md
@@ -0,0 +1,186 @@
+---
+type: reference
+---
+
+# Configuring Sidekiq
+
+This section discusses how to configure an external Sidekiq instance.
+
+Sidekiq requires connection to the Redis, PostgreSQL and Gitaly instance.
+To configure the Sidekiq node:
+
+1. SSH into the Sidekiq server.
+
+1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab package
+you want using steps 1 and 2 from the GitLab downloads page.
+**Do not complete any other steps on the download page.**
+
+1. Open `/etc/gitlab/gitlab.rb` with your editor.
+
+1. Generate the Sidekiq configuration:
+
+   ```ruby
+   sidekiq['listen_address'] = "10.10.1.48"
+
+   ## Optional: Enable extra Sidekiq processes
+   sidekiq_cluster['enable'] = true
+   sidekiq_cluster['enable'] = true
+     "elastic_indexer"
+   ]
+   ```
+
+1. Setup Sidekiq's connection to Redis:
+
+   ```ruby
+   ## Must be the same in every sentinel node
+   redis['master_name'] = 'gitlab-redis'
+
+   ## The same password for Redis authentication you set up for the master node.
+   redis['master_password'] = 'YOUR_PASSOWORD'
+
+   ## A list of sentinels with `host` and `port`
+   gitlab_rails['redis_sentinels'] = [
+       {'host' => '10.10.1.34', 'port' => 26379},
+       {'host' => '10.10.1.35', 'port' => 26379},
+       {'host' => '10.10.1.36', 'port' => 26379},
+     ]
+   ```
+
+1. Setup Sidekiq's connection to Gitaly:
+
+   ```ruby
+   git_data_dirs({
+     'default' => { 'gitaly_address' => 'tcp://gitaly:8075' },
+   })
+   gitlab_rails['gitaly_token'] = 'YOUR_TOKEN'
+   ```
+
+1. Setup Sidekiq's connection to Postgres:
+
+   ```ruby
+   gitlab_rails['db_host'] = '10.10.1.30'
+   gitlab_rails['db_password'] = 'YOUR_PASSOWORD'
+   gitlab_rails['db_port'] = '5432'
+   gitlab_rails['db_adapter'] = 'postgresql'
+   gitlab_rails['db_encoding'] = 'unicode'
+   gitlab_rails['auto_migrate'] = false
+   ```
+
+   Remember to add the Sidekiq nodes to the Postgres whitelist:
+
+   ```ruby
+   postgresql['trust_auth_cidr_addresses'] = %w(127.0.0.1/32 10.10.1.30/32 10.10.1.31/32 10.10.1.32/32 10.10.1.33/32 10.10.1.38/32)
+   ```
+
+1. Disable other services:
+
+   ```ruby
+   nginx['enable'] = false
+   grafana['enable'] = false
+   prometheus['enable'] = false
+   gitlab_rails['auto_migrate'] = false
+   alertmanager['enable'] = false
+   gitaly['enable'] = false
+   gitlab_monitor['enable'] = false
+   gitlab_workhorse['enable'] = false
+   nginx['enable'] = false
+   postgres_exporter['enable'] = false
+   postgresql['enable'] = false
+   redis['enable'] = false
+   redis_exporter['enable'] = false
+   unicorn['enable'] = false
+   gitlab_exporter['enable'] = false
+   ```
+
+1. Run `gitlab-ctl reconfigure`.
+
+## Example configuration
+
+Here's what the ending `/etc/gitlab/gitlab.rb` would look like:
+
+```ruby
+########################################
+#####        Services Disabled       ###
+########################################
+
+nginx['enable'] = false
+grafana['enable'] = false
+prometheus['enable'] = false
+gitlab_rails['auto_migrate'] = false
+alertmanager['enable'] = false
+gitaly['enable'] = false
+gitlab_monitor['enable'] = false
+gitlab_workhorse['enable'] = false
+nginx['enable'] = false
+postgres_exporter['enable'] = false
+postgresql['enable'] = false
+redis['enable'] = false
+redis_exporter['enable'] = false
+unicorn['enable'] = false
+gitlab_exporter['enable'] = false
+
+########################################
+####              Redis              ###
+########################################
+
+## Must be the same in every sentinel node
+redis['master_name'] = 'gitlab-redis'
+
+## The same password for Redis authentication you set up for the master node.
+redis['master_password'] = 'YOUR_PASSOWORD'
+
+## A list of sentinels with `host` and `port`
+gitlab_rails['redis_sentinels'] = [
+    {'host' => '10.10.1.34', 'port' => 26379},
+    {'host' => '10.10.1.35', 'port' => 26379},
+    {'host' => '10.10.1.36', 'port' => 26379},
+  ]
+
+#######################################
+###              Gitaly             ###
+#######################################
+
+git_data_dirs({
+  'default' => { 'gitaly_address' => 'tcp://gitaly:8075' },
+})
+gitlab_rails['gitaly_token'] = 'YOUR_TOKEN'
+
+#######################################
+###            Postgres             ###
+#######################################
+gitlab_rails['db_host'] = '10.10.1.30'
+gitlab_rails['db_password'] = 'YOUR_PASSOWORD'
+gitlab_rails['db_port'] = '5432'
+gitlab_rails['db_adapter'] = 'postgresql'
+gitlab_rails['db_encoding'] = 'unicode'
+gitlab_rails['auto_migrate'] = false
+
+#######################################
+###      Sidekiq configuration      ###
+#######################################
+sidekiq['listen_address'] = "10.10.1.48"
+
+#######################################
+###     Monitoring configuration    ###
+#######################################
+consul['enable'] = true
+consul['monitoring_service_discovery'] =  true
+
+consul['configuration'] = {
+  bind_addr: '10.10.1.48',
+  retry_join: %w(10.10.1.34 10.10.1.35 10.10.1.36)
+}
+
+# Set the network addresses that the exporters will listen on
+node_exporter['listen_address'] = '10.10.1.48:9100'
+
+# Rails Status for prometheus
+gitlab_rails['monitoring_whitelist'] = ['10.10.1.42', '127.0.0.1']
+```
+
+## Further reading
+
+Related Sidekiq configuration:
+
+1. [Extra Sidekiq processes](../operations/extra_sidekiq_processes.md)
+1. [Using the GitLab-Sidekiq chart](https://docs.gitlab.com/charts/charts/gitlab/sidekiq/)
diff --git a/doc/administration/packages/container_registry.md b/doc/administration/packages/container_registry.md
index 2b029859447..51c03f2edd0 100644
--- a/doc/administration/packages/container_registry.md
+++ b/doc/administration/packages/container_registry.md
@@ -353,10 +353,6 @@ configuring a different storage driver. By default the GitLab Container Registry
 is configured to use the filesystem driver, which makes use of [storage path](#container-registry-storage-path)
 configuration.
 
-NOTE: **Note:** Enabling a storage driver other than `filesystem` would mean
-that your Docker client needs to be able to access the storage backend directly.
-In that case, you must use an address that resolves and is accessible outside GitLab server. The Docker client will continue to authenticate via GitLab but data transfer will be direct to and from the storage backend.
-
 The different supported drivers are:
 
 | Driver     | Description                         |
@@ -425,6 +421,55 @@ storage:
 NOTE: **Note:**
 `your-s3-bucket` should only be the name of a bucket that exists, and can't include subdirectories.
 
+### Disable redirect for storage driver
+
+By default, users accessing a registry configured with a remote backend are redirected to the default backend for the storage driver. For example, registries can be configured using the `s3` storage driver, which redirects requests to a remote S3 bucket to alleviate load on the GitLab server.
+
+However, this behaviour is undesirable for registries used by internal hosts that usually can't access public servers. To disable redirects, set the `disable` flag to true as follows. This makes all traffic to always go through the Registry service. This results in improved security (less surface attack as the storage backend is not publicly accessible), but worse performance (all traffic is redirected via the service).
+
+**Omnibus GitLab installations**
+
+1. Edit `/etc/gitlab/gitlab.rb`:
+
+    ```ruby
+    registry['storage'] = {
+      's3' => {
+        'accesskey' => 's3-access-key',
+        'secretkey' => 's3-secret-key-for-access-key',
+        'bucket' => 'your-s3-bucket',
+        'region' => 'your-s3-region',
+        'regionendpoint' => 'your-s3-regionendpoint'
+      },
+      'redirect' => {
+        'disable' => true
+      }
+    }
+    ```
+
+1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
+
+**Installations from source**
+
+1. Add the `redirect` flag to your registry configuration YML file:
+
+    ```yml
+    storage:
+      s3:
+        accesskey: 'AKIAKIAKI'
+        secretkey: 'secret123'
+        bucket: 'gitlab-registry-bucket-AKIAKIAKI'
+        region: 'your-s3-region'
+        regionendpoint: 'your-s3-regionendpoint'
+      redirect:
+        disable: true
+      cache:
+        blobdescriptor: inmemory
+      delete:
+        enabled: true
+    ```
+
+1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect.
+
 ### Storage limitations
 
 Currently, there is no storage limitation, which means a user can upload an
diff --git a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md
index e90f6f36cc6..657f29cc789 100644
--- a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md
+++ b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md
@@ -840,7 +840,7 @@ GitLab Rails console:
 
 ```ruby
 projects_and_size = []
-# a list of projects you want to look at, can get these however
+# You need to specify the projects that you want to look through. You can get these in any manner.
 projects = Project.last(100)
 
 projects.each do |p|
diff --git a/doc/api/graphql/reference/gitlab_schema.graphql b/doc/api/graphql/reference/gitlab_schema.graphql
index 4c5bce005d5..65934e53260 100644
--- a/doc/api/graphql/reference/gitlab_schema.graphql
+++ b/doc/api/graphql/reference/gitlab_schema.graphql
@@ -7356,6 +7356,11 @@ type Snippet implements Noteable {
   fileName: String
 
   """
+  HTTP URL to the snippet repository
+  """
+  httpUrlToRepo: String
+
+  """
   Id of the snippet
   """
   id: ID!
@@ -7396,6 +7401,11 @@ type Snippet implements Noteable {
   rawUrl: String!
 
   """
+  SSH URL to the snippet repository
+  """
+  sshUrlToRepo: String
+
+  """
   Title of the snippet
   """
   title: String!
diff --git a/doc/api/graphql/reference/gitlab_schema.json b/doc/api/graphql/reference/gitlab_schema.json
index 94c0c17d218..6da44c36cae 100644
--- a/doc/api/graphql/reference/gitlab_schema.json
+++ b/doc/api/graphql/reference/gitlab_schema.json
@@ -22214,6 +22214,20 @@
               "deprecationReason": null
             },
             {
+              "name": "httpUrlToRepo",
+              "description": "HTTP URL to the snippet repository",
+              "args": [
+
+              ],
+              "type": {
+                "kind": "SCALAR",
+                "name": "String",
+                "ofType": null
+              },
+              "isDeprecated": false,
+              "deprecationReason": null
+            },
+            {
               "name": "id",
               "description": "Id of the snippet",
               "args": [
@@ -22321,6 +22335,20 @@
               "deprecationReason": null
             },
             {
+              "name": "sshUrlToRepo",
+              "description": "SSH URL to the snippet repository",
+              "args": [
+
+              ],
+              "type": {
+                "kind": "SCALAR",
+                "name": "String",
+                "ofType": null
+              },
+              "isDeprecated": false,
+              "deprecationReason": null
+            },
+            {
               "name": "title",
               "description": "Title of the snippet",
               "args": [
diff --git a/doc/api/graphql/reference/index.md b/doc/api/graphql/reference/index.md
index a5eeab127d0..2acb4822e95 100644
--- a/doc/api/graphql/reference/index.md
+++ b/doc/api/graphql/reference/index.md
@@ -1161,9 +1161,11 @@ Represents a snippet entry
 | `description` | String | Description of the snippet |
 | `descriptionHtml` | String | The GitLab Flavored Markdown rendering of `description` |
 | `fileName` | String | File Name of the snippet |
+| `httpUrlToRepo` | String | HTTP URL to the snippet repository |
 | `id` | ID! | Id of the snippet |
 | `project` | Project | The project the snippet is associated with |
 | `rawUrl` | String! | Raw URL of the snippet |
+| `sshUrlToRepo` | String | SSH URL to the snippet repository |
 | `title` | String! | Title of the snippet |
 | `updatedAt` | Time! | Timestamp this snippet was updated |
 | `userPermissions` | SnippetPermissions! | Permissions for the current user on the resource |
diff --git a/doc/security/README.md b/doc/security/README.md
index 20da1a2c77c..c21d99658b8 100644
--- a/doc/security/README.md
+++ b/doc/security/README.md
@@ -19,6 +19,7 @@ type: index
 - [Send email confirmation on sign-up](user_email_confirmation.md)
 - [Security of running jobs](https://docs.gitlab.com/runner/security/)
 - [Proxying images](asset_proxy.md)
+- [CI/CD environment variables](cicd_environment_variables.md)
 
 ## Securing your GitLab installation
 
diff --git a/doc/security/cicd_environment_variables.md b/doc/security/cicd_environment_variables.md
new file mode 100644
index 00000000000..ea597ea05f2
--- /dev/null
+++ b/doc/security/cicd_environment_variables.md
@@ -0,0 +1,11 @@
+---
+type: reference
+---
+
+# CI/CD Environment Variables
+
+Environment variables are applied to environments via the runner and can be set from the project's **Settings > CI/CD** page.
+
+The values are encrypted using [aes-256-cbc](https://en.wikipedia.org/wiki/Advanced_Encryption_Standard) and stored in the database.
+
+This data can only be decrypted with a valid [secrets file](../raketasks/backup_restore.md#when-the-secrets-file-is-lost).
diff --git a/doc/user/admin_area/activating_deactivating_users.md b/doc/user/admin_area/activating_deactivating_users.md
index 9c153497e74..7c8ab18ccef 100644
--- a/doc/user/admin_area/activating_deactivating_users.md
+++ b/doc/user/admin_area/activating_deactivating_users.md
@@ -36,7 +36,7 @@ A user can be deactivated from the Admin Area. To do this:
 Please note that for the deactivation option to be visible to an admin, the user:
 
 - Must be currently active.
-- Should not have any activity in the last 180 days.
+- Must not have any signins or activity in the last 180 days.
 
 Users can also be deactivated using the [GitLab API](../../api/users.md#deactivate-user).