Add latest changes from gitlab-org/gitlab@13-2-stable-ee

3 files changed, 98 insertions, 79 deletions

diff --git a/doc/administration/monitoring/prometheus/gitlab_exporter.md b/doc/administration/monitoring/prometheus/gitlab_exporter.md
index 3effca4a2bf..686ed14ba42 100644
--- a/doc/administration/monitoring/prometheus/gitlab_exporter.md
+++ b/doc/administration/monitoring/prometheus/gitlab_exporter.md
@@ -9,27 +9,25 @@ info: To determine the technical writer assigned to the Stage/Group associated w
 >- Available since [Omnibus GitLab 8.17](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/1132).
 >- Renamed from `GitLab monitor exporter` to `GitLab exporter` in [GitLab 12.3](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/16511).
 
-The [GitLab exporter](https://gitlab.com/gitlab-org/gitlab-exporter) allows you to
-measure various GitLab metrics, pulled from Redis and the database, in Omnibus GitLab
+The [GitLab exporter](https://gitlab.com/gitlab-org/gitlab-exporter) enables you to
+measure various GitLab metrics pulled from Redis and the database in Omnibus GitLab
 instances.
 
 NOTE: **Note:**
-For installations from source you'll have to install and configure it yourself.
+For installations from source you must install and configure it yourself.
 
 To enable the GitLab exporter in an Omnibus GitLab instance:
 
-1. [Enable Prometheus](index.md#configuring-prometheus)
-1. Edit `/etc/gitlab/gitlab.rb`
-1. Add or find and uncomment the following line, making sure it's set to `true`:
+1. [Enable Prometheus](index.md#configuring-prometheus).
+1. Edit `/etc/gitlab/gitlab.rb`.
+1. Add, or find and uncomment, the following line, making sure it's set to `true`:
 
    ```ruby
    gitlab_exporter['enable'] = true
    ```
 
 1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure)
-   for the changes to take effect
+   for the changes to take effect.
 
-Prometheus will now automatically begin collecting performance data from
-the GitLab exporter exposed under `localhost:9168`.
-
-[← Back to the main Prometheus page](index.md)
+Prometheus automatically begins collecting performance data from
+the GitLab exporter exposed at `localhost:9168`.
diff --git a/doc/administration/monitoring/prometheus/gitlab_metrics.md b/doc/administration/monitoring/prometheus/gitlab_metrics.md
index f3084b1732e..fa5e5da80c3 100644
--- a/doc/administration/monitoring/prometheus/gitlab_metrics.md
+++ b/doc/administration/monitoring/prometheus/gitlab_metrics.md
@@ -14,18 +14,18 @@ To enable the GitLab Prometheus metrics:
 1. [Restart GitLab](../../restart_gitlab.md#omnibus-gitlab-restart) for the changes to take effect.
 
 NOTE: **Note:**
-For installations from source you'll have to configure it yourself.
+For installations from source you must configure it yourself.
 
 ## Collecting the metrics
 
 GitLab monitors its own internal service metrics, and makes them available at the
 `/-/metrics` endpoint. Unlike other [Prometheus](https://prometheus.io) exporters, to access
-it, the client IP address needs to be [explicitly allowed](../ip_whitelist.md).
+the metrics, the client IP address must be [explicitly allowed](../ip_whitelist.md).
 
 For [Omnibus GitLab](https://docs.gitlab.com/omnibus/) and Chart installations,
 these metrics are enabled and collected as of
 [GitLab 9.4](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/1702).
-For source installations or earlier versions, these metrics must be enabled
+For source installations, these metrics must be enabled
 manually and collected by a Prometheus server.
 
 For enabling and viewing metrics from Sidekiq nodes, see [Sidekiq metrics](#sidekiq-metrics).
@@ -40,7 +40,7 @@ The following metrics are available:
 | `gitlab_banzai_cacheless_render_real_duration_seconds`         | Histogram |                    9.4 | Duration of rendering Markdown into HTML when cached output does not exist                          | `controller`, `action`                              |
 | `gitlab_cache_misses_total`                                    | Counter   |                   10.2 | Cache read miss                                                                                     | `controller`, `action`                              |
 | `gitlab_cache_operation_duration_seconds`                      | Histogram |                   10.2 | Cache access time                                                                                   |                                                     |
-| `gitlab_cache_operations_total`                                | Counter   |                   12.2 | Cache operations by controller/action                                                               | `controller`, `action`, `operation`                 |
+| `gitlab_cache_operations_total`                                | Counter   |                   12.2 | Cache operations by controller or action                                                               | `controller`, `action`, `operation`                 |
 | `gitlab_ci_pipeline_creation_duration_seconds`                 | Histogram |                   13.0 | Time in seconds it takes to create a CI/CD pipeline                                                 |                                                     |
 | `gitlab_ci_pipeline_size_builds`                               | Histogram |                   13.1 | Total number of builds within a pipeline grouped by a pipeline source                               | `source`                                            |
 | `job_waiter_started_total`                                     | Counter   |                   12.9 | Number of batches of jobs started where a web request is waiting for the jobs to complete           | `worker`                                            |
@@ -92,16 +92,16 @@ The following metrics are available:
 | `gitlab_view_rendering_duration_seconds`                       | Histogram |                   10.2 | Duration for views (histogram)                                                                      | `controller`, `action`, `view`                      |
 | `http_requests_total`                                          | Counter   |                    9.4 | Rack request count                                                                                  | `method`                                            |
 | `http_request_duration_seconds`                                | Histogram |                    9.4 | HTTP response time from rack middleware                                                             | `method`, `status`                                  |
-| `gitlab_transaction_db_count_total`                            | Counter   |                   13.1 | Counter for total number of sql calls                                                               | `controller`, `action`                              |
-| `gitlab_transaction_db_write_count_total`                      | Counter   |                   13.1 | Counter for total number of write sql calls                                                         | `controller`, `action`                              |
-| `gitlab_transaction_db_cached_count_total`                     | Counter   |                   13.1 | Counter for total number of cached sql calls                                                        | `controller`, `action`                              |
+| `gitlab_transaction_db_count_total`                            | Counter   |                   13.1 | Counter for total number of SQL calls                                                               | `controller`, `action`                              |
+| `gitlab_transaction_db_write_count_total`                      | Counter   |                   13.1 | Counter for total number of write SQL calls                                                         | `controller`, `action`                              |
+| `gitlab_transaction_db_cached_count_total`                     | Counter   |                   13.1 | Counter for total number of cached SQL calls                                                        | `controller`, `action`                              |
 | `http_redis_requests_duration_seconds`                         | Histogram |                   13.1 | Redis requests duration during web transactions                                                     | `controller`, `action`                              |
 | `http_redis_requests_total`                                    | Counter   |                   13.1 | Redis requests count during web transactions                                                        | `controller`, `action`                              |
 | `http_elasticsearch_requests_duration_seconds` **(STARTER)**   | Histogram |                   13.1 | Elasticsearch requests duration during web transactions                                             | `controller`, `action`                              |
 | `http_elasticsearch_requests_total` **(STARTER)**              | Counter   |                   13.1 | Elasticsearch requests count during web transactions                                                | `controller`, `action`                              |
 | `pipelines_created_total`                                      | Counter   |                    9.4 | Counter of pipelines created                                                                        |                                                     |
 | `rack_uncaught_errors_total`                                   | Counter   |                    9.4 | Rack connections handling uncaught errors count                                                     |                                                     |
-| `user_session_logins_total`                                    | Counter   |                    9.4 | Counter of how many users have logged in                                                            |                                                     |
+| `user_session_logins_total`                                    | Counter   |                    9.4 | Counter of how many users have logged in since GitLab was started or restarted                                                            |                                                     |
 | `upload_file_does_not_exist`                                   | Counter   | 10.7 in EE, 11.5 in CE | Number of times an upload record could not find its file                                            |                                                     |
 | `failed_login_captcha_total`                                   | Gauge     |                   11.0 | Counter of failed CAPTCHA attempts during login                                                     |                                                     |
 | `successful_login_captcha_total`                               | Gauge     |                   11.0 | Counter of successful CAPTCHA attempts during login                                                 |                                                     |
@@ -120,7 +120,7 @@ The following metrics can be controlled by feature flags:
 ## Sidekiq metrics
 
 Sidekiq jobs may also gather metrics, and these metrics can be accessed if the
-Sidekiq exporter is enabled (for example, using the `monitoring.sidekiq_exporter`
+Sidekiq exporter is enabled: for example, using the `monitoring.sidekiq_exporter`
 configuration option in `gitlab.yml`. These metrics are served from the
 `/metrics` path on the configured port.
 
@@ -173,6 +173,7 @@ configuration option in `gitlab.yml`. These metrics are served from the
 | `geo_repositories_retrying_verification_count` | Gauge   | 11.2  | Number of repositories verification failures that Geo is actively trying to correct on secondary  | `url` |
 | `geo_wikis_retrying_verification_count`        | Gauge   | 11.2  | Number of wikis verification failures that Geo is actively trying to correct on secondary | `url` |
 | `global_search_bulk_cron_queue_size`           | Gauge   | 12.10 | Number of database records waiting to be synchronized to Elasticsearch | |
+| `global_search_awaiting_indexing_queue_size`   | Gauge   | 13.2  | Number of database updates waiting to be synchronized to Elasticsearch while indexing is paused | |
 | `package_files_count`                          | Gauge   | 13.0  | Number of package files on primary | `url` |
 | `package_files_checksummed_count`              | Gauge   | 13.0  | Number of package files checksummed on primary | `url` |
 | `package_files_checksum_failed_count`          | Gauge   | 13.0  | Number of package files failed to calculate the checksum on primary
@@ -187,16 +188,16 @@ The following metrics are available:
 
 ## Connection pool metrics
 
-These metrics record the status of the database [connection pools](https://api.rubyonrails.org/classes/ActiveRecord/ConnectionAdapters/ConnectionPool.html).
+These metrics record the status of the database
+[connection pools](https://api.rubyonrails.org/classes/ActiveRecord/ConnectionAdapters/ConnectionPool.html),
+and the metrics all have these labels:
 
-They all have these labels:
-
-1. `class` - the Ruby class being recorded.
-    1. `ActiveRecord::Base` is the main database connection.
-    1. `Geo::TrackingBase` is the connection to the Geo tracking database, if
-       enabled.
-1. `host` - the host name used to connect to the database.
-1. `port` - the port used to connect to the database.
+- `class` - the Ruby class being recorded.
+  - `ActiveRecord::Base` is the main database connection.
+  - `Geo::TrackingBase` is the connection to the Geo tracking database, if
+    enabled.
+- `host` - the host name used to connect to the database.
+- `port` - the port used to connect to the database.
 
 | Metric                                        | Type  | Since | Description                                       |
 |:----------------------------------------------|:------|:------|:--------------------------------------------------|
@@ -251,15 +252,29 @@ When Puma is used instead of Unicorn, the following metrics are available:
 | `puma_idle_threads`               | Gauge   | 12.0  | Number of spawned threads which are not processing a request |
 | `puma_killer_terminations_total`  | Gauge   | 12.0  | Number of workers terminated by PumaWorkerKiller |
 
+## Redis metrics
+
+These client metrics are meant to complement Redis server metrics.
+These metrics are broken down per [Redis
+instance](https://docs.gitlab.com/omnibus/settings/redis.html#running-with-multiple-redis-instances).
+These metrics all have a `storage` label which indicates the Redis
+instance (`cache`, `shared_state` etc.).
+
+| Metric                            | Type    | Since | Description |
+|:--------------------------------- |:------- |:----- |:----------- |
+| `gitlab_redis_client_exceptions_total`                    | Counter   | 13.2  | Number of Redis client exceptions, broken down by exception class |
+| `gitlab_redis_client_requests_total`                    | Counter   | 13.2  | Number of Redis client requests |
+| `gitlab_redis_client_requests_duration_seconds`                    | Histogram   | 13.2  | Redis request latency, excluding blocking commands |
+
 ## Metrics shared directory
 
 GitLab's Prometheus client requires a directory to store metrics data shared between multi-process services.
 Those files are shared among all instances running under Unicorn server.
 The directory must be accessible to all running Unicorn's processes, or
-metrics won't function correctly.
+metrics can't function correctly.
 
 This directory's location is configured using environment variable `prometheus_multiproc_dir`.
 For best performance, create this directory in `tmpfs`.
 
 If GitLab is installed using [Omnibus GitLab](https://docs.gitlab.com/omnibus/)
-and `tmpfs` is available, then the metrics directory will be configured for you.
+and `tmpfs` is available, then GitLab configures the metrics directory for you.
diff --git a/doc/administration/monitoring/prometheus/index.md b/doc/administration/monitoring/prometheus/index.md
index 1e233b890a2..f0ad0a1a2e6 100644
--- a/doc/administration/monitoring/prometheus/index.md
+++ b/doc/administration/monitoring/prometheus/index.md
@@ -145,6 +145,12 @@ To use an external Prometheus server:
    gitlab_rails['monitoring_whitelist'] = ['127.0.0.0/8', '192.168.0.1']
    ```
 
+1. On **all** GitLab Rails(Puma/Unicorn, Sidekiq) servers, set the Prometheus server IP address and listen port. For example:
+
+   ```ruby
+   gitlab_rails['prometheus_address'] = '192.168.0.1:9090'
+   ```
+
 1. To scrape NGINX metrics, you'll also need to configure NGINX to allow the Prometheus server
    IP. For example:
 
@@ -165,54 +171,54 @@ To use an external Prometheus server:
 
    ```yaml
    scrape_configs:
-   - job_name: nginx
-     static_configs:
-     - targets:
-       - 1.1.1.1:8060
-   - job_name: redis
-     static_configs:
-     - targets:
-       - 1.1.1.1:9121
-   - job_name: postgres
-     static_configs:
-     - targets:
-       - 1.1.1.1:9187
-   - job_name: node
-     static_configs:
-     - targets:
-       - 1.1.1.1:9100
-   - job_name: gitlab-workhorse
-     static_configs:
-     - targets:
-       - 1.1.1.1:9229
-   - job_name: gitlab-rails
-     metrics_path: "/-/metrics"
-     static_configs:
-     - targets:
-       - 1.1.1.1:8080
-   - job_name: gitlab-sidekiq
-     static_configs:
-     - targets:
-       - 1.1.1.1:8082
-   - job_name: gitlab_exporter_database
-     metrics_path: "/database"
-     static_configs:
-     - targets:
-       - 1.1.1.1:9168
-   - job_name: gitlab_exporter_sidekiq
-     metrics_path: "/sidekiq"
-     static_configs:
-     - targets:
-       - 1.1.1.1:9168
-   - job_name: gitlab_exporter_process
-     metrics_path: "/process"
-     static_configs:
-     - targets:
-       - 1.1.1.1:9168
-   - job_name: gitaly
-     static_configs:
-     - targets:
-       - 1.1.1.1:9236
+     - job_name: nginx
+       static_configs:
+         - targets:
+           - 1.1.1.1:8060
+     - job_name: redis
+       static_configs:
+         - targets:
+           - 1.1.1.1:9121
+     - job_name: postgres
+       static_configs:
+         - targets:
+           - 1.1.1.1:9187
+     - job_name: node
+       static_configs:
+         - targets:
+           - 1.1.1.1:9100
+     - job_name: gitlab-workhorse
+       static_configs:
+         - targets:
+           - 1.1.1.1:9229
+     - job_name: gitlab-rails
+       metrics_path: "/-/metrics"
+       static_configs:
+         - targets:
+           - 1.1.1.1:8080
+     - job_name: gitlab-sidekiq
+       static_configs:
+         - targets:
+           - 1.1.1.1:8082
+     - job_name: gitlab_exporter_database
+       metrics_path: "/database"
+       static_configs:
+         - targets:
+           - 1.1.1.1:9168
+     - job_name: gitlab_exporter_sidekiq
+       metrics_path: "/sidekiq"
+       static_configs:
+         - targets:
+           - 1.1.1.1:9168
+     - job_name: gitlab_exporter_process
+       metrics_path: "/process"
+       static_configs:
+         - targets:
+           - 1.1.1.1:9168
+     - job_name: gitaly
+       static_configs:
+         - targets:
+           - 1.1.1.1:9236
    ```
 
 1. Reload the Prometheus server.