Merge branch 'docs-prometheus-service-discovery' into 'master'

Documentation for Prometheus Service Discovery

Closes #63110

See merge request gitlab-org/gitlab-ce!29811

(cherry picked from commit 46640168fa13e7e4b7b32adb003ff3a5b883192b)

95525a56 Adding documentation for Prometheus Service Discovery
781d05af Moved monitoring to its own step
f6c0fd2e Adding monitoring to number list in Redis
db2f83f3 Updates based on review
1105b8ea A couple of missed adjustments

6 files changed, 192 insertions, 2 deletions

diff --git a/doc/administration/high_availability/README.md b/doc/administration/high_availability/README.md
index d9c80b1ec59..0c4f926c579 100644
--- a/doc/administration/high_availability/README.md
+++ b/doc/administration/high_availability/README.md
@@ -65,6 +65,7 @@ larger one.
 - 1 Redis node
 - 1 NFS/Gitaly storage server
 - 2 or more GitLab application nodes (Unicorn, Workhorse, Sidekiq)
+- 1 Monitoring node (Prometheus, Grafana)
 
 #### Installation Instructions
 
@@ -76,6 +77,7 @@ you can continue with the next step.
 1. [Redis](redis.md#redis-in-a-scaled-environment)
 1. [Gitaly](gitaly.md) (recommended) or [NFS](nfs.md)
 1. [GitLab application nodes](gitlab.md)
+1. [Monitoring node (Prometheus and Grafana)](monitoring_node.md)
 
 ### Full Scaling
 
@@ -90,6 +92,7 @@ in size, indicating that there is contention or not enough resources.
 - 2 or more NFS/Gitaly storage servers
 - 2 or more Sidekiq nodes
 - 2 or more GitLab application nodes (Unicorn, Workhorse)
+- 1 Monitoring node (Prometheus, Grafana)
 
 ## High Availability Architecture Examples
 
@@ -133,6 +136,7 @@ the contention.
 - 3 Consul/Sentinel nodes
 - 2 or more GitLab application nodes (Unicorn, Workhorse, Sidekiq, PGBouncer)
 - 1 NFS/Gitaly server
+- 1 Monitoring node (Prometheus, Grafana)
 
 ![Horizontal architecture diagram](img/horizontal.png)
 
@@ -192,6 +196,7 @@ with the added complexity of many more nodes to configure, manage and monitor.
 - 2 or more API nodes (All requests to `/api`)
 - 2 or more Web nodes (All other web requests)
 - 2 or more NFS/Gitaly servers
+- 1 Monitoring node (Prometheus, Grafana)
 
 ![Fully Distributed architecture diagram](img/fully-distributed.png)
 
@@ -205,4 +210,5 @@ separately:
    1. [NFS Client and Host setup](nfs_host_client_setup.md)
 1. [Configure the GitLab application servers](gitlab.md)
 1. [Configure the load balancers](load_balancer.md)
+1. [Monitoring node (Prometheus and Grafana)](monitoring_node.md)
 
diff --git a/doc/administration/high_availability/database.md b/doc/administration/high_availability/database.md
index 3b874e5d312..0aab7d6a58f 100644
--- a/doc/administration/high_availability/database.md
+++ b/doc/administration/high_availability/database.md
@@ -82,7 +82,8 @@ deploy the bundled PostgreSQL.
 1. Note the PostgreSQL node's IP address or hostname, port, and
    plain text password. These will be necessary when configuring the GitLab
    application servers later.
-
+1. [Enable monitoring](#enable-monitoring)
+   
 Advanced configuration options are supported and can be added if
 needed.
 
@@ -399,6 +400,7 @@ check the [Troubleshooting section](#troubleshooting) before proceeding.
    ```
 
 1. [Reconfigure GitLab] for the changes to take effect.
+1. [Enable Monitoring](#enable-monitoring)
 
 > Please note:
 >
@@ -1086,6 +1088,25 @@ the previous section:
       the `gitlab` database user
    1. [Reconfigure GitLab] for the changes to take effect
 
+## Enable Monitoring
+
+> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/3786) in GitLab 12.0.
+
+If you enable Monitoring, it must be enabled on **all** database servers.
+
+1. Create/edit `/etc/gitlab/gitlab.rb` and add the following configuration:
+
+   ```ruby
+   # Enable service discovery for Prometheus
+   consul['monitoring_service_discovery'] = true
+
+   # Set the network addresses that the exporters will listen on
+   node_exporter['listen_address'] = '0.0.0.0:9100'
+   postgres_exporter['listen_address'] = '0.0.0.0:9187'
+   ```
+
+1. Run `sudo gitlab-ctl reconfigure` to compile the configuration.
+
 ## Troubleshooting
 
 #### Consul and PostgreSQL changes not taking effect.
diff --git a/doc/administration/high_availability/gitaly.md b/doc/administration/high_availability/gitaly.md
index 1d8e6c999cb..90e5f71d835 100644
--- a/doc/administration/high_availability/gitaly.md
+++ b/doc/administration/high_availability/gitaly.md
@@ -19,3 +19,28 @@ Continue configuration of other components by going back to:
 
 - [Scaled Architectures](README.md#scalable-architecture-examples)
 - [High Availability Architectures](README.md#high-availability-architecture-examples)
+
+## Enable Monitoring
+
+> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/3786) in GitLab 12.0.
+
+  1. Create/edit `/etc/gitlab/gitlab.rb` and add the following configuration:
+
+     ```ruby
+     # Enable service discovery for Prometheus
+     consul['enable'] = true
+     consul['monitoring_service_discovery'] =  true
+
+     # Replace placeholders
+     # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z
+     # with the addresses of the Consul server nodes
+     consul['configuration'] = {
+        retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z),
+     }
+
+     # Set the network addresses that the exporters will listen on
+     node_exporter['listen_address'] = '0.0.0.0:9100' 
+     gitaly['prometheus_listen_addr'] = "0.0.0.0:9236"
+     ```
+
+  1. Run `sudo gitlab-ctl reconfigure` to compile the configuration.
diff --git a/doc/administration/high_availability/gitlab.md b/doc/administration/high_availability/gitlab.md
index 888426ece5c..1a530bb6255 100644
--- a/doc/administration/high_availability/gitlab.md
+++ b/doc/administration/high_availability/gitlab.md
@@ -76,6 +76,8 @@
     registry['gid'] = 9002
     ```
 
+1. [Enable monitoring](#enable-monitoring)
+
     > **Note:** To maintain uniformity of links across HA clusters, the `external_url`
     on the first application server as well as the additional application
     servers should point to the external url that users will use to access GitLab.
@@ -88,7 +90,8 @@
     [Nginx documentation](http://docs.gitlab.com/omnibus/settings/nginx.html#enable-https)
     for more information.
     >
-    > **Note:** It is best to set the `uid` and `gid`s prior to the initial reconfigure of GitLab. Omnibus will not recursively `chown` directories if set after the initial reconfigure.
+    > **Note:** It is best to set the `uid` and `gid`s prior to the initial reconfigure
+    of GitLab. Omnibus will not recursively `chown` directories if set after the initial reconfigure.
 
 ## First GitLab application server
 
@@ -129,6 +132,46 @@ need some extra configuration.
 
 1. Run `sudo gitlab-ctl reconfigure` to compile the configuration.
 
+## Enable Monitoring
+
+> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/3786) in GitLab 12.0.
+
+If you enable Monitoring, it must be enabled on **all** GitLab servers.
+
+1. Create/edit `/etc/gitlab/gitlab.rb` and add the following configuration:
+
+   ```ruby
+   # Enable service discovery for Prometheus
+   consul['enable'] = true
+   consul['monitoring_service_discovery'] =  true
+
+   # Replace placeholders
+   # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z
+   # with the addresses of the Consul server nodes
+   consul['configuration'] = {
+      retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z),
+   }
+
+   # Set the network addresses that the exporters will listen on
+   node_exporter['listen_address'] = '0.0.0.0:9100'
+   gitlab_workhorse['prometheus_listen_addr'] = '0.0.0.0:9229'
+   sidekiq['listen_address'] = "0.0.0.0"
+   unicorn['listen'] = '0.0.0.0'
+
+   # Add the monitoring node's IP address to the monitoring whitelist and allow it to scrape the NGINX metrics
+   # Replace placeholder
+   # monitoring.gitlab.example.com
+   # with the addresses gathered for the monitoring node
+   gitlab_rails['monitoring_whitelist'] = ['monitoring.gitlab.example.com']
+   nginx['status']['options']['allow'] = ['monitoring.gitlab.example.com']
+   ```
+
+1. Run `sudo gitlab-ctl reconfigure` to compile the configuration.
+
+> **Warning:** After changing `unicorn['listen']` in `gitlab.rb`, and running `sudo gitlab-ctl reconfigure`,
+  it can take an extended period of time for unicorn to complete reloading after receiving a `HUP`.
+  For more information, see the [issue](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/4401).
+
 ## Troubleshooting
 
 - `mount: wrong fs type, bad option, bad superblock on`
diff --git a/doc/administration/high_availability/monitoring_node.md b/doc/administration/high_availability/monitoring_node.md
new file mode 100644
index 00000000000..d16bf7dc0f0
--- /dev/null
+++ b/doc/administration/high_availability/monitoring_node.md
@@ -0,0 +1,67 @@
+# Configuring a Monitoring node for Scaling and High Availability
+
+> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/3786) in GitLab 12.0.
+
+## Standalone Monitoring node using GitLab Omnibus
+
+The GitLab Omnibus package can be used to configure a standalone Monitoring node running Prometheus and Grafana.
+The monitoring node is not highly available. See [Scaling and High Availability](README.md)
+for an overview of GitLab scaling and high availability options.
+
+The steps below are the minimum necessary to configure a Monitoring node running Prometheus and Grafana with
+Omnibus:
+
+1. SSH into the Monitoring node.
+1. [Download/install](https://about.gitlab.com/installation) the Omnibus GitLab
+   package you want using **steps 1 and 2** from the GitLab downloads page.
+   - Do not complete any other steps on the download page.
+
+1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
+
+    ```ruby
+    external_url 'http://gitlab.example.com'
+
+    # Enable Prometheus
+    prometheus['enable'] = true
+    prometheus['listen_address'] = '0.0.0.0:9090'
+    prometheus['monitor_kubernetes'] = false
+
+    # Enable Grafana
+    grafana['enable'] = true
+    grafana['admin_password'] = 'toomanysecrets'
+
+    # Enable service discovery for Prometheus
+    consul['enable'] = true
+    consul['monitoring_service_discovery'] =  true
+
+    # Replace placeholders
+    # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z
+    # with the addresses of the Consul server nodes
+    consul['configuration'] = {
+       retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z),
+    }
+
+    # Disable all other services
+    gitlab_rails['auto_migrate'] = false
+    alertmanager['enable'] = false
+    gitaly['enable'] = false
+    gitlab_monitor['enable'] = false
+    gitlab_workhorse['enable'] = false
+    nginx['enable'] = true
+    postgres_exporter['enable'] = false
+    postgresql['enable'] = false
+    redis['enable'] = false
+    redis_exporter['enable'] = false
+    sidekiq['enable'] = false
+    unicorn['enable'] = false
+    node_exporter['enable'] = false
+    ```
+
+1. Run `sudo gitlab-ctl reconfigure` to compile the configuration.
+
+## Migrating to Service Discovery
+
+Once monitoring using Service Discovery is enabled with `consul['monitoring_service_discovery'] =  true`,
+ensure that `prometheus['scrape_configs']` is not set  in `/etc/gitlab/gitlab.rb`. Setting both
+`consul['monitoring_service_discovery'] =  true` and `prometheus['scrape_configs']` in `/etc/gitlab/gitlab.rb`
+will result in errors.
diff --git a/doc/administration/high_availability/redis.md b/doc/administration/high_availability/redis.md
index 1aaa709fc8f..f61a8834af3 100644
--- a/doc/administration/high_availability/redis.md
+++ b/doc/administration/high_availability/redis.md
@@ -74,6 +74,7 @@ Omnibus:
 1. Note the Redis node's IP address or hostname, port, and
    Redis password. These will be necessary when configuring the GitLab
    application servers later.
+1. [Enable Monitoring](#enable-monitoring)   
 
 Advanced configuration options are supported and can be added if
 needed.
@@ -749,6 +750,33 @@ gitlab_rails['redis_sentinels'] = [
 
 [Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect.
 
+## Enable Monitoring
+
+> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/3786) in GitLab 12.0.
+
+  If you enable Monitoring, it must be enabled on **all** Redis servers.
+
+  1. Create/edit `/etc/gitlab/gitlab.rb` and add the following configuration:
+
+     ```ruby
+     # Enable service discovery for Prometheus
+     consul['enable'] = true
+     consul['monitoring_service_discovery'] =  true
+
+     # Replace placeholders
+     # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z
+     # with the addresses of the Consul server nodes
+     consul['configuration'] = {
+        retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z),
+     }
+
+     # Set the network addresses that the exporters will listen on
+     node_exporter['listen_address'] = '0.0.0.0:9100'
+     redis_exporter['listen_address'] = '0.0.0.0:9121'
+     ```
+
+  1. Run `sudo gitlab-ctl reconfigure` to compile the configuration.
+
 ## Advanced configuration
 
 Omnibus GitLab configures some things behind the curtains to make the sysadmins'