Refactor Prometheus integration docs

[ci skip]

11 files changed, 145 insertions, 45 deletions

diff --git a/doc/administration/monitoring/prometheus/index.md b/doc/administration/monitoring/prometheus/index.md
index 3a394c561db..51ef14b88cb 100644
--- a/doc/administration/monitoring/prometheus/index.md
+++ b/doc/administration/monitoring/prometheus/index.md
@@ -96,6 +96,28 @@ Sample Prometheus queries:
 - **Data transmitted:** `irate(node_network_transmit_bytes[5m])`
 - **Data received:** `irate(node_network_receive_bytes[5m])`
 
+## Configuring Prometheus to monitor Kubernetes
+
+>**Note:**
+This is a project integration introduced in GitLab 9.0. Read more in the
+[Prometheus integration docs][].
+
+GitLab offers powerful integration with Prometheus for monitoring your apps.
+Metrics are retrieved from the configured Prometheus server, and then displayed
+within the GitLab interface.
+
+To enable the Kubernetes monitoring:
+
+1. Edit `/etc/gitlab/gitlab.rb`
+1. Add or find and uncomment the following line:
+
+    ```ruby
+    prometheus['monitor_kubernetes'] = true
+    ```
+
+1. Save the file and [reconfigure GitLab][reconfigure] for the changes to
+   take effect
+
 ## Prometheus exporters
 
 There are a number of libraries and servers which help in exporting existing
@@ -145,3 +167,4 @@ The GitLab monitor exporter allows you to measure various GitLab metrics.
 [reconfigure]: ../../restart_gitlab.md#omnibus-gitlab-reconfigure
 [#27069]: https://gitlab.com/gitlab-org/gitlab-ce/issues/27069
 [1261]: https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests/1261
+[prometheus integration docs]: ../../../user/project/integrations/prometheus.md
diff --git a/doc/user/project/integrations/img/configure_prometheus_service.png b/doc/user/project/integrations/img/configure_prometheus_service.png
deleted file mode 100644
index 75a7175b3a0..00000000000
--- a/doc/user/project/integrations/img/configure_prometheus_service.png
+++ /dev/null
diff --git a/doc/user/project/integrations/img/deploy_prometheus_yml.png b/doc/user/project/integrations/img/deploy_prometheus_yml.png
deleted file mode 100644
index 73eea5a137b..00000000000
--- a/doc/user/project/integrations/img/deploy_prometheus_yml.png
+++ /dev/null
diff --git a/doc/user/project/integrations/img/gcp_firewall_rule.png b/doc/user/project/integrations/img/gcp_firewall_rule.png
deleted file mode 100644
index cfd407a7479..00000000000
--- a/doc/user/project/integrations/img/gcp_firewall_rule.png
+++ /dev/null
diff --git a/doc/user/project/integrations/img/gcp_node_name.png b/doc/user/project/integrations/img/gcp_node_name.png
deleted file mode 100644
index 0bf88f58ec2..00000000000
--- a/doc/user/project/integrations/img/gcp_node_name.png
+++ /dev/null
diff --git a/doc/user/project/integrations/img/prometheus_gcp_firewall_rule.png b/doc/user/project/integrations/img/prometheus_gcp_firewall_rule.png
new file mode 100644
index 00000000000..e30cba211e6
--- /dev/null
+++ b/doc/user/project/integrations/img/prometheus_gcp_firewall_rule.png
diff --git a/doc/user/project/integrations/img/prometheus_gcp_node_name.png b/doc/user/project/integrations/img/prometheus_gcp_node_name.png
new file mode 100644
index 00000000000..ea289431454
--- /dev/null
+++ b/doc/user/project/integrations/img/prometheus_gcp_node_name.png
diff --git a/doc/user/project/integrations/img/prometheus_service_configuration.png b/doc/user/project/integrations/img/prometheus_service_configuration.png
new file mode 100644
index 00000000000..c7dfe874817
--- /dev/null
+++ b/doc/user/project/integrations/img/prometheus_service_configuration.png
diff --git a/doc/user/project/integrations/img/prometheus_yaml_deploy.png b/doc/user/project/integrations/img/prometheus_yaml_deploy.png
new file mode 100644
index 00000000000..978cd7eaa50
--- /dev/null
+++ b/doc/user/project/integrations/img/prometheus_yaml_deploy.png
diff --git a/doc/user/project/integrations/project_services.md b/doc/user/project/integrations/project_services.md
index c481f174157..3cf1cc704a2 100644
--- a/doc/user/project/integrations/project_services.md
+++ b/doc/user/project/integrations/project_services.md
@@ -51,7 +51,7 @@ Click on the service links to see further configuration instructions and details
 | [Slack Notifications](slack.md) | Receive event notifications in Slack |
 | [Slack slash commands](slack_slash_commands.md) | Slack chat and ChatOps slash commands |
 | PivotalTracker | Project Management Software (Source Commits Endpoint) |
-| Prometheus | Monitor the performance of your deployed apps |
+| [Prometheus](prometheus.md) | Monitor the performance of your deployed apps |
 | Pushover | Pushover makes it easy to get real-time notifications on your Android device, iPhone, iPad, and Desktop |
 | [Redmine](redmine.md) | Redmine issue tracker |
 
diff --git a/doc/user/project/integrations/prometheus.md b/doc/user/project/integrations/prometheus.md
index e507c18ec85..fb91d56086a 100644
--- a/doc/user/project/integrations/prometheus.md
+++ b/doc/user/project/integrations/prometheus.md
@@ -1,94 +1,171 @@
 # Prometheus integration
 
-GitLab offers powerful integration with Prometheus for monitoring your apps. Metrics are retrieved from the configured Prometheus server, and then displayed within the GitLab interface.
+>**Notes:**
+- [Introduced][ce-8935] in GitLab 9.0.
+- Parts of this integration require access to the GitLab server. If you don't
+  have access but want to enable Prometheus integration, ask your administrator.
 
-Each project can be configured with its own specific Prometheus server, see the [configuration](#configuration) section for more details. If you have a single Prometheus server which monitors all of your infrastructure, you can pre-fill the settings page with a default template. To configure the template, see the [Services Templates](services-templates) document.
+GitLab offers powerful integration with [Prometheus] for monitoring your apps.
+Metrics are retrieved from the configured Prometheus server, and then displayed
+within the GitLab interface.
+
+Each project can be configured with its own specific Prometheus server, see the
+[configuration](#configuration) section for more details. If you have a single
+Prometheus server which monitors all of your infrastructure, you can pre-fill
+the settings page with a default template. To configure the template, see the
+[Services templates](services_templates.md) document.
 
 ## Requirements
 
 Integration with Prometheus requires the following:
-1. GitLab 9.0 or higher.
-1. Your app must be deployed on [Kubernetes](kubernetes).
-1. Prometheus must be configured to collect Kubernetes metrics.
-1. Each metric must be have a label to indicate the environment.
-1. GitLab must have network connectivity to the Prometheus sever.
 
-## Configuration
+1. GitLab 9.0 or higher
+1. Your app must be deployed on [Kubernetes][]
+1. Prometheus must be configured to collect Kubernetes metrics
+1. Each metric must be have a label to indicate the environment
+1. GitLab must have network connectivity to the Prometheus sever
+
+There are a few steps necessary to set up integration between Prometheus and
+GitLab.
+
+## Configuring Prometheus to collect Kubernetes metrics
+
+In order for Prometheus to collect Kubernetes metrics, you first must have a
+Prometheus server up and running. You have two options here:
 
-There are a few steps necessary to set up integration between Prometheus and GitLab.
+1. If you installed Omnibus GitLab inside of Kubernetes, you can simply use the
+   [bundled version of Prometheus](#configuring-prometheus-to-collect-kubernetes-metrics).
+1. If you are using GitLab.com or installed GitLab outside of Kubernetes, you
+   will likely need to run a [Prometheus server within the Kubernetes cluster](#configuring-your-own-prometheus-server-within-kubernetes).
+   Once installed, the easiest way to monitor Kubernetes is to simply use
+   Prometheus' support for [Kubernetes Service Discovery][prometheus-k8s-sd].
 
-### Configuring Prometheus to collect Kubernetes metrics
+### Configuring Omnibus GitLab Prometheus to monitor Kubernetes
 
-In order for Prometheus to collect Kubernetes metrics, you first must have a Prometheus server. If you installed Omnibus GitLab inside of Kubernetes, you can simply use the bundled version of Prometheus. If you are using GitLab.com or installed GitLab outside of Kubernetes, you will likely need to run a Prometheus server within the Kubernetes cluster.
+With Omnibus GitLab running inside of Kubernetes, you can leverage the bundled
+version of Prometheus to collect the required metrics.
 
-Once installed, the easiest way to monitor Kubernetes is to simply use Prometheus' support for [Kubernetes Service Discovery](prometheus-k8s-sd).
+Read how to configure the bundled Prometheus server in the
+[Administration guide][gitlab-prometheus-k8s-monitor].
 
-#### Configuring Omnibus GitLab's Prometheus to monitor Kubernetes
+Now that Prometheus is configured, proceed on
+[configuring the Prometheus project service in GitLab](##configuration-in-gitlab).
 
-With GitLab running inside of Kubernetes, you can leverage the bundled version of Prometheus to collect the required metrics. To make this as easy as possible, we have added a new option within `gitlab.rb`. Simply set  `prometheus['monitor_kubernetes']` to true, and metrics will begin to be collected from Kubernetes.
+### Configuring your own Prometheus server within Kubernetes
 
-More detailed information on configuring the bundled Prometheus server is available in our [Administration guide](gitlab-prometheus-k8s-monitor).
+Setting up and configuring Prometheus within Kubernetes is quick and painless.
+The Prometheus project provides an [official Docker image][prometheus-docker-image]
+which we can use as a starting point.
 
-#### Setting up and configuring your own Prometheus server within Kubernetes
+To get started quickly, we have provided a [sample YML file][prometheus-yml]
+that can be used as a template. This file will create a `prometheus` **Namespace**,
+**Service**, **Deployment**, and **ConfigMap** in Kubernetes. You can upload
+this file to the Kubernetes dashboard using the **+ Create** at the top right.
 
-Setting up and configuring Prometheus within Kubernetes is quick and painless. The Prometheus project provides an [official Docker image](prometheus-docker-image) which we can use as a starting point.
+![Deploy Prometheus](img/prometheus_yaml_deploy.png)
 
-To get started quickly, We have provided a [sample YML file](prometheus-yml) can be used as a template. This file will create a `prometheus` Namespace, Service, Deployment, and ConfigMap. You can upload this file to the Kubernetes dashboard using the `+ Create` at the top right, or use `kubectl`.
+Or use `kubectl`:
 
-![Deploy Prometheus](img/deploy_prometheus_yml.png)
+```bash
+kubectl apply -f path/to/prometheus.yml
+```
 
-Once started, you should start to see the Prometheus service, deployment, and pod start within the `prometheus` namespace. The server will begin to collect metrics from each Kubernetes Node in the cluster, based on the configuration provided in the template.
+Once deployed, you should see the Prometheus service, deployment, and
+pod start within the `prometheus` namespace. The server will begin to collect
+metrics from each Kubernetes Node in the cluster, based on the configuration
+provided in the template.
 
-##### Enabling and controlling access to the deployed Prometheus server
+Since GitLab is not running within Kubernetes, the template provides external
+network access via a `NodePort` running on `30090`. This method allows access
+to be controlled using provider firewall rules, like within Google Compute Engine.
 
-Since GitLab is not running within Kubernetes, the template provides external network access via a `NodePort` running on `30090`. This method allows access to be controlled using provider firewall rules, like within Google Compute Engine.
+Since a `NodePort` does not automatically have firewall rules created for it,
+one will need to be created manually to allow access. In GCP/GKE, you will want
+to confirm the Node that the Prometheus pod is running on. This can be done
+either by looking at the Pod in the Kubernetes dashboard, or by running:
 
-Since a `NodePort` does not automatically have firewall rules created for it, one will need to be created manually to allow access. In GCP/GKE, you will want to confirm the Node that the Prometheus pod is running on. This can be done either by looking at the Pod in the Kubernetes dashboard, or by running `kubectl describe pods -n prometheus`.
+```bash
+kubectl describe pods -n prometheus
+```
 
-Next on GKE, we need to get the `tag` of the Node or VM Instance, so we can create an accurate firewall rule. The easiest way to do this is to go into the Google Cloud Platform Compute console and select the VM instance that matches the name of the Node gathered from the step above. In this case, the node tag needed is `gke-prometheus-demo-5d5ada10-node`. Also make a note of the `External IP`, which will be the IP address the Prometheus server is reachable on.
+Next on GKE, we need to get the `tag` of the Node or VM Instance, so we can
+create an accurate firewall rule. The easiest way to do this is to go into the
+Google Cloud Platform Compute console and select the VM instance that matches
+the name of the Node gathered from the step above. In this case, the node tag
+needed is `gke-prometheus-demo-5d5ada10-node`. Also make a note of the
+**External IP**, which will be the IP address the Prometheus server is reachable
+on.
 
-![GCP Node Detail](img/gcp_node_name.png)
+![GCP Node Detail](img/prometheus_gcp_node_name.png)
 
-Armed with the proper Node tag, the firewall rule can now be created specifically for this node. To create the firewall rule, open the Google Cloud Platform Networking console, and select `Firewall Rules`.
+Armed with the proper Node tag, the firewall rule can now be created
+specifically for this node. To create the firewall rule, open the Google Cloud
+Platform Networking console, and select **Firewall Rules**.
 
 Create a new rule:
-* Specify the source IP range to match your desired access list, which should include your GitLab server. GitLab.com's IP address range is available [here](gitlab.com-ip-range), note that GitLab.com's IP's are subject to change without notification.
-* Allowed protocol and port should be `tcp:30090`.
-* The target tags should match the Node tag identified earlier in this step.
 
-![GCP Firewall Rule](img/gcp_firewall_rule.png)
+- Specify the source IP range to match your desired access list, which should
+  include your GitLab server. A sample of GitLab.com's IP address range is
+  available [in this issue][gitlab.com-ip-range], but note that GitLab.com's IPs
+  are subject to change without prior notification.
+- Allowed protocol and port should be `tcp:30090`.
+- The target tags should match the Node tag identified earlier in this step.
+
+![GCP Firewall Rule](img/prometheus_gcp_firewall_rule.png)
+
+---
+
+Now that Prometheus is configured, proceed on
+[configuring the Prometheus project service in GitLab](##configuration-in-gitlab).
 
-### Configuration in GitLab
+## Configuration in GitLab
 
-Actual configuration of Prometheus integration is very simple. All you will need is the DNS or IP address of the Prometheus server you'd like to integrate with.
+The actual configuration of Prometheus integration within GitLab is very simple.
+All you will need is the DNS or IP address of the Prometheus server you'd like
+to integrate with.
 
-Navigate to the [Integrations page](project_services.md#accessing-the-project-services), click the **Prometheus** service, and then provide the base URL of the your server, for example http://prometheus.example.com/. The `Test Settings` button can be used to confirm connectivity from GitLab to the Prometheus server.
+1. Navigate to the [Integrations page](project_services.md#accessing-the-project-services)
+1. Click the **Prometheus** service
+1. Provide the base URL of the your server, for example `http://prometheus.example.com/`.
+   The **Test Settings** button can be used to confirm connectivity from GitLab
+   to the Prometheus server.
 
-![Configure Prometheus Service](img/configure_prometheus_service.png)
+![Configure Prometheus Service](img/prometheus_service_configuration.png)
 
-### Metrics and Labels
+## Metrics and Labels
 
-Presently GitLab retrieves performance data from two metrics, `container_cpu_usage_seconds_total` and `container_memory_usage_bytes`. These metrics are collected from the Kubernetes pods via Prometheus, and report CPU and Memory utilization of each container or Pod running in the cluster.
+GitLab retrieves performance data from two metrics, `container_cpu_usage_seconds_total`
+and `container_memory_usage_bytes`. These metrics are collected from the
+Kubernetes pods via Prometheus, and report CPU and Memory utilization of each
+container or Pod running in the cluster.
 
-In order to isolate and only display relevant metrics for a given environment however, GitLab needs a method to detect which pods are associated. To do that, GitLab will specifically request metrics that have an `environment` tag that matches [$CI_ENVIRONMENT_SLUG](ci-environment-slug).
+In order to isolate and only display relevant metrics for a given environment
+however, GitLab needs a method to detect which pods are associated. To do that,
+GitLab will specifically request metrics that have an `environment` tag that
+matches the [$CI_ENVIRONMENT_SLUG][ci-environment-slug].
 
-If you are using GitLab Auto-Deploy and one of the methods of configuring Prometheus above, the `environment` will be automatically added.
+If you are using [GitLab Auto-Deploy][autodeploy] and one of the methods of
+configuring Prometheus above, the `environment` will be automatically added.
 
-#### GitLab Prometheus Queries
+### GitLab Prometheus queries
 
-The queries utilized by GitLab are:
+The queries utilized by GitLab are shown in the following table.
 
 | Metric | Query |
-| --- | --- |
-| Average Memory (MB) | (sum(container_memory_usage_bytes{container_name="app",environment="$CI_ENVIRONMENT_SLUG"}) / count(container_memory_usage_bytes{container_name="app",environment="$CI_ENVIRONMENT_SLUG"})) /1024/1024 |
-| Average CPU Utilization (%) | sum(rate(container_cpu_usage_seconds_total{container_name="app",environment="$CI_ENVIRONMENT_SLUG"}[2m])) / count(container_cpu_usage_seconds_total{container_name="app",environment="$CI_ENVIRONMENT_SLUG"}) * 100 |
+| ------ | ----- |
+| Average Memory (MB) | `(sum(container_memory_usage_bytes{container_name="app",environment="$CI_ENVIRONMENT_SLUG"}) / count(container_memory_usage_bytes{container_name="app",environment="$CI_ENVIRONMENT_SLUG"})) /1024/1024` |
+| Average CPU Utilization (%) | `sum(rate(container_cpu_usage_seconds_total{container_name="app",environment="$CI_ENVIRONMENT_SLUG"}[2m])) / count(container_cpu_usage_seconds_total{container_name="app",environment="$CI_ENVIRONMENT_SLUG"}) * 100` |
 
 ## Troubleshooting
 
+[autodeploy]: ../../../ci/autodeploy/index.md
 [kubernetes]: https://kubernetes.io
 [prometheus-k8s-sd]: https://prometheus.io/docs/operating/configuration/#<kubernetes_sd_config>
-[gitlab-prometheus-k8s-monitor]: TODO
+[prometheus]: https://prometheus.io
+[gitlab-prometheus-k8s-monitor]: ../../../administration/monitoring/prometheus/index.md#configuring-prometheus-to-monitor-kubernetes
 [prometheus-docker-image]: https://hub.docker.com/r/prom/prometheus/
 [prometheus-yml]:samples/prometheus.yml
 [gitlab.com-ip-range]: https://gitlab.com/gitlab-com/infrastructure/issues/434
 [ci-environment-slug]: https://docs.gitlab.com/ce/ci/variables/#predefined-variables-environment-variables
+[ce-8935]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/8935
+[promgldocs]: ../../../administration/monitoring/prometheus/index.md