Merge branch '35129-update-prometheus-docs' into 'master'

Update Prometheus documentation for 9.4

Closes #35129

See merge request !12885

9 files changed, 182 insertions, 48 deletions

diff --git a/doc/ci/environments.md b/doc/ci/environments.md
index 3393030210e..df5c66a4c85 100644
--- a/doc/ci/environments.md
+++ b/doc/ci/environments.md
@@ -602,9 +602,8 @@ exist, you should see something like:
 >**Notes:**
 >
 - For the monitor dashboard to appear, you need to:
-  - Have enabled the [Kubernetes integration][kube]
-  - Have your app deployed on Kubernetes
   - Have enabled the [Prometheus integration][prom]
+  - Configured Prometheus to collect at least one [supported metric](../user/project/integrations/prometheus_library/metrics.md)
 - With GitLab 9.2, all deployments to an environment are shown directly on the
   monitoring dashboard
 
diff --git a/doc/ci/img/environments_monitoring.png b/doc/ci/img/environments_monitoring.png
index 387b6c54b61..d9c46ea4c95 100644
--- a/doc/ci/img/environments_monitoring.png
+++ b/doc/ci/img/environments_monitoring.png
diff --git a/doc/user/project/integrations/prometheus.md b/doc/user/project/integrations/prometheus.md
index 4ccc13b16ab..6f15765751c 100644
--- a/doc/user/project/integrations/prometheus.md
+++ b/doc/user/project/integrations/prometheus.md
@@ -17,32 +17,27 @@ the settings page with a default template. To configure the template, see the
 Integration with Prometheus requires the following:
 
 1. GitLab 9.0 or higher
-1. The [Kubernetes integration must be enabled][kube] on your project
-1. Your app must be deployed on [Kubernetes][]
-1. Prometheus must be configured to collect Kubernetes metrics
+1. Prometheus must be configured to collect one of the [supported metrics](prometheus_library/metrics.md)
 1. Each metric must be have a label to indicate the environment
-1. GitLab must have network connectivity to the Prometheus sever
+1. GitLab must have network connectivity to the Prometheus server
 
-There are a few steps necessary to set up integration between Prometheus and
-GitLab.
+## Getting started with Prometheus monitoring
 
-## Configuring Prometheus to collect Kubernetes metrics
+Depending on your deployment and where you have located your GitLab server, there are a few options to get started with Prometheus monitoring.
 
-In order for Prometheus to collect Kubernetes metrics, you first must have a
-Prometheus server up and running. You have two options here:
+* If both GitLab and your applications are installed in the same Kubernetes cluster, you can leverage the [bundled Prometheus server within GitLab](#configuring-omnibus-gitlab-prometheus-to-monitor-kubernetes).
+* If your applications are deployed on Kubernetes, but GitLab is not in the same cluster, then you can [configure a Prometheus server in your Kubernetes cluster](#configuring-your-own-prometheus-server-within-kubernetes).
+* If your applications are not running in Kubernetes, [get started with Prometheus](#getting-started-with-prometheus-outside-of-kubernetes).
 
-- If you installed Omnibus GitLab inside of Kubernetes, you can simply use the
-  [bundled version of Prometheus][promgldocs]. In that case, follow the info in the
-  [Omnibus GitLab section](#configuring-omnibus-gitlab-prometheus-to-monitor-kubernetes)
-  below.
-- If you are using GitLab.com or installed GitLab outside of Kubernetes, you
-  will likely need to run a Prometheus server within the Kubernetes cluster.
-  Once installed, the easiest way to monitor Kubernetes is to simply use
-  Prometheus' support for [Kubernetes Service Discovery][prometheus-k8s-sd].
-  In that case, follow the instructions on
-  [configuring your own Prometheus server within Kubernetes](#configuring-your-own-prometheus-server-within-kubernetes).
+### Getting started with Prometheus outside of Kubernetes
 
-### Configuring Omnibus GitLab Prometheus to monitor Kubernetes
+Installing and configuring Prometheus to monitor applications is fairly straight forward.
+
+1. [Install Prometheus](https://prometheus.io/docs/introduction/install/)
+1. Set up one of the [supported monitoring targets](prometheus_library/metrics.md)
+1. Configure the Prometheus server to [collect their metrics](https://prometheus.io/docs/operating/configuration/#scrape_config)
+
+### Configuring Omnibus GitLab Prometheus to monitor Kubernetes deployments
 
 With Omnibus GitLab running inside of Kubernetes, you can leverage the bundled
 version of Prometheus to collect the supported metrics. Once enabled, Prometheus will automatically begin monitoring Kubernetes Nodes and any [annotated Pods](https://prometheus.io/docs/operating/configuration/#<kubernetes_sd_config>). 
@@ -74,7 +69,7 @@ kubectl apply -f path/to/prometheus.yml
 Once deployed, you should see the Prometheus service, deployment, and
 pod start within the `prometheus` namespace. The server will begin to collect
 metrics from each Kubernetes Node in the cluster, based on the configuration
-provided in the template.
+provided in the template. It will also attempt to collect metrics from any Kubernetes Pods that have been [annotated for Prometheus](https://prometheus.io/docs/operating/configuration/#pod).
 
 Since GitLab is not running within Kubernetes, the template provides external
 network access via a `NodePort` running on `30090`. This method allows access
@@ -133,30 +128,6 @@ to integrate with.
 
 ![Configure Prometheus Service](img/prometheus_service_configuration.png)
 
-## Metrics and Labels
-
-GitLab retrieves performance data from two metrics, `container_cpu_usage_seconds_total`
-and `container_memory_usage_bytes`. These metrics are collected from the
-Kubernetes pods via Prometheus, and report CPU and Memory utilization of each
-container or Pod running in the cluster.
-
-In order to isolate and only display relevant metrics for a given environment
-however, GitLab needs a method to detect which pods are associated. To do that,
-GitLab will specifically request metrics that have an `environment` tag that
-matches the [$CI_ENVIRONMENT_SLUG][ci-environment-slug].
-
-If you are using [GitLab Auto-Deploy][autodeploy] and one of the methods of
-configuring Prometheus above, the `environment` will be automatically added.
-
-### GitLab Prometheus queries
-
-The queries utilized by GitLab are shown in the following table.
-
-| Metric | Query |
-| ------ | ----- |
-| Average Memory (MB) | `(sum(container_memory_usage_bytes{container_name!="POD",environment="$CI_ENVIRONMENT_SLUG"}) / count(container_memory_usage_bytes{container_name!="POD",environment="$CI_ENVIRONMENT_SLUG"})) /1024/1024` |
-| Average CPU Utilization (%) | `sum(rate(container_cpu_usage_seconds_total{container_name!="POD",environment="$CI_ENVIRONMENT_SLUG"}[2m])) / count(container_cpu_usage_seconds_total{container_name!="POD",environment="$CI_ENVIRONMENT_SLUG"}) * 100` |
-
 ## Monitoring CI/CD Environments
 
 Once configured, GitLab will attempt to retrieve performance metrics for any
@@ -168,8 +139,9 @@ environment which has had a successful deployment.
 
 > [Introduced][ce-10408] in GitLab 9.2.
 > GitLab 9.3 added the [numeric comparison](https://gitlab.com/gitlab-org/gitlab-ce/issues/27439) of the 30 minute averages.
+> Requires [Kubernetes](prometheus_library/kubernetes.md) metrics
 
-Developers can view the performance impact of their changes within the merge
+Developers can view theperformance impact of their changes within the merge
 request workflow. When a source branch has been deployed to an environment, a sparkline and numeric comparison of the average memory consumption will appear. On the sparkline, a dot
 indicates when the current changes were deployed, with up to 30 minutes of
 performance data displayed before and after. The comparison shows the difference between the 30 minute average before and after the deployment. This information is updated after
diff --git a/doc/user/project/integrations/prometheus_library/cloudwatch.md b/doc/user/project/integrations/prometheus_library/cloudwatch.md
new file mode 100644
index 00000000000..cc5cee36d28
--- /dev/null
+++ b/doc/user/project/integrations/prometheus_library/cloudwatch.md
@@ -0,0 +1,25 @@
+# Monitoring AWS Resources
+> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/12621) in GitLab 9.4
+
+GitLab has support for automatically detecting and monitoring AWS resources, starting with the [Elastic Load Balancer](https://aws.amazon.com/elasticloadbalancing/). This is provided by leveraging the official [Cloudwatch exporter](https://github.com/prometheus/cloudwatch_exporter), which translates [Cloudwatch metrics](https://aws.amazon.com/cloudwatch/) into a Prometheus readable form.
+
+## Metrics supported
+
+| Name | Query |
+| ---- | ----- |
+| Throughput (req/sec) | sum(aws_elb_request_count_sum{%{environment_filter}}) / 60 |
+| Latency (ms) | avg(aws_elb_latency_average{%{environment_filter}}) * 1000 |
+| HTTP Error Rate (%) | sum(aws_elb_httpcode_backend_5_xx_sum{%{environment_filter}}) / sum(aws_elb_request_count_sum{%{environment_filter}}) |
+
+## Configuring Prometheus to monitor for Cloudwatch metrics
+
+To get started with Cloudwatch monitoring, you should install and configure the [Cloudwatch exporter](https://github.com/hnlq715/nginx-vts-exporter) which retrieves and parses the specified Cloudwatch metrics and translates them into a Prometheus monitoring endpoint.
+
+Right now, the only AWS resource supported is the Elastic Load Balancer, whose Cloudwatch metrics can be found [here](http://docs.aws.amazon.com/elasticloadbalancing/latest/classic/elb-cloudwatch-metrics.html).
+
+A sample Cloudwatch Exporter configuration file, configured for basic AWS ELB monitoring, is [available for download](../samples/cloudwatch.yml).
+
+## Specifying the Environment label
+
+In order to isolate and only display relevant metrics for a given environment
+however, GitLab needs a method to detect which labels are associated. To do this, GitLab will [look for an `environment` label](metrics.md#identifying-environments).
diff --git a/doc/user/project/integrations/prometheus_library/kubernetes.md b/doc/user/project/integrations/prometheus_library/kubernetes.md
new file mode 100644
index 00000000000..eb8cd821ddc
--- /dev/null
+++ b/doc/user/project/integrations/prometheus_library/kubernetes.md
@@ -0,0 +1,26 @@
+# Monitoring Kubernetes
+> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/8935) in GitLab 9.0
+
+GitLab has support for automatically detecting and monitoring Kubernetes metrics. Kubernetes exposes Node level metrics out of the box via the built-in [Prometheus metrics support in cAdvisor](https://github.com/google/cadvisor). No additional services or exporters are needed.
+
+## Metrics supported
+
+| Name | Query |
+| ---- | ----- |
+| Average Memory Usage (MB) | (sum(container_memory_usage_bytes{container_name!="POD",%{environment_filter}}) / count(container_memory_usage_bytes{container_name!="POD",%{environment_filter}})) /1024/1024 |
+| Average CPU Utilization (%) | sum(rate(container_cpu_usage_seconds_total{container_name!="POD",%{environment_filter}}[2m])) / count(container_cpu_usage_seconds_total{container_name!="POD",%{environment_filter}}) * 100 |
+
+## Configuring Prometheus to monitor for Kubernetes node metrics
+
+In order for Prometheus to collect Kubernetes metrics, you first must have a
+Prometheus server up and running. You have two options here:
+
+- If you have an Omnibus based GitLab installation within your Kubernetes cluster, you can leverage the bundled Prometheus server to [monitor Kubernetes](../../../../administration/monitoring/prometheus/index.md#configuring-prometheus-to-monitor-kubernetes).
+- To configure your own Prometheus server, you can follow the [Prometheus documentation](https://prometheus.io/docs/introduction/overview/) or [our guide](../../../../administration/monitoring/prometheus/index.md#configuring-your-own-prometheus-server-within-kubernetes).
+
+## Specifying the Environment label
+
+In order to isolate and only display relevant metrics for a given environment
+however, GitLab needs a method to detect which labels are associated. To do this, GitLab will [look for an `environment` label](metrics.md#identifying-environments).
+
+If you are using [GitLab Auto-Deploy][autodeploy] and one of the two [provided Kubernetes monitoring solutions](../prometheus.md#getting-started-with-prometheus-monitoring), the `environment` label will be automatically added.
diff --git a/doc/user/project/integrations/prometheus_library/metrics.md b/doc/user/project/integrations/prometheus_library/metrics.md
new file mode 100644
index 00000000000..55146e57370
--- /dev/null
+++ b/doc/user/project/integrations/prometheus_library/metrics.md
@@ -0,0 +1,25 @@
+# Prometheus Metrics library
+> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/8935) in GitLab 9.0
+
+GitLab offers automatic detection of select [Prometheus exporters](https://prometheus.io/docs/instrumenting/exporters/). Currently supported exporters are:
+* [Kubernetes](kubernetes.md)
+* [NGINX](nginx.md)
+* [Amazon Cloud Watch](cloudwatch.md)
+
+We have tried to surface the most important metrics for each exporter, and will be continuing to add support for additional exporters in future releases. If you would like to add support for other official exporters, [contributions](#adding-to-the-library) are welcome.
+
+## Identifying Environments
+
+GitLab retrieves performance data from the configured Prometheus server, and attempts to identifying the presence of known metrics. Once identified, GitLab then needs to be able to map the data to a particular environment.
+
+In order to isolate and only display relevant metrics for a given environment, GitLab needs a method to detect which labels are associated. To do that,
+GitLab will look for the required metrics which have a label that
+matches the [$CI_ENVIRONMENT_SLUG][ci-environment-slug].
+
+For example if you are deploying to an environment named `production`, there must be a label for the metric with the value of `production`.
+
+## Adding to the library
+
+We strive to support the 2-4 most important metrics for each common system service that supports Prometheus. If you are looking for support for a particular exporter which has not yet been added to the library, additions can be made [to the `additional_metrics.yml`](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/config/prometheus/additional_metrics.yml) file.
+
+> Note: The library is only for monitoring public, common, system services which all customers can benefit from. Support for monitoring [customer proprietary metrics](https://gitlab.com/gitlab-org/gitlab-ee/issues/2273) will be added in a subsequent release.
diff --git a/doc/user/project/integrations/prometheus_library/nginx.md b/doc/user/project/integrations/prometheus_library/nginx.md
new file mode 100644
index 00000000000..fe238e74e36
--- /dev/null
+++ b/doc/user/project/integrations/prometheus_library/nginx.md
@@ -0,0 +1,23 @@
+# Monitoring NGINX
+> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/12621) in GitLab 9.4
+
+GitLab has support for automatically detecting and monitoring NGINX. This is provided by leveraging the [NGINX VTS exporter](https://github.com/hnlq715/nginx-vts-exporter), which translates [VTS statistics](https://github.com/vozlt/nginx-module-vts) into a Prometheus readable form.
+
+## Metrics supported
+
+| Name | Query |
+| ---- | ----- |
+| Throughput (req/sec) | sum(rate(nginx_requests_total{server_zone!="*", server_zone!="_", %{environment_filter}}[2m])) |
+| Latency (ms) | avg(nginx_upstream_response_msecs_avg{%{environment_filter}}) * 1000 |
+| HTTP Error Rate (%) | sum(nginx_responses_total{status_code="5xx", %{environment_filter}}) / sum(nginx_responses_total{server_zone!="*", server_zone!="_", %{environment_filter}}) |
+
+## Configuring Prometheus to monitor for NGINX metrics
+
+To get started with NGINX monitoring, you should first enable the [VTS statistics](https://github.com/vozlt/nginx-module-vts)) module for your NGINX server. This will capture and display statistics in an HTML readable form. Next, you should install and configure the [NGINX VTS exporter](https://github.com/hnlq715/nginx-vts-exporter) which parses these statistics and translates them into a Prometheus monitoring endpoint.
+
+If you are using NGINX as your Kubernetes ingress, there is [upcoming direct support](https://github.com/kubernetes/ingress/pull/423) for enabling Prometheus monitoring in the 0.9.0 release.
+
+## Specifying the Environment label
+
+In order to isolate and only display relevant metrics for a given environment
+however, GitLab needs a method to detect which labels are associated. To do this, GitLab will [look for an `environment` label](metrics.md#identifying-environments).
diff --git a/doc/user/project/integrations/samples/cloudwatch.yml b/doc/user/project/integrations/samples/cloudwatch.yml
new file mode 100644
index 00000000000..d9b58f52c32
--- /dev/null
+++ b/doc/user/project/integrations/samples/cloudwatch.yml
@@ -0,0 +1,26 @@
+region: us-east-1
+  metrics:
+    - aws_namespace: AWS/ELB
+      aws_metric_name: RequestCount
+      aws_dimensions: [AvailabilityZone, LoadBalancerName]
+      aws_dimension_select:
+        LoadBalancerName: [gitlab-ha-lb]
+      aws_statistics: [Sum]
+    - aws_namespace: AWS/ELB
+      aws_metric_name: Latency
+      aws_dimensions: [AvailabilityZone, LoadBalancerName]
+      aws_dimension_select:
+        LoadBalancerName: [gitlab-ha-lb]
+      aws_statistics: [Average]
+    - aws_namespace: AWS/ELB
+      aws_metric_name: HTTPCode_Backend_2XX
+      aws_dimensions: [AvailabilityZone, LoadBalancerName]
+      aws_dimension_select:
+        LoadBalancerName: [gitlab-ha-lb]
+      aws_statistics: [Sum]
+    - aws_namespace: AWS/ELB
+      aws_metric_name: HTTPCode_Backend_5XX
+      aws_dimensions: [AvailabilityZone, LoadBalancerName]
+      aws_dimension_select:
+        LoadBalancerName: [gitlab-ha-lb]
+      aws_statistics: [Sum]
diff --git a/doc/user/project/integrations/samples/prometheus.yml b/doc/user/project/integrations/samples/prometheus.yml
index 01bbcaffe1e..30b59e172a1 100644
--- a/doc/user/project/integrations/samples/prometheus.yml
+++ b/doc/user/project/integrations/samples/prometheus.yml
@@ -24,6 +24,44 @@ data:
           target_label: environment
           regex: (.+)-.+-.+
           replacement: $1
+      - job_name: kubernetes-pods
+        tls_config:
+          ca_file: "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
+          insecure_skip_verify: true
+        bearer_token_file: "/var/run/secrets/kubernetes.io/serviceaccount/token"
+        kubernetes_sd_configs:
+        - role: pod
+          api_server: https://kubernetes.default.svc:443
+          tls_config:
+            ca_file: "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
+          bearer_token_file: "/var/run/secrets/kubernetes.io/serviceaccount/token"
+        relabel_configs:
+        - source_labels:
+          - __meta_kubernetes_pod_annotation_prometheus_io_scrape
+          action: keep
+          regex: 'true'
+        - source_labels:
+          - __meta_kubernetes_pod_annotation_prometheus_io_path
+          action: replace
+          target_label: __metrics_path__
+          regex: "(.+)"
+        - source_labels:
+          - __address__
+          - __meta_kubernetes_pod_annotation_prometheus_io_port
+          action: replace
+          regex: "([^:]+)(?::[0-9]+)?;([0-9]+)"
+          replacement: "$1:$2"
+          target_label: __address__
+        - action: labelmap
+          regex: __meta_kubernetes_pod_label_(.+)
+        - source_labels:
+          - __meta_kubernetes_namespace
+          action: replace
+          target_label: kubernetes_namespace
+        - source_labels:
+          - __meta_kubernetes_pod_name
+          action: replace
+          target_label: kubernetes_pod_name
 ---
 apiVersion: v1
 kind: Service