From a55106d629bde3d7d39ca7ec126f955e9a08b7bd Mon Sep 17 00:00:00 2001 From: Joshua Lambert Date: Tue, 20 Feb 2018 01:48:06 -0500 Subject: Update Prometheus docs for enhanced integration --- .../integrations/img/prometheus_dashboard.png | Bin 0 -> 63234 bytes .../project/integrations/img/prometheus_deploy.png | Bin 0 -> 27258 bytes doc/user/project/integrations/prometheus.md | 128 +++++++-------------- .../integrations/prometheus_library/kubernetes.md | 9 +- .../project/integrations/samples/prometheus.yml | 107 ----------------- 5 files changed, 45 insertions(+), 199 deletions(-) create mode 100644 doc/user/project/integrations/img/prometheus_dashboard.png create mode 100644 doc/user/project/integrations/img/prometheus_deploy.png delete mode 100644 doc/user/project/integrations/samples/prometheus.yml diff --git a/doc/user/project/integrations/img/prometheus_dashboard.png b/doc/user/project/integrations/img/prometheus_dashboard.png new file mode 100644 index 00000000000..84eac5eb939 Binary files /dev/null and b/doc/user/project/integrations/img/prometheus_dashboard.png differ diff --git a/doc/user/project/integrations/img/prometheus_deploy.png b/doc/user/project/integrations/img/prometheus_deploy.png new file mode 100644 index 00000000000..d39081bcc7b Binary files /dev/null and b/doc/user/project/integrations/img/prometheus_deploy.png differ diff --git a/doc/user/project/integrations/prometheus.md b/doc/user/project/integrations/prometheus.md index 5fefb3b69c4..a0358a33f8d 100644 --- a/doc/user/project/integrations/prometheus.md +++ b/doc/user/project/integrations/prometheus.md @@ -2,119 +2,71 @@ > [Introduced][ce-8935] in GitLab 9.0. -GitLab offers powerful integration with [Prometheus] for monitoring your apps. -Metrics are retrieved from the configured Prometheus server, and then displayed +GitLab offers powerful integration with [Prometheus] for monitoring key metrics your apps, directly within GitLab. +Metrics for each environment are retrieved from Prometheus, and then displayed within the GitLab interface. -Each project can be configured with its own specific Prometheus server, see the -[configuration](#configuration) section for more details. If you have a single -Prometheus server which monitors all of your infrastructure, you can pre-fill -the settings page with a default template. To configure the template, see the -[Services templates](services_templates.md) document. +![Environment Dashboard](img/prometheus_dashboard.png) -## Requirements +There are two ways to setup Prometheus integration, depending on where your apps are running: +* For deployments on Kubernetes, GitLab can [deploy and manage Prometheus](#managed-prometheus-on-kubernetes) in a cluster +* For other deployment targets, simply [specify the Prometheus server](manual-configuration-of-prometheus). -Integration with Prometheus requires the following: - -1. GitLab 9.0 or higher -1. Prometheus must be configured to collect one of the [supported metrics](prometheus_library/metrics.md) -1. Each metric must be have a label to indicate the environment -1. GitLab must have network connectivity to the Prometheus server - -## Getting started with Prometheus monitoring - -Depending on your deployment and where you have located your GitLab server, there are a few options to get started with Prometheus monitoring. - -* If both GitLab and your applications are installed in the same Kubernetes cluster, you can leverage the [bundled Prometheus server within GitLab](#configuring-omnibus-gitlab-prometheus-to-monitor-kubernetes). -* If your applications are deployed on Kubernetes, but GitLab is not in the same cluster, then you can [configure a Prometheus server in your Kubernetes cluster](#configuring-your-own-prometheus-server-within-kubernetes). -* If your applications are not running in Kubernetes, [get started with Prometheus](#getting-started-with-prometheus-outside-of-kubernetes). - -### Getting started with Prometheus outside of Kubernetes - -Installing and configuring Prometheus to monitor applications is fairly straight forward. - -1. [Install Prometheus](https://prometheus.io/docs/introduction/install/) -1. Set up one of the [supported monitoring targets](prometheus_library/metrics.md) -1. Configure the Prometheus server to [collect their metrics](https://prometheus.io/docs/operating/configuration/#scrape_config) - -### Configuring Omnibus GitLab Prometheus to monitor Kubernetes deployments +## Managed Prometheus on Kubernetes -With Omnibus GitLab running inside of Kubernetes, you can leverage the bundled -version of Prometheus to collect the supported metrics. Once enabled, Prometheus will automatically begin monitoring Kubernetes Nodes and any [annotated Pods](https://prometheus.io/docs/operating/configuration/#). +GitLab can seamlessly deploy and manage Prometheus on a [connected Kubernetes cluster](../clusters/index.html), making monitoring the metrics of your deployed apps as easy as a single click. -1. Read how to configure the bundled Prometheus server in the - [Administration guide][gitlab-prometheus-k8s-monitor]. -1. Now that Prometheus is configured, proceed on - [configuring the Prometheus project service in GitLab](#configuration-in-gitlab). +### Requirements -### Configuring your own Prometheus server within Kubernetes +* GitLab [10.5 or above](https://gitlab.com/gitlab-org/gitlab-ce/issues/28916) +* A [connected Kubernetes cluster](../clusters/index.html) +* Helm Tiller [installed by GitLab](../clusters/index.html#installing-applications) -Setting up and configuring Prometheus within Kubernetes is quick and painless. -The Prometheus project provides an [official Docker image][prometheus-docker-image] -which we can use as a starting point. +### Getting started -To get started quickly, we have provided a [sample YML file][prometheus-yml] -that can be used as a template. This file will create a `prometheus` **Namespace**, -**Service**, **Deployment**, and **ConfigMap** in Kubernetes. You can upload -this file to the Kubernetes dashboard using **+ Create** at the top right. +Once you have a connected Kubernetes cluster with Helm installed, deploying a managed Prometheus is as easy as a single click. -![Deploy Prometheus](img/prometheus_yaml_deploy.png) +1. Go to the `CI/CD > Kubernetes` page, to view your connected clusters +1. Select the cluster you would like to deploy Prometheus to +1. Click the **Install** button to deploy Prometheus to the cluster -Or use `kubectl`: +![Managed Prometheus Deploy](img/prometheus_deploy.png) -```bash -kubectl apply -f path/to/prometheus.yml -``` +### About managed Prometheus deployments -Once deployed, you should see the Prometheus service, deployment, and -pod start within the `prometheus` namespace. The server will begin to collect -metrics from each Kubernetes Node in the cluster, based on the configuration -provided in the template. It will also attempt to collect metrics from any Kubernetes Pods that have been [annotated for Prometheus](https://prometheus.io/docs/operating/configuration/#pod). +Prometheus is deployed into the `gitlab-managed-apps` namespace, using the [official Helm chart](https://github.com/kubernetes/charts/tree/master/stable/prometheus). Prometheus is only accessible within the cluster, with GitLab communicating through the [Kubernetes API](https://kubernetes.io/docs/concepts/overview/kubernetes-api/). -Since GitLab is not running within Kubernetes, the template provides external -network access via a `NodePort` running on `30090`. This method allows access -to be controlled using provider firewall rules, like within Google Compute Engine. +The Prometheus server will [automatically detect and monitor](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Ckubernetes_sd_config%3E) nodes, pods, and endpoints. -Since a `NodePort` does not automatically have firewall rules created for it, -one will need to be created manually to allow access. In GCP/GKE, you will want -to confirm the Node that the Prometheus pod is running on. This can be done -either by looking at the Pod in the Kubernetes dashboard, or by running: +To configure a resource to be monitored by Prometheus, simply set the following [Kubernetes annotations](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/): +* `prometheus.io/scrape` to `true` to enable monitoring of the resource. +* `prometheus.io/port` to define the port of the metrics endpoint. +* `prometheus.io/path` to define the path of the metrics endpoint. Defaults to `/metrics`. -```bash -kubectl describe pods -n prometheus -``` +CPU and Memory consumption is also monitored, but requires [naming conventions](prometheus_library/kubernetes.html#specifying-the-environment) in order to determine the environment. If you are using [Auto DevOps](../../../topics/autodevops/), this is handled automatically. -Next on GKE, we need to get the `tag` of the Node or VM Instance, so we can -create an accurate firewall rule. The easiest way to do this is to go into the -Google Cloud Platform Compute console and select the VM instance that matches -the name of the Node gathered from the step above. In this case, the node tag -needed is `gke-prometheus-demo-5d5ada10-node`. Also make a note of the -**External IP**, which will be the IP address the Prometheus server is reachable -on. +The [NGINX Ingress]((../clusters/index.html#installing-applications)) that is deployed by GitLab to clusters, is automatically annotated for monitoring providing key response metrics: latency, throughput, and error rates. -![GCP Node Detail](img/prometheus_gcp_node_name.png) +## Manual configuration of Prometheus -Armed with the proper Node tag, the firewall rule can now be created -specifically for this node. To create the firewall rule, open the Google Cloud -Platform Networking console, and select **Firewall Rules**. +### Requirements -Create a new rule: +Integration with Prometheus requires the following: -- Specify the source IP range to match your desired access list, which should - include your GitLab server. A sample of GitLab.com's IP address range is - available [in this issue][gitlab.com-ip-range], but note that GitLab.com's IPs - are subject to change without prior notification. -- Allowed protocol and port should be `tcp:30090`. -- The target tags should match the Node tag identified earlier in this step. +1. GitLab 9.0 or higher +1. Prometheus must be configured to collect one of the [supported metrics](prometheus_library/metrics.md) +1. Each metric must be have a label to indicate the environment +1. GitLab must have network connectivity to the Prometheus server -![GCP Firewall Rule](img/prometheus_gcp_firewall_rule.png) +### Getting started ---- +Installing and configuring Prometheus to monitor applications is fairly straight forward. -Now that Prometheus is configured, proceed to -[configure the Prometheus project service in GitLab](##configuration-in-gitlab). +1. [Install Prometheus](https://prometheus.io/docs/introduction/install/) +1. Set up one of the [supported monitoring targets](prometheus_library/metrics.md) +1. Configure the Prometheus server to [collect their metrics](https://prometheus.io/docs/operating/configuration/#scrape_config) -## Configuration in GitLab +### Configuration in GitLab The actual configuration of Prometheus integration within GitLab is very simple. All you will need is the DNS or IP address of the Prometheus server you'd like diff --git a/doc/user/project/integrations/prometheus_library/kubernetes.md b/doc/user/project/integrations/prometheus_library/kubernetes.md index a6673fa2a00..81d022314c2 100644 --- a/doc/user/project/integrations/prometheus_library/kubernetes.md +++ b/doc/user/project/integrations/prometheus_library/kubernetes.md @@ -24,9 +24,10 @@ Prometheus server up and running. You have two options here: - If you have an Omnibus based GitLab installation within your Kubernetes cluster, you can leverage the bundled Prometheus server to [monitor Kubernetes](../../../../administration/monitoring/prometheus/index.md#configuring-prometheus-to-monitor-kubernetes). - To configure your own Prometheus server, you can follow the [Prometheus documentation](https://prometheus.io/docs/introduction/overview/) or [our guide](../../../../administration/monitoring/prometheus/index.md#configuring-your-own-prometheus-server-within-kubernetes). -## Specifying the Environment label +## Specifying the Environment -In order to isolate and only display relevant metrics for a given environment -however, GitLab needs a method to detect which labels are associated. To do this, GitLab will [look for an `environment` label](metrics.md#identifying-environments). +In order to isolate and only display relevant CPU and Memory metrics for a given environment, GitLab needs a method to detect which containers it is running. Because these metrics are tracked at the container level, traditional Kubernetes labels are not available. -If you are using [GitLab Auto-Deploy](../../../../ci/autodeploy/index.md) and one of the two [provided Kubernetes monitoring solutions](../prometheus.md#getting-started-with-prometheus-monitoring), the `environment` label will be automatically added. +Instead, the [Deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) or [DaemonSet](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/) name should begin with the name of the [enviroment](../../../../environments.html). It can be followed by a `-` and additional content if desired. + +If you are using [GitLab Auto-Deploy](../../../../ci/autodeploy/index.md) and one of the two [provided Kubernetes monitoring solutions](../prometheus.md#getting-started-with-prometheus-monitoring), the naming will be correctly set automatically. diff --git a/doc/user/project/integrations/samples/prometheus.yml b/doc/user/project/integrations/samples/prometheus.yml deleted file mode 100644 index 3a4735d282f..00000000000 --- a/doc/user/project/integrations/samples/prometheus.yml +++ /dev/null @@ -1,107 +0,0 @@ -apiVersion: v1 -kind: Namespace -metadata: - name: prometheus ---- -apiVersion: v1 -kind: ConfigMap -metadata: - name: prometheus - namespace: prometheus -data: - prometheus.yml: |- - scrape_configs: - - job_name: 'kubernetes-nodes' - scheme: https - tls_config: - ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt - insecure_skip_verify: true - bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token - kubernetes_sd_configs: - - role: node - metric_relabel_configs: - - source_labels: [pod_name] - target_label: environment - regex: (.+)-.+-.+ - replacement: $1 - - job_name: kubernetes-pods - tls_config: - ca_file: "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt" - insecure_skip_verify: true - bearer_token_file: "/var/run/secrets/kubernetes.io/serviceaccount/token" - kubernetes_sd_configs: - - role: pod - api_server: https://kubernetes.default.svc:443 - tls_config: - ca_file: "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt" - bearer_token_file: "/var/run/secrets/kubernetes.io/serviceaccount/token" - relabel_configs: - - source_labels: - - __meta_kubernetes_pod_annotation_prometheus_io_scrape - action: keep - regex: 'true' - - source_labels: - - __meta_kubernetes_pod_annotation_prometheus_io_path - action: replace - target_label: __metrics_path__ - regex: "(.+)" - - source_labels: - - __address__ - - __meta_kubernetes_pod_annotation_prometheus_io_port - action: replace - regex: "([^:]+)(?::[0-9]+)?;([0-9]+)" - replacement: "$1:$2" - target_label: __address__ - - action: labelmap - regex: __meta_kubernetes_pod_label_(.+) - - source_labels: - - __meta_kubernetes_namespace - action: replace - target_label: kubernetes_namespace - - source_labels: - - __meta_kubernetes_pod_name - action: replace - target_label: kubernetes_pod_name ---- -apiVersion: v1 -kind: Service -metadata: - name: prometheus - namespace: prometheus -spec: - selector: - app: prometheus - ports: - - name: prometheus - protocol: TCP - port: 9090 - nodePort: 30090 - type: NodePort ---- -apiVersion: extensions/v1beta1 -kind: Deployment -metadata: - name: prometheus - namespace: prometheus -spec: - replicas: 1 - template: - metadata: - labels: - app: prometheus - spec: - containers: - - name: prometheus - image: prom/prometheus:latest - args: - - '--config.file=/prometheus-data/prometheus.yml' - ports: - - name: prometheus - containerPort: 9090 - volumeMounts: - - name: data-volume - mountPath: /prometheus-data - volumes: - - name: data-volume - configMap: - name: prometheus -- cgit v1.2.1 From c7fd85a7790ac821ce5cdaee245fae7e6b9f8d5a Mon Sep 17 00:00:00 2001 From: Joshua Lambert Date: Tue, 20 Feb 2018 01:56:40 -0500 Subject: Fix links --- doc/user/project/integrations/prometheus.md | 10 +++++----- doc/user/project/integrations/prometheus_library/kubernetes.md | 2 +- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/doc/user/project/integrations/prometheus.md b/doc/user/project/integrations/prometheus.md index a0358a33f8d..8ddebf89c76 100644 --- a/doc/user/project/integrations/prometheus.md +++ b/doc/user/project/integrations/prometheus.md @@ -10,17 +10,17 @@ within the GitLab interface. There are two ways to setup Prometheus integration, depending on where your apps are running: * For deployments on Kubernetes, GitLab can [deploy and manage Prometheus](#managed-prometheus-on-kubernetes) in a cluster -* For other deployment targets, simply [specify the Prometheus server](manual-configuration-of-prometheus). +* For other deployment targets, simply [specify the Prometheus server](#manual-configuration-of-prometheus). ## Managed Prometheus on Kubernetes -GitLab can seamlessly deploy and manage Prometheus on a [connected Kubernetes cluster](../clusters/index.html), making monitoring the metrics of your deployed apps as easy as a single click. +GitLab can seamlessly deploy and manage Prometheus on a [connected Kubernetes cluster](../clusters/index.md), making monitoring the metrics of your deployed apps as easy as a single click. ### Requirements * GitLab [10.5 or above](https://gitlab.com/gitlab-org/gitlab-ce/issues/28916) -* A [connected Kubernetes cluster](../clusters/index.html) -* Helm Tiller [installed by GitLab](../clusters/index.html#installing-applications) +* A [connected Kubernetes cluster](../clusters/index.md) +* Helm Tiller [installed by GitLab](../clusters/index.md#installing-applications) ### Getting started @@ -45,7 +45,7 @@ To configure a resource to be monitored by Prometheus, simply set the following CPU and Memory consumption is also monitored, but requires [naming conventions](prometheus_library/kubernetes.html#specifying-the-environment) in order to determine the environment. If you are using [Auto DevOps](../../../topics/autodevops/), this is handled automatically. -The [NGINX Ingress]((../clusters/index.html#installing-applications)) that is deployed by GitLab to clusters, is automatically annotated for monitoring providing key response metrics: latency, throughput, and error rates. +The [NGINX Ingress](../clusters/index.md#installing-applications) that is deployed by GitLab to clusters, is automatically annotated for monitoring providing key response metrics: latency, throughput, and error rates. ## Manual configuration of Prometheus diff --git a/doc/user/project/integrations/prometheus_library/kubernetes.md b/doc/user/project/integrations/prometheus_library/kubernetes.md index 81d022314c2..8fbaa1a4a34 100644 --- a/doc/user/project/integrations/prometheus_library/kubernetes.md +++ b/doc/user/project/integrations/prometheus_library/kubernetes.md @@ -28,6 +28,6 @@ Prometheus server up and running. You have two options here: In order to isolate and only display relevant CPU and Memory metrics for a given environment, GitLab needs a method to detect which containers it is running. Because these metrics are tracked at the container level, traditional Kubernetes labels are not available. -Instead, the [Deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) or [DaemonSet](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/) name should begin with the name of the [enviroment](../../../../environments.html). It can be followed by a `-` and additional content if desired. +Instead, the [Deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) or [DaemonSet](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/) name should begin with the name of the [environment](../../../../ci/environments.md). It can be followed by a `-` and additional content if desired. If you are using [GitLab Auto-Deploy](../../../../ci/autodeploy/index.md) and one of the two [provided Kubernetes monitoring solutions](../prometheus.md#getting-started-with-prometheus-monitoring), the naming will be correctly set automatically. -- cgit v1.2.1 From 7c9070a96523c8746d4945cc2029c15f93b0e82b Mon Sep 17 00:00:00 2001 From: Joshua Lambert Date: Tue, 20 Feb 2018 02:08:57 -0500 Subject: Minor polish --- .../integrations/img/prometheus_dashboard.png | Bin 63234 -> 26112 bytes doc/user/project/integrations/prometheus.md | 10 ++++------ 2 files changed, 4 insertions(+), 6 deletions(-) diff --git a/doc/user/project/integrations/img/prometheus_dashboard.png b/doc/user/project/integrations/img/prometheus_dashboard.png index 84eac5eb939..bd19f1b44cc 100644 Binary files a/doc/user/project/integrations/img/prometheus_dashboard.png and b/doc/user/project/integrations/img/prometheus_dashboard.png differ diff --git a/doc/user/project/integrations/prometheus.md b/doc/user/project/integrations/prometheus.md index 8ddebf89c76..5d0d58eaf80 100644 --- a/doc/user/project/integrations/prometheus.md +++ b/doc/user/project/integrations/prometheus.md @@ -9,12 +9,12 @@ within the GitLab interface. ![Environment Dashboard](img/prometheus_dashboard.png) There are two ways to setup Prometheus integration, depending on where your apps are running: -* For deployments on Kubernetes, GitLab can [deploy and manage Prometheus](#managed-prometheus-on-kubernetes) in a cluster +* For deployments on Kubernetes, GitLab can automatically [deploy and manage Prometheus](#managed-prometheus-on-kubernetes) * For other deployment targets, simply [specify the Prometheus server](#manual-configuration-of-prometheus). ## Managed Prometheus on Kubernetes -GitLab can seamlessly deploy and manage Prometheus on a [connected Kubernetes cluster](../clusters/index.md), making monitoring the metrics of your deployed apps as easy as a single click. +GitLab can seamlessly deploy and manage Prometheus on a [connected Kubernetes cluster](../clusters/index.md), making monitoring of your apps easy. ### Requirements @@ -36,14 +36,12 @@ Once you have a connected Kubernetes cluster with Helm installed, deploying a ma Prometheus is deployed into the `gitlab-managed-apps` namespace, using the [official Helm chart](https://github.com/kubernetes/charts/tree/master/stable/prometheus). Prometheus is only accessible within the cluster, with GitLab communicating through the [Kubernetes API](https://kubernetes.io/docs/concepts/overview/kubernetes-api/). -The Prometheus server will [automatically detect and monitor](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Ckubernetes_sd_config%3E) nodes, pods, and endpoints. - -To configure a resource to be monitored by Prometheus, simply set the following [Kubernetes annotations](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/): +The Prometheus server will [automatically detect and monitor](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Ckubernetes_sd_config%3E) nodes, pods, and endpoints. To configure a resource to be monitored by Prometheus, simply set the following [Kubernetes annotations](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/): * `prometheus.io/scrape` to `true` to enable monitoring of the resource. * `prometheus.io/port` to define the port of the metrics endpoint. * `prometheus.io/path` to define the path of the metrics endpoint. Defaults to `/metrics`. -CPU and Memory consumption is also monitored, but requires [naming conventions](prometheus_library/kubernetes.html#specifying-the-environment) in order to determine the environment. If you are using [Auto DevOps](../../../topics/autodevops/), this is handled automatically. +CPU and Memory consumption is monitored, but requires [naming conventions](prometheus_library/kubernetes.html#specifying-the-environment) in order to determine the environment. If you are using [Auto DevOps](../../../topics/autodevops/), this is handled automatically. The [NGINX Ingress](../clusters/index.md#installing-applications) that is deployed by GitLab to clusters, is automatically annotated for monitoring providing key response metrics: latency, throughput, and error rates. -- cgit v1.2.1 From 39c56f41468edc034fd8a3cc437e9a1d7755e87b Mon Sep 17 00:00:00 2001 From: Joshua Lambert Date: Tue, 20 Feb 2018 09:33:38 -0500 Subject: Update NGINX docs --- .../integrations/prometheus_library/nginx_ingress.md | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-) diff --git a/doc/user/project/integrations/prometheus_library/nginx_ingress.md b/doc/user/project/integrations/prometheus_library/nginx_ingress.md index e6f13d0630b..a767ae9261f 100644 --- a/doc/user/project/integrations/prometheus_library/nginx_ingress.md +++ b/doc/user/project/integrations/prometheus_library/nginx_ingress.md @@ -2,11 +2,11 @@ > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/13438) in GitLab 9.5 -GitLab has support for automatically detecting and monitoring the Kubernetes NGINX ingress controller. This is provided by leveraging the built in Prometheus metrics included in [version 0.9.0](https://github.com/kubernetes/ingress/blob/master/controllers/nginx/Changelog.md#09-beta1) of the ingress. +GitLab has support for automatically detecting and monitoring the Kubernetes NGINX ingress controller. This is provided by leveraging the built in Prometheus metrics included in [version 0.9.0](https://github.com/kubernetes/ingress/blob/master/controllers/nginx/Changelog.md#09-beta1) and above of the ingress. ## Requirements -The [Prometheus service](../prometheus/index.md) must be enabled. +[Prometheus integration](../prometheus/index.md) must be active. ## Metrics supported @@ -18,24 +18,23 @@ The [Prometheus service](../prometheus/index.md) must be enabled. ## Configuring NGINX ingress monitoring -If you have deployed with the [gitlab-omnibus](https://docs.gitlab.com/ee/install/kubernetes/gitlab_omnibus.md) Helm chart, and your application is running in the same cluster, no further action is required. The ingress metrics will be automatically enabled and annotated for Prometheus monitoring. Simply ensure Prometheus monitoring is [enabled for your project](../prometheus.md), which is on by default. +If you have deployed NGINX Ingress using GitLab's [Kubernetes cluster integration](../../clusters/index.md#installing-applications), it will automatically be monitored by Prometheus. For other deployments, there is some configuration required depending on your installation: -* NGINX Ingress should be version 0.9.0 or above +* NGINX Ingress should be version 0.9.0 or above, with metrics enabled * NGINX Ingress should be annotated for Prometheus monitoring * Prometheus should be configured to monitor annotated pods -### Setting up NGINX Ingress for Prometheus monitoring +### Manually setting up NGINX Ingress for Prometheus monitoring Version 0.9.0 and above of [NGINX ingress](https://github.com/kubernetes/ingress/tree/master/controllers/nginx) have built-in support for exporting Prometheus metrics. To enable, a ConfigMap setting must be passed: `enable-vts-status: "true"`. Once enabled, a Prometheus metrics endpoint will start running on port 10254. -With metric data now available, Prometheus needs to be configured to collect it. The easiest way to do this is to leverage Prometheus' [built-in Kubernetes service discovery](https://prometheus.io/docs/operating/configuration/#kubernetes_sd_config), which automatically detects a variety of Kubernetes components and makes them available for monitoring. Since NGINX ingress metrics are exposed per pod, a scrape job for Kubernetes pods is required. A sample pod scraping configuration [is available](https://github.com/prometheus/prometheus/blob/master/documentation/examples/prometheus-kubernetes.yml#L248). This configuration will detect pods and enable collection of metrics **only if** they have been specifically annotated for monitoring. +Next, the ingress needs to be annotated for Prometheus monitoring. Two new annotations need to be added: -Depending on how NGINX ingress was deployed, typically a DaemonSet or Deployment, edit the corresponding YML spec. Two new annotations need to be added: * `prometheus.io/scrape: "true"` * `prometheus.io/port: "10254"` -Prometheus should now be collecting NGINX ingress metrics. To validate view the Prometheus Targets, available under `Status > Targets` on the Prometheus dashboard. New entries for NGINX should be listed in the kubernetes pod monitoring job, `kubernetes-pods`. +Managing these settings depends on how NGINX ingress has been deployed. If you have deployed via the [official Helm chart](https://github.com/kubernetes/charts/tree/master/stable/nginx-ingress), metrics can be enabled with `controller.stats.enabled` along with the required annotations. Alternatively it is possible edit the NGINX ingress YML directly in the [Kubernetes dashboard](https://github.com/kubernetes/dashboard). ## Specifying the Environment label -- cgit v1.2.1 From ed8dce703652f8c0958fe3fa95d5ff24cfe9b432 Mon Sep 17 00:00:00 2001 From: Joshua Lambert Date: Tue, 20 Feb 2018 15:10:57 -0500 Subject: Feedback --- .../integrations/img/prometheus_gcp_firewall_rule.png | Bin 15247 -> 0 bytes .../integrations/img/prometheus_gcp_node_name.png | Bin 52622 -> 0 bytes .../integrations/prometheus_library/kubernetes.md | 2 +- .../integrations/prometheus_library/nginx_ingress.md | 15 +++++++++++++-- 4 files changed, 14 insertions(+), 3 deletions(-) delete mode 100644 doc/user/project/integrations/img/prometheus_gcp_firewall_rule.png delete mode 100644 doc/user/project/integrations/img/prometheus_gcp_node_name.png diff --git a/doc/user/project/integrations/img/prometheus_gcp_firewall_rule.png b/doc/user/project/integrations/img/prometheus_gcp_firewall_rule.png deleted file mode 100644 index e30cba211e6..00000000000 Binary files a/doc/user/project/integrations/img/prometheus_gcp_firewall_rule.png and /dev/null differ diff --git a/doc/user/project/integrations/img/prometheus_gcp_node_name.png b/doc/user/project/integrations/img/prometheus_gcp_node_name.png deleted file mode 100644 index ea289431454..00000000000 Binary files a/doc/user/project/integrations/img/prometheus_gcp_node_name.png and /dev/null differ diff --git a/doc/user/project/integrations/prometheus_library/kubernetes.md b/doc/user/project/integrations/prometheus_library/kubernetes.md index 8fbaa1a4a34..106241ff191 100644 --- a/doc/user/project/integrations/prometheus_library/kubernetes.md +++ b/doc/user/project/integrations/prometheus_library/kubernetes.md @@ -28,6 +28,6 @@ Prometheus server up and running. You have two options here: In order to isolate and only display relevant CPU and Memory metrics for a given environment, GitLab needs a method to detect which containers it is running. Because these metrics are tracked at the container level, traditional Kubernetes labels are not available. -Instead, the [Deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) or [DaemonSet](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/) name should begin with the name of the [environment](../../../../ci/environments.md). It can be followed by a `-` and additional content if desired. +Instead, the [Deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) or [DaemonSet](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/) name should begin with the name of the [environment](../../../../ci/environments.md). It can be followed by a `-` and additional content if desired. For example, a deployment name of `review-homepage-5620p5` would match the `review/homepage` environment. If you are using [GitLab Auto-Deploy](../../../../ci/autodeploy/index.md) and one of the two [provided Kubernetes monitoring solutions](../prometheus.md#getting-started-with-prometheus-monitoring), the naming will be correctly set automatically. diff --git a/doc/user/project/integrations/prometheus_library/nginx_ingress.md b/doc/user/project/integrations/prometheus_library/nginx_ingress.md index a767ae9261f..49b34c82ae6 100644 --- a/doc/user/project/integrations/prometheus_library/nginx_ingress.md +++ b/doc/user/project/integrations/prometheus_library/nginx_ingress.md @@ -18,13 +18,24 @@ GitLab has support for automatically detecting and monitoring the Kubernetes NGI ## Configuring NGINX ingress monitoring -If you have deployed NGINX Ingress using GitLab's [Kubernetes cluster integration](../../clusters/index.md#installing-applications), it will automatically be monitored by Prometheus. +If you have deployed NGINX Ingress using GitLab's [Kubernetes cluster integration](../../clusters/index.md#installing-applications), it will [automatically be monitored](#about-managed-nginx-ingress-deployments) by Prometheus. -For other deployments, there is some configuration required depending on your installation: +For other deployments, there is [some configuration](#manually-setting-up-nginx-ingress-for-prometheus-monitoring) required depending on your installation: * NGINX Ingress should be version 0.9.0 or above, with metrics enabled * NGINX Ingress should be annotated for Prometheus monitoring * Prometheus should be configured to monitor annotated pods +### About managed NGINX Ingress deployments + +NGINX Ingress is deployed into the `gitlab-managed-apps` namespace, using the [official Helm chart](https://github.com/kubernetes/charts/tree/master/stable/nginx-ingress). NGINX Ingress will be [externally reachable via the Load Balancer's IP](https://docs.gitlab.com/ce/user/project/clusters/index.html#getting-the-external-ip-address). + +NGINX is configured for Prometheus monitoring, by setting: +* `enable-vts-status: "true"`, to export Prometheus metrics +* `prometheus.io/scrape: "true"`, to enable automatic discovery +* `prometheus.io/port: "10254"`, to specify the metrics port + +When used in conjunction with the GitLab deployed Prometheus service, response metrics will be automatically collected. + ### Manually setting up NGINX Ingress for Prometheus monitoring Version 0.9.0 and above of [NGINX ingress](https://github.com/kubernetes/ingress/tree/master/controllers/nginx) have built-in support for exporting Prometheus metrics. To enable, a ConfigMap setting must be passed: `enable-vts-status: "true"`. Once enabled, a Prometheus metrics endpoint will start running on port 10254. -- cgit v1.2.1 From 2f5985157241f27af09be1bc89b830625c874a4e Mon Sep 17 00:00:00 2001 From: Joshua Lambert Date: Tue, 20 Feb 2018 15:14:22 -0500 Subject: One last item of feedback --- doc/user/project/integrations/prometheus.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/user/project/integrations/prometheus.md b/doc/user/project/integrations/prometheus.md index 5d0d58eaf80..249463fb86e 100644 --- a/doc/user/project/integrations/prometheus.md +++ b/doc/user/project/integrations/prometheus.md @@ -13,12 +13,12 @@ There are two ways to setup Prometheus integration, depending on where your apps * For other deployment targets, simply [specify the Prometheus server](#manual-configuration-of-prometheus). ## Managed Prometheus on Kubernetes +> **Note**: [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/28916) in GitLab 10.5 GitLab can seamlessly deploy and manage Prometheus on a [connected Kubernetes cluster](../clusters/index.md), making monitoring of your apps easy. ### Requirements -* GitLab [10.5 or above](https://gitlab.com/gitlab-org/gitlab-ce/issues/28916) * A [connected Kubernetes cluster](../clusters/index.md) * Helm Tiller [installed by GitLab](../clusters/index.md#installing-applications) -- cgit v1.2.1