diff options
Diffstat (limited to 'doc/user/project/integrations/prometheus.md')
| -rw-r--r-- | doc/user/project/integrations/prometheus.md | 81 |
1 files changed, 80 insertions, 1 deletions
diff --git a/doc/user/project/integrations/prometheus.md b/doc/user/project/integrations/prometheus.md index 43a9e24526d..40113624430 100644 --- a/doc/user/project/integrations/prometheus.md +++ b/doc/user/project/integrations/prometheus.md @@ -13,7 +13,7 @@ There are two ways to set up Prometheus integration, depending on where your app - For deployments on Kubernetes, GitLab can automatically [deploy and manage Prometheus](#managed-prometheus-on-kubernetes). - For other deployment targets, simply [specify the Prometheus server](#manual-configuration-of-prometheus). -Once enabled, GitLab will automatically detect metrics from known services in the [metric library](#monitoring-cicd-environments). +Once enabled, GitLab will automatically detect metrics from known services in the [metric library](#monitoring-cicd-environments). You are also able to [add your own metrics](#adding-additional-metrics-premium) as well. ## Enabling Prometheus Integration @@ -93,6 +93,85 @@ GitLab will automatically scan the Prometheus server for metrics from known serv You can view the performance dashboard for an environment by [clicking on the monitoring button](../../../ci/environments.md#monitoring-environments). +### Adding additional metrics **[PREMIUM]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/3799) in [GitLab Premium](https://about.gitlab.com/pricing/) 10.6. + +Additional metrics can be monitored by adding them on the Prometheus integration page. Once saved, they will be displayed on the environment performance dashboard. + + + +A few fields are required: + +- **Name**: Chart title +- **Type**: Type of metric. Metrics of the same type will be shown together. +- **Query**: Valid [PromQL query](https://prometheus.io/docs/prometheus/latest/querying/basics/). +- **Y-axis label**: Y axis title to display on the dashboard. +- **Unit label**: Query units, for example `req / sec`. Shown next to the value. + +Multiple metrics can be displayed on the same chart if the fields **Name**, **Type**, and **Y-axis label** match between metrics. For example, a metric with **Name** `Requests Rate`, **Type** `Business`, and **Y-axis label** `rec / sec` would display on the same chart as a second metric with the same values. A **Legend label** is suggested if this feature used. + +#### Query Variables + +GitLab supports a limited set of [CI variables](../../../ci/variables/README.html) in the Prometheus query. This is particularly useful for identifying a specific environment, for example with `CI_ENVIRONMENT_SLUG`. The supported variables are: + +- CI_ENVIRONMENT_SLUG +- KUBE_NAMESPACE + +To specify a variable in a query, enclose it in curly braces with a leading percent. For example: `%{ci_environment_slug}`. + +### Setting up alerts for Prometheus metrics **[ULTIMATE]** + +#### Managed Prometheus instances + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/6590) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.2 for [custom metrics](#adding-additional-metrics-premium), and 11.3 for [library metrics](prometheus_library/metrics.md). + +For managed Prometheus instances using auto configuration, alerts for metrics [can be configured](#adding-additional-metrics-premium) directly in the performance dashboard. + +To set an alert, click on the alarm icon in the top right corner of the metric you want to create the alert for. A dropdown +will appear, with options to set the threshold and operator. Click **Add** to save and activate the alert. + + + +To remove the alert, click back on the alert icon for the desired metric, and click **Delete**. + +#### External Prometheus instances + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/9258) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.8. + +For manually configured Prometheus servers, a notify endpoint is provided to use with Prometheus webhooks. If you have manual configuration enabled, an **Alerts** section is added to **Settings > Integrations > Prometheus**. This contains the *URL* and *Authorization Key*. The **Reset Key** button will invalidate the key and generate a new one. + + + +To send GitLab alert notifications, copy the *URL* and *Authorization Key* into the [`webhook_configs`](https://prometheus.io/docs/alerting/configuration/#webhook_config) section of your Prometheus Alertmanager configuration: + +```yaml +receivers: + name: gitlab + webhook_configs: + - http_config: + bearer_token: 9e1cbfcd546896a9ea8be557caf13a76 + send_resolved: true + url: http://192.168.178.31:3001/root/manual_prometheus/prometheus/alerts/notify.json + ... +``` + +### Taking action on incidents **[ULTIMATE]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/4925) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.11. + +Alerts can be used to trigger actions, like open an issue automatically. To configure the actions: + +1. Navigate to your project's **Settings > Operations > Incidents**. +1. Enable the option to create issues. +1. Choose the [issue template](../description_templates.md) to create the issue from. +1. Optionally, select whether to send an email notification to the developers of the project. +1. Click **Save changes**. + +Once enabled, an issue will be opened automatically when an alert is triggered. To further customize the issue, you can add labels, mentions, or any other supported [quick action](../quick_actions.md) in the selected issue template. + +If the metric exceeds the threshold of the alert for over 5 minutes, an email will be sent to all [Maintainers and Owners](../../permissions.md#project-members-permissions) of the project. + ## Determining the performance impact of a merge > [Introduced][ce-10408] in GitLab 9.2. |