diff options
Diffstat (limited to 'doc/operations/metrics/alerts.md')
-rw-r--r-- | doc/operations/metrics/alerts.md | 110 |
1 files changed, 110 insertions, 0 deletions
diff --git a/doc/operations/metrics/alerts.md b/doc/operations/metrics/alerts.md new file mode 100644 index 00000000000..43debbd1b78 --- /dev/null +++ b/doc/operations/metrics/alerts.md @@ -0,0 +1,110 @@ +--- +stage: Monitor +group: APM +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + +# Set up alerts for Prometheus metrics + +After [configuring metrics for your CI/CD environment](index.md), you can set up +alerting for Prometheus metrics depending on the location of your instances, and +[trigger actions from alerts](#trigger-actions-from-alerts-ultimate) to notify +your team when environment performance falls outside of the boundaries you set. + +## Managed Prometheus instances + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/6590) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.2 for [custom metrics](index.md#adding-custom-metrics), and 11.3 for [library metrics](../../user/project/integrations/prometheus_library/metrics.md). + +For managed Prometheus instances using auto configuration, you can +[configure alerts for metrics](index.md#adding-custom-metrics) directly in the +[metrics dashboard](index.md). To set an alert: + +1. In your project, navigate to **{cloud-gear}** **Operations > Metrics**, +1. Identify the metric you want to create the alert for, and click the + **ellipsis** **{ellipsis_v}** icon in the top right corner of the metric. +1. Choose **Alerts**. +1. Set threshold and operator. +1. Click **Add** to save and activate the alert. + +![Adding an alert](../../user/project/integrations/img/prometheus_alert.png) + +To remove the alert, click back on the alert icon for the desired metric, and click **Delete**. + +## External Prometheus instances + +>- [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9258) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.8. +>- [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/42640) to [GitLab Core](https://about.gitlab.com/pricing/) in 12.10. + +For manually configured Prometheus servers, GitLab provides a notify endpoint for +use with Prometheus webhooks. If you have manual configuration enabled, an +**Alerts** section is added to **{settings}** **Settings > Integrations > Prometheus**. +This section contains the **URL** and **Authorization Key** you will need. The +**Reset Key** button will invalidate the key and generate a new one. + +![Prometheus service configuration of Alerts](../../user/project/integrations/img/prometheus_service_alerts.png) + +To send GitLab alert notifications, copy the **URL** and **Authorization Key** into the +[`webhook_configs`](https://prometheus.io/docs/alerting/latest/configuration/#webhook_config) +section of your Prometheus Alertmanager configuration: + +```yaml +receivers: + name: gitlab + webhook_configs: + - http_config: + bearer_token: 9e1cbfcd546896a9ea8be557caf13a76 + send_resolved: true + url: http://192.168.178.31:3001/root/manual_prometheus/prometheus/alerts/notify.json + ... +``` + +For GitLab to associate your alerts with an [environment](../../ci/environments/index.md), +you must configure a `gitlab_environment_name` label on the alerts you set up in +Prometheus. The value of this should match the name of your environment in GitLab. + +NOTE: **Note:** +In GitLab versions 13.1 and greater, you can configure your manually configured +Prometheus server to use the +[Generic alerts integration](../../user/project/integrations/generic_alerts.md). + +## Trigger actions from alerts **(ULTIMATE)** + +>- [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/4925) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.11. +>- [From GitLab Ultimate 12.5](https://gitlab.com/gitlab-org/gitlab/-/issues/13401), when GitLab receives a recovery alert, it will automatically close the associated issue. + +Alerts can be used to trigger actions, like opening an issue automatically +(disabled by default since `13.1`). To configure the actions: + +1. Navigate to your project's **{settings}** **Settings > Operations > Incidents**. +1. Enable the option to create issues. +1. Choose the [issue template](../../user/project/description_templates.md) to create the issue from. +1. Optionally, select whether to send an email notification to the developers of the project. +1. Click **Save changes**. + +After enabling, GitLab automatically opens an issue when an alert is triggered containing +values extracted from [alert's payload](https://prometheus.io/docs/alerting/latest/configuration/#webhook_config): + +- Issue author: `GitLab Alert Bot` +- Issue title: Extract from `annotations/title`, `annotations/summary` or `labels/alertname` +- Alert `Summary`: A list of properties + - `starts_at`: Alert start time via `startsAt` + - `full_query`: Alert query extracted from `generatorURL` + - Optional list of attached annotations extracted from `annotations/*` +- Alert [GFM](../../user/markdown.md): GitLab Flavored Markdown from `annotations/gitlab_incident_markdown` + +When GitLab receives a **Recovery Alert**, it closes the associated issue. +This action is recorded as a system message on the issue indicating that it +was closed automatically by the GitLab Alert bot. + +To further customize the issue, you can add labels, mentions, or any other supported +[quick action](../../user/project/quick_actions.md) in the selected issue template, +which applies to all incidents. To limit quick actions or other information to +only specific types of alerts, use the `annotations/gitlab_incident_markdown` field. + +Since [version 12.2](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/63373), +GitLab tags each incident issue with the `incident` label automatically. If the label +does not yet exist, it is also created automatically. + +If the metric exceeds the threshold of the alert for over 5 minutes, GitLab sends +an email to all [Maintainers and Owners](../../user/permissions.md#project-members-permissions) +of the project. |