summaryrefslogtreecommitdiff
path: root/doc/operations/incident_management/alert_integrations.md
blob: f2225ad99920aaacef23a5dbf0054ab6f2325d5a (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
---
stage: Monitor
group: Health
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers
---

# Generic alerts integration

> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13203) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 12.4.
> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/42640) to [GitLab Core](https://about.gitlab.com/pricing/) in 12.8.

GitLab can accept alerts from any source via a generic webhook receiver.
When you set up the generic alerts integration, a unique endpoint will
be created which can receive a payload in JSON format, and will in turn
create an issue with the payload in the body of the issue. You can always
[customize the payload](#customizing-the-payload) to your liking.

The entire payload will be posted in the issue discussion as a comment
authored by the GitLab Alert Bot.

NOTE: **Note:**
In GitLab versions 13.1 and greater, you can configure
[External Prometheus instances](../metrics/alerts.md#external-prometheus-instances)
to use this endpoint.

## Setting up generic alerts

To obtain credentials for setting up a generic alerts integration:

1. Sign in to GitLab as a user with maintainer [permissions](../../user/permissions.md)
   for a project.
1. Navigate to the **Operations** page for your project, depending on your
   installed version of GitLab:
   - *In GitLab versions 13.1 and greater,* navigate to **Settings > Operations**
     in your project.
   - *In GitLab versions prior to 13.1,* navigate to **Settings > Integrations**
     in your project. GitLab will display a banner encouraging you to enable
     the Alerts endpoint in **Settings > Operations** instead.
1. Click **Alerts endpoint**.
1. Toggle the **Active** alert setting to display the **URL** and **Authorization Key**
   for the webhook configuration.

## Customizing the payload

You can customize the payload by sending the following parameters. All fields
other than `title` are optional:

| Property                  | Type            | Description |
| ------------------------- | --------------- | ----------- |
| `title`                   | String          | The title of the incident. Required. |
| `description`             | String          | A high-level summary of the problem. |
| `start_time`              | DateTime        | The time of the incident. If none is provided, a timestamp of the issue will be used. |
| `end_time`                | DateTime        | For existing alerts only. When provided, the alert is resolved and the associated incident is closed. |
| `service`                 | String          | The affected service. |
| `monitoring_tool`         | String          |  The name of the associated monitoring tool. |
| `hosts`                   | String or Array | One or more hosts, as to where this incident occurred. |
| `severity`                | String          | The severity of the alert. Must be one of `critical`, `high`, `medium`, `low`, `info`, `unknown`. Default is `critical`. |
| `fingerprint`             | String or Array | The unique identifier of the alert. This can be used to group occurrences of the same alert. |
| `gitlab_environment_name` | String          | The name of the associated GitLab [environment](../../ci/environments/index.md). This can be used to associate your alert to your environment. |

You can also add custom fields to the alert's payload. The values of extra
parameters aren't limited to primitive types (such as strings or numbers), but
can be a nested JSON object. For example:

```json
{ "foo": { "bar": { "baz": 42 } } }
```

TIP: **Payload size:**
Ensure your requests are smaller than the [payload application limits](../../administration/instance_limits.md#generic-alert-json-payloads).

Example request:

```shell
curl --request POST \
  --data '{"title": "Incident title"}' \
  --header "Authorization: Bearer <authorization_key>" \
  --header "Content-Type: application/json" \
  <url>
```

The `<authorization_key>` and `<url>` values can be found when [setting up generic alerts](#setting-up-generic-alerts).

Example payload:

```json
{
  "title": "Incident title",
  "description": "Short description of the incident",
  "start_time": "2019-09-12T06:00:55Z",
  "service": "service affected",
  "monitoring_tool": "value",
  "hosts": "value",
  "severity": "high",
  "fingerprint": "d19381d4e8ebca87b55cda6e8eee7385",
  "foo": {
    "bar": {
      "baz": 42
    }
  }
}
```

## Triggering test alerts

> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3066) in GitLab Core in 13.2.

After a [project maintainer or owner](#setting-up-generic-alerts)
[configures generic alerts](#setting-up-generic-alerts), you can trigger a test
alert to confirm your integration works properly.

1. Sign in as a user with Developer or greater [permissions](../../user/permissions.md).
1. Navigate to **Settings > Operations** in your project.
1. Click **Alerts endpoint** to expand the section.
1. Enter a sample payload in **Alert test payload** (valid JSON is required).
1. Click **Test alert payload**.

GitLab displays an error or success message, depending on the outcome of your test.

## Automatic grouping of identical alerts **(PREMIUM)**

> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/214557) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.2.

In GitLab versions 13.2 and greater, GitLab groups alerts based on their
payload. When an incoming alert contains the same payload as another alert
(excluding the `start_time` and `hosts` attributes), GitLab groups these alerts
together and displays a counter on the [Alert Management List](./incidents.md)
and details pages.

If the existing alert is already `resolved`, GitLab creates a new alert instead.

![Alert Management List](./img/alert_list_v13_1.png)