1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
|
---
description: "GitLab - Incident Management. GitLab offers solutions for handling incidents in your applications and services"
---
# Incident Management
GitLab offers solutions for handling incidents in your applications and services,
from setting up an alert with Prometheus, to receiving a notification via a
monitoring tool like Slack, and automatically setting up Zoom calls with your
support team.
## Configuring incidents **(ULTIMATE)**
> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/4925) in GitLab Ultimate 11.11.
The Incident Management features can be enabled and disabled via your project's
**Settings > Operations > Incidents**.
### Automatically create issues from alerts
GitLab issues can automatically be created as a result of an alert notification.
An issue created this way will contain the error information to help you further
debug it.
### Issue templates
You can create your own [issue templates](../project/description_templates.md#creating-issue-templates)
that can be [used within Incident Management](../project/integrations/prometheus.md#taking-action-on-incidents-ultimate).
To select your issue template for use within Incident Management:
1. Visit your project's **Settings > Operations > Incidents**.
1. Select the template from the **Issue Template** dropdown.
## Alerting
GitLab can react to the alerts that your applications and services may be
triggering by automatically creating issues, and alerting developers via email.
The emails will be sent to [owners and maintainers](../permissions.md) of the project and will contain details on the alert as well as a link to see more information.
### Prometheus alerts
Prometheus alerts can be set up in both:
- [GitLab-managed Prometheus](../project/integrations/prometheus.md#setting-up-alerts-for-prometheus-metrics-ultimate) and
- [Self-managed Prometheus](../project/integrations/prometheus.md#external-prometheus-instances) installations.
### Alert endpoint
GitLab can accept alerts from any source via a generic webhook receiver. When
you set up the generic alerts integration, a unique endpoint will
be created which can receive a payload in JSON format.
[Read more on setting this up, including how to customize the payload](../project/integrations/generic_alerts.md).
### Recovery alerts
GitLab can [automatically close issues](../project/integrations/prometheus.md#taking-action-on-incidents-ultimate)
that have been automatically created when you receive notification that the
alert is resolved.
## Embedded metrics
Metrics can be embedded anywhere where GitLab Markdown is used, for example,
descriptions and comments on issues and merge requests.
This can be useful for when you're sharing metrics, such as for discussing
an incident or performance issues, so you can output the dashboard directly
into any issue, merge request, epic, or any other Markdown text field in GitLab
by simply [copying and pasting the link to the metrics dashboard](../project/integrations/prometheus.md#embedding-gitlab-managed-kubernetes-metrics).
TIP: **Tip:**
Both GitLab-hosted and Grafana metrics can also be
[embedded in issue templates](../project/integrations/prometheus.md#embedding-metrics-in-issue-templates).
### GitLab-hosted metrics
Learn how to embed [GitLab hosted metric charts](../project/integrations/prometheus.md#embedding-metric-charts-within-gitlab-flavored-markdown).
#### Context menu
From each of the embedded metrics panels, you can access more details
about the data you are viewing from a context menu.
You can access the context menu by clicking the **{ellipsis_v}** **More actions**
dropdown box above the upper right corner of the panel:
The options are:
- [View logs](#view-logs)
- [Download CSV](#download-csv)
##### View logs
> - [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/201846) in GitLab Ultimate 12.8.
> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/25455) to [GitLab Core](https://about.gitlab.com/pricing/) 12.9.
This can be useful if you are triaging an application incident and need to
[explore logs](../project/integrations/prometheus.md#view-logs-ultimate)
from across your application. It also helps you to understand
what is affecting your application's performance and quickly resolve any problems.
##### Download CSV
Data from embedded charts can be [downloaded as CSV](../project/integrations/prometheus.md#downloading-data-as-csv).
### Grafana metrics
Learn how to embed [Grafana hosted metric charts](../project/integrations/prometheus.md#embedding-grafana-charts).
## Slack integration
Slack slash commands allow you to control GitLab and view content right inside
Slack, without having to leave it.
Learn how to [set up Slack slash commands](../project/integrations/slack_slash_commands.md)
and how to [use them](../../integration/slash_commands.md).
### Slash commands
Please refer to a list of [available slash commands](../../integration/slash_commands.md) and associated descriptions.
## Zoom in issues
In order to communicate synchronously for incidents management, GitLab allows you to
associate a Zoom meeting with an issue. Once you start a Zoom call for a fire-fight,
you need a way to associate the conference call with an issue, so that your team
members can join swiftly without requesting a link.
Read more how to [add or remove a zoom meeting](../project/issues/associate_zoom_meeting.md).
### Configuring Incidents
Incident Management features can be easily enabled & disabled via the Project settings page. Head to Project -> Settings -> Operations -> Incidents.
#### Auto-creation
You can automatically create GitLab issues from an Alert notification. Issues created this way contain error information to help you debug the error. Appropriately configured alerts include an [embedded chart](../project/integrations/prometheus.md#embedding-metrics-based-on-alerts-in-incident-issues) for the query corresponding to the alert.
|
