diff options
Diffstat (limited to 'doc/operations/incident_management/alerts.md')
| -rw-r--r-- | doc/operations/incident_management/alerts.md | 300 |
1 files changed, 221 insertions, 79 deletions
diff --git a/doc/operations/incident_management/alerts.md b/doc/operations/incident_management/alerts.md index d908af63000..a6168386024 100644 --- a/doc/operations/incident_management/alerts.md +++ b/doc/operations/incident_management/alerts.md @@ -4,119 +4,261 @@ group: Health info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- -# Create and manage alerts in GitLab +# Alerts -Users with at least Developer [permissions](../../user/permissions.md) can access -the Alert Management list at **{cloud-gear}** **Operations > Alerts** in your -project's sidebar. The Alert Management list displays alerts sorted by start time, -but you can change the sort order by clicking the headers in the Alert Management list. +Alerts are a critical entity in your incident managment workflow. They represent a notable event that might indicate a service outage or disruption. GitLab provides a list view for triage and detail view for deeper investigation of what happened. + +## Alert List + +Users with at least Developer [permissions](../../user/permissions.md) can +access the Alert list at **Operations > Alerts** in your project's +sidebar. The Alert list displays alerts sorted by start time, but +you can change the sort order by clicking the headers in the Alert list. ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217745) in GitLab 13.1.) The alert list displays the following information: - + -- **Search** - The alert list supports a simple free text search on the title, +- **Search**: The alert list supports a simple free text search on the title, description, monitoring tool, and service fields. ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/213884) in GitLab 13.1.) -- **Severity** - The current importance of a alert and how much attention it should - receive. For a listing of all statuses, read [Alert Management severity](#alert-severity). -- **Start time** - How long ago the alert fired. This field uses the standard - GitLab pattern of `X time ago`, but is supported by a granular date/time tooltip - depending on the user's locale. -- **Alert description** - The description of the alert, which attempts to capture the most meaningful data. -- **Event count** - The number of times that an alert has fired. -- **Issue** - A link to the incident issue that has been created for the alert. -- **Status** - The current status of the alert: +- **Severity**: The current importance of a alert and how much attention it + should receive. For a listing of all statuses, read [Alert Management severity](#alert-severity). +- **Start time**: How long ago the alert fired. This field uses the standard + GitLab pattern of `X time ago`, but is supported by a granular date/time + tooltip depending on the user's locale. +- **Alert description**: The description of the alert, which attempts to + capture the most meaningful data. +- **Event count**: The number of times that an alert has fired. +- **Issue**: A link to the incident issue that has been created for the alert. +- **Status**: The current status of the alert: - **Triggered**: No one has begun investigation. - **Acknowledged**: Someone is actively investigating the problem. - **Resolved**: No further work is required. - + TIP: **Tip:** -Check out a [live example](https://gitlab.com/gitlab-examples/ops/incident-setup/everyone/tanuki-inc/-/alert_management) +Check out a live example available from the +[`tanuki-inc` project page](https://gitlab-examples-ops-incident-setup-everyone-tanuki-inc.34.69.64.147.nip.io/) in GitLab to examine alerts in action. -## Enable Alerts +## Alert severity + +Each level of alert contains a uniquely shaped and color-coded icon to help +you identify the severity of a particular alert. These severity icons help you +immediately identify which alerts you should prioritize investigating: + + -NOTE: **Note:** -You need at least Maintainer [permissions](../../user/permissions.md) to enable -the Alerts feature. +Alerts contain one of the following icons: -There are several ways to accept alerts into your GitLab project. -Enabling any of these methods enables the Alert list. After configuring -alerts, visit **{cloud-gear}** **Operations > Alerts** in your project's sidebar -to view the list of alerts. +| Severity | Icon | Color (hexadecimal) | +|----------|-------------------------|---------------------| +| Critical | **{severity-critical}** | `#8b2615` | +| High | **{severity-high}** | `#c0341d` | +| Medium | **{severity-medium}** | `#fca429` | +| Low | **{severity-low}** | `#fdbc60` | +| Info | **{severity-info}** | `#418cd8` | +| Unknown | **{severity-unknown}** | `#bababa` | -### Enable GitLab-managed Prometheus alerts +## Alert details page -You can install the GitLab-managed Prometheus application on your Kubernetes -cluster. For more information, read -[Managed Prometheus on Kubernetes](../../user/project/integrations/prometheus.md#managed-prometheus-on-kubernetes). -When GitLab-managed Prometheus is installed, the [Alerts list](alerts.md) -is also enabled. +Navigate to the Alert details view by visiting the [Alert list](./alerts.md) +and selecting an alert from the list. You need least Developer [permissions](../../user/permissions.md) +to access alerts. -To populate the alerts with data, read -[GitLab-Managed Prometheus instances](../metrics/alerts.md#managed-prometheus-instances). +TIP: **Tip:** +To review live examples of GitLab alerts, visit the +[alert list](https://gitlab.com/gitlab-examples/ops/incident-setup/everyone/tanuki-inc/-/alert_management) +for this demo project. Select any alert in the list to examine its alert details +page. -### Enable external Prometheus alerts +Alerts provide **Overview** and **Alert details** tabs to give you the right +amount of information you need. -You can configure an externally-managed Prometheus instance to send alerts -to GitLab. To set up this configuration, read the [configuring Prometheus](../metrics/alerts.md#external-prometheus-instances) documentation. Activating the external Prometheus -configuration also enables the [Alerts list](./alerts.md). +### Alert details tab -To populate the alerts with data, read -[External Prometheus instances](../metrics/alerts.md#external-prometheus-instances). +The **Alert details** tab has two sections. The top section provides a short list of critical details such as the severity, start time, number of events, and originating monitorting tool. The second section displays the full alert payload. -### Enable a Generic Alerts endpoint +### Metrics tab -GitLab provides the Generic Alerts endpoint so you can accept alerts from a third-party -alerts service. Read the -[instructions for toggling generic alerts](generic_alerts.md#setting-up-generic-alerts) -to add this option. After configuring the endpoint, the -[Alerts list](./alerts.md) is enabled. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217768) in GitLab 13.2. -To populate the alerts with data, read [Customizing the payload](./generic_alerts.md#customizing-the-payload) for requests to the alerts endpoint. +The **Metrics** tab will display a metrics chart for alerts coming from Prometheus. If the alert originated from any other tool, the **Metrics** tab will be empty. To set up alerts for GitLab-managed Prometheus instances, see [Managed Prometheus instances](../metrics/alerts.md#managed-prometheus-instances). For externally-managed Prometheus instances, you will need to configure your alerting +rules to display a chart in the alert. For information about how to configure +your alerting rules, see [Embedding metrics based on alerts in incident issues](../metrics/embed.md#embedding-metrics-based-on-alerts-in-incident-issues). See +[External Prometheus instances](../metrics/alerts.md#external-prometheus-instances) +for information about setting up alerts for your self-managed Prometheus +instance. -### Opsgenie integration **(PREMIUM)** +To view the metrics for an alert: -> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3066) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.2. +1. Sign in as a user with Developer or higher [permissions](../../user/permissions.md). +1. Navigate to **Operations > Alerts**. +1. Select the alert you want to view. +1. Below the title of the alert, select the **Metrics** tab. -A new way of monitoring Alerts via a GitLab integration is with -[Opsgenie](https://www.atlassian.com/software/opsgenie). + -NOTE: **Note:** -If you enable the Opsgenie integration, you can't have other GitLab alert services, -such as [Generic Alerts](./generic_alerts.md) or -Prometheus alerts, active at the same time. +#### View an alert's logs -To enable Opsgenie integration: +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/201846) in GitLab Ultimate 12.8. and [improved](https://gitlab.com/gitlab-org/gitlab/-/issues/217768) in GitLab 13.3. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/25455) to [GitLab Core](https://about.gitlab.com/pricing/) 12.9. -1. Sign in as a user with Maintainer or Owner [permissions](../../user/permissions.md). -1. Navigate to **{cloud-gear}** **Operations > Alerts**. -1. In the **Integrations** select box, select Opsgenie. -1. Click the **Active** toggle. -1. In the **API URL**, enter the base URL for your Opsgenie integration, such - as `https://app.opsgenie.com/alert/list`. -1. Click **Save changes**. +Viewing logs from a metrics panel can be useful if you're triaging an +application incident and need to [explore logs](../metrics/dashboards/index.md#chart-context-menu) +from across your application. These logs help you understand what's affecting +your application's performance and how to resolve any problems. -After enabling the integration, navigate to the Alerts list page at -**{cloud-gear}** **Operations > Alerts**, and click **View alerts in Opsgenie**. +To view the logs for an alert: -## Alert severity +1. Sign in as a user with Developer or higher [permissions](../../user/permissions.md). +1. Navigate to **Operations > Alerts**. +1. Select the alert you want to view. +1. Below the title of the alert, select the **Metrics** tab. +1. Select the [menu](../metrics/dashboards/index.md#chart-context-menu) of + the metric chart to view options. +1. Select **View logs**. -Each level of alert contains a uniquely shaped and color-coded icon to help -you identify the severity of a particular alert. These severity icons help you -immediately identify which alerts you should prioritize investigating: +### Activity feed tab - +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3066) in GitLab 13.1. -Alerts contain one of the following icons: +The **Activity feed** tab is a log of activity on the alert. When you take action on an alert, this is logged as a system note. This gives you a linear +timeline of the alert's investigation and assignment history. + +The following actions will result in a system note: + +- [Updating the status of an alert](#update-an-alerts-status) +- [Creating an incident based on an alert](#create-an-incident-from-an-alert) +- [Assignment of an alert to a user](#assign-an-alert) + + + +## Alert actions + +There are different actions avilable in GitLab to help triage and respond to alerts. + +### Update an alert's status + +The Alert detail view enables you to update the Alert Status. +See [Create and manage alerts in GitLab](./alerts.md) for more details. + +### Create an incident from an alert + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217745) in GitLab 13.1. + +The Alert detail view enables you to create an issue with a +description populated from an alert. To create the issue, +select the **Create Issue** button. You can then view the issue from the +alert by selecting the **View Issue** button. + +Closing a GitLab issue associated with an alert changes the alert's status to +Resolved. See [Create and manage alerts in GitLab](alerts.md) for more details +about alert statuses. + +### Assign an alert + +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3066) in GitLab 13.1. + +In large teams, where there is shared ownership of an alert, it can be +difficult to track who is investigating and working on it. Assigning alerts eases collaboration and delegation by indicating which user is owning the alert. GitLab supports only a single assignee per alert. + +To assign an alert: + +1. To display the list of current alerts, navigate to **Operations > Alerts**: + +  + +1. Select your desired alert to display its **Alert Details View**: + +  + +1. If the right sidebar is not expanded, select + **{angle-double-right}** **Expand sidebar** to expand it. +1. In the right sidebar, locate the **Assignee**, and then select **Edit**. + From the dropdown menu, select each user you want to assign to the alert. + GitLab creates a [to-do item](../../user/todos.md) for each user. + +  + +After completing their portion of investigating or fixing the alert, users can +unassign themselves from the alert. To remove an assignee, select **Edit** next to the **Assignee** dropdown menu +and deselect the user from the list of assignees, or select **Unassigned**. + +### Create a to do from an alert + +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3066) in GitLab 13.1. + +You can manually create [To-Do list items](../../user/todos.md) for yourself +from the Alert details screen, and view them later on your **To-Do List**. To +add a to do: + +1. To display the list of current alerts, navigate to **Operations > Alerts**. +1. Select your desired alert to display its **Alert Management Details View**. +1. Select the **Add a To-Do** button in the right sidebar: + +  + +Select the **To-Do List** **{todo-done}** in the navigation bar to view your current to-do list. + + + +## Link runbooks to alerts + +> Runbook URLs [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/39315) in GitLab 13.3. + +When creating alerts from the metrics dashboard for +[managed Prometheus instances](../metrics/alerts.md#managed-prometheus-instances), +you can link a runbook. When the alert triggers, you can access the runbook through +the [chart context menu](../metrics/dashboards/index.md#chart-context-menu) in the +upper-right corner of the metrics chart, making it easy for you to locate and access +the correct runbook: + + + +## View the environment that generated the alert + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/232492) in GitLab 13.5. +> - It's [deployed behind a feature flag](../../user/feature_flags.md), disabled by default. +> - It's disabled on GitLab.com. +> - It's not recommended for production use. +> - To use it in GitLab self-managed instances, ask a GitLab administrator to [enable it](#enable-or-disable-environment-link-in-alert-details). **(CORE ONLY)** + +CAUTION: **Warning:** +This feature might not be available to you. Check the **version history** note above for details. + +The environment information and the link are displayed in the [Alert Details tab](#alert-details-tab). + +### Enable or disable Environment Link in Alert Details **(CORE ONLY)** + +Viewing the environment is under development and not ready for production use. It is +deployed behind a feature flag that is **disabled by default**. +[GitLab administrators with access to the GitLab Rails console](../../administration/feature_flags.md) +can enable it. + +To enable it: + +```ruby +Feature.enable(:expose_environment_path_in_alert_details) +``` + +To enable for just a particular project: + +```ruby +project = Project.find_by_full_path('your-group/your-project') +Feature.enable(:expose_environment_path_in_alert_details, project) +``` + +To disable it: + +```ruby +Feature.disable(:expose_environment_path_in_alert_details) +``` + +To disable for just a particular project: -| Severity | Icon | Color (hexadecimal) | -|---|---|---| -| Critical | **{severity-critical}** | `#8b2615` | -| High | **{severity-high}** | `#c0341d` | -| Medium | **{severity-medium}** | `#fca429` | -| Low | **{severity-low}** | `#fdbc60` | -| Info | **{severity-info}** | `#418cd8` | -| Unknown | **{severity-unknown}** | `#bababa` | +```ruby +project = Project.find_by_full_path('your-group/your-project') +Feature.disable(:expose_environment_path_in_alert_details, project) +``` |