From 48aff82709769b098321c738f3444b9bdaa694c6 Mon Sep 17 00:00:00 2001 From: GitLab Bot Date: Wed, 21 Oct 2020 07:08:36 +0000 Subject: Add latest changes from gitlab-org/gitlab@13-5-stable-ee --- doc/operations/incident_management/incidents.md | 240 ++++++++++++++++-------- 1 file changed, 158 insertions(+), 82 deletions(-) (limited to 'doc/operations/incident_management/incidents.md') diff --git a/doc/operations/incident_management/incidents.md b/doc/operations/incident_management/incidents.md index 3ff02b3dc6b..3d85fa0ebd8 100644 --- a/doc/operations/incident_management/incidents.md +++ b/doc/operations/incident_management/incidents.md @@ -4,16 +4,88 @@ group: Health info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- -# Create and manage incidents in GitLab +# Incidents -While no configuration is required to use the [manual features](#create-an-incident-manually) -of incident management, some simple [configuration](#configure-incidents) is needed to automate incident creation. +Incidents are critical entities in incident management workflows. They represent a service disruption or outage that needs to be restored urgently. GitLab provides tools for the triage, response, and remediation of incidents. -For users with at least Developer [permissions](../../user/permissions.md), the -Incident Management list is available at **Operations > Incidents** +## Incident Creation + +You can create an incident manually or automatically. + +### Create incidents manually + +If you have at least Guest [permissions](../../user/permissions.md), to create an Incident, you have two options to do this manually. + +**From the Incidents List:** + +> [Moved](https://gitlab.com/gitlab-org/monitor/health/-/issues/24) to GitLab core in 13.3. + +- Navigate to **Operations > Incidents** and click **Create Incident**. +- Create a new issue using the `incident` template available when creating it. +- Create a new issue and assign the `incident` label to it. + +![Incident List Create](./img/incident_list_create_v13_3.png) + +**From the Issues List:** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/230857) in GitLab 13.4. + +- Navigate to **Issues > List** and click **Create Issue**. +- Create a new issue using the `type` drop-down and select `Incident`. +- The page refreshes and the page only displays fields relevant to Incidents. + +![Incident List Create](./img/new_incident_create_v13_4.png) + +### Create incidents automatically + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/4925) in GitLab Ultimate 11.11. + +With Maintainer or higher [permissions](../../user/permissions.md), you can enable + GitLab to create incident automatically whenever an alert is triggered: + +1. Navigate to **Settings > Operations > Incidents** and expand + **Incidents**: + + ![Incident Management Settings](./img/incident_management_settings_v13_3.png) + +1. Check the **Create an incident** + checkbox. +1. To customize the incident, select an [issue templates](../../user/project/description_templates.md#creating-issue-templates). +1. To send [an email notification](alert_notifications.md#email-notifications) to users + with [Developer permissions](../../user/permissions.md), select + **Send a separate email notification to Developers**. Email notifications will also be sent to users with **Maintainer** and **Owner** permissions. +1. Click **Save changes**. + +### Create incidents via the PagerDuty webhook + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/119018) in GitLab 13.3. + +You can set up a webhook with PagerDuty to automatically create a GitLab incident +for each PagerDuty incident. This configuration requires you to make changes +in both PagerDuty and GitLab: + +1. Sign in as a user with Maintainer [permissions](../../user/permissions.md). +1. Navigate to **Settings > Operations > Incidents** and expand **Incidents**. +1. Select the **PagerDuty integration** tab: + + ![PagerDuty incidents integration](./img/pagerduty_incidents_integration_v13_3.png) + +1. Activate the integration, and save the changes in GitLab. +1. Copy the value of **Webhook URL** for use in a later step. +1. Follow the steps described in the + [PagerDuty documentation](https://support.pagerduty.com/docs/webhooks) + to add the webhook URL to a PagerDuty webhook integration. + +To confirm the integration is successful, trigger a test incident from PagerDuty to +confirm that a GitLab incident is created from the incident. + +## Incident list + +For users with at least Guest [permissions](../../user/permissions.md), the +Incident list is available at **Operations > Incidents** in your project's sidebar. The list contains the following metrics: -![Incident List](img/incident_list_v13_4.png) +![Incident List](img/incident_list_v13_5.png) - **Status** - To filter incidents by their status, click **Open**, **Closed**, or **All** above the incident list. @@ -27,8 +99,8 @@ in your project's sidebar. The list contains the following metrics: - **{severity-low}** **Low - S4** - **{severity-unknown}** **Unknown** - NOTE: **Note:** - Editing incident severity on the incident details page was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/229402) in GitLab 13.4. + [Editing incident severity](#change-severity) on the incident details page was + [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/229402) in GitLab 13.4. - **Incident** - The description of the incident, which attempts to capture the most meaningful data. @@ -44,113 +116,117 @@ The Incident list displays incidents sorted by incident created date. To see if a column is sortable, point your mouse at the header. Sortable columns display an arrow next to the column name. +Incidents share the [Issues API](../../user/project/issues/index.md). + TIP: **Tip:** For a live example of the incident list in action, visit this [demo project](https://gitlab.com/gitlab-examples/ops/incident-setup/everyone/tanuki-inc/-/incidents). -NOTE: **Note:** -Incidents share the [Issues API](../../user/project/issues/index.md). +## Incident details + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/230847) in GitLab 13.4. -## Configure incidents +Users with at least Reporter [permissions](../../user/permissions.md) can view +the Incident Details page. Navigate to **Operations > Incidents** in your project's +sidebar, and select an incident from the list. -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/4925) in GitLab Ultimate 11.11. +When you take any of these actions on an incident, GitLab logs a system note and +displays it in the Incident Details view: -With Maintainer or higher [permissions](../../user/permissions.md), you can enable -or disable Incident Management features in the GitLab user interface -to create issues when alerts are triggered: +- Updating the severity of an incident + ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/42358) in GitLab 13.5.) -1. Navigate to **Settings > Operations > Incidents** and expand - **Incidents**: +For live examples of GitLab incidents, visit the `tanuki-inc` project's +[incident list page](https://gitlab.com/gitlab-examples/ops/incident-setup/everyone/tanuki-inc/-/incidents). +Click any incident in the list to display its incident details page. - ![Incident Management Settings](./img/incident_management_settings_v13_3.png) +### Summary -1. For GitLab versions 11.11 and greater, you can select the **Create an issue** - checkbox to create an issue based on your own - [issue templates](../../user/project/description_templates.md#creating-issue-templates). - For more information, see - [Trigger actions from alerts](../metrics/alerts.md#trigger-actions-from-alerts) **(ULTIMATE)**. -1. To create issues from alerts, select the template in the **Issue Template** - select box. -1. To send [separate email notifications](index.md#notify-developers-of-alerts) to users - with [Developer permissions](../../user/permissions.md), select - **Send a separate email notification to Developers**. -1. Click **Save changes**. +The summary section for incidents provides both critical details about and the +contents of the issue template (if one was used). The highlighted bar at the top +of the incident displays from left to right: -Appropriately configured alerts include an -[embedded chart](../metrics/embed.md#embedding-metrics-based-on-alerts-in-incident-issues) -for the query corresponding to the alert. You can also configure GitLab to -[close issues](../metrics/alerts.md#trigger-actions-from-alerts) -when you receive notification that the alert is resolved. +- The link to the original alert. +- The alert start time. +- The event count. -## Create an incident manually +Beneath the highlight bar, GitLab displays a summary that includes the following fields: -If you have at least Developer [permissions](../../user/permissions.md), to create an Incident, you have two options. +- Start time +- Severity +- `full_query` +- Monitoring tool -### From the Incidents List +Comments are displayed in threads, but can be displayed chronologically +[in a timeline view](#timeline-view). -> [Moved](https://gitlab.com/gitlab-org/monitor/health/-/issues/24) to GitLab core in 13.3. +### Alert details -- Navigate to **Operations > Incidents** and click **Create Incident**. -- Create a new issue using the `incident` template available when creating it. -- Create a new issue and assign the `incident` label to it. +Incidents show the details of linked alerts in a separate tab. To populate this +tab, the incident must have been created with a linked alert. Incidents +created automatically from alerts have this +field populated. -![Incident List Create](./img/incident_list_create_v13_3.png) +![Incident alert details](./img/incident_alert_details_v13_4.png) -### From the Issues List +### Timeline view -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/230857) in GitLab 13.4. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/227836) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.5. -- Navigate to **Issues > List** and click **Create Issue**. -- Create a new issue using the `type` drop-down and select `Incident`. -- The page refreshes and the page only displays fields relevant to Incidents. +To quickly see the latest updates on an incident, click +**{comments}** **Turn timeline view on** in the comment bar to display comments +un-threaded and ordered chronologically, newest to oldest: -![Incident List Create](./img/new_incident_create_v13_4.png) +![Timeline view toggle](./img/timeline_view_toggle_v13_5.png) -## Configure PagerDuty integration +### Service Level Agreement countdown timer -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/119018) in GitLab 13.3. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/241663) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.5. -You can set up a webhook with PagerDuty to automatically create a GitLab issue -for each PagerDuty incident. This configuration requires you to make changes -in both PagerDuty and GitLab: +After enabling **Incident SLA** in the Incident Management configuration, newly-created +incidents display a SLA (Service Level Agreement) timer showing the time remaining before +the SLA period expires. If the incident is not closed before the SLA period ends, GitLab +adds a `missed::SLA` label to the incident. -1. Sign in as a user with Maintainer [permissions](../../user/permissions.md). -1. Navigate to **Settings > Operations > Incidents** and expand **Incidents**. -1. Select the **PagerDuty integration** tab: +## Incident Actions - ![PagerDuty incidents integration](./img/pagerduty_incidents_integration_v13_3.png) +There are different actions avilable to help triage and respond to incidents. -1. Activate the integration, and save the changes in GitLab. -1. Copy the value of **Webhook URL** for use in a later step. -1. Follow the steps described in the - [PagerDuty documentation](https://support.pagerduty.com/docs/webhooks) - to add the webhook URL to a PagerDuty webhook integration. +### Assign incidents -To confirm the integration is successful, trigger a test incident from PagerDuty to -confirm that a GitLab issue is created from the incident. +Assign incidents to users that are actively responding. Select **Edit** in the right-hand side bar to select or deselect assignees. -## Incident details +### Change severity -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/230847) in GitLab 13.4. +See [Incident List](#incident-list) for a full description of the severities available. Select **Edit** in the right-hand side bar to change the severity of an incident. -### Summary +### Add a to do -The summary section for incidents provides both critical details about and the -contents of the issue template (if one was used). The highlighted bar at the top -of the incident displays from left to right: the link to the original alert, the -alert start time, and the event count. Beneath the highlight bar, GitLab -displays a summary that includes the following fields: +Add a to-do for incidents that you want to track in your to-do list. Clicke the **Add a to do** button at the top of the right-hand side bar to add a to do. -- Start time -- Severity -- `full_query` -- Monitoring tool +### Manage incidents from Slack -### Alert details +Slack slash commands allow you to control GitLab and view GitLab content without leaving Slack. -Incidents show the details of linked alerts in a separate tab. To populate this -tab, the incident must have been created with a linked alert. Incidents -[created automatically](#configure-incidents) from alerts will have this -field populated. +Learn how to [set up Slack slash commands](../../user/project/integrations/slack_slash_commands.md) +and how to [use the available slash commands](../../integration/slash_commands.md). -![Incident alert details](./img/incident_alert_details_v13_4.png) +### Associate Zoom calls + +GitLab enables you to [associate a Zoom meeting with an issue](../../user/project/issues/associate_zoom_meeting.md) +for synchronous communication during incident management. After starting a Zoom +call for an incident, you can associate the conference call with an issue. Your +team members can join the Zoom call without requesting a link. + +### Embed metrics in incidents + +You can embed metrics anywhere [GitLab Markdown](../../user/markdown.md) is +used, such as descriptions, comments on issues, and merge requests. Embedding +metrics helps you share them when discussing incidents or performance issues. +You can output the dashboard directly into any issue, merge request, epic, or +any other Markdown text field in GitLab by +[copying and pasting the link to the metrics dashboard](../metrics/embed.md#embedding-gitlab-managed-kubernetes-metrics). + +You can embed both [GitLab-hosted metrics](../metrics/embed.md) and +[Grafana metrics](../metrics/embed_grafana.md) in incidents and issue +templates. -- cgit v1.2.1