path: root/doc/operations/error_tracking.md

---
stage: Monitor
group: Observability
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
---

# Error Tracking **(FREE)**

Error Tracking allows developers to discover and view errors generated by their application. Because error information is surfaced where the code is being developed, efficiency and awareness are increased.

## How error tracking works

For error tracking to work, you need two pieces:

- **Your application with Sentry SDK:** when the error happens, Sentry SDK captures information
  about it and sends it over the network to the backend. The backend stores information about all
  errors.

- **Error tracking backend:** the backend can be either GitLab itself or Sentry. When it's GitLab,
  we name it _integrated error tracking_ because you don't need to set up a separate backend. It's
  already part of the product.

  - To use the GitLab backend, see [integrated error tracking](#integrated-error-tracking).
  - To use Sentry as the backend, see [Sentry error tracking](#sentry-error-tracking).

  No matter what backend you choose, the [error tracking UI](#error-tracking-list)
  is the same.

## Sentry error tracking

[Sentry](https://sentry.io/) is an open source error tracking system. GitLab allows administrators to connect Sentry to GitLab so users can view a list of Sentry errors in GitLab.

### Deploying Sentry

You can sign up to the cloud-hosted [Sentry](https://sentry.io) or deploy your own [on-premise instance](https://github.com/getsentry/onpremise/).

### Enabling Sentry

GitLab provides a way to connect Sentry to your project. You need at
least Maintainer [permissions](../user/permissions.md) to enable the Sentry integration.

1. Sign up to Sentry.io or [deploy your own](#deploying-sentry) Sentry instance.
1. [Create](https://docs.sentry.io/product/sentry-basics/guides/integrate-frontend/create-new-project/) a new Sentry project. For each GitLab project that you want to integrate, we recommend that you create a new Sentry project.
1. Find or generate a [Sentry auth token](https://docs.sentry.io/api/auth/#auth-tokens).
   For the SaaS version of Sentry, you can find or generate the auth token at [https://sentry.io/api/](https://sentry.io/api/).
   Make sure to give the token at least the following scopes: `project:read`, `event:read`, and
   `event:write` (for resolving events).
1. In GitLab, enable error tracking:
   1. On the top bar, select **Main menu > Projects** and find your project.
   1. On the left sidebar, select **Monitor > Error Tracking**.
   1. Select **Enable error tracking**.
1. In GitLab, ensure error tracking is active.
   1. On the left sidebar, select **Settings > Monitor**.
   1. Expand **Error Tracking**.
   1. Ensure the **Active** checkbox is selected.
1. In the **Sentry API URL** box, enter your Sentry hostname. For example, enter `https://sentry.example.com`. For the SaaS version of Sentry, the hostname is `https://sentry.io`.
1. In the **Auth Token** box, enter the token you previously generated.
1. To test the connection to Sentry and populate the **Project** dropdown list, select **Connect**.
1. From the **Project** list, choose a Sentry project to link to your GitLab project.
1. Select **Save changes**.

You can now visit **Monitor > Error Tracking** in your project's sidebar to [view a list](#error-tracking-list) of Sentry errors.

### Enabling GitLab issues links

You may also want to enable Sentry's GitLab integration by following the steps in the [Sentry documentation](https://docs.sentry.io/product/integrations/gitlab/)

### Enable GitLab Runner

To configure GitLab Runner with Sentry, you must add the value for `sentry_dsn` to your GitLab
Runner's `config.toml` configuration file, as referenced in [GitLab Runner Advanced Configuration](https://docs.gitlab.com/runner/configuration/advanced-configuration.html).
While setting up Sentry, select **Go** if you're asked for the project type.

If you see the following error in your GitLab Runner logs, then you should specify the deprecated
DSN in **Sentry.io > Project Settings > Client Keys (DSN) > Show deprecated DSN**.

```plaintext
ERROR: Sentry failure builds=0 error=raven: dsn missing private key
```

## Error Tracking list

Users with at least Reporter [permissions](../user/permissions.md)
can find the Error Tracking list at **Monitor > Error Tracking** in your project's sidebar.
Here, you can filter errors by title or by status (one of Ignored , Resolved, or Unresolved) and sort in descending order by Frequency, First Seen, or Last Seen. By default, the error list is ordered by Last Seen and filtered to Unresolved errors.

![Error Tracking list](img/error_tracking_list_v12_6.png)

## Error details

From error list, users can go to the error details page by selecting the title of any error.

This page includes:

- A link to the Sentry issue.
- A link to the GitLab commit if the Sentry [release ID/version](https://docs.sentry.io/product/releases/?platform=javascript#configure-sdk) on the Sentry Issue's first release matches a commit SHA in your GitLab hosted project.
- Other details about the issue, including a full stack trace.
- In [GitLab 12.7 and newer](https://gitlab.com/gitlab-org/gitlab/-/issues/36246), language and urgency are displayed.

By default, a **Create issue** button is displayed:

![Error Details without Issue Link](img/error_details_v12_7.png)

If you create a GitLab issue from the error, the **Create issue** button changes to a **View issue**
button and a link to the GitLab issue displays in the error detail section.

## Taking action on errors

You can take action on Sentry Errors in the GitLab UI. Marking errors as ignored or resolved requires at least Developer role.

### Ignoring errors

> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/39665) in GitLab 12.7.

In the [Error Details](#error-details) page you can ignore a Sentry error by selecting **Ignore** near the top of the page.

Ignoring an error prevents it from appearing in the [Error Tracking List](#error-tracking-list), and silences notifications that were set up in Sentry.

### Resolving errors

> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/39825) in GitLab 12.7.

In the [Error Details](#error-details) page you can resolve a Sentry error by
selecting **Resolve** near the top of the page.

Marking an error as resolved indicates that the error has stopped firing events. If a GitLab issue is linked to the error, then the issue closes.

If another event occurs, the error reverts to unresolved.

## Integrated error tracking **(FREE SAAS)**

> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/329596) in GitLab 14.4.
> - [Disabled](https://gitlab.com/gitlab-org/gitlab/-/issues/353639) in GitLab 14.9 [with a flag](../administration/feature_flags.md) named `integrated_error_tracking`. Disabled by default.
> - [Enabled on GitLab.com](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7586) in GitLab 15.6.

NOTE:
Available only on GitLab.com. This feature is in [Open Beta](https://about.gitlab.com/handbook/product/gitlab-the-product/#open-beta).

### Known limitations

Only basic support is provided with `capture_exception` as the holding method.
Additional features requests (see this [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/340178)) are added on a case-by-case basis.

### Debugging issues

The majority of languages are supported by Sentry expose `debug` option as part of initialization.
It is also possible to output JSON before it is sent to the API.
See the [Go example](#go) below for a suggested solution.

### Enabling error tracking

Regardless of the programming language, you need to enable error tracking for your project. This doc assumes you already have a project for which you want to enable error tracking.
This example uses the `gitlab.com` instance.

To enable error tracking, follow these steps:

1. On the top bar, select **Main menu > Projects** and find your project.
1. On the left sidebar, select **Settings > Monitor**.
1. Expand the **Error Tracking** tab.
1. Under **Enable error tracking**, select the **Active** checkbox.
1. Under **Error tracking backend**, select **GitLab**.
1. Select **Save changes**.
1. Copy the DSN string to use later.

### Listing captured errors

Once your application has emitted errors to the Error Tracking API through the Sentry SDK, they should be available under **Monitor > Error Tracking** tab/section.

For more details, refer to the [GitLab error tracking documentation](https://gitlab.com/help/operations/error_tracking#error-tracking-list).

This process assumes the GDK feature flag `integrated_error_tracking` is enabled. If you are running GDK locally and you do not see the option for error tracking, you can enable it by running the following commands:

```linux
cd <PATH_TO_GDK>
gdk rails console
Feature.enable(:integrated_error_tracking)
```

### Emitting errors

#### Supported Sentry types

According to the [data model](https://develop.sentry.dev/sdk/envelopes/#data-model), the available item types are:

- [Event](https://develop.sentry.dev/sdk/event-payloads/)
- [Transactions](https://develop.sentry.dev/sdk/event-payloads/transaction/)
- Attachment
- [Session](https://develop.sentry.dev/sdk/sessions/)
- [Sessions](https://develop.sentry.dev/sdk/sessions/)
- [User feedback](https://develop.sentry.dev/sdk/envelopes/#user-feedback) (also known as user report)
- [Client report](https://develop.sentry.dev/sdk/client-reports/)

Items of various types can be sent to the error tracking app, using either the Store endpoint, the envelope endpoint, or both. The following table lists all event types available through Sentry SDK. It also explains which endpoint can be used for ingestion and whether it is supported by GitLab Observability Backend.

Event item types can contain various interfaces, such as exception, message, stack trace, and template. You can read more about the core data interfaces in [Sentry documentation](https://develop.sentry.dev/sdk/event-payloads/#core-interfaces).

| Item type       | Interface   | Can be sent through the Store endpoint | Can be sent through the Envelope endpoint | Supported              |
| --------------- | ----------- | -------------------------------------- | ----------------------------------------- | ---------------------- |
| `event`         | exception   | **{check-circle}** Yes                 | **{check-circle}** Yes                    | **{check-circle}** Yes |
| `event`         | message     | **{check-circle}** Yes                 | **{check-circle}** Yes                    | **{check-circle}** Yes |
| `event`         | stack trace | **{check-circle}** Yes                 | **{check-circle}** Yes                    | **{check-circle}** Yes |
| `event`         | template    | **{check-circle}** Yes                 | **{check-circle}** Yes                    | **{dotted-circle}** No |
| `transaction`   | N/A         | **{dotted-circle}** No                 | **{check-circle}** Yes                    | **{dotted-circle}** No |
| `attachment`    | N/A         | **{dotted-circle}** No                 | **{check-circle}** Yes                    | **{dotted-circle}** No |
| `session`       | N/A         | **{dotted-circle}** No                 | **{check-circle}** Yes                    | **{dotted-circle}** No |
| `sessions`      | N/A         | **{dotted-circle}** No                 | **{check-circle}** Yes                    | **{dotted-circle}** No |
| `user_report`   | N/A         | **{dotted-circle}** No                 | **{check-circle}** Yes                    | **{dotted-circle}** No |
| `client_report` | N/A         | **{dotted-circle}** No                 | **{check-circle}** Yes                    | **{dotted-circle}** No |

#### Supported languages

Each language shows a basic example of how to capture exceptions with the respective SDK.
For more in-depth documentation, see [documentation for Sentry SDK](https://docs.sentry.io/). You can also find information for additional programming languages.

Only a subset of languages is supported.

The following table lists them:

| Sentry SDK  | Supported?  |
| ----------- | ----------- |
| Ruby        | Yes |
| Go          | Yes |
| JavaScript  | Yes |
| Java        | Yes |
| Python      | Yes |
| PHP         | Yes |
| .NET        | Not tested |
| Android     | Not tested |
| Apple       | Not tested |
| Perl        | Not tested |

A more up-to-date version of [this matrix can be found in this doc](https://gitlab.com/gitlab-org/opstrace/opstrace/-/issues/1737).

#### Go

1. `chdir` into folder `docs/guides/user/error_tracking_examples/go/`
1. Install the dependencies with the following command:

   ```shell
   go mod tidy
   ```

1. Run the following command:

   ```shell
   export SENTRY_DSN="<DSN string>"
   go run main.go <DSN string>
   ```

After you've run this program, there should be an error visible in the Error tracking tab from `Listing captured errors` section of this document.

#### Ruby

1. `chdir` into folder `docs/guides/user/error_tracking_examples/ruby/`
1. Install the dependencies with the following command:

   ```shell
   gem install bundler
   bundle install
   ```

1. Execute the example with the following command:

   ```shell
   export SENTRY_DSN="<DSN string>"
   ruby app.rb
   ```

After you've run this program, there should be an error visible in the Error tracking tab from `Listing captured errors` section of this document.

#### PHP

1. `chdir` into folder `docs/guides/user/error_tracking_examples/php/`

1. Build and run the Docker container with the following commands:

```shell
export SENTRY_DSN="<DSN string>"
docker build -t sentry-php .
docker run -e SENTRY_DSN --rm sentry-php
```

After you've run this program, there should be an error visible in the Error tracking tab from `Listing captured errors` section of this document.

#### Python

1. `chdir` into folder `docs/guides/user/error_tracking_examples/python/`

1. Install the dependencies with the following commands:

   ```shell
   virtualenv env
   source env/bin/activate
   pip install -r requirements.txt
   ```

1. Run the following commands:

```shell
export SENTRY_DSN="<DSN string>"
python send_exception.py
```

After you've run this program, there should be an error visible in the Error tracking tab from `Listing captured errors` section of this document.

#### Java

1. `chdir` into folder `docs/guides/user/error_tracking_examples/python/`

1. Run the following command:

```shell
export SENTRY_DSN="<DSN string>"
./gradlew run
```

#### Node.js

1. `chdir` into folder `docs/guides/user/error_tracking_examples/nodejs/`

1. Install the dependencies with the following command:

   ```shell
   npm install --save @sentry/node @sentry/tracing
   ```

1. Run the following command:

   ```shell
   export SENTRY_DSN="<DSN string>"
   node ./test.js
   ```

After you've run this program, there should be an error visible in the Error tracking tab from `Listing captured errors` section of this document.

### Rotating Sentry DSN

The Sentry DSN (client key) is a secret and it should not be exposed to the public. If it's leaked, you can rotate the Sentry DSN with the following steps:

1. [Create an access token](../user/profile/personal_access_tokens.md#create-a-personal-access-token) by clicking your profile picture in GitLab.com. Then choose Preferences,then Access Token. Make sure you add the API scope.
1. Using the [error tracking API](../api/error_tracking.md), create a new Sentry DSN with the following command:

   ```shell
   curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --header "Content-Type: application/json" \
     "https://gitlab.example.com/api/v4/projects/<your_project_number>/error_tracking/client_keys"
   ```

1. Get the available client keys (Sentry DSNs). Ensure that the newly created Sentry DSN is in place. Then note down the key ID of the old client key:

   ```shell
   curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/<your_project_number>/error_tracking/client_keys"
   ```

1. Delete the old client key with the following command:

   ```shell
   curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/<your_project_number>/error_tracking/client_keys/<key_id>"
   ```