diff options
Diffstat (limited to 'doc/administration/monitoring')
6 files changed, 125 insertions, 89 deletions
diff --git a/doc/administration/monitoring/gitlab_self_monitoring_project/index.md b/doc/administration/monitoring/gitlab_self_monitoring_project/index.md index b1ec74c2f40..f8764468256 100644 --- a/doc/administration/monitoring/gitlab_self_monitoring_project/index.md +++ b/doc/administration/monitoring/gitlab_self_monitoring_project/index.md @@ -4,43 +4,44 @@ group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# GitLab self monitoring project **(FREE SELF)** +# Self monitoring project **(FREE SELF)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/32351) in GitLab 12.7, behind a disabled feature flag (`self_monitoring_project`). -> - The feature flag was removed and the Self Monitoring Project was [made generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/198511) in GitLab 12.8. +> - The feature flag was removed and the self monitoring project was [made generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/198511) in GitLab 12.8. -GitLab has been adding the ability for administrators to see insights into the -health of their GitLab instance. To surface this experience in a native way -(similar to how you would interact with an application deployed using GitLab), -a base project called "GitLab self monitoring" with -[internal visibility](../../../public_access/public_access.md#internal-projects) -is added under a group called "GitLab Instance Administrators" -specifically created for visualizing and configuring the monitoring of your -GitLab instance. +GitLab provides administrators insights into the health of their GitLab instance. -All administrators at the time of creation of the project and group are -added as maintainers of the group and project, and as an administrator, you can -add new members to the group to give them the [Maintainer role](../../../user/permissions.md) for -the project. +To provide a native experience (similar interacting with an application deployed using GitLab), a +project called **Monitoring** is created: -This project is used to self monitor your GitLab instance. The metrics dashboard -of the project shows some basic resource usage charts, such as CPU and memory usage -of each server in [Omnibus GitLab](https://docs.gitlab.com/omnibus/) installations. +- With [internal visibility](../../../public_access/public_access.md#internal-projects). +- Under a group called **GitLab Instance**. -You can also use the project to configure your own -[custom metrics](../../../operations/metrics/index.md#adding-custom-metrics) using -metrics exposed by the [GitLab exporter](../prometheus/gitlab_metrics.md#metrics-available). +The project is created specifically for visualizing and configuring the monitoring of your GitLab +instance. -## Creating the self monitoring project +When the project and group are created, all administrators are added as maintainers. As an +administrator, you can add new members to the group to give them the +[Maintainer role](../../../user/permissions.md) for the project. + +This project can be used to: + +- Self monitor your GitLab instance. The metrics dashboard of the project shows some basic resource + usage charts, such as CPU and memory usage of each server in + [Omnibus GitLab](https://docs.gitlab.com/omnibus/) installations. +- Also configure your own [custom metrics](../../../operations/metrics/index.md#adding-custom-metrics) + using metrics exposed by the [GitLab exporter](../prometheus/gitlab_metrics.md#metrics-available). + +## Create the self monitoring project 1. On the top bar, select **Menu >** **{admin}** **Admin**. 1. On the left sidebar, select **Settings > Metrics and profiling** and expand **Self monitoring**. -1. Toggle the **Create Project** button on. +1. Toggle **Self monitoring** on. 1. After your GitLab instance creates the project, GitLab displays a link to the - project in the text above the **Create Project** toggle. You can also find it + project in the text above the **Self monitoring** toggle. You can also find it from the top bar by selecting **Menu > Project**, then selecting **Your projects**. -## Deleting the self monitoring project +## Delete the self monitoring project WARNING: Deleting the self monitoring project removes any changes made to the project. If @@ -48,8 +49,8 @@ you create the project again, it's created in its default state. 1. On the top bar, select **Menu >** **{admin}** **Admin**. 1. On the left sidebar, go to **Settings > Metrics and profiling** and expand **Self monitoring**. -1. Toggle the **Create Project** button off. -1. In the confirmation dialog that opens, click **Delete project**. +1. Toggle **Self monitoring** off. +1. In the confirmation dialog that opens, click **Delete self monitoring project**. It can take a few seconds for it to be deleted. 1. After the project is deleted, GitLab displays a message confirming your action. @@ -66,27 +67,24 @@ The dashboard uses metrics available in You can also [create your own dashboards](../../../operations/metrics/dashboards/index.md). -## Connection to Prometheus +## Connect to Prometheus The project is automatically configured to connect to the -[internal Prometheus](../prometheus/index.md) instance if the Prometheus -instance is present (should be the case if GitLab was installed via Omnibus -and you haven't disabled it). +[internal Prometheus](../prometheus/index.md) instance if the Prometheus instance is present. +This should be the case if GitLab was installed using Omnibus GitLab and you haven't disabled it. -If that's not the case or if you have an external Prometheus instance or a customized setup, -you should -[configure it manually](../../../user/project/integrations/prometheus.md#manual-configuration-of-prometheus). +If that's not the case, or if you have an external Prometheus instance or a customized setup, +you [configure it manually](../../../user/project/integrations/prometheus.md#manual-configuration-of-prometheus). -## Taking action on Prometheus alerts **(ULTIMATE)** +## Take action on Prometheus alerts **(ULTIMATE)** You can [add a webhook](../../../operations/metrics/alerts.md#external-prometheus-instances) -to the Prometheus configuration for GitLab to receive notifications of any -alerts. +to the Prometheus configuration for GitLab to receive notifications of any alerts. Once the webhook is setup, you can [take action on incoming alerts](../../../operations/metrics/alerts.md#trigger-actions-from-alerts). -## Adding custom metrics to the self monitoring project +## Add custom metrics to the self monitoring project You can add custom metrics in the self monitoring project by: @@ -95,11 +93,10 @@ You can add custom metrics in the self monitoring project by: ## Troubleshooting -### Getting error message in logs: `Could not create instance administrators group. Errors: ["You don't have permission to create groups."]` +### Error message in logs: `Could not create instance administrators group. Errors: ["You don't have permission to create groups."]` -There is [a bug](https://gitlab.com/gitlab-org/gitlab/-/issues/208676) which causes -project creation to fail with the following error (which appears in the log file) -when the first administrator user is an +A [bug](https://gitlab.com/gitlab-org/gitlab/-/issues/208676) causes project creation to fail with +the following error in the log file when the first administrator user is an [external user](../../../user/permissions.md#external-users): ```plaintext diff --git a/doc/administration/monitoring/ip_whitelist.md b/doc/administration/monitoring/ip_whitelist.md index 20c97a0df8f..b6df176ea87 100644 --- a/doc/administration/monitoring/ip_whitelist.md +++ b/doc/administration/monitoring/ip_whitelist.md @@ -9,7 +9,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w > Introduced in GitLab 9.4. NOTE: -We intend to [rename IP whitelist as `IP allowlist`](https://gitlab.com/gitlab-org/gitlab/-/issues/7554). +We intend to [rename IP whitelist as `IP allowlist`](https://gitlab.com/groups/gitlab-org/-/epics/3478). GitLab provides some [monitoring endpoints](../../user/admin_area/monitoring/health_check.md) that provide health check information when probed. diff --git a/doc/administration/monitoring/performance/grafana_configuration.md b/doc/administration/monitoring/performance/grafana_configuration.md index 3b82b0e4bb8..0f40fd3c0e7 100644 --- a/doc/administration/monitoring/performance/grafana_configuration.md +++ b/doc/administration/monitoring/performance/grafana_configuration.md @@ -65,15 +65,41 @@ GitLab sidebar: 1. On the top bar, select **Menu >** **{admin}** **Admin**. 1. On the left sidebar, select **Settings > Metrics and profiling** and expand **Metrics - Grafana**. -1. Check the **Enable access to Grafana** checkbox. +1. Select the **Add a link to Grafana** checkbox. 1. Configure the **Grafana URL**: - *If Grafana is enabled through Omnibus GitLab and on the same server,* leave **Grafana URL** unchanged. It should be `/-/grafana`. - *Otherwise,* enter the full URL of the Grafana instance. -1. Click **Save changes**. +1. Select **Save changes**. GitLab displays your link in the **Menu > Admin > Monitoring > Metrics Dashboard**. +## Required Scopes + +> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5822) in GitLab 13.10. + +When setting up Grafana through the process above, no scope shows in the screen at +**Menu >** **{admin}** **Admin > Applications > GitLab Grafana**. However, the `read_user` scope is +required and is provided to the application automatically. Setting any scope other than +`read_user` without also including `read_user` leads to this error when you try to log in using +GitLab as the OAuth provider: + +```plaintext +The requested scope is invalid, unknown, or malformed. +``` + +If you see this error, make sure that one of the following is true in the GitLab Grafana +configuration screen: + +- No scopes appear. +- The `read_user` scope is included. + +> Versions of GitLab prior 13.10 use the API scope instead of `read_user`. In versions of GitLab +> prior to 13.10, the API scope: +> +> - Is required to access Grafana through the GitLab OAuth provider. +> - Is set by enabling the Grafana application as shown in [Integration with GitLab UI](#integration-with-gitlab-ui). + ## Security Update Users running GitLab version 12.0 or later should immediately upgrade to one of the diff --git a/doc/administration/monitoring/performance/performance_bar.md b/doc/administration/monitoring/performance/performance_bar.md index 5a7e8e12a38..6d9418133d8 100644 --- a/doc/administration/monitoring/performance/performance_bar.md +++ b/doc/administration/monitoring/performance/performance_bar.md @@ -4,22 +4,37 @@ group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Performance Bar **(FREE SELF)** +# Performance bar **(FREE SELF)** -> The **Stats** field [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/271551) in GitLab 13.9. -> The **Memory** field [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/330736) in GitLab 14.0. +> - The **Stats** field [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/271551) in GitLab 13.9. +> - The **Memory** field [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/330736) in GitLab 14.0. -You can display the GitLab Performance Bar to see statistics for the performance -of a page. When activated, it looks as follows: +You can display the performance bar to see statistics for the performance of a GitLab UI page. +For example: - + -From left to right, it displays: +## Available information + +From left to right, the performance bar displays: - **Current Host**: the current host serving the page. - **Database queries**: the time taken (in milliseconds) and the total number of database queries, displayed in the format `00ms / 00 (00 cached) pg`. Click to display - a modal window with more details. + a modal window with more details. You can use this to see the following + details for each query: + - **In a transaction**: shows up below the query if it was executed in + the context of a transaction + - **Role**: shows up when [database load + balancing](../../database_load_balancing.md) is enabled. It shows + which server role was used for the query. "Primary" means that the query + was sent to the read/write primary server. "Replica" means it was sent + to a read-only replica. + - **Config name**: shows up only when the + `GITLAB_MULTIPLE_DATABASE_METRICS` environment variable is set. This is + used to distinguish between different databases configured for different + GitLab features. The name shown is the same name used to configure database + connections in GitLab. - **Gitaly calls**: the time taken (in milliseconds) and the total number of [Gitaly](../../gitaly/index.md) calls. Click to display a modal window with more details. @@ -57,8 +72,17 @@ From left to right, it displays: NOTE: Not all indicators are available in all environments. For instance, the memory view -requires to run Ruby with [specific patches](https://gitlab.com/gitlab-org/gitlab-build-images/-/blob/master/patches/ruby/2.7.2/thread-memory-allocations-2.7.patch) applied. -When running GitLab locally using the GDK this is typically not the case and the memory view cannot be used. +requires running Ruby with [specific patches](https://gitlab.com/gitlab-org/gitlab-build-images/-/blob/master/patches/ruby/2.7.2/thread-memory-allocations-2.7.patch) +applied. When running GitLab locally using [GDK](../../../development/contributing/index.md#gitlab-development-kit), +this is typically not the case and the memory view cannot be used. + +## Keyboard shortcut + +Press the [<kbd>p</kbd> + <kbd>b</kbd> keyboard shortcut](../../../user/shortcuts.md) to display +the performance bar, and again to hide it. + +For non-administrators to display the performance bar, it must be +[enabled for them](#enable-the-performance-bar-for-non-administrators). ## Request warnings @@ -74,23 +98,17 @@ appears next to requests with warnings.  -## Enable the Performance Bar via the Admin Area +## Enable the performance bar for non-administrators -The GitLab Performance Bar is disabled by default for non-administrators. To enable it +The performance bar is disabled by default for non-administrators. To enable it for a given group: -1. Sign in as a user with Administrator [permissions](../../../user/permissions.md). +1. Sign in as a user with Administrator [role](../../../user/permissions.md). 1. On the top bar, select **Menu >** **{admin}** **Admin**. 1. On the left sidebar, select **Settings > Metrics and profiling** (`admin/application_settings/metrics_and_profiling`), and expand **Profiling - Performance bar**. -1. Click **Enable access to the Performance Bar**. -1. In the **Allowed group** field, provide the full path of the group allowed - to access the GitLab Performance Bar. +1. Click **Allow non-administrators to access to the performance bar**. +1. In the **Allow access to members of the following group** field, provide the full path of the + group allowed to access the performance. 1. Click **Save changes**. - -## Keyboard shortcut for the Performance Bar - -After enabling the GitLab Performance Bar, press the [<kbd>p</kbd> + -<kbd>b</kbd> keyboard shortcut](../../../user/shortcuts.md) to display it, and -again to hide it. diff --git a/doc/administration/monitoring/prometheus/gitlab_metrics.md b/doc/administration/monitoring/prometheus/gitlab_metrics.md index 2aa95a2b0f1..459eb259498 100644 --- a/doc/administration/monitoring/prometheus/gitlab_metrics.md +++ b/doc/administration/monitoring/prometheus/gitlab_metrics.md @@ -8,10 +8,10 @@ info: To determine the technical writer assigned to the Stage/Group associated w To enable the GitLab Prometheus metrics: -1. Log into GitLab as a user with [administrator permissions](../../../user/permissions.md). +1. Log in to GitLab as a user with Administrator [role](../../../user/permissions.md). 1. On the top bar, select **Menu >** **{admin}** **Admin**. 1. On the left sidebar, select **Settings > Metrics and profiling**. -1. Find the **Metrics - Prometheus** section, and click **Enable Prometheus Metrics**. +1. Find the **Metrics - Prometheus** section, and select **Add link to Prometheus**. 1. [Restart GitLab](../../restart_gitlab.md#omnibus-gitlab-restart) for the changes to take effect. For installations from source you must configure it yourself. @@ -45,6 +45,7 @@ The following metrics are available: | `gitlab_ci_pipeline_size_builds` | Histogram | 13.1 | Total number of builds within a pipeline grouped by a pipeline source | `source` | | `job_waiter_started_total` | Counter | 12.9 | Number of batches of jobs started where a web request is waiting for the jobs to complete | `worker` | | `job_waiter_timeouts_total` | Counter | 12.9 | Number of batches of jobs that timed out where a web request is waiting for the jobs to complete | `worker` | +| `gitlab_ci_active_jobs` | Histogram | 14.2 | Count of active jobs when pipeline is created | | | `gitlab_database_transaction_seconds` | Histogram | 12.1 | Time spent in database transactions, in seconds | | | `gitlab_method_call_duration_seconds` | Histogram | 10.2 | Method calls real duration | `controller`, `action`, `module`, `method` | | `gitlab_page_out_of_bounds` | Counter | 12.8 | Counter for the PageLimiter pagination limit being hit | `controller`, `action`, `bot` | @@ -152,15 +153,8 @@ The following metrics can be controlled by feature flags: ## Praefect metrics -You can [configure Praefect to report metrics](../../gitaly/praefect.md#praefect). -These are some of the Praefect metrics served from the `/metrics` path on the [configured port](index.md#changing-the-port-and-address-prometheus-listens-on) -(9652 by default). - -| Metric | Type | Since | Description | Labels | -| :----- | :--- | ----: | :---------- | :----- | -| `gitaly_praefect_replication_latency_bucket` | Histogram | 12.10 | The amount of time it takes for replication to complete once the replication job starts. | | -| `gitaly_praefect_replication_delay_bucket` | Histogram | 12.10 | A measure of how much time passes between when the replication job is created and when it starts. | | -| `gitaly_praefect_node_latency_bucket` | Histogram | 12.10 | The latency in Gitaly returning health check information to Praefect. This indicates Praefect connection saturation. | | +You can [configure Praefect](../../gitaly/praefect.md#praefect) to report metrics. For information +on available metrics, see the [relevant documentation](../../gitaly/index.md#monitor-gitaly-cluster). ## Sidekiq metrics diff --git a/doc/administration/monitoring/prometheus/index.md b/doc/administration/monitoring/prometheus/index.md index dd402f800e3..dd81f71d418 100644 --- a/doc/administration/monitoring/prometheus/index.md +++ b/doc/administration/monitoring/prometheus/index.md @@ -8,18 +8,19 @@ info: To determine the technical writer assigned to the Stage/Group associated w [Prometheus](https://prometheus.io) is a powerful time-series monitoring service, providing a flexible platform for monitoring GitLab and other software products. + GitLab provides out-of-the-box monitoring with Prometheus, providing easy access to high quality time-series monitoring of GitLab services. -> **Notes:** -> -> - Prometheus and the various exporters listed in this page are bundled in the -> Omnibus GitLab package. Check each exporter's documentation for the timeline -> they got added. For installations from source you must install them -> yourself. Over subsequent releases additional GitLab metrics are captured. -> - Prometheus services are on by default with GitLab 9.0. -> - Prometheus and its exporters don't authenticate users, and are available -> to anyone who can access them. +Prometheus and the various exporters listed in this page are bundled in the +Omnibus GitLab package. Check each exporter's documentation for the timeline +they got added. For installations from source you must install them +yourself. Over subsequent releases additional GitLab metrics are captured. + +Prometheus services are on by default. + +Prometheus and its exporters don't authenticate users, and are available to anyone who can access +them. ## Overview @@ -33,7 +34,7 @@ dashboard tool like [Grafana](https://grafana.com). For installations from source, you must install and configure it yourself. -Prometheus and its exporters are on by default, starting with GitLab 9.0. +Prometheus and its exporters are on by default. Prometheus runs as the `gitlab-prometheus` user and listen on `http://localhost:9090`. By default, Prometheus is only accessible from the GitLab server itself. Each exporter is automatically set up as a @@ -294,8 +295,8 @@ To use an external Prometheus server: You can visit `http://localhost:9090` for the dashboard that Prometheus offers by default. If SSL has been enabled on your GitLab instance, you may not be able to access -Prometheus on the same browser as GitLab if using the same FQDN due to [HSTS](https://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security). We plan to -[provide access via GitLab](https://gitlab.com/gitlab-org/multi-user-prometheus), but in the interim there are +Prometheus on the same browser as GitLab if using the same FQDN due to [HSTS](https://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security). +[A test project exists](https://gitlab.com/gitlab-org/multi-user-prometheus) to provide access via GitLab, but in the interim there are some workarounds: using a separate FQDN, using server IP, using a separate browser for Prometheus, resetting HSTS, or having [NGINX proxy it](https://docs.gitlab.com/omnibus/settings/nginx.html#inserting-custom-nginx-settings-into-the-gitlab-server-block). |