diff options
| author | GitLab Bot <gitlab-bot@gitlab.com> | 2020-06-18 11:18:50 +0000 |
|---|---|---|
| committer | GitLab Bot <gitlab-bot@gitlab.com> | 2020-06-18 11:18:50 +0000 |
| commit | 8c7f4e9d5f36cff46365a7f8c4b9c21578c1e781 (patch) | |
| tree | a77e7fe7a93de11213032ed4ab1f33a3db51b738 /doc/administration/monitoring | |
| parent | 00b35af3db1abfe813a778f643dad221aad51fca (diff) | |
| download | gitlab-ce-8c7f4e9d5f36cff46365a7f8c4b9c21578c1e781.tar.gz | |
Add latest changes from gitlab-org/gitlab@13-1-stable-ee
Diffstat (limited to 'doc/administration/monitoring')
16 files changed, 248 insertions, 139 deletions
diff --git a/doc/administration/monitoring/github_imports.md b/doc/administration/monitoring/github_imports.md index 6d5915930b2..21cc4c708a8 100644 --- a/doc/administration/monitoring/github_imports.md +++ b/doc/administration/monitoring/github_imports.md @@ -1,3 +1,9 @@ +--- +stage: Monitor +group: APM +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Monitoring GitHub imports >**Note:** diff --git a/doc/administration/monitoring/gitlab_self_monitoring_project/index.md b/doc/administration/monitoring/gitlab_self_monitoring_project/index.md index 091da46f316..d05fb803c5c 100644 --- a/doc/administration/monitoring/gitlab_self_monitoring_project/index.md +++ b/doc/administration/monitoring/gitlab_self_monitoring_project/index.md @@ -1,7 +1,13 @@ +--- +stage: Monitor +group: APM +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # GitLab self monitoring project -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/32351) in GitLab 12.7, behind a disabled feature flag (`self_monitoring_project`). -> - The feature flag was removed and the Self Monitoring Project was [made generally available](https://gitlab.com/gitlab-org/gitlab/issues/198511) in GitLab 12.8. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/32351) in GitLab 12.7, behind a disabled feature flag (`self_monitoring_project`). +> - The feature flag was removed and the Self Monitoring Project was [made generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/198511) in GitLab 12.8. GitLab has been adding the ability for administrators to see insights into the health of their GitLab instance. In order to surface this experience in a native way, similar to how @@ -15,10 +21,12 @@ All administrators at the time of creation of the project and group will be adde as maintainers of the group and project, and as an admin, you'll be able to add new members to the group in order to give them maintainer access to the project. -This project is used to self monitor your GitLab instance. Metrics are not yet -fully integrated, and the dashboard does not aggregate any data on Omnibus installations. GitLab plans -to provide integrated self-monitoring metrics in a future release. You can -currently use the project to configure your own [custom metrics](../../../user/project/integrations/prometheus.md#adding-custom-metrics) using +This project is used to self monitor your GitLab instance. The metrics dashboard +of the project shows some basic resource usage charts, such as CPU and memory usage +of each server in [Omnibus GitLab](https://docs.gitlab.com/omnibus/) installations. + +You can also use the project to configure your own +[custom metrics](../../../user/project/integrations/prometheus.md#adding-custom-metrics) using metrics exposed by the [GitLab exporter](../prometheus/gitlab_metrics.md#metrics-available). ## Creating the self monitoring project @@ -53,7 +61,7 @@ you should ## Taking action on Prometheus alerts **(ULTIMATE)** You can [add a webhook](../../../user/project/integrations/prometheus.md#external-prometheus-instances) -to the Prometheus config in order for GitLab to receive notifications of any alerts. +to the Prometheus configuration in order for GitLab to receive notifications of any alerts. Once the webhook is setup, you can [take action on incoming alerts](../../../user/project/integrations/prometheus.md#taking-action-on-incidents-ultimate). @@ -69,7 +77,7 @@ You can add custom metrics in the self monitoring project by: ### Getting error message in logs: `Could not create instance administrators group. Errors: ["You don’t have permission to create groups."]` -There is [a bug](https://gitlab.com/gitlab-org/gitlab/issues/208676) which causes +There is [a bug](https://gitlab.com/gitlab-org/gitlab/-/issues/208676) which causes project creation to fail with the following error (which appears in the log file) when the first admin user is an [external user](../../../user/permissions.md#external-users-core-only): diff --git a/doc/administration/monitoring/index.md b/doc/administration/monitoring/index.md index 1d7c52a198f..a54c25450c6 100644 --- a/doc/administration/monitoring/index.md +++ b/doc/administration/monitoring/index.md @@ -1,3 +1,9 @@ +--- +stage: Monitor +group: APM +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Monitoring GitLab Explore our features to monitor your GitLab instance: diff --git a/doc/administration/monitoring/ip_whitelist.md b/doc/administration/monitoring/ip_whitelist.md index 1b0ea1744d9..862a9368be8 100644 --- a/doc/administration/monitoring/ip_whitelist.md +++ b/doc/administration/monitoring/ip_whitelist.md @@ -1,3 +1,9 @@ +--- +stage: Monitor +group: APM +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # IP whitelist > Introduced in GitLab 9.4. diff --git a/doc/administration/monitoring/performance/gitlab_configuration.md b/doc/administration/monitoring/performance/gitlab_configuration.md index 14119a5d8f3..d09dabab40d 100644 --- a/doc/administration/monitoring/performance/gitlab_configuration.md +++ b/doc/administration/monitoring/performance/gitlab_configuration.md @@ -1,3 +1,9 @@ +--- +stage: Monitor +group: APM +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # GitLab Configuration GitLab Performance Monitoring is disabled by default. To enable it and change any of its diff --git a/doc/administration/monitoring/performance/grafana_configuration.md b/doc/administration/monitoring/performance/grafana_configuration.md index 3438a564d2f..4dbe3aed84e 100644 --- a/doc/administration/monitoring/performance/grafana_configuration.md +++ b/doc/administration/monitoring/performance/grafana_configuration.md @@ -1,3 +1,9 @@ +--- +stage: Monitor +group: APM +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Grafana Configuration [Grafana](https://grafana.com/) is a tool that allows you to visualize time @@ -57,7 +63,7 @@ repository for more information on this process. ## Integration with GitLab UI -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/61005) in GitLab 12.1. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/61005) in GitLab 12.1. If you have set up Grafana, you can enable a link to access it easily from the sidebar: diff --git a/doc/administration/monitoring/performance/img/request_profiling_token.png b/doc/administration/monitoring/performance/img/request_profiling_token.png Binary files differdeleted file mode 100644 index ee819fcb437..00000000000 --- a/doc/administration/monitoring/performance/img/request_profiling_token.png +++ /dev/null diff --git a/doc/administration/monitoring/performance/index.md b/doc/administration/monitoring/performance/index.md index 02070287611..6f22327e499 100644 --- a/doc/administration/monitoring/performance/index.md +++ b/doc/administration/monitoring/performance/index.md @@ -1,3 +1,9 @@ +--- +stage: Monitor +group: APM +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # GitLab Performance Monitoring GitLab comes with its own application performance measuring system as of GitLab diff --git a/doc/administration/monitoring/performance/performance_bar.md b/doc/administration/monitoring/performance/performance_bar.md index 89246bc9782..c681dedbca8 100644 --- a/doc/administration/monitoring/performance/performance_bar.md +++ b/doc/administration/monitoring/performance/performance_bar.md @@ -1,3 +1,9 @@ +--- +stage: Monitor +group: APM +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Performance Bar A Performance Bar can be displayed, to dig into the performance of a page. When @@ -18,7 +24,7 @@ It allows you to see (from left to right):  - total load timings of the page; click through for details of these calls. Values in the following order: - Backend - Time that the actual base page took to load - - [First Contentful Paint](https://developers.google.com/web/tools/lighthouse/audits/first-contentful-paint) - Time until something was visible to the user + - [First Contentful Paint](hhttps://web.dev/first-contentful-paint/) - Time until something was visible to the user - [DomContentLoaded](https://developers.google.com/web/fundamentals/performance/critical-rendering-path/measure-crp) Event - Number of Requests that the page loaded  diff --git a/doc/administration/monitoring/performance/request_profiling.md b/doc/administration/monitoring/performance/request_profiling.md index c32edb60f9d..a3b29493d84 100644 --- a/doc/administration/monitoring/performance/request_profiling.md +++ b/doc/administration/monitoring/performance/request_profiling.md @@ -1,17 +1,39 @@ +--- +stage: Monitor +group: APM +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Request Profiling -## Procedure +To profile a request: + +1. Sign in to GitLab as a user with Administrator or Maintainer [permissions](../../../user/permissions.md). +1. In the navigation bar, click **{admin}** **Admin area**. +1. Navigate to **{monitor}** **Monitoring > Requests Profiles**. +1. In the **Requests Profiles** section, copy the token. +1. Pass the headers `X-Profile-Token: <token>` and `X-Profile-Mode: <mode>`(where + `<mode>` can be `execution` or `memory`) to the request you want to profile. When + passing headers, you can use: + + - Browser extensions such as the + [ModHeader](https://chrome.google.com/webstore/detail/modheader/idgpnmonknjnojddfkpgkljpfnnfcklj) + Chrome extension. + - `curl`. For example: + + ```shell + curl --header 'X-Profile-Token: <token>' --header 'X-Profile-Mode: <mode>' "https://gitlab.example.com/group/project" + ``` + + NOTE: **Note:** + Profiled requests can take longer than usual. + +After the request completes, you can view the profiling output from the +**{monitor}** **Monitoring > Requests Profiles** administration page: -1. Grab the profiling token from **Monitoring > Requests Profiles** admin page - (highlighted in a blue in the image below). -  -1. Pass the header `X-Profile-Token: <token>` and `X-Profile-Mode: <mode>`(where `<mode>` can be `execution` or `memory`) to the request you want to profile. You can use: - - Browser extensions. For example, [ModHeader](https://chrome.google.com/webstore/detail/modheader/idgpnmonknjnojddfkpgkljpfnnfcklj) Chrome extension. - - `curl`. For example, `curl --header 'X-Profile-Token: <token>' --header 'X-Profile-Mode: <mode>' https://gitlab.example.com/group/project`. -1. Once request is finished (which will take a little longer than usual), you can - view the profiling output from **Monitoring > Requests Profiles** admin page. -  + -## Cleaning up +## Cleaning up profiled requests -Profiling output will be cleared out every day via a Sidekiq worker. +The output from profiled requests is cleared out once each day through a +Sidekiq worker. diff --git a/doc/administration/monitoring/prometheus/gitlab_metrics.md b/doc/administration/monitoring/prometheus/gitlab_metrics.md index f725db9a039..f3084b1732e 100644 --- a/doc/administration/monitoring/prometheus/gitlab_metrics.md +++ b/doc/administration/monitoring/prometheus/gitlab_metrics.md @@ -6,30 +6,29 @@ info: To determine the technical writer assigned to the Stage/Group associated w # GitLab Prometheus metrics ->**Note:** -Available since [Omnibus GitLab 9.3](https://gitlab.com/gitlab-org/gitlab-foss/issues/29118). For -installations from source you'll have to configure it yourself. - To enable the GitLab Prometheus metrics: -1. Log into GitLab as an administrator. -1. Navigate to **Admin Area > Settings > Metrics and profiling**. +1. Log into GitLab as a user with [administrator permissions](../../../user/permissions.md). +1. Navigate to **{admin}** **Admin Area > Settings > Metrics and profiling**. 1. Find the **Metrics - Prometheus** section, and click **Enable Prometheus Metrics**. 1. [Restart GitLab](../../restart_gitlab.md#omnibus-gitlab-restart) for the changes to take effect. +NOTE: **Note:** +For installations from source you'll have to configure it yourself. + ## Collecting the metrics GitLab monitors its own internal service metrics, and makes them available at the -`/-/metrics` endpoint. Unlike other [Prometheus](https://prometheus.io) exporters, in order to access -it, the client IP needs to be [included in a whitelist](../ip_whitelist.md). +`/-/metrics` endpoint. Unlike other [Prometheus](https://prometheus.io) exporters, to access +it, the client IP address needs to be [explicitly allowed](../ip_whitelist.md). -For Omnibus and Chart installations, these metrics are automatically enabled -and collected as of [GitLab -9.4](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/1702). For -source installations or earlier versions, these metrics will need to be enabled +For [Omnibus GitLab](https://docs.gitlab.com/omnibus/) and Chart installations, +these metrics are enabled and collected as of +[GitLab 9.4](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/1702). +For source installations or earlier versions, these metrics must be enabled manually and collected by a Prometheus server. -See also [Sidekiq metrics](#sidekiq-metrics) for how to enable and view metrics from Sidekiq nodes. +For enabling and viewing metrics from Sidekiq nodes, see [Sidekiq metrics](#sidekiq-metrics). ## Metrics available @@ -43,32 +42,33 @@ The following metrics are available: | `gitlab_cache_operation_duration_seconds` | Histogram | 10.2 | Cache access time | | | `gitlab_cache_operations_total` | Counter | 12.2 | Cache operations by controller/action | `controller`, `action`, `operation` | | `gitlab_ci_pipeline_creation_duration_seconds` | Histogram | 13.0 | Time in seconds it takes to create a CI/CD pipeline | | +| `gitlab_ci_pipeline_size_builds` | Histogram | 13.1 | Total number of builds within a pipeline grouped by a pipeline source | `source` | | `job_waiter_started_total` | Counter | 12.9 | Number of batches of jobs started where a web request is waiting for the jobs to complete | `worker` | | `job_waiter_timeouts_total` | Counter | 12.9 | Number of batches of jobs that timed out where a web request is waiting for the jobs to complete | `worker` | | `gitlab_database_transaction_seconds` | Histogram | 12.1 | Time spent in database transactions, in seconds | | | `gitlab_method_call_duration_seconds` | Histogram | 10.2 | Method calls real duration | `controller`, `action`, `module`, `method` | | `gitlab_page_out_of_bounds` | Counter | 12.8 | Counter for the PageLimiter pagination limit being hit | `controller`, `action`, `bot` | | `gitlab_rails_queue_duration_seconds` | Histogram | 9.4 | Measures latency between GitLab Workhorse forwarding a request to Rails | | -| `gitlab_sql_duration_seconds` | Histogram | 10.2 | SQL execution time, excluding SCHEMA operations and BEGIN / COMMIT | | -| `gitlab_transaction_allocated_memory_bytes` | Histogram | 10.2 | Allocated memory for all transactions (gitlab_transaction_* metrics) | | +| `gitlab_sql_duration_seconds` | Histogram | 10.2 | SQL execution time, excluding `SCHEMA` operations and `BEGIN` / `COMMIT` | | +| `gitlab_transaction_allocated_memory_bytes` | Histogram | 10.2 | Allocated memory for all transactions (`gitlab_transaction_*` metrics) | | | `gitlab_transaction_cache_<key>_count_total` | Counter | 10.2 | Counter for total Rails cache calls (per key) | | | `gitlab_transaction_cache_<key>_duration_total` | Counter | 10.2 | Counter for total time (seconds) spent in Rails cache calls (per key) | | | `gitlab_transaction_cache_count_total` | Counter | 10.2 | Counter for total Rails cache calls (aggregate) | | | `gitlab_transaction_cache_duration_total` | Counter | 10.2 | Counter for total time (seconds) spent in Rails cache calls (aggregate) | | | `gitlab_transaction_cache_read_hit_count_total` | Counter | 10.2 | Counter for cache hits for Rails cache calls | `controller`, `action` | | `gitlab_transaction_cache_read_miss_count_total` | Counter | 10.2 | Counter for cache misses for Rails cache calls | `controller`, `action` | -| `gitlab_transaction_duration_seconds` | Histogram | 10.2 | Duration for all transactions (gitlab_transaction_* metrics) | `controller`, `action` | +| `gitlab_transaction_duration_seconds` | Histogram | 10.2 | Duration for all transactions (`gitlab_transaction_*` metrics) | `controller`, `action` | | `gitlab_transaction_event_build_found_total` | Counter | 9.4 | Counter for build found for API /jobs/request | | | `gitlab_transaction_event_build_invalid_total` | Counter | 9.4 | Counter for build invalid due to concurrency conflict for API /jobs/request | | | `gitlab_transaction_event_build_not_found_cached_total` | Counter | 9.4 | Counter for cached response of build not found for API /jobs/request | | | `gitlab_transaction_event_build_not_found_total` | Counter | 9.4 | Counter for build not found for API /jobs/request | | | `gitlab_transaction_event_change_default_branch_total` | Counter | 9.4 | Counter when default branch is changed for any repository | | | `gitlab_transaction_event_create_repository_total` | Counter | 9.4 | Counter when any repository is created | | -| `gitlab_transaction_event_etag_caching_cache_hit_total` | Counter | 9.4 | Counter for etag cache hit. | `endpoint` | -| `gitlab_transaction_event_etag_caching_header_missing_total` | Counter | 9.4 | Counter for etag cache miss - header missing | `endpoint` | -| `gitlab_transaction_event_etag_caching_key_not_found_total` | Counter | 9.4 | Counter for etag cache miss - key not found | `endpoint` | -| `gitlab_transaction_event_etag_caching_middleware_used_total` | Counter | 9.4 | Counter for etag middleware accessed | `endpoint` | -| `gitlab_transaction_event_etag_caching_resource_changed_total` | Counter | 9.4 | Counter for etag cache miss - resource changed | `endpoint` | +| `gitlab_transaction_event_etag_caching_cache_hit_total` | Counter | 9.4 | Counter for ETag cache hit. | `endpoint` | +| `gitlab_transaction_event_etag_caching_header_missing_total` | Counter | 9.4 | Counter for ETag cache miss - header missing | `endpoint` | +| `gitlab_transaction_event_etag_caching_key_not_found_total` | Counter | 9.4 | Counter for ETag cache miss - key not found | `endpoint` | +| `gitlab_transaction_event_etag_caching_middleware_used_total` | Counter | 9.4 | Counter for ETag middleware accessed | `endpoint` | +| `gitlab_transaction_event_etag_caching_resource_changed_total` | Counter | 9.4 | Counter for ETag cache miss - resource changed | `endpoint` | | `gitlab_transaction_event_fork_repository_total` | Counter | 9.4 | Counter for repository forks (RepositoryForkWorker). Only incremented when source repository exists | | | `gitlab_transaction_event_import_repository_total` | Counter | 9.4 | Counter for repository imports (RepositoryImportWorker) | | | `gitlab_transaction_event_push_branch_total` | Counter | 9.4 | Counter for all branch pushes | | @@ -92,6 +92,13 @@ The following metrics are available: | `gitlab_view_rendering_duration_seconds` | Histogram | 10.2 | Duration for views (histogram) | `controller`, `action`, `view` | | `http_requests_total` | Counter | 9.4 | Rack request count | `method` | | `http_request_duration_seconds` | Histogram | 9.4 | HTTP response time from rack middleware | `method`, `status` | +| `gitlab_transaction_db_count_total` | Counter | 13.1 | Counter for total number of sql calls | `controller`, `action` | +| `gitlab_transaction_db_write_count_total` | Counter | 13.1 | Counter for total number of write sql calls | `controller`, `action` | +| `gitlab_transaction_db_cached_count_total` | Counter | 13.1 | Counter for total number of cached sql calls | `controller`, `action` | +| `http_redis_requests_duration_seconds` | Histogram | 13.1 | Redis requests duration during web transactions | `controller`, `action` | +| `http_redis_requests_total` | Counter | 13.1 | Redis requests count during web transactions | `controller`, `action` | +| `http_elasticsearch_requests_duration_seconds` **(STARTER)** | Histogram | 13.1 | Elasticsearch requests duration during web transactions | `controller`, `action` | +| `http_elasticsearch_requests_total` **(STARTER)** | Counter | 13.1 | Elasticsearch requests count during web transactions | `controller`, `action` | | `pipelines_created_total` | Counter | 9.4 | Counter of pipelines created | | | `rack_uncaught_errors_total` | Counter | 9.4 | Rack connections handling uncaught errors count | | | `user_session_logins_total` | Counter | 9.4 | Counter of how many users have logged in | | @@ -99,7 +106,7 @@ The following metrics are available: | `failed_login_captcha_total` | Gauge | 11.0 | Counter of failed CAPTCHA attempts during login | | | `successful_login_captcha_total` | Gauge | 11.0 | Counter of successful CAPTCHA attempts during login | | | `auto_devops_pipelines_completed_total` | Counter | 12.7 | Counter of completed Auto DevOps pipelines, labeled by status | | -| `gitlab_metrics_dashboard_processing_time_ms` | Summary | 12.10 | Metrics dashboard processing time in milliseconds | service, stages | +| `gitlab_metrics_dashboard_processing_time_ms` | Summary | 12.10 | Metrics dashboard processing time in milliseconds | service, stages | ## Metrics controlled by a feature flag @@ -119,13 +126,17 @@ configuration option in `gitlab.yml`. These metrics are served from the | Metric | Type | Since | Description | Labels | |:---------------------------------------------- |:------- |:----- |:----------- |:------ | -| `sidekiq_jobs_cpu_seconds` | Histogram | 12.4 | Seconds of cpu time to run Sidekiq job | `queue`, `boundary`, `external_dependencies`, `feature_category`, `job_status`, `urgency` | +| `sidekiq_jobs_cpu_seconds` | Histogram | 12.4 | Seconds of CPU time to run Sidekiq job | `queue`, `boundary`, `external_dependencies`, `feature_category`, `job_status`, `urgency` | | `sidekiq_jobs_completion_seconds` | Histogram | 12.2 | Seconds to complete Sidekiq job | `queue`, `boundary`, `external_dependencies`, `feature_category`, `job_status`, `urgency` | | `sidekiq_jobs_db_seconds` | Histogram | 12.9 | Seconds of DB time to run Sidekiq job | `queue`, `boundary`, `external_dependencies`, `feature_category`, `job_status`, `urgency` | | `sidekiq_jobs_gitaly_seconds` | Histogram | 12.9 | Seconds of Gitaly time to run Sidekiq job | `queue`, `boundary`, `external_dependencies`, `feature_category`, `job_status`, `urgency` | +| `sidekiq_redis_requests_duration_seconds` | Histogram | 13.1 | Duration in seconds that a Sidekiq job spent querying a Redis server | `queue`, `boundary`, `external_dependencies`, `feature_category`, `job_status`, `urgency` | +| `sidekiq_elasticsearch_requests_duration_seconds` | Histogram | 13.1 | Duration in seconds that a Sidekiq job spent in requests to an Elasticsearch server | `queue`, `boundary`, `external_dependencies`, `feature_category`, `job_status`, `urgency` | | `sidekiq_jobs_queue_duration_seconds` | Histogram | 12.5 | Duration in seconds that a Sidekiq job was queued before being executed | `queue`, `boundary`, `external_dependencies`, `feature_category`, `urgency` | | `sidekiq_jobs_failed_total` | Counter | 12.2 | Sidekiq jobs failed | `queue`, `boundary`, `external_dependencies`, `feature_category`, `urgency` | | `sidekiq_jobs_retried_total` | Counter | 12.2 | Sidekiq jobs retried | `queue`, `boundary`, `external_dependencies`, `feature_category`, `urgency` | +| `sidekiq_redis_requests_total` | Counter | 13.1 | Redis requests during a Sidekiq job execution | `queue`, `boundary`, `external_dependencies`, `feature_category`, `job_status`, `urgency` | +| `sidekiq_elasticsearch_requests_total` | Counter | 13.1 | Elasticsearch requests during a Sidekiq job execution | `queue`, `boundary`, `external_dependencies`, `feature_category`, `job_status`, `urgency` | | `sidekiq_running_jobs` | Gauge | 12.2 | Number of Sidekiq jobs running | `queue`, `boundary`, `external_dependencies`, `feature_category`, `urgency` | | `sidekiq_concurrency` | Gauge | 12.5 | Maximum number of Sidekiq jobs | | | `geo_db_replication_lag_seconds` | Gauge | 10.2 | Database replication lag (seconds) | `url` | @@ -172,7 +183,29 @@ The following metrics are available: | Metric | Type | Since | Description | |:--------------------------------- |:--------- |:------------------------------------------------------------- |:-------------------------------------- | -| `db_load_balancing_hosts` | Gauge | [12.3](https://gitlab.com/gitlab-org/gitlab/issues/13630) | Current number of load balancing hosts | +| `db_load_balancing_hosts` | Gauge | [12.3](https://gitlab.com/gitlab-org/gitlab/-/issues/13630) | Current number of load balancing hosts | + +## Connection pool metrics + +These metrics record the status of the database [connection pools](https://api.rubyonrails.org/classes/ActiveRecord/ConnectionAdapters/ConnectionPool.html). + +They all have these labels: + +1. `class` - the Ruby class being recorded. + 1. `ActiveRecord::Base` is the main database connection. + 1. `Geo::TrackingBase` is the connection to the Geo tracking database, if + enabled. +1. `host` - the host name used to connect to the database. +1. `port` - the port used to connect to the database. + +| Metric | Type | Since | Description | +|:----------------------------------------------|:------|:------|:--------------------------------------------------| +| `gitlab_database_connection_pool_size` | Gauge | 13.0 | Total connection pool capacity | +| `gitlab_database_connection_pool_connections` | Gauge | 13.0 | Current connections in the pool | +| `gitlab_database_connection_pool_busy` | Gauge | 13.0 | Connections in use where the owner is still alive | +| `gitlab_database_connection_pool_dead` | Gauge | 13.0 | Connections in use where the owner is not alive | +| `gitlab_database_connection_pool_idle` | Gauge | 13.0 | Connections not in use | +| `gitlab_database_connection_pool_waiting` | Gauge | 13.0 | Threads currently waiting on this queue | ## Ruby metrics @@ -205,31 +238,28 @@ Unicorn specific metrics, when Unicorn is used. When Puma is used instead of Unicorn, the following metrics are available: -| Metric | Type | Since | Description | -|:---------------------------------------------- |:------- |:----- |:----------- | -| `puma_workers` | Gauge | 12.0 | Total number of workers | -| `puma_running_workers` | Gauge | 12.0 | Number of booted workers | -| `puma_stale_workers` | Gauge | 12.0 | Number of old workers | -| `puma_running` | Gauge | 12.0 | Number of running threads | -| `puma_queued_connections` | Gauge | 12.0 | Number of connections in that worker's "to do" set waiting for a worker thread | -| `puma_active_connections` | Gauge | 12.0 | Number of threads processing a request | -| `puma_pool_capacity` | Gauge | 12.0 | Number of requests the worker is capable of taking right now | -| `puma_max_threads` | Gauge | 12.0 | Maximum number of worker threads | -| `puma_idle_threads` | Gauge | 12.0 | Number of spawned threads which are not processing a request | -| `puma_killer_terminations_total` | Gauge | 12.0 | Number of workers terminated by PumaWorkerKiller | +| Metric | Type | Since | Description | +|:--------------------------------- |:------- |:----- |:----------- | +| `puma_workers` | Gauge | 12.0 | Total number of workers | +| `puma_running_workers` | Gauge | 12.0 | Number of booted workers | +| `puma_stale_workers` | Gauge | 12.0 | Number of old workers | +| `puma_running` | Gauge | 12.0 | Number of running threads | +| `puma_queued_connections` | Gauge | 12.0 | Number of connections in that worker's "to do" set waiting for a worker thread | +| `puma_active_connections` | Gauge | 12.0 | Number of threads processing a request | +| `puma_pool_capacity` | Gauge | 12.0 | Number of requests the worker is capable of taking right now | +| `puma_max_threads` | Gauge | 12.0 | Maximum number of worker threads | +| `puma_idle_threads` | Gauge | 12.0 | Number of spawned threads which are not processing a request | +| `puma_killer_terminations_total` | Gauge | 12.0 | Number of workers terminated by PumaWorkerKiller | ## Metrics shared directory GitLab's Prometheus client requires a directory to store metrics data shared between multi-process services. Those files are shared among all instances running under Unicorn server. -The directory needs to be accessible to all running Unicorn's processes otherwise -metrics will not function correctly. - -For best performance its advisable that this directory will be located in `tmpfs`. - -Its location is configured using environment variable `prometheus_multiproc_dir`. +The directory must be accessible to all running Unicorn's processes, or +metrics won't function correctly. -If GitLab is installed using Omnibus and `tmpfs` is available then metrics -directory will be automatically configured. +This directory's location is configured using environment variable `prometheus_multiproc_dir`. +For best performance, create this directory in `tmpfs`. -[← Back to the main Prometheus page](index.md) +If GitLab is installed using [Omnibus GitLab](https://docs.gitlab.com/omnibus/) +and `tmpfs` is available, then the metrics directory will be configured for you. diff --git a/doc/administration/monitoring/prometheus/index.md b/doc/administration/monitoring/prometheus/index.md index cb93aca6e4e..1e233b890a2 100644 --- a/doc/administration/monitoring/prometheus/index.md +++ b/doc/administration/monitoring/prometheus/index.md @@ -84,7 +84,7 @@ To change the address/port that Prometheus listens on: 1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect -### Adding custom scrape configs +### Adding custom scrape configurations You can configure additional scrape targets for the Omnibus GitLab-bundled Prometheus by editing `prometheus['scrape_configs']` in `/etc/gitlab/gitlab.rb` diff --git a/doc/administration/monitoring/prometheus/node_exporter.md b/doc/administration/monitoring/prometheus/node_exporter.md index 357303ee4e1..07f6b8b8e1e 100644 --- a/doc/administration/monitoring/prometheus/node_exporter.md +++ b/doc/administration/monitoring/prometheus/node_exporter.md @@ -6,27 +6,24 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Node exporter ->**Note:** -Available since Omnibus GitLab 8.16. For installations from source you'll -have to install and configure it yourself. - -The [node exporter](https://github.com/prometheus/node_exporter) allows you to measure +The [node exporter](https://github.com/prometheus/node_exporter) enables you to measure various machine resources such as memory, disk and CPU utilization. +NOTE: **Note:** +For installations from source you'll have to install and configure it yourself. + To enable the node exporter: -1. [Enable Prometheus](index.md#configuring-prometheus) -1. Edit `/etc/gitlab/gitlab.rb` -1. Add or find and uncomment the following line, making sure it's set to `true`: +1. [Enable Prometheus](index.md#configuring-prometheus). +1. Edit `/etc/gitlab/gitlab.rb`. +1. Add (or find and uncomment) the following line, making sure it's set to `true`: ```ruby node_exporter['enable'] = true ``` -1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) - for the changes to take effect - -Prometheus will now automatically begin collecting performance data from -the node exporter exposed under `localhost:9100`. +1. Save the file, and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) + for the changes to take effect. -[← Back to the main Prometheus page](index.md) +Prometheus will now begin collecting performance data from the node exporter +exposed at `localhost:9100`. diff --git a/doc/administration/monitoring/prometheus/pgbouncer_exporter.md b/doc/administration/monitoring/prometheus/pgbouncer_exporter.md index 92ba2d9bb52..62d0bf684b6 100644 --- a/doc/administration/monitoring/prometheus/pgbouncer_exporter.md +++ b/doc/administration/monitoring/prometheus/pgbouncer_exporter.md @@ -6,29 +6,30 @@ info: To determine the technical writer assigned to the Stage/Group associated w # PgBouncer exporter ->**Note:** -Available since [Omnibus GitLab 11.0](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/2493). -For installations from source you'll have to install and configure it yourself. +> Introduced in [Omnibus GitLab 11.0](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/2493). + +The [PgBouncer exporter](https://github.com/prometheus-community/pgbouncer_exporter) enables +you to measure various [PgBouncer](https://www.pgbouncer.org/) metrics. -The [PgBouncer exporter](https://github.com/stanhu/pgbouncer_exporter) allows you to measure various PgBouncer metrics. +NOTE: **Note:** +For installations from source you'll have to install and configure it yourself. To enable the PgBouncer exporter: -1. [Enable Prometheus](index.md#configuring-prometheus) -1. Edit `/etc/gitlab/gitlab.rb` -1. Add or find and uncomment the following line, making sure it's set to `true`: +1. [Enable Prometheus](index.md#configuring-prometheus). +1. Edit `/etc/gitlab/gitlab.rb`. +1. Add (or find and uncomment) the following line, making sure it's set to `true`: ```ruby pgbouncer_exporter['enable'] = true ``` -1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to - take effect. +1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) + for the changes to take effect. -Prometheus will now automatically begin collecting performance data from -the PgBouncer exporter exposed under `localhost:9188`. +Prometheus will now begin collecting performance data from the PgBouncer exporter +exposed at `localhost:9188`. -The PgBouncer exporter will also be enabled by default if the [`pgbouncer_role`](https://docs.gitlab.com/omnibus/roles/#postgres-roles) +The PgBouncer exporter will also be enabled by default if the +[`pgbouncer_role`](https://docs.gitlab.com/omnibus/roles/#postgresql-roles) role is enabled. - -[← Back to the main Prometheus page](index.md) diff --git a/doc/administration/monitoring/prometheus/postgres_exporter.md b/doc/administration/monitoring/prometheus/postgres_exporter.md index 77ca502b21d..e3fff45fce3 100644 --- a/doc/administration/monitoring/prometheus/postgres_exporter.md +++ b/doc/administration/monitoring/prometheus/postgres_exporter.md @@ -6,12 +6,11 @@ info: To determine the technical writer assigned to the Stage/Group associated w # PostgreSQL Server Exporter ->**Note:** -Available since [Omnibus GitLab 8.17](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/1131). -For installations from source you will have to install and configure it yourself. - The [PostgreSQL Server Exporter](https://github.com/wrouesnel/postgres_exporter) allows you to export various PostgreSQL metrics. +NOTE: **Note:** +For installations from source you will have to install and configure it yourself. + To enable the PostgreSQL Server Exporter: 1. [Enable Prometheus](index.md#configuring-prometheus). @@ -21,10 +20,10 @@ To enable the PostgreSQL Server Exporter: postgres_exporter['enable'] = true ``` -NOTE: **Note:** -If PostgreSQL Server Exporter is configured on a separate node, make sure that the local -address is [listed in `trust_auth_cidr_addresses`](../../high_availability/database.md#network-information) or the -exporter will not be able to connect to the database. + NOTE: **Note:** + If PostgreSQL Server Exporter is configured on a separate node, make sure that the local + address is [listed in `trust_auth_cidr_addresses`](../../postgresql/replication_and_failover.md#network-information) or the + exporter will not be able to connect to the database. 1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. @@ -34,31 +33,44 @@ the PostgreSQL Server Exporter exposed under `localhost:9187`. ## Advanced configuration -In most cases, PostgreSQL Server Exporter will work with the defaults and you should not -need to change anything. - -To further customize the PostgreSQL Server Exporter, use the following configuration options: +In most cases, PostgreSQL Server Exporter works with the defaults and you should not +need to change anything. To further customize the PostgreSQL Server Exporter, +use the following configuration options: 1. Edit `/etc/gitlab/gitlab.rb`: ```ruby - postgres_exporter['dbname'] = 'pgbouncer' # The name of the database to connect to. - postgres_exporter['user'] = 'gitlab-psql' # The user to sign in as. - postgres_exporter['password'] = '' # The user's password. - postgres_exporter['host'] = 'localhost' # The host to connect to. Values that start with '/' are for unix domain sockets (default is 'localhost'). - postgres_exporter['port'] = 5432 # The port to bind to (default is '5432'). - postgres_exporter['sslmode'] = 'require' # Whether or not to use SSL. Valid options are: - # 'disable' (no SSL), - # 'require' (always use SSL and skip verification, this is the default value), - # 'verify-ca' (always use SSL and verify that the certificate presented by the server was signed by a trusted CA), - # 'verify-full' (always use SSL and verify that the certification presented by the server was signed by a trusted CA and the server host name matches the one in the certificate). - postgres_exporter['fallback_application_name'] = '' # An application_name to fall back to if one isn't provided. - postgres_exporter['connect_timeout'] = '' # Maximum wait for connection, in seconds. Zero or not specified means wait indefinitely. - postgres_exporter['sslcert'] = 'ssl.crt' # Cert file location. The file must contain PEM encoded data. - postgres_exporter['sslkey'] = 'ssl.key' # Key file location. The file must contain PEM encoded data. - postgres_exporter['sslrootcert'] = 'ssl-root.crt' # The location of the root certificate file. The file must contain PEM encoded data. + # The name of the database to connect to. + postgres_exporter['dbname'] = 'pgbouncer' + # The user to sign in as. + postgres_exporter['user'] = 'gitlab-psql' + # The user's password. + postgres_exporter['password'] = '' + # The host to connect to. Values that start with '/' are for unix domain sockets + # (default is 'localhost'). + postgres_exporter['host'] = 'localhost' + # The port to bind to (default is '5432'). + postgres_exporter['port'] = 5432 + # Whether or not to use SSL. Valid options are: + # 'disable' (no SSL), + # 'require' (always use SSL and skip verification, this is the default value), + # 'verify-ca' (always use SSL and verify that the certificate presented by + # the server was signed by a trusted CA), + # 'verify-full' (always use SSL and verify that the certification presented + # by the server was signed by a trusted CA and the server host name matches + # the one in the certificate). + postgres_exporter['sslmode'] = 'require' + # An application_name to fall back to if one isn't provided. + postgres_exporter['fallback_application_name'] = '' + # Maximum wait for connection, in seconds. Zero or not specified means wait indefinitely. + postgres_exporter['connect_timeout'] = '' + # Cert file location. The file must contain PEM encoded data. + postgres_exporter['sslcert'] = 'ssl.crt' + # Key file location. The file must contain PEM encoded data. + postgres_exporter['sslkey'] = 'ssl.key' + # The location of the root certificate file. The file must contain PEM encoded data. + postgres_exporter['sslrootcert'] = 'ssl-root.crt' ``` -1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. - -[← Back to the main Prometheus page](index.md) +1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) + for the changes to take effect. diff --git a/doc/administration/monitoring/prometheus/redis_exporter.md b/doc/administration/monitoring/prometheus/redis_exporter.md index bef87400f5a..b7c66959349 100644 --- a/doc/administration/monitoring/prometheus/redis_exporter.md +++ b/doc/administration/monitoring/prometheus/redis_exporter.md @@ -6,19 +6,18 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Redis exporter ->**Note:** -Available since [Omnibus GitLab 8.17](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/1118). -For installations from source you'll have to install and configure it yourself. - -The [Redis exporter](https://github.com/oliver006/redis_exporter) allows you to measure -various [Redis](https://redis.io) metrics. For more information on what's exported, +The [Redis exporter](https://github.com/oliver006/redis_exporter) enables you to measure +various [Redis](https://redis.io) metrics. For more information on what is exported, [read the upstream documentation](https://github.com/oliver006/redis_exporter/blob/master/README.md#whats-exported). +NOTE: **Note:** +For installations from source you'll have to install and configure it yourself. + To enable the Redis exporter: -1. [Enable Prometheus](index.md#configuring-prometheus) -1. Edit `/etc/gitlab/gitlab.rb` -1. Add or find and uncomment the following line, making sure it's set to `true`: +1. [Enable Prometheus](index.md#configuring-prometheus). +1. Edit `/etc/gitlab/gitlab.rb`. +1. Add (or find and uncomment) the following line, making sure it's set to `true`: ```ruby redis_exporter['enable'] = true @@ -27,7 +26,5 @@ To enable the Redis exporter: 1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -Prometheus will now automatically begin collecting performance data from -the Redis exporter exposed under `localhost:9121`. - -[← Back to the main Prometheus page](index.md) +Prometheus will now begin collecting performance data from +the Redis exporter exposed at `localhost:9121`. |
