diff options
Diffstat (limited to 'doc/development/telemetry/usage_ping.md')
| -rw-r--r-- | doc/development/telemetry/usage_ping.md | 422 |
1 files changed, 57 insertions, 365 deletions
diff --git a/doc/development/telemetry/usage_ping.md b/doc/development/telemetry/usage_ping.md index 5e78e2c5f25..ea5eb6c389f 100644 --- a/doc/development/telemetry/usage_ping.md +++ b/doc/development/telemetry/usage_ping.md @@ -131,7 +131,7 @@ There are four types of counters which are all found in `usage_data.rb`: - **Ordinary Batch Counters:** Simple count of a given ActiveRecord_Relation - **Distinct Batch Counters:** Distinct count of a given ActiveRecord_Relation on given column - **Alternative Counters:** Used for settings and configurations -- **Redis Counters:** Used for in-memory counts. This method is being deprecated due to data inaccuracies and will be replaced with a persistent method. +- **Redis Counters:** Used for in-memory counts. NOTE: **Note:** Only use the provided counter methods. Each counter method contains a built in fail safe to isolate each counter to avoid breaking the entire Usage Ping. @@ -155,7 +155,7 @@ There are two batch counting methods provided, `Ordinary Batch Counters` and `Di Handles `ActiveRecord::StatementInvalid` error -Simple count of a given ActiveRecord_Relation +Simple count of a given ActiveRecord_Relation, does a non-distinct batch count, smartly reduces batch_size and handles errors. Method: `count(relation, column = nil, batch: true, start: nil, finish: nil)` @@ -179,15 +179,16 @@ count(::Clusters::Cluster.aws_installed.enabled, :cluster_id, start: ::Clusters: Handles `ActiveRecord::StatementInvalid` error -Distinct count of a given ActiveRecord_Relation on given column +Distinct count of a given ActiveRecord_Relation on given column, a distinct batch count, smartly reduces batch_size and handles errors. -Method: `distinct_count(relation, column = nil, batch: true, start: nil, finish: nil)` +Method: `distinct_count(relation, column = nil, batch: true, batch_size: nil, start: nil, finish: nil)` Arguments: - `relation` the ActiveRecord_Relation to perform the count - `column` the column to perform the distinct count, by default is the primary key - `batch`: default `true` in order to use batch counting +- `batch_size`: if none set it will use default value 10000 from `Gitlab::Database::BatchCounter` - `start`: custom start of the batch counting in order to avoid complex min calculations - `end`: custom end of the batch counting in order to avoid complex min calculations @@ -212,14 +213,48 @@ Arguments: - `counter`: a counter from `Gitlab::UsageDataCounters`, that has `fallback_totals` method implemented - or a `block`: which is evaluated +#### Ordinary Redis Counters + +Examples of implementation: + +- Using Redis methods [`INCR`](https://redis.io/commands/incr), [`GET`](https://redis.io/commands/get), and [`Gitlab::UsageDataCounters::WikiPageCounter`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/wiki_page_counter.rb) +- Using Redis methods [`HINCRBY`](https://redis.io/commands/hincrby), [`HGETALL`](https://redis.io/commands/hgetall), and [`Gitlab::UsageCounters::PodLogs`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_counters/pod_logs.rb) + +#### Redis HLL Counters + +With `Gitlab::Redis::HLL` we have available data structures used to count unique values. + +Implemented using Redis methods [PFADD](https://redis.io/commands/pfadd) and [PFCOUNT](https://redis.io/commands/pfcount). + +Recommendations: + +- Key should expire in 29 days. +- If possible, data granularity should be a week. For example a key could be composed from the metric's name and week of the year, `2020-33-{metric_name}`. +- Use a [feature flag](../../operations/feature_flags.md) in order to have a control over the impact when adding new metrics. +- If possible, data granularity should be week, for example a key could be composed from metric name and week of the year, 2020-33-{metric_name} +- Use a [feature flag](../../operations/feature_flags.md) in order to have a control over the impact when adding new metrics + +Examples of implementation: + +- [`Gitlab::UsageDataCounters::TrackUniqueActions`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/track_unique_actions.rb) +- [`Gitlab::Analytics::UniqueVisits`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/analytics/unique_visits.rb) + Example of usage: ```ruby +# Redis Counters redis_usage_data(Gitlab::UsageDataCounters::WikiPageCounter) redis_usage_data { ::Gitlab::UsageCounters::PodLogs.usage_totals[:total] } -``` -Note that Redis counters are in the [process of being deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/216330) and you should instead try to use Snowplow events instead. We're in the process of building [self-managed event tracking](https://gitlab.com/gitlab-org/telemetry/-/issues/373) and once this is available, we will convert all Redis counters into Snowplow events. +# Redis HLL counter +counter = Gitlab::UsageDataCounters::TrackUniqueActions +redis_usage_data do + counter.count_unique_events( + event_action: Gitlab::UsageDataCounters::TrackUniqueActions::PUSH_ACTION, + date_from: time_period[:created_at].first, + date_to: time_period[:created_at].last + ) +``` ### Alternative Counters @@ -313,14 +348,16 @@ In order to have an understanding of the query's execution we add in the MR desc We also use `#database-lab` and [explain.depesz.com](https://explain.depesz.com/). For more details, see the [database review guide](../database_review.md#preparation-when-adding-or-modifying-queries). -Examples of query optimization work: +#### Optimization recommendations and examples -- [Example 1](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/26445) -- [Example 2](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/26871) +- Use specialized indexes [example 1](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/26871), [example 2](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/26445). +- Use defined `start` and `finish`, and simple queries, because these values can be memoized and reused, [example](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/37155). +- Avoid joins and write the queries as simply as possible, [example](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/36316). +- Set a custom `batch_size` for `distinct_count`, [example](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/38000). ### 4. Add the metric definition -When adding, changing, or updating metrics, please update the [Usage Statistics definition table](#usage-statistics-definitions). +When adding, changing, or updating metrics, please update the [Event Dictionary's **Usage Ping** table](event_dictionary.md). ### 5. Add new metric to Versions Application @@ -340,6 +377,10 @@ Ensure you comply with the [Changelog entries guide](../changelog.md). On GitLab.com, we have DangerBot setup to monitor Telemetry related files and DangerBot will recommend a Telemetry review. Mention `@gitlab-org/growth/telemetry/engineers` in your MR for a review. +### 9. Verify your metric + +On GitLab.com, the Product Analytics team regularly monitors Usage Ping. They may alert you that your metrics need further optimization to run quicker and with greater success. You may also use the [Usage Ping QA dashboard](https://app.periscopedata.com/app/gitlab/632033/Usage-Ping-QA) to check how well your metric performs. The dashboard allows filtering by GitLab version, by "Self-managed" & "Saas" and shows you how many failures have occurred for each metric. Whenever you notice a high failure rate, you may re-optimize your metric. + ### Optional: Test Prometheus based Usage Ping If the data submitted includes metrics [queried from Prometheus](#prometheus-queries) that you would like to inspect and verify, @@ -386,358 +427,6 @@ with any of the other services that are running. That is not how node metrics ar always runs as a process alongside other GitLab components on any given node. From Usage Ping's perspective none of the node data would therefore appear to be associated to any of the services running, since they all appear to be running on different hosts. To alleviate this problem, the `node_exporter` in GCK was arbitrarily "assigned" to the `web` service, meaning only for this service `node_*` metrics will appear in Usage Ping. -## Usage Statistics definitions - -| Statistic | Section | Stage | Tier | Edition | Description | -| --------------------------------------------------------- | ------------------------------------ | ------------- | ---------------- | ------- | -------------------------------------------------------------------------- | -| `uuid` | | | | | | -| `hostname` | | | | | | -| `version` | | | | | | -| `installation_type` | | | | | | -| `active_user_count` | | | | | | -| `recorded_at` | | | | | | -| `recording_ce_finished_at` | | | | CE+EE | When the core features were computed | -| `recording_ee_finished_at` | | | | EE | When the EE-specific features were computed | -| `edition` | | | | | | -| `license_md5` | | | | | | -| `license_id` | | | | | | -| `historical_max_users` | | | | | | -| `Name` | `licensee` | | | | | -| `Email` | `licensee` | | | | | -| `Company` | `licensee` | | | | | -| `license_user_count` | | | | | | -| `license_starts_at` | | | | | | -| `license_expires_at` | | | | | | -| `license_plan` | | | | | | -| `license_trial` | | | | | | -| `assignee_lists` | `counts` | | | | | -| `boards` | `counts` | | | | | -| `ci_builds` | `counts` | `verify` | | | Unique builds in project | -| `ci_internal_pipelines` | `counts` | `verify` | | | Total pipelines in GitLab repositories | -| `ci_external_pipelines` | `counts` | `verify` | | | Total pipelines in external repositories | -| `ci_pipeline_config_auto_devops` | `counts` | `verify` | | | Total pipelines from an Auto DevOps template | -| `ci_pipeline_config_repository` | `counts` | `verify` | | | Total Pipelines from templates in repository | -| `ci_runners` | `counts` | `verify` | | | Total configured Runners in project | -| `ci_triggers` | `counts` | `verify` | | | Total configured Triggers in project | -| `ci_pipeline_schedules` | `counts` | `verify` | | | Pipeline schedules in GitLab | -| `auto_devops_enabled` | `counts` | `configure` | | | Projects with Auto DevOps template enabled | -| `auto_devops_disabled` | `counts` | `configure` | | | Projects with Auto DevOps template disabled | -| `deploy_keys` | `counts` | | | | | -| `deployments` | `counts` | `release` | | | Total deployments | -| `deployments` | `counts_monthly` | `release` | | | Total deployments last 28 days | -| `dast_jobs` | `counts` | | | | | -| `successful_deployments` | `counts` | `release` | | | Total successful deployments | -| `successful_deployments` | `counts_monthly` | `release` | | | Total successful deployments last 28 days | -| `failed_deployments` | `counts` | `release` | | | Total failed deployments | -| `failed_deployments` | `counts_monthly` | `release` | | | Total failed deployments last 28 days | -| `environments` | `counts` | `release` | | | Total available and stopped environments | -| `clusters` | `counts` | `configure` | | | Total GitLab Managed clusters both enabled and disabled | -| `clusters_enabled` | `counts` | `configure` | | | Total GitLab Managed clusters currently enabled | -| `project_clusters_enabled` | `counts` | `configure` | | | Total GitLab Managed clusters attached to projects | -| `group_clusters_enabled` | `counts` | `configure` | | | Total GitLab Managed clusters attached to groups | -| `instance_clusters_enabled` | `counts` | `configure` | | | Total GitLab Managed clusters attached to the instance | -| `clusters_disabled` | `counts` | `configure` | | | Total GitLab Managed disabled clusters | -| `project_clusters_disabled` | `counts` | `configure` | | | Total GitLab Managed disabled clusters previously attached to projects | -| `group_clusters_disabled` | `counts` | `configure` | | | Total GitLab Managed disabled clusters previously attached to groups | -| `instance_clusters_disabled` | `counts` | `configure` | | | Total GitLab Managed disabled clusters previously attached to the instance | -| `clusters_platforms_eks` | `counts` | `configure` | | | Total GitLab Managed clusters provisioned with GitLab on AWS EKS | -| `clusters_platforms_gke` | `counts` | `configure` | | | Total GitLab Managed clusters provisioned with GitLab on GCE GKE | -| `clusters_platforms_user` | `counts` | `configure` | | | Total GitLab Managed clusters that are user provisioned | -| `clusters_applications_helm` | `counts` | `configure` | | | Total GitLab Managed clusters with Helm enabled | -| `clusters_applications_ingress` | `counts` | `configure` | | | Total GitLab Managed clusters with Ingress enabled | -| `clusters_applications_cert_managers` | `counts` | `configure` | | | Total GitLab Managed clusters with Cert Manager enabled | -| `clusters_applications_crossplane` | `counts` | `configure` | | | Total GitLab Managed clusters with Crossplane enabled | -| `clusters_applications_prometheus` | `counts` | `configure` | | | Total GitLab Managed clusters with Prometheus enabled | -| `clusters_applications_runner` | `counts` | `configure` | | | Total GitLab Managed clusters with Runner enabled | -| `clusters_applications_knative` | `counts` | `configure` | | | Total GitLab Managed clusters with Knative enabled | -| `clusters_applications_elastic_stack` | `counts` | `configure` | | | Total GitLab Managed clusters with Elastic Stack enabled | -| `clusters_applications_cilium` | `counts` | `configure` | | | Total GitLab Managed clusters with Cilium enabled | -| `clusters_management_project` | `counts` | `configure` | | | Total GitLab Managed clusters with defined cluster management project | -| `in_review_folder` | `counts` | | | | | -| `grafana_integrated_projects` | `counts` | | | | | -| `groups` | `counts` | | | | | -| `issues` | `counts` | | | | | -| `issues_created_from_gitlab_error_tracking_ui` | `counts` | `monitor` | | | | -| `issues_with_associated_zoom_link` | `counts` | `monitor` | | | | -| `issues_using_zoom_quick_actions` | `counts` | `monitor` | | | | -| `issues_with_embedded_grafana_charts_approx` | `counts` | `monitor` | | | | -| `issues_with_health_status` | `counts` | | | | | -| `keys` | `counts` | | | | | -| `label_lists` | `counts` | | | | | -| `lfs_objects` | `counts` | | | | | -| `milestone_lists` | `counts` | | | | | -| `milestones` | `counts` | | | | | -| `pages_domains` | `counts` | `release` | | | Total GitLab Pages domains | -| `pool_repositories` | `counts` | | | | | -| `projects` | `counts` | | | | | -| `projects_imported_from_github` | `counts` | | | | | -| `projects_with_repositories_enabled` | `counts` | | | | | -| `projects_with_error_tracking_enabled` | `counts` | `monitor` | | | | -| `protected_branches` | `counts` | | | | | -| `releases` | `counts` | `release` | | | Unique release tags | -| `remote_mirrors` | `counts` | | | | | -| `requirements_created` | `counts` | | | | | -| `snippets` | `counts` | 'create' | | CE+EE | | -| `snippets` | `counts_monthly` | 'create' | | CE+EE | | -| `personal_snippets` | `counts` | 'create' | | CE+EE | | -| `personal_snippets` | `counts_monthly` | 'create' | | CE+EE | | -| `project_snippets` | `counts` | 'create' | | CE+EE | | -| `project_snippets` | `counts_monthly` | 'create' | | CE+EE | | -| `suggestions` | `counts` | | | | | -| `todos` | `counts` | | | | | -| `uploads` | `counts` | | | | | -| `web_hooks` | `counts` | | | | | -| `projects_alerts_active` | `counts` | | | | | -| `projects_asana_active` | `counts` | | | | | -| `projects_assembla_active` | `counts` | | | | | -| `projects_bamboo_active` | `counts` | | | | | -| `projects_bugzilla_active` | `counts` | | | | | -| `projects_buildkite_active` | `counts` | | | | | -| `projects_campfire_active` | `counts` | | | | | -| `projects_custom_issue_tracker_active` | `counts` | | | | | -| `projects_discord_active` | `counts` | | | | | -| `projects_drone_ci_active` | `counts` | | | | | -| `projects_emails_on_push_active` | `counts` | | | | | -| `projects_external_wiki_active` | `counts` | | | | | -| `projects_flowdock_active` | `counts` | | | | | -| `projects_github_active` | `counts` | | | | | -| `projects_hangouts_chat_active` | `counts` | | | | | -| `projects_hipchat_active` | `counts` | | | | | -| `projects_irker_active` | `counts` | | | | | -| `projects_jenkins_active` | `counts` | | | | | -| `projects_jira_active` | `counts` | | | | | -| `projects_mattermost_active` | `counts` | | | | | -| `projects_mattermost_slash_commands_active` | `counts` | | | | | -| `projects_microsoft_teams_active` | `counts` | | | | | -| `projects_packagist_active` | `counts` | | | | | -| `projects_pipelines_email_active` | `counts` | | | | | -| `projects_pivotaltracker_active` | `counts` | | | | | -| `projects_prometheus_active` | `counts` | | | | | -| `projects_pushover_active` | `counts` | | | | | -| `projects_redmine_active` | `counts` | | | | | -| `projects_slack_active` | `counts` | | | | | -| `projects_slack_slash_commands_active` | `counts` | | | | | -| `projects_teamcity_active` | `counts` | | | | | -| `projects_unify_circuit_active` | `counts` | | | | | -| `projects_webex_teams_active` | `counts` | | | | | -| `projects_youtrack_active` | `counts` | | | | | -| `projects_jira_server_active` | `counts` | | | | | -| `projects_jira_cloud_active` | `counts` | | | | | -| `projects_jira_dvcs_cloud_active` | `counts` | | | | | -| `projects_jira_dvcs_server_active` | `counts` | | | | | -| `projects_jira_issuelist_active` | `counts` | `create` | | EE | Total Jira Issue feature enabled | -| `labels` | `counts` | | | | | -| `merge_requests` | `counts` | | | | | -| `merge_requests_users` | `counts` | | | | | -| `notes` | `counts` | | | | | -| `wiki_pages_create` | `counts` | | | | | -| `wiki_pages_update` | `counts` | | | | | -| `wiki_pages_delete` | `counts` | | | | | -| `web_ide_commits` | `counts` | | | | | -| `web_ide_views` | `counts` | | | | | -| `web_ide_merge_requests` | `counts` | | | | | -| `web_ide_previews` | `counts` | | | | | -| `snippet_comment` | `counts` | | | | | -| `commit_comment` | `counts` | | | | | -| `merge_request_comment` | `counts` | | | | | -| `snippet_create` | `counts` | | | | | -| `snippet_update` | `counts` | | | | | -| `navbar_searches` | `counts` | | | | | -| `cycle_analytics_views` | `counts` | | | | | -| `productivity_analytics_views` | `counts` | | | | | -| `source_code_pushes` | `counts` | | | | | -| `merge_request_create` | `counts` | | | | | -| `design_management_designs_create` | `counts` | | | | | -| `design_management_designs_update` | `counts` | | | | | -| `design_management_designs_delete` | `counts` | | | | | -| `licenses_list_views` | `counts` | | | | | -| `user_preferences_group_overview_details` | `counts` | | | | | -| `user_preferences_group_overview_security_dashboard` | `counts` | | | | | -| `ingress_modsecurity_logging` | `counts` | | | | | -| `ingress_modsecurity_blocking` | `counts` | | | | | -| `ingress_modsecurity_disabled` | `counts` | | | | | -| `ingress_modsecurity_not_installed` | `counts` | | | | | -| `dependency_list_usages_total` | `counts` | | | | | -| `epics` | `counts` | | | | | -| `feature_flags` | `counts` | | | | | -| `geo_nodes` | `counts` | `geo` | | | Number of sites in a Geo deployment | -| `geo_event_log_max_id` | `counts` | `geo` | | | Number of replication events on a Geo primary | -| `incident_issues` | `counts` | `monitor` | | | Issues created by the alert bot | -| `alert_bot_incident_issues` | `counts` | `monitor` | | | Issues created by the alert bot | -| `incident_labeled_issues` | `counts` | `monitor` | | | Issues with the incident label | -| `issues_created_gitlab_alerts` | `counts` | `monitor` | | | Issues created from alerts by non-alert bot users | -| `issues_created_manually_from_alerts` | `counts` | `monitor` | | | Issues created from alerts by non-alert bot users | -| `issues_created_from_alerts` | `counts` | `monitor` | | | Issues created from Prometheus and alert management alerts | -| `ldap_group_links` | `counts` | | | | | -| `ldap_keys` | `counts` | | | | | -| `ldap_users` | `counts` | | | | | -| `pod_logs_usages_total` | `counts` | | | | | -| `projects_enforcing_code_owner_approval` | `counts` | | | | | -| `projects_mirrored_with_pipelines_enabled` | `counts` | `release` | | | Projects with repository mirroring enabled | -| `projects_reporting_ci_cd_back_to_github` | `counts` | `verify` | | | Projects with a GitHub service pipeline enabled | -| `projects_with_packages` | `counts` | `package` | | | Projects with package registry configured | -| `projects_with_prometheus_alerts` | `counts` | `monitor` | | | Projects with Prometheus alerting enabled | -| `projects_with_tracing_enabled` | `counts` | `monitor` | | | Projects with tracing enabled | -| `projects_with_alerts_service_enabled` | `counts` | `monitor` | | | Projects with alerting service enabled | -| `template_repositories` | `counts` | | | | | -| `container_scanning_jobs` | `counts` | | | | | -| `dependency_scanning_jobs` | `counts` | | | | | -| `license_management_jobs` | `counts` | | | | | -| `sast_jobs` | `counts` | | | | | -| `status_page_projects` | `counts` | `monitor` | | | Projects with status page enabled | -| `status_page_issues` | `counts` | `monitor` | | | Issues published to a Status Page | -| `status_page_incident_publishes` | `counts` | `monitor` | | | Cumulative count of usages of publish operation | -| `status_page_incident_unpublishes` | `counts` | `monitor` | | | Cumulative count of usages of unpublish operation | -| `epics_deepest_relationship_level` | `counts` | | | | | -| `operations_dashboard_default_dashboard` | `counts` | `monitor` | | | Active users with enabled operations dashboard | -| `operations_dashboard_users_with_projects_added` | `counts` | `monitor` | | | Active users with projects on operations dashboard | -| `container_registry_enabled` | | | | | | -| `dependency_proxy_enabled` | | | | | | -| `gitlab_shared_runners_enabled` | | | | | | -| `gravatar_enabled` | | | | | | -| `ldap_enabled` | | | | | | -| `mattermost_enabled` | | | | | | -| `omniauth_enabled` | | | | | | -| `prometheus_enabled` | | | | | Whether the bundled Prometheus is enabled | -| `prometheus_metrics_enabled` | | | | | | -| `reply_by_email_enabled` | | | | | | -| `average` | `avg_cycle_analytics - code` | | | | | -| `sd` | `avg_cycle_analytics - code` | | | | | -| `missing` | `avg_cycle_analytics - code` | | | | | -| `average` | `avg_cycle_analytics - test` | | | | | -| `sd` | `avg_cycle_analytics - test` | | | | | -| `missing` | `avg_cycle_analytics - test` | | | | | -| `average` | `avg_cycle_analytics - review` | | | | | -| `sd` | `avg_cycle_analytics - review` | | | | | -| `missing` | `avg_cycle_analytics - review` | | | | | -| `average` | `avg_cycle_analytics - staging` | | | | | -| `sd` | `avg_cycle_analytics - staging` | | | | | -| `missing` | `avg_cycle_analytics - staging` | | | | | -| `average` | `avg_cycle_analytics - production` | | | | | -| `sd` | `avg_cycle_analytics - production` | | | | | -| `missing` | `avg_cycle_analytics - production` | | | | | -| `total` | `avg_cycle_analytics` | | | | | -| `g_analytics_contribution` | `analytics_unique_visits` | `manage` | | | Visits to /groups/:group/-/contribution_analytics | -| `g_analytics_insights` | `analytics_unique_visits` | `manage` | | | Visits to /groups/:group/-/insights | -| `g_analytics_issues` | `analytics_unique_visits` | `manage` | | | Visits to /groups/:group/-/issues_analytics | -| `g_analytics_productivity` | `analytics_unique_visits` | `manage` | | | Visits to /groups/:group/-/analytics/productivity_analytics | -| `g_analytics_valuestream` | `analytics_unique_visits` | `manage` | | | Visits to /groups/:group/-/analytics/value_stream_analytics | -| `p_analytics_pipelines` | `analytics_unique_visits` | `manage` | | | Visits to /:group/:project/pipelines/charts | -| `p_analytics_code_reviews` | `analytics_unique_visits` | `manage` | | | Visits to /:group/:project/-/analytics/code_reviews | -| `p_analytics_valuestream` | `analytics_unique_visits` | `manage` | | | Visits to /:group/:project/-/value_stream_analytics | -| `p_analytics_insights` | `analytics_unique_visits` | `manage` | | | Visits to /:group/:project/insights | -| `p_analytics_issues` | `analytics_unique_visits` | `manage` | | | Visits to /:group/:project/-/analytics/issues_analytics | -| `p_analytics_repo` | `analytics_unique_visits` | `manage` | | | Visits to /:group/:project/-/graphs/master/charts | -| `u_analytics_todos` | `analytics_unique_visits` | `manage` | | | Visits to /dashboard/todos | -| `i_analytics_cohorts` | `analytics_unique_visits` | `manage` | | | Visits to /-/instance_statistics/cohorts | -| `i_analytics_dev_ops_score` | `analytics_unique_visits` | `manage` | | | Visits to /-/instance_statistics/dev_ops_score | -| `analytics_unique_visits_for_any_target` | `analytics_unique_visits` | `manage` | | | Visits to any of the pages listed above | -| `clusters_applications_cert_managers` | `usage_activity_by_stage` | `configure` | | CE+EE | Unique clusters with certificate managers enabled | -| `clusters_applications_helm` | `usage_activity_by_stage` | `configure` | | CE+EE | Unique clusters with Helm enabled | -| `clusters_applications_ingress` | `usage_activity_by_stage` | `configure` | | CE+EE | Unique clusters with Ingress enabled | -| `clusters_applications_knative` | `usage_activity_by_stage` | `configure` | | CE+EE | Unique clusters with Knative enabled | -| `clusters_management_project` | `usage_activity_by_stage` | `configure` | | CE+EE | Unique clusters with project management enabled | -| `clusters_disabled` | `usage_activity_by_stage` | `configure` | | CE+EE | Total non-"GitLab Managed clusters" | -| `clusters_enabled` | `usage_activity_by_stage` | `configure` | | CE+EE | Total GitLab Managed clusters | -| `clusters_platforms_gke` | `usage_activity_by_stage` | `configure` | | CE+EE | Unique clusters with Google Cloud installed | -| `clusters_platforms_eks` | `usage_activity_by_stage` | `configure` | | CE+EE | Unique clusters with AWS installed | -| `clusters_platforms_user` | `usage_activity_by_stage` | `configure` | | CE+EE | Unique clusters that are user provided | -| `instance_clusters_disabled` | `usage_activity_by_stage` | `configure` | | CE+EE | Unique clusters disabled on instance | -| `instance_clusters_enabled` | `usage_activity_by_stage` | `configure` | | CE+EE | Unique clusters enabled on instance | -| `group_clusters_disabled` | `usage_activity_by_stage` | `configure` | | CE+EE | Unique clusters disabled on group | -| `group_clusters_enabled` | `usage_activity_by_stage` | `configure` | | CE+EE | Unique clusters enabled on group | -| `project_clusters_disabled` | `usage_activity_by_stage` | `configure` | | CE+EE | Unique clusters disabled on project | -| `project_clusters_enabled` | `usage_activity_by_stage` | `configure` | | CE+EE | Unique clusters enabled on project | -| `projects_slack_notifications_active` | `usage_activity_by_stage` | `configure` | | EE | Unique projects with Slack service enabled | -| `projects_slack_slash_active` | `usage_activity_by_stage` | `configure` | | EE | Unique projects with Slack '/' commands enabled | -| `projects_with_prometheus_alerts` | `usage_activity_by_stage` | `configure` | | EE | Projects with Prometheus enabled and no alerts | -| `deploy_keys` | `usage_activity_by_stage` | `create` | | CE+EE | | -| `keys` | `usage_activity_by_stage` | `create` | | CE+EE | | -| `merge_requests` | `usage_activity_by_stage` | `create` | | CE+EE | | -| `projects_with_disable_overriding_approvers_per_merge_request` | `usage_activity_by_stage` | `create` | | CE+EE | | -| `projects_without_disable_overriding_approvers_per_merge_request` | `usage_activity_by_stage` | `create` | | CE+EE | | -| `remote_mirrors` | `usage_activity_by_stage` | `create` | | CE+EE | | -| `snippets` | `usage_activity_by_stage` | `create` | | CE+EE | | -| `merge_requests_users` | `usage_activity_by_stage_monthly` | `create` | | CE+EE | Unique count of users who used a merge request | -| `action_monthly_active_users_project_repo` | `usage_activity_by_stage_monthly` | `create` | | CE+EE | Unique count of users who pushed to a project repo | -| `action_monthly_active_users_design_management` | `usage_activity_by_stage_monthly` | `create` | | CE+EE | Unique count of users who interacted with the design system management | -| `action_monthly_active_users_wiki_repo` | `usage_activity_by_stage_monthly` | `create` | | CE+EE | Unique count of users who created or updated a wiki repo | -| `projects_enforcing_code_owner_approval` | `usage_activity_by_stage` | `create` | | EE | | -| `merge_requests_with_optional_codeowners` | `usage_activity_by_stage` | `create` | | EE | | -| `merge_requests_with_required_codeowners` | `usage_activity_by_stage` | `create` | | EE | | -| `projects_imported_from_github` | `usage_activity_by_stage` | `create` | | EE | | -| `projects_with_repositories_enabled` | `usage_activity_by_stage` | `create` | | EE | | -| `protected_branches` | `usage_activity_by_stage` | `create` | | EE | | -| `suggestions` | `usage_activity_by_stage` | `create` | | EE | | -| `approval_project_rules` | `usage_activity_by_stage` | `create` | | EE | Number of project approval rules | -| `approval_project_rules_with_target_branch` | `usage_activity_by_stage` | `create` | | EE | Number of project approval rules with not default target branch | -| `merge_requests_with_added_rules` | `usage_activity_by_stage` | `create` | | EE | Merge Requests with added rules | -| `clusters` | `usage_activity_by_stage` | `monitor` | | CE+EE | | -| `clusters_applications_prometheus` | `usage_activity_by_stage` | `monitor` | | CE+EE | | -| `operations_dashboard_default_dashboard` | `usage_activity_by_stage` | `monitor` | | CE+EE | | -| `operations_dashboard_users_with_projects_added` | `usage_activity_by_stage` | `monitor` | | EE | | -| `projects_prometheus_active` | `usage_activity_by_stage` | `monitor` | | EE | | -| `projects_with_error_tracking_enabled` | `usage_activity_by_stage` | `monitor` | | EE | | -| `projects_with_tracing_enabled` | `usage_activity_by_stage` | `monitor` | | EE | | -| `events` | `usage_activity_by_stage` | `manage` | | CE+EE | | -| `groups` | `usage_activity_by_stage` | `manage` | | CE+EE | | -| `users_created_at` | `usage_activity_by_stage` | `manage` | | CE+EE | | -| `omniauth_providers` | `usage_activity_by_stage` | `manage` | | CE+EE | | -| `ldap_keys` | `usage_activity_by_stage` | `manage` | | EE | | -| `ldap_users` | `usage_activity_by_stage` | `manage` | | EE | | -| `value_stream_management_customized_group_stages` | `usage_activity_by_stage` | `manage` | | EE | | -| `projects_with_compliance_framework` | `usage_activity_by_stage` | `manage` | | EE | | -| `ldap_servers` | `usage_activity_by_stage` | `manage` | | EE | | -| `ldap_group_sync_enabled` | `usage_activity_by_stage` | `manage` | | EE | | -| `ldap_admin_sync_enabled` | `usage_activity_by_stage` | `manage` | | EE | | -| `group_saml_enabled` | `usage_activity_by_stage` | `manage` | | EE | | -| `issues` | `usage_activity_by_stage` | `plan` | | CE+EE | | -| `notes` | `usage_activity_by_stage` | `plan` | | CE+EE | | -| `projects` | `usage_activity_by_stage` | `plan` | | CE+EE | | -| `todos` | `usage_activity_by_stage` | `plan` | | CE+EE | | -| `assignee_lists` | `usage_activity_by_stage` | `plan` | | EE | | -| `epics` | `usage_activity_by_stage` | `plan` | | EE | | -| `label_lists` | `usage_activity_by_stage` | `plan` | | EE | | -| `milestone_lists` | `usage_activity_by_stage` | `plan` | | EE | | -| `projects_jira_active` | `usage_activity_by_stage` | `plan` | | EE | | -| `projects_jira_dvcs_server_active` | `usage_activity_by_stage` | `plan` | | EE | | -| `projects_jira_dvcs_server_active` | `usage_activity_by_stage` | `plan` | | EE | | -| `service_desk_enabled_projects` | `usage_activity_by_stage` | `plan` | | CE+EE | | -| `service_desk_issues` | `usage_activity_by_stage` | `plan` | | CE+EE | | -| `deployments` | `usage_activity_by_stage` | `release` | | CE+EE | Total deployments | -| `failed_deployments` | `usage_activity_by_stage` | `release` | | CE+EE | Total failed deployments | -| `projects_mirrored_with_pipelines_enabled` | `usage_activity_by_stage` | `release` | | EE | Projects with repository mirroring enabled | -| `releases` | `usage_activity_by_stage` | `release` | | CE+EE | Unique release tags in project | -| `successful_deployments` | `usage_activity_by_stage` | `release` | | CE+EE | Total successful deployments | -| `user_preferences_group_overview_security_dashboard` | `usage_activity_by_stage` | `secure` | | | | -| `ci_builds` | `usage_activity_by_stage` | `verify` | | CE+EE | Unique builds in project | -| `ci_external_pipelines` | `usage_activity_by_stage` | `verify` | | CE+EE | Total pipelines in external repositories | -| `ci_internal_pipelines` | `usage_activity_by_stage` | `verify` | | CE+EE | Total pipelines in GitLab repositories | -| `ci_pipeline_config_auto_devops` | `usage_activity_by_stage` | `verify` | | CE+EE | Total pipelines from an Auto DevOps template | -| `ci_pipeline_config_repository` | `usage_activity_by_stage` | `verify` | | CE+EE | Pipelines from templates in repository | -| `ci_pipeline_schedules` | `usage_activity_by_stage` | `verify` | | CE+EE | Pipeline schedules in GitLab | -| `ci_pipelines` | `usage_activity_by_stage` | `verify` | | CE+EE | Total pipelines | -| `ci_triggers` | `usage_activity_by_stage` | `verify` | | CE+EE | Triggers enabled | -| `clusters_applications_runner` | `usage_activity_by_stage` | `verify` | | CE+EE | Unique clusters with Runner enabled | -| `projects_reporting_ci_cd_back_to_github` | `usage_activity_by_stage` | `verify` | | EE | Unique projects with a GitHub pipeline enabled | -| `merge_requests_users` | `usage_activity_by_stage_monthly` | `create` | | | Unique count of users who used a merge request | -| `duration_s` | `topology` | `enablement` | | | Time it took to collect topology data | -| `application_requests_per_hour` | `topology` | `enablement` | | | Number of requests to the web application per hour | -| `failures` | `topology` | `enablement` | | | Contains information about failed queries | -| `nodes` | `topology` | `enablement` | | | The list of server nodes on which GitLab components are running | -| `node_memory_total_bytes` | `topology > nodes` | `enablement` | | | The total available memory of this node | -| `node_cpus` | `topology > nodes` | `enablement` | | | The number of CPU cores of this node | -| `node_uname_info` | `topology > nodes` | `enablement` | | | The basic hardware architecture and OS release information on this node | -| `node_services` | `topology > nodes` | `enablement` | | | The list of GitLab services running on this node | -| `name` | `topology > nodes > node_services` | `enablement` | | | The name of the GitLab service running on this node | -| `process_count` | `topology > nodes > node_services` | `enablement` | | | The number of processes running for this service | -| `process_memory_rss` | `topology > nodes > node_services` | `enablement` | | | The average Resident Set Size of a service process | -| `process_memory_uss` | `topology > nodes > node_services` | `enablement` | | | The average Unique Set Size of a service process | -| `process_memory_pss` | `topology > nodes > node_services` | `enablement` | | | The average Proportional Set Size of a service process | -| `server` | `topology > nodes > node_services` | `enablement` | | | The type of web server used (Unicorn or Puma) | -| `network_policy_forwards` | `counts` | `defend` | | EE | Cumulative count of forwarded packets by Container Network | -| `network_policy_drops` | `counts` | `defend` | | EE | Cumulative count of dropped packets by Container Network | - ## Example Usage Ping payload The following is example content of the Usage Ping payload. @@ -811,13 +500,14 @@ The following is example content of the Usage Ping payload. "enabled": true, "version": "1.17.0" }, + "container_registry_server": { + "vendor": "gitlab", + "version": "2.9.1-gitlab" + }, "database": { "adapter": "postgresql", "version": "9.6.15" }, - "app_server": { - "type": "console" - }, "avg_cycle_analytics": { "issue": { "average": 999, @@ -941,7 +631,9 @@ The following is example content of the Usage Ping payload. "nodes": [ { "node_memory_total_bytes": 33269903360, + "node_memory_utilization": 0.35, "node_cpus": 16, + "node_cpu_utilization": 0.2, "node_uname_info": { "machine": "x86_64", "sysname": "Linux", |